Building facade extraction is the primary step in the recognition process in outdoor scenes. It is also achallenging task since each building can be viewed from different angles or under different lighting conditions. Inoutdoor imagery, regions, such as sky, trees, pavement cause interference for a successful building facade recognition.In this paper we propose a knowledge based approach to automatically segment out the whole facade or majorparts of the facade from outdoor scene. The found building regions are then subjected to recognition process. Thesystem is composed of two modules: segmentation of building facades region module and facade recognition module.In the facade segmentation module, color processing and objects position coordinates are used. In the facaderecognition module, Chamfer metrics are applied. In real time recognition scenario, the image with a building isfirst analyzed in order to extract the facade region, which is then compared to a database with feature descriptors inorder to find a match. The results show that the recognition rate is dependent on a precision of building extractionpart, which in turn, depends on a homogeneity of colors of facades.