Rooftop photovoltaic panels (RPVs) are being increasingly used in urban areas as a promising means of achieving energy sustainability. Determining proper layouts of RPVs that make the best use of rooftop areas is of importance as they have a considerable impact on the RPVs performance in efficiently producing energy. In this study, a new spatial methodology for automatically determining the proper layouts of RPVs is proposed. It aims to both extract planar rooftop segments and identify feasible layouts with the highest number of RPVs in highly irradiated areas. It leverages digital surface models (DSMs) to consider roof shapes and occlusions in placing RPVs. The innovations of the work are twofold: (a) a new method for plane segmentation, and (b) a new method for optimally placing RPVs based on metaheuristic optimization, which best utilizes the limited rooftop areas. The proposed methodology is evaluated on two test sites that differ in urban morphology, building size, and spatial resolution. The results show that the plane segmentation method can accurately extract planar segments, achieving 88.7% and 99.5% precision in the test sites. In addition, the results indicate that complex rooftops are adequately handled for placing RPVs, and overestimation of solar energy potential is avoided if detailed analysis based on panel placement is employed.