Building Attributes Extraction

Chen, Q., See, L., and Crooks, A. T. (2026). Evaluating the Feasibility of ChatGPT for Mapping Building Attributes. In K. Janowicz, R. Zhu, G. Mai, S. Gao, Y. Hu, Z. Wang, L. Cai, and L. Bennett (Eds.), Geography According to Foundation Models (pp. 107-120). IOS Press.

Introduction

As the spatial resolution of climate models becomes finer due to increases in computing power and storage, there is a need to represent urban areas and local-scale urban processes with greater detail than ever before. Although urban areas account for only 3% of the Earth’s surface, they have impacts in the form of urban heat islands, increased runoff from impervious surfaces and greater air pollution, while collectively accounting for around 70% of global CO₂ emissions. Moreover, with 55% of the world’s population currently living in urban areas and this number expected to increase to 68% by 2050, understanding and managing these urban impacts becomes even more critical.

At the city scale, urban climate models are now being used for different applications such as determining the impact of climate change in the future and modeling the effects of climate change adaptation and mitigation measures. However, to represent urban areas in these applications, much more information about cities is needed that captures microclimatic conditions, including building form and building function. By form, we mean the shape and configuration of the city, captured through building attributes such as height and building materials as well as the spatial layout of buildings and other infrastructure. In contrast, function refers to the activities that take place in a city, which can be captured through building function (e.g., residential or commercial use).

To infer some of the form parameters, the Local Climate Zone (LCZ) classification was developed. Using Landsat imagery, cities are classified into 10 urban types and 7 natural types, which are defined by the height of the buildings and vegetation as well as building density. This approach has been applied globally to aid the parameterization of urban areas in climate models. However, LCZs provide only some of the needed information. Another form parameter of importance is the age of the building, as this has been shown to be a key factor in modeling the building energy consumption. Using various urban morphological parameters as inputs, research into predicting building age with machine learning has yielded accuracies of up to 77% for buildings in Nottingham, UK, and mean absolute errors of between 15 and 20 years for buildings in the Netherlands and Spain.

In the same vein, knowing whether a building is residential, commercial, or mixed-use can also provide information on energy usage and carbon emissions, yet such information is not readily available at the city scale. For example, Points of Interest (POIs) in OpenStreetMap provide one source of building function, although there are issues with both completeness and consistency in tagging across cities, while another study has used POIs in combination with taxi trajectories and WeChat user location data to infer building function in Guangzhou, China. However, a more comprehensive and harmonized data set on building function is still not available.

Street-level imagery such as Google Street View and Mapillary provide a new, valuable source of building images from which form and function can be extracted. A number of studies have appeared that have used computer vision and deep learning approaches—including convolutional neural networks (CNNs)—to determine building function, height, and age. For example, pre-trained CNNs and Google Street View imagery were used to predict different building types, including residential and commercial in Calgary, Boston, and Toronto, with F1 scores ranging from 0.65 to 0.82 for residential and lower scores for commercial. Different approaches have been used to estimate building height, but in a comparison using the same data set, a pre-trained CNN using Google Street View imagery for Toronto outperformed other approaches including geometry-based methods, with a mean absolute error of 1.2 m. Similarly, there have been studies that attempted to predict building age. For example, one study used a CNN to extract features from street-level imagery and a support vector machine to predict construction year, with mean absolute errors between 10 and 12 years depending on the model. Finally, another study used CNN-based prediction from street-level imagery in Amsterdam, resulting in 81% accuracy.

There are two innovations that could potentially help fill the information gap on building form and function. The first is the growth of open data on building footprints, heights, and building age, and the second is advances in generative Artificial Intelligence (AI) in the form of multimodal large language models (MLLMs), which have been trained on large amounts of information. Unlike traditional language models that primarily process and generate text, MLLMs are capable of understanding and generating information across different modes, including text and images, and thus can be used to describe what they see in photographs. Such an approach has not yet been used to determine whether building height, age, and function can be extracted from photographs using MLLMs. Hence, the aim of this chapter is to investigate this feasibility. Using a sample of buildings across New York City from Mapillary images, matched to building footprints using a spatial-analytical approach and then classified with ChatGPT, this chapter highlights the advantages and limitations of such an approach for creating wall-to-wall datasets on building attributes in the future.

Results and Discussion

To investigate cases where ChatGPT misclassified the building types, we break down the classification patterns. The confusion matrix (see Fig. 2(a)) highlights that while 87% of residential buildings were correctly identified, 9% were mislabeled as mixed-use and 4% as commercial. Similarly, for commercial buildings, 24 out of 30 were correctly classified, but 5 were misidentified as mixed-use, indicating a certain overlap between these two categories. Mixed-use buildings showed the highest misclassification rate, with 32% (14 out of 44) incorrectly labeled as either residential or commercial. The chord diagram in Figure 2(b) visually emphasizes these trends, showing a high degree of agreement for residential buildings and greater confusion between the mixed-use and commercial categories.

To investigate the cases where ChatGPT misclassified building types, we first checked the in-depth building descriptions and found that the model mainly relies on visual cues and contextual signals rather than official zoning standards: residential buildings are identified by features such as windows, balconies, and private entrances, while commercial buildings are identified by larger windows, signage, and visible business activities; mixed-use buildings are often recognized through a combination of retail or office space at street level with residential units above. Figure 3(a) shows examples where ChatGPT’s labels align with expert labels. To further explore where disagreements occurred, we identified three main challenge patterns, shown in Figure 3(b): building-boundary confusion, dominance of visual features, and visual obstruction-induced assumptions.

To further investigate age classification limitations, Figure 4 highlights these cases. While Pre-WWII buildings showed the highest F1 score, 21 out of 84 buildings (25%) were misclassified as Post-WWII, partly because neighboring periods share similar architectural features and historical renovations can obscure original styles. Similar confusion was observed between Victorian and Pre-WWII categories. Conversely, the Postmodernist category had sparse agreement, with confusion spreading across other categories, and the chord diagram (Fig. 4(b)) visualizes these agreement and confusion patterns.

Turning to building height, Figure 5 summarizes both performance and error sources. Figure 5(a) compares ChatGPT-predicted heights with NYC Open Data, showing a moderate relationship (R² = 0.59). Figure 5(b) highlights two principal failure modes that explain the outliers: incomplete visibility, where only part of the building is seen and height is underestimated, and building-boundary confusion, where adjacent structures are treated as a single facade.