Benchmarking Image-based DSM Generation - 30/12/2013
The Potential for High-quality Airborne Data Collection
In view of the rapid advancement in photogrammetric technology and software for 3D reconstruction, the European Spatial Data Research Organisation (EuroSDR) initiated a benchmark project on image-based generation of Digital Surface Models (DSM). Such a comparative evaluation provides a platform for software developers to demonstrate the potential of their ongoing work. Furthermore, it can help users like National Mapping and Cadastral Agencies responsible for state-wide generation of high-quality DSMs to understand the applicability of such tools while triggering further developments based on their needs. The comparative results from 10 participants presented here provide comprehensive insight into the landscape of dense matching algorithms and demonstrate the current potential of image-based photogrammetric data collection.
To limit the data-processing effort required by potential participants, the test was restricted to subsets of aerial image flights. Thus, two representative datasets consisting of two aerial image sub-blocks with different land cover and block geometry were prepared. The first dataset, Vaihingen/Enz, is representative for data usually collected during state-wide DSM generation in areas with varying land cover. It covers a semi-rural area of undulating terrain. Both ground sampling distance and image overlap are rather moderate. The sub-block selected for the benchmark consists of three strips each with 12 images, resulting in four to nine images per object point (Figure 1). The DSM had to be generated for the block centre measuring 7.5km x 3km. Its pre-defined grid width of 0.2m corresponds to the ground sampling distance (GSD) of the aerial imagery used. The second test dataset is more typical for applications in densely built-up urban areas, where images are usually captured at a higher overlap and resolution. The Munich dataset covers the central part of the city and was captured at a GSD of 10cm with 80% in flight and 80% cross flight overlap. This results in up to 15 folded object points and thus a considerable redundancy. The image sub-block to be processed consists of three image strips each with five images. The area to be processed measures 1.5km x 1.7km (Figure 2).
Easy-to-Use Test Procedure
To enable a larger number of participants from varying backgrounds, the test was designed to be simple to implement while providing a basis for generating comparable datasets. All participants had to use the orientation parameters made available for the image blocks without modification. No 3D point clouds were taken into account, the evaluation was limited to a DSM raster in predefined size and resolution. The project was initiated in February 2013, and participants were given until May 2013 to submit their results. As a first review, these were then presented and discussed during the 2nd EuroSDR workshop on 'High Density Image Matching for DSM Computation' which was held at the Federal Office of Surveying and Metrology in Vienna, Austria, in June 2013. Software systems presented at the workshop were SocetSet 5.6 (NGATE) from BAE Systems, Microsoft’s UltraMap V3.1, Match-T DSM 5.5 from Trimble/inpho, ImageStation ISAE-Ext from GeoSystems, Pixel Factory from Astrium, the DSM Tool from the Royal Military Academy (RMA) of Brussels, the remote sensing software package from Joanneum Research, MicMac developed at IGN, SURE from the Institute for Photogrammetry (IfP), University of Stuttgart and the FPGA implementation of the SGM algorithm from the German Aerospace Center (DLR).
Evaluation of DSM Quality
Since it is difficult to retrieve high-quality reference surfaces from independent measurements, the participants’ results were used to generate a median DSM for evaluation purposes. While this median cannot be regarded as independent ground truth, differences between the respective solutions can nevertheless be captured.
Figure 3 shows an ortho image of a subset and a shaded relief representation of this median DSM for a part of the Vaihingen/Enz test area. A colour-coded representation of the root mean square (RMS) differences of all DSMs with respect to this median is overlaid additionally. As it is also visible from the corresponding ortho image in Figure 3 (right), larger differences occur in the shaded area of the quarry, in the river area, in the vicinity of patches of trees and in the area of the vineyards.
Figure 4 shows separate DSM differences for a number of single solutions. These differences between -2m and +2m are colour-coded differently than the RMS differences in Figure 3. For a quantitative analysis, elevation profiles were additionally extracted. These lines are overlaid on the ortho image in Figure 3 (left). The elevation profiles for line 0 are depicted in Figure 5. While all elevations are very similar for the well-illuminated area of the quarry, there are larger differences for some solutions in the area subject to shadow. For object surfaces not subject to image-matching problems, differences between almost all solutions are in the order of 20cm, i.e. in this case equal to the GSD.
Similarly, results for the test area Munich are presented in Figure 6. Differences between the respective solutions are mainly visible in small details and steep edges which occur close to buildings. Furthermore, shadows seem to result in larger differences and increased noise for the reconstructed surfaces.
Elevations for the exemplary profile 0, which is also overlaid on the ortho image of Figure 6, are depicted Figure 7. The first part of the profile is on the ground which is covered by a cast shadow. Especially in this area some solutions appear relatively noisy, and relatively large differences for another solution occur at the step edge defined by the façade. A closer view of the profile at two dormers is given in Figure 7 (right). As it is visible all solutions follow the shape of these structures correctly, while the difference is in the order of 10cm, which again is equal to the GSD of that dataset.
In addition to the geometric DSM quality, the quality of a software solution also depends on the computational efficiency and the processing time required. This is not only influenced by the implemented algorithm, but of course also results from the hardware environment used. Due to the considerable amount of reading and writing operations, data ingest, which depends on the available storage system and network environment, is another factor of influence. Therefore, participants were asked to provide information on their hardware environment, processing strategy and time effort for processing the benchmark data. The hardware environments varied from standard desktop PCs with single, multi-core or multiple high-end graphics cards to the application of larger computer clusters. Therefore the documented processing times do not allow for a clear ranking but rather demonstrate the general capacity of image-matching algorithms for area-covering DSM collection. Generally speaking, acceptable run-times were reached for a number of software systems even if a standard hardware environment was used. Thus, processing for the Munich dataset with 15 images at 220MP/image and Vaihingen/Enz with 36 images and 136MP/image could be realised in about 4 hours.
Even though stereo-matching for the automatic generation of elevation data from aerial images was originally introduced more than two decades ago, considerable progress has been made only recently. State-of-the-art digital airborne cameras provide imagery of good dynamic and signal-to-noise ratio. Furthermore, approaches like semi-global matching (SGM) allow for the efficient implementation of additional constraints, e.g. the preferences of smooth surfaces. These developments enable a per-pixel elevation estimation, even for difficult scenarios like low-texture areas.
The EuroSDR benchmark project shows that a growing number of software tools for detailed and reliable image-based DSM generation are available. Processing not only benefits from improved algorithms but also takes advantage of large image overlaps in order to efficiently eliminate erroneous matches. This provides a considerable reliability of DSM at vertical accuracy close to the sub-pixel level. However, some scenarios can still cause problems during image-based surface reconstruction. A few solutions showed decreasing accuracies when shadows were cast. Differences between the respective results also increased for fine object structures close to the resolution of the available images.
EuroSDR (2013): http://www.ifp.uni-stuttgart.de/eurosdr/ImageMatching/ Web page of the EuroSDR ‘Benchmark on Image Matching’ project.
Fritsch, D., Pfeifer, N. & Franzen, M. (Eds.) (2013): Proceedings of the 2nd EuroSDR workshop on High Density Image Matching for DSM Computation. EuroSDR Publication Series, No. 63. See also http://geo.tuwien.ac.at/news/2nd-workshop-on-high-density-image-matching-for-dsm-computation-successfully-completed-2013-06-19/
Haala, N. (2013): The Landscape of Dense Image Matching Algorithms, Photogrammetric Week 2013, Wichmann Verlag, pp. 271-284.
Current software development for image matching is advancing at a considerable pace. Thus, the major aim is not to rank the systems, but rather to demonstrate their capacity for area-covering quality data collection. Furthermore, continuous improvements can be expected, both for matching accuracy and computational performance. Thus the benchmark project will be continued for future updates.Last updated: 22/02/2020