Five Years of Benchmarking Automatic Scene Interpretation

In 2011, the ISPRS working group III/4 ‘3D Scene Analysis’ released a state of the art airborne dataset to the public. Two tasks were given to the scientific community: to detect urban objects, and to reconstruct 3D building models using the data provided. To date, more than 40 contributions were submitted to the first challenge, and more than 20 to the latter one. Reference data were used by the working group to independently evaluate the results submitted by peers all over the world. Results are available online and a special issue in the ISPRS Journal of Photogrammetry and Remote Sensing (Volume 93, July 2014) contains contributions from participants and summarises the main findings from those benchmarks. After five years we can now conclude that those challenges helped colleagues to compare their approaches with other authors, and to advance the state of the art in urban object detection and building reconstruction. We would like to announce that after the 2016 ISPRS Congress  all the reference data will be released, and that the evaluation will be discontinued. Please find updates on our website later this year.

In contrast, however, the three semantic labelling benchmarks that are maintained by our working group as well, will still be available, and we will continue evaluating results. The provided datasets (Vaihingen 2D, 3D and Potsdam 2D) are of high quality, and we encourage researchers to test their methods on scene understanding. Current works comprising deep learning methods, such as convolutional neural networks (CNN), achieve classification accuracies of more than 90% and – like the first benchmarks – the semantic labelling challenge is very well visible within the community. For this congress our working group received many interesting and innovative papers. In three oral sessions we will see high-quality contributions, from graph-based approaches to the latest developments in CNN, from 2D semantic labelling to 3D point cloud analysis.

We would also like to point out that in the autumn there will be another opportunity to discuss with colleagues and exchange experiences in scene analysis: the 6^th Geobia conference will take place in Enschede, the Netherlands from 14 to 16 September, hosted by the ITC Faculty of the University of Twente. In twelve oral and three poster sessions about 150 papers will be presented, ranging from method-driven works on segmentation or machine learning, to societal relevant applications and solutions regarding large scale operationalisation. Please visit www.geobia2016.com for further information and conference registration.

Markus Gerke on behalf of the WG III/4 and Geobia chairs.

Markus Gerke joined the ITC Faculty of the University of Twente in 2007 as an assistant professor for image sequence analysis. His focus is on capturing geometric and semantic information from images and image sequences.