Semantically Enriching Point Clouds - 19/05/2016
Exploring Colour and Geometrical Properties
Students in their second - and hence final - year of the Master in Geomatics programme at Delft University of Technology embarked on a ten-week Geomatics Synthesis Project as a group. The objective was to undertake a small but real-world research project where they could gain experience in the entire geoinformation chain (acquisition, processing and application).
(By Merwin Rook and Stella Psomadaki)
The Master in Geomatics programme at Delft University of Technology focuses on geographical information science. Geomatics at Delft differs from other geomatics programmes in terms of its close connection to the Faculty of Architecture and the Built Environment.
For the 2015 Geomatics Synthesis Project at Delft University of Technology, five students were asked to explore new ways for the direct use of point clouds, in order to fully exploit the data. Next to a geometry, the point cloud data holds colour information that is derived from matched terrestrial images, better describing the data collection environment. Other information usually available in point clouds, such as intensity, were not present due to the nature of the acquisition technique. Until now, point clouds are mainly used for 3D modelling, leading to a lot of this contextual information never being utilised. In most of the cases, the contextual data is discarded at the end of the project.
From the students' perspective, increasing the usability of point clouds comes down to incorporating semantics to the original xyz data. This means that the points are enriched with meaning derived from their urban environment. These additional point attributes can highly increase the value of the data by making it more descriptive. Features and parts of the data can be extracted more precisely, then shared and analysed for more applications and users. This approach creates new possibilities, such as high resolution map creation, water infiltration capacity analysis and other forms of studying the urban environment.
In this project, different approaches and techniques to adding semantics were explored and an optimal workflow was created by combining the best practices. These best practices consisted of indexing, colour transformation, region growing, region classification and the smoothing of the results. The focus in this workflow was on ground points, since the street level around buildings is usually left aside in 3D models.
Designing a workflow that would extract the road information and its surroundings first requires the definition of relevant classes for the application. In this case road extraction and material detection at street level were examined. The relevant subcategories used were: Grass, Road, Cycle path and Footpath. These categories represent how people generally interpret the street level. For more specific applications (e.g. the surface infiltration capacity), where the composition of materials is needed, a specialist would benefit from a classified point cloud with the classes asphalt, grass and tiles.
Data acquisition and indexing
The data acquisition was performed in collaboration with Cyclomedia using their mobile mappers. The mappers acquire panoramic images that are used in a stereo imaging process to create a coloured point cloud. The mobile mapper was driven around the Faculty of Architecture, collecting 93,436,125 points.
A subsection of the data was processed for the project, and ground points were extracted. These points were indexed by using a kd-tree for fast nearest neighbour search and the calculation of the geometrical properties. Current research in object recognition from point clouds is mainly focussed on making use of the geometric properties of the point and its environment. For ground points, however, a labelling process only on geometrical properties is more complex, as generally all ground points lay on the same plane. This is especially true for a flat country like the Netherlands. Therefore, the colour properties of the individual points were used to differentiate between neighbouring points, and identify which belong to the same class.
Region growing and colour similarity matching
In order to create groups of points with similar properties, a region growing algorithm was developed. The algorithm groups neighbouring points with similar colour and geometrical properties, the normal and curvature, using a predefined threshold of similarity. The colour information was transformed beforehand into a different colour space. A colour space is a colour indexing framework that assigns coordinates to colours based on their position in the colour space, like RGB (a 3D colour space). This transformation was done in order to make the data more suitable for spatial operations and especially colour matching. After experimenting with different colour spaces, the CIELab colour space was selected.
In the classification algorithm, the grown regions were given a semantic meaning by making use of supervised classification. This approach required a set of training sample data that had labels associated with them. This sample data came from prior knowledge about the scene. This meant that for the area of interest, samples were collected that belonged to the Road, Cycle Path, Foothpath and Grass. A Gaussian distribution of the colour properties was assumed to define the data classification algorithm. A 99% confidence level was chosen, as this gave the most optimal results.
To handle the noise generated from the supervised classification, a smoothing algorithm was developed. This method was based on image smoothing filters for noise removal. The algorithm traverses all points in the classified point cloud. For every point it checks its k-nearest neighbours and calculates the most frequent class among these neighbours. Next, the point gets reclassified if the original classification differs from the most prevalent class of it’s neighbours. This process offered both a visual and accuracy advantage to the final result.
The use of colour in combination with geometrical properties in today’s point cloud processing procedures can offer invaluable assistance in classifying objects. Normals and curvature are used to region grow, but not for directly labelling the point cloud. While most of the procedure can be automated, this approach still requires human interpretation in selecting training samples. Using external data, like the position of the car, could help in further automating the process.
Figure 1, The Geomatics Synthesis Project group, in front of the Cyclomedia mobile mapper.
Figure 2, The selected scene, before and after the ground filtering process.
Figure 3, The classified points, where each colour represents a different class.
Figure 4, Workflow of the classification process.
The group members Adrie Rovers, Tim Nagelkerke, Irene de Vreede, Stella Psomadaki and Merwin Rook would like to acknowledge the support of Bart Beers from Cyclomedia, as well as thank their mentor Wilko Quak.
Merwin Rook holds a BSc. in Urban planning and human geography and is a Geomatics Master student at Delft University of Technology, currently graduating on the subject of creating a workflow to automatically enrich 3D city models with semantic and thematic information.
Stella Psomadaki is a Geomatics Master student at TU Delft. She holds a BSc.in Rural and Surveying Engineering from the National Technical University of Athens, where she graduated first of her year. She is currently working on her graduation project on investigating the possibilities of integrating space and time in point clouds using a database management system.
Last updated: 01/06/2020