AI-powered solutions for reality capture data

Seamlessly integrating multiple artificial intelligence and traditional algorithms can save time, simplify functionality and improve performance during point cloud classification and feature extraction.

Point cloud classification and feature extraction are the critical building blocks of nearly every reality capture solution. But even with small datasets, traditional manual methods of data extraction are time-consuming and prone to human error. With today’s aerial, terrestrial and mobile mapping scanners collecting millions of points per second over large areas, the highly detailed point clouds can be overwhelming and difficult to manage. Streamlined workflows incorporating artificial intelligence (AI) help to avoid shifting the time saved in the field to lengthy processing of high-volume data in the office. Seamlessly integrating multiple AI techniques and traditional algorithms in one solution simplifies the functionality and improves performance. 

Point clouds captured by scanners integrated with processing software and analytical tools form end-to-end workflows for a wide range of applications, such as detailed topographic maps, asset management, pavement inspection, volume computations and more. A modern surveyor benefits from using complementary hardware and software that extracts the most information from the data. Embedded in the software, automated AI-based feature extraction and classification capabilities expedite processing and improve accuracy to fully leverage the value of collected data to address real-world problems.

As a critical part of the solution, AI can work continuously without interruptions, and the algorithms are consistent and reliable, ensuring information provided to stakeholders is accurate and timely. Rather than replacing a surveyor, it increases the value and efficiency of a surveyor’s skilled work. AI impacts traditional responsibilities by training machines to perform time-consuming, repetitive activities, leaving humans more time to complete the most valuable components of the workflow, such as data analysis and decision-making based on the deliverables. “Workers appreciate the improvements and benefits of AI,” says Pat L’Heureux, project engineer at Severino Trucking. “We can focus on design work and spend more time analysing data, which leads to smarter and quicker operational decision-making.”

Since AI can be quite complex to implement, a user-friendly interface is critical so that surveyors can leverage the advantages without having to be experts in AI technology. The most effective automated classification and feature extraction functionalities rely on the combination of different AI techniques, including 3D and 2D deep learning and traditional algorithms.

Advanced classification – the starting point

The first step in almost every information extraction workflow is classification of point clouds acquired by terrestrial laser scanners, aerial Lidar and mobile mapping systems, or generated from aerial imagery. Classification helps surveyors simplify and organize the point cloud data and provides a convenient visualization of the different elements, from ground to vegetation to structures. The operation also expedites the preparation of data for manual or automated feature extraction workflows. Additionally, classification allows more efficient data sharing between teams, which is especially critical for large datasets.

As an example of the benefits of classification, if the collected data will be used for extraction of poles and signs (point and geometric attributes for each instance of the object), it is much faster to work with just the poles class and signs class rather than the complete point cloud. When the point cloud is already classified, a surveyor can hide all irrelevant classes and work only with the relevant ones. As a further example of classification, a common task of ground surface modeling requires the point cloud to be separated into ‘ground’ and ‘everything else’.

When working with point clouds containing hundreds of millions of points, manual point cloud classification can be extremely time-consuming and cost-prohibitive. If processing is too expensive, it may exceed the value of the collected data. Therefore, accurate automated classification in the modern surveying world is not just ‘nice to have’; it is required to ensure the return on investment (ROI) for any large-scale reality capture project.

Automatic point cloud classification based on a 3D deep learning semantic segmentation model in Trimble Business Center (TBC) classifies each point into basic classes: ground, buildings, high vegetation, medium vegetation, poles, signs, powerlines, noise (cars, people, scanning artifacts), dividers and steps. 3D deep learning is the most robust solution for point cloud classification. The technology is based on neural networks that allow models to infer probability that a new point belongs to a particular class. The model can make an informed decision when confronted with an unknown object.

With AI doing the labour-intensive classification work, users benefit from a significant reduction in the time it takes to generate separate classes. “The comprehensive information generated adds value and helps us meet and sometimes exceed customer expectations,” says Alex Garcia, national manager mobile solutions at GeoVerra. “After we process mobile mapping data, we classify immediately and send the appropriate files to people just starting the project.”

Customized classification – for ultimate scaling

Generic classification offers great out-of-the-box functionality for immediate use of basic classes critical to most applications. But sometimes even the greatest deep learning model fails to recognize objects due to unique characteristics. Surveyors also face one-off tasks or recurring projects that require extraction of new objects.

Typically, there are two options for handling this situation: perform manual editing/classification, or submit a feature request to the software provider. These tactics are expensive and time-consuming, and both options can be unacceptable when trying to meet tight deadlines. Instead, do-it-yourself customization of the classification model can save time and improve performance when extracting nonconforming objects.

Training 3D deep learning models requires not only programming skills, but also extensive knowledge of AI. To provide access to non-AI experts, Trimble has developed a tool that allows users to train their own 3D deep learning model to add new classes or adapt existing classes and apply this classification on top of the generic classification in the base model. This allows a surveyor to take on new projects with more confidence than ever before. With access to customization tools that are easy to learn and use for individual domain-specific needs, the user is not dependent on a software provider’s roadmap and priorities.

Point clouds from terrestrial laser scanners, aerial Lidar, mobile mapping systems and aerial imagery are divided into samples which are used to train the classification model. Spherical point cloud samples rapidly created during training contain 25,000 points each. To expedite training, high-density point clouds are reduced, or ‘downsampled’. This process is called voxelization; it divides space into cubes of a user-defined size and picks one point from each cube to reduce resolution. A smaller voxel size means higher resolution and a more precise model.

A user loads annotated data and the model is trained to automate classification of any object. The submitted point clouds with annotated class of interest are divided into training and validation data. Training point clouds are used by the model to learn about a class, and the validation files are necessary for the model to automatically assess its accuracy after each training epoch. All complex training parameters are pre-set. However, a user is advised to select a voxel (3D pixel) size that preserves the shape of the object. Overly aggressive downsampling might result in unrecognizable objects, especially for small items like an insulator on a utility tower. The selected voxel size should also ensure that a model sees not only the object of interest but also the context around it.

Training the model is comparable to preparing for a test, with the training files like textbooks used for learning and the validation files like a pre-test. After studying the textbook (training files) for a certain period of time (one training epoch), it is time to take a pre-test (validation files) to understand how well the model is prepared. If the score is not satisfactory, studying continues for another epoch. In TBC, because the calculated accuracy is divided into training accuracy and validation accuracy, the user can automatically select an epoch where the model reached its best training and validation accuracies. Training a model might sound complex, but the simplified workflow allows any user to leverage AI without being an expert in the technology.

Delivering information, not data

Classification is an important first step in data processing. However, data alone is not sufficient to support decision-making and produce most deliverables. Improved classification with AI makes it faster and easier to produce useful, actionable information from large volumes of data.

Surveyors often require automated extraction of features as vectors with attributes and automated analysis of data based on the extracted features. Combining AI and traditional algorithms advances the capabilities from simple point and line extraction to decision-making based on the extracted features and their geometric attributes, such as points (sign, pole, manhole, tree) and lines (lane lines, overhead lines, kerbs). Geometric attributes include height and inclination of signs, diameter, height and crown spread of trees, and more. Access to this additional information helps a surveyor understand the state of the asset (e.g. a pole or sign exceeds the allowed inclination angle).

Supervising AI-enabled processes

With AI, the role of a surveyor shifts from performing manual feature extraction to supervising AI-enabled processes with convenient QA/QC tools. In addition to the time saved, the information gathered is more detailed and reliable than when the same task is performed manually. These tools, based on AI and traditional algorithms, are automatic for point extraction or semi-automatic for line extraction.

Combining different AI techniques and traditional algorithms is particularly critical for leveraging the collected data from complex systems. For example, mobile mapping systems collect both high-density point clouds and imagery, with each data type requiring different AI techniques to extract information.

For example, multiple techniques must be applied to deliver advanced AI-based tasks such as pavement analysis. This routine is based on point clouds collected with a mobile mapping system to detect and classify rutting, depression, potholes, bumps and corrugation, while imagery is used to detect cracking. Cracks, especially if not yet severe, are not visible in the point clouds but perfectly visible in imagery. Conversely, rutting might not be visible in the imagery. A set of algorithms extracts certain types of distresses from the point cloud, and 2D deep learning models extract cracks from the imagery. 3D deep learning is used to isolate only the relevant portion (i.e. pavement) of the point cloud.

The AI-driven tool not only provides information faster, it is more accurate than traditional pavement analysis approaches. Calculation of the Pavement Condition Index score for each road segment based on the international ASTM standard enables repairs to be prioritized and crews dispatched with the appropriate equipment.

Stockpile management is another application where automation powered by algorithms furnishes the information needed to make business decisions. Customers don’t use stockpile boundaries and classified stockpile points; they want the stockpile volumes in an easy-to-read report. Traditional methods involved drawing the boundary by hand, generating surfaces, and finally calculating volumes. “Artificial intelligence is having a huge impact on the work we do, specifically by reducing the time it takes to extract features and classify point clouds,” says René Bundgaard Christensen, land inspector at LE34. “The automated stockpile extraction and calculations in TBC take about half the time compared to manual calculations.”

Significant value to surveyors

As the amount of data being collected and managed continues to increase, streamlining workflows and scaling production with AI-assisted operations is critical for reducing costs and improving quality. AI excels at performing repetitive tasks like point cloud classification and feature extraction reliably and consistently, leaving surveyors free for other or additional field and office tasks.

Remaining competitive today includes offering a wider range of services such as managing assets, inspecting roads, creating digital terrain models (DTMs) and monitoring stockpile volumes at construction or mining sites. The integration of AI and traditional algorithms supports intelligent and confident decision-making by combining and transforming reality capture data into valuable information.