Mixed-reality photogrammetry in focus
Combining photogrammetry, drones and VR/AR technology in real time
The use of mixed reality in photogrammetry software has resulted in a real-time 3D inspection system that allows users to remotely access, visualize and measure the stereoscopic model simultaneously. An on-site operator runs a drone equipped with a stereo camera, and thanks to virtual reality headsets experts can observe the object of interest without leaving the office. This improves both cost-effectiveness and safety when inspecting large, critical structures.
Protecting critical infrastructure (such as power grids, transport networks, and information and communication systems) is vital for the security of countries and the well-being of their citizens. The European Commission pays attention to this fact and has launched the European Programme for Critical Infrastructure Protection (EPCIP) to reduce the vulnerabilities of such infrastructure. This is a package of measures aimed at improving the protection of critical infrastructure in Europe, across all Member States and in all relevant sectors of economic activity.
The infrastructure elements are under natural and/or anthropogenic pressure, and their monitoring is necessary for many reasons. Their construction is so complex and cost-intensive that maintenance and inspection is a must. Unexpected damages may cause economic losses and can also yield catastrophic results. Therefore, the monitoring of such objects, e.g. bridges, viaducts, dams, towers, is an important procedure and essential for both the public safety and the rehabilitation of the object.
Rich visualization experience
In 2022, a research project called REALTIME3D was set up. The goal was to design and develop livestreamed, multi-user and 3D stereoscopic view-based mixed-reality (MR) photogrammetry software for diverse kinds of inspection tasks. This led to the installation of a stereo camera rig as the image acquisition unit on an uncrewed aerial vehicle (UAV or ‘drone’) (Figure 1). The stereo base length follows the photogrammetric base-to-height ratio rules in order to get the best network geometry for stereoscopic viewing and mensuration.
The UAV system is operated on-site by an operator, while the inspection is done simultaneously in-office by the experts/engineers wearing virtual-reality (VR) headsets. Using the project’s innovative MR software, multiple users from multiple locations can connect to the system, and visualize the object of interest in 3D stereoscopic view mode with the capability of photogrammetric measurement.
REALTIME3D is a new system in the sense that both data acquisition and photogrammetric image processing is unified and done on the job in real time. The photogrammetric products are interactively and simultaneously shared by the multiple users who can even be in completely different geographical locations around the world. Additionally, REALTIME3D provides a very rich content and visualization experience – much greater than being located at the site itself.
System components
The main components of the REALTIME3D system are a stereo camera-equipped UAV and an MR-based photogrammetry software suite for real-time inspection of critical infrastructure (Figure 2). The entire hardware and software system is merged into a unified pipeline which has online (image) acquisition and (photogrammetric) processing capability. It includes the following components:
1. UAV carrying stereo camera and on-board software
2. Mixed-reality photogrammetry application running on VR headsets
3. The backend and the web application running on cloud servers.
Asynchronous monoscopic cameras (photo & video) are widely used onboard UAVs. However, the use of the simultaneous stereoscopic camera systems in the geospatial market is new and uncommon, and the few that are available are limited to academic purposes.
UAV carrying stereo camera and onboard software
The GGS AeroSpector (AS) 800 UAV is used as the aerial platform for the project. It has a good effective flight time, a good cost/performance ratio, and standard components in the case of repairs, replacements and modifications (Figure 3).
The AS-800 uses a foldable carbon quadcopter frame for a payload of up to 2kg. It is a complete, ready-to-fly (RTF) drone set with Herelink remote control. The AS-800 has a diameter of 800mm (motor centre to motor centre). This copter achieves a flight time of approximately 70 minutes without payload (and approx. 52 minutes with a 500g payload / approx. 45 minutes with a 1,000g payload / approx. 32 minutes with a 2,000g payload). The weight of the copter including all electronics is only 1,800g. Including a 1,000g payload and 17,000mah GensAce battery, a take-off weight of less than 4.5kg is possible.
Two Alvium cameras are used with a high-resolution image format of 20MP. The camera type is Allied Vision Alvium 1800 U-2040c Color Camera with USB 3.1 interface, C-Mount lens connector and a CMOS sensor of 4.512 x 4.512 pixels à 2.74µm pixel size. This camera has a low energy consumption, is lightweight and has very robust sensors typically used for industrial applications. The cameras are equipped with 16mm high-resolution lenses from the same manufacturer.
The cameras are fixed at the two ends of a 1m carbon tray base (stereo rig). The cameras are connected to an onboard PC (Latte Panda 3D) where the onboard software runs for image pre-processing and data transfer. The USB-3 interface is used to power the cameras (Power over Ethernet) and to control the cameras via a software development kit (Vimba SDK). An interface to trigger the cameras is connected on a separate port and connected to the UAV controller. GPS-RTK geotags with inertial measurement unit (IMU) data can be streamed with the images in real time if required.
Mixed-reality photogrammetry software running on VR headsets
Calibration of the stereo cameras and their relative orientation is done using Australis photogrammetry software with the calibration targets (Figure 4). Calibration information is passed to the backend software on the cloud server where distortion-free normalized image pairs are generated. These images are sent to the MR photogrammetry application for stereoscopic view generation. Users from multiple locations can access these stereo pairs with their VR headsets and interactively inspect the real-world objects (Figure 5).
The MR photogrammetry software, which has been developed in the Unity 3D engine, visualizes the 3D stereoscopic models for labelling, annotating, measuring and vectorizing the real-world object in detail. A VR native 3D graphical user interface (GUI) is implemented for gathering the data from the backend and displaying it to the user. Rather than the cartographic abstractions (2D maps and 3D models), augmented stereoscopic models are the primary format to be saved and retrieved in the database.
Web application
Data collected by all components can also be visualized and partially edited with the web application. The cloud-based backend is the central hub for the application pipeline. Users who do not have access to a VR headset are also able to use the software via this web application, and also have access to measurement and annotation functionalities.
Conclusions
The inspection and maintenance of infrastructure elements is often contracted out to service companies. If the team of experts are not based in the vicinity of the infrastructure, they may have to travel some distance to access the site and perform their tasks. Additionally, inspecting the infrastructure can entail checking relevant elements from up close, such as by climbing walls or pillars or abseiling from the top, with the associated risks (Figure 6).
REALTIME3D provides a technical solution that allows maintenance companies to perform regular infrastructure inspections without traveling to the site and without performing risky tasks. Besides infrastructure maintenance companies, other target groups for this solution are the public/private bodies and authorities managing the infrastructure, and any other organizations related to monitoring and maintenance services. Any type of generic customer who needs to carry out field visits or on-site observations (for qualitative and quantitative assessments) can also use this system to reduce their travelling and to acquire timely and highly detailed 3D data. As an additional benefit, the collected images can be stored and used for documentation in case problems emerge over time.
Acknowledgements
REALTIME3D is a research project co-funded by EUROSTARS and the European Commission for the period 2022-2024.
Further reading
Communication from the Commission of 12 December 2006 on a European Programme for Critical Infrastructure Protection, COM(2006) 786 final – Official Journal C 126 of 7.6.2007, https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=LEGISSUM:l33260
Critical infrastructure protection, EU Science Hub.
European Programme for Critical Infrastructure Protection, EUR-Lex Access to European Union Law.
Real-time 3D inspection of infrastructure using a stereo camera rig on a UAV. http://www.rtime3d.com/

Value staying current with geomatics?
Stay on the map with our expertly curated newsletters.
We provide educational insights, industry updates, and inspiring stories to help you learn, grow, and reach your full potential in your field. Don't miss out - subscribe today and ensure you're always informed, educated, and inspired.
Choose your newsletter(s)