Technology Convergence – Artificial Intelligence and Satellite Imaging - 25/01/2019
Artificial intelligence (AI) has been used for years on satellite images – so why all the excitement? The application of AI and machine learning (ML) to Earth observation (EO) data is a huge growth area, demonstrated by the many online competitions, GitHub repositories and entire businesses founded exclusively on this topic. GitHub is a development platform inspired by the way people work, from open source to business. Users can host and review code, manage projects and build software alongside 31 million developers.
However, taking the Oxford English Dictionary's definition of AI as ‘systems able to perform tasks normally requiring human intelligence’, AI has been available for many years to address the meatiest challenges in image interpretation.
The first automatic number plate recognition system was operational in the UK in the 1970s. Papers from 1985 describe automated 3D surface reconstruction from SAR images. From my earliest days working in space data analytics, AI supported orthorectification and 3D surface reconstructions have been available in off-the-shelf software. Formerly, these required the operator’s visual perception, but were already automated and replicated by computers. So, what has changed?
The amplified interest is due to the promise of:
- Systems that can cope with the wide variation in satellite images.
- Deeper semantic understanding of data replicating the human ability to instantly recognise features in overhead imagery, or even generate imagery.
- Automated interpretation of the amorphous and complex natural and built environment.
- Systems to make automated decisions at the ‘edge’ using resource limited computing.
- Workflows to better analyse information in multiple dimensionalities and data formats.
Each of these will be addressed in turn, with special attention paid to the last point. This is done through the lens of a satellite insights company which uses AI and ML in operational systems.
Variation in Satellite Images
A fundamental requirement is that systems cope with the varying nature of satellite images. Satellite data requires pre-processing before it can be interpreted, or algorithms must automatically cope. This is no mean feat with changing illumination, view angles, seasonality, atmospheric conditions, geometric, electromagnetic and other characteristics. Previously, approaches to this have varied but generally included modelling approaches to approximate the physical reality. Now though, we can throw many examples of a dataset at ML systems until it converges on a solution which always works. Hard-coded systems used to fail if, for example, they encountered a snow-covered image, and a new rule set would have to be created for that scenario, which would itself encounter ‘exceptions’. This made analytics difficult.
Deeper Semantic Understanding
The examples from the first paragraph use AI to perform very simple tasks at scale, making them perfect for automation. Orthorectification or 3D surface reconstruction run many billions of photogrammetric measurements quickly using a computer. Their scale made these tasks close to impossible to perform manually. People though excel at extracting semantics from images, not precise measurements, and this is where AI has advanced significantly. The untrained eye has little issue identifying a road in a satellite image in any country, whether a dirt road or a 10-lane motorway. Previously, we had to build complex process chains using characteristics such as geometry and size which would have trouble with less well-defined roads. Now, using neural networks we can find roads reliably in any imagery anywhere. The semantic depth can be taken further, for example automatically finding which roads are blocked by rubble following a disaster. Previously, this would require good geolocation of satellite images and change detection which is prone to errors. Now, AI can do this by looking at a single image, just like a human could.
Building on this, one of the more powerful recent developments in neural networks is the autoencoder. These are fascinating in that they take data, produce a reduced dimensionality representation and then recreate it in a different form. Simple applications of this intelligently remove noise from images, complex versions can fill in missing data in images, or even generate realistic satellite images from a sparse input. A popular example of this will generate a reasonable Google style map from a satellite image, interpreting all buildings and roads and redrawing them as styled map elements.
Machine learning can also analyse the complex natural and built environment aspects of images. One highly relevant subject is clouds in visible wavelength satellite imagery. These wispy, variable, opaque, transparent, moving features occur anywhere and can strike at any time! Previously, the solution was to use thermal sensors which discriminate well between clouds and terrain. But this is very expensive from a spacecraft perspective. Alternatively, we could threshold out bright white targets. But this results in all bright targets including roofs and snow to be wrongly classified as cloud. With new neural network structures however, an AI can spot cloud cover on a huge number of images in seconds. Even where the clouds are very thin, they work. This has been successfully developed for satellite videos with our partner Cortexica1.
AI is also allowing us to implement data processing closer to the source. There is more power in your smartphone than there was used to launch the first Apollo mission. We are taking advantage of this as we begin to distribute processing of satellite data into space. Using small onboard ML processors, we can make decisions on the spacecraft before downlink. For example, run the cloud detection to quickly determine if we can see through a hole in the cloud or extract all moving objects from a satellite video and only download those. This can substantially help handle the masses of data produced.
The Big Picture
Different tools achieve different ends. The power of these systems is not only found in the individual tools but rather how we choose to combine them to address the pressing needs of users. As such, much of the work at Earth-i is focused on creating the most efficient and effective ways to do this. Key to this is data harvesting from internal and external systems, the management of data and the extraction and transformation of data into formats that can be ingested by these algorithms. This is important as the newest AI and ML technologies are data hungry, requiring large volumes of data for training. Our systems must rise to the challenge.
The result of this development is the ability to address new challenges and serve a broader range of users. For example, we partner with Marex Spectron to produce services for traders where advanced analytics mean there is no need for any training or satellite image interpretation, the correct information is simply streamed into their systems.
A Bright Future for AI
The future is promising for AI and EO and there are many green field areas for research. There is an opportunity to look at old challenges in a new light. In other areas, AI helps make sense of unstructured data, detects anomalous network activity indicative of a hack, and has even been used to design satellite structures. New approaches for EO could include the automatic learning of satellite data formats, increasing satellite autonomy, automatic inference of valuable ground ‘truth’ data from online photos, using AI to infer geolocation data from disparate data sources and many more. We are just scratching the surface of the possibilities, the true power being in the ability to align and combine these tools to realise tomorrow’s solutions.
This article was published in Geomatics World January/February 2019
1 A. Francis, A. Broyelle and P. Sidiropoulos, Cortexica Vision SystemsLast updated: 25/08/2019