Game-based Crowdsourcing for Image Analysis

Complex image analysis is a task at which humans are still far better than computers. While computers can conduct simple analysis, they are not yet effective at reliable feature identification and classification, such as to identify where illegal logging is occurring for example. A huge amount of spatial imagery is available nowadays, both of the Earth and other planets, but – in view of the large number of possibilities and the sheer volume of data – there are not enough experts to analyse it all. A novel approach has now been developed that can increase image analysis and mapping capacity founded on game-based crowdsourcing.

(By Hans van ’t Woud, founder, BlackShore)

The approach presented in this article by the Dutch start-up BlackShore began as an experiment to help NASA scientists explore the surface of a planet similar to Earth. Deep canyons, traces of rivers, mineral deposits and evidence of past volcanic activity must be mapped in order to learn more about the history of the solar system. Those and many other geological features can be spotted in imagery collected continuously by NASA’s Mars Reconnaissance Orbiter. To extract more from all that valuable data, the then-novel concept of crowdsourcing was proposed to increase the number of ‘brains’ performing image analysis. The idea was to visualise the Martian satellite imagery in a computer game and ask the players, the ‘crowd’, to map out land features requested by researchers from the planetary exploration domain. While the players were learning about Mars, being inspired by the planet’s beauty and doing their mapping job, they were also continuously contributing to scientific research. The experiment showed that the crowd of players performed the task faster, more thoroughly and just as accurately as the experts, as long as the computer game taught the players what to look out for and kept them motivated. With the idea validated on the analysis of imagery of Mars, BlackShore entered the Business Incubation Programme of the European Space Agency (ESA). It proceeded to evolve its computer game called Cerberus, originally developed for Martian imagery, into a game ready to map Earth and to discover markets where there could be a need for crowd-generated data augmenting satellite imagery, such as in the case of disaster mapping.

Principle of operation

The purpose of the game is to make a map showing the features of interest, such as trees, based on satellite imagery. The images are divided into hexagonal tiles which are tagged by the players according to the dominant feature in each tile. The rationale is that, if a lot of users mark the same object in a tile, the Cerberus system treats it as a real object and then draws the object on the output map, which is compatible with any geographical information system (GIS). The algorithm which combines all the results from the crowd automatically filters out the outliers. For example, if only three out of a hundred people identify the feature in a particular location, the system ignores it and will not put the feature on the output map. However, when ten or more people click on the same spot, the system considers this to be the truth and the feature is drawn on the output map. It is important to ensure that the crowd delivers accurate results and that players do not just click randomly. Achieving the best results requires a mix of a smart scoring scheme, e-learning components to educate players prior to participation in the game, and smart game mechanics. Players are awarded a higher score for clicks which later result in a real feature, implying they have made a meaningful contribution. A player needs to have a minimum score to advance in the game, thereby creating an incentive for every player to do a good job. Interestingly, this results in the crowd noticing more features than individual experts and mapping being completed more rapidly and essentially for free, without quality being compromised.

Typhoon Haiyan

The Cerberus game was evolved to enter new markets for crowd-based image analysis, such as disaster mapping. It was used successfully to map the damage caused by Typhoon Haiyan which struck the Philippines in late 2013. Using very-high-resolution satellite imagery captured right after the skies cleared, the crowd managed to map out more than 8,000 features for an area of 25km² in the space of one and a half days. The map features included the level of destruction to property, blocked roads and damage to key infrastructure such as harbours. In addition, clouds visible in the image were mapped too. This indicated mapping areas with a reduced certainty level to field workers using the maps, which is crucial for people’s confidence in the maps.

Refugees

Another interesting application involved the attempt to help refugees in Iraq. In the summer of 2014, when villages surrounding Mount Sinjar were attacked by members of Islamic State (IS), many of the local population fled to the mountains. The crowd analysed ultra-high-resolution satellite imagery of the mountain range to help map suspicious activity, to identify the location of refugees and to designate safe drop areas for supplies. Within less than a day, the crowd had detected a number of refugees who would not have been spotted by sight alone and also discovered a location which may have been the site of a mass execution. The security aspect posed an extra challenge in this case: how could terrorists, for example, be prevented from participating in the game and making use of the fresh satellite imagery? To counter this, various security measures were put in place, including excluding entire countries from participation, preventing users from downloading the satellite products themselves and preventing the disclosure of exact geolocations to the crowd. Access to the generated maps is restricted to BlackShore and its clients only.

Using open satellite imagery

So far, Cerberus has relied on commercial and hence costly satellite imagery because of the ultra-high resolution. However, funded by the European Space Agency, a feasibility study was recently conducted of performing crowdsourcing using open satellite imagery originating from the Sentinel-2 satellite. The study focused on the detection of illegal forestry and aiding farmers in developing countries. Making use of the Sentinel-2 multispectral capabilities, the crowd was able to help generate a map covering an area of 500km² of Ghana under threat from deforestation. The long-term plan is to ask the crowd to monitor farming activities in Ghana to ensure that farmers can continue to work on their existing plots of land – for example by helping to detect diseases at an early stage – and to help plan irrigation systems. In this way, crowdsourcing technology helps to reduce the need to cut down trees to create fertile ground, helps to increase production yields and thus helps economies to grow while preserving the rainforests.

Artificial intelligence

Today, the human brain is still better at recognising objects in images than computers are, especially when logic is needed. Humans are also able to react to unexpected features which were not the original subject of the analysis – for example, in the case of a mass grave or a ship that has run aground. These are all anomalies that would be picked up by the crowd, whereas if a computer had not been programmed to expect such features beforehand it would simply not see them. Furthermore, even if a computer would learn from one case, that knowledge may not be applicable to the next case; damage signatures, buildings, typical vegetation imagery and even sunlight can all be different in each new case. While computers cannot completely take over from humans yet, humans and artificial intelligence (AI) can be combined by using crowd input to train artificial intelligence systems. In illustration of this, imagine that an area of 1,000km² has been struck by a tsunami. The crowd could map the first 100km², then an AI system would be trained using that map. The AI system could subsequently automatically analyse the remaining 900km² with the crowd then being engaged again to verify the computer’s output.

Author

Hans van ’t Woud is the founder of BlackShore, which is currently taking the crowdsourcing platform Cerberus to the next level. His educational background includes communication and multimedia design followed by human-centred multimedia completed at the University of Amsterdam with a focus on serious gaming, crowdsourcing and e-learning.

E-mail: hanswoud@blackshore.eu

Figure 1, Mapped destruction caused by Typhoon Haiyan. Dark purple and bright purple hexagons represent areas with completely demolished and damaged property, respectively. Bright blue hexagons represent clouds, indicating a lack of certainty about these areas.

Figure 2, Detecting IS activity on Mount Sinjar, Iraq (image courtesy: BlackShore & European Space Imaging).

Figure 3, Mapping logging and agriculture in Ghana using Sentinel-2 multispectral imagery and the Cerberus gaming platform (image courtesy: BlackShore & European Space Imaging).