Guest post by Steffen Fritz, Linda See & Ian McCallum of the International Institute for Applied Systems Analysis, Laxenburg, Austria

Steffen Fritz, Linda See & Ian McCallum

Citizen science, the collection or analysis of research data by the general public, has existed in one form or another for centuries, with contributions ranging from plant and animal observations to weather phenonmena¹. In the field of land cover and land use, however, its application is relatively new². Previously this was a task left largely to governments, research institutes and global bodies. With the recent availability of high resolution satellite imagery, this has changed, opening up new possibilities for citizen participation³. In our recent article in Nature Research’s Scientific Data⁴, we have made available a global dataset of crowdsourced land cover and land use reference data, containing the results of our first four citizen-science campaigns.

While citizen science’s potential in this field is enormous, there are numerous challenges. Two of the most common are attracting and retaining participants and the quality of the data they provided. Over time we have gained experience in how best to design citizen science projects in the field of land cover and land use to address these and other issues. In this blog post we describe some of these techniques briefly.

Each of our campaigns to date has been clearly driven by a research question, which we have communicated to the participants at the start of each campaign. In the study now published at Scientific Data, we communicated the research questions as: 1) determining land availability; 2) reducing land cover disagreement; 3) quantifying human impact; and 4) generating robust reference data. Having a clear research question, in turn, helps drive the sample design, which is vital to ensuring that we do not waste the time of our volunteers, but make effective use of their efforts. In this regard, one of the initiatives we are undertaking currently is to consider the quality (e.g. degree of cloudiness) of the available imagery for the task at hand when designing the sample.

In order to ensure a successful campaign, the selection of incentives is also crucial. Here, we employ a variety of techniques ranging from gamification, authorship, prizes and educational feedback. Gamification is the addition of game elements such as leaderboards and the ability to win prizes⁵, which can make participation more enjoyable. Prizes have included Amazon vouchers and co-authorship on a scientific paper for those participants who contributed the most classifications via Geo-Wiki, both in terms of quantity and quality. This latter incentive has been particularly successful in our past campaigns, and is something we are continuing to use. It is worth noting that in previous campaigns participants generally had an academic background, however more recently we have been targeting a wider, more diverse audience. Finally, we have tried different approaches to providing feedback to participants in order to help them learn over time and improve their visual identification skills, including a tutorial that we employed in the third mapping campaign, which all participants were required to view before they could start the competition. We also use online galleries with examples of different land cover types and a Facebook group to discuss images that are difficult to classify, providing rapid feedback to participants during each campaign.

Educational feedback and training during campaigns is one of the most important factors in quality control. For example, we have shown that non-experts in remote sensing and visual interpretation can improve over time⁶. In this context, it is also important to obtain as much information about the volunteers (e.g. profession, experience, etc.) as possible for post-processing. In these four campaigns, we used control or expert data after the fact to determine how well the participants performed. This quality information was then used to weight the data in further analyses⁷. In more recent campaigns, we have experimented with the idea of the wisdom of the crowd. That is, we have given the same location to multiple participants and determined correctness based on the majority⁸. This was shown to have limitations, and some combination of majority agreement with the use of expert data is important⁹. More recently, we have been using control or expert data during the campaign, rather than after, in a manner that feeds directly into the scoring mechanism. Feedback is thereby delivered to participants in real time, helping them improve. In addition, we have implemented a second near real-time feedback mechanism through Facebook that opens up a dialogue between the citizens and us, which we call ‘Ask the Expert’.

With each successive Geo-Wiki campaign, we have continued to learn more about how to run a successful crowdsourcing initiative. We look forward to continuing this fruitful interaction with our volunteers through future Geo-Wiki campaigns and we recognize the value of their contributions, without which it would not have been possible to create better land cover products.

References

1. Miller-Rushing, A., Primack, R. & Bonney, R. The history of public participation in ecological research. Front. Ecol. Environ. 10, 285–290 (2012).
2. Fritz, S. et al. Geo-Wiki.Org: The Use of Crowdsourcing to Improve Global Land Cover. Remote Sens. 1, 345–354 (2009).
3. See, L., Fritz, S. & McCallum, I. Satellite data: Beyond sharing Earth observations. Nature 514, 168–168 (2014).
4. Fritz, S. et al. A global dataset of crowdsourced land cover and land use reference data. Sci. Data 4, 170075 (2017).
5. Deterding, S., Sicart, M., Nacke, L., O’Hara, K. & Dixon, D. Gamification. Using game-design elements in non-gaming contexts. In CHI ’11 Extended Abstracts on Human Factors in Computing Systems 2425–2428 (ACM Press, 2011).
6. See, L. et al. Comparing the quality of crowdsourced data contributed by expert and non-experts. PLoS ONE 8, e69958 (2013).
7. Fritz, S. et al. Downgrading recent estimates of land available for biofuel production. Environ. Sci. Technol. 47, 1688–1694 (2013).
8. Salk, C. F., Sturn, T., See, L., Fritz, S. & Perger, C. Assessing quality of volunteer crowdsourcing contributions: lessons from the Cropland Capture game. Int. J. Digit. Earth 9, 410-426 (2016).
9. Salk, C. F., Sturn, T., See, L. & Fritz, S. Limitations of majority agreement in crowdsourced image interpretation. Trans. in GIS 21, 207–223 (2017).