When Are Crowdsourced Data Truthful, Accurate, and Representative/

Abstract

We trace crowdsourcing, as a business strategy to gather information, to Britain in the Industrial Revolution, when it was used to create trade directories. We show that the trade directories’ occupational snapshot was very highly correlated (≈0.99) with the 1851 census – a valuable objective metric of accuracy. Accuracy of modern crowdsourced data is more difficult to judge, but seems somewhat lower;  we make an explicit comparison to Yelp. We rationalize our results by considering:  construction of the sampling frame; incentives of the crowd to report correct
information; disincentives to report incorrect information (cost of contributing, presence of “gatekeepersâ€); and sampling strategy.