Crowdsourcing a HIT

Research on the motivations of the crowd often makes this mistake. No, not the creation of zombie hordes – though this is still an area of intense debate: wisdom of the crowds vs. groupthink – using the number of task units accepted in a given time as a proxy measure for the popularity of the task.

Workers’ favourite way to cook a steak is unlikely to be as well answered at 09:00 UTC, when Mechanical Turk is mainly staffed by South Asians, as it is at 18:00 UTC when Texas, and the rest of the United States, is looking at lunch.

Despite the huge sample sizes possible using Mechanical Turk, without an accurate estimate of the potential workforce for a given task, popularity is difficult to measure. To overcome this we consider the number of workers previewing the task.

Conversion Rate

Using turkmill, a tool we developed to extract preview data from standard web-server log files, and dividing the number of workers who complete a task by the number of workers who chose to preview the task we can find the conversion rate.

This metric eliminates workers who were either uninterested in the task or were simply unaware of it due to searches and filtering, or simply not being online and available.

Conversion rate can also be plotted over time. This allows an increased insight into worker behaviour and allows factors which cause variance to be identified.

Such graphs highlight the general linear trend and the effect of noise in what is a cumulative metric. Calculating the mean value for the inter-quartile time period allows a nominal conversion rate to be determined: a representative measure of popularity, allowing comparison between crowdsourced tasks to be made.