Science: you're doing it wrong.
Breaking News: Last week's weather
Blue (May 7): 50%; Green (May 8): 57%; Red (May 9): 66%

It's no surprise that umbrellas are more popular on wet days. We can factor out these effects by looking only at those that were interested in umbrellas to start with, people who entered the umbrella shop.

What percentage of shop visitors purchased an umbrella?

What has this got to do with

Crowdsourcing a HIT

Research on the motivations of the crowd often makes this mistake. No, not the creation of zombie hordes – though this is still an area of intense debate: wisdom of the crowds vs. groupthink – using the number of task units accepted in a given time as a proxy measure for the popularity of the task.

Workers’ favourite way to cook a steak is unlikely to be as well answered at 09:00 UTC, when Mechanical Turk is mainly staffed by South Asians, as it is at 18:00 UTC when Texas, and the rest of the United States, is looking at lunch.

Despite the huge sample sizes possible using Mechanical Turk, without an accurate estimate of the potential workforce for a given task, popularity is difficult to measure. To overcome this we consider the number of workers previewing the task.

Conversion Rate

Using turkmill, a tool we developed to extract preview data from standard web-server log files, and dividing the number of workers who complete a task by the number of workers who chose to preview the task we can find the conversion rate.

This metric eliminates workers who were either uninterested in the task or were simply unaware of it due to searches and filtering, or simply not being online and available.

Conversion rate can also be plotted over time. This allows an increased insight into worker behaviour and allows factors which cause variance to be identified.

Such graphs highlight the general linear trend and the effect of noise in what is a cumulative metric. Calculating the mean value for the inter-quartile time period allows a nominal conversion rate to be determined: a representative measure of popularity, allowing comparison between crowdsourced tasks to be made.

Graph of conversion rate for non-US workers Simple Mechanical Turk Task
  1. Umbrella, modified from: http://www.clker.com/clipart-mango-umbrella.html
  2. Umbrella Corporation logo, modified from: http://commons.wikimedia.org/wiki/File:Umbrella_Corporation_logo.png
  3. Biohazard, from: http://www.clker.com/clipart-boue.html
  4. Zombie, modified from: http://www.clker.com/clipart-zombie-2.html
  5. Weather icons, from: http://darkobra.deviantart.com/art/Tango-Weather-Icon-Pack-98024429
  6. Torn paper, modified from: http://www.clker.com/clipart-torn-paper3.html
  7. Sticky note (poster only), modified from: http://www.clker.com/clipart-blank-sticky-note-1.html
  8. QR code (poster only), courtesy of: http://goqr.me
SICSA logo EPSRC logo University of St Andrews logo

Get the poster: A4 Print of Poster

More about me: http://www.cs.st-and.ac.uk/~jtj2

© 2013 Jason T. Jacques