Marcus runs a small business, Umbrella Corp.
He wants to know why people chose to buy his products.
He decides to offer a different colour each day to see how many sell.
ConclusionPeople don't like
It's no surprise that umbrellas are more popular on wet days. We can factor out these effects by looking only at those that were interested in umbrellas to start with, people who entered the umbrella shop.
What percentage of shop visitors purchased an umbrella?
Crowdsourcing a HIT
Research on the motivations of the crowd often makes this mistake. No, not the creation of zombie hordes – though this is still an area of intense debate: wisdom of the crowds vs. groupthink – using the number of task units accepted in a given time as a proxy measure for the popularity of the task.
Workers’ favourite way to cook a steak is unlikely to be as well answered at 09:00 UTC, when Mechanical Turk is mainly staffed by South Asians, as it is at 18:00 UTC when Texas, and the rest of the United States, is looking at lunch.
Despite the huge sample sizes possible using Mechanical Turk, without an accurate estimate of the potential workforce for a given task, popularity is difficult to measure. To overcome this we consider the number of workers previewing the task.
Using turkmill, a tool we developed to extract preview data from standard web-server log files, and dividing the number of workers who complete a task by the number of workers who chose to preview the task we can find the conversion rate.
This metric eliminates workers who were either uninterested in the task or were simply unaware of it due to searches and filtering, or simply not being online and available.
Conversion rate can also be plotted over time. This allows an increased insight into worker behaviour and allows factors which cause variance to be identified.
Such graphs highlight the general linear trend and the effect of noise in what is a cumulative metric. Calculating the mean value for the inter-quartile time period allows a nominal conversion rate to be determined: a representative measure of popularity, allowing comparison between crowdsourced tasks to be made.
- Umbrella, modified from: http://www.clker.com/clipart-mango-umbrella.html
- Umbrella Corporation logo, modified from: http://commons.wikimedia.org/wiki/File:Umbrella_Corporation_logo.png
- Biohazard, from: http://www.clker.com/clipart-boue.html
- Zombie, modified from: http://www.clker.com/clipart-zombie-2.html
- Weather icons, from: http://darkobra.deviantart.com/art/Tango-Weather-Icon-Pack-98024429
- Torn paper, modified from: http://www.clker.com/clipart-torn-paper3.html
- Sticky note (poster only), modified from: http://www.clker.com/clipart-blank-sticky-note-1.html
- QR code (poster only), courtesy of: http://goqr.me
Get the poster: A4 Print of Poster
More about me: http://www.cs.st-and.ac.uk/~jtj2
© 2013 Jason T. Jacques