Crowdsourcing & Code Perception

What’s your problem?

Programming is hard. It requires that developers not only learn a new language, with new rigorously enforced syntax and semantics, but for many individuals it requires a new way of thinking with a rigidity and internalised consistency that they might not be used to. With binary computers everything is, very literally, black and white, high or low, one or zero. How can we reduce the cognitive burden on these individuals and help them reach their goals? Somebody think of the developers!

Code is poetry

Writing software is a skill, but it is also an art. So how exactly does creativity, design, and presentation fit into the raw, digital environment of an IDE? Aesthetics have long been used as a key influencer of human behavior – the entire print and multimedia advertising industries are built around this very concept – so can we use the aesthetic properties of the text developers type to influence them to write better code, faster? Is there is more to it than mono-space fonts?

Empowering people

As understanding and control of digital technology becomes a basic requirement to survive the modern world, programming gains a higher profile. While platforms like the Raspberry PI may bring flexible computing power to the masses, harnessing it remains a problem. For those that wish to take the plunge, coding must be easy to understand. How can code be made simpler and easier to understand, without changing the content itself?

He shoot! He scores?

The goal of this research is to improve the coding experience, not just of the experienced developer but also of those new to programming. By determining the presentational features that have measurable impact on developer performance and code perception, the aim is to use these features to inform the development of an assistive editor and improve the user experience of developers.

Good, fast, or cheap? Pick three.

Developers are expensive. And every one is unique. Attempting to discover general principles from a limited pool is difficult. Crowdsourcing has been shown to be a valuable way of gathering data for user studies quickly and cheaply. With carefully planned controls and objective filtering, crowdsourced studies have been show to be qualitatively and quantitatively equivalent to experiments with lab-based participants, making crowdsourcing a ‘holy grail’ for some types of user studies.

Why does the crowd work?

To better understand our potential participants, a more complete picture of the marketplace in which they work is required. In our long-term demographic study of crowd workers, comprising in excess of 3,000 participants, we have gathered both basic demographics and socio-economic status of US-based crowd workers in a way that is comparable to US labour statistics. This allows us to better understand crowd worker behaviour in the current economy and identify any causal relationship.

Making studies repeatable

For any results to be taken seriously, data collection must be repeatable. Part of the problem with crowdsourced data is the huge potential variability in participants due to factors such as time zone and culture. What do we really know about how workers choose to undertake a task?

To improve understanding of the worker pool and allow a more comprehensive understanding of worker behaviour we proposed a new methodology and new metrics to allow data to be captured about those workers who preview a task, but choose not to accept. Our key metric, conversion rate, allows the evaluation of how many potential workers decided to undertake the task. This insight is essential when attempting A/B testing using the crowd: do workers really prefer A, or were they all just asleep when B was tested? The ability to dynamically determine the effective worker population also provides advantages for commercial tasks, allowing requesters to measure the effect of actions taken to manipulate speed, cost, and accuracy.

The graph, right, shows the conversion rates for non-US worker for each of our task variations.

As expected the value proposition, or informing users of exactly how much work is required to receive the reward, was essential to a strong conversion rate. The most unexpected finding was a preference for tasks without University branding. We believe this may be due to poor worker experience with other institutions.