Amazon MTurk: advice for successful research

In 2005, Amazon introduced the Mechanical Turk (MTurk) platform for outsourcing repetitive tasks. MTurk is now exceedingly popular with market research agencies and universities. They find it an abundant source of human resources for, for example, answering surveys.

The platform employs 7,500 full-time and over 500,000 part-time workers worldwide, mainly in the USA and India. In this article, I analyze MTurk from a (market research) perspective and draw your attention to some specific features that could influence your results.

The MTurk workforce

Demographics and representation

The first thing we need to analyze is the demographic representation of MTurkers.

MTurk workers, or MTurkers, are generally younger, better educated, and more tech-savvy than the general population. A sizable proportion are under 30, often unemployed or underemployed. They belong to the lower revenue classes (remember this if you must respect quotas in your survey).

In terms of marital status, they are more likely to have never been married. Demographically, the platform sees more white individuals and fewer Hispanics/Latinos or African Americans. So, if ethnic representation is required in your research, conduct it with particular care. The MTurker population is less representative than national probability samples and some professional online panels but more representative than convenience samples.

Psychological traits

Secondly, we need to be aware that MTurkers have specific psychological profiles. They score higher on introversion. They have lower self-esteem and lower life satisfaction.

They also score higher on

• • Empathy
  • Social anxiety

Knowing these specificities is important, especially if you’re conducting questionnaires that mobilize these traits.

Challenges

The “Super Turkers” phenomenon

Super Turkers” are a small but continually active segment on the platform. They participate in an average of 10-17 weekly surveys, significantly more than typical panel respondents.

This group conducts 40% of all surveys. They are, of course, motivated by money. Reliance on these highly active respondents poses a risk of bias in research results, particularly in research sensitive to the experience effects of repeated survey participation.

Work ethics and motivation

For 80% of MTurkers, revenue is the main motivation. This takes precedence over interest in the subject (20%). This economic motivation leads them to complete tasks quickly to maximize their earnings.

This quest for speed of execution can compromise data quality. We, therefore, need to implement specific measures to identify speeders, i.e., those who respond too quickly to surveys. Training MTurkers could give them an advantage in terms of response time.

Finally, pay particular attention to the design and length of your survey. As with conventional panels, poorly designed or excessively lengthy surveys lead to:

• • Poor-quality responses
  • High dropout rates

My advice for our customers is to stay under 25 questions or 9 minutes.

Points of attention for your projects with MTurk

If you would like to research with MTurk, here are the points of attention that are most important:

False identity of respondents

Identity theft – or, more precisely, claiming qualities the respondent does not possess – is a real problem. It is not specific to MTurk but to all panel surveys.

There are various approaches to counter this problem:

• • removing financial incentives
  • asking screening questions

Sample bias

The specific profile of MTurkers will inevitably lead to bias if you would like to build a representative sample. You will, therefore, need to adjust your sample for the following aspects:

1. Age
2. Race and ethnicity
3. Gender
4. Revenue
5. Education
6. Marital status
7. Religion
8. Political ideology

Recommendations for report writing

For the transparency and integrity of your research project, I recommend that you be clear about:

• MTurkers’ compensation
• Country of residence
• respondents’ basic demographic data.

These factual elements will enable the reader to appreciate the reliability and generalizability of the results. However, this recommendation is not specific to MTurk. It is good practice for any qualitative or quantitative project.

Conclusion

MTurk has become a widespread tool for conducting research, particularly in the social sciences. However, the specificities of MTurkers require certain methodological adjustments. By understanding the demographic, motivational, and behavioral specificities of MTurkers, researchers can understand the limits of their research and anticipate problems.

Further reading

Here are some academic articles on the use of MTurk in research.

• Mason & Suri 2012
• Buchheit et al. 2018
• Hunt & Scheetz 2019
• Paolacci et al. 2010
• Wessling et al. 2017

By Pierre-Nicolas Schwab Dr. Pierre-Nicolas Schwab is the founder of IntoTheMinds. He specializes in e-commerce, retail and logistics. He is also a research fellow in the marketing department of the Free University of Brussels and acts as a coach for several startups and public organizations. He holds a PhD in Marketing, a MBA in Finance, and a MSc in Chemistry. He can be contacted by email, Linkedin or by phone (+32 486 42 79 42)