Using MTurk in Research
This guidance has been developed to ensure that the ethical principles of respect for persons, beneficence, and justice are being upheld for research involving MTurk Research participants. The guidance also applies to other popular survey firms such as Prolific.
What is MTurk in Research
Amazon Mechanical Turk (aka MTurk) is an online marketplace that matches “workers” with available work from various “requesters.” This work generally “requires human intelligence.” Many researchers have been utilizing MTurk to access a large heterogeneous population of participants for research studies. The MTurk site is designed for matching people with work and submitting payments to workers. It is not designed to facilitate the protection of human subjects in research studies.
Approval-related guidance for studies that use MTurk and other survey firms
In addition to the required elements of consent, consent materials for an MTurk study should include the following:
- A clear explanation of the task (e.g., writing, watching videos, taking a survey)
- Information about any screening processes to qualify for the study. This any “attention checks” within surveys.
- Information about additional software required to complete the task (e.g., JavaScript)
- The time commitment required to complete the task
- The full name of the researcher, contact information for the researcher/research team, and contact information for the IRB (215-898-2614—please do not provide an email address for the IRB)
If you do not state the full study purpose or include all research elements/procedures in your consent form, you are using deception or incomplete disclosure in your study. This includes scenarios where you do not tell participants that your study involves randomization. If you use deception or incomplete disclosure, please:
- Indicate in HS-ER that you will use Deception and answer associated questions
- Include plans for debriefing participants about the study’s true purpose in your HS-ERA application
- Include a statement that the study uses deception or incomplete disclosure in the consent language
- Attach your debriefing language to your HS-ERA application for IRB review
Many MTurk workers take their rejection rates seriously. The rejection rate indicates what HITs they can take, which in turn determines how much money they can make. This affects some participants’ livelihoods as they have no way to reverse rejections from their accounts. Given this, researchers should include a statement in the consent language about instances in which participants may fail to qualify or be rejected for a task (e.g., failed attention checks).
MTurk worker IDs are identifiable information. Research shows that they can be linked to individuals’ Amazon profiles. Please do not collect worker IDs unless necessary (e.g., to debrief participants). If you need to collect worker IDs, please ensure that the consent language describes the reason for their collection and confidentiality measures in place (e.g., worker IDs will be deleted after being used and not linked to survey data)
Please detail the following in the consent language:
- The amount of compensation and any bonuses. We recommend that compensation be commensurate to the complexity of the task.
- Whether participants will be compensated for any screening processes
- An estimate of how long it will take for the researcher to approve the HIT (i.e., how long it will take for a participant to receive payment)
HS-ERA Guidance for MTurk Studies
The HS-ERA applicat5ion should detail the full research design of the project, including:
- The methods you will use and topics you will cover. Please be specific.
- Any incomplete disclosures or deception being used
- Any plans for screening or “attention checks” that may cause a worker to be rejected. Potential financial risks to the participant due to rejection should be considered when applicable, including any plans to minimize them.
If there is a need to collect MTurk worker IDs:
- Ensure that HS-ERA describes the collection of MTurk worker IDs, the reason for their collection (e.g., debriefing), and confidentiality measures in place
- It is recommended that worker IDs be deleted after being used and not be linked to survey data, especially if the data collected is sensitive in any way
The IRB requires that the following be attached to the HS-ERA application for review:
- Your HIT Description (recruitment notice)
- Your survey instrument(s) or other data collection instrument(s), including photos and videos that you may show participants
- Your consent language, which could be submitted as the first page of the survey
- Debriefing language, which could be submitted as the last page of the survey
Privacy
MTurk lacks the security measures of survey software like Qualtrics. Moreover, collection of data by Amazon is subject to Amazon’s Privacy Policy and Terms of Service. Researchers should be aware of any policies or procedures in place by Amazon should a breach of data occur.