Amazon Mechanical Turk Studies

Is current?: 
Yes

Amazon Mechanical Turk provides a different pool of participants to use for research studies.  Its main advantage over the PI pool is that people will complete studies at a higher rate.   The disadvantages include a higher cost, as well as requiring more testing than a regular study.  Also, since workers are paid for completing mturk studies, it is more important to make sure that they won't run into technical issues while taking the study.

To convert a study into a mturk study, there are two changes that need to be made.  First, the startpage needs to not have a popup, since the popup closing causes errors on some computers.  The attached startpage does so. 

 

The other thing you need is to have a submit button on your final debriefing page.  This will send the data back to the mturk servers and let the participant get paid.  I suggest putting this button somewhere obvious on your debriefing page, since if they don't find it and don't click on it, they will not get paid and will get angry.

Just copy the following lines of code into your final debriefing page for the button.

 

<form method="post" action="https://www.mturk.com/mturk/externalSubmit">

<input type="hidden" name="assignmentId" value="<%=assignId%>" />

<input type="hidden" name="hitId" value="<%=hitId%>" />

<input type="hidden" name="workerId" value="<%=workerId%>" />

<tr><td align="center"><script language="JavaScript" type="text/javascript">writeButton("Finish");</script></td>

</form>

add the following code to the start of the page,and rename it to .jsp

 

 

<%@ taglib uri="/tags" prefix="pi" %> <%@page import="java.util.*, org.uva.*, org.uva.dao.DAOFactory" errorPage="/research/ShowError.jsp"%> <% StudySession studySession = (StudySession) session.getAttribute("studysession"); String params=studySession.getReferer();  String assignId=params.substring(0,params.indexOf("!!!")); String hitId=params.substring(params.indexOf("!!!")+3); String workerId=hitId.substring(hitId.indexOf("!!!")+3); hitId=hitId.substring(0,hitId.indexOf("!!!")); %>

 

Note that when testing this outside of Mturk, clicking on the button will give you an error.  Also, you will need to add &pID=test!!!test2!!!test3 to the launch url to get the debreifing button to display correctly.  So the launch url might end up as 

https://dw2.psyc.virginia.edu/implicit/Launch?study=/user/education/race/race.expt.xml&pID=test!!!test2!!!test3 

Next you will need some general info about the study:

 

title:What kind of website is this?
description:Please look at this website and indicate the type of website.
keywords:category, website
reward:0.02
assignments:1

Location:USA

Once you have everything else tested, you can email Andy Dzik and he will give you a link to the Mturk test server where you can run through your study on mturk and make sure that everything still works there.  The link will look something like:

http://workersandbox.mturk.com/mturk/preview?groupId=2OA57DZKPFE42XWHIHT5O8I9N1XE4W

 

IRB Approval

For UVa folks, the general Project Implicit IRB (Implicit Social Cognition on the Internet) has approval for MTurk studies, so new studies simply need to reference this approval (2003-0173-72), which includes the sample modification and a rationale for using MTurk. Submit a standard modification with any new materials/procedures and reference this protocol for the sample change.

 

Comments

Carlee Beth Hawkins's picture

This sample debriefing page didn't work for me.. Andy helped me troubleshoot, and in addition to adding the above botton to the standard debriefing, the following needs to be added to the top of the file: 

Below this line -- 

<%@ taglib uri="/tags" prefix="pi" %>

 

Add this text -- 


<%@page import="java.util.*, org.uva.*, org.uva.dao.DAOFactory"
errorPage="/research/ShowError.jsp"%>
<% StudySession studySession = (StudySession)
session.getAttribute("studysession");
 String params=studySession.getReferer();
 String assignId=params.substring(0,params.indexOf("!!!"));
 String hitId=params.substring(params.indexOf("!!!")+3);
 String workerId=hitId.substring(hitId.indexOf("!!!")+3);
 hitId=hitId.substring(0,hitId.indexOf("!!!")); %>