Study Development Guide

This page offers a detailed guide on how to create a study to run on Project Implicit. For a broad overview of all the steps involved in producing and launching your study, go to the Project Implicit research checklist.  

Before You Start

  1. Download the programs you'll need to program your study for PI. While the instructions are written for jEdit and Cyberduck respectively, you are able to use other ftp clients or text editors if that is your preference. Learn more at PI tools: Installation instructions.
  2. Use Cyberduck or another ftp client to connect to the dw2 server. Even if you do use a different client, the process for connecting should be the same.
  3. Learn some tips and tricks for the jEdit program.
  4. Learn about HTML and Javascript, which you will be using to program your study.

Project Implicit Study Conventions and Required Elements

  1. Learn the PI rules for naming folders, files, tasks, and variables so that your study can run properly. Failing to properly label elements of your study can result in numerous possible issues. 
  2. All studies need require realstart, lastpageconsent and debriefing pages.
  3. Examples of each of these pages can be found in the sample study
  4. Studies that do not include these elements will be removed from the research pool until they are added.

Organizing your study on the dev2 server

  1. All study files must be in a single folder that is for one and only one study[3]
  2.  File names for study materials should have informative yet concise titles.[4]   
  3. File names must be in lowercase letters, and cannot start with a number.[5]
  4. Only the final versions of files associated with your study should be in the study folder stored on the test and production servers.[6]
  5. If your study includes images, these images should be in a subfolder of the study folder entitled “images”

HTML programming

  1. All survey forms you create will include the HTML form element. Learn how to use it here.
  2. All studies need consent and debriefing pages.
  3. You may want to instruction pages to guide participants and help prevent attrition.
  4. You might also want to add an instructional manipulation check to ensure that participants are paying close attention to your study.
  5. There are many other elements you can program with HTML:
  6. Sample study: click here for a sample study and all its files:

Experiment Files

  1. Learn how to organize your study files into an experiment file, in which you'll define and map all of the tasks that participants will do. That page is redundant with this page on Experiment File Basics.
  2. Check out basic, moderately complex (with counterbalancing), and advanced (with "branching") examples of experiment files.
  3. Get more info on branching here.

Other & advanced programming

  1. JSP is another kind of file you can use to program questionnaires. Click here to learn how, and learn advanced techniques here.
  2. There are several types of implicit measures you can use. You can add an IAT to your study that already exists, or learn the XML code to create a race IAT. You can also learn advanced techniques to alter the IAT XML code, to use self-concept IATs, picture IATs, and more.
  3. You can also create a study login for certain participantsbut the page does not tell you how.
  4. You can also request email addresses from participants, for two-phase studies.
  5. Advanced study development topics include programming a cognitive load task, adding video to your experiment, and converting your study to Amazon MTurk.

Study Testing

1.  Validate all study files to confirm correct coding

2. Test that each study component appears and operates correctly in Firefox, Chrome, Safari, and Internet Explorer browsers.[7]

3. Confirm consistent formatting across all tasks, e.g. equal margins, consistent font, etc.

4. Test that data is recording correctly for each study component.  This includes: sending data to the database, following your variable naming decisions without overlapping variable names, and having the correct response options for each variable. To check the test data for a single page, use httpfox. To check the data for an entire study, use the Research Design Environment.

5. For studies that provide feedback to participants, test that the results show up in the debriefing and that the result is correct.[9]

6. Test every condition of the study from start to finish, ensuring that (a) all tasks that should show up do show up, (b) tasks appear in the expected order, (c) randomization is occurring when expected, and (d) random selection is occurring as expected.[10]

7. Confirm that the study ID is unique, i.e., not redundant with any other studies done on Project Implicit.[11]

8. Consider downloading Httpfox, an add-on for the Firefox browser to check that your data is being recorded correctly.

Study Approval

1. Obtain approval for the final, tested study from all collaborators before moving it to production. 

2. For studies to go into the Project Implicit research pool, send the validator link (see first bullet point under study testing) to to get feedback and, ultimately approval, from one of the research pool user experience reviewers: Kate Ratliff or Colin Smith. In your email, include the validator link for the study, your planned inclusion/exclusion rules, the planned sample size and justifcation for the sample size.[12]

Submitting/Moving Study to Production

1. You may need to restrict participation based on demographic or other variables. Learn how here.

2. Fill out and submit the Study Submission form. 



[1] Instructions and tasks should be clear, concise, and attentive to the capacity of the sample.  Study materials are usually created and partially tested on the researcher’s own machine and then moved to the test server (pi or dw2) for additional testing

[2] If your experiment file has several conditions, it is wise to create separate experiment files for each condition for easier testing later

[3] The only exception are files that are in the common folder.  Those files are used but not edited by individual researchers

[4] The experiment file itself should have the study name as part of its file name (e.g., affect2.expt.xml) but other files do not need to follow this practice.

[5] Avoid special characters (&,%,~,_), except for hyphens (-) and the occasional period (used only in the experiment file, e.g., affect2.expt.xml).

[6] You can create a pilot folder for preparation and then a final folder for the study materials. IMPORTANT: Test the study AFTER all extraneous files are removed from the study folder.

[7] Functionality testing can require some time - does each part operate correctly? Reusing existing materials can increase confidence that it will operate correctly across browsers.

[9]It is possible, for example, for the IAT feedback to be presented opposite the actual result if the IAT.xml file is not correct.  Testers should deliberately obtain a particular result and confirm that it is reported accurately.  This is particularly important for new IAT files, and one IAT file could be erroneous while others in the study are correct.

[10] Often this step requires running through the study many times quickly to check randomization.

[11] Use your username as part of the studyID.  That way, you need only compare against other studies that you have conducted.

[12] Visit the design principles wiki entry to anticipate what these reviewers need to see for studies to be approved for the researchers pool.