PI Research Pool: Design Principles & Policies

Is current?: 

This page covers design guidelines for Project Implicit Research Pool studies. It also has standard pages for informed consent, debriefings, and IAT instructions attached in one zip file, and a sample stand alone study.

Design Principles

This section is an on-going revision of best practices for running studies in the PI research pool. The principles are meant to: 
(a) enhance value of participation to the participant to maintain the vitality of the participant pool 
(b) address key methodological concerns with doing web-based research 
(c) encourage professional, consistent, and attractive study administration procedures

Key issue: Project Implicit participants are volunteers. They are participating in studies out of personal interest and for no other reason. Give them a reason to be bored, annoyed, or confused and they might dropout of the study or even out of the participant pool entirely.

* Study materials should provide clear expectations about what is coming - how much time, how many pages, what kinds of tasks. Setting clear expectations will reduce the "When will this be over".

* Instructions should be clear and brief. Too much, the participant will get bored or just not read it. Too vague, the participant will not do what you expect.

* Participants are diverse. They may never have been to college, they might not be American, and they may never have been asked the kinds of questions that are asked routinely in the laboratory. Do extensive perspective-taking for all of the eligible participants. Include selection rules to avoid getting participants that are not appropriate for the study (e.g., minors, non-U.S. residents or citizens, etc).

* Measures and tasks should engage the participant on interesting topics. And, provide ways for the participant to get something out of the experience. For example, if the study is about attitude malleability, and the topic is irrelevant for the hypothesis, then test attitude malleability on celebrities rather than on furniture.

* Provide feedback of some kind. Studies in the research pool almost always include an implicit measure, so feedback is usually associated with that implicit measure. Participants are there to get something out of the experience - debriefing and feedback is the primary means of meeting this promise to participants.

* Be educational. Explain what you are doing and why. This site integrates education and research. Give them a chance to learn something about psychological science.

* Emphasize the availability of researchers for comments and questions. Another way to engage the volunteers is to invite their comments, reactions, and alternative hypotheses. Always include a researcher email contact on the debriefing page, and reply to ALL messages. If only to thank them for the comment. Participants are our most valuable resource, and they are volunteering their time. Even if they write that they dislike the study, hypothesis or explanation, we owe them for their contribution.

* Keep the study as short as possible. Avoid going longer than 15 minutes.  Give clear indications of how long the study is, and how far they have gone or still have to go.

* Use planned incomplete study designs. Sample size is plentiful, participant time is not. Many complex designs can be implemented by cutting down the number of items that any one participant answers and just collecting data from more participants.

* Use simple language.

* Use a professional and consistent interface. White background, standard site font, clean and simple look and feel. Professionalism in material presentation is a good cue for participants to take the study seriously.

* If a question is not essential, don't ask it. This is not a lab study. Don't add superfluous items. It is much better to have a 5 minute study that provides a satisfying experience, so that they will want to do another, then it is to have a bloated 15 minute study that provides no additional interest value.

* Avoid using pictures of real people - except for public figures such as celebrities and politicians.

* If you want to run a study that will (a) use deception, (b) not use an implicit measure, or (c) violate one of the design principles above, then discuss it with Brian, Tony, or Mahzarin first. These will be considered, but are special cases (e.g., we have never done a web study with deception).


"Try to keep questionnaires to a single screen of questions"  Isn't this not true anymore, given Emily's PLOS One paper?  

Correct. Fixing the main text to reflect this.