Soliciting opinions about using Amazon's Mechanical Turk (aka Mturk) for survey participant recruitment
While Qualtrics and SurveyMonkey seem to be generally well respected, they also come with hefty price tags for participant recruitment ($5-10 per American respondent). Amazon’s Mechanical Turk (aka Mturk), on the other hand, can recruit participants for far less (<$1/respondent). What is the AoIR consensus on utilizing Mturk? I’ve seen articles both for and against using the service. Thanks in advance for all opinions/insights. - Cory Robinson -- Stephen Cory Robinson cory.robinson@colostate.edu<mailto:cory.robinson@colostate.edu> Office: Clark C258A http://colostate.academia.edu/StephenCoryRobinson
Hi Cory, Part of it depends on the type of data you are collecting. What is your research question? Do you need follow-up? How important is it that you have some way to verify that people are who they claim to be or in the location in they claim to be? I've been given to understand that the population that mechanical comes from tends to be younger and tends to have a lower income (please correct me if I am wrong). Is this a population you are interested in generalizing from? Another issue is data protection. I work in a medical school. We are very concerned with HIPAA and how we store data. Though I might be able to use Mturk for some purposes, I wouldn't ask about many health-related behaviors. However, we are seriously considering using Mturk to do some very preliminary pretesting on a couple of health-related messages geared toward young adults (this will be followed by other, more in-depth experiments). This probably doesn't help at all but at least if you clarify your research questions other people who know more can give you better help. Ashley -----Original Message----- From: air-l-bounces@listserv.aoir.org [mailto:air-l-bounces@listserv.aoir.org] On Behalf Of Robinson,Cory Sent: Wednesday, January 08, 2014 4:40 PM To: air-l@listserv.aoir.org Subject: [Air-L] Soliciting opinions about using Amazon's Mechanical Turk (aka Mturk) for survey participant recruitment While Qualtrics and SurveyMonkey seem to be generally well respected, they also come with hefty price tags for participant recruitment ($5-10 per American respondent). Amazon's Mechanical Turk (aka Mturk), on the other hand, can recruit participants for far less (<$1/respondent). What is the AoIR consensus on utilizing Mturk? I've seen articles both for and against using the service. Thanks in advance for all opinions/insights. - Cory Robinson -- Stephen Cory Robinson cory.robinson@colostate.edu<mailto:cory.robinson@colostate.edu> Office: Clark C258A http://colostate.academia.edu/StephenCoryRobinson _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/
If you're doing experiments, MTurk is great, especially if your likely alternative is a sample of undergrads. If you're doing surveys, it's not. The U.S. users are not representative of the American population, and I'd imagine the same is true of other countries' MTurk populations. And even if it were, there's no way to prove that to any kind of satisfaction. Basically, using it for a survey that you want to generalize to the population isn't getting you a sample that's any better than posting the survey to Facebook and Twitter and asking people to spread it around. FWIW, the skew I've seen with MTurk samples compared to the U.S. population is a) slightly too male, b) slightly too white, c) too young (though older and with more age variance than an undergrad sample), d) average income too low, and e) too liberal (though, again, less skew than I'd expect from undergrads). Aaron On Wed, Jan 8, 2014 at 6:40 PM, Robinson,Cory <cory.Robinson@colostate.edu> wrote:
While Qualtrics and SurveyMonkey seem to be generally well respected, they also come with hefty price tags for participant recruitment ($5-10 per American respondent). Amazon’s Mechanical Turk (aka Mturk), on the other hand, can recruit participants for far less (<$1/respondent).
What is the AoIR consensus on utilizing Mturk? I’ve seen articles both for and against using the service.
Thanks in advance for all opinions/insights.
- Cory Robinson -- Stephen Cory Robinson cory.robinson@colostate.edu<mailto:cory.robinson@colostate.edu> Office: Clark C258A http://colostate.academia.edu/StephenCoryRobinson
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
-- Aaron S. Veenstra Assistant Professor, Southern Illinois University Carbondale School of Journalism || 1232 Comm Building asveenstra@siu.edu || manytoomany.com
Additionally, keep in mind the nature of Mechanical Turk; is there an incentive for the specific group you are targeting to deliver high quality entries? With the amounts of projects executed on a weekly basis, problems of satisficing and routine knowledge of internal tests occur. I find this for surveys quite an important point to consider. See for more: Kapelner, A., & Chandler, D. (2010). Preventing Satisficing in online surveys. *----* Tim *W: *www.timmuntinga.com 2014/1/9 Aaron S. Veenstra <aaron@etchouse.com>
If you're doing experiments, MTurk is great, especially if your likely alternative is a sample of undergrads. If you're doing surveys, it's not. The U.S. users are not representative of the American population, and I'd imagine the same is true of other countries' MTurk populations. And even if it were, there's no way to prove that to any kind of satisfaction. Basically, using it for a survey that you want to generalize to the population isn't getting you a sample that's any better than posting the survey to Facebook and Twitter and asking people to spread it around.
FWIW, the skew I've seen with MTurk samples compared to the U.S. population is a) slightly too male, b) slightly too white, c) too young (though older and with more age variance than an undergrad sample), d) average income too low, and e) too liberal (though, again, less skew than I'd expect from undergrads).
Aaron
On Wed, Jan 8, 2014 at 6:40 PM, Robinson,Cory <cory.Robinson@colostate.edu> wrote:
While Qualtrics and SurveyMonkey seem to be generally well respected, they also come with hefty price tags for participant recruitment ($5-10 per American respondent). Amazon’s Mechanical Turk (aka Mturk), on the other hand, can recruit participants for far less (<$1/respondent).
What is the AoIR consensus on utilizing Mturk? I’ve seen articles both for and against using the service.
Thanks in advance for all opinions/insights.
- Cory Robinson -- Stephen Cory Robinson cory.robinson@colostate.edu<mailto:cory.robinson@colostate.edu> Office: Clark C258A http://colostate.academia.edu/StephenCoryRobinson
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
-- Aaron S. Veenstra Assistant Professor, Southern Illinois University Carbondale School of Journalism || 1232 Comm Building asveenstra@siu.edu || manytoomany.com _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
Other articles that address this issue are Berinsky, Huber, and Lenz 2012 (Political Analysis) and Chandler, Mueller, and Paolacci 2013 (Behavioral Research Methods). This is a worthwhile blog post in thinking through some of the issues too - http://www.culturalcognition.net/blog/2013/7/10/fooled-twice-shame-on-who-pr... . Broadly I agree with others that it depends on the question you're asking. On Wed, Jan 8, 2014 at 8:09 PM, Tim Muntinga <munt.tim@gmail.com> wrote:
Additionally, keep in mind the nature of Mechanical Turk; is there an incentive for the specific group you are targeting to deliver high quality entries? With the amounts of projects executed on a weekly basis, problems of satisficing and routine knowledge of internal tests occur. I find this for surveys quite an important point to consider.
See for more: Kapelner, A., & Chandler, D. (2010). Preventing Satisficing in online surveys.
*----* Tim *W: *www.timmuntinga.com
2014/1/9 Aaron S. Veenstra <aaron@etchouse.com>
If you're doing experiments, MTurk is great, especially if your likely alternative is a sample of undergrads. If you're doing surveys, it's not. The U.S. users are not representative of the American population, and I'd imagine the same is true of other countries' MTurk populations. And even if it were, there's no way to prove that to any kind of satisfaction. Basically, using it for a survey that you want to generalize to the population isn't getting you a sample that's any better than posting the survey to Facebook and Twitter and asking people to spread it around.
FWIW, the skew I've seen with MTurk samples compared to the U.S. population is a) slightly too male, b) slightly too white, c) too young (though older and with more age variance than an undergrad sample), d) average income too low, and e) too liberal (though, again, less skew than I'd expect from undergrads).
Aaron
On Wed, Jan 8, 2014 at 6:40 PM, Robinson,Cory <cory.Robinson@colostate.edu> wrote:
While Qualtrics and SurveyMonkey seem to be generally well respected, they also come with hefty price tags for participant recruitment ($5-10 per American respondent). Amazon’s Mechanical Turk (aka Mturk), on the other hand, can recruit participants for far less (<$1/respondent).
What is the AoIR consensus on utilizing Mturk? I’ve seen articles both for and against using the service.
Thanks in advance for all opinions/insights.
- Cory Robinson -- Stephen Cory Robinson cory.robinson@colostate.edu<mailto:cory.robinson@colostate.edu> Office: Clark C258A http://colostate.academia.edu/StephenCoryRobinson
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
-- Aaron S. Veenstra Assistant Professor, Southern Illinois University Carbondale School of Journalism || 1232 Comm Building asveenstra@siu.edu || manytoomany.com _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
hi, i believe there are also ethical concerns. i am not sure if this is what you were also asking about but i hope they are nevertheless useful. on orientalist underpinnings of the concept and labor issues: “Return of The Crowds: Mechanical Turk and Neoliberal States of Exception” in Digital Labor:The Internet as Playground and Factory. Ed. Trebor Scholz, Routledge, 2012. last year some researchers raised concerns about anonymity and irb issues related to using amazon mechanical turk: Blog post: http://crowdresearch.org/blog/?p=5177 Paper: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2228728 i also remember hearing about an amazon turk workers union that demanded better wages and better working conditions. here is a paper on a tool that was designed to support the process: http://www.ics.uci.edu/~lirani/Irani-Silberman-Turkopticon-camready.pdf and another article with respect to the problem with the labor conditions created by amazon and other similar crowdsourcing services: http://economix.blogs.nytimes.com/2013/03/18/the-unregulated-work-of-mechani... i am not sure if any of the ethical and labor related issues have been addressed over the last months and what ethical questions these raise for using mechanical turk in studies. i am sure there are others on this list who have a lot more to say. good luck! s. On Jan 9, 2014, at 10:42 AM, Leticia Bode <lb871@georgetown.edu> wrote:
Other articles that address this issue are Berinsky, Huber, and Lenz 2012 (Political Analysis) and Chandler, Mueller, and Paolacci 2013 (Behavioral Research Methods). This is a worthwhile blog post in thinking through some of the issues too - http://www.culturalcognition.net/blog/2013/7/10/fooled-twice-shame-on-who-pr... . Broadly I agree with others that it depends on the question you're asking.
On Wed, Jan 8, 2014 at 8:09 PM, Tim Muntinga <munt.tim@gmail.com> wrote:
Additionally, keep in mind the nature of Mechanical Turk; is there an incentive for the specific group you are targeting to deliver high quality entries? With the amounts of projects executed on a weekly basis, problems of satisficing and routine knowledge of internal tests occur. I find this for surveys quite an important point to consider.
See for more: Kapelner, A., & Chandler, D. (2010). Preventing Satisficing in online surveys.
*----* Tim *W: *www.timmuntinga.com
2014/1/9 Aaron S. Veenstra <aaron@etchouse.com>
If you're doing experiments, MTurk is great, especially if your likely alternative is a sample of undergrads. If you're doing surveys, it's not. The U.S. users are not representative of the American population, and I'd imagine the same is true of other countries' MTurk populations. And even if it were, there's no way to prove that to any kind of satisfaction. Basically, using it for a survey that you want to generalize to the population isn't getting you a sample that's any better than posting the survey to Facebook and Twitter and asking people to spread it around.
FWIW, the skew I've seen with MTurk samples compared to the U.S. population is a) slightly too male, b) slightly too white, c) too young (though older and with more age variance than an undergrad sample), d) average income too low, and e) too liberal (though, again, less skew than I'd expect from undergrads).
Aaron
On Wed, Jan 8, 2014 at 6:40 PM, Robinson,Cory <cory.Robinson@colostate.edu> wrote:
While Qualtrics and SurveyMonkey seem to be generally well respected, they also come with hefty price tags for participant recruitment ($5-10 per American respondent). Amazon’s Mechanical Turk (aka Mturk), on the other hand, can recruit participants for far less (<$1/respondent).
What is the AoIR consensus on utilizing Mturk? I’ve seen articles both for and against using the service.
Thanks in advance for all opinions/insights.
- Cory Robinson -- Stephen Cory Robinson cory.robinson@colostate.edu<mailto:cory.robinson@colostate.edu> Office: Clark C258A http://colostate.academia.edu/StephenCoryRobinson
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
-- Aaron S. Veenstra Assistant Professor, Southern Illinois University Carbondale School of Journalism || 1232 Comm Building asveenstra@siu.edu || manytoomany.com _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
participants (6)
-
Aaron S. Veenstra -
Ashley Nicoles Sanders-Jackson -
Leticia Bode -
Robinson,Cory -
Seda Gurses -
Tim Muntinga