I'm often frustrated by what I would term a "missing data problem" in computer-mediated qualitative research. I study internet-mediated political organizations in the US (not necessarily web-based, though they all have a web presence). I would generally characterize the field as featuring abundant open data, all of it heavily flawed, and then large swaths of firewalled data, which usually includes the good stuff. I'll use political blogs as an example. The good news is that these provide you with a time-stamped, readily accessible archive of who said what, to whom (with hyperlinks), using what frames, exactly when. Constructing case studies has gotten a lot easier, and this allows for high quality comparative case analysis. The bad news is that what goes into the networked public sphere through the blogs is only one portion of the communications around any given case. I'll be presenting a paper next month at the American Political Science Association annual meeting that uses the recent US controversy over the "JournoList" backchannel Google-Group to examine (among other things) the role that backchannels play among political communities-of-interest. Backchannels like JournoList are overwhelmingly common among the US political netroots. They provide a forum for networked organizations and individuals to share information, discuss/debate strategy, vent frustration, etc. The communication that occurs on these lists is entirely "missing data," for three reasons: 1. Restricted access list. These aren't public lists, they don't show up in google-group searches. Membership is provided to people who are invited by the list moderator. Moderators are free to establish list rules, which generally include an off-the-record/do-not-forward requirement. Mods have little enforcement power (public shaming, kick members off the list), but it still makes it impossible for the researcher to find the list unless s/he meets the right person, and even then the researcher probably only gets invited on if they agree not to publish anything off of those communications. 2. The "Fight Club rule." Most, if not all of, these backchannels include a rule that list members will not discuss the list in public or to the press. JournoList provides a cautionary tale for why this rule is used -- once a restricted-access list becomes publicly known, it is easy for opponents to dramatize it as the launching grounds for secret conspiracies. 3. The "Russian Nesting Doll effect." Since anyone can launch a Google-Group (it's a 2-screen setup process), once a single list gets large, individual members will break off and form their OWN also-secret sub-list. There are lists within lists, and thanks to the Fight Club rule, it is technically impossible to determine how many of them there are in total. A lot of that backchannel communication (like a lot of blog content) is basically noise. But some of it is vitally important deliberation or coordination. It is, in other words, *communications infrastructure* for communities-of-interest, mimicking the affordances provided by formal organizational listservs. And swimming in the abundance of available blog data, its easy for us to pretend as though "everything is public/open." We're missing huge swaths of communication, both within organizations and within these networked backchannels. As a qualitative researcher, that pretty seriously worries me. I have a couple of crazy ideas for how to cobble together a research project on these backchannels, but it'll be six months at least before it's a solid enough plan to share publicly. [As someone who dabbles in quantitative research as well, I'll note that the problems are even bigger there. So much data, all of it low-quality. Then there's plenty of high-quality data, but it's all firewalled/proprietary. I could go on a much longer rant about another study I have under way, of email communications from 70 advocacy groups, and just how much more data they possess and share with each other than will ever be made publicly available to you and me.] ...That was a much longer response than I'd intended to provide. I'm pretty animated about backchannel lists these days, hope 2 or 3 people found it to be an interesting tangent. -Dave On Sat, Aug 7, 2010 at 4:01 PM, Mohammad H. Hasani <mh_hasani@yahoo.com>wrote:
--- On Sat, 8/7/10, Mohammad H. Hasani <mh_hasani@yahoo.com> wrote:
From: Mohammad H. Hasani <mh_hasani@yahoo.com> Subject: Re: [Air-L] Missing Data in Qualitative and Online Research To: "William Dutton" <william.dutton@oii.ox.ac.uk> Date: Saturday, August 7, 2010, 12:59 PM
Generally speaking,
1. I think missing data in a Qual research is data could make a significant contribution to the grounded concept or theory but researcher neglects or fails to collect. Repeating some phases in data collection/analysis, conducting parallel Qual research or reverse previewing of the research phases seems to help.
2. The same for CMR, but here, the researcher should also pay more attention to the cases; inappropriate cases may cause collecting poor data could be seen as missed.
Triangulated observations may help in this case.
Mohammad H. Hasani,
Instructor in Sociology
Payame Noor University
--- On Fri, 8/6/10, William Dutton <william.dutton@oii.ox.ac.uk> wrote:
From: William Dutton <william.dutton@oii.ox.ac.uk> Subject: [Air-L] Missing Data in Qualitative and Online Research To: "air-l@listserv.aoir.org" <air-l@listserv.aoir.org> Date: Friday, August 6, 2010, 8:17 AM
Dear List Members,
My colleagues and I would greatly appreciate your input to an exploratory project on missing data in qualitative and online research.* We would appreciate anything from your immediate reactions while reading this e-mail to detailed references to literature that has addressed our questions.
There are two very general but heuristically useful questions. Your views on either would be welcomed. Feel free to reply privately or to the list, as you wish.
1. What is ‘missing data’ in the context of qualitative research and how is it dealt with?
2. Likewise, in computer-mediated research, are researchers missing particular kinds of data, or believe that they are missing particular kinds of observations, and how are they compensating or otherwise addressing this gap?
Thoughts? Thank you,
Bill
*This is a collaborative project between the ESRC’s National Centre for Research Methods (NCRM) ‘hub’ (Graham Crow, Rose Wiles), WISERD (Amanda Coffey), Oxford eSocial Science ‘node’ of NCeSS (Bill Dutton, Alison Powell), and Qualidata/Timescapes (Libby Bishop), based on our recognition of a shared but not well developed problem.
William Dutton, Director Professor of Internet Studies Oxford Internet Institute 1 St Giles', Oxford OX1 3JS UK
e-mail: william.dutton at oii.ox.ac.uk Web: http://www.oii.ox.ac.uk/people/?id=1 Phone: +44 (0)1865 287 212 Cell: +44 (0)7768 823906
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
-- Dave Karpf, PhD Assistant Professor Journalism and Media Studies Department School of Communication and Information Rutgers University, New Brunswick www.davidkarpf.com davekarpf@gmail.com