I’d like to second Dave’s comments. My research concerns political behavior in/arising from anonymous social locations online with very large user bases (1 million+). It’s a different context with many of the same issues. The illusion of massive amounts of open data: One of the beautiful/horrible things about researching anything online is it is like standing in front of a firehose. The amount of data seems to be enormous, but in line with what Dave stated, often the “good” data—-or in my cases-—very important additional data are not visible without passing through gatekeepers. In the cases of private companies, the gatekeepers are often unwilling to provide this data for proprietary reasons. In the cases of anonymous spaces, it can be difficult to even find who the gatekeepers/moderators/admins are in the masses to inquire about the things you cannot see. And, in the case of splinter communities such as those Dave mentioned-—unless you are part of their “in-group” you may never find yourself invited to view their conversations (or even know they exist). Frustratingly, many of these splinter groups can have huge effects on online spaces-—and be engaging in really weird and interesting behavior that has some serious theoretical importance. (Further, to even be invited/know they exist you'd have to be engaging in participant-observation research--which might not be appropriate/preferable for all researchers.) The ephemeral nature of the data: We think of things on the internet as living forever (photos destroying your chances of employment forever!)—-but in many spaces it does not. Some online spaces are constructed in such a way that information disappears quickly and is difficult/impossible to re-acquire (ex: 4chan, temporary sites to organize protest, etc.) and the information that is left behind is not necessarily to be trusted as there is an awareness that “outsiders” will see it (ex: Encyclopedia Dramatica). Others-—and this feeds into point one-—have a regulation system in which posts just disappear without note or comment. (Friends who study authority online, particularly religion, find this to be challenging.) And, others involve real time conversations of which there is no record-—such as IRC channels. Meaning, if you aren’t watching 24/7 you are bound to miss something that could be important-—and later must go back and attempt to reconstruct it. And, reconstructing can be difficult even if there is a record because it’s possible to find the same event retold with small variations across sources. Anyway, thank you Dave for your post. Best to all, Jessica -------------------------------------- Jessica L. Beyer Doctoral Candidate University of Washington Department of Political Science http://students.washington.edu/jlbeyer -------------------------------------- On Sun, 8 Aug 2010, Dave Karpf wrote:

I'm often frustrated by what I would term a "missing data problem" in computer-mediated qualitative research. I study internet-mediated political organizations in the US (not necessarily web-based, though they all have a web presence). I would generally characterize the field as featuring abundant open data, all of it heavily flawed, and then large swaths of firewalled data, which usually includes the good stuff.

I'll use political blogs as an example. The good news is that these provide you with a time-stamped, readily accessible archive of who said what, to whom (with hyperlinks), using what frames, exactly when. Constructing case studies has gotten a lot easier, and this allows for high quality comparative case analysis. The bad news is that what goes into the networked public sphere through the blogs is only one portion of the communications around any given case.

I'll be presenting a paper next month at the American Political Science Association annual meeting that uses the recent US controversy over the "JournoList" backchannel Google-Group to examine (among other things) the role that backchannels play among political communities-of-interest. Backchannels like JournoList are overwhelmingly common among the US political netroots. They provide a forum for networked organizations and individuals to share information, discuss/debate strategy, vent frustration, etc. The communication that occurs on these lists is entirely "missing data," for three reasons:

1. Restricted access list. These aren't public lists, they don't show up in google-group searches. Membership is provided to people who are invited by the list moderator. Moderators are free to establish list rules, which generally include an off-the-record/do-not-forward requirement. Mods have little enforcement power (public shaming, kick members off the list), but it still makes it impossible for the researcher to find the list unless s/he meets the right person, and even then the researcher probably only gets invited on if they agree not to publish anything off of those communications.

2. The "Fight Club rule." Most, if not all of, these backchannels include a rule that list members will not discuss the list in public or to the press. JournoList provides a cautionary tale for why this rule is used -- once a restricted-access list becomes publicly known, it is easy for opponents to dramatize it as the launching grounds for secret conspiracies.

3. The "Russian Nesting Doll effect." Since anyone can launch a Google-Group (it's a 2-screen setup process), once a single list gets large, individual members will break off and form their OWN also-secret sub-list. There are lists within lists, and thanks to the Fight Club rule, it is technically impossible to determine how many of them there are in total.

A lot of that backchannel communication (like a lot of blog content) is basically noise. But some of it is vitally important deliberation or coordination. It is, in other words, *communications infrastructure* for communities-of-interest, mimicking the affordances provided by formal organizational listservs. And swimming in the abundance of available blog data, its easy for us to pretend as though "everything is public/open." We're missing huge swaths of communication, both within organizations and within these networked backchannels. As a qualitative researcher, that pretty seriously worries me. I have a couple of crazy ideas for how to cobble together a research project on these backchannels, but it'll be six months at least before it's a solid enough plan to share publicly.

[As someone who dabbles in quantitative research as well, I'll note that the problems are even bigger there. So much data, all of it low-quality. Then there's plenty of high-quality data, but it's all firewalled/proprietary. I could go on a much longer rant about another study I have under way, of email communications from 70 advocacy groups, and just how much more data they possess and share with each other than will ever be made publicly available to you and me.]

...That was a much longer response than I'd intended to provide. I'm pretty animated about backchannel lists these days, hope 2 or 3 people found it to be an interesting tangent. -Dave

On Sat, Aug 7, 2010 at 4:01 PM, Mohammad H. Hasani <mh_hasani@yahoo.com>wrote:

...

--- On Sat, 8/7/10, Mohammad H. Hasani <mh_hasani@yahoo.com> wrote:

From: Mohammad H. Hasani <mh_hasani@yahoo.com> Subject: Re: [Air-L] Missing Data in Qualitative and Online Research To: "William Dutton" <william.dutton@oii.ox.ac.uk> Date: Saturday, August 7, 2010, 12:59 PM

Generally speaking,

1. I think missing data in a Qual research is data could make a significant contribution to the grounded concept or theory but researcher neglects or fails to collect. Repeating some phases in data collection/analysis, conducting parallel Qual research or reverse previewing of the research phases seems to help.

2. The same for CMR, but here, the researcher should also pay more attention to the cases; inappropriate cases may cause collecting poor data could be seen as missed.

Triangulated observations may help in this case.

Mohammad H. Hasani,

Instructor in Sociology

Payame Noor University

--- On Fri, 8/6/10, William Dutton <william.dutton@oii.ox.ac.uk> wrote:

From: William Dutton <william.dutton@oii.ox.ac.uk> Subject: [Air-L] Missing Data in Qualitative and Online Research To: "air-l@listserv.aoir.org" <air-l@listserv.aoir.org> Date: Friday, August 6, 2010, 8:17 AM

Dear List Members,

My colleagues and I would greatly appreciate your input to an exploratory project on missing data in qualitative and online research.* We would appreciate anything from your immediate reactions while reading this e-mail to detailed references to literature that has addressed our questions.

There are two very general but heuristically useful questions. Your views on either would be welcomed. Feel free to reply privately or to the list, as you wish.

1. What is ‘missing data’ in the context of qualitative research and how is it dealt with?

2. Likewise, in computer-mediated research, are researchers missing particular kinds of data, or believe that they are missing particular kinds of observations, and how are they compensating or otherwise addressing this gap?

Thoughts? Thank you,

Bill

*This is a collaborative project between the ESRC’s National Centre for Research Methods (NCRM) ‘hub’ (Graham Crow, Rose Wiles), WISERD (Amanda Coffey), Oxford eSocial Science ‘node’ of NCeSS (Bill Dutton, Alison Powell), and Qualidata/Timescapes (Libby Bishop), based on our recognition of a shared but not well developed problem.

William Dutton, Director Professor of Internet Studies Oxford Internet Institute 1 St Giles', Oxford OX1 3JS UK

e-mail: william.dutton at oii.ox.ac.uk Web: http://www.oii.ox.ac.uk/people/?id=1 Phone: +44 (0)1865 287 212 Cell: +44 (0)7768 823906

_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org

Join the Association of Internet Researchers: http://www.aoir.org/

_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org

Join the Association of Internet Researchers: http://www.aoir.org/

-- Dave Karpf, PhD

Assistant Professor Journalism and Media Studies Department School of Communication and Information Rutgers University, New Brunswick

www.davidkarpf.com davekarpf@gmail.com _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org

Join the Association of Internet Researchers: http://www.aoir.org/