2/25/08 TO: Political Science Methodolgy-POLMETH Association of Internet Researchers-Air-L Please find attached the consolidated submissions from my inquiry: “I am looking for reference material for use in my work at EPA. We are planning to launch a discussion board with a limited number of comment fields. But we also want to analyze discussion content in order to better understand the needs and interests of our customers--in particular those who come to EPA seeking environmental information. 1) Have any of you been teaching about content analysis methods lately? If so, what's your favorite reference? 2) Has anyone been working on content analysis of discussion boards as a research method?” Thanks to everyone for all the helpful responses! Content Analysis Reference 3) Sent by: Harald Klein <intext@GMX.DE> , Alexander Semenov <semenoffalex@googlemail.com if you are looking for an overview on content analysis research methodes, Klaus Krippendorff's book is excellent: Krippendorff, Klaus (2004): Content Analysis. An Introduction ot Its Methodology. Thousand Oaks. Sage. 411 pages 2) Sent by: "Callahan, Ewa S. Prof." <Ewa.Callahan@quinnipiac.edu> Following Alexander's note, I am a big fan of Krippendorff, especially his method of intercoder reliability testing. For his, and other methods you can check: http://www.temple.edu/sct/mmc/reliability/ followed up by: http://www.asc.upenn.edu/usr/krippendorff/dogs.html Content Analysis Research 0) Sent by: Muhammad Abdul-Mageed <mumageed@yahoo.com> Herring, S. C. (2007). Web(log) content analysis: Expanding the paradigm. Submitted to J. Hunsinger, M. Allen, & L. Klastrup (Eds.), The International Handbook of Internet Research. Springer Verlag. 1) Sent by: James Honaker <tercer@UCLA.EDU> There is a project that is quite close to what you want at the ISI at USC. Their page, with a number of project papers and abstracts, is: http://www.isi.edu/division3/discourse/ Their nlp goals are analysis of classroom discussion boards to create some degree of automated answering of student questions, direction to previous threads, and links between students with similar interests or research problems. The project is specifically about discussion boards commonly used for class teaching, but other than the classroom focus it is very close to the subject you are seeking. (Full disclosure, my wife worked on this project.) 2) Sent by : Henry Kim <h27kim@EMAIL.ARIZONA.EDU> There is a group called "Dark Web" here at Univ of AZ where computer scientists are doing content analysis on web sites connected to terrorism and more. Some of the techniques they are using may be of use for your purposes. Content Analysis Teaching 1) Sent by: Kevin Guidry <krguidry@gmail.com> , Muhammad Abdul-Mageed <mumageed@yahoo.com> , Alexander Semenov <semenoffalex@googlemail.com> You may like to have a look at the course description, and syllabus, of Prof. Susan Herring in Indiana University, for her course titled "Content Analysis for the World Wide Web". Herring has been teaching it for a long time and she has been trying to develop methodologies for analyzing the WWW. Web site at http://www.slis.indiana.edu/faculty/herring/. Specifically: http://ella.slis.indiana.edu/~herring/web.syll.06.html The following article is her last written paper on the issue, as far as I know: Herring, S. C. (2007). Web(log) content analysis: Expanding the paradigm. Submitted to J. Hunsinger, M. Allen, & L. Klastrup (Eds.), The International Handbook of Internet Research. Springer Verlag. Content Analysis Software 0) Sent by: Harald Klein <intext@GMX.DE> An overview on text analysis software gives http://www.textanalysis.info 1) Sent by: Paul Kellstedt <kellstedt@politics.tamu.edu> I'm the co-editor of The Political Methodologist. Tyler Johnson wrote a review for us of the software package WordStat, which he liked. You could find the back issue of TPM as a .pdf at the polmeth web site. http://polmeth.wustl.edu/thepolmeth.php 2) Sent by: Michael Evans <evans.michael@gmail.com Over the past few years I have experimented with various methods of computer-assisted textual data analysis, including traditional and new approaches to content analysis. In the process, I have developed an expertise in two particularly useful software packages produced by Provalis Research: QDA Miner and WordStat. Provalis' approach is to closely integrate several methods of textual data analysis that have traditionally (and misleadingly) been thought of as solely "qualitative" or "quantitative." The result is a mixed-methods approach that does not merely provide the convenience of having several quantitative and qualitative functions in one package, but also allows for creative synthesizing of the functions in order to accomplish entirely new analytical tasks. You can learn more about QDA Miner here (http://www.provalisresearch.com/QDAMiner/QDAMinerDesc.html) and WordStat here (http://www.provalisresearch.com/wordstat/Wordstat.html). Related Topic – Analysis of Public Comments on Regulations 1) Sent by: schrodt <schrodt@ku.edu> Contact Stuart Shulman at the University of Pittsburgh -- he has some quite sophisticated software that does this sort of thing with public input on proposed regulations, and I assume it would work for your project as well. In fact I think he has already done contract work for EPA -- or at least I know he has analyzed environmental regulations 2) Sent by: John D Wilkerson <jwilker@u.washington.edu> There are a lot of people working on public commenting in agencies in the field of information sciences. you should contact Stu Shulman (shulman@pitt.edu) who works with Jamie Callan. But as far as I know this is a pretty rich area of research in the e-government field. 3) Sent by: Russell Winn <ruwinn@nmsu.edu> Last summer I worked with public comments about designating Wilderness Areas in southern New Mexico. The approach we took was to treat each comment (e-mail, letter, etc.) as a separate document and then used NVIVO to code and organize the responses(NVIVO is a nice program for analyzing qualitative data but there are other programs as well). Most of the ‘hand coding’ was done by a group of graduate students who were enrolled in an Environmental Policy class I was teaching at the time. As a class we reviewed some of the documents, met with staff from the agency that had collected the comments, and drew up a list of potential codes (NVIVO calls them nodes) along with a brief description of when that node would be appropriate. I had every document separately coded by 2 students. If there was a disagreement about the coding the students met and agreed on a coding. Even with this reliability check there was some confusion about the coding. I ended up recoding many of the documents myself. If I were doing this again I would probably do all the coding myself, or go through a more lengthy training with the students that involved group coding of a number of the documents. In the end we had too many nodes and the comments were a bit difficult to analyze although we did find some interesting patterns. Your project sounds more limited looking for needs and interests. 4) Sent by: Stephen Purpura <stevepurpura@gmail.com> I'm a PhD student at Cornell (and graduate of the Kennedy School of Government) and I pretty much just work on automating content analysis of government data. I have two papers set for publication at dgo.2008 (digital government research) on automating finding issues in regulatory rule comments (I can send the drafts) and another in press with JITP (see http://www.stephenpurpura.com/20080116_HillardPurpuraWilkerson-revised.pdf). My other publications are listed at http://www.purpuras.net/publications.html. Laura B. Nielsen, Ph.D. Adjunct Professor of Public Policy Georgetown Public Policy Institute 3520 Prospect Street, Suite 308F Washington, DC 20007 WEB http://www.georgetown.edu/faculty/lbn4/ Program Analyst Environmental Analysis Division, MC 2842T Office of Information Analysis and Access Office of Environmental Information U.S. Environmental Protection Agency (202) 566-0621