- Air-L - lists.aoir.org

Software to extract content of Facebook & Twitter (some conclusions)
by Cristina Aced 28 Aug '14

28 Aug '14

Hi everyone! and thank you for your suggestions!! Many thanks José Manuel, Noha, Tijana, Stu, Tobias, Patricia, Harju, Craig, Tim & Bernhard! As promised, here you have my first impressions of the options you have suggested to extract data from Facebook. As I explained in previous messages, I need to gather the content published by companies on blogs, Twitter and Facebook (posts, tweets, status) with their date information and all the feedback received: number of comments/ RT/ mentions/ likes /shares and so on. After considering all your suggestions and trying some of them: • I'm using *OutWit Hub* <http://www.outwit.com/> (the free version is very complete) to scrape blogs and Twitter accounts. It is necessary to create scrapers to gather the data you want but it is easy to learn how it works and you can personalize the scraping. It's a great tool but it doesn't work properly with Facebook due to Facebook's API limitations :( • *NodeXL* is a useful tool specially for analyzing connections between users/ contents on social media and for creating amazing graphs. It also downloads status updates but the way it shows the information is a little bit chaotic (i.e. it is not easy to connect the status update with the comments it has received). Another interesting option: it is possible to get demographic information about FB's users (genre, location...). • *NVivo* is fantastic to work with Facebook and Twitter. Very easy to extract FB's status updates, date & time, comments and number of likes. To sum up: I have decided to use OutWit Hub to work with blogs and Twitter and NVivo to work with Facebook. If you have any doubt about this topic, maybe I can help you, so feel free to send your doubts and I'll try to answer them! *Cristina Aced * PhD Candidate Doctoral Programme in Knowledge and Information Society Universitat Oberta de Catalunya | caced(a)uoc.edu www.cristinaaced.com/blog | Twitter: @blogocorp

1 0

Re: [Air-L] Software to extract content of Facebook & Twitter
by Tim Libert 28 Aug '14

28 Aug '14

I’d quickly point out two additional considerations when ingesting fb/twitter data: 1) APIs generally exclude ads (which are ‘targeted’) - so depending on what you want to study and/or model an API will never give you an accurate view of what users really see. APIs are easy, but incomplete. 2) The trick with scraping content directly from the web is accounting for processing/executing javascript as that is how many pages pull content dynamically (there may also be other factors: redirects, iframes, canvas, etc). If your tool (e.g. Python urllib,etc). can only access static HTML you will not be able to pull the content you want as you will be accessing instruction sets of how to dynamically render content rather than the actual content. I am not sure how your tool in R works, but I imagine this is a likely issue you may be facing. I have developed some software that solves problem #2 by leveraging http://phantomjs.org/, but it’s not ready for public release quite yet; however, you may want to consider using an automation framework like selenium (http://www.seleniumhq.org/) - tim, phd student, upenn

5 5

Re: [Air-L] Software to extract content of Facebook & Twitter
by Tim Libert 27 Aug '14

27 Aug '14

I once figured out a way to get the youtube to spit out all of the comments at once by tweaking an ajax request, but I can’t figure out how to do it again on quick inspection. it was possible though, you may have to bribe your coder friend with some club mate. ;-) shawn is totally on the mark re: rate-limiting, google in particular is strict on scraping/botishness; anything you set against their properties needs some built-in politeness, I think a random interval of 5-15 seconds was working for me at one time. another way is to just pay somebody on amazon turk to copy/paste for you - could be the cheapest route time- and resource-wise and not requiring new tools. - t On Aug 27, 2014, at 5:41 PM, Shawn Walker <stw3(a)uw.edu> wrote: > Hi Anu, > > To extract YouTube comments, consider TubeKit (http://tubekit.org/) I've used it in a few projects to extract YouTube video metadata, videos, and comment data with great success. > > Another consideration with respect to the discussion of these tools for FB or Twitter data collection is to evaluate what APIs each tool uses. Depending on the API, you might only receive a small sample of data or may be rate limited with others. So, it's important to understand how any tool you use works and what implications or limitations that might have on your research. Historical data is notoriously difficult to get -- purchasing historical data is an option, but adds in a new set of limitations too (deleted accounts, posts, URL decay, etc.). > > These issues need to be discussed more openly and critically addressed. :) > > -- > Shawn Walker > PhD Candidate > Information School > University of Washington > stw3(a)uw.edu — students.washington.edu/stw3 > SoMe Lab @ UW - somelab.net > > ________________________________________ > From: Air-L <air-l-bounces(a)listserv.aoir.org> on behalf of Harju Anu <anu.harju(a)aalto.fi> > Sent: Wednesday, August 27, 2014 11:26 AM > To: Tim Libert > Cc: air-l(a)listserv.aoir.org > Subject: Re: [Air-L] Software to extract content of Facebook & Twitter > > Hi everyone, > > and I'm also grateful for all these suggestions for various tools. For a paper for my PhD I'm looking at YouTube comment threads and I was wondering if any one of you might know a tool that can extract those? It's a very laborious process to do manually and it drives me insane. I once asked a coder friend of mine, but he said it was more complicated than he initially thought, and we left it at that. > > Thank you in advance, and thanks for a great list! I've been a lurker for quite some time now and find it very useful. > > Best, > Anu > > > Anu Harju > Doctoral Candidate > Aalto University > Helsinki > Finland > > Sent from my iPhone > > On 27.8.2014, at 18.06, "Tim Libert" <tlibert(a)asc.upenn.edu> wrote: > >> I’d quickly point out two additional considerations when ingesting fb/twitter data: 1) APIs generally exclude ads (which are ‘targeted’) - so depending on what you want to study and/or model an API will never give you an accurate view of what users really see. APIs are easy, but incomplete. 2) The trick with scraping content directly from the web is accounting for processing/executing javascript as that is how many pages pull content dynamically (there may also be other factors: redirects, iframes, canvas, etc). If your tool (e.g. Python urllib,etc). can only access static HTML you will not be able to pull the content you want as you will be accessing instruction sets of how to dynamically render content rather than the actual content. I am not sure how your tool in R works, but I imagine this is a likely issue you may be facing. I have developed some software that solves problem #2 by leveraging http://phantomjs.org/, but it’s not ready for public release quite yet; however, you may want to consider using an automation framework like selenium (http://www.seleniumhq.org/) >> >> - tim, phd student, upenn >> _______________________________________________ >> The Air-L(a)listserv.aoir.org mailing list >> is provided by the Association of Internet Researchers http://aoir.org >> Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org >> >> Join the Association of Internet Researchers: >> http://www.aoir.org/ > _______________________________________________ > The Air-L(a)listserv.aoir.org mailing list > is provided by the Association of Internet Researchers http://aoir.org > Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org > > Join the Association of Internet Researchers: > http://www.aoir.org/

1 0

Governance or Ungovernance: A Strategy Workshop for Internet Activists
by Seda Gurses 27 Aug '14

27 Aug '14

hey airers, this year the internet governance forum (igf) is taking place in istanbul, which has raised some eyebrows given the government’s track record also with respect to the internet over the last 4 years. alternatif bilisim dernegi, a local association, is organizing the internet ungovernance forum in parallel to this year’s igf in istanbul (https://iuf.alternatifbilisim.org) in preparation of the ungovernance forum, we also put together a workshop which may be of interest to those of you who plan to be there and are asking similar questions about the future of internet governance. please see details below and feel free to spread widely in your networks. thank you, s. Governance or Ungovernance: A Strategy Workshop for Internet Activists 3rd of September, 2014 at MMO [1] Istanbul, Turkey https://iuf.alternatifbilisim.org/gov_ungov_workshop.html * Are you confused about all the complicated and overlapping institutions that claim to be working on Internet governance? * Are you unsure about how activists concerned with local censorship and surveillance fit into these global structures? * Are you wondering how the IGF, ICANN, IETF, the ITU, the CSTD, Net Mundial are relevant to your collective engagements? If these are questions you are asking yourself and you would like to think together with other people asking similar questions, then please join us at our strategy workshop for Internet activists! In preparation for the Internet Ungovernance Forum[2], the half-day workshop will provide an an opportunity for activists working towards a free, secure and open Internet to get to know Internet governance processes and to explore strategies for collective action. While topics abound, the focus of this workshop will be on matters of surveillance, privacy, the securitization of the Internet, and censorship. Together with invited guests, we plan to spend an afternoon developing strategies to address these problems through national and transnational Internet governance institutions, and/or through organizing outside of these structures. The participants will be expected to report important outcomes of the workshop at the final session of the Internet Ungovernance Forum. The workshop will consist of three parts: * A mapping and diagnosis of existing Internet governance processes that may be relevant to activists. This will include an overview of current shifts in internet governance and exploration of immediate points of intervention. * A discussion of levels of governance, especially the relative merits of participating in local, national, or transnational institutions. We will work through strategies available to activists to address the selected topics through transnational and national institutions that make up the current Internet governance landscape. * A discussion of initiatives and topics that the activists are currently involved in, and a mapping of their potential strategies for collective action within or outside of the Internet governance structures. Format: The workshop will consist of three discussion sessions structured and moderated by invited guests with experience in internet activism or in internet governance structures. Participation: This workshop is for activists that want to collectively develop strategies for action at the local or international level on the topics of surveillance, privacy, the securitization of the Internet and censorship. We expect activists from all disciplinary/non-disciplined backgrounds to join us. A commitment to collective action and curiosity about Internet governance processes is a plus. We ask those interested in participating to send an email to ungovWorkshop(a)alternatifbilisim.org by 1st of September 2014. Please include in the email a short biography and a list of topics you would like to see discussed at the workshop. Invited Guests: Melih Kirlidog, Alternatif Bilisim Dernegi, Marmara University Milton Mueller, Syracuse University, Internet Governance Project Robin Gross, IP Justice Meryem Marzouki, CNRS Reading Materials: We recommend reading the following materials and studying the following websites in preparation for the workshop: * Internet Governance Project blog [3] * The Citizen Lab [4] * Paper: Finding a Formula for Brazil: Representation and Legitimacy in Internet Governance [5] Schedule: 3. of September, 2014. 13.00 - 18.00 hours Location: Makina Muhensidleri Odasi Istanbul Subesi (Beyoglu/Taksim) Katip Mustafa Celebi Mahallesi Ipek Sokak No: 9 Beyoglu, Istanbul, Language: The workshop will take place in English. We encourage those who have reservations attending the workshop due to their command of the English language to couple up with other participants who can support them throughout the meeting. Please visit the Workshop webpage for updates.[6] [1] MMO Beyoglu Ofice http://www.mmo.org.tr/genel/bizden_detay.php?kod=26967&tipi=2&sube=10 [2] Internet Ungovernance Forum https://iuf.alternatifbilisim.org [3] http://internetgovernance.org [4] https://citizenlab.org/ [5] http://www.internetgovernance.org/wordpress/wp-content/uploads/MiltonBenWPd… [6] https://iuf.alternatifbilisim.org/gov_ungov_workshop.html

1 0

Due Today Aug 27 - GovLab's Academy Course Online Participation - Solving Public Problems with Technology
by Steven Clift 27 Aug '14

27 Aug '14

This crossed my radar today just in time for you to apply - by Aug 27: http://bit.ly/govlabacademycourse The GovLab Academy’s new course can help you solve problems that matter. Are you passionate about civic tech? Are you… Working in government with an innovative project you want to bring to life? An independent social innovator who wants to expand your toolkit for real-world change? Someone with an important idea for tackling a social problem but who needs to develop the key skills to take your vision from idea to implementation? Spaces for online participation are now available for the GovLab Academy’s flagship 14-week, Masters-level course, Solving Public Problems with Technology. Learn about the program here. http://bit.ly/govlabacademycourse In the Fall Semester of 2014 this course is being taught live for students at the MIT Media Lab (Wednesdays from 11am to 1pm) and NYU students (Thursdays from 6:30 pm to 9:30 pm). You can apply to join either session as an online participant in the program (credit will not be granted at either institution for online participants). The course will be taught by Professor Beth Simone Noveck (Director of the Governance Lab, author of Wiki Government and the forthcoming Smart Citizen, Smarter State, and former Deputy Chief Technology Officer in the Obama Administration), with contributions from the GovLab Faculty and a wide range experts in the fields of open governance and civic technology. Details on the subject matter covered, the work assigned, the supplementary readings and videos, and participation logistics are all contained in the online syllabus. Solving Public Problems with Technology is not a traditional course. We offer a hands-on learning and mentoring program designed to help you shape and implement an innovative project using civic technology. To that end, we will prepare you to take advantage of the latest innovations in open and participatory problem-solving, including the application of open data, crowdsourcing, expert networks and systems, game mechanics, and prizes. We will prepare you to work with real-world institutions and partners, such as agencies and NGOs, to develop more effective and scalable initiatives. We will also give you ready access to a community of like-minded practitioners and experts, who will provide a network of support as you take your project through to implementation. Individuals and teams are both eligible to apply. Online participation is free of charge thanks to a generous grant from the John S. and James L. Knight Foundation. While online participants will not receive academic credit the GovLab will recognize satisfactory completion of all course requirements with a certificate and demonstration of relevant skills and capabilities with appropriate open badges. Students enrolled at other institutions wishing to explore the possibility of being granted independent study credit at their home institution should contact us. We will be happy to collaborate. If you wish to apply, please fill out our application form here. Applications for Fall 2014 semester are due by August 27th, but additional courses will be offered later this year and throughout 2015. Can’t participate in this semester’s course? Stay updated on future offerings by signing up here. If you have questions please contact us at: info-academy(a)thegovlab.org. About The GovLab The Governance Lab (The GovLab) strives to improve people’s lives by changing how we govern. We endeavor to strengthen the ability of people and institutions to work together to solve problems, make decisions, resolve conflict and govern themselves more effectively and legitimately. We design technology, policy and strategies for fostering more open and collaborative approaches to governance and we test what works. The goal of the GovLab Academy is to support a new generation of public entrepreneurs to tap the intelligence of citizens and create actionable projects that improve people’s lives. Follow us on Twitter @TheGovLab Steven Clift - http://stevenclift.com Executive Director - http://E-Democracy.org Twitter: http://twitter.com/democracy Tel/Text: +1.612.234.7072

1 0

text analysis -nostalgia
by Barry Wellman 27 Aug '14

27 Aug '14

The discussion for Corpus Analysis reminds me of the General Inquirer, created (almost solely) by the late great Philip J Stone III. It was a major basis for my 1969 dissertation, analyzing "Who Am I?" statements of 9th graders in Pittsburgh. Did a few early papers on it too. There's a book on the GI that some libraries may have. While Phil is gone, supposedly there is a Foundation. I was initially asked to be on the board, but I never heard anything. So everything old is new again Barry Wellman _______________________________________________________________________ FRSC NetLab Network INSNA Founder Faculty of Information (iSchool) University of Toronto Toronto Canada M5S 3G6 http://www.chass.utoronto.ca/~wellman twitter: @barrywellman NETWORKED:The New Social Operating System. Lee Rainie & Barry Wellman MIT Press http://amzn.to/zXZg39 Print $15 Kindle $9 Old/NewCyberTimes http://bit.ly/c8N9V8 ________________________________________________________________________

1 0

Re: [Air-L] Software to extract content of Facebook & Twitter
by Cristina Aced 27 Aug '14

27 Aug '14

Many thanks for all your messages and suggestions. I'm now checking all the messages. I'll test the tools you have proposed and I promise to share which is the best option to extract these data from Facebook! Cristina Aced 2014-08-27 12:52 GMT+02:00 Shulman, Stu <stu(a)texifter.com>: > DiscoverText has been connected to the Facebook Open Graph API and the > Twitter Search API since 2010. Here are some reasons to test it for > collecting Facebook or other social and related non-social data: > > - there is nothing to install - use it in a browser > - pull data in from a variety of sources, including SurveyMonkey directly > via an API > - keystroke human coding is fast and connects to other human coders via a > peer network > - measurement tools developed open source in 2007 for inter-rater > reliability and adjudication of coder disagreement (CAT) > - search, filtering, and bucketing capabilities > - automated duplicate detection and near duplicate clustering > - machine learning classifiers that help sift out irrelevant data to > 'clean' social data sets > - patent pending "CoderRank" technology for enhanced machine-learning > - free for the first 30 days > https://www.discovertext.com/Home/TrialRegistration > > We are also in the last week of the final drawing to win 1,000,000 > historical Tweets and a year's worth of Enterprise grade software worth > more than $15,000. It takes about five minutes to beta test the free > historical Twitter estimator "Sifter" (http://sifter.texifter.com) and > then enter: http://bit.ly/1pGuUJo. > > These are high-grade tools for text and metadata developed in a research > lab specifically to improve measurement. > > > > > On Wed, Aug 27, 2014 at 3:41 AM, Cristina Aced <blogocorp(a)gmail.com> > wrote: > >> Hello all, >> >> I am looking for software that facilitates the extraction of content of >> Facebook pages. I am doing a research about how a sample of companies are >> using blogs, Twitter and Facebook. I want to apply content analysis but >> previously I need to extract the content. >> >> I am using R and the software OutWit Hub Pro to scrape the data on blogs >> and Twitter, but they are not working properly with Facebook (I think it >> is >> due to some restrictions of Facebook's API). >> >> Any other suggestions? >> >> Many thanks! >> >> *Cristina Aced * >> >> PhD Candidate >> Doctoral Programme in Knowledge and Information Society >> Universitat Oberta de Catalunya >> | caced(a)uoc.edu >> www.cristinaaced.com/blog | Twitter: @blogocorp >> _______________________________________________ >> The Air-L(a)listserv.aoir.org mailing list >> is provided by the Association of Internet Researchers http://aoir.org >> Subscribe, change options or unsubscribe at: >> http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org >> >> Join the Association of Internet Researchers: >> http://www.aoir.org/ >> > > > > -- > Dr. Stuart W. Shulman > http://people.umass.edu/stu > > Founder and CEO, Texifter > http://texifter.com > > LinkedIn > http://www.linkedin.com/in/stuartwshulman > > Twitter > https://twitter.com/StuartWShulman >

2 1

Looking for CATA program for tweet analysis
by Dave Voelker 27 Aug '14

27 Aug '14

I'd like to know if someone can recommend -- or knows of -- any computer assisted text analysis (CATA) programs capable of applying predefined concept dictionaries (lists of words and phrases representing a concept) to large datasets of tweets. I'm referring here to traditional, conceptual, content analysis (social science), not the bottom-up, data mining methods (computer science) primarily being used to analyze big data. An example of the kind of program I'm talking about is the freeware Yoshikoder (http://sourceforge.net/p/yoshikoder/wiki/Home/) but I'm unable to open a large (~ 3 GB) text file of tweets with it. - - - - - - - - - - - - - - David Voelker, Ph.D. Department of Communication Stanford University http://comm.stanford.edu/faculty/voelker/

6 6

2014 ASIS&T Annual Meeting now open
by Richard Hill 27 Aug '14

27 Aug '14

ASIST 2014 Annual Meeting See http://www.asis.org/asist2014/ for full information and to register October 31-November 5, 2014, Seattle, WA Come to Seattle for the 77th ASIS&T Annual Meeting. The theme for this year�s conference is Connecting Collections, Cultures, and Communities - we chose this theme to celebrate the breadth of information science, its historical roots, its user-centeredness, and its unique aim of bringing people together around ideas, thoughts, and the exchange of information and knowledge. The program includes 42 papers, 26 panels, 102 posters, 12 workshops, and many opportunities for socializing at receptions. All in all this conference is packed with intellectually stimulating sessions, lots of opportunities to network and meet new people, and engage in the health and well-being of this wonderful association. We are delighted to present two excellent and accomplished keynote speakers: Kris M. Kutchera is Vice President, Information Technology for the Alaska Air Group Alessandro Acquisti is Associate Professor at Carnegie Mellon University Richard Hill Executive Director Association for Information Science and Technology 1320 Fenwick Lane, Suite 510 Silver Spring, MD 20910 FAX: (301) 495-0810 (301) 495-0900

1 0

Teaching Aid for you to steal: My student workbook for class on Cultural/Media Studies Research Approaches
by Terri Senft 27 Aug '14

27 Aug '14

Well, hi, Pals. I finally put together the whole 67 page student workbook for my class on Cultural and Media studies Research Approaches.It is on my site, here: http://www.terrisenft.net/wordpress/wp-content/uploads/2012/08/Approaches-S… Note: This is a PDF document that includes all the student exercises tailored to my particular class. I will email you an MS Word copy if you want to mess around with it and make it suit your own needs. Just send a mail my way asking, and throw out a thanks or a howdy somewhere on your doc, and I'm happy :) Btw, I have another workbook for Senior Thesis projects that includes a bit of the stuff in the Approaches doc, but also includes a section on writing formal research proposals and literature reviews that will be in that one. That should be ready by the weekend, I think. Excelsior! T -- <http://goog_689013053> <http://goog_689013053> Dr. Theresa M. Senft Global Liberal Studies Program School of Arts & Sciences New York University 726 Broadway NY NY 10003 home: *www.terrisenft.net <http://goog_689013053>* (needs a serious updating) facebook: www.facebook.com/theresa.senft twitter: @terrisenft

2 1