Twitter data collection tools
Hello list members: I am looking for tools which can help extract all possible Twitter statistics (such as, number of tweets, followers, followings, mentions, re-tweets, favorites) for a list of Twitter handlers (around 120 accounts). *In particular, I look for a tool that can take the IDs as a single file and provide the desired statistics for each ID. * The Webometrics Analyst has this functionality, but unfortunately it only provides followers and followings data. I am also familiar with the several other tools including the ones mentioned in the Dean Freelon's curated list <https://docs.google.com/document/d/1UaERzROI986HqcwrBDLaqGG8X_lYwctj6ek6ryqDOiQ/edit>, but non of these can extract all the information I need. Some tools provide more statistics, but they work with one ID at time. I will greatly appreciate any suggestions. Thank you, -- Gohar Feroz Khan, PhD Adjunct Faculty & Research Adviser Korea Advance Institute of Science & Technology (KAIST) Global Information and Telecommunication Technology Program (ITTP) 291 Daehak-ro, Yuseong-gu, Daejeon, South Korea. ------------------------------------------------------------------------------ Check out my new book on social media analytics <http://7layersanalytics.com/>! ------------------------------------------------------------ -------------------- Please consider submitting your work to the social media analytics track at PACIS201 <http://www.pacis2016.org/Page/Index/71>6. ------------------------------------------------------------ -------------------- Social Identities: || Blog <http://gfkhan.wordpress.com> || Twitter <https://twitter.com/gfkhan> || LinkedIn <https://www.linkedin.com/pub/gohar-feroz-khan/7/62b/42> || Research Centre <http://centreforsocialtech.com/>||
http://www.screencast.com/t/VKLqaekPz Bulk Twitter Handle Follow On Tuesday, November 10, 2015, Gohar F. Khan <gohar.feroz@gmail.com> wrote:
Hello list members:
I am looking for tools which can help extract all possible Twitter statistics (such as, number of tweets, followers, followings, mentions, re-tweets, favorites) for a list of Twitter handlers (around 120 accounts). *In particular, I look for a tool that can take the IDs as a single file and provide the desired statistics for each ID. *
The Webometrics Analyst has this functionality, but unfortunately it only provides followers and followings data. I am also familiar with the several other tools including the ones mentioned in the Dean Freelon's curated list < https://docs.google.com/document/d/1UaERzROI986HqcwrBDLaqGG8X_lYwctj6ek6ryqD...
, but non of these can extract all the information I need. Some tools provide more statistics, but they work with one ID at time.
I will greatly appreciate any suggestions.
Thank you,
--
Gohar Feroz Khan, PhD
Adjunct Faculty & Research Adviser Korea Advance Institute of Science & Technology (KAIST) Global Information and Telecommunication Technology Program (ITTP) 291 Daehak-ro, Yuseong-gu, Daejeon, South Korea.
------------------------------------------------------------------------------ Check out my new book on social media analytics <http://7layersanalytics.com/>! ------------------------------------------------------------ -------------------- Please consider submitting your work to the social media analytics track at PACIS201 <http://www.pacis2016.org/Page/Index/71>6. ------------------------------------------------------------ -------------------- Social Identities: || Blog <http://gfkhan.wordpress.com> || Twitter <https://twitter.com/gfkhan> || LinkedIn <https://www.linkedin.com/pub/gohar-feroz-khan/7/62b/42> || Research Centre <http://centreforsocialtech.com/>|| _______________________________________________ The Air-L@listserv.aoir.org <javascript:;> mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
-- Dr. Stuart W. Shulman Founder and CEO, Texifter http://texifter.com LinkedIn http://www.linkedin.com/in/stuartwshulman Twitter https://twitter.com/StuartWShulman
There are two basic ways to do this. You can use a programming module like Twarc (https://github.com/edsu/twarc) or Twitter for Python (https://pypi.python.org/pypi/twitter) to pull the data yourself, which is free but takes some time to learn how to do. The other option is to use an analytics platform like Discovertext or Crimson Hexagon, which is more user-friendly but more expensive. ~DEEN On 11/10/2015 9:59 PM, Gohar F. Khan wrote:
Hello list members:
I am looking for tools which can help extract all possible Twitter statistics (such as, number of tweets, followers, followings, mentions, re-tweets, favorites) for a list of Twitter handlers (around 120 accounts). *In particular, I look for a tool that can take the IDs as a single file and provide the desired statistics for each ID. *
The Webometrics Analyst has this functionality, but unfortunately it only provides followers and followings data. I am also familiar with the several other tools including the ones mentioned in the Dean Freelon's curated list <https://docs.google.com/document/d/1UaERzROI986HqcwrBDLaqGG8X_lYwctj6ek6ryqDOiQ/edit>, but non of these can extract all the information I need. Some tools provide more statistics, but they work with one ID at time.
I will greatly appreciate any suggestions.
Thank you,
-- Deen Freelon, Ph.D. Assistant Professor American University School of Communication Office: McKinley 325 http://dfreelon.org/ @dfreelon
I wrote a script that gets you part way: https://github.com/casmlab/user-timeline-tools If you setup and run getUserTimeline.py, you'll have all the data you want, just not in the format you want. If you keep going and run parseUserTimeline.py, you'll end up with a MySQL table that has the info you listed, but you'll also have a bunch of stuff you don't need. Though, looking back at your list, I guess this doesn't get/save retweets, but you could probably mod it to do so. It worked 3 months ago. Issues and bug fixes welcome! On Tue, Nov 10, 2015 at 9:33 PM, Deen Freelon <dfreelon@gmail.com> wrote:
There are two basic ways to do this. You can use a programming module like Twarc (https://github.com/edsu/twarc) or Twitter for Python ( https://pypi.python.org/pypi/twitter) to pull the data yourself, which is free but takes some time to learn how to do. The other option is to use an analytics platform like Discovertext or Crimson Hexagon, which is more user-friendly but more expensive.
~DEEN
On 11/10/2015 9:59 PM, Gohar F. Khan wrote:
Hello list members:
I am looking for tools which can help extract all possible Twitter statistics (such as, number of tweets, followers, followings, mentions, re-tweets, favorites) for a list of Twitter handlers (around 120 accounts). *In particular, I look for a tool that can take the IDs as a single file and provide the desired statistics for each ID. *
The Webometrics Analyst has this functionality, but unfortunately it only provides followers and followings data. I am also familiar with the several other tools including the ones mentioned in the Dean Freelon's curated list < https://docs.google.com/document/d/1UaERzROI986HqcwrBDLaqGG8X_lYwctj6ek6ryqD...
, but non of these can extract all the information I need. Some tools provide more statistics, but they work with one ID at time.
I will greatly appreciate any suggestions.
Thank you,
-- Deen Freelon, Ph.D. Assistant Professor American University School of Communication Office: McKinley 325 http://dfreelon.org/ @dfreelon
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
Gohar, Retweets are not in the metadata payload. The "favorites" in Twitter data refer to the number of favorites by that user over the complete history of their Twitter account at the time they wrote that Tweet. The thing about RTs and favorites of a particular tweet is that they are dynamic variables. When you look at a collected Tweet using the Twitter display you see the counts for both live in realtime. There is no method I know of to capture that data in the aggregate. There are technical and legal barriers. ~Stu Stu Shulman <https://twitter.com/StuartWShulman> FC Massachusetts U14B, Head Coach Amherst Regional High School, JV-B Head CoachMA Olympic Development Program U12B, Assistant Coach On Tue, Nov 10, 2015 at 11:00 PM, Libby Hemphill <libbyh@gmail.com> wrote:
I wrote a script that gets you part way: https://github.com/casmlab/user-timeline-tools
If you setup and run getUserTimeline.py, you'll have all the data you want, just not in the format you want. If you keep going and run parseUserTimeline.py, you'll end up with a MySQL table that has the info you listed, but you'll also have a bunch of stuff you don't need.
Though, looking back at your list, I guess this doesn't get/save retweets, but you could probably mod it to do so. It worked 3 months ago. Issues and bug fixes welcome!
On Tue, Nov 10, 2015 at 9:33 PM, Deen Freelon <dfreelon@gmail.com> wrote:
There are two basic ways to do this. You can use a programming module like Twarc (https://github.com/edsu/twarc) or Twitter for Python ( https://pypi.python.org/pypi/twitter) to pull the data yourself, which is free but takes some time to learn how to do. The other option is to use an analytics platform like Discovertext or Crimson Hexagon, which is more user-friendly but more expensive.
~DEEN
On 11/10/2015 9:59 PM, Gohar F. Khan wrote:
Hello list members:
I am looking for tools which can help extract all possible Twitter statistics (such as, number of tweets, followers, followings, mentions, re-tweets, favorites) for a list of Twitter handlers (around 120 accounts). *In particular, I look for a tool that can take the IDs as a single file and provide the desired statistics for each ID. *
The Webometrics Analyst has this functionality, but unfortunately it only provides followers and followings data. I am also familiar with the several other tools including the ones mentioned in the Dean Freelon's curated list <
https://docs.google.com/document/d/1UaERzROI986HqcwrBDLaqGG8X_lYwctj6ek6ryqD...
, but non of these can extract all the information I need. Some tools provide more statistics, but they work with one ID at time.
I will greatly appreciate any suggestions.
Thank you,
-- Deen Freelon, Ph.D. Assistant Professor American University School of Communication Office: McKinley 325 http://dfreelon.org/ @dfreelon
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
participants (5)
-
Deen Freelon -
Gohar F. Khan -
Libby Hemphill -
Shulman, Stu -
Stuart Shulman