Extracting Twitter Data
Dear colleagues, I hope this email finds you all well and safe. Help needed! I am trying to download the videos and tweets shared in a specific period. I only need the information on public accounts that is accessible for any Twitter user: the name of the user who has shared it, the date, the video, and the tweet itself. I have been using the advanced search tool on Twitter, entering the date, and desired hashtags and accounts. It is a huge amount of data and I have to redo the process once more to get some more information and complete what I currently have. I have already looked for codes; however, a friend informed me that such information is only available for free, even with coding, for the last 30 days. I am looking to retrieve data from last November though. Has anyone gone through the same obstacle? Is that true that the only way for me is to do it manually? If not, is there any code for R? I appreciate your help! Best, Afrooz Mosallaeipour Drexel University Ph.D. Student in Communication, Culture, & Media
Hi it used to be possible to gather older tweets for free through ad hoc scrapers that leveraged a ' bug' in the advanced search, as far as i know that particular issue has been patched very recently, so at the present moment ( the last 2 days afaik) there is no way of getting what you want for free. i have no doubt that a way of restoring previous access will be found quickly but at the present moment the only way of getting what you want is through screen scraping, probably using some kind of browser emulation like Selenium which is kinda unwieldy. there are multiple project of twitter scrapers on github, in the coming weeks probably one of the will find a way to acquire data your best chance is to bookmark them and wait best Guido Il giorno mar 22 set 2020 alle ore 05:00 Mosallaeipour,Fatemeh < fm455@drexel.edu> ha scritto:
Dear colleagues,
I hope this email finds you all well and safe. Help needed! I am trying to download the videos and tweets shared in a specific period. I only need the information on public accounts that is accessible for any Twitter user: the name of the user who has shared it, the date, the video, and the tweet itself. I have been using the advanced search tool on Twitter, entering the date, and desired hashtags and accounts. It is a huge amount of data and I have to redo the process once more to get some more information and complete what I currently have. I have already looked for codes; however, a friend informed me that such information is only available for free, even with coding, for the last 30 days. I am looking to retrieve data from last November though.
Has anyone gone through the same obstacle? Is that true that the only way for me is to do it manually? If not, is there any code for R?
I appreciate your help!
Best,
Afrooz Mosallaeipour Drexel University Ph.D. Student in Communication, Culture, & Media
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
Thank you Fabio, Marc, and Guido, I’ll try your suggestions and will let you know about the results. However, I think what Guido says is correct! But will try and keep you updated. Cheers, Afrooz ________________________________ From: Guido Anselmi <guido.anselmi@unimib.it> Sent: Tuesday, September 22, 2020 3:48:46 AM To: Mosallaeipour,Fatemeh <fm455@drexel.edu> Cc: air-l@listserv.aoir.org <air-l@listserv.aoir.org> Subject: Re: [Air-L] Extracting Twitter Data External. Hi it used to be possible to gather older tweets for free through ad hoc scrapers that leveraged a ' bug' in the advanced search, as far as i know that particular issue has been patched very recently, so at the present moment ( the last 2 days afaik) there is no way of getting what you want for free. i have no doubt that a way of restoring previous access will be found quickly but at the present moment the only way of getting what you want is through screen scraping, probably using some kind of browser emulation like Selenium which is kinda unwieldy. there are multiple project of twitter scrapers on github, in the coming weeks probably one of the will find a way to acquire data your best chance is to bookmark them and wait best Guido Il giorno mar 22 set 2020 alle ore 05:00 Mosallaeipour,Fatemeh <fm455@drexel.edu<mailto:fm455@drexel.edu>> ha scritto: Dear colleagues, I hope this email finds you all well and safe. Help needed! I am trying to download the videos and tweets shared in a specific period. I only need the information on public accounts that is accessible for any Twitter user: the name of the user who has shared it, the date, the video, and the tweet itself. I have been using the advanced search tool on Twitter, entering the date, and desired hashtags and accounts. It is a huge amount of data and I have to redo the process once more to get some more information and complete what I currently have. I have already looked for codes; however, a friend informed me that such information is only available for free, even with coding, for the last 30 days. I am looking to retrieve data from last November though. Has anyone gone through the same obstacle? Is that true that the only way for me is to do it manually? If not, is there any code for R? I appreciate your help! Best, Afrooz Mosallaeipour Drexel University Ph.D. Student in Communication, Culture, & Media _______________________________________________ The Air-L@listserv.aoir.org<mailto:Air-L@listserv.aoir.org> mailing list is provided by the Association of Internet Researchers http://aoir.org<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Faoir.org%2F&data=02%7C01%7Cfm455%40drexel.edu%7C31f81963f6864308366308d85ecbf76e%7C3664e6fa47bd45a696708c4f080f8ca6%7C0%7C0%7C637363577426896014&sdata=jWP8KTQFOmAjyVvSg%2BOyfkto%2FpfNXluiXQ%2BvQ%2BYC9Ys%3D&reserved=0> Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flistserv.aoir.org%2Flistinfo.cgi%2Fair-l-aoir.org&data=02%7C01%7Cfm455%40drexel.edu%7C31f81963f6864308366308d85ecbf76e%7C3664e6fa47bd45a696708c4f080f8ca6%7C0%7C0%7C637363577426896014&sdata=VB6Rp0B5E783ILAyJxqDw1%2BDVzrUY5ayWPOV%2BYI3B5E%3D&reserved=0> Join the Association of Internet Researchers: http://www.aoir.org/<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.aoir.org%2F&data=02%7C01%7Cfm455%40drexel.edu%7C31f81963f6864308366308d85ecbf76e%7C3664e6fa47bd45a696708c4f080f8ca6%7C0%7C0%7C637363577426896014&sdata=rsjHPgka6gbLbZCeV9MaydW0o%2BznVrgz%2BXTp%2BPNHbdE%3D&reserved=0>
Hi all, Just briefly, I think it is still possible to do targeted searches via advanced search, it just seems the actual advanced search popup interface is broken (??). Although I am not sure what Guido is referring to. It seems as though you will need to type in the search string into the search field. Here is a topical example for Australia today (with policy backflip to fund a full fibre broadband network, aka NBN), looking at "NBN" mentions during the election period in 2013: https://twitter.com/search?q=%22NBN%22%20since%3A2013-07-13%20until%3A2013-0... I've had a quick look in Outwit Hub and it is still pulling the individual tweets for the above search string. (My old scrapers are not working, which is not surprising as they would only last until the next update anyway, and I haven't used since around 2018ish. Outwit Hub is pulling the individual tweets in the 'guess' field.) To get more useful information and take into account the ~4500 tweet limit will need a refined search string strategy. Cheers, Glen From: Air-L <air-l-bounces@listserv.aoir.org> On Behalf Of Mosallaeipour,Fatemeh Sent: Tuesday, 22 September 2020 10:42 PM To: Guido Anselmi <guido.anselmi@unimib.it> Cc: air-l@listserv.aoir.org Subject: Re: [Air-L] Extracting Twitter Data Thank you Fabio, Marc, and Guido, I'll try your suggestions and will let you know about the results. However, I think what Guido says is correct! But will try and keep you updated. Cheers, Afrooz ________________________________ From: Guido Anselmi <guido.anselmi@unimib.it<mailto:guido.anselmi@unimib.it>> Sent: Tuesday, September 22, 2020 3:48:46 AM To: Mosallaeipour,Fatemeh <fm455@drexel.edu<mailto:fm455@drexel.edu>> Cc: air-l@listserv.aoir.org<mailto:air-l@listserv.aoir.org> <air-l@listserv.aoir.org<mailto:air-l@listserv.aoir.org>> Subject: Re: [Air-L] Extracting Twitter Data External. Hi it used to be possible to gather older tweets for free through ad hoc scrapers that leveraged a ' bug' in the advanced search, as far as i know that particular issue has been patched very recently, so at the present moment ( the last 2 days afaik) there is no way of getting what you want for free. i have no doubt that a way of restoring previous access will be found quickly but at the present moment the only way of getting what you want is through screen scraping, probably using some kind of browser emulation like Selenium which is kinda unwieldy. there are multiple project of twitter scrapers on github, in the coming weeks probably one of the will find a way to acquire data your best chance is to bookmark them and wait best Guido Il giorno mar 22 set 2020 alle ore 05:00 Mosallaeipour,Fatemeh <fm455@drexel.edu<mailto:fm455@drexel.edu<mailto:fm455@drexel.edu%3cmailto:fm455@drexel.edu>>> ha scritto: Dear colleagues, I hope this email finds you all well and safe. Help needed! I am trying to download the videos and tweets shared in a specific period. I only need the information on public accounts that is accessible for any Twitter user: the name of the user who has shared it, the date, the video, and the tweet itself. I have been using the advanced search tool on Twitter, entering the date, and desired hashtags and accounts. It is a huge amount of data and I have to redo the process once more to get some more information and complete what I currently have. I have already looked for codes; however, a friend informed me that such information is only available for free, even with coding, for the last 30 days. I am looking to retrieve data from last November though. Has anyone gone through the same obstacle? Is that true that the only way for me is to do it manually? If not, is there any code for R? I appreciate your help! Best, Afrooz Mosallaeipour Drexel University Ph.D. Student in Communication, Culture, & Media _______________________________________________ The Air-L@listserv.aoir.org<mailto:Air-L@listserv.aoir.org<mailto:Air-L@listserv.aoir.org%3cmailto:Air-L@listserv.aoir.org>> mailing list is provided by the Association of Internet Researchers http://aoir.org<http://aoir.org><https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Faoir.org%2F&data=02%7C01%7Cfm455%40drexel.edu%7C31f81963f6864308366308d85ecbf76e%7C3664e6fa47bd45a696708c4f080f8ca6%7C0%7C0%7C637363577426896014&sdata=jWP8KTQFOmAjyVvSg%2BOyfkto%2FpfNXluiXQ%2BvQ%2BYC9Ys%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Faoir.org%2F&data=02%7C01%7Cfm455%40drexel.edu%7C31f81963f6864308366308d85ecbf76e%7C3664e6fa47bd45a696708c4f080f8ca6%7C0%7C0%7C637363577426896014&sdata=jWP8KTQFOmAjyVvSg%2BOyfkto%2FpfNXluiXQ%2BvQ%2BYC9Ys%3D&reserved=0>> Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org<http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org><https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flistserv.aoir.org%2Flistinfo.cgi%2Fair-l-aoir.org&data=02%7C01%7Cfm455%40drexel.edu%7C31f81963f6864308366308d85ecbf76e%7C3664e6fa47bd45a696708c4f080f8ca6%7C0%7C0%7C637363577426896014&sdata=VB6Rp0B5E783ILAyJxqDw1%2BDVzrUY5ayWPOV%2BYI3B5E%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flistserv.aoir.org%2Flistinfo.cgi%2Fair-l-aoir.org&data=02%7C01%7Cfm455%40drexel.edu%7C31f81963f6864308366308d85ecbf76e%7C3664e6fa47bd45a696708c4f080f8ca6%7C0%7C0%7C637363577426896014&sdata=VB6Rp0B5E783ILAyJxqDw1%2BDVzrUY5ayWPOV%2BYI3B5E%3D&reserved=0>> Join the Association of Internet Researchers: http://www.aoir.org/<http://www.aoir.org><https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.aoir.org%2F&data=02%7C01%7Cfm455%40drexel.edu%7C31f81963f6864308366308d85ecbf76e%7C3664e6fa47bd45a696708c4f080f8ca6%7C0%7C0%7C637363577426896014&sdata=rsjHPgka6gbLbZCeV9MaydW0o%2BznVrgz%2BXTp%2BPNHbdE%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.aoir.org%2F&data=02%7C01%7Cfm455%40drexel.edu%7C31f81963f6864308366308d85ecbf76e%7C3664e6fa47bd45a696708c4f080f8ca6%7C0%7C0%7C637363577426896014&sdata=rsjHPgka6gbLbZCeV9MaydW0o%2BznVrgz%2BXTp%2BPNHbdE%3D&reserved=0>> _______________________________________________ The Air-L@listserv.aoir.org<mailto:Air-L@listserv.aoir.org> mailing list is provided by the Association of Internet Researchers http://aoir.org<http://aoir.org> Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org<http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org> Join the Association of Internet Researchers: http://www.aoir.org/<http://www.aoir.org/>
participants (3)
-
Glen.Fuller -
Guido Anselmi -
Mosallaeipour,Fatemeh