Hi All, Can anyone recommend a way to download the feeds (historical tweets and re-tweets) of Twitter users? Is scraping for example using Data Miner a good idea? With the changes to Twitter's API rules, API-based websites such as Vicinitas.io have stopped working (or will stop soon), and I'm looking for other approaches. Thank you for your help. Kind regards, Rotem ----------- Dr Rotem Perach (he/him) Senior Research Fellow School of Social Sciences University of Westminster ORCID<https://orcid.org/0000-0002-8647-4367> | Twitter<https://twitter.com/DrRotemPerach/> The University of Westminster is a charity and a company limited by guarantee. Registration number: 977818 England. Registered Office: 309 Regent Street, London W1B 2HW. This message and its attachments are private and confidential. If you have received this message in error, please notify the sender and remove it and its attachments from your system.
Hi Rotem and All, I am also looking for solutions to this. I hope somebody has an idea of how to keep doing research on Twitter. Eleonora *--* Dr. Maria Eleonora Sciubba (she/her) (2022): “Adesso m’incazzo!”: Swearwords as resources for managing negative emotions in interaction <https://mediazioni.unibo.it/article/view/15263> (2021) https://rolsi.net/2021/06/02/guest-blog-em-ca-for-racial-justice/ *TSHD - Grant Design & Writing * *Department of Culture Studies - Senior Researcher* Editor, Internship Organizational Supervisor *Diggit Magazine* <https://www.diggitmagazine.com/> Tilburg University Twitter: @LolaSciubba *Be green. Keep it on the screen.* Il giorno mar 18 apr 2023 alle ore 11:15 Rotem Perach via Air-L < air-l@listserv.aoir.org> ha scritto:
Hi All,
Can anyone recommend a way to download the feeds (historical tweets and re-tweets) of Twitter users? Is scraping for example using Data Miner a good idea? With the changes to Twitter's API rules, API-based websites such as Vicinitas.io have stopped working (or will stop soon), and I'm looking for other approaches.
Thank you for your help.
Kind regards,
Rotem
-----------
Dr Rotem Perach (he/him) Senior Research Fellow School of Social Sciences University of Westminster
ORCID<https://orcid.org/0000-0002-8647-4367> | Twitter< https://twitter.com/DrRotemPerach/>
The University of Westminster is a charity and a company limited by guarantee. Registration number: 977818 England. Registered Office: 309 Regent Street, London W1B 2HW.
This message and its attachments are private and confidential. If you have received this message in error, please notify the sender and remove it and its attachments from your system. _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
The answer is complex, uncertain, and shifting. It is more than just a technical issue of "Where can I get data now?" There are fundamental legal, jurisdictional, market, equity, and political questions unresolved. Resolution seems unlikely in the short term. As a political scientist interested in election interference, I do not anticipate a positive resolution before the 2024 U.S. election. I would love to be wrong on that prediction. I have heard there are a few firms that retain agreements to market real time and historical Twitter data. You might ask your librarian about access to Crimson Hexagon, Meltwater, or other similar services with long term agreements Mr. Musk has not broken, yet. Nobody on the AIR-L list has confirmed whether the academic API is shut down. Is it? I don't have those credentials to check. As I have written previously, there are massive troves of Tweets in every computer science department and many social science departments. Any one of the raw JSON Tweet archives could be loaded into a free account on DiscoverText and then shared for teaching or research. Talk to the data folks on your campus and across your discipline about what is extant. Do not look for spreadsheets; find the raw JSON. Spreadsheets are significantly degraded historical objects that cannot be considered accurate representations of Tweets. Tweets live in the Twitter display and die in spreadsheets. I remain befuddled by the Python/R nexus. Those spreadsheets of Tweets simply are not Twitter. Any qualitative researcher knows what I mean. Meaning is produced in the interaction of the display elements. Meaning is destroyed, diluted, and bent out of shape in spreadsheets. The loss of real time access to new data is a problem and dangerous for democratic systems. This does not change the fact that so much data has been gathered and stored, like so many old newspapers. For example, here is a list of the Top 50 datasets in my DiscoverText account that any academic on this list or anywhere in the world can access via the "peer network" that enables collaboration, annotation, measurement of inter-rater reliability, and the creation of gold standard training sets for machine-learning via a graphical user interface that features the Twitter display. https://docs.google.com/spreadsheets/d/1j-Y40WmwFIX8pidfAcxgAt5JB3A3UyB2psh_... "The Case Against Spreadsheets as a Primary Twitter Research Tool" https://vimeo.com/526218014 Question: When is a Tweet not a Tweet? Answer: When it appears in a spreadsheet, or is deleted, or the account has been suspended. On Tue, Apr 18, 2023 at 5:56 AM M.E.Sciubba via Air-L < air-l@listserv.aoir.org> wrote:
Hi Rotem and All,
I am also looking for solutions to this. I hope somebody has an idea of how to keep doing research on Twitter.
Eleonora *--*
Dr. Maria Eleonora Sciubba (she/her) (2022): “Adesso m’incazzo!”: Swearwords as resources for managing negative emotions in interaction <https://mediazioni.unibo.it/article/view/15263> (2021) https://rolsi.net/2021/06/02/guest-blog-em-ca-for-racial-justice/
*TSHD - Grant Design & Writing * *Department of Culture Studies - Senior Researcher* Editor, Internship Organizational Supervisor *Diggit Magazine* <https://www.diggitmagazine.com/> Tilburg University Twitter: @LolaSciubba
*Be green. Keep it on the screen.*
Il giorno mar 18 apr 2023 alle ore 11:15 Rotem Perach via Air-L < air-l@listserv.aoir.org> ha scritto:
Hi All,
Can anyone recommend a way to download the feeds (historical tweets and re-tweets) of Twitter users? Is scraping for example using Data Miner a good idea? With the changes to Twitter's API rules, API-based websites such as Vicinitas.io have stopped working (or will stop soon), and I'm looking for other approaches.
Thank you for your help.
Kind regards,
Rotem
-----------
Dr Rotem Perach (he/him) Senior Research Fellow School of Social Sciences University of Westminster
ORCID<https://orcid.org/0000-0002-8647-4367> | Twitter< https://twitter.com/DrRotemPerach/>
The University of Westminster is a charity and a company limited by guarantee. Registration number: 977818 England. Registered Office: 309 Regent Street, London W1B 2HW.
This message and its attachments are private and confidential. If you have received this message in error, please notify the sender and remove it and its attachments from your system. _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
-- Dr. Stuart W. Shulman Founder and CEO, Texifter Editor Emeritus, *Journal of Information Technology & Politics*
The answer is complex, uncertain, and shifting. It is more than just a technical issue of "Where can I get data now?" There are fundamental legal, jurisdictional, market, equity, and political questions unresolved. Resolution seems unlikely in the short term. As a political scientist interested in election interference, I do not anticipate a positive resolution before the 2024 U.S. election. I would love to be wrong on that prediction. I have heard there are a few firms that retain agreements to market real time and historical Twitter data. You might ask your librarian about access to Crimson Hexagon, Meltwater, or other similar services with long term agreements Mr. Musk has not broken, yet. Nobody on the AIR-L list has confirmed whether the academic API is shut down. Is it? I don't have those credentials to check. As I have written previously, there are massive troves of Tweets in every computer science department and many social science departments. Any one of the raw JSON Tweet archives could be loaded into a free account on DiscoverText and then shared for teaching or research. Talk to the data folks on your campus and across your discipline about what is extant. Do not look for spreadsheets; find the raw JSON. Spreadsheets are significantly degraded historical objects that cannot be considered accurate representations of Tweets. Tweets live in the Twitter display and die in spreadsheets. I remain befuddled by the Python/R nexus. Those spreadsheets of Tweets simply are not Twitter. Any qualitative researcher knows what I mean. Meaning is produced in the interaction of the display elements. Meaning is destroyed, diluted, and bent out of shape in spreadsheets. The loss of real time access to new data is a problem and dangerous for democratic systems. This does not change the fact that so much data has been gathered and stored, like so many old newspapers. For example, here is a list of the Top 50 datasets in my DiscoverText account that any academic on this list or anywhere in the world can access via the "peer network" that enables collaboration, annotation, measurement of inter-rater reliability, and the creation of gold standard training sets for machine-learning via a graphical user interface that features the Twitter display. https://docs.google.com/spreadsheets/d/1j-Y40WmwFIX8pidfAcxgAt5JB3A3UyB2psh_... "The Case Against Spreadsheets as a Primary Twitter Research Tool" https://vimeo.com/526218014 Question: When is a Tweet not a Tweet? Answer: When it appears in a spreadsheet, or is deleted, or the account has been suspended. On Tue, Apr 18, 2023 at 5:57 AM M.E.Sciubba via Air-L < air-l@listserv.aoir.org> wrote:
Hi Rotem and All,
I am also looking for solutions to this. I hope somebody has an idea of how to keep doing research on Twitter.
Eleonora *--*
Dr. Maria Eleonora Sciubba (she/her) (2022): “Adesso m’incazzo!”: Swearwords as resources for managing negative emotions in interaction <https://mediazioni.unibo.it/article/view/15263> (2021) https://rolsi.net/2021/06/02/guest-blog-em-ca-for-racial-justice/
*TSHD - Grant Design & Writing * *Department of Culture Studies - Senior Researcher* Editor, Internship Organizational Supervisor *Diggit Magazine* <https://www.diggitmagazine.com/> Tilburg University Twitter: @LolaSciubba
*Be green. Keep it on the screen.*
Il giorno mar 18 apr 2023 alle ore 11:15 Rotem Perach via Air-L < air-l@listserv.aoir.org> ha scritto:
Hi All,
Can anyone recommend a way to download the feeds (historical tweets and re-tweets) of Twitter users? Is scraping for example using Data Miner a good idea? With the changes to Twitter's API rules, API-based websites such as Vicinitas.io have stopped working (or will stop soon), and I'm looking for other approaches.
Thank you for your help.
Kind regards,
Rotem
-----------
Dr Rotem Perach (he/him) Senior Research Fellow School of Social Sciences University of Westminster
ORCID<https://orcid.org/0000-0002-8647-4367> | Twitter< https://twitter.com/DrRotemPerach/>
The University of Westminster is a charity and a company limited by guarantee. Registration number: 977818 England. Registered Office: 309 Regent Street, London W1B 2HW.
This message and its attachments are private and confidential. If you have received this message in error, please notify the sender and remove it and its attachments from your system. _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
I can confirm that the Academic API v2 for Twitter is working, at least outside US. Use a python library like Tweepy and a code like the following you can get a stream of data from tweets in US. You will need to create a project in the Dev Dashboard and add your previous Apps to the project. The project should have been created for research purposes and the app too: import tweepy import operator class MyStream2(tweepy.StreamingClient): def on_tweet(self, tweet): print(tweet.text) streaming_client = MyStream2(BEARER_TOKEN) res = streaming_client.get_rules() if res.data: rules = list(map(operator.attrgetter("id"),res.data)) streaming_client.delete_rules(rules) streaming_client.add_rules(tweepy.StreamRule("place_country:US")) streaming_client.filter() Marc -----Original Message----- From: Air-L <air-l-bounces@listserv.aoir.org> On Behalf Of Stuart Shulman via Air-L Sent: Tuesday, April 18, 2023 12:46 PM To: M.E.Sciubba <mesciubba@gmail.com> Cc: air-l@listserv.aoir.org Subject: Re: [Air-L] Downloading Twitter feeds The answer is complex, uncertain, and shifting. It is more than just a technical issue of "Where can I get data now?" There are fundamental legal, jurisdictional, market, equity, and political questions unresolved. Resolution seems unlikely in the short term. As a political scientist interested in election interference, I do not anticipate a positive resolution before the 2024 U.S. election. I would love to be wrong on that prediction. I have heard there are a few firms that retain agreements to market real time and historical Twitter data. You might ask your librarian about access to Crimson Hexagon, Meltwater, or other similar services with long term agreements Mr. Musk has not broken, yet. Nobody on the AIR-L list has confirmed whether the academic API is shut down. Is it? I don't have those credentials to check. As I have written previously, there are massive troves of Tweets in every computer science department and many social science departments. Any one of the raw JSON Tweet archives could be loaded into a free account on DiscoverText and then shared for teaching or research. Talk to the data folks on your campus and across your discipline about what is extant. Do not look for spreadsheets; find the raw JSON. Spreadsheets are significantly degraded historical objects that cannot be considered accurate representations of Tweets. Tweets live in the Twitter display and die in spreadsheets. I remain befuddled by the Python/R nexus. Those spreadsheets of Tweets simply are not Twitter. Any qualitative researcher knows what I mean. Meaning is produced in the interaction of the display elements. Meaning is destroyed, diluted, and bent out of shape in spreadsheets. The loss of real time access to new data is a problem and dangerous for democratic systems. This does not change the fact that so much data has been gathered and stored, like so many old newspapers. For example, here is a list of the Top 50 datasets in my DiscoverText account that any academic on this list or anywhere in the world can access via the "peer network" that enables collaboration, annotation, measurement of inter-rater reliability, and the creation of gold standard training sets for machine-learning via a graphical user interface that features the Twitter display. https://docs.google.com/spreadsheets/d/1j-Y40WmwFIX8pidfAcxgAt5JB3A3UyB2psh_... "The Case Against Spreadsheets as a Primary Twitter Research Tool" https://vimeo.com/526218014 Question: When is a Tweet not a Tweet? Answer: When it appears in a spreadsheet, or is deleted, or the account has been suspended. On Tue, Apr 18, 2023 at 5:57 AM M.E.Sciubba via Air-L < air-l@listserv.aoir.org> wrote:
Hi Rotem and All,
I am also looking for solutions to this. I hope somebody has an idea of how to keep doing research on Twitter.
Eleonora *--*
Dr. Maria Eleonora Sciubba (she/her) (2022): “Adesso m’incazzo!”: Swearwords as resources for managing negative emotions in interaction <https://mediazioni.unibo.it/article/view/15263> (2021) https://rolsi.net/2021/06/02/guest-blog-em-ca-for-racial-justice/
*TSHD - Grant Design & Writing * *Department of Culture Studies - Senior Researcher* Editor, Internship Organizational Supervisor *Diggit Magazine* <https://www.diggitmagazine.com/> Tilburg University Twitter: @LolaSciubba
*Be green. Keep it on the screen.*
Il giorno mar 18 apr 2023 alle ore 11:15 Rotem Perach via Air-L < air-l@listserv.aoir.org> ha scritto:
Hi All,
Can anyone recommend a way to download the feeds (historical tweets and re-tweets) of Twitter users? Is scraping for example using Data Miner a good idea? With the changes to Twitter's API rules, API-based websites such as Vicinitas.io have stopped working (or will stop soon), and I'm looking for other approaches.
Thank you for your help.
Kind regards,
Rotem
-----------
Dr Rotem Perach (he/him) Senior Research Fellow School of Social Sciences University of Westminster
ORCID<https://orcid.org/0000-0002-8647-4367> | Twitter< https://twitter.com/DrRotemPerach/>
The University of Westminster is a charity and a company limited by guarantee. Registration number: 977818 England. Registered Office: 309 Regent Street, London W1B 2HW.
This message and its attachments are private and confidential. If you have received this message in error, please notify the sender and remove it and its attachments from your system. _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/
It works with Twitter Developer credentials with NodeXL's import function for social network analysis. On Wed, Apr 19, 2023 at 4:40 PM Marc Gallofré Ocaña via Air-L < air-l@listserv.aoir.org> wrote:
I can confirm that the Academic API v2 for Twitter is working, at least outside US.
Use a python library like Tweepy and a code like the following you can get a stream of data from tweets in US. You will need to create a project in the Dev Dashboard and add your previous Apps to the project. The project should have been created for research purposes and the app too:
import tweepy import operator
class MyStream2(tweepy.StreamingClient):
def on_tweet(self, tweet): print(tweet.text)
streaming_client = MyStream2(BEARER_TOKEN) res = streaming_client.get_rules() if res.data: rules = list(map(operator.attrgetter("id"),res.data)) streaming_client.delete_rules(rules) streaming_client.add_rules(tweepy.StreamRule("place_country:US")) streaming_client.filter()
Marc
-----Original Message----- From: Air-L <air-l-bounces@listserv.aoir.org> On Behalf Of Stuart Shulman via Air-L Sent: Tuesday, April 18, 2023 12:46 PM To: M.E.Sciubba <mesciubba@gmail.com> Cc: air-l@listserv.aoir.org Subject: Re: [Air-L] Downloading Twitter feeds
The answer is complex, uncertain, and shifting. It is more than just a technical issue of "Where can I get data now?" There are fundamental legal, jurisdictional, market, equity, and political questions unresolved. Resolution seems unlikely in the short term. As a political scientist interested in election interference, I do not anticipate a positive resolution before the 2024 U.S. election. I would love to be wrong on that prediction. I have heard there are a few firms that retain agreements to market real time and historical Twitter data. You might ask your librarian about access to Crimson Hexagon, Meltwater, or other similar services with long term agreements Mr. Musk has not broken, yet. Nobody on the AIR-L list has confirmed whether the academic API is shut down. Is it? I don't have those credentials to check.
As I have written previously, there are massive troves of Tweets in every computer science department and many social science departments. Any one of the raw JSON Tweet archives could be loaded into a free account on DiscoverText and then shared for teaching or research. Talk to the data folks on your campus and across your discipline about what is extant. Do not look for spreadsheets; find the raw JSON. Spreadsheets are significantly degraded historical objects that cannot be considered accurate representations of Tweets. Tweets live in the Twitter display and die in spreadsheets. I remain befuddled by the Python/R nexus. Those spreadsheets of Tweets simply are not Twitter. Any qualitative researcher knows what I mean. Meaning is produced in the interaction of the display elements. Meaning is destroyed, diluted, and bent out of shape in spreadsheets.
The loss of real time access to new data is a problem and dangerous for democratic systems. This does not change the fact that so much data has been gathered and stored, like so many old newspapers. For example, here is a list of the Top 50 datasets in my DiscoverText account that any academic on this list or anywhere in the world can access via the "peer network" that enables collaboration, annotation, measurement of inter-rater reliability, and the creation of gold standard training sets for machine-learning via a graphical user interface that features the Twitter display.
https://docs.google.com/spreadsheets/d/1j-Y40WmwFIX8pidfAcxgAt5JB3A3UyB2psh_...
"The Case Against Spreadsheets as a Primary Twitter Research Tool" https://vimeo.com/526218014 Question: When is a Tweet not a Tweet? Answer: When it appears in a spreadsheet, or is deleted, or the account has been suspended.
On Tue, Apr 18, 2023 at 5:57 AM M.E.Sciubba via Air-L < air-l@listserv.aoir.org> wrote:
Hi Rotem and All,
I am also looking for solutions to this. I hope somebody has an idea of how to keep doing research on Twitter.
Eleonora *--*
Dr. Maria Eleonora Sciubba (she/her) (2022): “Adesso m’incazzo!”: Swearwords as resources for managing negative emotions in interaction <https://mediazioni.unibo.it/article/view/15263> (2021) https://rolsi.net/2021/06/02/guest-blog-em-ca-for-racial-justice/
*TSHD - Grant Design & Writing * *Department of Culture Studies - Senior Researcher* Editor, Internship Organizational Supervisor *Diggit Magazine* <https://www.diggitmagazine.com/> Tilburg University Twitter: @LolaSciubba
*Be green. Keep it on the screen.*
Il giorno mar 18 apr 2023 alle ore 11:15 Rotem Perach via Air-L < air-l@listserv.aoir.org> ha scritto:
Hi All,
Can anyone recommend a way to download the feeds (historical tweets and re-tweets) of Twitter users? Is scraping for example using Data Miner a good idea? With the changes to Twitter's API rules, API-based websites such as Vicinitas.io have stopped working (or will stop soon), and I'm looking for other approaches.
Thank you for your help.
Kind regards,
Rotem
-----------
Dr Rotem Perach (he/him) Senior Research Fellow School of Social Sciences University of Westminster
ORCID<https://orcid.org/0000-0002-8647-4367> | Twitter< https://twitter.com/DrRotemPerach/>
The University of Westminster is a charity and a company limited by guarantee. Registration number: 977818 England. Registered Office: 309 Regent Street, London W1B 2HW.
This message and its attachments are private and confidential. If you have received this message in error, please notify the sender and remove it and its attachments from your system. _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/ _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
Hello I have an ethical question about scraping data from FB (that is already in the public domain) I am preparing a project for ethical clearance. The research focuses on two community FB groups of 3000 and 7000 members respectively and of which I am a member of both. As part of the research methods I will automate the collection of data for analysis (partly on spreadsheets), all of this would be possible manually if I had the time – automation allows me to collect a lot of data quickly – it is not extending access. If this data is anonymised and not published would that be enough to protect the privacy of members within the group? I wondered if anyone had experience of doing this and what additional safeguards they needed to implement for ethical clearance. The problem here is that a closed group is semi-private, individuals may feel like it is private even if it spans the whole community (this time a small town). Thanks From: Air-L <air-l-bounces@listserv.aoir.org> on behalf of Stuart Shulman via Air-L <air-l@listserv.aoir.org> Date: Tuesday, 18 April 2023 at 12:10 To: M.E.Sciubba <mesciubba@gmail.com> Cc: air-l@listserv.aoir.org <air-l@listserv.aoir.org> Subject: Re: [Air-L] Downloading Twitter feeds CAUTION: This message was sent from outside the University, purportedly from air-l-bounces@listserv.aoir.org . Please check the sender is legitimate before responding. Please treat any links or attachments with care - do not follow or open them unless you are sure they are genuine. The answer is complex, uncertain, and shifting. It is more than just a technical issue of "Where can I get data now?" There are fundamental legal, jurisdictional, market, equity, and political questions unresolved. Resolution seems unlikely in the short term. As a political scientist interested in election interference, I do not anticipate a positive resolution before the 2024 U.S. election. I would love to be wrong on that prediction. I have heard there are a few firms that retain agreements to market real time and historical Twitter data. You might ask your librarian about access to Crimson Hexagon, Meltwater, or other similar services with long term agreements Mr. Musk has not broken, yet. Nobody on the AIR-L list has confirmed whether the academic API is shut down. Is it? I don't have those credentials to check. As I have written previously, there are massive troves of Tweets in every computer science department and many social science departments. Any one of the raw JSON Tweet archives could be loaded into a free account on DiscoverText and then shared for teaching or research. Talk to the data folks on your campus and across your discipline about what is extant. Do not look for spreadsheets; find the raw JSON. Spreadsheets are significantly degraded historical objects that cannot be considered accurate representations of Tweets. Tweets live in the Twitter display and die in spreadsheets. I remain befuddled by the Python/R nexus. Those spreadsheets of Tweets simply are not Twitter. Any qualitative researcher knows what I mean. Meaning is produced in the interaction of the display elements. Meaning is destroyed, diluted, and bent out of shape in spreadsheets. The loss of real time access to new data is a problem and dangerous for democratic systems. This does not change the fact that so much data has been gathered and stored, like so many old newspapers. For example, here is a list of the Top 50 datasets in my DiscoverText account that any academic on this list or anywhere in the world can access via the "peer network" that enables collaboration, annotation, measurement of inter-rater reliability, and the creation of gold standard training sets for machine-learning via a graphical user interface that features the Twitter display. https://docs.google.com/spreadsheets/d/1j-Y40WmwFIX8pidfAcxgAt5JB3A3UyB2psh_... "The Case Against Spreadsheets as a Primary Twitter Research Tool" https://vimeo.com/526218014 Question: When is a Tweet not a Tweet? Answer: When it appears in a spreadsheet, or is deleted, or the account has been suspended. On Tue, Apr 18, 2023 at 5:57 AM M.E.Sciubba via Air-L < air-l@listserv.aoir.org> wrote:
Hi Rotem and All,
I am also looking for solutions to this. I hope somebody has an idea of how to keep doing research on Twitter.
Eleonora *--*
Dr. Maria Eleonora Sciubba (she/her) (2022): “Adesso m’incazzo!”: Swearwords as resources for managing negative emotions in interaction <https://mediazioni.unibo.it/article/view/15263> (2021) https://rolsi.net/2021/06/02/guest-blog-em-ca-for-racial-justice/
*TSHD - Grant Design & Writing * *Department of Culture Studies - Senior Researcher* Editor, Internship Organizational Supervisor *Diggit Magazine* <https://www.diggitmagazine.com/> Tilburg University Twitter: @LolaSciubba
*Be green. Keep it on the screen.*
Il giorno mar 18 apr 2023 alle ore 11:15 Rotem Perach via Air-L < air-l@listserv.aoir.org> ha scritto:
Hi All,
Can anyone recommend a way to download the feeds (historical tweets and re-tweets) of Twitter users? Is scraping for example using Data Miner a good idea? With the changes to Twitter's API rules, API-based websites such as Vicinitas.io have stopped working (or will stop soon), and I'm looking for other approaches.
Thank you for your help.
Kind regards,
Rotem
-----------
Dr Rotem Perach (he/him) Senior Research Fellow School of Social Sciences University of Westminster
ORCID<https://orcid.org/0000-0002-8647-4367> | Twitter< https://twitter.com/DrRotemPerach/><https://twitter.com/DrRotemPerach/%3e>
The University of Westminster is a charity and a company limited by guarantee. Registration number: 977818 England. Registered Office: 309 Regent Street, London W1B 2HW.
This message and its attachments are private and confidential. If you have received this message in error, please notify the sender and remove it and its attachments from your system. _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/
Hi Rose, I am very interested in how you solve that puzzle. I have been working lately with WhatsApp groups that are publicly available and anonymizing and performing aggregate analysis have gotten me through ethics approval. If you are going to perform more qualitative research it might be very advisable that users are aware of it and consent, which might prove very complicated. Let me know how this moves forward. Best wishes Marcelo Em sex., 21 de abr. de 2023 às 09:32, Butler, Rose via Air-L < air-l@listserv.aoir.org> escreveu:
Hello I have an ethical question about scraping data from FB (that is already in the public domain)
I am preparing a project for ethical clearance. The research focuses on two community FB groups of 3000 and 7000 members respectively and of which I am a member of both. As part of the research methods I will automate the collection of data for analysis (partly on spreadsheets), all of this would be possible manually if I had the time – automation allows me to collect a lot of data quickly – it is not extending access.
If this data is anonymised and not published would that be enough to protect the privacy of members within the group? I wondered if anyone had experience of doing this and what additional safeguards they needed to implement for ethical clearance. The problem here is that a closed group is semi-private, individuals may feel like it is private even if it spans the whole community (this time a small town).
Thanks
From: Air-L <air-l-bounces@listserv.aoir.org> on behalf of Stuart Shulman via Air-L <air-l@listserv.aoir.org> Date: Tuesday, 18 April 2023 at 12:10 To: M.E.Sciubba <mesciubba@gmail.com> Cc: air-l@listserv.aoir.org <air-l@listserv.aoir.org> Subject: Re: [Air-L] Downloading Twitter feeds CAUTION: This message was sent from outside the University, purportedly from air-l-bounces@listserv.aoir.org .
Please check the sender is legitimate before responding. Please treat any links or attachments with care - do not follow or open them unless you are sure they are genuine.
The answer is complex, uncertain, and shifting. It is more than just a technical issue of "Where can I get data now?" There are fundamental legal, jurisdictional, market, equity, and political questions unresolved. Resolution seems unlikely in the short term. As a political scientist interested in election interference, I do not anticipate a positive resolution before the 2024 U.S. election. I would love to be wrong on that prediction. I have heard there are a few firms that retain agreements to market real time and historical Twitter data. You might ask your librarian about access to Crimson Hexagon, Meltwater, or other similar services with long term agreements Mr. Musk has not broken, yet. Nobody on the AIR-L list has confirmed whether the academic API is shut down. Is it? I don't have those credentials to check.
As I have written previously, there are massive troves of Tweets in every computer science department and many social science departments. Any one of the raw JSON Tweet archives could be loaded into a free account on DiscoverText and then shared for teaching or research. Talk to the data folks on your campus and across your discipline about what is extant. Do not look for spreadsheets; find the raw JSON. Spreadsheets are significantly degraded historical objects that cannot be considered accurate representations of Tweets. Tweets live in the Twitter display and die in spreadsheets. I remain befuddled by the Python/R nexus. Those spreadsheets of Tweets simply are not Twitter. Any qualitative researcher knows what I mean. Meaning is produced in the interaction of the display elements. Meaning is destroyed, diluted, and bent out of shape in spreadsheets.
The loss of real time access to new data is a problem and dangerous for democratic systems. This does not change the fact that so much data has been gathered and stored, like so many old newspapers. For example, here is a list of the Top 50 datasets in my DiscoverText account that any academic on this list or anywhere in the world can access via the "peer network" that enables collaboration, annotation, measurement of inter-rater reliability, and the creation of gold standard training sets for machine-learning via a graphical user interface that features the Twitter display.
https://docs.google.com/spreadsheets/d/1j-Y40WmwFIX8pidfAcxgAt5JB3A3UyB2psh_...
"The Case Against Spreadsheets as a Primary Twitter Research Tool" https://vimeo.com/526218014 Question: When is a Tweet not a Tweet? Answer: When it appears in a spreadsheet, or is deleted, or the account has been suspended.
On Tue, Apr 18, 2023 at 5:57 AM M.E.Sciubba via Air-L < air-l@listserv.aoir.org> wrote:
Hi Rotem and All,
I am also looking for solutions to this. I hope somebody has an idea of how to keep doing research on Twitter.
Eleonora *--*
Dr. Maria Eleonora Sciubba (she/her) (2022): “Adesso m’incazzo!”: Swearwords as resources for managing negative emotions in interaction <https://mediazioni.unibo.it/article/view/15263> (2021) https://rolsi.net/2021/06/02/guest-blog-em-ca-for-racial-justice/
*TSHD - Grant Design & Writing * *Department of Culture Studies - Senior Researcher* Editor, Internship Organizational Supervisor *Diggit Magazine* <https://www.diggitmagazine.com/> Tilburg University Twitter: @LolaSciubba
*Be green. Keep it on the screen.*
Il giorno mar 18 apr 2023 alle ore 11:15 Rotem Perach via Air-L < air-l@listserv.aoir.org> ha scritto:
Hi All,
Can anyone recommend a way to download the feeds (historical tweets and re-tweets) of Twitter users? Is scraping for example using Data Miner a good idea? With the changes to Twitter's API rules, API-based websites such as Vicinitas.io have stopped working (or will stop soon), and I'm looking for other approaches.
Thank you for your help.
Kind regards,
Rotem
-----------
Dr Rotem Perach (he/him) Senior Research Fellow School of Social Sciences University of Westminster
ORCID<https://orcid.org/0000-0002-8647-4367> | Twitter< https://twitter.com/DrRotemPerach/><; https://twitter.com/DrRotemPerach/%3e>
The University of Westminster is a charity and a company limited by guarantee. Registration number: 977818 England. Registered Office: 309 Regent Street, London W1B 2HW.
This message and its attachments are private and confidential. If you have received this message in error, please notify the sender and remove it and its attachments from your system. _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/ _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
Hi Rose I suspect you will not be able to collect any data from Facebook, no matter what ethic approval you have, as their terms of service<https://www.facebook.com/legal/terms> say (in 3.2.3): You may not access or collect data from our Products using automated means (without our prior permission) or attempt to access data that you do not have permission to access. We also reserve all of our rights against text and data mining. which basically says you cannot scrape content, but I would be interested to know if the legal team at your institution thinks otherwise? Floriana ________________________________ From: Air-L <air-l-bounces@listserv.aoir.org> on behalf of Marcelo Santos via Air-L <air-l@listserv.aoir.org> Sent: 22 April 2023 18:50 To: Butler, Rose <R.Butler@shu.ac.uk> Cc: air-l@listserv.aoir.org <air-l@listserv.aoir.org> Subject: Re: [Air-L] Downloading Twitter feeds Hi Rose, I am very interested in how you solve that puzzle. I have been working lately with WhatsApp groups that are publicly available and anonymizing and performing aggregate analysis have gotten me through ethics approval. If you are going to perform more qualitative research it might be very advisable that users are aware of it and consent, which might prove very complicated. Let me know how this moves forward. Best wishes Marcelo Em sex., 21 de abr. de 2023 às 09:32, Butler, Rose via Air-L < air-l@listserv.aoir.org> escreveu:
Hello I have an ethical question about scraping data from FB (that is already in the public domain)
I am preparing a project for ethical clearance. The research focuses on two community FB groups of 3000 and 7000 members respectively and of which I am a member of both. As part of the research methods I will automate the collection of data for analysis (partly on spreadsheets), all of this would be possible manually if I had the time – automation allows me to collect a lot of data quickly – it is not extending access.
If this data is anonymised and not published would that be enough to protect the privacy of members within the group? I wondered if anyone had experience of doing this and what additional safeguards they needed to implement for ethical clearance. The problem here is that a closed group is semi-private, individuals may feel like it is private even if it spans the whole community (this time a small town).
Thanks
From: Air-L <air-l-bounces@listserv.aoir.org> on behalf of Stuart Shulman via Air-L <air-l@listserv.aoir.org> Date: Tuesday, 18 April 2023 at 12:10 To: M.E.Sciubba <mesciubba@gmail.com> Cc: air-l@listserv.aoir.org <air-l@listserv.aoir.org> Subject: Re: [Air-L] Downloading Twitter feeds CAUTION: This message was sent from outside the University, purportedly from air-l-bounces@listserv.aoir.org .
Please check the sender is legitimate before responding. Please treat any links or attachments with care - do not follow or open them unless you are sure they are genuine.
The answer is complex, uncertain, and shifting. It is more than just a technical issue of "Where can I get data now?" There are fundamental legal, jurisdictional, market, equity, and political questions unresolved. Resolution seems unlikely in the short term. As a political scientist interested in election interference, I do not anticipate a positive resolution before the 2024 U.S. election. I would love to be wrong on that prediction. I have heard there are a few firms that retain agreements to market real time and historical Twitter data. You might ask your librarian about access to Crimson Hexagon, Meltwater, or other similar services with long term agreements Mr. Musk has not broken, yet. Nobody on the AIR-L list has confirmed whether the academic API is shut down. Is it? I don't have those credentials to check.
As I have written previously, there are massive troves of Tweets in every computer science department and many social science departments. Any one of the raw JSON Tweet archives could be loaded into a free account on DiscoverText and then shared for teaching or research. Talk to the data folks on your campus and across your discipline about what is extant. Do not look for spreadsheets; find the raw JSON. Spreadsheets are significantly degraded historical objects that cannot be considered accurate representations of Tweets. Tweets live in the Twitter display and die in spreadsheets. I remain befuddled by the Python/R nexus. Those spreadsheets of Tweets simply are not Twitter. Any qualitative researcher knows what I mean. Meaning is produced in the interaction of the display elements. Meaning is destroyed, diluted, and bent out of shape in spreadsheets.
The loss of real time access to new data is a problem and dangerous for democratic systems. This does not change the fact that so much data has been gathered and stored, like so many old newspapers. For example, here is a list of the Top 50 datasets in my DiscoverText account that any academic on this list or anywhere in the world can access via the "peer network" that enables collaboration, annotation, measurement of inter-rater reliability, and the creation of gold standard training sets for machine-learning via a graphical user interface that features the Twitter display.
https://docs.google.com/spreadsheets/d/1j-Y40WmwFIX8pidfAcxgAt5JB3A3UyB2psh_...
"The Case Against Spreadsheets as a Primary Twitter Research Tool" https://vimeo.com/526218014 Question: When is a Tweet not a Tweet? Answer: When it appears in a spreadsheet, or is deleted, or the account has been suspended.
On Tue, Apr 18, 2023 at 5:57 AM M.E.Sciubba via Air-L < air-l@listserv.aoir.org> wrote:
Hi Rotem and All,
I am also looking for solutions to this. I hope somebody has an idea of how to keep doing research on Twitter.
Eleonora *--*
Dr. Maria Eleonora Sciubba (she/her) (2022): “Adesso m’incazzo!”: Swearwords as resources for managing negative emotions in interaction <https://mediazioni.unibo.it/article/view/15263> (2021) https://rolsi.net/2021/06/02/guest-blog-em-ca-for-racial-justice/
*TSHD - Grant Design & Writing * *Department of Culture Studies - Senior Researcher* Editor, Internship Organizational Supervisor *Diggit Magazine* <https://www.diggitmagazine.com/> Tilburg University Twitter: @LolaSciubba
*Be green. Keep it on the screen.*
Il giorno mar 18 apr 2023 alle ore 11:15 Rotem Perach via Air-L < air-l@listserv.aoir.org> ha scritto:
Hi All,
Can anyone recommend a way to download the feeds (historical tweets and re-tweets) of Twitter users? Is scraping for example using Data Miner a good idea? With the changes to Twitter's API rules, API-based websites such as Vicinitas.io have stopped working (or will stop soon), and I'm looking for other approaches.
Thank you for your help.
Kind regards,
Rotem
-----------
Dr Rotem Perach (he/him) Senior Research Fellow School of Social Sciences University of Westminster
ORCID<https://orcid.org/0000-0002-8647-4367> | Twitter< https://twitter.com/DrRotemPerach/><; https://twitter.com/DrRotemPerach/%3e>
The University of Westminster is a charity and a company limited by guarantee. Registration number: 977818 England. Registered Office: 309 Regent Street, London W1B 2HW.
This message and its attachments are private and confidential. If you have received this message in error, please notify the sender and remove it and its attachments from your system. _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/ _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/
Good point Floriana, but again: are a private company's terms of service a legal boundary, sufficient to prevent researchers to deal with issues of public interest. If you ask me, no. But I know many IRBs follow the terms of service, which is a very lazy shortcut, IMHO, to solve complex ethical issues. I would very much like to know what others think, though. Best Marcelo Em sáb., 22 de abr. de 2023 às 14:11, Grasso, Floriana < floriana@liverpool.ac.uk> escreveu:
Hi Rose
I suspect you will not be able to collect *any* data from Facebook, no matter what ethic approval you have, as their terms of service <https://www.facebook.com/legal/terms> say (in 3.2.3):
You may not access or collect data from our Products using automated means (without our prior permission) or attempt to access data that you do not have permission to access. We also reserve all of our rights against text and data mining.
which basically says you cannot scrape content, but I would be interested to know if the legal team at your institution thinks otherwise?
Floriana
------------------------------ *From:* Air-L <air-l-bounces@listserv.aoir.org> on behalf of Marcelo Santos via Air-L <air-l@listserv.aoir.org> *Sent:* 22 April 2023 18:50 *To:* Butler, Rose <R.Butler@shu.ac.uk> *Cc:* air-l@listserv.aoir.org <air-l@listserv.aoir.org> *Subject:* Re: [Air-L] Downloading Twitter feeds
Hi Rose, I am very interested in how you solve that puzzle.
I have been working lately with WhatsApp groups that are publicly available and anonymizing and performing aggregate analysis have gotten me through ethics approval. If you are going to perform more qualitative research it might be very advisable that users are aware of it and consent, which might prove very complicated.
Let me know how this moves forward.
Best wishes Marcelo
Em sex., 21 de abr. de 2023 às 09:32, Butler, Rose via Air-L < air-l@listserv.aoir.org> escreveu:
Hello I have an ethical question about scraping data from FB (that is already in the public domain)
I am preparing a project for ethical clearance. The research focuses on two community FB groups of 3000 and 7000 members respectively and of which I am a member of both. As part of the research methods I will automate the collection of data for analysis (partly on spreadsheets), all of this would be possible manually if I had the time – automation allows me to collect a lot of data quickly – it is not extending access.
If this data is anonymised and not published would that be enough to protect the privacy of members within the group? I wondered if anyone had experience of doing this and what additional safeguards they needed to implement for ethical clearance. The problem here is that a closed group is semi-private, individuals may feel like it is private even if it spans the whole community (this time a small town).
Thanks
From: Air-L <air-l-bounces@listserv.aoir.org> on behalf of Stuart Shulman via Air-L <air-l@listserv.aoir.org> Date: Tuesday, 18 April 2023 at 12:10 To: M.E.Sciubba <mesciubba@gmail.com> Cc: air-l@listserv.aoir.org <air-l@listserv.aoir.org> Subject: Re: [Air-L] Downloading Twitter feeds CAUTION: This message was sent from outside the University, purportedly from air-l-bounces@listserv.aoir.org .
Please check the sender is legitimate before responding. Please treat any links or attachments with care - do not follow or open them unless you are sure they are genuine.
The answer is complex, uncertain, and shifting. It is more than just a technical issue of "Where can I get data now?" There are fundamental legal, jurisdictional, market, equity, and political questions unresolved. Resolution seems unlikely in the short term. As a political scientist interested in election interference, I do not anticipate a positive resolution before the 2024 U.S. election. I would love to be wrong on that prediction. I have heard there are a few firms that retain agreements to market real time and historical Twitter data. You might ask your librarian about access to Crimson Hexagon, Meltwater, or other similar services with long term agreements Mr. Musk has not broken, yet. Nobody on the AIR-L list has confirmed whether the academic API is shut down. Is it? I don't have those credentials to check.
As I have written previously, there are massive troves of Tweets in every computer science department and many social science departments. Any one of the raw JSON Tweet archives could be loaded into a free account on DiscoverText and then shared for teaching or research. Talk to the data folks on your campus and across your discipline about what is extant. Do not look for spreadsheets; find the raw JSON. Spreadsheets are significantly degraded historical objects that cannot be considered accurate representations of Tweets. Tweets live in the Twitter display and die in spreadsheets. I remain befuddled by the Python/R nexus. Those spreadsheets of Tweets simply are not Twitter. Any qualitative researcher knows what I mean. Meaning is produced in the interaction of the display elements. Meaning is destroyed, diluted, and bent out of shape in spreadsheets.
The loss of real time access to new data is a problem and dangerous for democratic systems. This does not change the fact that so much data has been gathered and stored, like so many old newspapers. For example, here is a list of the Top 50 datasets in my DiscoverText account that any academic on this list or anywhere in the world can access via the "peer network" that enables collaboration, annotation, measurement of inter-rater reliability, and the creation of gold standard training sets for machine-learning via a graphical user interface that features the Twitter display.
https://docs.google.com/spreadsheets/d/1j-Y40WmwFIX8pidfAcxgAt5JB3A3UyB2psh_...
"The Case Against Spreadsheets as a Primary Twitter Research Tool" https://vimeo.com/526218014 Question: When is a Tweet not a Tweet? Answer: When it appears in a spreadsheet, or is deleted, or the account has been suspended.
On Tue, Apr 18, 2023 at 5:57 AM M.E.Sciubba via Air-L < air-l@listserv.aoir.org> wrote:
Hi Rotem and All,
I am also looking for solutions to this. I hope somebody has an idea of how to keep doing research on Twitter.
Eleonora *--*
Dr. Maria Eleonora Sciubba (she/her) (2022): “Adesso m’incazzo!”: Swearwords as resources for managing negative emotions in interaction <
https://mediazioni.unibo.it/article/view/15263>
(2021) https://rolsi.net/2021/06/02/guest-blog-em-ca-for-racial-justice/
*TSHD - Grant Design & Writing * *Department of Culture Studies - Senior Researcher* Editor, Internship Organizational Supervisor *Diggit Magazine* <https://www.diggitmagazine.com/> Tilburg University Twitter: @LolaSciubba
*Be green. Keep it on the screen.*
Il giorno mar 18 apr 2023 alle ore 11:15 Rotem Perach via Air-L < air-l@listserv.aoir.org> ha scritto:
Hi All,
Can anyone recommend a way to download the feeds (historical tweets and re-tweets) of Twitter users? Is scraping for example using Data Miner a good idea? With the changes to Twitter's API rules, API-based websites such as Vicinitas.io have stopped working (or will stop soon), and I'm looking for other approaches.
Thank you for your help.
Kind regards,
Rotem
-----------
Dr Rotem Perach (he/him) Senior Research Fellow School of Social Sciences University of Westminster
ORCID<https://orcid.org/0000-0002-8647-4367> | Twitter< https://twitter.com/DrRotemPerach/><; https://twitter.com/DrRotemPerach/%3e> <https://twitter.com/DrRotemPerach/%3e%3E>
The University of Westminster is a charity and a company limited by guarantee. Registration number: 977818 England. Registered Office: 309 Regent Street, London W1B 2HW.
This message and its attachments are private and confidential. If you have received this message in error, please notify the sender and remove it and its attachments from your system. _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/ _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
Thanks Marcelo This was my question too and I think that this ethical discussion may well test some of those parameters. How might research for investigative journalism or security address these questions for example? From: Marcelo Santos <marcelolbsantos@gmail.com> Date: Sunday, 23 April 2023 at 14:50 To: Grasso, Floriana <floriana@liverpool.ac.uk> Cc: Butler, Rose <R.Butler@shu.ac.uk>, air-l@listserv.aoir.org <air-l@listserv.aoir.org> Subject: Re: [Air-L] Downloading Twitter feeds CAUTION: This message was sent from outside the University, purportedly from marcelolbsantos@gmail.com . Please check the sender is legitimate before responding. Please treat any links or attachments with care - do not follow or open them unless you are sure they are genuine. Good point Floriana, but again: are a private company's terms of service a legal boundary, sufficient to prevent researchers to deal with issues of public interest. If you ask me, no. But I know many IRBs follow the terms of service, which is a very lazy shortcut, IMHO, to solve complex ethical issues. I would very much like to know what others think, though. Best Marcelo Em sáb., 22 de abr. de 2023 às 14:11, Grasso, Floriana <floriana@liverpool.ac.uk<mailto:floriana@liverpool.ac.uk>> escreveu: Hi Rose I suspect you will not be able to collect any data from Facebook, no matter what ethic approval you have, as their terms of service<https://www.facebook.com/legal/terms> say (in 3.2.3): You may not access or collect data from our Products using automated means (without our prior permission) or attempt to access data that you do not have permission to access. We also reserve all of our rights against text and data mining. which basically says you cannot scrape content, but I would be interested to know if the legal team at your institution thinks otherwise? Floriana ________________________________ From: Air-L <air-l-bounces@listserv.aoir.org<mailto:air-l-bounces@listserv.aoir.org>> on behalf of Marcelo Santos via Air-L <air-l@listserv.aoir.org<mailto:air-l@listserv.aoir.org>> Sent: 22 April 2023 18:50 To: Butler, Rose <R.Butler@shu.ac.uk<mailto:R.Butler@shu.ac.uk>> Cc: air-l@listserv.aoir.org<mailto:air-l@listserv.aoir.org> <air-l@listserv.aoir.org<mailto:air-l@listserv.aoir.org>> Subject: Re: [Air-L] Downloading Twitter feeds Hi Rose, I am very interested in how you solve that puzzle. I have been working lately with WhatsApp groups that are publicly available and anonymizing and performing aggregate analysis have gotten me through ethics approval. If you are going to perform more qualitative research it might be very advisable that users are aware of it and consent, which might prove very complicated. Let me know how this moves forward. Best wishes Marcelo Em sex., 21 de abr. de 2023 às 09:32, Butler, Rose via Air-L < air-l@listserv.aoir.org<mailto:air-l@listserv.aoir.org>> escreveu:
Hello I have an ethical question about scraping data from FB (that is already in the public domain)
I am preparing a project for ethical clearance. The research focuses on two community FB groups of 3000 and 7000 members respectively and of which I am a member of both. As part of the research methods I will automate the collection of data for analysis (partly on spreadsheets), all of this would be possible manually if I had the time – automation allows me to collect a lot of data quickly – it is not extending access.
If this data is anonymised and not published would that be enough to protect the privacy of members within the group? I wondered if anyone had experience of doing this and what additional safeguards they needed to implement for ethical clearance. The problem here is that a closed group is semi-private, individuals may feel like it is private even if it spans the whole community (this time a small town).
Thanks
From: Air-L <air-l-bounces@listserv.aoir.org<mailto:air-l-bounces@listserv.aoir.org>> on behalf of Stuart Shulman via Air-L <air-l@listserv.aoir.org<mailto:air-l@listserv.aoir.org>> Date: Tuesday, 18 April 2023 at 12:10 To: M.E.Sciubba <mesciubba@gmail.com<mailto:mesciubba@gmail.com>> Cc: air-l@listserv.aoir.org<mailto:air-l@listserv.aoir.org> <air-l@listserv.aoir.org<mailto:air-l@listserv.aoir.org>> Subject: Re: [Air-L] Downloading Twitter feeds CAUTION: This message was sent from outside the University, purportedly from air-l-bounces@listserv.aoir.org<mailto:air-l-bounces@listserv.aoir.org> .
Please check the sender is legitimate before responding. Please treat any links or attachments with care - do not follow or open them unless you are sure they are genuine.
The answer is complex, uncertain, and shifting. It is more than just a technical issue of "Where can I get data now?" There are fundamental legal, jurisdictional, market, equity, and political questions unresolved. Resolution seems unlikely in the short term. As a political scientist interested in election interference, I do not anticipate a positive resolution before the 2024 U.S. election. I would love to be wrong on that prediction. I have heard there are a few firms that retain agreements to market real time and historical Twitter data. You might ask your librarian about access to Crimson Hexagon, Meltwater, or other similar services with long term agreements Mr. Musk has not broken, yet. Nobody on the AIR-L list has confirmed whether the academic API is shut down. Is it? I don't have those credentials to check.
As I have written previously, there are massive troves of Tweets in every computer science department and many social science departments. Any one of the raw JSON Tweet archives could be loaded into a free account on DiscoverText and then shared for teaching or research. Talk to the data folks on your campus and across your discipline about what is extant. Do not look for spreadsheets; find the raw JSON. Spreadsheets are significantly degraded historical objects that cannot be considered accurate representations of Tweets. Tweets live in the Twitter display and die in spreadsheets. I remain befuddled by the Python/R nexus. Those spreadsheets of Tweets simply are not Twitter. Any qualitative researcher knows what I mean. Meaning is produced in the interaction of the display elements. Meaning is destroyed, diluted, and bent out of shape in spreadsheets.
The loss of real time access to new data is a problem and dangerous for democratic systems. This does not change the fact that so much data has been gathered and stored, like so many old newspapers. For example, here is a list of the Top 50 datasets in my DiscoverText account that any academic on this list or anywhere in the world can access via the "peer network" that enables collaboration, annotation, measurement of inter-rater reliability, and the creation of gold standard training sets for machine-learning via a graphical user interface that features the Twitter display.
https://docs.google.com/spreadsheets/d/1j-Y40WmwFIX8pidfAcxgAt5JB3A3UyB2psh_...
"The Case Against Spreadsheets as a Primary Twitter Research Tool" https://vimeo.com/526218014 Question: When is a Tweet not a Tweet? Answer: When it appears in a spreadsheet, or is deleted, or the account has been suspended.
On Tue, Apr 18, 2023 at 5:57 AM M.E.Sciubba via Air-L < air-l@listserv.aoir.org<mailto:air-l@listserv.aoir.org>> wrote:
Hi Rotem and All,
I am also looking for solutions to this. I hope somebody has an idea of how to keep doing research on Twitter.
Eleonora *--*
Dr. Maria Eleonora Sciubba (she/her) (2022): “Adesso m’incazzo!”: Swearwords as resources for managing negative emotions in interaction <https://mediazioni.unibo.it/article/view/15263> (2021) https://rolsi.net/2021/06/02/guest-blog-em-ca-for-racial-justice/
*TSHD - Grant Design & Writing * *Department of Culture Studies - Senior Researcher* Editor, Internship Organizational Supervisor *Diggit Magazine* <https://www.diggitmagazine.com/> Tilburg University Twitter: @LolaSciubba
*Be green. Keep it on the screen.*
Il giorno mar 18 apr 2023 alle ore 11:15 Rotem Perach via Air-L < air-l@listserv.aoir.org<mailto:air-l@listserv.aoir.org>> ha scritto:
Hi All,
Can anyone recommend a way to download the feeds (historical tweets and re-tweets) of Twitter users? Is scraping for example using Data Miner a good idea? With the changes to Twitter's API rules, API-based websites such as Vicinitas.io have stopped working (or will stop soon), and I'm looking for other approaches.
Thank you for your help.
Kind regards,
Rotem
-----------
Dr Rotem Perach (he/him) Senior Research Fellow School of Social Sciences University of Westminster
ORCID<https://orcid.org/0000-0002-8647-4367> | Twitter< https://twitter.com/DrRotemPerach/><<https://twitter.com/DrRotemPerach/%3E%3C> https://twitter.com/DrRotemPerach/%3e><https://twitter.com/DrRotemPerach/%3e%3E>
The University of Westminster is a charity and a company limited by guarantee. Registration number: 977818 England. Registered Office: 309 Regent Street, London W1B 2HW.
This message and its attachments are private and confidential. If you have received this message in error, please notify the sender and remove it and its attachments from your system. _______________________________________________ The Air-L@listserv.aoir.org<mailto:Air-L@listserv.aoir.org> mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org<mailto:Air-L@listserv.aoir.org> mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org<mailto:Air-L@listserv.aoir.org> mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/ _______________________________________________ The Air-L@listserv.aoir.org<mailto:Air-L@listserv.aoir.org> mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org<mailto:Air-L@listserv.aoir.org> mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/
I work in government research with massive surveys and tax data in Canada. Privacy and protecting confidentiality is a large part of my job. This topic is also well developed in census, health research and other areas in many countries in the USA, Europe and elsewhere. You can look for these best practices in these fields where you will find simple to complex methods to assess privacy invasion risk. I have to analyse researchers models to assess them for publication. This is generally quantitative models but that does not change the nature of the rights to privacy. There are methods of aggregating data or anonimizing data even around qualitative methods. My experience attending AoIR and related conferences is some new researchers are not very informed of privacy rights of respondents to the same degree as I am. I have legal obligations to the resondents of our surveys. Example what may be publicly on the Internet may also seem to useful for research purposes. When in fact the author of the data may have felt a sense of privacy when writing it. All subjects of research should be informed that they are subjects of research. A researcher finding an interesting medical self help Facebook group or vulnerable population may find a lot to theorize about. It may seem a great research topic. But if the membership of these people in this group were disclosed perhaps someone's life or rental contract or job may be at risk. This needs to be thought about. Again look for topics on anonymizing data, disclosure analysis, synthetic data, masking data, suppression of data, confidentiality protection. On Sun., Apr. 23, 2023, 9:58 a.m. Marcelo Santos via Air-L, < air-l@listserv.aoir.org> wrote:
Good point Floriana, but again: are a private company's terms of service a legal boundary, sufficient to prevent researchers to deal with issues of public interest. If you ask me, no. But I know many IRBs follow the terms of service, which is a very lazy shortcut, IMHO, to solve complex ethical issues.
I would very much like to know what others think, though.
Best Marcelo
Em sáb., 22 de abr. de 2023 às 14:11, Grasso, Floriana < floriana@liverpool.ac.uk> escreveu:
Hi Rose
I suspect you will not be able to collect *any* data from Facebook, no matter what ethic approval you have, as their terms of service <https://www.facebook.com/legal/terms> say (in 3.2.3):
You may not access or collect data from our Products using automated means (without our prior permission) or attempt to access data that you do not have permission to access. We also reserve all of our rights against text and data mining.
which basically says you cannot scrape content, but I would be interested to know if the legal team at your institution thinks otherwise?
Floriana
------------------------------ *From:* Air-L <air-l-bounces@listserv.aoir.org> on behalf of Marcelo Santos via Air-L <air-l@listserv.aoir.org> *Sent:* 22 April 2023 18:50 *To:* Butler, Rose <R.Butler@shu.ac.uk> *Cc:* air-l@listserv.aoir.org <air-l@listserv.aoir.org> *Subject:* Re: [Air-L] Downloading Twitter feeds
Hi Rose, I am very interested in how you solve that puzzle.
I have been working lately with WhatsApp groups that are publicly available and anonymizing and performing aggregate analysis have gotten me through ethics approval. If you are going to perform more qualitative research it might be very advisable that users are aware of it and consent, which might prove very complicated.
Let me know how this moves forward.
Best wishes Marcelo
Em sex., 21 de abr. de 2023 às 09:32, Butler, Rose via Air-L < air-l@listserv.aoir.org> escreveu:
Hello I have an ethical question about scraping data from FB (that is already in the public domain)
I am preparing a project for ethical clearance. The research focuses on two community FB groups of 3000 and 7000 members respectively and of which I am a member of both. As part of the research methods I will automate the collection of data for analysis (partly on spreadsheets), all of this would be possible manually if I had the time – automation allows me to collect a lot of data quickly – it is not extending access.
If this data is anonymised and not published would that be enough to protect the privacy of members within the group? I wondered if anyone had experience of doing this and what additional safeguards they needed to implement for ethical clearance. The problem here is that a closed group is semi-private, individuals may feel like it is private even if it spans the whole community (this time a small town).
Thanks
From: Air-L <air-l-bounces@listserv.aoir.org> on behalf of Stuart Shulman via Air-L <air-l@listserv.aoir.org> Date: Tuesday, 18 April 2023 at 12:10 To: M.E.Sciubba <mesciubba@gmail.com> Cc: air-l@listserv.aoir.org <air-l@listserv.aoir.org> Subject: Re: [Air-L] Downloading Twitter feeds CAUTION: This message was sent from outside the University, purportedly from air-l-bounces@listserv.aoir.org .
Please check the sender is legitimate before responding. Please treat any links or attachments with care - do not follow or open them unless you are sure they are genuine.
The answer is complex, uncertain, and shifting. It is more than just a technical issue of "Where can I get data now?" There are fundamental legal, jurisdictional, market, equity, and political questions unresolved. Resolution seems unlikely in the short term. As a political scientist interested in election interference, I do not anticipate a positive resolution before the 2024 U.S. election. I would love to be wrong on that prediction. I have heard there are a few firms that retain agreements to market real time and historical Twitter data. You might ask your librarian about access to Crimson Hexagon, Meltwater, or other similar services with long term agreements Mr. Musk has not broken, yet. Nobody on the AIR-L list has confirmed whether the academic API is shut down. Is it? I don't have those credentials to check.
As I have written previously, there are massive troves of Tweets in every computer science department and many social science departments. Any one of the raw JSON Tweet archives could be loaded into a free account on DiscoverText and then shared for teaching or research. Talk to the data folks on your campus and across your discipline about what is extant. Do not look for spreadsheets; find the raw JSON. Spreadsheets are significantly degraded historical objects that cannot be considered accurate representations of Tweets. Tweets live in the Twitter display and die in spreadsheets. I remain befuddled by the Python/R nexus. Those spreadsheets of Tweets simply are not Twitter. Any qualitative researcher knows what I mean. Meaning is produced in the interaction of the display elements. Meaning is destroyed, diluted, and bent out of shape in spreadsheets.
The loss of real time access to new data is a problem and dangerous for democratic systems. This does not change the fact that so much data has been gathered and stored, like so many old newspapers. For example, here is a list of the Top 50 datasets in my DiscoverText account that any academic on this list or anywhere in the world can access via the "peer network" that enables collaboration, annotation, measurement of inter-rater reliability, and the creation of gold standard training sets for machine-learning via a graphical user interface that features the Twitter display.
https://docs.google.com/spreadsheets/d/1j-Y40WmwFIX8pidfAcxgAt5JB3A3UyB2psh_...
"The Case Against Spreadsheets as a Primary Twitter Research Tool" https://vimeo.com/526218014 Question: When is a Tweet not a Tweet? Answer: When it appears in a spreadsheet, or is deleted, or the account has been suspended.
On Tue, Apr 18, 2023 at 5:57 AM M.E.Sciubba via Air-L < air-l@listserv.aoir.org> wrote:
Hi Rotem and All,
I am also looking for solutions to this. I hope somebody has an idea
of
how
to keep doing research on Twitter.
Eleonora *--*
Dr. Maria Eleonora Sciubba (she/her) (2022): “Adesso m’incazzo!”: Swearwords as resources for managing negative emotions in interaction < https://mediazioni.unibo.it/article/view/15263> (2021) https://rolsi.net/2021/06/02/guest-blog-em-ca-for-racial-justice/
*TSHD - Grant Design & Writing * *Department of Culture Studies - Senior Researcher* Editor, Internship Organizational Supervisor *Diggit Magazine* <https://www.diggitmagazine.com/> Tilburg University Twitter: @LolaSciubba
*Be green. Keep it on the screen.*
Il giorno mar 18 apr 2023 alle ore 11:15 Rotem Perach via Air-L < air-l@listserv.aoir.org> ha scritto:
Hi All,
Can anyone recommend a way to download the feeds (historical tweets and re-tweets) of Twitter users? Is scraping for example using Data Miner a good idea? With the changes to Twitter's API rules, API-based websites such as Vicinitas.io have stopped working (or will stop soon), and I'm looking for other approaches.
Thank you for your help.
Kind regards,
Rotem
-----------
Dr Rotem Perach (he/him) Senior Research Fellow School of Social Sciences University of Westminster
ORCID<https://orcid.org/0000-0002-8647-4367> | Twitter< https://twitter.com/DrRotemPerach/><; https://twitter.com/DrRotemPerach/%3e> <https://twitter.com/DrRotemPerach/%3e%3E>
The University of Westminster is a charity and a company limited by guarantee. Registration number: 977818 England. Registered Office: 309 Regent Street, London W1B 2HW.
This message and its attachments are private and confidential. If you have received this message in error, please notify the sender and remove it and its attachments from your system. _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/ _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
Thanks, it would not be possible to make users aware without compromising the research. Thanks for your interest Rose From: Marcelo Santos <marcelolbsantos@gmail.com> Date: Saturday, 22 April 2023 at 18:51 To: Butler, Rose <R.Butler@shu.ac.uk> Cc: stuart.shulman@gmail.com <stuart.shulman@gmail.com>, M.E.Sciubba <mesciubba@gmail.com>, air-l@listserv.aoir.org <air-l@listserv.aoir.org> Subject: Re: [Air-L] Downloading Twitter feeds CAUTION: This message was sent from outside the University, purportedly from marcelolbsantos@gmail.com . Please check the sender is legitimate before responding. Please treat any links or attachments with care - do not follow or open them unless you are sure they are genuine. Hi Rose, I am very interested in how you solve that puzzle. I have been working lately with WhatsApp groups that are publicly available and anonymizing and performing aggregate analysis have gotten me through ethics approval. If you are going to perform more qualitative research it might be very advisable that users are aware of it and consent, which might prove very complicated. Let me know how this moves forward. Best wishes Marcelo Em sex., 21 de abr. de 2023 às 09:32, Butler, Rose via Air-L <air-l@listserv.aoir.org<mailto:air-l@listserv.aoir.org>> escreveu: Hello I have an ethical question about scraping data from FB (that is already in the public domain) I am preparing a project for ethical clearance. The research focuses on two community FB groups of 3000 and 7000 members respectively and of which I am a member of both. As part of the research methods I will automate the collection of data for analysis (partly on spreadsheets), all of this would be possible manually if I had the time – automation allows me to collect a lot of data quickly – it is not extending access. If this data is anonymised and not published would that be enough to protect the privacy of members within the group? I wondered if anyone had experience of doing this and what additional safeguards they needed to implement for ethical clearance. The problem here is that a closed group is semi-private, individuals may feel like it is private even if it spans the whole community (this time a small town). Thanks From: Air-L <air-l-bounces@listserv.aoir.org<mailto:air-l-bounces@listserv.aoir.org>> on behalf of Stuart Shulman via Air-L <air-l@listserv.aoir.org<mailto:air-l@listserv.aoir.org>> Date: Tuesday, 18 April 2023 at 12:10 To: M.E.Sciubba <mesciubba@gmail.com<mailto:mesciubba@gmail.com>> Cc: air-l@listserv.aoir.org<mailto:air-l@listserv.aoir.org> <air-l@listserv.aoir.org<mailto:air-l@listserv.aoir.org>> Subject: Re: [Air-L] Downloading Twitter feeds CAUTION: This message was sent from outside the University, purportedly from air-l-bounces@listserv.aoir.org<mailto:air-l-bounces@listserv.aoir.org> . Please check the sender is legitimate before responding. Please treat any links or attachments with care - do not follow or open them unless you are sure they are genuine. The answer is complex, uncertain, and shifting. It is more than just a technical issue of "Where can I get data now?" There are fundamental legal, jurisdictional, market, equity, and political questions unresolved. Resolution seems unlikely in the short term. As a political scientist interested in election interference, I do not anticipate a positive resolution before the 2024 U.S. election. I would love to be wrong on that prediction. I have heard there are a few firms that retain agreements to market real time and historical Twitter data. You might ask your librarian about access to Crimson Hexagon, Meltwater, or other similar services with long term agreements Mr. Musk has not broken, yet. Nobody on the AIR-L list has confirmed whether the academic API is shut down. Is it? I don't have those credentials to check. As I have written previously, there are massive troves of Tweets in every computer science department and many social science departments. Any one of the raw JSON Tweet archives could be loaded into a free account on DiscoverText and then shared for teaching or research. Talk to the data folks on your campus and across your discipline about what is extant. Do not look for spreadsheets; find the raw JSON. Spreadsheets are significantly degraded historical objects that cannot be considered accurate representations of Tweets. Tweets live in the Twitter display and die in spreadsheets. I remain befuddled by the Python/R nexus. Those spreadsheets of Tweets simply are not Twitter. Any qualitative researcher knows what I mean. Meaning is produced in the interaction of the display elements. Meaning is destroyed, diluted, and bent out of shape in spreadsheets. The loss of real time access to new data is a problem and dangerous for democratic systems. This does not change the fact that so much data has been gathered and stored, like so many old newspapers. For example, here is a list of the Top 50 datasets in my DiscoverText account that any academic on this list or anywhere in the world can access via the "peer network" that enables collaboration, annotation, measurement of inter-rater reliability, and the creation of gold standard training sets for machine-learning via a graphical user interface that features the Twitter display. https://docs.google.com/spreadsheets/d/1j-Y40WmwFIX8pidfAcxgAt5JB3A3UyB2psh_... "The Case Against Spreadsheets as a Primary Twitter Research Tool" https://vimeo.com/526218014 Question: When is a Tweet not a Tweet? Answer: When it appears in a spreadsheet, or is deleted, or the account has been suspended. On Tue, Apr 18, 2023 at 5:57 AM M.E.Sciubba via Air-L < air-l@listserv.aoir.org<mailto:air-l@listserv.aoir.org>> wrote:
Hi Rotem and All,
I am also looking for solutions to this. I hope somebody has an idea of how to keep doing research on Twitter.
Eleonora *--*
Dr. Maria Eleonora Sciubba (she/her) (2022): “Adesso m’incazzo!”: Swearwords as resources for managing negative emotions in interaction <https://mediazioni.unibo.it/article/view/15263> (2021) https://rolsi.net/2021/06/02/guest-blog-em-ca-for-racial-justice/
*TSHD - Grant Design & Writing * *Department of Culture Studies - Senior Researcher* Editor, Internship Organizational Supervisor *Diggit Magazine* <https://www.diggitmagazine.com/> Tilburg University Twitter: @LolaSciubba
*Be green. Keep it on the screen.*
Il giorno mar 18 apr 2023 alle ore 11:15 Rotem Perach via Air-L < air-l@listserv.aoir.org<mailto:air-l@listserv.aoir.org>> ha scritto:
Hi All,
Can anyone recommend a way to download the feeds (historical tweets and re-tweets) of Twitter users? Is scraping for example using Data Miner a good idea? With the changes to Twitter's API rules, API-based websites such as Vicinitas.io have stopped working (or will stop soon), and I'm looking for other approaches.
Thank you for your help.
Kind regards,
Rotem
-----------
Dr Rotem Perach (he/him) Senior Research Fellow School of Social Sciences University of Westminster
ORCID<https://orcid.org/0000-0002-8647-4367> | Twitter< https://twitter.com/DrRotemPerach/><https://twitter.com/DrRotemPerach/%3e>
The University of Westminster is a charity and a company limited by guarantee. Registration number: 977818 England. Registered Office: 309 Regent Street, London W1B 2HW.
This message and its attachments are private and confidential. If you have received this message in error, please notify the sender and remove it and its attachments from your system. _______________________________________________ The Air-L@listserv.aoir.org<mailto:Air-L@listserv.aoir.org> mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org<mailto:Air-L@listserv.aoir.org> mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org<mailto:Air-L@listserv.aoir.org> mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/ _______________________________________________ The Air-L@listserv.aoir.org<mailto:Air-L@listserv.aoir.org> mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/
participants (10)
-
Butler, Rose -
Grasso, Floriana -
James Danowski -
M.E.Sciubba -
Marc Gallofré Ocaña -
Marcelo Santos -
Peter Timusk -
Rotem Perach -
Shulman, Stu -
Stuart Shulman