cases in which data-driven decision-making went awry
I apologize in advance that this is an imperfectly phrased query. In short, I'm looking for literature about terrible data governance and related issues. Basically, what happens when there are errors in automated data systems, how those errors might have occurred, and what institutions do (or don't) when they discover those errors. Ideally, cases would describe the technical bits as well as the human choices made. Another way to say it is that my colleagues and I are looking for investigations into data-driven decision-making gone awry. I've read Kathy O'Neill's Weapons of Math Destruction, which was excellent, and now I'm looking for more specific cases, if they exist. Thanks, Sheryl Grant
Hi Sheryl, My colleague Joanna Redden at the Data Justice Lab has put together a Data Harm Record that might be of use: https://datajusticelab.org/data-harm-record/ Best, Lina -- Dr Lina Dencik Senior Lecturer/Director MA Journalism, Media and Communication Co-Founder Data Justice Lab PI Data Justice: Understanding datafication in relation to social justice (DATAJUSTICE) ERC Starting Grant 2018-2023 School of Journalism, Media and Culture, Cardiff University Bute Building, King Edward VII Avenue, Cardiff CF10 3NB Email: DencikL@cardiff.ac.uk, Tel: +44 (0)29 208 75461 Twitter: @LinaDencik Fellow, Center for Media, Data and Society, Central European University ________________________________ From: Air-L <air-l-bounces@listserv.aoir.org> on behalf of Sheryl Grant <sherylgrant@gmail.com> Sent: 14 May 2018 22:28:51 To: air-l@listserv.aoir.org Subject: [Air-L] cases in which data-driven decision-making went awry I apologize in advance that this is an imperfectly phrased query. In short, I'm looking for literature about terrible data governance and related issues. Basically, what happens when there are errors in automated data systems, how those errors might have occurred, and what institutions do (or don't) when they discover those errors. Ideally, cases would describe the technical bits as well as the human choices made. Another way to say it is that my colleagues and I are looking for investigations into data-driven decision-making gone awry. I've read Kathy O'Neill's Weapons of Math Destruction, which was excellent, and now I'm looking for more specific cases, if they exist. Thanks, Sheryl Grant _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/
I'm not sure if this is what you mean, but there is a great paper on unexpected and sometimes disturbing behavior from machine-learning algorithms here: https://arxiv.org/pdf/1803.03453.pdf and a more popular press write-up of their study here: http://aiweirdness.com/post/172894792687/when-algorithms-surprise-us Safiya Noble's book *Algorithms of Oppression* is fantastic on the more "here is the harm that results from certain kinds of data-driven decision making" aspect of your question. You might also look at Eden Medina's book *Cybernetic Revolutionaries: Technology and Politics in Allende's Chile* on how and why an early attempt to run a country based on data and algorithmic decision went caput. On Mon, May 14, 2018 at 4:28 PM, Sheryl Grant <sherylgrant@gmail.com> wrote:
I apologize in advance that this is an imperfectly phrased query.
In short, I'm looking for literature about terrible data governance and related issues. Basically, what happens when there are errors in automated data systems, how those errors might have occurred, and what institutions do (or don't) when they discover those errors. Ideally, cases would describe the technical bits as well as the human choices made.
Another way to say it is that my colleagues and I are looking for investigations into data-driven decision-making gone awry.
I've read Kathy O'Neill's Weapons of Math Destruction, which was excellent, and now I'm looking for more specific cases, if they exist.
Thanks,
Sheryl Grant _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/ listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
Perhaps of interest: I make reference to a few “bad data cases” in my short presentation on "why we should care about bad data” available at : http://thegovlab.org/why-we-should-care-about-bad-data/ There are also a few “challenging data case studies” at http://odimpact.org/ e.g. http://odimpact.org/case-united-states-eightmaps.html On May 14, 2018, at 5:28 PM, Sheryl Grant <sherylgrant@gmail.com<mailto:sherylgrant@gmail.com>> wrote: I apologize in advance that this is an imperfectly phrased query. In short, I'm looking for literature about terrible data governance and related issues. Basically, what happens when there are errors in automated data systems, how those errors might have occurred, and what institutions do (or don't) when they discover those errors. Ideally, cases would describe the technical bits as well as the human choices made. Another way to say it is that my colleagues and I are looking for investigations into data-driven decision-making gone awry. I've read Kathy O'Neill's Weapons of Math Destruction, which was excellent, and now I'm looking for more specific cases, if they exist. Thanks, Sheryl Grant _______________________________________________ The Air-L@listserv.aoir.org<mailto:Air-L@listserv.aoir.org> mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/
Highly recommend "Automating Inequality" by Virginia Eubanks. https://us.macmillan.com/books/9781250074317 Cheers! On 5/14/18, 6:10 PM, "Air-L on behalf of Stefaan Verhulst" <air-l-bounces@listserv.aoir.org on behalf of SVerhulst@markle.org> wrote: Perhaps of interest: I make reference to a few “bad data cases” in my short presentation on "why we should care about bad data” available at : http://thegovlab.org/why-we-should-care-about-bad-data/ There are also a few “challenging data case studies” at http://odimpact.org/ e.g. http://odimpact.org/case-united-states-eightmaps.html On May 14, 2018, at 5:28 PM, Sheryl Grant <sherylgrant@gmail.com<mailto:sherylgrant@gmail.com>> wrote: I apologize in advance that this is an imperfectly phrased query. In short, I'm looking for literature about terrible data governance and related issues. Basically, what happens when there are errors in automated data systems, how those errors might have occurred, and what institutions do (or don't) when they discover those errors. Ideally, cases would describe the technical bits as well as the human choices made. Another way to say it is that my colleagues and I are looking for investigations into data-driven decision-making gone awry. I've read Kathy O'Neill's Weapons of Math Destruction, which was excellent, and now I'm looking for more specific cases, if they exist. Thanks, Sheryl Grant _______________________________________________ The Air-L@listserv.aoir.org<mailto:Air-L@listserv.aoir.org> mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/ _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/
The Australian Government's social services stuff-up 'robo-debt' is a good example: https://www.smh.com.au/politics/federal/robo-debt-an-unlawful-exercise-forme... On Tue, May 15, 2018 at 7:28 AM, Sheryl Grant <sherylgrant@gmail.com> wrote:
I apologize in advance that this is an imperfectly phrased query.
In short, I'm looking for literature about terrible data governance and related issues. Basically, what happens when there are errors in automated data systems, how those errors might have occurred, and what institutions do (or don't) when they discover those errors. Ideally, cases would describe the technical bits as well as the human choices made.
Another way to say it is that my colleagues and I are looking for investigations into data-driven decision-making gone awry.
I've read Kathy O'Neill's Weapons of Math Destruction, which was excellent, and now I'm looking for more specific cases, if they exist.
Thanks,
Sheryl Grant _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/ listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
Not specifically on data-driven errors, per se, but more generally about systems that either are dysfunctional or create dysfunctionalities, partly based on hwo they are designed, but also on who benefits or suffers from the problems, and what factors reinforce, propagate, and embed those dysfunctions: Rice, R. E. & Cooper, S. (2010). *Organizations and unusual routines: A systems analysis of dysfunctional feedback processes*. Cambridge, UK: Cambridge University Press. On Mon, May 14, 2018 at 3:17 PM, Deborah Lupton <deborah.lupton@gmail.com> wrote:
The Australian Government's social services stuff-up 'robo-debt' is a good example:
https://www.smh.com.au/politics/federal/robo-debt-an- unlawful-exercise-former-appeals-tribunal-member-says-20180405-p4z7x9.html
On Tue, May 15, 2018 at 7:28 AM, Sheryl Grant <sherylgrant@gmail.com> wrote:
I apologize in advance that this is an imperfectly phrased query.
In short, I'm looking for literature about terrible data governance and related issues. Basically, what happens when there are errors in automated data systems, how those errors might have occurred, and what institutions do (or don't) when they discover those errors. Ideally, cases would describe the technical bits as well as the human choices made.
Another way to say it is that my colleagues and I are looking for investigations into data-driven decision-making gone awry.
I've read Kathy O'Neill's Weapons of Math Destruction, which was excellent, and now I'm looking for more specific cases, if they exist.
Thanks,
Sheryl Grant _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/ listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/ listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
-- Ronald E. Rice Arthur N. Rupe Professor in the Social Effects of Mass Communication ICA President, 2006-2007 Department of Communication 4005 Social Sciences & Media Studies Bldg University of California Santa Barbara, CA 93106-4020 805-893-8696
Re RoboDebt See my conference paper Henman, Paul. (2017, September 4). The computer says 'DEBT': Towards a critical sociology of algorithms and algorithmic governance. Data for Policy Conference 2017. Zenodo. 10.5281/zenodo.884116 at https://zenodo.org/record/884117#.WcTlEsh97IU I'm happy to discuss technical bits further if required. Paul Henman Associate Professor of Digital Sociology and Social Policy Director, Bachelor of Social Science School of Social Science University of Queensland QLD 4072 T: +61 7 3365 2765 | E: P.Henman@uq.edu.au | W: www.digitalsocialpolicy.com UQ ALLY - Supporting the diversity of sexuality and gender identity at UQ. CRICOS Provider Number: 00025B -----Original Message----- From: Air-L <air-l-bounces@listserv.aoir.org> On Behalf Of Deborah Lupton Sent: Tuesday, 15 May 2018 8:18 AM To: Sheryl Grant <sherylgrant@gmail.com> Cc: <air-l@listserv.aoir.org> <air-l@listserv.aoir.org> Subject: Re: [Air-L] cases in which data-driven decision-making went awry The Australian Government's social services stuff-up 'robo-debt' is a good example: https://www.smh.com.au/politics/federal/robo-debt-an-unlawful-exercise-forme... On Tue, May 15, 2018 at 7:28 AM, Sheryl Grant <sherylgrant@gmail.com> wrote:
I apologize in advance that this is an imperfectly phrased query.
In short, I'm looking for literature about terrible data governance and related issues. Basically, what happens when there are errors in automated data systems, how those errors might have occurred, and what institutions do (or don't) when they discover those errors. Ideally, cases would describe the technical bits as well as the human choices made.
Another way to say it is that my colleagues and I are looking for investigations into data-driven decision-making gone awry.
I've read Kathy O'Neill's Weapons of Math Destruction, which was excellent, and now I'm looking for more specific cases, if they exist.
Thanks,
Sheryl Grant _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/ listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/
Thanks for the replies, all. I'm working my way through them and having a heyday. This is what I am trying to drill into: The accuracy of big data applications will be affected by the accuracy of small data. And all the things that can go wrong in the data lifecycle that thwart accuracy. Many thanks for all your suggestions so I could articulate this better. Sheryl On Mon, May 14, 2018 at 7:10 PM, Paul Henman <p.henman@uq.edu.au> wrote:
Re RoboDebt
See my conference paper Henman, Paul. (2017, September 4). The computer says 'DEBT': Towards a critical sociology of algorithms and algorithmic governance. Data for Policy Conference 2017. Zenodo. 10.5281/zenodo.884116 at https://zenodo.org/record/884117#.WcTlEsh97IU
I'm happy to discuss technical bits further if required.
Paul Henman Associate Professor of Digital Sociology and Social Policy Director, Bachelor of Social Science School of Social Science University of Queensland QLD 4072 T: +61 7 3365 2765 | E: P.Henman@uq.edu.au | W: www.digitalsocialpolicy.com
UQ ALLY - Supporting the diversity of sexuality and gender identity at UQ. CRICOS Provider Number: 00025B -----Original Message----- From: Air-L <air-l-bounces@listserv.aoir.org> On Behalf Of Deborah Lupton Sent: Tuesday, 15 May 2018 8:18 AM To: Sheryl Grant <sherylgrant@gmail.com> Cc: <air-l@listserv.aoir.org> <air-l@listserv.aoir.org> Subject: Re: [Air-L] cases in which data-driven decision-making went awry
The Australian Government's social services stuff-up 'robo-debt' is a good example:
https://www.smh.com.au/politics/federal/robo-debt-an- unlawful-exercise-former-appeals-tribunal-member-says-20180405-p4z7x9.html
On Tue, May 15, 2018 at 7:28 AM, Sheryl Grant <sherylgrant@gmail.com> wrote:
I apologize in advance that this is an imperfectly phrased query.
In short, I'm looking for literature about terrible data governance and related issues. Basically, what happens when there are errors in automated data systems, how those errors might have occurred, and what institutions do (or don't) when they discover those errors. Ideally, cases would describe the technical bits as well as the human choices made.
Another way to say it is that my colleagues and I are looking for investigations into data-driven decision-making gone awry.
I've read Kathy O'Neill's Weapons of Math Destruction, which was excellent, and now I'm looking for more specific cases, if they exist.
Thanks,
Sheryl Grant _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/ listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
I have written the section on risks of psychosocial and physical violence in digitalized workplaces for the International Labour Organization (of the UN) discussions at the forthcoming International Labour Conference in Geneva end of this month toward a new labour convention. One of the points I make (frequently) is that decisionmaking based on algorithm in the workplace brings up extensive ethical questions and can even lead to risky environments for workers. This includes e.g. AI in people analytics for recruitment, appraisal, talent spotting and use of algorithms in other human resource decisions in gig work. I can send the final draft if you are interested. An earlier draft is linked here at the launch of the report http://www.ilo.org/actrav/events/WCMS_616826/lang--en/index.htm Dr Phoebe V Moore Personal email pvm.doc@gmail.com Work email pm358@le.ac.uk <p.moore@mdx.ac.uk> Twitter @phoebemoore Biolog http://phoebevmoore.wordpress.com/ On 15 May 2018 at 19:49, Sheryl Grant <sherylgrant@gmail.com> wrote:
Thanks for the replies, all. I'm working my way through them and having a heyday.
This is what I am trying to drill into: The accuracy of big data applications will be affected by the accuracy of small data.
And all the things that can go wrong in the data lifecycle that thwart accuracy.
Many thanks for all your suggestions so I could articulate this better.
Sheryl
On Mon, May 14, 2018 at 7:10 PM, Paul Henman <p.henman@uq.edu.au> wrote:
Re RoboDebt
See my conference paper Henman, Paul. (2017, September 4). The computer says 'DEBT': Towards a critical sociology of algorithms and algorithmic governance. Data for Policy Conference 2017. Zenodo. 10.5281/zenodo.884116 at https://zenodo.org/record/884117#.WcTlEsh97IU
I'm happy to discuss technical bits further if required.
Paul Henman Associate Professor of Digital Sociology and Social Policy Director, Bachelor of Social Science School of Social Science University of Queensland QLD 4072 T: +61 7 3365 2765 | E: P.Henman@uq.edu.au | W: www.digitalsocialpolicy.com
UQ ALLY - Supporting the diversity of sexuality and gender identity at UQ. CRICOS Provider Number: 00025B -----Original Message----- From: Air-L <air-l-bounces@listserv.aoir.org> On Behalf Of Deborah Lupton Sent: Tuesday, 15 May 2018 8:18 AM To: Sheryl Grant <sherylgrant@gmail.com> Cc: <air-l@listserv.aoir.org> <air-l@listserv.aoir.org> Subject: Re: [Air-L] cases in which data-driven decision-making went awry
The Australian Government's social services stuff-up 'robo-debt' is a good example:
https://www.smh.com.au/politics/federal/robo-debt-an- unlawful-exercise-former-appeals-tribunal-member-says- 20180405-p4z7x9.html
On Tue, May 15, 2018 at 7:28 AM, Sheryl Grant <sherylgrant@gmail.com> wrote:
I apologize in advance that this is an imperfectly phrased query.
In short, I'm looking for literature about terrible data governance and related issues. Basically, what happens when there are errors in automated data systems, how those errors might have occurred, and what institutions do (or don't) when they discover those errors. Ideally, cases would describe the technical bits as well as the human choices made.
Another way to say it is that my colleagues and I are looking for investigations into data-driven decision-making gone awry.
I've read Kathy O'Neill's Weapons of Math Destruction, which was excellent, and now I'm looking for more specific cases, if they exist.
Thanks,
Sheryl Grant _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/ listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/ listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
participants (9)
-
Bridges, Lauren -
Deborah Lupton -
Lina Dencik -
Paul Henman -
Phoebe Moore -
Ronald Rice -
Samantha Close -
Sheryl Grant -
Stefaan Verhulst