emotion detection machine?
Dear colleagues, One of our students is wanting to analyze emotional content in in the comment fields of a major newspaper vis-a-vis specific hot-button issues. She has a good tool (I think) for scrapping the data - but she is stymied over the choice of an emotion analysis tool. She has looked at Senpy (http://senpy.gsi.upm.es/#test) and Twinword <https://www.twinword.com/api/emotion-analysis.php> - the latter seems the most accurate, but it is also expensive. She has recently discovered DepecheMood emotion lexicons (Staiano, J., & Guerini, M. (2014). Depechemood: a lexicon for emotion analysis from crowd-annotated news. arXiv preprint arXiv:1405.1605.) - but this suffers from a lack of clarity in terms of explaining its emotional categories: awe, indifference, sad, amusement , annoyance, joy, fear and anger. For my part, I am entirely clueless. Any suggestions that she might pursue would be greatly appreciated. best, - charles ess -- Professor in Media Studies Department of Media and Communication University of Oslo <http://www.hf.uio.no/imk/english/people/aca/charlees/index.html> Postboks 1093 Blindern 0317 Oslo, Norway c.m.ess@media.uio.no
Dear Charles, I'm not an expert, but I think she should be talking with linguists - and I think that what she's looking for is typically called sentiment analysis, not emotion analysis. There are probably tools for social media marketing that might be more readily accessible, but probably less scientifically transparent. https://en.wikipedia.org/wiki/Sentiment_analysis Jill Air-L på vegne av Charles M. Ess <air-l-bounces@listserv.aoir.org på vegne av c.m.ess@media.uio.no> skrev følgende den 05.09.2019, 12:53: Dear colleagues, One of our students is wanting to analyze emotional content in in the comment fields of a major newspaper vis-a-vis specific hot-button issues. She has a good tool (I think) for scrapping the data - but she is stymied over the choice of an emotion analysis tool. She has looked at Senpy (http://senpy.gsi.upm.es/#test) and Twinword <https://www.twinword.com/api/emotion-analysis.php> - the latter seems the most accurate, but it is also expensive. She has recently discovered DepecheMood emotion lexicons (Staiano, J., & Guerini, M. (2014). Depechemood: a lexicon for emotion analysis from crowd-annotated news. arXiv preprint arXiv:1405.1605.) - but this suffers from a lack of clarity in terms of explaining its emotional categories: awe, indifference, sad, amusement , annoyance, joy, fear and anger. For my part, I am entirely clueless. Any suggestions that she might pursue would be greatly appreciated. best, - charles ess -- Professor in Media Studies Department of Media and Communication University of Oslo <http://www.hf.uio.no/imk/english/people/aca/charlees/index.html> Postboks 1093 Blindern 0317 Oslo, Norway c.m.ess@media.uio.no _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/
Some tools for this work are born in academic research labs instead of corporate meeting rooms. For more than a decade we have offered free and commercial web-based tools for content labeling and annotator measurement in an academic research setting. To date there are 400+ interdisciplinary and multilingual citations: https://discovertext.com/2018/03/31/scholarly-citations-of-the-coding-analys... https://discovertext.com/publications/ Free demos: https://calendly.com/discovertext ~Stu *Stu Shulman* <https://twitter.com/StuartWShulman>U.S. Soccer Federation C-Licensed Coach Valeo FC & Capacidad <http://capacidadprograms.org/?page_id=13> Volunteer Coach *Is your player ready to give back to the game? *Contact Coach Stu about fall 2019 volunteer efforts. [image: Capacidad] <http://capacidadprograms.org/?page_id=13> On Thu, Sep 5, 2019 at 7:25 AM Jill Walker Rettberg < Jill.Walker.Rettberg@uib.no> wrote:
Dear Charles,
I'm not an expert, but I think she should be talking with linguists - and I think that what she's looking for is typically called sentiment analysis, not emotion analysis. There are probably tools for social media marketing that might be more readily accessible, but probably less scientifically transparent. https://en.wikipedia.org/wiki/Sentiment_analysis
Jill
Air-L på vegne av Charles M. Ess <air-l-bounces@listserv.aoir.org på vegne av c.m.ess@media.uio.no> skrev følgende den 05.09.2019, 12:53:
Dear colleagues,
One of our students is wanting to analyze emotional content in in the comment fields of a major newspaper vis-a-vis specific hot-button issues.
She has a good tool (I think) for scrapping the data - but she is stymied over the choice of an emotion analysis tool. She has looked at Senpy (http://senpy.gsi.upm.es/#test) and Twinword <https://www.twinword.com/api/emotion-analysis.php> - the latter seems the most accurate, but it is also expensive. She has recently discovered DepecheMood emotion lexicons (Staiano, J., & Guerini, M. (2014). Depechemood: a lexicon for emotion analysis from crowd-annotated news. arXiv preprint arXiv:1405.1605.) - but this suffers from a lack of clarity in terms of explaining its emotional categories: awe, indifference, sad, amusement , annoyance, joy, fear and anger.
For my part, I am entirely clueless. Any suggestions that she might pursue would be greatly appreciated.
best, - charles ess -- Professor in Media Studies Department of Media and Communication University of Oslo <http://www.hf.uio.no/imk/english/people/aca/charlees/index.html>
Postboks 1093 Blindern 0317 Oslo, Norway c.m.ess@media.uio.no _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
Dear Charles, There are many social listening tools on the corporate level: crimson hexagon and infegy which are media industry research methods for determining sentiment. What’s most important is making sure that you have a keyword set that is accurately parsing through the social media posts for sentiment. What’s always difficult is having a lexicon for local terms and also have a method for determining ambiguous posts where the person may mean something different then the keyword that is recognized. Best Wiebke Wiebke Reile, Ph.D. University of Hawai'i at Mānoa Communication and Information Sciences Adjunct Lecturer Brooklyn College Department of Television and Radio Wiebke.Reile@brooklyn.cuny.edu Online Adjunct Grand Canyon University College of Fine Arts and Production Wiebke.Reile@my.gcu.edu
On Sep 5, 2019, at 7:47 AM, Stuart Shulman <stuart.shulman@gmail.com> wrote:
Some tools for this work are born in academic research labs instead of corporate meeting rooms. For more than a decade we have offered free and commercial web-based tools for content labeling and annotator measurement in an academic research setting. To date there are 400+ interdisciplinary and multilingual citations:
https://discovertext.com/2018/03/31/scholarly-citations-of-the-coding-analys...
https://discovertext.com/publications/
Free demos: https://calendly.com/discovertext
~Stu
*Stu Shulman* <https://twitter.com/StuartWShulman>U.S. Soccer Federation C-Licensed Coach Valeo FC & Capacidad <http://capacidadprograms.org/?page_id=13> Volunteer Coach *Is your player ready to give back to the game? *Contact Coach Stu about fall 2019 volunteer efforts.
[image: Capacidad] <http://capacidadprograms.org/?page_id=13>
On Thu, Sep 5, 2019 at 7:25 AM Jill Walker Rettberg < Jill.Walker.Rettberg@uib.no> wrote:
Dear Charles,
I'm not an expert, but I think she should be talking with linguists - and I think that what she's looking for is typically called sentiment analysis, not emotion analysis. There are probably tools for social media marketing that might be more readily accessible, but probably less scientifically transparent. https://en.wikipedia.org/wiki/Sentiment_analysis
Jill
Air-L på vegne av Charles M. Ess <air-l-bounces@listserv.aoir.org på vegne av c.m.ess@media.uio.no> skrev følgende den 05.09.2019, 12:53:
Dear colleagues,
One of our students is wanting to analyze emotional content in in the comment fields of a major newspaper vis-a-vis specific hot-button issues.
She has a good tool (I think) for scrapping the data - but she is stymied over the choice of an emotion analysis tool. She has looked at Senpy (http://senpy.gsi.upm.es/#test) and Twinword <https://www.twinword.com/api/emotion-analysis.php> - the latter seems the most accurate, but it is also expensive. She has recently discovered DepecheMood emotion lexicons (Staiano, J., & Guerini, M. (2014). Depechemood: a lexicon for emotion analysis from crowd-annotated news. arXiv preprint arXiv:1405.1605.) - but this suffers from a lack of clarity in terms of explaining its emotional categories: awe, indifference, sad, amusement , annoyance, joy, fear and anger.
For my part, I am entirely clueless. Any suggestions that she might pursue would be greatly appreciated.
best, - charles ess -- Professor in Media Studies Department of Media and Communication University of Oslo <http://www.hf.uio.no/imk/english/people/aca/charlees/index.html>
Postboks 1093 Blindern 0317 Oslo, Norway c.m.ess@media.uio.no _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
Dear Charles Meltwater is a commercial tool that lets you analyse sentiment. It allows you to track social media as well as regular media using specific search strings and you can compare different publications and it can also search behind paywalls. It is normally quite pricey, but they have a classroom project that universities can sign up to where students enrolled in a specific course use it for free. Best Alette Alette Schoon (PhD) Senior Lecturer Documentary Filmmaking, Mobile and Internet Studies School of Journalism and Media Studies Rhodes University South Africa https://www.researchgate.net/profile/Alette_Schoon -----Original Message----- From: Air-L <air-l-bounces@listserv.aoir.org> On Behalf Of Charles M. Ess Sent: 05 September 2019 12:52 PM To: air-l <air-l@listserv.aoir.org> Subject: [Air-L] emotion detection machine? Dear colleagues, One of our students is wanting to analyze emotional content in in the comment fields of a major newspaper vis-a-vis specific hot-button issues. She has a good tool (I think) for scrapping the data - but she is stymied over the choice of an emotion analysis tool. She has looked at Senpy (http://senpy.gsi.upm.es/#test) and Twinword <https://www.twinword.com/api/emotion-analysis.php> - the latter seems the most accurate, but it is also expensive. She has recently discovered DepecheMood emotion lexicons (Staiano, J., & Guerini, M. (2014). Depechemood: a lexicon for emotion analysis from crowd-annotated news. arXiv preprint arXiv:1405.1605.) - but this suffers from a lack of clarity in terms of explaining its emotional categories: awe, indifference, sad, amusement , annoyance, joy, fear and anger. For my part, I am entirely clueless. Any suggestions that she might pursue would be greatly appreciated. best, - charles ess -- Professor in Media Studies Department of Media and Communication University of Oslo <http://www.hf.uio.no/imk/english/people/aca/charlees/index.html> Postboks 1093 Blindern 0317 Oslo, Norway c.m.ess@media.uio.no _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/
Charles, Another tool you might want to look into is LIWC http://liwc.wpengine.com/. Jess On Thu, Sep 5, 2019 at 6:52 AM Charles M. Ess <c.m.ess@media.uio.no> wrote:
Dear colleagues,
One of our students is wanting to analyze emotional content in in the comment fields of a major newspaper vis-a-vis specific hot-button issues.
She has a good tool (I think) for scrapping the data - but she is stymied over the choice of an emotion analysis tool. She has looked at Senpy (http://senpy.gsi.upm.es/#test) and Twinword <https://www.twinword.com/api/emotion-analysis.php> - the latter seems the most accurate, but it is also expensive. She has recently discovered DepecheMood emotion lexicons (Staiano, J., & Guerini, M. (2014). Depechemood: a lexicon for emotion analysis from crowd-annotated news. arXiv preprint arXiv:1405.1605.) - but this suffers from a lack of clarity in terms of explaining its emotional categories: awe, indifference, sad, amusement , annoyance, joy, fear and anger.
For my part, I am entirely clueless. Any suggestions that she might pursue would be greatly appreciated.
best, - charles ess -- Professor in Media Studies Department of Media and Communication University of Oslo <http://www.hf.uio.no/imk/english/people/aca/charlees/index.html>
Postboks 1093 Blindern 0317 Oslo, Norway c.m.ess@media.uio.no _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
-- Jessica A. Pater PhD Candidate, Human Centered Computing Georgia Institute of Technology pater@gatech.edu www.jesspater.com
Charles, given that there are myriad sentiment tools out there, ranging from traditional presence/absence dictionary-based through value-based dictionary through statistical and now neural approaches and both commercial and academic systems, the most important methodological consideration is the alignment of the specific tool's performance characteristics with both the medium (in your case a newspaper), the language/grammar (the formality and editorial structure of the paper and how closely that aligns with the tool's training dataset), the era (many tools are only updated periodically and/or were trained on content from a specific period and can have severe mismatches even for MSM), the domain (most tools are not domain-adapted and this can cause severe problems in certain domains, such as when a news source refers to "the democratic party" and just "republicans" with "party" systematically yielding a more positive score for the former), and most importantly, the definition of the specific emotion (ie, there is no universal "anxiety" score). Typically this involves reviewing the validation studies for each potential tool and comparing them along these dimensions, though methodologically the definition of the specific measure is a very important piece that is often missed. With GDELT, we've run 40 common tools totaling a few thousand dimensions over around a billion or so global news articles ( https://blog.gdeltproject.org/?s=gcam), including multilingual versions of some of the tools to test the impact of various translation approaches on emotional recovery, so you can compare dimensions and see, for example, how different "anxiety" dimensions compare for outlets and domains most similar to yours - often the differences in the response curves can be a quite informative signal for some applications as well. All of the scores are open data, so you can get a sense for how they respond. Kalev On Thu, Sep 5, 2019 at 6:52 AM Charles M. Ess <c.m.ess@media.uio.no> wrote:
Dear colleagues,
One of our students is wanting to analyze emotional content in in the comment fields of a major newspaper vis-a-vis specific hot-button issues.
She has a good tool (I think) for scrapping the data - but she is stymied over the choice of an emotion analysis tool. She has looked at Senpy (http://senpy.gsi.upm.es/#test) and Twinword <https://www.twinword.com/api/emotion-analysis.php> - the latter seems the most accurate, but it is also expensive. She has recently discovered DepecheMood emotion lexicons (Staiano, J., & Guerini, M. (2014). Depechemood: a lexicon for emotion analysis from crowd-annotated news. arXiv preprint arXiv:1405.1605.) - but this suffers from a lack of clarity in terms of explaining its emotional categories: awe, indifference, sad, amusement , annoyance, joy, fear and anger.
For my part, I am entirely clueless. Any suggestions that she might pursue would be greatly appreciated.
best, - charles ess -- Professor in Media Studies Department of Media and Communication University of Oslo <http://www.hf.uio.no/imk/english/people/aca/charlees/index.html>
Postboks 1093 Blindern 0317 Oslo, Norway c.m.ess@media.uio.no _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
Dear Charles (and List), I see this as an ethics issue. How reliable are “emotion analysis” tools? How would outcomes from them be used? As you say, there is a lack of clarity in some in terms of “explaining emotional categories.” To me, this signals (along with obvious knowledge about the limitations and problems with algorithms), that there is opportunity here to be very, very, very wrong about people’s opinions, and any algorithmically interpreted “emotional” state. For example, how would one interpret or finesse “frustration,” vs “anger”? The written word is contained within a language. Not all commenters will be native speakers to that language, and not all native speakers have the language tools required (even within their own language) to adequately express themselves, even in the best of times. What makes anyone think an algorithm would do better at this than a human trained in qualitative methods and with cultural and media and language knowledge? There is way too much margin of potential error here for this to be automated, or “useful.” It is much more likely that things will be assumed incorrectly by limited algorithms in the first place. Furthermore, does your student see any problem with this exercise? That their tool analysis might get it very wrong? That the wrong might lead to assumptions or outcomes that are harmful to entities, people, governments? What safeguards are in place for wrong assumptions and outcomes? Kind regards, Sally Sally Applin, Ph.D. .......... Research Fellow HRAF Advanced Research Centres (EU), Canterbury Centre for Social Anthropology and Computing (CSAC) .......... Research Associate Human Relations Area Files (HRAF) Yale University .......... Associate Editor, IEEE Consumer Electronics Magazine Member, IoT Council Executive Board Member: The Edward H. and Rosamond B. Spicer Foundation .......... http://www.posr.org http://www.sally.com I am based in Silicon Valley .......... sally@sally.com | 650.339.5236
On Sep 5, 2019, at 3:52 AM, Charles M. Ess <c.m.ess@media.uio.no> wrote:
Dear colleagues,
One of our students is wanting to analyze emotional content in in the comment fields of a major newspaper vis-a-vis specific hot-button issues.
She has a good tool (I think) for scrapping the data - but she is stymied over the choice of an emotion analysis tool. She has looked at Senpy (http://senpy.gsi.upm.es/#test) and Twinword <https://www.twinword.com/api/emotion-analysis.php> - the latter seems the most accurate, but it is also expensive. She has recently discovered DepecheMood emotion lexicons (Staiano, J., & Guerini, M. (2014). Depechemood: a lexicon for emotion analysis from crowd-annotated news. arXiv preprint arXiv:1405.1605.) - but this suffers from a lack of clarity in terms of explaining its emotional categories: awe, indifference, sad, amusement , annoyance, joy, fear and anger.
For my part, I am entirely clueless. Any suggestions that she might pursue would be greatly appreciated.
best, - charles ess -- Professor in Media Studies Department of Media and Communication University of Oslo <http://www.hf.uio.no/imk/english/people/aca/charlees/index.html>
Postboks 1093 Blindern 0317 Oslo, Norway c.m.ess@media.uio.no _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
Deare Sally and all, fascinating conversation about this topic. I am not an academic, but am professionally always on the lookout for scientific debates around emotional analysis and psychological testing. May I slightly hijack this thread to ask if there is a consensus around context v single word analysis. There is a company called Precire in Germany (where I am currently based) that claims to be able to detect a person's personality based on a 15 minute speech sample about any topic, so independent of context and based on individual words. My question: can you deduce personality from a conversation regardless of the topic (say, you talk about an argument with your neighbour and the test may well infer that you're a stressed / anxious person, whereas you talk about your last holiday, then the test may describe you as relaxed). This is on the cross line between emotion detection and character analysis. My understanding/argument is that these tests are generally not suitable for use in an everyday setting (whereas they may well support diagnoses of psychological problems/help in therapies), and that analysis outside of context should be disregarded. If anyone can direct me to studies discussing this in detail, I'd be very grateful - the literature is vast and therefore it's difficult to know who the authorities are in this field and what keywords to search for. Please feel free to contact me directly as well. Thanks and best, Veronika Quoting "Dr. S.A. Applin" <sally@sally.com>:
Dear Charles (and List),
I see this as an ethics issue.
How reliable are “emotion analysis” tools? How would outcomes from them be used?
As you say, there is a lack of clarity in some in terms of “explaining emotional categories.” To me, this signals (along with obvious knowledge about the limitations and problems with algorithms), that there is opportunity here to be very, very, very wrong about people’s opinions, and any algorithmically interpreted “emotional” state.
For example, how would one interpret or finesse “frustration,” vs “anger”? The written word is contained within a language. Not all commenters will be native speakers to that language, and not all native speakers have the language tools required (even within their own language) to adequately express themselves, even in the best of times. What makes anyone think an algorithm would do better at this than a human trained in qualitative methods and with cultural and media and language knowledge?
There is way too much margin of potential error here for this to be automated, or “useful.” It is much more likely that things will be assumed incorrectly by limited algorithms in the first place.
Furthermore, does your student see any problem with this exercise? That their tool analysis might get it very wrong? That the wrong might lead to assumptions or outcomes that are harmful to entities, people, governments?
What safeguards are in place for wrong assumptions and outcomes?
Kind regards,
Sally
Sally Applin, Ph.D. .......... Research Fellow HRAF Advanced Research Centres (EU), Canterbury Centre for Social Anthropology and Computing (CSAC) .......... Research Associate Human Relations Area Files (HRAF) Yale University .......... Associate Editor, IEEE Consumer Electronics Magazine Member, IoT Council Executive Board Member: The Edward H. and Rosamond B. Spicer Foundation .......... http://www.posr.org http://www.sally.com I am based in Silicon Valley .......... sally@sally.com | 650.339.5236
On Sep 5, 2019, at 3:52 AM, Charles M. Ess <c.m.ess@media.uio.no> wrote:
Dear colleagues,
One of our students is wanting to analyze emotional content in in the comment fields of a major newspaper vis-a-vis specific hot-button issues.
She has a good tool (I think) for scrapping the data - but she is stymied over the choice of an emotion analysis tool. She has looked at Senpy (http://senpy.gsi.upm.es/#test) and Twinword <https://www.twinword.com/api/emotion-analysis.php> - the latter seems the most accurate, but it is also expensive. She has recently discovered DepecheMood emotion lexicons (Staiano, J., & Guerini, M. (2014). Depechemood: a lexicon for emotion analysis from crowd-annotated news. arXiv preprint arXiv:1405.1605.) - but this suffers from a lack of clarity in terms of explaining its emotional categories: awe, indifference, sad, amusement , annoyance, joy, fear and anger.
For my part, I am entirely clueless. Any suggestions that she might pursue would be greatly appreciated.
best, - charles ess -- Professor in Media Studies Department of Media and Communication University of Oslo <http://www.hf.uio.no/imk/english/people/aca/charlees/index.html>
Postboks 1093 Blindern 0317 Oslo, Norway c.m.ess@media.uio.no _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
-- Veronika Thiel Senior Researcher https://algorithmwatch.org/ vt@algorithmwatch.org Tel.: +49 30 99 40 49 002 AW AlgorithmWatch gGmbH Bergstr. 22 10115 Berlin Sitz der Gesellschaft: Oranienstraße 19A, 10999 Berlin Amtsgericht Berlin Charlottenburg HRB 186522 B Geschäftsführung: Matthias Spielkamp
Sally, Machine generated sentiment analysis scores are sometimes abused as a shortcut to avoid certain forms of manual/mental labor in a variety of commercial and academic contexts. Language tools are in this scenario treated as a magic buttons to be deployed against corpora in the name of charts untouched by serious validation. I prefer it when humans are in-the-loop, which itself is recursive (meaning you repeat until there is no room to improve), using tools as filters to generate purposive samples that humans annotate and collectively validate using a systematic process. Sentiment problems range from hard to harder and hardest, where hardest means you cannot do it in a manner that can be validated by any means. There is no easy on this scale of tasks if false positive or negatives could cost a life or some other serious consequence, but to make it easier, requires a process, grossly boiled down below: 1. Collect a relevant and representative corpus of data, 2. Build a SPAM detection classifier to remove non-relevant data (ex., wrong language OR no discernible sentiment), 3. Build a topic classifier and focus on one key topic first (not all topics at once), 4. Solve the Rubik's cube of how many codes and what they really mean (ex., happy/sad OR angry/frustrated/both/neither...), 5. Test the topic-specific annotation scheme with a group of no less than five independent annotators (not just two), 6. Crowd source the task to larger groups when possible, using memo writing to identify boundary cases that kill/modify models, 7. Use iteration to identify elite annotators through recursive validation, memo reviews, and scoring against a gold standard. The goal is to build task- and language-specific machine classifiers using the best possible human experts in the process. The main idea, however, is to keep a critical role for humans. ~Stu On Thu, Sep 5, 2019 at 4:11 PM Dr. S.A. Applin <sally@sally.com> wrote:
Dear Charles (and List),
I see this as an ethics issue.
How reliable are “emotion analysis” tools? How would outcomes from them be used?
As you say, there is a lack of clarity in some in terms of “explaining emotional categories.” To me, this signals (along with obvious knowledge about the limitations and problems with algorithms), that there is opportunity here to be very, very, very wrong about people’s opinions, and any algorithmically interpreted “emotional” state.
For example, how would one interpret or finesse “frustration,” vs “anger”? The written word is contained within a language. Not all commenters will be native speakers to that language, and not all native speakers have the language tools required (even within their own language) to adequately express themselves, even in the best of times. What makes anyone think an algorithm would do better at this than a human trained in qualitative methods and with cultural and media and language knowledge?
There is way too much margin of potential error here for this to be automated, or “useful.” It is much more likely that things will be assumed incorrectly by limited algorithms in the first place.
Furthermore, does your student see any problem with this exercise? That their tool analysis might get it very wrong? That the wrong might lead to assumptions or outcomes that are harmful to entities, people, governments?
What safeguards are in place for wrong assumptions and outcomes?
Kind regards,
Sally
Sally Applin, Ph.D. .......... Research Fellow HRAF Advanced Research Centres (EU), Canterbury Centre for Social Anthropology and Computing (CSAC) .......... Research Associate Human Relations Area Files (HRAF) Yale University .......... Associate Editor, IEEE Consumer Electronics Magazine Member, IoT Council Executive Board Member: The Edward H. and Rosamond B. Spicer Foundation .......... http://www.posr.org http://www.sally.com I am based in Silicon Valley .......... sally@sally.com | 650.339.5236
On Sep 5, 2019, at 3:52 AM, Charles M. Ess <c.m.ess@media.uio.no> wrote:
Dear colleagues,
One of our students is wanting to analyze emotional content in in the comment fields of a major newspaper vis-a-vis specific hot-button issues.
She has a good tool (I think) for scrapping the data - but she is stymied over the choice of an emotion analysis tool. She has looked at Senpy (http://senpy.gsi.upm.es/#test) and Twinword < https://www.twinword.com/api/emotion-analysis.php> - the latter seems the most accurate, but it is also expensive. She has recently discovered DepecheMood emotion lexicons (Staiano, J., & Guerini, M. (2014). Depechemood: a lexicon for emotion analysis from crowd-annotated news. arXiv preprint arXiv:1405.1605.) - but this suffers from a lack of clarity in terms of explaining its emotional categories: awe, indifference, sad, amusement , annoyance, joy, fear and anger.
For my part, I am entirely clueless. Any suggestions that she might pursue would be greatly appreciated.
best, - charles ess -- Professor in Media Studies Department of Media and Communication University of Oslo <http://www.hf.uio.no/imk/english/people/aca/charlees/index.html>
Postboks 1093 Blindern 0317 Oslo, Norway c.m.ess@media.uio.no _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
-- Dr. Stuart W. Shulman Founder and CEO, Texifter Cell: 413-992-8513 LinkedIn: http://www.linkedin.com/in/stuartwshulman
Dear all - a belated but most sincere thanks for all of this! I'll bundle up the thread, forward it to my student, and see what sense we might be able to make of it all. Again, many thanks indeed and all best, - charles On 07/09/2019 13:53, Shulman, Stu wrote:
Sally,
Machine generated sentiment analysis scores are sometimes abused as a shortcut to avoid certain forms of manual/mental labor in a variety of commercial and academic contexts. Language tools are in this scenario treated as a magic buttons to be deployed against corpora in the name of charts untouched by serious validation. I prefer it when humans are in-the-loop, which itself is recursive (meaning you repeat until there is no room to improve), using tools as filters to generate purposive samples that humans annotate and collectively validate using a systematic process.
Sentiment problems range from hard to harder and hardest, where hardest means you cannot do it in a manner that can be validated by any means. There is no easy on this scale of tasks if false positive or negatives could cost a life or some other serious consequence, but to make it easier, requires a process, grossly boiled down below:
1. Collect a relevant and representative corpus of data, 2. Build a SPAM detection classifier to remove non-relevant data (ex., wrong language OR no discernible sentiment), 3. Build a topic classifier and focus on one key topic first (not all topics at once), 4. Solve the Rubik's cube of how many codes and what they really mean (ex., happy/sad OR angry/frustrated/both/neither...), 5. Test the topic-specific annotation scheme with a group of no less than five independent annotators (not just two), 6. Crowd source the task to larger groups when possible, using memo writing to identify boundary cases that kill/modify models, 7. Use iteration to identify elite annotators through recursive validation, memo reviews, and scoring against a gold standard.
The goal is to build task- and language-specific machine classifiers using the best possible human experts in the process. The main idea, however, is to keep a critical role for humans.
~Stu
On Thu, Sep 5, 2019 at 4:11 PM Dr. S.A. Applin <sally@sally.com <mailto:sally@sally.com>> wrote:
Dear Charles (and List),
I see this as an ethics issue.
How reliable are “emotion analysis” tools? How would outcomes from them be used?
As you say, there is a lack of clarity in some in terms of “explaining emotional categories.” To me, this signals (along with obvious knowledge about the limitations and problems with algorithms), that there is opportunity here to be very, very, very wrong about people’s opinions, and any algorithmically interpreted “emotional” state.
For example, how would one interpret or finesse “frustration,” vs “anger”? The written word is contained within a language. Not all commenters will be native speakers to that language, and not all native speakers have the language tools required (even within their own language) to adequately express themselves, even in the best of times. What makes anyone think an algorithm would do better at this than a human trained in qualitative methods and with cultural and media and language knowledge?
There is way too much margin of potential error here for this to be automated, or “useful.” It is much more likely that things will be assumed incorrectly by limited algorithms in the first place.
Furthermore, does your student see any problem with this exercise? That their tool analysis might get it very wrong? That the wrong might lead to assumptions or outcomes that are harmful to entities, people, governments?
What safeguards are in place for wrong assumptions and outcomes?
Kind regards,
Sally
Sally Applin, Ph.D. .......... Research Fellow HRAF Advanced Research Centres (EU), Canterbury Centre for Social Anthropology and Computing (CSAC) .......... Research Associate Human Relations Area Files (HRAF) Yale University .......... Associate Editor, IEEE Consumer Electronics Magazine Member, IoT Council Executive Board Member: The Edward H. and Rosamond B. Spicer Foundation .......... http://www.posr.org http://www.sally.com I am based in Silicon Valley .......... sally@sally.com <mailto:sally@sally.com> | 650.339.5236
> On Sep 5, 2019, at 3:52 AM, Charles M. Ess <c.m.ess@media.uio.no <mailto:c.m.ess@media.uio.no>> wrote: > > Dear colleagues, > > One of our students is wanting to analyze emotional content in in the comment fields of a major newspaper vis-a-vis specific hot-button issues. > > She has a good tool (I think) for scrapping the data - but she is stymied over the choice of an emotion analysis tool. She has looked at Senpy (http://senpy.gsi.upm.es/#test) and Twinword <https://www.twinword.com/api/emotion-analysis.php> - the latter seems the most accurate, but it is also expensive. > She has recently discovered DepecheMood emotion lexicons (Staiano, J., & Guerini, M. (2014). Depechemood: a lexicon for emotion analysis from crowd-annotated news. arXiv preprint arXiv:1405.1605.) - but this suffers from a lack of clarity in terms of explaining its emotional categories: awe, indifference, sad, amusement , annoyance, joy, fear and anger. > > For my part, I am entirely clueless. Any suggestions that she might pursue would be greatly appreciated. > > best, > - charles ess > -- > Professor in Media Studies > Department of Media and Communication > University of Oslo > <http://www.hf.uio.no/imk/english/people/aca/charlees/index.html> > > Postboks 1093 > Blindern 0317 > Oslo, Norway > c.m.ess@media.uio.no <mailto:c.m.ess@media.uio.no> > _______________________________________________ > The Air-L@listserv.aoir.org <mailto:Air-L@listserv.aoir.org> mailing list > is provided by the Association of Internet Researchers http://aoir.org > Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org > > Join the Association of Internet Researchers: > http://www.aoir.org/ >
_______________________________________________ The Air-L@listserv.aoir.org <mailto:Air-L@listserv.aoir.org> mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
-- Dr. Stuart W. Shulman Founder and CEO, Texifter Cell: 413-992-8513 LinkedIn: http://www.linkedin.com/in/stuartwshulman
-- Professor in Media Studies Department of Media and Communication University of Oslo <http://www.hf.uio.no/imk/english/people/aca/charlees/index.html> Postboks 1093 Blindern 0317 Oslo, Norway c.m.ess@media.uio.no
Dear all, Just revisiting this thread, and noticed some folk raised ethical issues in relation to sentiment analysis and emotional AI. I am looking for literature on the ethics of research using sentiment analysis tools and social media data (but also other types of data) and can¹t find much out there - have you come across anything or, better even, are you working on these issues? Andrew, I had a look at your website and approach - the critical issues you raise are interesting (summed up as 'is this OK?¹) and I¹d like to know more about how your team answers these questions but I can¹t find any answers. Are there any papers on methodologies and ethics at all coming out of the project? Many thanks, Aristea ----------------------------- Dr Aristea Fotopoulou UKRI-AHRC Innovation Leadership Fellow PI ART/DATA/HEALTH: Data as creative material for Health & Wellbeing 2019-2021 University of Brighton, School of Media Watts Building, Lewes Road, Brighton BN2 4GJ A.Fotopoulou@brighton.ac.uk |@aristeaf | https://aristeafotopoulou.org ART/DATA/HEALTH Research project: http://artdatahealth.org <http://artsdatahealth.org> On 12/09/2019 11:01, "Air-L on behalf of Charles M. Ess" <air-l-bounces@listserv.aoir.org on behalf of c.m.ess@media.uio.no> wrote:
Dear all - a belated but most sincere thanks for all of this!
I'll bundle up the thread, forward it to my student, and see what sense we might be able to make of it all.
Again, many thanks indeed and all best, - charles
On 07/09/2019 13:53, Shulman, Stu wrote:
Sally,
Machine generated sentiment analysis scores are sometimes abused as a shortcut to avoid certain forms of manual/mental labor in a variety of commercial and academic contexts. Language tools are in this scenario treated as a magic buttons to be deployed against corpora in the name of charts untouched by serious validation. I prefer it when humans are in-the-loop, which itself is recursive (meaning you repeat until there is no room to improve), using tools as filters to generate purposive samples that humans annotate and collectively validate using a systematic process.
Sentiment problems range from hard to harder and hardest, where hardest means you cannot do it in a manner that can be validated by any means. There is no easy on this scale of tasks if false positive or negatives could cost a life or some other serious consequence, but to make it easier, requires a process, grossly boiled down below:
1. Collect a relevant and representative corpus of data, 2. Build a SPAM detection classifier to remove non-relevant data (ex., wrong language OR no discernible sentiment), 3. Build a topic classifier and focus on one key topic first (not all topics at once), 4. Solve the Rubik's cube of how many codes and what they really mean (ex., happy/sad OR angry/frustrated/both/neither...), 5. Test the topic-specific annotation scheme with a group of no less than five independent annotators (not just two), 6. Crowd source the task to larger groups when possible, using memo writing to identify boundary cases that kill/modify models, 7. Use iteration to identify elite annotators through recursive validation, memo reviews, and scoring against a gold standard.
The goal is to build task- and language-specific machine classifiers using the best possible human experts in the process. The main idea, however, is to keep a critical role for humans.
~Stu
On Thu, Sep 5, 2019 at 4:11 PM Dr. S.A. Applin <sally@sally.com <mailto:sally@sally.com>> wrote:
Dear Charles (and List),
I see this as an ethics issue.
How reliable are ³emotion analysis² tools? How would outcomes from them be used?
As you say, there is a lack of clarity in some in terms of ³explaining emotional categories.² To me, this signals (along with obvious knowledge about the limitations and problems with algorithms), that there is opportunity here to be very, very, very wrong about people¹s opinions, and any algorithmically interpreted ³emotional² state.
For example, how would one interpret or finesse ³frustration,² vs ³anger²? The written word is contained within a language. Not all commenters will be native speakers to that language, and not all native speakers have the language tools required (even within their own language) to adequately express themselves, even in the best of times. What makes anyone think an algorithm would do better at this than a human trained in qualitative methods and with cultural and media and language knowledge?
There is way too much margin of potential error here for this to be automated, or ³useful.² It is much more likely that things will be assumed incorrectly by limited algorithms in the first place.
Furthermore, does your student see any problem with this exercise? That their tool analysis might get it very wrong? That the wrong might lead to assumptions or outcomes that are harmful to entities, people, governments?
What safeguards are in place for wrong assumptions and outcomes?
Kind regards,
Sally
Sally Applin, Ph.D. .......... Research Fellow HRAF Advanced Research Centres (EU), Canterbury Centre for Social Anthropology and Computing (CSAC) .......... Research Associate Human Relations Area Files (HRAF) Yale University .......... Associate Editor, IEEE Consumer Electronics Magazine Member, IoT Council Executive Board Member: The Edward H. and Rosamond B. Spicer Foundation .......... http://www.posr.org http://www.sally.com I am based in Silicon Valley .......... sally@sally.com <mailto:sally@sally.com> | 650.339.5236
> On Sep 5, 2019, at 3:52 AM, Charles M. Ess <c.m.ess@media.uio.no <mailto:c.m.ess@media.uio.no>> wrote: > > Dear colleagues, > > One of our students is wanting to analyze emotional content in in the comment fields of a major newspaper vis-a-vis specific hot-button issues. > > She has a good tool (I think) for scrapping the data - but she is stymied over the choice of an emotion analysis tool. She has looked at Senpy (http://senpy.gsi.upm.es/#test) and Twinword <https://www.twinword.com/api/emotion-analysis.php> - the latter seems the most accurate, but it is also expensive. > She has recently discovered DepecheMood emotion lexicons (Staiano, J., & Guerini, M. (2014). Depechemood: a lexicon for emotion analysis from crowd-annotated news. arXiv preprint arXiv:1405.1605.) - but this suffers from a lack of clarity in terms of explaining its emotional categories: awe, indifference, sad, amusement , annoyance, joy, fear and anger. > > For my part, I am entirely clueless. Any suggestions that she might pursue would be greatly appreciated. > > best, > - charles ess > -- > Professor in Media Studies > Department of Media and Communication > University of Oslo > <http://www.hf.uio.no/imk/english/people/aca/charlees/index.html> > > Postboks 1093 > Blindern 0317 > Oslo, Norway > c.m.ess@media.uio.no <mailto:c.m.ess@media.uio.no> > _______________________________________________ > The Air-L@listserv.aoir.org <mailto:Air-L@listserv.aoir.org> mailing list > is provided by the Association of Internet Researchers http://aoir.org > Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org > > Join the Association of Internet Researchers: > http://www.aoir.org/ >
_______________________________________________ The Air-L@listserv.aoir.org <mailto:Air-L@listserv.aoir.org> mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
-- Dr. Stuart W. Shulman Founder and CEO, Texifter Cell: 413-992-8513 LinkedIn: http://www.linkedin.com/in/stuartwshulman
-- Professor in Media Studies Department of Media and Communication University of Oslo <http://www.hf.uio.no/imk/english/people/aca/charlees/index.html>
Postboks 1093 Blindern 0317 Oslo, Norway c.m.ess@media.uio.no _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
__________________________________________________________________________ _____________________________________________________________________ CAUTION: This email may have originated from outside of the university. Do not click links or open attachments unless you recognise the sender and know the content is safe.
Hi Aristea, The following may help: - Leidner, J. L., & Plachouras, V. (2017, April). Ethical by design: ethics best practices for natural language processing. In *Proceedings of the First ACL Workshop on Ethics in Natural Language Processing* (pp. 30-40). - Hovy, D., & Spruit, S. L. (2016, August). The social impact of natural language processing. In *Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)* (pp. 591-598). Best, Ali Aristea Fotopoulou <A.Fotopoulou@brighton.ac.uk>, 27 Eyl 2019 Cum, 14:43 tarihinde şunu yazdı:
Dear all,
Just revisiting this thread, and noticed some folk raised ethical issues in relation to sentiment analysis and emotional AI. I am looking for literature on the ethics of research using sentiment analysis tools and social media data (but also other types of data) and can¹t find much out there - have you come across anything or, better even, are you working on these issues?
Andrew, I had a look at your website and approach - the critical issues you raise are interesting (summed up as 'is this OK?¹) and I¹d like to know more about how your team answers these questions but I can¹t find any answers. Are there any papers on methodologies and ethics at all coming out of the project?
Many thanks, Aristea
-----------------------------
Dr Aristea Fotopoulou UKRI-AHRC Innovation Leadership Fellow PI ART/DATA/HEALTH: Data as creative material for Health & Wellbeing 2019-2021 University of Brighton, School of Media Watts Building, Lewes Road, Brighton BN2 4GJ
A.Fotopoulou@brighton.ac.uk |@aristeaf | https://aristeafotopoulou.org ART/DATA/HEALTH Research project: http://artdatahealth.org <http://artsdatahealth.org>
On 12/09/2019 11:01, "Air-L on behalf of Charles M. Ess" <air-l-bounces@listserv.aoir.org on behalf of c.m.ess@media.uio.no> wrote:
Dear all - a belated but most sincere thanks for all of this!
I'll bundle up the thread, forward it to my student, and see what sense we might be able to make of it all.
Again, many thanks indeed and all best, - charles
On 07/09/2019 13:53, Shulman, Stu wrote:
Sally,
Machine generated sentiment analysis scores are sometimes abused as a shortcut to avoid certain forms of manual/mental labor in a variety of commercial and academic contexts. Language tools are in this scenario treated as a magic buttons to be deployed against corpora in the name of charts untouched by serious validation. I prefer it when humans are in-the-loop, which itself is recursive (meaning you repeat until there is no room to improve), using tools as filters to generate purposive samples that humans annotate and collectively validate using a systematic process.
Sentiment problems range from hard to harder and hardest, where hardest means you cannot do it in a manner that can be validated by any means. There is no easy on this scale of tasks if false positive or negatives could cost a life or some other serious consequence, but to make it easier, requires a process, grossly boiled down below:
1. Collect a relevant and representative corpus of data, 2. Build a SPAM detection classifier to remove non-relevant data (ex., wrong language OR no discernible sentiment), 3. Build a topic classifier and focus on one key topic first (not all topics at once), 4. Solve the Rubik's cube of how many codes and what they really mean (ex., happy/sad OR angry/frustrated/both/neither...), 5. Test the topic-specific annotation scheme with a group of no less than five independent annotators (not just two), 6. Crowd source the task to larger groups when possible, using memo writing to identify boundary cases that kill/modify models, 7. Use iteration to identify elite annotators through recursive validation, memo reviews, and scoring against a gold standard.
The goal is to build task- and language-specific machine classifiers using the best possible human experts in the process. The main idea, however, is to keep a critical role for humans.
~Stu
On Thu, Sep 5, 2019 at 4:11 PM Dr. S.A. Applin <sally@sally.com <mailto:sally@sally.com>> wrote:
Dear Charles (and List),
I see this as an ethics issue.
How reliable are ³emotion analysis² tools? How would outcomes from them be used?
As you say, there is a lack of clarity in some in terms of ³explaining emotional categories.² To me, this signals (along with obvious knowledge about the limitations and problems with algorithms), that there is opportunity here to be very, very, very wrong about people¹s opinions, and any algorithmically interpreted ³emotional² state.
For example, how would one interpret or finesse ³frustration,² vs ³anger²? The written word is contained within a language. Not all commenters will be native speakers to that language, and not all native speakers have the language tools required (even within their own language) to adequately express themselves, even in the best of times. What makes anyone think an algorithm would do better at this than a human trained in qualitative methods and with cultural and media and language knowledge?
There is way too much margin of potential error here for this to be automated, or ³useful.² It is much more likely that things will be assumed incorrectly by limited algorithms in the first place.
Furthermore, does your student see any problem with this exercise? That their tool analysis might get it very wrong? That the wrong might lead to assumptions or outcomes that are harmful to entities, people, governments?
What safeguards are in place for wrong assumptions and outcomes?
Kind regards,
Sally
Sally Applin, Ph.D. .......... Research Fellow HRAF Advanced Research Centres (EU), Canterbury Centre for Social Anthropology and Computing (CSAC) .......... Research Associate Human Relations Area Files (HRAF) Yale University .......... Associate Editor, IEEE Consumer Electronics Magazine Member, IoT Council Executive Board Member: The Edward H. and Rosamond B. Spicer Foundation .......... http://www.posr.org http://www.sally.com I am based in Silicon Valley .......... sally@sally.com <mailto:sally@sally.com> | 650.339.5236
> On Sep 5, 2019, at 3:52 AM, Charles M. Ess <c.m.ess@media.uio.no <mailto:c.m.ess@media.uio.no>> wrote: > > Dear colleagues, > > One of our students is wanting to analyze emotional content in in the comment fields of a major newspaper vis-a-vis specific hot-button issues. > > She has a good tool (I think) for scrapping the data - but she is stymied over the choice of an emotion analysis tool. She has looked at Senpy (http://senpy.gsi.upm.es/#test) and Twinword <https://www.twinword.com/api/emotion-analysis.php> - the latter seems the most accurate, but it is also expensive. > She has recently discovered DepecheMood emotion lexicons (Staiano, J., & Guerini, M. (2014). Depechemood: a lexicon for emotion analysis from crowd-annotated news. arXiv preprint arXiv:1405.1605.) - but this suffers from a lack of clarity in terms of explaining its emotional categories: awe, indifference, sad, amusement , annoyance, joy, fear and anger. > > For my part, I am entirely clueless. Any suggestions that she might pursue would be greatly appreciated. > > best, > - charles ess > -- > Professor in Media Studies > Department of Media and Communication > University of Oslo > <http://www.hf.uio.no/imk/english/people/aca/charlees/index.html
> > Postboks 1093 > Blindern 0317 > Oslo, Norway > c.m.ess@media.uio.no <mailto:c.m.ess@media.uio.no> > _______________________________________________ > The Air-L@listserv.aoir.org <mailto:Air-L@listserv.aoir.org> mailing list > is provided by the Association of Internet Researchers http://aoir.org > Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org > > Join the Association of Internet Researchers: > http://www.aoir.org/ >
_______________________________________________ The Air-L@listserv.aoir.org <mailto:Air-L@listserv.aoir.org>
mailing
list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
-- Dr. Stuart W. Shulman Founder and CEO, Texifter Cell: 413-992-8513 LinkedIn: http://www.linkedin.com/in/stuartwshulman
-- Professor in Media Studies Department of Media and Communication University of Oslo <http://www.hf.uio.no/imk/english/people/aca/charlees/index.html>
Postboks 1093 Blindern 0317 Oslo, Norway c.m.ess@media.uio.no _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
__________________________________________________________________________ _____________________________________________________________________ CAUTION: This email may have originated from outside of the university. Do not click links or open attachments unless you recognise the sender and know the content is safe.
_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers: http://www.aoir.org/
participants (12)
-
Alette Schoon -
ali hürriyetoglu -
Aristea Fotopoulou -
Charles M. Ess -
Dr. S.A. Applin -
Jessica Pater -
Jill Walker Rettberg -
kalev leetaru -
Shulman, Stu -
Stuart Shulman -
Veronika Thiel -
Wiebke Reile