Re: [Air-L] Request: Stop sending reviewer scores to authors
Hi Jean, That's definitely a good point and something we'll take on board looking at reviewing for next year. (As you say, I fear the fact that we're not already might be a technical limitation, but we'll investigate all the same.) On that front, it has been lovely that more and more journals are moving in this direction, too. (Such a relief when we turn out not to be reviewer 2!) Cheers, Tama On Fri, Jun 18, 2021 at 11:21 AM Jean Burgess <jean.burgess@qut.edu.au> wrote:
On that, Tama (and I know this isn't the first time someone has raised it), one of the most powerful aids to social learning would be to email the outcomes to reviewers as well, so that we can see our scores and comments alongside those of other reviewers. I understand Conftool may not let you do this, but it would be my top request (and like you say, if only our major granting bodies would do the same!).
On 17/6/21, 9:14 pm, "Air-L on behalf of Tama Leaver" < air-l-bounces@listserv.aoir.org on behalf of tamaleaver@gmail.com> wrote:
Dear Jill,
I'm sorry to hear receiving review scores was a bad experience for you.
Thanks for your suggestion about blinding the scores themselves and only providing comments, we will take it on board as part of the discussion and planning for next year's conference.
Since you asked I should add, though, that there are at least two functions currently achieved by having scores visible.
One is simple transparency: we do use the scores provided as part of the review process. It isn't the only signal, and this year there was a larger team so we could spend more time focusing on the qualitative comments in much more detail, but it is part of the review process and people often feel uncomfortable that they can't see part of what is being used to judge their submission. And of course, in spite of the many hours the conference committee spends on organising and supervising the review process, sometimes a bad review still slips through the cracks. The transparency of providing the review scores enables our participants to dispute them, if they feel a score is inappropriate.
(Just from personal experience, I hate that our major grants don't show the actual scores, just the comments, which clearly sometimes don't align well at all with the comments people make.)
The second reason is social: seeing our own numbers (whether fair or otherwise) is part of the process by which reviewers calibrate what fair numbers look like. While we do provide guidance on what the numbers indicate, sometimes your own past scores are one of the best ways to help people think about the numbers they provide in their reviews.
Those two points aside, I would also add, there is an internal conversation already underway about potentially moving away from a 1 to 100 scale to something else, and this feedback will definitely be considered in that discussion. If you have other suggestions of a productive reviewing system / review communication system that you’ve experienced elsewhere and think would be manageable for our organization, being mindful of the scale of our conference and that we stretch across many disciplines including quantitative ones, please do feel free to share your ideas with us.
Many thanks, Tama
On Thu, Jun 17, 2021 at 2:19 PM Jill Walker Rettberg < Jill.Walker.Rettberg@uib.no> wrote:
> I would like to make a formal request to the AoIR board that for future > conferences, the organization NOT send out reviewer scores to authors. > > I get why scores are useful for the selection process but it’s hard to see > how they are helpful for the author, and easy to see how they can be > harmful and demotivating. I’m in a secure position and can handle bad > reviews, but if I had received the scores I received this year as a junior > scholar I would have been devastated. > > The reviewer comments are useful to authors – thank you to all the > reviewers for the hard work, and for many constructive comments that will > be helpful for authors developing their ideas. But there is no reason to > send the scores to authors. > > There is also so much research showing the arbitrariness of peer review > (it’s not random but there is a LOT of subjectivity involved) that putting > such apparently objective scores out is really misleading. I mean, even > national and transnational research funders who pay reviewers and have > really robust systems for checking and double checking reviews have > problems setting scores. It’s basically impossible to do in a completely > fair way, and even more so in a volunteer system like ours where people > don’t have much time and reviewers are often assigned to assess papers on > topics and using methodologles they’re not familiar with. So let’s not make > the scores part of the feedback to authors. > > Not sending scores to authors is an easy fix, and we don’t even have to > get into all the annual discussions about do 1200 word abstracts even work, > are humanities perspectives properly evaluated by reviewers, how big should > the conference be, is it cool that some people present multiple things > while others are rejected completely etc. > > Are there ANY reasons to send scores to authors? > > Jill > > > Jill Walker Rettberg > Professor of Digital Culture > University of Bergen > > Principal Investigator of the ERC Consolidator grant Machine Vision in > Everyday Life: Playful Interactions with Visual Technologies in Digital > Art, Games, Narratives and Social Media. > https://urldefense.com/v3/__https://www.uib.no/en/machinevision/__;!!NVzLfOp... > > > _______________________________________________ > The Air-L@listserv.aoir.org mailing list > is provided by the Association of Internet Researchers https://urldefense.com/v3/__http://aoir.org__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFP... > Subscribe, change options or unsubscribe at: > https://urldefense.com/v3/__http://listserv.aoir.org/listinfo.cgi/air-l-aoir... > > Join the Association of Internet Researchers: > https://urldefense.com/v3/__http://www.aoir.org/__;!!NVzLfOphnbDXSw!T_Z5pmiA... >
-- Professor Tama Leaver Internet Studies Discipline Lead School of Media, Creative Arts and Social Inquiry (MCASI) Faculty of Humanities
Curtin University GPO Box U1987 Perth WA Australia 6845 Ph: (+61 8) 9266 1258 Email: t.leaver@curtin.edu.au Web: https://urldefense.com/v3/__http://www.tamaleaver.net__;!!NVzLfOphnbDXSw!T_Z... Twitter: @tamaleaver < https://urldefense.com/v3/__https://twitter.com/tamaleaver__;!!NVzLfOphnbDXS...
CRICOS Provider Code: 00301J (WA) _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers https://urldefense.com/v3/__http://aoir.org__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFP... Subscribe, change options or unsubscribe at: https://urldefense.com/v3/__http://listserv.aoir.org/listinfo.cgi/air-l-aoir...
Join the Association of Internet Researchers:
https://urldefense.com/v3/__http://www.aoir.org/__;!!NVzLfOphnbDXSw!T_Z5pmiA...
-- Professor Tama Leaver Internet Studies Discipline Lead School of Media, Creative Arts and Social Inquiry (MCASI) Faculty of Humanities Curtin University GPO Box U1987 Perth WA Australia 6845 Ph: (+61 8) 9266 1258 Email: t.leaver@curtin.edu.au Web: www.tamaleaver.net Twitter: @tamaleaver <https://twitter.com/tamaleaver> CRICOS Provider Code: 00301J (WA)
I’ve only used the Easychair system as an administrator - they don’t have this level of detailed scores, it’s more like journal reviewing, with strong accept, accept, weak accept, borderline, weak reject, reject, strong reject as options, plus comments. Reviewers also score their own confidence in their judgement (expert, high, medium, low, none) and the system generates rankings and simple scores (only visible to the admins as a support, not sent to authors) weighted accordingly. Both as a reviewer and as a program chair the reviewers’ own rating of their confidence has been EXTREMELY helpful. As a program chair dealing with several hundred submissions I found this more than detailed enough, and as a reviewer I find it more precise that the AoIR system. It’s easier to say whether I think an abstract is a weak reject or a strong accept or whatever than you give it a bunch of numerical scores for excellence and coherence - even if I’m not a subject expert. Jill ________________________________ From: Air-L <air-l-bounces@listserv.aoir.org> on behalf of Tama Leaver <tamaleaver@gmail.com> Sent: Friday, June 18, 2021 5:38 AM To: Jean Burgess Cc: aoir list Subject: Re: [Air-L] Request: Stop sending reviewer scores to authors Hi Jean, That's definitely a good point and something we'll take on board looking at reviewing for next year. (As you say, I fear the fact that we're not already might be a technical limitation, but we'll investigate all the same.) On that front, it has been lovely that more and more journals are moving in this direction, too. (Such a relief when we turn out not to be reviewer 2!) Cheers, Tama On Fri, Jun 18, 2021 at 11:21 AM Jean Burgess <jean.burgess@qut.edu.au> wrote:
On that, Tama (and I know this isn't the first time someone has raised it), one of the most powerful aids to social learning would be to email the outcomes to reviewers as well, so that we can see our scores and comments alongside those of other reviewers. I understand Conftool may not let you do this, but it would be my top request (and like you say, if only our major granting bodies would do the same!).
On 17/6/21, 9:14 pm, "Air-L on behalf of Tama Leaver" < air-l-bounces@listserv.aoir.org on behalf of tamaleaver@gmail.com> wrote:
Dear Jill,
I'm sorry to hear receiving review scores was a bad experience for you.
Thanks for your suggestion about blinding the scores themselves and only providing comments, we will take it on board as part of the discussion and planning for next year's conference.
Since you asked I should add, though, that there are at least two functions currently achieved by having scores visible.
One is simple transparency: we do use the scores provided as part of the review process. It isn't the only signal, and this year there was a larger team so we could spend more time focusing on the qualitative comments in much more detail, but it is part of the review process and people often feel uncomfortable that they can't see part of what is being used to judge their submission. And of course, in spite of the many hours the conference committee spends on organising and supervising the review process, sometimes a bad review still slips through the cracks. The transparency of providing the review scores enables our participants to dispute them, if they feel a score is inappropriate.
(Just from personal experience, I hate that our major grants don't show the actual scores, just the comments, which clearly sometimes don't align well at all with the comments people make.)
The second reason is social: seeing our own numbers (whether fair or otherwise) is part of the process by which reviewers calibrate what fair numbers look like. While we do provide guidance on what the numbers indicate, sometimes your own past scores are one of the best ways to help people think about the numbers they provide in their reviews.
Those two points aside, I would also add, there is an internal conversation already underway about potentially moving away from a 1 to 100 scale to something else, and this feedback will definitely be considered in that discussion. If you have other suggestions of a productive reviewing system / review communication system that you’ve experienced elsewhere and think would be manageable for our organization, being mindful of the scale of our conference and that we stretch across many disciplines including quantitative ones, please do feel free to share your ideas with us.
Many thanks, Tama
On Thu, Jun 17, 2021 at 2:19 PM Jill Walker Rettberg < Jill.Walker.Rettberg@uib.no> wrote:
> I would like to make a formal request to the AoIR board that for future > conferences, the organization NOT send out reviewer scores to authors. > > I get why scores are useful for the selection process but it’s hard to see > how they are helpful for the author, and easy to see how they can be > harmful and demotivating. I’m in a secure position and can handle bad > reviews, but if I had received the scores I received this year as a junior > scholar I would have been devastated. > > The reviewer comments are useful to authors – thank you to all the > reviewers for the hard work, and for many constructive comments that will > be helpful for authors developing their ideas. But there is no reason to > send the scores to authors. > > There is also so much research showing the arbitrariness of peer review > (it’s not random but there is a LOT of subjectivity involved) that putting > such apparently objective scores out is really misleading. I mean, even > national and transnational research funders who pay reviewers and have > really robust systems for checking and double checking reviews have > problems setting scores. It’s basically impossible to do in a completely > fair way, and even more so in a volunteer system like ours where people > don’t have much time and reviewers are often assigned to assess papers on > topics and using methodologles they’re not familiar with. So let’s not make > the scores part of the feedback to authors. > > Not sending scores to authors is an easy fix, and we don’t even have to > get into all the annual discussions about do 1200 word abstracts even work, > are humanities perspectives properly evaluated by reviewers, how big should > the conference be, is it cool that some people present multiple things > while others are rejected completely etc. > > Are there ANY reasons to send scores to authors? > > Jill > > > Jill Walker Rettberg > Professor of Digital Culture > University of Bergen > > Principal Investigator of the ERC Consolidator grant Machine Vision in > Everyday Life: Playful Interactions with Visual Technologies in Digital > Art, Games, Narratives and Social Media. > https://urldefense.com/v3/__https://www.uib.no/en/machinevision/__;!!NVzLfOp... > > > _______________________________________________ > The Air-L@listserv.aoir.org mailing list > is provided by the Association of Internet Researchers https://urldefense.com/v3/__http://aoir.org__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFP... > Subscribe, change options or unsubscribe at: > https://urldefense.com/v3/__http://listserv.aoir.org/listinfo.cgi/air-l-aoir... > > Join the Association of Internet Researchers: > https://urldefense.com/v3/__http://www.aoir.org/__;!!NVzLfOphnbDXSw!T_Z5pmiA... >
-- Professor Tama Leaver Internet Studies Discipline Lead School of Media, Creative Arts and Social Inquiry (MCASI) Faculty of Humanities
Curtin University GPO Box U1987 Perth WA Australia 6845 Ph: (+61 8) 9266 1258 Email: t.leaver@curtin.edu.au Web: https://urldefense.com/v3/__http://www.tamaleaver.net__;!!NVzLfOphnbDXSw!T_Z... Twitter: @tamaleaver < https://urldefense.com/v3/__https://twitter.com/tamaleaver__;!!NVzLfOphnbDXS...
CRICOS Provider Code: 00301J (WA) _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers https://urldefense.com/v3/__http://aoir.org__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFP... Subscribe, change options or unsubscribe at: https://urldefense.com/v3/__http://listserv.aoir.org/listinfo.cgi/air-l-aoir...
Join the Association of Internet Researchers:
https://urldefense.com/v3/__http://www.aoir.org/__;!!NVzLfOphnbDXSw!T_Z5pmiA...
-- Professor Tama Leaver Internet Studies Discipline Lead School of Media, Creative Arts and Social Inquiry (MCASI) Faculty of Humanities Curtin University GPO Box U1987 Perth WA Australia 6845 Ph: (+61 8) 9266 1258 Email: t.leaver@curtin.edu.au Web: www.tamaleaver.net<http://www.tamaleaver.net> Twitter: @tamaleaver <https://twitter.com/tamaleaver> CRICOS Provider Code: 00301J (WA) _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/
participants (2)
-
Jill Walker Rettberg -
Tama Leaver