I'd love to join as well! Associate Professor and Director of Graduate Studies, Department of Communication, Media, and the Arts College of Human Development, Culture, Media Seton Hall University, NJ, USA Keeping Women in Their Digital Place (2024, Penn State University Press) Editor-in-Chief, Journal of Religion, Media and Digital Culture (Brill) For more publications: https://works.bepress.com/ruth-tsuria/ -----Original Message----- From: Air-L <air-l-bounces@listserv.aoir.org> On Behalf Of Hüseyin Özçinar via Air-L Sent: Thursday, July 11, 2024 1:06 PM To: F vandenboom <f.vandenboom@hotmail.com> Cc: AoIR mailing list <air-l-aoir.org@listserv.aoir.org> Subject: Re: [Air-L] Trust Games - Annotation Tasks The topic is also of interest to me and I will be very happy to cooperate. Hüseyin Özçınar Pamukkale University Denizli/Turkey 11 Tem 2024 Per, saat 17:17 tarihinde F vandenboom via Air-L < air-l@listserv.aoir.org> şunu yazdı:

Hi Stu Count me in ! I use and am also developing several cardbased games on ai as part of my practice based research on anticipatory ai governance so would love to collaborate

Best Freyja

Sent from Outlook for Android<https://nam12.safelinks.protection.outlook.com/?url=https%3A%252 F%2Faka.ms%2FAAb9ysg&data=05%7C02%7Cruth.tsuria%40shu.edu%7Ce6745582b9 1b46ef515208dca1cc2a5a%7C51f07c2253b744dfb97ca13261d71075%7C1%7C0%7C63 8563145488969367%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=tHl%2FnUO%2BX XBbEOVjHciZspJsqj4JEJrJRC0ooMa3ki0%3D&reserved=0> ________________________________ From: Air-L <air-l-bounces@listserv.aoir.org> on behalf of Shulman, Stu via Air-L <air-l@listserv.aoir.org> Sent: Thursday, July 11, 2024 9:38:18 PM To: AoIR mailing list <air-l-aoir.org@listserv.aoir.org> Subject: [Air-L] Trust Games - Annotation Tasks

Trust Games

I am looking for collaborators to help prepare a free educational online game suitable for secondary and collegiate classrooms focused on whether content is trustworthy or not. Please contact me if you want to be a part of this effort. It might operate something like the "Which Face is Real" application, but might be used instead for identifying and discussing untrustworthy accounts on Twitter as a gamified learning module for classes this fall. I have most of the pieces ready, but I am not an expert in games. I'd like to form an ad hoc team and have this operational for September and October of 2024. My goal is to offer an IRB-compliant game platform that generates usable research results and better informed student discussions in advance of the U.S. election in November.

Annotation Tasks

I have a new set of annotation tasks related to planning for the game development. I need motivated undergraduates willing to label batches of Tweets under conditions that test core features of gamification in labeling, starting with speed and accuracy. In addition to getting paid more for being the fastest/most accurate labelers, students will see some remarkable datasets that have historical significance. If you know Jr. or Sr. undergraduates in the US or Canada with a >3.9 GPA, tell them to send me a resume.

Thanks AoIR!

-- Dr. Stuart W. Shulman Founder and CEO, Texifter Editor Emeritus, *Journal of Information Technology & Politics* _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir/. org%2F&data=05%7C02%7Cruth.tsuria%40shu.edu%7Ce6745582b91b46ef515208dc a1cc2a5a%7C51f07c2253b744dfb97ca13261d71075%7C1%7C0%7C6385631454889814 32%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI 6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=OlN6ffyi4caTij%2B%2BmIyWu1t Hv%2B9IosmTj893EIEvMK4%3D&reserved=0 <http://aoir/ .org%2F&data=05%7C02%7Cruth.tsuria%40shu.edu%7Ce6745582b91b46ef515208d ca1cc2a5a%7C51f07c2253b744dfb97ca13261d71075%7C1%7C0%7C638563145488987 244%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTi I6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=fc%2Bp9b1mk%2FPNY%2Bv3KsjA Q%2BFy11XfEznEu1GFz2QwmDw%3D&reserved=0> Subscribe, change options or unsubscribe at: http://lists/ erv.aoir.org%2Flistinfo.cgi%2Fair-l-aoir.org&data=05%7C02%7Cruth.tsuri a%40shu.edu%7Ce6745582b91b46ef515208dca1cc2a5a%7C51f07c2253b744dfb97ca 13261d71075%7C1%7C0%7C638563145488992567%7CUnknown%7CTWFpbGZsb3d8eyJWI joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7 C%7C&sdata=Z2RNaJsvB7rfH7%2FzcJRM%2BDLYoHyAO%2FgvTDRkg%2FwyzZI%3D&rese rved=0 <http://list/ serv.aoir.org%2Flistinfo.cgi%2Fair-l-aoir.org&data=05%7C02%7Cruth.tsur ia%40shu.edu%7Ce6745582b91b46ef515208dca1cc2a5a%7C51f07c2253b744dfb97c a13261d71075%7C1%7C0%7C638563145488997869%7CUnknown%7CTWFpbGZsb3d8eyJW IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C% 7C%7C&sdata=lQxm50L0ij0PKUADu6AXPkybw2B3I0QcX7PR%2F2HuySY%3D&reserved= 0>

Join the Association of Internet Researchers:

http://www.a/ oir.org%2F&data=05%7C02%7Cruth.tsuria%40shu.edu%7Ce6745582b91b46ef5152 08dca1cc2a5a%7C51f07c2253b744dfb97ca13261d71075%7C1%7C0%7C638563145489 004561%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=JD%2B3ZMANNDe40fkF2sySb O6WunqVkfNN%2BiVDiUjHUII%3D&reserved=0 <http://www/. aoir.org%2F&data=05%7C02%7Cruth.tsuria%40shu.edu%7Ce6745582b91b46ef515 208dca1cc2a5a%7C51f07c2253b744dfb97ca13261d71075%7C1%7C0%7C63856314548 9010646%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLC JBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=s%2FVsNzTB6bW0qxzr8YAV 2L1bim4h%2FqMGdSDmftidfUQ%3D&reserved=0> _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir/. org%2F&data=05%7C02%7Cruth.tsuria%40shu.edu%7Ce6745582b91b46ef515208dc a1cc2a5a%7C51f07c2253b744dfb97ca13261d71075%7C1%7C0%7C6385631454890159 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI 6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=YDAiWzFrUzdYMJle9n%2FChQT%2 BTtWWfPu9BAFNDIg0TBo%3D&reserved=0 Subscribe, change options or unsubscribe at: http://lists/ erv.aoir.org%2Flistinfo.cgi%2Fair-l-aoir.org&data=05%7C02%7Cruth.tsuri a%40shu.edu%7Ce6745582b91b46ef515208dca1cc2a5a%7C51f07c2253b744dfb97ca 13261d71075%7C1%7C0%7C638563145489021253%7CUnknown%7CTWFpbGZsb3d8eyJWI joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7 C%7C&sdata=Q9ieh15kkGpJFoSIGZr0aIvjG2p57%2FRYuJWyi3VSs1I%3D&reserved=0

Join the Association of Internet Researchers: http://www.a/ oir.org%2F&data=05%7C02%7Cruth.tsuria%40shu.edu%7Ce6745582b91b46ef5152 08dca1cc2a5a%7C51f07c2253b744dfb97ca13261d71075%7C1%7C0%7C638563145489 026589%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=e6dMPn8LuOuJGYeEY%2F0nz JpSNheS3G9WqgAK%2F4Ab6dM%3D&reserved=0

_______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org/ Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/ ** WARNING: This email originated from outside of Seton Hall University. Do not click links or open attachments unless you recognize the sender and know the content is safe. **

Thank you for your email. I am currently out of the office on vacation and will have limited access to email. I will return on August 15th and will respond to your message as soon as possible after that date. Best regards Vielen Dank für Ihre E-Mail. Ich bin derzeit im Urlaub und habe nur eingeschränkten Zugang zu meinen E-Mails. Ich kehre am 15. August zurück und werde Ihre Nachricht so bald wie möglich nach diesem Datum beantworten. Mit freundlichen Grüßen, Paloma On 17 Jul 2024, at 14:18, Shulman, Stu via Air-L <air-l@listserv.aoir.org> wrote:

Follow Up

I am grateful to the list for generating excellent people willing to work on projects. Already we have a nice group forming with novel exchanges of ideas in every meeting. Collaboration like this is fun and hopefully the game, Trust Defender, will be as well.

I do need more students, including some willing to label Twitter data as soon as today. I am testing a new approach to remunerating "micro" tasks that are limited to 5 minutes. I need to know the average speed, as well as the approximate best task size, and how fast the best human annotators can record reliable observations. I just did a small test and my rate was 1.45 seconds per item. For this initial "live-deleted-suspended" task, the key is a combined speed and accuracy score. If you make a keystroke error, you need to go back and fix it, which slows you down. If you let mistakes go unfixed, we will find them using measurement against a known gold standard during the initial rounds of training on this task.

The base remuneration is $20 for completion of tasks that have a total of 40 minutes allotted (8 tasks) during a roughly 24 hour period. It is definitely possible to complete all of these current tasks with close to 100% accuracy in less than 40 minutes. I am planning to start with a requirement for labeling 60 items in five minutes (5 seconds per item) in the early rounds. Then we will test the effect of increasing the number of items to 75 and 100 in later rounds. I'm trying to work out a prize system for the fastest and most accurate annotators over time. We have recently enhanced our IRB compliance architecture for this work. AS previously, deleted and suspended Tweets are not visible and now all Tweet metadata are also hidden from annotators. Our datasets have, in some cases, more than 70% of the items suspended. In these cases, it can be fascinating (also shocking) and definitely educational to see what is still live on Twitter.

One goal is to better understand how DiscoverText can be further modified to have default, ready-to-play games, as well as the traditional framework where it is simply a flexible tool, like a spreadsheet. Leaders of the games will create new tasks, codes, rules, parameters, code books, assignments, peer groups, and focus the students. Much of the content of the games can be created by the professors or teams of students. Our role now is creating archetypal game formats  such as the deductive "live, deleted, suspended" or a key new hybrid deductive/inductive "trust, don't trust, need more info" game that has nothing to do with speed or accuracy. More fully inductive educational games such as "bot, troll, or citizen" are possible. Small adaptations to the existing crowdsource software will make the "labeling" task more game-like. Students will train or play on default games out-of-the-box that are easy to launch, but professors will also adapt the framework with very minor training to define parameters for their own games. I will be working with professors to design rubrics so that students can play very short games in the fall then use the experience as fodder for discussion.

As a reminder: DiscoverText is 100% free for academic research and teaching. Anyone using DiscoverText to test or implement these annotation games can also pursue their own independent research agenda with other academics. We are creating a process that will streamline the onboarding of entire classes to enable "quick launch" games. There will be leaderboards and anonymization options.

https://discovertext.com/mentions/ https://calendly.com/discovertext

Stu

On Thu, Jul 11, 2024 at 8:38 AM Shulman, Stu <stu@texifter.com> wrote:

Trust Games

I am looking for collaborators to help prepare a free educational online game suitable for secondary and collegiate classrooms focused on whether content is trustworthy or not. Please contact me if you want to be a part of this effort. It might operate something like the "Which Face is Real" application, but might be used instead for identifying and discussing untrustworthy accounts on Twitter as a gamified learning module for classes this fall. I have most of the pieces ready, but I am not an expert in games. I'd like to form an ad hoc team and have this operational for September and October of 2024. My goal is to offer an IRB-compliant game platform that generates usable research results and better informed student discussions in advance of the U.S. election in November.

Annotation Tasks

I have a new set of annotation tasks related to planning for the game development. I need motivated undergraduates willing to label batches of Tweets under conditions that test core features of gamification in labeling, starting with speed and accuracy. In addition to getting paid more for being the fastest/most accurate labelers, students will see some remarkable datasets that have historical significance. If you know Jr. or Sr. undergraduates in the US or Canada with a >3.9 GPA, tell them to send me a resume.

Thanks AoIR!

-- Dr. Stuart W. Shulman Founder and CEO, Texifter Editor Emeritus, *Journal of Information Technology & Politics*

-- Dr. Stuart W. Shulman Founder and CEO, Texifter Editor Emeritus, *Journal of Information Technology & Politics* ResearchGate Profile <https://www.researchgate.net/profile/Stuart-Shulman> _______________________________________________ The Air-L@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org

Join the Association of Internet Researchers: http://www.aoir.org/