Gender information for academic purposes
Forum rules
Feel free to discuss any topics here. Please use the Politics sub-forum for political conversations. While most topics will be allowed please be sure to be respectful and follow our normal site rules at http://www.webdiplomacy.net/rules.php.
Feel free to discuss any topics here. Please use the Politics sub-forum for political conversations. While most topics will be allowed please be sure to be respectful and follow our normal site rules at http://www.webdiplomacy.net/rules.php.
Gender information for academic purposes
Hello!
I'm writing a paper studying (among other things) gender bias in Cicero, an AI trained using data from webDiplomacy: https://ai.meta.com/blog/cicero-ai-nego ... tent=video
The authors say that they noticed Cicero showed gender bias, and that the data they were given had about 75% masculine pronouns and 1% feminine pronouns, but they didn't mention how that relates to the actual genders of webDiplomacy players.
So, I'd like to know if you know a survey about Diplomacy players which recorded genders, or if the webDiplomacy admins have some stats on the subject.
In addition, here's a poll on the subject, so I can maybe collect my own data!
Sincerely,
I'm writing a paper studying (among other things) gender bias in Cicero, an AI trained using data from webDiplomacy: https://ai.meta.com/blog/cicero-ai-nego ... tent=video
The authors say that they noticed Cicero showed gender bias, and that the data they were given had about 75% masculine pronouns and 1% feminine pronouns, but they didn't mention how that relates to the actual genders of webDiplomacy players.
So, I'd like to know if you know a survey about Diplomacy players which recorded genders, or if the webDiplomacy admins have some stats on the subject.
In addition, here's a poll on the subject, so I can maybe collect my own data!
Sincerely,
-
- Posts: 4028
- Joined: Fri Sep 29, 2017 4:16 pm
- Location: The Five Valleys, Gloucestershire
- Contact:
Re: Gender information for academic purposes
Out of curiosity, is it not safe to assume that people who prefer not to say will exercise their preference by simply not saying, rather than actively saying that they'd prefer not to say?
I eat cookies to improve my snacking experience
-
- Posts: 206
- Joined: Mon Sep 12, 2022 10:20 pm
- Contact:
Re: Gender information for academic purposes
I think that OP just copied without thought a standard survey form designed to cover all edge cases.
¶ Keep thy heart with all diligence; for out of it are the issues of life.
-- Proverbs of Solomon, chapter 4, verse 23
-- Proverbs of Solomon, chapter 4, verse 23
- Jamiet99uk
- Posts: 32404
- Joined: Sat Dec 30, 2017 11:42 pm
- Location: Durham, UK
- Contact:
Re: Gender information for academic purposes
I have an opinion on your question, Octavious.... but I prefer not to share it.
The only person you're truly competing against, Wesley, is yourself.
Re: Gender information for academic purposes
As a fellow Academic I will say you perhaps aren't going to get the best data in the world from the forum, which has a different demographics set than random pick up games that make up the majority of Cicero's dataset.
Of course,I have no clue how you COULD actually get a gender breakdown of Web diplomacy. But it's something to be aware of.
Of course,I have no clue how you COULD actually get a gender breakdown of Web diplomacy. But it's something to be aware of.
- Esquire Bertissimmo
- Posts: 567
- Joined: Fri May 05, 2023 11:44 pm
- Contact:
Re: Gender information for academic purposes
The full article is behind a paywall so we're missing some details. I'd be interested in knowing how the authors concluded there was gender bias. That's certainly a huge skew towards masculine pronouns, but what if it turns out that WebDip is genuinely overwhelmingly played by men? Is it still "biased" if it accurately reflects the data it was trained on and the group it will play with?
Re: Gender information for academic purposes
<facepalm>Esquire Bertissimmo wrote: ↑Tue Aug 01, 2023 8:37 pmThat's certainly a huge skew towards masculine pronouns, but what if it turns out that WebDip is genuinely overwhelmingly played by men? Is it still "biased" if it accurately reflects the data it was trained on and the group it will play with?
Of course it's still gender bias. If the AI was trained on overwhelmingly masculine data then that just helps measure the extent of the gender bias.
See my full Profile:
http://www.webdiplomacy.net/profile.php?userID=17421
http://www.webdiplomacy.net/profile.php?userID=17421
- Esquire Bertissimmo
- Posts: 567
- Joined: Fri May 05, 2023 11:44 pm
- Contact:
Re: Gender information for academic purposes
Maybe there's something I'm missing. If an advertisement for feminine hygiene products doesn't have any men in it I'd hesitate to call that biased. If a bot trained heavily on male-skewed WebDip data and then defaults more often to masculine pronouns is that "bias" or just an accurate reflection of its training set?JECE wrote: ↑Tue Aug 01, 2023 11:51 pm<facepalm>Esquire Bertissimmo wrote: ↑Tue Aug 01, 2023 8:37 pmThat's certainly a huge skew towards masculine pronouns, but what if it turns out that WebDip is genuinely overwhelmingly played by men? Is it still "biased" if it accurately reflects the data it was trained on and the group it will play with?
Of course it's still gender bias. If the AI was trained on overwhelmingly masculine data then that just helps measure the extent of the gender bias.
Said another way, would artificially feeding the bot data such that half the data came from female players and the other half from men reduce the bias if that is intact very unlike the actual gender balance on the site?
-
- Posts: 619
- Joined: Fri Sep 29, 2017 3:06 pm
- Location: Manchester, UK
- Contact:
Re: Gender information for academic purposes
It's both. Bias in AI generally means it's accurately reflecting biased training data.Esquire Bertissimmo wrote: ↑Wed Aug 02, 2023 3:53 amIf a bot trained heavily on male-skewed WebDip data and then defaults more often to masculine pronouns is that "bias" or just an accurate reflection of its training set?
Good AI needs to be able to work well in a variety of situations, which is why this is a major issue. If it very specifically works well with WebDiplomacy's past demographics then that's great if it's playing on webDip and the demographics have stayed the same, but it's a problem if it's playing anywhere else or if the demographics change.
- Esquire Bertissimmo
- Posts: 567
- Joined: Fri May 05, 2023 11:44 pm
- Contact:
Re: Gender information for academic purposes
Thanks, I found this clarifying!captainmeme wrote: ↑Wed Aug 02, 2023 10:14 amIt's both. Bias in AI generally means it's accurately reflecting biased training data.Esquire Bertissimmo wrote: ↑Wed Aug 02, 2023 3:53 amIf a bot trained heavily on male-skewed WebDip data and then defaults more often to masculine pronouns is that "bias" or just an accurate reflection of its training set?
Good AI needs to be able to work well in a variety of situations, which is why this is a major issue. If it very specifically works well with WebDiplomacy's past demographics then that's great if it's playing on webDip and the demographics have stayed the same, but it's a problem if it's playing anywhere else or if the demographics change.
That said, I'm still a little confused. To consider an edge case, is an AI that's trained to spot cervical cancer female-sex biased? The answer is "yes" by definition, but it's incomprehensible to me to call that a bias because it perfectly reflects the training data and the group it is going to be used on. The "bias" in this case would only be an issue if you wanted to extend the model to see if it can detect other cancers that both sexes suffer from.
Maybe it's something similar here? If the model accurately reflects the gender balance of the site (I'm assuming it's skewed male-identifying) and it's only used on this site, wouldn't it just hurt the efficiency of the model to try to de-bias it? Maybe the researchers just want to better know the extent of the bias, or maybe they want to extend the model to other games or sites where they expect the preponderance of man-generated data to be a drawback. All of this would probably be more clear if we had access to the scientific paper linked in the article posted by OP.
Re: Gender information for academic purposes
Well, the reason the survey is here is to see if it accurately reflects the gender balance of the website. We can assume it accurately reflects the gender bias of messages received in random games. The bias is that the model that exists to send diplomacy messages will default to he/him for all players, so if one player doesn't use he/him, and every other player can tell, the AI will be, somewhat obviously, faulty.
( A cervical cancer ai wouldn't be female sex-biases because an AI that detects cervical cancer doesn't need to sex the photos it's looking at. It's not useful information. In diplomacy however, knowing what to say about the other players is pretty important - you need to talk in the third-person all the time. )
( A cervical cancer ai wouldn't be female sex-biases because an AI that detects cervical cancer doesn't need to sex the photos it's looking at. It's not useful information. In diplomacy however, knowing what to say about the other players is pretty important - you need to talk in the third-person all the time. )
- Esquire Bertissimmo
- Posts: 567
- Joined: Fri May 05, 2023 11:44 pm
- Contact:
Re: Gender information for academic purposes
Thanks for this, totally clears it up for me!Doom427 wrote: ↑Wed Aug 02, 2023 3:50 pmWell, the reason the survey is here is to see if it accurately reflects the gender balance of the website. We can assume it accurately reflects the gender bias of messages received in random games. The bias is that the model that exists to send diplomacy messages will default to he/him for all players, so if one player doesn't use he/him, and every other player can tell, the AI will be, somewhat obviously, faulty.
( A cervical cancer ai wouldn't be female sex-biases because an AI that detects cervical cancer doesn't need to sex the photos it's looking at. It's not useful information. In diplomacy however, knowing what to say about the other players is pretty important - you need to talk in the third-person all the time. )
Re: Gender information for academic purposes
Who would have ever guessed that the majority of players are male?
What a huge shock that must be to everyone.
What a huge shock that must be to everyone.
Re: Gender information for academic purposes
It is pretty male heavy, which makes sense! A game about gossiping behind someone's back while pretending to be friends is classic male behavior. Women just don't get much experience with that (I'm an outlier I know)
returning players realize this is a rewritten joke I used last year during the feminist wars
Still, only one woman? That's kinda crazy.
returning players realize this is a rewritten joke I used last year during the feminist wars
Still, only one woman? That's kinda crazy.
Re: Gender information for academic purposes
If I look up the stats in analytics it looks more varied at 75% male 25% female over the last 12 months.. But I'm not sure how accurate it is, not sure how they know the gender of users. I imagine it's based on search patterns or some such?
The age stats also seem dubious, I feel like our user based would be skewed a bit older?
The age stats also seem dubious, I feel like our user based would be skewed a bit older?
Re: Gender information for academic purposes
The age breakdown makes sense to me. We just pretend to be old, ha ha
See my full Profile:
http://www.webdiplomacy.net/profile.php?userID=17421
http://www.webdiplomacy.net/profile.php?userID=17421
-
- Posts: 4028
- Joined: Fri Sep 29, 2017 4:16 pm
- Location: The Five Valleys, Gloucestershire
- Contact:
Re: Gender information for academic purposes
The stats seem bang on to me. They show that webDip users come predominantly from the three great cities of the world.
London, New York, and Ashburn.
When you're tired of Ashburn you're tired of life
London, New York, and Ashburn.
When you're tired of Ashburn you're tired of life
I eat cookies to improve my snacking experience
- Jamiet99uk
- Posts: 32404
- Joined: Sat Dec 30, 2017 11:42 pm
- Location: Durham, UK
- Contact:
Re: Gender information for academic purposes
Are you thinking of Ashbourne?
The only person you're truly competing against, Wesley, is yourself.
-
- Posts: 4028
- Joined: Fri Sep 29, 2017 4:16 pm
- Location: The Five Valleys, Gloucestershire
- Contact:
Re: Gender information for academic purposes
Jamie, mate. I was born in Plymouth. UK geographic knowledge for people from Devon is pretty good for Devon and Cornwall, passable for Dorset and Somerset, and completely ignorant of everywhere else. I can assure you that I was not thinking of Ashbourne.
I eat cookies to improve my snacking experience
- Jamiet99uk
- Posts: 32404
- Joined: Sat Dec 30, 2017 11:42 pm
- Location: Durham, UK
- Contact:
Re: Gender information for academic purposes
Apparently Ashburn (in the USA) is a major hub for Internet data centres, so I'd guess we don't actually have hundreds of users living there.
The only person you're truly competing against, Wesley, is yourself.
Who is online
Users browsing this forum: No registered users