Gender information for academic purposes

General discussions that don't fit in other forums can go here.
Forum rules
Feel free to discuss any topics here. Please use the Politics sub-forum for political conversations. While most topics will be allowed please be sure to be respectful and follow our normal site rules at http://www.webdiplomacy.net/rules.php.

Which gender are you?

You may select 1 option

 
 
View results

Message
Author
momom2
Posts: 2
Joined: Mon Jul 31, 2023 6:48 am
Contact:

Gender information for academic purposes

#1 Post by momom2 » Mon Jul 31, 2023 6:49 am

Hello!
I'm writing a paper studying (among other things) gender bias in Cicero, an AI trained using data from webDiplomacy: https://ai.meta.com/blog/cicero-ai-nego ... tent=video

The authors say that they noticed Cicero showed gender bias, and that the data they were given had about 75% masculine pronouns and 1% feminine pronouns, but they didn't mention how that relates to the actual genders of webDiplomacy players.
So, I'd like to know if you know a survey about Diplomacy players which recorded genders, or if the webDiplomacy admins have some stats on the subject.

In addition, here's a poll on the subject, so I can maybe collect my own data!

Sincerely,

Octavious
Posts: 4028
Joined: Fri Sep 29, 2017 4:16 pm
Location: The Five Valleys, Gloucestershire
Contact:

Re: Gender information for academic purposes

#2 Post by Octavious » Mon Jul 31, 2023 9:01 am

Out of curiosity, is it not safe to assume that people who prefer not to say will exercise their preference by simply not saying, rather than actively saying that they'd prefer not to say?
I eat cookies to improve my snacking experience

learnedSloth
Posts: 206
Joined: Mon Sep 12, 2022 10:20 pm
Contact:

Re: Gender information for academic purposes

#3 Post by learnedSloth » Mon Jul 31, 2023 9:56 am

Octavious wrote:
Mon Jul 31, 2023 9:01 am
Out of curiosity, is it not safe to assume that people who prefer not to say will exercise their preference by simply not saying, rather than actively saying that they'd prefer not to say?
I think that OP just copied without thought a standard survey form designed to cover all edge cases.
¶ Keep thy heart with all diligence; for out of it are the issues of life.
-- Proverbs of Solomon, chapter 4, verse 23

User avatar
Jamiet99uk
Posts: 32404
Joined: Sat Dec 30, 2017 11:42 pm
Location: Durham, UK
Contact:

Re: Gender information for academic purposes

#4 Post by Jamiet99uk » Mon Jul 31, 2023 11:38 am

Octavious wrote:
Mon Jul 31, 2023 9:01 am
Out of curiosity, is it not safe to assume that people who prefer not to say will exercise their preference by simply not saying, rather than actively saying that they'd prefer not to say?
I have an opinion on your question, Octavious.... but I prefer not to share it.
The only person you're truly competing against, Wesley, is yourself.

User avatar
Doom427
Posts: 813
Joined: Fri Jan 18, 2019 6:54 am
Contact:

Re: Gender information for academic purposes

#5 Post by Doom427 » Mon Jul 31, 2023 12:58 pm

As a fellow Academic I will say you perhaps aren't going to get the best data in the world from the forum, which has a different demographics set than random pick up games that make up the majority of Cicero's dataset.

Of course,I have no clue how you COULD actually get a gender breakdown of Web diplomacy. But it's something to be aware of.

User avatar
Esquire Bertissimmo
Posts: 567
Joined: Fri May 05, 2023 11:44 pm
Contact:

Re: Gender information for academic purposes

#6 Post by Esquire Bertissimmo » Tue Aug 01, 2023 8:37 pm

The full article is behind a paywall so we're missing some details. I'd be interested in knowing how the authors concluded there was gender bias. That's certainly a huge skew towards masculine pronouns, but what if it turns out that WebDip is genuinely overwhelmingly played by men? Is it still "biased" if it accurately reflects the data it was trained on and the group it will play with?

User avatar
JECE
Posts: 1704
Joined: Sun Dec 31, 2017 4:35 pm
Contact:

Re: Gender information for academic purposes

#7 Post by JECE » Tue Aug 01, 2023 11:51 pm

Esquire Bertissimmo wrote:
Tue Aug 01, 2023 8:37 pm
That's certainly a huge skew towards masculine pronouns, but what if it turns out that WebDip is genuinely overwhelmingly played by men? Is it still "biased" if it accurately reflects the data it was trained on and the group it will play with?
<facepalm>

Of course it's still gender bias. If the AI was trained on overwhelmingly masculine data then that just helps measure the extent of the gender bias.
See my full Profile:
http://www.webdiplomacy.net/profile.php?userID=17421

User avatar
Esquire Bertissimmo
Posts: 567
Joined: Fri May 05, 2023 11:44 pm
Contact:

Re: Gender information for academic purposes

#8 Post by Esquire Bertissimmo » Wed Aug 02, 2023 3:53 am

JECE wrote:
Tue Aug 01, 2023 11:51 pm
Esquire Bertissimmo wrote:
Tue Aug 01, 2023 8:37 pm
That's certainly a huge skew towards masculine pronouns, but what if it turns out that WebDip is genuinely overwhelmingly played by men? Is it still "biased" if it accurately reflects the data it was trained on and the group it will play with?
<facepalm>

Of course it's still gender bias. If the AI was trained on overwhelmingly masculine data then that just helps measure the extent of the gender bias.
Maybe there's something I'm missing. If an advertisement for feminine hygiene products doesn't have any men in it I'd hesitate to call that biased. If a bot trained heavily on male-skewed WebDip data and then defaults more often to masculine pronouns is that "bias" or just an accurate reflection of its training set?

Said another way, would artificially feeding the bot data such that half the data came from female players and the other half from men reduce the bias if that is intact very unlike the actual gender balance on the site?

captainmeme
Posts: 619
Joined: Fri Sep 29, 2017 3:06 pm
Location: Manchester, UK
Contact:

Re: Gender information for academic purposes

#9 Post by captainmeme » Wed Aug 02, 2023 10:14 am

Esquire Bertissimmo wrote:
Wed Aug 02, 2023 3:53 am
If a bot trained heavily on male-skewed WebDip data and then defaults more often to masculine pronouns is that "bias" or just an accurate reflection of its training set?
It's both. Bias in AI generally means it's accurately reflecting biased training data.

Good AI needs to be able to work well in a variety of situations, which is why this is a major issue. If it very specifically works well with WebDiplomacy's past demographics then that's great if it's playing on webDip and the demographics have stayed the same, but it's a problem if it's playing anywhere else or if the demographics change.

User avatar
Esquire Bertissimmo
Posts: 567
Joined: Fri May 05, 2023 11:44 pm
Contact:

Re: Gender information for academic purposes

#10 Post by Esquire Bertissimmo » Wed Aug 02, 2023 3:32 pm

captainmeme wrote:
Wed Aug 02, 2023 10:14 am
Esquire Bertissimmo wrote:
Wed Aug 02, 2023 3:53 am
If a bot trained heavily on male-skewed WebDip data and then defaults more often to masculine pronouns is that "bias" or just an accurate reflection of its training set?
It's both. Bias in AI generally means it's accurately reflecting biased training data.

Good AI needs to be able to work well in a variety of situations, which is why this is a major issue. If it very specifically works well with WebDiplomacy's past demographics then that's great if it's playing on webDip and the demographics have stayed the same, but it's a problem if it's playing anywhere else or if the demographics change.
Thanks, I found this clarifying!

That said, I'm still a little confused. To consider an edge case, is an AI that's trained to spot cervical cancer female-sex biased? The answer is "yes" by definition, but it's incomprehensible to me to call that a bias because it perfectly reflects the training data and the group it is going to be used on. The "bias" in this case would only be an issue if you wanted to extend the model to see if it can detect other cancers that both sexes suffer from.

Maybe it's something similar here? If the model accurately reflects the gender balance of the site (I'm assuming it's skewed male-identifying) and it's only used on this site, wouldn't it just hurt the efficiency of the model to try to de-bias it? Maybe the researchers just want to better know the extent of the bias, or maybe they want to extend the model to other games or sites where they expect the preponderance of man-generated data to be a drawback. All of this would probably be more clear if we had access to the scientific paper linked in the article posted by OP.

User avatar
Doom427
Posts: 813
Joined: Fri Jan 18, 2019 6:54 am
Contact:

Re: Gender information for academic purposes

#11 Post by Doom427 » Wed Aug 02, 2023 3:50 pm

Well, the reason the survey is here is to see if it accurately reflects the gender balance of the website. We can assume it accurately reflects the gender bias of messages received in random games. The bias is that the model that exists to send diplomacy messages will default to he/him for all players, so if one player doesn't use he/him, and every other player can tell, the AI will be, somewhat obviously, faulty.

( A cervical cancer ai wouldn't be female sex-biases because an AI that detects cervical cancer doesn't need to sex the photos it's looking at. It's not useful information. In diplomacy however, knowing what to say about the other players is pretty important - you need to talk in the third-person all the time. )

User avatar
Esquire Bertissimmo
Posts: 567
Joined: Fri May 05, 2023 11:44 pm
Contact:

Re: Gender information for academic purposes

#12 Post by Esquire Bertissimmo » Wed Aug 02, 2023 4:08 pm

Doom427 wrote:
Wed Aug 02, 2023 3:50 pm
Well, the reason the survey is here is to see if it accurately reflects the gender balance of the website. We can assume it accurately reflects the gender bias of messages received in random games. The bias is that the model that exists to send diplomacy messages will default to he/him for all players, so if one player doesn't use he/him, and every other player can tell, the AI will be, somewhat obviously, faulty.

( A cervical cancer ai wouldn't be female sex-biases because an AI that detects cervical cancer doesn't need to sex the photos it's looking at. It's not useful information. In diplomacy however, knowing what to say about the other players is pretty important - you need to talk in the third-person all the time. )
Thanks for this, totally clears it up for me!

Ginge86
Posts: 209
Joined: Thu Nov 11, 2021 5:06 pm
Location: In your mums bed
Contact:

Re: Gender information for academic purposes

#13 Post by Ginge86 » Thu Aug 03, 2023 12:05 pm

Who would have ever guessed that the majority of players are male?
What a huge shock that must be to everyone.

User avatar
Doom427
Posts: 813
Joined: Fri Jan 18, 2019 6:54 am
Contact:

Re: Gender information for academic purposes

#14 Post by Doom427 » Sat Aug 05, 2023 1:45 pm

It is pretty male heavy, which makes sense! A game about gossiping behind someone's back while pretending to be friends is classic male behavior. Women just don't get much experience with that (I'm an outlier I know)

returning players realize this is a rewritten joke I used last year during the feminist wars

Still, only one woman? That's kinda crazy.

User avatar
kestasjk
Developer
Developer
Posts: 534
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Contact:

Re: Gender information for academic purposes

#15 Post by kestasjk » Sat Aug 05, 2023 6:28 pm

If I look up the stats in analytics it looks more varied at 75% male 25% female over the last 12 months.. But I'm not sure how accurate it is, not sure how they know the gender of users. I imagine it's based on search patterns or some such?

The age stats also seem dubious, I feel like our user based would be skewed a bit older?
Screenshot 2023-08-06 022544.jpg

User avatar
JECE
Posts: 1704
Joined: Sun Dec 31, 2017 4:35 pm
Contact:

Re: Gender information for academic purposes

#16 Post by JECE » Sat Aug 05, 2023 8:06 pm

The age breakdown makes sense to me. We just pretend to be old, ha ha
See my full Profile:
http://www.webdiplomacy.net/profile.php?userID=17421

Octavious
Posts: 4028
Joined: Fri Sep 29, 2017 4:16 pm
Location: The Five Valleys, Gloucestershire
Contact:

Re: Gender information for academic purposes

#17 Post by Octavious » Sun Aug 06, 2023 12:02 am

The stats seem bang on to me. They show that webDip users come predominantly from the three great cities of the world.

London, New York, and Ashburn.

When you're tired of Ashburn you're tired of life
I eat cookies to improve my snacking experience

User avatar
Jamiet99uk
Posts: 32404
Joined: Sat Dec 30, 2017 11:42 pm
Location: Durham, UK
Contact:

Re: Gender information for academic purposes

#18 Post by Jamiet99uk » Sun Aug 06, 2023 11:06 am

Octavious wrote:
Sun Aug 06, 2023 12:02 am
The stats seem bang on to me. They show that webDip users come predominantly from the three great cities of the world.

London, New York, and Ashburn.

When you're tired of Ashburn you're tired of life
Are you thinking of Ashbourne?
The only person you're truly competing against, Wesley, is yourself.

Octavious
Posts: 4028
Joined: Fri Sep 29, 2017 4:16 pm
Location: The Five Valleys, Gloucestershire
Contact:

Re: Gender information for academic purposes

#19 Post by Octavious » Sun Aug 06, 2023 11:52 am

Jamie, mate. I was born in Plymouth. UK geographic knowledge for people from Devon is pretty good for Devon and Cornwall, passable for Dorset and Somerset, and completely ignorant of everywhere else. I can assure you that I was not thinking of Ashbourne. ;)
I eat cookies to improve my snacking experience

User avatar
Jamiet99uk
Posts: 32404
Joined: Sat Dec 30, 2017 11:42 pm
Location: Durham, UK
Contact:

Re: Gender information for academic purposes

#20 Post by Jamiet99uk » Sun Aug 06, 2023 4:36 pm

Apparently Ashburn (in the USA) is a major hub for Internet data centres, so I'd guess we don't actually have hundreds of users living there.
The only person you're truly competing against, Wesley, is yourself.

Post Reply

Who is online

Users browsing this forum: No registered users