webdiplomacy.net

Hello!
I'm writing a paper studying (among other things) gender bias in Cicero, an AI trained using data from webDiplomacy: https://ai.meta.com/blog/cicero-ai-nego ... tent=video

The authors say that they noticed Cicero showed gender bias, and that the data they were given had about 75% masculine pronouns and 1% feminine pronouns, but they didn't mention how that relates to the actual genders of webDiplomacy players.
So, I'd like to know if you know a survey about Diplomacy players which recorded genders, or if the webDiplomacy admins have some stats on the subject.

In addition, here's a poll on the subject, so I can maybe collect my own data!

Sincerely,

Out of curiosity, is it not safe to assume that people who prefer not to say will exercise their preference by simply not saying, rather than actively saying that they'd prefer not to say?

Octavious wrote: ↑

Mon Jul 31, 2023 9:01 am

Out of curiosity, is it not safe to assume that people who prefer not to say will exercise their preference by simply not saying, rather than actively saying that they'd prefer not to say?

I think that OP just copied without thought a standard survey form designed to cover all edge cases.

Octavious wrote: ↑

Mon Jul 31, 2023 9:01 am

Out of curiosity, is it not safe to assume that people who prefer not to say will exercise their preference by simply not saying, rather than actively saying that they'd prefer not to say?

I have an opinion on your question, Octavious.... but I prefer not to share it.

As a fellow Academic I will say you perhaps aren't going to get the best data in the world from the forum, which has a different demographics set than random pick up games that make up the majority of Cicero's dataset.

Of course,I have no clue how you COULD actually get a gender breakdown of Web diplomacy. But it's something to be aware of.

The full article is behind a paywall so we're missing some details. I'd be interested in knowing how the authors concluded there was gender bias. That's certainly a huge skew towards masculine pronouns, but what if it turns out that WebDip is genuinely overwhelmingly played by men? Is it still "biased" if it accurately reflects the data it was trained on and the group it will play with?

Esquire Bertissimmo wrote: ↑

Tue Aug 01, 2023 8:37 pm

That's certainly a huge skew towards masculine pronouns, but what if it turns out that WebDip is genuinely overwhelmingly played by men? Is it still "biased" if it accurately reflects the data it was trained on and the group it will play with?

<facepalm>

Of course it's still gender bias. If the AI was trained on overwhelmingly masculine data then that just helps measure the extent of the gender bias.

JECE wrote: ↑

Tue Aug 01, 2023 11:51 pm

Esquire Bertissimmo wrote: ↑

Tue Aug 01, 2023 8:37 pm

That's certainly a huge skew towards masculine pronouns, but what if it turns out that WebDip is genuinely overwhelmingly played by men? Is it still "biased" if it accurately reflects the data it was trained on and the group it will play with?

<facepalm>

Of course it's still gender bias. If the AI was trained on overwhelmingly masculine data then that just helps measure the extent of the gender bias.

Maybe there's something I'm missing. If an advertisement for feminine hygiene products doesn't have any men in it I'd hesitate to call that biased. If a bot trained heavily on male-skewed WebDip data and then defaults more often to masculine pronouns is that "bias" or just an accurate reflection of its training set?

Said another way, would artificially feeding the bot data such that half the data came from female players and the other half from men reduce the bias if that is intact very unlike the actual gender balance on the site?

Esquire Bertissimmo wrote: ↑

Wed Aug 02, 2023 3:53 am

If a bot trained heavily on male-skewed WebDip data and then defaults more often to masculine pronouns is that "bias" or just an accurate reflection of its training set?

It's both. Bias in AI generally means it's accurately reflecting biased training data.

Good AI needs to be able to work well in a variety of situations, which is why this is a major issue. If it very specifically works well with WebDiplomacy's past demographics then that's great if it's playing on webDip and the demographics have stayed the same, but it's a problem if it's playing anywhere else or if the demographics change.

captainmeme wrote: ↑

Wed Aug 02, 2023 10:14 am

Esquire Bertissimmo wrote: ↑

Wed Aug 02, 2023 3:53 am

If a bot trained heavily on male-skewed WebDip data and then defaults more often to masculine pronouns is that "bias" or just an accurate reflection of its training set?

It's both. Bias in AI generally means it's accurately reflecting biased training data.

Good AI needs to be able to work well in a variety of situations, which is why this is a major issue. If it very specifically works well with WebDiplomacy's past demographics then that's great if it's playing on webDip and the demographics have stayed the same, but it's a problem if it's playing anywhere else or if the demographics change.

Thanks, I found this clarifying!

That said, I'm still a little confused. To consider an edge case, is an AI that's trained to spot cervical cancer female-sex biased? The answer is "yes" by definition, but it's incomprehensible to me to call that a bias because it perfectly reflects the training data and the group it is going to be used on. The "bias" in this case would only be an issue if you wanted to extend the model to see if it can detect other cancers that both sexes suffer from.

Maybe it's something similar here? If the model accurately reflects the gender balance of the site (I'm assuming it's skewed male-identifying) and it's only used on this site, wouldn't it just hurt the efficiency of the model to try to de-bias it? Maybe the researchers just want to better know the extent of the bias, or maybe they want to extend the model to other games or sites where they expect the preponderance of man-generated data to be a drawback. All of this would probably be more clear if we had access to the scientific paper linked in the article posted by OP.

Well, the reason the survey is here is to see if it accurately reflects the gender balance of the website. We can assume it accurately reflects the gender bias of messages received in random games. The bias is that the model that exists to send diplomacy messages will default to he/him for all players, so if one player doesn't use he/him, and every other player can tell, the AI will be, somewhat obviously, faulty.

( A cervical cancer ai wouldn't be female sex-biases because an AI that detects cervical cancer doesn't need to sex the photos it's looking at. It's not useful information. In diplomacy however, knowing what to say about the other players is pretty important - you need to talk in the third-person all the time. )

Doom427 wrote: ↑

Wed Aug 02, 2023 3:50 pm

Well, the reason the survey is here is to see if it accurately reflects the gender balance of the website. We can assume it accurately reflects the gender bias of messages received in random games. The bias is that the model that exists to send diplomacy messages will default to he/him for all players, so if one player doesn't use he/him, and every other player can tell, the AI will be, somewhat obviously, faulty.

( A cervical cancer ai wouldn't be female sex-biases because an AI that detects cervical cancer doesn't need to sex the photos it's looking at. It's not useful information. In diplomacy however, knowing what to say about the other players is pretty important - you need to talk in the third-person all the time. )

Thanks for this, totally clears it up for me!

Who would have ever guessed that the majority of players are male?
What a huge shock that must be to everyone.

It is pretty male heavy, which makes sense! A game about gossiping behind someone's back while pretending to be friends is classic male behavior. Women just don't get much experience with that (I'm an outlier I know)

returning players realize this is a rewritten joke I used last year during the feminist wars

Still, only one woman? That's kinda crazy.

If I look up the stats in analytics it looks more varied at 75% male 25% female over the last 12 months.. But I'm not sure how accurate it is, not sure how they know the gender of users. I imagine it's based on search patterns or some such?

The age stats also seem dubious, I feel like our user based would be skewed a bit older?

The age breakdown makes sense to me. We just pretend to be old, ha ha

The stats seem bang on to me. They show that webDip users come predominantly from the three great cities of the world.

London, New York, and Ashburn.

When you're tired of Ashburn you're tired of life

Octavious wrote: ↑

Sun Aug 06, 2023 12:02 am

The stats seem bang on to me. They show that webDip users come predominantly from the three great cities of the world.

London, New York, and Ashburn.

When you're tired of Ashburn you're tired of life

Are you thinking of Ashbourne?

Jamie, mate. I was born in Plymouth. UK geographic knowledge for people from Devon is pretty good for Devon and Cornwall, passable for Dorset and Somerset, and completely ignorant of everywhere else. I can assure you that I was not thinking of Ashbourne.

Apparently Ashburn (in the USA) is a major hub for Internet data centres, so I'd guess we don't actually have hundreds of users living there.