AI bots on webDiplomacy: Skynet edition!

[ppaquette]

Thanks very much Kestas. Congrats to you and your team.

Restitution, you can have access to the code on the following repos on Github:

- https://github.com/diplomacy/diplomacy/
- https://github.com/diplomacy/research/

Just the beginning of a new world!

[Murder Cat]

This is awesome! Hopefully the server starts processing again soon so I can try this out--I'm practically giddy!

[captainmeme]

Thanks very much Kestas. Congrats to you and your team.

Restitution, you can have access to the code on the following repos on Github:

- https://github.com/diplomacy/diplomacy/
- https://github.com/diplomacy/research/

Just the beginning of a new world!

I've read the paper, and a lot went over my head, but this is incredible and massive thanks for doing it!

I had one question about it - the paper says "orders from the last movementphase are enough to infer the current relationship between the powers". I know it's not easy to tell exactly what it would do in any given situation, but does this generally mean that if you have been supporting a bot for the whole game but then stop for one round, that bot will not consider you to be an ally anymore?

[peterwiggin]

I've read the paper, and a lot went over my head, but this is incredible and massive thanks for doing it!

I had one question about it - the paper says "orders from the last movementphase are enough to infer the current relationship between the powers". I know it's not easy to tell exactly what it would do in any given situation, but does this generally mean that if you have been supporting a bot for the whole game but then stop for one round, that bot will not consider you to be an ally anymore?

That's exactly what that means.

[peterwiggin]

The bots we have on webDip use a model developed by a group at MILA and described in this paper: https://arxiv.org/abs/1909.02128. Here, I'm going to give a summary of how the model works. For details, go read the paper!

In traditional computing, the programmer writes down what the program should do given specific inputs. In machine learning, the programmer instead defines a model and then uses examples (data) to teach the model how it should behave given certain inputs. For example, if I wanted a program to look at pictures of cats and dogs and tell me which ones were cats and which were dogs, I could attempt to write down a procedure by which my program could make that judgement. If I wanted to do this with machine learning, I would instead program a mathematical function that maps from a picture to a cat/dog prediction, and then use examples of cat and dog pictures to "train" the function's parameters so that it gives accurate predictions.

For a Diplomacy bot, we need to learn a policy that maps from a state (the placement of units on the board and possibly some history) to an action (orders for the bot's units). I'm going to start by describing the model used to map from state to action, and then describe how the model was trained.

We begin with an input that represents the board state and the orders from the previous phase. To represent the board state, for each of the 75 provinces and 6 coasts we encode some dynamic features such as whether there's a unit there, what type of unit, and who owns that unit, as well as some static features, such as whether it's a coast or province or sea space, and whether it's a supply center or not. The orders from the previous state are similarly encoded.

The encodings for the board state and previous-phase orders are then put into a model called a graph convolutional network, which is just a complicated mathematical function that looks at the relationships between the provinces and previous orders to arrive at a numerical representation of each province that incorporates information from the rest of the board state. The most important thing to know about this function is that it also takes the connections between provinces as input in order to take into account the movement rules.

A different function then looks at the output from the first network to produce orders, mapping from the state for that province to a probability for each of the possible orders for the unit in that province. The final order for the unit is then drawn from those probabilities. For powers with more than one unit needing orders, the orders are predicted one at a time, and the orders already chosen are taken into account when predicting the order for each proceeding unit.

The functions that encode the current state and use that encoding to predict orders both contain lots of parameters, which must be learned. Most parameter values would result in very bad orders, so the trick to all this is finding the parameters that lead to strong play. To do this, we get lots of examples of board states from human games as well as the orders that the humans submitted in each of those states. We pick a few board states and feed them into our models to get predicted orders out. We then compare those predictions to what the humans actually did and adjust the parameters to favor the human orders a little bit. By repeating this process, the models eventually output orders similar to what a human would in a given situation.

The drawback of this training method is that it requires lots of human examples. We can also train the models by having them play themselves and adjusting the parameters to make them more likely to capture centers and win. In this case, the two training methods yield bots with pretty similar strengths.

[dargorygel]

This is incredible.
HEARTY thanks and congrats to the team that PW has written about, and to our Devs, Admins, and Mods for seeing this, imagining this, working for this, building this, checking this, testing this and giving us this.

But supporting awards go to the whole mod team for keeping their mouths (and keyboards) shut about this. I am proud of the sound of silence.

And of course, special something goes to jmo, for giving up his robot-children for this worthy cause.

[MrAsiago]

https://webdiplomacy.net/board.php?gameID=246267

Perhaps this isn't the place for it, but my game just forced a draw against my will. I'm guessing this has to do with the outages that are happening right now, but is there any way I can get the game to restart?

[jmo1121109]

It was due to the outage. No, sorry but you can start another one.

[jmo1121109]

This is pretty amazing. I can only imagine the effort behind the scenes to make it happen

There's six figures worth of work that has gone into this. Maybe more.

This is not a joking estimate either. The developmental cost of this if the site had been required to hire a development team for this project would have been over 100,000 USD hands down. The immense time of the people at the University and the time we put in here was extensive. And we as a site are thrilled that we are able to offer these types of features without cost and without ads to our users.

Thank you to the donators who's donations cover our hosting costs and allow that to be possible as well.

[ibuprofen]

The new AI are causing the server outages! The Singularity is here! Resist the Machines!!!!!

[bo_sox48]

This is pretty amazing. I can only imagine the effort behind the scenes to make it happen

There's six figures worth of work that has gone into this. Maybe more.

This is not a joking estimate either. The developmental cost of this if the site had been required to hire a development team for this project would have been over 100,000 USD hands down. The immense time of the people at the University and the time we put in here was extensive. And we as a site are thrilled that we are able to offer these types of features without cost and without ads to our users.

Thank you to the donators who's donations cover our hosting costs and allow that to be possible as well.

And a bigger thank you to the developers who do this for free out of love for this platform and its members, and out of a desire to make it the best it can be.

Margaritas all around in Cabo this winter are on me.

[BobMcBob]

This is pretty amazing. I can only imagine the effort behind the scenes to make it happen

There's six figures worth of work that has gone into this. Maybe more.

This is not a joking estimate either. The developmental cost of this if the site had been required to hire a development team for this project would have been over 100,000 USD hands down. The immense time of the people at the University and the time we put in here was extensive. And we as a site are thrilled that we are able to offer these types of features without cost and without ads to our users.

Thank you to the donators who's donations cover our hosting costs and allow that to be possible as well.

I presume we didn't actually get to $100000 of donations though. Is it also partially because it was a university thing that this was possible?

[jmo1121109]

There's six figures worth of work that has gone into this. Maybe more.

This is not a joking estimate either. The developmental cost of this if the site had been required to hire a development team for this project would have been over 100,000 USD hands down. The immense time of the people at the University and the time we put in here was extensive. And we as a site are thrilled that we are able to offer these types of features without cost and without ads to our users.

Thank you to the donators who's donations cover our hosting costs and allow that to be possible as well.

I presume we didn't actually get to $100000 of donations though. Is it also partially because it was a university thing that this was possible?

Nope, this project was entirely free. The University gets publications, and I get..uhhh...hmmm, anyway enjoy.

[bo_sox48]

You get paid in +1s.

[BobMcBob]

So yeah, because it's a publication it's free. Makes sense. And don't worry, I'm sure you'll ge your share of +1's. I can't wait until I can finally actually play the bots though. As you may be able to tell, I'm very excited. Still waiting...

[jmo1121109]

So yeah, because it's a publication it's free. Makes sense. And don't worry, I'm sure you'll ge your share of +1's. I can't wait until I can finally actually play the bots though. As you may be able to tell, I'm very excited. Still waiting...

For what? We turned them back on a little bit ago, can’t swear it won’t go down again but we’re fixing issues as they come up

[BobMcBob]

Yeah, I missed that. I didn't realise you had to go into the game to start them. I'm losing now... Oh well, this change is great. Thanks guys!

[Temasek22]

I found out that the AI just stalemates itself eternally...

My game was Let's go bots!-2

[Claesar]

As part of the testing team (i.e. all mods) that got to try these before it went live, I have to say I've been very impressed by the level of play from the bots! Of course they're not perfect, but neither are humans. They should fit right in.

[MadMarx]

WTF I don’t log on for three months and all hell breaks loose... First thing I did was donate everything I had in PayPal $41.13 not a ton but if more do something similar it will add up...