AI bots on webDiplomacy: Skynet edition!
Re: AI bots on webDiplomacy: Skynet edition!
Thanks very much Kestas. Congrats to you and your team.
Restitution, you can have access to the code on the following repos on Github:
- https://github.com/diplomacy/diplomacy/
- https://github.com/diplomacy/research/
Just the beginning of a new world!
Restitution, you can have access to the code on the following repos on Github:
- https://github.com/diplomacy/diplomacy/
- https://github.com/diplomacy/research/
Just the beginning of a new world!
13
-
- Posts: 2
- Joined: Mon Jul 29, 2019 9:32 pm
- Karma: 3
- Contact:
Re: AI bots on webDiplomacy: Skynet edition!
This is awesome! Hopefully the server starts processing again soon so I can try this out--I'm practically giddy!
2
-
- Posts: 617
- Joined: Fri Sep 29, 2017 3:06 pm
- Location: Manchester, UK
- Karma: 768
- Contact:
Re: AI bots on webDiplomacy: Skynet edition!
I've read the paper, and a lot went over my head, but this is incredible and massive thanks for doing it!ppaquette wrote: ↑Sat Sep 14, 2019 1:15 amThanks very much Kestas. Congrats to you and your team.
Restitution, you can have access to the code on the following repos on Github:
- https://github.com/diplomacy/diplomacy/
- https://github.com/diplomacy/research/
Just the beginning of a new world!
I had one question about it - the paper says "orders from the last movementphase are enough to infer the current relationship between the powers". I know it's not easy to tell exactly what it would do in any given situation, but does this generally mean that if you have been supporting a bot for the whole game but then stop for one round, that bot will not consider you to be an ally anymore?
2
-
- Site Admin
- Posts: 172
- Joined: Fri Jun 02, 2017 7:55 am
- Location: Boston, MA
- Karma: 427
- Contact:
Re: AI bots on webDiplomacy: Skynet edition!
That's exactly what that means.captainmeme wrote: ↑Sat Sep 14, 2019 1:37 am
I've read the paper, and a lot went over my head, but this is incredible and massive thanks for doing it!
I had one question about it - the paper says "orders from the last movementphase are enough to infer the current relationship between the powers". I know it's not easy to tell exactly what it would do in any given situation, but does this generally mean that if you have been supporting a bot for the whole game but then stop for one round, that bot will not consider you to be an ally anymore?
4
-
- Site Admin
- Posts: 172
- Joined: Fri Jun 02, 2017 7:55 am
- Location: Boston, MA
- Karma: 427
- Contact:
Re: AI bots on webDiplomacy: Skynet edition!
The bots we have on webDip use a model developed by a group at MILA and described in this paper: https://arxiv.org/abs/1909.02128. Here, I'm going to give a summary of how the model works. For details, go read the paper!
In traditional computing, the programmer writes down what the program should do given specific inputs. In machine learning, the programmer instead defines a model and then uses examples (data) to teach the model how it should behave given certain inputs. For example, if I wanted a program to look at pictures of cats and dogs and tell me which ones were cats and which were dogs, I could attempt to write down a procedure by which my program could make that judgement. If I wanted to do this with machine learning, I would instead program a mathematical function that maps from a picture to a cat/dog prediction, and then use examples of cat and dog pictures to "train" the function's parameters so that it gives accurate predictions.
For a Diplomacy bot, we need to learn a policy that maps from a state (the placement of units on the board and possibly some history) to an action (orders for the bot's units). I'm going to start by describing the model used to map from state to action, and then describe how the model was trained.
We begin with an input that represents the board state and the orders from the previous phase. To represent the board state, for each of the 75 provinces and 6 coasts we encode some dynamic features such as whether there's a unit there, what type of unit, and who owns that unit, as well as some static features, such as whether it's a coast or province or sea space, and whether it's a supply center or not. The orders from the previous state are similarly encoded.
The encodings for the board state and previous-phase orders are then put into a model called a graph convolutional network, which is just a complicated mathematical function that looks at the relationships between the provinces and previous orders to arrive at a numerical representation of each province that incorporates information from the rest of the board state. The most important thing to know about this function is that it also takes the connections between provinces as input in order to take into account the movement rules.
A different function then looks at the output from the first network to produce orders, mapping from the state for that province to a probability for each of the possible orders for the unit in that province. The final order for the unit is then drawn from those probabilities. For powers with more than one unit needing orders, the orders are predicted one at a time, and the orders already chosen are taken into account when predicting the order for each proceeding unit.
The functions that encode the current state and use that encoding to predict orders both contain lots of parameters, which must be learned. Most parameter values would result in very bad orders, so the trick to all this is finding the parameters that lead to strong play. To do this, we get lots of examples of board states from human games as well as the orders that the humans submitted in each of those states. We pick a few board states and feed them into our models to get predicted orders out. We then compare those predictions to what the humans actually did and adjust the parameters to favor the human orders a little bit. By repeating this process, the models eventually output orders similar to what a human would in a given situation.
The drawback of this training method is that it requires lots of human examples. We can also train the models by having them play themselves and adjusting the parameters to make them more likely to capture centers and win. In this case, the two training methods yield bots with pretty similar strengths.
In traditional computing, the programmer writes down what the program should do given specific inputs. In machine learning, the programmer instead defines a model and then uses examples (data) to teach the model how it should behave given certain inputs. For example, if I wanted a program to look at pictures of cats and dogs and tell me which ones were cats and which were dogs, I could attempt to write down a procedure by which my program could make that judgement. If I wanted to do this with machine learning, I would instead program a mathematical function that maps from a picture to a cat/dog prediction, and then use examples of cat and dog pictures to "train" the function's parameters so that it gives accurate predictions.
For a Diplomacy bot, we need to learn a policy that maps from a state (the placement of units on the board and possibly some history) to an action (orders for the bot's units). I'm going to start by describing the model used to map from state to action, and then describe how the model was trained.
We begin with an input that represents the board state and the orders from the previous phase. To represent the board state, for each of the 75 provinces and 6 coasts we encode some dynamic features such as whether there's a unit there, what type of unit, and who owns that unit, as well as some static features, such as whether it's a coast or province or sea space, and whether it's a supply center or not. The orders from the previous state are similarly encoded.
The encodings for the board state and previous-phase orders are then put into a model called a graph convolutional network, which is just a complicated mathematical function that looks at the relationships between the provinces and previous orders to arrive at a numerical representation of each province that incorporates information from the rest of the board state. The most important thing to know about this function is that it also takes the connections between provinces as input in order to take into account the movement rules.
A different function then looks at the output from the first network to produce orders, mapping from the state for that province to a probability for each of the possible orders for the unit in that province. The final order for the unit is then drawn from those probabilities. For powers with more than one unit needing orders, the orders are predicted one at a time, and the orders already chosen are taken into account when predicting the order for each proceeding unit.
The functions that encode the current state and use that encoding to predict orders both contain lots of parameters, which must be learned. Most parameter values would result in very bad orders, so the trick to all this is finding the parameters that lead to strong play. To do this, we get lots of examples of board states from human games as well as the orders that the humans submitted in each of those states. We pick a few board states and feed them into our models to get predicted orders out. We then compare those predictions to what the humans actually did and adjust the parameters to favor the human orders a little bit. By repeating this process, the models eventually output orders similar to what a human would in a given situation.
The drawback of this training method is that it requires lots of human examples. We can also train the models by having them play themselves and adjusting the parameters to make them more likely to capture centers and win. In this case, the two training methods yield bots with pretty similar strengths.
18
- dargorygel
- Site Moderator
- Posts: 6212
- Joined: Mon Dec 18, 2017 1:55 pm
- Location: Over the rainbow
- Karma: 6434
- Contact:
Re: AI bots on webDiplomacy: Skynet edition!
This is incredible.
HEARTY thanks and congrats to the team that PW has written about, and to our Devs, Admins, and Mods for seeing this, imagining this, working for this, building this, checking this, testing this and giving us this.
But supporting awards go to the whole mod team for keeping their mouths (and keyboards) shut about this. I am proud of the sound of silence.
And of course, special something goes to jmo, for giving up his robot-children for this worthy cause.
HEARTY thanks and congrats to the team that PW has written about, and to our Devs, Admins, and Mods for seeing this, imagining this, working for this, building this, checking this, testing this and giving us this.
But supporting awards go to the whole mod team for keeping their mouths (and keyboards) shut about this. I am proud of the sound of silence.
And of course, special something goes to jmo, for giving up his robot-children for this worthy cause.
4
Re: AI bots on webDiplomacy: Skynet edition!
https://webdiplomacy.net/board.php?gameID=246267
Perhaps this isn't the place for it, but my game just forced a draw against my will. I'm guessing this has to do with the outages that are happening right now, but is there any way I can get the game to restart?
Perhaps this isn't the place for it, but my game just forced a draw against my will. I'm guessing this has to do with the outages that are happening right now, but is there any way I can get the game to restart?
1
-
- Lifetime Site Contributor
- Posts: 1099
- Joined: Fri Sep 29, 2017 4:20 pm
- Karma: 2944
- Contact:
Re: AI bots on webDiplomacy: Skynet edition!
It was due to the outage. No, sorry but you can start another one.
1
-
- Lifetime Site Contributor
- Posts: 1099
- Joined: Fri Sep 29, 2017 4:20 pm
- Karma: 2944
- Contact:
Re: AI bots on webDiplomacy: Skynet edition!
This is not a joking estimate either. The developmental cost of this if the site had been required to hire a development team for this project would have been over 100,000 USD hands down. The immense time of the people at the University and the time we put in here was extensive. And we as a site are thrilled that we are able to offer these types of features without cost and without ads to our users.bo_sox48 wrote: ↑Fri Sep 13, 2019 11:14 pmThere's six figures worth of work that has gone into this. Maybe more.Tom Bombadil wrote: ↑Fri Sep 13, 2019 11:06 pmThis is pretty amazing. I can only imagine the effort behind the scenes to make it happen
Thank you to the donators who's donations cover our hosting costs and allow that to be possible as well.
12
Re: AI bots on webDiplomacy: Skynet edition!
The new AI are causing the server outages! The Singularity is here! Resist the Machines!!!!!
4
Re: AI bots on webDiplomacy: Skynet edition!
And a bigger thank you to the developers who do this for free out of love for this platform and its members, and out of a desire to make it the best it can be.jmo1121109 wrote: ↑Sat Sep 14, 2019 3:10 amThis is not a joking estimate either. The developmental cost of this if the site had been required to hire a development team for this project would have been over 100,000 USD hands down. The immense time of the people at the University and the time we put in here was extensive. And we as a site are thrilled that we are able to offer these types of features without cost and without ads to our users.bo_sox48 wrote: ↑Fri Sep 13, 2019 11:14 pmThere's six figures worth of work that has gone into this. Maybe more.Tom Bombadil wrote: ↑Fri Sep 13, 2019 11:06 pmThis is pretty amazing. I can only imagine the effort behind the scenes to make it happen
Thank you to the donators who's donations cover our hosting costs and allow that to be possible as well.
Margaritas all around in Cabo this winter are on me.
8
-
- Posts: 3150
- Joined: Mon Feb 11, 2019 6:28 am
- Location: Also Not Austalia
- Karma: 2123
- Contact:
Re: AI bots on webDiplomacy: Skynet edition!
I presume we didn't actually get to $100000 of donations though. Is it also partially because it was a university thing that this was possible?jmo1121109 wrote: ↑Sat Sep 14, 2019 3:10 amThis is not a joking estimate either. The developmental cost of this if the site had been required to hire a development team for this project would have been over 100,000 USD hands down. The immense time of the people at the University and the time we put in here was extensive. And we as a site are thrilled that we are able to offer these types of features without cost and without ads to our users.bo_sox48 wrote: ↑Fri Sep 13, 2019 11:14 pmThere's six figures worth of work that has gone into this. Maybe more.Tom Bombadil wrote: ↑Fri Sep 13, 2019 11:06 pmThis is pretty amazing. I can only imagine the effort behind the scenes to make it happen
Thank you to the donators who's donations cover our hosting costs and allow that to be possible as well.
1
-
- Lifetime Site Contributor
- Posts: 1099
- Joined: Fri Sep 29, 2017 4:20 pm
- Karma: 2944
- Contact:
Re: AI bots on webDiplomacy: Skynet edition!
Nope, this project was entirely free. The University gets publications, and I get..uhhh...hmmm, anyway enjoy.BobMcBob wrote: ↑Sat Sep 14, 2019 3:27 amI presume we didn't actually get to $100000 of donations though. Is it also partially because it was a university thing that this was possible?jmo1121109 wrote: ↑Sat Sep 14, 2019 3:10 amThis is not a joking estimate either. The developmental cost of this if the site had been required to hire a development team for this project would have been over 100,000 USD hands down. The immense time of the people at the University and the time we put in here was extensive. And we as a site are thrilled that we are able to offer these types of features without cost and without ads to our users.
Thank you to the donators who's donations cover our hosting costs and allow that to be possible as well.
17
-
- Posts: 3150
- Joined: Mon Feb 11, 2019 6:28 am
- Location: Also Not Austalia
- Karma: 2123
- Contact:
Re: AI bots on webDiplomacy: Skynet edition!
So yeah, because it's a publication it's free. Makes sense. And don't worry, I'm sure you'll ge your share of +1's. I can't wait until I can finally actually play the bots though. As you may be able to tell, I'm very excited. Still waiting...
2
-
- Lifetime Site Contributor
- Posts: 1099
- Joined: Fri Sep 29, 2017 4:20 pm
- Karma: 2944
- Contact:
Re: AI bots on webDiplomacy: Skynet edition!
For what? We turned them back on a little bit ago, can’t swear it won’t go down again but we’re fixing issues as they come up
4
-
- Posts: 3150
- Joined: Mon Feb 11, 2019 6:28 am
- Location: Also Not Austalia
- Karma: 2123
- Contact:
Re: AI bots on webDiplomacy: Skynet edition!
Yeah, I missed that. I didn't realise you had to go into the game to start them. I'm losing now... Oh well, this change is great. Thanks guys!
2
Re: AI bots on webDiplomacy: Skynet edition!
I found out that the AI just stalemates itself eternally...
My game was Let's go bots!-2
My game was Let's go bots!-2
2
Re: AI bots on webDiplomacy: Skynet edition!
As part of the testing team (i.e. all mods) that got to try these before it went live, I have to say I've been very impressed by the level of play from the bots! Of course they're not perfect, but neither are humans. They should fit right in.
2
Re: AI bots on webDiplomacy: Skynet edition!
WTF I don’t log on for three months and all hell breaks loose... First thing I did was donate everything I had in PayPal $41.13 not a ton but if more do something similar it will add up...
3
Who is online
Users browsing this forum: No registered users and 81 guests