Introducing the first ever webDiplomacy Terminator Tournament!

Find the tournament calendar, sign up for tournaments, track tournament results, and fill out the tournament director questionnaire here.
Message
Author
Scarabus
Posts: 37
Joined: Sun Dec 09, 2018 4:27 pm
Karma: 18
Contact:

Re: Introducing the first ever webDiplomacy Terminator Tournament!

#261 Post by Scarabus » Sun Oct 20, 2019 2:02 pm

Ok,my thoughts on the bots:-

- firstly, and in a broad sense, they were chllenging to an extent, and I recognise the previous comments that they are "learning" from moves made in all games. I would be interested in how this works. Will they eventually become "super players", or is there a limit on what/how much they can? I will focus on the areas I think they need to improve, although not really knowing whether the programming can be changed in any way.

-the main area for improvement is their seeming inability to sense when a player is about to solo. I am talking about when one country gets to say 14 centres. It would be natural for most decent players to gang up, even to a small degree. In one of my games, playing Germany, the Turkish bot simply carried on taking french and english centres while I soloed. I wouls expect that the bots should have some level of sense of danger in this respect. Even if allied to a player, this should be overwritten with a desire to get a draw, rather than to keep growing at the expense of losing the game.

-the next area I saw as quite weak was tactically. It seemed too easy to hold a line against a bot, where the bot could easily have broken my lines. They seemed not to recognise when cutting support could give them the edge in taking a centre. Again, this was evident even when I was about to solo. Maybe when calculating the odds of success they fail to understand that cutting support reduces the opposing "support count". Without knowing how they work it out, it is difficult to say why, but this is how I saw it.

-the other area I saw as questionable was their seeming willingness to perpetually bounce, rather than to introduce some level of "randomness". In a few situations, it was clear to me that I could gradually grow, while two or three other countries were effectively static, because their lack of frustration with a position meant their moves were entirely predictable.

These were my main points. I don't intend any of this as a criticism, because I found it to be a fun experience, but I am sure everyone would like to see them improve. I hope so, because it would give more of a challenge, and more of a feel that you are playing against a real opponent. If I think of anything else, I will post more later.

As has already been said, thanks for organising.
3

TheGhostmaker
Posts: 15
Joined: Sun Feb 04, 2018 2:13 pm
Karma: 19
Contact:

Re: Introducing the first ever webDiplomacy Terminator Tournament!

#262 Post by TheGhostmaker » Sun Oct 20, 2019 3:19 pm

Thanks for running the tournament bo, I had a blast! It was way more fun playing the bots with something on the line. They've got flaws, but the bots are really impressive to me.

Chapeau to Scarabus for the highest total score across both rounds.

Some unsolicited format thoughts, if this is done again (maybe against v2 of the bots?):

I'd like to have more games per round (e.g. 7 per round, and play each country once). 3 makes it too luck based for me. For example, in round 2, Scarabus and RoganJosh had draws as England, which I didn’t have to play; genghiz and xorxes’s Austria games had Italy opening to Tyr or Tri, whereas mine had a Lepanto. Extra games wouldn't even make the tournament take much longer for me, since I played ~5 practice games per country anyway.

In both rounds I think a lot of players were tracking other players' scores. I know I did, particularly in round 1 where I wanted to figure out if a draw was good enough. Now we know what a good score looks like, in round 1 you could use a fixed score cut off (e.g. 100 points per game), so we don't have to check scores.

Finally, size of draw felt a little bit arbitrary in the scoring system. Scarabus took a 4-way in this game: http://webdiplomacy.net/board.php?gameID=250817 because Russia was still alive. Maybe scoring of drawn games should be based on counting supply centres, because the bots don't go for eliminations?
1

Scarabus
Posts: 37
Joined: Sun Dec 09, 2018 4:27 pm
Karma: 18
Contact:

Re: Introducing the first ever webDiplomacy Terminator Tournament!

#263 Post by Scarabus » Sun Oct 20, 2019 3:28 pm

I didn't have to play Austria in the tournament either.

That game was Floodgates, not me.

Scarabus
Posts: 37
Joined: Sun Dec 09, 2018 4:27 pm
Karma: 18
Contact:

Re: Introducing the first ever webDiplomacy Terminator Tournament!

#264 Post by Scarabus » Sun Oct 20, 2019 3:29 pm

But I agree the draw criteria for the bots are a bit arbitrary.

TheGhostmaker
Posts: 15
Joined: Sun Feb 04, 2018 2:13 pm
Karma: 19
Contact:

Re: Introducing the first ever webDiplomacy Terminator Tournament!

#265 Post by TheGhostmaker » Sun Oct 20, 2019 3:51 pm

Scarabus wrote:
Sun Oct 20, 2019 3:28 pm
That game was Floodgates, not me.
Lol, whoops.

User avatar
Chaqa
Bronze Donator
Bronze Donator
Posts: 13229
Joined: Fri Sep 29, 2017 7:33 pm
Location: Allentown, PA, USA
Karma: 7985
Contact:

Re: Introducing the first ever webDiplomacy Terminator Tournament!

#266 Post by Chaqa » Sun Oct 20, 2019 6:09 pm

The bots have some good plays occasionally, but what I find annoying is their penchant for completely dumbfounding opening moves. Any Austria or Italy who suicide rushes the other is going to doom them both, and the bots love to do this.

Ruined my game with Austria in round 1 and the last (and last ever) bot game as Italy I played a few days ago.

Claesar
Posts: 1965
Joined: Tue Oct 03, 2017 10:34 am
Karma: 1490
Contact:

Re: Introducing the first ever webDiplomacy Terminator Tournament!

#267 Post by Claesar » Sun Oct 20, 2019 6:41 pm

Chaqa wrote:
Sun Oct 20, 2019 6:09 pm
...
Any Austria or Italy who suicide rushes the other is going to doom them both, and the bots love to do this.
...
There's plenty of humans who do the same. That's where they've learned it.
3

mhsmith0
Posts: 318
Joined: Wed Jan 03, 2018 12:35 am
Karma: 186
Contact:

Re: Introducing the first ever webDiplomacy Terminator Tournament!

#268 Post by mhsmith0 » Sun Oct 20, 2019 7:31 pm

It's also really nice to see the bots do this when you rand Turkey (and sometimes Russia) so it's not ALL loss...

gimix
Site Moderator
Site Moderator
Posts: 349
Joined: Tue Nov 13, 2018 8:56 am
Location: My Mountains, the highest in Europe
Karma: 343
Contact:

Re: Introducing the first ever webDiplomacy Terminator Tournament!

#269 Post by gimix » Sun Oct 20, 2019 8:05 pm

Claesar wrote:
Sun Oct 20, 2019 6:41 pm
Chaqa wrote:
Sun Oct 20, 2019 6:09 pm
...
Any Austria or Italy who suicide rushes the other is going to doom them both, and the bots love to do this.
...
There's plenty of humans who do the same. That's where they've learned it.
Were you one of the teachers? :razz:
1

bo_sox48
Site Admin
Site Admin
Posts: 3901
Joined: Thu Jun 08, 2017 1:01 am
Karma: 2785
Contact:

Re: Introducing the first ever webDiplomacy Terminator Tournament!

#270 Post by bo_sox48 » Sun Oct 20, 2019 9:12 pm

The bots definitely have flaws and will never be perfect, so there are probably ways to take advantage of them. Someone who is deeply invested in doing so will without question find these holes and learn to exploit them, so hosting a meaningful tournament playing against them is a little bit dangerous in nature (I don't know why people would do that outside of a tournament setting, but I don't know why people cheat in single player Minecraft either...). That doesn't mean a tournament with the bots can't be fun, which is why I hosted this one anyway and why I hope it will continue into the future. That said, it could definitely use some fine tuning and any criticism is welcome.

To answer a few things:
Scarabus wrote:
Sun Oct 20, 2019 2:02 pm
Will they eventually become "super players", or is there a limit on what/how much they can? I will focus on the areas I think they need to improve, although not really knowing whether the programming can be changed in any way.
No, they're done learning. The bots got their training pre-release, and the second training set of the bots that you all encountered after the first day or two of this tournament also got its training prior to being released. I don't know whether or not there will be any future iterations of the bots that are more "knowledgable" but the nature of the project is such that there probably won't be many changes to them going forward.

Most things that the bots do are representative of what humans have done, so the weak spots tactically are suggestive of human behavior. While the bots were trained on as many games as could be handled in order to widen the breadth of what they "know" to do, that also means that not-so-skilled humans of questionable competence would have also influenced their learning, and thus some behaviors that are more common among said subgroup of humans may also be more common among the bots. I don't know if that explains all of their flaws, but I have seen enough highly irrational behaviors patterned by real people in opening phases and late game scenarios to believe that that's probably where some of their struggles stem from.
TheGhostmaker wrote:
Sun Oct 20, 2019 3:19 pm
Some unsolicited format thoughts, if this is done again (maybe against v2 of the bots?):

I'd like to have more games per round (e.g. 7 per round, and play each country once). 3 makes it too luck based for me. For example, in round 2, Scarabus and RoganJosh had draws as England, which I didn’t have to play; genghiz and xorxes’s Austria games had Italy opening to Tyr or Tri, whereas mine had a Lepanto. Extra games wouldn't even make the tournament take much longer for me, since I played ~5 practice games per country anyway.
I'd like 7 games per round as well and actually planned it out that way when I thought up this tournament. I was then reminded that we set a hard limit of 3 ongoing bot games at one time per person, the reason for this being that we don't want to make actual human games and competition obsolete or do anything that could affect the number of games people want to play against other people. Despite this, future iterations of this tournament should absolutely mandate each player play all seven countries in every round. That would require a couple of preparatory things that we didn't have here, though, the most important of which being time. I only thought of this tournament about 24 hours before the bots were released, and knowing that the Masters was going to be starting around the beginning of November I was not comfortable letting this impromptu tournament run into the time those sign ups happened and games began. That meant that the rounds had to be fairly quick, and since players are limited to three ongoing bot games at a time that would require substantially longer rounds by at least a couple of weeks apiece. That just wasn't feasible here. Regardless of who runs this tournament in the future, I'm going to be a big proponent of longer rounds so that everyone can play seven games in each.

Each player played 6 of the 7 countries, and anyone who needed a tiebreaker to win would have played the 7th. That said, that's one more reason why in a tournament where everything is supposed to be equal it would definitely be better to play all seven countries in each round.
TheGhostmaker wrote:
Sun Oct 20, 2019 3:19 pm
Finally, size of draw felt a little bit arbitrary in the scoring system. Scarabus took a 4-way in this game: http://webdiplomacy.net/board.php?gameID=250817 because Russia was still alive. Maybe scoring of drawn games should be based on counting supply centres, because the bots don't go for eliminations?
For the scoring system to be as effective as it could be, we probably need to do some analytical investigation into bot games to see how impactful center count actually is on bot behavior. If they don't change their behavior significantly when you get closer to a stalemate or a solo, center count is probably not the best way to go about it because one can easily game the bots to pick up every center possible when it becomes apparent that a game won't be a solo. On that same note, draw size is a poor choice for the reason you mentioned. Both center count and draw size are subjected to one pretty major flaw - if a player gets their butt kicked soundly enough, they can draw when it becomes obvious they're going to lose and actually score points, whereas someone who isn't defeated until later on would be prevented from drawing by the bot user with a center lead.
2

KCelec
Posts: 9
Joined: Sat May 18, 2019 8:10 am
Karma: 10
Contact:

Re: Introducing the first ever webDiplomacy Terminator Tournament!

#271 Post by KCelec » Sun Oct 20, 2019 10:55 pm

About non-optimal bot moves, I once encounter a very special thing : only Hold orders.
It was in autumn 04 in this game : http://webdiplomacy.net/board.php?gameID=251334
Maybe the bots learned also civil disorders ?
1

Scarabus
Posts: 37
Joined: Sun Dec 09, 2018 4:27 pm
Karma: 18
Contact:

Re: Introducing the first ever webDiplomacy Terminator Tournament!

#272 Post by Scarabus » Mon Oct 21, 2019 1:28 pm

KCelec wrote:
Sun Oct 20, 2019 10:55 pm
About non-optimal bot moves, I once encounter a very special thing : only Hold orders.
It was in autumn 04 in this game : http://webdiplomacy.net/board.php?gameID=251334
Maybe the bots learned also civil disorders ?
Strange

gimix
Site Moderator
Site Moderator
Posts: 349
Joined: Tue Nov 13, 2018 8:56 am
Location: My Mountains, the highest in Europe
Karma: 343
Contact:

Re: Introducing the first ever webDiplomacy Terminator Tournament!

#273 Post by gimix » Mon Oct 21, 2019 11:07 pm

One thing that may be worth noting... when this tournament was starting we talked about a 50% expected solo rate by the human player.
Now i didn't look at all the single game outcomes, but just by looking at the total scores of all the players i would say the solo rate was a bit under 30%!
And even the 20-something players who made it to the second round, and who had of course a far better rate in their first three games, didn't fare that good (on average) in the second triplet: so that at the end of the six games their mean rate seems to have returned well under the 50% mark
1

User avatar
DougJoe
Posts: 897
Joined: Thu Aug 15, 2019 9:02 pm
Location: Alto, MI, USA
Karma: 208
Contact:

Re: Introducing the first ever webDiplomacy Terminator Tournament!

#274 Post by DougJoe » Wed Aug 26, 2020 1:31 am

Has there been any thought or talk of doing a second one of these?

Claesar
Posts: 1965
Joined: Tue Oct 03, 2017 10:34 am
Karma: 1490
Contact:

Re: Introducing the first ever webDiplomacy Terminator Tournament!

#275 Post by Claesar » Wed Aug 26, 2020 5:21 am

Yes, that has always been Bo's intention. However, he is very busy currently so it seems unlikely to happen right now.
1

Post Reply

Who is online

Users browsing this forum: No registered users and 40 guests