Introducing the first ever webDiplomacy Terminator Tournament!

Post a reply

Confirmation code
Enter the code exactly as it appears. All letters are case insensitive.
Smilies
:points: :-D :eyeroll: :neutral: :nmr: :razz: :raging: :-) ;) :( :sick: :o :? 8-) :x :shock: :lol: :cry: :evil: :?: :smirk: :!:
View more smilies

BBCode is ON
[img] is ON
[flash] is OFF
[url] is OFF
Smilies are ON

Topic review
   

If you wish to attach one or more files enter the details below.

Expand view Topic review: Introducing the first ever webDiplomacy Terminator Tournament!

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by Claesar » Wed Aug 26, 2020 5:21 am

Yes, that has always been Bo's intention. However, he is very busy currently so it seems unlikely to happen right now.

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by DougJoe » Wed Aug 26, 2020 1:31 am

Has there been any thought or talk of doing a second one of these?

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by gimix » Mon Oct 21, 2019 11:07 pm

One thing that may be worth noting... when this tournament was starting we talked about a 50% expected solo rate by the human player.
Now i didn't look at all the single game outcomes, but just by looking at the total scores of all the players i would say the solo rate was a bit under 30%!
And even the 20-something players who made it to the second round, and who had of course a far better rate in their first three games, didn't fare that good (on average) in the second triplet: so that at the end of the six games their mean rate seems to have returned well under the 50% mark

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by Scarabus » Mon Oct 21, 2019 1:28 pm

KCelec wrote:
Sun Oct 20, 2019 10:55 pm
About non-optimal bot moves, I once encounter a very special thing : only Hold orders.
It was in autumn 04 in this game : http://webdiplomacy.net/board.php?gameID=251334
Maybe the bots learned also civil disorders ?
Strange

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by KCelec » Sun Oct 20, 2019 10:55 pm

About non-optimal bot moves, I once encounter a very special thing : only Hold orders.
It was in autumn 04 in this game : http://webdiplomacy.net/board.php?gameID=251334
Maybe the bots learned also civil disorders ?

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by bo_sox48 » Sun Oct 20, 2019 9:12 pm

The bots definitely have flaws and will never be perfect, so there are probably ways to take advantage of them. Someone who is deeply invested in doing so will without question find these holes and learn to exploit them, so hosting a meaningful tournament playing against them is a little bit dangerous in nature (I don't know why people would do that outside of a tournament setting, but I don't know why people cheat in single player Minecraft either...). That doesn't mean a tournament with the bots can't be fun, which is why I hosted this one anyway and why I hope it will continue into the future. That said, it could definitely use some fine tuning and any criticism is welcome.

To answer a few things:
Scarabus wrote:
Sun Oct 20, 2019 2:02 pm
Will they eventually become "super players", or is there a limit on what/how much they can? I will focus on the areas I think they need to improve, although not really knowing whether the programming can be changed in any way.
No, they're done learning. The bots got their training pre-release, and the second training set of the bots that you all encountered after the first day or two of this tournament also got its training prior to being released. I don't know whether or not there will be any future iterations of the bots that are more "knowledgable" but the nature of the project is such that there probably won't be many changes to them going forward.

Most things that the bots do are representative of what humans have done, so the weak spots tactically are suggestive of human behavior. While the bots were trained on as many games as could be handled in order to widen the breadth of what they "know" to do, that also means that not-so-skilled humans of questionable competence would have also influenced their learning, and thus some behaviors that are more common among said subgroup of humans may also be more common among the bots. I don't know if that explains all of their flaws, but I have seen enough highly irrational behaviors patterned by real people in opening phases and late game scenarios to believe that that's probably where some of their struggles stem from.
TheGhostmaker wrote:
Sun Oct 20, 2019 3:19 pm
Some unsolicited format thoughts, if this is done again (maybe against v2 of the bots?):

I'd like to have more games per round (e.g. 7 per round, and play each country once). 3 makes it too luck based for me. For example, in round 2, Scarabus and RoganJosh had draws as England, which I didn’t have to play; genghiz and xorxes’s Austria games had Italy opening to Tyr or Tri, whereas mine had a Lepanto. Extra games wouldn't even make the tournament take much longer for me, since I played ~5 practice games per country anyway.
I'd like 7 games per round as well and actually planned it out that way when I thought up this tournament. I was then reminded that we set a hard limit of 3 ongoing bot games at one time per person, the reason for this being that we don't want to make actual human games and competition obsolete or do anything that could affect the number of games people want to play against other people. Despite this, future iterations of this tournament should absolutely mandate each player play all seven countries in every round. That would require a couple of preparatory things that we didn't have here, though, the most important of which being time. I only thought of this tournament about 24 hours before the bots were released, and knowing that the Masters was going to be starting around the beginning of November I was not comfortable letting this impromptu tournament run into the time those sign ups happened and games began. That meant that the rounds had to be fairly quick, and since players are limited to three ongoing bot games at a time that would require substantially longer rounds by at least a couple of weeks apiece. That just wasn't feasible here. Regardless of who runs this tournament in the future, I'm going to be a big proponent of longer rounds so that everyone can play seven games in each.

Each player played 6 of the 7 countries, and anyone who needed a tiebreaker to win would have played the 7th. That said, that's one more reason why in a tournament where everything is supposed to be equal it would definitely be better to play all seven countries in each round.
TheGhostmaker wrote:
Sun Oct 20, 2019 3:19 pm
Finally, size of draw felt a little bit arbitrary in the scoring system. Scarabus took a 4-way in this game: http://webdiplomacy.net/board.php?gameID=250817 because Russia was still alive. Maybe scoring of drawn games should be based on counting supply centres, because the bots don't go for eliminations?
For the scoring system to be as effective as it could be, we probably need to do some analytical investigation into bot games to see how impactful center count actually is on bot behavior. If they don't change their behavior significantly when you get closer to a stalemate or a solo, center count is probably not the best way to go about it because one can easily game the bots to pick up every center possible when it becomes apparent that a game won't be a solo. On that same note, draw size is a poor choice for the reason you mentioned. Both center count and draw size are subjected to one pretty major flaw - if a player gets their butt kicked soundly enough, they can draw when it becomes obvious they're going to lose and actually score points, whereas someone who isn't defeated until later on would be prevented from drawing by the bot user with a center lead.

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by gimix » Sun Oct 20, 2019 8:05 pm

Claesar wrote:
Sun Oct 20, 2019 6:41 pm
Chaqa wrote:
Sun Oct 20, 2019 6:09 pm
...
Any Austria or Italy who suicide rushes the other is going to doom them both, and the bots love to do this.
...
There's plenty of humans who do the same. That's where they've learned it.
Were you one of the teachers? :razz:

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by mhsmith0 » Sun Oct 20, 2019 7:31 pm

It's also really nice to see the bots do this when you rand Turkey (and sometimes Russia) so it's not ALL loss...

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by Claesar » Sun Oct 20, 2019 6:41 pm

Chaqa wrote:
Sun Oct 20, 2019 6:09 pm
...
Any Austria or Italy who suicide rushes the other is going to doom them both, and the bots love to do this.
...
There's plenty of humans who do the same. That's where they've learned it.

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by Chaqa » Sun Oct 20, 2019 6:09 pm

The bots have some good plays occasionally, but what I find annoying is their penchant for completely dumbfounding opening moves. Any Austria or Italy who suicide rushes the other is going to doom them both, and the bots love to do this.

Ruined my game with Austria in round 1 and the last (and last ever) bot game as Italy I played a few days ago.

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by TheGhostmaker » Sun Oct 20, 2019 3:51 pm

Scarabus wrote:
Sun Oct 20, 2019 3:28 pm
That game was Floodgates, not me.
Lol, whoops.

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by Scarabus » Sun Oct 20, 2019 3:29 pm

But I agree the draw criteria for the bots are a bit arbitrary.

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by Scarabus » Sun Oct 20, 2019 3:28 pm

I didn't have to play Austria in the tournament either.

That game was Floodgates, not me.

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by TheGhostmaker » Sun Oct 20, 2019 3:19 pm

Thanks for running the tournament bo, I had a blast! It was way more fun playing the bots with something on the line. They've got flaws, but the bots are really impressive to me.

Chapeau to Scarabus for the highest total score across both rounds.

Some unsolicited format thoughts, if this is done again (maybe against v2 of the bots?):

I'd like to have more games per round (e.g. 7 per round, and play each country once). 3 makes it too luck based for me. For example, in round 2, Scarabus and RoganJosh had draws as England, which I didn’t have to play; genghiz and xorxes’s Austria games had Italy opening to Tyr or Tri, whereas mine had a Lepanto. Extra games wouldn't even make the tournament take much longer for me, since I played ~5 practice games per country anyway.

In both rounds I think a lot of players were tracking other players' scores. I know I did, particularly in round 1 where I wanted to figure out if a draw was good enough. Now we know what a good score looks like, in round 1 you could use a fixed score cut off (e.g. 100 points per game), so we don't have to check scores.

Finally, size of draw felt a little bit arbitrary in the scoring system. Scarabus took a 4-way in this game: http://webdiplomacy.net/board.php?gameID=250817 because Russia was still alive. Maybe scoring of drawn games should be based on counting supply centres, because the bots don't go for eliminations?

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by Scarabus » Sun Oct 20, 2019 2:02 pm

Ok,my thoughts on the bots:-

- firstly, and in a broad sense, they were chllenging to an extent, and I recognise the previous comments that they are "learning" from moves made in all games. I would be interested in how this works. Will they eventually become "super players", or is there a limit on what/how much they can? I will focus on the areas I think they need to improve, although not really knowing whether the programming can be changed in any way.

-the main area for improvement is their seeming inability to sense when a player is about to solo. I am talking about when one country gets to say 14 centres. It would be natural for most decent players to gang up, even to a small degree. In one of my games, playing Germany, the Turkish bot simply carried on taking french and english centres while I soloed. I wouls expect that the bots should have some level of sense of danger in this respect. Even if allied to a player, this should be overwritten with a desire to get a draw, rather than to keep growing at the expense of losing the game.

-the next area I saw as quite weak was tactically. It seemed too easy to hold a line against a bot, where the bot could easily have broken my lines. They seemed not to recognise when cutting support could give them the edge in taking a centre. Again, this was evident even when I was about to solo. Maybe when calculating the odds of success they fail to understand that cutting support reduces the opposing "support count". Without knowing how they work it out, it is difficult to say why, but this is how I saw it.

-the other area I saw as questionable was their seeming willingness to perpetually bounce, rather than to introduce some level of "randomness". In a few situations, it was clear to me that I could gradually grow, while two or three other countries were effectively static, because their lack of frustration with a position meant their moves were entirely predictable.

These were my main points. I don't intend any of this as a criticism, because I found it to be a fun experience, but I am sure everyone would like to see them improve. I hope so, because it would give more of a challenge, and more of a feel that you are playing against a real opponent. If I think of anything else, I will post more later.

As has already been said, thanks for organising.

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by Floodgates » Sun Oct 20, 2019 3:15 am

Thanks for organising this Bo, it was fun stomping and being stomped by the bots. I wouldn't usually play bot games, so it was nice to have an excuse!

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by bo_sox48 » Sat Oct 19, 2019 2:50 pm

I would love to hear more about what everyone thought. Definitely post it.

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by Scarabus » Sat Oct 19, 2019 6:34 am

Well done all. If the powers that be are interested, I have some observations on the bots' performance.

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by RoganJosh » Sat Oct 19, 2019 12:00 am

Close but no cigar..

Thanks bo_sox for organizing, and congratulation to TGM!

Re: Introducing the first ever webDiplomacy Terminator Tournament!

by tobieee911 » Fri Oct 18, 2019 11:13 pm

bo_sox48 wrote:
Fri Oct 18, 2019 11:10 pm
You're in there now. You got your points.
Thank you! :-D

Top