Austria vs Bots

[kestasjk]

#1 development flaw with the bots is designers programming Italy to attack Austria in extremely unrealistic percentage of games. Zero way the bots have ‘learned this’ as no real Italy diplomacy player plays like that….

Yep when the FAIR research ended unfortunately so did the bots. We were a bit spoiled by Philip Paquette who ran his MILA bots himself for us on his system for years. Also when his hardware couldn't process orders fast enough anymore he had a docker image that made it pretty straightforward to get the bots running (even just running on CPU, no GPU needed).

With the FAIR bots it's quite different; their bots are available but the models aren't public. I do have access to them but it's set up to run CICERO on a supercomputer, and needs a powerful graphics card.
I've spent the weekend and took today off work to try and get the FAIR bots running on my desktop machine at home where I have a 2080 GPU; this way I'll have a better understanding of the requirements for the various models and how it'll integrate in, but getting it set up is much more complicated.

I have got it to install and compile, the unit tests have passed, now I'm porting the model data over and will try and spin it up. Fingers crossed.
If it does work I'll get my machine to submit orders to the production site until we host it in the cloud.

[Bladerunners]

Nice job! Austria very hard to play as majority of starts has Italy very very strangely programmed to attack Austria in most games (which is not common diplomacy play / really bad opening normally for Italy)

[kestasjk]

FYI the AvF bot is currently down, going to replace with the DORA configuration over the weekend

[Trigfea63]

FYI the AvF bot is currently down, going to replace with the DORA configuration over the weekend

That's great news, Kestas. I'm already looking forward to my thrashing(s) at DORA's hands!

[kestasjk]

FYI the AvF bot is currently down, going to replace with the DORA configuration over the weekend

That's great news, Kestas. I'm already looking forward to my thrashing(s) at DORA's hands!

Well let's not count our chickens.. but I'm running the new config now, interested to see if you notice a difference

[Trigfea63]

FYI the AvF bot is currently down, going to replace with the DORA configuration over the weekend

That's great news, Kestas. I'm already looking forward to my thrashing(s) at DORA's hands!

Well let's not count our chickens.. but I'm running the new config now, interested to see if you notice a difference

Yes, clearly this is a superior AI. I only played one game, but the difference was unmistakable. For some reason, it still opened with the Russian opening, Albanian variation (TRI->ALB, BUD->RUM, VIE->GAL). Unlike the previous AI, it actually followed through on those opening moves in a more sensible way, to GRE, SEV, and WAR in Autumn 1901. It made another major blunder in 1902, though, sending the SEV army to Turkey instead of north to StP. Those early, unforced errors put it too far behind, and I was able to secure the stalemate lines and win.

https://webdiplomacy.net/board.php?gameID=546552

Except for the weird opening errors, this AI was highly effective throughout the game, similar to "FB" (the old Fairbot from 2021-22). The MILA AIs have difficulty once the game position gets too complex in the mid-game. This AI was tough to outwit. I expect I would have been hard pressed to beat it if it had just played a normal opening.

Also, the speed was great. FB used to take a long time to enter orders during the mid-game, sometimes as long as 15 minutes. This AI entered orders immediately every time. No different from the MILA AIs in terms of speed.

I'll play a few more games and provide more feedback in another week or so.

[kestasjk]

Hmm, interesting. Really like that we can start playing with the bot difficulties now. Will try to see what other FvA options there are for some clues about the opening, worst case scenario I'll reach out to the FAIR team but they're pretty hard to get hold of

[MerryBBuck]

When you know your opponent's opening: https://webdiplomacy.net/board.php?gameID=550660

[Trigfea63]

Looks like you knew more than its opening.

This and other data are leading me to doubt my original conclusion about the new AI. I don't see how a sophisticated AI on the level of DORA gets smoked like that. Or like this:
https://webdiplomacy.net/board.php?game ... #gamePanel
https://webdiplomacy.net/board.php?game ... #gamePanel

Yeah, I know, not as impressive as Merry's. Still, easy wins for the human player. I didn't have to think about my moves at all, and won both games in 1905.

Kestas, Fairbot2's overall record since you made the AI switch is only 15 wins and 20 losses. Plus a bunch of draws (mostly incomplete games). Is it possible for DORA to perform that badly?

[Trigfea63]

Kestas, Fairbot2's overall record since you made the AI switch is only 15 wins and 20 losses. Plus a bunch of draws (mostly incomplete games). Is it possible for DORA to perform that badly?

The DORA AI paper (Bakhtin et al. 2021) says DORA won all 13 games as Austria and 73% as France against top FvA players on webdip. See Table 1 on page 7:
https://arxiv.org/pdf/2110.02924

[ShipOfTheseus]

When you know your opponent's opening: https://webdiplomacy.net/board.php?gameID=550660

Another great example of why we need a better final screen on the Beta edition -- you should be able to screenshot the all-34 blue map.

[kestasjk]

Looks like you knew more than its opening.

This and other data are leading me to doubt my original conclusion about the new AI. I don't see how a sophisticated AI on the level of DORA gets smoked like that. Or like this:
https://webdiplomacy.net/board.php?game ... #gamePanel
https://webdiplomacy.net/board.php?game ... #gamePanel

Yeah, I know, not as impressive as Merry's. Still, easy wins for the human player. I didn't have to think about my moves at all, and won both games in 1905.

Kestas, Fairbot2's overall record since you made the AI switch is only 15 wins and 20 losses. Plus a bunch of draws (mostly incomplete games). Is it possible for DORA to perform that badly?

It's definitely not unlikely that it's a different bot, I just loaded the model that looked like it was probably the FvA DORA model. Definitely need people's feedback to get this right.
Do you think there was any difference at all in the performance before / after?

Also I've got the cicero codebase set up and running and I loaded the DORA config and model using the cicero codebase, assuming that the bots would run the same it was just configuration, but I may need to get the original searchbot repo running. I'll take a closer look into the exact model used in that paper you listed. (I'm hoping I'm not going to find they used some crazy supercluster of GPUs with a crazy set of parameters I can't match)

By the way the FvA bot will be up and down today; I've added code that will hopefully allow a single running bot instance to play as multiple accounts and want to test that, and also want to try and get some functionality so the bots will be more reliable.
I may need to get a server machine for running the bots here, as running these bots under Linux under Windows on my main desktop is always going to be pretty flaky, but I'll see how far I can go with the current setup first

[kestasjk]

Interested to see how you go now; I think I've matched it up to the exact config they referred to in that paper but let's see

[Trigfea63]

[/quote]
Do you think there was any difference at all in the performance before / after?
[/quote]

At first I thought so. The after version seemed improved. But then, after 2 more games, I'm not sure. It's not easy to separate out the poor opening from sub-par play generally. In 1v1, if one player gets ahead by too much, it tends to snowball.

Interested to see how you go now; I think I've matched it up to the exact config they referred to in that paper but let's see

That's the old Fairbot alright! Knows what moves I'm gonna make before I do. I'm not quite dead yet, but it's a struggle to stay afloat

https://webdiplomacy.net/board.php?gameID=558795

[kestasjk]

Interested to see how you go now; I think I've matched it up to the exact config they referred to in that paper but let's see

That's the old Fairbot alright! Knows what moves I'm gonna make before I do. I'm not quite dead yet, but it's a struggle to stay afloat

https://webdiplomacy.net/board.php?gameID=558795

Ah good stuff, that's that sorted.

Since the Searchbot / Cicero / Diplodicus / DORA bots are so resource intensive I've changed the API so that one bot can handle multiple bot accounts.

Here's a game of 7 Cicero's all playing against each other, and the one Cicero bot thinks it's in 7 different games:

Some of the things it says that you see before it gets filtered out it can be quite sassy..
There are so many ways it can be adjusted.. I understand it well enough now that I know we can have a single bot with multiple difficulty levels, press and no-press, carefully filtered or blunt, quick and sloppy or slow and careful, all running from a single bot instance.

Exciting stuff.. I thought getting Cicero/full-press bots running here was years away.

[JECE]

There is something not right about seeing an AI send the message "Nah I'm gonna kill you".

[kestasjk]

There is something not right about seeing an AI send the message "Nah I'm gonna kill you".

Oh trust me.. that made it through the filter The AI says the darndest things that don't make it through. jmo would ban an unfiltered cicero in a heartbeat!

[mongster_03]

Quick question, why is it that every game I play as Turkey, I get Lepantoed, while every game I play as Austria, Italy attacks me? This is starting to feel rigged to give the most difficult possible scenarios.

[DougJoe]

Quick question, why is it that every game I play as Turkey, I get Lepantoed, while every game I play as Austria, Italy attacks me? This is starting to feel rigged to give the most difficult possible scenarios.

It's not. How many games have you played?

[Bladerunners]

He's right re. Italy being "rigged -- the developers mistakenly programmed Italy to attack Austria in greater than 75% of games rather than real life percentages....Italy Bots programmed not to learn from optimal play..