webDiplomacy.net new server

[kestasjk]

Hi all,

We have moved to a new server; the last one was several years old and was struggling with new features and the database size. We also are hoping to be able to add exciting features which require more server resources, make use of modern container-based deployment to encourage new developers, and to integrate the new AI bots into the system.

But as with any major change there are bound to be a few problems, so if anything seems to have broken please let us know.


A good time to recognise those who have contributed to webDiplomacy.net; whether donating to keep the server up and free of obnoxious ads, making improvements to the software, or patiently keeping cheaters out and answering support requests: 15 years is a long time in web-years and I think we’ve exposed a lot of new people to the hobby.

Regards,
Kestas

Edit: Looks like there was an issue with map rendering but no other errors. I've resolved that and turned game processing back on.

[jmo1121109]

Thanks for the upgrade Kestas!

[cdngooner]

Seems extra slow loading lately, compared to its old self and to other websites. Something with the new system?

[kestasjk]

Thanks for the info. When you say lately how lately are you talking about? Is there anything in particular that is repeatably slow? Does it vary throughout the day or is it fairly consistently slow?

I've found it's a fairly common issue people have had that queries that used to run fast on older MySQL versions have become slower after upgrades. A query might work fine, then there's an optimizer change and it performs terribly.


We have identified a few issues with variant maps and some other bits and pieces; will post here as I go.

[Gowron]

No pages would load about 20 or 30 minutes ago.

[cdngooner]

Its been hit or miss for about 2 days now.

[kestasjk]

A while ago while tuning the database for the new system I did accidentally take the database down for a few minutes, so I think that's all that was.

There are logs whenever there is a failure, and I have also enabled logging for slow queries. I made some config changes to make the system use the increased memory space and disk speed, plus a few optimizer recommendations.

Let me know if things remain sluggish. It would be especially helpful if you could record the time, the page you viewed, and copy and paste the data at the bottom of the page that looks like this:

Code: Select all

Rendered in: 0.01sec - Data retrievals: 112 - Data insertions: 9 - Memory used: 0.545MB

Logged on: 77 - Playing: 1292 - Registered: 133889 - Pages served: 310854973
Starting games: 38 - Joinable games: 0 - Active games: 514 - Finished games: 168188

webDiplomacy version 1.66
Enable Desktop Mode

(Note you don't see this at the bottom of the forum, only non-forum pages)

The render times etc and the query numbers, plus the time, will let me look up whether there were any slow queries going on at the time.

We are still experiencing a few deadlocks every now and then (every time an error occurs it is logged, so we should at least know if things break), but a lot less regularly than on the old server.

[Gowron]

All pages would not load at all 12 minutes ago.

[kestasjk]

I just sorted the last known issue, and had to wipe the map/order cache as maps/orders were being generated with an inconsistent database state causing javascript errors. Might be a bit slow while it regenerates.

Keep letting me know when you experience issues; looking at the logs I had rebooted the database to apply a config change at that time. Let's wait and see how we go from here forward; I'm collecting any errors or slow queries that occur, and all the causes of previous issues are resolved.


I expect we'll need to do some more tuning of the database. It looks like MySQL's internals have changed quite a lot in the latest couple of versions, so that queries that used to function well might now be locking huge sections of the database and causing deadlocks etc.

[Wusti]

Can you do us a favour? Can you give a longer recovery time for order placement when you put it back up? Some games got hit by missed turns when it came back up. Cheers.

[kestasjk]

Yep fair point; the downtime was 3-4 hours and I think I added 12? Should have done 24.

[teccles]

You may well already be aware, but just in case: the website seems to intermittently very slow. Maybe once or twice a day since the server move, I've been experiencing very long load times (maybe a minute?), on an otherwise good internet connection.

[Chaqa]

You may well already be aware, but just in case: the website seems to intermittently very slow. Maybe once or twice a day since the server move, I've been experiencing very long load times (maybe a minute?), on an otherwise good internet connection.

Same issue. I know Eden has it as wwell, site seems to go down sometimes

[jmo1121109]

You may well already be aware, but just in case: the website seems to intermittently very slow. Maybe once or twice a day since the server move, I've been experiencing very long load times (maybe a minute?), on an otherwise good internet connection.

Same issue. I know Eden has it as wwell, site seems to go down sometimes

Yes, I am also having this issue and not seeing any direct errors tied to it when I've experienced it.

[Changez]

Hey Mods, my game against bots is totally stuck and gives me this message at the top of the screen:

"The last process time was over 12 minutes ago (at Saturday, 19-Jun-21 16:39:06 UTC UTC+0); the server is not processing games until the cause is found and games are given extra time."

Don't know if this is related or if it's an update of some kind.

Thanks

[kestasjk]

Apologies I've restarted processing. Clearly need to dedicate some time to this locking issue

[jmo1121109]

Has anyone ran into the issue of the site being slow to load who is NOT on dark mode? I noticed that part of the delay was a missing css file in the KnownWorld variant and created a blank file, and I immediately stopped seeing the delayed load times. If you run into this again please let me know here.

[teccles]

I'm afraid I just had another slow load time (several minutes).

I am using dark mode.

[jmo1121109]

I'm afraid I just had another slow load time (several minutes).

I am using dark mode.

Was it the home page or something else on the site?

[Yonni]

I believe it was a game and the home page for me but definitely the home page. Not using dark mode.