Reliability Ratings

Members can make suggestions for improving the site and improving the forum as well as submit bug reports to be reviewed by our support team here.
Message
Author
RoganJosh
Posts: 121
Joined: Sun Dec 31, 2017 1:02 am
Location: Stockholm
Karma: 28
Contact:

Re: Reliability Ratings

#21 Post by RoganJosh » Wed Aug 08, 2018 7:21 am

Mercy wrote:
Wed Aug 08, 2018 4:43 am
As I understand it, the discussion was about making RR a better indicator of someones true reliability.
Yes, but the premises of the discussion was that a .66 RR for someone who went in CD 13 times in ~40 games, all within one year, is unfair. I think Swordsman was arguing that this is actually a fair rating. Hence, there is no problem.

RoganJosh
Posts: 121
Joined: Sun Dec 31, 2017 1:02 am
Location: Stockholm
Karma: 28
Contact:

Re: Reliability Ratings

#22 Post by RoganJosh » Wed Aug 08, 2018 7:24 am

That said, the suggestions you quoted are all reasonable. But notice how they focus on the actual problem: newbies joining too many games and then CD'ing in them.

Nephthys
Posts: 155
Joined: Sat Feb 10, 2018 5:47 am
Karma: 7

Re: Reliability Ratings

#23 Post by Nephthys » Wed Aug 08, 2018 8:29 am

Just gonna quickly jump back in here, I'm not saying that my rating is unfair (although I do find it an inaccurate representation of my reliability) I was just saying that I feel as though my (naive) views when I first joined the site are costing me now. Most higher stakes (100+ bet) games are reliability locked and not only these games significantly more interesting and fun and I find that in the case of 95% of these I am unable to join.

What I ideally would like to see (while understanding that new changes take time, of which is limited) is;

1) A way that protects newer users from CDing, I personally really like the lock of games until X number of phases has been played (also a possible deterrent for multi's as it requires a little bit of extra effort).

2) A more clear algorithm for RR. I have no clue what it takes for me to rise by a certain percentage. Surely at the very least this can be made public (as far as I know it isn't, may be wrong though)

Claesar
Moderator
Moderator
Posts: 488
Joined: Tue Oct 03, 2017 10:34 am
Karma: 172

Re: Reliability Ratings

#24 Post by Claesar » Wed Aug 08, 2018 9:19 am

Nephthys wrote:
Wed Aug 08, 2018 8:29 am
...

2) A more clear algorithm for RR. I have no clue what it takes for me to rise by a certain percentage. Surely at the very least this can be made public (as far as I know it isn't, may be wrong though)
I think I've seen the formula before, but I can't remember it. Jmo, Ghug or ATC should be able to provide it, once they read this thread.
1

ChippeRock
Posts: 264
Joined: Thu Oct 19, 2017 5:36 pm
Karma: 71

Re: Reliability Ratings

#25 Post by ChippeRock » Wed Aug 08, 2018 9:38 am

swordsman3003 wrote:
Mon Aug 06, 2018 9:23 pm
In my opinion, after multiple CDs, it make sense that you would have to establish yourself over many games before your rating would change. Even one CD can be a terrible experience for your fellow players. The passage of time does not establish that a player is reliable.
I had 40 some CDs... And after playing 100+ games with a near perfect record I was only able to raise it 10 percentage points. I feel like you should be able to advance higher in RR quicker than this.
1

Foxcastle
Posts: 1345
Joined: Sat Dec 30, 2017 11:48 pm
Location: Night Vale
Karma: 203

Re: Reliability Ratings

#26 Post by Foxcastle » Wed Aug 08, 2018 3:25 pm

Perhaps there should be some sort of rehabilitation program for players who are serious about fixing their RR. Something like: after you've played consistently without an NMR/CD for a period of time (a month? more? X number of games?), you can apply to the mods, and for every X consecutive turns you go without NMR'ing after applying, they'll erase one NRM/CD from your record. It doesn't require any coding, and it's some effort for the mods, but it's not that hard to check, especially if you make the players compile the documentation of their performance.

Claesar
Moderator
Moderator
Posts: 488
Joined: Tue Oct 03, 2017 10:34 am
Karma: 172

Re: Reliability Ratings

#27 Post by Claesar » Wed Aug 08, 2018 4:28 pm

Actually, I'm not sure we can erase NMRs.. If we can't, it would require a lot of coding. Perhaps admins can, though.

swordsman3003
Posts: 58
Joined: Tue Jan 02, 2018 2:51 pm
Karma: 46

Re: Reliability Ratings

#28 Post by swordsman3003 » Wed Aug 08, 2018 6:01 pm

ChippeRock wrote:
Wed Aug 08, 2018 9:38 am
I had 40 some CDs... And after playing 100+ games with a near perfect record I was only able to raise it 10 percentage points. I feel like you should be able to advance higher in RR quicker than this.
You went into CD in almost 1/3 of all the games you have played...40 CDs is a terrifying number. That represents maybe 200+ bad experiences you caused for your fellow players, which is equivalent to a player's-entire-webDip-career's worth of damage (I've been a member of this website for years and have barely played over 300 games).

You dug yourself deep, deep in a hole my friend, I presume because you didn't value your RR for a long time.

But your RR is more than just your CDs; it also accounts for your NMRs. You have NMR'd as recently as games that are still going on right now as I write this post. I think a reliability rating of 72% is more than fair...

To get my RR to 99%, I reached a point where I actually had <1% of my games be CDs (3/300+).
7

PRINCE WILLIAM
Posts: 139
Joined: Sun Dec 31, 2017 11:17 am
Karma: 23

Re: Reliability Ratings

#29 Post by PRINCE WILLIAM » Wed Aug 08, 2018 6:06 pm

Certainly, it is not the same a CD in a live game and the CD in a phased game of a day ie. In a live game an internet malfunction or an unexpected real-life need, from a call to the call of nature, can result in losing two turns. My two CDs were because of calls from clients. It shouldn't weight the same on the RR.
Can someone explain the mechanics of RR? My NMR to the whole is less than 1% and yet my RR is 97%. I had tried many calculation forms to find how it is computed but I failed to solve it.
1

jmo1121109
Developer
Developer
Posts: 354
Joined: Fri Sep 29, 2017 4:20 pm
Karma: 431

Re: Reliability Ratings

#30 Post by jmo1121109 » Thu Aug 09, 2018 4:40 am

The actual query controlling RR is:

Code: Select all

UPDATE wD_Users u 
			SET u.cdCount = (SELECT COUNT(1) FROM wD_CivilDisorders c WHERE c.userID = u.id AND c.forcedByMod=0),
				u.nmrCount = (SELECT COUNT(1) FROM wD_NMRs n WHERE n.userID = u.id),
				u.gameCount = (
					SELECT COUNT(1) 
					FROM wD_Members m
					WHERE m.userID = u.id) + (
					SELECT COUNT(1) 
					FROM wD_CivilDisorders c LEFT JOIN wD_Members m ON c.gameID = m.gameID AND c.userID = m.userID AND c.countryID = m.countryID
					WHERE m.id IS NULL AND c.userID = u.id),
				u.cdTakenCount = (
					SELECT COUNT(1)
					FROM wD_Members ct
					INNER JOIN wD_CivilDisorders c ON c.gameID = ct.gameID AND c.countryID = ct.countryID AND NOT c.userID = ct.userID
					WHERE ct.userID = u.id AND c.turn = (
						SELECT MAX(sc.turn) 
						FROM wD_CivilDisorders sc 
						WHERE sc.gameID = c.gameID AND sc.countryID = c.countryID
					)
				),
				u.reliabilityRating = (POW( (
					(100 * ( 1.0 - ((cast(u.cdCount as signed) + u.deletedCDs) / (u.gameCount+1)) ))
				    +  (100 * (1.0 -   ((u.nmrCount)/(u.phaseCount+1))))
			    )/2 , 3)/10000)"
I'll try and find time this weekend to post the English friendly breakdown but NMR's are weighted to about 1/3 of a CD. CD's and NMR's are each counted. CD's forced by mods are ignored. NMR's and CD's for cancelled games are still counted.

jmo1121109
Developer
Developer
Posts: 354
Joined: Fri Sep 29, 2017 4:20 pm
Karma: 431

Re: Reliability Ratings

#31 Post by jmo1121109 » Thu Aug 09, 2018 4:47 am

Nephthys wrote:
Wed Aug 08, 2018 8:29 am
Surely at the very least this can be made public (as far as I know it isn't, may be wrong though)
As site code is public for anyone curious at https://github.com/kestasjk/webDiplomacy

By searching "NMR" there you can see all the code relating to NMR's. Not super easy if you aren't technical, but it is public. Code contributions are also taken. So if anyone with a little SQL skill wants to take a shot at writing a better formula I'd be happy to review it and get it live.
1

Mercy
Posts: 130
Joined: Thu Oct 19, 2017 4:03 pm
Karma: 68

Re: Reliability Ratings

#32 Post by Mercy » Thu Aug 09, 2018 5:52 am

Mentioning vDiplomacy is really becoming a meme of mine. Here we go again.

If you play at vDiplomacy and hover the mouse above your own reliability rating on your profile, you see the text:

''RR = ((noNMR + noCD)/2)^3"

This means:

"RR = 100 * ( ([# of phases played - # of NMRs]/[# of phases played] + [# of games played - # of CDs]/[# of games played])/2 )^3"
jmo1121109 wrote:
Thu Aug 09, 2018 4:40 am
NMR's are weighted to about 1/3 of a CD.
Really? That is a difference between webdip and vdip then.
1

jollycream
Posts: 3
Joined: Thu Aug 09, 2018 6:33 pm
Karma: 3

Re: Reliability Ratings

#33 Post by jollycream » Thu Aug 09, 2018 9:27 pm

As someone who joined today, I noticed when I had joined my first game that I straight away could join games with a reliability rating of 100%. It feels wrong to say the least, as it allows any newbie to join 5-15 games straight away without having proven any reliability whatsoever.

One way to fix it would perhaps be to have a minimum value on the divisor in the calculation of reliability? That way, newbies cannot come in and ruin too many new games without having proven themselves reliable first.
2

Squigs44
Posts: 1173
Joined: Thu Jul 12, 2018 11:50 pm
Location: America
Karma: 126

Re: Reliability Ratings

#34 Post by Squigs44 » Fri Aug 10, 2018 2:56 am

Yeah the code jmo posted isn't too hard to translate, and it basically looks like what mercy described vdip uses. The only thing that seems different is the +1 in the divisors. If anyone wants to know how many more games they need to boost their RR to a certain percentage, I'll be happy to calculate it for you.

Squigs44
Posts: 1173
Joined: Thu Jul 12, 2018 11:50 pm
Location: America
Karma: 126

Re: Reliability Ratings

#35 Post by Squigs44 » Fri Aug 10, 2018 3:51 am

For example on Nephthys:
*Disclaimer - I don't know what his NMR and phases are so this is just an estimate. It actually looks like his NMR count probably isn't too high compared to his CD's.

To raise your RR to 70% would take about 8 clean games
75% would take about 21 clean games
80% would take about 40 clean games
85% would take about 72 clean games
90% would take about 133 clean games

The way the formula is set up, the cubing power makes it pretty unforgiving with that many CD's to get into the higher (85%+) RRs.
1

jmo1121109
Developer
Developer
Posts: 354
Joined: Fri Sep 29, 2017 4:20 pm
Karma: 431

Re: Reliability Ratings

#36 Post by jmo1121109 » Fri Aug 10, 2018 3:54 am

I'm an admin over on vDip too, Oli made a variety of changes and experiments pretty frequently with new ideas there. He's added a few (like limited the number of games people can join as new members), letting mods give forgiveness points, etc. Eventually some of those will be brought over but a few are too new to see how they work yet. And I'd like to only have to make one round of changes. That said, if anyone comes up with a creative formula change I'd be happy to look into it.
1

bo_sox48
Admin
Admin
Posts: 1144
Joined: Thu Jun 08, 2017 1:01 am
Karma: 344

Re: Reliability Ratings

#37 Post by bo_sox48 » Fri Aug 10, 2018 5:05 am

Feel free to run these changes by the rest of the gang before you make any promises, vDip admin.

Correct me if I'm wrong, but haven't we done some sort of RR forgiveness in the past?

SuperSteve
Posts: 23
Joined: Thu Jan 04, 2018 10:37 pm
Karma: 4

Re: Reliability Ratings

#38 Post by SuperSteve » Fri Aug 10, 2018 5:10 am

I don't care about the "fairness" of ratings. I'm for a strict system that rewards reliability and punishes unreliability.

Still, it would be nice if the system weighed recent results more heavily than results in the distant past. Not for the benefit of the unreliable players but in the interest of accuracy.
1

CptMike
Posts: 217
Joined: Sun Dec 31, 2017 8:12 am
Location: Liège, BE
Karma: 23

Re: Reliability Ratings

#39 Post by CptMike » Fri Aug 10, 2018 5:50 am

A few words about the 1v1 case :

I have been playing a lot of /live/ 1v1 games.

First point : it is really frustrating to see players just dropping the game, often without even leaving a message to inform you. I don't create them any more without requiring an RR of 85-90 %. But this does not prevent apparently fresh players to come (with 100 % RR) and behave that way...

Second point : when you create a live 1v1 or when you register to this, it does not start immediately. And a player "forgets" about this. I don't think it is done purposedly (even if it is very frustrating). I have experienced this a dozen of times and I have done so myself several times. I remember a game with Nealkochhar where we *both* forgot the games.

I'd suggest that the first behaviour is "punished" strongly a way or the other. You could afford to the remaining player (players is standard games) to triple the sentence of the drop of RR due to this ?

The second case on the other side is an accident. Maybe it should be weighted another way even if it is frustrating as well to the other player !

A 'resign' button may offer to the "bad player" a "clear way" to leave a game that is lost. But then I suggest 'resign' should be confirmed (to avoid accidents :cry: ...)
1

ssorenn
Posts: 326
Joined: Thu Oct 19, 2017 4:51 pm
Karma: 53

Re: Reliability Ratings

#40 Post by ssorenn » Wed Aug 22, 2018 7:12 pm

ssorenn wrote:
Wed Aug 08, 2018 12:31 am
RR should not be given to any new player till x amount of game have been finished by the player. Then the aggregate of his moves from the X amount of games is complied giving him a RR score.
This stops newbs from joining games with RR requirement. Too many games, especially gunboat are ruined by new player who don’t know the ethics of finishing what they start. I think that was swordsmans point. Or part of it at least.

I will say for the record that i committed these transgressions when i first joined. This was the first real competitive site i ever joined online. But as a new player, i thought”it’s just a stupid game, who cares”. Only later to figure out it really matters to a lot of people.

So, having to earn a score may be better, and some type of testimonial when you join, explaining that these games take a long amounts time and people take them seriously.

What the X number is , up for debate.
instead of creating the algo to fit X in, why not just create another peramiter in the game creation field that allows for some to determine that X for themselves

ex. RR85%
X#games on site >20

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest