Reliability Ratings

Members can make suggestions for improving the site and improving the forum as well as submit bug reports to be reviewed by our support team here.
Message
Author
Nephthys
Posts: 2908
Joined: Sat Feb 10, 2018 5:47 am
Location: Not a GM
Karma: 842
Contact:

Reliability Ratings

#1 Post by Nephthys » Mon Aug 06, 2018 8:21 pm

Hey all,

Just thinking about the Reliability Rating system as something that could potentially be improved and curious to hear peoples thoughts.

When I first joined this site I seriously underestimated the time it would require and as such was unable to continue due to work commitments (resulting in multiple CDs). Many (9ish?) months later I had the time and decided to re-join. Since then I have missed two turns and taken over 28 Civil Disorders, yet my RR remains horribly low (66%).

Would it be a improvement/possible to have the RR only take in the last year as a more accurate picture or have it reset after a defined period of inactivity?
1

swordsman3003
Gold Donator
Gold Donator
Posts: 264
Joined: Tue Jan 02, 2018 2:51 pm
Location: Texas
Karma: 607
Contact:

Re: Reliability Ratings

#2 Post by swordsman3003 » Mon Aug 06, 2018 9:23 pm

In my opinion, after multiple CDs, it make sense that you would have to establish yourself over many games before your rating would change. Even one CD can be a terrible experience for your fellow players. The passage of time does not establish that a player is reliable.
2

Octavious
Posts: 3844
Joined: Fri Sep 29, 2017 4:16 pm
Location: The Five Valleys, Gloucestershire
Karma: 2605
Contact:

Re: Reliability Ratings

#3 Post by Octavious » Mon Aug 06, 2018 9:38 pm

Personally I reckon your last ten games is a far more reliable indicator of future performance than what you did when you first joined and only had a vague idea of what was going on. If time and effort doesn't establish that a formerly awful member has become reliable, then what on earth does?
4

User avatar
ghug
Bronze Donator
Bronze Donator
Posts: 17581
Joined: Mon Mar 20, 2017 3:51 pm
Location: Seattle
Karma: 11400
Contact:

Re: Reliability Ratings

#4 Post by ghug » Mon Aug 06, 2018 9:41 pm

The issue is that ten games can be a week for someone or a year for someone else. In the latter case, it's meaningful. In the former, not so much. On the flip side, if I leave the site and play no games for a year, that doesn't suddenly make me reliable.

That's not an insurmountable problem, but we tend to have minimal time for development.
4

ziran
Posts: 211
Joined: Sun Dec 31, 2017 3:21 am
Karma: 75
Contact:

Re: Reliability Ratings

#5 Post by ziran » Tue Aug 07, 2018 3:08 am

i think a system in which the first ten or so games or the first six months are dropped wouldn't be too radically different, but still help those who didn't understand the consequences of NMRs and CDs.
1

Nephthys
Posts: 2908
Joined: Sat Feb 10, 2018 5:47 am
Location: Not a GM
Karma: 842
Contact:

Re: Reliability Ratings

#6 Post by Nephthys » Tue Aug 07, 2018 4:19 am

As a response and to add some more thoughts;

I like ziran's idea of the first few games not counting, I was only aware that people could take over game, and so, wrongly assumed that it wouldn't lead to much disruption.

For the other points, regardless of how long a game is does it not require, to some degree, the same level of commitment to finish a game, regardless of phase length. While I think that 10 games may be too few I think that the most recent 25-30 games would be sufficient to show whether someone tends to be reliable or not while being slightly more forgiving that the current system.

Claesar
Posts: 1965
Joined: Tue Oct 03, 2017 10:34 am
Karma: 1490
Contact:

Re: Reliability Ratings

#7 Post by Claesar » Tue Aug 07, 2018 5:57 am

On a sidenote, 1v1s help to boost your RR as you can get a lot of turns done much faster. With time and effort, you can get back to a high RR.

RoganJosh
Silver Donator
Silver Donator
Posts: 556
Joined: Sun Dec 31, 2017 1:02 am
Location: Stockholm
Karma: 464
Contact:

Re: Reliability Ratings

#8 Post by RoganJosh » Tue Aug 07, 2018 7:25 am

Nephthys went into CD in 13 games. Which is about 25% of all games he played. I don't see the problem here, a reliability rating of 66% sounds about accurate to me.
1

Octavious
Posts: 3844
Joined: Fri Sep 29, 2017 4:16 pm
Location: The Five Valleys, Gloucestershire
Karma: 2605
Contact:

Re: Reliability Ratings

#9 Post by Octavious » Tue Aug 07, 2018 8:34 am

Claesar wrote:
Tue Aug 07, 2018 5:57 am
On a sidenote, 1v1s help to boost your RR as you can get a lot of turns done much faster. With time and effort, you can get back to a high RR.
On that point, if a dismal reliability rating can be masked by spending a couple of evenings playing draughts, is there any point to having it at all?
3

Claesar
Posts: 1965
Joined: Tue Oct 03, 2017 10:34 am
Karma: 1490
Contact:

Re: Reliability Ratings

#10 Post by Claesar » Tue Aug 07, 2018 9:56 am

Octavious wrote:
Tue Aug 07, 2018 8:34 am
Claesar wrote:
Tue Aug 07, 2018 5:57 am
On a sidenote, 1v1s help to boost your RR as you can get a lot of turns done much faster. With time and effort, you can get back to a high RR.
On that point, if a dismal reliability rating can be masked by spending a couple of evenings playing draughts, is there any point to having it at all?
There are several arguments to make for possible improvements on the RR system (which are not on top of the update list, so do't hold your breath). Of course even in its current form, it provides an indication of reliability and gives some measure of the likelihood a player will submit moves rather than NMR.
Furthermore, you can always get more detailed statistics from their profile if you wish to investigate a certain player.
So yes, it is useful.

Octavious
Posts: 3844
Joined: Fri Sep 29, 2017 4:16 pm
Location: The Five Valleys, Gloucestershire
Karma: 2605
Contact:

Re: Reliability Ratings

#11 Post by Octavious » Tue Aug 07, 2018 10:33 am

If you'll forgive me for going on about it, I feel that there is an element of bias at play here. The site leadership is, by its nature, made up of established and reliable players of the game. The drawbacks of RR are therefore something that are seldom or never experienced, whilst the benefits are enjoyed.

The reality for many players, such as Nephthys who after a disaster of a start appears to be trying to do things right, is that RR is an absolute pain in the arse that greatly reduces their site experience. I can't help but feel that, if the site leadership had more first-hand experience of the negative side of RR, doing something about it would be a lot closer to the top of the list.

swordsman3003
Gold Donator
Gold Donator
Posts: 264
Joined: Tue Jan 02, 2018 2:51 pm
Location: Texas
Karma: 607
Contact:

Re: Reliability Ratings

#12 Post by swordsman3003 » Tue Aug 07, 2018 1:22 pm

All this talk about how hard it is to join games with a low reliability rating or to restore reliability rating once it is lowered, but what about the perspective of the victims of a civil disorder? Let me advocate for them.

The experience of multiple players (everyone else in the match) is damaged or ruined by a CD. Someone with 10 to 15 CDs, assuming these are merely classic games (and not ones with even more players) has produced something like 60-90 bad experiences for other players on the website.

You say you didn't know it was wrong? So what? How does that help the players whose experiences you damaged?

Online diplomacy games often take 2-6 months to finish, and many players play very few games (like less than 10 in a year). 10-15 CDs can cause massive damage to the experience of other players, especially those who have played very few games.

One swift, automatic way that a player can learn that going into CD is unsportsmanlike is to find out that that other players can block them from joining their new games. That induces someone to find out why they don't have reliability rating, and to stop causing CDs in their games if they want join other players' games.

The simple passage of time does not mean a player has become reliable, and resetting reliability rating every so often undermines the purpose of such a system. The effects of the games ruined by a CD can never be undone, so why should it be trivial and automatic to get a clean slate?

Nephthys has a reliability rating of 66% after going into civil disorder in 13 of the 37 games he's played on the website. That actually sounds pretty generous to me.
3

Claesar
Posts: 1965
Joined: Tue Oct 03, 2017 10:34 am
Karma: 1490
Contact:

Re: Reliability Ratings

#13 Post by Claesar » Tue Aug 07, 2018 1:46 pm

You raise a good point, Octavious, but the #1 grievance among members (as far as I can tell) is NMRs. Therefore I appreciate measures that try to counter this.

I too had a low RR at some point (I think 70%) because I lost internet connection for a weekend, but I've recovered by playing a lot.

Yonni
Silver Donator
Silver Donator
Posts: 408
Joined: Thu Oct 19, 2017 6:55 pm
Karma: 368
Contact:

Re: Reliability Ratings

#14 Post by Yonni » Tue Aug 07, 2018 2:35 pm

@Naph, I'm guessing that you're testing the outer limits of the RR as you have a really high ratio of CDs to completed games. I wonder if someone with knowledge of the inner workings of the RR can shed some light on how many more games he needs to play to get to a 70%, 80%, or 90% RR.

10? That might be reasonable. 100? That's probably too much.

Also, are live games rated differently? Needing to log in every day or two to enter orders for a non-live game is a better indication of commitment, imho.

breaca
Gold Donator
Gold Donator
Posts: 57
Joined: Tue Jan 02, 2018 5:48 am
Karma: 26
Contact:

Re: Reliability Ratings

#15 Post by breaca » Tue Aug 07, 2018 6:59 pm

swordsman3003 wrote:
Tue Aug 07, 2018 1:22 pm
All this talk about how hard it is to join games with a low reliability rating or to restore reliability rating once it is lowered, but what about the perspective of the victims of a civil disorder? Let me advocate for them.

The experience of multiple players (everyone else in the match) is damaged or ruined by a CD. Someone with 10 to 15 CDs, assuming these are merely classic games (and not ones with even more players) has produced something like 60-90 bad experiences for other players on the website.

You say you didn't know it was wrong? So what? How does that help the players whose experiences you damaged?

Online diplomacy games often take 2-6 months to finish, and many players play very few games (like less than 10 in a year). 10-15 CDs can cause massive damage to the experience of other players, especially those who have played very few games.

One swift, automatic way that a player can learn that going into CD is unsportsmanlike is to find out that that other players can block them from joining their new games. That induces someone to find out why they don't have reliability rating, and to stop causing CDs in their games if they want join other players' games.

The simple passage of time does not mean a player has become reliable, and resetting reliability rating every so often undermines the purpose of such a system. The effects of the games ruined by a CD can never be undone, so why should it be trivial and automatic to get a clean slate?

Nephthys has a reliability rating of 66% after going into civil disorder in 13 of the 37 games he's played on the website. That actually sounds pretty generous to me.
I agree. I'd like to see the reliability rating be the metric displayed next to each players name (rather than points). To me it is a more significant measure.
3

Nodnyl
Silver Donator
Silver Donator
Posts: 5
Joined: Sun Dec 31, 2017 11:56 am
Karma: 7
Contact:

Re: Reliability Ratings

#16 Post by Nodnyl » Tue Aug 07, 2018 8:51 pm

As the initial loss of RR was due to underestimating the commitment required to play in multiple games as a new player, maybe there should be some restriction on the number of games a new player may join. This would reduce the number of spoilt games which in my experience are most frustrating.

Claesar
Posts: 1965
Joined: Tue Oct 03, 2017 10:34 am
Karma: 1490
Contact:

Re: Reliability Ratings

#17 Post by Claesar » Tue Aug 07, 2018 9:23 pm

Nodnyl wrote:
Tue Aug 07, 2018 8:51 pm
As the initial loss of RR was due to underestimating the commitment required to play in multiple games as a new player, maybe there should be some restriction on the number of games a new player may join. This would reduce the number of spoilt games which in my experience are most frustrating.
Technically this restriction is in place, as you have only 100 points.. But perhaps we should impose more restrictions, yes.
1

celaph
Posts: 2144
Joined: Sun Dec 31, 2017 8:41 pm
Location: Bay Area, CA, USA
Karma: 1141
Contact:

Re: Reliability Ratings

#18 Post by celaph » Tue Aug 07, 2018 11:25 pm

Claesar wrote:
Tue Aug 07, 2018 9:23 pm
Nodnyl wrote:
Tue Aug 07, 2018 8:51 pm
As the initial loss of RR was due to underestimating the commitment required to play in multiple games as a new player, maybe there should be some restriction on the number of games a new player may join. This would reduce the number of spoilt games which in my experience are most frustrating.
Technically this restriction is in place, as you have only 100 points.. But perhaps we should impose more restrictions, yes.
But there are many games that have 10-15 buy in. I know when I first joined WebDip I joined too many games and could not really play well in any of them as my time was stretched too thin (it hurt my GR more than my RR, but who really cares). I would support additional restrictions.

ssorenn
Gold Donator
Gold Donator
Posts: 469
Joined: Thu Oct 19, 2017 4:51 pm
Karma: 109
Contact:

Re: Reliability Ratings

#19 Post by ssorenn » Wed Aug 08, 2018 12:31 am

RR should not be given to any new player till x amount of game have been finished by the player. Then the aggregate of his moves from the X amount of games is complied giving him a RR score.
This stops newbs from joining games with RR requirement. Too many games, especially gunboat are ruined by new player who don’t know the ethics of finishing what they start. I think that was swordsmans point. Or part of it at least.

I will say for the record that i committed these transgressions when i first joined. This was the first real competitive site i ever joined online. But as a new player, i thought”it’s just a stupid game, who cares”. Only later to figure out it really matters to a lot of people.

So, having to earn a score may be better, and some type of testimonial when you join, explaining that these games take a long amounts time and people take them seriously.

What the X number is , up for debate.
1

Mercy
Posts: 257
Joined: Thu Oct 19, 2017 4:03 pm
Karma: 220
Contact:

Re: Reliability Ratings

#20 Post by Mercy » Wed Aug 08, 2018 4:43 am

Swordsman, I don't think that anyone was arguing to make it easier for unreliable people to join games. As I understand it, the discussion was about making RR a better indicator of someones true reliability.

Under the current system, someone who consistently NMRs a few times in every game he plays can have the same RR as someone who had a couple of CD's when he started playing, but who has played 10 games in the past year and never NMRd in them. Since the latter is more reliable, you want the latter to have a higher RR in comparison to the former and make it easier for him to join games in comparison to the former.

The only reason I can think of for not wanting that is if you are carrying grudges against players who ruined your games, even if they did so unknowingly, and I see that as a bad reason. Can't we have some mercy?


I saw a couple of suggestions:
Nodnyl wrote:
Tue Aug 07, 2018 8:51 pm
maybe there should be some restriction on the number of games a new player may join.
This is already implemented on vDiplomacy. New players can join two games at a maximum. After they have completed a small number of phases (19 or so, I believe) they can join more games (though they get an RR rating at that point, so they may still be restricted if they missed phases).
ssorenn wrote:
Wed Aug 08, 2018 12:31 am
RR should not be given to any new player till x amount of game have been finished by the player. Then the aggregate of his moves from the X amount of games is complied giving him a RR score.
This stops newbs from joining games with RR requirement.
VDiplomacy always has the answer. Not only can you have RR restrictions on your game, you can also have restrictions on the minimum number of phases played. This is needed though, because brand new players can join games with RR restrictions.
1

Post Reply

Who is online

Users browsing this forum: mOctave and 77 guests