@zultar: The problem is, it doesn't measure approximated true skills with respect to the level you've played at. What it measures is a combination of approximated true skills and number of games played, so that for the typical numbers of games played for the regulars on the site, the latter is the more important of the two.

To put it another way: imagine two players, one of whom plays gunboat against average webDip players 100 times with 20 solos, 40 draws, and 40 losses, and another of whom plays gunboat against average webDip players 20 times with 12 solos, 4 draws, and 4 losses. The former will be the higher GR player. Which one would you rather draw as your opponent in a tournament game?

@cs "longevity."

I don't understand that at all. Can you explain it to me with an example of how a player who plays longer has an advantage over an equally skilled player who hasn't played longer - that can't be accounted for by sampling density (e.g. you could make the argument my GR is higher than my true skill should indicate because there are too few samples to be sure - I could have gotten lucky)?

I mean of course it's impossible for a guy who's only played 4 games to have a GR of 800, is that all you mean? In that sense longevity helps, but as soon as your rating is on the order of where your true skill would predict, that is no longer a handicap.


@Zully

too much stress. Those last few games made me way too upset while playing to be justifiable on real life human terms. Something about the game makes me take it WAAAY to seriously. I'm tempted to switch to gunboat - even though I'll suck at it, I won't care that I suck at it.

Ahh OK CS, I get it now. Yes, that makes sense. I tend to look at the effectiveness of systems as n -> big.

Still, I think it's unfair to regard that as a reward towards longevity, but rather it should be viewed as a penalty toward new players. The "reward" experiences diminishing returns as you approach your natural place on the scale.

@goldie

I read the press again. I was played, that's what happened :)

@YJ: The problem is that for, say, a gunboat player winning 20% of the time, drawing 40% of the time, and losing 40% of the time, at random, GR will take something in the neighborhood of 400 games to converge to the correct value, and until you have converged, it will be lower than the correct value such that the fewer games you have played, the more you will be underrated.

In other words, if we have identical skills, and I play 100 games and you play 200 games, you will be rated higher than I will be. (Unless we're sub-100 players, in which case I will be rated higher)

One way that you can see this: if GRs were generally indicative of skill, then if I look at experienced players, it should be equally likely that their GR goes up or down in a given month. However, in practice you will find that a player rated over 100 is more likely to go up than down. To put it simply: if you look at the top 50 players in GR this month, I will be willing to bet you that more of them will have their GR go up than down comparing March and April. Would you be willing to take the other side of that bet?

@YJ - haha we both were

No, no I do follow the argument, CS. But again, I don't really have a problem with penalizing new players. That is a different phenomenon than rewarding longevity, and I think it's important to make the distinction.

If one player has played 1000 games, and another player has played 2000, neither has a longevity advantage, and I guess that's all I care about. Admittedly, maybe I'm oversimplifying the implications for practical, everyday, use. By the way, isn't there a scaling factor in the formula that could be tweaked so that you'd get more fluctuation in GR but also reduce this penalty? Could it be we're not just at the optimal point?

But you also hinted that it would be easy to design a metric that didn't have this flaw? Can you illuminate a bit? Alternatives that I've come across seem inferior to me because they don't reward appropriately for relative strength of competition.

@YJ: "Ahh OK CS, I get it now. Yes, that makes sense. I tend to look at the effectiveness of systems as n -> big."

That's a good way to look at it as long as most players are in the limit where n -> big. The problem is that on webDip, most players are not in that limit, hence the reward for longevity. If we wanted to measure skill, we still could -- what most systems do is to calculate two numbers:

(1) Just based upon the information given so far, what rating represents the skill displayed?

(2) What is the uncertainty in that measurement (this will be much higher for players with fewer games/fewer recent games)?

It would then be up to you to decide how to report the results. For example, consider:

Player A: GR of 440 +/- 130, only played a few games
Player B: GR of 400 +/- 10, played a lot of games

You might think Player B is actually stronger, based upon your experience that most players with a GR of 440 +/- 130 end up actually being overrated. Some systems set a threshold where if the uncertainty is too high, the rating is not reported, while others report it but sort by one SD below the actual rating (meaning, 390 vs. 310).

The current GR system is also misguided in terms of how it deals with new data. If a player has only played 3 games before, one new game is much more useful in forming an impression of their skill than if they have already played 300. However, the current GR formula treats both as equally useful data.

So, like I said, it's designed to measure a combination of skill and longevity. If the goal is to design something purely measuring skill, meaning something predictive (the strongest possible correlation between this month's rating set and the results of games played in the next month), it would be easy for me to design something more predictive by not giving a longevity bonus. This is also the reason that I don't go around trying to get a strong board by advertising for games with high GR players.

I'll let you catch up to my last post before I reply. I understand completely your mean +- SD argument - I'm fairly competent with probabilities, distributions, and sampling.

I really feel like calling it a "bonus" isn't quite reflective of the truth, and I'm not sure how you would eradicate the penalty on newer players with any predictive system (though I am hardly an expert at this). It seems like any system will be introducing inaccuracies while n is small. You could make it so that it's more of an uncertainty than a straight up penalty, but I'm not certain that's hugely any better.

OK, letting you catch up, for real :)

Ok, since I'm a big nerd, I actually went through the current GR ranking. We have about 52000 users, and so I used the 26000th user as a marker (who joined in may of 2010, almost a year before I did) and I calculated the different in the current rating minus the peak rating for the top 40 current players. So a negative score means that their current rating is lower by that much relative to their peak rating.

Here's what I have so far:

MM: -315
Barn3tt: -16
anlari: -26
uclabb: -5
jmo: 0
THM: -28
LanGaidin: -57
Tusky: 0
Hellenic: -92
djbent: -197
Azogar: 0
Octavius: -20
sweetwatersam: -125
Indy: -32
MKE: 0
FloatingLakes: -49
TomB: -55
rdrivera: -166
Alderian: -7
Laoban: -103
opripom: -4
AsianKnight: -16
Ethanol: 0
Troodonte: -78

So, for the top current 40 people, 24 of them joined before may 2010), and the average difference in current-peak GR is -58 with 8 people whose loss is than 10 rating. Out of those eight, I can attest that uclabb, jmo, THM, and MKE are all excellent players.

So, what do people think now?

@YJ: I'm caught up now. Yes, I think I could write something which would solve the problem. Reducing the convergence time in the current GR formula is definitely going to improve it, but still has the flaw that it's treating one game as equally important when you've played 3 or 300 games, and that's unreasonable.

Let me try and separate the GR algorithm into three pieces, because I think it does at least one and possibly two of them well. There are three components to the system:

1) There is a decision on how to treat the result of a game. For example, that in a WTA game, a 4-way draw is worth 1/4 of a win while a survive, defeat, and resign are all worth 0 of a win. But, in a game which is not WTA, that a 16-center defeat is better than a 3-center, 5-way draw.

This design decision seems reasonable, and I see no reason to change it.

2) The decision to have GR reflect the average "share" of the pot, meaning that if player A has 2x the GR of player B, they should, in an average game, get twice the "result".

It's not actually clear to me that this it's possible for this to converge. Most systems instead say something like "if A is rated 100 higher than B, then A will get a better result than B 2/3 of the time". So, this might need to be changed. But, for the moment, let's presume that it actually *is* possible to converge. Note that no matter how this system works, it will be the case that playing better opposition means that the result needed to maintain your rating will be lower, which is the property you want.

3) The design decision to let each game be worth the same amount of information, regardless of how much information already exists, and to have a very slow convergence.

This is the one that needs changing.

What I think I'd try first if I designed a rating set, given that TrueSkill appears to be patented, is adapt a version of Glicko (http://www.freechess.org/Help/HelpFiles/glicko.html). If I can get my hands on a set of game results, I'd be happy to take a shot at it and let you know the result.

@zultar- It is pretty hard to get meaningful information from the stats you posted, just because by definition people's current scores are less than their peak GR, and it is hard to say what is a big difference. What I would suggest that you do if you want to get a slightly more meaningful result is compare the GRs of players who were in the top 40 a year or two with their GRs now. In this case, we might expect that players' GRs might have slightly increased because GR converges from below, but we also might expect it to decrease since there might be some selection bias of the top 40 in GR being especially lucky recently. Thus, we might reasonably expect the change to be about zero and a deviation from that in either direction could be evidence for your claim.

It's easier to pass the buck on to someone else to do the work, uclabb.
It would be great if you could do this yourself and present the result. I did what I could in the limited time that I have while discussing peripheral issues with others.

I do think your idea is valid. Could you do a march 2013 comparison to march 2012?

So how many games does it take to get to the point you there is no longevity bias? Or does it go on infinitely?

I'm just trying to help, zultar.

Actually, after thinking about it some more, I think you do want to compare someone's recent rating as opposed to a year or two ago with their overall rating because the claim has to do with how they would fare NOW against the current crop. If you compare GR from a year ago, that will give you information about how they did in the field a year ago. It would probably be a better system to calculate the AVERAGE of the top 40 players for their last 3-6 months and compare that with their peak rating.

Would you agree that:
If the difference is small, say less than 5-10% or by a fixed amount of 10-20 , then you can say that they fare just as well in any given time frame.
If the difference is big, say more than 15-20% or by 50 , then I think it's fair to say that the they fare better in the past than currently (which for an individual you can say that they might have ran into a string of bad luck, but for a group of people, it's probably more likely that there is a significant difference in the current crop and the previous one with a negative value indicating that the current top echelon is better than the previous one.
Is that fair?

Can we get some volunteers to calculate the average rating over a period of 3-6 months of the current top 40 minus their peak rating?

I think I would agree with that to a degree. To use PE as an example (because I think you might have), PE's GR went through the roof at one point (I think it was about a year ago) because he is an unbelievable live game player (I think he is #1 in live game GR but I didn't check) and he had a stretch where he played a bunch of live games. Perhaps you could argue that live games were worse back then than they are now (and I think you would be right), but the real reason why his peak GR is so high is because he had that one great stretch of results. I imagine that pulling peak GR does a lot of things like that:

For another example, it used to be the case that my peak GR came from after I soloed a World Cup game in which all the players needed a specific result in order to move on to the next round. I was able to play that to my advantage to turn what probably should have been a draw into a solo. I have since passed that GR mark, but the point is that I imagine that a lot of peak GRs come from a specific lucky streak.

Having said that, I do agree with you that obviously large changes or obviously large changes in your framework would be meaningful. I just think it is not clear what it obviously large or obviously small a priori.

And just to clarify what I was saying: I was saying to take the top 40 people x years ago, then see which of those players are still playing (maybe there are 24) and then compare the GR of those players in the two time frames. I think maybe you are right that this isn't the best solution either, because skill isn't constant over time. It is also confusing because GR is in some sense about relative skill and not absolute skill. I will think about it.

I've played on webDip for a little while now and I'm not sure I'd say the "current upper echelon" is all that much better than the "old upper echelon", but there is no question the two groups are different. Take a top player from a few years back and throw him in a game with six current top players and the result will be pretty similar to if you took one current top player and put him in a game with six top players from a few years ago.

As far as the GR analysis, I'm not sure what you'll be able to take away from the efforts. GR can be inflated by playing weaker competition, since raising your GR significantly is highly dependent on solos, and the better competition you play the fewer solos there will be, which is just as true now as it was a few years back.

All that said, I think there are more quality players around these days, but probably because there are just more players in general. If the pool to choose from is twice as big now, and you just want to compare the top 40 guys (rather than the top 1%), then I'd say, yeah, the top 40 now are very likely better than the too 40 back then, but I'd argue the top 1% from now and then are pretty similar.

I think I see your point, uclabb.

I do want to make an addendum to what I was proposing:
Calculate the average rating for the last 6 months (if only 3 exist, then calculate only for the three). IF the peak rating was earned before those 6 months, then calculate average minus peak, and IF the peak rating was earned during those 6 months, then calculate peak minus average. This is to indicate whether one's own level of performance has increased or decreased relative to peak rating for the last 6 months.
To give it an example, if Mr. A's average rating was 400 and his peak rating was 500 two years ago, then his "score" would be a -100, which indicated that he has performed worse currently. If Mrs. B's average rating was 400 and her peak rating was 500 within the last 6 months, then her "score" would be +100, which indicated that she has performed better currently. We are, of course, making the assumption that these people's true skills have not changed significantly over the last 6 months (and as a whole, that seems to be a reasonable assumption to make as some people would improve and some would not).

Then we would compare to see if there is a general trend for the old top players vs the new top players.

I don't know if I would buy your argument, MM. I think your argument works well if there wasn't a continuity between the current upper echelon vs the old upper echelon. If these were two separate groups, then I would buy your argument since if you are picking the top 40 or even 1% from the population, then it is likely that the skills or whatever metric you want to use would be comparable.

However, there is continuity here. The top 40 all time great have remained relatively stable with few changes. If a top 1% player in 2010 does as well as he does in 2013, then his rating would be relatively comparable without significant changes (or if you were optimistic, then you could argue that the latest average rating should be higher because he would have learned and improved since then). If many upper echelon players continue to fare as well now as they did then, then I would submit that the level of competition has not changed. However, if the upper echelon players do worse now and remember, they would have had the extra time to refine and improve their diplomatic skills, then I would submit the level of competition has gotten stiffer. Moreover, if we take the 1% of the old, we can compare to see how many of them are still active and how they are doing now relative to the new 1%.

I think arguments and anecdotes/personal feelings are less viable here than numerical data as this seems to be a posteriori issue rather than a priori one. The justification here is dependent on empirical evidence, not independent of experience.

I also disagree that there are as many solos now than there were years ago. Whether it's the change in the level of the competition or people "settling for draws" as some might argue, there are more draws now than there were in the early years. As jmo pointed out and I agree, people are more knowledgeable now and they generally have a better sense of stalemate lines.

Wish I had more time to debate, but for now I'll have to leave you with this: Nuh-Uhhhh!!

@CSteinhardt
I am not so sure that Trueskill is unable to be used. They have quite a bit of material on it that describes how to implement it. I think at one point I saw a public github that implemented it. Since Webdip isn't making money, I think the restrictions might allow it.

I'm not sure what your language of choice is, but Infer.net is designed by Microsoft for "probabilistic programming", so it might be useful. I have never used it, so I can't vouch for it. (http://research.microsoft.com/en-us/um/cambridge/projects/infernet/docs/InferNet_Intro.pdf)

I haven't read the article that describes Glicko, but I think it is also uses Bayesian probability, so I am not sure what the differences are. Trueskill was designed for multiple teams and draws, which is an obstacle for other ranking algorithms.

oh shit

I think the competition level remains about the same, based on my experience here over the past few years. Good new players join, good old players leave. Diplomacy skill is such a touchy-feely unmeasurable, it's hard to say straight up that Player A is better than Player B when both are good players. So much depends on what country Player A drew, who else is in the game, how their styles match up, what happened in their recent games, etc. There are still plenty of players that I consider to be high-quality based on reputation or stats, who I haven't played yet. Some are "new" players who are still active and some are "old" players who aren't around anymore (or periodically re-enter the ghost rankings just to push my rank down for a month or two, bastards).

Solos are very difficult to achieve in high-GR games. Some of it is the all-around skill level of the players, but I think some of it is very conservative play where a 3wd is an acceptable goal. I like to push a game to take a solo shot if I think I have a realistic chance, but there's much less incentive to do so if when I fail there is no counterbalance of someone else trying for a solo. In tournament games, such as the Masters or World Cup, it seems to me like solos are still very possible.

I also think zultar is a big nerd.

I second the sentiment that Zultar is a big nerd. :P

Seriously, though, thanks for the mention! I don't feel like doing a bunch of math at the moment, but I will say that you Zultar were definitely one of the players I identified as skilled early on and looked up to.

What did nerds do that was so offensive?

I'll tell you what they did. Nerds invented mathematics, science, philosophy, all things academic. Nerds gave us modern technologies and conveniences. Nerds invented the internet, Diplomacy, Webdiplomacy, and 2WL, the most advance funny artificial intelligence this century. Nerds gave you ghostrating! Nerds gave YOU everything you can ever wish for.

What did you guys do? You opposed them on every one of those things, every one.

So when you try to hurl that label at my feet, 'nerd,' as if it were something to be ashamed of, something dirty, something to run away from, it won't work, THM and Tasnica. Because I will pick up that label and I will wear it as a badge of honor.

Same here zultar. Nerd, geek, dweeb, dork. I got called all those throughout school and now I proudly say "Why thank you!" When someone tries to insult me with those words.

Sorry, but I'm a computer. Quit the illegal downloading.