Do you know that B is a better player than we thought after step 2, though? We don't have any information about C so my prior on B won't change given any information about his performance against C.

So, in 2, if C beat B 4-1, would A's rating go down because it makes B look like a weaker player?

Okay, that's not in line with the point I was trying to make. Let's say that we know C's ability well, and he is an average player.

After step 1 we think B is below average, and A above average. After step 2, we should probably think that B is better than average, and A is much better than average

@2ndWhiteLine

Yes, that is correct.

I wouldn't think so. For all we know Player B just drastically improved as a player and has turned into a superior player then Player A with his additional experience.

I guess, then, I would say I don't like that your rating can change a lot if you're not playing games, which is your main point. Ability and effort are also not static. If B loses a bunch to A, and then A gets really drunk and CDs in his next 6 games, I don't think A should suffer in rating.

The anxioma of any rating system must be that it is a rating *in a certain point in time*. You're trying to apply it as if the rating would apply to somebody, regardless of where it is in time. To me, this seems to go contradictory to, well, life. Everything is in motion - I can improve, I can become better. Person B can also improve or become worse. Whether or not person B can up his game should not factor into mine.

tl;dr: ratings are by nature points in time.

Er, B. B shouldn't suffer because the guy he bested him drank bourbon instead of entering orders.

Wtf why shouldn't he stqxkle?

RE the not-static ability: I agree, and in fact would want to take into account the time delay between the games to decide how much of an effect it has. (So if the games are finishing 1 day apart, it makes a big difference, but if the gap is 2 years, there is no difference at all)

I'm not sure time is a good proxy for potential for improvement. Your results in diplomacy, like anything else, can be non-linear and non-monotonic. Skill probably zigzags in weird ways for a lot of players that doesn't have a dependable relationship with time or games played.

Agree with Stackle's last comment. And I think that zigzagging is not a bad thing per se, in fact, it probably encourages consistent good play, as well as opens up the market for new players that don't have a huge trackrecord yet.

Also, it'd probably be a bitch to calculate :-)

@Stackelberg

Skill probably zigzags in weird ways for a lot of players that doesn't have a dependable relationship with time or games played.

This is exactly correct. If I know your rating today is 100, and I want to say how good you will be tomorrow, its probably going to be close to 100 still. If I want to say what your rating will be in 1 year, my best guess is 100, but there's a very broad range.

To be clear, I can still model changes in ability, you don't need to worry about that. The scenario I am describing above has all the games happening in a small time frame, even partially overlapping.

@Ghost - if you implement the method posited by your hypothetical situation, you're going to increase the complexity of your rating system exponentially. I run a college football computer ranking system that iterates until the system is stable. It typically requires 15-20 iterations with 128 teams and about 840 data points. You have thousands of registered players here with tens of thousands of games. The complexity of re-iterating to advantage player A because of player B's subsequent games would probably mean it's not worth your while.

Additionally, players get better over time. Why should you be advantaged for beating player B back when he was a lousy player later?

"Additionally, players get better over time. Why should you be advantaged for beating player B back when he was a lousy player later?"

Alternatively, why should you benefit from beating a great player back when he was new and inexperienced? If I beat peterwiggin in his first game on the site, I would reap the benefits for years afterward. Conceivably, I could play one game on the site, beat peterwiggin or MM, then sit back and watch my GR increase without doing anything. I would much prefer games be treated individually.

That's what I was trying to say but I don't think I phrased it very well.

I think it's a good idea Ghostie, especially w.r.t. current GR as GR in its current implementation isn't actually a predictor of a player's current skill per se (it's closer to something like a lower bound on that skill). This goes back to the discussion a long time ago about how a fundamental problem (maybe problem is strong) with GR is that it is a single value rather than a ranking and an uncertainty. It sounds like this idea could at least partially address the issue.

No one that we respect actually thinks that ratings/rankings don't have an uncertainty right? Do we need to cater to those people by putting little * or ± into the system? That would boggle my mind.

@Valis- the point is that GR isn't actually a best guess as to a player's current skill. It's a self correcting ratings system like ELO.

For example, if a new player soloed their first game against 6 players with GR 500 players on the site, they would get 2/37 (I think this is the right k value?) * 500 * 6 = 162 bringing their GR to 262. Probably an unbiased estimate of their skill at this point would be closer to at least 400, possibly higher, but with high uncertainty. But a perfect guess of skill is not what GR is meant to capture. It's meant to be a rating system in which a good player can work their rating up and eventually reach their true value.

I really dislike something like this. New players want to at least be able to roughly understand why their rating is improving or falling - game results do that. A system like this has no transparency and I don't really think it's much better - if you beat someone on their first game, then they win 4 in a row against other people because they've got used to it, it doesn't mean it required more skill on your part to beat them in their first game.

I agree with the "no's". Past games are a sunk cost and should not affect future ratings based on factors outside of your control (when you are not in that future game).

I like that the GR system allows you to "work [your] rating up and eventually reach [your] true value" as @uclabb wrote, or if you continue to improve (or perform worse) the GR finds its way to your new equilibrium.

"For all we know Player B just drastically improved as a player and has turned into a superior player then Player A with his additional experience."

"The anxioma of any rating system must be that it is a rating *in a certain point in time*. You're trying to apply it as if the rating would apply to somebody, regardless of where it is in time"

"Why should you be advantaged for beating player B back when he was a lousy player later?"

"If I beat peterwiggin in his first game on the site, I would reap the benefits for years afterward."

"A system like this has no transparency and I don't really think it's much better - if you beat someone on their first game, then they win 4 in a row against other people because they've got used to it, it doesn't mean it required more skill on your part to beat them in their first game."

This comment is coming up a lot, and it's not at the core of what I am asking. Let's nip it in the bud:

The simplest way of doing this is to assume each player has a rating which is constant across time. As you correctly point out, this is limited.

What I would actually do is model each player's rating as a function of time. I will have a prior over possible functions that could be a players rating. Smooth functions with ratings closer to average will be more likely, but if someone changes ability, it will model that.

________________________________________________

What I want to know is:
Suppose there were a rating system which made these sorts of updates, and was more successful at predicting results than when you remove this. Would you prefer this, or having ratings only change when you've played?

So yeah, I don't want to know if you guys think it will work: I can evaluate that numerically. I want to know how you feel about it *given that is does work*

So you're asking if we want something that's strictly better than another option?

Does this mean that players wouldn't have to drop off the list due to inactivity?

More is it worth the trouble of removing the old system and replacing it with a new system, knowing some people will be genuinely resentful of it, for the sake of a small improvement in how it works. The whole "is it worth moving house to live 10 minutes closer to work?" argument.

They have to re-make it for integration into the site, no problem also trying to make it better as well. Nothing sacred about the current formula/system.

There is always value in consistency

A few things here: any rating premised solely on past results can only be retrodictive, not predictive. Predictive rankings (Vegas-style) would have to take into consideration a whole number of non-quantifiable factors for a game like Diplomacy.

Second: @Valis - I don't think the question here is if we want something better than the old option. I think the real question is whether it's worth it to the administrators of the GR to exponentially increase the difficulty in compiling these rankings. I'm sure we'd all love to have the most finely tuned system in the world. Is it worth that much extra labor to the administrators?

The problem of rating player or sports teams isn't nearly as simple as it sounds. A new system can't simply be "Let's go back and adjust past game results with today's rankings." Because the results from those games will create a new set of today's rankings. Then we have to go back through all the games again with the new set of rankings and we'll find that yet another new list of today's rankings is generated.

To do that long enough until the initial and final rankings are the same will take a ton of iterations. In the world of computer sports rankings (yes, there is such a world and I'm an amateur in it), this is called infinite depth. To transform a system with this many games and players into an infinite depth ranking would be a huge amount of time. Only the administrators can determine if that's worth it.