For some time now, I've been getting complaints that the MDL rating system is too harsh, and ratings drop too much on the back of expired wins. However, ratings do not rise on the back of expired losses. I will briefly explain the previous system, it's flaws and what this current change is all about.

Old System:

• Games expire after 5 months.
  
• At any given date, the CLOT gets all games with finish date >= (today - 5 months). All games prior to this are considered as expired.
  
• All of these games are used to compute the Elo rating for every player from scratch(using the standard Elo algorithm). Due to this approach, the set of games used to compute Elo is always changing, which leaves players with high ratings very susceptible to rating changes on the back of expired games. Case in point - myself during my long winning streak(I couldn't sustain my rating, let alone improve on it). Another recent example is timon, who was falling quite a bit without new finished games.

The Elo rating system doesn't handle game expiration very well and is generally considered a continuous rating system. We've all seen RTL where players come back after a year or two with very high ratings and hog the top spots on the ladder without justifying it with their recent results. The game expiration component on MDL was one of the ways I chose to deal with this problem. By making games expire, I ensured that any player with a high rating had to justify it with current results. This wasn't an ideal solution, but it was the only one I could think of when I first created MDL.

Over time, I've observed the rating system on MDL and learned more about such systems. Recently, I had a long chat with Math Wolf about MDL's rating system, and we came up with a slightly different rating system. I'm calling it the MWElo rating system as it's mainly Math Wolf's idea.

MWElo System:

• Games will never expire in this system.
  
• The underlying principle of any rating system is that it needs enough data to be able to accurately estimate a player's skill level. So we need to provide incentives to be active on MDL.
  
• To achieve that, this system uses the concept of an activity bonus. It awards an activity bonus of k/8 = 4 points(k-factor = 32 on MDL) for each finished game. This bonus is awarded to both the winner and the loser at game completion.
  
• The activity bonus is capped at a max of 80 points. This ensures that if you can guarantee a certain level of activity, you will receive most of the possible activity points. It also prevents someone who is very active from getting too many activity points.
  
• Every day, the total activity bonus points for every player will decay by 2%. This decay ensures that a player must remain active in order to keep their activity points. 34 days of complete inactivity(0 games finished) will halve your activity bonus [(1-0.02) ^34 ~= 0.5]
  
• The displayed rating on MDL is the sum of the true Elo rating of a player and their activity bonus(similar to the seasonal ladder). The Elo rating is used to determine player match-ups. This ensures that players of equal skill level are more likely to be matched up. However, the ranks are computed using the displayed rating.
  
• This system also introduces an additional component which will converge your Elo rating towards 1500 if you have been inactive for too long. The criteria* is that if you haven't finished a game in the last 50 days, your rating will converge towards 1500 by 1 point every day after the 50th day. For ex - If my rating is 1800, and I've been inactive for 80 days, my rating starts decreasing(if rating was <1500, it increases) by 1 every day after the 51st day. So by the 80th day, my rating would be 1800-30 = 1770. Over time, if I go completely inactive,I will converge to 1500.

This system is in preview. We will evaluate its performance over the next week and decide if we want to keep it, adjust it or switch back to the old system. I would like to thank Math Wolf for spending a lot of time and coming up with this system. I've compiled a list of the old/new rating for every player on MDL which can be found here - https://goo.gl/X1gJEn

I will post a change-log in my next post. Keep in mind that this may system may contain some bugs which I will fix as soon as I find them. The change touches a large portion of MDL's code-base. If you notice any issues with other MDL features, please let me know.


*(current criteria is if you haven't finished a game in the last 5 months.. This will be changed to finish a game in last 50 days at a later date).

Edited 7/31/2017 04:28:19

Changes

• Add activity bonus component to the rating system.
  
• Decay activity bonus by 2% every day.
  
• Decay Elo rating every day if player is deemed inactive
  
• Database changes to support the new rating system
  
• Perform incremental Elo updates instead of considering all games on every run. The previous Elo rating and the recently finished games are used to compute the new Elo score.
  
• Remove Unexpired games table on MDL.
  
• Remove Least Vetoed templates table on MDL.
  
• Update the following pages to reflect the new rating system.
  
  • Rating distribution graph on the MDL Stats page.
    
  • Weekly Report page.
    
  • All players page
    
  • Clan page - Show Most Games table instead of the Unexpired Games table
    
  • Template page
    
  • Show DisplayedRating, EloRating and Activity bonus on the player page
    
    

Annoyed personally because I was planning on re-joining the ladder after my games expired but now I feel highly agitated since in the past I joined MDL and surrendered all games when leaving.. Should I get penalised for inconsistency possibly so... Though I feel I'm more consistent handling my games without surrendering like a bitch as I'm actually completing my 2v2 and 3v3 Ladder games and commiting reasonably fast and feel it's unfair that my past is going to possibly hold me back.

I'm still grateful for the "improvements" though regardless if I personally agree/disagree but wanted to say this.

Annoyed personally because I was planning on re-joining the ladder after my games expired but now I feel highly agitated since in the past I joined MDL and surrendered all games when leaving.. Should I get penalised for inconsistency possibly so... Though I feel I'm more consistent handling my games without surrendering like a bitch as I'm actually completing my 2v2 and 3v3 Ladder games and commiting reasonably fast and feel it's unfair that my past is going to possibly hold me back.

Your past is not "holding you back". You currently have an Elo Rating of 1554. How you got to that rating has no bearing on your future. All that matters is that your rating stands at 1554. This is not like the standard WL ladders which use BayesElo(past results continue to impact you in BayesElo). You could get to 1554 via 3-0 or get to 1554 via a 25-22 record. As far as this rating system is concerned, it's all the same.

If you were to let your games "expire" in the old system, you would have to work harder to get to the rating you're aiming for. I would recommend reading how the standard Elo rating system works.

Edited 7/31/2017 04:43:29

I don't have a understanding of how rating systems work and maybe it was naive to make such a comment above however most probably don't know either and I just based my logic off 1v1 ladder which is also Elo? Bit confusing tbh.

Would you mind linking a source for someone noob to understand how the Elo rating system works and maybe put it on the MDL of some sorts cause education of how the rating system works would be good? Instead of me and maybe others assuming it has similarities like 1v1 ladder for expiration assuming they are roughly the same... because that's the only rating system they are exposed to.. sorta?

Edited 7/31/2017 04:35:56

There is a link under FAQs > Ranking.
https://metinmediamath.wordpress.com/2013/11/27/how-to-calculate-the-elo-rating-including-example/

EDIT : You can also use this site to calculate the effect of an individual game. Just remember to use k=32.
http://www.3dkingdoms.com/chess/elo.htm

Edited 8/1/2017 04:42:58

If all that matters is that your current rating is 1554, and that rating was computed from past events, then doesn't this mean that the past actually does matter?

Of course, your results determine your Elo rating. But the point I was trying to make was that how you get to a rating is irrelevant to your future. i.e, 10-0 can have the same effect as 20-10 if it meant that both those records yield the same rating(depending on your opponents).

This is not what you see on regular WL ladders(BayesElo). Losses bog you down there. So people usually make runs and wait for losses to expire to achieve "higher ratings/rank". Platinum was sort of hinting at that, which is something he doesn't need to worry about on this ladder because it doesn't work that way here.

Great change, MotD. I really like, that you talked to an expert like MathWolf and improved the rating system.

Hey I lost 8 ranks with this new system! Replace it immediately back with the old one that rewarded snail players like me!
/end{trolling}

I'm very happy MotD contacted me to discuss this! Overall, I think this is an improvement (obviously), although small bugs and kinks may need to be ironed out over time.

About "expiration" in general: the technical argument goes that games either "expire immediately" or "expire at infinity". Once a game is counted into the rating, it will never be used again. This makes sense as the rating is updated with the game when it happens. All systems using expiring games update the rating using games that finished several months in the past, which, if you think about it, doesn't make all that much sense.
This is also what MotD means above when he says that "how you get to a rating is irrelevant to your future." Once the game is counted and you have the new rating, the game won't be used again. How the rating was obtained is irrelevant when the next game comes along (which isn't the case in BayesELO).

The actual impact of a game on the rating will slowly decrease over time. (i.e. what would the rating be without this game?) Every new game against an opponent of similar skill has theoretically a larger impact than every previous game because it applies a change on the updated ranking. This can be practically tested immediately if you'd flip the order of a win and a loss against oppenents rated exactly the same: win, then lose gives a lower rating that lose, then win. (Sidenote: ELO actually rewards delaying wins rather than delaying losses, which is a great feature I think!)
This is an advantage over TrueSkill (used in RT ladder), where the impact of early games can be large and linger for quite a while, not even taking matchmaking into consideration.

Experts generally agree that the three main drawbacks (outside some technical issues) of ELO are
(1) rewarding inactivity
(2) the (subjective) choice of a K-factor
(3) the relative importance matchmaking

For MDL, the main problem was (1), which we tried to fix with the activity bonus. As currently implemented, this bonus is similar to a time-varying Poisson process (count data, here "counting" recent games). Additionally, convergence towards 1500 seems generally appropriate for long absence as there is no guarantee the rating is still correct.

I think (2) is less relevant for MDL. The subjective choice here simply means how much fluctuation MotD (or the players) want. A little more fluctuation is good and adds excitement for these kinds of competition, so a high K seems suitable. If the idea would be to have a rigorous ranking where you need many games to move substantially, a lower K would be needed.

From what I experienced myself, the matchmaking at MDL is automatic and quite good, which makes gaming the system (choosing one's opponents) more difficult and (3) irrelevant.

Great uppdate motd, and great job Mat wolf aswell, and also a very good comment by you above! Im exited for the future of MDL! This is a well needed update and its amazing!

I like the general idea. I really do. I'm not sure about the math aspect of it though.

Finishing a game grants 4 activity points. The points are capped at 80. From equation (1-0.02)^d=1-(4/80) you get d=2,5. That means that to get full rating bonus from activity, you need to finish a game every 2.5 days on average.

As a player who plays just a few games at the time (I usually limit myself to 6 MD games total), the idea that I'll always be behind in points compared to players of same skill level who play 9 simultaneous games is a little bit off-putting. I can already tell you I'm not the only one. (I know I'm not even participating in MDL now, but I was going to join again after I leave 2v2 ladder)

Did you consider delaying that counter? Let's say you start losing activity points a week after your last game finishes. That + 2,5 would give you over 9 days to finish a game on average. You could change the point loss to 2,5% on every day after the first week to get the same result of 0.5 after 34 days.

Just my suggestions. Though I don't believe you and Math Wolf did not consider it already. So may I ask why did you decide on such a move? I agree with not promoting non-activity. But are you sure that blatantly giving more points to players who play faster is a good idea?

PS: great job with the ladder overall. We all appreciate the effort you put into this.

@Rento: I'm a snail myself, so we did carefully consider these numbers. I knew it would penalise me as well. :-)

The impact should be reasonable. To use myself as an example: I play few games at the time and slow, barely enough to remain ranked actually. Yet, at this moment, I have 25 activity points. A criterion that I proposed to MotD was that anyone putting it at least some minimum effort needs to get half the activity points (40). A total of 30 games in 5 months (you needed 20 to remain ranked) gives you (slightly more than) that. The difference between a very active player and someone who achieves this is less than 40 points, which seems to make a difference of only a few ranks.

I personally think this is correct and fair because the rating of someone who plays more is more accurate.
Someone who wins his last 10 games, but didn't play all that much during that period (and may be delaying losses), does not deserve the same activity bonus as someone who goes 20-5 during the same period. Yet, without activity bonus, they might get the same rating. For the first player, there is a much larger risk that this rating is inflated. Note that TrueSkill, used on RT, also penalises for (lower) activity and more severely than this system. From a mathematical and theoretical point of view, I don't see it as an activity bonus, but as an accuracy bonus. My first proposal to MotD was about using a variance measure to achieve this same effect, but that was not as practical.

Either how, as far as I understood, these numbers are not final yet and may be adapted based on how much activity should translate to a full activity bonus. I'm sure MotD doesn't want it to be a reason for people not to join, but at the same time it should remain an incentive to play more than just two games at a time.

I'm personally not a fan of delaying the decay. This would give an incentive to stall losses with exactly a week in between, which we definitely want to avoid!

EDIT: some more details and example added in second real paragraph.

Edited 7/31/2017 15:45:44

Well here's where we disagree: 40 or 50 points difference is a big deal if you ask me. I mean, I was never in top of that ladder, but from what I've heard you get literally a few points for every win up there. (Can I ask for confirmation MotD?)

To put things in perspective:
- finishing a game every 2,5 day gives you full 80 points
- every 3 days - 70 points (-10 points already! just half a day difference in playing speed)
- every 5 days - 40 points (40 points behind)
- every 7 days - 30 points (50 points behind!)

Does that really seem fine to you guys?

I personally think this is correct and fair because the rating of someone who plays more is more accurate.
Someone who wins his last 10 games, but didn't play all that much during that period (and may be delaying losses), does not deserve the same activity bonus as someone who goes 20-5 during the same period

I agree partly, however sometimes a person is being penalised for not finishing games because their opponents are playing ridiculously slow. Is it fair to penalise the other person who would play fast given the chance?

Currently I have 5 games ongoing. (1 just finished because I asked the other person to play faster) but 4 out of those 5 games are being played ridiculously slow by my opponents. The one that just finished today was over two weeks and the other 4 are about to be 2 weeks. That means I have not finished a single game in two weeks, not from any fault of my own, yet I am penalised for those extra points I could have had. I did talk to Motd about it and he explained it doesn't have that much impact and it will even out due to getting faster opponents another time and I'm fine with that. However I just wanted to comment on your statement where you said it was fair.

Is there any way the "Best Rating Achieved" can be recalculated factoring in this new rating system? Ex. I had 1750+ on the old rating system, but with this new one, my current rating (below my former max in relation to the old rating system) is automatically above the old max rating I had on the old rating system. I'm not sure whether the data has been lost, but I think the conversion of old -> new should also apply to the former max rating achieved.

Well here's where we disagree: 40 or 50 points difference is a big deal if you ask me. I mean, I was never in top of that ladder, but from what I've heard you get literally a few points for every win up there. (Can I ask for confirmation MotD?)

That's not completely true anymore. This had to do with the expiration, which pulled the rating back down again and again whenever a previous win expired. A win while ranked at the top should now actually make a difference. Note that the top rated players all have a much higher rating now because of this (as it should be!)

To put things in perspective:
- finishing a game every 2,5 day gives you full 80 points
- every 3 days - 70 points (-10 points already! just half a day difference in playing speed)
- every 5 days - 40 points (40 points behind)
- every 7 days - 30 points (50 points behind!)

Does that really seem fine to you guys?

It's of course up to MotD to decide if he wants to change the numbers (I'm in contact with him and open for discussion about it at all times). Personally, I'm happy to talk about them here and discuss the pros and cons of them.

That said, I personally agree on these numbers and they look fine to me (as a slow player!).
Because:
Every 2.5 days means almost 3 times more games than every 7 days! That's 3 times as much chance of an upset loss, because more games also has a higher chance of getting an opponent that is somewhat further away in rating. A difference of 50 points to offset that seems fair to me with a K-factor of 32. After all, a game against an equally rated opponent nets you 16 points while a big upset loss can cost you 30. Lower amounts of games do lead to reduced precision and associated behaviour to game the system.

Also every 2.5 days versus every 3 days is 20% more games as well. It's not half a day difference in play speed necessarily, it can also be 6 games instead of 5, which is a tangible difference, I'd say.

Again, they are not final and as far as I know it's open for discussion what should be enough to warrant a maximum bonus. Giving 5 points per game for example, would already mean you need 20% less games to achieve the maximum (and would give everyone except those at the cap a 25% higher bonus). A total of 8 points per game (or alternatively, 4 when started, 4 when finished), would mean 30 games in 5 months is enough to reach the max.

I personally think this is correct and fair because the rating of someone who plays more is more accurate.
Someone who wins his last 10 games, but didn't play all that much during that period (and may be delaying losses), does not deserve the same activity bonus as someone who goes 20-5 during the same period

I agree partly, however sometimes a person is being penalised for not finishing games because their opponents are playing ridiculously slow. Is it fair to penalise the other person who would play fast given the chance?

Currently I have 5 games ongoing. (1 just finished because I asked the other person to play faster) but 4 out of those 5 games are being played ridiculously slow by my opponents. The one that just finished today was over two weeks and the other 4 are about to be 2 weeks. That means I have not finished a single game in two weeks, not from any fault of my own, yet I am penalised for those extra points I could have had. I did talk to Motd about it and he explained it doesn't have that much impact and it will even out due to getting faster opponents another time and I'm fine with that. However I just wanted to comment on your statement where you said it was fair.

That's obviously a fair point. There are three (four) things I can say about this:
* I agree with MotD that over time it will offset each other, but this will indeed lead to small changes during periods when many games finish close after one another and periods where few expire. I'm not aware of any reasonable solution to this.
* After reading about the update, comparing the numbers and thinking everything through again, we discussed a small (future) back-end change that should make sure that this has no impact on very active players who are/were at the cap. If you accumulated points above the cap, some of those are retained in the back-end, so that you can keep maximum bonus for a few days after finishing the last game (not all to avoid the most extremely active players to remain at +80 a month after they leave!)
* Splitting the bonus in 2 points when starting a game and 2 when finishing could offset this and allow active players to add games in such situations? I'm not sure if this would be easy to program however.
* You are lucky I am not one of your opponents. ;-)

EDIT: minor text fixes and included reply to Styxie as well.

Edited 7/31/2017 16:19:03

* You are lucky I am not one of your opponents. ;-)

Haha yup. I'm not complaining as such, I understand some players play slower than others especially when busy and I don't really have a problem with people playing slowly. Just wanted to point out that while slow players are being penalised which may be deserved, this impacts their opponents too which may or may not be deserved.

* After reading about the update, comparing the numbers and thinking everything through again, we discussed a small (future) back-end change that should make sure that this has no impact on very active players who are/were at the cap. If you accumulated points above the cap, some of those are retained in the back-end, so that you can keep maximum bonus for a few days after finishing the last game (not all to avoid the most extremely active players to remain at +80 a month after they leave!)

If it does fix the issue, then that's great and appreciate you and motd taking it into consideration. Thanks for all the effort and discussion.

"Haha yup. I'm not complaining as such, I understand some players play slower than others especially when busy and I don't really have a problem with people playing slowly. Just wanted to point out that while slow players are being penalised which may be deserved, this impacts their opponents too which may or may not be deserved."


Well this is so easy to fix. whenever you end up in such situation, and that cant be to often u end up in 5/5 opponents being real slowpokes, but when that happen its just to incresse tha gameamount to 6 or 7, get 2 new games and then reduce it again. Playing 5 games agains fast players takes more effort thab 7 games against slow players so it shouldnt be a problem?

I don't see it as an activity bonus, but as an accuracy bonus

I think this is the best way to think about it. Also, if you look at rating differences on the ladder, especially at the very top, the difference between places is often (though certainly not always) more than 40 points anyway, so you might not lose out in placement based on the accuracy/activity bonus anyway. I think it's a positive thing, and I speak as a person who has somehow been playing even slower on this ladder than Math Wolf! *gasp*

Is there any way the "Best Rating Achieved" can be recalculated factoring in this new rating system?

I agree with this also, though probably don't do it until you finalize the fine-tuning of the rating system. As is, the historical graphs are pretty meaningless, so it would be nice to have it back dated.

Edited 7/31/2017 17:40:28