(1) Figure out a matchup discrepancy matrix
e.g. Peach vs Puff winrate is 0.43
(2) Use an Elo head-to-head variant where the Elo update function takes matchup discrepency into account
e.g.
- A vs B has an expected 0.9 winrate
- A is Peach and B is Puff
- Elo update is done expecting A to have a winrate of 1 - (1 - 0.9) * (0.5 / 0.43) = 0.884
The next best Puff player is #38, and doesn't have any wins against top 10 foxes. Is HBox just the best player ever, consistently winning a "bad" matchup, or is Puff a better character than people commonly believe? Who's to say?
Well, TFA had no bones about calculating one.
> Is HBox just the best player ever
The current data says pretty definitively, yes.
If other players can learn how to get his winrates vs Fox, then the matchup matrix would end up reflecting that. The matchup matrix doesn't need to reflect the perfect ("objective") state of the matchup, just the current one.
(The system I'm talking about would look more suspicious if HBox wasn't considered the best, because it would probably put him at #1 anyway.)
I didn't do a good job of clarifying what I meant. Hbox is obviously the #1 player right now. The question is if he's just totally on another level of every other player, or if we're underestimating puff as a character.
Note that this is a really deep question. There are strong arguments (parry) that in the "20XX" yoshi would be the most viable character right after fox. Given that, is Amsa overrated because he's underperforming how his character should, or underrated since he's overperforimg the "average" Yoshi player?
The system you describe basically just ends up rewarding above average players who use unusual characters. Should Abate be ranked top 20? Probably not, but considering how much he outperforms the "average" luigi (same thing for Amsa, does he deserve to be, say, top 10), he probably would be.
It really depends on what you want the ranking to mean.
If you want it to mean: "If all the players in the world played in a tournament, what would the expected result be", then a normal Elo-like rating system (e.g. glicko-2) should be fine, because all the data available is from real tournaments, and it's not really feasible for players to strategically dodge bad matchups to pad their ratings.
But one criticism TFA has of this method is matchup discrepancy. I'm not sure that's actually important (players choose their mains freely), but if it is can't you just correct for it?
I think you're right that this correction would create an undesirable result. That just means that the matchup discrepancy criticism isn't good.