Seeking Criticisms and Suggestions for Personal Ratings (Elo W25)

[u/TigerWave01]

Hi all! I am reposting a post that I just made because of changes I made from the original post. To recap what I said then, recently, I have been getting into making my own ranking system, with the goal of creating a ranking system that is as transparent and simple as something like the Colley Matrix while still taking things like margin of victory and power of a team into account. This is mostly out of boredom but also because I just really, really like sports analytics. However, I am by *no* means a mathematician (I had to talk to other, smarter friends of mine to get some of the terminology right here) and therefore need to keep it simple for me to understand a lot of this, or at least have something that can be explained in an easy to understand way. With all of that in mind, I settled on making what I call, for now, "Elo W25". In very simple terms, it works like this:

Method

1. Preseason: This includes two parts: finding a team's last season Simple Rating System (SRS) rating and converting that rating into an Elo score.
   • Last season's SRS = average margin + average opponent SRS.
   • Last season's SRS is then converted into an Elo score with Elo = 1500 + 18.5 × (Last season's SRS − FBS average SRS).
2. In-season Elo: Prior to a game, a calculation is done to determine the expected result from the game. After the game, another calculation is done which gives or takes points from a team depending on whether that team won or lost (and if the team was expected to win or lose) as well as the margin of victory.
   • Expected result = 1 / (1 + 10^(−((Team's pregame Elo − Opponent's pregame Elo) + home edge) / 400)),
     • Home edge equals 46 points (0 at neutral sites).
   • After each game, change = 75 × ln(margin + 1) × (result − expected).
     • If a team wins, result = 1. If a team loses, result = 0.
     • Margin is simply winner's score - loser's score.
3. W25 (wins above No. 25): At the end of the season, teams are ranked based on a benchmark equal to the 25th-best Elo at the end of the regular season.
   • To calculate the probability that the No. 25 team would win a game, run the calculation p = 1 / (1 + 10^((Opponent's final Elo − (benchmark + home edge)) / 400)) for every game.
   • A win earns 1 − p and a loss costs p. W25 = a team's actual wins − Σp, and teams are ranked by W25.

The results can be found here for FBS teams and here for lower division teams. Like I said before, the point of this is to 1. reward strength of schedule and strength of resume, 2. be relatively accurate compared to other computer polling, and 3. be incredibly simple and not beholden to information other than game data, which preserves flexibility. However, I am open to any criticism, even harsh criticism! I want to do anything I can to make this rating better while still sticking to those three main goals. Thanks for taking the time to read this!

[u/importantbrian]

Not bad.

Why start the season with the prior seasons SRS and then convert that to Elo? Seems like an unnecessary step. You could just use their end of season elo from the prior year. You can regress their end of season elo to the mean by some amount between seasons to account for turnover, and season to season variability.

The W25 part doesn’t make a ton of sense to me. How is this different from just ranking by elo?

[u/TigerWave01 → u/importantbrian]

Thank you! From the tests I did, I think starting from the prior seasons SRS was slightly more accurate for pregame picks, albeit by an incredibly miniscule amount. That part I've been very on the fence on because doing an SRS calculation, while it may be more accurate, is also much more complicated and only slightly more accurate, if at all. SRS is also more accurate to human polls, but it's not necessarily a goal to stick to human polls, they just provide a point of reference.

As for W25, my reasons are because I believe Elo and W25 are doing two different things. If I ranked just by Elo, you end up getting much more of a power rating rather than a ranking system. And if a power rating is what you want, I think Elo does a pretty decent job with that. But for a resume-based ranking, not so much. Take 2023, for example. Ranking by Elo gives you a top four of Michigan, Alabama, Georgia, and Ohio State. Washington, Texas, and Florida State are 5th, 6th, and 7th respectively. What W25 does, at least in my view, is rank teams not based off of necessarily who is the "best", but rather what each team has accomplished in a season. Tbf, there's probably no reason to worry about pregame picks if I'm trying to make a ranking system, but that's the reasoning I had for the addition. Thank you again for the feedback!

[u/importantbrian → u/TigerWave01]

Ah okay I reread what W25 is doing and I get it now. It’s a SOR metric. I’m still not 100% on how it works but I see the general concept.

Depending on how small the gain from SRS is it might be worth considering if it’s random noise or maybe overfitting. Is it an improvement for every season in your sample or just some?

Something I’ve done with elo systems in the past is to regress the season end elo to the mean by some amount. I’ve even built a model using CFBD’s returning production to do the adjustment. The only prior I’ve found that works better is Vegas win totals.

I tend to prefer my ranking systems to be predictive but that’s a philosophical question I don’t think there’s a right answer.

[u/TigerWave01 → u/importantbrian]

Yea, W25 is a bit of a difficult one to explain (actually part of the reason I posted this here, since I wasn't sure if there was maybe a simpler option that was as effective), but it's the best system I've tested so far. To put it in less number-y terms, it's basically seeing how the #25 Elo ranked team would fair against another team's schedule, then comparing that to that team's actual performance. So, for example, say the #25's Elo score is 1800 and, after going through that formula in the OP, you determine that a team with an 1800 Elo is predicted to have 8 wins against, say, Oregon's schedule. Then let's assume that Oregon went 11-1. From there, you subtract the number of wins Oregon has (11) by the number of wins the #25 team is predicted to have (8) to get the final W25 score (3). The team with the highest score is the one that most outperformed the benchmark team and is, according to this ranking system, the team with the best resume. You can set the benchmark to any team you'd want (for example, Wins Above the Bubble is an NCAA metric that uses the same formula, as far as I understand, but sets the benchmark somewhere in the 40s to represent the last team into the tournament), but it's basically a way to compare resumes based on some preestablished benchmark.

I believe SRS is an improvement for every season, but at least in regard to pregame picks, it only increases accuracy by 0.2% (total accuracy of 71.0%) compared to keeping 2/3rds of the distance from 1500 (total accuracy of 70.8%). The difference in the CFP and AP ranking gap (basically how close the Elo W25 rankings are to those two polls) is also miniscule. The carryover (3.66 gap from CFP/3.44 gap from AP) has a difference of only 0.33 and 0.23 compared to the SRS's ranking gap (3.33 gap from CFP/3.21 gap from AP), respectively. The carryover also seems to reward G5 teams more, for whatever that's worth.

[u/Quaizimo]

Nice work, and the writeup is clear. A few things from building something similar:

1. Margin of victory multiplier. ln(margin+1) on its own over-rewards favorites. A good team beating a bad team by 40 was already expected to win big, but it still collects a large swing, so ratings drift upward and spread out over a season. FiveThirtyEight's Elo divides by the rating gap to correct for that: ln(MOV+1) × 2.2 / ((winner Elo − loser Elo) × 0.001 + 2.2). It's a small change and it usually helps.
2. 71.0% vs 70.8% is probably noise. Count how many games the two versions actually picked differently. If it's a few dozen across all your seasons, 0.2% is a handful of games. Brier score or log loss on the win probability is a fairer test than pick %, because a 51% call and a 90% call count the same in accuracy.
3. Benchmark against the spread. On the same games, check how often the closing-line favorite wins. That's the bar any predictive rating is really competing with. If Elo lands close to it, that's a good result. If it beats it, recheck your inputs before believing it.
4. Your 46-point home edge works out to about 2.5 points on your SRS scale (46 / 18.5). That's in the right range, but it's worth checking each season, because home field in CFB has shrunk over the last decade.

W25 makes sense as a resume metric. It's the same idea as strength of record: how a benchmark team would do against that schedule. Keeping it separate from the predictive Elo is the right call. What does your preseason regression do for teams that changed coaches?

[u/TigerWave01 → u/Quaizimo]

Thank you! I've been toying with the model since posting this, but so far it seems like the parameters I set, but changing from SRS to a carryover model has led to the most accurate results while maximizing simplicity, at least with regard to pregame picks, so I appreciate the suggestions to use other metrics, that's super helpful. I'll address each point as they're numbered in your post

1. Thank you! Testing that new formula, nothing much changes, but it does put Clemson 4th (rather than 5th) in 2019, which was actually one of several benchmarks I've been checking (along with general agreement with the polls in 2019, agreement with the top 5 in 2023, and Cincinnati in the top 4 in 2021, among others). At the same time, it moves Ohio State out of the top 4 and puts Tennessee in in 2022, so that strays away from the 2022 CFP and makes the Elo stray further away from the end-of-season human polls, which again isn't necessarily a must-have goal but is a point of reference for me.

2. Yea, it was only a difference of five games over ten years. Almost certainly noise, so I think I'll do a 2/3rds carryover, which seems to be the "best" system based on pregame picks and of course far simpler than the SRS model. The Brier score and log loss difference is equally infinitesimal, with the gap between them being 0.0003 in Brier and 0.0005 in log loss, about 0.2% of either score (similar to pregame picks, funnily enough).

3. This one is kind of a long one so I'll try to break it up:

Both the SRS Elo and carryover Elo usually has an ATS between 49%-51% of the time, the average being 50.1% and 50.2%, respectively, which ranks it around 15-20 against other models on The Prediction Tracker. The simpler, carryover Elo being 0.1% better seems to be more evidence that SRS beating out carryover most of the time is the result of noise. Notably, carryover Elo had a great performance last year at 52.5%, but that was very much an outlier. After some tuning (using 2013 to 2019 data to try and beat spreads from 2020 to 2025), resulting in variables of K = 55, h = 30, carryover = 0.5, and √margin, the best it could do was an average ATS of 50.3%, and the tuned results never crossed above 52.4%. Even with hindsight, the average ATS for 2020-2025 was 50.5%, which suggests to me that the ~50% mark is going to be the best you'll get using a simpler Elo system like the one I made here.

I've been toying with another model, basically an SRS rating that uses a similar chain system that Elo does plus a ton more preseason data. It's pretty complicated for my non-math brain to handle, and while it beats Elo by two percentage points in correct picks, it seems to be a wash when comparing the two ATS. The SRS Chain model had some crazy good years, like 53% in 2017 and 53.6% in 2025, but also had some of the lowest lows, like 47.1% in 2019 and 48.2% in 2021. Elo doesn't quite get that high, but it more consistently stays in the 15th-20th rankings against all other computer models.

All of this to say, it's probably not anything to write home about for betting uses and, frankly, any Elo system will probably max out at around 50.5%+ over the ten year window. However, the carryover Elo being extremely consistent and comparable to other models is still pretty decent for it being such a simple formula and only using prior game scores.

1. I actually did do several tests when crafting this rating and found that 2.5 points gave me the most accurate results, then translated that to Elo. However, that was with SRS, not with the new carryover. Doing some tests on the newer version, it seems like a home edge of 50 picks the most winners, but 60 has the lowest Brier score and log loss. With that said, these differences are pretty small and very well could be within noise; I'll probably need to do some more tests to see if any of them cause some significant change.

2. You know, I actually didn't consider regression for things like coach movement or, especially nowadays, player movement. The carryover includes a 2/3rds regression to the mean, which is based on the fact that average turnover of a CFB team is 30% to 38% (I decided to go with 2/3rds as it's simpler to explain and, at least from what I remember, it tended to do the best). But of course certain teams lose a lot more after major events like that. Frankly, I'm not sure if there's a way to incorporate those more unique occurances that maintains the system's simplicity, but if you know of a way to do that, please let me know because I'd be interested to at least try it out.

[u/Quaizimo → u/TigerWave01]

Good stuff, thanks for running all of it. Going by your numbers:

1. The Clemson/OSU shuffle is pretty much what that multiplier is meant to do: blowouts by favorites get pulled down. I'd judge it on Brier/log loss rather than poll agreement, but your call since polls are a reference point for you. Since your tuned set landed on √margin, it's worth testing the 538 correction on top of √ as well.
2. Five games over ten years is noise. Going with 2/3 carryover makes sense.
3. 50.1–50.2% ATS is about what you'd expect when grading against the close. Your best tuned result topping out at 52.4% is basically break-even at -110 (52.38%). On the straight-up side, how does your 71% compare to the closing favorite's win rate on the same games? If you're within a point or so of it using scores only, that's a solid predictive rating.
4. With picks pointing to 50 and Brier pointing to 60, I'd go with Brier. The gap is small enough that 55 is probably fine. Just make sure neutral-site games (bowls, kickoff games, conference title games) are getting 0.
5. importantbrian already pointed to the simplest fix: CFBD returning production. One way to keep it to a single extra parameter:

carryover = 2/3 + b × (team returning % − FBS average returning %)

Fit b on 2013–2019 and test on 2020–2025 like you did with the spread. Teams that lose most of their production regress harder, and that covers portal and coaching turnover without needing a separate flag. If you're open to one outside input, he also mentioned Vegas win totals as a better prior, but that breaks your game-data-only rule.

Did the SRS chain's 2017 and 2025 seasons have anything in common, or does it look random?

[u/TigerWave01 → u/Quaizimo]

1. Fwiw, the only reason they're a reference point is for purely subjective reasons; it's those results that I think the committee got "right" and so I was trying to mirror them while still having an objective way of getting there. But that's also more of the job of W25 than the Elo score; whatever I can do to get the Elo more accurate, which in turn makes the W25 more representative of who truly has the best resume, I'll take it. What's interesting, though, is that Brier/log loss is actually *worse* with the 538 formula, which was definitely a surprise to me. It isn't a huge difference, but it is consistently worse than ln(margin + 1). Why do you think that is? Like I said, that was a big surprise to me.
2. Ehhh, yea it's not the best when it comes to picking winners. At least compared to the other predictors, it's one of the lowest ones in the Prediction Tracker. Although I think that's just by virtue of being an Elo system; the other Elo systems, like Beck Elo, are all in that 70%-71% range. Honestly, it makes sense; unlike SRS, which recalculates the entire season, Elo has bias towards the last game played and only includes game scores, which kills it compared to the other, more complete systems. That's why, I think, the SRS Chain does so much better; not only does it recalculate the entire season, it also includes a lot more preseason data. The problem is that pure single season SRS is even worse than Elo, but weighting the last season in whichever way I do it makes it exponentially more complicated. If I could find a way to make SRS that includes preseason data or at the very least last season's scores but is less complicated than what I've tested so far, I would probably use that as a predictor before Elo.
3. Similar to the modified margin formula, accounting for returning production doesn't do much at all. It corrects for about 13 games over ten years, which I imagine is just noise, and tuning for 2013-2019 also didn't make much of a difference for 2020 to 2025. I would be willing to test for Vegas win totals, but I'm not too sure where to find that kind of information. I've been using CollegeFootballData.com to get all of my data, but I couldn't find that there.

For the SRS chain, 2017 and 2025 look pretty random compared to each other. If anything, they each did good in something completely different (for example, 2017 did good in the early season and in picking power vs power schools, while 2025 did much better in the later season and in games involving G5 teams)

[u/TigerWave01 → u/Quaizimo]

I kept toying around with that SRS system and seemed to make something far better, both in terms of picks, ATS, and as a basis for resume building, with it. It is slightly more complex, but still uses 100% open source data and can probably be recreated relatively easily, at least with a spreadsheet.

The preseason rating is last season's final rating plus a returning production adjustment, all from public box scores and rosters: 20 x (share of last year's rushing and receiving yards from returning players - the division average) + 20 x (share of tackles from returning players - the division average) + 5 x (share of passing yards from returning players - the division average). On CFBD, FBS data had returning production going back to at least 2014, but FCS data started much later, around 2022. In those cases, the FCS teams didn't have an adjustment for returning players.

Once in the season begins, every team's rating (both FBS and FCS) follows the standard SRS formula (average adjusted margin + average rating of its opponents - h), with "h" being the 2.5-point home edge. Each team also carries 5 "imaginary games" at its preseason rating to prevent early-season ratings swings. After doing a few tests, it seems like two prior seasons is needed to maximize accuracy, while any more doesn't add any sort of positives or negatives (so with this rating being set between 2016-2025, the data starts in 2014).

To try and see how this model performed compared to others, I compared the Chained SRS to results from other models on ThePredictionTracker (FBS vs FBS, bowls included), over two five year windows and one ten year window. The reason for the multiple windows is that this formula's weights were determined by using 2016-2019 data, which suggests to me that the numbers from 2020-2025 will be more accurate to what you would expect going forward:

• 2016-2019: 74.2% picks, 1st of 51 systems; 51.7% against the closing spread, 4th of 48
• 2020-2025: 71.4% picks, 14th of 30 (the betting lines lead at about 72.3%); 50.9% ATS, 1st of 27
• Full 2016-2025: 72.5% picks, 7th of 28; 51.2% ATS, 2nd of 25

It also produces a résumé ranking, SRS Chain W25: wins above what an average No. 25 team would be expected to win against the same schedule. What's different from the other W25 systems I had, this one includes a head-to-head swap when two adjacent teams played each other and the lower ranked team wins. However, this seems a little forced to me so I'm on the fence about whether to include it (it doesn't change anything about picks or ATS calculations, just personal preference on that one). Against the final pre-bowl CFP rankings it matched 224 of 250 top-25 spots and the No. 1 team in 7 of 10 seasons.

I've made two separate spreadsheets, which I'll attach here: an FBS spreadsheet and an FCS spreadsheet. Both spreadsheets are sorted by W25 rankings, but can be sorted by SRS Chain ratings and they both include a Formulas sheet at the end, which includes more correctly written out formulas and, in some instances, a brief note for clarity. This is the best system I've tried so far; I'm still going to toy around with it to see if I can get the picks from 2020-2025 in the top 10 computer systems, which I think taking your suggestion about home edge may be helpful for. Any other suggestions would be appreciated!