r/CFBAnalysis • u/Upper_Sympathy6862 • 12d ago

Updated my CFB model: backtested rankings, a public pick ledger, and a Discord bot you can ask questions

I've been building Fourth & Data (fourth-data.com), a free college football analytics site, and just shipped a round of updates. Looking for feedback from people who care about the methodology.
What's new:
• Rebuilt rankings: a blend of résumé (Colley), margin (Massey, capped at 28), and SP+ power, with power weighted heavily early and fading as the season goes. I walk-forward backtested it on 2017–2025 (~5,900 games). Surprising finding: pure résumé was the weakest method, so it now carries much less weight than margin.
• Public pick ledger: every pick is logged and graded, misses included. Currently 229–62 straight up (78.7%). ATS is 30–26, which is too small a sample to mean much yet, and I'm tracking against the line at pick time.
• Filters: sort the ledger by team, conference, and season to see where the model is strong or blind.
• Ask Fourth & Data on Discord: a bot that answers questions about matchups, rankings, and teams using the site's data, and posts alerts. Join here: fourth-data.com/api/discord/invite
I'd genuinely like to hear where the model looks wrong to you, or what you'd want added. Happy to answer methodology questions in the comments.

8 Upvotes
  • permalink
  • reddit

84% Upvoted

6 comments sorted by

4

u/investigatormaker 12d ago

Fourth & Data's walk-forward backtest needs one detail made explicit: whether the SP+ input for each historical game was the rating available before kickoff. Using a later rating would let future information into the test, even if the blending weights were applied chronologically.

I make ThreadFox. I can post the Reddit communities whose rules allow a post about Fourth & Data here, with each rule quoted, if you want them.

Were those SP+ inputs archived weekly snapshots?

1

u/Upper_Sympathy6862 12d ago

Good question, and no, they weren't weekly snapshots. CFBD only serves end-of-season SP+, so there's no pre-kickoff version to pull for past weeks. The results I posted used the previous season's final SP+ as the power input, so nothing from the season being tested leaks in, but it's a weaker input than a true weekly rating would be. I also ran it with same-season final SP+, and you're right that that version leaks future info, so I don't lean on it. If anyone knows a source of archived weekly SP+, I'd happily rerun with it.

1

u/ReleaseEuphoric6166 11d ago

Nice work putting the full ledger up with the misses in it. Most people don't.

A few things from running a similar CFB pipeline:

  1. Team efficiency can leak too, not just SP+. If any of your efficiency inputs come from CFBD's /stats/season/advanced, that endpoint ignores the week parameter and returns full-season numbers. We pulled weeks 3, 8 and 14 and got identical values. In a backtest that's the same problem as same-season SP+. We rebuilt ours from per-game box scores using prior weeks only, and most of our "early-season edge" went away when we did. (quaizimo.com)
  2. Are weeks 2–5 in the ledger logged live before kickoff, or rebuilt from the model afterwards? If they're live, posting a hash or a dated commit of each week's file before Saturday lets anyone verify it without having to trust the site.
  3. Since CFBD only keeps final SP+, saving your own snapshot every week this season gives you a clean pre-kickoff input to backtest on next year.
  4. The 229–62 is mostly the 90%+ band (87–3). Splitting out FBS-vs-FBS games would show what the model really does. Straight-up also only pays if you beat the price: a 96% favorite is priced around −2000, so 87–3 is roughly break-even after the vig. Your calibration table is the right tool for this. Adding ROI at the posted moneyline next to it would answer the betting question. Your 80–89% band hits 77.8% against an 85.6% average forecast, so that's where I'd look first.
  5. For the ATS picks, track closing line value alongside W-L. At 30–26 it'll tell you much sooner than the record whether the 8-point threshold is real.

Does the weekly re-tune change past projections, or only future weeks?

1

u/Upper_Sympathy6862 11d ago

Thanks, this is the kind of feedback I was hoping for.
No season-level stats in the backtest. Efficiency is built per game from prior weeks only, SP+ is last season's, and returning production/talent are preseason. I'm saving weekly SP+, line movement, and pregame weather forecasts going forward so next year's backtest has clean inputs.
Picks are stored before kickoff and never edited. Starting this week I'll post a hash of each week's locked picks before Saturday so anyone can verify.
You're right on the 80–89% band. It's the older system, which said 85.6% and went 35–10. That's only 45 picks, so it's partly noise, but it's overconfident there and underconfident at 60–79%. The new model checks out better on 2022–25 holdout: 84.9% stated vs 84.8% actual in that band. Its weak spot is the 50–59% bucket (said 55%, won 50%), so I'm recalibrating that now.
Agreed on the FBS-only split and moneyline ROI. Adding both next to the calibration table.
CLV is tracked on the ATS picks. Honestly, the backtest found no proven ATS edge at any threshold, and the site says so.
Re-tunes only affect future weeks. Locked picks never change.

1

u/Upper_Sympathy6862 11d ago

Thanks, this was genuinely useful. Did all of it today:
Backtest has no season-level stats. Efficiency is built per game from prior weeks only, and SP+ is last season's. I'm now snapshotting SP+, line movement, and pregame weather forecasts weekly for next year's backtest.
Picks are stored before kickoff and never edited, and now they're verifiable. Every Tuesday before the first kickoff, a SHA-256 of the week's picks gets posted to our Discord, and the exact file is downloadable from the site. Week 6: ad2580cc76b74cff6a574553e3e337f3b817a579ac49ddb8f5a8775c1e67feea.
You were right about the record. FBS vs FBS only is 166–54 (75.5%). Betting every pick at the closing moneyline is −2.7% ROI, and the 90%+ band (87–3) is +0.5%, basically break-even like you said. Both are on the Accuracy page now, next to calibration.
The 80–89% band is the older system (35–10 vs 85.6% stated). The new model checks out at 84.9% stated vs 84.8% actual on 2022–25 holdout. I also tested recalibrating its 50–59% band, but across 8 seasons it hits 55.6%, so the recent dip is noise and I left it alone.
CLV is tracked on the ATS picks. The backtest found no proven ATS edge, and the site says so.
Re-tunes only affect future weeks. Locked picks never change.