Football Probability Calibration: Brier Score, Log Loss and Reliability

Football probability calibration asks whether stated probabilities match observed frequencies. Evaluate football 1X2 forecasts with multiclass Brier score, log loss and classwise reliability tables. The calculator accepts one match or a batch, while the methodology below explains calibration, sharpness, resolution and defensible out-of-sample validation.

In one minute
  • Calibration: events forecast at 60% should occur about 60% of the time.
  • Sharpness: forecasts should be informative rather than clustered near the base rate.
  • Brier score: squared error across the full probability vector; lower is better.
  • Log loss: penalizes low probability assigned to the outcome that actually occurred.
  • Reliability diagrams: show where probabilities are systematically too high or too low.
  • Out-of-sample testing: is required; in-sample fit overstates performance.

Football Probability Evaluation Calculator

Paste one or more 1X2 forecasts as Home, Draw, Away, Result. An optional fifth column can contain a group label such as league, season or model version. Probabilities may be entered as decimals (0.50, 0.28, 0.22) or percentages (50, 28, 22). Use one unit convention per row. A % suffix on any of the three values explicitly selects percentage units for that row; for example, 1%, 0%, 99% means 0.01, 0, 0.99. Without a suffix, values above 1 select percentage units; otherwise the row is read as decimal probabilities.

Scoring convention used here: multiclass Brier is the unscaled sum across Home/Draw/Away, so a single-match score ranges from 0 to 2. Log loss uses the natural logarithm. The primary log loss is −log of the probability assigned to the observed result: it is ∞ when that probability is zero. A separate ε-smoothed comparison floors every component below 10−15 and renormalizes the complete vector. Brier score and reliability use the submitted probabilities after sum normalization.

Accepted results: H / Home / 1, D / Draw / X, A / Away / 2. Use one comma, semicolon or tab separator consistently throughout the batch. With semicolons or tabs, decimal commas are accepted. Decimal scientific notation is also accepted. Quote single-line fields containing separators and double any literal quote inside a quoted field.

6

Matches

0.5656

Mean Brier

0.9418

Mean raw log loss

66.67%

Top-pick accuracy

0

Rows affected by ε smoothing

No rows required probability-sum rescaling.

No probabilities were changed by ε smoothing; raw and smoothed scores agree to display precision.

Per-match scores

# H D A Result Actual p Brier Raw log loss Top pick Group
150.00%28.00%22.00%H50.00%0.37680.6931CorrectSample A
245.00%30.00%25.00%D30.00%0.75501.2040WrongSample A
362.00%23.00%15.00%H62.00%0.21980.4780CorrectSample A
434.00%31.00%35.00%A35.00%0.63421.0498CorrectSample B
555.00%27.00%18.00%D27.00%0.86781.3093WrongSample B
640.00%30.00%30.00%H40.00%0.54000.9163CorrectSample B

Group summary

Group n Mean Brier Mean raw log loss Top-pick accuracy
Sample A30.45050.791766.67%
Sample B30.68071.091866.67%

Fixed-width reliability table

Home win: 10 percentage-point bins using the submitted probabilities.

Forecast bin n Mean forecast Observed rate Gap
30–40%134.00%0.00%−34.00pp
40–50%242.50%50.00%+7.50pp
50–60%252.50%50.00%−2.50pp
60–70%162.00%100.00%+38.00pp

If two outcomes share the highest submitted probability, top-pick accuracy uses the first maximum in Home / Draw / Away order. The reliability table reports sample size but no uncertainty band; with few matches per bin, apparent gaps can reflect sampling noise. Bins include their lower boundary and exclude their upper boundary, except that the final bin includes 100%.

Rows are rejected if a probability is outside 0–100% or if the three values sum outside 99.5–100.5%. Small rounding discrepancies inside that band are rescaled to exactly 100% and counted in the note above. The calculator does not infer missing probabilities or outcomes.

Why prediction accuracy is not enough

Classification accuracy reduces every match to one selected outcome: home, draw or away. It ignores the rest of the probability distribution. How often top-ranked exact scores should land at all is a separate question, analysed in the most-likely-score accuracy study.

Forecast Home Draw Away Top pick
Model A 51% 28% 21% Home
Model B 90% 7% 3% Home

If the home team wins, a simple accuracy measure gives both models the same result. If the away team wins, it again gives both the same result: wrong. But Model B made a much stronger and more damaging error.

Accuracy also depends heavily on the mix of matches. A model evaluated on a league or sample containing many strong home favourites can achieve a high hit rate by repeatedly choosing the home team. That does not prove that its probabilities are accurate.

A probability forecast makes two claims: which outcomes are more likely, and how much more likely they are. Evaluation should test both.

What does calibration mean?

A forecast is calibrated when its probabilities agree with long-run observed frequencies. For a binary event:

P(Y = 1 | forecast probability = p) = p

If a model assigns a 60% home-win probability to many matches, approximately 60% of those home teams should win. Calibration is a property of a set of forecasts and outcomes, not of one match.

Average forecast Matches Home wins Observed rate Calibration gap
60% 100 59 59% −1pp
60% 100 42 42% −18pp

The first group is close to calibrated. The second is overconfident in home wins: it states 60%, but only 42% occur.

Calibration is conditional

A model can look calibrated overall while failing within important subgroups. For example, its total home-win probability may match the overall home-win rate, yet it may systematically:

  • overrate favourites and underrate underdogs;
  • underestimate draws;
  • work in one league but not another;
  • work early in a season but drift after team strength changes;
  • miscalibrate neutral-venue matches, congested schedules or fixtures with elevated pre-match disciplinary risk.

Useful analysis therefore checks calibration by probability range, outcome, league, season and other pre-specified segments. Segments must be defined from pre-match information: a pre-match model cannot be graded on subgroups selected by in-match events such as an actual red card, because the event itself depends on how the match unfolded. Stratify by pre-match risk instead, or treat event-defined splits as descriptive post-match diagnostics only.

Calibration, sharpness and resolution

Calibration alone is not sufficient. A forecast that always reports the historical base rates can be calibrated but uninformative.

Uninformative but potentially calibrated

Suppose a league historically produces 45% home wins, 27% draws and 28% away wins. A forecast that issues 45/27/28 for every fixture may be calibrated in aggregate, but it does not distinguish a dominant home favourite from an evenly matched game.

Calibration Do observed frequencies agree with stated probabilities?
Sharpness How concentrated and decisive are the forecasts? This is a property of the forecasts alone.
Resolution Does the model separate groups with genuinely different observed outcome rates?

The classical forecasting objective is to maximize sharpness subject to calibration, a framework developed by Gneiting, Balabdaoui & Raftery (2007). Resolution is a related but outcome-dependent property: it measures whether the forecasts separate groups with genuinely different observed frequencies. Confidence is useful only when it is justified.

Sharpness is not automatically good. An overconfident model also produces extreme probabilities. Proper scoring rules penalize unsupported sharpness.

Why use proper scoring rules?

A scoring rule converts a probability forecast and the subsequent result into a numerical loss or reward. A rule is proper when the forecaster minimizes expected loss by reporting its true probability assessment. It is strictly proper when truthful reporting is uniquely optimal.

This matters because some intuitive metrics can be improved by distorting probabilities. Brier score and logarithmic score are strictly proper under their standard definitions; see Gneiting & Raftery (2007) for the general scoring-rule framework.

Use the Football Probability Evaluation Calculator above to apply both scores to a single 1X2 forecast or a batch.

Metric Uses full probabilities? Proper? Main behaviour
Classification accuracy No Proper, but not strictly proper Uses only the highest-probability outcome
Profit or ROI Not necessarily Not a probability scoring rule Depends on prices, staking, limits and sample variance
Brier score Yes Yes Quadratic penalty across every outcome
Log loss Uses probability of realized outcome Yes Strong penalty for confident errors
Ranked probability score Yes Yes Uses an assumed ordering of categories

Propriety applies to probability forecasts. A hard class label on its own has no propriety status; when accuracy is induced from a probability vector through its highest-probability class, the zero-one score is proper but not strictly proper — the true distribution is among the optimal reports, but so is any distortion that keeps the same top class. That is exactly why accuracy does not reward honest probability magnitudes.

The ranked probability score is sometimes used for 1X2 forecasts because home–draw–away can be placed in an order. Whether an away win should be treated as “farther” from a home win than a draw is a methodological choice, not an uncontested fact; Wheatcroft (2021) argues against the ranked probability score for football forecasts. Brier score and log loss avoid that particular assumption.

Brier score for football 1X2 probabilities

For N matches and K mutually exclusive outcomes, the multiclass Brier score can be written as:

BS = (1/N) Σi=1…N Σk=1…K (pik − oik)²

For football 1X2, K = 3. The forecast vector contains probabilities for home, draw and away. The observed vector is one-hot encoded: the realized outcome receives 1 and the others receive 0.

ImpliedScore convention: all football 1X2 evaluation on this site uses the unscaled three-class sum shown above. We do not divide the per-match result by 2 or by K. Under this convention the single-match range is 0 to 2.

Worked Brier example

A model forecasts 50% home, 28% draw and 22% away. The home team wins, so the outcome vector is 1/0/0:

BS = (0.50 − 1)² + (0.28 − 0)² + (0.22 − 0)² = 0.3768

A more confident 70/18/12 forecast would score 0.1368 if the home team won. The stronger forecast is rewarded because the outcome supported it.

If a model instead forecast 60/25/15 and the away team won:

BS = 0.60² + 0.25² + (0.15 − 1)² = 1.1450

Lower is better—but state the convention

Under the original multiclass sum shown above, the score ranges from 0 to 2. Some software divides the result by 2 or by the number of classes, producing a different numerical scale. Model comparisons are valid only when the same convention is used.

Reporting rule: publish the formula with the score. “Brier score = 0.18” is ambiguous without the normalization convention and the outcome definition.

What Brier score captures

Brier score reflects both calibration and the ability to issue informative probabilities. It is less extreme than log loss because its penalty grows quadratically and remains bounded in the categorical formulation.

Log loss: the cost of being confidently wrong

Multiclass log loss—also called logarithmic loss, cross-entropy or ignorance loss—depends on the probability assigned to the outcome that occurred:

Log loss = −(1/N) Σi=1…N log(pi,actual)

Lower is better. A perfect forecast assigns probability 1 to the observed outcome and has loss 0.

Probability assigned to actual outcome Single-match log loss
70% 0.3567
50% 0.6931
15% 1.8971
5% 2.9957
1% 4.6052

Log loss is unbounded. Assigning zero probability to an outcome that occurs produces infinite loss. This is a meaningful failure of the forecast, so it should not be silently replaced by a finite number. An optional smoothed comparison can floor probabilities at ε, but that transforms the submitted distribution. Report raw scores separately, apply the same ε to every model, and normalize the full transformed vector:

p′k = max(pk, ε) / Σj max(pj, ε)

And report how many forecasts the clipping actually touched. An overly large ε can materially change model rankings, because extremely small probabilities are exactly where log-loss differences concentrate.

The calculator uses the natural logarithm. Its headline, per-match and group log losses use the original normalized forecast, including infinite loss for an observed zero-probability outcome. It also reports the ε = 10−15 smoothed mean, labels affected per-match and group comparisons with ε, and counts every row containing a probability below that floor. Smoothing can affect tiny positive values as well as zeros. Standard log loss is strictly proper for the probability vector being scored; applying a non-invertible floor first does not preserve strict propriety for the original submitted vector. Brier score and reliability are calculated before this separate smoothing step.

Brier score vs log loss: which should you use?

Property Brier score Raw log loss
Penalty shape Quadratic Logarithmic
Confident errors Penalized, but bounded Penalized very strongly
Uses non-realized classes Yes, directly Only through the probability-sum constraint
Interpretability Squared probability error Average surprise of the realized outcomes
Recommended use Report alongside calibration diagnostics Report when overconfidence is a major concern

No single proper score answers every diagnostic question. A defensible evaluation normally reports both Brier score and log loss, then uses reliability diagrams to identify where the errors occur.

How to build a football reliability diagram

A reliability diagram plots average forecast probability against observed event frequency. Perfect calibration lies on the 45-degree line.

Binary, classwise approach

For 1X2 forecasts, create three one-vs-rest diagrams:

  • home win versus not home win;
  • draw versus not draw;
  • away win versus not away win.

For each outcome:

  1. Collect the forecast probability from every match.
  2. Split the probabilities into bins.
  3. Calculate the mean forecast within each bin.
  4. Calculate the observed event rate within each bin.
  5. Plot observed rate against mean forecast.
  6. Show the number of matches and uncertainty for each point.

Classwise one-vs-rest diagrams test the marginal calibration of each outcome. Passing all three checks is necessary but does not establish that the complete home/draw/away vector is calibrated conditional on the full forecast: Vaicenavicius and co-authors give an explicit example in which every marginal prediction is calibrated while full reliability fails. A deeper multiclass analysis can group forecasts within the probability simplex or use multidimensional calibration tests; for practical league work, the three classwise diagrams remain the standard first tool.

Be precise about which uncertainty the bands show. A confidence band asks where the true calibration curve lies; a consistency band asks how far the points could stray from the diagonal by chance alone if the model were perfectly calibrated — the CORP methodology of Dimitriadis, Gneiting and Jordan draws this distinction explicitly. Within one bin the forecast probabilities usually differ. If matches are also independent, under perfect calibration the success count follows a Poisson-binomial distribution built from the individual probabilities, not a plain binomial at the bin average; it can be evaluated exactly or by simulating Bernoulli outcomes match by match. Matches within a league season can also be dependent, so purely independent resampling may understate the width of either band.

Home-probability bin n Mean forecast Observed home wins Gap
10–20% 420 15.4% 14.8% −0.6pp
20–30% 510 25.2% 28.0% +2.8pp
30–40% 590 35.1% 34.6% −0.5pp

Illustrative values only. A point above the diagonal means the event occurred more often than forecast; a point below it means the model assigned too much probability.

Fixed-width bins versus quantile bins

Method Advantage Risk
Fixed-width bins Easy to interpret across models and datasets Extreme bins may contain very few matches
Equal-count / quantile bins More stable sample size per point Bin ranges differ between models and can hide local structure
Isotonic or smooth calibration curve Avoids some arbitrary visual jumps Adds modelling choices and can overfit small samples

Reliability diagrams are sensitive to bin choice. Changing boundaries can change the visual conclusion, particularly in sparse probability regions. Publish the bin definitions, sample counts and confidence intervals rather than showing an unqualified line.

Do not show only top-pick confidence. A model can calibrate its selected favourite while systematically mispricing draws. Classwise diagrams expose that failure.

Brier score decomposition

For a binary event, the classical Murphy decomposition separates the Brier score into three components:

Brier score = Uncertainty − Resolution + Reliability
Uncertainty The inherent variability of the event. It depends on the base rate and is outside the forecaster’s control.
Resolution How much observed event rates differ across forecast groups. Higher resolution reduces the score.
Reliability Squared disagreement between forecast probabilities and observed frequencies. Lower is better.

Consider two calibrated systems. One always forecasts the league base rate. The other identifies matches with genuinely low and high home-win rates. Both may have low reliability error, but the second has better resolution.

In a finite-sample Murphy decomposition, grouping by identical forecast probabilities gives the classical exact identity. If distinct forecasts are pooled into broad bins and replaced by bin means, the simple three-term calculation instead describes the coarsened forecasts; it need not equal the raw Brier score. For example, forecasts 0.2 and 0.8 with outcomes 0 and 1 have raw binary Brier score 0.04. Putting both in one bin and replacing both probabilities by 0.5 gives score 0.25. This is a change in the forecasts being scored, not a rounding error.

Empirical decomposition depends on how forecasts are grouped or calibrated. Bin-based estimates can be biased or unstable, and multiclass extensions require explicit definitions; Foulley (2021) applies score-decomposition and reliability analysis to football outcomes. The decomposition should therefore supplement—not replace—the raw proper scores and diagrams.

A defensible football forecast validation workflow

1. Freeze the forecast before the match

Store the full probability vector, timestamp and model version. A backtest is invalid if lineups, closing odds or final results leak into a forecast that is supposed to represent an earlier time.

2. Use chronological out-of-sample splits

Train on earlier matches and evaluate on later matches. Randomly mixing past and future fixtures can leak season and team information across the split. Rolling or expanding windows are usually more realistic.

3. Compare the same matches

Two models must be scored on the identical fixture set. Excluding hard matches from one model can create a false performance advantage.

4. Include meaningful reference forecasts

  • league or season base rates;
  • a simple Poisson model (the candidate structures are compared in Poisson vs Dixon-Coles vs bivariate Poisson);
  • margin-free closing bookmaker probabilities;
  • the previous production model.

5. Report multiple diagnostics

  • multiclass Brier score with the exact convention;
  • log loss and probability-clipping rule;
  • classwise reliability diagrams;
  • sample counts and uncertainty intervals;
  • performance by league, season and probability range.

6. Quantify uncertainty in model differences

A lower average score does not automatically mean a meaningful improvement. Match-level losses are paired observations, so model differences can be assessed with a paired bootstrap or another time-aware resampling method. Resampling by match alone may understate uncertainty when fixtures within a season are dependent.

7. Keep evaluation separate from recalibration

Platt scaling, isotonic regression and other calibration methods must be fitted on training or validation data, then tested on untouched matches. Fitting and evaluating a calibration curve on the same sample creates optimistic results.

For 1X2 forecasts, recalibration should also preserve a coherent three-outcome vector. Platt scaling and isotonic regression are binary methods; applying them separately to home, draw and away can produce probabilities that no longer sum to one or that reorder the outcomes, so the result requires renormalization and full-vector validation. Dedicated multinomial approaches — multinomial logistic recalibration, temperature scaling or Dirichlet calibration — keep the vector coherent by construction.

How to evaluate bookmaker-implied probabilities

For a complete standard market from one bookmaker at one timestamp, reciprocal odds will normally sum to more than 100% because of the overround. Promotional, stale or cross-book prices can produce a booksum at or below 100%, so the input set and its source should always be checked, and the outcomes must be mutually exclusive, exhaustive and taken from one market snapshot. The overround must be removed before calibration or proper scores are calculated; the main approaches are compared in bookmaker margin removal methods.

qk = 1 / decimal oddsk
pk = qk / Σq

Proportional normalization is transparent, but it is not the only de-vig method. Shin, power and other methods distribute margin differently. If two studies use different methods, part of the score difference may come from de-vigging rather than forecasting quality.

Opening odds and closing odds answer different questions

  • Opening odds test early market information and are more exposed to later news.
  • Closing odds usually represent the market after more information and liquidity.
  • Best available odds are useful for betting execution but may not form one coherent probability vector.
  • Average market odds reduce bookmaker-specific noise but can blur timing differences.

The source, timestamp and margin-removal method should be recorded for every observation. For an introduction to what these evaluations can establish, read how accurate betting odds are: calibration and the Brier score.

Calibration is not profitability. A well-calibrated market can still offer negative expected returns after margin. Conversely, a short profitable backtest can be driven by variance while the underlying probabilities remain poor.

How this applies to the ImpliedScore calibration study

ImpliedScore currently compares margin-free bookmaker probabilities with observed results across 19,381 matches from five major European leagues. Prices come from football-data.co.uk market-average AvgH/AvgD/AvgA non-closing columns and are de-vigged proportionally. Matches are grouped by implied home-win probability, and each group’s mean implied rate is compared with its actual home-win rate.

That is a valid calibration diagnostic. It can reveal patterns such as favourites winning more often than implied or draws occurring more often in certain match classes. See the football market calibration study.

But maximum bin deviation is not a complete forecast score. It depends on the binning scheme, ignores how errors are distributed across all matches and does not by itself compare the full home/draw/away probability vectors.

The current study should also be read as descriptive. Its headline gaps are the largest found across many leagues, outcome classes and probability groups, and the largest of many gaps is expected to exceed the typical gap even when no persistent pattern exists. Treating them as formal evidence of market bias would require pre-specified bins, uncertainty intervals, season-level stability checks and an accounting for multiple comparisons.

What the evaluation calculator measures

The calculator at the top of this page scores complete Home/Draw/Away probability vectors against realized 1X2 outcomes. It reports match-level and mean multiclass Brier score, natural-log loss, top-pick accuracy, optional group summaries and fixed-width classwise reliability tables. Those outputs are diagnostics for a supplied forecast sample; they do not replace chronological validation, uncertainty intervals or external reference forecasts.

Comparing the market with the ImpliedScore matrix

ImpliedScore fits home and away goal rates to reproduce margin-free 1X2 and Over/Under market probabilities. The inversion from de-vigged prices to those goal rates is documented in market-implied goals from betting odds. Therefore, its resulting 1X2 probabilities are not independent forecasts in the same sense as a historical team model: the market probabilities are inputs to the fit.

Comparing ImpliedScore’s fitted 1X2 output with the source market on the same matches mainly measures approximation error and the constraints of the Poisson or Dixon–Coles matrix. It does not test whether ImpliedScore contains new predictive information beyond the market.

The more meaningful tests are:

  • how closely the fitted matrix reproduces all input markets;
  • whether Poisson or Dixon–Coles gives better out-of-sample exact-score probabilities;
  • whether league-specific ρ values remain stable over time;
  • whether the score matrix is calibrated at 0-0, 1-0, 0-1, 1-1 and other scorelines;
  • whether adding Over/Under information improves scoreline log loss.

The site’s methodology documents how market odds are de-vigged and converted into λ values. The Dixon–Coles guide explains the low-score correction, while the Maher model guide describes the historical attack–defence alternative.

Convert margin-free 1X2 and Over/Under probabilities into a complete football score matrix.

Open the score probability calculator

Common evaluation mistakes

Using only the percentage of correct favourites

This discards probability magnitude and rewards models for exploiting class imbalance.

Calculating calibration on raw bookmaker odds

The overround must be removed first. Otherwise the “probabilities” do not sum to one.

Choosing bins after seeing the result

Repeatedly changing boundaries until a pattern appears is a form of researcher discretion. Pre-specify the main binning method and show robustness checks.

Ignoring sample size in extreme bins

Probabilities above 80% or below 10% may be rare. Large apparent gaps can be ordinary sampling noise. Always display n and an uncertainty interval.

Pooling seasons without checking drift

A model can appear calibrated over ten seasons while being systematically wrong in the most recent two. Report rolling or season-level results.

Evaluating on training data

A flexible model can fit historical outcomes well without forecasting new matches well.

Comparing different fixture sets

Missing odds, promoted teams or data cleaning can cause models to be scored on different samples. Use paired comparisons.

Treating ROI as proof of probability quality

ROI combines forecast error, offered price, staking and variance. It is an economic outcome, not a proper probability score.

Publishing one score without a baseline

A Brier score has little meaning in isolation. Compare it with base rates, bookmaker probabilities and simpler models on the same matches.

Recommended reporting template

  • Forecast: full home/draw/away probability vector saved before kickoff.
  • Sample: leagues, seasons, exclusions and number of matches.
  • Odds: source, timestamp and de-vig method.
  • Brier: exact multiclass formula and normalization.
  • Log loss: logarithm base, raw score, any separately reported smoothing threshold and affected-row count.
  • Calibration: classwise diagrams, bin rules, n and uncertainty bands.
  • Validation: chronological out-of-sample design.
  • Comparison: same matches, paired score differences and reference models.
  • Limitations: drift, dependence, missing data and retrospective choices.

References

  1. Brier, G. W. (1950). Verification of Forecasts Expressed in Terms of Probability . Monthly Weather Review, 78(1), 1–3.
  2. Murphy, A. H. (1973). A New Vector Partition of the Probability Score . Journal of Applied Meteorology, 12(4), 595–600.
  3. Gneiting, T. & Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation . Journal of the American Statistical Association, 102(477), 359–378.
  4. Gneiting, T., Balabdaoui, F. & Raftery, A. E. (2007). Probabilistic Forecasts, Calibration and Sharpness . Journal of the Royal Statistical Society: Series B, 69(2), 243–268.
  5. Vaicenavicius, J., Widmann, D., Andersson, C., Lindsten, F., Roll, J. & Schön, T. B. (2019). Evaluating Model Calibration in Classification . Proceedings of Machine Learning Research, 89, 3459–3467.
  6. Dimitriadis, T., Gneiting, T. & Jordan, A. I. (2021). Stable Reliability Diagrams for Probabilistic Classifiers . Proceedings of the National Academy of Sciences, 118(8), e2016191118.
  7. Foulley, J.-L. (2021). More on Verification of Probability Forecasts for Football Outcomes: Score Decompositions, Reliability, and Discrimination Analyses . arXiv:2106.14345.
  8. Wheatcroft, E. (2021). Evaluating Probabilistic Forecasts of Football Matches: The Case Against the Ranked Probability Score . Journal of Quantitative Analysis in Sports, 17(4), 273–287.
Scroll to Top