A benchmark score needs a denominator

Abstract

"85% accurate" is not a result. It is the end of a sentence whose beginning is missing: accurate on which task, over which population, measured how, and out of how many cases? Specialised models — small models adapted to one job, such as triaging support requests or ranking expiring domains for review — make the missing part more important, not less. They are usually compared with a larger general model, on data the team chose, under conditions the team controls. This article sets out the method we use at Namecase to keep those comparisons honest. It separates three measurements that are often blended into one number (quality, coverage and calibration), shows how to compare a specialised model with its base on equal terms, and argues that the final evaluation belongs to the workflow the model serves, not to a leaderboard. It closes with the minimum record an evaluation should keep to be reproducible. It reports no benchmark result for any of our models: it describes a method.

1. Four things every result needs

A score needs a task, a population, a metric and a denominator. Remove any of them and the number stops meaning anything a reader can check.

Consider a model that reads a support email and fills seven fields: the language, the product, the intent, the urgency, the customer reference, the domain name involved and whether a human must answer. A report that the model is "95% accurate" may be counting fields. If each field were right 95% of the time and the errors were independent, the share of emails with all seven fields right would be

0.95⁷ ≈ 0.70

so roughly seven emails in ten would come out entirely correct. In practice errors are not independent: a hard email tends to produce several mistakes at once, which pushes the case-level figure up, while a missing reference can cascade into a wrong product and a wrong intent. The only way to know is to measure both, and to say which one a headline number is.

The population matters as much. A model evaluated on emails a human had already sorted into "clear" cases will look better than one evaluated on the full inbox. Both numbers can be true. Only one describes what the model will meet in production.

2. Preserve the task

External benchmarks are useful precisely because they ask questions outside the training domain. They become misleading when an adapter quietly makes the question easier. Three adaptations are common and rarely reported:

• Truncation. Long inputs are cut to fit a context window. The benchmark now measures the beginning of the documents, not the documents.
• Narrowed options. Candidate labels are dropped to fit a prompt or a classification head. Choosing among five labels is not choosing among twenty.
• Removed context. Links, attachments or previous turns of a conversation are stripped because the harness cannot carry them.

None of these is forbidden. Each must be named. A function-calling benchmark adapted to check only which tool a model picks — not the arguments it fills, nor whether the call succeeds — measures tool selection, and the report should say "tool-selection subset of …" instead of borrowing the reputation of the full benchmark. When we adapt an external benchmark we keep the full inputs and the full set of options wherever the model can accept them, and we record every request it could not accept rather than dropping it.

3. Three measurements, kept apart

3.1 Quality

Quality asks whether the predictions match a reference. The right metric depends on the question:

Macro-averaged scores deserve particular care with imbalanced data. A support model that never predicts the rare "legal complaint" class can still post a high accuracy; its macro-F1 will not hide the gap.

3.2 Coverage

Coverage asks how much of the workload the system answers at all. Specialised systems often abstain — by design, when confidence is low, or by accident, when an input is too long or in an unsupported language. Abstaining can be the right behaviour. It still has to be counted.

Suppose a model answers 800 of 1,000 requests and is right on 90% of those it answers. Its accuracy on the answered requests is 90%. Its accuracy over the workload is

720 correct ÷ 1,000 requests = 72%.

A second model that answers everything at 80% is better over the workload and worse on what it chooses to answer. Which one is preferable depends on what happens to the 200 unanswered requests: if they go to a person who handles them well, the first model may be the better system. The report has to show both numbers, and the trade-off between them: a risk–coverage curve, which plots the error rate on the answered cases as the abstention threshold moves, is the standard way to do so (Geifman and El-Yaniv, 2017).

3.3 Calibration

Calibration asks whether a stated confidence means what it says. If a model reports 0.9 on a hundred predictions, about ninety of them should be right. If seventy-two are, the model is over-confident by eighteen points, and any rule that acts on "confidence above 0.9" is acting on a fiction.

Two measures are common. The Brier score is the mean squared difference between the stated probability and the outcome (1 if right, 0 if wrong); lower is better, and it rewards both accuracy and honest probabilities (Brier, 1950). The expected calibration error (ECE) groups predictions into confidence bins and averages the gap between each bin's confidence and its accuracy, weighted by the size of the bin:

ECE = Σₘ (|Bₘ| ÷ n) · |acc(Bₘ) − conf(Bₘ)|

ECE is easy to read and easy to game: the number of bins and their edges change it. We report the binning with the figure and, where volumes allow, a reliability diagram rather than a single number (Naeini et al., 2015; Guo et al., 2017).

Calibration also has a scope. A temperature fitted on Italian support emails with eight intents is not automatically valid for Spanish emails or for twelve intents. Re-measure it where it will be used.

Finally, a calibrated probability is still a probability. A prediction at 0.97 is not a verified fact, and it does not authorise an operation. Wherever a model's output can trigger an action — a refund, a domain transfer, a message to a customer — the action needs its own check, outside the model.

4. Compare the same system

A "model" in an evaluation is really a configuration: the checkpoint, the tokenizer, the runtime and its version, the prompt template and how it is rendered, the decoding parameters, the token budget and any calibration applied afterwards. Change one and you are evaluating a different system. Comparisons that look like model A against model B are often model A on a warm GPU against model B behind an HTTP call on a CPU; the latency gap is then a property of the set-up, not of the models.

Before a result is published we freeze the configuration and record it with the result (section 7). When we compare a specialised model with the general model it was adapted from, we hold everything else fixed and run both on the same items.

Running both on the same items allows a paired comparison, which is far more sensitive than comparing two accuracies. Count the items where the models disagree: b items the base model gets right and the specialised model gets wrong, c items the other way round. Items both get right or both get wrong say nothing about the difference. With 1,000 items, b = 40 and c = 95, the specialised model wins 55 more items than it loses, a gain of 5.5 points. McNemar's test (McNemar, 1947) checks whether such an imbalance could be chance:

χ² = (|b − c| − 1)² ÷ (b + c) = 54² ÷ 135 ≈ 21.6

which is far beyond what chance produces (p < 0.001). A bootstrap over items gives a confidence interval for the 5.5-point gain itself (Efron and Tibshirani, 1993). Looking at the 135 disagreements one by one is usually more instructive than either statistic: it shows what specialisation changed.

5. Guard against leakage

A specialised model is trained on the same kind of data it is evaluated on, which makes leakage the most likely way to fool ourselves. Duplicates and near-duplicates between training and test sets, a customer whose emails appear on both sides, a label that was derived from information only known after the fact — each inflates results that will not survive production. Leakage is widespread enough in applied machine learning to be called a reproducibility crisis (Kapoor and Narayanan, 2023).

Three habits help. Split by the unit that will be new in production — the customer, the domain, the month — not by row. Keep a test set that nobody looks at while iterating. And when the data has a time order, test on a period that comes after everything the model was trained or tuned on.

6. Measure the workflow

General benchmarks are not a substitute for evaluating the product. A model exists to change how work gets done, and errors do not cost the same in that work. In support triage, routing a newsletter to the wrong queue costs a few seconds; filing a genuine buyer's enquiry as advertising can lose a sale. A model with better macro-F1 can be the worse choice if its errors fall on the expensive side. Weigh errors by their cost, and agree the weights with the people who own the outcome before looking at the results.

For an assistant that drafts replies, the useful measurements are about the operator, not the text:

• the share of drafts accepted without changes, and with small edits;
• how much the operator changed (an edit distance is enough);
• sensitive cases correctly routed to a person instead of drafted;
• opportunities missed — the buyer who was never answered;
• net operator time per case, including the time spent reading and fixing drafts that were not used.

A draft that the operator rewrites from scratch is not a time saving, however fluent it reads. These measurements connect the model's behaviour back to the obligation the system was built to keep, which is the only reason to build it.

7. The minimum evaluation record

An evaluation that cannot be repeated cannot be trusted, including by the team that ran it six months later. We keep, for every published result:

Model cards (Mitchell et al., 2019) formalise much of this for publication. The per-request records are the part teams most often skip and most often need.

8. How we apply it

At Namecase we build compact, specialised models for narrow jobs — reading support requests, putting expiring domains into a reviewable order, suggesting price ranges for a human to check — and we serve them next to the people who make the final call. The method above is how we decide whether a specialised model earns its place: same items as the general model it is compared with, quality and coverage and calibration reported separately, leakage-safe splits, and a last word that belongs to the workflow. When we publish results for one of our models, they will come with their denominators and their record.

References

• Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3.
• Efron, B., and Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall.
• Geifman, Y., and El-Yaniv, R. (2017). Selective classification for deep neural networks. Advances in Neural Information Processing Systems 30.
• Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning.
• Järvelin, K., and Kekäläinen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4), 422–446.
• Kapoor, S., and Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9).
• McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2), 153–157.
• Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., and Gebru, T. (2019). Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency.
• Naeini, M. P., Cooper, G. F., and Hauskrecht, M. (2015). Obtaining well calibrated probabilities using Bayesian binning. Proceedings of the AAAI Conference on Artificial Intelligence.