Let's start with a small story.
You just joined the software team at a hospital lab. A pathologist stops by your desk. They have a request.
The lab runs a genetic test called NGS, short for next-generation sequencing. It reads the DNA of a tumor sample to find mutations, and those mutations help decide which treatment a patient gets. But the test has a catch. It only works reliably if enough of the sample is actually tumor. The usual rule is at least 20%.
That share is called tumor cellularity, or TC. Right now a pathologist estimates it by eye for every sample. It is slow, and two experts often give two different numbers. So the request is simple. Can you build a model that estimates TC for them?
You build it. For each sample, the model outputs a probability: how likely is it that this sample has at least 20% tumor. You test it on 1,000 samples it has never seen.
Accuracy is 91%. AUC is 0.97. (Don't worry, we'll get to what those mean.) Everyone is happy, and you make it to the demo just in time.
This works well for now. Then the pathologist asks one more question.
"Great numbers. But should we actually use it?"
You open your mouth to answer and realize that 91% says nothing about that.
Two things that are both called a threshold
Before anything else, let's fix a naming problem. This field uses the word "threshold" for two different things, and mixing them up is the fastest way to get lost.
The first is the clinical cutoff. It is the 20% rule. It is a fact about tissue, and it decides what counts as a positive case. A sample with 20% tumor or more is positive, meaning, fit for sequencing. Anything below is negative.
The second is the decision threshold, which I'll write as pt. It is a fact about you and your model. It is how sure the model must be before you act on its answer. If the model says 0.83 and your pt is 0.5, the sample goes to sequencing. If your pt is 0.9, it doesn't.
The cutoff stays fixed at 20%. The pt is a dial you can turn. Everything in this article is about that dial.
Four ways to be right or wrong
Every prediction lands in one of four boxes. If you have ever built a spam filter, you already know this table. Here it is for our lab.
True positive (TP): the model says the sample is fit, and it really is. The test runs and gives a reliable answer.
False positive (FP): the model says fit, but the sample has too little tumor. The test runs anyway and may miss a real mutation. That wastes money and can leave a falsely reassuring result.
False negative (FN): the model says unfit, but the sample was fine. It gets rejected, which usually means another biopsy (a needle taking a fresh tissue sample) and a delay in treatment.
True negative (TN): the model says unfit, and it really is. The lab skips a wasted test.
Papers turn these four boxes into a handful of metrics. Sensitivity is the share of truly fit samples the model caught. You may know it as recall. Specificity is the share of truly unfit samples the model correctly rejected. Accuracy is the share of all predictions that were right.
Where accuracy breaks
Accuracy has two problems here.
The first is the base rate. Doctors call it prevalence: the share of samples that are truly positive. In our data, 72% of samples are fit. A "model" that says yes to everything scores 72% accuracy without any intelligence at all. So 91% is less impressive than it sounds.
The second problem is bigger. Accuracy treats a false positive and a false negative as equally bad. In the lab they are not, and which one hurts more changes from day to day. If sequencing is expensive and the queue is long, wasted tests hurt. If a re-biopsy is delaying someone's treatment, missed samples hurt.
What about AUC? It has a different blind spot. Here is the friendly version. Pick one fit sample and one unfit sample at random. AUC is the chance that the model gives the fit one the higher score. A coin flip gets 0.5 and a perfect sorter gets 1.0. So AUC measures how well the model sorts, like a search engine putting the right results in the right order. It doesn't care whether the scores themselves mean anything, and it knows nothing about costs.
So we need something that speaks in decisions. That is what Decision Curve Analysis does.
What DCA asks
Statisticians Andrew Vickers and Elena Elkin introduced Decision Curve Analysis (DCA) in 2006. The question it asks is refreshingly practical. At each level of caution, is using the model better than the two laziest policies you could follow instead?
Those two policies are:
Treat all: send every sample to sequencing. You never miss a fit sample, but you pay for every unfit one.
Treat none: send nothing. No wasted tests, and no useful ones either.
"Treat" is medical language for "act on it." In our lab it means "send to sequencing." In your world it might mean "page the on-call engineer" or "block the transaction."
If the model can't beat both of these across the range of pt values you care about, it hasn't earned its place.
Net benefit, one piece at a time
To compare policies, DCA scores each one with a single number called net benefit. Here it is, with n as the total number of samples.
net_benefit = TP/n - (FP/n) × (pt / (1 - pt))Let's read it left to right. TP/n is the good: the share of all samples that were correctly sent for testing. FP/n is the bad: the share that were wrongly sent. But we can't just subtract the bad, because one wasted test is not the same size as one useful test. So it gets multiplied by an exchange rate, pt / (1 - pt).
Where does that come from? Think of a smoke detector in your kitchen. How many burnt-toast false alarms would you put up with to never miss a real fire? Say your answer is four. That means you would act whenever the chance of a real fire is 1 in 5, which is 20%. That is your pt. Four false alarms per real fire means one false alarm costs a quarter of a real catch, and 0.20 / (1 - 0.20) is exactly 0.25.
So pt is a compact way of saying how much you dislike false alarms. A low pt means they are cheap. A high pt means they are expensive.
The two baselines are easy. Treat none has a net benefit of exactly zero, since nothing is sent. Treat all flags everyone, so TP/n becomes the base rate and FP/n becomes one minus the base rate.
Let's build it
First we need data. We don't have real patients here, so let's fake a model. The code below invents the true tumor fraction for 1,000 samples and labels each one positive if it clears the 20% cutoff. Then it builds a "model" that sees a noisy version of the truth and turns it into a probability using a sigmoid, the same S-shaped function that logistic regression uses.
import numpy as np
from sklearn.metrics import roc_auc_score
rng = np.random.default_rng(42)
n = 1000
# The truth: what fraction of each sample is really tumor
true_tc = rng.beta(1.6, 3.2, n)
# Label: 1 if the sample clears the 20% cutoff, otherwise 0
y_true = (true_tc >= 0.20).astype(int)
# The "model": a noisy estimate of TC, turned into P(TC >= 20%)
estimated_tc = np.clip(true_tc + rng.normal(0, 0.07, n), 0, 1)
y_prob = 1 / (1 + np.exp(-(estimated_tc - 0.20) / 0.04))
accuracy = np.mean((y_prob >= 0.5) == y_true)
print(f"Accuracy: {accuracy:.3f}") # 0.907
print(f"AUC: {roc_auc_score(y_true, y_prob):.3f}") # 0.965
print(f"Base rate: {y_true.mean():.3f}") # 0.716
Those are the numbers from our story. Now for the part that matters. We need two small functions, one for the model and one for the treat-all baseline.
def net_benefit(y_true, y_prob, pt):
n = len(y_true)
flagged = y_prob >= pt
true_positives = np.sum(flagged & (y_true == 1))
false_positives = np.sum(flagged & (y_true == 0))
exchange_rate = pt / (1 - pt)
return true_positives / n - (false_positives / n) * exchange_rate
def net_benefit_treat_all(y_true, pt):
base_rate = y_true.mean()
exchange_rate = pt / (1 - pt)
return base_rate - (1 - base_rate) * exchange_rate
Now let's sweep the pt dial from 0.05 to 0.90 and plot all three policies.
import matplotlib.pyplot as plt
thresholds = np.linspace(0.05, 0.90, 86)
model_curve = [net_benefit(y_true, y_prob, pt) for pt in thresholds]
all_curve = [net_benefit_treat_all(y_true, pt) for pt in thresholds]
plt.plot(thresholds, model_curve, label="Model", color="steelblue", linewidth=2)
plt.plot(thresholds, all_curve, label="Treat all", color="gray", linestyle="--")
plt.axhline(0, label="Treat none", color="black", linestyle=":")
plt.xlabel("Decision threshold (pt)")
plt.ylabel("Net benefit")
plt.ylim(-0.1, 0.8)
plt.legend()
plt.show()
Reading the curve
Three lines. The dotted black one at zero is treat none. The dashed gray one is treat all. It starts high and then falls, because the more you dislike false alarms, the worse it is to flag everything. The blue one is our model.
At pt = 0.20, the model scores 0.672 and treat all scores 0.645. That is a modest win. At 0.50 it is 0.623 against 0.432. At 0.80, treat all has fallen to -0.420 (off the bottom of the chart), which means it does more harm than good, while the model still holds 0.566.
Here is a handy way to read those gaps. The difference at 0.50 is about 0.19. That is like finding 19 extra truly fit samples per 100, without a single extra false alarm.
Notice the far left too. When pt is 0.05, false alarms are nearly free, so the model barely beats treat all. Of course it does. If wasted tests cost almost nothing, you may as well test everything.
So here is the answer for the pathologist. Across the whole range, the model beats both simple policies, and the gap grows as wasted tests get more expensive. That is a far better answer than "91%".
Same AUC, different answer
AUC and DCA are not rivals. They answer different questions, and the difference is easiest to see by building a model that keeps one answer identical while changing the other. Let's construct a second model that AUC insists is exactly as good as the first, and watch DCA disagree.
Here's the trick. Take every probability our model produced and squeeze it into a narrow band near the top of the scale, 0.90 to 0.99, with this transform:
squeezed = 0.90 + 0.09 * y_prob
Walk through what that line does. y_prob ranges from 0 to 1. Multiplying by 0.09 compresses that down to a range of 0 to 0.09. Adding 0.90 shifts the whole thing up, so the output now ranges from 0.90 to 0.99. A sample that used to score 0.10 now scores 0.909. A sample that used to score 0.95 now scores 0.9855. Every sample got relabeled as "almost certainly fit," regardless of how confident the original model actually was.
The one thing this transform does not touch is order. If sample A scored higher than sample B before, 0.90 + 0.09 * A > 0.90 + 0.09 * B still holds after, because we did the same multiply-and-add to every sample. This is called a monotonic transform: it can slide and stretch the values, but it can never make a lower-ranked sample overtake a higher-ranked one. Think of it like sorting a spreadsheet column, then giving every row a raise proportional to nothing but its row number. The order of the rows on screen never changes.
That matters because AUC, remember, only asks one thing: pick a random fit sample and a random unfit sample, what's the chance the model scores the fit one higher? That's purely a question about order. So:
print(f"AUC: {roc_auc_score(y_true, squeezed):.3f}") # 0.965, unchanged
squeezed_curve = [net_benefit(y_true, squeezed, pt) for pt in thresholds]
Same 0.965 as before, down to the third decimal. AUC genuinely cannot tell these two models apart, because as far as ranking goes, they aren't different models. That's not a flaw in AUC, it's exactly what AUC is built to measure: can this model sort fit from unfit. Yes, it still can, as well as it ever could.
Now feed the same squeezed numbers into DCA, which doesn't care about order, it cares about the actual value next to pt.
Look at the squeezed model's curve. It sits exactly on top of treat all for the entire range we swept, pt from 0.05 to 0.90, and never once separates from it. Here's why, worked through the same way we built net_benefit earlier. Every squeezed value is at least 0.90. So for any pt in that 0.05-to-0.90 range, the condition flagged = y_prob >= pt is true for every single sample, fit or not, because 0.90-something is always greater than a pt of, say, 0.30 or 0.60. Flagging every sample is, by definition, the treat-all policy. The squeezed model isn't making decisions differently from treat-all anywhere on this chart, it's mechanically incapable of making a different decision, because across our entire plotted range it never once outputs a number low enough to say "skip this one." (Push pt past 0.90, outside what we plotted, and the squeezed numbers do start spreading back out, but they land somewhere worse and noisier, not back at the original model's performance. That range is past what a pathologist here would plausibly use anyway, so we left it off the chart.)
So we've built a model that ranks exactly as well as the original, and yet, at any threshold a pathologist would actually use (0.50, say, or wherever your pt lands), it behaves identically to sending everything to sequencing. That's the gap AUC can't see and DCA can.
The name for this gap is calibration. A well-calibrated model is right about 90% of the time when it says 90%. A well-calibrated model tells you the truth about how confident it is. Ours doesn't anymore. Think of a weather app that says "95% chance of rain" every single day. It might still correctly rank tomorrow as more likely to rain than today. That's ranking, and it's intact. But you can no longer read "95%" and trust that it means what it says, and if you were deciding whether to carry an umbrella based on that number crossing some threshold, you'd get it wrong constantly. That's the decision-making layer, and it's broken.
AUC tells you whether a model can, in principle, tell fit from unfit apart. DCA tells you whether the numbers it hands you are safe to act on at the threshold you actually use. Both questions are worth asking. You just need DCA (or a calibration check) to answer the second one, because AUC was never built to. Calibration is a big topic on its own, and it's why this newsletter is called Recalibrated, so we'll be back to it often.
Now the shortcut
Once you understand the math, you don't need to maintain it yourself. Oncothresh is an open-source Python library I wrote for evaluating clinical models at decision thresholds, and DCA is built in.
from oncothresh import ThresholdEvaluator
evaluator = ThresholdEvaluator(y_true=true_tc, y_pred=y_prob)
result = evaluator.decision_curve(clinical_threshold=0.20, thresholds=thresholds)
print(result)
# DecisionCurveResult(clinical_threshold=0.20, prevalence=0.716, pt=[0.05-0.90], n_points=86)
print(np.allclose(result.net_benefit_model, model_curve)) # True
Two details are worth knowing. First, y_true is the raw tumor fraction, not 0 or 1. You give the library the clinical cutoff and it does the labeling for you. Second, y_pred must be a probability between 0 and 1. Pass it a raw score and it raises an error rather than guessing.
The numbers match our hand-rolled version exactly, and the library is checked against dcurves, the reference R package for DCA. Install it with pip install oncothresh. The source is at github.com/omkaradhali/oncothresh.
When to use it
A model's output triggers an action, such as sending a test, flagging for review or raising an alert, and the mistakes cost different amounts.
You need to convince someone outside ML (a pathologist, a PM, a compliance officer) that the model is worth using. A curve in their language beats an AUC.
You are comparing two models and want to know which is better for the decision, not just which one sorts better.
When not to use it
Nothing acts on the output, or only the ranking matters, like search results.
You have very few samples. A curve from 50 cases is mostly noise.
The costs are obviously equal and one sensible threshold exists. Plain sensitivity and specificity will do. Keep it simple.
Next time, let's look at why a model can rank perfectly and still lie about its probabilities, and what to do about it.
Hope this was useful.
Recalibrated is a newsletter about building and validating clinical AI, written by Omkar Adhali (ORCID). If someone sent you this, you can subscribe at recalibrated.ai.