Cut scores on licensing exams

Why must the pass mark that separates a competent practitioner from an incompetent one be argued for rather than measured?

A licensing board publishes a pass mark, and a candidate one point below it does not practise. The number looks like a finding — as though someone determined where competence begins, the way a laboratory determines a melting point.

But boards do not describe it that way. They convene panels, run structured procedures, and publish a defence of the number rather than a derivation. Suppose that is honest rather than evasive. What would have to be true of competence for the line to be genuinely unfindable — something a board must choose, with no fact of the matter it could have got wrong?

Ask first what a discoverable cut score would look like. There would have to be two populations on the score scale — the competent and the incompetent — with a gap between them. If it existed, the trough would be the answer, every panel would find the same place, and the argument would be over.

So look at the distribution. Candidate scores on a licensing exam form a single smooth hump: no valley along it, no shoulder, no place where the density drops away and picks up again. That is what you should expect, since the underlying thing is an accumulation — of knowledge, drilled procedure, supervised practice — and accumulations vary continuously. Competence is not a natural kind with a boundary the way a chemical element is, so the data cannot say where to cut: nothing in its shape distinguishes one point from its neighbour.

Could an outside criterion rescue us? It is the right instinct: forget the distribution and look at what candidates later do. Regress malpractice claims, disciplinary findings or supervisor ratings on exam score and look for a step — flat risk above some score, a cliff below. Where such relationships have been examined they come out smooth and monotonic, with no break to point at.

A structural obstacle also deserves stating, because it is why more data can never settle this. Only candidates above the line practise, so the outcomes of everyone below are unobservable by construction — the evidence is truncated at exactly the place we want to look.

If the line is not out there, what is a board actually doing? It is trading two errors whose costs cannot be put in the same units. A false pass places someone unsafe in front of the public; a false fail takes a livelihood from a competent person after years of training and debt. Push the line up and you convert the second error into the first; push it down and the reverse. No position avoids both, and no arithmetic makes the two costs commensurable — which is what makes this a policy judgement wearing a number's clothes.

What standard-setting methods do, then, is not locate the truth but make the judgement auditable. In the Angoff procedure a panel considers each item and estimates the probability that a "minimally competent candidate" would answer it correctly; the sum of those probabilities is the cut score. In the bookmark method items are ordered by difficulty and each panellist marks the point where such a candidate's chance of success falls below a stated criterion — commonly around two-thirds, though boards choose differently.

Both route the decision through a hypothetical person nobody has met, and different panels picture that person differently. The documented consequence is what the argument predicts: different methods on the same exam yield different cut scores, and so do different panels using the same method. That variation is not a scandal to be engineered away; it is the signature of a judgement.

One more layer sits on top. Even granted a chosen line, a candidate one mark below and one mark above are not reliably different — the conditional standard error of measurement near the cut is usually larger than the gap. So a board faces a second decision about that noise: some shift the line in the candidate's favour by a standard error, some publish a decision-consistency figure, some do neither.

It is worth marking how this differs from a threshold set on measurement noise, since the two look identical from a distance. When a laboratory declares a detection limit it also draws a line on a continuum and chooses its error rates. But there it can measure a blank — material known to contain none of the substance — and the underlying state really is binary. That makes the false-positive rate calculable and the threshold defensible against an outside fact.

A licensing board has no blank. There is no group known to be truly incompetent against which to characterise the scale, and no true dichotomy behind the score to be right or wrong about. The detection limit is a decision problem with a knowable answer rate; the cut score is one in which even the two categories are outputs of the procedure rather than inputs to it.

Two nearby ideas should be kept separate. Equating — putting this year's paper on last year's scale — lets a fixed cut score keep its meaning across forms, but presupposes the line rather than justifying it. Validity asks whether the score supports the inference at all, and a well-argued line on a badly-aimed test is still a bad decision.

Finally, the claim here is falsifiable, not merely sceptical. It predicts a unimodal score distribution with no discontinuity, and a smooth relationship between score and any external competence criterion. Demonstrate a genuine step in such a criterion at some score, replicating across cohorts and jurisdictions, and standard setting becomes measurement overnight. One near-miss: candidates who just passed do differ later from those who just failed, but that follows from holding the licence — the practice and supervision it permits — not from the sliver of ability between them.

Every term the collection defines is gathered in the glossary.