The Number Moved, the Model Didn't

Sleep staging is a well-trodden benchmark. You take a single EEG channel, cut it into 30-second epochs, and predict which of five stages a human scorer assigned: wake, N1, N2, N3, REM. The published literature on Sleep-EDF reports 82–85% accuracy. It is an ideal task for the question I actually care about, which is not "can I hit the number" but "what would I have to do to the evaluation to make the number mean nothing."

The answer, it turns out, is: very little, and none of it requires dishonesty. Every lever below is a defensible methodological choice. Several are standard practice. Each one moves the headline number, and only one of them moves the model.

This writeup reports a classical-feature baseline that reaches 79.1% ± 5.7% accuracy (Cohen's kappa 0.707, macro-F1 0.687) under subject-wise leave-one-out cross-validation on the 20-subject Sleep-EDF Cassette subset. It also reports the same pipeline reaching 90.4% with one preprocessing change that makes the model no better at its job — macro-F1 moves by 0.0002. Both numbers are real. Only one of them is honest, and the reason is the point of the article.

Every result below was produced twice, on two machines under different operating systems, and agreed to four decimal places in both the cropped and uncropped conditions.

The setup

Sleep-EDF Expanded, Sleep Cassette subset, subjects 00–19: 39 recordings, 20 subjects, two nights each except subject 13, who has one. Single channel, Fpz-Cz, sampled at 100 Hz. Labels remapped from R&K to AASM five-class, with S3 and S4 merged into N3 and movement/unscored epochs dropped. After preprocessing: 42,307 epochs.

Class balance, as expected, is poor:

Features are classical and hand-built: log and relative band power in five bands via Welch PSD, five band ratios, spectral edge frequencies and spectral entropy, Hjorth mobility and complexity, and a handful of amplitude statistics. Twenty-six features per epoch. Classifiers are a random forest and a histogram gradient-boosting ensemble. No neural network appears anywhere in this article.

Decision 1: the split, which is the one that matters

Sleep Cassette filenames encode subject and night: SC4<ss><n>E0. SC4001E0 and SC4002E0 are not two subjects. They are one person, sleeping twice.

Split on filename, or on record index, or with a plain shuffled k-fold, and one person's two nights land on opposite sides of the boundary. The model is then evaluated on a night from someone whose EEG it has already memorized — their electrode impedance, their amplitude scale, their spindle morphology. The number goes up. Nothing was learned.

The loader parses subject ID out of the filename, groups before splitting, and asserts twice: that the train and test subject sets have empty intersection, and that both nights of every subject fall on the same side. Those assertions run on every fold.

This is the failure that invalidates everything downstream, and it is invisible in the results. A leaked evaluation and a clean one produce output of identical shape.

Decision 2: the crop, and what it is worth

Sleep Cassette recordings run about 20 hours. The subject is wired up in the afternoon and unhooked the next morning, so the file contains many hours of lights-on wake at both ends. Nearly every published result crops this — typically to 30 minutes of wake on either side of the sleep period — and so does this one.

The justification is sound: lights-on wake is not the task. Sleep staging is.

But it is worth being precise about the size of the effect. Uncropped, the same 39 recordings yield 106,414 epochs, of which 68.0% are wake. A classifier that ignores its input and always predicts "awake" scores 68.0%.

Running the identical pipeline on uncropped data, gradient boosting:

Accuracy gains 11.3 points. Macro-F1 gains two ten-thousandths. For the random forest, run identically, accuracy gains 11.2 points and macro-F1 drops by 0.0022.

The model did not improve. The denominator changed. Both numbers come from the same code with one flag flipped, and if I reported only the second one, every sentence in this article would still be true.

Three details are worth dwelling on, because they are what a careful reader would check and they all point the wrong way.

Cohen's kappa is not a defense. Kappa exists specifically to correct for chance agreement under class imbalance. It rose 0.095. More robust than accuracy here; not immune.

The variance improves too. Fold-to-fold standard deviation halves, from 0.057 to 0.025. The inflated number is also the more stable-looking one, so a reviewer checking for suspicious variance would find the uncropped result more trustworthy, not less.

All of the gain is in one class. W's F1 climbs 0.153 while every other class is flat or worse. The classifier got better at identifying eight extra hours of a subject lying awake in a lab with the lights on, and that improvement is now 68% of the test set.

Macro-F1 is what catches this, and it catches it because it refuses to let a prevalent easy class vote twice.

Decision 3: which subset, and who is in it

The 82–85% figures in the literature are generally reported on this 20-subject subset, not on all 78 subjects in the full cassette set. Benchmarking against papers that used a different denominator is its own kind of lie, so the subset is chosen to match the claim.

Having chosen it, one property is worth stating plainly, because I did not know it until I checked the demographics file: the 20-subject subset is ages 25 to 34, median 28. The remaining 58 subjects are ages 50 to 101, median 67. The subset is not a random sample of the cohort. It is the young half, entirely.

So the benchmark everyone cites is a healthy-young-adult benchmark. Sleep architecture changes substantially with age — slow-wave sleep declines, arousals increase — and a model validated here has not been shown to generalize to the population that actually gets referred for sleep studies. That is not a flaw in the dataset. It is a flaw in how the number is usually described.

Decision 4: gamma is not a band you have

The data is sampled at 100 Hz. Nyquist is 50. Above roughly 35 Hz, a forehead electrode is reading jaw and scalp muscle, not cortex. A "gamma power" feature here is EMG contamination wearing a neuroscience label.

It is dropped. In its place: sigma, 12–16 Hz — the spindle band, which is the defining graphoelement of N2, which is 42% of the data. This is not a tuning choice. It is reading the sampling rate before choosing the features.

Decision 5: normalization, disclosed

Features are robust-z-scored within each recording, using only that recording's own median and interquartile range. This cancels per-subject amplitude and impedance offsets, and it is worth several points.

It is not cross-subject leakage — no test subject is scaled by statistics from the training set. It is, however, transductive within a recording: the test night's own epochs determine its scale. For offline staging, where the whole night is in hand, that is legitimate. For a real-time system it would not be. The distinction matters and is easy to leave unmentioned, so: it is mentioned. The --no-norm flag reproduces the pipeline without it.

Decision 6: metrics fixed in advance, including the failure

Before seeing any result, three metrics were fixed — accuracy, Cohen's kappa, and macro-F1 — along with an explicit prediction: that N1 would fail, with F1 in the 40–50 range, because N1 is rare, transitional, and has poor inter-rater agreement even among human scorers.

The direction was right. The number was not. N1 came in at F1 0.269, recall 21%. The prediction was off by roughly half.

That miss is reported here rather than quietly dropped, because a pre-registration that only appears when it succeeds is not a pre-registration.

Results

Subject-wise leave-one-subject-out, 20 folds, both models:

For reference, the same two models on uncropped data: 0.9024 ± 0.0265 and 0.9043 ± 0.0251, macro-F1 0.6839 and 0.6868.

Per-class, gradient boosting:

Confusion matrix, row-normalized (rows true, columns predicted):

The plateau, and why it is the finding

The two classifiers differ by 0.0005 accuracy — five ten-thousandths, against a fold-to-fold standard deviation of 0.06. Their per-fold accuracies correlate at 0.972. A bagged ensemble and a boosted ensemble, tuned independently, land on top of each other and fail on the same nights.

Given these features, the classifier is not the bottleneck.

That claim needs its boundary stated: both models consume the same 26 hand-built features, so what has been shown is that model choice within a classical-feature pipeline does not matter. A convolutional network on the raw epoch is a different hypothesis class and could in principle recover structure the band powers discard. The stronger claim — that architecture does not matter at all — is not supported by this experiment and is not made.

What is supported is the shape of the ceiling. Single-epoch classification on Sleep-EDF tops out in the high seventies, and the reason is structural rather than architectural: a human scorer does not classify epochs in isolation. They read the night as a sequence, and the AASM rules themselves are transition rules. An N1 epoch is largely defined by what preceded it. Strip the context and you have removed information the labels were generated with.

The confusion matrix shows exactly this. N1's errors are not concentrated — 34% to REM, 26% to W, 19% to N2. It is not being confused with one neighbor. It is being confused with all of them, which is what you would expect from a class whose definition is positional.

What is not explained

Per-subject accuracy runs from 0.684 to 0.866 — an 18-point spread. Model choice is worth 0.0005. Subject identity is worth 18 points.

That alone justifies subject-wise k-fold over a single held-out split: with this much between-subject variance, a single split can be made to report anything in an 18-point range, and every value in it is defensible.

What drives it is unclear. A subject's N1 share does not predict their accuracy (r = −0.10). Neither does age (r = 0.21, p = 0.37), though the subset's narrow age band limits that test. Electrode quality, arousal frequency, and scorer variation are all plausible and none were tested here. It is left open.

Scope

Out of scope for this piece, and deliberately: multichannel fusion, temporal context models, real-time inference, my own recordings, and beating the state of the art.

Temporal context is the known lift and is the obvious next artifact. It is a different data pipeline — sequence windowing, sequence-boundary leakage, different batching — and folding it in here would have turned a bounded result into an unbounded one.

Limits. One dataset, one channel, single-epoch classification, self-graded. The reproduction across two machines guards against implementation error, not against a shared mistake in method. The age-stratification of the subset is stated but its effect on generalization is not measured here — that is the next artifact, not this one. If your pipeline survives the six decisions above, I'd like to hear about it. If it doesn't, I'd like to hear about that more.

Assistance note. Portions of the analysis pipeline and this manuscript were prepared with AI assistance; every number was produced by the author's own runs, reproduced on a second machine, and verified against the frozen results manifest.