The metadata predicted the patient, until the site changed

A model that never hears the heartbeat sorts normal from abnormal about as well as the literature reports. Move it to an unseen cohort and it does worse than guessing.

1What it was given, and what it decided to ask

The system received 2,000 single-channel heart-sound recordings at roughly 2 kHz, each labelled normal or abnormal and drawn from one of six cohorts collected at different sites and under different conditions. Nothing else. It chose its own question: how much of the reported normal-versus-abnormal separability in this corpus is carried by non-physiological recording bookkeeping rather than by murmur-band energy, and does whichever signal drives it survive when you hold out an entire cohort?

2The metadata alone works, then collapses

Under ordinary pooled 5-fold cross-validation the metadata-only model reaches AUC 0.596. Under strict leave-one-cohort-out validation it falls to 0.347, a drop of 0.248 (bootstrap 95% CI 0.176 to 0.297, p < 0.0001, surviving Bonferroni correction across 11 tests at α = 0.00455). The murmur-band model does not behave this way: it goes from 0.701 to 0.656, a drop of 0.045 that is not distinguishable from zero.

3Why: the metadata is a name tag for the hospital

The same five metadata features identify which of the six cohorts a recording came from with 87.2% accuracy, against a 16.7% chance baseline. And simply scoring every recording by the abnormal-prevalence of its predicted cohort reproduces AUC 0.642, which already exceeds the metadata model's own pooled 0.596. An oracle that knows each cohort's true base rate reaches 0.766. So the metadata model was never reading the patient. It was reading the recording protocol, inferring the site, and quoting that site's base rate.

4The table that explains the whole thing

These are the six cohorts' own averages, computed by the run. Read across a row and each site has a recording signature: cohort b keeps clips down to 8 seconds and records five times louder than cohort d; cohort c runs 51 seconds; cohort f's microphone is five times brighter than cohort b's, with twelve times the zero-crossing rate. Five numbers are more than enough to tell them apart.

Now read the second column. Abnormal prevalence runs from 9.8% in cohort e to 88.9% in cohort c, a ninefold spread. Identify the site and you have already made most of a diagnosis, without touching the heart.

The last column is where the sub-chance number comes from. Cohort e alone is 1,333 of the 2,000 recordings and is the least abnormal cohort at 9.8%. A model trained on the other five has learned that this recording profile belongs to sicker populations, so on cohort e it ranks the wrong way round and scores 0.272. Because cohort e dominates the weighted average, the whole leave-one-cohort-out figure lands at 0.347, below chance. The inversion is not a bug in the evaluation; it is the confound being read backwards.

5The split runs through the features themselves

An analysis of variance over each feature shows that the protocol statistics carry a far larger fraction of between-cohort variance than the murmur-band feature does. All seven tests land at p < 10⁻³⁰. Held out cohort by cohort the metadata model is wildly unstable, from 0.272 on the largest cohort to 0.844 on a cohort of only 18 recordings; the gap between the two models is nonetheless stable across 10 independent cross-validation seeds and after dropping the two smallest cohorts.

6Every number on this page