Hold-outs, proxies, and real-life evidence¶

A frozen hold-out is powerful when the target can be sampled cleanly. It is not the only rigorous evidence model for a lifetime curriculum.

Contamination discipline¶

For any unseen benchmark: freeze the graph, preregister scoring, keep scored material hidden, use independent curators/reviewers, preserve disagreements, and classify failures as missing family, missing node, hidden prerequisite, missing fact, mastery failure, ambiguous rubric, or scorer disagreement.

Criterion proxies¶

The package preserves 417 imported All Souls Philosophy/General prompt mappings. Their independent freeze and scoring provenance has not been established. Public past-paper mappings are not automatically unseen hold-outs or evidence of learner performance.

Expansion to Economics, English Literature, History, Law and Politics is proposed work, not a completed evaluation. Before testing, preregister a fixed graph version, scoring rules, independent reviewers and contamination controls. See current evidence limits.

Prospective performance proxies¶

Some branches are better tested by performance than by exam archives:

• Mind/Learning - unfamiliar-domain acquisition, delayed retention, transfer, teaching and AI-offloading comparisons.
• Design - blind briefs, blind comparison, expert/user review and measurable usability/reality checks.
• Technical Systems - hidden derivations, builds, experiments, failure injections and operational perturbations.
• Life competence - live/simulated tasks plus the Surprise Resilience Battery.

Longitudinal evidence¶

The strongest outcome claims require prospective studies of real learners over years. Observational comparisons remain vulnerable to learner selection and life circumstance. Publish outcome definitions, comparison strategy, attrition and null or negative findings rather than compressing “a good life” into one score.