Examiner calibration¶
Use with the general examination and the technical portfolio. This is an implementation protocol, not an empirically calibrated scale.
Before assessment¶
Two reviewers independently score an agreed sample without seeing author identity or AI condition when feasible. Record judgments before discussing them. Include a fluent but false submission, a rough but sound submission, and an honest incomplete submission. Establish access arrangements and permitted tools in advance.
Retain separate judgments for knowledge, internal model, transfer, practical execution, judgment, communication, AI stewardship, reality contact, responsibility and renewal. Use the canonical dimension sets appropriate to the assessment; do not merge their different constructs into one percentage.
Performance anchors¶
| Dimension | Weak | Adequate | Strong |
|---|---|---|---|
| Reconstruction | Repeats conclusions; cannot recover assumptions | Rebuilds the essential argument or procedure | Rebuilds it by a second route and identifies dependency failures |
| Evidence | Treats an attractive source or correlation as proof | Separates claim, evidence and inference | Locates a discriminating observation and explains residual uncertainty |
| Transfer | Reuses vocabulary despite a changed mechanism | Adapts the model to an unfamiliar case | Detects the boundary of transfer and chooses a better model |
| Execution | Describes what should happen | Performs the authorized task and checks the result | Recovers from a realistic perturbation and leaves maintainable evidence |
| Responsibility | Ignores affected people, permissions or downside | Identifies consent, scope, hazards and referral | Preserves recourse, reversibility and independent challenge |
| Taste | States preference or copies prestigious style | Names criteria and compares actual alternatives | Explains a worthwhile tradeoff, revises and defends the result in context |
An unsupported novel claim is not originality. Reward a useful new question, defensible synthesis or informative counterexample. Do not reward uncertainty as a verbal ritual: it must change the decision, bounds or next investigation.
Error handling¶
Classify an error before grading it: factual, conceptual, inferential, procedural, ethical/scope, or communicative. A correct answer reached through invalid reasoning cannot establish the reasoning construct. A transcription error differs from a wrong causal model, but a safety-critical transcription error can still require a stop. Record consequences separately from conceptual understanding.
AI and oral defense¶
Require provenance for delegated work and a declared independent contribution. Select a small unannounced perturbation: remove a source, change a boundary condition, introduce a misleading output, or disable the tool. Ask the learner to reconstruct the affected reasoning. Do not use surprise to humiliate or defeat reasonable disability accommodations.
AI polish earns no independent-capability credit. A model may propose critique; the examiner owns scoring, factual checks, privacy and appeal. Do not submit confidential learner work to a third party without authorization.
Disagreement and retest¶
Preserve both original ratings and their rationales. Identify whether disagreement concerns facts, constructs, standards or values. Resolve factual disputes with sources; document legitimate interpretive disagreement. Use a third reviewer for material unresolved differences, and give the learner a correction/appeal route.
Retest a new instance after a declared delay. Record the task, conditions, reviewer and rubric version so a later study can measure reliability without rewriting the original evidence. A successful retest is evidence about that capability under those conditions, not a guarantee of character or flourishing.