Notebook page: structured technical interviews — rubric, question design, calibration, think-aloud

CAREER & LEARNING CRAFT

Structured Technical Interviews: Rubrics and Panel Calibration Over Gut Feel

An unstructured panel is a low-reliability instrument. Structure is how you make it measure the same thing in every candidate.

📌Core Concept

When every interviewer asks their own questions and scores by impression, they are measuring different constructs. The panel's disagreement is then mostly instrument noise, not signal about the candidate. Structure attacks that noise on three axes: a fixed question set with a written rubric, advance calibration on a shared example, and independent scoring before any discussion.

Mental model: a panel is a measuring instrument. The rubric is its scale, calibration is checking the scale on a known sample, and independent scores are the reason the instrument is worth running at all.

🔑Key Components

SPINE 1 · STRUCTURE

Fixed questions + written rubric

Every candidate meets the same stimulus, and each score level is defined by observable behaviour before anyone is interviewed — so the scale means the same thing to every rater.

SPINE 2 · QUESTION DESIGN

Trajectory over trivia

A trivia question is perfectly structured and still low-signal: its output is one bit (knew it / did not). An open problem produces a trajectory — clarifying questions, assumptions surfaced, trade-offs named, recovery after a wrong turn.

SPINE 3 · CALIBRATION

Argue on a sample, not a candidate

The panel scores a shared example — ideally a real past transcript — and argues to consensus on what strong / average / weak looks like. Drift is discovered cheaply, before it lands on a real person.

SPINE 4 · ENGAGEMENT

Think aloud, or you score the noise

If the reasoning stays in the candidate's head, you score the outcome — the noisiest proxy available. “I'm here to help you succeed, please think aloud” is the cheapest fix, and an adversarial interviewer destroys their own measurement by scaring the candidate.

⚙️How It Works

Build the spine. Same problem, same rubric, same opening for every candidate — with licensed freedom in follow-ups. A rigid script run by a bored panel is its own failure mode.
Calibrate in advance. The whole panel scores one shared past transcript and argues to consensus. If they cannot agree on the sample, they will not agree on the candidate.
Interview with the trajectory in view. Ask open questions, take hints off the table, watch for assumptions and trade-offs rather than right answers.
Score independently, in writing, first. Then — only then — discuss.

Why the order matters: discussing first collapses the panel onto the loudest or most senior voice, and destroys the independence that made multiple interviews worth running in the first place.

The distinction that keeps the honest: a calibrated panel can converge on a consistently wrong bar. Consistency is reliability, not validity — it makes the instrument repeatable and says nothing about whether the score predicts job performance. The missing feedback loop is checking hire outcomes against interview scores over a cohort.

🌎Real-World Applications

High-stakes panels

Where a single mis-hire is expensive, a fixed spine plus pre-committed rubric is what makes a rejection defensible — to the candidate, the panel, and the record.

Bias as a process property

Telling interviewers to “de-bias” moves little. Structure — fixed questions, written rubric, independent scores, evidence quoted before ratings — reduces variance mechanically.

Calibration as rehearsal

Scoring a past transcript together is a cheap, no-stakes rehearsal of the disagreement you would otherwise have live — on a candidate.

Validity check over time

Track later performance of hires against their interview scores. If the correlation is flat, the panel is reliable but invalid — exactly the failure consistency alone cannot detect.

🧪Checkpoint — answer before you move on

Why must the rubric's score levels be written down before any candidate is interviewed?What goes wrong if each rater defines “strong” in their own head, after the fact?
Why does the panel commit to written scores before discussing — and what does discussing first do to the measurement?Name the force that pulls the panel together once conversation starts.
A calibrated panel scores three candidates identically — and all three hires underperform. What property is missing, and which measurement would have caught it?Reliability is not validity. What closes the loop?