How to score interviews consistently: eight controls, and where AI helps
On this page
To score interviews consistently, ask every candidate the same core questions, rate each answer against a written anchor rather than against other candidates, write down the evidence before you choose a number, score alone and straight after each interview, and review your scores across the shortlist for drift. The U.S. Office of Personnel Management's structured interview guide puts the first two plainly: all candidates are asked the same questions in the same order and evaluated using a common rating scale. AI can help with the parts people forget, but the rating itself should stay with the interviewer.
This page is written for hiring managers who interview a few times a month and want their scores to mean the same thing in week four of a search as in week one. Building the scale itself is covered in interview rating scale, and ready-made anchors are in behavioral anchors for interview scores.
Why do interview scores drift?
Inconsistent scores rarely come from carelessness. They come from small process gaps that let a different standard creep in for each candidate.
| Cause | What it looks like | Control below |
|---|---|---|
| Different questions for different candidates | One candidate got the hard technical question, another got a friendly chat | 1 |
| No written anchors | "3" means "fine" to one interviewer and "good" to another | 2 |
| Rating from impressions | The number came first and the reasons were found later | 3 |
| Rating days later | Monday's candidate is scored from memory on Thursday | 4 |
| Contrast with the previous candidate | An average candidate after a weak one scores high | 5 |
| Hearing other views first | Scores converge on whoever spoke first | 6 |
| Weights changed mid-search | A competency matters more once a favorite candidate is strong in it | 7 |
| Slow drift in strictness | Everyone after week three gets a point lower | 8 |
The eight controls
1. Ask the same core questions, in the same order
Write four to six core questions, one or two per competency, and ask all of them in every interview for the role. Follow-ups can differ, because they respond to what the candidate said, but the core cannot. The OPM guide also recommends using the same interviewers across all candidates when feasible, for the same reason.
2. Rate against written anchors
An anchor describes what an answer at each level sounds like, for each competency. Without anchors every interviewer rates against a private picture of the job. With them, the question becomes "which description does this answer match?", which two people can answer the same way.
3. Write the evidence before the number
For each competency, note what the candidate said or did, then choose the rating, then write one line on why it is not a level higher or lower. The full method is in evidence-based interview feedback. If you cannot write a single observation for a competency, mark it not assessed rather than guessing a middle score.
4. Score within the hour, alone
The OPM guide says to review your notes and rate immediately after the candidate leaves. Memory of the middle of an interview fades fast and is replaced by an overall impression, which is exactly what anchors are meant to avoid. Block ten minutes after each interview on your calendar.
5. Compare the answer with the anchor, never with the last candidate
"Much better than this morning's" is a contrast, not a rating. Before you settle on a number, reread the anchor for that level and the one above it. If you interview several candidates in one day, put the anchor sheet in front of you for each one, not only the first.
6. Keep scores private until everyone has submitted
The first number said aloud in a debrief pulls the others toward it. Collect all scorecards before anyone speaks, then discuss the evidence, starting with the competencies where scores differ most. The interview debrief template sets the speaking order. If new evidence changes your view, record both scores and the reason.
7. Fix the weights before the first interview
Decide how much each competency counts, and which ones are must-pass, at the start of the search. Changing weights once you have a favorite candidate turns the scorecard into a justification. How to set them is in the weighted interview scorecard guide.
8. Check for drift after every batch
Every five or six interviews, lay your scores side by side. Look for a row of identical middle scores, a competency where you never give the top or bottom level, or scores that slide in one direction over time. Bring patterns to a short calibration session, where interviewers rate the same sample answer and compare.
A drift check you can run on your own scores
An invented example: one hiring manager's ratings on "problem solving" for six candidates, in interview order, on a 1 to 4 scale.
| Candidate | Week | Score | Evidence written? |
|---|---|---|---|
| A | 1 | 3 | Yes, two quotes |
| B | 1 | 4 | Yes, two quotes |
| C | 2 | 3 | One line, no quote |
| D | 3 | 2 | "Not as strong as B" |
| E | 3 | 2 | One quote |
| F | 4 | 2 | No evidence |
Two things stand out. The evidence gets thinner as the search goes on, and the scores fall at the same time. D's justification is a comparison with B, not the anchor. Before the debrief, this manager should reread the anchors, revisit D, E and F against them using their notes or transcripts, and record any changed score with its reason. The numbers here are invented to show the pattern to look for in your own scores.
Where can AI make scoring more consistent?
The most useful AI features for consistency do not produce scores. They make sure each interviewer has the same information in front of them when they score.
| AI aid | Consistency problem it addresses | Caution |
|---|---|---|
| Question guide that tracks which core questions have been asked | Control 1: no candidate misses a core question | The guide should come from the agreed questions, not new ones the tool invents per candidate |
| Transcript search and evidence pulled per competency | Controls 3 and 4: rating from what was said, not from memory | Check every quote against the transcript |
| "Not covered" flags at the end of the call | Prevents middle scores given with no evidence | Only as good as the competency list you gave it |
| Drift reports across your scorecards | Control 8: shows patterns by interviewer and over time | Patterns are prompts for review, not proof of bias |
| Draft feedback organized by competency | Makes evidence-first scoring faster | The rating stays with you; see how to use AI to write interview feedback |
Should AI suggest the score?
Many interview tools now propose a rating. Software applies the same rule every time, which looks like consistency, but there are three problems with letting it set the number.
- Anchoring. Once you have seen a suggested 3, you look for reasons it is a 3. Your rating stops being independent, and the debrief loses the disagreement that catches errors.
- Consistent is not the same as right. A model that misreads one kind of answer will misread it the same way for every candidate. Consistent error is harder to spot than random error.
- Legal weight. In New York City, whether an AI score is an automated employment decision tool depends partly on how it is used, for example whether it is weighted more than any other criterion or used to overrule people. The rules are summarized, with links to the statutes, in is it legal to use AI note takers in interviews.
If your tool suggests scores, use this order: write your evidence and your own rating first, then look at the suggestion, and if they differ, go back to the transcript rather than splitting the difference. Record your reason when you keep your own number. Never let a tool's score break a tie between candidates.
A consistency checklist for each search
- Core questions written and shared with every interviewer before the first interview.
- Anchors written for every level of every competency.
- Weights and must-pass competencies agreed and written down.
- Same interviewers across candidates where possible.
- Ten minutes blocked after every interview for scoring.
- Evidence written before each rating; "not assessed" used when there is none.
- Scores submitted before the debrief, and not shared until all are in.
- Drift check every five or six interviews, and a calibration session if patterns appear.
- Any changed score recorded with the new evidence that justified it.
- AI suggestions, if any, looked at only after your own rating.
Questions people ask
How do I make sure I score every candidate the same way?
Ask every candidate the same core questions, rate each answer against a written anchor rather than against the other candidates, write down the evidence before you choose the number, and rate straight after each interview while it is fresh. Then look at your scores across the whole shortlist to check you have not drifted harsher or softer.
Why do two interviewers give the same candidate different scores?
Usually because they asked different questions, read the scale differently, or one of them heard the other's view first. Shared anchors, independent scoring before the debrief, and a calibration session where everyone rates the same sample answer close most of the gap.
Can AI score interviews more consistently than people?
Software applies the same rule every time, but consistency is not the same as accuracy, and a score you adopt without review can carry legal weight in places that regulate automated hiring tools. Use AI to make people consistent: tracking which questions were asked, pulling the evidence for each competency, and flagging drift across scorecards.
Should I change a score after the debrief?
Only if you learned new evidence, such as something another interviewer heard that you missed, and then record both scores and the reason. Changing a score because the room disagreed with you, without new evidence, makes the scores less consistent, not more.