Can you fairly compare candidates who had different interviewers?
On this page
- When can you compare candidates who had different interviewers?
- How do you spot a lenient or severe interviewer?
- Should you adjust the scores, re-score, or re-interview?
- A worked example
- What about recruiters who run many screens a day?
- A neutral comparison checklist you can copy
- How do you avoid the problem next time?
- Questions people ask
Yes, if every interviewer asked the same questions and scored against the same written anchors, with the candidate's own words under each score. Then the main risk is that some interviewers score more leniently or more severely than others, which you can check: compare each interviewer's scoring pattern, and for close finalists have one person re-score the key answers from the notes or transcripts. Going forward, keep one interviewer, or one pair, per stage for every candidate, which is what the U.S. Office of Personnel Management's structured interview guide recommends.
Inter-rater reliability and calibration sessions measure and raise agreement across a team in general. This page is for a specific moment: your finalists were seen by different people, and you have to compare them now.
When can you compare candidates who had different interviewers?
Check four conditions before you look at any numbers:
| Condition | If yes | If no |
|---|---|---|
| Same questions for the competency, in the same stage | Scores describe the same thing | You are comparing different interviews; the scores cannot be lined up |
| Same anchored scale, with a description for each point | A 3 should mean the same to everyone | Each interviewer used a private scale; re-score from the notes against one set of anchors |
| Scorecards written independently, before any discussion | Each score is that interviewer's own reading | Scores may echo whoever spoke first; see independent feedback before the debrief |
| Evidence written under each score | You can re-score without the interviewer in the room | You are comparing opinions; a fresh, shared step is the only fix |
If the first two conditions fail, stop: no adjustment makes the comparison fair. Give the finalists still in the running a common step with the same interviewer and the same questions, as described below.
How do you spot a lenient or severe interviewer?
OPM's structured interview guide names both errors: leniency, a tendency to give high ratings to all candidates whatever their performance, and strictness, the reverse. Both show up in the numbers if you lay out every candidate each interviewer scored for the role, not only the finalists.
| Interviewer | Candidates scored | Average (1–4) | Share of 4s | Share of 1s and 2s |
|---|---|---|---|---|
| Interviewer A | 14 | 3.2 | 36% | 7% |
| Interviewer B | 12 | 2.4 | 8% | 50% |
| Interviewer C | 13 | 2.8 | 15% | 23% |
These numbers are invented, to show what to look for. Interviewer B looks severe and A lenient. Before acting on that, check two things:
- Did they see similar candidates? If B interviewed only the later rounds, or only applicants from one source, B may simply have seen a different group. Patterns mean something only when candidates reached interviewers more or less by schedule.
- Do they disagree on the same candidates? The strongest evidence is a candidate both interviewers scored on the same competency. If B is a point lower on most shared candidates, it is B's reading of the scale.
With only three or four candidates per interviewer, differences like these are mostly chance. Treat them as a reason to read the scorecards, not as a finding.
Should you adjust the scores, re-score, or re-interview?
| Option | How it works | When to use it | Limits |
|---|---|---|---|
| Rank within each interviewer | Compare a candidate with others the same interviewer saw, then combine the rankings | Each interviewer saw several candidates for the role | Breaks down when one interviewer saw only strong or only weak candidates |
| Subtract each interviewer's average | Express every score as above or below that interviewer's own average | As a flag that a gap between finalists may be about interviewers | Same problem, more confidently stated. Never the deciding step |
| Re-score key answers | One reviewer re-scores the must-have answers for every finalist, from notes or transcripts, against the anchors, without seeing the original scores | Finalists within about a point of each other on a must-have | Needs written evidence; recordings need the candidate's consent |
| Common final step | Every finalist still in contention meets the same interviewer, with the same questions | The evidence is thin, or the first two conditions above failed | Costs time; offer it to every finalist, not only the one in doubt |
For most teams the sequence is: use the interviewer pattern to see whether there is a problem, re-score the must-have answers to settle it, and add a common step only when the notes are too thin to re-score. The reviewer should be someone who did not interview the finalists, working from the candidate's words rather than the interviewer's adjectives; evidence-based interview feedback covers what those notes need to contain.
A worked example
Invented case. Two finalists for an operations manager role. Avery was interviewed in round 2 by Interviewer B and scored 2 on "process improvement" (a must-have). Blake was interviewed by Interviewer A and scored 4 on the same competency.
Pattern: across the role, B averages 2.4 and A averages 3.2, and on the two candidates they both scored, B was lower by a point each time.
Re-score: the recruiter asks the hiring manager, who saw neither round 2, to re-score both answers from the written notes against the anchor ("identified a bottleneck from data, changed the process, measured the result"). Avery's notes show a measured reduction in order errors after a change she led: 3. Blake's notes describe an initiative with no measurement: 3.
Result: on process improvement the finalists are level, and the decision moves to the other competencies. Interviewer B's reading of the anchor goes on the agenda for the next calibration session.
What about recruiters who run many screens a day?
Agency and high-volume recruiters face the reverse problem. The interviewer is constant, but by the afternoon the candidates blur, and the client's interviewers in the next round are all different people. A few habits keep the comparison neutral:
- One screen template for the role, with the must-ask questions in the same order for every call, and the anchors printed beside them.
- Score immediately after each call, against the anchors, with two or three of the candidate's own phrases under each score. Memory at 5 p.m. is not evidence.
- Compare at the end of the day by competency, one row at a time across candidates, not one candidate at a time.
- Send client interviewers the same scorecard, and ask each for evidence, separately, before you relay anyone's verdict.
A neutral comparison checklist you can copy
CANDIDATE COMPARISON CHECK: [role]
[ ] Same must-ask questions for each competency, same stage
[ ] Same anchored scale, descriptions for every point
[ ] Scorecards submitted before any discussion
[ ] Candidate's words written under every must-have score
[ ] Interviewer pattern reviewed (average, share of top and
bottom scores, gaps on shared candidates)
[ ] Close finalists (within 1 point on a must-have) re-scored
by one reviewer from the notes, blind to original scores
[ ] Common final step offered to every finalist if notes are thin
[ ] Interviewers with a consistent gap added to calibration
[ ] Decision record names the evidence, not the interviewerHow do you avoid the problem next time?
- One interviewer, or one pair, per stage. OPM's guide says that when feasible the same interviewers should be used across all candidates, in a panel or one after another, to keep ratings consistent.
- If volume forces a split, assign candidates by calendar, not by who seems strongest, and keep at least one shared interviewer in the final stage.
- Calibrate before the search, by having every interviewer score the same recorded or written answer and comparing; see interview calibration.
- Watch drift during the search; how to score interviews consistently has a check you can run on your own scores.
Different interviewers are not the only variable that can tilt a comparison. If some finalists were seen in person and others on video, see how to compare candidates interviewed on video and in person; for the full finalist decision, use how to compare candidates after interviews.
Questions people ask
Can I fairly compare candidates if different people interviewed them?
Yes, if every interviewer asked the same questions and scored against the same written anchors, with the candidate's words under each score. Then check whether one interviewer scores consistently higher or lower than the others, and for close calls have one person re-score the key answers from the notes. Without shared questions and anchors, no adjustment makes the scores comparable.
How do I know if one interviewer scores more harshly than the others?
Compare each interviewer's average score and their share of top and bottom ratings across every candidate they scored for the role, and look at whether they differ from colleagues on candidates both of them saw. A consistent gap across many candidates suggests strictness; a gap based on three or four candidates is mostly noise.
Should I adjust scores for a tough interviewer?
Use the adjustment as a flag, not a verdict. Subtracting each interviewer's average only works when interviewers saw similar candidates, which is rarely true when a senior interviewer handles later rounds. For finalists who are close, re-scoring their answers with one reviewer against the anchors is more reliable than arithmetic.
What is the fairest way to split interviews across a team?
Keep one interviewer, or one pair, per stage for every candidate on the role wherever you can, as the U.S. Office of Personnel Management's structured interview guide recommends. If volume forces you to split, assign candidates to interviewers by schedule rather than by impression, and have every interviewer use the same questions and anchors.