How to evaluate your interviewers: the measures to track and how to give the feedback

To evaluate your interviewers, track a few measures for each person over a quarter: how fast and how completely they submit scorecards, whether each rating cites specific evidence, how many of the planned questions they actually ask, how their pass rate and scores compare with other panelists who saw the same candidates, and what candidates say about them. No single number grades an interviewer. The pattern across the measures, read next to a sample of their scorecards, tells you who needs calibration, who needs coaching and who is quietly doing the best work on your panel.

This page covers the measures, how to calculate them without fooling yourself, how to read the patterns, and how to give interviewers the feedback. It assumes people already interview alone; getting them there is the interviewer training checklist, and the statistics of whether two interviewers agree are in inter-rater reliability in interviews.

What makes an interviewer good, in terms you can measure?

Strip away style and a good interviewer does five things: asks the questions the role needs, collects job-related evidence, scores that evidence against the agreed scale, does it independently and on time, and treats the candidate in a way that does not cost you the hire. Each of those leaves a trace in data you already have. The U.S. Office of Personnel Management's structured interview guide (2008) is a useful reference: it says notes should "document the interviewer's reasoning for each rating on each competency," that interviewers should rate immediately after the candidate leaves, and that the date, length, questions, scores, notes and interviewer training for each interview should be recorded and kept. Those records are the raw material for everything below.

Which measures should you track for each interviewer?

Two more are worth adding once the basics run: talk-time share in recorded or transcribed interviews (see interviewer talk-time ratio), and agreement with co-panelists, which the inter-rater reliability page shows how to compute. Pick three or four measures to start with; a dashboard with twelve columns gets read once.

How do you calculate pass rates and drift without fooling yourself?

Raw pass rates mislead because interviewers see different candidates. Someone who only runs final rounds will pass more people than someone who runs first screens, because the weak candidates never reach them. Compare each interviewer with the panelists who saw the same candidates. An example with invented numbers, for one quarter on one role:

Read it in that order: C passes far more people than colleagues who heard similar candidates, so check C's evidence for leniency or a halo effect. A is stricter than the panel; that is only a problem if A's scorecards do not support the lower ratings. D looks extreme, but six candidates is too few to read anything into. As a working rule, wait until an interviewer has a couple of dozen scorecards on comparable candidates before you draw a conclusion; that is a judgment call, not a statistical threshold.

Drift works the same way. If an interviewer's average rating rose half a point on a four-point scale while the panel's stayed flat, the interviewer's reading of the anchors moved. That is what an interview calibration session exists to fix.

One legal check belongs here too. The federal Uniform Guidelines treat a selection rate for any race, sex or ethnic group that is less than four-fifths of the rate for the group with the highest rate as generally regarded as evidence of adverse impact, while noting that differences based on small numbers may not be statistically significant (29 CFR 1607.4(D), as of October 2026). If you can see pass rates by interviewer, it is worth having whoever holds your applicant demographic data look at them by group as well, with the small-numbers caution in mind. This is not legal advice; involve counsel before acting on a disparity.

Where does the data come from?

• Your ATS. Scorecard timestamps, completeness and ratings are already there. In Greenhouse, the Interviewer Calibration report shows total scorecards completed, adherence to highlighted focus attributes, ratings and no-decisions per interviewer; Greenhouse's help center says it needs Job Admin permissions or above and is available on the Core, Plus and Pro tiers (as of October 2026). Without such a report, an export of scorecards with submission times, pivoted in a spreadsheet, gives the same view.
• A sample of scorecards. Evidence quality cannot be counted by software reliably. Read ten recent scorecards per interviewer each quarter and mark each rating "cites evidence" or "does not." Ten minutes per person.
• Candidate surveys. Add one or two questions about the interview itself to your candidate experience survey, and record which interviewers the candidate met so answers can be grouped.
• Call reviews. Where interviews are recorded with consent, listening to one interview per person per quarter shows question coverage and manner better than any field. The routine is in how to coach recruiters with call reviews.

If your recruiters run interviews in Interview Signal, which we make, the Team plan's weekly coaching report shows each recruiter's must-ask coverage, suggested follow-ups used, scorecard completeness and average interview length. It is built from counts only, not transcripts, so it covers coverage and completeness but cannot judge how well a question was asked.

How do you read the patterns?

Most of these causes are listed, with fixes, in common interviewer mistakes. Use that page as the reference when you sit down with someone.

How do you give interviewers this feedback?

1. Make it routine and private. A short quarterly note to every interviewer, not only the ones with problems, so receiving feedback is normal.
2. Show their numbers next to the panel's. "Your advance rate was 78% on candidates the rest of the panel advanced 44% of the time" is a fact to discuss, not a verdict.
3. Bring one scorecard. Pick a rating and ask what the candidate said that earned it. The conversation about evidence does more than the percentage.
4. Agree one change. One quote per rating, ask the third required question, submit before the debrief. Write it down.
5. Check next quarter. If the same pattern persists after calibration and a coaching conversation, take the person out of the loop for that role until they have shadowed again. Say so plainly.
6. Recognize the good ones. Interviewers with fast, evidenced, well-calibrated scorecards are the people to ask to reverse-shadow new interviewers; see interviewer shadowing.

Wording matters. "Your scores are wrong" ends the conversation; "your 4s and the panel's 4s seem to mean different things, can we look at one together?" starts it. For interviewers who are senior to you, frame the data as being about the process: the team is checking that a 4 means the same thing in every room.

What should you not do with these numbers?

• Do not publish a league table. Ranking interviewers publicly teaches them to match the average, which kills the independent dissent a panel needs.
• Do not reward a target pass rate. There is no right rate; it depends on how good your pipeline is.
• Do not punish disagreement. A well-evidenced "no" against four "yes" votes is the panel working. Judge the evidence, not the vote.
• Do not read small samples. A handful of interviews says almost nothing.
• Do not judge interviewers only by later hire performance. Whether their "yes" candidates succeed is worth tracking over a year or more with a quality of hire measure, but it is slow, sees only the people you hired, and is shaped by everyone else in the loop.

Done this way, evaluating interviewers stops being an awkward performance conversation and becomes part of how the panel keeps its scale honest: a few numbers, a sample of real scorecards, one agreed change and a check next quarter.