Inter-rater reliability in interviews: how to measure whether your interviewers agree

Inter-rater reliability in interviews is how consistently different interviewers score the same candidate. If two trained interviewers would reach different verdicts on the same evidence, the hiring decision depends on who happened to be scheduled. Research gives a baseline: a meta-analysis of 125 reliability coefficients found a mean interrater reliability of .74 for panel interviews and .44 for separate interviews by different interviewers (Huffcutt, Culbertson and Weyhrauch, 2013). You can measure your own with percent agreement, Cohen's kappa or an intraclass correlation, using scores you already collect.

This page explains what the number means, what research found, how to calculate it with a worked example, and how to raise it. The session that fixes disagreement, scoring sample answers and rewriting anchors, is the interview calibration session.

What does inter-rater reliability measure in an interview?

It answers two different questions, and it helps to know which one you are asking:

• Do interviewers who hear the same answers score them the same way? This is the panel question. Everyone heard identical evidence, so disagreement comes only from how they read it against the scale.
• Do separate interviews of the same candidate reach the same conclusion? This is the loop question. Different interviewers ask different questions on different days, so disagreement also comes from the questions and from how the candidate performed each time.

Huffcutt and colleagues describe three sources of measurement error behind this: random response error, transient error (the candidate's day-to-day variation) and conspect error (differences between raters). Panel estimates capture mostly the last; separate-interview estimates capture all three, which is why they come out lower. The authors' practical warning is that you should not judge your process on estimates that leave sources out.

How reliable are interviews, according to the research?

The validity ceiling is the reason reliability matters. A score that two interviewers cannot agree on cannot predict much about job performance either, because part of it is noise about the interviewer. Raising reliability does not guarantee better hires, but low reliability guarantees a cap on how good they can be.

How do you measure interviewer agreement yourself?

You need scores that were submitted independently (see independent interview feedback before the debrief), and pairs of interviewers who rated the same thing. In most loops each interviewer covers different competencies, so measure agreement on the overall hire or no-hire recommendation, on a competency two interviewers both assess on purpose, or on panel interviews.

1. Percent agreement

The share of candidates on whom two interviewers land on the same side of the hire line. An example with invented numbers: over a quarter, 40 candidates were each seen by the same pair of panelists.

They agreed on 18 + 14 = 32 of 40 candidates: 32 ÷ 40 = 80%.

2. Cohen's kappa

Percent agreement includes agreement that would happen by chance. If both interviewers said "hire" to most candidates, they would often agree without reading the evidence the same way at all. Cohen's kappa removes the chance part:

kappa = (observed agreement - chance agreement) / (1 - chance agreement)

Observed agreement   = 32 / 40 = 0.800
A says hire          = 23 / 40 = 0.575     A says no = 0.425
B says hire          = 21 / 40 = 0.525     B says no = 0.475
Chance agreement     = (0.575 x 0.525) + (0.425 x 0.475)
                     = 0.302 + 0.202 = 0.504
kappa                = (0.800 - 0.504) / (1 - 0.504) = 0.60

How good is 0.60? It depends on whose scale you use. In a widely cited open-access review, McHugh (2012, Biochemia Medica) describes the traditional interpretation, where 0.41 to 0.60 counts as "moderate," and argues it is too lenient; her stricter table calls 0.40 to 0.59 "weak" and 0.60 to 0.79 "moderate." Her context is health research, not hiring, so treat either scale as a reference, not a pass mark. The useful comparison is your own kappa before and after a calibration session.

3. Intraclass correlation for 1 to 4 scores

When you want to use the full scale rather than just the hire line, and more than two interviewers are involved, the usual statistic is the intraclass correlation (ICC). There are several forms, and the right one depends on whether the same interviewers rate every candidate; Shrout and Fleiss (1979) set out the choices, and Koo and Li (2016) give an open-access guide that treats values below 0.5 as poor, 0.5 to 0.75 as moderate, 0.75 to 0.9 as good and above 0.9 as excellent. Most statistics packages compute it; a people analytics colleague can set it up once.

Doing it in a spreadsheet

Put one row per candidate, with interviewer A's verdict in column B and interviewer B's in column C (1 = hire, 0 = no hire), rows 2 to 41:

Observed agreement   =SUMPRODUCT(--(B2:B41=C2:C41))/COUNT(B2:B41)
A hire rate          =AVERAGE(B2:B41)
B hire rate          =AVERAGE(C2:C41)
Chance agreement     =F2*G2+(1-F2)*(1-G2)      (F2, G2 = the two hire rates)
Kappa                =(E2-H2)/(1-H2)            (E2 = observed, H2 = chance)
Within one point     =SUMPRODUCT(--(ABS(J2:J41-K2:K41)<=1))/COUNT(J2:J41)

The last line works on 1 to 4 scores in columns J and K. "Within one point" is a rough check, not a reliability coefficient, but it is easy to explain to a hiring team.

What does low agreement usually point to?

Read the pattern before you fix anything. Low agreement has a few typical shapes:

Check every quarter, or after every 20 or so candidates for a high-volume role, whichever comes first. That cadence is a practical habit, not a research threshold; with only a handful of candidates, differences are mostly noise.

How do you raise inter-rater reliability?

Conway's meta-analysis points to the levers, and they are all within a hiring manager's control:

1. Standardize the questions. Same core questions, same order, for every candidate. See what is a structured interview.
2. Standardize how answers are scored. Anchors that describe what a 2, 3 and 4 answer contains. Where two interviewers split by two points, the anchor wording is usually the cause.
3. Combine ratings by rule, not by discussion alone. A written rule, like the one in the hire or no-hire decision framework, turns scores into a verdict the same way every time.
4. Train interviewers, with practice scoring and feedback. The calibration session is the practice; interviewer shadowing brings new interviewers onto the same scale.
5. Keep the same interviewers across candidates for a role where you can. OPM's guide recommends it for consistency, and Huffcutt and Woehr (1999) concluded from 120 studies that the same interviewer should be used across all applicants, especially when the interview itself is not highly structured.

Interviewer behavior matters as much as the paperwork. An interviewer who skips the kit's questions or scores on first impressions will drag agreement down; the most frequent patterns are in common interviewer mistakes.

What does high agreement not tell you?

• That the standard is right. A panel can agree reliably on something that does not predict the job. Check outcomes separately; see how to measure quality of hire.
• That the scores were independent. Agreement reached by reading each other's feedback is not reliability. Lorenz and colleagues (2011) found that social influence made groups converge and feel more confident without becoming more accurate.
• That every competency is reliable. An overall verdict can agree while one competency is scored at random. Break it down by competency before you celebrate.
• That disagreement is bad. A split on a candidate is information. The goal is not unanimity; it is that disagreements come from different evidence, not different readings of the scale.