How to build an interview scorecard with AI, step by step

To build an interview scorecard with AI, give an AI tool the job description and your intake notes, ask it in stages for the role's outcomes, four to six competencies, a 1 to 4 behaviorally anchored rating scale for each, and questions mapped to each competency, then have the hiring manager cut and edit the draft and test it on one sample answer before the first interview. AI is fast at the first draft. The judgment about what matters, and the check that every question is job-related and lawful, stays with people.

The finished product should look like a normal structured scorecard. If you want the target format first, see the interview scorecard template; this page is about getting there with AI and not letting the AI's habits into the result.

What does AI do well here, and what does it get wrong?

Step 1: what should you give the AI?

The quality of the draft depends on the inputs. Give it four things:

1. The job description, with the company name removed if you prefer.
2. Intake notes: what the person must achieve in the first six to twelve months, the hardest part of the job, and why the last person succeeded or struggled. If you do not have these, run the intake first with these intake meeting questions.
3. Level and context: seniority, team size, who the role reports to.
4. The interview plan: how many interviews, and which one this scorecard is for.

Use an AI tool your organization has approved. There is no candidate data at this stage, but intake notes can contain things you would not post publicly. The AI use policy for interviews template has a section for approved tools.

Step 2: how do you get outcomes and competencies?

Ask for outcomes first. Competencies drafted from outcomes are more specific than ones drafted from a list of duties.

You are helping a hiring manager design a structured interview.
Use only the job description and intake notes below. If you need to
assume something, list the assumption separately.

1. List the 3 to 5 outcomes this person must deliver in their first
   12 months, each in one sentence.
2. From those outcomes, propose 4 to 6 competencies that can be
   assessed in an interview. For each: a one-line definition, which
   outcome it supports, and whether it is a must-have.
3. Do not include personality traits, "culture fit", or anything
   that cannot be shown through past behavior or a work sample.

JOB DESCRIPTION:
[paste]

INTAKE NOTES:
[paste]

Then edit as a person. Strike any competency that is really a personality trait, merge two that overlap, and confirm the must-haves with the hiring manager. If the list is still longer than six, the interview plan should split it across interviewers rather than ask one interviewer to cover everything.

Step 3: how do you get rating anchors that interviewers can use?

For each competency below, write a 1 to 4 rating scale:
4 = strong evidence, 3 = meets the bar, 2 = below the bar,
1 = no evidence or contrary evidence.
Each anchor must describe something an interviewer could hear the
candidate say or describe doing, at the level of this role. Avoid
words like "good", "strong", "excellent" or "solid" unless the anchor
says what that looks like. Keep each anchor to two sentences.

COMPETENCIES:
[paste the edited list with definitions]

Read every anchor and ask: would two interviewers hearing the same answer pick the same number? If an anchor still says "demonstrates good judgment", rewrite it with an example of the behavior. The behavioral anchor library has tested anchors for common competencies you can use as a reference or paste in directly.

Step 4: how do you get questions mapped to each competency?

For each competency, write:
- one primary behavioral question asking about a specific past
  situation,
- two follow-up questions that ask what the candidate personally did
  and what the result was,
- one sign of a strong answer and one sign of a weak answer, using
  the anchors above.
Do not write questions about age, family, health, religion,
nationality, or anything not related to the job. Prefer past examples
to hypotheticals.

Check every question yourself against illegal interview questions. AI drafts sometimes produce a "getting to know you" question that strays into family or background. Also check the questions fit the time: one primary question and follow-ups per competency usually takes eight to twelve minutes.

Step 5: how do you assemble the scorecard?

Put the edited pieces into one page that interviewers fill in during or straight after the interview:

INTERVIEW SCORECARD: [role]     Interview: [round]     Interviewer: [name]
Candidate: [name]               Date: [date]

COMPETENCY 1: [name]   Must-have: [yes/no]   Weight: [%]
  Question: [primary question]
  Follow-ups: [1] [2]
  Anchors: 4 [..]  3 [..]  2 [..]  1 [..]
  Evidence (candidate's words, with "I" or "we"):
  ____________________________________________________________
  Rating: [ ]

COMPETENCY 2 ... (same fields)

MUST-HAVE CHECK:  all must-haves rated 3 or more?  [yes / no: which]
NOT ASSESSED:     [competencies you ran out of time for]
FOR THE NEXT INTERVIEWER: [what to probe]
Submitted at: [time]

If you weight competencies, set the weights now, with the hiring manager, before anyone is interviewed. The weighted interview scorecard shows how to set them and when weighting changes nothing.

Step 6: how do you test it before the first interview?

1. Write one sample answer per must-have competency, or take one from a past interview with identifying details removed.
2. Have two interviewers score it separately using the anchors.
3. Compare. If they differ by more than one point, the anchor needs rewriting, not the interviewers.
4. Time the questions by reading them aloud with follow-ups.
5. Lock the scorecard for the requisition, so every candidate is scored on the same version.

For a fuller version of this exercise across a whole interview team, see how to run an interview calibration session.

A short worked example

One competency from an invented scorecard for a payroll specialist, after human editing:

The AI draft for this competency said "Attention to detail: demonstrates strong accuracy." Everything useful in the final version came from the intake notes and the editing.

What are the common mistakes?

• One giant prompt. Asking for competencies, anchors and questions at once produces a long, shallow document. Ask in stages and edit between them.
• Accepting invented requirements. If the draft mentions a certification or tool the job description never named, delete it or confirm it with the hiring manager.
• Keeping every competency. Ten competencies means none is assessed properly. Cut to the ones the outcomes depend on.
• Different scorecards for different candidates. Regenerating the draft halfway through a search gives later candidates a different test. Lock it.
• Skipping the hiring manager. A scorecard the hiring manager did not edit is one they will not use in the debrief.
• Pasting candidate data into the drafting tool. Building the scorecard needs no candidate information at all; keep it that way.

Should AI help score the interview too?

That is a separate decision from building the scorecard. A tool that drafts evidence from a transcript for the interviewer to confirm is different from one that produces scores or rankings on its own, and the second can bring bias audit and notice duties in some places. Many teams let AI organize the evidence and keep the rating with the interviewer. Interview Signal works that way: it builds the question guide from the job description, fills the scorecard during the call, and every score carries the candidate's own quotes, checked against the transcript, for the interviewer to accept or change. Whatever you use, the scorecard you built here is what keeps the output comparable across candidates; the candidate comparison matrix template picks up from there, and the AI recruiting glossary explains terms such as bias audit and AEDT.