How AI interview scoring works: transcripts, rubrics, evidence and the risks to check
On this page
- What happens, step by step, when AI scores an interview?
- What are the two main kinds of AI interview scoring?
- Where does AI interview scoring go wrong?
- What laws apply to AI interview scoring?
- What should you ask a vendor about AI interview scoring?
- How do you test AI scoring before you rely on it?
- How should people and AI scores work together?
- Questions people ask
AI interview scoring works in four stages: speech recognition turns the interview into a transcript, the system matches each answer to the question it answers, a model rates each answer against a rubric of competencies and written anchors, ideally quoting the candidate's own words as evidence, and the ratings are combined into a score that a person should review before any decision. Each stage can fail: transcripts are less accurate for some speakers than others, models can cite evidence the candidate never gave or reward longer answers, and a score that screens out a protected group at a higher rate must be job-related under Title VII. As of November 2026, New York City requires a bias audit and candidate notice when such a tool substantially assists a hiring decision, Illinois regulates AI analysis of video interviews and AI with discriminatory effects, and California's civil rights regulations cover automated-decision systems.
Not legal advice. The legal points summarize the statutes, regulations and agency pages linked below as checked in November 2026. AI hiring rules change quickly; the dated state-by-state table is on AI hiring laws by state.
What happens, step by step, when AI scores an interview?
- Capture. Audio, and in some products video, from a live call, a recorded one-way interview or an AI interviewer.
- Transcription and speaker labels. Speech-to-text produces the words, and the system works out who said what. This matters more than it sounds: if the interviewer's words land on the candidate's side, the candidate gets credit for the question.
- Matching answers to questions. Each stretch of the candidate's speech is tied to the question or follow-up it answers.
- The rubric. Competencies, their weights, and anchors that say what a low, middle and high answer looks like for this role. A model scores against whatever rubric it is given, so a vague rubric produces vague scores; see behaviorally anchored rating scales and how to build an interview scorecard with AI.
- Scoring with evidence. A language model reads the answer and the rubric and returns a score and a rationale. Better designs require the candidate's own words as evidence for every score, and mark a competency "not assessed" when the answers never spoke to it.
- Evidence check. Every quote is matched back to the transcript. A quote that is not there is a paraphrase or an invention and should be removed, and a score left without evidence should be flagged or capped.
- Combining scores. Usually a weighted average across competencies, with a clear note when only some competencies were scored.
- Human review. The interviewer reads the evidence, accepts or changes each score, and a named person makes the decision.
- Records. The transcript, the rubric version, the model's scores, any overrides and the final decision are kept together.
We make Interview Signal. Its scorecard is one example of steps 5 to 8: each area is scored from the candidate's own words, every quote is checked word for word against the candidate's side of the transcript and removed if it is not there, an area left with no quote is capped at 5 of 10 and marked "Probe further," and the interviewer can replace any score with their own, as described in how the scorecard with verified quotes works.
What are the two main kinds of AI interview scoring?
Products differ most in what the score is compared with. Ask which kind you are buying before anything else:
| Rubric scoring of the transcript | Predictive scoring from past data | |
|---|---|---|
| What the model reads | The words of each answer, the question and your rubric | Features of the interview (words, and in some products voice or video) compared with patterns from past employees or past ratings |
| What a score means | How well the answer matches the written anchors | How closely the candidate resembles people previously rated as successful |
| Can you check a score? | Yes, if the rationale cites quotes you can find in the transcript | Often hard: the reasons for an individual score may not be visible |
| Main risk | Invented or misread evidence; a model that rewards fluency or length | Learning past bias from the training data; features that stand in for protected traits |
| Extra legal questions | New York City's audit if the score substantially assists the decision | Illinois's video interview act for AI analysis of recorded video; the federal lie detector rules if the tool analyzes voice stress |
The last row is not hypothetical. The Labor Department's polygraph rule says voice stress analyzers "include any systems that utilize voice stress analysis, whether or not an opinion on honesty or dishonesty is specifically rendered" (29 CFR 801.2(d)(1)), and most private employers may not use lie detectors on applicants; see lie detector tests in hiring.
Where does AI interview scoring go wrong?
| Risk | What happens | How to check |
|---|---|---|
| Transcription errors that differ by speaker | A study of five commercial speech recognition systems on interview audio found an average word error rate of 0.35 for Black speakers against 0.19 for white speakers (Koenecke et al., PNAS, 2020; the systems were 2019 versions). Worse transcripts mean worse evidence for some candidates | Compare transcripts with the audio for a sample of interviews across different speakers; see how accurate are AI interview transcripts |
| Invented or misattributed evidence | Speech recognition can insert phrases nobody said, and a model can put a paraphrase in quotation marks or quote the interviewer | Every quote must appear, word for word, on the candidate's side of the transcript; see AI note taker hallucinations |
| Rewarding length and fluency | Research on language models used as judges documented "position, verbosity, and self-enhancement biases" (Zheng et al., NeurIPS 2023). In an interview, a long polished answer can outscore a short specific one | Score a short, specific answer and a long, general one to the same question and see which wins; anchors should reward checkable specifics |
| Proxies for protected traits | Accent, pauses, disfluency or word choice can track national origin or disability without being job-related | Ask which features are scored; the rubric should score content, not delivery, and candidates need an accommodation route |
| Guessing on missing evidence | A competency the candidate never discussed gets a middling score anyway | The tool should show "not assessed" and say how many competencies the overall score covers |
| Inconsistent results | The same transcript scores differently on two runs or after a model update | Rescore a sample twice, and again after each vendor update |
| Rubber-stamp review | Reviewers accept every score without reading the evidence | Log overrides; if nobody ever changes a score, the review is not real |
What laws apply to AI interview scoring?
- Title VII. A score that causes a disparate impact by race, color, religion, sex or national origin must be "job related for the position in question and consistent with business necessity," and scores and cutoffs may not be adjusted by group (42 U.S.C. 2000e-2(k) and (l)). Software does not move that duty to the vendor; see Title VII disparate impact and AI hiring and the four-fifths rule. What the EEOC withdrew and still says is on EEOC guidance on AI in hiring.
- The ADA. Employment tests that screen out people with disabilities must be job-related, and tests must be given so the result reflects the skill measured rather than "impaired sensory, manual, or speaking skills" (42 U.S.C. 12112(b)(6) and (b)(7)). A candidate with a speech impairment may need a different route, such as a person scoring the interview.
- New York City Local Law 144. An automated employment decision tool is one that uses machine learning, statistical modeling, data analytics or AI, helps make employment decisions, and "substantially assists or replaces discretionary decision-making"; "employment decision" includes screening. Before use, the tool needs a bias audit within the past year with a published summary, and candidates who are city residents must get notice 10 business days before use, including how to request an accommodation (DCWP FAQ). Whether a given tool is covered is worked through on what is an automated employment decision tool and the bias audit page.
- Illinois Artificial Intelligence Video Interview Act. Since January 1, 2020, an employer that asks applicants for Illinois-based positions to record video interviews and uses AI to analyze them must, before the interview, notify the applicant, explain "how the artificial intelligence works and what general types of characteristics it uses to evaluate applicants," and get consent; it may share the videos only with people whose expertise is needed, must delete them within 30 days of a request, and must report demographic data if it relies solely on the AI to decide who gets an in-person interview (820 ILCS 42). Detail on our Illinois page.
- Illinois Human Rights Act. Since January 1, 2026, it is a civil rights violation, with respect to recruitment, hiring and other employment decisions, to use AI "that has the effect of subjecting employees to discrimination on the basis of protected classes," to use zip codes as a proxy for protected classes, or to fail to give notice of AI use (Public Act 103-0804).
- California. The Civil Rights Council's regulations, in effect since October 1, 2025, make clear that an automated-decision system may violate state law if it harms applicants based on protected characteristics, and require records "including automated-decision data" to be kept for at least four years (Civil Rights Department; our California page).
- Colorado. SB 26-189, signed May 14, 2026, covers automated decision-making technology used in consequential decisions, which include employment, with notice, an explanation within 30 days of an adverse outcome and a right to request human review; the bill summary ties January 1, 2027 to developers' documentation duties and the attorney general's rules (Colorado General Assembly; our Colorado page).
- Outside the U.S. For candidates in Europe, see the EU AI Act in recruiting and GDPR Article 22.
What should you ask a vendor about AI interview scoring?
- What exactly is scored: the words only, or also voice, facial or other video features?
- Where does the rubric come from, and can we edit the competencies, weights and anchors for each role?
- Does every score cite the candidate's own words, and are quotes checked against the transcript automatically?
- What happens when an answer gives no evidence for a competency?
- Which speech recognition system do you use, and what accuracy data do you have across accents, dialects and speech conditions?
- How do you separate speakers in panels, phone screens and in-person interviews?
- What adverse impact testing have you done, on what data, and will you support our own pass-rate monitoring and any bias audit a city or state requires?
- Can interviewers override scores, and are overrides logged with a reason?
- What is the alternative route for a candidate who needs an accommodation or who will not be scored this way?
- What data do you keep, where, for how long, and is any of it used to train models? See do AI note takers train on interview recordings.
How do you test AI scoring before you rely on it?
- Pick past interviews you already scored by hand. For example, 20 to 30 interviews for one role, handled under your own notice and retention policy. The number is an example; more is better.
- Check the transcripts against the audio for a subset that covers different speakers.
- Score them with the tool and with two people working independently. Study the disagreements competency by competency rather than one overall agreement figure.
- Check every quote the tool cites. One invented quote in a pilot is a finding, not noise.
- Rescore the same transcripts a second time to see whether the scores hold.
- Compare pass rates by group at the stage where the score will be used, with the four-fifths rule as a first check and care with small numbers.
- Write down how the score will be used: advisory evidence for the interviewer, or a threshold. The second carries far more legal weight in places like New York City.
- Repeat the check after any change to the rubric or the vendor's model.
How should people and AI scores work together?
The safest pattern keeps the model as an evidence gatherer and the interviewer as the scorer of record. Ask interviewers to read the quoted evidence before the number, require a short reason for every override, and never reject a candidate on an AI score alone. Colorado's 2026 law gives people a right to request meaningful human review and reconsideration after an adverse decision (when its duties apply is on our Colorado page), and Illinois adds a reporting duty for employers that rely solely on AI to screen video interviews, so a documented human decision is worth having everywhere. Notes about the candidate's evidence belong in the scorecard; impressions about anything else do not, as what not to write in interview notes explains.
Staffing agencies carry these duties in their own right. New York City's law names employment agencies alongside employers, and it applies when the agency itself is located in the city. When an agency sends a client a scored candidate, send the evidence with the score, say which parts were model-generated and which were the recruiter's judgment, and keep the transcript and rubric version on file in case the client or the candidate asks how the score was reached.
Questions people ask
How does AI score a job interview?
Speech recognition turns the interview into a transcript, the system matches each answer to the question it answers, and a model rates each answer against a rubric of competencies with written anchors for each score level, ideally quoting the candidate's own words as evidence. The ratings are combined, usually as a weighted average, into an overall score that a person should review before any decision.
Is AI interview scoring accurate?
It is only as accurate as its weakest step. Transcripts can contain errors, and a 2020 study of five commercial speech recognition systems found an average word error rate of 0.35 for Black speakers against 0.19 for white speakers. Models can also cite evidence the candidate never gave or favor longer answers. Test any tool on your own past interviews, check every quoted piece of evidence against the transcript, and keep a person responsible for the score.
Is it legal to use AI to score job interviews?
Yes in the United States, but the employer stays responsible for the outcome. Title VII requires a score that screens out a protected group at a higher rate to be job related and consistent with business necessity, the ADA requires accommodations, and some places add their own rules: New York City requires a bias audit and candidate notice for automated employment decision tools, Illinois regulates AI analysis of video interviews and AI with discriminatory effects, and California's civil rights regulations cover automated-decision systems.
Does AI interview scoring need a bias audit?
In New York City it does if the tool is an automated employment decision tool: one that uses machine learning, statistical modeling, data analytics or AI to help make employment decisions and substantially assists or replaces discretionary decision-making. The audit must be done within a year before use and its summary published, and candidates must get notice 10 business days before use. Elsewhere, no general audit law applies as of November 2026, but adverse impact checks are still the way to show a score is fair.
What should a recruiter check before trusting an AI interview score?
Read the evidence, not the number. Check that every quote behind the score appears in the candidate's own words in the transcript, that the score cites specifics rather than how fluent or long the answer was, that competencies the candidate never addressed are marked not assessed rather than guessed, and that the transcript itself is accurate where it matters.