How to evaluate pre-employment assessment vendors: validity, fairness, accessibility and contract terms

Compare pre-employment assessment vendors on evidence, not demos. First define what the job requires through a job analysis; then ask every vendor for the same things: validity evidence for jobs like yours, reliability, score and pass-rate differences between groups, the accommodation process, accessibility, how any AI scoring works and what happens to candidate data. Score the answers on one weighted scorecard, pilot the top one or two on your own roles, and write the evidence and audit duties into the contract. The employer, not the vendor, answers for the test it uses: as of November 2026 federal agencies have stepped back from the Uniform Guidelines, but Title VII, the ADA, state laws and New York City's bias-audit law still expect a test that screens people out to be job related.

For which vendors offer what, see the best pre-employment assessment tools; for what each kind of test measures, see types of pre-employment assessments. This page is the evaluation process itself.

What should you decide before you talk to vendors?

1. What the job requires. A short job analysis listing the tasks and the knowledge, skills and abilities that matter most; the job analysis guide shows how. Vendors should map their tests to this list, not the other way around.
2. Where the test sits. Before or after the first interview, for every applicant or a shortlist. Early, high-volume use magnifies both savings and any adverse impact.
3. How scores will be used. A pass mark, a ranking, or one input among several. A hard cut score needs the strongest evidence.
4. The candidate's time budget. For example, 20 minutes for hourly roles and 45 for professional ones (example limits); long tests lose candidates.
5. Who decides. HR, the hiring manager, legal and IT security each sign off on their part of the scorecard below.

Which questions should you ask every assessment vendor?

Two of those rows come straight from statute. Title VII makes it unlawful "to adjust the scores of, use different cutoff scores for, or otherwise alter the results of, employment related tests on the basis of race, color, religion, sex, or national origin" (42 U.S.C. 2000e-2(l)). The ADA bars tests that "screen out or tend to screen out" people with disabilities unless job-related and consistent with business necessity, and requires tests to be administered so results reflect the skill measured "rather than reflecting the impaired sensory, manual, or speaking skills" of the candidate (42 U.S.C. 12112(b)(6) and (7)).

What counts as validity evidence, and what doesn't?

The clearest written test is still the Uniform Guidelines on Employee Selection Procedures. Section 1607.9 says that "under no circumstances will the general reputation of a test or other selection procedures, its author or its publisher, or casual reports of its validity be accepted in lieu of evidence of validity," and rules out "all forms of promotional literature," usage figures and "testimonial statements and credentials of sellers, users, or consultants." Section 1607.7 lets you rely on a publisher's studies only when the jobs are shown to be similar, and warns that "users are cautioned that they are responsible for compliance" (29 CFR part 1607).

Their status changed in 2026: as of November 2026 the Justice Department considers the Guidelines unlawful insofar as they impose disparate-impact liability without intent, OPM removed its references to them (91 FR 48234), and OFCCP's copy was removed effective October 26, 2026 (91 FR 54444). The statute's disparate-impact provision, state laws and private claims remain, and the professional bar has not moved: the Society for Industrial and Organizational Psychology's December 2025 statement recommends that, "regardless of regulatory and guidance changes," organizations "continue to follow established professional standards." So ask for the same evidence you would have asked for in 2025.

How do you score vendors side by side?

Rate every finalist on the same rubric, 1 to 4 per criterion, before anyone sees pricing. The weights below are an example; set your own before the first demo.

Have the hiring manager, an HR lead and someone from legal or compliance score independently, then compare; a vendor that wins on demo polish and loses on evidence should not make the shortlist.

How do you run a pilot before you buy?

1. Pick one role with enough hiring volume to learn from.
2. Run the test without using the scores. Give it to current employees in the role, or to applicants in parallel with your existing process, so nobody is rejected on an unproven test.
3. Compare scores with outcomes you trust: structured interview ratings, work samples, later performance. If high scorers and low scorers look the same, the test is not telling you anything for this job.
4. Check pass rates by group at the proposed cut score where you lawfully hold the data, and read the four-fifths rule page on why small samples mislead.
5. Collect candidate feedback and completion rates, including accommodation requests and how fast they were handled.

When test results and interviews point in different directions during the pilot, the guide to conflicting results covers how to weigh them.

What should the contract say?

• Evidence on request: the vendor supplies validation documentation, group-level results for your candidates and model or test changes before they go live.
• Audit support: if the tool falls under New York City's Local Law 144, the vendor provides the data an independent auditor needs; see the NYC bias audit guide.
• Records: California's automated-decision regulations, effective October 1, 2025, require employers to keep employment records, "including automated-decision data, for a minimum of four years," per the Civil Rights Council, and they define "agent" as a covered term, so make sure the vendor keeps or hands over what you need.
• Accommodations: who handles requests, and how quickly.
• Data: retention periods, deletion, subprocessors, no training on your candidates' data without consent, and an export when you leave.
• Notices: who writes the candidate notices that state and city laws require; the AI hiring laws by state page lists them.

What are the red flags?

• A claim that the test is "bias-free" or "approved," offered instead of data; treat it as the promotional literature that section 1607.9 says is no substitute for evidence.
• No technical manual, or one that only covers different jobs or a different version.
• Refusal to share group-level data, even under a confidentiality agreement.
• Any adjustment of scores or cut scores by group.
• Scoring of faces, voices or emotions that the vendor cannot explain; see the Illinois AI Video Interview Act and Maryland's facial recognition consent rule for two state laws aimed at exactly this.
• Pressure to use the score as the only decision, with no human review.