Interview scorecard for QA engineers, with anchors for each competency
On this page
- Which competencies should a QA engineer scorecard cover?
- The scorecard: weights and what a 1, 3 and 5 look like
- Copy-ready QA engineer scorecard
- What should each interviewer ask, and what counts as evidence?
- Two work samples that produce scoreable evidence
- A filled example
- How do you adjust it for the seat and the level?
- Mistakes specific to scoring QA engineers
- Where does this scorecard fit in the hiring process?
- Questions people ask
A QA engineer interview scorecard should score six things: risk-based test design, exploratory testing and defect finding, bug reporting and triage, test automation design and maintenance, CI and release judgment, and collaboration with developers and product. Below is a copy-ready scorecard with example weights, what a 1, 3 and 5 look like on a 1–5 scale, two work samples and a filled example. Score what the candidate decides to test and why, not the tool names on their resume; an ISTQB certificate, if the job requires one, is a pass/fail check, never points.
The five points follow the 1–5 interview rating scale: 1 is no evidence or evidence of the opposite, 3 is a clear example that meets the bar for the level, 5 is rare. The layout comes from the interview scorecard template. The full loop, with a test design session on a discount-code spec, a live bug hunt and a flaky-test code review, is in the interview guide for QA engineers; this page adds the anchors each row is scored against and two different exercises, a CI failure triage and a test-level sort. The scorecard builder prints a 1–4 version if your team uses four points.
Which competencies should a QA engineer scorecard cover?
- Risk-based test design. Do they decide what to test first by what would hurt users or the business most, and use boundaries and negative cases rather than only the happy path?
- Exploratory testing and defect finding. Given an unfamiliar build, do they find the serious problems, or many trivial ones?
- Bug reporting and triage. Can a developer reproduce their report without a follow-up question, and do they separate severity from priority?
- Test automation design and maintenance. Do they put each check at the cheapest level that catches the risk, and keep the suite fast and trustworthy?
- CI and release judgment. When the pipeline is red on release day, can they tell a real regression from a flaky test or a broken environment, and say what the risk is in plain words?
- Collaboration and quality advocacy. Do they get involved at acceptance criteria and design, and push back on a risky release without becoming the gatekeeper everyone works around?
The scorecard: weights and what a 1, 3 and 5 look like
Weights are an example for a mid-level QA automation engineer on a web product. Agree yours with the engineering or QA manager at intake and lock them before the first interview; see weighted interview scorecard for how.
| Competency | Weight | 1 looks like | 3 looks like | 5 looks like |
|---|---|---|---|---|
| Risk-based test design | 20% | Lists happy-path cases in the order the spec was written; no boundaries, no negative cases, no sense of what matters most. | Orders tests by user and business impact, covers boundaries and negative cases, and asks at least one question that exposes a gap in the spec. | All of a 3, plus explains what they would deliberately not test given the time, and what they would watch in production to cover that gap. |
| Exploratory testing and defect finding | 15% | Clicks through the main flow once; finds only cosmetic issues. | Works from a stated charter (an area and a risk), varies inputs and states, and finds the high-severity planted bug. | Finds the high-severity bug and a hidden one, and explains the pattern that led them there (for example, "anything with money and a boundary"). |
| Bug reporting and triage | 15% | "It's broken on mobile": no steps, no expected versus actual, no environment. | Numbered steps, expected and actual results, environment and build, evidence, and a severity with a reason a developer can reproduce from. | All of a 3, plus narrows the conditions (which inputs, which browsers) before filing and separates severity from priority unprompted. |
| Test automation design and maintenance | 20% | Wants every check as a UI end-to-end test; describes fixed sleeps or rerunning until green as normal. | Places checks at unit, API or UI level by what each catches and costs; uses explicit waits and isolated test data; has fixed a flaky test at its cause. | Designed or reshaped a suite others rely on (structure, data setup, run time) and can say how flakiness or run time changed after. |
| CI and release judgment | 15% | Treats any red build as "do not ship" or any green build as "safe", with no reasoning about what failed or what was not tested. | Separates real regressions from flaky tests and environment failures with evidence, and states the remaining risk in customer terms to the person who decides. | All of a 3, plus a change they made to the pipeline or release process (quarantine with an owner, a smoke gate, a rollback check) and what it changed. |
| Collaboration and quality advocacy | 15% | Describes developers as the opposition, or only tests what is handed over at the end. | Joins refinement to shape acceptance criteria, pairs with developers on testability, and raises risk without blocking by default. | Changed how a team builds quality in (developers writing more of their own tests, a shared definition of done) and can show the result. |
Copy-ready QA engineer scorecard
Candidate: [name] Role: [QA engineer / SDET / manual tester], [level]
Interviewer: [name] Stage: [screen / test design / triage exercise / developer panel]
Pass/fail (checked before scoring)
Certification required by the job description (e.g. ISTQB CTFL): [ ] verified [ ] n/a
Can meet the schedule stated in the job description (release windows, on-call if any):
[ ] yes [ ] no
Risk-based test design (20%) Score 1-5: [ ]
Evidence: [first tests chosen and why, boundaries, questions about the spec]
Exploratory testing and defect finding (15%) Score 1-5: [ ]
Evidence: [charter stated, bugs found and their severity]
Bug reporting and triage (15%) Score 1-5: [ ]
Evidence: [steps, expected vs actual, environment, severity vs priority]
Test automation design and maintenance (20%) Score 1-5: [ ]
Evidence: [test levels chosen in the sort, a flaky test fixed at its cause]
CI and release judgment (15%) Score 1-5: [ ]
Evidence: [how each failure in the triage was classified, what they told the team]
Collaboration and quality advocacy (15%) Score 1-5: [ ]
Evidence: [involvement before testing, a release they pushed back on]
Weighted score: sum of (weight x score) / sum of weights scored = [ ]
Recommendation: [Strong yes / Yes / No / Strong no]
What should each interviewer ask, and what counts as evidence?
The table gives the 1, 3 and 5. Here is the question that tests each row and what a 2 and a 4 sound like.
Risk-based test design
Ask: "A password reset feature ships Friday. You have one day. What do you test first, and what do you leave out?" Evidence: token expiry and reuse, rate limits and account enumeration ranked above copy changes; a stated reason for what is left out.
- 2: Covers the main cases and a few negatives, but cannot say which matter most.
- 4: Ranks by risk and names a spec gap, such as what happens to an old reset link after the password changes.
Exploratory testing and defect finding
Ask: "Walk me through the last serious bug you found that no test caught. How did you find it?" More in interview questions for attention to detail.
- 2: Found it by chance and cannot describe an approach they would repeat.
- 4: Describes a charter or heuristic, the bug's impact, and a test added afterward so it stays fixed.
Bug reporting and triage
Ask: "What is the difference between severity and priority? Give me a bug from your work where they were different." Evidence: a real example, such as a crash in a rarely used admin screen (high severity, lower priority).
- 2: Knows the definitions but the example blurs them.
- 4: Clear example, and describes how they agreed priority with the product owner rather than setting it alone.
Test automation design and maintenance
Ask: "Tell me about a flaky test you fixed. What was the cause, and how did you prove it?" Then the test-level sort below.
- 2: Added retries or a longer wait and the test "stopped failing".
- 4: Found the cause (shared data, timing, test order), fixed it, and moved part of the check to a faster level.
CI and release judgment
Ask: the CI failure triage below, then "Who decides whether to ship, and what do you give them?"
- 2: Classifies the failures correctly but frames the answer as "QA says no" without the customer risk.
- 4: Correct classification, a short written risk statement, and a follow-up so the flaky test does not stay quarantined forever.
Collaboration and quality advocacy
Ask: "Tell me about a release you thought should not go out. What did you do, and what happened?" More in interview questions for quality focus.
- 2: Raised the concern late, or escalated over the team's head without talking to the developer first.
- 4: Raised it early with evidence, offered options (ship behind a flag, ship with a known-issue note), and accepted the decision once it was made by the right person.
Two work samples that produce scoreable evidence
Both are invented and the same for every candidate. Tell candidates in advance how long each takes and how to ask for an adjustment, such as extra time or a screen reader; see ADA accommodations in interviews. If you send either as a take-home, read how to evaluate a take-home assignment first.
CI failure triage (20 minutes). Show an invented pipeline summary for a release candidate: 412 tests, four failures. (1) A checkout test expects a total of 107.10 and gets 107.09; the commit list shows a change to how tax is rounded. (2) A login UI test timed out waiting for a button; its history shows it failed and passed on rerun three times this week. (3) Both email tests failed with "connection refused" to the mail service stub. (4) A report test asserts on yesterday's date written into the test as a fixed value. Brief: "Release is at 3 p.m. What do you do about each failure, in what order, and what do you tell the team?"
What to listen for: failure 1 treated as a likely real regression that touches money and goes to the developer first; failure 2 as flaky, with a plan to find the cause rather than just rerun; failure 3 as an environment problem to fix and rerun before judging the code; failure 4 as a broken test, not a broken product. Strong candidates say what they would tell the release owner in one or two sentences. This scores CI and release judgment, and automation maintenance.
Test-level sort (15 minutes). Hand over ten invented checks for a password reset feature, for example: the token expires after 30 minutes; the email template renders the user's name; the reset page works on a small phone screen; five wrong attempts lock the form; the new password must meet the policy; an old token stops working after a successful reset; the link in the email goes to the right environment. Brief: "Put each check at unit, API, UI end-to-end, exploratory or not worth automating, and tell me why."
What to listen for: rules such as expiry and password policy placed at unit or API level; one or two end-to-end tests for the full journey, not ten; layout on a small screen left to exploratory or visual checks; and a reason for each choice based on what the check catches and what it costs to keep. This scores automation design and risk-based test design.
A filled example
A made-up candidate for a mid-level QA automation engineer on a subscription billing product, scored after the test design session, the CI triage, the test-level sort and the developer panel:
Risk-based test design — 4. Put proration on plan changes and failed-payment retries first; asked what happens to a discount when a customer downgrades mid-cycle, which the spec did not say.
Exploratory testing and defect finding — 3. Found the planted negative-total bug working from a "money at the boundaries" charter; missed the hidden date bug.
Bug reporting and triage — 4. Report was reproducible by the panel developer without questions; narrowed the bug to fixed-amount codes before filing.
Test automation design and maintenance — 3. Sorted eight of ten checks to sensible levels; described fixing a flaky test caused by shared test accounts.
CI and release judgment — 4. Classified all four failures correctly; sent the rounding change to its author first and drafted a two-line risk note for the release owner.
Collaboration and quality advocacy — 2. Examples were all about testing finished work; no involvement in acceptance criteria or design described.
Weighted score: 3.35 ((4×20 + 3×15 + 4×15 + 3×20 + 4×15 + 2×15) ÷ 100). Recommendation: Yes. Strong on risk and release judgment; ask the hiring manager whether the team can bring the new hire into refinement from the first sprint, and carry that into the first month's goals.
How do you adjust it for the seat and the level?
| Seat or level | Adjustment |
|---|---|
| Manual or exploratory tester | Raise exploratory testing and bug reporting to 25% each; replace automation design with a row on test documentation (charters, session notes, test cases others can run). |
| SDET or test automation engineer | Raise automation to 30% and score it partly on test code in a live review; compare with the software engineer scorecard if the seat writes product code too. |
| Performance or load tester | Replace exploratory testing with a row on workload modeling and reading results (which percentile, which bottleneck, what changed between runs). |
| Mobile QA | Add device and OS coverage choices to test design, and app-store release steps to release judgment. |
| Junior or career changer | Accept course, bootcamp or open-source examples; score design and bug reporting at full weight and accept smaller scope for a 3. |
| QA lead | Add a row on test strategy across teams and coaching; a 3 on collaboration needs a change to how a team works, not one release. |
Mistakes specific to scoring QA engineers
| Mistake | Why it fails | Fix |
|---|---|---|
| Scoring tool names (one framework over another) | Tools change; the judgment about what to test at which level carries over | Score the reasoning in the test-level sort, not the framework named |
| Counting bugs found | Rewards volume of cosmetic findings over the bug that loses money | Score against the planted list and its severity, written before the first candidate |
| Rewarding "I would block the release" | Blocking by default makes QA the team everyone routes around | Score how clearly they state risk and who they give the decision to |
| Points for a certificate | A certificate shows a syllabus was studied, not how someone tests | Pass/fail line only if the job requires it |
| "Digital native" or "young team" as a plus | Both are proxies for age; the EEOC lists age (40 or older) among protected bases | Score current practice shown in the exercises |
| Asking about family to judge release-night availability | Not job-related, and invites a discrimination claim | State the schedule from the job description and ask every candidate if they can meet it |
Where does this scorecard fit in the hiring process?
The QA engineer phone screen places the candidate (manual tester, automation engineer or SDET) and runs the knockouts. Keep the rows consistent with the must-haves in the QA engineer job description template, run the loop from the interview guide, and score each stage on these six rows so the debrief compares like with like. If the job requires an ISTQB certificate, such as the Certified Tester Foundation Level (CTFL v4.0), check it before scoring: ISTQB's Successful Candidate Register searches by name or certificate number, but says inclusion is voluntary, so a holder may not be listed (as of November 2026); ask the candidate for the certificate and the board that issued it. Carry the weakest row into the first month of the 30-60-90 day plan for QA engineers.
If you run the developer panel or the test design session on a video call with Interview Signal, the desktop app we make, these rows can become the scorecard areas in its kit, each with a weight of 1, 2 or 3 and rubric lines on a 1–10 scale; each score carries the candidate's own quotes, checked word for word against the transcript, and an area with no surviving quote is capped at 5 and marked "Probe further" (see how the scorecard with verified quotes works). It reads only the words spoken, so a triage written on a whiteboard is scored from what the candidate says about it. Whatever you use, have each interviewer score before reading the others, then compare in a calibration session.
Questions people ask
What should a QA engineer interview scorecard measure?
Six things for most QA seats: risk-based test design, exploratory testing and defect finding, bug reporting and triage, test automation design and maintenance, CI and release judgment, and collaboration with developers and product. Weight them for the seat, and write what a 1, 3 and 5 sound like for each row before the first interview.
Should an ISTQB certification add points on a QA engineer scorecard?
No. If the job description requires an ISTQB certificate, check it as a pass/fail line before scoring, for example in ISTQB's Successful Candidate Register, where listing is voluntary. The scored rows should measure how the candidate designs tests, finds bugs and judges release risk, which a certificate does not show on its own.
How do you score a QA candidate who found fewer bugs in the exercise?
Score severity and reasoning, not the count. A candidate who found the one bug that loses money and explained why it matters has shown more than one who logged ten cosmetic issues. Write the planted bugs and their severity before the first candidate so every interviewer scores against the same list.
How is a QA engineer scorecard different from a software engineer scorecard?
A software engineer scorecard weights writing correct code and system design. A QA engineer scorecard weights deciding what to test and at which level, finding and reporting defects, keeping an automated suite trustworthy, and telling the team plainly what risk a release carries. Coding is scored inside the automation row, on test code rather than algorithms.