Interview scorecard for data scientists, with anchors for each competency
On this page
- Which competencies should a data scientist scorecard cover?
- The scorecard: weights and what a 1, 3 and 5 look like
- Copy-ready data scientist scorecard
- What should each interviewer ask, and what counts as evidence?
- Two scenarios that produce scoreable evidence
- A filled example
- How do you adjust it for the seat and the level?
- Mistakes specific to scoring data scientists
- Where does this scorecard fit in the hiring process?
- Questions people ask
A data scientist interview scorecard should score six things separately: problem framing, statistics and experiment design, modeling and data hygiene, communication and influence, code quality and reproducibility, and ownership. Each line gets a weight that adds up to 100 and a 1–5 scale whose 1, 3 and 5 are written in data science terms (a leaked feature caught or missed, a sample size fixed before the test, a result explained without dropping the caveat), so two interviewers who hear the same answer give the same score. Below is the copy-ready scorecard, the question and evidence for each line, two scenarios, a filled example and the scoring mistakes specific to data scientists.
The scale follows the 1–5 interview rating scale, where 3 is a real pass for the level and 5 is rare, and the layout follows the interview scorecard template. The loop that produces the evidence (an experiment readout review, a modeling session with planted leakage, a stakeholder session) is in the interview guide for data scientists. This page is the sheet every interviewer in that loop fills in.
Which competencies should a data scientist scorecard cover?
- Problem framing. Turns a vague request into a question the data can answer, with the decision it feeds named first.
- Statistics and experiment design. Fixes the metric, effect size and sample size before looking, and knows when a result is not a result.
- Modeling and data hygiene. Starts from a baseline, picks a metric that matches how the output is used, and catches leakage and bad joins.
- Communication and influence. Explains a result to a non-technical listener without losing the caveat, and declines a claim the data does not support.
- Code quality and reproducibility. Writes SQL and Python a teammate can rerun and trust.
- Ownership and collaboration. Follows work into the decision or into production, checks it afterward and owns what went wrong.
Score communication on clarity and accuracy, never on accent or speaking style. Tools named in the job description (Python, SQL, a cloud platform) are requirements to confirm, not lines to score.
The scorecard: weights and what a 1, 3 and 5 look like
Example weights for a mid-level product data scientist, matching the loop in the interview guide. Agree yours with the hiring manager and lock them before the first candidate (see interview scorecard weighting).
| Competency | Weight | 1 looks like | 3 looks like | 5 looks like |
|---|---|---|---|---|
| Problem framing | 20% | Starts querying or modeling without asking how the result will be used | Asks who acts on the result, when and what would change their decision, then restates the request as a measurable question | Reframes the request into a cheaper or better question (a test instead of a model, a rule instead of a forecast) and explains why before any data work |
| Statistics and experiment design | 20% | Reads a p-value as the chance the change works; no sample size or duration logic | Sets the metric, minimum effect and sample size in advance; spots early stopping or an unbalanced split | Also weighs practical against statistical significance, guardrail metrics and multiple comparisons, and says what to do when a test cannot be run |
| Modeling and data hygiene | 20% | Reports a near-perfect score with pride; no baseline; accuracy on imbalanced data | Builds a simple baseline first, uses a metric tied to the use (for example precision in the top N cases the team can act on) and checks for leakage | Finds leakage or a label defined after the fact, explains how it got in, and validates the way the model will run (time-based splits where time matters) |
| Communication and influence | 20% | A methods tour with no recommendation, or agrees to an unsupported causal claim to keep the peace | Leads with the decision, says what the analysis can and cannot show in plain words, keeps the main caveat | Changes a stakeholder's mind with evidence and offers a way to test the claim they wanted, without losing the relationship |
| Code quality and reproducibility | 10% | Code only runs in their own notebook state; joins unchecked; no way to rerun | Readable SQL and Python, row counts checked after joins, results reproduce from a clean run | Adds assertions on data assumptions, version-controls the work and leaves it so another analyst can extend it |
| Ownership and collaboration | 10% | Work ends at the notebook or the slide; cannot say whether it was used | Followed one model or analysis into use, checked it afterward and describes their own part honestly | Found their own shipped work was wrong, raised it before anyone asked and changed the process so it would be caught sooner |
A 2 shows part of the 3 behavior, or needs heavy prompting to get there; a 4 shows the 3 behavior plus part of the 5. When a line never came up, mark it "not assessed" rather than giving it a 1.
Copy-ready data scientist scorecard
Candidate: [name] Seat: [analytics / applied ML / research] Level: [ ]
Interviewer: [name] Stage: [readout review / modeling / stakeholder / HM]
Competency Weight Score (1-5 or N/A) Evidence (their words, numbers, choices)
Problem framing 20% [ ] [questions asked before starting; the decision named]
Statistics and experiment design 20% [ ] [sample size, duration, issues spotted in the readout]
Modeling and data hygiene 20% [ ] [baseline, metric chosen and why, leakage found or missed]
Communication and influence 20% [ ] [how they opened; the caveat kept; response to pushback]
Code quality and reproducibility 10% [ ] [join checks, rerun from clean state, readability]
Ownership and collaboration 10% [ ] [work followed into use; a mistake owned]
Requirements (pass/fail, not scored): [degree if required, verified] [work authorization asked the same way for all]
Weighted score: [sum of score x weight] Open questions for the next round: [ ]
Recommendation: [Strong yes / Yes / No / Strong no]
What should each interviewer ask, and what counts as evidence?
Problem framing
Ask: "Our head of sales wants a model that predicts which leads will close. What do you ask before you start?" Probe: "What if the answers made a model unnecessary?" Evidence: questions about who uses the output, how often and what they do today, and a simpler option named. A list of algorithms is not framing.
Statistics and experiment design
Ask: "We want to test a new onboarding email. Design the test." Probe: "Halfway through, the result is significant. Do we stop?" Evidence: a primary metric, a minimum effect worth detecting, sample size and duration fixed in advance, and a clear no to stopping early.
Modeling and data hygiene
Ask: "Tell me about a model whose first offline score was too good to be true." Probe: "How did you find the cause?" Evidence: a named feature or join, how it leaked, and the validation they changed. A candidate who has never seen a suspicious score has usually not looked.
Communication and influence
Ask: "Explain the last model you built to a store manager in two minutes." Probe, in role: "So the new price caused the drop, right?" Evidence: the decision first, no jargon, and a polite no to the causal claim with a way to test it. Use the stakeholder management questions for more probes.
Code quality and reproducibility
Use the query review below. Evidence: they check row counts before and after the join and explain the fix in a sentence a reviewer would accept.
Ownership and collaboration
Ask: "Tell me about analysis of yours that was wrong after people used it. How did you find out?" Evidence: monitoring or a check they set up, who they told and what changed. Blaming the data team with no action of their own is a 1 or 2.
Two scenarios that produce scoreable evidence
Metric drop diagnosis (15 minutes). "Weekly active users in the mobile app fell 9% last week (an invented figure). The vice president wants to know why by Friday. Walk me through your first hour."
What to listen for: whether they rule out measurement first (a logging change, a new app release, a late data pipeline) before guessing at user behavior; whether they segment by platform, app version, region and new versus returning users; what result would confirm or rule out each idea; and how they would report a partial answer on Friday. Scores framing, data hygiene and communication.
Query review (10 minutes). "Revenue on this dashboard is about 30% higher than the finance report (invented). Here is the query."
SELECT o.order_date, SUM(o.order_total) AS revenue, COUNT(i.item_id) AS items
FROM orders o
JOIN order_items i ON i.order_id = o.order_id
GROUP BY o.order_date;
What to listen for: that joining an order-level total to item-level rows repeats the total once per item, so revenue is overstated; a fix (sum orders before joining, or count items in a separate subquery); and a habit worth a 5: reconciling to the finance number before publishing. Scores code quality and data hygiene.
A filled example
A made-up candidate for a mid-level product data scientist seat, scored across the loop:
Problem framing: 4 (20%). In the metric drop scenario, asked whether a release had shipped last week before touching data, then proposed a segment cut to confirm it.
Statistics and experiment design: 3 (20%). Found the early stop and the unbalanced split in the readout; did not raise the fall in average order value until asked.
Modeling and data hygiene: 3 (20%). Built a logistic regression baseline first and chose precision in the top 200; missed the leaked field but noticed the score was too high and went looking.
Communication and influence: 4 (20%). Opened with "call these 200 customers first" and declined the pricing claim, offering a holdout test instead.
Code quality and reproducibility: 2 (10%). Found the join fan-out only after the interviewer suggested checking row counts.
Ownership and collaboration: 3 (10%). Described a forecast that drifted after a pricing change, the alert they added and the retraining schedule they agreed with engineering.
Weighted score: 3.3 of 5. (0.2 x 4 + 0.2 x 3 + 0.2 x 3 + 0.2 x 4 + 0.1 x 2 + 0.1 x 3.) Recommendation: Yes, with one open question for the hiring manager: how much unsupervised SQL the seat needs in the first quarter.
How do you adjust it for the seat and the level?
| Seat or level | What changes |
|---|---|
| Analytics or decision science | Move weight to statistics and experiment design and to communication (for example 25% each); modeling drops to 10%. |
| Applied machine learning | Move weight to modeling and code quality; add a line for deployment and monitoring if the seat owns models in production. |
| Research-leaning | Add a line for depth in the method area, scored on a walk-through of their own paper or project; keep communication. |
| Junior or new graduate | Use coursework, internships and personal projects as evidence; scale the scenarios down; a 3 means sound reasoning with a few prompts. |
| Senior or staff | A 3 requires raising the framing problem unprompted; weight ownership higher and add influence across teams. |
Mistakes specific to scoring data scientists
| Mistake | Why it fails | Fix |
|---|---|---|
| Scoring model sophistication | A complex model with no baseline beats a well-validated simple one on the sheet but not on the job | Score whether the method fits the decision and was validated |
| Crediting headline results | "Lifted conversion 12%" says nothing about how the test was run | Score the design the candidate can explain, not the number |
| Letting a polished presentation raise every line | The halo effect turns one strong session into six high scores | Score each line only from its own evidence, before the debrief |
| Scoring degrees and tool lists | Neither shows judgment, and both proxy for background rather than skill | Check required items pass or fail; score what they did with them |
| Asking to see an employer's code or data | Rewards candidates willing to share confidential work | Use the scenarios above; ask for the approach, not the artifacts |
Where does this scorecard fit in the hiring process?
The recruiter's screen uses a shorter rubric (see data scientist phone screen questions). In the loop, each interviewer scores only the lines their stage owns, in writing, before the debrief, so no one anchors on the loudest opinion (independent feedback before the debrief). Write the rows from the must-haves in the data scientist job description, and carry the lowest-scoring line of the person you hire into their 30-60-90 day plan.
If you interview on a video call with Interview Signal, our desktop app, you can enter these competencies as the role's scorecard areas: it scores each area from 1 to 10 during the call with the candidate's own quotes as evidence, deletes any quote not found word for word in the transcript, and caps an area with no surviving quote at 5, marked "Probe further" (see how the verified-quote scorecard works).
Questions people ask
What should be on an interview scorecard for a data scientist?
Score six things separately: problem framing, statistics and experiment design, modeling and data hygiene, communication and influence, code quality and reproducibility, and ownership. Give each a weight that adds up to 100 and write what a 1, a 3 and a 5 sound like in data science terms, such as catching a leaked feature or fixing the sample size before a test starts.
How should I weight statistics versus machine learning on a data scientist scorecard?
Weight by the seat, not by the title. An analytics or decision science seat should put more weight on statistics, experiment design and communication; an applied machine learning seat should put more on modeling, validation and code quality. Agree the weights with the hiring manager before the first interview and keep them the same for every candidate.
Should a PhD raise a data scientist's interview score?
No. If the job description requires a degree, check it as a pass or fail item and verify it with the school. The scorecard should score what the candidate showed in the interview: how they framed a problem, validated a model and explained a result. A degree is not evidence of any of those on its own.
Can I use the same scorecard for a data scientist take-home assignment?
Yes. Score the take-home on the same competency lines and anchors, then spend part of the next interview asking the candidate to walk through their choices and change one assumption live. That follow-up is where you confirm the work and the reasoning are theirs.