Data scientist phone screen questions: modeling, experiments and stakeholders
On this page
- Which data scientist seat: analytics, applied ML or research-leaning
- One project, end to end
- Statistics and experiment design
- Feature engineering and data reality
- Explaining results to non-technical stakeholders
- Certifications, portfolios and technical tests
- How data scientist resumes overstate the work
- Knockout checklist and scorecard
- Questions people ask
A data scientist phone screen should find out whether the candidate can design an analysis, choose a model with a reason, and know whether the result is actually trustworthy. Ask for one project where they went from a business question to a model to a decision someone made because of it. Candidates who did the work can walk through the data, the method they chose and why, how they validated it, and what the stakeholder did differently afterward. Overstated resumes say "built predictive models" and "drove data-driven decisions" without naming a single model, dataset or outcome.
This is not the same screen as data analyst phone screen questions, which tests SQL and dashboards rather than modeling and experiment design, or machine learning engineer screening questions, which tests production systems rather than the analysis itself. Titles blur across all three; ask what the candidate actually built and shipped before assuming the title tells you the skill set.
Which data scientist seat: analytics, applied ML or research-leaning
| Seat | What the work usually is | Facts to ask for | Common overstatement |
|---|---|---|---|
| Analytics / decision-science data scientist | Experiments, causal analysis, statistical modeling that informs a business decision | The decision it informed, the experiment design, the sample size | Correlation analysis presented as causal proof |
| Applied ML data scientist | Builds predictive or classification models that get used, sometimes deployed by engineering | The model type, the features, how it was validated, what happened after launch | A notebook model with no validation presented as production-ready work |
| Research-leaning data scientist | Longer-horizon exploration, novel methods, sometimes publication-adjacent work | What was actually applied versus explored, and by whom | Research interest relabeled as applied production experience |
| Data scientist doing analyst work | Dashboards and reporting under a data scientist title | Whether any modeling or experimentation happened at all | Title inflation with no statistics or modeling work behind it |
One project, end to end
Ask: "Walk me through one project from the business question to the decision someone made because of your work." This single question carries most of the screen.
| Question | What a strong answer sounds like | Red flags |
|---|---|---|
| What was the business question, in one sentence? | A specific decision: whether to launch a feature, which customers to target, how to price something. | "We wanted insights into the data." |
| What model or method did you use, and why that one? | Names the method (logistic regression, gradient boosting, a causal inference technique) and explains the trade-off against at least one alternative. | Names a model with no reasoning for choosing it over anything else. |
| How did you validate it? | A holdout set, cross-validation, or a proper A/B test, with the metric that mattered for the decision, not just accuracy. | "It looked right" with no validation method. |
| What happened after the result went to the stakeholder? | A specific decision or action that followed, and whether the impact was measured afterward. | The project ends at "we presented the findings," with no known outcome. |
Statistics and experiment design
Worked example: reading an A/B test claim (invented numbers)
A candidate says: "I ran an A/B test that increased conversion by 12 percent."
- Ask for the sample size and duration: 12 percent on 40,000 users per arm over three weeks is a very different claim than 12 percent on 400 users over two days.
- Ask whether the result was statistically significant, and at what threshold. A candidate who owned the test knows the p-value or confidence interval without looking it up.
- Ask about novelty effects and seasonality: did they check whether the lift held after the first week, or during a period unaffected by a holiday or promotion?
A candidate who designed the test can answer all three. A candidate repeating a result from a dashboard usually cannot.
- "How do you decide sample size before running a test?" Strong: power analysis based on expected effect size and baseline rate. Red flag: "we ran it until the numbers looked good," which is peeking, a real source of false positives.
- "Tell me about a result that looked significant but wasn't real." Strong: a specific example — multiple comparisons, a seasonal spike, a segment that was too small — and what they did about it. This question is hard to fake.
- "When would you use a causal method instead of a straightforward regression?" Strong: names a situation with confounding (users who opted in are different from users who didn't) and a method like propensity matching or a natural experiment.
Feature engineering and data reality
- "What features did you build for your model, and which ones actually mattered?" Strong: specific features, and which ones the model relied on most, with a method for checking (feature importance, ablation).
- "Tell me about a data quality problem that changed your approach." Strong: a specific issue (leakage, a mislabeled target, missing data that wasn't random) and how they caught it.
- "Did you ever have data leakage in a model? How did you find it?" Strong: describes a specific case, such as a feature that used information not available at prediction time, and how the unusually high accuracy tipped them off.
Explaining results to non-technical stakeholders
Ask the candidate to explain one of their own projects in under a minute, as if to a stakeholder with no statistics background, on the call itself.
- Strong: simplifies without losing the substance — states the decision, the finding, and the confidence level in plain language, and can go deeper if asked.
- Red flag: cannot simplify at all, or the simplified version drops a caveat that mattered (small sample, correlation not causation, a limited population).
Certifications, portfolios and technical tests
Certifications are not a strong signal for data science roles; nothing analogous to a licensing body exists for the field, and cloud vendor certificates (AWS, Azure, Google) test tooling, not modeling judgment. A portfolio or a take-home is more useful.
- "Do you have a project you can walk me through visually after this call — a notebook, a writeup, a repo?" If yes, review it before the next round rather than relying only on the verbal description.
- "What would you want to know before I gave you a take-home?" Strong candidates ask about the data, the deadline and what "good" looks like, which is itself a signal of how they scope ambiguous work.
How data scientist resumes overstate the work
| Claim | Question to test it |
|---|---|
| "Built machine learning models to drive business impact" | "Name one model, the decision it fed, and what changed because of it." |
| "Increased revenue by $2M using predictive analytics" | "How was that $2M measured, and against what baseline or control group?" |
| "Deep learning expert" | "What architecture, on what data, and did you train it or fine-tune an existing one?" |
| "Advanced statistical analysis" | "Walk me through the actual test or method, not the name of the category." |
Knockout checklist and scorecard
Must-ask on every data scientist phone screen
- Seat: analytics/decision science, applied ML, research-leaning, or analyst-in-disguise.
- One project end to end: business question, method, validation, outcome.
- Sample size and validation method behind any headline result.
- One data quality or leakage problem they personally caught.
- Ability to explain a result simply without dropping the caveat.
- Whether the role needs production ML skills this candidate does not have.
Knock out, or flag before submitting, if the candidate cannot name a model or method they personally chose and why; if every result they cite has no sample size, baseline or validation attached; if the role needs production deployment skills and the candidate has only worked in notebooks; or if they cannot simplify their own work for a non-technical listener at all.
| Area | 1 | 2 | 3 | 4 |
|---|---|---|---|---|
| Modeling depth | Buzzwords only | Names methods, thin reasoning | Chose and validated a method with reasoning | Compares trade-offs across methods fluently |
| Statistical rigor | No validation described | Basic validation | Proper test design, sample size aware | Catches false positives and confounders unprompted |
| Business impact | No decision tied to the work | A decision, unmeasured impact | A decision with measured outcome | Tracked impact after the fact, adjusted approach |
| Communication | Cannot simplify | Simplifies, drops caveats | Simplifies accurately | Adjusts depth fluently for the audience |
| Logistics fit | Deal-breaker | Two open questions | One open question | All aligned |
Write the evidence next to each score in the candidate's own words: "Ran an A/B test on the checkout flow, 38,000 users per arm, held for three weeks past the initial spike, significant at 95 percent confidence." Interview Signal attaches quotes like that to each score from the call itself, so a statistics claim in the submittal is the number the candidate actually said, not a rounded memory of it.
Questions people ask
How is a data scientist screen different from a data analyst screen?
A data analyst screen tests SQL, dashboards and turning stakeholder questions into reports. A data scientist screen tests statistical reasoning and modeling: how they designed an experiment, chose a model, and validated it, with a result that fed a real decision rather than a dashboard someone reads.
How is a data scientist screen different from a machine learning engineer screen?
A machine learning engineer screen focuses on production systems: serving, monitoring, latency, retraining pipelines. A data scientist screen focuses on the analysis and modeling that answers a business question, whether or not it ever gets deployed as a live system. Many roles blend both; ask which one this job actually needs before you screen.
Do data scientists need a graduate degree?
Often preferred but not universal. Ask about the work instead of the credential: a strong candidate can describe a model they built, why they chose it over alternatives, and how they validated it, regardless of degree.
What's the fastest way to spot a data scientist who can't explain their own work?
Ask them to explain a model they built to a non-technical stakeholder in under a minute, on the call. Someone who actually built and validated the model can simplify it without losing the substance. Someone repeating a term they read cannot.