PracHub
QuestionsPremiumLearningGuidesInterview PrepNEWCoaches
|Home/Analytics & Experimentation/Anthropic

How do you design an A/B experiment?

Last updated: Mar 29, 2026

Quick Overview

This question evaluates competency in online experiment design, including hypothesis framing, metric selection and guardrails, unit of randomization and interference control, power/sample-size reasoning, and identification and mitigation of common biases.

  • hard
  • Anthropic
  • Analytics & Experimentation
  • Software Engineer

How do you design an A/B experiment?

Company: Anthropic

Role: Software Engineer

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Onsite

You have an interview on your agenda titled **“Experiment Design.”** You are asked to design an online experiment (A/B test) for a product change. Describe, step-by-step: - The **goal** and **hypothesis**. - Primary/secondary **metrics** and guardrails. - **Unit of randomization** and how to avoid interference. - **Sample size / power** considerations (what inputs you need). - How you would handle **biases** (selection effects, novelty, seasonality, logging issues). - Decision rules, segmentation, and how you would interpret ambiguous results.

Quick Answer: This question evaluates competency in online experiment design, including hypothesis framing, metric selection and guardrails, unit of randomization and interference control, power/sample-size reasoning, and identification and mitigation of common biases.

Related Interview Questions

  • Design a profiling plan for kernels - Anthropic (hard)
Anthropic logo
Anthropic
Mar 1, 2026, 12:00 AM
Software Engineer
Onsite
Analytics & Experimentation
15
0
Loading...

You have an interview on your agenda titled “Experiment Design.”

You are asked to design an online experiment (A/B test) for a product change.

Describe, step-by-step:

  • The goal and hypothesis .
  • Primary/secondary metrics and guardrails.
  • Unit of randomization and how to avoid interference.
  • Sample size / power considerations (what inputs you need).
  • How you would handle biases (selection effects, novelty, seasonality, logging issues).
  • Decision rules, segmentation, and how you would interpret ambiguous results.

Solution

Show

Comments (0)

Sign in to leave a comment

Loading comments...

Browse More Questions

More Analytics & Experimentation•More Anthropic•More Software Engineer•Anthropic Software Engineer•Anthropic Analytics & Experimentation•Software Engineer Analytics & Experimentation
PracHub

Master your tech interviews with 7,500+ real questions from top companies.

Product

  • Questions
  • Learning Tracks
  • Interview Guides
  • Resources
  • Premium
  • For Universities
  • Student Access

Browse

  • By Company
  • By Role
  • By Category
  • Topic Hubs
  • SQL Questions
  • Compare Platforms
  • Discord Community

Support

  • support@prachub.com
  • (916) 541-4762

Legal

  • Privacy Policy
  • Terms of Service
  • About Us

© 2026 PracHub. All rights reserved.