A/B Testing

When to Use A/B Testing

Good Use Cases

✅ Testing button colors or copy
✅ Comparing layout variations
✅ Evaluating new features
✅ Optimizing conversion funnels
✅ Testing pricing displays
✅ Comparing onboarding flows

Bad Use Cases

❌ Testing completely different concepts
   (Use qualitative research instead)
   
❌ Small sample sizes
   (Not enough statistical power)
   
❌ Testing broken experiences
   (Fix bugs first)
   
❌ Testing without clear hypothesis
   ("Let's just see what happens")
   
❌ Multiple simultaneous changes
   (Use multivariate testing instead)

The A/B Testing Process

┌─────────────────────────────────────────────────────┐
│  1. Identify Opportunity                            │
│     ↓                                               │
│  2. Form Hypothesis                                 │
│     ↓                                               │
│  3. Design Variants                                 │
│     ↓                                               │
│  4. Determine Sample Size                           │
│     ↓                                               │
│  5. Run Test                                        │
│     ↓                                               │
│  6. Analyze Results                                 │
│     ↓                                               │
│  7. Implement Winner / Iterate                      │
└─────────────────────────────────────────────────────┘

Step 1: Identify Opportunity

Data Sources

Quantitative:

• Funnel drop-off analysis
• Heatmaps and session recordings
• Conversion rate trends
• User flow analysis
• Performance metrics

Qualitative:

• User feedback
• Support tickets
• Usability testing
• Customer interviews

Example opportunity:

Analytics shows:
• Checkout page: 10,000 visits/day
• 40% drop-off at payment step
• 15% complete purchase

Opportunity: Reduce payment step friction
Potential impact: +2% conversion = +$X revenue

Step 2: Form Hypothesis

Structure

IF we [change],
THEN [metric] will [increase/decrease],
BECAUSE [reasoning].

Example hypotheses:

H1: Simplified form
"If we reduce the checkout form from 10 fields to 5,
then conversion rate will increase by 3%,
because users abandon long forms."

H2: Trust signals
"If we add security badges near the payment button,
then conversion rate will increase by 5%,
because users need reassurance about security."

H3: Progress indicator
"If we add a progress bar to checkout,
then completion rate will increase by 2%,
because users need to see progress."

Hypothesis Quality Check

□ Testable (can measure outcome)
□ Specific (clear what changes)
□ Impactful (worth the effort)
□ Reasonable (based on research)
□ Time-bound (clear duration)

Step 3: Design Variants

Control vs Variant

Control (A): Current version
└── Baseline for comparison

Variant (B): New version
└── One specific change

Example:
Control:  [Buy Now] - Blue button
Variant:  [Buy Now] - Green button
         (Only color changes, nothing else)

Design Principles

✅ Change ONE element at a time
   → Isolates impact
   → Clear attribution

✅ Make meaningful differences
   → Small tweaks may not register
   → Test substantial changes

✅ Keep everything else identical
   → Same traffic sources
   → Same time period
   → Same user segments

❌ Change multiple things
   → Can't tell what caused effect
   → Confounding variables

Step 4: Determine Sample Size

Statistical Requirements

Sample Size depends on:
├── Baseline conversion rate
├── Minimum detectable effect (MDE)
├── Statistical power (typically 80%)
├── Significance level (typically 95%)
└── Number of variants

Sample Size Calculation

Example:
• Current conversion: 5%
• Want to detect: 10% relative lift (5% → 5.5%)
• Power: 80%
• Significance: 95%

Result: ~30,000 visitors per variant
Total: ~60,000 visitors needed

Use online calculators:

• Optimizely Sample Size Calculator
• Evan Miller Sample Size Calculator
• VWO A/B Test Calculator

Traffic Considerations

If you have 10,000 visitors/day:
→ 60,000 visitors needed
→ 6 days to reach significance

If you have 1,000 visitors/day:
→ 60,000 visitors needed
→ 60 days to reach significance

Consider: Is this test worth running?

Step 5: Run the Test

Traffic Allocation

Option 1: 50/50 Split
Control: 50% of traffic
Variant: 50% of traffic
→ Fastest to significance
→ Highest risk

Option 2: Gradual Rollout
Control: 90% → 70% → 50%
Variant: 10% → 30% → 50%
→ Lower risk
→ Takes longer

Randomization

Critical: Users must be randomly assigned

Good randomization:
• User ID hash
• Session-based
• Consistent (same user sees same variant)

Bad randomization:
• Time-based (hour of day)
• Geographic (without reason)
• First-come-first-served

Test Duration

Minimum: 1-2 business cycles
• Account for weekly patterns
• Include weekends if relevant
• Avoid holidays/anomalies

Typical: 1-4 weeks
• Depends on traffic volume
• Run until significance reached
• Don't peek early (p-hacking)

Step 6: Analyze Results

Key Metrics

Primary Metric:

The main success measure
Example: Conversion rate, Revenue per user

Secondary Metrics:

Guardrail metrics (watch for negative impacts)
Examples: 
• Time on site
• Pages per session
• Return visits
• Support tickets

Segmentation:

Analyze by:
• Device type
• Traffic source
• New vs returning
• Geographic region
• User segment

Statistical Significance

Confidence Level: 95% (industry standard)
→ 5% chance of false positive

P-value < 0.05 = Statistically significant

Example result:
"Variant B increased conversion by 12%
with 98% confidence (p = 0.02)"

Interpreting Results

Scenario 1: Clear winner
Control:  5.0% conversion
Variant:  5.6% conversion (+12%)
Confidence: 99%
→ Implement variant

Scenario 2: No significant difference
Control:  5.0% conversion
Variant:  5.1% conversion (+2%)
Confidence: 78%
→ Inconclusive
→ Run longer or try different variant

Scenario 3: Loser
Control:  5.0% conversion
Variant:  4.5% conversion (-10%)
Confidence: 95%
→ Stick with control
→ Analyze why it failed

Common Pitfalls

1. Peeking at Results

❌ Checking daily, stopping when "significant"
❌ Multiple testing problem
❌ False positives

✅ Set sample size beforehand
✅ Run for predetermined duration
✅ Look once at the end

2. Multiple Testing Problem

❌ Running 20 tests simultaneously
   → 1 will be significant by chance (5%)
   
✅ Prioritize tests
✅ Use Bonferroni correction
✅ Focus on primary metric

3. Sample Pollution

❌ Same user sees both variants
❌ Users switch devices
❌ Clearing cookies changes variant

✅ Consistent user assignment
✅ Logged-in user tracking
✅ Cross-device consistency

4. Testing Too Long

❌ Running for months
   → Seasonality effects
   → External factors change
   → Sample pollution accumulates

✅ 1-4 week typical duration
✅ Account for business cycles
✅ Stop at significance

Tools and Platforms

A/B Testing Tools

Full-Stack Platforms:

• Optimizely
• VWO (Visual Website Optimizer)
• Google Optimize (sunsetted, alternatives: Convert, AB Tasty)
• Adobe Target

Developer-Focused:

• GrowthBook
• Statsig
• LaunchDarkly
• Unleash

Mobile:

• Firebase Remote Config
• Split.io
• Apptimize

Analytics Integration

Essential integrations:
• Google Analytics / Amplitude / Mixpanel
• Heatmaps (Hotjar, FullStory)
• Session recordings
• Error tracking
• Revenue tracking

Advanced Techniques

Sequential Testing

Check results multiple times with adjustments:
• Group Sequential Design
• Always Valid P-values
• Bayesian updating

Benefit: Can stop early if effect is large
Risk: Requires statistical expertise

Multi-Armed Bandit

Adaptive testing:
• Start with 50/50 split
• Gradually shift traffic to winning variant
• Balance exploration vs exploitation

Best for: Continuous optimization
Not for: One-time decisions

Bayesian A/B Testing

Different statistical approach:
• Provides probability distribution
• Easier to interpret
• Allows for priors

Result: "95% probability B is better than A"
vs
"95% confidence B is different from A"

Documenting Tests

Test Record Template

Test ID: EXP-2024-001
Date: Jan 15 - Feb 1, 2024
Owner: Jane Smith

Hypothesis:
[Clear statement]

Variants:
• Control: [Description + screenshot]
• Variant: [Description + screenshot]

Results:
• Primary metric: +X% (p=0.0X)
• Secondary metrics: [Impact summary]
• Segments: [Any differences]

Decision: [Implement/Iterate/Discard]
Learnings: [Key insights]
Next Steps: [Follow-up tests]

Building Test Library

Maintain repository of:
• All past tests
• Results and learnings
• Successful patterns
• Failed hypotheses

Use for:
• Avoiding repeated mistakes
• Informing new hypotheses
• Onboarding new team members
• Building organizational knowledge