Marketing A/B Test Analysis

Analyzing a marketing experiment with 588,000+ users to determine whether advertisements drive product conversions, using statistical hypothesis testing and segmented analysis.

Key Findings

• Ads increase conversion by 43% relative to control (2.55% vs 1.79%, p < 0.001), confirmed by chi-squared, z-test, and bootstrap methods
• Optimal ad frequency: 51-200 exposures — below 50 shows no significant effect, above 200 shows saturation
• Best timing: Monday-Wednesday afternoons — statistically significant after Bonferroni correction
• Simpson's paradox detected — low-exposure segments show reversed effects, highlighting the importance of segmented analysis

Methodology

Analysis	Method	Purpose
A/B Test	Chi-squared, Z-test, Bootstrap CI	Confirm ads vs PSA effect
Effect Size	Cohen's h	Quantify practical significance
Power Analysis	Statsmodels	Validate experiment adequacy
Dose-Response	Binned conversion + Bonferroni	Find optimal ad frequency
Timing Analysis	Segment-level hypothesis tests	Identify best days/hours
Paradox Check	Simpson's paradox detection	Validate aggregate results

Visualizations

Conversion Rates by Group

Dose-Response Curve

Conversion Heatmap: Day × Time

Note: All visualizations are saved to outputs/figures/ in high resolution (300 DPI).

Tech Stack

• Analysis: Python, Pandas, NumPy, SciPy, Statsmodels
• Visualization: Matplotlib, Seaborn
• Dashboard: Streamlit
• Statistical Methods: Chi-squared test, Two-proportion Z-test, Bootstrap CI, Cohen's h, Bonferroni correction, Power analysis

Project Structure

marketing-ab-test-analysis/
├── data/
│   └── marketing_AB.csv          # Dataset (588K rows, 7 columns)
├── notebooks/
│   └── analysis.ipynb            # Complete analysis report
├── src/
│   ├── __init__.py               # Package initialization
│   ├── data_loader.py            # Data loading and cleaning
│   ├── eda.py                    # Exploratory data analysis visualizations
│   ├── statistical_tests.py      # Hypothesis testing functions
│   ├── power_analysis.py         # Power analysis functions
│   └── segmentation.py           # Segmented analysis functions
├── app/
│   └── streamlit_app.py          # Interactive Streamlit dashboard
├── outputs/
│   └── figures/                  # Saved visualization figures
├── requirements.txt              # Python dependencies
├── .gitignore                    # Git ignore rules
└── README.md                     # This file

Quick Start

# Clone the repo
git clone https://github.com/Aditya292248/marketing-ab-test-analysis.git
cd marketing-ab-test-analysis

# Install dependencies
pip install -r requirements.txt

# Run the notebook
jupyter notebook notebooks/analysis.ipynb

# Launch the dashboard
streamlit run app/streamlit_app.py

Dataset

Marketing A/B Testing by Favio Vazquez on Kaggle.

• 588,101 users split into treatment (ads, 96%) and control (PSA, 4%)
• Binary outcome: converted (purchased) or not
• Features: total ads seen, peak day, peak hour

Author

Aditya Bisht — Computer Science, Northeastern University
LinkedIn | GitHub

License

This project is for educational and analysis purposes. Dataset attribution: Favio Vazquez (Kaggle).