Live demo → https://pricing-promotion-decision-engine.streamlit.app/
Stack: Python · Streamlit ·linearmodels·scipy·pandas·matplotlib·plotly
A deployed decision-support web app that turns 4.65M weekly retail scanner observations into a ranked list of price and promotion changes worth A/B testing. Demand model is IV-tested within 3% of OLS; every recommendation is paired with a powered, store-level test plan and a constraint diagnostic. Built on the Dominick's Finer Foods cereal panel.
The hero above is generated directly from the project's data artifacts — see scripts/build_readme_hero.py.
A reproducible decision-support pipeline, not a new econometric estimator. It translates classic scanner-data demand estimation, pricing optimization, and A/B test design into one product surface:
- Demand estimation — log-log fixed-effects model on 2.59M brand-size-store-week observations.
- Counterfactual simulation — model-implied units, revenue, profit at any candidate price, with optional demand / cost / competitor / inventory shocks.
- Profit optimization — vectorized search across 5,896 product-store combinations, ranked by expected weekly lift.
- Experiment design — risk-tiered, store-randomized A/B test plan with explicit power and underpowered-flag.
- Robustness — Hausman + over-ID IV move β_own by ≤3.0%; store-week FE moves it 4.5% — same sign across all four → Robust OLS.
The methodology deliberately separates formula provenance: DFF price, quantity, movement, and gross-margin fields are used for accounting transformations; the demand model is a standard reduced-form log-log fixed-effects approximation; Duan smearing is used for log-outcome retransformation; and the competitor-price index plus anchored counterfactual optimizer are project-specific implementation choices for decision support.
Current stage: Streamlit MVP and experiment-design workflow completed. The project now has cleaned panels, EDA, demand estimation, counterfactual simulation, profit optimization, A/B validation design, and an interactive local app.
- Raw data dictionary:
rawData/README.md - Cleaning notebook:
notebooks/01_data_cleaning.ipynb - EDA notebook:
notebooks/02_eda.ipynb - Demand estimation notebook:
notebooks/03_demand_estimation.ipynb - Counterfactual notebook:
notebooks/04_counterfactual.ipynb - A/B design notebook:
notebooks/05_ab_testing_design.ipynb - Cleaning diagnostics:
reports/data_cleaning_summary.md - Demand summary:
reports/demand_model_summary.md - Counterfactual summary:
reports/counterfactual_summary.md - A/B validation plan:
reports/ab_test_plan.md - Methodology:
docs/methodology.md,docs/methodology.tex - Processed panel:
data/processed/sku_store_week_panel.parquet - Brand-size panel:
data/processed/brand_size_store_week_panel.parquet - Streamlit app:
app.py
Next stages are cannibalization robustness, packaging/deployment polish, and optional causal-IV upgrades.
Retail pricing teams need to answer questions that pure sales forecasting does not answer:
- Is the current price too high or too low?
- Does a promotion increase units, revenue, or profit?
- How would quantity, revenue, and profit change under an alternative price?
- Which price and promotion status are worth testing under business constraints?
- What experiment is needed before rolling out a candidate action?
The final product lets a user select a product or brand, adjust effective price, promotion status, competitor price, unit cost, and inventory, then see predicted demand, revenue, profit, and a raise-and-test candidate action.
The Streamlit MVP supports user-defined demand, cost, competitor-price, inventory, and promo-fixed-cost scenarios layered on top of the frozen demand model; defaults are inert (BASELINE = offline notebook results), and any non-zero shock triggers live re-optimization across all eligible cells with risk-flag banners when the inputs leave a sane range. Multiplicative shocks must satisfy demand_shock > -1, cost_shock > -1, and competitor_price_shock > -1; risk flags use symmetric thresholds for demand, cost, and competitor-price shocks. See docs/methodology.md §8.1 for the overlay equations.
The project uses the Dominick's Finer Foods database from the James M. Kilts Center for Marketing at the University of Chicago Booth School of Business. The MVP focuses on the Ready-to-Eat Cereals category.
Raw files currently used:
rawData/wcer.csv: weekly UPC-store movement data for cerealsrawData/upccer.csv: cereal UPC attributesrawData/dominicks_manual.pdf: official codebook and field definitions
Core fields include UPC, STORE, WEEK, MOVE, PRICE, QTY, SALE, PROFIT, and OK. See rawData/README.md for the data dictionary, field rules, SALE code notes, PROFIT interpretation, and validation assertions.
The first cleaning pass produced:
wcer.csv: 6,602,582 rows loadedOK=1: 97.86% of raw rows- Zero
MOVErows: 28.0% - Rows retained after
OK=1, valid margin, and positive price filters: 4,654,156 - Cleaned panel covers 486 UPCs, 93 stores, 366 weeks, and 36,199 UPC-store pairs
- Median UPC-store coverage is 78 weeks, which supports SKU-store fixed effects
- Zero-movement rows are all zero-price rows; a small number of additional zero-price rows have positive movement and are excluded as invalid price records
- UPC metadata join miss rate: 0.00%
- Cleaned panel saved to
data/processed/sku_store_week_panel.parquet
Full diagnostics are in reports/data_cleaning_summary.md.
The source-of-truth panel is UPC x store x week. The baseline demand model uses brand x size_oz x store x week, with the UPC-level panel retained for robustness.
At the brand-size level, quantity is summed, dollar sales are aggregated from DFF bundle prices, and effective price is sales / quantity, not a simple average. The aggregated unit-cost proxy is quantity-weighted. The baseline promotion variable is promo_any, with promo_share retained as a diagnostic.
The project is positioned as an applied decision product rather than a structural IO replication.
- Scanner-data retail pricing: Dominick's is a classic public retail panel for shelf management and pricing research.
- Differentiated demand: BLP and Nevo's cereal work motivate the structural demand context and cereal-market substitution logic, but they are not the source of the MVP fixed-effects equation.
- Promotion behavior: Hendel and Nevo highlight that temporary sales may involve stockpiling, so a static weekly model should be interpreted carefully.
- Pricing optimization: revenue management and price optimization literature motivate combining demand, cost, inventory, and constraints in one decision problem; the MVP grid search is an implementation choice, not a new optimization method.
- Experimentation: observational price and promotion variation is not final causal evidence, so candidate actions need randomized tests or credible rollout designs.
The baseline demand model is a transparent fixed-effects model:
log(quantity) ~ log(own effective price)
+ log(competitor price)
+ promo
+ SKU-store fixed effects
+ week fixed effects
+ optional controls
The app uses the fitted model to:
- estimate own-price elasticity, cross-price elasticity, and conditional sale-code effects;
- apply smearing correction when transforming log demand predictions back to quantity;
- simulate price and promotion counterfactuals;
- optimize profit with price band, margin, inventory, and optional promotion-cost constraints;
- surface uncertainty and identification caveats;
- design an A/B test for rollout validation.
Formula provenance matters for interpretation. Effective price and dollar sales are DFF accounting transformations (PRICE/QTY and PRICE*MOVE/QTY); unit_cost is derived from the DFF gross-margin field and should be read as an average-acquisition-cost proxy, not observed marginal cost. The competitor-price index is a project-specific competitive-environment measure, not a structural substitution matrix. The anchored counterfactual demand function is an algebraic implementation of the fitted log-log model, and revenue/profit use inventory-capped sold quantity before computing dollars.
When the app reports absolute candidate profit, the promo fixed cost enters as profit = (price - cost) * sold_units - F * promo. When it reports profit lift against the observed baseline, the same fixed cost is subtracted from both candidate and baseline profits, so the fixed cost only changes incremental lift when the candidate and baseline promotion states differ.
The main fixed-effects specification uses product-store and week fixed effects. A stricter robustness check should estimate a store-week fixed-effects version to absorb local store demand shocks, campaigns, traffic, and inventory conditions; the competitor-price index may be excluded from that check when it is mechanically absorbed or unstable.
For the full demand model, counterfactual simulation logic, profit objective, optimization design, and identification caveats, see docs/methodology.md. A LaTeX version is available at docs/methodology.tex.
The project uses an anchored fixed-effects demand model plus a constrained price-promotion grid optimizer. The design intentionally trades some behavioral richness for interpretability, fast scenario simulation, and deployment readiness.
| Model / algorithm | Core idea | Best suited for | Complexity / engineering cost | How this project differs |
|---|---|---|---|---|
| Business rules / margin markup | Set prices from target margin, historical price bands, or buyer rules | Fast manual pricing guardrails | Very low | More data-driven: learns own-price, competitor-price, and promo response before ranking actions |
| Simple log-log OLS demand | Estimate average elasticity with log(Q) ~ log(P) + promo |
Transparent elasticity baseline | Low | Adds product-store and week fixed effects to reduce persistent product, store, and seasonality confounding |
| Panel fixed-effects demand | Absorb product-store and week heterogeneity while estimating price and promo effects | Scanner-data demand estimation | Medium-low | This is the offline estimation core, then extended into counterfactual profit optimization |
| This project: anchored FE + constrained grid optimizer | Estimate elasticities offline; simulate each cell from its observed baseline; search bounded price and promo candidates | Auditable price/promo actions worth A/B testing | Online complexity O(C x G x M); here 5,896 x 21 x 2 candidates, vectorized and interactive |
Adds a decision layer while staying interpretable and lightweight enough for real-time scenario recomputation |
| Tree / boosting demand forecast | Predict sales with nonlinear features using Random Forest, XGBoost, etc. | Forecast accuracy and nonlinear pattern capture | Medium; tuning and explanation cost | May forecast better, but price effects are less causal and less directly usable for counterfactual pricing |
| Time-series forecasting | Forecast each item/store path from trend and seasonality | Baseline demand planning | Medium; scales awkwardly across many sparse SKU-store series | Forecasts what will happen, but does not naturally answer "what if price changes?" |
| Multinomial / nested logit | Model consumer choice across products or nests | Market-share and substitution analysis | Medium-high | Captures substitution more explicitly, but requires stronger choice-structure assumptions and more complex estimation |
| BLP / random-coefficients demand | Estimate heterogeneous preferences and structural substitution with nonlinear GMM and simulation | Structural IO, welfare, market-power analysis | High | Richer behavioral model, but much heavier than an MVP decision-support system |
| Full cross-elasticity matrix optimization | Estimate all product-product interactions and jointly optimize prices | Multi-product portfolio pricing | High; can face combinatorial growth as products and price grids expand | This project optimizes per cell, avoiding joint-optimization blowups while surfacing cannibalization as a diagnostic |
| Bayesian hierarchical demand | Pool information across brands, stores, and SKUs with uncertainty estimates | Sparse products and uncertainty quantification | High; MCMC/VI and model monitoring | More expressive uncertainty model, but more expensive to fit and operate |
| Bandit / reinforcement learning pricing | Learn prices online through exploration and exploitation | Live dynamic pricing systems with traffic and safety controls | High; needs online experiments and guardrails | This project is more conservative: offline ranking first, then controlled A/B validation |
In short, the model is not claiming a new estimator. Its advantage is the productized workflow: a standard, interpretable fixed-effects demand model is converted into a fast, bounded, auditable optimizer that ranks price and promotion tests without requiring a full structural demand system.
pricing/
|
|-- README.md
|-- app.py
|-- requirements.txt
|
|-- docs/
| |-- app_wireframe.md
| |-- methodology.md
| |-- methodology.tex
| |-- methodology.pdf
|
|-- rawData/
| |-- README.md
| |-- dominicks_manual.pdf
| |-- wcer.csv
| |-- upccer.csv
|
|-- data/
| |-- processed/
| | |-- sku_store_week_panel.parquet
| | |-- brand_size_store_week_panel.parquet
| | |-- demand_modeling_dataset.parquet
| | |-- model_coefficients.csv
| | |-- top_recommendations.csv
| | |-- experiment_candidates.csv
|
|-- notebooks/
| |-- 01_data_cleaning.ipynb
| |-- 02_eda.ipynb
| |-- 03_demand_estimation.ipynb
| |-- 04_counterfactual.ipynb
| |-- 05_ab_testing_design.ipynb
|
|-- pages/
| |-- 1_Evidence.py # Model Evidence
| |-- 2_Simulate.py # What-If Simulator
| |-- 3_Optimize.py # Candidate Finder
| |-- 4_Validate.py # Test Planner
| |-- 5_Boundaries.py # Trust & Boundaries
| |-- 6_Upload.py # Upload & Score
|
|-- reports/
| |-- data_cleaning_summary.md
| |-- eda_summary.md
| |-- demand_model_summary.md
| |-- counterfactual_summary.md
| |-- ab_test_plan.md
| |-- figures/
|
|-- src/
| |-- data.py
| |-- features.py
| |-- validation.py
| |-- scenario.py
| |-- simulation.py
| |-- optimization.py
| |-- upload.py
To rebuild from raw data, run the cleaning notebook first:
jupyter lab notebooks/01_data_cleaning.ipynbThen run the downstream notebooks in order:
jupyter lab notebooks/02_eda.ipynb
jupyter lab notebooks/03_demand_estimation.ipynb
jupyter lab notebooks/04_counterfactual.ipynb
jupyter lab notebooks/05_ab_testing_design.ipynbTo launch the local Streamlit MVP:
streamlit run app.py- The data are historical and should be used for portfolio demonstration, not current business decisions.
- Inventory is not directly observed in DFF and will be treated as a scenario input.
PROFITsupports a useful gross-margin-derived unit-cost proxy, but it is not observed marginal cost or replacement cost.- Price, promotion, and competitor price are observational and may be endogenous.
- The MVP cross-price term is an aggregate competitor price index, not a full brand-pair substitution matrix.
promo_anyis a cleaned sale-code measure, not a randomized treatment; missing sale codes may still hide promotions.- Store-week demand shocks can still confound observational price and promotion variation; the planned store-week-FE robustness check is meant to stress-test this risk.
- Any candidate action should be validated with randomized or quasi-experimental rollout before production use.
- Add
07_cannibalization_robustness.ipynbto test same-brand cross-size substitution before treating upper-guardrail optimizer output as robust. - Add focused tests for
src/data.py,src/features.py,src/scenario.py,src/optimization.py, andsrc/upload.py. - Package the Streamlit app for sharing and write a short case-study page tying data, model, optimizer, and experiment design together.
- Optionally add
08_causal_iv.ipynbwith Hausman-style other-store price instruments and cost-shock instruments.