Predictive Analytics – Project 5: A/B Test a New Menu Launch

Topic: A/B Test | Tool: Alteryx

Project Overview:

The coffee restaurant would like to conduct a market test with a new menu and needs to figure whether the new menu can drive enough sales to offset the cost of marketing the new menu.

To minimize the risk, the management team decides to test the changes in two cities with new television advertising. Denver and Chicago were chosen to participate in this test because the stores in these two cities perform similarly to all stores across the entire chain of stores; performance in these two markets would be a good proxy to predict how well the updated menu performs.

You’ve been asked to analyze the results of the experiment to determine whether the menu changes should be applied to all store. The predicted impact to profitability should be enough to justify the increased marketing budget: at least 18% increase in profit growth compared to the comparative period while compared to the control stores; otherwise known as incremental lift.

Step 1: Plan Your Analysis

What is the performance metric you’ll use to evaluate the results of your test?

Gross margin is the performance metric used to evaluate the results of the test. The predicted impact to profitability should be enough to justify the increased marketing budget: at least 18% increase in profit growth compared to the comparative period while compared to the control stores; otherwise known as incremental lift. In the data, profit is represented in the gross_margin variable.

What is the test period?

The test period is 4/29/2016 – 7/21/2016. The test ran for a period of 12 weeks (4/29/2016 – 7/21/201) where five stores in each of the test markets offered the updated menu along with television advertising.

At what level (day, week, month, etc.) should the data be aggregated?

The data should be aggregated at the week level.

Step 2: Clean Up Your Data

In this step, you should prepare the data for steps 3 and 4. You should aggregate the transaction data to the appropriate level and filter on the appropriate data ranges. You can assume that there is no missing, incomplete, duplicate, or dirty data. You’re ready to move on to the next step when you have weekly transaction data for all stores.

Use these three raw data files to create three files used for A/B Analysis.These files are:

Weekly store traffic data for A/B Trend Tool: Produces our seasonality and trends indices to help us match our treatment and control stores
Store list data for A/B Controls tool: Produces which control stores to match with our treatment stores along with results from the A/B Trends Tool
Store sales analysis data for A/B Analysis tool: Produces the final results

Below please find the workflow in Alteryx for data preparation:

Step 3: Match Treatment and Control Units

In this step, you should create the trend and seasonality variables, and use them along with you other control variable(s) to match two control units to each treatment unit. Note: Calculate the number of transactions per store per week to calculate trend and seasonality.

Apart from trend and seasonality…

What control variables should be considered? Note: Only consider variables in the RoundRoastersStore file.

AvgMonthSales and Sq_Ft should be considered as control variable apart from trend and seasonality.

What is the correlation between your each potential control variable and your performance metric?

I created a workflow using Pearson Correlation Analysis tool to look at the correlation between the appropriate numeric variables in the round roasters stores file AvgMonthSales and Sq_Ft with the performance metric gross margin. Below please find the workflow in Alteryx:

What control variables will you use to match treatment and control stores?

AvgMonthSales is the control variable that I will use to match treatment and control stores. Based on the Pearson Correlation Analysis report below, AvgMonthSales is statistically significant because the p-value < 0.05.

Please fill out the table below with your treatment and control stores pairs:

Treatment Store	Control Store 1	Control Store 2
1664	7162	8112
1675	1580	1807
1696	1964	1863
1700	2014	1630
1712	8162	7434
2288	9081	2568
2293	12219	9524
2301	3102	9238
2322	2409	3235
2341	12536	2383

For reference, below please find the workflow for A/B Trends and A/B Control:

Step 4: Analysis and Writeup

What is your recommendation – Should the company roll out the updated menu to all stores?

The company should roll out the updated menu to all stores. Based on the project details, the predicted impact to profitability should be enough to justify the increased marketing budget: at least 18% increase in profit growth compared to the comparative period while compared to the control stores. If we look at the result of average lift (see details in the next two sections) for each region and overall, they are all higher than 18%. Therefore, the company should roll out the updated menu.

What is the lift from the new menu for West and Central regions (include statistical significance)?

West region: the Average Lift is 37.9% and the Significance Level is 99.5%. See the A/B Test analysis model and report below: