How to perform an A/B test

An A/B test is a method used to compare two versions of a product or service to determine which performs better. In this example, an A/B test was conducted on a mobile game called Cookie Cats to examine the effect of moving the first gate in the game from level 30 to level 40. The dataset included information about user retention for both groups (gate 30 and gate 40) at two time periods - 1 day and 7 days after a new player hits their first gate. The null hypothesis was that there is no effect on retention when the first gate is moved to level 40, while the alternative hypothesis was that there is an effect on retention when the first gate is moved to level 40. A sample ratio mismatch (SRM) test was performed to ensure the data was balanced before running the chi-square tests. The results showed no significant difference in retention after 1 day, but a statistically significant difference in retention after 7 days between the control and treatment groups. This indicates that users who play the game on a weekly basis have lower overall user retention if the first gate is moved up to level 40. The A/B test results can be used to make informed decisions about product design and improvements.