An Alternative to Statistical Significance for Making Decisions with Experiments

The article discusses the shift from using statistical significance (stat sig) in online experimentation to alternative approaches, such as expected utility. Statistical significance helps reduce false positives but may miss valuable gains due to increased false negatives. Expected utility focuses on making the best decision for each experiment based on available evidence. The article presents a comparison between stat sig and expected utility using an A/B test example and demonstrates that expected utility generally results in more correct decisions. It also addresses Type I and Type II errors, showing how expected utility handles them differently than statistical significance.