Book contents
- Trustworthy Online Controlled Experiments
- Reviews
- Trustworthy Online Controlled Experiments
- Copyright page
- Contents
- Preface
- Acknowledgments
- Part I Introductory Topics for Everyone
- Part II Selected Topics for Everyone
- Part III Complementary and Alternative Techniques to Controlled Experiments
- Part IV Advanced Topics for Building an Experimentation Platform
- Part V Advanced Topics for Analyzing Experiments
- 17 The Statistics behind Online Controlled Experiments
- 18 Variance Estimation and Improved Sensitivity: Pitfalls and Solutions
- 19 The A/A Test
- 20 Triggering for Improved Sensitivity
- 21 Sample Ratio Mismatch and Other Trust-Related Guardrail Metrics
- 22 Leakage and Interference between Variants
- 23 Measuring Long-Term Treatment Effects
- References
- Index
19 - The A/A Test
from Part V - Advanced Topics for Analyzing Experiments
Published online by Cambridge University Press: 13 March 2020
- Trustworthy Online Controlled Experiments
- Reviews
- Trustworthy Online Controlled Experiments
- Copyright page
- Contents
- Preface
- Acknowledgments
- Part I Introductory Topics for Everyone
- Part II Selected Topics for Everyone
- Part III Complementary and Alternative Techniques to Controlled Experiments
- Part IV Advanced Topics for Building an Experimentation Platform
- Part V Advanced Topics for Analyzing Experiments
- 17 The Statistics behind Online Controlled Experiments
- 18 Variance Estimation and Improved Sensitivity: Pitfalls and Solutions
- 19 The A/A Test
- 20 Triggering for Improved Sensitivity
- 21 Sample Ratio Mismatch and Other Trust-Related Guardrail Metrics
- 22 Leakage and Interference between Variants
- 23 Measuring Long-Term Treatment Effects
- References
- Index
Summary
Why you care: Running A/A tests is a critical part of establishing trust in an experimentation platform. The idea is so useful because the tests fail many times in practice, which leads to re-evaluating assumptions and identifying bugs.
- Type
- Chapter
- Information
- Trustworthy Online Controlled ExperimentsA Practical Guide to A/B Testing, pp. 200 - 208Publisher: Cambridge University PressPrint publication year: 2020