This self-contained guide introduces two pillars of data science, probability theory and statistics, side by side, illuminating the connections between probabilistic concepts and the statistical techniques they employ, such as the relationship between nonparametric and parametric models and random variables. Other topics covered include hypothesis testing, principal component analysis, correlation, and regression. Examples throughout the book draw from real-world datasets, quickly demonstrating concepts in practice and confronting readers with fundamental challenges in data science, such as overfitting, the curse of dimensionality, and causal inference. Code in Python reproducing these examples is available on the book's website, along with videos, slides, and solutions to exercises. This accessible book is ideal for undergraduate and graduate students, data science practitioners, and others interested in the theoretical concepts underlying data science methods.