I vaguely remember ever learned about Bayes’ Theorem in college. Until I read this book I never thought there are so many applications of Bayes’ Theorem in our daily life. The drug companies uses it to get FDA approvals on drugs. The food companies applies the theorem to “prove” certain foods are good for us. In a way, the theorem allows us to change our perception of certain events’ probability of occurring. I can imagine more applications than the ones he outlines in the book. By the way, the pdf format of this book can be freely downloaded here, thanks to the author’s generosity.

Because of this book and his Youtube videos, I got interested in re-learning Python as a practical/useful programming language. It opened my eyes to the wonderful world of Python and its ecosystem.

The book gets pretty technical fast after the first chapter which piqued my interest with his Cookie, M&M, and Monte Hall problems. At the end I don’t know if it’s really necessary to know all the various applications in prediction and simulation, but it could come in handy for people who work in the statistical field.

The key thing to remember is that Bayes Theorem gives a way to update or arrive at the posterior probability of a prior/known hypothesis, H, after learning some new piece of data. The equation is simple: P(H|D)=P(H)*P(D|H)/P(D)
As it turn out, the most difficult part is P(D) or the normalizing factor, which is the probability of seeing the data under any hypotheses at all.

Overall, all the concepts presented by the author make sense to me. I’m not sure I can actually implement and devise the models when the actual problem arrives. But at least I can recognize the Bayesian problem when I see one and know where to find help.

Chapter Summary:
1. Bayes’s Theorem was derived easily and discussed. He introduced the probability calculator of getting a heart attack.

2. Computational Statistics: The author provides a set of Python codes/tools to calculate the result quickly. I had to re-take a Python refresher course before getting any further. But the tools are powerful and help me understand the subject better. All the codes and downloadable from thinkbayes.com website. It took me a while to figure out Monte Hall’s problem – one of those paradox that’s hard to sink in.

3. Estimation: This chapter estimates the posterior probability upon rolling of a multi-face die (4-/6-/8-/12-/20-sides die) and successive rolling of dice (more data points changes the probability distribution), and probability of the number of locomotive given the observation of one locomotive number. This is when you really need to have the computer do the work for you. The intuitive part is that the more information you have, the better your estimation is going to be, as some data points would be ruled out. For example, finding 6 eliminates the 4-sided die.

4. More Estimation: In this chapter, the author covers the “biased” Euro coin problem. The interesting phenomenon is that the prior hypothesis makes little difference (provided you don’t rule things out with 0 probability), the posterior distribution will likely converge with more data points.

5. Odds and Addends: The author covers the “odds” form of the Bayesian Theorem, o(A|D)=o(A)*P(D/A)/P(D|B), which is probably easier to understand for most people. The Addends part of the chapter goes into getting the density and distribution functions.

6. Decision Analysis: Given a prior distribution function, what’s the best decision to make given some data. The author presented a Price Is Right scenario and solve the best price to bid for a show case showdown based the past data and best guess to adjust the bid. The PDF, Probability Density Function, is introduced here. Also the use of KDE (Kernel Density Estimation) is used to smooth PDF that fits the data. This is an interesting application of Bayes Theorem. The math is complicated and couldn’t be done without a computer. I guess it would be good to bring your computer to the Price Is Right game.

7. Prediction: Here the author presented a better way to predict the outcome of a playoff game score based on past data between two teams (Boston Bruin vs. Vancouver Canucks.) This should be no difference from how the gambling industry computes the odds before a big game. Just imagine all the gambling earning you could’ve made by mastering this chapter! I’m sure someone has applied the same idea/theory to the financial market like stocks, bonds and futures.

8. Observer Bias: This is another angle of predicting the outcome (like the wait time for the next train at a given time) taking into account of the observer bias.

9. Two Dimensions: Using the paintball example, the author applies the Bayesian framework to the two-dimensional problem. In addition, the joint distribution , marginal distribution, and conditional distribution. And you don’t think one dimensional Bayesian framework is confusing enough…

10. Approximate Bayesian Computation (ABC): When the likelihood of any particular dataset is 1) very small, 2) expensive to compute, 3) not really what we want. We don’t care of about the likelihood of seeing the exact dataset we saw but any dataset like that.

11. Hypothesis Testing: The Bayes Factor, the ratio of likelihood of a new scenario to that of the baseline, can be used to test the likelihood of a particular hypothesis, e.g. fairness/cheat of an Euro coin. Bayes factor of 1~3 is barely worth mentioning, 3~10 is substantial, 10~30 strong, 30~100 very strong, >100 decisive.

12. Evidence: Test for the strength of evidence. “How strong is the evidence that Alice is better than Bob, given their SAT score?”

13. Simulation: Simulate the tumor growth rate based on prior growth rate and the data points of current tumor size, age and etc.

14. A Hierarchical Model: reflects the structure of the system, with causes at the top and effects at the bottom. Instead of solving the “forward” problem, we can reverse engineer the distribution of the parameters given the data. The Gaiger counter problem demonstrates the connection between causation and hierarchical modeling.

15. Dealing with Dimensions: This last chapter combines all of the lessons so far and applies to the “Belly Button Bacteria” prediction and simulation. This is a very difficult chapter to understand and probably requires the full understanding of the previous lessons.