Category Archives: Engineering

Book Review: “Think Bayes” by Allen Downey

I vaguely remember ever learned about Bayes’ Theorem in college. Until I read this book I never thought there are so many applications of Bayes’ Theorem in our daily life. The drug companies uses it to get FDA approvals on drugs. The food companies applies the theorem to “prove” certain foods are good for us. In a way, the theorem allows us to change our perception of certain events’ probability of occurring. I can imagine more applications than the ones he outlines in the book. By the way, the pdf format of this book can be freely downloaded here, thanks to the author’s generosity.

Because of this book and his Youtube videos, I got interested in re-learning Python as a practical/useful programming language. It opened my eyes to the wonderful world of Python and its ecosystem.

The book gets pretty technical fast after the first chapter which piqued my interest with his Cookie, M&M, and Monte Hall problems. At the end I don’t know if it’s really necessary to know all the various applications in prediction and simulation, but it could come in handy for people who work in the statistical field.

The key thing to remember is that Bayes Theorem gives a way to update or arrive at the posterior probability of a prior/known hypothesis, H, after learning some new piece of data. The equation is simple: P(H|D)=P(H)*P(D|H)/P(D)
As it turn out, the most difficult part is P(D) or the normalizing factor, which is the probability of seeing the data under any hypotheses at all.

Overall, all the concepts presented by the author make sense to me. I’m not sure I can actually implement and devise the models when the actual problem arrives. But at least I can recognize the Bayesian problem when I see one and know where to find help.

Chapter Summary:
1. Bayes’s Theorem was derived easily and discussed. He introduced the probability calculator of getting a heart attack.

2. Computational Statistics: The author provides a set of Python codes/tools to calculate the result quickly. I had to re-take a Python refresher course before getting any further. But the tools are powerful and help me understand the subject better. All the codes and downloadable from thinkbayes.com website. It took me a while to figure out Monte Hall’s problem – one of those paradox that’s hard to sink in.

3. Estimation: This chapter estimates the posterior probability upon rolling of a multi-face die (4-/6-/8-/12-/20-sides die) and successive rolling of dice (more data points changes the probability distribution), and probability of the number of locomotive given the observation of one locomotive number. This is when you really need to have the computer do the work for you. The intuitive part is that the more information you have, the better your estimation is going to be, as some data points would be ruled out. For example, finding 6 eliminates the 4-sided die.

4. More Estimation: In this chapter, the author covers the “biased” Euro coin problem. The interesting phenomenon is that the prior hypothesis makes little difference (provided you don’t rule things out with 0 probability), the posterior distribution will likely converge with more data points.

5. Odds and Addends: The author covers the “odds” form of the Bayesian Theorem, o(A|D)=o(A)*P(D/A)/P(D|B), which is probably easier to understand for most people. The Addends part of the chapter goes into getting the density and distribution functions.

6. Decision Analysis: Given a prior distribution function, what’s the best decision to make given some data. The author presented a Price Is Right scenario and solve the best price to bid for a show case showdown based the past data and best guess to adjust the bid. The PDF, Probability Density Function, is introduced here. Also the use of KDE (Kernel Density Estimation) is used to smooth PDF that fits the data. This is an interesting application of Bayes Theorem. The math is complicated and couldn’t be done without a computer. I guess it would be good to bring your computer to the Price Is Right game.

7. Prediction: Here the author presented a better way to predict the outcome of a playoff game score based on past data between two teams (Boston Bruin vs. Vancouver Canucks.) This should be no difference from how the gambling industry computes the odds before a big game. Just imagine all the gambling earning you could’ve made by mastering this chapter! I’m sure someone has applied the same idea/theory to the financial market like stocks, bonds and futures.

8. Observer Bias: This is another angle of predicting the outcome (like the wait time for the next train at a given time) taking into account of the observer bias.

9. Two Dimensions: Using the paintball example, the author applies the Bayesian framework to the two-dimensional problem. In addition, the joint distribution , marginal distribution, and conditional distribution. And you don’t think one dimensional Bayesian framework is confusing enough…

10. Approximate Bayesian Computation (ABC): When the likelihood of any particular dataset is 1) very small, 2) expensive to compute, 3) not really what we want. We don’t care of about the likelihood of seeing the exact dataset we saw but any dataset like that.

11. Hypothesis Testing: The Bayes Factor, the ratio of likelihood of a new scenario to that of the baseline, can be used to test the likelihood of a particular hypothesis, e.g. fairness/cheat of an Euro coin. Bayes factor of 1~3 is barely worth mentioning, 3~10 is substantial, 10~30 strong, 30~100 very strong, >100 decisive.

12. Evidence: Test for the strength of evidence. “How strong is the evidence that Alice is better than Bob, given their SAT score?”

13. Simulation: Simulate the tumor growth rate based on prior growth rate and the data points of current tumor size, age and etc.

14. A Hierarchical Model: reflects the structure of the system, with causes at the top and effects at the bottom. Instead of solving the “forward” problem, we can reverse engineer the distribution of the parameters given the data. The Gaiger counter problem demonstrates the connection between causation and hierarchical modeling.

15. Dealing with Dimensions: This last chapter combines all of the lessons so far and applies to the “Belly Button Bacteria” prediction and simulation. This is a very difficult chapter to understand and probably requires the full understanding of the previous lessons.


Book Review: “Think Python” by Allen Downey

It’s been a while since I used Python last. See my review on another Python book: “Invent Your Own Computer Game with Python” Unfortunately, like most spoken languages, you’d lose the ability to use a computer language if you don’t use it often. And I did forget most of it. This time, inspired by “Think Bayes,” my original intention to learn about Bayesian Statistics (more on this in future blogs), I decided to learn Python using Allen Downey’s book – Think Python.

Thanks to Downey’s teacher style, this time I think it’s going to stick with me. Also it helps to have the Enthought Canopy Express – a free tool with an integrated IDE to practice Python along the way.

Due to my previous programming experience with C, this time it took me just a week to finish the book during my “leisure” time. Also, having the free e-book (Thanks to Allen Downey’s generosity) side by side with Canopy makes a good hands-on experience. All the examples and downloadable directly from thinkpython.com.

It’s very helpful to do the exercises assigned to enhance the learning experience. Nothing beats actually coming up with the solutions on your own, then check against the downloaded solutions. Learning by doing it. That’s the ticket to retained learning. Also I refreshed my understanding of the object programming and learned a few things about GUI programming with Python and Tkinter.

“Think Python” accomplishes the goal of teaching the users how to program in Python – an excellent book and resource. Now moving on to “Think Bayes” next!


Book Review: “What is a p-value anyway? 34 Stories to Help You Actually Understand Statistics” by Andrew J. Vickers

Review on Video:

When I was studying engineering, Statistics was not a required subject. It wasn’t until I started working that I appreciated the power of statistics. In this imprecise world, many things can be explained only in statistic terms like confidence level, insufficient sample size and etc. I got a full lesson of statistics as part of the my MBA degree curriculum. Ever after taking many more statistics-related class and going through a full Six Sigma Green Belt training, there are a few things about statistics that are still hard to grasp. It’s not just p-value that confused people, there are simply too many pitfalls when novices or even experts apply statistics to real life problems.

The author organizes 34 stories across 34 chapters. Since the author works in the medical field, he mentioned quite a few tidbits about how drugs were clinical tried. It’s a good book for beginners as well as people who used statistics regularly to watch out for its pitfalls. You might get a different perspective about statistics. I did.

My key takeaways – some refresher and something new:
1. Many things in life doesn’t follow normal distribution, especially the ones involve physical ability (pregnancy duration, body BMI). Sometimes log scale fits better.
2. Two sorts of variation: observable natural variations (reproducibility), variation of study results (repeatibility).
3. Statistical ties, e.g. in election poll, means that the confidence interval includes/overlaps with no difference.(Chap. 12)
4. P-values test hypotheses. (Chap. 13)
5. Statistics are mainly used for inference (test hypotheses) or prediction (extrapolation, interpolation).
6. Null hypotheses is a statement suggesting that nothing interesting is going on (status quo) that there is no difference between that observed data and what was expected or no difference between two groups. The P-value is the probability that the data would be at least was extreme as those observed if the null hypotheses were true.
6. T-test vs. Wilcoxon test (new to me). If the data is very skewed, use Wilcoxon test whose data must be converted to ranks first. (Chap. 16)
7. Precision (width of confidence interval) = variation/ sqrt(sample size). To reduce the confidence interval (enhance precision) by half, you’d need 4 times of the sample size – very expensive. To get the sample size for a specific test = (noise or variation / signal or confidence interval )^2.
8. “Adjust the results” can be applied to multi-variable regression to help with confounding (confusing). (Chap. 19).
9. Sensitivity is the probability of a positive diagnostic test given that you have the disease (true positive). Specificity is the probability that a negative diagnostic test given that you don’t have the disease (true negative). The most worrisome situations are when the test comes back positive if they indeed have the disease (positive predictive value) or when the test comes back negative and the patient is truly free of disease is the negative predictive value. (Chap. 20)
10. Don’t accept the null hypothesis. Instead say “we could not show a difference.” Don’t use a p-value of 0, say “P < 0.001" instead. 11. Some test methods, e.g. chi-squired and ANOVA, only provides P-value - no estimates. Correlation provides estimates but no inferences. 12. One common error is to calculate the probability of something that has already happened. Then come into conclusions about what caused it based on whether that probability is how or low. E.g. calculation of the odd OJ killed his wife. Instead, the question to ask is "if a woman has been murdered and has been previous beaten by her husband, what is the chance of he was the murderer." 13. Conditional probability depends on both the probability before the information was obtained (prior probability of a heart disease) and the value of he information (such the accuracy of the heart test). 14. The more statistical tests you conduct, the greater the chance that one will come up statistically significant, even if the null hypothesis is true. 15. A smaller study has a good chance of failing to reject the null hypothesis, even if it's false. Subgroup analysis increases both the rise of falsely rejecting the null hypothesis when it's true and falsely failing to reject the null hypothesis when it's false. 16. P-values measure strength of evidence, not size of an effect.
17. Don’t compare p-values.
18. Many statistical errors occur because of starting the clock at the wrong time.
19. Lead time bias. If you find a way to find the problem earlier, then the time between the problem and the end result will be longer.
20. Statistics is used to help scientists analyze data, but is itself a science.
21. Statistics should be about linking math to science: a. think through the science and develop statistical hypotheses in the light of specific question. b. interpret the results of the analysis in terms of their implications for those questions.
22. Statistics is about people even if you can’t see the tears.

Simulink World Tour

Last week, I had an opportunity to participate in the Simulink World Tour, hosted by Mathworks. I was attracted by the agenda of showing all different applications of Simulink. Personally, I have been fascinated by the easy and intuitive way to build a system and solve for the solution. That’s the way mathematic should be – to help people visualize and construct the problem. The actual mechanics of solving for the mathematic solutions is less than interesting, at least for me.

In the morning session, they showed off a bunch of stateflow, design verifiers modules, VHDL code generation, which are good for logic verification. They also added PolySpace to check for code correctness.

What stood out for me are the embedded Matlab C code, which are great to speed up execution speed once it’s compiled. The image processing demo in the video surveillance application was quite interesting. I didn’t know the execution speed could be fast enough to do that. They also showed a hybrid car’s electro-mechanical system built with simulink and how the gas mileage fluctuated when the gas peddle is floored. This demonstrated how the system modules can be built and simulated when the interface behavior are properly modeled.

During the buffet lunch, I observed that most of the participants came from the defense industry. They probably deal with system simulations a lot due to the analog/RF nature and the interactions between electrical and mechanical systems. I can see how powerful simulink can be if put to good use. I would like to play with simulink but haven’t seen a good fit for my line of work, where things tend to be mostly digital. The analog-digital interface has been largely shielded by the chipsets. I’m still looking…

Intel Developer Forum (IDF) and Moore

On Tuesday, 9/18/07, I attended Intel Developer Forum held at the large Moscone Center West in the beautiful San Francisco. IDF is celebrating its 10-year anniversary this year and yet I have never attended one before, perhaps due to my association with Sun. Intel is a relatively new partner to Sun and we’re ramping up the learning curve on Intel’s offerings so we can design server products around it.

I came away very impressed with Intel’s speed in execution. Looks like they have all the ducks lined up and firing on all cylinders: Penryn, Nehalem, Tolapai, and etc. The attendance was very good – around a thousand or so. The topics that stood out for me:

1. All the 3-week-old wafers that are showing off Nehalem, Tolapai, and etc. Looks like they’re working just fine. Excellent execution, Intel. Being paranoid and having a good competitor like AMD really helps to spear things along. 🙂

2. USB3: 10x the speed of USB2. It has optical and copper phy layer. This could potentially put some of the PCEe devices out of business due to the parity in performance. Of course, we may see an explosion of I/O devices that normally resides within the box. I can just picture HDTV, HD camera, high-performance storage array, and etc.

3. Mobility: WiMAX and etc. We may well be cutting out all the cables soon on PC’s.

4. I didn’t get the chance to see the show cases. I heard lots of good product offerings though.

The highlight of the day is the appearance of Gordon Moore, the originator of “Moore’s Law.” By his own account, Moore’s Law may soon hit a wall in 10 or 15 years when we may hit the physical limits of material properties. When asked about making tough decisions, he said, “Tough decisions are normally very close. You may as well toss a coin.” Dr. Moore suggested that life science may be the way to go for today’s engineers. Being in a new industry has its reward as in the in case of Gordon Moore himself. He projects that computer still needs to be able to interact with human as the next wave of computer innovation.

I felt a sense of excitement as in the early days of the computer shows. Intel is leading the effort to re-shape the computer industry. I think I’ll go back again next year.

Why do we choke?

I have been the designated tutor of my 5-year-old daughter’s piano lessons at home (my wife does the driving to/from the piano teacher). One thing I have noticed is that there would be times when she really aced one part of the song consistently then afterward she would choke or fumble hopelessly on that particular part some days later. This is not unusual for myself, and most people or even athletes (remember the disappointment we felt when we saw the ice skaters fell miserably on a simple routine?). We sometimes make mistakes where we or others least expected that we would make. Why is that? I’ll defer the scientific explanation to the brain expert. My theory is that we often choke where we haven’t experienced failures yet or we haven’t figured out how to fail. And failures are our enforcement for continuous success.

Drawing a similarity to why a hardware system would sometimes fail where it’s least expected to fail – like I2C (the “slow” system management bus) that does the housekeeping within the system, or simple power sequencing circuits involving power supplies. And the simple answer is often that we haven’t figured out how to make it fail or test to fail – whether intentionally or unintentionally.

As part of my job at Sun Microsystems – System Validation or hardware quality assurance of sorts, we need to harden/toughen our hardware and correct/catch all of the hardware bugs before we release our hardware to our customers. After the hardware goes out to the field (customers), any hardware bug could have serious financial and logistic consequences for Sun, not to mention having to answer to our Quality Office 🙂 Financial consequence entails many order of magnitude to fix the problems using field engineers or recalling the products altogether. Logistic consequences sometimes mean living with the minor hardware bugs and resulted software legacy issues when an improved hardware is phased in. It’s all risk and disaster management. The complexity can get really nasty.

So how do we prevent “choking” at all? For the computer hardware, we would need to stress each of the circuits, even the most unlikely candidates, to failure if possible. The means looking at the boundary conditions of the design and coming up with creative ways to “break” the design and check if we have sufficient design margin to weather the constant shocks of component variations, manufacturing process variations, operation condition variations and etc, during the entire product life at the customers’ site. This is whole new topic for a separate blog at a different time.

For my 5-year old daughter’s piano lessons, it means more practices and further drill down to where she shows sign of “breaking” or carelessness. Sometimes, I would make her play a certain segment of the piece in the middle to break up the routine a bit and see how she reacts to it. It’s all about making her fail before the “big” day – the May 20, 2007 recital. It’s so good to practice your theories on your children 🙂