I have been the designated tutor of my 5-year-old daughter’s piano lessons at home (my wife does the driving to/from the piano teacher). One thing I have noticed is that there would be times when she really aced one part of the song consistently then afterward she would choke or fumble hopelessly on that particular part some days later. This is not unusual for myself, and most people or even athletes (remember the disappointment we felt when we saw the ice skaters fell miserably on a simple routine?). We sometimes make mistakes where we or others least expected that we would make. Why is that? I’ll defer the scientific explanation to the brain expert. My theory is that we often choke where we haven’t experienced failures yet or we haven’t figured out how to fail. And failures are our enforcement for continuous success.

Drawing a similarity to why a hardware system would sometimes fail where it’s least expected to fail – like I2C (the “slow” system management bus) that does the housekeeping within the system, or simple power sequencing circuits involving power supplies. And the simple answer is often that we haven’t figured out how to make it fail or test to fail – whether intentionally or unintentionally.

As part of my job at Sun Microsystems – System Validation or hardware quality assurance of sorts, we need to harden/toughen our hardware and correct/catch all of the hardware bugs before we release our hardware to our customers. After the hardware goes out to the field (customers), any hardware bug could have serious financial and logistic consequences for Sun, not to mention having to answer to our Quality Office 🙂 Financial consequence entails many order of magnitude to fix the problems using field engineers or recalling the products altogether. Logistic consequences sometimes mean living with the minor hardware bugs and resulted software legacy issues when an improved hardware is phased in. It’s all risk and disaster management. The complexity can get really nasty.

So how do we prevent “choking” at all? For the computer hardware, we would need to stress each of the circuits, even the most unlikely candidates, to failure if possible. The means looking at the boundary conditions of the design and coming up with creative ways to “break” the design and check if we have sufficient design margin to weather the constant shocks of component variations, manufacturing process variations, operation condition variations and etc, during the entire product life at the customers’ site. This is whole new topic for a separate blog at a different time.

For my 5-year old daughter’s piano lessons, it means more practices and further drill down to where she shows sign of “breaking” or carelessness. Sometimes, I would make her play a certain segment of the piece in the middle to break up the routine a bit and see how she reacts to it. It’s all about making her fail before the “big” day – the May 20, 2007 recital. It’s so good to practice your theories on your children 🙂