Win Some, Lose Some … For Now

This week had both. The free upgrade from Hertz was decidedly nice but the customer’s insistence on writing his ISR handler in C++ wasn’t.

For the non-geek types, you probably know that computer programs are written in different languages. This is because you can do so many different things with computers that “no one size fits all”. The choice of language is often about the engineer’s time and how long it takes him or her to write it. Some languages assume a lot and, when those assumptions are appropriate, the engineer can write the program and be done with it very quickly.

ISRs (Interrupt Service Routines) are a very special category of software. They have a very strict list of assumptions that must be carefully observed. And ISRs are so special that many software engineers will go their entire career and never write one. They are rare and difficult to write.

The company I work for has, for years, told its customers that they must write ISRs in the C language, not C++, because the latter makes several assumptions that are wrong when it comes to writing ISRs. (Those two languages are very popular in the niche our customers occupy.)

But customers can be insistent, and when they are good ($) customers, you don’t want to say “No.” Instead, you try to dissuade them by saying “Yes, but” and then explaining why something is a bad idea.

So, we wrote up a fairly long laundry list of DOs and DON’Ts, show how to write ISRs in C and then describe where C++ would do it wrong and what extraordinary steps you would use to keep C++ from doing that. We also add the caveat that, even if you make it work, a decade later and long after you are gone, another engineer might come along, not be aware of that laundry list and, in the ISR you wrote “way back when”, decide to add one of the NO-NO things.

“Boom!” the computer will crash, but only rarely, very rarely.

The errors that cause these crashes are called “race conditions.” What we mean is that a sequence of things has to happen in a very precise order for the failure to happen. And with ISRs, the odds of that sequence happening are extremely small. We’re talking odds of 1 in a billion or 1 in a thousand-billion. Odds so rare you might think, “Gee, that’ll never happen!”

But computers do things billions of times per second so that exceedingly low probability actually does happen. It’s still rare, but when the computer is executing that program day and night, 24/7 for years at a time, it happens.

“Boom!” the computer crashes.

(I can’t say what this customer’s products do but I guess it’s fair to ask if you have you seen any 30 second drop-outs in your cable TV or internet on-demand service? If so, then … but I’m not sayin’.)

Worse, when you try to analyze what went wrong, the damage inflicted could be in hundreds of places so the odds of two (rare) crashes looking alike is almost nil. To all appearances, you don’t have one problem, you have a whole bunch.

Late Wednesday when I informed them that C++ in an ISR was going to be a problem and that we strongly recommended they use plain C instead, this week’s customer said, in essence, “Well, we know all that but decided to do it anyway and our product has worked just fine for more than a decade.”

I congratulated them on their good fortune and we moved on to another topic.

At the end of the day as I was packing equipment and getting ready to leave, that same engineer asked, “Say, after you’re done with that, could we get you to look at some mysterious crashes we’ve been accumulating for a couple of years?”

Having been an occasional stage performer and, in my role as a teacher who sometimes tells a funny story to illustrate a technical point, I’m familiar with “the look” – you know the one that says, “Are you kidding me?”

I used it but only got back a blank stare.

So, after a couple of beats I said, “I’m sure our services group could be hired to give you a hand but I’m afraid my time here is up.”

It was after 7:00PM and I was ready to be “out the door, thank you very much!”

Training went well and we helped them acquire lots of new skills. As often requested, we skipped some key topics at their request – “My engineers already know that” (but no, they really don’t) – and those too are now glaringly obvious to them. They now know where they are ignorant, and that their jobs will be hard if they don’t fix that. This is common in my experience as a paid trainer and I confidently expect they will book additional training somewhere in the next couple of months.

Assuming that return visit happens, it’ll be interesting to see if they’ve replaced the C++ in their ISRs or if their foreheads are still bloody from banging them against that brick wall.

(And can I get another free upgrade to that convertible Mustang? Boy, that was nice!)