Posts Tagged ‘ethics’

Breathalyzer Source Code Analysis

Thursday, November 5th, 2009 Michael Barr

Firmware bugs seem to be everywhere these days. So much so that firmware source code analysis is even entering the courtroom in criminal cases involving data collection devices with software inside. Consider the precedent-setting case of the Alcotest 7110. After a two-year legal fight, seven defendants in New Jersey DUI cases successfully won the right to have their experts review the source code for the Alcotest firmware.

The state and the defendants both ultimately produced expert reports evaluating the quality of the firmware source code. Though each side’s experts reached divergent opinions as to the overall code quality, several facts seem to have emerged as a result of the analysis:

- Of the available 12-bits of A/D precision, just 4-bits (most-significant) are used in the actual calculation. This sorts each raw blood-alcohol reading into one of 16 buckets. (I wonder how they biased the rounding on that.)
- Out of range A/D readings are forced to the high or low limit. This must happen with at least 32 consecutive readings before any flags are raised.
- There is no feedback mechanism for the software to ensure that actuated devices, such as an air pump and infrared sensor, are actually on or off when they are supposed to be.
- The software first averages the initial two readings. Then it averages the third reading with that average. Then the fourth reading is averaged in, etc. No comments or documentation explains the use of this formula, which causes the final reading to have a weight of 0.5 in the final value and the one before that to have a weight of 0.25, etc.
- Out of range averages are forced to the high or low limit too.
- Static analysis with lint produced over 19,000 warnings about the code (that’s about three errors for ever five lines of source code).

What would you infer about the reliability of a defendant’s blood-alcohol level if you were on that jury? If you’re so inclined you can read the full expert reports for yourself: at defendants and state.

This Code Stinks! The Worst Embedded Code Ever

Thursday, November 5th, 2009 Michael Barr

At the Embedded Systems Conference Boston in September, I gave a popular ESC Theater talk titled “This Code Stinks! The Worst Embedded Code Ever” that used lousy code from real products as a teaching tool. The example code was gathered by a number of engineers from a broad swath of companies over several years. (Minor details, including variable names and function names, were changed as needed to hide the specifics of applications, companies, or programmers.)

Here’s just one example of the bad code in that presentation:

y = (x + 305) / 146097 * 400 + (x + 305) % 146097 / 36524 * 100 + (x + 305) % 146097 % 36524 / 1461 * 4 + (x + 305) % 146097 % 36524 % 1461 / 365;

I don’t know if the above snippet contains any bugs, as most of the other examples were found to. And that’s a problem. Where are we supposed to begin an analysis of the above? What is this code supposed to do when it works? What range of input values is appropriate to test? What are the correct output values for a given input? Is this code responsible for handling out of range inputs gracefully? In the original listing, there were no comments on or around this line to help.

I eventually learned that this code computes the year, with accounting for extra days in leap years, given the number of days since a known reference date (e.g., January 1, 1970). But I note that we still don’t know if it works in all cases; despite it being present in an FDA-regulated medical device. I note too that the Microsoft Zune Bug was buried in a much better formatted snippet of code that performed a very similar calculation.

Here’s another example, this time in C++, with the bug finding left as an exercise for the reader:

bool Probe::getParam(uint32_t param_id, int32_t idx)
{
int32_t val = 0;
int32_t ret = 0;

ret = m_pParam->readParam(param_id, idx, &val);

if (!ret)
{
logMsg(“attempt to read parameter failed\n”);
exit(1);
}
else …

Hint: This code was embedded in a piece of factory automation equipment.

I’ve placed the full set of slides online at http://bit.ly/badcode.

Firmware Disasters

Tuesday, June 23rd, 2009 Michael Barr

First, an Airbus A330 fell out of the sky. Then two D.C. Metro trains collided. Several hundred people have been killed and injured in these disastrous system failures. Did bugs in embedded software play a role in either or both disasters?

An incident on an earlier (October 2006) Airbus A330 flight may offer clues to the crash of Air France 447:

Qantas Flight 72 had been airborne for three hours, flying uneventfully on autopilot from Singapore to Perth, Australia. But as the in-flight dinner service wrapped up, the aircraft’s flight-control computer went crazy. The plane abruptly entered a smooth 650-ft. dive (which the crew sensed was not being caused by turbulence) that sent dozens of people smashing into the airplane’s luggage bins and ceiling. More than 100 of the 300 people on board were hurt, with broken bones, neck and spinal injuries, and severe lacerations splattering blood throughout the cabin. (Article, Time Magazine, June 3, 2009)

Authorities have blamed a pair of simultaneous computer failures for that event in the fly-by-wire A330. First, one of three redundant air data inertial reference units (ADIRUs) began giving bad angle of attack (AOA) data. Simultaneously, a voting algorithm intended to handle precisely such a failure in 1 of the 3 units by relying only on the other matching data failed to work as designed; the flight computer instead made decisions only on the basis of the one failed ADIRU!

(A later analysis by Airbus “found data fingerprints suggesting similar ADIRU problems had occurred on a total of four flights. One of the earlier instances, in fact, included a September 2006 event on the same [equipment] that entered the uncommanded dive in October [2006].” Ibid.)

Much of the attention in the publicly disclosed details of the Air France 447 crash has focused on the failure of one of several air speed indicators. Were there three of those as well? If so, was the same flight computer to blame for failing to recognize which to trust and which was unreliable?

It is very early in the investigation of yesterday’s collision between two D.C. Metro red line trains, in which a stopped train was rear-ended and heavily damaged by a moving train on the same track, to place blame. But a WashingtonPost.com article headlined “Collision was Supposed to be Impossible” says it all:

Metro was designed with a fail-safe computerized signal system that is supposed to prevent trains from colliding.

and

During morning and afternoon rush hours, all trains except longer eight-car trains typically operate in automatic mode, meaning their movements are controlled by computerized systems and the central Operations Control Center. Both trains in yesterday’s crash [about 5pm] were six-car trains. (Article, Washington Post, June 23, 2009)

Are bugs in embedded software to blame for these two disasters? You can bet the lawyers are already looking into it.

Resume Inflation

Wednesday, January 14th, 2009 Michael Barr

Like many companies, Netrino is perpetually interviewing candidates for software engineering positions. However, since our work is primarily in the area of embedded systems design we tend to meet many software candidates who come originally from electrical engineering backgrounds. Among this group of candidates there is a prevalent bias toward a kind of resume inflation, in the technical skills section.

Almost all of the resumes we receive for firmware engineering positions contain one of the strings “C, C++” or “C/C++” at or near the front of a longer list of programming language skills (e.g., “C/C++, Fortran, Assembly (various), Forth”). C is generally at the front of that language list for good reason–because it is the one programming language with which the candidate has the most hands-on experience. Too often, though, our interviews have revealed that C++ is listed second for no good reason.

C and C++ are fundamentally different languages. Many of these “resume inflation” job candidates don’t even realize that C++ is as different as it is from C. Often, their actual experience with C++ turns out to be that they’ve used C++ compilers to build C programs. And when they really have used C++, their practical experience usually ends at the use of classes.

Object-oriented C++ has three important aspects: encapsulation, inheritance, and polymorphism. Mere use of encapsulation is insufficient to claim you have C++ programming skills. When we include C++ in a job listing it is because the work may involve the use of every part of C++. To get the job, you must be knowledgeable in all of C++.

What can you do if you are submitting your resume and know some C++ rather than a lot? Be clear and humble. If you merely worked with C++ in school or on a single project, list it in a separate section. Or put C++ apart from C in your language list.

Interviewers (as well as resume reviewers) are impressed by modesty and clear communication.

Programming as Profession

Wednesday, January 14th, 2009 Michael Barr

Doing my daily reading around the embedded systems press, I happened across this gem in an article by Jim Turley:

Programming is often treated as a creative endeavor, undertaken by spirited and talented artistes who cannot or should not be shackled to convention, regulations, or reasonable hygiene. Get over it. Programming is a job like any other, and an employer’s responsibility is to ship a profitable product, not coddle and babysit self-indulgent hackers.

I couldn’t agree with Jim more. It is especially unfortunate when a lack of professionalism shows itself in the realm of embedded systems, such as medical devices, that put human lives at risk. I regret to inform the reader that from Netrino‘s vantage point as consultants we see everyday the train wreck that results from poorly managed and unprofessional programmers working in the embedded systems space.

You need a license from the state to to be a barber. To write embedded software, you simply need to be able to spell ‘C’. But I’m on a mission to change that.