Posts Tagged ‘ethics’

Firmware Disasters

Tuesday, June 23rd, 2009 Michael Barr

First, an Airbus A330 fell out of the sky. Then two D.C. Metro trains collided. Several hundred people have been killed and injured in these disastrous system failures. Did bugs in embedded software play a role in either or both disasters?

An incident on an earlier (October 2006) Airbus A330 flight may offer clues to the crash of Air France 447:

Qantas Flight 72 had been airborne for three hours, flying uneventfully on autopilot from Singapore to Perth, Australia. But as the in-flight dinner service wrapped up, the aircraft’s flight-control computer went crazy. The plane abruptly entered a smooth 650-ft. dive (which the crew sensed was not being caused by turbulence) that sent dozens of people smashing into the airplane’s luggage bins and ceiling. More than 100 of the 300 people on board were hurt, with broken bones, neck and spinal injuries, and severe lacerations splattering blood throughout the cabin. (Article, Time Magazine, June 3, 2009)

Authorities have blamed a pair of simultaneous computer failures for that event in the fly-by-wire A330. First, one of three redundant air data inertial reference units (ADIRUs) began giving bad angle of attack (AOA) data. Simultaneously, a voting algorithm intended to handle precisely such a failure in 1 of the 3 units by relying only on the other matching data failed to work as designed; the flight computer instead made decisions only on the basis of the one failed ADIRU!

(A later analysis by Airbus “found data fingerprints suggesting similar ADIRU problems had occurred on a total of four flights. One of the earlier instances, in fact, included a September 2006 event on the same [equipment] that entered the uncommanded dive in October [2006].” Ibid.)

Much of the attention in the publicly disclosed details of the Air France 447 crash has focused on the failure of one of several air speed indicators. Were there three of those as well? If so, was the same flight computer to blame for failing to recognize which to trust and which was unreliable?

It is very early in the investigation of yesterday’s collision between two D.C. Metro red line trains, in which a stopped train was rear-ended and heavily damaged by a moving train on the same track, to place blame. But a WashingtonPost.com article headlined “Collision was Supposed to be Impossible” says it all:

Metro was designed with a fail-safe computerized signal system that is supposed to prevent trains from colliding.

and

During morning and afternoon rush hours, all trains except longer eight-car trains typically operate in automatic mode, meaning their movements are controlled by computerized systems and the central Operations Control Center. Both trains in yesterday’s crash [about 5pm] were six-car trains. (Article, Washington Post, June 23, 2009)

Are bugs in embedded software to blame for these two disasters? You can bet the lawyers are already looking into it.

Resume Inflation

Wednesday, January 14th, 2009 Michael Barr

Like many companies, Netrino is perpetually interviewing candidates for software engineering positions. However, since our work is primarily in the area of embedded systems design we tend to meet many software candidates who come originally from electrical engineering backgrounds. Among this group of candidates there is a prevalent bias toward a kind of resume inflation, in the technical skills section.

Almost all of the resumes we receive for firmware engineering positions contain one of the strings “C, C++” or “C/C++” at or near the front of a longer list of programming language skills (e.g., “C/C++, Fortran, Assembly (various), Forth”). C is generally at the front of that language list for good reason–because it is the one programming language with which the candidate has the most hands-on experience. Too often, though, our interviews have revealed that C++ is listed second for no good reason.

C and C++ are fundamentally different languages. Many of these “resume inflation” job candidates don’t even realize that C++ is as different as it is from C. Often, their actual experience with C++ turns out to be that they’ve used C++ compilers to build C programs. And when they really have used C++, their practical experience usually ends at the use of classes.

Object-oriented C++ has three important aspects: encapsulation, inheritance, and polymorphism. Mere use of encapsulation is insufficient to claim you have C++ programming skills. When we include C++ in a job listing it is because the work may involve the use of every part of C++. To get the job, you must be knowledgeable in all of C++.

What can you do if you are submitting your resume and know some C++ rather than a lot? Be clear and humble. If you merely worked with C++ in school or on a single project, list it in a separate section. Or put C++ apart from C in your language list.

Interviewers (as well as resume reviewers) are impressed by modesty and clear communication.

Programming as Profession

Wednesday, January 14th, 2009 Michael Barr

Doing my daily reading around the embedded systems press, I happened across this gem in an article by Jim Turley:

Programming is often treated as a creative endeavor, undertaken by spirited and talented artistes who cannot or should not be shackled to convention, regulations, or reasonable hygiene. Get over it. Programming is a job like any other, and an employer’s responsibility is to ship a profitable product, not coddle and babysit self-indulgent hackers.

I couldn’t agree with Jim more. It is especially unfortunate when a lack of professionalism shows itself in the realm of embedded systems, such as medical devices, that put human lives at risk. I regret to inform the reader that from Netrino‘s vantage point as consultants we see everyday the train wreck that results from poorly managed and unprofessional programmers working in the embedded systems space.

You need a license from the state to to be a barber. To write embedded software, you simply need to be able to spell ‘C’. But I’m on a mission to change that.

Breathalizer Source Code to Get a Day in Court

Thursday, February 7th, 2008 Michael Barr

Here’s an interesting news story at the intersection of embedded systems and due process:

http://www.news.com/Police-Blotter-Intoxilyzer-code-must-be-disclosed/2100-1030_3-6227951.html

How many potential bugs might a knowledgable expert witness spot in your code? Are the comments in your source accurate and clean enough for a judge or jury to read?

Building Reliable Systems

Wednesday, September 19th, 2007 Michael Barr

I’m writing this live from the Embedded Systems Conference in Boston, while participating in a birds of a feather discussion moderated by Jack Ganssle. The subject of the session is Building Reliable Systems.

The discussion here amongst perhaps 80 engineers (about 75% electrical engineers by education) initially focused on resources and schedules and the inevitability of bugs, but has now turned to what seems to a more productive thread: specific processes and tools that produce higher reliability.

One gentleman, had a great way of summarizing what needs to be done at a high level:

* Prioritization
* Process
* Metrics

In this view, Prioritization is an input to the Process. That is, prioritization of features relative to one another–as well as the accurate definition of properties such as quality and reliability. These are to be provided by the customer or from engineering management. The Process used for design and development and testing are then guided by these input parameters. Metrics are an output of such a Process, if you strive to generate them. Metrics include the schedule time it took to complete specific feature implementations, as well as bug rates.

On the specific point of process and tools, we’ve discovered that just a few companies represented in the room are using code coverage tools and test-driven development, though most appear to agree these might be helpful in raising reliability.

My bottom line fear: One of the products designed by someone in the room with me right now will kill or maim eventually!