Posts Tagged ‘programming’

Firmware Wall of Shame: Welch Allyn Defibrillator Recall

Tuesday, March 17th, 2009 Michael Barr

The FDA has just announced a Class I (the most serious human risk category) recall of the Welch Allyn AED 10 automatic external defibrillator (shown).

Among the reasons for the recall are the following problems that are either caused by embedded software bugs or hardware problems able to be fixed entirely through a firmware upgrade:

  • “Units serviced in 2007 and upgraded with software version 02.06.00 have a remote possibility of shut down during use in cold environmental conditions. There are no known injuries or deaths associated with this issue. The units will be updated with the current version of software.”
  • “All of the recalled units will be upgraded with software that corrects [another] unexpected shutdown problem. In the meantime … it is vital to follow the step 1-2-3 operating procedure which directs attachment of the pads after the device has been turned on. This procedure is described on the back of your device and also in the Quick Reference material inside the AED 10 case. Some pages in the user’s manual may erroneously describe or show illustrations of [a different] operating procedure… Please disregard these erroneous instructions.”

There has been at least one death at a time when the second type of unexpected software shutdown occurred. Are bugs in the embedded software to blame? Of what sort? Could the authors of that firmware be sued in relation to the death? Were they negligent? Are they sure that there are Zero Bugs (or even just fewer bugs) in the “current version of the software”?

Expect more of this type of firmware-involved death as embedded systems continue to proliferate.

RTOS Myth #5: You Need One

Monday, February 23rd, 2009 Michael Barr

The Myth: You need a real-time operating system (RTOS) to make your embedded software easy to implement and maintain.

The Truth: Three positive implications of the use of a preemptive priority-based RTOS must be weighed against ten negative implications. An RTOS works well in some scenarios, but overly complicates the design of many other systems.

Annual surveys of the readers of EETimes and Embedded Systems Design generally find that about half (50%) of embedded software developers work on projects that utilize a commercial RTOS, such as VxWorks or uC/OS-II. Due to a sampling bias stemming from the correlation between big teams and RTOS use, the number of new products containing an RTOS is likely much lower than 50%.

Contrary to popular belief, a real-time operating system (RTOS) is not the answer to all of your design problems. When chosen for the wrong reasons, the presence an RTOS can make the firmware design more complicated rather than less. In addition, preemption increases the opportunity for race condition and non-reentrancy bugs. Finally, the inclusion of an RTOS has other costs, such as additional RAM and ROM/Flash.

By my calculations, preemption has three principal positive implications and ten negative implications. For example, one positive is that an RTOS can help separate the timing (i.e., which code is running on the CPU when?) from the application-level algorithmics. The problem is decomposed into a less “fragile”/more maintainable firmware design.

However, the use of separately-coded parallel tasks also complicates the software. Without the RTOS, the only possible race conditions occur between communicating interrupt service routines and the background loop in main(). Additional tasks increase the need for communication and synchronization factorially.

Perhaps the most interesting tradeoff concerns responsiveness to interrupts. Although what an RTOS does is to divide up the spare CPU time not used by any interrupt service routine (ISR) between pseudo-parallel running tasks (i.e., a set of C functions that don’t return), one negative side effect is slower CPU response to interrupts. The interrupt latency is higher with an RTOS than without.

Don’t get me wrong, sometimes an RTOS is a valuable tool. But don’t go using one simply to put its name on your resume. You may instead find yourself languishing in an overly-complicated and buggy embedded software design at your present job.

The “RTOS Alternatives” course by Netrino Institute includes full coverage of these tradeoffs, including details of each of preemptions three positives and all ten of the negative implications of a commercial RTOS.

Go back to RTOS Myth #4.

Requirements vs. Design

Wednesday, February 4th, 2009 Michael Barr

Over the years, I have found that many engineers (as well as their managers) struggle to separate the various elements or layers of firmware engineering. For example, we are barraged with requests for “design reviews” that turn out to be “code reviews” because the customer is confused about the meaning of “design”.

In the hopes of clearing this up, I propose a concise set of definitions and an architectural analogy.

Requirements
The requirements are the WHAT of the system. A set of requirements is a list of statements each of which begins “The system shall…” Each such statement must be objective and testable. The requirements should not unnecessarily restrict the HOW of the architecture, design, or implementation.

Architecture
The architecture of a system is the outermost layer of HOW. The architecture is a block diagram. The architecture of a system describes dataflow and workflow partitioning at the hardware vs. software level. The architecture of firmware features subsystem-level blocks such as device drivers, middleware, RTOS, etc. The architecture does not include function or variable names. It should be extensible in the direction of anticipated future changes.

Analogy: An architect describes a new building very broadly. A scale model and drawings show the outer dimensions, foundation, and number of floors. The number of rooms and their specific uses are not included at this level.

Design
The design of a system is the middle layer of HOW. A firmware design document identifies finer structural details, such as the names and responsibilities of tasks within the specific subsystems or device drivers, the brand of RTOS (if one is used), and the various interfaces between subsystems. The design does include class, task, function, and variable names that must be agreed upon by all implementers.

Analogy: A designer describes the interior and exterior of the new building in finer detail than the architect. He locates and names the rooms and gives them purposes. The location of pipes and vents and outlets are not included at this level.

Implementation
An implementation is the lowest layer of HOW. There is no document, other than the source code or schematics, to describe the implementation details. If the interfaces are defined sufficiently at the design level above, individual engineers are able to begin implementation in parallel.

Analogy: The carpenter, plumber, and electrician work in parallel and apply their own judgement about the finer details of component placement.

Constructive feedback is welcome via the blog comments or e-mail.

RTOS Myth #4: The RTOS is in Charge

Friday, June 6th, 2008 Michael Barr

The Myth: The operating system is in charge and it decides when to switch from one application task to another.

The Truth: A real-time operating system (RTOS) is a very different beast than a multi-user desktop operating system, such as Linux. In fact, an RTOS is simply a library of functions plus a timer tick interrupt handler.

The only opportunities for an RTOS to effect a context switch from one task to another are:

1. If a running task deletes itself (or exits, if your OS allows that). In this case, a function in the RTOS library detects the lack of a running task and can directly invoke the scheduler function to select the next task to run.

2. When a running task blocks, which can only happen by making a function call into the RTOS library.

3. If a running task creates a new task with a priority higher than it’s own.

4. When a previously blocked task of higher priority is unblocked, which could happen as a result of:

a. The running task made a function call into the RTOS library (e.g., semaphore post).

b. An interrupt service routine executed with that side effect.

These four (or, five, if you prefer) points of entry into the RTOS are the only mechanisms by which control of the CPU can be transferred from one task to another. They are called “scheduling points”.

The implication here is that your application code is actually in charge. If it were to avoid calling into the RTOS library while simultaneously disabling interrupts, the running task could steal control of the CPU for any length of time.

Go back to RTOS Myth #3 or forward to RTOS Myth #5.

Real-Time Java is Dead

Friday, April 25th, 2008 Michael Barr

A few months less than ten years ago, I presented a paper at the Embedded Systems Conference (ESC) for the first time. My 1.5 hour course was entitled “Embedded Java” or something similar. This was in Silicon Valley, and the audience was standing room only–despite a rather large room to start with.

I’ve tracked technical and business developments in the world of embedded and real-time Java for even longer–and written a number of articles on the subject. And so I didn’t miss that over the years the audiences for courses on either variant of Java dwindled. After a hopeful year or two too long, I gave up on Java in our space and stopped proposing the topic at ESC.

Then, last Spring, I had the refreshing experience of teaching a two-day hands-on real-time Java programming class in the Netherlands. The room was packed. There was enthusiasm. And these guys were really using Java to develop software (though it wasn’t truly embedded code). So I thought I’d try again at ESC and reproposed the topic.

Last week I taught a course called “Real-Time Java Programming” at the ESC Silicon Valley venue. A lot had changed about the audience size. This time, in a room of a similar size, there were many empty chairs and tables. I think there were perhaps 15 people attending this time.

By my reckoning, Java is officially dead in the embedded systems community–especially in the U.S.