Archive for October, 2009

Help Bring the Embedded Software Boot Camp to Your City

Friday, October 30th, 2009 Michael Barr

When Netrino announced the first public offering of the Embedded Software Boot Camp a year and a half ago, I had no idea how popular it would be. Or just how much I could love teaching the intensive hands-on week-long version of the training we had developed over many years.

At this point, we have educated hundreds of engineers about embedded software architecture and related best practices through the topics of Hardware Interfacing in C, Multithreaded RTOS Programming, and RTOS Alternatives and ARM-based programming exercises.

Here’s just a small sampling of feedback from recent attendees:

“I would like to thank you again for the Embedded Software Boot Camp. I brought all the books back to the company and showed my boss the slides and all the handouts and all that good stuff and he was very impressed. Needless to say he was happy with the investment he made in Netrino.” — Garrett

“A better use of time and money than the Wind River VxWorks training course I took last month!” — David, IBM

“Hands on exercises are well thought out.” — Mahesh

“This is one of the best trainings I have ever attended.” — H., Hughes Network Systems

“Fabulous, pertinent, comprehensive and articulate collection of the most important things needed practically. Awesome!” — Sourabh

“Complete and correct embedded software training.” — P. Sipika

For 2010, we are planning a multi-city worldwide road-show for this popular event. I plan to teach as many of them personally as I can. I’d love to have you join us, but we first need your input to select the best cities and dates. If you’ve got a minute, please take our quick 5-question online survey at:

No personally identifying information is gathered in the survey. Thus if you want to be the first to know what cities and dates we choose, be sure to sign up for our mailing list or bookmark our public training calendar.

Logs and the Logging Loggers who Log Them

Wednesday, October 7th, 2009 Michael Barr

There is some excellent concise advice about creating a logging framework for embedded software in the recent MD&DI article “Software Design for Test“.

Portions of the advice are so good, they bear restatement in the context of our ongoing discussion about reliable firmware architecture:

1. Verbosity The API for your logger module should allow for the selection of a verbosity level, within a range from debug-only to critical. That way, application code that desires to have log something need not be changed even as you shift to a production release environment in which you wish to minimize log memory size.

2. Timestamps All logged events should be timestamped. Furthermore, these timestamps should be added as close as possible to the time of the actual event’s occurrence. Note that timer ticks since reset may be an acceptable form of a timestamp in systems not otherwise requiring date/time knowledge.

3. Readability The underlying log format should favor readability by humans. Event logs will ideally be parse-able by automated testing tools as well as by humans who must occasionally develop new automated tests. If the underlying log format cannot itself be human readable, then an offline tool should be provided to translate the raw log data to a human readable format.

If your logger module does only these three things, it will provide a powerful foundation on which much additional functionality can be built.

Slack Scheduling vs. Rate Monotonic Analysis

Friday, October 2nd, 2009 Michael Barr

The “slack scheduling” technique described in a recent article by Bill Cronk is interesting to me for a few reasons. First, because a traditional priority-based preemptive RTOS used in conjunction with RMA priority assignment offers all of the pros and none of the cons of the described slack scheduling method. And second because slack scheduling may still be valuable when working within a corporate or industry regime, such ARINC-653, that legislates a cyclic executive approach to achieving determinism.

I don’t think the original article addresses either of these points, so I will address them here.

I have written and spoken about RMA extensively in the past. For readers wanting background, start with Introduction to Rate Monotonic Schedule. Then read Why RMA is Not Just for Academics, from which this important and relevant quote:

The principal benefit of RMA is that the performance of a set of tasks thus prioritized degrades gracefully. Your key “critical set” of tasks can be guaranteed (even proven a priori) to always meet its deadlines–even during periods of transient overload. Dynamic-priority operating systems cannot make this guarantee. Nor can static-priority RTOSes running tasks prioritized in other ways.

In plain English, RMA helps you prove (via math) that your interrupt service routines (ISRs) plus the subset of tasks with deadlines will always complete their work on time. Then you can do whatever the heck you want at lower priority levels without messing that up. Consider that with this architecture a ventilator could feature a low priority video game. If the CPU becomes overloaded, only the game’s performance is at risk. The rest (most) of the time the game gets to use all the “slack” CPU cycles left behind by the critical set.

The pros of slack scheduling appear to be:

  • Non-critical tasks (i.e., those without deadline miss consequences) can’t steal CPU cycles needed by critical tasks.
  • Allows slack CPU cycles to be used by the non-critical tasks.
  • No need to ever “terminate any lower criticality [code] that exceeds” its time slot.
  • Client tasks never need to “wait for their server thread to be scheduled”.

All of these pros are common to the RMA+RTOS approach as well.

The cons of slack scheduling appear to be:

  • There is overhead associated with tracking, granting, and requesting slack time (the article doesn’t provide enough info to quantify these).
  • There is overhead associated with waking tasks (or polling instead of interrupts) to see if they need even use any of their allocated time in this cycle.
  • Like all time-partitioned code structures, it is fragile in the face of late-added work items (“the introduction of a new thread may be difficult if there is no room available on the timeline”.

The RMA+RTOS approach suffers none of these downsides.

It is important to understand that the two really powerful and beautiful aspects of RMA are first that tasks outside the critical set need not be budgeted/analyzed at all (they don’t have deadlines anyway) and second that these low priority non-critical tasks have automatic free (no-overhead) use of all the CPU cycles not used by those with deadlines. All of the critical set schedulability analysis is done on worst-case CPU use. Best case and average case may be substantially less. For example, an interrupt that could first worst-case every 1 ms eats up a lot of CPU time on paper only (well, at least until that worst case when it does really consume that much CPU); the actual implementation need not be changed to poll; and every last unused CPU cycle is available for other uses in the real system.

Unfortunately, few embedded software engineers apply RMA at all. And some of those who do, may get the math wrong. There are assumptions and calculations to be handled properly. And that’s where a time-partitioned code structure (a.k.a., cyclic executive or real-time executive) is demonstrably better than RMA+RTOS. And that is probably why ARINC-653 mandates time-partitioned scheduling.

Under a mandate of time-partitioned, the described slack scheduling approach may aid implementers.