First, you need to realize that this book is not really about testing–it is about software development. The central idea behind TDD (Test-Driven Development) is that software, as any complex system in nature, has to evolve gradually and has to keep working throughout all the development stages. This idea is of course not new and goes back all the way to the Darwin’s “On the Origin of Species”. More recently, in his 1977 book “Systemantics: How Systems Work and Especially How They Fail” John Gall wrote:

“A complex system that works is invariably found to have evolved from a simple system that worked…. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system”.

The key point of TDD is to subject the software to constant “struggle for existence” to actually see if it indeed is still working and in the process weed out any undesired mutations.

Of course, in developing software we don’t have the deep evolutionary time, so we need to accelerate the pace of software evolution. We do this by automating the testing.

For embedded development this means avoiding the target system bottleneck (James calls it DOH-Development On Hardware). The embedded TDD strategy is to develop embedded software on the desktop and only occasionally check it on the real embedded hardware. This means that the C/C++ compilers and tools for the desktop (such as Visual C++, MinGW, or Cygwin for Windows and GCC for Linux and Mac OS X) are important for us.

The book comes with testing frameworks (Unity and CppUTest) and plenty of example code. The code works right of the box on Linux, but I had some issues running it on Windows. In the process of learning the tools, I’ve prepared a small template for Visual C++ 2008, which is available for download from:

http://www.state-machine.com/attachments/blinky_tdd.zip

This demo assumes that you download and install the CppUTest framework (http://sourceforge.net/projects/cpputest/) and that you define the environment variable CPP_U_TEST to point to the directory where you installed CppUTest. The Visual Studio solution AllTest.sln is located in the blinky\tests directory.

I’d love to hear about your experiences with TDD in embedded programming. I’m sure I will blog more about it in the future.

Arduino (see arduino.cc) is an open-source electronics prototyping platform, designed to make digital electronics more accessible to non-specialists in multidisciplinary projects. Arduino has gained popularity, because it provides both hardware and compatible software focused on developing working prototypes. By making it easier to build the first prototype, Arduino lowers the barrier of entry to the field of modern microelectronics and enables a host of new applications.

However, programming the Arduino microcontroller (Atmel AVRmega) remains a challenge. Traditionally, Arduino programs are written in a sequential manner, which means that whenever an Arduino program needs to synchronize with some external event, such as a button press, arrival of a character through the serial port, or a time delay, it explicitly waits in-line for the occurrence of the event. Waiting “in-line” means that the Arduino processor spends all of its cycles constantly checking for some condition in a tight loop (called the polling loop).

Although this approach is functional in many situations, it doesn’t work very well when there are multiple possible sources of events whose arrival times and order you cannot predict and where it is important to handle the events in a timely manner. The fundamental problem is that while a sequential program is waiting for one kind of event (e.g., a button press), it is not doing any other work and is not responsive to other events (e.g., characters from the serial port).

Another big problem with the sequential program structure is wastefulness in terms of power dissipation. Regardless of how much or how little actual work is being done, the Arduino processor is always running at top speed, which drains the battery quickly and prevents you from making truly long-lasting battery-powered devices.

Event-Driven Programming for Arduino

For these and other reasons experienced programmers turn to the long-know design strategy called event-driven programming, which requires a distinctly different way of thinking than conventional sequential programs. All event-driven programs are naturally divided into the application, which actually handles the events, and the supervisory event-driven infrastructure (framework), which waits for events and dispatches them to the application. The control resides in the event-driven framework, so from the application standpoint, the control is inverted compared to a traditional sequential program.

It turns out that the QP/C++ state machine framework beautifully complements the Arduino platform and provides everything you need to build responsive, robust, and power-efficient Arduino programs based on modern hierarchical state machines. In many ways the open source QP/C++ state machine framework is like a modern real-time operating system (RTOS) specifically designed for executing event-driven state machines. The free QM™ graphical modeling tool takes Arduino programming to the next level, by enabling automatic code generation of complete Arduino sketches.

The QP Development Kit (QDK) for Arduino is different from most other QDKs available from state-machine.com in that it is self-contained and includes the simplified and compacted source-code version of the QP/C++ framework. The QDK-Arduino is designed to plug directly into the Arduino IDE to become immediately useful without building a binary library. The extensive Application Note “Event Driven Arduino Programming with QP” describes the main concepts and how to get started.

For example, recently an Embedded.com reader attacked my comment to the article “Back to the Basics – Practical Embedded Coding Tips: Part 1 Reentrancy, atomic variables and recursion“, in which I advised against using the heap. Here is this reader’s argumentation:

I have no idea why did you bring up the pledge not to use the heap, on modern 32-bit MCUs (ARMs etc) there is no reason – and no justification – to avoid using the heap. The only reason not to use the heap is to avoid memory fragmentation, but good heap implementation and careful memory allocation planning will overcome that.

As I cannot disagree more with the statements above, I decided that it’s perhaps the time to re-post my “heap of problems” list, which goes as follows:

• Dynamically allocating and freeing memory can fragment the heap over time to the point that the program crashes because of an inability to allocate more RAM. The total remaining heap storage might be more than adequate, but no single piece satisfies a specific malloc() request.
• Heap-based memory management is wasteful. All heap management algorithms must maintain some form of header information for each block allocated. At the very least, this information includes the size of the block. For example, if the header causes a four-byte overhead, then a four-byte allocation requires at least eight bytes, so only 50 percent of the allocated memory is usable to the application. Because of these overheads and the aforementioned fragmentation, determining the minimum size of the heap is difficult. Even if you were to know the worst-case mix of objects simultaneously allocated on the heap (which you typically don’t), the required heap storage is much more than a simple sum of the object sizes. As a result, the only practical way to make the heap more reliable is to massively oversize it.
• Both malloc() and free() can be (and often are) nondeterministic, meaning that they potentially can take a long (hard to quantify) time to execute, which conflicts squarely with real-time constraints. Although many RTOSs have heap management algorithms with bounded, or even deterministic performance, they don’t necessarily handle multiple small allocations efficiently.

Unfortunately, the list of heap problems doesn’t stop there. A new class of problems appears when you use heap in a multithreaded environment. The heap becomes a shared resource and consequently causes all the headaches associated with resource sharing, so the list goes on:

• Both malloc() and free() can be (and often are) non-reentrant; that is, they cannot be safely called simultaneously from multiple threads of execution.
• The reentrancy problem can be remedied by protecting malloc(), free(), realloc(), and so on internally with a mutex, which lets only one thread at a time access the shared heap. However, this scheme could cause excessive blocking of threads (especially if memory management is nondeterministic) and can significantly reduce parallelism. Mutexes can also be subject to priority inversion. Naturally, the heap management functions protected by a mutex are not available to interrupt service routines (ISRs) because ISRs cannot block.

Finally, all the problems listed previously come on top of the usual pitfalls associated with dynamic memory allocation. For completeness, I’ll mention them here as well.

• If you destroy all pointers to an object and fail to free it or you simply leave objects lying about well past their useful lifetimes, you create a memory leak. If you leak enough memory, your storage allocation eventually fails.
• Conversely, if you free a heap object but the rest of the program still believes that pointers to the object remain valid, you have created dangling pointers. If you dereference such a dangling pointer to access the recycled object (which by that time might be already allocated to somebody else), your application can crash.
• Most of the heap-related problems are notoriously difficult to test. For example, a brief bout of testing often fails to uncover a storage leak that kills a program after a few hours, or weeks, of operation. Similarly, exceeding a real-time deadline because of nondeterminism can show up only when the heap reaches a certain fragmentation pattern. These types of problems are extremely difficult to reproduce.

I suppose that the main difficulties in understanding assertions lay in the fact that while the implementation of assertions is trivial, the effective use of assertions requires a paradigm shift in the view of software construction and the nature of software errors in particular.

Perhaps the most important point to understand about assertions is that they neither handle nor prevent errors, in the same way as fuses in electrical circuits don’t prevent accidents or abuse. In fact, a fuse is an intentionally introduced weak spot in the circuit that is designed to fail sooner than anything else, so actually the whole circuit with a fuse is less robust than without it.

I believe that the analogy between assertions and fuses (which, by the way has been originally proposed by Niall Murphy in a private conversation at one of the Embedded Systems Conferences) is accurate and valuable, because it helps in making the paradigm shift in understanding many aspects of using assertions. Here I’d only like to elaborate just two aspects.

First, the analogy to fuses correctly suggests that assertions work best in the “weakest” spots. Such “weak spots” are often found at the interface between components (e.g., preconditions in a function) but there are many others. The best assertions are those that protect the most of the system. In other words, the best assertions catch errors that would have the most impact on the rest of the system.

The second important implication of the fuse analogy is the issue of disabling assertions in the production code. As the comments to the aforementioned article suggest, most engineers tend to disable assertions before shipping the code, especially in the safety critical products. I believe that this is exactly backwards.

I understand that the standard “assert.h” header file is designed to use assertions only in a debug build, so the macro assert() compiles to nothing when the symbol NDEBUG is defined. I strongly suggest rethinking this philosophy, because disabling assertions in the release configuration is like using nails, paper clips, or coins for fuses. Just imagine finding a nail in place of a fuse in a hospital’s operating room or in a dashboard of an airliner? What would you think of this sort of “repairs”?

Yet, by disabling assertions in our code we do exactly this.

I believe it is very important to understand that assertions have a very important role to play, especially in the filed and especially in the mission-critical systems, because they add additional safety layer in the software. Perhaps the biggest fallacy of our profession is the naïve optimism that our software will not fail. In a nutshell we somehow believe that when we stop checking for errors, they will stop occurring. After all–we don’t see them anymore. But this is not how computer systems work. An error, no matter how small, can cause catastrophic failure. With software, there are no “small” errors. Our software is either in complete control over the machine or it isn’t. Assertions help us know when we lose control.

So what do I suggest we do when the assertion fires in the filed? The proper course of action requires a lot of thinking and sometimes a lot of work. In safety-critical systems software failure should be part of the fault-tree analysis. Sometimes, reaching a fail-safe state requires some redundancy in the hardware. In any case, the assertion failures should be extensively tested.

But this is really the best we can do.

In this post I’d like to tell you about my recent big hope for a truly productive IDE, which is the Qt Creator from qtsoftware.com. Qt Creator is based on the popular cross-platform Qt framework and runs natively on Windows, Linux, BSD, Mac OS X, and some embedded platforms. No Java (as in the case of Eclipse) means speed and snappy interface. Qt Software (previously Trolltech, acquired in 2008 by Nokia) offers free downloads of Qt Creator for all major platforms.

Qt Creator is primarily targeted as the IDE for Qt-related development. However, the recently released version 1.1 (April 23, 2009) supports external projects, so adding your embedded or any other projects unrelated to Qt is easy.

For example, I’ve created an embedded project for a “game” shown in the screen shot below (click on the image to see it full-size):

QtCreator

The editing surface maximizes the screen real-estate for file viewing and supports sophisticated splitting, so that my favorite side-by-side code editing is easy.

As shown in the left pane, you can add to your project as many files in different directories as you like. Given this information, Qt Creator builds an internal database of all symbols in your code to allow you exploring and navigating through your source code quickly. For example, you can jump from symbol usage to its definition by pressing F2 (press Alt-back-arrow to jump back to the previous context).

Everything in the editor is designed to enhance quick navigation. For example, every editor pane has a drop-down list of functions and other elements in the file. The editor also supports selective viewing with collapsible/expandable code sections, so you can fit more information on the screen. To quickly view the collapsed section you can simply hover your mouse cursor over it.

I immensely like the support for project-wide searching (as well as search-and-replace), which is available at the bottom of the screen. This feature alone is worth installing the tool.

Even though it is so new, Qt Creator is already very interesting, free, cross-platform IDE with features comparable to Visual Studio 2008 and other best-in-class tools. Qt Software seems very committed to enhancing Qt Creator and I hope that Qt Creator will soon catch up with Eclipse as third-party plug-ins will be developed. One feature that I will be looking forward to is side-by-side code differencing. But already, it is a powerful, free, cross-platform tool that you should try.

First of all, to benefit from side-by-side diff you need to limit the width of your lines so that you don’t need to scroll horizontally to see all the code. Countless bugs slip into a VCS, because they are hidden off screen during the final merge and people are simply tired of constantly scrolling back and forth. (All GUI usability studies agree that horizontal scrolling of text is always a bad idea.)

Granted, the modern high-resolution wide screens offer a lot of horizontal pixels, but ultimately you’ll always run out of the screen real estate if you allow lines to go on for miles. The column width must obviously allow comfortable viewing two code listings side-by-side, but you should also budget some horizontal space for the directory-tree view, vertical sliders, line numbers, and line margins, as shown in the screen shot below. I’ve been using the column width limit of no more than 78 characters. Your limit could perhaps be higher, but you must set such a limit and then enforce it without exceptions.

I can see two main reasons why people write very long lines. The first is long strings in the code. But C or C++ allow writing wide string constants in the following way:

char const s1[] = "This long string is acc\

eptable to all C compilers.";

char const s2[] = "This long string is permissible "

"in ANSI C.";

In other words, you can either use a backslash ‘\’ to terminate a string and continue in the next line, or you can terminate a string normally with a double quote ‘”‘, and an ANSI C compiler will concatenate such adjacent strings into a single zero-terminated string.

The second reason for long lines are preprocessor macros. Here again, you can use the backslash ‘\’ to break up a longer macro into lines. For example:
#define err(flag, msg) if (flag) \ printf(msg)

is the same as

#define err(flag, msg) if (flag) printf(msg)

The use of a backslash for breaking up longer lines brings up the issue of the end-of-line convention and the use of white space in your source code in general.

Let me start with the end-of-line convention. The issue here is that the backslash continuation won’t work unless the ‘\’ character is immediately followed by the end-of-line. Unfortunately, at lest two incompatible end-of-line conventions are in widespread use. The DOS/Windows end-of-line convention consists of the pair of characters CR-LF (0x0D, 0x0A in hex) to terminate lines. In contrast the UNIX™ end-of-line convention uses only one LF character (0x0A). As it turns out, Unix-like machines (e.g. Linux) are confused by the DOS end-of-line convention and will not correctly recognize the backslash-continuation, which looks like ‘\’-CR-LF (0x5C, 0x0D, 0x0A), instead of ‘\’-LF (0x5C, 0x0A).

My recommendation is to use consistently only the UNIX end-of-line convention, even on Windows machines. In my experience all Windows-based compilers have no problems with the UNIX convention, including the ancient tools from the DOS-era. As I mentioned, the converse is not true.

And finally, let me talk about the use of white space (spaces, tabs, end-of-line) in general. Obviously, to benefit from source code differencing you’d like to see only the relevant differences and differences in white space only are typically not relevant. Many code-differencing tools offer an option to ignore white space, but I would not recommend relying on it. Are files with different sizes really identical? And also, as I said before, extra spaces or tabs after the backslash, but before the end-of-line, are not allowed.

As far as tabs are concerned, I’d strongly recommend not to use them at all. Tabs are rendered differently by different editors and printers and bring only insignificant memory savings. Preferably, you should disable tabs at the editor level. At the very least, you should replace all tabs by spaces (“untabify”) before saving the file. As for spaces, I recommend removing any trailing spaces that precede the end-of-line character (LF).

Obviously, you can and should automate the source code cleanup. I use the QCLEAN utility (available here under the GPL license) for cleaning up the code from tabs, trailing blanks, and to enforce the Unix end-of-line convention. The simple console QCLEAN Windows executable scanns recursively all source files (.C, .CPP, .H, .ASM, .S, Makefile, etc.) down from the directory in which it is invoked. The following two listings show a code snippet before and after cleanup with the QCLEAN utility (spaces are shown as dots, tabs as \t, DOS end-of-lines as \r\n, UNIX end-of-lines as \n).

before cleanup:
.\t...\r\n

class.Foo.:.public.Bar.{...\n

public:.\r\n

\tFoo(int8_t.x,.int16_t.y,.int32_t z).//..ctor..\n

....:.Bar(x,.y),.m_z(z)....\n

....{}.............\n

.\t..\n

....virtual.~Foo();\t... //.xtor........\r\n

....virtual int32_t doSomething(int8_t.x);.//.method..\r\n

after cleanup with QCLEAN:
\n

class.Foo.:.public.Bar.{\n

public:\n

....Foo(int8_t.x,.int16_t.y,.int32_t z).//..ctor\n

....:.Bar(x,.y),.m_z(z)\n

....{}\n

\n

....virtual.~Foo();... //.xtor\n

....virtual int32_t doSomething(int8_t.x);.//.method\n


Even though the last year was most dramatic, the trend has been actually continuing for a number of years. This couldn’t go unnoticed by UML tool vendors, who desparately have been trying to cater to C programmers. For example, you can check out the DDJ article “UML for C Programmers” (which seems to be pretty exact re-print of the Embedded Systems Conference paper “UML for C-Based Embedded Systems“). To my surprise, nether this article, nor the ESC class mention any well-known techniques of mapping objects and classes to C. I’m sure that it is not what UML vendors like. After all, UML is crippled without objects. (The only real meat remaining are state machines.) But I suppose that the marketing departments of I-Logix/Telelogic have done their homework. Apparently embedded developers don’t like to hear about objects anymore.

I find this really disturbing. It seems that “object” and (pardon my language) “class” are becoming dirty words in the embedded circles. C++ decline is one thing. But abandoning objects is a different story. Aren’t we throwing out the baby with the bath water?

One would assume that the 21st-century software developers have objects in their bones and everyone knows how to program with objects in any language, including C. Apparently increasing number of us don’t know that object technology is a way of design, not the use of any particular language or tool. Most design and implementation techniques now associated with C++, Smalltalk, or Java, actually long predate these languages.

So here is how you implement a Point class in C (a Point that you can put on a screen):
typedef struct PointTag {

int16_t x; /* x-coordinate */

int16_t y; /* y-coordinate */

} Point;

void Point_ctor(Point *me, int16_t x, int16_t y) {

me->x = x;

me->y = y;

}

void Point_move(Point *me, int16_t dx, int16_t dy) {

me->x += dx;

me->y += dy;

}

int16_t Point_dist(Point const *me, Point const *other) {

int16_t dx = me->x – other->x;

int16_t dy = me->y – other->y;

return (int16_t)sqrt(dx*dx + dy*dy);

}. . .

/* example of using Point objects */

Point foo, bar, tar; /* multiple instances of Point */

int16_t dist;

Point_ctor(&foo, 0, 0);Point_ctor(&bar, 1, 1);

Point_ctor(&tar, -1, 2);

dist = Point_dist(&foo, &bar);

Point_move(&tar, 2, 4);dist = Point_dist(&bar, &tar);. . .

You can create any number of Point objects as instances of the Point struct. You need to initialize each point with the “constructor” Point_ctor(). You manipulate the Points only through the provided functions, which take the pointer “me” as the first argument. The “me” pointer corresponds directly to the implicit “this” pointer in C++.

Moreover, you can as easily implement single inheritance. Assume for example, that you need to add a color attribute to Points. Instead of developing such a colored-Point from scratch, you can inherit most what’s common from Point and add only what’s different. Here’s how you do it:
typedef struct ColoredPointTag {

Point super; /* derives from Point */

uint16_t color; /* 16-bit color */

} ColoredPoint;

void ColoredPoint_ctor(ColoredPoint *me,

int16_t x, int16_t y, uint16_t color) {

Point_ctor(&me->super, x, y); /* call superclass’ ctor */

me->color = color;

}
...

/* example of using ColoredPoint objects */

ColoredPoint p1, p2;int16_t dist;
ColoredPoint_ctor(&p1, 0, 2, RED);

ColoredPoint_ctor(&p2, 0, 2, BLUE);
/* re-use inherited function */

dist = Point_dist((Point *)&p1, (Point *)&p2);

As you can see, you implement inheritance by literally embedding the superclass (Point) as the first member of the subclass (ColoredPoint). Such nesting of structures always aligns the first data member ‘super’ at the beginning of every instance of the derived structure. This alignment is guaranteed by the C standard. Specifically, WG14/N1124 Section 6.7.2.1.13 says: “… A pointer to a structure object, suitably converted, points to its initial member. There may be unnamed padding within a structure object, but not at its beginning”. This alignment lets you treat a pointer to the derived ColoredPoint struct as a pointer to the Point base struct. All this is legal, portable, and blessed by the Standard.

With this arrangement, you can always safely pass a pointer to ColoredPoint to any C function that expects a pointer to Point. Consequently, all functions designed for the Point structure are automatically available to the ColoredPoint structure. They are all inherited.

There is really nothing to it.

…one day came two swindlers. They let it be known they were weavers, and they said they could weave the most magnificent fabrics imaginable. Not only were their colors and patterns uncommonly fine, but clothes made of this cloth had a wonderful way of becoming invisible to anyone who was unfit for his office, or who was unusually stupid.

…so off went the Emperor in his new clothes that were nothing at all. Everyone in the streets and the windows said, “Oh, how fine are the Emperor’s new clothes! Don’t they fit him to perfection? And see his long train!” Nobody would confess that he couldn’t see anything, for that would prove him either unfit for his position, or a fool. No costume the Emperor had worn before was ever such a complete success.”

–Hans Christian Andersen, “The Emperor’s New Clothes”

To me this little story has a lot to do with Eclipse (www.eclipse.org), which apparently is taking our industry by storm. Obviously, I must be the poor fool, unfit to see the remarkable benefits of Eclipse, but as an embedded developer I really, honestly don’t.

Admittedly, I’m a very naïve user of Eclipse, with experience limited just to two tools: the Altera Nios II Integrated Development Environment (IDE) and the Texas Instruments Code Composer Essentials for MSP430. Both these tools are based on Eclipse, and because of this both are just terrible.

I’m really not impressed with the CDT (C/C++ Development Tooling). The CDT workspaces, project files, and makefiles are notoriously difficult to move from one development workstation to another because they contain absolute paths. Even for the simplest project the CDT manages somehow to produce hundreds of files in a directory tree 3-level deep. You tell me how am I supposed to save this in any VCS (Version Control System).

The make process takes ages.

But probably, the worst part is the GDB interface to the remote target. Not only is the connection flaky and dreadfully slow (no comparison at all to other commercial offerings.) The target connectivity spawns some GDB server processes that tend to be “pigs” (i.e., take 100% of your host CPU, even if not talking to the target.) This isn’t the highest level of professionalism…

Sure, the CDT allows you to forego the automatic makefiles generation and use external Makefiles instead (which I would actually recommend). In principle, I could also go ahead and fix any problems in Eclipse, the CDT plugin, or the GDB server, because they are all available as open source. But, then I must ask if Eclipse is really such a great productivity booster? Don’t I really have a bigger fish to fry than fighting the tool?

So, as it stands, the Eclipse Emperor is naked for me.

What do you think? What are your experiences with Eclipse in the embedded system space?

But tools or no tools, the real problem in my view is not so much with creating new code, as it is in getting rid of the old code.

In every company I worked for, we had to maintain just one broad code base for all products of that particular division of the company. We only kept adding to this code base, as new features, product variants, and entirely new products were released. But we never removed anything. Needless to say, the code was a kitchen sink of everything that the company ever did, including prototypes and dead ends. Most of the stuff was long obsolete, but it lived on in our code forever.

Adding code is easy. Removing dead code (without breaking the actually used parts of the code) is hard. But without the mechanisms for dropping the old baggage, we face a real Software Crisis.

Yet most managers don’t get it. I remember one day my boss came to my desk wanting to know how much code I have just cranked out. I proudly showed him that I managed to actually remove an ugly function. He was clearly disappointed in my negative productivity.

From all my experience, I’m convinced that getting rid of code is more important than creating new code. As I said, it’s not easy, but rather requires careful planning and actual design for obsolescence. In the future installments of this blog, I plan to provide a few concrete design strategies to allow easy (or at least easier) removing of obsolete code. Stay tuned.

If you accept arguments made in Seybold’s article (and many other articles and books about agile development and extreme programming), then you must look at testing in an entirely new light. Testing is not some pain-in-the-neck chore performed long after the design and coding by the most junior and inexperienced team members. Rather, continuous testing is the primary activity that drives everything else that’s going on in the project. In fact, if you truly put so understood Testing (with capital T) at the center, the whole agile process falls out more or less automatically from this single principle.

Testing, in the agile sense, has been notoriously difficult in the embedded space. The desktop guys have powerful, commodity hardware with plenty of standard development tools. We embedded folks, on the other hand, by definition work on some custom design interfaced to proprietary, often buggy (or not even yet existing) hardware.

But it doesn’t mean that embedded developers cannot dramatically improve Testability of their software. If you truly, seriously think about Testing, you need to bend everything in the project toward the Testing, not the other way around.

Let’s start with the design. Everybody knows that modular software with independently testable pieces is good. The trick, of course, is to build it that way.

The conventional approaches, unfortunately, aren’t helping here. Take for example a traditional RTOS. The natural units of decomposition are tasks. But when you try to unit-test any real-world task, you quickly notice that it is hopelessly intertwined with other tasks by means of semaphores, shared resources, mutexes, condition variables, event flags, message mailboxes, message queues, and so on. Surely, traditional RTOSes provide no shortage of mechanisms to tie the application in a hopeless knot.

Experienced embedded gurus know to be wary of most of the RTOS mechanisms, and strictly build applications around the message-passing paradigm. Strict encapsulation is the name of the game. A task hides all its internal data and resources and communicates with the outside world only by sending and receiving events. Such systems use only a tiny fraction of the RTOS, namely message queues, and have really no need for all the other tricky RTOS mechanisms. Software components designed that way are not only easier to unit-test. They are also safer, more reusable, maintainable, and extensible.

But at this point I need to ask the nagging questions. Why structuring all systems that way is not somehow enforced in the RTOS itself? Why RTOS vendors bend over backwards to keep adding even more ways to couple the tasks?

The second aspect of software development that can make or break any successful Testing strategy is the error and exception handling policy. I’m really amazed how much complexity is added to the code by “defensive programming” techniques that somehow attempt to “handle” erroneous situations that never should have occurred in the first place, like overrunning an array index or dereferencing a NULL-pointer. The problem is that defensive programming hinders Testing… and demoralizes the testers.

You see, defensively written code accepts much wider range of inputs than it should and by doing so hides bugs. Your tests don’t appear to uncover evident errors. Yet such tests don’t build much confidence in the system, because the code might be wondering around all nights and weekends silently sweeping the errors under the rug.

A much better alternative is to confront errors head-on, by liberally using assertions (or more scientifically the Design By Contract philosophy). Testing a piece of code peppered with assertions is an entirely different experience than “defensive” code. Every successful Test run means that the program passed all its assertions. Every Test failure is much harder to dismiss as “not reproducible”, because you have a record in form of a file name and line number where the assertion fired. This information gives you an excellent starting point for understanding and ultimately fixing the bug.

And finally, Testing almost always requires instrumenting the code to give the tester additional visibility into the inner workings of the software. Unfortunately, in many embedded systems even the primitive printf() facility is unavailable (no screen to print to). Obviously, you can do much better than printf() (e.g., see the Quantum Spy software trace facility).

As you can see, Testing in the agile sense requires serious upfront investments and rethinking many of the time-honored embedded practices. You can no longer build a system without accounting for Testing right from the start.

What do you think about agile embedded software development? What do you do to improve Testability of your systems?