Technically, the problem of a non-reentrant functions is a special case of the problem of a race condition.  For that reason the run-time errors caused by a non-reentrant function are similar and also don’t occur in a reproducible way—making them just as hard to debug.  Unfortunately, a non-reentrant function is also more difficult to spot in a code review than other types of race conditions.

The figure below shows a typical scenario.  Here the software entities subject to preemption are RTOS tasks.  But rather than manipulating a shared object directly, they do so by way of function call indirection.  For example, suppose that Task A calls a sockets-layer protocol function, which calls a TCP-layer protocol function, which calls an IP-layer protocol function, which calls an Ethernet driver.  In order for the system to behave reliably, all of these functions must be reentrant.

But the functions of the driver module manipulate the same global object in the form of the registers of the Ethernet Controller chip.  If preemption is permitted during these register manipulations, Task B may preempt Task A after the Packet A data has been queued but before the transmit is begun.  Then Task B calls the sockets-layer function, which calls the TCP-layer function, which calls the IP-layer function, which calls the Ethernet driver, which queues and transmits Packet B.  When control of the CPU returns to Task A, it finally requests its transmission.  Depending on the design of the Ethernet controller chip, this may either retransmit Packet B or generate an error.  Either way, Packet A’s data is lost and does not go out onto the network.

In order for the functions of this Ethernet driver to be callable from multiple RTOS tasks (near-)simultaneously, those functions must be made reentrant.  If each function uses only stack variables, there is nothing to do; each RTOS task has its own private stack.  But drivers and some other functions will be non-reentrant unless carefully designed.

The key to making functions reentrant is to suspend preemption around all accesses of peripheral registers, global variables (including static local variables), persistent heap objects, and shared memory areas.  This can be done either by disabling one or more interrupts or by acquiring and releasing a mutex; the specifics of the type of shared data usually dictate the best solution.

Best Practice: Create and hide a mutex within each library or driver module that is not intrinsically reentrant.  Make acquisition of this mutex a pre-condition for the manipulation of any persistent data or shared registers used within the module as a whole.  For example, the same mutex may be used to prevent race conditions involving both the Ethernet controller registers and a global (or static local) packet counter.  All functions in the module that access this data, must follow the protocol to acquire the mutex before manipulating these objects.

Beware that non-reentrant functions may come into your code base as part of third party middleware, legacy code, or device drivers.  Disturbingly, non-reentrant functions may even be part of the standard C or C++ library provided with your compiler.  For example, if you are using the GNU compiler to build RTOS-based applications, take note that you should be using the reentrant “newlib” standard C library rather than the default.

Firmware-Specific Bug #1

Firmware-Specific Bug #3

A race condition is any situation in which the combined outcome of two or more threads of execution (which can be either RTOS tasks or main() plus an ISR) varies depending on the precise order in which the instructions of each are interleaved.

For example, suppose you have two threads of execution in which one regularly increments a global variable (g_counter += 1;) and the other occasionally resets it (g_counter = 0;). There is a race condition here if the increment cannot always be executed atomically (i.e., in a single instruction cycle). A collision between the two updates of the counter variable may never or only very rarely occur. But when it does, the counter will not actually be reset in memory; its value is henceforth corrupt. The effect of this may have serious consequences for the system, though perhaps not until a long time after the actual collision.

Best Practice: Race conditions can be prevented by surrounding the “critical sections” of code that must be executed atomically with an appropriate preemption-limiting pair of behaviors. To prevent a race condition involving an ISR, at least one interrupt signal must be disabled for the duration of the other code’s critical section. In the case of a race between RTOS tasks, the best practice is the creation of a mutex specific to that shared object, which each task must acquire before entering the critical section. Note that it is not a good idea to rely on the capabilities of a specific CPU to ensure atomicity, as that only prevents the race condition until a change of compiler or CPU.

Shared data and the random timing of preemption are culprits that cause the race condition. But the error might not always occur, making tracking down such bugs from symptoms to root causes incredibly difficult. It is, therefore, important to be ever-vigilant about protecting all shared objects.

Best Practice: Name all potentially shared objects—including global variables, heap objects, or peripheral registers and pointers to the same—in a way that the risk is immediately obvious to every future reader of the code. Netrino’s Embedded C Coding Standard advocates the use of a ‘g_‘ prefix for this purpose.

Locating all potentially shared objects is the first step in a code audit for race conditions.

Firmware-Specific Bug #2: Non-Reentrant Function