Last month, NHTSA and the NASA Engineering and Safety Center (NESC) published reports of their joint investigation into the causes of unintended acceleration in Toyota vehicles. NASA’s multi-disciplinary NESC technical team was asked, by Congress, to assist NHTSA by performing a review of Toyota’s electronic throttle control and the associated embedded software. In carefully worded concluding statement, NASA stated that it “found no electronic flaws in Toyota vehicles capable of producing the large throttle openings required to create dangerous high-speed unintended acceleration incidents.” (The official reports and a number of supporting files are available for download at http://www.nhtsa.gov/UA.)

The first thing you will notice if you join me in trying to judge the technical issues for yourself are the redactions: pages and pages of them. In parts and entirely for unexplained reasons, this report on automotive electronics reads like the public version of a CIA Training Manual. I’ve observed that approximately 193 of the 1,061 pages released so far feature some level of redaction (via black boxes, which obscure from a single number, word, or phrase to a full table, page, or section). The redactions are at their worst in NASA’s Appendix A, which describes NASA’s review of Toyota’s embedded software in detail. More than half of all the pages with redactions (including the vast majority of fully redacted tables, pages, and sections) are in that Appendix.

Despite the redactions, we can still learn some interesting facts about Toyota’s embedded software and NASA’s technical review of the same. The bulk of the below outlines what I’ve been able to make sense of in about two days of reading. Throughout, my focus is on embedded software inside the electronic throttle control, so I’m leaving out considerations of other potential causes, including EMI (which NASA also investigated). First a little background on the investigation.

Background

Although the inquiry was taken to examine unintended acceleration reports across all Toyota, Scion, and Lexus models, NASA focused its technical inquiry almost entirely on Toyota Camry models equipped with the Electronic Throttle Control System, Intelligent (ETCS-i). The Camry has long been among the top cars bought in the U.S., so this choice probably made finding relevant complaint data and affected vehicles easier for NHTSA. (BTW, NASA says the voluntary complaint database shows both that unintended accelerations were reported before the introduction of electronic throttle control and that press coverage and Congressional hearings can increase the volume of complaints.)

According to a press release by the company made upon publication of the NHTSA and NASA reports, Toyota’s ETCS-i has been installed in “more than 40 million cars and trucks sold around the world, including more than 16 million in the United States.” Undoubtedly, ETCS-i has also “made possible significant safety advances such as vehicle stability control and traction control.” But as with any other embedded system there have been refinements made through the years to both the electronics and the embedded software.

Though Toyota apparently made available, under agreed terms and via its attorneys, schematics, design documents, and source code “for multiple Camry years and versions” (Appendix A, p. 9) as well as many of the Japanese engineers involved in its design and evolution, NASA only closely examined one version. In NASA’s words, “The area of emphasis will be the 2005 Toyota Camry because this vehicle has a consistently high rate of reported ‘UA events’ over all Toyota models and all years, when normalized to the number of each model and year, according to NHTSA data.” (p. 7) Except as otherwise stated, everything else in this column concerns the electronics and firmware found in that year, make, and model.

Event Data Recorders

Event Data Recorder (EDR) is the generic term for the automotive equivalent of an aircraft black box flight data recorder. EDRs were first installed in cars in the early 1990s and have increased in use as well as sophistication in the years since. Generally speaking, the event data recorder is an embedded system residing within the airbag control module located in the front center of the engine compartment. The event data recorder is connected to other parts of the car’s electronics via the CAN bus and is always monitoring vehicle speed, the position of the brake and accelerator pedals, and other key parameters.

In the event of an impossibly high (for the vehicle operating normally) acceleration or deceleration sensor reading, Toyota’s latest event data recorders save the prior five 1Hz samples of these parameters in a non-volatile memory area. Once saved, an event record can be read over the car’s On-Board Diagnostics (OBD) port (or, in the event of a more severe accident, directly from the airbag control module) via a special cable and PC software. If the airbag actually deploys, the event record will be permanently locked. The last 2 or 3 (depending on version) lesser “bump” records are also stored, but may be overwritten in a FIFO manner.

This investigation of Toyota’s unintended acceleration marked the first time that anyone from NHTSA had ever read data from a Toyota event data recorder. (Toyota representatives apparently testified in Congress that there had previously just been one copy of the necessary PC software in the U.S.) As part of this study, NHTSA validated and used tools provided by Toyota to extract historical data from 52 vehicles involved in incidents of unintended acceleration, with acknowledged bias toward geographically reachable recent events. After reviewing driver and other witness statements and examining said black box data, NHTSA concluded that 39 of these 52 events were explainable as “pedal misapplications.” That’s a very nice way of saying that whenever the driver reported “stepping on the brake” he or she had pressed the accelerator pedal by mistake. Figure 5 of a supplemental report describing these facts portrays an increasing likelihood of such incidents with driver age vs. the bell curve of Camry ownership by age.

Note that no record is apparently ever made, in the event data recorder or elsewhere, of any events or state changes within the ETCS-i firmware. So-called “Diagnostic Trouble Codes” concerning sensor and other hardware failures are recorded in non-volatile memory and the presence of one or more such codes enables the “Check Engine” light on the dashboard. But no logging is done of significant software faults, including but not limited to watchdog-initiated resets.

Engine Control Module

ETCS-i is a collection of components and features that was changed in the basic engine design when Toyota switched from mechanical to electronic throttle control. (Electronic throttle control is also known as “throttle-by-wire”.) Toyota has used two different types of pedal sensors in the ETCS-i system, always in a redundant fashion. The earlier design, pre-2007, using potentiometers was susceptible to current leakage via growth of tin whiskers. Though this type of failure was not known to cause sudden high-speed behaviors, it did seem to be associated with a higher number of warranty claims. The newer pedal sensor design uses Hall effect sensors.

Importantly, the brakes are not a part of the ETCS-i system. In the 2005 Camry, Toyota’s brake pedal was mechanically controlled. (It may still be.) It appears this is one of the reasons the NASA team felt comfortable with their conclusion that driver reports of wide open throttle behavior that could not be stopped with the brakes were not caused by software failures (alone). “The NESC team did not find an electrical path from the ETCS-i that could disable braking.” (NASA Report, p. 15) It is clear, though, that power assisted brakes lose the enabling vacuum pressure when the throttle is wide open and the driver subsequently pumps the brakes; thus any system failure that opened the throttle could indirectly make bringing the vehicle to a stop considerably harder.

The Engine Control Module at the heart of the ETCS-i consists of a Main-CPU and a Sub-CPU located within a pair of ASICs. The Sub-CPU contains a set of A/D converters that translates raw sensor inputs, such as voltages VPA and VPA1 from the accelerator pedal, into digital position values and sends them to the Main-CPU via a serial interface. In addition, the Sub-CPU monitors the outputs of the Main-CPU and is able to reset (in the manner of a watchdog timer) the Main-CPU.

The Main-CPU is reported to be a V850E1 microcontroller, which is “a 32-bit RISC CPU core for ASIC” designed by Renesas (nee NEC). The V850E1 processor has a 64MB program address space, which is part of an overall 4GB linear address space. The Main-CPU also keeps tabs on the Sub-CPU and can reset it if anything is found wrong.

NASA reports that the embedded software in the Main-CPU is written (mostly) in ANSI C and compiled using a GreenHills C compiler (Appendix A, p. 14). Furthermore, an OSEK-compliant real-time operating system with fixed-priority preemptive scheduling is used to manage a redacted (but apparently larger than ten, based on the size of the redaction) number of real-time tasks. The actual firmware development (design, coding and unit testing) was outsourced to Denso (p. 19). Toyota apparently performed integration testing and ran several commercial and in-house static analysis tools, including QAC (p. 20). The code was written in English, with Japanese comments and design documents, and follows a proprietary Toyota naming convention/coding standard that predates but half overlaps with the 1998 version of MISRA-C.

Are There Bugs in Toyota’s Firmware?

In the NASA Report’s executive summary it is made clear that “because proof that the ETCS-i caused the reported UAs was not found does not mean it could not occur.” (NASA Report, p. 17) The report also states that NASA’s analysis was time-limited and top-down, remarking “The Toyota Electronic Throttle Control (ETC) was far more complex than expected involving hundreds of thousands of lines of software code” and that this affected the quality of a planned peer review.

It’s stated that “Reported [Unintended Accelerations (UAs)] are rare events. Typically, the reporting of UAs is about 1/100,000 vehicles/year.” But there are millions of cars on the road, and so NHTSA has collected some “831 UA reports for Camry” alone. “Over one-half of the reported events described large (greater than 25 degrees) high-throttle opening UAs of unknown cause” (NASA Report, p. 14), the causes of which are never fully explained in these reports.

The NASA apparently identified some lesser firmware bugs themselves, saying “[our] logic model verifications identified a number of potential issues. All of these issues involved unrealistic timing delays in the multiprocessing, asynchronous software control flow.” (Appendix A, p. 11) NASA also spent time simulating possible race conditions due to worrisome “recursively nested interrupt masking” (pp, 44-46); note, though, that simulation success is not a sufficient proof of lack of races. As well, the NASA team seems to recommend “reducing the amount of global data” (p. 38) and eliminating “dead code” (p. 40).

Additionally, the redacted text in other parts of Appendix A seems to be obscuring that:

• “The standard gcc compiler version 4″ generated a redacted number of warnings (probably larger than 100) about the code, in 11 different warning categories. (p. 25)
• “Coverity version 4.2″ generated a redacted number of warnings (probably larger than 154) about the code, in 10 different warning categories. (p. 27)
• “Codesonar version 3.6p1″ generated a redacted number of warnings (probably larger than 136) about the code, in 10 different warning categories.
• “Uno version 2.12″ generated a redacted number of warnings (probably larger than 72) about the code, in 9 different warning categories.
• The code contained at least 347 deviations from a subset of 14 of the MISRA-C rules.
• The code contained at least 243 violations of a subset of 9 of the 10 “Power of 10–Rules for Developing Safety Critical Code,” which was published in IEEE Computer in 2006 by NASA team member Gerard Holzmann.

It looks to me like Figure 6.2.3-1 of the NASA Report (p. 30) shows that UA complaints filed with NHTSA increased in the year of introduction of electronic throttle control for the vast majority of Toyota, Scion, and Lexus models–and that complaint counts have remained higher but generally declined over time since those transitions years. Such a complaint data pattern is perhaps consistent with firmware bugs. (Note to NHTSA: It would be helpful to see this same chart normalized by number of vehicles sold by model year and with the rows sorted by the year of ETC introduction. It would also be nice to see a chart of ETCS-i firmware versions and updates, which vehicles they apply to, and the dates on which each was put into new production vehicles or distributed through dealers.)

Final Thoughts

I am not privy to all of the facts considered by the NHTSA or NASA review teams and thus cannot say if I agree or disagree with their overall conclusion that embedded software bugs are not to blame for reports of unintended acceleration in Toyota vehicles. How about you? If you’ve spotted something I missed in the reports from NHTSA or NASA, please send me an e-mail or leave a comment below. Let’s keep the conversation going.