embedded software boot camp

An Update on Toyota and Unintended Acceleration

Saturday, October 26th, 2013 by Michael Barr

In early 2011, I wrote a couple of blog posts (here and here) as well as a later article (here) describing my initial thoughts on skimming NASA’s official report on its analysis of Toyota’s electronic throttle control system. Half a year later, I was contacted and retained by attorneys for numerous parties involved in suing Toyota for personal injuries and economic losses stemming from incidents of unintended acceleration. As a result, I got to look at Toyota’s engine source code directly and judge for myself.

From January 2012, I’ve led a team of seven experienced engineers, including three others from Barr Group, in reviewing Toyota’s electronic throttle and some other source code as well as related documents, in a secure room near my home in Maryland. This work proceeded in two rounds, with a first round of expert reports and depositions issued in July 2012 that led to a billion-dollar economic loss settlement as well as an undisclosed settlement of the first personal injury case set for trial in U.S. Federal Court. The second round began with an over 750 page formal written expert report by me in April 2013 and culminated this week in an Oklahoma jury’s decision that the multiple defects in Toyota’s engine software directly caused a September 2007 single vehicle crash that injured the driver and killed her passenger.

It is significant that this was the first and only jury so far to hear any opinions about Toyota’s software defects. Earlier cases either predated our source code access, applied a non-software theory, or was settled by Toyota for an undisclosed sum.

In our analysis of Toyota’s source code, we built upon the prior analysis by NASA. First, we looked more closely at more lines of the source code for more vehicles for more man months. And we also did a lot of things that NASA didn’t have time to do, including reviewing Toyota’s operating system’s internals, reviewing the source code for Toyota’s “monitor CPU”, performing an independent worst-case stack depth analysis, running portions of the main CPU software including the RTOS in a processor simulator, and demonstrating–in 2005 and 2008 Toyota Camry vehicles–a link between loss of throttle control and the numerous defects we found in the software.

In a nutshell, the team led by Barr Group found what the NASA team sought but couldn’t find: “a systematic software malfunction in the Main CPU that opens the throttle without operator action and continues to properly control fuel injection and ignition” that is not reliably detected by any fail-safe. To be clear, NASA never concluded software wasn’t at least one of the causes of Toyota’s high complaint rate for unintended acceleration; they just said they weren’t able to find the specific software defect(s) that caused unintended acceleration. We did.

Now it’s your turn to judge for yourself. Though I don’t think you can find my expert report outside the Court system, here are links to the trial transcript of my expert testimony to the Oklahoma jury and a (redacted) copy of the slides I shared with the jury in Bookout, et.al. v. Toyota.

Note that the jury in Oklahoma found that Toyota owed each victim $1.5 million in compensatory damages and also found that Toyota acted with “reckless disregard”. The latter legal standard meant the jury was headed toward deliberations on additional punitive damages when Toyota called the plaintiffs to settle (for yet another undisclosed amount). It has been reported that an additional 400+ personal injury cases are still working their way through various courts.

Related Stories

Updates

On December 13, 2013, Toyota settled the case that was set for the next trial, in West Virginia in January 2014, and announced an “intensive” settlement process to try to resolve approximately 300 of the remaining personal injury case, which are consolidated in U.S. and California courts.

Toyota continues to publicly deny there is a problem and seems to have no plans to address the unsafe design and inadequate fail safes in its drive-by-wire vehicles–the electronics and software design of which is similar in most of the Toyota and Lexus (and possibly Scion) vehicles manufactured over at least about the last ten model years. Meanwhile, incidents of unintended acceleration continue to be reported in these vehicles (see also the NHTSA complaint database) and these new incidents, when injuries are severe, continue to result in new personal injury lawsuits against Toyota.

In March 2014, the U.S. Department of Justice announced a $1.2 billion settlement in a criminal case against Toyota. As part of that settlement, Toyota admitted to past lying to NHTSA, Congress, and the public about unintended acceleration and also to putting its brand before public safety. Yet Toyota still has made no safety recalls for the defective engine software.

On April 1, 2014, I gave a keynote speech at the EE Live conference, which touched on the Toyota litigation in the context of lethal embedded software failures of the past and the coming era of self-driving vehicles. The slides from that presentation are available for download at http://www.barrgroup.com/killer-apps/.

On September 18, 2014, Professor Phil Koopman, of Carnegie Mellon University, presented a talk about his public findings in these Toyota cases entitled “A Case Study of Toyota Unintended Acceleration and Software Safety“.

On October 30, 2014, Italian computer scientist Roberto Bagnara presented a talk entitled “On the Toyota UA Case
and the Redefinition of Product Liability for Embedded Software
” at the 12th Workshop on Automotive Software & Systems, in Milan.

Tags: , , , , , , , , , , , ,

12 Responses to “An Update on Toyota and Unintended Acceleration”

  1. Miro Samek says:

    Hi Michael,

    Thank you for posting the link to your court deposition. I found it fascinating and couldn’t stop reading late into the night…

    There is no doubt in my mind that exposing the inadequacies in the Toyota firmware is a very important development for the whole embedded software profession.

    It is also interesting to see old mistakes repeated time and time again. For example a timed task degenerating into a kitchen sink.

    I also bet my shirt that there were no assertions in the Toyota firmware. Assertions in software work like fuses in electrical systems and beyond certain density of assertions in the code all failures (including hardware failures) manifest themselves as assertion violations. I’m sure that this could have saved the day (besides making software development so much faster).

    Anyway, there are tons of valuable lessons to learn here. From now on I will imagine that all my software is on trial…

    –Miro

    • Tom Betka says:

      Indeed! I’ve been reading about this story for the better part of the past four hours, and am now on page 64 of his 286-page deposition. FASCINATING stuff, and there’s a wealth of knowledge to be gained from simply reading through these documents.

      Absolutely incredible sequence of events–thanks so much to all who’ve written about this story, and obviously to Mr. Barr for the very interesting trial testimony!

  2. Dear Mr. Barr,

    Nicely done! I found your testimony very interesting, and while I am not a software expert, I can certainly verify the inability of Toyota vehicles to detect certain malfunctions in the electronic throttle controls. And few malfunctions are more apparent than tin whiskers growing inside the APP sensors!

    Since my 2010 testimony in the Washington Toyota hearings, I have learned much. Your testimony certainly adds to that knowledge and I am pleased that it has received much needed media attention.

    Maybe our paths will cross someday.

    DWG

  3. Mr. Barr,

    Wow! Finally, the official, reliable truth has emerged at long last. Thank you for your hard work.

    I am not an expert of any sort. I am just a Japanese to English translator. Through my work, I saw hundreds of Toyota’s internal documents that strongly suggested that UA was rooted in problems in the software (and also some in hardware) and that Toyota knew about these problems and was attempting to identify them and fix them. But meanwhile the company denied anything was wrong, including in the testimony before both the US House and Senate by Mr. Toyoda, Mr. T. Uchiyamada (the company’s current chairman), other executives, and two of Toyota’s engineers.

    I have recently published the internal documents in the public interest. You can find them through my Facebook page. Engineers might enjoy poring over them.

    Mr. Barr, it is a relief to see that the true state of the software is now fully understood. I hope and pray that the US government, including Congress and NHTSA, will now take action to help ensure public safety. I also wish you the greatest success in presenting your findings to the juries of many upcoming trials to help bring justice to consumers who relied on Toyota’s and NHTSA’s assurances all this time, but whose trust has been badly misplaced.

    Carry on!
    BZB

  4. Christenson says:

    What’s with all the stupid redactions about “Task X” (Kitchen sink task), Y millisecond tasks and Z second watchdogs? Not to mention the task count itself? Subtracting those details does nothing to alter the conclusions of the testimony, especially the parts about the technical debt, and doesn’t conceal anything from anyone that has even momentarily thought about the kind of software involved. It only proves that secrecy is a coverup strategy for Toyota! And TWO PAGES of source code being secret? Just petty….

    Me, I’m glad there’s a hard-wired, stop-whether-or-not-the-CPU-cooperates E-stop on the stuff I program.

    Can the report (in 800 pages of gory detail) be published and linked here, since it is now evidence in a court of law and a presumption of openness applies?

  5. John Wheeler says:

    Wow, the courtroom transcripts are a great read. I’m on page 98 right now, and I’ve been glued to my screen for the past hour and a half. The analogies with race conditions, overflows, and spaghetti code are all very good. You also allude the Toyota engineers didn’t have separation of concerns in the ‘kitchen sink’ task–It’s very scary.

    While reading this testimony and the egregious details, I can’t help but think one thing – the electronic throttle control shouldn’t be 100% software without some type of mechanical fault protection as a backup. I’ve read about the Therac 25 case, and the problem there was 100% software control of critical systems without hardware interlocks. My questions is: what has Toyota done since this aside from damage control and misguided firmware updates?

  6. Doug says:

    Up until now, Toyota has systematically suppressed from the public any mention of problems with their engine control software, by either settling cases out of court (which is effectively buying secrecy) or getting judges to allow outrageously restrictive secrecy rules in these court cases.

    You have to wonder why, this time, they allowed your testimony in the public record? This public testimony has told us all what is in your 800 page report that still is “secret” but not really any more.

    Toyota rolled the dice and this time they lost, and the loss could be enormous since there are still about 500 cases to be tried, and I would hope that you will testify in every one of those cases.

    So I have to speculate on what to expect in the next UA case to be tried: Toyota will mount a personal attack against you – after all, in America, if you don’t like the message then discredit the messenger.

  7. Parris Boyd says:

    Thanks for setting the record straight. It certainly needed to be done.

    There’s been a news blackout of your findings, anonymous personal attacks in comments on the Internet, and misleading reports from mainstream media. Bloomberg removed a comment I posted about Toyota’s recent software-related recalls (Prius, RAV4, Tacoma, Lexus RX350) and complaints of computer-related brake problems in Camry Hybrids lending support to your findings. It seems that your findings are being circulated primarily by bloggers, trade journals, and engineering conferences.

    Apparently, the Recall King is now offering another billion-dollar “settlement” in an effort to buy its way out of the federal criminal investigation regarding the way it handled complaints of sudden unintended acceleration. There’s no excuse for the way Toyota, the government. and mainstream media have behaved.
    Talk about a corporate-controlled police state…

    I’ve been blogging about Toyota for quite some time. My blog is titled “Beware of Toyota. Their next victim may be YOU…”

  8. Greg says:

    I read over the court presentation slides from Mr. Barr and it is clear the code is flawed.

    I have a 2006 Camry LE and there is no recall on it….how is this possible?

  9. Dana Tognini says:

    3 young men were killed yesterday while driving a 2013 Toyota Corolla driving on a windy road. It is difficult to attain high enough speeds on this curvy country road to cut a car in half. Here is a link to the article and video from a local news channel
    and you can see the car. I am not a physicist, but looking at the car it’s as if it was traveling more than phenomenal speed.
    http://sanfrancisco.cbslocal.com/2016/10/18/marin-county-fatal-crash-sir-francis-drake-blvd-lagunitas/

    I remembered hearing the UA issue and got on the computer and found your name as an expert that discovered this
    imbedded software flaw where other government agencies could not. What would be the next step if the parents wanted to look into the sudden acceleration as a possible cause? I know the family personally and I happen to be a court reporter. They are too bereft right now to think in legal terms or retention of evidence, so I am reaching out to you in this exploratory manner.
    Thank you,
    Dana Tognini

  10. Suyuan Wang says:

    Dear Mr. Barr,

    Thank you soooooo much for your hard-working reports.
    I’m from Taiwan. My husband’s 2010 Camry had an unintended acceleration on November 20th, 2016, when he went birding in a Metropolitan park.

    His car already turned into the parking section & about to get into the 6th parking space—then the car ran forward suddenly. He tried to step on the brake but it didn’t work at all– until the car hit the flower bed wall then it stopped with a crash front part of the Camry.

    He’s lucky without personal injury but scarred to death!

    Toyota told us last week there’s no any mistake information showed in their checking report.
    According to them, there’s no any brake record recorded on their reports.

    I showed them your reports on Oklahoma case. They said you were testing 2005-2008 Camry, they were different from my husband’s 2010 Camry. I knew they were fooling me. What would you suggest me to do?

  11. Paul Penrose says:

    Like Mr. Barr I am also an embedded software engineer. I have worked on pacemaker software for Medtronic (for which I am named in a patent involving the first RTOS in an embedded medical device) and other safety critical software for the likes of Guident (now Boston Scientific) and Lockheed Martin. Because of this experience I have a deep understanding of the issues involved in developing these kind of systems, especially the firmware. While I am a bit shocked at Toyota’s failure to use a certified RTOS and industry best development processes, I am not surprised. In my 35 years in this industry I have witnessed many companies and software engineers, with little or no experience, attempt to develop real time embedded systems; often with disastrous results. With the increasing use of microprocessors in our modern devices, there is a greater and greater need for embedded engineers. However it is a difficult specialty to master and most software engineers opt for something easier like phone or internet apps. I took the Embedded and Real-Time Systems Programming certification course from the University of Washington in 2006. Twenty five people started, but fifteen months later only five people passed; the rest dropped out because it was too difficult. This does not bode well for our future.

Leave a Reply