embedded software boot camp

Is Toyota’s Accelerator Problem Caused by Embedded Software Bugs?

Thursday, January 28th, 2010 by Michael Barr

Last month I received an interesting e-mail in response to a column I wrote for Embedded Systems Design called The Lawyers are Coming! My column was partly about the poor state of embedded software quality across all industries, and my correspondent was writing to say my observations were accurate from his perch within the automotive industry. Included in his e-mail was this interesting tidbit:

I read something about the big Toyota recall being related to floor mats interfering with the accelerator, but I was told that the problem appears to be software (firmware) for the control-by-wire pedal.  Me thinks somebody probably forgot to check ranges, overflows, or stability properly when implementing the “algorithm”.

As background for those of you who have been working in SCIFs or other labs, the “big Toyota recall” was first announced in September 2009. It was said to concern removable floor mats causing the accelerator pedal to be pressed down. Some 3.8 million Toyota and Lexus vehicles were involved and owners were told to remove floor mats immediately.

This week several related major news events have transpired, including:

But none of the articles I’ve read have talked about software being a cause. And it’s not clear if the affected models are drive-by-wire. However, at least one article I read yesterday suggested that one fix being worked on is a software interlock to ensure that if both the brake and the gas pedal are depressed, the brake will override the accelerator. On the one hand, that seems to mean that software is already in the middle; on the other, I would be extremely surprised to learn that such an interlock wasn’t already present in a drive-by-wire system.

So what’s the story? Are embedded software bugs to blame for this massive recall? Do you know? Have you found any helpful articles pointing at software problems? Please share what you know in the comments below, or e-mail me privately.

Tags: , , ,

14 Responses to “Is Toyota’s Accelerator Problem Caused by Embedded Software Bugs?”

  1. Michael Barr says:

    Here's another blogger asking a similar question:
    http://www.dvorak.org/blog/2010/01/28/toyota-recall-is-it-the-computer-not-the-gas-pedal/

    And here’s a longer story about accelerator design in general, which includes some comments on the role of software as well as safety interlocks:
    http://www.ktbb.com/thecarshow/2010/01/29/toyotas-big-recall/

  2. Michael Barr says:

    Here’s a November 29, 2009 L.A. Times article that correlates the increase in “sudden acceleration” reports and the switch to “drive-by-wire” systems at Toyota:

    http://articles.latimes.com/2009/nov/29/business/la-fi-toyota-throttle29-2009nov29

    In addition, there’s a “teardown” of the accelerator pedal’s mechanical parts at

    http://www.thetruthaboutcars.com

  3. Michael Barr says:

    Finally this grim detail from the Washington Post's Friday edition:

    In the accident that has drawn perhaps the most publicity, a 2009 Lexus ES 350 raced through San Diego, weaving at 120 miles an hour through rush-hour freeway traffic. Veteran California Highway Patrol officer Mark Saylor was at the wheel, with his wife, teenage daughter and brother-in-law aboard. “We're in trouble. . . . There’s no brakes,” Saylor’s brother-in-law told a police dispatcher over a cellphone. As they approached an intersection, and the end of the road, the passengers could be heard urging each other to pray. All four died.

    Afterward, investigators said that it appeared the brakes had been applied for so long that the brake pads melted, according to a report by the National Highway Traffic Safety Administration.

  4. Kyle Bostian says:

    I assume that the CHP officer didn't switch into Neutral for fear of blowing up the engine. In this day and age, though, I would expect the ECU to shut the fuel injectors off if the rpm gets too high. Naturally I'm not inclined to experiment with this on my own cars, but does anyone know the answer?

  5. Prasanna says:

    http://www.cnn.com/2010/WORLD/asiapcf/02/04/japan.prius.complaints/index.html?hpt=T1

    Anti-lock breaking software issue in Toyota Prius 2010. I have worked (not for Toyota) on ABS and VSC embedded software . It is very complex.

  6. Anonymous says:

    I have been pondering the same question: do Toyota vehicles have a throttle-kill function activated by brake application? If not, they are clearly criminally defective by definition, no matter what the factual answers may be to the alleged runaway engines. Most web commentary, particularly the claim that such a function WILL be standard for them starting with 2011 models, would lead one to believe that they DO NOT now have such a function.

    You asked for information on this topic from readers. I have nothing first-hand, but did come across the following Toyota statement in several places, including CNBC from which this particular version was taken:

    UPDATE: CNBC has just received the following statement from Toyota: “After many years of exhaustive testing—by us and other outside agencies—we have found no evidence of a problem with our electronic throttle control system that could have caused unwanted acceleration. Our vehicles go through extensive electromagnetic radiation testing dynamically. We have our own test facility in Japan, we are also building one in Ann Arbor. The testing examines microwave radiation and every other type of magnetic wave and we have never been able to force our systems to fail through any of the tests that are done on them. There are many redundancies and fail safes that are built into our system. If the accelerator pedal and the throttle on the engine don’t match in their communication to each other the throttle returns to an idle position.”

    It is the last sentence that is of interest here; the rest is meaningless. I haven't had a Toyota since they stopped selling the AWD/manual gearbox combination in any but freak vehicles, but any owner could run the test in a few minutes to see if the function is indeed incorporated, (although that wouldn't prove that it actually works when needed). If the brake pads really melted in the California Cop Case, then it would be rather unambiguously established that the override function, if present, failed utterly, and also – albeit somewhat less clearly – that the throttle was held open without operator command.For the longer run: does this current kerfuffle, with all its murkiness, indicate that it's about time to mandate "black box" (think: flight data recorder) functions in automotive electronic control units? One would need to remain mindful that such recorders can also fail.

    Web note: you invite responses “privately, by email” yet I could find no email address to use for such a submission. And I feel forced to chose "anonymous" as a comment-poster, since it doesn't seem wise to plaster email addresses about in public.t.r. jackson

  7. Michael Barr says:

    Thanks for your thoughtful comments. Anyone can contact me via my Google Profile at
    http://www.michaelbarr.info

    Or search the web a little and you're bound to find my e-mail address at the end of an article on embedded.com or elsewhere.

  8. Michael Barr says:

    Regarding the engine blowing up if you put it into neutral, here's an excerpt from an interview with one of the Car Talk guys:

    Q: They were saying if you have that stuck accelerator, you should throw your car into neutral and turn it off?

    A: Not turn it off! I think there has been a lot of misinformation out there — we have a thing on our Web site that tells people what to do, cartalk.com. But do not turn the key off, under any circumstance. You’re going to lose your power brakes, you’re going to lose your power steering.So, the secret is, if the car is running away, move the shifter from drive into neutral — it’s one step away — and the engine will race like crazy. It will be scary because the engine will sound like it’s going to blow up, but you will not harm the engine, and engines will not blow up if they are over-revved now.

  9. Anonymous says:

    I’ve been an automotive control systems engineer for several years in the area of engine and transmission control systems. My opinion is that there is a very high likelyhood that this unintended acceleration issue is caused by an intermittent software and/or computer hardware issue…maybe both interacting with each other.

    Intermittent issues are always difficult to diagnose, even more so when computer hardware/software is involved. And with this system, the main component of the control system is computer hardware/software, which is complex to say the least. I can guarantee that Toyota engineers suspect the computer hardware/software to be the cause of the problem. Duplicating an issue like this in an environment where instrumentation can be used to determine the root cause is a very difficult thing to do. Sometimes the instrumentation itself can change the system enough to eliminate duplication of the problem.

    So, Toyota engineers have either found the root cause of the software/hardware issue or they haven't been able to duplicate it in an engineering environment yet, and are still trying. If the root cause has been found and corrected, they'll most likely slip a software change in when the vehicles while they are being updated with the so-called "pedal" fix.

    Hopefully our government is paying attention here because the public really should be aware of the root cause of a computer software/hardware issue like this. Obviously, this will look bad on the Toyota engineering that's supposedly so great, and that's what will keep them from acknowledging this.

    And BTW, this isn't something that an external lab could test and determine the cause of. With software, the source code is needed, in addition to an instrumented control computer and associated software debug equipment. Basically, you would need to be in their software development lab to know what was really going on.

  10. bandit says:

    This article does not do a very good job about the Toyota problems (read the comments):
    http://www.techeye.net/chips/electronic-tin-whiskers-may-be-behind-toyota-recalls

    This is the paper it refers to (which tells a different story):
    http://www.nutwooduk.co.uk/downloads/Toyota.doc

    which tells a different story than the article.

  11. Joe Richard says:

    I just brought my 2007 Camry Hybrid in for the “Accelerator Recall”. The service tech told me that, although I didn’t need the redesigned accelerator pedal, they were installing a reinforced back – AND they were reprogramming the software to include the mod where the engine is put in” neutral” when the brakes are applied. Also, this software fix was already in the 2010 Prius.
    Can anyone verify this?

  12. Mark says:

    Here is a reprot I found online that would explain a lot of the problem they are having.

    http://www.safetyresearch.net/Library/Preliminary_Report022110.pdf

    The link didn’t show up on my first post.

Leave a Reply to Michael Barr