Last month I received an interesting e-mail in response to a column I wrote for Embedded Systems Design called The Lawyers are Coming! My column was partly about the poor state of embedded software quality across all industries, and my correspondent was writing to say my observations were accurate from his perch within the automotive industry. Included in his e-mail was this interesting tidbit:
I read something about the big Toyota recall being related to floor mats interfering with the accelerator, but I was told that the problem appears to be software (firmware) for the control-by-wire pedal. Me thinks somebody probably forgot to check ranges, overflows, or stability properly when implementing the “algorithm”.
As background for those of you who have been working in SCIFs or other labs, the “big Toyota recall” was first announced in September 2009. It was said to concern removable floor mats causing the accelerator pedal to be pressed down. Some 3.8 million Toyota and Lexus vehicles were involved and owners were told to remove floor mats immediately.
This week several related major news events have transpired, including:
- Toyota recalled millions of additional vehicles in the U.S.,
- Under pressure from the U.S. NHTSA, Toyota halted production and sales of eight models,
- Avis, Hertz, and Enterprise pulled affected Toyota models from their rental fleets,
- Toyota’s recall spread to Europe and China, and
- Ford stopped production of a full-size commercial vehicle after discovering that the gas pedal came from the supplier involved in the Toyota recall.
But none of the articles I’ve read have talked about software being a cause. And it’s not clear if the affected models are drive-by-wire. However, at least one article I read yesterday suggested that one fix being worked on is a software interlock to ensure that if both the brake and the gas pedal are depressed, the brake will override the accelerator. On the one hand, that seems to mean that software is already in the middle; on the other, I would be extremely surprised to learn that such an interlock wasn’t already present in a drive-by-wire system.
So what’s the story? Are embedded software bugs to blame for this massive recall? Do you know? Have you found any helpful articles pointing at software problems? Please share what you know in the comments below, or e-mail me privately.
Tags: embedded, programming, safety, trends
Here's another blogger asking a similar question:
http://www.dvorak.org/blog/2010/01/28/toyota-recall-is-it-the-computer-not-the-gas-pedal/
And here’s a longer story about accelerator design in general, which includes some comments on the role of software as well as safety interlocks:
http://www.ktbb.com/thecarshow/2010/01/29/toyotas-big-recall/
Here’s a November 29, 2009 L.A. Times article that correlates the increase in “sudden acceleration” reports and the switch to “drive-by-wire” systems at Toyota:
http://articles.latimes.com/2009/nov/29/business/la-fi-toyota-throttle29-2009nov29
In addition, there’s a “teardown” of the accelerator pedal’s mechanical parts at
http://www.thetruthaboutcars.com
Finally this grim detail from the Washington Post's Friday edition:
There’s a report on this particular accident, which seems to implicate a floor mat issue, at http://www-odi.nhtsa.dot.gov/acms/docservlet/Artemis/Public/Pursuits/2007/EA/INFR-EA07010-28888.pdf
I assume that the CHP officer didn't switch into Neutral for fear of blowing up the engine. In this day and age, though, I would expect the ECU to shut the fuel injectors off if the rpm gets too high. Naturally I'm not inclined to experiment with this on my own cars, but does anyone know the answer?
http://www.cnn.com/2010/WORLD/asiapcf/02/04/japan.prius.complaints/index.html?hpt=T1
Anti-lock breaking software issue in Toyota Prius 2010. I have worked (not for Toyota) on ABS and VSC embedded software . It is very complex.
Interesting article about the s/w process at Toyota:
http://www.bestbrains.dk/Blog/2009/04/22/LeanStudyTour2009Day2FeelingPrivileged.aspx
I have been pondering the same question: do Toyota vehicles have a throttle-kill function activated by brake application? If not, they are clearly criminally defective by definition, no matter what the factual answers may be to the alleged runaway engines. Most web commentary, particularly the claim that such a function WILL be standard for them starting with 2011 models, would lead one to believe that they DO NOT now have such a function.
You asked for information on this topic from readers. I have nothing first-hand, but did come across the following Toyota statement in several places, including CNBC from which this particular version was taken:
It is the last sentence that is of interest here; the rest is meaningless. I haven't had a Toyota since they stopped selling the AWD/manual gearbox combination in any but freak vehicles, but any owner could run the test in a few minutes to see if the function is indeed incorporated, (although that wouldn't prove that it actually works when needed). If the brake pads really melted in the California Cop Case, then it would be rather unambiguously established that the override function, if present, failed utterly, and also – albeit somewhat less clearly – that the throttle was held open without operator command.For the longer run: does this current kerfuffle, with all its murkiness, indicate that it's about time to mandate "black box" (think: flight data recorder) functions in automotive electronic control units? One would need to remain mindful that such recorders can also fail.
Web note: you invite responses “privately, by email” yet I could find no email address to use for such a submission. And I feel forced to chose "anonymous" as a comment-poster, since it doesn't seem wise to plaster email addresses about in public.t.r. jackson
Thanks for your thoughtful comments. Anyone can contact me via my Google Profile at
http://www.michaelbarr.info
Or search the web a little and you're bound to find my e-mail address at the end of an article on embedded.com or elsewhere.
Regarding the engine blowing up if you put it into neutral, here's an excerpt from an interview with one of the Car Talk guys:
I’ve been an automotive control systems engineer for several years in the area of engine and transmission control systems. My opinion is that there is a very high likelyhood that this unintended acceleration issue is caused by an intermittent software and/or computer hardware issue…maybe both interacting with each other.
Intermittent issues are always difficult to diagnose, even more so when computer hardware/software is involved. And with this system, the main component of the control system is computer hardware/software, which is complex to say the least. I can guarantee that Toyota engineers suspect the computer hardware/software to be the cause of the problem. Duplicating an issue like this in an environment where instrumentation can be used to determine the root cause is a very difficult thing to do. Sometimes the instrumentation itself can change the system enough to eliminate duplication of the problem.
So, Toyota engineers have either found the root cause of the software/hardware issue or they haven't been able to duplicate it in an engineering environment yet, and are still trying. If the root cause has been found and corrected, they'll most likely slip a software change in when the vehicles while they are being updated with the so-called "pedal" fix.
Hopefully our government is paying attention here because the public really should be aware of the root cause of a computer software/hardware issue like this. Obviously, this will look bad on the Toyota engineering that's supposedly so great, and that's what will keep them from acknowledging this.
And BTW, this isn't something that an external lab could test and determine the cause of. With software, the source code is needed, in addition to an instrumented control computer and associated software debug equipment. Basically, you would need to be in their software development lab to know what was really going on.
This article does not do a very good job about the Toyota problems (read the comments):
http://www.techeye.net/chips/electronic-tin-whiskers-may-be-behind-toyota-recalls
This is the paper it refers to (which tells a different story):
http://www.nutwooduk.co.uk/downloads/Toyota.doc
which tells a different story than the article.
I just brought my 2007 Camry Hybrid in for the “Accelerator Recall”. The service tech told me that, although I didn’t need the redesigned accelerator pedal, they were installing a reinforced back – AND they were reprogramming the software to include the mod where the engine is put in” neutral” when the brakes are applied. Also, this software fix was already in the 2010 Prius.
Can anyone verify this?
Here is a reprot I found online that would explain a lot of the problem they are having.
http://www.safetyresearch.net/Library/Preliminary_Report022110.pdf
The link didn’t show up on my first post.