Posts Tagged ‘embedded’

« Older Entries
Newer Entries »

An Update on Toyota and Unintended Acceleration

Saturday, October 26th, 2013 Michael Barr

In early 2011, I wrote a couple of blog posts (here and here) as well as a later article (here) describing my initial thoughts on skimming NASA’s official report on its analysis of Toyota’s electronic throttle control system. Half a year later, I was contacted and retained by attorneys for numerous parties involved in suing Toyota for personal injuries and economic losses stemming from incidents of unintended acceleration. As a result, I got to look at Toyota’s engine source code directly and judge for myself.

From January 2012, I’ve led a team of seven experienced engineers, including three others from Barr Group, in reviewing Toyota’s electronic throttle and some other source code as well as related documents, in a secure room near my home in Maryland. This work proceeded in two rounds, with a first round of expert reports and depositions issued in July 2012 that led to a billion-dollar economic loss settlement as well as an undisclosed settlement of the first personal injury case set for trial in U.S. Federal Court. The second round began with an over 750 page formal written expert report by me in April 2013 and culminated this week in an Oklahoma jury’s decision that the multiple defects in Toyota’s engine software directly caused a September 2007 single vehicle crash that injured the driver and killed her passenger.

It is significant that this was the first and only jury so far to hear any opinions about Toyota’s software defects. Earlier cases either predated our source code access, applied a non-software theory, or was settled by Toyota for an undisclosed sum.

In our analysis of Toyota’s source code, we built upon the prior analysis by NASA. First, we looked more closely at more lines of the source code for more vehicles for more man months. And we also did a lot of things that NASA didn’t have time to do, including reviewing Toyota’s operating system’s internals, reviewing the source code for Toyota’s “monitor CPU”, performing an independent worst-case stack depth analysis, running portions of the main CPU software including the RTOS in a processor simulator, and demonstrating–in 2005 and 2008 Toyota Camry vehicles–a link between loss of throttle control and the numerous defects we found in the software.

In a nutshell, the team led by Barr Group found what the NASA team sought but couldn’t find: “a systematic software malfunction in the Main CPU that opens the throttle without operator action and continues to properly control fuel injection and ignition” that is not reliably detected by any fail-safe. To be clear, NASA never concluded software wasn’t at least one of the causes of Toyota’s high complaint rate for unintended acceleration; they just said they weren’t able to find the specific software defect(s) that caused unintended acceleration. We did.

Now it’s your turn to judge for yourself. Though I don’t think you can find my expert report outside the Court system, here are links to the trial transcript of my expert testimony to the Oklahoma jury and a (redacted) copy of the slides I shared with the jury in Bookout, et.al. v. Toyota.

Note that the jury in Oklahoma found that Toyota owed each victim $1.5 million in compensatory damages and also found that Toyota acted with “reckless disregard”. The latter legal standard meant the jury was headed toward deliberations on additional punitive damages when Toyota called the plaintiffs to settle (for yet another undisclosed amount). It has been reported that an additional 400+ personal injury cases are still working their way through various courts.

Related Stories

Updates

On December 13, 2013, Toyota settled the case that was set for the next trial, in West Virginia in January 2014, and announced an “intensive” settlement process to try to resolve approximately 300 of the remaining personal injury case, which are consolidated in U.S. and California courts.

Toyota continues to publicly deny there is a problem and seems to have no plans to address the unsafe design and inadequate fail safes in its drive-by-wire vehicles–the electronics and software design of which is similar in most of the Toyota and Lexus (and possibly Scion) vehicles manufactured over at least about the last ten model years. Meanwhile, incidents of unintended acceleration continue to be reported in these vehicles (see also the NHTSA complaint database) and these new incidents, when injuries are severe, continue to result in new personal injury lawsuits against Toyota.

In March 2014, the U.S. Department of Justice announced a $1.2 billion settlement in a criminal case against Toyota. As part of that settlement, Toyota admitted to past lying to NHTSA, Congress, and the public about unintended acceleration and also to putting its brand before public safety. Yet Toyota still has made no safety recalls for the defective engine software.

On April 1, 2014, I gave a keynote speech at the EE Live conference, which touched on the Toyota litigation in the context of lethal embedded software failures of the past and the coming era of self-driving vehicles. The slides from that presentation are available for download at http://www.barrgroup.com/killer-apps/.

On September 18, 2014, Professor Phil Koopman, of Carnegie Mellon University, presented a talk about his public findings in these Toyota cases entitled “A Case Study of Toyota Unintended Acceleration and Software Safety“.

On October 30, 2014, Italian computer scientist Roberto Bagnara presented a talk entitled “On the Toyota UA Case
and the Redefinition of Product Liability for Embedded Software” at the 12th Workshop on Automotive Software & Systems, in Milan.

Tags: , , , , , , , , , , , ,
Posted in Firmware Bugs | 12 Comments »

Intellectual Property Protections for Embedded Software: A Primer

Tuesday, June 11th, 2013 Michael Barr

This article is now located here: https://experts.barrgroup.com/articles/intellectual-property-protections-embedded-software-primer

Tags: , , , , , ,
Posted in Intellectual Property | 1 Comment »

Dead Code, the Law, and Unintended Consequences

Wednesday, February 6th, 2013 Michael Barr

This article is now located here: https://experts.barrgroup.com/articles/dead-code-law-and-unintended-consequences

Tags: , , , , , ,
Posted in Intellectual Property | 5 Comments »

Introducing Barr Group

Wednesday, December 26th, 2012 Michael Barr

In the ten months since forming Barr Group, I have received many questions about the new company. As we enter the new year, I thought it a good time to use this blog post to answer the most frequently asked questions, such as:

  • What does Barr Group do?
  • Who are Barr Group’s clients?
  • How is Barr Group different than my former company?
  • Who is our CEO and what skills does he bring?
  • What is my role in Barr Group?

If I had to describe Barr Group (http://www.barrgroup.com) in a single sentence, I would say that “Barr Group helps companies that design embedded systems make their products more reliable and more secure.” We do sell a few small items–such as the Embedded C Coding Standard book and Embedded Software Training in a Box kit–but our company is not really about our own products. Rather, we achieve our mission of improving embedded systems reliability and security by delivering business-to-business services of primarily three types: (1) consulting, (2) training, and (3) engineering.

Barr Group serves clients from small startups to well-known Fortune 100 companies that make embedded systems used in a wide range of industries. Representative clients include: Adtran, Medtronic, Renesas, TI, and Xerox. Barr Group’s staff has expertise and experience in the design of medical devices, industrial controls, consumer electronics, telecommunications, transportation equipment, smart grid technologies, and many other types of embedded systems.

Barr Group’s consulting services are sold to engineering managers, engineering directors, or vice presidents of engineering. Typical consulting engagements are short-duration/high-value projects aimed at answering strategically important questions related to embedded systems architecture and embedded software development processes. For example, in the area of architecture for reliability and security we offer services specifically in the following areas: system design review, software design review, system (re)architecture, software (re)architecture, source code review, cost reduction, reverse engineering, and security analysis. Of course, we often address more targeted issues as well, including embedded software development process improvements. Because we are unaffiliated with any processor, RTOS, or tool vendor, all of our advice is independent of any external influence; we aim only to find the best path forward for our clients, favoring alternatives that require only 20% of the effort to achieve 80% of the available benefits.

Barr Group’s training courses are designed to raise the quality of engineers and engineering teams and many of them include hands-on programming exercises. We teach these courses both privately and publicly. Private training is held at the client’s office and every engineer in attendance works for the client. By contrast, any individual or small group of engineers can purchase a ticket to our public training courses. Our Spring 2013 training calendar includes four week-long hands-on courses: Embedded Software Boot Camp (Maryland), Embedded Security Boot Camp (Silicon Valley), Embedded Android Boot Camp (Maryland), and Agile and Test-Driven Embedded Development (Florida).

Barr Group’s engineering design services include outsourced development of: electronics (including FPGA and PCB design); device drivers for operating systems such as MicroC/OS, VxWorks, Windows, Linux, Android, and others; embedded software; mechanical enclosures; and everything in between. In one representative project that was recently completed, a cross-functional team of talented Barr Group engineers worked together to perform all of the mechanical, electrical, software, reliability, and security engineering for a long-lived high voltage electrical switching system for deployment in a modern “smart grid” electrical distribution network.

In relation to my earlier company, which was founded in 1999, the principal difference in all of the above is Barr Group’s additional focus on embedded systems security, compared with reliability alone. Like Netrino, some members of our engineering staff also work as expert witnesses in complex technical litigation–with a range of cases involving allegations of product liability, patent infringement, and source code copyright infringement.

Finally, under the new leadership of seasoned technology executive (and fellow electrical engineer) Andrew Girson, Barr Group has added a suite of Engineer-Centric Market ResearchTM services, which assist IC makers, RTOS vendors, and other companies serving the embedded systems design community improve their products and marketing by better understanding the mind of the engineer. These services have been specifically enabled by the combination of Mr. Girson’s skills and expertise in strategic technical marketing with Barr Group’s extensive contacts in the embedded systems industry, including the over 20,000 Firmware Update newsletter subscribers.

My role in Barr Group is chief technology officer. The switch from my role as president of the old company to CTO of the new company has freed up considerably more of my time to work on engineering and expert witness projects. The extra time allows me to focus on sharing my technical expertise with as many clients as possible while also developing the other engineers who work individuals projects.

All in all, it has been great fun (if a lot of work) launching the new company this year. I look forward to another successful year for Barr Group in 2013. Please don’t hesitate to contact me or call us at (866) 653-6233 if we can be of assistance to your company. And happy new year!

Tags: , , , , , , , , , , ,
Posted in Uncategorized | No Comments »

How to Combine Volatile with Struct

Friday, November 9th, 2012 Michael Barr

C’s volatile keyword is a qualifier that can be used to declare a variable in such a way that the compiler will never optimize away any of the reads and writes. Though there are several important types of variables to declare volatile, this obscure keyword is especially valuable when you are interacting with hardware peripheral registers and such via memory-mapped I/O.

Sometimes a memory-mapped I/O device could be as simple as just having a single 8-bit control register at a fixed address. In that case, the placement of volatile should be to the left of the * operator in the declaration of the pointer to that address, as in:


uint8_t volatile * p_ledreg = 0x10000000;

In the above code, the variable p_legreg is a pointer to a volatile 8-bit unsigned register located at address 0x10000000.

However, it is far more common that memory-mapped peripherals have at least a half dozen registers. In this more complicated scenario, a C struct can be defined to encapsulate these registers as a set and a pointer to said data structure can be declared. Here’s an example of such a declaration that does not feature the volatile keyword at all:


typedef struct
{
uint8_t reg1;
uint8_t reg2;
uint8_t _reserved;
uint8_t reg3;

} mydevice_t;

mydevice_t * p_mydevice = 0x10000000;

In this scenario, there are three possible places for the volatile keyword. First, the first line could be modified so that the new type “mydevice_t” always contains the volatile keyword, as in:


typedef volatile struct

Or the last line could be modified so that the pointer “p_mydevice” is a pointer to a volatile mydevice_t:


mydevice_t volatile * p_mydevice = 0x10000000;

Note that the difference between these first two volatile placements is whether all instances of said struct are volatile or only the pointer’s instance is volatile. If there is only one instance of the struct in the whole program and it is the pointer p_mydevice, then that difference is immaterial.

Finally, the third option is to place one or more volatile keywords within the struct definition itself. With this placement, only the specific registers within the struct that are declared volatile will be treated, by the compiler, as subject to asynchronous change. Reads and writes from or to other, non-volatile-declared, registers in the struct may potentially be optimized away. Here’s an example:


typedef struct
{
uint8_t volatile reg1;
uint8_t volatile reg2;
uint8_t const _reserved;
uint8_t reg3;
}

mydevice_t * p_mydevice = 0x10000000;

Given that there are multiple choices for the placement of volatile, where is the best place to put the volatile keyword in practice? My preferred placement is typically in the pointer declaration. That way, all of the registers in the struct will be treated, by the compiler, as volatile and yet it is possible to have other (e.g. RAM-based shadows) instances of said struct that are not volatile because they are not actually hardware registers underneath.

Tags: , , ,
Posted in Coding Standards, Efficient C/C++ | 13 Comments »

« Older Entries
Newer Entries »