Archive for November, 2009

Embedded Programmers Worldwide Earn Failing Grades in C and C++

Tuesday, November 24th, 2009 Michael Barr

In industry surveys, over 80% of embedded software developers report using C or C++ as their primary programming language. Yet as a group, these programmers earned a failing grade on a multiple-choice quiz testing firmware-related C programming skills. A scary result, considering that embedded software inside medical devices, industrial controls, anti-lock brakes, and cockpits place human lives at risk every day.

In a February 2008 blog post, I examined the first few hundred results from the “Embedded C Quiz” on the Netrino website. That analysis compared the performance of programmers in the U.S. and India with the rest of the world (the only three data sets large enough for meaningful analysis). I concluded that the average embedded programmers in the U.S. and India don’t know C very well, but do know it better than programmers in the rest of the world.

Two years now since launching the quiz, we have collected thousands of data points, so it’s time for an update on programmer performance. In total, 3,870 programmers have taken the short 10-question multiple-choice C skills test. A few (a bit less than 3%) didn’t answer all of the questions; the analysis below is based on just the 3,755 completed quizzes. (Note that each website user can only take the quiz once.)

Across all countries, the mean result was 60.8%–a grade of ‘D-‘ at best. That is to say that the average embedded programmer answered just 6 out of 10 multiple-choice questions correctly. A rather scary fact, given that C is the language of choice for most embedded projects and that C++ is even harder to master.

Programmers in the United States scored slightly above average. But they still earned a failing grade of 61.8%. Programmers in India scored slightly below the worldwide average, at 58.9%. Together, programmers from these two large English-speaking countries accounted for the majority of all quiz takers.

The number of completed quizzes, mean scores, and standard deviations for all countries with more than 20 completed quizzes are shown in the table below, sorted by average score. In general, programmers from European countries scored best.

Country Completed Mean Std Dev
Poland 23 68.7 19.2
Sweden 26 67.7 15.8
Australia 45 67.3 22.3
Germany 57 67.2 17.2
France 35 66.9 24.0
United Kingdom 109 66.1 22.8
Spain 24 65.0 18.3
Canada 114 64.5 19.3
China 51 64.1 23.4
Israel 22 62.3 21.7
United States 1346 61.8 20.4
Egypt 28 59.3 22.8
India 1288 58.9 22.4
Romania 45 58.9 23.0
Singapore 24 58.3 20.1
Italy 44 56.4 20.8
Turkey 57 55.6 23.3
Brazil 47 55.1 24.1
Pakistan 25 44.0 21.7

How are your embedded C programming skills? Test them by taking the Embedded C Quiz yourself now at http://www.netrino.com/Embedded-Systems/Embedded-C-Quiz?

P.S.  We recently launched an Embedded C++ Quiz and the results so far look downright abysmal.  I’ll write something about that in a future post.  Do you have a few minutes to take that one too?

Breathalyzer Source Code Analysis

Thursday, November 5th, 2009 Michael Barr

Firmware bugs seem to be everywhere these days. So much so that firmware source code analysis is even entering the courtroom in criminal cases involving data collection devices with software inside. Consider the precedent-setting case of the Alcotest 7110. After a two-year legal fight, seven defendants in New Jersey DUI cases successfully won the right to have their experts review the source code for the Alcotest firmware.

The state and the defendants both ultimately produced expert reports evaluating the quality of the firmware source code. Though each side’s experts reached divergent opinions as to the overall code quality, several facts seem to have emerged as a result of the analysis:

– Of the available 12-bits of A/D precision, just 4-bits (most-significant) are used in the actual calculation. This sorts each raw blood-alcohol reading into one of 16 buckets. (I wonder how they biased the rounding on that.)
– Out of range A/D readings are forced to the high or low limit. This must happen with at least 32 consecutive readings before any flags are raised.
– There is no feedback mechanism for the software to ensure that actuated devices, such as an air pump and infrared sensor, are actually on or off when they are supposed to be.
– The software first averages the initial two readings. Then it averages the third reading with that average. Then the fourth reading is averaged in, etc. No comments or documentation explains the use of this formula, which causes the final reading to have a weight of 0.5 in the final value and the one before that to have a weight of 0.25, etc.
– Out of range averages are forced to the high or low limit too.
– Static analysis with lint produced over 19,000 warnings about the code (that’s about three errors for ever five lines of source code).

What would you infer about the reliability of a defendant’s blood-alcohol level if you were on that jury? If you’re so inclined you can read the full expert reports for yourself: at defendants and state.

This Code Stinks! The Worst Embedded Code Ever

Thursday, November 5th, 2009 Michael Barr

At the Embedded Systems Conference Boston in September, I gave a popular ESC Theater talk titled “This Code Stinks! The Worst Embedded Code Ever” that used lousy code from real products as a teaching tool. The example code was gathered by a number of engineers from a broad swath of companies over several years. (Minor details, including variable names and function names, were changed as needed to hide the specifics of applications, companies, or programmers.)

Here’s just one example of the bad code in that presentation:

y = (x + 305) / 146097 * 400 + (x + 305) % 146097 / 36524 * 100 + (x + 305) % 146097 % 36524 / 1461 * 4 + (x + 305) % 146097 % 36524 % 1461 / 365;

I don’t know if the above snippet contains any bugs, as most of the other examples were found to. And that’s a problem. Where are we supposed to begin an analysis of the above? What is this code supposed to do when it works? What range of input values is appropriate to test? What are the correct output values for a given input? Is this code responsible for handling out of range inputs gracefully? In the original listing, there were no comments on or around this line to help.

I eventually learned that this code computes the year, with accounting for extra days in leap years, given the number of days since a known reference date (e.g., January 1, 1970). But I note that we still don’t know if it works in all cases; despite it being present in an FDA-regulated medical device. I note too that the Microsoft Zune Bug was buried in a much better formatted snippet of code that performed a very similar calculation.

Here’s another example, this time in C++, with the bug finding left as an exercise for the reader:

bool Probe::getParam(uint32_t param_id, int32_t idx)
{
int32_t val = 0;
int32_t ret = 0;

ret = m_pParam->readParam(param_id, idx, &val);

if (!ret)
{
logMsg(“attempt to read parameter failed\n”);
exit(1);
}
else …

Hint: This code was embedded in a piece of factory automation equipment.

I’ve placed the full set of slides online at http://bit.ly/badcode.

TechBites – A Collaborative Community for Engineers

Tuesday, November 3rd, 2009 Michael Barr

Since at least as far back as January 1999, when I wrote a white paper describing the idea to some colleagues, I’ve wanted to participate in a collaborative community for embedded systems design engineers. I felt then, as I do now, that the collective wisdom spread across the community is largely untapped. This feeling was confirmed when I served as editor-in-chief of Embedded Systems Programming magazine, where I tried to turn bits and pieces of your knowledge into articles.

But what if there were a way to reap the benefits of both any willing contributor and an editor able to clean the conversation up, keep things organized, and move threads along? Finally, there is.

A new site launched today called TechBites (http://www.techbites.com) aims to provide a Web 2.0 collaborative community for engineers, where every site member can contribute to the conversation. Yet the site has an impressive list of industry experts already signed up as editors for specific topic-focused communities (e.g., I’ll be helping out with the RTOS and Embedded Medical communities).

At TechBites.com you can:

* Read content including Blogs, Articles, Reviews, and Tech-Tips.
* Create your own content and participate in group discussions.
* Connect with your peers and industry experts.

Join now (it’s free) to take advantage of these opportunities and connect with your friends!

Participate in one or more of the managed communities or start your own user community around a product or technology you know well.