embedded software boot camp

The Sad State of Embedded Software Process

Wednesday, September 15th, 2010 by Michael Barr

Today VDC is sharing their 2010 Embedded System Engineering Survey Results.  I’m sad to say that the collective immaturity of the embedded software community continues to be on display.

Consider these two depressing statistics from the survey results:

  • Only 1 out of 5 embedded software developers are using a static analysis tool on their current project.  That’s about the same percent as last year, so it’s not even like the trend is positive.  I find this unbelievable.  There are at least a dozen great static analysis tools on the market.  One of them, PC-Lint, costs just $389.  Yet 4 out of 5 firmware developers don’t use any such bug-killing tool!?!
  • Over 40% of embedded software developers either report following “no methodology” (27%) on their current project or say they “don’t know” which one (15%).  Choosing and following a software development process is not brain surgery.  It can be as straightforward as the waterfall model (design before you code, code before you test) or as adaptive as agile methods.  But please pick something and follow it.  How can 4 out of 10 firmware developers even manage to avoid it!?!

You can’t make this kind of stuff up!

It is 2010, my friends.  The first embedded system was designed almost 40 years ago.  We need to get our act together.

Check out my Firmware Code of Ethics, get yourself a static analysis tool, and start following a development process.

19 Responses to “The Sad State of Embedded Software Process”

  1. D Wellesley says:

    My background is most PLC programming. I became interested in micro-controllers about ten years ago. In the PLC world, I always preferred Rockwell or Siemens PLCs. The reason is because the logic can be exported to a text file. You can then run a tool on that text file to find discrepancies and/or modify/expand it to re-import. I always hated working with PLC platforms where this wasn’t a text import/export option. It’s funny to read that so few C programmers use a text tool.

  2. GroovyD says:

    i have pc-lint, and even set it up sometimes, but sometime later i find myself not wanting to run it because that usually means hours of adjusting settings that i thought i already set up but for some reason they aren’t working again and it is usually that the environment has changed in some way perhaps some path has changed or something just can find something or some pre-processor directive. i know it is an excuse and the tool needs to be setup properly, but it would be really helpful if they made it all a bit easier.

    usually i try to load the plug-ins for integrating the tool with my ide and that is usually when things don’t quite work right and i find myself just wandering elsewhere. i hate using the command line pc-lint since it is a real pain going back and forth to the ide to fix the problems. really having it integrated with the ide is critical. so if anyone has pc-lint config files setup for the avr32 uc3b lineup using the free avr studio 32 (based on eclipse) with the free gcc tools I would appreciate it.

    i hear your plee bro, and i know it is all a big excuse so i will give it another shot…

    • Nigel Jones says:

      PC-Lint is indeed hard to setup for an embedded system. I tend to use just IAR compilers and so porting my PC-Lint configuration files from one architecture to another is relatively straight forward. Having said that it is still close to a day every time I do it. Despite this I find it very worthwhile. If PC-Lint finds one bug then the day has been worthwhile. Given that it typically finds a lot more than that just ups its value.

      Incidentally I have recently found a perhaps not unsurprising use of PC-Lint. Like many in the industry I am moving towards buying as much software as I can (particularly protocol stacks). Given that these are usually supplied in source code form, it’s a rather interesting experience to run PC-Lint over the purchased code. You can learn a lot about what you have just bought!

  3. Kyle Bostian says:

    I could have written GroovyD’s comment. Maybe getting this tool set up separates the men from the boys and shows which side of that fence I’m on. However, I struggled even with a known good .lnt file. Time passed, another project popped up, and static analysis went right back to the back burner.

    This is a real chicken and egg problem, though – embedded compiler vendors will not do shrink-wrap integration of static analysis until there is higher market adoption. Market adoption will not be higher until the tools are easier to set up.

  4. Peter Bushell says:

    Process is important, but I find myself worrying more, and more often, about the state of some embedded programmers…

  5. Jason Malarkey says:

    I use PC-Lint every day. It is a godsend. I use level 2 as a minimum warning level, and typically only go up to level 3. The number of warnings is easily manageable. Level 4 is kind of worthless in my opinion, so I don’t bother. This seems to be where all the “useless” warnings come from. I’m picky and require the code to be lint clean before check-in to the version control system. Don’t even get me started on people that don’t use version control…

    For new code, I truly do not understand why people get a ton of warnings. If you are not abusing the language, you really shouldn’t have that many warnings. Keep your code clean from the beginning and the tool adds minimal overhead. If you are maintaining an existing code base, that was not linted to begin with, then I understand.

    It terrifies me that people don’t use it. It is cheap and fast and catches a ton of bugs.

    • Edgar says:

      I’ve just set GCC to report warnings as errors on my project. It took 2 days to clean up everything, but now I am a happy camper.

  6. Bill says:

    My entire team uses pc-lint. It was incredibly difficult to get it set up and running without issue, but it has been worth 10 times the effort we put in. A clean lint output is a requirement for passing our design review, and as a result people now write code that passes almost right away. *That* was the real benefit – we never even create the bug in the first place.

  7. Susan says:

    PC-Lint is hard to set up? C’mon boys, really? If setting up lint is hard for you to do, then I really don’t want you writing safety critical, embedded software. I run PC-Lint every day that I write C/C++ code. I run it on software that I purchase (if I get the source code). I run it on code that I did not write that I have been asked to review. I don’t love setting up the options everytime I do this, but that is trivial compared to the issues lint catches for me. Once you are in the habit of setting up and using lint, it is easy to keep up with the fixes that are necessary to keep code clean. It’s called being a professional.

  8. James Kosin says:

    Not true for me.

    My compiler supports MISRA C and I use that to guard totally from silly mistakes.
    I’ve been doing embedded programming for many years now and have learned many things over the years.
    I’ve been using version control also for about 7 years now, before that ZIP files and the like with dates and sometimes times involved.

  9. Allen Moore says:

    I’ve been using PC-lint since 1997 and it’s saved my bacon many times, catching what the compiler misses and saving countless hours of debugging. Even if you’re introducing it to pre-existing code (and getting thousands of messages), you can still prune them down and weed out the relevant ones. However it can be time-consuming to set up, particularly if third-party libraries are involved. My current project is like that, with include files surreptitiously squirreled away by installers all over the disk.

    Getting back to the point Mr. Barr raised, I’d have to agree with him on the importance of process. I’ve been developing embedded software professionally since 1983, and it still seems to be pretty much of a seat-of-the-pants activity, more art than engineering. I would say one of the bigger problems is the difficulty in estimating a project, which often results in unrealistic deadlines, leading to intense schedule pressure, which pretty much precludes taking time to adopt new practices and refine process.

  10. Steve Karg says:

    I used to use PC-Lint, but now I use the free and open source splint from splint.org (Secure Programming Lint). It is certainly challenging to create the lint configuration files for either PC-Lint or splint for various embedded architectures and compilers.

  11. Joe S says:

    I don’t think the state of embedded software development is so bad everywhere … it certainly is better than that where I work. We work on a LARGE embedded system and I don’t think we could do it if we didn’t have a defined process (including methodology), version control, code reviews before check in, nightly regression tests with results broadcast by email every morning, bug tracking database, and more. If I ever went anywhere that didn’t have these things I think I’d feel responsible for bringing them in.

  12. Ken says:

    C is an awful programming language, really a high-level assembler. It’s left embedded computing back in the dark ages! As a computer engineer, I am consistently twice as productive using other prog. languages and thank God embedded systems have enough memory and MCU muscle to support this. I love that Microsoft still has security breaches on their OS due to the lack of array bound checking in C!

  13. Something are meant to change in big-way, not incremental small steps.

    I believe C should be used lesser and lesser, for low level code only. Success of Android confirms this in loud way.
    The ratio of programming efficiency is about 2.5:1 between Java:C.

    The only way to create/maintain big C projects is to use a rich graphics environment like MATLAB, with C primarily an output language for code generator. Only such rich editing environments can enforce the LINT kind of issues, right there at the time of writing the code.

    C was actually designed for backend language for lex/yacc, but people started (mis)using it and continue to do it.
    [Ref : Unix programming by Kernighan/Pike].

    My exp: 10 yrs of programming experience and using C (mostly), C++, Java, C#, MATLAB and some other code generators. Gone through lots of pain of maintaining big SW, where I observed all violations of design/coding principles.

  14. David says:

    I think that gcc -Wall -Wextra can be subtituted by lint tool (if you use gcc).
    Iterative waterfall is the development methodology. Usually, two iterations are enough, the prototype and the final version, but then new versions are being generated from time to time to solve bugs undetected or add new features (and bugs).

  15. Ram Chadalavada says:

    Anyone know how the open source splint rank up with the likes of PC-Lint such tools in terms of features and accuracy? It has an Eclipse plugin as a plus.
    And Michael, thank you for bringing up this sad but true state of affairs. I have had to don the role of the process/methodology ‘nazi’ at my workplace. Tough love is required these days.

  16. Matthew Khouzam says:

    I would be very curious to se how pc lint compares to some of the free open source software out there. I do embedded develpment mostly using the gcc toolchain. I actually tried another lint, (lclint http://lclint.cs.virginia.edu/) and Codan (an eclipse cdt plugin) and they are useful to perform a full audit. I think the learning curve for pc lint may be smaller, I haven’t tried it yet. What I would like to note is that the tests were on more boundary based conditions than actual design abilities. It would be very interesting to run a test on the software process.

    The hidden advantage of a static analyzer is that it trains people to write easier to verify code. It’s kind of like having a teacher next to you while you type but that is not judgemental.

  17. Anthony Wong says:

    In my group, we’ve been using Coverity Prevent (static code analysis tool) for over a year, now. Initially, most people were hesitant and disagreed with the analysis (if the code works, then it’s not a defect!).

    Now, everyone pretty much accepts it.

Leave a Reply to Peter Bushell