http://embeddedgurus.com/stack-overflow/2009/10/a-taxonomy-of-bug-types-in-embedded-systems/ Thoughts on embedded systems by Nigel Jones Mon, 06 Sep 2010 14:57:18 +0000 http://wordpress.org/?v=2.9.1 hourly 1 http://embeddedgurus.com/stack-overflow/2009/10/a-taxonomy-of-bug-types-in-embedded-systems/comment-page-1/#comment-303 Nigel Jones Sat, 10 Oct 2009 12:20:50 +0000 http://www.gfcdev.org/test-stack/2009/10/07/a-taxonomy-of-bug-types-in-embedded-systems/#comment-303 You make some interesting points GrayGaffer. I essentially have no experience of inter-processor communications in a multi-core design, which probably goes a long way to explaining why this class of problem doesn't appear on my taxonomy. I found the problem you described very interesting - particularly as it illustrates something that has long been apparent to me - namely that completely testing a product is virtually impossible. I think most of us would treat with great skepticism a bug report three years after the product shipped. While I've done my fair share of sitting on factory floors, I long ago came to the conclusion that the only real way to prevent bugs of this type is via a formal code review that explicitly looks for these sorts of race conditions. You make some interesting points GrayGaffer. I essentially have no experience of inter-processor communications in a multi-core design, which probably goes a long way to explaining why this class of problem doesn't appear on my taxonomy. I found the problem you described very interesting – particularly as it illustrates something that has long been apparent to me – namely that completely testing a product is virtually impossible. I think most of us would treat with great skepticism a bug report three years after the product shipped. While I've done my fair share of sitting on factory floors, I long ago came to the conclusion that the only real way to prevent bugs of this type is via a formal code review that explicitly looks for these sorts of race conditions.

]]> http://embeddedgurus.com/stack-overflow/2009/10/a-taxonomy-of-bug-types-in-embedded-systems/comment-page-1/#comment-302 GrayGaffer Thu, 08 Oct 2009 23:17:39 +0000 http://www.gfcdev.org/test-stack/2009/10/07/a-taxonomy-of-bug-types-in-embedded-systems/#comment-302 Projects are increasingly moving up in scale of CPU types and count, e.g. PowerPc multi-cores. Which brings into prominence another bug type: thread safety violations. This is not quite the same thing as re-entrancy bugs. The latter usually involve un-necessary global state, while threading issues usually revolve around poorly- (or un-) protected critical regions around access to global or shared resources. Interrupts must be consider as pre-emptive threads, even with co-operative tasking architectures. As must resource handlers shared amongst multiple CPU cores.I had my exemplar experience in this with a buffer queue bug in a queue from interrupt handler to routing layer. It did not rear its head until three years after the product started shipping. Up till then all uses had been stochastically well distributed, but in this one instance it was being used for printing pick-tickets in an automated warehouse. It just so happened that the interval between tickets, at full load, raised the probability from insignificant to one in twenty messages, because of a synchronicity arising from the processing time between one chunk interrupt to the next lining up with the task level access to the same queue. I spent three days sitting on the factory floor glued to an oscilloscope looking for this one. In the end I just had to DI - EI around two assembly instructions to fix it.Lesson learned. Projects are increasingly moving up in scale of CPU types and count, e.g. PowerPc multi-cores. Which brings into prominence another bug type: thread safety violations. This is not quite the same thing as re-entrancy bugs. The latter usually involve un-necessary global state, while threading issues usually revolve around poorly- (or un-) protected critical regions around access to global or shared resources. Interrupts must be consider as pre-emptive threads, even with co-operative tasking architectures. As must resource handlers shared amongst multiple CPU cores.I had my exemplar experience in this with a buffer queue bug in a queue from interrupt handler to routing layer. It did not rear its head until three years after the product started shipping. Up till then all uses had been stochastically well distributed, but in this one instance it was being used for printing pick-tickets in an automated warehouse. It just so happened that the interval between tickets, at full load, raised the probability from insignificant to one in twenty messages, because of a synchronicity arising from the processing time between one chunk interrupt to the next lining up with the task level access to the same queue. I spent three days sitting on the factory floor glued to an oscilloscope looking for this one. In the end I just had to DI – EI around two assembly instructions to fix it.Lesson learned.

]]>