Archive for May, 2009

The TOP SECRET Social Class

Sunday, May 24th, 2009

This is America – the land of opportunity and home of the brave. This country was created on the premise that all men (and women) are equal. We don’t have kings and queens and we don’t tolerate discrimination due to race, creed, or color. Yet we have a classed society. We have gotten ourselves into a situation where there are two very distinct classes of engineers – those with a security clearance and those without.

Engineers have been suffering just like everyone else during this economic downturn. Some have been laid off more than once and many are worried about their job security. During these troubling times I often hear “If only I could get my security clearance”. The opposite perspective was well stated by a friend: “Got a TS clearance which has helped my employability to no end”.

While some engineers are simply not interested in any type of work that requires a security clearance, others are not so picky. They would welcome the opportunity to level the playing field when competing for security-oriented jobs. Currently they are at a severe disadvantage when the job description contains the phrases “Active Secret Clearance is required” or perhaps “Clearance required: Secret; TS or TS with Full-Scope Poly desired”.

This problem is rooted in the two pervasive challenges of the engineering workplace: time and money. Companies want their new employee available for work immediately. They minimize lengthy delays by restricting the candidate pool to those already cleared. In reality, few companies can afford to hold a highly paid employee on the sideline until the needed clearance comes through.

But what can be done about this problem? Well, They Say We Landed a Man on The Moon, but it doesn’t take a rocket scientist to solve this. To level the job market playing field the government needs to offer the service of providing a security clearance in a timely fashion to those citizens that request one. Like being prequalified for a mortgage, an individual would be pre-cleared up to a certain level. Such “pre-cleared” individuals would be equivalent in every way to those currently enjoying an active clearance.

How can I ask the already overworked and cash strapped government to do more? Easy. I ask this in the interests of eliminating a classed society and maintaining the personal equality envisioned by the founders of this country – oh, and also to provide the purely practical advantage of removing the disruption cause as companies trade the same cleared employees back and forth from project to project.

To those who may suggest offering clearance to so many people would endanger our security infrastructure I answer in one work – balderdash. If anything, the government would learn a great deal about its citizens by looking closely into the backgrounds of those who request such scrutiny. It is even possible; perhaps likely, this work would detect malfeasance and avert attacks and other mishaps.

Unfortunately, this task is large. By some estimates there are about 300,000 working engineers. I have no idea how many engineers already have security clearances or how many additional are cleared per year. That information may even be classified. However, a quick look at a list of engineering job openings shows many of them require a currently active clearance. Clearly the magnitude of this problem, at least for the first few years clearing the backlog, requires prioritizing which engineers are cleared first. I suggest the following initial prioritized scheme. It can be review periodically.

1. Individuals never convicted of a major crime.

2. Individuals who were previously cleared at any time in the past. This shows a demonstrated interest in this type of work and the high likelihood that the individual will be successfully cleared once again.

3. Individuals with the most needed skills as determined by open job requisitions during the previous six months.

4. Special circumstances.

5. Date of the application.

A web site would be created to allow interested engineers to apply online for a security clearance. Applicants would check appropriate boxes and list any special circumstances to establish their clearance priority. Any applicant found to be lying would be rejected. The web site would log the information and provide the applicant’s position on the list and an estimate of when to expect their clearance. This information would be periodically updated.

The mission of enabling citizens to obtain security clearances should begin immediately. We need to remove the classed society that establishes engineering royalty while holding others unwilling attendants to lesser tasks. It is unacceptable that we have a large number of quality engineers sitting on the sideline instead of working to make our country safer.

They Say We Landed a Man on The Moon

Saturday, May 16th, 2009

We’ve all heard them – the persistent rumors that the United States faked the moon landing. These rumors have gained enough traction that you see them mentioned in the mainstream press and even see documentaries focused on refuting specific claims of fabrication.  How can this possibly happen given that millions of people, including me, grew up watching live broadcasts of spacecraft launches and lunar missions.

The Mercury, Gemini, and Apollo programs that culminated with landing a man on the moon were done in public view.  For some reason, back in the 1960s, it was not a security threat for every foreign power in the world to know the American aerospace capabilities.  The public nature of what had happened and what was planned provided an opportunity for everyone, the knowledgeable and the uninformed, to critique the approach.

With information freely available, the academic community provided detailed positioning information for the in flight trajectory of the spacecraft.  I was an amateur astronomer (yep… once a geek, always a geek) and made use of this information.  The brightness was near the limit of my inexpensive equipment, but with young eyes and a strong desire I was able to imagine that I was watching a craft created on earth en route to the moon.  In addition, ham radio enthusiasts with sophisticated equipment claimed to be able to hear unscrambled conversations between the astronauts and ground control.

The launches themselves were open to the public.  As a young student I had the opportunity to go with some friends to see one of the launches.  Although I lived on the east coast it was a long drive to Florida and I had other things to do.  Perhaps I had to mow the lawn or study for some upcoming tests.  Spending a couple of days in a car to see the launch just didn’t seem very important.  After all, I thought at the time, they would be sending people to the moon for the rest of my life.  I, and others, thought we would have plenty of money to expand the space program once we stop spending a fortune in Vietnam.

Having lived through one of the greatest adventures the human race has ever undertaken, how could I entertain any doubts whatsoever about the veracity of the lunar landings?  For me this is not about discrepancies in photos or flags waving in the non-breeze.  Sadly, this amazing achievement conflicts with my years of experience in engineering.

Do I actually doubt that the United States landed a man on the moon?  I guess not, but I am at a loss to explain how the engineering was done.  While I worked with engineers who could have done this, I never felt great confidence that many of the organizations I’ve seen could have accomplished such an incredible feat.  I joked that some of the places could have embedded a man in the moon – but bringing him back alive was out of the question.

Organizations I’ve experienced, some with great commitment to process, universally suffer from over optimism and excessive focus on short-term goals.  They are universally worried about the schedule, the next demo, and what the customer will think of delays.  At times they lose sight of the bigger picture and the importance of actually getting the engineering right.

A friend of mine once said the Apollo moon landings were not a triumph of engineering but of management.  He was absolutely correct.  While you need good engineers, without good management and leadership your project is going nowhere.  This is why Netrino and other companies offer courses in engineering management to compliment their training for engineers.  In particular, Netrino’s Team-Based Firmware Development course goes beyond traditional management topics of setting requirements and managing the schedule to help companies large and small establish proper management attitudes and corporate culture.

Commercial engineering is not an academic environment.  You have a schedule and finite money.  You may not have time to find a perfect solution, but it is your job to find a good solution.  Getting it right with some engineering elegance is never a waste of time.  Don’t be part of the problem.  Do it right.  If your corporate culture doesn’t support this, help change it or find another company.  Good engineering is not about embedding a man in the moon, but about returning him safely to home.

Memory Loss

Thursday, May 14th, 2009

I recently bought a new digital camera.  A well-known brand, it was on sale but listed for nearly $200.  When I bought it the salesman tried to sell me a memory card.  His sales pitch was that the camera could only store 10 pictures.  Laughing to myself I thought how good it was to be an engineer and know such a thing was impossible given the price and density of modern memory.

 When I got home I extracted my new treasure from its packaging.  Luckily the battery was mostly charged, so I ignored the user manual and started taking pictures.  One, two, three, then nothing.  With some investigation I discovered to my amazement that the salesman wasn’t lying to make a sale.  The 15 MB of internal camera memory could only store 3 high quality pictures.  It could store 10 in mediocre quality.  Further investigating the user manual I found a table estimating the number of pictures that could be stored for various external memory cards.  The manual estimated that a 2GB memory card could hold well over 1,000 mediocre quality pictures.

Let’s think about this.  Those of you old enough to be familiar with 35mm cameras might remember film came in rolls that could take 24 or 36 pictures.  Building a camera that can only take 10 pictures defies reason.  What engineer would build such a thing?  What project manager with an engineering background would approve the release of such a decrepit product?  In fact, for an extra couple of dollars, maybe 1% of the retail price, the basic camera could store hundreds of pictures.

Ah, but marketing trouble looms.  Far few customers would buy a memory card if the camera, fresh out of the box, could store hundreds of pictures.  It seems memory cards have a high profit margin and the marketing team insists that digital cameras be designed and manufactured in an intentionally crippled form so that the company can sell more digital film.  Of course once someone buys one memory card they are more likely to buy a second and third.  Building a camera that can only store 10 pictures is not a function of bad engineering but of good marketing – at least in the view of the corporate leaders making the decisions.

The occasional chasm between what is viewed as good engineering and good marketing can be discouraging and stressful to both groups.  This difference in vision can also translate to vastly different product concepts.  Engineers tend to want cool features but, sadly, it is rare for marketing to be on the side of the consumer.  Marketing tends to focus on extracting more money from the consumer and getting them to buy accessories and to pay for additional features. 

In the case of the camera, the marketing approach was effective in getting me to buy a memory card.  Clearly the decision makers and the marketing team earned their bonuses.

But wait, there’s more.  This marketing ploy left a bad taste in my mouth and I would have to guess this is not just because I’m an engineer.  Like me, most people don’t like being taken advantage of.

Over time the public becomes bitter and turns against manufactures, institutions, and individuals whom they feel have abused their position and taken undue advantage.  In contrast some people develop a near religious devotion to manufactures they perceive as building products that, while expensive, provide value.  These manufacturers don’t nickel and dime their customers.  In turn their customers form long lines at stores the night before the release of a new product.  While these elite manufactures may lose out on the sale of a few memory cards, they are rewarded with market dominating products that sell millions.

Penny-wise and pound-foolish says much about far too many modern companies.

Progeny of a Grand Old Processor

Wednesday, May 13th, 2009

I’m proud to be the latest member of the embeddedgurus.net team.  Welcome to my blog, Area 0x51.

Given the name Area 0x51, there can only be two topic choices for my first blog.  I can either discuss alien embedded engineering techniques (AEET) or the venerable 8051 processor.  Let’s go with the 8051.  As luck would have it I’ve recently been working on a ZigBee one-chip-wonder based on the 8051.  In general this is an amazing chip.  It runs at a speed that could only be imagined when the original 8051 was introduced.  In addition to the ZigBee radio it contains a pile of peripherals and a reasonable amount of RAM and FLASH ROM.  Indeed, this chip allows a fully functional ZigBee device to be built with very few components.

But all is not well in this 8051 kingdom.  The chip is true to its 8-bit CPU ancestry and suffers from the 64K address limit… then things go down hill.

One of the chip’s on-board peripherals is a relatively standard UART.  It has a holding register to buffer the next character while one is being sent from the transmit register.  When sending of the current byte is finished, the content of the holding register is loaded into the transmit register and the process of sending the next character begins.  When the byte in the holding register is moved into the transmit register, a bit is set in the UART status register indicating the holding register has room for a new byte.  All this is very nice and very traditional.

Unfortunately, the geniuses that designed the chip decide the power up initialization default value of the UART status register would indicate the holding register was full.  This is downright silly.  What were they thinking?  The default state of the chip says the UART cannot accept a character to transmit.  The programmer has to just KNOW that no meaningful byte will be overwritten.  Sending the very first byte on the UART peripheral is an act of faith.  One must write to a register when the status says that register is full.

A more serious challenge is the basic chip architecture.  The interrupt vector table resides in FLASH memory.  For most programs this presents no problem.  Just burn the program and the associated interrupt vectors into FLASH and start running.  However, this architecture makes an entire class of programs much more difficult to implement.  Programs that may load different interrupt service routines must use an indirection table since writing a new interrupt vector into FLASH is prohibitively complex.  This complexity includes the necessity of erasing the FLASH page before it can be written which means content must be saved, interrupts must be disable, and more.  Writing a bootloader (my recent task) becomes extraordinarily difficult.  The bootloader must never be overwritten, so its interrupt vectors must always be available in some fashion.  The solution amounts to an arcane mixture of tables and function pointers, intercepted interrupts, and manual tracking of bootloader and application interrupt vectors.  It would have been so, SO much easier if the interrupt vectors were in RAM and initialized by a few bytes of code executed early in the power up process.

Both the UART and the interrupt vector architecture issues come under the category of “what were they thinking”?  Like a T-shirt manufacturer sewing a scratchy tag into the neck of the shirt, it is clear the manufacture did not actually use the product themselves, or perhaps has no structure in place to hear complaints of the early testers.  In a competitive world, why would you work so hard to get such amazing capabilities into the chip, yet stumble on something as simple as a UART – a design that has been standardized for decades.  It only makes sense that a set of experienced eyes should take a close look at new designs before they go into production.  This is an extraordinarily cheap insurance policy that would prevent “stupid mistakes” and “dumb ideas” from being shipped.  You have to wonder why management didn’t make use of experienced internal reviewers or perhaps an outside review by a company like Netrino.  Was it hubris or was it the ever-present short schedule that allowed no time for such niceties as a review or a “stop-and-think”.

In my career, I’ve seen so many occasions where project management has driven an immature project to market that I can hear the echoes of their demands as I type:  “Hurry, hurry, hurry…  We have no time for a design review, or that code review, or to fix that bug.  We must ship TODAY”.  So many of the companies where I heard this are out of business now that you would think it would be blatantly obvious that rushing bad products to market eventually has a devastating effect on the company.

Unfortunately, this learning rarely takes place since those who rushed buggy products to premature market are, in fact, often congratulated for their skill and ability to drive a project to completion.  Indeed, the corporate bonus structure is often based on the ability to deliver on commitments and stay on schedule.  Furthermore, hiding bugs or perhaps not looking very hard for them allows the team and team leaders to bask in the glow of successfully meeting their schedule.  The damage inflicted on their company is subtle and may take years to kill the company.  By that time the culprits are too far removed to be tarnished by their misdeeds.

I’ve seen over and over job descriptions that require candidates to have a demonstrated ability to keep projects on schedule.  I must say that I have NEVER seen a job description that required candidates to have a demonstrated ability to deliver quality products.  Many companies talk the talk of quality but few actually care enough to walk the walk of ensuring their engineers have the training and time to deliver quality products.

I’m sure the 8051 wonder chip I recently used will sell because it has amazing capabilities.  The problem is this chip, as good as it is, did not achieve its true potential.  Like so many products, it can only dream of what might have been.