Archive for June, 2009

Schedules I – Bozos & Clowns

Saturday, June 27th, 2009

My post, They Say We Landed a Man on The Moon, resulted in a number of private conversations about schedules. Enough private chats; let’s light the fire and really get this discussion going. This is part 1 of a two-part discussion on schedules, scheduling, and adhering to schedules.

Let me start by saying that schedules are as important as requirements. Proper planning dictates the project must have a schedule and everyone on the team must understand his or her contribution to keeping the project on that schedule. In addition, the schedule should be a commitment on the part of the developers and that cannot happen unless the components and milestones in the schedule have been specified by mutual agreement between developers and management. A schedule cannot be dictated to the developers if one desires their commitment to stay on that schedule.

It is common sense why the team must adhere to the schedule. However, common sense is sometimes not as common as we would like so we should specifically state some of the reasons:

· Working on a project means the team is getting paid. Working longer than expected means more money must be spent than initially budgeted. Companies have finite money, so more money on one project means something else suffers.

· Working on a project longer than expected also means the team members are NOT working on the next project.

· Taking longer to finish a project means a delay in the additional revenue the late project was intended to produce. The company gets hit with a double-whammy. It spends more and makes less. Obviously this cannot continue indefinitely or the company will be forced into bankruptcy.

· Other groups within the company may be depending on the engineers to stay on their timeline to make any number of related events possible. For example, an advertising campaign won’t make much sense if there is not yet a product; being late for the Christmas selling season may seriously damage projected revenue; and missing a planetary alignment pretty much kills the mission.

· Managers look bad when their project is late

· Developers look bad when their project is late

· The available time provides a context in which to make engineering tradeoffs

· The available time frames the acceptability of certain project risks

Given all these very good reasons, how can there ever be a justification for a schedule slip? Well, sometimes the insidious universe just won’t cooperate and problems take unexpectedly and unpredictably long to solve. In other cases the original timetable was just not reasonable. On occasion the schedule was arbitrary and externally imposed and never really intended to be realistic. In other cases there was a good faith effort to make an achievable schedule but the effort was badly underestimated. My 30 years of engineering experience says two things are absolutely critical to creating a pragmatic schedule to which the team has some chance of adhering:

· Previous experience with the project technology and architecture

· A calibration of the productivity and work quality of the team members

There are always surprises when the team lacks experience in the technology or system architecture. In this scenario, schedule estimates amount to no more than educated guesses and these guesses can be wildly inaccurate. Likewise, the difference in the volume of robust and reliable output of software, hardware, and firmware engineers seems to vary by nearly two orders of magnitude. Unfortunately, most engineering of new products is explicitly about new technology, new ways of doing things, and often with new teammates. Despite the importance of staying on the established timetable, we have a real-world environment where development of the newest and most advanced technology is seemingly destined to fall behind schedule.

* * *

Consider a project that is starting to fall behind schedule. Here is where bozos and clowns make their grand entrance. Schedule deadlines, realistic or not, provide an excuse for the inept to make ridiculous and damaging demands. True leadership manages to a schedule but through experience and knowledge understands the tipping point. Real leaders understand when a slip is justified and when extreme efforts to stay on the established timeline simply inflict further damage on the project.

It is important to frame this problem in the corporate context. Superiors must have an extraordinary amount of faith in a manager to tolerate schedule slips. The situation can exist where many in corporate executive management lack personal expertise in the technology being developed and correspondingly lack interest in problems along the way. For a number of executives, they only want to know their money is being well spent and the only way they can tell is to check off the milestones. Understanding the nature of a problem is not in their job description and most would not have the necessary time to invest in understanding even if they had the interest. When milestones are missed the easiest option is to increase the pressure. This means there can be a conflict between what is right for the project and what is right for the manager’s career.

Executive management can easily view a manager slipping the schedule as weak and simply not driving the team hard enough. Hence, in the normal corporate structure, we have an inherent defect. This defect forces upwardly mobile individuals to ignore or hide problems instead of confronting and solving them. In this environment, management weasels continually divert resources and attention from the righteous goal of a quality product so they can provide glowing status reports to senior management. Many times executive management never actually use products made in this atmosphere and are completely unaware of the buggy garbage being shipped. Instead of being chastised for shipping junk, the weasel manager is congratulated for delivering his project on schedule. Other managers observe this and we have created a corporate feedback loop that encourages unwarranted herculean and damaging efforts to stay on unrealistic schedules – often at the expense of product quality and supportability.

Many managers buckle under this perceived pressure and start skipping steps and taking progressively riskier shortcuts. It is incontrovertible that forcefully holding the line against schedule slips causes damage to quality and often results in actually lengthening the schedule as payment on risky shortcuts comes due. I have seen this too many times in too many organizations. I’ve seen complete disasters that resulted from managers doing insane things to stay on an impossible schedule.

Patents and Copyrights and Sharks… Oh My!

Thursday, June 25th, 2009

It can be educational and stressful when an engineer goes through the corporate patent process for the first time. Some companies emphasize creating patents and some don’t. Some engineers work their entire career and never file a patent while others seem to be involved in a new patent almost monthly.

As an engineer, not a lawyer, I see the differences between patents, copyrights, and trademarks to be:

· A patent protects the process, method, or apparatus of an invention (like an algorithm or mechanism)

· A copyright protects the expression of an idea (like a book or picture)

· A trademark protects the name of something (like Mickey Mouse™)

There can be extenuating circumstances, but most engineers sign away their right to personally hold a technical patent related to the business of their employer. In a way this is very reasonable. The company pays the engineer and provides training and an environment focused on creating and implementing technology. It makes sense that a company would claim ownership of ideas fostered by the environment it created. Hence, nearly all patents are assigned to the company for which the engineer works. Some companies pay a token amount to employees that come up with a patent. Some pay a little to thank the engineer for the effort of writing down a good idea; more for ideas they like enough to submit an application to the United States Patent Office; still more if the Patent Office awards a patent.

The process of obtaining a patent starts when an employee has an idea for something new and not obvious to a person of ordinary skill. The idea can be revolutionary or a substantial improvement to something that already exists. Patents cover any number of methods and devices but this blog is most interested in technical advancements. When an engineer has an idea that significantly advances technology, management will instruct the engineer to submit an invention disclosure form. A company of reasonable size may have an invention disclosure review board. This group may meet on a regular or ad hoc basis. Some days, weeks, or months later the group will review the invention disclosures that have been submitted. Corporate executives use their skill and experience to decide which ideas have sufficient merit to continue with the patent application process.

Not all engineers have equal access to the process described above. I’ve worked with or been associated with a reasonably large number of companies but have never seen a company that publically detailed an actual patent submission process. Newcomers must occasionally submit a number of invention disclosures before they are recognized and start getting attention. The corporate patent holders can be something of a club and like many clubs you must sometimes prove you are worthy before you can join.

An interesting quirk of the patent system is that there is no need to actually implement the idea. One must only describe how to implement the idea in the patent application. A few companies exist to exploit this quirk. They never build anything yet make good, albeit annoying, income by extracting royalty payments from companies that build and sell products that use their patented ideas.

The critical elements of a patent are the claims. Claims specify exactly the intellectual property for which protection is sought. My experience is that revolutionary ideas are much harder to patent than innovative technological improvements. The reason is that revolutionary patent applications are often accompanied by broad, sweeping claims. The Patent Office is generally resistant to broad claims and prefers to leave room for more innovation and growth in new technologies. Writing good (i.e. likely to be approved by the Patent Office) claims requires specialized experience and this is where the attorneys get involved.

For engineers (unless they’ve been through a divorce), working on a patent may be the first time they work closely with lawyers. While the lawyers may be knowledgeable about technology, they are not as expert as the expert(s) inventing the technology and applying for the patents. It can take hours of meetings with the attorneys to sit and weave thoughts and ideas into specific written claims of the patent. It can take weeks to prepare the final patent application as claims are written, reviewed, and revised.

It can then take several years for the Patent Office to make a decision on the patentability of the invention.

Dude! I just want my email

Monday, June 22nd, 2009

I have several computers and as a consultant often work at different locations. Because of this I tend to pick up my email using a web browser instead of one of the email applications. The problem is some of the email hosting services use accessing email as an opportunity to sell things. My browser recently declared “Completed loading 84 of 97 items” while I was waiting for the chance to enter my password. My browser was cluttered with the likes of:

· How to whiten my teeth

· How to get rid of cellulite

· Learn my credit score

· Help with my bladder, gas, and erectile dysfunction problems

· How to earn $300,000 per year working from home

· Hot stock tips

· How to pay off my income tax debt at a fraction of what I owe

· Learn about college affordability aid

· How to refinance my mortgage

This was not spam email, but advertisements thrown at me by the company I pay to provide Internet service. Look, I just want my email. There is NO chance I’m buying any of this junk – especially since trying to sell me this stuff makes it take longer to get to my email.

Capable people and successful companies operate with grace and dignity. They don’t beat you over the head with fast deals and large volumes of trash. Give me one or two advertisements and I may read them – especially if the ads are done with some amount of elegance and taste. Bombard me with junk and you have an irritated customer. This doesn’t seem complicated to me. A shotgun blast of a large volume of advertisements is not marketing – it is wishful thinking.

Forked Dreams

Saturday, June 20th, 2009

I bought a fork the other day. It came in a plastic bubble that required the use of a chainsaw to open. After the bubble was opened I found I could not use the fork until I accepted the fork licensing agreement. The agreement was seven pages long and included the following:

· This fork is leased, not sold. You paid good money but you don’t own anything. In a little while, the folk will become obsolete but the lease continues indefinitely.

· We do not guarantee this fork to be good for any purpose. You should not use it for anything important. Any attempt to do so is your poor judgment and comes at your own risk.

· A plague of lawyers will be released upon you should you try to reverse engineer this fork.

· It may be necessary to upgrade your spoons to version 3.7 to work with this fork. This is your problem, not ours.

Ok, I’m exaggerating. It wasn’t a fork but a USB to serial port converter – essentially a smart cable. How in the world did we get to the point that I have to agree to a multipage license before I can use a simple electronic trinket… or to receive a critical operating system update? My GPS system makes me agree that I’m an idiot for using it every time it powers up. I’m not allowed to trust its directions and I’m not allowed to pay attention to it while I’m driving.

When I was a kid I didn’t have to sign a licensing agreement to use a typewriter; why should I have to sign one to use a word processor? Software has become as everyday as a fork and far more critical. In fact, I use software many, many more hours per day than I use a fork.

Stop it.

Stop it!!!

We’ve moved from American Innovation to American Litigation. Why should anyone have to print “caution hot” on the side of a coffee cup? Are you a moron?

Several years ago I had a bag in each hand leaving a fast-food establishment. I went to put one bag in my mouth to free a hand so I could open the door to leave. I inadvertently inserted the corner of the bag in my eye. After staggering around and drying my eye I looked closely at the bag. Nowhere did it say, “Do not insert in eye”. I HAD them. I was going to be rich!!

Common laws need to protect people and corporations from malicious intent and true negligence. These laws need to be interpreted in a rational fashion by the courts. If you are a moron it is not society’s duty to protect you from yourself. We should not have to sign a licensing agreement to use an electronic device just as we do not (yet…) have to sign an agreement to use a razor blade.

As an extreme example, consider the components used to construct modern aircraft. Software and hardware developers must sign dozens of agreements before compiling code, making an FPGA, or even reading a PDF of component documentation. What consequences would result if a developer refused to upgrade some needed tool because he didn’t like a clause in the agreement? How long until the manufactures pass that liability unto the customers? Can you imagine the day when we must sign an agreement that the aircraft is provided as-is and not intended for any specific purpose? Can you imagine that getting on a commercial flight might be interpreted as an indication of bad judgment? These agreements cost serious money to create but are they even enforceable? Most people click ACCEPT and never even read them.

In the old days there was an implicit assumption that when we bought a typewriter we could use it to type; when we bought a fork we could use it to eat; when we bought a slide rule it would give reasonably accurate answers when used correctly. All engineers, and the companies that hire them, should have to worry about is building a good product that works well. Engineers, and the customers that buy their products, should never have to consider multi-page licensing agreements created to give bizarre rights to the manufacture or to protect the manufacture from improper or moronic use of their product. Common laws, interpreted in rational fashion, should provide all the needed protection for both manufactures and their customers.

Ah… I’m such a dreamer…


Friday, June 12th, 2009

I was in the lab talking to the manager of the test team. A member of the team was running tests at a nearby workbench. As the manager and I continued our discussion we could see the test failing. Once, twice, three times it failed. On the forth try the test passed and the staff member could be seen making a notation in the testing logbook. “Excuse me”, said the manager. He got up and walked over to the other workbench where he saw the test was marked “passed” in the logbook. The manager commented to the team member that he had seen the test fail several times. “Oh yes”, said the tester proudly. “I had to run it a bunch of times to get it to pass”.

This has been one of the great stories of my engineering career for several years. I’ve told it over and over and it always gets a knowing smile from listeners. Testing optimism is pandemic. One can suffer from it even if you have been inoculated against the disease by the hard lessons of a long engineering career. Your brain knows that seeing something work once means nothing, yet your heart yearns to believe your work is complete. The temptation of easy success beckons. It whispers to you. It is a siren luring you to crash on the rocks of prematurely declaring project victory.

Real-time systems are notorious for having intermittent problems that occur when the planets, and systems tasks, align in an unfavorable fashion. They can also align in a favorable fashion, allowing the system to randomly work. The tester in the preceding story learned this and used it to improve the testing success rate. This effort, by the way, was not malicious or a shirking of duty. The tester genuinely believed that the job was to get the device to pass and this belief exposed several problems. Clearly, the tester was not adequately trained. Although the tester was taught how to run the tests, it was taken as common sense that bugs were to be found and reported. In reality the job was to find bugs so the developers could fix them and make the product better. Unfortunately, it seemed to the tester, every discovered problem resulted in great angst and tension among management and developers. Common sense was thwarted by the entire surrounding culture that subliminally conveyed that bugs were not wanted.

The tester was inadequately trained but the situation exposed another problem. That is, running a test once may allow a statistically infrequent problem to escape detection. Assuming the goal is a quality product, intermittent problems create a serious dilemma for the test team. Intermittent problems don’t always happen. They may occur frequently or rarely and they may be serious or minor. It is their statistical or problematic nature that makes them difficult to reproduce, evil, and so very tempting to ignore. Finding them is hard but proving the problem has been fixed is even harder. Developers are really in a difficult situation. How can they prove the problem is fixed when they could not make it happen in the first place? It costs time and money to test, test, test, and then retest. Some companies cannot afford to do this. Some do not have an interest in doing this.

The conscientious engineer is only happy if something works every time. Unfortunately there is a conflict between this honorable goal and the financial and schedule constraints of the real world. The more complicated the device, the more tasks and interrupts and events happening in cyclic and asynchronous fashion, the more likely the system harbors intermittent problems. This scenario cannot be dumped at the feet of the test team. A quality design implemented by well-trained and talented developers must precede work by the test team. Finally, since we are all susceptible to the disease of optimism, management must continually reinforce that bugs, quirks, and problems are to be found, reported, and resolved.