Because the ASIC had not been implemented per the standard, other components had to deviate from the standard to interface with it. In this case, the engineers had to hack up the firmware discovery process to say, “If the device being queried is an endpoint and the vendor and device id matches this particular chip, then treat it as a root complex and search for another bus and its devices.”

Designing a block in strict compliance to a standard has several benefits:

• The compliant block will work, without changes, with compliant off-the-shelf components (e.g. blocks, chips, device drivers, test suites, debuggers, analyzers, development platforms).
• Well-defined specifications already exist for verifying and testing compliant blocks.
• Expertise in standards exists and is available.

Best Practice: Design the chip or block exactly to the specifications of the standard, even down to the details of the characteristics of the registers.

Some standards have standard subsets. For example, the RS-232 serial interface has transmit and receive lines and hardware handshaking lines. The hardware handshaking lines are optional. Hardware and drivers should be designed such that, if they are fully implemented, they will work with other components that do not have the optional part implemented.

A non-standard subset of a standard will create problems. Implementing a transmit-only RS-232 interface can cause compatibility problems with associated compliant components, such as RS-232 drivers, other RS-232 interfaces connected to it, and test suites trying to verify functionality. These interfacing components would all have to be customized, increasing the time and cost of development, and the risk of introducing bugs.

Best Practice: When implementing a subset of a standard, implement a standard subset, not a custom-designed standard.

Remember, there is no such thing as a customized standard. Either it is customized or it is a standard – not both.

Until the next standard issue…

It is a problem because one group (hardware engineers in this case) has to produce documentation for a different group (firmware engineers). They generally are in different physical locations, come from different engineering backgrounds, have different terminology, and have different perspectives on the end products. To overcome these differences, hardware and firmware engineers need to collaborate with each other.

Writing documentation is difficult. When I write, I understand perfectly well what I meant to say. I know and understand unwritten details and nuances that are second nature to me without realizing that my reader might not. I’m reminded of that when I have someone else review my writing and they bring up missing and incomplete sections.

While both hardware and firmware engineers should be actively involved in the collaboration, it is primarily the hardware engineers who do the majority of the work with documentation and design. When ready for review, they should give the documentation to firmware engineers for review. Firmware engineers should, in a timely fashion, review the documentation and provide hardware engineers with comments on incomplete, unclear, and incorrect sections and any issues that they might discover. Firmware review of hardware documentation should be part of the checklist for milestone completions.

Best Practice: Give hardware documentation to firmware engineers to review and respond with comments and issues.

Before firmware engineers can understand the specifics of how the block works, they need to have an overall picture of the purpose and operations of the block. If they focus too much on details, specific registers, and bits, it is hard to see what the overall operation should be. Not having a high-level description can be compared to seeing a box full of nuts, bolts, springs, levers, and other parts and not being able to recognize that it is supposed to be a toaster. But by first stepping back and looking at a high-level description of the toaster as a whole and understanding how it toasts bread, then it is easier to understand how to assemble the components to make a functional toaster.

The same concept applies to a block on the chip. Firmware engineers need to see and understand the big picture of how that block should operate, within itself and in conjunction with other parts of the system, before they can understand the detailed registers and bits.

Best Practice: Produce and have reviewed a high-level description of the block that describes its theory of operation, its function in the system, and its parts.

In addition to the high-level documentation, a detailed documentation is needed and should contain both a reference section and a tutorial section. A reference section has a list of all registers in the block, typically in address order. It gives details for each register and the bits and/or bit fields in that register. The tutorial section describes when, how, and in what order to use those registers and bits to carry out a task.

An example of reference documentations are the UNIX man pages (and Linux and other variants) which contains all commands and functions in alphabetical order, each one describing their respective options in order. On the other hand, the well-known book by Kernighan and Ritchie, “The C Programming Language,” is written in tutorial style, explaining how to do various tasks, using necessary C constructs to accomplish the tasks.

Best Practice: Provide both a reference section and a tutorial section in the detailed documentation of a block.

Until the next writing…

One of the reasons programmers can get away with that type of thinking is because they can and are able to change software rapidly by simply recompiling and re-executing. Hardware engineers do not have that luxury. It takes months and millions of dollars to produce a different version of the chip. Future costs can be reduced if the chip is designed with the future in mind, with the possibility that it might be used in future products that are currently unplanned.

There is also an aspect of planning for the future where you prepare by building a framework but not putting in the features. That makes it easier to implement the new features in the future but still requires putting out a new version of the chip. But that will be the subject for another issue. For this issue, I am talking about what can be done to make this version of the chip more likely to be used in the future.

Not all known future functionality should be put into the design; time, effort, space and cost will not permit it. But some of it has little risk in terms of complexity, implementation and verification. If a three-bit number might need four bits in the future, making it a four-bit number now has little risk to the program. Adding a couple of extra GPIO lines is also low-risk (assuming there are pins for it.) Adding a block that is new, big and complex could add a lot of risk to the program and needs to be weighed against the potential gain if it were needed.

Extreme Programming says that “only 10% of that extra stuff will ever get used.” But if you put 10 “extra” features in the chip and only one gets used, you could save your company months and millions of dollars.

Best Practice: Include low-risk features in the design of the chip that might be used in future products.

Aside from features that might be needed in the future, faster speed is characteristic of future products. Chips are designed with a performance and speed budget. Planning for the future would mean that you also look at the speed of the chip. I have seen cases where existing chips were limited in their usefulness because they were not fast enough and we had to spend lots of time and money to produce a new version just to get the speeds we needed.

Best Practice: Increase the performance margins in the design to allow the chip to be used in faster products in the future.

Always looking toward the future…

After a presentation I gave at a conference, one of the attendees came up and told me about his ASIC design team that consisted of young engineers. They had completed their design and told him that they were done. He then asked the question, “Six months from now when you get the silicon back and it does not work, what are you going to need to diagnose and solve the problems?” They went back to their desks and worked some more.

He taught them a very important principle: Think about what could go wrong and what would be needed to figure that out. Too often, hardware designs assume that nothing will go wrong. It is akin to a software function that does not check the validity of parameters being passed in, or a hardware module that does not synchronize an incoming signal to its clock.

Hardware engineers are good at troubleshooting chips by mounting them to test fixtures and attaching probes. However, when the chip is mounted inside a prototype device, it is often at the expense of not being able to attach the test fixtures. Unfortunately, some problems will not reveal themselves until the chip is inside the actual device running the actual firmware. For situations like this, it’s important for hardware to be designed to assist troubleshooting.

A technique that I have successfully employed is to build firmware-accessible debugging resources into the chip. It is like having a built-in logic analyzer. Here are three types of debugging resources I’ve found particularly useful.

Some devices use a specific number of signal pulses to control certain features. For example, a laser printer generates a horizontal sync pulse to set the size of the paper to be printed. When the printer is operating properly, you probably do not need to know how many pulses occurred. But if something is wrong, you might want to know how many pulses were generated. Maybe only enough pulses for a Letter-size sheet of paper was generated when you are trying to print on a Legal-size sheet. In this case, having a pulse counter on the signal that firmware can read and reset would help solve that problem.

Best Practice: Provide firmware-readable and resettable event counters to track the occurrences of key events in the hardware.

Well-designed UARTs allow firmware to read the current levels of the handshaking signals. That helps troubleshoot the RS-232 communications. The same technique can be used for any other I/O signals where knowing the current levels of those signals could help troubleshoot problems.

Best Practice: Provide read access to view the current state of key input and output signal pins.

As a general rule, firmware does not need to know the current state of a state machine in hardware. However when there are problems, knowing the state can prove useful. A co-worker was trying to get a new chip going and it would not work. He read the current state of the state machine and discovered that it was stuck in a state waiting for an external signal. He looked at the prototype board and discovered that a resister was missing on that signal line. He solved that problem in a matter of minutes whereas it probably would have taken him hours if that state register had not been there.

Best Practice: Provide a register that shows the current state of each state machine.

Until the next bug…

I tried three different algorithms over a three-week period and finally settled on one, even though under rare circumstances, it had a severe performance penalty. Unfortunately, one particular customer bought several of these printers for a specific application and experienced this rare circumstance that cut print speed in half for every page. Obviously the customer was dissatisfied and threatened to return the printers and cancel the large order, which included other printer models. I spent another two weeks to come up with a fourth algorithm that avoided the penalty. We upgraded the customer’s printers and they were happy.

Had this particular block had the early termination support (at a low cost of a few hundred gates), we could have accommodated this late product change with a one-hour firmware change. Instead, beyond nearly costing us a major contract, it also cost five weeks of engineering effort, thousands of warranty dollars and some damage to our reputation.

In contrast, we had an ASIC that was designed for portrait-format printers that was being investigated for use in landscape-format printers. The ASIC was based off previous ASICs that did support landscape-format printers. The engineers did not remove the landscape-format support even though the ASIC was not targeted for landscape-format printers. ASICs for landscape-format printers require more onboard memory in several places in the pipeline to accommodate longer scan lines. Reducing the memory requirement would have reduced their gate count, but they decided not to do that when they made this ASIC.

After verifying that landscape-format support was still in the ASIC, we used them in landscape-format printers. This saved us millions of dollars and several months from having to develop and produce a new ASIC.

Ideally, every ASIC will support any product, whether old or new, high-end or low-end. Since that is not realistic, choices have to be made. Old functionality can be removed once you are sure that it will never be needed again, even for mid-life kicker products. Functionality with large gate-count requirements that is only needed for specific family lines of products could be removed. But where possible, leave as much functionality in an ASIC, even if not all of it is needed for the targeted products.

Best Practice: Implement and retain all known low-overhead functionality in a block, even if the current requirements do not call for it.

Until the next shipment…

Interrupts are a fundamental part of chips, but there is no standard way of implementing them. This leads to several problems:

• A chip with blocks leveraged from several sources could have mixed interrupt methods.
• Firmware drivers leveraged from one chip to another, or from one block to another, are prone to introducing defects.
• Firmware engineers must first study the documentation to determine which method is being used.

Having firmware write a 1 to ack an interrupt is the best method to use. Most chips use it so it is more commonly known by firmware engineers. When the hardware issues an interrupt causing firmware to read the value in the interrupt register, there are 1s in the appropriate positions indicating which interrupt or interrupts occurred. So it is an easy step to use that value to ack the interrupt. Otherwise, firmware has to take a 1’s compliment of the value before writing it out. And firmware engineers have to remember to take that 1’s compliment everywhere the acking is done, whether acking with the value read from the interrupt register or using a constant (such as a #define) to specify which bit to use.

Best Practice: Design the interrupt module so that a 1 written to the interrupt register acks the interrupt.

Interrupt modules also have the ability to let firmware control whether or not a specific interrupt is allowed to propagate. I have seen this register called the “Enable” register for some chips and the “Mask” register for other chips. And I have also seen the “Mask” register in some chips use a 1 as a mask for allowing interrupts to propagate and other chips use a 0 to mask out the interrupt. Every time I come across an Interrupt “Mask” register, I have to first study the documentation to see if I should use a 0 or a 1.

The best term to use is “Enable” because it is clear in the reader’s mind that to “Enable” an interrupt, you write a 1.

Best Practice: Use “Enable” as the name of the register that controls which interrupts will propagate.

These are just a couple of aspects regarding interrupts that should be set as a standard. I will discuss more in future newsletters.

Until the next enabled interrupt…

Based on the hardware and firmware implementation required, we can group these designs into three categories:

• No Delay – Hardware completes the task almost immediately. Firmware can assume the task is immediately completed and can safely take the next step.
• Short Delay – Hardware completes the task after a short delay. Firmware must wait momentarily for the task to complete before taking the next step.
• Long Delay – Hardware completes the task after a long delay. The wait time is long enough that firmware should do other processing while waiting for the task to complete before it can take the next step.

Let’s take aborts in hardware as an example, since implementations exist in each of the three categories – no, short, and long delays. For some aborts there is no delay; it is a simple matter of returning back to the home or idle state, clearing counters and buffers, and completing other activities that can be done quickly. Such an operation is so quick that it is not necessary for hardware to add extra logic for a status or interrupt bit. In these cases, firmware can initiate the abort and simply move on to the next step, which may be to set up the hardware for the next job. The key is for hardware to complete the abort before firmware tries to access it again.

Best Practice: When the task in hardware is fast enough to complete before the next firmware access, hardware does not need to implement a status or interrupt bit for task completion.

Some abort implementations can take several clock cycles to complete, which means that firmware must wait for completion before accessing the block again. If it is a short delay, hardware should provide a status bit that firmware could poll, looping a few times until the task is done, then move on to the next step. If there is a long delay, then hardware should provide an interrupt bit that firmware will enable. Firmware will then do other processing while waiting for the interrupt to occur. Setting up, waiting and responding to an interrupt requires several CPU cycles with task swaps, context switches and semaphore handling. Thus, for firmware, polling a status bit is preferable to managing an interrupt if the task will be done after a short delay.

Where that line should be between short and long delays must be determined on a case-by-case basis and depends on the hardware platform, operating system and performance requirements. The dividing line could even move dynamically depending on the current operating conditions of the product. To give engineers the flexibility of moving that dividing line, the hardware for short and long delays should be the same, implemented with both a status bit and a maskable interrupt. This flexibility allows engineers to calculate or take measurements to count how many loops the polling is taking and determine if polling is acceptable or if interrupts are needed.

Best Practice: Implement both a status bit and a maskable interrupt bit to indicate completion of hardware tasks that take time to complete, whether a short or a long time.

For some blocks, the time the abort takes can vary from a short delay if the block is in an idle state to a long delay if the block is busy and needs to gracefully terminate. Since firmware cannot know the current state, it must always assume the worse case. If firmware wants to take advantage of the shorter aborts when they do occur, it could poll for several loops in case the task completes quickly. If not, then enable the interrupt and switch to another task.

To help engineers know how to implement the firmware, put in the block’s documentation the min and max abort times and the conditions in which they will occur. It could be something such as, “if the block is already idle, the abort will complete in 20ns, otherwise it will take 2-3us to complete.”

Best Practice: Document the min and max times that a hardware task will take, including the conditions and states that affect those times.

I used aborts for these examples, but the concepts apply for any firmware-initiated hardware task that could take time to complete. Implementing both status and interrupt bits for short- and long-delay hardware tasks allows firmware to balance the system load and performance by using polling loops or interrupts as appropriate.

Until the next interrupt (which will not occur for at least 1,000,000,000,000us)…

Although the engineer had valid reasons for designing the register that way, he or she had not anticipated the impact on firmware of mixing different types of bits in the same register. To avoid complexity and risk of firmware defects, different types of bits should be located in different registers. To see why, let’s examine how firmware manages two types of bits – read/write bits and interrupt bits.

With read/write bits, firmware sets and clears bits when needed. It typically first reads the register, modifies the desired bit, then writes the modified value back out. Here is a sample code fragment:

tmp = ReadReg (regA);  /* Get the register contents */
tmp |= 0x01;           /* Set bit 0 */
RegWrite (regA, tmp);  /* Write it back out */

In the case of interrupt bits, firmware often writes a value with one bit set to acknowledge the desired interrupt while leaving any other pending interrupts untouched.

RegWrite (regB, 0x10); /* Ack bit 4 */

Mixing the two types of bits in the same register could cause problems. Using the read/write code on interrupt bits causes pending interrupts to inadvertently be acknowledged. Using the interrupt code on read/write bits clears all read/write bits that used to be set. Firmware must take special care to ensure that it does not inadvertently change the wrong bits.

Here is a code fragment that acknowledges bit 4 while taking care not to acknowledge other pending interrupts or modify any read/write bits. In this example, read/write bits are located in positions 0x0f and interrupt bits are located in positions 0x70.

tmp = RegRead (regC);  /* Get the register contents */
tmp &= 0x0f;           /* Keep R/W bits but zero any intr bits */
tmp |= 0x10;           /* Set bit 4 to ack */
RegWrite (regC, tmp);  /* Write it back out */

Acknowledging an interrupt changed from a one-step to a four-step operation. A similar code fragment is needed to modify desired read/write bits while leaving alone any pending interrupts.

While there is a way for firmware to safely handle this, it is out of the ordinary and prone to firmware defects. Combining different types of bits into the same register may save registers but it adds unnecessary burden and complexity to firmware. Looking ahead and anticipating the firmware impact can lead to a more reliable and robust solution of placing different types of bits in separate registers.

Best Practice: Segregate different types of bits (read/write, read-only, interrupt, etc.) into different registers.

If necessary, read-only bits could be combined with any one of the other types of bits. This is acceptable because no matter how the other bits are handled, firmware writes to the register will not affect the read-only bits.

Until the next bit…

This incident illustrates the benefits of early collaboration between hardware and firmware engineers. All too often, though, the early phases of embedded systems projects consist solely of hardware engineers working on the design of the hardware without firmware engineers involved. This practice makes it difficult to produce designs flexible and robust enough to support features that also rely on firmware. By working together, however, hardware and firmware engineers can avoid these problems.

Early collaboration of hardware and firmware engineers (both formally and informally) gets both groups talking to each other. Each group has a different perspective of their respective needs and of the embedded system. Here are some benefits you can expect:

• Reviewing the system requirements together helps reconcile differences and produce a more comprehensive view of the requirements. This helps assure the design meets the requirements.
• Some features can be implemented in either hardware or firmware. Working together, engineers can discuss where the line should be drawn between hardware and firmware, taking into account trade-offs such as performance, flexibility and ease of implementation.
• Collaboration ensures the design of the hardware/firmware interface leads to a smoother integration of hardware and firmware into the embedded product as a whole.

Hardware and firmware engineers reap these and other benefits when they collaborate early.

Best Practice: Start collaboration between hardware and firmware engineers during the initial hardware high-level design phase.

Formal hardware development processes typically include checkpoints with formal reviews and signoffs by key personnel, assuring successful completion of all criteria. Firmware participation in the hardware checkpoints encourages hardware and firmware engineers to review the documentation together and jointly resolve any issues they uncover.

Best Practice: Include firmware representation in reviews and signoffs of hardware checkpoints throughout the life cycle.

Until our next collaboration…

I used “bridge” in the conceptual sense to enable two sides to work together, bridging the gap between the two, and establishing processes, procedures, and protocols between the two. I have written much (book and newsletters) about hardware and firmware (a.k.a. embedded software), both in terms of their respective engineers and in terms of the design of the interface between hardware and firmware. But I also like to discuss bridging the gap between any two other entities, such as management and engineers or the end user and their embedded systems device.

In addition I’ll post other musings in the embedded and engineering space. Until next time …