High Performance Scalable Hardware Configuration Management

Shiv Sikand, IC Manage

International Cadence Users Group Conference
September 15-17, 2003

Introduction

As process feature sizes shrink, design data requirements have increased exponentially. The number of team members needed to build a chip has increased as well. These teams are no longer in single locations and need to collaborate, track changes, and coordinate effectively. Multiple design variants are often needed and the design team must therefore track and propagate changes in shared blocks across their entire portfolio. Many teams are aware of these challenges and have built in-house systems or purchased solutions to help overcome these problems. However, most of these systems are derived from systems that were originally intended for software management. The performance and scalability requirements of modern semiconductor designs are much larger than these systems were designed to handle. This paper describes the key requirements for the configuration management (CM) process from a functionality, performance and scalability perspective for effective hardware design management.

History

Most of the experiences recounted in this paper comes from the design, implementation and deployment of cdsp4, the Cadence-Perforce integration[1][2]. This was developed by the author in 1998 to meet the requirements of a large microprocessor design team at Silicon Graphics, Inc. This CM solution entered the public domain in 2000 and has since been deployed at several companies for large scale design. In some instances, this new CM technology was used to replace proprietary systems and is continuing to prove its value at companies like SGI, nVidia, Apple and Matrix Semiconductor as well as a number of exciting startups such as Aeluros, Pixim, TriCN and Raza Microelectronics. Due to continued strong demand, IC Manage, Inc. was founded in 2003 to develop a major evolution of cdsp4. The company will announce details, availability, and beta programs for this new software in Q4 2003.

Hardware Configuration Management

Hardware configuration management is not always well understood in the IC Industry. It is often associated with version control. Version control is an essential building block of configuration management, but it is not the entire problem. Configuration management should provide a powerful method to manage variants of a design, spanning the entire design cycle from specification to delivery, across the wide variety of engineering disciplines used by a successful semiconductor or IP vendor. Managing variants of a design is key to efficient parallel development and design reuse. [3]

Variant management has some very specific requirements to be effective. It must be fast in its operations such that the computation and data transfer overhead is negligible. It needs to be scalable to support extremely large datasets and numbers of users. It needs to be incremental because otherwise, the task can quickly become over-complicated and a burden for the users, requiring extensive management of policies and procedures.

IC Variant Management

For any given design, there will typically be a large number of variants or sub-variants that share some common design element. The reasons for these variants include process issues; system bugs, timing and cross talk problems, and changed customer requirements. Propagating the right set of changes to the appropriate variant is a very difficult problem without a CM system in place. In many cases, the work is repeated across multiple variants since there is no easy way to apply sets of changes to multiple data sets. This is extremely inefficient and error prone and is often the cause of wasted mask sets in the fab. Version control can help these problems, but much more is really needed.

An example of this issue is a bug that has been found in a particular design that has already been taped out some time earlier. This design was sourced to multiple foundries or even multiple customers, perhaps with a different pad ring. The designers have already fixed the bug in the current tree some time ago. The process of applying the changes related to this bug from the current tree into a number of historical databases is a task that cannot easily be accomplished without a large amount of rework. Since the trees were all just system copies of each other, it is very hard to determine which versions were used. While version control can help identify this problem and reduce the possibility of errors, the rework must still be done in each individual tree. However, with change propagation and dependency tracking support, the task is significantly simplified, allowing the designers to focus on verification and tape out schedules instead of repeating the changes in every tree and hoping they got it right.

As the design progresses towards completion, an exponential growth in the overall amount of data and the number of associated files occur. This typically takes the form of change dump files, various simulation and verification output, and parasitic data. The generation time can be long for this data so it is important to have it under some form of control. In addition, tools may be upgraded and generate slightly incompatible results from the previous versions often making it impossible to reproduce the exact behavior of an older design. Tracking and managing this data is a very important part of the whole IC configuration problem.

As companies tend to produce more than one chip, many design components are shared. Managing these shared components is a time consuming task since the producers of the data have enough to worry about without the extra burden of taking into account all the places where the data might be used. Without powerful techniques at the highest level of abstraction, replication of data sets is unavoidable and adds an extra degree of complexity to IC variant management when it comes to change propagation.

A key feature of true variant management is the ability to handle automatic change propagation between any and all components in an incremental fashion. This includes the requirement for a file, or set of files, to change names and directory structure across any dataset boundary and back again as many times as necessary, while still referring to the same objects. This allows the producers of the data to be decoupled from the low-level implementation requirements, which is a key issue for streamlined and de -centralized development. A simple example is that one project identifies latches with a particular prefix, so that their downstream tools can perform the appropriate timing checks. The library group may release a new latch library, but the project must copy and rename them to suit their needs, creating a fork. If the library group updates the latch library, this process must be repeated until the design is stable, resulting in very slow turnaround times. With incremental change propagation and name mapping, the project can simply synchronize their data with the changed objects and not suffer productivity losses.

Allowing designers to control their workspaces with respect to changes from everyone else is a feature that also reduces time to completion. If, for example, a new power grid is created that has an error in it and one set of designers is working hard to complete LVS or DRC, this change will prevent them from completing the task due to shorts even though the shorts are not related to the actual construction of the block that they are working on. Instead of waiting for the fix, they can synchronize to a previous state, complete their work and move on to another task while a new power grid is being generated. When this data is ready, they can synchronize to it and re-run top level verification. Another advantage of controlled workspaces is the ability to maintain constant data for regression testing of tools. The tool developer can keep the data constant while debugging the flow and then make incremental updates to that workspace to test it further on new sequences of data. Trying to hit a moving target during such testing is difficult and frustrating, significantly slowing time to completion.

CM Architecture

Many of the current commercial offerings for IC data management are based on SCCS follow-ons, such as RCS, RCE or CVS or alternative architectures such as ClearCase. The SCCS derivatives typically have some minimally attached configuration management capabilities or overly complex ones such as in ClearCase that require kernel level interaction to control the variants, resulting in significant speed penalties.

CM is the ability to manage sets of files and their associated dependencies automatically. The traditional first generation method of providing configuration management utilized symbolic names or “tags”. The use of tags for simplistic design and organization is adequate. It fails, however, when design and reuse becomes even slightly complex.

Even though the existing underlying archiving tools have been extended with client server and networking protocols to make them more versatile, they are still based on simplistic mechanisms that are not adequate given the complexity of today’s tasks.

Next generation configuration management solutions are, instead, typically based on a highly scalable relational databases with advanced networking protocols. Both the Open Source cdsp4 DFII integration and IC Manage, the future of cdsp4, use the Perforce solution as the underlying engine. Each of these integrations leverage the unique architectural features of Perforce to provide significant benefits when applied to the hardware design problem. Many of these features require powerful encapsulation to make them usable for IC design, but a technical discussion is crucial to understanding the significant advantages offered by this new architecture.

CM Key Architectural Features

Data integrity

In software builds, the end result is almost always a target or set of targets, compiled together and linked. Usually the compiler can easily detect gross errors in a sample configuration. In hardware design, these errors may often not appear until much later in the build process due to the complexity of the problem and the different disciplines required. Since there is no single compilation process as there is in software design, but instead a highly complex set of inter -dependent transformations it is very important to know early on that a configuration is valid. In RCS- like systems using symbolic names, there is no way to submit sets of file together and guarantee that the set remains as such. Perforce solves this problem through Atomic Transactions. This feature allows sets of files to be tracked and submitted together. Either all of the files will be transferred to the server, or none at all due to all, guaranteeing data integrity.

The key to a good CM system is its ability to track change packages and the ease by which these change packages can be propagated. Even though each file in a tree has its revision history, each revision in its history is only useful in the context of a set of related files. The question “What other source files were changed along with this particular change to a module?” can’t be answered unless the system can track change packages, or sets of files related by a logical change. Change packages, not individual file changes, are the visible manifestation of software and hardware development.

It is important that every transaction in the system is automatically assigned a unique Change Identifier, making it very easy to track changes throughout the entire enterprise. Since Perforce follows a true client-server architecture and metadata is stored on the server, all state can be queried directly through the database rather than having to collect file status information from distributed workspaces across a network.

Variant creation and incremental change propagation

Variants are typically created using some form of system copy command. In traditional archivers that support branching, the archiver is responsible for copying the source object to the target object, thus maintaining a pointer to the parent object version. While this technique is useful for the initial fork, name transformation issues can often complicate the true discovery of parent objects so it is often impossible to decode the parentage after as few as three or four forks. It is also not possible to make incremental updates to the child tree if the parent tree changes. Perforce solves both these problems with a technique known as InterFile Branching(IFB) [5].

IFB is the cornerstone of working with configurations. It begins with the natural model of copying files and renaming them – and finishes with a collection of techniques that make its model usable. In practice, if branching a file means copying it in the repository then the replication of data can make storage space a concern. A simple antidote is to perform a virtual copy, where the newly branched file makes use of the contents of the original file. This requires a level of indirection between the repository name space and the actual underlying object store. When adding a new revision extends the branch, the branched file can acquire its own separate entity in the object store.

Supporting variants with virtual copies reduces the space requirement of a variant to be a mere record for the newly created variant that points to the original file. This virtual copy can also be used when one variant is explicitly synchronized with another. If the user wishes to fold a branch back into a trunk and make the two identical, both can reference the same content at that point.

The change from virtual copy to entity is tracked internally and provides a complete audit trail and can make for comprehensive reporting. Through transitive closure, it is possible to compute for any revision of any file whatever deltas it incorporates from other files through first, second, third, etc. generation merges or replaces.

This powerful feature allows changes between branches to be propagated automatically. There is no need for diffs (often an unusable technique for determining changes on binary objects), or other manual techniques to determine the difference between trees that shared a common fork point. It works seamlessly across design hierarchies too, because Perforce is able to compute and apply the incremental updates between data sets and also report them with ease.

Tags, as used by the currently available commercial solutions in the DFII space are generally very inefficient in comparison to IFB techniques for a number of reasons:

  • They are not incremental, so data must be continually re-tagged
  • It is difficult to visualize or get reporting information on the difference between two or more sets of tags
  • They proliferate and make tree management more complex than it needs to be since policies, scripts and tools must be created to manage the tags themselves.
  • It is a generally slow operation in most databases, both for the creation and recovery of file sets
  • It is an explicit operation – if something wasn’t tagged, it is next to impossible to return to that state. With Perforce all state is recoverable at all times.

Distributed file system approach

Perforce uses stateless connections over TCP to connect workspaces together. All metadata and file system state is centrally stored on the server allowing for easy synchronization of state between clients. IC Manage offers the ability to go back and forward in version space for a single file all the way up to an entire project or even the entire configuration state of a complete enterprise, including recovery of deleted revisions through a powerful user interface.

Performance

Performance is crucial for configuration operations and data delivery. If the data management system becomes a bottleneck, designers and teams will bypass it. The combination of the IC Manage architecture and its integration with Perforce provides unprecedented levels of data throughput, both on the LAN, WAN, VPN or Internet with low cost PC hardware [6].

When a full duplex pipe is available between server and client, three architectural techniques are available for data transfer operations.

Simple RPC

The protocol is to send a request and wait for the response. This incurs the latency of the network for every remote request, which is generally a poor performer over long, high latency networks like the Internet. This is the technique used by HTTP 1.0. To make matters worse, HTTP 1.0 requires a separate connection for each message, and the connection establishment and teardown protocol costs about as much as a message exchange. The performance of this is too slow for any significant effort.

Pipelining RPC

This protocol sends multiple requests and waits for responses. This is the technique used by HTTP 1.1. It’s better than simple RPC, because the latency can be divided by the number of messages that can be sent at once. However, to take real advantage of this scheme, the programmer has to batch up all requests. For Web Pages, the effort is low since all the inline images can be detected and sent in a pipeline of requests. For more complicated transactions it is difficult to do this batching. For other applications using HTTP, or a protocol modeled on HTTP, batching requests requires programmer effort for each request that is going to be launched with a delayed response. Apache’s low -level network I/O is built with support for HTTP 1.1 and will typically yield nearly ideal performance for this type of protocol. However, the performance is still quite limited. Multi-threading can be applied to the problem to improve performance, but the complexity and the bugs introduced grow exponentially.

Message queuing

This protocol builds messages that contain the necessary context so that the receiver does not have to remember its context. The sender simply starts sending them in a stream. The underlying system has to be architected based on a message queuing model, but if this is the case, performance is maximized. The latency penalty is paid only once, and thereafter all transactions are bandwidth limited. Bandwidth limited is the place to be for optimal throughput. Perforce is built on an underlying message queuing architecture and the systems that we have implemented for hardware configuration management utilize it completely. We have obtained performance that is significantly higher than that possible with an HTTP based application.

Geographically distributed sites

Since IC Manage is optimized to run over any kind of TCP link with maximum performance, remote site setup is simple. A proxy-caching server can be setup on the remote LAN which itself communicates to the master server on the WAN or VPN. The proxy makes sure that only one copy of a particular version of an object crosses the long wire and serves it locally to all requesting clients. The administrator of the master site simply sets up permissions and access privileges for the remote site. No other networking, security or NFS support needs to be configured since only a single TCP port connection is required between the master and proxy. This can even be setup with a single command line argument to Secure Shell (ssh) allowing an encrypted communication session completely decoupled from the IT infrastructure.

This is a particularly powerful solution when the remote site is a contractor, outsourcing provider or IP customer allowing them to be active in minutes rather than hours or days.

Integrated Defect Tracking

Change propagation must be tied very closely to defect tracking in order to improve the collaboration and communication aspects of CM. The defect tracking discipline includes change requirements, enhancement requests, design specification changes and bugs. It is crucial to tightly integrate defect tracking with the actual changes in the data set so that designers and project managers can easily synchronize their workspaces to reflect the defect states reported in the tracker. Without integration with the CM system, a defect tracker is at best a searchable text database. With integration, the defect tracker becomes a key component in building configurations that can quickly be tested and verified. IC Manage provides a full cross-linked, scalable search and report mechanism that allows defects to be searched on individual files.

Hierarchical support and ease of use

Ease of use is crucial to the successful implementation of a CM system. If it is complicated or hard to use, adoption by the design teams can hamper success. IC Manage is fully hierarchical and has support for multiple database types and objects. This approach offers a lot of elegance since it can deliver just the right amount of data and configuration parameters that a specific designer needs to complete a task without worrying about the overall assembly of the data and its structure. This is a re-iteration of the producer/ consumer relationship that IC Manage works hard to decouple at all times. It does not require additional layering by in-house design teams to provide a customized solution. Project managers and top-level integration specialists can assemble the appropriate data representations with ease. This fluidity is crucial to avoiding time consuming restructuring and data organization to suit a particular DM solution. IC Manage works by fitting in to customer use models whilst still applying Best Principles for CM.[7]

Software quality and reliability

May studies have found the general quality of EDA software to be low [4]. In the DFII space particularly, customers have had serious issues with database corruption as well as revision control and configuration management induced damage to data. Many project leaders are reluctant to put their trust in CM software because of past experiences, preferring instead to trust in the quality of their low level UNIX system tools such as tar and copy, and then building in-house solutions to provide basic CM-type capabilities. Failures in historic CM systems have serious ramifications for IC completion and have made design teams hesitant to adopt.

There are four key components to solving the quality and reliability problem for CM: Atomicity, Consistency, Isolation, Durability (ACID). It is crucial that the CM system be built using a basis and data storage model that implements these properly and robustly.

Atomicity

The key feature of atomicity in a CM system is that either all the objects are transferred, or none at all. This prevents data from ending up in an inconsistent state due to a software failure, network failure, or hardware failure. There are two specific examples: the first example is in the case where multiple files make up a database representation. Unless all the files are transferred, the underlying database will be corrupt. The second example is the delivery of important configuration state over local or distant networks. If the link is disrupted in some way, the configuration state will be invalid, but this may not be clear to the data consumer or producer until much later resulting in significant losses in productivity.

Consistency

A consistent state of the database can be expected at all times. Without database journalling, checkpoint, and replay support, it is impossible to guarantee consistency in a CM system. A journal is typically a log of all the event transactions that take place. This log, which can be stored in a separate location from the actual file store, is crucial in recovering the CM database state in the event of problems. A checkpoint is the ability to convert the entire binary representation into an ASCII file. This file can then be used to recreate all state by replaying it, along with the journal, if necessary.

Isolation

Transactions are isolated from one another; race conditions are eliminated to ensure that multiple transaction attempts do not collide.

Durable

Once a transaction commits, its updates survive, even if the system experiences a memory or other hardware failure. This overall level of support is not available in any of the existing commercial tools for design data management (DM) in the hardware space and is a major weakness in the approaches currently offered. Failures in the system result in serious consequences to the underlying data.

At the current time, none of the supported customers for cdsp4/IC Manage solution have reported any failures, either in the software, or in the actual configurations that have been built. The uptime reports indicate no restarts for the server software, other than for scheduled maintenance or upgrades. The cdsp4/IC Manage solution delivers all four characteristics for reliability listed above. This results in truly robust and reliable system.

DFII Integration

Intuitive interactive behavior is the key driver of productivity for CM tools. cdsp4 took a very early architectural decision to provide a single point of access for all data management related functionality and IC Manage takes this idea significantly further. cdsp4 uses SKILL List Boxes to implement the Library Manager. Due to the limited GUI abilities available within SKILL, IC Manage uses state of the art C++ based GUI technology to provide a powerful visual environment for interactive use while improving performance considerably over the cdsp4 implementation. In addition, IC Manage directly integrates Perforce seamlessly using its native API, which avoids the need for SKILL based data handling and parsing operations and decouples its own memory usage from that of DFII.

This new GUI appears as a clone of the Cadence Library Manager. It provides direct access to the IC Manage DM functionality but provides key visual information to the users about objects in the browser that is typically not available in standard GDM integrations.

IC Manage facilitates the assembly of large and complex data sets from diverse data sources, not just DFII. All data sources are treated incrementally, so that designers and project managers can quickly see what objects have changed between releases. In existing DM tools that use tag based approaches for releases, the users only see the availability of a new tag and they can either sync to it or not. In IC Manage, users can get reporting information on the changes that make up the differences in releases, allowing them to quickly understand the impact of new data. Since the data can be mapped into any arbitrary name space, it effectively decouples the name space and any hierarchy from what is required by the consumer. This is particularly important when large numbers of projects are involved each with their own specific requirements.

The advantages of such an approach are that the designers can decide how to construct their projects without having tool imposed restrictions. It also makes the support and management of IP resource significantly simpler since different consumers simply have different mapping rules, which allow direct linkage back to the source parent objects.

Co-managed sets

CDB data consists of sets of files that make up a particular view. These files need to be operated on as a group in order to maintain the consistency of the database. Currently available commercial tools typically do not understand the concept of groups of files, nor can they treat them as atomic objects to guarantee consistency of the underlying tool database. These tools handle co-managed sets by introducing additional files throughout the entire design hierarchy to represent the co-managed sets and their associated versions. This additional data significantly increases the file counts in a Cadence library. Performance is also reduced since these files need to be read in and processed for every cellview that needs to be checked out or written for every checkin. In direct contrast, IC Manage uses unique metadata techniques to handle co-managed sets without any additional files. The only measurable or identifiable manifestation of a managed library is the change of the string ‘dmtype none’ to ‘dmtype icm’ in the cdsinfo.tag file.

Resulting Performance

A customer with a full custom design provided the data for this benchmark run. The benchmark was conducted independently by senior industry EDA experts. This data consists of a single DFII library of approximately 30K files and 50 Mbytes of data. The five most common CM tools, currently used in hardware design data management, were compared on identical hardware with 2GB of RAM. The graphs below show the amount of Time to perform the operation. The different Tools are labeled A, B, C, D, and E. The IC Manage solution using Perforce is labeled “p4”.

hardware-configuration-management-1


hardware-configuration-management-2


hardware-configuration-management-3

Analysis of result

s It is clear from the above data that the Perforce architecture is significantly superior. The results clearly indicate the clear symmetry of all operations in Perforce, indicating that it performs equally well on a wide range of operations in contrast to some systems that attempt to optimize certain operations at the expense of others. The most marked example is Tool A that performs well on incremental operations for synchronization, but has very poor checkin performance. It is actually so slow in checkin performance that it is nearly unusable. Tool D appears to perform well with respect to the other contenders, but on closer inspection of the actual numbers, it is slower for too many of the operations by a significant margin.

This simple benchmark does not show performance differentials as the number of connections to the server increases, but our field tests and installed customer base report excellent scalability in this area.

Data scalability and reliability

hardware-configuration-management-4

The table above shows some customer data archive sizes as of February 2003. Neither these customers nor any of the other customers have observed a failure in the software since inception. This is in direct contrast to user reports on popular IC forums for the other CM software packages evaluated.

Conclusion

As designs get larger and more complex, as design processes get more complex, as re-use of internal and external IP becomes crucial to a company’s success, the requirement of utilizing a comprehensive CM system designed for hardware management is a necessity. Cadence-Perforce integrations as provided by the Open Source cdsp4 software and it’s commercial evolution IC Manage provide powerful features that enable designers and managers to improve their design flows, share data worldwide and collaborate with individuals, teams, vendors and companies. The performance, scalability and feature set of these solutions is unmatched.

References

  • Shiv Sikand: Advanced Open Source Design Management for 4.4, International Cadence Users Group, San Jose 2000
  • Shiv Sikand: The Hardware Design Challenge, White Paper, 2002
  • Shiv Sikand, Roger March: File Systems and Perforce, Perforce User Conference, Las Vegas 2003
  • Laura Wingerd, Christopher Seiwald: High-level Best Practices in Software Configuration Management.
2017-02-03T20:09:16+00:00