This paper describes an ultra high performance OpenSource Software Configuration Management (SCM) system for Cadence DFII 4.4 based on the Perforce Fast SCM System.
SCM techniques are well defined for software development efforts. Hardware design has a different set of requirements mainly due to the complex mix of both ASCII and binary data that eventually defines the physical mask layout of an integrated circuit.
SCM is not to be confused with revision control. Revision control is an important part of SCM, but only one piece of the overall puzzle. An intuitive configuration and release scheme is missing from many of the available tools and particularly those in the DFII space. The range of skills required to build a chip are very wide and many different tools are required to complete the design. There is also a wide range of users, in each skill set, ranging from highly experienced to novice. In order for an SCM system to be successful in such an environment, it needs to be robust, high performance, scalable and have low maintenance requirements. This paper describes such a system developed at SGI and now available as OpenSource, which overcomes the feature and performance limitations of all currently available DFII integrations.
SGI has been wrestling with introducing SCM to the hardware design activity to match the processes used in the IRIX Operating System and Advanced Graphics software divisions. These processes have resulted in an efficient, highly productive and streamlined product design and delivery platform for large software systems.
In the chip design space, we started investigating the use of TDM−LD for 4.3 with Spectrum Services after finding the native 4.3 system to be quite limiting. We then tested a pure TDM environment for 4.4. The biggest obstacle to successful TDM deployment was its speed and complexity. In addition, our goal was to have a common system for both the software and hardware aspects of the chip process. The RTL, CAD and system software teams were extremely reluctant to use it since it did not meet their targets in terms of functionality and performance. We needed a solution that would allow us to manage all aspects of the hardware design flow, not just the layout and schematic data.
Our first goal was to find a tool that would meet our requirements. A plethora of tools are available which make a large number of claims as to their capabilities, but in practice many of these tools are fundamentally flawed both in architecture and execution.
The specifics of the Cadence database, namely co− managed sets of files, was an interesting issue since we wanted a solution that would not hamper performance. We observed that all existing design management (DM) implementations for DFII use an additional file for each co−managed set to express the version dependencies. This almost always has a large performance hit and prevents easy manipulation of the CDB data.
The global requirements were as follows:
- A common tool for all files in the hardware design process. This was a particularly important requirement so that we could have a consistent methodology across Software, RTL, CAD, Verification and Physical Design.
- High performance and scalability since the repository sizes were expected to approach many hundreds of gigabytes with over a hundred users.
- A powerful and flexible release and configuration management architecture.
- Support for geographically distributed design teams.
- A strong procedural interface and/or API for tool integration.
- Low cost (capital and recurring) and low maintenance.
Requirement 1 was the hardest to fulfill due to the political considerations involved, since it touched such a large group of people. However, we felt this was very important ( and the approach has since been vindicated) and a lot of effort was put into co−ordinating the individual team requirements.
Within SGI, a number of tools were already in use.
- Ptools, an in−house RCS derivative
Each of these tools had their own champions and we needed to find a common ground. Ptools was easy to reject since it had no support for binary data. ClearCase was attractive from a feature set point of view but had proved to be a major problem at SGI. The performance was very poor and requiring custom kernels for every OS release was a big issue for machine administration. We experienced corruption with the VOBS on numerous occasions and we also had a full time administrator solely for ClearCase and were anxious to have a better solution. CVS was extremely popular and well liked, but it fundamentally lacked SCM features for release and configuration management which were added on by each group according to their needs. Most of these solution were symbolic name or “tag” based and were unwieldy and inconsistent.
We were already unhappy with TDM for DFII use and thus did not consider expanding its use outside of DFII.
Perforce was originally brought in to SGI because of its ability to run on both NT and IRIX as well as its remote depot ability to support distributed teams. It had a loyal following and the number of licenses in use was growing on a monthly basis.
In addition, we considered
- A custom integration by Spectrum Services CRCS was rejected because it was just RCS as the name suggests. We were impressed by the marketing claims of Synchronicity but were not able to find many people who were willing to recommend it to us, mainly due to its extreme slowness, poor scalability and lack of a cohesive release and configuration scheme. A custom integration was instantly rejected based on a very large six figure quote.
The field was now down to Perforce or a choice of some 30 other available solutions including Continuus, Razor, PVCS, StarTeam and SourceIntegrity.
We chose Perforce over these solutions for a number of reasons:
- Unmatched performance and scalability.
- Atomic transactions. This feature guarantees a group of files that are submitted together will stay that way, and that this information is persistent, unlike ClearCase groups.
- InterFile Branching. A very powerful feature for handling configurations.
- Change based client server architecture. Every transaction in the system is assigned a unique change number, making it very easy to track changes throughout the entire tree. Since state is stored centrally in the server and not in the workspaces, data recovery, mining and reporting tasks are fast and easy to perform.
- Remote depots for geographically distributed teams.
- Appears to run on every single hardware platform known to man.
- Low cost, including a free API and demonstrated low maintenance.
- A very positive and growing user base, both internally and externally to SGI.
The Perforce Advantage
InterFile Branching (IFB) is the cornerstone of this implementation and the reader is directed to  for a more detailed technical explanation.
But first, why branch at all ? 
The answer is quite simple. In order to create workspaces that are controlled by the owner of the workspace and not by the tyranny of the tree, one must branch to create a specific configuration that represents some given state. I use the term tyranny since in a single codeline model where there is just a bunch of versions of a large collection of files, the workspace is a slave to its state of synchronization with that codeline. It can either be in sync, or not. It can easily go into a sync state but will often struggle to return to some previous known state. Most SCCS derivatives (RCS, CVS, RCE) are archivers with such a primitive branching mechanism that they are difficult to use. Instead, symbolic names or “tags” are used to mark collections of files in order to represent state.
Tags are generally very awkward for a number of reasons:
- They are not incremental, so data must be continually re−tagged
- It is difficult to visualize or get reporting information on the difference between two or more sets of tags
- They proliferate and make tree management more complex than it needs to be.
- It is a generally slow operation in most databases, both for the creation and recovery of file sets.
- It is an explicit operation. I.e. if you didn’t tag something, it is next to impossible to return to that state.
- In DFII (or any other database), traversing the hierarchy using Skill or ITK (or any other internal tool) to build a tag set is an inefficient way to map data sets and requires the database to be active in order to make a configuration.
InterFile Branching, coupled with Atomic Transactions and a Change Based architecture, eliminate the need for the majority of tagging operations. Specific state is always recoverable, since every change has a transaction associated with and the Perforce relational database allows you to recover sets of files to any point in time. This is the essence of well implemented configuration management from a tool perspective.
Prior to the introduction of commercial tools to the data management problem, most companies used some combination of tar, copy or move, which we call TCM. This usually requires the notion of a ‘freeze’ followed by a copying or archiving activity. Custom built TCM systems are typically highly effective, but are expensive to maintain and run.
Perforce begins with this natural model − copying software files and renaming them − and finishes it with a collection of techniques that make its model usable. Apart from its fundamental difference in the way files are renamed (which is part of the key to the branching technology) are some key features:
In practice, if branching a file means copying it in the repository then storage space can be a concern. A simple antidote is to perform a virtual copy, where the newly branched file makes use of the contents of the original file. This requires a level of indirection between the repository name space and the actual underlying object store. When the branch is extended by adding a new revision, the branched file can acquire its own separate entity in the object store.
Supporting variants with virtual copies reduces the space requirement of a variant to be merely a record for the newly created variant that points to the original file. This is, more or less, no greater than the cost of a traditional variant.
This virtual copy can also be used when one variant is explicitly synchronized with another. If the user wishes to fold a branch back into a trunk and make the two identical, both can reference the same content at that point.
The change from virtual copy to entity is tracked internally and provides a complete audit trail and can make for comprehensive reporting. Through transitive closure, it is possible to compute for any revision of any file whatever deltas it incorporates from other files through first, second, third, etc. generation merges or replaces.
They allow sets of files to be tracked together. The key to a good SCM system is its ability to track change packages and the ease by which these change packages can be propagated. Even though each file in a codeline has its revision history, each revision in its history is only useful in the context of a set of related files. The question “What other source files were changed along with this particular change to foo.c?” can’t be answered unless you track change packages, or sets of files related by a logical change. Change packages, not individual file changes, are the visible manifestation of software (and by extension, hardware) development.
Perforce and CAD
Since much of CAD data is binary, we choose a branching model that is different to pure software development, namely branch and replace instead of branch and merge. However, it is important to understand that while branch and replace is the underlying configuration model for chip development it does not exclude or limit branch and merge in any way. In fact, our software and RTL teams use branch and merge within their own branches, but a branch and replace model is used to manage the main codeline or mainline.
In previous design flows, we observed that the copying of frozen data to create snapshots was a major requirement, but that these snapshots became stale very quickly, particularly in the early stages of a project. New data was introduced at a rapid rate which invalidated the current snapshot. As we moved closer and closer to tapeout, snapshots tended to be taken less frequently and change was continually feared since small perturbations could very easily result in design corruption. The chip managers were reluctant to allow the introduction of new data while the engineers felt it was imperative. In the end, a balance was usually struck, but was a time consuming process since it involved continuous face to face discussions and the manual tracking of a large number of file dependencies.
Our vision for the snapshot data was to use IFB as the underlying method for replicating managed data in the system.
The integration history automatically allows one to see the pending changes to a snapshot, i.e. only the files that are different in a branch need be integrated back to the mainline and vice−versa.
This allows a very powerful, incremental approach to managing configuration data.
The transaction based nature of branch updates allow transactions in either direction to be easily undone. The importance of this cannot be overstated. In a traditional mainline model, the users have some part of the design tree in their own workspace. Once the mainline is updated and the workspace synchronized, the user typically has no way of going back to the state that used to exist prior to the synchronization. Some form of release management can be used to mitigate this to some extent, but the granularity of such a system is usually very coarse. It does not allow the fine control of a workspace that users typically need to remain productive with respect to mass perturbations introduced from the mainline.
By using IFB to create the workspace, all transactions that update the branch are part of the branch history. This allows total control of the workspace by the user. The synchronization with the mainline is now controlled and updates can be undone at any time to preserve the integrity of the branch.
The user can make edits to files and perform intermediate checkins as a checkpoint mechanism. These edits are essentially invisible to the mainline, but allows a detailed version history to be maintained in the branch. This is typically impossible in a mainline model, since the checkin shows up in the mainline as soon as it is complete. When the user is happy with a certain set of edits, they can typically run some verification procedures and then integrate the set of changes in their branch back into the mainline as an atomic transaction. The mainline changes then typically appear as sets of complete data, rather than a random set of changes to arbitrary files, which is a very elegant alternative to continual file tagging.
An integration policy prevents stale data from being reverse integrated into the mainline. Simply stated, all pending changes into a branch must be forward integrated before any changes can be reverse integrated. However, the policy is site specific and not a tool requirement. Again, since the integration record tracks dependencies, the list of files in either direction is generated automatically, allowing the user to analyze and review the impact of all changes in either direction.
Design Framework II
In 4.4.1, prior to the release of GDM, the customer integration method was through the use of Skill triggers. The Skill trigger model is an excellent fit with the Perforce edit/submit model . We evaluated GDM with the 4.4.2 release and decided to continue to use the Skill triggers to perform our integration since GDM lacked an interactive interface as well as an error recovery mechanism. The only catch with the Skill integration was that tools that did not run Skill would be excluded (i.e. pipo, nino). However, since the majority of our use was Virtuoso and Composer we decided to press ahead and later build a lightweight GDM implementation for the non−Skill tools.
The interface treats all Cadence checkins and checkouts as atomic transactions. This guarantees the correctness of the co−managed sets at all times and gives us the powerful feature set described. 
An additional issue was the Library Manager. We wanted to have access to the versioned system in a similar way to 4.3 and there was very little customization possible with the existing tool, so we decided to build our own browser. These two decisions were key to decoupling us from requiring new releases related to GDM issues and allowing us to build a very high performance interface.
We used the Perforce API to build a custom client that could be run as an IPC process. The new (to 4.4) ipcProcess interface was significantly faster than the hiProcess interface in 4.3 and using the ‘fisrvDoesOp’ feature of GDM 2.0, we were able to avoid any unnecessary forks to execute Perforce commands.
The Skill triggers that we used are as follows:
PostCreateLib, PreDeleteObj, PostDeleteObj, PostCreateObj, PreCheckout, PostCheckin and ddRegUserTrigger.
The file information server that is required by GDM is essentially a specially crafted NULL daemon and thus has virtually no impact on the overall performance of the system.
Our only major gripe with the DFII DM architecture is that both ddCheckin and ddCheckout use filesystem read/write permissions to set their actions. This is also true in a pure GDM system and we would prefer it if there were two additional triggers or other mechanism that could be used to query the file status rather than rely on a permission bit. Additional code had to be added to the user trigger to prevent loss of data when permission bits were accidentally toggled.
- A new library browser (using Skill list boxes) allowing a central point for all DM functions.
- A file synchronization interface that allows the free movement in revision space of a library file, category or cellview, as many times backward of forward as the user may require.
- A library synchronization interface that allows the entire, or selective contents of a library to be restored to any point in time or change number space, as many times backward or forward as the user may require.
- The ability to create unlimited, un−managed versions of the same cell for simultaneous viewing purposes.
- An Integration Wizard that allows change package propagation to be handled automatically, both in the forward and reverse direction with full user control.
- Full support for all auto−checkin and auto− checkout functions as well as the the Skill ddCheckin/ddCheckout interface
- A library/branch−specific change number browser.
- Show checkouts, show versions, checkin, checkout and cancel checkout functionality bound directly to the browser, similar to the 4.3 user interface.
- Interactive operation of all features allowing for error trapping, reporting and recovery in both hiGraphicMode or in nograph mode.
- Ultra high performance does not require any waiting for operations to complete, unlike the commercially available solutions. Most operations complete in centiseconds, including version recovery and branch synchronization.
- Fast, lightweight utilities for CDB files. e.g. showcheckouts, buildbranch, etc.. These can be run from a simple low bandwidth terminal. The average reporting time for our showcheckouts Perl script in our environment, with typically 400−500 checkouts distributed among 50+ users is6 seconds.
- CDB data appears as any other data in the system and can thus be manipulated just like any other file using native Perforce commands.
This is an image of the main Library Browser interface.
Multiple objects can be selected to have the same operation performed on them.
The text field at the top either shows the name of the library client or reports “Not Revisioned”.
The browser is split into four sections, one each for available libraries, categories, cells and cellviews.