NVIDIA uses the Perforce SCM tools as their primary data management solution. The flexibility and open architecture of this system allows customization for the various needs and methodologies of all departments throughout the company. Its scalability enables efficient support of almost four thousand users. The robustness of the Perforce server allows for extended system uptime with 24/7 usage, limited only by scheduled hardware and OS maintenance requirements.
IC Manage (ICM) solutions further extend the capabilities of Perforce. ICM replication software creates live copies of the Perforce server. These copies provide read-only access to the Perforce database, fail-over nodes, and backup systems that allow daily checkpoint creation with no interruption of Perforce user activity. Use of read-only servers dramatically reduces the load on the main Perforce server, allowing large sync jobs from automated build and test farms to occur without interfering with other Perforce tasks. Overall IC Manage solutions greatly enhance the reliability of the Perforce environment by providing “hot” fail-over nodes and daily backups, and by distributing the server load away from the primary server to multiple read-only nodes.
Initially Perforce was deployed at NVIDIA as a single repository. Its use grew rapidly as users began to appreciate its functionality and ease of use. Source code, documents, binary files – all data that represented value to the company was submitted to Perforce. This made data easy to manage as it was secure and available to all interested users throughout the company. The server was installed on a Sun/Solaris Starfire host with SAN storage connected via a fiber channel link. The Perforce database files and depot were located on a SAN volume which was continuously backed up.
Ultimately the Perforce depot size, large number of users and heavy usage patterns began to cause performance problems. Large numbers of users running sync operations resulted in file system lock contentions on the database and long wait times as a result. A decision was made to split the single Perforce repository into multiple independent servers in order to reduce the load on each individual server, and to help alleviate the lock contention problem. The two largest Perforce server instances after the split were the p4hw and p4sw servers (used primarily by the Hardware engineering and Software engineering groups respectively). A number of additional Perforce servers were created and assigned to other departments. All servers resided on a single Sun/Solaris host with SAN storage for database and depot files.
As the p4hw and p4sw Perforce servers grew larger the performance problems resulting from CPU load, high virtual memory use and file system lock contentions returned. At peak usage times there were hundreds of p4d processes all sharing a single set of host resources. The first action of the IC Manage plan to improve the NVIDIA Perforce environment was to move each Perforce server to a dedicated Linux/x86 host with expanded RAM and SCSI direct attached storage (DAS) for Perforce database files.
The DAS approach was taken specifically to improve file system random I/O performance and reduce access latencies. It was configured as a 10-disk RAID-10 array in an attempt to maximize the sequential throughput and minimize file seek times. Currently the storage subsystem of each Perforce server at NVIDIA is configured as a 12 SAS disk RAID-10 array driven in parallel by two P800 (HP) controllers. Such a configuration yields a 400MB/sec sequential read/write rate and 100MB/sec random read/write rate (as measured by bonnie++).
Sixty four GB of physical memory together with the XFS file system provided good file system caching and further improved overall random I/O performance. In addition eight x86 CPUs were installed in each server to eliminate the possibility of a CPU bottleneck when running large numbers of p4d processes. Perforce depot volumes were left on a Netapp NAS filer mounted over NFS and with Gigabit Ethernet connection to each server. The large size of the depot (over 10 TB) made it impractical to move it to the DAS. The random I/O performance of these volumes was not nearly as critical for Perforce operation as that of the database volume. Currently Perforce servers at NVIDIA are powered by 8 CPUs and 256 GB RAM.
The result of this re-architecture was a significant improvement of Perforce performance immediately evident to the user community. One metric of the improved performance was journal file growth at three to four times the prior rate, suggesting a much larger number of Perforce transactions were being executed daily.
Another IC Manage innovation introduced to the NVIDIA Perforce environment was a fail-over configuration. Each Perforce server and its live replica were paired with a Linux high availability system. The IC Manage software enabled near real-time Perforce replication to its fail-over pair with only a few seconds of maximum lag time at peak loads. In the event of a hardware/software failure on the main node the Linux high availability system would redirect Perforce service to the fail-over node, making this failure practically transparent to users (it cannot be completely transparent since pending Perforce transactions at the moment of failure would break and have to be repeated).
Fail-over instances were installed for the most critical and most heavily used p4hw and p4sw servers. Another feature of the Perforce fail-over configuration was the convenience of software/hardware system upgrades and maintenance, such as CPU, RAM or Linux kernel upgrades. The upgrades were first done on a fail-over node then the Perforce service was switched to the fail-over node allowing the upgrade to be performed on the main server. This approach allowed server downtime to occur without interruption to the user community.
The read-only server was another enhancement initiated by IC Manage. A read-only server is a separate Perforce server running on a dedicated host powered by IC Manage replication software. Practice showed that there were a large number of Perforce clients (both users and automated scripts) that did not submit data or otherwise modify the Perforce database, other than using sync commands. A sync does modify the db.have table in the database and as such is not strictly a read-only operation. A large number of such clients were switched from using the main Perforce server to using its read-only replica. There were two read-only servers installed initially – one for p4hw and another for the p4sw Perforce server. The result was an improvement of the user experience for both the read-only server clients and clients of the primary server. Read-only servers helped to significantly reduce the load on main Perforce servers.
3. Network architecture
All Perforce servers, fail-over nodes and read-only servers are mounted in the same rack cabinets in the data-center. It is preferable that the Perforce server and its fail-over pair are located adjacent to each other because they need to be connected with an Ethernet crossover cable and a serial cable required for the Linux high availability service. Location of the read-only servers is not critical as long as they are connected to the same LAN. Currently the Perforce journal growth rate at NVIDIA is up to 8 GB per day. Such a high rate can cause high replication latencies at peak times when replicating over a WAN, but will not cause a network bottleneck when replicating over a Gigabit Ethernet even if connected to different subnets. Centralized location of all Perforce servers makes it convenient for IT personnel to perform administrative tasks.
4. Fail‐over Perforce servers
One of the first applications of the IC Manage replication technology in NVIDIA was the introduction of fail-over servers. Based on a Linux High-Availability technology the failover server is acting as a “hot swap” for the entire server hosting the Perforce service. The Linux High-Availability cluster typically consists of two identical servers.