Unique Underlying Technology
IC Manage PeerCache has three key elements to enable massively parallel workflows.
With Virtual workspaces, the file system interface remains the same, however, the descriptive file metadata is separated from the actual file content. PeerCache allows independent control of the metadata and the file content. It has a separate transport layer to move the file content bytes around.
Virtual workspaces eliminate the need to make physical copies, reducing both copy time and storage requirements by 10-100x, since there is no duplication of content.
They also offer on-demand access. All files are accessible immediately, with content delivered as needed.
IC Manage PeerCache provides a unique peer-to-peer network which takes advantage of the SSD caches that are increasingly finding their way into these compute farms for use as scratch or local storage. It then uses this peer-to-peer network to accelerate delivery of workspace content.
Traditional serial file data transport mechanisms have limited, on demand access speed (reads), and “write” and metadata “stat” performance. Even if you add more file storage to the central filer, you are still limited by the speed of a single NFS thread. It still doesn’t give consistent I/O performance across very large data sets, as most shared storage systems suffer from variable latency as they try to scale, slowing application performance.
Semiconductor companies have compute farms with large numbers of nodes. However, while phones and laptops today use very fast local SSD, either as drives or NVMe (non-volatile memory express), compute nodes today can only use SSD as temporary scratch space. The files cannot be shared between nodes because local storage is not networked. Further, these local silos must be manually managed, copied and backed up.
PeerCache’s breakthrough technology creates a peer-to-peer network out of your compute farm, using the compute nodes’ scratch HDD and/or SSD local storage as caches, delivering parallel clones for almost free.
Virtual workspaces plus this peer-to-peer network with local caching gives development teams near-instant content delivery (reads).
Local Read *and* Write Caching
IC Manage PeerCache has a hybrid architecture, which combines the existing NFS storage infrastructure and local SSD performance — all using software only.
So instead of streaming to a central network filer, we write to the local SSD which automatically backs up to the filer. Thus all “stats”, “reads” and “writes” become local.
Further, the writes are copied back to your existing NFS infrastructure and/or DM system for safe keeping and backup. This maintains the integrity of your data while also making it accessible from any host on the network.
Because the data is written back to the filer, it is fully compatible with your existing infrastructure and best practices.
The local drive serves as a high performance cache for both reads and writes, delivering higher throughput for all of your analysis jobs.
PeerCache supports all major DM systems, both commercial and open source, because it works at the file level, presenting an identical file system interface.