Zettar to Showcase Accelerating AI Data Migration at Scale at SC21

PALO ALTO, Calif., November 14, 2021 – Zettar today announced that it will showcase a fully optimized, large-scale enterprise AI data migration solution, developed in collaboration with DDN and NVIDIA, at Supercomputing 2021 ( November 14-19, 2021).

Research focused on NVIDIA technology is gaining more and more importance and momentum in industries and universities. NVIDIA DGX SuperPOD, the first cloud-native multi-tenant AI supercomputer and turnkey data center integrated with DDN AI400X AI storage devices using NVIDIA Quantum InfiniBand Networking, is the de facto standard for large-scale AI supercomputing in data-intensive enterprises.

With the rapidly growing and increasingly large data sets for AI training and processing, powering GPUs in a timely and adequate manner has become a challenge. This is especially true when data needs to be migrated from the edge, retrieved from the cloud, ingested from other on-premises storage, or balanced between existing storage and newly added storage. In such circumstances, the use of simple, scalable and efficient data transfer software is essential to increase the productivity of large-scale AI research.

Zettar’s AI data migration solution, developed in collaboration with DDN and NVIDIA, is production ready, using commercially available hardware and software, and achieves full storage and network performance. It is compact, consisting of only 11 rack units: 8 RU for the 4 AI400x, 1 RU for the NVIDIA Quantum 8700 InfiniBand switch and 2 RU for the servers. It is also linearly scalable and highly available, with a distributed and modular architecture that offers intelligent task management, is easy to use and secure, and provides detailed reports. Plus, it covers all common use cases: single-site, multi-site, and multi-cloud.

The companies began a solution test bed on October 15. Ten days later, the test bench was moving at 2.9PB / day (33Gb / s). With further tuning, it is expected to achieve a data migration rate of 4PB / day (50Gb / s) in NVIDIA GTC next spring, a 34% increase using the same hardware and software. A recorded flash speech is available at NVIDIA SC21 Virtual Theater. There will be a follow-up white paper that describes the solution, test procedures and results, as well as Zettar’s open source tools, including the elbencho storage scan tools.

“When data needs to be migrated from any other storage service, such as NAS systems, a parallel file system, or object storage, over any distance, the basic architecture of the solution is still applicable. Once the data is in DDN storage, excellent NVIDIA DGX SuperPOD AI processing efficiency is expected, ”said Chin Fang, Founder and CEO of Zettar Inc.

Zettar’s software uses NVIDIA Quantum InfiniBand technology to deliver high-performance, large-scale data migration even over long distances when using NVIDIA MetroX-2 for edge-to-data center use cases Said Gilad Shainer, SVP, Marketing, NVIDIA.

“The Zettar solution is an ideal complement to our A3I storage appliances,” said Kurt Kuckein, senior vice president of marketing at DDN. “The movement of data is a frequent challenge for our customers, as they operate at a scale that typical data management tools do not capture. High-speed data transfer to a centralized repository allows customers to access more of their data faster, which in turn generates faster and more accurate results.

Link to the presentation: here.

About Zettar

Zettar’s Data transfer software has gained a worldwide and unmatched reputation in the high-end HPC space for being the simplest, most efficient and versatile data manager to move data at scale, at the speed of the light. For more information, contact [email protected], [email protected] or [email protected]

Source: Zettar

Sean N. Ayres