Setting Up Managed Lustre Storage on Google Cloud (1/3)

Building Managed Lustre Storage on Google Cloud (1/3) Table of Contents Overview Official Documentation Highlights Hands-On: Running Commands and Troubleshooting 1. Overview 1-1 What Is Google Cloud Managed Lustre? Google Cloud Managed Lustre is a fully managed parallel file system designed for large-scale high-performance computing (HPC) and AI/ML workloads. Powered by the industry-standard Lustre file system (built on DDN EXAScaler) and delivered as a cloud-native service, it provides petabyte-scale capacity and hundreds of GB/s of throughput out of the box—without the operational burden of managing complex storage infrastructure. 1-2 Google Cloud High-Performance Storage Options and Lustre Selection Criteria Google Cloud provides several high-performance storage options tailored to different workload characteristics: Local SSD: Directly attached to the host VM to deliver the lowest latency and highest IOPS. However, the data is ephemeral (temporary) and cannot be shared across multiple VMs. Managed Lustre: Distributes data across numerous storage servers to provide high-throughput parallel I/O. Filestore (High Scale/Enterprise): A fully managed Network File System (NFS) ideal for general-purpose file sharing. While its high-performance tiers deliver strong throughput, it faces scalability limits when thousands of clients access the system simultaneously. Hyperdisk (Extreme Throughput): Next-generation block storage providing high IOPS and throughput, primarily allocated to single VMs with limited shared access capabilities. Cloud Storage FUSE: Mounts nearly infinitely scalable object storage like a local file system via GCS FUSE. While it offers high throughput, its metadata operations and latency are typically slower than dedicated HPC file systems. Storage Option Primary Use Case Key Performance Indicators Local SSD Maximum-performance scratch storage for individual VMs (ephemeral data) Ultra-low latency, highest IOPS per VM Managed Lustre Workloads demanding extreme parallel and sequential I/O (HPC, AI/ML training, media rendering, simulations) Low latency, hundreds of GB/s throughput Filestore (High Scale) Enterprise workloads, GKE persistent volumes, shared file systems High availability, high IOPS, solid throughput (lower than Lustre) Persistent Disk (Extreme/Hyperdisk) Persistent block storage requiring high IOPS Flexible performance tuning, persistence, reliability Cloud Storage FUSE Cost-effective AI/ML data ingestion, checkpoint storage, unlimited-capacity data lakes Infinite scalability, high sequential read throughput (higher latency for metadata operations and random I/O compared to Lustre) 1-3. When Should You Choose Lustre? With multiple high-performance storage options available, deciding when to choose Lustre comes down to specific architectural demands. Managed Lustre is the ideal choice when your workload requires: Extreme Concurrency: Hundreds to thousands of clients (VMs or containers) reading or writing to the exact same file system simultaneously. Maximum Aggregate Throughput: Tens to hundreds of GB/s of evenly distributed throughput that exceeds the limits of Filestore or individual persistent disks. Strict POSIX Compliance: Seamless migration of existing HPC applications to the cloud without rewriting file I/O operations. 1.4 Use Case: Vertex AI Training Cluster and Managed Lustre Vertex AI Training Cluster (VTC) is a product within Vertex AI, Google Cloud's fully managed service, that allows you to provision and operate clusters dedicated to machine learning training. Because it is fully managed, it removes the operational overhead on internal AI infrastructure teams typically needed to deploy and maintain large-scale HPC clusters. You can easily provision clusters using AI accelerator resources, such as GPUs and TPUs. For example, you can deploy and run a GPU cluster with thousands or tens of thousands of data center GPUs without taking on ongoing infrastructure maintenance burdens or their associated operational costs. Large-scale HPC clusters represent one of the primary use cases requiring high-performance storage. In these environments, Managed Lustre serves as a critical storage tier to drive overall cluster efficiency: Maximizing GPU/TPU Utilization (Eliminating I/O Bottlenecks): High read throughput ensures training data feeds continuously into GPUs/TPUs, preventing expensive accelerators from sitting idle waiting on data loads. High-Speed Checkpointing: Large language model (LLM) training on distributed accelerators requires frequent checkpointing to save model states. These checkpoints trigger massive write bursts. Managed Lustre's parallel write capabilities drastically cut checkpoint duration, improving overall training efficiency. Streamlined Configuration: Specifying Managed Lustre as the storage option when creating a VTC connects the storage directly to the cluster's VPC via Private Service Access, enabling immediate access to high-performance storage without complex manual mounting steps. For more details on VTC, which launched in Public Preview in Q4 2025, refer to the Vertex AI Training cluster overview. For information on storage options in VTC, see Vertex AI > Storage. Next: Building Managed Lustre Storage on Google Cloud (2/3)