Deploying Lustre as a Self-Hosted AWS FSx Alternative

Lustre is an open-source, POSIX-compliant parallel distributed file system for HPC, ML training, and large-scale data processing, it separates metadata from data across dedicated server nodes, scaling aggregate throughput linearly as you add storage targets. AWS FSx for Lustre is the fully-managed equivalent, with per-TiB pricing and lock-in to AWS. This guide deploys a self-hosted Lustre cluster (1 Metadata Server + 2 Object Storage Servers + a client), configures striping/PFL, LNET, quotas/ACLs, Prometheus+Grafana monitoring, Pacemaker HA, IOR/mdtest benchmarks, and covers migrating off FSx. Architecture Lustre separates metadata (filenames, directory structure, permissions, stripe layout) from file data. The Metadata Server (MDS) hosts the Metadata Target (MDT); Object Storage Server (OSS) nodes host Object Storage Targets (OSTs) holding actual file data. Clients talk directly to both MDS and OSS over the LNET network protocol, so I/O flows in parallel across OSTs without a central gateway. AWS FSx Feature Self-Hosted Lustre Equivalent FSx for Lustre File System Lustre file system (MDS + OSS cluster) FSx Metadata MDS + MDT FSx Data Storage OSS + OSTs FSx SSD Storage Class Lustre with LDISKFS on NVMe/SSD FSx Intelligent-Tiering Lustre HSM + Progressive File Layout (PFL) FSx S3 Data Repository Lustre HSM with an S3-compatible backend FSx Auto Import/Export Robinhood Policy Engine or lhsmtool_posix FSx Throughput Scaling Adding OSS/OST pairs FSx Client Mount Lustre kernel client module CloudWatch Metrics Lustre Jobstats + Prometheus exporter Prerequisites: 3 Linux servers (1 MDS + 2 OSS) plus a separate client server, all non-root sudo; a dedicated block storage volume on the MDS (≥10GB) and each OSS (≥20GB each); all nodes connected over a private network (10GbE minimum, InfiniBand recommended for production HPC); comfort with kernel modules and matching kernel-devel to the running kernel. Install Lustre Packages Lustre 2.15 LTS needs a Whamcloud-patched kernel. This uses the Enterprise Linux 8 build (Rocky/AlmaLinux/RHEL 8), check the Whamcloud download site for other distros. Add the Repository On every MDS, OSS, and client node: $ sudo nano /etc/yum.repos.d/lustre.repo [lustre-server] name=Lustre Server baseurl=https://downloads.whamcloud.com/public/lustre/lustre-2.15.8/el8.10/server/ gpgcheck=0 enabled=1 [lustre-client] name=Lustre Client baseurl=https://downloads.whamcloud.com/public/lustre/lustre-2.15.8/el8.10/client/ gpgcheck=0 enabled=1 [e2fsprogs-wc] name=Lustre e2fsprogs baseurl=https://downloads.whamcloud.com/public/e2fsprogs/latest/el8/ gpgcheck=0 enabled=1 Install Server Packages On the MDS and both OSS nodes: $ sudo dnf install -y epel-release $ sudo dnf install -y kernel-4.18.0-553.82.1.el8_lustre kernel-devel-4.18.0-553.82.1.el8_lustre --nogpgcheck $ sudo grubby --set-default /boot/vmlinuz-4.18.0-553.82.1.el8_lustre.x86_64 $ sudo dnf install -y e2fsprogs --enablerepo=e2fsprogs-wc --nogpgcheck $ sudo dnf install -y lustre-osd-ldiskfs-mount lustre kmod-lustre-osd-ldiskfs --nogpgcheck $ echo "exclude=kernel-4.18.0*" | sudo tee -a /etc/dnf/dnf.conf The kernel pin is critical — Lustre modules build against a specific kernel, and a routine distro kernel update will break the cluster on next reboot. $ sudo reboot Wait ~60s, reconnect: $ uname -r Should show 4.18.0-553.82.1.el8_lustre.x86_64. $ sudo modprobe lustre $ sudo lsmod | grep lustre Confirms lustre, lnet, obdclass, etc. loaded. Install Client Packages Client node only — uses the stock distro kernel: $ sudo dnf install -y epel-release $ sudo dnf install -y kmod-lustre-client lustre-client --nogpgcheck $ sudo modprobe lustre $ sudo lsmod | grep lustre $ echo "exclude=kernel-4.18.0*" | sudo tee -a /etc/dnf/dnf.conf Open the LNET Port On every node — firewalld blocks TCP 988 by default: $ sudo firewall-cmd --permanent --add-port=988/tcp $ sudo firewall-cmd --reload Important: LNET binds to each server's primary interface by default — typically the public interface on a cloud instance. The MGS registers its network ID (NID) on that interface, so whatever MDS-IP-ADDRESS you use to format OSTs and mount clients must match it. To route Lustre traffic over a private network instead, configure LNET on every node before formatting any target: create /etc/modprobe.d/lustre.conf with options lnet networks=tcp0(INTERFACE) (replace INTERFACE with your private-network interface name from ip addr, e.g. enp8s0), then reboot. Use that node's private address as MDS-IP-ADDRESS throughout. Deploy the Metadata Server The MDS hosts both MGS (coordinates file system registration) and MDT (stores namespace/stripe layout) roles. $ lsblk Identify the dedicated block device — commonly /dev/vdb on virtio-based cloud block storage, with the OS on /dev/vda. Use whatever your lsblk output shows. $ sudo mkfs.lustre --fsname=lustrefs --mgs --mdt --index=0 /dev/vdb $ sudo mkdir -p /mnt/mdt $ sudo mount -t lustre /dev/vdb /mnt/mdt $ sudo lctl dl Should list mgs, mds, mdt, osd-ldiskfs as active. Deploy Object Storage Servers First OSS (index 0) $ lsblk $ sudo mkfs.lustre --fsname=lustrefs --ost --index=0 --mgsnode=MDS-IP-ADDRESS@tcp /dev/vdb $ sudo mkdir -p /mnt/ost0 $ sudo mount -t lustre /dev/vdb /mnt/ost0 $ sudo lctl dl Should show obdfilter, ost, and an mgc connection to the MGS. Second OSS (index 1) $ sudo mkfs.lustre --fsname=lustrefs --ost --index=1 --mgsnode=MDS-IP-ADDRESS@tcp /dev/vdb $ sudo mkdir -p /mnt/ost1 $ sudo mount -t lustre /dev/vdb /mnt/ost1 From the MDS, confirm both OSTs registered: $ sudo lctl get_param osc.*.ost_server_uuid Both should show FULL state. Mount on Clients $ sudo mkdir -p /mnt/lustre $ sudo mount -t lustre MDS-IP-ADDRESS@tcp:/lustrefs /mnt/lustre $ df -h /mnt/lustre $ lfs df -h /mnt/lustre lfs df breaks down capacity per MDT/OST — useful for spotting imbalances. $ sudo dd if=/dev/zero of=/mnt/lustre/testfile bs=1M count=100 $ sudo rm /mnt/lustre/testfile Configure Striping and PFL Striping spreads a file across multiple OSTs for aggregate throughput; Progressive File Layout (PFL) applies different stripe counts per file-size range — the self-hosted equivalent of FSx Intelligent-Tiering. Run on the client. Uniform striping: $ sudo mkdir -p /mnt/lustre/striped $ sudo lfs setstripe -c 2 -S 1M /mnt/lustre/striped $ sudo lfs getstripe -d /mnt/lustre/striped -c 2 spreads each file across 2 OSTs; -S 1M sets a 1MiB stripe size. Progressive layout (3-tier): $ sudo mkdir -p /mnt/lustre/pfl $ sudo lfs setstripe -E 4M -c 1 -E 64M -c 2 -E -1 -c 2 /mnt/lustre/pfl $ sudo lfs getstripe -d /mnt/lustre/pfl Files ≤4MiB use 1 stripe (avoids overhead on small files), ≤64MiB use 2, larger use 2 for remaining extents. LNET Networking $ sudo lnetctl net show $ sudo lnetctl net show --verbose $ sudo lctl ping MDS-IP-ADDRESS@tcp A successful ping confirms LNET routes to the MDS. Multi-rail bonds multiple interfaces for redundancy/bandwidth. InfiniBand deployments use o2ib instead of tcp. Quotas and ACLs Enable and Apply Quotas From the MDS: $ sudo lctl conf_param lustrefs.quota.mdt=ug $ sudo lctl conf_param lustrefs.quota.ost=ug From the client: $ sudo lfs setquota -u USER-NAME -b 100M -B 200M -i 10000 -I 20000 /mnt/lustre $ sudo lfs quota -u USER-NAME /mnt/lustre -b/-B are soft/hard block limits; -i/-I are soft/hard inode (file count) limits. POSIX ACLs $ sudo mkdir -p /mnt/lustre/acltest $ sudo setfacl -m u:nobody:rwx /mnt/lustre/acltest $ sudo getfacl /mnt/lustre/acltest Monitoring with Prometheus + Grafana The self-hosted equivalent of CloudWatch for FSx. Enable Jobstats On MDS and client: $ sudo lctl set_param jobid_var=procname_uid Tags each I/O op with process name + UID (e.g. dd.0). Build and Run the Lustre Exporter On the MDS: $ sudo dnf install -y golang git wget --nogpgcheck $ git clone https://github.com/GSI-HPC/lustre_exporter.git ~/lustre_exporter $ cd ~/lustre_exporter $ GOTOOLCHAIN=auto go build $ ls -la lustre_exporter $ sudo cp ~/lustre_exporter/lustre_exporter /usr/local/bin/ $ sudo nano /etc/systemd/system/lustre_exporter.service [Unit] Description=Lustre Prometheus Exporter After=network.target [Service] Type=simple ExecStart=/usr/local/bin/lustre_exporter Restart=on-failure [Install] WantedBy=multi-user.target $ sudo systemctl daemon-reload $ sudo systemctl enable --now lustre_exporter $ curl -s http://localhost:9169/metrics | grep -E "^lustre_" | head -20 Deploy Prometheus $ wget https://github.com/prometheus/prometheus/releases/download/v3.13.1/prometheus-3.13.1.linux-amd64.tar.gz $ tar -xzf prometheus-3.13.1.linux-amd64.tar.gz $ sudo mv prometheus-3.13.1.linux-amd64 /opt/prometheus $ sudo nano /opt/prometheus/prometheus.yml global: scrape_interval: 15s scrape_configs: - job_name: 'lustre' static_configs: - targets: ['localhost:9169'] $ sudo nano /etc/systemd/system/prometheus.service [Unit] Description=Prometheus After=network.target [Service] Type=simple ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --storage.tsdb.path=/opt/prometheus/data Restart=on-failure [Install] WantedBy=multi-user.target $ sudo chcon -t bin_t /opt/prometheus/prometheus $ sudo systemctl daemon-reload $ sudo systemctl enable --now prometheus $ curl -s http://localhost:9090/api/v1/targets | grep '"health"' Should show "health":"up". Install Grafana $ sudo nano /etc/yum.repos.d/grafana.repo [grafana] name=grafana baseurl=https://rpm.grafana.com repo_gpgcheck=1 enabled=1 gpgcheck=1 gpgkey=https://rpm.grafana.com/gpg.key sslverify=1 sslcacert=/etc/pki/tls/certs/ca-bundle.crt $ sudo dnf install -y grafana --nogpgcheck $ sudo systemctl enable --now grafana-server $ sudo grafana cli --homepath /usr/share/grafana admin reset-admin-password ADMIN-PASSWORD $ sudo firewall-cmd --permanent --add-port=3000/tcp $ sudo firewall-cmd --reload Open http://MDS-IP-ADDRESS:3000, log in as admin: Connections → Data sources → Add data source → Prometheus, URL http://localhost:9090, Save & test. Dashboards → New → New dashboard → Add visualization, Prometheus source, Code mode, query lustre_capacity_kibibytes, Run queries, Save. High Availability with Pacemaker Active-passive MDS failover via Pacemaker/Corosync — requires a second MDS and a block device both nodes can reach. Shared-storage failover needs a block device attached to both MDS nodes simultaneously (SAN, iSCSI target, or multi-attach volume). Most cloud block storage attaches to one instance at a time, so treat the shared MDT resource here as illustrative unless your infra supports multi-attach — the cluster stack itself still deploys and forms quorum as shown, ready to pair with shared storage where available. Install Cluster Packages On both MDS nodes: $ sudo dnf config-manager --set-enabled ha $ sudo dnf install -y pacemaker pcs corosync fence-agents-all --nogpgcheck $ sudo firewall-cmd --permanent --add-service=high-availability $ sudo firewall-cmd --reload $ sudo systemctl enable --now pcsd $ echo "hacluster:CLUSTER-PASSWORD" | sudo chpasswd Important: Corosync needs each node's hostname to resolve to one consistent IPv4 address on the cluster network — not loopback. Cloud-init often maps hostnames to 127.0.0.1/::1 in /etc/hosts, which breaks Corosync with a "different IP families" error. Fix: set manage_etc_hosts: false in /etc/cloud/cloud.cfg, remove loopback entries for the node hostnames, and add one IPv4 entry per node pointing at its private-network address (e.g. 10.42.0.4 lustre-mds, 10.42.0.7 lustre-mds2). Form the Cluster From the primary MDS: $ sudo pcs host auth MDS-PRIMARY-HOSTNAME MDS-SECONDARY-HOSTNAME -u hacluster -p CLUSTER-PASSWORD $ sudo pcs cluster setup lustre-cluster MDS-PRIMARY-HOSTNAME MDS-SECONDARY-HOSTNAME --force $ sudo pcs cluster start --all $ sudo pcs cluster enable --all $ sudo pcs status Both nodes should show Online with quorum. Define the MDT Resource $ sudo pcs resource create mdt-resource ocf:heartbeat:Filesystem device=/dev/SHARED-DEVICE directory=/mnt/mdt fstype=lustre op monitor interval=30s Production needs STONITH fencing matched to your infrastructure — the default stonith-enabled=false is for initial verification only. Verify with IOR and mdtest Build $ sudo dnf install -y gcc gcc-c++ make openmpi openmpi-devel git autoconf automake libtool --nogpgcheck $ source /etc/profile.d/modules.sh $ module load mpi/openmpi-x86_64 $ git clone https://github.com/hpc/ior.git ~/ior $ cd ~/ior $ git checkout 4.0.0 $ ./bootstrap $ ./configure $ make Run $ sudo mkdir -p /mnt/lustre/iortest-dir $ sudo chmod 777 /mnt/lustre/iortest-dir $ cd ~/ior/src $ ./ior -w -r -t 1M -b 100M -F -o /mnt/lustre/iortest-dir/testfile -w -r write+read test, -t 1M transfer size, -b 100M block size/process, -F file-per-process. $ sudo mkdir -p /mnt/lustre/mdtest-dir $ sudo chmod 777 /mnt/lustre/mdtest-dir $ ./mdtest -n 100 -i 2 -d /mnt/lustre/mdtest-dir Reports directory/file creation, stat, and removal rates. PSM3 warnings like Failed to get eth0 (unit 0) cpu set on cloud Ethernet are harmless — just OpenMPI falling back from InfiniBand-style transport. Migrating from AWS FSx for Lustre Data: FSx exports to S3 via a Data Repository Association (aws fsx create-data-repository-task --type EXPORT_TO_REPOSITORY); pull it down on a Lustre-mounted client with aws s3 sync --endpoint-url , which works against any S3-compatible store. Parallelize across clients/prefixes for large datasets, or use lfs migrate + rsync between two mounted file systems directly. Clients: unmount FSx, mount -t lustre MDS-IP-ADDRESS@tcp:/lustrefs in its place — same POSIX interface, no app code changes. S3 tiering: Lustre HSM replaces the FSx S3 Data Repository. HSM coordinator runs on the MDS (mdt.*.hsm_control), a copytool (lhsmtool_posix) runs on a client, backend is any S3-compatible bucket exposed via FUSE (e.g. s3fs). lfs hsm_archive copies to S3, lfs hsm_release frees the local copy, reads transparently restore from S3. Robinhood automates archive/release policy. Note: s3fs backends log harmless fsetxattr ... Operation not supported during archive (no xattr support) — archive/restore still succeed. Sizing: ~2KiB MDT space per file (budget 1GiB MDT per million files); sum OST capacity + 20% overhead; add capacity by formatting new OSTs with the next sequential index — the MGS picks them up automatically. Watch for: Scaling: FSx auto-scales. Self-hosted needs manual OST additions — size for projected growth upfront S3 sync: FSx syncs bidirectionally automatically. Self-hosted HSM needs explicit archive/restore or a policy engine Kernel compat: client modules must match the client kernel; pin server kernels LNET: default TCP is fine for most cloud setups. InfiniBand needs matching config on every node Metrics: swap CloudWatch for the Prometheus exporter + Grafana Backups: no automated snapshots like FSx — use lfs find + tar/rsync for data, LVM/LDISKFS snapshots for metadata Cost: FSx is per-TiB. Self-hosted cost is compute+storage — calculate your breakeven Next Steps The cluster delivers parallel I/O across OSTs with the full Lustre POSIX interface FSx workloads expect. From here: Explore Distributed Namespace (DNE) for scaling metadata across multiple MDS nodes Tune network striping for multi-rail LNET configurations Automate HSM archive/release policy with Robinhood instead of manual lfs hsm_* calls For the full guide, visit the original article on Vultr Docs.