Amazon FSx for Lustre

Amazon FSx for Lustre delivers the performance to satisfy a wide variety of high-performance workloads. The Lustre file system is optimized for data processing, with sub-millisecond latencies and throughput that scales to hundreds of gigabytes per second. In just a few clicks, you can access high-performance shared storage from tens of thousands of compute instances and scale up storage capacity on-demand.

Amazon FSx for Lustre makes it easy to process your cloud datasets in S3. When linked to an S3 bucket, FSx for Lustre transparently presents objects as files, allowing you to run your workload without managing data transfer from S3. As the contents of your S3 bucket change, FSx for Lustre automatically updates your file system with the latest data available to run your workload.

Amazon FSx for Lustre offers a range of storage options and features to optimize price and performance. Solid-State Drive (SSD) storage is optimized for latency-sensitive workloads. Hard Disk Drive (HDD) storage is optimized for throughput-focused workloads. You can also choose unreplicated, scratch file systems to further lower costs for shorter-term processing of data and data compression to reduce storage consumption of both your file system storage and your  file system backups.

Amazon FSx for Lustre is POSIX-compliant, so you can use your current Linux-based applications without having to make any changes. FSx for Lustre provides a native file system interface and works as any file system does with your Linux operating system. It provides read-after-write consistency and supports file locking. You can access your file systems from Amazon EC2 instances, Amazon EKS clusters, and from on premises using Amazon Direct Connect.

Amazon FSx for Lustre makes it easy to launch and run the high-performance Lustre file system in the cloud. You no longer need to worry about hardware provisioning and maintenance, software configuration, backup management, monitoring storage consumption, and complex performance tuning of file systems. In minutes, you can create and launch an Amazon FSx for Lustre file system by using the Amazon Web Services Management Console, the Amazon CLI, or an Amazon SDK.

Machine learning workloads use massive amounts of training data. These workloads often use shared file storage because multiple compute instances need to process the training datasets concurrently. FSx for Lustre is optimal for machine learning workloads, because it provides shared file storage with high throughput and consistent, low latencies to process the ML training datasets. FSx for Lustre is also integrated with Amazon SageMaker, allowing you to accelerate your training jobs.

High performance computing (HPC) enables scientists and engineers to solve complex, compute-intensive problems. HPC workloads, like oil & gas discovery, and genome analysis, process massive amounts of data that need to be accessed by multiple compute instances with high levels of throughput. FSx for Lustre is ideal for HPC and scientific computing workloads because it provides a file system that’s optimized for the performance and costs of high-performance workloads, with file system access across thousands of EC2 instances. FSx for Lustre also integrates with Amazon Web Services ParallelCluster and Amazon Web Services Batch, making it easy to use with your HPC workloads.

Media data processing workflows, like video rendering, visual effects, and media production, need the compute and storage resources to handle the massive amounts of data being created. FSx for Lustre provides the high performance and low latencies needed for processing, distributing, and analyzing digital media files.

Customers developing autonomous vehicle systems often test models by running simulations and training on massive amounts of vehicle sensor and camera data to ensure vehicle safety. FSx for Lustre enables you to access that data simultaneously from thousands of nodes with high levels of performance, allowing you to more easily run simulations at scale and to accelerate model development.

Big data analytics use cases, including SAS Grid workloads and financial analysis, produce massive volumes of data that need high-performance storage to power data-intensive applications. Processing the ever-increasing volume of data can be costly and complicated. Amazon FSx for Lustre can process large amounts of data at scale and is performance and cost-optimized, providing faster time to discovery and value to your organization.

Electronic design automation (EDA) is a high-performance application used to simulate performance and failures during the design phase of silicon chip production. FSx for Lustre provides the performance and flexibility that enables you to innovate faster, design and verify new products, and scale to meet demand.