Amazon FSx for Lustre
Fast and scalable shared storage to power your compute workloads
Amazon FSx for Lustre is a fully managed service that provides cost-effective, high-performance, scalable storage for compute workloads. Many workloads such as machine learning, high performance computing (HPC), video rendering, and financial simulations depend on compute instances accessing the same set of data through high-performance shared storage.
Powered by Lustre, the popular high-performance file system, FSx for Lustre offers sub-millisecond latencies, up to hundreds of gigabytes per second of throughput, and millions of IOPS. It provides multiple deployment options and storage types to optimize cost and performance for your workload requirements.
FSx for Lustre file systems can also be linked to Amazon S3 buckets, allowing you to access and process data concurrently from both a high-performance file system and from the S3 API.
Benefits
Highly-performant and scalable
Accelerate compute workloads with shared storage that provides sub-millisecond latencies, up to hundreds of GBs/s of throughput, and millions of IOPS.
Easy to use with S3 data
Access and process Amazon S3 data from a high-performance file system by linking your file systems to S3 buckets.
Cost-effective
Optimize price and performance with a range of deployment options, including storage type, performance tier, and replication level.
Simple and fully managed
You no longer need to worry about hardware provisioning and maintenance, software configuration, backup management, monitoring storage consumption, and complex performance tuning of file systems. In minutes, you can create and launch an Amazon FSx for Lustre file system.
How it works
Amazon FSx for Lustre provides fully managed shared storage with the scalability and performance of the popular Lustre file system.
Use cases
Machine learning
Machine learning workloads use massive amounts of training data. These workloads often use shared file storage because multiple compute instances need to process the training datasets concurrently. FSx for Lustre is optimal for machine learning workloads, because it provides shared file storage with high throughput and consistent, low latencies to process the ML training datasets. FSx for Lustre is also integrated with Amazon SageMaker, allowing you to accelerate your training jobs.
High performance computing
High performance computing (HPC) enables scientists and engineers to solve complex, compute-intensive problems. HPC workloads, like oil & gas discovery, and genome analysis, process massive amounts of data that need to be accessed by multiple compute instances with high levels of throughput. FSx for Lustre is ideal for HPC and scientific computing workloads because it provides a file system that’s optimized for the performance and costs of high-performance workloads, with file system access across thousands of EC2 instances. FSx for Lustre also integrates with Amazon Web Services ParallelCluster and Amazon Web Services Batch, making it easy to use with your HPC workloads.
Media processing and transcoding
Media data processing workflows, like video rendering, visual effects, and media production, need the compute and storage resources to handle the massive amounts of data being created. FSx for Lustre provides the high performance and low latencies needed for processing, distributing, and analyzing digital media files.
Autonomous vehicles
Customers developing autonomous vehicle systems often test models by running simulations and training on massive amounts of vehicle sensor and camera data to ensure vehicle safety. FSx for Lustre enables you to access that data simultaneously from thousands of nodes with high levels of performance, allowing you to more easily run simulations at scale and to accelerate model development.
Big data and financial analytics
Big data analytics use cases, including SAS Grid workloads and financial analysis, produce massive volumes of data that need high-performance storage to power data-intensive applications. Processing the ever-increasing volume of data can be costly and complicated. Amazon FSx for Lustre can process large amounts of data at scale and is performance and cost-optimized, providing faster time to discovery and value to your organization.
Electronic design automation
Electronic design automation (EDA) is a high-performance application used to simulate performance and failures during the design phase of silicon chip production. FSx for Lustre provides the performance and flexibility that enables you to innovate faster, design and verify new products, and scale to meet demand.
