What does this Amazon Web Services Solution do?

JuiceFS is a distributed shared file system. JuiceFS provides standard, flexible and fully managed storage service for the Hadoop ecosystem enabling big data platforms to maximize performance on the cloud. In EMR environment, it can support almost all computing engines and it is fully compatible with HDFS. JuiceFS, with its own metadata service combined with Amazon S3, can ensure data consistency and provide better read and write performance, especially in ETL and data analysis scenarios that use Parquet and ORC column storage data formats.

Amazon Web Services Solution overview

This solution allows you to quickly get started with JuiceFS and learn how to use it as the storage backend for Amazon EMR. In addition, you can also run performance test with the script attached in the solution.

Architecture of Serverless Image Handler

Architecture Description

This solution deploys an Amazon EMR cluster on a private subnet, and installs the JuiceFS Plugins on each node of the cluster. The private subnet is accessed through the NAT Gateway on the public network. JuiceFS stores metadata, while Amazon S3 stores user data.

Amazon EMR with JuiceFS

Version 1.0.0
Last updated: 01/2021
Author: Amazon Web Services

Estimated deployment time: 20 min

Source code 

Features

HDFS Compatible

Turns Amazon S3 into a shared file system, POSIX, HDFS and NFS compatible.

Cost Saving

Compared with self-built HDFS, this solution achieves similar performance while saves about 50%-70% of the cost.

Metadata Management

JuiceFS Metadata Service managers the metadata information of your filesystem. The leverages Amazon S3 as object storage.
Standard Product Icons (Features) Squid Ink
Explore all Amazon Web Services Solutions

Browse our portfolio of Amazon Web Services-built solutions to common architectural problems.

Learn more 
Next-Steps-Icon_Find-a-Partner-B
Find a Partner

Find Amazon Web Services certified consulting and technology partners to help you get started.

Learn more 
Standard Product Icons (Start Building) Squid Ink
Start building in the console

Sign-up and start exploring our services.

Get started