We use machine learning technology to do auto-translation. Click "English" on top navigation bar to check Chinese version.
Amazon Web Services Glue crawlers support cross-account crawling to support data mesh architecture
Data lakes have come a long way, and there’s been tremendous innovation in this space. Today’s modern data lakes are cloud native, work with multiple data types, and make this data easily available to diverse stakeholders across the business. As time has gone by, data lakes have grown significantly and have evolved to data meshes as a way to scale.
Data mesh advocates for decentralized ownership and delivery of enterprise data management systems that benefit several personas. Data producers can use the data mesh platform to create datasets and share them across business teams to ensure data availability, reliability, and interoperability across functions and data subject areas. Data consumers now have better data sharing with data mesh and federation across business units without compromising data security. The data governance team can support distributed data, where all data is accessible to those with the proper authority to access it. With data mesh, data doesn’t have to be consolidated into a single data lake or account and can remain within different databases and data lakes. An essential capability needed in such a data lake architecture is the ability to continuously understand changes in the data lakes in various other domains and make those available to data consumers. Without such a capability, manual work is needed to understand producers’ updates and make them available to consumers and governance.
Amazon Web Services customers use a
In the post
In this post, we walk through the creation of a simplified data mesh architecture that shows how to use an Amazon Web Services Glue crawler with Lake Formation to automate bringing changes from data producer domains to data consumers while maintaining centralized governance.
Solution overview
In a data mesh architecture, you have several producer accounts that own S3 buckets, several consumer accounts who wants to access shared datasets, and a central governance account to manage data shares between producers and consumers. This central governance account doesn’t own any S3 bucket or actual tables.
The following figure shows a simplified data mesh architecture with a single producer account, a centralized governance account, and a single consumer account. The data mesh producer account hosts the encrypted S3 bucket, which is shared with the central governance account. The central governance account registers the S3 bucket with Lake Formation using an
In the following sections, we provide
Prerequisites
Complete the following steps in each account (producer, central governance, and consumer) to update the Data Catalog settings to use Lake Formation permissions to control catalog resources instead of IAM-based access control:
- Sign in to the Lake Formation console as admin.
- If this is the first time accessing the Lake Formation console, add yourself as the data lake administrator.
- In the navigation pane, under Data catalog , choose Settings .
- Uncheck Use only IAM access control for new databases .
- Uncheck Use only IAM access control for new tables in new databases .
- Keep Version 3 as the current cross-account version.
- Choose Save .
Set up resources in the central governance account
The CloudFormation template for the central account creates a CentralDataMeshOwner
user assigned as Lake Formation admin. The CentralDataMeshOwner
user in the central governance account performs the necessary steps to share the central catalogs with the producer and consumer accounts. The CentralDataMeshOwner
user also sets up a custom Lake Formation service role to register the S3 data lake location. Complete the following steps:
- Log in to the central governance account console as IAM administrator.
- Choose Launch Stack to deploy the CloudFormation template:
- For DataMeshOwnerUserName , keep the default (
CentralDataMeshOwner
). - For ProducerAmazon Web ServicesAccount , enter the producer account ID.
- Create the stack.
- After the stack launches, on the Amazon Web Services CloudFormation console, navigate to the Resources tab of the stack.
- Note down the value of
RegisterLocationServiceRole
. - Choose the
LFUsersPassword
value to navigate to theAmazon Web Services Secrets Manager console. - In the Secret value section, choose Retrieve secret value .
- Note down the secret value for the password for IAM user
CentralDataMeshOwner
.
Set up resources in the producer account
The CloudFormation template for the producer account creates the following resources:
- IAM user
LOBProducerSteward
- S3 bucket
retail-datalake-<producer account id >-<producer region>
- KMS key used for bucket encryption
- Required S3 bucket policies to provide access to the central governance account
- Amazon Web Services Glue crawler and crawler IAM role with necessary permissions
Complete the following steps:
- Log in to the producer account console as IAM administrator.
- Choose Launch Stack to deploy the CloudFormation template:
- For CentralAccountID , enter the central account ID.
- For CentralAccountLFServiceRole , enter the value of
RegisterLocationServiceRole
from CloudFormation noted earlier. - Create the stack.
- When the stack is complete, on the Amazon Web Services CloudFormation console, navigate to the Resources tab of the stack.
- Note down the
AWSGlueServiceRole
value. - Choose the
ProducerStewardUserCredentials
value to navigate to the Secrets Manager console. - In the Secret value section, choose Retrieve secret value .
- Note down the secret value for the password for IAM user
LOBProducerSteward
. - On the Amazon S3 console, check the bucket policies for
retail-datalake-<producer account id >-<producer region>
and make sure it is shared with the central governance account IAM role.
This is required for registering the bucket with Lake Formation in the central account so that the account can manage the data sharing.
- On the Amazon Web Services KMS console, check that the bucket is encrypted with the customer managed key and the key is shared with the central governance account.
Set up resources in the consumer account
The CloudFormation template for the consumer account creates the following resources:
- IAM user
ConsumerAdminUser
assigned to the data lake admin - IAM user
LFBusinessAnalyst1
- S3 bucket for Athena output
- Athena workgroup
Complete the following steps:
- Log in to the consumer account console as IAM administrator.
- Choose Launch Stack to deploy the CloudFormation template:
- Create the stack.
- When the stack is complete, on the Amazon Web Services CloudFormation console, navigate to the Resources tab of the stack.
- Choose the
AllConsumerUsersCredentials
value to navigate to the Secrets Manager console. - In the Secret value section, choose Retrieve secret value .
- Note down the secret value for the password for the IAM user
ConsumerAdminUser
.
Now that all the accounts have been set up, we set up cross-account sharing on Amazon Web Services with a central governance account to manage sharing of permissions across producers and consumers.
Configure the central governance account to manage sharing with the producer account
Sign in to the central governance account as CentralDataMeshOwner
using the password noted earlier through the central governance account CloudFormation stack. Then complete the following steps:
- On Lake Formation console, choose Data lake locations under Register and ingest in the navigation pane.
- For Amazon S3 path , provide the path
retail-datalake-<producer account id >-<region>
. - For IAM role , choose the IAM role created using the CloudFormation stack.
This role has permissions for the accessing the encrypted S3 bucket and its key. Do not choose the role AWSServiceRoleForLakeFormationDataAccess
.
- Choose Register location .
- In the navigation pane, choose Databases .
- Choose Create database .
- For Database name ¸ enter
datameshtestdatabase
. - Choose Create database .
- In the navigation pane, choose Data locations and choose Grant .
- Select External account and provide the producer account for Amazon Web Services account ID, Amazon Web Services organization ID, or IAM principal ARN .
- For Storage location , provide the data lake bucket path.
- Select Grantable , then choose Grant .
- Choose Data lake permissions , then choose Grant .
- Select External accounts and provide the producer account number.
- For Databases, choose
datameshtestdatabase
. - For Database permissions and Grantable permissions , select Create table , Alter , and Describe .
- Choose Grant .
Configure the crawler in the producer account to populate the schema
Sign in to producer account as LOBProducerSteward
with the password noted earlier through the producer account CloudFormation stack, then complete the following steps:
- On the Amazon Web Services RAM console, accept the pending resource share from the central account.
- On the Lake Formation console, choose Databases under Data catalog in the navigation pane.
- Choose
datameshtestdatabase
, and on the Action menu, choose Create resource link . - For Resource link name , enter
datameshtestdatabaselink
. - Choose Create .
- On the Amazon Web Services Glue console, choose Crawlers in the navigation pane.
- Choose the crawler
CrossAccountCrawler-<accountid>
. - Choose Edit , then choose Configure security settings .
- Select Use Lake Formation credentials for crawling S3 data source .
- Select In a different account and provide the account ID of the central governance account.
- Choose Next .
- Choose
datameshtestdatabaselink
as the database and choose Update .
- In the navigation pane, choose Data locations and choose Grant .
- Select My account , and choose the crawler IAM role for IAM users and roles .
- For Storage locations , choose the bucket
retail-datalake-<accountid>-<region>
. - For Registered account location , enter the central account ID.
- Choose Grant .
Alternatively, you can also use the Amazon Web Services CLI to grant data location permission on bucket registered in central account to the crawler role using below command:For using CLI, refer to Installing or updating the latest version of the
Amazon Web Services CLI . - In the navigation pane, choose Data lake permissions .
- Choose the crawler IAM role for the principal account.
- Choose
datameshtestdatabase
for the database. - For Database permissions , select Create , Describe , and Alter .
- Choose Grant .
- Choose the crawler IAM role for the principal account.
- Choose
datameshtestdatabaselink
for the database. - For Resource link permissions , select Describe .
- Choose Grant .
- Run the crawler.
The following screenshot shows the details after a successful run.
When the crawler is complete, you can validate the table created under the database datameshtestdatabaselink
.
This table is owned by the producer account and available in the central governance account under the shared database datameshtestdatabase
. Now the data lake admin in the central governance account can share the database and populated table with the consumer account.
Configure the central governance account to manage sharing of read-only access with the consumer account
Sign in to the central governance account as CentralDataMeshOwner
with the password noted earlier through the central governance account CloudFormation stack, then complete the following steps:
- Grant database permissions to the consumer account.
- For Principals , choose external account and provide <consumer accountID>
- For Databases , select
datameshtestdatabase
. - For Database permissions , select Describe .
- For Grantable permissions ¸ select Describe .
- Choose Grant .
- Grant table permissions to the consumer account.
- For Principals , choose external account and provide
<consumer accountID>
. - For Databases , select
datameshtestdatabase
. - For Tables , select
retail_datalake_<accountID>_<region>
. - For Table permissions , select Select and Describe .
- For Grantable permissions ¸ select Select and Describe .
- Choose Grant .
Configure the consumer account as the consumer account data lake admin
Sign to the consumer account as ConsumerAdminUser
with the password noted earlier through the consumer account CloudFormation stack. (Note that in the consumer account Lake Formation configuration, both ConsumerAdminUser
and LFBusinessAnalyst1
have the same password.)
- On the Amazon Web Services RAM console, accept the resource share from the central account.
- On the Lake Formation console, validate that the shared database
datameshtestdatabase
is available and create the resource linkdatameshtestdatabaselink
using the shared database.
The following screenshot shows the details after the resource link is created.
- On the Lake Formation console, choose Grant .
- Choose LFBusinessAnalyst1 for IAM users and roles .
- Choose
datameshtestdatabase
for the database under Named data catalog resources . - Select Describe for Database permissions .
- On the Lake Formation console, choose Grant .
- Choose
LFBusinessAnalyst1
for IAM users and roles . - Choose
datameshtestdatabaselink
for the database under Named data catalog resources . - Select Describe for Resource link permissions .
- On the Lake Formation console, choose Grant .
- Choose
LFBusinessAnalyst1
for IAM users and roles . - Choose
retail_datalake_<accountid>_<region>
for the table under Named data catalog resources . - Select Select and Describe for Table permissions .
Run queries in the consumer account
Sign to the consumer account console as LFBusinessAnalyst1
with the password noted earlier through the consumer account CloudFormation stack, then complete the following steps:
- On the Athena console, and choose
lfconsumer-workgroup
as the Athena workgroup. - Run the following query to validate access:
We have successfully registered the dataset and created a Data Catalog in the central governance account. We crawled the data lake that was registered with the central governance account using Lake Formation permissions from the producer account and populated the schema. We granted Lake Formation permission on the database and table from the central account to the consumer user and validated consumer user access to the data using Athena.
Clean up
To avoid unwanted charges to your Amazon Web Services account, delete the Amazon Web Services resources:
- Sign in to the CloudFormation console as the IAM admin used for creating the CloudFormation stack in all three accounts.
- Delete the stacks you created.
Conclusion
In this post, we showed how to set up cross-account crawling using a central governance account with the new Amazon Web Services Glue crawler capability of Lake Formation integration. This capability allows data producers to set up crawling capabilities in their own domain so that changes are seamlessly available to data governance and data consumers. Implementing a data mesh with Amazon Web Services Glue crawlers, Lake Formation, Athena, and other analytical services provide a well-understood, performant, scalable, and cost-effective solution to integrate, prepare, and serve data.
If you have questions or suggestions, submit them in the comments section.
For more resources, refer to the following:
-
Design a data mesh architecture using Amazon Web Services Lake Formation and Amazon Web Services Glue -
Introducing Amazon Web Services Glue crawlers using Amazon Web Services Lake Formation permission management
About the authors
Sandeep Adwankar is a Senior Technical Product Manager at Amazon Web Services. Based in the California Bay Area, he works with customers around the globe to translate business and technical requirements into products that enable customers to improve how they manage, secure, and access data.
Srividya Parthasarathy is a Senior Big Data Architect on the Amazon Web Services Lake Formation team. She enjoys building data mesh solutions and sharing them with the community.
Piyali Kamra is a seasoned enterprise architect and a hands-on technologist who believes that building large scale enterprise systems is not an exact science but more like an art, in which tools and technologies must be carefully selected based on the team’s culture , strengths , weaknesses and risks , in tandem with having a futuristic vision as to how you want to shape your product a few years down the road.
The mentioned AWS GenAI Services service names relating to generative AI are only available or previewed in the Global Regions. Amazon Web Services China promotes AWS GenAI Services relating to generative AI solely for China-to-global business purposes and/or advanced technology introduction.