Posted On: Apr 10, 2024
Amazon DataSync introduces manifests, a new feature that enables you to provide a definitive list of source files or objects to be transferred by your DataSync tasks. Using manifests, you can decrease your task execution times by specifying only the files or objects that need to be processed by your task.
Amazon DataSync is an online data movement service. DataSync automates and accelerates copying data between Amazon Storage services, on-premises storage, edge locations, or other clouds. When a DataSync task executes, it scans and compares source and destination locations to determine which files or objects should be transferred. For large file systems or object stores, the scan and compare operation can significantly increase the overall duration of the task execution. If you have well-known datasets, such as those moved as part of an automated workflow, you can use manifests to avoid scanning your entire file or object storage systems to determine changes. Using a manifest file, you can specify millions of source files or objects to be transferred, and DataSync will only scan and compare the files listed in the manifest. You can also use manifests to copy specific versions of objects from your Amazon S3 bucket.
For more information, view the documentation.