Amazon S3
The S3 (Amazon Simple Storage Service) connection in Prophecy enables you to integrate with AWS S3, which is a scalable object storage service for storing and retrieving any amount of data.
Feature support
The table below outlines whether the connection supports certain Prophecy features.
| Feature | Supported |
|---|---|
| Read data with a Source gem | Yes |
| Write data with a Target gem | Yes |
| Browse data in the Environment browser | Yes |
Parameters
To create a connection with your Amazon S3 buckets, enter the following parameters:
| Parameter | Description |
|---|---|
| Connection Name | Unique name for the connection |
| Access Key ID | Your AWS access key ID |
| Secret Access Key (Secret required) | Your AWS secret access key |
| Region | AWS region where your S3 bucket is located Example: us-east-1 |
| Bucket Name | Name of your S3 bucket |
S3 permissions
When you create an S3 connection in Prophecy, access permissions are tied to the AWS credentials you use. This is because Prophecy uses your credentials to execute all data operations, such as reading or writing files.
To fully leverage an S3 connection in Prophecy, you need the following permissions:
- For a bucket:
List,Write - For an object/file:
Read,Write
Sharing connections within teams
Connections in Prophecy are stored within fabrics, which are assigned to specific teams. Once an S3 connection is added to a fabric, all team members that have access to the fabric can use the connection in their projects. No additional authentication is required—team members automatically inherit the access and permissions of the original connection.
Be mindful of the access level granted by the stored credentials. Anyone on the team will have the same permissions—including access to sensitive data if allowed.
To manage this securely, consider creating a dedicated fabric and team for high-sensitivity connections. This way, only approved users have access to those credentials.
Fetching data
Prophecy fetches data from S3 connections in the following ways:
-
When you browse an S3 connection in the Environment browser, Prophecy fetches data on demand as you expand folders. You can manually refresh the Environment browser to see updated files.
-
When a pipeline runs, Source gems will read the latest available version of the data. If the schema of the data stored in S3 changes, you will need to re-infer the schema in Prophecy.