Amazon S3
Prophecy supports direct integration with Amazon S3, allowing you to read from and write to S3 buckets as part of your data pipelines. This page explains how to configure the connection, what permissions are required, and how S3 connections are managed and shared within your team.
Prerequisites
Prophecy connects to Amazon S3 using the AWS credentials you provide. These credentials are used to authenticate requests and authorize all file operations during pipeline execution. To ensure Prophecy can read from and write to S3 as needed, the credentials must grant the following permissions:
s3:ListBucketto list the contents of the bucket.s3:PutObjectto write files to the bucket.s3:GetObjectto read files.
To learn more, visit Required permissions for Amazon S3 API operations in the AWS documentation.
Feature support
The table below outlines whether the connection supports certain Prophecy features.
| Feature | Supported |
|---|---|
| Read data with a Source gem | Yes |
| Write data with a Target gem | Yes |
| Browse data in the Environment browser | Yes |
Connection parameters
To create a connection with your Amazon S3 buckets, enter the following parameters:
| Parameter | Description |
|---|---|
| Connection Name | Unique name for the connection |
| Access Key ID | Your AWS access key ID |
| Secret Access Key (Secret required) | Your AWS secret access key |
| Region | AWS region where your S3 bucket is located Example: us-east-1 |
| Bucket Name | Name of your S3 bucket |
Sharing connections within teams
Connections in Prophecy are stored within fabrics, which are assigned to specific teams. Once an S3 connection is added to a fabric, all team members who have access to the fabric can use the connection in their projects. No additional authentication is required—team members automatically inherit the access and permissions of the stored connection credentials.
Be mindful of the access level granted by the stored credentials. Anyone on the team will have the same permissions—including access to sensitive data if allowed.
To manage this securely, consider creating a dedicated fabric and team for high-sensitivity connections. This way, only approved users have access to those credentials.
Fetching data
Prophecy fetches data from S3 connections in the following ways:
-
When you browse an S3 connection in the Environment browser, Prophecy fetches data on demand as you expand folders. You can manually refresh the Environment browser to see updated files.
-
When a pipeline runs, Source gems will read the latest available version of the data. If the schema of the data stored in S3 changes, you will need to re-infer the schema in Prophecy.