Google Cloud Storage connection
Prophecy supports direct integration with Google Cloud Storage (GCS), allowing you to read from and write to GCS buckets as part of your data pipelines. This page explains how to configure the connection, what permissions are required, and how GCS connections are managed and shared within your team.
Prerequisites
Prophecy connects to GCS using a Google Cloud service account key that you provide. This key is used to authenticate requests and authorize all file operations during pipeline execution.
To ensure Prophecy can read from and write to GCS as needed, the service account must have the following permissions:
storage.objects.list— to list the contents of the bucketstorage.objects.get— to read files from the bucketstorage.objects.create— to write files to the bucket
To learn more, visit IAM permissions for Cloud Storage in the Google Cloud documentation.
Feature support
The table below outlines whether the connection supports certain Prophecy features.
| Feature | Supported |
|---|---|
| Read data with a Source gem | Yes |
| Write data with a Target gem | Yes |
| Browse data in the Environment browser | Yes |
| Trigger scheduled pipeline upon file arrival or change | Yes |
Connection parameters
To create a connection with your GCS buckets, enter the following parameters:
| Parameter | Description |
|---|---|
| Connection Name | Unique name for the connection. |
| Service Account Key (Secret required) | Key used to authenticate the connection. See Create and delete service account keys for more information. |
| Project ID | Google Cloud project ID that owns the bucket. |
| Bucket Name | Name of your GCS bucket. |
Paste the full JSON content of your GCP service account key into a Prophecy secret as text. Binary upload is not supported.
Sharing connections within teams
Connections in Prophecy are stored within fabrics, which are assigned to specific teams. Once a GCS connection is added to a fabric, all team members who have access to the fabric can use the connection in their projects. No additional authentication is required—team members automatically inherit the access and permissions of the stored service account credentials.
Be mindful of the access level granted by the stored service account key. Anyone on the team will have the same permissions—including access to sensitive data if allowed.
To manage this securely, consider creating a dedicated fabric and team for high-sensitivity connections. This way, only approved users have access to those credentials.
Fetching data
Prophecy fetches data from GCS connections in the following ways:
- When you browse a GCS connection in the Environment browser, Prophecy fetches data on demand as you expand folders. You can manually refresh the Environment browser to see updated files.
- When a pipeline runs, Source gems will read the latest available version of the data. If the schema of the data stored in GCS changes, you will need to re-infer the schema in Prophecy.