Amazon Redshift connection
Connect Prophecy to your Amazon Redshift data warehouse to read from and write to tables from your pipelines. This page explains how to configure the connection, including required parameters, necessary permissions, and how connections are shared across teams.
Prerequisites
Prophecy connects to Amazon Redshift using the database credentials you provide. These credentials are used to authenticate your session and authorize all data operations performed during pipeline execution. To use a Redshift connection effectively, your user must have the following permissions:
SELECT,INSERT,UPDATE, andDELETEon the tables used in your Prophecy pipelines.CREATE TABLE,DROP TABLE, orALTER TABLEif your pipelines create or replace tables.- Access to specific schemas or databases where your tables reside.
To learn more about user permissions, visit Default database user permissions in the Amazon Redshift documentation.
Feature support
The table below outlines whether the connection supports certain Prophecy features.
| Feature | Supported |
|---|---|
| Read data with a Source gem | Yes |
| Write data with a Target gem | Yes |
| Browse data in the Environment browser | Yes |
Data type mapping
When Prophecy processes data from Amazon Redshift using SQL warehouses, it converts Redshift-specific data types to formats compatible with your target warehouse. This table shows how Amazon Redshift data types are transformed for Databricks and BigQuery.
| Redshift | Databricks | BigQuery |
|---|---|---|
| SMALLINT | INT Alias: Integer | INT64 Alias: Integer |
| INTEGER | BIGINT Alias: Bigint | INT64 Alias: Integer |
| BIGINT | BIGINT Alias: Bigint | INT64 Alias: Integer |
| REAL | DOUBLE Alias: Double | FLOAT64 Alias: Float |
| DOUBLE PRECISION | DOUBLE Alias: Double | FLOAT64 Alias: Float |
| DECIMAL | DECIMAL(38,5) Alias: Decimal | NUMERIC Alias: Numeric |
| BOOLEAN | BOOLEAN Alias: Boolean | BOOL Alias: Boolean |
| CHAR | STRING Alias: String | STRING Alias: String |
| VARCHAR | STRING Alias: String | STRING Alias: String |
| DATE | DATE Alias: Date | DATE Alias: Date |
| TIME | TIMESTAMP Alias: Timestamp | TIME Alias: Time |
| TIMETZ | TIMESTAMP Alias: Timestamp | TIME Alias: Time |
| TIMESTAMP | TIMESTAMP Alias: Timestamp | TIMESTAMP Alias: Timestamp |
| TIMESTAMPTZ | TIMESTAMP Alias: Timestamp | TIMESTAMP Alias: Timestamp |
| VARBYTE | BINARY Alias: Binary | BYTES Alias: Bytes |
| GEOMETRY | STRING Alias: String | STRING Alias: String |
| GEOGRAPHY | STRING Alias: String | STRING Alias: String |
| SUPER | STRING Alias: String | STRING Alias: String |
| HLLSKETCH | STRING Alias: String | STRING Alias: String |
| INTERVAL | STRING Alias: String | STRING Alias: String |
Learn more in Supported data types.
Connection parameters
To create a connection with Redshift, enter the following parameters:
| Parameter | Description |
|---|---|
| Connection Name | Name to identify your connection |
| Server | Redshift cluster server Example: redshift-cluster-1.abc123xyz789.us-west-2.redshift.amazonaws.com |
| Port | Port used by Redshift (default is 5439) |
| Username | Your Redshift username |
| Database | Name of the Redshift database you want to connect to Example: analytics_db |
| Password (Secret required) | Your Redshift password |
Sharing connections within teams
Connections in Prophecy are stored within fabrics, which are assigned to specific teams. Once a Redshift connection is added to a fabric, all team members who have access to the fabric can use the connection in their projects. No additional authentication is required—team members automatically inherit the access and permissions of the stored connection credentials.
Be mindful of the access level granted by the stored credentials. Anyone on the team will have the same permissions—including access to sensitive data if allowed.
To manage this securely, consider creating a dedicated fabric and team for high-sensitivity connections. This way, only approved users have access to those credentials.
Fetching data
Prophecy fetches data from Redshift connections in the following ways:
-
When you browse a Redshift connection in the Environment browser, Prophecy fetches data on demand as you expand folders. You can manually refresh the Environment browser to see updated files.
-
When a pipeline runs, Source gems will read the latest available version of the data. If the schema of your data in Redshift changes, Prophecy will automatically use the new schema.