Prophecy 4.2.1 Extended Maintenance (EM) Release

October 25, 2025

Prophecy for Analysts

The following features are available in this release.

We have released two new editions of Prophecy for analysts: Free and Professional Edition. Free and Professional Editions are designed for data analysts who want to collaborate without managing infrastructure. Both editions provide Prophecy-managed resources that are metered by credits.

Review the Prophecy Editions page for a full comparison between Free, Professional, and Enterprise Editions of Prophecy.
You can now use Prophecy's AI Agent to generate pipeline documentation. Instead of writing documentation manually, this lets you create detailed specifications automatically.
Prophecy provides the following new gems in this release.
- Regex: Enables pattern matching and text extraction using regular expressions.
- GCS Source and Target: Read from or write to Google Cloud Storage (GCS) buckets.
- Salesforce Target: Supports writing to Salesforce Objects (CRM Analytics datasets not supported for targets).

The following enhancements are available in this release.

The Email gem now supports a Use column option for several parameters, including To, Cc, Bc, Subject, and Body. When enabled, these fields can be dynamically populated from columns in your input dataset. Each row in your dataset can specify a different recipient and/or custom message while sending the same attachment to all recipients.
The RestAPI gem has been updated in the following ways:
- The gem now supports parsing JSON responses into multiple output columns.
- There are two new authentication methods — Basic Auth and Bearer Token — that automatically send credentials to APIs using new Prophecy secret types.
XLSX (Excel) Source and Target gems have been enhanced:
- Read from multiple sheets in a single Excel file and union all rows into one output table.
- Write to multiple sheets by providing a column whose values are target sheet names. Sheets are created if they don’t exist.
We have added an option to the Directory gem that enables you to list all the sheet names in XLSX files detected in a specified directory.
You can now use the DynamicInput gem to union data from files defined in a column of file paths. Schemas are matched based on column names.

The following enhancements are available in this release.

You can use Selective data sampling mode when running pipelines on EMR and EMR Serverless clusters.
The ProphecyScalaLibs dependency now supports Scala 2.13. Update the dependency in your existing projects to leverage this change.

You can now run the following gems in Databricks Serverless:

Gem	Requirements
XLSX Source and Target	ProphecySparkBasicsPython >= 0.2.5 ProphecyLibsPython >= 2.1.5 Pandas Library Type only (Crealytics not supported)
Text Source and Target	ProphecySparkBasicsPython >= 0.2.5

When using EMR SAML authentication with Okta, you will see updated job size configurations in EMR fabrics.
For Databricks Jobs, we have expanded configuration options and improved integration with Prophecy orchestration.
- Email notifications: Now includes a duration threshold, providing email alerts when jobs exceed expected runtimes.
- If Else gem: Implement conditional branching within a job to control execution flow between two paths.
- RunJob gem: Allow one Prophecy job to trigger another job programmatically.
- Notebook gem: Execute a Databricks notebook from a Prophecy job.
- Delta Live Tables Pipeline gem: Run a Databricks Delta Live Tables pipeline from a Prophecy job.
- Run Conditions for job gems: Control when a job gem executes based on the outcome of upstream job gems.
- For Each execution for job gems: Configure most job gems to run multiple iterations dynamically using the For Each option in the gem's Conditional tab.

The following are the newest available Prophecy Library versions in this release.

You can now enable automatic team creation when using Microsoft Entra ID (Azure Active Directory) SSO.
We've added the ability to enable or disable Copilot at the team level.