Prophecy 4.1.3.x

August 18, 2025

Prophecy for Analysts

The following features are available in this release.

The new Pipeline gem lets you trigger another pipeline from within the current pipeline. You can use trigger conditions, pass parameters, and receive metadata for each run. Each gem execution can trigger the target pipeline multiple times, running sequentially until all runs finish.
We've added a feature that lets you divide your pipeline into logical sections using visual containers for canvas organization.
You can now move projects hosted in a Prophecy-managed Git repository into an external repository without having to clone the project.
Prophecy provides the following new gems in this release.
- CountRecords: Count the number of rows in a dataset using different methods.
- Directory: List files and folders of a specified directory from a data ingress/egress connection.
- DataMasking: Obfuscate sensitive string data in one or more columns.
- DataEncoderDecoder: Encode or decode data in selected columns using a variety of techniques.
- FindDuplicates: Filters rows in a dataset based on how frequently they appear.
- RecordID: Generate row-level IDs using UUIDs for randomly generated values or incremental IDs for sequential values.
- DynamicInput: Run SQL queries that update automatically based on your incoming data.
- Fixed-width Source: Read fixed-width files from various file storage accessed from connections.
- Azure Synapse Source: Using the new Azure Synapse connection, you can read data from Microsoft SQL Server hosted on Azure Synapse dedicated SQL pools.

The following enhancements are available in this release.

You can now enable or disable auto-run for Prophecy Apps in the app settings.
We have improved the Knowledge Graph in the following ways:
- After the first full crawl of a fabric, subsequent crawls are now incremental, based on differences between cached Knowledge Graph data and the latest warehouse data. This greatly improves crawl performance.
- Progress is now displayed at the schema level instead of the database level, providing more granular tracking.
- Crawls can now be cancelled while in progress.
When you open a gem, you can now collapse the Ports panel of the gem configuration dialog.
We've added a File Encoding property to CSV Source and Target gems. This lets you decode CSV files when reading and encode CSV files when writing.
SFTP and S3 Source gems have the following enhancements that allow you to (Added in 4.1.3.3):
- Use wildcard patterns in file paths.
- Move or delete files after they are successfully read.
- Dynamically read files provided by file arrival/change triggers through the new Configuration mode.

The following features are available in this release.

You can now define project-level configurations that are shared across all pipelines within a project.

The following enhancements are available in this release.

Prophecy can now capture execution metrics for pipelines that run using Selective data sampling mode.
The Vanilla data sampling mode has been removed from the UI. Pipelines that currently use Vanilla will continue to run as expected; however, once the sampling mode is changed, you won’t be able to switch back to Vanilla. We recommend updating all pipelines from Vanilla to Selective mode.
The Amazon EMR Spark fabric now accepts SAML as an authentication method, using Okta as the identity provider (Added in 4.1.3.3).
When configuring a Databricks job size in Prophecy, you can now set the cluster access mode to Auto (Added in 4.1.3.3). This aligns with recent Databricks changes around cluster access modes. Note that pipeline behavior may vary depending on the cluster Databricks allocates.

The following are the newest available Prophecy Library versions in this release.

Prophecy uses external LLMs for Copilot and AI agents. We've added support for additional LLMs if you want to switch from the default OpenAI endpoint:
- Google Gemini 2.5
- Claud Sonnet 4

note

OpenAI GPT-4o and GPT-4o models mini are fully supported. Gemini and Claud models are under integration testing at this time.

We have addressed several security vulnerabilities in 4.1.3.4 by upgrading dependencies to their latest patched versions: