Skip to main content

Prophecy 3.4.4.x

January 31, 2025

  • Prophecy Python libs version: 1.9.33
  • Prophecy Scala libs version: 8.8.0

Features​

Expanded business rules capabilities​

The new business rule capabilities include:

  • Setting a timeframe in which business rules should apply.
  • Automatic detection of conflicting rule logic.
  • Defining multiple output columns per business rule.

Enhanced Data Explorer experience​

The Data Explorer has been enhanced in the following ways:

  • Filter and sort happen in the frontend. It no longer requires execution in the backend.
  • You can search your data using the new Search function.
  • You can load larger data samples at a time using the new Selective data sampling setting.

Updates​

New custom gem function​

When you are building a custom Spark gem, you can include the customOutputSchemaEnabled function in your gem code to enable or disable to Custom Schema toggle by default. This toggle can be found in the Output tab of a gem.

Advanced mode for SQL target model​

When you want to overwrite the default database for your target model, you can now use advanced mode to dynamically populate the field. Rather than inputting a simple string, you can now use functions, values, and variables in your database definition.

Use multiple unique keys​

Previously when writing a SQL model using an SCD 2 merge approach, you could only define one unique key. This update lets you define one or more unique keys.

Read CSV files with Pandas​

When you read a CSV file using a Source gem, you can now read the file with Pandas. To do so, you must be using a Sharepoint or SFTP location. This feature is for Python projects only.

Snowflake for Prophecy-managed Airflow​

As of Prophecy 3.4.4.1, we support Snowflake on our Prophecy-managed Airflow fabric.

Fixes​

Improved performance on UC shared clusters​

We have removed some redundancy and have fixed how partial runs execute on Databricks UC shared clusters.

Allow use of tables with quotes​

This fix lets you use Snowflake tables that include quotation marks in the table name.

Better error detection in Gem Builder​

Errors captured from sandbox are now displayed with a stacktrace showing the lines in your code which caused an exception to be thrown.

Single user clusters used by a Databricks group​

As of Prophecy 3.4.4.1, users in Databricks groups assigned to single user clusters will have the permission to attach to the cluster in Prophecy without any further action.