Skip to main content

Prophecy 4.1.2.x

July 22, 2025

Prophecy for Analysts​

Features​

The following features are available in this release.

  • You can now use Google BigQuery as a SQL Warehouse connection in Prophecy fabrics. This means that you can run SQL transformations in pipelines in your BigQuery warehouse. Learn how to configure a BigQuery connection.

  • You can now define and call stored procedures within SQL projects running on BigQuery. This is particularly useful when migrating workflows from other systems.

  • Pipeline schedules can now be triggered by file arrivals or changes in a specified SFTP or S3 location.

  • Prophecy provides the following new SQL gems in this release.

    • Pivot: Reshape your data by transforming unique row values from one column into new columns.
    • Unpivot: Convert multiple columns into row-level key-value pairs to simplify columnar data.
    • Buffer: Take any polygon or line and expand or contract its boundaries.
    • FindNearest: Find the closest spatial point(s) between two datasets based on geographic distance.
    • Generalize: Reduce the number of vertices in polygons and polylines while preserving overall shape.
    • HeatMap: Transform latitude and longitude point data into a spatial heatmap using hexagonal tiling.
    • PolyBuild: Build spatial shapes from coordinate data by grouping and ordering points into either polygons or polylines.
    • SpatialMatch: Compare geometries and return only the pairs that have the spatial relationship you specify.

Enhancements​

The following enhancement is available in this release.

  • We now support uploading Parquet files to Databricks connections.

Prophecy for Engineers​

Features​

The following feature is available in this release.

Enhancements​

The following enhancements are available in this release.

  • We have improved the release flow for protected branches. When a branch like main is protected, direct commits to it are not allowed. Our previous release process required a direct commit to main, which caused issues in these scenarios. We now provide an option on the Open Pull Request screen to bump the version of the source branch before merging into main. To learn more, visit Pull requests.

  • You can now merge from the default branch (e.g., main) into another branch. Previously, the merge UI only allowed merging into the default branch. This enhancement is available in both the Git workflow dialog within the project editor and the Commits tab on the project metadata page.

  • The Catalog Table Target gem includes the following enhancements. Update the ProphecySparkBasicsPython package to version 0.2.47 to access these updates.

    • Soft Delete for SCD Type 2 (SCD2) Merge: You can now enable soft delete behavior during SCD2 merges. When the source system deletes a record, the target table retains its history by updating the record’s end-time. This treats the deletion as a Type 2 change instead of removing the row.

    • Schema Evolution for Merge: When writing to a Delta table on Databricks using merge mode, the table can now automatically evolve its schema to accommodate changes in the incoming data (such as added or modified columns). This feature helps prevent merge failures due to schema mismatches. To enable it, turn on the "Schema evolution for merge" property in the Catalog Table Target gem. (Requires Databricks Runtime 15.4 or later.)

  • The XLSX gem now runs on Databricks UC clusters configured with standard (formerly shared) access mode.

    GemConditionPackageMinimum version
    XLSXRead file with Pandas. The openpyxl library is needed for Pandas to work. Files can be read from and written to UC volumes only.ProphecySparkBasicsPython0.2.47

Prophecy Library versions​

The following are the newest available Prophecy Library versions in this release.

  • ProphecyLibsPython 2.0.11

  • ProphecyLibsScala 8.11.0

Global Updates​

  • You can now enable auditing for file downloads from Prophecy. When this feature is turned on via an environment flag, audit logs will capture the file name, the user who downloaded it, and the page in Prophecy where the download occurred. Contact your administrator or support team to enable this feature.