Skip to main content

Prophecy 3.4.3.x

January 8, 2025

  • Prophecy Python libs version: 1.9.29
  • Prophecy Scala libs version: 8.7.0

Features​

Import functions from the Databricks Unity Catalog​

SQL UDFs stored in Databricks Unity Catalog are now available for use in Prophecy Python Projects.

File upload in Spark pipelines​

There are now three ways to upload a file and incorporate it into your Spark pipeline:

  • Drag and drop the file directly onto the pipeline canvas.
  • Open the Source/Target gem drawer and click Upload file.
  • Create a new Source gem, click + New Dataset, and then select Upload file.

Updates​

SCIM user experience enhancements​

When SCIM is enabled, there is a new option under Settings > SSO that lets you easily generate a SCIM token.

Machine-to-Machine functionality with Databricks OAuth​

This feature lets you use Databricks OAuth for authentication during scheduled Jobs, as well as for Project releases and deployments. Visit the Databricks documentation to learn more about access to Databricks with a Service Principal using OAuth.