Skip to main content

December 2024

3.4.2.* (December 13, 2024)​

  • Prophecy Python libs version: 1.9.28
  • Prophecy Scala libs version: 8.6.0

Features​

Databricks OAuth integration​

Prophecy has integrated with Databricks OAuth in order to provide you with increased security via industry-standard authentication flows.

You will see a login overlay in Prophecy, such as when selecting a Fabric, where Databricks API interactions are required.

Select a Fabric

The Databricks OAuth setup must be completed by both your Databricks Account Admin and your Prophecy Team Admin.

For more information on how it works and how to set it up, see Databricks OAuth.

Active and Total Users APIs​

You can use the Active Users and Total Users APIs to find out the number of current and total users who are logged in to Prophecy. This can help you understand usage and times of high traffic, enabling you to manage capacity more effectively.

For more information on the APIs, see Active and Total Users.

Minor Improvements​

  • "Offset" column name bug fix: We fixed a bug where naming a column using a Snowflake reserved keyword, such as "Offset", would break when using a Reformat Gem.

  • Support for null values in unit tests: When upgrading the Scala prophecy-libs version to 8.6.0 or later and the Python prophecy-libs version to 1.9.27 or later, you may notice differences in the Prophecy-managed files related to unit tests. Specifically, changes might occur in the prophecy/tests/*.json files. Any null values in these files will be replaced with empty strings (""). This change does not affect existing unit tests, and they will continue to function as before. These files are managed by Prophecy and are not used during Pipeline execution.

    There is a new option under the three dots of the unit test data table to Set value as Null in columns.

  • New Spark Fabric diagnostic error codes: There are new diagnostic error codes for the following failures:

    • Unable to reach Databricks endpoint.
    • Unable to write execution metrics because Hive Metastore is not enabled on your Spark.
    • Authentication fails while attempting to test a Spark Fabric connection.

    For more information, see Diagnostics.