Skip to main content

July 2024

3.3.8.* (July 8, 2024)​

  • Prophecy Python libs version: 1.9.9
  • Prophecy Scala libs version: 8.0.22

Features​

Supporting maven dependencies for Livy​

Prophecy now supports adding Maven dependencies to Pipelines, which are automatically installed on a cluster during Pipeline attachment. This feature, previously available for Databricks, is now extended to Livy Fabrics.

Airflow Copilot enhancements​

  • New Categories And Gems: We have introduced new categories and Gems, including SFTP Sensor, SFTP to Snowflake, Tableau Extract, and SFTP to S3. The Gem Palette has been re-categorized with new sections for Data Transfer, Trigger/Notify, etc., to enhance Gem discovery.
  • Simplified Model Gem: The UX for the Model Gem in Airflow has been simplified. The Model Gem now automatically handles running the current working branch for the selected model and schedules the same upon release. Users can still choose to override these settings in the UI.

Pipeline Monitoring (beta)​

Pipeline Monitoring enables Spark Copilot users to monitor and debug their scheduled and interactive Pipeline runs through Prophecy. When enabled, Prophecy highlights the Gem that causes the failure at runtime. Users can view log stack traces in the Prophecy UI to debug runtime issues faster. This feature is available for all interactive runs through the Prophecy UI and any scheduled Job runs.

To enable this, update the Prophecy Library version in your Project dependencies to the latest (1.9.9), and add the Enable Pipeline monitoring flag in Pipeline Settings. See Enable Pipeline Monitoring. Note that this feature is currently in beta.

For more information, see Pipeline Monitoring and Use Pipeline Monitoring.

Minor Improvements​

  • Support for Apache Iceberg in Sources and Targets: Users can now use Iceberg Catalog tables for reading and writing data in Spark Copilot. This feature is currently available only in Python projects.
  • Duplicate Shared Pipelines: Users can now duplicate a shared Pipeline from a dependency project into their current project. This is useful for teams sharing a "starter" Pipeline with basic Gems and configurations, allowing other users to copy and start new Pipeline development.
  • Support DBR 14.x and 15.x: Prophecy now supports Databricks Runtime versions 14.x and 15.x for all interactive and scheduled runs via Prophecy.