Skip to main content

February 2024

3.3.1.* (Feb 28, 2024)​

  • Prophecy Python libs version: 1.8.7
  • Prophecy Scala libs version: 7.1.72

Features​

Low code Airflow Enhancements​

With this Prophecy release, you can connect your self-hosted Apache Airflow Fabric to Prophecy for DAG creation and management in Low code Airflow. Also, we have added a new Gem for OnPremPipeline, easing deployment of Spark pipelines to On Prem Spark.

Copilot Enhancements​

We have further enhanced your Copilot experience by enabling using AI to generate Macros in Low code SQL. Just click on the Copilot icon and effortlessly generate any required function using the magic of AI.

3.3.0.* (Feb 05, 2024)​

  • Prophecy Python libs version: 1.8.4
  • Prophecy Scala libs version: 7.1.66

Features​

Table Iterator Subgraph And Custom Subgraphs​

Introducing the Table Iterator Subgraph in Prophecy version 3.3. This new subgraph empowers users to iterate over one or more Gems for each row of the first input DataFrame. Additionally, users can create their own Custom Subgraph to incorporate custom Iterators or control flow logic such as try-catch, conditional, etc. Read here for detailed documentation.

Automatic Code Regeneration​

Prophecy now facilitates automatic code regeneration whenever a user makes changes that could potentially impact the generated code for other Pipelines and Projects. For instance, when updating dependencies or reusable components like UDFs, Subgraph, Datasets, etc., the system automatically triggers updates to all Pipelines . This ensures seamless synchronization and alerts users to any potential impacts on their codebase.

Secret Management​

Enhancing security, Prophecy now supports three different Secret Providers: Databricks Secrets, HashiCorp Vault, and Environment Variables. Users can link these providers in the Fabric they want to use and securely access secrets in their Gems and Pipelines. This ensures that sensitive authentication credentials are stored in their Secret Providers and used securely by Prophecy.

info

Prophecy never accesses the value of these secrets. It only uses the keys to generate and execute correct code.

Read here for detailed documentation.

Dependency Revamp​

A revamp in the user experience and management of dependencies in Prophecy. Users can now customize Package hub and Custom dependencies at the Project level or override them for a specific Pipeline within the Project. User-defined Gems can also have third-party custom dependencies for pipelines using the Gem.

info

Existing dependencies, previously managed at the Pipeline level, will now be moved to the Project level for easier management. Users may notice uncommitted changes in their POM files. This change does not impact the functionality of pipelines or jobs.

Minor Improvements​

  • Node Search: The Search in the Top left corner now includes searching for any Gem in the open Pipeline. It highlights the Gem Name on the canvas, and clicking it pans the Pipeline to bring the Gem into the user's view.