Skip to main content

November 2023

3.2.5.* (November 28, 2023)​

  • Prophecy Python libs version: 1.7.0
  • Prophecy Scala libs version: 7.1.37

Features​

Low Code Airflow Enhancements​

We're excited to introduce several new features in our Low Code Airflow Support to enhance your orchestration experience:

  • For-Loop Support with Subgraphs Now, you can easily incorporate loops into your task orchestration. Looping over tasks is a common use case in orchestrations, especially when running Spark Pipelines with different configurations. Our Low Code Airflow introduces support for For-Loops through the concept of Subgraphs.

  • Project-Level Functions Reusing common Python functions across operators or even across different DAGs is now a breeze. With the introduction of Project-Level Functions in Low Code Airflow, you can define functions at the project level, making them accessible to all DAGs within the project. This promotes code reusability and simplifies DAG development.

  • Direct Snowflake Connection and Gem Support Expanding our SQL capabilities, we now offer direct support for executing SQL queries on connected Snowflake instances. While we already supported SQL execution via DBT, this release allows users to run SQLs directly, providing more flexibility in working with Snowflake.

  • Trigger DAG Operator In orchestration, it's common to initiate jobs from a parent Job. Airflow facilitates this through the Trigger DAG Run Operator. We're thrilled to announce that Low Code Airflow now seamlessly incorporates this functionality. Users can leverage this Gem to effortlessly trigger additional jobs in Prophecy or any DAG they've created in Airflow.

Release enhancements​

Last Month we release some advance settings to give more granular control to User for their Releases. Building upon that, we have added small enhancements in the Release path.

  • Show Deployment history: Previously we used to show the List of all releases made through Prophecy and the current deployed version of all Jobs per Fabric. With this release, we have added a new tab to show Deployment History of the project too. User can also check the logs for past deployments from the Deployment History tab.

Deployment_history

  • Delete Branch while Merging: Most of the Git providers have the functionality to delete the branch while merging to the Main/Master branch. This options helps to keep the number of development branches in check and helps in keeping the tree clean. On the merge screen we have added a checkbox to delete the branch after Merge. Please note, this only deletes the branch from Prophecy and not from your Git.

Delete_branch

Data Explorer for Spark​

The Data Explorer feature empowers users to explore and analyze their data samples directly within the UI. We already supported this in our Low-code SQL product. With this release, we extended the same capabilities in the Low code Spark product too. With this feature, users can gain insights, verify data accuracy, and make informed decisions by leveraging the filter, sort, download and Save as Gem capabilities directly in UI. Please refer here for detailed documentation.