Skip to main content

July 2023

3.1.2.* (July 26, 2023)​

  • Prophecy Python libs version: 1.5.9
  • Prophecy Scala libs version: 7.1.3

Features​

Livy Fabric Users can now enable Execution Metrics​

Livy Fabric users can now enable Execution Metrics and historical runs to track the performance of their Spark pipelines. To enable Execution Metrics, users need to create the following tables:

  • Pipeline Metrics Table
  • Component (Dataset) Metrics Table
  • Interim Table

The instructions for creating these tables can be found here

info

Note: Existing Livy users who had Execution Metrics enabled before this release may experience failures in their pipelines if the above tables are not created. To mitigate this issue, users can either:

  • Set the EXECUTION_METRICS_DISABLED flag to true in their deployment.
  • Create the above-mentioned tables.

If you have any questions, please contact us at contact us.

Users can now select all-purpose Databricks clusters for testing their Jobs​

Users can now select all-purpose Databricks clusters for testing their Jobs. This can be useful for quickly testing changes to a Job without affecting the production environment.

info

Note: Selecting an all-purpose Databricks cluster for testing will not change the Fabric/Job size configured in the Job settings for release and scheduling. This means that the Job will still run on the specified Fabric/Job-size when it is released or scheduled.

Minor Improvements​

Low code Airflow Enhancements​

We have made a few minor enhancements to our low-code Airflow support. These enhancements include:

  • Support for DAG level params/configs at Job level: Users can now add configs for Airflow Jobs similar to Spark Pipelines and use Jinja templating to refer them in their Gems.
  • Support for BranchPythonOperator: This allows a User to “branch” or follow a path following the execution of this task.

Low code SQL Enhancements​

We have made a few minor enhancements to our low-code SQL support. These enhancements include:

  • SetOperations Gem: This Gem will allow users to add Union, Intersection or Exception functionality while modifying their Target models.
  • Adding Trino as a SQL Provider in our Low Code SQL product as Beta. Stay tuned for more comprehensive documentation

Low code Spark Enhancements​

We have made a few minor enhancements to our low-code Spark support. These enhancements include:

  • Moved Salesforce Type under Application section in Source and Target Gems: The Salesforce type has been moved from the Warehouse tab to the Application tab in both source and target gems. This change makes it easier for users to find the Salesforce type and to understand its purpose.
  • Added Information type diagnostic in Script gem: In script gem, output Schema cannot be inferred automatically. If you need an output schema, you will need to infer the schema from a cluster. Without doing this, the output schema is empty. We have now added an Information Diagnostics message to guide Users about inferring Schema manually if they need an output schema.
  • Added keyboard Shortcut: We have added a keyboard shortcut to select all gems in the low-code platform. The shortcut is Ctrl+A.

3.1.1.* (July 6, 2023)​

  • Prophecy Python libs version: 1.5.6
  • Prophecy Scala libs version: 7.0.50

Features​

Conditional Execution In Spark Pipelines​

We are thrilled to announce the addition of support for conditional execution in Spark Gems within our Pipeline framework. This highly anticipated feature provides users with the ability to effortlessly build conditional transformation scenarios directly on our user interface (UI).

Conditional execution allows users to define specific criteria or rules that determine whether a particular transformation should be applied or skipped during Pipeline execution. By leveraging conditional execution, users can easily implement complex data processing logic and dynamically adapt their Pipelines based on varying conditions. This empowers users to efficiently handle different scenarios, such as filtering out irrelevant data, applying specific transformations based on business rules, or performing conditional aggregations. More details on this feature can be found here.

Revamped Prophecy Fabrics UI​

We are excited to announce the revamped Prophecy Fabrics user interface (UI), designed to achieve feature parity with the Databricks cluster creation flow. This significant update ensures that users have a seamless experience and consistent options when creating clusters within Prophecy. The new UI introduces an enhanced Job sizes interface that captures all the options and flexibility provided by Databricks for cluster creation. These Job sizes can be leveraged to create clusters when executing Pipelines through Prophecy or scheduling Jobs through Prophecy. With this improvement, users can easily configure and customize clusters to meet their specific requirements, ensuring optimal performance and resource allocation for their data processing tasks. Please read more about this feature here.

New Features in Low Code Airflow​

Following our recent General Availability Announcement for Low Code Airflow, we are excited to introduce additional features that further enhance the capabilities of the platform.

In this release, we have introduced new Gems in Low Code Airflow Jobs, enabling users to incorporate more functionality into their Airflow DAGs. Users can now easily add the following operators to their DAGs: Slack Operator, Email Operator, HTTP Sensor, and File Sensors. These operators expand the range of actions and triggers available, providing users with more flexibility in designing their data workflows. Furthermore, we are pleased to announce that we have extended support for scheduling DBT projects from private Git repositories. This enhancement allows users to seamlessly integrate and schedule DBT projects hosted in private repositories, providing greater security and control over their data transformation processes.

Minor Improvements​

Ability to Attach Cluster to multiple Pipelines for Interactive Run​

Previously, users could only connect a cluster to a single Pipeline, limiting their ability to use Prophecy in multiple tabs. With this enhancement, users can now attach clusters to different Pipelines in different tabs, enabling concurrent connections without interference. This improvement offers increased flexibility and productivity, allowing users to seamlessly switch between Pipeline configurations, perform parallel analyses, and efficiently manage data workflows.

Alert and warning diagnostics for Users passing plain-text Username password in Gems​

Using hardcoded usernames and passwords is not recommended when working with Gems. This approach can result in unsafe code, as the credentials may be exposed in your configured Git repository for the Project. We have added Warnings and Diagnostics if Users are using plain text Passwords in Gems. Read about the recommended usage here

Improved Entity Discovery on Info Pages​

To enhance the discoverability of our entities, we try to provide comprehensive information on related entities directly on the info page. When viewing a Project's info page, users can now see all associated Pipelines, Jobs, Datasets, and Subgraphs contained within the Project. Likewise, when accessing a Pipeline's info page, users can easily identify the Jobs utilizing the Pipeline, as well as all Published Subgraphs that are used within the Pipeline. For Datasets, users can quickly discover all Pipelines that utilize the Dataset.