Skip to main content

April 2024

3.3.5.* (April 30, 2024)​

  • Prophecy Python libs version: 1.8.15
  • Prophecy Scala libs version: 7.1.93

Features​

Parametrized Gems in Spark​

Many enterprises use generic Pipelines that act like frameworks. These Pipelines have the same basic transformation rules but work with files that have different schemas. They generally have different output columns, computed using different expressions. To help with this, Prophecy now lets you completely parameterize your transformation gems. This means the logic for all the expressions along with output columns can come from a Pipeline config. So, you can use one Pipeline as a framework and run it with different setups.

Copilot Fix It for Spark and SQL expressions​

Also, in our last update, we'd enabled Copilot to suggest expressions automatically based on output column names. With this release, we've added a feature called Copilot Fix It. It can fix any errors in your expression logic with just a click of the "Fix with AI" button. It can correct things like syntax mistakes, wrong function calls, or typos.

3.3.4.* (April 12, 2024)​

  • Prophecy Python libs version: 1.8.15
  • Prophecy Scala libs version: 7.1.88

Minor Improvements​

  • Streamlined Expression Suggestions: Enhancing the Copilot feature, we now automatically suggest expressions when users add a new column in the Gem. These suggestions are tailored to the target column name and can be accepted with a simple press of the Tab button.
  • Improved Filtering UX for Problem and Runtime Logs: To enhance the debugging experience in Pipelines, we've refined the user interface for logging. This includes the addition of search, filter, sort, and download options.

3.3.3.* (April 1, 2024)​

  • Prophecy Python libs version: 1.8.13
  • Prophecy Scala libs version: 7.1.82

Features​

Prophecy Libs as First class Project Dependency​

Prophecy Libs is a Prophecy-provided library designed to enhance your Pipelines. These libraries provide essential features such as interims, monitoring, and secret management while maintaining clean and concise Pipeline code. When executing your Pipelines, Prophecy Libs need to be installed on your clusters. In our latest release, we’ve made Prophecy Libs a Project Dependency configurable by the user. This gives you control over when to update dependencies in your Pipelines and on long-running clusters.

Secrets in Pipeline Config​

Users can now create Pipeline configurations of type Secrets. These configurations can be utilized within their Gems. This flexibility allows users to manage username/password changes across different environments by creating distinct config instances. With this, even when working with various Fabrics, users can continue using the same Gem and Pipeline.

Minor Improvements​

  • Code Generation Improvements: We’ve fine-tuned the code generation process for Spark Pipelines, ensuring top-notch quality. You might notice some minor uncommitted changes when you revisit your Pipeline Code.
  • Low Code SQL Improvements: We’ve revamped the UI for Port schema in SQL Gems, enhancing the user experience for modifying and managing Input/Output Ports. Additionally, our Copilot fix-it feature now helps users rectify expression errors interactively.
  • Airflow Improvements: While building Jobs interactively in Airflow, users can now utilize the Trigger DAG run operator to test unreleased DAGs.
  • Git Improvements: Git Merge screens now allow merging to branches other than the main branch. Plus, if you’re rolling back on the main branch, you’ll be prompted to create a new branch for the rollback.
info

Please be aware that when you next open your Projects for editing, you may notice uncommitted changes due to this update.