April 2024
3.3.5.* (April 30, 2024)
- Prophecy Python libs version: 1.8.15
- Prophecy Scala libs version: 7.1.93
Features
Parametrized Gems in Spark
Many enterprises use generic pipelines that act like frameworks. These pipelines have the same basic transformation rules but work with files that have different schemas. They generally have different output columns, computed using different expressions. To help with this, Prophecy now lets you completely parameterize your transformation gems. This means the logic for all the expressions along with output columns can come from a pipeline config. So, you can use one pipeline as a framework and run it with different setups.
Copilot Fix It for Spark and SQL expressions
Also, in our last update, we'd enabled Copilot to suggest expressions automatically based on output column names. With this release, we've added a feature called Copilot Fix It. It can fix any errors in your expression logic with just a click of the "Fix with AI" button. It can correct things like syntax mistakes, wrong function calls, or typos.
3.3.4.* (April 12, 2024)
- Prophecy Python libs version: 1.8.15
- Prophecy Scala libs version: 7.1.88
Minor Improvements
- Streamlined Expression Suggestions: Enhancing the Copilot feature, we now automatically suggest expressions when users add a new column in the gem. These suggestions are tailored to the target column name and can be accepted with a simple press of the Tab button.
- Improved Filtering UX for Problem and Runtime Logs: To enhance the debugging experience in pipelines, we've refined the user interface for logging. This includes the addition of search, filter, sort, and download options.
3.3.3.* (April 1, 2024)
- Prophecy Python libs version: 1.8.13
- Prophecy Scala libs version: 7.1.82