Skip to main content

September 2024

3.3.10.* (September 2, 2024)​

  • Prophecy Python libs version: 1.9.14
  • Prophecy Scala libs version: 8.1.4

Features​

Spark Copilot Enhancements​

  • Source suggestion: While working on your Pipeline, if there is an unconnected source, Data Copilot will now suggest other sources that you can add to it on the graph.

SQL Copilot Enhancements​

  • SQL Subgraph: We now support Subgraphs in your SQL models. Subgraphs allow you to wrap multiple different Gems under a single parent Gem. These logical groupings can simplify the Visual view of your model.

  • Support for configurations in the Target Model: The Target Model supports column selection and project-level configurations.

  • Schedule button: Just as you can schedule a Pipeline in Spark, you can now use the Schedule button to schedule a model in SQL. Opening the Schedule screen shows all existing Jobs. You can also create a new Job.

  • Support dragging function parameters: You can drag and drop function parameters in the Visual view of the expression builder.

UX Improvements​

  • Support teams search: You can use search on the teams select box during Fabric creation.

  • SQL upload file button: The Upload File button is now always present on the Environment tab.

Minor Improvements​

  • Fix with AI while saving Gems: When saving a Gem, Data Copilot provides a one-click option to automatically resolve any diagnostic errors within the Gem.

  • Auto describe Datasets: We now automatically describe Datasets and their columns when you drop a Dataset from the Environment tab. Additionally, we improved the quality of the descriptions using interim data when you run your Pipeline.

  • UDFs and Macros in Copilot expression suggestions: The Data Copilot can now generate expressions using custom UDFs and Macros defined in Spark and SQL projects.

  • Prevent unnecessary code changes: Unnecessary code changes are no longer made for the following cases:

    • after a new user loads a Job that was created by another user in the Pipeline
    • after multiple users open a Pipeline with published subgraphs

  • UI doesn't refresh after pull origin fix: We've fixed an issue where merged changes didn't appear after pulling origin in Prophecy until you've refreshed the UI.

  • Fabric configuration reset fix: We've fixed an issue where the Fabric configuration would reset during creation when adding a Job size and selecting JSON.

  • Upgrade dbt version: We've upgraded our dbt supported version to v1.8.