June 2024

3.3.7.* (June 7, 2024)

Auto documentation and description: The AI experience in Spark and SQL copilot is now automated. This includes automatic documentation and descriptions for gems, datasets, columns, projects, and more. Additionally, Git commit and release messages are now auto-generated.
Conversational interface: Users can now interactively modify existing models after generating them with the Copilot, allowing for continuous, incremental prompts through a chat like interface.
UDFs and Macros in copilot expression predictions: The copilot can now predict expressions using custom UDFs and Macros defined in Spark and SQL projects.
Data Context in Prompts: Users can enable data context for expression prediction, ensuring the use of customer data context in Gems. This feature can be enabled by a Team admin in the Team level settings.

Along with above, Various UX improvements have been made to ensure a seamless AI experience in our Data Transformation Copilots.

Business Rule Engine: Added support for Business Rules in Spark Copilot. Users can define business rules as functions and use them in any of their transformation Gems to apply. Note User defined functions are also moved to the left side project Browser with this change.

Auto AI suggestions in SQL code editor: The SQL copilot now features AI auto-suggestions in the SQL code view too, facilitating easier model modification directly in code.
Auto suggest Expression based on Target schema: When a target Gem schema is already defined, expressions for the preceding Gem can be auto-predicted based on the connected input and output Gems.
SQL File upload: Users can directly upload files such as CSV, Excel, JSON, and Parquet to Snowflake or Databricks through Prophecy. Drag and drop files onto the canvas or switch to the environment tab to upload.

Along with above features, Execution and load times for SQL models have been significantly reduced, with a 60% improvement.