Skip to main content

June 2023

3.1.0.* (June 22, 2023)​

  • Prophecy Python libs version: 1.5.5
  • Prophecy Scala libs version: 7.0.48

Features​

Data Copilot​

We're thrilled to announce a major update that aims to revolutionize the process of building models and pipelines. With our latest release, creating a new Model from scratch has never been easier—all you need are simple English prompts. But that's not all! Our Data Copilot can also predict granular expressions, making it simpler to write any transformations in your Gems. Please refer to below video for quick intro.

Generative-AI​

With this Release, Prophecy is empowering you to build generative AI applications directly on top of your enterprise private data. With this exciting feature, you can create pipelines that seamlessly ingest data from various sources such as Slack, Asana, Documentation, and web scraping. The information is then vectorized and fed into a vector database. Utilizing these vectors, you can do many thing like develop dynamic and interactive live chatbot applications by effortlessly streaming messages through Kafka.

Here is a quick video demonstrating the same.

Here are two example templates that serve as valuable references as you embark on your generative AI application journey.

These templates leverage the power of our toolbox built on top of Apache Spark, providing you with robust data infrastructure to kickstart your development process. With Prophecy and its toolbox, you can confidently build and scale data-driven solutions for generative AI applications.

General Availability Announcement: Low Code Airflow​

We are delighted to announce the General Availability (GA) release of our highly anticipated Low Code Airflow Jobs Support feature! Following a successful beta phase, we have refined and optimized this functionality to provide you with a robust and user-friendly experience. Here are the key highlights of this feature:

  • Prophecy Managed Airflow: We are excited to introduce Prophecy Managed Airflow, which is now available for trials and even production use cases. You can seamlessly utilize it with DBT or Spark Pipeline. Stay tuned for upcoming updates as we continue to expand our support for more operators.
  • Support for MWAA, Composer 1 and 2 : With our Low Code Airflow Jobs Support, you can easily connect and schedule jobs to your own MWAA/Composer managed Airflow environment. Enjoy the convenience of leveraging your existing infrastructure.
  • Gem Builder for Airflow : To further enhance customization and extensibility, we have added support for Airflow Gems in Gem Builder. This feature empowers you to add any operators you desire, allowing for greater versatility and tailored workflows with Airflow.

For detailed documentation on this feature, including setup instructions and best practices, please refer to our comprehensive documentation available here

Data Explorer For SQL​

The Data Explorer feature empowers users to explore and analyze their data samples directly within the user interface (UI). With this feature, users can gain insights, verify data accuracy, and make informed decisions by leveraging the filter/sort/download capabilities directly in UI. Please refer here for detailed documentation.

DLT Jobs Support​

Delta Live Tables (DLT)(https://www.databricks.com/product/delta-live-tables) is a powerful feature that empowers you to effortlessly build and manage reliable batch and streaming data Pipelines on the Databricks Lakehouse Platform. We are excited to announce that Prophecy now supports scheduling DLT Pipelines through Prophecy Jobs.

Minor Improvements​

In addition to resolving minor issues in our Low-Code-Spark and Low-Code-SQL features, we have introduced below enhancements and fixes.

PingID and AAD​

Enhanced user authentication experience by adding PingID support for seamless integration with Azure Active Directory (AAD), ensuring secure and streamlined access to your applications.

Config Package Name support in Pipelines​

Introduced a new parameter in Pipeline Settings called "Config Package Name," allowing users to specify a folder where config files will be generated. This enhancement addresses potential conflicts when running multiple Pipelines with identical config names in the same cluster by organizing the config files under designated folders.

Other Minor Improvements​

  • We have improved the user experience by relocating the options to edit configuration names, delete, and edit as JSON under the "..." menu. Additionally, we have introduced the functionality to duplicate Pipeline configurations.
  • Along with fixing minor issues in CSV, Fixed format and Flatten schema Gems, we have added two new Gems on our Spark IDE: Compare Columns and Hashing.

3.0.2.* (June 02, 2023)​

  • Prophecy Python libs version: 1.5.4
  • Prophecy Scala libs version: 7.0.32

Minor Improvements​

EMR Support (Beta)​

Users may connect to Amazon EMR clusters by creating EMR Fabrics.

Databricks support enhancements​

  • Catalog Table Gem now supports selecting catalog for unity-enabled Databricks workspaces.
  • Added Info, description and a link to docs for and Gems in UI