Prophecy 4.0.0.x

April 15, 2025

Starting in Prophecy 4.0.0, our product caters to two audiences: data engineers and business analysts. As we keep making our Spark offerings better for engineers, we're also adding new, easy-to-use tools for business analysts to fulfill a comprehensive platform approach. While Prophecy for Analysts encompasses a completely new SQL-dependent project type, Prophecy for Engineers refers to our traditional PySpark/Scala projects.

We shared this release live in our recent customer webinar! See the recap below.

Prophecy for Analysts

Features

The following are features available in this release.

• Prophecy fabrics to accommodate native orchestration with Prophecy Automate.

• Connections to a wide variety of data providers that can be used throughout pipelines and shared with teammates.

• Business applications to enable non-technical users to run pipelines with built-in guardrails.

• A new set of gems to let you ingest, transform, parse, clean, and write data in the visual pipeline canvas.

• Simplified versioning to streamline your project workflow, while still following Git best practices.

• Monitoring in the Observability page for tracking deployed projects and pipeline schedules.

• Data profiling in the Data Explorer for quick statistics on your data.

• Collaboration capabilities to bridge the gap between teams and speed up time to production.

• Extensibility features to let you build custom components for projects and add additional functionality to Prophecy.

note

These features are only available for projects that leverage Prophecy fabrics.

Prophecy for Engineers

Features

The following are features and improvements available in this release.

• Data profiling in the Data Explorer for quick statistics on your data.

• Data diff is a new feature that lets you see if your target data at the end of your pipeline matches your expectations.

• You can now set a specific Spark Config in a Livy fabric for a job size configuration.

• The CSV Source gem in Python/PySpark projects now includes a property that lets you skip n number of first or last lines when reading in a CSV file.

• More gems in Python/PySpark projects are now compatible with Databricks UC clusters configured with standard (formerly shared) access mode. The table below shows the minimum package version required to enable compatibility.

  Gem	Condition	Package	Minimum version
  CSV Source	Read file with Pandas	ProphecySparkBasicsPython	0.2.44
  BigQuery Source and Target	None	ProphecyWarehousePython	0.0.9
  EmailData	None	ProphecyWebAppPython	0.1.2
  Seed Source	None	ProphecySparkBasicsPython	0.2.39

Fixes

The following are fixes available in this release.

• Selective data sampling mode now works with Databricks UC standard clusters in Scala projects (Python already supported).

Prophecy Library versions

The following are the newest available Prophecy Library versions in this release.

• ProphecyLibsPython: 1.9.45

• ProphecyLibsScala: 8.9.0

Webinar Recap

This is one of our most feature-packed releases yet. We showcased some of these features in an exclusive, live webinar with our customers. If you missed the webinar, don't worry! Click for the full videos (EMEA / AMER) and find the slides here.