Overview

Features

Prophecy Deployment Models

Prophecy Version Description
SaaS Prophecy is deployed on cloud (AWS) in Prophecy Network, and it connects to your existing Spark cluster in any public cloud - that runs in your account. Your data remains in your network
Enterprise Trial Prophecy is deployed on cloud (AWS) in Prophecy Network, and it uses Prophecy Databricks cluster
Enterprise Prophecy is deployed on any cloud in Customer Network (VPC) or on-premises, as a Kubernetes operator. This is extremely secure and used by largest Fortune 500 customers such as banks

Low Code Spark

Spark ISV Prophecy Distro Spark Versions
Databricks SaaS, Enterprise 2.4+, 3.0+
AWS EMR Enterprise 2.4+, 3.0+
Hortonworks, Cloudera, MapR Enterprise 2.4+, 3.0+
Feature Description
Low Code Development Visual drag-and-drop development of Spark workflows
Standard Gems Sources, Targets and Transformation built in
Interactive Execution Interactive step-by-step execution on Spark cluster
Configuration You can define configurations per project, workflow and execution environment
Tests Prophecy makes it easy to write unit tests per gem, and entire workflows tests
UDFs and UDAFs You can define UDFs, UDAFs and use them in your workflow
Note: For more Spark details, see Spark Features

Low Code Airflow

Prophecy deploys it’s own Airflow, since it requires Airflow version > 2.0+ and that is not available as a managed service. On request we’ll support any managed Airflow after a few months as services upgrade to 2.0

Feature Description
Introduction Why we chose Airflow and what our goal here is
Low Code Development Interactive step-by-step execution on Airflow
Standard Gems Airflow operators built-in into Prophecy
Interactive Run Interactive step-by-step execution on Spark cluster
Configuration You can define configurations for the schedule and various execution environments
Deployment Deploy to Airflow to run at scheduled time
Monitoring Monitor all the deployed schedules and their runs
Note: For Airflow Operators Supported, see Airflow Features

AbInitio to Spark Transpiler

Prophecy relies on reverse engineering AbInitio formats. No one in our company has ever used AbInitio, but we’re compiler folks and we have reverse engineered all their formats.

AbInitio Versions Spark Versions
2.4+, 3.0+ 2.4+, 3.0+