Overview

Overview

Prophecy Definitions
Name Definition
Project A project contains workflows, datasets and schedules created by a team in Prophecy
– It is stored as a folder in Git
Workflow A Spark ETL dataflow created in Prophecy
– It can be edited in a visual drag-and-drop editor in Prophecy
– It can be edited in a code editor in Prophecy
– It is stored in Git in Prophecy (or users' git)
Dataset A link in Prophecy to underlying data
– It is created from Source/Target components in a Workflow
– It can be a file on S3 or DBFS
– It can be a table in a database read using JDBC connection
– It can be a Hive/Delta catalog table read directly from storage
Schedule An Airflow DAG created in Prophecy to run Spark workflow in production
– It is deployed on a Fabric
Fabric An execution environment such as dev, test or prod
– There is one Git branch created per Fabric (except dev - which is master)
– There is one Airflow running per Fabric (you can schedule in test or prod)