Low Code Airflow

Airflow Features

Feature Description
Standard Gems Airflow operators built-in into Prophecy
Introduction Why we chose Airflow and what our goal here is
Low Code Development Interactive step-by-step execution on Airflow
Interactive Run Interactive step-by-step execution on Spark cluster
Configuration You can define configurations for the schedule and various execution environments
Deployment Deploy to Airflow to run at scheduled time
Monitoring Monitor all the deployed schedules and their runs

Standard Gems

Component Description
Workflow (Spark) Choose a Spark workflow to run from a Prophecy project
Sensor (S3) Add a sensor that triggers when an S3 file is added
Script (Python) Generic Python gem that can run any operator or script
Email Send a mail on failure or success
Branch Add a branchbased on a condition

Custom Gems

Prophecy has a Gem Builder to be released in fall 2021 that allows you to build your own gems.