Low Code Airflow

Development

Create a New Schedule

You can create a schedule from the + button on the top right of the metadata screen

Visual Development

Prophecy lets you visually drop gems on the schedule screen. The gems become operators for the airflow DAG.

Editor Screen Controls

Once you click that, you can click on Open Schedule to arrive at this screen:

This is an empty schedule, and you can now edit it visually.

Gem Menu

On hovering on the three dots on the top right of a gem, you see this menu. Here you can rename or remove a gem.

Gem Ports & Connections

A gem has an incoming port and an outgoing port. You can create edges on the screen to denote sequencing between the gems.

If there is an edge from (Gem1, outport) to (Gem2, inport), for the airflow DAG, it translates to python code Gem1_Operator >> Gem2_Operator. It means that Gem1 will run first and then Gem2 will run. Gem2 run will also be based on a trigger rule.

Gem Trigger Rules

Airflow lets you define a trigger rule on your Operators. So all our gems have this as an argument. This is visible in gems' screenshots below. Here is what the trigger rules mean, from the airflow docs: