Low Code Airflow

Introduction

Once you have developed a spark workflow using prophecy, you will want to schedule it to run at some frequency. To support this, prophecy integrates with Airflow and provides a low-code interface to develop Airflow DAGs.

Why Airflow?

The orchestration tools market right now has multiple players, including

  • Airflow
  • Luigi
  • Prefect
  • Argo
  • KubeFlow
  • MLFlow

Among these tools, we found Airflow to be the most mature and most popular tool with a a lot of breadth in its feature-set.

Now Airflow also comes with its own quirks - it has a high learning curve and is difficult to operate and manage. We believe that we can make using Airflow easier than it currently is, by providing a visual development interface.

Prophecy for Airflow

Prophecy aims to make development, deployment and monitoring on Airflow very easy.

  • Prophecy has inbuilt gems that you can drag-and-drop on a visual canvas for common tasks to build an Airflow Schedule-Dag
  • Every visual gems translates to an Operator in the Airflow DAG and python code is generated for it
  • Interactive execution allows you to do dry runs of the Schedule-Dag before you deploy it, this functionality is not there in Airflow
  • You can inspect log of each gem/operator as it runs - making it very easy to test