Once you have developed a spark workflow using prophecy, you will want to schedule it to run at some frequency. To support this, prophecy integrates with Airflow and provides a low-code interface to develop Airflow DAGs.
The orchestration tools market right now has multiple players, including
Among these tools, we found Airflow to be the most mature and most popular tool with a a lot of breadth in its feature-set.
Now Airflow also comes with its own quirks - it has a high learning curve and is difficult to operate and manage. We believe that we can make using Airflow easier than it currently is, by providing a visual development interface.
Prophecy for Airflow
Prophecy aims to make development, deployment and monitoring on Airflow very easy.
- Prophecy has inbuilt gems that you can drag-and-drop on a visual canvas for common tasks to build an Airflow Schedule-Dag
- Every visual gems translates to an Operator in the Airflow DAG and python code is generated for it
- Interactive execution allows you to do dry runs of the Schedule-Dag before you deploy it, this functionality is not there in Airflow
- You can inspect log of each gem/operator as it runs - making it very easy to test