Once you are done developing your schedule, the next thing you’d want to do is to test it. Prophecy lets you do that with one click. You have to choose a fabric to run it on (top right) and click on the Play button on bottom right. Once you do that, you see this
You can open the detailed logs section to see what’s happening.
- There are some initial steps, which prepare a DAG code for airflow.
- If you are using any workflow components, those workflows will be compiled into jar files and the jars will be pushed to a filesystem that can be accessed by spark, which is sitting inside the fabric.
- The DAG is given a temporary
dag_idand put as a file in Airflow DAG Bag
- Airflow picks up the DAG
- The dag is manually triggered
- The manual run is then monitored, with logs being streamed back to the UI. You can also see the logs on hovering over the components.