Prophecy Metadata System provides search and column-level lineage. The simplicity of the interface is backed by a very sophisticated Metadata system.
Here is a deep dive of Lineage with Search
Prophecy Metadata Service, that is supported by a Search Service and a Lineage Service.
An example chain of events can be:
- User changes a workflow, and commits it to the production branch
- This will trigger a metadata event - Event-1
- Two Services will respond
- Lineage service will recompute lineage and commit that to git, triggering Event-2
- Schedule service will see that this workflow is set to run at it’s next scheduled time. It will rebuild the binaries (jar) for this workflow and redeploy it to the Airflow scheduler.
- Search service will see both events Event-1 and Event-2 and pull the latest data in from both to ensure search is up to date