Search & Lineage

Lineage with Search

Prophecy provides very powerful column level lineage with high quality search!

How to Start with Lineage

Lineage is a graph - that at a column-level granuarity shows how values are flowing across datasets and workflows. When you want to lookup lineage, you’ll start with a particular dataset or workflow. There are 3 ways of picking a starting point for your traversal:

Lineage icon

Start at a known Dataset or Workflow

You can view datasets in a project, or all the datasets you have access to from the Metadata Left-tab, and click the lineage icon in the actions

Similarly, you can click lineage icon on a workflow

When you start at a Dataset or a Workflow, you can then select a column

You can also start with search by clicking the Lineage icon on the left

Once you search, you’ll get search results that are rich and precise


Search will match

  • Names of Datasets & Workflows
  • Columns inside Datasets & Workflows
  • Expressions - any transformation inside workflows

Please note that if you match expressions/transformations, there is a direct lineage link to the exact gem where that transform happened

How to understand Lineage

Lineage results have the following controls

  • Search History Left Panel: On the left there is a panel that shows the last lineage that was looked up. It has an Entity that is a Workflow or a Dataset and the Context - that is the column in the context of which it was looked up. Only the Entities show up in the history panel.
  • Dataset and/or Column Selector: This is the toolbar on the top that enables the user to focus on a particular dataset and a particular column within that dataset
  • Color Scheme: The color scheme works as so
    • Gray: This shows the absence of the searched column
    • Yellow: This shows the column is present, but pass through
    • Red: This shows that the column is present and involved in a transformation
  • Upstream and Downstram Summary: When you’re looking at a column across datasets and workflows, there is an upstream and downstream summary panel at the bottom. This has universal scope and covers all datasets and workflows known to Prophecy.

Summary Screen

Here, you’ll see

  • Search History on the left
  • a chosen Dataset in the middle that has a dashed border
  • The chosen Column in the top bar
  • Colors showing the absence (gray), presence (yellow), and modification (red) involving the chosen column
  • Upstream and Downstream transform summary for this column

Workflow Detail Screen

You can do a deep dive within a workflow and you’ll see

  • Selected Dataset.Column on the top
  • Colors showing the absence (gray), presence (yellow), and modification (red) involving the chosen column
  • Code showing the transform highlighted for the clicked gem