Skip to main content

Lineage extractor

The Prophecy lineage extractor is a Python tool that retrieves and exports lineage information from Prophecy projects and pipelines. It supports project, pipeline, and branch-level lineage extraction, with optional features like emailing reports.

You can run the lineage extractor manually or integrate it into a CI workflow to automate report generation. This page covers how to run the extractor via command line and how to automate it using GitHub Actions or GitLab CI.

info

The lineage extractor only supports extraction from Spark pipelines and SQL pipelines. It does not support SQL models.

Prerequisites​

To use the lineage extractor for SQL pipelines:

  • knowledge-graph must be enabled in your Prophecy deployment.

Command​

Use the lineage extractor Python command to export the lineage of a specific pipeline.

ArgumentTypeRequiredDescription
--project-idStringYesProphecy project ID. You can find it in the project URL. Example: https://app.prophecy.io/metadata/entity/projects/57040 where 57040 is the project ID.
--pipeline-idStringDependsOne or more pipeline IDs in ProjectID/PipelineName format, comma-separated.
Required for the lineage reader; optional for the knowledge-graph reader.
--output-dirStringYesDirectory path where the extractor writes the lineage report.
--readerStringNoReader to use. Set to lineage for Spark projects or knowledge-graph for SQL projects.
--fmtStringNoOutput format. Use excel (default) or openlineage (JSON in OpenLineage format).
--branchStringNoBranch to extract lineage from. Defaults to main.
--send-emailFlagNoSends the report by email. Requires SMTP configuration.
Learn more in Integration with GitHub Actions or GitLab CI.
--run-for-allFlagNoGenerates lineage for all pipelines in the project, rather than just one pipeline.
python -m prophecy_lineage_extractor \
  --project-id 6493 \
  --reader knowledge-graph \
  --branch test1234 \
  --output-dir ./test \
  --run-for-all \
  --fmt openlineage

Integration with GitHub Actions or GitLab CI​

This section walks you through automating the extraction of lineage reports from your Prophecy pipelines using a CI workflow in GitHub Actions or GitLab CI. You'll set up a script that pulls lineage data, generates an Excel report, and optionally sends it by email or commits it back to your repository.

Prerequisites​

  • A Prophecy project hosted in an external GitHub or GitLab repository.
  • Access to the repository and permissions to set up CI/CD pipelines.
  • A Prophecy Personal Access Token (PAT).
  • (Optional) To enable email reports, you must have SMTP credentials.

Set environment variables and secrets​

To configure lineage extraction behavior related to authentication, email delivery, and output settings, you’ll need to provide several inputs. While you can hardcode these values directly into your CI workflow YAML, it's strongly recommended to store them as environment variables or secrets. This approach keeps sensitive data like access tokens and SMTP credentials secure, avoids leaking secrets into version control, and makes it easier to update values across environments without modifying the workflow file.

  1. Go to your repository’s Settings > Secrets and variables > Actions.
  2. Add the listed variables under Secrets and Variables tabs.
Variable/SecretDescription
PROPHECY_PATPersonal Access Token used to authenticate with Prophecy.
SMTP_USERNAMEUsername for the email account used to send reports.
SMTP_PASSWORDPassword needed for the email account.
MONITOR_TIME_ENVDuration of the monitoring window in minutes (default: 150).
GIT_COMMITSet to 1 to enable committing generated output to Git.
OUTPUT_DIRDirectory path where lineage files are stored.
OPENLINEAGE_URLURL for sending OpenLineage events. If not set, and format is openlineage, events are written as JSON files in OUTPUT_DIR/<PROJECT-ID>/.

Set up workflow configuration​

To automate lineage extraction and optionally email or commit the resulting reports, you’ll need to set up a CI workflow in your repository. The configuration below provides templates for both GitHub Actions and GitLab CI, which install the extractor, run it with your parameters, and optionally commit the results. These templates assume you've already configured the required environment variables and secrets. Customize them with your specific project and pipeline details before running.

In your GitHub repository:

  1. Select Add file > Create new file.
  2. Name the file .github/workflows/prophecy_lineage_extractor.yml.
  3. Paste the following YAML into the file.
name: Run Prophecy Lineage extractor on main

on:
  push:
    branches:
      - main
    paths:
      - "datasets/**"
      - "pipelines/**"
      - "pbt_project.yml"
      - ".github/workflows/prophecy_lineage_extractor.yml"

permissions:
  contents: write

jobs:
  extract-and-mail-prophecy-lineage:
    runs-on: ubuntu-latest
    env:
      OUTPUT_DIR: "output"
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.9"

      - name: Install Package
        run: |
          pip install --no-cache-dir prophecy-lineage-extractor

      - name: Extract and Send Prophecy Lineage
        env:
          PROPHECY_URL: "https://<custom>.prophecy.io"
          MONITOR_TIME_ENV: ${{ vars.MONITOR_TIME_ENV }}
          PROPHECY_PAT: ${{ secrets.PROPHECY_PAT }}
          SMTP_HOST: "smtp.gmail.com"
          SMTP_PORT: "587"
          SMTP_USERNAME: ${{ secrets.SMTP_USERNAME }}
          SMTP_PASSWORD: ${{ secrets.SMTP_PASSWORD }}
          RECEIVER_EMAIL: "<myRecipient@company.com>"
        run: |
          python -m prophecy_lineage_extractor --project-id <YOUR_PROJECT_ID> --pipeline-id <YOUR_PROJECT_ID>/pipelines/<YOUR_PIPELINE_NAME> --send-email --output-dir $OUTPUT_DIR --branch main

      - name: Commit file to output directory
        env:
          GIT_COMMIT: ${{ vars.GIT_COMMIT }}
        run: |
          if [[ $GIT_COMMIT == "1" ]]; then
              git config --global user.name '<YOUR_GIT_USERNAME>'
              git config --global user.email '<YOUR_GIT_EMAIL>'
              git add $OUTPUT_DIR/*
              git commit -m "[Github Action: main]: Adding Excel lineage report"
              git push
          else
              echo "Committing to Git is not enabled"

Make sure you modify the template with your own details:

  • Replace PROPHECY_URL with your Prophecy URL.
  • Update with your ProjectID and PipelineID.
  • Modify the receiver email.
  • Set your global Git username and email.

Verify lineage file creation​

After a successful run, you should see a directory matching OUTPUT_DIR in your repo containing Excel lineage files like pipeline_name_lineage.xlsx. This XLSX file will show detailed lineage information about your pipeline.

Lineage extractor output

Troubleshooting​

If your workflow doesn't run as expected:

  • Check for error messages in GitHub workflow run logs or GitLab job logs.
  • Verify that you have set all environment variables and secrets correctly.
  • Ensure your Prophecy access token is valid and has the necessary permissions.
  • Confirm that the Project ID and Pipeline ID are correct in the workflow file.