Overview

Commit code for a Project

Prophecy has Agile development process. In the context of data engineering this means

  • Workflows are stored as code in Git
  • Tests are stored as code in Git
  • During development, all changes are saved in a user branch
  • User commits the code to master when ready

Steps

A Project in Prophecy represents a folder in Git.

  1. Edit a Workflow
    • Edits are made as commits to user@company.com branch
    • Everytime you click save from visual or code editor - you get a local commit
  2. Test a Workflow
    • You’ll test a workflow by running on interactive Spark cluster and let’s say that now you’re happy with it
  3. Commit
    • You’ll go to the project page in metadata
    • You’ll click on CREATE PULL REQUEST button
    • You’ll merge from user@company.com to master
    • During the merge unit tests will be run
  4. You’re done!!
    • You made an edit and it has been saved to your Git repo in master