Skip to main content

Pipelines

A Pipeline (formerly known as a Workflow) is a type of entity within Prophecy that is used to represent the flow of data. They are similar to a map you might use on a road trip: You have a Start and Finish (Datasets) and the stops to make along the way (Gems).

Pipeline list​

In the Metadata view of the UI you'll find the Pipeline list

Pipeline list

Follow the instructions below to create a new one

Creating a Pipeline​

Pipelines can be created using the Create Entity view.

Create entity

Clicking the Pipeline button will pop up the Pipeline Creation UI.

Pipeline Creation

Field NameDescription
ProjectWhich project to create the Pipeline in. This controls who has access to the Pipeline, groups Pipelines together for lineage, and allows you to use Datasets already published within that project.
BranchWhich Git branch to use when developing this Pipeline.
NamePipeline name
ModeWhether the Pipeline will be Batch mode or Streaming
DescriptionDescription for the Pipeline. Put whatever you want here that will help others (or yourself) understand the purpose of the Pipeline.

Editing a Pipeline​

When editing a Pipeline, you'll be using the editor shown below.

Editing a Pipeline

  1. Pipeline Config : Buttons that bring up views related to the configuration of your Pipeline:
    1. Config
    2. Scheduling
    3. Unit Tests
    4. UDFs/UDAFs
    5. Dependencies
    6. Spark Version
    7. Visual Language
  2. View switch : Switch between the Visual development and the Generated Code. Note: The generated Code will only update if there are no errors detected in the Visual view.
  3. Fabric Switch : Select the Fabric to use during development, start a new cluster or connect to an existing one
  4. Gem drawer : Click to select a category, then click on the Gem you wish to insert into the editor. See here for more information on the Gems
  5. Workspace : Everything in the middle of the view is the Workspace. This is where you'll create and manipulate the Gems that make up this Pipeline.
  6. Git Status : In Prophecy almost everything is stored in Git, and this is where you can interact with the version control for your Pipeline. See Projects & Git for more information
  7. Run : If you just want to run your Pipeline from start to finish, click this button