Pipeline (formerly known as a Workflow) is a type of entity within Prophecy that is used to represent the flow of data. They are similar to a map you might use on a road trip: You have a Start and Finish (Datasets) and the stops to make along the way (Gems).
In the Metadata view of the UI you'll find the Pipeline list
Follow the instructions below to create a new one
Creating a Pipeline
Pipelines can be created using the Create Entity view.
Pipeline button will pop up the Pipeline Creation UI.
|Project||Which project to create the |
|Branch||Which Git branch to use when developing this |
|Mode||Whether the |
|Description||Description for the |
Editing a Pipeline
When editing a
Pipeline, you'll be using the editor shown below.
- Pipeline Config : Buttons that bring up views related to the configuration of your
- Unit Tests
- Spark Version
- Visual Language
- View switch : Switch between the Visual development and the Generated Code. Note: The generated Code will only update if there are no errors detected in the Visual view.
- Fabric Switch : Select the Fabric to use during development, start a new cluster or connect to an existing one
- Gem drawer : Click to select a category, then click on the Gem you wish to insert into the editor. See here for more information on the Gems
- Workspace : Everything in the middle of the view is the Workspace. This is where you'll create and manipulate the Gems that make up this
- Git Status : In Prophecy almost everything is stored in Git, and this is where you can interact with the version control for your Pipeline. See Projects & Git for more information
- Run : If you just want to run your
Pipelinefrom start to finish, click this button