Developer

Configuration

Configuration Menu

Configuration menu is on the top-left of the editor screen, represented by 3 dots, it is followed by the name of the workflow

Clicking the three dots opens up the menu

Auto Set Layout

Auto set Layout re-arranges the components on the screen according to a built-in layout algorithm. This algorithm is a work in progress, and we’re woring on improving it.

Unit Tests

Unit tests allows you to

  • Coverage: See the existing unit tests per component
  • Run: Run all unit tests for this workflow
  • Generate: Generate unit tests if the workflow has run and sample data is available

Configurations

Confiugration variables can be defined here (let’s say foo), and used in any path or expression string in the workflow as {Config.foo}. Configuration variables can be common to the workflow, and then there can be some values overriden for each execution environment (or Fabric). You’ll see a tab for each

UDFs

User Defined Functions can be written for the workflow here. They are written in the code language of the workflow (Scala or Python). These can be used in any component and the expression builder will suggest these along with in-built functions in various components.

Note: Central library of UDFs is a feature on the near-term product roadmap

Options

This shows the options used when creating the workflow - for example whether it is batch or streaming, and the backend language to be written when you’re visually developing the workflow (scala or python).

However, Interactive development options can be changed. When you’re running a workflow in development mode (the normal mode in visual development), Prophecy will read sample rows after each component. This also disables all optimizations - since here the focus is on correctness. While this is normally ok, you will sometimes want to reduce the sampling or diable it altogether when running with very large datasets of when trying to understand workflow performance.

Note: If your workflow has very large datasets, or you’re trying to understand performance, you can disable data sampling.