Low Code Spark

Configuration

Configuration Menu

Configuration menu is on the top-left of the editor screen, represented by 3 dots, it is followed by the name of the workflow

Clicking the three dots opens up the menu

Auto Set Layout

Auto set Layout re-arranges the components on the screen according to a built-in layout algorithm. This algorithm is a work in progress, and we’re woring on improving it.

Configurations

Confiugration variables can be defined here (let’s say foo), and used in any path or expression string in the workflow as {Config.foo}. Configuration variables can be common to the workflow, and then there can be some values overriden for each execution environment (or Fabric). You’ll see a tab for each

Options

This shows the options used when creating the workflow - for example whether it is batch or streaming, and the backend language to be written when you’re visually developing the workflow (scala or python).

However, Interactive development options can be changed. When you’re running a workflow in development mode (the normal mode in visual development), Prophecy will read sample rows after each component. This also disables all optimizations - since here the focus is on correctness. While this is normally ok, you will sometimes want to reduce the sampling or diable it altogether when running with very large datasets of when trying to understand workflow performance.

Note: If your workflow has very large datasets, or you’re trying to understand performance, you can disable data sampling.