Developer

Interactive Execution

Interactive Execution requires you to spin up a cluster from the top-right of the workflow.

Cluster Connector

When you click to attach, you’ll see any existing clusters (this image has none), and the various Job Sizes you can spin up (Small and Medium here), along with the CPUs and Memory for each Job Size.

The Job Sizes are shown for a particular Fabric. Fabric is the execution environment and you can click it to change the execution environment if you have multiple ones.

When you click on a Job Size, it will spin up a cluster of this size. For existing clusters, this will spin up a job of this size. This will often take 1-5 minutes. 0 minutes to attach to an existing cluster that shows up in the drop-down. 1 minute for spinning up a job on existing cluster. 3 minutes to spin up a new Databricks cluster. 5 minutes to spin up a new EMR cluster.

Once the cluster is spun up, you’ll see two links, one to the UI for this application (Prophecy is running a Spark job with interpreter for your workflow), and one for the cluster. You can click and it will open this UI on another screen.

Save and Play Buttons

Once you have a cluster attached, you can run a workflow by hitting the Play button on the bottom right of the screen. This is only enabled after you save the workflow by hitting the Save button.

You can alternately hit the Small Play button on the bottom right of a any component (shows on mouse-over), and it will only run that component and any previous components that are required to get the inputs for the chosen component.

Job Status

When you run a job, the job id and progress bar shows at the bottom. Typically, one job runs per output. The collection of interim data does not cause any extra jobs, unless you ask for detailed stats - that causes a few extra jobs to be run to compute these statistics.

Interim - Samples and Stats

Interim is the sample flowing between components

Double clicking on interim shows the data in detail. You can see the component and output port names in the title bar. You can also see the number of rows and partitions (for performance). Then there are columns with data types and some sample rows of data.

NOTE

If you have clicked Detailed Stats in the component menu, you can further click on the gray stats icon in the title bar for the interim data, that next to the data type.