Skip to main content

Script

Spark Gem

Provides a SparkSession and allows you to run custom code.

Parameters​

ParameterMeaningRequired
Input DataFrame(s)Input DataFrame(s)False
Output DataFrame(s)Output DataFrame(s)False
CodeCustom code to be executedTrue
info

To remove input/output DataFrame(s), simply click icon on the left sidebar

Script - Remove inputs

Schema​

When executing a custom script Gem, the output schema is not known by Prophecy so it must be inferred from a sample computation result. Click the Custom Schema button and Infer from cluster as shown in the Gems --> Outputs description here. The schema will be inferred according to the script and the Spark version running on the connected cluster.

Examples​

Script Gem with Input and Output: Un-pivoting a DataFrame​

We'll perform the unpivot operation using our custom code

Script - Unpivot

Script Gem with only Output: Generating a DataFrame​

We'll use the provided SparkSession to create and return a DataFrame

note

Since we removed the input port, we don't see input DataFrame in the method signature

Script - Unpivot