Skip to main content


Provides a SparkSession and allows you to run custom code


Input DataFrame(s)Input DataFrame(s)False
Output DataFrame(s)Output DataFrame(s)False
CodeCustom code to be executedTrue

To remove input/output DataFrame(s), simply click icon on the left sidebar

Script - Remove inputs


When executing a custom script Gem, the output schema is not known by Prophecy so it must be inferred from a sample computation result. Click the Custom Schema button and Infer from cluster as shown in the Gems --> Outputs description here. The schema will be inferred according to the script and the Spark version running on the connected cluster.


Script Gem with Input and Output: Un-pivoting a DataFrame

We'll perform the unpivot operation using our custom code

Script - Unpivot

Script Gem with only Output: Generating a DataFrame

We'll use the provided SparkSession to create and return a DataFrame


Since we removed the input port, we don't see input DataFrame in the method signature

Script - Unpivot