Low Code Spark

Spark Features

Feature Description
Standard Gems Sources, Targets and Transformation built in
Low Code Development Visual drag-and-drop development of Spark workflows
Interactive Execution Interactive step-by-step execution on Spark cluster
Configuration You can define configurations per project, workflow and execution environment
Tests Prophecy makes it easy to write unit tests per gem, and entire workflows tests
UDFs and UDAFs You can define UDFs, UDAFs and use them in your workflow

Standard Gems

Section Components
Source Introduction Files Catalog Database
Target Introduction Files Catalog Database
Transform Reformat Filter OrderBy SetOperation
SchemaTransformer FlattenSchema Aggregate WindowFunction
Deduplicate Limit
Join / Split Join MultiJoin RowDsitributor Repartition
Custom Script SubGraph SQLStatement

Custom Gems

Prophecy has a Gem Builder to be released in summer 2021 that allows you to build your own gems.