Spark Gems

Gems are functional units in a Pipeline that perform tasks such as reading, transforming, writing, or handling other data operations.

The table below outlines the different Spark Gem categories.

Gem	Category	Description
	Source and Target	The set of Gems that help with loading and saving data.
	Transform	The set of Gems that help with transforming data.
	Join and Split	The set of Gems that help with the process of merging or splitting DataFrame(s) to create new DataFrame(s).
	Custom	The set of Gems that our creative teams build using Expression Builder to extend the Propehcy capabilities.
	Machine Learning	The set of Gems that prepare data or use data for Machine Learning.
	Subgraph	A Gem that can contain many other Gems within it.