Custom

Includes a set of gems that do not fall into the other available categories of gems.

📄️ Script

Provide a place to use a SparkSession and whatever code you wish to use

📄️ SQLStatement

Create DataFrames based on custom SQL queries

📄️ DeltaTableOperations

Gem that encompasses some of the import side operations of Delta

📄️ FileOperation

Perform file operations on different file systems

📄️ RestAPIEnrich

Enrich DataFrame with content from rest API response based on configuration

📄️ Directory

Return a listing of all the files in a specified directory