77 docs tagged with "gems"

Group and pivot your data

Group data and apply aggregation methods or pivot operations

Parameters and properties to read from and write to Avro files

Parameters and properties to read from and write to the BigQuery warehouse

Change the data type of multiple columns at once

Change the data type of multiple columns at once

Rename multiple columns in your dataset in a systematic way

Rename multiple columns in your dataset in a systematic way

Parse XML or JSON inside a table

Compare columns between two dataframes

Parameters and properties to read from and write to the CosmosDB warehouse

Parameters and properties to read from and write to CSV files

Standardize data formats

Standardize data formats and address missing or null values in the data

Ensure your data adhere to predefined constraints

DB2

Remove duplicates from your data

Remove rows with duplicate values of specified columns

Parameters and properties to read from and write to Delta files

Read from or write to tables managed by a Delta table metastore

Return a listing of all the files in a specified directory

Dynamically generate values depending on certain conditions

Dynamically filter columns of your dataset based on a set of conditions

Dynamically filter columns of your dataset based on a set of conditions

Filter the data

Filter your data based on a custom filter condition

Parameters and properties to read from and write to Fixed Format files

Flatten nested columns

Flatten nested data

Build functions with SQL macros to be used in gem expressions

Identify non-identical duplicates in your data

Power your pipelines with gems

Transform your data with Prophecy gems

Read from or write to tables managed by a Hive metastore

Read from or write to tables managed by Iceberg

Parameters and properties to read from and write to the JDBC warehouse

Join two or more datasets

Join one or more DataFrames on conditions

Parameters and properties to read from and write to JSON files

Parameters and properties to read from and write to Kafka files

Limit the number of columns processed

Limit the number of rows

Lookup

Use dbt macros in your pipelines

Parameters and properties to read from and write to the MongoDB warehouse.

Oracle

Parameters and properties to read from and write to ORC files

Sort the data

Sort your data based on one or more columns

Parameters and properties to read from and write to Parquet files

Parameters and properties to read from and write to the Redshift warehouse.

Use expressions to reformat column names and values

Select one or more columns or values using expressions and functions

Repartition or coalesce a DataFrame

Call APIs from your pipeline.

Enrich DataFrame with content from rest API response based on configuration

Create multiple DataFrames based on filter conditions

Salesforce

Sample records by choosing a specific number or percentage of records

Add, Edit, Rename or Drop Columns

Parameters and properties to read from Seed files

Union, Intersect and Difference

Parameters and properties to read from and write to the Snowflake warehouse.

Set of gems related to reading and writing data

Prophecy Streaming Gems

Gems are data seeds, sources, transformations, and targets

Use a custom SQL statement

Create DataFrames based on custom SQL queries

Teradata

Parameters and properties to read from and write to Text file

Use the Unpivot gem to transform your data from a wide format to a long format

Learn how to upload files to your Spark pipeline

Recursively processes rows

Create moving aggregations and transformation

Aggregate and transform Windowed data

Parameters and properties to read from and write too XLSX (Excel) files

Parameters and properties to read from and write to XML files