69 docs tagged with "gems"

Group data and apply aggregation methods or pivot operations

Avro

BigQuery

Change the data type of multiple columns at once

Rename multiple columns in your dataset in a systematic way

Parse XML or JSON inside a table

Compare columns between two dataframes

CosmosDB

CSV

Standardize data formats and address missing or null values in the data

Ensure your data adhere to predefined constraints

DB2

Remove rows with duplicate values of specified columns

Remove rows with duplicate values of specified columns

Delta

Delta Table

Return a listing of all the files in a specified directory

Dynamically generate values depending on certain conditions

Dynamically filter columns of your dataset based on a set of conditions

Filter your data based on a custom filter condition

Filter your data based on a custom filter condition

Fixed Format

Flatten nested data

Flatten nested data

Identify non-identical duplicates in your data

Transform your data with Prophecy gems

Read from or write to Tables managed by a Hive metastore

Iceberg

JDBC

Join one or more DataFrames on conditions

JSON

Reading and writing data from Apache Kafka in batch mode

Limit the number of rows

Limit the number of rows

Lookup

MongoDB

Oracle

ORC

Sort your data based on one or more columns

Sort your data based on one or more Columns

Parquet

Redshift

Select one or more columns or values using expressions and functions

Select one or more columns or values using expressions and functions.

Repartition or coalesce a DataFrame

Enrich DataFrame with content from rest API response based on configuration

Create multiple DataFrames based on filter conditions

Salesforce

Sample records by choosing a specific number or percentage of records

Add, Edit, Rename or Drop Columns

Seed

Union, Intersect and Difference

Union, Intersect and Difference

Snowflake

Prophecy Streaming Gems

Gems are data seeds, sources, transformations, and targets

Create DataFrames based on custom SQL queries

Create models based on custom SQL queries

Teradata

Text

Data transformation steps in SQL

Use the Unpivot gem to transform your data from a wide format to a long format

Use the Unpivot gem to transform your data from a wide format to a long format.

Upload files to your Spark pipeline

Recursively processes rows

Aggregate and transform Windowed data

Aggregate and transform Windowed data

XLSX (Excel)

Seed