107 docs tagged with "gems"

Group and pivot your data

Group data and apply aggregation methods or pivot operations

Parameters and properties to read from and write to Avro files

Parameters and properties to read from and write to the BigQuery warehouse

Expand or contracts the boundaries of a polygon or line

Change the data type of multiple columns at once

Rename multiple columns in your dataset in a systematic way

Parse XML or JSON inside a table

Compare columns between two dataframes

Parameters and properties to read from and write to the CosmosDB warehouse

Returns one integer that represents the count of records in the input dataset

Create geographic points with longitude and latitude coordinates

Parameters and properties to read from and write to CSV files

Standardize data formats

Standardize data formats and address missing or null values in the data

Encode and decode data using different techniques

Obfuscate data in one or more columns

Ensure your data adhere to predefined constraints

DB2

Remove duplicates from your data

Remove rows with duplicate values of specified columns

Parameters and properties to read from and write to Delta files

Read from or write to tables managed by a Delta table metastore

Return a listing of all the files in a specified directory

Calculate the distance between two points

Run SQL queries that update dynamically at runtime

Dynamically generate values depending on certain conditions

Dynamically filter columns of your dataset based on a set of conditions

Dynamically filter columns of your dataset based on a set of conditions

Send your pipeline output tables to others via email

Send emails via your Spark pipeline

Send data from your Spark pipeline to others by email

Filter the data

Filter your data based on a custom filter condition

Return rows that match a certain count

Identify the shortest distance between spatial objects

Parameters and properties to read from and write to Fixed Format files

Flatten nested columns

Flatten nested data

Build functions with SQL macros to be used in gem expressions

Match records that are not exactly identical

Identify non-identical duplicates in your data

Power your pipelines with gems

Transform your data with Prophecy gems

Generate spatial heatmaps from geo point data using hexagons

Read from or write to tables managed by a Hive metastore

Read from or write to tables managed by Iceberg

Parameters and properties to read from and write to the JDBC warehouse

Join two or more datasets

Join one or more DataFrames on conditions

Parameters and properties to read from and write to JSON files

Parse JSON inside a table

Parameters and properties to read from and write to Kafka files

Limit the number of columns processed

Limit the number of rows

Lookup

Use dbt macros in your pipelines

Parameters and properties to read from and write to the MongoDB warehouse.

Change the data type of multiple columns at once

Rename multiple columns in your dataset in a systematic way

Oracle

Parameters and properties to read from and write to ORC files

Sort the data

Sort your data based on one or more columns

Parameters and properties to read from and write to Parquet files

Convert your table from long to wide format

Create a polygon or polyline from a set of coordinates

Send your pipeline output directly to PowerBI

Assign each row a unique ID

Parameters and properties to read from and write to the Redshift warehouse.

Use expressions to reformat column names and values

Select one or more columns or values using expressions and functions

Repartition or coalesce a DataFrame

Call APIs from your pipeline

Enrich DataFrame with content from rest API response based on configuration

Create multiple DataFrames based on filter conditions

Salesforce

Sample records by choosing a specific number or percentage of records

Add, Edit, Rename or Drop Columns

Leverage a Python script in your pipeline

Parameters and properties to read from Seed files

Union, Intersect and Difference

Decrease the number of nodes that make up a polygon or polyline

Use data from Smartsheet in your Spark pipeline

Parameters and properties to read from and write to the Snowflake warehouse.

Set of gems related to reading and writing data

Prophecy Streaming Gems

Find relationships between geographic features

Use a custom SQL statement

Create DataFrames based on custom SQL queries

Create and call stored procedures to use in pipelines

Send data from your Spark pipeline to Tableau

Send data to automatically update your Tableau dashboards

Teradata

Parameters and properties to read from and write to Text file

Convert text into a column in your table

Create a placeholder gem in your pipeline

Convert your table from wide to long format

Convert your table from wide to long format

Use the Unpivot gem to transform your data from a wide format to a long format

Learn how to upload files to your Spark pipeline

Recursively processes rows

Create moving aggregations and transformation

Aggregate and transform Windowed data

Parameters and properties to read from and write too XLSX (Excel) files

Parameters and properties to read from and write to XML files

Parse XML inside a table