Source & Target
Constitutes the set of Gems that help with loading and saving data.
File
A collection of Gems related to working with various file-based formats.
| Name | Description |
|---|---|
| CSV | Allows you to read or write a delimited file (often called Comma Separated File, CSV) |
| Parquet | Parquet is an open source file format built to handle flat columnar storage data formats. |
| Avro | Avro format is a row-based storage format for Hadoop, which is widely used as a serialization platform. |
| Text | This Gem allows you to read from or write to text file. |
| Delta | Reads data from Delta files present at a path and writes Delta files to a path based on configuration. |
| JSON | Allows you to read or write a delimited file (often called Comma Separated File, CSV) |
| ORC | ORC (Optimized Row Columnar) is a columnar file format designed for Spark/Hadoop workloads. |
| Fixed Format | Read data from fixed format files with expected schema, or write data to fixed format files with expected schema. |
| Kafka | This source currently connects with Kafka Brokers in Batch mode. |
| XLSX (Excel) | Allows you to read or write Excel-compatible files. |
| FTP | Allows you to read or write files (csv, text and binary) on a remote location |
Warehouse
A collection of Gems specializing in connecting to warehouse-style data sources.
| Name | Description |
|---|---|
| Snowflake | Allows you to read or write data to the Snowflake warehouse, using a high-performance connector. Enterprise only. |
| Redshift | Allows you to read or write data to the Redshift warehouse, using a high-performance connector. Enterprise only. |
| Teradata | Allows you to read or write data to the Teradata warehouse, using a high-performance connector. Enterprise only. |
| JDBC | Allows you to read or write data to the JDBC database. |
| Oracle | Allows you to read or write data to the Oracle warehouse, using a high-performance connector. Enterprise only. |
| DB2 | Allows you to read or write data to the DB2 warehouse, using a high-performance connector. Enterprise only. |
| BigQuery | Allows you to read or write data to the BigQuery warehouse, using a high-performance connector. Enterprise only. |
Catalog
A collection of Gems related to working with various table-based formats.
| Name | Description |
|---|---|
| Hive | Read from or write to Tables managed by a Hive metastore |
| Delta | Reads data from Delta tables saved in data catalog and writes data into Delta table in a managed Metastore. |
Lookup
Lookup is a special component that allows you to broadcast any data, to later be used anywhere in your Pipeline.