Source And Target
The Source and Target gems to help you work with various file-based formats, connect to a warehouse-style data source, and work with various table-based formats.
File
You can use the Source and Target gems to help you work with following file-based formats:
| Formats | ||
|---|---|---|
| Avro | JSON | Seed |
| CSV | Kafka | Text |
| Delta | ORC | XLSX (Excel) |
| Fixed Format | Parquet | XML |
Synthetic data generator
You can generate synthetic data with the data generator file type. It allows you to specify the data type of each column and populates them with randomly generated data. You can also specify additional requirements such as the boundaries for each row, and the percentage of rows that must have null values.
Warehouse
You can use the Source and Target gems to help you connect to the following warehouse-style data sources:
| Data Sources | ||
|---|---|---|
| BigQuery | MongoDB | Snowflake |
| CosmosDB | Oracle | Teradata |
| DB2 | Redshift | |
| JDBC | Salesforce |
Catalog
You can use the Source and Target gems to help you work with the following table-based formats:
| Formats | ||
|---|---|---|
| Delta | Hive | Iceberg |
Lookup
Lookup is a special component that allows you to broadcast any data and use it anywhere in your pipeline at a later time.