Skip to main content

Parquet

Parquet is an open-source Columnar storage data format. It handles large volumes of data by supporting complex pushdown predicates, nested schemas and a wide variety of column encoding types.

This Gem allows you to read from or write to Parquet files.

Source

Reads data from Parquet files at the given path.

Source Parameters

ParameterDescriptionRequiredDefault
LocationFile path where parquet files are presentTrueNone
SchemaSchema to be applied on the loaded data. Can be defined/edited as json or inferred using Infer Schema button.TrueNone
Recursive File LookupThis is used to recursively load files from the given Location. Disables partition discovery. An exception will be thrown if this option and a partitionSpec are specified.FalseFalse
Path Global FilterAn optional glob pattern to only include files with paths matching the pattern. The syntax follows GlobFilter. It does not change the behavior of partition discovery.FalseNone
Modified BeforeAn optional Timestamp to only include files with modification times occurring before the specified Time. The provided timestamp must be in YYYY-MM-DDTHH:mm:ss form (e.g. 2020-06-01T13:00:00)FalseNone
Modified AfterAn optional timestamp to only include files with modification times occurring after the specified Time. The provided timestamp must be in YYYY-MM-DDTHH:mm:ss form (e.g. 2020-06-01T13:00:00)FalseNone
Merge SchemaSets whether schemas should be merged from all collected Parquet part-files. This will override spark.sql.parquet.mergeSchema.False(value of spark.sql.parquet.
mergeSchema)
Int96 Rebase modeThe int96RebaseMode option allows to specify the rebasing mode for INT96 timestamps from the Julian to Proleptic Gregorian calendar.

Currently supported modes are:

EXCEPTION: fails in reads of ancient INT96 timestamps that are ambiguous between the two calendars.

CORRECTED: loads INT96 timestamps without rebasing.

LEGACY: performs rebasing of ancient timestamps from the Julian to Proleptic Gregorian calendar.
False(value of spark.sql.parquet
.int96RebaseModeInRead)
Datetime Rebase modeThe datetimeRebaseMode option allows to specify the rebasing mode for the values of the DATE, TIMESTAMP_MILLIS, TIMESTAMP_MICROS logical types from the Julian to Proleptic Gregorian calendar.
Currently supported modes are:

EXCEPTION: fails in reads of ancient dates/timestamps that are ambiguous between the two calendars.

CORRECTED: loads dates/timestamps without rebasing.

LEGACY: performs rebasing of ancient dates/timestamps from the Julian to Proleptic Gregorian calendar.
False(value of spark.sql.parquet
.datetimeRebaseModeInRead)

Example

Generated Code

def read_parquet(spark: SparkSession) -> DataFrame:
return spark.read\
.format("parquet")\
.option("mergeSchema", True)\
.load("dbfs:/FileStore/Users/parquet/test.parquet")


Target

Target Parameters

Write data as Parquet files at the specified path.

ParameterDescriptionRequiredDefault
LocationFile path where the Parquet files will be writtenTrueNone
CompressionCompression codec to use when saving to file. This can be one of the known case-insensitive shorten names (none, uncompressed, snappy, gzip, lzo, brotli, lz4, and zstd). This will override spark.sql.parquet.compression.codec.False`snappy
Write ModeHow to handle existing data. See this table for a list of available options.Trueerror
Partition ColumnsList of columns to partition the Parquet files byFalseNone

Supported Write Modes

Write ModeDescription
overwriteIf data already exists, overwrite with the contents of the Dataframe.
appendIf data already exists, append the contents of the Dataframe.
ignoreIf data already exists, do nothing with the contents of the Dataframe. This is similar to a CREATE TABLE IF NOT EXISTS in SQL.
errorIf data already exists, throw an exception.

Example

Generated Code

def write_parquet(spark: SparkSession, in0: DataFrame):
in0.write\
.format("parquet")\
.mode("overwrite")\
.save("dbfs:/data/test_output.parquet")
info

To know more about tweaking Parquet related properties in Spark config click here.