Repartition
ProphecySparkBasicsPython0.0.1+ProphecySparkBasicsScala0.0.1+Databricks UC Single Cluster14.3+Databricks UC Shared14.3+Livy3.0.1+
This will repartition or coalesce the input DataFrame based on the specified configuration. There are four different repartitioning options:
Hash Repartitoning
Repartitions the data evenly across various partitions based on the hash value of the specified key.
Parameters
Parameter | Description | Required |
---|---|---|
DataFrame | Input DataFrame | True |
Overwrite default partitions | Flag to overwrite default partitions | False |
Number of partitions | Integer value specifying number of partitions | False |
Repartition expression(s) | List of expressions to repartition by | True |