CompareColumns gem
ProphecySparkBasicsPython 0.0.1+ProphecySparkBasicsScala 0.0.1+UC Dedicated Cluster 14.3+UC Standard Cluster 14.3+Livy 3.0.1+
The CompareColumns gem lets you compare columns between two DataFrames based on the key id columns defined.
Parameters
Parameter | Description | Required |
---|---|---|
DataFrame 1 | First input DataFrame | True |
DataFrame 2 | Second input DataFrame | True |
ID columns to retain(Select Id Columns) | List of columns that are used joining two dataframes | True |
Output Column Name(Select Output Columns) | In the output, alias name of the column name that was compared among dataframes | True |
Match Count Column Name(Select Output Columns) | In the output, alias name of the column that shows the count of rows that matched between two dataframes | True |
Mismatch Count Column Name(Select Output Columns) | In the output, alias name of the column that shows the count of rows that mismatched between two dataframes | True |
Mismatch Example Left Column Name(Select Output Columns) | In the output, alias name of the column displaying an incorrect left column value | True |
Mismatch Example Right Column Name(Select Output Columns) | In the output, alias name of the column displaying an incorrect right column value | True |
Mismatch Example ID Column Prefix(Select Output Columns) | In the output, alias name of the ID column value that mismatched between two dataframes | True |