SetOperation

ProphecySparkBasicsPython 0.0.1+ProphecySparkBasicsScala 0.0.1+UC Dedicated Cluster 14.3+UC Standard Cluster 14.3+Livy 3.0.1+

Use the SetOperation gem to perform addition or subtraction of rows from DataFrames with identical schemas and different data.

Parameters

Parameter	Description
DataFrame 1	First input DataFrame
DataFrame 2	Second input DataFrame
DataFrame N	Nth input DataFrame
Operation type	Union: Returns a DataFrame containing rows in any one of the input DataFrames, while preserving duplicates. Intersect All: Returns a DataFrame containing rows in all of the input DataFrames, while preserving duplicates. Except All: Returns a DataFrames containing rows in the first DataFrame, but not in the other DataFrames, while preserving duplicates.

To add more input DataFrames, you can click the + icon on the left sidebar.

Examples

Operation Type: Union

Python

def union(spark: SparkSession, in0: DataFrame, in1: DataFrame, ) -> DataFrame:
    return in0.unionAll(in1)

Scala

object union {
  def apply(spark: SparkSession, in0: DataFrame, in1: DataFrame): DataFrame =
    in0.unionAll(in1)
}

Operation Type: Intersect All

Python

def intersectAll(spark: SparkSession, in0: DataFrame, in1: DataFrame, ) -> DataFrame:
    return in0.intersectAll(in1)

Scala

object intersectAll {
  def apply(spark: SparkSession, in0: DataFrame, in1: DataFrame): DataFrame =
    in0.intersectAll(in1)
}

Operation Type: Except All

Python

def exceptAll(spark: SparkSession, in0: DataFrame, in1: DataFrame, ) -> DataFrame:
    return in0.exceptAll(in1)

Scala

object exceptAll {
  def apply(spark: SparkSession, in0: DataFrame, in1: DataFrame): DataFrame =
    in0.exceptAll(in1)
}