Low Code Spark

SetOperation

Set Operation provides the simple set operations to add or subtract rows from datasets with identical schemas and different data. It has variable number of ports. Following are the supported operations

  • UnionAll
  • Except
  • ExceptAll

.

Example

Let' see an example where we want to add rows from two sources

.

.

The details are pretty simple, we just have to choose the operation

.

.

Example Code

The code is pretty simple:


object Union {

  def apply(spark: SparkSession, in: DataFrame, in0: DataFrame): SetOperation = {
    import spark.implicits._

    lazy val out = unionAll(in, in0)

    out

  }

}
    

# Python code coming soon!