Developer

Filter

Filter is a fairly simple component that outputs the rows for which the added filter condition is true.

.

Example

Here, we’re filtering the data for a particular manager, and for a particular month - since we’re only interested in that data for now.

.

.

Example Code

Here is the backend code for this filter condition:


object MgrMonth {

  def apply(spark: SparkSession, in: DataFrame): Filter = {
    import spark.implicits._

    val out =
      in.filter((col("i_manager_id") === lit(38)).and(col("d_moy") === lit(12)).and(col("d_year") === lit(1998)))

    out

  }

}
    

# Python code coming soon!