Low Code Spark


Filter is a fairly simple component that outputs the rows for which the added filter condition is true.



Here, we’re filtering the data for a particular manager, and for a particular month - since we’re only interested in that data for now.



Example Code

Here is the backend code for this filter condition:

object MgrMonth {

  def apply(spark: SparkSession, in: DataFrame): Filter = {
    import spark.implicits._

    val out =
      in.filter((col("i_manager_id") === lit(38)).and(col("d_moy") === lit(12)).and(col("d_year") === lit(1998)))




# Python code coming soon!