Configuration

Allows you to define configurations to control various aspects of your Pipeline.

Prophecy IDE allows you to define three kinds of configurations:

Spark Configuration

Set runtime Spark configurations as name-value pairs. The name-value pairs will be set inside the Spark runtime configurations as spark.conf.set(name, value)

Hadoop Configuration

Hadoop configurations as name-value pairs. The name-value pairs will be set inside the Hadoop configuration as spark.sparkContext.hadoopConfiguration.set(name, value)

Pipeline Configuration

Config values which can be set at Pipeline level and then be accessed inside any component in the Pipeline. Multiple instances of configuration can be created per Pipeline.

---

Syntax for using configuration inside Gems:

For visual language SQL : '$config_name'

For visual language Scala/Python : Config.config_name

For using Spark expression with visual language SQL : expr('$config_name')

For using Spark expression with visual language Scala/Python : expr(Config.config_name)

Examples for Pipeline level configurations

Now let's use the above defined configurations in the below Pipeline.

Using Config in limit Gem

SQL Visual Language

In the below image '$num_top_customers' is fetching the integer value defined in configurations.

Scala/Python Visual Language

In the below image Config.num_top_customers is fetching the integer value defined in configurations.

Using Spark-expression Config type in Gem

Here we have used Spark expression directly from our config value to populate a column.

SQL Visual Language

In the below image:

(1) amounts -> expr('$test_expression') is coming from configuration type defined as Spark-expression
(2) report_name -> '$report_name' is coming configuration type defined as string

Scala/Python Visual Language

In the below image:

(1) amounts -> expr(Config.test_expression) is coming from configuration type defined as Spark-expression
(2) report_name -> Config.report_name is coming configuration type defined as string

note

Similarly configurations defined as type Spark-expression can be used directly in filter, join, reformat etc Gems directly.

Using config in paths for Source/Target Gems

Config can also be used to refer to paths. This type of configuration comes in handy in situation where we have DEV, QA and PROD data. And want to configure Dataset (or in general the Job runs) based on which environment we are running it in.

Edit Pipeline Name

To change the Pipeline name itself, go to Prophecy's metadata page. Locate the Pipeline within a Project, and click the pencil icon.

Pipeline Configuration instances

Different configuration instances can be defined as per requirement. This comes in handy when Pipeline needs to run with different configurations in different environments or different users.

New instances can be configured to override default values as shown in image below:

Using a particular configuration instance for interactive runs

For interactive runs, configuration can be selected as shown in image below.

Using configuration instances in Jobs

Particular instances can also be configured in Databricks Jobs.

Overriding configuration values in Jobs

Specific values from configuration instance can be overridden as shown in images below:

Code

All configuration instances and values are automatically converted to code as well. Default configurations are stored as code and specific instance overrides are stored as JSON files as shown in image below.

Scala Config code

Python Config code

Component code

Python

def Reformat(spark: SparkSession, in0: DataFrame) -> DataFrame:
    return in0.select(
        col("customer_id"),
        col("orders"),
        col("account_length_days"),
        expr(Config.test_expression).as("amounts"),
        lit(Config.report_name).as("report_name")
    )

Scala

object Reformat {

  def apply(spark: SparkSession, in: DataFrame): DataFrame =
    in.select(col("customer_id"),
              col("orders"),
              col("account_length_days"),
              expr(Config.test_expression).as("amounts"),
              lit(Config.report_name).as("report_name")
    )

}

📄️ Conditional Execution in Spark

Guide on conditional Execution of a Gem