Subgraph
Subgraph Gems let you take multiple different Gems and wrap them under a single reusable parent Gem. In other words, they allow you to decompose complex logic into reusable components. See how to create and publish a Subgraph in the video below.
Types of Subgraphs
There are three types of Subgraph available by default. The Basic Subgraph is a great introduction to using Subgraphs.
Name | Description |
---|---|
Basic | Captures one or more Gems within a Pipeline to reuse across other Pipelines and Projects. |
Table Iterator | Iterates over one or more Gems for each row of the first input DataFrame. Table iterator is available for Python Projects only. |
Novel Type | Apply any custom logic to the group of Gems present inside the Subgraph. |
Create your own type of Subgraph
You can also create your own Subgraph to apply any custom logic on the group of Gems present inside it. For example Try Catch, other kinds of Iterators, etc.
To create your own Subgraph type, Go to the project you want to create the Gem in. Click on the (1) Create Gem button. This will open up the Create Gem Form. Provide a (2) Name for the Gem, and select the mode as Control Flow Gem.
This takes to you the Gem Code Editor with a basic structure of the code generated. Here you can start modifying your dialog, validation and the actual logic of the Gem. Read here for more details of the Gem code. More detailed docs on writing Subgraph Gems to follow soon.
The newly constructed Subgraph Gem can be utilized within any Pipeline of this Project, accessible through the Subgraph menu as demonstrated below. Furthermore, you have the option to Release this project, enabling its use as a dependency in other projects, thus incorporating the created Gem into various projects. Read here for more details on project as a dependency.
What's next
To learn more about Spark Subgraphs, see the following pages:
📄️ Basic subgraph
Basic Subgraph, Group your Gems in reusable Parent Gems.
📄️ TableIterator
Loop over each row of an input Dataframe