Skip to main content

Subgraph

Subgraph allows you to take multiple different Gems and wrap them under a single reusable parent Gem. It allows you to decompose complex logic into reusable components. See how to create and publish a Subgraph in the video below.

Types of Subgraphs​

There are three types of Subgraph available by default. The Basic Subgraph is a great introduction to using Subgraphs.

NameDescription
BasicCaptures one or more Gems within a Pipeline to reuse across other Pipelines and Projects.
Table IteratorIterates over one or more Gems for each row of the first input DataFrame. Table iterator is available for Python Projects only.
Novel TypeApply any custom logic to the group of Gems present inside the Subgraph.

Create your own type of Subgraph​

You can also create your own Subgraph to apply any custom logic on the group of Gems present inside it. For example Try Catch, other kinds of Iterators, etc.

To create your own Subgraph type, Go to the project you want to create the Gem in. Click on the (1) Create Gem button. This will open up the Create Gem Form. Provide a (2) Name for the Gem, and select the mode as Control Flow Gem.

Create_subgraph_gem

This takes to you the Gem Code Editor with a basic structure of the code generated. Here you can start modifying your dialog, validation and the actual logic of the Gem. Read here for more details of the Gem code. More detailed docs on writing Subgraph Gems to follow soon.

The newly constructed Subgraph Gem can be utilized within any Pipeline of this Project, accessible through the Subgraph menu as demonstrated below. Furthermore, you have the option to Release this project, enabling its use as a dependency in other projects, thus incorporating the created Gem into various projects. Read here for more details on project as a dependency.

Use_subgraph

What's next​

To learn more about Spark Subgraphs, see the following pages: