Build Shareable Pipelines
Users can share the Pipelines from one Project to another project and run it with a config in the dependent Projects. This way Data admins can Create deployment templates for the pipelines that have the best practices baked into them for authorization, notifications, handling of errors, and logging the correct information.
An Admin User will create the Pipeline in a Project, let's call it Base Project. They can add config variables to the Pipeline, values for which can be provided while using the Pipeline in another Project called App Project.
Users can run these pipelines then interactively, or also schedule them in Jobs.
To use a Pipeline from Base Project for interactive runs, the User would need to create a Config in App Project. Once added, this Pipeline would be visible in App Project. Please see the below video for interactive runs.
Please note, this Pipeline is marked as read-only in App Project. Only configs can be added and edited for this Pipeline here in App Project.
For Jobs, a User doesn't need to import a Pipeline. When they create a Job in App project they would be able to select any Pipeline from Base Project in the Pipeline operator. All configs from Base Project and App project are then available here in Job.
Please see the below video for sharing Configured Pipelines in Jobs.
Please note, if a Pipeline has a config with the same name in App project and Base Project, the one in App project would be considered final.
Any Modification to existing Pipelines or new Pipelines will be updated in AppProject only after Releasing BaseProject, and updating the dependency in AppProject.