Fabrics
Before you can run a pipeline, you need to define an execution environment where the pipelines will run. In Prophecy, you do this by creating fabrics.
A fabric specifies everything required for execution. Depending on the fabric type, it may include:
- External compute: This requires manual effort to connect an external environment, such as Databricks or Snowflake.
- Prophecy Automate: This is the built-in Prophecy runtime included in Prophecy fabrics.
- Connections: These are optional configurations that let you integrate with additional data providers for ingestion and egress.
When you open a project and attach a fabric, you'll have access to all of the fabric's resources to run your pipelines.
Fabric types
Different fabrics are designed to support specific project types. Use the table below to identify which fabric best aligns with your project's execution requirements.
| Fabric type | Description | Usage |
|---|---|---|
| Prophecy | Compute with Prophecy Automate and a SQL warehouse | Run pipelines in SQL projects. |
| Spark | Compute with a Spark engine | Run pipelines in PySpark/Scala projects and Databricks jobs. |
| SQL | Compute with a SQL warehouse | Run models in SQL projects. You cannot run pipelines. |
| Airflow | Compute with an Airflow-compatible engine | Run Airflow jobs. |
Fabrics are configured by team admins. To learn how to create each type of fabric listed here, see Fabric setup in the Administration documentation.
Separate environments and access
Fabrics provide a structured way to separate working environments with team-based access. At minimum, you might have a development and production environment, but you may also have fabrics for QA, Staging, etc.
- Development fabric: Environment used for testing and experimentation, where pipelines can be iterated on without impacting live data or processes.
- Production fabric: Environment hosts stable, validated pipelines that process live data for business operations.
When you create a fabric, you assign it to a team. Only users in that team can access the fabric. To control access in your production environment, create a team with only those users who can execute and deploy pipelines in production.
To learn more about the relationship between fabrics, projects, and teams, visit Team-based access.
Components
Fabrics define all the necessary components that Prophecy needs to communicate with a specific execution environment. Each execution environment will require its own unique fabric configuration. Fabric creation, outlined in the administration section of the documentation, is typically handled by team admins.
Find the key components below that correspond to distinct sections within the fabric's settings.
| Component | Description | Required |
|---|---|---|
| Credentials | Prophecy securely encrypts your credentials and uses them to connect to your external environments. | Credentials are required to establish connections to external execution environments. |
| Cluster configuration | This will determine the computation resources allocated to you. | Cluster configurations are required. Defaults are pre-configured based on your credentials. |
| Connections | To connect to multiple data providers for use in your pipelines, you can add additional connections to your fabric. | One connection to a primary execution environment is required. Additional connections are optional. |
| Secrets | Fabrics can store secrets from different secret providers such as HashiCorp Vault. | Secrets are not required for fabric creation. However, sometimes secrets are required for connections. If a secret is required, you will be prompted to create one. |
Fabric usage
To execute your pipelines, you must attach a fabric to your project, as fabrics define the project's execution environment. You can switch between different fabrics within a project based on factors such as whether you want to use a development or production execution environment. All fabrics created for your teams will be available for you to view and use.
To attach a fabric to a project:
- Open a project from the Prophecy metadata page.
- Open a pipeline or model that you want to work on.
- Expand the Attach Cluster menu. This menu will differ slightly between Spark and SQL projects.
- Select a fabric. You will be shown fabrics that have the same data provider as your project (e.g., Databricks).
- Attach to a cluster or create a new cluster.
- Run your pipeline or model. This executes the data transformation on the environment defined in the fabric!
Fabric metadata
In Prophecy, metadata refers to structured information about all the assets in your workspace. In other words, metadata in Prophecy is the control plane information. You can see this information in the Metadata interface in Prophecy.
If you want to see all of the fabrics owned by your teams, navigate to the Fabrics tab of the Metadata page. Then, click into each fabric to access the fabric settings. These will resemble the settings that appear during fabric creation.
Only team admins can update the fabric settings. Other users in the team may only view the fabric settings as readers.
What's next
If you want to quickly get started with Prophecy, you can use a trial fabric to run pipelines. These are Prophecy-managed execution environments that allow you to get started with building pipelines without needing to connect to your own external execution environment. They are designed for experimentation and initial development before connecting to external, production-grade execution environments.