Skip to main content

Projects

A Project in Prophecy is the core unit for developing, organizing, and deploying data workflows to production. It encompasses all the components needed for building and running data processes.

Project types​

When you create a project, you must choose a project type: Spark/Python, Spark/Scala, or SQL. Visually designed pipelines will generate code in the selected Project output language.

Project language

Spark projects contain Pipelines, Datasets, and Jobs. SQL Projects contain Models, Datasets, and Jobs. You can view these components in the Metadata tab of the Prophecy user interface.

note

It is not currently possible to switch the output language of a Project after it has been created.

Projects and Git​

Each project must be stored in a Git repository. You can either choose to host your project on a Prophecy-managed repository, or you can connect your own external repository, like one on GitHub. Additionally, all assets within a Project (like Pipelines, Models, Datasets, and Jobs) are stored as code in the Project's Git repository.

Open any Pipeline (left) or Model (right) in your Project. Simply toggle from visual to code to see the underlying code for that Pipeline or Model, as well as the rest of the Project components.

Visual To Code

Version control lets users and teams collaborate on Projects, contribute simultaneously, and reuse code.

What's next​

To learn about the Project lifecycle, visit Develop and deploy a Project.

To explore different Project components, view the following pages: