Skip to main content

Datasets

In Prophecy, datasets are grouped by projects and rely on the following:

  • Schema: The structure or shape of the data, including column names, data types, and the method for reading and writing the data in this format.
  • Fabric: The execution environment in which the data resides.

Create datasets​

Datasets are created where they are first used in a Source or Target gems. A dataset definition includes its:

  • Type: The type of data you are reading/writing like CSV, Parquet files or catalog tables.
  • Location: The location of your data. It could be a file path for CSV or a table name.
  • Properties: Properties consists of Schema and some other attributes specific to the file format. For example, in case of CSV, you can give Column delimiter in additional attributes. You can also define Metadata for each column here like description, tags, and mappings.

Datasets can be used by any pipeline within the same project, and in some cases by other projects within the same team.

View datasets​

There are two ways to view a list of datasets:

  • To see all datasets, navigate to Metadata > Datasets.
  • To see only one project's datasets, navigate to Metadata > Projects. Then, open a project. Click on the Content tab, and then the Datasets subtab.

Dataset Metadata​

If you open the metadata page for one of the datasets, you'll find the following information:

NameDescription
Dataset nameThe name of this dataset, which is editable.
Dataset descriptionThe description of this dataset, which is editable.
Dataset propertiesA subset of properties used for reading or writing to this Dataset.
Dataset schemaThe columns of this dataset and their data types.
Delete DatasetThe option to delete this dataset. Use with caution.

In the Relations tab, there is additional information about where and how this dataset is used.

NameDescription
Physical DatasetsLocation of the dataset in relation to a fabric.
PipelinesA list of pipelines that use this dataset, with the Relation column indicating if it is for Read or Write purposes.
JobsA list of jobs that use this dataset, with the Relation column indicating if it is for Read or Write purposes.
Open Lineage ViewerThe option to open this dataset in the Lineage viewer, showing column-level lineage for this dataset.

Publishing and sharing datasets​

As part of the project release process, datasets within that project are published to other projects within the same Team, and can be published to other Teams in read-only mode. This allows you to share your dataset configurations with other Teams without allowing them to make changes to the original dataset definitions. Let's see this in action:

  1. DI_TEAM is the central Data Infrastructure team. They have defined a common project named DI_Common_Python.
  2. DI_Common_Python has a number of datasets defined within it: DI Common Datasets
  3. The DI_Team merges and releases the DI_Common_Python project, tagging it 0.1. DI Common Release
  4. As you can see, the DI_Team has published the DI_Common_Python project to the DE_Team, the Data Engineering Team.
  5. Now, whenever the DE_Team builds pipelines, they can see the following: Common Datasets

We can see the DI_Common_Python project's datasets, and the fact that they're listed as Read-only. This means that DE_Team can use the datasets, but cannot edit them.

info

For regular usage, we suggest having only one instance of a particular dataset within a pipeline, as the dataset's properties and underlying data can change each time the dataset is read or written.