Databricks Volumes gem

Prophecy Automate

Use a Source or Target gem to read from or write to Databricks Volumes in Prophecy pipelines. This page covers supported file formats, how to create the gem, and how to configure connection details and paths for both Source and Target gems.

Supported file formats

Format	Read	Write
CSV	Yes	Yes
Fixed width	Yes	No
JSON	Yes	Yes
Parquet	Yes	Yes
XLSX	Yes	Yes
XML	Yes	Yes

Create a Databricks Volumes gem

To create a Databricks Volumes Source or Target gem in your pipeline:

1. Set up your Databricks connection.

2. Add a new Source or Target gem to your pipeline canvas and open the configuration.

3. In the Type tab, select Databricks under File. Do not select Databricks under Table.

4. In the Location tab, choose your file format and location.

   info

   For more information on how to configure this screen, jump to Location.

5. In the Properties tab, set the file properties. These vary based on the file type that you are working with.

   info

   See the list of properties per file type.

6. (Source only) In the Preview tab, load a sample of the data and verify that it looks correct.

Location

Parameter	Description
Format type	Type of file to read or write, such as csv or json.
Select or create connection	Select an existing Databricks Volumes connection or create a new one.
Filepath	Path to the file in Databricks Volumes. Example: /Volumes/catalog/schema/volume/file.csv