Skip to main content

Set up Prophecy libraries in Databricks volumes

If Maven or PyPI is blocked in your Databricks environment, you can add Prophecy libraries directly to Databricks volumes.

Prerequisites​

Before you set up Prophecy libraries in a Databricks volume, you need to know which library version your project uses. That version must be present in the volume for the setup to work.

To check your project’s Prophecy library version:

  1. Open your project metadata.

  2. Click on the Dependencies tab.

  3. Find the relevant Prophecy library (Scala or Python) and note the version.

note

If using ProphecyLibsScala, you can look up versions on Maven Repository that are compatible with your Spark version.

Prepare a Databricks volume​

To get started, you'll need to define a Databricks volume where you will store the Prophecy libraries.

  1. Create a volume in your workspace where you'll maintain Prophecy libraries:

    databricks volumes create --name prophecy-libs --storage-location <your-volume-location>

  2. Download pyhocon from PyPI or an internal artifactory.

  3. Upload the dependency to the volumes path.

    databricks fs cp pyhocon-0.3.61-py3-none-any.whl /Volumes/<your-volume-location>/prophecy-libs/

  4. Create a local Init Script. For example, create install_pyhocon.sh with the following content:

    #!/bin/bash
    pip install /Volumes/<your-volume-location>/prophecy-libs/pyhocon-0.3.61-py3-none-any.whl

  5. Upload the script to your volumes path.

    databricks fs cp install_pyhocon.sh /Volumes/<your-volume-location>/prophecy-libs/

Upload Prophecy libraries​

Download the appropriate Prophecy library versions from the public buckets and upload them to the same volume path. Replace <version> in the example URLs based on what's used in your Prophecy project.

  • ProphecyPythonLibs

    wget https://prophecy-public-bucket.s3.us-east-2.amazonaws.com/python-prophecy-libs/prophecy_libs-<version>-py3-none-any.whl

    databricks fs cp prophecy_libs-<version>-py3-none-any.whl /Volumes/<your-volume-location>/prophecy-libs/

  • ProphecyScalaLibs

    wget https://prophecy-public-bucket.s3.us-east-2.amazonaws.com/prophecy-libs/prophecy-libs-assembly-<version>.jar

    databricks fs cp prophecy-libs-assembly-<version>.jar /Volumes/<your-volume-location>/prophecy-libs/

You can also download Prophecy libraries from Azure and GCP. For a full list of paths, visit Prophecy libraries.

Update fabric configuration​

For all relevant Databricks fabrics, you'll need to update the location of the Prophecy libraries.

  1. Open the fabric settings.

  2. Toward the end of the page, find the Prophecy Library settings.

  3. For Scala and/or Python, change the Resolution mode to File System.

  4. Under Path, add the volume path that you created earlier.

    Example: /Volumes/<your-volume-location>/prophecy-libs/

  5. Click Update to save this change.

Whitelist for UC standard clusters​

To whitelist the Init Script and the JAR path, follow the instructions to add Prophecy libraries to your Databricks allowlist.