Databricks OAuth

Prophecy provides Databricks OAuth to align with industry-standard authentication flows. This gives you more granular access control and better security, making it a good alternative to Personal Access Tokens (PATs). This integration works with both Spark clusters and SQL warehouses.

Requirements

To leverage Databricks OAuth in your Prophecy deployment, you need:

• Prophecy 3.4.3.1 or later.
• An app connection between Databricks and Prophecy. This connection is set up by default in SaaS environments and no further action is needed.

For Dedicated SaaS and self-hosted deployments, follow the steps below. Prior to performing this one-time registration process, users will see the OAuth option in fabrics but will be unable to use it.

Register Prophecy as an app connection in Databricks

First, a Databricks account admin needs to complete the following steps once for your Prophecy deployment:

1. On Databricks, navigate to Account Settings > App connections in your account console.
2. Create a new App connection for Prophecy. Ensure that:
   • Access scopes are set to ALL APIs.
   • The redirect URL contains <your_prophecy_url>/api/databricks/oauthredirect.
3. This process generates Databricks OAuth Application fields on the Prophecy side.
4. Under Client ID, copy your OAuth Client ID for the application, and share it with your Prophecy Cluster Admin.
5. Under Client secret, select Generate a client secret. Share it with your Prophecy Cluster Admin.
6. Click Save.

Then, the Prophecy cluster admin has to add the Databricks credentials to Prophecy:

1. Navigate to Admin Settings > Security.
2. Under Databricks OAuth Application (U2M), paste the Client ID and the Client Secret into the respective fields.
3. Finally, the Prophecy Kubernetes cluster must be restarted to enact these changes.

Use cases supported by Databricks

In Prophecy, you can use Databricks OAuth in two ways: either using user identities or a service principal.

Type	Identity	Use Cases	Behavior	Notes
User-based OAuth (U2M)	Individual user accounts	Pipeline development, job development	Executes actions with the user's Databricks permissions. Users sign in via the Databricks login screen.	Requires periodic login based on Databricks OAuth timeout settings. Default timeouts are used in SaaS, but are configurable in Dedicated SaaS/self-hosted deployments.
Service Principal OAuth (M2M)	Service principal	Project deployment, scheduled pipeline execution	Authenticates using service principal credentials. All scheduled jobs run as the service principal.	Required for automation and CI/CD. You will see a warning if service principal credentials are missing.

note

The minimum level of access that your service principal requires (including access to clusters, tables, etc.) will vary per use case.

To learn how to configure Databricks OAuth for different fabric types, see:

• Databricks connection in a Prophecy fabric
• Databricks connection in a Spark fabric
• Databricks connection in a SQL fabric

Security

Prophecy ensures robust security measures during the OAuth authentication process, following best practices documented by Databricks. For example:

• The authentication process follows a three-step OAuth flow to generate tokens, leveraging Proof Key for Code Exchange (PKCE) to enhance security. Prophecy uses a Prophecy-hosted callback URL to capture and process authorization codes, issuing and securely storing access tokens.
• Prophecy securely stores the refresh token, which is used to renew itself and obtain new access tokens, to ensure uninterrupted authenticated connectivity to Databricks. Tokens are encrypted before being stored, following the same stringent encryption standards applied to other credentials managed by Prophecy.