Skip to main content

Self-hosted

Prophecy is written as a set of microservices that run on a Kubernetes cluster. Our recommended platforms to run Kubernetes are:

  • Amazon Web Services managed EKS
  • Microsoft Azure managed AKS
  • Google Cloud Platform managed GKE

There are a couple of ways to install a self-hosted Prophecy deployment:

  • Installation via Helm
  • Installation via Marketplaces

Once Prophecy is installed, you'll have to manually perform upgrades, backups, restores, etc.

info

For personalized help, contact support. If applicable, send along your logs to help us troubleshoot.

Spark​

After installation, Prophecy requires the following for interactive development:

  • Databricks API 1.2 for Databricks-based Spark deployments, or
  • Livy 0.7.x for any other Spark deployment support (like CDP, HDP, MapR, Spark on Kubernetes).

Airflow​

For interactive and jobs deployment to Airflow, Prophecy requires a customer-managed Airflow deployment version 2.x.x (latest recommended). Astronomer's managed Airflow offering is supported.

Logging / Metrics​

  • Prophecy comes with a built-in lightweight infrastructure for monitoring (based on Loki & Grafana) and logging (based on Prometheus, Grafana and alert-manager, etc.).
  • You can optionally redirect the logs and metrics to your own logging services.

Prophecy services​

You can use the Running Services API to retrieve the status of your Prophecy services. 200 denotes that the service is running, while 404 denotes that the service is disabled. Any other error code denotes a service failure.

Running Services API​

Example:

curl 'https://<prophecy-env-url>/athena/api/v1/prophecy/status'
Response:
{
 "anyServiceDown": false,
 "data": {
  "services": [
   {
    "isPrimary": true,
    "name": "App",
    "statusCode": 200
   },
   {
    "isPrimary": true,
    "name": "Metadata",
    "statusCode": 200
   },
   {
    "isPrimary": false,
    "name": "Pipeline/Jobs Editor",
    "statusCode": 200
   },
   {
    "isPrimary": false,
    "name": "Prophecy Managed Git",
    "statusCode": 200
   },
   {
    "isPrimary": false,
    "name": "Execution",
    "statusCode": 200
   },
   {
    "isPrimary": false,
    "name": "CI / CD",
    "statusCode": 200
   },
   {
    "isPrimary": false,
    "name": "Lineage",
    "statusCode": 200
   },
   {
    "isPrimary": false,
    "name": "Search",
    "statusCode": 404
   },
   {
    "isPrimary": false,
    "name": "Kafka",
    "statusCode": 404
   },
   {
    "isPrimary": false
   },
   {
    "isPrimary": false,
    "name": "Transpiler",
    "statusCode": 404
   },
   {
    "isPrimary": false,
    "name": "Data Quality",
    "statusCode": 404
   },
   {
    "isPrimary": false,
    "name": "Data Copilot",
    "statusCode": 404
   },
   {
    "isPrimary": false,
    "name": "Sandboxing",
    "statusCode": 200
   },
   {
    "isPrimary": false,
    "name": "Database Connectivity",
    "statusCode": 200
   }
  ]
 },
 "isProphecyDown": false,
 "success": true
}