Databricks Configuration

This page discusses how to configure Databricks as an external compute platform.

Page Contents

1. Mandatory properties:
2. Optional properties:
3. Credentials from the cluster environment
   1. Store the credentials in a secret scope
   2. Map the secrets into the cluster environment
   3. Enable the property on the data source

---

A Databricks Data Source added in Stardog using the data-source add CLI command or Stardog Studio can be registered as an external compute platform. Add specific properties in the data source definition to configure the Databricks data source as an external compute platform.

Mandatory properties:

Optional properties:

Credentials from the cluster environment

Stardog passes the Stardog credential and the source database password to the Databricks job as job parameters. Set external.compute.credentials.from.environment to true on the external compute data source to leave them out. The job then reads them from environment variables on the cluster, which you populate from a Databricks secret scope.

This requires stardog-spark-connector 3.4.0 or later.

Store the credentials in a secret scope

The job writes back to Stardog as a Stardog user. It can authenticate either with that user’s name and password, or with a token. Store one or the other: a job that finds both fails, because it cannot tell which to use.

A token is generated with the user token CLI command:

$ stardog-admin user token -u etl-service -p "..."

Create a scope in the Databricks workspace and put the Stardog credential and the source database credentials in it:

$ databricks secrets create-scope stardog-external-compute
$ databricks secrets put-secret stardog-external-compute stardog-token --string-value "eyJhbGciOiJIUzI1NiJ9..."
$ databricks secrets put-secret stardog-external-compute jdbc-user --string-value "reporting"
$ databricks secrets put-secret stardog-external-compute jdbc-password --string-value "..."

To authenticate with a password instead of a token, store stardog-username and stardog-password in the scope and leave stardog-token out.

Map the secrets into the cluster environment

On the cluster, under Advanced options > Spark > Environment variables, reference the secrets. The same values can be set through the spark_env_vars field of the Clusters API:

STARDOG_TOKEN={{secrets/stardog-external-compute/stardog-token}}
STARDOG_JDBC_USER={{secrets/stardog-external-compute/jdbc-user}}
STARDOG_JDBC_PASSWORD={{secrets/stardog-external-compute/jdbc-password}}

With a password instead of a token, set STARDOG_USERNAME and STARDOG_PASSWORD from the scope in place of STARDOG_TOKEN.

These are the variables the connector reads:

The variable that holds the credential decides how the job authenticates. A token authenticates as its own owner, so STARDOG_USERNAME is needed only alongside STARDOG_PASSWORD.

An environment value is used as it stands and is never decoded, so store each credential exactly as it was issued.

Databricks substitutes a {{secrets/scope/key}} reference when the cluster starts, so a rotated secret reaches the job only after the cluster is restarted. The substitution works in the cluster specification fields only, never in job parameters.

Spark’s spark.redaction.regex masks values whose key contains secret, password or token. The variable names above already match it, which keeps the values out of the Environment tab of the Spark UI.

Enable the property on the data source

Add the property to the external compute data source:

external.compute=true
external.compute.host.name=adb-XXXXXXXXXXXXXX.XX.azuredatabricks.net
databricks.cluster.id=0704-XXXXXX-XXXXXdir
stardog.host.url=https://myhost.stardog.cloud:5820
external.compute.credentials.from.environment=true

$ stardog-admin data-source add databricks.properties

The job authenticates to Stardog as the identity held in the secret scope rather than as the user who triggered the operation, so the write back to Stardog carries that identity’s permissions rather than the triggering user’s. Grant it only what the target database requires.

---