Link Search Menu Expand Document
Start for Free

External Compute

This chapter discusses External Compute - one of Stardog’s features for pushing heavy workloads to compute platforms like Databricks, EMR Serverless, and Spark on Kubernetes. This page primarily discusses what is external compute, how it works, and what are the supported operations. See the Chapter Contents for a short description of what else is included in this chapter.

Page Contents
  1. Overview
  2. How it Works
  3. Compatibility Table:
  4. Architecture
  5. OAuth and Authentication
    1. Identity provider users
    2. Keeping credentials out of the job parameters
  6. Chapter Contents

Overview

Stardog supports operations where the workload can be pushed to external compute platforms.

The supported compute platforms are:

  1. Databricks
  2. EMR Serverless
  3. Spark on Kubernetes (via the Spark Operator)

Supported Operations are:

  1. Virtual Graph Materialization using virtual-import CLI command and SPARQL Update Queries (add/copy)
  2. Virtual Graph Cache using cache-create and cache-refresh CLI commands
  3. Resolve Entities using entity-resolution resolve CLI command

How it Works

Stardog converts the supported operation into a Spark job, connects to the external compute platform, uploads the stardog-spark-connector.jar, and then creates and triggers the Spark job. This Spark job does the required computation on the external compute platform and then connects back to the Stardog server (using the user’s credentials that triggered the operation) to write the results to Stardog.

The Stardog Platform uploads the latest compatible stardog-spark-connector.jar version when the external compute supported operation is triggered if this jar is not on the compute platform. For more information on various configurations available around the stardog-spark-connector.jar, please refer to Configuring External Compute Datasource

If the older version of stardog-spark-connector.jar is available on the compute platform, uninstall the older version. Refer to the compute platform documentation on how to uninstall the libraries.

Compatibility Table:

Stardog Platform Versions Compatible stardog-spark-connector Versions Spark versions
8.2.* 2.0.0 3.2.2
9.0.* 3.0.0 3.2.2
9.1.* 3.1.0 3.2.2
10.*, 11.* 3.2.0 3.5.0
11.* 3.3.0 3.5.0
12.0.*, 12.1.* 3.3.0 3.5.0
12.2.* 3.4.0 3.5.0

Architecture

The high-level architecture for external compute is as shown:

spark multiple run

OAuth and Authentication

An external compute job connects back to Stardog to write its results. By default, it authenticates with a token that Stardog signs at submission time for the user who triggered the operation, on every compute platform and for every supported operation.

The exception is a job that reads its credentials from the environment of the cluster it runs on. Stardog signs no token for it, and it authenticates as the identity held in that environment rather than as the triggering user. The rest of this section describes the default.

Stardog ships a default JWT configuration that includes a signer, so a server that has never been configured for JWT issues and accepts these tokens with no setup. Two configurations stop it:

  1. jwt.disable=true in stardog.properties. Stardog cannot create the token, so no external compute operation can run unless it reads its credentials from the environment.
  2. A custom JWT configuration file, set through the jwt.conf property, with no signer entry. A custom file must define a signer of its own.

Identity provider users

External Compute supports OAuth for Identity Providers (IDPs) that are integrated with Stardog, including Azure, Okta, and others. See OAuth 2.0 Integration for how an IdP is configured. With IDP OAuth (e.g. Azure, Okta), the user is created in Stardog on first sign-in and roles assigned in the IDP are mapped to Stardog roles. When an external compute job is run, Stardog passes the triggering user’s username and a token (created for that user and their roles) to the Spark job; the job uses it to connect back to Stardog and perform the operation.

The token carries the user’s roles in a stardogRoles.externalCompute claim, and a default configuration does not read that claim. A user who has a password in Stardog is unaffected, because their permissions come from their own role assignments. A user created by an identity provider has no password, so the job is refused unless the roles arrive with the token. Add an issuer entry naming that claim to the JWT configuration file, alongside the signer entry:

https://${ip_address_where_stardog_is_running}:${port_number}:
  usernameField: stardog-username
  rolesClaimPath: stardogRoles.externalCompute

These OAuth credentials can be used to connect to Stardog and for the job to call back to Stardog. Pass-through authentication for the Virtual Graph is not supported in External Compute.

Keeping credentials out of the job parameters

Stardog passes the Stardog credential and the source database password to the job as job parameters. Set external.compute.credentials.from.environment to true in the external compute configuration to leave them out. The job then reads them from environment variables on the machine it runs on, which you provide. How those variables are set depends on the compute platform: see Databricks and Spark on Kubernetes.

In this mode the job authenticates as the identity you supply rather than as the user who triggered the operation.

Chapter Contents