External Catalogs

This page discusses importing and using metadata from external catalog systems.

Page Contents

1. Overview
   1. External Credentials
      1. Adding a credential
      2. Listing existing stored credentials
2. Databricks Unity Catalog
   1. Configuration
   2. Data Model
3. Collibra
   1. Configuration
   2. Data Model
4. Microsoft Purview
   1. Configuration
   2. Exporting Stardog Metadata
   3. Data Model
5. JDBC
   1. Configuration
   2. Data Model

---

Overview

The Knowledge Catalog can import metadata from external catalog systems to enable a single unified semantic layer over multiple catalogs.

External Credentials

Metadata providers that call external REST APIs cannot use Data Sources to hold credentials. The Catalog provides CLI commands to store encrypted usernames and passwords in the catalog credential store. A token is returned that is used in provider configurations. The token is exchanged for the stored credentials during server processing.

The catalog credential store is a user configurable location for storing credential strings. The stored strings can be configured to be stored as keyless or encryption key hashes.

The following options can be used to override the defaults.

By default the credential store expectes a DER encoded RSA key pair located in the Stardog home directory. The keys need to be named catalog.priv and catalog.pub.

The following example uses openssl to generate an RSA key pair:

• Create an RSA PEM encoded file

  openssl genrsa -out catalog.pem 2048
  

• Extract the private key as a DER encoded file

  openssl pkcs8 -topk8 -nocrypt -in catalog.pem -outform der -out catalog.priv
  

• Extract the public key as a DER encoded file

  openssl pkey -in catalog.pem -pubout -outform der -out catalog.pub
  

To add credentials to the store use the CLI administrator commands catalog credentials-add and catalog credentials-list.

Adding a credential

In the following example a username my_user_name is added along with the corresponding password. The returned UUID 04bee4c7-26cc-4c97-b817-dbb3298fa842 is the value that is the used in the access_key property for the metadata provider using these credentials.

$ stardog-admin catalog credentials-add
Username: my_user_name
Password: 
Description (optional): My External System Credentials

04bee4c7-26cc-4c97-b817-dbb3298fa842

Listing existing stored credentials

In the following example the account credentials that were added are now listed. Only the UUID and the user provided description are retrieved. There is no way to retrieve the original credential values. The credential values can only be retrived by the catalog server once they are stored.

$ stardog-admin catalog credentials-list

Catalog Stored Credentials
+--------------------------------------+--------------------------------+
|              Access Key              |          Description           |
+--------------------------------------+--------------------------------+
| 04bee4c7-26cc-4c97-b817-dbb3298fa842 | My External System Credentials |
+--------------------------------------+--------------------------------+

Databricks Unity Catalog

The Knowledge Catalog can be configured to import Unity Catalog metadata using a Databricks account. You can configure the import to occur on a customizable schedule. Databricks Unity metadata is written to the Stardog Catalog, where it can be queried in conjunction with your Stardog databases.

Configuration

To import Databricks Unity Catalog metadata, you insert a DatabricksProvider configuration into the Knowledge Catalog’s stardog:catalog:providers named graph. The configuration describes how Databricks can be accessed and how often the Knowledge Catalog should refresh the metadata.

insert data {
    graph stardog:catalog:providers 
    { 
        <urn:myDBricksProvider> a <tag:stardog:api:catalog:DatabricksProvider> ;
            <tag:stardog:api:catalog:provider:dataSource> <DATA_SOURCE_HERE> ;
            <tag:stardog:api:catalog:provider:schedule> "SCHEDULE_HERE"  .
    }
} 

This table details the property values that need to be set for configuring a Databricks metadata provider.

After the configuration is inserted, a job is automatically created to run on the specified schedule. The job will import Databricks Unity metadata and load a general data model for viewing the metadata in Explorer.

Data Model

This table contains the classes used for modeling the Databricks metadata. Prefix bricks is namespace tag:stardog:api:catalog:databricks:.

Collibra

The Knowledge Catalog can be configured to import data from a Collibra Data Intelligence Cloud account. Collibra is a data catalog product that collects business glossary, data governance, lineage and compliance metadata.

Configuration

The configuration for a Collibra provider requires that Collibra credentials be stored in the catalog credential store. See storing credentials for details.

Collibra Data Intelligence Cloud uses HTTP Basic Auth with a username and password for authentication. Add your account username and password to the catalog credential store and use the returned access key to configure the Collibra provider.

insert data {
    graph stardog:catalog:providers 
    { 
        <urn:collibra> a <tag:stardog:api:catalog:CollibraProvider> ;
            <tag:stardog:api:catalog:provider:accessKey> "ACCESS_KEY_HERE" ;
            <tag:stardog:api:catalog:provider:serverAddress> "CLOUD_URL_HERE" ;  
            <tag:stardog:api:catalog:provider:schedule> "SCHEDULE_HERE"  .
    }
} 

This table details the property values that need to be set for configuring a Collibra metadata provider.

Data Model

This table contains the classes used for modeling the Collibra metadata. Prefix collibra is namespace tag:stardog:api:catalog:collibra:.

Microsoft Purview

The Knowledge Catalog can be configured to both import data from an Microsoft Purview application running on Azure and export Stardog catalog data back into it. Purview is Microsoft’s data governance, cataloging and protection product.

Configuration

The configuration for a Purview provider requires that Purview credentials be stored in the catalog credential store. See storing credentials for details.

Microsoft Purview on Azure uses the OAuth client_credentials grant type for authorization. Add your Azure client id and application client secret to the catalog credential store and use the returned access key to configure the Purview provider.

insert data {
    graph stardog:catalog:providers 
    { 
        <urn:purview> a <tag:stardog:api:catalog:PurviewProvider> ;
            <tag:stardog:api:catalog:provider:accessKey> "ACCESS_KEY_HERE" ;
            <tag:stardog:api:catalog:provider:serverAddress> "CLOUD_URL_HERE" ;  
            <tag:stardog:api:catalog:provider:tenantId> "AZURE_TENTANT_ID_HERE" ;  
            <tag:stardog:api:catalog:provider:schedule> "SCHEDULE_HERE"  .
    }
} 

This table details the property values that need to be set for configuring a Purview metadata provider.

Exporting Stardog Metadata

The Purview provider exports data source metadata into the configured Purview server. There is no extra configuration required to export Stardog metadata. When the provider’s scheduled job is run the export will automatically occur after the import is completed.

The following Purview asset types are added for the Stardog data source metadata:

After the scheduled provider job has run you can log into your Purview account and view the Stardog metadata. It is located in a custom stardogcatalog collection.

Data Model

This table contains the classes used for modeling the Purview metadata. Prefix purview is namespace tag:stardog:api:catalog:purview: and the atlas prefix is namespace tag:stardog:api:catalog:atlas:.

JDBC

The Knowledge Catalog can be configured to use a JDBC driver to import database metadata into the Knowledge Catalog. Any JDBC compliant driver should work. Be sure to first add the JAR file to the classpath of the Stardog server.

Configuration

The configuration for a JDBC provider requires that credentials be stored in the catalog credential store. See storing credentials for details.

Currently, the JDBC provider expects a username and password for authentication. Add your database username and password to the catalog credential store and use the returned access key to configure the JDBC provider. When the provider import job is run, the standard jdbc.username and jdbc.password properties will be injected to the JDBC connection string.

insert data {
    graph stardog:catalog:providers 
    { 
        <urn:purview> a <tag:stardog:api:catalog:JdbcProvider> ;
            <tag:stardog:api:catalog:provider:accessKey> "ACCESS_KEY_HERE" ;
            <tag:stardog:api:catalog:provider:jdbcDriver> "DRIVER_CLASSNAME_HERE" ;  
            <tag:stardog:api:catalog:provider:jdbcURL> "JDBC_CONNECTION_STRING_HERE" ;  
            <tag:stardog:api:catalog:provider:schedule> "SCHEDULE_HERE"  .
    }
} 

This table details the property values that need to be set for configuring a JDBC metadata provider.

Data Model

This table contains the classes used for modeling the Purview metadata. Prefix jdbc is namespace tag:stardog:api:catalog:jdbc: and the catalog prefix is namespace tag:stardog:api:catalog:.

---