Data Source Configuration

This page covers the configuration options for the various types of Data Sources.

Page Contents

Overview
Configuration Options

Overview

The available configuration options for a Data Source depend on the type of the Data Source. Supported Data Source types include JDBC, MongoDB, Elasticsearch, Cassandra, and SPARQL.

Configuration options are stored in a Java properties file.

A typical properties file for a JDBC-type Data Source (in this case, MySQL) looks like:

jdbc.url=jdbc:mysql://localhost/dept
jdbc.username=MySqlUserName
jdbc.password=MyPassword
jdbc.driver=com.mysql.jdbc.Driver

Configuration Options

Options Common to All Data Source Types

The following option applies to Data Sources of all types. The options apply to all Virtual Graphs for a given Data Source.

`unique.key.sets`

Default
Required	No
Description	For data sources that do not express unique constraints in their metadata, either because unique constraints are not supported or because the data source did not include some or all of the valid constraints for reasons such as performance concerns, this property is used to define additional constraints manually. The property value is a comma-separated list of keys that define unique rows in a table. Each key is itself a comma-separated list of schema-qualified columns, enclosed in parentheses. For example, if table `APP.CUSTOMERS` has an `ID` column that serves as a primary key and a pair of columns, `FNAME` and `LNAME`, that together are a unique key, the value to express that is: `(APP.CUSTOMERS.ID),(APP.CUSTOMERS.FNAME,APP.CUSTOMERS.LNAME)`

`foreign.key.sets`

Default
Required	No
Description	For JDBC data sources that do not express foreign key constraints in their metadata, or where additional column relationships exist that are not declared in the schema, this property is used to declare inclusion dependencies between tables. An inclusion dependency declares that every value in the child column(s) has a matching value in the parent column(s). Unlike SQL foreign keys, the parent columns do not need to be a primary or unique key. Each declaration is a parenthesized list of `child_table.column->parent_table.column` pairs. Multiple declarations are separated by commas. For example, if every value in `ORDERS.CUSTOMER_ID` exists in `CUSTOMERS.ID`, the value is: `(ORDERS.CUSTOMER_ID->CUSTOMERS.ID)`. For two separate single-column declarations, the value is: `(ORDER_ITEMS.ORDER_ID->ORDERS.ID),(ORDER_ITEMS.PRODUCT_ID->PRODUCTS.ID)`. For a single multi-column declaration, list all pairs in one group: `(ORDER_ITEMS.ORDER_ID->ORDER_DETAILS.ORDER_ID,ORDER_ITEMS.LINE_NO->ORDER_DETAILS.LINE_NO)`. Stardog uses these declarations to simplify generated SQL queries by eliminating unnecessary joins when the required data can be obtained from the child table alone. When the parent columns are also a unique key, more aggressive join elimination is possible. Declarations from this property are merged with any foreign keys discovered from the database metadata.

`functional.dependency.sets`

Default
Required	No
Description	Declares functional dependencies for JDBC data sources: columns whose values determine the values of other columns in the same table, without being unique in the table. Each declaration is a parenthesized `determinant columns -> dependent columns` group whose table-qualified columns all reference the same table. Multiple declarations are separated by commas. For example, if all rows of `APP.ROLES` with the same `MOVIE_ID` have the same `TITLE` and `RELEASE_YEAR`, the value to express that is: `(APP.ROLES.MOVIE_ID->APP.ROLES.TITLE,APP.ROLES.RELEASE_YEAR)`. A multi-column determinant lists its columns before the arrow: `(APP.ROLES.MOVIE_ID,APP.ROLES.ACTOR_ID->APP.ROLES.CHAR_NAME)`. Stardog uses these declarations to eliminate self-joins over denormalized tables; see Declaring Functional Dependencies. Unlike `unique.key.sets`, the determinant does not need to be unique in the table. Any two rows that agree on the determinant columns must agree on the dependent columns, with NULL treated as a value. Stardog does not validate declarations. If a declared dependency does not hold in the data, query results are undefined.

`batch.size`

Default	`100`
Required	Yes
Description	The `batch.size` option determines how many solutions will be evaluated at a Virtual Graph per request of the `ServiceJoin` operator. A larger batch size can reduce the number of requests to a data source but can make the evaluation of each request more expensive. By default, the batch size is set to `100`. The value can be adjusted in the range `[0, 10000]`. A value of `0` will disable batch requests to the data source.

JDBC Options

The following properties are used for all relational data sources.

`jdbc.url`

Default
Required	Yes
Description	The URL of the JDBC connection.

`jdbc.username`

Default
Required	No
Description	The username used to make the JDBC connection.

`jdbc.password`

Default
Required	No
Description	The password used to make the JDBC connection.

`jdbc.driver`

Default
Required	No
Description	The driver class name used to make the JDBC connection.

`sql.dialect`

Default	Inferred from supported JDBC drivers. `ORACLE` for unsupported drivers.
Required	No
Description	When using an unsupported JDBC driver, this option can be used to specify the format of the generated SQL. The options supported are `ATHENA`, `BIGQUERY`, `DB2`, `DERBY`, `EXASOL`, `H2`, `HANA`, `HIVE`, `IMPALA`, `MSSQL`, `MYSQL`, `ORACLE`, `POSTGRESQL`, `REDSHIFT`, `REST`, `SALESFORCE_CDATA`, `SALESFORCE_SIMBA`, `SINGLESTORE`, `SNOWFLAKE`, `SPARKSQL`, `SPARKSQL_DATABRICKS`, `SPARKSQL_IOMETE`, `SYNAPSE_DEDICATED`, `SYNAPSE_SERVERLESS`, `TERADATA`, `TRINO`

`sql.default.catalog`

Default	(taken from JDBC driver)
Required	No
Description	Overrides the default catalog for the connected user. Schemas in the default catalog may be referenced without qualification (`myschema` rather than `mycatalog.myschema`). For use with databases (like Databricks) that support 3-level (catalog/schema/table) identifiers. See also: Databases and Schemas

`sql.catalogs`

Default
Required	No
Description	A comma-separated list of catalogs. This list plus the default catalog define the set of catalogs represented by a `.` wildcard for the `sql.schemas` option. An asterisk (`*`) can be used as a wildcard for this, the `sql.catalogs` option, meaning all catalogs accessible to the JDBC connection. For use with databases (like Databricks) that support 3-level (catalog/schema/table) identifiers. See also: Databases and Schemas

`sql.default.schema`

Default	(taken from JDBC driver)
Required	No
Description	Overrides the default schema for the connected user. Tables in the default schema may be referenced without qualification (`mytable` or `"mytable"` rather than `myschema.mytable`, `"myschema"."mytable"`, or, for Databricks, `mycatalog.myschema.mytable`). See also: `sql.default.catalog` and Databases and Schemas

`sql.schemas`

Default
Required	No
Description	A comma-separated list of schemas to append to the schema search path. This option allows SQL queries from the virtual graph mappings to reference tables that are outside of the default schema for the connected user. An asterisk can be used as a wildcard. A lone asterisk (``) means all schemas in the default catalog, `mycatalog.` means all schemas in the `mycatalog` catalog, and `.` means all schemas in all the accessible catalogs. See also: `sql.default.catalog` and Databases and Schemas

`sql.skip.validation`

Default	`false`
Required	No
Description	Set this option to `true` to bypass the validation of catalog, schema, table, and column references against metadata from the backing database. These checks are made in order to catch configuration problems early. Should not be necessary with any supported database and driver but can be used with a JDBC driver or database that does not support returning schema metadata.

`jdbc.row.count.estimate.strategy`

Default	`JDBC_GET_INDEX_INFO`
Required	No
Description	Determines what methods for estimating a table’s row count will be used and in what fallback order. Each dialect has a method for getting row count from database metadata, which typically involves querying indexes or system tables. This option controls whether a brute-force `SELECT COUNT()` query should be used. Note: The index/system-table method is considered a failure if it returns a row count estimate less than 2. Valid options are: • `JDBC_GET_INDEX_INFO`: Use only JDBC metadata or dialect-specific system tables. Do not use `SELECT COUNT()`. • `COUNT_FALLBACK`: Try JDBC metadata first, fall back to `SELECT COUNT()` if the metadata method fails. • `COUNT_FIRST`: Try `SELECT COUNT()` first, fall back to JDBC metadata if the query fails.

`jdbc.table.sample.percentage`

Default	`0.1` (varies by dialect)
Required	No
Description	When a brute-force `SELECT COUNT()` is used to determine a table’s row count (see `jdbc.row.count.estimate.strategy`), and the dialect supports the SQL `TABLESAMPLE` keyword, this option determines the percentage of rows to include in the sample. Valid range is 0-100. A value of 0 disables sampling (performs a full table scan). For example, with a value of 10, the query would be `SELECT COUNT() FROM table_name TABLESAMPLE SYSTEM (10 PERCENT)`. The sampled count is automatically scaled to estimate the full table size. Not all dialects support table sampling; for those that do not, this option is ignored.

Tomcat Connection Pool Options

Additionally, connection pool properties for the built-in Tomcat connection pool are allowed. This set of additional allowed properties is listed in the Tomcat JDBC Connection Pool documentation. Stardog sets these connection pool defaults:

initialSize=3
testWhileIdle=true

# 4 hours
timeBetweenEvictionRunsMillis=14400000

validationQueryTimeout=10

To disable connection pooling, one can set the following connection pool properties:

testOnBorrow=true
timeBetweenEvictionRunsMillis=0
maxIdle=0
minIdle=0

This may be useful for troubleshooting some configurations.

Passing Options Directly to the JDBC Driver

Any options with the prefix ext. will be passed directly to the JDBC Driver. Tomcat connection pool properties do not need to be prefixed with ext.

Any unknown options will be ignored.

Supplying Driver Files as Options

Some JDBC drivers take configuration through a property whose value is a path on the machine running the driver, rather than the configuration itself. The CData drivers are the common case: the REST driver’s Location names a directory of .rsd schema files, and properties such as OAuthJWTCert and SSLClientCert name certificate or keystore files. Setting these with ext. options requires putting the files on the Stardog server’s file system, which requires system administrator access to the server, and makes the data source depend on files that exist on only one node.

As of Stardog 12.2, options prefixed with driver.file. supply the contents of those files instead. Stardog writes the contents to a directory of its own on the server and sets the driver property to the path of what it wrote. An option takes one of three forms:

# A single file: the driver property is set to the path of a file holding this content
driver.file.OAuthJWTCert=-----BEGIN PRIVATE KEY----- ...

# A directory: the part after '/' is the file's path within the directory, and the
# driver property is set to the path of the directory itself
driver.file.Location/people.rsd=<rsd content>
driver.file.Location/cars.rsd=<rsd content>

# Binary content, encoded in base64
driver.file.base64.SSLClientCert=MIIKuAIBAzCCCnQGCS ...

Writing a certificate or schema file onto one line of a properties file is awkward, so in a data source or virtual graph properties file given to the CLI, a driver.file. value that starts with @ names a file or directory to read the contents from instead:

driver.file.Location=@schemas/
driver.file.OAuthJWTCert=@certs/jwt.pem
driver.file.base64.SSLClientCert=@certs/client.p12

A relative path is resolved against the directory that holds the properties file.
A directory expands to one option per file it contains. In the example above, driver.file.Location=@schemas/ becomes driver.file.Location/people.rsd, driver.file.Location/cars.rsd, and so on.
Files are read as UTF-8 text, or as bytes for a driver.file.base64. option.
To start a value with a literal @, write @@.

The data-source add, virtual add, virtual import and virtual source_metadata commands expand these references on the machine running the command, so the server receives ordinary driver.file. options. Clients that call the HTTP API pass the contents directly.

Things to keep in mind:

Precedence. A driver.file. option takes precedence over an ext. option for the same driver property. ext. options pointing at server paths keep working wherever they are allowed.
Size. The contents of all the driver.file. options of one data source may total at most 1 MB. They are meant for configuration such as schemas, certificates and keystores, not for data.
Storage and replication. The contents are part of the data source definition, which is stored encrypted in the system database and replicated to every node of a cluster. Each node writes its own copy of the files under STARDOG_HOME. The files are cleared and recreated when the server starts, and removed when the data source is removed or its contents change.
Updating. To change a file, update the data source, for example with data-source add --overwrite. The new contents are written to a new directory alongside the old ones, and the old files are removed once the previous version of the data source is closed.
Property names. The name after driver.file. must be the driver property name: letters, digits, ., _ and -. base64 is reserved as a marker and cannot be used as a property name.
Write targets. Properties that name a file the driver writes, such as Logfile, cannot be supplied this way.

On servers that disable server file paths with virtual.datasource.server.file.paths.enabled, driver.file. options are the only way to configure these properties.

MongoDB Options

`mongodb.uri`

Default
Required	Yes
Description	The URI for the MongoDB connection. Examples: `mongodb://localhost/mydb` or `mongodb+srv://myUserName:myP4ssw0rd@cluster0-kgprod.company.com/mydb`

Elasticsearch Options

`elasticsearch.rest.urls`

Default
Required	Yes
Description	Whitespace-delimited list of connection host/port values for Elasticsearch. Example: `server1:9200 server2:9200 server3:9200`

`elasticsearch.indexes`

Default	All accessible indexes available at `elasticsearch.rest.urls`
Required	No
Description	A comma-delimited list of indexes to make available from this data source.

`elasticsearch.username`

Default
Required	No
Description	Username for Elasticsearch connections.

`elasticsearch.password`

Default
Required	No
Description	Password for Elasticsearch connections.

`elasticsearch.connect.timeout`

Default	`1s`
Required	No
Description	Time to wait when establishing a TCP connection to an Elasticsearch host, as a duration such as `500ms`, `1s`, or `1m`; a bare number is milliseconds. `0` disables the timeout.

`elasticsearch.socket.timeout`

Default	`30s`
Required	No
Description	Longest time to wait for data on an established connection: for Elasticsearch to start answering a search, scroll, or clear-scroll request, or to send the next part of a response. It does not cap the total time a request takes. Set as a duration such as `30s` or `2m`; a bare number is milliseconds. Raise this if searches against a busy cluster time out before Elasticsearch starts responding. `0` disables the timeout, in which case a request to an unresponsive node waits indefinitely and cannot be interrupted by a query timeout or by cancelling the query, because the Elasticsearch connector does not support cancelling a request that is in flight.

`elasticsearch.connection.idle.timeout`

Default	`30s`
Required	No
Description	Longest a pooled connection may stay idle after its last response before Stardog discards it instead of reusing it, as a duration such as `30s` or `5m`; a bare number is milliseconds. A connection that is in use is not affected, however old it is. Load balancers, NAT gateways, and firewalls between Stardog and Elasticsearch commonly drop idle connections without notifying the client, and a request sent on such a connection fails with `Connection reset`. Keep this value below the idle timeout of any device in the path. `0` keeps connections indefinitely.

Cassandra Options

`cassandra.contact.point`

Default
Required	Yes
Description	The address of the Cassandra node(s) that the driver uses to discover the cluster topology. Example: `cassandra.abc.com`

`cassandra.port`

Default	9042
Required	No
Description	The port to use to connect to the Cassandra host.

`cassandra.keyspace`

Default
Required	Yes
Description	The Cassandra keyspace to use for this session.

`cassandra.username`

Default
Required	No
Description	The username for the Cassandra cluster.

`cassandra.password`

Default
Required	No
Description	The password for the Cassandra cluster.

`cassandra.allow.filtering`

Default	`false`
Required	No
Description	Whether to include the `ALLOW FILTERING` clause at the end of Cassandra CQL queries. Not recommended for production use.

SPARQL Engine/Service Options

`sparql.url`

Default
Required	Yes
Description	SPARQL query endpoint/connection string with database specified, i.e. `http://myhost:26023/testdb/query`

`sparql.username`

Default
Required	No
Description	The username to access the SPARQL endpoint.

`sparql.password`

Default
Required	No
Description	The password to access the SPARQL endpoint.

`sparql.graphname`

Default
Required	Yes
Description	The graph name on the SPARQL endpoint to be mapped as virtual graph.

`sparql.statsbasedoptimization`

Default	`true`
Required	No
Description	Whether to enable statistics-based optimization while accessing the SPARQL endpoint.

Overview
Configuration Options