Server Monitoring

This page discusses how to monitor the Stardog server.

Page Contents

1. Overview
2. Accessing Monitoring Information
   1. Prometheus
      1. Prometheus Metrics Filters
      2. Filter examples
3. JMX Monitoring
4. Disabling Monitoring
5. Knowledge Graph Metrics
   1. Instance Wide Metrics
6. Block Cache Metrics
7. Database Metrics
   1. Per Index Metrics
8. HTTP Server Metrics
9. Memory Usage Metrics
   1. Memory Management Metrics
   2. Java Memory Metrics
   3. Process Memory Metrics
10. Process Metrics

Overview

Stardog provides server monitoring via the Metrics library. In addition to providing some basic JVM information, Stardog also exports information about the Stardog DBMS configuration, as well as stats for all databases within the system (e.g., the total number of open connections, database size, and average query time).

Accessing Monitoring Information

Monitoring information is available via the Java API, the HTTP API, the CLI, or (if configured) the JMX interface.

Performing a GET on the /admin/status endpoint will return a JSON object containing all the information available about the server and its databases.

$ curl -u admin:admin "http://localhost:5820/admin/status/"

The stardog-admin server status CLI command will print a subset of this information to the console.

The endpoint /{yourDatabaseName}/status endpoint will return the monitoring information about that database’s status.

$ curl -u admin:admin "http://localhost:5820/{yourDatabaseName}/status/"

Prometheus

Monitoring information is also available for Prometheus via the /admin/status/prometheus endpoint, allowing Prometheus servers to scrape Stardog directly. This endpoint requires authentication.

In some environments, it can be advantageous to scrape metrics from an unauthenticated endpoint. The /admin/status/prometheus/internal endpoint allows a service from a private address space to scrape the metrics without Stardog authentication. By default, it is restricted to connections from 127.0.0.1/32. The allowed CIDR can be changed via stardog.properties with the prometheus.allowCIDR option.

$ curl -u admin:admin "http://localhost:5820/admin/status/prometheus/"

READ permission on dbms-admin:metrics is required to access both the /admin/status and /admin/status/prometheus endpoints in order to consume server metrics information. This is to prevent giving access to sensitive information in server metrics to any unauthenticated users. If the user does not have this permission, admin/status will only return a few metrics, such as the version of the server.

Prometheus Metrics Filters

The Prometheus API offers ways to limit the number of metrics returned. This may be achieved by supplying a regex either in the stardog.properties or directly via a query parameter. The API endpoint supports the query parameters include and exclude, which can be configured in the scrape_config section of the Prometheus configuration file. stardog.properties has corresponding config options named metrics.prometheus.include and metrics.prometheus.exclude. When both parameters are supplied, the exclusion will win. Query parameters take precedence over the configuration option.

Filter examples

Get all metrics starting with dbms_ and com_ with the regex ^(dbms|com)_.*:

$ curl -u admin:admin "http://localhost:5820/admin/status/prometheus?include=dbms_.%2A&include=%5E%28dbms_%7Ccom%29_.%2A"

Or add the following line to stardog.properties:

metrics.prometheus.include=^(dbms|com)_.*

Exclude database specific metrics and kga metrics with the regex ^(databases|kga)_.*:

$ curl -u admin:admin "http://localhost:5820/admin/status/prometheus?exclude=%5E%28databases%7Ckga%29_.%2A"

Or add the following line to stardog.properties:

metrics.prometheus.exclude=^(databases|kga)_.*

JMX Monitoring

By default, JMX monitoring is not enabled. You can enable it by setting metrics.reporter=jmx in the stardog.properties file. Then, you can use a tool like VisualVM or JConsole to attach to the process running the JVM, or connect directly to the JMX server.

If you want to connect to the JMX server remotely, you need to set metrics.jmx.remote.access=true in stardog.properties. Stardog will bind an RMI server for remote access on port 5833. If you want to change the port Stardog binds the remote server to, you can set the property metrics.jmx.port in stardog.properties.

Disabling Monitoring

If you wish to disable monitoring completely, set metrics.enabled to false in stardog.properties.

Knowledge Graph Metrics

These metrics focus on measuring a few key aspects of the knowledge graph.

Instance Wide Metrics

These metrics are written out for the entire Stardog instance and capture triple counts for each virtual graph and the total triple count. For the purposes of this document, virtual graph IRIs such as virtual://name consist of UriScheme (virtual) and VirtualGraphName (name).

Block Cache Metrics

These are metrics for the 3 global block caches: Data, Dictionary, and TXN. Each Block cache is global and shared by all databases simultaneously. There are three distinct caches with distinct purposes, but each has their own set of metrics.

1. The Data Cache stores data from indices.
2. The Dictionary Cache stores entries from dictionary mappings.
3. The Txn Cache stores transaction entries. This speeds up access to transaction metadata.

For convenience, the metrics system rolls up statistics for all three Block Caches into a “total” metrics. Thus, we have four prefix forms:

1. dbms.memory.blockcache.data
2. dbms.memory.blockcache.dictionary
3. dbms.memory.blockcache.txn
4. dbms.memory.blockcache.total

Since all metrics have the same definition for each time, we will only list the metrics once, using the form dbms.memory.blockcache.CACHE.<metric>, where CACHE can be data,dictionary,txn, or total.

Each Block cache has three internal components:

1. The “data” component is where actual bytes from files are stored.
2. The “index” component is where file indices are stored.
3. The “filter” component is where bloom filters are stored.

If storage.cacheIndexBlocks, storage.cacheDictionaryIndexBlocks, and storage.cacheTxnIndexBlocks are false, then the “index” and “filter” sections will not be populated for that block cache and will instead have all zeros.

Database Metrics

These metrics are written out once per database and are written in the form of databases.*. In JMX, these metrics are collected in a separate folder. For reference purposes, metrics in this table will use YourDb as the database name.

Per Index Metrics

These metrics are written out once per index within a database (i.e. SPOC, C, CPO, etc.). They are of the form databases.*. For the purposes of this document, we will use YourDb as the database name, and INAME as the index name.

There are 8 different kinds of indices in Stardog:

Table of Index Metrics:

HTTP Server Metrics

These metrics are used to monitor the HTTP subsystem. They are general to the process itself (since there is only one HTTP layer per process).

Memory Usage Metrics

Memory Management Metrics

The Memory Management subsystem is responsible for efficiently managing Stardog’s internal memory usage, especially during query answering. Memory is broken down into a set of reusable memory “blocks”.

Java Memory Metrics

These are metrics about (or related to) the JVM’s memory usage. They are usually accessible through other JVM tools (like JMX) but are provided as explicit metrics for end-user convenience.

Thread dumps for the server can be retrieved with the metric jvm.threads, but only if the threads parameter is set to true in the HTTP request. Using the --threads option in the stardog-admin server metrics CLI command will achieve this. This capability is useful as an alternative to jstack, as it does not require login access to the server.

When metrics.jvm.enabled is set to true in stardog.properties, Stardog additionally reports a set of JVM metrics. They have the following prefixes:

• jvm.gc.* for GC related metrics
• jvm.memory.* for JVM heap related metrics
• jvm.memory.buffers.* for JVM metrics related to use of memory buffers

Process Memory Metrics

These are metrics about the process itself, ignoring the JVM. These are almost always accessible through other means (such as ps on Linux systems) but are provided as metrics within Stardog both for end-user convenience and for automatic management (such as warning when memory usage exceeds a threshold).

Process Metrics

Process metrics are metrics that are unique to the Stardog process currently running and its environment. They contain information about the process itself without referencing any specific database.

---