Link Search Menu Expand Document
Start for Free

Logging

This page discusses the different types of logging available Stardog.

Page Contents
  1. stardog.log
  2. Access and Audit Logging
    1. Access Log Configuration
    2. Audit Log Configuration
    3. Analyzing the Access and Audit Log
      1. Event Types
      2. Print all events
      3. Print all events with specified fields
      4. Filter events
      5. Output formats
        1. JSON Output
        2. Delimited Output
  3. Slow Query Log

stardog.log

Stardog’s server log is the stardog.log which can be found inside your $STARDOG_HOME directory.

Unlike the other logs mentioned in this section (access, audit, and slow query), the stardog.log can be configured via log4j2. A default log4j2.xml can be found in the <stardog-installation-directory>/server/dbms directory. You can copy this file to your $STARDOG_HOME directory to make any modifications you’d like (e.g. log rotation).

$ cp <stardog/installation-directory>/server/dbms/log4j2.xml $STARDOG_HOME

Access and Audit Logging

Audit logging is a superset of the events in access logging. Access logging covers the most often required logging events; you should consider enabling audit logging if you really need to log every server event. Logging generally doesn’t have much impact on performance; but the safest way to insure that impact is negligible is to log to a separate disk (or to a centralized logging server, etc.).

The important configuration choices are whether logs should be binary or plain text (both based on ProtocolBuffer message formats); the type of logging (audit or access); the logging location (which may be “off disk” or even “off machine”). Logging to a centralized logging facility requires a Java plugin that implements the Stardog Server logging interface; see Java Programming for more information; and the log rotation policy (file size or time).

We walk through how to setup both the Access and Audit log sections below and discuss their configuration in more detail.

An example stardog.properties file containing configuration for access and audit is provided in our stardog-examples Github repository.

Access Log Configuration

The access log logs all connection-based events, such as queries and transactional changes, and authentication events. Connection boundary events, that is open and closing of the actual connection, are not logged.

The access log is disabled by default and should be enabled explicitly in the stardog.properties configuration file by adding:

logging.access.enabled = true

The type of the access log is required. There are two allowed values: text and binary, both based on Protobuf.

  • In text logs, entries are serialized as key-value pairs in plain text and log entries are delimited by the ASCII Start of Text and ASCII End of Text bytes.
  • In binary logs, entries are written as size delimited protobuf binary messages.

We can add the type of access log by adding the following to the stardog.properties:

logging.access.type = text

Here we’ve chosed to set the type of log as text as opposed to binary.

Optionally you can specify a custom file name for the access log. By default, the access log is saved as a file named access.log inside the $STARDOG_HOME directory. If desired, a different file name can be provided as follows. An absolute path can be provided to save the file in a different directory.

logging.access.file = my_access.log

It is possible to define a file rotation strategy so that log entries will be split into multiple files. One possibility is to define a limit on file size and start a new file when the size limit is reached. The file that exceeds the limit will be renamed with a unique suffix and all the subsequent entries will be added to the newly created file. The size limit is defined in bytes as follows. See the audit log example for defining an alternative way of rotating files based on fixed-time intervals.

logging.access.rotation.type = size
logging.access.rotation.limit = 1000000

The server configuration properties for access logging are documented in completion in the server configuration

Audit Log Configuration

As noted earlier, the audit log logs all server events and is a superset of the access log. Access logging covers the most often required logging events; you should only consider enabling audit logging if you really need to log every server event. Logging generally doesn’t have much impact on performance; but the safest way to insure that impact is negligible is to log to a separate disk (or to a centralized logging server, etc.).

The audit log is disabled by default and should be enabled explicitly in the stardog.properties configuration file by adding:

logging.audit.enabled = true

Similar to the access log, the type of the audit log is required. There are two allowed values: text and binary, both based on Protobuf.

  • In text logs, entries are serialized as key-value pairs in plain text and log entries are delimited by the ASCII Start of Text and ASCII End of Text bytes.
  • In binary logs, entries are written as size delimited protobuf binary messages.

We can add the type of audit log by adding the following to the stardog.properties:

logging.audit.type = binary

By default, the audit log is saved as a file named audit.log inside the $STARDOG_HOME directory. Optionally, a different file name can be provided as follows. An absolute path can be provided to save the file in a different directory.

logging.audit.file = my_audit.log

Audit logs can be rotated similar to access logs. Size limit can be used as described above. Alternately, log files can be rotated at fixed time intervals. The interval value should be a positive integer followed by either letter ‘d’ (for days) or letter ‘h’ (for hours). Below is an example to get daily logs:

logging.audit.rotation.type = time
logging.audit.rotation.interval = 1d

Rotated log files can be compressed with gzip if the following option is set:

logging.audit.rotation.compress = true

Compression happens asynchronously. Once the rotation condition is triggered, the audit file will be rotated in an uncompressed format and will be compressed in a background thread. Once the compression completes, the uncompressed rotated file will be deleted. Compressed rotated file will use the .gz file extension.

The number of rotated files can be limited as well to avoid running out of disk space by accumulating rotated files. The following setting, for example, will limit the number of rotated files to 5:

logging.audit.rotation.file.count = 5

Note that the actual audit log is not included in this count. Therefore, if the limit is set to 5, there will be at most 6 log files at any time. When the number of rotated files exceeds this limit, the latest rotated file will be deleted. If this limit is reduced in the configuration at any point, the server needs to be restarted and the change will take affect the next time the log file is rotated. For example, suppose the file count was set to 10 and there are 10 rotated log files in $STARDOG_HOME directory. The file count is reduced to 5 and the server is restarted. The next time the log file is rotated, the 6 oldest log files will be deleted and there will be 5 rotated log files including the newly rotated file.

The server configuration properties for audit logging are documented in completion in the server configuration

Analyzing the Access and Audit Log

The contents of the access and audit log can be printed using the log print CLI utility. The utility has multiple options (discussed below) to filter and customize the log contents that are outputted and their format.

Event Types

Stardog logs multiple different types of events. The events can be provided to the log print utility if you wish to filter for certain events such as a ‘ChangePassword’ event. These events are listed and described in the table below:

Event Description
AddRole Adds a role to the system.
AddUser Adds a user to the system.
AddUserRole Assigns a user to a role.
Authentication Authenicates a user.
Backup Takes a backup or one database or the entire server. The ‘Database’ field for a server backup will have a value of ‘AllDatabases’.
ChangePassword Changes a user’s password.
ChangeUserEnabled Enables or disables a user.
ConnectionClose Closes a connection to a database.
ConnectionOpen Opens a connection to the database.
CreateDatabase Creates a database.
DataAdd Adds data to a database.
DataRemove Removes data from a database.
DataRemoveAll Removes all data from a database
DataRemoveContext Removes data from a specific named graph (context) of the database.
DeleteRole Deletes a role.
DeleteUser Deletes a user.
DeleteUserRole Unassigns a role from a user.
DropDatabase Drops a database.
GetEffectiveUserPermissions Gets all permissions assigned to a given user as well as those granted by assigned roles.
GetEnabledStatus Checks whether the specified user is enabled.
GetOption Checks a database’s configuration options (metadata).
GetRolePermissions Gets the permissions associated with the specified role.
GetSuperStatus Checks whether the specified user is a super user.
GetUserPermissions Gets the current user permissions (explicit only).
GetUserRoles Retrieves all roles of the specified user.
GetUsersWithRole Gets the list of all users with the given role.
ListRoles Gets the names of all roles in the system.
ListUsers Gets the names of all users in the system that the authenticated user is allowed to see.
Migrate Migrates the contents of a Stardog home directory from an old version to the current version.
Offline Takes the database offline.
Online Takes the database online.
Optimize Optimizes a database.
Query Query execution started.
QueryEnd Query execution terminated.
Repair Repairs a database if it is corrupted.
Restore Restores a database backup.
RolePermissionGrant Grants a permission to a role.
RolePermissionRevoke Deletes a permission from a role.
ServerPropertyUpdated Updateable server property is updated.
SetEnabledStatus Enables/disables the user.
SetOption Sets a database’s configuration option (metadata).
SetUserRoles Changes the user roles.
Shutdown Shuts down Stardog.
Startup Starts up Stardog.
StoredQueryAdded Adds a stored query.
StoredQueryRemoved Removes a stored query.
StoredQueryUpdated Updates a stored query.
TxBegin Begins a transaction.
TxCommit End a transaction.
TxRollback Rolls back a transaction.
UserPermissionGrant Grants a permission to a user.
UserPermissionRevoke Revokes a permission from a user.
Verify Verifies a database’s integrity.
WildcardPermissionRevoke Revokes multiple permissions simultaneously.

To print all events in the audit/access log provide the absolute path to the audit/access log as the last argument to the log print command:

  • stardog-admin log print $STARDOG_HOME/audit.log
    

    If the access or audit log is written as text (e.g. logging.audit.type=text in stardog.properties) add the --text flag to the command:

    stardog-admin log print --text $STARDOG_HOME/audit.log
    

    If you do not specify the --text flag when it is needed, you may see the following error message in your console:

    Log header is missing or malformed
    The detailed stack trace for the error is:
    java.io.IOException: Log header is missing or malformed
    	at com.complexible.stardog.logging.protobuf.ProtobufLogReader.read(ProtobufLogReader.java:46)
    	at com.complexible.stardog.logging.cli.LogPrint.call(LogPrint.java:132)
    	at com.complexible.stardog.logging.cli.LogPrint.call(LogPrint.java:59)
    	at com.complexible.stardog.cli.CLIBase.execute(CLIBase.java:56)
    	at com.complexible.stardog.cli.admin.CLI.main(CLI.java:218)
    
  • +--------------------------------+----------------------+----------------------+----------------------+--------------------------+--------------------------------------------------------------------------+------------+
    |              Time              |        Event         |       Database       |         User         |            Ip            |                                  Value                                   |   Flags    |
    +--------------------------------+----------------------+----------------------+----------------------+--------------------------+--------------------------------------------------------------------------+------------+
    | 2021-05-19T17:21:42.244-04:00  | Startup              |                      | root                 | local:ephemeral          |                                                                          |            |
    | 2021-05-19T17:21:42.247-04:00  | ListUsers            |                      | root                 | local:ephemeral          |                                                                          |            |
    | 2021-05-19T17:23:08.373-04:00  | Authentication       |                      | admin                | /127.0.0.1:58730         |                                                                          |            |
    | 2021-05-19T17:23:08.417-04:00  | Authentication       |                      | admin                | /127.0.0.1:58730         |                                                                          |            |
    | 2021-05-19T17:23:08.435-04:00  | GetUserPermissions   |                      | admin                | /127.0.0.1:58730         | testUser                                                                 |            |
    | 2021-05-19T17:23:08.495-04:00  | Authentication       |                      | admin                | /127.0.0.1:58730         |                                                                          |            |
    | 2021-05-19T17:23:08.496-04:00  | GetUserRoles         |                      | admin                | /127.0.0.1:58730         | testUser                                                                 |            |
    | 2021-05-19T17:23:08.501-04:00  | Authentication       |                      | admin                | /127.0.0.1:58730         |                                                                          |            |
    | 2021-05-19T17:23:08.502-04:00  | GetRolePermissions   |                      | admin                | /127.0.0.1:58730         | someRole                                                                 |            |
    | 2021-05-19T17:23:24.638-04:00  | Authentication       |                      | admin                | /127.0.0.1:58732         |                                                                          |            |
    | 2021-05-19T17:23:24.652-04:00  | Authentication       |                      | admin                | /127.0.0.1:58732         |                                                                          |            |
    | 2021-05-19T17:23:24.692-04:00  | Shutdown             |                      | admin                | /127.0.0.1:58732         |                                                                          |            |
    +--------------------------------+----------------------+----------------------+----------------------+--------------------------+--------------------------------------------------------------------------+------------+
    

One can choose to only include certain fields in the output. Allowed values are:

  • Time
  • Event
  • Database
  • User
  • IP
  • Value
  • Flags
  • Query

Fields follow the --field option in the log print command:

  • stardog-admin log print --field time user database query -- $STARDOG_HOME/access.log
    
  • +--------------------------------+----------------------+----------------------+----------------------------------------------------------------------------------+
    |              Time              |         User         |       Database       |                                      Query                                       |
    +--------------------------------+----------------------+----------------------+----------------------------------------------------------------------------------+
    | 2021-05-20T10:36:41.717-04:00  | admin                |                      |                                                                                  |
    | 2021-05-20T10:36:41.723-04:00  | admin                | testDatabase         | COPY <virtual://employees> to <urn:g1>                                           |
    | 2021-05-20T10:36:41.723-04:00  | admin                | testDatabase         |                                                                                  |
    | 2021-05-20T10:36:41.758-04:00  | admin                | testDatabase         | Virtual graph import of <virtual://employees>                                    |
    | 2021-05-20T10:45:24.963-04:00  | admin                | testDatabase         |                                                                                  |
    +--------------------------------+----------------------+----------------------+----------------------------------------------------------------------------------+
    

Filter events

It’s possible to provide filter expressions to the log print command to hide certain events in the output.

  • A filter expression is in the form field operator value where field is one of the fields defined in --field option, operator is one of =, <, <=, >, >=, or ~, and value should be a valid value for the field.
    • The operator ~ represents regular expression matching where value should be a valid Java regular expression and should match the entire value of the given field.

    • Valid values for the for the Event field can be found in the event types table above.

    • Filter expression for Time field should use xsd:date or xsd:dateTime values.

Filter expressions follow the --filter option in the log print command:

  • stardog-admin log print --filter event=query 'time>=2021-05-20' -- $STARDOG_HOME/audit.log
    
  • +--------------------------------+----------------------+----------------------+----------------------+--------------------------+----------------------------------------------------------------------------------+------------+
    |              Time              |        Event         |       Database       |         User         |            Ip            |                                      Value                                       |   Flags    |
    +--------------------------------+----------------------+----------------------+----------------------+--------------------------+----------------------------------------------------------------------------------+------------+
    | 2021-05-20T15:20:50.595-04:00  | Query                | testDb               | admin                | /0:0:0:0:0:0:0:1:62420   | Query(bfa1137a-42c2-458f-a83e-561fb93b480d, 24, DB=testDb, User=admin,           |            |
    |                                |                      |                      |                      |                          | Reasoning=false, Started=2021-05-20 03:20:50 PM, Elapsed=00:00:00.-01,           |            |
    |                                |                      |                      |                      |                          | Timeout=00:05:00.000):                                                           |            |
    |                                |                      |                      |                      |                          | select * { ?s ?p ?o }                                                            |            |
    +--------------------------------+----------------------+----------------------+----------------------+--------------------------+----------------------------------------------------------------------------------+------------+
    
    

Output formats

The audit/access log can be printed in a tabular format (default), JSON, or a delimited format like CSV.

JSON Output

To format the output of the log print command as JSON, pass the --json flag in.

  • stardog-admin log print --json --filter event=query 'time>=2021-05-20' -- $STARDOG_HOME/audit.log
    
  • {"DATABASE":"testDb","IP":"/0:0:0:0:0:0:0:1:60517","TIME":"2021-05-20T10:36:41.723-04:00","FLAGS":" ","VALUE":"Query(bfa1137a-42c2-458f-a83e-561fb93b480d, 14, DB=testDb, User=admin, Reasoning=false, Started=2021-05-20 10:36:41 AM, Elapsed=00:00:00.-01, Timeout=00:05:00.000):\nCOPY <virtual://employees> to <urn:g1>","USER":"admin","EVENT":"Query"}
    {"DATABASE":"testDb","IP":"/0:0:0:0:0:0:0:1:60517","TIME":"2021-05-20T10:36:41.758-04:00","FLAGS":" ","VALUE":"Query(bfa1137a-42c2-458f-a83e-561fb93b480d, 15, DB=testDb, User=admin, Reasoning=false, Started=2021-05-20 10:36:41 AM, Elapsed=00:00:00.-01, Timeout=00:00:00.-01):\nVirtual graph import of <virtual://employees>","USER":"admin","EVENT":"Query"}
    
Delimited Output

One can use a delimeter to separate fields in the output. Specify the delimiter of your choice (e.g. , for a CSV) after the -d option.

  • stardog-admin log print -d , --filter event=query 'time>=2021-05-20' -- $STARDOG_HOME/audit.log
    
  • Time,Event,Database,User,Ip,Value,Flags
    2021-05-20T10:36:41.723-04:00,Query,testDb,admin,/0:0:0:0:0:0:0:1:60517,"Query(bfa1137a-42c2-458f-a83e-561fb93b480d, 14, DB=testDb, User=admin, Reasoning=false, Started=2021-05-20 10:36:41 AM, Elapsed=00:00:00.-01, Timeout=00:05:00.000):
    COPY <virtual://employees> to <urn:g1>", 
    2021-05-20T10:36:41.758-04:00,Query,testDb,admin,/0:0:0:0:0:0:0:1:60517,"Query(bfa1137a-42c2-458f-a83e-561fb93b480d, 15, DB=testDb, User=admin, Reasoning=false, Started=2021-05-20 10:36:41 AM, Elapsed=00:00:00.-01, Timeout=00:00:00.-01):
    Virtual graph import of <virtual://employees>",
    

Slow Query Log

As the name suggests, the slow query logs all slow queries. A query is considered to be slow if its execution time exceeds the logging.slow_query.time property defined below.

The slow log query should be enabled similar to the access and audit logs:

logging.slow_query.enabled = true

Slow query time is specified as follows. Any query whose execution time exceed this limit will be logged in the slow query log. The time value should be a positive integer followed by either letter ‘h’ (for hours), letter ‘m’ (for minutes), letter ‘s’ (for seconds), or letters ‘ms’ (for milliseconds). Examples intervals are 1h for 1 hour, 5m for 5 minutes, 90s for 90 seconds, and 500ms for 500 milliseconds. If no time is specified in the configuration, the default value of 1 minute will be used.

logging.slow_query.time = 10s

Unlike access or audit logs, slow query log has a default type for formatting the slow query events which is intended to be human-readable. If no log type is specified, this default type will be used. This default format is not suitable for machine-processing and meant to be only for developers to read. For a machine-processable format, the type of the slow query log can be set similar to the access and audit logs (binary).

logging.slow_query.type = text

The server configuration properties for slow query logging are documented in completion in the server configuration