Logging
This page discusses the different types of logging available Stardog.
Page Contents
stardog.log
Stardog’s server log is the stardog.log
which can be found inside your $STARDOG_HOME
directory.
Unlike the other logs mentioned in this section (access, audit, and slow query), the stardog.log
can be configured via log4j2. A default log4j2.xml
can be found in the <stardog-installation-directory>/server/dbms
directory. You can copy this file to your $STARDOG_HOME
directory to make any modifications you’d like (e.g. log rotation).
$ cp <stardog/installation-directory>/server/dbms/log4j2.xml $STARDOG_HOME
Access and Audit Logging
Audit logging is a superset of the events in access logging. Access logging covers the most often required logging events; you should consider enabling audit logging if you really need to log every server event. Logging generally doesn’t have much impact on performance; but the safest way to insure that impact is negligible is to log to a separate disk (or to a centralized logging server, etc.).
The important configuration choices are whether logs should be binary or plain text (both based on ProtocolBuffer message formats); the type of logging (audit or access); the logging location (which may be “off disk” or even “off machine”). Logging to a centralized logging facility requires a Java plugin that implements the Stardog Server logging interface; see Java Programming for more information; and the log rotation policy (file size or time).
We walk through how to setup both the Access and Audit log sections below and discuss their configuration in more detail.
An example stardog.properties
file containing configuration for access and audit is provided in our stardog-examples Github repository.
Access Log Configuration
The access log logs all connection-based events, such as queries and transactional changes, and authentication events. Connection boundary events, that is open and closing of the actual connection, are not logged.
The access log is disabled by default and should be enabled explicitly in the stardog.properties
configuration file by adding:
logging.access.enabled = true
The type of the access log is required. There are two allowed values: text
and binary
, both based on Protobuf.
- In
text
logs, entries are serialized as key-value pairs in plain text and log entries are delimited by the ASCII Start of Text and ASCII End of Text bytes. - In
binary
logs, entries are written as size delimited protobuf binary messages.
We can add the type of access log by adding the following to the stardog.properties
:
logging.access.type = text
Here we’ve chosed to set the type of log as text
as opposed to binary
.
Optionally you can specify a custom file name for the access log. By default, the access log is saved as a file named access.log
inside the $STARDOG_HOME
directory. If desired, a different file name can be provided as follows. An absolute path can be provided to save the file in a different directory.
logging.access.file = my_access.log
It is possible to define a file rotation strategy so that log entries will be split into multiple files. One possibility is to define a limit on file size and start a new file when the size limit is reached. The file that exceeds the limit will be renamed with a unique suffix and all the subsequent entries will be added to the newly created file. The size limit is defined in bytes as follows. See the audit log example for defining an alternative way of rotating files based on fixed-time intervals.
logging.access.rotation.type = size
logging.access.rotation.limit = 1000000
The server configuration properties for access logging are documented in completion in the server configuration
Audit Log Configuration
As noted earlier, the audit log logs all server events and is a superset of the access log. Access logging covers the most often required logging events; you should only consider enabling audit logging if you really need to log every server event. Logging generally doesn’t have much impact on performance; but the safest way to insure that impact is negligible is to log to a separate disk (or to a centralized logging server, etc.).
The audit log is disabled by default and should be enabled explicitly in the stardog.properties
configuration file by adding:
logging.audit.enabled = true
Similar to the access log, the type of the audit log is required. There are two allowed values: text
and binary
, both based on Protobuf.
- In
text
logs, entries are serialized as key-value pairs in plain text and log entries are delimited by the ASCII Start of Text and ASCII End of Text bytes. - In
binary
logs, entries are written as size delimited protobuf binary messages.
We can add the type of audit log by adding the following to the stardog.properties
:
logging.audit.type = binary
By default, the audit log is saved as a file named audit.log
inside the $STARDOG_HOME
directory. Optionally, a different file name can be provided as follows. An absolute path can be provided to save the file in a different directory.
logging.audit.file = my_audit.log
Audit logs can be rotated similar to access logs. Size limit can be used as described above. Alternately, log files can be rotated at fixed time intervals. The interval value should be a positive integer followed by either letter ‘d’ (for days) or letter ‘h’ (for hours). Below is an example to get daily logs:
logging.audit.rotation.type = time
logging.audit.rotation.interval = 1d
Rotated log files can be compressed with gzip if the following option is set:
logging.audit.rotation.compress = true
Compression happens asynchronously. Once the rotation condition is triggered, the audit file will be rotated in an uncompressed format and will be compressed in a background thread. Once the compression completes, the uncompressed rotated file will be deleted. Compressed rotated file will use the .gz
file extension.
The number of rotated files can be limited as well to avoid running out of disk space by accumulating rotated files. The following setting, for example, will limit the number of rotated files to 5:
logging.audit.rotation.file.count = 5
Note that the actual audit log is not included in this count. Therefore, if the limit is set to 5, there will be at most 6 log files at any time. When the number of rotated files exceeds this limit, the latest rotated file will be deleted. If this limit is reduced in the configuration at any point, the server needs to be restarted and the change will take affect the next time the log file is rotated. For example, suppose the file count was set to 10 and there are 10 rotated log files in $STARDOG_HOME
directory. The file count is reduced to 5 and the server is restarted. The next time the log file is rotated, the 6 oldest log files will be deleted and there will be 5 rotated log files including the newly rotated file.
The server configuration properties for audit logging are documented in completion in the server configuration
Analyzing the Access and Audit Log
The contents of the access and audit log can be printed using the log print
CLI utility. The utility has multiple options (discussed below) to filter and customize the log contents that are outputted and their format.
Event Types
Stardog logs multiple different types of events. The events can be provided to the log print
utility if you wish to filter for certain events such as a ‘ChangePassword’ event. These events are listed and described in the table below:
Event | Description |
---|---|
AddRole | Adds a role to the system. |
AddUser | Adds a user to the system. |
AddUserRole | Assigns a user to a role. |
Authentication | Authenicates a user. |
Backup | Takes a backup or one database or the entire server. The ‘Database’ field for a server backup will have a value of ‘AllDatabases’. |
ChangePassword | Changes a user’s password. |
ChangeUserEnabled | Enables or disables a user. |
ConnectionClose | Closes a connection to a database. |
ConnectionOpen | Opens a connection to the database. |
CreateDatabase | Creates a database. |
DataAdd | Adds data to a database. |
DataRemove | Removes data from a database. |
DataRemoveAll | Removes all data from a database |
DataRemoveContext | Removes data from a specific named graph (context) of the database. |
DeleteRole | Deletes a role. |
DeleteUser | Deletes a user. |
DeleteUserRole | Unassigns a role from a user. |
DropDatabase | Drops a database. |
GetEffectiveUserPermissions | Gets all permissions assigned to a given user as well as those granted by assigned roles. |
GetEnabledStatus | Checks whether the specified user is enabled. |
GetOption | Checks a database’s configuration options (metadata). |
GetRolePermissions | Gets the permissions associated with the specified role. |
GetSuperStatus | Checks whether the specified user is a super user. |
GetUserPermissions | Gets the current user permissions (explicit only). |
GetUserRoles | Retrieves all roles of the specified user. |
GetUsersWithRole | Gets the list of all users with the given role. |
ListRoles | Gets the names of all roles in the system. |
ListUsers | Gets the names of all users in the system that the authenticated user is allowed to see. |
Migrate | Migrates the contents of a Stardog home directory from an old version to the current version. |
Offline | Takes the database offline. |
Online | Takes the database online. |
Optimize | Optimizes a database. |
Query | Query execution started. |
QueryEnd | Query execution terminated. |
Repair | Repairs a database if it is corrupted. |
Restore | Restores a database backup. |
RolePermissionGrant | Grants a permission to a role. |
RolePermissionRevoke | Deletes a permission from a role. |
ServerPropertyUpdated | Updateable server property is updated. |
SetEnabledStatus | Enables/disables the user. |
SetOption | Sets a database’s configuration option (metadata). |
SetUserRoles | Changes the user roles. |
Shutdown | Shuts down Stardog. |
Startup | Starts up Stardog. |
StoredQueryAdded | Adds a stored query. |
StoredQueryRemoved | Removes a stored query. |
StoredQueryUpdated | Updates a stored query. |
TxBegin | Begins a transaction. |
TxCommit | End a transaction. |
TxRollback | Rolls back a transaction. |
UserPermissionGrant | Grants a permission to a user. |
UserPermissionRevoke | Revokes a permission from a user. |
Verify | Verifies a database’s integrity. |
WildcardPermissionRevoke | Revokes multiple permissions simultaneously. |
Print all events
To print all events in the audit/access log provide the absolute path to the audit/access log as the last argument to the log print
command:
stardog-admin log print $STARDOG_HOME/audit.log
If the access or audit log is written as text (e.g.
logging.audit.type=text
instardog.properties
) add the--text
flag to the command:stardog-admin log print --text $STARDOG_HOME/audit.log
If you do not specify the
--text
flag when it is needed, you may see the following error message in your console:Log header is missing or malformed The detailed stack trace for the error is: java.io.IOException: Log header is missing or malformed at com.complexible.stardog.logging.protobuf.ProtobufLogReader.read(ProtobufLogReader.java:46) at com.complexible.stardog.logging.cli.LogPrint.call(LogPrint.java:132) at com.complexible.stardog.logging.cli.LogPrint.call(LogPrint.java:59) at com.complexible.stardog.cli.CLIBase.execute(CLIBase.java:56) at com.complexible.stardog.cli.admin.CLI.main(CLI.java:218)
+--------------------------------+----------------------+----------------------+----------------------+--------------------------+--------------------------------------------------------------------------+------------+ | Time | Event | Database | User | Ip | Value | Flags | +--------------------------------+----------------------+----------------------+----------------------+--------------------------+--------------------------------------------------------------------------+------------+ | 2021-05-19T17:21:42.244-04:00 | Startup | | root | local:ephemeral | | | | 2021-05-19T17:21:42.247-04:00 | ListUsers | | root | local:ephemeral | | | | 2021-05-19T17:23:08.373-04:00 | Authentication | | admin | /127.0.0.1:58730 | | | | 2021-05-19T17:23:08.417-04:00 | Authentication | | admin | /127.0.0.1:58730 | | | | 2021-05-19T17:23:08.435-04:00 | GetUserPermissions | | admin | /127.0.0.1:58730 | testUser | | | 2021-05-19T17:23:08.495-04:00 | Authentication | | admin | /127.0.0.1:58730 | | | | 2021-05-19T17:23:08.496-04:00 | GetUserRoles | | admin | /127.0.0.1:58730 | testUser | | | 2021-05-19T17:23:08.501-04:00 | Authentication | | admin | /127.0.0.1:58730 | | | | 2021-05-19T17:23:08.502-04:00 | GetRolePermissions | | admin | /127.0.0.1:58730 | someRole | | | 2021-05-19T17:23:24.638-04:00 | Authentication | | admin | /127.0.0.1:58732 | | | | 2021-05-19T17:23:24.652-04:00 | Authentication | | admin | /127.0.0.1:58732 | | | | 2021-05-19T17:23:24.692-04:00 | Shutdown | | admin | /127.0.0.1:58732 | | | +--------------------------------+----------------------+----------------------+----------------------+--------------------------+--------------------------------------------------------------------------+------------+
Print all events with specified fields
One can choose to only include certain fields in the output. Allowed values are:
- Time
- Event
- Database
- User
- IP
- Value
- Flags
- Query
Fields follow the --field
option in the log print
command:
stardog-admin log print --field time user database query -- $STARDOG_HOME/access.log
+--------------------------------+----------------------+----------------------+----------------------------------------------------------------------------------+ | Time | User | Database | Query | +--------------------------------+----------------------+----------------------+----------------------------------------------------------------------------------+ | 2021-05-20T10:36:41.717-04:00 | admin | | | | 2021-05-20T10:36:41.723-04:00 | admin | testDatabase | COPY <virtual://employees> to <urn:g1> | | 2021-05-20T10:36:41.723-04:00 | admin | testDatabase | | | 2021-05-20T10:36:41.758-04:00 | admin | testDatabase | Virtual graph import of <virtual://employees> | | 2021-05-20T10:45:24.963-04:00 | admin | testDatabase | | +--------------------------------+----------------------+----------------------+----------------------------------------------------------------------------------+
Filter events
It’s possible to provide filter expressions to the log print
command to hide certain events in the output.
- A filter expression is in the form
field operator value
where field is one of the fields defined in--field
option, operator is one of=
,<
,<=
,>
,>=
, or~
, and value should be a valid value for the field.-
The operator
~
represents regular expression matching where value should be a valid Java regular expression and should match the entire value of the given field. -
Valid values for the for the Event field can be found in the event types table above.
-
Filter expression for Time field should use xsd:date or xsd:dateTime values.
-
Filter expressions follow the --filter
option in the log print
command:
stardog-admin log print --filter event=query 'time>=2021-05-20' -- $STARDOG_HOME/audit.log
+--------------------------------+----------------------+----------------------+----------------------+--------------------------+----------------------------------------------------------------------------------+------------+ | Time | Event | Database | User | Ip | Value | Flags | +--------------------------------+----------------------+----------------------+----------------------+--------------------------+----------------------------------------------------------------------------------+------------+ | 2021-05-20T15:20:50.595-04:00 | Query | testDb | admin | /0:0:0:0:0:0:0:1:62420 | Query(bfa1137a-42c2-458f-a83e-561fb93b480d, 24, DB=testDb, User=admin, | | | | | | | | Reasoning=false, Started=2021-05-20 03:20:50 PM, Elapsed=00:00:00.-01, | | | | | | | | Timeout=00:05:00.000): | | | | | | | | select * { ?s ?p ?o } | | +--------------------------------+----------------------+----------------------+----------------------+--------------------------+----------------------------------------------------------------------------------+------------+
Output formats
The audit/access log can be printed in a tabular format (default), JSON, or a delimited format like CSV.
JSON Output
To format the output of the log print
command as JSON, pass the --json
flag in.
stardog-admin log print --json --filter event=query 'time>=2021-05-20' -- $STARDOG_HOME/audit.log
{"DATABASE":"testDb","IP":"/0:0:0:0:0:0:0:1:60517","TIME":"2021-05-20T10:36:41.723-04:00","FLAGS":" ","VALUE":"Query(bfa1137a-42c2-458f-a83e-561fb93b480d, 14, DB=testDb, User=admin, Reasoning=false, Started=2021-05-20 10:36:41 AM, Elapsed=00:00:00.-01, Timeout=00:05:00.000):\nCOPY <virtual://employees> to <urn:g1>","USER":"admin","EVENT":"Query"} {"DATABASE":"testDb","IP":"/0:0:0:0:0:0:0:1:60517","TIME":"2021-05-20T10:36:41.758-04:00","FLAGS":" ","VALUE":"Query(bfa1137a-42c2-458f-a83e-561fb93b480d, 15, DB=testDb, User=admin, Reasoning=false, Started=2021-05-20 10:36:41 AM, Elapsed=00:00:00.-01, Timeout=00:00:00.-01):\nVirtual graph import of <virtual://employees>","USER":"admin","EVENT":"Query"}
Delimited Output
One can use a delimeter to separate fields in the output. Specify the delimiter of your choice (e.g. ,
for a CSV) after the -d
option.
stardog-admin log print -d , --filter event=query 'time>=2021-05-20' -- $STARDOG_HOME/audit.log
Time,Event,Database,User,Ip,Value,Flags 2021-05-20T10:36:41.723-04:00,Query,testDb,admin,/0:0:0:0:0:0:0:1:60517,"Query(bfa1137a-42c2-458f-a83e-561fb93b480d, 14, DB=testDb, User=admin, Reasoning=false, Started=2021-05-20 10:36:41 AM, Elapsed=00:00:00.-01, Timeout=00:05:00.000): COPY <virtual://employees> to <urn:g1>", 2021-05-20T10:36:41.758-04:00,Query,testDb,admin,/0:0:0:0:0:0:0:1:60517,"Query(bfa1137a-42c2-458f-a83e-561fb93b480d, 15, DB=testDb, User=admin, Reasoning=false, Started=2021-05-20 10:36:41 AM, Elapsed=00:00:00.-01, Timeout=00:00:00.-01): Virtual graph import of <virtual://employees>",
Slow Query Log
As the name suggests, the slow query logs all slow queries. A query is considered to be slow if its execution time exceeds the logging.slow_query.time
property defined below.
The slow log query should be enabled similar to the access and audit logs:
logging.slow_query.enabled = true
Slow query time is specified as follows. Any query whose execution time exceed this limit will be logged in the slow query log. The time value should be a positive integer followed by either letter ‘h’ (for hours), letter ‘m’ (for minutes), letter ‘s’ (for seconds), or letters ‘ms’ (for milliseconds). Examples intervals are 1h
for 1 hour, 5m
for 5 minutes, 90s
for 90 seconds, and 500ms
for 500 milliseconds. If no time is specified in the configuration, the default value of 1 minute will be used.
logging.slow_query.time = 10s
Unlike access or audit logs, slow query log has a default type for formatting the slow query events which is intended to be human-readable. If no log type is specified, this default type will be used. This default format is not suitable for machine-processing and meant to be only for developers to read. For a machine-processable format, the type of the slow query log can be set similar to the access and audit logs (binary
).
logging.slow_query.type = text
The server configuration properties for slow query logging are documented in completion in the server configuration