This chapter discusses operating a Stardog Cluster. This page discusses basic operations like adding nodes to a cluster and taking backups. More advanced topics like managing a distributed cache, standby nodes, and troubleshooting the cluster are discussed in separate pages.
Stardog cluster stores the UUID of the last committed transaction for each database in ZooKeeper. When a new node is joining the cluster it will compare the local transaction ID of each database with the corresponding transaction ID stored in ZooKeeper. If there is a mismatch the node will synchronize the database contents from another node in the cluster. If there are no nodes in the cluster the new node cannot join the cluster and will shut itself down. For this reason, if you are starting a new cluster then you should make sure that the ZooKeeper state is cleared. If you are retaining an existing cluster then new nodes should be started when there is at least one node in the cluster.
If there are active transactions in the cluster joining node will wait for those transactions to finish and then synchronize its databases. More transactions may take place during synchronization and in that case the joining node will continue synchronization and retrieve the data from new transactions. Thus, it will take longer for a node to join the cluster if there are continuous transactions. Note that, the new node will not be available for requests until all the databases are synchronized. The proxy/load-balancer should perform a health check before forwarding the requests to a new node (as shown in the above configuration) so user requests will always be forwarded to available nodes.
Please see the open transactions section of the troubleshooting guide for further guidance when investigating if open transactions are the cause of a node being unable to join.
It is important to shutdown the cluster in a safe and controlled manner in order to prevent possible inconsistencies or corruption.
Before shutting down the cluster you should ensure that there are no transactions running, e.g.,
stardog-admin db status <db name> will show if there are any open transactions for a database. This step is not strictly required, however, it can reduce the risk of open transactions being killed, allowing the cluster to stop quickly and helping avoid non-coordinator nodes from having to re-sync when they attempt to rejoin the cluster next time it is started.
In order to shut down the cluster you only need to execute the following command once:
$ stardog-admin cluster stop
cluster stop request will cause all available nodes in the cluster to shutdown. If a node was expelled from the cluster due to a failure it would not receive this command and might need to be shutdown manually. In order to shutdown a single node in the cluster use the regular
server stop command and be sure to specify the server address:
$ stardog-admin --server http://localhost:5821 server stop
If you send the
server stop command to the load balancer then a random node selected by the load balancer will shutdown.
Monitor the logs and processes of all nodes to ensure the Stardog server process has stopped before restarting or terminating the underlying system.
If you are running Stardog in a container you should stop the cluster via
cluster stop or make sure to configure the shutdown timeout for your container runtime sufficiently high to avoid possible corruption. By default, container runtimes such as Docker or Kubernetes first send a SIGTERM before waiting a short amount of time (e.g. 10s-30s) and then sending a SIGKILL. In some instances this can result in corruption if Stardog has not shutdown cleanly before it is forcibly killed by the container runtime.
The process to upgrade Stardog Cluster is straightfoward; however, there are a few extra steps you should take to ensure the upgrade goes as quickly and smoothly as possible. Before you begin the upgrade, make sure to place the new Stardog binaries on all of the cluster nodes.
First, note which node is the coordinator since this is the first node that will be started as part of the upgrade.
stardog-admin cluster info will show the nodes in the cluster and which one is the coordinator.
Next, follow the process in the shutdown section and stop the cluster before continuing.
Once all nodes have stopped, backup the
STARDOG_HOME directories on all of the nodes.
With the new version of Stardog, bring the cluster up one node at a time, starting with the previous coordinator. As each node starts make sure that it is able to join the cluster cleanly before moving on to the next node.
Backing up the cluster is similar to single-node backups. However, there are a few points to be aware of. All nodes in the cluster will perform a backup unless S3 is the backup location, in which case only a single node will perform the backup to the S3 bucket.
If you are backing up to S3 then the
backup.location server property should be the same on all nodes in the cluster since any node may perform the backup.
You can also disable backup replication in the cluster by setting the following option:
backup.location is used to specify a backup directory mounted on the Stardog nodes then
backup.location can specify different directories on each node in the cluster, if required. In this case you should disable replicated backups and issue the backup command to each node individually.
--to is passed to the backup CLI commands, it will take precedence over either
backup.location specified in
stardog.properties and all nodes will perform a backup to the specified location.
Always remember to follow best practices for backups. Transfer any backups out of the infrastructure where Stardog runs (ideally to a separate data center, if possible) and validate them in a separate environment before assuming they are correct.
Similar to single-node Stardog, you can restore individual databases with
db restore. However, db restore from a file-based backup (not an s3 or gcp backup, for instance) can only be run on a single node cluster to ensure the cluster remains consistent. You can follow the same instructions for scaling the cluster down and up as you would for repairing a database in the cluster. Restoring from non-file-based backups will be replicated to all nodes and can safely be run in multi-node clusters.
Because Stardog Cluster uses ZooKeeper to ensure strong consistency between all of the nodes in the cluster, we recommend
server restore be done on only one node with a fresh deploy of ZooKeeper (i.e., clear ZooKeeper’s state once it is no longer in use).
The operational process is to:
- Shutdown Stardog on all nodes in the cluster
- Shutdown the ZooKeeper ensemble, if possible. If that’s not possible we recommend backing up ZooKeeper’s state and wiping the contents stored by Stardog.
- Create an empty
$STARDOG_HOMEdirectory on all of the Stardog Cluster nodes.
$STARDOG_HOMEto the empty home and run
server restore(the same as you would for a single node) on a single node.
- Start a fresh ZooKeeper ensemble with an empty data directory.
- Start ONLY the Stardog node where you performed
server restore. Verify the node starts and is in the cluster with the
cluster infocommand before continuing to step 7.
- Start a second node in the cluster with its empty home directory, wait for it to sync and join the cluster, as reported by
cluster info. Wait until the node joins before moving to step 8.
- Repeat step 7, one node at a time, for the remaining cluster nodes.
db repair command can only be run on a single node cluster to ensure the cluster remains consistent. Therefore, you must shut down the additional nodes before you can run repair. We recommend you terminate the non-coordinator nodes in your cluster by issuing the
server stop command to each server individually. Terminating writes to the cluster before shutting down the nodes will help speed the process along. You can do this by offlining any databases with open write transactions or by putting the cluster into read only mode.
Once you scale the cluster down to a single node, you can run the repair command. After it completes you can start the other cluster nodes again, one at a time, allowing each to join before moving on to the next one until your cluster is back at full strength.
Stardog Cluster can run in read only mode as a preview feature starting in 7.7.0. In read only mode the cluster will accept reads, such as listing roles and users as well as read queries, and reject writes, including:
- Data adds
- Database create and drop
- Restore a backup
- SPARQL update queries
- User and role modifications
- VG operations to add, remove, and refresh VGs
In addition to read operations, other operations that are permitted in read only mode, include:
- Disable read only mode
- Shutdown the cluster
- Kill a query
- Generating the cluster diagnostic report
Read only mode is a cluster-only feature. It is not available for single node Stardog.
You can start read only mode on a running cluster with the following command:
$ stardog-admin cluster readonly-start
If the command is unable to acquire the necessary locks due to other write transactions, you can stop them by offlining those databases and then starting read only mode. Be advised that offlining a database will terminate all read and write queries for that database.
Stop read only mode, returning the cluster to read/write, with the following command:
$ stardog-admin cluster readonly-stop
A user must be an admin or have the permission
[EXECUTE, dbms-admin:read-only] to start and stop read only mode.
ZooKeeper is independent from the Stardog Cluster nodes. Stardog officially supports ZooKeeper 3.4.14 and Stardog 7.4.0+ supports ZooKeeper 3.5.9 in preview mode. Please refer to the ZooKeeper documentation for more details managing ZooKeeper and following its best practices.
To upgrade ZooKeeper, we recommend that you first stop all writes to Stardog and then backup all Stardog and ZooKeeper nodes. Once the backup is complete you should stop all Stardog nodes, following the instructions in the shutdown section.
After the cluster is stopped, you can upgrade ZooKeeper, either by performing a rolling upgrade of the ZooKeeper nodes (if allowed between the ZooKeeper versions you are upgrading) or by deploying a new, empty ZooKeeper ensemble with the new version.
If you deploy a new, empty ZooKeeper make sure to start a Stardog cluster node that contains all of your data. If you start a Stardog node that is out of date (e.g., if it was previously expelled from the cluster before you stopped writes and shut down the cluster) then it’s possible you will lose data when the remaining Stardog cluster nodes start and sync.
Stardog cluster nodes can be started one at a time once the new version of ZooKeeper is running successfully on all ZooKeeper nodes.