Replica Clusters

This page describes Replica Clusters in Stardog - a read-only HA cluster that replicates from a primary cluster and can be promoted to take its place.

This feature is in beta.

Page Contents

1. Overview
2. Configuring a Replica Cluster
3. Using a Replica Cluster
   1. Reads and writes
   2. Syncing
   3. Monitoring
4. Promoting a Replica Cluster

---

Overview

Stardog 12.2.0 introduces the replica cluster, which is an ordinary Stardog Cluster, with its own ZooKeeper ensemble, that runs in replica mode: it serves reads, rejects client writes, and periodically pulls updates from another cluster, the primary. It typically runs in a different data center or region from the primary. Its main uses are:

1. Disaster recovery. If the primary’s region is lost, you can promote the replica cluster to a writable primary with one command. The promoted cluster does not need to restart in order to begin accepting writes.

2. High availability at the remote site. The replica cluster survives the loss of any node, just like the primary. If the coordinator fails, a new coordinator is elected and it resumes syncing, starting from the last transaction the cluster committed.

3. Remote reads. Clients at the remote site can run queries against the replica cluster. Results may lag the primary by up to one sync interval.

Only the replica cluster’s coordinator contacts the primary. It pulls each transaction committed on the primary and applies it to every node in the replica cluster through the cluster’s normal two-phase commit, so all nodes hold the same data. Connections go outbound from the replica cluster to the primary only; the primary never connects to the replica site. The primary needs no configuration change.

A replica cluster differs from a geo replica in these ways:

A replica cluster is licensed exactly like a standard cluster: every node needs the same license key, and the license must allow at least as many nodes as the replica cluster has. Replica cluster nodes are not standby nodes and do not count toward the standby node limit. Promotion does not change the license requirements.

Configuring a Replica Cluster

To set up a replica cluster:

1. Set up a cluster at the remote site as described in Installation and Setup, with its own ZooKeeper ensemble. Do not point it at the primary’s ZooKeeper.
2. Add the replica settings below to stardog.properties on every node.
3. Start the nodes.

A complete stardog.properties for one node of a replica cluster looks like this:

pack.enabled=true
pack.node.address=10.2.0.4
pack.zookeeper.address=10.2.0.1:2180,10.2.0.2:2180,10.2.0.3:2180

pack.replicaCluster=true
pack.replicaCluster.primary.address=<primary cluster endpoint>:5820
pack.replicaCluster.primary.user=<primary cluster superuser>
pack.replicaCluster.primary.password=<primary cluster superuser password>

The replica settings are the same on every node. The primary address is the IP or domain name and port of the primary cluster’s endpoint, typically its load balancer. The user must be a superuser on the primary. Because the credentials are in stardog.properties, we recommend that the file is only readable to Stardog and that the file system is encrypted.

Do not set pack.replicaCluster.primary.insecure=true in production environments or across wide-area networks with sensitive data. All Stardog data and metadata will be sent unencrypted over HTTP.

pack.replicaCluster cannot be combined with pack.standby, pack.readReplica, or pack.geoReplica. A node with both set refuses to start.

A replica cluster holds exactly the primary’s databases. The first sync copies every database from the primary in full, and any database on the replica cluster that the primary does not have is dropped.

Using a Replica Cluster

Reads and writes

A replica cluster accepts reads and read queries on every node. It rejects every client write, including data updates and admin operations such as creating a database or adding a user, with this error:

The cluster is in replica mode and does not accept client writes; write to the primary cluster instead

Users, roles, permissions, and database settings come from the primary. Make those changes on the primary; they reach the replica cluster with the next sync.

This rejection is separate from read-only mode. cluster readonly-stop does not make a replica cluster writable; only promotion does.

The query log does not run on a replica cluster. It starts when the cluster is promoted.

Syncing

Every pack.replicaCluster.sync.interval, the coordinator compares each database’s last transaction with the primary’s and pulls any transactions it is missing. Each transaction commits on all nodes of the replica cluster at once, so the replica cluster always holds the primary’s data as of some committed transaction.

The coordinator copies a database in full instead of replaying transactions when:

• The database is new to the replica cluster, including on the first sync.
• The primary’s transaction log cannot be replayed from the replica cluster’s position.
• The system database changed on the primary. The system database, which holds users, roles, permissions, and database settings, is always copied in full.

A full copy is built alongside the existing database, which serves reads throughout. When the copy finishes, reads of that database are refused on every node for a brief window, typically under a second. A full copy of the system database briefly interrupts reads of every database.

Monitoring

cluster status shows these fields for each node of a replica cluster:

Only the coordinator syncs, so its entry describes what the cluster has pulled. The other nodes report NOT_COORDINATOR.

The standby commands, such as cluster standby-status, do not apply to a replica cluster.

Promoting a Replica Cluster

Promotion turns a replica cluster into an ordinary writable cluster. The promotion process runs online: no node restarts or leaves the cluster, no data is copied, and reads stay available throughout.

Promotion does not stop the old primary from accepting writes. You must stop clients from writing to it, through routing, a load balancer, or DNS. Otherwise both clusters accept writes and their data diverges.

Transactions committed on the old primary after the replica cluster’s last sync are not present on the promoted cluster, and are never fetched afterwards.

To promote a replica cluster:

1. If the old primary is still running, stop clients from writing to it. If the old primary is down, go to step 3.
2. If the old primary is still reachable, wait until cluster status on the replica cluster shows the coordinator’s Last Sync Result as IN_SYNC.
3. Run cluster status on the replica cluster and record each database’s Synced Transaction IDs. This is the point in the old primary’s history the new primary starts from. If the old primary is still reachable, run cluster status on it and compare these IDs with each database’s Local Last Txn ID in its Last Transaction IDs table. If any database differs, repeat steps 2 and 3. The query log database (querylog by default) keeps changing while the old primary serves queries, so exclude it from the comparison.

4. Promote the replica cluster. Send the command to any node:

   stardog-admin --server http://<replica cluster endpoint>:5820 cluster promote
   

   The user running stardog-admin cluster promote must be a superuser.

5. Point clients at the promoted cluster.
6. Set pack.replicaCluster=false in stardog.properties on every node. No restart is needed; the change takes effect at each node’s next restart.

Do not skip step 6. The cluster records the promotion only in ZooKeeper. If ZooKeeper’s data is lost while pack.replicaCluster=true is still set, a restarted node reconnects to the old primary as a replica and overwrites every write made since promotion.

cluster promote works in two steps:

1. Detach. Every node stops syncing from the primary. A sync already in progress, which may be a full copy crossing the network, is allowed to finish, for up to pack.replicaCluster.promote.quiesce.timeout.
2. Accept writes. Every node starts accepting client writes.

If a node cannot detach within the timeout, the command fails and no node accepts writes. The cluster is left detached, and cluster status shows Replica State: DETACHED. Once the in-progress sync has ended, run cluster promote again to complete the promotion process.

The command also refuses to lift the write gate while the cluster is in read-only mode. Run cluster readonly-stop, then run cluster promote again.

If a node is unreachable when writes are enabled, the command reports it and does not expel it. The node starts accepting writes when it reconnects.

Replica State: PRIMARY on a promoted cluster means that node still has pack.replicaCluster=true in stardog.properties. Set it to false (step 6).

A promoted cluster cannot be turned back into a replica cluster. To create a new replica cluster from its nodes, stop every node, clear ZooKeeper’s state, and remove the databases from each node’s Stardog home directory, keeping the license key and stardog.properties. Then configure the nodes as described in Configuring a Replica Cluster and start them. The first sync copies every database from the primary.

---