Database Checkpoints
This page discusses using database checkpoints to preserve point in time database images.
Page Contents
Overview
A checkpoint is a point in time database image. Stardog is able to quickly create a parallel directory within the STARDOG_HOME directory that contains all the files of the selected database. The active and parallel data directory share the same disk space via a technique called “hard links”. Restated: Only directory entries are duplicated, not the actual files. The total disk storage is the equivalent of the active database only, not two databases (active and parallel). The parallel data directory represents a point in time copy. Any data later added to or deleted from the active database has no impact on the parallel data directory.
Limitations:
- Stardog 8.0 introduces checkpoints only for the system database, not user databases. User database checkpoints are planned for a future release.
- Checkpoints on Windows systems are limited to certain types of storage volumes.
- Stardog’s checkpoints are not yet operational in cluster configurations.
Listing available checkpoints
EXAMPLE
List checkpoints available for the Stardog database called system:
$ stardog-admin db checkpoint list system
+-------------------------+
| Checkpoints |
+-------------------------+
| system_2022-05-18_12.00 |
| system_2022-05-18_11.48 |
| system_2022-05-12_11.05 |
| system_2022-05-05_10.14 |
+-------------------------+
Creating a checkpoint
Stardog has two methods for creating a checkpoint: automatic and manual. Stardog activates automatic checkpoints for the system database on non-Windows implementations. The user can also request a checkpoint via the stardog-admin command whenever desired.
Checkpoints are activated on Windows platforms, but may not work. Checkpoints depend upon a storage feature call “hard links”. Not every Windows storage volume supports this feature.
Automatic checkpoints
Automatic checkpoints are controlled via the stardog.properties file. The default properties are:
storage.checkpoint.system.interval.minutes = 360
storage.checkpoint.system.intervals.keep = 3
storage.checkpoint.system.days.keep = 3
storage.checkpoint.system.weeks.keep = 2
Interval minutes: how often should a checkpoint be generated (the default of 360 minutes is 6 hours). Setting this value to zero disables automatic checkpoints. For non-zero settings, the intervals align from midnight based upon the server’s clock. First checkpoint occurs at midnight plus the interval minutes. Another checkpoint occurs every subsequent interval minutes. There is no attempt to make interval minutes fit evenly within 24 hours. Example: using 420 minutes (7 hours), checkpoints occur at 7am, 2pm, and 9pm. Next one is 7am the following day. The alignment to local midnight keeps this server’s checkpoints consistent across restarts. The alignment also allows checkpoints of all servers in a cluster to be “close” to each other, though not explicitly synchronized.
Intervals keep: how many of the checkpoints that were generated at each interval should be maintained. Older checkpoints get removed, except for those needed to satisfy the days keep and weeks keep settings.
Days keep: how many of the last checkpoints generated on a calendar day, i.e. last checkpoint before midnight, should be maintained. This setting is incremental to the “intervals keep” setting. It is possible for the last checkpoint of a day to temporarily count toward both intervals keep and days keep.
Weeks keep: how many of the last checkpoints generated on a Sunday should be maintained. This setting is incremental to both “intervals keep” and “days keep”. It is possible for the last checkpoint on a Sunday to temporarily count toward intervals keep, days keep, and weeks keep as their respective times overlap.
EXAMPLE
You can disable automatic checkpoints by setting storage.checkpoint.system.interval.minutes to 0. You must add
storage.checkpoint.system.interval.minutes = 0
to your stardog.properties file, then restart Stardog.
Manual checkpoints
You manually initiate a checkpoint using the stardog-admin command. Checkpoints happen almost instantaneously and do not impact other Stardog activities already processing.
EXAMPLE
Create checkpoint for the Stardog database called system:
$ stardog-admin db checkpoint create system
Successfully created checkpoint: system_2022-05-18_11.48
Restoring a system database checkpoint
Crisis restore
A crisis restore is only used if Stardog will not restart. This procedure is a last hope process for bringing Stardog back from a failure that has corrupted its system database. The steps are listed below, but it is recommended that you contact Stardog Support first.
Step 1. Add the following line to the stardog.properties file.
storage.system.db.restore.enabled = true
Step 2. Start Stardog.
$ stardog-admin server start
Step 3. Stop Stardog.
$ stardog-admin server stop
Step 4. Remove the “.restore.enabled” line from stardog.properties file.
Step 5. Start Stardog again.
$ stardog-admin server start
Step 2 above has Stardog restore the most recent system database checkpoint. If Stardog successfully starts in step 2, the restore was successful. Otherwise, you need to contact Stardog Support to see if there are any possible, manual alternatives.
Step 4 is essential. You must remove the extra line from the stardog.properties file. Otherwise Stardog will restore the most recent checkpoint on every future restart.
Manual restore
A manual checkpoint restore of a Stardog system database is possible only if Stardog is operational. The procedure is only used if some portion of the system database is corrupted, but not to the extent the corruption prevents restart. Please discuss this procedure and its options with Stardog Support prior to use.
Step 1. List the checkpoints available.
$ stardog-admin db checkpoint list system
+-------------------------+
| Checkpoints |
+-------------------------+
| system_2022-05-18_12.00 |
| system_2022-05-18_11.48 |
| system_2022-05-12_11.05 |
| system_2022-05-05_10.14 |
+-------------------------+
Step 2. Select the system database checkpoint to restore.
$ stardog-admin db checkpoint restore system system_2022-05-12_11.05
System checkpoint system_2022-05-12_11.05 ready. Please 'stardog-admin server stop', then 'stardog-admin server start' to apply.
Step 3. Restart Stardog.
$ stardog-admin server stop
$ stardog-admin server start
Manual restore options for system database
Please discuss these options with Stardog Support prior to use. Inappropriate use may cause data loss across all databases, not just the system database. These options exist to help partially restore Stardog when faced with severe data corruption.
$ stardog-admin db checkpoint restore system system_2022-05-12_11.05 --restore-transactions --restore-naming
--restore-transactions and --restore-naming options may be used alone or together. They control whether or not two extremely important sections of the system database get restored or not. The default is to NOT restore the two sections. It is recommended to first attempt restore without the options. It is possible to retry the same checkpoint restore using the options incrementally. However, there is no way to revert one of these options once used.
--restore-naming can result in any user databases that were added after the checkpoint to become inaccessible.
--restore-transactions can result in any data changes and/or whole databases added after the checkpoint to become inaccessible.