The primary system resources used by Stardog are CPU, memory, and disk. Stardog also uses file handles and sockets, but we don’t discuss those here. Stardog will take advantage of more than one CPU, core, and core-based thread in data loading and in throughput-heavy or multi-user loads. Stardog performance is influenced by the speed of CPUs and cores. But some workloads are bound by main memory or by disk I/O (or both) more than by CPU. Use the fastest CPUs you can afford with the largest secondary caches and the most number of cores and core-based threads of execution, especially in multi-user workloads.
Stardog uses system memory aggressively and the total system memory available to Stardog is often the most important factor in performance. Stardog uses both JVM memory (heap memory) and also the operating system memory outside the JVM (direct or native memory). Having more system memory available is always good; however, increasing the total memory limit too close to total system memory is not prudent as operating system will not have enough memory for its own operations (see guidelines below).
The following table shows recommended system memory for Stardog based on the graph locally stored in Stardog and how the system memory should be divided between the JVM heap memory and the direct memory. Note that, the exact amount of needed memory can vary a lot depending on many factors other than the graph size such as the characteristics of your graph, amount of data processed by queries, virtual graph access patterns, transactional load on the system, amount of data bulk loaded into new databases, number of concurrent users and so on.
The values in this table can be used as a guideline but the only way to make sure you have optimal settings is trying your workload on your data and analyzing memory metrics provided by Stardog.
|Number of Triples||JVM Heap Memory||Direct Memory||Total System Memory|
Out of the box, Stardog sets the maximum JVM memory to 2GB and direct memory to 1GB which works fine for most small databases (less than 100 million triples) but should be increased as recommended above for larger datasets. You can increase the memory for Stardog by setting the system property
STARDOG_SERVER_JAVA_ARGS using the standard JVM options. For example, you can set this property to
"-Xms4g -Xmx4g -XX:MaxDirectMemorySize=8g" to increase the JVM memory to 4GB and off-heap to 8GB. We recommend setting the minimum heap size (
-Xms option) and max heap size (
-Xmx option) to the same value.
Some general guidelines that can be used in addition to the above table:
- Heap memory should not be less than 2GB and setting it higher than 100GB is typically not recommended due to increased GC pauses.
- JVM uses Compressed OOPs optimization if heap limit is less than 32GB. If you want to set the heap limit to higher than 32GB then you will only see noticeable benefits if you go to 50-60GB or higher.
- Direct memory should be set higher than heap memory except for very small scales to prevent the heap size going below the recommended 2GB limit.
- The sum of heap and direct memory settings should be around 90% of the total system memory available so operating system has enough memory for its own operations.
- It is not recommended to run any other memory-intensive application on the same machine that a Stardog server is running as those applications would compete for the same resources and if the overall memory usage in the system increases to dangerously high levels the operating system or the container will kill the Stardog process.
Stardog stores data on disk in a compressed format. The disk space needed for a database depends on many factors besides the number of triples, including the number of unique resources and literals in the data, average length of resource identifiers and literals, and how much the data is compressed. As a general rule of thumb, every million triples require 70 MB to 100 MB of disk space. The actual disk usage for a database may be different in practice. It is also important to realize the amount of disk space needed at creation time for bulk loading data is higher as temporary files will be created. The extra disk space needed at bulk loading time can be 40% to 70% of the final database size.
The disk space used by Stardog is additive for more than one database and there is little disk space used other than what is required for the databases. To calculate the total disk space needed for more than one database, one may sum the disk space needed by each database.