Testing Queries

This page discusses query testing capabilities provided by Stardog to perform regression testing for correctness and performance using declarative test definitions.

Page Contents

1. Overview
2. Test Definitions
   1. Test Configuration
   2. Specifying Queries
   3. Correctness Testing
   4. Performance Testing
   5. Test Modules
   6. Test Definition Vocabulary
3. Running Tests
4. Auto Test Creation

---

Overview

Testing the performance and correctness of software code is an indispensable part of software development process. When developing a graph-based solution, a similar capability is required for testing the performance and correctness of queries as the data models and the queries evolve over time. This kind of testing can certainly be done by using your choice of Stardog API with a software testing library. However, Stardog also provides a way to declaratively define your tests in a test definition file, automatically create test definitions from your existing queries and run these tests via CLI command and/or using the API.

Test Definitions

A test definition is an RDF file commonly serialized using the Turtle syntax. A test definition file contains a list of tests and each test is for testing a single query against a single database with configuration parameters. Typically, most of the configuration parameters are defined at the file level but each test might define its configuration parameters as shown in the following example:

:SelectAlbums a :Test ;
    :server "https://express.stardog.cloud" ;
    :database "stardog-tutorial-music" ;
    :queryString "SELECT ?album { ?album a :Album }" ;
    :resultCount 1037 ;
    :warmups 2;
    :runs 3;
    :expectedTime 100 .      

As seen a single test definition is a set of properties attached to the same subject. The local name of the subject is used as the test name and each test definition should have the Test type declaration shown in the first line of this example.

We explain what the configuration parameters mean and how they can be organized in definition files in the following sections.

Test Configuration

Each test definition file can optionally contain a configuration section at the beginning that specifies the configuration options that will be used by default. The default value are attached to a special subject named defaults but otherwise look exactly like how they would look like when attached to regular tests.

:defaults
    :server "https://express.stardog.cloud" ;
    :database "stardog-tutorial-music" ;
    :warmups 2;
    :runs 3 .      

Typically, server and database information will be shared between the tests and defined in the default configuration section. The default values will be used for all the tests unless the test overrides these configuration options.

The credentials to access the server are not specified as part of the test definitions in order to avoid exposing sensitive information in plain files. The credentials are specified while running tests and can take advantage of features like password files.

Specifying Queries

There are three different ways to specify the query for a test and only one should be used for each query.

1. First option is to specify a query string inline in the test definition file using the queryString property:

    :SelectAlbums a :Test ;
        :queryString "SELECT ?album { ?album a :Album }" ;
        ...
   

   Multi-line query strings can be specified using the triple-quote syntax of Turtle:

    :SelectAlbums a :Test ;
        :queryString """SELECT ?album 
    WHERE { 
       ?album a :Album 
    }""" ;
        ...
   

   When the query string is sent to the database it will use the stored namespaces from the database. The query string may include additional prefix declarations too. But the prefix declarations defined in the test file will not be applied to the query.

2. The second option is to specify a path to a query using the queryFile property:

    :SelectAlbums a :Test ;
        :queryFile "selectAlbums.sparql" ;
        ...
   

   The property value is a path for the query file. If a relative path is specified then it will be resolved using the path of the test definition file. That is, in the above example selectAlbums.sparql should exist in the same directory as the definition file. Paths may point to subdirectories or parent directories as needed.

3. The third option is to specify the name of a stored query using the queryStored property:

    :SelectAlbums a :Test ;
        :queryStored "selectAlbums" ; 
        ...
   

   The stored query with the given name should exist in the server and should be executable against the database.

Correctness Testing

There are two different ways to check the correctness of a query.

1. The first way is to save the expected query results into a file and specify the file path in the test definition using the resultFile property:

    :SelectAlbums a :Test ;
        :resultFile "selectAlbums.srx" ;
        ...
   

   The file path is again relative to the test definition file similar to queryFile explained above. The file format will be selected based on the extension of the file name. The extensions .srx or .xml can be used if the query results are in the SPARQL/XML format and the extensions .srj or .json can be used if the query results are in the SPARQL/JSON format. The query results should not contain any bnodes and very large result sets, e.g. more than million bindings, should be avoided.

2. The second option is to specify the number of expected results using the resultCount property:

    :SelectAlbums a :Test ;
        :resultCount 1037 ;
        ...
   

   This option is not as reliable because a query may return the same number of results but the contents of the query results might not be same. But in some cases such as the very large result set use case mentioned above this would be a reasonable option.

   The result count for queries are described as follows:

   • SELECT: The number of bindings, i.e. rows, returned
   • CONSTRUCT and DESCRIBE: The number of unique triples returned
   • ASK: Count is 1 if the query returns true and 0 if false
   • PATHS: The number of paths returned

Update queries (COPY, CLEAR, INSERT, …) do not return any results; they update the database. The resultCount or resultFile values specified for update queries are simply ignored. since an update query can never fail in a test by definition. The correctness of update queries can be checked by adding a read query after the update query in the test definitions. The tests defined in a file are executed in the order they have been defined in the file so update and read queries can be organized in a way to allow this.

Performance Testing

Testing the performance of a query is done by running the query multiple times (preferably after a number of warmup runs) and comparing the average execution time with the expected query answering time. Expected query time is specified in milliseconds using the expectedTime property:

:SelectAlbums a :Test ;
    :expectedTime 100 ;
    ...      

It is expected that the query execution times fluctuate slightly between runs so a slight difference compared to the expected time does not necessarily indicate a slowdown. For this reason, there is a notion of a “failure threshold” that is take into account before a test is considered to fail. Threshold is expressed as a percentage point (integer value) and the default failure threshold value is 10. For example, if the expectedTime for a test is 100ms as in the above example then the test will fail if average query execution is more than 110ms. The percentage can be set to a value higher than 100 if the query execution time is known to fluctuate highly especially for queries that complete very quickly.

The default warmups are 0 and the default number of runs 1 which is geared towards correctness testing more than performance testing. You should increase these values if you will do performance testing. By default, auto test creator uses 3 warmups and 2 runs but based on query characteristics more warmups or runs might be needed. Running the queries multiple times and observing query execution times is the most reliable way to determine optimal number of warmups and runs.

By default, update queries are executed only once regardless of the warmups and runs setting. It is typically not meaningful to run an update query multiple times since the first execution applies the changes and subsequent executions have no effect. But in some cases it might make sense to run update queries multiple queries

Test Modules

Test definitions can be grouped in different files and a test definition file can be included in another one using the include property:

[] :include "album-test.ttl" , "artist-test.ttl" , "path-test.ttl" .

If a test file defines defaults then those defaults will be applied to the included test file and overridden by any defaults defined in the included files. Included test files will be run in the order they have been defined in the file. A test file can contain both include statements and test definitions. Again execution ordering will follow the definition order.

Test Definition Vocabulary

The RDF vocabulary used for test definition is defined in the namespace tag:stardog:api:test: but there is no need to declare this namespace in the test definition file because the test reader utility uses this default namespace while parsing the test definition files.

There is only a single class Test defined in the vocabulary and all the properties except for include are attached to test instances. The special test defaults is used to define default properties that will apply to all the tests as explained above.

Running Tests

Tests can be run with the following command:

$ stardog test run test-file.ttl

The command will print progress information as test are rerunning and print a summary of test results:

+--------------------+------------------------------+-------------------------------------------------------+
|       Result       |             Test             |                        Message                        |
+--------------------+------------------------------+-------------------------------------------------------+
| PASSED             | query1                       |                                                       |
| FAILED_ERROR       | query2                       | java.nio.file.NoSuchFileException: query2.sparql      |
| FAILED_TIMING      | query2                       | Query Time Expected : 184 Actual: 226 (Slowdown: 22%) |
| FAILED_CORRECTNESS | query4                       | Query Results Expected : 1036 Actual: 1037            |
| PASSED             | query5                       |                                                       |
+--------------------+------------------------------+-------------------------------------------------------+

Finished running 5 tests in 00:00:01.389
	PASSED: 2
	FAILED_CORRECTNESS: 1
	FAILED_ERROR: 1
	FAILED_TIMING: 1

As shown in the output above there are four possible outcomes for a test:

• PASSED: Test passed
• FAILED_CORRECTNESS: The results returned by the query did not match expected results
• FAILED_TIMING: The average execution timing exceeded expected query time more than the threshold percentage
• FAILED_ERROR: An error occurred during the execution of a test

The test run command by default uses the admin/admin credentials to connect to the server specified in the test definition file. A different username and password can be provided just like any other CLI command:

$ stardog test run -u myuser -p mypass test-file.ttl

The test run command provides additional options, e.g. to print query plans. Please check out the help page for a complete list of options supported.

Auto Test Creation

If you have a set of SPARQL queries saved in files then you can use the stardog test create command to automatically create a test definition using the following command:

$ stardog test create myDb path/to/queries

The command will recursively traverse the directory, find the query files matching the provided glob expression (by default files with extension .sparql or .rq), execute each query against the provided database and generate one test definition file for each directory. The name of the test definition file will be same as the directory name. The test file can be renamed as long as it is not used in an include statement. Test definition file will contain one test for each query file found. By default, the command runs each query 3 times as a warmup and then runs the query times to compute the expected time. The results for each query is saved in a file in the same directory as the query file.

Here is another example creating a test file for the SPARQL queries from the Stardog tutorials against the publicly accessible Stardog Express server:

Note that, we are specifying the server URL as part of the connection string with the database name. We have also changed the credentials used to connect to the server since the default admin/admin credentials are not allowed for Stardog Express. Update queries are not allowed against Stardog express. Stardog tutorial contains two update queries (20-bands-members-insert.sparql and 21-songs-length-delete.sparql) so we provided a glob expression that tells the command to only include SPARQL files that start with the character 0 or 1 effectively excluding the update queries. Finally, the --only-counts tell the command to not save query results in files and instead use the resultCount property to only record expected number of results.

If we want to create a test file that only checks correctness and not performance we can use the --no-timings option and run each query only once without the warmups:

---