This page discusses Stardog’s NiFi integration support to ETL data sources to Stardog.
Stardog supports integration with Apache Nifi, a scalable, robust ETL platform that can be used to load and transform data from Enterprise data sources to Stardog. Stardog includes three Processors for loading, querying and updating a Stardog database.
To install NiFi and the Stardog connector:
- Go to http://nifi.apache.org/download.html and download the latest binary release (nifi-1.12.0-bin.zip as of this writing)
- Decompress the zip file to a local folder
- Download the Stardog NiFi nar files from http://downloads.stardog.com/extras/stardog-extras-
STARDOG_VERSION_NUMBER.zip into the
libfolder in the NiFi installation folder, where
STARDOG_VERSION_NUMBERis the version of your Stardog server.
Start the NiFi server by running the command
bin/nifi.sh start in the NiFi installation folder. It takes up to a minute for the NiFi server to start. Once the server is running you can go to the URL http://localhost:8080/nifi in your browser, which will show the empty workflow. You can drag the processor icon from the top left to the empty canvas and add a Stardog processor:
Once the processor is added you can change the parameters to specify the Stardog server to connect to, credentials, etc. See the following example for more details:
An example NiFi workflow is provided in the Stardog Examples GitHub repository. The workflow is for loading the Coivd19 dataset published by New York Times on GitHub into Stardog. It contains three processors:
- NiFi built-in processor to retrieve the CSV file from GitHub.
- StardogPut processor that ingests the CSV file into a staging graph in Stardog using the Stardog mappings available in the examples repository.
- StardogQuery processor that copies the staging graph to the default graph and updates the last modification time.
Follow these steps to upload this workflow to your NiFi instance (see the screencast below and refer to Apache NiFi user interface for terminology):
- From the Operate Palette, click the “Upload Template” button and select the
- Drag the “Template” icon from the Components Toolbar onto the canvas.
- Unselect the processors by clicking an empty spot on the canvas and then select the
StardogPutprocessor to configure the connection details and point to the correct location for the mappings file
- Modify the connection details for the
StardogQueryprocessor in a similar way.
The example is created to run every hour so if you have leave NiFi running the data will be fetched, transformed and uploaded into Stardog every hour.
As an alterative to supplying the Stardog url and credentials in every Stardog processor, you can configure the Stardog Connection Service just once and then reference that service in each Stardog processor.