Can Tracker provide a stand-alone RDF data store with a SPARQL query interface?

[dnn (Daniele Nicolodi)]

Hello,

I am looking for a solution for a cross-platform application to store a RDF database and query it using SPARQL (or a custom query language that gets translated into SPARQL). I am prototyping the thing with Python RDFlib, but I expect it to be too slow when it comes to handle the real datasets.

I haven’t looked at the software components at all, but it seems that Tracker implements most of what I need. Is it feasible to have a “private” instance of a Tracker database (possibly backed by an in-memory SQLite database) and an interface of sort to run queries? From where should I start to try to put together something like this?

Thank you.

Cheers,
Dan

[sthursfield (Sam Thursfield)]

Hi Dan
Tracker 3 supports private instances, use tracker_sparql_connection_new() to create a private database connection, and tracker_sparql_connection_query() to run queries. All Tracker databases are backed by SQlite, and if you pass NULL as the store parameter then it’ll be an in-memory database.

If you have more questions, we are also in tracker on IRC and Matrix.

[dnn (Daniele Nicolodi)]

Thanks Sam. This is interesting. How hard is to get Tracker on macOS and Windows? I had a quick look and, for exampe, macports only what looks like an ancient 1.12.4 version. Does Tracker require a daemon running or can it be used as a pure library? After a quick look to the documentation, it seems that Tracker does not support all XSD data types. Can it be extended to, say, support decimals?

[sthursfield (Sam Thursfield)]

Tracker is two things - a database and a filesystem indexer. I think you are interested in the database, which we call “Tracker SPARQL”, and lives in the tracker.git project. The filesystem indexer is “Tracker Miners” and lives in the tracker-miners.git project.

Historically the two parts were lumped together in one project and the whole thing was called “Tracker”. Nobody made an effort to port to Windows or Mac because they have built-in filesystem indexing anyway,

Now that Tracker SPARQL is separated, we could look at what’s needed to run on other platforms. The build system in Meson which is designed to work nicely on Windows and Mac, so give it a try and see if it builds Portability patches for Tracker SPARQL would be welcome.

The daemon is for the indexing part only, the database is a library (libtracker-sparql).

As for XSD support, the database can in principle support anything that SQLite supports. Tracker SPARQL aims to be small, fast and standards-compliant, so patches that bring it inline with existing standards are welcome for sure.

edit: I saw you briefly in tracker, myself and Carlos Garnacho are both in Europe / CEST timezone, if that helps

[dnn (Daniele Nicolodi)]

Thank you for the very informative reply. I’ll start exploring. I would like to use tracker-sparql for a side project of a side project, thus hacking on tracker itself is a bit too much far on the side to be an effective way forward. I am afraid that getting it to work on Windows may be a bit too much work (I don’t even know how hard it is to get valac running in Windows).

For giving a bit of context, I am exploring RDF ans SPARQL as an alternative or as a different foundation for bean-query GitHub - beancount/beanquery: A customizable light-weight SQL query tool that works on tabular data, including Beancount. I started familiarizing myself with RDF and SPARQL using Python RDFlib but the performance is definitely not what I would like (unless I am doing something wrong with it).

[dnn (Daniele Nicolodi)]

It seems that macports has all the build dependencies for

$ meson setup build/ --reconfigure -Ddocs=false -Dman=false -Dsystemd_user_services=false

to work. However, compilation does not go very far: macOS does not have pipe2():

[4/137] Compiling C object src/libtracker-common/libtracker-common.a.p/tracker-term-utils.c.o
FAILED: src/libtracker-common/libtracker-common.a.p/tracker-term-utils.c.o 
ccache cc -Isrc/libtracker-common/libtracker-common.a.p -Isrc/libtracker-common -I../src/libtracker-common -I. -I.. -Isrc -I../src -I/opt/local/include/glib-2.0 -I/opt/local/lib/glib-2.0/include -I/opt/local/include -I/opt/local/include/gio-unix-2.0/ -fvisibility=hidden -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -g -Wformat -Wformat-security -Wno-pointer-sign -DTRACKER_COMPILATION '-DG_LOG_DOMAIN="Tracker"' -DG_LOG_STRUCTURED=1 -DG_ENABLE_DEBUG -DG_ENABLE_CONSISTENCY_CHECKS -D_REENTRANT -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -MD -MQ src/libtracker-common/libtracker-common.a.p/tracker-term-utils.c.o -MF src/libtracker-common/libtracker-common.a.p/tracker-term-utils.c.o.d -o src/libtracker-common/libtracker-common.a.p/tracker-term-utils.c.o -c ../src/libtracker-common/tracker-term-utils.c
../src/libtracker-common/tracker-term-utils.c:160:6: error: implicit declaration of function 'pipe2' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        if (pipe2 (fds, O_CLOEXEC) < 0)
            ^
../src/libtracker-common/tracker-term-utils.c:160:6: note: did you mean 'pipe'?
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/unistd.h:470:6: note: 'pipe' declared here
int      pipe(int [2]);
         ^
1 error generated.

I think the port to Windows would be a major overtake.

[sthursfield (Sam Thursfield)]

Right, RDFLib isn’t designed for performance indeed.

Thanks for the Mac info, I suspect that with a few well chosen #ifdefs we could get tracker-sparql to compile there. I do have access to a Mac at present, but not much time either unfortunately. I don’t have access to Windows so that will remain a mystery!

[garnacho (Carlos Garnacho)]

GTK CI builds and tests on mac/mingw as well. It might be nice to adopt that for Tracker.

I too think it shouldn’t be hard to make Tracker build and run correctly on these. The specific tracker-term-utils.c file is quite unix-oriented, but it’s also fairly superfluous (contains helper terminal-oriented functions for CLI utilities, redirecting to pager, figuring out terminal WxH, etc). Everything else is (best effort) OS independent.

[dnn (Daniele Nicolodi)]

I’m quickly hacking something together to get the build going on macOS. The pipe2 issue is trivial to solve, however, next I hit an undefined function: json_generator_to_gstring. I assume it comes from json-glib, but I don’t find it in the exported API and there is indeed a bug open about adding it [BZ#785750] Add json_generator_to_gstring() helpers (#23) · Issues · GNOME / json-glib · GitLab. Is my json-glib too old?

[dnn (Daniele Nicolodi)]

Yup. json-glib too old. Maybe the dependency check needs to be adjusted in meson.build. Strangely enough, reading the json-glib changelog, the missing function was added in version 1.2 and I have version 1.8.

However, I added jsdon-glib as a subproject for now.

Now I have a link error related to iconv…