Using Apache Hive to directly evaluate SPARQL queries.
Overview: SPARQL is the W3C standard query language for querying data expressed in RDF (Resource Description Framework). The increasing amounts of RDF data available raise a major need and research interest in building efficient and scalable distributed SPARQL query evaluators.
In this context, we propose and share RDFHive: a simple implementation of a distributed RDF datastore benefiting from Apache Hive. RDFHive is designed to leverage existing Hadoop infrastructures for evaluating SPARQL queries. RDFHive relies on a translation of SPARQL queries into SQL queries that Hive is able to evaluate.
Technically, RDFHive directly evaluates SPARQL queries i.e. there is no preprocessing step, indeed an RDF triple file is seen by Hive as a three-column table. Thus, the bash translator simply translates SPARQL queries according to this scheme. This method has two advantages: first, creating a database is very fast; second, since the upfront investment is light, RDFHive is an interesting tool to evaluate a few SPARQL queries at once.
Version: 1.0
- Apache Hadoop (+HDFS) version 2.6.0-cdh5.7.0
- Apache Hive version 1.1.0
In this package, we provide sources to load and query RDF datasets with RDFHive. We also present a simple test-suite based on the popular RDF/SPARQL benchmark: LUBM. For space reasons, this dataset only contains a few hundred of thousand RDF triples.
git clone github.com/tyrex-team/rdfhive.git ;
cd rdfhive/ ;
RDFHive can only load RDF data written according to the N-Triples format. This file has to be uploaded first on the HDFS.
hadoop fs -copyFromLocal local_file.nt hdfs_file.nt ;
bash bin/load.sh dbName hdfs_file.nt ;
To execute a SPARQL query over a loaded RDF dataset, RDFHive first
translates it into SQL and then evaluates the generated query. If
--debug is specified, RDFHive will be more verbose.
bash bin/eval.sh dbName LocalQueryFile ;
An already created database can also be removed.
bash bin/remove.sh dbName ;
Finally, a very basic test suite is included in this repository to demonstrate RDFHive.
cd tests/ ;
bash preliminaries.sh ;
bash run-benchmarks.sh ;
bash clean-all.sh ;
Moreover, two scripts are also part of the project (in bin/):
lubmqueries.sh and watdivqueries.sh which already contain
translation of LUBM and
WatDiv SPARQL queries.
digit := [1-9]
alphanum := [a-z|A-Z|1-9]
prefix := PREFIX (alphanum)*: <(alphanum)*>
var := ("?"|"$")(alphanum)*
tp := (var|(alphanum)*) (var|(alphanum)*) (var|(alphanum)*)
selectQuery :=
(prefix)*
SELECT (REDUCED|DISTINCT)? ("*"|(var)+)
WHERE { (tp) (" . "tp)* }
(LIMIT (digit)*)? (OFFSET (digit)*)?
This project is under the CeCILL license.
Damien Graux
[email protected]
Pierre Genevès
Nabil Layaïda
Tyrex Team, Inria (France), 2016