You can star this work, if you find it interesting or want to come back.
Star

Abstract

This document describe a method to execute the SPARQL test suite (of w3c/rdf-tests) with in input several docker's instances of SPARQL services. This document is also a demonstrator uses SPARQL queries directly in the HTML.

Introduction

This document is a draft (work in progress) to visualize the actual interoperability between several editors of SPARQL service (another version to look these results with more details : SPARQLScore). It uses a new method to execute the SPARQL test suite in the cloud (reproducible free).

Already, this document describes several scripts to execute a part of SPARQL (core) test suite (of w3c/rdf-tests) with docker's instances of RDF databases. These scripts can be used locally or in the cloud (via GitHub and the continuous integration service of Travis CI).

This document is also a demonstrator : all tables in this HTML document are the results of SPARQL queries (look the code). This document is a demonstrator and a draft to visualize the actual interoperability between several editors of SPARQL service. (another version to look these results with more details : SPARQLScore).

Tests, tools and softwares used

This document is a demonstrator for the W3C working groups and uses :

To execute the tests of W3C in this document, we use : Only a part of tests of W3C are useable by TFT : Thanks at editors of RDF databases who made this work possible : Thanks at services who host/execute this project : Thanks at the University Paris-Saclay that hosts the SPARQL service to share the results of these tests.

How to run the Test Cases

How to run the Test Cases in local with Docker
This script calculates an indicator of interoperability for SPARQL 1.1 with the TFT framework (Test-For-Triplestore). Here, it is applied to the RDF Blazegraph database. These steps are: (1) upload the Docker images and (2) compile them, (3) deploy five nodes of Linked Data locally, (4) install TFT, (5) load the tests, (6) run the tests by saving the results and (7) calculate the interoperability indicator. 
# (1) Upload the Docker images
docker pull bordercloud/tft-jena-fuseki
docker pull bordercloud/tft-virtuoso7-stable

# (2) Compile the Docker images of SPARQL service to test
docker build -t tft-stardog .

# (3) Deploy a Linked Data locally
# 172.17.0.2
docker run --privileged --name instance.tft-stardog -h tft-stardog -d tft-stardog
# 172.17.0.3
docker run --privileged --name instance.tft.example.org \
           -h example.org -d bordercloud/tft-virtuoso7-stable
# 172.17.0.4
docker run --privileged --name instance.tft.example1.org \
           -h example1.org -d bordercloud/tft-virtuoso7-stable
# 172.17.0.5
docker run --privileged --name instance.tft.example2.org \
           -h example2.org -d bordercloud/tft-virtuoso7-stable
# 172.17.0.6 local database to save tests and results
docker run --privileged --name instance.tft_database -d tft-jena-fuseki

# (4) Install TFT
git clone --recursive https://github.com/BorderCloud/TFT.git
cd TFT
# Install the SPARQL client of TFT
composer install
wget http://mirrors.standaloneinstaller.com/apache//jmeter/binaries/apache-jmeter-4.0.tgz
tar xvzf apache-jmeter-4.0.tgz
mv  apache-jmeter-4.0 jmeter
rm apache-jmeter-4.0.tgz

# (5) Load all the tests
php ./tft-testsuite -a -t fuseki -q http://172.17.0.6:8080/test/query \
                    -u http://172.17.0.6:8080/test/update

# (6) Execute these tests
php ./tft -t fuseki -q http://172.17.0.6:8080/test/query \
                    -u http://172.17.0.6:8080/test/update \
          -tt fuseki -te http://172.17.0.2/blazegraph/namespace/test/sparql \
          -r http://example.org/buildid   \
          -o ./junit  \
          --softwareName="Jena" \
          --softwareDescribeTag=X.X.X \
          --softwareDescribe="Name"

# (7) Calculate a score/indicator of interoperability
php ./tft-score -t fuseki -q http://172.17.0.6:8080/test/query \
                          -u http://172.17.0.6:8080/test/update \
                -r  http://example.org/buildid
How to run the Test Cases in GitHub via Travis CI
https://github.com/BorderCloud/tft-jena-fuseki/blob/master/.travis.yml 
sudo: required
# notifications:
#   email:
#     recipients:
#      - yourEmail@example.com
#     on_success: always # default: change (other : never)
#     on_failure: always # default: always

env:
  global:
    - URI="https://travis-ci.org/BorderCloud/tft-blazegraph/builds/$((TRAVIS_JOB_ID - 1))"
    - NAME="Blazegraph"
    - VERSION="v2.2.0"
#    - SPARQLSCORE_DATABASE="134.158.74.247"
    - SPARQLSCORE_DATABASE="172.17.0.6:8080" #local

language: php
php:
  - '7.1'

services:
  - docker

before_install:
# Download docker's images
  - docker pull bordercloud/tft-jena-fuseki
  - docker pull bordercloud/tft-virtuoso7-stable

# Compile the docker's project
  - docker build -t tft-blazegraph .

script:
# Deploy TFT

# 172.17.0.2
  -   docker run --privileged --name instance.tft-blazegraph -h tft-blazegraph -d tft-blazegraph
# 172.17.0.3
  -   docker run --privileged --name instance.tft.example.org -h example.org -d bordercloud/tft-virtuoso7-stable
# 172.17.0.4
  -   docker run --privileged --name instance.tft.example1.org -h example1.org -d bordercloud/tft-virtuoso7-stable
# 172.17.0.5
  -   docker run --privileged --name instance.tft.example2.org -h example2.org -d bordercloud/tft-virtuoso7-stable

# Run RDF database in order to save tests and results in local (decomment these line and change the variable SPARQLSCORE_DATABASE
## 172.17.0.6 for local
  -   docker run --privileged --name instance.tft_database -d bordercloud/tft-jena-fuseki

# Wait...
  -   sleep 30
#  -   docker network inspect bridge

  -   git clone --recursive https://github.com/BorderCloud/TFT.git
  -   cd TFT
# install SPARQL client
  -   composer install
# install JMeter
  - wget http://mirrors.standaloneinstaller.com/apache//jmeter/binaries/apache-jmeter-4.0.tgz
  - tar xzf apache-jmeter-4.0.tgz
  - mv  apache-jmeter-4.0 jmeter
  - rm apache-jmeter-4.0.tgz

  -   php ./tft-testsuite -a -t fuseki -q http://${SPARQLSCORE_DATABASE}/test/query -u http://${SPARQLSCORE_DATABASE}/test/update
  -   php ./tft -t fuseki -q http://${SPARQLSCORE_DATABASE}/test/query -u http://${SPARQLSCORE_DATABASE}/test/update -tt fuseki -te http://172.17.0.2/blazegraph/namespace/test/sparql -r ${URI} -o ./junit --softwareName="${NAME}" --softwareDescribeTag=${VERSION}  --softwareDescribe="${TRAVIS_COMMIT}"
  -   php ./tft-score -t fuseki -q http://${SPARQLSCORE_DATABASE}/test/query -u http://${SPARQLSCORE_DATABASE}/test/update -r  ${URI}

after_success:
#  - if [ "$TRAVIS_BRANCH" == "master" ]; then
#    docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD ;
#    docker push bordercloud/tft-blazegraph;
#    fi
Share the report
In the Travis script, you can replace the local IP by the IP of a SPARQL service online, such as, SPARQLScore (134.158.74.247).

Reports

This section summarizes the outcomes of last test reports of SPARQL implementation in docker images hosted by GitHub. List of Docker images used:

Implementation by specification

Implementation by software

Tests

Tests by Specifications

SPARQL 1.1 Protocol

SPARQL 1.1 Query Language

SPARQL 1.1 Update

SPARQL 1.1 Query Results CSV and TSV Formats

SPARQL 1.1 Query Results JSON Format

SPARQL 1.1 Federation Extensions

SPARQL 1.1 Entailment Regimes

SPARQL 1.1 Service Description

There are not tests. todo ?