You can star this work, if you find it interesting or want to come back.
Star

Abstract

This document describe a method to execute the SPARQL test suite (of w3c/rdf-tests) with in input several docker's instances of SPARQL services. This document is also a demonstrator uses SPARQL queries directly in the HTML.

Introduction

This document is a draft to visualize the actual interoperability between several editors of SPARQL service (another version to look these results with more details : SPARQLScore). It uses a new method to execute the SPARQL test suite in the cloud (reproducible free).

Already, this document describes several scripts to execute a part of SPARQL (core) test suite (of w3c/rdf-tests) with docker's instances of RDF databases. These scripts can be used locally or in the cloud (via GitHub and the continuous integration service of Travis CI).

This document is also a demonstrator : all tables in this HTML document are the results of SPARQL queries (look the code). This document is a demonstrator and a draft to visualize the actual interoperability between several editors of SPARQL service. (another version to look these results with more details : SPARQLScore).

Tests, tools and softwares used

This document is a demonstrator for the W3C working groups and uses :

To execute the tests of W3C in this document, we use : Only a part of tests of W3C are useable by TFT : Thanks at editors of RDF databases who made this work possible : Thanks at services who host/execute this project : Thanks at the University Paris-Saclay that hosts the SPARQL service to share the results of these tests.

How to run the Test Cases

How to run the Test Cases in local with Docker
This script calculates an indicator of interoperability for SPARQL 1.1 with the TFT framework (Test-For-Triplestore). Here, it is applied to the RDF Jena-Fuseki database. These steps are: (1) compile the Docker image, (2) deploy five nodes of Linked Data locally, (3) install TFT with JMeter, (4) load the tests, (5) run the tests by saving the results and (6) calculate the interoperability indicator. 
# (1) Compile the Docker images of SPARQL service to test
docker build -t tft-jena-fuseki .

# (2) Deploy all SPARQL services with docker-compose
docker-compose up -d

# (3) Install TFT
git clone --recursive https://github.com/BorderCloud/TFT.git
cd TFT
# Install the SPARQL client of TFT
composer install
# Install Jmeter for testing the minimal protocol
wget http://mirrors.standaloneinstaller.com/apache//jmeter/binaries/apache-jmeter-5.4.1.tgz
tar xvzf apache-jmeter-5.4.1.tgz
mv  apache-jmeter-5.4.1 jmeter
rm apache-jmeter-5.4.1.tgz

# (4) Load all the tests
php ./tft-testsuite -a -t fuseki -q http://172.18.0.6:8080/test/query \
                    -u http://172.18.0.6:8080/test/update

# (5) Execute these tests
php ./tft -t fuseki -q http://172.18.0.6:8080/test/query \
                    -u http://172.18.0.6:8080/test/update \
          -tt fuseki -te http://172.18.0.2/sparql \
          -r http://example.org/buildid   \
          -o ./junit  \
          --softwareName="Jena" \
          --softwareDescribeTag=X.X.X \
          --softwareDescribe="Name"

# (6) Calculate a score/indicator of interoperability
php ./tft-score -t fuseki -q http://172.17.0.6:8080/test/query \
                          -u http://172.17.0.6:8080/test/update \
                -r  http://example.org/buildid
How to run the Test Cases in GitHub
GitHub Actions
With this file, your can execute manually the Test Cases directly in GitHub. https://github.com/BorderCloud/tft-jena-fuseki/blob/master/.github/workflows/test.yml 
name: Test

on: workflow_dispatch

env:
    NAME: "Jena Fuseki"
    SOFTWARE_DESCRIBE: "Apache Jena Fuseki is a RDF storage with a SPARQL server"
    VERSION: "4.1.0"
    SPARQLSCORE_DATABASE: "134.158.74.239"
#   SPARQLSCORE_DATABASE= "172.18.0.6:8080" #local
    URI: ${{github.server_url}}/${{github.repository}}/actions/runs/${{github.run_id}}       
    COMMIT: ${{github.sha}}
    
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Setup PHP
        uses: shivammathur/setup-php@v2
        with:
            php-version: '7.4'
            tools: composer
      - name: Setup Java
        uses: actions/setup-java@v2
        with:
            distribution: 'adopt'
            java-version: '8'
      - name: Build image docker
        run: >-
            docker build -t tft-jena-fuseki .
      - name: Start servers
        run:  |
            sudo docker-compose up -d
            sleep 30s
      - name: Setup TFT with Jmeter
        run: |
            git clone --recursive https://github.com/BorderCloud/TFT.git
            cd ./TFT
            composer install
            wget http://mirrors.standaloneinstaller.com/apache//jmeter/binaries/apache-jmeter-5.4.1.tgz
            tar xvzf apache-jmeter-5.4.1.tgz
            mv  apache-jmeter-5.4.1 jmeter
            rm apache-jmeter-5.4.1.tgz
            php -version
      - name: Import tests
        run: |
            cd ./TFT
            php ./tft-testsuite -a -t fuseki -q http://${SPARQLSCORE_DATABASE}/test/query -u http://${SPARQLSCORE_DATABASE}/test/update
      - name: Execute tests
        run: |
            cd ./TFT
            php ./tft -t fuseki -q http://${SPARQLSCORE_DATABASE}/test/query -u http://${SPARQLSCORE_DATABASE}/test/update -tt fuseki -te http://172.18.0.2/sparql -r ${URI} -o ./junit --softwareName="${NAME}"  --softwareDescribeTag=${VERSION}  --softwareDescribe="${COMMIT}"
      - name: Read result
        run: |
            cd ./TFT
            php ./tft-score -t fuseki -q http://${SPARQLSCORE_DATABASE}/test/query -u http://${SPARQLSCORE_DATABASE}/test/update -r  ${URI}
      - name: Stop servers
        run:  |
          docker-compose stop
Travis CI
It's also possible to use the cloud of Travis CI. https://github.com/BorderCloud/tft-jena-fuseki/blob/master/.travis.yml 
sudo: required

language: php
php:
  -  7.4.15

notifications:
  email:
    recipients:
      - karima.rafes@gmail.com
    on_success: always # default: change (other : never)
    on_failure: always # default: always

env:
  global:
    - URI="https://travis-ci.org/BorderCloud/tft-jena-fuseki/builds/$((TRAVIS_JOB_ID - 1))"
    - NAME="Jena Fuseki"
    - SOFTWARE_DESCRIBE="Apache Jena Fuseki is a RDF storage with a SPARQL server."
    - VERSION="4.0.0"
    - SPARQLSCORE_DATABASE="172.18.0.6:8080" #local

services:
  - docker

before_install:
# Download docker's images
  - docker pull bordercloud/tft-virtuoso7-stable
#  - docker pull bordercloud/tft-jena-fuseki

# Compile the docker's project
  - docker build -t tft-jena-fuseki .
  - sudo docker-compose up -d

script:
  - sleep 30s
  - git clone --recursive https://github.com/BorderCloud/TFT.git
  - cd TFT
# install SPARQL client
  - composer install
# install JMeter
  - wget http://mirrors.standaloneinstaller.com/apache//jmeter/binaries/apache-jmeter-5.4.1.tgz
  - tar xvzf apache-jmeter-5.4.1.tgz
  - mv  apache-jmeter-5.4.1 jmeter
  - rm apache-jmeter-5.4.1.tgz

  -   php ./tft-testsuite -a -t fuseki -q http://${SPARQLSCORE_DATABASE}/test/query -u http://${SPARQLSCORE_DATABASE}/test/update
  -   php ./tft -t fuseki -q http://${SPARQLSCORE_DATABASE}/test/query -u http://${SPARQLSCORE_DATABASE}/test/update -tt fuseki -te http://172.18.0.2/sparql -r ${URI} -o ./junit --softwareName="${NAME}"  --softwareDescribeTag=${VERSION}  --softwareDescribe="${TRAVIS_COMMIT}"
  -   php ./tft-score -t fuseki -q http://${SPARQLSCORE_DATABASE}/test/query -u http://${SPARQLSCORE_DATABASE}/test/update -r  ${URI}

  -   docker-compose stop

# Save the Docker image
after_success:
   - if [ "$TRAVIS_BRANCH" == "master" ]; then
     echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin;
     docker tag tft-jena-fuseki bordercloud/tft-jena-fuseki:latest ;
     docker tag tft-jena-fuseki bordercloud/tft-jena-fuseki:${VERSION} ;
     docker push bordercloud/tft-jena-fuseki ;
     fi
Share the report
In the script, you can replace the local IP by the IP of a SPARQL service online, such as, SPARQLScore (134.158.74.239).

Reports

This section summarizes the outcomes of last test reports of SPARQL implementation in docker images hosted by GitHub. List of Docker images used:

Implementation by specification

Implementation by software

Tests

Tests by Specifications

SPARQL 1.1 Protocol

SPARQL 1.1 Query Language

SPARQL 1.1 Update

SPARQL 1.1 Query Results CSV and TSV Formats

SPARQL 1.1 Query Results JSON Format

SPARQL 1.1 Federation Extensions

SPARQL 1.1 Entailment Regimes

TFT cannot test correctly the tests about "Entailment Regimes" (see details).

SPARQL 1.1 Service Description

There are not tests. todo ?