Running qlever-ui behind Traefik

[tobiasschweizer]

Hi,

I am trying to run qlever-ui behind Traefik in a docker compose setup (got inspired by #99).

I have an issue when loading the JS files. Somehow, the app seems to serve HTML files instead of JS files, resulting in parsing errors such as:

Uncaught SyntaxError: Unexpected token '<' (at helper.js:1:1)
raphael.js:1 Uncaught SyntaxError: Unexpected token '<' (at raphael.js:1:1)
treant.js:1 Uncaught SyntaxError: Unexpected token '<' (at treant.js:1:1)
qleverUI.js:1 Uncaught SyntaxError: Unexpected token '<' (at qleverUI.js:1:1)
codemirror.js:1 Uncaught SyntaxError: Unexpected token '<' (at codemirror.js:1:1)
autocomplete.js:1 Uncaught SyntaxError: Unexpected token '<' (at autocomplete.js:1:1)
active-line.js:1 Uncaught SyntaxError: Unexpected token '<' (at active-line.js:1:1)
show-hint.js:1 Uncaught SyntaxError: Unexpected token '<' (at show-hint.js:1:1)
search.js:1 Uncaught SyntaxError: Unexpected token '<' (at search.js:1:1)
sparql.js:1 Uncaught SyntaxError: Unexpected token '<' (at sparql.js:1:1)
sparql-hint.js:1 Uncaught SyntaxError: Unexpected token '<' (at sparql-hint.js:1:1)
index.js:1 Failed to load module script: Expected a JavaScript-or-Wasm module script but the server responded with a MIME type of "text/html". Strict MIME type checking is enforced for module scripts per HTML spec.
keywordSearch.js:1 Uncaught SyntaxError: Unexpected token '<' (at keywordSearch.js:1:1)

I think this related to some settings regarding static files.
AI says that there is something wrong with https://github.com/qlever-dev/qlever-ui/blob/4e3d057b9214f4f2fec868532c6291bbb15ec78f/backend/urls.py and suggests to exclude static:

from django.urls import include, path, re_path
from rest_framework import routers
from rest_framework.generics import ListAPIView
from . import views

urlpatterns = [
    path(
        "api/backends/",
        views.BackendList.as_view(),
        name="backend-list",
    ),
    path(
        "api/backends/<slug:slug>/",
        views.BackendViewSet.as_view({"get": "retrieve"}),
        name="backend-detail",
    ),
    re_path(r"^api/share$", views.shareLink, name="shareLink"),
    re_path(
        r"^api/warmup/(?P<backend>[^/]+)/(?P<target>[a-zA-Z0-9_-]+)$",
        views.warmup,
        name="warmup",
    ),
    re_path(
        r"^api/examples/(?P<backend>[^/]+)$", views.examples, name="examples"
    ),
    re_path(
        r"^api/prefixes/(?P<backend>[^/]+)$", views.prefixes, name="prefixes"
    ),
    re_path(r"^api/config/(?P<backend>[^/]+)$", views.config, name="config"),
    
    # Catch-all for index - MUST BE LAST and exclude static/media/admin/api
    re_path(
        r"^(?!static/)(?!media/)(?!admin/)(?!api/)(?P<backend>[A-Za-z0-9\+@:()%\-_]*)(/(?P<short>[A-Za-z0-9]{6})?)?$",
        views.index,
        name="index",
    ),
]

The key change is adding (?!static/)(?!media/)(?!admin/)(?!api/) at the beginning, which are negative lookaheads that say "don't match if the path starts with static/, media/, admin/, or api/".

I tried this but it did not solve the issue. Any hints? Thanks!

[valentinedwv]

basic docker stack below

### basic services and a single container for qlver
# PROJECT MUST BE A DNS COMPLAINT NAME. NO UNDER_SCORES
# Environment variables need to be set in a .env file:
version: '3.9'
networks:
  qlever_network:
      name: qlever_network_${QLEVER_NET:-base}
      external: true
  traefik_proxy:
      name: traefik_proxy
      external: true
#export USER_ID=$(id -u)
#export GROUP_ID=$(id -g)
volumes:
  qleverflow_data:
    name: qleverflow_data_${PROJECT:-geocodesexamples}
    #name: qlever

configs:
  qlever_config:
    name: qlever-config-${QLEVER_CONFIG:-geocodesexamples}
    #file: /Users/valentin/development/dev_earthcube/qleverflow/catalogues/data-geocodesexamples/Qleverfile.geocodesexamples
    external: true
services:
# PROJECT MUST BE A DNS COMPLAINT NAME. NO UNDER_SCORES
 qlever-server-base:
    image: docker.io/adfreiburg/qlever:latest
    hostname: qlever-server-${PROJECT:-geocodesexamples}
    user: "root:root"  # Will be set via .env file
    #user: "1011:1011"
    working_dir: /data
#    ports:
#      - "7019:7019"
    environment:
      - PATH="/qlever:/qlever/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
      - LANG="C.UTF-8"
      - LC_ALL="C.UTF-8"
      - LC_CTYPE="C.UTF-8"
      - DEBIAN_FRONTEND="noninteractive"
      - QLEVER_PROFILE="/etc/profile.d/qlever.sh"
      - QLEVER_ARGCOMPLETE_ENABLED="1"
      - QLEVER_IS_RUNNING_IN_CONTAINER="1"
    volumes:
      - /etc/localtime:/etc/localtime:ro
     # - /home/fils/scratch/qleverflow/configs/marineregions:/data
      # - type: volume
      #   source: qleverflow_data
      #   target: /data
      #   o: uid=1000,gid=1000
      #- qleverflow_data:/data

    configs:
      - source: qlever_config
        target: /data/Qleverfile
    entrypoint: /qlever/docker-entrypoint.sh
    command: >
               ' ls -l && qlever get-data  --log-level DEBUG && qlever index --text-index from_text_records_and_literals && qlever start --use-text-index yes --run-in-foreground '

    # run in forground, otherwise docker thinks run is complete.
    # 'qlever setup-config ${PROJECT:-geocodesexamples} && qlever get-data && qlever index &&  qlever start'
    # -c 'ServerMain -i ${PROJECT:-geocodesexamples} -j 8 -p 7019 -m 25G -c 20G -e 1G -k 200 -s 240s'
    #restart: unless-stopped
    networks: &network
      - qlever_network
      - traefik_proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.qlever-base-${PROJECT:-geocodesexamples}.entrypoints=http"
      - "traefik.http.routers.qlever-base-${PROJECT:-geocodesexamples}.rule=Host(`qlever.${HOST}`) && PathPrefix(`/graphspace/${PROJECT}`)"
      - "traefik.http.middlewares.qlever-base-${PROJECT:-geocodesexamples}-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.qlever-base-${PROJECT:-geocodesexamples}.middlewares=qlever-base-${PROJECT:-geocodesexamples}-https-redirect"
      - "traefik.http.routers.qlever-base-${PROJECT:-geocodesexamples}-secure.entrypoints=https"
      - "traefik.http.routers.qlever-base-${PROJECT:-geocodesexamples}-secure.rule=Host(`qlever.${HOST}`) && PathPrefix(`/graphspace/${PROJECT}`)"
      - "traefik.http.routers.qlever-base-${PROJECT:-geocodesexamples}-secure.tls=true"
      - "traefik.http.routers.qlever-base-${PROJECT:-geocodesexamples}-secure.tls.certresolver=httpresolver"
      - "traefik.http.routers.qlever-base-${PROJECT:-geocodesexamples}-secure.service=qlever-base-${PROJECT:-geocodesexamples}"
      - "traefik.http.services.qlever-base-${PROJECT:-geocodesexamples}.loadbalancer.server.port=7019"
      - "traefik.docker.network=traefik_proxy"


 qlever-ui-base:
    image: docker.io/adfreiburg/qlever-ui:latest
    hostname: qlever-ui-${PROJECT:-geocodesexamples}
    #init: true
    volumes:
      - type: volume
        source: qleverflow_data
        target: /app/db/
    #   - qleverflow_data:/app/db
    # ports:
    #   - "8176:7000"
    environment:
      # this will hopefully be a hostname from the above service
      QLEVER_HOST: "http://qlever-server-${PROJECT:-geocodesexamples}:7007"
      QLEVERUI_ALLOWED_HOSTS: "*"
      QLEVERUI_CSRF_TRUSTED_ORIGINS: ${QLEVERUI_CSRF_TRUSTED_ORIGINS:-https://*.geocodes-aws-dev.earthcube.org}
      QLEVERUI_SECRET_KEY: ${QLEVERUI_SECRET_KEY:-!!super_secret!!}
      QLEVERUI_DATABASE_URL: "sqlite:////app/db/qleverui.sqlite3"
#      QLEVER_HOST: "http://0.0.0.0:7007"
   #    entrypoint: bash
    command:
      - gunicorn
      - "--bind"
      - ":7000"
      - "--workers"
      - "3"
      - "--limit-request-line"
      - "10000"
      - "qlever.wsgi:application"
    #restart: unless-stopped
    networks: *network
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.qlever-ui-${PROJECT:-geocodesexamples}.entrypoints=http"
      - "traefik.http.routers.qlever-ui-${PROJECT:-geocodesexamples}.rule=Host(`qlever-ui.${HOST}`)"
      - "traefik.http.middlewares.qlever-ui-${PROJECT:-geocodesexamples}-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.qlever-ui-${PROJECT:-geocodesexamples}.middlewares=qlever-ui-${PROJECT:-geocodesexamples}-https-redirect"
      - "traefik.http.routers.qlever-ui-${PROJECT:-geocodesexamples}-secure.entrypoints=https"
      - "traefik.http.routers.qlever-ui-${PROJECT:-geocodesexamples}-secure.rule=Host(`qlever-ui.${HOST}`)"
      - "traefik.http.routers.qlever-ui-${PROJECT:-geocodesexamples}-secure.tls=true"
      - "traefik.http.routers.qlever-ui-${PROJECT:-geocodesexamples}-secure.tls.certresolver=httpresolver"
      - "traefik.http.routers.qlever-ui-${PROJECT:-geocodesexamples}-secure.service=qlever-ui-${PROJECT:-geocodesexamples}"
      - "traefik.http.services.qlever-ui-${PROJECT:-geocodesexamples}.loadbalancer.server.port=7000"
      - "traefik.docker.network=traefik_proxy"

 qlever-petrimaps-base:
    image: docker.io/adfreiburg/qlever-petrimaps:latest
    hostname: qlever-petrimaps
    # ports:
    #   - "9090:9090"
    networks: *network
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.qlever-petrimaps-${PROJECT:-geocodesexamples}.entrypoints=http"
      - "traefik.http.routers.qlever-petrimaps-${PROJECT:-geocodesexamples}.rule=Host(`qlever-petrimaps.${HOST}`)"
      - "traefik.http.middlewares.qlever-petrimaps-${PROJECT:-geocodesexamples}-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.qlever-petrimaps-${PROJECT:-geocodesexamples}.middlewares=qlever-petrimaps-${PROJECT:-geocodesexamples}-https-redirect"
      - "traefik.http.routers.qlever-petrimaps-${PROJECT:-geocodesexamples}-secure.entrypoints=https"
      - "traefik.http.routers.qlever-petrimaps-${PROJECT:-geocodesexamples}-secure.rule=Host(`qlever-petrimaps.${HOST}`)"
      - "traefik.http.routers.qlever-petrimaps-${PROJECT:-geocodesexamples}-secure.tls=true"
      - "traefik.http.routers.qlever-petrimaps-${PROJECT:-geocodesexamples}-secure.tls.certresolver=httpresolver"
      - "traefik.http.routers.qlever-petrimaps-${PROJECT:-geocodesexamples}-secure.service=qlever-petrimaps-${PROJECT:-geocodesexamples}"
      - "traefik.http.services.qlever-petrimaps-${PROJECT:-geocodesexamples}.loadbalancer.server.port=9090"
      - "traefik.docker.network=traefik_proxy"

[tobiasschweizer]

@valentinedwv Thanks for this!

I think I have now figured out what the problem is.

The UI URL is this: https://myqleverui.localhost/qleverui/default

But it requests the static files from this URL: https://myqleverui.localhost/static/js/helper.js

The problem is that the prefix qleverui is missing (it should be https://myqleverui.localhost/qleverui/static/js/helper.js). So is there a way to instruct the UI to use the prefix when loading those files (base path)?

Looking at your setup, I tried setting QLEVER_HOST: "https://myqleverui.localhost/qleverui". Do I have to include that env var somewhere in the Python entrypoint?

This is how my entry file looks like:

#!/bin/sh

cp /app/db/qleverui.sqlite3 /app/db/olympics.ui-db.sqlite3

python manage.py configure default "${QLEVER_SERVER_ENDPOINT}"

gunicorn \
  --bind ":${QLEVER_UI_PORT}" \
  --workers 1 \
  --limit-request-line 10000 \
  qlever.wsgi:application

[valentinedwv]

@valentinedwv Thanks for this!

I think I have now figured out what the problem is.

The UI URL is this: https://myqleverui.localhost/qleverui/default

But it requests the static files from this URL: https://myqleverui.localhost/static/js/helper.js

The problem is that the prefix qleverui is missing (it should be https://myqleverui.localhost/qleverui/static/js/helper.js). So is there a way to instruct the UI to use the prefix when loading those files (base path)?

it's a django app, so try setting a env that gets passed to the qleverui:
https://stackoverflow.com/questions/17420478/how-to-set-base-url-in-django

[tobiasschweizer]

Ok, I tried this:

I map an adapted version of /app/qlever/urls.py in the container:

from django.urls import include, re_path, path
from django.contrib import admin
from django.contrib.staticfiles.urls import staticfiles_urlpatterns

urlpatterns = [
    re_path(r"qleverui/", include([
        re_path(r"^admin/", admin.site.urls),
        re_path(r"^", include("backend.urls")),
        ])
    )
]
urlpatterns += staticfiles_urlpatterns()

The idea is to set the prefix qleverui for all paths.

This is also documented here:

We can improve this by stating the common path prefix only once and grouping the suffixes that differ:

https://docs.djangoproject.com/en/5.1/topics/http/urls/

And it seems similar to https://stackoverflow.com/questions/17420478/how-to-set-base-url-in-django

However, now I get:

Page not found (404)
Request Method: GET
Request URL: http://myqleverui.localhost/default
Using the URLconf defined in qlever.urls, Django tried these URL patterns, in this order:

^qleverui/
^static/(?P.*)$
The current path, default, didn’t match any of these.

An issue might be that traefik strips the prefix and it never arrives at Django.

traefik.http.middlewares.qleverui-sec-stripprefix.stripprefix.prefixes: /qleverui

But at the same time the browser should request the static files using the prefix. So now I am wondering whether the template would have to be changed.

qlever-ui/backend/templates/partials/head.html

Line 83 in 4e3d057

<script src="{% static "js/helper.js" %}"></script>

The idea would be to prepend the prefix to /static/helper.js etc. so the browser fetches the files from https://myqleverui.localhost/qleverui/static/js/helper.js and not https://myqleverui.localhost/static/js/helper.js. https://myqleverui.localhost/ is another app and serves a static HTML page. Any ideas?

[tobiasschweizer]

I made some progress changing the config for static, however I have two remaining problems:

• wasm/formatter/index.js contains: function wasmModule(imports){return _loadWasmModule(0, '/static/wasm/formatter/11b51eb4b0b0cb77.wasm', null, imports)} -> seems to come from formatter/index.js
• /api/share

These two URLs are still starting from /static without the prefix. Where could this be changed?

@hannahbast If I get this to work, should I try to submit a PR? I actually made changes to /app/qlever/urls.py and /app/qlever/settings.py and added a /app/qlever/middleware.py:

class ScriptNameMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        script_name = request.META.get('HTTP_X_SCRIPT_NAME', '')
        if script_name:
            request.META['SCRIPT_NAME'] = script_name
        return self.get_response(request)

I think middleware.py would not be part of a PR but the adaptions to urls.py and settings.py would ideally be config options.

[tobiasschweizer]

Ok, I am getting even closer. I manually hacked

./staticfiles/wasm/formatter/index.js:function wasmModule(imports){return _loadWasmModule(0, '/static/wasm/formatter/11b51eb4b0b0cb77.wasm', null, imports)}

So now it says "qleverui/static/wasm/formatter/11b51eb4b0b0cb77.wasm" and now the file is loaded correctly.

The other thing that does not work is the call to api/share

qlever-ui/backend/static/js/qleverUI.js

Line 252 in 4e3d057

const response = await fetch("/api/share", {

Maybe that would need the static template tag too?

I'd really like to make a PR so these things ca be made configurable in a clean way but I need some guidance.

[hannahbast]

@tobiasschweizer We are in the process of switching to a new, completely refactored UI. I have just added you to the repository, which is currently still private. Feel free to try it out and talk with @IoannisNezis