Issue with reverseProxy traefik

[toxic0berliner]

I have a nice setup running jellyfin on docker with traefik as reverse proxy, it works extremely well in any browser, but kodi plugin has a lot of issues.
To be clear, when I setup the kodi plugin to use server "jellyfin.lan:8096" everything works perfectly.
But when I set the server to "jellyfin.example.com" it has a lot of issues :

• thumbs are not loaded
• playback in plugin mode takes ages to start and almost always crashes when stoping playback
• playback in direct-path mode works (I setup smb path properly) but stoping the playback crashed kodi as well.

I have a valid letsencrypt certificate for jellyfin.example.com, http always redirects to https, and as explained, it works very very wel in a browser.

I have tried a lot of things but cannot find the issue... I started kodi both n my mibox3, but also on my windows10 machine and same in both places.
I suspect jellyfin plugin could either refuse to follow http to https redirects, or it does not use SNI when making requests and traefik will server the default cert which is invalid... I have even tried to disbale "verify connection" in the settings but it does not help...

Any help is appreciated.

Here is my compose.yml to deploy jellyfin : I left the comments for you to see what I have tried but unsucessfully each time)

version: "3"
services:
  jellyfin:
    image: ghcr.io/linuxserver/jellyfin
    container_name: jellyfin
    hostname: jellyfin
    # privileged: true
    environment:
      - PUID=1026
      - PGID=100
      - TZ=Europe/Paris
      - UMASK=022
      - JELLYFIN_PublishedServerUrl=https://jellyfin.example.com/
    labels:
      traefik.http.services.jellyfin.loadBalancer.server.port: 8096
      subdomain: jellyfin
      traefik.http.routers.jellyfin.middlewares: jellyfin-mw
      traefik.http.middlewares.jellyfin-mw.headers.customResponseHeaders.X-Robots-Tag: noindex,nofollow,nosnippet,noarchive,notranslate,noimageindex
      traefik.http.middlewares.jellyfin-mw.headers.customrequestheaders.X-Forwarded-Proto: https
      traefik.http.middlewares.jellyfin-mw.headers.customrequestheaders.X-Forwarded-Protocol: https
      traefik.http.middlewares.jellyfin-mw.headers.customrequestheaders.X-Forwarded-Host: jellyfin.example.com
      traefik.http.middlewares.jellyfin-mw.headers.SSLRedirect: true
      traefik.http.middlewares.jellyfin-mw.headers.SSLHost: jellyfin.example.com
      traefik.http.middlewares.jellyfin-mw.headers.SSLForceHost: true
      traefik.http.middlewares.jellyfin-mw.headers.STSSeconds: 315360000
      traefik.http.middlewares.jellyfin-mw.headers.STSIncludeSubdomains: true
      traefik.http.middlewares.jellyfin-mw.headers.STSPreload: true
      traefik.http.middlewares.jellyfin-mw.headers.forceSTSHeader: true
      traefik.http.middlewares.jellyfin-mw.headers.frameDeny: false
      traefik.http.middlewares.jellyfin-mw.headers.contentTypeNosniff: true
      traefik.http.middlewares.jellyfin-mw.headers.browserXSSFilter: true
      traefik.http.middlewares.jellyfin-mw.headers.customFrameOptionsValue: 'allow-from https://organizr.example.com'
      traefik.http.services.jellyfin.loadBalancer.passHostHeader: true
    volumes:
      - jellyfin:/notreallyused
      - /var/lib/docker/volumes/jellyfin/_data/config/:/config
      - /mnt/nas/video:/data/video
      - /mnt/nas/surveillance:/data/surveillance:ro
      - /mnt/nas/books:/data/books
      - /mnt/nas/downloads:/data/downloads
      - /mnt/nas/homes/toxic/Drive/Moments:/data/moments
      - /mnt/nas/music:/data/music
      - /mnt/nas/photo:/data/photo
    group_add:
      - 107
    ports:
      - 8096:8096
      #- 8920:8920 #optional
      - 7359:7359/udp #optional
      - 1900:1900/udp #optional
    devices:
      - /dev/dri:/dev/dri #optional
    restart: unless-stopped
    networks:
      traefiknet:
        aliases: 
          - jellyfin
#  jellyfixer:
#    image: quay.io/xsteadfastx/jellyfixer:latest
#    container_name: jellyfixer
#    hostname: jellyfixer
#    command: http://jellyfin:8096
#    networks:
#      traefiknet:
#        aliases: 
#          - jellyfixer
#    environment:
#      - JELLYFIXER_INTERNAL_URL=http://jellyfin:8096
#      - JELLYFIXER_EXTERNAL_URL=https://jellyfin.example.com
#    labels:
#      traefik.http.routers.jellyfixer.rule: Host(`jellyfin.foo.tld`) && Path(`/System/Info/Public`)
volumes:
  jellyfin: 
    external: true
networks:
  traefiknet:
    name: traefiknet
    driver: bridge
    external: true

Note : JELLYFIN_PublishedServerUrl I tried with or without https...

Here is what I get in https://jellyfin.example.com/System/Info/Public

{"LocalAddress":"https://jellyfin.example.com","ServerName":"jellyfin-andromeda","Version":"10.7.2","ProductName":"Jellyfin Server","OperatingSystem":"Linux","Id":"xxxxxxxxxxxxxxxxxxxxxxx","StartupWizardCompleted":true}

I even got hardware decoding/encoding working, web UI is really snappy to start/stop reading movies and tc shows, even big high-def ones...

I tried to run fiddler on my windows machine running kodi and inspecting https, when connecting to jellyfin.example.com I do the the redirect from http to https, but after that I see a TLS start that gets closed right away. I suspect it's due to fiddler having a cert not trusted by kodi despite beeing in the windows trusted store... When fiddler is disabled I do get connectivity, the library is synced, but as explained, no thumbs, and very buggy playback.

Again, thanks a lot for any idea on how to get this working.

[mcarlton00]

Check Kodi logs for any errors. I'm going through a nginx reverse proxy and have no issues. https://kodi.wiki/view/Log_file/Easy

Also, if you're switching server URL, make sure you do a full database reset from the addon menu and "Reset addon data". Changing it while media is already synced doesn't work currently.

[toxic0berliner]

I did the full reset.
Finally found some kodi logs that seem to be able to help : it's full of this:

2021-04-26 01:40:04.780 T:9160    ERROR <general>: Handshake Status 301
                                                   Traceback (most recent call last):
                                                     File "C:\Users\Etienne\AppData\Roaming\Kodi\addons\plugin.video.jellyfin\jellyfin_kodi\jellyfin\websocket.py", line 891, in run_forever
                                                       self.sock.connect(self.url, header=self.header)
                                                     File "C:\Users\Etienne\AppData\Roaming\Kodi\addons\plugin.video.jellyfin\jellyfin_kodi\jellyfin\websocket.py", line 483, in connect
                                                       self._handshake(hostname, port, resource, **options)
                                                     File "C:\Users\Etienne\AppData\Roaming\Kodi\addons\plugin.video.jellyfin\jellyfin_kodi\jellyfin\websocket.py", line 528, in _handshake
                                                       raise WebSocketException("Handshake Status %d" % status)
                                                   jellyfin.websocket.WebSocketException: Handshake Status 301

This looks to be an issue with websockets and if it gets a 301 status I expect it is beeing redirected to use https... But I have no difficulties with websockets on other docker containers (wetty for instance) behind the same traefik with the same automatic docker settings with labels...

Also, I got some messages like this:

2021-04-26 01:38:39.725 T:11488   ERROR <general>: CCurlFile::Stat - Failed: Timeout was reached(28) for https://jellyfin.example.com/Items/XXX/Images/Backdrop/0?Format=original&Tag=XXX

And when I take the very same url in my browser on the same windows machine I get an almost instant response showing me the picture...

Could it be that kodi does not recognize letsencrypt as a trusted cert ? Probably the log would be different..

I'm relly not seeing what is happening here.. Do you happen to know in in what way the playback in kodi is so different from the web ui of jellyfin ?

Any help is very appreciated. Thanks in advance !

Edit: I read here that some ws libraries just don't follow redirect and instead throw an error. Maybe you're not using the same lib as thee guys but could have the same issue... Hope someone can look into it and either fix by following the redirects, or maybe help me find some more logs to see where it is being redirected to (as of now I'm not even seeing where ws_client tries to connect... The server url is explicitely https://jellyfin.example.com so if it uses only https it should never see a 301 redirect anyway...)

Edit 2: I found where it's trying to connect :

2021-04-26 01:34:40.917 T:9160     INFO <general>: JELLYFIN.jellyfin.ws_client -> INFO::jellyfin_kodi\jellyfin\ws_client.py:50 Websocket url: wss://jellyfin.example.com/socket?api_key=XXXX&device_id=XXXX

And using this I have been able to confirm that the very url this log is showing answers me with a 101 code switching to websocket protocol, no 301 redirect. I'm not able to go further and chat with the websocket since I don't know it's language, but hey, that looks good to me ;) So no idea why this keeps crashing... Even enabling log level debug in jellyfin I'm not getting much more info, fails with status 301 but that's all it says, 301 usually gives a location to go to but jellyfin ws logic does not log it even in debug...

Edit 3: I looked at my traefik logs and I also see the 301 without much more info...

10.0.30.112 - - [26/Apr/2021:10:30:58 +0000] "GET /socket?api_key=XXX&device_id=XXX HTTP/1.1" 301 164 "-" "-" 22030 "websecure-jellyfin@docker" "-" 0ms
10.0.30.112 - - [26/Apr/2021:10:30:58 +0000] "GET /Sessions?DeviceId=XXX HTTP/1.1" 200 745 "-" "-" 22032 "websecure-jellyfin@docker" "http://172.19.0.21:8096" 1ms

Looking at the log-format of traefik:
<remote_IP_address> - <client_user_name_if_available> [<timestamp>] "<request_method> <request_path> <request_protocol>" <origin_server_HTTP_status> <origin_server_content_size> "<request_referrer>" "<request_user_agent>" <number_of_requests_received_since_Traefik_started> "<Traefik_router_name>" "<Traefik_server_URL>" <request_duration_in_ms>ms

I see that somehow the "<Traefik_server_URL>" is empty for the socket request where it it not for a standard GET...

So it might still be a traefik issue... Should I use a specific port for ws on jellyfin ? I do think traefik is perfectly capable of proxying a wss request into a ws backend and it works for many other containers I run with the same conf...

[toxic0berliner]

I tested out someone's traefik config and I'm still getting the same error, I agree that I might be able somewhere to avoid 301 redirect, but I also think it would make sense for Kodi plugin to simply follow 301 redirects for the wss and not simply fail with an error.
Any chance of getting that changed to handle 301 redirects in the wss query ?
Thanks in advance.

[mcarlton00]

A lot of things aren't adding up here. There's no configuration available for the websocket URL. It's based off the saved server URL that you're using. And to answer your question, if everything is configured properly, there's exactly 0 reason for us to ever handle a 301 redirect on the websocket side, unless the server admin is doing some crazy rewriting that I don't understand.

jellyfin-kodi/jellyfin_kodi/jellyfin/ws_client.py

Line 47 in f64b250

server = server.replace('https', "wss") if server.startswith('https') else server.replace('http', "ws")

If your server is using https, it will automatically use wss://. No redirect needed. Which means something else is going wrong. Furthermore, whether websockets are working or not has no effect on whether kodi will play successfully or not. It's used for in progress reporting, metadata updates from the server, and casting, not general playback. I suspect this is a red herring of a different problem, but without full logs we won't be able to find any evidence.

[toxic0berliner]

Thanks for your help, Let me provide some more logs to help :

This is the startup of kodi after a "local database reset" keeping the jellyfin addon settings : https://pastebin.ubuntu.com/p/hDwnZDPy92/

I've cut it after retrieving the first show since it's getting huge...

One unexplainable issue is that leter on in the log I see lots of this:

2021-05-01 20:12:49.397 T:12040    INFO <general>: JELLYFIN.objects.tvshows -> DEBUG::jellyfin_kodi\objects\tvshows.py:380 ADD episode [17/483] 7a5b0bb782920cb60890abc21e64ccca: 12 and a Half Angry Men
2021-05-01 20:12:49.400 T:12040    INFO <general>: JELLYFIN.objects.kodi.artwork -> DEBUG::jellyfin_kodi\objects\kodi\artwork.py:36 ADD to kodi_id 482 art: https://{jellyfin-server}/Items/7a5b0bb782920cb60890abc21e64ccca/Images/Primary/0?Format=original&Tag=8efc143cfbc739e5233dc1d1b592273f
2021-05-01 20:12:49.404 T:12040    INFO <general>: JELLYFIN.objects.tvshows -> DEBUG::jellyfin_kodi\objects\tvshows.py:278 EpisodeId 4c8cd887a43c63d28cd0faf06851ab56 not found
2021-05-01 20:12:49.411 T:12040    INFO <general>: JELLYFIN.objects.tvshows -> DEBUG::jellyfin_kodi\objects\tvshows.py:380 ADD episode [17/484] 4c8cd887a43c63d28cd0faf06851ab56: Bigfat
2021-05-01 20:12:49.413 T:12040    INFO <general>: JELLYFIN.objects.kodi.artwork -> DEBUG::jellyfin_kodi\objects\kodi\artwork.py:36 ADD to kodi_id 483 art: https://{jellyfin-server}/Items/4c8cd887a43c63d28cd0faf06851ab56/Images/Primary/0?Format=original&Tag=cd1da39c7722d2bb411b381aa28a24db
2021-05-01 20:12:49.418 T:12040    INFO <general>: JELLYFIN.objects.tvshows -> DEBUG::jellyfin_kodi\objects\tvshows.py:278 EpisodeId 1a409c526bdcf0746eb0f1c8d6fbb073 not found

And this is probably the source of my not having any thumbnails in the library.
But the url in this log is good ! If I replace my jellyfin-server by the proper hostname and point my browser to it, it serves me the proper picture... Here is my getting it with curl without any issue :

 curl --output delme4.jpg -v  https://jellyfin-server/Items/4c8cd887a43c63d28cd0faf06851ab56/Images/Primary/0?Format=original&Tag=cd1da39c7722d2bb411b381aa28a24db
 * TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [70 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2730 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [589 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=example.com
*  start date: Apr 17 22:42:52 2021 GMT
*  expire date: Jul 16 22:42:52 2021 GMT
*  subjectAltName: host "jellyfin-server" matched cert's "*.example.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
} [5 bytes data]
> GET /Items/4c8cd887a43c63d28cd0faf06851ab56/Images/Primary/0?Format=original HTTP/1.1
> Host: jellyfin-server
> User-Agent: curl/7.65.0
> Accept: */*
>
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Age: 113461827
< Cache-Control: public
< Content-Length: 40382
< Content-Security-Policy: frame-ancestors 'self' *.example.com *.other-example.com example.com other-example.com www.example.com organizr.example.com www.other-example.com organizr.other-example.com;
< Content-Type: image/jpeg
< Date: Sat, 01 May 2021 18:34:42 GMT
< Last-Modified: Tue, 26 Sep 2017 13:24:16 GMT
< Realtimeinfo.dlna.org: DLNA.ORG_TLAG=*
< Server: Kestrel
< Strict-Transport-Security: max-age=315360000; includeSubDomains; preload
< Transfermode.dlna.org: Interactive
< Vary: Accept
< X-Content-Type-Options: nosniff
< X-Frame-Options: allow-from https://organizr.example.com
< X-Response-Time-Ms: 0
< X-Robots-Tag: noindex,nofollow,nosnippet,noarchive,notranslate,noimageindex
< X-Xss-Protection: 1; mode=block
<
{ [398 bytes data]
100 40382  100 40382    0     0   443k      0 --:--:-- --:--:-- --:--:--  448k
* Connection #0 to host jellyfin-server left intact

                                                                                                                                                                                           ✔
[1]+  Done                    /bin/curl.exe --output delme4.jpg -v https://jellyfin-server/Items/4c8cd887a43c63d28cd0faf06851ab56/Images/Primary/0?Format=original

Now I understand that wss is not required to play the file, but indeed it still fails, here is some log of me attempting to play a file :
https://pastebin.ubuntu.com/p/wTcVhpRSSN/

Also, since this wss is failing somehow, probably the plugin is not giving up on reporting progress, and it keeps displaying me a rotating waiting animation instead of playing the show.

As you can see, I have turned down loglevel to info otherwise I get lost...
And I don't see any other issue than this 301 thing...

[toxic0berliner]

I've edited websocket.py to trace the request headers and response, here is what I got :

Request Headers: 

GET 
/socket?api_key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&device_id=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy 
HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: jellyfin.example.com:443
Origin: http://jellyfin.example.com:443
Sec-WebSocket-Key: zzzzzzzzzzzzzzzzzzzzzzzzz==
Sec-WebSocket-Version: 13


Response:
{
'content-security-policy': "frame-ancestors 'self' *.example.com *.other-example.com example.com other-example.com www.example.com organizr.example.com www.other-example.com organizr.other-example.com;"
'content-type': 'text/html; charset=utf-8'
'location': 'https://jellyfin.example.com/socket?api_key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&device_id=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy'
'x-robots-tag': 'noindex,nofollow,nosnippet,noarchive,notranslate,noimageindex'
'date': 'Sun
02 May 2021 08:49:21 GMT'
'content-length': '164'
}

Probably the origin still beeing http is causing this, I'm going to try to tweak the requets headers, but otherwise location seems the same as source...

I'm not that good with python but I can still edit the file ;) so if you can help me try to make it follow redirects just for test, I could see if it helps...

Thanks in advance.

[toxic0berliner]

Ok, seems I solved it by cleaning up my traefik labels and adding a general X-Forwarded-Proto middleware on traefik, now the handshake gets a proper 101 response.
But still I a missing the thumbnails, I see timeouts trying to download the thumbs :

2021-05-02 11:15:18.263 T:7472    ERROR <general>: CCurlFile::Stat - Failed: Timeout was reached(28) for https://jellyfin.example.com/Items/3261e1edabeffb15711c717f88bdfaed/Images/Primary/0?Format=original&Tag=add3dbced7ecd77ef762430233258655

And this url works properly in my browser as I said.

Maybe CCurlFile isn't doing SNI properly ? Anyone had the issue too ? Or an idea to solve it ?

[mcarlton00]

Since you have it working now, can you verify if our traefik docs are correct?

Traefik 1 - https://github.com/jellyfin/jellyfin-docs/blob/master/general/networking/traefik.md
Traefik 2 - https://github.com/jellyfin/jellyfin-docs/blob/master/general/networking/traefik2.md

As for the thumbnails, there's another open issue (that I also can't replicate) where we're trying to track down what's happening there #505

[toxic0berliner]

So, traefik1 I can't check, but for traefik 2 in fact I had all the labels from your doc and somehow it was not working, I trashed everything and ended up only with the following labels :

    labels:
      traefik.http.services.jellyfin.loadBalancer.server.port: 8096
      traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto: https
      traefik.http.routers.app-secure.middlewares: sslheader@docker

Rest looks fine.

I'll try again your mw to see why it failed for me...

[toxic0berliner]

Going one by one, I've managed to identify that the SSLForceHost label is the one causing the wss query to get a 301...
traefik.http.middlewares.jellyfinmw.headers.SSLForceHost: true
Leave that one out, or indeed, implement following a 301 in the WSS query, since in fact, I think traefik is doing what it's promising, looking at the traefik doc :

sslForceHost¶

Set sslForceHost to true and set sslHost to force requests to use SSLHost regardless of whether they already use SSL.

Probably getting mixed up with host jellyfin.example.com and jellyfin.example.com:443 as well...

Now that did not help me with the missing thumbnails... Trying to dig into it now..

[mcarlton00]

Sounds like this issue is resolved, we'll discuss the thumbnails in the other issue so we don't get split brain on it.