Web App Access: fails with Gzip Content-Encoding

[imcatwhocode]

Internal Server Error occurs when a user tries to connect to a gzip-encoded endpoint via Web App Access. Browser, cURL, and Go (net/http with TLSClient) successfully connect and decode the body from the same endpoint.

Expected behavior: Web App Access successfully proxies connections to endpoints with gzip content encoding.

Current behavior: Web App Access returns an Internal Server Error for such endpoints.

Steps to reproduce

1. Obtain an example endpoint with Content-Encoding: gzip. You can use example server attached below.
2. Add a configuration entry for Web App Access pointing to this server.
3. Open this resource via Teleport in your browser.

Details

Version

# teleport version
Teleport v14.1.1 git:api/v14.1.1-0-gfb6429e go1.21.3

# uname -a
Linux example-node 5.4.0-165-generic #182-Ubuntu SMP Mon Oct 2 19:43:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.6 LTS
Release:	20.04
Codename:	focal

Error log

2023-10-26T21:31:43+03:00 ERRO [APP:SERVI] Error forwarding to https://127.0.0.1:3443/, err: dial tcp 127.0.0.1:3443: connect: connection refused reverseproxy/reverse_proxy.go:231

Configuration entry

app_service:
  ...
  apps:
    - name: test-gzip
      uri: "https://127.0.0.1:3443"
      insecure_skip_verify: true

Example endpoint server

You might need an example server to reproduce this issue. Golang is outside of my knowledge, I can only provide a zero-deps Node.js example.

Requirements: Node.js 16 or newer.

const zlib = require('zlib');
const https = require('https');

const key = `-----BEGIN PRIVATE KEY-----
MIIJQgIBADANBgkqhkiG9w0BAQEFAASCCSwwggkoAgEAAoICAQCRD/mtxsEUidiD
A4Uk23Emul3sSP9tTvkJ8iuiMT4XcgnzTeqJWx15v0hZF1+e5oQKevT2hA+kIABm
0lQEg8M7dDPPvdhTz0ew6ezDeR3itbs/YpDaHm9zCQmeEjWIEmD9GWk36vsm8s+0
2L4WOsQ1x98TFaYopcYHL3W+79YTbzEYX+y+fGNbz/gLyGKlYj0WqWQaz+uJcE0P
r+6VdHso4e+YKQxjNJGhOfofdhZ+dZPkjpJZB/PxGixAnh/T4enz962mSJ7zeDWL
2tlWt0GQhod5G86zzUAQbb1n31ObhUu4r6Dd2QxshAYTfs9aHzSjpHczPVe+o8w7
cYI7hTeoCLmvAtLNG3HxKuSqzIoS+e5kf5tkVGaZu2HhDrvfq4CeqiUi5iDIxhpr
n5LwMIL6eMbi2m9r0KYae3Wy7VuY+CgT9HPoG2H5CQTmzB/Q3eyv/LPxVeQ/QpBm
56BUBTrKp0rh5WiPbYM3UP8buOLKATeRVDwOXH7hUceLNVLVgsswYBQ1iDE6xQqR
fYuB7d8WHioUke/A4WnFZKZufXYS5BcgeIyYtrdJ6UNRQCQHjgi6MyNIXMG1b9I6
hhC+KSL3YDV2SBH8Oi4uPqBH89Fni+QKeT4r/aKnaG1Hu0copsRk90oZ5/PrkydW
tv2FFBKLUsk5F6SAKrYQ1YUM5K3HzQIDAQABAoICAATjkbHkYrH/QlOKDRT0q1/X
hJ5XyHoF4RlPdt6Aw9u07XZsgBTQhBIMGxNbmQ6pLLaQtcxQH5nTtUfB23CKg0wv
6qYb+xw5/mvNOnxNCu+O2eh1OTYMtAtfuNoBuZmkKu4kytnGWuARa2QoMOHdaar5
WUCSqHayvZWU6SY9/YxKoVA1vWwfE5cuz2iQQJjb0ny+yT+pFcJcTVCTyJnzYso6
MSHC8mauiSDyrAYe6/unmUwo7PBAtx0/WVH1MlPpB4JALPNPe8HvYXic79m/E9/b
1NmFUSZYqsnS9QD48x+o8qgfDTAk5kkK+wxxp+nc5uCEBLtunBQx/4paVT4Ny5Ya
cgJjd5wokQgTbZLRBPHC2H6qiLOpCDd3Oa2bEjFfpM1q5sBcUdXPq+v5I5gWwjUF
U93cG0xEsYS0qiBfqUxSNS/kEwwJ28d/9QhyM5a1cuG2z2vTqnH4WNoQ5fLyUf4f
h4zKDm9KH1MocYsX6v1aNHvpW48WEPFPWXYI1eQZxdBm8dx9wb/QAwfm4N3KJru7
J9NLlPZPLBCTJ8D5L0+RbP3mbvAs6rm+ng8liRhrg4Zmr6BAsUWeqz0YADb+CIHj
eeUfWkNdj6/8BBzRH/LJSAg7nhFqcrJ58bTDosbTlIHjqK3HykEYXILNAnhkXKB1
VXqgvbYXhCW3mcKYZ/YRAoIBAQDKMvCb+vmaA82vTYs595nozsVw8m675Nx/Iiom
1Uj8053aTBPNLumDwPE57A6lgUiWKo+dP/k7uHKh52+8wPrM8OJr2NsF2djdx+/G
bm2S2Fuh5v56dKeSLVGBfv8PJ6AdMpmHcFuacRsF6F9XyOhmCNjKSRUN190XHcbF
sxdHnckNTB1QmvubDjf8S4BEkILZsuVAJeBBb6mGV9ZeqqfppVvwLgBEH9HjMXxh
wR2bu80+9yBTkUS8w8ilRod+/0AU14UlAK5QEx4fCWdZnxijR8F4BvaKH19kHpcP
6j1mCf+hsQFwrQsejWMLf6dKsfI5Ic0DkdLQhsVNyszKoTK9AoIBAQC3qRljVlHC
1dN06JqBatSngaxw0N2E0U5/QL+Ka2th9PFQtklqzq4z3RkII71elONI4PnLUllJ
eLFNvRUsEFkXE5mVemHoJeK2jKFXzlDjrwpqX7HL9dmaj15Up/iG0LtFMxsugqqA
kFMzN/AMIoN/6AkjdzJCRsYSTDB7G2PACmjjFDEuMyu/wAhEg9Gs9IhT2FiQBwdZ
jT9lYef0krZRC7Tk/pYleE0hAsM2J5OsXA+OJWIYXp7HEOgDWeAsJ0KVE8C4C3R4
8xmpalUHLBa6J5HNohEzZ9I24LQMg31sNfsISHaciRl3V33pEACE2QNFh8kBv//+
VUi4Q7AweEJRAoIBACQOt3epZu9/NCPJ2UUqEWlBzlXY9cROCMAnMIsiR9TGTAUM
wg86CWx0cNAad3YuOuqEhn3NVb2/o7mZkwR/CXqYjlgbINvHQHQ0Xl7tfF7OdHDH
x6PAv81ChaFpK0ThlRz/0/8iG0PMXbxM9wupiUAA3eAc7kM8tnGT3smMqV8dFSho
5WVlbe71tygnsqUHfD4kN9ubpYUly35olZLmLfFtipqnKWb8oao5PewUyxnlgLF0
0m/0nORoqZRHoowUKrJMyLKTj4I6FooEXEjQJU2g0zB7VWXGOHWbOuccp3xV1h9Y
eGBGi5AUOJBpYETivMZRb8TuMUD8d5zqyGBuJKUCggEASTIciymQxky/zJ0TTrbL
78JPflrPnipo1XBxpfHvxwd+0UME+YtLhCH+4u6Tf0lpV9dJ1o2IRi7Swv+zm28c
goTuNdOjh4jp1QUxHDR+qm9NgF2JF5dw1ReEacrT0q8Ho6mdkKjE2R4r+oK1qbBr
piNM2RSdNCrOr+YrgDTHSXccnDuk6hu8uNwIr321p++BEaFskmIqvLb9aCNf9Wov
PRekTYtlPaO3YYufT0rwoMXD8UvWz42OTBnhzcwY+p/e+lj3+qGeFsIUHR/iqUwZ
9rDkfAa2Qj2DWxmsyDvzVZgh5wgyNWqyeoylAqi8/DUdB635HzhtBujoZX+cjYSw
MQKCAQEAoTsZtz1c63n/Lvz2BG2Ffr8BIuCW+UtD62AoiOygJJ5ROSBLVwR/lIiD
gZlaTUGvzO7zE+VUdjhU9bc0mQQVuabonBFxo+ggMGyXNZQMB1UyuOkJlcRfQNMX
6RE/hjJz18GQGAObgQqZDxzzLXps7ThUqpSqFlI9r2m8lck1qSkSdGWA8j+g1ZVH
7LL9uY15GyIlat6IOcerM/lK7LsThqnjQn5C0Gqrh9ma6uIWJMDdCGzMQp5DViC6
sFJofYTi6+VlgWtgqwl8DXHsbAU9d5rYUuKem6O8eXydKigYvtGXVf+As8h43cIB
1MulIuQWWj0g9rpttlvJoZfPk7b4TQ==
-----END PRIVATE KEY-----
`;

const cert = `-----BEGIN CERTIFICATE-----
MIIFpzCCA4+gAwIBAgIUHNgXXh3mEOqRJpncOeT03VoUYHYwDQYJKoZIhvcNAQEL
BQAwYzELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM
GEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDEcMBoGA1UEAwwTZ3ppcC10ZXN0Lmxv
Y2FsaG9zdDAeFw0yMzEwMjYxODI4MTZaFw0yNDEwMjUxODI4MTZaMGMxCzAJBgNV
BAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBX
aWRnaXRzIFB0eSBMdGQxHDAaBgNVBAMME2d6aXAtdGVzdC5sb2NhbGhvc3QwggIi
MA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCRD/mtxsEUidiDA4Uk23Emul3s
SP9tTvkJ8iuiMT4XcgnzTeqJWx15v0hZF1+e5oQKevT2hA+kIABm0lQEg8M7dDPP
vdhTz0ew6ezDeR3itbs/YpDaHm9zCQmeEjWIEmD9GWk36vsm8s+02L4WOsQ1x98T
FaYopcYHL3W+79YTbzEYX+y+fGNbz/gLyGKlYj0WqWQaz+uJcE0Pr+6VdHso4e+Y
KQxjNJGhOfofdhZ+dZPkjpJZB/PxGixAnh/T4enz962mSJ7zeDWL2tlWt0GQhod5
G86zzUAQbb1n31ObhUu4r6Dd2QxshAYTfs9aHzSjpHczPVe+o8w7cYI7hTeoCLmv
AtLNG3HxKuSqzIoS+e5kf5tkVGaZu2HhDrvfq4CeqiUi5iDIxhprn5LwMIL6eMbi
2m9r0KYae3Wy7VuY+CgT9HPoG2H5CQTmzB/Q3eyv/LPxVeQ/QpBm56BUBTrKp0rh
5WiPbYM3UP8buOLKATeRVDwOXH7hUceLNVLVgsswYBQ1iDE6xQqRfYuB7d8WHioU
ke/A4WnFZKZufXYS5BcgeIyYtrdJ6UNRQCQHjgi6MyNIXMG1b9I6hhC+KSL3YDV2
SBH8Oi4uPqBH89Fni+QKeT4r/aKnaG1Hu0copsRk90oZ5/PrkydWtv2FFBKLUsk5
F6SAKrYQ1YUM5K3HzQIDAQABo1MwUTAdBgNVHQ4EFgQU4jNT8/j9fe4y0Tiy8i8z
H8vO9sswHwYDVR0jBBgwFoAU4jNT8/j9fe4y0Tiy8i8zH8vO9sswDwYDVR0TAQH/
BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAgEAf1CXhmSRakFacWaHzMcr/xvRJses
YABjxdMSxIQvot1tAw5HATje20GiUks88WjTf786en+Cepgx4+/mYC5/h9pTNKtT
K/yggeeq508Z9ce60e2dl1+DekVZO7yicadOU6lxTFgUJFNx/EuEyqM3ilxdk6rv
krzAEngsGc3liG7/SvEvtGn2DcPmEhyckNn61cPLHRFBga9lHQ5yqB7AONs/f3/d
UgLB4JgsM4zDXuRCahw/l1ZNlJwyNGM1eAH/0yg0WitWHI6BvleAJj7ZPiDA3Idd
nbYtSLh0HMxuwz+ZYqbBpdpKls/zg4jHtmBHCxXQ9FN6Ont0czMjasNRNbPsTI80
OMgDt4JXVj/Zxt5l7YQLszbZyFtpHoMXhk+y0AH6LeL9SLTgEp0qGk9PPWzIcVRV
4PLGv1lWj6SQe9Rc20QWDxvA0eG0gtunCyjeISbuNuCjW1DQkjyrBP6v0Gnm0W4h
CGqPYuUqyLpKPgCJhXOptb+Zw066Zb11QnhRedbx5Hh0qDQGtvuwrMUxF/VXkZc/
S/dBnVApJ5F+5dD2B0mel3SaDVIF2RoCMmkmzxQDX56MqEgfZN6PsvNJkRggqSHZ
VrG4WEihMCp/DSgP63KilqhDauKvv6WLaiib2fhKY0IO59Nuo2KwBMTNAliiIS4W
saA0IsdaGexa1+c=
-----END CERTIFICATE-----
`;

https.createServer({ key, cert }, (req, res) => {
  res.writeHead(200, { 'content-encoding': 'gzip' });
  zlib.createGzip().end('<h1>Hello, Secure World!</h1>').pipe(res)
}).listen(3443);

Example cURL request

curl -kv --compressed https://localhost:3443
*   Trying 127.0.0.1:3443...
* Connected to localhost (127.0.0.1) port 3443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=gzip-test.localhost
*  start date: Oct 26 18:28:16 2023 GMT
*  expire date: Oct 25 18:28:16 2024 GMT
*  issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=gzip-test.localhost
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* using HTTP/1.1
> GET / HTTP/1.1
> Host: localhost:3443
> User-Agent: curl/8.1.2
> Accept: */*
> Accept-Encoding: deflate, gzip
> 
< HTTP/1.1 200 OK
< content-encoding: gzip
< Date: Thu, 26 Oct 2023 19:09:25 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
< Transfer-Encoding: chunked
< 
* Connection #0 to host localhost left intact
<h1>Hello, Secure World!</h1>

Example Go client

I've used it to verify that the issue is not in crypto/tls or net/http modules.

package main
import (
	"crypto/tls"
	"fmt"
	"io"
	"net/http"
)
func httpsClient(url string) []byte {
	tr := &http.Transport{
		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
	}
	client := &http.Client{Transport: tr}
	resp, err := client.Get(url)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()
	fmt.Println("Status =", resp.Status)
	msg, _ := io.ReadAll(resp.Body)
	return msg
}

func main() {
	msg := httpsClient("https://localhost:3443")
	fmt.Println("Body =", string(msg))
}

[hicasper]

I've noticed that as well. Also, I found that the teleport application doesn't seem to support gzip compression from the server side to the client/browser and can't use nginx as a forward proxy. Nowadays, the static assets of the application (such as javascript, css files, etc.) are very large in file size, and not supporting gzip will slow down the loading speed.

[webvictim]

I tested a gzip-enabled nginx proxy behind Teleport recently and can confirm that it does correctly pass gzip-encoded traffic through if the correct headers (Accept-Encoding: gzip) are present in the request made to the Teleport proxy.

I would advise anyone reporting this bug to try this functionality again and report back if there is any further issue.

2023-10-26T21:31:43+03:00 ERRO [APP:SERVI] Error forwarding to https://127.0.0.1:3443/, err: dial tcp 127.0.0.1:3443: connect: connection refused reverseproxy/reverse_proxy.go:231

This error is not related to gzip encoding - it indicates that the downstream application was not accepting connections on port 3443 when Teleport attempted to connect to it.