From 828abd15f2c8407e2d2a0154a34d25cc2e57127d Mon Sep 17 00:00:00 2001 From: Alex Leong Date: Wed, 4 Dec 2024 10:00:35 -0800 Subject: [PATCH] apply protocol detection docs update to edge (#1873) Signed-off-by: Alex Leong Co-authored-by: Flynn --- .../2-edge/features/protocol-detection.md | 101 ++++++++++-------- 1 file changed, 54 insertions(+), 47 deletions(-) diff --git a/linkerd.io/content/2-edge/features/protocol-detection.md b/linkerd.io/content/2-edge/features/protocol-detection.md index 541230e3b7..b19b1508fa 100644 --- a/linkerd.io/content/2-edge/features/protocol-detection.md +++ b/linkerd.io/content/2-edge/features/protocol-detection.md @@ -9,17 +9,17 @@ Linkerd is capable of proxying all TCP traffic, including TLS connections, WebSockets, and HTTP tunneling. In most cases, Linkerd can do this without configuration. To accomplish this, -Linkerd performs *protocol detection* to determine whether traffic is HTTP or -HTTP/2 (including gRPC). If Linkerd detects that a connection is HTTP or -HTTP/2, Linkerd automatically provides HTTP-level metrics and routing. - -If Linkerd *cannot* determine that a connection is using HTTP or HTTP/2, -Linkerd will proxy the connection as a plain TCP connection, applying -[mTLS](../automatic-mtls/) and providing byte-level metrics as usual. - -(Note that HTTPS calls to or from meshed pods are treated as TCP, not as HTTP. -Because the client initiates the TLS connection, Linkerd is not be able to -decrypt the connection to observe the HTTP transactions.) +Linkerd performs *protocol detection* to determine whether traffic is HTTP +(including HTTP/2 and gRPC). If Linkerd detects that a connection is HTTP, it +will automatically provide HTTP-level metrics and routing. If Linkerd *cannot* +determine that a connection is using HTTP, Linkerd will proxy the connection as +a plain TCP connection without HTTP metrics and routing. (In both cases, +non-HTTP features such as [mutual TLS](../automatic-mtls/) and byte-level +metrics are still applied.) + +Protocol detection can only happen if the HTTP traffic is unencrypted from the +client. If the application itself initiates a TLS call, Linkerd will not be able +to decrypt the connection, and will treat it as an opaque TCP connection. ## Configuring protocol detection @@ -30,45 +30,53 @@ connections, you are likely running into a protocol detection timeout. This section will help you understand how to fix this. {{< /note >}} -In some cases, Linkerd's protocol detection will time out because it doesn't see -any bytes from the client. This situation is commonly encountered when using -protocols where the server sends data before the client does (such as SMTP) or -protocols that proactively establish connections without sending data (such as -Memcache). In this case, the connection will proceed as a TCP connection after a -10-second protocol detection delay. +To do protocol detection, Linkerd waits for up to 10 seconds to see bytes sent +from the client. Note that until the protocol has been determined, Linkerd +cannot even establish a connection to the destination, since HTTP routing +configuration may inform where this connection is established to. + +If Linkerd does not see enough data from the client within 10 seconds from +connection establishment to determine the protocol, Linkerd will treat the +connection as an opaque TCP connection and will proceed as normal, establishing +the connection to the destination and proxying the data. -To avoid this delay, you will need to provide some configuration for Linkerd. -There are two basic mechanisms for configuring protocol detection: _opaque -ports_ and _skip ports_: +In practice, protocol detection timeouts typically happen when the application +is using a protocol where the server sends data before the client does (such as +SMTP) or a protocol that proactively establishes connections without sending data +(such as Memcache). In this case, everything will work, but Linkerd will +introduce an unnecessary 10 second delay before connection establishment. + +To avoid this delay, you can provide some configuration for Linkerd. There are +two basic mechanisms for configuring protocol detection: _opaque ports_ and +_skip ports_: * Opaque ports instruct Linkerd to skip protocol detection and proxy the - connection as a TCP stream + connection as a TCP stream. * Skip ports bypass the proxy entirely. -Opaque ports are generally preferred as they allow Linkerd to provide mTLS, -TCP-level metrics, policy, etc. Skip ports circumvent Linkerd's ability to -provide security features. +Opaque ports are generally preferred as they only skip protocol detection, +without interfering with Linkerd's ability to provide mTLS, TCP-level metrics, +policy, etc. Skip ports, by contrast, create networking rules that avoid the +proxy entirely, circumventing Linkerd's ability to provide security features. Linkerd maintains a default list of opaque ports that corresponds to the standard ports used by protocols that interact poorly with protocol detection. -As of the 2.12 release, that list is: **25** (SMTP), **587** (SMTP), **3306** -(MySQL), **4444** (Galera), **5432** (Postgres), **6379** (Redis), **9300** -(ElasticSearch), and **11211** (Memcache). ## Protocols that may require configuration The following table contains common protocols that may require additional configuration. -| Protocol | Standard port(s) | In default list? | Notes | +| Protocol | Standard ports | In default list? | Notes | |-----------------|------------------|------------------|-------| | SMTP | 25, 587 | Yes | | | MySQL | 3306 | Yes | | -| MySQL with Galera | 3306, 4444, 4567, 4568 | Partially | Ports 4567 and 4568 are not in Linkerd's default set of opaque ports | +| MySQL with Galera | 3306, 4444, 4567, 4568 | Partially | Ports 4567 and 4568 are not in Linkerd's default list of opaque ports | | PostgreSQL | 5432 | Yes | | | Redis | 6379 | Yes | | | ElasticSearch | 9300 | Yes | | | Memcache | 11211 | Yes | | +| NATS | 4222, 6222, 8222 | No | | If you are using one of those protocols, follow this decision tree to determine which configuration you need to apply. @@ -78,27 +86,32 @@ which configuration you need to apply. ## Marking ports as opaque You can use the `config.linkerd.io/opaque-ports` annotation to mark a port as -opaque. Note that this annotation should be set on the _destination_, not on the -source, of the traffic. +opaque. Linkerd will skip protocol detection on opaque ports, and treat +connections to them as TCP streams. -This annotation can be set in a variety of ways: +This annotation should be set on the _destination_, not on the source, of the +traffic. This is true even if the destination is unmeshed, as it controls the +behavior of meshed clients. -1. On the workload itself, e.g. on the Deployment's Pod spec receiving the traffic. -1. On the Service receiving the traffic. -1. On a namespace (in which it will apply to all workloads in the namespace). -1. In an [authorization policy](../server-policy/) `Server` object's - `proxyProtocol` field, in which case it will apply to all pods targeted by that - `Server`. +This annotation *must* be set in two places: -When set, Linkerd will skip protocol detection both on the client side and on -the server side. Note that since this annotation informs the behavior of meshed -_clients_, it can be applied to unmeshed workloads as well as meshed ones. +1. On the Service receiving the traffic. +2. On the workload itself (e.g. on the Deployment's Pod spec receiving the +traffic), or on enclosing namespace, in which it will apply to all workloads in +the namespace. {{< note >}} Multiple ports can be provided as a comma-delimited string. The values you provide will _replace_, not augment, the default list of opaque ports. {{< /note >}} +{{< note >}} +If you are using [authorization policies](../server-policy/), the `Server`'s +`proxyProtocol` field which can be used to control protocol detection behavior +and can be used instead of a Service annotation. Regardless, we suggest +annotating the Service object for clarity. +{{< /note >}} + ## Marking ports as skip ports Sometimes it is necessary to bypass the proxy altogether. In this case, you can @@ -125,12 +138,6 @@ Note that the default set of opaque ports can be configured at install time, e.g. by using `--set proxy.opaquePorts`. This may be helpful in conjunction with `enable-external-profiles`. -{{< note >}} -There was a bug in Linkerd 2.11.0 and 2.11.1 that prevented the opaque ports -behavior of `enable-external-profiles` from working. This was fixed in Linkerd -2.11.2. -{{< /note >}} - ## Using `NetworkPolicy` resources with opaque ports When a service has a port marked as opaque, any `NetworkPolicy` resources that