Skip to content

Commit

Permalink
apply protocol detection docs update to edge (#1873)
Browse files Browse the repository at this point in the history
Signed-off-by: Alex Leong <[email protected]>
Co-authored-by: Flynn <[email protected]>
  • Loading branch information
adleong and kflynn authored Dec 4, 2024
1 parent 7a02962 commit 828abd1
Showing 1 changed file with 54 additions and 47 deletions.
101 changes: 54 additions & 47 deletions linkerd.io/content/2-edge/features/protocol-detection.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,17 @@ Linkerd is capable of proxying all TCP traffic, including TLS connections,
WebSockets, and HTTP tunneling.

In most cases, Linkerd can do this without configuration. To accomplish this,
Linkerd performs *protocol detection* to determine whether traffic is HTTP or
HTTP/2 (including gRPC). If Linkerd detects that a connection is HTTP or
HTTP/2, Linkerd automatically provides HTTP-level metrics and routing.

If Linkerd *cannot* determine that a connection is using HTTP or HTTP/2,
Linkerd will proxy the connection as a plain TCP connection, applying
[mTLS](../automatic-mtls/) and providing byte-level metrics as usual.

(Note that HTTPS calls to or from meshed pods are treated as TCP, not as HTTP.
Because the client initiates the TLS connection, Linkerd is not be able to
decrypt the connection to observe the HTTP transactions.)
Linkerd performs *protocol detection* to determine whether traffic is HTTP
(including HTTP/2 and gRPC). If Linkerd detects that a connection is HTTP, it
will automatically provide HTTP-level metrics and routing. If Linkerd *cannot*
determine that a connection is using HTTP, Linkerd will proxy the connection as
a plain TCP connection without HTTP metrics and routing. (In both cases,
non-HTTP features such as [mutual TLS](../automatic-mtls/) and byte-level
metrics are still applied.)

Protocol detection can only happen if the HTTP traffic is unencrypted from the
client. If the application itself initiates a TLS call, Linkerd will not be able
to decrypt the connection, and will treat it as an opaque TCP connection.

## Configuring protocol detection

Expand All @@ -30,45 +30,53 @@ connections, you are likely running into a protocol detection timeout.
This section will help you understand how to fix this.
{{< /note >}}

In some cases, Linkerd's protocol detection will time out because it doesn't see
any bytes from the client. This situation is commonly encountered when using
protocols where the server sends data before the client does (such as SMTP) or
protocols that proactively establish connections without sending data (such as
Memcache). In this case, the connection will proceed as a TCP connection after a
10-second protocol detection delay.
To do protocol detection, Linkerd waits for up to 10 seconds to see bytes sent
from the client. Note that until the protocol has been determined, Linkerd
cannot even establish a connection to the destination, since HTTP routing
configuration may inform where this connection is established to.

If Linkerd does not see enough data from the client within 10 seconds from
connection establishment to determine the protocol, Linkerd will treat the
connection as an opaque TCP connection and will proceed as normal, establishing
the connection to the destination and proxying the data.

To avoid this delay, you will need to provide some configuration for Linkerd.
There are two basic mechanisms for configuring protocol detection: _opaque
ports_ and _skip ports_:
In practice, protocol detection timeouts typically happen when the application
is using a protocol where the server sends data before the client does (such as
SMTP) or a protocol that proactively establishes connections without sending data
(such as Memcache). In this case, everything will work, but Linkerd will
introduce an unnecessary 10 second delay before connection establishment.

To avoid this delay, you can provide some configuration for Linkerd. There are
two basic mechanisms for configuring protocol detection: _opaque ports_ and
_skip ports_:

* Opaque ports instruct Linkerd to skip protocol detection and proxy the
connection as a TCP stream
connection as a TCP stream.
* Skip ports bypass the proxy entirely.

Opaque ports are generally preferred as they allow Linkerd to provide mTLS,
TCP-level metrics, policy, etc. Skip ports circumvent Linkerd's ability to
provide security features.
Opaque ports are generally preferred as they only skip protocol detection,
without interfering with Linkerd's ability to provide mTLS, TCP-level metrics,
policy, etc. Skip ports, by contrast, create networking rules that avoid the
proxy entirely, circumventing Linkerd's ability to provide security features.

Linkerd maintains a default list of opaque ports that corresponds to the
standard ports used by protocols that interact poorly with protocol detection.
As of the 2.12 release, that list is: **25** (SMTP), **587** (SMTP), **3306**
(MySQL), **4444** (Galera), **5432** (Postgres), **6379** (Redis), **9300**
(ElasticSearch), and **11211** (Memcache).

## Protocols that may require configuration

The following table contains common protocols that may require additional
configuration.

| Protocol | Standard port(s) | In default list? | Notes |
| Protocol | Standard ports | In default list? | Notes |
|-----------------|------------------|------------------|-------|
| SMTP | 25, 587 | Yes | |
| MySQL | 3306 | Yes | |
| MySQL with Galera | 3306, 4444, 4567, 4568 | Partially | Ports 4567 and 4568 are not in Linkerd's default set of opaque ports |
| MySQL with Galera | 3306, 4444, 4567, 4568 | Partially | Ports 4567 and 4568 are not in Linkerd's default list of opaque ports |
| PostgreSQL | 5432 | Yes | |
| Redis | 6379 | Yes | |
| ElasticSearch | 9300 | Yes | |
| Memcache | 11211 | Yes | |
| NATS | 4222, 6222, 8222 | No | |

If you are using one of those protocols, follow this decision tree to determine
which configuration you need to apply.
Expand All @@ -78,27 +86,32 @@ which configuration you need to apply.
## Marking ports as opaque

You can use the `config.linkerd.io/opaque-ports` annotation to mark a port as
opaque. Note that this annotation should be set on the _destination_, not on the
source, of the traffic.
opaque. Linkerd will skip protocol detection on opaque ports, and treat
connections to them as TCP streams.

This annotation can be set in a variety of ways:
This annotation should be set on the _destination_, not on the source, of the
traffic. This is true even if the destination is unmeshed, as it controls the
behavior of meshed clients.

1. On the workload itself, e.g. on the Deployment's Pod spec receiving the traffic.
1. On the Service receiving the traffic.
1. On a namespace (in which it will apply to all workloads in the namespace).
1. In an [authorization policy](../server-policy/) `Server` object's
`proxyProtocol` field, in which case it will apply to all pods targeted by that
`Server`.
This annotation *must* be set in two places:

When set, Linkerd will skip protocol detection both on the client side and on
the server side. Note that since this annotation informs the behavior of meshed
_clients_, it can be applied to unmeshed workloads as well as meshed ones.
1. On the Service receiving the traffic.
2. On the workload itself (e.g. on the Deployment's Pod spec receiving the
traffic), or on enclosing namespace, in which it will apply to all workloads in
the namespace.

{{< note >}}
Multiple ports can be provided as a comma-delimited string. The values you
provide will _replace_, not augment, the default list of opaque ports.
{{< /note >}}

{{< note >}}
If you are using [authorization policies](../server-policy/), the `Server`'s
`proxyProtocol` field which can be used to control protocol detection behavior
and can be used instead of a Service annotation. Regardless, we suggest
annotating the Service object for clarity.
{{< /note >}}

## Marking ports as skip ports

Sometimes it is necessary to bypass the proxy altogether. In this case, you can
Expand All @@ -125,12 +138,6 @@ Note that the default set of opaque ports can be configured at install
time, e.g. by using `--set proxy.opaquePorts`. This may be helpful in
conjunction with `enable-external-profiles`.

{{< note >}}
There was a bug in Linkerd 2.11.0 and 2.11.1 that prevented the opaque ports
behavior of `enable-external-profiles` from working. This was fixed in Linkerd
2.11.2.
{{< /note >}}

## Using `NetworkPolicy` resources with opaque ports

When a service has a port marked as opaque, any `NetworkPolicy` resources that
Expand Down

0 comments on commit 828abd1

Please sign in to comment.