feat(tracing): open-telemetry traces

[seriousben]

What

This PR enables exporting opentelemetry traces to a given endpoint. The endpoint can be either configured in the config file or using the standard opentelemetry environment variables.

config.yml example:

tracing:
  enabled: true

Changes:

• Config change to support enabling tracing
• Change to how the config parsing work to merge config values with the default

A look at existing traces

As we start using traces more and more, we'll need to format and add new traces as needed.

Tracing propagation

We are using W3C Tracecontext to propagate traces. It reads a parent trace-id from the traceparent header and returns a new trace + span back in the response headers.

curl -v -H "traceparent: 00-d5fe1dc9035165ce36952daf29686b6c-14330be33197dd1a-01" http://localhost:8900/namespaces
* Host localhost:8900 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8900...
* connect to ::1 port 8900 from ::1 port 53044 failed: Connection refused
*   Trying 127.0.0.1:8900...
* Connected to localhost (127.0.0.1) port 8900
> GET /namespaces HTTP/1.1
> Host: localhost:8900
> User-Agent: curl/8.7.1
> Accept: */*
> traceparent: 00-d5fe1dc9035165ce36952daf29686b6c-14330be33197dd1a-01
>
* Request completely sent off
< HTTP/1.1 200 OK
< content-type: application/json
< traceparent: 00-d5fe1dc9035165ce36952daf29686b6c-574acef8aad9a667-01
< tracestate:
< vary: origin, access-control-request-method, access-control-request-headers
< access-control-allow-origin: *
< content-length: 62
< date: Wed, 04 Dec 2024 20:21:45 GMT
<
* Connection #0 to host localhost left intact
{"namespaces":[{"name":"default","created_at":1733342892666}]}

Testing

Run server with the config:

tracing:
  enabled: true

run --rm --name jaeger \
                                        -e COLLECTOR_OTLP_ENABLED:true \
                                        -p 16686:16686 \
                                        -p 4317:4317 \
                                        -p 4318:4318 \
                                       jaegertracing/all-in-one:1.49

Open http://localhost:16686

Contribution Checklist

• If the python-sdk was changed, please run make fmt in python-sdk/.
• If the server was changed, please run make fmt in server/.
• Make sure all PR Checks are passing.

[seriousben]

This needs better integration with axum's traces, especially with regards to error results before being ready.

It also needs configuration to toggle tracing export.

Otherwise, this PR works well with Jaeger.

[diptanu]

Whats left here?

[seriousben]

This now only needs more spans and more instrument in the code as well as a configuration toggle for tracing.

[seriousben]

unrelated change triggered by clippy.