Implement OpenTelemetry tracing

[brianehlert]

The NGINX team recently released a native OpenTelemetry tracing module: https://nginx.org/en/docs/ngx_otel_module.html
This finally allows the NGINX Ingress Controller to implement OpenTelemetry tracing and set a deprecation path for the currently included OpenTracing module.

There will be a period of time where both modules should be present in the images so that users have a chance to do in place upgrades through modifying their resources and converting to the opentelemetry tracing configuration.
If we followed the 'normal' K8s API deprecation process this would be three releases of compatibility where both modules would be present. Following the current quarterly release practices, this would be an overlap period 9 months.
For example: opentracing only -> opentracing / open telemetry -> opentracing / open telemetry -> opentracing / open telemetry -> open telemetry only

We don't know how many customers implement OpenTracing today and thus are unsure of the real impact, but estimate that 9 months should give adequate planning and modification time.
Also, considering the number of times that OpenTelemetry adoption has been raised, we are unclear if any users have tools that do not support OpenTelemetry where removing OpenTracting would be considered a burden.

[brianehlert]

The NGINX OTEL module mentioned above is currently exclusive to NGINX Plus (unfortunately) and does not meet the needs of our NGINX OSS user base. We will re-evaluate this as we don't want to implement different solutions for the paid and free edition.
We have the hope that someone in the wider NGINX community will port the module to NGINX OSS which will solve the problem.

[ryepup]

exclusive to NGINX Plus (unfortunately)

This has changed; the OTEL module now supports NGINX OSS. From https://github.com/nginxinc/nginx-otel:

The ngx_otel_module dynamic module enables NGINX Open Source or NGINX Plus to send telemetry data to an OTel collector.

[brianehlert]

Yes, that has changed from Aug 11 when I posted that additional comment.
We do want to enable this by default in the project and there is a community PR in-flight.
#4508

I think the key is currently the How we achieve this for both OSS and N+.

[tinhtq]

do you have any article to run this with grafana?