Metrics missing - Prometheus remote_write (experimental-prometheus-rw) (e.g. k6_http_req_duration_seconds and many others)

[Nachtfalkeaw]

Brief summary

Running the k6 tool with the option experimental-prometheus-rw works but it is not exporting all (needed) metrics.
The referenced Grafana Dashboard ist not fully working because metrics are missing.

k6 version

k6 v0.50.0 (commit/f18209a5e3, go1.21.8, linux/amd64)

OS

Linux 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux

Docker version and image (if applicable)

No response

Steps to reproduce the problem

I am using these environment variables:

export K6_PROMETHEUS_RW_SERVER_URL="http://localhost:9090/api/v1/write"
export K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM="true"
export K6_PROMETHEUS_RW_TREND_STATS="p(95),p(99),min,max,sum,avg,med"

Running this command
./k6 run -o experimental-prometheus-rw --traces-output otel="http://192.168.178.51:4317" --insecure-skip-tls-verify --no-connection-reuse --no-vu-connection-reuse --tag testid=pfsense_01 --vus 10 --duration 3m --profiling-enabled ./script.js

Missing at least these metrics in remote_write export:

k6_http_req_duration_seconds
k6_iteration_duration_seconds
k6_dropped_iterations_total
k6_http_req_blocked_seconds
k6_http_req_tls_handshaking_seconds
k6_http_req_sending_seconds
k6_http_req_waiting_seconds
k6_http_req_receiving_seconds
k6_http_req_duration_seconds
k6_checks_rate

I am using this grafana dashboard:
https://grafana.com/grafana/dashboards/18030-k6-prometheus-native-histograms/

Expected behaviour

The missing metrics should be pushed to Prometheus remote_write because they are needed to visualize the dashboard.
if I use -o web-dashboard and open the Web-Dashboard on localhost:5665 I can see these metrics.

Actual behaviour

Missing these metrics in remote_write output:

k6_http_req_duration_seconds
k6_iteration_duration_seconds
k6_dropped_iterations_total
k6_http_req_blocked_seconds
k6_http_req_tls_handshaking_seconds
k6_http_req_sending_seconds
k6_http_req_waiting_seconds
k6_http_req_receiving_seconds
k6_http_req_duration_seconds
k6_checks_rate

[codebien]

Hi @Nachtfalkeaw,

export K6_PROMETHEUS_RW_TREND_STATS="p(95),p(99),min,max,sum,avg,med"

this env variable is not required when you use native histograms.

Did you check that instead your test is flushing the gauges and counters as stats metrics? https://k6.io/docs/results-output/real-time/prometheus-remote-write/#counter-and-gauges

Can you share a basic docker-compose to simulate your environment and your configuration, please? It might be you have a misconfigured environment. You can use as a base the docker-compose provided in the output's repo https://github.com/grafana/xk6-output-prometheus-remote/blob/main/docker-compose.yml.

[Nachtfalkeaw]

Hello,

I checked my config again and you were right. It conflicts if I have both env variables set together.

I used these 2 only:
export K6_PROMETHEUS_RW_SERVER_URL="http://localhost:9090/api/v1/write"
export K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM="true"

and removed this
export K6_PROMETHEUS_RW_TREND_STATS="p(95),p(99),min,max,sum,avg,med"

and then the dashboard works. I probably expected to have both type of metrics at the same time so I enabled both.

PS:
Is it possible to set these two values as config flag instead on env variables?
export K6_PROMETHEUS_RW_SERVER_URL="http://localhost:9090/api/v1/write"
export K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM="true"

However, this can be closed as I think it is solved.

[codebien]

Is it possible to set these two values as config flag instead on env variables?

@Nachtfalkeaw yes, you should be able to set them by using url and trendAsNativeHistogram flags.

[maxiar]

Same issue .. I tried with only those environment vars:

export K6_PROMETHEUS_RW_SERVER_URL="http://localhost:9090/api/v1/write"
export K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM="true"

but some visualization does not work:

only those metrics are available :(

Any suggestion?

EDIT: I found the solution. I'm using MIMIR prometheus endpoint. I need to activate the native histogram ingestion adding a new param in my config:

https://grafana.com/docs/mimir/latest/configure/configure-native-histograms-ingestion/