monitor issues

[vagetablechicken]

origin vagetablechicken#13

• taskmanager has no prometheus api, so don't add it in prometheus config

  • / or /status print a html page, not good for prometheus, maybe openmldb-exporter can handle it, but info is simple.

• what means openmldb_connected_seconds_total? https://github.com/4paradigm/openmldb-exporter/blob/53792409f9a53cceaacf0091f658ca124692d356/openmldb_exporter/collector/collectors.py#L80
  current time - connect time in show component incorrect: when openmldb-exporter is alive, stop clear start all, it can't reconnect to zk, and fail to execute show components or table status, won't update counters, so the openmldb_connected_seconds_total won't change. only zk restart can be handled, reset(start from small zxid) can't, check if needs improve
  why grafana var role doesn't have tm? fix grafana dashboard template

• Table Partition Number show deleted tables partitions, we know it's deleted in graph, but the big number make me confused.

• openmldb_table_rows will up and down when insert data

test case is insert data into disk table

• two panel are the same metric, but different titile, confused

TODO all metrics in grafana, and how to read them when do perf profiling

About doc

It's better to give a simple and clear steps for monitor deployment, current doc is hard to read.

insert

When we want to know insert perf, we can get total time by client who do insertion.
DOING: kafka->openmldb perf test in openmldb-compose

Grafana Update

• show taskmanager status in first row
• put metrics of tablets: we can know the insertion performance xxx_openmldb_api_tablet_server_put[_xxx], latency and qps

new grafana dashboard:
https://gist.github.com/vagetablechicken/9428268b8cb8ba50d5867e6f0326da64

[vagetablechicken]

• Process IO READ/WRITE Second label is instance, we can't know which is read, which is write
• IO ... is the same, confused label
• tablet query latency doesn't contains subquery, subquery is another metric pattern