feat(opentelemetry): support variable resource attributes #13839

SuzyWangIBMer · 2024-11-06T19:40:50Z

Summary

This change allows render resource attributes as lua code.
For example,

  config:
    traces_endpoint: "https://otlp-red-saas.instana.io:4318/v1/traces"
    resource_attributes:
      host.id: "$(headers.host)"

host.id is 0.0.0.0:9000 or depends on headers.host value.

Checklist

The Pull Request has tests
A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Issue reference

Fix #12538

samugi · 2024-11-15T09:00:03Z

failing test is expected, let's rebase after #13877 is merged

samugi

Thank you so much for your contribution, this is great! I left a few comments for things that I feel could be improved a bit. There's some code duplication with request-transformer that we may want to take care of, but that refactor can be done later.

samugi · 2024-11-18T08:27:37Z

kong/plugins/opentelemetry/utils.lua

@@ -13,6 +21,8 @@ local DEFAULT_HEADERS = {

 local _log_prefix = "[otel] "

+local compiled_resource_attributes = {}


keeping this variable at the module level's scope can be problematic because it will persist across different requests, which may introduce cross-request pollution. I'd recommend we just return the compiled resource attributes from the function that computes them.

@samugi Thanks for quick response. I have updated the pr as suggested.

samugi · 2024-11-18T08:38:25Z

kong/plugins/opentelemetry/utils.lua

+    end
+
+    if res then
+      current_name = current_name:lower()


why is this needed?

samugi · 2024-11-18T08:52:19Z

kong/plugins/opentelemetry/traces.lua

@@ -122,7 +122,7 @@ end

 local function http_export_traces(conf, spans)
  local headers = get_headers(conf.headers)
-  local payload = encode_traces(spans, conf.resource_attributes)
+  local payload = encode_traces(spans, otel_utils.compiled_resource_attributes)


(see other comments)

samugi · 2024-11-18T08:52:38Z

kong/plugins/opentelemetry/logs.lua

@@ -21,7 +21,7 @@ local prepare_logs = otlp.prepare_logs

 local function http_export_logs(conf, logs_batch)
  local headers = get_headers(conf.headers)
-  local payload = encode_logs(logs_batch, conf.resource_attributes)
+  local payload = encode_logs(logs_batch, otel_utils.compiled_resource_attributes)


(see my comments in the traces.lua file)

samugi · 2024-11-18T08:55:11Z

kong/plugins/opentelemetry/handler.lua

+  if conf.resource_attributes then
+    otel_utils.read_resource_attributes(conf.resource_attributes)
+  end


instead of reading resource attributes in the access() phase, we can do the rendering of the attributes in log() and pass the result as part of the conf parameter, when we enqueue the entry to be processed by http_export_traces. This way we don't need a global variable anymore (see my other comment in utils.lua), and is also more performant because the rendering will happen out of the hot path, after the response is already returned to the client.

samugi · 2024-11-18T09:00:57Z

kong/plugins/opentelemetry/utils.lua

+  local compiled_templates = template_cache[resource_attributes]
+  if not compiled_templates then
+    compiled_templates = {}
+    -- store it by `config_array` which is part of the plugin `conf` table


caching looks good here. Please update the comment which still refers to the original code (mentioning config_array).

samugi · 2024-11-18T09:17:24Z

kong/plugins/opentelemetry/utils.lua

+  for current_name, current_value in pairs(resource_attributes) do
+    local res, err = param_value(current_value, resource_attributes, template_env)
+    if err then
+      return error(" failed to render the template " ..


error is a bit harsh here, it can make sense in request-transformer where a failure in modifying the request can have security implications, but here I would just log an error and then leave the original (raw) value for the attribute, wdyt?

samugi · 2024-11-18T09:22:00Z

spec/03-plugins/37-opentelemetry/02-schema_spec.lua

@@ -33,4 +33,16 @@ describe("Plugin: OpenTelemetry (schema)", function()
      }
    }, err)
  end)
+
+  it("accepts variable values", function()


could you add a test for a failing template as well?

I haven't touch original resource attributes schema much, except than adding a compiled template. It is still a map with boolean/string/number type. In the future, we might add validator limiting variable to be existing env vars.

pull-request-size bot added the size/L label Nov 6, 2024

github-actions bot added plugins/opentelemetry schema-change-noteworthy cherry-pick kong-ee schedule this PR for cherry-picking to kong/kong-ee labels Nov 6, 2024

team-eng-enablement added the author/community PRs from the open-source community (not Kong Inc) label Nov 6, 2024

samugi self-assigned this Nov 15, 2024

samugi force-pushed the feat/otel_variable_resource_attributes branch from 6dc12b9 to eaf8f25 Compare November 15, 2024 13:31

samugi self-requested a review November 15, 2024 17:07

samugi requested changes Nov 18, 2024

View reviewed changes

SuzyWangIBMer added 2 commits November 18, 2024 13:33

feat(opentelemetry):set variable resource attributes

8f3ff38

test(opentelemetry): support variable resource attributes

8a185af

SuzyWangIBMer force-pushed the feat/otel_variable_resource_attributes branch from eaf8f25 to 8a185af Compare November 18, 2024 22:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(opentelemetry): support variable resource attributes #13839

feat(opentelemetry): support variable resource attributes #13839

SuzyWangIBMer commented Nov 6, 2024 •

edited

Loading

samugi commented Nov 15, 2024

samugi left a comment

samugi Nov 18, 2024

SuzyWangIBMer Nov 18, 2024

samugi Nov 18, 2024

samugi Nov 18, 2024

samugi Nov 18, 2024

samugi Nov 18, 2024

samugi Nov 18, 2024

samugi Nov 18, 2024

samugi Nov 18, 2024

SuzyWangIBMer Nov 18, 2024

		@@ -13,6 +21,8 @@ local DEFAULT_HEADERS = {

		local _log_prefix = "[otel] "

		local compiled_resource_attributes = {}

feat(opentelemetry): support variable resource attributes #13839

Are you sure you want to change the base?

feat(opentelemetry): support variable resource attributes #13839

Conversation

SuzyWangIBMer commented Nov 6, 2024 • edited Loading

Summary

Checklist

Issue reference

samugi commented Nov 15, 2024

samugi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SuzyWangIBMer commented Nov 6, 2024 •

edited

Loading