LWP::UserAgent integration doesn't add OT headers

[marrem]

I followed the synopsis for the LWP::UserAgent integration.

But I don't see any OT headers being added to the request. The only headers I see are:

TE: deflate,gzip;q=0.3
Connection: TE, close
Host: localhost
User-Agent: libwww-perl/6.68

How am I supposed to use this integration?

Do I still need to explicitly obtain a tracer provider, a tracer and call tracer->in_span?

First attempt

Just 'use'd the OpenTelemetry::Integration with LWP::UserAgent and used the ua to do an http request. That didn't add any header at all to the request.

Second attempt

I added:

use OpenTelemetry; 
use OpenTelemetry::SDK;

That didn't change anything. Still no OT headers on the request.

Third attempt

I also added:

# Obtain the current default tracer provider 
my $provider = OpenTelemetry->tracer_provider; 
 
# Create a trace 
my $tracer = $provider->tracer( name => 'my_app', version => '1.0' ); 
 
$tracer->in_span( outer => sub ( $span, $context ) { 
    # In outer span 
    print "Hi I'm in a span\n"; 
    }); 

That didn't add any OT header to my request either.

I know I might add custom stuff to the span by calling methods on the $span, but is that needed just for basic tracing?

So how do I use OpenTelemetry in perl? Any pointers to some simple tutorials. I couldn't find them. Most stuff I found was quite abstract.

[marrem]

Addition: I forgot the 'use feature qw(signatures)'. So the in_span call went wrong. But after adding the feature the warning disappeared. But still no headers.

[marrem]

OS: Linux, Debian 12, 4.19.0-18-amd64
Perl v5.36.0
OpenTelemetry 0.019
LWP::UserAgent 6.68

[marrem]

Doing the call inside a span does generate headers:

$tracer->in_span( outer => sub ( $span, $context ) {
        print "========CONTEXT======\n";
        print Dumper $context;
        print "========SPAN=========\n";
        print Dumper $span;
        $response = $ua->get('http://localhost/test');
});

Generates these headers:

-------request------
                 '_headers' => bless( {
                                        'user-agent' => 'libwww-perl/6.68',
                                        'traceparent' => '00-0a59a102c547ff171705206d7ba22013-9ebac056114bdb2d-01',
                                        '::std_case' => {
                                                          'traceparent' => 'Traceparent',
                                                          'tracestate' => 'Tracestate'
                                                        },
                                        'tracestate' => ''
                                      }, 'HTTP::Headers' )

The only thing I do not yet understand, if the span I create is the first (root) span, why does the parent span id in the traceparent have a value?
Or is the http call seen a child span of the span I declared in code ('outer')?

[marrem]

I close the issue.

I hope to find some more information on how to use it in our application, which is an Apache-mod-perl application.

Because somehow I have to wrap everything a request handler does inside of a span.

[jjatria]

Thank you for your interest, and for the detailed explanation of what you did, and what you expected. You are correct that there is little in the way of actual usage examples, but experiences like yours will help to create them. With this in mind, I still think it might be useful to try to answer some of your questions, even if it's just to leave a record for future readers.

(For brevity, please assume all snippets below start with importing strict, warnings, and say)

---

I followed the synopsis for the LWP::UserAgent integration.

But I don't see any OT headers being added to the request.

The synopsis at the moment is equivalent to the following:

use OpenTelemetry::Integration 'LWP::UserAgent';

LWP::UserAgent->new->get('https://httpbin.org/anything');

This is maybe less useful than it could be. This is indeed all that a library writer needs to write to enable the integration, but not all that an application writer needs to see actual work being done.

Let's try first to connect some of our expectations to OpenTelemetry terms.

We want to see some headers attached to our request. These headers are used by OpenTelemetry to propagate context information further down the pipeline. In order for OpenTelemetry to do this, it needs two things:

1. Something to propagate things with (a propagator)
2. Something that can be propagated (a valid span context)

None of these things are true yet, so let's fix that.

Propagators are responsible for propagating details of your context into a carrier. In more concrete terms, responsible for setting the OpenTelemetry headers in your request. This matters because OpenTelemetry does not require any specific propagation headers: you could be using any of a growing number of supported protocols (although note that only some of the protocols supported by the OpenTelemetry spec are supported by this Perl implementation at the moment).

Since the propagator defines which headers need to be set, if no propagator has been configured, there is nothing to add to your request.

The easiest way to set up a propagator is to import the SDK:

use OpenTelemetry::SDK;
use OpenTelemetry::Integration 'LWP::UserAgent';

LWP::UserAgent->new->get('https://httpbin.org/anything');

You can confirm that the propagators have been set up:

...
say for OpenTelemetry->propagator->keys;
# traceparent
# tracestate
# baggage

(Incidentally, these defaults come from the value of the OTEL_PROPAGATORS environment variable, which defaults to tracecontext,baggage).

But this is still only half the problem. Now that we can propagate things, we need something to propagate: the SpanContext. But not just any SpanContext. We need a SpanContext that is valid. From CPAN:

[A SpanContext is valid] if the span context's "span_id" and "trace_id"
are both valid.

This helps a little. In our snippet above we never started a span, so the current context is still just the empty root context, which does not hold a span. When we try to retrieve a span, we get back an invalid span, which is a span with an invalid
span ID.

Let's fix that.

We can do this by hand, but it can get a little cumbersome:

use LWP::UserAgent;
use OpenTelemetry qw( 
    otel_tracer_provider
    otel_current_context
    otel_context_with_span
);

use OpenTelemetry::Integration 'LWP::UserAgent';
use OpenTelemetry::SDK;

my $span = otel_tracer_provider->tracer->create_span( name => 'outer' );

# In a real application, we would need to make sure the original
# context is kept and restored at the end, but since this is the only thing that
# will run we can be a little more cavalier
otel_current_context = otel_context_with_span($span);

LWP::UserAgent->new->get('https://httpbin.org/anything');

This will do it. You can confirm this by examining the response:

my $res = LWP::UserAgent->new->get('https://httpbin.org/anything');
say $res->content;
# {
#  ...
#  "headers": {
#    "Host": "httpbin.org", 
#    "Traceparent": "00-adadf85bb2057a22ad3b374b17703fed-65b640ee4c885f62-01", 
#    "Tracestate": "", 
#    "User-Agent": "libwww-perl/6.72", 
#    "X-Amzn-Trace-Id": "..."
#  }, 
#  "method": "GET", 
#  "url": "https://httpbin.org/anything"
# }

In most cases, it will be more convenient (and safer) to do this with the in_span wrapper that you ended up using. This is because that takes care of restoring the original context even when an error occurs, etc. But this in particular is an area where help will be needed before we settle on an interface that is convenient to use. See for example #9.

---

if the span I create is the first (root) span, why does the parent
span id in the traceparent have a value? Or is the http call seen a
child span of the span I declared in code ('outer')?

You are precisely right. The span you created (outer) is the root span, so it does not have a parent ID. You can check this for yourself:

...
OpenTelemetry->tracer_provider->tracer->in_span( outer => sub {
    # We need a snapshot because spans are write-only
    my $span = shift->snapshot;

    say $span->hex_parent_span_id; # Will print 0000000000000000
});

Like you correctly guessed, the integration is wrapping the request
in a child span, so the parent span ID is the ID of the span you created
(in your case, the root span). This is so that you don't have to bother
setting up all the things that need to go into a client HTTP span to make
it compliant with OpenTelemetry expects. You can see what we do to this
span in the code.

---

I hope this answers at least some of your questions. I haven't used mod_perl in a while, so I haven't spent a lot of effort in trying to make it more convenient to use, but it could be that writing an integration for it can help.