[feat] langfuse tracing layer #498

ahau-square · 2024-12-18T21:45:53Z

Added a tracing layer that translates Rust tracing events to locally hosted Langfuse server via Langfuse API.

The layer:

Converts tracing spans and events to Langfuse observations, capturing model completions and tool calls
Maintains parent-child span relationships
Batches events for efficient sending
Records metadata and input/output data
Tracing layer used in both CLI and Goose UI (instantiated in server)

Usage
Set LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY env variables for auth to your local Langfuse server or just run just langfuse-server

The traces are also written to a log file in ~/.config/goose/logs. The name of the log for the CLI is the same as the session name the session gets saved to. TODO outside of this PR to do the same for the Goose UI.

ahau-square · 2024-12-19T22:27:57Z

@ajgray-stripe as you may have seen we're porting goose over to rust. Wanted to get your thoughts on this first pass at adding in tracing and how we might adapt it to make your observability system easily pluggable

ajgray-stripe · 2024-12-29T21:55:24Z

Hey @ahau-square, thanks for the heads-up on this work! I rearranged what you've got into a couple different files (nothing too major as regards functionality, hopefully) and came up with this. (fft cherry-pick; I would have committed to your branch but I don't have the permissions to)

basically, the idea is that ObservationLayer struct handles the common parts + glue code e.g. methods that the native tracing crate requires, and BatchManager implementations get specific details. Most of the new code in different observers will just have to implement different versions of add_event and send in there, and the code in observation_layer.rs abstracts over everything else needed.

please lmk if that makes sense!

confirmed that the refactor works on my machine

One thing that this would still be missing is a "plugin system" of registering observer layers without needing to explicitly enumerate them in main.rs. Not really sure yet what the best way to do that is, but that doesn't necessarily have to block this

salman1993 · 2025-01-06T16:35:11Z

crates/goose/src/tracing/observation_layer.rs

+        // Create the span observation
+        let mut batch = self.batch_manager.lock().await;
+        batch.add_event(
+            "observation-create",


probably don't need this now but i am guessing these are Langfuse specific event types. do we want Observation/Observer to be a trait and this would be LangfuseObserver?

these are indeed langfuse-specific. currently the observation layer implement the layer trait, so might be confusing to have both that and an additional observe trait?

with @ajgray-stripe 's suggestion, folks can map from this schema to their own layer implementations in add_event

I think we can kick the can down the road now and revisit a refactor when someone wants to add another observability provider

baxen

Looks great!

baxen · 2025-01-06T23:29:55Z

crates/goose/src/agent.rs

@@ -5,6 +5,7 @@ use rust_decimal_macros::dec;
 use serde_json::json;
 use std::collections::HashMap;
 use tokio::sync::Mutex;
+use tracing::{debug, instrument, warn};


will have some conflicts with this file with the mcp changes, but we can forward port easily i think

baxen · 2025-01-06T23:31:56Z

crates/goose/src/tracing/langfuse_layer.rs

+            status if status.is_success() => {
+                let response_body: LangfuseIngestionResponse = response.json().await?;
+
+                for error in &response_body.errors {


kind of silly question, but what happens when you use tracing in response to a tracing integration? does it recurse? i wonder where this should go but it makes sense that we can't panic on this because the external server might be down at any given time

I think this should be fine because in on_event in observation_layer.rs, we only process the event and add to the Langfuse batch if there's a current span context, but the background task that runs send_async doesn't inherit any span context

So in this case the error trace will pass through all the layers on the trace subscriber including the langfuse one, file, console, but langfuse one will not find a current span attached to it and won't process it, and the file/console layers will process it normally

github-actions · 2025-01-07T21:17:17Z

Desktop App for this PR

The following build is available for testing:

📱 macOS Desktop App (Universal, signed)

The app is signed and notarized for macOS. After downloading, unzip the file and drag the Goose.app to your Applications folder.

This link is provided by nightly.link and will work even if you're not logged into GitHub.

* v1.0: hotfix: use same executable path for dev system

github-actions · 2025-01-07T22:05:15Z

Desktop App for this PR

The following build is available for testing:

📱 macOS Desktop App (Universal, signed)

The app is signed and notarized for macOS. After downloading, unzip the file and drag the Goose.app to your Applications folder.

This link is provided by nightly.link and will work even if you're not logged into GitHub.

github-actions · 2025-01-07T23:21:16Z

Desktop App for this PR

The following build is available for testing:

📱 macOS Desktop App (Universal, signed)

The app is signed and notarized for macOS. After downloading, unzip the file and drag the Goose.app to your Applications folder.

This link is provided by nightly.link and will work even if you're not logged into GitHub.

crates/goose/src/agent.rs

github-actions · 2025-01-08T01:08:03Z

Desktop App for this PR

The following build is available for testing:

📱 macOS Desktop App (Universal, signed)

The app is signed and notarized for macOS. After downloading, unzip the file and drag the Goose.app to your Applications folder.

This link is provided by nightly.link and will work even if you're not logged into GitHub.

github-actions · 2025-01-08T01:20:40Z

Desktop App for this PR

The following build is available for testing:

📱 macOS Desktop App (Universal, signed)

The app is signed and notarized for macOS. After downloading, unzip the file and drag the Goose.app to your Applications folder.

This link is provided by nightly.link and will work even if you're not logged into GitHub.

Co-authored-by: Alistair Gray <[email protected]>

ahau-square added 4 commits December 18, 2024 16:32

working langfuse initial implementation

2c4d9d1

simplified tracing implementation

8cf258b

merged

7e19d72

remove file logging

ed5a321

ahau-square added 2 commits December 21, 2024 21:26

add file writing layer to tracing

928398c

use generation type instead

ff8238d

ajgray-stripe and others added 5 commits January 2, 2025 12:31

refactor observation layer

c67af0b

consolidate tool traces

e3f5ce2

consolidate tool tracing

fe67482

migrated tracing into core goose

e96c084

add langfuse starter script and update justfile

697b1e2

salman1993 reviewed Jan 6, 2025

View reviewed changes

tracing added to server

926e412

ahau-square marked this pull request as ready for review January 6, 2025 22:07

baxen approved these changes Jan 6, 2025

View reviewed changes

ahau-square force-pushed the ahau/1.0-tracing-langfuse branch 2 times, most recently from 9bf6b81 to 926e412 Compare January 7, 2025 01:14

ahau-square and others added 9 commits January 6, 2025 20:40

Merge branch 'v1.0' into ahau/1.0-tracing-langfuse

6c21f7b

formatting

3ca25e7

save cli session traces to have same session name for log file name

7cd41f9

add tracing on other providers

7fa0ebc

reformat

6beb3f6

add tests

c5df60a

formatting

375b33e

merged

cc509aa

Merge v1.0: Resolve conflicts with tracing implementation

8876ae9

Merge branch 'v1.0' into ahau/1.0-tracing-langfuse

75f9719

* v1.0: hotfix: use same executable path for dev system

ahau-square added 2 commits January 7, 2025 16:53

merged with cli working with tracing

7652e27

tracing added back into server after merge

b1efec5

fix tests

9409351

zakiali reviewed Jan 7, 2025

View reviewed changes

crates/goose/src/agent.rs Outdated Show resolved Hide resolved

zakiali approved these changes Jan 8, 2025

View reviewed changes

remove agent.rs mistakenly added in merge

a25ed46

ahau-square added 2 commits January 7, 2025 20:13

fix log levels on log file

f646c13

format

aa7780f

ahau-square merged commit 798e410 into v1.0 Jan 8, 2025
6 checks passed

lily-de pushed a commit that referenced this pull request Jan 8, 2025

[feat] langfuse tracing layer (#498)

43dd61a

Co-authored-by: Alistair Gray <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feat] langfuse tracing layer #498

[feat] langfuse tracing layer #498

ahau-square commented Dec 18, 2024 •

edited

Loading

ahau-square commented Dec 19, 2024 •

edited

Loading

ajgray-stripe commented Dec 29, 2024

salman1993 Jan 6, 2025 •

edited

Loading

ahau-square Jan 6, 2025

baxen left a comment

baxen Jan 6, 2025

baxen Jan 6, 2025

ahau-square Jan 7, 2025

github-actions bot commented Jan 7, 2025

github-actions bot commented Jan 7, 2025

github-actions bot commented Jan 7, 2025

github-actions bot commented Jan 8, 2025

github-actions bot commented Jan 8, 2025

[feat] langfuse tracing layer #498

[feat] langfuse tracing layer #498

Conversation

ahau-square commented Dec 18, 2024 • edited Loading

ahau-square commented Dec 19, 2024 • edited Loading

ajgray-stripe commented Dec 29, 2024

salman1993 Jan 6, 2025 • edited Loading

Choose a reason for hiding this comment

ahau-square Jan 6, 2025

Choose a reason for hiding this comment

baxen left a comment

Choose a reason for hiding this comment

baxen Jan 6, 2025

Choose a reason for hiding this comment

baxen Jan 6, 2025

Choose a reason for hiding this comment

ahau-square Jan 7, 2025

Choose a reason for hiding this comment

github-actions bot commented Jan 7, 2025

Desktop App for this PR

github-actions bot commented Jan 7, 2025

Desktop App for this PR

github-actions bot commented Jan 7, 2025

Desktop App for this PR

github-actions bot commented Jan 8, 2025

Desktop App for this PR

github-actions bot commented Jan 8, 2025

Desktop App for this PR

ahau-square commented Dec 18, 2024 •

edited

Loading

ahau-square commented Dec 19, 2024 •

edited

Loading

salman1993 Jan 6, 2025 •

edited

Loading