feat: track cost and token usage in log file #80
Conversation
| logger = logging.getLogger(_LOGGER_NAME) | ||
| logger.setLevel(level) | ||
| log_file_directory.mkdir(parents=True, exist_ok=True) | ||
| file_handler = logging.FileHandler(log_file_directory / _LOGGER_FILE_NAME) |
There was a problem hiding this comment.
what do you think about bucketing the logs by time (e.g. goose.yyyy-mm-dd.log or something)? not sure how verbose the logging could be, but we sorta also get rolling logs for free too
There was a problem hiding this comment.
Yeah, definitely we can use the TimedRotatingFileHandler when the log gets verbose. At the moment we only log the cost, but I think it will be helpful to log errors.
There was a problem hiding this comment.
makes sense! if we wanna go long-term we could also go one step further extending logging.Filter to some intelligent routing [or segment] (e.g. api-costs.log vs goose.log)
There was a problem hiding this comment.
yeah, like tags
src/goose/_logger.py
Outdated
|
|
||
| def setup_logging(log_file_directory: Path, level: int = logging.INFO) -> None: | ||
| logger = logging.getLogger(_LOGGER_NAME) | ||
| logger.setLevel(level) |
There was a problem hiding this comment.
if we want to control this via the cli , i've been playing with click and we could maybe add @click.option("--log-level", ...) so it's easier to dial up the messages up/down without rebuilding goose
* main: feat: add version options (#74)
lifeizhou-ap
changed the title
|
I like this - and has been asked for a bit |
|
fyi since this is a common feature in Q/A or eval tools (langtrace, openlit, traceloop etc) you can have a look to validate approach vs what they are doing. Not suggesting change, just for context: e.g. from openlit |
Why
Users can view the cost and token usage in the log file
What
exchange[PR](feat: collect total token usage square/exchange#32)