Latency issue fix with prepare_inference #535
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zakiali
force-pushed
the
zaki/instantiate_token_counter_once
branch
from
5f7ba9f to
45a48f4
Compare
baxen
approved these changes
Jan 2, 2025
Desktop App for this PRThe following build is available for testing: The app is signed and notarized for macOS. After downloading, unzip the file and drag the Goose.app to your Applications folder. This link is provided by nightly.link and will work even if you're not logged into GitHub. |
michaelneale
added a commit
that referenced
this pull request
Jan 3, 2025
michaelneale
added a commit
that referenced
this pull request
Jan 6, 2025
* v1.0: (43 commits) feat: openrouter provider (#538) [ui] chore: tidy up gui providers (#537) [ui]: Polish and system theme fix (#533) [ui]: General ui polish to more closely match designs (#532) Latency issue fix with prepare_inference (#535) chore: use cross to build binaries (#507) feat: a system for non developers to augment developer system (#524) fix: Broken open directory and new session buttons in more menu (#520) refactor: move get_usage to provider trait (#506) fix: Make stop button more obvious (#516) fix: Enhance Dark mode menu dots visibility (#517) working on automating release of .zip and binaries and having them on each PR as well (#509) conditionally load memory system in goose server (#508) Adds 'instructions' field to InitializeResult (#511) feat: MCP client sdk (#505) Update cli-release.yml feat: added groq provider (#494) fix: use rust tls (#500) fix: Ldelalande/fix scroll (#504) feat: MCP server sdk (simple version first) (#499) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TokenCounter takes about a second to instantiate causing a latency hit every time prepare_infernece is called. This was causing a significant latency hit on every agent interaction. This PR moves that the instantiation to the Agent struct to minimize this hit