Currently, throughput of ~300 tokens/sec is achieved using a batch size of 34. Throughput drops to 30 tokens/sec for single input.
Next - To use INT8 or fp8 quantised models or use TensorRT
-
Notifications
You must be signed in to change notification settings - Fork 0
Satvik713/Mistral-7B-inference-optimisation-
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
No description, website, or topics provided.
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published