TTFX optimization of aigenerate("whats the meaning of life"; model)

[Sixzero]

Recently I decided to cut down on TTFX of EasyContext.jl and realized that TTFX of PromptingTools needs to improve a lot:

time julia -e 'using PromptingTools; @time ai"Hi there"gpt4om;'
[ Info: Tokens: 28 @ Cost: $0.0 in 6.8 seconds
  7.875881 seconds (11.58 M allocations: 785.347 MiB, 5.01% gc time, 99.61% compilation time)
julia -e 'using PromptingTools; @time ai"Hi there"gpt4om;'  8.28s user 0.73s system 106% cpu 8.469 total

Correct me if I did something wrong here.

[Sixzero]

I wonder if we could somehow bring it down 0.3 seconds, what is the time for ai"Hello"echo in the precompilation.jl .

[svilupp]

There is already mocking like this: https://github.com/svilupp/PromptingTools.jl/blob/main/src/precompilation.jl

It seems that the majority of the time is spent on the HTTP call (as per our Slack chat), so we would need to make sure the right HTTP paths get precompiled, perhaps with a mock server to make sure the HTTP stack gets called.

Did you manage to isolate how much is the compilation vs the API request itself?

[Sixzero]

JuliaWeb/HTTP.jl#1194

Yes, it looks like HTTP.jl takes up 6 seconds and 0.3-0.4 on PromptingTools, so hopefully we will have a solution for this issue, it seems surreal, I hope I am just missing here something.

[Sixzero]

It got fixed in HTTP.jl

[svilupp]

FYI.
It was fixed here: JuliaWeb/HTTP.jl#1201

It's in the patch release 1.10.12, so set the dep if you want to enforce it.