How does "Accelerated Guidance" work?

[RamiAwar]

Tried looking into the code for this but it was too complicated to unwrap in 10 minutes. I'm not sure I understand this part of the docs well, it's covered too briefly I think. Can you explain in more detail how this works? How can you ask the LLM to only generate the values of a JSON structure for example and have that be accurate? This library went beyond prompt engineering into prompt sorcery

[slundberg]

GPT style LLMs are all auto-regressive and process tokens in two modes:

1. they suck in chunks of prompt tokens to fill in a KV-cache that represents the prompt to the model.
2. they generate a single new token (and add one position to the KV-cache). This generation is done by generating a probability vector over a set of tokens, then sampling from that.

Guidance can "accelerate" inference because "prompt tokens" are way cheaper/faster than "generation tokens" (due to lots of factors like GPU batching). Because a guidance program specifies much of the structure of the output we can convert many of the output tokens into batches of prompt-like tokens that are cheaper. We also can use the structure of the template to dynamically bias the next token probabilities to make sure the text that comes next aligns with the template and is optimally tokenized (e.g. token healing).

Note that we can only do this for models we have control over (currently local models in Transformers), though we are working on exposing a server version as well.

[RamiAwar]

Ah I see, I was trying to understand how this was possible without having access to the model internals. Now it makes sense :)

[linonetwo]

Can you point out the implementation code lines for this? Thanks!

[roy-pstr]

Are there any plans for LLMs models over API to support this feature? such as OpenAI?