It looks like the prompt wasn't removed from the response when using the Hugging Face provider.

[danny-su]

[Blarc]

Hey @danny-su, I'm pretty sure this is an issue with the LLM and not the plugin, since the plugin sets the message from the LLM's response.

Edit: Huh, unless this LLM returns the prompt in the response on purpose. Is this something that can be configured?

[danny-su]

@Blarc You can remove the leading prompt by its length.

https://huggingface.co/docs/transformers/main/llm_tutorial

[Blarc]

@danny-su I am still not exactly sure how to implement this.

• Should I remove the first n characters, where n is the length of the prompt?
• From the screenshot that you sent, I can see the prompt and the response are split by ---. This should also probably be removed? Maybe it would be more reliable to just remove everything before and including --- ?
• Should I add option to all client configurations or do I enable this just for HuggingFace?

[danny-su]

@Blarc You need to remove the first n characters; there is no option for this purpose.
You can check the python code from the transformers pipelines. https://github.com/huggingface/transformers/blob/19dabe96362803fb0a9ae7073d03533966598b17/src/transformers/pipelines/text_generation.py#L55

[danny-su]

@Blarc It seems not to work as expected.

[Blarc]

@danny-su That is a bit odd, since it works fine for me. Does the generated message contain the whole prompt or only a part?

[danny-su]

@Blarc It only contains part of the prompt.

[danny-su]

@Blarc, I got the connection time-out error when using Llama 3.3.