chat-ui seems to ignore chatPromptTemplate

[hapatrick]

I'm trying to use chat-ui to talk to a vLLM server, using vLLM's OpenAI API, serving Mixtral-8x22B.

vLLM was throwing an error on every request from chat-ui: "Conversation roles must alternate user/assistant/user/assistant/..."

After googling I learned that this was because Mixtral doesn't support the "system" role, and I should use a chatPromptTemplate like the one below.

But setting the chatPromptTemplate had no effect on the queries chat-ui was sending to vLLM.

Can anyone tell me what I'm doing wrong?

Here is my MODELS config:

 MODELS: |
    [
      {
        "name": "vllm",
        "id": "mixtral",
        "preprompt" : "",
        "chatPromptTemplate": "<s> {{#each messages}}{{#ifUser}}[INST]{{#if @first}}{{#if @root.preprompt}}{{@root.preprompt}}\n{{/if}}{{/if}} {{content}} [/INST]{{/ifUser}}{{#ifAssistant}} {{content}}</s> {{/ifAssistant}}{{/each}}",
        "parameters" : {
          "temperature" : 0.6,
          "top_p" : 0.95,
          "repetition_penalty" : 1.2,
          "top_k" : 50,
          "truncate" : 24576,
          "max_new_tokens" : 8192,
          "stop" : ["</s>"]
        },
        "endpoints": [{
          "type" : "openai",
          "baseURL": "http://vllm:8000/v1"
        }]
      }
    ]

[hapatrick]

I don't know why chat-ui seemed to be ignoring the chatPromptTemplate. But I eventually fixed the problem in vLLM by using this template to override the default Mixtral template:

{{ bos_token }}
{%- set vars = namespace(system_message = None) -%}
{%- for message in messages -%}
    {%- if message['role'] == 'system' -%}
        {%- set vars.system_message = message['content'] -%}
    {%- elif message['role'] == 'user' -%}
        {%- if vars.system_message -%}
            {{ ' [INST] ' + vars.system_message + ' ' + message['content'] + ' [/INST]' }}
            {%- set vars.system_message = None -%}
        {%- else -%}
            {{ ' [INST] ' + message['content'] + ' [/INST]' }}
        {%- endif -%}
    {%- elif message['role'] == 'assistant' -%}
        {{ ' ' + message['content'] + ' ' + eos_token }}
    {%- else -%}
        {{ raise_exception('Only user, assistant, and system roles are supported!') }}
    {%- endif -%}
{%- endfor -%}