Task freeze

[damienlaine]

If for some reason the llm service fails to respond. Jobs might get stuck forever

See logs :

linto_llm-gateway.1.1qeqpejy2j4n@linagora-linto-bm-02    | 25/11/2024 15:46:13 http_server INFO: Task 0f78d8f8-38b9-4623-a227-98d19bc01527 queued
linto_llm-gateway.1.1qeqpejy2j4n@linagora-linto-bm-02    | 25/11/2024 15:46:13 http_server INFO: Task 0f78d8f8-38b9-4623-a227-98d19bc01527 processing started
linto_llm-gateway.1.1qeqpejy2j4n@linagora-linto-bm-02    | 25/11/2024 15:46:13 backend INFO: Loading prompt for service: summary
linto_llm-gateway.1.1qeqpejy2j4n@linagora-linto-bm-02    | 25/11/2024 15:46:13 backend INFO: Prompt fields: 2
linto_llm-gateway.1.1qeqpejy2j4n@linagora-linto-bm-02    | 25/11/2024 15:46:13 backend INFO: Setting up backend with params: {'name': 'llama3', 'modelName': 'casperhansen/llama-3-8b-instruct-awq', 'totalContextLength': 8192, 'maxGenerationLength': 2048, 'tokenizerClass': 'LlamaTokenizer', 'createNewTurnAfter': 300, 'summaryTurns': 2, 'maxNewTurns': 10, 'temperature': 0.1, 'top_p': 0.8} for task: 0f78d8f8-38b9-4623-a227-98d19bc01527
linto_llm-gateway.1.1qeqpejy2j4n@linagora-linto-bm-02    | You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message.
linto_llm-gateway.1.1qeqpejy2j4n@linagora-linto-bm-02    | 25/11/2024 15:46:15 backend ERROR: Error publishing: Connection error.
linto_llm-gateway.1.1qeqpejy2j4n@linagora-linto-bm-02    | 25/11/2024 15:46:15 http_server ERROR: An error occurred in processing tasks : cannot unpack non-iterable NoneType object

[damienlaine]

@htagourti

[htagourti]

Fixed in "houssem_llm_backend" branch.
When an exception is raised in the backend, the celery task will fail and not block the queue
PR coming soon