can't not load model..

[melodysdreamj]

wait 1 hours, but did not load model..

in windows10

[JosephSBoyle]

I'm having a similar problem. Win10, ggml Alpaca 7B downloaded from huggingface:
https://huggingface.co/Pi3141

[ViperX7]

can you try the alpaca-native-enhanced model
and can you tell me how many cores are there in your system

[Teragron]

I've had the same problem with the "ggml-model-q4_0.bin" but had no issue with "ggml-model-q4_0_unfiltered.bin".
So it's probably about the model.

[llkj11]

Same issue here with: gpt4all-lora-quantized.bin and ggml-alpaca-7b-q4.bin

[OzGoober]

Using 2 files from Pi3141, gpt4-x-alpaca-native-13B-ggml worked, alpaca-native-7B-ggml didn't load

gpt4-x-alpaca-native-13B-ggml:
'models\ggml-model-q4_1.bin',
Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00

alpaca-native-7B-ggml:
'models\ggml-model-q4_0.bin',
ERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoR

[andresdelcampo]

Similar, won't load any of my 3 quantized 7B and 13B alpaca variants that worked in dalai or alpaca.cpp.

...
'-m',
'models\ggml-model-q4_0.bin',
'--interactive-first'
]
ERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoRERRoR

[boehm-e]

Same here ggml-alpaca-7b-q4.bin not loading

[AndreasKunar]

Same here, llama7B, llama13B, alpaca,... - all working locally with llama.cpp on the commandline. All hanging on load.

Parameters for invoking llama.cpp commandline seem right and commandline status shows apparent completion:
Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00

But web-ui dialogue hangs with loading.

EDIT: running on macOS/Apple Silicon via current git-clone + copying templates folder from ZIP.

[sjackp]

Same here, endless -loading model "ggml-model-q4_0.bin"

[AndreasKunar]

Updated / totally edited for better clarification.

• I'm on macOS/Apple silicon. Running the current/latest llama.cpp with several models from terminal. It all works fine in terminal, even when testing in alpaca-turbo's environment with its parameters from the terminal
• Because I want the latest llama.cpp+models, I can't just run the docker or other images. Needed to git-clone (+ copy templates folder from ZIP). Linked my working llama.cpp "main" to ./bin/mac, and its models' *.bin into ./models with nice names for the UI.

Cause of model-hang for me:
Alpaca-Turbo expects a certain string "RSTsr" to be displayed by it's invocation of llama.cpp once its finished loading (set in alpaca_turbo.py line 46). The current llama.cpp + llama 7B e.g. does not show this string in its invocation. So the load hangs. For me it seems, it's not how the model is, it's more how Alpaca-Turbo was constructed. The current llama.cpp+models produces very different responses - I'm trying to understand alpaca_turbo.py, so that maybe I can fix it (at least for me).

Hope this clarifies/helps.

[AndreasKunar]

Update - I got it to work (most of the time) on my Mac by changing alpaca_turbo.py quite a bit. But I don't think it is mergeable into a pull-request, because my solution seems to be one-off just for my situation.