examples : do not use common library in simple example #9803
Conversation
examples/simple/simple.cpp
Outdated
| char buf[128]; | ||
| int n = llama_token_to_piece(model, new_token_id, buf, sizeof(buf), 0, true); | ||
| std::string s(buf, n); | ||
| fprintf(stderr, "%s", s.c_str()); |
There was a problem hiding this comment.
I recently found that some models can have tokens with length of up to 256. So it would be a good idea to check for negative n here.
Btw, is std::string necessary here? can we just set null terminator manually and print buf directly?
The logic for the LOG_ messages was to output the generated text to stdout and everything else to stderr. This has the benefit of being able to redirect info messages to somewhere else (e.g. /dev/null).
There was a problem hiding this comment.
I do not really understand why llama_token_to_piece does not return a NUL-terminated string in the first place. I assume that it is because some tokens may contain \0 characters, in which case it is necessary to use std::string to represent them, even though they are discarded again in the call to printf.
Printing the generated text to stderr was a mistake, I will fix that.
slaren
force-pushed
the
sl/simplify-simple-example
branch
from
c4276b0 to
8f83141
Compare
slaren
force-pushed
the
sl/simplify-simple-example
branch
from
8f83141 to
1c4d573
Compare
|
I have changed |
* examples : do not use common library in simple example * add command line parser, simplify code
* examples : do not use common library in simple example * add command line parser, simplify code
* examples : do not use common library in simple example * add command line parser, simplify code
|
@slaren Appreciate this change! |
Using the common library defeats the purpose of having an example that is supposed to serve as a simple starting point.