A few questions about vision models (multimodal)

[DanilZittser]

Hello, developers of ollama-python!

Thank you for your hard work and dedication to building such an awesome library. Keep it up! 🚀

I have a few questions about using multimodal models.

1️⃣ Where can I find up-to-date information about what prompt techniques the model supports?

For example, the llama3.2-vision description page doesn't say anything about the model supporting prompt techniques like system prompt, few-shot examples or structured output. Am I right in thinking that for each model you need to go to the official GitHub page?

For example, on this issue I learned that the model does not support few-shot prompting.

===

2️⃣ Should the same way of representing an image as bytes be used for few-shot examples and the actual request?

messages = [
    # <system_prompt>
    {
        'role': 'system',
        'content': system_prompt,
    },
    # </system_prompt>
    # <few_shot>
    {
        'role': 'user',
        'images': ['dummy.jpg']  # <-- 1 path-like str
    },
    {
        'role': 'assistant',
        'content': '{"dummy_output": [1, 2]}',
    },
    # </few_shot>
    # <actual_request>
    {
        'role': 'user',
        'images': [numpy_image.tobytes()]  # <-- 2 numpy built-in method
    }
    # </actual_request>
]

The problem is this:

def main() -> None:
    img = np.random.randint(low=0, high=256, size=(240, 320, 3), dtype=np.uint8)
    cv2.imwrite('dummy.jpg', img)

    assert Path('dummy.jpg').read_bytes() == img.tobytes()  # AssertionError
    assert Path('dummy.jpg').read_bytes() == cv2.imread('dummy.jpg').tobytes()  # AssertionError

===

ollama server version

ollama version is 0.5.2-rc3-0-g581a4a5-dirty

ollama client version

Name: ollama
Version: 0.4.4
Summary: The official Python client for Ollama.
Home-page: https://ollama.com
Author: Ollama
Author-email: [email protected]
License: MIT
Requires: httpx, pydantic
Required-by: 

[TemugeB]

Wouldn't this also cause assertion error assert cv.imread('dummy.jpg').tobytes() == img.tobytes() ? Imwrite saves in RGB format while internally opencv uses BGR?

[TemugeB]

Nevermind. The problem is you're using JPEG to save, which compresses the image. So the decompression is not exactly the same as the original image. Try using PNG to save.