Add new OllamaSchema

CHANGELOG.md

@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 - `@ai_str` macros now support multi-turn conversations. The `ai"something"` call will automatically remember the last conversation, so you can simply reply with `ai!"my-reply"`. If you send another message with `ai""`, you'll start a new conversation. Same for the asynchronous versions `aai""` and `aai!""`.
+- Created a new default schema for Ollama models `OllamaSchema` (replacing `OllamaManagedSchema`), which allows multi-turn conversations and conversations with images (eg, with Llava and Bakllava models). `OllamaManagedSchema` has been kept for compatibility and as an example of a schema where one provides prompt as a string (not dictionaries like OpenAI API).
 
 ### Fixed
 - Removed template `RAG/CreateQAFromContext` because it's a duplicate of `RAG/RAGCreateQAFromContext`

README.md

@@ -384,13 +384,19 @@ We can use it with the `aigenerate` function:
 
 ```julia
 const PT = PromptingTools
-schema = PT.OllamaManagedSchema() # notice the different schema!
+schema = PT.OllamaSchema() # notice the different schema!
 
 msg = aigenerate(schema, "Say hi!"; model="openhermes2.5-mistral")
 # [ Info: Tokens: 69 in 0.9 seconds
 # AIMessage("Hello! How can I assist you today?")
 ```
 
+For common models that have been registered (see `?PT.MODEL_REGISTRY`), you do not need to provide the schema explicitly:
+
+```julia
+msg = aigenerate("Say hi!"; model="openhermes2.5-mistral")
+```
+
 And we can also use the `aiembed` function:
 
 ```julia
@@ -401,6 +407,8 @@ msg = aiembed(schema, ["Embed me", "Embed me"]; model="openhermes2.5-mistral")
 msg.content # 4096×2 Matrix{Float64}:
 ```
 
+You can now also use `aiscan` to provide images to Ollama models! See the docs for more information.
+
 If you're getting errors, check that Ollama is running - see the [Setup Guide for Ollama](#setup-guide-for-ollama) section below.
 
 ### Using MistralAI API and other OpenAI-compatible APIs

docs/src/examples/working_with_ollama.md

@@ -25,7 +25,7 @@ Note: You must download these models prior to using them with `ollama pull <mode
 
 > [!TIP]
 > If you use Apple Mac M1-3, make sure to provide `api_kwargs=(; options=(; num_gpu=99))` to make sure the whole model is offloaded on your GPU. Current default is 1, which makes some models unusable. Example for running Mixtral:
-> `msg = aigenerate(PT.OllamaManagedSchema(), "Count from 1 to 5 and then say hi."; model="dolphin-mixtral:8x7b-v2.5-q4_K_M", api_kwargs=(; options=(; num_gpu=99)))`
+> `msg = aigenerate(PT.OllamaSchema(), "Count from 1 to 5 and then say hi."; model="dolphin-mixtral:8x7b-v2.5-q4_K_M", api_kwargs=(; options=(; num_gpu=99)))`
 
 ## Text Generation with aigenerate
 
@@ -86,7 +86,7 @@ If you're using some model that is not in the registry, you can either add it:
 ````julia
 PT.register_model!(;
     name = "llama123",
-    schema = PT.OllamaManagedSchema(),
+    schema = PT.OllamaSchema(),
     description = "Some model")
 PT.MODEL_ALIASES["l123"] = "llama123" # set an alias you like for it
 ````
@@ -98,19 +98,31 @@ PT.MODEL_ALIASES["l123"] = "llama123" # set an alias you like for it
 OR define the schema explicitly (to avoid dispatch on global `PT.PROMPT_SCHEMA`):
 
 ````julia
-schema = PT.OllamaManagedSchema()
+schema = PT.OllamaSchema()
 aigenerate(schema, "Say hi!"; model = "llama2")
 ````
 
 ````
 AIMessage("Hello there! *smiling face* It's nice to meet you! I'm here to help you with any questions or tasks you may have, so feel free to ask me anything. Is there something specific you need assistance with today? 😊")
 ````
 
-Note: If you only use Ollama, you can change the default schema to `PT.OllamaManagedSchema()`
-via `PT.set_preferences!("PROMPT_SCHEMA" => "OllamaManagedSchema", "MODEL_CHAT"=>"llama2")`
+Note: If you only use Ollama, you can change the default schema to `PT.OllamaSchema()`
+via `PT.set_preferences!("PROMPT_SCHEMA" => "OllamaSchema", "MODEL_CHAT"=>"llama2")`
 
 Restart your session and run `aigenerate("Say hi!")` to test it.
 
+! Note that in version 0.6, we've introduced `OllamaSchema`, which superseded `OllamaManagedSchema` and allows multi-turn conversations and conversations with images (eg, with Llava and Bakllava models). `OllamaManagedSchema` has been kept for compatibility and as an example of a schema where one provides a prompt as a string (not dictionaries like OpenAI API).
+
+## Providing Images with aiscan
+
+It's as simple as providing a local image path (keyword `image_path`). You can provide one or more images:
+
+````julia
+msg = aiscan("Describe the image"; image_path=["julia.png","python.png"] model="bakllava")
+````
+
+`image_url` keyword is not supported at the moment (use `Downloads.download` to download the image locally).
+
 ## Embeddings with aiembed
 
 ### Simple embedding for one document
@@ -165,7 +177,7 @@ Add normalization as postprocessing function to normalize embeddings on receptio
 
 ````julia
 using LinearAlgebra
-schema = PT.OllamaManagedSchema()
+schema = PT.OllamaSchema()
 
 msg = aiembed(schema,
     ["embed me", "and me too"],

examples/working_with_ollama.jl

@@ -39,19 +39,27 @@ msg = aigenerate(conversation; model)
 # If you're using some model that is not in the registry, you can either add it:
 PT.register_model!(;
     name = "llama123",
-    schema = PT.OllamaManagedSchema(),
+    schema = PT.OllamaSchema(),
     description = "Some model")
 PT.MODEL_ALIASES["l123"] = "llama123" # set an alias you like for it
 
 # OR define the schema explicitly (to avoid dispatch on global `PT.PROMPT_SCHEMA`):
-schema = PT.OllamaManagedSchema()
+schema = PT.OllamaSchema()
 aigenerate(schema, "Say hi!"; model = "llama2")
 
-# Note: If you only use Ollama, you can change the default schema to `PT.OllamaManagedSchema()` 
-# via `PT.set_preferences!("PROMPT_SCHEMA" => "OllamaManagedSchema", "MODEL_CHAT"=>"llama2")`
+# Note: If you only use Ollama, you can change the default schema to `PT.OllamaSchema()` 
+# via `PT.set_preferences!("PROMPT_SCHEMA" => "OllamaSchema", "MODEL_CHAT"=>"llama2")`
 #
 # Restart your session and run `aigenerate("Say hi!")` to test it.
 
+# ! Note that in version 0.6, we've introduced `OllamaSchema`, which superseded `OllamaManagedSchema` and allows multi-turn conversations and conversations with images (eg, with Llava and Bakllava models). `OllamaManagedSchema` has been kept for compatibility and as an example of a schema where one provides a prompt as a string (not dictionaries like OpenAI API).
+
+# ## Providing Images with aiscan
+
+# It's as simple as providing an image URL (keyword `image_url`) or a local path (keyword `image_path`). You can provide one or more images:
+
+msg = aiscan("Describe the image"; image_path = ["/test/data/julia.png"]model = "bakllava")
+
 # ## Embeddings with aiembed
 
 # ### Simple embedding for one document
@@ -75,7 +83,7 @@ size(embedding)
 # ### Using postprocessing function
 # Add normalization as postprocessing function to normalize embeddings on reception (for easy cosine similarity later)
 using LinearAlgebra
-schema = PT.OllamaManagedSchema()
+schema = PT.OllamaSchema()
 
 msg = aiembed(schema,
     ["embed me", "and me too"],

src/PromptingTools.jl

@@ -60,6 +60,7 @@ include("code_generation.jl")
 include("llm_shared.jl")
 include("llm_openai.jl")
 include("llm_ollama_managed.jl")
+include("llm_ollama.jl")
 
 ## Convenience utils
 export @ai_str, @aai_str, @ai!_str, @aai!_str

src/llm_interface.jl

@@ -86,6 +86,28 @@ See `?PREFERENCES` for more details on how to set your API key permanently.
 """
 struct MistralOpenAISchema <: AbstractOpenAISchema end
 
+abstract type AbstractOllamaSchema <: AbstractPromptSchema end
+
+"""
+OllamaSchema is the default schema for Olama models.
+
+It uses the following conversation template:
+```
+[Dict(role="system",content="..."),Dict(role="user",content="..."),Dict(role="assistant",content="...")]
+```
+
+It's very similar to OpenAISchema, but it appends images differently.
+"""
+struct OllamaSchema <: AbstractOllamaSchema end
+
+"Echoes the user's input back to them. Used for testing the implementation"
+@kwdef mutable struct TestEchoOllamaSchema <: AbstractOllamaSchema
+    response::AbstractDict
+    status::Integer
+    model_id::String = ""
+    inputs::Any = nothing
+end
+
 abstract type AbstractChatMLSchema <: AbstractPromptSchema end
 """
 ChatMLSchema is used by many open-source chatbots, by OpenAI models (under the hood) and by several models and inferfaces (eg, Ollama, vLLM)