v0.28.0

PromptingTools v0.28.0

Diff since v0.27.0

### Added

• Added Mistral coding-oriented Codestral to the model registry, aliased as codestral or mistralc. It's very fast, performant and much cheaper than similar models.

### Commits

Merged pull requests:

• Add Mistral Codestral (#160) (@svilupp)

v0.27.0

PromptingTools v0.27.0

Diff since v0.26.2

Added

• Added a keyword-based search similarity to RAGTools to serve both for baseline evaluation and for advanced performance (by having a hybrid index with both embeddings and BM25). See ?RT.KeywordsIndexer and ?RT.BM25Similarity for more information, to build use build_index(KeywordsIndexer(), texts) or convert an existing embeddings-based index ChunkKeywordsIndex(index).

Updated

• For naming consistency, ChunkIndex in RAGTools has been renamed to ChunkEmbeddingsIndex (with an alias ChunkIndex for backwards compatibility). There are now two main index types: ChunkEmbeddingsIndex and ChunkKeywordsIndex (=BM25), which can be combined into a MultiIndex to serve as a hybrid index.

Commits

Merged pull requests:

• Add BM25 Index (#157) (@svilupp)
• Implement E2E Hybrid retrieval (BM25 + embeddings) (#159) (@svilupp)

Closed issues:

• [FR] RAG: Add HybridChunkIndex with BM25 support (#119)

v0.26.2

PromptingTools v0.26.2

Diff since v0.26.1

Fixed

• Fixed a rare bug where prompt templates created on MacOS will come with metadata that breaks the prompt loader. From now on, it ignores any dotfiles (hidden files starting with ".").

Commits

Merged pull requests:

• Template file parsing fix (#156) (@svilupp)

v0.26.1

PromptingTools v0.26.1

Diff since v0.26.0

Fixed

• Fixed a bug where utility length_longest_common_subsequence was not working with complex Unicode characters

Commits

Merged pull requests:

• Fix LCS utility (#155) (@svilupp)

v0.26.0

PromptingTools v0.26.0

Diff since v0.25.0

BREAKING CHANGES

• Added new field meta to TracerMessage and TracerMessageLike to hold metadata in a simply dictionary. Change is backward-compatible.
• Changed behaviour of aitemplates(name::Symbol) to look for the exact match on the template name, not just a partial match. This is a breaking change for the aitemplates function only. Motivation is that having multiple matches could have introduced subtle bugs when looking up valid placeholders for a template.

Added

• Improved support for aiclassify with OpenAI models (you can now encode upto 40 choices).
• Added a template for routing questions :QuestionRouter (to be used with aiclassify)
• Improved tracing by TracerSchema to automatically capture crucial metadata such as any LLM API kwargs (api_kwargs), use of prompt templates and its version. Information is captured in meta(tracer) dictionary. See ?TracerSchema for more information.
• New tracing schema SaverSchema allows to automatically serialize all conversations. It can be composed with other tracing schemas, eg, TracerSchema to automatically capture necessary metadata and serialize. See ?SaverSchema for more information.
• Updated options for Binary embeddings (refer to release v0.18 for motivation). Adds utility functions pack_bits and unpack_bits to move between binary and UInt64 representations of embeddings. RAGTools adds the corresponding BitPackedBatchEmbedder and BitPackedCosineSimilarity for fast retrieval on these Bool<->UInt64 embeddings (credit to domluna's tinyRAG).

Fixed

• Fixed a bug where aiclassify would not work when returning the full conversation for choices with extra descriptions

Commits

Merged pull requests:

• Improvements to aiclassify and aitemplates (#150) (@svilupp)
• Improve tracer schema / automated logging (#151) (@svilupp)
• Add BitPacked embeddings for RAG retrieval (#152) (@svilupp)
• add tracer kwargs (#153) (@svilupp)

Closed issues:

• [FR] Improve performance of Bool embeddings (#144)

v0.25.0

PromptingTools v0.25.0

Diff since v0.24.0

Added

• Added model registry record for the latest OpenAI GPT4 Omni model (gpt4o) - it's as good as GPT4, faster and cheaper.

Commits

Merged pull requests:

• Add GPT4-Omni (#149) (@svilupp)

v0.24.0

PromptingTools v0.24.0

Diff since v0.23.0

Added

• Added support for DeepSeek models via the dschat and dscode aliases. You can set the DEEPSEEK_API_KEY environment variable to your DeepSeek API key.

Commits

Merged pull requests:

• Add DeepSeek models (#147) (@svilupp)
• Update Changelog (#148) (@svilupp)

v0.23.0

PromptingTools v0.23.0

Diff since v0.22.0

Added

• Added new prompt templates for "Expert" tasks like LinuxBashExpertAsk, JavascriptExpertTask, etc.
• Added new prompt templates for self-critiquing agents like ChiefEditorTranscriptCritic, JuliaExpertTranscriptCritic, etc.

Updated

• Extended aicodefixer_feedback methods to work with AICode and AIGenerate.

Commits

Merged pull requests:

• Add model providers and Supported functions (#134) (@adarshpalaskar1)
• Mention that the account requires funding (#135) (@KronosTheLate)
• Add templates and minor improvements (#142) (@svilupp)

Closed issues:

• Unable to get started (#132)

v0.22.0

PromptingTools v0.22.0

Diff since v0.21.0

### Added

• Added support for Groq, the fastest LLM provider out there. It's free for now, so you can try it out - you just need to set your GROQ_API_KEY. We've added Llama3 8b (alias "gllama3"), 70b (alias "gllama370") and Mixtral 8x7b (alias "gmixtral"). For the shortcut junkies, we also added a shorthand Llama3 8b = "gl3" (first two letters and the last digit), Llama3 70b = "gl70" (first two letters and the last two digits).

Commits

Merged pull requests:

• Add support for groq (#139) (@svilupp)
• Update Project.toml (#140) (@svilupp)

v0.21.0

PromptingTools v0.21.0

Diff since v0.20.1

Added

• New models added to the model registry: Llama3 8b on Ollama (alias "llama3" for convenience) and on Together.ai (alias "tllama3", "t" stands for Together.ai), also adding the llama3 70b on Together.ai (alias "tllama370") and the powerful Mixtral-8x22b on Together.ai (alias "tmixtral22").

Fixed

• Fixed a bug where pretty-printing RAGResult would forget a newline between the sources and context sections.

### Commits

Merged pull requests:

• Add Llama 3 (#138) (@svilupp)