v0.44.0

PromptingTools v0.44.0

Diff since v0.43.0

Added

• Added Mistral Large 2 and Mistral-Nemo to the model registry (alias mistral-nemo).

Fixed

• Fixed a bug where wrap_string would not correctly split very long Unicode words.

Commits

Merged pull requests:

• Add mistral large 2 (#184) (@svilupp)
• Fix wrap_string for long Unicode (#185) (@svilupp)

v0.43.0

PromptingTools v0.43.0

Diff since v0.42.0

Added

• Added Llama 3.1 registry records for Fireworks.ai (alias fllama3, fllama370, fllama3405 and fls, flm, fll for small/medium/large similar to the other providers).

Commits

Merged pull requests:

• Add Fireworks.ai llama3.1 (#183) (@svilupp)

v0.42.0

PromptingTools v0.42.0

Diff since v0.41.0

Added

• Registered new Meta Llama 3.1 models hosted on GroqCloud and Together.ai (eg, Groq-hosted gllama370 has been updated to point to the latest available model and 405b model now has alias gllama3405). Because that's quite clunky, I've added abbreviations based on sizes small/medium/large (that is 8b, 70b, 405b) under gls/glm/gll for Llama 3.1 hosted on GroqCloud (similarly, we now have tls/tlm/tll for Llama3.1 on Together.ai).
• Generic model aliases for Groq and Together.ai for Llama3 models have been updated to point to the latest available models (Llama 3.1).
• Added Gemma2 9b model hosted on GroqCloud to the model registry (alias ggemma9).

Updated

• Minor optimizations to SubDocumentTermMatrix to reduce memory allocations and improve performance.

Commits

Merged pull requests:

• Register Llama 3.1 models + minor retrieval improvement (#182) (@svilupp)

v0.41.0

PromptingTools v0.41.0

Diff since v0.40.0

Added

• Introduced a "view" of DocumentTermMatrix (=SubDocumentTermMatrix) to allow views of Keyword-based indices (ChunkKeywordsIndex). It's not a pure view (TF matrix is materialized to prevent performance degradation).

Fixed

• Fixed a bug in find_closest(finder::BM25Similarity, ...) where the view of DocumentTermMatrix (ie, view(DocumentTermMatrix(...), ...)) was undefined.
• Fixed a bug where a view of a view of a ChunkIndex wouldn't intersect the positions (it was returning only the latest requested positions).

Commits

Merged pull requests:

• Add SubDocumentTermMatrix (#181) (@svilupp)

v0.40.0

PromptingTools v0.40.0

Diff since v0.39.0

Added

• Introduces RAGTools.SubChunkIndex to allow projecting views of various indices. Useful for pre-filtering your data (faster and more precise retrieval). See ?RT.SubChunkIndex for more information and how to use it.

Updated

• CandidateChunks and MultiCandidateChunks intersection methods updated to be an order of magnitude faster (useful for large sets like tag filters).

Fixed

• Fixed a bug in find_closest(finder::BM25Similarity, ...) where minimum_similarity kwarg was not implemented.

Commits

Merged pull requests:

• Add SubChunkIndex (view of index) (#179) (@svilupp)

v0.39.0

PromptingTools v0.39.0

Diff since v0.38.0

Breaking Changes

• Changed the default model for ai* chat functions (PT.MODEL_CHAT) from gpt3t to gpt4om (GPT-4o-mini). See the LLM-Leaderboard results and the release blog post.

Added

• Added the new GPT-4o-mini to the model registry (alias gpt4om). It's the smallest and fastest model based on GPT4 that is cheaper than GPT3.5Turbo.

Commits

Merged pull requests:

• Add GPT-4o-mini + set as default (#180) (@svilupp)

v0.38.0

PromptingTools v0.38.0

Diff since v0.37.1

Added

• Added a new tagging filter RT.AllTagFilter to RT.find_tags, which requires all tags to be present in a chunk.
• Added an option in RT.get_keywords to set the minimum length of the keywords.
• Added a new method for reciprocal_rank_fusion and utility for standardizing candidate chunk scores (score_to_unit_scale).

Commits

Merged pull requests:

• Template update (#177) (@svilupp)
• Add AllTagFilter (#178) (@svilupp)

v0.37.1

PromptingTools v0.37.1

Diff since v0.37.0

Fixed

• Fixed a bug in CohereReranker when it wouldn't handle correctly CandidateChunks index_ids.

Commits

Merged pull requests:

• Fix CohereReranker bug (#176) (@svilupp)

v0.37.0

PromptingTools v0.37.0

Diff since v0.36.0

Updated

• Increase compat bound for FlashRank to 0.4

Commits

Merged pull requests:

• Increase compat for FlashRank (#175) (@svilupp)

v0.36.0

PromptingTools v0.36.0

Diff since v0.35.0

Added

• Added a prompt template for RAG query expansion for BM25 (RAGQueryKeywordExpander)

Fixed

• Fixed a small bug in the truncation step of the RankGPT's permutation_step! (bad indexing of string characters).
• Fixed a bug where a certain combination of rank_start and rank_end would not result the last sliding window.
• Fixed a bug where partially filled RAGResult would fail pretty-printing with pprint

Commits

Merged pull requests:

• Update rankGPT implementation + add a template (#174) (@svilupp)