2.18.0.0

Version 2.18.0.0 Release Notes

Compatible with OpenSearch 2.18.0

Features

• Add AVX512 support to k-NN for FAISS library #2069

Enhancements

• Introducing a loading layer in FAISS #2033
• Add short circuit if no live docs are in segments #2059
• Optimize reduceToTopK in ResultUtil by removing pre-filling and reducing peek calls #2146
• Update Default Rescore Context based on Dimension #2149
• KNNIterators should support with and without filters #2155
• Adding Support to Enable/Disble Share level Rescoring and Update Oversampling Factor#2172
• Add support to build vector data structures greedily and perform exact search when there are no engine files #1942
• Add CompressionLevel Calculation for PQ #2200
• Remove FSDirectory dependency from native engine constructing side and deprecated FileWatcher #2182
• Update approximate_threshold to 15K documents #2229
• Update default engine to FAISS #2221

Bug Fixes

• Add DocValuesProducers for releasing memory when close index #1946
• KNN80DocValues should only be considered for BinaryDocValues fields #2147
• Score Fix for Binary Quantized Vector and Setting Default value in case of shard level rescoring is disabled for oversampling factor#2183
• Java Docs Fix For 2.x#2190

Documentation

• Fix sed command in DEVELOPER_GUIDE.md to append a new line character '\n'. #2181

Maintenance

• Remove benchmarks folder from k-NN repo #2127
• Fix lucene codec after lucene version bumped to 9.12. #2195

Refactoring

• Does not create additional KNNVectorValues in NativeEngines990KNNVectorWriter when quantization is not needed #2133
• Minor refactoring and refactored some unit test #2167

2.17.0.0

Version 2.17.0.0 Release Notes

Compatible with OpenSearch 2.17.0

Features

• Integrate Lucene Vector field with native engines to use KNNVectorFormat during segment creation #1945
• k-NN query rescore support for native engines #1984
• Add support for byte vector with Faiss Engine HNSW algorithm #1823
• Add support for byte vector with Faiss Engine IVF algorithm #2002
• Add mode/compression configuration support for disk-based vector search #2034
• Add spaceType as a top level optional parameter while creating vector field. #2044

Enhancements

• Adds iterative graph build capability into a faiss index to improve the memory footprint during indexing and Integrates KNNVectorsFormat for native engines#1950
• Add model version to model metadata and change model metadata reads to be from cluster metadata #2005

Bug Fixes

• Corrected search logic for scenario with non-existent fields in filter #1874
• Add script_fields context to KNNAllowlist [#1917] (#1917)
• Fix graph merge stats size calculation #1844
• Disallow a vector field to have an invalid character for a physical file name. #1936
• Fix memory overflow caused by cache behavior #2015
• Use correct type for binary vector in ivf training #2086
• Switch MINGW32 to MINGW64 #2090

Infrastructure

• Parallelize make to reduce build time [#2006] (#2006)

Maintenance

• Fix a flaky unit test:testMultiFieldsKnnIndex, which was failing due to inconsistent merge behaviors #1924

Refactoring

• Introduce KNNVectorValues interface to iterate on different types of Vector values during indexing and search #1897
• Integrate KNNVectorValues with vector ANN Search flow #1952
• Clean up parsing for query #1824
• Refactor engine package structure #1913
• Refactor method structure and definitions #1920
• Refactor KNNVectorFieldType from KNNVectorFieldMapper to a separate class for better readability. #1931
• Generalize lib interface to return context objects #1925
• Restructure mappers to better handle null cases and avoid branching in parsing #1939
• Added Quantization Framework and implemented 1Bit and multibit quantizer#1889
• Encapsulate dimension, vector data type validation/processing inside Library #1957
• Add quantization state cache #1960
• Add quantization state reader and writer #1997

2.16.0.0

Version 2.16.0.0 Release Notes

Compatible with OpenSearch 2.16.0

Features

• Adds dynamic query parameter ef_search #1783
• Adds dynamic query parameter ef_search in radial search faiss engine #1790
• Adds dynamic query parameter nprobes #1792
• Add binary format support with HNSW method in Faiss Engine #1781
• Add script scoring support for knn field with binary data type #1826
• Add painless script support for hamming with binary vector data type #1839
• Add binary format support with IVF method in Faiss Engine #1784
• Add support for Lucene inbuilt Scalar Quantizer #1848

Enhancements

• Switch from byte stream to byte ref for serde #1825

Bug Fixes

• Fixing the arithmetic to find the number of vectors to stream from java to jni layer.#1804
• Fixed LeafReaders casting errors to SegmentReaders when segment replication is enabled during search.#1808
• Release memory properly for an array type #1820
• FIX Same Suffix Cause Recall Drop to zero #1802

Infrastructure

• Apply custom patch only once by comparing the last patch id #1833

Documentation

• Update dev guide to fix clang linking issue on arm #1746

Maintenance

• Bump faiss commit to 33c0ba5 #1796

2.15.0.0

Version 2.15.0.0 Release Notes

Compatible with OpenSearch 2.15.0

Features

• Use the Lucene Distance Calculation Function in Script Scoring for doing exact search #1699

Enhancements

• Add KnnCircuitBreakerException and modify exception message #1688
• Add stats for radial search #1684
• Support script score when doc value is disabled and fix misusing DISI #1696
• Add validation for pq m parameter before training starts #1713
• Block delete model requests if an index uses the model #1722

Bug Fixes

• Block commas in model description #1692
• Update threshold value after new result is added #1715

2.14.0.0

Version 2.14.0.0 Release Notes

Compatible with OpenSearch 2.14.0

Features

• Add k-NN clear cache api #740
• Support radial search in k-NN plugin #1617
• Support filter and nested field in faiss engine radial search #1652

Enhancements

• Make the HitQueue size more appropriate for exact search #1549
• Implement the Streaming Feature to stream vectors from Java to JNI layer to enable creation of larger segments for vector indices #1604
• Remove unnecessary toString conversion of vector field and added some minor optimization in KNNCodec 1613
• Serialize all models into cluster metadata #1499

Bug Fixes

• Add stored fields for knn_vector type #1630
• Enable script score to work with model based indices #1649

Infrastructure

• Add micro-benchmark module in k-NN plugin for benchmark streaming vectors to JNI layer functionality. #1583
• Add arm64 check when SIMD is disabled #1618
• Skip rebuild from scratch after cmake is run #1636

2.13.0.0

Version 2.13.0.0 Release Notes

Compatible with OpenSearch 2.13.0

Enhancements

• Optize Faiss Query With Filters: Reduce iteration and memory for id filter #1402
• Detect AVX2 Dynamically on the System #1502
• Validate zero vector when using cosine metric #1501
• Persist model definition in model metadata [#1527] (#1527)
• Added Inner Product Space type support for Lucene Engine #1551
• Add Range Validation for Faiss SQFP16 #1493
• SQFP16 Range Validation for Faiss IVF Models #1557

Bug Fixes

• Disable sdc table for HNSWPQ read-only indices #1518
• Switch SpaceType.INNERPRODUCT's vector similarity function to MAXIMUM_INNER_PRODUCT #1532
• Add patch to fix arm segfault in nmslib during ingestion #1541
• Share ivfpq-l2 table allocations across indices on load #1558

Infrastructure

• Manually install zlib for win CI #1513
• Update k-NN build artifact script to enable SIMD on ARM for Faiss #1543

Maintenance

• Bump faiss lib commit to 32f0e8cf92cd2275b60364517bb1cce67aa29a55 #1443
• Fix FieldInfo Parameters Mismatch #1490
• Upgrade faiss to 12b92e9 #1509

2.12.0.0

Version 2.12.0.0 Release Notes

Compatible with OpenSearch 2.12.0

Features

• Add parent join support for lucene knn #1182
• Add parent join support for faiss hnsw #1398

Enhancements

• Increase Lucene max dimension limit to 16,000 #1346
• Tuned default values for ef_search and ef_construction for better indexing and search performance for vector search #1353
• Enabled Filtering on Nested Vector fields with top level filters #1372
• Throw proper exception to invalid k-NN query #1380

Bug Fixes

• Fix use-after-free case on nmslib search path #1305
• Allow nested knn field mapping when train model #1318
• Properly designate model state for actively training models when nodes crash or leave cluster #1317
• Fix script score queries not getting cached #1367
• Fix KNNScorer to apply boost #1403
• Fix equals and hashCode methods for KNNQuery and KNNQueryBuilder #1397
• Pass correct value on IDSelectorBitmap initialization #1444

Infrastructure

• Upgrade gradle to 8.4 1289
• Refactor security testing to install from individual components #1307
• Refactor integ tests that access model index #1423
• Fix flaky model tests #1429

Maintenance

• Update developer guide to include M1 Setup #1222
• Upgrade urllib to 1.26.17 #1278
• Upgrade urllib to 1.26.18 #1319
• Upgrade guava to 32.1.3 #1319
• Bump lucene codec to 99 #1383
• Update spotless and eclipse dependencies #1450

2.11.0.0

Version 2.11.0.0 Release Notes

Compatible with OpenSearch 2.11.0

Enhancements

• Added support for ignore_unmapped in KNN queries. #1071
• Add graph creation stats to the KNNStats API. #1141

Maintenance

• Update bytebuddy to 1.14.7 #1135

2.10.0.0

Version 2.10.0.0 Release Notes

Compatible with OpenSearch 2.10.0

Features

• Add Clear Cache API (#740)

Enhancements

• Enabled the IVF algorithm to work with Filters of K-NN Query. (#1013)
• Improved the logic to switch to exact search for restrictive filters search for better recall. (#1059)
• Added max distance computation logic to enhance the switch to exact search in filtered Nearest Neighbor Search. (#1066)

Bug Fixes

• Update Faiss parameter construction to allow HNSW+PQ to work (#1074)

Maintenance

• Update Guava Version to 32.0.1 (#1019)

Refactoring

• Fix TransportAddress Refactoring Changes in Core (#1020)

2.9.0.0

Version 2.9.0.0 Release Notes

Compatible with OpenSearch 2.9.0

Features

• Added support for Efficient Pre-filtering for Faiss Engine (#936)
• Add Support for Lucene Byte Sized Vector (#971)