Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VectorDBBench running on Milvus DB giving error - Resource temporarily unavailable #368

Open
kaminis85 opened this issue Sep 17, 2024 · 8 comments
Open

VectorDBBench running on Milvus DB giving error - Resource temporarily unavailable #368

kaminis85 opened this issue Sep 17, 2024 · 8 comments
Assignees
@alwayslove2013

Comments

@kaminis85
Copy link

We are running a VectorDB benchmark test on Milvus DB. Below are the specification of the test ran-
Database - Milvus
Case - Search Performance Test (500K Dataset, 1536 Dim)
Index type - DiskANN
search_list - 100 (default)

Sharing the full log here-

#init_bench
2024-09-17 09:59:38,437 | INFO: all configs: [('ALIYUN_OSS_URL', 'assets.zilliz.com.cn/benchmark/'), ('AWS_S3_URL', 'assets.zilliz.com/benchmark/'), ('CONCURRENCY_DURATION', 30), ('CONFIG_LOCAL_DIR', PosixPath('/usr/local/lib/python3.11/site-packages/vectordb_bench/config-files')), ('CUSTOM_CONFIG_DIR', PosixPath('/usr/local/lib/python3.11/site-packages/vectordb_bench/custom/custom_case.json')), ('DATASET_LOCAL_DIR', PosixPath('/tmp/vectordb_bench/dataset')), ('DEFAULT_DATASET_URL', 'assets.zilliz.com/benchmark/'), ('DROP_OLD', True), ('K_DEFAULT', 100), ('LOG_LEVEL', 'INFO'), ('NUM_CONCURRENCY', [1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100]), ('NUM_PER_BATCH', 5000), ('RESULTS_LOCAL_DIR', PosixPath('/usr/local/lib/python3.11/site-packages/vectordb_bench/results')), ('USE_SHUFFLED_DATA', True)] (main.py:11) (15519)

Collecting usage statistics. To deactivate, set browser.gatherUsageStats to false.

You can now view your Streamlit app in your browser.

Local URL: http://localhost:8501
Network URL: http://10.129.4.79:8501
External URL: http://192.133.242.70:8501

2024-09-17 10:01:34,491 | INFO: generated uuid for the tasks: e2aaa4749e1b41499c63d75622b32a74 (interface.py:66) (15520)
2024-09-17 10:01:34,552 | INFO | DB | CaseType Dataset Filter | task_label (task_runner.py:338)
2024-09-17 10:01:34,552 | INFO | ----------- | ------------ -------------------- ------- | ------- (task_runner.py:338)
2024-09-17 10:01:34,552 | INFO | Milvus | Performance OpenAI-MEDIUM-500K None | test2_nfs_17 (task_runner.py:338)
2024-09-17 10:01:34,552 | INFO: task submitted: id=e2aaa4749e1b41499c63d75622b32a74, test2_nfs_17, case number: 1 (interface.py:231) (15520)
2024-09-17 10:01:34.775 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:01:35,220 | INFO: [1/1] start case: {'label': <CaseLabel.Performance: 2>, 'dataset': {'data': {'name': 'OpenAI', 'size': 500000, 'dim': 1536, 'metric_type': <MetricType.COSINE: 'COSINE'>}}, 'db': 'Milvus'}, drop_old=True (interface.py:164) (15635)
2024-09-17 10:01:35,220 | INFO: Starting run (task_runner.py:100) (15635)
2024-09-17 10:01:35,297 | INFO: Milvus client drop_old collection: VectorDBBenchCollection (milvus.py:45) (15635)
2024-09-17 10:01:35,302 | INFO: Milvus create collection: VectorDBBenchCollection (milvus.py:55) (15635)
2024-09-17 10:01:35,950 | INFO: local dataset root path not exist, creating it: /tmp/vectordb_bench/dataset/openai/openai_medium_500k (data_source.py:119) (15635)
2024-09-17 10:01:35,950 | INFO: Start to downloading files, total count: 3 (data_source.py:135) (15635)
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [02:20<00:00, 46.86s/it]
2024-09-17 10:03:56,531 | INFO: Succeed to download all files, downloaded file count = 3 (data_source.py:140) (15635)
2024-09-17 10:03:56,531 | INFO: Read the entire file into memory: test.parquet (dataset.py:229) (15635)
2024-09-17 10:03:56,665 | INFO: Read the entire file into memory: neighbors.parquet (dataset.py:229) (15635)
2024-09-17 10:03:56,690 | INFO: Start performance case (task_runner.py:158) (15635)
2024-09-17 10:03:56.912 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:03:57,480 | INFO: (SpawnProcess-1:1) Start inserting embeddings in batch 5000 (serial_runner.py:35) (15962)
2024-09-17 10:03:57,480 | INFO: Get iterator for shuffle_train.parquet (dataset.py:247) (15962)
2024-09-17 10:04:37,844 | INFO: (SpawnProcess-1:1) Loaded 100000 embeddings into VectorDB (serial_runner.py:59) (15962)
2024-09-17 10:05:20,985 | INFO: (SpawnProcess-1:1) Loaded 200000 embeddings into VectorDB (serial_runner.py:59) (15962)
2024-09-17 10:06:12,291 | INFO: (SpawnProcess-1:1) Loaded 300000 embeddings into VectorDB (serial_runner.py:59) (15962)
2024-09-17 10:07:06,173 | INFO: (SpawnProcess-1:1) Loaded 400000 embeddings into VectorDB (serial_runner.py:59) (15962)
2024-09-17 10:07:58,594 | INFO: (SpawnProcess-1:1) Loaded 500000 embeddings into VectorDB (serial_runner.py:59) (15962)
2024-09-17 10:07:58,603 | INFO: (SpawnProcess-1:1) Finish loading all dataset into VectorDB, dur=241.12337821099936 (serial_runner.py:61) (15962)
2024-09-17 10:07:59.174 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:08:00,333 | INFO: Milvus post insert before optimize (milvus.py:101) (18207)
2024-09-17 10:18:08,647 | INFO: Milvus optimizing before search (milvus.py:93) (18207)
2024-09-17 10:18:18,257 | INFO: Finish loading the entire dataset into VectorDB, insert_duration=242.15658646000156, optimize_duration=617.8485961160004 load_duration(insert + optimize) = 860.0052 (task_runner.py:167) (15635)
2024-09-17 10:18:18.525 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:18:19,117 | INFO: SpawnProcess-1:3 start search the entire test_data to get recall and latency (serial_runner.py:171) (19339)
2024-09-17 10:18:52,124 | INFO: SpawnProcess-1:3 search entire test_data: cost=25.8579s, queries=1000, avg_recall=0.9896, avg_ndcg=0.9911,avg_latency=0.0259, p99=0.0557 (serial_runner.py:209) (19339)
2024-09-17 10:18:52.526 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:18:52,972 | INFO: Start search 30s in concurrency 1, filters: None (mp_runner.py:99) (15635)
2024-09-17 10:18:53.196 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing
2024-09-17 10:19:29,975 | INFO: Syncing all process and start concurrency search, concurrency=5 (mp_runner.py:108) (15635)
2024-09-17 10:19:59,998 | INFO: SpawnProcess-1:8 search 30s: actual_dur=30.0072s, count=1636, qps in this process: 54.5202 (mp_runner.py:76) (20009)
2024-09-17 10:20:00,006 | INFO: SpawnProcess-1:7 search 30s: actual_dur=30.0141s, count=1626, qps in this process: 54.1745 (mp_runner.py:76) (20006)
2024-09-17 10:20:00,008 | INFO: SpawnProcess-1:11 search 30s: actual_dur=30.0159s, count=1629, qps in this process: 54.2712 (mp_runner.py:76) (20012)
2024-09-17 10:20:00,010 | INFO: SpawnProcess-1:10 search 30s: actual_dur=30.0189s, count=1631, qps in this process: 54.3324 (mp_runner.py:76) (20011)
2024-09-17 10:20:00,011 | INFO: SpawnProcess-1:9 search 30s: actual_dur=30.0186s, count=1630, qps in this process: 54.2997 (mp_runner.py:76) (20010)
2024-09-17 10:20:00,013 | INFO: End search in concurrency 5: dur=30.037750658000732s, total_count=8152, qps=271.3918 (mp_runner.py:120) (15635)
2024-09-17 10:20:00,362 | INFO: Update largest qps with concurrency 5: current max_qps=271.3918 (mp_runner.py:124) (15635)
2024-09-17 10:20:00.585 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:20:01,031 | INFO: Start search 30s in concurrency 10, filters: None (mp_runner.py:99) (15635)
2024-09-17 10:20:11,068 | INFO: Syncing all process and start concurrency search, concurrency=10 (mp_runner.py:108) (15635)
2024-09-17 10:20:41,098 | INFO: SpawnProcess-1:18 search 30s: actual_dur=30.011s, count=885, qps in this process: 29.4892 (mp_runner.py:76) (20990)
2024-09-17 10:20:41,099 | INFO: SpawnProcess-1:13 search 30s: actual_dur=30.0114s, count=887, qps in this process: 29.5554 (mp_runner.py:76) (20983)
2024-09-17 10:20:41,104 | INFO: SpawnProcess-1:15 search 30s: actual_dur=30.0177s, count=895, qps in this process: 29.8157 (mp_runner.py:76) (20987)
2024-09-17 10:20:41,104 | INFO: SpawnProcess-1:17 search 30s: actual_dur=30.0173s, count=892, qps in this process: 29.7162 (mp_runner.py:76) (20989)
2024-09-17 10:20:41,104 | INFO: SpawnProcess-1:21 search 30s: actual_dur=30.0171s, count=891, qps in this process: 29.6831 (mp_runner.py:76) (20993)
2024-09-17 10:20:41,105 | INFO: SpawnProcess-1:22 search 30s: actual_dur=30.0176s, count=892, qps in this process: 29.7159 (mp_runner.py:76) (20994)
2024-09-17 10:20:41,106 | INFO: SpawnProcess-1:20 search 30s: actual_dur=30.0184s, count=890, qps in this process: 29.6485 (mp_runner.py:76) (20992)
2024-09-17 10:20:41,106 | INFO: SpawnProcess-1:19 search 30s: actual_dur=30.0193s, count=894, qps in this process: 29.7808 (mp_runner.py:76) (20991)
2024-09-17 10:20:41,106 | INFO: SpawnProcess-1:16 search 30s: actual_dur=30.0177s, count=885, qps in this process: 29.4826 (mp_runner.py:76) (20988)
2024-09-17 10:20:41,106 | INFO: SpawnProcess-1:14 search 30s: actual_dur=30.0178s, count=890, qps in this process: 29.6491 (mp_runner.py:76) (20986)
2024-09-17 10:20:41,108 | INFO: End search in concurrency 10: dur=30.040207265999925s, total_count=8901, qps=296.3029 (mp_runner.py:120) (15635)
2024-09-17 10:20:41,512 | INFO: Update largest qps with concurrency 10: current max_qps=296.3029 (mp_runner.py:124) (15635)
2024-09-17 10:20:41.734 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:20:42,178 | INFO: Start search 30s in concurrency 15, filters: None (mp_runner.py:99) (15635)
2024-09-17 10:20:42.420 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:20:52,215 | INFO: Syncing all process and start concurrency search, concurrency=15 (mp_runner.py:108) (15635)
2024-09-17 10:21:22,247 | INFO: SpawnProcess-1:31 search 30s: actual_dur=30.0064s, count=578, qps in this process: 19.2626 (mp_runner.py:76) (22862)
2024-09-17 10:21:22,248 | INFO: SpawnProcess-1:27 search 30s: actual_dur=30.005s, count=578, qps in this process: 19.2635 (mp_runner.py:76) (22858)
2024-09-17 10:21:22,249 | INFO: SpawnProcess-1:38 search 30s: actual_dur=30.009s, count=577, qps in this process: 19.2276 (mp_runner.py:76) (22869)
2024-09-17 10:21:22,250 | INFO: SpawnProcess-1:34 search 30s: actual_dur=30.0074s, count=581, qps in this process: 19.3619 (mp_runner.py:76) (22865)
2024-09-17 10:21:22,253 | INFO: SpawnProcess-1:36 search 30s: actual_dur=30.0156s, count=574, qps in this process: 19.1234 (mp_runner.py:76) (22867)
2024-09-17 10:21:22,254 | INFO: SpawnProcess-1:24 search 30s: actual_dur=30.0121s, count=576, qps in this process: 19.1923 (mp_runner.py:76) (22853)
2024-09-17 10:21:22,255 | INFO: SpawnProcess-1:29 search 30s: actual_dur=30.012s, count=577, qps in this process: 19.2256 (mp_runner.py:76) (22860)
2024-09-17 10:21:22,256 | INFO: SpawnProcess-1:33 search 30s: actual_dur=30.0164s, count=581, qps in this process: 19.3561 (mp_runner.py:76) (22864)
2024-09-17 10:21:22,260 | INFO: SpawnProcess-1:32 search 30s: actual_dur=30.0173s, count=584, qps in this process: 19.4554 (mp_runner.py:76) (22863)
2024-09-17 10:21:22,261 | INFO: SpawnProcess-1:28 search 30s: actual_dur=30.0191s, count=587, qps in this process: 19.5542 (mp_runner.py:76) (22859)
2024-09-17 10:21:22,263 | INFO: SpawnProcess-1:25 search 30s: actual_dur=30.0217s, count=583, qps in this process: 19.4193 (mp_runner.py:76) (22856)
2024-09-17 10:21:22,264 | INFO: SpawnProcess-1:26 search 30s: actual_dur=30.0245s, count=576, qps in this process: 19.1843 (mp_runner.py:76) (22857)
2024-09-17 10:21:22,264 | INFO: SpawnProcess-1:37 search 30s: actual_dur=30.0236s, count=587, qps in this process: 19.5513 (mp_runner.py:76) (22868)
2024-09-17 10:21:22,264 | INFO: SpawnProcess-1:35 search 30s: actual_dur=30.0258s, count=581, qps in this process: 19.35 (mp_runner.py:76) (22866)
2024-09-17 10:21:22,266 | INFO: SpawnProcess-1:30 search 30s: actual_dur=30.0242s, count=576, qps in this process: 19.1845 (mp_runner.py:76) (22861)
2024-09-17 10:21:22,267 | INFO: End search in concurrency 15: dur=30.05192263400022s, total_count=8696, qps=289.3658 (mp_runner.py:120) (15635)
2024-09-17 10:21:22.933 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:21:23,382 | INFO: Start search 30s in concurrency 20, filters: None (mp_runner.py:99) (15635)
2024-09-17 10:21:23.624 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:21:23.705 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing
2024-09-17 10:21:33,434 | INFO: Syncing all process and start concurrency search, concurrency=20 (mp_runner.py:108) (15635)
2024-09-17 10:22:03,471 | INFO: SpawnProcess-1:40 search 30s: actual_dur=30.0123s, count=437, qps in this process: 14.5607 (mp_runner.py:76) (25609)
2024-09-17 10:22:03,472 | INFO: SpawnProcess-1:50 search 30s: actual_dur=30.0141s, count=435, qps in this process: 14.4932 (mp_runner.py:76) (25621)
2024-09-17 10:22:03,474 | INFO: SpawnProcess-1:54 search 30s: actual_dur=30.0157s, count=433, qps in this process: 14.4258 (mp_runner.py:76) (25625)
2024-09-17 10:22:03,474 | INFO: SpawnProcess-1:59 search 30s: actual_dur=30.0168s, count=435, qps in this process: 14.4919 (mp_runner.py:76) (25630)
2024-09-17 10:22:03,475 | INFO: SpawnProcess-1:44 search 30s: actual_dur=30.0151s, count=438, qps in this process: 14.5927 (mp_runner.py:76) (25615)
2024-09-17 10:22:03,476 | INFO: SpawnProcess-1:45 search 30s: actual_dur=30.018s, count=435, qps in this process: 14.4913 (mp_runner.py:76) (25616)
2024-09-17 10:22:03,478 | INFO: SpawnProcess-1:58 search 30s: actual_dur=30.0182s, count=433, qps in this process: 14.4246 (mp_runner.py:76) (25629)
2024-09-17 10:22:03,480 | INFO: SpawnProcess-1:49 search 30s: actual_dur=30.0205s, count=436, qps in this process: 14.5234 (mp_runner.py:76) (25620)
2024-09-17 10:22:03,482 | INFO: SpawnProcess-1:51 search 30s: actual_dur=30.0238s, count=440, qps in this process: 14.655 (mp_runner.py:76) (25622)
2024-09-17 10:22:03,483 | INFO: SpawnProcess-1:57 search 30s: actual_dur=30.0271s, count=439, qps in this process: 14.6201 (mp_runner.py:76) (25628)
2024-09-17 10:22:03,486 | INFO: SpawnProcess-1:48 search 30s: actual_dur=30.0287s, count=439, qps in this process: 14.6193 (mp_runner.py:76) (25619)
2024-09-17 10:22:03,488 | INFO: SpawnProcess-1:46 search 30s: actual_dur=30.0304s, count=435, qps in this process: 14.4853 (mp_runner.py:76) (25617)
2024-09-17 10:22:03,489 | INFO: SpawnProcess-1:53 search 30s: actual_dur=30.0343s, count=435, qps in this process: 14.4834 (mp_runner.py:76) (25624)
2024-09-17 10:22:03,493 | INFO: SpawnProcess-1:43 search 30s: actual_dur=30.0324s, count=440, qps in this process: 14.6508 (mp_runner.py:76) (25614)
2024-09-17 10:22:03,494 | INFO: SpawnProcess-1:52 search 30s: actual_dur=30.0388s, count=438, qps in this process: 14.5811 (mp_runner.py:76) (25623)
2024-09-17 10:22:03,498 | INFO: SpawnProcess-1:47 search 30s: actual_dur=30.0409s, count=434, qps in this process: 14.447 (mp_runner.py:76) (25618)
2024-09-17 10:22:03,498 | INFO: SpawnProcess-1:55 search 30s: actual_dur=30.0375s, count=433, qps in this process: 14.4153 (mp_runner.py:76) (25626)
2024-09-17 10:22:03,498 | INFO: SpawnProcess-1:41 search 30s: actual_dur=30.0386s, count=439, qps in this process: 14.6145 (mp_runner.py:76) (25612)
2024-09-17 10:22:03,499 | INFO: SpawnProcess-1:42 search 30s: actual_dur=30.0421s, count=432, qps in this process: 14.3798 (mp_runner.py:76) (25613)
2024-09-17 10:22:03,499 | INFO: SpawnProcess-1:56 search 30s: actual_dur=30.0395s, count=437, qps in this process: 14.5475 (mp_runner.py:76) (25627)
2024-09-17 10:22:03,501 | INFO: End search in concurrency 20: dur=30.067121991000022s, total_count=8723, qps=290.1176 (mp_runner.py:120) (15635)
2024-09-17 10:22:04.157 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:22:04,613 | INFO: Start search 30s in concurrency 25, filters: None (mp_runner.py:99) (15635)
2024-09-17 10:22:04.868 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:22:04.868 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:22:14,655 | INFO: Syncing all process and start concurrency search, concurrency=25 (mp_runner.py:108) (15635)
OpenBLAS blas_thread_init: pthread_create failed for thread 22 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4194304 current, 4194304 max
OpenBLAS blas_thread_init: pthread_create failed for thread 23 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4194304 current, 4194304 max

Usingpymilvus, we validated that the db connection, creation of collections and indexing are successful.
@alwayslove2013
Copy link
Collaborator

@kaminis85 This error seems to indicate a performance bottleneck on the machine running VectorDBBench rather than Milvus.

I would like to confirm the following:

  • Is the machine running VectorDBBench the same as the one deploying Milvus?
  • If not, how many CPU cores and how much memory does the machine running VectorDBBench have?

@alwayslove2013 alwayslove2013 self-assigned this Sep 18, 2024
@kaminis85
Copy link
Author

@alwayslove2013 Thanks for replying here.

  1. Both Milvus and Vector DB are running as pods on worker nodes. Vector DB pod is running on ocp-worker2 node. Milvus data node pod is running on ocp-worker0 node.
  2. ocp-worker2 node where VectorDB pod is running, has 128 CPU cores. Out of which 127.2 is available

Not sure what is causing here performance bottleneck.

@alwayslove2013
Copy link
Collaborator

@kaminis85 When testing with a concurrency of 20, it was successful, but it failed at 25. From the logs, it appears that the client running VectorDBBench encountered a resource bottleneck. The specific cause is still unclear. It is recommended to deploy VectorDBBench and Milvus on different machines for further testing.

2024-09-17 10:22:03,501 | INFO: End search in concurrency 20: dur=30.067121991000022s, total_count=8723, qps=290.1176 (mp_runner.py:120) (15635)
...
2024-09-17 10:22:14,655 | INFO: Syncing all process and start concurrency search, concurrency=25 (mp_runner.py:108) (15635)
OpenBLAS blas_thread_init: pthread_create failed for thread 22 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4194304 current, 4194304 max

@kaminis85
Copy link
Author

kaminis85 commented Sep 19, 2024 •
edited
Loading

@alwayslove2013
We are running VectorDB bench test on one of the worker nodes that is running RHCOS. We suspect it has to do something with OS here which is RHCOS, as the workload is getting crashed after some time, and sometimes we get some issue to install some python dependencies.
Is there any specific requirement while running VectorDB test on RHCOS?

@alwayslove2013
Copy link
Collaborator

@kaminis85 Sorry but we don't have any debugging experience with RHCOS. If you have any findings, feel free to share them with us~

@kaminis85
Copy link
Author

Sure @alwayslove2013
Thanks!

@kaminis85
Copy link
Author

kaminis85 commented Sep 23, 2024 •
edited
Loading

@alwayslove2013
Steps to run performance test with VectorDB Bench (Milvus DB on OCP)-

  1. Deployed Milvus DB on OCP Cluster (with GPUs)
  2. Edited Milvus service from ClusterIP to NodePort
  3. Created a route for Milvus service with target port
  4. We installed VectorDB bench tool on OCP Installer VM that is running RHEL 9.4
  5. Somehow Vector DB GUI was not loading when we ran on OCP installer VM. So, had to disable the firewall
  6. To connect Milvus DB from VectorDB Bench, used one of the worker nodes IP and port (can check under Milvus service after adding the NodePort)

After executing above steps, the tests were running fine.

@alwayslove2013
Copy link
Collaborator

@kaminis85 Excellent! I want to express my heartfelt gratitude for sharing your successful experience with VectorDBBench. Your insights not only highlight the effectiveness of our work but also inspire others in the community~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alwayslove2013 alwayslove2013

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@alwayslove2013 @kaminis85