Optimize pgvector test for semi-recent enhancements

[jkatz]

This commit adds several changes to the pgvector test to create a more representative test environment based on recent and older changes to pgvector. Notable changes include allowing for testing of parallel index buiding parameters, using loading with the recommended binary loading method, and other changes to better emulate what a typical user of pgvector would do.

This commit also has some general cleanups as well.

Co-authored-by: Mark Greenhalgh [email protected]
Co-authored-by: Tyler House [email protected]

[XuanYang-cn]

/assign @alwayslove2013
/assign

[jkatz]

@XuanYang-cn @alwayslove2013 Please let us know if this PR requires additional work. There are some other changes we'd like to include for testing other configurations of pgvector, but we'd like to baseline it against the flat implementation first. Thanks!

[alwayslove2013]

Recommend avoiding adding specific dependencies in the config.py. Users only install the corresponding toolkit when they are conducting tests.

However, in the scenario where the default results page is opened solely for result display, VDBBench will load the standardized result (json). Serializing this data requires the config.py file from all clients.

Currently, if a user hasn't installed psycopg, they won't be able to open the results page.

[jkatz]

This is fixed in ea29f47

[jkatz]

@alwayslove2013 Thanks for the feedback! This is resolved in the latest push.

Overall, I would suggest moving to psycopg3 (psycopg) as it's now the maintained version of psycopg; however, that change could be made in a separate pull request.

[alwayslove2013]

@jkatz Thank you so much for your contribution! We greatly appreciate it and are thrilled to receive your pull request. We look forward to collaborating with you and driving the project forward together!

Overall, I would suggest moving to psycopg3 (psycopg) as it's now the maintained version of psycopg; however, that change could be made in a separate pull request.

[jkatz]

@alwayslove2013 Likewise. I personally appreciate the approach VectorDBBench takes around testing concurrency, which resembles how users interact with databases.

I've pushed up the fix to the latest patch to handle the merge conflict that remained (which I'm still baffled how that got in, but I'll triple check next time).

[alwayslove2013]

/approve

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: alwayslove2013, jkatz
To complete the pull request process, please assign xuanyang-cn after the PR has been reviewed.
You can assign the PR to them by writing /assign @xuanyang-cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[alwayslove2013]

@jkatz I would like to express my sincere gratitude for your support. Our primary goal has been to ensure that the test data reflects the performance characteristics of the real-world usage scenarios as accurately as possible.

... testing concurrency, which resembles how users interact with databases.

If you have any suggestions or innovative ideas with VDBBench, we would be more than happy to discuss them with you. Your valuable input is crucial for us to enhance the functionality and user experience of the tool.

[wahajali]

@jkatz I'm trying to see how we can optimize the pgvector and pgvecto.rs client further and one thought I had was to use functions for making vector queries. Do you think its something to explore further?
I've previously seen something similar for HammerDB where they use stored procedures (although its not for vector benchmarking): https://www.hammerdb.com/blog/uncategorized/why-you-should-benchmark-your-database-using-stored-procedures/

[greenhal]

This benchmark is pretty simple, run the query, get the result, compare the result.

To improve the performance, using stored procedures you would have to either reduce network round trips, or reduce the data transmitted, neither of which is much overhead in this benchmark. Using stored procedures HammerDB does both of these, because there is a lot of potential network traffic associated with a tpcc benchmark, for example see new order transactions, to create a new order (the main tpcc performance measurement), it could take up to 6 round trips to create an order, if all of the logic was on the client, using stored procedures in this case, it's 1 or even less, because hammer db can send a single request that says, create 100 orders.

With that in mind, there could be 2 ways that could possibly improve pgvector performance, or any engines performance. (which could be implemented with stored procedures.)

1. Reduce the network trips.
   This would be accomplished by batching the request/results, instead of sending 1 vector to query, send 100 or even all 1k, then return all of the results. But since the same amount of data would be sent/received. The only savings would be the reduce round trips and the overhead associated with those round trips and the sending/receiving would still be done over multiple trips. Overall, the benefit would be minimal, if any at all.

2. Reduce the data transferred.
   There are a few ways of doing this, the most extreme would be:

   • to load the data, ground truth and test vector onto the target
   • a "run benchmark" procedure on the target would run all the queries and just return the recall and qps.

Both would require a pretty significant change to vectordbbench, they would be engine specific and would not represent a real world use case.