Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sparse: accelerate the writing of index files #1017

Merged
merged 1 commit into from
Jan 9, 2025

Conversation

sparknack
Copy link
Contributor

@sparknack sparknack commented Jan 8, 2025

Since we remove forward index in #1011 , sparse row data for index file should reconstructed from inverted index. But it is too slow to construct the index file using the DAAT cursor method. Switch to the TAAT traversal method instead.

Copy link

mergify bot commented Jan 8, 2025

@sparknack 🔍 Important: PR Classification Needed!

For efficient project management and a seamless review process, it's essential to classify your PR correctly. Here's how:

  1. If you're fixing a bug, label it as kind/bug.
  2. For small tweaks (less than 20 lines without altering any functionality), please use kind/improvement.
  3. Significant changes that don't modify existing functionalities should be tagged as kind/enhancement.
  4. Adjusting APIs or changing functionality? Go with kind/feature.

For any PR outside the kind/improvement category, ensure you link to the associated issue using the format: “issue: #”.

Thanks for your efforts and contribution to the community!.

Copy link

codecov bot commented Jan 9, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.91%. Comparing base (3c46f4c) to head (52ee7e2).
Report is 284 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##           main    #1017       +/-   ##
=========================================
+ Coverage      0   73.91%   +73.91%     
=========================================
  Files         0       82       +82     
  Lines         0     6981     +6981     
=========================================
+ Hits          0     5160     +5160     
- Misses        0     1821     +1821     

see 82 files with indirect coverage changes

It is too slow to construct the index file using the DAAT cursor
method. Switch to the TAAT traversal method instead.

Signed-off-by: Shawn Wang <[email protected]>
@sparknack sparknack changed the title sparse: accelerate the writing of index files by using inverted index sparse: accelerate the writing of index files Jan 9, 2025
@mergify mergify bot removed the ci-passed label Jan 9, 2025
@zhengbuqian
Copy link
Collaborator

/lgtm
/approve

@sre-ci-robot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sparknack, zhengbuqian

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sparknack
Copy link
Contributor Author

issue: #967

@sparknack
Copy link
Contributor Author

/kind improvement

@sre-ci-robot sre-ci-robot merged commit 104b211 into zilliztech:main Jan 9, 2025
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants