-
-
Notifications
You must be signed in to change notification settings - Fork 219
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reduce disk space with CouchDB Nouveau (TCO) #9542
Comments
I'll try to answer here the questions asked in the original issue and any question that might come up along the way.
Much much faster than our regular couch views! I remade the TODO: also index
TBD
Easy using the docker image the couchdb team publishes.
It's all contained in a single docker volume, it's easy to check manually.
First, upgrading nouveau alongside couchdb right now can break existing indexes. It broke during the 3.4.1 => 3.4.2 upgrade and the couchdb team quickly fixed the underlying problem. With that said, they are committed to not cause a nouveau view reindexing on upgrades and are working towards automatic view reindexing, from their slack:
So hopefully upgrades will be smooth when nouveau stabilizes. And second, changing the view by modifying the code and index documents differently should be as straightforward as changing a regular couch view. It's essentially a JS function that lives in a ddoc, same as regular views we already have.
TL;DR: I saw ~25% disk savings for MoH Zanzibar. Starting with the existing snapshot of MoH Zanzibar after compaction and views cleanup, disk usage of CouchDB was 55GB. I haven't noticed Nouveau's disk usage going above 1GB during the 5 minutes it took to index the views but I'm planning to make changes to https://github.com/jkuester/chtoolbox/ to monitor that as well. As mentioned earlier, couch exposes an API to help with this. This will come in handy on top of our current disk monitoring functionality.
var order = dead + ' ' + muted + ' ' + idx + ' ' + (doc.name && doc.name.toLowerCase()); It seems to be used to put the dead and muted contacts at the end of the search results and order the results alphabetically.
|
this is great - thanks @m5r ! One question that didn't ask, which i've just added to the body is: How much disk savings do we see? In more detail: If we start with the freetext view we have now for online users, delete it, recreate it in Nouveau, what's the savings? As well, when upgrading/recreating the index in Nouveau, how much spare ephemeral disk do we need? Do we know the percent of total disk the freetext view takes up so we can try and compare to what Nouveau will take up? Feel free to break this into it's own sub-ticket if the research seems a rabbit hole unto itself! |
Great question @mrjones-plip 👀 I updated my previous comment with this week's updates and an answer to your question |
Thanks for the write up @m5r !
To be clear, this is a 25% savings on the
yes! i love this idea. |
No this is a 25% savings on all couchdb + Nouveau. |
wow!!! awesome news - thanks Mokhtar |
Aside from the sub-issues already assigned to this issue, we're also dependent on apache/couchdb#5354 being released |
What feature do you want to improve?
Reduce total disk space that CHT takes up, thus reduce the total cost of ownership (TCO)
Describe the improvement you'd like
Use Nouveau search instead of existing CouchDB search
Describe alternatives you've considered
refactoring/removing freetext views
Additional context
We should answer how viable it will be to use CouchDB's Nouveau search to both improve search but to also reduce disk use of the indexes. This should include, but not be limited to:
The text was updated successfully, but these errors were encountered: