Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chronicle log files use a lot of disk space #115

Open
etedpet opened this issue Aug 23, 2019 · 2 comments
Open

Chronicle log files use a lot of disk space #115

etedpet opened this issue Aug 23, 2019 · 2 comments
Labels
enhancement New feature or request

Comments

@etedpet
Copy link
Contributor

etedpet commented Aug 23, 2019

We should investigate the possibility to make Chronicle disk footprint smaller.
Comparing the log size of the audit files created when running the run_ccm_performance_test.sh (runs cassandra-stress with 3 million requests), we can see that the SLF4J logfiles take ~ 1.6G disc space, while Chronicle take ~2.1G.

We should investigate ways to lower this... A quick test showed that shortening the WireTag keys (the Strings describing the fields when writing data) made the usage much lower ~ 1.8G.
Maybe we don't even need those tags (not all of them)?

@etedpet
Copy link
Contributor Author

etedpet commented Aug 26, 2019

If no fields are needed, then a "fieldless" ChronicleQueue could be used...
ChronicleQueue chronicle = SingleChronicleQueueBuilder.fieldlessBinary(config.getLogPath().toFile())...

@eperott eperott added the enhancement New feature or request label Sep 26, 2019
@etedpet
Copy link
Contributor Author

etedpet commented Oct 3, 2019

Ran some more tests regarding the Chronicle log file size, when comparing ecAudit with the audit logging in C4.0. Below table compares the log size produced by different setups.

Size (kB) | Setup | Diff
1832532 | ecAudit (default setup - including bound values to prepared statements) | 260%
774808 | C4.0 (have some more fields than below) | 110%
703348 | ecAudit (with OPERATION_NAKED - most similar to C4.0) | 100% <---Use this as baseline
691428 | ecAudit (OPERATION_NAKED + write single field to chronicle - Similar to C4.0) | 98%
486184 | ecAudit (OPERATION_NAKED + short field-names) | 69%
438832 | ecAudit (OPERATION_NAKED + NO field-names) | 62%
408876 | ecAudit (OPERATION_NAKED + fieldless ChronicleQueue) | 58%

  • Example of the "default" log entry (printout from eclog):
    1570025092487|127.0.0.1:45176|127.0.0.1|cassandra|ATTEMPT|UPDATE "standard1" SET "C0" = ?,"C1" = ?,"C2" = ?,"C3" = ?,"C4" = ? WHERE KEY=?[0x28029cc8622a9724bf1cf40452459cffcf8d59c9ab5ebb00c91ff72de420d1a346b9, 0x6b986a3cb238eafd3953c7109ca2b92c7ece025076ffa573623b7994a915fb9ea209, 0x11c6d2fb29c9f643dae00c72e9f79a9f0858fd93bf1bbd79cd24d4124971a99ba9e8, 0xdb2591aed6efa7b9081965abf9eb5e719714b3478036afb640a7997ac149a10aa4df, 0x286882bb1dc0a02ed00d336effa840144ec533361247b50bc551a7f8e93b06f99ed8, 0x36384c384b314b313930]

  • Corresponding example with "OPERATION_NAKED" configured:
    1570106774994|127.0.0.1:55944|127.0.0.1|cassandra|ATTEMPT|UPDATE "standard1" SET "C0" = ?,"C1" = ?,"C2" = ?,"C3" = ?,"C4" = ? WHERE KEY=?

@RhoderickGalero RhoderickGalero moved this to Todo issue in C2T Sep 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants