You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In my access pattern [1], I notice that I now have a large and fast-growing metadata footprint that I would prefer to avoid [2], but which, if needed, I'll adapt.
When running gc --compact it seems that there are large numbers of pending slices (which gc --compact eliminates).
Hence, I wondered:
Is approximately linear rapid (and potentially not bounded) metadata growth expected given an access pattern as described [1]?
Would the metadata growth pattern be more likely explained by accumulated pending slices or normal "fragmentation" wherein I've accumulated a number of chunks where only a portion is active (since juicefs avoids merging as a design goal)?
I know that running an explicit gc --compact seems to address this, but is this something that would be expected to be cleaned up as part of client background maintenance?
[1] I'm storing a virtual hard disk using juicefs mount with writeback and upload-delay of 5s.
[2] After starting with metadata of 50 MB, my metadata steadily grew until recently reaching 2.5GB.
Here's a table-level view of my metadata payload (sqlite3_analyzer output):
/** Disk-Space Utilization Report
Page size in bytes................................ 4096
Pages in the whole file (measured)................ 574350
Pages in the whole file (calculated).............. 574350
Pages that store data............................. 502527 87.5%
Pages on the freelist (per header)................ 71822 12.5%
Pages on the freelist (calculated)................ 71822 12.5%
Pages of auto-vacuum overhead..................... 0 0.0%
Number of tables in the database.................. 20
Number of indices................................. 9
Number of defined indices......................... 7
Number of implied indices......................... 2
Size of the file in bytes......................... 2352537600
Bytes of user payload stored...................... 1519413804 64.6%
*** Page counts for all tables with their indices *****************************
JFS_CHUNK......................................... 288493 50.2%
JFS_CHUNK_REF..................................... 209230 36.4%
JFS_XATTR......................................... 3814 0.66%
JFS_NODE.......................................... 544 0.095%
JFS_EDGE.......................................... 424 0.074%
SQLITE_SCHEMA..................................... 3 0.0%
JFS_COUNTER....................................... 2 0.0%
JFS_FLOCK......................................... 2 0.0%
JFS_PLOCK......................................... 2 0.0%
JFS_SETTING....................................... 2 0.0%
JFS_SUSTAINED..................................... 2 0.0%
JFS_DELFILE....................................... 1 0.0%
JFS_DELSLICES..................................... 1 0.0%
JFS_DETACHED_NODE................................. 1 0.0%
JFS_DIR_QUOTA..................................... 1 0.0%
JFS_DIR_STATS..................................... 1 0.0%
JFS_SESSION2...................................... 1 0.0%
JFS_SYMLINK....................................... 1 0.0%
SQLITE_SEQUENCE................................... 1 0.0%
SQLITE_STAT1...................................... 1 0.0%
*** Page counts for all tables and indices separately *************************
JFS_CHUNK......................................... 288370 50.2%
JFS_CHUNK_REF..................................... 209230 36.4%
JFS_XATTR......................................... 3339 0.58%
JFS_NODE.......................................... 544 0.095%
UQE_JFS_XATTR_NAME................................ 475 0.083%
JFS_EDGE.......................................... 183 0.032%
UQE_JFS_EDGE_EDGE................................. 139 0.024%
UQE_JFS_CHUNK_CHUNK............................... 123 0.021%
IDX_JFS_EDGE_INODE................................ 102 0.018%
SQLITE_SCHEMA..................................... 3 0.0%
UQE_JFS_FLOCK_FLOCK............................... 1 0.0%
UQE_JFS_PLOCK_PLOCK............................... 1 0.0%
UQE_JFS_SUSTAINED_SUSTAINED....................... 1 0.0%
JFS_COUNTER....................................... 1 0.0%
JFS_DELFILE....................................... 1 0.0%
JFS_DELSLICES..................................... 1 0.0%
JFS_DETACHED_NODE................................. 1 0.0%
JFS_DIR_QUOTA..................................... 1 0.0%
JFS_DIR_STATS..................................... 1 0.0%
JFS_FLOCK......................................... 1 0.0%
JFS_PLOCK......................................... 1 0.0%
JFS_SESSION2...................................... 1 0.0%
JFS_SETTING....................................... 1 0.0%
JFS_SUSTAINED..................................... 1 0.0%
JFS_SYMLINK....................................... 1 0.0%
SQLITE_AUTOINDEX_JFS_COUNTER_1.................... 1 0.0%
SQLITE_AUTOINDEX_JFS_SETTING_1.................... 1 0.0%
SQLITE_SEQUENCE................................... 1 0.0%
SQLITE_STAT1...................................... 1 0.0%
*** All tables and indices ****************************************************
Percentage of total database...................... 87.5%
Number of entries................................. 55058670
Bytes of storage consumed......................... 2058350592
Bytes of payload.................................. 1522250773 74.0%
Bytes of metadata................................. 440965650 21.4%
Average payload per entry......................... 27.65
Average unused bytes per entry.................... 1.75
Average metadata per entry........................ 8.01
Average fanout.................................... 315.00
Maximum payload per entry......................... 3584194
Entries that use overflow......................... 4477 0.008%
Index pages used.................................. 693
Primary pages used................................ 218006
Overflow pages used............................... 283828
Total pages used.................................. 502527
Unused bytes on index pages....................... 456074 16.1%
Unused bytes on primary pages..................... 95636524 10.7%
Unused bytes on overflow pages.................... 158975 0.014%
Unused bytes on all pages......................... 96251573 4.7%
*** All tables ****************************************************************
Percentage of total database...................... 87.3%
Number of entries................................. 54870298
Bytes of storage consumed......................... 2054893568
Bytes of payload.................................. 1519418080 73.9%
Bytes of metadata................................. 440390442 21.4%
Average payload per entry......................... 27.69
Average unused bytes per entry.................... 1.75
Average metadata per entry........................ 8.03
Average fanout.................................... 318.00
Maximum payload per entry......................... 3584194
Entries that use overflow......................... 4477 0.008%
Index pages used.................................. 685
Primary pages used................................ 217170
Overflow pages used............................... 283828
Total pages used.................................. 501683
Unused bytes on index pages....................... 443774 15.8%
Unused bytes on primary pages..................... 95599701 10.7%
Unused bytes on overflow pages.................... 158975 0.014%
Unused bytes on all pages......................... 96202450 4.7%
*** All indices ***************************************************************
Percentage of total database...................... 0.15%
Number of entries................................. 188372
Bytes of storage consumed......................... 3457024
Bytes of payload.................................. 2832693 81.9%
Bytes of metadata................................. 575208 16.6%
Average payload per entry......................... 15.04
Average unused bytes per entry.................... 0.26
Average metadata per entry........................ 3.05
Average fanout.................................... 105.00
Maximum payload per entry......................... 77
Entries that use overflow......................... 0 0.0%
Index pages used.................................. 8
Primary pages used................................ 836
Overflow pages used............................... 0
Total pages used.................................. 844
Unused bytes on index pages....................... 12300 37.5%
Unused bytes on primary pages..................... 36823 1.1%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 49123 1.4%
*** Table JFS_CHUNK and all its indices ***************************************
Percentage of total database...................... 50.2%
Number of entries................................. 82266
Bytes of storage consumed......................... 1181667328
Bytes of payload.................................. 1173837192 99.34%
Bytes of metadata................................. 2724700 0.23%
Average payload per entry......................... 14268.80
Average unused bytes per entry.................... 75.64
Average metadata per entry........................ 33.12
Average fanout.................................... 358.00
Maximum payload per entry......................... 3584194
Entries that use overflow......................... 4477 5.4%
Index pages used.................................. 13
Primary pages used................................ 4652
Overflow pages used............................... 283828
Total pages used.................................. 288493
Unused bytes on index pages....................... 10652 20.0%
Unused bytes on primary pages..................... 6053213 31.8%
Unused bytes on overflow pages.................... 158975 0.014%
Unused bytes on all pages......................... 6222840 0.53%
*** Table JFS_CHUNK w/o any indices *******************************************
Percentage of total database...................... 50.2%
Number of entries................................. 41133
Bytes of storage consumed......................... 1181163520
Bytes of payload.................................. 1173463886 99.35%
Bytes of metadata................................. 2599829 0.22%
B-tree depth...................................... 3
Average payload per entry......................... 28528.53
Average unused bytes per entry.................... 151.15
Average metadata per entry........................ 63.21
Average fanout.................................... 378.00
Non-sequential pages.............................. 490 0.17%
Maximum payload per entry......................... 3584194
Entries that use overflow......................... 4477 10.9%
Index pages used.................................. 12
Primary pages used................................ 4530
Overflow pages used............................... 283828
Total pages used.................................. 288370
Unused bytes on index pages....................... 8524 17.3%
Unused bytes on primary pages..................... 6049710 32.6%
Unused bytes on overflow pages.................... 158975 0.014%
Unused bytes on all pages......................... 6217209 0.53%
*** Index UQE_JFS_CHUNK_CHUNK of table JFS_CHUNK ******************************
Percentage of total database...................... 0.021%
Number of entries................................. 41133
Bytes of storage consumed......................... 503808
Bytes of payload.................................. 373306 74.1%
Bytes of metadata................................. 124871 24.8%
B-tree depth...................................... 2
Average payload per entry......................... 9.08
Average unused bytes per entry.................... 0.14
Average metadata per entry........................ 3.04
Average fanout.................................... 123.00
Non-sequential pages.............................. 1 0.82%
Maximum payload per entry......................... 10
Entries that use overflow......................... 0 0.0%
Index pages used.................................. 1
Primary pages used................................ 122
Overflow pages used............................... 0
Total pages used.................................. 123
Unused bytes on index pages....................... 2128 52.0%
Unused bytes on primary pages..................... 3503 0.70%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 5631 1.1%
*** Table JFS_CHUNK_REF *******************************************************
Percentage of total database...................... 36.4%
Number of entries................................. 54681884
Bytes of storage consumed......................... 857006080
Bytes of payload.................................. 330409280 38.6%
Bytes of metadata................................. 436825863 51.0%
B-tree depth...................................... 4
Average payload per entry......................... 6.04
Average unused bytes per entry.................... 1.64
Average metadata per entry........................ 7.99
Average fanout.................................... 317.00
Non-sequential pages.............................. 5338 2.6%
Maximum payload per entry......................... 8
Entries that use overflow......................... 0 0.0%
Index pages used.................................. 659
Primary pages used................................ 208571
Overflow pages used............................... 0
Total pages used.................................. 209230
Unused bytes on index pages....................... 414125 15.3%
Unused bytes on primary pages..................... 89356812 10.5%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 89770937 10.5%
*** Table JFS_COUNTER and all its indices *************************************
Percentage of total database...................... 0.0%
Number of entries................................. 20
Bytes of storage consumed......................... 8192
Bytes of payload.................................. 357 4.4%
Bytes of metadata................................. 86 1.0%
Average payload per entry......................... 17.85
Average unused bytes per entry.................... 387.45
Average metadata per entry........................ 4.30
Maximum payload per entry......................... 26
Entries that use overflow......................... 0 0.0%
Primary pages used................................ 2
Overflow pages used............................... 0
Total pages used.................................. 2
Unused bytes on primary pages..................... 7749 94.6%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 7749 94.6%
*** Table JFS_COUNTER w/o any indices *****************************************
Percentage of total database...................... 0.0%
Number of entries................................. 10
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 192 4.7%
Bytes of metadata................................. 48 1.2%
B-tree depth...................................... 1
Average payload per entry......................... 19.20
Average unused bytes per entry.................... 385.60
Average metadata per entry........................ 4.80
Maximum payload per entry......................... 26
Entries that use overflow......................... 0 0.0%
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 3856 94.1%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 3856 94.1%
*** Index SQLITE_AUTOINDEX_JFS_COUNTER_1 of table JFS_COUNTER *****************
Percentage of total database...................... 0.0%
Number of entries................................. 10
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 165 4.0%
Bytes of metadata................................. 38 0.93%
B-tree depth...................................... 1
Average payload per entry......................... 16.50
Average unused bytes per entry.................... 389.30
Average metadata per entry........................ 3.80
Maximum payload per entry......................... 23
Entries that use overflow......................... 0 0.0%
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 3893 95.0%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 3893 95.0%
*** Table JFS_DELFILE *********************************************************
Percentage of total database...................... 0.0%
Number of entries................................. 0
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 0 0.0%
Bytes of metadata................................. 8 0.20%
B-tree depth...................................... 1
Average payload per entry......................... 0.0
Average unused bytes per entry.................... 0.0
Average metadata per entry........................ 0.0
Maximum payload per entry......................... 0
Entries that use overflow......................... 0
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 4088 99.80%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 4088 99.80%
*** Table JFS_DELSLICES *******************************************************
Percentage of total database...................... 0.0%
Number of entries................................. 0
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 0 0.0%
Bytes of metadata................................. 8 0.20%
B-tree depth...................................... 1
Average payload per entry......................... 0.0
Average unused bytes per entry.................... 0.0
Average metadata per entry........................ 0.0
Maximum payload per entry......................... 0
Entries that use overflow......................... 0
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 4088 99.80%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 4088 99.80%
*** Table JFS_DETACHED_NODE ***************************************************
Percentage of total database...................... 0.0%
Number of entries................................. 0
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 0 0.0%
Bytes of metadata................................. 8 0.20%
B-tree depth...................................... 1
Average payload per entry......................... 0.0
Average unused bytes per entry.................... 0.0
Average metadata per entry........................ 0.0
Maximum payload per entry......................... 0
Entries that use overflow......................... 0
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 4088 99.80%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 4088 99.80%
*** Table JFS_DIR_QUOTA *******************************************************
Percentage of total database...................... 0.0%
Number of entries................................. 0
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 0 0.0%
Bytes of metadata................................. 8 0.20%
B-tree depth...................................... 1
Average payload per entry......................... 0.0
Average unused bytes per entry.................... 0.0
Average metadata per entry........................ 0.0
Maximum payload per entry......................... 0
Entries that use overflow......................... 0
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 4088 99.80%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 4088 99.80%
*** Table JFS_DIR_STATS *******************************************************
Percentage of total database...................... 0.0%
Number of entries................................. 0
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 0 0.0%
Bytes of metadata................................. 8 0.20%
B-tree depth...................................... 1
Average payload per entry......................... 0.0
Average unused bytes per entry.................... 0.0
Average metadata per entry........................ 0.0
Maximum payload per entry......................... 0
Entries that use overflow......................... 0
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 4088 99.80%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 4088 99.80%
*** Table JFS_EDGE and all its indices ****************************************
Percentage of total database...................... 0.074%
Number of entries................................. 110733
Bytes of storage consumed......................... 1736704
Bytes of payload.................................. 1276758 73.5%
Bytes of metadata................................. 440998 25.4%
Average payload per entry......................... 11.53
Average unused bytes per entry.................... 0.17
Average metadata per entry........................ 3.98
Average fanout.................................... 141.00
Maximum payload per entry......................... 79
Entries that use overflow......................... 0 0.0%
Index pages used.................................. 3
Primary pages used................................ 421
Overflow pages used............................... 0
Total pages used.................................. 424
Unused bytes on index pages....................... 6447 52.5%
Unused bytes on primary pages..................... 12501 0.72%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 18948 1.1%
*** Table JFS_EDGE w/o any indices ********************************************
Percentage of total database...................... 0.032%
Number of entries................................. 36911
Bytes of storage consumed......................... 749568
Bytes of payload.................................. 524841 70.0%
Bytes of metadata................................. 216648 28.9%
B-tree depth...................................... 2
Average payload per entry......................... 14.22
Average unused bytes per entry.................... 0.22
Average metadata per entry........................ 5.87
Average fanout.................................... 182.00
Non-sequential pages.............................. 1 0.55%
Maximum payload per entry......................... 79
Entries that use overflow......................... 0 0.0%
Index pages used.................................. 1
Primary pages used................................ 182
Overflow pages used............................... 0
Total pages used.................................. 183
Unused bytes on index pages....................... 2490 60.8%
Unused bytes on primary pages..................... 5589 0.75%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 8079 1.1%
*** Indices of table JFS_EDGE *************************************************
Percentage of total database...................... 0.042%
Number of entries................................. 73822
Bytes of storage consumed......................... 987136
Bytes of payload.................................. 751917 76.2%
Bytes of metadata................................. 224350 22.7%
Average payload per entry......................... 10.19
Average unused bytes per entry.................... 0.15
Average metadata per entry........................ 3.04
Average fanout.................................... 120.00
Maximum payload per entry......................... 77
Entries that use overflow......................... 0 0.0%
Index pages used.................................. 2
Primary pages used................................ 239
Overflow pages used............................... 0
Total pages used.................................. 241
Unused bytes on index pages....................... 3957 48.3%
Unused bytes on primary pages..................... 6912 0.71%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 10869 1.1%
*** Index IDX_JFS_EDGE_INODE of table JFS_EDGE ********************************
Percentage of total database...................... 0.018%
Number of entries................................. 36911
Bytes of storage consumed......................... 417792
Bytes of payload.................................. 300110 71.8%
Bytes of metadata................................. 111953 26.8%
B-tree depth...................................... 2
Average payload per entry......................... 8.13
Average unused bytes per entry.................... 0.16
Average metadata per entry........................ 3.03
Average fanout.................................... 102.00
Non-sequential pages.............................. 1 0.99%
Maximum payload per entry......................... 9
Entries that use overflow......................... 0 0.0%
Index pages used.................................. 1
Primary pages used................................ 101
Overflow pages used............................... 0
Total pages used.................................. 102
Unused bytes on index pages....................... 2562 62.5%
Unused bytes on primary pages..................... 3167 0.77%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 5729 1.4%
*** Index UQE_JFS_EDGE_EDGE of table JFS_EDGE *********************************
Percentage of total database...................... 0.024%
Number of entries................................. 36911
Bytes of storage consumed......................... 569344
Bytes of payload.................................. 451807 79.4%
Bytes of metadata................................. 112397 19.7%
B-tree depth...................................... 2
Average payload per entry......................... 12.24
Average unused bytes per entry.................... 0.14
Average metadata per entry........................ 3.05
Average fanout.................................... 139.00
Non-sequential pages.............................. 1 0.72%
Maximum payload per entry......................... 77
Entries that use overflow......................... 0 0.0%
Index pages used.................................. 1
Primary pages used................................ 138
Overflow pages used............................... 0
Total pages used.................................. 139
Unused bytes on index pages....................... 1395 34.1%
Unused bytes on primary pages..................... 3745 0.66%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 5140 0.90%
*** Table JFS_FLOCK and all its indices ***************************************
Percentage of total database...................... 0.0%
Number of entries................................. 0
Bytes of storage consumed......................... 8192
Bytes of payload.................................. 0 0.0%
Bytes of metadata................................. 16 0.20%
Average payload per entry......................... 0.0
Average unused bytes per entry.................... 0.0
Average metadata per entry........................ 0.0
Maximum payload per entry......................... 0
Entries that use overflow......................... 0
Primary pages used................................ 2
Overflow pages used............................... 0
Total pages used.................................. 2
Unused bytes on primary pages..................... 8176 99.80%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 8176 99.80%
*** Table JFS_FLOCK w/o any indices *******************************************
Percentage of total database...................... 0.0%
Number of entries................................. 0
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 0 0.0%
Bytes of metadata................................. 8 0.20%
B-tree depth...................................... 1
Average payload per entry......................... 0.0
Average unused bytes per entry.................... 0.0
Average metadata per entry........................ 0.0
Maximum payload per entry......................... 0
Entries that use overflow......................... 0
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 4088 99.80%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 4088 99.80%
*** Index UQE_JFS_FLOCK_FLOCK of table JFS_FLOCK ******************************
Percentage of total database...................... 0.0%
Number of entries................................. 0
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 0 0.0%
Bytes of metadata................................. 8 0.20%
B-tree depth...................................... 1
Average payload per entry......................... 0.0
Average unused bytes per entry.................... 0.0
Average metadata per entry........................ 0.0
Maximum payload per entry......................... 0
Entries that use overflow......................... 0
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 4088 99.80%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 4088 99.80%
*** Table JFS_NODE ************************************************************
Percentage of total database...................... 0.095%
Number of entries................................. 36913
Bytes of storage consumed......................... 2228224
Bytes of payload.................................. 1976398 88.7%
Bytes of metadata................................. 222000 10.0%
B-tree depth...................................... 3
Average payload per entry......................... 53.54
Average unused bytes per entry.................... 0.81
Average metadata per entry........................ 6.01
Average fanout.................................... 181.00
Non-sequential pages.............................. 1 0.18%
Maximum payload per entry......................... 61
Entries that use overflow......................... 0 0.0%
Index pages used.................................. 3
Primary pages used................................ 541
Overflow pages used............................... 0
Total pages used.................................. 544
Unused bytes on index pages....................... 7512 61.1%
Unused bytes on primary pages..................... 22314 1.0%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 29826 1.3%
*** Table JFS_PLOCK and all its indices ***************************************
Percentage of total database...................... 0.0%
Number of entries................................. 0
Bytes of storage consumed......................... 8192
Bytes of payload.................................. 0 0.0%
Bytes of metadata................................. 16 0.20%
Average payload per entry......................... 0.0
Average unused bytes per entry.................... 0.0
Average metadata per entry........................ 0.0
Maximum payload per entry......................... 0
Entries that use overflow......................... 0
Primary pages used................................ 2
Overflow pages used............................... 0
Total pages used.................................. 2
Unused bytes on primary pages..................... 8176 99.80%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 8176 99.80%
*** Table JFS_PLOCK w/o any indices *******************************************
Percentage of total database...................... 0.0%
Number of entries................................. 0
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 0 0.0%
Bytes of metadata................................. 8 0.20%
B-tree depth...................................... 1
Average payload per entry......................... 0.0
Average unused bytes per entry.................... 0.0
Average metadata per entry........................ 0.0
Maximum payload per entry......................... 0
Entries that use overflow......................... 0
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 4088 99.80%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 4088 99.80%
*** Index UQE_JFS_PLOCK_PLOCK of table JFS_PLOCK ******************************
Percentage of total database...................... 0.0%
Number of entries................................. 0
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 0 0.0%
Bytes of metadata................................. 8 0.20%
B-tree depth...................................... 1
Average payload per entry......................... 0.0
Average unused bytes per entry.................... 0.0
Average metadata per entry........................ 0.0
Maximum payload per entry......................... 0
Entries that use overflow......................... 0
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 4088 99.80%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 4088 99.80%
*** Table JFS_SESSION2 ********************************************************
Percentage of total database...................... 0.0%
Number of entries................................. 0
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 0 0.0%
Bytes of metadata................................. 8 0.20%
B-tree depth...................................... 1
Average payload per entry......................... 0.0
Average unused bytes per entry.................... 0.0
Average metadata per entry........................ 0.0
Maximum payload per entry......................... 0
Entries that use overflow......................... 0
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 4088 99.80%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 4088 99.80%
*** Table JFS_SETTING and all its indices *************************************
Percentage of total database...................... 0.0%
Number of entries................................. 2
Bytes of storage consumed......................... 8192
Bytes of payload.................................. 2691 32.8%
Bytes of metadata................................. 24 0.29%
Average payload per entry......................... 1345.50
Average unused bytes per entry.................... 2738.50
Average metadata per entry........................ 12.00
Maximum payload per entry......................... 2682
Entries that use overflow......................... 0 0.0%
Primary pages used................................ 2
Overflow pages used............................... 0
Total pages used.................................. 2
Unused bytes on primary pages..................... 5477 66.9%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 5477 66.9%
*** Table JFS_SETTING w/o any indices *****************************************
Percentage of total database...................... 0.0%
Number of entries................................. 1
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 2682 65.5%
Bytes of metadata................................. 13 0.32%
B-tree depth...................................... 1
Average payload per entry......................... 2682.00
Average unused bytes per entry.................... 1401.00
Average metadata per entry........................ 13.00
Maximum payload per entry......................... 2682
Entries that use overflow......................... 0 0.0%
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 1401 34.2%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 1401 34.2%
*** Index SQLITE_AUTOINDEX_JFS_SETTING_1 of table JFS_SETTING *****************
Percentage of total database...................... 0.0%
Number of entries................................. 1
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 9 0.22%
Bytes of metadata................................. 11 0.27%
B-tree depth...................................... 1
Average payload per entry......................... 9.00
Average unused bytes per entry.................... 4076.00
Average metadata per entry........................ 11.00
Maximum payload per entry......................... 9
Entries that use overflow......................... 0 0.0%
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 4076 99.51%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 4076 99.51%
*** Table JFS_SUSTAINED and all its indices ***********************************
Percentage of total database...................... 0.0%
Number of entries................................. 0
Bytes of storage consumed......................... 8192
Bytes of payload.................................. 0 0.0%
Bytes of metadata................................. 16 0.20%
Average payload per entry......................... 0.0
Average unused bytes per entry.................... 0.0
Average metadata per entry........................ 0.0
Maximum payload per entry......................... 0
Entries that use overflow......................... 0
Primary pages used................................ 2
Overflow pages used............................... 0
Total pages used.................................. 2
Unused bytes on primary pages..................... 8176 99.80%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 8176 99.80%
*** Table JFS_SUSTAINED w/o any indices ***************************************
Percentage of total database...................... 0.0%
Number of entries................................. 0
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 0 0.0%
Bytes of metadata................................. 8 0.20%
B-tree depth...................................... 1
Average payload per entry......................... 0.0
Average unused bytes per entry.................... 0.0
Average metadata per entry........................ 0.0
Maximum payload per entry......................... 0
Entries that use overflow......................... 0
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 4088 99.80%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 4088 99.80%
*** Index UQE_JFS_SUSTAINED_SUSTAINED of table JFS_SUSTAINED ******************
Percentage of total database...................... 0.0%
Number of entries................................. 0
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 0 0.0%
Bytes of metadata................................. 8 0.20%
B-tree depth...................................... 1
Average payload per entry......................... 0.0
Average unused bytes per entry.................... 0.0
Average metadata per entry........................ 0.0
Maximum payload per entry......................... 0
Entries that use overflow......................... 0
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 4088 99.80%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 4088 99.80%
*** Table JFS_SYMLINK *********************************************************
Percentage of total database...................... 0.0%
Number of entries................................. 0
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 0 0.0%
Bytes of metadata................................. 8 0.20%
B-tree depth...................................... 1
Average payload per entry......................... 0.0
Average unused bytes per entry.................... 0.0
Average metadata per entry........................ 0.0
Maximum payload per entry......................... 0
Entries that use overflow......................... 0
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 4088 99.80%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 4088 99.80%
*** Table JFS_XATTR and all its indices ***************************************
Percentage of total database...................... 0.66%
Number of entries................................. 146812
Bytes of storage consumed......................... 15622144
Bytes of payload.................................. 14743458 94.4%
Bytes of metadata................................. 751549 4.8%
Average payload per entry......................... 100.42
Average unused bytes per entry.................... 0.87
Average metadata per entry........................ 5.12
Average fanout.................................... 272.00
Maximum payload per entry......................... 350
Entries that use overflow......................... 0 0.0%
Index pages used.................................. 14
Primary pages used................................ 3800
Overflow pages used............................... 0
Total pages used.................................. 3814
Unused bytes on index pages....................... 13361 23.3%
Unused bytes on primary pages..................... 113776 0.73%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 127137 0.81%
*** Table JFS_XATTR w/o any indices *******************************************
Percentage of total database...................... 0.58%
Number of entries................................. 73406
Bytes of storage consumed......................... 13676544
Bytes of payload.................................. 13036162 95.3%
Bytes of metadata................................. 525635 3.8%
B-tree depth...................................... 3
Average payload per entry......................... 177.59
Average unused bytes per entry.................... 1.56
Average metadata per entry........................ 7.16
Average fanout.................................... 370.00
Non-sequential pages.............................. 1 0.030%
Maximum payload per entry......................... 350
Entries that use overflow......................... 0 0.0%
Index pages used.................................. 9
Primary pages used................................ 3330
Overflow pages used............................... 0
Total pages used.................................. 3339
Unused bytes on index pages....................... 7146 19.4%
Unused bytes on primary pages..................... 107601 0.79%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 114747 0.84%
*** Index UQE_JFS_XATTR_NAME of table JFS_XATTR *******************************
Percentage of total database...................... 0.083%
Number of entries................................. 73406
Bytes of storage consumed......................... 1945600
Bytes of payload.................................. 1707296 87.8%
Bytes of metadata................................. 225914 11.6%
B-tree depth...................................... 3
Average payload per entry......................... 23.26
Average unused bytes per entry.................... 0.17
Average metadata per entry........................ 3.08
Average fanout.................................... 95.00
Non-sequential pages.............................. 1 0.21%
Maximum payload per entry......................... 55
Entries that use overflow......................... 0 0.0%
Index pages used.................................. 5
Primary pages used................................ 470
Overflow pages used............................... 0
Total pages used.................................. 475
Unused bytes on index pages....................... 6215 30.3%
Unused bytes on primary pages..................... 6175 0.32%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 12390 0.64%
*** Table SQLITE_SCHEMA *******************************************************
Percentage of total database...................... 0.0%
Number of entries................................. 28
Bytes of storage consumed......................... 12288
Bytes of payload.................................. 4276 34.8%
Bytes of metadata................................. 262 2.1%
B-tree depth...................................... 2
Average payload per entry......................... 152.71
Average unused bytes per entry.................... 276.79
Average metadata per entry........................ 9.36
Average fanout.................................... 2.00
Non-sequential pages.............................. 2 100.0%
Maximum payload per entry......................... 510
Entries that use overflow......................... 0 0.0%
Index pages used.................................. 1
Primary pages used................................ 2
Overflow pages used............................... 0
Total pages used.................................. 3
Unused bytes on index pages....................... 3977 97.1%
Unused bytes on primary pages..................... 3773 46.1%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 7750 63.1%
*** Table SQLITE_SEQUENCE *****************************************************
Percentage of total database...................... 0.0%
Number of entries................................. 3
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 44 1.1%
Bytes of metadata................................. 20 0.49%
B-tree depth...................................... 1
Average payload per entry......................... 14.67
Average unused bytes per entry.................... 1344.00
Average metadata per entry........................ 6.67
Maximum payload per entry......................... 15
Entries that use overflow......................... 0 0.0%
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 4032 98.4%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 4032 98.4%
*** Table SQLITE_STAT1 ********************************************************
Percentage of total database...................... 0.0%
Number of entries................................. 9
Bytes of storage consumed......................... 4096
Bytes of payload.................................. 319 7.8%
Bytes of metadata................................. 44 1.1%
B-tree depth...................................... 1
Average payload per entry......................... 35.44
Average unused bytes per entry.................... 414.78
Average metadata per entry........................ 4.89
Maximum payload per entry......................... 49
Entries that use overflow......................... 0 0.0%
Primary pages used................................ 1
Overflow pages used............................... 0
Total pages used.................................. 1
Unused bytes on primary pages..................... 3733 91.1%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 3733 91.1%
*** Definitions ***************************************************************
Page size in bytes
The number of bytes in a single page of the database file.
Usually 1024.
Number of pages in the whole file
The number of 4096-byte pages that go into forming the complete
database
Pages that store data
The number of pages that store data, either as primary B*Tree pages or
as overflow pages. The number at the right is the data pages divided by
the total number of pages in the file.
Pages on the freelist
The number of pages that are not currently in use but are reserved for
future use. The percentage at the right is the number of freelist pages
divided by the total number of pages in the file.
Pages of auto-vacuum overhead
The number of pages that store data used by the database to facilitate
auto-vacuum. This is zero for databases that do not support auto-vacuum.
Number of tables in the database
The number of tables in the database, including the SQLITE_SCHEMA table
used to store schema information.
Number of indices
The total number of indices in the database.
Number of defined indices
The number of indices created using an explicit CREATE INDEX statement.
Number of implied indices
The number of indices used to implement PRIMARY KEY or UNIQUE constraints
on tables.
Size of the file in bytes
The total amount of disk space used by the entire database files.
Bytes of user payload stored
The total number of bytes of user payload stored in the database. The
schema information in the SQLITE_SCHEMA table is not counted when
computing this number. The percentage at the right shows the payload
divided by the total file size.
Percentage of total database
The amount of the complete database file that is devoted to storing
information described by this category.
Number of entries
The total number of B-Tree key/value pairs stored under this category.
Bytes of storage consumed
The total amount of disk space required to store all B-Tree entries
under this category. The is the total number of pages used times
the pages size.
Bytes of payload
The amount of payload stored under this category. Payload is the data
part of table entries and the key part of index entries. The percentage
at the right is the bytes of payload divided by the bytes of storage
consumed.
Bytes of metadata
The amount of formatting and structural information stored in the
table or index. Metadata includes the btree page header, the cell pointer
array, the size field for each cell, the left child pointer or non-leaf
cells, the overflow pointers for overflow cells, and the rowid value for
rowid table cells. In other words, metadata is everything that is neither
unused space nor content. The record header in the payload is counted as
content, not metadata.
Average payload per entry
The average amount of payload on each entry. This is just the bytes of
payload divided by the number of entries.
Average unused bytes per entry
The average amount of free space remaining on all pages under this
category on a per-entry basis. This is the number of unused bytes on
all pages divided by the number of entries.
Non-sequential pages
The number of pages in the table or index that are out of sequence.
Many filesystems are optimized for sequential file access so a small
number of non-sequential pages might result in faster queries,
especially for larger database files that do not fit in the disk cache.
Note that after running VACUUM, the root page of each table or index is
at the beginning of the database file and all other pages are in a
separate part of the database file, resulting in a single non-
sequential page.
Maximum payload per entry
The largest payload size of any entry.
Entries that use overflow
The number of entries that user one or more overflow pages.
Total pages used
This is the number of pages used to hold all information in the current
category. This is the sum of index, primary, and overflow pages.
Index pages used
This is the number of pages in a table B-tree that hold only key (rowid)
information and no data.
Primary pages used
This is the number of B-tree pages that hold both key and data.
Overflow pages used
The total number of overflow pages used for this category.
Unused bytes on index pages
The total number of bytes of unused space on all index pages. The
percentage at the right is the number of unused bytes divided by the
total number of bytes on index pages.
Unused bytes on primary pages
The total number of bytes of unused space on all primary pages. The
percentage at the right is the number of unused bytes divided by the
total number of bytes on primary pages.
Unused bytes on overflow pages
The total number of bytes of unused space on all overflow pages. The
percentage at the right is the number of unused bytes divided by the
total number of bytes on overflow pages.
Unused bytes on all pages
The total number of bytes of unused space on all primary and overflow
pages. The percentage at the right is the number of unused bytes
divided by the total number of bytes.
*******************************************************************************
The entire text of this report can be sourced into any SQL database
engine for further analysis. All of the text above is an SQL comment.
The data used to generate this report follows:
*/
BEGIN;
CREATE TABLE space_used(
name clob, -- Name of a table or index in the database file
tblname clob, -- Name of associated table
is_index boolean, -- TRUE if it is an index, false for a table
is_without_rowid boolean, -- TRUE if WITHOUT ROWID table
nentry int, -- Number of entries in the BTree
leaf_entries int, -- Number of leaf entries
depth int, -- Depth of the b-tree
payload int, -- Total amount of data stored in this table or index
ovfl_payload int, -- Total amount of data stored on overflow pages
ovfl_cnt int, -- Number of entries that use overflow
mx_payload int, -- Maximum payload size
int_pages int, -- Number of interior pages used
leaf_pages int, -- Number of leaf pages used
ovfl_pages int, -- Number of overflow pages used
int_unused int, -- Number of unused bytes on interior pages
leaf_unused int, -- Number of unused bytes on primary pages
ovfl_unused int, -- Number of unused bytes on overflow pages
gap_cnt int, -- Number of gaps in the page layout
compressed_size int -- Total bytes stored on disk
);
INSERT INTO space_used VALUES('sqlite_schema','sqlite_schema',0,0,29,28,2,4276,0,0,510,1,2,0,3977,3773,0,2,12288);
INSERT INTO space_used VALUES('jfs_setting','jfs_setting',0,0,1,1,1,2682,0,0,2682,0,1,0,0,1401,0,0,4096);
INSERT INTO space_used VALUES('sqlite_autoindex_jfs_setting_1','jfs_setting',1,0,1,1,1,9,0,0,9,0,1,0,0,4076,0,0,4096);
INSERT INTO space_used VALUES('jfs_counter','jfs_counter',0,0,10,10,1,192,0,0,26,0,1,0,0,3856,0,0,4096);
INSERT INTO space_used VALUES('sqlite_autoindex_jfs_counter_1','jfs_counter',1,0,10,10,1,165,0,0,23,0,1,0,0,3893,0,0,4096);
INSERT INTO space_used VALUES('jfs_edge','jfs_edge',0,0,37092,36911,2,524841,0,0,79,1,182,0,2490,5589,0,1,749568);
INSERT INTO space_used VALUES('sqlite_sequence','sqlite_sequence',0,0,3,3,1,44,0,0,15,0,1,0,0,4032,0,0,4096);
INSERT INTO space_used VALUES('jfs_node','jfs_node',0,0,37453,36913,3,1976398,0,0,61,3,541,0,7512,22314,0,1,2228224);
INSERT INTO space_used VALUES('jfs_symlink','jfs_symlink',0,0,0,0,1,0,0,0,0,0,1,0,0,4088,0,0,4096);
INSERT INTO space_used VALUES('jfs_xattr','jfs_xattr',0,0,76735,73406,3,13036162,0,0,350,9,3330,0,7146,107601,0,1,13676544);
INSERT INTO space_used VALUES('jfs_chunk','jfs_chunk',0,0,45662,41133,3,1173463886,1161265201,4477,3584194,12,4530,283828,8524,6049710,158975,490,1181163520);
INSERT INTO space_used VALUES('jfs_chunk_ref','jfs_chunk_ref',0,0,54890454,54681884,4,330409280,0,0,8,659,208571,0,414125,89356812,0,5338,857006080);
INSERT INTO space_used VALUES('jfs_delslices','jfs_delslices',0,0,0,0,1,0,0,0,0,0,1,0,0,4088,0,0,4096);
INSERT INTO space_used VALUES('jfs_session2','jfs_session2',0,0,0,0,1,0,0,0,0,0,1,0,0,4088,0,0,4096);
INSERT INTO space_used VALUES('jfs_sustained','jfs_sustained',0,0,0,0,1,0,0,0,0,0,1,0,0,4088,0,0,4096);
INSERT INTO space_used VALUES('jfs_delfile','jfs_delfile',0,0,0,0,1,0,0,0,0,0,1,0,0,4088,0,0,4096);
INSERT INTO space_used VALUES('jfs_flock','jfs_flock',0,0,0,0,1,0,0,0,0,0,1,0,0,4088,0,0,4096);
INSERT INTO space_used VALUES('jfs_plock','jfs_plock',0,0,0,0,1,0,0,0,0,0,1,0,0,4088,0,0,4096);
INSERT INTO space_used VALUES('sqlite_stat1','sqlite_stat1',0,0,9,9,1,319,0,0,49,0,1,0,0,3733,0,0,4096);
INSERT INTO space_used VALUES('jfs_dir_stats','jfs_dir_stats',0,0,0,0,1,0,0,0,0,0,1,0,0,4088,0,0,4096);
INSERT INTO space_used VALUES('jfs_detached_node','jfs_detached_node',0,0,0,0,1,0,0,0,0,0,1,0,0,4088,0,0,4096);
INSERT INTO space_used VALUES('jfs_dir_quota','jfs_dir_quota',0,0,0,0,1,0,0,0,0,0,1,0,0,4088,0,0,4096);
INSERT INTO space_used VALUES('UQE_jfs_edge_edge','jfs_edge',1,0,36911,36774,2,451807,0,0,77,1,138,0,1395,3745,0,1,569344);
INSERT INTO space_used VALUES('IDX_jfs_edge_inode','jfs_edge',1,0,36911,36811,2,300110,0,0,9,1,101,0,2562,3167,0,1,417792);
INSERT INTO space_used VALUES('UQE_jfs_xattr_name','jfs_xattr',1,0,73406,72937,3,1707296,0,0,55,5,470,0,6215,6175,0,1,1945600);
INSERT INTO space_used VALUES('UQE_jfs_chunk_chunk','jfs_chunk',1,0,41133,41012,2,373306,0,0,10,1,122,0,2128,3503,0,1,503808);
INSERT INTO space_used VALUES('UQE_jfs_sustained_sustained','jfs_sustained',1,0,0,0,1,0,0,0,0,0,1,0,0,4088,0,0,4096);
INSERT INTO space_used VALUES('UQE_jfs_flock_flock','jfs_flock',1,0,0,0,1,0,0,0,0,0,1,0,0,4088,0,0,4096);
INSERT INTO space_used VALUES('UQE_jfs_plock_plock','jfs_plock',1,0,0,0,1,0,0,0,0,0,1,0,0,4088,0,0,4096);
COMMIT;
> juicefs -V
juicefs version 1.1.1+2023-11-28.437f4e629a09
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
In my access pattern [1], I notice that I now have a large and fast-growing metadata footprint that I would prefer to avoid [2], but which, if needed, I'll adapt.
When running
gc --compactit seems that there are large numbers of pending slices (which gc --compact eliminates).Hence, I wondered:
gc --compactseems to address this, but is this something that would be expected to be cleaned up as part of client background maintenance?[1] I'm storing a virtual hard disk using juicefs mount with writeback and upload-delay of 5s.
[2] After starting with metadata of 50 MB, my metadata steadily grew until recently reaching 2.5GB.
juicefs status --moreoutput:{ "Setting": { "Name": "XXX", "UUID": "XXX", "Storage": "b2", "Bucket": "https://XXX.s3.us-east-005.backblazeb2.com", "AccessKey": "XXX", "SecretKey": "removed", "BlockSize": 4096, "Compression": "zstd", "EncryptKey": "removed", "KeyEncrypted": true, "TrashDays": 0, "MetaVersion": 1 }, "Sessions": [], "Statistic": { "UsedSpace": 1523944566784, "AvailableSpace": 1124375962275840, "UsedInodes": 36909, "AvailableInodes": 10485760 } }Here's a table-level view of my metadata payload (sqlite3_analyzer output):
> juicefs -V juicefs version 1.1.1+2023-11-28.437f4e629a09Beta Was this translation helpful? Give feedback.
All reactions