JuiceFS report OOM command not allowed

[wutz]

Today, i noticed that our juicefs cannot accessed with error input/output error.

So i checked juicefs logs and found that it's output juicefs[59] <ERROR>: error: OOM command not allowed when used memory > 'maxmemory'.

This is a Redis error, I then checked Aliyun Redis and found that the maximum memory reached 1G.

$ df -h /jfs
Filesystem       Size  Used Avail Use% Mounted on
JuiceFS:juicefs  1.0P  302G  1.0P   1% /jfs

$ df -i /jfs/
Filesystem        Inodes IUsed    IFree IUse% Mounted on
JuiceFS:juicefs 10495179  9419 10485760    1% /jfs

Only 9149 inodes consume 1 GB of Redis memory, is this normal ?

How many files need to be configured with a reference for Redis memory ?

We use juicefs as prometheus and fluentd backend store.

[xiaogaozi]

Looks like 115 KiB per inode, could you restart Redis to observe if the memory load will down?

[davies]

Again executing juicefs gc --compact:

• fs: inode - 10083, size - 355G
• redis: key.count - 25616, total.allocated - 44M
• oss: count - 105211, size - 45G

These number looks reasonable now.

[davies]

@wutz Here is the memory stats in my case:

# Memory
used_memory_human:44.85M
used_memory_rss_human:232.88M
used_memory_peak_human:705.21M
used_memory_peak_perc:6.36%
used_memory_overhead:41028030
used_memory_startup:4995800
used_memory_dataset:5999498
used_memory_dataset_perc:14.27%

The actual memory used by JuiceFS is only 6MB (used_memory_dataset), most of the used_memory_human are used_memory_overhead (41MB)

[wutz]

Me too, Redis overhead is too big.

used_memory_human:46.48M
used_memory_dataset:5858728
used_memory_dataset_perc:13.30%

[wutz]

Another question, is the total amount of filesystem inodes (same as total size) a hard limit ?

The current inode total is about 10m：

$  df -i /jfs
Filesystem        Inodes IUsed    IFree IUse% Mounted on
JuiceFS:juicefs 10496053 10293 10485760    1% /jfs

We are considering juicefs for HPC scenarios, where it typically uses inode levels of 10 billion and size levels of 10 PiB. Is there a good suggestion?

[davies]

No, there is no hard limit. Redis can't handle 10 billions of files, it will need 300+GB of memory. A SQL database can support 10 billions of files, but slower than Redis, may not be fast enough for HPC scenarios.

We plan to support TiKV as meta engine, that will be a better choice than Redis and SQL database.