How to set or enable small file merging in Ozone?

[Dong-GuoSheng]

How to set or enable small file merging in Ozone?
In ozone, whether using the OozoneClient or S3 interface to write 100000 small files (each of 1KB), at least 100000 inodes will be added in hdds (viewed through free-i)
What parameters can be set to merge these small files into medium to large file storage to reduce inode usage in Linux?
Looking forward to your help.

在ozone中怎么设置或启用小文件合并？
在ozone中无论是使用OozoneClient还是S3接口写入100000个小文件（每个都是1KB），hdds下最少会增加100000个inode（通过free -i查看的）。
通过什么参数可以设置将这些小文件合并中大文件存储，以减少linux中的inode占用？
期待能获得您的帮助。

[ChenSammi]

So far Ozone doesn‘t support this feature yet.

[ivandika3]

This probably requires a new block file format, similar to Facebook's Haystack / SeaweedFS where a single large append-only file contains multiple blocks. However the tradeoff is that there will be more bookkeeping and GC logic during deletion that needs to be handled.

Currently having a single file per block is simpler.

[jojochuang]

My team discussed creating special files for files less than a predefined value. The files could reside in Ozone Manager rocksdb, or in DataNode rocksdb.

That said, Ozone doesn't optimize for this scenario.

[ivandika3]

My team discussed creating special files for files less than a predefined value. The files could reside in Ozone Manager rocksdb, or in DataNode rocksdb.

Thanks for the info. I saw some previous community discussions regarding this.

Speaking about RocksDB, maybe we can try to explore https://github.com/facebook/rocksdb/wiki/BlobDB since it seems to be able to transparently store the blobs in either directly in LSM tree or in a separate file depending on the size. However, I'm not yet familiar with this and its possible limitations.

[jojochuang]

I had encountered this problem in a lab environments.
You could increase inodes but it requires reformat the disk. https://www.tecmint.com/increase-disk-inode-number-in-linux/

[kerneltime]

@Dong-GuoSheng are you using HCFS interface or S3 APIs?