v1.0.7

Relicensing

One notable thing in this release is that we cleaned up our license situation, in a nutshell:

• Relicense most code under GPL-2.0-or-later OR Apache-2.0 by @cgwalters in #376

Much more in the linked issue: #344

Features

• mkcomposefs: make it cpuset aware by @giuseppe in #355

Other changes

Other than the above: a few build and static analyzer fixes, some documentation improvements.

• README.md: Add a "tagline" by @cgwalters in #348
• ci: Drop "with meson" from description by @cgwalters in #350
• ci: Add revdep-ci for bootc by @cgwalters in #351
• spec: Sync from Fedora by @cgwalters in #363
• mkcomposefs: Document external sandboxing by @cgwalters in #362
• meson.build: raise used C standard to C11 by @ziyao233 in #365
• libcomposefs: include linux/limits.h to satisfy usage of XATTR_NAME_MAX by @ziyao233 in #366
• README: minor tweaks by @jlebon in #370
• tests: Fix should-fail-self-hardlink to conform to latest syntax by @cgwalters in #374
• Fix clang warnings by @cgwalters in #368
• Add a libcomposefs-internal by @cgwalters in #367
• internal: Rewrite byteswapping macros by @cgwalters in #375
• tests: Add a flow for integration test with fsverity enabled by @cgwalters in #377
• README.md: Link to community forums by @cgwalters in #379
• Add a CONTRIBUTING.md by @cgwalters in #381
• MAINTAINERS.md: New file by @cgwalters in #382
• ci: Update run-on-arch by @cgwalters in #383
• writer: Add O_DIRECTORY by @eriksjolund in #385
• writer: Use blocksize constant by @cgwalters in #388
• Fixed typos in README.md by @marrusl in #390
• lib: Add expansion comments to header by @cgwalters in #384
• Release 1.0.7 by @cgwalters in #386

New Contributors

• @ziyao233 made their first contribution in #365
• @jlebon made their first contribution in #370
• @marrusl made their first contribution in #390

Full Changelog: v1.0.6...v1.0.7

v1.0.6

Feature: mkcomposefs --from-file fuzzing

A while ago while working on the Rust composefs bindings I found a bug in the dumpfile parser where the process would crash if provided an unexpected value.

Today for the Rust bindings and the containers/image library because we're forking off a separate process (that we can apply isolation to and low privilege overall) even an assertion failure is relatively benign. And such tools usually wouldn't produce a dumpfile malformed in that way. But still, it caused me to question whether there were other bugs.

For codebases that are linking in the libcomposefs library, process aborts are obviously more problematic.

This series is the result of running honggfuzz and quite a lot of time spent learning this codebase.

Thanks to @hsiangkao @alexlarsson @jeckersb @giuseppe @allisonkarlitskaya for their assistance with reviews and discussion of this!

Composefs and security

I don't think there's cause to classify this as a CVE yet, as we didn't really formally declare that mkcomposefs --from-file is a "security boundary". At the current time, if you are using composefs in a context where it is being provided potentially malicious input, it is strongly recommended to ensure that the input is validated and sanitized as much as possible, alongside use of hardening and fuzzing techniques. We have developed Rust bindings for example which do some of this, and work will continue on that.

This said, we will likely aim to make "mkcomposefs --from-file" a security boundary in the future (e.g. guaranteeing that it will always either return an error or generate a valid EROFS) and allocate CVEs for any further flaws.

Composefs and reliability

composefs has been used reliably in many cases for a while now; in general almost all of these issues could only occur for extreme boundary conditions such as:

• symlinks close to the PATH_MAX (4096 byte) limit
• extremely large extended attributes
• Files claiming to be extremely large (> 5 EiB)

Hence, I do not think there is a need for existing composefs users to consider this a critical fix.

Most of these cases also again would fail with an assertion - at most a denial of service. However in some cases we would produce a corrupted EROFS blob - and in the cases I tested, the Linux kernel EROFS parser successfully detected the corruption.

The EROFS kernel code is actively fuzzed and fixed when bugs are found (e.g. this recent one) but still a goal for this project is to avoid exposing the Linux kernel code to a potentially corrupted EROFS at all.

(A core part of the idea of composefs is that the EROFS metadata doesn't need to be transported "on the wire", it can be re-synthesized reproducibly from other metadata, such as existing OCI tar layers, or ostree commits, etc.)

Autotools build system dropped

• build-sys: Remove autotools by @cgwalters in #313

The only supported build system is now meson.

Potentially incompatible changes

• mkcomposefs: Process last content in dumpfile without newline by @cgwalters in #331

We believe this is unlikely to break anyone, but if it does we are open to reverting this change.

New features

• dump: Support --filter by @cgwalters in #320

Other changes

• lib: Bail on xattr names > XATTR_NAME_MAX by @cgwalters in #314
• lib: Verify mode values by @cgwalters in #315
• Reject an empty name by @cgwalters in #318
• Two more testing improvements by @cgwalters in #317
• More validation by @cgwalters in #319
• rust/dumpfile: Minor optimization for unescape_to_path_canonical by @cgwalters in #321
• rust: Attempt to fix build on docs.rs by @cgwalters in #322
• dump: Support --filter by @cgwalters in #320
• rust: Add wrapping for composefs-info dump by @cgwalters in #323
• Various fixes to the fsverity part of test-units.sh by @alexlarsson in #325
• Replace custom clang-format bits with stock meson targets by @cgwalters in #326
• Add 64bit rdev APIs by @cgwalters in #329
• libcomposefs: detect short erofs files by @allisonkarlitskaya in #333
• rust: Accept 64 bit values for rdev by @cgwalters in #330
• A few cleanups around symlinks by @cgwalters in #337
• tests: Move sha verification after other checks by @cgwalters in #338
• ci: update deprecated {up,down}load-artifact@v2 to v4 by @jeckersb in #339
• dumpfile: Add some test code to convert to tar by @cgwalters in #328
• mkcomposefs: Add a hidden CFS_PARSE_STRICT by @cgwalters in #340
• Results from running honggfuzz, plus some more manual auditing by @cgwalters in #342
• lib: Move validation out of lcfs_node_add_child() by @cgwalters in #341
• writer: Clarifications around our usage of EROFS_NULL_ADDR by @cgwalters in #345
• Release 1.0.6 by @cgwalters in #346

New Contributors

• @allisonkarlitskaya made their first contribution in #333
• @jeckersb made their first contribution in #339

Full Changelog: v1.0.5...v1.0.6

v1.0.5

A notable change here is that composefs now supports meson as a build system. We are likely to remove the autotools build system in the next release, so if you're including composefs in your build system, please let us know if there's any problems with that in #301

There's a few minor internal fixes and hardening, some CI cleanups. Also included in the list below is changes to the Rust code, which has already been published to crates.io and is versioned separately from this release.

Thanks to all contributors!

What's Changed

• test: use correct composefs-info binary by @Kiskae in #291
• rust: Some doc and misc tweaks by @cgwalters in #293
• ci: Test Rust code against current C code by @cgwalters in #292
• test: remove hardcoded /tmp by @Kiskae in #296
• rust: Add mandatory description field, other misc changes by @cgwalters in #295
• composefs-core: Drop incorrect include by @cgwalters in #297
• rust: Two minor patches by @cgwalters in #298
• composefs-info: specify locale to isgraph/isprint by @giuseppe in #299
• Add meson as an alternative to autotools with corresponding CI by @jluebbe in #300
• info: Add missing terminator for longopts by @cgwalters in #302
• mount: Consistently use composefs as source by @cgwalters in #303
• meson.build: Allow use as a meson subproject by @jluebbe in #304
• mkcompose: report close() errors by @giuseppe in #306
• libcomposefs: do not override errno by @giuseppe in #307
• libcomposefs: drop using "ro" option by @giuseppe in #309
• Minor rust lints by @cgwalters in #308
• Verify payload length early on by @cgwalters in #310
• Release 1.0.5 by @cgwalters in #312

New Contributors

• @Kiskae made their first contribution in #291

Full Changelog: v1.0.4...v1.0.5

Release 1.0.4

Changes since 1.0.4:

• Added LCFS_MOUNT_FLAGS_TRY_VERITY/tryverity mount option to do best-effort fs-verity
• Use the new loewerdir+ and datadir+ overlayfs mount options from linux 6.6
• mkcomposefs is now multi-threaded when computing digests and copying files
• mkcomposefs now uses copy_file_range when copying files
• Added some initial rust crates to handle composefs dump files
• Added fuzz tests
• Documentation updates
• Fix some crashes and leaks
• Improve error reporting
• Fix build on various libc types and versions

Release 1.0.3

Changes since 1.0.2:

• Image format version 1 was added that allows storing overlay whiteouts in
  a composefs image.
• libcomposefs added max_version option
• mkcomposefs added --version and --max-version options
• composefs can now be build without go-md2man
• Minor bugfix handling file read errors

Release version 1.0.2

Changes since 1.0.1:

• Dropped composefs-from-json in tests in favour of using the
  composefs-info dump format.
• libyajl dependency dropped
• libcomposefs now limits the number of xattrs per file to 64k
• Fixed build against libc without reallocarray
• Performance fixes
• go-md2man is used instead of pandoc for manpages
• Minor fixes to spec file

Release version 1.0.1

Changes since 1.0.0:

• mkcomposefs now supports --from-file which reads image descriptions
  in the format of composefs-info dump output.
• composefs-info dump output now shows content and hardlink target
• mount.composefs now uses the new mount api on recent kernels, which
  fixes a regression in escaping commas in upperdir/lowerdir/workdir
  options.
• lcfs_node_set_payload() now accepts NULL to unset
• Some performance improvements
• Fixed errors in error reporting
• Added more manpages

Release version 1.0.0

This is the first stable release of composefs. Starting now, we
guarantee a stable library ABI and a binary stable file format. The
later means that any image build from an identical lcfs_node tree and
identical write options, will produce a file that is binary identical
to a later run even with a different version. The same is true for
a mkcomposefs run with the same options.

Major changes since 0.1.4:

• Added a soname with version to libcomposefs

• All required overlayfs xattr changes are now upstream and the
  corresponding image generation changes have been made in
  composefs. This includes support for escaping overlayfs xattrs and
  whiteouts for nested overlayfs mounts.

• fs-verity built-in signature support was dropped in favour of
  userspace signatures.

• The erofs images now uses the new bloom filter for faster xattr
  lookups. This is backward compatible and old erofs version still
  work (sans the speedup).

• Files can now be inlined in the erofs image to avoid overhead of
  using redirections for small files.

• There is a new API to regenerate a lcfs_node tree from a composefs
  image file.

• There is a new composefs-info tool that lets you dump info about
  images, including what objects it refers to and which ones are
  missing from a given basedir.

• Various fixes, cleanups and new tests

Signed-off-by: Alexander Larsson [email protected]

Release 0.1.4

Major changes in this version:

• kernel module was dropped
• overlayfs digest xattrs used change to match the final (v5) patchset
• Updated overlayfs mounts to use the final syntax for datalower dirs
• library now exposes API to compute fs-verity digests
• library dropped ctime and node removal API
• Fixed mkcomposefs on big endian
• mkcomposefs now tries to use reflinks if possible
• writer-json was renamed to composefs-from-json
• composefs-from-json sandboxes the json parsing
• composefs-from-json honors mtime
• mount.composefs now supports --help
• Added initial version of composefs-fuse
• Various code cleanups
• Updated docs

Release 0.1.3

Added option to specify image mount point to lcfs_mount().