erofs / erofs-utils

erofs-utils =========== Userspace tools for EROFS filesystem, currently including: mkfs.erofs filesystem formatter mount.erofs mount helper for EROFS erofsfuse FUSE daemon alternative dump.erofs filesystem analyzer fsck.erofs filesystem compatibility & consistency checker as well as extractor EROFS filesystem overview ------------------------- EROFS filesystem stands for Enhanced Read-Only File System. It aims to form a generic read-only filesystem solution for various read-only use cases instead of just focusing on storage space saving without considering any side effects of runtime performance. Typically EROFS could be considered in the following use scenarios: • Firmwares in performance-sensitive systems, such as system partitions of Android smartphones; • Mountable immutable images such as container images for effective metadata & data access compared with tar, cpio or other local filesystems (e.g. ext4, XFS, btrfs, etc.) • FSDAX-enabled rootfs for secure containers (Linux 5.15+); • Live CDs which need a set of files with another high-performance algorithm to optimize startup time; others files for archival purposes only are not needed; • and more. Note that all EROFS metadata is uncompressed by design, so that you could take EROFS as a drop-in read-only replacement of ext4, XFS, btrfs, etc. without any compression-based dependencies and EROFS can bring more effective filesystem accesses to users with reduced metadata. For more details of EROFS filesystem itself, please refer to: https://www.kernel.org/doc/html/next/filesystems/erofs.html For more details on how to build erofs-utils, see . For more details about filesystem performance, see . mkfs.erofs ---------- Two main kinds of EROFS images can be generated: (un)compressed images. • For uncompressed images, there will be no compressed files in these images. However, an EROFS image can contain files which consist of various aligned data blocks and then a tail that is stored inline in order to compact images [1]. • For compressed images, it will try to use the given algorithms first for each regular file and see if storage space can be saved with compression. If not, it will fall back to an uncompressed file. Note that EROFS supports per-file compression configuration, proper configuration options need to be enabled to parse compressed files by the Linux kernel. How to generate EROFS images ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Compression algorithms could be specified with the command-line option to build a compressed EROFS image from a local directory: $ mkfs.erofs -zlz4hc foo.erofs.img foo/ Supported algorithms by the Linux kernel: • LZ4 (Linux 5.3+); • LZMA (Linux 5.16+); • DEFLATE (Linux 6.6+); • Zstandard (Linux 6.10+). Alternatively, generate an uncompressed EROFS from a local directory: $ mkfs.erofs foo.erofs.img foo/ Additionally, you can specify a higher compression level to get a (slightly) smaller image than the default level: $ mkfs.erofs -zlz4hc,12 foo.erofs.img foo/ Multi-threaded support can be explicitly enabled with the ./configure option ; otherwise, single-threaded compression will be used for now. It may take more time on multiprocessor platforms if multi-threaded support is not enabled. Currently, doesn't support multi-threading due to limited development resources. Reproducible builds ~~~~~~~~~~~~~~~~~~~ Reproducible builds are typically used for verification and security, ensuring the same binaries/distributions to be reproduced in a deterministic way. Images generated by the same version of will be identical to previous runs if the same input is specified, and the same options are used. Specifically, variable timestamps and filesystem UUIDs can result in unreproducible EROFS images. and can be used to fix them. How to generate EROFS big pcluster images (Linux 5.13+) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ By default, EROFS formatter compresses data into separate one-block (e.g. 4KiB) filesystem physical clusters for outstanding random read performance. In other words, each EROFS filesystem block can be independently decompressed. However, other similar filesystems typically compress data into "blocks" of 128KiB or more for much smaller images. Users may prefer smaller images for archiving purposes, even if random performance is compromised with those configurations, and even worse when using 4KiB blocks. In order to fulfill users' needs, big plusters has been introduced since Linux 5.13, in which each physical clusters will be more than one blocks. Specifically, is used to specify the maximum size of each pcluster in bytes: $ mkfs.erofs -zlz4hc -C65536 foo.erofs.img foo/ Thus, in this case, pcluster sizes can be up to 64KiB. Note that large pcluster size can degrade random performance (though it may improve sequential read performance for typical storage devices), so please evaluate carefully in advance. Alternatively, you can make per-(sub)file compression strategies according to file access patterns if needed. How to generate EROFS images with multiple algorithms ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It's possible to generate an EROFS image with files in different algorithms due to various purposes. For example, LZMA for archival purposes and LZ4 for runtime purposes. In order to use alternative algorithms, just specify two or more compressing configurations together separated by ':' like below: -zlzma:lz4hc,12:lzma,9 -C32768 Although mkfs still choose the first one by default, you could try to write a compress-hints file like below: 4096 1 .*\.so$ 32768 2 .*\.txt$ 4096 sbin/.*$ 16384 0 .* and specify with so that ".so" files will use "lz4hc,12" compression with 4k pclusters, ".txt" files will use "lzma,9" compression with 32k pclusters, files under "/sbin" will use the default "lzma" compression with 4k plusters and other files will use "lzma" compression with 16k pclusters. Note that the largest pcluster size should be s…