Documentation/filesystems/erofs.txt

   1 Overview
   2 ========
   3
   4 EROFS file-system stands for Enhanced Read-Only File System. Different
   5 from other read-only file systems, it aims to be designed for flexibility,
   6 scalability, but be kept simple and high performance.
   7
   8 It is designed as a better filesystem solution for the following scenarios:
   9  - read-only storage media or
  10
  11  - part of a fully trusted read-only solution, which means it needs to be
  12    immutable and bit-for-bit identical to the official golden image for
  13    their releases due to security and other considerations and
  14
  15  - hope to save some extra storage space with guaranteed end-to-end performance
  16    by using reduced metadata and transparent file compression, especially
  17    for those embedded devices with limited memory (ex, smartphone);
  18
  19 Here is the main features of EROFS:
  20  - Little endian on-disk design;
  21
  22  - Currently 4KB block size (nobh) and therefore maximum 16TB address space;
  23
  24  - Metadata & data could be mixed by design;
  25
  26  - 2 inode versions for different requirements:
  27                           compact (v1)  extended (v2)
  28    Inode metadata size:   32 bytes      64 bytes
  29    Max file size:         4 GB          16 EB (also limited by max. vol size)
  30    Max uids/gids:         65536         4294967296
  31    File change time:      no            yes (64 + 32-bit timestamp)
  32    Max hardlinks:         65536         4294967296
  33    Metadata reserved:     4 bytes       14 bytes
  34
  35  - Support extended attributes (xattrs) as an option;
  36
  37  - Support xattr inline and tail-end data inline for all files;
  38
  39  - Support POSIX.1e ACLs by using xattrs;
  40
  41  - Support transparent file compression as an option:
  42    LZ4 algorithm with 4 KB fixed-sized output compression for high performance.
  43
  44 The following git tree provides the file system user-space tools under
  45 development (ex, formatting tool mkfs.erofs):
  46 >> git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git
  47
  48 Bugs and patches are welcome, please kindly help us and send to the following
  49 linux-erofs mailing list:
  50 >> linux-erofs mailing list   <linux-erofs@lists.ozlabs.org>
  51
  52 Mount options
  53 =============
  54
  55 (no)user_xattr         Setup Extended User Attributes. Note: xattr is enabled
  56                        by default if CONFIG_EROFS_FS_XATTR is selected.
  57 (no)acl                Setup POSIX Access Control List. Note: acl is enabled
  58                        by default if CONFIG_EROFS_FS_POSIX_ACL is selected.
  59 cache_strategy=%s      Select a strategy for cached decompression from now on:
  60                          disabled: In-place I/O decompression only;
  61                         readahead: Cache the last incomplete compressed physical
  62                                    cluster for further reading. It still does
  63                                    in-place I/O decompression for the rest
  64                                    compressed physical clusters;
  65                        readaround: Cache the both ends of incomplete compressed
  66                                    physical clusters for further reading.
  67                                    It still does in-place I/O decompression
  68                                    for the rest compressed physical clusters.
  69
  70 On-disk details
  71 ===============
  72
  73 Summary
  74 -------
  75 Different from other read-only file systems, an EROFS volume is designed
  76 to be as simple as possible:
  77
  78                                 |-> aligned with the block size
  79    ____________________________________________________________
  80   | |SB| | ... | Metadata | ... | Data | Metadata | ... | Data |
  81   |_|__|_|_____|__________|_____|______|__________|_____|______|
  82   0 +1K
  83
  84 All data areas should be aligned with the block size, but metadata areas
  85 may not. All metadatas can be now observed in two different spaces (views):
  86  1. Inode metadata space
  87     Each valid inode should be aligned with an inode slot, which is a fixed
  88     value (32 bytes) and designed to be kept in line with compact inode size.
  89
  90     Each inode can be directly found with the following formula:
  91          inode offset = meta_blkaddr * block_size + 32 * nid
  92
  93                                 |-> aligned with 8B
  94                                            |-> followed closely
  95     + meta_blkaddr blocks                                      |-> another slot
  96      _____________________________________________________________________
  97     |  ...   | inode |  xattrs  | extents  | data inline | ... | inode ...
  98     |________|_______|(optional)|(optional)|__(optional)_|_____|__________
  99              |-> aligned with the inode slot size
 100                   .                   .
 101                 .                         .
 102               .                              .
 103             .                                    .
 104           .                                         .
 105         .                                              .
 106       .____________________________________________________|-> aligned with 4B
 107       | xattr_ibody_header | shared xattrs | inline xattrs |
 108       |____________________|_______________|_______________|
 109       |->    12 bytes    <-|->x * 4 bytes<-|               .
 110                           .                .                 .
 111                     .                      .                   .
 112                .                           .                     .
 113            ._______________________________.______________________.
 114            | id | id | id | id |  ... | id | ent | ... | ent| ... |
 115            |____|____|____|____|______|____|_____|_____|____|_____|
 116                                            |-> aligned with 4B
 117                                                        |-> aligned with 4B
 118
 119     Inode could be 32 or 64 bytes, which can be distinguished from a common
 120     field which all inode versions have -- i_format:
 121
 122         __________________               __________________
 123        |     i_format     |             |     i_format     |
 124        |__________________|             |__________________|
 125        |        ...       |             |        ...       |
 126        |                  |             |                  |
 127        |__________________| 32 bytes    |                  |
 128                                         |                  |
 129                                         |__________________| 64 bytes
 130
 131     Xattrs, extents, data inline are followed by the corresponding inode with
 132     proper alignment, and they could be optional for different data mappings.
 133     _currently_ total 4 valid data mappings are supported:
 134
 135      0  flat file data without data inline (no extent);
 136      1  fixed-sized output data compression (with non-compacted indexes);
 137      2  flat file data with tail packing data inline (no extent);
 138      3  fixed-sized output data compression (with compacted indexes, v5.3+).
 139
 140     The size of the optional xattrs is indicated by i_xattr_count in inode
 141     header. Large xattrs or xattrs shared by many different files can be
 142     stored in shared xattrs metadata rather than inlined right after inode.
 143
 144  2. Shared xattrs metadata space
 145     Shared xattrs space is similar to the above inode space, started with
 146     a specific block indicated by xattr_blkaddr, organized one by one with
 147     proper align.
 148
 149     Each share xattr can also be directly found by the following formula:
 150          xattr offset = xattr_blkaddr * block_size + 4 * xattr_id
 151
 152                            |-> aligned by  4 bytes
 153     + xattr_blkaddr blocks                     |-> aligned with 4 bytes
 154      _________________________________________________________________________
 155     |  ...   | xattr_entry |  xattr data | ... |  xattr_entry | xattr data  ...
 156     |________|_____________|_____________|_____|______________|_______________
 157
 158 Directories
 159 -----------
 160 All directories are now organized in a compact on-disk format. Note that
 161 each directory block is divided into index and name areas in order to support
 162 random file lookup, and all directory entries are _strictly_ recorded in
 163 alphabetical order in order to support improved prefix binary search
 164 algorithm (could refer to the related source code).
 165
 166                  ___________________________
 167                 /                           |
 168                /              ______________|________________
 169               /              /              | nameoff1       | nameoffN-1
 170  ____________.______________._______________v________________v__________
 171 | dirent | dirent | ... | dirent | filename | filename | ... | filename |
 172 |___.0___|____1___|_____|___N-1__|____0_____|____1_____|_____|___N-1____|
 173      \                           ^
 174       \                          |                           * could have
 175        \                         |                             trailing '\0'
 176         \________________________| nameoff0
 177
 178                              Directory block
 179
 180 Note that apart from the offset of the first filename, nameoff0 also indicates
 181 the total number of directory entries in this block since it is no need to
 182 introduce another on-disk field at all.
 183
 184 Compression
 185 -----------
 186 Currently, EROFS supports 4KB fixed-sized output transparent file compression,
 187 as illustrated below:
 188
 189          |---- Variant-Length Extent ----|-------- VLE --------|----- VLE -----
 190          clusterofs                      clusterofs            clusterofs
 191          |                               |                     |   logical data
 192 _________v_______________________________v_____________________v_______________
 193 ... |    .        |             |        .    |             |  .          | ...
 194 ____|____.________|_____________|________.____|_____________|__.__________|____
 195     |-> cluster <-|-> cluster <-|-> cluster <-|-> cluster <-|-> cluster <-|
 196          size          size          size          size          size
 197           .                             .                .                   .
 198            .                       .               .                  .
 199             .                  .              .                .
 200       _______._____________._____________._____________._____________________
 201          ... |             |             |             | ... physical data
 202       _______|_____________|_____________|_____________|_____________________
 203              |-> cluster <-|-> cluster <-|-> cluster <-|
 204                   size          size          size
 205
 206 Currently each on-disk physical cluster can contain 4KB (un)compressed data
 207 at most. For each logical cluster, there is a corresponding on-disk index to
 208 describe its cluster type, physical cluster address, etc.
 209
 210 See "struct z_erofs_vle_decompressed_index" in erofs_fs.h for more details.
 211