Documentation/filesystems/aufs/README

   1
   2 Aufs4 -- advanced multi layered unification filesystem version 4.x
   3 http://aufs.sf.net
   4 Junjiro R. Okajima
   5
   6
   7 0. Introduction
   8 ----------------------------------------
   9 In the early days, aufs was entirely re-designed and re-implemented
  10 Unionfs Version 1.x series. Adding many original ideas, approaches,
  11 improvements and implementations, it becomes totally different from
  12 Unionfs while keeping the basic features.
  13 Recently, Unionfs Version 2.x series begin taking some of the same
  14 approaches to aufs1's.
  15 Unionfs is being developed by Professor Erez Zadok at Stony Brook
  16 University and his team.
  17
  18 Aufs4 supports linux-4.0 and later, and for linux-3.x series try aufs3.
  19 If you want older kernel version support, try aufs2-2.6.git or
  20 aufs2-standalone.git repository, aufs1 from CVS on SourceForge.
  21
  22 Note: it becomes clear that "Aufs was rejected. Let's give it up."
  23       According to Christoph Hellwig, linux rejects all union-type
  24       filesystems but UnionMount.
  25 <http://marc.info/?l=linux-kernel&m=123938533724484&w=2>
  26
  27 PS. Al Viro seems have a plan to merge aufs as well as overlayfs and
  28     UnionMount, and he pointed out an issue around a directory mutex
  29     lock and aufs addressed it. But it is still unsure whether aufs will
  30     be merged (or any other union solution).
  31 <http://marc.info/?l=linux-kernel&m=136312705029295&w=1>
  32
  33
  34 1. Features
  35 ----------------------------------------
  36 - unite several directories into a single virtual filesystem. The member
  37   directory is called as a branch.
  38 - you can specify the permission flags to the branch, which are 'readonly',
  39   'readwrite' and 'whiteout-able.'
  40 - by upper writable branch, internal copyup and whiteout, files/dirs on
  41   readonly branch are modifiable logically.
  42 - dynamic branch manipulation, add, del.
  43 - etc...
  44
  45 Also there are many enhancements in aufs, such as:
  46 - test only the highest one for the directory permission (dirperm1)
  47 - copyup on open (coo=)
  48 - 'move' policy for copy-up between two writable branches, after
  49   checking free space.
  50 - xattr, acl
  51 - readdir(3) in userspace.
  52 - keep inode number by external inode number table
  53 - keep the timestamps of file/dir in internal copyup operation
  54 - seekable directory, supporting NFS readdir.
  55 - whiteout is hardlinked in order to reduce the consumption of inodes
  56   on branch
  57 - do not copyup, nor create a whiteout when it is unnecessary
  58 - revert a single systemcall when an error occurs in aufs
  59 - remount interface instead of ioctl
  60 - maintain /etc/mtab by an external command, /sbin/mount.aufs.
  61 - loopback mounted filesystem as a branch
  62 - kernel thread for removing the dir who has a plenty of whiteouts
  63 - support copyup sparse file (a file which has a 'hole' in it)
  64 - default permission flags for branches
  65 - selectable permission flags for ro branch, whether whiteout can
  66   exist or not
  67 - export via NFS.
  68 - support <sysfs>/fs/aufs and <debugfs>/aufs.
  69 - support multiple writable branches, some policies to select one
  70   among multiple writable branches.
  71 - a new semantics for link(2) and rename(2) to support multiple
  72   writable branches.
  73 - no glibc changes are required.
  74 - pseudo hardlink (hardlink over branches)
  75 - allow a direct access manually to a file on branch, e.g. bypassing aufs.
  76   including NFS or remote filesystem branch.
  77 - userspace wrapper for pathconf(3)/fpathconf(3) with _PC_LINK_MAX.
  78 - and more...
  79
  80 Currently these features are dropped temporary from aufs4.
  81 See design/08plan.txt in detail.
  82 - nested mount, i.e. aufs as readonly no-whiteout branch of another aufs
  83   (robr)
  84 - statistics of aufs thread (/sys/fs/aufs/stat)
  85
  86 Features or just an idea in the future (see also design/*.txt),
  87 - reorder the branch index without del/re-add.
  88 - permanent xino files for NFSD
  89 - an option for refreshing the opened files after add/del branches
  90 - light version, without branch manipulation. (unnecessary?)
  91 - copyup in userspace
  92 - inotify in userspace
  93 - readv/writev
  94
  95
  96 2. Download
  97 ----------------------------------------
  98 There are three GIT trees for aufs4, aufs4-linux.git,
  99 aufs4-standalone.git, and aufs-util.git. Note that there is no "4" in
 100 "aufs-util.git."
 101 While the aufs-util is always necessary, you need either of aufs4-linux
 102 or aufs4-standalone.
 103
 104 The aufs4-linux tree includes the whole linux mainline GIT tree,
 105 git://git.kernel.org/.../torvalds/linux.git.
 106 And you cannot select CONFIG_AUFS_FS=m for this version, eg. you cannot
 107 build aufs4 as an external kernel module.
 108 Several extra patches are not included in this tree. Only
 109 aufs4-standalone tree contains them. They are described in the later
 110 section "Configuration and Compilation."
 111
 112 On the other hand, the aufs4-standalone tree has only aufs source files
 113 and necessary patches, and you can select CONFIG_AUFS_FS=m.
 114 But you need to apply all aufs patches manually.
 115
 116 You will find GIT branches whose name is in form of "aufs4.x" where "x"
 117 represents the linux kernel version, "linux-4.x". For instance,
 118 "aufs4.0" is for linux-4.0. For latest "linux-4.x-rcN", use
 119 "aufs4.x-rcN" branch.
 120
 121 o aufs4-linux tree
 122 $ git clone --reference /your/linux/git/tree \
 123         git://github.com/sfjro/aufs4-linux.git aufs4-linux.git
 124 - if you don't have linux GIT tree, then remove "--reference ..."
 125 $ cd aufs4-linux.git
 126 $ git checkout origin/aufs4.0
 127
 128 Or You may want to directly git-pull aufs into your linux GIT tree, and
 129 leave the patch-work to GIT.
 130 $ cd /your/linux/git/tree
 131 $ git remote add aufs4 git://github.com/sfjro/aufs4-linux.git
 132 $ git fetch aufs4
 133 $ git checkout -b my4.0 v4.0
 134 $ (add your local change...)
 135 $ git pull aufs4 aufs4.0
 136 - now you have v4.0 + your_changes + aufs4.0 in you my4.0 branch.
 137 - you may need to solve some conflicts between your_changes and
 138   aufs4.0. in this case, git-rerere is recommended so that you can
 139   solve the similar conflicts automatically when you upgrade to 4.1 or
 140   later in the future.
 141
 142 o aufs4-standalone tree
 143 $ git clone git://github.com/sfjro/aufs4-standalone.git aufs4-standalone.git
 144 $ cd aufs4-standalone.git
 145 $ git checkout origin/aufs4.0
 146
 147 o aufs-util tree
 148 $ git clone git://git.code.sf.net/p/aufs/aufs-util aufs-util.git
 149 - note that the public aufs-util.git is on SourceForge instead of
 150   GitHUB.
 151 $ cd aufs-util.git
 152 $ git checkout origin/aufs4.0
 153
 154 Note: The 4.x-rcN branch is to be used with `rc' kernel versions ONLY.
 155 The minor version number, 'x' in '4.x', of aufs may not always
 156 follow the minor version number of the kernel.
 157 Because changes in the kernel that cause the use of a new
 158 minor version number do not always require changes to aufs-util.
 159
 160 Since aufs-util has its own minor version number, you may not be
 161 able to find a GIT branch in aufs-util for your kernel's
 162 exact minor version number.
 163 In this case, you should git-checkout the branch for the
 164 nearest lower number.
 165
 166 For (an unreleased) example:
 167 If you are using "linux-4.10" and the "aufs4.10" branch
 168 does not exist in aufs-util repository, then "aufs4.9", "aufs4.8"
 169 or something numerically smaller is the branch for your kernel.
 170
 171 Also you can view all branches by
 172         $ git branch -a
 173
 174
 175 3. Configuration and Compilation
 176 ----------------------------------------
 177 Make sure you have git-checkout'ed the correct branch.
 178
 179 For aufs4-linux tree,
 180 - enable CONFIG_AUFS_FS.
 181 - set other aufs configurations if necessary.
 182
 183 For aufs4-standalone tree,
 184 There are several ways to build.
 185
 186 1.
 187 - apply ./aufs4-kbuild.patch to your kernel source files.
 188 - apply ./aufs4-base.patch too.
 189 - apply ./aufs4-mmap.patch too.
 190 - apply ./aufs4-standalone.patch too, if you have a plan to set
 191   CONFIG_AUFS_FS=m. otherwise you don't need ./aufs4-standalone.patch.
 192 - copy ./{Documentation,fs,include/uapi/linux/aufs_type.h} files to your
 193   kernel source tree. Never copy $PWD/include/uapi/linux/Kbuild.
 194 - enable CONFIG_AUFS_FS, you can select either
 195   =m or =y.
 196 - and build your kernel as usual.
 197 - install the built kernel.
 198   Note: Since linux-3.9, every filesystem module requires an alias
 199   "fs-<fsname>". You should make sure that "fs-aufs" is listed in your
 200   modules.aliases file if you set CONFIG_AUFS_FS=m.
 201 - install the header files too by "make headers_install" to the
 202   directory where you specify. By default, it is $PWD/usr.
 203   "make help" shows a brief note for headers_install.
 204 - and reboot your system.
 205
 206 2.
 207 - module only (CONFIG_AUFS_FS=m).
 208 - apply ./aufs4-base.patch to your kernel source files.
 209 - apply ./aufs4-mmap.patch too.
 210 - apply ./aufs4-standalone.patch too.
 211 - build your kernel, don't forget "make headers_install", and reboot.
 212 - edit ./config.mk and set other aufs configurations if necessary.
 213   Note: You should read $PWD/fs/aufs/Kconfig carefully which describes
 214   every aufs configurations.
 215 - build the module by simple "make".
 216   Note: Since linux-3.9, every filesystem module requires an alias
 217   "fs-<fsname>". You should make sure that "fs-aufs" is listed in your
 218   modules.aliases file.
 219 - you can specify ${KDIR} make variable which points to your kernel
 220   source tree.
 221 - install the files
 222   + run "make install" to install the aufs module, or copy the built
 223     $PWD/aufs.ko to /lib/modules/... and run depmod -a (or reboot simply).
 224   + run "make install_headers" (instead of headers_install) to install
 225     the modified aufs header file (you can specify DESTDIR which is
 226     available in aufs standalone version's Makefile only), or copy
 227     $PWD/usr/include/linux/aufs_type.h to /usr/include/linux or wherever
 228     you like manually. By default, the target directory is $PWD/usr.
 229 - no need to apply aufs4-kbuild.patch, nor copying source files to your
 230   kernel source tree.
 231
 232 Note: The header file aufs_type.h is necessary to build aufs-util
 233       as well as "make headers_install" in the kernel source tree.
 234       headers_install is subject to be forgotten, but it is essentially
 235       necessary, not only for building aufs-util.
 236       You may not meet problems without headers_install in some older
 237       version though.
 238
 239 And then,
 240 - read README in aufs-util, build and install it
 241 - note that your distribution may contain an obsoleted version of
 242   aufs_type.h in /usr/include/linux or something. When you build aufs
 243   utilities, make sure that your compiler refers the correct aufs header
 244   file which is built by "make headers_install."
 245 - if you want to use readdir(3) in userspace or pathconf(3) wrapper,
 246   then run "make install_ulib" too. And refer to the aufs manual in
 247   detail.
 248
 249 There several other patches in aufs4-standalone.git. They are all
 250 optional. When you meet some problems, they will help you.
 251 - aufs4-loopback.patch
 252   Supports a nested loopback mount in a branch-fs. This patch is
 253   unnecessary until aufs produces a message like "you may want to try
 254   another patch for loopback file".
 255 - vfs-ino.patch
 256   Modifies a system global kernel internal function get_next_ino() in
 257   order to stop assigning 0 for an inode-number. Not directly related to
 258   aufs, but recommended generally.
 259 - tmpfs-idr.patch
 260   Keeps the tmpfs inode number as the lowest value. Effective to reduce
 261   the size of aufs XINO files for tmpfs branch. Also it prevents the
 262   duplication of inode number, which is important for backup tools and
 263   other utilities. When you find aufs XINO files for tmpfs branch
 264   growing too much, try this patch.
 265 - lockdep-debug.patch
 266   Because aufs is not only an ordinary filesystem (callee of VFS), but
 267   also a caller of VFS functions for branch filesystems, subclassing of
 268   the internal locks for LOCKDEP is necessary. LOCKDEP is a debugging
 269   feature of linux kernel. If you enable CONFIG_LOCKDEP, then you will
 270   need to apply this debug patch to expand several constant values.
 271   If don't know what LOCKDEP, then you don't have apply this patch.
 272
 273
 274 4. Usage
 275 ----------------------------------------
 276 At first, make sure aufs-util are installed, and please read the aufs
 277 manual, aufs.5 in aufs-util.git tree.
 278 $ man -l aufs.5
 279
 280 And then,
 281 $ mkdir /tmp/rw /tmp/aufs
 282 # mount -t aufs -o br=/tmp/rw:${HOME} none /tmp/aufs
 283
 284 Here is another example. The result is equivalent.
 285 # mount -t aufs -o br=/tmp/rw=rw:${HOME}=ro none /tmp/aufs
 286   Or
 287 # mount -t aufs -o br:/tmp/rw none /tmp/aufs
 288 # mount -o remount,append:${HOME} /tmp/aufs
 289
 290 Then, you can see whole tree of your home dir through /tmp/aufs. If
 291 you modify a file under /tmp/aufs, the one on your home directory is
 292 not affected, instead the same named file will be newly created under
 293 /tmp/rw. And all of your modification to a file will be applied to
 294 the one under /tmp/rw. This is called the file based Copy on Write
 295 (COW) method.
 296 Aufs mount options are described in aufs.5.
 297 If you run chroot or something and make your aufs as a root directory,
 298 then you need to customize the shutdown script. See the aufs manual in
 299 detail.
 300
 301 Additionally, there are some sample usages of aufs which are a
 302 diskless system with network booting, and LiveCD over NFS.
 303 See sample dir in CVS tree on SourceForge.
 304
 305
 306 5. Contact
 307 ----------------------------------------
 308 When you have any problems or strange behaviour in aufs, please let me
 309 know with:
 310 - /proc/mounts (instead of the output of mount(8))
 311 - /sys/module/aufs/*
 312 - /sys/fs/aufs/* (if you have them)
 313 - /debug/aufs/* (if you have them)
 314 - linux kernel version
 315   if your kernel is not plain, for example modified by distributor,
 316   the url where i can download its source is necessary too.
 317 - aufs version which was printed at loading the module or booting the
 318   system, instead of the date you downloaded.
 319 - configuration (define/undefine CONFIG_AUFS_xxx)
 320 - kernel configuration or /proc/config.gz (if you have it)
 321 - behaviour which you think to be incorrect
 322 - actual operation, reproducible one is better
 323 - mailto: aufs-users at lists.sourceforge.net
 324
 325 Usually, I don't watch the Public Areas(Bugs, Support Requests, Patches,
 326 and Feature Requests) on SourceForge. Please join and write to
 327 aufs-users ML.
 328
 329
 330 6. Acknowledgements
 331 ----------------------------------------
 332 Thanks to everyone who have tried and are using aufs, whoever
 333 have reported a bug or any feedback.
 334
 335 Especially donators:
 336 Tomas Matejicek(slax.org) made a donation (much more than once).
 337         Since Apr 2010, Tomas M (the author of Slax and Linux Live
 338         scripts) is making "doubling" donations.
 339         Unfortunately I cannot list all of the donators, but I really
 340         appreciate.
 341         It ends Aug 2010, but the ordinary donation URL is still available.
 342         <http://sourceforge.net/donate/index.php?group_id=167503>
 343 Dai Itasaka made a donation (2007/8).
 344 Chuck Smith made a donation (2008/4, 10 and 12).
 345 Henk Schoneveld made a donation (2008/9).
 346 Chih-Wei Huang, ASUS, CTC donated Eee PC 4G (2008/10).
 347 Francois Dupoux made a donation (2008/11).
 348 Bruno Cesar Ribas and Luis Carlos Erpen de Bona, C3SL serves public
 349         aufs2 GIT tree (2009/2).
 350 William Grant made a donation (2009/3).
 351 Patrick Lane made a donation (2009/4).
 352 The Mail Archive (mail-archive.com) made donations (2009/5).
 353 Nippy Networks (Ed Wildgoose) made a donation (2009/7).
 354 New Dream Network, LLC (www.dreamhost.com) made a donation (2009/11).
 355 Pavel Pronskiy made a donation (2011/2).
 356 Iridium and Inmarsat satellite phone retailer (www.mailasail.com), Nippy
 357         Networks (Ed Wildgoose) made a donation for hardware (2011/3).
 358 Max Lekomcev (DOM-TV project) made a donation (2011/7, 12, 2012/3, 6 and
 359 11).
 360 Sam Liddicott made a donation (2011/9).
 361 Era Scarecrow made a donation (2013/4).
 362 Bor Ratajc made a donation (2013/4).
 363 Alessandro Gorreta made a donation (2013/4).
 364 POIRETTE Marc made a donation (2013/4).
 365 Alessandro Gorreta made a donation (2013/4).
 366 lauri kasvandik made a donation (2013/5).
 367 "pemasu from Finland" made a donation (2013/7).
 368 The Parted Magic Project made a donation (2013/9 and 11).
 369 Pavel Barta made a donation (2013/10).
 370 Nikolay Pertsev made a donation (2014/5).
 371 James B made a donation (2014/7 and 2015/7).
 372 Stefano Di Biase made a donation (2014/8).
 373 Daniel Epellei made a donation (2015/1).
 374 OmegaPhil made a donation (2016/1).
 375 Tomasz Szewczyk made a donation (2016/4).
 376 James Burry made a donation (2016/12).
 377
 378 Thank you very much.
 379 Donations are always, including future donations, very important and
 380 helpful for me to keep on developing aufs.
 381
 382
 383 7.
 384 ----------------------------------------
 385 If you are an experienced user, no explanation is needed. Aufs is
 386 just a linux filesystem.
 387
 388
 389 Enjoy!
 390
 391 # Local variables: ;
 392 # mode: text;
 393 # End: ;