From: Miklos Szeredi <miklos@szeredi.hu>
To: viro@ZenIV.linux.org.uk, torvalds@linux-foundation.org
Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org,
hch@infradead.org, akpm@linux-foundation.org, apw@canonical.com,
nbd@openwrt.org, neilb@suse.de, jordipujolp@gmail.com,
ezk@fsl.cs.sunysb.edu, dhowells@redhat.com,
sedat.dilek@googlemail.com, hooanon05@yahoo.co.jp,
mszeredi@suse.cz
Subject: [PATCH 07/13] overlay-overlay-filesystem-documentation
Date: Tue, 12 Mar 2013 16:41:42 +0100 [thread overview]
Message-ID: <1363102908-28956-8-git-send-email-miklos@szeredi.hu> (raw)
In-Reply-To: <1363102908-28956-1-git-send-email-miklos@szeredi.hu>
From: Neil Brown <neilb@suse.de>
Document the overlay filesystem.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
---
Documentation/filesystems/overlayfs.txt | 199 +++++++++++++++++++++++++++++++
MAINTAINERS | 7 ++
2 files changed, 206 insertions(+)
create mode 100644 Documentation/filesystems/overlayfs.txt
diff --git a/Documentation/filesystems/overlayfs.txt b/Documentation/filesystems/overlayfs.txt
new file mode 100644
index 0000000..00dbab0
--- /dev/null
+++ b/Documentation/filesystems/overlayfs.txt
@@ -0,0 +1,199 @@
+Written by: Neil Brown <neilb@suse.de>
+
+Overlay Filesystem
+==================
+
+This document describes a prototype for a new approach to providing
+overlay-filesystem functionality in Linux (sometimes referred to as
+union-filesystems). An overlay-filesystem tries to present a
+filesystem which is the result over overlaying one filesystem on top
+of the other.
+
+The result will inevitably fail to look exactly like a normal
+filesystem for various technical reasons. The expectation is that
+many use cases will be able to ignore these differences.
+
+This approach is 'hybrid' because the objects that appear in the
+filesystem do not all appear to belong to that filesystem. In many
+cases an object accessed in the union will be indistinguishable
+from accessing the corresponding object from the original filesystem.
+This is most obvious from the 'st_dev' field returned by stat(2).
+
+While directories will report an st_dev from the overlay-filesystem,
+all non-directory objects will report an st_dev from the lower or
+upper filesystem that is providing the object. Similarly st_ino will
+only be unique when combined with st_dev, and both of these can change
+over the lifetime of a non-directory object. Many applications and
+tools ignore these values and will not be affected.
+
+Upper and Lower
+---------------
+
+An overlay filesystem combines two filesystems - an 'upper' filesystem
+and a 'lower' filesystem. When a name exists in both filesystems, the
+object in the 'upper' filesystem is visible while the object in the
+'lower' filesystem is either hidden or, in the case of directories,
+merged with the 'upper' object.
+
+It would be more correct to refer to an upper and lower 'directory
+tree' rather than 'filesystem' as it is quite possible for both
+directory trees to be in the same filesystem and there is no
+requirement that the root of a filesystem be given for either upper or
+lower.
+
+The lower filesystem can be any filesystem supported by Linux and does
+not need to be writable. The lower filesystem can even be another
+overlayfs. The upper filesystem will normally be writable and if it
+is it must support the creation of trusted.* extended attributes, and
+must provide valid d_type in readdir responses, at least for symbolic
+links - so NFS is not suitable.
+
+A read-only overlay of two read-only filesystems may use any
+filesystem type.
+
+Directories
+-----------
+
+Overlaying mainly involves directories. If a given name appears in both
+upper and lower filesystems and refers to a non-directory in either,
+then the lower object is hidden - the name refers only to the upper
+object.
+
+Where both upper and lower objects are directories, a merged directory
+is formed.
+
+At mount time, the two directories given as mount options are combined
+into a merged directory:
+
+ mount -t overlayfs overlayfs -olowerdir=/lower,upperdir=/upper /overlay
+
+Then whenever a lookup is requested in such a merged directory, the
+lookup is performed in each actual directory and the combined result
+is cached in the dentry belonging to the overlay filesystem. If both
+actual lookups find directories, both are stored and a merged
+directory is created, otherwise only one is stored: the upper if it
+exists, else the lower.
+
+Only the lists of names from directories are merged. Other content
+such as metadata and extended attributes are reported for the upper
+directory only. These attributes of the lower directory are hidden.
+
+whiteouts and opaque directories
+--------------------------------
+
+In order to support rm and rmdir without changing the lower
+filesystem, an overlay filesystem needs to record in the upper filesystem
+that files have been removed. This is done using whiteouts and opaque
+directories (non-directories are always opaque).
+
+The overlay filesystem uses extended attributes with a
+"trusted.overlay." prefix to record these details.
+
+A whiteout is created as a symbolic link with target
+"(overlay-whiteout)" and with xattr "trusted.overlay.whiteout" set to "y".
+When a whiteout is found in the upper level of a merged directory, any
+matching name in the lower level is ignored, and the whiteout itself
+is also hidden.
+
+A directory is made opaque by setting the xattr "trusted.overlay.opaque"
+to "y". Where the upper filesystem contains an opaque directory, any
+directory in the lower filesystem with the same name is ignored.
+
+readdir
+-------
+
+When a 'readdir' request is made on a merged directory, the upper and
+lower directories are each read and the name lists merged in the
+obvious way (upper is read first, then lower - entries that already
+exist are not re-added). This merged name list is cached in the
+'struct file' and so remains as long as the file is kept open. If the
+directory is opened and read by two processes at the same time, they
+will each have separate caches. A seekdir to the start of the
+directory (offset 0) followed by a readdir will cause the cache to be
+discarded and rebuilt.
+
+This means that changes to the merged directory do not appear while a
+directory is being read. This is unlikely to be noticed by many
+programs.
+
+seek offsets are assigned sequentially when the directories are read.
+Thus if
+ - read part of a directory
+ - remember an offset, and close the directory
+ - re-open the directory some time later
+ - seek to the remembered offset
+
+there may be little correlation between the old and new locations in
+the list of filenames, particularly if anything has changed in the
+directory.
+
+Readdir on directories that are not merged is simply handled by the
+underlying directory (upper or lower).
+
+
+Non-directories
+---------------
+
+Objects that are not directories (files, symlinks, device-special
+files etc.) are presented either from the upper or lower filesystem as
+appropriate. When a file in the lower filesystem is accessed in a way
+the requires write-access, such as opening for write access, changing
+some metadata etc., the file is first copied from the lower filesystem
+to the upper filesystem (copy_up). Note that creating a hard-link
+also requires copy_up, though of course creation of a symlink does
+not.
+
+The copy_up may turn out to be unnecessary, for example if the file is
+opened for read-write but the data is not modified.
+
+The copy_up process first makes sure that the containing directory
+exists in the upper filesystem - creating it and any parents as
+necessary. It then creates the object with the same metadata (owner,
+mode, mtime, symlink-target etc.) and then if the object is a file, the
+data is copied from the lower to the upper filesystem. Finally any
+extended attributes are copied up.
+
+Once the copy_up is complete, the overlay filesystem simply
+provides direct access to the newly created file in the upper
+filesystem - future operations on the file are barely noticed by the
+overlay filesystem (though an operation on the name of the file such as
+rename or unlink will of course be noticed and handled).
+
+
+Non-standard behavior
+---------------------
+
+The copy_up operation essentially creates a new, identical file and
+moves it over to the old name. The new file may be on a different
+filesystem, so both st_dev and st_ino of the file may change.
+
+Any open files referring to this inode will access the old data and
+metadata. Similarly any file locks obtained before copy_up will not
+apply to the copied up file.
+
+On a file opened with O_RDONLY fchmod(2), fchown(2), futimesat(2) and
+fsetxattr(2) will fail with EROFS.
+
+If a file with multiple hard links is copied up, then this will
+"break" the link. Changes will not be propagated to other names
+referring to the same inode.
+
+Symlinks in /proc/PID/ and /proc/PID/fd which point to a non-directory
+object in overlayfs will not contain valid absolute paths, only
+relative paths leading up to the filesystem's root. This will be
+fixed in the future.
+
+Some operations are not atomic, for example a crash during copy_up or
+rename will leave the filesystem in an inconsistent state. This will
+be addressed in the future.
+
+Changes to underlying filesystems
+---------------------------------
+
+Offline changes, when the overlay is not mounted, are allowed to either
+the upper or the lower trees.
+
+Changes to the underlying filesystems while part of a mounted overlay
+filesystem are not allowed. If the underlying filesystem is changed,
+the behavior of the overlay is undefined, though it will not result in
+a crash or deadlock.
diff --git a/MAINTAINERS b/MAINTAINERS
index 9561658..9ea89b8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -5872,6 +5872,13 @@ F: drivers/scsi/osd/
F: include/scsi/osd_*
F: fs/exofs/
+OVERLAYFS FILESYSTEM
+M: Miklos Szeredi <miklos@szeredi.hu>
+L: linux-fsdevel@vger.kernel.org
+S: Supported
+F: fs/overlayfs/*
+F: Documentation/filesystems/overlayfs.txt
+
P54 WIRELESS DRIVER
M: Christian Lamparter <chunkeey@googlemail.com>
L: linux-wireless@vger.kernel.org
--
1.7.10.4
next prev parent reply other threads:[~2013-03-12 15:45 UTC|newest]
Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-03-12 15:41 [PATCH 00/13] overlay filesystem: request for inclusion (v16) Miklos Szeredi
2013-03-12 15:41 ` [PATCH 01/13] vfs-add-i_op-dentry_open Miklos Szeredi
2013-03-12 15:41 ` [PATCH 02/13] vfs-export-do_splice_direct-to-modules Miklos Szeredi
2013-03-12 15:41 ` [PATCH 03/13] vfs-introduce-clone_private_mount Miklos Szeredi
2013-03-12 15:41 ` [PATCH 04/13] overlay filesystem Miklos Szeredi
2013-03-12 15:41 ` [PATCH 05/13] overlayfs-add-statfs-support Miklos Szeredi
2013-03-12 15:41 ` [PATCH 06/13] overlayfs-implement-show_options Miklos Szeredi
2013-03-12 15:41 ` Miklos Szeredi [this message]
2013-03-12 15:41 ` [PATCH 08/13] fs-limit-filesystem-stacking-depth Miklos Szeredi
2013-03-12 15:41 ` [PATCH 09/13] overlayfs-fix-possible-leak-in-ovl_new_inode Miklos Szeredi
2013-03-12 15:41 ` [PATCH 10/13] overlayfs-create-new-inode-in-ovl_link Miklos Szeredi
2013-03-12 15:41 ` [PATCH 11/13] vfs-export-inode_permission-to-modules Miklos Szeredi
2013-03-12 15:41 ` [PATCH 12/13] ovl-switch-to-inode_permission Miklos Szeredi
2013-03-12 15:41 ` [PATCH 13/13] overlayfs-copy-up-i_uid-i_gid-from-the-underlying-inode Miklos Szeredi
2013-03-12 16:49 ` [PATCH 00/13] overlay filesystem: request for inclusion (v16) Sedat Dilek
2013-03-12 20:00 ` Miklos Szeredi
2013-03-12 17:22 ` J. R. Okajima
2013-03-12 20:01 ` Miklos Szeredi
2013-03-12 21:50 ` Linus Torvalds
2013-03-12 22:23 ` Al Viro
2013-03-13 9:42 ` Sedat Dilek
2013-03-13 15:24 ` Miklos Szeredi
2013-03-13 18:52 ` Al Viro
2013-03-13 22:09 ` Miklos Szeredi
2013-03-13 23:19 ` Al Viro
2013-03-14 10:37 ` Miklos Szeredi
2013-03-14 12:57 ` DT_WHT (Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)) J. R. Okajima
2013-03-14 22:59 ` [PATCH 00/13] overlay filesystem: request for inclusion (v16) Al Viro
2013-03-18 11:27 ` Miklos Szeredi
2013-03-13 10:09 ` Sedat Dilek
2013-03-13 10:57 ` Miklos Szeredi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1363102908-28956-8-git-send-email-miklos@szeredi.hu \
--to=miklos@szeredi.hu \
--cc=akpm@linux-foundation.org \
--cc=apw@canonical.com \
--cc=dhowells@redhat.com \
--cc=ezk@fsl.cs.sunysb.edu \
--cc=hch@infradead.org \
--cc=hooanon05@yahoo.co.jp \
--cc=jordipujolp@gmail.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mszeredi@suse.cz \
--cc=nbd@openwrt.org \
--cc=neilb@suse.de \
--cc=sedat.dilek@googlemail.com \
--cc=torvalds@linux-foundation.org \
--cc=viro@ZenIV.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).