From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from zeniv.linux.org.uk ([195.92.253.2]:39364 "EHLO ZenIV.linux.org.uk" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932540AbcEQG1J (ORCPT ); Tue, 17 May 2016 02:27:09 -0400 Date: Tue, 17 May 2016 07:27:06 +0100 From: Al Viro To: Linus Torvalds Cc: Linux Kernel Mailing List , linux-fsdevel Subject: Re: [git pull] vfs.git Message-ID: <20160517062705.GF14480@ZenIV.linux.org.uk> References: <20160516033258.GE14480@ZenIV.linux.org.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: linux-fsdevel-owner@vger.kernel.org List-ID: On Mon, May 16, 2016 at 08:43:33AM -0700, Linus Torvalds wrote: > On Sun, May 15, 2016 at 8:32 PM, Al Viro wrote: > > FWIW, I considered sending that pile in several pull requests, but for some > > reason git request-pull v4.6 vfs work.lookups spews something very odd into > > diffstat - files that have never been touched by it and, in fact, doing > > merge with mainline does *not* end up with those files anywhere in the > > diff. > > This is "normal" if you have multiple merge bases (which in turn > happens various ways, but all of them involved the branch having > merges in itself, either back-merges or just merging two or more topic > branches that had different starting points). OK... What happened is that I started these branches off -rc1, then by -rc3 realized that PAGE_CACHE_SIZE->PAGE_SIZE stuff was going to create one hell of merge noise and rebased work.lookups to it. I would've done the same to work.xattr, but by then I had a never-rebase branch on top of it (cifs xattr stuff) and that meant that work.xattr (including its initial segment needed in work.lookups) was stuck at -rc1. Oh, well... > > If you prefer that stuff to go in separate pulls, please say so. > > This time I'd really prefer it, especially that parallel lookup > branch. That's such a big fundamental change that it definitely merits > being merged separately rather than as part of "here's all the vfs > changes for 4.7".. OK. What I've got looks so: #work.const-path - rc1-based, no conflicts with anything, independent from everything else #work.misc - ditto #work.iov_iter - ditto, but with merge from #for-linus at some point. No conflicts either (and that #for-linus had also been rc1-based). #work.preadv2 - rc3-based, no conflicts with anything, independent from everything else #sendmsg.cifs - rc1-based, trims quite a bit of now-unneeded crap from cifs. One obvious conflict with PAGE_CACHE_SIZE->PAGE_SIZE in there. Independent from everything else. #work.xattr - rc1-based, some starts with some acl fixes, then switches ->getxattr to passing inode and dentry separately. This is the point where the things start to get tricky - that got merged into the very beginning of the -rc3-based #work.lookups, to allow untangling the security_d_instantiate() mess. That merge actually got one of the PAGE_CACHE_SIZE conflicts resolved. #work.xattr itself proceeds to switch a lot of filesystems to generic_...xattr(); no complications there. #work.lookups, after that initial merge from #work.xattr does the following: * untangle security_d_instantiate() * convert a bunch of open-coded lookup_one_len_unlocked() to calls of that thing; one such place (in overlayfs) actually yields a trivial conflict with overlayfs fixes later in the cycle - overlayfs ended up switching to a variant of lookup_one_len_unlocked() sans the permission checks. I would've dropped that commit (it gets overridden on merge from #ovl-fixes in #for-next; proper resolution is to use the variant in mainline fs/overlayfs/super.c), but I didn't want to rebase the damn thing - it was fairly late in the cycle... * some filesystems had managed to depend on lookup/lookup exclusion for *fs-internal* data structures in a way that would break if we relaxed the VFS exclusion. Fixing hadn't been hard, fortunately. * core of that series - parallel lookup machinery, replacing ->i_mutex with rwsem, making lookup_slow() take it only shared. At that point lookups happen in parallel; lookups on the same name wait for the in-progress one to be done with that dentry. Surprisingly little code, at that - almost all of it is in fs/dcache.c, with fs/namei.c changes limited to lookup_slow() - making it use the new primitive and actually switching to locking shared. * parallel readdir stuff - first of all, we provide the exclusion on per-struct file basis, same as we do for read() vs. lseek() for regular files. That takes care of most of the needed exclusion in readdir/readdir; however, these guys are trickier than lookups, so I went for switching them one-by-one - new method (->iterate_shared()) is added and filesystems are switched to it as they are either confirmed to be OK with shared lock on directory or fixed to be OK with that. I hope to kill the original method come next cycle (almost all in-tree filesystems are switched already), but it's still not quite finished. * several filesystems get switched to parallel readdir. The interesting part here is dealing with dcache preseeding by readdir; that needs minor adjustment to be safe with directory locked only shared. Most of the filesystems doing that got switched to in those commits. Important exception: NFS. Turns out that NFS folks, with their, er, insistence on VFS getting the fuck out of the way of the Smart Filesystem Code That Knows How And What To Lock(tm) have grown the locking of their own. Homegrown rwsem, with lookup/readdir/atomic_open being *writers* (sillyunlink is the reader there). Of course, with VFS getting the fuck out of the way, as requested, the actual smarts of the smart filesystem code etc. had become exposed... * do_last/lookup_open/atomic_open cleanups. As the result, open() without O_CREAT locks the directory only shared. Including the ->atomic_open() case. Backmerge from #for-linus in the middle of that - atomic_open() fix got brought in. * then comes NFS switch to saner (VFS-based ;-) locking, killing the homegrown "lookup and readdir are writers" kinda-sorta rwsem. All exclusion for sillyunlink/lookup is done by the parallel lookups mechanism. Exclusion between sillyunlink and rmdir is a real rwsem now - rmdir being the writer. Result: NFS lookups/readdirs/O_CREAT-less opens happen in parallel now. * the rest of the series consists of switching a lot of filesystems to parallel readdir; in a lot of cases ->llseek() gets simplified as well. One backmerge in there (again, #for-linus - rockridge fix). That's it for #work.lookups. There's one conflict in that pile (overlayfs one mentioned above) and there's a conflict resolved in the first merge (rc3 vs. beginning of #work.xattr). FWIW, I think the least PITA would be if I send #work.lookups + backmerge from #ovl-fixes to resolve the fs/overlay/super.c conflict. If you think that it needs to done in even smaller steps (after all, it's about 3/4 of the entire pile), please say so. Otherwise, see below: The following changes since commit 38b78a5f18584db6fa7441e0f4531b283b0e6725: ovl: ignore permissions on underlying lookup (2016-05-10 23:58:18 -0400) are available in the git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs.git for-linus for you to fetch changes up to 0e0162bb8c008fa7742f69d4d4982c8a37b88f95: Merge branch 'ovl-fixes' into for-linus (2016-05-17 02:17:59 -0400) ---------------------------------------------------------------- Al Viro (75): reiserfs_cache_default_acl(): use get_acl() don't bother with ->d_inode->i_sb - it's always equal to ->d_sb cifs: kill more bogus checks in ->...xattr() methods reiserfs: switch to generic_{get,set,remove}xattr() xattr_handler: pass dentry and inode as separate arguments of ->get() ->getxattr(): pass dentry and inode as separate arguments Merge getxattr prototype change into work.lookups security_d_instantiate(): move to the point prior to attaching dentry to inode kernfs: use lookup_one_len_unlocked() configfs_detach_prep(): make sure that wait_mutex won't go away ocfs2: don't open-code inode_lock/inode_unlock orangefs: don't open-code inode_lock/inode_unlock reiserfs: open-code reiserfs_mutex_lock_safe() in reiserfs_unpack() reconnect_one(): use lookup_one_len_unlocked() ovl_lookup_real(): use lookup_one_len_unlocked() make ext2_get_page() and friends work without external serialization nfs: missing wakeup in nfs_unblock_sillyrename() lookup_slow(): bugger off on IS_DEADDIR() from the very beginning __d_add(): don't drop/regain ->d_lock beginning of transition to parallel lookups - marking in-lookup dentries parallel lookups machinery, part 2 parallel lookups machinery, part 3 parallel lookups machinery, part 4 (and last) parallel lookups: actual switch to rwsem give readdir(2)/getdents(2)/etc. uniform exclusion with lseek() introduce a parallel variant of ->iterate() proc_fill_cache(): switch to d_alloc_parallel() proc_sys_fill_cache(): switch to d_alloc_parallel() switch all procfs directories ->iterate_shared() fuse: switch to ->iterate_shared() cifs: switch to ->iterate_shared() dcache_{readdir,dir_lseek}() users: switch to ->iterate_shared simple local filesystems: switch to ->iterate_shared() path_openat(): take O_PATH handling out of do_last() lookup_open(): expand the call of vfs_create() Merge branch 'for-linus' into work.lookups atomic_open(): don't bother with EEXIST check - it's done in do_last() atomic_open(): consolidate "overridden ENOENT" in open-yourself cases atomic_open(): massage the create_error logics a bit do_last(): get rid of duplicate ELOOP check do_last(): take fput() on error after opening to out: atomic_open(): delay open_to_namei_flags() until the method call atomic_open(): be paranoid about may_open() return value lookup_open(): lift the "fallback to !O_CREAT" logics from atomic_open() atomic_open(): reorder and clean up a bit lookup_open(): expand the call of real_lookup() lookup_open(): put the dentry fed to ->lookup() or ->atomic_open() into in-lookup hash lookup_open(): lock the parent shared unless O_CREAT is given nfs: switch to ->iterate_shared() nfs: per-name sillyunlink exclusion configfs_readdir(): make safe under shared lock kernfs: no point locking directory around that generic_file_llseek() lustre: don't need to lock inode in directory lseek more trivial ->iterate_shared conversions romfs, squashfs: switch to ->iterate_shared() fat: switch to ->iterate_shared() 9p: switch to ->iterate_shared() Merge branch 'for-linus' into work.lookups switch ecryptfs to ->iterate_shared logfs: no need to lock directory in lseek btrfs: switch to ->iterate_shared() get_acorn_filename(): deobfuscate a bit isofs: switch to ->iterate_shared() befs: constify stuff a bit befs: switch to ->iterate_shared() afs: switch to ->iterate_shared() f2fs: switch to ->iterate_shared() gfs2: switch to ->iterate_shared() hpfs: handle allocation failures in hpfs_add_pos() hpfs: switch to ->iterate_shared() hostfs: switch to ->iterate_shared() hfsplus: switch to ->iterate_shared() hfs: switch to ->iterate_shared() ext4: switch to ->iterate_shared() Merge branch 'ovl-fixes' into for-linus Andreas Gruenbacher (3): jfs: Remove unnecessary code in jfs_get_acl posix_acl: Inode acl caching fixes posix_acl: Unexport acl_by_type and make it static Documentation/filesystems/porting | 53 +++ arch/alpha/kernel/osf_sys.c | 4 +- arch/powerpc/platforms/cell/spufs/inode.c | 2 +- block/blk-map.c | 47 +-- drivers/staging/lustre/lustre/llite/dir.c | 4 +- .../staging/lustre/lustre/llite/llite_internal.h | 4 +- drivers/staging/lustre/lustre/llite/xattr.c | 6 +- fs/9p/acl.c | 8 +- fs/9p/vfs_dir.c | 4 +- fs/9p/vfs_inode.c | 2 +- fs/9p/xattr.c | 4 +- fs/affs/dir.c | 2 +- fs/afs/dir.c | 16 +- fs/autofs4/root.c | 4 +- fs/bad_inode.c | 4 +- fs/befs/befs.h | 4 +- fs/befs/btree.c | 16 +- fs/befs/btree.h | 4 +- fs/befs/datastream.c | 26 +- fs/befs/datastream.h | 11 +- fs/befs/linuxvfs.c | 6 +- fs/bfs/dir.c | 2 +- fs/btrfs/acl.c | 3 - fs/btrfs/inode.c | 2 +- fs/btrfs/ioctl.c | 18 +- fs/btrfs/tree-log.c | 6 +- fs/btrfs/xattr.c | 6 +- fs/ceph/acl.c | 2 + fs/ceph/super.h | 2 +- fs/ceph/xattr.c | 8 +- fs/cifs/cifs_dfs_ref.c | 2 +- fs/cifs/cifsfs.c | 2 +- fs/cifs/cifsfs.h | 2 +- fs/cifs/inode.c | 3 +- fs/cifs/readdir.c | 57 +-- fs/cifs/xattr.c | 56 +-- fs/coda/dir.c | 18 +- fs/compat.c | 12 +- fs/configfs/dir.c | 37 +- fs/configfs/inode.c | 2 +- fs/cramfs/inode.c | 2 +- fs/dcache.c | 267 ++++++++++++-- fs/ecryptfs/crypto.c | 5 +- fs/ecryptfs/ecryptfs_kernel.h | 4 +- fs/ecryptfs/file.c | 73 +++- fs/ecryptfs/inode.c | 23 +- fs/ecryptfs/mmap.c | 3 +- fs/efs/dir.c | 3 +- fs/efs/namei.c | 2 +- fs/exofs/dir.c | 16 +- fs/exofs/super.c | 2 +- fs/exportfs/expfs.c | 12 +- fs/ext2/acl.c | 3 - fs/ext2/dir.c | 16 +- fs/ext2/namei.c | 2 +- fs/ext2/xattr_security.c | 6 +- fs/ext2/xattr_trusted.c | 6 +- fs/ext2/xattr_user.c | 8 +- fs/ext4/acl.c | 3 - fs/ext4/dir.c | 4 +- fs/ext4/namei.c | 4 +- fs/ext4/xattr_security.c | 6 +- fs/ext4/xattr_trusted.c | 6 +- fs/ext4/xattr_user.c | 8 +- fs/f2fs/acl.c | 3 - fs/f2fs/dir.c | 2 +- fs/f2fs/namei.c | 2 +- fs/f2fs/xattr.c | 14 +- fs/fat/dir.c | 6 +- fs/file.c | 5 + fs/freevxfs/vxfs_lookup.c | 2 +- fs/fuse/dir.c | 99 +++-- fs/gfs2/file.c | 4 +- fs/gfs2/inode.c | 9 +- fs/gfs2/ops_fstype.c | 4 +- fs/gfs2/super.c | 2 +- fs/gfs2/xattr.c | 6 +- fs/hfs/attr.c | 5 +- fs/hfs/catalog.c | 3 + fs/hfs/dir.c | 12 +- fs/hfs/hfs_fs.h | 5 +- fs/hfs/inode.c | 2 + fs/hfsplus/catalog.c | 3 + fs/hfsplus/dir.c | 12 +- fs/hfsplus/hfsplus_fs.h | 1 + fs/hfsplus/inode.c | 1 + fs/hfsplus/posix_acl.c | 3 - fs/hfsplus/super.c | 1 + fs/hfsplus/xattr.c | 10 +- fs/hfsplus/xattr.h | 2 +- fs/hfsplus/xattr_security.c | 6 +- fs/hfsplus/xattr_trusted.c | 6 +- fs/hfsplus/xattr_user.c | 6 +- fs/hostfs/hostfs_kern.c | 2 +- fs/hpfs/dir.c | 12 +- fs/hpfs/dnode.c | 8 +- fs/hpfs/hpfs_fn.h | 2 +- fs/inode.c | 17 +- fs/isofs/dir.c | 4 +- fs/isofs/rock.c | 13 +- fs/jffs2/acl.c | 2 - fs/jffs2/dir.c | 4 +- fs/jffs2/security.c | 6 +- fs/jffs2/super.c | 2 +- fs/jffs2/xattr_trusted.c | 6 +- fs/jffs2/xattr_user.c | 6 +- fs/jfs/acl.c | 6 - fs/jfs/jfs_xattr.h | 2 +- fs/jfs/namei.c | 2 +- fs/jfs/xattr.c | 8 +- fs/kernfs/dir.c | 17 +- fs/kernfs/inode.c | 6 +- fs/kernfs/kernfs-internal.h | 4 +- fs/kernfs/mount.c | 5 +- fs/libfs.c | 11 +- fs/logfs/dir.c | 4 +- fs/minix/dir.c | 2 +- fs/namei.c | 399 ++++++++++----------- fs/nfs/dir.c | 80 +++-- fs/nfs/direct.c | 2 +- fs/nfs/inode.c | 4 +- fs/nfs/nfs3acl.c | 43 ++- fs/nfs/nfs4proc.c | 14 +- fs/nfs/nfstrace.h | 2 +- fs/nfs/unlink.c | 192 +++------- fs/nfsd/nfs3proc.c | 4 +- fs/nfsd/nfs3xdr.c | 2 +- fs/nfsd/nfsfh.c | 2 +- fs/nilfs2/dir.c | 16 +- fs/nilfs2/namei.c | 2 +- fs/ocfs2/aops.c | 4 +- fs/ocfs2/dlmglue.c | 3 + fs/ocfs2/file.c | 2 +- fs/ocfs2/inode.c | 2 +- fs/ocfs2/xattr.c | 20 +- fs/omfs/dir.c | 2 +- fs/open.c | 2 +- fs/openpromfs/inode.c | 2 +- fs/orangefs/file.c | 4 +- fs/orangefs/orangefs-kernel.h | 4 +- fs/orangefs/xattr.c | 10 +- fs/overlayfs/inode.c | 4 +- fs/overlayfs/overlayfs.h | 4 +- fs/overlayfs/readdir.c | 4 +- fs/overlayfs/super.c | 2 +- fs/posix_acl.c | 116 +++--- fs/proc/base.c | 35 +- fs/proc/fd.c | 8 +- fs/proc/generic.c | 2 +- fs/proc/namespaces.c | 3 +- fs/proc/proc_net.c | 2 +- fs/proc/proc_sysctl.c | 17 +- fs/proc/root.c | 4 +- fs/qnx4/dir.c | 2 +- fs/qnx6/dir.c | 2 +- fs/read_write.c | 12 - fs/readdir.c | 37 +- fs/reiserfs/dir.c | 2 +- fs/reiserfs/file.c | 6 +- fs/reiserfs/ioctl.c | 6 +- fs/reiserfs/namei.c | 18 +- fs/reiserfs/xattr.c | 54 --- fs/reiserfs/xattr.h | 9 +- fs/reiserfs/xattr_acl.c | 8 +- fs/reiserfs/xattr_security.c | 19 +- fs/reiserfs/xattr_trusted.c | 19 +- fs/reiserfs/xattr_user.c | 19 +- fs/romfs/super.c | 4 +- fs/splice.c | 3 + fs/squashfs/dir.c | 4 +- fs/squashfs/xattr.c | 6 +- fs/sysv/dir.c | 2 +- fs/ubifs/dir.c | 2 +- fs/ubifs/ubifs.h | 4 +- fs/ubifs/xattr.c | 6 +- fs/udf/dir.c | 2 +- fs/udf/namei.c | 2 +- fs/ufs/dir.c | 16 +- fs/ufs/super.c | 2 +- fs/xattr.c | 12 +- fs/xfs/xfs_acl.c | 20 +- fs/xfs/xfs_file.c | 2 +- fs/xfs/xfs_xattr.c | 6 +- include/linux/dcache.h | 26 +- include/linux/file.h | 13 + include/linux/fs.h | 51 ++- include/linux/nfs_fs.h | 11 +- include/linux/nfs_xdr.h | 4 +- include/linux/posix_acl.h | 1 - include/linux/uio.h | 1 + include/linux/xattr.h | 5 +- include/trace/events/ext4.h | 6 +- kernel/audit_watch.c | 2 +- lib/iov_iter.c | 19 + mm/shmem.c | 9 +- net/socket.c | 2 +- security/commoncap.c | 6 +- security/integrity/evm/evm_main.c | 6 +- security/selinux/hooks.c | 11 +- security/smack/smack_lsm.c | 6 +- 200 files changed, 1545 insertions(+), 1300 deletions(-)