[RFC] vfs: avoid sb->s_umount lock while changing bind-mount flags

[RFC] vfs: avoid sb->s_umount lock while changing bind-mount flags

[Aditya Kali]

During remount of a bind mount (mount -o remount,bind,ro,... /mnt/mntpt),
we currently take down_write(&sb->s_umount). This causes the remount
operation to get blocked behind writes occuring on device (possibly
mounted somewhere else). We have observed that simply trying to change
the bind-mount from read-write to read-only can take several seconds
becuase writeback is in progress. Looking at the code it seems to me that
we need s_umount lock only around the do_remount_sb() call.
vfsmount_lock seems enough to protect the flag change on the mount.
So this patch fixes the locking so that changing of flags can happen
outside the down_write(&sb->s_umount).

I wanted to get comments if I am violating any assumption around this code.
Another thing that I was curious about was if we need the
{lock|unlock}_mount(path) around this code. Please advise.

Signed-off-by: Aditya Kali <adityakali@google.com>
---
 fs/namespace.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/fs/namespace.c b/fs/namespace.c
index da5c494..4b9c839 100644
--- a/fs/namespace.c
+++ b/fs/namespace.c
@@ -1838,20 +1838,21 @@ static int do_remount(struct path *path, int flags, int mnt_flags,
 	if (err)
 		return err;
 
-	down_write(&sb->s_umount);
 	if (flags & MS_BIND)
 		err = change_mount_flags(path->mnt, flags);
 	else if (!capable(CAP_SYS_ADMIN))
 		err = -EPERM;
-	else
+	else {
+		down_write(&sb->s_umount);
 		err = do_remount_sb(sb, flags, data, 0);
+		up_write(&sb->s_umount);
+	}
 	if (!err) {
 		br_write_lock(&vfsmount_lock);
 		mnt_flags |= mnt->mnt.mnt_flags & MNT_PROPAGATION_MASK;
 		mnt->mnt.mnt_flags = mnt_flags;
 		br_write_unlock(&vfsmount_lock);
 	}
-	up_write(&sb->s_umount);
 	if (!err) {
 		br_write_lock(&vfsmount_lock);
 		touch_mnt_namespace(mnt->mnt_ns);
-- 
1.8.4

Re: [RFC] vfs: avoid sb->s_umount lock while changing bind-mount flags

[Al Viro]

On Mon, Sep 16, 2013 at 10:42:30AM -0700, Aditya Kali wrote:
> During remount of a bind mount (mount -o remount,bind,ro,... /mnt/mntpt),
> we currently take down_write(&sb->s_umount). This causes the remount
> operation to get blocked behind writes occuring on device (possibly
> mounted somewhere else). We have observed that simply trying to change
> the bind-mount from read-write to read-only can take several seconds
> becuase writeback is in progress. Looking at the code it seems to me that
> we need s_umount lock only around the do_remount_sb() call.
> vfsmount_lock seems enough to protect the flag change on the mount.
> So this patch fixes the locking so that changing of flags can happen
> outside the down_write(&sb->s_umount).

What's to prevent mount -o remount,ro /mnt and mount -o remount,rw,nodev /mnt
racing and ending up with that sucker rw and without nodev?  As for
lock_mount... nope - we carefully do *not* hold namespace_sem over any kind
of fs operations.  Anything getting stuck while holding it will have
really nasty consequences.

So ->s_umount here is inelegant, but alternatives sucks worse...

Re: [RFC] vfs: avoid sb->s_umount lock while changing bind-mount flags

[Aditya Kali]

On 09/16/2013 07:40 PM, Al Viro wrote:
> On Mon, Sep 16, 2013 at 10:42:30AM -0700, Aditya Kali wrote:
>> During remount of a bind mount (mount -o remount,bind,ro,... /mnt/mntpt),
>> we currently take down_write(&sb->s_umount). This causes the remount
>> operation to get blocked behind writes occuring on device (possibly
>> mounted somewhere else). We have observed that simply trying to change
>> the bind-mount from read-write to read-only can take several seconds
>> becuase writeback is in progress. Looking at the code it seems to me that
>> we need s_umount lock only around the do_remount_sb() call.
>> vfsmount_lock seems enough to protect the flag change on the mount.
>> So this patch fixes the locking so that changing of flags can happen
>> outside the down_write(&sb->s_umount).
>
> What's to prevent mount -o remount,ro /mnt and mount -o remount,rw,nodev /mnt
> racing and ending up with that sucker rw and without nodev?

Thanks for the reply! I see the problem in my patch. Please find the 
second attempt at this patch below. I have tried to keep the non-MS_BIND 
remount semantics same while moving the MS_BIND remount code outside of 
s_umount lock. Is it OK to not synchronize the non-MS_BIND 
do_remount_sb() call with change of mnt_flags in MS_BIND case?


---
  fs/namespace.c | 37 +++++++++++++++++++++++--------------
  1 file changed, 23 insertions(+), 14 deletions(-)

diff --git a/fs/namespace.c b/fs/namespace.c
index da5c494..25c4faf 100644
--- a/fs/namespace.c
+++ b/fs/namespace.c
@@ -454,11 +454,13 @@ void mnt_drop_write_file(struct file *file)
  }
  EXPORT_SYMBOL(mnt_drop_write_file);

+/*
+ * Must be called under br_write_lock(&vfsmount_lock);
+ */
  static int mnt_make_readonly(struct mount *mnt)
  {
  	int ret = 0;

-	br_write_lock(&vfsmount_lock);
  	mnt->mnt.mnt_flags |= MNT_WRITE_HOLD;
  	/*
  	 * After storing MNT_WRITE_HOLD, we'll read the counters. This store
@@ -492,15 +494,15 @@ static int mnt_make_readonly(struct mount *mnt)
  	 */
  	smp_wmb();
  	mnt->mnt.mnt_flags &= ~MNT_WRITE_HOLD;
-	br_write_unlock(&vfsmount_lock);
  	return ret;
  }

+/*
+ * Must be called under br_write_lock(&vfsmount_lock);
+ */
  static void __mnt_unmake_readonly(struct mount *mnt)
  {
-	br_write_lock(&vfsmount_lock);
  	mnt->mnt.mnt_flags &= ~MNT_READONLY;
-	br_write_unlock(&vfsmount_lock);
  }

  int sb_prepare_remount_readonly(struct super_block *sb)
@@ -1838,20 +1840,27 @@ static int do_remount(struct path *path, int 
flags, int mnt_flags,
  	if (err)
  		return err;

-	down_write(&sb->s_umount);
-	if (flags & MS_BIND)
+	if (flags & MS_BIND) {
+		br_write_lock(&vfsmount_lock);
  		err = change_mount_flags(path->mnt, flags);
-	else if (!capable(CAP_SYS_ADMIN))
+		if (!err) {
+			mnt_flags |= mnt->mnt.mnt_flags & MNT_PROPAGATION_MASK;
+			mnt->mnt.mnt_flags = mnt_flags;
+		}
+		br_write_unlock(&vfsmount_lock);
+	} else if (!capable(CAP_SYS_ADMIN))
  		err = -EPERM;
-	else
+	else {
+		down_write(&sb->s_umount);
  		err = do_remount_sb(sb, flags, data, 0);
-	if (!err) {
-		br_write_lock(&vfsmount_lock);
-		mnt_flags |= mnt->mnt.mnt_flags & MNT_PROPAGATION_MASK;
-		mnt->mnt.mnt_flags = mnt_flags;
-		br_write_unlock(&vfsmount_lock);
+		if (!err) {
+			br_write_lock(&vfsmount_lock);
+			mnt_flags |= mnt->mnt.mnt_flags & MNT_PROPAGATION_MASK;
+			mnt->mnt.mnt_flags = mnt_flags;
+			br_write_unlock(&vfsmount_lock);
+		}
+		up_write(&sb->s_umount);
  	}
-	up_write(&sb->s_umount);
  	if (!err) {
  		br_write_lock(&vfsmount_lock);
  		touch_mnt_namespace(mnt->mnt_ns);
-- 
1.8.4

Re: [RFC] vfs: avoid sb->s_umount lock while changing bind-mount flags

[Aditya Kali]

Hi Al and other fs-developers,

Please let me know what you think about this patch.

Thanks,
-- 
Aditya

On Thu, Sep 19, 2013 at 1:13 PM, Aditya Kali <adityakali@google.com> wrote:
>
>
> On 09/16/2013 07:40 PM, Al Viro wrote:
>>
>> On Mon, Sep 16, 2013 at 10:42:30AM -0700, Aditya Kali wrote:
>>>
>>> During remount of a bind mount (mount -o remount,bind,ro,... /mnt/mntpt),
>>> we currently take down_write(&sb->s_umount). This causes the remount
>>> operation to get blocked behind writes occuring on device (possibly
>>> mounted somewhere else). We have observed that simply trying to change
>>> the bind-mount from read-write to read-only can take several seconds
>>> becuase writeback is in progress. Looking at the code it seems to me that
>>> we need s_umount lock only around the do_remount_sb() call.
>>> vfsmount_lock seems enough to protect the flag change on the mount.
>>> So this patch fixes the locking so that changing of flags can happen
>>> outside the down_write(&sb->s_umount).
>>
>>
>> What's to prevent mount -o remount,ro /mnt and mount -o remount,rw,nodev
>> /mnt
>> racing and ending up with that sucker rw and without nodev?
>
>
> Thanks for the reply! I see the problem in my patch. Please find the second
> attempt at this patch below. I have tried to keep the non-MS_BIND remount
> semantics same while moving the MS_BIND remount code outside of s_umount
> lock. Is it OK to not synchronize the non-MS_BIND do_remount_sb() call with
> change of mnt_flags in MS_BIND case?
>
>
> ---
>  fs/namespace.c | 37 +++++++++++++++++++++++--------------
>  1 file changed, 23 insertions(+), 14 deletions(-)
>
> diff --git a/fs/namespace.c b/fs/namespace.c
> index da5c494..25c4faf 100644
> --- a/fs/namespace.c
> +++ b/fs/namespace.c
>
> @@ -454,11 +454,13 @@ void mnt_drop_write_file(struct file *file)
>  }
>  EXPORT_SYMBOL(mnt_drop_write_file);
>
> +/*
> + * Must be called under br_write_lock(&vfsmount_lock);
> + */
>  static int mnt_make_readonly(struct mount *mnt)
>  {
>         int ret = 0;
>
> -       br_write_lock(&vfsmount_lock);
>         mnt->mnt.mnt_flags |= MNT_WRITE_HOLD;
>         /*
>          * After storing MNT_WRITE_HOLD, we'll read the counters. This store
> @@ -492,15 +494,15 @@ static int mnt_make_readonly(struct mount *mnt)
>          */
>         smp_wmb();
>         mnt->mnt.mnt_flags &= ~MNT_WRITE_HOLD;
> -       br_write_unlock(&vfsmount_lock);
>         return ret;
>  }
>
> +/*
> + * Must be called under br_write_lock(&vfsmount_lock);
> + */
>  static void __mnt_unmake_readonly(struct mount *mnt)
>  {
> -       br_write_lock(&vfsmount_lock);
>         mnt->mnt.mnt_flags &= ~MNT_READONLY;
> -       br_write_unlock(&vfsmount_lock);
>  }
>
>  int sb_prepare_remount_readonly(struct super_block *sb)
> @@ -1838,20 +1840,27 @@ static int do_remount(struct path *path, int flags,
> int mnt_flags,
>
>         if (err)
>                 return err;
>
> -       down_write(&sb->s_umount);
> -       if (flags & MS_BIND)
> +       if (flags & MS_BIND) {
> +               br_write_lock(&vfsmount_lock);
>                 err = change_mount_flags(path->mnt, flags);
> -       else if (!capable(CAP_SYS_ADMIN))
> +               if (!err) {
> +                       mnt_flags |= mnt->mnt.mnt_flags &
> MNT_PROPAGATION_MASK;
> +                       mnt->mnt.mnt_flags = mnt_flags;
> +               }
> +               br_write_unlock(&vfsmount_lock);
> +       } else if (!capable(CAP_SYS_ADMIN))
>
>                 err = -EPERM;
> -       else
> +       else {
> +               down_write(&sb->s_umount);
>                 err = do_remount_sb(sb, flags, data, 0);
> -       if (!err) {
> -               br_write_lock(&vfsmount_lock);
> -               mnt_flags |= mnt->mnt.mnt_flags & MNT_PROPAGATION_MASK;
> -               mnt->mnt.mnt_flags = mnt_flags;
> -               br_write_unlock(&vfsmount_lock);
> +               if (!err) {
> +                       br_write_lock(&vfsmount_lock);
> +                       mnt_flags |= mnt->mnt.mnt_flags &
> MNT_PROPAGATION_MASK;
> +                       mnt->mnt.mnt_flags = mnt_flags;
> +                       br_write_unlock(&vfsmount_lock);
> +               }
> +               up_write(&sb->s_umount);
>         }
> -       up_write(&sb->s_umount);
>         if (!err) {
>                 br_write_lock(&vfsmount_lock);
>                 touch_mnt_namespace(mnt->mnt_ns);
> --
> 1.8.4
>
>
>

Re: [RFC] vfs: avoid sb->s_umount lock while changing bind-mount flags

[Aditya Kali]

+Ted Ts'o, Tejun Heo, Jens Axboe


On 09/30/2013 10:54 AM, Aditya Kali wrote:
> Hi Al and other fs-developers,
>
> Please let me know what you think about this patch.
>
> Thanks,
>
 > On Thu, Sep 19, 2013 at 1:13 PM, Aditya Kali <adityakali@google.com> 
wrote:
 >>
 >>
 >> On 09/16/2013 07:40 PM, Al Viro wrote:
 >>>
 >>> On Mon, Sep 16, 2013 at 10:42:30AM -0700, Aditya Kali wrote:
 >>>>
 >>>> During remount of a bind mount (mount -o remount,bind,ro,... 
/mnt/mntpt),
 >>>> we currently take down_write(&sb->s_umount). This causes the remount
 >>>> operation to get blocked behind writes occuring on device (possibly
 >>>> mounted somewhere else). We have observed that simply trying to change
 >>>> the bind-mount from read-write to read-only can take several seconds
 >>>> becuase writeback is in progress. Looking at the code it seems to 
me that
 >>>> we need s_umount lock only around the do_remount_sb() call.
 >>>> vfsmount_lock seems enough to protect the flag change on the mount.
 >>>> So this patch fixes the locking so that changing of flags can happen
 >>>> outside the down_write(&sb->s_umount).
 >>>
 >>>
 >>> What's to prevent mount -o remount,ro /mnt and mount -o 
remount,rw,nodev
 >>> /mnt
 >>> racing and ending up with that sucker rw and without nodev?
 >>
 >>
 >> Thanks for the reply! I see the problem in my patch. Please find the 
second
 >> attempt at this patch below. I have tried to keep the non-MS_BIND 
remount
 >> semantics same while moving the MS_BIND remount code outside of s_umount
 >> lock. Is it OK to not synchronize the non-MS_BIND do_remount_sb() 
call with
 >> change of mnt_flags in MS_BIND case?
 >>


---
  fs/namespace.c | 37 +++++++++++++++++++++++--------------
  1 file changed, 23 insertions(+), 14 deletions(-)

diff --git a/fs/namespace.c b/fs/namespace.c
index da5c494..25c4faf 100644
--- a/fs/namespace.c
+++ b/fs/namespace.c
@@ -454,11 +454,13 @@ void mnt_drop_write_file(struct file *file)
  }
  EXPORT_SYMBOL(mnt_drop_write_file);

+/*
+ * Must be called under br_write_lock(&vfsmount_lock);
+ */
  static int mnt_make_readonly(struct mount *mnt)
  {
  	int ret = 0;

-	br_write_lock(&vfsmount_lock);
  	mnt->mnt.mnt_flags |= MNT_WRITE_HOLD;
  	/*
  	 * After storing MNT_WRITE_HOLD, we'll read the counters. This store
@@ -492,15 +494,15 @@ static int mnt_make_readonly(struct mount *mnt)
  	 */
  	smp_wmb();
  	mnt->mnt.mnt_flags &= ~MNT_WRITE_HOLD;
-	br_write_unlock(&vfsmount_lock);
  	return ret;
  }

+/*
+ * Must be called under br_write_lock(&vfsmount_lock);
+ */
  static void __mnt_unmake_readonly(struct mount *mnt)
  {
-	br_write_lock(&vfsmount_lock);
  	mnt->mnt.mnt_flags &= ~MNT_READONLY;
-	br_write_unlock(&vfsmount_lock);
  }

  int sb_prepare_remount_readonly(struct super_block *sb)
@@ -1838,20 +1840,27 @@ static int do_remount(struct path *path, int 
flags, int mnt_flags,
  	if (err)
  		return err;

-	down_write(&sb->s_umount);
-	if (flags & MS_BIND)
+	if (flags & MS_BIND) {
+		br_write_lock(&vfsmount_lock);
  		err = change_mount_flags(path->mnt, flags);
-	else if (!capable(CAP_SYS_ADMIN))
+		if (!err) {
+			mnt_flags |= mnt->mnt.mnt_flags & MNT_PROPAGATION_MASK;
+			mnt->mnt.mnt_flags = mnt_flags;
+		}
+		br_write_unlock(&vfsmount_lock);
+	} else if (!capable(CAP_SYS_ADMIN))
  		err = -EPERM;
-	else
+	else {
+		down_write(&sb->s_umount);
  		err = do_remount_sb(sb, flags, data, 0);
-	if (!err) {
-		br_write_lock(&vfsmount_lock);
-		mnt_flags |= mnt->mnt.mnt_flags & MNT_PROPAGATION_MASK;
-		mnt->mnt.mnt_flags = mnt_flags;
-		br_write_unlock(&vfsmount_lock);
+		if (!err) {
+			br_write_lock(&vfsmount_lock);
+			mnt_flags |= mnt->mnt.mnt_flags & MNT_PROPAGATION_MASK;
+			mnt->mnt.mnt_flags = mnt_flags;
+			br_write_unlock(&vfsmount_lock);
+		}
+		up_write(&sb->s_umount);
  	}
-	up_write(&sb->s_umount);
  	if (!err) {
  		br_write_lock(&vfsmount_lock);
  		touch_mnt_namespace(mnt->mnt_ns);
-- 
1.8.4

Re: [RFC] vfs: avoid sb->s_umount lock while changing bind-mount flags

[Al Viro]

On Mon, Sep 30, 2013 at 11:13:23AM -0700, Aditya Kali wrote:
> +Ted Ts'o, Tejun Heo, Jens Axboe
> 
> 
> On 09/30/2013 10:54 AM, Aditya Kali wrote:
> >Hi Al and other fs-developers,
> >
> >Please let me know what you think about this patch.

Don't top-post, please...  What prevents a race between MS_BIND remounts
and plain ones?  Used to be serialized on ->s_umount, but...

Re: [RFC] vfs: avoid sb->s_umount lock while changing bind-mount flags

[Aditya Kali]

On Mon, Sep 30, 2013 at 1:03 PM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> On Mon, Sep 30, 2013 at 11:13:23AM -0700, Aditya Kali wrote:
>> +Ted Ts'o, Tejun Heo, Jens Axboe
>>
>>
>> On 09/30/2013 10:54 AM, Aditya Kali wrote:
>> >Hi Al and other fs-developers,
>> >
>> >Please let me know what you think about this patch.
>
> Don't top-post, please...  What prevents a race between MS_BIND remounts
> and plain ones?  Used to be serialized on ->s_umount, but...

The rational here is that MS_BIND remounts need to act on the bind
mount-point only (struct vfsmount *) and not on the underlying
superblock (struct super_block *). So, intuitively, it should not be
necessary to take a superblock level lock for the MS_BIND case. As for
the modification of mnt_flags on vfsmount, there is already
vfsmount_lock taken in both MS_BIND and non-MS_BIND case to prevent
the race.

Following is the example that demonstrates the problem that I am
trying to address with this patch:

(1) /dev/sda is mounted at /mnt/sda
(2) /mnt/sda/users/user1 is bind mounted at /home/user1/ ;
/mnt/sda/users/user2 is bind mounted at /home/user2/
(3) user1 is doing buffered writes in /home/user1/logs/
(4) admin tries to make /home/user2/bin/ read-only using a bind-mounts:
    $ mount --bind /home/user2/bin/ /home/user2/bin/    # this is fast
    $ mount --bind -o remount,ro /home/user2/bin/    # this blocks
behind any writeback happening on sda (because of sb->s_umount write
lock)

Please advise.

Thanks,
-- 
Aditya