loop subsystem corrupted after mounting multiple btrfs sub-volumes

loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Stanislav Brabec]

While writing a test suite for util-linux[1], I experienced a a strange
behavior of loop device:

When two loop devices refer to the same file, and two btrfs mounts are
called on them, the second mount changes loop device of the first,
already mounted sub-volume. (Note that the current implementation of
util-linux mount -oloop works exactly in this way, and it allocates new
loop device for each mount command, so this bug can be easily
reproduced without losetup, just using "mount -oloop" or fstab.)

/proc/self/mountinfo after first btrfs loop mount:

107 59 0:59 /d0/dd0/ddd0/s1/d1/dd1/ddd1/s2 /mnt/1 rw,relatime shared:45 - btrfs /dev/loop0 rw,space_cache,subvolid=257,subvol=/d0/dd0/ddd0/s1/d1/dd1/ddd1/s2

This line changes after second first btrfs loop to:

07 59 0:59 /d0/dd0/ddd0/s1/d1/dd1/ddd1/s2 /mnt/1 rw,relatime shared:45 - btrfs /dev/loop1 rw,space_cache,subvolid=257,subvol=/d0/dd0/ddd0/s1/d1/dd1/ddd1/s2

See the change of /dev/loop0 to /dev/loop1!

It is apparently not only proc file change, but it also causes a
corruption of loop device subsystem, as I observed severe problems
on the affected system later:

- mount(2) returning 0 but doing nothing.

- mount(8) entering an infinite loop while searching for free loop
device.


Here is a main reproducer:

=====================
#!/bin/sh
# Prepare the environment:
/btrfs.sh
mkdir -p /mnt/1 /mnt/2
losetup /dev/loop0 /btrfs.img
# Verify that nothing is mounted:
cat /proc/self/mountinfo | grep /mnt
mount /dev/loop0 /mnt/1
echo "One file system should be mounted now."
cat /proc/self/mountinfo | grep /mnt
# Create another loop.
losetup /dev/loop1 /btrfs.img
echo "Going to mount second one."
mount -osubvol=/ /dev/loop1 /mnt/2 2>&1
echo "Two file system should be mounted now."
cat /proc/self/mountinfo | grep /mnt
echo "Strange. First mount changed its loop device!"
umount /mnt/2
echo "And now check, whether it remains changed after umount."
cat /proc/self/mountinfo | grep /mnt
umount /mnt/1
losetup -d /dev/loop1
losetup -d /dev/loop0
rmdir /mnt/1 /mnt/2
=====================

And here is its output:

One file system should be mounted now.
107 59 0:59 /d0/dd0/ddd0/s1/d1/dd1/ddd1/s2 /mnt/1 rw,relatime shared:45 - btrfs /dev/loop0 rw,space_cache,subvolid=257,subvol=/d0/dd0/ddd0/s1/d1/dd1/ddd1/s2
Going to mount second one.
Two file system should be mounted now.
107 59 0:59 /d0/dd0/ddd0/s1/d1/dd1/ddd1/s2 /mnt/1 rw,relatime shared:45 - btrfs /dev/loop1 rw,space_cache,subvolid=257,subvol=/d0/dd0/ddd0/s1/d1/dd1/ddd1/s2
108 59 0:59 / /mnt/2 rw,relatime shared:47 - btrfs /dev/loop1 rw,space_cache,subvolid=5,subvol=/
Strange. First mount changed its loop device!
And now check, whether it remains changed after umount.
107 59 0:59 /d0/dd0/ddd0/s1/d1/dd1/ddd1/s2 /mnt/1 rw,relatime shared:45 - btrfs /dev/loop1 rw,space_cache,subvolid=257,subvol=/d0/dd0/ddd0/s1/d1/dd1/ddd1/s2

It was actually reproduced on linux-4.4.1 on openSUSE Tumbleweed.


Test image creator:

===== /btrfs.sh =====
#!/bin/sh
truncate -s 42M /btrfs.img
mkfs.btrfs -f -d single -m single /btrfs.img >/dev/null
mount -o loop /btrfs.img /mnt
pushd . >/dev/null
cd /mnt
mkdir -p d0/dd0/ddd0
cd ./d0/dd0/ddd0
touch file{1..5}
btrfs subvol create s1 >/dev/null
cd ./s1
touch file{1..5}
mkdir bind-point
mkdir -p d1/dd1/ddd1
cd ./d1/dd1/ddd1
btrfs subvol create s2 >/dev/null
DEFAULT_SUBVOLID=$(btrfs inspect rootid s2)
btrfs subvol set-default $DEFAULT_SUBVOLID . >/dev/null
NON_DEFAULT_SUBVOLID=$(btrfs subvol list /mnt |
while read dummy id rest ; do if test $id = $DEFAULT_SUBVOLID ; then
continue ; fi ; echo $id ; done)
cd ../../../..
mkdir -p d2/dd2/ddd2
cd ./d2/dd2/ddd2
btrfs subvol create s3 >/dev/null
mkdir -p s3/bind-mnt
popd >/dev/null
NON_DEFAULT_SUBVOL=d0/dd0/ddd0/d2/dd2/ddd2/s3
umount /mnt
=====================

[1] http://marc.info/?l=util-linux-ng&m=145590643206663&w=2

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                         e-mail: sbrabec@suse.com
Lihovarská 1060/12                            tel: +49 911 7405384547
190 00 Praha 9                                 fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Austin S. Hemmelgarn]

Added linux-btrfs as this should be documented there as a known issue 
until it gets fixed (although I have no idea which side is the issue).
On 2016-02-25 14:22, Stanislav Brabec wrote:
> While writing a test suite for util-linux[1], I experienced a a strange
> behavior of loop device:
>
> When two loop devices refer to the same file, and two btrfs mounts are
> called on them, the second mount changes loop device of the first,
> already mounted sub-volume. (Note that the current implementation of
> util-linux mount -oloop works exactly in this way, and it allocates new
> loop device for each mount command, so this bug can be easily
> reproduced without losetup, just using "mount -oloop" or fstab.)
I'm not 100% certain, but I think this is a interaction between how 
BTRFS handles multiple mounts of the same filesystem on a given system 
and how mount handles loop mounts.  AFAIUI, all instances of a given 
BTRFS filesystem being mounted on a given system are internally 
identical to bind mounts of a hidden mount of that filesystem.  This is 
what allows both manual mounting of sub-volumes, and multiple mounting 
of the FS in general.
>
> /proc/self/mountinfo after first btrfs loop mount:
>
> 107 59 0:59 /d0/dd0/ddd0/s1/d1/dd1/ddd1/s2 /mnt/1 rw,relatime shared:45 - btrfs /dev/loop0 rw,space_cache,subvolid=257,subvol=/d0/dd0/ddd0/s1/d1/dd1/ddd1/s2
>
> This line changes after second first btrfs loop to:
>
> 07 59 0:59 /d0/dd0/ddd0/s1/d1/dd1/ddd1/s2 /mnt/1 rw,relatime shared:45 - btrfs /dev/loop1 rw,space_cache,subvolid=257,subvol=/d0/dd0/ddd0/s1/d1/dd1/ddd1/s2
>
> See the change of /dev/loop0 to /dev/loop1!
>
> It is apparently not only proc file change, but it also causes a
> corruption of loop device subsystem, as I observed severe problems
> on the affected system later:
>
> - mount(2) returning 0 but doing nothing.
>
> - mount(8) entering an infinite loop while searching for free loop
> device.
This seems odd that it would cause such a degree of inconsistency in the 
kernel itself.  My guess though is that mount(8) sees that you're trying 
to mount a file and unconditionally tries to bind it to a loop device 
without checking any in-use loop devices to see if it's already bound to 
them, and then when it calls mount(2), this ends up somehow confusing 
the BTRFS driver (probably because you've now mounted two filesystems 
with effectively identical super-blocks, BTRFS already has issues if 
multiple filesystems have the same UUID, and I have no idea how it might 
react to filesystems that appear identical but are on separate devices).
>
>
> Here is a main reproducer:
>
> =====================
> #!/bin/sh
> # Prepare the environment:
> /btrfs.sh
> mkdir -p /mnt/1 /mnt/2
> losetup /dev/loop0 /btrfs.img
> # Verify that nothing is mounted:
> cat /proc/self/mountinfo | grep /mnt
> mount /dev/loop0 /mnt/1
> echo "One file system should be mounted now."
> cat /proc/self/mountinfo | grep /mnt
> # Create another loop.
> losetup /dev/loop1 /btrfs.img
> echo "Going to mount second one."
> mount -osubvol=/ /dev/loop1 /mnt/2 2>&1
> echo "Two file system should be mounted now."
> cat /proc/self/mountinfo | grep /mnt
> echo "Strange. First mount changed its loop device!"
> umount /mnt/2
> echo "And now check, whether it remains changed after umount."
> cat /proc/self/mountinfo | grep /mnt
> umount /mnt/1
> losetup -d /dev/loop1
> losetup -d /dev/loop0
> rmdir /mnt/1 /mnt/2
> =====================
>
> And here is its output:
>
> One file system should be mounted now.
> 107 59 0:59 /d0/dd0/ddd0/s1/d1/dd1/ddd1/s2 /mnt/1 rw,relatime shared:45 - btrfs /dev/loop0 rw,space_cache,subvolid=257,subvol=/d0/dd0/ddd0/s1/d1/dd1/ddd1/s2
> Going to mount second one.
> Two file system should be mounted now.
> 107 59 0:59 /d0/dd0/ddd0/s1/d1/dd1/ddd1/s2 /mnt/1 rw,relatime shared:45 - btrfs /dev/loop1 rw,space_cache,subvolid=257,subvol=/d0/dd0/ddd0/s1/d1/dd1/ddd1/s2
> 108 59 0:59 / /mnt/2 rw,relatime shared:47 - btrfs /dev/loop1 rw,space_cache,subvolid=5,subvol=/
> Strange. First mount changed its loop device!
> And now check, whether it remains changed after umount.
> 107 59 0:59 /d0/dd0/ddd0/s1/d1/dd1/ddd1/s2 /mnt/1 rw,relatime shared:45 - btrfs /dev/loop1 rw,space_cache,subvolid=257,subvol=/d0/dd0/ddd0/s1/d1/dd1/ddd1/s2
>
> It was actually reproduced on linux-4.4.1 on openSUSE Tumbleweed.
>
>
> Test image creator:
>
> ===== /btrfs.sh =====
> #!/bin/sh
> truncate -s 42M /btrfs.img
> mkfs.btrfs -f -d single -m single /btrfs.img >/dev/null
> mount -o loop /btrfs.img /mnt
> pushd . >/dev/null
> cd /mnt
> mkdir -p d0/dd0/ddd0
> cd ./d0/dd0/ddd0
> touch file{1..5}
> btrfs subvol create s1 >/dev/null
> cd ./s1
> touch file{1..5}
> mkdir bind-point
> mkdir -p d1/dd1/ddd1
> cd ./d1/dd1/ddd1
> btrfs subvol create s2 >/dev/null
> DEFAULT_SUBVOLID=$(btrfs inspect rootid s2)
> btrfs subvol set-default $DEFAULT_SUBVOLID . >/dev/null
> NON_DEFAULT_SUBVOLID=$(btrfs subvol list /mnt |
> while read dummy id rest ; do if test $id = $DEFAULT_SUBVOLID ; then
> continue ; fi ; echo $id ; done)
> cd ../../../..
> mkdir -p d2/dd2/ddd2
> cd ./d2/dd2/ddd2
> btrfs subvol create s3 >/dev/null
> mkdir -p s3/bind-mnt
> popd >/dev/null
> NON_DEFAULT_SUBVOL=d0/dd0/ddd0/d2/dd2/ddd2/s3
> umount /mnt
> =====================
>
> [1] http://marc.info/?l=util-linux-ng&m=145590643206663&w=2
>

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Stanislav Brabec]

Austin S. Hemmelgarn wrote:
 > Added linux-btrfs as this should be documented there as a known issue
 > until it gets fixed (although I have no idea which side is the issue).

This is a very bad behavior, as it makes impossible to safely use btrfs
loop bind mounts in fstab. (Well, it is possible to write a work-around
in util-linux: Remember the source file, and if -oloop is specified
next time, and source file is already assigned to a loop device, use
existing loop device.)

> I'm not 100% certain, but I think this is a interaction between how
> BTRFS handles multiple mounts of the same filesystem on a given system
> and how mount handles loop mounts.  AFAIUI, all instances of a given
> BTRFS filesystem being mounted on a given system are internally
> identical to bind mounts of a hidden mount of that filesystem.  This is
> what allows both manual mounting of sub-volumes, and multiple mounting
> of the FS in general.

Yes, internal implementation is the same.

But here it causes a  real trouble: However both mounts point to the
same file, first and second mount use different loop device. To create
a bind mount, something ugly needs to be done. And it is done in an
incorrect way.


I already found another inconsistency caused by this implementation:

/proc/self/mountinfo reports subvolid of the nearest upper sub-volume
root for the bind mount, not the sub-volume that was used for creating
this bind mount, and subvolid that potentially does not correspond to
any subvolume root.

This could causes problem for evaluation of order of umount(2) that
should prevent EBUSY.


I was talking about it with David Sterba, and he told, that in the
current implementation is not optimal. btrfs driver does not have
sufficient information to evaluate true root of the bind mount.

Maybe the same is valid for the reported loop issue, and this is just
an ugly side effect.


P. S.: There are some use differences between bind mounts and btrfs
sub-volumes:

- Bind mounts can be created for any file or directory.
- Sub-volume mounts can be created only for inodes marked as sub-volume
   root.

- Bind mounts can be mounted only if any of upper sub-volume root is
   mounted.
- Sub-volumes can be mounted even if volume root is not mounted.

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                         e-mail: sbrabec@suse.com
Lihovarská 1060/12                            tel: +49 911 7405384547
190 00 Praha 9                                 fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Austin S. Hemmelgarn]

On 2016-02-26 10:50, Stanislav Brabec wrote:
> Austin S. Hemmelgarn wrote:
>  > Added linux-btrfs as this should be documented there as a known issue
>  > until it gets fixed (although I have no idea which side is the issue).
>
> This is a very bad behavior, as it makes impossible to safely use btrfs
> loop bind mounts in fstab. (Well, it is possible to write a work-around
> in util-linux: Remember the source file, and if -oloop is specified
> next time, and source file is already assigned to a loop device, use
> existing loop device.)
>
>> I'm not 100% certain, but I think this is a interaction between how
>> BTRFS handles multiple mounts of the same filesystem on a given system
>> and how mount handles loop mounts.  AFAIUI, all instances of a given
>> BTRFS filesystem being mounted on a given system are internally
>> identical to bind mounts of a hidden mount of that filesystem.  This is
>> what allows both manual mounting of sub-volumes, and multiple mounting
>> of the FS in general.
>
> Yes, internal implementation is the same.
>
> But here it causes a  real trouble: However both mounts point to the
> same file, first and second mount use different loop device. To create
> a bind mount, something ugly needs to be done. And it is done in an
> incorrect way.
That's just it though, from what I can tell based on what I've seen and 
what you said above, mount(8) isn't doing things correctly in this case. 
  If we were to do this with something like XFS or ext4, the filesystem 
would probably end up completely messed up just because of the log 
replay code (assuming they actually mount the second time, I'm not sure 
what XFS would do in this case, but I believe that ext4 would allow the 
mount as long as the mmp feature is off).  It would make sense that this 
behavior wouldn't have been noticed before (and probably wouldn't have 
mattered even if it had been), because most filesystems don't allow 
multiple mounts even if they're all RO, and most people don't try to 
mount other filesystems multiple times as a result of this.  If this 
behavior of allocating a new loop device for each call on a given file 
is in fact not BTRFS specific (as implied by your statement about a 
possible workaround in mount(8)), then mount(8) really should be fixed 
to not do that before we even consider looking at the issues in BTRFS, 
as that is behavior that has serious potential to result in data 
corruption for any filesystem, not just BTRFS.

Now, if this does get fixed, mount(8) doesn't necessarily need to 
maintain it's own copy of the state of /dev/loop mappings, it could 
simply check the currently allocated loop devices.  You would of course 
need some form of locking relative to other mount -o loop instances and 
losetup, and it would be slow, but if you're using enough loop devices 
that this causes noticeable delays, then you really shouldn't be 
complaining all that much about performance.
>
>
> I already found another inconsistency caused by this implementation:
>
> /proc/self/mountinfo reports subvolid of the nearest upper sub-volume
> root for the bind mount, not the sub-volume that was used for creating
> this bind mount, and subvolid that potentially does not correspond to
> any subvolume root.
>
> This could causes problem for evaluation of order of umount(2) that
> should prevent EBUSY.
>
> I was talking about it with David Sterba, and he told, that in the
> current implementation is not optimal. btrfs driver does not have
> sufficient information to evaluate true root of the bind mount.
I've noticed this before myself, but I've never seen any issues 
resulting from it; however, I've also not tried calling BTRFS related 
ioctls on or from such a mount, so I may just have been lucky.
>
> Maybe the same is valid for the reported loop issue, and this is just
> an ugly side effect.
I'd be more than willing to bet that that isn't the case, loop mounts 
and bind mounts are entirely different inside the kernel, and I think 
the loop mount issue on the BTRFS side is a result of the issues it has 
when dealing with filesystems with the same UUID (if this is in fact the 
case, similar behavior should be seen when trying to either mount 
multiple lower level components of a multi-path device, or by manually 
creating multiple /dev/loop associations for the same file and mounting 
them all at once using the /dev/loop names instead of the file).
>
>
> P. S.: There are some use differences between bind mounts and btrfs
> sub-volumes:
>
> - Bind mounts can be created for any file or directory.
> - Sub-volume mounts can be created only for inodes marked as sub-volume
>    root.
>
> - Bind mounts can be mounted only if any of upper sub-volume root is
>    mounted.
> - Sub-volumes can be mounted even if volume root is not mounted.
FWIW, it's actually possible to simulate this behavior with bind mounts 
by mounting the root at the eventual mount point, then bind mounting the 
desired directory from that root over top of it.  Of course, there is 
almost zero practical purpose to anyone doing this on most traditional 
filesystems unless they're actively trying to hide data.

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Stanislav Brabec]

Austin S. Hemmelgarn wrote:
 > On 2016-02-26 10:50, Stanislav Brabec wrote:
> That's just it though, from what I can tell based on what I've seen and
> what you said above, mount(8) isn't doing things correctly in this case.
>   If we were to do this with something like XFS or ext4, the filesystem
> would probably end up completely messed up just because of the log
> replay code (assuming they actually mount the second time, I'm not sure
> what XFS would do in this case, but I believe that ext4 would allow the
> mount as long as the mmp feature is off).  It would make sense that this
> behavior wouldn't have been noticed before (and probably wouldn't have
> mattered even if it had been), because most filesystems don't allow
> multiple mounts even if they're all RO, and most people don't try to
> mount other filesystems multiple times as a result of this.  If this
> behavior of allocating a new loop device for each call on a given file
> is in fact not BTRFS specific (as implied by your statement about a
> possible workaround in mount(8)), then mount(8) really should be fixed
> to not do that before we even consider looking at the issues in BTRFS,
> as that is behavior that has serious potential to result in data
> corruption for any filesystem, not just BTRFS.

Well, kernel could "fix" it in a simple way:

- don't allow two loop devices pointing to the same file
or
- don't allow two loop devices pointing to the same file being used by
   mount(2).

Then util-linux would need a behavior change for sure.

>> I already found another inconsistency caused by this implementation:
>>
>> /proc/self/mountinfo reports subvolid of the nearest upper sub-volume
>> root for the bind mount, not the sub-volume that was used for creating
>> this bind mount, and subvolid that potentially does not correspond to
>> any subvolume root.
>>
>> This could causes problem for evaluation of order of umount(2) that
>> should prevent EBUSY.
>>
>> I was talking about it with David Sterba, and he told, that in the
>> current implementation is not optimal. btrfs driver does not have
>> sufficient information to evaluate true root of the bind mount.
> I've noticed this before myself, but I've never seen any issues
> resulting from it; however, I've also not tried calling BTRFS related
> ioctls on or from such a mount, so I may just have been lucky.

I can imagine two side effects deeply inside mount(8):

- "mount -a" uses subvol internally for a path lookup of the default
   volume or volume corresponding to subvolid. (Only the GIT version,
   not yet in 2.27.1.) I could imagine that the lookup is confused by a
   bind mount reporting the searched subvolid and a "random" subvol
   subvol.  But I don't have a reproducer yet, and I am not sure,
   whether it is really possible.

- "umount -a" could have a problem to find a proper order to umount(2)
   without EBUSY. I did not check the algorithm, so I am not sure,
   whether it is a real issue.


P. S.: There were many problems with btrfs in mount(8):

https://git.kernel.org/cgit/utils/util-linux/util-linux.git/commit/?id=c4af75a84ef3430003c77be2469869aaf3a63e2a
https://git.kernel.org/cgit/utils/util-linux/util-linux.git/commit/?id=618a88140e26a134727a39c906c9cdf6d0c04513
https://git.kernel.org/cgit/utils/util-linux/util-linux.git/commit/?id=d2f8267847ecbe763a3b63af1289bf1179cd8c45
https://git.kernel.org/cgit/utils/util-linux/util-linux.git/commit/?id=2cd28fc82d0c947472a4700d5e764265916fba1e
https://git.kernel.org/cgit/utils/util-linux/util-linux.git/commit/?id=352740e88e2c9cb180fe845ce210b1c7b5ad88c7

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                         e-mail: sbrabec@suse.com
Lihovarská 1060/12                            tel: +49 911 7405384547
190 00 Praha 9                                 fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Al Viro]

On Fri, Feb 26, 2016 at 11:39:11AM -0500, Austin S. Hemmelgarn wrote:

> That's just it though, from what I can tell based on what I've seen
> and what you said above, mount(8) isn't doing things correctly in
> this case.  If we were to do this with something like XFS or ext4,
> the filesystem would probably end up completely messed up just
> because of the log replay code (assuming they actually mount the
> second time, I'm not sure what XFS would do in this case, but I
> believe that ext4 would allow the mount as long as the mmp feature
> is off).  It would make sense that this behavior wouldn't have been
> noticed before (and probably wouldn't have mattered even if it had
> been), because most filesystems don't allow multiple mounts even if
> they're all RO, and most people don't try to mount other filesystems
> multiple times as a result of this.

They most certainly do.  The problem is mount(8) treatment of -o loop -
you can mount e.g. ext4 many times, it'll just get you extra references
to the same struct super_block from those new vfsmounts.  IOW, that'll
behave the same way as if you were doing mount --bind on subsequent ones.

And as far as kernel is concerned, /dev/loop* isn't special in any respects;
if you do explicit losetup and mount the resulting /dev/loop<n> as many
times as you wish, it'll work just fine.  And from the kernel POV it's not
different from what it sees with -o loop; setting the loop device up is
done first by separate syscall, then mount(2) for that device is issued.

It's mount(8) that screws up here.

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Austin S. Hemmelgarn]

On 2016-02-26 12:07, Stanislav Brabec wrote:
> Austin S. Hemmelgarn wrote:
>  > On 2016-02-26 10:50, Stanislav Brabec wrote:
>> That's just it though, from what I can tell based on what I've seen and
>> what you said above, mount(8) isn't doing things correctly in this case.
>>   If we were to do this with something like XFS or ext4, the filesystem
>> would probably end up completely messed up just because of the log
>> replay code (assuming they actually mount the second time, I'm not sure
>> what XFS would do in this case, but I believe that ext4 would allow the
>> mount as long as the mmp feature is off).  It would make sense that this
>> behavior wouldn't have been noticed before (and probably wouldn't have
>> mattered even if it had been), because most filesystems don't allow
>> multiple mounts even if they're all RO, and most people don't try to
>> mount other filesystems multiple times as a result of this.  If this
>> behavior of allocating a new loop device for each call on a given file
>> is in fact not BTRFS specific (as implied by your statement about a
>> possible workaround in mount(8)), then mount(8) really should be fixed
>> to not do that before we even consider looking at the issues in BTRFS,
>> as that is behavior that has serious potential to result in data
>> corruption for any filesystem, not just BTRFS.
>
> Well, kernel could "fix" it in a simple way:
>
> - don't allow two loop devices pointing to the same file
> or
> - don't allow two loop devices pointing to the same file being used by
>    mount(2).
This has legitimate usage in testing multipath configuration and 
operation, and in testing that filesystems handle this correctly.  On 
top of that, it becomes decidedly non-trivial to handle when you 
consider that loop devices can map a fixed range of a file independent 
of the rest of the file (this used to be the way to pull partitions out 
of raw disk images before the device mapper became as commonplace as it 
is now).
>
> Then util-linux would need a behavior change for sure.
>
>>> I already found another inconsistency caused by this implementation:
>>>
>>> /proc/self/mountinfo reports subvolid of the nearest upper sub-volume
>>> root for the bind mount, not the sub-volume that was used for creating
>>> this bind mount, and subvolid that potentially does not correspond to
>>> any subvolume root.
>>>
>>> This could causes problem for evaluation of order of umount(2) that
>>> should prevent EBUSY.
>>>
>>> I was talking about it with David Sterba, and he told, that in the
>>> current implementation is not optimal. btrfs driver does not have
>>> sufficient information to evaluate true root of the bind mount.
>> I've noticed this before myself, but I've never seen any issues
>> resulting from it; however, I've also not tried calling BTRFS related
>> ioctls on or from such a mount, so I may just have been lucky.
>
> I can imagine two side effects deeply inside mount(8):
>
> - "mount -a" uses subvol internally for a path lookup of the default
>    volume or volume corresponding to subvolid. (Only the GIT version,
>    not yet in 2.27.1.) I could imagine that the lookup is confused by a
>    bind mount reporting the searched subvolid and a "random" subvol
>    subvol.  But I don't have a reproducer yet, and I am not sure,
>    whether it is really possible.
>
> - "umount -a" could have a problem to find a proper order to umount(2)
>    without EBUSY. I did not check the algorithm, so I am not sure,
>    whether it is a real issue.
If BTRFS can't get the correct ref on the FS root internally, then there 
are all kinds of things that could go wrong when you try to do any of 
the typical maintenance stuff on it (like balancing, scrub, defrag, 
snapshot/subvolume creation/deletion, etc).  In essence, if you try to 
do almost anything using the btrfs command line tools on that mount 
point, it might fail in new and interesting ways.
>
>
> P. S.: There were many problems with btrfs in mount(8):
>
> https://git.kernel.org/cgit/utils/util-linux/util-linux.git/commit/?id=c4af75a84ef3430003c77be2469869aaf3a63e2a
> https://git.kernel.org/cgit/utils/util-linux/util-linux.git/commit/?id=618a88140e26a134727a39c906c9cdf6d0c04513
> https://git.kernel.org/cgit/utils/util-linux/util-linux.git/commit/?id=d2f8267847ecbe763a3b63af1289bf1179cd8c45
> https://git.kernel.org/cgit/utils/util-linux/util-linux.git/commit/?id=2cd28fc82d0c947472a4700d5e764265916fba1e
> https://git.kernel.org/cgit/utils/util-linux/util-linux.git/commit/?id=352740e88e2c9cb180fe845ce210b1c7b5ad88c7
>
The first commit is just test cases, and the others are specific issues 
that only affected BTRFS which have nothing to do with this thread at 
all other than involving mount(8) and BTRFS.  The originally stated 
issue that this thread is about is specific to loop mounting a BTRFS 
filesystem stored in a file multiple times.  The issue can be 
empirically demonstrated to be a result of an interaction between BTRFS 
behavior regarding duplicate filesystems and an implementation detail of 
mount(8).  The BTRFS behavior WRT duplicate FS UUID's is not going away 
any time soon (believe me, it's been discussed _a lot_ on the mailing 
list in the context of almost everything except loop devices, and the 
developers have pretty much stated that there is no sane way to handle 
it), and the mount(8) behavior has the potential to cause either data 
corruption or similar behavior in the future (I would expect that XFS 
with metadata checksumming enabled would cause a similar interaction, 
although they probably would handle it better).

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Stanislav Brabec]

Al Viro wrote:
> On Fri, Feb 26, 2016 at 11:39:11AM -0500, Austin S. Hemmelgarn wrote:
> 
>> That's just it though, from what I can tell based on what I've seen
>> and what you said above, mount(8) isn't doing things correctly in
>> this case.  If we were to do this with something like XFS or ext4,
>> the filesystem would probably end up completely messed up just
>> because of the log replay code (assuming they actually mount the
>> second time, I'm not sure what XFS would do in this case, but I
>> believe that ext4 would allow the mount as long as the mmp feature
>> is off).  It would make sense that this behavior wouldn't have been
>> noticed before (and probably wouldn't have mattered even if it had
>> been), because most filesystems don't allow multiple mounts even if
>> they're all RO, and most people don't try to mount other filesystems
>> multiple times as a result of this.

Well, in such case kernel should return an error when mount(8) is
trying to use multiple mount devices for a single file for mount(2).

But kernel does not return error, it starts to do strange things.

> They most certainly do.  The problem is mount(8) treatment of -o loop -
> you can mount e.g. ext4 many times, it'll just get you extra references
> to the same struct super_block from those new vfsmounts.  IOW, that'll
> behave the same way as if you were doing mount --bind on subsequent ones.

I just tested the same with ext4. The rewriting of mountinfo happens
only with btrfs.

But after that mount(2) stops to work. See the last mount(2). It
returns 0, but nothing is mounted! Kernel mount(2) refuses to work!

# mount -oloop /ext4.img /mnt/1
# cat /proc/self/mountinfo | grep /mnt
238 59 7:0 / /mnt/1 rw,relatime shared:153 - ext4 /dev/loop0 rw,data=ordered
# mount -oloop /ext4.img /mnt/2
# cat /proc/self/mountinfo | grep /mnt
238 59 7:0 / /mnt/1 rw,relatime shared:153 - ext4 /dev/loop0 rw,data=ordered
243 59 7:1 / /mnt/2 rw,relatime shared:156 - ext4 /dev/loop1 rw,data=ordered
# umount /mnt/*
# mount -oloop /btrfs.img /mnt/1
# cat /proc/self/mountinfo | grep /mnt
238 59 0:94 /d0/dd0/ddd0/s1/d1/dd1/ddd1/s2 /mnt/1 rw,relatime shared:153 - btrfs /dev/loop0 rw,space_cache,subvolid=257,subvol=/d0/dd0/ddd0/s1/d1/dd1/ddd1/s2
# mount -oloop,subvol=/ /btrfs.img /mnt/2
# cat /proc/self/mountinfo | grep /mnt
238 59 0:94 /d0/dd0/ddd0/s1/d1/dd1/ddd1/s2 /mnt/1 rw,relatime shared:153 - btrfs /dev/loop1 rw,space_cache,subvolid=257,subvol=/d0/dd0/ddd0/s1/d1/dd1/ddd1/s2

I is really strange! Mount was called, but nothing appeared in the
mountinfo. Just a rewritten /dev/loop0 -> /dev/loop1 in the existing
mount.

To be sure, that it is mount(2) issue and not mount(8), let's try it
again with strace.

# strace mount -oloop,subvol=/ /btrfs.img /mnt/2 2>&1 | tail -n 7
mount("/dev/loop1", "/mnt/2", "btrfs", MS_MGC_VAL, "subvol=/") = 0
access("/mnt/2", W_OK)                  = 0
close(4)                                = 0
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++
# cat /proc/self/mountinfo | grep /mnt
238 59 0:94 /d0/dd0/ddd0/s1/d1/dd1/ddd1/s2 /mnt/1 rw,relatime shared:153 - btrfs /dev/loop1 rw,space_cache,subvolid=257,subvol=/d0/dd0/ddd0/s1/d1/dd1/ddd1/s2

Where is /mnt/2?

> And as far as kernel is concerned, /dev/loop* isn't special in any respects;
> if you do explicit losetup and mount the resulting /dev/loop<n> as many
> times as you wish, it'll work just fine.

mount(8) just calls losetup internally for every -o loop. Once per
"loop" option. Nobody probably tried to loop mount the same ext4 volume
more times, so no problems appeared.

But for btrfs, one would. And mounting two btrfs subvolumes with two
"-oloop" calls losetup twice for the same file.

> And from the kernel POV it's not
> different from what it sees with -o loop; setting the loop device up is
> done first by separate syscall, then mount(2) for that device is issued.

Yes, it is different.
- You have one file.
- You have two loop devices pointing to the same file.
- btrfs subvolumes are internally handled similarly like bind mounts.
  It means, that all subvolumes should have the same mount source. But
  these two mounts don't have.

> It's mount(8) that screws up here.

Yes mount(8) screws mount(2). And it corrupts kernel:

1) /proc/self/mountinfo changes its contents.

2) mount(2) called after the reproducer returns OK but does nothing.

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                         e-mail: sbrabec@suse.com
Lihovarská 1060/12                            tel: +49 911 7405384547
190 00 Praha 9                                 fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Stanislav Brabec]

On Feb 26, 2016 at 19:22 Austin S. Hemmelgarn wrote:
> The first commit is just test cases, and the others are specific issues
> that only affected BTRFS which have nothing to do with this thread at
> all other than involving mount(8) and BTRFS.

Yes, it is a bit off topic. It just demonstrates, how complex mount(8) 
for btrfs is.

The test case is all-fail in util-linux 2.27.1.

> mount(8) behavior has the potential to cause either data
> corruption or similar behavior in the future (I would expect that XFS
> with metadata checksumming enabled would cause a similar interaction,
> although they probably would handle it better).

Especially "mount -a" has a hard time to recognize what was already 
mounted and what still needs to be mounted.

The only information mount(8) has is the one from mountinfo. 
Interpreting of the mountinfo contents to reconstruct possible mount 
options used is a task far from being trivial.

Some of them are even impossible to discriminate:

Suppose you have:
mount -osubvol=/ /dev/sda2 /mnt/1
mount -osubvol=/sbv /dev/sda2 /mnt/2

Case 1:
mount -obind /mnt/1/sbv/bnd /mnt/3

Case 2:
mount -obind /mnt/2/bnd /mnt/3

Case 1 and case 2 have exactly the same mountinfo, but different 
reference counts for /mnt/1 and /mnt/2.

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                         e-mail: sbrabec@suse.com
Lihovarská 1060/12                            tel: +49 911 7405384547
190 00 Praha 9                                 fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Austin S. Hemmelgarn]

On 2016-02-26 14:12, Stanislav Brabec wrote:
> Al Viro wrote:
>> On Fri, Feb 26, 2016 at 11:39:11AM -0500, Austin S. Hemmelgarn wrote:
>>
>>> That's just it though, from what I can tell based on what I've seen
>>> and what you said above, mount(8) isn't doing things correctly in
>>> this case.  If we were to do this with something like XFS or ext4,
>>> the filesystem would probably end up completely messed up just
>>> because of the log replay code (assuming they actually mount the
>>> second time, I'm not sure what XFS would do in this case, but I
>>> believe that ext4 would allow the mount as long as the mmp feature
>>> is off).  It would make sense that this behavior wouldn't have been
>>> noticed before (and probably wouldn't have mattered even if it had
>>> been), because most filesystems don't allow multiple mounts even if
>>> they're all RO, and most people don't try to mount other filesystems
>>> multiple times as a result of this.
>
> Well, in such case kernel should return an error when mount(8) is
> trying to use multiple mount devices for a single file for mount(2).
As I said in my other e-mail, there are perfectly legitimate reasons to 
be doing this.  And I should also point out that anybody who has one of 
those reasons for doing this should be setting up the loop devices 
themselves, so mount(8) behaving this way is still wrong.
>
> But kernel does not return error, it starts to do strange things.
>
>> They most certainly do.  The problem is mount(8) treatment of -o loop -
>> you can mount e.g. ext4 many times, it'll just get you extra references
>> to the same struct super_block from those new vfsmounts.  IOW, that'll
>> behave the same way as if you were doing mount --bind on subsequent ones.
>
> I just tested the same with ext4. The rewriting of mountinfo happens
> only with btrfs.
>
> But after that mount(2) stops to work. See the last mount(2). It
> returns 0, but nothing is mounted! Kernel mount(2) refuses to work!
>
> # mount -oloop /ext4.img /mnt/1
> # cat /proc/self/mountinfo | grep /mnt
> 238 59 7:0 / /mnt/1 rw,relatime shared:153 - ext4 /dev/loop0 rw,data=ordered
> # mount -oloop /ext4.img /mnt/2
> # cat /proc/self/mountinfo | grep /mnt
> 238 59 7:0 / /mnt/1 rw,relatime shared:153 - ext4 /dev/loop0 rw,data=ordered
> 243 59 7:1 / /mnt/2 rw,relatime shared:156 - ext4 /dev/loop1 rw,data=ordered
> # umount /mnt/*
> # mount -oloop /btrfs.img /mnt/1
> # cat /proc/self/mountinfo | grep /mnt
> 238 59 0:94 /d0/dd0/ddd0/s1/d1/dd1/ddd1/s2 /mnt/1 rw,relatime shared:153 - btrfs /dev/loop0 rw,space_cache,subvolid=257,subvol=/d0/dd0/ddd0/s1/d1/dd1/ddd1/s2
> # mount -oloop,subvol=/ /btrfs.img /mnt/2
> # cat /proc/self/mountinfo | grep /mnt
> 238 59 0:94 /d0/dd0/ddd0/s1/d1/dd1/ddd1/s2 /mnt/1 rw,relatime shared:153 - btrfs /dev/loop1 rw,space_cache,subvolid=257,subvol=/d0/dd0/ddd0/s1/d1/dd1/ddd1/s2
>
> I is really strange! Mount was called, but nothing appeared in the
> mountinfo. Just a rewritten /dev/loop0 -> /dev/loop1 in the existing
> mount.
>
> To be sure, that it is mount(2) issue and not mount(8), let's try it
> again with strace.
>
> # strace mount -oloop,subvol=/ /btrfs.img /mnt/2 2>&1 | tail -n 7
> mount("/dev/loop1", "/mnt/2", "btrfs", MS_MGC_VAL, "subvol=/") = 0
> access("/mnt/2", W_OK)                  = 0
> close(4)                                = 0
> close(1)                                = 0
> close(2)                                = 0
> exit_group(0)                           = ?
> +++ exited with 0 +++
> # cat /proc/self/mountinfo | grep /mnt
> 238 59 0:94 /d0/dd0/ddd0/s1/d1/dd1/ddd1/s2 /mnt/1 rw,relatime shared:153 - btrfs /dev/loop1 rw,space_cache,subvolid=257,subvol=/d0/dd0/ddd0/s1/d1/dd1/ddd1/s2
>
> Where is /mnt/2?
It's kind of interesting, but I can't reproduce _any_ of this behavior 
with either ext4 or BTRFS when I manually set up the loop devices and 
point mount(8) at those instead of using -o loop on a file. That really 
seems to indicate that this is caused by something mount(8) is doing 
when it's calling losetup. I'm running a mostly unmodified version of 
4.4.2 (the only modification that would come even remotely close to this 
is that I changed the default mount options for everything from relatime 
to noatime), and util-linux 2.27.1 from Gentoo.
>
>> And as far as kernel is concerned, /dev/loop* isn't special in any respects;
>> if you do explicit losetup and mount the resulting /dev/loop<n> as many
>> times as you wish, it'll work just fine.
>
> mount(8) just calls losetup internally for every -o loop. Once per
> "loop" option. Nobody probably tried to loop mount the same ext4 volume
> more times, so no problems appeared.
>
> But for btrfs, one would. And mounting two btrfs subvolumes with two
> "-oloop" calls losetup twice for the same file.
>
>> And from the kernel POV it's not
>> different from what it sees with -o loop; setting the loop device up is
>> done first by separate syscall, then mount(2) for that device is issued.
>
> Yes, it is different.
> - You have one file.
> - You have two loop devices pointing to the same file.
> - btrfs subvolumes are internally handled similarly like bind mounts.
>    It means, that all subvolumes should have the same mount source. But
>    these two mounts don't have.
There is insufficient information given just the context of the syscall 
to differentiate this particular case in kernel code.
>
>> It's mount(8) that screws up here.
>
> Yes mount(8) screws mount(2). And it corrupts kernel:
>
> 1) /proc/self/mountinfo changes its contents.
>
> 2) mount(2) called after the reproducer returns OK but does nothing.
>
OK, we've determined that mount(2) is misbehaving.  That doesn't change 
the fact that mount(8) is triggering this, and therefore should itself 
be corrected.  Assume that mount(2) gets fixed so it doesn't lose it's 
mind and /proc/self/mountinfo doesn't change.  There will still be 
issues resulting from mount(8)'s behavior:
1. BTRFS will lose it's mind and corrupt data when using a multi-device 
filesystem (due to the problems with duplicate FS UUID's).
2. XFS might have similar issues to 1 when using metadata checksumming, 
although it's more likely that it won't allow the second mount to succeed.
3. Most other filesystems will likely end up corrupting data.

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Al Viro]

On Fri, Feb 26, 2016 at 03:05:27PM -0500, Austin S. Hemmelgarn wrote:
> >Where is /mnt/2?
> It's kind of interesting, but I can't reproduce _any_ of this
> behavior with either ext4 or BTRFS when I manually set up the loop
> devices and point mount(8) at those instead of using -o loop on a
> file. That really seems to indicate that this is caused by something
> mount(8) is doing when it's calling losetup. I'm running a mostly
> unmodified version of 4.4.2 (the only modification that would come
> even remotely close to this is that I changed the default mount
> options for everything from relatime to noatime), and util-linux
> 2.27.1 from Gentoo.

Sigh...  sys_mount() (mount_bdev(), actually) has no way to tell if two
loop devices refer to the same underlying object.  As far as it's
concerned, you are asking to mount a completely unrelated block device.
Which just happens to see the data (living in separate pagecache, even)
modified behind its back (with some delay) after it gets written to another
device.  Filesystem drivers generally don't like when something is screwing
the underlying data, to put it mildly...

When you ask to mount the _same_ device, mount_bdev(), as well as btrfs
counterpart, makes sure that you get a reference to the same struct
super_block, which avoids all coherency problems - all mounted instances
refer to the same in-core objects (dentries, inodes, page cache, etc.).
They get separate struct vfsmount instances, but that only matters for
mountpoint crossing.

As soon as you've set the second /dev/loop alias for the same underlying
file, you are asking for all kinds of trouble.  If you use the same one
consistently, you are OK.  BTW, even
losetup /dev/loop0 /dev/sda1
mount -t ext2 /dev/sda1 /mnt/1
mount -t ext2 /dev/loop0 /mnt/2
is enough for trouble - you get (as far as ext2 knows) unrelated devices
screwing each other, with no good way to predict that.  And you need to
check propagation through more than one layer - loop over loop over block
is also possible.

IMO on-demand losetup a-la -o loop is simply a bad idea...

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Austin S. Hemmelgarn]

On 2016-02-26 15:30, Al Viro wrote:
> On Fri, Feb 26, 2016 at 03:05:27PM -0500, Austin S. Hemmelgarn wrote:
>>> Where is /mnt/2?
>> It's kind of interesting, but I can't reproduce _any_ of this
>> behavior with either ext4 or BTRFS when I manually set up the loop
>> devices and point mount(8) at those instead of using -o loop on a
>> file. That really seems to indicate that this is caused by something
>> mount(8) is doing when it's calling losetup. I'm running a mostly
>> unmodified version of 4.4.2 (the only modification that would come
>> even remotely close to this is that I changed the default mount
>> options for everything from relatime to noatime), and util-linux
>> 2.27.1 from Gentoo.
>
> Sigh...  sys_mount() (mount_bdev(), actually) has no way to tell if two
> loop devices refer to the same underlying object.  As far as it's
> concerned, you are asking to mount a completely unrelated block device.
> Which just happens to see the data (living in separate pagecache, even)
> modified behind its back (with some delay) after it gets written to another
> device.  Filesystem drivers generally don't like when something is screwing
> the underlying data, to put it mildly...
>
> When you ask to mount the _same_ device, mount_bdev(), as well as btrfs
> counterpart, makes sure that you get a reference to the same struct
> super_block, which avoids all coherency problems - all mounted instances
> refer to the same in-core objects (dentries, inodes, page cache, etc.).
> They get separate struct vfsmount instances, but that only matters for
> mountpoint crossing.
>
> As soon as you've set the second /dev/loop alias for the same underlying
> file, you are asking for all kinds of trouble.  If you use the same one
> consistently, you are OK.  BTW, even
> losetup /dev/loop0 /dev/sda1
> mount -t ext2 /dev/sda1 /mnt/1
> mount -t ext2 /dev/loop0 /mnt/2
> is enough for trouble - you get (as far as ext2 knows) unrelated devices
> screwing each other, with no good way to predict that.  And you need to
> check propagation through more than one layer - loop over loop over block
> is also possible.
>
> IMO on-demand losetup a-la -o loop is simply a bad idea...
>
I agree wholeheartedly and wasn't disputing any of this, I meant I'm not 
seeing any of the odd mount(2) or /proc/self/mountinfo behavior that 
Stanislav started the thread about.  It was entirely trivial to get the 
filesystem images I used into a state where they couldn't be mounted 
again afterwards.

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Stanislav Brabec]

On Feb 26, 2016 at 21:05 Austin S. Hemmelgarn wrote:

> It's kind of interesting, but I can't reproduce _any_ of this behavior
> with either ext4 or BTRFS when I manually set up the loop devices and
> point mount(8) at those instead of using -o loop on a file. That really
> seems to indicate that this is caused by something mount(8) is doing
> when it's calling losetup.

Behavior of "-oloop" is more similar to "losetup -f /fs.img"? than to
"losetup /dev/loop0 /fs.img".

Anyway, I can reproduce without -oloop:
# losetup /dev/loop0 /btrfs.img
# mount /dev/loop0 /mnt/1
# grep /mnt /proc/self/mountinfo
107 59 0:59 /d0/dd0/ddd0/s1/d1/dd1/ddd1/s2 /mnt/1 rw,relatime shared:45 - btrfs /dev/loop0 rw,space_cache,subvolid=257,subvol=/d0/dd0/ddd0/s1/d1/dd1/ddd1/s2
# losetup /dev/loop1 /btrfs.img
# mount -osubvol=/ /dev/loop1 /mnt/2
# grep /mnt /proc/self/mountinfo
107 59 0:59 /d0/dd0/ddd0/s1/d1/dd1/ddd1/s2 /mnt/1 rw,relatime shared:45 - btrfs /dev/loop1 rw,space_cache,subvolid=257,subvol=/d0/dd0/ddd0/s1/d1/dd1/ddd1/s2
108 59 0:59 / /mnt/2 rw,relatime shared:48 - btrfs /dev/loop1 rw,space_cache,subvolid=5,subvol=/
# uname -a
Linux oct 4.4.1-1-default #1 SMP PREEMPT Mon Feb 15 11:03:27 UTC 2016 (6398c2d) x86_64 x86_64 x86_64 GNU/Linux

(Note that the system was freshly rebooted. After other experiments,
the second line of mountinfo can be missing completely.)

>> 2) mount(2) called after the reproducer returns OK but does nothing.
>>
> OK, we've determined that mount(2) is misbehaving.  That doesn't change
> the fact that mount(8) is triggering this, and therefore should itself
> be corrected.

> Assume that mount(2) gets fixed so it doesn't lose it's
> mind and /proc/self/mountinfo doesn't change.  There will still be
> issues resulting from mount(8)'s behavior:
> 1. BTRFS will lose it's mind and corrupt data when using a multi-device
> filesystem (due to the problems with duplicate FS UUID's).
> 2. XFS might have similar issues to 1 when using metadata checksumming,
> although it's more likely that it won't allow the second mount to succeed.
> 3. Most other filesystems will likely end up corrupting data.

Do I understand, that you are saying:

Yes, mounting multiple loop devices associated with one file is a
legitimate use, but mount(8) should never do it, because it has other
ugly side effects?

OK, it looks like a next task for mount(8) to fix.

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                         e-mail: sbrabec@suse.com
Lihovarská 1060/12                            tel: +49 911 7405384547
190 00 Praha 9                                 fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Stanislav Brabec]

On Feb 26, 2016 at 21:30 Al Viro wrote:
> IMO on-demand losetup a-la -o loop is simply a bad idea...

So the correct behavior of -o loop should:

Check, whether another mount command already did losetup.

If not, allocate new loop device.

If yes, reuse existing loop device.

Well, it seems to be safe, even if the loop device was not allocated by 
mount(8) itself, as
ioctl(fd, LOOP_CLR_FD)
never returns EBUSY:

# losetup /dev/loop2 /ext4.img
# mount /dev/loop2 /mnt
# strace losetup -d /dev/loop2 2>&1 | tail -n7 | head -n3
open("/dev/loop2", O_RDONLY|O_CLOEXEC)  = 3
ioctl(3, LOOP_CLR_FD)                   = 0
close(3)                                = 0

If the recycling "alien" loop devices will not be considered as a good 
idea, then (if possible):

If the loop device was allocated by mount(8) itself, recycle it.

If the loop device was not allocated by mount(8) itself, return error.

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                         e-mail: sbrabec@suse.com
Lihovarská 1060/12                            tel: +49 911 7405384547
190 00 Praha 9                                 fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Al Viro]

On Fri, Feb 26, 2016 at 09:37:22PM +0100, Stanislav Brabec wrote:

> Do I understand, that you are saying:
> 
> Yes, mounting multiple loop devices associated with one file is a
> legitimate use, but mount(8) should never do it, because it has other
> ugly side effects?

It's on the same level as "hey, let's have an nbd daemon run in qemu
guest, exporting a host file over nbd, import it to host as /dev/nbd69,
set a loopback device over the underlying file as /dev/loop42 and
ask e.g. xfs to recognize that it's dealing with the same underlying array
of bytes in both cases - wouldn't it be neat if it could do that?"

There's no magic.  Really.  Unexpected sharing of backing store between
apparently unrelated devices can cause trouble.  And I'm not sure how
to deal with -o loop in a sane way, TBH - automagical losetup is bloody
hard to get right.  Keep in mind that loop-over-loop is also possible...

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Stanislav Brabec]

On Feb 26, 2016 at 22:03 Al Viro wrote:
And I'm not sure how
> to deal with -o loop in a sane way, TBH - automagical losetup is bloody
> hard to get right.

See another reply in this thread for the idea:
Fri, 26 Feb 2016 22:00:44 +0100

> Keep in mind that loop-over-loop is also possible...

Indeed! Let's remember that mount(8) should never do it.

# losetup /dev/loop0 /dev/sda2
# losetup /dev/loop1 /dev/loop0
# losetup -l
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
/dev/loop0         0      0         0  0 /dev/sda2
/dev/loop1         0      0         0  0 /dev/loop0


But it actually does, if the command line is "overlooped":

oct:~ # mount -oloop /dev/loop1 /mnt

as it does exactly that:
oct:~ # losetup -l
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
/dev/loop0         0      0         0  0 /dev/sda2
/dev/loop1         0      0         0  0 /dev/loop0
/dev/loop2         0      0         1  0 /dev/loop1

It should definitely report error whenever trying -oloop on top of
anything else than a file. Or at least a warning.

Well, even losetup should report a warning.

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                         e-mail: sbrabec@suse.com
Lihovarská 1060/12                            tel: +49 911 7405384547
190 00 Praha 9                                 fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Al Viro]

On Fri, Feb 26, 2016 at 10:36:50PM +0100, Stanislav Brabec wrote:

> It should definitely report error whenever trying -oloop on top of
> anything else than a file. Or at least a warning.
> 
> Well, even losetup should report a warning.

Keep in mind that with crypto in the game it just might be useful to have
loop-over-loop - it might be _not_ a no-op (hell, you might have two
layers of encryption - not the smartest thing to do, but if that's what
got dumped on your lap, you deal with what you've got).  So such warnings
shouldn't be hard errors.

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 282 bytes --]

On Fri, 26 Feb 2016 22:00:44 +0100, Stanislav Brabec said:

> Well, it seems to be safe, even if the loop device was not allocated by 
> mount(8) itself, as
> ioctl(fd, LOOP_CLR_FD)
> never returns EBUSY:

The fact you don't get an EBUSY doesn't mean it's actually safe....

[-- Attachment #2: Type: application/pgp-signature, Size: 848 bytes --]

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Austin S. Hemmelgarn]

On 2016-02-26 16:45, Al Viro wrote:
> On Fri, Feb 26, 2016 at 10:36:50PM +0100, Stanislav Brabec wrote:
>
>> It should definitely report error whenever trying -oloop on top of
>> anything else than a file. Or at least a warning.
>>
>> Well, even losetup should report a warning.
>
> Keep in mind that with crypto in the game it just might be useful to have
> loop-over-loop - it might be _not_ a no-op (hell, you might have two
> layers of encryption - not the smartest thing to do, but if that's what
> got dumped on your lap, you deal with what you've got).  So such warnings
> shouldn't be hard errors.
>
That and that using a loop device is one of the ways to expose 
partitions on a device the kernel doesn't normally expose them from.  In 
general, I'm pretty certain the preferred method is to use DM based 
mappings, but those aren't always available, and I've had multiple cases 
where I had to use nested loop devices to get to a filesystem.

You can't protect against everything, but you very much should not be 
doing something that is known to cause issues, and the current behavior 
of mount(8) WRT -o loop definitely can cause issues.

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Stanislav Brabec]

On Feb 26, 2016 at 23:00 Valdis.Kletnieks@vt.edu wrote:
> On Fri, 26 Feb 2016 22:00:44 +0100, Stanislav Brabec said:
>
>> Well, it seems to be safe, even if the loop device was not allocated by
>> mount(8) itself, as
>> ioctl(fd, LOOP_CLR_FD)
>> never returns EBUSY:
>
> The fact you don't get an EBUSY doesn't mean it's actually safe....
>
Then, what should mount do, when -oloop is used and loop for the file
is already set?

1) Verify that the loop device is a plain loop without encryption, and
recycle it.

2) Allocate new loop device? (Known to cause issues, and corrupts
structures on a current kernel.)

3) Trace loop devices allocated by mount itself and report error, if it
was not allocated by mount. (But there can still be legitimate uses,
e. g. two filesystems in one file, each at different offset, one of
them is encrypted.)

3) Report error and recommend direct use of /dev/loop*. (See 3 plus
setup of a system with /dev/loop* in fstab needs a non-standard
actions.)

4) Other ideas?

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                         e-mail: sbrabec@suse.com
Lihovarská 1060/12                            tel: +49 911 7405384547
190 00 Praha 9                                 fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Ming Lei]

On Mon, Feb 29, 2016 at 10:56 PM, Stanislav Brabec <sbrabec@suse.cz> wrote:
> On Feb 26, 2016 at 23:00 Valdis.Kletnieks@vt.edu wrote:
>>
>> On Fri, 26 Feb 2016 22:00:44 +0100, Stanislav Brabec said:
>>
>>> Well, it seems to be safe, even if the loop device was not allocated by
>>> mount(8) itself, as
>>> ioctl(fd, LOOP_CLR_FD)
>>> never returns EBUSY:
>>
>>
>> The fact you don't get an EBUSY doesn't mean it's actually safe....
>>
> Then, what should mount do, when -oloop is used and loop for the file
> is already set?
>
> 1) Verify that the loop device is a plain loop without encryption, and
> recycle it.
>
> 2) Allocate new loop device? (Known to cause issues, and corrupts
> structures on a current kernel.)
>
> 3) Trace loop devices allocated by mount itself and report error, if it
> was not allocated by mount. (But there can still be legitimate uses,
> e. g. two filesystems in one file, each at different offset, one of
> them is encrypted.)
>
> 3) Report error and recommend direct use of /dev/loop*. (See 3 plus
> setup of a system with /dev/loop* in fstab needs a non-standard
> actions.)
>
> 4) Other ideas?

One idea is to just take the loop which has been set for the file, and
it should work well no matter if the orignal loop is mounted or not.

Thanks,

>
>
> --
> Best Regards / S pozdravem,
>
> Stanislav Brabec
> software developer
> ---------------------------------------------------------------------
> SUSE LINUX, s. r. o.                         e-mail: sbrabec@suse.com
> Lihovarská 1060/12                            tel: +49 911 7405384547
> 190 00 Praha 9                                 fax:  +420 284 084 001
> Czech Republic                                    http://www.suse.cz/
> PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76



-- 
Ming Lei

Re: loop subsystem corrupted after mounting multiple btrfs sub-volumes

[Stanislav Brabec]

On Feb 26, 2016 at 21:30 Al Viro wrote:
> Sigh...  sys_mount() (mount_bdev(), actually) has no way to tell if two
> loop devices refer to the same underlying object.  As far as it's
> concerned, you are asking to mount a completely unrelated block device.
> Which just happens to see the data (living in separate pagecache, even)
> modified behind its back (with some delay) after it gets written to another
> device.  Filesystem drivers generally don't like when something is screwing
> the underlying data, to put it mildly...
>

I wrote a loop device reuse patch for mount -oloop.

[PATCH 0/3] btrfs-safe implementation of -oloop
http://marc.info/?l=util-linux-ng&m=146048532307963&w=2

[PATCH 1/3] libmount: Re-organize is_mounted_same_loopfile()
http://marc.info/?l=util-linux-ng&m=146048535907971&w=2

[PATCH 2/3] libmount: reuse existing loop device
http://marc.info/?l=util-linux-ng&m=146048537807980&w=2

[PATCH 3/3] mount: Handle EROFS before calling mount() syscall
http://marc.info/?l=util-linux-ng&m=146048542007990&w=2

However it works for me, there are still some controversial issues
described in [PATCH 0/3].

These patches will hide corruption of kernel loop control structures
mentioned earlier in this thread and in most cases prevent data
corruption.

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                         e-mail: sbrabec@suse.com
Lihovarská 1060/12                            tel: +49 911 7405384547
190 00 Praha 9                                 fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76