Linux-BTRFS Archive on lore.kernel.org
 help / color / Atom feed
* updatedb does not index /home when /home is Btrfs
@ 2017-11-04  0:15 Chris Murphy
  2017-11-04  4:49 ` Adam Borowski
  2017-11-05  7:47 ` Fixed subject: updatedb does not index separately mounted btrfs subvolumes Duncan
  0 siblings, 2 replies; 14+ messages in thread
From: Chris Murphy @ 2017-11-04  0:15 UTC (permalink / raw)
  To: Btrfs BTRFS

Ancient bug, still seems to be a bug.
https://bugzilla.redhat.com/show_bug.cgi?id=906591

The issue is that updatedb by default will not index bind mounts, but
by default on Fedora and probably other distros, put /home on a
subvolume and then mount that subvolume which is in effect a bind
mount.

There's a lot of early discussion in 2013 about it, but then it's
dropped off the radar as nobody has any ideas how to fix this in
mlocate.


-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: updatedb does not index /home when /home is Btrfs
  2017-11-04  0:15 updatedb does not index /home when /home is Btrfs Chris Murphy
@ 2017-11-04  4:49 ` Adam Borowski
  2017-11-04  6:26   ` Andrei Borzenkov
  2017-11-05  7:47 ` Fixed subject: updatedb does not index separately mounted btrfs subvolumes Duncan
  1 sibling, 1 reply; 14+ messages in thread
From: Adam Borowski @ 2017-11-04  4:49 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Btrfs BTRFS

On Fri, Nov 03, 2017 at 06:15:53PM -0600, Chris Murphy wrote:
> Ancient bug, still seems to be a bug.
> https://bugzilla.redhat.com/show_bug.cgi?id=906591
> 
> The issue is that updatedb by default will not index bind mounts, but
> by default on Fedora and probably other distros, put /home on a
> subvolume and then mount that subvolume which is in effect a bind
> mount.
> 
> There's a lot of early discussion in 2013 about it, but then it's
> dropped off the radar as nobody has any ideas how to fix this in
> mlocate.

I don't see how this would be a bug in btrfs.  The same happens if you
bind-mount /home (or individual homes), which is a valid and non-rare setup.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ Laws we want back: Poland, Dz.U. 1921 nr.30 poz.177 (also Dz.U. 
⣾⠁⢰⠒⠀⣿⡁ 1920 nr.11 poz.61): Art.2: An official, guilty of accepting a gift
⢿⡄⠘⠷⠚⠋⠀ or another material benefit, or a promise thereof, [in matters
⠈⠳⣄⠀⠀⠀⠀ relevant to duties], shall be punished by death by shooting.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: updatedb does not index /home when /home is Btrfs
  2017-11-04  4:49 ` Adam Borowski
@ 2017-11-04  6:26   ` Andrei Borzenkov
  2017-11-04  7:05     ` Adam Borowski
  0 siblings, 1 reply; 14+ messages in thread
From: Andrei Borzenkov @ 2017-11-04  6:26 UTC (permalink / raw)
  To: Adam Borowski, Chris Murphy; +Cc: Btrfs BTRFS

04.11.2017 07:49, Adam Borowski пишет:
> On Fri, Nov 03, 2017 at 06:15:53PM -0600, Chris Murphy wrote:
>> Ancient bug, still seems to be a bug.
>> https://bugzilla.redhat.com/show_bug.cgi?id=906591
>>
>> The issue is that updatedb by default will not index bind mounts, but
>> by default on Fedora and probably other distros, put /home on a
>> subvolume and then mount that subvolume which is in effect a bind
>> mount.
>>
>> There's a lot of early discussion in 2013 about it, but then it's
>> dropped off the radar as nobody has any ideas how to fix this in
>> mlocate.
> 
> I don't see how this would be a bug in btrfs.  The same happens if you
> bind-mount /home (or individual homes), which is a valid and non-rare setup.
> 

It is the problem *on* btrfs because - as opposed to normal bind mount -
those mount points do *not* refer to the same content. As was commented
in mentioned bug report:

mount -o subvol=root /dev/sdb1 /root
mount -o subvol=foo /dev/sdb1 /root/foo
mount -o subvol bar /dev/sdb1 /bar/bar

Both /root/foo and /root/bar, will be skipped even though they are not
accessible via any other path (on mounted filesystem)


191 25 0:54 /root /home/bor/tmp/root rw,relatime shared:131 - btrfs
/dev/loop0 rw,space_cache,subvolid=258,subvol=/root
285 191 0:54 /foo /home/bor/tmp/root/foo rw,relatime shared:239 - btrfs
/dev/loop0 rw,space_cache,subvolid=256,subvol=/foo
325 191 0:54 /bar /home/bor/tmp/root/bar rw,relatime shared:279 - btrfs
/dev/loop0 rw,space_cache,subvolid=257,subvol=/bar

bor@bor-Latitude-E5450:~/tmp$ sudo updatedb --debug-pruning -l 0 -o
../db -U root
...
Matching bind_mount_paths:
 => adding `/home/bor/tmp/root/foo'
 => adding `/home/bor/tmp/root/bar'
...done


It is a problem *of* btrfs because it does not offer any easy way to
distinguish between subvolume mount and bind mount. If you are aware of
one, please comment on mentioned bug report.

And note that updatedb can be run as non-root as well, so it probably
cannot use btrfs specific ioctls to extract information.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: updatedb does not index /home when /home is Btrfs
  2017-11-04  6:26   ` Andrei Borzenkov
@ 2017-11-04  7:05     ` Adam Borowski
  2017-11-04 18:27       ` Andrei Borzenkov
  0 siblings, 1 reply; 14+ messages in thread
From: Adam Borowski @ 2017-11-04  7:05 UTC (permalink / raw)
  To: Andrei Borzenkov; +Cc: Chris Murphy, Btrfs BTRFS

On Sat, Nov 04, 2017 at 09:26:36AM +0300, Andrei Borzenkov wrote:
> 04.11.2017 07:49, Adam Borowski пишет:
> > On Fri, Nov 03, 2017 at 06:15:53PM -0600, Chris Murphy wrote:
> >> Ancient bug, still seems to be a bug.
> >> https://bugzilla.redhat.com/show_bug.cgi?id=906591
> >>
> >> The issue is that updatedb by default will not index bind mounts, but
> >> by default on Fedora and probably other distros, put /home on a
> >> subvolume and then mount that subvolume which is in effect a bind
> >> mount.
> >>
> >> There's a lot of early discussion in 2013 about it, but then it's
> >> dropped off the radar as nobody has any ideas how to fix this in
> >> mlocate.
> > 
> > I don't see how this would be a bug in btrfs.  The same happens if you
> > bind-mount /home (or individual homes), which is a valid and non-rare setup.
> 
> It is the problem *on* btrfs because - as opposed to normal bind mount -
> those mount points do *not* refer to the same content.

Neither do they refer to in a "normal" bind mount.

> As was commented in mentioned bug report:
> 
> mount -o subvol=root /dev/sdb1 /root
> mount -o subvol=foo /dev/sdb1 /root/foo
> mount -o subvol bar /dev/sdb1 /bar/bar
> 
> Both /root/foo and /root/bar, will be skipped even though they are not
> accessible via any other path (on mounted filesystem)

losetup -D
truncate -s 4G junk
losetup -f junk
mkfs.ext4 /dev/loop0
mkdir -p foo bar
mount /dev/loop0 foo
mkdir foo/bar
touch foo/fileA foo/bar/fileB
mount --bind foo/bar bar
umount foo

> It is a problem *of* btrfs because it does not offer any easy way to
> distinguish between subvolume mount and bind mount. If you are aware of
> one, please comment on mentioned bug report.

Well, subvolume mounts are indistinguishable from bind mounts because they
_are_ bind mounts.  You merely don't need to mount the "master" first.

The only way such a "master" mount is special is that, on most filesystems,
its root was accessible at least at some point (but it might no longer be,
thanks to chroot, pivot_root, etc).


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ Laws we want back: Poland, Dz.U. 1921 nr.30 poz.177 (also Dz.U. 
⣾⠁⢰⠒⠀⣿⡁ 1920 nr.11 poz.61): Art.2: An official, guilty of accepting a gift
⢿⡄⠘⠷⠚⠋⠀ or another material benefit, or a promise thereof, [in matters
⠈⠳⣄⠀⠀⠀⠀ relevant to duties], shall be punished by death by shooting.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: updatedb does not index /home when /home is Btrfs
  2017-11-04  7:05     ` Adam Borowski
@ 2017-11-04 18:27       ` Andrei Borzenkov
  2017-11-04 18:55         ` Chris Murphy
  0 siblings, 1 reply; 14+ messages in thread
From: Andrei Borzenkov @ 2017-11-04 18:27 UTC (permalink / raw)
  To: Adam Borowski; +Cc: Chris Murphy, Btrfs BTRFS

04.11.2017 10:05, Adam Borowski пишет:
> On Sat, Nov 04, 2017 at 09:26:36AM +0300, Andrei Borzenkov wrote:
>> 04.11.2017 07:49, Adam Borowski пишет:
>>> On Fri, Nov 03, 2017 at 06:15:53PM -0600, Chris Murphy wrote:
>>>> Ancient bug, still seems to be a bug.
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=906591
>>>>
>>>> The issue is that updatedb by default will not index bind mounts, but
>>>> by default on Fedora and probably other distros, put /home on a
>>>> subvolume and then mount that subvolume which is in effect a bind
>>>> mount.
>>>>
>>>> There's a lot of early discussion in 2013 about it, but then it's
>>>> dropped off the radar as nobody has any ideas how to fix this in
>>>> mlocate.
>>>
>>> I don't see how this would be a bug in btrfs.  The same happens if you
>>> bind-mount /home (or individual homes), which is a valid and non-rare setup.
>>
>> It is the problem *on* btrfs because - as opposed to normal bind mount -
>> those mount points do *not* refer to the same content.
> 
> Neither do they refer to in a "normal" bind mount.
> 
>> As was commented in mentioned bug report:
>>
>> mount -o subvol=root /dev/sdb1 /root
>> mount -o subvol=foo /dev/sdb1 /root/foo
>> mount -o subvol bar /dev/sdb1 /bar/bar
>>
>> Both /root/foo and /root/bar, will be skipped even though they are not
>> accessible via any other path (on mounted filesystem)
> 
> losetup -D
> truncate -s 4G junk
> losetup -f junk
> mkfs.ext4 /dev/loop0
> mkdir -p foo bar
> mount /dev/loop0 foo
> mkdir foo/bar
> touch foo/fileA foo/bar/fileB
> mount --bind foo/bar bar
> umount foo
> 

Indeed. I can build the same configuration on non-btrfs and updatedb
would skip non-overlapping mounts just as it would on btrfs. It is just
that it is rather more involved on other filesystems (and as you
mentioned this requires top-level to be mounted at some point), while on
btrfs it is much easier to get (and is default on number of distributions).

So yes, it really appears that updatedb check for duplicated mounts is
wrong in general and needs rethinking.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: updatedb does not index /home when /home is Btrfs
  2017-11-04 18:27       ` Andrei Borzenkov
@ 2017-11-04 18:55         ` Chris Murphy
  2017-11-04 19:37           ` Nicholas D Steeves
  2017-11-05  8:01           ` Andrei Borzenkov
  0 siblings, 2 replies; 14+ messages in thread
From: Chris Murphy @ 2017-11-04 18:55 UTC (permalink / raw)
  To: Andrei Borzenkov; +Cc: Adam Borowski, Chris Murphy, Btrfs BTRFS

On Sat, Nov 4, 2017 at 12:27 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> 04.11.2017 10:05, Adam Borowski пишет:
>> On Sat, Nov 04, 2017 at 09:26:36AM +0300, Andrei Borzenkov wrote:
>>> 04.11.2017 07:49, Adam Borowski пишет:
>>>> On Fri, Nov 03, 2017 at 06:15:53PM -0600, Chris Murphy wrote:
>>>>> Ancient bug, still seems to be a bug.
>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=906591
>>>>>
>>>>> The issue is that updatedb by default will not index bind mounts, but
>>>>> by default on Fedora and probably other distros, put /home on a
>>>>> subvolume and then mount that subvolume which is in effect a bind
>>>>> mount.
>>>>>
>>>>> There's a lot of early discussion in 2013 about it, but then it's
>>>>> dropped off the radar as nobody has any ideas how to fix this in
>>>>> mlocate.
>>>>
>>>> I don't see how this would be a bug in btrfs.  The same happens if you
>>>> bind-mount /home (or individual homes), which is a valid and non-rare setup.
>>>
>>> It is the problem *on* btrfs because - as opposed to normal bind mount -
>>> those mount points do *not* refer to the same content.
>>
>> Neither do they refer to in a "normal" bind mount.
>>
>>> As was commented in mentioned bug report:
>>>
>>> mount -o subvol=root /dev/sdb1 /root
>>> mount -o subvol=foo /dev/sdb1 /root/foo
>>> mount -o subvol bar /dev/sdb1 /bar/bar
>>>
>>> Both /root/foo and /root/bar, will be skipped even though they are not
>>> accessible via any other path (on mounted filesystem)
>>
>> losetup -D
>> truncate -s 4G junk
>> losetup -f junk
>> mkfs.ext4 /dev/loop0
>> mkdir -p foo bar
>> mount /dev/loop0 foo
>> mkdir foo/bar
>> touch foo/fileA foo/bar/fileB
>> mount --bind foo/bar bar
>> umount foo
>>
>
> Indeed. I can build the same configuration on non-btrfs and updatedb
> would skip non-overlapping mounts just as it would on btrfs. It is just
> that it is rather more involved on other filesystems (and as you
> mentioned this requires top-level to be mounted at some point), while on
> btrfs it is much easier to get (and is default on number of distributions).
>
> So yes, it really appears that updatedb check for duplicated mounts is
> wrong in general and needs rethinking.

Yes, even if it's not a Btrfs bug, I think it's useful to get a
different set of eyes on this than just the mlocate folks. Maybe it
should get posted to fs-devel?

-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: updatedb does not index /home when /home is Btrfs
  2017-11-04 18:55         ` Chris Murphy
@ 2017-11-04 19:37           ` Nicholas D Steeves
  2017-11-05  8:01           ` Andrei Borzenkov
  1 sibling, 0 replies; 14+ messages in thread
From: Nicholas D Steeves @ 2017-11-04 19:37 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Andrei Borzenkov, Adam Borowski, Btrfs BTRFS

On 4 November 2017 at 14:55, Chris Murphy <lists@colorremedies.com> wrote:
> On Sat, Nov 4, 2017 at 12:27 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>> 04.11.2017 10:05, Adam Borowski пишет:
>>> On Sat, Nov 04, 2017 at 09:26:36AM +0300, Andrei Borzenkov wrote:
>>>> 04.11.2017 07:49, Adam Borowski пишет:
>>>>> On Fri, Nov 03, 2017 at 06:15:53PM -0600, Chris Murphy wrote:
>>>>>> Ancient bug, still seems to be a bug.
>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=906591
>>>>>>
>>>>>> The issue is that updatedb by default will not index bind mounts, but
>>>>>> by default on Fedora and probably other distros, put /home on a
>>>>>> subvolume and then mount that subvolume which is in effect a bind
>>>>>> mount.
>>>>>>
>>>>>> There's a lot of early discussion in 2013 about it, but then it's
>>>>>> dropped off the radar as nobody has any ideas how to fix this in
>>>>>> mlocate.
>>>>>
>>>>> I don't see how this would be a bug in btrfs.  The same happens if you
>>>>> bind-mount /home (or individual homes), which is a valid and non-rare setup.
>>>>
>>>> It is the problem *on* btrfs because - as opposed to normal bind mount -
>>>> those mount points do *not* refer to the same content.
>>>
>>> Neither do they refer to in a "normal" bind mount.
>>>
>>>> As was commented in mentioned bug report:
>>>>
>>>> mount -o subvol=root /dev/sdb1 /root
>>>> mount -o subvol=foo /dev/sdb1 /root/foo
>>>> mount -o subvol bar /dev/sdb1 /bar/bar
>>>>
>>>> Both /root/foo and /root/bar, will be skipped even though they are not
>>>> accessible via any other path (on mounted filesystem)
>>>
>>> losetup -D
>>> truncate -s 4G junk
>>> losetup -f junk
>>> mkfs.ext4 /dev/loop0
>>> mkdir -p foo bar
>>> mount /dev/loop0 foo
>>> mkdir foo/bar
>>> touch foo/fileA foo/bar/fileB
>>> mount --bind foo/bar bar
>>> umount foo
>>>
>>
>> Indeed. I can build the same configuration on non-btrfs and updatedb
>> would skip non-overlapping mounts just as it would on btrfs. It is just
>> that it is rather more involved on other filesystems (and as you
>> mentioned this requires top-level to be mounted at some point), while on
>> btrfs it is much easier to get (and is default on number of distributions).
>>
>> So yes, it really appears that updatedb check for duplicated mounts is
>> wrong in general and needs rethinking.
>
> Yes, even if it's not a Btrfs bug, I think it's useful to get a
> different set of eyes on this than just the mlocate folks. Maybe it
> should get posted to fs-devel?

How is this not a configuration issue?  For btrfs users why not just
recommend PRUNE_BIND_MOUNTS="no" and a particular set of PRUNEPATHS?

I have each top-level subvolume (id=5 or subvol=/) mounted at
/btrfs-admin/$LABEL, where /btrfs-admin is root:sudo 750, and this is
what I use in /etc/updatedb.conf:

PRUNE_BIND_MOUNTS="no"
PRUNENAMES=".git .bzr .hg .svn"
PRUNEPATHS="/tmp /var/spool /media /btrfs-admin /var/cache /var/lib/lxc"
PRUNEFS="NFS nfs nfs4 rpc_pipefs afs binfmt_misc proc smbfs autofs
iso9660 ncpfs coda devpts ftpfs devfs mfs shfs sysfs cifs lustre tmpfs
usbfs udf fuse.glusterfs fuse.sshfs curlftpfs"

With the exception of LXC rootfss I have a flat subvolume structure
under each subvol=/.  These subvolumes are mounted at specific
mountpoints using fstab.  Given that updatedb and locate work
flawlessly, and that I've only had two issues (freespacecache) while
using LTS kernels, I'm inclined to conclude that this is the least
disruptive configuration.  If I used snapper I'd add it to PRUNEPATHS
and rely on its facilities to find files that had been deleted,
because I don't want to see n-duplicates-for-file when I use locate.
A user who wanted to see those duplicates could remove the path from
PRUNEPATHS.

Sincerely,
Nicholas

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Fixed subject: updatedb does not index separately mounted btrfs subvolumes
  2017-11-04  0:15 updatedb does not index /home when /home is Btrfs Chris Murphy
  2017-11-04  4:49 ` Adam Borowski
@ 2017-11-05  7:47 ` Duncan
  2017-11-05 14:02   ` Chris Murphy
  2017-11-06  0:27   ` Peter Grandi
  1 sibling, 2 replies; 14+ messages in thread
From: Duncan @ 2017-11-05  7:47 UTC (permalink / raw)
  To: linux-btrfs

Chris Murphy posted on Fri, 03 Nov 2017 18:15:53 -0600 as excerpted:

> Ancient bug, still seems to be a bug.
> https://bugzilla.redhat.com/show_bug.cgi?id=906591
> 
> The issue is that updatedb by default will not index bind mounts, but by
> default on Fedora and probably other distros, put /home on a subvolume
> and then mount that subvolume which is in effect a bind mount.

<gripe-mode: inaccurate subject lines>

So the issue isn't /home being btrfs (as you said in the subject), but 
rather, it's /home being an explicitly mounted subvolume, since btrfs 
uses bind-mounts internally for subvolume mounts.

Makes a difference for folks like me who have /home on a fully separate 
btrfs.  You had me wondering for a moment, tho as it happens I don't use 
updatedb either, but I was still wondering what weird bug and how it 
might otherwise affect me.  An accurate subject line would have helped...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: updatedb does not index /home when /home is Btrfs
  2017-11-04 18:55         ` Chris Murphy
  2017-11-04 19:37           ` Nicholas D Steeves
@ 2017-11-05  8:01           ` Andrei Borzenkov
  2017-11-06 13:51             ` Austin S. Hemmelgarn
  1 sibling, 1 reply; 14+ messages in thread
From: Andrei Borzenkov @ 2017-11-05  8:01 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Adam Borowski, Btrfs BTRFS

04.11.2017 21:55, Chris Murphy пишет:
> On Sat, Nov 4, 2017 at 12:27 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>> 04.11.2017 10:05, Adam Borowski пишет:
>>> On Sat, Nov 04, 2017 at 09:26:36AM +0300, Andrei Borzenkov wrote:
>>>> 04.11.2017 07:49, Adam Borowski пишет:
>>>>> On Fri, Nov 03, 2017 at 06:15:53PM -0600, Chris Murphy wrote:
>>>>>> Ancient bug, still seems to be a bug.
>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=906591
>>>>>>
>>>>>> The issue is that updatedb by default will not index bind mounts, but
>>>>>> by default on Fedora and probably other distros, put /home on a
>>>>>> subvolume and then mount that subvolume which is in effect a bind
>>>>>> mount.
>>>>>>
>>>>>> There's a lot of early discussion in 2013 about it, but then it's
>>>>>> dropped off the radar as nobody has any ideas how to fix this in
>>>>>> mlocate.
>>>>>
>>>>> I don't see how this would be a bug in btrfs.  The same happens if you
>>>>> bind-mount /home (or individual homes), which is a valid and non-rare setup.
>>>>
>>>> It is the problem *on* btrfs because - as opposed to normal bind mount -
>>>> those mount points do *not* refer to the same content.
>>>
>>> Neither do they refer to in a "normal" bind mount.
>>>
>>>> As was commented in mentioned bug report:
>>>>
>>>> mount -o subvol=root /dev/sdb1 /root
>>>> mount -o subvol=foo /dev/sdb1 /root/foo
>>>> mount -o subvol bar /dev/sdb1 /bar/bar
>>>>
>>>> Both /root/foo and /root/bar, will be skipped even though they are not
>>>> accessible via any other path (on mounted filesystem)
>>>
>>> losetup -D
>>> truncate -s 4G junk
>>> losetup -f junk
>>> mkfs.ext4 /dev/loop0
>>> mkdir -p foo bar
>>> mount /dev/loop0 foo
>>> mkdir foo/bar
>>> touch foo/fileA foo/bar/fileB
>>> mount --bind foo/bar bar
>>> umount foo
>>>
>>
>> Indeed. I can build the same configuration on non-btrfs and updatedb
>> would skip non-overlapping mounts just as it would on btrfs. It is just
>> that it is rather more involved on other filesystems (and as you
>> mentioned this requires top-level to be mounted at some point), while on
>> btrfs it is much easier to get (and is default on number of distributions).
>>
>> So yes, it really appears that updatedb check for duplicated mounts is
>> wrong in general and needs rethinking.
> 
> Yes, even if it's not a Btrfs bug, I think it's useful to get a
> different set of eyes on this than just the mlocate folks. Maybe it
> should get posted to fs-devel?
> 

Looking at mlocate history, initial bind detection was extremely
simplistic but actually correct, and would still work even with btrfs -
just look in /etc/mtab for mount with "bind" option where what != where.
This covers any sort of bind mount.

Later /etc/mtab disappeared and code was rewritten to use mountinfo.
Intentionally or not, this rewrite only works for bind mounts inside the
same filesystem subtree. I.e. it also won't catch cross filesystem bind
mounts. Failure on btrfs is side effect of this assumption.

So it actually can be considered regression in mlocate code.

I suppose first mlocate folks need to get clear answer what they want to
test here, then it makes sense to discuss how to do it.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Fixed subject: updatedb does not index separately mounted btrfs subvolumes
  2017-11-05  7:47 ` Fixed subject: updatedb does not index separately mounted btrfs subvolumes Duncan
@ 2017-11-05 14:02   ` Chris Murphy
  2017-11-06  0:27   ` Peter Grandi
  1 sibling, 0 replies; 14+ messages in thread
From: Chris Murphy @ 2017-11-05 14:02 UTC (permalink / raw)
  To: Duncan; +Cc: Btrfs BTRFS

On Sun, Nov 5, 2017 at 1:47 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> Chris Murphy posted on Fri, 03 Nov 2017 18:15:53 -0600 as excerpted:
>
>> Ancient bug, still seems to be a bug.
>> https://bugzilla.redhat.com/show_bug.cgi?id=906591
>>
>> The issue is that updatedb by default will not index bind mounts, but by
>> default on Fedora and probably other distros, put /home on a subvolume
>> and then mount that subvolume which is in effect a bind mount.
>
> <gripe-mode: inaccurate subject lines>
>
> So the issue isn't /home being btrfs (as you said in the subject), but
> rather, it's /home being an explicitly mounted subvolume, since btrfs
> uses bind-mounts internally for subvolume mounts.
>
> Makes a difference for folks like me who have /home on a fully separate
> btrfs.  You had me wondering for a moment, tho as it happens I don't use
> updatedb either, but I was still wondering what weird bug and how it
> might otherwise affect me.  An accurate subject line would have helped...


It is accurate, it's an exactly copied title from the bug report. That
the bug report's title is confusing is sorta beside the point, I
certainly can't do anything about that.

-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Fixed subject: updatedb does not index separately mounted btrfs subvolumes
  2017-11-05  7:47 ` Fixed subject: updatedb does not index separately mounted btrfs subvolumes Duncan
  2017-11-05 14:02   ` Chris Murphy
@ 2017-11-06  0:27   ` Peter Grandi
  1 sibling, 0 replies; 14+ messages in thread
From: Peter Grandi @ 2017-11-06  0:27 UTC (permalink / raw)
  To: Linux fs Btrfs

>> The issue is that updatedb by default will not index bind
>> mounts, but by default on Fedora and probably other distros,
>> put /home on a subvolume and then mount that subvolume which
>> is in effect a bind mount.

> <gripe-mode: inaccurate subject lines>

> So the issue isn't /home being btrfs (as you said in the
> subject), but rather, it's /home being an explicitly mounted
> subvolume, since btrfs uses bind-mounts internally for
> subvolume mounts.

That to me seems like rather improper terminology and notes, and
I would consider these to be more appropriate:

* There are entities known as "root directories", and their main
  property is that all inodes reachable from one in the same
  filesystem have the same "device id".
* Each "filesystem" has at least one, and a Btrfs "volume" has
  one for every "subvolume", including the "top subvolume".
* A "root directory" can be "mounted" on a "mount point"
  directory of another "filesystem", which allows navigating
  from one filesystem to another.
* A "mounted" root directory can be identified by the device id
  of '.' being different from that of '..'.
* In Linux a "root directory" can be "mounted" onto several
  "mount point" directories at the same time.
* In Linux a "bind" operation is not a "mount" operation, it is
  in effect a kind of temporary "hard link", one that makes a
  directory aliased to a "bind point" directory.

Looking at this:

  tree#  tail -3 /proc/mounts                                                                                                                                            
  /dev/mapper/sda7 /fs/sda7 btrfs rw,nodiratime,relatime,nossd,nospace_cache,user_subvol_rm_allowed,subvolid=5,subvol=/ 0 0
  /dev/mapper/sda7 /fs/sda7/bind btrfs rw,nodiratime,relatime,nossd,nospace_cache,user_subvol_rm_allowed,subvolid=431,subvol=/= 0 0
  /dev/mapper/sda7 /fs/sda7/bind-tmp btrfs rw,nodiratime,relatime,nossd,nospace_cache,user_subvol_rm_allowed,subvolid=431,subvol=/=/tmp 0 0

  tree#  stat --format "%3D %6i %N" {,/fs,/fs/sda7}/{.,..} /fs/sda7/{=,=/subvol,=/subvol/dir,=/tmp,bind,bind-tmp}/{.,..} 
  806      2 ‘/.’
  806      2 ‘/..’
   23  36176 ‘/fs/.’
  806      2 ‘/fs/..’
   26    256 ‘/fs/sda7/.’
   23  36176 ‘/fs/sda7/..’
   27    256 ‘/fs/sda7/=/.’
   26    256 ‘/fs/sda7/=/..’
   2b    256 ‘/fs/sda7/=/subvol/.’
   27    256 ‘/fs/sda7/=/subvol/..’
   2b    258 ‘/fs/sda7/=/subvol/dir/.’
   2b    256 ‘/fs/sda7/=/subvol/dir/..’
   27 344618 ‘/fs/sda7/=/tmp/.’
   27    256 ‘/fs/sda7/=/tmp/..’
   27    256 ‘/fs/sda7/bind/.’
   26    256 ‘/fs/sda7/bind/..’
   27 344618 ‘/fs/sda7/bind-tmp/.’
   26    256 ‘/fs/sda7/bind-tmp/..’

It shows that subvolume root directories are "mount points" and
not "bind points" (note that ‘/fs/sda7/=/subvol’ is not
explicitly mounted, yet its '.' and '..' have different device
ids), and that "bind points" appear as if they were ordinary
directories (an unwise decision I suspect).

Many tools for UNIX-like systems don't cross "mount point"
directories (or follow symbolic links), by default or with an
option, but will cross "bind point" directories as they look
like ordinary directories.

For 'mlocate' the "bind point" directories are a special case,
handled by looking up every directory examined in the list of
"bind point" directories, as per line 381 here:

https://pagure.io/mlocate/blob/master/f/src/bind-mount.c#_381

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: updatedb does not index /home when /home is Btrfs
  2017-11-05  8:01           ` Andrei Borzenkov
@ 2017-11-06 13:51             ` Austin S. Hemmelgarn
  2017-11-06 18:35               ` Chris Murphy
  0 siblings, 1 reply; 14+ messages in thread
From: Austin S. Hemmelgarn @ 2017-11-06 13:51 UTC (permalink / raw)
  To: Andrei Borzenkov, Chris Murphy; +Cc: Adam Borowski, Btrfs BTRFS

On 2017-11-05 03:01, Andrei Borzenkov wrote:
> 04.11.2017 21:55, Chris Murphy пишет:
>> On Sat, Nov 4, 2017 at 12:27 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>>> 04.11.2017 10:05, Adam Borowski пишет:
>>>> On Sat, Nov 04, 2017 at 09:26:36AM +0300, Andrei Borzenkov wrote:
>>>>> 04.11.2017 07:49, Adam Borowski пишет:
>>>>>> On Fri, Nov 03, 2017 at 06:15:53PM -0600, Chris Murphy wrote:
>>>>>>> Ancient bug, still seems to be a bug.
>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=906591
>>>>>>>
>>>>>>> The issue is that updatedb by default will not index bind mounts, but
>>>>>>> by default on Fedora and probably other distros, put /home on a
>>>>>>> subvolume and then mount that subvolume which is in effect a bind
>>>>>>> mount.
>>>>>>>
>>>>>>> There's a lot of early discussion in 2013 about it, but then it's
>>>>>>> dropped off the radar as nobody has any ideas how to fix this in
>>>>>>> mlocate.
>>>>>>
>>>>>> I don't see how this would be a bug in btrfs.  The same happens if you
>>>>>> bind-mount /home (or individual homes), which is a valid and non-rare setup.
>>>>>
>>>>> It is the problem *on* btrfs because - as opposed to normal bind mount -
>>>>> those mount points do *not* refer to the same content.
>>>>
>>>> Neither do they refer to in a "normal" bind mount.
>>>>
>>>>> As was commented in mentioned bug report:
>>>>>
>>>>> mount -o subvol=root /dev/sdb1 /root
>>>>> mount -o subvol=foo /dev/sdb1 /root/foo
>>>>> mount -o subvol bar /dev/sdb1 /bar/bar
>>>>>
>>>>> Both /root/foo and /root/bar, will be skipped even though they are not
>>>>> accessible via any other path (on mounted filesystem)
>>>>
>>>> losetup -D
>>>> truncate -s 4G junk
>>>> losetup -f junk
>>>> mkfs.ext4 /dev/loop0
>>>> mkdir -p foo bar
>>>> mount /dev/loop0 foo
>>>> mkdir foo/bar
>>>> touch foo/fileA foo/bar/fileB
>>>> mount --bind foo/bar bar
>>>> umount foo
>>>>
>>>
>>> Indeed. I can build the same configuration on non-btrfs and updatedb
>>> would skip non-overlapping mounts just as it would on btrfs. It is just
>>> that it is rather more involved on other filesystems (and as you
>>> mentioned this requires top-level to be mounted at some point), while on
>>> btrfs it is much easier to get (and is default on number of distributions).
>>>
>>> So yes, it really appears that updatedb check for duplicated mounts is
>>> wrong in general and needs rethinking.
>>
>> Yes, even if it's not a Btrfs bug, I think it's useful to get a
>> different set of eyes on this than just the mlocate folks. Maybe it
>> should get posted to fs-devel?
>>
> 
> Looking at mlocate history, initial bind detection was extremely
> simplistic but actually correct, and would still work even with btrfs -
> just look in /etc/mtab for mount with "bind" option where what != where.
> This covers any sort of bind mount.
> 
> Later /etc/mtab disappeared and code was rewritten to use mountinfo.
> Intentionally or not, this rewrite only works for bind mounts inside the
> same filesystem subtree. I.e. it also won't catch cross filesystem bind
> mounts. Failure on btrfs is side effect of this assumption.
This brings to mind another 'feature' of BTRFS that I came across 
recently, namely that subvolumes that aren't explicitly mounted still 
show up as mount points according to how most CLI tools differentiate 
what's a mount point.

In particular, the st_dev field in stat() results for the subvolume 
differs from the containing directory, and the f_fsid field in statvfs() 
results for the subvolume differs from the containing directory (a side 
effect of the differing st_dev field, which is part of what's used to 
calculate f_fsid on Linux), which means the only way to know if 
something actually is a mount point is to make this check, and then 
verify it in /proc/mounts or /proc/self/mountinfo.

That particular 'feature' means that GNU find, xargs, and du will never 
cross subvolume boundaries if you tell them to stay on one filesystem, 
and some other tools may misidentify where things are mounted.
> 
> So it actually can be considered regression in mlocate code.
> 
> I suppose first mlocate folks need to get clear answer what they want to
> test here, then it makes sense to discuss how to do it.
Agreed.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: updatedb does not index /home when /home is Btrfs
  2017-11-06 13:51             ` Austin S. Hemmelgarn
@ 2017-11-06 18:35               ` Chris Murphy
  2017-11-06 19:44                 ` Austin S. Hemmelgarn
  0 siblings, 1 reply; 14+ messages in thread
From: Chris Murphy @ 2017-11-06 18:35 UTC (permalink / raw)
  To: Austin S. Hemmelgarn
  Cc: Andrei Borzenkov, Chris Murphy, Adam Borowski, Btrfs BTRFS

On Mon, Nov 6, 2017 at 6:51 AM, Austin S. Hemmelgarn
<ahferroin7@gmail.com> wrote:

> This brings to mind another 'feature' of BTRFS that I came across recently,
> namely that subvolumes that aren't explicitly mounted still show up as mount
> points according to how most CLI tools differentiate what's a mount point.
>
> In particular, the st_dev field in stat() results for the subvolume differs
> from the containing directory, and the f_fsid field in statvfs() results for
> the subvolume differs from the containing directory (a side effect of the
> differing st_dev field, which is part of what's used to calculate f_fsid on
> Linux), which means the only way to know if something actually is a mount
> point is to make this check, and then verify it in /proc/mounts or
> /proc/self/mountinfo.
>
> That particular 'feature' means that GNU find, xargs, and du will never
> cross subvolume boundaries if you tell them to stay on one filesystem, and
> some other tools may misidentify where things are mounted.


Elsewhere I brought up that mountinfo gives bogus subvol= information
that conflicts with the subvolid= information, when doing bind mounts
of directories on Btrfs. Trivially reproduced in my home directory:


[chris@f26h ~]$ mkdir directory1
[chris@f26h ~]$ mkdir directory2
[chris@f26h ~]$ sudo mount -B directory2 directory1
[chris@f26h ~]$ mount:
/dev/nvme0n1p8 on /home type btrfs
(rw,relatime,seclabel,ssd,space_cache,subvolid=257,subvol=/home)
/dev/nvme0n1p8 on /home/chris/directory1 type btrfs
(rw,relatime,seclabel,ssd,space_cache,subvolid=257,subvol=/home/chris/directory2)



The first mount item is correct, there is a subvolume home with
subvolume ID of 257, mounted at /home.
The second mount item considers directory2 a subvolume (wrong), with
subvolid 257 (kinda right, it's on subvolid 257, but kinda wrong, it
is not itself a subvolume).


-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: updatedb does not index /home when /home is Btrfs
  2017-11-06 18:35               ` Chris Murphy
@ 2017-11-06 19:44                 ` Austin S. Hemmelgarn
  0 siblings, 0 replies; 14+ messages in thread
From: Austin S. Hemmelgarn @ 2017-11-06 19:44 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Andrei Borzenkov, Adam Borowski, Btrfs BTRFS

On 2017-11-06 13:35, Chris Murphy wrote:
> On Mon, Nov 6, 2017 at 6:51 AM, Austin S. Hemmelgarn
> <ahferroin7@gmail.com> wrote:
> 
>> This brings to mind another 'feature' of BTRFS that I came across recently,
>> namely that subvolumes that aren't explicitly mounted still show up as mount
>> points according to how most CLI tools differentiate what's a mount point.
>>
>> In particular, the st_dev field in stat() results for the subvolume differs
>> from the containing directory, and the f_fsid field in statvfs() results for
>> the subvolume differs from the containing directory (a side effect of the
>> differing st_dev field, which is part of what's used to calculate f_fsid on
>> Linux), which means the only way to know if something actually is a mount
>> point is to make this check, and then verify it in /proc/mounts or
>> /proc/self/mountinfo.
>>
>> That particular 'feature' means that GNU find, xargs, and du will never
>> cross subvolume boundaries if you tell them to stay on one filesystem, and
>> some other tools may misidentify where things are mounted.
> 
> 
> Elsewhere I brought up that mountinfo gives bogus subvol= information
> that conflicts with the subvolid= information, when doing bind mounts
> of directories on Btrfs. Trivially reproduced in my home directory:
> 
> 
> [chris@f26h ~]$ mkdir directory1
> [chris@f26h ~]$ mkdir directory2
> [chris@f26h ~]$ sudo mount -B directory2 directory1
> [chris@f26h ~]$ mount:
> /dev/nvme0n1p8 on /home type btrfs
> (rw,relatime,seclabel,ssd,space_cache,subvolid=257,subvol=/home)
> /dev/nvme0n1p8 on /home/chris/directory1 type btrfs
> (rw,relatime,seclabel,ssd,space_cache,subvolid=257,subvol=/home/chris/directory2)
> 
> 
> 
> The first mount item is correct, there is a subvolume home with
> subvolume ID of 257, mounted at /home.
> The second mount item considers directory2 a subvolume (wrong), with
> subvolid 257 (kinda right, it's on subvolid 257, but kinda wrong, it
> is not itself a subvolume).
Yeah, thankfully the only stuff I personally need to worry about this 
for doesn't care whether it's a regular mount, a bind mount, or an 
explicitly mounted subvolume, just that's it's actually a real mount 
point, not an implicit subvolume mount.

Ideally, all of this needs to be looked at eventually and cleaned up 
such that things are reasonably sane.  I understand why st_dev changes 
for each subvolume (they all show an inode number of 256, so things 
might break if st_dev matched too), but the rest of this is just 
unnecessary fallout from that.

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, back to index

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-11-04  0:15 updatedb does not index /home when /home is Btrfs Chris Murphy
2017-11-04  4:49 ` Adam Borowski
2017-11-04  6:26   ` Andrei Borzenkov
2017-11-04  7:05     ` Adam Borowski
2017-11-04 18:27       ` Andrei Borzenkov
2017-11-04 18:55         ` Chris Murphy
2017-11-04 19:37           ` Nicholas D Steeves
2017-11-05  8:01           ` Andrei Borzenkov
2017-11-06 13:51             ` Austin S. Hemmelgarn
2017-11-06 18:35               ` Chris Murphy
2017-11-06 19:44                 ` Austin S. Hemmelgarn
2017-11-05  7:47 ` Fixed subject: updatedb does not index separately mounted btrfs subvolumes Duncan
2017-11-05 14:02   ` Chris Murphy
2017-11-06  0:27   ` Peter Grandi

Linux-BTRFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-btrfs/0 linux-btrfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-btrfs linux-btrfs/ https://lore.kernel.org/linux-btrfs \
		linux-btrfs@vger.kernel.org
	public-inbox-index linux-btrfs

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-btrfs


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git