mlocate/updatedb and btrfs subvolume mounts

mlocate/updatedb and btrfs subvolume mounts

[G. Richard Bellamy]

I've just noticed that I'm having issues with finding files using
"locate" when those files are on btrfs subvolume mounts.

The issue is that updatedb cannot discern the difference between a
btrfs bind mount and btrfs subvolume [1][2]. This generally means that
if you're using btrfs subvolume mounts and updatedb at the same time,
and you want to index those subvolumes, you'll need to set
PRUNE_BIND_MOUNTS to 0 or "no". And then deal with all the cruft that
causes.

>From the bug above, you can see that the RedHat dev Michal Sekletar is
out of ideas. I'm not sure if he's reached out here or not... and if
not, he might welcome some help from the folks on this list.

Regrads,
Richard

[1] https://bugzilla.redhat.com/show_bug.cgi?id=906591#c3
[2] http://www.spinics.net/lists/linux-btrfs/msg42510.html

Re: mlocate/updatedb and btrfs subvolume mounts

[G. Richard Bellamy]

Is disabling PRUNE_BIND_MOUNTS for updatedb really the only solution here?

On Fri, Apr 3, 2015 at 1:07 PM, G. Richard Bellamy
<rbellamy@pteradigm.com> wrote:
> I've just noticed that I'm having issues with finding files using
> "locate" when those files are on btrfs subvolume mounts.
>
> The issue is that updatedb cannot discern the difference between a
> btrfs bind mount and btrfs subvolume [1][2]. This generally means that
> if you're using btrfs subvolume mounts and updatedb at the same time,
> and you want to index those subvolumes, you'll need to set
> PRUNE_BIND_MOUNTS to 0 or "no". And then deal with all the cruft that
> causes.
>
> From the bug above, you can see that the RedHat dev Michal Sekletar is
> out of ideas. I'm not sure if he's reached out here or not... and if
> not, he might welcome some help from the folks on this list.
>
> Regrads,
> Richard
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=906591#c3
> [2] http://www.spinics.net/lists/linux-btrfs/msg42510.html

Re: mlocate/updatedb and btrfs subvolume mounts

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 2191 bytes --]

I generally set up that kind of service on bind mounts of its own,
e.g.

        mount --bind / /.binds/root
        mount --bind /boot /.binds/root/boot
        mount --bind /home /.binds/root/home
        ... repeat for everything else you care about

then I tell backup / indexing software to process everything it finds
under /.binds/root.  If it's not possible to have the software drop
the /.binds/root prefix, run it chroot inside /.binds/root instead.
Tell locate to go ahead and process bind mounts since there's nothing
else it will have access to.

Since these are simple binds (not rbinds), they don't see any
other filesystems, period.  They do see btrfs subvolumes, though.

There are plenty of non-btrfs reasons to do things this way.  You can
let users mount SMB shares, FUSE filesystems, or whatever other insane
crap they like, and it won't affect locate or similar services (but make
sure your users have no access to /.binds--chmod it 700, or put them
under an isolated bind mount hierarchy of their own).

On Fri, Apr 03, 2015 at 01:07:37PM -0700, G. Richard Bellamy wrote:
> I've just noticed that I'm having issues with finding files using
> "locate" when those files are on btrfs subvolume mounts.
> 
> The issue is that updatedb cannot discern the difference between a
> btrfs bind mount and btrfs subvolume [1][2]. This generally means that
> if you're using btrfs subvolume mounts and updatedb at the same time,
> and you want to index those subvolumes, you'll need to set
> PRUNE_BIND_MOUNTS to 0 or "no". And then deal with all the cruft that
> causes.
> 
> >From the bug above, you can see that the RedHat dev Michal Sekletar is
> out of ideas. I'm not sure if he's reached out here or not... and if
> not, he might welcome some help from the folks on this list.
> 
> Regrads,
> Richard
> 
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=906591#c3
> [2] http://www.spinics.net/lists/linux-btrfs/msg42510.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: mlocate/updatedb and btrfs subvolume mounts

[Duncan]

Zygo Blaxell posted on Sun, 12 Apr 2015 16:11:07 -0400 as excerpted:

> Since these are simple binds (not rbinds), they don't see any other
> filesystems, period.  They do see btrfs subvolumes, though.

Tho... for systemd users...

Do note that systemd plays with the kernel's shared-subtree "private" 
default, switching it to shared for /.  The idea is that this makes 
containers "just work", since mounts on / get propagated into containers 
as well.  Unfortunately, the kernel default is private because that's the 
way the kernel behaved before the shared-subtree feature was introduced, 
and systemd changed that, breaking a lot of assumptions and scripts, 
including my backup script, along the way.

I don't use subvolumes, but do use a mount --bind of / elsewhere, so I 
can backup /, ALL of / (including subtrees hidden on the primary mount by 
overmounts), without any danger of crossing mount-points, since the bind-
mount was always private.  I've been doing this since well before I 
switched to either btrfs or systemd, and probably since before the kernel 
and mount introduced the shared-subtree feature, and it has worked very 
well.

But then systemd came around and decided that the "private" default-mount 
wasn't good enough for systemd, and that it was now going to default to 
shared.  Of course that broke existing solutions all over the place, but 
as we know, care to keep from breaking existing solutions isn't a 
priority, to say the least, with systemd, so...

So I had to add "private" to my / mount options, so mounting the backup 
partition on /mnt/bk wouldn't cause it to mount on the already bind-
mounted /mnt/rootbind that I was trying to backup, as well.

Link to a systemd-devel thread describing the change (with a link to the 
commit) and the systemd-ish way to undo it using a unit file (as opposed 
to the option I added to the fstab / entry instead).

http://lists.freedesktop.org/archives/systemd-devel/2013-
February/008558.html

Also see the mount(8) manpage under shared subtree operations, and the 
kernel documentation file in Documentation/filesystems/sharedsubtree.txt.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: mlocate/updatedb and btrfs subvolume mounts

[G. Richard Bellamy]

On Sun, Apr 12, 2015 at 9:12 PM, Duncan <1i5t5.duncan@cox.net> wrote:
>
> Zygo Blaxell posted on Sun, 12 Apr 2015 16:11:07 -0400 as excerpted:
>
> > Since these are simple binds (not rbinds), they don't see any other
> > filesystems, period.  They do see btrfs subvolumes, though.
>
> Tho... for systemd users...
>
> Do note that systemd plays with the kernel's shared-subtree "private"
> default, switching it to shared for /.

I like to think I know a bit about a thing or two. I've read both Zygo
and Duncan's probably cogent responses.

My brain hurts, and I am just not getting it.

If I am to understand correctly, and specific to my OP, the problem
with updatedb et al. on my system is that /, /home, /var, /opt and
/var/lib/libvirt are all subvolume mounts - which appear as bind
mounts. Thus the PRUNE_BIND_MOUNTS prunes ALL mounts, and updatedb
sees nothing.

Everything else is a blur when we start talking about "mount --bind /
/.binds/root"

Oh wait, maybe I get it now...

By running "mount --bind / /.binds/root" and running updatedb in
"/.binds/root" while still allowing it to see binds, I am certain to
get a an mlocate db, and it won't ever see anything outside that
explicit hierarchy.

Something like that?

-rb

Re: mlocate/updatedb and btrfs subvolume mounts

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 1887 bytes --]

On Mon, May 04, 2015 at 04:40:08PM -0700, G. Richard Bellamy wrote:
> On Sun, Apr 12, 2015 at 9:12 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> >
> > Zygo Blaxell posted on Sun, 12 Apr 2015 16:11:07 -0400 as excerpted:
> >
> > > Since these are simple binds (not rbinds), they don't see any other
> > > filesystems, period.  They do see btrfs subvolumes, though.
> >
> > Tho... for systemd users...
> >
> > Do note that systemd plays with the kernel's shared-subtree "private"
> > default, switching it to shared for /.
> 
> I like to think I know a bit about a thing or two. I've read both Zygo
> and Duncan's probably cogent responses.
> 
> My brain hurts, and I am just not getting it.
> 
> If I am to understand correctly, and specific to my OP, the problem
> with updatedb et al. on my system is that /, /home, /var, /opt and
> /var/lib/libvirt are all subvolume mounts - which appear as bind
> mounts. Thus the PRUNE_BIND_MOUNTS prunes ALL mounts, and updatedb
> sees nothing.
> 
> Everything else is a blur when we start talking about "mount --bind /
> /.binds/root"
> 
> Oh wait, maybe I get it now...
> 
> By running "mount --bind / /.binds/root" and running updatedb in
> "/.binds/root" while still allowing it to see binds, I am certain to
> get a an mlocate db, and it won't ever see anything outside that
> explicit hierarchy.
> 
> Something like that?

Something like that.  You might also have to do 
'chroot /.binds/root <your-updatedb-script-here>' in order to get path
names that make sense from / (i.e. to prevent updatedb from adding
'/.binds/root' to everything).

Bind mounts are awesome.  And also evil.  ;)

> -rb
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]