rsync -ax and subvolumes

rsync -ax and subvolumes

[Ulli Horlacher]

I run into the problem that "rsync -ax" sees btrfs subvolumes as "other
filesystems" and ignores them.

framstag@tux:~: rsync -h | grep one-file-system
 -x, --one-file-system       don't cross filesystem boundaries

How could I tell rsync to include btrfs subvolumes, but exclude
other mounted filesystems?

Do I have to write a rsync wrapper script and find all subvolumes to have
them included? Nasty task...

-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<20191010172011.GA3392@tik.uni-stuttgart.de>

Re: rsync -ax and subvolumes

[Kai Krakow]

Hi!

Am Do., 10. Okt. 2019 um 19:32 Uhr schrieb Ulli Horlacher
<framstag@rus.uni-stuttgart.de>:
>
> I run into the problem that "rsync -ax" sees btrfs subvolumes as "other
> filesystems" and ignores them.

I worked around it by mounting the btrfs-pool at a special directory:

mount -o subvolid=0 /dev/disk/by-label/rootfs /mnt/btrfs-pool

Then run rsync from /mnt/btrfs-pool without the "-x" flag. It won't
see mount points with foreign filesystems anyway below that mount
point.

> How could I tell rsync to include btrfs subvolumes, but exclude
> other mounted filesystems?

You can't, btrfs subvolumes are distinctive filesystems when viewed
from the API: they have their own set of inodes and thus their own
dev-id (and when that changes between two directories, you're crossing
a filesystem boundary).

> Do I have to write a rsync wrapper script and find all subvolumes to have
> them included? Nasty task...

The trick above should do, as long as you don't mind seeing another
view of your btrfs-pool opposed to your root mount view. It has the
benefit of seeing all your stuff. And it has the benefit of no longer
seeing API mounts there (like proc, sys, dev) and you can should rsync
without any more excludes needed.

Actually, you could also just bind-mount into /mnt/btrfs, bind-mounts
won't inherit other mounts but will still see pure subvolumes.

HTH
Kai

Re: rsync -ax and subvolumes

[Ulli Horlacher]

On Thu 2019-10-10 (20:47), Kai Krakow wrote:

> > I run into the problem that "rsync -ax" sees btrfs subvolumes as "other
> > filesystems" and ignores them.
> 
> I worked around it by mounting the btrfs-pool at a special directory:
> 
> mount -o subvolid=0 /dev/disk/by-label/rootfs /mnt/btrfs-pool

This is only possible by root, but not by regular users.
Yes, there are true multi-user systems still out there :-)


> Actually, you could also just bind-mount into /mnt/btrfs, bind-mounts
> won't inherit other mounts but will still see pure subvolumes.

Again, only possible by root.

-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<CAMthOuMV7MgB4b87RsijYr9e0UsjMUDNk+QRXeauFdb3cZcTjw@mail.gmail.com>

Re: rsync -ax and subvolumes

[Ulli Horlacher]

On Thu 2019-10-10 (20:47), Kai Krakow wrote:

> Actually, you could also just bind-mount into /mnt/btrfs, bind-mounts
> won't inherit other mounts but will still see pure subvolumes.

Next problem:

root@ptm1:~# sysinfo 
System:        Linux ptm1 4.4.180-94.100-default x86_64
Distribution:  SUSE Linux Enterprise Server 12 SP3

root@ptm1:~# mount | grep sda
/dev/sda2 on / type btrfs (rw,relatime,space_cache,subvolid=453,subvol=/@/.snapshots/128/snapshot)
/dev/sda2 on /.snapshots type btrfs (rw,relatime,space_cache,subvolid=270,subvol=/@/.snapshots)
/dev/sda2 on /opt type btrfs (rw,relatime,space_cache,subvolid=259,subvol=/@/opt)
/dev/sda2 on /var/log type btrfs (rw,relatime,space_cache,subvolid=264,subvol=/@/var/log)
/dev/sda2 on /srv type btrfs (rw,relatime,space_cache,subvolid=260,subvol=/@/srv)
/dev/sda2 on /var/tmp type btrfs (rw,relatime,space_cache,subvolid=267,subvol=/@/var/tmp)
/dev/sda2 on /var/spool type btrfs (rw,relatime,space_cache,subvolid=266,subvol=/@/var/spool)
/dev/sda2 on /usr/local type btrfs (rw,relatime,space_cache,subvolid=262,subvol=/@/usr/local)
/dev/sda2 on /var/opt type btrfs (rw,relatime,space_cache,subvolid=265,subvol=/@/var/opt)
/dev/sda2 on /home type btrfs (rw,relatime,space_cache,subvolid=258,subvol=/@/home)
/dev/sda2 on /var/lib/machines type btrfs (rw,relatime,space_cache,subvolid=1235,subvol=/@/var/lib/machines)
/dev/sda2 on /tmp type btrfs (rw,relatime,space_cache,subvolid=261,subvol=/@/tmp)
/dev/sda2 on /var/crash type btrfs (rw,relatime,space_cache,subvolid=263,subvol=/@/var/crash)

root@ptm1:~# mount -o bind / /mnt/tmp

root@ptm1:~# find /opt | wc -l
564

root@ptm1:~# find /mnt/tmp/opt | wc -l
1


I want to copy everything from /dev/sda2 with rsync, but not from other devices.

rsynv -avH  / /mnt/usb # copies also /proc /dev etc
rsynv -avHx / /mnt/usb # copies ONLY / but not /opt /var/log etc

And yes, it must be rsync, because I run this command often, for syncing.

-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<CAMthOuMV7MgB4b87RsijYr9e0UsjMUDNk+QRXeauFdb3cZcTjw@mail.gmail.com>

Re: rsync -ax and subvolumes

[Austin S. Hemmelgarn]

On 2019-10-10 17:21, Ulli Horlacher wrote:
> On Thu 2019-10-10 (20:47), Kai Krakow wrote:
> 
>>> I run into the problem that "rsync -ax" sees btrfs subvolumes as "other
>>> filesystems" and ignores them.
>>
>> I worked around it by mounting the btrfs-pool at a special directory:
>>
>> mount -o subvolid=0 /dev/disk/by-label/rootfs /mnt/btrfs-pool
> 
> This is only possible by root, but not by regular users.
> Yes, there are true multi-user systems still out there :-)
And that is what `sudo` or capabilities are for.

`sudo` will even let you get as specific as command line arguments, so 
you can specify an exact mount command that can be run (including 
ensuring that it's read-only) and an exact unmount command that can be run.

If you want to go the capabilities route, you'll need CAP_MOUNT.  This 
is a lot riskier than using `sudo` though.

That said, if you really want _everything_, you're going to need to 
either be root anyway, or have the CAP_DAC_READ_SEARCH (or 
CAP_DAC_OVERRIDE) capability, because there are some files you just 
won't be able to read otherwise (at minimum the contents of `/root` and 
any properly secured authentication logs in `/var/log`, as well as 
possibly other things under `/var` and possibly some things under `/etc`).
> 
> 
>> Actually, you could also just bind-mount into /mnt/btrfs, bind-mounts
>> won't inherit other mounts but will still see pure subvolumes.
> 
> Again, only possible by root.
> 

Re: rsync -ax and subvolumes

[Andrei Borzenkov]

11.10.2019 9:15, Ulli Horlacher пишет:
> On Thu 2019-10-10 (20:47), Kai Krakow wrote:
> 
>> Actually, you could also just bind-mount into /mnt/btrfs, bind-mounts
>> won't inherit other mounts but will still see pure subvolumes.
> 
> Next problem:
> 
> root@ptm1:~# sysinfo 
> System:        Linux ptm1 4.4.180-94.100-default x86_64
> Distribution:  SUSE Linux Enterprise Server 12 SP3
> 
> root@ptm1:~# mount | grep sda
> /dev/sda2 on / type btrfs (rw,relatime,space_cache,subvolid=453,subvol=/@/.snapshots/128/snapshot)
> /dev/sda2 on /.snapshots type btrfs (rw,relatime,space_cache,subvolid=270,subvol=/@/.snapshots)
> /dev/sda2 on /opt type btrfs (rw,relatime,space_cache,subvolid=259,subvol=/@/opt)
> /dev/sda2 on /var/log type btrfs (rw,relatime,space_cache,subvolid=264,subvol=/@/var/log)
> /dev/sda2 on /srv type btrfs (rw,relatime,space_cache,subvolid=260,subvol=/@/srv)
> /dev/sda2 on /var/tmp type btrfs (rw,relatime,space_cache,subvolid=267,subvol=/@/var/tmp)
> /dev/sda2 on /var/spool type btrfs (rw,relatime,space_cache,subvolid=266,subvol=/@/var/spool)
> /dev/sda2 on /usr/local type btrfs (rw,relatime,space_cache,subvolid=262,subvol=/@/usr/local)
> /dev/sda2 on /var/opt type btrfs (rw,relatime,space_cache,subvolid=265,subvol=/@/var/opt)
> /dev/sda2 on /home type btrfs (rw,relatime,space_cache,subvolid=258,subvol=/@/home)
> /dev/sda2 on /var/lib/machines type btrfs (rw,relatime,space_cache,subvolid=1235,subvol=/@/var/lib/machines)
> /dev/sda2 on /tmp type btrfs (rw,relatime,space_cache,subvolid=261,subvol=/@/tmp)
> /dev/sda2 on /var/crash type btrfs (rw,relatime,space_cache,subvolid=263,subvol=/@/var/crash)
> 
> root@ptm1:~# mount -o bind / /mnt/tmp
> 
> root@ptm1:~# find /opt | wc -l
> 564
> 
> root@ptm1:~# find /mnt/tmp/opt | wc -l
> 1
> 

Of course. /opt is mount point. You will need to replicate the same
mount hierarchy under /mnt/tmp.

Re: rsync -ax and subvolumes

[Kai Krakow]

Am Fr., 11. Okt. 2019 um 08:17 Uhr schrieb Ulli Horlacher
<framstag@rus.uni-stuttgart.de>:
>
> On Thu 2019-10-10 (20:47), Kai Krakow wrote:
>
> > Actually, you could also just bind-mount into /mnt/btrfs, bind-mounts
> > won't inherit other mounts but will still see pure subvolumes.
>
> Next problem:

The other problem first: As already pointed out, there's "sudo", and
for the job you're trying to do, you probably need root permissions
anyway. OTOH, I don't see the problem why you don't just statically
mount subvolid=0 into /mnt/btrfs-pool permanently. Either that way, or
you could use automount. It's easy with systemd:

$ grep btrfs-pool /etc/fstab
LABEL=system /mnt/btrfs-pool btrfs
noauto,compress=zstd,subvolid=0,x-systemd.automount

You could create an intermediate directory in /mnt to allow one
specific user only:

/mnt/for-user/btrfs-pool

Then adjust chown/chmod on /mnt/for-user to limit it to just one user
(or group if you want), that is: remove rwx from the others.

Apparently, with btrfs, the "ro" flag afaik is not per subvolume
mount. But I'm using systemd to limit access to read-only for the
service doing the backup:

$ egrep '^(Read|Privat|Protect)' /etc/systemd/system/system-backup.service
ProtectSystem=strict
PrivateTmp=yes
ReadWriteDirectories=/root
ReadWriteDirectories=/srv/laniakea/backups/jupiter.sol.local.borg
ReadOnlyDirectories=/mnt/btrfs-pool

I'm using borg instead of rsync (because it is much faster and storage
is used much more efficient), so it needs write access to /root. If
you're using rsync, you don't need that. I used rsync a few years back
for the same job, it was just too slow and resource intensive.

Now let's get to your job:

> root@ptm1:~# mount | grep sda
> /dev/sda2 on / type btrfs (rw,relatime,space_cache,subvolid=453,subvol=/@/.snapshots/128/snapshot)
> /dev/sda2 on /.snapshots type btrfs (rw,relatime,space_cache,subvolid=270,subvol=/@/.snapshots)
> /dev/sda2 on /opt type btrfs (rw,relatime,space_cache,subvolid=259,subvol=/@/opt)
> /dev/sda2 on /var/log type btrfs (rw,relatime,space_cache,subvolid=264,subvol=/@/var/log)
> /dev/sda2 on /srv type btrfs (rw,relatime,space_cache,subvolid=260,subvol=/@/srv)
> /dev/sda2 on /var/tmp type btrfs (rw,relatime,space_cache,subvolid=267,subvol=/@/var/tmp)
> /dev/sda2 on /var/spool type btrfs (rw,relatime,space_cache,subvolid=266,subvol=/@/var/spool)
> /dev/sda2 on /usr/local type btrfs (rw,relatime,space_cache,subvolid=262,subvol=/@/usr/local)
> /dev/sda2 on /var/opt type btrfs (rw,relatime,space_cache,subvolid=265,subvol=/@/var/opt)
> /dev/sda2 on /home type btrfs (rw,relatime,space_cache,subvolid=258,subvol=/@/home)
> /dev/sda2 on /var/lib/machines type btrfs (rw,relatime,space_cache,subvolid=1235,subvol=/@/var/lib/machines)
> /dev/sda2 on /tmp type btrfs (rw,relatime,space_cache,subvolid=261,subvol=/@/tmp)
> /dev/sda2 on /var/crash type btrfs (rw,relatime,space_cache,subvolid=263,subvol=/@/var/crash)

Ah, that funny OpenSuSE structure...


> root@ptm1:~# mount -o bind / /mnt/tmp

Bind mounting the "/" mount namespace will exclude all sub mounts, and
OpenSuSE uses a lot. It would work if subvols would be embedded into
the normal FS hierarchy.


> root@ptm1:~# find /opt | wc -l
> 564
>
> root@ptm1:~# find /mnt/tmp/opt | wc -l
> 1

Expected given your mtab.


> I want to copy everything from /dev/sda2 with rsync, but not from other devices.
>
> rsynv -avH  / /mnt/usb # copies also /proc /dev etc
> rsynv -avHx / /mnt/usb # copies ONLY / but not /opt /var/log etc

Try this, pretending you mounted subvolid=0 to /mnt/btrfs-pool:

# NOTE: do not use "-x" here:
rsync -av $(findmnt -nr /dev/sda2 | perl -ne'/subvol=([^,]+)/ && \
  print "/mnt/btrfs-pool$1"') /srv/backups/test-backup/

Or:

# NOTE: you should use "-x" here:
rsync -avx $(findmnt -nr /dev/sda2 | awk '{ print $1 }' \
  /srv/backups/test-backup/

The first example may explode if your mountpoints contain funny
characters (like "," or spaces). I didn't really test, I did just a
quick rsync run.

Both examples may have surprising consequences if source and
destination is the same device. You have been warned. But that's not
really different from how you used it.


> And yes, it must be rsync, because I run this command often, for syncing.

That's not exactly explaining your use-case: I don't see the
correlation between "often" and "rsync", but I guess you mean the
process should be fast because you want to run the command often. But
then again, there's faster software than rsync.


HTH
Kai