why is the same mount point repeatedly mounted in nested manner?

why is the same mount point repeatedly mounted in nested manner?

[Dave T]

I recently switched from snapper and snapsync to btrbk, which I
generally prefer. However, I am running into one issue.

Background - from https://digint.ch/btrbk/doc/faq.html
Btrbk is designed to operate on the subvolumes within a root
subvolume. The author recommend booting linux from a btrfs subvolume,
and using the btrfs root only as a container for subvolumes (i.e. NOT
booting from "subvolid=5").

That's exactly what I'm doing.

The key point is that btrbk requires mounting the toplevel subvolume
to perform its tasks.

I'm using btrbk via a systemd timer. I have a daily and hourly timer
set up. Each timer starts by mounting the btrfs root, performing the
btrbk task, and unmounting the btrfs root.

I create hourly snapshots and on a daily basis I use btrbk to transfer
those to two different USB HDD's as well as to a remote server via
SSH.

Here's the problem. I now end up (after some time) with a nested list
of mounts for one particular mount point as shown below. I don't
understand why or how this is happening. It does have side effects
(disk access can hang). The apparent "cure" is to restart the nfs
server service, but I'm still trying to understand the issue fully.

# cat /etc/fstab
UUID=28D099A-9D92-487C-8113-A231CAD0EEF2  /     btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvol=/@btrtop/snapshot
0 0 #mounts my root filesystem
UUID=28D099A-9D92-487C-8113-A231CAD0EEF2  /mnt/btrtop/root  btrfs
noauto,nofail,rw,noatime,nodiratime,compress=lzo,space_cache    0 0
#mounts the top btrfs volume for btrbk access to all snapshots
/var/cache/pacman       /srv/nfs/var/cache/pacman       none  bind  0 0

# findmnt -t btrfs
TARGET                                      SOURCE
                                       FSTYPE OPTIONS
/
/dev/mapper/xyz[/@btrtop/snapshot]                     btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
├─/srv/nfs/var/cache/pacman
/dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
│ └─/srv/nfs/var/cache/pacman
/dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
│   └─/srv/nfs/var/cache/pacman
/dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
│     └─/srv/nfs/var/cache/pacman
/dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
│       └─/srv/nfs/var/cache/pacman
/dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
│         └─/srv/nfs/var/cache/pacman
/dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
│           └─/srv/nfs/var/cache/pacman
/dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
│             └─/srv/nfs/var/cache/pacman
/dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
├─/var/cache/pacman
/dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
│ └─/var/cache/pacman
/dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
│   └─/var/cache/pacman
/dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
│     └─/var/cache/pacman
/dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
│       └─/var/cache/pacman
/dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
│         └─/var/cache/pacman
/dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
│           └─/var/cache/pacman
/dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot

Re: why is the same mount point repeatedly mounted in nested manner?

[Phillip Susi]

Dave T <davestechshop@gmail.com> writes:

> I'm using btrbk via a systemd timer. I have a daily and hourly timer
> set up. Each timer starts by mounting the btrfs root, performing the
> btrbk task, and unmounting the btrfs root.

It looks like the unmounting is not being done, so it just keeps
mounting it over again on the next backup.

Re: why is the same mount point repeatedly mounted in nested manner?

[Dave T]

On Thu, Aug 5, 2021 at 1:49 PM Phillip Susi <phill@thesusis.net> wrote:
>
>
> Dave T <davestechshop@gmail.com> writes:
>
> > I'm using btrbk via a systemd timer. I have a daily and hourly timer
> > set up. Each timer starts by mounting the btrfs root, performing the
> > btrbk task, and unmounting the btrfs root.
>
> It looks like the unmounting is not being done, so it just keeps
> mounting it over again on the next backup.
>

Not exactly... that path should not be unmounted. I do not mount that
location explicitly for the btrbk tasks. It is mounted when the server
boots up with the bind mount line I showed in fstab:

/var/cache/pacman       /srv/nfs/var/cache/pacman       none  bind  0 0

It should not be unmounted (or remounted) just because the top level
btrfs volume is (un)mounted for btrbk tasks. That's the part that is
confusing me.

There is no command associated with my btrbk tasks that mounts
/srv/nfs/var/cache/pacman (or /var/cache/pacman). There's no other
entry in fstab for it except the bind mount I showed.

Re: why is the same mount point repeatedly mounted in nested manner?

[Adam Borowski]

On Thu, Aug 05, 2021 at 04:41:03PM -0400, Dave T wrote:
> On Thu, Aug 5, 2021 at 1:49 PM Phillip Susi <phill@thesusis.net> wrote:
> > Dave T <davestechshop@gmail.com> writes:
> >
> > > I'm using btrbk via a systemd timer. I have a daily and hourly timer
> > > set up. Each timer starts by mounting the btrfs root, performing the
> > > btrbk task, and unmounting the btrfs root.

> Not exactly... that path should not be unmounted. I do not mount that
> location explicitly for the btrbk tasks. It is mounted when the server
> boots up with the bind mount line I showed in fstab:
> 
> /var/cache/pacman       /srv/nfs/var/cache/pacman       none  bind  0 0
> 
> It should not be unmounted (or remounted) just because the top level
> btrfs volume is (un)mounted for btrbk tasks. That's the part that is
> confusing me.
> 
> There is no command associated with my btrbk tasks that mounts
> /srv/nfs/var/cache/pacman (or /var/cache/pacman). There's no other
> entry in fstab for it except the bind mount I showed.

I've seen systemd randomly mount and/or unmount filesystems listed in fstab
when their state somehow differs from what systemd thinks should be (which
can happen either by you or a program doing a mount, or due to systemd
misreading the state reported by the kernel).

So I wonder, can you test with an init/rc other than systemd?


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ If you ponder doing what Jesus did, remember than flipping tables
⢿⡄⠘⠷⠚⠋⠀ and chasing people with a whip is a prime choice.
⠈⠳⣄⠀⠀⠀⠀

Re: why is the same mount point repeatedly mounted in nested manner?

[Chris Murphy]

On Sat, Aug 7, 2021 at 6:25 PM Adam Borowski <kilobyte@angband.pl> wrote:
>
> On Thu, Aug 05, 2021 at 04:41:03PM -0400, Dave T wrote:
> > On Thu, Aug 5, 2021 at 1:49 PM Phillip Susi <phill@thesusis.net> wrote:
> > > Dave T <davestechshop@gmail.com> writes:
> > >
> > > > I'm using btrbk via a systemd timer. I have a daily and hourly timer
> > > > set up. Each timer starts by mounting the btrfs root, performing the
> > > > btrbk task, and unmounting the btrfs root.
>
> > Not exactly... that path should not be unmounted. I do not mount that
> > location explicitly for the btrbk tasks. It is mounted when the server
> > boots up with the bind mount line I showed in fstab:
> >
> > /var/cache/pacman       /srv/nfs/var/cache/pacman       none  bind  0 0
> >
> > It should not be unmounted (or remounted) just because the top level
> > btrfs volume is (un)mounted for btrbk tasks. That's the part that is
> > confusing me.
> >
> > There is no command associated with my btrbk tasks that mounts
> > /srv/nfs/var/cache/pacman (or /var/cache/pacman). There's no other
> > entry in fstab for it except the bind mount I showed.
>
> I've seen systemd randomly mount and/or unmount filesystems listed in fstab
> when their state somehow differs from what systemd thinks should be (which
> can happen either by you or a program doing a mount, or due to systemd
> misreading the state reported by the kernel).
>
> So I wonder, can you test with an init/rc other than systemd?

We just need logs, systemd will put a message that it's mounting
something into the journal: journalctl -b --no-hostname | grep mount


-- 
Chris Murphy

Re: why is the same mount point repeatedly mounted in nested manner?

[devel]

On 05/08/2021 17:46, Dave T wrote:
> I recently switched from snapper and snapsync to btrbk, which I
> generally prefer. However, I am running into one issue.
>
> Background - from https://digint.ch/btrbk/doc/faq.html
> Btrbk is designed to operate on the subvolumes within a root
> subvolume. The author recommend booting linux from a btrfs subvolume,
> and using the btrfs root only as a container for subvolumes (i.e. NOT
> booting from "subvolid=5").
>
> That's exactly what I'm doing.
>
> The key point is that btrbk requires mounting the toplevel subvolume
> to perform its tasks.
>
> I'm using btrbk via a systemd timer. I have a daily and hourly timer
> set up. Each timer starts by mounting the btrfs root, performing the
> btrbk task, and unmounting the btrfs root.
>
> I create hourly snapshots and on a daily basis I use btrbk to transfer
> those to two different USB HDD's as well as to a remote server via
> SSH.
>
> Here's the problem. I now end up (after some time) with a nested list
> of mounts for one particular mount point as shown below. I don't
> understand why or how this is happening. It does have side effects
> (disk access can hang). The apparent "cure" is to restart the nfs
> server service, but I'm still trying to understand the issue fully.
>
> # cat /etc/fstab
> UUID=28D099A-9D92-487C-8113-A231CAD0EEF2  /     btrfs
> rw,noatime,nodiratime,compress=lzo,space_cache,subvol=/@btrtop/snapshot
> 0 0 #mounts my root filesystem
> UUID=28D099A-9D92-487C-8113-A231CAD0EEF2  /mnt/btrtop/root  btrfs
> noauto,nofail,rw,noatime,nodiratime,compress=lzo,space_cache    0 0
> #mounts the top btrfs volume for btrbk access to all snapshots
> /var/cache/pacman       /srv/nfs/var/cache/pacman       none  bind  0 0
>
> # findmnt -t btrfs
> TARGET                                      SOURCE
>                                        FSTYPE OPTIONS
> /
> /dev/mapper/xyz[/@btrtop/snapshot]                     btrfs
> rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> ├─/srv/nfs/var/cache/pacman
> /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> │ └─/srv/nfs/var/cache/pacman
> /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> │   └─/srv/nfs/var/cache/pacman
> /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> │     └─/srv/nfs/var/cache/pacman
> /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> │       └─/srv/nfs/var/cache/pacman
> /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> │         └─/srv/nfs/var/cache/pacman
> /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> │           └─/srv/nfs/var/cache/pacman
> /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> │             └─/srv/nfs/var/cache/pacman
> /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> ├─/var/cache/pacman
> /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> │ └─/var/cache/pacman
> /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> │   └─/var/cache/pacman
> /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> │     └─/var/cache/pacman
> /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> │       └─/var/cache/pacman
> /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> │         └─/var/cache/pacman
> /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> │           └─/var/cache/pacman
> /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot

Try mounting the subvolume with it's subvolume ID. System only generates
unit files from the fstab it does not follow them , so if you are vague
in your fstab the systemd unit files will also be vague.

-- 
==

D Alexander

Re: why is the same mount point repeatedly mounted in nested manner?

[Dave T]

On Sun, Aug 8, 2021 at 8:10 AM <devel@roosoft.ltd.uk> wrote:
>
> On 05/08/2021 17:46, Dave T wrote:
> > I recently switched from snapper and snapsync to btrbk, which I
> > generally prefer. However, I am running into one issue.
> >
> > Background - from https://digint.ch/btrbk/doc/faq.html
> > Btrbk is designed to operate on the subvolumes within a root
> > subvolume. The author recommend booting linux from a btrfs subvolume,
> > and using the btrfs root only as a container for subvolumes (i.e. NOT
> > booting from "subvolid=5").
> >
> > That's exactly what I'm doing.
> >
> > The key point is that btrbk requires mounting the toplevel subvolume
> > to perform its tasks.
> >
> > I'm using btrbk via a systemd timer. I have a daily and hourly timer
> > set up. Each timer starts by mounting the btrfs root, performing the
> > btrbk task, and unmounting the btrfs root.
> >
> > I create hourly snapshots and on a daily basis I use btrbk to transfer
> > those to two different USB HDD's as well as to a remote server via
> > SSH.
> >
> > Here's the problem. I now end up (after some time) with a nested list
> > of mounts for one particular mount point as shown below. I don't
> > understand why or how this is happening. It does have side effects
> > (disk access can hang). The apparent "cure" is to restart the nfs
> > server service, but I'm still trying to understand the issue fully.
> >
> > # cat /etc/fstab
> > UUID=28D099A-9D92-487C-8113-A231CAD0EEF2  /     btrfs
> > rw,noatime,nodiratime,compress=lzo,space_cache,subvol=/@btrtop/snapshot
> > 0 0 #mounts my root filesystem
> > UUID=28D099A-9D92-487C-8113-A231CAD0EEF2  /mnt/btrtop/root  btrfs
> > noauto,nofail,rw,noatime,nodiratime,compress=lzo,space_cache    0 0
> > #mounts the top btrfs volume for btrbk access to all snapshots
> > /var/cache/pacman       /srv/nfs/var/cache/pacman       none  bind  0 0
> >
> > # findmnt -t btrfs
> > TARGET                                      SOURCE
> >                                        FSTYPE OPTIONS
> > /
> > /dev/mapper/xyz[/@btrtop/snapshot]                     btrfs
> > rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> > ├─/srv/nfs/var/cache/pacman
> > /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> > rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> > │ └─/srv/nfs/var/cache/pacman
> > /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> > rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> > │   └─/srv/nfs/var/cache/pacman
> > /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> > rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> > │     └─/srv/nfs/var/cache/pacman
> > /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> > rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> > │       └─/srv/nfs/var/cache/pacman
> > /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> > rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> > │         └─/srv/nfs/var/cache/pacman
> > /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> > rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> > │           └─/srv/nfs/var/cache/pacman
> > /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> > rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> > │             └─/srv/nfs/var/cache/pacman
> > /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> > rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> > ├─/var/cache/pacman
> > /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> > rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> > │ └─/var/cache/pacman
> > /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> > rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> > │   └─/var/cache/pacman
> > /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> > rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> > │     └─/var/cache/pacman
> > /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> > rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> > │       └─/var/cache/pacman
> > /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> > rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> > │         └─/var/cache/pacman
> > /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> > rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
> > │           └─/var/cache/pacman
> > /dev/mapper/xyz[/@btrtop/snapshot/var/cache/pacman]    btrfs
> > rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=3194,subvol=/@btrtop/snapshot
>
> Try mounting the subvolume with it's subvolume ID. System only generates
> unit files from the fstab it does not follow them , so if you are vague
> in your fstab the systemd unit files will also be vague.

Thank you for the tip. I appreciate your interest in my issue.
However, I don't fully understand what to change.
Here are the relevant lines from my fstab. I added line numbers
because the lines will get wrapped in email.  I don't see what part of
this is vague:

1. # cat /etc/fstab
2. UUID=28D099A-9D92-487C-8113-A231CAD0EEF2  /     btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvol=/@btrtop/snapshot
0 0
3. UUID=28D099A-9D92-487C-8113-A231CAD0EEF2  /mnt/btrtop/root  btrfs
noauto,nofail,rw,noatime,nodiratime,compress=lzo,space_cache    0 0
4. /var/cache/pacman       /srv/nfs/var/cache/pacman       none  bind  0 0

The path /var/cache/pacman is not a subvolume, but it resides on btrfs
subvolume @btrtop/snapshot. @btrtop/snapshot is normally mounted at
"/" but for btrfs tasks, it is also mounted at /mnt/btrtop/root. This
additional mount operation seems to be causing these nested mounts of
my bind mount for  /srv/nfs/var/cache/pacman .

P.S. I cannot test without using systemd. (I'm not even sure I
remember how to use a non-systemd distro anymore!)

Re: why is the same mount point repeatedly mounted in nested manner?

[Goffredo Baroncelli]

On 8/8/21 9:48 PM, Dave T wrote:
> On Sun, Aug 8, 2021 at 8:10 AM <devel@roosoft.ltd.uk> wrote:
>>
>> On 05/08/2021 17:46, Dave T wrote:
[...]
>>
>> Try mounting the subvolume with it's subvolume ID. System only generates
>> unit files from the fstab it does not follow them , so if you are vague
>> in your fstab the systemd unit files will also be vague.
> 
> Thank you for the tip. I appreciate your interest in my issue.
> However, I don't fully understand what to change.

I think that Alexander is suggesting to add something like ',subvolid=5' to the line of fstab where /mnt/btrtop/root is mounted.

I add that if it is a systemd bug, it would help to look at the .mount files generated by systemd:

$ sudo ls /run/systemd/generator/*.mount
....

Can you share the content of the  systemd units where you ask to mount '/mnt/btrtop/root' ? It could help the diagnose.

And finally, what happens if you mount/unmount from command line (e.g. manually) /mnt/btrtop/root ?

BR

> Here are the relevant lines from my fstab. I added line numbers
> because the lines will get wrapped in email.  I don't see what part of
> this is vague:
> 
> 1. # cat /etc/fstab
> 2. UUID=28D099A-9D92-487C-8113-A231CAD0EEF2  /     btrfs
> rw,noatime,nodiratime,compress=lzo,space_cache,subvol=/@btrtop/snapshot
> 0 0
> 3. UUID=28D099A-9D92-487C-8113-A231CAD0EEF2  /mnt/btrtop/root  btrfs
> noauto,nofail,rw,noatime,nodiratime,compress=lzo,space_cache    0 0
> 4. /var/cache/pacman       /srv/nfs/var/cache/pacman       none  bind  0 0

I don't know if it matters, but why you set as 'none' the filesystem type ? However according to askubuntu/stackoverflow it seem the right thing to do...

> 
> The path /var/cache/pacman is not a subvolume, but it resides on btrfs
> subvolume @btrtop/snapshot. @btrtop/snapshot is normally mounted at
> "/" but for btrfs tasks, it is also mounted at /mnt/btrtop/root. This
> additional mount operation seems to be causing these nested mounts of
> my bind mount for  /srv/nfs/var/cache/pacman .
> 
> P.S. I cannot test without using systemd. (I'm not even sure I
> remember how to use a non-systemd distro anymore!)
> 


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: why is the same mount point repeatedly mounted in nested manner?

[Dave T]

On Mon, Aug 9, 2021 at 3:29 PM Goffredo Baroncelli <kreijack@libero.it> wrote:
>
> On 8/8/21 9:48 PM, Dave T wrote:
> > On Sun, Aug 8, 2021 at 8:10 AM <devel@roosoft.ltd.uk> wrote:
> >>
> >> On 05/08/2021 17:46, Dave T wrote:
> [...]
> >>
> >> Try mounting the subvolume with it's subvolume ID. System only generates
> >> unit files from the fstab it does not follow them , so if you are vague
> >> in your fstab the systemd unit files will also be vague.
> >
> > Thank you for the tip. I appreciate your interest in my issue.
> > However, I don't fully understand what to change.
>
> I think that Alexander is suggesting to add something like ',subvolid=5' to the line of fstab where /mnt/btrtop/root is mounted.

I see. Thank you (and Alexander). I will add that! EDIT: I just added
it and tested mounting, umounting. It seems to work exactly the same
as previously. findmnt now reports:

├─/mnt/btrtop/root                       /dev/mapper/rootluks
                               btrfs
rw,noatime,nodiratime,compress=lzo,space_cache,subvolid=5,subvol=/

>
> I add that if it is a systemd bug, it would help to look at the .mount files generated by systemd:
>
> $ sudo ls /run/systemd/generator/*.mount
>
> Can you share the content of the  systemd units where you ask to mount '/mnt/btrtop/root' ? It could help the diagnose.
>
Thank you. Good tip. I had not looked at those files before. Here are
the generator files (before my change to add ,subvolid=5).

# cat /run/systemd/generator/-.mount
# Automatically generated by systemd-fstab-generator
[Unit]
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
SourcePath=/etc/fstab
Before=local-fs.target
After=blockdev@dev-disk-by\x2duuid-28D099A\x2d49D92\x2d4487C\x2d48113\x2d4A231CAD0EEF2.target
[Mount]
Where=/
What=/dev/disk/by-uuid/28D099A-9D92-487C-8113-A231CAD0EEF2
Type=btrfs
Options=rw,noatime,nodiratime,compress=lzo,space_cache,subvol=/@btrtop/snapshot



# cat /run/systemd/generator/mnt-btrtop-root.mount
# Automatically generated by systemd-fstab-generator
[Unit]
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
SourcePath=/etc/fstab
After=blockdev@dev-disk-by\x2duuid-28D099A\x2d49D92\x2d4487C\x2d48113\x2d4A231CAD0EEF2.target
[Mount]
Where=/mnt/btrtop/root
What=/dev/disk/by-uuid/28D099A-9D92-487C-8113-A231CAD0EEF2
Type=btrfs
Options=noauto,nofail,rw,noatime,nodiratime,compress=lzo,space_cache




# cat /run/systemd/generator/srv-nfs-var-cache-pacman.mount
# Automatically generated by systemd-fstab-generator
[Unit]
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
SourcePath=/etc/fstab
Before=local-fs.target
[Mount]
Where=/srv/nfs/var/cache/pacman
What=/var/cache/pacman
Type=none
Options=bind


> And finally, what happens if you mount/unmount from command line (e.g. manually) /mnt/btrtop/root ?

It works as expected. It mounts and unmounts and there are no errors
and the extra nested mounting doesn't occur. I just ran through a
testing cycle of multiple mounts and unmounts to verify this.

Also, in recent days I stopped mounting and umounting /mnt/btrtop/root
and just left it mounted all the time. However, when checking today, I
still found a nested mount:

├─/srv/nfs/var/cache/pacman
│
│ └─/srv/nfs/var/cache/pacman


>
> BR
>
> > Here are the relevant lines from my fstab. I added line numbers
> > because the lines will get wrapped in email.  I don't see what part of
> > this is vague:
> >
> > 1. # cat /etc/fstab
> > 2. UUID=28D099A-9D92-487C-8113-A231CAD0EEF2  /     btrfs
> > rw,noatime,nodiratime,compress=lzo,space_cache,subvol=/@btrtop/snapshot
> > 0 0
> > 3. UUID=28D099A-9D92-487C-8113-A231CAD0EEF2  /mnt/btrtop/root  btrfs
> > noauto,nofail,rw,noatime,nodiratime,compress=lzo,space_cache    0 0
> > 4. /var/cache/pacman       /srv/nfs/var/cache/pacman       none  bind  0 0
>
> I don't know if it matters, but why you set as 'none' the filesystem type ? However according to askubuntu/stackoverflow it seem the right thing to do...

That is the way I have always done bind mounts.

I can send more complete systemd logs if needed, but this is what it
looked like hourly (before I stopped unmounting):

Aug 03 15:00:12 systemd[1]: mnt-btrtop-root.mount: Deactivated successfully.
Aug 03 16:00:06 btrbk_run.sh[2376252]: mounting btrbk btrtop volumes...
Aug 03 16:00:06 btrbk_run.sh[2376252]: OK: mounted [/mnt/btrtop/root] (1 of 3)
Aug 03 16:00:06 btrbk_run.sh[2376252]: OK: mounted [/mnt/btrtop/home] (2 of 3)
Aug 03 16:00:06 btrbk_run.sh[2376252]: OK: mounted mount
[/mnt/btrtop/user] (3 of 3)
Aug 03 16:00:06 btrbk_run.sh[2376252]: Finished mounting btrbk btrtop volumes.
Aug 03 16:00:11 btrbk_run.sh[2376356]: un-mounting btrbk btrtop volumes...
Aug 03 16:00:11 btrbk_run.sh[2376356]: OK: un-mounted
[/mnt/btrtop/user] (1 of 3)
Aug 03 16:00:11 systemd[1]: mnt-btrtop-user.mount: Deactivated successfully.
Aug 03 16:00:11 btrbk_run.sh[2376356]: OK: un-mounted
[/mnt/btrtop/home] (2 of 3)
Aug 03 16:00:11 systemd[1]: mnt-btrtop-home.mount: Deactivated successfully.
Aug 03 16:00:11 btrbk_run.sh[2376356]: OK: un-mounted
[/mnt/btrtop/root] (3 of 3)
Aug 03 16:00:11 btrbk_run.sh[2376356]: Finished un-mounting btrbk
btrtop volumes.
Aug 03 16:00:11 systemd[1]: mnt-btrtop-root.mount: Deactivated successfully.
Aug 03 17:00:06 btrbk_run.sh[2387185]: mounting btrbk btrtop volumes...
Aug 03 17:00:06 btrbk_run.sh[2387185]: OK: mounted [/mnt/btrtop/root] (1 of 3)
Aug 03 17:00:06 btrbk_run.sh[2387185]: OK: mounted [/mnt/btrtop/home] (2 of 3)
Aug 03 17:00:06 btrbk_run.sh[2387185]: OK: mounted mount
[/mnt/btrtop/user] (3 of 3)
Aug 03 17:00:06 btrbk_run.sh[2387185]: Finished mounting btrbk btrtop volumes.
Aug 03 17:00:10 btrbk_run.sh[2387275]: un-mounting btrbk btrtop volumes...
Aug 03 17:00:10 btrbk_run.sh[2387275]: OK: un-mounted
[/mnt/btrtop/user] (1 of 3)
Aug 03 17:00:10 systemd[1]: mnt-btrtop-user.mount: Deactivated successfully.
Aug 03 17:00:10 btrbk_run.sh[2387275]: OK: un-mounted
[/mnt/btrtop/home] (2 of 3)
Aug 03 17:00:10 systemd[1]: mnt-btrtop-home.mount: Deactivated successfully.
Aug 03 17:00:10 btrbk_run.sh[2387275]: OK: un-mounted
[/mnt/btrtop/root] (3 of 3)
Aug 03 17:00:10 btrbk_run.sh[2387275]: Finished un-mounting btrbk
btrtop volumes.
Aug 03 17:00:10 systemd[1]: mnt-btrtop-root.mount: Deactivated successfully.
Aug 03 18:00:06 btrbk_run.sh[2398488]: mounting btrbk btrtop volumes...

As mentioned, I have (temporarily) stopped unmounting these volumes
and I just leave them mounted all the time. The logs now look like
this:

Aug 06 03:00:14 btrbk_run.sh[3022708]: mounting btrbk btrtop volumes...
Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/root] (1 of
3) was already mounted. Nothing to do.
Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/home] (2 of
3) was already mounted. Nothing to do.
Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/user] (3 of
3) was already mounted. Nothing to do.
Aug 06 03:00:14 btrbk_run.sh[3022708]: Finished mounting btrbk btrtop volumes.
Aug 06 04:00:14 btrbk_run.sh[3033520]: mounting btrbk btrtop volumes...
Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/root] (1 of
3) was already mounted. Nothing to do.
Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/home] (2 of
3) was already mounted. Nothing to do.
Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/user] (3 of
3) was already mounted. Nothing to do.
Aug 06 04:00:14 btrbk_run.sh[3033520]: Finished mounting btrbk btrtop volumes.

>
> >
> > The path /var/cache/pacman is not a subvolume, but it resides on btrfs
> > subvolume @btrtop/snapshot. @btrtop/snapshot is normally mounted at
> > "/" but for btrfs tasks, it is also mounted at /mnt/btrtop/root. This
> > additional mount operation seems to be causing these nested mounts of
> > my bind mount for  /srv/nfs/var/cache/pacman .
> >
> > P.S. I cannot test without using systemd. (I'm not even sure I
> > remember how to use a non-systemd distro anymore!)
> >
>
>
> --
> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: why is the same mount point repeatedly mounted in nested manner?

[Goffredo Baroncelli]

On 8/9/21 10:15 PM, Dave T wrote:
> On Mon, Aug 9, 2021 at 3:29 PM Goffredo Baroncelli <kreijack@libero.it> wrote:
>>
[...]

> 
> Also, in recent days I stopped mounting and umounting /mnt/btrtop/root
> and just left it mounted all the time. However, when checking today, I
> still found a nested mount:
> 
> ├─/srv/nfs/var/cache/pacman
> │
> │ └─/srv/nfs/var/cache/pacman
> 
> 

Ok, so it seems that these mounts are triggered not by the mount of /mnt/btrtop/root (see below). What is the output of

$ systemctl --reverse list-dependencies -- srv-nfs-var-cache-pacman.mount



[...]
> 
> As mentioned, I have (temporarily) stopped unmounting these volumes
> and I just leave them mounted all the time. The logs now look like
> this:
> 
> Aug 06 03:00:14 btrbk_run.sh[3022708]: mounting btrbk btrtop volumes...
> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/root] (1 of
> 3) was already mounted. Nothing to do.
> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/home] (2 of
> 3) was already mounted. Nothing to do.
> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/user] (3 of
> 3) was already mounted. Nothing to do.

This told another story. It seems that "btrbk" itself already try to mount the btrfs subvolume. I understood that it was the systemd unit to do that. Could you share the content of btrbk_run.sh ?


> Aug 06 03:00:14 btrbk_run.sh[3022708]: Finished mounting btrbk btrtop volumes.
> Aug 06 04:00:14 btrbk_run.sh[3033520]: mounting btrbk btrtop volumes...
> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/root] (1 of
> 3) was already mounted. Nothing to do.
> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/home] (2 of
> 3) was already mounted. Nothing to do.
> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/user] (3 of
> 3) was already mounted. Nothing to do.
> Aug 06 04:00:14 btrbk_run.sh[3033520]: Finished mounting btrbk btrtop volumes.
> 
>>
>>>
>>> The path /var/cache/pacman is not a subvolume, but it resides on btrfs
>>> subvolume @btrtop/snapshot. @btrtop/snapshot is normally mounted at
>>> "/" but for btrfs tasks, it is also mounted at /mnt/btrtop/root. This
>>> additional mount operation seems to be causing these nested mounts of
>>> my bind mount for  /srv/nfs/var/cache/pacman .
>>>
>>> P.S. I cannot test without using systemd. (I'm not even sure I
>>> remember how to use a non-systemd distro anymore!)
>>>
>>
>>
>> --
>> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
>> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: why is the same mount point repeatedly mounted in nested manner?

[Dave T]

On Tue, Aug 10, 2021 at 11:43 AM Goffredo Baroncelli <kreijack@inwind.it> wrote:
>
> On 8/9/21 10:15 PM, Dave T wrote:
> > On Mon, Aug 9, 2021 at 3:29 PM Goffredo Baroncelli <kreijack@libero.it> wrote:
> >>
> [...]
>
> >
> > Also, in recent days I stopped mounting and umounting /mnt/btrtop/root
> > and just left it mounted all the time. However, when checking today, I
> > still found a nested mount:
> >
> > ├─/srv/nfs/var/cache/pacman
> > │
> > │ └─/srv/nfs/var/cache/pacman
> >
> >
>
> Ok, so it seems that these mounts are triggered not by the mount of /mnt/btrtop/root (see below). What is the output of
>
> $ systemctl --reverse list-dependencies -- srv-nfs-var-cache-pacman.mount
>

srv-nfs-var-cache-pacman.mount
● ├─nfs-server.service
● └─local-fs.target
●   └─sysinit.target
○     ├─btrbk-daily.service
●     ├─btrbk-daily.timer
○     ├─btrbk.service
●     ├─btrbk.timer
●     ├─dbus.service
●     ├─dbus.socket
●     ├─dhcpcd.service
●     ├─getty@tty1.service
●     ├─gssproxy.service
○     ├─keymap_ds.service
○     ├─logrotate.service
●     ├─logrotate.timer
○     ├─man-db.service
●     ├─man-db.timer
○     ├─nfs-utils.service
○     ├─users_permissions.service
●     ├─users_permissions.timer
○     ├─shadow.service
●     ├─shadow.timer
○     ├─snapper-cleanup.service
●     ├─snapper-cleanup.timer
●     ├─sshd.service
○     ├─sshdgenkeys.service
○     ├─systemd-ask-password-wall.service
●     ├─systemd-logind.service
●     ├─systemd-tmpfiles-clean.timer
●     ├─systemd-user-sessions.service
●     ├─user-runtime-dir@1000.service
●     ├─user@1000.service
●     ├─basic.target
○     │ ├─initrd.target
●     │ └─multi-user.target
●     │   └─graphical.target
○     └─rescue.target


This looks interesting -- some of those dependencies are very
unexpected to me. I look forward to your take on what this output
indicates. I don't understand why there is a connection between
srv-nfs-var-cache-pacman.mount and btrbk.service, for example. (I
included btrbk.service with source listings you requested below.)

>
> [...]
> >
> > As mentioned, I have (temporarily) stopped unmounting these volumes
> > and I just leave them mounted all the time. The logs now look like
> > this:
> >
> > Aug 06 03:00:14 btrbk_run.sh[3022708]: mounting btrbk btrtop volumes...
> > Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/root] (1 of
> > 3) was already mounted. Nothing to do.
> > Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/home] (2 of
> > 3) was already mounted. Nothing to do.
> > Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/user] (3 of
> > 3) was already mounted. Nothing to do.
>
> This told another story. It seems that "btrbk" itself already try to mount the btrfs subvolume. I understood that it was the systemd unit to do that. Could you share the content of btrbk_run.sh ?
>

# systemctl cat btrbk.service
# /usr/lib/systemd/system/btrbk.service
[Unit]
Description=btrbk backup
Documentation=man:btrbk(1)

[Service]
Type=oneshot
ExecStart=/usr/bin/btrbk run

# /etc/systemd/system/btrbk.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/local/bin/btrbk_run.sh


# cat /usr/local/bin/btrbk_run.sh
#!/bin/bash

/usr/local/bin/btrbk_mount

/usr/bin/btrbk --config /etc/btrbk/btrbk.conf run

# 2021-08-05 My first troubleshooting step is to disable unmounting
these shares.
# /usr/local/bin/btrbk-umount


# cat /usr/local/bin/btrbk_mount
#!/bin/bash

btrbk_mount() {

echo "mounting btrbk btrtop volumes..."

findmnt /mnt/btrtop/root
if [[ $? -ne 0 ]]; then \
  mount /mnt/btrtop/root
  if [[ $? -ne 0 ]]; then \
    echo "ERROR: failed to mount [/mnt/btrtop/root] (1 of 3)"
  else
    echo "OK: mounted [/mnt/btrtop/root] (1 of 3)"
  fi
else
  echo "INFO: [/mnt/btrtop/root] (1 of 3) was already mounted. Nothing to do."
fi
findmnt /mnt/btrtop/home
if [[ $? -ne 0 ]]; then \
  mount /mnt/btrtop/home
  if [[ $? -ne 0 ]]; then \
    echo "ERROR: failed to mount [/mnt/btrtop/home] (2 of 3)"
  else
    echo "OK: mounted [/mnt/btrtop/home] (2 of 3)"
  fi
else
  echo "INFO: [/mnt/btrtop/home] (2 of 3) was already mounted. Nothing to do."
fi
findmnt /mnt/btrtop/user
if [[ $? -ne 0 ]]; then \
  mount /mnt/btrtop/user
  if [[ $? -ne 0 ]]; then \
    echo "ERROR: failed to mount [/mnt/btrtop/user] (3 of 3)"
  else
    echo "OK: mounted mount [/mnt/btrtop/user] (3 of 3)"
  fi
else
  echo "INFO: [/mnt/btrtop/user] (3 of 3) was already mounted. Nothing to do."
fi

echo "Finished mounting btrbk btrtop volumes."

}

btrbk_mount

# end of file /usr/local/bin/btrbk_mount


>
> > Aug 06 03:00:14 btrbk_run.sh[3022708]: Finished mounting btrbk btrtop volumes.
> > Aug 06 04:00:14 btrbk_run.sh[3033520]: mounting btrbk btrtop volumes...
> > Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/root] (1 of
> > 3) was already mounted. Nothing to do.
> > Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/home] (2 of
> > 3) was already mounted. Nothing to do.
> > Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/user] (3 of
> > 3) was already mounted. Nothing to do.
> > Aug 06 04:00:14 btrbk_run.sh[3033520]: Finished mounting btrbk btrtop volumes.
> >
> >>
> >>>
> >>> The path /var/cache/pacman is not a subvolume, but it resides on btrfs
> >>> subvolume @btrtop/snapshot. @btrtop/snapshot is normally mounted at
> >>> "/" but for btrfs tasks, it is also mounted at /mnt/btrtop/root. This
> >>> additional mount operation seems to be causing these nested mounts of
> >>> my bind mount for  /srv/nfs/var/cache/pacman .
> >>>
> >>> P.S. I cannot test without using systemd. (I'm not even sure I
> >>> remember how to use a non-systemd distro anymore!)
> >>>
> >>
> >>
> >> --
> >> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> >> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
>
>
> --
> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: why is the same mount point repeatedly mounted in nested manner?

[Goffredo Baroncelli]

On 8/10/21 6:03 PM, Dave T wrote:
> On Tue, Aug 10, 2021 at 11:43 AM Goffredo Baroncelli <kreijack@inwind.it> wrote:
>>
>> On 8/9/21 10:15 PM, Dave T wrote:
>>> On Mon, Aug 9, 2021 at 3:29 PM Goffredo Baroncelli <kreijack@libero.it> wrote:
>>>>
>> [...]
>>
>>>
>>> Also, in recent days I stopped mounting and umounting /mnt/btrtop/root
>>> and just left it mounted all the time. However, when checking today, I
>>> still found a nested mount:
>>>
>>> ├─/srv/nfs/var/cache/pacman
>>> │
>>> │ └─/srv/nfs/var/cache/pacman
>>>
>>>
>>
>> Ok, so it seems that these mounts are triggered not by the mount of /mnt/btrtop/root (see below). What is the output of
>>
>> $ systemctl --reverse list-dependencies -- srv-nfs-var-cache-pacman.mount
>>
> 
> srv-nfs-var-cache-pacman.mount
> ● ├─nfs-server.service
> ● └─local-fs.target
> ●   └─sysinit.target
> ○     ├─btrbk-daily.service
> ●     ├─btrbk-daily.timer
> ○     ├─btrbk.service
> ●     ├─btrbk.timer
> ●     ├─dbus.service
> ●     ├─dbus.socket
> ●     ├─dhcpcd.service
> ●     ├─getty@tty1.service
> ●     ├─gssproxy.service
> ○     ├─keymap_ds.service
> ○     ├─logrotate.service
> ●     ├─logrotate.timer
> ○     ├─man-db.service
> ●     ├─man-db.timer
> ○     ├─nfs-utils.service
> ○     ├─users_permissions.service
> ●     ├─users_permissions.timer
> ○     ├─shadow.service
> ●     ├─shadow.timer
> ○     ├─snapper-cleanup.service
> ●     ├─snapper-cleanup.timer
> ●     ├─sshd.service
> ○     ├─sshdgenkeys.service
> ○     ├─systemd-ask-password-wall.service
> ●     ├─systemd-logind.service
> ●     ├─systemd-tmpfiles-clean.timer
> ●     ├─systemd-user-sessions.service
> ●     ├─user-runtime-dir@1000.service
> ●     ├─user@1000.service
> ●     ├─basic.target
> ○     │ ├─initrd.target
> ●     │ └─multi-user.target
> ●     │   └─graphical.target
> ○     └─rescue.target
> 
> 
> This looks interesting -- some of those dependencies are very
> unexpected to me. I look forward to your take on what this output
> indicates. I don't understand why there is a connection between
> srv-nfs-var-cache-pacman.mount and btrbk.service, for example. (I
> included btrbk.service with source listings you requested below.)

The btrbk.* units are triggered by sysinit.target. And sysinit requires local-fs.target, which in turn requires that all fstab mounts are mounted (then the link to srv-nfs-var-cache-pacman.mount).
There no is any direct connection between btrbk and srv-nfs-var-cache-pacman.mount.

What is more interesting is the dependencies between srv-nfs-var-cache-pacman.mount and nfs-server.service. I suspect (but I don't have any proof) that systemd is confused by the tuple {btrfs subvolume, bind mount, nfs dependecies}.

What happens if you restart the nfs-server ?
> 
>>
>> [...]
>>>
>>> As mentioned, I have (temporarily) stopped unmounting these volumes
>>> and I just leave them mounted all the time. The logs now look like
>>> this:
>>>
>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: mounting btrbk btrtop volumes...
>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/root] (1 of
>>> 3) was already mounted. Nothing to do.
>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/home] (2 of
>>> 3) was already mounted. Nothing to do.
>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/user] (3 of
>>> 3) was already mounted. Nothing to do.
>>
>> This told another story. It seems that "btrbk" itself already try to mount the btrfs subvolume. I understood that it was the systemd unit to do that. Could you share the content of btrbk_run.sh ?
>>
> 
> # systemctl cat btrbk.service
> # /usr/lib/systemd/system/btrbk.service
> [Unit]
> Description=btrbk backup
> Documentation=man:btrbk(1)
> 
> [Service]
> Type=oneshot
> ExecStart=/usr/bin/btrbk run
> 
> # /etc/systemd/system/btrbk.service.d/override.conf
> [Service]
> ExecStart=
> ExecStart=/usr/local/bin/btrbk_run.sh
> 
> 
> # cat /usr/local/bin/btrbk_run.sh
> #!/bin/bash
> 
> /usr/local/bin/btrbk_mount
> 
> /usr/bin/btrbk --config /etc/btrbk/btrbk.conf run
> 
> # 2021-08-05 My first troubleshooting step is to disable unmounting
> these shares.
> # /usr/local/bin/btrbk-umount
> 
> 
> # cat /usr/local/bin/btrbk_mount
> #!/bin/bash
> 
> btrbk_mount() {
> 
> echo "mounting btrbk btrtop volumes..."
> 
> findmnt /mnt/btrtop/root
> if [[ $? -ne 0 ]]; then \
>    mount /mnt/btrtop/root
>    if [[ $? -ne 0 ]]; then \
>      echo "ERROR: failed to mount [/mnt/btrtop/root] (1 of 3)"
>    else
>      echo "OK: mounted [/mnt/btrtop/root] (1 of 3)"
>    fi
> else
>    echo "INFO: [/mnt/btrtop/root] (1 of 3) was already mounted. Nothing to do."
> fi
> findmnt /mnt/btrtop/home
> if [[ $? -ne 0 ]]; then \
>    mount /mnt/btrtop/home
>    if [[ $? -ne 0 ]]; then \
>      echo "ERROR: failed to mount [/mnt/btrtop/home] (2 of 3)"
>    else
>      echo "OK: mounted [/mnt/btrtop/home] (2 of 3)"
>    fi
> else
>    echo "INFO: [/mnt/btrtop/home] (2 of 3) was already mounted. Nothing to do."
> fi
> findmnt /mnt/btrtop/user
> if [[ $? -ne 0 ]]; then \
>    mount /mnt/btrtop/user
>    if [[ $? -ne 0 ]]; then \
>      echo "ERROR: failed to mount [/mnt/btrtop/user] (3 of 3)"
>    else
>      echo "OK: mounted mount [/mnt/btrtop/user] (3 of 3)"
>    fi
> else
>    echo "INFO: [/mnt/btrtop/user] (3 of 3) was already mounted. Nothing to do."
> fi
> 
> echo "Finished mounting btrbk btrtop volumes."
> 
> }
> 
> btrbk_mount
> 
> # end of file /usr/local/bin/btrbk_mount
> 
> 
>>
>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: Finished mounting btrbk btrtop volumes.
>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: mounting btrbk btrtop volumes...
>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/root] (1 of
>>> 3) was already mounted. Nothing to do.
>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/home] (2 of
>>> 3) was already mounted. Nothing to do.
>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/user] (3 of
>>> 3) was already mounted. Nothing to do.
>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: Finished mounting btrbk btrtop volumes.
>>>
>>>>
>>>>>
>>>>> The path /var/cache/pacman is not a subvolume, but it resides on btrfs
>>>>> subvolume @btrtop/snapshot. @btrtop/snapshot is normally mounted at
>>>>> "/" but for btrfs tasks, it is also mounted at /mnt/btrtop/root. This
>>>>> additional mount operation seems to be causing these nested mounts of
>>>>> my bind mount for  /srv/nfs/var/cache/pacman .
>>>>>
>>>>> P.S. I cannot test without using systemd. (I'm not even sure I
>>>>> remember how to use a non-systemd distro anymore!)
>>>>>
>>>>
>>>>
>>>> --
>>>> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
>>>> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
>>
>>
>> --
>> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
>> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: why is the same mount point repeatedly mounted in nested manner?

[Dave T]

On Tue, Aug 10, 2021 at 12:17 PM Goffredo Baroncelli <kreijack@inwind.it> wrote:
>
> On 8/10/21 6:03 PM, Dave T wrote:
> > On Tue, Aug 10, 2021 at 11:43 AM Goffredo Baroncelli <kreijack@inwind.it> wrote:
> >>
> >> On 8/9/21 10:15 PM, Dave T wrote:
> >>> On Mon, Aug 9, 2021 at 3:29 PM Goffredo Baroncelli <kreijack@libero.it> wrote:
> >>>>
> >> [...]
> >>
> >>>
> >>> Also, in recent days I stopped mounting and umounting /mnt/btrtop/root
> >>> and just left it mounted all the time. However, when checking today, I
> >>> still found a nested mount:
> >>>
> >>> ├─/srv/nfs/var/cache/pacman
> >>> │
> >>> │ └─/srv/nfs/var/cache/pacman
> >>>
> >>>
> >>
> >> Ok, so it seems that these mounts are triggered not by the mount of /mnt/btrtop/root (see below). What is the output of
> >>
> >> $ systemctl --reverse list-dependencies -- srv-nfs-var-cache-pacman.mount
> >>
> >
> > srv-nfs-var-cache-pacman.mount
> > ● ├─nfs-server.service
> > ● └─local-fs.target
> > ●   └─sysinit.target
> > ○     ├─btrbk-daily.service
> > ●     ├─btrbk-daily.timer
> > ○     ├─btrbk.service
> > ●     ├─btrbk.timer
> > ●     ├─dbus.service
> > ●     ├─dbus.socket
> > ●     ├─dhcpcd.service
> > ●     ├─getty@tty1.service
> > ●     ├─gssproxy.service
> > ○     ├─keymap_ds.service
> > ○     ├─logrotate.service
> > ●     ├─logrotate.timer
> > ○     ├─man-db.service
> > ●     ├─man-db.timer
> > ○     ├─nfs-utils.service
> > ○     ├─users_permissions.service
> > ●     ├─users_permissions.timer
> > ○     ├─shadow.service
> > ●     ├─shadow.timer
> > ○     ├─snapper-cleanup.service
> > ●     ├─snapper-cleanup.timer
> > ●     ├─sshd.service
> > ○     ├─sshdgenkeys.service
> > ○     ├─systemd-ask-password-wall.service
> > ●     ├─systemd-logind.service
> > ●     ├─systemd-tmpfiles-clean.timer
> > ●     ├─systemd-user-sessions.service
> > ●     ├─user-runtime-dir@1000.service
> > ●     ├─user@1000.service
> > ●     ├─basic.target
> > ○     │ ├─initrd.target
> > ●     │ └─multi-user.target
> > ●     │   └─graphical.target
> > ○     └─rescue.target
> >
> >
> > This looks interesting -- some of those dependencies are very
> > unexpected to me. I look forward to your take on what this output
> > indicates. I don't understand why there is a connection between
> > srv-nfs-var-cache-pacman.mount and btrbk.service, for example. (I
> > included btrbk.service with source listings you requested below.)
>
> The btrbk.* units are triggered by sysinit.target. And sysinit requires local-fs.target, which in turn requires that all fstab mounts are mounted (then the link to srv-nfs-var-cache-pacman.mount).
> There no is any direct connection between btrbk and srv-nfs-var-cache-pacman.mount.
>
> What is more interesting is the dependencies between srv-nfs-var-cache-pacman.mount and nfs-server.service. I suspect (but I don't have any proof) that systemd is confused by the tuple {btrfs subvolume, bind mount, nfs dependecies}.
>
> What happens if you restart the nfs-server ?

As part of this issue, nfs clients have been experiencing slowness and
sometimes hangs. I have restarted the nfs server service a few times
while this issue was happening and it seems to temporarily resolve the
client issues, but I'm not 100% sure because a specific incident will
resolve eventually on its own without any intervention by me. But
future incidences of slowness or temporary hangs are continuing and I
do think this is all related.

Restarting the nfs server service does not make the nested mounts we
are discussing go away. I have to unwind them with multiple calls to
umount -- although the number of calls is less than the number of
nested mounts, which I find confusing. Typically just 2-3 calls to
umount will unmount a nesting 5-6 deep.

Clients connect to a number of different nfs shares from this server,
yet none of the others wind up with nested mounts like
srv-nfs-var-cache-pacman.mount . All mounts are configured the same
way, using the same bind mount parameters in fstab.

> >
> >>
> >> [...]
> >>>
> >>> As mentioned, I have (temporarily) stopped unmounting these volumes
> >>> and I just leave them mounted all the time. The logs now look like
> >>> this:
> >>>
> >>> Aug 06 03:00:14 btrbk_run.sh[3022708]: mounting btrbk btrtop volumes...
> >>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/root] (1 of
> >>> 3) was already mounted. Nothing to do.
> >>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/home] (2 of
> >>> 3) was already mounted. Nothing to do.
> >>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/user] (3 of
> >>> 3) was already mounted. Nothing to do.
> >>
> >> This told another story. It seems that "btrbk" itself already try to mount the btrfs subvolume. I understood that it was the systemd unit to do that. Could you share the content of btrbk_run.sh ?
> >>
> >
> > # systemctl cat btrbk.service
> > # /usr/lib/systemd/system/btrbk.service
> > [Unit]
> > Description=btrbk backup
> > Documentation=man:btrbk(1)
> >
> > [Service]
> > Type=oneshot
> > ExecStart=/usr/bin/btrbk run
> >
> > # /etc/systemd/system/btrbk.service.d/override.conf
> > [Service]
> > ExecStart=
> > ExecStart=/usr/local/bin/btrbk_run.sh
> >
> >
> > # cat /usr/local/bin/btrbk_run.sh
> > #!/bin/bash
> >
> > /usr/local/bin/btrbk_mount
> >
> > /usr/bin/btrbk --config /etc/btrbk/btrbk.conf run
> >
> > # 2021-08-05 My first troubleshooting step is to disable unmounting
> > these shares.
> > # /usr/local/bin/btrbk-umount
> >
> >
> > # cat /usr/local/bin/btrbk_mount
> > #!/bin/bash
> >
> > btrbk_mount() {
> >
> > echo "mounting btrbk btrtop volumes..."
> >
> > findmnt /mnt/btrtop/root
> > if [[ $? -ne 0 ]]; then \
> >    mount /mnt/btrtop/root
> >    if [[ $? -ne 0 ]]; then \
> >      echo "ERROR: failed to mount [/mnt/btrtop/root] (1 of 3)"
> >    else
> >      echo "OK: mounted [/mnt/btrtop/root] (1 of 3)"
> >    fi
> > else
> >    echo "INFO: [/mnt/btrtop/root] (1 of 3) was already mounted. Nothing to do."
> > fi
> > findmnt /mnt/btrtop/home
> > if [[ $? -ne 0 ]]; then \
> >    mount /mnt/btrtop/home
> >    if [[ $? -ne 0 ]]; then \
> >      echo "ERROR: failed to mount [/mnt/btrtop/home] (2 of 3)"
> >    else
> >      echo "OK: mounted [/mnt/btrtop/home] (2 of 3)"
> >    fi
> > else
> >    echo "INFO: [/mnt/btrtop/home] (2 of 3) was already mounted. Nothing to do."
> > fi
> > findmnt /mnt/btrtop/user
> > if [[ $? -ne 0 ]]; then \
> >    mount /mnt/btrtop/user
> >    if [[ $? -ne 0 ]]; then \
> >      echo "ERROR: failed to mount [/mnt/btrtop/user] (3 of 3)"
> >    else
> >      echo "OK: mounted mount [/mnt/btrtop/user] (3 of 3)"
> >    fi
> > else
> >    echo "INFO: [/mnt/btrtop/user] (3 of 3) was already mounted. Nothing to do."
> > fi
> >
> > echo "Finished mounting btrbk btrtop volumes."
> >
> > }
> >
> > btrbk_mount
> >
> > # end of file /usr/local/bin/btrbk_mount
> >
> >
> >>
> >>> Aug 06 03:00:14 btrbk_run.sh[3022708]: Finished mounting btrbk btrtop volumes.
> >>> Aug 06 04:00:14 btrbk_run.sh[3033520]: mounting btrbk btrtop volumes...
> >>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/root] (1 of
> >>> 3) was already mounted. Nothing to do.
> >>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/home] (2 of
> >>> 3) was already mounted. Nothing to do.
> >>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/user] (3 of
> >>> 3) was already mounted. Nothing to do.
> >>> Aug 06 04:00:14 btrbk_run.sh[3033520]: Finished mounting btrbk btrtop volumes.
> >>>
> >>>>
> >>>>>
> >>>>> The path /var/cache/pacman is not a subvolume, but it resides on btrfs
> >>>>> subvolume @btrtop/snapshot. @btrtop/snapshot is normally mounted at
> >>>>> "/" but for btrfs tasks, it is also mounted at /mnt/btrtop/root. This
> >>>>> additional mount operation seems to be causing these nested mounts of
> >>>>> my bind mount for  /srv/nfs/var/cache/pacman .
> >>>>>
> >>>>> P.S. I cannot test without using systemd. (I'm not even sure I
> >>>>> remember how to use a non-systemd distro anymore!)
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> >>>> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
> >>
> >>
> >> --
> >> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> >> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
>
>
> --
> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: why is the same mount point repeatedly mounted in nested manner?

[Goffredo Baroncelli]

On 8/10/21 6:27 PM, Dave T wrote:
> On Tue, Aug 10, 2021 at 12:17 PM Goffredo Baroncelli <kreijack@inwind.it> wrote:
>>
>> On 8/10/21 6:03 PM, Dave T wrote:
>>> On Tue, Aug 10, 2021 at 11:43 AM Goffredo Baroncelli <kreijack@inwind.it> wrote:
[...]
>> What is more interesting is the dependencies between srv-nfs-var-cache-pacman.mount and nfs-server.service. I suspect (but I don't have any proof) that systemd is confused by the tuple {btrfs subvolume, bind mount, nfs dependecies}.
>>
>> What happens if you restart the nfs-server ?
> 
> As part of this issue, nfs clients have been experiencing slowness and
> sometimes hangs. I have restarted the nfs server service a few times
> while this issue was happening and it seems to temporarily resolve the
> client issues, but I'm not 100% sure because a specific incident will
> resolve eventually on its own without any intervention by me. But
> future incidences of slowness or temporary hangs are continuing and I
> do think this is all related.
> 
> Restarting the nfs server service does not make the nested mounts we
> are discussing go away. I have to unwind them with multiple calls to
> umount -- although the number of calls is less than the number of
> nested mounts, which I find confusing. Typically just 2-3 calls to
> umount will unmount a nesting 5-6 deep.
> 
> Clients connect to a number of different nfs shares from this server,
> yet none of the others wind up with nested mounts like
> srv-nfs-var-cache-pacman.mount . All mounts are configured the same
> way, using the same bind mount parameters in fstab.
> 

Unfortunately I don't have any more suggestions..

My opinion is that the problem is not related to the btrfs itself, but it is more a side effect of the interaction between systemd and btrfs. What about looking to the log ?

$ sudo journalctl -u  srv-nfs-var-cache-pacman.mount

Then check the log near the mount/unmount

To avoid this problem enterely, what about using a symlink instead of the bind mount ?

Something like

# rmdir /srv/nfs/var/cache/pacman ; ln -sf /var/cache/pacman /srv/nfs/var/cache/pacman


BR
GB

>>>
>>>>
>>>> [...]
>>>>>
>>>>> As mentioned, I have (temporarily) stopped unmounting these volumes
>>>>> and I just leave them mounted all the time. The logs now look like
>>>>> this:
>>>>>
>>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: mounting btrbk btrtop volumes...
>>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/root] (1 of
>>>>> 3) was already mounted. Nothing to do.
>>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/home] (2 of
>>>>> 3) was already mounted. Nothing to do.
>>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/user] (3 of
>>>>> 3) was already mounted. Nothing to do.
>>>>
>>>> This told another story. It seems that "btrbk" itself already try to mount the btrfs subvolume. I understood that it was the systemd unit to do that. Could you share the content of btrbk_run.sh ?
>>>>
>>>
>>> # systemctl cat btrbk.service
>>> # /usr/lib/systemd/system/btrbk.service
>>> [Unit]
>>> Description=btrbk backup
>>> Documentation=man:btrbk(1)
>>>
>>> [Service]
>>> Type=oneshot
>>> ExecStart=/usr/bin/btrbk run
>>>
>>> # /etc/systemd/system/btrbk.service.d/override.conf
>>> [Service]
>>> ExecStart=
>>> ExecStart=/usr/local/bin/btrbk_run.sh
>>>
>>>
>>> # cat /usr/local/bin/btrbk_run.sh
>>> #!/bin/bash
>>>
>>> /usr/local/bin/btrbk_mount
>>>
>>> /usr/bin/btrbk --config /etc/btrbk/btrbk.conf run
>>>
>>> # 2021-08-05 My first troubleshooting step is to disable unmounting
>>> these shares.
>>> # /usr/local/bin/btrbk-umount
>>>
>>>
>>> # cat /usr/local/bin/btrbk_mount
>>> #!/bin/bash
>>>
>>> btrbk_mount() {
>>>
>>> echo "mounting btrbk btrtop volumes..."
>>>
>>> findmnt /mnt/btrtop/root
>>> if [[ $? -ne 0 ]]; then \
>>>     mount /mnt/btrtop/root
>>>     if [[ $? -ne 0 ]]; then \
>>>       echo "ERROR: failed to mount [/mnt/btrtop/root] (1 of 3)"
>>>     else
>>>       echo "OK: mounted [/mnt/btrtop/root] (1 of 3)"
>>>     fi
>>> else
>>>     echo "INFO: [/mnt/btrtop/root] (1 of 3) was already mounted. Nothing to do."
>>> fi
>>> findmnt /mnt/btrtop/home
>>> if [[ $? -ne 0 ]]; then \
>>>     mount /mnt/btrtop/home
>>>     if [[ $? -ne 0 ]]; then \
>>>       echo "ERROR: failed to mount [/mnt/btrtop/home] (2 of 3)"
>>>     else
>>>       echo "OK: mounted [/mnt/btrtop/home] (2 of 3)"
>>>     fi
>>> else
>>>     echo "INFO: [/mnt/btrtop/home] (2 of 3) was already mounted. Nothing to do."
>>> fi
>>> findmnt /mnt/btrtop/user
>>> if [[ $? -ne 0 ]]; then \
>>>     mount /mnt/btrtop/user
>>>     if [[ $? -ne 0 ]]; then \
>>>       echo "ERROR: failed to mount [/mnt/btrtop/user] (3 of 3)"
>>>     else
>>>       echo "OK: mounted mount [/mnt/btrtop/user] (3 of 3)"
>>>     fi
>>> else
>>>     echo "INFO: [/mnt/btrtop/user] (3 of 3) was already mounted. Nothing to do."
>>> fi
>>>
>>> echo "Finished mounting btrbk btrtop volumes."
>>>
>>> }
>>>
>>> btrbk_mount
>>>
>>> # end of file /usr/local/bin/btrbk_mount
>>>
>>>
>>>>
>>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: Finished mounting btrbk btrtop volumes.
>>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: mounting btrbk btrtop volumes...
>>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/root] (1 of
>>>>> 3) was already mounted. Nothing to do.
>>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/home] (2 of
>>>>> 3) was already mounted. Nothing to do.
>>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/user] (3 of
>>>>> 3) was already mounted. Nothing to do.
>>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: Finished mounting btrbk btrtop volumes.
>>>>>
>>>>>>
>>>>>>>
>>>>>>> The path /var/cache/pacman is not a subvolume, but it resides on btrfs
>>>>>>> subvolume @btrtop/snapshot. @btrtop/snapshot is normally mounted at
>>>>>>> "/" but for btrfs tasks, it is also mounted at /mnt/btrtop/root. This
>>>>>>> additional mount operation seems to be causing these nested mounts of
>>>>>>> my bind mount for  /srv/nfs/var/cache/pacman .
>>>>>>>
>>>>>>> P.S. I cannot test without using systemd. (I'm not even sure I
>>>>>>> remember how to use a non-systemd distro anymore!)
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
>>>>>> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
>>>>
>>>>
>>>> --
>>>> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
>>>> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
>>
>>
>> --
>> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
>> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: why is the same mount point repeatedly mounted in nested manner?

[Dave T]

On Tue, Aug 10, 2021 at 4:17 PM Goffredo Baroncelli <kreijack@inwind.it> wrote:
>
> On 8/10/21 6:27 PM, Dave T wrote:
> > On Tue, Aug 10, 2021 at 12:17 PM Goffredo Baroncelli <kreijack@inwind.it> wrote:
> >>
> >> On 8/10/21 6:03 PM, Dave T wrote:
> >>> On Tue, Aug 10, 2021 at 11:43 AM Goffredo Baroncelli <kreijack@inwind.it> wrote:
> [...]
> >> What is more interesting is the dependencies between srv-nfs-var-cache-pacman.mount and nfs-server.service. I suspect (but I don't have any proof) that systemd is confused by the tuple {btrfs subvolume, bind mount, nfs dependecies}.
> >>
> >> What happens if you restart the nfs-server ?
> >
> > As part of this issue, nfs clients have been experiencing slowness and
> > sometimes hangs. I have restarted the nfs server service a few times
> > while this issue was happening and it seems to temporarily resolve the
> > client issues, but I'm not 100% sure because a specific incident will
> > resolve eventually on its own without any intervention by me. But
> > future incidences of slowness or temporary hangs are continuing and I
> > do think this is all related.
> >
> > Restarting the nfs server service does not make the nested mounts we
> > are discussing go away. I have to unwind them with multiple calls to
> > umount -- although the number of calls is less than the number of
> > nested mounts, which I find confusing. Typically just 2-3 calls to
> > umount will unmount a nesting 5-6 deep.
> >
> > Clients connect to a number of different nfs shares from this server,
> > yet none of the others wind up with nested mounts like
> > srv-nfs-var-cache-pacman.mount . All mounts are configured the same
> > way, using the same bind mount parameters in fstab.
> >
>
> Unfortunately I don't have any more suggestions..

The replies you have given so far have already helped me understand
the issue a little better. Thank you.
>
> My opinion is that the problem is not related to the btrfs itself, but it is more a side effect of the interaction between systemd and btrfs. What about looking to the log ?

Yes, I agree.

>
> $ sudo journalctl -u  srv-nfs-var-cache-pacman.mount
>
> Then check the log near the mount/unmount

    -- Boot 48833b09370b46a4ade0c1979438be90 --
    Jul 24 22:29:00 nfssrvr systemd[1]: Mounting /srv/nfs/var/cache/pacman...
    Jul 24 22:29:00 nfssrvr systemd[1]: Mounted /srv/nfs/var/cache/pacman.
    -- Boot d11dc578c1c342b3a44bbf7d266c4049 --
    Jul 25 12:37:31 nfssrvr systemd[1]: Mounting /srv/nfs/var/cache/pacman...
    Jul 25 12:37:31 nfssrvr systemd[1]: Mounted /srv/nfs/var/cache/pacman.
    -- Boot a4e09d4b507c4e4e949f03c3ffe11081 --
    Jul 25 14:15:33 nfssrvr systemd[1]: Mounting /srv/nfs/var/cache/pacman...
    Jul 25 14:15:33 nfssrvr systemd[1]: Mounted /srv/nfs/var/cache/pacman.
    Jul 25 18:27:56 nfssrvr systemd[1]:
srv-nfs-var-cache-pacman.mount: Deactivated successfully.
    Aug 05 11:39:10 nfssrvr systemd[1]:
srv-nfs-var-cache-pacman.mount: Deactivated successfully.

Those last two lines are my intervention (manually unmounting). I
don't see anything else of interest to this in the logs.
>
> To avoid this problem enterely, what about using a symlink instead of the bind mount ?
>
> Something like
>
> # rmdir /srv/nfs/var/cache/pacman ; ln -sf /var/cache/pacman /srv/nfs/var/cache/pacman

As far as I know, that's not a recommended practice for NFS shares. I
don't know why. I'll look into it. Thanks for the idea.
>
>
> BR
> GB
>
> >>>
> >>>>
> >>>> [...]
> >>>>>
> >>>>> As mentioned, I have (temporarily) stopped unmounting these volumes
> >>>>> and I just leave them mounted all the time. The logs now look like
> >>>>> this:
> >>>>>
> >>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: mounting btrbk btrtop volumes...
> >>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/root] (1 of
> >>>>> 3) was already mounted. Nothing to do.
> >>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/home] (2 of
> >>>>> 3) was already mounted. Nothing to do.
> >>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/user] (3 of
> >>>>> 3) was already mounted. Nothing to do.
> >>>>
> >>>> This told another story. It seems that "btrbk" itself already try to mount the btrfs subvolume. I understood that it was the systemd unit to do that. Could you share the content of btrbk_run.sh ?
> >>>>
> >>>
> >>> # systemctl cat btrbk.service
> >>> # /usr/lib/systemd/system/btrbk.service
> >>> [Unit]
> >>> Description=btrbk backup
> >>> Documentation=man:btrbk(1)
> >>>
> >>> [Service]
> >>> Type=oneshot
> >>> ExecStart=/usr/bin/btrbk run
> >>>
> >>> # /etc/systemd/system/btrbk.service.d/override.conf
> >>> [Service]
> >>> ExecStart=
> >>> ExecStart=/usr/local/bin/btrbk_run.sh
> >>>
> >>>
> >>> # cat /usr/local/bin/btrbk_run.sh
> >>> #!/bin/bash
> >>>
> >>> /usr/local/bin/btrbk_mount
> >>>
> >>> /usr/bin/btrbk --config /etc/btrbk/btrbk.conf run
> >>>
> >>> # 2021-08-05 My first troubleshooting step is to disable unmounting
> >>> these shares.
> >>> # /usr/local/bin/btrbk-umount
> >>>
> >>>
> >>> # cat /usr/local/bin/btrbk_mount
> >>> #!/bin/bash
> >>>
> >>> btrbk_mount() {
> >>>
> >>> echo "mounting btrbk btrtop volumes..."
> >>>
> >>> findmnt /mnt/btrtop/root
> >>> if [[ $? -ne 0 ]]; then \
> >>>     mount /mnt/btrtop/root
> >>>     if [[ $? -ne 0 ]]; then \
> >>>       echo "ERROR: failed to mount [/mnt/btrtop/root] (1 of 3)"
> >>>     else
> >>>       echo "OK: mounted [/mnt/btrtop/root] (1 of 3)"
> >>>     fi
> >>> else
> >>>     echo "INFO: [/mnt/btrtop/root] (1 of 3) was already mounted. Nothing to do."
> >>> fi
> >>> findmnt /mnt/btrtop/home
> >>> if [[ $? -ne 0 ]]; then \
> >>>     mount /mnt/btrtop/home
> >>>     if [[ $? -ne 0 ]]; then \
> >>>       echo "ERROR: failed to mount [/mnt/btrtop/home] (2 of 3)"
> >>>     else
> >>>       echo "OK: mounted [/mnt/btrtop/home] (2 of 3)"
> >>>     fi
> >>> else
> >>>     echo "INFO: [/mnt/btrtop/home] (2 of 3) was already mounted. Nothing to do."
> >>> fi
> >>> findmnt /mnt/btrtop/user
> >>> if [[ $? -ne 0 ]]; then \
> >>>     mount /mnt/btrtop/user
> >>>     if [[ $? -ne 0 ]]; then \
> >>>       echo "ERROR: failed to mount [/mnt/btrtop/user] (3 of 3)"
> >>>     else
> >>>       echo "OK: mounted mount [/mnt/btrtop/user] (3 of 3)"
> >>>     fi
> >>> else
> >>>     echo "INFO: [/mnt/btrtop/user] (3 of 3) was already mounted. Nothing to do."
> >>> fi
> >>>
> >>> echo "Finished mounting btrbk btrtop volumes."
> >>>
> >>> }
> >>>
> >>> btrbk_mount
> >>>
> >>> # end of file /usr/local/bin/btrbk_mount
> >>>
> >>>
> >>>>
> >>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: Finished mounting btrbk btrtop volumes.
> >>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: mounting btrbk btrtop volumes...
> >>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/root] (1 of
> >>>>> 3) was already mounted. Nothing to do.
> >>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/home] (2 of
> >>>>> 3) was already mounted. Nothing to do.
> >>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/user] (3 of
> >>>>> 3) was already mounted. Nothing to do.
> >>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: Finished mounting btrbk btrtop volumes.
> >>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> The path /var/cache/pacman is not a subvolume, but it resides on btrfs
> >>>>>>> subvolume @btrtop/snapshot. @btrtop/snapshot is normally mounted at
> >>>>>>> "/" but for btrfs tasks, it is also mounted at /mnt/btrtop/root. This
> >>>>>>> additional mount operation seems to be causing these nested mounts of
> >>>>>>> my bind mount for  /srv/nfs/var/cache/pacman .
> >>>>>>>
> >>>>>>> P.S. I cannot test without using systemd. (I'm not even sure I
> >>>>>>> remember how to use a non-systemd distro anymore!)
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> >>>>>> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
> >>>>
> >>>>
> >>>> --
> >>>> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> >>>> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
> >>
> >>
> >> --
> >> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> >> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
>
>
> --
> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: why is the same mount point repeatedly mounted in nested manner?

[Dave T]

Following up on this topic, I came across this github issue today:

[bind entries in fstab are made recursive regardless of options ·
Issue #1204 · systemd/systemd ·
GitHub](https://github.com/systemd/systemd/issues/1204)

I think that is the explanation. I will do more testing to confirm.

On Tue, Aug 10, 2021 at 4:36 PM Dave T <davestechshop@gmail.com> wrote:
>
> On Tue, Aug 10, 2021 at 4:17 PM Goffredo Baroncelli <kreijack@inwind.it> wrote:
> >
> > On 8/10/21 6:27 PM, Dave T wrote:
> > > On Tue, Aug 10, 2021 at 12:17 PM Goffredo Baroncelli <kreijack@inwind.it> wrote:
> > >>
> > >> On 8/10/21 6:03 PM, Dave T wrote:
> > >>> On Tue, Aug 10, 2021 at 11:43 AM Goffredo Baroncelli <kreijack@inwind.it> wrote:
> > [...]
> > >> What is more interesting is the dependencies between srv-nfs-var-cache-pacman.mount and nfs-server.service. I suspect (but I don't have any proof) that systemd is confused by the tuple {btrfs subvolume, bind mount, nfs dependecies}.
> > >>
> > >> What happens if you restart the nfs-server ?
> > >
> > > As part of this issue, nfs clients have been experiencing slowness and
> > > sometimes hangs. I have restarted the nfs server service a few times
> > > while this issue was happening and it seems to temporarily resolve the
> > > client issues, but I'm not 100% sure because a specific incident will
> > > resolve eventually on its own without any intervention by me. But
> > > future incidences of slowness or temporary hangs are continuing and I
> > > do think this is all related.
> > >
> > > Restarting the nfs server service does not make the nested mounts we
> > > are discussing go away. I have to unwind them with multiple calls to
> > > umount -- although the number of calls is less than the number of
> > > nested mounts, which I find confusing. Typically just 2-3 calls to
> > > umount will unmount a nesting 5-6 deep.
> > >
> > > Clients connect to a number of different nfs shares from this server,
> > > yet none of the others wind up with nested mounts like
> > > srv-nfs-var-cache-pacman.mount . All mounts are configured the same
> > > way, using the same bind mount parameters in fstab.
> > >
> >
> > Unfortunately I don't have any more suggestions..
>
> The replies you have given so far have already helped me understand
> the issue a little better. Thank you.
> >
> > My opinion is that the problem is not related to the btrfs itself, but it is more a side effect of the interaction between systemd and btrfs. What about looking to the log ?
>
> Yes, I agree.
>
> >
> > $ sudo journalctl -u  srv-nfs-var-cache-pacman.mount
> >
> > Then check the log near the mount/unmount
>
>     -- Boot 48833b09370b46a4ade0c1979438be90 --
>     Jul 24 22:29:00 nfssrvr systemd[1]: Mounting /srv/nfs/var/cache/pacman...
>     Jul 24 22:29:00 nfssrvr systemd[1]: Mounted /srv/nfs/var/cache/pacman.
>     -- Boot d11dc578c1c342b3a44bbf7d266c4049 --
>     Jul 25 12:37:31 nfssrvr systemd[1]: Mounting /srv/nfs/var/cache/pacman...
>     Jul 25 12:37:31 nfssrvr systemd[1]: Mounted /srv/nfs/var/cache/pacman.
>     -- Boot a4e09d4b507c4e4e949f03c3ffe11081 --
>     Jul 25 14:15:33 nfssrvr systemd[1]: Mounting /srv/nfs/var/cache/pacman...
>     Jul 25 14:15:33 nfssrvr systemd[1]: Mounted /srv/nfs/var/cache/pacman.
>     Jul 25 18:27:56 nfssrvr systemd[1]:
> srv-nfs-var-cache-pacman.mount: Deactivated successfully.
>     Aug 05 11:39:10 nfssrvr systemd[1]:
> srv-nfs-var-cache-pacman.mount: Deactivated successfully.
>
> Those last two lines are my intervention (manually unmounting). I
> don't see anything else of interest to this in the logs.
> >
> > To avoid this problem enterely, what about using a symlink instead of the bind mount ?
> >
> > Something like
> >
> > # rmdir /srv/nfs/var/cache/pacman ; ln -sf /var/cache/pacman /srv/nfs/var/cache/pacman
>
> As far as I know, that's not a recommended practice for NFS shares. I
> don't know why. I'll look into it. Thanks for the idea.
> >
> >
> > BR
> > GB
> >
> > >>>
> > >>>>
> > >>>> [...]
> > >>>>>
> > >>>>> As mentioned, I have (temporarily) stopped unmounting these volumes
> > >>>>> and I just leave them mounted all the time. The logs now look like
> > >>>>> this:
> > >>>>>
> > >>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: mounting btrbk btrtop volumes...
> > >>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/root] (1 of
> > >>>>> 3) was already mounted. Nothing to do.
> > >>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/home] (2 of
> > >>>>> 3) was already mounted. Nothing to do.
> > >>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/user] (3 of
> > >>>>> 3) was already mounted. Nothing to do.
> > >>>>
> > >>>> This told another story. It seems that "btrbk" itself already try to mount the btrfs subvolume. I understood that it was the systemd unit to do that. Could you share the content of btrbk_run.sh ?
> > >>>>
> > >>>
> > >>> # systemctl cat btrbk.service
> > >>> # /usr/lib/systemd/system/btrbk.service
> > >>> [Unit]
> > >>> Description=btrbk backup
> > >>> Documentation=man:btrbk(1)
> > >>>
> > >>> [Service]
> > >>> Type=oneshot
> > >>> ExecStart=/usr/bin/btrbk run
> > >>>
> > >>> # /etc/systemd/system/btrbk.service.d/override.conf
> > >>> [Service]
> > >>> ExecStart=
> > >>> ExecStart=/usr/local/bin/btrbk_run.sh
> > >>>
> > >>>
> > >>> # cat /usr/local/bin/btrbk_run.sh
> > >>> #!/bin/bash
> > >>>
> > >>> /usr/local/bin/btrbk_mount
> > >>>
> > >>> /usr/bin/btrbk --config /etc/btrbk/btrbk.conf run
> > >>>
> > >>> # 2021-08-05 My first troubleshooting step is to disable unmounting
> > >>> these shares.
> > >>> # /usr/local/bin/btrbk-umount
> > >>>
> > >>>
> > >>> # cat /usr/local/bin/btrbk_mount
> > >>> #!/bin/bash
> > >>>
> > >>> btrbk_mount() {
> > >>>
> > >>> echo "mounting btrbk btrtop volumes..."
> > >>>
> > >>> findmnt /mnt/btrtop/root
> > >>> if [[ $? -ne 0 ]]; then \
> > >>>     mount /mnt/btrtop/root
> > >>>     if [[ $? -ne 0 ]]; then \
> > >>>       echo "ERROR: failed to mount [/mnt/btrtop/root] (1 of 3)"
> > >>>     else
> > >>>       echo "OK: mounted [/mnt/btrtop/root] (1 of 3)"
> > >>>     fi
> > >>> else
> > >>>     echo "INFO: [/mnt/btrtop/root] (1 of 3) was already mounted. Nothing to do."
> > >>> fi
> > >>> findmnt /mnt/btrtop/home
> > >>> if [[ $? -ne 0 ]]; then \
> > >>>     mount /mnt/btrtop/home
> > >>>     if [[ $? -ne 0 ]]; then \
> > >>>       echo "ERROR: failed to mount [/mnt/btrtop/home] (2 of 3)"
> > >>>     else
> > >>>       echo "OK: mounted [/mnt/btrtop/home] (2 of 3)"
> > >>>     fi
> > >>> else
> > >>>     echo "INFO: [/mnt/btrtop/home] (2 of 3) was already mounted. Nothing to do."
> > >>> fi
> > >>> findmnt /mnt/btrtop/user
> > >>> if [[ $? -ne 0 ]]; then \
> > >>>     mount /mnt/btrtop/user
> > >>>     if [[ $? -ne 0 ]]; then \
> > >>>       echo "ERROR: failed to mount [/mnt/btrtop/user] (3 of 3)"
> > >>>     else
> > >>>       echo "OK: mounted mount [/mnt/btrtop/user] (3 of 3)"
> > >>>     fi
> > >>> else
> > >>>     echo "INFO: [/mnt/btrtop/user] (3 of 3) was already mounted. Nothing to do."
> > >>> fi
> > >>>
> > >>> echo "Finished mounting btrbk btrtop volumes."
> > >>>
> > >>> }
> > >>>
> > >>> btrbk_mount
> > >>>
> > >>> # end of file /usr/local/bin/btrbk_mount
> > >>>
> > >>>
> > >>>>
> > >>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: Finished mounting btrbk btrtop volumes.
> > >>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: mounting btrbk btrtop volumes...
> > >>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/root] (1 of
> > >>>>> 3) was already mounted. Nothing to do.
> > >>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/home] (2 of
> > >>>>> 3) was already mounted. Nothing to do.
> > >>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/user] (3 of
> > >>>>> 3) was already mounted. Nothing to do.
> > >>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: Finished mounting btrbk btrtop volumes.
> > >>>>>
> > >>>>>>
> > >>>>>>>
> > >>>>>>> The path /var/cache/pacman is not a subvolume, but it resides on btrfs
> > >>>>>>> subvolume @btrtop/snapshot. @btrtop/snapshot is normally mounted at
> > >>>>>>> "/" but for btrfs tasks, it is also mounted at /mnt/btrtop/root. This
> > >>>>>>> additional mount operation seems to be causing these nested mounts of
> > >>>>>>> my bind mount for  /srv/nfs/var/cache/pacman .
> > >>>>>>>
> > >>>>>>> P.S. I cannot test without using systemd. (I'm not even sure I
> > >>>>>>> remember how to use a non-systemd distro anymore!)
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> > >>>>>> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
> > >>>>
> > >>>>
> > >>>> --
> > >>>> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> > >>>> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
> > >>
> > >>
> > >> --
> > >> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> > >> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
> >
> >
> > --
> > gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> > Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5