Re: why is the same mount point repeatedly mounted in nested manner?

[Goffredo Baroncelli <kreijack@inwind.it>]

On 8/10/21 6:03 PM, Dave T wrote:
> On Tue, Aug 10, 2021 at 11:43 AM Goffredo Baroncelli <kreijack@inwind.it> wrote:
>>
>> On 8/9/21 10:15 PM, Dave T wrote:
>>> On Mon, Aug 9, 2021 at 3:29 PM Goffredo Baroncelli <kreijack@libero.it> wrote:
>>>>
>> [...]
>>
>>>
>>> Also, in recent days I stopped mounting and umounting /mnt/btrtop/root
>>> and just left it mounted all the time. However, when checking today, I
>>> still found a nested mount:
>>>
>>> ├─/srv/nfs/var/cache/pacman
>>> │
>>> │ └─/srv/nfs/var/cache/pacman
>>>
>>>
>>
>> Ok, so it seems that these mounts are triggered not by the mount of /mnt/btrtop/root (see below). What is the output of
>>
>> $ systemctl --reverse list-dependencies -- srv-nfs-var-cache-pacman.mount
>>
> 
> srv-nfs-var-cache-pacman.mount
> ● ├─nfs-server.service
> ● └─local-fs.target
> ●   └─sysinit.target
> ○     ├─btrbk-daily.service
> ●     ├─btrbk-daily.timer
> ○     ├─btrbk.service
> ●     ├─btrbk.timer
> ●     ├─dbus.service
> ●     ├─dbus.socket
> ●     ├─dhcpcd.service
> ●     ├─getty@tty1.service
> ●     ├─gssproxy.service
> ○     ├─keymap_ds.service
> ○     ├─logrotate.service
> ●     ├─logrotate.timer
> ○     ├─man-db.service
> ●     ├─man-db.timer
> ○     ├─nfs-utils.service
> ○     ├─users_permissions.service
> ●     ├─users_permissions.timer
> ○     ├─shadow.service
> ●     ├─shadow.timer
> ○     ├─snapper-cleanup.service
> ●     ├─snapper-cleanup.timer
> ●     ├─sshd.service
> ○     ├─sshdgenkeys.service
> ○     ├─systemd-ask-password-wall.service
> ●     ├─systemd-logind.service
> ●     ├─systemd-tmpfiles-clean.timer
> ●     ├─systemd-user-sessions.service
> ●     ├─user-runtime-dir@1000.service
> ●     ├─user@1000.service
> ●     ├─basic.target
> ○     │ ├─initrd.target
> ●     │ └─multi-user.target
> ●     │   └─graphical.target
> ○     └─rescue.target
> 
> 
> This looks interesting -- some of those dependencies are very
> unexpected to me. I look forward to your take on what this output
> indicates. I don't understand why there is a connection between
> srv-nfs-var-cache-pacman.mount and btrbk.service, for example. (I
> included btrbk.service with source listings you requested below.)

The btrbk.* units are triggered by sysinit.target. And sysinit requires local-fs.target, which in turn requires that all fstab mounts are mounted (then the link to srv-nfs-var-cache-pacman.mount).
There no is any direct connection between btrbk and srv-nfs-var-cache-pacman.mount.

What is more interesting is the dependencies between srv-nfs-var-cache-pacman.mount and nfs-server.service. I suspect (but I don't have any proof) that systemd is confused by the tuple {btrfs subvolume, bind mount, nfs dependecies}.

What happens if you restart the nfs-server ?
> 
>>
>> [...]
>>>
>>> As mentioned, I have (temporarily) stopped unmounting these volumes
>>> and I just leave them mounted all the time. The logs now look like
>>> this:
>>>
>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: mounting btrbk btrtop volumes...
>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/root] (1 of
>>> 3) was already mounted. Nothing to do.
>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/home] (2 of
>>> 3) was already mounted. Nothing to do.
>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/user] (3 of
>>> 3) was already mounted. Nothing to do.
>>
>> This told another story. It seems that "btrbk" itself already try to mount the btrfs subvolume. I understood that it was the systemd unit to do that. Could you share the content of btrbk_run.sh ?
>>
> 
> # systemctl cat btrbk.service
> # /usr/lib/systemd/system/btrbk.service
> [Unit]
> Description=btrbk backup
> Documentation=man:btrbk(1)
> 
> [Service]
> Type=oneshot
> ExecStart=/usr/bin/btrbk run
> 
> # /etc/systemd/system/btrbk.service.d/override.conf
> [Service]
> ExecStart=
> ExecStart=/usr/local/bin/btrbk_run.sh
> 
> 
> # cat /usr/local/bin/btrbk_run.sh
> #!/bin/bash
> 
> /usr/local/bin/btrbk_mount
> 
> /usr/bin/btrbk --config /etc/btrbk/btrbk.conf run
> 
> # 2021-08-05 My first troubleshooting step is to disable unmounting
> these shares.
> # /usr/local/bin/btrbk-umount
> 
> 
> # cat /usr/local/bin/btrbk_mount
> #!/bin/bash
> 
> btrbk_mount() {
> 
> echo "mounting btrbk btrtop volumes..."
> 
> findmnt /mnt/btrtop/root
> if [[ $? -ne 0 ]]; then \
>    mount /mnt/btrtop/root
>    if [[ $? -ne 0 ]]; then \
>      echo "ERROR: failed to mount [/mnt/btrtop/root] (1 of 3)"
>    else
>      echo "OK: mounted [/mnt/btrtop/root] (1 of 3)"
>    fi
> else
>    echo "INFO: [/mnt/btrtop/root] (1 of 3) was already mounted. Nothing to do."
> fi
> findmnt /mnt/btrtop/home
> if [[ $? -ne 0 ]]; then \
>    mount /mnt/btrtop/home
>    if [[ $? -ne 0 ]]; then \
>      echo "ERROR: failed to mount [/mnt/btrtop/home] (2 of 3)"
>    else
>      echo "OK: mounted [/mnt/btrtop/home] (2 of 3)"
>    fi
> else
>    echo "INFO: [/mnt/btrtop/home] (2 of 3) was already mounted. Nothing to do."
> fi
> findmnt /mnt/btrtop/user
> if [[ $? -ne 0 ]]; then \
>    mount /mnt/btrtop/user
>    if [[ $? -ne 0 ]]; then \
>      echo "ERROR: failed to mount [/mnt/btrtop/user] (3 of 3)"
>    else
>      echo "OK: mounted mount [/mnt/btrtop/user] (3 of 3)"
>    fi
> else
>    echo "INFO: [/mnt/btrtop/user] (3 of 3) was already mounted. Nothing to do."
> fi
> 
> echo "Finished mounting btrbk btrtop volumes."
> 
> }
> 
> btrbk_mount
> 
> # end of file /usr/local/bin/btrbk_mount
> 
> 
>>
>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: Finished mounting btrbk btrtop volumes.
>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: mounting btrbk btrtop volumes...
>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/root] (1 of
>>> 3) was already mounted. Nothing to do.
>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/home] (2 of
>>> 3) was already mounted. Nothing to do.
>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/user] (3 of
>>> 3) was already mounted. Nothing to do.
>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: Finished mounting btrbk btrtop volumes.
>>>
>>>>
>>>>>
>>>>> The path /var/cache/pacman is not a subvolume, but it resides on btrfs
>>>>> subvolume @btrtop/snapshot. @btrtop/snapshot is normally mounted at
>>>>> "/" but for btrfs tasks, it is also mounted at /mnt/btrtop/root. This
>>>>> additional mount operation seems to be causing these nested mounts of
>>>>> my bind mount for  /srv/nfs/var/cache/pacman .
>>>>>
>>>>> P.S. I cannot test without using systemd. (I'm not even sure I
>>>>> remember how to use a non-systemd distro anymore!)
>>>>>
>>>>
>>>>
>>>> --
>>>> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
>>>> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
>>
>>
>> --
>> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
>> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5