Kernel 5.4 breaks fuse 2.X nonempty mount option

Kernel 5.4 breaks fuse 2.X nonempty mount option

[Stefan Priebe - Profihost AG]

Hi,

while using fuse 2.x and nonempty mount option - fuse mounts breaks
after upgrading from kernel 4.19 to 5.4.

While everything works fine under 4.19 - a mount on kernel 5.4 gives me
the following kernel message and an unmountable fuse mount:
[  201.289586] fuse: Unknown parameter 'nonempty'

While nonempty is clearly a fuse 2.x mount option, which should be
handled in userspace it seems it is passed to the kernel and older
kernels just ignored it.

It might be that those reworks have caused this:

commit 9bc61ab18b1d41f26dc06b9e6d3c203e65f83fe6
Author: David Howells <dhowells@redhat.com>
Date:   Sun Nov 4 03:19:03 2018 -0500

    vfs: Introduce fs_context, switch vfs_kern_mount() to it.

...

commit c7eb6869632a5d33b41d0a00d683b8395392b7ee
Author: David Howells <dhowells@redhat.com>
Date:   Mon Mar 25 16:38:31 2019 +0000

    vfs: subtype handling moved to fuse

Can anybody help?

Greets,
Stefan

Re: Kernel 5.4 breaks fuse 2.X nonempty mount option

[Stefan Priebe - Profihost AG]

added: miklos@szeredi.hu

Am 18.06.20 um 22:38 schrieb Stefan Priebe - Profihost AG:
> Hi,
> 
> while using fuse 2.x and nonempty mount option - fuse mounts breaks
> after upgrading from kernel 4.19 to 5.4.
> 
> While everything works fine under 4.19 - a mount on kernel 5.4 gives me
> the following kernel message and an unmountable fuse mount:
> [  201.289586] fuse: Unknown parameter 'nonempty'
> 
> While nonempty is clearly a fuse 2.x mount option, which should be
> handled in userspace it seems it is passed to the kernel and older
> kernels just ignored it.
> 
> It might be that those reworks have caused this:
> 
> commit 9bc61ab18b1d41f26dc06b9e6d3c203e65f83fe6
> Author: David Howells <dhowells@redhat.com>
> Date:   Sun Nov 4 03:19:03 2018 -0500
> 
>     vfs: Introduce fs_context, switch vfs_kern_mount() to it.
> 
> ...
> 
> commit c7eb6869632a5d33b41d0a00d683b8395392b7ee
> Author: David Howells <dhowells@redhat.com>
> Date:   Mon Mar 25 16:38:31 2019 +0000
> 
>     vfs: subtype handling moved to fuse
> 
> Can anybody help?
> 
> Greets,
> Stefan
> 

Re: Kernel 5.4 breaks fuse 2.X nonempty mount option

[David Howells]

Stefan Priebe - Profihost AG <s.priebe@profihost.ag> wrote:

> while using fuse 2.x and nonempty mount option - fuse mounts breaks
> after upgrading from kernel 4.19 to 5.4.

Can you give us an example mount commandline to try?

Thanks,
David

Re: Kernel 5.4 breaks fuse 2.X nonempty mount option

[Daniel Aberger - Profihost AG]

Am 19.06.20 um 09:47 schrieb David Howells:
> Stefan Priebe - Profihost AG <s.priebe@profihost.ag> wrote:
> 
>> while using fuse 2.x and nonempty mount option - fuse mounts breaks
>> after upgrading from kernel 4.19 to 5.4.
> 
> Can you give us an example mount commandline to try?


It's happening with the following /etc/fstab entry:

ceph-fuse		/var/log/pve/tasks	fuse.ceph
defaults,_netdev,noauto,x-systemd.automount,noatime,nonempty	0 0


Cheers,
Daniel

Re: Kernel 5.4 breaks fuse 2.X nonempty mount option

[Stefan Priebe - Profihost AG]

Am 19.06.20 um 09:47 schrieb David Howells:
> Stefan Priebe - Profihost AG <s.priebe@profihost.ag> wrote:
> 
>> while using fuse 2.x and nonempty mount option - fuse mounts breaks
>> after upgrading from kernel 4.19 to 5.4.
> 
> Can you give us an example mount commandline to try?

see fstab which daniel sent or:
ceph-fuse  /var/log/pve/tasks nonempty

Greets,
Stefan

> 
> Thanks,
> David
> 

Re: Kernel 5.4 breaks fuse 2.X nonempty mount option

[Stefan Priebe - Profihost AG]

may be more details:

Am 19.06.20 um 21:02 schrieb Stefan Priebe - Profihost AG:
> 
> Am 19.06.20 um 09:47 schrieb David Howells:
>> Stefan Priebe - Profihost AG <s.priebe@profihost.ag> wrote:
>>
>>> while using fuse 2.x and nonempty mount option - fuse mounts breaks
>>> after upgrading from kernel 4.19 to 5.4.
>>
>> Can you give us an example mount commandline to try?
> 
> see fstab which daniel sent or:
> ceph-fuse  /var/log/pve/tasks nonempty
> 
> Greets,
> Stefan


# ceph-fuse /var/log/pve/tasks -o rw,nonempty,noatime
2020-06-19 21:20:18.092 7f26af394f40 -1 init, newargv = 0x5627e08cc3d0
newargc=9fuse: bad mount point `/var/log/pve/tasks': Transport endpoint
is not connected

ceph-fuse[6955032020-06-19 21:20:18.092 7f26af394f40 -1
fuse_parse_cmdline failed.
]: fuse failed to initialize

# dmesg
[1226534.081647] fuse: Unknown parameter 'nonempty'

works fine with vanilla 4.19 kernel.

>>
>> Thanks,
>> David
>>

Re: Kernel 5.4 breaks fuse 2.X nonempty mount option

[Stefan Priebe - Profihost AG]

Hi David,

i did a git bisect and the breaking commit is:

commit c30da2e981a703c6b1d49911511f7ade8dac20be
Author: David Howells <dhowells@redhat.com>
Date:   Mon Mar 25 16:38:31 2019 +0000

    fuse: convert to use the new mount API

    Convert the fuse filesystem to the new internal mount API as the old
    one will be obsoleted and removed.  This allows greater flexibility in
    communication of mount parameters between userspace, the VFS and the
    filesystem.

    See Documentation/filesystems/mount_api.txt for more information.

    Signed-off-by: David Howells <dhowells@redhat.com>
    Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
    Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>

most probably due to the following diffences:


old default:
               default:
-                       return 0;


new default:
+       default:
+               return -EINVAL;


it seems the old API silently did ignore unknown parameters while the
new one fails with EINVAL.

I'm not sure who to blame here but this seems to break existing usespace
fuse programs or may be libfuse 2.9.X in general.

Greets,
Stefan

Am 19.06.20 um 21:02 schrieb Stefan Priebe - Profihost AG:
> 
> Am 19.06.20 um 09:47 schrieb David Howells:
>> Stefan Priebe - Profihost AG <s.priebe@profihost.ag> wrote:
>>
>>> while using fuse 2.x and nonempty mount option - fuse mounts breaks
>>> after upgrading from kernel 4.19 to 5.4.
>>
>> Can you give us an example mount commandline to try?
> 
> see fstab which daniel sent or:
> ceph-fuse  /var/log/pve/tasks nonempty
> 
> Greets,
> Stefan
> 
>>
>> Thanks,
>> David
>>

Re: Kernel 5.4 breaks fuse 2.X nonempty mount option

[Stefan Priebe - Profihost AG]

Hi,

while the commit is the right one my guess for the cause was only
correct by 50%.

Old code was an own parser for the fs params, which silently ignores
unknown params.

New code uses:
 476         opt = fs_parse(fc, &fuse_fs_parameters, param, &result);


which fails on unknown params.

I've no idea how to fix the kernel for libfuse 2.9.x again to allow
unknown parameters - it seems there is no option for fs_parse todo so.

Stefan
Am 21.06.20 um 21:33 schrieb Stefan Priebe - Profihost AG:
> Hi David,
> 
> i did a git bisect and the breaking commit is:
> 
> commit c30da2e981a703c6b1d49911511f7ade8dac20be
> Author: David Howells <dhowells@redhat.com>
> Date:   Mon Mar 25 16:38:31 2019 +0000
> 
>     fuse: convert to use the new mount API
> 
>     Convert the fuse filesystem to the new internal mount API as the old
>     one will be obsoleted and removed.  This allows greater flexibility in
>     communication of mount parameters between userspace, the VFS and the
>     filesystem.
> 
>     See Documentation/filesystems/mount_api.txt for more information.
> 
>     Signed-off-by: David Howells <dhowells@redhat.com>
>     Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
>     Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
> 
> most probably due to the following diffences:
> 
> 
> old default:
>                default:
> -                       return 0;
> 
> 
> new default:
> +       default:
> +               return -EINVAL;
> 
> 
> it seems the old API silently did ignore unknown parameters while the
> new one fails with EINVAL.
> 
> I'm not sure who to blame here but this seems to break existing usespace
> fuse programs or may be libfuse 2.9.X in general.
> 
> Greets,
> Stefan
> 
> Am 19.06.20 um 21:02 schrieb Stefan Priebe - Profihost AG:
>>
>> Am 19.06.20 um 09:47 schrieb David Howells:
>>> Stefan Priebe - Profihost AG <s.priebe@profihost.ag> wrote:
>>>
>>>> while using fuse 2.x and nonempty mount option - fuse mounts breaks
>>>> after upgrading from kernel 4.19 to 5.4.
>>>
>>> Can you give us an example mount commandline to try?
>>
>> see fstab which daniel sent or:
>> ceph-fuse  /var/log/pve/tasks nonempty
>>
>> Greets,
>> Stefan
>>
>>>
>>> Thanks,
>>> David
>>>

Re: Kernel 5.4 breaks fuse 2.X nonempty mount option

[Nikolaus Rath]

On Jun 18 2020, Stefan Priebe - Profihost AG <s.priebe@profihost.ag> wrote:
> Hi,
>
> while using fuse 2.x and nonempty mount option - fuse mounts breaks
> after upgrading from kernel 4.19 to 5.4.

IIRC nonempty is not processed by the kernel, but libfuse. This sounds like
you did a partial upgrade to libfuse 3.x (which dropped the option).

Best,
Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: Kernel 5.4 breaks fuse 2.X nonempty mount option

[Stefan Priebe - Profihost AG]

Am 22.06.20 um 07:18 schrieb Nikolaus Rath:
> On Jun 18 2020, Stefan Priebe - Profihost AG <s.priebe@profihost.ag> wrote:
>> Hi,
>>
>> while using fuse 2.x and nonempty mount option - fuse mounts breaks
>> after upgrading from kernel 4.19 to 5.4.
> 
> IIRC nonempty is not processed by the kernel, but libfuse. This sounds like
> you did a partial upgrade to libfuse 3.x (which dropped the option).

no for sure not. I already posted all details.

Greets,
Stefan

> 
> Best,
> Nikolaus
> 

Re: Kernel 5.4 breaks fuse 2.X nonempty mount option

[Miklos Szeredi]

On Sun, Jun 21, 2020 at 9:33 PM Stefan Priebe - Profihost AG
<s.priebe@profihost.ag> wrote:
>
> Hi David,
>
> i did a git bisect and the breaking commit is:
>
> commit c30da2e981a703c6b1d49911511f7ade8dac20be
> Author: David Howells <dhowells@redhat.com>
> Date:   Mon Mar 25 16:38:31 2019 +0000
>
>     fuse: convert to use the new mount API
>
>     Convert the fuse filesystem to the new internal mount API as the old
>     one will be obsoleted and removed.  This allows greater flexibility in
>     communication of mount parameters between userspace, the VFS and the
>     filesystem.
>
>     See Documentation/filesystems/mount_api.txt for more information.
>
>     Signed-off-by: David Howells <dhowells@redhat.com>
>     Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
>     Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
>
> most probably due to the following diffences:
>
>
> old default:
>                default:
> -                       return 0;
>
>
> new default:
> +       default:
> +               return -EINVAL;
>
>
> it seems the old API silently did ignore unknown parameters while the
> new one fails with EINVAL.

v4.19 has this:

static int parse_fuse_opt(char *opt, struct fuse_mount_data *d, int is_bdev,
                          struct user_namespace *user_ns)
{
[...]
        while ((p = strsep(&opt, ",")) != NULL) {
[...]
                token = match_token(p, tokens, args);
                switch (token) {
[...]
                default:
                        return 0;

and

static int fuse_fill_super(struct super_block *sb, void *data, int silent)
{
[...]
        err = -EINVAL;
        if (sb->s_flags & SB_MANDLOCK)
                goto err;

        sb->s_flags &= ~(SB_NOSEC | SB_I_VERSION);

        if (!parse_fuse_opt(data, &d, is_bdev, sb->s_user_ns))
                goto err;
[...]
 err:
        return err;
}

That looks like it returns -EINVAL for unknown options.  Can you
perform a "strace" on the old and the new kernel to see what the
difference is?

BTW, the hard rule is: userspace regressions caused by kernel changes
must be fixed.  It's just not clear where exactly this is coming from.

Thanks,
Miklos

Re: Kernel 5.4 breaks fuse 2.X nonempty mount option

[Stefan Priebe - Profihost AG]

Am 25.06.20 um 21:54 schrieb Miklos Szeredi:
> On Sun, Jun 21, 2020 at 9:33 PM Stefan Priebe - Profihost AG
> <s.priebe@profihost.ag> wrote:
>>
>> Hi David,
>>
>> i did a git bisect and the breaking commit is:
>>
>> commit c30da2e981a703c6b1d49911511f7ade8dac20be
>> Author: David Howells <dhowells@redhat.com>
>> Date:   Mon Mar 25 16:38:31 2019 +0000
>>
>>     fuse: convert to use the new mount API
>>
>>     Convert the fuse filesystem to the new internal mount API as the old
>>     one will be obsoleted and removed.  This allows greater flexibility in
>>     communication of mount parameters between userspace, the VFS and the
>>     filesystem.
>>
>>     See Documentation/filesystems/mount_api.txt for more information.
>>
>>     Signed-off-by: David Howells <dhowells@redhat.com>
>>     Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
>>     Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
>>
>> most probably due to the following diffences:
>>
>>
>> old default:
>>                default:
>> -                       return 0;
>>
>>
>> new default:
>> +       default:
>> +               return -EINVAL;
>>
>>
>> it seems the old API silently did ignore unknown parameters while the
>> new one fails with EINVAL.
> 
> v4.19 has this:
> 
> static int parse_fuse_opt(char *opt, struct fuse_mount_data *d, int is_bdev,
>                           struct user_namespace *user_ns)
> {
> [...]
>         while ((p = strsep(&opt, ",")) != NULL) {
> [...]
>                 token = match_token(p, tokens, args);
>                 switch (token) {
> [...]
>                 default:
>                         return 0;
> 
> and
> 
> static int fuse_fill_super(struct super_block *sb, void *data, int silent)
> {
> [...]
>         err = -EINVAL;
>         if (sb->s_flags & SB_MANDLOCK)
>                 goto err;
> 
>         sb->s_flags &= ~(SB_NOSEC | SB_I_VERSION);
> 
>         if (!parse_fuse_opt(data, &d, is_bdev, sb->s_user_ns))
>                 goto err;
> [...]
>  err:
>         return err;
> }
> 
> That looks like it returns -EINVAL for unknown options.  Can you
> perform a "strace" on the old and the new kernel to see what the
> difference is?
> 
> BTW, the hard rule is: userspace regressions caused by kernel changes
> must be fixed.  It's just not clear where exactly this is coming from.

Does a userspace strace really help? I did a git bisect between kernel
v5.3 (working) und v5.4 (not working) and it shows


commit c30da2e981a703c6b1d49911511f7ade8dac20be
Author: David Howells <dhowells@redhat.com>
Date:   Mon Mar 25 16:38:31 2019 +0000

    fuse: convert to use the new mount API

    Convert the fuse filesystem to the new internal mount API as the old
    one will be obsoleted and removed.  This allows greater flexibility in
    communication of mount parameters between userspace, the VFS and the
    filesystem.

    See Documentation/filesystems/mount_api.txt for more information.

    Signed-off-by: David Howells <dhowells@redhat.com>
    Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
    Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>

as the regression commit.

Stefan

> 
> Thanks,
> Miklos
> 

Re: Kernel 5.4 breaks fuse 2.X nonempty mount option

[Miklos Szeredi]

On Thu, Jun 25, 2020 at 10:10 PM Stefan Priebe - Profihost AG
<s.priebe@profihost.ag> wrote:
>
> Does a userspace strace really help? I did a git bisect between kernel
> v5.3 (working) und v5.4 (not working) and it shows

I cannot reproduce this with the libfuse2 examples.  Passing
"nonempty" as a mount(2) in either v5.3 or v5.4 results in -EINVAL.
So without an strace I cannot tell what is causing the regression.

Thanks,
Miklos

Re: Kernel 5.4 breaks fuse 2.X nonempty mount option

[Stefan Priebe - Profihost AG]

Hi Miklos,

Am 26.06.20 um 10:34 schrieb Miklos Szeredi:
> On Thu, Jun 25, 2020 at 10:10 PM Stefan Priebe - Profihost AG
> <s.priebe@profihost.ag> wrote:
>>
>> Does a userspace strace really help? I did a git bisect between kernel
>> v5.3 (working) und v5.4 (not working) and it shows
> 
> I cannot reproduce this with the libfuse2 examples.  Passing
> "nonempty" as a mount(2) in either v5.3 or v5.4 results in -EINVAL.
> So without an strace I cannot tell what is causing the regression.

the exact mount command is:
ceph-fuse /var/log/pve/tasks -o noatime,nonempty

so you mean i should just run strace -ff -s0 on it and throw the output
to pastebin?

Greets,
Stefan

Re: Kernel 5.4 breaks fuse 2.X nonempty mount option

[Miklos Szeredi]

On Mon, Jun 29, 2020 at 2:24 PM Stefan Priebe - Profihost AG
<s.priebe@profihost.ag> wrote:
>
> Hi Miklos,
>
> Am 26.06.20 um 10:34 schrieb Miklos Szeredi:
> > On Thu, Jun 25, 2020 at 10:10 PM Stefan Priebe - Profihost AG
> > <s.priebe@profihost.ag> wrote:
> >>
> >> Does a userspace strace really help? I did a git bisect between kernel
> >> v5.3 (working) und v5.4 (not working) and it shows
> >
> > I cannot reproduce this with the libfuse2 examples.  Passing
> > "nonempty" as a mount(2) in either v5.3 or v5.4 results in -EINVAL.
> > So without an strace I cannot tell what is causing the regression.
>
> the exact mount command is:
> ceph-fuse /var/log/pve/tasks -o noatime,nonempty
>
> so you mean i should just run strace -ff -s0 on it and throw the output
> to pastebin?

Yep.   I might need the arguments for the mount() syscalls though.

Thanks,
Miklos

Re: Kernel 5.4 breaks fuse 2.X nonempty mount option

[Miklos Szeredi]

[Back on-list]

On Mon, Jun 29, 2020 at 8:19 PM Stefan Priebe - Profihost AG
<s.priebe@profihost.ag> wrote:
>
> Hi,
>
> here we go (Off list):
>
> 4.19 kernel:
> # timeout 20 strace -ff -s1024 -- ceph-fuse -f /var/log/pve/tasks -o
> noatime,nonempty
>
> Output: see attached file
>
>
> 5.4.47 kernel:
> # timeout 20 strace -ff -s1024 -- ceph-fuse -f /var/log/pve/tasks -o
> noatime,nonempty
>
> Output: see attached file
>

Thanks.  This is the relevant line:

[pid  6448] mount("ceph-fuse", "/var/log/pve/tasks", 0x557e68f75cb0,
MS_MGC_VAL|MS_REMOUNT|MS_NOATIME, "nonempty") = 0

It's "mount -o remount -o unknownoption" that is broken.   Fix is a
little complex, because we want the old behavior on the old mount(2)
api, but the new behavior on the fsconfig(2) api.  Will post a patch
tomorrow.

Thanks,
Miklos