btrfs fails to mount on kernel 5.11.4 but works on 5.10.19

btrfs fails to mount on kernel 5.11.4 but works on 5.10.19

[Norbert Preining]

Dear all

(please cc)

not sure this is the right mailing list, but I cannot boot into 5.11.4 
it gives me
	devid 9 uui .....
	failed to read the system array: -2
	open_ctree failed
(only partial, typed in from photo)

OTOH, 5.10.19 boots without a hinch
$ btrfs fi show /
Label: none  uuid: 911600cb-bd76-4299-9445-666382e8ad20
        Total devices 8 FS bytes used 3.28TiB
        devid    1 size 899.01GiB used 670.00GiB path /dev/sdb3
        devid    2 size 489.05GiB used 271.00GiB path /dev/sdd
        devid    3 size 1.82TiB used 1.58TiB path /dev/sde1
        devid    4 size 931.51GiB used 708.00GiB path /dev/sdf1
        devid    5 size 1.82TiB used 1.58TiB path /dev/sdc1
        devid    7 size 931.51GiB used 675.00GiB path /dev/nvme2n1p1
        devid    8 size 931.51GiB used 680.03GiB path /dev/nvme1n1p1
        devid    9 size 931.51GiB used 678.03GiB path /dev/nvme0n1p1

That is a multi disk array with all data duplicated data/sys:

$ btrfs fi us -T /
Overall:
    Device size:                   8.63TiB
    Device allocated:              6.76TiB
    Device unallocated:            1.87TiB
    Device missing:                  0.00B
    Used:                          6.57TiB
    Free (estimated):              1.03TiB      (min: 1.03TiB)
    Free (statfs, df):             1.01TiB
    Data ratio:                       2.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

                  Data      Metadata System               
Id Path           RAID1     RAID1    RAID1     Unallocated
-- -------------- --------- -------- --------- -----------
 1 /dev/sdb3      662.00GiB  8.00GiB         -   229.01GiB
 2 /dev/sdd       271.00GiB        -         -     1.55TiB
 3 /dev/sde1        1.58TiB  7.00GiB         -   241.02GiB
 4 /dev/sdf1      701.00GiB  7.00GiB         -   223.51GiB
 5 /dev/sdc1        1.57TiB 10.00GiB         -   241.02GiB
 7 /dev/nvme2n1p1 675.00GiB        -         -   256.51GiB
 8 /dev/nvme1n1p1 673.00GiB  7.00GiB  32.00MiB   251.48GiB
 9 /dev/nvme0n1p1 671.00GiB  7.00GiB  32.00MiB   253.48GiB
-- -------------- --------- -------- --------- -----------
   Total            3.36TiB 23.00GiB  32.00MiB     3.21TiB
   Used             3.27TiB 15.70GiB 528.00KiB            
$

Is there something wrong with the filesystem? Or the kernel?
Any hint how to debug this?

(Please cc)

Thanks

Norbert

--
PREINING Norbert                              https://www.preining.info
Fujitsu Research Labs  +  IFMGA Guide + TU Wien + TeX Live + Debian Dev
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

Re: btrfs fails to mount on kernel 5.11.4 but works on 5.10.19

[Wang Yugui]

Hi,

If this is a boot btrfs filesystem, please try to mount it manually with
a live linux with the kernel 5.11.4?

When a boot btrfs filesystem with multiple disks, I have a question for
a little long time.

If some but not all of the disks are scaned, systemd will try to mount
it? and then btrfs mount will try to wait other disks to be scaned?

some but not all disks already make '/dev/disk/by-uuid/' or '/dev/disk/by-label/'
ready, and then systemd will try to mount it?

I have another problem about boot btrfs filesystem with multiple disks.
https://github.com/systemd-rhel/rhel-7/issues/129

Best Regards
Wang Yugui (wangyugui@e16-tech.com)
2021/03/08

> Dear all
> 
> (please cc)
> 
> not sure this is the right mailing list, but I cannot boot into 5.11.4 
> it gives me
> 	devid 9 uui .....
> 	failed to read the system array: -2
> 	open_ctree failed
> (only partial, typed in from photo)
> 
> OTOH, 5.10.19 boots without a hinch
> $ btrfs fi show /
> Label: none  uuid: 911600cb-bd76-4299-9445-666382e8ad20
>         Total devices 8 FS bytes used 3.28TiB
>         devid    1 size 899.01GiB used 670.00GiB path /dev/sdb3
>         devid    2 size 489.05GiB used 271.00GiB path /dev/sdd
>         devid    3 size 1.82TiB used 1.58TiB path /dev/sde1
>         devid    4 size 931.51GiB used 708.00GiB path /dev/sdf1
>         devid    5 size 1.82TiB used 1.58TiB path /dev/sdc1
>         devid    7 size 931.51GiB used 675.00GiB path /dev/nvme2n1p1
>         devid    8 size 931.51GiB used 680.03GiB path /dev/nvme1n1p1
>         devid    9 size 931.51GiB used 678.03GiB path /dev/nvme0n1p1
> 
> That is a multi disk array with all data duplicated data/sys:
> 
> $ btrfs fi us -T /
> Overall:
>     Device size:                   8.63TiB
>     Device allocated:              6.76TiB
>     Device unallocated:            1.87TiB
>     Device missing:                  0.00B
>     Used:                          6.57TiB
>     Free (estimated):              1.03TiB      (min: 1.03TiB)
>     Free (statfs, df):             1.01TiB
>     Data ratio:                       2.00
>     Metadata ratio:                   2.00
>     Global reserve:              512.00MiB      (used: 0.00B)
>     Multiple profiles:                  no
> 
>                   Data      Metadata System               
> Id Path           RAID1     RAID1    RAID1     Unallocated
> -- -------------- --------- -------- --------- -----------
>  1 /dev/sdb3      662.00GiB  8.00GiB         -   229.01GiB
>  2 /dev/sdd       271.00GiB        -         -     1.55TiB
>  3 /dev/sde1        1.58TiB  7.00GiB         -   241.02GiB
>  4 /dev/sdf1      701.00GiB  7.00GiB         -   223.51GiB
>  5 /dev/sdc1        1.57TiB 10.00GiB         -   241.02GiB
>  7 /dev/nvme2n1p1 675.00GiB        -         -   256.51GiB
>  8 /dev/nvme1n1p1 673.00GiB  7.00GiB  32.00MiB   251.48GiB
>  9 /dev/nvme0n1p1 671.00GiB  7.00GiB  32.00MiB   253.48GiB
> -- -------------- --------- -------- --------- -----------
>    Total            3.36TiB 23.00GiB  32.00MiB     3.21TiB
>    Used             3.27TiB 15.70GiB 528.00KiB            
> $
> 
> Is there something wrong with the filesystem? Or the kernel?
> Any hint how to debug this?
> 
> (Please cc)
> 
> Thanks
> 
> Norbert
> 
> --
> PREINING Norbert                              https://www.preining.info
> Fujitsu Research Labs  +  IFMGA Guide + TU Wien + TeX Live + Debian Dev
> GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

Re: btrfs fails to mount on kernel 5.11.4 but works on 5.10.19

[Norbert Preining]

Hi

(please cc)

thanks for your email. First some additional information. Since this
happened I searched and realized that there seem to have been a problem
with 5.12-rc1, which I tried for short time (checking whether AMD-GPU
hangs are fixed). Now I read that -rc1 is a btrfs-killer. I have swap
partition, not swap file, and 64G or RAM, so normally swap is not used,
though.

Then for your email:

On Mon, 08 Mar 2021, Wang Yugui wrote:
> If this is a boot btrfs filesystem, please try to mount it manually with
> a live linux with the kernel 5.11.4?

Hmmm, good idea, but how do I get a live system with that kernel? Not
that I know any having this as default by now.

I have booted systemrescue and run now
	btrfsck --check-data-csum
to see whether there are some other errors.

> When a boot btrfs filesystem with multiple disks, I have a question for
> a little long time.

What kind of problems do you see?

> If some but not all of the disks are scaned, systemd will try to mount
> it? and then btrfs mount will try to wait other disks to be scaned?

Hmm, but nothing has changed in the underlying boot configuration
(Debian/sid). Same system only kernel changes.

(Please cc)

Thanks and all the best

Norbert

> > not sure this is the right mailing list, but I cannot boot into 5.11.4 
> > it gives me
> > 	devid 9 uui .....  
> > 	failed to read the system array: -2
> > 	open_ctree failed
> > (only partial, typed in from photo)
> > 
> > OTOH, 5.10.19 boots without a hinch
> > $ btrfs fi show /
> > Label: none  uuid: 911600cb-bd76-4299-9445-666382e8ad20
> >         Total devices 8 FS bytes used 3.28TiB
> >         devid    1 size 899.01GiB used 670.00GiB path /dev/sdb3
> >         devid    2 size 489.05GiB used 271.00GiB path /dev/sdd
> >         devid    3 size 1.82TiB used 1.58TiB path /dev/sde1
> >         devid    4 size 931.51GiB used 708.00GiB path /dev/sdf1
> >         devid    5 size 1.82TiB used 1.58TiB path /dev/sdc1
> >         devid    7 size 931.51GiB used 675.00GiB path /dev/nvme2n1p1
> >         devid    8 size 931.51GiB used 680.03GiB path /dev/nvme1n1p1
> >         devid    9 size 931.51GiB used 678.03GiB path /dev/nvme0n1p1
> > 
> > That is a multi disk array with all data duplicated data/sys:
> > 
> > $ btrfs fi us -T /
> > Overall:
> >     Device size:                   8.63TiB
> >     Device allocated:              6.76TiB
> >     Device unallocated:            1.87TiB
> >     Device missing:                  0.00B
> >     Used:                          6.57TiB
> >     Free (estimated):              1.03TiB      (min: 1.03TiB)
> >     Free (statfs, df):             1.01TiB
> >     Data ratio:                       2.00
> >     Metadata ratio:                   2.00
> >     Global reserve:              512.00MiB      (used: 0.00B)
> >     Multiple profiles:                  no
> > 
> >                   Data      Metadata System               
> > Id Path           RAID1     RAID1    RAID1     Unallocated
> > -- -------------- --------- -------- --------- -----------
> >  1 /dev/sdb3      662.00GiB  8.00GiB         -   229.01GiB
> >  2 /dev/sdd       271.00GiB        -         -     1.55TiB
> >  3 /dev/sde1        1.58TiB  7.00GiB         -   241.02GiB
> >  4 /dev/sdf1      701.00GiB  7.00GiB         -   223.51GiB
> >  5 /dev/sdc1        1.57TiB 10.00GiB         -   241.02GiB
> >  7 /dev/nvme2n1p1 675.00GiB        -         -   256.51GiB
> >  8 /dev/nvme1n1p1 673.00GiB  7.00GiB  32.00MiB   251.48GiB
> >  9 /dev/nvme0n1p1 671.00GiB  7.00GiB  32.00MiB   253.48GiB
> > -- -------------- --------- -------- --------- -----------
> >    Total            3.36TiB 23.00GiB  32.00MiB     3.21TiB
> >    Used             3.27TiB 15.70GiB 528.00KiB            
> > $
> > 
> > Is there something wrong with the filesystem? Or the kernel?
> > Any hint how to debug this?

--
PREINING Norbert                              https://www.preining.info
Fujitsu Research Labs  +  IFMGA Guide + TU Wien + TeX Live + Debian Dev
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

Re: btrfs fails to mount on kernel 5.11.4 but works on 5.10.19

[Chris Murphy]

On Sun, Mar 7, 2021 at 4:28 PM Norbert Preining <norbert@preining.info> wrote:
>
> Dear all
>
> (please cc)
>
> not sure this is the right mailing list, but I cannot boot into 5.11.4
> it gives me
>         devid 9 uui .....
>         failed to read the system array: -2
>         open_ctree failed
> (only partial, typed in from photo)

Post the photo? This is a generic message and we need to see more
information. Is devid 9 missing?

Does the initrd on this system contain?

/usr/lib/udev/rules.d/64-btrfs.rules

That will wait until all devices are available before attempting to
mount. If it's not in the initrd, it won't wait and it's prone to
races, and you can often get mount failures because not all devices
are ready to be mounted.







>
> OTOH, 5.10.19 boots without a hinch
> $ btrfs fi show /
> Label: none  uuid: 911600cb-bd76-4299-9445-666382e8ad20
>         Total devices 8 FS bytes used 3.28TiB
>         devid    1 size 899.01GiB used 670.00GiB path /dev/sdb3
>         devid    2 size 489.05GiB used 271.00GiB path /dev/sdd
>         devid    3 size 1.82TiB used 1.58TiB path /dev/sde1
>         devid    4 size 931.51GiB used 708.00GiB path /dev/sdf1
>         devid    5 size 1.82TiB used 1.58TiB path /dev/sdc1
>         devid    7 size 931.51GiB used 675.00GiB path /dev/nvme2n1p1
>         devid    8 size 931.51GiB used 680.03GiB path /dev/nvme1n1p1
>         devid    9 size 931.51GiB used 678.03GiB path /dev/nvme0n1p1

This seems to be a somewhat risky setup or at least highly performance
variable. Any single device that fails will result in boot failure.


-- 
Chris Murphy

Re: btrfs fails to mount on kernel 5.11.4 but works on 5.10.19

[Chris Murphy]

On Sun, Mar 7, 2021 at 5:25 PM Norbert Preining <norbert@preining.info> wrote:
>
> Hi
>
> (please cc)
>
> thanks for your email. First some additional information. Since this
> happened I searched and realized that there seem to have been a problem
> with 5.12-rc1, which I tried for short time (checking whether AMD-GPU
> hangs are fixed). Now I read that -rc1 is a btrfs-killer. I have swap
> partition, not swap file, and 64G or RAM, so normally swap is not used,
> though.

That bug should not have affected the dedication swap partition case.



-- 
Chris Murphy

Re: btrfs fails to mount on kernel 5.11.4 but works on 5.10.19

[Norbert Preining]

[-- Attachment #1: Type: text/plain, Size: 1706 bytes --]

Hi Chris,

thanks a lot for your email!

> Post the photo? This is a generic message and we need to see more
> information. Is devid 9 missing?

Yes it is missing. I attach the screenshot.

> Does the initrd on this system contain?
>   /usr/lib/udev/rules.d/64-btrfs.rules

Good question, I don't have access now since I am doing the 
	btrfsck --check-data-csum
which takes a bit of time. On a similar system (but without multiple
disks and no raid), but otherwise the same Debian/sid, I don't have
the rules file in the initrd (only in /lib/udev/rules.d/64-btrfs.rules
on the running system)

> That will wait until all devices are available before attempting to
> mount. If it's not in the initrd, it won't wait and it's prone to
> races, and you can often get mount failures because not all devices
> are ready to be mounted.

Ahh, ok that could be an issue - I will try to include the udev rules
and try again with 5.11.4. Strange that it didn't before, though.

> This seems to be a somewhat risky setup or at least highly performance
> variable. Any single device that fails will result in boot failure.

I had that, and it can normally be easily recovered via mounting in
degraded mode, removing or replacing the device, and reboot.
Worked last time ;-) What would your suggestion be instead?

> > hangs are fixed). Now I read that -rc1 is a btrfs-killer. I have swap

> That bug should not have affected the dedication swap partition case.

Good to hear, thanks!

Best

Norbert

--
PREINING Norbert                              https://www.preining.info
Fujitsu Research Labs  +  IFMGA Guide + TU Wien + TeX Live + Debian Dev
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

[-- Attachment #2: boot-error-btrfs.jpg --]
[-- Type: image/jpeg, Size: 351571 bytes --]

Re: btrfs fails to mount on kernel 5.11.4 but works on 5.10.19

[Norbert Preining]

Hi Chris,

once more ..

> > Does the initrd on this system contain?
> >   /usr/lib/udev/rules.d/64-btrfs.rules

No, it didn't.

Now I added it, and with 64-btrfs.rules available in the initrd I still
get the same error (see previous screenshot) :-(

Best

Norbert

--
PREINING Norbert                              https://www.preining.info
Fujitsu Research Labs  +  IFMGA Guide + TU Wien + TeX Live + Debian Dev
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

Re: btrfs fails to mount on kernel 5.11.4 but works on 5.10.19

[Chris Murphy]

On Sun, Mar 7, 2021 at 7:18 PM Norbert Preining <norbert@preining.info> wrote:
>
> Hi Chris,
>
> once more ..
>
> > > Does the initrd on this system contain?
> > >   /usr/lib/udev/rules.d/64-btrfs.rules
>
> No, it didn't.
>
> Now I added it, and with 64-btrfs.rules available in the initrd I still
> get the same error (see previous screenshot) :-(

I suspect something is wrong with devid 9 in that case. If it's a
dracut system, then it waits indefinitely for sysroot. You'll need to
boot with something like rd.break=pre-mount and see first if you can
mount normally to /sysroot, but if devid 9 is still missing then mount
degraded and replace that device. Or otherwise find out why it's
missing.

I don't think the scrub helps right now, the issue is the device is
missing. Where scrub does help is if the device reappears for normal
mount following previous degraded mount - the scrub is needed to get
the missing device caught up with the rest.


-- 
Chris Murphy

Re: btrfs fails to mount on kernel 5.11.4 but works on 5.10.19

[Norbert Preining]

Hi Chris,

On Sun, 07 Mar 2021, Chris Murphy wrote:
> I suspect something is wrong with devid 9 in that case. If it's a
> dracut system, then it waits indefinitely for sysroot. You'll need to
> boot with something like rd.break=pre-mount and see first if you can
> mount normally to /sysroot, but if devid 9 is still missing then mount
> degraded and replace that device. Or otherwise find out why it's
> missing.

Ok, I will see what I can do. It isn't a dracut system, but
initramfs-tools AFAIS (not that I ever touched or decided anything in
this respect).

I will try to get to that state. (Only that USB devices seem to be hosed
in Debian's initramfs ... no kbd atm :-(

As said, it is just puzzling that 5.10.19 boots without a hinch and
without a warning, just running it now.

> I don't think the scrub helps right now, the issue is the device is

Ok, I have already killed it ;-)

Thanks a lot

Norbert

--
PREINING Norbert                              https://www.preining.info
Fujitsu Research Labs  +  IFMGA Guide + TU Wien + TeX Live + Debian Dev
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

[SOLVED] Re: btrfs fails to mount on kernel 5.11.4 but works on 5.10.19

[Norbert Preining]

Hi Chris, hi all,

case closed. The reason was that some NVMe devices weren't recognized.
I don't have any idea **why** this happened, but I cleaned out the whole
models tree, made a complete rebuild of the kernel, and now it works.

Sorry for the noise.

And I also switched to dracut on the go, thanks for the suggestions.

Best

Norbert

--
PREINING Norbert                              https://www.preining.info
Fujitsu Research Labs  +  IFMGA Guide + TU Wien + TeX Live + Debian Dev
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13