mounting raid1 btrfs - occasional devid missing / open_ctree failed

mounting raid1 btrfs - occasional devid missing / open_ctree failed

[Michal Soltys]

Hi,

I have btrfs set for root filesystem using two separate partitions, 
essentially:

abel: 'xs22_root'  uuid: 3754849d-46ab-4db6-8434-2c578b1f6bba
         Total devices 2 FS bytes used 8.39GiB
         devid    1 size 34.00GiB used 14.01GiB path /dev/sdb2
         devid    2 size 34.00GiB used 14.01GiB path /dev/sda2


Data,RAID1: Size:12.00GiB, Used:8.13GiB
    /dev/sda2      12.00GiB
    /dev/sdb2      12.00GiB

Metadata,RAID1: Size:2.00GiB, Used:256.62MiB
    /dev/sda2       2.00GiB
    /dev/sdb2       2.00GiB

System,RAID1: Size:8.00MiB, Used:16.00KiB
    /dev/sda2       8.00MiB
    /dev/sdb2       8.00MiB

Unallocated:
    /dev/sda2      19.99GiB
    /dev/sdb2      19.99GiB


Occasionally, during reboots (all of them clean of course) I get:

devid 2 uuid <uuid> is missing
failed to ready system array: -2
open_ctree faield

Which drops me into initramfs commandline.

When that happens, one of the partitions is mountable, the other is not. 
Mounting the filesystem via the "good" one also fixes the "bad" one.

Tested with 4.20.11 and 5.0.2 kernels.

Any idea what could be wrong ?
Does btrfs require full umount via pivoted initramfs during shutdown ?

Re: mounting raid1 btrfs - occasional devid missing / open_ctree failed

[Hans van Kranenburg]

On 3/14/19 4:44 PM, Michal Soltys wrote:
> Hi,
> 
> I have btrfs set for root filesystem using two separate partitions,
> essentially:
> 
> abel: 'xs22_root'  uuid: 3754849d-46ab-4db6-8434-2c578b1f6bba
>         Total devices 2 FS bytes used 8.39GiB
>         devid    1 size 34.00GiB used 14.01GiB path /dev/sdb2
>         devid    2 size 34.00GiB used 14.01GiB path /dev/sda2
> 
> 
> Data,RAID1: Size:12.00GiB, Used:8.13GiB
>    /dev/sda2      12.00GiB
>    /dev/sdb2      12.00GiB
> 
> Metadata,RAID1: Size:2.00GiB, Used:256.62MiB
>    /dev/sda2       2.00GiB
>    /dev/sdb2       2.00GiB
> 
> System,RAID1: Size:8.00MiB, Used:16.00KiB
>    /dev/sda2       8.00MiB
>    /dev/sdb2       8.00MiB
> 
> Unallocated:
>    /dev/sda2      19.99GiB
>    /dev/sdb2      19.99GiB
> 
> 
> Occasionally, during reboots (all of them clean of course) I get:
> 
> devid 2 uuid <uuid> is missing
> failed to ready system array: -2
> open_ctree faield
> 
> Which drops me into initramfs commandline.

It might try to mount before all disks are visible. This can happen when
e.g. a raid controller needs a little bit more time to initialize.

Try adding rootdelay=10 to your linux command line, or any other number,
and then look at logging when booting (so no quiet on your linux command
line).

> When that happens, one of the partitions is mountable, the other is not.
> Mounting the filesystem via the "good" one also fixes the "bad" one.
> 
> Tested with 4.20.11 and 5.0.2 kernels.
> 
> Any idea what could be wrong ?
> Does btrfs require full umount via pivoted initramfs during shutdown ?
Hans

Re: mounting raid1 btrfs - occasional devid missing / open_ctree failed

[Michal Soltys]

On 3/14/19 4:57 PM, Hans van Kranenburg wrote:
> On 3/14/19 4:44 PM, Michal Soltys wrote:
>> Hi,
>>
>> I have btrfs set for root filesystem using two separate partitions,
>> essentially:
>>
>> abel: 'xs22_root'  uuid: 3754849d-46ab-4db6-8434-2c578b1f6bba
>>          Total devices 2 FS bytes used 8.39GiB
>>          devid    1 size 34.00GiB used 14.01GiB path /dev/sdb2
>>          devid    2 size 34.00GiB used 14.01GiB path /dev/sda2
>>
>>
>> Data,RAID1: Size:12.00GiB, Used:8.13GiB
>>     /dev/sda2      12.00GiB
>>     /dev/sdb2      12.00GiB
>>
>> Metadata,RAID1: Size:2.00GiB, Used:256.62MiB
>>     /dev/sda2       2.00GiB
>>     /dev/sdb2       2.00GiB
>>
>> System,RAID1: Size:8.00MiB, Used:16.00KiB
>>     /dev/sda2       8.00MiB
>>     /dev/sdb2       8.00MiB
>>
>> Unallocated:
>>     /dev/sda2      19.99GiB
>>     /dev/sdb2      19.99GiB
>>
>>
>> Occasionally, during reboots (all of them clean of course) I get:
>>
>> devid 2 uuid <uuid> is missing
>> failed to ready system array: -2
>> open_ctree faield
>>
>> Which drops me into initramfs commandline.
> 
> It might try to mount before all disks are visible. This can happen when
> e.g. a raid controller needs a little bit more time to initialize.
> 
> Try adding rootdelay=10 to your linux command line, or any other number,
> and then look at logging when booting (so no quiet on your linux command
> line).
> 

It remains unmountable from initramfs's commandline with both disks 
(partitions) present. Filesystem is mounted formally via uuid, so it 
needs both disks present before udev/blkid can pick the correct devices.

>> When that happens, one of the partitions is mountable, the other is not.
>> Mounting the filesystem via the "good" one also fixes the "bad" one.
>>
>> Tested with 4.20.11 and 5.0.2 kernels.
>>
>> Any idea what could be wrong ?
>> Does btrfs require full umount via pivoted initramfs during shutdown ?
> Hans
> 

Re: mounting raid1 btrfs - occasional devid missing / open_ctree failed

[Michal Soltys]

On 3/14/19 5:09 PM, Michal Soltys wrote:
> On 3/14/19 4:57 PM, Hans van Kranenburg wrote:
>> On 3/14/19 4:44 PM, Michal Soltys wrote:
>>> 
>>> <cut>
>>>
>>> Occasionally, during reboots (all of them clean of course) I get:
>>>
>>> devid 2 uuid <uuid> is missing
>>> failed to ready system array: -2
>>> open_ctree faield
>>>
>>> Which drops me into initramfs commandline.
>>
>> It might try to mount before all disks are visible. This can happen when
>> e.g. a raid controller needs a little bit more time to initialize.
>>
>> Try adding rootdelay=10 to your linux command line, or any other number,
>> and then look at logging when booting (so no quiet on your linux command
>> line).
>>
> 
> It remains unmountable from initramfs's commandline with both disks 
> (partitions) present. Filesystem is mounted formally via uuid, so it 
> needs both disks present before udev/blkid can pick the correct devices.
> 

E.g.

mount /dev/sdr2 /newroot -> fails
mount /dev/sds2 /newroot -> ok
umount /newroot
mount /dev/sdr2 /newroot -> ok

>>> When that happens, one of the partitions is mountable, the other is not.
>>> Mounting the filesystem via the "good" one also fixes the "bad" one.
>>>
>>> Tested with 4.20.11 and 5.0.2 kernels.
>>>
>>> Any idea what could be wrong ?
>>> Does btrfs require full umount via pivoted initramfs during shutdown ?
>> Hans
>>
> 
> 

Re: mounting raid1 btrfs - occasional devid missing / open_ctree failed

[Hans van Kranenburg]

On 3/14/19 5:17 PM, Michal Soltys wrote:
> On 3/14/19 5:09 PM, Michal Soltys wrote:
>> On 3/14/19 4:57 PM, Hans van Kranenburg wrote:
>>> On 3/14/19 4:44 PM, Michal Soltys wrote:
>>>>
>>>> <cut>
>>>>
>>>> Occasionally, during reboots (all of them clean of course) I get:
>>>>
>>>> devid 2 uuid <uuid> is missing
>>>> failed to ready system array: -2
>>>> open_ctree faield
>>>>
>>>> Which drops me into initramfs commandline.
>>>
>>> It might try to mount before all disks are visible. This can happen when
>>> e.g. a raid controller needs a little bit more time to initialize.
>>>
>>> Try adding rootdelay=10 to your linux command line, or any other number,
>>> and then look at logging when booting (so no quiet on your linux command
>>> line).
>>>
>>
>> It remains unmountable from initramfs's commandline with both disks
>> (partitions) present. Filesystem is mounted formally via uuid, so it
>> needs both disks present before udev/blkid can pick the correct devices.

When using UUID in fstab/initramfs, all btrfs devices show up having the
same UUID, so if any of them is present, it can try mounting.

> E.g.
> 
> mount /dev/sdr2 /newroot -> fails

Are you typing this manually? And what's the error? Does it complain
about the devid of sds2?

> mount /dev/sds2 /newroot -> ok

While doing this you trigger the system to have a look, and at that
moment it will suddenly discover the disk, and now it sees them both and
it mounts...

> umount /newroot
> mount /dev/sdr2 /newroot -> ok

...even when using sdr2, because when doing the things above all disks
got discovered.

It's of course possible you have another problem.My suggestion was just
a been-there-done-that. But, in that case posting more logging might help.

>>>> When that happens, one of the partitions is mountable, the other is
>>>> not.
>>>> Mounting the filesystem via the "good" one also fixes the "bad" one.
>>>>
>>>> Tested with 4.20.11 and 5.0.2 kernels.
>>>>
>>>> Any idea what could be wrong ?
>>>> Does btrfs require full umount via pivoted initramfs during shutdown ?

Hans

Re: mounting raid1 btrfs - occasional devid missing / open_ctree failed

[Michal Soltys]

On 3/14/19 5:21 PM, Hans van Kranenburg wrote:
> On 3/14/19 5:17 PM, Michal Soltys wrote:
>> On 3/14/19 5:09 PM, Michal Soltys wrote:
>>> On 3/14/19 4:57 PM, Hans van Kranenburg wrote:
>>>> On 3/14/19 4:44 PM, Michal Soltys wrote:
>>>>>
>>>>> <cut>
>>>>>
>>>>> Occasionally, during reboots (all of them clean of course) I get:
>>>>>
>>>>> devid 2 uuid <uuid> is missing
>>>>> failed to ready system array: -2
>>>>> open_ctree faield
>>>>>
>>>>> Which drops me into initramfs commandline.
>>>>
>>>> It might try to mount before all disks are visible. This can happen when
>>>> e.g. a raid controller needs a little bit more time to initialize.
>>>>
>>>> Try adding rootdelay=10 to your linux command line, or any other number,
>>>> and then look at logging when booting (so no quiet on your linux command
>>>> line).
>>>>
>>>
>>> It remains unmountable from initramfs's commandline with both disks
>>> (partitions) present. Filesystem is mounted formally via uuid, so it
>>> needs both disks present before udev/blkid can pick the correct devices.
> 
> When using UUID in fstab/initramfs, all btrfs devices show up having the
> same UUID, so if any of them is present, it can try mounting.
> 
>> E.g.
>>
>> mount /dev/sdr2 /newroot -> fails
> 
> Are you typing this manually? And what's the error? Does it complain
> about the devid of sds2?

Yes.

Output captured via serial console (debug level) below.

> 
>> mount /dev/sds2 /newroot -> ok
> 
> While doing this you trigger the system to have a look, and at that
> moment it will suddenly discover the disk, and now it sees them both and
> it mounts...
> 
>> umount /newroot
>> mount /dev/sdr2 /newroot -> ok
> 
> ...even when using sdr2, because when doing the things above all disks
> got discovered.
> 
> It's of course possible you have another problem.My suggestion was just
> a been-there-done-that. But, in that case posting more logging might help.
> 

[   25.579637] md127: detected capacity change from 0 to 5401587744768
[   25.728050] PM: Image not found (code -22)
[   25.779465] BTRFS: device label xs22_root devid 1 transid 25364 /dev/sds2
[   25.810606] BTRFS info (device sds2): disk space caching is enabled
[   25.841015] BTRFS info (device sds2): has skinny extents
[   25.870085] BTRFS error (device sds2): devid 2 uuid c93d387c-1acc-4bea-aeae-59a440ac7991 is missing
[   25.903337] BTRFS error (device sds2): failed to read the system array: -2
[   25.957884] BTRFS error (device sds2): open_ctree failed
[   26.002614] printk: systemd-udevd: 53 output lines suppressed due to ratelimiting


BusyBox v1.22.1 (Debian 1:1.22.0-19+b3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs) mount /dev/sds2 /root
[   42.638417] BTRFS info (device sds2): disk space caching is enabled
[   42.673164] BTRFS info (device sds2): has skinny extents
[   42.706947] BTRFS error (device sds2): devid 2 uuid c93d387c-1acc-4bea-aeae-59a440ac7991 is missing
[   42.743085] BTRFS error (device sds2): failed to read the system array: -2
[   42.789905] BTRFS error (device sds2): open_ctree failed
mount: mounting /dev/sds2 on /root failed: Invalid argument
(initramfs) mount /dev/sdr2 /root
[   54.216006] BTRFS: device label xs22_root devid 2 transid 25364 /dev/sdr2
[   54.249603] BTRFS info (device sds2): disk space caching is enabled
[   54.281158] BTRFS info (device sds2): has skinny extents
[   54.318920] BTRFS info (device sds2): enabling ssd optimizations
[   54.352494] BTRFS info (device sds2): checking UUID tree
(initramfs) umount /root
(initramfs) mount /dev/sds2 /root
[   63.458869] BTRFS info (device sds2): disk space caching is enabled
[   63.490016] BTRFS info (device sds2): has skinny extents
[   63.529345] BTRFS info (device sds2): enabling ssd optimizations

Re: mounting raid1 btrfs - occasional devid missing / open_ctree failed

[Hans van Kranenburg]

Hi,

On 3/14/19 6:54 PM, Michal Soltys wrote:
> On 3/14/19 5:21 PM, Hans van Kranenburg wrote:
>> On 3/14/19 5:17 PM, Michal Soltys wrote:
>>> On 3/14/19 5:09 PM, Michal Soltys wrote:
>>>> On 3/14/19 4:57 PM, Hans van Kranenburg wrote:
>>>>> On 3/14/19 4:44 PM, Michal Soltys wrote:
>>>>>>
>>>>>> <cut>
>>>>>>
>>>>>> Occasionally, during reboots (all of them clean of course) I get:
>>>>>>
>>>>>> devid 2 uuid <uuid> is missing
>>>>>> failed to ready system array: -2
>>>>>> open_ctree faield
>>>>>>
>>>>>> Which drops me into initramfs commandline.
>>>>>
>>>>> It might try to mount before all disks are visible. This can happen when
>>>>> e.g. a raid controller needs a little bit more time to initialize.
>>>>>
>>>>> Try adding rootdelay=10 to your linux command line, or any other number,
>>>>> and then look at logging when booting (so no quiet on your linux command
>>>>> line).
>>>>>
>>>>
>>>> It remains unmountable from initramfs's commandline with both disks
>>>> (partitions) present. Filesystem is mounted formally via uuid, so it
>>>> needs both disks present before udev/blkid can pick the correct devices.
>>
>> When using UUID in fstab/initramfs, all btrfs devices show up having the
>> same UUID, so if any of them is present, it can try mounting.
>>
>>> E.g.
>>>
>>> mount /dev/sdr2 /newroot -> fails
>>
>> Are you typing this manually? And what's the error? Does it complain
>> about the devid of sds2?
> 
> Yes.

Did you try rootdelay? The stuff below looks like it tried mounting too
soon indeed.

> Output captured via serial console (debug level) below.
> 
>>
>>> mount /dev/sds2 /newroot -> ok
>>
>> While doing this you trigger the system to have a look, and at that
>> moment it will suddenly discover the disk, and now it sees them both and
>> it mounts...
>>
>>> umount /newroot
>>> mount /dev/sdr2 /newroot -> ok
>>
>> ...even when using sdr2, because when doing the things above all disks
>> got discovered.
>>
>> It's of course possible you have another problem.My suggestion was just
>> a been-there-done-that. But, in that case posting more logging might help.
>>
> 
> [   25.579637] md127: detected capacity change from 0 to 5401587744768
> [   25.728050] PM: Image not found (code -22)
> [   25.779465] BTRFS: device label xs22_root devid 1 transid 25364 /dev/sds2

/dev/sds2 is available, a mount is attempted...

> [   25.810606] BTRFS info (device sds2): disk space caching is enabled
> [   25.841015] BTRFS info (device sds2): has skinny extents
> [   25.870085] BTRFS error (device sds2): devid 2 uuid c93d387c-1acc-4bea-aeae-59a440ac7991 is missing
> [   25.903337] BTRFS error (device sds2): failed to read the system array: -2
> [   25.957884] BTRFS error (device sds2): open_ctree failed

...which fails because devid 2 can't be found.

> [   26.002614] printk: systemd-udevd: 53 output lines suppressed due to ratelimiting
> 
> 
> BusyBox v1.22.1 (Debian 1:1.22.0-19+b3) built-in shell (ash)
> Enter 'help' for a list of built-in commands.

Here...

> (initramfs) mount /dev/sds2 /root
> [   42.638417] BTRFS info (device sds2): disk space caching is enabled
> [   42.673164] BTRFS info (device sds2): has skinny extents
> [   42.706947] BTRFS error (device sds2): devid 2 uuid c93d387c-1acc-4bea-aeae-59a440ac7991 is missing
> [   42.743085] BTRFS error (device sds2): failed to read the system array: -2
> [   42.789905] BTRFS error (device sds2): open_ctree failed
> mount: mounting /dev/sds2 on /root failed: Invalid argument

...the same happens again. devid 2 is not in the picture yet.

But now...

> (initramfs) mount /dev/sdr2 /root
> [   54.216006] BTRFS: device label xs22_root devid 2 transid 25364 /dev/sdr2

...you explicitly point to /dev/sdr2, which triggers having a look at it
and it is discovered as being a btrfs block device.

Now the mount succeeds:

> [   54.249603] BTRFS info (device sds2): disk space caching is enabled
> [   54.281158] BTRFS info (device sds2): has skinny extents
> [   54.318920] BTRFS info (device sds2): enabling ssd optimizations
> [   54.352494] BTRFS info (device sds2): checking UUID tree

And after this...

> (initramfs) umount /root

...they're both still visible, so any other mount pointing at any of the
two succeeds.

> (initramfs) mount /dev/sds2 /root
> [   63.458869] BTRFS info (device sds2): disk space caching is enabled
> [   63.490016] BTRFS info (device sds2): has skinny extents
> [   63.529345] BTRFS info (device sds2): enabling ssd optimizations
> 

Hans

Re: mounting raid1 btrfs - occasional devid missing / open_ctree failed

[Andrei Borzenkov]

14.03.2019 19:17, Michal Soltys пишет:
> On 3/14/19 5:09 PM, Michal Soltys wrote:
>> On 3/14/19 4:57 PM, Hans van Kranenburg wrote:
>>> On 3/14/19 4:44 PM, Michal Soltys wrote:
>>>>
>>>> <cut>
>>>>
>>>> Occasionally, during reboots (all of them clean of course) I get:
>>>>
>>>> devid 2 uuid <uuid> is missing
>>>> failed to ready system array: -2
>>>> open_ctree faield
>>>>
>>>> Which drops me into initramfs commandline.
>>>
>>> It might try to mount before all disks are visible. This can happen when
>>> e.g. a raid controller needs a little bit more time to initialize.
>>>
>>> Try adding rootdelay=10 to your linux command line, or any other number,
>>> and then look at logging when booting (so no quiet on your linux command
>>> line).
>>>
>>
>> It remains unmountable from initramfs's commandline with both disks
>> (partitions) present. Filesystem is mounted formally via uuid, so it
>> needs both disks present before udev/blkid can pick the correct devices.
>>
> 
> E.g.
> 
> mount /dev/sdr2 /newroot -> fails
> mount /dev/sds2 /newroot -> ok
> umount /newroot
> mount /dev/sdr2 /newroot -> ok
> 

This implies that neither device was scanned with equivalent of "btrfs
device scan". After attempt to mount both partitions kernel is aware of
them and from now on it succeeds.

What "dmesg | grep -i btrfs" shows before and after these commands?

>>>> When that happens, one of the partitions is mountable, the other is
>>>> not.
>>>> Mounting the filesystem via the "good" one also fixes the "bad" one.
>>>>
>>>> Tested with 4.20.11 and 5.0.2 kernels.
>>>>
>>>> Any idea what could be wrong ?
>>>> Does btrfs require full umount via pivoted initramfs during shutdown ?
>>> Hans
>>>
>>
>>
> 

Re: mounting raid1 btrfs - occasional devid missing / open_ctree failed

[Michal Soltys]

On 19/03/15 04:59, Andrei Borzenkov wrote:
> 14.03.2019 19:17, Michal Soltys пишет:
>> On 3/14/19 5:09 PM, Michal Soltys wrote:
>>> On 3/14/19 4:57 PM, Hans van Kranenburg wrote:
>>>> On 3/14/19 4:44 PM, Michal Soltys wrote:
>>>>>
>>>>> <cut>
>>>>>
>>>>> Occasionally, during reboots (all of them clean of course) I get:
>>>>>
>>>>> devid 2 uuid <uuid> is missing
>>>>> failed to ready system array: -2
>>>>> open_ctree faield
>>>>>
>>>>> Which drops me into initramfs commandline.
>>>>
>>>> It might try to mount before all disks are visible. This can happen when
>>>> e.g. a raid controller needs a little bit more time to initialize.
>>>>
>>>> Try adding rootdelay=10 to your linux command line, or any other number,
>>>> and then look at logging when booting (so no quiet on your linux command
>>>> line).
>>>>
>>>
>>> It remains unmountable from initramfs's commandline with both disks
>>> (partitions) present. Filesystem is mounted formally via uuid, so it
>>> needs both disks present before udev/blkid can pick the correct devices.
>>>
>> 
>> E.g.
>> 
>> mount /dev/sdr2 /newroot -> fails
>> mount /dev/sds2 /newroot -> ok
>> umount /newroot
>> mount /dev/sdr2 /newroot -> ok
>> 
> 
> This implies that neither device was scanned with equivalent of "btrfs
> device scan". After attempt to mount both partitions kernel is aware of
> them and from now on it succeeds.
> 

That would point to some really weird thing [not] being done in 
initramfs in its debian variation. I'll look closer into it.

Thanks for pointing that out, the 'btrfs device scan' requirement 
completely slipped my mind.

> What "dmesg | grep -i btrfs" shows before and after these commands?
> 

Will verify and report.

>>>>> When that happens, one of the partitions is mountable, the other is
>>>>> not.
>>>>> Mounting the filesystem via the "good" one also fixes the "bad" one.
>>>>>
>>>>> Tested with 4.20.11 and 5.0.2 kernels.
>>>>>
>>>>> Any idea what could be wrong ?
>>>>> Does btrfs require full umount via pivoted initramfs during shutdown ?
>>>> Hans
>>>>
>>>
>>>
>> 
> 
> 

Re: mounting raid1 btrfs - occasional devid missing / open_ctree failed

[Michal Soltys]

On 3/15/19 10:36 AM, Michal Soltys wrote:
> On 19/03/15 04:59, Andrei Borzenkov wrote:
>> 14.03.2019 19:17, Michal Soltys пишет:
>>> On 3/14/19 5:09 PM, Michal Soltys wrote:
>>>> On 3/14/19 4:57 PM, Hans van Kranenburg wrote:
>>>>> On 3/14/19 4:44 PM, Michal Soltys wrote:
>>>>>>
>>>>>> <cut>
> 
> That would point to some really weird thing [not] being done in 
> initramfs in its debian variation. I'll look closer into it.
> 
> Thanks for pointing that out, the 'btrfs device scan' requirement 
> completely slipped my mind.
> 
>> What "dmesg | grep -i btrfs" shows before and after these commands?
>>
> 
> Will verify and report.
> 

Turned out to be a peculiar race issue with debian's initrams-tools's btrfs script (basically executing device scan during its local-premount stage). As it turned out it can be executed very early - in my case, before any block device is even visible in /dev. So no wonder later root mounting didn't work correctly.

Hans, Andrei - thanks for help.

For the record:

[   15.445002] scsi 0:0:7:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[   15.545118] input:   USB Keyboard as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.1/2-1.1.3/2-1.1.3:1.0/0003:09DA:0260.0003/input/input2
->>> [   15.754962] ::: Starting BTRFS pre-mount
[   15.801450] scsi 0:0:8:0: Direct-Access     ATA      ST2000VX000-1CU1 CV22 PQ: 0 ANSI: 6
->>> [   15.803632] ::: Finishing BTRFS pre-mount
[   15.851614] scsi 0:0:8:0: SATA: handle(0x000f), sas_addr(0x5003048000b28b90), phy(16), device_name(0x0000000000000000)
[   15.854133] hid-generic 0003:09DA:0260.0003: input,hidraw2: USB HID v1.10 Keyboard [  USB Keyboard] on usb-0000:00:1d.0-1.1.3/input0

.... then a bit later devices become available:

[   19.691841] sd 0:0:0:0: Power-on or device reset occurred
[   19.692125] sd 0:0:1:0: Power-on or device reset occurred
[   19.740664] sd 0:0:2:0: Power-on or device reset occurred
[   19.740803] scsi 1:0:0:0: Direct-Access     ATA      INTEL SSDSC2BB08 0370 PQ: 0 ANSI: 5
[   19.741009] sd 0:0:12:0: Power-on or device reset occurred
[   19.741166] scsi 2:0:0:0: Direct-Access     ATA      INTEL SSDSC2BB08 0370 PQ: 0 ANSI: 5
[   19.741348] sd 0:0:3:0: Power-on or device reset occurred
[   19.742136] sd 0:0:0:0: [sda] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)

... and finally:

[   25.221537] md127: detected capacity change from 0 to 5401587744768
[   25.363978] PM: Image not found (code -22)
[   25.424970] BTRFS: device label xs22_root devid 1 transid 33775 /dev/sds2
[   25.455095] BTRFS info (device sds2): disk space caching is enabled
[   25.483435] BTRFS info (device sds2): has skinny extents
[   25.513225] BTRFS error (device sds2): devid 2 uuid c93d387c-1acc-4bea-aeae-59a440ac7991 is missing
[   25.546619] BTRFS error (device sds2): failed to read the system array: -2
[   25.602175] BTRFS error (device sds2): open_ctree failed