All of lore.kernel.org
 help / color / mirror / Atom feed
* Recover from "couldn't read tree root"?
@ 2021-06-20 20:37 Nathan Dehnel
  2021-06-20 21:09 ` Chris Murphy
  2021-06-20 21:19 ` Chris Murphy
  0 siblings, 2 replies; 8+ messages in thread
From: Nathan Dehnel @ 2021-06-20 20:37 UTC (permalink / raw)
  To: Btrfs BTRFS

A machine failed to boot, so I tried to mount its root partition from
systemrescuecd, which failed:

[ 5404.240019] BTRFS info (device bcache3): disk space caching is enabled
[ 5404.240022] BTRFS info (device bcache3): has skinny extents
[ 5404.243195] BTRFS error (device bcache3): parent transid verify
failed on 3004631449600 wanted 1420882 found 1420435
[ 5404.243279] BTRFS error (device bcache3): parent transid verify
failed on 3004631449600 wanted 1420882 found 1420435
[ 5404.243362] BTRFS error (device bcache3): parent transid verify
failed on 3004631449600 wanted 1420882 found 1420435
[ 5404.243432] BTRFS error (device bcache3): parent transid verify
failed on 3004631449600 wanted 1420882 found 1420435
[ 5404.243435] BTRFS warning (device bcache3): couldn't read tree root
[ 5404.244114] BTRFS error (device bcache3): open_ctree failed

btrfs rescue super-recover -v /dev/bcache0 returned this:

parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
Ignoring transid failure
ERROR: could not setup extent tree
Failed to recover bad superblocks

uname -a:

Linux sysrescue 5.10.34-1-lts #1 SMP Sun, 02 May 2021 12:41:09 +0000
x86_64 GNU/Linux

btrfs --version:

btrfs-progs v5.10.1

btrfs fi show:

Label: none  uuid: 76189222-b60d-4402-a7ff-141f057e8574
        Total devices 10 FS bytes used 1.50TiB
        devid    1 size 931.51GiB used 311.03GiB path /dev/bcache3
        devid    2 size 931.51GiB used 311.00GiB path /dev/bcache2
        devid    3 size 931.51GiB used 311.00GiB path /dev/bcache1
        devid    4 size 931.51GiB used 311.00GiB path /dev/bcache0
        devid    5 size 931.51GiB used 311.00GiB path /dev/bcache4
        devid    6 size 931.51GiB used 311.00GiB path /dev/bcache8
        devid    7 size 931.51GiB used 311.00GiB path /dev/bcache6
        devid    8 size 931.51GiB used 311.03GiB path /dev/bcache9
        devid    9 size 931.51GiB used 311.03GiB path /dev/bcache7
        devid   10 size 931.51GiB used 311.03GiB path /dev/bcache5

Is this filesystem recoverable?

(Sorry, re-sending because I forgot to add a subject)

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: Recover from "couldn't read tree root"?
  2021-06-20 20:37 Recover from "couldn't read tree root"? Nathan Dehnel
@ 2021-06-20 21:09 ` Chris Murphy
  2021-06-20 21:31   ` Nathan Dehnel
  2021-06-20 21:19 ` Chris Murphy
  1 sibling, 1 reply; 8+ messages in thread
From: Chris Murphy @ 2021-06-20 21:09 UTC (permalink / raw)
  To: Nathan Dehnel; +Cc: Btrfs BTRFS

On Sun, Jun 20, 2021 at 2:38 PM Nathan Dehnel <ncdehnel@gmail.com> wrote:
>
> A machine failed to boot, so I tried to mount its root partition from
> systemrescuecd, which failed:
>
> [ 5404.240019] BTRFS info (device bcache3): disk space caching is enabled
> [ 5404.240022] BTRFS info (device bcache3): has skinny extents
> [ 5404.243195] BTRFS error (device bcache3): parent transid verify
> failed on 3004631449600 wanted 1420882 found 1420435
> [ 5404.243279] BTRFS error (device bcache3): parent transid verify
> failed on 3004631449600 wanted 1420882 found 1420435
> [ 5404.243362] BTRFS error (device bcache3): parent transid verify
> failed on 3004631449600 wanted 1420882 found 1420435
> [ 5404.243432] BTRFS error (device bcache3): parent transid verify
> failed on 3004631449600 wanted 1420882 found 1420435
> [ 5404.243435] BTRFS warning (device bcache3): couldn't read tree root
> [ 5404.244114] BTRFS error (device bcache3): open_ctree failed
>
> btrfs rescue super-recover -v /dev/bcache0 returned this:
>
> parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> Ignoring transid failure
> ERROR: could not setup extent tree
> Failed to recover bad superblocks
>
> uname -a:
>
> Linux sysrescue 5.10.34-1-lts #1 SMP Sun, 02 May 2021 12:41:09 +0000
> x86_64 GNU/Linux
>
> btrfs --version:
>
> btrfs-progs v5.10.1
>
> btrfs fi show:
>
> Label: none  uuid: 76189222-b60d-4402-a7ff-141f057e8574
>         Total devices 10 FS bytes used 1.50TiB
>         devid    1 size 931.51GiB used 311.03GiB path /dev/bcache3
>         devid    2 size 931.51GiB used 311.00GiB path /dev/bcache2
>         devid    3 size 931.51GiB used 311.00GiB path /dev/bcache1
>         devid    4 size 931.51GiB used 311.00GiB path /dev/bcache0
>         devid    5 size 931.51GiB used 311.00GiB path /dev/bcache4
>         devid    6 size 931.51GiB used 311.00GiB path /dev/bcache8
>         devid    7 size 931.51GiB used 311.00GiB path /dev/bcache6
>         devid    8 size 931.51GiB used 311.03GiB path /dev/bcache9
>         devid    9 size 931.51GiB used 311.03GiB path /dev/bcache7
>         devid   10 size 931.51GiB used 311.03GiB path /dev/bcache5
>
> Is this filesystem recoverable?
>> (Sorry, re-sending because I forgot to add a subject)

Definitely don't write any irreversible changes, such as a repair
attempt, to anything until you understand what what wrong or it'll
make recovery harder or impossible.

Was bcache in write back or write through mode?

What's the configuration? Can you supply something like

lsblk -o NAME,FSTYPE,SIZE,FSUSE%,MOUNTPOINT,UUID,MIN-IO,SCHED,DISC-GRAN,MODEL



-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: Recover from "couldn't read tree root"?
  2021-06-20 20:37 Recover from "couldn't read tree root"? Nathan Dehnel
  2021-06-20 21:09 ` Chris Murphy
@ 2021-06-20 21:19 ` Chris Murphy
  2021-06-20 21:48   ` Nathan Dehnel
  1 sibling, 1 reply; 8+ messages in thread
From: Chris Murphy @ 2021-06-20 21:19 UTC (permalink / raw)
  To: Nathan Dehnel; +Cc: Btrfs BTRFS

On Sun, Jun 20, 2021 at 2:38 PM Nathan Dehnel <ncdehnel@gmail.com> wrote:
>
> A machine failed to boot, so I tried to mount its root partition from
> systemrescuecd, which failed:
>
> [ 5404.240019] BTRFS info (device bcache3): disk space caching is enabled
> [ 5404.240022] BTRFS info (device bcache3): has skinny extents
> [ 5404.243195] BTRFS error (device bcache3): parent transid verify
> failed on 3004631449600 wanted 1420882 found 1420435
> [ 5404.243279] BTRFS error (device bcache3): parent transid verify
> failed on 3004631449600 wanted 1420882 found 1420435
> [ 5404.243362] BTRFS error (device bcache3): parent transid verify
> failed on 3004631449600 wanted 1420882 found 1420435
> [ 5404.243432] BTRFS error (device bcache3): parent transid verify
> failed on 3004631449600 wanted 1420882 found 1420435
> [ 5404.243435] BTRFS warning (device bcache3): couldn't read tree root
> [ 5404.244114] BTRFS error (device bcache3): open_ctree failed

This is generally bad, and means some lower layer did something wrong,
such as getting write order incorrect, i.e. failing to properly honor
flush/fua. Recovery can be difficult and take a while.
https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#parent_transid_verify_failed

I suggest searching logs since the last time this file system was
working, because the above error is indicating a problem that's
already happened and what we need to know is what happened, if
possible. Something like this:

journalctl --since=-5d -k -o short-monotonic --no-hostname | grep
"Linux version\| ata\|bcache\|Btrfs\|BTRFS\|] hd\| scsi\| sd\| sdhci\|
mmc\| nvme\| usb\| vd"



> btrfs rescue super-recover -v /dev/bcache0 returned this:
>
> parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> Ignoring transid failure
> ERROR: could not setup extent tree
> Failed to recover bad superblocks

OK something is really wrong if you're not able to see a single
superblock on any of the bcache devices. Every member device has  3
super blocks, given the sizes you've provided. For there to not be a
single one is a spectacular failure as if the bcache cache device
isn't returning correct information for any of them. So I'm gonna
guess a single shared SSD, which is a single point of failure, and
it's spitting out garbage or zeros. But I'm not even close to a bcache
expert so you might want to ask bcache developers how to figure out
what state bcache is in and whether and how to safely decouple it from
the backing drives so that you can engage in recovery attempts.

If bcache mode is write through, there's a chance the backing drives
have valid btrfs metadata, and it's just that on read the SSD is
returning bogus information.





-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: Recover from "couldn't read tree root"?
  2021-06-20 21:09 ` Chris Murphy
@ 2021-06-20 21:31   ` Nathan Dehnel
  2021-06-20 22:19     ` Chris Murphy
  2021-06-20 22:53     ` Chris Murphy
  0 siblings, 2 replies; 8+ messages in thread
From: Nathan Dehnel @ 2021-06-20 21:31 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Btrfs BTRFS

>Was bcache in write back or write through mode?
Writeback.

>What's the configuration?
NAME        FSTYPE              SIZE FSUSE% MOUNTPOINT
UUID                                  MIN-IO SCHED       DISC-GRAN
MODEL
loop0       squashfs          655.6M   100% /run/archiso/sfs/airootfs
                                        512 mq-deadline        0B
sda                           238.5G
                                        512 mq-deadline      512B
C300-CTFDDAC256MAG
├─sda1                            2M
                                        512 mq-deadline      512B
├─sda2      linux_raid_member   512M
325a2f12-18b8-27f7-2f81-f554a9b0fccc     512 mq-deadline      512B
│ └─md126   vfat              511.9M
EF35-0411                                512                  512B
└─sda3      linux_raid_member    16G
93ed641f-394b-2122-7525-b3311aaac6b8     512 mq-deadline      512B
  └─md125   swap                 16G
9ea84fb7-8bd7-4a0e-91fe-398790643066 1048576                  512B
sdb                           232.9G
                                        512 mq-deadline      512B
Samsung_SSD_850_EVO_250GB
├─sdb1                            2M
                                        512 mq-deadline      512B
├─sdb2      linux_raid_member   512M
325a2f12-18b8-27f7-2f81-f554a9b0fccc     512 mq-deadline      512B
│ └─md126   vfat              511.9M
EF35-0411                                512                  512B
└─sdb3      linux_raid_member    16G
93ed641f-394b-2122-7525-b3311aaac6b8     512 mq-deadline      512B
  └─md125   swap                 16G
9ea84fb7-8bd7-4a0e-91fe-398790643066 1048576                  512B
sdc         btrfs             931.5G
12bcde5c-b3ae-4fa6-8e17-0a4b564f1ba1     512 mq-deadline        0B
WDC_WD1002FAEX-00Z3A0
└─sdc1      bcache            931.5G
f34b26ea-8229-4f3f-bdc5-29c5fe16eaae     512 mq-deadline        0B
  └─bcache0 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
sdd         btrfs             931.5G
12bcde5c-b3ae-4fa6-8e17-0a4b564f1ba1     512 mq-deadline        0B
WDC_WD1002FAEX-00Z3A0
└─sdd1      bcache            931.5G
beb25260-1b36-473f-93c4-7ef016a62f44     512 mq-deadline        0B
  └─bcache1 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
sde         btrfs             931.5G
12bcde5c-b3ae-4fa6-8e17-0a4b564f1ba1    4096 mq-deadline        0B
WDC_WD1003FZEX-00MK2A0
└─sde1      bcache            931.5G
21b55c83-c951-4e4f-affc-0b9bf54c8783    4096 mq-deadline        0B
  └─bcache2 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
sdf         btrfs             931.5G
12bcde5c-b3ae-4fa6-8e17-0a4b564f1ba1     512 mq-deadline        0B
WDC_WD1002FAEX-00Z3A0
└─sdf1      bcache            931.5G
d4d2b9d6-077d-4328-b2cd-14f6db259955     512 mq-deadline        0B
  └─bcache3 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
sdg         btrfs             931.5G
12bcde5c-b3ae-4fa6-8e17-0a4b564f1ba1     512 mq-deadline        0B
ST1000NM0011
└─sdg1      bcache            931.5G
a8513a01-c6be-4bec-b3f9-a5797225d304     512 mq-deadline        0B
  └─bcache4 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
sdh                           931.5G
                                        512 mq-deadline        0B
WDC_WD1002FAEX-00Z3A0
└─sdh1      bcache            931.5G
ffeacab7-ff42-453c-b012-58b119236fa5     512 mq-deadline        0B
  └─bcache5 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
sdi         btrfs             931.5G
12bcde5c-b3ae-4fa6-8e17-0a4b564f1ba1     512 mq-deadline        0B
WDC_WD1002FAEX-00Y9A0
└─sdi1      bcache            931.5G
f3f4d706-7d73-4b48-a5b3-9802fc0de978     512 mq-deadline        0B
  └─bcache6 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
sdj         btrfs             931.5G
12bcde5c-b3ae-4fa6-8e17-0a4b564f1ba1    4096 mq-deadline        0B
WDC_WD1003FZEX-00MK2A0
└─sdj1      bcache            931.5G
64d10dda-4ac2-44d4-941a-362ccb5ddbba    4096 mq-deadline        0B
  └─bcache7 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
sdk         btrfs             931.5G
12bcde5c-b3ae-4fa6-8e17-0a4b564f1ba1     512 mq-deadline        0B
WDC_WD1002FAEX-00Y9A0
└─sdk1      bcache            931.5G
c3ddc718-f700-4360-82c9-7db76114e3f6     512 mq-deadline        0B
  └─bcache8 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
sdl         btrfs             931.5G
12bcde5c-b3ae-4fa6-8e17-0a4b564f1ba1     512 mq-deadline        0B
WDC_WD1002FAEX-00Z3A0
└─sdl1      bcache            931.5G
2bf5ac80-cdf6-4c0c-9434-bcdc4626abff     512 mq-deadline        0B
  └─bcache9 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
sdm         iso9660            14.9G
2021-05-08-11-22-02-00                   512 mq-deadline        0B
USB_2.0_FD
├─sdm1      iso9660             717M   100% /run/archiso/bootmnt
2021-05-08-11-22-02-00                   512 mq-deadline        0B
└─sdm2      vfat                1.4M
0A52-44A0                                512 mq-deadline        0B
nvme0n1     linux_raid_member  13.4G
4703551c-4570-b6c8-7dda-991b93b99c9a     512 none             512B
INTEL MEMPEK1W016GA
└─md127     bcache             13.4G
dfda7dc0-07a4-40bf-b5b8-e3458c181ce4   16384                  512B
  ├─bcache0 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
  ├─bcache1 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
  ├─bcache2 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
  ├─bcache3 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
  ├─bcache4 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
  ├─bcache5 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
  ├─bcache6 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
  ├─bcache7 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
  ├─bcache8 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
  └─bcache9 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
nvme1n1     linux_raid_member  13.4G
4703551c-4570-b6c8-7dda-991b93b99c9a     512 none             512B
INTEL MEMPEK1W016GA
└─md127     bcache             13.4G
dfda7dc0-07a4-40bf-b5b8-e3458c181ce4   16384                  512B
  ├─bcache0 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
  ├─bcache1 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
  ├─bcache2 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
  ├─bcache3 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
  ├─bcache4 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
  ├─bcache5 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
  ├─bcache6 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
  ├─bcache7 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
  ├─bcache8 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B
  └─bcache9 btrfs             931.5G
76189222-b60d-4402-a7ff-141f057e8574     512                  512B

On Sun, Jun 20, 2021 at 9:09 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sun, Jun 20, 2021 at 2:38 PM Nathan Dehnel <ncdehnel@gmail.com> wrote:
> >
> > A machine failed to boot, so I tried to mount its root partition from
> > systemrescuecd, which failed:
> >
> > [ 5404.240019] BTRFS info (device bcache3): disk space caching is enabled
> > [ 5404.240022] BTRFS info (device bcache3): has skinny extents
> > [ 5404.243195] BTRFS error (device bcache3): parent transid verify
> > failed on 3004631449600 wanted 1420882 found 1420435
> > [ 5404.243279] BTRFS error (device bcache3): parent transid verify
> > failed on 3004631449600 wanted 1420882 found 1420435
> > [ 5404.243362] BTRFS error (device bcache3): parent transid verify
> > failed on 3004631449600 wanted 1420882 found 1420435
> > [ 5404.243432] BTRFS error (device bcache3): parent transid verify
> > failed on 3004631449600 wanted 1420882 found 1420435
> > [ 5404.243435] BTRFS warning (device bcache3): couldn't read tree root
> > [ 5404.244114] BTRFS error (device bcache3): open_ctree failed
> >
> > btrfs rescue super-recover -v /dev/bcache0 returned this:
> >
> > parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> > parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> > parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> > parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> > parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> > Ignoring transid failure
> > ERROR: could not setup extent tree
> > Failed to recover bad superblocks
> >
> > uname -a:
> >
> > Linux sysrescue 5.10.34-1-lts #1 SMP Sun, 02 May 2021 12:41:09 +0000
> > x86_64 GNU/Linux
> >
> > btrfs --version:
> >
> > btrfs-progs v5.10.1
> >
> > btrfs fi show:
> >
> > Label: none  uuid: 76189222-b60d-4402-a7ff-141f057e8574
> >         Total devices 10 FS bytes used 1.50TiB
> >         devid    1 size 931.51GiB used 311.03GiB path /dev/bcache3
> >         devid    2 size 931.51GiB used 311.00GiB path /dev/bcache2
> >         devid    3 size 931.51GiB used 311.00GiB path /dev/bcache1
> >         devid    4 size 931.51GiB used 311.00GiB path /dev/bcache0
> >         devid    5 size 931.51GiB used 311.00GiB path /dev/bcache4
> >         devid    6 size 931.51GiB used 311.00GiB path /dev/bcache8
> >         devid    7 size 931.51GiB used 311.00GiB path /dev/bcache6
> >         devid    8 size 931.51GiB used 311.03GiB path /dev/bcache9
> >         devid    9 size 931.51GiB used 311.03GiB path /dev/bcache7
> >         devid   10 size 931.51GiB used 311.03GiB path /dev/bcache5
> >
> > Is this filesystem recoverable?
> >> (Sorry, re-sending because I forgot to add a subject)
>
> Definitely don't write any irreversible changes, such as a repair
> attempt, to anything until you understand what what wrong or it'll
> make recovery harder or impossible.
>
> Was bcache in write back or write through mode?
>
> What's the configuration? Can you supply something like
>
> lsblk -o NAME,FSTYPE,SIZE,FSUSE%,MOUNTPOINT,UUID,MIN-IO,SCHED,DISC-GRAN,MODEL
>
>
>
> --
> Chris Murphy

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: Recover from "couldn't read tree root"?
  2021-06-20 21:19 ` Chris Murphy
@ 2021-06-20 21:48   ` Nathan Dehnel
  0 siblings, 0 replies; 8+ messages in thread
From: Nathan Dehnel @ 2021-06-20 21:48 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Btrfs BTRFS

>I suggest searching logs since the last time this file system was
working, because the above error is indicating a problem that's
already happened and what we need to know is what happened, if
possible. Something like this:

>journalctl --since=-5d -k -o short-monotonic --no-hostname | grep
"Linux version\| ata\|bcache\|Btrfs\|BTRFS\|] hd\| scsi\| sd\| sdhci\|
mmc\| nvme\| usb\| vd"

Unfortunately I put my journal logs in a different subvolume so they
wouldn't bloat my snapshots so they weren't included in my backups.

>So I'm gonna guess a single shared SSD, which is a single point of failure, and
it's spitting out garbage or zeros.
It's 2 SSDs in mdraid RAID10.

>But I'm not even close to a bcache expert so you might want to ask bcache developers how to figure out
what state bcache is in and whether and how to safely decouple it from
the backing drives so that you can engage in recovery attempts.
They didn't respond the last couple of times I've asked a question on
their irc or mailing list.

On Sun, Jun 20, 2021 at 9:19 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sun, Jun 20, 2021 at 2:38 PM Nathan Dehnel <ncdehnel@gmail.com> wrote:
> >
> > A machine failed to boot, so I tried to mount its root partition from
> > systemrescuecd, which failed:
> >
> > [ 5404.240019] BTRFS info (device bcache3): disk space caching is enabled
> > [ 5404.240022] BTRFS info (device bcache3): has skinny extents
> > [ 5404.243195] BTRFS error (device bcache3): parent transid verify
> > failed on 3004631449600 wanted 1420882 found 1420435
> > [ 5404.243279] BTRFS error (device bcache3): parent transid verify
> > failed on 3004631449600 wanted 1420882 found 1420435
> > [ 5404.243362] BTRFS error (device bcache3): parent transid verify
> > failed on 3004631449600 wanted 1420882 found 1420435
> > [ 5404.243432] BTRFS error (device bcache3): parent transid verify
> > failed on 3004631449600 wanted 1420882 found 1420435
> > [ 5404.243435] BTRFS warning (device bcache3): couldn't read tree root
> > [ 5404.244114] BTRFS error (device bcache3): open_ctree failed
>
> This is generally bad, and means some lower layer did something wrong,
> such as getting write order incorrect, i.e. failing to properly honor
> flush/fua. Recovery can be difficult and take a while.
> https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#parent_transid_verify_failed
>
> I suggest searching logs since the last time this file system was
> working, because the above error is indicating a problem that's
> already happened and what we need to know is what happened, if
> possible. Something like this:
>
> journalctl --since=-5d -k -o short-monotonic --no-hostname | grep
> "Linux version\| ata\|bcache\|Btrfs\|BTRFS\|] hd\| scsi\| sd\| sdhci\|
> mmc\| nvme\| usb\| vd"
>
>
>
> > btrfs rescue super-recover -v /dev/bcache0 returned this:
> >
> > parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> > parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> > parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> > parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> > parent transid verify failed on 3004631449600 wanted 1420882 found 1420435
> > Ignoring transid failure
> > ERROR: could not setup extent tree
> > Failed to recover bad superblocks
>
> OK something is really wrong if you're not able to see a single
> superblock on any of the bcache devices. Every member device has  3
> super blocks, given the sizes you've provided. For there to not be a
> single one is a spectacular failure as if the bcache cache device
> isn't returning correct information for any of them. So I'm gonna
> guess a single shared SSD, which is a single point of failure, and
> it's spitting out garbage or zeros. But I'm not even close to a bcache
> expert so you might want to ask bcache developers how to figure out
> what state bcache is in and whether and how to safely decouple it from
> the backing drives so that you can engage in recovery attempts.
>
> If bcache mode is write through, there's a chance the backing drives
> have valid btrfs metadata, and it's just that on read the SSD is
> returning bogus information.
>
>
>
>
>
> --
> Chris Murphy

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: Recover from "couldn't read tree root"?
  2021-06-20 21:31   ` Nathan Dehnel
@ 2021-06-20 22:19     ` Chris Murphy
  2021-06-20 22:53     ` Chris Murphy
  1 sibling, 0 replies; 8+ messages in thread
From: Chris Murphy @ 2021-06-20 22:19 UTC (permalink / raw)
  To: Nathan Dehnel; +Cc: Chris Murphy, Btrfs BTRFS

The two Intel MEMPEK1W016GA's are in raid10, but you aren't really
protected unless the drive reports a discrete read error. Only in that
case would the md driver know to use the mirror copy. While it
certainly should sooner report a read error than return zeros or
garbage, this is the situation we're in with SSDs. Is that what's
happening? *shrug* Needs more investigation. But it's at either the
bcache or mdadm level, near as I can tell.

Was there a crash or power failure while using this array by any chance?


Chris Murphy

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: Recover from "couldn't read tree root"?
  2021-06-20 21:31   ` Nathan Dehnel
  2021-06-20 22:19     ` Chris Murphy
@ 2021-06-20 22:53     ` Chris Murphy
  2021-06-22  3:26       ` Nathan Dehnel
  1 sibling, 1 reply; 8+ messages in thread
From: Chris Murphy @ 2021-06-20 22:53 UTC (permalink / raw)
  To: Nathan Dehnel; +Cc: Chris Murphy, Btrfs BTRFS

On Sun, Jun 20, 2021 at 3:31 PM Nathan Dehnel <ncdehnel@gmail.com> wrote:
>
> >Was bcache in write back or write through mode?
> Writeback.

Ok that's bad in this configuration because it means all the writes go
to the SSD and could be there for minutes, hours, days, or longer.
That means it's even possible the current supers are only on the SSDs,
as well as other critical btrfs metadata.

My best guess now is to assume one of the drives is bad and spewing
garbage or zeros. And assemble the array degraded with just one SSD
drive, and see if you can mount. If not, then it's the other SSD you
need to assemble degraded. There's a way to set a drive manually as
faulty so it won't assemble; I also thought of using sysfs but on my
own system, /sys/block/nvme0n1/device/delete does not exist like it
does for SATA SSDs.

Next you have to wrestle with this dilemma. If you pick the bad SSD,
you don't want bcache flushing anything from it to your HDDs or it'll
just corrupt them, right? if you pick the good SSD, you actually do
want bcache to flush it all to the drives, so they're in a good state
and you can optionally decouple the SSD entirely so that you're left
with just the individual drives again.

I think you might want to use 'blockdev --setro' on all the block
devices, SSD and HDD, to prevent any changes. You might get some
complaints from bcache if it can't write to HDDs or even to the SSDs,
so that might look like you've picked the bad SSD. But the real test
is if you can mount the btrfs. Try that with 'mount -o
ro,nologreplay,usebackuproot' and if you can at least get that far and
do some basic navigation, that's probably the good SSD. If you still
get mount failure, it's probably the bad one.

If you get a successful ro mount, I'd take advantage of it and backup
anything important. Just get it out now. And then you can try it all
again with everything read write; but with the bad SSD still disabled
and md array assemble degraded with the good SSD; and see if you can
mount read-write again. You need to be read write at the block device
layer to get bcache to flush SSD state to the drives, which I think is
done by setting the mode to writethrough and then waiting until
bcache/state is clean. HDDs need to be writable but btrfs doesn't need
to be mounted for this.

The other possibility is that there some bad data on both SSDs, in
which case it fails and chances are the btrfs is toast.


-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: Recover from "couldn't read tree root"?
  2021-06-20 22:53     ` Chris Murphy
@ 2021-06-22  3:26       ` Nathan Dehnel
  0 siblings, 0 replies; 8+ messages in thread
From: Nathan Dehnel @ 2021-06-22  3:26 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Btrfs BTRFS

I couldn't figure out how to salvage the filesystem, so I wiped it. Oh well.


On Sun, Jun 20, 2021 at 5:53 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sun, Jun 20, 2021 at 3:31 PM Nathan Dehnel <ncdehnel@gmail.com> wrote:
> >
> > >Was bcache in write back or write through mode?
> > Writeback.
>
> Ok that's bad in this configuration because it means all the writes go
> to the SSD and could be there for minutes, hours, days, or longer.
> That means it's even possible the current supers are only on the SSDs,
> as well as other critical btrfs metadata.
>
> My best guess now is to assume one of the drives is bad and spewing
> garbage or zeros. And assemble the array degraded with just one SSD
> drive, and see if you can mount. If not, then it's the other SSD you
> need to assemble degraded. There's a way to set a drive manually as
> faulty so it won't assemble; I also thought of using sysfs but on my
> own system, /sys/block/nvme0n1/device/delete does not exist like it
> does for SATA SSDs.
>
> Next you have to wrestle with this dilemma. If you pick the bad SSD,
> you don't want bcache flushing anything from it to your HDDs or it'll
> just corrupt them, right? if you pick the good SSD, you actually do
> want bcache to flush it all to the drives, so they're in a good state
> and you can optionally decouple the SSD entirely so that you're left
> with just the individual drives again.
>
> I think you might want to use 'blockdev --setro' on all the block
> devices, SSD and HDD, to prevent any changes. You might get some
> complaints from bcache if it can't write to HDDs or even to the SSDs,
> so that might look like you've picked the bad SSD. But the real test
> is if you can mount the btrfs. Try that with 'mount -o
> ro,nologreplay,usebackuproot' and if you can at least get that far and
> do some basic navigation, that's probably the good SSD. If you still
> get mount failure, it's probably the bad one.
>
> If you get a successful ro mount, I'd take advantage of it and backup
> anything important. Just get it out now. And then you can try it all
> again with everything read write; but with the bad SSD still disabled
> and md array assemble degraded with the good SSD; and see if you can
> mount read-write again. You need to be read write at the block device
> layer to get bcache to flush SSD state to the drives, which I think is
> done by setting the mode to writethrough and then waiting until
> bcache/state is clean. HDDs need to be writable but btrfs doesn't need
> to be mounted for this.
>
> The other possibility is that there some bad data on both SSDs, in
> which case it fails and chances are the btrfs is toast.
>
>
> --
> Chris Murphy

^ permalink raw reply	[flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-06-22  3:26 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-20 20:37 Recover from "couldn't read tree root"? Nathan Dehnel
2021-06-20 21:09 ` Chris Murphy
2021-06-20 21:31   ` Nathan Dehnel
2021-06-20 22:19     ` Chris Murphy
2021-06-20 22:53     ` Chris Murphy
2021-06-22  3:26       ` Nathan Dehnel
2021-06-20 21:19 ` Chris Murphy
2021-06-20 21:48   ` Nathan Dehnel

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.