* parent transid verify failed / ERROR: could not setup extent tree
@ 2021-03-20 6:33 Dave T
2021-03-20 6:51 ` Dave T
2021-03-21 2:03 ` Chris Murphy
0 siblings, 2 replies; 10+ messages in thread
From: Dave T @ 2021-03-20 6:33 UTC (permalink / raw)
To: Btrfs BTRFS
I hope to get some expert advice before I proceed. I don't want to
make things worse. Here's my situation now:
This problem is with an external USB drive and it is encrypted.
cryptsetup open succeeds. But mount fails.k
mount /backup
mount: /backup: wrong fs type, bad option, bad superblock on
/dev/mapper/xusbluks, missing codepage or helper program, or other
error.
Next the following command succeeds:
mount -o ro,recovery /dev/mapper/xusbluks /backup
This is my backup disk (5TB), and I don't have another 5TB disk to
copy all the data to. I hope I can fix the issue without losing my
backups.
Next step I did:
# btrfs check /dev/mapper/xyz
Opening filesystem to check...
parent transid verify failed on 2853827608576 wanted 29436 found 29433
parent transid verify failed on 2853827608576 wanted 29436 found 29433
parent transid verify failed on 2853827608576 wanted 29436 found 29433
Ignoring transid failure
leaf parent key incorrect 2853827608576
ERROR: could not setup extent tree
ERROR: cannot open file system
BTW, this command returns no result:
which btrfs-zero-log
I don't have that script/application installed. I'm running Arch Linux. I have
core/btrfs-progs 5.11-1
llinux 5.11.7-arch1-1
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: parent transid verify failed / ERROR: could not setup extent tree
2021-03-20 6:33 parent transid verify failed / ERROR: could not setup extent tree Dave T
@ 2021-03-20 6:51 ` Dave T
2021-03-21 2:03 ` Chris Murphy
1 sibling, 0 replies; 10+ messages in thread
From: Dave T @ 2021-03-20 6:51 UTC (permalink / raw)
To: Btrfs BTRFS
On Sat, Mar 20, 2021 at 2:33 AM Dave T <davestechshop@gmail.com> wrote:
>
> I hope to get some expert advice before I proceed. I don't want to
> make things worse. Here's my situation now:
>
> This problem is with an external USB drive and it is encrypted.
> cryptsetup open succeeds. But mount fails.k
>
> mount /backup
> mount: /backup: wrong fs type, bad option, bad superblock on
> /dev/mapper/xusbluks, missing codepage or helper program, or other
> error.
>
> Next the following command succeeds:
>
> mount -o ro,recovery /dev/mapper/xusbluks /backup
>
> This is my backup disk (5TB), and I don't have another 5TB disk to
> copy all the data to. I hope I can fix the issue without losing my
> backups.
>
> Next step I did:
>
> # btrfs check /dev/mapper/xyz
> Opening filesystem to check...
> parent transid verify failed on 2853827608576 wanted 29436 found 29433
> parent transid verify failed on 2853827608576 wanted 29436 found 29433
> parent transid verify failed on 2853827608576 wanted 29436 found 29433
> Ignoring transid failure
> leaf parent key incorrect 2853827608576
> ERROR: could not setup extent tree
> ERROR: cannot open file system
>
> BTW, this command returns no result:
>
> which btrfs-zero-log
>
> I don't have that script/application installed. I'm running Arch Linux. I have
>
> core/btrfs-progs 5.11-1
> llinux 5.11.7-arch1-1
I'm following up my own post with additional data. I read some other
threads that requested this, so I thought I could proactively include
it.
# btrfs insp dump-s /dev/mapper/xzy
superblock: bytenr=65536, device=/dev/mapper/xzy
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x44b6abfd [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid e23b0b43-0c36-4c7d-b9e1-e821acb259be
metadata_uuid e23b0b43-0c36-4c7d-b9e1-e821acb259be
label usb_backup
generation 29436
root 2853792677888
sys_array_size 129
chunk_root_generation 29396
root_level 1
chunk_root 22052864
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 5000928428032
bytes_used 2822988496896
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x169
( MIXED_BACKREF |
COMPRESS_LZO |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )
cache_generation 29436
uuid_tree_generation 29436
dev_item.uuid b98c68ff-2e12-41c4-97d3-81d488175dcd
dev_item.fsid e23b0b43-0c36-4c7d-b9e1-e821acb259be [match]
dev_item.type 0
dev_item.total_bytes 5000928428032
dev_item.bytes_used 2848662224896
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
# btrfs inspect-internal dump-super --full /dev/mapper/xzy
superblock: bytenr=65536, device=/dev/mapper/xzy
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x44b6abfd [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid e23b0b43-0c36-4c7d-b9e1-e821acb259be
metadata_uuid e23b0b43-0c36-4c7d-b9e1-e821acb259be
label usb_backup
generation 29436
root 2853792677888
sys_array_size 129
chunk_root_generation 29396
root_level 1
chunk_root 22052864
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 5000928428032
bytes_used 2822988496896
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x169
( MIXED_BACKREF |
COMPRESS_LZO |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )
cache_generation 29436
uuid_tree_generation 29436
dev_item.uuid b98c68ff-2e12-41c4-97d3-81d488175dcd
dev_item.fsid e23b0b43-0c36-4c7d-b9e1-e821acb259be [match]
dev_item.type 0
dev_item.total_bytes 5000928428032
dev_item.bytes_used 2848662224896
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:
item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096)
length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 1
stripe 0 devid 1 offset 22020096
dev_uuid b98c68ff-2e12-41c4-97d3-81d488175dcd
stripe 1 devid 1 offset 30408704
dev_uuid b98c68ff-2e12-41c4-97d3-81d488175dcd
backup_roots[4]:
backup 0:
backup_tree_root: 2853865455616 gen: 29435 level: 1
backup_chunk_root: 22052864 gen: 29396 level: 1
backup_extent_root: 2853827903488 gen: 29435 level: 2
backup_fs_root: 31408128 gen: 24 level: 0
backup_dev_root: 2593527267328 gen: 29423 level: 1
backup_csum_root: 2853819629568 gen: 29435 level: 3
backup_total_bytes: 5000928428032
backup_bytes_used: 2822988496896
backup_num_devices: 1
backup 1:
backup_tree_root: 2853792677888 gen: 29436 level: 1
backup_chunk_root: 22052864 gen: 29396 level: 1
backup_extent_root: 2853792727040 gen: 29436 level: 2
backup_fs_root: 31408128 gen: 24 level: 0
backup_dev_root: 2593527267328 gen: 29423 level: 1
backup_csum_root: 2853816270848 gen: 29436 level: 3
backup_total_bytes: 5000928428032
backup_bytes_used: 2822988496896
backup_num_devices: 1
backup 2:
backup_tree_root: 2853787942912 gen: 29433 level: 1
backup_chunk_root: 22052864 gen: 29396 level: 1
backup_extent_root: 2853788942336 gen: 29433 level: 2
backup_fs_root: 31408128 gen: 24 level: 0
backup_dev_root: 2593527267328 gen: 29423 level: 1
backup_csum_root: 2853806047232 gen: 29433 level: 3
backup_total_bytes: 5000928428032
backup_bytes_used: 2822957436928
backup_num_devices: 1
backup 3:
backup_tree_root: 2853792727040 gen: 29434 level: 1
backup_chunk_root: 22052864 gen: 29396 level: 1
backup_extent_root: 2853796118528 gen: 29434 level: 2
backup_fs_root: 31408128 gen: 24 level: 0
backup_dev_root: 2593527267328 gen: 29423 level: 1
backup_csum_root: 2853832540160 gen: 29434 level: 3
backup_total_bytes: 5000928428032
backup_bytes_used: 2822957436928
backup_num_devices: 1
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: parent transid verify failed / ERROR: could not setup extent tree
2021-03-20 6:33 parent transid verify failed / ERROR: could not setup extent tree Dave T
2021-03-20 6:51 ` Dave T
@ 2021-03-21 2:03 ` Chris Murphy
[not found] ` <CAGdWbB4dN45=4T_WbF7tXmm2UsmF0bh=Lb_z-H=QVQWaW=C=Bw@mail.gmail.com>
1 sibling, 1 reply; 10+ messages in thread
From: Chris Murphy @ 2021-03-21 2:03 UTC (permalink / raw)
To: Dave T; +Cc: Btrfs BTRFS
On Sat, Mar 20, 2021 at 5:15 AM Dave T <davestechshop@gmail.com> wrote:
>
> I hope to get some expert advice before I proceed. I don't want to
> make things worse. Here's my situation now:
>
> This problem is with an external USB drive and it is encrypted.
> cryptsetup open succeeds. But mount fails.k
>
> mount /backup
> mount: /backup: wrong fs type, bad option, bad superblock on
> /dev/mapper/xusbluks, missing codepage or helper program, or other
> error.
>
> Next the following command succeeds:
>
> mount -o ro,recovery /dev/mapper/xusbluks /backup
>
> This is my backup disk (5TB), and I don't have another 5TB disk to
> copy all the data to. I hope I can fix the issue without losing my
> backups.
>
> Next step I did:
>
> # btrfs check /dev/mapper/xyz
> Opening filesystem to check...
> parent transid verify failed on 2853827608576 wanted 29436 found 29433
> parent transid verify failed on 2853827608576 wanted 29436 found 29433
> parent transid verify failed on 2853827608576 wanted 29436 found 29433
> Ignoring transid failure
> leaf parent key incorrect 2853827608576
> ERROR: could not setup extent tree
> ERROR: cannot open file system
From your superblock:
backup 2:
backup_tree_root: 2853787942912 gen: 29433 level: 1
Do this:
btrfs check -r 2853787942912 /dev/xyz
If it comes up clean it's safe to do: mount -o usebackuproot, without
needing to use ro. And in that case it'll self recover. You will lose
some data, between the commits. It is possible there's partial loss,
so it's not enough to just do a scrub, you'll want to freshen the
backups as well - if that's what was happening at the time that the
trouble happened (the trouble causing the subsequent transid
failures).
Sometimes backup roots are already stale and inconsistent due to
overwrites, so the btrfs check might find problems with that older
root.
What you eventually need to look at is what precipitated the transid
failures, and avoid it. Typical is a drive firmware bug where it gets
write ordering wrong and then there's a crash or power fail. Possibly
one way to work around the bug is disabling the drive's write cache
(use a udev rule to make sure it's always applied). Another way is add
a different make/model drive to it, and convert to raid1 profile. And
hopefully they won't have overlapping firmware bugs.
--
Chris Murphy
^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: parent transid verify failed / ERROR: could not setup extent tree
[not found] ` <CAGdWbB4dN45=4T_WbF7tXmm2UsmF0bh=Lb_z-H=QVQWaW=C=Bw@mail.gmail.com>
@ 2021-03-21 5:53 ` Dave T
2021-03-21 18:03 ` Chris Murphy
0 siblings, 1 reply; 10+ messages in thread
From: Dave T @ 2021-03-21 5:53 UTC (permalink / raw)
To: Btrfs BTRFS
On Sat, Mar 20, 2021 at 10:04 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sat, Mar 20, 2021 at 5:15 AM Dave T <davestechshop@gmail.com> wrote:
> >
> > I hope to get some expert advice before I proceed. I don't want to
> > make things worse. Here's my situation now:
> >
> > This problem is with an external USB drive and it is encrypted.
> > cryptsetup open succeeds. But mount fails.k
> >
> > mount /backup
> > mount: /backup: wrong fs type, bad option, bad superblock on
> > /dev/mapper/xusbluks, missing codepage or helper program, or other
> > error.
> >
> > Next the following command succeeds:
> >
> > mount -o ro,recovery /dev/mapper/xusbluks /backup
> >
> > This is my backup disk (5TB), and I don't have another 5TB disk to
> > copy all the data to. I hope I can fix the issue without losing my
> > backups.
> >
> > Next step I did:
> >
> > # btrfs check /dev/mapper/xyz
> > Opening filesystem to check...
> > parent transid verify failed on 2853827608576 wanted 29436 found 29433
> > parent transid verify failed on 2853827608576 wanted 29436 found 29433
> > parent transid verify failed on 2853827608576 wanted 29436 found 29433
> > Ignoring transid failure
> > leaf parent key incorrect 2853827608576
> > ERROR: could not setup extent tree
> > ERROR: cannot open file system
>
>
> From your superblock:
>
> backup 2:
> backup_tree_root: 2853787942912 gen: 29433 level: 1
>
> Do this:
>
> btrfs check -r 2853787942912 /dev/xyz
>
> If it comes up clean it's safe to do: mount -o usebackuproot, without
> needing to use ro. And in that case it'll self recover. You will lose
> some data, between the commits. It is possible there's partial loss,
> so it's not enough to just do a scrub, you'll want to freshen the
> backups as well - if that's what was happening at the time that the
> trouble happened (the trouble causing the subsequent transid
> failures).
>
> Sometimes backup roots are already stale and inconsistent due to
> overwrites, so the btrfs check might find problems with that older
> root.
# btrfs check -r 2853787942912 /dev/mapper/xyz
Opening filesystem to check...
parent transid verify failed on 2853787942912 wanted 29436 found 29433
parent transid verify failed on 2853787942912 wanted 29436 found 29433
parent transid verify failed on 2853787942912 wanted 29436 found 29433
Ignoring transid failure
parent transid verify failed on 2853827723264 wanted 29433 found 29435
parent transid verify failed on 2853827723264 wanted 29433 found 29435
parent transid verify failed on 2853827723264 wanted 29433 found 29435
Ignoring transid failure
leaf parent key incorrect 2853827723264
ERROR: could not setup extent tree
ERROR: cannot open file system
It appears the backup root is already stale.
>
> What you eventually need to look at is what precipitated the transid
> failures, and avoid it.
The USB drive was disconnected by the user (an accident). I have other
devices with the same hardware that have never experienced this issue.
Do you have further ideas or suggestions I can try? Thank you for your
time and for sharing your expertise.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: parent transid verify failed / ERROR: could not setup extent tree
2021-03-21 5:53 ` Dave T
@ 2021-03-21 18:03 ` Chris Murphy
2021-03-22 6:32 ` Dave T
0 siblings, 1 reply; 10+ messages in thread
From: Chris Murphy @ 2021-03-21 18:03 UTC (permalink / raw)
To: Dave T; +Cc: Btrfs BTRFS, Qu Wenruo
On Sat, Mar 20, 2021 at 11:54 PM Dave T <davestechshop@gmail.com> wrote:
>
> # btrfs check -r 2853787942912 /dev/mapper/xyz
> Opening filesystem to check...
> parent transid verify failed on 2853787942912 wanted 29436 found 29433
> parent transid verify failed on 2853787942912 wanted 29436 found 29433
> parent transid verify failed on 2853787942912 wanted 29436 found 29433
> Ignoring transid failure
> parent transid verify failed on 2853827723264 wanted 29433 found 29435
> parent transid verify failed on 2853827723264 wanted 29433 found 29435
> parent transid verify failed on 2853827723264 wanted 29433 found 29435
> Ignoring transid failure
> leaf parent key incorrect 2853827723264
> ERROR: could not setup extent tree
> ERROR: cannot open file system
btrfs insp dump-t -t 2853827723264 /dev/
> It appears the backup root is already stale.
I'm not sure. If you can post the contents of that leaf (I don't think
it will contain filenames but double check) Qu might have an idea if
it's safe to try a read-write mount with -o usebackuproot without
causing problems later.
> > What you eventually need to look at is what precipitated the transid
> > failures, and avoid it.
>
> The USB drive was disconnected by the user (an accident). I have other
> devices with the same hardware that have never experienced this issue.
>
> Do you have further ideas or suggestions I can try? Thank you for your
> time and for sharing your expertise.
The drive could be getting write ordering wrong all the time, and it
only turns into a problem with a crash, power fail, or accidental
disconnect. More common is the write ordering is only sometimes
wrong, and a crash or powerfail is usually survivable, but leads to a
false sense of security about the drive.
The simple theory of write order is data->metadata->sync->super->sync.
It shouldn't ever be the case that a newer superblock generation is on
stable media before the metadata it points to.
--
Chris Murphy
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: parent transid verify failed / ERROR: could not setup extent tree
2021-03-21 18:03 ` Chris Murphy
@ 2021-03-22 6:32 ` Dave T
2021-03-22 19:48 ` Chris Murphy
0 siblings, 1 reply; 10+ messages in thread
From: Dave T @ 2021-03-22 6:32 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS, Qu Wenruo
On Sun, Mar 21, 2021 at 2:03 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sat, Mar 20, 2021 at 11:54 PM Dave T <davestechshop@gmail.com> wrote:
> >
> > # btrfs check -r 2853787942912 /dev/mapper/xyz
> > Opening filesystem to check...
> > parent transid verify failed on 2853787942912 wanted 29436 found 29433
> > parent transid verify failed on 2853787942912 wanted 29436 found 29433
> > parent transid verify failed on 2853787942912 wanted 29436 found 29433
> > Ignoring transid failure
> > parent transid verify failed on 2853827723264 wanted 29433 found 29435
> > parent transid verify failed on 2853827723264 wanted 29433 found 29435
> > parent transid verify failed on 2853827723264 wanted 29433 found 29435
> > Ignoring transid failure
> > leaf parent key incorrect 2853827723264
> > ERROR: could not setup extent tree
> > ERROR: cannot open file system
>
> btrfs insp dump-t -t 2853827723264 /dev/
# btrfs insp dump-t -t 2853827723264 /dev/mapper/xzy
btrfs-progs v5.11
parent transid verify failed on 2853827608576 wanted 29436 found 29433
parent transid verify failed on 2853827608576 wanted 29436 found 29433
parent transid verify failed on 2853827608576 wanted 29436 found 29433
Ignoring transid failure
leaf parent key incorrect 2853827608576
WARNING: could not setup extent tree, skipping it
parent transid verify failed on 2853827608576 wanted 29436 found 29433
Ignoring transid failure
leaf parent key incorrect 2853827608576
Couldn't setup device tree
ERROR: unable to open /dev/mapper/xzy
# btrfs insp dump-t -t 2853787942912 /dev/mapper/xzy
btrfs-progs v5.11
parent transid verify failed on 2853827608576 wanted 29436 found 29433
parent transid verify failed on 2853827608576 wanted 29436 found 29433
parent transid verify failed on 2853827608576 wanted 29436 found 29433
Ignoring transid failure
leaf parent key incorrect 2853827608576
WARNING: could not setup extent tree, skipping it
parent transid verify failed on 2853827608576 wanted 29436 found 29433
Ignoring transid failure
leaf parent key incorrect 2853827608576
Couldn't setup device tree
ERROR: unable to open /dev/mapper/xzy
# btrfs insp dump-t -t 2853827608576 /dev/mapper/xzy
btrfs-progs v5.11
parent transid verify failed on 2853827608576 wanted 29436 found 29433
parent transid verify failed on 2853827608576 wanted 29436 found 29433
parent transid verify failed on 2853827608576 wanted 29436 found 29433
Ignoring transid failure
leaf parent key incorrect 2853827608576
WARNING: could not setup extent tree, skipping it
parent transid verify failed on 2853827608576 wanted 29436 found 29433
Ignoring transid failure
leaf parent key incorrect 2853827608576
Couldn't setup device tree
ERROR: unable to open /dev/mapper/xzy
>
> > It appears the backup root is already stale.
>
> I'm not sure. If you can post the contents of that leaf (I don't think
> it will contain filenames but double check) Qu might have an idea if
> it's safe to try a read-write mount with -o usebackuproot without
> causing problems later.
>
> > > What you eventually need to look at is what precipitated the transid
> > > failures, and avoid it.
> >
> > The USB drive was disconnected by the user (an accident). I have other
> > devices with the same hardware that have never experienced this issue.
> >
> > Do you have further ideas or suggestions I can try? Thank you for your
> > time and for sharing your expertise.
>
> The drive could be getting write ordering wrong all the time, and it
> only turns into a problem with a crash, power fail, or accidental
> disconnect. More common is the write ordering is only sometimes
> wrong, and a crash or powerfail is usually survivable, but leads to a
> false sense of security about the drive.
>
> The simple theory of write order is data->metadata->sync->super->sync.
> It shouldn't ever be the case that a newer superblock generation is on
> stable media before the metadata it points to.
>
The drive is a Western Digital Elements 5TB. I searched a bit on write
order under Linux, but I did not find any helpful articles regarding
any configuration changes I could make. Is this purely a hardware
issue, or are there steps I can take to ensure the correct write
ordering?
Thank you!
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: parent transid verify failed / ERROR: could not setup extent tree
2021-03-22 6:32 ` Dave T
@ 2021-03-22 19:48 ` Chris Murphy
2021-03-23 6:50 ` Dave T
0 siblings, 1 reply; 10+ messages in thread
From: Chris Murphy @ 2021-03-22 19:48 UTC (permalink / raw)
To: Dave T; +Cc: Chris Murphy, Btrfs BTRFS, Qu Wenruo
On Mon, Mar 22, 2021 at 12:32 AM Dave T <davestechshop@gmail.com> wrote:
>
> On Sun, Mar 21, 2021 at 2:03 PM Chris Murphy <lists@colorremedies.com> wrote:
> >
> > On Sat, Mar 20, 2021 at 11:54 PM Dave T <davestechshop@gmail.com> wrote:
> > >
> > > # btrfs check -r 2853787942912 /dev/mapper/xyz
> > > Opening filesystem to check...
> > > parent transid verify failed on 2853787942912 wanted 29436 found 29433
> > > parent transid verify failed on 2853787942912 wanted 29436 found 29433
> > > parent transid verify failed on 2853787942912 wanted 29436 found 29433
> > > Ignoring transid failure
> > > parent transid verify failed on 2853827723264 wanted 29433 found 29435
> > > parent transid verify failed on 2853827723264 wanted 29433 found 29435
> > > parent transid verify failed on 2853827723264 wanted 29433 found 29435
> > > Ignoring transid failure
> > > leaf parent key incorrect 2853827723264
> > > ERROR: could not setup extent tree
> > > ERROR: cannot open file system
> >
> > btrfs insp dump-t -t 2853827723264 /dev/
>
> # btrfs insp dump-t -t 2853827723264 /dev/mapper/xzy
> btrfs-progs v5.11
> parent transid verify failed on 2853827608576 wanted 29436 found 29433
> parent transid verify failed on 2853827608576 wanted 29436 found 29433
> parent transid verify failed on 2853827608576 wanted 29436 found 29433
> Ignoring transid failure
> leaf parent key incorrect 2853827608576
> WARNING: could not setup extent tree, skipping it
> parent transid verify failed on 2853827608576 wanted 29436 found 29433
> Ignoring transid failure
> leaf parent key incorrect 2853827608576
> Couldn't setup device tree
> ERROR: unable to open /dev/mapper/xzy
>
> # btrfs insp dump-t -t 2853787942912 /dev/mapper/xzy
> btrfs-progs v5.11
> parent transid verify failed on 2853827608576 wanted 29436 found 29433
> parent transid verify failed on 2853827608576 wanted 29436 found 29433
> parent transid verify failed on 2853827608576 wanted 29436 found 29433
> Ignoring transid failure
> leaf parent key incorrect 2853827608576
> WARNING: could not setup extent tree, skipping it
> parent transid verify failed on 2853827608576 wanted 29436 found 29433
> Ignoring transid failure
> leaf parent key incorrect 2853827608576
> Couldn't setup device tree
> ERROR: unable to open /dev/mapper/xzy
>
> # btrfs insp dump-t -t 2853827608576 /dev/mapper/xzy
> btrfs-progs v5.11
> parent transid verify failed on 2853827608576 wanted 29436 found 29433
> parent transid verify failed on 2853827608576 wanted 29436 found 29433
> parent transid verify failed on 2853827608576 wanted 29436 found 29433
> Ignoring transid failure
> leaf parent key incorrect 2853827608576
> WARNING: could not setup extent tree, skipping it
> parent transid verify failed on 2853827608576 wanted 29436 found 29433
> Ignoring transid failure
> leaf parent key incorrect 2853827608576
> Couldn't setup device tree
> ERROR: unable to open /dev/mapper/xzy
That does not look promising. I don't know whether a read-write mount
with usebackuproot will recover, or end up with problems.
Options:
a. btrfs check --repair
This probably fails on the same problem, it can't setup the extent tree.
b. btrfs check --init-extent-tree
This is a heavy hammer, it might succeed, but takes a long time. On 5T
it might take double digit hours or even single digit days. It's
generally faster to just wipe the drive and restore from backups than
use init-extent-tree (I understand this *is* your backup).
c. Setup an overlay file on device mapper, to redirect the writes from
a read-write mount with usebackup root. I think it's sufficient to
just mount, optionally write some files (empty or not), and umount.
Then do a btrfs check to see if the current tree is healthy.
https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
That guide is a bit complex to deal with many drives with mdadm raid,
so you can simplify it for just one drive. The gist is no writes go to
the drive itself, it's treated as read-only by device-mapper (in fact
you can optionally add a pre-step with the blockdev command and
--setro to make sure the entire drive is read-only; just make sure to
make it rw once you're done testing). All the writes with this overlay
go into a loop mounted file which you intentionally just throw away
after testing.
d. Just skip the testing and try usebackuproot with a read-write
mount. It might make things worse, but at least it's fast to test. If
it messes things up, you'll have to recreate this backup from scratch.
As for how to prevent this? I'm not sure. About the best we can do is
disable the drive write cache with a udev rule, and/or raid1 with
another make/model drive, and let Btrfs detect occasional corruption
and self heal from the good copy. Another obvious way to avoid the
problem is, stop having power failures, crashes, and accidental USB
cable disconnections :)
It's not any one thing that's the problem. It's a sequence of problems
happening in just the right (or wrong) order that causes the problem.
Bugs + mistake + bad luck = problem.
--
Chris Murphy
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: parent transid verify failed / ERROR: could not setup extent tree
2021-03-22 19:48 ` Chris Murphy
@ 2021-03-23 6:50 ` Dave T
2021-03-23 22:39 ` Chris Murphy
0 siblings, 1 reply; 10+ messages in thread
From: Dave T @ 2021-03-23 6:50 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS, Qu Wenruo
On Mon, Mar 22, 2021 at 3:49 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Mon, Mar 22, 2021 at 12:32 AM Dave T <davestechshop@gmail.com> wrote:
> >
> > On Sun, Mar 21, 2021 at 2:03 PM Chris Murphy <lists@colorremedies.com> wrote:
> > >
> > > On Sat, Mar 20, 2021 at 11:54 PM Dave T <davestechshop@gmail.com> wrote:
> > > >
> > > > # btrfs check -r 2853787942912 /dev/mapper/xyz
> > > > Opening filesystem to check...
> > > > parent transid verify failed on 2853787942912 wanted 29436 found 29433
> > > > parent transid verify failed on 2853787942912 wanted 29436 found 29433
> > > > parent transid verify failed on 2853787942912 wanted 29436 found 29433
> > > > Ignoring transid failure
> > > > parent transid verify failed on 2853827723264 wanted 29433 found 29435
> > > > parent transid verify failed on 2853827723264 wanted 29433 found 29435
> > > > parent transid verify failed on 2853827723264 wanted 29433 found 29435
> > > > Ignoring transid failure
> > > > leaf parent key incorrect 2853827723264
> > > > ERROR: could not setup extent tree
> > > > ERROR: cannot open file system
> > >
> > > btrfs insp dump-t -t 2853827723264 /dev/
> >
> > # btrfs insp dump-t -t 2853827723264 /dev/mapper/xzy
> > btrfs-progs v5.11
> > parent transid verify failed on 2853827608576 wanted 29436 found 29433
> > parent transid verify failed on 2853827608576 wanted 29436 found 29433
> > parent transid verify failed on 2853827608576 wanted 29436 found 29433
> > Ignoring transid failure
> > leaf parent key incorrect 2853827608576
> > WARNING: could not setup extent tree, skipping it
> > parent transid verify failed on 2853827608576 wanted 29436 found 29433
> > Ignoring transid failure
> > leaf parent key incorrect 2853827608576
> > Couldn't setup device tree
> > ERROR: unable to open /dev/mapper/xzy
> >
> > # btrfs insp dump-t -t 2853787942912 /dev/mapper/xzy
> > btrfs-progs v5.11
> > parent transid verify failed on 2853827608576 wanted 29436 found 29433
> > parent transid verify failed on 2853827608576 wanted 29436 found 29433
> > parent transid verify failed on 2853827608576 wanted 29436 found 29433
> > Ignoring transid failure
> > leaf parent key incorrect 2853827608576
> > WARNING: could not setup extent tree, skipping it
> > parent transid verify failed on 2853827608576 wanted 29436 found 29433
> > Ignoring transid failure
> > leaf parent key incorrect 2853827608576
> > Couldn't setup device tree
> > ERROR: unable to open /dev/mapper/xzy
> >
> > # btrfs insp dump-t -t 2853827608576 /dev/mapper/xzy
> > btrfs-progs v5.11
> > parent transid verify failed on 2853827608576 wanted 29436 found 29433
> > parent transid verify failed on 2853827608576 wanted 29436 found 29433
> > parent transid verify failed on 2853827608576 wanted 29436 found 29433
> > Ignoring transid failure
> > leaf parent key incorrect 2853827608576
> > WARNING: could not setup extent tree, skipping it
> > parent transid verify failed on 2853827608576 wanted 29436 found 29433
> > Ignoring transid failure
> > leaf parent key incorrect 2853827608576
> > Couldn't setup device tree
> > ERROR: unable to open /dev/mapper/xzy
>
> That does not look promising. I don't know whether a read-write mount
> with usebackuproot will recover, or end up with problems.
>
> Options:
>
> a. btrfs check --repair
> This probably fails on the same problem, it can't setup the extent tree.
>
> b. btrfs check --init-extent-tree
> This is a heavy hammer, it might succeed, but takes a long time. On 5T
> it might take double digit hours or even single digit days. It's
> generally faster to just wipe the drive and restore from backups than
> use init-extent-tree (I understand this *is* your backup).
>
> c. Setup an overlay file on device mapper, to redirect the writes from
> a read-write mount with usebackup root. I think it's sufficient to
> just mount, optionally write some files (empty or not), and umount.
> Then do a btrfs check to see if the current tree is healthy.
> https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
>
> That guide is a bit complex to deal with many drives with mdadm raid,
> so you can simplify it for just one drive. The gist is no writes go to
> the drive itself, it's treated as read-only by device-mapper (in fact
> you can optionally add a pre-step with the blockdev command and
> --setro to make sure the entire drive is read-only; just make sure to
> make it rw once you're done testing). All the writes with this overlay
> go into a loop mounted file which you intentionally just throw away
> after testing.
>
> d. Just skip the testing and try usebackuproot with a read-write
> mount. It might make things worse, but at least it's fast to test. If
> it messes things up, you'll have to recreate this backup from scratch.
I took this approach. My command was simply:
mount -o usebackuproot /dev/mapper/xzy /backup
It appears to have succeeded because it mounted without errors. I
completed a new incremental backup (with btrbk) and it finished
without errors.
I'll be pleased if my backup history is preserved, as appears to be the case.
I will run some checks on those backup subvolumes tomorrow. Are there
specific checks you would recommend?
>
> As for how to prevent this? I'm not sure. About the best we can do is
> disable the drive write cache with a udev rule,
That sounds like a suitable solution for me.
Thank you for this information. BTW, I have been using BTRFS for many
years. This is the first serious issue I have had, and as you said
there is a large element of user error and bad luck involved in this
case.
> and/or raid1 with
> another make/model drive, and let Btrfs detect occasional corruption
> and self heal from the good copy. Another obvious way to avoid the
> problem is, stop having power failures, crashes, and accidental USB
> cable disconnections :)
>
> It's not any one thing that's the problem. It's a sequence of problems
> happening in just the right (or wrong) order that causes the problem.
> Bugs + mistake + bad luck = problem.
>
> --
> Chris Murphy
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: parent transid verify failed / ERROR: could not setup extent tree
2021-03-23 6:50 ` Dave T
@ 2021-03-23 22:39 ` Chris Murphy
0 siblings, 0 replies; 10+ messages in thread
From: Chris Murphy @ 2021-03-23 22:39 UTC (permalink / raw)
To: Dave T; +Cc: Chris Murphy, Btrfs BTRFS, Qu Wenruo
On Tue, Mar 23, 2021 at 12:50 AM Dave T <davestechshop@gmail.com> wrote:
>
> > d. Just skip the testing and try usebackuproot with a read-write
> > mount. It might make things worse, but at least it's fast to test. If
> > it messes things up, you'll have to recreate this backup from scratch.
>
> I took this approach. My command was simply:
>
> mount -o usebackuproot /dev/mapper/xzy /backup
>
> It appears to have succeeded because it mounted without errors. I
> completed a new incremental backup (with btrbk) and it finished
> without errors.
> I'll be pleased if my backup history is preserved, as appears to be the case.
>
> I will run some checks on those backup subvolumes tomorrow. Are there
> specific checks you would recommend?
It will have replaced all the root nodes and super blocks within a
minute, or immediately upon umount. So you can just do a 'btrfs check'
and see if that comes up clean now. It's basically a kind of rollback
and if it worked, there will be no inconsistencies found by btrfs
check.
--
Chris Murphy
^ permalink raw reply [flat|nested] 10+ messages in thread
* parent transid verify failed / ERROR: could not setup extent tree
@ 2023-05-06 10:05 Marcin Wesołowski
0 siblings, 0 replies; 10+ messages in thread
From: Marcin Wesołowski @ 2023-05-06 10:05 UTC (permalink / raw)
To: linux-btrfs
Hi everyone,
I faced a power outage while copying large files between my
LUKS-encrypted btrfs partitions on a USB WD Elements drive (with an
external PSU). Later, it failed to mount or do basic btrfs check. I've
found a quite similar (I think) problem discussed here
https://www.spinics.net/lists/linux-btrfs/msg111522.html, looking less
fatal though.
This is a backup drive and I think I should have all its content
spread across other drives or PCs, but I was just about to finish a
long manual clean-up process, so if it was possible to recover I'd
spare quite a bit of time. So I just have two questions:
- is there anything else I could try to do to recover most of the
files? I made a mirror with dd, so it's safe to go wild with
experiments. I'm also not afraid of letting it run for a few days
(it's attached to a UPS now ;)
- if it doesn't work or is not feasible, is it possible to somehow
recover just the file and folder names? I'd then know what to look for
on other backups.
I'm running Ubuntu 20.04.4 LTS, btrfs-progs v5.4.1.
Thanks in advance!
Here's what I've tried so far (inspired by the thread pasted above).
# btrfs check /dev/mapper/luks-3870aa81-a158-47d0-86f1-530de7284d1a
Opening filesystem to check...
parent transid verify failed on 5090687057920 wanted 4073 found 4075
parent transid verify failed on 5090687057920 wanted 4073 found 4075
parent transid verify failed on 5090687057920 wanted 4073 found 4075
Ignoring transid failure
ERROR: could not setup extent tree
ERROR: cannot open file system
root@dell:/home/wesol# mount -o usebackuproot
/dev/mapper/luks-3870aa81-a158-47d0-86f1-530de7284d1a
/media/experiments
mount: /media/experiments: wrong fs type, bad option, bad superblock
on /dev/mapper/luks-3870aa81-a158-47d0-86f1-530de7284d1a, missing
codepage or helper program, or other error.
root@dell:/home/wesol# mount -o ro,recovery
/dev/mapper/luks-3870aa81-a158-47d0-86f1-530de7284d1a
/media/experiments
mount: /media/experiments: wrong fs type, bad option, bad superblock
on /dev/mapper/luks-3870aa81-a158-47d0-86f1-530de7284d1a, missing
codepage or helper program, or other error.
root@dell:/home/wesol# btrfs insp dump-s
/dev/mapper/luks-3870aa81-a158-47d0-86f1-530de7284d1a
superblock: bytenr=65536,
device=/dev/mapper/luks-3870aa81-a158-47d0-86f1-530de7284d1a
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x64c7c8ad [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid b8ddc54a-94eb-46e8-a65d-6fc02eb04c72
metadata_uuid b8ddc54a-94eb-46e8-a65d-6fc02eb04c72
label smb00
generation 4073
root 5090687057920
sys_array_size 129
chunk_root_generation 4073
root_level 1
chunk_root 22052864
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 9654574776320
bytes_used 5761471766528
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x141
( MIXED_BACKREF |
EXTENDED_IREF |
SKINNY_METADATA )
cache_generation 4073
uuid_tree_generation 4073
dev_item.uuid 3607cd4c-fb8a-4f57-bba4-85352db5c418
dev_item.fsid b8ddc54a-94eb-46e8-a65d-6fc02eb04c72 [match]
dev_item.type 0
dev_item.total_bytes 9654574776320
dev_item.bytes_used 5781051146240
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
root@dell:/home/wesol# btrfs inspect-internal dump-super --full
/dev/mapper/luks-3870aa81-a158-47d0-86f1-530de7284d1a
superblock: bytenr=65536,
device=/dev/mapper/luks-3870aa81-a158-47d0-86f1-530de7284d1a
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x64c7c8ad [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid b8ddc54a-94eb-46e8-a65d-6fc02eb04c72
metadata_uuid b8ddc54a-94eb-46e8-a65d-6fc02eb04c72
label smb00
generation 4073
root 5090687057920
sys_array_size 129
chunk_root_generation 4073
root_level 1
chunk_root 22052864
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 9654574776320
bytes_used 5761471766528
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x141
( MIXED_BACKREF |
EXTENDED_IREF |
SKINNY_METADATA )
cache_generation 4073
uuid_tree_generation 4073
dev_item.uuid 3607cd4c-fb8a-4f57-bba4-85352db5c418
dev_item.fsid b8ddc54a-94eb-46e8-a65d-6fc02eb04c72 [match]
dev_item.type 0
dev_item.total_bytes 9654574776320
dev_item.bytes_used 5781051146240
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:
item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096)
length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 1
stripe 0 devid 1 offset 22020096
dev_uuid 3607cd4c-fb8a-4f57-bba4-85352db5c418
stripe 1 devid 1 offset 30408704
dev_uuid 3607cd4c-fb8a-4f57-bba4-85352db5c418
backup_roots[4]:
backup 0:
backup_tree_root: 5090646720512 gen: 4070 level: 1
backup_chunk_root: 22511616 gen: 4070 level: 1
backup_extent_root: 5090641772544 gen: 4070 level: 2
backup_fs_root: 5090652487680 gen: 4071 level: 2
backup_dev_root: 5090642771968 gen: 4070 level: 1
backup_csum_root: 5090652536832 gen: 4071 level: 3
backup_total_bytes: 9654574776320
backup_bytes_used: 5757456617472
backup_num_devices: 1
backup 1:
backup_tree_root: 5090660990976 gen: 4071 level: 1
backup_chunk_root: 22052864 gen: 4071 level: 1
backup_extent_root: 5090653126656 gen: 4071 level: 2
backup_fs_root: 5090652487680 gen: 4071 level: 2
backup_dev_root: 5090653175808 gen: 4071 level: 1
backup_csum_root: 5090652536832 gen: 4071 level: 3
backup_total_bytes: 9654574776320
backup_bytes_used: 5758862508032
backup_num_devices: 1
backup 2:
backup_tree_root: 5090674573312 gen: 4072 level: 1
backup_chunk_root: 22511616 gen: 4072 level: 1
backup_extent_root: 5090650488832 gen: 4072 level: 2
backup_fs_root: 5090683322368 gen: 4073 level: 2
backup_dev_root: 5090651111424 gen: 4072 level: 1
backup_csum_root: 5090651209728 gen: 4072 level: 3
backup_total_bytes: 9654574776320
backup_bytes_used: 5760037130240
backup_num_devices: 1
backup 3:
backup_tree_root: 5090687057920 gen: 4073 level: 1
backup_chunk_root: 22052864 gen: 4073 level: 1
backup_extent_root: 5090661482496 gen: 4073 level: 2
backup_fs_root: 5090683322368 gen: 4073 level: 2
backup_dev_root: 5090661531648 gen: 4073 level: 1
backup_csum_root: 5090683453440 gen: 4073 level: 3
backup_total_bytes: 9654574776320
backup_bytes_used: 5761471766528
backup_num_devices: 1
root@dell:/home/wesol# btrfs check -r 5090646720512
/dev/mapper/luks-3870aa81-a158-47d0-86f1-530de7284d1a
Opening filesystem to check...
parent transid verify failed on 5090646720512 wanted 4073 found 4070
parent transid verify failed on 5090646720512 wanted 4073 found 4070
parent transid verify failed on 5090646720512 wanted 4073 found 4070
Ignoring transid failure
parent transid verify failed on 5090651602944 wanted 4070 found 4072
parent transid verify failed on 5090651602944 wanted 4070 found 4072
parent transid verify failed on 5090651602944 wanted 4070 found 4072
Ignoring transid failure
leaf parent key incorrect 5090651602944
ERROR: could not setup extent tree
ERROR: cannot open file system
root@dell:/home/wesol# btrfs check -r 5090660990976
/dev/mapper/luks-3870aa81-a158-47d0-86f1-530de7284d1a
Opening filesystem to check...
'ERROR: could not setup extent tree
ERROR: cannot open file system
root@dell:/home/wesol# btrfs check -r 5090674573312
/dev/mapper/luks-3870aa81-a158-47d0-86f1-530de7284d1a
Opening filesystem to check...
parent transid verify failed on 5090674573312 wanted 4073 found 4074
parent transid verify failed on 5090674573312 wanted 4073 found 4074
parent transid verify failed on 5090674573312 wanted 4073 found 4074
Ignoring transid failure
ERROR: could not setup extent tree
ERROR: cannot open file system
root@dell:/home/wesol# btrfs check -r 5090687057920
/dev/mapper/luks-3870aa81-a158-47d0-86f1-530de7284d1a
Opening filesystem to check...
parent transid verify failed on 5090687057920 wanted 4073 found 4075
parent transid verify failed on 5090687057920 wanted 4073 found 4075
parent transid verify failed on 5090687057920 wanted 4073 found 4075
Ignoring transid failure
ERROR: could not setup extent tree
ERROR: cannot open file system
root@dell:/home/wesol# btrfs check --init-extent-tree
/dev/mapper/luks-3870aa81-a158-47d0-86f1-530de7284d1a
WARNING:
Do not use --repair unless you are advised to do so by a developer
or an experienced user, and then only after having accepted that no
fsck can successfully repair all types of filesystem corruption. Eg.
some software or hardware bugs can fatally damage a volume.
The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 5090687057920 wanted 4073 found 4075
parent transid verify failed on 5090687057920 wanted 4073 found 4075
parent transid verify failed on 5090687057920 wanted 4073 found 4075
Ignoring transid failure
WARNING: could not setup extent tree, skipping it
Couldn't setup device tree
ERROR: cannot open file system
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2023-05-06 10:05 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-20 6:33 parent transid verify failed / ERROR: could not setup extent tree Dave T
2021-03-20 6:51 ` Dave T
2021-03-21 2:03 ` Chris Murphy
[not found] ` <CAGdWbB4dN45=4T_WbF7tXmm2UsmF0bh=Lb_z-H=QVQWaW=C=Bw@mail.gmail.com>
2021-03-21 5:53 ` Dave T
2021-03-21 18:03 ` Chris Murphy
2021-03-22 6:32 ` Dave T
2021-03-22 19:48 ` Chris Murphy
2021-03-23 6:50 ` Dave T
2021-03-23 22:39 ` Chris Murphy
2023-05-06 10:05 Marcin Wesołowski
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.