external harddisk: bogus corrupt leaf error?

external harddisk: bogus corrupt leaf error?

[Martin Steigerwald]

Hi!

I have an external 500 GB harddisk with BTRFS. On mounting it the kernel 
(5.9-rc5, vanilla, self compiled) reports:

[282409.344208] BTRFS info (device sdc1): enabling auto defrag
[282409.344222] BTRFS info (device sdc1): use zstd compression, level 3
[282409.344225] BTRFS info (device sdc1): disk space caching is enabled
[282409.465837] BTRFS critical (device sdc1): corrupt leaf: root=1 
block=906756096 slot=204, invalid root item size, have 239 expect 439

Note: It has used LZO compression before, but I switched mount option to 
zstd meanwhile.

However, btrfs-progds 5.7 gives:

% btrfs check /dev/sdc1
Opening filesystem to check...
Checking filesystem on /dev/sdc1
UUID: […]
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 249031409664 bytes used, no error found
total csum bytes: 242738928
total tree bytes: 352387072
total fs tree bytes: 67747840
total extent tree bytes: 14565376
btree space waste bytes: 37691414
file data blocks allocated: 1067158315008
 referenced 247077785600


Is this kernel message in error? Or does 'btrfs check' not check for 
this error yet?

Here some more information:

% btrfs filesystem df /mnt/[…]
Data, single: total=444.67GiB, used=231.60GiB
System, DUP: total=64.00MiB, used=80.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=3.00GiB, used=335.98MiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=271.09MiB, used=0.00B
WARNING: Multiple block group profiles detected, see 'man btrfs(5)'.
WARNING:   Metadata: single, dup
WARNING:   System: single, dup

Ah, there is something old I can clean up, it seems:


% merkaba:~> btrfs filesystem usage -T /mnt/amazon
Overall:
    Device size:                 465.76GiB
    Device allocated:            450.81GiB
    Device unallocated:           14.95GiB
    Device missing:                  0.00B
    Used:                        232.26GiB
    Free (estimated):            228.02GiB      (min: 220.55GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              271.09MiB      (used: 0.00B)
    Multiple profiles:                 yes      (metadata, system)

             Data      Metadata Metadata  System  System               
Id Path      single    single   DUP       single  DUP       Unallocated
-- --------- --------- -------- --------- ------- --------- -----------
 1 /dev/sdc1 444.67GiB  8.00MiB   6.00GiB 4.00MiB 128.00MiB    14.95GiB
-- --------- --------- -------- --------- ------- --------- -----------
   Total     444.67GiB  8.00MiB   3.00GiB 4.00MiB  64.00MiB    14.95GiB
   Used      231.60GiB    0.00B 335.98MiB   0.00B  80.00KiB  

Will run a scrub next to see whether all data can be read.

Device stats are okay:

% btrfs device stats /mnt/amazon
[/dev/sdc1].write_io_errs    0
[/dev/sdc1].read_io_errs     0
[/dev/sdc1].flush_io_errs    0
[/dev/sdc1].corruption_errs  0
[/dev/sdc1].generation_errs  0

Device reports SMART status as: passed.

There are some "Error: ICRC, ABRT 24 sectors at LBA" like errors in 
SMART error log, but they are from a very early time about 290 to 310 
hours of disk power-on lifetime. I believe this may have been me 
disconnecting the device while it was still being written to or 
something, but I will run a long SMART test as well.

Best,
-- 
Martin

Re: external harddisk: bogus corrupt leaf error?

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 4787 bytes --]



On 2020/9/21 下午5:29, Martin Steigerwald wrote:
> Hi!
> 
> I have an external 500 GB harddisk with BTRFS. On mounting it the kernel 
> (5.9-rc5, vanilla, self compiled) reports:
> 
> [282409.344208] BTRFS info (device sdc1): enabling auto defrag
> [282409.344222] BTRFS info (device sdc1): use zstd compression, level 3
> [282409.344225] BTRFS info (device sdc1): disk space caching is enabled
> [282409.465837] BTRFS critical (device sdc1): corrupt leaf: root=1 
> block=906756096 slot=204, invalid root item size, have 239 expect 439

This one can only be detected by kernel, not btrfs check yet.

Recently kernel has much more strict checks than btrfs-check, sometimes
it can be too strict, as some error is not really going to cause
problems, but just against on-disk format.

And this is the case.

In theory, you can mount the fs with older kernel, any kernel older than
commit 259ee7754b67 ("btrfs: tree-checker: Add ROOT_ITEM check") should
still be able to mount the fs.

For workaround, you can dump the tree block 906756096, locate the slot
204, see what tree root it is.
If it's a subvolume/snapshot, deleting it should solve the problem, even
for the latest kernel.


For the root cause, it should be some older kernel creating the wrong
root item size.
I can't find the commit but it should be pretty old, as after v5.4 we
have mandatory write time tree checks, which will reject such write
directly.

Thanks,
Qu
> 
> Note: It has used LZO compression before, but I switched mount option to 
> zstd meanwhile.
> 
> However, btrfs-progds 5.7 gives:
> 
> % btrfs check /dev/sdc1
> Opening filesystem to check...
> Checking filesystem on /dev/sdc1
> UUID: […]
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 249031409664 bytes used, no error found
> total csum bytes: 242738928
> total tree bytes: 352387072
> total fs tree bytes: 67747840
> total extent tree bytes: 14565376
> btree space waste bytes: 37691414
> file data blocks allocated: 1067158315008
>  referenced 247077785600
> 
> 
> Is this kernel message in error? Or does 'btrfs check' not check for 
> this error yet?
> 
> Here some more information:
> 
> % btrfs filesystem df /mnt/[…]
> Data, single: total=444.67GiB, used=231.60GiB
> System, DUP: total=64.00MiB, used=80.00KiB
> System, single: total=4.00MiB, used=0.00B
> Metadata, DUP: total=3.00GiB, used=335.98MiB
> Metadata, single: total=8.00MiB, used=0.00B
> GlobalReserve, single: total=271.09MiB, used=0.00B
> WARNING: Multiple block group profiles detected, see 'man btrfs(5)'.
> WARNING:   Metadata: single, dup
> WARNING:   System: single, dup
> 
> Ah, there is something old I can clean up, it seems:
> 
> 
> % merkaba:~> btrfs filesystem usage -T /mnt/amazon
> Overall:
>     Device size:                 465.76GiB
>     Device allocated:            450.81GiB
>     Device unallocated:           14.95GiB
>     Device missing:                  0.00B
>     Used:                        232.26GiB
>     Free (estimated):            228.02GiB      (min: 220.55GiB)
>     Data ratio:                       1.00
>     Metadata ratio:                   2.00
>     Global reserve:              271.09MiB      (used: 0.00B)
>     Multiple profiles:                 yes      (metadata, system)
> 
>              Data      Metadata Metadata  System  System               
> Id Path      single    single   DUP       single  DUP       Unallocated
> -- --------- --------- -------- --------- ------- --------- -----------
>  1 /dev/sdc1 444.67GiB  8.00MiB   6.00GiB 4.00MiB 128.00MiB    14.95GiB
> -- --------- --------- -------- --------- ------- --------- -----------
>    Total     444.67GiB  8.00MiB   3.00GiB 4.00MiB  64.00MiB    14.95GiB
>    Used      231.60GiB    0.00B 335.98MiB   0.00B  80.00KiB  
> 
> Will run a scrub next to see whether all data can be read.
> 
> Device stats are okay:
> 
> % btrfs device stats /mnt/amazon
> [/dev/sdc1].write_io_errs    0
> [/dev/sdc1].read_io_errs     0
> [/dev/sdc1].flush_io_errs    0
> [/dev/sdc1].corruption_errs  0
> [/dev/sdc1].generation_errs  0
> 
> Device reports SMART status as: passed.
> 
> There are some "Error: ICRC, ABRT 24 sectors at LBA" like errors in 
> SMART error log, but they are from a very early time about 290 to 310 
> hours of disk power-on lifetime. I believe this may have been me 
> disconnecting the device while it was still being written to or 
> something, but I will run a long SMART test as well.
> 
> Best,
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: external harddisk: bogus corrupt leaf error?

[Martin Steigerwald]

Qu Wenruo - 21.09.20, 12:09:01 CEST:
> On 2020/9/21 下午5:29, Martin Steigerwald wrote:
> > Hi!
> > 
> > I have an external 500 GB harddisk with BTRFS. On mounting it the
> > kernel (5.9-rc5, vanilla, self compiled) reports:
> > 
> > [282409.344208] BTRFS info (device sdc1): enabling auto defrag
> > [282409.344222] BTRFS info (device sdc1): use zstd compression,
> > level 3 [282409.344225] BTRFS info (device sdc1): disk space
> > caching is enabled [282409.465837] BTRFS critical (device sdc1):
> > corrupt leaf: root=1 block=906756096 slot=204, invalid root item
> > size, have 239 expect 439
> This one can only be detected by kernel, not btrfs check yet.
> 
> Recently kernel has much more strict checks than btrfs-check,
> sometimes it can be too strict, as some error is not really going to
> cause problems, but just against on-disk format.
> 
> And this is the case.
> 
> In theory, you can mount the fs with older kernel, any kernel older
> than commit 259ee7754b67 ("btrfs: tree-checker: Add ROOT_ITEM check")
> should still be able to mount the fs.

Oh, I can still mount the filesystem just fine, so no problem there.

> For workaround, you can dump the tree block 906756096, locate the slot
> 204, see what tree root it is.

While mounted, as the scrub is still running:

btrfs-progs v5.7 
leaf 906756096 items 205 free space 2555 generation 12080 owner ROOT_TREE
leaf 906756096 flags 0x1(WRITTEN) backref revision 1
fs uuid […]

[…]

        item 204 key (DATA_RELOC_TREE ROOT_ITEM 0) itemoff 7680 itemsize 239
                generation 4 root_dirid 256 bytenr 29442048 level 0 refs 1
                lastsnap 0 byte_limit 0 bytes_used 16384 flags 0x0(none)
                drop key (0 UNKNOWN.0 0) level 0

Now what does that tell me?

> If it's a subvolume/snapshot, deleting it should solve the problem,
> even for the latest kernel.

The device has just one subvolume except root subvolume:

% btrfs subvol list /mnt/amazon 
ID 258 gen 12560 top level 5 path daten

> For the root cause, it should be some older kernel creating the wrong
> root item size.
> I can't find the commit but it should be pretty old, as after v5.4 we
> have mandatory write time tree checks, which will reject such write
> directly.

So eventually I would have to backup the disk and create FS from scratch
to get rid of the error? Or can I, even if its no subvolume involved, find the
item affected, copy it somewhere else and then write it to the disk again?

> Thanks,
> Qu

Somehow I am reminded of mister Q in Star Trek… :)

Thank you!
Martin
 
> > Note: It has used LZO compression before, but I switched mount
> > option to zstd meanwhile.
> > 
> > However, btrfs-progds 5.7 gives:
> > 
> > % btrfs check /dev/sdc1
> > Opening filesystem to check...
> > Checking filesystem on /dev/sdc1
> > UUID: […]
> > [1/7] checking root items
> > [2/7] checking extents
> > [3/7] checking free space cache
> > [4/7] checking fs roots
> > [5/7] checking only csums items (without verifying data)
> > [6/7] checking root refs
> > [7/7] checking quota groups skipped (not enabled on this FS)
> > found 249031409664 bytes used, no error found
> > total csum bytes: 242738928
> > total tree bytes: 352387072
> > total fs tree bytes: 67747840
> > total extent tree bytes: 14565376
> > btree space waste bytes: 37691414
> > file data blocks allocated: 1067158315008
> > 
> >  referenced 247077785600
> > 
> > Is this kernel message in error? Or does 'btrfs check' not check for
> > this error yet?
> > 
> > Here some more information:
[…]
-- 
Martin

Re: external harddisk: bogus corrupt leaf error?

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 4378 bytes --]



On 2020/9/21 下午6:30, Martin Steigerwald wrote:
> Qu Wenruo - 21.09.20, 12:09:01 CEST:
>> On 2020/9/21 下午5:29, Martin Steigerwald wrote:
>>> Hi!
>>>
>>> I have an external 500 GB harddisk with BTRFS. On mounting it the
>>> kernel (5.9-rc5, vanilla, self compiled) reports:
>>>
>>> [282409.344208] BTRFS info (device sdc1): enabling auto defrag
>>> [282409.344222] BTRFS info (device sdc1): use zstd compression,
>>> level 3 [282409.344225] BTRFS info (device sdc1): disk space
>>> caching is enabled [282409.465837] BTRFS critical (device sdc1):
>>> corrupt leaf: root=1 block=906756096 slot=204, invalid root item
>>> size, have 239 expect 439
>> This one can only be detected by kernel, not btrfs check yet.
>>
>> Recently kernel has much more strict checks than btrfs-check,
>> sometimes it can be too strict, as some error is not really going to
>> cause problems, but just against on-disk format.
>>
>> And this is the case.
>>
>> In theory, you can mount the fs with older kernel, any kernel older
>> than commit 259ee7754b67 ("btrfs: tree-checker: Add ROOT_ITEM check")
>> should still be able to mount the fs.
> 
> Oh, I can still mount the filesystem just fine, so no problem there.
> 
>> For workaround, you can dump the tree block 906756096, locate the slot
>> 204, see what tree root it is.
> 
> While mounted, as the scrub is still running:
> 
> btrfs-progs v5.7 
> leaf 906756096 items 205 free space 2555 generation 12080 owner ROOT_TREE
> leaf 906756096 flags 0x1(WRITTEN) backref revision 1
> fs uuid […]
> 
> […]
> 
>         item 204 key (DATA_RELOC_TREE ROOT_ITEM 0) itemoff 7680 itemsize 239
>                 generation 4 root_dirid 256 bytenr 29442048 level 0 refs 1
>                 lastsnap 0 byte_limit 0 bytes_used 16384 flags 0x0(none)
>                 drop key (0 UNKNOWN.0 0) level 0
> 
> Now what does that tell me?

Oh, that's data reloc tree. It can't be easily fixed by regular
operations, we may have to craft a special repairer for you to repair that.

> 
>> If it's a subvolume/snapshot, deleting it should solve the problem,
>> even for the latest kernel.
> 
> The device has just one subvolume except root subvolume:
> 
> % btrfs subvol list /mnt/amazon 
> ID 258 gen 12560 top level 5 path daten

Yep, that data reloc tree is not visible to the user, thus much tricker
to handle.

> 
>> For the root cause, it should be some older kernel creating the wrong
>> root item size.
>> I can't find the commit but it should be pretty old, as after v5.4 we
>> have mandatory write time tree checks, which will reject such write
>> directly.
> 
> So eventually I would have to backup the disk and create FS from scratch
> to get rid of the error? Or can I, even if its no subvolume involved, find the
> item affected, copy it somewhere else and then write it to the disk again?

That's the theory.

We can easily rebuild that data reloc tree, since it should be empty if
balance is not running.

But we don't have it ready at hand in btrfs-progs...

So you may either want to wait until some quick dirty fixer arrives, or
can start backup right now.
All the data/files shouldn't be affected at all.

Thanks,
Qu

> 
>> Thanks,
>> Qu
> 
> Somehow I am reminded of mister Q in Star Trek… :)
> 
> Thank you!
> Martin
>  
>>> Note: It has used LZO compression before, but I switched mount
>>> option to zstd meanwhile.
>>>
>>> However, btrfs-progds 5.7 gives:
>>>
>>> % btrfs check /dev/sdc1
>>> Opening filesystem to check...
>>> Checking filesystem on /dev/sdc1
>>> UUID: […]
>>> [1/7] checking root items
>>> [2/7] checking extents
>>> [3/7] checking free space cache
>>> [4/7] checking fs roots
>>> [5/7] checking only csums items (without verifying data)
>>> [6/7] checking root refs
>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>> found 249031409664 bytes used, no error found
>>> total csum bytes: 242738928
>>> total tree bytes: 352387072
>>> total fs tree bytes: 67747840
>>> total extent tree bytes: 14565376
>>> btree space waste bytes: 37691414
>>> file data blocks allocated: 1067158315008
>>>
>>>  referenced 247077785600
>>>
>>> Is this kernel message in error? Or does 'btrfs check' not check for
>>> this error yet?
>>>
>>> Here some more information:
> […]
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: external harddisk: bogus corrupt leaf error?

[Martin Steigerwald]

Qu Wenruo - 21.09.20, 13:14:05 CEST:
> >> For the root cause, it should be some older kernel creating the
> >> wrong
> >> root item size.
> >> I can't find the commit but it should be pretty old, as after v5.4
> >> we
> >> have mandatory write time tree checks, which will reject such write
> >> directly.
> > 
> > So eventually I would have to backup the disk and create FS from
> > scratch to get rid of the error? Or can I, even if its no subvolume
> > involved, find the item affected, copy it somewhere else and then
> > write it to the disk again?
> That's the theory.
> 
> We can easily rebuild that data reloc tree, since it should be empty
> if balance is not running.
> 
> But we don't have it ready at hand in btrfs-progs...
> 
> So you may either want to wait until some quick dirty fixer arrives,
> or can start backup right now.
> All the data/files shouldn't be affected at all.

Hmmm, do you have an idea if and when such a quick dirty fixer would be 
available?

Also, is it still safe to write to the filesystem? I looked at the disk, 
cause I wanted to move some large files over to it to free up some space 
on my laptop's internal SSDs.

If its still safe to write to the filesystem, I may just wait. I will 
refresh the backup of the disk anyway. But if its not safe to write to 
it anymore, I would redo the filesystem from scratch. Would give the 
added benefit of having everything zstd compressed and I could also go 
for XXHASH or what one of the faster of the new checksum algorithms was.

Best,
-- 
Martin

Re: external harddisk: bogus corrupt leaf error?

[Chris Murphy]

On Mon, Sep 21, 2020 at 5:46 AM Martin Steigerwald <martin@lichtvoll.de> wrote:
>
> Qu Wenruo - 21.09.20, 13:14:05 CEST:
> > >> For the root cause, it should be some older kernel creating the
> > >> wrong
> > >> root item size.
> > >> I can't find the commit but it should be pretty old, as after v5.4
> > >> we
> > >> have mandatory write time tree checks, which will reject such write
> > >> directly.
> > >
> > > So eventually I would have to backup the disk and create FS from
> > > scratch to get rid of the error? Or can I, even if its no subvolume
> > > involved, find the item affected, copy it somewhere else and then
> > > write it to the disk again?
> > That's the theory.
> >
> > We can easily rebuild that data reloc tree, since it should be empty
> > if balance is not running.
> >
> > But we don't have it ready at hand in btrfs-progs...
> >
> > So you may either want to wait until some quick dirty fixer arrives,
> > or can start backup right now.
> > All the data/files shouldn't be affected at all.
>
> Hmmm, do you have an idea if and when such a quick dirty fixer would be
> available?
>
> Also, is it still safe to write to the filesystem? I looked at the disk,
> cause I wanted to move some large files over to it to free up some space
> on my laptop's internal SSDs.
>
> If its still safe to write to the filesystem, I may just wait. I will
> refresh the backup of the disk anyway. But if its not safe to write to
> it anymore, I would redo the filesystem from scratch. Would give the
> added benefit of having everything zstd compressed and I could also go
> for XXHASH or what one of the faster of the new checksum algorithms was.

xxhash is about tied with crc32c, but it's hardware specific. Chances
are the computation of the checksums will have no affect on
performance. But the size of the checksum can. The xxhash64 checksum
is twice the size of crc32c, so it means the csum tree is double. As
there can be some latency as a result of csum writes, when doing heavy
writes, moving to bigger csums increases this latency. If you don't
have heavy writes workloads, it's unlikely to matter. To avoid
collisions with crc32c, it probably starts to makes sense to use
xxhash or better once the file system size gets above 8TB.



-- 
Chris Murphy

Re: external harddisk: bogus corrupt leaf error?

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 1802 bytes --]



On 2020/9/21 下午7:46, Martin Steigerwald wrote:
> Qu Wenruo - 21.09.20, 13:14:05 CEST:
>>>> For the root cause, it should be some older kernel creating the
>>>> wrong
>>>> root item size.
>>>> I can't find the commit but it should be pretty old, as after v5.4
>>>> we
>>>> have mandatory write time tree checks, which will reject such write
>>>> directly.
>>>
>>> So eventually I would have to backup the disk and create FS from
>>> scratch to get rid of the error? Or can I, even if its no subvolume
>>> involved, find the item affected, copy it somewhere else and then
>>> write it to the disk again?
>> That's the theory.
>>
>> We can easily rebuild that data reloc tree, since it should be empty
>> if balance is not running.
>>
>> But we don't have it ready at hand in btrfs-progs...
>>
>> So you may either want to wait until some quick dirty fixer arrives,
>> or can start backup right now.
>> All the data/files shouldn't be affected at all.
> 
> Hmmm, do you have an idea if and when such a quick dirty fixer would be 
> available?

If you need, I guess in 24 hours.

> 
> Also, is it still safe to write to the filesystem? I looked at the disk, 
> cause I wanted to move some large files over to it to free up some space 
> on my laptop's internal SSDs.

Yes. If you want to be extra safe, just don't utilize balance until it's
fixed.

Thanks,
Qu

> 
> If its still safe to write to the filesystem, I may just wait. I will 
> refresh the backup of the disk anyway. But if its not safe to write to 
> it anymore, I would redo the filesystem from scratch. Would give the 
> added benefit of having everything zstd compressed and I could also go 
> for XXHASH or what one of the faster of the new checksum algorithms was.
> 
> Best,
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: external harddisk: bogus corrupt leaf error?

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2593 bytes --]



On 2020/9/22 上午7:48, Qu Wenruo wrote:
> 
> 
> On 2020/9/21 下午7:46, Martin Steigerwald wrote:
>> Qu Wenruo - 21.09.20, 13:14:05 CEST:
>>>>> For the root cause, it should be some older kernel creating the
>>>>> wrong
>>>>> root item size.
>>>>> I can't find the commit but it should be pretty old, as after v5.4
>>>>> we
>>>>> have mandatory write time tree checks, which will reject such write
>>>>> directly.
>>>>
>>>> So eventually I would have to backup the disk and create FS from
>>>> scratch to get rid of the error? Or can I, even if its no subvolume
>>>> involved, find the item affected, copy it somewhere else and then
>>>> write it to the disk again?
>>> That's the theory.
>>>
>>> We can easily rebuild that data reloc tree, since it should be empty
>>> if balance is not running.
>>>
>>> But we don't have it ready at hand in btrfs-progs...
>>>
>>> So you may either want to wait until some quick dirty fixer arrives,
>>> or can start backup right now.
>>> All the data/files shouldn't be affected at all.
>>
>> Hmmm, do you have an idea if and when such a quick dirty fixer would be 
>> available?
> 
> If you need, I guess in 24 hours.

Here you go:
https://github.com/adam900710/btrfs-progs/tree/dirty_fix

You need to compile the btrfs-progs (in fact, you need to compile
btrfs-corrupt-block).
Then execute:
# ./btrfs-corrupt-block -X <device>

It should solve the problem.
If nothing is output, and no crash, then the repair is done.
Or you will see a crash with calltrace, and your on-disk data is untouched.


The root problem turns out to be a false alert.

It's possible to have an old root item, which is smaller than current
root_item.
In that case, current kernel can handle it without problem.

I'll fix the problem in the kernel too to prevent further false alerts.

Thanks,
Qu

> 
>>
>> Also, is it still safe to write to the filesystem? I looked at the disk, 
>> cause I wanted to move some large files over to it to free up some space 
>> on my laptop's internal SSDs.
> 
> Yes. If you want to be extra safe, just don't utilize balance until it's
> fixed.
> 
> Thanks,
> Qu
> 
>>
>> If its still safe to write to the filesystem, I may just wait. I will 
>> refresh the backup of the disk anyway. But if its not safe to write to 
>> it anymore, I would redo the filesystem from scratch. Would give the 
>> added benefit of having everything zstd compressed and I could also go 
>> for XXHASH or what one of the faster of the new checksum algorithms was.
>>
>> Best,
>>
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: external harddisk: bogus corrupt leaf error?

[Martin Steigerwald]

Qu Wenruo - 22.09.20, 04:14:34 CEST:
> On 2020/9/22 上午7:48, Qu Wenruo wrote:
> > On 2020/9/21 下午7:46, Martin Steigerwald wrote:
> >> Qu Wenruo - 21.09.20, 13:14:05 CEST:
> >>>>> For the root cause, it should be some older kernel creating the
> >>>>> wrong
> >>>>> root item size.
> >>>>> I can't find the commit but it should be pretty old, as after
> >>>>> v5.4
> >>>>> we
> >>>>> have mandatory write time tree checks, which will reject such
> >>>>> write
> >>>>> directly.
> >>>> 
> >>>> So eventually I would have to backup the disk and create FS from
> >>>> scratch to get rid of the error? Or can I, even if its no
> >>>> subvolume
> >>>> involved, find the item affected, copy it somewhere else and then
> >>>> write it to the disk again?
> >>> 
> >>> That's the theory.
> >>> 
> >>> We can easily rebuild that data reloc tree, since it should be
> >>> empty
> >>> if balance is not running.
> >>> 
> >>> But we don't have it ready at hand in btrfs-progs...
> >>> 
> >>> So you may either want to wait until some quick dirty fixer
> >>> arrives,
> >>> or can start backup right now.
> >>> All the data/files shouldn't be affected at all.
> >> 
> >> Hmmm, do you have an idea if and when such a quick dirty fixer
> >> would be available?
> > 
> > If you need, I guess in 24 hours.
> 
> Here you go:
> https://github.com/adam900710/btrfs-progs/tree/dirty_fix
> 
> You need to compile the btrfs-progs (in fact, you need to compile
> btrfs-corrupt-block).
> Then execute:
> # ./btrfs-corrupt-block -X <device>
> 
> It should solve the problem.
> If nothing is output, and no crash, then the repair is done.
> Or you will see a crash with calltrace, and your on-disk data is
> untouched.

Thank you very much for the prompt delivery of that tool.

No, in case its safe enough to write it would have not been that urgent.

I hope to find time to try it out this evening. Currently in a training.

Ciao,
-- 
Martin

Re: external harddisk: bogus corrupt leaf error?

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2313 bytes --]



On 2020/9/22 下午4:40, Martin Steigerwald wrote:
> Qu Wenruo - 22.09.20, 04:14:34 CEST:
>> On 2020/9/22 上午7:48, Qu Wenruo wrote:
>>> On 2020/9/21 下午7:46, Martin Steigerwald wrote:
>>>> Qu Wenruo - 21.09.20, 13:14:05 CEST:
>>>>>>> For the root cause, it should be some older kernel creating the
>>>>>>> wrong
>>>>>>> root item size.
>>>>>>> I can't find the commit but it should be pretty old, as after
>>>>>>> v5.4
>>>>>>> we
>>>>>>> have mandatory write time tree checks, which will reject such
>>>>>>> write
>>>>>>> directly.
>>>>>>
>>>>>> So eventually I would have to backup the disk and create FS from
>>>>>> scratch to get rid of the error? Or can I, even if its no
>>>>>> subvolume
>>>>>> involved, find the item affected, copy it somewhere else and then
>>>>>> write it to the disk again?
>>>>>
>>>>> That's the theory.
>>>>>
>>>>> We can easily rebuild that data reloc tree, since it should be
>>>>> empty
>>>>> if balance is not running.
>>>>>
>>>>> But we don't have it ready at hand in btrfs-progs...
>>>>>
>>>>> So you may either want to wait until some quick dirty fixer
>>>>> arrives,
>>>>> or can start backup right now.
>>>>> All the data/files shouldn't be affected at all.
>>>>
>>>> Hmmm, do you have an idea if and when such a quick dirty fixer
>>>> would be available?
>>>
>>> If you need, I guess in 24 hours.
>>
>> Here you go:
>> https://github.com/adam900710/btrfs-progs/tree/dirty_fix
>>
>> You need to compile the btrfs-progs (in fact, you need to compile
>> btrfs-corrupt-block).
>> Then execute:
>> # ./btrfs-corrupt-block -X <device>
>>
>> It should solve the problem.
>> If nothing is output, and no crash, then the repair is done.
>> Or you will see a crash with calltrace, and your on-disk data is
>> untouched.
> 
> Thank you very much for the prompt delivery of that tool.
> 
> No, in case its safe enough to write it would have not been that urgent.

Since I have manually crafted an image with the same situation, and
tested, it should be OK.
> 
> I hope to find time to try it out this evening. Currently in a training.

Another solution is waiting for the kernel patch to arrive.
That one should be 100% safe.

I hope that one can be merged in v5.9-rc cycle.

Thanks,
Qu

> 
> Ciao,
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]