* Fwd: Read time tree block corruption detected
[not found] <CAJheHN0FUe-ijMco1ZOc6iKF2zbPocOw+iiVNeTT1r-JuXOJww@mail.gmail.com>
@ 2020-05-06 21:54 ` Tyler Richmond
2020-05-06 23:55 ` Chris Murphy
2020-05-07 1:13 ` Fwd: " Qu Wenruo
0 siblings, 2 replies; 56+ messages in thread
From: Tyler Richmond @ 2020-05-06 21:54 UTC (permalink / raw)
To: linux-btrfs
Hello,
I looked up this error and it basically says ask a developer to
determine if it's a false error or not. I just started getting some
slow response times, and looked at the dmesg log to find a ton of
these errors.
[192088.446299] BTRFS critical (device sdh): corrupt leaf: root=5
block=203510940835840 slot=4 ino=1311670, invalid inode generation:
has 18446744073709551492 expect [0, 6875827]
[192088.449823] BTRFS error (device sdh): block=203510940835840 read
time tree block corruption detected
[192088.459238] BTRFS critical (device sdh): corrupt leaf: root=5
block=203510940835840 slot=4 ino=1311670, invalid inode generation:
has 18446744073709551492 expect [0, 6875827]
[192088.462773] BTRFS error (device sdh): block=203510940835840 read
time tree block corruption detected
[192088.464711] BTRFS critical (device sdh): corrupt leaf: root=5
block=203510940835840 slot=4 ino=1311670, invalid inode generation:
has 18446744073709551492 expect [0, 6875827]
[192088.468457] BTRFS error (device sdh): block=203510940835840 read
time tree block corruption detected
btrfs device stats, however, doesn't show any errors.
Is there anything I should do about this, or should I just continue
using my array as normal?
Thank you!
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Read time tree block corruption detected
2020-05-06 21:54 ` Fwd: Read time tree block corruption detected Tyler Richmond
@ 2020-05-06 23:55 ` Chris Murphy
2020-05-07 0:51 ` Tyler Richmond
2020-05-07 1:13 ` Fwd: " Qu Wenruo
1 sibling, 1 reply; 56+ messages in thread
From: Chris Murphy @ 2020-05-06 23:55 UTC (permalink / raw)
To: Tyler Richmond; +Cc: Btrfs BTRFS
On Wed, May 6, 2020 at 3:54 PM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>
> Hello,
>
> I looked up this error and it basically says ask a developer to
> determine if it's a false error or not. I just started getting some
> slow response times, and looked at the dmesg log to find a ton of
> these errors.
>
> [192088.446299] BTRFS critical (device sdh): corrupt leaf: root=5
> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> has 18446744073709551492 expect [0, 6875827]
> [192088.449823] BTRFS error (device sdh): block=203510940835840 read
> time tree block corruption detected
> [192088.459238] BTRFS critical (device sdh): corrupt leaf: root=5
> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> has 18446744073709551492 expect [0, 6875827]
> [192088.462773] BTRFS error (device sdh): block=203510940835840 read
> time tree block corruption detected
> [192088.464711] BTRFS critical (device sdh): corrupt leaf: root=5
> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> has 18446744073709551492 expect [0, 6875827]
> [192088.468457] BTRFS error (device sdh): block=203510940835840 read
> time tree block corruption detected
>
> btrfs device stats, however, doesn't show any errors.
>
> Is there anything I should do about this, or should I just continue
> using my array as normal?
What kernel version? This looks like relatively recent kernel
reporting format. Can you search for inode 1311670? e.g.
$ sudo btrfs insp ino -v 1311670 /mountpoint
Note that each subvolume has its own set of inodes. You need to point
the command to the correct subvolume. In this case it's root=5 which
is the default/top-level. As long as you haven't changed the default
subvolume, and you've mounted the file system without subvol or
subvolid option, it should point to the correct file that's affected
by this. And also maybe useful:
$ sudo btrfs insp dump-t -b 203510940835840 /mountpoint
--
Chris Murphy
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Read time tree block corruption detected
2020-05-06 23:55 ` Chris Murphy
@ 2020-05-07 0:51 ` Tyler Richmond
2020-05-07 1:06 ` Chris Murphy
0 siblings, 1 reply; 56+ messages in thread
From: Tyler Richmond @ 2020-05-07 0:51 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS
Kernel version is 5.4.0-28. Just upgraded to Ubuntu 20.04.
btrfs-progs is version 5.6
That inode search returned an error:
ERROR: ino paths ioctl: Input/output error
I don't use any subvolumes.
The next command also resulted in an error. Replaced the actual
mountpoint in the error with /mountpoint.
ERROR: not a block device or regular file: /mountpoint/
ERROR: device scan /mountpoint/: Input/output error
Something definitely doesn't seem right.
On Wed, May 6, 2020 at 7:55 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Wed, May 6, 2020 at 3:54 PM Tyler Richmond <t.d.richmond@gmail.com> wrote:
> >
> > Hello,
> >
> > I looked up this error and it basically says ask a developer to
> > determine if it's a false error or not. I just started getting some
> > slow response times, and looked at the dmesg log to find a ton of
> > these errors.
> >
> > [192088.446299] BTRFS critical (device sdh): corrupt leaf: root=5
> > block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> > has 18446744073709551492 expect [0, 6875827]
> > [192088.449823] BTRFS error (device sdh): block=203510940835840 read
> > time tree block corruption detected
> > [192088.459238] BTRFS critical (device sdh): corrupt leaf: root=5
> > block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> > has 18446744073709551492 expect [0, 6875827]
> > [192088.462773] BTRFS error (device sdh): block=203510940835840 read
> > time tree block corruption detected
> > [192088.464711] BTRFS critical (device sdh): corrupt leaf: root=5
> > block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> > has 18446744073709551492 expect [0, 6875827]
> > [192088.468457] BTRFS error (device sdh): block=203510940835840 read
> > time tree block corruption detected
> >
> > btrfs device stats, however, doesn't show any errors.
> >
> > Is there anything I should do about this, or should I just continue
> > using my array as normal?
>
> What kernel version? This looks like relatively recent kernel
> reporting format. Can you search for inode 1311670? e.g.
>
> $ sudo btrfs insp ino -v 1311670 /mountpoint
>
> Note that each subvolume has its own set of inodes. You need to point
> the command to the correct subvolume. In this case it's root=5 which
> is the default/top-level. As long as you haven't changed the default
> subvolume, and you've mounted the file system without subvol or
> subvolid option, it should point to the correct file that's affected
> by this. And also maybe useful:
>
> $ sudo btrfs insp dump-t -b 203510940835840 /mountpoint
>
>
>
> --
> Chris Murphy
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Read time tree block corruption detected
2020-05-07 0:51 ` Tyler Richmond
@ 2020-05-07 1:06 ` Chris Murphy
0 siblings, 0 replies; 56+ messages in thread
From: Chris Murphy @ 2020-05-07 1:06 UTC (permalink / raw)
To: Tyler Richmond; +Cc: Chris Murphy, Btrfs BTRFS
On Wed, May 6, 2020 at 6:51 PM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>
> Kernel version is 5.4.0-28. Just upgraded to Ubuntu 20.04.
> btrfs-progs is version 5.6
>
> That inode search returned an error:
> ERROR: ino paths ioctl: Input/output error
>
> I don't use any subvolumes.
>
> The next command also resulted in an error. Replaced the actual
> mountpoint in the error with /mountpoint.
>
> ERROR: not a block device or regular file: /mountpoint/
> ERROR: device scan /mountpoint/: Input/output error
>
> Something definitely doesn't seem right.
You need to substitute the /mountpoint with the actual mountpoint of
the file system, be it / or /home or /mnt
--
Chris Murphy
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-05-06 21:54 ` Fwd: Read time tree block corruption detected Tyler Richmond
2020-05-06 23:55 ` Chris Murphy
@ 2020-05-07 1:13 ` Qu Wenruo
2020-05-07 1:30 ` Tyler Richmond
1 sibling, 1 reply; 56+ messages in thread
From: Qu Wenruo @ 2020-05-07 1:13 UTC (permalink / raw)
To: Tyler Richmond, linux-btrfs
[-- Attachment #1.1: Type: text/plain, Size: 1734 bytes --]
On 2020/5/7 上午5:54, Tyler Richmond wrote:
> Hello,
>
> I looked up this error and it basically says ask a developer to
> determine if it's a false error or not. I just started getting some
> slow response times, and looked at the dmesg log to find a ton of
> these errors.
>
> [192088.446299] BTRFS critical (device sdh): corrupt leaf: root=5
> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> has 18446744073709551492 expect [0, 6875827]
> [192088.449823] BTRFS error (device sdh): block=203510940835840 read
> time tree block corruption detected
> [192088.459238] BTRFS critical (device sdh): corrupt leaf: root=5
> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> has 18446744073709551492 expect [0, 6875827]
> [192088.462773] BTRFS error (device sdh): block=203510940835840 read
> time tree block corruption detected
> [192088.464711] BTRFS critical (device sdh): corrupt leaf: root=5
> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> has 18446744073709551492 expect [0, 6875827]
> [192088.468457] BTRFS error (device sdh): block=203510940835840 read
> time tree block corruption detected
>
> btrfs device stats, however, doesn't show any errors.
>
> Is there anything I should do about this, or should I just continue
> using my array as normal?
This is caused by older kernel underflow inode generation.
Latest btrfs-progs can fix it, using btrfs check --repair.
Or you can go safer, by manually locating the inode using its inode
number (1311670), and copy it to some new location using previous
working kernel, then delete the old file, copy the new one back to fix it.
Thanks,
Qu
>
> Thank you!
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-05-07 1:13 ` Fwd: " Qu Wenruo
@ 2020-05-07 1:30 ` Tyler Richmond
2020-05-07 5:43 ` Tyler Richmond
0 siblings, 1 reply; 56+ messages in thread
From: Tyler Richmond @ 2020-05-07 1:30 UTC (permalink / raw)
To: Qu Wenruo, lists; +Cc: linux-btrfs
Chris, I had used the correct mountpoint in the command. I just edited
it in the email to be /mountpoint for consistency.
Qu, I'll try the repair. Fingers crossed!
On Wed, May 6, 2020 at 9:13 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2020/5/7 上午5:54, Tyler Richmond wrote:
> > Hello,
> >
> > I looked up this error and it basically says ask a developer to
> > determine if it's a false error or not. I just started getting some
> > slow response times, and looked at the dmesg log to find a ton of
> > these errors.
> >
> > [192088.446299] BTRFS critical (device sdh): corrupt leaf: root=5
> > block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> > has 18446744073709551492 expect [0, 6875827]
> > [192088.449823] BTRFS error (device sdh): block=203510940835840 read
> > time tree block corruption detected
> > [192088.459238] BTRFS critical (device sdh): corrupt leaf: root=5
> > block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> > has 18446744073709551492 expect [0, 6875827]
> > [192088.462773] BTRFS error (device sdh): block=203510940835840 read
> > time tree block corruption detected
> > [192088.464711] BTRFS critical (device sdh): corrupt leaf: root=5
> > block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> > has 18446744073709551492 expect [0, 6875827]
> > [192088.468457] BTRFS error (device sdh): block=203510940835840 read
> > time tree block corruption detected
> >
> > btrfs device stats, however, doesn't show any errors.
> >
> > Is there anything I should do about this, or should I just continue
> > using my array as normal?
>
> This is caused by older kernel underflow inode generation.
>
> Latest btrfs-progs can fix it, using btrfs check --repair.
>
> Or you can go safer, by manually locating the inode using its inode
> number (1311670), and copy it to some new location using previous
> working kernel, then delete the old file, copy the new one back to fix it.
>
> Thanks,
> Qu
>
> >
> > Thank you!
> >
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-05-07 1:30 ` Tyler Richmond
@ 2020-05-07 5:43 ` Tyler Richmond
2020-05-07 5:52 ` Qu Wenruo
0 siblings, 1 reply; 56+ messages in thread
From: Tyler Richmond @ 2020-05-07 5:43 UTC (permalink / raw)
To: Qu Wenruo, Chris Murphy; +Cc: Btrfs BTRFS
Well, the repair doesn't look terribly successful.
parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
parent level=1
child level=4
parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
parent level=1
child level=4
parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
parent level=1
child level=4
parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
parent level=1
child level=4
parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
parent level=1
child level=4
parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
parent level=1
child level=4
parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
parent level=1
child level=4
parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
parent level=1
child level=4
parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
parent level=1
child level=4
parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
parent level=1
child level=4
parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
parent level=1
child level=4
ERROR: failed to zero log tree: -17
ERROR: attempt to start transaction over already running one
WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
extent buffer leak: start 225049066086400 len 4096
extent buffer leak: start 225049066086400 len 4096
WARNING: dirty eb leak (aborted trans): start 225049066086400 len 4096
extent buffer leak: start 225049066094592 len 4096
extent buffer leak: start 225049066094592 len 4096
WARNING: dirty eb leak (aborted trans): start 225049066094592 len 4096
extent buffer leak: start 225049066102784 len 4096
extent buffer leak: start 225049066102784 len 4096
WARNING: dirty eb leak (aborted trans): start 225049066102784 len 4096
extent buffer leak: start 225049066131456 len 4096
extent buffer leak: start 225049066131456 len 4096
WARNING: dirty eb leak (aborted trans): start 225049066131456 len 4096
What is going on?
On Wed, May 6, 2020 at 9:30 PM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>
> Chris, I had used the correct mountpoint in the command. I just edited
> it in the email to be /mountpoint for consistency.
>
> Qu, I'll try the repair. Fingers crossed!
>
> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >
> >
> >
> > On 2020/5/7 上午5:54, Tyler Richmond wrote:
> > > Hello,
> > >
> > > I looked up this error and it basically says ask a developer to
> > > determine if it's a false error or not. I just started getting some
> > > slow response times, and looked at the dmesg log to find a ton of
> > > these errors.
> > >
> > > [192088.446299] BTRFS critical (device sdh): corrupt leaf: root=5
> > > block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> > > has 18446744073709551492 expect [0, 6875827]
> > > [192088.449823] BTRFS error (device sdh): block=203510940835840 read
> > > time tree block corruption detected
> > > [192088.459238] BTRFS critical (device sdh): corrupt leaf: root=5
> > > block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> > > has 18446744073709551492 expect [0, 6875827]
> > > [192088.462773] BTRFS error (device sdh): block=203510940835840 read
> > > time tree block corruption detected
> > > [192088.464711] BTRFS critical (device sdh): corrupt leaf: root=5
> > > block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> > > has 18446744073709551492 expect [0, 6875827]
> > > [192088.468457] BTRFS error (device sdh): block=203510940835840 read
> > > time tree block corruption detected
> > >
> > > btrfs device stats, however, doesn't show any errors.
> > >
> > > Is there anything I should do about this, or should I just continue
> > > using my array as normal?
> >
> > This is caused by older kernel underflow inode generation.
> >
> > Latest btrfs-progs can fix it, using btrfs check --repair.
> >
> > Or you can go safer, by manually locating the inode using its inode
> > number (1311670), and copy it to some new location using previous
> > working kernel, then delete the old file, copy the new one back to fix it.
> >
> > Thanks,
> > Qu
> >
> > >
> > > Thank you!
> > >
> >
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-05-07 5:43 ` Tyler Richmond
@ 2020-05-07 5:52 ` Qu Wenruo
2020-05-07 15:52 ` Tyler Richmond
0 siblings, 1 reply; 56+ messages in thread
From: Qu Wenruo @ 2020-05-07 5:52 UTC (permalink / raw)
To: Tyler Richmond, Chris Murphy; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 6567 bytes --]
On 2020/5/7 下午1:43, Tyler Richmond wrote:
> Well, the repair doesn't look terribly successful.
>
> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> parent level=1
> child level=4
This means there are more problems, not only the hash name mismatch.
This means the fs is already corrupted, the name hash is just one
unrelated symptom.
The only good news is, btrfs-progs abort the transaction, thus no
further damage to the fs.
Please run a plain btrfs-check to show what's the problem first.
Thanks,
Qu
> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> parent level=1
> child level=4
> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> parent level=1
> child level=4
> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> parent level=1
> child level=4
> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> parent level=1
> child level=4
> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> parent level=1
> child level=4
> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> parent level=1
> child level=4
> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> parent level=1
> child level=4
> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> parent level=1
> child level=4
> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> parent level=1
> child level=4
> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> parent level=1
> child level=4
> ERROR: failed to zero log tree: -17
> ERROR: attempt to start transaction over already running one
> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
> extent buffer leak: start 225049066086400 len 4096
> extent buffer leak: start 225049066086400 len 4096
> WARNING: dirty eb leak (aborted trans): start 225049066086400 len 4096
> extent buffer leak: start 225049066094592 len 4096
> extent buffer leak: start 225049066094592 len 4096
> WARNING: dirty eb leak (aborted trans): start 225049066094592 len 4096
> extent buffer leak: start 225049066102784 len 4096
> extent buffer leak: start 225049066102784 len 4096
> WARNING: dirty eb leak (aborted trans): start 225049066102784 len 4096
> extent buffer leak: start 225049066131456 len 4096
> extent buffer leak: start 225049066131456 len 4096
> WARNING: dirty eb leak (aborted trans): start 225049066131456 len 4096
>
> What is going on?
>
> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>>
>> Chris, I had used the correct mountpoint in the command. I just edited
>> it in the email to be /mountpoint for consistency.
>>
>> Qu, I'll try the repair. Fingers crossed!
>>
>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>
>>>
>>>
>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
>>>> Hello,
>>>>
>>>> I looked up this error and it basically says ask a developer to
>>>> determine if it's a false error or not. I just started getting some
>>>> slow response times, and looked at the dmesg log to find a ton of
>>>> these errors.
>>>>
>>>> [192088.446299] BTRFS critical (device sdh): corrupt leaf: root=5
>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
>>>> has 18446744073709551492 expect [0, 6875827]
>>>> [192088.449823] BTRFS error (device sdh): block=203510940835840 read
>>>> time tree block corruption detected
>>>> [192088.459238] BTRFS critical (device sdh): corrupt leaf: root=5
>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
>>>> has 18446744073709551492 expect [0, 6875827]
>>>> [192088.462773] BTRFS error (device sdh): block=203510940835840 read
>>>> time tree block corruption detected
>>>> [192088.464711] BTRFS critical (device sdh): corrupt leaf: root=5
>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
>>>> has 18446744073709551492 expect [0, 6875827]
>>>> [192088.468457] BTRFS error (device sdh): block=203510940835840 read
>>>> time tree block corruption detected
>>>>
>>>> btrfs device stats, however, doesn't show any errors.
>>>>
>>>> Is there anything I should do about this, or should I just continue
>>>> using my array as normal?
>>>
>>> This is caused by older kernel underflow inode generation.
>>>
>>> Latest btrfs-progs can fix it, using btrfs check --repair.
>>>
>>> Or you can go safer, by manually locating the inode using its inode
>>> number (1311670), and copy it to some new location using previous
>>> working kernel, then delete the old file, copy the new one back to fix it.
>>>
>>> Thanks,
>>> Qu
>>>
>>>>
>>>> Thank you!
>>>>
>>>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-05-07 5:52 ` Qu Wenruo
@ 2020-05-07 15:52 ` Tyler Richmond
2020-05-08 0:11 ` Qu Wenruo
0 siblings, 1 reply; 56+ messages in thread
From: Tyler Richmond @ 2020-05-07 15:52 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Chris Murphy, Btrfs BTRFS
Thank you for helping. The end result of the scan was:
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
there are no extents for csum range 0-69632
csum exists for 0-69632 but there is no extent record
...
...
there are no extents for csum range 946692096-946827264
csum exists for 946692096-946827264 but there is no extent record
there are no extents for csum range 946831360-947912704
csum exists for 946831360-947912704 but there is no extent record
ERROR: errors found in csum tree
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 44157956026368 bytes used, error(s) found
total csum bytes: 42038602716
total tree bytes: 49688616960
total fs tree bytes: 1256427520
total extent tree bytes: 1709105152
btree space waste bytes: 3172727316
file data blocks allocated: 261625653436416
referenced 47477768499200
What do I need to do to fix all of this?
On Thu, May 7, 2020 at 1:52 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2020/5/7 下午1:43, Tyler Richmond wrote:
> > Well, the repair doesn't look terribly successful.
> >
> > parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > Ignoring transid failure
> > ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > parent level=1
> > child level=4
>
> This means there are more problems, not only the hash name mismatch.
>
> This means the fs is already corrupted, the name hash is just one
> unrelated symptom.
>
> The only good news is, btrfs-progs abort the transaction, thus no
> further damage to the fs.
>
> Please run a plain btrfs-check to show what's the problem first.
>
> Thanks,
> Qu
>
> > parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > Ignoring transid failure
> > ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > parent level=1
> > child level=4
> > parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > Ignoring transid failure
> > ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > parent level=1
> > child level=4
> > parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > Ignoring transid failure
> > ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > parent level=1
> > child level=4
> > parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > Ignoring transid failure
> > ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > parent level=1
> > child level=4
> > parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > Ignoring transid failure
> > ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > parent level=1
> > child level=4
> > parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > Ignoring transid failure
> > ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > parent level=1
> > child level=4
> > parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > Ignoring transid failure
> > ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > parent level=1
> > child level=4
> > parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > Ignoring transid failure
> > ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > parent level=1
> > child level=4
> > parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > Ignoring transid failure
> > ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > parent level=1
> > child level=4
> > parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > Ignoring transid failure
> > ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > parent level=1
> > child level=4
> > ERROR: failed to zero log tree: -17
> > ERROR: attempt to start transaction over already running one
> > WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
> > extent buffer leak: start 225049066086400 len 4096
> > extent buffer leak: start 225049066086400 len 4096
> > WARNING: dirty eb leak (aborted trans): start 225049066086400 len 4096
> > extent buffer leak: start 225049066094592 len 4096
> > extent buffer leak: start 225049066094592 len 4096
> > WARNING: dirty eb leak (aborted trans): start 225049066094592 len 4096
> > extent buffer leak: start 225049066102784 len 4096
> > extent buffer leak: start 225049066102784 len 4096
> > WARNING: dirty eb leak (aborted trans): start 225049066102784 len 4096
> > extent buffer leak: start 225049066131456 len 4096
> > extent buffer leak: start 225049066131456 len 4096
> > WARNING: dirty eb leak (aborted trans): start 225049066131456 len 4096
> >
> > What is going on?
> >
> > On Wed, May 6, 2020 at 9:30 PM Tyler Richmond <t.d.richmond@gmail.com> wrote:
> >>
> >> Chris, I had used the correct mountpoint in the command. I just edited
> >> it in the email to be /mountpoint for consistency.
> >>
> >> Qu, I'll try the repair. Fingers crossed!
> >>
> >> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>
> >>>
> >>>
> >>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
> >>>> Hello,
> >>>>
> >>>> I looked up this error and it basically says ask a developer to
> >>>> determine if it's a false error or not. I just started getting some
> >>>> slow response times, and looked at the dmesg log to find a ton of
> >>>> these errors.
> >>>>
> >>>> [192088.446299] BTRFS critical (device sdh): corrupt leaf: root=5
> >>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> >>>> has 18446744073709551492 expect [0, 6875827]
> >>>> [192088.449823] BTRFS error (device sdh): block=203510940835840 read
> >>>> time tree block corruption detected
> >>>> [192088.459238] BTRFS critical (device sdh): corrupt leaf: root=5
> >>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> >>>> has 18446744073709551492 expect [0, 6875827]
> >>>> [192088.462773] BTRFS error (device sdh): block=203510940835840 read
> >>>> time tree block corruption detected
> >>>> [192088.464711] BTRFS critical (device sdh): corrupt leaf: root=5
> >>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> >>>> has 18446744073709551492 expect [0, 6875827]
> >>>> [192088.468457] BTRFS error (device sdh): block=203510940835840 read
> >>>> time tree block corruption detected
> >>>>
> >>>> btrfs device stats, however, doesn't show any errors.
> >>>>
> >>>> Is there anything I should do about this, or should I just continue
> >>>> using my array as normal?
> >>>
> >>> This is caused by older kernel underflow inode generation.
> >>>
> >>> Latest btrfs-progs can fix it, using btrfs check --repair.
> >>>
> >>> Or you can go safer, by manually locating the inode using its inode
> >>> number (1311670), and copy it to some new location using previous
> >>> working kernel, then delete the old file, copy the new one back to fix it.
> >>>
> >>> Thanks,
> >>> Qu
> >>>
> >>>>
> >>>> Thank you!
> >>>>
> >>>
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-05-07 15:52 ` Tyler Richmond
@ 2020-05-08 0:11 ` Qu Wenruo
2020-05-08 4:23 ` Tyler Richmond
0 siblings, 1 reply; 56+ messages in thread
From: Qu Wenruo @ 2020-05-08 0:11 UTC (permalink / raw)
To: Tyler Richmond; +Cc: Chris Murphy, Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 8357 bytes --]
On 2020/5/7 下午11:52, Tyler Richmond wrote:
> Thank you for helping. The end result of the scan was:
>
>
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> [4/7] checking fs roots
Good news is, your fs is still mostly fine.
> [5/7] checking only csums items (without verifying data)
> there are no extents for csum range 0-69632
> csum exists for 0-69632 but there is no extent record
> ...
> ...
> there are no extents for csum range 946692096-946827264
> csum exists for 946692096-946827264 but there is no extent record
> there are no extents for csum range 946831360-947912704
> csum exists for 946831360-947912704 but there is no extent record
> ERROR: errors found in csum tree
Only extent tree is corrupted.
Normally btrfs check --init-csum-tree should be able to handle it.
But still, please be sure you're using the latest btrfs-progs to fix it.
Thanks,
Qu
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 44157956026368 bytes used, error(s) found
> total csum bytes: 42038602716
> total tree bytes: 49688616960
> total fs tree bytes: 1256427520
> total extent tree bytes: 1709105152
> btree space waste bytes: 3172727316
> file data blocks allocated: 261625653436416
> referenced 47477768499200
>
> What do I need to do to fix all of this?
>
> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
>>> Well, the repair doesn't look terribly successful.
>>>
>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>> Ignoring transid failure
>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>> parent level=1
>>> child level=4
>>
>> This means there are more problems, not only the hash name mismatch.
>>
>> This means the fs is already corrupted, the name hash is just one
>> unrelated symptom.
>>
>> The only good news is, btrfs-progs abort the transaction, thus no
>> further damage to the fs.
>>
>> Please run a plain btrfs-check to show what's the problem first.
>>
>> Thanks,
>> Qu
>>
>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>> Ignoring transid failure
>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>> parent level=1
>>> child level=4
>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>> Ignoring transid failure
>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>> parent level=1
>>> child level=4
>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>> Ignoring transid failure
>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>> parent level=1
>>> child level=4
>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>> Ignoring transid failure
>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>> parent level=1
>>> child level=4
>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>> Ignoring transid failure
>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>> parent level=1
>>> child level=4
>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>> Ignoring transid failure
>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>> parent level=1
>>> child level=4
>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>> Ignoring transid failure
>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>> parent level=1
>>> child level=4
>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>> Ignoring transid failure
>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>> parent level=1
>>> child level=4
>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>> Ignoring transid failure
>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>> parent level=1
>>> child level=4
>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>> Ignoring transid failure
>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>> parent level=1
>>> child level=4
>>> ERROR: failed to zero log tree: -17
>>> ERROR: attempt to start transaction over already running one
>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
>>> extent buffer leak: start 225049066086400 len 4096
>>> extent buffer leak: start 225049066086400 len 4096
>>> WARNING: dirty eb leak (aborted trans): start 225049066086400 len 4096
>>> extent buffer leak: start 225049066094592 len 4096
>>> extent buffer leak: start 225049066094592 len 4096
>>> WARNING: dirty eb leak (aborted trans): start 225049066094592 len 4096
>>> extent buffer leak: start 225049066102784 len 4096
>>> extent buffer leak: start 225049066102784 len 4096
>>> WARNING: dirty eb leak (aborted trans): start 225049066102784 len 4096
>>> extent buffer leak: start 225049066131456 len 4096
>>> extent buffer leak: start 225049066131456 len 4096
>>> WARNING: dirty eb leak (aborted trans): start 225049066131456 len 4096
>>>
>>> What is going on?
>>>
>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>>>>
>>>> Chris, I had used the correct mountpoint in the command. I just edited
>>>> it in the email to be /mountpoint for consistency.
>>>>
>>>> Qu, I'll try the repair. Fingers crossed!
>>>>
>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I looked up this error and it basically says ask a developer to
>>>>>> determine if it's a false error or not. I just started getting some
>>>>>> slow response times, and looked at the dmesg log to find a ton of
>>>>>> these errors.
>>>>>>
>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt leaf: root=5
>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>> [192088.449823] BTRFS error (device sdh): block=203510940835840 read
>>>>>> time tree block corruption detected
>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt leaf: root=5
>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>> [192088.462773] BTRFS error (device sdh): block=203510940835840 read
>>>>>> time tree block corruption detected
>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt leaf: root=5
>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>> [192088.468457] BTRFS error (device sdh): block=203510940835840 read
>>>>>> time tree block corruption detected
>>>>>>
>>>>>> btrfs device stats, however, doesn't show any errors.
>>>>>>
>>>>>> Is there anything I should do about this, or should I just continue
>>>>>> using my array as normal?
>>>>>
>>>>> This is caused by older kernel underflow inode generation.
>>>>>
>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
>>>>>
>>>>> Or you can go safer, by manually locating the inode using its inode
>>>>> number (1311670), and copy it to some new location using previous
>>>>> working kernel, then delete the old file, copy the new one back to fix it.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>>>
>>>>>> Thank you!
>>>>>>
>>>>>
>>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-05-08 0:11 ` Qu Wenruo
@ 2020-05-08 4:23 ` Tyler Richmond
2020-05-08 5:07 ` Qu Wenruo
0 siblings, 1 reply; 56+ messages in thread
From: Tyler Richmond @ 2020-05-08 4:23 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Btrfs BTRFS
Something went wrong:
Reinitialize checksum tree
Unable to find block group for 0
Unable to find block group for 0
Unable to find block group for 0
ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
btrfs(+0x6dd94)[0x55a933af7d94]
btrfs(+0x71b94)[0x55a933afbb94]
btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
btrfs(+0x360b2)[0x55a933ac00b2]
btrfs(+0x46a3e)[0x55a933ad0a3e]
btrfs(main+0x98)[0x55a933a9fe88]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
btrfs(_start+0x2e)[0x55a933a9fa0e]
Aborted
I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
available. I'll let that try overnight?
On Thu, May 7, 2020 at 8:11 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2020/5/7 下午11:52, Tyler Richmond wrote:
> > Thank you for helping. The end result of the scan was:
> >
> >
> > [1/7] checking root items
> > [2/7] checking extents
> > [3/7] checking free space cache
> > [4/7] checking fs roots
>
> Good news is, your fs is still mostly fine.
>
> > [5/7] checking only csums items (without verifying data)
> > there are no extents for csum range 0-69632
> > csum exists for 0-69632 but there is no extent record
> > ...
> > ...
> > there are no extents for csum range 946692096-946827264
> > csum exists for 946692096-946827264 but there is no extent record
> > there are no extents for csum range 946831360-947912704
> > csum exists for 946831360-947912704 but there is no extent record
> > ERROR: errors found in csum tree
>
> Only extent tree is corrupted.
>
> Normally btrfs check --init-csum-tree should be able to handle it.
>
> But still, please be sure you're using the latest btrfs-progs to fix it.
>
> Thanks,
> Qu
>
> > [6/7] checking root refs
> > [7/7] checking quota groups skipped (not enabled on this FS)
> > found 44157956026368 bytes used, error(s) found
> > total csum bytes: 42038602716
> > total tree bytes: 49688616960
> > total fs tree bytes: 1256427520
> > total extent tree bytes: 1709105152
> > btree space waste bytes: 3172727316
> > file data blocks allocated: 261625653436416
> > referenced 47477768499200
> >
> > What do I need to do to fix all of this?
> >
> > On Thu, May 7, 2020 at 1:52 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>
> >>
> >>
> >> On 2020/5/7 下午1:43, Tyler Richmond wrote:
> >>> Well, the repair doesn't look terribly successful.
> >>>
> >>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>> Ignoring transid failure
> >>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>> parent level=1
> >>> child level=4
> >>
> >> This means there are more problems, not only the hash name mismatch.
> >>
> >> This means the fs is already corrupted, the name hash is just one
> >> unrelated symptom.
> >>
> >> The only good news is, btrfs-progs abort the transaction, thus no
> >> further damage to the fs.
> >>
> >> Please run a plain btrfs-check to show what's the problem first.
> >>
> >> Thanks,
> >> Qu
> >>
> >>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>> Ignoring transid failure
> >>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>> parent level=1
> >>> child level=4
> >>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>> Ignoring transid failure
> >>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>> parent level=1
> >>> child level=4
> >>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>> Ignoring transid failure
> >>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>> parent level=1
> >>> child level=4
> >>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>> Ignoring transid failure
> >>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>> parent level=1
> >>> child level=4
> >>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>> Ignoring transid failure
> >>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>> parent level=1
> >>> child level=4
> >>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>> Ignoring transid failure
> >>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>> parent level=1
> >>> child level=4
> >>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>> Ignoring transid failure
> >>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>> parent level=1
> >>> child level=4
> >>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>> Ignoring transid failure
> >>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>> parent level=1
> >>> child level=4
> >>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>> Ignoring transid failure
> >>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>> parent level=1
> >>> child level=4
> >>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>> Ignoring transid failure
> >>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>> parent level=1
> >>> child level=4
> >>> ERROR: failed to zero log tree: -17
> >>> ERROR: attempt to start transaction over already running one
> >>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
> >>> extent buffer leak: start 225049066086400 len 4096
> >>> extent buffer leak: start 225049066086400 len 4096
> >>> WARNING: dirty eb leak (aborted trans): start 225049066086400 len 4096
> >>> extent buffer leak: start 225049066094592 len 4096
> >>> extent buffer leak: start 225049066094592 len 4096
> >>> WARNING: dirty eb leak (aborted trans): start 225049066094592 len 4096
> >>> extent buffer leak: start 225049066102784 len 4096
> >>> extent buffer leak: start 225049066102784 len 4096
> >>> WARNING: dirty eb leak (aborted trans): start 225049066102784 len 4096
> >>> extent buffer leak: start 225049066131456 len 4096
> >>> extent buffer leak: start 225049066131456 len 4096
> >>> WARNING: dirty eb leak (aborted trans): start 225049066131456 len 4096
> >>>
> >>> What is going on?
> >>>
> >>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond <t.d.richmond@gmail.com> wrote:
> >>>>
> >>>> Chris, I had used the correct mountpoint in the command. I just edited
> >>>> it in the email to be /mountpoint for consistency.
> >>>>
> >>>> Qu, I'll try the repair. Fingers crossed!
> >>>>
> >>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
> >>>>>> Hello,
> >>>>>>
> >>>>>> I looked up this error and it basically says ask a developer to
> >>>>>> determine if it's a false error or not. I just started getting some
> >>>>>> slow response times, and looked at the dmesg log to find a ton of
> >>>>>> these errors.
> >>>>>>
> >>>>>> [192088.446299] BTRFS critical (device sdh): corrupt leaf: root=5
> >>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> >>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>> [192088.449823] BTRFS error (device sdh): block=203510940835840 read
> >>>>>> time tree block corruption detected
> >>>>>> [192088.459238] BTRFS critical (device sdh): corrupt leaf: root=5
> >>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> >>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>> [192088.462773] BTRFS error (device sdh): block=203510940835840 read
> >>>>>> time tree block corruption detected
> >>>>>> [192088.464711] BTRFS critical (device sdh): corrupt leaf: root=5
> >>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> >>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>> [192088.468457] BTRFS error (device sdh): block=203510940835840 read
> >>>>>> time tree block corruption detected
> >>>>>>
> >>>>>> btrfs device stats, however, doesn't show any errors.
> >>>>>>
> >>>>>> Is there anything I should do about this, or should I just continue
> >>>>>> using my array as normal?
> >>>>>
> >>>>> This is caused by older kernel underflow inode generation.
> >>>>>
> >>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
> >>>>>
> >>>>> Or you can go safer, by manually locating the inode using its inode
> >>>>> number (1311670), and copy it to some new location using previous
> >>>>> working kernel, then delete the old file, copy the new one back to fix it.
> >>>>>
> >>>>> Thanks,
> >>>>> Qu
> >>>>>
> >>>>>>
> >>>>>> Thank you!
> >>>>>>
> >>>>>
> >>
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-05-08 4:23 ` Tyler Richmond
@ 2020-05-08 5:07 ` Qu Wenruo
2020-05-08 5:12 ` Tyler Richmond
0 siblings, 1 reply; 56+ messages in thread
From: Qu Wenruo @ 2020-05-08 5:07 UTC (permalink / raw)
To: Tyler Richmond; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 9918 bytes --]
On 2020/5/8 下午12:23, Tyler Richmond wrote:
> Something went wrong:
>
> Reinitialize checksum tree
> Unable to find block group for 0
> Unable to find block group for 0
> Unable to find block group for 0
> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
> btrfs(+0x6dd94)[0x55a933af7d94]
> btrfs(+0x71b94)[0x55a933afbb94]
> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
> btrfs(+0x360b2)[0x55a933ac00b2]
> btrfs(+0x46a3e)[0x55a933ad0a3e]
> btrfs(main+0x98)[0x55a933a9fe88]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
> btrfs(_start+0x2e)[0x55a933a9fa0e]
> Aborted
This means no space for extra metadata...
Anyway the csum tree problem shouldn't be a big thing, you could leave
it and call it a day.
BTW, as long as btrfs check reports no extra problem for the inode
generation, it should be pretty safe to use the fs.
Thanks,
Qu
>
> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
> available. I'll let that try overnight?
>
> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
>>> Thank you for helping. The end result of the scan was:
>>>
>>>
>>> [1/7] checking root items
>>> [2/7] checking extents
>>> [3/7] checking free space cache
>>> [4/7] checking fs roots
>>
>> Good news is, your fs is still mostly fine.
>>
>>> [5/7] checking only csums items (without verifying data)
>>> there are no extents for csum range 0-69632
>>> csum exists for 0-69632 but there is no extent record
>>> ...
>>> ...
>>> there are no extents for csum range 946692096-946827264
>>> csum exists for 946692096-946827264 but there is no extent record
>>> there are no extents for csum range 946831360-947912704
>>> csum exists for 946831360-947912704 but there is no extent record
>>> ERROR: errors found in csum tree
>>
>> Only extent tree is corrupted.
>>
>> Normally btrfs check --init-csum-tree should be able to handle it.
>>
>> But still, please be sure you're using the latest btrfs-progs to fix it.
>>
>> Thanks,
>> Qu
>>
>>> [6/7] checking root refs
>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>> found 44157956026368 bytes used, error(s) found
>>> total csum bytes: 42038602716
>>> total tree bytes: 49688616960
>>> total fs tree bytes: 1256427520
>>> total extent tree bytes: 1709105152
>>> btree space waste bytes: 3172727316
>>> file data blocks allocated: 261625653436416
>>> referenced 47477768499200
>>>
>>> What do I need to do to fix all of this?
>>>
>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>
>>>>
>>>>
>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
>>>>> Well, the repair doesn't look terribly successful.
>>>>>
>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>> Ignoring transid failure
>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>> parent level=1
>>>>> child level=4
>>>>
>>>> This means there are more problems, not only the hash name mismatch.
>>>>
>>>> This means the fs is already corrupted, the name hash is just one
>>>> unrelated symptom.
>>>>
>>>> The only good news is, btrfs-progs abort the transaction, thus no
>>>> further damage to the fs.
>>>>
>>>> Please run a plain btrfs-check to show what's the problem first.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>> Ignoring transid failure
>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>> parent level=1
>>>>> child level=4
>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>> Ignoring transid failure
>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>> parent level=1
>>>>> child level=4
>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>> Ignoring transid failure
>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>> parent level=1
>>>>> child level=4
>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>> Ignoring transid failure
>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>> parent level=1
>>>>> child level=4
>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>> Ignoring transid failure
>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>> parent level=1
>>>>> child level=4
>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>> Ignoring transid failure
>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>> parent level=1
>>>>> child level=4
>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>> Ignoring transid failure
>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>> parent level=1
>>>>> child level=4
>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>> Ignoring transid failure
>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>> parent level=1
>>>>> child level=4
>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>> Ignoring transid failure
>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>> parent level=1
>>>>> child level=4
>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>> Ignoring transid failure
>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>> parent level=1
>>>>> child level=4
>>>>> ERROR: failed to zero log tree: -17
>>>>> ERROR: attempt to start transaction over already running one
>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
>>>>> extent buffer leak: start 225049066086400 len 4096
>>>>> extent buffer leak: start 225049066086400 len 4096
>>>>> WARNING: dirty eb leak (aborted trans): start 225049066086400 len 4096
>>>>> extent buffer leak: start 225049066094592 len 4096
>>>>> extent buffer leak: start 225049066094592 len 4096
>>>>> WARNING: dirty eb leak (aborted trans): start 225049066094592 len 4096
>>>>> extent buffer leak: start 225049066102784 len 4096
>>>>> extent buffer leak: start 225049066102784 len 4096
>>>>> WARNING: dirty eb leak (aborted trans): start 225049066102784 len 4096
>>>>> extent buffer leak: start 225049066131456 len 4096
>>>>> extent buffer leak: start 225049066131456 len 4096
>>>>> WARNING: dirty eb leak (aborted trans): start 225049066131456 len 4096
>>>>>
>>>>> What is going on?
>>>>>
>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>>>>>>
>>>>>> Chris, I had used the correct mountpoint in the command. I just edited
>>>>>> it in the email to be /mountpoint for consistency.
>>>>>>
>>>>>> Qu, I'll try the repair. Fingers crossed!
>>>>>>
>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I looked up this error and it basically says ask a developer to
>>>>>>>> determine if it's a false error or not. I just started getting some
>>>>>>>> slow response times, and looked at the dmesg log to find a ton of
>>>>>>>> these errors.
>>>>>>>>
>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt leaf: root=5
>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>> [192088.449823] BTRFS error (device sdh): block=203510940835840 read
>>>>>>>> time tree block corruption detected
>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt leaf: root=5
>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>> [192088.462773] BTRFS error (device sdh): block=203510940835840 read
>>>>>>>> time tree block corruption detected
>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt leaf: root=5
>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>> [192088.468457] BTRFS error (device sdh): block=203510940835840 read
>>>>>>>> time tree block corruption detected
>>>>>>>>
>>>>>>>> btrfs device stats, however, doesn't show any errors.
>>>>>>>>
>>>>>>>> Is there anything I should do about this, or should I just continue
>>>>>>>> using my array as normal?
>>>>>>>
>>>>>>> This is caused by older kernel underflow inode generation.
>>>>>>>
>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
>>>>>>>
>>>>>>> Or you can go safer, by manually locating the inode using its inode
>>>>>>> number (1311670), and copy it to some new location using previous
>>>>>>> working kernel, then delete the old file, copy the new one back to fix it.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Qu
>>>>>>>
>>>>>>>>
>>>>>>>> Thank you!
>>>>>>>>
>>>>>>>
>>>>
>>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-05-08 5:07 ` Qu Wenruo
@ 2020-05-08 5:12 ` Tyler Richmond
2020-05-08 5:47 ` Qu Wenruo
0 siblings, 1 reply; 56+ messages in thread
From: Tyler Richmond @ 2020-05-08 5:12 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Btrfs BTRFS
If this is saying there's no extra space for metadata, is that why
adding more files often makes the system hang for 30-90s? Is there
anything I should do about that?
Thank you so much for all of your help. I love how flexible BTRFS is
but when things go wrong it's very hard for me to troubleshoot.
On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2020/5/8 下午12:23, Tyler Richmond wrote:
> > Something went wrong:
> >
> > Reinitialize checksum tree
> > Unable to find block group for 0
> > Unable to find block group for 0
> > Unable to find block group for 0
> > ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
> > btrfs(+0x6dd94)[0x55a933af7d94]
> > btrfs(+0x71b94)[0x55a933afbb94]
> > btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
> > btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
> > btrfs(+0x360b2)[0x55a933ac00b2]
> > btrfs(+0x46a3e)[0x55a933ad0a3e]
> > btrfs(main+0x98)[0x55a933a9fe88]
> > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
> > btrfs(_start+0x2e)[0x55a933a9fa0e]
> > Aborted
>
> This means no space for extra metadata...
>
> Anyway the csum tree problem shouldn't be a big thing, you could leave
> it and call it a day.
>
> BTW, as long as btrfs check reports no extra problem for the inode
> generation, it should be pretty safe to use the fs.
>
> Thanks,
> Qu
> >
> > I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
> > available. I'll let that try overnight?
> >
> > On Thu, May 7, 2020 at 8:11 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>
> >>
> >>
> >> On 2020/5/7 下午11:52, Tyler Richmond wrote:
> >>> Thank you for helping. The end result of the scan was:
> >>>
> >>>
> >>> [1/7] checking root items
> >>> [2/7] checking extents
> >>> [3/7] checking free space cache
> >>> [4/7] checking fs roots
> >>
> >> Good news is, your fs is still mostly fine.
> >>
> >>> [5/7] checking only csums items (without verifying data)
> >>> there are no extents for csum range 0-69632
> >>> csum exists for 0-69632 but there is no extent record
> >>> ...
> >>> ...
> >>> there are no extents for csum range 946692096-946827264
> >>> csum exists for 946692096-946827264 but there is no extent record
> >>> there are no extents for csum range 946831360-947912704
> >>> csum exists for 946831360-947912704 but there is no extent record
> >>> ERROR: errors found in csum tree
> >>
> >> Only extent tree is corrupted.
> >>
> >> Normally btrfs check --init-csum-tree should be able to handle it.
> >>
> >> But still, please be sure you're using the latest btrfs-progs to fix it.
> >>
> >> Thanks,
> >> Qu
> >>
> >>> [6/7] checking root refs
> >>> [7/7] checking quota groups skipped (not enabled on this FS)
> >>> found 44157956026368 bytes used, error(s) found
> >>> total csum bytes: 42038602716
> >>> total tree bytes: 49688616960
> >>> total fs tree bytes: 1256427520
> >>> total extent tree bytes: 1709105152
> >>> btree space waste bytes: 3172727316
> >>> file data blocks allocated: 261625653436416
> >>> referenced 47477768499200
> >>>
> >>> What do I need to do to fix all of this?
> >>>
> >>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
> >>>>> Well, the repair doesn't look terribly successful.
> >>>>>
> >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>> Ignoring transid failure
> >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>> parent level=1
> >>>>> child level=4
> >>>>
> >>>> This means there are more problems, not only the hash name mismatch.
> >>>>
> >>>> This means the fs is already corrupted, the name hash is just one
> >>>> unrelated symptom.
> >>>>
> >>>> The only good news is, btrfs-progs abort the transaction, thus no
> >>>> further damage to the fs.
> >>>>
> >>>> Please run a plain btrfs-check to show what's the problem first.
> >>>>
> >>>> Thanks,
> >>>> Qu
> >>>>
> >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>> Ignoring transid failure
> >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>> parent level=1
> >>>>> child level=4
> >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>> Ignoring transid failure
> >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>> parent level=1
> >>>>> child level=4
> >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>> Ignoring transid failure
> >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>> parent level=1
> >>>>> child level=4
> >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>> Ignoring transid failure
> >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>> parent level=1
> >>>>> child level=4
> >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>> Ignoring transid failure
> >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>> parent level=1
> >>>>> child level=4
> >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>> Ignoring transid failure
> >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>> parent level=1
> >>>>> child level=4
> >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>> Ignoring transid failure
> >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>> parent level=1
> >>>>> child level=4
> >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>> Ignoring transid failure
> >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>> parent level=1
> >>>>> child level=4
> >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>> Ignoring transid failure
> >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>> parent level=1
> >>>>> child level=4
> >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>> Ignoring transid failure
> >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>> parent level=1
> >>>>> child level=4
> >>>>> ERROR: failed to zero log tree: -17
> >>>>> ERROR: attempt to start transaction over already running one
> >>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
> >>>>> extent buffer leak: start 225049066086400 len 4096
> >>>>> extent buffer leak: start 225049066086400 len 4096
> >>>>> WARNING: dirty eb leak (aborted trans): start 225049066086400 len 4096
> >>>>> extent buffer leak: start 225049066094592 len 4096
> >>>>> extent buffer leak: start 225049066094592 len 4096
> >>>>> WARNING: dirty eb leak (aborted trans): start 225049066094592 len 4096
> >>>>> extent buffer leak: start 225049066102784 len 4096
> >>>>> extent buffer leak: start 225049066102784 len 4096
> >>>>> WARNING: dirty eb leak (aborted trans): start 225049066102784 len 4096
> >>>>> extent buffer leak: start 225049066131456 len 4096
> >>>>> extent buffer leak: start 225049066131456 len 4096
> >>>>> WARNING: dirty eb leak (aborted trans): start 225049066131456 len 4096
> >>>>>
> >>>>> What is going on?
> >>>>>
> >>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond <t.d.richmond@gmail.com> wrote:
> >>>>>>
> >>>>>> Chris, I had used the correct mountpoint in the command. I just edited
> >>>>>> it in the email to be /mountpoint for consistency.
> >>>>>>
> >>>>>> Qu, I'll try the repair. Fingers crossed!
> >>>>>>
> >>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
> >>>>>>>> Hello,
> >>>>>>>>
> >>>>>>>> I looked up this error and it basically says ask a developer to
> >>>>>>>> determine if it's a false error or not. I just started getting some
> >>>>>>>> slow response times, and looked at the dmesg log to find a ton of
> >>>>>>>> these errors.
> >>>>>>>>
> >>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt leaf: root=5
> >>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> >>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>>>> [192088.449823] BTRFS error (device sdh): block=203510940835840 read
> >>>>>>>> time tree block corruption detected
> >>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt leaf: root=5
> >>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> >>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>>>> [192088.462773] BTRFS error (device sdh): block=203510940835840 read
> >>>>>>>> time tree block corruption detected
> >>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt leaf: root=5
> >>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> >>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>>>> [192088.468457] BTRFS error (device sdh): block=203510940835840 read
> >>>>>>>> time tree block corruption detected
> >>>>>>>>
> >>>>>>>> btrfs device stats, however, doesn't show any errors.
> >>>>>>>>
> >>>>>>>> Is there anything I should do about this, or should I just continue
> >>>>>>>> using my array as normal?
> >>>>>>>
> >>>>>>> This is caused by older kernel underflow inode generation.
> >>>>>>>
> >>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
> >>>>>>>
> >>>>>>> Or you can go safer, by manually locating the inode using its inode
> >>>>>>> number (1311670), and copy it to some new location using previous
> >>>>>>> working kernel, then delete the old file, copy the new one back to fix it.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Qu
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Thank you!
> >>>>>>>>
> >>>>>>>
> >>>>
> >>
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-05-08 5:12 ` Tyler Richmond
@ 2020-05-08 5:47 ` Qu Wenruo
2020-05-08 13:52 ` Tyler Richmond
0 siblings, 1 reply; 56+ messages in thread
From: Qu Wenruo @ 2020-05-08 5:47 UTC (permalink / raw)
To: Tyler Richmond; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 11146 bytes --]
On 2020/5/8 下午1:12, Tyler Richmond wrote:
> If this is saying there's no extra space for metadata, is that why
> adding more files often makes the system hang for 30-90s? Is there
> anything I should do about that?
I'm not sure about the hang though.
It would be nice to give more info to diagnosis.
The output of 'btrfs fi usage' is useful for space usage problem.
But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
space in vanilla df command) space for btrfs.
Thanks,
Qu
>
> Thank you so much for all of your help. I love how flexible BTRFS is
> but when things go wrong it's very hard for me to troubleshoot.
>
> On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2020/5/8 下午12:23, Tyler Richmond wrote:
>>> Something went wrong:
>>>
>>> Reinitialize checksum tree
>>> Unable to find block group for 0
>>> Unable to find block group for 0
>>> Unable to find block group for 0
>>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
>>> btrfs(+0x6dd94)[0x55a933af7d94]
>>> btrfs(+0x71b94)[0x55a933afbb94]
>>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
>>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
>>> btrfs(+0x360b2)[0x55a933ac00b2]
>>> btrfs(+0x46a3e)[0x55a933ad0a3e]
>>> btrfs(main+0x98)[0x55a933a9fe88]
>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
>>> btrfs(_start+0x2e)[0x55a933a9fa0e]
>>> Aborted
>>
>> This means no space for extra metadata...
>>
>> Anyway the csum tree problem shouldn't be a big thing, you could leave
>> it and call it a day.
>>
>> BTW, as long as btrfs check reports no extra problem for the inode
>> generation, it should be pretty safe to use the fs.
>>
>> Thanks,
>> Qu
>>>
>>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
>>> available. I'll let that try overnight?
>>>
>>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>
>>>>
>>>>
>>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
>>>>> Thank you for helping. The end result of the scan was:
>>>>>
>>>>>
>>>>> [1/7] checking root items
>>>>> [2/7] checking extents
>>>>> [3/7] checking free space cache
>>>>> [4/7] checking fs roots
>>>>
>>>> Good news is, your fs is still mostly fine.
>>>>
>>>>> [5/7] checking only csums items (without verifying data)
>>>>> there are no extents for csum range 0-69632
>>>>> csum exists for 0-69632 but there is no extent record
>>>>> ...
>>>>> ...
>>>>> there are no extents for csum range 946692096-946827264
>>>>> csum exists for 946692096-946827264 but there is no extent record
>>>>> there are no extents for csum range 946831360-947912704
>>>>> csum exists for 946831360-947912704 but there is no extent record
>>>>> ERROR: errors found in csum tree
>>>>
>>>> Only extent tree is corrupted.
>>>>
>>>> Normally btrfs check --init-csum-tree should be able to handle it.
>>>>
>>>> But still, please be sure you're using the latest btrfs-progs to fix it.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>> [6/7] checking root refs
>>>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>>>> found 44157956026368 bytes used, error(s) found
>>>>> total csum bytes: 42038602716
>>>>> total tree bytes: 49688616960
>>>>> total fs tree bytes: 1256427520
>>>>> total extent tree bytes: 1709105152
>>>>> btree space waste bytes: 3172727316
>>>>> file data blocks allocated: 261625653436416
>>>>> referenced 47477768499200
>>>>>
>>>>> What do I need to do to fix all of this?
>>>>>
>>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
>>>>>>> Well, the repair doesn't look terribly successful.
>>>>>>>
>>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>>>> Ignoring transid failure
>>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>>>> parent level=1
>>>>>>> child level=4
>>>>>>
>>>>>> This means there are more problems, not only the hash name mismatch.
>>>>>>
>>>>>> This means the fs is already corrupted, the name hash is just one
>>>>>> unrelated symptom.
>>>>>>
>>>>>> The only good news is, btrfs-progs abort the transaction, thus no
>>>>>> further damage to the fs.
>>>>>>
>>>>>> Please run a plain btrfs-check to show what's the problem first.
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>
>>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>>>> Ignoring transid failure
>>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>>>> parent level=1
>>>>>>> child level=4
>>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>>>> Ignoring transid failure
>>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>>>> parent level=1
>>>>>>> child level=4
>>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>>>> Ignoring transid failure
>>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>>>> parent level=1
>>>>>>> child level=4
>>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>>>> Ignoring transid failure
>>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>>>> parent level=1
>>>>>>> child level=4
>>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>>>> Ignoring transid failure
>>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>>>> parent level=1
>>>>>>> child level=4
>>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>>>> Ignoring transid failure
>>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>>>> parent level=1
>>>>>>> child level=4
>>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>>>> Ignoring transid failure
>>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>>>> parent level=1
>>>>>>> child level=4
>>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>>>> Ignoring transid failure
>>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>>>> parent level=1
>>>>>>> child level=4
>>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>>>> Ignoring transid failure
>>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>>>> parent level=1
>>>>>>> child level=4
>>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
>>>>>>> Ignoring transid failure
>>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
>>>>>>> parent level=1
>>>>>>> child level=4
>>>>>>> ERROR: failed to zero log tree: -17
>>>>>>> ERROR: attempt to start transaction over already running one
>>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
>>>>>>> extent buffer leak: start 225049066086400 len 4096
>>>>>>> extent buffer leak: start 225049066086400 len 4096
>>>>>>> WARNING: dirty eb leak (aborted trans): start 225049066086400 len 4096
>>>>>>> extent buffer leak: start 225049066094592 len 4096
>>>>>>> extent buffer leak: start 225049066094592 len 4096
>>>>>>> WARNING: dirty eb leak (aborted trans): start 225049066094592 len 4096
>>>>>>> extent buffer leak: start 225049066102784 len 4096
>>>>>>> extent buffer leak: start 225049066102784 len 4096
>>>>>>> WARNING: dirty eb leak (aborted trans): start 225049066102784 len 4096
>>>>>>> extent buffer leak: start 225049066131456 len 4096
>>>>>>> extent buffer leak: start 225049066131456 len 4096
>>>>>>> WARNING: dirty eb leak (aborted trans): start 225049066131456 len 4096
>>>>>>>
>>>>>>> What is going on?
>>>>>>>
>>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Chris, I had used the correct mountpoint in the command. I just edited
>>>>>>>> it in the email to be /mountpoint for consistency.
>>>>>>>>
>>>>>>>> Qu, I'll try the repair. Fingers crossed!
>>>>>>>>
>>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I looked up this error and it basically says ask a developer to
>>>>>>>>>> determine if it's a false error or not. I just started getting some
>>>>>>>>>> slow response times, and looked at the dmesg log to find a ton of
>>>>>>>>>> these errors.
>>>>>>>>>>
>>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt leaf: root=5
>>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
>>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>> [192088.449823] BTRFS error (device sdh): block=203510940835840 read
>>>>>>>>>> time tree block corruption detected
>>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt leaf: root=5
>>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
>>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>> [192088.462773] BTRFS error (device sdh): block=203510940835840 read
>>>>>>>>>> time tree block corruption detected
>>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt leaf: root=5
>>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
>>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>> [192088.468457] BTRFS error (device sdh): block=203510940835840 read
>>>>>>>>>> time tree block corruption detected
>>>>>>>>>>
>>>>>>>>>> btrfs device stats, however, doesn't show any errors.
>>>>>>>>>>
>>>>>>>>>> Is there anything I should do about this, or should I just continue
>>>>>>>>>> using my array as normal?
>>>>>>>>>
>>>>>>>>> This is caused by older kernel underflow inode generation.
>>>>>>>>>
>>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
>>>>>>>>>
>>>>>>>>> Or you can go safer, by manually locating the inode using its inode
>>>>>>>>> number (1311670), and copy it to some new location using previous
>>>>>>>>> working kernel, then delete the old file, copy the new one back to fix it.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Qu
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thank you!
>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-05-08 5:47 ` Qu Wenruo
@ 2020-05-08 13:52 ` Tyler Richmond
2020-08-18 3:36 ` Tyler Richmond
[not found] ` <CAJheHN3qwDAGY=z14zfO4LBrxNJZZ_rvAMsWLwe-k+4+t3zLog@mail.gmail.com>
0 siblings, 2 replies; 56+ messages in thread
From: Tyler Richmond @ 2020-05-08 13:52 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Btrfs BTRFS
5.6.1 also failed the same way. Here's the usage output. This is the
part where you see I've been using RAID5 haha
WARNING: RAID56 detected, not implemented
Overall:
Device size: 60.03TiB
Device allocated: 98.06GiB
Device unallocated: 59.93TiB
Device missing: 0.00B
Used: 92.56GiB
Free (estimated): 0.00B (min: 8.00EiB)
Data ratio: 0.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no
Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
/dev/sdh 8.07TiB
/dev/sdf 8.07TiB
/dev/sdg 8.07TiB
/dev/sdd 8.07TiB
/dev/sdc 8.07TiB
/dev/sde 8.07TiB
Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
/dev/sdh 34.00GiB
/dev/sdf 32.00GiB
/dev/sdg 32.00GiB
System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
/dev/sdf 32.00MiB
/dev/sdg 32.00MiB
Unallocated:
/dev/sdh 2.81TiB
/dev/sdf 2.81TiB
/dev/sdg 2.81TiB
/dev/sdd 1.03TiB
/dev/sdc 1.03TiB
/dev/sde 1.03TiB
On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2020/5/8 下午1:12, Tyler Richmond wrote:
> > If this is saying there's no extra space for metadata, is that why
> > adding more files often makes the system hang for 30-90s? Is there
> > anything I should do about that?
>
> I'm not sure about the hang though.
>
> It would be nice to give more info to diagnosis.
> The output of 'btrfs fi usage' is useful for space usage problem.
>
> But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
> space in vanilla df command) space for btrfs.
>
> Thanks,
> Qu
>
> >
> > Thank you so much for all of your help. I love how flexible BTRFS is
> > but when things go wrong it's very hard for me to troubleshoot.
> >
> > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>
> >>
> >>
> >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
> >>> Something went wrong:
> >>>
> >>> Reinitialize checksum tree
> >>> Unable to find block group for 0
> >>> Unable to find block group for 0
> >>> Unable to find block group for 0
> >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
> >>> btrfs(+0x6dd94)[0x55a933af7d94]
> >>> btrfs(+0x71b94)[0x55a933afbb94]
> >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
> >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
> >>> btrfs(+0x360b2)[0x55a933ac00b2]
> >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
> >>> btrfs(main+0x98)[0x55a933a9fe88]
> >>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
> >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
> >>> Aborted
> >>
> >> This means no space for extra metadata...
> >>
> >> Anyway the csum tree problem shouldn't be a big thing, you could leave
> >> it and call it a day.
> >>
> >> BTW, as long as btrfs check reports no extra problem for the inode
> >> generation, it should be pretty safe to use the fs.
> >>
> >> Thanks,
> >> Qu
> >>>
> >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
> >>> available. I'll let that try overnight?
> >>>
> >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
> >>>>> Thank you for helping. The end result of the scan was:
> >>>>>
> >>>>>
> >>>>> [1/7] checking root items
> >>>>> [2/7] checking extents
> >>>>> [3/7] checking free space cache
> >>>>> [4/7] checking fs roots
> >>>>
> >>>> Good news is, your fs is still mostly fine.
> >>>>
> >>>>> [5/7] checking only csums items (without verifying data)
> >>>>> there are no extents for csum range 0-69632
> >>>>> csum exists for 0-69632 but there is no extent record
> >>>>> ...
> >>>>> ...
> >>>>> there are no extents for csum range 946692096-946827264
> >>>>> csum exists for 946692096-946827264 but there is no extent record
> >>>>> there are no extents for csum range 946831360-947912704
> >>>>> csum exists for 946831360-947912704 but there is no extent record
> >>>>> ERROR: errors found in csum tree
> >>>>
> >>>> Only extent tree is corrupted.
> >>>>
> >>>> Normally btrfs check --init-csum-tree should be able to handle it.
> >>>>
> >>>> But still, please be sure you're using the latest btrfs-progs to fix it.
> >>>>
> >>>> Thanks,
> >>>> Qu
> >>>>
> >>>>> [6/7] checking root refs
> >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
> >>>>> found 44157956026368 bytes used, error(s) found
> >>>>> total csum bytes: 42038602716
> >>>>> total tree bytes: 49688616960
> >>>>> total fs tree bytes: 1256427520
> >>>>> total extent tree bytes: 1709105152
> >>>>> btree space waste bytes: 3172727316
> >>>>> file data blocks allocated: 261625653436416
> >>>>> referenced 47477768499200
> >>>>>
> >>>>> What do I need to do to fix all of this?
> >>>>>
> >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
> >>>>>>> Well, the repair doesn't look terribly successful.
> >>>>>>>
> >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>>>> Ignoring transid failure
> >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>>>> parent level=1
> >>>>>>> child level=4
> >>>>>>
> >>>>>> This means there are more problems, not only the hash name mismatch.
> >>>>>>
> >>>>>> This means the fs is already corrupted, the name hash is just one
> >>>>>> unrelated symptom.
> >>>>>>
> >>>>>> The only good news is, btrfs-progs abort the transaction, thus no
> >>>>>> further damage to the fs.
> >>>>>>
> >>>>>> Please run a plain btrfs-check to show what's the problem first.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Qu
> >>>>>>
> >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>>>> Ignoring transid failure
> >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>>>> parent level=1
> >>>>>>> child level=4
> >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>>>> Ignoring transid failure
> >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>>>> parent level=1
> >>>>>>> child level=4
> >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>>>> Ignoring transid failure
> >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>>>> parent level=1
> >>>>>>> child level=4
> >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>>>> Ignoring transid failure
> >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>>>> parent level=1
> >>>>>>> child level=4
> >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>>>> Ignoring transid failure
> >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>>>> parent level=1
> >>>>>>> child level=4
> >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>>>> Ignoring transid failure
> >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>>>> parent level=1
> >>>>>>> child level=4
> >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>>>> Ignoring transid failure
> >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>>>> parent level=1
> >>>>>>> child level=4
> >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>>>> Ignoring transid failure
> >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>>>> parent level=1
> >>>>>>> child level=4
> >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>>>> Ignoring transid failure
> >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>>>> parent level=1
> >>>>>>> child level=4
> >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> >>>>>>> Ignoring transid failure
> >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> >>>>>>> parent level=1
> >>>>>>> child level=4
> >>>>>>> ERROR: failed to zero log tree: -17
> >>>>>>> ERROR: attempt to start transaction over already running one
> >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
> >>>>>>> extent buffer leak: start 225049066086400 len 4096
> >>>>>>> extent buffer leak: start 225049066086400 len 4096
> >>>>>>> WARNING: dirty eb leak (aborted trans): start 225049066086400 len 4096
> >>>>>>> extent buffer leak: start 225049066094592 len 4096
> >>>>>>> extent buffer leak: start 225049066094592 len 4096
> >>>>>>> WARNING: dirty eb leak (aborted trans): start 225049066094592 len 4096
> >>>>>>> extent buffer leak: start 225049066102784 len 4096
> >>>>>>> extent buffer leak: start 225049066102784 len 4096
> >>>>>>> WARNING: dirty eb leak (aborted trans): start 225049066102784 len 4096
> >>>>>>> extent buffer leak: start 225049066131456 len 4096
> >>>>>>> extent buffer leak: start 225049066131456 len 4096
> >>>>>>> WARNING: dirty eb leak (aborted trans): start 225049066131456 len 4096
> >>>>>>>
> >>>>>>> What is going on?
> >>>>>>>
> >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond <t.d.richmond@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>> Chris, I had used the correct mountpoint in the command. I just edited
> >>>>>>>> it in the email to be /mountpoint for consistency.
> >>>>>>>>
> >>>>>>>> Qu, I'll try the repair. Fingers crossed!
> >>>>>>>>
> >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
> >>>>>>>>>> Hello,
> >>>>>>>>>>
> >>>>>>>>>> I looked up this error and it basically says ask a developer to
> >>>>>>>>>> determine if it's a false error or not. I just started getting some
> >>>>>>>>>> slow response times, and looked at the dmesg log to find a ton of
> >>>>>>>>>> these errors.
> >>>>>>>>>>
> >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt leaf: root=5
> >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>>>>>> [192088.449823] BTRFS error (device sdh): block=203510940835840 read
> >>>>>>>>>> time tree block corruption detected
> >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt leaf: root=5
> >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>>>>>> [192088.462773] BTRFS error (device sdh): block=203510940835840 read
> >>>>>>>>>> time tree block corruption detected
> >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt leaf: root=5
> >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>>>>>> [192088.468457] BTRFS error (device sdh): block=203510940835840 read
> >>>>>>>>>> time tree block corruption detected
> >>>>>>>>>>
> >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
> >>>>>>>>>>
> >>>>>>>>>> Is there anything I should do about this, or should I just continue
> >>>>>>>>>> using my array as normal?
> >>>>>>>>>
> >>>>>>>>> This is caused by older kernel underflow inode generation.
> >>>>>>>>>
> >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
> >>>>>>>>>
> >>>>>>>>> Or you can go safer, by manually locating the inode using its inode
> >>>>>>>>> number (1311670), and copy it to some new location using previous
> >>>>>>>>> working kernel, then delete the old file, copy the new one back to fix it.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Qu
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Thank you!
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> >>
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-05-08 13:52 ` Tyler Richmond
@ 2020-08-18 3:36 ` Tyler Richmond
[not found] ` <CAJheHN3qwDAGY=z14zfO4LBrxNJZZ_rvAMsWLwe-k+4+t3zLog@mail.gmail.com>
1 sibling, 0 replies; 56+ messages in thread
From: Tyler Richmond @ 2020-08-18 3:36 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Btrfs BTRFS
Qu,
Sorry to resurrect this thread, but I just ran into something that I
can't really just ignore. I've found a folder that is full of files
which I guess have been broken somehow. I found a backup and restored
them, but I want to delete this folder of broken files. But whenever I
try, the fs is forced into readonly mode again. I just finished
another btrfs check --repair but it didn't fix the problem.
https://pastebin.com/eTV3s3fr
I'm already on btrfs-progs v5.7. Any new suggestions?
On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>
> 5.6.1 also failed the same way. Here's the usage output. This is the
> part where you see I've been using RAID5 haha
>
> WARNING: RAID56 detected, not implemented
> Overall:
> Device size: 60.03TiB
> Device allocated: 98.06GiB
> Device unallocated: 59.93TiB
> Device missing: 0.00B
> Used: 92.56GiB
> Free (estimated): 0.00B (min: 8.00EiB)
> Data ratio: 0.00
> Metadata ratio: 2.00
> Global reserve: 512.00MiB (used: 0.00B)
> Multiple profiles: no
>
> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
> /dev/sdh 8.07TiB
> /dev/sdf 8.07TiB
> /dev/sdg 8.07TiB
> /dev/sdd 8.07TiB
> /dev/sdc 8.07TiB
> /dev/sde 8.07TiB
>
> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
> /dev/sdh 34.00GiB
> /dev/sdf 32.00GiB
> /dev/sdg 32.00GiB
>
> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
> /dev/sdf 32.00MiB
> /dev/sdg 32.00MiB
>
> Unallocated:
> /dev/sdh 2.81TiB
> /dev/sdf 2.81TiB
> /dev/sdg 2.81TiB
> /dev/sdd 1.03TiB
> /dev/sdc 1.03TiB
> /dev/sde 1.03TiB
>
> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >
> >
> >
> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
> > > If this is saying there's no extra space for metadata, is that why
> > > adding more files often makes the system hang for 30-90s? Is there
> > > anything I should do about that?
> >
> > I'm not sure about the hang though.
> >
> > It would be nice to give more info to diagnosis.
> > The output of 'btrfs fi usage' is useful for space usage problem.
> >
> > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
> > space in vanilla df command) space for btrfs.
> >
> > Thanks,
> > Qu
> >
> > >
> > > Thank you so much for all of your help. I love how flexible BTRFS is
> > > but when things go wrong it's very hard for me to troubleshoot.
> > >
> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > >>
> > >>
> > >>
> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
> > >>> Something went wrong:
> > >>>
> > >>> Reinitialize checksum tree
> > >>> Unable to find block group for 0
> > >>> Unable to find block group for 0
> > >>> Unable to find block group for 0
> > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
> > >>> btrfs(+0x71b94)[0x55a933afbb94]
> > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
> > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
> > >>> btrfs(main+0x98)[0x55a933a9fe88]
> > >>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
> > >>> Aborted
> > >>
> > >> This means no space for extra metadata...
> > >>
> > >> Anyway the csum tree problem shouldn't be a big thing, you could leave
> > >> it and call it a day.
> > >>
> > >> BTW, as long as btrfs check reports no extra problem for the inode
> > >> generation, it should be pretty safe to use the fs.
> > >>
> > >> Thanks,
> > >> Qu
> > >>>
> > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
> > >>> available. I'll let that try overnight?
> > >>>
> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > >>>>
> > >>>>
> > >>>>
> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
> > >>>>> Thank you for helping. The end result of the scan was:
> > >>>>>
> > >>>>>
> > >>>>> [1/7] checking root items
> > >>>>> [2/7] checking extents
> > >>>>> [3/7] checking free space cache
> > >>>>> [4/7] checking fs roots
> > >>>>
> > >>>> Good news is, your fs is still mostly fine.
> > >>>>
> > >>>>> [5/7] checking only csums items (without verifying data)
> > >>>>> there are no extents for csum range 0-69632
> > >>>>> csum exists for 0-69632 but there is no extent record
> > >>>>> ...
> > >>>>> ...
> > >>>>> there are no extents for csum range 946692096-946827264
> > >>>>> csum exists for 946692096-946827264 but there is no extent record
> > >>>>> there are no extents for csum range 946831360-947912704
> > >>>>> csum exists for 946831360-947912704 but there is no extent record
> > >>>>> ERROR: errors found in csum tree
> > >>>>
> > >>>> Only extent tree is corrupted.
> > >>>>
> > >>>> Normally btrfs check --init-csum-tree should be able to handle it.
> > >>>>
> > >>>> But still, please be sure you're using the latest btrfs-progs to fix it.
> > >>>>
> > >>>> Thanks,
> > >>>> Qu
> > >>>>
> > >>>>> [6/7] checking root refs
> > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
> > >>>>> found 44157956026368 bytes used, error(s) found
> > >>>>> total csum bytes: 42038602716
> > >>>>> total tree bytes: 49688616960
> > >>>>> total fs tree bytes: 1256427520
> > >>>>> total extent tree bytes: 1709105152
> > >>>>> btree space waste bytes: 3172727316
> > >>>>> file data blocks allocated: 261625653436416
> > >>>>> referenced 47477768499200
> > >>>>>
> > >>>>> What do I need to do to fix all of this?
> > >>>>>
> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
> > >>>>>>> Well, the repair doesn't look terribly successful.
> > >>>>>>>
> > >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>
> > >>>>>> This means there are more problems, not only the hash name mismatch.
> > >>>>>>
> > >>>>>> This means the fs is already corrupted, the name hash is just one
> > >>>>>> unrelated symptom.
> > >>>>>>
> > >>>>>> The only good news is, btrfs-progs abort the transaction, thus no
> > >>>>>> further damage to the fs.
> > >>>>>>
> > >>>>>> Please run a plain btrfs-check to show what's the problem first.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Qu
> > >>>>>>
> > >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> ERROR: failed to zero log tree: -17
> > >>>>>>> ERROR: attempt to start transaction over already running one
> > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> > >>>>>>> WARNING: dirty eb leak (aborted trans): start 225049066086400 len 4096
> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> > >>>>>>> WARNING: dirty eb leak (aborted trans): start 225049066094592 len 4096
> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> > >>>>>>> WARNING: dirty eb leak (aborted trans): start 225049066102784 len 4096
> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> > >>>>>>> WARNING: dirty eb leak (aborted trans): start 225049066131456 len 4096
> > >>>>>>>
> > >>>>>>> What is going on?
> > >>>>>>>
> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond <t.d.richmond@gmail.com> wrote:
> > >>>>>>>>
> > >>>>>>>> Chris, I had used the correct mountpoint in the command. I just edited
> > >>>>>>>> it in the email to be /mountpoint for consistency.
> > >>>>>>>>
> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
> > >>>>>>>>
> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
> > >>>>>>>>>> Hello,
> > >>>>>>>>>>
> > >>>>>>>>>> I looked up this error and it basically says ask a developer to
> > >>>>>>>>>> determine if it's a false error or not. I just started getting some
> > >>>>>>>>>> slow response times, and looked at the dmesg log to find a ton of
> > >>>>>>>>>> these errors.
> > >>>>>>>>>>
> > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt leaf: root=5
> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> > >>>>>>>>>> [192088.449823] BTRFS error (device sdh): block=203510940835840 read
> > >>>>>>>>>> time tree block corruption detected
> > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt leaf: root=5
> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> > >>>>>>>>>> [192088.462773] BTRFS error (device sdh): block=203510940835840 read
> > >>>>>>>>>> time tree block corruption detected
> > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt leaf: root=5
> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> > >>>>>>>>>> [192088.468457] BTRFS error (device sdh): block=203510940835840 read
> > >>>>>>>>>> time tree block corruption detected
> > >>>>>>>>>>
> > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
> > >>>>>>>>>>
> > >>>>>>>>>> Is there anything I should do about this, or should I just continue
> > >>>>>>>>>> using my array as normal?
> > >>>>>>>>>
> > >>>>>>>>> This is caused by older kernel underflow inode generation.
> > >>>>>>>>>
> > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
> > >>>>>>>>>
> > >>>>>>>>> Or you can go safer, by manually locating the inode using its inode
> > >>>>>>>>> number (1311670), and copy it to some new location using previous
> > >>>>>>>>> working kernel, then delete the old file, copy the new one back to fix it.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> Qu
> > >>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Thank you!
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
[not found] ` <CAJheHN3qwDAGY=z14zfO4LBrxNJZZ_rvAMsWLwe-k+4+t3zLog@mail.gmail.com>
@ 2020-08-18 6:07 ` Qu Wenruo
2020-08-18 12:18 ` Tyler Richmond
0 siblings, 1 reply; 56+ messages in thread
From: Qu Wenruo @ 2020-08-18 6:07 UTC (permalink / raw)
To: Tyler Richmond; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 16984 bytes --]
On 2020/8/18 上午11:35, Tyler Richmond wrote:
> Qu,
>
> Sorry to resurrect this thread, but I just ran into something that I
> can't really just ignore. I've found a folder that is full of files
> which I guess have been broken somehow. I found a backup and restored
> them, but I want to delete this folder of broken files. But whenever I
> try, the fs is forced into readonly mode again. I just finished another
> btrfs check --repair but it didn't fix the problem.
>
> https://pastebin.com/eTV3s3fr
Is that the full output?
No inode generation bugs?
>
> I'm already on btrfs-progs v5.7. Any new suggestions?
Strange.
The detection and repair should have been merged into v5.5.
If your fs is small enough, would you please provide the "btrfs-image
-c9" dump?
It would contain the filenames and directories names, but doesn't
contain file contents.
Thanks,
Qu
>
> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com
> <mailto:t.d.richmond@gmail.com>> wrote:
>
> 5.6.1 also failed the same way. Here's the usage output. This is the
> part where you see I've been using RAID5 haha
>
> WARNING: RAID56 detected, not implemented
> Overall:
> Device size: 60.03TiB
> Device allocated: 98.06GiB
> Device unallocated: 59.93TiB
> Device missing: 0.00B
> Used: 92.56GiB
> Free (estimated): 0.00B (min: 8.00EiB)
> Data ratio: 0.00
> Metadata ratio: 2.00
> Global reserve: 512.00MiB (used: 0.00B)
> Multiple profiles: no
>
> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
> /dev/sdh 8.07TiB
> /dev/sdf 8.07TiB
> /dev/sdg 8.07TiB
> /dev/sdd 8.07TiB
> /dev/sdc 8.07TiB
> /dev/sde 8.07TiB
>
> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
> /dev/sdh 34.00GiB
> /dev/sdf 32.00GiB
> /dev/sdg 32.00GiB
>
> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
> /dev/sdf 32.00MiB
> /dev/sdg 32.00MiB
>
> Unallocated:
> /dev/sdh 2.81TiB
> /dev/sdf 2.81TiB
> /dev/sdg 2.81TiB
> /dev/sdd 1.03TiB
> /dev/sdc 1.03TiB
> /dev/sde 1.03TiB
>
> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >
> >
> >
> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
> > > If this is saying there's no extra space for metadata, is that why
> > > adding more files often makes the system hang for 30-90s? Is there
> > > anything I should do about that?
> >
> > I'm not sure about the hang though.
> >
> > It would be nice to give more info to diagnosis.
> > The output of 'btrfs fi usage' is useful for space usage problem.
> >
> > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
> > space in vanilla df command) space for btrfs.
> >
> > Thanks,
> > Qu
> >
> > >
> > > Thank you so much for all of your help. I love how flexible BTRFS is
> > > but when things go wrong it's very hard for me to troubleshoot.
> > >
> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > >>
> > >>
> > >>
> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
> > >>> Something went wrong:
> > >>>
> > >>> Reinitialize checksum tree
> > >>> Unable to find block group for 0
> > >>> Unable to find block group for 0
> > >>> Unable to find block group for 0
> > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
> > >>> btrfs(+0x71b94)[0x55a933afbb94]
> > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
> > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
> > >>> btrfs(main+0x98)[0x55a933a9fe88]
> > >>>
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
> > >>> Aborted
> > >>
> > >> This means no space for extra metadata...
> > >>
> > >> Anyway the csum tree problem shouldn't be a big thing, you
> could leave
> > >> it and call it a day.
> > >>
> > >> BTW, as long as btrfs check reports no extra problem for the inode
> > >> generation, it should be pretty safe to use the fs.
> > >>
> > >> Thanks,
> > >> Qu
> > >>>
> > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
> > >>> available. I'll let that try overnight?
> > >>>
> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > >>>>
> > >>>>
> > >>>>
> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
> > >>>>> Thank you for helping. The end result of the scan was:
> > >>>>>
> > >>>>>
> > >>>>> [1/7] checking root items
> > >>>>> [2/7] checking extents
> > >>>>> [3/7] checking free space cache
> > >>>>> [4/7] checking fs roots
> > >>>>
> > >>>> Good news is, your fs is still mostly fine.
> > >>>>
> > >>>>> [5/7] checking only csums items (without verifying data)
> > >>>>> there are no extents for csum range 0-69632
> > >>>>> csum exists for 0-69632 but there is no extent record
> > >>>>> ...
> > >>>>> ...
> > >>>>> there are no extents for csum range 946692096-946827264
> > >>>>> csum exists for 946692096-946827264 but there is no extent
> record
> > >>>>> there are no extents for csum range 946831360-947912704
> > >>>>> csum exists for 946831360-947912704 but there is no extent
> record
> > >>>>> ERROR: errors found in csum tree
> > >>>>
> > >>>> Only extent tree is corrupted.
> > >>>>
> > >>>> Normally btrfs check --init-csum-tree should be able to
> handle it.
> > >>>>
> > >>>> But still, please be sure you're using the latest btrfs-progs
> to fix it.
> > >>>>
> > >>>> Thanks,
> > >>>> Qu
> > >>>>
> > >>>>> [6/7] checking root refs
> > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
> > >>>>> found 44157956026368 bytes used, error(s) found
> > >>>>> total csum bytes: 42038602716
> > >>>>> total tree bytes: 49688616960
> > >>>>> total fs tree bytes: 1256427520
> > >>>>> total extent tree bytes: 1709105152
> > >>>>> btree space waste bytes: 3172727316
> > >>>>> file data blocks allocated: 261625653436416
> > >>>>> referenced 47477768499200
> > >>>>>
> > >>>>> What do I need to do to fix all of this?
> > >>>>>
> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
> > >>>>>>> Well, the repair doesn't look terribly successful.
> > >>>>>>>
> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> 6875841 found 6876224
> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> 6875841 found 6876224
> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>
> > >>>>>> This means there are more problems, not only the hash name
> mismatch.
> > >>>>>>
> > >>>>>> This means the fs is already corrupted, the name hash is
> just one
> > >>>>>> unrelated symptom.
> > >>>>>>
> > >>>>>> The only good news is, btrfs-progs abort the transaction,
> thus no
> > >>>>>> further damage to the fs.
> > >>>>>>
> > >>>>>> Please run a plain btrfs-check to show what's the problem
> first.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Qu
> > >>>>>>
> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> 6875841 found 6876224
> > >>>>>>> Ignoring transid failure
> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> item=84
> > >>>>>>> parent level=1
> > >>>>>>> child level=4
> > >>>>>>> ERROR: failed to zero log tree: -17
> > >>>>>>> ERROR: attempt to start transaction over already running one
> > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> 225049066086400 len 4096
> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> 225049066094592 len 4096
> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> 225049066102784 len 4096
> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> 225049066131456 len 4096
> > >>>>>>>
> > >>>>>>> What is going on?
> > >>>>>>>
> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond
> <t.d.richmond@gmail.com <mailto:t.d.richmond@gmail.com>> wrote:
> > >>>>>>>>
> > >>>>>>>> Chris, I had used the correct mountpoint in the command.
> I just edited
> > >>>>>>>> it in the email to be /mountpoint for consistency.
> > >>>>>>>>
> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
> > >>>>>>>>
> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
> > >>>>>>>>>> Hello,
> > >>>>>>>>>>
> > >>>>>>>>>> I looked up this error and it basically says ask a
> developer to
> > >>>>>>>>>> determine if it's a false error or not. I just started
> getting some
> > >>>>>>>>>> slow response times, and looked at the dmesg log to
> find a ton of
> > >>>>>>>>>> these errors.
> > >>>>>>>>>>
> > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt
> leaf: root=5
> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> generation:
> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> > >>>>>>>>>> [192088.449823] BTRFS error (device sdh):
> block=203510940835840 read
> > >>>>>>>>>> time tree block corruption detected
> > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt
> leaf: root=5
> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> generation:
> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> > >>>>>>>>>> [192088.462773] BTRFS error (device sdh):
> block=203510940835840 read
> > >>>>>>>>>> time tree block corruption detected
> > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt
> leaf: root=5
> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> generation:
> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> > >>>>>>>>>> [192088.468457] BTRFS error (device sdh):
> block=203510940835840 read
> > >>>>>>>>>> time tree block corruption detected
> > >>>>>>>>>>
> > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
> > >>>>>>>>>>
> > >>>>>>>>>> Is there anything I should do about this, or should I
> just continue
> > >>>>>>>>>> using my array as normal?
> > >>>>>>>>>
> > >>>>>>>>> This is caused by older kernel underflow inode generation.
> > >>>>>>>>>
> > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
> > >>>>>>>>>
> > >>>>>>>>> Or you can go safer, by manually locating the inode
> using its inode
> > >>>>>>>>> number (1311670), and copy it to some new location using
> previous
> > >>>>>>>>> working kernel, then delete the old file, copy the new
> one back to fix it.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> Qu
> > >>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Thank you!
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-08-18 6:07 ` Qu Wenruo
@ 2020-08-18 12:18 ` Tyler Richmond
2020-08-23 1:15 ` Tyler Richmond
0 siblings, 1 reply; 56+ messages in thread
From: Tyler Richmond @ 2020-08-18 12:18 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Btrfs BTRFS
I didn't check dmesg during the btrfs check, but that was the only
output during the rm -f before it was forced readonly. I just checked
dmesg for inode generation values, and there are a lot of them.
https://pastebin.com/stZdN0ta
The dmesg output had 990 lines containing inode generation.
However, these were at least later. I tried to do a btrfs balance
-mconvert raid1 and it failed with an I/O error. That is probably what
generated these specific errors, but maybe they were also happening
during the btrfs repair.
The FS is ~45TB, but the btrfs-image -c9 failed anway with:
ERROR: either extent tree is corrupted or deprecated extent ref format
ERROR: create failed: -5
On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2020/8/18 上午11:35, Tyler Richmond wrote:
> > Qu,
> >
> > Sorry to resurrect this thread, but I just ran into something that I
> > can't really just ignore. I've found a folder that is full of files
> > which I guess have been broken somehow. I found a backup and restored
> > them, but I want to delete this folder of broken files. But whenever I
> > try, the fs is forced into readonly mode again. I just finished another
> > btrfs check --repair but it didn't fix the problem.
> >
> > https://pastebin.com/eTV3s3fr
>
> Is that the full output?
>
> No inode generation bugs?
> >
> > I'm already on btrfs-progs v5.7. Any new suggestions?
>
> Strange.
>
> The detection and repair should have been merged into v5.5.
>
> If your fs is small enough, would you please provide the "btrfs-image
> -c9" dump?
>
> It would contain the filenames and directories names, but doesn't
> contain file contents.
>
> Thanks,
> Qu
> >
> > On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com
> > <mailto:t.d.richmond@gmail.com>> wrote:
> >
> > 5.6.1 also failed the same way. Here's the usage output. This is the
> > part where you see I've been using RAID5 haha
> >
> > WARNING: RAID56 detected, not implemented
> > Overall:
> > Device size: 60.03TiB
> > Device allocated: 98.06GiB
> > Device unallocated: 59.93TiB
> > Device missing: 0.00B
> > Used: 92.56GiB
> > Free (estimated): 0.00B (min: 8.00EiB)
> > Data ratio: 0.00
> > Metadata ratio: 2.00
> > Global reserve: 512.00MiB (used: 0.00B)
> > Multiple profiles: no
> >
> > Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
> > /dev/sdh 8.07TiB
> > /dev/sdf 8.07TiB
> > /dev/sdg 8.07TiB
> > /dev/sdd 8.07TiB
> > /dev/sdc 8.07TiB
> > /dev/sde 8.07TiB
> >
> > Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
> > /dev/sdh 34.00GiB
> > /dev/sdf 32.00GiB
> > /dev/sdg 32.00GiB
> >
> > System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
> > /dev/sdf 32.00MiB
> > /dev/sdg 32.00MiB
> >
> > Unallocated:
> > /dev/sdh 2.81TiB
> > /dev/sdf 2.81TiB
> > /dev/sdg 2.81TiB
> > /dev/sdd 1.03TiB
> > /dev/sdc 1.03TiB
> > /dev/sde 1.03TiB
> >
> > On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> > <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > >
> > >
> > >
> > > On 2020/5/8 下午1:12, Tyler Richmond wrote:
> > > > If this is saying there's no extra space for metadata, is that why
> > > > adding more files often makes the system hang for 30-90s? Is there
> > > > anything I should do about that?
> > >
> > > I'm not sure about the hang though.
> > >
> > > It would be nice to give more info to diagnosis.
> > > The output of 'btrfs fi usage' is useful for space usage problem.
> > >
> > > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
> > > space in vanilla df command) space for btrfs.
> > >
> > > Thanks,
> > > Qu
> > >
> > > >
> > > > Thank you so much for all of your help. I love how flexible BTRFS is
> > > > but when things go wrong it's very hard for me to troubleshoot.
> > > >
> > > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> > <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > > >>
> > > >>
> > > >>
> > > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
> > > >>> Something went wrong:
> > > >>>
> > > >>> Reinitialize checksum tree
> > > >>> Unable to find block group for 0
> > > >>> Unable to find block group for 0
> > > >>> Unable to find block group for 0
> > > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
> > > >>> btrfs(+0x6dd94)[0x55a933af7d94]
> > > >>> btrfs(+0x71b94)[0x55a933afbb94]
> > > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
> > > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
> > > >>> btrfs(+0x360b2)[0x55a933ac00b2]
> > > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
> > > >>> btrfs(main+0x98)[0x55a933a9fe88]
> > > >>>
> > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
> > > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
> > > >>> Aborted
> > > >>
> > > >> This means no space for extra metadata...
> > > >>
> > > >> Anyway the csum tree problem shouldn't be a big thing, you
> > could leave
> > > >> it and call it a day.
> > > >>
> > > >> BTW, as long as btrfs check reports no extra problem for the inode
> > > >> generation, it should be pretty safe to use the fs.
> > > >>
> > > >> Thanks,
> > > >> Qu
> > > >>>
> > > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
> > > >>> available. I'll let that try overnight?
> > > >>>
> > > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
> > <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
> > > >>>>> Thank you for helping. The end result of the scan was:
> > > >>>>>
> > > >>>>>
> > > >>>>> [1/7] checking root items
> > > >>>>> [2/7] checking extents
> > > >>>>> [3/7] checking free space cache
> > > >>>>> [4/7] checking fs roots
> > > >>>>
> > > >>>> Good news is, your fs is still mostly fine.
> > > >>>>
> > > >>>>> [5/7] checking only csums items (without verifying data)
> > > >>>>> there are no extents for csum range 0-69632
> > > >>>>> csum exists for 0-69632 but there is no extent record
> > > >>>>> ...
> > > >>>>> ...
> > > >>>>> there are no extents for csum range 946692096-946827264
> > > >>>>> csum exists for 946692096-946827264 but there is no extent
> > record
> > > >>>>> there are no extents for csum range 946831360-947912704
> > > >>>>> csum exists for 946831360-947912704 but there is no extent
> > record
> > > >>>>> ERROR: errors found in csum tree
> > > >>>>
> > > >>>> Only extent tree is corrupted.
> > > >>>>
> > > >>>> Normally btrfs check --init-csum-tree should be able to
> > handle it.
> > > >>>>
> > > >>>> But still, please be sure you're using the latest btrfs-progs
> > to fix it.
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Qu
> > > >>>>
> > > >>>>> [6/7] checking root refs
> > > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
> > > >>>>> found 44157956026368 bytes used, error(s) found
> > > >>>>> total csum bytes: 42038602716
> > > >>>>> total tree bytes: 49688616960
> > > >>>>> total fs tree bytes: 1256427520
> > > >>>>> total extent tree bytes: 1709105152
> > > >>>>> btree space waste bytes: 3172727316
> > > >>>>> file data blocks allocated: 261625653436416
> > > >>>>> referenced 47477768499200
> > > >>>>>
> > > >>>>> What do I need to do to fix all of this?
> > > >>>>>
> > > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
> > <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
> > > >>>>>>> Well, the repair doesn't look terribly successful.
> > > >>>>>>>
> > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > 6875841 found 6876224
> > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > 6875841 found 6876224
> > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > 6875841 found 6876224
> > > >>>>>>> Ignoring transid failure
> > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > item=84
> > > >>>>>>> parent level=1
> > > >>>>>>> child level=4
> > > >>>>>>
> > > >>>>>> This means there are more problems, not only the hash name
> > mismatch.
> > > >>>>>>
> > > >>>>>> This means the fs is already corrupted, the name hash is
> > just one
> > > >>>>>> unrelated symptom.
> > > >>>>>>
> > > >>>>>> The only good news is, btrfs-progs abort the transaction,
> > thus no
> > > >>>>>> further damage to the fs.
> > > >>>>>>
> > > >>>>>> Please run a plain btrfs-check to show what's the problem
> > first.
> > > >>>>>>
> > > >>>>>> Thanks,
> > > >>>>>> Qu
> > > >>>>>>
> > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > 6875841 found 6876224
> > > >>>>>>> Ignoring transid failure
> > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > item=84
> > > >>>>>>> parent level=1
> > > >>>>>>> child level=4
> > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > 6875841 found 6876224
> > > >>>>>>> Ignoring transid failure
> > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > item=84
> > > >>>>>>> parent level=1
> > > >>>>>>> child level=4
> > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > 6875841 found 6876224
> > > >>>>>>> Ignoring transid failure
> > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > item=84
> > > >>>>>>> parent level=1
> > > >>>>>>> child level=4
> > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > 6875841 found 6876224
> > > >>>>>>> Ignoring transid failure
> > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > item=84
> > > >>>>>>> parent level=1
> > > >>>>>>> child level=4
> > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > 6875841 found 6876224
> > > >>>>>>> Ignoring transid failure
> > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > item=84
> > > >>>>>>> parent level=1
> > > >>>>>>> child level=4
> > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > 6875841 found 6876224
> > > >>>>>>> Ignoring transid failure
> > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > item=84
> > > >>>>>>> parent level=1
> > > >>>>>>> child level=4
> > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > 6875841 found 6876224
> > > >>>>>>> Ignoring transid failure
> > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > item=84
> > > >>>>>>> parent level=1
> > > >>>>>>> child level=4
> > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > 6875841 found 6876224
> > > >>>>>>> Ignoring transid failure
> > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > item=84
> > > >>>>>>> parent level=1
> > > >>>>>>> child level=4
> > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > 6875841 found 6876224
> > > >>>>>>> Ignoring transid failure
> > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > item=84
> > > >>>>>>> parent level=1
> > > >>>>>>> child level=4
> > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > 6875841 found 6876224
> > > >>>>>>> Ignoring transid failure
> > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > item=84
> > > >>>>>>> parent level=1
> > > >>>>>>> child level=4
> > > >>>>>>> ERROR: failed to zero log tree: -17
> > > >>>>>>> ERROR: attempt to start transaction over already running one
> > > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
> > > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> > > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> > > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> > 225049066086400 len 4096
> > > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> > > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> > > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> > 225049066094592 len 4096
> > > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> > > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> > > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> > 225049066102784 len 4096
> > > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> > > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> > > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> > 225049066131456 len 4096
> > > >>>>>>>
> > > >>>>>>> What is going on?
> > > >>>>>>>
> > > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond
> > <t.d.richmond@gmail.com <mailto:t.d.richmond@gmail.com>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> Chris, I had used the correct mountpoint in the command.
> > I just edited
> > > >>>>>>>> it in the email to be /mountpoint for consistency.
> > > >>>>>>>>
> > > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
> > > >>>>>>>>
> > > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
> > <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
> > > >>>>>>>>>> Hello,
> > > >>>>>>>>>>
> > > >>>>>>>>>> I looked up this error and it basically says ask a
> > developer to
> > > >>>>>>>>>> determine if it's a false error or not. I just started
> > getting some
> > > >>>>>>>>>> slow response times, and looked at the dmesg log to
> > find a ton of
> > > >>>>>>>>>> these errors.
> > > >>>>>>>>>>
> > > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt
> > leaf: root=5
> > > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> > generation:
> > > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> > > >>>>>>>>>> [192088.449823] BTRFS error (device sdh):
> > block=203510940835840 read
> > > >>>>>>>>>> time tree block corruption detected
> > > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt
> > leaf: root=5
> > > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> > generation:
> > > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> > > >>>>>>>>>> [192088.462773] BTRFS error (device sdh):
> > block=203510940835840 read
> > > >>>>>>>>>> time tree block corruption detected
> > > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt
> > leaf: root=5
> > > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> > generation:
> > > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> > > >>>>>>>>>> [192088.468457] BTRFS error (device sdh):
> > block=203510940835840 read
> > > >>>>>>>>>> time tree block corruption detected
> > > >>>>>>>>>>
> > > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Is there anything I should do about this, or should I
> > just continue
> > > >>>>>>>>>> using my array as normal?
> > > >>>>>>>>>
> > > >>>>>>>>> This is caused by older kernel underflow inode generation.
> > > >>>>>>>>>
> > > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
> > > >>>>>>>>>
> > > >>>>>>>>> Or you can go safer, by manually locating the inode
> > using its inode
> > > >>>>>>>>> number (1311670), and copy it to some new location using
> > previous
> > > >>>>>>>>> working kernel, then delete the old file, copy the new
> > one back to fix it.
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks,
> > > >>>>>>>>> Qu
> > > >>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thank you!
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> >
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-08-18 12:18 ` Tyler Richmond
@ 2020-08-23 1:15 ` Tyler Richmond
2020-08-23 1:51 ` Qu Wenruo
0 siblings, 1 reply; 56+ messages in thread
From: Tyler Richmond @ 2020-08-23 1:15 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Btrfs BTRFS
Is my best bet just to downgrade the kernel and then try to delete the
broken files? Or should I rebuild from scratch? Just don't know
whether it's worth the time to try and figure this out or if the
problems stem from the FS being too old and it's beyond trying to
repair.
On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>
> I didn't check dmesg during the btrfs check, but that was the only
> output during the rm -f before it was forced readonly. I just checked
> dmesg for inode generation values, and there are a lot of them.
>
> https://pastebin.com/stZdN0ta
> The dmesg output had 990 lines containing inode generation.
>
> However, these were at least later. I tried to do a btrfs balance
> -mconvert raid1 and it failed with an I/O error. That is probably what
> generated these specific errors, but maybe they were also happening
> during the btrfs repair.
>
> The FS is ~45TB, but the btrfs-image -c9 failed anway with:
> ERROR: either extent tree is corrupted or deprecated extent ref format
> ERROR: create failed: -5
>
>
> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >
> >
> >
> > On 2020/8/18 上午11:35, Tyler Richmond wrote:
> > > Qu,
> > >
> > > Sorry to resurrect this thread, but I just ran into something that I
> > > can't really just ignore. I've found a folder that is full of files
> > > which I guess have been broken somehow. I found a backup and restored
> > > them, but I want to delete this folder of broken files. But whenever I
> > > try, the fs is forced into readonly mode again. I just finished another
> > > btrfs check --repair but it didn't fix the problem.
> > >
> > > https://pastebin.com/eTV3s3fr
> >
> > Is that the full output?
> >
> > No inode generation bugs?
> > >
> > > I'm already on btrfs-progs v5.7. Any new suggestions?
> >
> > Strange.
> >
> > The detection and repair should have been merged into v5.5.
> >
> > If your fs is small enough, would you please provide the "btrfs-image
> > -c9" dump?
> >
> > It would contain the filenames and directories names, but doesn't
> > contain file contents.
> >
> > Thanks,
> > Qu
> > >
> > > On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com
> > > <mailto:t.d.richmond@gmail.com>> wrote:
> > >
> > > 5.6.1 also failed the same way. Here's the usage output. This is the
> > > part where you see I've been using RAID5 haha
> > >
> > > WARNING: RAID56 detected, not implemented
> > > Overall:
> > > Device size: 60.03TiB
> > > Device allocated: 98.06GiB
> > > Device unallocated: 59.93TiB
> > > Device missing: 0.00B
> > > Used: 92.56GiB
> > > Free (estimated): 0.00B (min: 8.00EiB)
> > > Data ratio: 0.00
> > > Metadata ratio: 2.00
> > > Global reserve: 512.00MiB (used: 0.00B)
> > > Multiple profiles: no
> > >
> > > Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
> > > /dev/sdh 8.07TiB
> > > /dev/sdf 8.07TiB
> > > /dev/sdg 8.07TiB
> > > /dev/sdd 8.07TiB
> > > /dev/sdc 8.07TiB
> > > /dev/sde 8.07TiB
> > >
> > > Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
> > > /dev/sdh 34.00GiB
> > > /dev/sdf 32.00GiB
> > > /dev/sdg 32.00GiB
> > >
> > > System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
> > > /dev/sdf 32.00MiB
> > > /dev/sdg 32.00MiB
> > >
> > > Unallocated:
> > > /dev/sdh 2.81TiB
> > > /dev/sdf 2.81TiB
> > > /dev/sdg 2.81TiB
> > > /dev/sdd 1.03TiB
> > > /dev/sdc 1.03TiB
> > > /dev/sde 1.03TiB
> > >
> > > On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> > > <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > > >
> > > >
> > > >
> > > > On 2020/5/8 下午1:12, Tyler Richmond wrote:
> > > > > If this is saying there's no extra space for metadata, is that why
> > > > > adding more files often makes the system hang for 30-90s? Is there
> > > > > anything I should do about that?
> > > >
> > > > I'm not sure about the hang though.
> > > >
> > > > It would be nice to give more info to diagnosis.
> > > > The output of 'btrfs fi usage' is useful for space usage problem.
> > > >
> > > > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
> > > > space in vanilla df command) space for btrfs.
> > > >
> > > > Thanks,
> > > > Qu
> > > >
> > > > >
> > > > > Thank you so much for all of your help. I love how flexible BTRFS is
> > > > > but when things go wrong it's very hard for me to troubleshoot.
> > > > >
> > > > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> > > <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > > > >>
> > > > >>
> > > > >>
> > > > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
> > > > >>> Something went wrong:
> > > > >>>
> > > > >>> Reinitialize checksum tree
> > > > >>> Unable to find block group for 0
> > > > >>> Unable to find block group for 0
> > > > >>> Unable to find block group for 0
> > > > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
> > > > >>> btrfs(+0x6dd94)[0x55a933af7d94]
> > > > >>> btrfs(+0x71b94)[0x55a933afbb94]
> > > > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
> > > > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
> > > > >>> btrfs(+0x360b2)[0x55a933ac00b2]
> > > > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
> > > > >>> btrfs(main+0x98)[0x55a933a9fe88]
> > > > >>>
> > > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
> > > > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
> > > > >>> Aborted
> > > > >>
> > > > >> This means no space for extra metadata...
> > > > >>
> > > > >> Anyway the csum tree problem shouldn't be a big thing, you
> > > could leave
> > > > >> it and call it a day.
> > > > >>
> > > > >> BTW, as long as btrfs check reports no extra problem for the inode
> > > > >> generation, it should be pretty safe to use the fs.
> > > > >>
> > > > >> Thanks,
> > > > >> Qu
> > > > >>>
> > > > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
> > > > >>> available. I'll let that try overnight?
> > > > >>>
> > > > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
> > > <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
> > > > >>>>> Thank you for helping. The end result of the scan was:
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> [1/7] checking root items
> > > > >>>>> [2/7] checking extents
> > > > >>>>> [3/7] checking free space cache
> > > > >>>>> [4/7] checking fs roots
> > > > >>>>
> > > > >>>> Good news is, your fs is still mostly fine.
> > > > >>>>
> > > > >>>>> [5/7] checking only csums items (without verifying data)
> > > > >>>>> there are no extents for csum range 0-69632
> > > > >>>>> csum exists for 0-69632 but there is no extent record
> > > > >>>>> ...
> > > > >>>>> ...
> > > > >>>>> there are no extents for csum range 946692096-946827264
> > > > >>>>> csum exists for 946692096-946827264 but there is no extent
> > > record
> > > > >>>>> there are no extents for csum range 946831360-947912704
> > > > >>>>> csum exists for 946831360-947912704 but there is no extent
> > > record
> > > > >>>>> ERROR: errors found in csum tree
> > > > >>>>
> > > > >>>> Only extent tree is corrupted.
> > > > >>>>
> > > > >>>> Normally btrfs check --init-csum-tree should be able to
> > > handle it.
> > > > >>>>
> > > > >>>> But still, please be sure you're using the latest btrfs-progs
> > > to fix it.
> > > > >>>>
> > > > >>>> Thanks,
> > > > >>>> Qu
> > > > >>>>
> > > > >>>>> [6/7] checking root refs
> > > > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
> > > > >>>>> found 44157956026368 bytes used, error(s) found
> > > > >>>>> total csum bytes: 42038602716
> > > > >>>>> total tree bytes: 49688616960
> > > > >>>>> total fs tree bytes: 1256427520
> > > > >>>>> total extent tree bytes: 1709105152
> > > > >>>>> btree space waste bytes: 3172727316
> > > > >>>>> file data blocks allocated: 261625653436416
> > > > >>>>> referenced 47477768499200
> > > > >>>>>
> > > > >>>>> What do I need to do to fix all of this?
> > > > >>>>>
> > > > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
> > > <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
> > > > >>>>>>> Well, the repair doesn't look terribly successful.
> > > > >>>>>>>
> > > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > > 6875841 found 6876224
> > > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > > 6875841 found 6876224
> > > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > > 6875841 found 6876224
> > > > >>>>>>> Ignoring transid failure
> > > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > > item=84
> > > > >>>>>>> parent level=1
> > > > >>>>>>> child level=4
> > > > >>>>>>
> > > > >>>>>> This means there are more problems, not only the hash name
> > > mismatch.
> > > > >>>>>>
> > > > >>>>>> This means the fs is already corrupted, the name hash is
> > > just one
> > > > >>>>>> unrelated symptom.
> > > > >>>>>>
> > > > >>>>>> The only good news is, btrfs-progs abort the transaction,
> > > thus no
> > > > >>>>>> further damage to the fs.
> > > > >>>>>>
> > > > >>>>>> Please run a plain btrfs-check to show what's the problem
> > > first.
> > > > >>>>>>
> > > > >>>>>> Thanks,
> > > > >>>>>> Qu
> > > > >>>>>>
> > > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > > 6875841 found 6876224
> > > > >>>>>>> Ignoring transid failure
> > > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > > item=84
> > > > >>>>>>> parent level=1
> > > > >>>>>>> child level=4
> > > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > > 6875841 found 6876224
> > > > >>>>>>> Ignoring transid failure
> > > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > > item=84
> > > > >>>>>>> parent level=1
> > > > >>>>>>> child level=4
> > > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > > 6875841 found 6876224
> > > > >>>>>>> Ignoring transid failure
> > > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > > item=84
> > > > >>>>>>> parent level=1
> > > > >>>>>>> child level=4
> > > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > > 6875841 found 6876224
> > > > >>>>>>> Ignoring transid failure
> > > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > > item=84
> > > > >>>>>>> parent level=1
> > > > >>>>>>> child level=4
> > > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > > 6875841 found 6876224
> > > > >>>>>>> Ignoring transid failure
> > > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > > item=84
> > > > >>>>>>> parent level=1
> > > > >>>>>>> child level=4
> > > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > > 6875841 found 6876224
> > > > >>>>>>> Ignoring transid failure
> > > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > > item=84
> > > > >>>>>>> parent level=1
> > > > >>>>>>> child level=4
> > > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > > 6875841 found 6876224
> > > > >>>>>>> Ignoring transid failure
> > > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > > item=84
> > > > >>>>>>> parent level=1
> > > > >>>>>>> child level=4
> > > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > > 6875841 found 6876224
> > > > >>>>>>> Ignoring transid failure
> > > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > > item=84
> > > > >>>>>>> parent level=1
> > > > >>>>>>> child level=4
> > > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > > 6875841 found 6876224
> > > > >>>>>>> Ignoring transid failure
> > > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > > item=84
> > > > >>>>>>> parent level=1
> > > > >>>>>>> child level=4
> > > > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > > 6875841 found 6876224
> > > > >>>>>>> Ignoring transid failure
> > > > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > > item=84
> > > > >>>>>>> parent level=1
> > > > >>>>>>> child level=4
> > > > >>>>>>> ERROR: failed to zero log tree: -17
> > > > >>>>>>> ERROR: attempt to start transaction over already running one
> > > > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
> > > > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> > > > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> > > > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> > > 225049066086400 len 4096
> > > > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> > > > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> > > > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> > > 225049066094592 len 4096
> > > > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> > > > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> > > > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> > > 225049066102784 len 4096
> > > > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> > > > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> > > > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> > > 225049066131456 len 4096
> > > > >>>>>>>
> > > > >>>>>>> What is going on?
> > > > >>>>>>>
> > > > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond
> > > <t.d.richmond@gmail.com <mailto:t.d.richmond@gmail.com>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>> Chris, I had used the correct mountpoint in the command.
> > > I just edited
> > > > >>>>>>>> it in the email to be /mountpoint for consistency.
> > > > >>>>>>>>
> > > > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
> > > > >>>>>>>>
> > > > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
> > > <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
> > > > >>>>>>>>>> Hello,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> I looked up this error and it basically says ask a
> > > developer to
> > > > >>>>>>>>>> determine if it's a false error or not. I just started
> > > getting some
> > > > >>>>>>>>>> slow response times, and looked at the dmesg log to
> > > find a ton of
> > > > >>>>>>>>>> these errors.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt
> > > leaf: root=5
> > > > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> > > generation:
> > > > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> > > > >>>>>>>>>> [192088.449823] BTRFS error (device sdh):
> > > block=203510940835840 read
> > > > >>>>>>>>>> time tree block corruption detected
> > > > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt
> > > leaf: root=5
> > > > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> > > generation:
> > > > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> > > > >>>>>>>>>> [192088.462773] BTRFS error (device sdh):
> > > block=203510940835840 read
> > > > >>>>>>>>>> time tree block corruption detected
> > > > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt
> > > leaf: root=5
> > > > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> > > generation:
> > > > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> > > > >>>>>>>>>> [192088.468457] BTRFS error (device sdh):
> > > block=203510940835840 read
> > > > >>>>>>>>>> time tree block corruption detected
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Is there anything I should do about this, or should I
> > > just continue
> > > > >>>>>>>>>> using my array as normal?
> > > > >>>>>>>>>
> > > > >>>>>>>>> This is caused by older kernel underflow inode generation.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Or you can go safer, by manually locating the inode
> > > using its inode
> > > > >>>>>>>>> number (1311670), and copy it to some new location using
> > > previous
> > > > >>>>>>>>> working kernel, then delete the old file, copy the new
> > > one back to fix it.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Thanks,
> > > > >>>>>>>>> Qu
> > > > >>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Thank you!
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>
> > > >
> > >
> >
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-08-23 1:15 ` Tyler Richmond
@ 2020-08-23 1:51 ` Qu Wenruo
2020-08-23 2:31 ` Qu Wenruo
2020-08-23 2:32 ` Tyler Richmond
0 siblings, 2 replies; 56+ messages in thread
From: Qu Wenruo @ 2020-08-23 1:51 UTC (permalink / raw)
To: Tyler Richmond; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 19171 bytes --]
On 2020/8/23 上午9:15, Tyler Richmond wrote:
> Is my best bet just to downgrade the kernel and then try to delete the
> broken files? Or should I rebuild from scratch? Just don't know
> whether it's worth the time to try and figure this out or if the
> problems stem from the FS being too old and it's beyond trying to
> repair.
All invalid inode generations, should be able to be repaired by latest
btrfs-check.
If not, please provide the btrfs-image dump for us to determine what's
going wrong.
Thanks,
Qu
>
> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>>
>> I didn't check dmesg during the btrfs check, but that was the only
>> output during the rm -f before it was forced readonly. I just checked
>> dmesg for inode generation values, and there are a lot of them.
>>
>> https://pastebin.com/stZdN0ta
>> The dmesg output had 990 lines containing inode generation.
>>
>> However, these were at least later. I tried to do a btrfs balance
>> -mconvert raid1 and it failed with an I/O error. That is probably what
>> generated these specific errors, but maybe they were also happening
>> during the btrfs repair.
>>
>> The FS is ~45TB, but the btrfs-image -c9 failed anway with:
>> ERROR: either extent tree is corrupted or deprecated extent ref format
>> ERROR: create failed: -5
>>
>>
>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>
>>>
>>>
>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
>>>> Qu,
>>>>
>>>> Sorry to resurrect this thread, but I just ran into something that I
>>>> can't really just ignore. I've found a folder that is full of files
>>>> which I guess have been broken somehow. I found a backup and restored
>>>> them, but I want to delete this folder of broken files. But whenever I
>>>> try, the fs is forced into readonly mode again. I just finished another
>>>> btrfs check --repair but it didn't fix the problem.
>>>>
>>>> https://pastebin.com/eTV3s3fr
>>>
>>> Is that the full output?
>>>
>>> No inode generation bugs?
>>>>
>>>> I'm already on btrfs-progs v5.7. Any new suggestions?
>>>
>>> Strange.
>>>
>>> The detection and repair should have been merged into v5.5.
>>>
>>> If your fs is small enough, would you please provide the "btrfs-image
>>> -c9" dump?
>>>
>>> It would contain the filenames and directories names, but doesn't
>>> contain file contents.
>>>
>>> Thanks,
>>> Qu
>>>>
>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com
>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>
>>>> 5.6.1 also failed the same way. Here's the usage output. This is the
>>>> part where you see I've been using RAID5 haha
>>>>
>>>> WARNING: RAID56 detected, not implemented
>>>> Overall:
>>>> Device size: 60.03TiB
>>>> Device allocated: 98.06GiB
>>>> Device unallocated: 59.93TiB
>>>> Device missing: 0.00B
>>>> Used: 92.56GiB
>>>> Free (estimated): 0.00B (min: 8.00EiB)
>>>> Data ratio: 0.00
>>>> Metadata ratio: 2.00
>>>> Global reserve: 512.00MiB (used: 0.00B)
>>>> Multiple profiles: no
>>>>
>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
>>>> /dev/sdh 8.07TiB
>>>> /dev/sdf 8.07TiB
>>>> /dev/sdg 8.07TiB
>>>> /dev/sdd 8.07TiB
>>>> /dev/sdc 8.07TiB
>>>> /dev/sde 8.07TiB
>>>>
>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
>>>> /dev/sdh 34.00GiB
>>>> /dev/sdf 32.00GiB
>>>> /dev/sdg 32.00GiB
>>>>
>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
>>>> /dev/sdf 32.00MiB
>>>> /dev/sdg 32.00MiB
>>>>
>>>> Unallocated:
>>>> /dev/sdh 2.81TiB
>>>> /dev/sdf 2.81TiB
>>>> /dev/sdg 2.81TiB
>>>> /dev/sdd 1.03TiB
>>>> /dev/sdc 1.03TiB
>>>> /dev/sde 1.03TiB
>>>>
>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com
>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>> >
>>>> >
>>>> >
>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
>>>> > > If this is saying there's no extra space for metadata, is that why
>>>> > > adding more files often makes the system hang for 30-90s? Is there
>>>> > > anything I should do about that?
>>>> >
>>>> > I'm not sure about the hang though.
>>>> >
>>>> > It would be nice to give more info to diagnosis.
>>>> > The output of 'btrfs fi usage' is useful for space usage problem.
>>>> >
>>>> > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
>>>> > space in vanilla df command) space for btrfs.
>>>> >
>>>> > Thanks,
>>>> > Qu
>>>> >
>>>> > >
>>>> > > Thank you so much for all of your help. I love how flexible BTRFS is
>>>> > > but when things go wrong it's very hard for me to troubleshoot.
>>>> > >
>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com
>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>> > >>
>>>> > >>
>>>> > >>
>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
>>>> > >>> Something went wrong:
>>>> > >>>
>>>> > >>> Reinitialize checksum tree
>>>> > >>> Unable to find block group for 0
>>>> > >>> Unable to find block group for 0
>>>> > >>> Unable to find block group for 0
>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
>>>> > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
>>>> > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
>>>> > >>>
>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
>>>> > >>> Aborted
>>>> > >>
>>>> > >> This means no space for extra metadata...
>>>> > >>
>>>> > >> Anyway the csum tree problem shouldn't be a big thing, you
>>>> could leave
>>>> > >> it and call it a day.
>>>> > >>
>>>> > >> BTW, as long as btrfs check reports no extra problem for the inode
>>>> > >> generation, it should be pretty safe to use the fs.
>>>> > >>
>>>> > >> Thanks,
>>>> > >> Qu
>>>> > >>>
>>>> > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
>>>> > >>> available. I'll let that try overnight?
>>>> > >>>
>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>> > >>>>
>>>> > >>>>
>>>> > >>>>
>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
>>>> > >>>>> Thank you for helping. The end result of the scan was:
>>>> > >>>>>
>>>> > >>>>>
>>>> > >>>>> [1/7] checking root items
>>>> > >>>>> [2/7] checking extents
>>>> > >>>>> [3/7] checking free space cache
>>>> > >>>>> [4/7] checking fs roots
>>>> > >>>>
>>>> > >>>> Good news is, your fs is still mostly fine.
>>>> > >>>>
>>>> > >>>>> [5/7] checking only csums items (without verifying data)
>>>> > >>>>> there are no extents for csum range 0-69632
>>>> > >>>>> csum exists for 0-69632 but there is no extent record
>>>> > >>>>> ...
>>>> > >>>>> ...
>>>> > >>>>> there are no extents for csum range 946692096-946827264
>>>> > >>>>> csum exists for 946692096-946827264 but there is no extent
>>>> record
>>>> > >>>>> there are no extents for csum range 946831360-947912704
>>>> > >>>>> csum exists for 946831360-947912704 but there is no extent
>>>> record
>>>> > >>>>> ERROR: errors found in csum tree
>>>> > >>>>
>>>> > >>>> Only extent tree is corrupted.
>>>> > >>>>
>>>> > >>>> Normally btrfs check --init-csum-tree should be able to
>>>> handle it.
>>>> > >>>>
>>>> > >>>> But still, please be sure you're using the latest btrfs-progs
>>>> to fix it.
>>>> > >>>>
>>>> > >>>> Thanks,
>>>> > >>>> Qu
>>>> > >>>>
>>>> > >>>>> [6/7] checking root refs
>>>> > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>>> > >>>>> found 44157956026368 bytes used, error(s) found
>>>> > >>>>> total csum bytes: 42038602716
>>>> > >>>>> total tree bytes: 49688616960
>>>> > >>>>> total fs tree bytes: 1256427520
>>>> > >>>>> total extent tree bytes: 1709105152
>>>> > >>>>> btree space waste bytes: 3172727316
>>>> > >>>>> file data blocks allocated: 261625653436416
>>>> > >>>>> referenced 47477768499200
>>>> > >>>>>
>>>> > >>>>> What do I need to do to fix all of this?
>>>> > >>>>>
>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>> > >>>>>>
>>>> > >>>>>>
>>>> > >>>>>>
>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
>>>> > >>>>>>> Well, the repair doesn't look terribly successful.
>>>> > >>>>>>>
>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>> 6875841 found 6876224
>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>> 6875841 found 6876224
>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>> 6875841 found 6876224
>>>> > >>>>>>> Ignoring transid failure
>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>> item=84
>>>> > >>>>>>> parent level=1
>>>> > >>>>>>> child level=4
>>>> > >>>>>>
>>>> > >>>>>> This means there are more problems, not only the hash name
>>>> mismatch.
>>>> > >>>>>>
>>>> > >>>>>> This means the fs is already corrupted, the name hash is
>>>> just one
>>>> > >>>>>> unrelated symptom.
>>>> > >>>>>>
>>>> > >>>>>> The only good news is, btrfs-progs abort the transaction,
>>>> thus no
>>>> > >>>>>> further damage to the fs.
>>>> > >>>>>>
>>>> > >>>>>> Please run a plain btrfs-check to show what's the problem
>>>> first.
>>>> > >>>>>>
>>>> > >>>>>> Thanks,
>>>> > >>>>>> Qu
>>>> > >>>>>>
>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>> 6875841 found 6876224
>>>> > >>>>>>> Ignoring transid failure
>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>> item=84
>>>> > >>>>>>> parent level=1
>>>> > >>>>>>> child level=4
>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>> 6875841 found 6876224
>>>> > >>>>>>> Ignoring transid failure
>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>> item=84
>>>> > >>>>>>> parent level=1
>>>> > >>>>>>> child level=4
>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>> 6875841 found 6876224
>>>> > >>>>>>> Ignoring transid failure
>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>> item=84
>>>> > >>>>>>> parent level=1
>>>> > >>>>>>> child level=4
>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>> 6875841 found 6876224
>>>> > >>>>>>> Ignoring transid failure
>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>> item=84
>>>> > >>>>>>> parent level=1
>>>> > >>>>>>> child level=4
>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>> 6875841 found 6876224
>>>> > >>>>>>> Ignoring transid failure
>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>> item=84
>>>> > >>>>>>> parent level=1
>>>> > >>>>>>> child level=4
>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>> 6875841 found 6876224
>>>> > >>>>>>> Ignoring transid failure
>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>> item=84
>>>> > >>>>>>> parent level=1
>>>> > >>>>>>> child level=4
>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>> 6875841 found 6876224
>>>> > >>>>>>> Ignoring transid failure
>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>> item=84
>>>> > >>>>>>> parent level=1
>>>> > >>>>>>> child level=4
>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>> 6875841 found 6876224
>>>> > >>>>>>> Ignoring transid failure
>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>> item=84
>>>> > >>>>>>> parent level=1
>>>> > >>>>>>> child level=4
>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>> 6875841 found 6876224
>>>> > >>>>>>> Ignoring transid failure
>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>> item=84
>>>> > >>>>>>> parent level=1
>>>> > >>>>>>> child level=4
>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>> 6875841 found 6876224
>>>> > >>>>>>> Ignoring transid failure
>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>> item=84
>>>> > >>>>>>> parent level=1
>>>> > >>>>>>> child level=4
>>>> > >>>>>>> ERROR: failed to zero log tree: -17
>>>> > >>>>>>> ERROR: attempt to start transaction over already running one
>>>> > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>> 225049066086400 len 4096
>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>> 225049066094592 len 4096
>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>> 225049066102784 len 4096
>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>> 225049066131456 len 4096
>>>> > >>>>>>>
>>>> > >>>>>>> What is going on?
>>>> > >>>>>>>
>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond
>>>> <t.d.richmond@gmail.com <mailto:t.d.richmond@gmail.com>> wrote:
>>>> > >>>>>>>>
>>>> > >>>>>>>> Chris, I had used the correct mountpoint in the command.
>>>> I just edited
>>>> > >>>>>>>> it in the email to be /mountpoint for consistency.
>>>> > >>>>>>>>
>>>> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
>>>> > >>>>>>>>
>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>> > >>>>>>>>>
>>>> > >>>>>>>>>
>>>> > >>>>>>>>>
>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
>>>> > >>>>>>>>>> Hello,
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> I looked up this error and it basically says ask a
>>>> developer to
>>>> > >>>>>>>>>> determine if it's a false error or not. I just started
>>>> getting some
>>>> > >>>>>>>>>> slow response times, and looked at the dmesg log to
>>>> find a ton of
>>>> > >>>>>>>>>> these errors.
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt
>>>> leaf: root=5
>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>> generation:
>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>> > >>>>>>>>>> [192088.449823] BTRFS error (device sdh):
>>>> block=203510940835840 read
>>>> > >>>>>>>>>> time tree block corruption detected
>>>> > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt
>>>> leaf: root=5
>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>> generation:
>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>> > >>>>>>>>>> [192088.462773] BTRFS error (device sdh):
>>>> block=203510940835840 read
>>>> > >>>>>>>>>> time tree block corruption detected
>>>> > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt
>>>> leaf: root=5
>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>> generation:
>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>> > >>>>>>>>>> [192088.468457] BTRFS error (device sdh):
>>>> block=203510940835840 read
>>>> > >>>>>>>>>> time tree block corruption detected
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> Is there anything I should do about this, or should I
>>>> just continue
>>>> > >>>>>>>>>> using my array as normal?
>>>> > >>>>>>>>>
>>>> > >>>>>>>>> This is caused by older kernel underflow inode generation.
>>>> > >>>>>>>>>
>>>> > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
>>>> > >>>>>>>>>
>>>> > >>>>>>>>> Or you can go safer, by manually locating the inode
>>>> using its inode
>>>> > >>>>>>>>> number (1311670), and copy it to some new location using
>>>> previous
>>>> > >>>>>>>>> working kernel, then delete the old file, copy the new
>>>> one back to fix it.
>>>> > >>>>>>>>>
>>>> > >>>>>>>>> Thanks,
>>>> > >>>>>>>>> Qu
>>>> > >>>>>>>>>
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> Thank you!
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>
>>>> > >>>>>>
>>>> > >>>>
>>>> > >>
>>>> >
>>>>
>>>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-08-23 1:51 ` Qu Wenruo
@ 2020-08-23 2:31 ` Qu Wenruo
2020-08-23 2:49 ` Tyler Richmond
2020-08-23 2:32 ` Tyler Richmond
1 sibling, 1 reply; 56+ messages in thread
From: Qu Wenruo @ 2020-08-23 2:31 UTC (permalink / raw)
To: Tyler Richmond; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 20129 bytes --]
On 2020/8/23 上午9:51, Qu Wenruo wrote:
>
>
> On 2020/8/23 上午9:15, Tyler Richmond wrote:
>> Is my best bet just to downgrade the kernel and then try to delete the
>> broken files? Or should I rebuild from scratch? Just don't know
>> whether it's worth the time to try and figure this out or if the
>> problems stem from the FS being too old and it's beyond trying to
>> repair.
>
> All invalid inode generations, should be able to be repaired by latest
> btrfs-check.
>
> If not, please provide the btrfs-image dump for us to determine what's
> going wrong.
>
> Thanks,
> Qu
>>
>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>>>
>>> I didn't check dmesg during the btrfs check, but that was the only
>>> output during the rm -f before it was forced readonly. I just checked
>>> dmesg for inode generation values, and there are a lot of them.
>>>
>>> https://pastebin.com/stZdN0ta
>>> The dmesg output had 990 lines containing inode generation.
>>>
>>> However, these were at least later. I tried to do a btrfs balance
>>> -mconvert raid1 and it failed with an I/O error. That is probably what
>>> generated these specific errors, but maybe they were also happening
>>> during the btrfs repair.
>>>
>>> The FS is ~45TB, but the btrfs-image -c9 failed anway with:
>>> ERROR: either extent tree is corrupted or deprecated extent ref format
>>> ERROR: create failed: -5
Oh, forgot this part.
This means you have v0 ref?!
Then the fs is too old, no progs/kernel support after all.
In that case, please rollback to the last working kernel and copy your data.
In fact, that v0 ref should only be in the code base for several weeks
before 2010, thus it's really too old.
The good news is, with tree-checker, we should never experience such
too-old-to-be-usable problem (at least I hope so)
Thanks,
Qu
>>>
>>>
>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>
>>>>
>>>>
>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
>>>>> Qu,
>>>>>
>>>>> Sorry to resurrect this thread, but I just ran into something that I
>>>>> can't really just ignore. I've found a folder that is full of files
>>>>> which I guess have been broken somehow. I found a backup and restored
>>>>> them, but I want to delete this folder of broken files. But whenever I
>>>>> try, the fs is forced into readonly mode again. I just finished another
>>>>> btrfs check --repair but it didn't fix the problem.
>>>>>
>>>>> https://pastebin.com/eTV3s3fr
>>>>
>>>> Is that the full output?
>>>>
>>>> No inode generation bugs?
>>>>>
>>>>> I'm already on btrfs-progs v5.7. Any new suggestions?
>>>>
>>>> Strange.
>>>>
>>>> The detection and repair should have been merged into v5.5.
>>>>
>>>> If your fs is small enough, would you please provide the "btrfs-image
>>>> -c9" dump?
>>>>
>>>> It would contain the filenames and directories names, but doesn't
>>>> contain file contents.
>>>>
>>>> Thanks,
>>>> Qu
>>>>>
>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com
>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>
>>>>> 5.6.1 also failed the same way. Here's the usage output. This is the
>>>>> part where you see I've been using RAID5 haha
>>>>>
>>>>> WARNING: RAID56 detected, not implemented
>>>>> Overall:
>>>>> Device size: 60.03TiB
>>>>> Device allocated: 98.06GiB
>>>>> Device unallocated: 59.93TiB
>>>>> Device missing: 0.00B
>>>>> Used: 92.56GiB
>>>>> Free (estimated): 0.00B (min: 8.00EiB)
>>>>> Data ratio: 0.00
>>>>> Metadata ratio: 2.00
>>>>> Global reserve: 512.00MiB (used: 0.00B)
>>>>> Multiple profiles: no
>>>>>
>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
>>>>> /dev/sdh 8.07TiB
>>>>> /dev/sdf 8.07TiB
>>>>> /dev/sdg 8.07TiB
>>>>> /dev/sdd 8.07TiB
>>>>> /dev/sdc 8.07TiB
>>>>> /dev/sde 8.07TiB
>>>>>
>>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
>>>>> /dev/sdh 34.00GiB
>>>>> /dev/sdf 32.00GiB
>>>>> /dev/sdg 32.00GiB
>>>>>
>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
>>>>> /dev/sdf 32.00MiB
>>>>> /dev/sdg 32.00MiB
>>>>>
>>>>> Unallocated:
>>>>> /dev/sdh 2.81TiB
>>>>> /dev/sdf 2.81TiB
>>>>> /dev/sdg 2.81TiB
>>>>> /dev/sdd 1.03TiB
>>>>> /dev/sdc 1.03TiB
>>>>> /dev/sde 1.03TiB
>>>>>
>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com
>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>> >
>>>>> >
>>>>> >
>>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
>>>>> > > If this is saying there's no extra space for metadata, is that why
>>>>> > > adding more files often makes the system hang for 30-90s? Is there
>>>>> > > anything I should do about that?
>>>>> >
>>>>> > I'm not sure about the hang though.
>>>>> >
>>>>> > It would be nice to give more info to diagnosis.
>>>>> > The output of 'btrfs fi usage' is useful for space usage problem.
>>>>> >
>>>>> > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
>>>>> > space in vanilla df command) space for btrfs.
>>>>> >
>>>>> > Thanks,
>>>>> > Qu
>>>>> >
>>>>> > >
>>>>> > > Thank you so much for all of your help. I love how flexible BTRFS is
>>>>> > > but when things go wrong it's very hard for me to troubleshoot.
>>>>> > >
>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com
>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>> > >>
>>>>> > >>
>>>>> > >>
>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
>>>>> > >>> Something went wrong:
>>>>> > >>>
>>>>> > >>> Reinitialize checksum tree
>>>>> > >>> Unable to find block group for 0
>>>>> > >>> Unable to find block group for 0
>>>>> > >>> Unable to find block group for 0
>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
>>>>> > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
>>>>> > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
>>>>> > >>>
>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
>>>>> > >>> Aborted
>>>>> > >>
>>>>> > >> This means no space for extra metadata...
>>>>> > >>
>>>>> > >> Anyway the csum tree problem shouldn't be a big thing, you
>>>>> could leave
>>>>> > >> it and call it a day.
>>>>> > >>
>>>>> > >> BTW, as long as btrfs check reports no extra problem for the inode
>>>>> > >> generation, it should be pretty safe to use the fs.
>>>>> > >>
>>>>> > >> Thanks,
>>>>> > >> Qu
>>>>> > >>>
>>>>> > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
>>>>> > >>> available. I'll let that try overnight?
>>>>> > >>>
>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>> > >>>>
>>>>> > >>>>
>>>>> > >>>>
>>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
>>>>> > >>>>> Thank you for helping. The end result of the scan was:
>>>>> > >>>>>
>>>>> > >>>>>
>>>>> > >>>>> [1/7] checking root items
>>>>> > >>>>> [2/7] checking extents
>>>>> > >>>>> [3/7] checking free space cache
>>>>> > >>>>> [4/7] checking fs roots
>>>>> > >>>>
>>>>> > >>>> Good news is, your fs is still mostly fine.
>>>>> > >>>>
>>>>> > >>>>> [5/7] checking only csums items (without verifying data)
>>>>> > >>>>> there are no extents for csum range 0-69632
>>>>> > >>>>> csum exists for 0-69632 but there is no extent record
>>>>> > >>>>> ...
>>>>> > >>>>> ...
>>>>> > >>>>> there are no extents for csum range 946692096-946827264
>>>>> > >>>>> csum exists for 946692096-946827264 but there is no extent
>>>>> record
>>>>> > >>>>> there are no extents for csum range 946831360-947912704
>>>>> > >>>>> csum exists for 946831360-947912704 but there is no extent
>>>>> record
>>>>> > >>>>> ERROR: errors found in csum tree
>>>>> > >>>>
>>>>> > >>>> Only extent tree is corrupted.
>>>>> > >>>>
>>>>> > >>>> Normally btrfs check --init-csum-tree should be able to
>>>>> handle it.
>>>>> > >>>>
>>>>> > >>>> But still, please be sure you're using the latest btrfs-progs
>>>>> to fix it.
>>>>> > >>>>
>>>>> > >>>> Thanks,
>>>>> > >>>> Qu
>>>>> > >>>>
>>>>> > >>>>> [6/7] checking root refs
>>>>> > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>>>> > >>>>> found 44157956026368 bytes used, error(s) found
>>>>> > >>>>> total csum bytes: 42038602716
>>>>> > >>>>> total tree bytes: 49688616960
>>>>> > >>>>> total fs tree bytes: 1256427520
>>>>> > >>>>> total extent tree bytes: 1709105152
>>>>> > >>>>> btree space waste bytes: 3172727316
>>>>> > >>>>> file data blocks allocated: 261625653436416
>>>>> > >>>>> referenced 47477768499200
>>>>> > >>>>>
>>>>> > >>>>> What do I need to do to fix all of this?
>>>>> > >>>>>
>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>> > >>>>>>
>>>>> > >>>>>>
>>>>> > >>>>>>
>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
>>>>> > >>>>>>> Well, the repair doesn't look terribly successful.
>>>>> > >>>>>>>
>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>> 6875841 found 6876224
>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>> 6875841 found 6876224
>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>> 6875841 found 6876224
>>>>> > >>>>>>> Ignoring transid failure
>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>> item=84
>>>>> > >>>>>>> parent level=1
>>>>> > >>>>>>> child level=4
>>>>> > >>>>>>
>>>>> > >>>>>> This means there are more problems, not only the hash name
>>>>> mismatch.
>>>>> > >>>>>>
>>>>> > >>>>>> This means the fs is already corrupted, the name hash is
>>>>> just one
>>>>> > >>>>>> unrelated symptom.
>>>>> > >>>>>>
>>>>> > >>>>>> The only good news is, btrfs-progs abort the transaction,
>>>>> thus no
>>>>> > >>>>>> further damage to the fs.
>>>>> > >>>>>>
>>>>> > >>>>>> Please run a plain btrfs-check to show what's the problem
>>>>> first.
>>>>> > >>>>>>
>>>>> > >>>>>> Thanks,
>>>>> > >>>>>> Qu
>>>>> > >>>>>>
>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>> 6875841 found 6876224
>>>>> > >>>>>>> Ignoring transid failure
>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>> item=84
>>>>> > >>>>>>> parent level=1
>>>>> > >>>>>>> child level=4
>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>> 6875841 found 6876224
>>>>> > >>>>>>> Ignoring transid failure
>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>> item=84
>>>>> > >>>>>>> parent level=1
>>>>> > >>>>>>> child level=4
>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>> 6875841 found 6876224
>>>>> > >>>>>>> Ignoring transid failure
>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>> item=84
>>>>> > >>>>>>> parent level=1
>>>>> > >>>>>>> child level=4
>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>> 6875841 found 6876224
>>>>> > >>>>>>> Ignoring transid failure
>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>> item=84
>>>>> > >>>>>>> parent level=1
>>>>> > >>>>>>> child level=4
>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>> 6875841 found 6876224
>>>>> > >>>>>>> Ignoring transid failure
>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>> item=84
>>>>> > >>>>>>> parent level=1
>>>>> > >>>>>>> child level=4
>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>> 6875841 found 6876224
>>>>> > >>>>>>> Ignoring transid failure
>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>> item=84
>>>>> > >>>>>>> parent level=1
>>>>> > >>>>>>> child level=4
>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>> 6875841 found 6876224
>>>>> > >>>>>>> Ignoring transid failure
>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>> item=84
>>>>> > >>>>>>> parent level=1
>>>>> > >>>>>>> child level=4
>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>> 6875841 found 6876224
>>>>> > >>>>>>> Ignoring transid failure
>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>> item=84
>>>>> > >>>>>>> parent level=1
>>>>> > >>>>>>> child level=4
>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>> 6875841 found 6876224
>>>>> > >>>>>>> Ignoring transid failure
>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>> item=84
>>>>> > >>>>>>> parent level=1
>>>>> > >>>>>>> child level=4
>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>> 6875841 found 6876224
>>>>> > >>>>>>> Ignoring transid failure
>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>> item=84
>>>>> > >>>>>>> parent level=1
>>>>> > >>>>>>> child level=4
>>>>> > >>>>>>> ERROR: failed to zero log tree: -17
>>>>> > >>>>>>> ERROR: attempt to start transaction over already running one
>>>>> > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>> 225049066086400 len 4096
>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>> 225049066094592 len 4096
>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>> 225049066102784 len 4096
>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>> 225049066131456 len 4096
>>>>> > >>>>>>>
>>>>> > >>>>>>> What is going on?
>>>>> > >>>>>>>
>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond
>>>>> <t.d.richmond@gmail.com <mailto:t.d.richmond@gmail.com>> wrote:
>>>>> > >>>>>>>>
>>>>> > >>>>>>>> Chris, I had used the correct mountpoint in the command.
>>>>> I just edited
>>>>> > >>>>>>>> it in the email to be /mountpoint for consistency.
>>>>> > >>>>>>>>
>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
>>>>> > >>>>>>>>
>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
>>>>> > >>>>>>>>>> Hello,
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> I looked up this error and it basically says ask a
>>>>> developer to
>>>>> > >>>>>>>>>> determine if it's a false error or not. I just started
>>>>> getting some
>>>>> > >>>>>>>>>> slow response times, and looked at the dmesg log to
>>>>> find a ton of
>>>>> > >>>>>>>>>> these errors.
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt
>>>>> leaf: root=5
>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>> generation:
>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>> > >>>>>>>>>> [192088.449823] BTRFS error (device sdh):
>>>>> block=203510940835840 read
>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt
>>>>> leaf: root=5
>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>> generation:
>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>> > >>>>>>>>>> [192088.462773] BTRFS error (device sdh):
>>>>> block=203510940835840 read
>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt
>>>>> leaf: root=5
>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>> generation:
>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>> > >>>>>>>>>> [192088.468457] BTRFS error (device sdh):
>>>>> block=203510940835840 read
>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> Is there anything I should do about this, or should I
>>>>> just continue
>>>>> > >>>>>>>>>> using my array as normal?
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>> This is caused by older kernel underflow inode generation.
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>> Or you can go safer, by manually locating the inode
>>>>> using its inode
>>>>> > >>>>>>>>> number (1311670), and copy it to some new location using
>>>>> previous
>>>>> > >>>>>>>>> working kernel, then delete the old file, copy the new
>>>>> one back to fix it.
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>> Thanks,
>>>>> > >>>>>>>>> Qu
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> Thank you!
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>
>>>>> > >>>>>>
>>>>> > >>>>
>>>>> > >>
>>>>> >
>>>>>
>>>>
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-08-23 1:51 ` Qu Wenruo
2020-08-23 2:31 ` Qu Wenruo
@ 2020-08-23 2:32 ` Tyler Richmond
1 sibling, 0 replies; 56+ messages in thread
From: Tyler Richmond @ 2020-08-23 2:32 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Btrfs BTRFS
OK, I'll get that. btrfs-image fails without the -w flag because of
tree extent corruption. If I use -w and -ss, is that fine?
On Sat, Aug 22, 2020 at 9:51 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2020/8/23 上午9:15, Tyler Richmond wrote:
> > Is my best bet just to downgrade the kernel and then try to delete the
> > broken files? Or should I rebuild from scratch? Just don't know
> > whether it's worth the time to try and figure this out or if the
> > problems stem from the FS being too old and it's beyond trying to
> > repair.
>
> All invalid inode generations, should be able to be repaired by latest
> btrfs-check.
>
> If not, please provide the btrfs-image dump for us to determine what's
> going wrong.
>
> Thanks,
> Qu
> >
> > On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
> >>
> >> I didn't check dmesg during the btrfs check, but that was the only
> >> output during the rm -f before it was forced readonly. I just checked
> >> dmesg for inode generation values, and there are a lot of them.
> >>
> >> https://pastebin.com/stZdN0ta
> >> The dmesg output had 990 lines containing inode generation.
> >>
> >> However, these were at least later. I tried to do a btrfs balance
> >> -mconvert raid1 and it failed with an I/O error. That is probably what
> >> generated these specific errors, but maybe they were also happening
> >> during the btrfs repair.
> >>
> >> The FS is ~45TB, but the btrfs-image -c9 failed anway with:
> >> ERROR: either extent tree is corrupted or deprecated extent ref format
> >> ERROR: create failed: -5
> >>
> >>
> >> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>
> >>>
> >>>
> >>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
> >>>> Qu,
> >>>>
> >>>> Sorry to resurrect this thread, but I just ran into something that I
> >>>> can't really just ignore. I've found a folder that is full of files
> >>>> which I guess have been broken somehow. I found a backup and restored
> >>>> them, but I want to delete this folder of broken files. But whenever I
> >>>> try, the fs is forced into readonly mode again. I just finished another
> >>>> btrfs check --repair but it didn't fix the problem.
> >>>>
> >>>> https://pastebin.com/eTV3s3fr
> >>>
> >>> Is that the full output?
> >>>
> >>> No inode generation bugs?
> >>>>
> >>>> I'm already on btrfs-progs v5.7. Any new suggestions?
> >>>
> >>> Strange.
> >>>
> >>> The detection and repair should have been merged into v5.5.
> >>>
> >>> If your fs is small enough, would you please provide the "btrfs-image
> >>> -c9" dump?
> >>>
> >>> It would contain the filenames and directories names, but doesn't
> >>> contain file contents.
> >>>
> >>> Thanks,
> >>> Qu
> >>>>
> >>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com
> >>>> <mailto:t.d.richmond@gmail.com>> wrote:
> >>>>
> >>>> 5.6.1 also failed the same way. Here's the usage output. This is the
> >>>> part where you see I've been using RAID5 haha
> >>>>
> >>>> WARNING: RAID56 detected, not implemented
> >>>> Overall:
> >>>> Device size: 60.03TiB
> >>>> Device allocated: 98.06GiB
> >>>> Device unallocated: 59.93TiB
> >>>> Device missing: 0.00B
> >>>> Used: 92.56GiB
> >>>> Free (estimated): 0.00B (min: 8.00EiB)
> >>>> Data ratio: 0.00
> >>>> Metadata ratio: 2.00
> >>>> Global reserve: 512.00MiB (used: 0.00B)
> >>>> Multiple profiles: no
> >>>>
> >>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
> >>>> /dev/sdh 8.07TiB
> >>>> /dev/sdf 8.07TiB
> >>>> /dev/sdg 8.07TiB
> >>>> /dev/sdd 8.07TiB
> >>>> /dev/sdc 8.07TiB
> >>>> /dev/sde 8.07TiB
> >>>>
> >>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
> >>>> /dev/sdh 34.00GiB
> >>>> /dev/sdf 32.00GiB
> >>>> /dev/sdg 32.00GiB
> >>>>
> >>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
> >>>> /dev/sdf 32.00MiB
> >>>> /dev/sdg 32.00MiB
> >>>>
> >>>> Unallocated:
> >>>> /dev/sdh 2.81TiB
> >>>> /dev/sdf 2.81TiB
> >>>> /dev/sdg 2.81TiB
> >>>> /dev/sdd 1.03TiB
> >>>> /dev/sdc 1.03TiB
> >>>> /dev/sde 1.03TiB
> >>>>
> >>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> >>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>> >
> >>>> >
> >>>> >
> >>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
> >>>> > > If this is saying there's no extra space for metadata, is that why
> >>>> > > adding more files often makes the system hang for 30-90s? Is there
> >>>> > > anything I should do about that?
> >>>> >
> >>>> > I'm not sure about the hang though.
> >>>> >
> >>>> > It would be nice to give more info to diagnosis.
> >>>> > The output of 'btrfs fi usage' is useful for space usage problem.
> >>>> >
> >>>> > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
> >>>> > space in vanilla df command) space for btrfs.
> >>>> >
> >>>> > Thanks,
> >>>> > Qu
> >>>> >
> >>>> > >
> >>>> > > Thank you so much for all of your help. I love how flexible BTRFS is
> >>>> > > but when things go wrong it's very hard for me to troubleshoot.
> >>>> > >
> >>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> >>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>> > >>
> >>>> > >>
> >>>> > >>
> >>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
> >>>> > >>> Something went wrong:
> >>>> > >>>
> >>>> > >>> Reinitialize checksum tree
> >>>> > >>> Unable to find block group for 0
> >>>> > >>> Unable to find block group for 0
> >>>> > >>> Unable to find block group for 0
> >>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
> >>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
> >>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
> >>>> > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
> >>>> > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
> >>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
> >>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
> >>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
> >>>> > >>>
> >>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
> >>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
> >>>> > >>> Aborted
> >>>> > >>
> >>>> > >> This means no space for extra metadata...
> >>>> > >>
> >>>> > >> Anyway the csum tree problem shouldn't be a big thing, you
> >>>> could leave
> >>>> > >> it and call it a day.
> >>>> > >>
> >>>> > >> BTW, as long as btrfs check reports no extra problem for the inode
> >>>> > >> generation, it should be pretty safe to use the fs.
> >>>> > >>
> >>>> > >> Thanks,
> >>>> > >> Qu
> >>>> > >>>
> >>>> > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
> >>>> > >>> available. I'll let that try overnight?
> >>>> > >>>
> >>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
> >>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>> > >>>>
> >>>> > >>>>
> >>>> > >>>>
> >>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
> >>>> > >>>>> Thank you for helping. The end result of the scan was:
> >>>> > >>>>>
> >>>> > >>>>>
> >>>> > >>>>> [1/7] checking root items
> >>>> > >>>>> [2/7] checking extents
> >>>> > >>>>> [3/7] checking free space cache
> >>>> > >>>>> [4/7] checking fs roots
> >>>> > >>>>
> >>>> > >>>> Good news is, your fs is still mostly fine.
> >>>> > >>>>
> >>>> > >>>>> [5/7] checking only csums items (without verifying data)
> >>>> > >>>>> there are no extents for csum range 0-69632
> >>>> > >>>>> csum exists for 0-69632 but there is no extent record
> >>>> > >>>>> ...
> >>>> > >>>>> ...
> >>>> > >>>>> there are no extents for csum range 946692096-946827264
> >>>> > >>>>> csum exists for 946692096-946827264 but there is no extent
> >>>> record
> >>>> > >>>>> there are no extents for csum range 946831360-947912704
> >>>> > >>>>> csum exists for 946831360-947912704 but there is no extent
> >>>> record
> >>>> > >>>>> ERROR: errors found in csum tree
> >>>> > >>>>
> >>>> > >>>> Only extent tree is corrupted.
> >>>> > >>>>
> >>>> > >>>> Normally btrfs check --init-csum-tree should be able to
> >>>> handle it.
> >>>> > >>>>
> >>>> > >>>> But still, please be sure you're using the latest btrfs-progs
> >>>> to fix it.
> >>>> > >>>>
> >>>> > >>>> Thanks,
> >>>> > >>>> Qu
> >>>> > >>>>
> >>>> > >>>>> [6/7] checking root refs
> >>>> > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
> >>>> > >>>>> found 44157956026368 bytes used, error(s) found
> >>>> > >>>>> total csum bytes: 42038602716
> >>>> > >>>>> total tree bytes: 49688616960
> >>>> > >>>>> total fs tree bytes: 1256427520
> >>>> > >>>>> total extent tree bytes: 1709105152
> >>>> > >>>>> btree space waste bytes: 3172727316
> >>>> > >>>>> file data blocks allocated: 261625653436416
> >>>> > >>>>> referenced 47477768499200
> >>>> > >>>>>
> >>>> > >>>>> What do I need to do to fix all of this?
> >>>> > >>>>>
> >>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
> >>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>> > >>>>>>
> >>>> > >>>>>>
> >>>> > >>>>>>
> >>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
> >>>> > >>>>>>> Well, the repair doesn't look terribly successful.
> >>>> > >>>>>>>
> >>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>> 6875841 found 6876224
> >>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>> 6875841 found 6876224
> >>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>> 6875841 found 6876224
> >>>> > >>>>>>> Ignoring transid failure
> >>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>> item=84
> >>>> > >>>>>>> parent level=1
> >>>> > >>>>>>> child level=4
> >>>> > >>>>>>
> >>>> > >>>>>> This means there are more problems, not only the hash name
> >>>> mismatch.
> >>>> > >>>>>>
> >>>> > >>>>>> This means the fs is already corrupted, the name hash is
> >>>> just one
> >>>> > >>>>>> unrelated symptom.
> >>>> > >>>>>>
> >>>> > >>>>>> The only good news is, btrfs-progs abort the transaction,
> >>>> thus no
> >>>> > >>>>>> further damage to the fs.
> >>>> > >>>>>>
> >>>> > >>>>>> Please run a plain btrfs-check to show what's the problem
> >>>> first.
> >>>> > >>>>>>
> >>>> > >>>>>> Thanks,
> >>>> > >>>>>> Qu
> >>>> > >>>>>>
> >>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>> 6875841 found 6876224
> >>>> > >>>>>>> Ignoring transid failure
> >>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>> item=84
> >>>> > >>>>>>> parent level=1
> >>>> > >>>>>>> child level=4
> >>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>> 6875841 found 6876224
> >>>> > >>>>>>> Ignoring transid failure
> >>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>> item=84
> >>>> > >>>>>>> parent level=1
> >>>> > >>>>>>> child level=4
> >>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>> 6875841 found 6876224
> >>>> > >>>>>>> Ignoring transid failure
> >>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>> item=84
> >>>> > >>>>>>> parent level=1
> >>>> > >>>>>>> child level=4
> >>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>> 6875841 found 6876224
> >>>> > >>>>>>> Ignoring transid failure
> >>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>> item=84
> >>>> > >>>>>>> parent level=1
> >>>> > >>>>>>> child level=4
> >>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>> 6875841 found 6876224
> >>>> > >>>>>>> Ignoring transid failure
> >>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>> item=84
> >>>> > >>>>>>> parent level=1
> >>>> > >>>>>>> child level=4
> >>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>> 6875841 found 6876224
> >>>> > >>>>>>> Ignoring transid failure
> >>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>> item=84
> >>>> > >>>>>>> parent level=1
> >>>> > >>>>>>> child level=4
> >>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>> 6875841 found 6876224
> >>>> > >>>>>>> Ignoring transid failure
> >>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>> item=84
> >>>> > >>>>>>> parent level=1
> >>>> > >>>>>>> child level=4
> >>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>> 6875841 found 6876224
> >>>> > >>>>>>> Ignoring transid failure
> >>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>> item=84
> >>>> > >>>>>>> parent level=1
> >>>> > >>>>>>> child level=4
> >>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>> 6875841 found 6876224
> >>>> > >>>>>>> Ignoring transid failure
> >>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>> item=84
> >>>> > >>>>>>> parent level=1
> >>>> > >>>>>>> child level=4
> >>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>> 6875841 found 6876224
> >>>> > >>>>>>> Ignoring transid failure
> >>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>> item=84
> >>>> > >>>>>>> parent level=1
> >>>> > >>>>>>> child level=4
> >>>> > >>>>>>> ERROR: failed to zero log tree: -17
> >>>> > >>>>>>> ERROR: attempt to start transaction over already running one
> >>>> > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
> >>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> >>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> >>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>> 225049066086400 len 4096
> >>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> >>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> >>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>> 225049066094592 len 4096
> >>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> >>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> >>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>> 225049066102784 len 4096
> >>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> >>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> >>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>> 225049066131456 len 4096
> >>>> > >>>>>>>
> >>>> > >>>>>>> What is going on?
> >>>> > >>>>>>>
> >>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond
> >>>> <t.d.richmond@gmail.com <mailto:t.d.richmond@gmail.com>> wrote:
> >>>> > >>>>>>>>
> >>>> > >>>>>>>> Chris, I had used the correct mountpoint in the command.
> >>>> I just edited
> >>>> > >>>>>>>> it in the email to be /mountpoint for consistency.
> >>>> > >>>>>>>>
> >>>> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
> >>>> > >>>>>>>>
> >>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
> >>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>> > >>>>>>>>>
> >>>> > >>>>>>>>>
> >>>> > >>>>>>>>>
> >>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
> >>>> > >>>>>>>>>> Hello,
> >>>> > >>>>>>>>>>
> >>>> > >>>>>>>>>> I looked up this error and it basically says ask a
> >>>> developer to
> >>>> > >>>>>>>>>> determine if it's a false error or not. I just started
> >>>> getting some
> >>>> > >>>>>>>>>> slow response times, and looked at the dmesg log to
> >>>> find a ton of
> >>>> > >>>>>>>>>> these errors.
> >>>> > >>>>>>>>>>
> >>>> > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt
> >>>> leaf: root=5
> >>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> >>>> generation:
> >>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>> > >>>>>>>>>> [192088.449823] BTRFS error (device sdh):
> >>>> block=203510940835840 read
> >>>> > >>>>>>>>>> time tree block corruption detected
> >>>> > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt
> >>>> leaf: root=5
> >>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> >>>> generation:
> >>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>> > >>>>>>>>>> [192088.462773] BTRFS error (device sdh):
> >>>> block=203510940835840 read
> >>>> > >>>>>>>>>> time tree block corruption detected
> >>>> > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt
> >>>> leaf: root=5
> >>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> >>>> generation:
> >>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>> > >>>>>>>>>> [192088.468457] BTRFS error (device sdh):
> >>>> block=203510940835840 read
> >>>> > >>>>>>>>>> time tree block corruption detected
> >>>> > >>>>>>>>>>
> >>>> > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
> >>>> > >>>>>>>>>>
> >>>> > >>>>>>>>>> Is there anything I should do about this, or should I
> >>>> just continue
> >>>> > >>>>>>>>>> using my array as normal?
> >>>> > >>>>>>>>>
> >>>> > >>>>>>>>> This is caused by older kernel underflow inode generation.
> >>>> > >>>>>>>>>
> >>>> > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
> >>>> > >>>>>>>>>
> >>>> > >>>>>>>>> Or you can go safer, by manually locating the inode
> >>>> using its inode
> >>>> > >>>>>>>>> number (1311670), and copy it to some new location using
> >>>> previous
> >>>> > >>>>>>>>> working kernel, then delete the old file, copy the new
> >>>> one back to fix it.
> >>>> > >>>>>>>>>
> >>>> > >>>>>>>>> Thanks,
> >>>> > >>>>>>>>> Qu
> >>>> > >>>>>>>>>
> >>>> > >>>>>>>>>>
> >>>> > >>>>>>>>>> Thank you!
> >>>> > >>>>>>>>>>
> >>>> > >>>>>>>>>
> >>>> > >>>>>>
> >>>> > >>>>
> >>>> > >>
> >>>> >
> >>>>
> >>>
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-08-23 2:31 ` Qu Wenruo
@ 2020-08-23 2:49 ` Tyler Richmond
2020-08-23 4:28 ` Qu Wenruo
0 siblings, 1 reply; 56+ messages in thread
From: Tyler Richmond @ 2020-08-23 2:49 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Btrfs BTRFS
Well, I can guarantee that I didn't create this fs before 2015 (just
checked the order confirmation from when I bought the server), but I
may have just used whatever was in the Ubuntu package manager at the
time. So maybe I don't have a v0 ref?
On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2020/8/23 上午9:51, Qu Wenruo wrote:
> >
> >
> > On 2020/8/23 上午9:15, Tyler Richmond wrote:
> >> Is my best bet just to downgrade the kernel and then try to delete the
> >> broken files? Or should I rebuild from scratch? Just don't know
> >> whether it's worth the time to try and figure this out or if the
> >> problems stem from the FS being too old and it's beyond trying to
> >> repair.
> >
> > All invalid inode generations, should be able to be repaired by latest
> > btrfs-check.
> >
> > If not, please provide the btrfs-image dump for us to determine what's
> > going wrong.
> >
> > Thanks,
> > Qu
> >>
> >> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
> >>>
> >>> I didn't check dmesg during the btrfs check, but that was the only
> >>> output during the rm -f before it was forced readonly. I just checked
> >>> dmesg for inode generation values, and there are a lot of them.
> >>>
> >>> https://pastebin.com/stZdN0ta
> >>> The dmesg output had 990 lines containing inode generation.
> >>>
> >>> However, these were at least later. I tried to do a btrfs balance
> >>> -mconvert raid1 and it failed with an I/O error. That is probably what
> >>> generated these specific errors, but maybe they were also happening
> >>> during the btrfs repair.
> >>>
> >>> The FS is ~45TB, but the btrfs-image -c9 failed anway with:
> >>> ERROR: either extent tree is corrupted or deprecated extent ref format
> >>> ERROR: create failed: -5
>
> Oh, forgot this part.
>
> This means you have v0 ref?!
>
> Then the fs is too old, no progs/kernel support after all.
>
> In that case, please rollback to the last working kernel and copy your data.
>
> In fact, that v0 ref should only be in the code base for several weeks
> before 2010, thus it's really too old.
>
> The good news is, with tree-checker, we should never experience such
> too-old-to-be-usable problem (at least I hope so)
>
> Thanks,
> Qu
>
> >>>
> >>>
> >>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
> >>>>> Qu,
> >>>>>
> >>>>> Sorry to resurrect this thread, but I just ran into something that I
> >>>>> can't really just ignore. I've found a folder that is full of files
> >>>>> which I guess have been broken somehow. I found a backup and restored
> >>>>> them, but I want to delete this folder of broken files. But whenever I
> >>>>> try, the fs is forced into readonly mode again. I just finished another
> >>>>> btrfs check --repair but it didn't fix the problem.
> >>>>>
> >>>>> https://pastebin.com/eTV3s3fr
> >>>>
> >>>> Is that the full output?
> >>>>
> >>>> No inode generation bugs?
> >>>>>
> >>>>> I'm already on btrfs-progs v5.7. Any new suggestions?
> >>>>
> >>>> Strange.
> >>>>
> >>>> The detection and repair should have been merged into v5.5.
> >>>>
> >>>> If your fs is small enough, would you please provide the "btrfs-image
> >>>> -c9" dump?
> >>>>
> >>>> It would contain the filenames and directories names, but doesn't
> >>>> contain file contents.
> >>>>
> >>>> Thanks,
> >>>> Qu
> >>>>>
> >>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com
> >>>>> <mailto:t.d.richmond@gmail.com>> wrote:
> >>>>>
> >>>>> 5.6.1 also failed the same way. Here's the usage output. This is the
> >>>>> part where you see I've been using RAID5 haha
> >>>>>
> >>>>> WARNING: RAID56 detected, not implemented
> >>>>> Overall:
> >>>>> Device size: 60.03TiB
> >>>>> Device allocated: 98.06GiB
> >>>>> Device unallocated: 59.93TiB
> >>>>> Device missing: 0.00B
> >>>>> Used: 92.56GiB
> >>>>> Free (estimated): 0.00B (min: 8.00EiB)
> >>>>> Data ratio: 0.00
> >>>>> Metadata ratio: 2.00
> >>>>> Global reserve: 512.00MiB (used: 0.00B)
> >>>>> Multiple profiles: no
> >>>>>
> >>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
> >>>>> /dev/sdh 8.07TiB
> >>>>> /dev/sdf 8.07TiB
> >>>>> /dev/sdg 8.07TiB
> >>>>> /dev/sdd 8.07TiB
> >>>>> /dev/sdc 8.07TiB
> >>>>> /dev/sde 8.07TiB
> >>>>>
> >>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
> >>>>> /dev/sdh 34.00GiB
> >>>>> /dev/sdf 32.00GiB
> >>>>> /dev/sdg 32.00GiB
> >>>>>
> >>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
> >>>>> /dev/sdf 32.00MiB
> >>>>> /dev/sdg 32.00MiB
> >>>>>
> >>>>> Unallocated:
> >>>>> /dev/sdh 2.81TiB
> >>>>> /dev/sdf 2.81TiB
> >>>>> /dev/sdg 2.81TiB
> >>>>> /dev/sdd 1.03TiB
> >>>>> /dev/sdc 1.03TiB
> >>>>> /dev/sde 1.03TiB
> >>>>>
> >>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> >>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
> >>>>> > > If this is saying there's no extra space for metadata, is that why
> >>>>> > > adding more files often makes the system hang for 30-90s? Is there
> >>>>> > > anything I should do about that?
> >>>>> >
> >>>>> > I'm not sure about the hang though.
> >>>>> >
> >>>>> > It would be nice to give more info to diagnosis.
> >>>>> > The output of 'btrfs fi usage' is useful for space usage problem.
> >>>>> >
> >>>>> > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
> >>>>> > space in vanilla df command) space for btrfs.
> >>>>> >
> >>>>> > Thanks,
> >>>>> > Qu
> >>>>> >
> >>>>> > >
> >>>>> > > Thank you so much for all of your help. I love how flexible BTRFS is
> >>>>> > > but when things go wrong it's very hard for me to troubleshoot.
> >>>>> > >
> >>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> >>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>> > >>
> >>>>> > >>
> >>>>> > >>
> >>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
> >>>>> > >>> Something went wrong:
> >>>>> > >>>
> >>>>> > >>> Reinitialize checksum tree
> >>>>> > >>> Unable to find block group for 0
> >>>>> > >>> Unable to find block group for 0
> >>>>> > >>> Unable to find block group for 0
> >>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
> >>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
> >>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
> >>>>> > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
> >>>>> > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
> >>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
> >>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
> >>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
> >>>>> > >>>
> >>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
> >>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
> >>>>> > >>> Aborted
> >>>>> > >>
> >>>>> > >> This means no space for extra metadata...
> >>>>> > >>
> >>>>> > >> Anyway the csum tree problem shouldn't be a big thing, you
> >>>>> could leave
> >>>>> > >> it and call it a day.
> >>>>> > >>
> >>>>> > >> BTW, as long as btrfs check reports no extra problem for the inode
> >>>>> > >> generation, it should be pretty safe to use the fs.
> >>>>> > >>
> >>>>> > >> Thanks,
> >>>>> > >> Qu
> >>>>> > >>>
> >>>>> > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
> >>>>> > >>> available. I'll let that try overnight?
> >>>>> > >>>
> >>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
> >>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>> > >>>>
> >>>>> > >>>>
> >>>>> > >>>>
> >>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
> >>>>> > >>>>> Thank you for helping. The end result of the scan was:
> >>>>> > >>>>>
> >>>>> > >>>>>
> >>>>> > >>>>> [1/7] checking root items
> >>>>> > >>>>> [2/7] checking extents
> >>>>> > >>>>> [3/7] checking free space cache
> >>>>> > >>>>> [4/7] checking fs roots
> >>>>> > >>>>
> >>>>> > >>>> Good news is, your fs is still mostly fine.
> >>>>> > >>>>
> >>>>> > >>>>> [5/7] checking only csums items (without verifying data)
> >>>>> > >>>>> there are no extents for csum range 0-69632
> >>>>> > >>>>> csum exists for 0-69632 but there is no extent record
> >>>>> > >>>>> ...
> >>>>> > >>>>> ...
> >>>>> > >>>>> there are no extents for csum range 946692096-946827264
> >>>>> > >>>>> csum exists for 946692096-946827264 but there is no extent
> >>>>> record
> >>>>> > >>>>> there are no extents for csum range 946831360-947912704
> >>>>> > >>>>> csum exists for 946831360-947912704 but there is no extent
> >>>>> record
> >>>>> > >>>>> ERROR: errors found in csum tree
> >>>>> > >>>>
> >>>>> > >>>> Only extent tree is corrupted.
> >>>>> > >>>>
> >>>>> > >>>> Normally btrfs check --init-csum-tree should be able to
> >>>>> handle it.
> >>>>> > >>>>
> >>>>> > >>>> But still, please be sure you're using the latest btrfs-progs
> >>>>> to fix it.
> >>>>> > >>>>
> >>>>> > >>>> Thanks,
> >>>>> > >>>> Qu
> >>>>> > >>>>
> >>>>> > >>>>> [6/7] checking root refs
> >>>>> > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
> >>>>> > >>>>> found 44157956026368 bytes used, error(s) found
> >>>>> > >>>>> total csum bytes: 42038602716
> >>>>> > >>>>> total tree bytes: 49688616960
> >>>>> > >>>>> total fs tree bytes: 1256427520
> >>>>> > >>>>> total extent tree bytes: 1709105152
> >>>>> > >>>>> btree space waste bytes: 3172727316
> >>>>> > >>>>> file data blocks allocated: 261625653436416
> >>>>> > >>>>> referenced 47477768499200
> >>>>> > >>>>>
> >>>>> > >>>>> What do I need to do to fix all of this?
> >>>>> > >>>>>
> >>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
> >>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>> > >>>>>>
> >>>>> > >>>>>>
> >>>>> > >>>>>>
> >>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
> >>>>> > >>>>>>> Well, the repair doesn't look terribly successful.
> >>>>> > >>>>>>>
> >>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>> 6875841 found 6876224
> >>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>> 6875841 found 6876224
> >>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>> 6875841 found 6876224
> >>>>> > >>>>>>> Ignoring transid failure
> >>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>> item=84
> >>>>> > >>>>>>> parent level=1
> >>>>> > >>>>>>> child level=4
> >>>>> > >>>>>>
> >>>>> > >>>>>> This means there are more problems, not only the hash name
> >>>>> mismatch.
> >>>>> > >>>>>>
> >>>>> > >>>>>> This means the fs is already corrupted, the name hash is
> >>>>> just one
> >>>>> > >>>>>> unrelated symptom.
> >>>>> > >>>>>>
> >>>>> > >>>>>> The only good news is, btrfs-progs abort the transaction,
> >>>>> thus no
> >>>>> > >>>>>> further damage to the fs.
> >>>>> > >>>>>>
> >>>>> > >>>>>> Please run a plain btrfs-check to show what's the problem
> >>>>> first.
> >>>>> > >>>>>>
> >>>>> > >>>>>> Thanks,
> >>>>> > >>>>>> Qu
> >>>>> > >>>>>>
> >>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>> 6875841 found 6876224
> >>>>> > >>>>>>> Ignoring transid failure
> >>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>> item=84
> >>>>> > >>>>>>> parent level=1
> >>>>> > >>>>>>> child level=4
> >>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>> 6875841 found 6876224
> >>>>> > >>>>>>> Ignoring transid failure
> >>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>> item=84
> >>>>> > >>>>>>> parent level=1
> >>>>> > >>>>>>> child level=4
> >>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>> 6875841 found 6876224
> >>>>> > >>>>>>> Ignoring transid failure
> >>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>> item=84
> >>>>> > >>>>>>> parent level=1
> >>>>> > >>>>>>> child level=4
> >>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>> 6875841 found 6876224
> >>>>> > >>>>>>> Ignoring transid failure
> >>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>> item=84
> >>>>> > >>>>>>> parent level=1
> >>>>> > >>>>>>> child level=4
> >>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>> 6875841 found 6876224
> >>>>> > >>>>>>> Ignoring transid failure
> >>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>> item=84
> >>>>> > >>>>>>> parent level=1
> >>>>> > >>>>>>> child level=4
> >>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>> 6875841 found 6876224
> >>>>> > >>>>>>> Ignoring transid failure
> >>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>> item=84
> >>>>> > >>>>>>> parent level=1
> >>>>> > >>>>>>> child level=4
> >>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>> 6875841 found 6876224
> >>>>> > >>>>>>> Ignoring transid failure
> >>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>> item=84
> >>>>> > >>>>>>> parent level=1
> >>>>> > >>>>>>> child level=4
> >>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>> 6875841 found 6876224
> >>>>> > >>>>>>> Ignoring transid failure
> >>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>> item=84
> >>>>> > >>>>>>> parent level=1
> >>>>> > >>>>>>> child level=4
> >>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>> 6875841 found 6876224
> >>>>> > >>>>>>> Ignoring transid failure
> >>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>> item=84
> >>>>> > >>>>>>> parent level=1
> >>>>> > >>>>>>> child level=4
> >>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>> 6875841 found 6876224
> >>>>> > >>>>>>> Ignoring transid failure
> >>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>> item=84
> >>>>> > >>>>>>> parent level=1
> >>>>> > >>>>>>> child level=4
> >>>>> > >>>>>>> ERROR: failed to zero log tree: -17
> >>>>> > >>>>>>> ERROR: attempt to start transaction over already running one
> >>>>> > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
> >>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> >>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> >>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>> 225049066086400 len 4096
> >>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> >>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> >>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>> 225049066094592 len 4096
> >>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> >>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> >>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>> 225049066102784 len 4096
> >>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> >>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> >>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>> 225049066131456 len 4096
> >>>>> > >>>>>>>
> >>>>> > >>>>>>> What is going on?
> >>>>> > >>>>>>>
> >>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond
> >>>>> <t.d.richmond@gmail.com <mailto:t.d.richmond@gmail.com>> wrote:
> >>>>> > >>>>>>>>
> >>>>> > >>>>>>>> Chris, I had used the correct mountpoint in the command.
> >>>>> I just edited
> >>>>> > >>>>>>>> it in the email to be /mountpoint for consistency.
> >>>>> > >>>>>>>>
> >>>>> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
> >>>>> > >>>>>>>>
> >>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
> >>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>> > >>>>>>>>>
> >>>>> > >>>>>>>>>
> >>>>> > >>>>>>>>>
> >>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
> >>>>> > >>>>>>>>>> Hello,
> >>>>> > >>>>>>>>>>
> >>>>> > >>>>>>>>>> I looked up this error and it basically says ask a
> >>>>> developer to
> >>>>> > >>>>>>>>>> determine if it's a false error or not. I just started
> >>>>> getting some
> >>>>> > >>>>>>>>>> slow response times, and looked at the dmesg log to
> >>>>> find a ton of
> >>>>> > >>>>>>>>>> these errors.
> >>>>> > >>>>>>>>>>
> >>>>> > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt
> >>>>> leaf: root=5
> >>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> >>>>> generation:
> >>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>> > >>>>>>>>>> [192088.449823] BTRFS error (device sdh):
> >>>>> block=203510940835840 read
> >>>>> > >>>>>>>>>> time tree block corruption detected
> >>>>> > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt
> >>>>> leaf: root=5
> >>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> >>>>> generation:
> >>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>> > >>>>>>>>>> [192088.462773] BTRFS error (device sdh):
> >>>>> block=203510940835840 read
> >>>>> > >>>>>>>>>> time tree block corruption detected
> >>>>> > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt
> >>>>> leaf: root=5
> >>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> >>>>> generation:
> >>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>> > >>>>>>>>>> [192088.468457] BTRFS error (device sdh):
> >>>>> block=203510940835840 read
> >>>>> > >>>>>>>>>> time tree block corruption detected
> >>>>> > >>>>>>>>>>
> >>>>> > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
> >>>>> > >>>>>>>>>>
> >>>>> > >>>>>>>>>> Is there anything I should do about this, or should I
> >>>>> just continue
> >>>>> > >>>>>>>>>> using my array as normal?
> >>>>> > >>>>>>>>>
> >>>>> > >>>>>>>>> This is caused by older kernel underflow inode generation.
> >>>>> > >>>>>>>>>
> >>>>> > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
> >>>>> > >>>>>>>>>
> >>>>> > >>>>>>>>> Or you can go safer, by manually locating the inode
> >>>>> using its inode
> >>>>> > >>>>>>>>> number (1311670), and copy it to some new location using
> >>>>> previous
> >>>>> > >>>>>>>>> working kernel, then delete the old file, copy the new
> >>>>> one back to fix it.
> >>>>> > >>>>>>>>>
> >>>>> > >>>>>>>>> Thanks,
> >>>>> > >>>>>>>>> Qu
> >>>>> > >>>>>>>>>
> >>>>> > >>>>>>>>>>
> >>>>> > >>>>>>>>>> Thank you!
> >>>>> > >>>>>>>>>>
> >>>>> > >>>>>>>>>
> >>>>> > >>>>>>
> >>>>> > >>>>
> >>>>> > >>
> >>>>> >
> >>>>>
> >>>>
> >
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-08-23 2:49 ` Tyler Richmond
@ 2020-08-23 4:28 ` Qu Wenruo
2020-08-24 2:47 ` Tyler Richmond
0 siblings, 1 reply; 56+ messages in thread
From: Qu Wenruo @ 2020-08-23 4:28 UTC (permalink / raw)
To: Tyler Richmond; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 22031 bytes --]
On 2020/8/23 上午10:49, Tyler Richmond wrote:
> Well, I can guarantee that I didn't create this fs before 2015 (just
> checked the order confirmation from when I bought the server), but I
> may have just used whatever was in the Ubuntu package manager at the
> time. So maybe I don't have a v0 ref?
Then btrfs-image shouldn't report that.
There is an item smaller than any valid btrfs item, normally it means
it's a v0 ref.
If not, then it could be a bigger problem.
Could you please provide the full btrfs-check output?
Also, if possible result from "btrfs check --mode=lowmem" would also help.
Also, if you really go "--repair", then the full output would also be
needed to determine what's going wrong.
There is a report about "btrfs check --repair" didn't repair the inode
generation, if that's the case we must have a bug then.
Thanks,
Qu
>
> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
>>>
>>>
>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
>>>> Is my best bet just to downgrade the kernel and then try to delete the
>>>> broken files? Or should I rebuild from scratch? Just don't know
>>>> whether it's worth the time to try and figure this out or if the
>>>> problems stem from the FS being too old and it's beyond trying to
>>>> repair.
>>>
>>> All invalid inode generations, should be able to be repaired by latest
>>> btrfs-check.
>>>
>>> If not, please provide the btrfs-image dump for us to determine what's
>>> going wrong.
>>>
>>> Thanks,
>>> Qu
>>>>
>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>>>>>
>>>>> I didn't check dmesg during the btrfs check, but that was the only
>>>>> output during the rm -f before it was forced readonly. I just checked
>>>>> dmesg for inode generation values, and there are a lot of them.
>>>>>
>>>>> https://pastebin.com/stZdN0ta
>>>>> The dmesg output had 990 lines containing inode generation.
>>>>>
>>>>> However, these were at least later. I tried to do a btrfs balance
>>>>> -mconvert raid1 and it failed with an I/O error. That is probably what
>>>>> generated these specific errors, but maybe they were also happening
>>>>> during the btrfs repair.
>>>>>
>>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway with:
>>>>> ERROR: either extent tree is corrupted or deprecated extent ref format
>>>>> ERROR: create failed: -5
>>
>> Oh, forgot this part.
>>
>> This means you have v0 ref?!
>>
>> Then the fs is too old, no progs/kernel support after all.
>>
>> In that case, please rollback to the last working kernel and copy your data.
>>
>> In fact, that v0 ref should only be in the code base for several weeks
>> before 2010, thus it's really too old.
>>
>> The good news is, with tree-checker, we should never experience such
>> too-old-to-be-usable problem (at least I hope so)
>>
>> Thanks,
>> Qu
>>
>>>>>
>>>>>
>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
>>>>>>> Qu,
>>>>>>>
>>>>>>> Sorry to resurrect this thread, but I just ran into something that I
>>>>>>> can't really just ignore. I've found a folder that is full of files
>>>>>>> which I guess have been broken somehow. I found a backup and restored
>>>>>>> them, but I want to delete this folder of broken files. But whenever I
>>>>>>> try, the fs is forced into readonly mode again. I just finished another
>>>>>>> btrfs check --repair but it didn't fix the problem.
>>>>>>>
>>>>>>> https://pastebin.com/eTV3s3fr
>>>>>>
>>>>>> Is that the full output?
>>>>>>
>>>>>> No inode generation bugs?
>>>>>>>
>>>>>>> I'm already on btrfs-progs v5.7. Any new suggestions?
>>>>>>
>>>>>> Strange.
>>>>>>
>>>>>> The detection and repair should have been merged into v5.5.
>>>>>>
>>>>>> If your fs is small enough, would you please provide the "btrfs-image
>>>>>> -c9" dump?
>>>>>>
>>>>>> It would contain the filenames and directories names, but doesn't
>>>>>> contain file contents.
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>>
>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com
>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>
>>>>>>> 5.6.1 also failed the same way. Here's the usage output. This is the
>>>>>>> part where you see I've been using RAID5 haha
>>>>>>>
>>>>>>> WARNING: RAID56 detected, not implemented
>>>>>>> Overall:
>>>>>>> Device size: 60.03TiB
>>>>>>> Device allocated: 98.06GiB
>>>>>>> Device unallocated: 59.93TiB
>>>>>>> Device missing: 0.00B
>>>>>>> Used: 92.56GiB
>>>>>>> Free (estimated): 0.00B (min: 8.00EiB)
>>>>>>> Data ratio: 0.00
>>>>>>> Metadata ratio: 2.00
>>>>>>> Global reserve: 512.00MiB (used: 0.00B)
>>>>>>> Multiple profiles: no
>>>>>>>
>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
>>>>>>> /dev/sdh 8.07TiB
>>>>>>> /dev/sdf 8.07TiB
>>>>>>> /dev/sdg 8.07TiB
>>>>>>> /dev/sdd 8.07TiB
>>>>>>> /dev/sdc 8.07TiB
>>>>>>> /dev/sde 8.07TiB
>>>>>>>
>>>>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
>>>>>>> /dev/sdh 34.00GiB
>>>>>>> /dev/sdf 32.00GiB
>>>>>>> /dev/sdg 32.00GiB
>>>>>>>
>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
>>>>>>> /dev/sdf 32.00MiB
>>>>>>> /dev/sdg 32.00MiB
>>>>>>>
>>>>>>> Unallocated:
>>>>>>> /dev/sdh 2.81TiB
>>>>>>> /dev/sdf 2.81TiB
>>>>>>> /dev/sdg 2.81TiB
>>>>>>> /dev/sdd 1.03TiB
>>>>>>> /dev/sdc 1.03TiB
>>>>>>> /dev/sde 1.03TiB
>>>>>>>
>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com
>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
>>>>>>> > > If this is saying there's no extra space for metadata, is that why
>>>>>>> > > adding more files often makes the system hang for 30-90s? Is there
>>>>>>> > > anything I should do about that?
>>>>>>> >
>>>>>>> > I'm not sure about the hang though.
>>>>>>> >
>>>>>>> > It would be nice to give more info to diagnosis.
>>>>>>> > The output of 'btrfs fi usage' is useful for space usage problem.
>>>>>>> >
>>>>>>> > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
>>>>>>> > space in vanilla df command) space for btrfs.
>>>>>>> >
>>>>>>> > Thanks,
>>>>>>> > Qu
>>>>>>> >
>>>>>>> > >
>>>>>>> > > Thank you so much for all of your help. I love how flexible BTRFS is
>>>>>>> > > but when things go wrong it's very hard for me to troubleshoot.
>>>>>>> > >
>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com
>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
>>>>>>> > >>> Something went wrong:
>>>>>>> > >>>
>>>>>>> > >>> Reinitialize checksum tree
>>>>>>> > >>> Unable to find block group for 0
>>>>>>> > >>> Unable to find block group for 0
>>>>>>> > >>> Unable to find block group for 0
>>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
>>>>>>> > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
>>>>>>> > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
>>>>>>> > >>>
>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
>>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
>>>>>>> > >>> Aborted
>>>>>>> > >>
>>>>>>> > >> This means no space for extra metadata...
>>>>>>> > >>
>>>>>>> > >> Anyway the csum tree problem shouldn't be a big thing, you
>>>>>>> could leave
>>>>>>> > >> it and call it a day.
>>>>>>> > >>
>>>>>>> > >> BTW, as long as btrfs check reports no extra problem for the inode
>>>>>>> > >> generation, it should be pretty safe to use the fs.
>>>>>>> > >>
>>>>>>> > >> Thanks,
>>>>>>> > >> Qu
>>>>>>> > >>>
>>>>>>> > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
>>>>>>> > >>> available. I'll let that try overnight?
>>>>>>> > >>>
>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>> > >>>>
>>>>>>> > >>>>
>>>>>>> > >>>>
>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
>>>>>>> > >>>>> Thank you for helping. The end result of the scan was:
>>>>>>> > >>>>>
>>>>>>> > >>>>>
>>>>>>> > >>>>> [1/7] checking root items
>>>>>>> > >>>>> [2/7] checking extents
>>>>>>> > >>>>> [3/7] checking free space cache
>>>>>>> > >>>>> [4/7] checking fs roots
>>>>>>> > >>>>
>>>>>>> > >>>> Good news is, your fs is still mostly fine.
>>>>>>> > >>>>
>>>>>>> > >>>>> [5/7] checking only csums items (without verifying data)
>>>>>>> > >>>>> there are no extents for csum range 0-69632
>>>>>>> > >>>>> csum exists for 0-69632 but there is no extent record
>>>>>>> > >>>>> ...
>>>>>>> > >>>>> ...
>>>>>>> > >>>>> there are no extents for csum range 946692096-946827264
>>>>>>> > >>>>> csum exists for 946692096-946827264 but there is no extent
>>>>>>> record
>>>>>>> > >>>>> there are no extents for csum range 946831360-947912704
>>>>>>> > >>>>> csum exists for 946831360-947912704 but there is no extent
>>>>>>> record
>>>>>>> > >>>>> ERROR: errors found in csum tree
>>>>>>> > >>>>
>>>>>>> > >>>> Only extent tree is corrupted.
>>>>>>> > >>>>
>>>>>>> > >>>> Normally btrfs check --init-csum-tree should be able to
>>>>>>> handle it.
>>>>>>> > >>>>
>>>>>>> > >>>> But still, please be sure you're using the latest btrfs-progs
>>>>>>> to fix it.
>>>>>>> > >>>>
>>>>>>> > >>>> Thanks,
>>>>>>> > >>>> Qu
>>>>>>> > >>>>
>>>>>>> > >>>>> [6/7] checking root refs
>>>>>>> > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>>>>>> > >>>>> found 44157956026368 bytes used, error(s) found
>>>>>>> > >>>>> total csum bytes: 42038602716
>>>>>>> > >>>>> total tree bytes: 49688616960
>>>>>>> > >>>>> total fs tree bytes: 1256427520
>>>>>>> > >>>>> total extent tree bytes: 1709105152
>>>>>>> > >>>>> btree space waste bytes: 3172727316
>>>>>>> > >>>>> file data blocks allocated: 261625653436416
>>>>>>> > >>>>> referenced 47477768499200
>>>>>>> > >>>>>
>>>>>>> > >>>>> What do I need to do to fix all of this?
>>>>>>> > >>>>>
>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>> > >>>>>>
>>>>>>> > >>>>>>
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
>>>>>>> > >>>>>>> Well, the repair doesn't look terribly successful.
>>>>>>> > >>>>>>>
>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>> 6875841 found 6876224
>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>> 6875841 found 6876224
>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>> 6875841 found 6876224
>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>> item=84
>>>>>>> > >>>>>>> parent level=1
>>>>>>> > >>>>>>> child level=4
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> This means there are more problems, not only the hash name
>>>>>>> mismatch.
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> This means the fs is already corrupted, the name hash is
>>>>>>> just one
>>>>>>> > >>>>>> unrelated symptom.
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> The only good news is, btrfs-progs abort the transaction,
>>>>>>> thus no
>>>>>>> > >>>>>> further damage to the fs.
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> Please run a plain btrfs-check to show what's the problem
>>>>>>> first.
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> Thanks,
>>>>>>> > >>>>>> Qu
>>>>>>> > >>>>>>
>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>> 6875841 found 6876224
>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>> item=84
>>>>>>> > >>>>>>> parent level=1
>>>>>>> > >>>>>>> child level=4
>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>> 6875841 found 6876224
>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>> item=84
>>>>>>> > >>>>>>> parent level=1
>>>>>>> > >>>>>>> child level=4
>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>> 6875841 found 6876224
>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>> item=84
>>>>>>> > >>>>>>> parent level=1
>>>>>>> > >>>>>>> child level=4
>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>> 6875841 found 6876224
>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>> item=84
>>>>>>> > >>>>>>> parent level=1
>>>>>>> > >>>>>>> child level=4
>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>> 6875841 found 6876224
>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>> item=84
>>>>>>> > >>>>>>> parent level=1
>>>>>>> > >>>>>>> child level=4
>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>> 6875841 found 6876224
>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>> item=84
>>>>>>> > >>>>>>> parent level=1
>>>>>>> > >>>>>>> child level=4
>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>> 6875841 found 6876224
>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>> item=84
>>>>>>> > >>>>>>> parent level=1
>>>>>>> > >>>>>>> child level=4
>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>> 6875841 found 6876224
>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>> item=84
>>>>>>> > >>>>>>> parent level=1
>>>>>>> > >>>>>>> child level=4
>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>> 6875841 found 6876224
>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>> item=84
>>>>>>> > >>>>>>> parent level=1
>>>>>>> > >>>>>>> child level=4
>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>> 6875841 found 6876224
>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>> item=84
>>>>>>> > >>>>>>> parent level=1
>>>>>>> > >>>>>>> child level=4
>>>>>>> > >>>>>>> ERROR: failed to zero log tree: -17
>>>>>>> > >>>>>>> ERROR: attempt to start transaction over already running one
>>>>>>> > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>> 225049066086400 len 4096
>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>> 225049066094592 len 4096
>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>> 225049066102784 len 4096
>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>> 225049066131456 len 4096
>>>>>>> > >>>>>>>
>>>>>>> > >>>>>>> What is going on?
>>>>>>> > >>>>>>>
>>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond
>>>>>>> <t.d.richmond@gmail.com <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>> Chris, I had used the correct mountpoint in the command.
>>>>>>> I just edited
>>>>>>> > >>>>>>>> it in the email to be /mountpoint for consistency.
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
>>>>>>> > >>>>>>>>>> Hello,
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> I looked up this error and it basically says ask a
>>>>>>> developer to
>>>>>>> > >>>>>>>>>> determine if it's a false error or not. I just started
>>>>>>> getting some
>>>>>>> > >>>>>>>>>> slow response times, and looked at the dmesg log to
>>>>>>> find a ton of
>>>>>>> > >>>>>>>>>> these errors.
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt
>>>>>>> leaf: root=5
>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>> generation:
>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error (device sdh):
>>>>>>> block=203510940835840 read
>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt
>>>>>>> leaf: root=5
>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>> generation:
>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error (device sdh):
>>>>>>> block=203510940835840 read
>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt
>>>>>>> leaf: root=5
>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>> generation:
>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error (device sdh):
>>>>>>> block=203510940835840 read
>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> Is there anything I should do about this, or should I
>>>>>>> just continue
>>>>>>> > >>>>>>>>>> using my array as normal?
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>> This is caused by older kernel underflow inode generation.
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>> Or you can go safer, by manually locating the inode
>>>>>>> using its inode
>>>>>>> > >>>>>>>>> number (1311670), and copy it to some new location using
>>>>>>> previous
>>>>>>> > >>>>>>>>> working kernel, then delete the old file, copy the new
>>>>>>> one back to fix it.
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>> Thanks,
>>>>>>> > >>>>>>>>> Qu
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> Thank you!
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>
>>>>>>> > >>>>
>>>>>>> > >>
>>>>>>> >
>>>>>>>
>>>>>>
>>>
>>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-08-23 4:28 ` Qu Wenruo
@ 2020-08-24 2:47 ` Tyler Richmond
2020-08-24 8:26 ` Qu Wenruo
0 siblings, 1 reply; 56+ messages in thread
From: Tyler Richmond @ 2020-08-24 2:47 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Btrfs BTRFS
Qu,
Finally finished another repair and captured the output.
https://pastebin.com/ffcbwvd8
Does that show you what you need? Or should I still do one in lowmem mode?
Thanks for your help!
On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2020/8/23 上午10:49, Tyler Richmond wrote:
> > Well, I can guarantee that I didn't create this fs before 2015 (just
> > checked the order confirmation from when I bought the server), but I
> > may have just used whatever was in the Ubuntu package manager at the
> > time. So maybe I don't have a v0 ref?
>
> Then btrfs-image shouldn't report that.
>
> There is an item smaller than any valid btrfs item, normally it means
> it's a v0 ref.
> If not, then it could be a bigger problem.
>
> Could you please provide the full btrfs-check output?
> Also, if possible result from "btrfs check --mode=lowmem" would also help.
>
> Also, if you really go "--repair", then the full output would also be
> needed to determine what's going wrong.
> There is a report about "btrfs check --repair" didn't repair the inode
> generation, if that's the case we must have a bug then.
>
> Thanks,
> Qu
> >
> > On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>
> >>
> >>
> >> On 2020/8/23 上午9:51, Qu Wenruo wrote:
> >>>
> >>>
> >>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
> >>>> Is my best bet just to downgrade the kernel and then try to delete the
> >>>> broken files? Or should I rebuild from scratch? Just don't know
> >>>> whether it's worth the time to try and figure this out or if the
> >>>> problems stem from the FS being too old and it's beyond trying to
> >>>> repair.
> >>>
> >>> All invalid inode generations, should be able to be repaired by latest
> >>> btrfs-check.
> >>>
> >>> If not, please provide the btrfs-image dump for us to determine what's
> >>> going wrong.
> >>>
> >>> Thanks,
> >>> Qu
> >>>>
> >>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
> >>>>>
> >>>>> I didn't check dmesg during the btrfs check, but that was the only
> >>>>> output during the rm -f before it was forced readonly. I just checked
> >>>>> dmesg for inode generation values, and there are a lot of them.
> >>>>>
> >>>>> https://pastebin.com/stZdN0ta
> >>>>> The dmesg output had 990 lines containing inode generation.
> >>>>>
> >>>>> However, these were at least later. I tried to do a btrfs balance
> >>>>> -mconvert raid1 and it failed with an I/O error. That is probably what
> >>>>> generated these specific errors, but maybe they were also happening
> >>>>> during the btrfs repair.
> >>>>>
> >>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway with:
> >>>>> ERROR: either extent tree is corrupted or deprecated extent ref format
> >>>>> ERROR: create failed: -5
> >>
> >> Oh, forgot this part.
> >>
> >> This means you have v0 ref?!
> >>
> >> Then the fs is too old, no progs/kernel support after all.
> >>
> >> In that case, please rollback to the last working kernel and copy your data.
> >>
> >> In fact, that v0 ref should only be in the code base for several weeks
> >> before 2010, thus it's really too old.
> >>
> >> The good news is, with tree-checker, we should never experience such
> >> too-old-to-be-usable problem (at least I hope so)
> >>
> >> Thanks,
> >> Qu
> >>
> >>>>>
> >>>>>
> >>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
> >>>>>>> Qu,
> >>>>>>>
> >>>>>>> Sorry to resurrect this thread, but I just ran into something that I
> >>>>>>> can't really just ignore. I've found a folder that is full of files
> >>>>>>> which I guess have been broken somehow. I found a backup and restored
> >>>>>>> them, but I want to delete this folder of broken files. But whenever I
> >>>>>>> try, the fs is forced into readonly mode again. I just finished another
> >>>>>>> btrfs check --repair but it didn't fix the problem.
> >>>>>>>
> >>>>>>> https://pastebin.com/eTV3s3fr
> >>>>>>
> >>>>>> Is that the full output?
> >>>>>>
> >>>>>> No inode generation bugs?
> >>>>>>>
> >>>>>>> I'm already on btrfs-progs v5.7. Any new suggestions?
> >>>>>>
> >>>>>> Strange.
> >>>>>>
> >>>>>> The detection and repair should have been merged into v5.5.
> >>>>>>
> >>>>>> If your fs is small enough, would you please provide the "btrfs-image
> >>>>>> -c9" dump?
> >>>>>>
> >>>>>> It would contain the filenames and directories names, but doesn't
> >>>>>> contain file contents.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Qu
> >>>>>>>
> >>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com
> >>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
> >>>>>>>
> >>>>>>> 5.6.1 also failed the same way. Here's the usage output. This is the
> >>>>>>> part where you see I've been using RAID5 haha
> >>>>>>>
> >>>>>>> WARNING: RAID56 detected, not implemented
> >>>>>>> Overall:
> >>>>>>> Device size: 60.03TiB
> >>>>>>> Device allocated: 98.06GiB
> >>>>>>> Device unallocated: 59.93TiB
> >>>>>>> Device missing: 0.00B
> >>>>>>> Used: 92.56GiB
> >>>>>>> Free (estimated): 0.00B (min: 8.00EiB)
> >>>>>>> Data ratio: 0.00
> >>>>>>> Metadata ratio: 2.00
> >>>>>>> Global reserve: 512.00MiB (used: 0.00B)
> >>>>>>> Multiple profiles: no
> >>>>>>>
> >>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
> >>>>>>> /dev/sdh 8.07TiB
> >>>>>>> /dev/sdf 8.07TiB
> >>>>>>> /dev/sdg 8.07TiB
> >>>>>>> /dev/sdd 8.07TiB
> >>>>>>> /dev/sdc 8.07TiB
> >>>>>>> /dev/sde 8.07TiB
> >>>>>>>
> >>>>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
> >>>>>>> /dev/sdh 34.00GiB
> >>>>>>> /dev/sdf 32.00GiB
> >>>>>>> /dev/sdg 32.00GiB
> >>>>>>>
> >>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
> >>>>>>> /dev/sdf 32.00MiB
> >>>>>>> /dev/sdg 32.00MiB
> >>>>>>>
> >>>>>>> Unallocated:
> >>>>>>> /dev/sdh 2.81TiB
> >>>>>>> /dev/sdf 2.81TiB
> >>>>>>> /dev/sdg 2.81TiB
> >>>>>>> /dev/sdd 1.03TiB
> >>>>>>> /dev/sdc 1.03TiB
> >>>>>>> /dev/sde 1.03TiB
> >>>>>>>
> >>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> >>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>> >
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
> >>>>>>> > > If this is saying there's no extra space for metadata, is that why
> >>>>>>> > > adding more files often makes the system hang for 30-90s? Is there
> >>>>>>> > > anything I should do about that?
> >>>>>>> >
> >>>>>>> > I'm not sure about the hang though.
> >>>>>>> >
> >>>>>>> > It would be nice to give more info to diagnosis.
> >>>>>>> > The output of 'btrfs fi usage' is useful for space usage problem.
> >>>>>>> >
> >>>>>>> > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
> >>>>>>> > space in vanilla df command) space for btrfs.
> >>>>>>> >
> >>>>>>> > Thanks,
> >>>>>>> > Qu
> >>>>>>> >
> >>>>>>> > >
> >>>>>>> > > Thank you so much for all of your help. I love how flexible BTRFS is
> >>>>>>> > > but when things go wrong it's very hard for me to troubleshoot.
> >>>>>>> > >
> >>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> >>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>> > >>
> >>>>>>> > >>
> >>>>>>> > >>
> >>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
> >>>>>>> > >>> Something went wrong:
> >>>>>>> > >>>
> >>>>>>> > >>> Reinitialize checksum tree
> >>>>>>> > >>> Unable to find block group for 0
> >>>>>>> > >>> Unable to find block group for 0
> >>>>>>> > >>> Unable to find block group for 0
> >>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
> >>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
> >>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
> >>>>>>> > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
> >>>>>>> > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
> >>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
> >>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
> >>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
> >>>>>>> > >>>
> >>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
> >>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
> >>>>>>> > >>> Aborted
> >>>>>>> > >>
> >>>>>>> > >> This means no space for extra metadata...
> >>>>>>> > >>
> >>>>>>> > >> Anyway the csum tree problem shouldn't be a big thing, you
> >>>>>>> could leave
> >>>>>>> > >> it and call it a day.
> >>>>>>> > >>
> >>>>>>> > >> BTW, as long as btrfs check reports no extra problem for the inode
> >>>>>>> > >> generation, it should be pretty safe to use the fs.
> >>>>>>> > >>
> >>>>>>> > >> Thanks,
> >>>>>>> > >> Qu
> >>>>>>> > >>>
> >>>>>>> > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
> >>>>>>> > >>> available. I'll let that try overnight?
> >>>>>>> > >>>
> >>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
> >>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>> > >>>>
> >>>>>>> > >>>>
> >>>>>>> > >>>>
> >>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
> >>>>>>> > >>>>> Thank you for helping. The end result of the scan was:
> >>>>>>> > >>>>>
> >>>>>>> > >>>>>
> >>>>>>> > >>>>> [1/7] checking root items
> >>>>>>> > >>>>> [2/7] checking extents
> >>>>>>> > >>>>> [3/7] checking free space cache
> >>>>>>> > >>>>> [4/7] checking fs roots
> >>>>>>> > >>>>
> >>>>>>> > >>>> Good news is, your fs is still mostly fine.
> >>>>>>> > >>>>
> >>>>>>> > >>>>> [5/7] checking only csums items (without verifying data)
> >>>>>>> > >>>>> there are no extents for csum range 0-69632
> >>>>>>> > >>>>> csum exists for 0-69632 but there is no extent record
> >>>>>>> > >>>>> ...
> >>>>>>> > >>>>> ...
> >>>>>>> > >>>>> there are no extents for csum range 946692096-946827264
> >>>>>>> > >>>>> csum exists for 946692096-946827264 but there is no extent
> >>>>>>> record
> >>>>>>> > >>>>> there are no extents for csum range 946831360-947912704
> >>>>>>> > >>>>> csum exists for 946831360-947912704 but there is no extent
> >>>>>>> record
> >>>>>>> > >>>>> ERROR: errors found in csum tree
> >>>>>>> > >>>>
> >>>>>>> > >>>> Only extent tree is corrupted.
> >>>>>>> > >>>>
> >>>>>>> > >>>> Normally btrfs check --init-csum-tree should be able to
> >>>>>>> handle it.
> >>>>>>> > >>>>
> >>>>>>> > >>>> But still, please be sure you're using the latest btrfs-progs
> >>>>>>> to fix it.
> >>>>>>> > >>>>
> >>>>>>> > >>>> Thanks,
> >>>>>>> > >>>> Qu
> >>>>>>> > >>>>
> >>>>>>> > >>>>> [6/7] checking root refs
> >>>>>>> > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
> >>>>>>> > >>>>> found 44157956026368 bytes used, error(s) found
> >>>>>>> > >>>>> total csum bytes: 42038602716
> >>>>>>> > >>>>> total tree bytes: 49688616960
> >>>>>>> > >>>>> total fs tree bytes: 1256427520
> >>>>>>> > >>>>> total extent tree bytes: 1709105152
> >>>>>>> > >>>>> btree space waste bytes: 3172727316
> >>>>>>> > >>>>> file data blocks allocated: 261625653436416
> >>>>>>> > >>>>> referenced 47477768499200
> >>>>>>> > >>>>>
> >>>>>>> > >>>>> What do I need to do to fix all of this?
> >>>>>>> > >>>>>
> >>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
> >>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>> > >>>>>>
> >>>>>>> > >>>>>>
> >>>>>>> > >>>>>>
> >>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
> >>>>>>> > >>>>>>> Well, the repair doesn't look terribly successful.
> >>>>>>> > >>>>>>>
> >>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>> 6875841 found 6876224
> >>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>> 6875841 found 6876224
> >>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>> 6875841 found 6876224
> >>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>> item=84
> >>>>>>> > >>>>>>> parent level=1
> >>>>>>> > >>>>>>> child level=4
> >>>>>>> > >>>>>>
> >>>>>>> > >>>>>> This means there are more problems, not only the hash name
> >>>>>>> mismatch.
> >>>>>>> > >>>>>>
> >>>>>>> > >>>>>> This means the fs is already corrupted, the name hash is
> >>>>>>> just one
> >>>>>>> > >>>>>> unrelated symptom.
> >>>>>>> > >>>>>>
> >>>>>>> > >>>>>> The only good news is, btrfs-progs abort the transaction,
> >>>>>>> thus no
> >>>>>>> > >>>>>> further damage to the fs.
> >>>>>>> > >>>>>>
> >>>>>>> > >>>>>> Please run a plain btrfs-check to show what's the problem
> >>>>>>> first.
> >>>>>>> > >>>>>>
> >>>>>>> > >>>>>> Thanks,
> >>>>>>> > >>>>>> Qu
> >>>>>>> > >>>>>>
> >>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>> 6875841 found 6876224
> >>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>> item=84
> >>>>>>> > >>>>>>> parent level=1
> >>>>>>> > >>>>>>> child level=4
> >>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>> 6875841 found 6876224
> >>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>> item=84
> >>>>>>> > >>>>>>> parent level=1
> >>>>>>> > >>>>>>> child level=4
> >>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>> 6875841 found 6876224
> >>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>> item=84
> >>>>>>> > >>>>>>> parent level=1
> >>>>>>> > >>>>>>> child level=4
> >>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>> 6875841 found 6876224
> >>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>> item=84
> >>>>>>> > >>>>>>> parent level=1
> >>>>>>> > >>>>>>> child level=4
> >>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>> 6875841 found 6876224
> >>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>> item=84
> >>>>>>> > >>>>>>> parent level=1
> >>>>>>> > >>>>>>> child level=4
> >>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>> 6875841 found 6876224
> >>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>> item=84
> >>>>>>> > >>>>>>> parent level=1
> >>>>>>> > >>>>>>> child level=4
> >>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>> 6875841 found 6876224
> >>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>> item=84
> >>>>>>> > >>>>>>> parent level=1
> >>>>>>> > >>>>>>> child level=4
> >>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>> 6875841 found 6876224
> >>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>> item=84
> >>>>>>> > >>>>>>> parent level=1
> >>>>>>> > >>>>>>> child level=4
> >>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>> 6875841 found 6876224
> >>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>> item=84
> >>>>>>> > >>>>>>> parent level=1
> >>>>>>> > >>>>>>> child level=4
> >>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>> 6875841 found 6876224
> >>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>> item=84
> >>>>>>> > >>>>>>> parent level=1
> >>>>>>> > >>>>>>> child level=4
> >>>>>>> > >>>>>>> ERROR: failed to zero log tree: -17
> >>>>>>> > >>>>>>> ERROR: attempt to start transaction over already running one
> >>>>>>> > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
> >>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> >>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> >>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>>>> 225049066086400 len 4096
> >>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> >>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> >>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>>>> 225049066094592 len 4096
> >>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> >>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> >>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>>>> 225049066102784 len 4096
> >>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> >>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> >>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>>>> 225049066131456 len 4096
> >>>>>>> > >>>>>>>
> >>>>>>> > >>>>>>> What is going on?
> >>>>>>> > >>>>>>>
> >>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond
> >>>>>>> <t.d.richmond@gmail.com <mailto:t.d.richmond@gmail.com>> wrote:
> >>>>>>> > >>>>>>>>
> >>>>>>> > >>>>>>>> Chris, I had used the correct mountpoint in the command.
> >>>>>>> I just edited
> >>>>>>> > >>>>>>>> it in the email to be /mountpoint for consistency.
> >>>>>>> > >>>>>>>>
> >>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
> >>>>>>> > >>>>>>>>
> >>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
> >>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>> > >>>>>>>>>
> >>>>>>> > >>>>>>>>>
> >>>>>>> > >>>>>>>>>
> >>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
> >>>>>>> > >>>>>>>>>> Hello,
> >>>>>>> > >>>>>>>>>>
> >>>>>>> > >>>>>>>>>> I looked up this error and it basically says ask a
> >>>>>>> developer to
> >>>>>>> > >>>>>>>>>> determine if it's a false error or not. I just started
> >>>>>>> getting some
> >>>>>>> > >>>>>>>>>> slow response times, and looked at the dmesg log to
> >>>>>>> find a ton of
> >>>>>>> > >>>>>>>>>> these errors.
> >>>>>>> > >>>>>>>>>>
> >>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt
> >>>>>>> leaf: root=5
> >>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> >>>>>>> generation:
> >>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error (device sdh):
> >>>>>>> block=203510940835840 read
> >>>>>>> > >>>>>>>>>> time tree block corruption detected
> >>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt
> >>>>>>> leaf: root=5
> >>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> >>>>>>> generation:
> >>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error (device sdh):
> >>>>>>> block=203510940835840 read
> >>>>>>> > >>>>>>>>>> time tree block corruption detected
> >>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt
> >>>>>>> leaf: root=5
> >>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> >>>>>>> generation:
> >>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error (device sdh):
> >>>>>>> block=203510940835840 read
> >>>>>>> > >>>>>>>>>> time tree block corruption detected
> >>>>>>> > >>>>>>>>>>
> >>>>>>> > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
> >>>>>>> > >>>>>>>>>>
> >>>>>>> > >>>>>>>>>> Is there anything I should do about this, or should I
> >>>>>>> just continue
> >>>>>>> > >>>>>>>>>> using my array as normal?
> >>>>>>> > >>>>>>>>>
> >>>>>>> > >>>>>>>>> This is caused by older kernel underflow inode generation.
> >>>>>>> > >>>>>>>>>
> >>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
> >>>>>>> > >>>>>>>>>
> >>>>>>> > >>>>>>>>> Or you can go safer, by manually locating the inode
> >>>>>>> using its inode
> >>>>>>> > >>>>>>>>> number (1311670), and copy it to some new location using
> >>>>>>> previous
> >>>>>>> > >>>>>>>>> working kernel, then delete the old file, copy the new
> >>>>>>> one back to fix it.
> >>>>>>> > >>>>>>>>>
> >>>>>>> > >>>>>>>>> Thanks,
> >>>>>>> > >>>>>>>>> Qu
> >>>>>>> > >>>>>>>>>
> >>>>>>> > >>>>>>>>>>
> >>>>>>> > >>>>>>>>>> Thank you!
> >>>>>>> > >>>>>>>>>>
> >>>>>>> > >>>>>>>>>
> >>>>>>> > >>>>>>
> >>>>>>> > >>>>
> >>>>>>> > >>
> >>>>>>> >
> >>>>>>>
> >>>>>>
> >>>
> >>
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-08-24 2:47 ` Tyler Richmond
@ 2020-08-24 8:26 ` Qu Wenruo
2020-08-25 5:25 ` Tyler Richmond
0 siblings, 1 reply; 56+ messages in thread
From: Qu Wenruo @ 2020-08-24 8:26 UTC (permalink / raw)
To: Tyler Richmond; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 23569 bytes --]
On 2020/8/24 上午10:47, Tyler Richmond wrote:
> Qu,
>
> Finally finished another repair and captured the output.
>
> https://pastebin.com/ffcbwvd8
>
> Does that show you what you need? Or should I still do one in lowmem mode?
Lowmem mode (no need for --repair) is recommended since original mode
doesn't detect the inode generation problem.
And it's already btrfs-progs v5.7 right?
THanks,
Qu
>
> Thanks for your help!
>
> On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2020/8/23 上午10:49, Tyler Richmond wrote:
>>> Well, I can guarantee that I didn't create this fs before 2015 (just
>>> checked the order confirmation from when I bought the server), but I
>>> may have just used whatever was in the Ubuntu package manager at the
>>> time. So maybe I don't have a v0 ref?
>>
>> Then btrfs-image shouldn't report that.
>>
>> There is an item smaller than any valid btrfs item, normally it means
>> it's a v0 ref.
>> If not, then it could be a bigger problem.
>>
>> Could you please provide the full btrfs-check output?
>> Also, if possible result from "btrfs check --mode=lowmem" would also help.
>>
>> Also, if you really go "--repair", then the full output would also be
>> needed to determine what's going wrong.
>> There is a report about "btrfs check --repair" didn't repair the inode
>> generation, if that's the case we must have a bug then.
>>
>> Thanks,
>> Qu
>>>
>>> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>
>>>>
>>>>
>>>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
>>>>>
>>>>>
>>>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
>>>>>> Is my best bet just to downgrade the kernel and then try to delete the
>>>>>> broken files? Or should I rebuild from scratch? Just don't know
>>>>>> whether it's worth the time to try and figure this out or if the
>>>>>> problems stem from the FS being too old and it's beyond trying to
>>>>>> repair.
>>>>>
>>>>> All invalid inode generations, should be able to be repaired by latest
>>>>> btrfs-check.
>>>>>
>>>>> If not, please provide the btrfs-image dump for us to determine what's
>>>>> going wrong.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>>
>>>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>>>>>>>
>>>>>>> I didn't check dmesg during the btrfs check, but that was the only
>>>>>>> output during the rm -f before it was forced readonly. I just checked
>>>>>>> dmesg for inode generation values, and there are a lot of them.
>>>>>>>
>>>>>>> https://pastebin.com/stZdN0ta
>>>>>>> The dmesg output had 990 lines containing inode generation.
>>>>>>>
>>>>>>> However, these were at least later. I tried to do a btrfs balance
>>>>>>> -mconvert raid1 and it failed with an I/O error. That is probably what
>>>>>>> generated these specific errors, but maybe they were also happening
>>>>>>> during the btrfs repair.
>>>>>>>
>>>>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway with:
>>>>>>> ERROR: either extent tree is corrupted or deprecated extent ref format
>>>>>>> ERROR: create failed: -5
>>>>
>>>> Oh, forgot this part.
>>>>
>>>> This means you have v0 ref?!
>>>>
>>>> Then the fs is too old, no progs/kernel support after all.
>>>>
>>>> In that case, please rollback to the last working kernel and copy your data.
>>>>
>>>> In fact, that v0 ref should only be in the code base for several weeks
>>>> before 2010, thus it's really too old.
>>>>
>>>> The good news is, with tree-checker, we should never experience such
>>>> too-old-to-be-usable problem (at least I hope so)
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
>>>>>>>>> Qu,
>>>>>>>>>
>>>>>>>>> Sorry to resurrect this thread, but I just ran into something that I
>>>>>>>>> can't really just ignore. I've found a folder that is full of files
>>>>>>>>> which I guess have been broken somehow. I found a backup and restored
>>>>>>>>> them, but I want to delete this folder of broken files. But whenever I
>>>>>>>>> try, the fs is forced into readonly mode again. I just finished another
>>>>>>>>> btrfs check --repair but it didn't fix the problem.
>>>>>>>>>
>>>>>>>>> https://pastebin.com/eTV3s3fr
>>>>>>>>
>>>>>>>> Is that the full output?
>>>>>>>>
>>>>>>>> No inode generation bugs?
>>>>>>>>>
>>>>>>>>> I'm already on btrfs-progs v5.7. Any new suggestions?
>>>>>>>>
>>>>>>>> Strange.
>>>>>>>>
>>>>>>>> The detection and repair should have been merged into v5.5.
>>>>>>>>
>>>>>>>> If your fs is small enough, would you please provide the "btrfs-image
>>>>>>>> -c9" dump?
>>>>>>>>
>>>>>>>> It would contain the filenames and directories names, but doesn't
>>>>>>>> contain file contents.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Qu
>>>>>>>>>
>>>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com
>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>
>>>>>>>>> 5.6.1 also failed the same way. Here's the usage output. This is the
>>>>>>>>> part where you see I've been using RAID5 haha
>>>>>>>>>
>>>>>>>>> WARNING: RAID56 detected, not implemented
>>>>>>>>> Overall:
>>>>>>>>> Device size: 60.03TiB
>>>>>>>>> Device allocated: 98.06GiB
>>>>>>>>> Device unallocated: 59.93TiB
>>>>>>>>> Device missing: 0.00B
>>>>>>>>> Used: 92.56GiB
>>>>>>>>> Free (estimated): 0.00B (min: 8.00EiB)
>>>>>>>>> Data ratio: 0.00
>>>>>>>>> Metadata ratio: 2.00
>>>>>>>>> Global reserve: 512.00MiB (used: 0.00B)
>>>>>>>>> Multiple profiles: no
>>>>>>>>>
>>>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
>>>>>>>>> /dev/sdh 8.07TiB
>>>>>>>>> /dev/sdf 8.07TiB
>>>>>>>>> /dev/sdg 8.07TiB
>>>>>>>>> /dev/sdd 8.07TiB
>>>>>>>>> /dev/sdc 8.07TiB
>>>>>>>>> /dev/sde 8.07TiB
>>>>>>>>>
>>>>>>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
>>>>>>>>> /dev/sdh 34.00GiB
>>>>>>>>> /dev/sdf 32.00GiB
>>>>>>>>> /dev/sdg 32.00GiB
>>>>>>>>>
>>>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
>>>>>>>>> /dev/sdf 32.00MiB
>>>>>>>>> /dev/sdg 32.00MiB
>>>>>>>>>
>>>>>>>>> Unallocated:
>>>>>>>>> /dev/sdh 2.81TiB
>>>>>>>>> /dev/sdf 2.81TiB
>>>>>>>>> /dev/sdg 2.81TiB
>>>>>>>>> /dev/sdd 1.03TiB
>>>>>>>>> /dev/sdc 1.03TiB
>>>>>>>>> /dev/sde 1.03TiB
>>>>>>>>>
>>>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com
>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
>>>>>>>>> > > If this is saying there's no extra space for metadata, is that why
>>>>>>>>> > > adding more files often makes the system hang for 30-90s? Is there
>>>>>>>>> > > anything I should do about that?
>>>>>>>>> >
>>>>>>>>> > I'm not sure about the hang though.
>>>>>>>>> >
>>>>>>>>> > It would be nice to give more info to diagnosis.
>>>>>>>>> > The output of 'btrfs fi usage' is useful for space usage problem.
>>>>>>>>> >
>>>>>>>>> > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
>>>>>>>>> > space in vanilla df command) space for btrfs.
>>>>>>>>> >
>>>>>>>>> > Thanks,
>>>>>>>>> > Qu
>>>>>>>>> >
>>>>>>>>> > >
>>>>>>>>> > > Thank you so much for all of your help. I love how flexible BTRFS is
>>>>>>>>> > > but when things go wrong it's very hard for me to troubleshoot.
>>>>>>>>> > >
>>>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com
>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>> > >>
>>>>>>>>> > >>
>>>>>>>>> > >>
>>>>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
>>>>>>>>> > >>> Something went wrong:
>>>>>>>>> > >>>
>>>>>>>>> > >>> Reinitialize checksum tree
>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
>>>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
>>>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
>>>>>>>>> > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
>>>>>>>>> > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
>>>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
>>>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
>>>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
>>>>>>>>> > >>>
>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
>>>>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
>>>>>>>>> > >>> Aborted
>>>>>>>>> > >>
>>>>>>>>> > >> This means no space for extra metadata...
>>>>>>>>> > >>
>>>>>>>>> > >> Anyway the csum tree problem shouldn't be a big thing, you
>>>>>>>>> could leave
>>>>>>>>> > >> it and call it a day.
>>>>>>>>> > >>
>>>>>>>>> > >> BTW, as long as btrfs check reports no extra problem for the inode
>>>>>>>>> > >> generation, it should be pretty safe to use the fs.
>>>>>>>>> > >>
>>>>>>>>> > >> Thanks,
>>>>>>>>> > >> Qu
>>>>>>>>> > >>>
>>>>>>>>> > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
>>>>>>>>> > >>> available. I'll let that try overnight?
>>>>>>>>> > >>>
>>>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>> > >>>>
>>>>>>>>> > >>>>
>>>>>>>>> > >>>>
>>>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
>>>>>>>>> > >>>>> Thank you for helping. The end result of the scan was:
>>>>>>>>> > >>>>>
>>>>>>>>> > >>>>>
>>>>>>>>> > >>>>> [1/7] checking root items
>>>>>>>>> > >>>>> [2/7] checking extents
>>>>>>>>> > >>>>> [3/7] checking free space cache
>>>>>>>>> > >>>>> [4/7] checking fs roots
>>>>>>>>> > >>>>
>>>>>>>>> > >>>> Good news is, your fs is still mostly fine.
>>>>>>>>> > >>>>
>>>>>>>>> > >>>>> [5/7] checking only csums items (without verifying data)
>>>>>>>>> > >>>>> there are no extents for csum range 0-69632
>>>>>>>>> > >>>>> csum exists for 0-69632 but there is no extent record
>>>>>>>>> > >>>>> ...
>>>>>>>>> > >>>>> ...
>>>>>>>>> > >>>>> there are no extents for csum range 946692096-946827264
>>>>>>>>> > >>>>> csum exists for 946692096-946827264 but there is no extent
>>>>>>>>> record
>>>>>>>>> > >>>>> there are no extents for csum range 946831360-947912704
>>>>>>>>> > >>>>> csum exists for 946831360-947912704 but there is no extent
>>>>>>>>> record
>>>>>>>>> > >>>>> ERROR: errors found in csum tree
>>>>>>>>> > >>>>
>>>>>>>>> > >>>> Only extent tree is corrupted.
>>>>>>>>> > >>>>
>>>>>>>>> > >>>> Normally btrfs check --init-csum-tree should be able to
>>>>>>>>> handle it.
>>>>>>>>> > >>>>
>>>>>>>>> > >>>> But still, please be sure you're using the latest btrfs-progs
>>>>>>>>> to fix it.
>>>>>>>>> > >>>>
>>>>>>>>> > >>>> Thanks,
>>>>>>>>> > >>>> Qu
>>>>>>>>> > >>>>
>>>>>>>>> > >>>>> [6/7] checking root refs
>>>>>>>>> > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>>>>>>>> > >>>>> found 44157956026368 bytes used, error(s) found
>>>>>>>>> > >>>>> total csum bytes: 42038602716
>>>>>>>>> > >>>>> total tree bytes: 49688616960
>>>>>>>>> > >>>>> total fs tree bytes: 1256427520
>>>>>>>>> > >>>>> total extent tree bytes: 1709105152
>>>>>>>>> > >>>>> btree space waste bytes: 3172727316
>>>>>>>>> > >>>>> file data blocks allocated: 261625653436416
>>>>>>>>> > >>>>> referenced 47477768499200
>>>>>>>>> > >>>>>
>>>>>>>>> > >>>>> What do I need to do to fix all of this?
>>>>>>>>> > >>>>>
>>>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
>>>>>>>>> > >>>>>>> Well, the repair doesn't look terribly successful.
>>>>>>>>> > >>>>>>>
>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>> 6875841 found 6876224
>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>> 6875841 found 6876224
>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>> 6875841 found 6876224
>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>> item=84
>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>> This means there are more problems, not only the hash name
>>>>>>>>> mismatch.
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>> This means the fs is already corrupted, the name hash is
>>>>>>>>> just one
>>>>>>>>> > >>>>>> unrelated symptom.
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>> The only good news is, btrfs-progs abort the transaction,
>>>>>>>>> thus no
>>>>>>>>> > >>>>>> further damage to the fs.
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>> Please run a plain btrfs-check to show what's the problem
>>>>>>>>> first.
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>> > >>>>>> Qu
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>> 6875841 found 6876224
>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>> item=84
>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>> 6875841 found 6876224
>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>> item=84
>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>> 6875841 found 6876224
>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>> item=84
>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>> 6875841 found 6876224
>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>> item=84
>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>> 6875841 found 6876224
>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>> item=84
>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>> 6875841 found 6876224
>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>> item=84
>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>> 6875841 found 6876224
>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>> item=84
>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>> 6875841 found 6876224
>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>> item=84
>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>> 6875841 found 6876224
>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>> item=84
>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>> 6875841 found 6876224
>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>> item=84
>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>> > >>>>>>> ERROR: failed to zero log tree: -17
>>>>>>>>> > >>>>>>> ERROR: attempt to start transaction over already running one
>>>>>>>>> > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>> 225049066086400 len 4096
>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>> 225049066094592 len 4096
>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>> 225049066102784 len 4096
>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>> 225049066131456 len 4096
>>>>>>>>> > >>>>>>>
>>>>>>>>> > >>>>>>> What is going on?
>>>>>>>>> > >>>>>>>
>>>>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond
>>>>>>>>> <t.d.richmond@gmail.com <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> Chris, I had used the correct mountpoint in the command.
>>>>>>>>> I just edited
>>>>>>>>> > >>>>>>>> it in the email to be /mountpoint for consistency.
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
>>>>>>>>> > >>>>>>>>>> Hello,
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> I looked up this error and it basically says ask a
>>>>>>>>> developer to
>>>>>>>>> > >>>>>>>>>> determine if it's a false error or not. I just started
>>>>>>>>> getting some
>>>>>>>>> > >>>>>>>>>> slow response times, and looked at the dmesg log to
>>>>>>>>> find a ton of
>>>>>>>>> > >>>>>>>>>> these errors.
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt
>>>>>>>>> leaf: root=5
>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>> generation:
>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error (device sdh):
>>>>>>>>> block=203510940835840 read
>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt
>>>>>>>>> leaf: root=5
>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>> generation:
>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error (device sdh):
>>>>>>>>> block=203510940835840 read
>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt
>>>>>>>>> leaf: root=5
>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>> generation:
>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error (device sdh):
>>>>>>>>> block=203510940835840 read
>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> Is there anything I should do about this, or should I
>>>>>>>>> just continue
>>>>>>>>> > >>>>>>>>>> using my array as normal?
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>> This is caused by older kernel underflow inode generation.
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>> Or you can go safer, by manually locating the inode
>>>>>>>>> using its inode
>>>>>>>>> > >>>>>>>>> number (1311670), and copy it to some new location using
>>>>>>>>> previous
>>>>>>>>> > >>>>>>>>> working kernel, then delete the old file, copy the new
>>>>>>>>> one back to fix it.
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>>> > >>>>>>>>> Qu
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> Thank you!
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>
>>>>>>>>> > >>
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>
>>>>>
>>>>
>>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-08-24 8:26 ` Qu Wenruo
@ 2020-08-25 5:25 ` Tyler Richmond
2020-08-25 6:37 ` Qu Wenruo
0 siblings, 1 reply; 56+ messages in thread
From: Tyler Richmond @ 2020-08-25 5:25 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Btrfs BTRFS
Qu,
Yes, it's btrfs-progs 5.7. Here is the result of the lowmem check:
https://pastebin.com/8Tzx23EX
Thanks!
On Mon, Aug 24, 2020 at 4:26 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2020/8/24 上午10:47, Tyler Richmond wrote:
> > Qu,
> >
> > Finally finished another repair and captured the output.
> >
> > https://pastebin.com/ffcbwvd8
> >
> > Does that show you what you need? Or should I still do one in lowmem mode?
>
> Lowmem mode (no need for --repair) is recommended since original mode
> doesn't detect the inode generation problem.
>
> And it's already btrfs-progs v5.7 right?
>
> THanks,
> Qu
> >
> > Thanks for your help!
> >
> > On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>
> >>
> >>
> >> On 2020/8/23 上午10:49, Tyler Richmond wrote:
> >>> Well, I can guarantee that I didn't create this fs before 2015 (just
> >>> checked the order confirmation from when I bought the server), but I
> >>> may have just used whatever was in the Ubuntu package manager at the
> >>> time. So maybe I don't have a v0 ref?
> >>
> >> Then btrfs-image shouldn't report that.
> >>
> >> There is an item smaller than any valid btrfs item, normally it means
> >> it's a v0 ref.
> >> If not, then it could be a bigger problem.
> >>
> >> Could you please provide the full btrfs-check output?
> >> Also, if possible result from "btrfs check --mode=lowmem" would also help.
> >>
> >> Also, if you really go "--repair", then the full output would also be
> >> needed to determine what's going wrong.
> >> There is a report about "btrfs check --repair" didn't repair the inode
> >> generation, if that's the case we must have a bug then.
> >>
> >> Thanks,
> >> Qu
> >>>
> >>> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
> >>>>>
> >>>>>
> >>>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
> >>>>>> Is my best bet just to downgrade the kernel and then try to delete the
> >>>>>> broken files? Or should I rebuild from scratch? Just don't know
> >>>>>> whether it's worth the time to try and figure this out or if the
> >>>>>> problems stem from the FS being too old and it's beyond trying to
> >>>>>> repair.
> >>>>>
> >>>>> All invalid inode generations, should be able to be repaired by latest
> >>>>> btrfs-check.
> >>>>>
> >>>>> If not, please provide the btrfs-image dump for us to determine what's
> >>>>> going wrong.
> >>>>>
> >>>>> Thanks,
> >>>>> Qu
> >>>>>>
> >>>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
> >>>>>>>
> >>>>>>> I didn't check dmesg during the btrfs check, but that was the only
> >>>>>>> output during the rm -f before it was forced readonly. I just checked
> >>>>>>> dmesg for inode generation values, and there are a lot of them.
> >>>>>>>
> >>>>>>> https://pastebin.com/stZdN0ta
> >>>>>>> The dmesg output had 990 lines containing inode generation.
> >>>>>>>
> >>>>>>> However, these were at least later. I tried to do a btrfs balance
> >>>>>>> -mconvert raid1 and it failed with an I/O error. That is probably what
> >>>>>>> generated these specific errors, but maybe they were also happening
> >>>>>>> during the btrfs repair.
> >>>>>>>
> >>>>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway with:
> >>>>>>> ERROR: either extent tree is corrupted or deprecated extent ref format
> >>>>>>> ERROR: create failed: -5
> >>>>
> >>>> Oh, forgot this part.
> >>>>
> >>>> This means you have v0 ref?!
> >>>>
> >>>> Then the fs is too old, no progs/kernel support after all.
> >>>>
> >>>> In that case, please rollback to the last working kernel and copy your data.
> >>>>
> >>>> In fact, that v0 ref should only be in the code base for several weeks
> >>>> before 2010, thus it's really too old.
> >>>>
> >>>> The good news is, with tree-checker, we should never experience such
> >>>> too-old-to-be-usable problem (at least I hope so)
> >>>>
> >>>> Thanks,
> >>>> Qu
> >>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
> >>>>>>>>> Qu,
> >>>>>>>>>
> >>>>>>>>> Sorry to resurrect this thread, but I just ran into something that I
> >>>>>>>>> can't really just ignore. I've found a folder that is full of files
> >>>>>>>>> which I guess have been broken somehow. I found a backup and restored
> >>>>>>>>> them, but I want to delete this folder of broken files. But whenever I
> >>>>>>>>> try, the fs is forced into readonly mode again. I just finished another
> >>>>>>>>> btrfs check --repair but it didn't fix the problem.
> >>>>>>>>>
> >>>>>>>>> https://pastebin.com/eTV3s3fr
> >>>>>>>>
> >>>>>>>> Is that the full output?
> >>>>>>>>
> >>>>>>>> No inode generation bugs?
> >>>>>>>>>
> >>>>>>>>> I'm already on btrfs-progs v5.7. Any new suggestions?
> >>>>>>>>
> >>>>>>>> Strange.
> >>>>>>>>
> >>>>>>>> The detection and repair should have been merged into v5.5.
> >>>>>>>>
> >>>>>>>> If your fs is small enough, would you please provide the "btrfs-image
> >>>>>>>> -c9" dump?
> >>>>>>>>
> >>>>>>>> It would contain the filenames and directories names, but doesn't
> >>>>>>>> contain file contents.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Qu
> >>>>>>>>>
> >>>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com
> >>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
> >>>>>>>>>
> >>>>>>>>> 5.6.1 also failed the same way. Here's the usage output. This is the
> >>>>>>>>> part where you see I've been using RAID5 haha
> >>>>>>>>>
> >>>>>>>>> WARNING: RAID56 detected, not implemented
> >>>>>>>>> Overall:
> >>>>>>>>> Device size: 60.03TiB
> >>>>>>>>> Device allocated: 98.06GiB
> >>>>>>>>> Device unallocated: 59.93TiB
> >>>>>>>>> Device missing: 0.00B
> >>>>>>>>> Used: 92.56GiB
> >>>>>>>>> Free (estimated): 0.00B (min: 8.00EiB)
> >>>>>>>>> Data ratio: 0.00
> >>>>>>>>> Metadata ratio: 2.00
> >>>>>>>>> Global reserve: 512.00MiB (used: 0.00B)
> >>>>>>>>> Multiple profiles: no
> >>>>>>>>>
> >>>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
> >>>>>>>>> /dev/sdh 8.07TiB
> >>>>>>>>> /dev/sdf 8.07TiB
> >>>>>>>>> /dev/sdg 8.07TiB
> >>>>>>>>> /dev/sdd 8.07TiB
> >>>>>>>>> /dev/sdc 8.07TiB
> >>>>>>>>> /dev/sde 8.07TiB
> >>>>>>>>>
> >>>>>>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
> >>>>>>>>> /dev/sdh 34.00GiB
> >>>>>>>>> /dev/sdf 32.00GiB
> >>>>>>>>> /dev/sdg 32.00GiB
> >>>>>>>>>
> >>>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
> >>>>>>>>> /dev/sdf 32.00MiB
> >>>>>>>>> /dev/sdg 32.00MiB
> >>>>>>>>>
> >>>>>>>>> Unallocated:
> >>>>>>>>> /dev/sdh 2.81TiB
> >>>>>>>>> /dev/sdf 2.81TiB
> >>>>>>>>> /dev/sdg 2.81TiB
> >>>>>>>>> /dev/sdd 1.03TiB
> >>>>>>>>> /dev/sdc 1.03TiB
> >>>>>>>>> /dev/sde 1.03TiB
> >>>>>>>>>
> >>>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> >>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>>>> >
> >>>>>>>>> >
> >>>>>>>>> >
> >>>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
> >>>>>>>>> > > If this is saying there's no extra space for metadata, is that why
> >>>>>>>>> > > adding more files often makes the system hang for 30-90s? Is there
> >>>>>>>>> > > anything I should do about that?
> >>>>>>>>> >
> >>>>>>>>> > I'm not sure about the hang though.
> >>>>>>>>> >
> >>>>>>>>> > It would be nice to give more info to diagnosis.
> >>>>>>>>> > The output of 'btrfs fi usage' is useful for space usage problem.
> >>>>>>>>> >
> >>>>>>>>> > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
> >>>>>>>>> > space in vanilla df command) space for btrfs.
> >>>>>>>>> >
> >>>>>>>>> > Thanks,
> >>>>>>>>> > Qu
> >>>>>>>>> >
> >>>>>>>>> > >
> >>>>>>>>> > > Thank you so much for all of your help. I love how flexible BTRFS is
> >>>>>>>>> > > but when things go wrong it's very hard for me to troubleshoot.
> >>>>>>>>> > >
> >>>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> >>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>>>> > >>
> >>>>>>>>> > >>
> >>>>>>>>> > >>
> >>>>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
> >>>>>>>>> > >>> Something went wrong:
> >>>>>>>>> > >>>
> >>>>>>>>> > >>> Reinitialize checksum tree
> >>>>>>>>> > >>> Unable to find block group for 0
> >>>>>>>>> > >>> Unable to find block group for 0
> >>>>>>>>> > >>> Unable to find block group for 0
> >>>>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
> >>>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
> >>>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
> >>>>>>>>> > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
> >>>>>>>>> > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
> >>>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
> >>>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
> >>>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
> >>>>>>>>> > >>>
> >>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
> >>>>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
> >>>>>>>>> > >>> Aborted
> >>>>>>>>> > >>
> >>>>>>>>> > >> This means no space for extra metadata...
> >>>>>>>>> > >>
> >>>>>>>>> > >> Anyway the csum tree problem shouldn't be a big thing, you
> >>>>>>>>> could leave
> >>>>>>>>> > >> it and call it a day.
> >>>>>>>>> > >>
> >>>>>>>>> > >> BTW, as long as btrfs check reports no extra problem for the inode
> >>>>>>>>> > >> generation, it should be pretty safe to use the fs.
> >>>>>>>>> > >>
> >>>>>>>>> > >> Thanks,
> >>>>>>>>> > >> Qu
> >>>>>>>>> > >>>
> >>>>>>>>> > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
> >>>>>>>>> > >>> available. I'll let that try overnight?
> >>>>>>>>> > >>>
> >>>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
> >>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>>>> > >>>>
> >>>>>>>>> > >>>>
> >>>>>>>>> > >>>>
> >>>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
> >>>>>>>>> > >>>>> Thank you for helping. The end result of the scan was:
> >>>>>>>>> > >>>>>
> >>>>>>>>> > >>>>>
> >>>>>>>>> > >>>>> [1/7] checking root items
> >>>>>>>>> > >>>>> [2/7] checking extents
> >>>>>>>>> > >>>>> [3/7] checking free space cache
> >>>>>>>>> > >>>>> [4/7] checking fs roots
> >>>>>>>>> > >>>>
> >>>>>>>>> > >>>> Good news is, your fs is still mostly fine.
> >>>>>>>>> > >>>>
> >>>>>>>>> > >>>>> [5/7] checking only csums items (without verifying data)
> >>>>>>>>> > >>>>> there are no extents for csum range 0-69632
> >>>>>>>>> > >>>>> csum exists for 0-69632 but there is no extent record
> >>>>>>>>> > >>>>> ...
> >>>>>>>>> > >>>>> ...
> >>>>>>>>> > >>>>> there are no extents for csum range 946692096-946827264
> >>>>>>>>> > >>>>> csum exists for 946692096-946827264 but there is no extent
> >>>>>>>>> record
> >>>>>>>>> > >>>>> there are no extents for csum range 946831360-947912704
> >>>>>>>>> > >>>>> csum exists for 946831360-947912704 but there is no extent
> >>>>>>>>> record
> >>>>>>>>> > >>>>> ERROR: errors found in csum tree
> >>>>>>>>> > >>>>
> >>>>>>>>> > >>>> Only extent tree is corrupted.
> >>>>>>>>> > >>>>
> >>>>>>>>> > >>>> Normally btrfs check --init-csum-tree should be able to
> >>>>>>>>> handle it.
> >>>>>>>>> > >>>>
> >>>>>>>>> > >>>> But still, please be sure you're using the latest btrfs-progs
> >>>>>>>>> to fix it.
> >>>>>>>>> > >>>>
> >>>>>>>>> > >>>> Thanks,
> >>>>>>>>> > >>>> Qu
> >>>>>>>>> > >>>>
> >>>>>>>>> > >>>>> [6/7] checking root refs
> >>>>>>>>> > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
> >>>>>>>>> > >>>>> found 44157956026368 bytes used, error(s) found
> >>>>>>>>> > >>>>> total csum bytes: 42038602716
> >>>>>>>>> > >>>>> total tree bytes: 49688616960
> >>>>>>>>> > >>>>> total fs tree bytes: 1256427520
> >>>>>>>>> > >>>>> total extent tree bytes: 1709105152
> >>>>>>>>> > >>>>> btree space waste bytes: 3172727316
> >>>>>>>>> > >>>>> file data blocks allocated: 261625653436416
> >>>>>>>>> > >>>>> referenced 47477768499200
> >>>>>>>>> > >>>>>
> >>>>>>>>> > >>>>> What do I need to do to fix all of this?
> >>>>>>>>> > >>>>>
> >>>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
> >>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>>>> > >>>>>>
> >>>>>>>>> > >>>>>>
> >>>>>>>>> > >>>>>>
> >>>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
> >>>>>>>>> > >>>>>>> Well, the repair doesn't look terribly successful.
> >>>>>>>>> > >>>>>>>
> >>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>> 6875841 found 6876224
> >>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>> 6875841 found 6876224
> >>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>> 6875841 found 6876224
> >>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>> item=84
> >>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>> > >>>>>>
> >>>>>>>>> > >>>>>> This means there are more problems, not only the hash name
> >>>>>>>>> mismatch.
> >>>>>>>>> > >>>>>>
> >>>>>>>>> > >>>>>> This means the fs is already corrupted, the name hash is
> >>>>>>>>> just one
> >>>>>>>>> > >>>>>> unrelated symptom.
> >>>>>>>>> > >>>>>>
> >>>>>>>>> > >>>>>> The only good news is, btrfs-progs abort the transaction,
> >>>>>>>>> thus no
> >>>>>>>>> > >>>>>> further damage to the fs.
> >>>>>>>>> > >>>>>>
> >>>>>>>>> > >>>>>> Please run a plain btrfs-check to show what's the problem
> >>>>>>>>> first.
> >>>>>>>>> > >>>>>>
> >>>>>>>>> > >>>>>> Thanks,
> >>>>>>>>> > >>>>>> Qu
> >>>>>>>>> > >>>>>>
> >>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>> 6875841 found 6876224
> >>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>> item=84
> >>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>> 6875841 found 6876224
> >>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>> item=84
> >>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>> 6875841 found 6876224
> >>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>> item=84
> >>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>> 6875841 found 6876224
> >>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>> item=84
> >>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>> 6875841 found 6876224
> >>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>> item=84
> >>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>> 6875841 found 6876224
> >>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>> item=84
> >>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>> 6875841 found 6876224
> >>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>> item=84
> >>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>> 6875841 found 6876224
> >>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>> item=84
> >>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>> 6875841 found 6876224
> >>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>> item=84
> >>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>> 6875841 found 6876224
> >>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>> item=84
> >>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>> > >>>>>>> ERROR: failed to zero log tree: -17
> >>>>>>>>> > >>>>>>> ERROR: attempt to start transaction over already running one
> >>>>>>>>> > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
> >>>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> >>>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> >>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>>>>>> 225049066086400 len 4096
> >>>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> >>>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> >>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>>>>>> 225049066094592 len 4096
> >>>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> >>>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> >>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>>>>>> 225049066102784 len 4096
> >>>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> >>>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> >>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>>>>>> 225049066131456 len 4096
> >>>>>>>>> > >>>>>>>
> >>>>>>>>> > >>>>>>> What is going on?
> >>>>>>>>> > >>>>>>>
> >>>>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond
> >>>>>>>>> <t.d.richmond@gmail.com <mailto:t.d.richmond@gmail.com>> wrote:
> >>>>>>>>> > >>>>>>>>
> >>>>>>>>> > >>>>>>>> Chris, I had used the correct mountpoint in the command.
> >>>>>>>>> I just edited
> >>>>>>>>> > >>>>>>>> it in the email to be /mountpoint for consistency.
> >>>>>>>>> > >>>>>>>>
> >>>>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
> >>>>>>>>> > >>>>>>>>
> >>>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
> >>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>>>> > >>>>>>>>>
> >>>>>>>>> > >>>>>>>>>
> >>>>>>>>> > >>>>>>>>>
> >>>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
> >>>>>>>>> > >>>>>>>>>> Hello,
> >>>>>>>>> > >>>>>>>>>>
> >>>>>>>>> > >>>>>>>>>> I looked up this error and it basically says ask a
> >>>>>>>>> developer to
> >>>>>>>>> > >>>>>>>>>> determine if it's a false error or not. I just started
> >>>>>>>>> getting some
> >>>>>>>>> > >>>>>>>>>> slow response times, and looked at the dmesg log to
> >>>>>>>>> find a ton of
> >>>>>>>>> > >>>>>>>>>> these errors.
> >>>>>>>>> > >>>>>>>>>>
> >>>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt
> >>>>>>>>> leaf: root=5
> >>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> >>>>>>>>> generation:
> >>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error (device sdh):
> >>>>>>>>> block=203510940835840 read
> >>>>>>>>> > >>>>>>>>>> time tree block corruption detected
> >>>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt
> >>>>>>>>> leaf: root=5
> >>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> >>>>>>>>> generation:
> >>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error (device sdh):
> >>>>>>>>> block=203510940835840 read
> >>>>>>>>> > >>>>>>>>>> time tree block corruption detected
> >>>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt
> >>>>>>>>> leaf: root=5
> >>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> >>>>>>>>> generation:
> >>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error (device sdh):
> >>>>>>>>> block=203510940835840 read
> >>>>>>>>> > >>>>>>>>>> time tree block corruption detected
> >>>>>>>>> > >>>>>>>>>>
> >>>>>>>>> > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
> >>>>>>>>> > >>>>>>>>>>
> >>>>>>>>> > >>>>>>>>>> Is there anything I should do about this, or should I
> >>>>>>>>> just continue
> >>>>>>>>> > >>>>>>>>>> using my array as normal?
> >>>>>>>>> > >>>>>>>>>
> >>>>>>>>> > >>>>>>>>> This is caused by older kernel underflow inode generation.
> >>>>>>>>> > >>>>>>>>>
> >>>>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
> >>>>>>>>> > >>>>>>>>>
> >>>>>>>>> > >>>>>>>>> Or you can go safer, by manually locating the inode
> >>>>>>>>> using its inode
> >>>>>>>>> > >>>>>>>>> number (1311670), and copy it to some new location using
> >>>>>>>>> previous
> >>>>>>>>> > >>>>>>>>> working kernel, then delete the old file, copy the new
> >>>>>>>>> one back to fix it.
> >>>>>>>>> > >>>>>>>>>
> >>>>>>>>> > >>>>>>>>> Thanks,
> >>>>>>>>> > >>>>>>>>> Qu
> >>>>>>>>> > >>>>>>>>>
> >>>>>>>>> > >>>>>>>>>>
> >>>>>>>>> > >>>>>>>>>> Thank you!
> >>>>>>>>> > >>>>>>>>>>
> >>>>>>>>> > >>>>>>>>>
> >>>>>>>>> > >>>>>>
> >>>>>>>>> > >>>>
> >>>>>>>>> > >>
> >>>>>>>>> >
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>>
> >>
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-08-25 5:25 ` Tyler Richmond
@ 2020-08-25 6:37 ` Qu Wenruo
2020-08-25 13:30 ` Tyler Richmond
0 siblings, 1 reply; 56+ messages in thread
From: Qu Wenruo @ 2020-08-25 6:37 UTC (permalink / raw)
To: Tyler Richmond; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 25385 bytes --]
On 2020/8/25 下午1:25, Tyler Richmond wrote:
> Qu,
>
> Yes, it's btrfs-progs 5.7. Here is the result of the lowmem check:
>
> https://pastebin.com/8Tzx23EX
That doesn't detect any inode generation problem at all, which is not a
good sign.
Would you also pvode the dump for the offending block?
> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
has 18446744073709551492 expect [0, 6875827]
For this case, would you please provide the tree dump of "203510940835840" ?
# btrfs ins dump-tree -b 203510940835840 <device>
And, since btrfs-image can't dump with regular extent tree, the "-w"
dump would also help.
Thanks,
Qu
> Thanks!
>
> On Mon, Aug 24, 2020 at 4:26 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2020/8/24 上午10:47, Tyler Richmond wrote:
>>> Qu,
>>>
>>> Finally finished another repair and captured the output.
>>>
>>> https://pastebin.com/ffcbwvd8
>>>
>>> Does that show you what you need? Or should I still do one in lowmem mode?
>>
>> Lowmem mode (no need for --repair) is recommended since original mode
>> doesn't detect the inode generation problem.
>>
>> And it's already btrfs-progs v5.7 right?
>>
>> THanks,
>> Qu
>>>
>>> Thanks for your help!
>>>
>>> On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>
>>>>
>>>>
>>>> On 2020/8/23 上午10:49, Tyler Richmond wrote:
>>>>> Well, I can guarantee that I didn't create this fs before 2015 (just
>>>>> checked the order confirmation from when I bought the server), but I
>>>>> may have just used whatever was in the Ubuntu package manager at the
>>>>> time. So maybe I don't have a v0 ref?
>>>>
>>>> Then btrfs-image shouldn't report that.
>>>>
>>>> There is an item smaller than any valid btrfs item, normally it means
>>>> it's a v0 ref.
>>>> If not, then it could be a bigger problem.
>>>>
>>>> Could you please provide the full btrfs-check output?
>>>> Also, if possible result from "btrfs check --mode=lowmem" would also help.
>>>>
>>>> Also, if you really go "--repair", then the full output would also be
>>>> needed to determine what's going wrong.
>>>> There is a report about "btrfs check --repair" didn't repair the inode
>>>> generation, if that's the case we must have a bug then.
>>>>
>>>> Thanks,
>>>> Qu
>>>>>
>>>>> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
>>>>>>>> Is my best bet just to downgrade the kernel and then try to delete the
>>>>>>>> broken files? Or should I rebuild from scratch? Just don't know
>>>>>>>> whether it's worth the time to try and figure this out or if the
>>>>>>>> problems stem from the FS being too old and it's beyond trying to
>>>>>>>> repair.
>>>>>>>
>>>>>>> All invalid inode generations, should be able to be repaired by latest
>>>>>>> btrfs-check.
>>>>>>>
>>>>>>> If not, please provide the btrfs-image dump for us to determine what's
>>>>>>> going wrong.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Qu
>>>>>>>>
>>>>>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> I didn't check dmesg during the btrfs check, but that was the only
>>>>>>>>> output during the rm -f before it was forced readonly. I just checked
>>>>>>>>> dmesg for inode generation values, and there are a lot of them.
>>>>>>>>>
>>>>>>>>> https://pastebin.com/stZdN0ta
>>>>>>>>> The dmesg output had 990 lines containing inode generation.
>>>>>>>>>
>>>>>>>>> However, these were at least later. I tried to do a btrfs balance
>>>>>>>>> -mconvert raid1 and it failed with an I/O error. That is probably what
>>>>>>>>> generated these specific errors, but maybe they were also happening
>>>>>>>>> during the btrfs repair.
>>>>>>>>>
>>>>>>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway with:
>>>>>>>>> ERROR: either extent tree is corrupted or deprecated extent ref format
>>>>>>>>> ERROR: create failed: -5
>>>>>>
>>>>>> Oh, forgot this part.
>>>>>>
>>>>>> This means you have v0 ref?!
>>>>>>
>>>>>> Then the fs is too old, no progs/kernel support after all.
>>>>>>
>>>>>> In that case, please rollback to the last working kernel and copy your data.
>>>>>>
>>>>>> In fact, that v0 ref should only be in the code base for several weeks
>>>>>> before 2010, thus it's really too old.
>>>>>>
>>>>>> The good news is, with tree-checker, we should never experience such
>>>>>> too-old-to-be-usable problem (at least I hope so)
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
>>>>>>>>>>> Qu,
>>>>>>>>>>>
>>>>>>>>>>> Sorry to resurrect this thread, but I just ran into something that I
>>>>>>>>>>> can't really just ignore. I've found a folder that is full of files
>>>>>>>>>>> which I guess have been broken somehow. I found a backup and restored
>>>>>>>>>>> them, but I want to delete this folder of broken files. But whenever I
>>>>>>>>>>> try, the fs is forced into readonly mode again. I just finished another
>>>>>>>>>>> btrfs check --repair but it didn't fix the problem.
>>>>>>>>>>>
>>>>>>>>>>> https://pastebin.com/eTV3s3fr
>>>>>>>>>>
>>>>>>>>>> Is that the full output?
>>>>>>>>>>
>>>>>>>>>> No inode generation bugs?
>>>>>>>>>>>
>>>>>>>>>>> I'm already on btrfs-progs v5.7. Any new suggestions?
>>>>>>>>>>
>>>>>>>>>> Strange.
>>>>>>>>>>
>>>>>>>>>> The detection and repair should have been merged into v5.5.
>>>>>>>>>>
>>>>>>>>>> If your fs is small enough, would you please provide the "btrfs-image
>>>>>>>>>> -c9" dump?
>>>>>>>>>>
>>>>>>>>>> It would contain the filenames and directories names, but doesn't
>>>>>>>>>> contain file contents.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Qu
>>>>>>>>>>>
>>>>>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com
>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> 5.6.1 also failed the same way. Here's the usage output. This is the
>>>>>>>>>>> part where you see I've been using RAID5 haha
>>>>>>>>>>>
>>>>>>>>>>> WARNING: RAID56 detected, not implemented
>>>>>>>>>>> Overall:
>>>>>>>>>>> Device size: 60.03TiB
>>>>>>>>>>> Device allocated: 98.06GiB
>>>>>>>>>>> Device unallocated: 59.93TiB
>>>>>>>>>>> Device missing: 0.00B
>>>>>>>>>>> Used: 92.56GiB
>>>>>>>>>>> Free (estimated): 0.00B (min: 8.00EiB)
>>>>>>>>>>> Data ratio: 0.00
>>>>>>>>>>> Metadata ratio: 2.00
>>>>>>>>>>> Global reserve: 512.00MiB (used: 0.00B)
>>>>>>>>>>> Multiple profiles: no
>>>>>>>>>>>
>>>>>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
>>>>>>>>>>> /dev/sdh 8.07TiB
>>>>>>>>>>> /dev/sdf 8.07TiB
>>>>>>>>>>> /dev/sdg 8.07TiB
>>>>>>>>>>> /dev/sdd 8.07TiB
>>>>>>>>>>> /dev/sdc 8.07TiB
>>>>>>>>>>> /dev/sde 8.07TiB
>>>>>>>>>>>
>>>>>>>>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
>>>>>>>>>>> /dev/sdh 34.00GiB
>>>>>>>>>>> /dev/sdf 32.00GiB
>>>>>>>>>>> /dev/sdg 32.00GiB
>>>>>>>>>>>
>>>>>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
>>>>>>>>>>> /dev/sdf 32.00MiB
>>>>>>>>>>> /dev/sdg 32.00MiB
>>>>>>>>>>>
>>>>>>>>>>> Unallocated:
>>>>>>>>>>> /dev/sdh 2.81TiB
>>>>>>>>>>> /dev/sdf 2.81TiB
>>>>>>>>>>> /dev/sdg 2.81TiB
>>>>>>>>>>> /dev/sdd 1.03TiB
>>>>>>>>>>> /dev/sdc 1.03TiB
>>>>>>>>>>> /dev/sde 1.03TiB
>>>>>>>>>>>
>>>>>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com
>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
>>>>>>>>>>> > > If this is saying there's no extra space for metadata, is that why
>>>>>>>>>>> > > adding more files often makes the system hang for 30-90s? Is there
>>>>>>>>>>> > > anything I should do about that?
>>>>>>>>>>> >
>>>>>>>>>>> > I'm not sure about the hang though.
>>>>>>>>>>> >
>>>>>>>>>>> > It would be nice to give more info to diagnosis.
>>>>>>>>>>> > The output of 'btrfs fi usage' is useful for space usage problem.
>>>>>>>>>>> >
>>>>>>>>>>> > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
>>>>>>>>>>> > space in vanilla df command) space for btrfs.
>>>>>>>>>>> >
>>>>>>>>>>> > Thanks,
>>>>>>>>>>> > Qu
>>>>>>>>>>> >
>>>>>>>>>>> > >
>>>>>>>>>>> > > Thank you so much for all of your help. I love how flexible BTRFS is
>>>>>>>>>>> > > but when things go wrong it's very hard for me to troubleshoot.
>>>>>>>>>>> > >
>>>>>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com
>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>> > >>
>>>>>>>>>>> > >>
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
>>>>>>>>>>> > >>> Something went wrong:
>>>>>>>>>>> > >>>
>>>>>>>>>>> > >>> Reinitialize checksum tree
>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
>>>>>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
>>>>>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
>>>>>>>>>>> > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
>>>>>>>>>>> > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
>>>>>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
>>>>>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
>>>>>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
>>>>>>>>>>> > >>>
>>>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
>>>>>>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
>>>>>>>>>>> > >>> Aborted
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> This means no space for extra metadata...
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> Anyway the csum tree problem shouldn't be a big thing, you
>>>>>>>>>>> could leave
>>>>>>>>>>> > >> it and call it a day.
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> BTW, as long as btrfs check reports no extra problem for the inode
>>>>>>>>>>> > >> generation, it should be pretty safe to use the fs.
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> Thanks,
>>>>>>>>>>> > >> Qu
>>>>>>>>>>> > >>>
>>>>>>>>>>> > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
>>>>>>>>>>> > >>> available. I'll let that try overnight?
>>>>>>>>>>> > >>>
>>>>>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
>>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
>>>>>>>>>>> > >>>>> Thank you for helping. The end result of the scan was:
>>>>>>>>>>> > >>>>>
>>>>>>>>>>> > >>>>>
>>>>>>>>>>> > >>>>> [1/7] checking root items
>>>>>>>>>>> > >>>>> [2/7] checking extents
>>>>>>>>>>> > >>>>> [3/7] checking free space cache
>>>>>>>>>>> > >>>>> [4/7] checking fs roots
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>> Good news is, your fs is still mostly fine.
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>>> [5/7] checking only csums items (without verifying data)
>>>>>>>>>>> > >>>>> there are no extents for csum range 0-69632
>>>>>>>>>>> > >>>>> csum exists for 0-69632 but there is no extent record
>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>> > >>>>> there are no extents for csum range 946692096-946827264
>>>>>>>>>>> > >>>>> csum exists for 946692096-946827264 but there is no extent
>>>>>>>>>>> record
>>>>>>>>>>> > >>>>> there are no extents for csum range 946831360-947912704
>>>>>>>>>>> > >>>>> csum exists for 946831360-947912704 but there is no extent
>>>>>>>>>>> record
>>>>>>>>>>> > >>>>> ERROR: errors found in csum tree
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>> Only extent tree is corrupted.
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>> Normally btrfs check --init-csum-tree should be able to
>>>>>>>>>>> handle it.
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>> But still, please be sure you're using the latest btrfs-progs
>>>>>>>>>>> to fix it.
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>> Thanks,
>>>>>>>>>>> > >>>> Qu
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>>> [6/7] checking root refs
>>>>>>>>>>> > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>>>>>>>>>> > >>>>> found 44157956026368 bytes used, error(s) found
>>>>>>>>>>> > >>>>> total csum bytes: 42038602716
>>>>>>>>>>> > >>>>> total tree bytes: 49688616960
>>>>>>>>>>> > >>>>> total fs tree bytes: 1256427520
>>>>>>>>>>> > >>>>> total extent tree bytes: 1709105152
>>>>>>>>>>> > >>>>> btree space waste bytes: 3172727316
>>>>>>>>>>> > >>>>> file data blocks allocated: 261625653436416
>>>>>>>>>>> > >>>>> referenced 47477768499200
>>>>>>>>>>> > >>>>>
>>>>>>>>>>> > >>>>> What do I need to do to fix all of this?
>>>>>>>>>>> > >>>>>
>>>>>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
>>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
>>>>>>>>>>> > >>>>>>> Well, the repair doesn't look terribly successful.
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>> item=84
>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>> This means there are more problems, not only the hash name
>>>>>>>>>>> mismatch.
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>> This means the fs is already corrupted, the name hash is
>>>>>>>>>>> just one
>>>>>>>>>>> > >>>>>> unrelated symptom.
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>> The only good news is, btrfs-progs abort the transaction,
>>>>>>>>>>> thus no
>>>>>>>>>>> > >>>>>> further damage to the fs.
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>> Please run a plain btrfs-check to show what's the problem
>>>>>>>>>>> first.
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>> > >>>>>> Qu
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>> item=84
>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>> item=84
>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>> item=84
>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>> item=84
>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>> item=84
>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>> item=84
>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>> item=84
>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>> item=84
>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>> item=84
>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>> item=84
>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>> > >>>>>>> ERROR: failed to zero log tree: -17
>>>>>>>>>>> > >>>>>>> ERROR: attempt to start transaction over already running one
>>>>>>>>>>> > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > >>>>>>> What is going on?
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond
>>>>>>>>>>> <t.d.richmond@gmail.com <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>> > >>>>>>>> Chris, I had used the correct mountpoint in the command.
>>>>>>>>>>> I just edited
>>>>>>>>>>> > >>>>>>>> it in the email to be /mountpoint for consistency.
>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
>>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
>>>>>>>>>>> > >>>>>>>>>> Hello,
>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>> > >>>>>>>>>> I looked up this error and it basically says ask a
>>>>>>>>>>> developer to
>>>>>>>>>>> > >>>>>>>>>> determine if it's a false error or not. I just started
>>>>>>>>>>> getting some
>>>>>>>>>>> > >>>>>>>>>> slow response times, and looked at the dmesg log to
>>>>>>>>>>> find a ton of
>>>>>>>>>>> > >>>>>>>>>> these errors.
>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt
>>>>>>>>>>> leaf: root=5
>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>>> generation:
>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error (device sdh):
>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt
>>>>>>>>>>> leaf: root=5
>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>>> generation:
>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error (device sdh):
>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt
>>>>>>>>>>> leaf: root=5
>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>>> generation:
>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error (device sdh):
>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>> > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>> > >>>>>>>>>> Is there anything I should do about this, or should I
>>>>>>>>>>> just continue
>>>>>>>>>>> > >>>>>>>>>> using my array as normal?
>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>> > >>>>>>>>> This is caused by older kernel underflow inode generation.
>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>> > >>>>>>>>> Or you can go safer, by manually locating the inode
>>>>>>>>>>> using its inode
>>>>>>>>>>> > >>>>>>>>> number (1311670), and copy it to some new location using
>>>>>>>>>>> previous
>>>>>>>>>>> > >>>>>>>>> working kernel, then delete the old file, copy the new
>>>>>>>>>>> one back to fix it.
>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>>>>> > >>>>>>>>> Qu
>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>> > >>>>>>>>>> Thank you!
>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>
>>>>>>>>>>> >
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-08-25 6:37 ` Qu Wenruo
@ 2020-08-25 13:30 ` Tyler Richmond
2020-08-25 13:38 ` Qu Wenruo
0 siblings, 1 reply; 56+ messages in thread
From: Tyler Richmond @ 2020-08-25 13:30 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Btrfs BTRFS
Qu,
The dump of the block is:
https://pastebin.com/ran85JJv
I've also completed the btrfs-image, but it's almost 50gb. What's the
best way to get it to you? Also, does it work with -ss or are the
original filenames important?
Thanks again!
On Tue, Aug 25, 2020 at 2:37 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2020/8/25 下午1:25, Tyler Richmond wrote:
> > Qu,
> >
> > Yes, it's btrfs-progs 5.7. Here is the result of the lowmem check:
> >
> > https://pastebin.com/8Tzx23EX
>
> That doesn't detect any inode generation problem at all, which is not a
> good sign.
>
> Would you also pvode the dump for the offending block?
>
> > block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> has 18446744073709551492 expect [0, 6875827]
>
> For this case, would you please provide the tree dump of "203510940835840" ?
>
> # btrfs ins dump-tree -b 203510940835840 <device>
>
> And, since btrfs-image can't dump with regular extent tree, the "-w"
> dump would also help.
>
> Thanks,
> Qu
>
>
> > Thanks!
> >
> > On Mon, Aug 24, 2020 at 4:26 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>
> >>
> >>
> >> On 2020/8/24 上午10:47, Tyler Richmond wrote:
> >>> Qu,
> >>>
> >>> Finally finished another repair and captured the output.
> >>>
> >>> https://pastebin.com/ffcbwvd8
> >>>
> >>> Does that show you what you need? Or should I still do one in lowmem mode?
> >>
> >> Lowmem mode (no need for --repair) is recommended since original mode
> >> doesn't detect the inode generation problem.
> >>
> >> And it's already btrfs-progs v5.7 right?
> >>
> >> THanks,
> >> Qu
> >>>
> >>> Thanks for your help!
> >>>
> >>> On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 2020/8/23 上午10:49, Tyler Richmond wrote:
> >>>>> Well, I can guarantee that I didn't create this fs before 2015 (just
> >>>>> checked the order confirmation from when I bought the server), but I
> >>>>> may have just used whatever was in the Ubuntu package manager at the
> >>>>> time. So maybe I don't have a v0 ref?
> >>>>
> >>>> Then btrfs-image shouldn't report that.
> >>>>
> >>>> There is an item smaller than any valid btrfs item, normally it means
> >>>> it's a v0 ref.
> >>>> If not, then it could be a bigger problem.
> >>>>
> >>>> Could you please provide the full btrfs-check output?
> >>>> Also, if possible result from "btrfs check --mode=lowmem" would also help.
> >>>>
> >>>> Also, if you really go "--repair", then the full output would also be
> >>>> needed to determine what's going wrong.
> >>>> There is a report about "btrfs check --repair" didn't repair the inode
> >>>> generation, if that's the case we must have a bug then.
> >>>>
> >>>> Thanks,
> >>>> Qu
> >>>>>
> >>>>> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
> >>>>>>>> Is my best bet just to downgrade the kernel and then try to delete the
> >>>>>>>> broken files? Or should I rebuild from scratch? Just don't know
> >>>>>>>> whether it's worth the time to try and figure this out or if the
> >>>>>>>> problems stem from the FS being too old and it's beyond trying to
> >>>>>>>> repair.
> >>>>>>>
> >>>>>>> All invalid inode generations, should be able to be repaired by latest
> >>>>>>> btrfs-check.
> >>>>>>>
> >>>>>>> If not, please provide the btrfs-image dump for us to determine what's
> >>>>>>> going wrong.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Qu
> >>>>>>>>
> >>>>>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> I didn't check dmesg during the btrfs check, but that was the only
> >>>>>>>>> output during the rm -f before it was forced readonly. I just checked
> >>>>>>>>> dmesg for inode generation values, and there are a lot of them.
> >>>>>>>>>
> >>>>>>>>> https://pastebin.com/stZdN0ta
> >>>>>>>>> The dmesg output had 990 lines containing inode generation.
> >>>>>>>>>
> >>>>>>>>> However, these were at least later. I tried to do a btrfs balance
> >>>>>>>>> -mconvert raid1 and it failed with an I/O error. That is probably what
> >>>>>>>>> generated these specific errors, but maybe they were also happening
> >>>>>>>>> during the btrfs repair.
> >>>>>>>>>
> >>>>>>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway with:
> >>>>>>>>> ERROR: either extent tree is corrupted or deprecated extent ref format
> >>>>>>>>> ERROR: create failed: -5
> >>>>>>
> >>>>>> Oh, forgot this part.
> >>>>>>
> >>>>>> This means you have v0 ref?!
> >>>>>>
> >>>>>> Then the fs is too old, no progs/kernel support after all.
> >>>>>>
> >>>>>> In that case, please rollback to the last working kernel and copy your data.
> >>>>>>
> >>>>>> In fact, that v0 ref should only be in the code base for several weeks
> >>>>>> before 2010, thus it's really too old.
> >>>>>>
> >>>>>> The good news is, with tree-checker, we should never experience such
> >>>>>> too-old-to-be-usable problem (at least I hope so)
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Qu
> >>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
> >>>>>>>>>>> Qu,
> >>>>>>>>>>>
> >>>>>>>>>>> Sorry to resurrect this thread, but I just ran into something that I
> >>>>>>>>>>> can't really just ignore. I've found a folder that is full of files
> >>>>>>>>>>> which I guess have been broken somehow. I found a backup and restored
> >>>>>>>>>>> them, but I want to delete this folder of broken files. But whenever I
> >>>>>>>>>>> try, the fs is forced into readonly mode again. I just finished another
> >>>>>>>>>>> btrfs check --repair but it didn't fix the problem.
> >>>>>>>>>>>
> >>>>>>>>>>> https://pastebin.com/eTV3s3fr
> >>>>>>>>>>
> >>>>>>>>>> Is that the full output?
> >>>>>>>>>>
> >>>>>>>>>> No inode generation bugs?
> >>>>>>>>>>>
> >>>>>>>>>>> I'm already on btrfs-progs v5.7. Any new suggestions?
> >>>>>>>>>>
> >>>>>>>>>> Strange.
> >>>>>>>>>>
> >>>>>>>>>> The detection and repair should have been merged into v5.5.
> >>>>>>>>>>
> >>>>>>>>>> If your fs is small enough, would you please provide the "btrfs-image
> >>>>>>>>>> -c9" dump?
> >>>>>>>>>>
> >>>>>>>>>> It would contain the filenames and directories names, but doesn't
> >>>>>>>>>> contain file contents.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Qu
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com
> >>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> 5.6.1 also failed the same way. Here's the usage output. This is the
> >>>>>>>>>>> part where you see I've been using RAID5 haha
> >>>>>>>>>>>
> >>>>>>>>>>> WARNING: RAID56 detected, not implemented
> >>>>>>>>>>> Overall:
> >>>>>>>>>>> Device size: 60.03TiB
> >>>>>>>>>>> Device allocated: 98.06GiB
> >>>>>>>>>>> Device unallocated: 59.93TiB
> >>>>>>>>>>> Device missing: 0.00B
> >>>>>>>>>>> Used: 92.56GiB
> >>>>>>>>>>> Free (estimated): 0.00B (min: 8.00EiB)
> >>>>>>>>>>> Data ratio: 0.00
> >>>>>>>>>>> Metadata ratio: 2.00
> >>>>>>>>>>> Global reserve: 512.00MiB (used: 0.00B)
> >>>>>>>>>>> Multiple profiles: no
> >>>>>>>>>>>
> >>>>>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
> >>>>>>>>>>> /dev/sdh 8.07TiB
> >>>>>>>>>>> /dev/sdf 8.07TiB
> >>>>>>>>>>> /dev/sdg 8.07TiB
> >>>>>>>>>>> /dev/sdd 8.07TiB
> >>>>>>>>>>> /dev/sdc 8.07TiB
> >>>>>>>>>>> /dev/sde 8.07TiB
> >>>>>>>>>>>
> >>>>>>>>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
> >>>>>>>>>>> /dev/sdh 34.00GiB
> >>>>>>>>>>> /dev/sdf 32.00GiB
> >>>>>>>>>>> /dev/sdg 32.00GiB
> >>>>>>>>>>>
> >>>>>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
> >>>>>>>>>>> /dev/sdf 32.00MiB
> >>>>>>>>>>> /dev/sdg 32.00MiB
> >>>>>>>>>>>
> >>>>>>>>>>> Unallocated:
> >>>>>>>>>>> /dev/sdh 2.81TiB
> >>>>>>>>>>> /dev/sdf 2.81TiB
> >>>>>>>>>>> /dev/sdg 2.81TiB
> >>>>>>>>>>> /dev/sdd 1.03TiB
> >>>>>>>>>>> /dev/sdc 1.03TiB
> >>>>>>>>>>> /dev/sde 1.03TiB
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> >>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>>>>>> >
> >>>>>>>>>>> >
> >>>>>>>>>>> >
> >>>>>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
> >>>>>>>>>>> > > If this is saying there's no extra space for metadata, is that why
> >>>>>>>>>>> > > adding more files often makes the system hang for 30-90s? Is there
> >>>>>>>>>>> > > anything I should do about that?
> >>>>>>>>>>> >
> >>>>>>>>>>> > I'm not sure about the hang though.
> >>>>>>>>>>> >
> >>>>>>>>>>> > It would be nice to give more info to diagnosis.
> >>>>>>>>>>> > The output of 'btrfs fi usage' is useful for space usage problem.
> >>>>>>>>>>> >
> >>>>>>>>>>> > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
> >>>>>>>>>>> > space in vanilla df command) space for btrfs.
> >>>>>>>>>>> >
> >>>>>>>>>>> > Thanks,
> >>>>>>>>>>> > Qu
> >>>>>>>>>>> >
> >>>>>>>>>>> > >
> >>>>>>>>>>> > > Thank you so much for all of your help. I love how flexible BTRFS is
> >>>>>>>>>>> > > but when things go wrong it's very hard for me to troubleshoot.
> >>>>>>>>>>> > >
> >>>>>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> >>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>>>>>> > >>
> >>>>>>>>>>> > >>
> >>>>>>>>>>> > >>
> >>>>>>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
> >>>>>>>>>>> > >>> Something went wrong:
> >>>>>>>>>>> > >>>
> >>>>>>>>>>> > >>> Reinitialize checksum tree
> >>>>>>>>>>> > >>> Unable to find block group for 0
> >>>>>>>>>>> > >>> Unable to find block group for 0
> >>>>>>>>>>> > >>> Unable to find block group for 0
> >>>>>>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
> >>>>>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
> >>>>>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
> >>>>>>>>>>> > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
> >>>>>>>>>>> > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
> >>>>>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
> >>>>>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
> >>>>>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
> >>>>>>>>>>> > >>>
> >>>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
> >>>>>>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
> >>>>>>>>>>> > >>> Aborted
> >>>>>>>>>>> > >>
> >>>>>>>>>>> > >> This means no space for extra metadata...
> >>>>>>>>>>> > >>
> >>>>>>>>>>> > >> Anyway the csum tree problem shouldn't be a big thing, you
> >>>>>>>>>>> could leave
> >>>>>>>>>>> > >> it and call it a day.
> >>>>>>>>>>> > >>
> >>>>>>>>>>> > >> BTW, as long as btrfs check reports no extra problem for the inode
> >>>>>>>>>>> > >> generation, it should be pretty safe to use the fs.
> >>>>>>>>>>> > >>
> >>>>>>>>>>> > >> Thanks,
> >>>>>>>>>>> > >> Qu
> >>>>>>>>>>> > >>>
> >>>>>>>>>>> > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
> >>>>>>>>>>> > >>> available. I'll let that try overnight?
> >>>>>>>>>>> > >>>
> >>>>>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
> >>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>>>>>> > >>>>
> >>>>>>>>>>> > >>>>
> >>>>>>>>>>> > >>>>
> >>>>>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
> >>>>>>>>>>> > >>>>> Thank you for helping. The end result of the scan was:
> >>>>>>>>>>> > >>>>>
> >>>>>>>>>>> > >>>>>
> >>>>>>>>>>> > >>>>> [1/7] checking root items
> >>>>>>>>>>> > >>>>> [2/7] checking extents
> >>>>>>>>>>> > >>>>> [3/7] checking free space cache
> >>>>>>>>>>> > >>>>> [4/7] checking fs roots
> >>>>>>>>>>> > >>>>
> >>>>>>>>>>> > >>>> Good news is, your fs is still mostly fine.
> >>>>>>>>>>> > >>>>
> >>>>>>>>>>> > >>>>> [5/7] checking only csums items (without verifying data)
> >>>>>>>>>>> > >>>>> there are no extents for csum range 0-69632
> >>>>>>>>>>> > >>>>> csum exists for 0-69632 but there is no extent record
> >>>>>>>>>>> > >>>>> ...
> >>>>>>>>>>> > >>>>> ...
> >>>>>>>>>>> > >>>>> there are no extents for csum range 946692096-946827264
> >>>>>>>>>>> > >>>>> csum exists for 946692096-946827264 but there is no extent
> >>>>>>>>>>> record
> >>>>>>>>>>> > >>>>> there are no extents for csum range 946831360-947912704
> >>>>>>>>>>> > >>>>> csum exists for 946831360-947912704 but there is no extent
> >>>>>>>>>>> record
> >>>>>>>>>>> > >>>>> ERROR: errors found in csum tree
> >>>>>>>>>>> > >>>>
> >>>>>>>>>>> > >>>> Only extent tree is corrupted.
> >>>>>>>>>>> > >>>>
> >>>>>>>>>>> > >>>> Normally btrfs check --init-csum-tree should be able to
> >>>>>>>>>>> handle it.
> >>>>>>>>>>> > >>>>
> >>>>>>>>>>> > >>>> But still, please be sure you're using the latest btrfs-progs
> >>>>>>>>>>> to fix it.
> >>>>>>>>>>> > >>>>
> >>>>>>>>>>> > >>>> Thanks,
> >>>>>>>>>>> > >>>> Qu
> >>>>>>>>>>> > >>>>
> >>>>>>>>>>> > >>>>> [6/7] checking root refs
> >>>>>>>>>>> > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
> >>>>>>>>>>> > >>>>> found 44157956026368 bytes used, error(s) found
> >>>>>>>>>>> > >>>>> total csum bytes: 42038602716
> >>>>>>>>>>> > >>>>> total tree bytes: 49688616960
> >>>>>>>>>>> > >>>>> total fs tree bytes: 1256427520
> >>>>>>>>>>> > >>>>> total extent tree bytes: 1709105152
> >>>>>>>>>>> > >>>>> btree space waste bytes: 3172727316
> >>>>>>>>>>> > >>>>> file data blocks allocated: 261625653436416
> >>>>>>>>>>> > >>>>> referenced 47477768499200
> >>>>>>>>>>> > >>>>>
> >>>>>>>>>>> > >>>>> What do I need to do to fix all of this?
> >>>>>>>>>>> > >>>>>
> >>>>>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
> >>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>>>>>> > >>>>>>
> >>>>>>>>>>> > >>>>>>
> >>>>>>>>>>> > >>>>>>
> >>>>>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
> >>>>>>>>>>> > >>>>>>> Well, the repair doesn't look terribly successful.
> >>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>> item=84
> >>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>> > >>>>>>
> >>>>>>>>>>> > >>>>>> This means there are more problems, not only the hash name
> >>>>>>>>>>> mismatch.
> >>>>>>>>>>> > >>>>>>
> >>>>>>>>>>> > >>>>>> This means the fs is already corrupted, the name hash is
> >>>>>>>>>>> just one
> >>>>>>>>>>> > >>>>>> unrelated symptom.
> >>>>>>>>>>> > >>>>>>
> >>>>>>>>>>> > >>>>>> The only good news is, btrfs-progs abort the transaction,
> >>>>>>>>>>> thus no
> >>>>>>>>>>> > >>>>>> further damage to the fs.
> >>>>>>>>>>> > >>>>>>
> >>>>>>>>>>> > >>>>>> Please run a plain btrfs-check to show what's the problem
> >>>>>>>>>>> first.
> >>>>>>>>>>> > >>>>>>
> >>>>>>>>>>> > >>>>>> Thanks,
> >>>>>>>>>>> > >>>>>> Qu
> >>>>>>>>>>> > >>>>>>
> >>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>> item=84
> >>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>> item=84
> >>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>> item=84
> >>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>> item=84
> >>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>> item=84
> >>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>> item=84
> >>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>> item=84
> >>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>> item=84
> >>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>> item=84
> >>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>> item=84
> >>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>> > >>>>>>> ERROR: failed to zero log tree: -17
> >>>>>>>>>>> > >>>>>>> ERROR: attempt to start transaction over already running one
> >>>>>>>>>>> > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
> >>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> >>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> >>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>>>>>>>> 225049066086400 len 4096
> >>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> >>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> >>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>>>>>>>> 225049066094592 len 4096
> >>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> >>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> >>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>>>>>>>> 225049066102784 len 4096
> >>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> >>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> >>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>>>>>>>> 225049066131456 len 4096
> >>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>> > >>>>>>> What is going on?
> >>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond
> >>>>>>>>>>> <t.d.richmond@gmail.com <mailto:t.d.richmond@gmail.com>> wrote:
> >>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>> > >>>>>>>> Chris, I had used the correct mountpoint in the command.
> >>>>>>>>>>> I just edited
> >>>>>>>>>>> > >>>>>>>> it in the email to be /mountpoint for consistency.
> >>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
> >>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
> >>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
> >>>>>>>>>>> > >>>>>>>>>> Hello,
> >>>>>>>>>>> > >>>>>>>>>>
> >>>>>>>>>>> > >>>>>>>>>> I looked up this error and it basically says ask a
> >>>>>>>>>>> developer to
> >>>>>>>>>>> > >>>>>>>>>> determine if it's a false error or not. I just started
> >>>>>>>>>>> getting some
> >>>>>>>>>>> > >>>>>>>>>> slow response times, and looked at the dmesg log to
> >>>>>>>>>>> find a ton of
> >>>>>>>>>>> > >>>>>>>>>> these errors.
> >>>>>>>>>>> > >>>>>>>>>>
> >>>>>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt
> >>>>>>>>>>> leaf: root=5
> >>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> >>>>>>>>>>> generation:
> >>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error (device sdh):
> >>>>>>>>>>> block=203510940835840 read
> >>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
> >>>>>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt
> >>>>>>>>>>> leaf: root=5
> >>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> >>>>>>>>>>> generation:
> >>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error (device sdh):
> >>>>>>>>>>> block=203510940835840 read
> >>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
> >>>>>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt
> >>>>>>>>>>> leaf: root=5
> >>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> >>>>>>>>>>> generation:
> >>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error (device sdh):
> >>>>>>>>>>> block=203510940835840 read
> >>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
> >>>>>>>>>>> > >>>>>>>>>>
> >>>>>>>>>>> > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
> >>>>>>>>>>> > >>>>>>>>>>
> >>>>>>>>>>> > >>>>>>>>>> Is there anything I should do about this, or should I
> >>>>>>>>>>> just continue
> >>>>>>>>>>> > >>>>>>>>>> using my array as normal?
> >>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>> > >>>>>>>>> This is caused by older kernel underflow inode generation.
> >>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
> >>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>> > >>>>>>>>> Or you can go safer, by manually locating the inode
> >>>>>>>>>>> using its inode
> >>>>>>>>>>> > >>>>>>>>> number (1311670), and copy it to some new location using
> >>>>>>>>>>> previous
> >>>>>>>>>>> > >>>>>>>>> working kernel, then delete the old file, copy the new
> >>>>>>>>>>> one back to fix it.
> >>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>> > >>>>>>>>> Thanks,
> >>>>>>>>>>> > >>>>>>>>> Qu
> >>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>> > >>>>>>>>>>
> >>>>>>>>>>> > >>>>>>>>>> Thank you!
> >>>>>>>>>>> > >>>>>>>>>>
> >>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>> > >>>>>>
> >>>>>>>>>>> > >>>>
> >>>>>>>>>>> > >>
> >>>>>>>>>>> >
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-08-25 13:30 ` Tyler Richmond
@ 2020-08-25 13:38 ` Qu Wenruo
2020-08-25 13:43 ` Tyler Richmond
0 siblings, 1 reply; 56+ messages in thread
From: Qu Wenruo @ 2020-08-25 13:38 UTC (permalink / raw)
To: Tyler Richmond; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 27388 bytes --]
On 2020/8/25 下午9:30, Tyler Richmond wrote:
> Qu,
>
> The dump of the block is:
>
> https://pastebin.com/ran85JJv
>
> I've also completed the btrfs-image, but it's almost 50gb. What's the
> best way to get it to you? Also, does it work with -ss or are the
> original filenames important?
50G is too big for me to even receive.
But your dump shows the problem!
It's not inode generation, but inode transid, which would affect send.
This is not even checked in btrfs-progs, thus no wonder why it doesn't
detect them.
And copy-pasted kernel message shares the same "generation" word, not
using proper transid to show the problem.
Your dump really saved the day!
The fix for kernel and btrfs-progs would come in next few days.
Thanks,
Qu
>
> Thanks again!
>
>
> On Tue, Aug 25, 2020 at 2:37 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2020/8/25 下午1:25, Tyler Richmond wrote:
>>> Qu,
>>>
>>> Yes, it's btrfs-progs 5.7. Here is the result of the lowmem check:
>>>
>>> https://pastebin.com/8Tzx23EX
>>
>> That doesn't detect any inode generation problem at all, which is not a
>> good sign.
>>
>> Would you also pvode the dump for the offending block?
>>
>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
>> has 18446744073709551492 expect [0, 6875827]
>>
>> For this case, would you please provide the tree dump of "203510940835840" ?
>>
>> # btrfs ins dump-tree -b 203510940835840 <device>
>>
>> And, since btrfs-image can't dump with regular extent tree, the "-w"
>> dump would also help.
>>
>> Thanks,
>> Qu
>>
>>
>>> Thanks!
>>>
>>> On Mon, Aug 24, 2020 at 4:26 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>
>>>>
>>>>
>>>> On 2020/8/24 上午10:47, Tyler Richmond wrote:
>>>>> Qu,
>>>>>
>>>>> Finally finished another repair and captured the output.
>>>>>
>>>>> https://pastebin.com/ffcbwvd8
>>>>>
>>>>> Does that show you what you need? Or should I still do one in lowmem mode?
>>>>
>>>> Lowmem mode (no need for --repair) is recommended since original mode
>>>> doesn't detect the inode generation problem.
>>>>
>>>> And it's already btrfs-progs v5.7 right?
>>>>
>>>> THanks,
>>>> Qu
>>>>>
>>>>> Thanks for your help!
>>>>>
>>>>> On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 2020/8/23 上午10:49, Tyler Richmond wrote:
>>>>>>> Well, I can guarantee that I didn't create this fs before 2015 (just
>>>>>>> checked the order confirmation from when I bought the server), but I
>>>>>>> may have just used whatever was in the Ubuntu package manager at the
>>>>>>> time. So maybe I don't have a v0 ref?
>>>>>>
>>>>>> Then btrfs-image shouldn't report that.
>>>>>>
>>>>>> There is an item smaller than any valid btrfs item, normally it means
>>>>>> it's a v0 ref.
>>>>>> If not, then it could be a bigger problem.
>>>>>>
>>>>>> Could you please provide the full btrfs-check output?
>>>>>> Also, if possible result from "btrfs check --mode=lowmem" would also help.
>>>>>>
>>>>>> Also, if you really go "--repair", then the full output would also be
>>>>>> needed to determine what's going wrong.
>>>>>> There is a report about "btrfs check --repair" didn't repair the inode
>>>>>> generation, if that's the case we must have a bug then.
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>>
>>>>>>> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
>>>>>>>>>> Is my best bet just to downgrade the kernel and then try to delete the
>>>>>>>>>> broken files? Or should I rebuild from scratch? Just don't know
>>>>>>>>>> whether it's worth the time to try and figure this out or if the
>>>>>>>>>> problems stem from the FS being too old and it's beyond trying to
>>>>>>>>>> repair.
>>>>>>>>>
>>>>>>>>> All invalid inode generations, should be able to be repaired by latest
>>>>>>>>> btrfs-check.
>>>>>>>>>
>>>>>>>>> If not, please provide the btrfs-image dump for us to determine what's
>>>>>>>>> going wrong.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Qu
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I didn't check dmesg during the btrfs check, but that was the only
>>>>>>>>>>> output during the rm -f before it was forced readonly. I just checked
>>>>>>>>>>> dmesg for inode generation values, and there are a lot of them.
>>>>>>>>>>>
>>>>>>>>>>> https://pastebin.com/stZdN0ta
>>>>>>>>>>> The dmesg output had 990 lines containing inode generation.
>>>>>>>>>>>
>>>>>>>>>>> However, these were at least later. I tried to do a btrfs balance
>>>>>>>>>>> -mconvert raid1 and it failed with an I/O error. That is probably what
>>>>>>>>>>> generated these specific errors, but maybe they were also happening
>>>>>>>>>>> during the btrfs repair.
>>>>>>>>>>>
>>>>>>>>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway with:
>>>>>>>>>>> ERROR: either extent tree is corrupted or deprecated extent ref format
>>>>>>>>>>> ERROR: create failed: -5
>>>>>>>>
>>>>>>>> Oh, forgot this part.
>>>>>>>>
>>>>>>>> This means you have v0 ref?!
>>>>>>>>
>>>>>>>> Then the fs is too old, no progs/kernel support after all.
>>>>>>>>
>>>>>>>> In that case, please rollback to the last working kernel and copy your data.
>>>>>>>>
>>>>>>>> In fact, that v0 ref should only be in the code base for several weeks
>>>>>>>> before 2010, thus it's really too old.
>>>>>>>>
>>>>>>>> The good news is, with tree-checker, we should never experience such
>>>>>>>> too-old-to-be-usable problem (at least I hope so)
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Qu
>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sorry to resurrect this thread, but I just ran into something that I
>>>>>>>>>>>>> can't really just ignore. I've found a folder that is full of files
>>>>>>>>>>>>> which I guess have been broken somehow. I found a backup and restored
>>>>>>>>>>>>> them, but I want to delete this folder of broken files. But whenever I
>>>>>>>>>>>>> try, the fs is forced into readonly mode again. I just finished another
>>>>>>>>>>>>> btrfs check --repair but it didn't fix the problem.
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://pastebin.com/eTV3s3fr
>>>>>>>>>>>>
>>>>>>>>>>>> Is that the full output?
>>>>>>>>>>>>
>>>>>>>>>>>> No inode generation bugs?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm already on btrfs-progs v5.7. Any new suggestions?
>>>>>>>>>>>>
>>>>>>>>>>>> Strange.
>>>>>>>>>>>>
>>>>>>>>>>>> The detection and repair should have been merged into v5.5.
>>>>>>>>>>>>
>>>>>>>>>>>> If your fs is small enough, would you please provide the "btrfs-image
>>>>>>>>>>>> -c9" dump?
>>>>>>>>>>>>
>>>>>>>>>>>> It would contain the filenames and directories names, but doesn't
>>>>>>>>>>>> contain file contents.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Qu
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com
>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 5.6.1 also failed the same way. Here's the usage output. This is the
>>>>>>>>>>>>> part where you see I've been using RAID5 haha
>>>>>>>>>>>>>
>>>>>>>>>>>>> WARNING: RAID56 detected, not implemented
>>>>>>>>>>>>> Overall:
>>>>>>>>>>>>> Device size: 60.03TiB
>>>>>>>>>>>>> Device allocated: 98.06GiB
>>>>>>>>>>>>> Device unallocated: 59.93TiB
>>>>>>>>>>>>> Device missing: 0.00B
>>>>>>>>>>>>> Used: 92.56GiB
>>>>>>>>>>>>> Free (estimated): 0.00B (min: 8.00EiB)
>>>>>>>>>>>>> Data ratio: 0.00
>>>>>>>>>>>>> Metadata ratio: 2.00
>>>>>>>>>>>>> Global reserve: 512.00MiB (used: 0.00B)
>>>>>>>>>>>>> Multiple profiles: no
>>>>>>>>>>>>>
>>>>>>>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
>>>>>>>>>>>>> /dev/sdh 8.07TiB
>>>>>>>>>>>>> /dev/sdf 8.07TiB
>>>>>>>>>>>>> /dev/sdg 8.07TiB
>>>>>>>>>>>>> /dev/sdd 8.07TiB
>>>>>>>>>>>>> /dev/sdc 8.07TiB
>>>>>>>>>>>>> /dev/sde 8.07TiB
>>>>>>>>>>>>>
>>>>>>>>>>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
>>>>>>>>>>>>> /dev/sdh 34.00GiB
>>>>>>>>>>>>> /dev/sdf 32.00GiB
>>>>>>>>>>>>> /dev/sdg 32.00GiB
>>>>>>>>>>>>>
>>>>>>>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
>>>>>>>>>>>>> /dev/sdf 32.00MiB
>>>>>>>>>>>>> /dev/sdg 32.00MiB
>>>>>>>>>>>>>
>>>>>>>>>>>>> Unallocated:
>>>>>>>>>>>>> /dev/sdh 2.81TiB
>>>>>>>>>>>>> /dev/sdf 2.81TiB
>>>>>>>>>>>>> /dev/sdg 2.81TiB
>>>>>>>>>>>>> /dev/sdd 1.03TiB
>>>>>>>>>>>>> /dev/sdc 1.03TiB
>>>>>>>>>>>>> /dev/sde 1.03TiB
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
>>>>>>>>>>>>> > > If this is saying there's no extra space for metadata, is that why
>>>>>>>>>>>>> > > adding more files often makes the system hang for 30-90s? Is there
>>>>>>>>>>>>> > > anything I should do about that?
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > I'm not sure about the hang though.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > It would be nice to give more info to diagnosis.
>>>>>>>>>>>>> > The output of 'btrfs fi usage' is useful for space usage problem.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
>>>>>>>>>>>>> > space in vanilla df command) space for btrfs.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>> > Qu
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > > Thank you so much for all of your help. I love how flexible BTRFS is
>>>>>>>>>>>>> > > but when things go wrong it's very hard for me to troubleshoot.
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>> > >>
>>>>>>>>>>>>> > >>
>>>>>>>>>>>>> > >>
>>>>>>>>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
>>>>>>>>>>>>> > >>> Something went wrong:
>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>> > >>> Reinitialize checksum tree
>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
>>>>>>>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
>>>>>>>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
>>>>>>>>>>>>> > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
>>>>>>>>>>>>> > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
>>>>>>>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
>>>>>>>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
>>>>>>>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
>>>>>>>>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
>>>>>>>>>>>>> > >>> Aborted
>>>>>>>>>>>>> > >>
>>>>>>>>>>>>> > >> This means no space for extra metadata...
>>>>>>>>>>>>> > >>
>>>>>>>>>>>>> > >> Anyway the csum tree problem shouldn't be a big thing, you
>>>>>>>>>>>>> could leave
>>>>>>>>>>>>> > >> it and call it a day.
>>>>>>>>>>>>> > >>
>>>>>>>>>>>>> > >> BTW, as long as btrfs check reports no extra problem for the inode
>>>>>>>>>>>>> > >> generation, it should be pretty safe to use the fs.
>>>>>>>>>>>>> > >>
>>>>>>>>>>>>> > >> Thanks,
>>>>>>>>>>>>> > >> Qu
>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>> > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
>>>>>>>>>>>>> > >>> available. I'll let that try overnight?
>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
>>>>>>>>>>>>> > >>>>> Thank you for helping. The end result of the scan was:
>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>> > >>>>> [1/7] checking root items
>>>>>>>>>>>>> > >>>>> [2/7] checking extents
>>>>>>>>>>>>> > >>>>> [3/7] checking free space cache
>>>>>>>>>>>>> > >>>>> [4/7] checking fs roots
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>>> Good news is, your fs is still mostly fine.
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>>>> [5/7] checking only csums items (without verifying data)
>>>>>>>>>>>>> > >>>>> there are no extents for csum range 0-69632
>>>>>>>>>>>>> > >>>>> csum exists for 0-69632 but there is no extent record
>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>> > >>>>> there are no extents for csum range 946692096-946827264
>>>>>>>>>>>>> > >>>>> csum exists for 946692096-946827264 but there is no extent
>>>>>>>>>>>>> record
>>>>>>>>>>>>> > >>>>> there are no extents for csum range 946831360-947912704
>>>>>>>>>>>>> > >>>>> csum exists for 946831360-947912704 but there is no extent
>>>>>>>>>>>>> record
>>>>>>>>>>>>> > >>>>> ERROR: errors found in csum tree
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>>> Only extent tree is corrupted.
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>>> Normally btrfs check --init-csum-tree should be able to
>>>>>>>>>>>>> handle it.
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>>> But still, please be sure you're using the latest btrfs-progs
>>>>>>>>>>>>> to fix it.
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>>> Thanks,
>>>>>>>>>>>>> > >>>> Qu
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>>>> [6/7] checking root refs
>>>>>>>>>>>>> > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>>>>>>>>>>>> > >>>>> found 44157956026368 bytes used, error(s) found
>>>>>>>>>>>>> > >>>>> total csum bytes: 42038602716
>>>>>>>>>>>>> > >>>>> total tree bytes: 49688616960
>>>>>>>>>>>>> > >>>>> total fs tree bytes: 1256427520
>>>>>>>>>>>>> > >>>>> total extent tree bytes: 1709105152
>>>>>>>>>>>>> > >>>>> btree space waste bytes: 3172727316
>>>>>>>>>>>>> > >>>>> file data blocks allocated: 261625653436416
>>>>>>>>>>>>> > >>>>> referenced 47477768499200
>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>> > >>>>> What do I need to do to fix all of this?
>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
>>>>>>>>>>>>> > >>>>>>> Well, the repair doesn't look terribly successful.
>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>> item=84
>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>> This means there are more problems, not only the hash name
>>>>>>>>>>>>> mismatch.
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>> This means the fs is already corrupted, the name hash is
>>>>>>>>>>>>> just one
>>>>>>>>>>>>> > >>>>>> unrelated symptom.
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>> The only good news is, btrfs-progs abort the transaction,
>>>>>>>>>>>>> thus no
>>>>>>>>>>>>> > >>>>>> further damage to the fs.
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>> Please run a plain btrfs-check to show what's the problem
>>>>>>>>>>>>> first.
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>> > >>>>>> Qu
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>> item=84
>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>> item=84
>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>> item=84
>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>> item=84
>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>> item=84
>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>> item=84
>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>> item=84
>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>> item=84
>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>> item=84
>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>> item=84
>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>> > >>>>>>> ERROR: failed to zero log tree: -17
>>>>>>>>>>>>> > >>>>>>> ERROR: attempt to start transaction over already running one
>>>>>>>>>>>>> > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>> > >>>>>>> What is going on?
>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond
>>>>>>>>>>>>> <t.d.richmond@gmail.com <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>> > >>>>>>>> Chris, I had used the correct mountpoint in the command.
>>>>>>>>>>>>> I just edited
>>>>>>>>>>>>> > >>>>>>>> it in the email to be /mountpoint for consistency.
>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
>>>>>>>>>>>>> > >>>>>>>>>> Hello,
>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>> > >>>>>>>>>> I looked up this error and it basically says ask a
>>>>>>>>>>>>> developer to
>>>>>>>>>>>>> > >>>>>>>>>> determine if it's a false error or not. I just started
>>>>>>>>>>>>> getting some
>>>>>>>>>>>>> > >>>>>>>>>> slow response times, and looked at the dmesg log to
>>>>>>>>>>>>> find a ton of
>>>>>>>>>>>>> > >>>>>>>>>> these errors.
>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt
>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>>>>> generation:
>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error (device sdh):
>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt
>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>>>>> generation:
>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error (device sdh):
>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt
>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>>>>> generation:
>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error (device sdh):
>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>> > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>> > >>>>>>>>>> Is there anything I should do about this, or should I
>>>>>>>>>>>>> just continue
>>>>>>>>>>>>> > >>>>>>>>>> using my array as normal?
>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>> > >>>>>>>>> This is caused by older kernel underflow inode generation.
>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>> > >>>>>>>>> Or you can go safer, by manually locating the inode
>>>>>>>>>>>>> using its inode
>>>>>>>>>>>>> > >>>>>>>>> number (1311670), and copy it to some new location using
>>>>>>>>>>>>> previous
>>>>>>>>>>>>> > >>>>>>>>> working kernel, then delete the old file, copy the new
>>>>>>>>>>>>> one back to fix it.
>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>>>>>>> > >>>>>>>>> Qu
>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>> > >>>>>>>>>> Thank you!
>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>
>>>>>>>>>>>>> >
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-08-25 13:38 ` Qu Wenruo
@ 2020-08-25 13:43 ` Tyler Richmond
2020-11-05 7:01 ` Tyler Richmond
0 siblings, 1 reply; 56+ messages in thread
From: Tyler Richmond @ 2020-08-25 13:43 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Btrfs BTRFS
Great, glad we got somewhere! I'll look forward to the fix!
On Tue, Aug 25, 2020 at 9:38 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2020/8/25 下午9:30, Tyler Richmond wrote:
> > Qu,
> >
> > The dump of the block is:
> >
> > https://pastebin.com/ran85JJv
> >
> > I've also completed the btrfs-image, but it's almost 50gb. What's the
> > best way to get it to you? Also, does it work with -ss or are the
> > original filenames important?
>
> 50G is too big for me to even receive.
>
> But your dump shows the problem!
>
> It's not inode generation, but inode transid, which would affect send.
>
> This is not even checked in btrfs-progs, thus no wonder why it doesn't
> detect them.
>
> And copy-pasted kernel message shares the same "generation" word, not
> using proper transid to show the problem.
>
> Your dump really saved the day!
>
> The fix for kernel and btrfs-progs would come in next few days.
>
> Thanks,
> Qu
> >
> > Thanks again!
> >
> >
> > On Tue, Aug 25, 2020 at 2:37 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>
> >>
> >>
> >> On 2020/8/25 下午1:25, Tyler Richmond wrote:
> >>> Qu,
> >>>
> >>> Yes, it's btrfs-progs 5.7. Here is the result of the lowmem check:
> >>>
> >>> https://pastebin.com/8Tzx23EX
> >>
> >> That doesn't detect any inode generation problem at all, which is not a
> >> good sign.
> >>
> >> Would you also pvode the dump for the offending block?
> >>
> >>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> >> has 18446744073709551492 expect [0, 6875827]
> >>
> >> For this case, would you please provide the tree dump of "203510940835840" ?
> >>
> >> # btrfs ins dump-tree -b 203510940835840 <device>
> >>
> >> And, since btrfs-image can't dump with regular extent tree, the "-w"
> >> dump would also help.
> >>
> >> Thanks,
> >> Qu
> >>
> >>
> >>> Thanks!
> >>>
> >>> On Mon, Aug 24, 2020 at 4:26 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 2020/8/24 上午10:47, Tyler Richmond wrote:
> >>>>> Qu,
> >>>>>
> >>>>> Finally finished another repair and captured the output.
> >>>>>
> >>>>> https://pastebin.com/ffcbwvd8
> >>>>>
> >>>>> Does that show you what you need? Or should I still do one in lowmem mode?
> >>>>
> >>>> Lowmem mode (no need for --repair) is recommended since original mode
> >>>> doesn't detect the inode generation problem.
> >>>>
> >>>> And it's already btrfs-progs v5.7 right?
> >>>>
> >>>> THanks,
> >>>> Qu
> >>>>>
> >>>>> Thanks for your help!
> >>>>>
> >>>>> On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 2020/8/23 上午10:49, Tyler Richmond wrote:
> >>>>>>> Well, I can guarantee that I didn't create this fs before 2015 (just
> >>>>>>> checked the order confirmation from when I bought the server), but I
> >>>>>>> may have just used whatever was in the Ubuntu package manager at the
> >>>>>>> time. So maybe I don't have a v0 ref?
> >>>>>>
> >>>>>> Then btrfs-image shouldn't report that.
> >>>>>>
> >>>>>> There is an item smaller than any valid btrfs item, normally it means
> >>>>>> it's a v0 ref.
> >>>>>> If not, then it could be a bigger problem.
> >>>>>>
> >>>>>> Could you please provide the full btrfs-check output?
> >>>>>> Also, if possible result from "btrfs check --mode=lowmem" would also help.
> >>>>>>
> >>>>>> Also, if you really go "--repair", then the full output would also be
> >>>>>> needed to determine what's going wrong.
> >>>>>> There is a report about "btrfs check --repair" didn't repair the inode
> >>>>>> generation, if that's the case we must have a bug then.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Qu
> >>>>>>>
> >>>>>>> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
> >>>>>>>>>> Is my best bet just to downgrade the kernel and then try to delete the
> >>>>>>>>>> broken files? Or should I rebuild from scratch? Just don't know
> >>>>>>>>>> whether it's worth the time to try and figure this out or if the
> >>>>>>>>>> problems stem from the FS being too old and it's beyond trying to
> >>>>>>>>>> repair.
> >>>>>>>>>
> >>>>>>>>> All invalid inode generations, should be able to be repaired by latest
> >>>>>>>>> btrfs-check.
> >>>>>>>>>
> >>>>>>>>> If not, please provide the btrfs-image dump for us to determine what's
> >>>>>>>>> going wrong.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Qu
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> I didn't check dmesg during the btrfs check, but that was the only
> >>>>>>>>>>> output during the rm -f before it was forced readonly. I just checked
> >>>>>>>>>>> dmesg for inode generation values, and there are a lot of them.
> >>>>>>>>>>>
> >>>>>>>>>>> https://pastebin.com/stZdN0ta
> >>>>>>>>>>> The dmesg output had 990 lines containing inode generation.
> >>>>>>>>>>>
> >>>>>>>>>>> However, these were at least later. I tried to do a btrfs balance
> >>>>>>>>>>> -mconvert raid1 and it failed with an I/O error. That is probably what
> >>>>>>>>>>> generated these specific errors, but maybe they were also happening
> >>>>>>>>>>> during the btrfs repair.
> >>>>>>>>>>>
> >>>>>>>>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway with:
> >>>>>>>>>>> ERROR: either extent tree is corrupted or deprecated extent ref format
> >>>>>>>>>>> ERROR: create failed: -5
> >>>>>>>>
> >>>>>>>> Oh, forgot this part.
> >>>>>>>>
> >>>>>>>> This means you have v0 ref?!
> >>>>>>>>
> >>>>>>>> Then the fs is too old, no progs/kernel support after all.
> >>>>>>>>
> >>>>>>>> In that case, please rollback to the last working kernel and copy your data.
> >>>>>>>>
> >>>>>>>> In fact, that v0 ref should only be in the code base for several weeks
> >>>>>>>> before 2010, thus it's really too old.
> >>>>>>>>
> >>>>>>>> The good news is, with tree-checker, we should never experience such
> >>>>>>>> too-old-to-be-usable problem (at least I hope so)
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Qu
> >>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
> >>>>>>>>>>>>> Qu,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Sorry to resurrect this thread, but I just ran into something that I
> >>>>>>>>>>>>> can't really just ignore. I've found a folder that is full of files
> >>>>>>>>>>>>> which I guess have been broken somehow. I found a backup and restored
> >>>>>>>>>>>>> them, but I want to delete this folder of broken files. But whenever I
> >>>>>>>>>>>>> try, the fs is forced into readonly mode again. I just finished another
> >>>>>>>>>>>>> btrfs check --repair but it didn't fix the problem.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> https://pastebin.com/eTV3s3fr
> >>>>>>>>>>>>
> >>>>>>>>>>>> Is that the full output?
> >>>>>>>>>>>>
> >>>>>>>>>>>> No inode generation bugs?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I'm already on btrfs-progs v5.7. Any new suggestions?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Strange.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The detection and repair should have been merged into v5.5.
> >>>>>>>>>>>>
> >>>>>>>>>>>> If your fs is small enough, would you please provide the "btrfs-image
> >>>>>>>>>>>> -c9" dump?
> >>>>>>>>>>>>
> >>>>>>>>>>>> It would contain the filenames and directories names, but doesn't
> >>>>>>>>>>>> contain file contents.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Qu
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com
> >>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 5.6.1 also failed the same way. Here's the usage output. This is the
> >>>>>>>>>>>>> part where you see I've been using RAID5 haha
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> WARNING: RAID56 detected, not implemented
> >>>>>>>>>>>>> Overall:
> >>>>>>>>>>>>> Device size: 60.03TiB
> >>>>>>>>>>>>> Device allocated: 98.06GiB
> >>>>>>>>>>>>> Device unallocated: 59.93TiB
> >>>>>>>>>>>>> Device missing: 0.00B
> >>>>>>>>>>>>> Used: 92.56GiB
> >>>>>>>>>>>>> Free (estimated): 0.00B (min: 8.00EiB)
> >>>>>>>>>>>>> Data ratio: 0.00
> >>>>>>>>>>>>> Metadata ratio: 2.00
> >>>>>>>>>>>>> Global reserve: 512.00MiB (used: 0.00B)
> >>>>>>>>>>>>> Multiple profiles: no
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
> >>>>>>>>>>>>> /dev/sdh 8.07TiB
> >>>>>>>>>>>>> /dev/sdf 8.07TiB
> >>>>>>>>>>>>> /dev/sdg 8.07TiB
> >>>>>>>>>>>>> /dev/sdd 8.07TiB
> >>>>>>>>>>>>> /dev/sdc 8.07TiB
> >>>>>>>>>>>>> /dev/sde 8.07TiB
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
> >>>>>>>>>>>>> /dev/sdh 34.00GiB
> >>>>>>>>>>>>> /dev/sdf 32.00GiB
> >>>>>>>>>>>>> /dev/sdg 32.00GiB
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
> >>>>>>>>>>>>> /dev/sdf 32.00MiB
> >>>>>>>>>>>>> /dev/sdg 32.00MiB
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Unallocated:
> >>>>>>>>>>>>> /dev/sdh 2.81TiB
> >>>>>>>>>>>>> /dev/sdf 2.81TiB
> >>>>>>>>>>>>> /dev/sdg 2.81TiB
> >>>>>>>>>>>>> /dev/sdd 1.03TiB
> >>>>>>>>>>>>> /dev/sdc 1.03TiB
> >>>>>>>>>>>>> /dev/sde 1.03TiB
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> >>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>>>>>>>> >
> >>>>>>>>>>>>> >
> >>>>>>>>>>>>> >
> >>>>>>>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
> >>>>>>>>>>>>> > > If this is saying there's no extra space for metadata, is that why
> >>>>>>>>>>>>> > > adding more files often makes the system hang for 30-90s? Is there
> >>>>>>>>>>>>> > > anything I should do about that?
> >>>>>>>>>>>>> >
> >>>>>>>>>>>>> > I'm not sure about the hang though.
> >>>>>>>>>>>>> >
> >>>>>>>>>>>>> > It would be nice to give more info to diagnosis.
> >>>>>>>>>>>>> > The output of 'btrfs fi usage' is useful for space usage problem.
> >>>>>>>>>>>>> >
> >>>>>>>>>>>>> > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
> >>>>>>>>>>>>> > space in vanilla df command) space for btrfs.
> >>>>>>>>>>>>> >
> >>>>>>>>>>>>> > Thanks,
> >>>>>>>>>>>>> > Qu
> >>>>>>>>>>>>> >
> >>>>>>>>>>>>> > >
> >>>>>>>>>>>>> > > Thank you so much for all of your help. I love how flexible BTRFS is
> >>>>>>>>>>>>> > > but when things go wrong it's very hard for me to troubleshoot.
> >>>>>>>>>>>>> > >
> >>>>>>>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> >>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>>>>>>>> > >>
> >>>>>>>>>>>>> > >>
> >>>>>>>>>>>>> > >>
> >>>>>>>>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
> >>>>>>>>>>>>> > >>> Something went wrong:
> >>>>>>>>>>>>> > >>>
> >>>>>>>>>>>>> > >>> Reinitialize checksum tree
> >>>>>>>>>>>>> > >>> Unable to find block group for 0
> >>>>>>>>>>>>> > >>> Unable to find block group for 0
> >>>>>>>>>>>>> > >>> Unable to find block group for 0
> >>>>>>>>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
> >>>>>>>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
> >>>>>>>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
> >>>>>>>>>>>>> > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
> >>>>>>>>>>>>> > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
> >>>>>>>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
> >>>>>>>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
> >>>>>>>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
> >>>>>>>>>>>>> > >>>
> >>>>>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
> >>>>>>>>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
> >>>>>>>>>>>>> > >>> Aborted
> >>>>>>>>>>>>> > >>
> >>>>>>>>>>>>> > >> This means no space for extra metadata...
> >>>>>>>>>>>>> > >>
> >>>>>>>>>>>>> > >> Anyway the csum tree problem shouldn't be a big thing, you
> >>>>>>>>>>>>> could leave
> >>>>>>>>>>>>> > >> it and call it a day.
> >>>>>>>>>>>>> > >>
> >>>>>>>>>>>>> > >> BTW, as long as btrfs check reports no extra problem for the inode
> >>>>>>>>>>>>> > >> generation, it should be pretty safe to use the fs.
> >>>>>>>>>>>>> > >>
> >>>>>>>>>>>>> > >> Thanks,
> >>>>>>>>>>>>> > >> Qu
> >>>>>>>>>>>>> > >>>
> >>>>>>>>>>>>> > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
> >>>>>>>>>>>>> > >>> available. I'll let that try overnight?
> >>>>>>>>>>>>> > >>>
> >>>>>>>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
> >>>>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
> >>>>>>>>>>>>> > >>>>> Thank you for helping. The end result of the scan was:
> >>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>> > >>>>> [1/7] checking root items
> >>>>>>>>>>>>> > >>>>> [2/7] checking extents
> >>>>>>>>>>>>> > >>>>> [3/7] checking free space cache
> >>>>>>>>>>>>> > >>>>> [4/7] checking fs roots
> >>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>> > >>>> Good news is, your fs is still mostly fine.
> >>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>> > >>>>> [5/7] checking only csums items (without verifying data)
> >>>>>>>>>>>>> > >>>>> there are no extents for csum range 0-69632
> >>>>>>>>>>>>> > >>>>> csum exists for 0-69632 but there is no extent record
> >>>>>>>>>>>>> > >>>>> ...
> >>>>>>>>>>>>> > >>>>> ...
> >>>>>>>>>>>>> > >>>>> there are no extents for csum range 946692096-946827264
> >>>>>>>>>>>>> > >>>>> csum exists for 946692096-946827264 but there is no extent
> >>>>>>>>>>>>> record
> >>>>>>>>>>>>> > >>>>> there are no extents for csum range 946831360-947912704
> >>>>>>>>>>>>> > >>>>> csum exists for 946831360-947912704 but there is no extent
> >>>>>>>>>>>>> record
> >>>>>>>>>>>>> > >>>>> ERROR: errors found in csum tree
> >>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>> > >>>> Only extent tree is corrupted.
> >>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>> > >>>> Normally btrfs check --init-csum-tree should be able to
> >>>>>>>>>>>>> handle it.
> >>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>> > >>>> But still, please be sure you're using the latest btrfs-progs
> >>>>>>>>>>>>> to fix it.
> >>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>> > >>>> Thanks,
> >>>>>>>>>>>>> > >>>> Qu
> >>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>> > >>>>> [6/7] checking root refs
> >>>>>>>>>>>>> > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
> >>>>>>>>>>>>> > >>>>> found 44157956026368 bytes used, error(s) found
> >>>>>>>>>>>>> > >>>>> total csum bytes: 42038602716
> >>>>>>>>>>>>> > >>>>> total tree bytes: 49688616960
> >>>>>>>>>>>>> > >>>>> total fs tree bytes: 1256427520
> >>>>>>>>>>>>> > >>>>> total extent tree bytes: 1709105152
> >>>>>>>>>>>>> > >>>>> btree space waste bytes: 3172727316
> >>>>>>>>>>>>> > >>>>> file data blocks allocated: 261625653436416
> >>>>>>>>>>>>> > >>>>> referenced 47477768499200
> >>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>> > >>>>> What do I need to do to fix all of this?
> >>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
> >>>>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
> >>>>>>>>>>>>> > >>>>>>> Well, the repair doesn't look terribly successful.
> >>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>>>> item=84
> >>>>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>> > >>>>>> This means there are more problems, not only the hash name
> >>>>>>>>>>>>> mismatch.
> >>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>> > >>>>>> This means the fs is already corrupted, the name hash is
> >>>>>>>>>>>>> just one
> >>>>>>>>>>>>> > >>>>>> unrelated symptom.
> >>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>> > >>>>>> The only good news is, btrfs-progs abort the transaction,
> >>>>>>>>>>>>> thus no
> >>>>>>>>>>>>> > >>>>>> further damage to the fs.
> >>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>> > >>>>>> Please run a plain btrfs-check to show what's the problem
> >>>>>>>>>>>>> first.
> >>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>> > >>>>>> Thanks,
> >>>>>>>>>>>>> > >>>>>> Qu
> >>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>>>> item=84
> >>>>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>>>> item=84
> >>>>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>>>> item=84
> >>>>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>>>> item=84
> >>>>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>>>> item=84
> >>>>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>>>> item=84
> >>>>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>>>> item=84
> >>>>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>>>> item=84
> >>>>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>>>> item=84
> >>>>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> >>>>>>>>>>>>> 6875841 found 6876224
> >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> >>>>>>>>>>>>> item=84
> >>>>>>>>>>>>> > >>>>>>> parent level=1
> >>>>>>>>>>>>> > >>>>>>> child level=4
> >>>>>>>>>>>>> > >>>>>>> ERROR: failed to zero log tree: -17
> >>>>>>>>>>>>> > >>>>>>> ERROR: attempt to start transaction over already running one
> >>>>>>>>>>>>> > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
> >>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> >>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> >>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>>>>>>>>>> 225049066086400 len 4096
> >>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> >>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> >>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>>>>>>>>>> 225049066094592 len 4096
> >>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> >>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> >>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>>>>>>>>>> 225049066102784 len 4096
> >>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> >>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> >>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> >>>>>>>>>>>>> 225049066131456 len 4096
> >>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>> > >>>>>>> What is going on?
> >>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond
> >>>>>>>>>>>>> <t.d.richmond@gmail.com <mailto:t.d.richmond@gmail.com>> wrote:
> >>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>> > >>>>>>>> Chris, I had used the correct mountpoint in the command.
> >>>>>>>>>>>>> I just edited
> >>>>>>>>>>>>> > >>>>>>>> it in the email to be /mountpoint for consistency.
> >>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
> >>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
> >>>>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
> >>>>>>>>>>>>> > >>>>>>>>>> Hello,
> >>>>>>>>>>>>> > >>>>>>>>>>
> >>>>>>>>>>>>> > >>>>>>>>>> I looked up this error and it basically says ask a
> >>>>>>>>>>>>> developer to
> >>>>>>>>>>>>> > >>>>>>>>>> determine if it's a false error or not. I just started
> >>>>>>>>>>>>> getting some
> >>>>>>>>>>>>> > >>>>>>>>>> slow response times, and looked at the dmesg log to
> >>>>>>>>>>>>> find a ton of
> >>>>>>>>>>>>> > >>>>>>>>>> these errors.
> >>>>>>>>>>>>> > >>>>>>>>>>
> >>>>>>>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt
> >>>>>>>>>>>>> leaf: root=5
> >>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> >>>>>>>>>>>>> generation:
> >>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>>>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error (device sdh):
> >>>>>>>>>>>>> block=203510940835840 read
> >>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
> >>>>>>>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt
> >>>>>>>>>>>>> leaf: root=5
> >>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> >>>>>>>>>>>>> generation:
> >>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>>>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error (device sdh):
> >>>>>>>>>>>>> block=203510940835840 read
> >>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
> >>>>>>>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt
> >>>>>>>>>>>>> leaf: root=5
> >>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> >>>>>>>>>>>>> generation:
> >>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> >>>>>>>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error (device sdh):
> >>>>>>>>>>>>> block=203510940835840 read
> >>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
> >>>>>>>>>>>>> > >>>>>>>>>>
> >>>>>>>>>>>>> > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
> >>>>>>>>>>>>> > >>>>>>>>>>
> >>>>>>>>>>>>> > >>>>>>>>>> Is there anything I should do about this, or should I
> >>>>>>>>>>>>> just continue
> >>>>>>>>>>>>> > >>>>>>>>>> using my array as normal?
> >>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>> > >>>>>>>>> This is caused by older kernel underflow inode generation.
> >>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
> >>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>> > >>>>>>>>> Or you can go safer, by manually locating the inode
> >>>>>>>>>>>>> using its inode
> >>>>>>>>>>>>> > >>>>>>>>> number (1311670), and copy it to some new location using
> >>>>>>>>>>>>> previous
> >>>>>>>>>>>>> > >>>>>>>>> working kernel, then delete the old file, copy the new
> >>>>>>>>>>>>> one back to fix it.
> >>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>> > >>>>>>>>> Thanks,
> >>>>>>>>>>>>> > >>>>>>>>> Qu
> >>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>> > >>>>>>>>>>
> >>>>>>>>>>>>> > >>>>>>>>>> Thank you!
> >>>>>>>>>>>>> > >>>>>>>>>>
> >>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>> > >>
> >>>>>>>>>>>>> >
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-08-25 13:43 ` Tyler Richmond
@ 2020-11-05 7:01 ` Tyler Richmond
2020-11-05 7:19 ` Qu Wenruo
0 siblings, 1 reply; 56+ messages in thread
From: Tyler Richmond @ 2020-11-05 7:01 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Btrfs BTRFS
Qu,
I'm wondering, was a fix for this ever implemented? I recently added a
new drive to expand the array, and during the rebalance it dropped
itself back to a read only filesystem. I suspect it's related to the
issues discussed earlier in this thread. Is there anything I can do to
complete the balance? The error that caused it to drop to read only is
here: https://pastebin.com/GGYVMaiG
Thanks!
On Tue, Aug 25, 2020 at 9:43 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>
> Great, glad we got somewhere! I'll look forward to the fix!
>
> On Tue, Aug 25, 2020 at 9:38 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >
> >
> >
> > On 2020/8/25 下午9:30, Tyler Richmond wrote:
> > > Qu,
> > >
> > > The dump of the block is:
> > >
> > > https://pastebin.com/ran85JJv
> > >
> > > I've also completed the btrfs-image, but it's almost 50gb. What's the
> > > best way to get it to you? Also, does it work with -ss or are the
> > > original filenames important?
> >
> > 50G is too big for me to even receive.
> >
> > But your dump shows the problem!
> >
> > It's not inode generation, but inode transid, which would affect send.
> >
> > This is not even checked in btrfs-progs, thus no wonder why it doesn't
> > detect them.
> >
> > And copy-pasted kernel message shares the same "generation" word, not
> > using proper transid to show the problem.
> >
> > Your dump really saved the day!
> >
> > The fix for kernel and btrfs-progs would come in next few days.
> >
> > Thanks,
> > Qu
> > >
> > > Thanks again!
> > >
> > >
> > > On Tue, Aug 25, 2020 at 2:37 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > >>
> > >>
> > >>
> > >> On 2020/8/25 下午1:25, Tyler Richmond wrote:
> > >>> Qu,
> > >>>
> > >>> Yes, it's btrfs-progs 5.7. Here is the result of the lowmem check:
> > >>>
> > >>> https://pastebin.com/8Tzx23EX
> > >>
> > >> That doesn't detect any inode generation problem at all, which is not a
> > >> good sign.
> > >>
> > >> Would you also pvode the dump for the offending block?
> > >>
> > >>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> > >> has 18446744073709551492 expect [0, 6875827]
> > >>
> > >> For this case, would you please provide the tree dump of "203510940835840" ?
> > >>
> > >> # btrfs ins dump-tree -b 203510940835840 <device>
> > >>
> > >> And, since btrfs-image can't dump with regular extent tree, the "-w"
> > >> dump would also help.
> > >>
> > >> Thanks,
> > >> Qu
> > >>
> > >>
> > >>> Thanks!
> > >>>
> > >>> On Mon, Aug 24, 2020 at 4:26 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > >>>>
> > >>>>
> > >>>>
> > >>>> On 2020/8/24 上午10:47, Tyler Richmond wrote:
> > >>>>> Qu,
> > >>>>>
> > >>>>> Finally finished another repair and captured the output.
> > >>>>>
> > >>>>> https://pastebin.com/ffcbwvd8
> > >>>>>
> > >>>>> Does that show you what you need? Or should I still do one in lowmem mode?
> > >>>>
> > >>>> Lowmem mode (no need for --repair) is recommended since original mode
> > >>>> doesn't detect the inode generation problem.
> > >>>>
> > >>>> And it's already btrfs-progs v5.7 right?
> > >>>>
> > >>>> THanks,
> > >>>> Qu
> > >>>>>
> > >>>>> Thanks for your help!
> > >>>>>
> > >>>>> On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On 2020/8/23 上午10:49, Tyler Richmond wrote:
> > >>>>>>> Well, I can guarantee that I didn't create this fs before 2015 (just
> > >>>>>>> checked the order confirmation from when I bought the server), but I
> > >>>>>>> may have just used whatever was in the Ubuntu package manager at the
> > >>>>>>> time. So maybe I don't have a v0 ref?
> > >>>>>>
> > >>>>>> Then btrfs-image shouldn't report that.
> > >>>>>>
> > >>>>>> There is an item smaller than any valid btrfs item, normally it means
> > >>>>>> it's a v0 ref.
> > >>>>>> If not, then it could be a bigger problem.
> > >>>>>>
> > >>>>>> Could you please provide the full btrfs-check output?
> > >>>>>> Also, if possible result from "btrfs check --mode=lowmem" would also help.
> > >>>>>>
> > >>>>>> Also, if you really go "--repair", then the full output would also be
> > >>>>>> needed to determine what's going wrong.
> > >>>>>> There is a report about "btrfs check --repair" didn't repair the inode
> > >>>>>> generation, if that's the case we must have a bug then.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Qu
> > >>>>>>>
> > >>>>>>> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
> > >>>>>>>>>> Is my best bet just to downgrade the kernel and then try to delete the
> > >>>>>>>>>> broken files? Or should I rebuild from scratch? Just don't know
> > >>>>>>>>>> whether it's worth the time to try and figure this out or if the
> > >>>>>>>>>> problems stem from the FS being too old and it's beyond trying to
> > >>>>>>>>>> repair.
> > >>>>>>>>>
> > >>>>>>>>> All invalid inode generations, should be able to be repaired by latest
> > >>>>>>>>> btrfs-check.
> > >>>>>>>>>
> > >>>>>>>>> If not, please provide the btrfs-image dump for us to determine what's
> > >>>>>>>>> going wrong.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> Qu
> > >>>>>>>>>>
> > >>>>>>>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> I didn't check dmesg during the btrfs check, but that was the only
> > >>>>>>>>>>> output during the rm -f before it was forced readonly. I just checked
> > >>>>>>>>>>> dmesg for inode generation values, and there are a lot of them.
> > >>>>>>>>>>>
> > >>>>>>>>>>> https://pastebin.com/stZdN0ta
> > >>>>>>>>>>> The dmesg output had 990 lines containing inode generation.
> > >>>>>>>>>>>
> > >>>>>>>>>>> However, these were at least later. I tried to do a btrfs balance
> > >>>>>>>>>>> -mconvert raid1 and it failed with an I/O error. That is probably what
> > >>>>>>>>>>> generated these specific errors, but maybe they were also happening
> > >>>>>>>>>>> during the btrfs repair.
> > >>>>>>>>>>>
> > >>>>>>>>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway with:
> > >>>>>>>>>>> ERROR: either extent tree is corrupted or deprecated extent ref format
> > >>>>>>>>>>> ERROR: create failed: -5
> > >>>>>>>>
> > >>>>>>>> Oh, forgot this part.
> > >>>>>>>>
> > >>>>>>>> This means you have v0 ref?!
> > >>>>>>>>
> > >>>>>>>> Then the fs is too old, no progs/kernel support after all.
> > >>>>>>>>
> > >>>>>>>> In that case, please rollback to the last working kernel and copy your data.
> > >>>>>>>>
> > >>>>>>>> In fact, that v0 ref should only be in the code base for several weeks
> > >>>>>>>> before 2010, thus it's really too old.
> > >>>>>>>>
> > >>>>>>>> The good news is, with tree-checker, we should never experience such
> > >>>>>>>> too-old-to-be-usable problem (at least I hope so)
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> Qu
> > >>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
> > >>>>>>>>>>>>> Qu,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Sorry to resurrect this thread, but I just ran into something that I
> > >>>>>>>>>>>>> can't really just ignore. I've found a folder that is full of files
> > >>>>>>>>>>>>> which I guess have been broken somehow. I found a backup and restored
> > >>>>>>>>>>>>> them, but I want to delete this folder of broken files. But whenever I
> > >>>>>>>>>>>>> try, the fs is forced into readonly mode again. I just finished another
> > >>>>>>>>>>>>> btrfs check --repair but it didn't fix the problem.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> https://pastebin.com/eTV3s3fr
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Is that the full output?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> No inode generation bugs?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I'm already on btrfs-progs v5.7. Any new suggestions?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Strange.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> The detection and repair should have been merged into v5.5.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> If your fs is small enough, would you please provide the "btrfs-image
> > >>>>>>>>>>>> -c9" dump?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> It would contain the filenames and directories names, but doesn't
> > >>>>>>>>>>>> contain file contents.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>> Qu
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com
> > >>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> 5.6.1 also failed the same way. Here's the usage output. This is the
> > >>>>>>>>>>>>> part where you see I've been using RAID5 haha
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> WARNING: RAID56 detected, not implemented
> > >>>>>>>>>>>>> Overall:
> > >>>>>>>>>>>>> Device size: 60.03TiB
> > >>>>>>>>>>>>> Device allocated: 98.06GiB
> > >>>>>>>>>>>>> Device unallocated: 59.93TiB
> > >>>>>>>>>>>>> Device missing: 0.00B
> > >>>>>>>>>>>>> Used: 92.56GiB
> > >>>>>>>>>>>>> Free (estimated): 0.00B (min: 8.00EiB)
> > >>>>>>>>>>>>> Data ratio: 0.00
> > >>>>>>>>>>>>> Metadata ratio: 2.00
> > >>>>>>>>>>>>> Global reserve: 512.00MiB (used: 0.00B)
> > >>>>>>>>>>>>> Multiple profiles: no
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
> > >>>>>>>>>>>>> /dev/sdh 8.07TiB
> > >>>>>>>>>>>>> /dev/sdf 8.07TiB
> > >>>>>>>>>>>>> /dev/sdg 8.07TiB
> > >>>>>>>>>>>>> /dev/sdd 8.07TiB
> > >>>>>>>>>>>>> /dev/sdc 8.07TiB
> > >>>>>>>>>>>>> /dev/sde 8.07TiB
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
> > >>>>>>>>>>>>> /dev/sdh 34.00GiB
> > >>>>>>>>>>>>> /dev/sdf 32.00GiB
> > >>>>>>>>>>>>> /dev/sdg 32.00GiB
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
> > >>>>>>>>>>>>> /dev/sdf 32.00MiB
> > >>>>>>>>>>>>> /dev/sdg 32.00MiB
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Unallocated:
> > >>>>>>>>>>>>> /dev/sdh 2.81TiB
> > >>>>>>>>>>>>> /dev/sdf 2.81TiB
> > >>>>>>>>>>>>> /dev/sdg 2.81TiB
> > >>>>>>>>>>>>> /dev/sdd 1.03TiB
> > >>>>>>>>>>>>> /dev/sdc 1.03TiB
> > >>>>>>>>>>>>> /dev/sde 1.03TiB
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> > >>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > >>>>>>>>>>>>> >
> > >>>>>>>>>>>>> >
> > >>>>>>>>>>>>> >
> > >>>>>>>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
> > >>>>>>>>>>>>> > > If this is saying there's no extra space for metadata, is that why
> > >>>>>>>>>>>>> > > adding more files often makes the system hang for 30-90s? Is there
> > >>>>>>>>>>>>> > > anything I should do about that?
> > >>>>>>>>>>>>> >
> > >>>>>>>>>>>>> > I'm not sure about the hang though.
> > >>>>>>>>>>>>> >
> > >>>>>>>>>>>>> > It would be nice to give more info to diagnosis.
> > >>>>>>>>>>>>> > The output of 'btrfs fi usage' is useful for space usage problem.
> > >>>>>>>>>>>>> >
> > >>>>>>>>>>>>> > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
> > >>>>>>>>>>>>> > space in vanilla df command) space for btrfs.
> > >>>>>>>>>>>>> >
> > >>>>>>>>>>>>> > Thanks,
> > >>>>>>>>>>>>> > Qu
> > >>>>>>>>>>>>> >
> > >>>>>>>>>>>>> > >
> > >>>>>>>>>>>>> > > Thank you so much for all of your help. I love how flexible BTRFS is
> > >>>>>>>>>>>>> > > but when things go wrong it's very hard for me to troubleshoot.
> > >>>>>>>>>>>>> > >
> > >>>>>>>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> > >>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > >>>>>>>>>>>>> > >>
> > >>>>>>>>>>>>> > >>
> > >>>>>>>>>>>>> > >>
> > >>>>>>>>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
> > >>>>>>>>>>>>> > >>> Something went wrong:
> > >>>>>>>>>>>>> > >>>
> > >>>>>>>>>>>>> > >>> Reinitialize checksum tree
> > >>>>>>>>>>>>> > >>> Unable to find block group for 0
> > >>>>>>>>>>>>> > >>> Unable to find block group for 0
> > >>>>>>>>>>>>> > >>> Unable to find block group for 0
> > >>>>>>>>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
> > >>>>>>>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
> > >>>>>>>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
> > >>>>>>>>>>>>> > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
> > >>>>>>>>>>>>> > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
> > >>>>>>>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
> > >>>>>>>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
> > >>>>>>>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
> > >>>>>>>>>>>>> > >>>
> > >>>>>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
> > >>>>>>>>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
> > >>>>>>>>>>>>> > >>> Aborted
> > >>>>>>>>>>>>> > >>
> > >>>>>>>>>>>>> > >> This means no space for extra metadata...
> > >>>>>>>>>>>>> > >>
> > >>>>>>>>>>>>> > >> Anyway the csum tree problem shouldn't be a big thing, you
> > >>>>>>>>>>>>> could leave
> > >>>>>>>>>>>>> > >> it and call it a day.
> > >>>>>>>>>>>>> > >>
> > >>>>>>>>>>>>> > >> BTW, as long as btrfs check reports no extra problem for the inode
> > >>>>>>>>>>>>> > >> generation, it should be pretty safe to use the fs.
> > >>>>>>>>>>>>> > >>
> > >>>>>>>>>>>>> > >> Thanks,
> > >>>>>>>>>>>>> > >> Qu
> > >>>>>>>>>>>>> > >>>
> > >>>>>>>>>>>>> > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
> > >>>>>>>>>>>>> > >>> available. I'll let that try overnight?
> > >>>>>>>>>>>>> > >>>
> > >>>>>>>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
> > >>>>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > >>>>>>>>>>>>> > >>>>
> > >>>>>>>>>>>>> > >>>>
> > >>>>>>>>>>>>> > >>>>
> > >>>>>>>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
> > >>>>>>>>>>>>> > >>>>> Thank you for helping. The end result of the scan was:
> > >>>>>>>>>>>>> > >>>>>
> > >>>>>>>>>>>>> > >>>>>
> > >>>>>>>>>>>>> > >>>>> [1/7] checking root items
> > >>>>>>>>>>>>> > >>>>> [2/7] checking extents
> > >>>>>>>>>>>>> > >>>>> [3/7] checking free space cache
> > >>>>>>>>>>>>> > >>>>> [4/7] checking fs roots
> > >>>>>>>>>>>>> > >>>>
> > >>>>>>>>>>>>> > >>>> Good news is, your fs is still mostly fine.
> > >>>>>>>>>>>>> > >>>>
> > >>>>>>>>>>>>> > >>>>> [5/7] checking only csums items (without verifying data)
> > >>>>>>>>>>>>> > >>>>> there are no extents for csum range 0-69632
> > >>>>>>>>>>>>> > >>>>> csum exists for 0-69632 but there is no extent record
> > >>>>>>>>>>>>> > >>>>> ...
> > >>>>>>>>>>>>> > >>>>> ...
> > >>>>>>>>>>>>> > >>>>> there are no extents for csum range 946692096-946827264
> > >>>>>>>>>>>>> > >>>>> csum exists for 946692096-946827264 but there is no extent
> > >>>>>>>>>>>>> record
> > >>>>>>>>>>>>> > >>>>> there are no extents for csum range 946831360-947912704
> > >>>>>>>>>>>>> > >>>>> csum exists for 946831360-947912704 but there is no extent
> > >>>>>>>>>>>>> record
> > >>>>>>>>>>>>> > >>>>> ERROR: errors found in csum tree
> > >>>>>>>>>>>>> > >>>>
> > >>>>>>>>>>>>> > >>>> Only extent tree is corrupted.
> > >>>>>>>>>>>>> > >>>>
> > >>>>>>>>>>>>> > >>>> Normally btrfs check --init-csum-tree should be able to
> > >>>>>>>>>>>>> handle it.
> > >>>>>>>>>>>>> > >>>>
> > >>>>>>>>>>>>> > >>>> But still, please be sure you're using the latest btrfs-progs
> > >>>>>>>>>>>>> to fix it.
> > >>>>>>>>>>>>> > >>>>
> > >>>>>>>>>>>>> > >>>> Thanks,
> > >>>>>>>>>>>>> > >>>> Qu
> > >>>>>>>>>>>>> > >>>>
> > >>>>>>>>>>>>> > >>>>> [6/7] checking root refs
> > >>>>>>>>>>>>> > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
> > >>>>>>>>>>>>> > >>>>> found 44157956026368 bytes used, error(s) found
> > >>>>>>>>>>>>> > >>>>> total csum bytes: 42038602716
> > >>>>>>>>>>>>> > >>>>> total tree bytes: 49688616960
> > >>>>>>>>>>>>> > >>>>> total fs tree bytes: 1256427520
> > >>>>>>>>>>>>> > >>>>> total extent tree bytes: 1709105152
> > >>>>>>>>>>>>> > >>>>> btree space waste bytes: 3172727316
> > >>>>>>>>>>>>> > >>>>> file data blocks allocated: 261625653436416
> > >>>>>>>>>>>>> > >>>>> referenced 47477768499200
> > >>>>>>>>>>>>> > >>>>>
> > >>>>>>>>>>>>> > >>>>> What do I need to do to fix all of this?
> > >>>>>>>>>>>>> > >>>>>
> > >>>>>>>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
> > >>>>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > >>>>>>>>>>>>> > >>>>>>
> > >>>>>>>>>>>>> > >>>>>>
> > >>>>>>>>>>>>> > >>>>>>
> > >>>>>>>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
> > >>>>>>>>>>>>> > >>>>>>> Well, the repair doesn't look terribly successful.
> > >>>>>>>>>>>>> > >>>>>>>
> > >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > >>>>>>>>>>>>> 6875841 found 6876224
> > >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > >>>>>>>>>>>>> 6875841 found 6876224
> > >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > >>>>>>>>>>>>> 6875841 found 6876224
> > >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> > >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > >>>>>>>>>>>>> item=84
> > >>>>>>>>>>>>> > >>>>>>> parent level=1
> > >>>>>>>>>>>>> > >>>>>>> child level=4
> > >>>>>>>>>>>>> > >>>>>>
> > >>>>>>>>>>>>> > >>>>>> This means there are more problems, not only the hash name
> > >>>>>>>>>>>>> mismatch.
> > >>>>>>>>>>>>> > >>>>>>
> > >>>>>>>>>>>>> > >>>>>> This means the fs is already corrupted, the name hash is
> > >>>>>>>>>>>>> just one
> > >>>>>>>>>>>>> > >>>>>> unrelated symptom.
> > >>>>>>>>>>>>> > >>>>>>
> > >>>>>>>>>>>>> > >>>>>> The only good news is, btrfs-progs abort the transaction,
> > >>>>>>>>>>>>> thus no
> > >>>>>>>>>>>>> > >>>>>> further damage to the fs.
> > >>>>>>>>>>>>> > >>>>>>
> > >>>>>>>>>>>>> > >>>>>> Please run a plain btrfs-check to show what's the problem
> > >>>>>>>>>>>>> first.
> > >>>>>>>>>>>>> > >>>>>>
> > >>>>>>>>>>>>> > >>>>>> Thanks,
> > >>>>>>>>>>>>> > >>>>>> Qu
> > >>>>>>>>>>>>> > >>>>>>
> > >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > >>>>>>>>>>>>> 6875841 found 6876224
> > >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> > >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > >>>>>>>>>>>>> item=84
> > >>>>>>>>>>>>> > >>>>>>> parent level=1
> > >>>>>>>>>>>>> > >>>>>>> child level=4
> > >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > >>>>>>>>>>>>> 6875841 found 6876224
> > >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> > >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > >>>>>>>>>>>>> item=84
> > >>>>>>>>>>>>> > >>>>>>> parent level=1
> > >>>>>>>>>>>>> > >>>>>>> child level=4
> > >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > >>>>>>>>>>>>> 6875841 found 6876224
> > >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> > >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > >>>>>>>>>>>>> item=84
> > >>>>>>>>>>>>> > >>>>>>> parent level=1
> > >>>>>>>>>>>>> > >>>>>>> child level=4
> > >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > >>>>>>>>>>>>> 6875841 found 6876224
> > >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> > >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > >>>>>>>>>>>>> item=84
> > >>>>>>>>>>>>> > >>>>>>> parent level=1
> > >>>>>>>>>>>>> > >>>>>>> child level=4
> > >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > >>>>>>>>>>>>> 6875841 found 6876224
> > >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> > >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > >>>>>>>>>>>>> item=84
> > >>>>>>>>>>>>> > >>>>>>> parent level=1
> > >>>>>>>>>>>>> > >>>>>>> child level=4
> > >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > >>>>>>>>>>>>> 6875841 found 6876224
> > >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> > >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > >>>>>>>>>>>>> item=84
> > >>>>>>>>>>>>> > >>>>>>> parent level=1
> > >>>>>>>>>>>>> > >>>>>>> child level=4
> > >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > >>>>>>>>>>>>> 6875841 found 6876224
> > >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> > >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > >>>>>>>>>>>>> item=84
> > >>>>>>>>>>>>> > >>>>>>> parent level=1
> > >>>>>>>>>>>>> > >>>>>>> child level=4
> > >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > >>>>>>>>>>>>> 6875841 found 6876224
> > >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> > >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > >>>>>>>>>>>>> item=84
> > >>>>>>>>>>>>> > >>>>>>> parent level=1
> > >>>>>>>>>>>>> > >>>>>>> child level=4
> > >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > >>>>>>>>>>>>> 6875841 found 6876224
> > >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> > >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > >>>>>>>>>>>>> item=84
> > >>>>>>>>>>>>> > >>>>>>> parent level=1
> > >>>>>>>>>>>>> > >>>>>>> child level=4
> > >>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
> > >>>>>>>>>>>>> 6875841 found 6876224
> > >>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
> > >>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
> > >>>>>>>>>>>>> item=84
> > >>>>>>>>>>>>> > >>>>>>> parent level=1
> > >>>>>>>>>>>>> > >>>>>>> child level=4
> > >>>>>>>>>>>>> > >>>>>>> ERROR: failed to zero log tree: -17
> > >>>>>>>>>>>>> > >>>>>>> ERROR: attempt to start transaction over already running one
> > >>>>>>>>>>>>> > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
> > >>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> > >>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
> > >>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> > >>>>>>>>>>>>> 225049066086400 len 4096
> > >>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> > >>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
> > >>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> > >>>>>>>>>>>>> 225049066094592 len 4096
> > >>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> > >>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
> > >>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> > >>>>>>>>>>>>> 225049066102784 len 4096
> > >>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> > >>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
> > >>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
> > >>>>>>>>>>>>> 225049066131456 len 4096
> > >>>>>>>>>>>>> > >>>>>>>
> > >>>>>>>>>>>>> > >>>>>>> What is going on?
> > >>>>>>>>>>>>> > >>>>>>>
> > >>>>>>>>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond
> > >>>>>>>>>>>>> <t.d.richmond@gmail.com <mailto:t.d.richmond@gmail.com>> wrote:
> > >>>>>>>>>>>>> > >>>>>>>>
> > >>>>>>>>>>>>> > >>>>>>>> Chris, I had used the correct mountpoint in the command.
> > >>>>>>>>>>>>> I just edited
> > >>>>>>>>>>>>> > >>>>>>>> it in the email to be /mountpoint for consistency.
> > >>>>>>>>>>>>> > >>>>>>>>
> > >>>>>>>>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
> > >>>>>>>>>>>>> > >>>>>>>>
> > >>>>>>>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
> > >>>>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
> > >>>>>>>>>>>>> > >>>>>>>>>
> > >>>>>>>>>>>>> > >>>>>>>>>
> > >>>>>>>>>>>>> > >>>>>>>>>
> > >>>>>>>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
> > >>>>>>>>>>>>> > >>>>>>>>>> Hello,
> > >>>>>>>>>>>>> > >>>>>>>>>>
> > >>>>>>>>>>>>> > >>>>>>>>>> I looked up this error and it basically says ask a
> > >>>>>>>>>>>>> developer to
> > >>>>>>>>>>>>> > >>>>>>>>>> determine if it's a false error or not. I just started
> > >>>>>>>>>>>>> getting some
> > >>>>>>>>>>>>> > >>>>>>>>>> slow response times, and looked at the dmesg log to
> > >>>>>>>>>>>>> find a ton of
> > >>>>>>>>>>>>> > >>>>>>>>>> these errors.
> > >>>>>>>>>>>>> > >>>>>>>>>>
> > >>>>>>>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt
> > >>>>>>>>>>>>> leaf: root=5
> > >>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> > >>>>>>>>>>>>> generation:
> > >>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> > >>>>>>>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error (device sdh):
> > >>>>>>>>>>>>> block=203510940835840 read
> > >>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
> > >>>>>>>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt
> > >>>>>>>>>>>>> leaf: root=5
> > >>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> > >>>>>>>>>>>>> generation:
> > >>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> > >>>>>>>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error (device sdh):
> > >>>>>>>>>>>>> block=203510940835840 read
> > >>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
> > >>>>>>>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt
> > >>>>>>>>>>>>> leaf: root=5
> > >>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
> > >>>>>>>>>>>>> generation:
> > >>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
> > >>>>>>>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error (device sdh):
> > >>>>>>>>>>>>> block=203510940835840 read
> > >>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
> > >>>>>>>>>>>>> > >>>>>>>>>>
> > >>>>>>>>>>>>> > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
> > >>>>>>>>>>>>> > >>>>>>>>>>
> > >>>>>>>>>>>>> > >>>>>>>>>> Is there anything I should do about this, or should I
> > >>>>>>>>>>>>> just continue
> > >>>>>>>>>>>>> > >>>>>>>>>> using my array as normal?
> > >>>>>>>>>>>>> > >>>>>>>>>
> > >>>>>>>>>>>>> > >>>>>>>>> This is caused by older kernel underflow inode generation.
> > >>>>>>>>>>>>> > >>>>>>>>>
> > >>>>>>>>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
> > >>>>>>>>>>>>> > >>>>>>>>>
> > >>>>>>>>>>>>> > >>>>>>>>> Or you can go safer, by manually locating the inode
> > >>>>>>>>>>>>> using its inode
> > >>>>>>>>>>>>> > >>>>>>>>> number (1311670), and copy it to some new location using
> > >>>>>>>>>>>>> previous
> > >>>>>>>>>>>>> > >>>>>>>>> working kernel, then delete the old file, copy the new
> > >>>>>>>>>>>>> one back to fix it.
> > >>>>>>>>>>>>> > >>>>>>>>>
> > >>>>>>>>>>>>> > >>>>>>>>> Thanks,
> > >>>>>>>>>>>>> > >>>>>>>>> Qu
> > >>>>>>>>>>>>> > >>>>>>>>>
> > >>>>>>>>>>>>> > >>>>>>>>>>
> > >>>>>>>>>>>>> > >>>>>>>>>> Thank you!
> > >>>>>>>>>>>>> > >>>>>>>>>>
> > >>>>>>>>>>>>> > >>>>>>>>>
> > >>>>>>>>>>>>> > >>>>>>
> > >>>>>>>>>>>>> > >>>>
> > >>>>>>>>>>>>> > >>
> > >>>>>>>>>>>>> >
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-05 7:01 ` Tyler Richmond
@ 2020-11-05 7:19 ` Qu Wenruo
2020-11-05 20:08 ` Ferry Toth
0 siblings, 1 reply; 56+ messages in thread
From: Qu Wenruo @ 2020-11-05 7:19 UTC (permalink / raw)
To: Tyler Richmond; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 30000 bytes --]
On 2020/11/5 下午3:01, Tyler Richmond wrote:
> Qu,
>
> I'm wondering, was a fix for this ever implemented?
Already implemented the --repair ability in latest btrfs-progs.
> I recently added a
> new drive to expand the array, and during the rebalance it dropped
> itself back to a read only filesystem. I suspect it's related to the
> issues discussed earlier in this thread. Is there anything I can do to
> complete the balance? The error that caused it to drop to read only is
> here: https://pastebin.com/GGYVMaiG
Yep, the same cause.
Thanks,
Qu
>
> Thanks!
>
>
> On Tue, Aug 25, 2020 at 9:43 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>>
>> Great, glad we got somewhere! I'll look forward to the fix!
>>
>> On Tue, Aug 25, 2020 at 9:38 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>
>>>
>>>
>>> On 2020/8/25 下午9:30, Tyler Richmond wrote:
>>>> Qu,
>>>>
>>>> The dump of the block is:
>>>>
>>>> https://pastebin.com/ran85JJv
>>>>
>>>> I've also completed the btrfs-image, but it's almost 50gb. What's the
>>>> best way to get it to you? Also, does it work with -ss or are the
>>>> original filenames important?
>>>
>>> 50G is too big for me to even receive.
>>>
>>> But your dump shows the problem!
>>>
>>> It's not inode generation, but inode transid, which would affect send.
>>>
>>> This is not even checked in btrfs-progs, thus no wonder why it doesn't
>>> detect them.
>>>
>>> And copy-pasted kernel message shares the same "generation" word, not
>>> using proper transid to show the problem.
>>>
>>> Your dump really saved the day!
>>>
>>> The fix for kernel and btrfs-progs would come in next few days.
>>>
>>> Thanks,
>>> Qu
>>>>
>>>> Thanks again!
>>>>
>>>>
>>>> On Tue, Aug 25, 2020 at 2:37 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 2020/8/25 下午1:25, Tyler Richmond wrote:
>>>>>> Qu,
>>>>>>
>>>>>> Yes, it's btrfs-progs 5.7. Here is the result of the lowmem check:
>>>>>>
>>>>>> https://pastebin.com/8Tzx23EX
>>>>>
>>>>> That doesn't detect any inode generation problem at all, which is not a
>>>>> good sign.
>>>>>
>>>>> Would you also pvode the dump for the offending block?
>>>>>
>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>
>>>>> For this case, would you please provide the tree dump of "203510940835840" ?
>>>>>
>>>>> # btrfs ins dump-tree -b 203510940835840 <device>
>>>>>
>>>>> And, since btrfs-image can't dump with regular extent tree, the "-w"
>>>>> dump would also help.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> On Mon, Aug 24, 2020 at 4:26 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 2020/8/24 上午10:47, Tyler Richmond wrote:
>>>>>>>> Qu,
>>>>>>>>
>>>>>>>> Finally finished another repair and captured the output.
>>>>>>>>
>>>>>>>> https://pastebin.com/ffcbwvd8
>>>>>>>>
>>>>>>>> Does that show you what you need? Or should I still do one in lowmem mode?
>>>>>>>
>>>>>>> Lowmem mode (no need for --repair) is recommended since original mode
>>>>>>> doesn't detect the inode generation problem.
>>>>>>>
>>>>>>> And it's already btrfs-progs v5.7 right?
>>>>>>>
>>>>>>> THanks,
>>>>>>> Qu
>>>>>>>>
>>>>>>>> Thanks for your help!
>>>>>>>>
>>>>>>>> On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2020/8/23 上午10:49, Tyler Richmond wrote:
>>>>>>>>>> Well, I can guarantee that I didn't create this fs before 2015 (just
>>>>>>>>>> checked the order confirmation from when I bought the server), but I
>>>>>>>>>> may have just used whatever was in the Ubuntu package manager at the
>>>>>>>>>> time. So maybe I don't have a v0 ref?
>>>>>>>>>
>>>>>>>>> Then btrfs-image shouldn't report that.
>>>>>>>>>
>>>>>>>>> There is an item smaller than any valid btrfs item, normally it means
>>>>>>>>> it's a v0 ref.
>>>>>>>>> If not, then it could be a bigger problem.
>>>>>>>>>
>>>>>>>>> Could you please provide the full btrfs-check output?
>>>>>>>>> Also, if possible result from "btrfs check --mode=lowmem" would also help.
>>>>>>>>>
>>>>>>>>> Also, if you really go "--repair", then the full output would also be
>>>>>>>>> needed to determine what's going wrong.
>>>>>>>>> There is a report about "btrfs check --repair" didn't repair the inode
>>>>>>>>> generation, if that's the case we must have a bug then.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Qu
>>>>>>>>>>
>>>>>>>>>> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
>>>>>>>>>>>>> Is my best bet just to downgrade the kernel and then try to delete the
>>>>>>>>>>>>> broken files? Or should I rebuild from scratch? Just don't know
>>>>>>>>>>>>> whether it's worth the time to try and figure this out or if the
>>>>>>>>>>>>> problems stem from the FS being too old and it's beyond trying to
>>>>>>>>>>>>> repair.
>>>>>>>>>>>>
>>>>>>>>>>>> All invalid inode generations, should be able to be repaired by latest
>>>>>>>>>>>> btrfs-check.
>>>>>>>>>>>>
>>>>>>>>>>>> If not, please provide the btrfs-image dump for us to determine what's
>>>>>>>>>>>> going wrong.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Qu
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I didn't check dmesg during the btrfs check, but that was the only
>>>>>>>>>>>>>> output during the rm -f before it was forced readonly. I just checked
>>>>>>>>>>>>>> dmesg for inode generation values, and there are a lot of them.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://pastebin.com/stZdN0ta
>>>>>>>>>>>>>> The dmesg output had 990 lines containing inode generation.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> However, these were at least later. I tried to do a btrfs balance
>>>>>>>>>>>>>> -mconvert raid1 and it failed with an I/O error. That is probably what
>>>>>>>>>>>>>> generated these specific errors, but maybe they were also happening
>>>>>>>>>>>>>> during the btrfs repair.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway with:
>>>>>>>>>>>>>> ERROR: either extent tree is corrupted or deprecated extent ref format
>>>>>>>>>>>>>> ERROR: create failed: -5
>>>>>>>>>>>
>>>>>>>>>>> Oh, forgot this part.
>>>>>>>>>>>
>>>>>>>>>>> This means you have v0 ref?!
>>>>>>>>>>>
>>>>>>>>>>> Then the fs is too old, no progs/kernel support after all.
>>>>>>>>>>>
>>>>>>>>>>> In that case, please rollback to the last working kernel and copy your data.
>>>>>>>>>>>
>>>>>>>>>>> In fact, that v0 ref should only be in the code base for several weeks
>>>>>>>>>>> before 2010, thus it's really too old.
>>>>>>>>>>>
>>>>>>>>>>> The good news is, with tree-checker, we should never experience such
>>>>>>>>>>> too-old-to-be-usable problem (at least I hope so)
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Qu
>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
>>>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sorry to resurrect this thread, but I just ran into something that I
>>>>>>>>>>>>>>>> can't really just ignore. I've found a folder that is full of files
>>>>>>>>>>>>>>>> which I guess have been broken somehow. I found a backup and restored
>>>>>>>>>>>>>>>> them, but I want to delete this folder of broken files. But whenever I
>>>>>>>>>>>>>>>> try, the fs is forced into readonly mode again. I just finished another
>>>>>>>>>>>>>>>> btrfs check --repair but it didn't fix the problem.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> https://pastebin.com/eTV3s3fr
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is that the full output?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> No inode generation bugs?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm already on btrfs-progs v5.7. Any new suggestions?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Strange.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The detection and repair should have been merged into v5.5.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If your fs is small enough, would you please provide the "btrfs-image
>>>>>>>>>>>>>>> -c9" dump?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It would contain the filenames and directories names, but doesn't
>>>>>>>>>>>>>>> contain file contents.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 5.6.1 also failed the same way. Here's the usage output. This is the
>>>>>>>>>>>>>>>> part where you see I've been using RAID5 haha
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> WARNING: RAID56 detected, not implemented
>>>>>>>>>>>>>>>> Overall:
>>>>>>>>>>>>>>>> Device size: 60.03TiB
>>>>>>>>>>>>>>>> Device allocated: 98.06GiB
>>>>>>>>>>>>>>>> Device unallocated: 59.93TiB
>>>>>>>>>>>>>>>> Device missing: 0.00B
>>>>>>>>>>>>>>>> Used: 92.56GiB
>>>>>>>>>>>>>>>> Free (estimated): 0.00B (min: 8.00EiB)
>>>>>>>>>>>>>>>> Data ratio: 0.00
>>>>>>>>>>>>>>>> Metadata ratio: 2.00
>>>>>>>>>>>>>>>> Global reserve: 512.00MiB (used: 0.00B)
>>>>>>>>>>>>>>>> Multiple profiles: no
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
>>>>>>>>>>>>>>>> /dev/sdh 8.07TiB
>>>>>>>>>>>>>>>> /dev/sdf 8.07TiB
>>>>>>>>>>>>>>>> /dev/sdg 8.07TiB
>>>>>>>>>>>>>>>> /dev/sdd 8.07TiB
>>>>>>>>>>>>>>>> /dev/sdc 8.07TiB
>>>>>>>>>>>>>>>> /dev/sde 8.07TiB
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
>>>>>>>>>>>>>>>> /dev/sdh 34.00GiB
>>>>>>>>>>>>>>>> /dev/sdf 32.00GiB
>>>>>>>>>>>>>>>> /dev/sdg 32.00GiB
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
>>>>>>>>>>>>>>>> /dev/sdf 32.00MiB
>>>>>>>>>>>>>>>> /dev/sdg 32.00MiB
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Unallocated:
>>>>>>>>>>>>>>>> /dev/sdh 2.81TiB
>>>>>>>>>>>>>>>> /dev/sdf 2.81TiB
>>>>>>>>>>>>>>>> /dev/sdg 2.81TiB
>>>>>>>>>>>>>>>> /dev/sdd 1.03TiB
>>>>>>>>>>>>>>>> /dev/sdc 1.03TiB
>>>>>>>>>>>>>>>> /dev/sde 1.03TiB
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
>>>>>>>>>>>>>>>> > > If this is saying there's no extra space for metadata, is that why
>>>>>>>>>>>>>>>> > > adding more files often makes the system hang for 30-90s? Is there
>>>>>>>>>>>>>>>> > > anything I should do about that?
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > I'm not sure about the hang though.
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > It would be nice to give more info to diagnosis.
>>>>>>>>>>>>>>>> > The output of 'btrfs fi usage' is useful for space usage problem.
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
>>>>>>>>>>>>>>>> > space in vanilla df command) space for btrfs.
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>> > Qu
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>> > > Thank you so much for all of your help. I love how flexible BTRFS is
>>>>>>>>>>>>>>>> > > but when things go wrong it's very hard for me to troubleshoot.
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
>>>>>>>>>>>>>>>> > >>> Something went wrong:
>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>> > >>> Reinitialize checksum tree
>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
>>>>>>>>>>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
>>>>>>>>>>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
>>>>>>>>>>>>>>>> > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
>>>>>>>>>>>>>>>> > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
>>>>>>>>>>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
>>>>>>>>>>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
>>>>>>>>>>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
>>>>>>>>>>>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
>>>>>>>>>>>>>>>> > >>> Aborted
>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>> > >> This means no space for extra metadata...
>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>> > >> Anyway the csum tree problem shouldn't be a big thing, you
>>>>>>>>>>>>>>>> could leave
>>>>>>>>>>>>>>>> > >> it and call it a day.
>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>> > >> BTW, as long as btrfs check reports no extra problem for the inode
>>>>>>>>>>>>>>>> > >> generation, it should be pretty safe to use the fs.
>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>> > >> Thanks,
>>>>>>>>>>>>>>>> > >> Qu
>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>> > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
>>>>>>>>>>>>>>>> > >>> available. I'll let that try overnight?
>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
>>>>>>>>>>>>>>>> > >>>>> Thank you for helping. The end result of the scan was:
>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>> > >>>>> [1/7] checking root items
>>>>>>>>>>>>>>>> > >>>>> [2/7] checking extents
>>>>>>>>>>>>>>>> > >>>>> [3/7] checking free space cache
>>>>>>>>>>>>>>>> > >>>>> [4/7] checking fs roots
>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>> > >>>> Good news is, your fs is still mostly fine.
>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>> > >>>>> [5/7] checking only csums items (without verifying data)
>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range 0-69632
>>>>>>>>>>>>>>>> > >>>>> csum exists for 0-69632 but there is no extent record
>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range 946692096-946827264
>>>>>>>>>>>>>>>> > >>>>> csum exists for 946692096-946827264 but there is no extent
>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range 946831360-947912704
>>>>>>>>>>>>>>>> > >>>>> csum exists for 946831360-947912704 but there is no extent
>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>> > >>>>> ERROR: errors found in csum tree
>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>> > >>>> Only extent tree is corrupted.
>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>> > >>>> Normally btrfs check --init-csum-tree should be able to
>>>>>>>>>>>>>>>> handle it.
>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>> > >>>> But still, please be sure you're using the latest btrfs-progs
>>>>>>>>>>>>>>>> to fix it.
>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>> > >>>> Thanks,
>>>>>>>>>>>>>>>> > >>>> Qu
>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>> > >>>>> [6/7] checking root refs
>>>>>>>>>>>>>>>> > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>>>>>>>>>>>>>>> > >>>>> found 44157956026368 bytes used, error(s) found
>>>>>>>>>>>>>>>> > >>>>> total csum bytes: 42038602716
>>>>>>>>>>>>>>>> > >>>>> total tree bytes: 49688616960
>>>>>>>>>>>>>>>> > >>>>> total fs tree bytes: 1256427520
>>>>>>>>>>>>>>>> > >>>>> total extent tree bytes: 1709105152
>>>>>>>>>>>>>>>> > >>>>> btree space waste bytes: 3172727316
>>>>>>>>>>>>>>>> > >>>>> file data blocks allocated: 261625653436416
>>>>>>>>>>>>>>>> > >>>>> referenced 47477768499200
>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>> > >>>>> What do I need to do to fix all of this?
>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
>>>>>>>>>>>>>>>> > >>>>>>> Well, the repair doesn't look terribly successful.
>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>> > >>>>>> This means there are more problems, not only the hash name
>>>>>>>>>>>>>>>> mismatch.
>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>> > >>>>>> This means the fs is already corrupted, the name hash is
>>>>>>>>>>>>>>>> just one
>>>>>>>>>>>>>>>> > >>>>>> unrelated symptom.
>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>> > >>>>>> The only good news is, btrfs-progs abort the transaction,
>>>>>>>>>>>>>>>> thus no
>>>>>>>>>>>>>>>> > >>>>>> further damage to the fs.
>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>> > >>>>>> Please run a plain btrfs-check to show what's the problem
>>>>>>>>>>>>>>>> first.
>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>>>> > >>>>>> Qu
>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>> > >>>>>>> ERROR: failed to zero log tree: -17
>>>>>>>>>>>>>>>> > >>>>>>> ERROR: attempt to start transaction over already running one
>>>>>>>>>>>>>>>> > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>> What is going on?
>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond
>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>>> Chris, I had used the correct mountpoint in the command.
>>>>>>>>>>>>>>>> I just edited
>>>>>>>>>>>>>>>> > >>>>>>>> it in the email to be /mountpoint for consistency.
>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
>>>>>>>>>>>>>>>> > >>>>>>>>>> Hello,
>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>>>>> I looked up this error and it basically says ask a
>>>>>>>>>>>>>>>> developer to
>>>>>>>>>>>>>>>> > >>>>>>>>>> determine if it's a false error or not. I just started
>>>>>>>>>>>>>>>> getting some
>>>>>>>>>>>>>>>> > >>>>>>>>>> slow response times, and looked at the dmesg log to
>>>>>>>>>>>>>>>> find a ton of
>>>>>>>>>>>>>>>> > >>>>>>>>>> these errors.
>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt
>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error (device sdh):
>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt
>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error (device sdh):
>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt
>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error (device sdh):
>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>>>>> Is there anything I should do about this, or should I
>>>>>>>>>>>>>>>> just continue
>>>>>>>>>>>>>>>> > >>>>>>>>>> using my array as normal?
>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>>>> This is caused by older kernel underflow inode generation.
>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>>>> Or you can go safer, by manually locating the inode
>>>>>>>>>>>>>>>> using its inode
>>>>>>>>>>>>>>>> > >>>>>>>>> number (1311670), and copy it to some new location using
>>>>>>>>>>>>>>>> previous
>>>>>>>>>>>>>>>> > >>>>>>>>> working kernel, then delete the old file, copy the new
>>>>>>>>>>>>>>>> one back to fix it.
>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> > >>>>>>>>> Qu
>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-05 7:19 ` Qu Wenruo
@ 2020-11-05 20:08 ` Ferry Toth
2020-11-05 23:00 ` Qu Wenruo
0 siblings, 1 reply; 56+ messages in thread
From: Ferry Toth @ 2020-11-05 20:08 UTC (permalink / raw)
To: Qu Wenruo, Tyler Richmond; +Cc: Btrfs BTRFS
I am in a similar spot, during updating my distro (Kubuntu), I am unable
to update a certain package. I know which file it is:
~$ ls -l /usr/share/doc/libatk1.0-data
ls: kan geen toegang krijgen tot '/usr/share/doc/libatk1.0-data':
Invoer-/uitvoerfout
This creates the following in journal:
kernel: BTRFS critical (device sda2): corrupt leaf: root=294
block=1169152675840 slot=1 ino=915987, invalid inode generation: has
18446744073709551492 expect [0, 5851353]
kernel: BTRFS error (device sda2): block=1169152675840 read time tree
block corruption detected
Now, the problem: this file is on my rootfs, which is mounted. apt
(distribution updated) installed all packages but can't continue
configuring, because libatk is a dependancy. I can't delete the file
because of the I/O error. And btrfs check complains (I tried running RO)
because the file system is mounted.
But, on the sunny side, the file system is not RO.
Is there any way to forcefully remove the file? Or do you have a
recommendation how to proceed?
Linux = 5.6.0-1032-oem
Thanks,
Ferry
Op 05-11-2020 om 08:19 schreef Qu Wenruo:
>
>
> On 2020/11/5 下午3:01, Tyler Richmond wrote:
>> Qu,
>>
>> I'm wondering, was a fix for this ever implemented?
>
> Already implemented the --repair ability in latest btrfs-progs.
>
>> I recently added a
>> new drive to expand the array, and during the rebalance it dropped
>> itself back to a read only filesystem. I suspect it's related to the
>> issues discussed earlier in this thread. Is there anything I can do to
>> complete the balance? The error that caused it to drop to read only is
>> here: https://pastebin.com/GGYVMaiG
>
> Yep, the same cause.
>
> Thanks,
> Qu
>>
>> Thanks!
>>
>>
>> On Tue, Aug 25, 2020 at 9:43 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>>>
>>> Great, glad we got somewhere! I'll look forward to the fix!
>>>
>>> On Tue, Aug 25, 2020 at 9:38 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>
>>>>
>>>>
>>>> On 2020/8/25 下午9:30, Tyler Richmond wrote:
>>>>> Qu,
>>>>>
>>>>> The dump of the block is:
>>>>>
>>>>> https://pastebin.com/ran85JJv
>>>>>
>>>>> I've also completed the btrfs-image, but it's almost 50gb. What's the
>>>>> best way to get it to you? Also, does it work with -ss or are the
>>>>> original filenames important?
>>>>
>>>> 50G is too big for me to even receive.
>>>>
>>>> But your dump shows the problem!
>>>>
>>>> It's not inode generation, but inode transid, which would affect send.
>>>>
>>>> This is not even checked in btrfs-progs, thus no wonder why it doesn't
>>>> detect them.
>>>>
>>>> And copy-pasted kernel message shares the same "generation" word, not
>>>> using proper transid to show the problem.
>>>>
>>>> Your dump really saved the day!
>>>>
>>>> The fix for kernel and btrfs-progs would come in next few days.
>>>>
>>>> Thanks,
>>>> Qu
>>>>>
>>>>> Thanks again!
>>>>>
>>>>>
>>>>> On Tue, Aug 25, 2020 at 2:37 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 2020/8/25 下午1:25, Tyler Richmond wrote:
>>>>>>> Qu,
>>>>>>>
>>>>>>> Yes, it's btrfs-progs 5.7. Here is the result of the lowmem check:
>>>>>>>
>>>>>>> https://pastebin.com/8Tzx23EX
>>>>>>
>>>>>> That doesn't detect any inode generation problem at all, which is not a
>>>>>> good sign.
>>>>>>
>>>>>> Would you also pvode the dump for the offending block?
>>>>>>
>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>
>>>>>> For this case, would you please provide the tree dump of "203510940835840" ?
>>>>>>
>>>>>> # btrfs ins dump-tree -b 203510940835840 <device>
>>>>>>
>>>>>> And, since btrfs-image can't dump with regular extent tree, the "-w"
>>>>>> dump would also help.
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>
>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> On Mon, Aug 24, 2020 at 4:26 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2020/8/24 上午10:47, Tyler Richmond wrote:
>>>>>>>>> Qu,
>>>>>>>>>
>>>>>>>>> Finally finished another repair and captured the output.
>>>>>>>>>
>>>>>>>>> https://pastebin.com/ffcbwvd8
>>>>>>>>>
>>>>>>>>> Does that show you what you need? Or should I still do one in lowmem mode?
>>>>>>>>
>>>>>>>> Lowmem mode (no need for --repair) is recommended since original mode
>>>>>>>> doesn't detect the inode generation problem.
>>>>>>>>
>>>>>>>> And it's already btrfs-progs v5.7 right?
>>>>>>>>
>>>>>>>> THanks,
>>>>>>>> Qu
>>>>>>>>>
>>>>>>>>> Thanks for your help!
>>>>>>>>>
>>>>>>>>> On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 2020/8/23 上午10:49, Tyler Richmond wrote:
>>>>>>>>>>> Well, I can guarantee that I didn't create this fs before 2015 (just
>>>>>>>>>>> checked the order confirmation from when I bought the server), but I
>>>>>>>>>>> may have just used whatever was in the Ubuntu package manager at the
>>>>>>>>>>> time. So maybe I don't have a v0 ref?
>>>>>>>>>>
>>>>>>>>>> Then btrfs-image shouldn't report that.
>>>>>>>>>>
>>>>>>>>>> There is an item smaller than any valid btrfs item, normally it means
>>>>>>>>>> it's a v0 ref.
>>>>>>>>>> If not, then it could be a bigger problem.
>>>>>>>>>>
>>>>>>>>>> Could you please provide the full btrfs-check output?
>>>>>>>>>> Also, if possible result from "btrfs check --mode=lowmem" would also help.
>>>>>>>>>>
>>>>>>>>>> Also, if you really go "--repair", then the full output would also be
>>>>>>>>>> needed to determine what's going wrong.
>>>>>>>>>> There is a report about "btrfs check --repair" didn't repair the inode
>>>>>>>>>> generation, if that's the case we must have a bug then.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Qu
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
>>>>>>>>>>>>>> Is my best bet just to downgrade the kernel and then try to delete the
>>>>>>>>>>>>>> broken files? Or should I rebuild from scratch? Just don't know
>>>>>>>>>>>>>> whether it's worth the time to try and figure this out or if the
>>>>>>>>>>>>>> problems stem from the FS being too old and it's beyond trying to
>>>>>>>>>>>>>> repair.
>>>>>>>>>>>>>
>>>>>>>>>>>>> All invalid inode generations, should be able to be repaired by latest
>>>>>>>>>>>>> btrfs-check.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If not, please provide the btrfs-image dump for us to determine what's
>>>>>>>>>>>>> going wrong.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I didn't check dmesg during the btrfs check, but that was the only
>>>>>>>>>>>>>>> output during the rm -f before it was forced readonly. I just checked
>>>>>>>>>>>>>>> dmesg for inode generation values, and there are a lot of them.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://pastebin.com/stZdN0ta
>>>>>>>>>>>>>>> The dmesg output had 990 lines containing inode generation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> However, these were at least later. I tried to do a btrfs balance
>>>>>>>>>>>>>>> -mconvert raid1 and it failed with an I/O error. That is probably what
>>>>>>>>>>>>>>> generated these specific errors, but maybe they were also happening
>>>>>>>>>>>>>>> during the btrfs repair.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway with:
>>>>>>>>>>>>>>> ERROR: either extent tree is corrupted or deprecated extent ref format
>>>>>>>>>>>>>>> ERROR: create failed: -5
>>>>>>>>>>>>
>>>>>>>>>>>> Oh, forgot this part.
>>>>>>>>>>>>
>>>>>>>>>>>> This means you have v0 ref?!
>>>>>>>>>>>>
>>>>>>>>>>>> Then the fs is too old, no progs/kernel support after all.
>>>>>>>>>>>>
>>>>>>>>>>>> In that case, please rollback to the last working kernel and copy your data.
>>>>>>>>>>>>
>>>>>>>>>>>> In fact, that v0 ref should only be in the code base for several weeks
>>>>>>>>>>>> before 2010, thus it's really too old.
>>>>>>>>>>>>
>>>>>>>>>>>> The good news is, with tree-checker, we should never experience such
>>>>>>>>>>>> too-old-to-be-usable problem (at least I hope so)
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Qu
>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sorry to resurrect this thread, but I just ran into something that I
>>>>>>>>>>>>>>>>> can't really just ignore. I've found a folder that is full of files
>>>>>>>>>>>>>>>>> which I guess have been broken somehow. I found a backup and restored
>>>>>>>>>>>>>>>>> them, but I want to delete this folder of broken files. But whenever I
>>>>>>>>>>>>>>>>> try, the fs is forced into readonly mode again. I just finished another
>>>>>>>>>>>>>>>>> btrfs check --repair but it didn't fix the problem.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> https://pastebin.com/eTV3s3fr
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Is that the full output?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> No inode generation bugs?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm already on btrfs-progs v5.7. Any new suggestions?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Strange.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The detection and repair should have been merged into v5.5.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If your fs is small enough, would you please provide the "btrfs-image
>>>>>>>>>>>>>>>> -c9" dump?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It would contain the filenames and directories names, but doesn't
>>>>>>>>>>>>>>>> contain file contents.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 5.6.1 also failed the same way. Here's the usage output. This is the
>>>>>>>>>>>>>>>>> part where you see I've been using RAID5 haha
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> WARNING: RAID56 detected, not implemented
>>>>>>>>>>>>>>>>> Overall:
>>>>>>>>>>>>>>>>> Device size: 60.03TiB
>>>>>>>>>>>>>>>>> Device allocated: 98.06GiB
>>>>>>>>>>>>>>>>> Device unallocated: 59.93TiB
>>>>>>>>>>>>>>>>> Device missing: 0.00B
>>>>>>>>>>>>>>>>> Used: 92.56GiB
>>>>>>>>>>>>>>>>> Free (estimated): 0.00B (min: 8.00EiB)
>>>>>>>>>>>>>>>>> Data ratio: 0.00
>>>>>>>>>>>>>>>>> Metadata ratio: 2.00
>>>>>>>>>>>>>>>>> Global reserve: 512.00MiB (used: 0.00B)
>>>>>>>>>>>>>>>>> Multiple profiles: no
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
>>>>>>>>>>>>>>>>> /dev/sdh 8.07TiB
>>>>>>>>>>>>>>>>> /dev/sdf 8.07TiB
>>>>>>>>>>>>>>>>> /dev/sdg 8.07TiB
>>>>>>>>>>>>>>>>> /dev/sdd 8.07TiB
>>>>>>>>>>>>>>>>> /dev/sdc 8.07TiB
>>>>>>>>>>>>>>>>> /dev/sde 8.07TiB
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB (94.44%)
>>>>>>>>>>>>>>>>> /dev/sdh 34.00GiB
>>>>>>>>>>>>>>>>> /dev/sdf 32.00GiB
>>>>>>>>>>>>>>>>> /dev/sdg 32.00GiB
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
>>>>>>>>>>>>>>>>> /dev/sdf 32.00MiB
>>>>>>>>>>>>>>>>> /dev/sdg 32.00MiB
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Unallocated:
>>>>>>>>>>>>>>>>> /dev/sdh 2.81TiB
>>>>>>>>>>>>>>>>> /dev/sdf 2.81TiB
>>>>>>>>>>>>>>>>> /dev/sdg 2.81TiB
>>>>>>>>>>>>>>>>> /dev/sdd 1.03TiB
>>>>>>>>>>>>>>>>> /dev/sdc 1.03TiB
>>>>>>>>>>>>>>>>> /dev/sde 1.03TiB
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>> > > If this is saying there's no extra space for metadata, is that why
>>>>>>>>>>>>>>>>> > > adding more files often makes the system hang for 30-90s? Is there
>>>>>>>>>>>>>>>>> > > anything I should do about that?
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > I'm not sure about the hang though.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > It would be nice to give more info to diagnosis.
>>>>>>>>>>>>>>>>> > The output of 'btrfs fi usage' is useful for space usage problem.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > But the common idea is, to keep at 1~2 Gi unallocated (not avaiable
>>>>>>>>>>>>>>>>> > space in vanilla df command) space for btrfs.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>> > Qu
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > > Thank you so much for all of your help. I love how flexible BTRFS is
>>>>>>>>>>>>>>>>> > > but when things go wrong it's very hard for me to troubleshoot.
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>> > >>> Something went wrong:
>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>> > >>> Reinitialize checksum tree
>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1
>>>>>>>>>>>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
>>>>>>>>>>>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
>>>>>>>>>>>>>>>>> > >>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
>>>>>>>>>>>>>>>>> > >>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
>>>>>>>>>>>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
>>>>>>>>>>>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
>>>>>>>>>>>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
>>>>>>>>>>>>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
>>>>>>>>>>>>>>>>> > >>> Aborted
>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>> > >> This means no space for extra metadata...
>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>> > >> Anyway the csum tree problem shouldn't be a big thing, you
>>>>>>>>>>>>>>>>> could leave
>>>>>>>>>>>>>>>>> > >> it and call it a day.
>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>> > >> BTW, as long as btrfs check reports no extra problem for the inode
>>>>>>>>>>>>>>>>> > >> generation, it should be pretty safe to use the fs.
>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>> > >> Thanks,
>>>>>>>>>>>>>>>>> > >> Qu
>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>> > >>> I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is
>>>>>>>>>>>>>>>>> > >>> available. I'll let that try overnight?
>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>> > >>>>> Thank you for helping. The end result of the scan was:
>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>> > >>>>> [1/7] checking root items
>>>>>>>>>>>>>>>>> > >>>>> [2/7] checking extents
>>>>>>>>>>>>>>>>> > >>>>> [3/7] checking free space cache
>>>>>>>>>>>>>>>>> > >>>>> [4/7] checking fs roots
>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>> > >>>> Good news is, your fs is still mostly fine.
>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>> > >>>>> [5/7] checking only csums items (without verifying data)
>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range 0-69632
>>>>>>>>>>>>>>>>> > >>>>> csum exists for 0-69632 but there is no extent record
>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range 946692096-946827264
>>>>>>>>>>>>>>>>> > >>>>> csum exists for 946692096-946827264 but there is no extent
>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range 946831360-947912704
>>>>>>>>>>>>>>>>> > >>>>> csum exists for 946831360-947912704 but there is no extent
>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>> > >>>>> ERROR: errors found in csum tree
>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>> > >>>> Only extent tree is corrupted.
>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>> > >>>> Normally btrfs check --init-csum-tree should be able to
>>>>>>>>>>>>>>>>> handle it.
>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>> > >>>> But still, please be sure you're using the latest btrfs-progs
>>>>>>>>>>>>>>>>> to fix it.
>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>> > >>>> Thanks,
>>>>>>>>>>>>>>>>> > >>>> Qu
>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>> > >>>>> [6/7] checking root refs
>>>>>>>>>>>>>>>>> > >>>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>>>>>>>>>>>>>>>> > >>>>> found 44157956026368 bytes used, error(s) found
>>>>>>>>>>>>>>>>> > >>>>> total csum bytes: 42038602716
>>>>>>>>>>>>>>>>> > >>>>> total tree bytes: 49688616960
>>>>>>>>>>>>>>>>> > >>>>> total fs tree bytes: 1256427520
>>>>>>>>>>>>>>>>> > >>>>> total extent tree bytes: 1709105152
>>>>>>>>>>>>>>>>> > >>>>> btree space waste bytes: 3172727316
>>>>>>>>>>>>>>>>> > >>>>> file data blocks allocated: 261625653436416
>>>>>>>>>>>>>>>>> > >>>>> referenced 47477768499200
>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>> > >>>>> What do I need to do to fix all of this?
>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>> > >>>>>>> Well, the repair doesn't look terribly successful.
>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>> This means there are more problems, not only the hash name
>>>>>>>>>>>>>>>>> mismatch.
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>> This means the fs is already corrupted, the name hash is
>>>>>>>>>>>>>>>>> just one
>>>>>>>>>>>>>>>>> > >>>>>> unrelated symptom.
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>> The only good news is, btrfs-progs abort the transaction,
>>>>>>>>>>>>>>>>> thus no
>>>>>>>>>>>>>>>>> > >>>>>> further damage to the fs.
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>> Please run a plain btrfs-check to show what's the problem
>>>>>>>>>>>>>>>>> first.
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>>>>> > >>>>>> Qu
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on 218620880703488 wanted
>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent bytenr=225049956061184
>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>> > >>>>>>> child level=4
>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: failed to zero log tree: -17
>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: attempt to start transaction over already running one
>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096
>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066086400 len 4096
>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066094592 len 4096
>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066102784 len 4096
>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start 225049066131456 len 4096
>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans): start
>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>> What is going on?
>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond
>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>>> Chris, I had used the correct mountpoint in the command.
>>>>>>>>>>>>>>>>> I just edited
>>>>>>>>>>>>>>>>> > >>>>>>>> it in the email to be /mountpoint for consistency.
>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>> > >>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>>>>> I looked up this error and it basically says ask a
>>>>>>>>>>>>>>>>> developer to
>>>>>>>>>>>>>>>>> > >>>>>>>>>> determine if it's a false error or not. I just started
>>>>>>>>>>>>>>>>> getting some
>>>>>>>>>>>>>>>>> > >>>>>>>>>> slow response times, and looked at the dmesg log to
>>>>>>>>>>>>>>>>> find a ton of
>>>>>>>>>>>>>>>>> > >>>>>>>>>> these errors.
>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt
>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error (device sdh):
>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt
>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error (device sdh):
>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt
>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error (device sdh):
>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>>>>> btrfs device stats, however, doesn't show any errors.
>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>>>>> Is there anything I should do about this, or should I
>>>>>>>>>>>>>>>>> just continue
>>>>>>>>>>>>>>>>> > >>>>>>>>>> using my array as normal?
>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>>>> This is caused by older kernel underflow inode generation.
>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair.
>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>>>> Or you can go safer, by manually locating the inode
>>>>>>>>>>>>>>>>> using its inode
>>>>>>>>>>>>>>>>> > >>>>>>>>> number (1311670), and copy it to some new location using
>>>>>>>>>>>>>>>>> previous
>>>>>>>>>>>>>>>>> > >>>>>>>>> working kernel, then delete the old file, copy the new
>>>>>>>>>>>>>>>>> one back to fix it.
>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> > >>>>>>>>> Qu
>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-05 20:08 ` Ferry Toth
@ 2020-11-05 23:00 ` Qu Wenruo
2020-11-05 23:12 ` Ferry Toth
0 siblings, 1 reply; 56+ messages in thread
From: Qu Wenruo @ 2020-11-05 23:00 UTC (permalink / raw)
To: Ferry Toth, Tyler Richmond; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 38537 bytes --]
On 2020/11/6 上午4:08, Ferry Toth wrote:
> I am in a similar spot, during updating my distro (Kubuntu), I am unable
> to update a certain package. I know which file it is:
>
> ~$ ls -l /usr/share/doc/libatk1.0-data
> ls: kan geen toegang krijgen tot '/usr/share/doc/libatk1.0-data':
> Invoer-/uitvoerfout
>
> This creates the following in journal:
>
> kernel: BTRFS critical (device sda2): corrupt leaf: root=294
> block=1169152675840 slot=1 ino=915987, invalid inode generation: has
> 18446744073709551492 expect [0, 5851353]
> kernel: BTRFS error (device sda2): block=1169152675840 read time tree
> block corruption detected
>
> Now, the problem: this file is on my rootfs, which is mounted. apt
> (distribution updated) installed all packages but can't continue
> configuring, because libatk is a dependancy. I can't delete the file
> because of the I/O error. And btrfs check complains (I tried running RO)
> because the file system is mounted.
>
> But, on the sunny side, the file system is not RO.
>
> Is there any way to forcefully remove the file? Or do you have a
> recommendation how to proceed?
Newer kernel will reject to even read the item, thus will not be able to
remove it.
I guess you have to use some distro ISO to fix the fs.
Thanks,
Qu
>
> Linux = 5.6.0-1032-oem
>
> Thanks,
> Ferry
>
> Op 05-11-2020 om 08:19 schreef Qu Wenruo:
>>
>>
>> On 2020/11/5 下午3:01, Tyler Richmond wrote:
>>> Qu,
>>>
>>> I'm wondering, was a fix for this ever implemented?
>>
>> Already implemented the --repair ability in latest btrfs-progs.
>>
>>> I recently added a
>>> new drive to expand the array, and during the rebalance it dropped
>>> itself back to a read only filesystem. I suspect it's related to the
>>> issues discussed earlier in this thread. Is there anything I can do to
>>> complete the balance? The error that caused it to drop to read only is
>>> here: https://pastebin.com/GGYVMaiG
>>
>> Yep, the same cause.
>>
>> Thanks,
>> Qu
>>>
>>> Thanks!
>>>
>>>
>>> On Tue, Aug 25, 2020 at 9:43 AM Tyler Richmond
>>> <t.d.richmond@gmail.com> wrote:
>>>>
>>>> Great, glad we got somewhere! I'll look forward to the fix!
>>>>
>>>> On Tue, Aug 25, 2020 at 9:38 AM Qu Wenruo <quwenruo.btrfs@gmx.com>
>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 2020/8/25 下午9:30, Tyler Richmond wrote:
>>>>>> Qu,
>>>>>>
>>>>>> The dump of the block is:
>>>>>>
>>>>>> https://pastebin.com/ran85JJv
>>>>>>
>>>>>> I've also completed the btrfs-image, but it's almost 50gb. What's the
>>>>>> best way to get it to you? Also, does it work with -ss or are the
>>>>>> original filenames important?
>>>>>
>>>>> 50G is too big for me to even receive.
>>>>>
>>>>> But your dump shows the problem!
>>>>>
>>>>> It's not inode generation, but inode transid, which would affect send.
>>>>>
>>>>> This is not even checked in btrfs-progs, thus no wonder why it doesn't
>>>>> detect them.
>>>>>
>>>>> And copy-pasted kernel message shares the same "generation" word, not
>>>>> using proper transid to show the problem.
>>>>>
>>>>> Your dump really saved the day!
>>>>>
>>>>> The fix for kernel and btrfs-progs would come in next few days.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>>
>>>>>> Thanks again!
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 25, 2020 at 2:37 AM Qu Wenruo <quwenruo.btrfs@gmx.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 2020/8/25 下午1:25, Tyler Richmond wrote:
>>>>>>>> Qu,
>>>>>>>>
>>>>>>>> Yes, it's btrfs-progs 5.7. Here is the result of the lowmem check:
>>>>>>>>
>>>>>>>> https://pastebin.com/8Tzx23EX
>>>>>>>
>>>>>>> That doesn't detect any inode generation problem at all, which is
>>>>>>> not a
>>>>>>> good sign.
>>>>>>>
>>>>>>> Would you also pvode the dump for the offending block?
>>>>>>>
>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>
>>>>>>> For this case, would you please provide the tree dump of
>>>>>>> "203510940835840" ?
>>>>>>>
>>>>>>> # btrfs ins dump-tree -b 203510940835840 <device>
>>>>>>>
>>>>>>> And, since btrfs-image can't dump with regular extent tree, the "-w"
>>>>>>> dump would also help.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Qu
>>>>>>>
>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> On Mon, Aug 24, 2020 at 4:26 AM Qu Wenruo
>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2020/8/24 上午10:47, Tyler Richmond wrote:
>>>>>>>>>> Qu,
>>>>>>>>>>
>>>>>>>>>> Finally finished another repair and captured the output.
>>>>>>>>>>
>>>>>>>>>> https://pastebin.com/ffcbwvd8
>>>>>>>>>>
>>>>>>>>>> Does that show you what you need? Or should I still do one in
>>>>>>>>>> lowmem mode?
>>>>>>>>>
>>>>>>>>> Lowmem mode (no need for --repair) is recommended since
>>>>>>>>> original mode
>>>>>>>>> doesn't detect the inode generation problem.
>>>>>>>>>
>>>>>>>>> And it's already btrfs-progs v5.7 right?
>>>>>>>>>
>>>>>>>>> THanks,
>>>>>>>>> Qu
>>>>>>>>>>
>>>>>>>>>> Thanks for your help!
>>>>>>>>>>
>>>>>>>>>> On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo
>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 2020/8/23 上午10:49, Tyler Richmond wrote:
>>>>>>>>>>>> Well, I can guarantee that I didn't create this fs before
>>>>>>>>>>>> 2015 (just
>>>>>>>>>>>> checked the order confirmation from when I bought the
>>>>>>>>>>>> server), but I
>>>>>>>>>>>> may have just used whatever was in the Ubuntu package
>>>>>>>>>>>> manager at the
>>>>>>>>>>>> time. So maybe I don't have a v0 ref?
>>>>>>>>>>>
>>>>>>>>>>> Then btrfs-image shouldn't report that.
>>>>>>>>>>>
>>>>>>>>>>> There is an item smaller than any valid btrfs item, normally
>>>>>>>>>>> it means
>>>>>>>>>>> it's a v0 ref.
>>>>>>>>>>> If not, then it could be a bigger problem.
>>>>>>>>>>>
>>>>>>>>>>> Could you please provide the full btrfs-check output?
>>>>>>>>>>> Also, if possible result from "btrfs check --mode=lowmem"
>>>>>>>>>>> would also help.
>>>>>>>>>>>
>>>>>>>>>>> Also, if you really go "--repair", then the full output would
>>>>>>>>>>> also be
>>>>>>>>>>> needed to determine what's going wrong.
>>>>>>>>>>> There is a report about "btrfs check --repair" didn't repair
>>>>>>>>>>> the inode
>>>>>>>>>>> generation, if that's the case we must have a bug then.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Qu
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo
>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
>>>>>>>>>>>>>>> Is my best bet just to downgrade the kernel and then try
>>>>>>>>>>>>>>> to delete the
>>>>>>>>>>>>>>> broken files? Or should I rebuild from scratch? Just
>>>>>>>>>>>>>>> don't know
>>>>>>>>>>>>>>> whether it's worth the time to try and figure this out or
>>>>>>>>>>>>>>> if the
>>>>>>>>>>>>>>> problems stem from the FS being too old and it's beyond
>>>>>>>>>>>>>>> trying to
>>>>>>>>>>>>>>> repair.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> All invalid inode generations, should be able to be
>>>>>>>>>>>>>> repaired by latest
>>>>>>>>>>>>>> btrfs-check.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If not, please provide the btrfs-image dump for us to
>>>>>>>>>>>>>> determine what's
>>>>>>>>>>>>>> going wrong.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond
>>>>>>>>>>>>>>> <t.d.richmond@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I didn't check dmesg during the btrfs check, but that
>>>>>>>>>>>>>>>> was the only
>>>>>>>>>>>>>>>> output during the rm -f before it was forced readonly. I
>>>>>>>>>>>>>>>> just checked
>>>>>>>>>>>>>>>> dmesg for inode generation values, and there are a lot
>>>>>>>>>>>>>>>> of them.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> https://pastebin.com/stZdN0ta
>>>>>>>>>>>>>>>> The dmesg output had 990 lines containing inode generation.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> However, these were at least later. I tried to do a
>>>>>>>>>>>>>>>> btrfs balance
>>>>>>>>>>>>>>>> -mconvert raid1 and it failed with an I/O error. That is
>>>>>>>>>>>>>>>> probably what
>>>>>>>>>>>>>>>> generated these specific errors, but maybe they were
>>>>>>>>>>>>>>>> also happening
>>>>>>>>>>>>>>>> during the btrfs repair.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway with:
>>>>>>>>>>>>>>>> ERROR: either extent tree is corrupted or deprecated
>>>>>>>>>>>>>>>> extent ref format
>>>>>>>>>>>>>>>> ERROR: create failed: -5
>>>>>>>>>>>>>
>>>>>>>>>>>>> Oh, forgot this part.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This means you have v0 ref?!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Then the fs is too old, no progs/kernel support after all.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In that case, please rollback to the last working kernel
>>>>>>>>>>>>> and copy your data.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In fact, that v0 ref should only be in the code base for
>>>>>>>>>>>>> several weeks
>>>>>>>>>>>>> before 2010, thus it's really too old.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The good news is, with tree-checker, we should never
>>>>>>>>>>>>> experience such
>>>>>>>>>>>>> too-old-to-be-usable problem (at least I hope so)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo
>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sorry to resurrect this thread, but I just ran into
>>>>>>>>>>>>>>>>>> something that I
>>>>>>>>>>>>>>>>>> can't really just ignore. I've found a folder that is
>>>>>>>>>>>>>>>>>> full of files
>>>>>>>>>>>>>>>>>> which I guess have been broken somehow. I found a
>>>>>>>>>>>>>>>>>> backup and restored
>>>>>>>>>>>>>>>>>> them, but I want to delete this folder of broken
>>>>>>>>>>>>>>>>>> files. But whenever I
>>>>>>>>>>>>>>>>>> try, the fs is forced into readonly mode again. I just
>>>>>>>>>>>>>>>>>> finished another
>>>>>>>>>>>>>>>>>> btrfs check --repair but it didn't fix the problem.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> https://pastebin.com/eTV3s3fr
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is that the full output?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> No inode generation bugs?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'm already on btrfs-progs v5.7. Any new suggestions?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Strange.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The detection and repair should have been merged into
>>>>>>>>>>>>>>>>> v5.5.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If your fs is small enough, would you please provide
>>>>>>>>>>>>>>>>> the "btrfs-image
>>>>>>>>>>>>>>>>> -c9" dump?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It would contain the filenames and directories names,
>>>>>>>>>>>>>>>>> but doesn't
>>>>>>>>>>>>>>>>> contain file contents.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond
>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 5.6.1 also failed the same way. Here's the usage
>>>>>>>>>>>>>>>>>> output. This is the
>>>>>>>>>>>>>>>>>> part where you see I've been using RAID5 haha
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> WARNING: RAID56 detected, not implemented
>>>>>>>>>>>>>>>>>> Overall:
>>>>>>>>>>>>>>>>>> Device size: 60.03TiB
>>>>>>>>>>>>>>>>>> Device allocated: 98.06GiB
>>>>>>>>>>>>>>>>>> Device unallocated: 59.93TiB
>>>>>>>>>>>>>>>>>> Device missing: 0.00B
>>>>>>>>>>>>>>>>>> Used: 92.56GiB
>>>>>>>>>>>>>>>>>> Free (estimated): 0.00B
>>>>>>>>>>>>>>>>>> (min: 8.00EiB)
>>>>>>>>>>>>>>>>>> Data ratio: 0.00
>>>>>>>>>>>>>>>>>> Metadata ratio: 2.00
>>>>>>>>>>>>>>>>>> Global reserve: 512.00MiB
>>>>>>>>>>>>>>>>>> (used: 0.00B)
>>>>>>>>>>>>>>>>>> Multiple profiles: no
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
>>>>>>>>>>>>>>>>>> /dev/sdh 8.07TiB
>>>>>>>>>>>>>>>>>> /dev/sdf 8.07TiB
>>>>>>>>>>>>>>>>>> /dev/sdg 8.07TiB
>>>>>>>>>>>>>>>>>> /dev/sdd 8.07TiB
>>>>>>>>>>>>>>>>>> /dev/sdc 8.07TiB
>>>>>>>>>>>>>>>>>> /dev/sde 8.07TiB
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB
>>>>>>>>>>>>>>>>>> (94.44%)
>>>>>>>>>>>>>>>>>> /dev/sdh 34.00GiB
>>>>>>>>>>>>>>>>>> /dev/sdf 32.00GiB
>>>>>>>>>>>>>>>>>> /dev/sdg 32.00GiB
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
>>>>>>>>>>>>>>>>>> /dev/sdf 32.00MiB
>>>>>>>>>>>>>>>>>> /dev/sdg 32.00MiB
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Unallocated:
>>>>>>>>>>>>>>>>>> /dev/sdh 2.81TiB
>>>>>>>>>>>>>>>>>> /dev/sdf 2.81TiB
>>>>>>>>>>>>>>>>>> /dev/sdg 2.81TiB
>>>>>>>>>>>>>>>>>> /dev/sdd 1.03TiB
>>>>>>>>>>>>>>>>>> /dev/sdc 1.03TiB
>>>>>>>>>>>>>>>>>> /dev/sde 1.03TiB
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo
>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>> > > If this is saying there's no extra space for
>>>>>>>>>>>>>>>>>> metadata, is that why
>>>>>>>>>>>>>>>>>> > > adding more files often makes the system hang
>>>>>>>>>>>>>>>>>> for 30-90s? Is there
>>>>>>>>>>>>>>>>>> > > anything I should do about that?
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > I'm not sure about the hang though.
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > It would be nice to give more info to diagnosis.
>>>>>>>>>>>>>>>>>> > The output of 'btrfs fi usage' is useful for
>>>>>>>>>>>>>>>>>> space usage problem.
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > But the common idea is, to keep at 1~2 Gi
>>>>>>>>>>>>>>>>>> unallocated (not avaiable
>>>>>>>>>>>>>>>>>> > space in vanilla df command) space for btrfs.
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>> > Qu
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> > > Thank you so much for all of your help. I
>>>>>>>>>>>>>>>>>> love how flexible BTRFS is
>>>>>>>>>>>>>>>>>> > > but when things go wrong it's very hard for
>>>>>>>>>>>>>>>>>> me to troubleshoot.
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo
>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>> > >>> Something went wrong:
>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>> > >>> Reinitialize checksum tree
>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1`
>>>>>>>>>>>>>>>>>> triggered, value 1
>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
>>>>>>>>>>>>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
>>>>>>>>>>>>>>>>>> > >>> Aborted
>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>> > >> This means no space for extra metadata...
>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>> > >> Anyway the csum tree problem shouldn't be a
>>>>>>>>>>>>>>>>>> big thing, you
>>>>>>>>>>>>>>>>>> could leave
>>>>>>>>>>>>>>>>>> > >> it and call it a day.
>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>> > >> BTW, as long as btrfs check reports no extra
>>>>>>>>>>>>>>>>>> problem for the inode
>>>>>>>>>>>>>>>>>> > >> generation, it should be pretty safe to use
>>>>>>>>>>>>>>>>>> the fs.
>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>> > >> Thanks,
>>>>>>>>>>>>>>>>>> > >> Qu
>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>> > >>> I just noticed I have btrfs-progs 5.6
>>>>>>>>>>>>>>>>>> installed and 5.6.1 is
>>>>>>>>>>>>>>>>>> > >>> available. I'll let that try overnight?
>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>> > >>>>> Thank you for helping. The end result of
>>>>>>>>>>>>>>>>>> the scan was:
>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>> > >>>>> [1/7] checking root items
>>>>>>>>>>>>>>>>>> > >>>>> [2/7] checking extents
>>>>>>>>>>>>>>>>>> > >>>>> [3/7] checking free space cache
>>>>>>>>>>>>>>>>>> > >>>>> [4/7] checking fs roots
>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>> > >>>> Good news is, your fs is still mostly fine.
>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>> > >>>>> [5/7] checking only csums items (without
>>>>>>>>>>>>>>>>>> verifying data)
>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range 0-69632
>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 0-69632 but there is no
>>>>>>>>>>>>>>>>>> extent record
>>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>> 946692096-946827264
>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 946692096-946827264 but
>>>>>>>>>>>>>>>>>> there is no extent
>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>> 946831360-947912704
>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 946831360-947912704 but
>>>>>>>>>>>>>>>>>> there is no extent
>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>> > >>>>> ERROR: errors found in csum tree
>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>> > >>>> Only extent tree is corrupted.
>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>> > >>>> Normally btrfs check --init-csum-tree
>>>>>>>>>>>>>>>>>> should be able to
>>>>>>>>>>>>>>>>>> handle it.
>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>> > >>>> But still, please be sure you're using the
>>>>>>>>>>>>>>>>>> latest btrfs-progs
>>>>>>>>>>>>>>>>>> to fix it.
>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>> > >>>> Thanks,
>>>>>>>>>>>>>>>>>> > >>>> Qu
>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>> > >>>>> [6/7] checking root refs
>>>>>>>>>>>>>>>>>> > >>>>> [7/7] checking quota groups skipped (not
>>>>>>>>>>>>>>>>>> enabled on this FS)
>>>>>>>>>>>>>>>>>> > >>>>> found 44157956026368 bytes used, error(s)
>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>> > >>>>> total csum bytes: 42038602716
>>>>>>>>>>>>>>>>>> > >>>>> total tree bytes: 49688616960
>>>>>>>>>>>>>>>>>> > >>>>> total fs tree bytes: 1256427520
>>>>>>>>>>>>>>>>>> > >>>>> total extent tree bytes: 1709105152
>>>>>>>>>>>>>>>>>> > >>>>> btree space waste bytes: 3172727316
>>>>>>>>>>>>>>>>>> > >>>>> file data blocks allocated: 261625653436416
>>>>>>>>>>>>>>>>>> > >>>>> referenced 47477768499200
>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>> > >>>>> What do I need to do to fix all of this?
>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>> > >>>>>>> Well, the repair doesn't look terribly
>>>>>>>>>>>>>>>>>> successful.
>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>> This means there are more problems, not
>>>>>>>>>>>>>>>>>> only the hash name
>>>>>>>>>>>>>>>>>> mismatch.
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>> This means the fs is already corrupted,
>>>>>>>>>>>>>>>>>> the name hash is
>>>>>>>>>>>>>>>>>> just one
>>>>>>>>>>>>>>>>>> > >>>>>> unrelated symptom.
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>> The only good news is, btrfs-progs abort
>>>>>>>>>>>>>>>>>> the transaction,
>>>>>>>>>>>>>>>>>> thus no
>>>>>>>>>>>>>>>>>> > >>>>>> further damage to the fs.
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>> Please run a plain btrfs-check to show
>>>>>>>>>>>>>>>>>> what's the problem
>>>>>>>>>>>>>>>>>> first.
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>>>>>> > >>>>>> Qu
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: failed to zero log tree: -17
>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: attempt to start transaction
>>>>>>>>>>>>>>>>>> over already running one
>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: reserved space leaked,
>>>>>>>>>>>>>>>>>> flag=0x4 bytes_reserved=4096
>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans):
>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans):
>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans):
>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans):
>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>> What is going on?
>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler
>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>>> Chris, I had used the correct
>>>>>>>>>>>>>>>>>> mountpoint in the command.
>>>>>>>>>>>>>>>>>> I just edited
>>>>>>>>>>>>>>>>>> > >>>>>>>> it in the email to be /mountpoint for
>>>>>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> I looked up this error and it
>>>>>>>>>>>>>>>>>> basically says ask a
>>>>>>>>>>>>>>>>>> developer to
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> determine if it's a false error or
>>>>>>>>>>>>>>>>>> not. I just started
>>>>>>>>>>>>>>>>>> getting some
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> slow response times, and looked at
>>>>>>>>>>>>>>>>>> the dmesg log to
>>>>>>>>>>>>>>>>>> find a ton of
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> these errors.
>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical
>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0,
>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error (device
>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical
>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0,
>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error (device
>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical
>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0,
>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error (device
>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> btrfs device stats, however, doesn't
>>>>>>>>>>>>>>>>>> show any errors.
>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Is there anything I should do about
>>>>>>>>>>>>>>>>>> this, or should I
>>>>>>>>>>>>>>>>>> just continue
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> using my array as normal?
>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>>>> This is caused by older kernel
>>>>>>>>>>>>>>>>>> underflow inode generation.
>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it, using
>>>>>>>>>>>>>>>>>> btrfs check --repair.
>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>>>> Or you can go safer, by manually
>>>>>>>>>>>>>>>>>> locating the inode
>>>>>>>>>>>>>>>>>> using its inode
>>>>>>>>>>>>>>>>>> > >>>>>>>>> number (1311670), and copy it to some
>>>>>>>>>>>>>>>>>> new location using
>>>>>>>>>>>>>>>>>> previous
>>>>>>>>>>>>>>>>>> > >>>>>>>>> working kernel, then delete the old
>>>>>>>>>>>>>>>>>> file, copy the new
>>>>>>>>>>>>>>>>>> one back to fix it.
>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> > >>>>>>>>> Qu
>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-05 23:00 ` Qu Wenruo
@ 2020-11-05 23:12 ` Ferry Toth
2020-11-05 23:32 ` Qu Wenruo
0 siblings, 1 reply; 56+ messages in thread
From: Ferry Toth @ 2020-11-05 23:12 UTC (permalink / raw)
To: Qu Wenruo, Tyler Richmond; +Cc: Btrfs BTRFS
Hi,
Op 06-11-2020 om 00:00 schreef Qu Wenruo:
>
> On 2020/11/6 上午4:08, Ferry Toth wrote:
>> I am in a similar spot, during updating my distro (Kubuntu), I am unable
>> to update a certain package. I know which file it is:
>>
>> ~$ ls -l /usr/share/doc/libatk1.0-data
>> ls: kan geen toegang krijgen tot '/usr/share/doc/libatk1.0-data':
>> Invoer-/uitvoerfout
>>
>> This creates the following in journal:
>>
>> kernel: BTRFS critical (device sda2): corrupt leaf: root=294
>> block=1169152675840 slot=1 ino=915987, invalid inode generation: has
>> 18446744073709551492 expect [0, 5851353]
>> kernel: BTRFS error (device sda2): block=1169152675840 read time tree
>> block corruption detected
>>
>> Now, the problem: this file is on my rootfs, which is mounted. apt
>> (distribution updated) installed all packages but can't continue
>> configuring, because libatk is a dependancy. I can't delete the file
>> because of the I/O error. And btrfs check complains (I tried running RO)
>> because the file system is mounted.
>>
>> But, on the sunny side, the file system is not RO.
>>
>> Is there any way to forcefully remove the file? Or do you have a
>> recommendation how to proceed?
> Newer kernel will reject to even read the item, thus will not be able to
> remove it.
That's already the case. (input / output error)
> I guess you have to use some distro ISO to fix the fs.
And then? btrfs check --repair the disk offline?
> Thanks,
> Qu
>
>> Linux = 5.6.0-1032-oem
>>
>> Thanks,
>> Ferry
>>
>> Op 05-11-2020 om 08:19 schreef Qu Wenruo:
>>>
>>> On 2020/11/5 下午3:01, Tyler Richmond wrote:
>>>> Qu,
>>>>
>>>> I'm wondering, was a fix for this ever implemented?
>>> Already implemented the --repair ability in latest btrfs-progs.
>>>
>>>> I recently added a
>>>> new drive to expand the array, and during the rebalance it dropped
>>>> itself back to a read only filesystem. I suspect it's related to the
>>>> issues discussed earlier in this thread. Is there anything I can do to
>>>> complete the balance? The error that caused it to drop to read only is
>>>> here: https://pastebin.com/GGYVMaiG
>>> Yep, the same cause.
>>>
>>> Thanks,
>>> Qu
>>>> Thanks!
>>>>
>>>>
>>>> On Tue, Aug 25, 2020 at 9:43 AM Tyler Richmond
>>>> <t.d.richmond@gmail.com> wrote:
>>>>> Great, glad we got somewhere! I'll look forward to the fix!
>>>>>
>>>>> On Tue, Aug 25, 2020 at 9:38 AM Qu Wenruo <quwenruo.btrfs@gmx.com>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> On 2020/8/25 下午9:30, Tyler Richmond wrote:
>>>>>>> Qu,
>>>>>>>
>>>>>>> The dump of the block is:
>>>>>>>
>>>>>>> https://pastebin.com/ran85JJv
>>>>>>>
>>>>>>> I've also completed the btrfs-image, but it's almost 50gb. What's the
>>>>>>> best way to get it to you? Also, does it work with -ss or are the
>>>>>>> original filenames important?
>>>>>> 50G is too big for me to even receive.
>>>>>>
>>>>>> But your dump shows the problem!
>>>>>>
>>>>>> It's not inode generation, but inode transid, which would affect send.
>>>>>>
>>>>>> This is not even checked in btrfs-progs, thus no wonder why it doesn't
>>>>>> detect them.
>>>>>>
>>>>>> And copy-pasted kernel message shares the same "generation" word, not
>>>>>> using proper transid to show the problem.
>>>>>>
>>>>>> Your dump really saved the day!
>>>>>>
>>>>>> The fix for kernel and btrfs-progs would come in next few days.
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>> Thanks again!
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 25, 2020 at 2:37 AM Qu Wenruo <quwenruo.btrfs@gmx.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2020/8/25 下午1:25, Tyler Richmond wrote:
>>>>>>>>> Qu,
>>>>>>>>>
>>>>>>>>> Yes, it's btrfs-progs 5.7. Here is the result of the lowmem check:
>>>>>>>>>
>>>>>>>>> https://pastebin.com/8Tzx23EX
>>>>>>>> That doesn't detect any inode generation problem at all, which is
>>>>>>>> not a
>>>>>>>> good sign.
>>>>>>>>
>>>>>>>> Would you also pvode the dump for the offending block?
>>>>>>>>
>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>
>>>>>>>> For this case, would you please provide the tree dump of
>>>>>>>> "203510940835840" ?
>>>>>>>>
>>>>>>>> # btrfs ins dump-tree -b 203510940835840 <device>
>>>>>>>>
>>>>>>>> And, since btrfs-image can't dump with regular extent tree, the "-w"
>>>>>>>> dump would also help.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Qu
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> On Mon, Aug 24, 2020 at 4:26 AM Qu Wenruo
>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 2020/8/24 上午10:47, Tyler Richmond wrote:
>>>>>>>>>>> Qu,
>>>>>>>>>>>
>>>>>>>>>>> Finally finished another repair and captured the output.
>>>>>>>>>>>
>>>>>>>>>>> https://pastebin.com/ffcbwvd8
>>>>>>>>>>>
>>>>>>>>>>> Does that show you what you need? Or should I still do one in
>>>>>>>>>>> lowmem mode?
>>>>>>>>>> Lowmem mode (no need for --repair) is recommended since
>>>>>>>>>> original mode
>>>>>>>>>> doesn't detect the inode generation problem.
>>>>>>>>>>
>>>>>>>>>> And it's already btrfs-progs v5.7 right?
>>>>>>>>>>
>>>>>>>>>> THanks,
>>>>>>>>>> Qu
>>>>>>>>>>> Thanks for your help!
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo
>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 2020/8/23 上午10:49, Tyler Richmond wrote:
>>>>>>>>>>>>> Well, I can guarantee that I didn't create this fs before
>>>>>>>>>>>>> 2015 (just
>>>>>>>>>>>>> checked the order confirmation from when I bought the
>>>>>>>>>>>>> server), but I
>>>>>>>>>>>>> may have just used whatever was in the Ubuntu package
>>>>>>>>>>>>> manager at the
>>>>>>>>>>>>> time. So maybe I don't have a v0 ref?
>>>>>>>>>>>> Then btrfs-image shouldn't report that.
>>>>>>>>>>>>
>>>>>>>>>>>> There is an item smaller than any valid btrfs item, normally
>>>>>>>>>>>> it means
>>>>>>>>>>>> it's a v0 ref.
>>>>>>>>>>>> If not, then it could be a bigger problem.
>>>>>>>>>>>>
>>>>>>>>>>>> Could you please provide the full btrfs-check output?
>>>>>>>>>>>> Also, if possible result from "btrfs check --mode=lowmem"
>>>>>>>>>>>> would also help.
>>>>>>>>>>>>
>>>>>>>>>>>> Also, if you really go "--repair", then the full output would
>>>>>>>>>>>> also be
>>>>>>>>>>>> needed to determine what's going wrong.
>>>>>>>>>>>> There is a report about "btrfs check --repair" didn't repair
>>>>>>>>>>>> the inode
>>>>>>>>>>>> generation, if that's the case we must have a bug then.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Qu
>>>>>>>>>>>>> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo
>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
>>>>>>>>>>>>>>>> Is my best bet just to downgrade the kernel and then try
>>>>>>>>>>>>>>>> to delete the
>>>>>>>>>>>>>>>> broken files? Or should I rebuild from scratch? Just
>>>>>>>>>>>>>>>> don't know
>>>>>>>>>>>>>>>> whether it's worth the time to try and figure this out or
>>>>>>>>>>>>>>>> if the
>>>>>>>>>>>>>>>> problems stem from the FS being too old and it's beyond
>>>>>>>>>>>>>>>> trying to
>>>>>>>>>>>>>>>> repair.
>>>>>>>>>>>>>>> All invalid inode generations, should be able to be
>>>>>>>>>>>>>>> repaired by latest
>>>>>>>>>>>>>>> btrfs-check.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If not, please provide the btrfs-image dump for us to
>>>>>>>>>>>>>>> determine what's
>>>>>>>>>>>>>>> going wrong.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond
>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com> wrote:
>>>>>>>>>>>>>>>>> I didn't check dmesg during the btrfs check, but that
>>>>>>>>>>>>>>>>> was the only
>>>>>>>>>>>>>>>>> output during the rm -f before it was forced readonly. I
>>>>>>>>>>>>>>>>> just checked
>>>>>>>>>>>>>>>>> dmesg for inode generation values, and there are a lot
>>>>>>>>>>>>>>>>> of them.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> https://pastebin.com/stZdN0ta
>>>>>>>>>>>>>>>>> The dmesg output had 990 lines containing inode generation.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> However, these were at least later. I tried to do a
>>>>>>>>>>>>>>>>> btrfs balance
>>>>>>>>>>>>>>>>> -mconvert raid1 and it failed with an I/O error. That is
>>>>>>>>>>>>>>>>> probably what
>>>>>>>>>>>>>>>>> generated these specific errors, but maybe they were
>>>>>>>>>>>>>>>>> also happening
>>>>>>>>>>>>>>>>> during the btrfs repair.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway with:
>>>>>>>>>>>>>>>>> ERROR: either extent tree is corrupted or deprecated
>>>>>>>>>>>>>>>>> extent ref format
>>>>>>>>>>>>>>>>> ERROR: create failed: -5
>>>>>>>>>>>>>> Oh, forgot this part.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This means you have v0 ref?!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Then the fs is too old, no progs/kernel support after all.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In that case, please rollback to the last working kernel
>>>>>>>>>>>>>> and copy your data.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In fact, that v0 ref should only be in the code base for
>>>>>>>>>>>>>> several weeks
>>>>>>>>>>>>>> before 2010, thus it's really too old.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The good news is, with tree-checker, we should never
>>>>>>>>>>>>>> experience such
>>>>>>>>>>>>>> too-old-to-be-usable problem (at least I hope so)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo
>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Sorry to resurrect this thread, but I just ran into
>>>>>>>>>>>>>>>>>>> something that I
>>>>>>>>>>>>>>>>>>> can't really just ignore. I've found a folder that is
>>>>>>>>>>>>>>>>>>> full of files
>>>>>>>>>>>>>>>>>>> which I guess have been broken somehow. I found a
>>>>>>>>>>>>>>>>>>> backup and restored
>>>>>>>>>>>>>>>>>>> them, but I want to delete this folder of broken
>>>>>>>>>>>>>>>>>>> files. But whenever I
>>>>>>>>>>>>>>>>>>> try, the fs is forced into readonly mode again. I just
>>>>>>>>>>>>>>>>>>> finished another
>>>>>>>>>>>>>>>>>>> btrfs check --repair but it didn't fix the problem.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> https://pastebin.com/eTV3s3fr
>>>>>>>>>>>>>>>>>> Is that the full output?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> No inode generation bugs?
>>>>>>>>>>>>>>>>>>> I'm already on btrfs-progs v5.7. Any new suggestions?
>>>>>>>>>>>>>>>>>> Strange.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The detection and repair should have been merged into
>>>>>>>>>>>>>>>>>> v5.5.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> If your fs is small enough, would you please provide
>>>>>>>>>>>>>>>>>> the "btrfs-image
>>>>>>>>>>>>>>>>>> -c9" dump?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It would contain the filenames and directories names,
>>>>>>>>>>>>>>>>>> but doesn't
>>>>>>>>>>>>>>>>>> contain file contents.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond
>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 5.6.1 also failed the same way. Here's the usage
>>>>>>>>>>>>>>>>>>> output. This is the
>>>>>>>>>>>>>>>>>>> part where you see I've been using RAID5 haha
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> WARNING: RAID56 detected, not implemented
>>>>>>>>>>>>>>>>>>> Overall:
>>>>>>>>>>>>>>>>>>> Device size: 60.03TiB
>>>>>>>>>>>>>>>>>>> Device allocated: 98.06GiB
>>>>>>>>>>>>>>>>>>> Device unallocated: 59.93TiB
>>>>>>>>>>>>>>>>>>> Device missing: 0.00B
>>>>>>>>>>>>>>>>>>> Used: 92.56GiB
>>>>>>>>>>>>>>>>>>> Free (estimated): 0.00B
>>>>>>>>>>>>>>>>>>> (min: 8.00EiB)
>>>>>>>>>>>>>>>>>>> Data ratio: 0.00
>>>>>>>>>>>>>>>>>>> Metadata ratio: 2.00
>>>>>>>>>>>>>>>>>>> Global reserve: 512.00MiB
>>>>>>>>>>>>>>>>>>> (used: 0.00B)
>>>>>>>>>>>>>>>>>>> Multiple profiles: no
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
>>>>>>>>>>>>>>>>>>> /dev/sdh 8.07TiB
>>>>>>>>>>>>>>>>>>> /dev/sdf 8.07TiB
>>>>>>>>>>>>>>>>>>> /dev/sdg 8.07TiB
>>>>>>>>>>>>>>>>>>> /dev/sdd 8.07TiB
>>>>>>>>>>>>>>>>>>> /dev/sdc 8.07TiB
>>>>>>>>>>>>>>>>>>> /dev/sde 8.07TiB
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB
>>>>>>>>>>>>>>>>>>> (94.44%)
>>>>>>>>>>>>>>>>>>> /dev/sdh 34.00GiB
>>>>>>>>>>>>>>>>>>> /dev/sdf 32.00GiB
>>>>>>>>>>>>>>>>>>> /dev/sdg 32.00GiB
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
>>>>>>>>>>>>>>>>>>> /dev/sdf 32.00MiB
>>>>>>>>>>>>>>>>>>> /dev/sdg 32.00MiB
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Unallocated:
>>>>>>>>>>>>>>>>>>> /dev/sdh 2.81TiB
>>>>>>>>>>>>>>>>>>> /dev/sdf 2.81TiB
>>>>>>>>>>>>>>>>>>> /dev/sdg 2.81TiB
>>>>>>>>>>>>>>>>>>> /dev/sdd 1.03TiB
>>>>>>>>>>>>>>>>>>> /dev/sdc 1.03TiB
>>>>>>>>>>>>>>>>>>> /dev/sde 1.03TiB
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>> > > If this is saying there's no extra space for
>>>>>>>>>>>>>>>>>>> metadata, is that why
>>>>>>>>>>>>>>>>>>> > > adding more files often makes the system hang
>>>>>>>>>>>>>>>>>>> for 30-90s? Is there
>>>>>>>>>>>>>>>>>>> > > anything I should do about that?
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > I'm not sure about the hang though.
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > It would be nice to give more info to diagnosis.
>>>>>>>>>>>>>>>>>>> > The output of 'btrfs fi usage' is useful for
>>>>>>>>>>>>>>>>>>> space usage problem.
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > But the common idea is, to keep at 1~2 Gi
>>>>>>>>>>>>>>>>>>> unallocated (not avaiable
>>>>>>>>>>>>>>>>>>> > space in vanilla df command) space for btrfs.
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>> > Qu
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>> > > Thank you so much for all of your help. I
>>>>>>>>>>>>>>>>>>> love how flexible BTRFS is
>>>>>>>>>>>>>>>>>>> > > but when things go wrong it's very hard for
>>>>>>>>>>>>>>>>>>> me to troubleshoot.
>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>> > >>> Something went wrong:
>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>> > >>> Reinitialize checksum tree
>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1`
>>>>>>>>>>>>>>>>>>> triggered, value 1
>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
>>>>>>>>>>>>>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
>>>>>>>>>>>>>>>>>>> > >>> Aborted
>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>> > >> This means no space for extra metadata...
>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>> > >> Anyway the csum tree problem shouldn't be a
>>>>>>>>>>>>>>>>>>> big thing, you
>>>>>>>>>>>>>>>>>>> could leave
>>>>>>>>>>>>>>>>>>> > >> it and call it a day.
>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>> > >> BTW, as long as btrfs check reports no extra
>>>>>>>>>>>>>>>>>>> problem for the inode
>>>>>>>>>>>>>>>>>>> > >> generation, it should be pretty safe to use
>>>>>>>>>>>>>>>>>>> the fs.
>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>> > >> Thanks,
>>>>>>>>>>>>>>>>>>> > >> Qu
>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>> > >>> I just noticed I have btrfs-progs 5.6
>>>>>>>>>>>>>>>>>>> installed and 5.6.1 is
>>>>>>>>>>>>>>>>>>> > >>> available. I'll let that try overnight?
>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>> > >>>>> Thank you for helping. The end result of
>>>>>>>>>>>>>>>>>>> the scan was:
>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>> > >>>>> [1/7] checking root items
>>>>>>>>>>>>>>>>>>> > >>>>> [2/7] checking extents
>>>>>>>>>>>>>>>>>>> > >>>>> [3/7] checking free space cache
>>>>>>>>>>>>>>>>>>> > >>>>> [4/7] checking fs roots
>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>> > >>>> Good news is, your fs is still mostly fine.
>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>> > >>>>> [5/7] checking only csums items (without
>>>>>>>>>>>>>>>>>>> verifying data)
>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range 0-69632
>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 0-69632 but there is no
>>>>>>>>>>>>>>>>>>> extent record
>>>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>> 946692096-946827264
>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 946692096-946827264 but
>>>>>>>>>>>>>>>>>>> there is no extent
>>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>> 946831360-947912704
>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 946831360-947912704 but
>>>>>>>>>>>>>>>>>>> there is no extent
>>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>>> > >>>>> ERROR: errors found in csum tree
>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>> > >>>> Only extent tree is corrupted.
>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>> > >>>> Normally btrfs check --init-csum-tree
>>>>>>>>>>>>>>>>>>> should be able to
>>>>>>>>>>>>>>>>>>> handle it.
>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>> > >>>> But still, please be sure you're using the
>>>>>>>>>>>>>>>>>>> latest btrfs-progs
>>>>>>>>>>>>>>>>>>> to fix it.
>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>> > >>>> Thanks,
>>>>>>>>>>>>>>>>>>> > >>>> Qu
>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>> > >>>>> [6/7] checking root refs
>>>>>>>>>>>>>>>>>>> > >>>>> [7/7] checking quota groups skipped (not
>>>>>>>>>>>>>>>>>>> enabled on this FS)
>>>>>>>>>>>>>>>>>>> > >>>>> found 44157956026368 bytes used, error(s)
>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>> > >>>>> total csum bytes: 42038602716
>>>>>>>>>>>>>>>>>>> > >>>>> total tree bytes: 49688616960
>>>>>>>>>>>>>>>>>>> > >>>>> total fs tree bytes: 1256427520
>>>>>>>>>>>>>>>>>>> > >>>>> total extent tree bytes: 1709105152
>>>>>>>>>>>>>>>>>>> > >>>>> btree space waste bytes: 3172727316
>>>>>>>>>>>>>>>>>>> > >>>>> file data blocks allocated: 261625653436416
>>>>>>>>>>>>>>>>>>> > >>>>> referenced 47477768499200
>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>> > >>>>> What do I need to do to fix all of this?
>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>> > >>>>>>> Well, the repair doesn't look terribly
>>>>>>>>>>>>>>>>>>> successful.
>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>> This means there are more problems, not
>>>>>>>>>>>>>>>>>>> only the hash name
>>>>>>>>>>>>>>>>>>> mismatch.
>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>> This means the fs is already corrupted,
>>>>>>>>>>>>>>>>>>> the name hash is
>>>>>>>>>>>>>>>>>>> just one
>>>>>>>>>>>>>>>>>>> > >>>>>> unrelated symptom.
>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>> The only good news is, btrfs-progs abort
>>>>>>>>>>>>>>>>>>> the transaction,
>>>>>>>>>>>>>>>>>>> thus no
>>>>>>>>>>>>>>>>>>> > >>>>>> further damage to the fs.
>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>> Please run a plain btrfs-check to show
>>>>>>>>>>>>>>>>>>> what's the problem
>>>>>>>>>>>>>>>>>>> first.
>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> > >>>>>> Qu
>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: failed to zero log tree: -17
>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: attempt to start transaction
>>>>>>>>>>>>>>>>>>> over already running one
>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: reserved space leaked,
>>>>>>>>>>>>>>>>>>> flag=0x4 bytes_reserved=4096
>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans):
>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans):
>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans):
>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans):
>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>> What is going on?
>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler
>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>>> Chris, I had used the correct
>>>>>>>>>>>>>>>>>>> mountpoint in the command.
>>>>>>>>>>>>>>>>>>> I just edited
>>>>>>>>>>>>>>>>>>> > >>>>>>>> it in the email to be /mountpoint for
>>>>>>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers crossed!
>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo
>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> I looked up this error and it
>>>>>>>>>>>>>>>>>>> basically says ask a
>>>>>>>>>>>>>>>>>>> developer to
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> determine if it's a false error or
>>>>>>>>>>>>>>>>>>> not. I just started
>>>>>>>>>>>>>>>>>>> getting some
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> slow response times, and looked at
>>>>>>>>>>>>>>>>>>> the dmesg log to
>>>>>>>>>>>>>>>>>>> find a ton of
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> these errors.
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical
>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0,
>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error (device
>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical
>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0,
>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error (device
>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical
>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0,
>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error (device
>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> btrfs device stats, however, doesn't
>>>>>>>>>>>>>>>>>>> show any errors.
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Is there anything I should do about
>>>>>>>>>>>>>>>>>>> this, or should I
>>>>>>>>>>>>>>>>>>> just continue
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> using my array as normal?
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>>>> This is caused by older kernel
>>>>>>>>>>>>>>>>>>> underflow inode generation.
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it, using
>>>>>>>>>>>>>>>>>>> btrfs check --repair.
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Or you can go safer, by manually
>>>>>>>>>>>>>>>>>>> locating the inode
>>>>>>>>>>>>>>>>>>> using its inode
>>>>>>>>>>>>>>>>>>> > >>>>>>>>> number (1311670), and copy it to some
>>>>>>>>>>>>>>>>>>> new location using
>>>>>>>>>>>>>>>>>>> previous
>>>>>>>>>>>>>>>>>>> > >>>>>>>>> working kernel, then delete the old
>>>>>>>>>>>>>>>>>>> file, copy the new
>>>>>>>>>>>>>>>>>>> one back to fix it.
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-05 23:12 ` Ferry Toth
@ 2020-11-05 23:32 ` Qu Wenruo
2020-11-05 23:37 ` Ferry Toth
0 siblings, 1 reply; 56+ messages in thread
From: Qu Wenruo @ 2020-11-05 23:32 UTC (permalink / raw)
To: Ferry Toth, Tyler Richmond; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 40986 bytes --]
On 2020/11/6 上午7:12, Ferry Toth wrote:
> Hi,
>
> Op 06-11-2020 om 00:00 schreef Qu Wenruo:
>>
>> On 2020/11/6 上午4:08, Ferry Toth wrote:
>>> I am in a similar spot, during updating my distro (Kubuntu), I am unable
>>> to update a certain package. I know which file it is:
>>>
>>> ~$ ls -l /usr/share/doc/libatk1.0-data
>>> ls: kan geen toegang krijgen tot '/usr/share/doc/libatk1.0-data':
>>> Invoer-/uitvoerfout
>>>
>>> This creates the following in journal:
>>>
>>> kernel: BTRFS critical (device sda2): corrupt leaf: root=294
>>> block=1169152675840 slot=1 ino=915987, invalid inode generation: has
>>> 18446744073709551492 expect [0, 5851353]
>>> kernel: BTRFS error (device sda2): block=1169152675840 read time tree
>>> block corruption detected
>>>
>>> Now, the problem: this file is on my rootfs, which is mounted. apt
>>> (distribution updated) installed all packages but can't continue
>>> configuring, because libatk is a dependancy. I can't delete the file
>>> because of the I/O error. And btrfs check complains (I tried running RO)
>>> because the file system is mounted.
>>>
>>> But, on the sunny side, the file system is not RO.
>>>
>>> Is there any way to forcefully remove the file? Or do you have a
>>> recommendation how to proceed?
>> Newer kernel will reject to even read the item, thus will not be able to
>> remove it.
> That's already the case. (input / output error)
>> I guess you have to use some distro ISO to fix the fs.
> And then? btrfs check --repair the disk offline?
Yep.
You would want the latest btrfs-progs though.
THanks,
Qu
>> Thanks,
>> Qu
>>
>>> Linux = 5.6.0-1032-oem
>>>
>>> Thanks,
>>> Ferry
>>>
>>> Op 05-11-2020 om 08:19 schreef Qu Wenruo:
>>>>
>>>> On 2020/11/5 下午3:01, Tyler Richmond wrote:
>>>>> Qu,
>>>>>
>>>>> I'm wondering, was a fix for this ever implemented?
>>>> Already implemented the --repair ability in latest btrfs-progs.
>>>>
>>>>> I recently added a
>>>>> new drive to expand the array, and during the rebalance it dropped
>>>>> itself back to a read only filesystem. I suspect it's related to the
>>>>> issues discussed earlier in this thread. Is there anything I can do to
>>>>> complete the balance? The error that caused it to drop to read only is
>>>>> here: https://pastebin.com/GGYVMaiG
>>>> Yep, the same cause.
>>>>
>>>> Thanks,
>>>> Qu
>>>>> Thanks!
>>>>>
>>>>>
>>>>> On Tue, Aug 25, 2020 at 9:43 AM Tyler Richmond
>>>>> <t.d.richmond@gmail.com> wrote:
>>>>>> Great, glad we got somewhere! I'll look forward to the fix!
>>>>>>
>>>>>> On Tue, Aug 25, 2020 at 9:38 AM Qu Wenruo <quwenruo.btrfs@gmx.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 2020/8/25 下午9:30, Tyler Richmond wrote:
>>>>>>>> Qu,
>>>>>>>>
>>>>>>>> The dump of the block is:
>>>>>>>>
>>>>>>>> https://pastebin.com/ran85JJv
>>>>>>>>
>>>>>>>> I've also completed the btrfs-image, but it's almost 50gb.
>>>>>>>> What's the
>>>>>>>> best way to get it to you? Also, does it work with -ss or are the
>>>>>>>> original filenames important?
>>>>>>> 50G is too big for me to even receive.
>>>>>>>
>>>>>>> But your dump shows the problem!
>>>>>>>
>>>>>>> It's not inode generation, but inode transid, which would affect
>>>>>>> send.
>>>>>>>
>>>>>>> This is not even checked in btrfs-progs, thus no wonder why it
>>>>>>> doesn't
>>>>>>> detect them.
>>>>>>>
>>>>>>> And copy-pasted kernel message shares the same "generation" word,
>>>>>>> not
>>>>>>> using proper transid to show the problem.
>>>>>>>
>>>>>>> Your dump really saved the day!
>>>>>>>
>>>>>>> The fix for kernel and btrfs-progs would come in next few days.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Qu
>>>>>>>> Thanks again!
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Aug 25, 2020 at 2:37 AM Qu Wenruo <quwenruo.btrfs@gmx.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2020/8/25 下午1:25, Tyler Richmond wrote:
>>>>>>>>>> Qu,
>>>>>>>>>>
>>>>>>>>>> Yes, it's btrfs-progs 5.7. Here is the result of the lowmem
>>>>>>>>>> check:
>>>>>>>>>>
>>>>>>>>>> https://pastebin.com/8Tzx23EX
>>>>>>>>> That doesn't detect any inode generation problem at all, which is
>>>>>>>>> not a
>>>>>>>>> good sign.
>>>>>>>>>
>>>>>>>>> Would you also pvode the dump for the offending block?
>>>>>>>>>
>>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>> generation:
>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>
>>>>>>>>> For this case, would you please provide the tree dump of
>>>>>>>>> "203510940835840" ?
>>>>>>>>>
>>>>>>>>> # btrfs ins dump-tree -b 203510940835840 <device>
>>>>>>>>>
>>>>>>>>> And, since btrfs-image can't dump with regular extent tree, the
>>>>>>>>> "-w"
>>>>>>>>> dump would also help.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Qu
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 24, 2020 at 4:26 AM Qu Wenruo
>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 2020/8/24 上午10:47, Tyler Richmond wrote:
>>>>>>>>>>>> Qu,
>>>>>>>>>>>>
>>>>>>>>>>>> Finally finished another repair and captured the output.
>>>>>>>>>>>>
>>>>>>>>>>>> https://pastebin.com/ffcbwvd8
>>>>>>>>>>>>
>>>>>>>>>>>> Does that show you what you need? Or should I still do one in
>>>>>>>>>>>> lowmem mode?
>>>>>>>>>>> Lowmem mode (no need for --repair) is recommended since
>>>>>>>>>>> original mode
>>>>>>>>>>> doesn't detect the inode generation problem.
>>>>>>>>>>>
>>>>>>>>>>> And it's already btrfs-progs v5.7 right?
>>>>>>>>>>>
>>>>>>>>>>> THanks,
>>>>>>>>>>> Qu
>>>>>>>>>>>> Thanks for your help!
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo
>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 2020/8/23 上午10:49, Tyler Richmond wrote:
>>>>>>>>>>>>>> Well, I can guarantee that I didn't create this fs before
>>>>>>>>>>>>>> 2015 (just
>>>>>>>>>>>>>> checked the order confirmation from when I bought the
>>>>>>>>>>>>>> server), but I
>>>>>>>>>>>>>> may have just used whatever was in the Ubuntu package
>>>>>>>>>>>>>> manager at the
>>>>>>>>>>>>>> time. So maybe I don't have a v0 ref?
>>>>>>>>>>>>> Then btrfs-image shouldn't report that.
>>>>>>>>>>>>>
>>>>>>>>>>>>> There is an item smaller than any valid btrfs item, normally
>>>>>>>>>>>>> it means
>>>>>>>>>>>>> it's a v0 ref.
>>>>>>>>>>>>> If not, then it could be a bigger problem.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Could you please provide the full btrfs-check output?
>>>>>>>>>>>>> Also, if possible result from "btrfs check --mode=lowmem"
>>>>>>>>>>>>> would also help.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also, if you really go "--repair", then the full output would
>>>>>>>>>>>>> also be
>>>>>>>>>>>>> needed to determine what's going wrong.
>>>>>>>>>>>>> There is a report about "btrfs check --repair" didn't repair
>>>>>>>>>>>>> the inode
>>>>>>>>>>>>> generation, if that's the case we must have a bug then.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo
>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>> Is my best bet just to downgrade the kernel and then try
>>>>>>>>>>>>>>>>> to delete the
>>>>>>>>>>>>>>>>> broken files? Or should I rebuild from scratch? Just
>>>>>>>>>>>>>>>>> don't know
>>>>>>>>>>>>>>>>> whether it's worth the time to try and figure this out or
>>>>>>>>>>>>>>>>> if the
>>>>>>>>>>>>>>>>> problems stem from the FS being too old and it's beyond
>>>>>>>>>>>>>>>>> trying to
>>>>>>>>>>>>>>>>> repair.
>>>>>>>>>>>>>>>> All invalid inode generations, should be able to be
>>>>>>>>>>>>>>>> repaired by latest
>>>>>>>>>>>>>>>> btrfs-check.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If not, please provide the btrfs-image dump for us to
>>>>>>>>>>>>>>>> determine what's
>>>>>>>>>>>>>>>> going wrong.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond
>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> I didn't check dmesg during the btrfs check, but that
>>>>>>>>>>>>>>>>>> was the only
>>>>>>>>>>>>>>>>>> output during the rm -f before it was forced readonly. I
>>>>>>>>>>>>>>>>>> just checked
>>>>>>>>>>>>>>>>>> dmesg for inode generation values, and there are a lot
>>>>>>>>>>>>>>>>>> of them.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> https://pastebin.com/stZdN0ta
>>>>>>>>>>>>>>>>>> The dmesg output had 990 lines containing inode
>>>>>>>>>>>>>>>>>> generation.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> However, these were at least later. I tried to do a
>>>>>>>>>>>>>>>>>> btrfs balance
>>>>>>>>>>>>>>>>>> -mconvert raid1 and it failed with an I/O error. That is
>>>>>>>>>>>>>>>>>> probably what
>>>>>>>>>>>>>>>>>> generated these specific errors, but maybe they were
>>>>>>>>>>>>>>>>>> also happening
>>>>>>>>>>>>>>>>>> during the btrfs repair.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway
>>>>>>>>>>>>>>>>>> with:
>>>>>>>>>>>>>>>>>> ERROR: either extent tree is corrupted or deprecated
>>>>>>>>>>>>>>>>>> extent ref format
>>>>>>>>>>>>>>>>>> ERROR: create failed: -5
>>>>>>>>>>>>>>> Oh, forgot this part.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This means you have v0 ref?!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Then the fs is too old, no progs/kernel support after all.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In that case, please rollback to the last working kernel
>>>>>>>>>>>>>>> and copy your data.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In fact, that v0 ref should only be in the code base for
>>>>>>>>>>>>>>> several weeks
>>>>>>>>>>>>>>> before 2010, thus it's really too old.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The good news is, with tree-checker, we should never
>>>>>>>>>>>>>>> experience such
>>>>>>>>>>>>>>> too-old-to-be-usable problem (at least I hope so)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo
>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Sorry to resurrect this thread, but I just ran into
>>>>>>>>>>>>>>>>>>>> something that I
>>>>>>>>>>>>>>>>>>>> can't really just ignore. I've found a folder that is
>>>>>>>>>>>>>>>>>>>> full of files
>>>>>>>>>>>>>>>>>>>> which I guess have been broken somehow. I found a
>>>>>>>>>>>>>>>>>>>> backup and restored
>>>>>>>>>>>>>>>>>>>> them, but I want to delete this folder of broken
>>>>>>>>>>>>>>>>>>>> files. But whenever I
>>>>>>>>>>>>>>>>>>>> try, the fs is forced into readonly mode again. I just
>>>>>>>>>>>>>>>>>>>> finished another
>>>>>>>>>>>>>>>>>>>> btrfs check --repair but it didn't fix the problem.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> https://pastebin.com/eTV3s3fr
>>>>>>>>>>>>>>>>>>> Is that the full output?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> No inode generation bugs?
>>>>>>>>>>>>>>>>>>>> I'm already on btrfs-progs v5.7. Any new
>>>>>>>>>>>>>>>>>>>> suggestions?
>>>>>>>>>>>>>>>>>>> Strange.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The detection and repair should have been merged into
>>>>>>>>>>>>>>>>>>> v5.5.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> If your fs is small enough, would you please provide
>>>>>>>>>>>>>>>>>>> the "btrfs-image
>>>>>>>>>>>>>>>>>>> -c9" dump?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> It would contain the filenames and directories names,
>>>>>>>>>>>>>>>>>>> but doesn't
>>>>>>>>>>>>>>>>>>> contain file contents.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond
>>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 5.6.1 also failed the same way. Here's the usage
>>>>>>>>>>>>>>>>>>>> output. This is the
>>>>>>>>>>>>>>>>>>>> part where you see I've been using RAID5 haha
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> WARNING: RAID56 detected, not implemented
>>>>>>>>>>>>>>>>>>>> Overall:
>>>>>>>>>>>>>>>>>>>> Device size: 60.03TiB
>>>>>>>>>>>>>>>>>>>> Device allocated: 98.06GiB
>>>>>>>>>>>>>>>>>>>> Device unallocated: 59.93TiB
>>>>>>>>>>>>>>>>>>>> Device missing: 0.00B
>>>>>>>>>>>>>>>>>>>> Used: 92.56GiB
>>>>>>>>>>>>>>>>>>>> Free (estimated): 0.00B
>>>>>>>>>>>>>>>>>>>> (min: 8.00EiB)
>>>>>>>>>>>>>>>>>>>> Data ratio: 0.00
>>>>>>>>>>>>>>>>>>>> Metadata ratio: 2.00
>>>>>>>>>>>>>>>>>>>> Global reserve: 512.00MiB
>>>>>>>>>>>>>>>>>>>> (used: 0.00B)
>>>>>>>>>>>>>>>>>>>> Multiple profiles: no
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
>>>>>>>>>>>>>>>>>>>> /dev/sdh 8.07TiB
>>>>>>>>>>>>>>>>>>>> /dev/sdf 8.07TiB
>>>>>>>>>>>>>>>>>>>> /dev/sdg 8.07TiB
>>>>>>>>>>>>>>>>>>>> /dev/sdd 8.07TiB
>>>>>>>>>>>>>>>>>>>> /dev/sdc 8.07TiB
>>>>>>>>>>>>>>>>>>>> /dev/sde 8.07TiB
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB
>>>>>>>>>>>>>>>>>>>> (94.44%)
>>>>>>>>>>>>>>>>>>>> /dev/sdh 34.00GiB
>>>>>>>>>>>>>>>>>>>> /dev/sdf 32.00GiB
>>>>>>>>>>>>>>>>>>>> /dev/sdg 32.00GiB
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
>>>>>>>>>>>>>>>>>>>> /dev/sdf 32.00MiB
>>>>>>>>>>>>>>>>>>>> /dev/sdg 32.00MiB
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Unallocated:
>>>>>>>>>>>>>>>>>>>> /dev/sdh 2.81TiB
>>>>>>>>>>>>>>>>>>>> /dev/sdf 2.81TiB
>>>>>>>>>>>>>>>>>>>> /dev/sdg 2.81TiB
>>>>>>>>>>>>>>>>>>>> /dev/sdd 1.03TiB
>>>>>>>>>>>>>>>>>>>> /dev/sdc 1.03TiB
>>>>>>>>>>>>>>>>>>>> /dev/sde 1.03TiB
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>>> > > If this is saying there's no extra space for
>>>>>>>>>>>>>>>>>>>> metadata, is that why
>>>>>>>>>>>>>>>>>>>> > > adding more files often makes the system hang
>>>>>>>>>>>>>>>>>>>> for 30-90s? Is there
>>>>>>>>>>>>>>>>>>>> > > anything I should do about that?
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > I'm not sure about the hang though.
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > It would be nice to give more info to
>>>>>>>>>>>>>>>>>>>> diagnosis.
>>>>>>>>>>>>>>>>>>>> > The output of 'btrfs fi usage' is useful for
>>>>>>>>>>>>>>>>>>>> space usage problem.
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > But the common idea is, to keep at 1~2 Gi
>>>>>>>>>>>>>>>>>>>> unallocated (not avaiable
>>>>>>>>>>>>>>>>>>>> > space in vanilla df command) space for btrfs.
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>>> > Qu
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>>> > > Thank you so much for all of your help. I
>>>>>>>>>>>>>>>>>>>> love how flexible BTRFS is
>>>>>>>>>>>>>>>>>>>> > > but when things go wrong it's very hard for
>>>>>>>>>>>>>>>>>>>> me to troubleshoot.
>>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>>> > >>> Something went wrong:
>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>> > >>> Reinitialize checksum tree
>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1`
>>>>>>>>>>>>>>>>>>>> triggered, value 1
>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
>>>>>>>>>>>>>>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
>>>>>>>>>>>>>>>>>>>> > >>> Aborted
>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>> > >> This means no space for extra metadata...
>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>> > >> Anyway the csum tree problem shouldn't be a
>>>>>>>>>>>>>>>>>>>> big thing, you
>>>>>>>>>>>>>>>>>>>> could leave
>>>>>>>>>>>>>>>>>>>> > >> it and call it a day.
>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>> > >> BTW, as long as btrfs check reports no extra
>>>>>>>>>>>>>>>>>>>> problem for the inode
>>>>>>>>>>>>>>>>>>>> > >> generation, it should be pretty safe to use
>>>>>>>>>>>>>>>>>>>> the fs.
>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>> > >> Thanks,
>>>>>>>>>>>>>>>>>>>> > >> Qu
>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>> > >>> I just noticed I have btrfs-progs 5.6
>>>>>>>>>>>>>>>>>>>> installed and 5.6.1 is
>>>>>>>>>>>>>>>>>>>> > >>> available. I'll let that try overnight?
>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> > >>>>> Thank you for helping. The end result of
>>>>>>>>>>>>>>>>>>>> the scan was:
>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>> [1/7] checking root items
>>>>>>>>>>>>>>>>>>>> > >>>>> [2/7] checking extents
>>>>>>>>>>>>>>>>>>>> > >>>>> [3/7] checking free space cache
>>>>>>>>>>>>>>>>>>>> > >>>>> [4/7] checking fs roots
>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>> > >>>> Good news is, your fs is still mostly
>>>>>>>>>>>>>>>>>>>> fine.
>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>> > >>>>> [5/7] checking only csums items (without
>>>>>>>>>>>>>>>>>>>> verifying data)
>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>>> 0-69632
>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 0-69632 but there is no
>>>>>>>>>>>>>>>>>>>> extent record
>>>>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>>> 946692096-946827264
>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 946692096-946827264 but
>>>>>>>>>>>>>>>>>>>> there is no extent
>>>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>>> 946831360-947912704
>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 946831360-947912704 but
>>>>>>>>>>>>>>>>>>>> there is no extent
>>>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>>>> > >>>>> ERROR: errors found in csum tree
>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>> > >>>> Only extent tree is corrupted.
>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>> > >>>> Normally btrfs check --init-csum-tree
>>>>>>>>>>>>>>>>>>>> should be able to
>>>>>>>>>>>>>>>>>>>> handle it.
>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>> > >>>> But still, please be sure you're using the
>>>>>>>>>>>>>>>>>>>> latest btrfs-progs
>>>>>>>>>>>>>>>>>>>> to fix it.
>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>> > >>>> Thanks,
>>>>>>>>>>>>>>>>>>>> > >>>> Qu
>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>> > >>>>> [6/7] checking root refs
>>>>>>>>>>>>>>>>>>>> > >>>>> [7/7] checking quota groups skipped (not
>>>>>>>>>>>>>>>>>>>> enabled on this FS)
>>>>>>>>>>>>>>>>>>>> > >>>>> found 44157956026368 bytes used, error(s)
>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>> > >>>>> total csum bytes: 42038602716
>>>>>>>>>>>>>>>>>>>> > >>>>> total tree bytes: 49688616960
>>>>>>>>>>>>>>>>>>>> > >>>>> total fs tree bytes: 1256427520
>>>>>>>>>>>>>>>>>>>> > >>>>> total extent tree bytes: 1709105152
>>>>>>>>>>>>>>>>>>>> > >>>>> btree space waste bytes: 3172727316
>>>>>>>>>>>>>>>>>>>> > >>>>> file data blocks allocated:
>>>>>>>>>>>>>>>>>>>> 261625653436416
>>>>>>>>>>>>>>>>>>>> > >>>>> referenced 47477768499200
>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>> What do I need to do to fix all of this?
>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> > >>>>>>> Well, the repair doesn't look terribly
>>>>>>>>>>>>>>>>>>>> successful.
>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>> This means there are more problems, not
>>>>>>>>>>>>>>>>>>>> only the hash name
>>>>>>>>>>>>>>>>>>>> mismatch.
>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>> This means the fs is already corrupted,
>>>>>>>>>>>>>>>>>>>> the name hash is
>>>>>>>>>>>>>>>>>>>> just one
>>>>>>>>>>>>>>>>>>>> > >>>>>> unrelated symptom.
>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>> The only good news is, btrfs-progs abort
>>>>>>>>>>>>>>>>>>>> the transaction,
>>>>>>>>>>>>>>>>>>>> thus no
>>>>>>>>>>>>>>>>>>>> > >>>>>> further damage to the fs.
>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>> Please run a plain btrfs-check to show
>>>>>>>>>>>>>>>>>>>> what's the problem
>>>>>>>>>>>>>>>>>>>> first.
>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> > >>>>>> Qu
>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: failed to zero log tree: -17
>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: attempt to start transaction
>>>>>>>>>>>>>>>>>>>> over already running one
>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: reserved space leaked,
>>>>>>>>>>>>>>>>>>>> flag=0x4 bytes_reserved=4096
>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans):
>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans):
>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans):
>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans):
>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>> What is going on?
>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler
>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>>> Chris, I had used the correct
>>>>>>>>>>>>>>>>>>>> mountpoint in the command.
>>>>>>>>>>>>>>>>>>>> I just edited
>>>>>>>>>>>>>>>>>>>> > >>>>>>>> it in the email to be /mountpoint for
>>>>>>>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers
>>>>>>>>>>>>>>>>>>>> crossed!
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu
>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> I looked up this error and it
>>>>>>>>>>>>>>>>>>>> basically says ask a
>>>>>>>>>>>>>>>>>>>> developer to
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> determine if it's a false error or
>>>>>>>>>>>>>>>>>>>> not. I just started
>>>>>>>>>>>>>>>>>>>> getting some
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> slow response times, and looked at
>>>>>>>>>>>>>>>>>>>> the dmesg log to
>>>>>>>>>>>>>>>>>>>> find a ton of
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> these errors.
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical
>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0,
>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error (device
>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical
>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0,
>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error (device
>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical
>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0,
>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error (device
>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> btrfs device stats, however, doesn't
>>>>>>>>>>>>>>>>>>>> show any errors.
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Is there anything I should do about
>>>>>>>>>>>>>>>>>>>> this, or should I
>>>>>>>>>>>>>>>>>>>> just continue
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> using my array as normal?
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> This is caused by older kernel
>>>>>>>>>>>>>>>>>>>> underflow inode generation.
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it, using
>>>>>>>>>>>>>>>>>>>> btrfs check --repair.
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Or you can go safer, by manually
>>>>>>>>>>>>>>>>>>>> locating the inode
>>>>>>>>>>>>>>>>>>>> using its inode
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> number (1311670), and copy it to some
>>>>>>>>>>>>>>>>>>>> new location using
>>>>>>>>>>>>>>>>>>>> previous
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> working kernel, then delete the old
>>>>>>>>>>>>>>>>>>>> file, copy the new
>>>>>>>>>>>>>>>>>>>> one back to fix it.
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-05 23:32 ` Qu Wenruo
@ 2020-11-05 23:37 ` Ferry Toth
2020-11-05 23:40 ` Qu Wenruo
0 siblings, 1 reply; 56+ messages in thread
From: Ferry Toth @ 2020-11-05 23:37 UTC (permalink / raw)
To: Qu Wenruo, Tyler Richmond; +Cc: Btrfs BTRFS
Hi
Op 06-11-2020 om 00:32 schreef Qu Wenruo:
>
> On 2020/11/6 上午7:12, Ferry Toth wrote:
>> Hi,
>>
>> Op 06-11-2020 om 00:00 schreef Qu Wenruo:
>>> On 2020/11/6 上午4:08, Ferry Toth wrote:
>>>> I am in a similar spot, during updating my distro (Kubuntu), I am unable
>>>> to update a certain package. I know which file it is:
>>>>
>>>> ~$ ls -l /usr/share/doc/libatk1.0-data
>>>> ls: kan geen toegang krijgen tot '/usr/share/doc/libatk1.0-data':
>>>> Invoer-/uitvoerfout
>>>>
>>>> This creates the following in journal:
>>>>
>>>> kernel: BTRFS critical (device sda2): corrupt leaf: root=294
>>>> block=1169152675840 slot=1 ino=915987, invalid inode generation: has
>>>> 18446744073709551492 expect [0, 5851353]
>>>> kernel: BTRFS error (device sda2): block=1169152675840 read time tree
>>>> block corruption detected
>>>>
>>>> Now, the problem: this file is on my rootfs, which is mounted. apt
>>>> (distribution updated) installed all packages but can't continue
>>>> configuring, because libatk is a dependancy. I can't delete the file
>>>> because of the I/O error. And btrfs check complains (I tried running RO)
>>>> because the file system is mounted.
>>>>
>>>> But, on the sunny side, the file system is not RO.
>>>>
>>>> Is there any way to forcefully remove the file? Or do you have a
>>>> recommendation how to proceed?
>>> Newer kernel will reject to even read the item, thus will not be able to
>>> remove it.
>> That's already the case. (input / output error)
>>> I guess you have to use some distro ISO to fix the fs.
>> And then? btrfs check --repair the disk offline?
> Yep.
>
> You would want the latest btrfs-progs though.
Groovy has 5.7. Would that be good enough? Otherwise will be difficult
to build on/for live usb image.
>
> THanks,
> Qu
>>> Thanks,
>>> Qu
>>>
>>>> Linux = 5.6.0-1032-oem
>>>>
>>>> Thanks,
>>>> Ferry
>>>>
>>>> Op 05-11-2020 om 08:19 schreef Qu Wenruo:
>>>>> On 2020/11/5 下午3:01, Tyler Richmond wrote:
>>>>>> Qu,
>>>>>>
>>>>>> I'm wondering, was a fix for this ever implemented?
>>>>> Already implemented the --repair ability in latest btrfs-progs.
>>>>>
>>>>>> I recently added a
>>>>>> new drive to expand the array, and during the rebalance it dropped
>>>>>> itself back to a read only filesystem. I suspect it's related to the
>>>>>> issues discussed earlier in this thread. Is there anything I can do to
>>>>>> complete the balance? The error that caused it to drop to read only is
>>>>>> here: https://pastebin.com/GGYVMaiG
>>>>> Yep, the same cause.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 25, 2020 at 9:43 AM Tyler Richmond
>>>>>> <t.d.richmond@gmail.com> wrote:
>>>>>>> Great, glad we got somewhere! I'll look forward to the fix!
>>>>>>>
>>>>>>> On Tue, Aug 25, 2020 at 9:38 AM Qu Wenruo <quwenruo.btrfs@gmx.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> On 2020/8/25 下午9:30, Tyler Richmond wrote:
>>>>>>>>> Qu,
>>>>>>>>>
>>>>>>>>> The dump of the block is:
>>>>>>>>>
>>>>>>>>> https://pastebin.com/ran85JJv
>>>>>>>>>
>>>>>>>>> I've also completed the btrfs-image, but it's almost 50gb.
>>>>>>>>> What's the
>>>>>>>>> best way to get it to you? Also, does it work with -ss or are the
>>>>>>>>> original filenames important?
>>>>>>>> 50G is too big for me to even receive.
>>>>>>>>
>>>>>>>> But your dump shows the problem!
>>>>>>>>
>>>>>>>> It's not inode generation, but inode transid, which would affect
>>>>>>>> send.
>>>>>>>>
>>>>>>>> This is not even checked in btrfs-progs, thus no wonder why it
>>>>>>>> doesn't
>>>>>>>> detect them.
>>>>>>>>
>>>>>>>> And copy-pasted kernel message shares the same "generation" word,
>>>>>>>> not
>>>>>>>> using proper transid to show the problem.
>>>>>>>>
>>>>>>>> Your dump really saved the day!
>>>>>>>>
>>>>>>>> The fix for kernel and btrfs-progs would come in next few days.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Qu
>>>>>>>>> Thanks again!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Aug 25, 2020 at 2:37 AM Qu Wenruo <quwenruo.btrfs@gmx.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> On 2020/8/25 下午1:25, Tyler Richmond wrote:
>>>>>>>>>>> Qu,
>>>>>>>>>>>
>>>>>>>>>>> Yes, it's btrfs-progs 5.7. Here is the result of the lowmem
>>>>>>>>>>> check:
>>>>>>>>>>>
>>>>>>>>>>> https://pastebin.com/8Tzx23EX
>>>>>>>>>> That doesn't detect any inode generation problem at all, which is
>>>>>>>>>> not a
>>>>>>>>>> good sign.
>>>>>>>>>>
>>>>>>>>>> Would you also pvode the dump for the offending block?
>>>>>>>>>>
>>>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>>> generation:
>>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>>
>>>>>>>>>> For this case, would you please provide the tree dump of
>>>>>>>>>> "203510940835840" ?
>>>>>>>>>>
>>>>>>>>>> # btrfs ins dump-tree -b 203510940835840 <device>
>>>>>>>>>>
>>>>>>>>>> And, since btrfs-image can't dump with regular extent tree, the
>>>>>>>>>> "-w"
>>>>>>>>>> dump would also help.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Qu
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Aug 24, 2020 at 4:26 AM Qu Wenruo
>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> On 2020/8/24 上午10:47, Tyler Richmond wrote:
>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Finally finished another repair and captured the output.
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://pastebin.com/ffcbwvd8
>>>>>>>>>>>>>
>>>>>>>>>>>>> Does that show you what you need? Or should I still do one in
>>>>>>>>>>>>> lowmem mode?
>>>>>>>>>>>> Lowmem mode (no need for --repair) is recommended since
>>>>>>>>>>>> original mode
>>>>>>>>>>>> doesn't detect the inode generation problem.
>>>>>>>>>>>>
>>>>>>>>>>>> And it's already btrfs-progs v5.7 right?
>>>>>>>>>>>>
>>>>>>>>>>>> THanks,
>>>>>>>>>>>> Qu
>>>>>>>>>>>>> Thanks for your help!
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo
>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 2020/8/23 上午10:49, Tyler Richmond wrote:
>>>>>>>>>>>>>>> Well, I can guarantee that I didn't create this fs before
>>>>>>>>>>>>>>> 2015 (just
>>>>>>>>>>>>>>> checked the order confirmation from when I bought the
>>>>>>>>>>>>>>> server), but I
>>>>>>>>>>>>>>> may have just used whatever was in the Ubuntu package
>>>>>>>>>>>>>>> manager at the
>>>>>>>>>>>>>>> time. So maybe I don't have a v0 ref?
>>>>>>>>>>>>>> Then btrfs-image shouldn't report that.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> There is an item smaller than any valid btrfs item, normally
>>>>>>>>>>>>>> it means
>>>>>>>>>>>>>> it's a v0 ref.
>>>>>>>>>>>>>> If not, then it could be a bigger problem.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Could you please provide the full btrfs-check output?
>>>>>>>>>>>>>> Also, if possible result from "btrfs check --mode=lowmem"
>>>>>>>>>>>>>> would also help.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also, if you really go "--repair", then the full output would
>>>>>>>>>>>>>> also be
>>>>>>>>>>>>>> needed to determine what's going wrong.
>>>>>>>>>>>>>> There is a report about "btrfs check --repair" didn't repair
>>>>>>>>>>>>>> the inode
>>>>>>>>>>>>>> generation, if that's the case we must have a bug then.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo
>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
>>>>>>>>>>>>>>>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>> Is my best bet just to downgrade the kernel and then try
>>>>>>>>>>>>>>>>>> to delete the
>>>>>>>>>>>>>>>>>> broken files? Or should I rebuild from scratch? Just
>>>>>>>>>>>>>>>>>> don't know
>>>>>>>>>>>>>>>>>> whether it's worth the time to try and figure this out or
>>>>>>>>>>>>>>>>>> if the
>>>>>>>>>>>>>>>>>> problems stem from the FS being too old and it's beyond
>>>>>>>>>>>>>>>>>> trying to
>>>>>>>>>>>>>>>>>> repair.
>>>>>>>>>>>>>>>>> All invalid inode generations, should be able to be
>>>>>>>>>>>>>>>>> repaired by latest
>>>>>>>>>>>>>>>>> btrfs-check.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If not, please provide the btrfs-image dump for us to
>>>>>>>>>>>>>>>>> determine what's
>>>>>>>>>>>>>>>>> going wrong.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond
>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> I didn't check dmesg during the btrfs check, but that
>>>>>>>>>>>>>>>>>>> was the only
>>>>>>>>>>>>>>>>>>> output during the rm -f before it was forced readonly. I
>>>>>>>>>>>>>>>>>>> just checked
>>>>>>>>>>>>>>>>>>> dmesg for inode generation values, and there are a lot
>>>>>>>>>>>>>>>>>>> of them.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> https://pastebin.com/stZdN0ta
>>>>>>>>>>>>>>>>>>> The dmesg output had 990 lines containing inode
>>>>>>>>>>>>>>>>>>> generation.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> However, these were at least later. I tried to do a
>>>>>>>>>>>>>>>>>>> btrfs balance
>>>>>>>>>>>>>>>>>>> -mconvert raid1 and it failed with an I/O error. That is
>>>>>>>>>>>>>>>>>>> probably what
>>>>>>>>>>>>>>>>>>> generated these specific errors, but maybe they were
>>>>>>>>>>>>>>>>>>> also happening
>>>>>>>>>>>>>>>>>>> during the btrfs repair.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway
>>>>>>>>>>>>>>>>>>> with:
>>>>>>>>>>>>>>>>>>> ERROR: either extent tree is corrupted or deprecated
>>>>>>>>>>>>>>>>>>> extent ref format
>>>>>>>>>>>>>>>>>>> ERROR: create failed: -5
>>>>>>>>>>>>>>>> Oh, forgot this part.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This means you have v0 ref?!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Then the fs is too old, no progs/kernel support after all.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In that case, please rollback to the last working kernel
>>>>>>>>>>>>>>>> and copy your data.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In fact, that v0 ref should only be in the code base for
>>>>>>>>>>>>>>>> several weeks
>>>>>>>>>>>>>>>> before 2010, thus it's really too old.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The good news is, with tree-checker, we should never
>>>>>>>>>>>>>>>> experience such
>>>>>>>>>>>>>>>> too-old-to-be-usable problem (at least I hope so)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Sorry to resurrect this thread, but I just ran into
>>>>>>>>>>>>>>>>>>>>> something that I
>>>>>>>>>>>>>>>>>>>>> can't really just ignore. I've found a folder that is
>>>>>>>>>>>>>>>>>>>>> full of files
>>>>>>>>>>>>>>>>>>>>> which I guess have been broken somehow. I found a
>>>>>>>>>>>>>>>>>>>>> backup and restored
>>>>>>>>>>>>>>>>>>>>> them, but I want to delete this folder of broken
>>>>>>>>>>>>>>>>>>>>> files. But whenever I
>>>>>>>>>>>>>>>>>>>>> try, the fs is forced into readonly mode again. I just
>>>>>>>>>>>>>>>>>>>>> finished another
>>>>>>>>>>>>>>>>>>>>> btrfs check --repair but it didn't fix the problem.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> https://pastebin.com/eTV3s3fr
>>>>>>>>>>>>>>>>>>>> Is that the full output?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> No inode generation bugs?
>>>>>>>>>>>>>>>>>>>>> I'm already on btrfs-progs v5.7. Any new
>>>>>>>>>>>>>>>>>>>>> suggestions?
>>>>>>>>>>>>>>>>>>>> Strange.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The detection and repair should have been merged into
>>>>>>>>>>>>>>>>>>>> v5.5.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> If your fs is small enough, would you please provide
>>>>>>>>>>>>>>>>>>>> the "btrfs-image
>>>>>>>>>>>>>>>>>>>> -c9" dump?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> It would contain the filenames and directories names,
>>>>>>>>>>>>>>>>>>>> but doesn't
>>>>>>>>>>>>>>>>>>>> contain file contents.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond
>>>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 5.6.1 also failed the same way. Here's the usage
>>>>>>>>>>>>>>>>>>>>> output. This is the
>>>>>>>>>>>>>>>>>>>>> part where you see I've been using RAID5 haha
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> WARNING: RAID56 detected, not implemented
>>>>>>>>>>>>>>>>>>>>> Overall:
>>>>>>>>>>>>>>>>>>>>> Device size: 60.03TiB
>>>>>>>>>>>>>>>>>>>>> Device allocated: 98.06GiB
>>>>>>>>>>>>>>>>>>>>> Device unallocated: 59.93TiB
>>>>>>>>>>>>>>>>>>>>> Device missing: 0.00B
>>>>>>>>>>>>>>>>>>>>> Used: 92.56GiB
>>>>>>>>>>>>>>>>>>>>> Free (estimated): 0.00B
>>>>>>>>>>>>>>>>>>>>> (min: 8.00EiB)
>>>>>>>>>>>>>>>>>>>>> Data ratio: 0.00
>>>>>>>>>>>>>>>>>>>>> Metadata ratio: 2.00
>>>>>>>>>>>>>>>>>>>>> Global reserve: 512.00MiB
>>>>>>>>>>>>>>>>>>>>> (used: 0.00B)
>>>>>>>>>>>>>>>>>>>>> Multiple profiles: no
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB (99.42%)
>>>>>>>>>>>>>>>>>>>>> /dev/sdh 8.07TiB
>>>>>>>>>>>>>>>>>>>>> /dev/sdf 8.07TiB
>>>>>>>>>>>>>>>>>>>>> /dev/sdg 8.07TiB
>>>>>>>>>>>>>>>>>>>>> /dev/sdd 8.07TiB
>>>>>>>>>>>>>>>>>>>>> /dev/sdc 8.07TiB
>>>>>>>>>>>>>>>>>>>>> /dev/sde 8.07TiB
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB
>>>>>>>>>>>>>>>>>>>>> (94.44%)
>>>>>>>>>>>>>>>>>>>>> /dev/sdh 34.00GiB
>>>>>>>>>>>>>>>>>>>>> /dev/sdf 32.00GiB
>>>>>>>>>>>>>>>>>>>>> /dev/sdg 32.00GiB
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB (6.87%)
>>>>>>>>>>>>>>>>>>>>> /dev/sdf 32.00MiB
>>>>>>>>>>>>>>>>>>>>> /dev/sdg 32.00MiB
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Unallocated:
>>>>>>>>>>>>>>>>>>>>> /dev/sdh 2.81TiB
>>>>>>>>>>>>>>>>>>>>> /dev/sdf 2.81TiB
>>>>>>>>>>>>>>>>>>>>> /dev/sdg 2.81TiB
>>>>>>>>>>>>>>>>>>>>> /dev/sdd 1.03TiB
>>>>>>>>>>>>>>>>>>>>> /dev/sdc 1.03TiB
>>>>>>>>>>>>>>>>>>>>> /dev/sde 1.03TiB
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>>>> > > If this is saying there's no extra space for
>>>>>>>>>>>>>>>>>>>>> metadata, is that why
>>>>>>>>>>>>>>>>>>>>> > > adding more files often makes the system hang
>>>>>>>>>>>>>>>>>>>>> for 30-90s? Is there
>>>>>>>>>>>>>>>>>>>>> > > anything I should do about that?
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > I'm not sure about the hang though.
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > It would be nice to give more info to
>>>>>>>>>>>>>>>>>>>>> diagnosis.
>>>>>>>>>>>>>>>>>>>>> > The output of 'btrfs fi usage' is useful for
>>>>>>>>>>>>>>>>>>>>> space usage problem.
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > But the common idea is, to keep at 1~2 Gi
>>>>>>>>>>>>>>>>>>>>> unallocated (not avaiable
>>>>>>>>>>>>>>>>>>>>> > space in vanilla df command) space for btrfs.
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>>>> > Qu
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>>>> > > Thank you so much for all of your help. I
>>>>>>>>>>>>>>>>>>>>> love how flexible BTRFS is
>>>>>>>>>>>>>>>>>>>>> > > but when things go wrong it's very hard for
>>>>>>>>>>>>>>>>>>>>> me to troubleshoot.
>>>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>>>> > >>> Something went wrong:
>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>> > >>> Reinitialize checksum tree
>>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1`
>>>>>>>>>>>>>>>>>>>>> triggered, value 1
>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
>>>>>>>>>>>>>>>>>>>>> > >>> Aborted
>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>> > >> This means no space for extra metadata...
>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>> > >> Anyway the csum tree problem shouldn't be a
>>>>>>>>>>>>>>>>>>>>> big thing, you
>>>>>>>>>>>>>>>>>>>>> could leave
>>>>>>>>>>>>>>>>>>>>> > >> it and call it a day.
>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>> > >> BTW, as long as btrfs check reports no extra
>>>>>>>>>>>>>>>>>>>>> problem for the inode
>>>>>>>>>>>>>>>>>>>>> > >> generation, it should be pretty safe to use
>>>>>>>>>>>>>>>>>>>>> the fs.
>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>> > >> Thanks,
>>>>>>>>>>>>>>>>>>>>> > >> Qu
>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>> > >>> I just noticed I have btrfs-progs 5.6
>>>>>>>>>>>>>>>>>>>>> installed and 5.6.1 is
>>>>>>>>>>>>>>>>>>>>> > >>> available. I'll let that try overnight?
>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> > >>>>> Thank you for helping. The end result of
>>>>>>>>>>>>>>>>>>>>> the scan was:
>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>> [1/7] checking root items
>>>>>>>>>>>>>>>>>>>>> > >>>>> [2/7] checking extents
>>>>>>>>>>>>>>>>>>>>> > >>>>> [3/7] checking free space cache
>>>>>>>>>>>>>>>>>>>>> > >>>>> [4/7] checking fs roots
>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>> > >>>> Good news is, your fs is still mostly
>>>>>>>>>>>>>>>>>>>>> fine.
>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>> [5/7] checking only csums items (without
>>>>>>>>>>>>>>>>>>>>> verifying data)
>>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>>>> 0-69632
>>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 0-69632 but there is no
>>>>>>>>>>>>>>>>>>>>> extent record
>>>>>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>>>> 946692096-946827264
>>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 946692096-946827264 but
>>>>>>>>>>>>>>>>>>>>> there is no extent
>>>>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>>>> 946831360-947912704
>>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 946831360-947912704 but
>>>>>>>>>>>>>>>>>>>>> there is no extent
>>>>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>>>>> > >>>>> ERROR: errors found in csum tree
>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>> > >>>> Only extent tree is corrupted.
>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>> > >>>> Normally btrfs check --init-csum-tree
>>>>>>>>>>>>>>>>>>>>> should be able to
>>>>>>>>>>>>>>>>>>>>> handle it.
>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>> > >>>> But still, please be sure you're using the
>>>>>>>>>>>>>>>>>>>>> latest btrfs-progs
>>>>>>>>>>>>>>>>>>>>> to fix it.
>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>> > >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> > >>>> Qu
>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>> [6/7] checking root refs
>>>>>>>>>>>>>>>>>>>>> > >>>>> [7/7] checking quota groups skipped (not
>>>>>>>>>>>>>>>>>>>>> enabled on this FS)
>>>>>>>>>>>>>>>>>>>>> > >>>>> found 44157956026368 bytes used, error(s)
>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>> > >>>>> total csum bytes: 42038602716
>>>>>>>>>>>>>>>>>>>>> > >>>>> total tree bytes: 49688616960
>>>>>>>>>>>>>>>>>>>>> > >>>>> total fs tree bytes: 1256427520
>>>>>>>>>>>>>>>>>>>>> > >>>>> total extent tree bytes: 1709105152
>>>>>>>>>>>>>>>>>>>>> > >>>>> btree space waste bytes: 3172727316
>>>>>>>>>>>>>>>>>>>>> > >>>>> file data blocks allocated:
>>>>>>>>>>>>>>>>>>>>> 261625653436416
>>>>>>>>>>>>>>>>>>>>> > >>>>> referenced 47477768499200
>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>> What do I need to do to fix all of this?
>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Well, the repair doesn't look terribly
>>>>>>>>>>>>>>>>>>>>> successful.
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>> This means there are more problems, not
>>>>>>>>>>>>>>>>>>>>> only the hash name
>>>>>>>>>>>>>>>>>>>>> mismatch.
>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>> This means the fs is already corrupted,
>>>>>>>>>>>>>>>>>>>>> the name hash is
>>>>>>>>>>>>>>>>>>>>> just one
>>>>>>>>>>>>>>>>>>>>> > >>>>>> unrelated symptom.
>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>> The only good news is, btrfs-progs abort
>>>>>>>>>>>>>>>>>>>>> the transaction,
>>>>>>>>>>>>>>>>>>>>> thus no
>>>>>>>>>>>>>>>>>>>>> > >>>>>> further damage to the fs.
>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>> Please run a plain btrfs-check to show
>>>>>>>>>>>>>>>>>>>>> what's the problem
>>>>>>>>>>>>>>>>>>>>> first.
>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> > >>>>>> Qu
>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: failed to zero log tree: -17
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: attempt to start transaction
>>>>>>>>>>>>>>>>>>>>> over already running one
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: reserved space leaked,
>>>>>>>>>>>>>>>>>>>>> flag=0x4 bytes_reserved=4096
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans):
>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans):
>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans):
>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted trans):
>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> What is going on?
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler
>>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> Chris, I had used the correct
>>>>>>>>>>>>>>>>>>>>> mountpoint in the command.
>>>>>>>>>>>>>>>>>>>>> I just edited
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> it in the email to be /mountpoint for
>>>>>>>>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers
>>>>>>>>>>>>>>>>>>>>> crossed!
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu
>>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler Richmond
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> I looked up this error and it
>>>>>>>>>>>>>>>>>>>>> basically says ask a
>>>>>>>>>>>>>>>>>>>>> developer to
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> determine if it's a false error or
>>>>>>>>>>>>>>>>>>>>> not. I just started
>>>>>>>>>>>>>>>>>>>>> getting some
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> slow response times, and looked at
>>>>>>>>>>>>>>>>>>>>> the dmesg log to
>>>>>>>>>>>>>>>>>>>>> find a ton of
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> these errors.
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical
>>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0,
>>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error (device
>>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical
>>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0,
>>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error (device
>>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical
>>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492 expect [0,
>>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error (device
>>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption detected
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> btrfs device stats, however, doesn't
>>>>>>>>>>>>>>>>>>>>> show any errors.
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Is there anything I should do about
>>>>>>>>>>>>>>>>>>>>> this, or should I
>>>>>>>>>>>>>>>>>>>>> just continue
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> using my array as normal?
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> This is caused by older kernel
>>>>>>>>>>>>>>>>>>>>> underflow inode generation.
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it, using
>>>>>>>>>>>>>>>>>>>>> btrfs check --repair.
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Or you can go safer, by manually
>>>>>>>>>>>>>>>>>>>>> locating the inode
>>>>>>>>>>>>>>>>>>>>> using its inode
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> number (1311670), and copy it to some
>>>>>>>>>>>>>>>>>>>>> new location using
>>>>>>>>>>>>>>>>>>>>> previous
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> working kernel, then delete the old
>>>>>>>>>>>>>>>>>>>>> file, copy the new
>>>>>>>>>>>>>>>>>>>>> one back to fix it.
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-05 23:37 ` Ferry Toth
@ 2020-11-05 23:40 ` Qu Wenruo
2020-11-06 10:09 ` Ferry Toth
0 siblings, 1 reply; 56+ messages in thread
From: Qu Wenruo @ 2020-11-05 23:40 UTC (permalink / raw)
To: Ferry Toth, Qu Wenruo, Tyler Richmond; +Cc: Btrfs BTRFS
On 2020/11/6 上午7:37, Ferry Toth wrote:
> Hi
>
> Op 06-11-2020 om 00:32 schreef Qu Wenruo:
>>
>> On 2020/11/6 上午7:12, Ferry Toth wrote:
>>> Hi,
>>>
>>> Op 06-11-2020 om 00:00 schreef Qu Wenruo:
>>>> On 2020/11/6 上午4:08, Ferry Toth wrote:
>>>>> I am in a similar spot, during updating my distro (Kubuntu), I am
>>>>> unable
>>>>> to update a certain package. I know which file it is:
>>>>>
>>>>> ~$ ls -l /usr/share/doc/libatk1.0-data
>>>>> ls: kan geen toegang krijgen tot '/usr/share/doc/libatk1.0-data':
>>>>> Invoer-/uitvoerfout
>>>>>
>>>>> This creates the following in journal:
>>>>>
>>>>> kernel: BTRFS critical (device sda2): corrupt leaf: root=294
>>>>> block=1169152675840 slot=1 ino=915987, invalid inode generation: has
>>>>> 18446744073709551492 expect [0, 5851353]
>>>>> kernel: BTRFS error (device sda2): block=1169152675840 read time tree
>>>>> block corruption detected
>>>>>
>>>>> Now, the problem: this file is on my rootfs, which is mounted. apt
>>>>> (distribution updated) installed all packages but can't continue
>>>>> configuring, because libatk is a dependancy. I can't delete the file
>>>>> because of the I/O error. And btrfs check complains (I tried
>>>>> running RO)
>>>>> because the file system is mounted.
>>>>>
>>>>> But, on the sunny side, the file system is not RO.
>>>>>
>>>>> Is there any way to forcefully remove the file? Or do you have a
>>>>> recommendation how to proceed?
>>>> Newer kernel will reject to even read the item, thus will not be
>>>> able to
>>>> remove it.
>>> That's already the case. (input / output error)
>>>> I guess you have to use some distro ISO to fix the fs.
>>> And then? btrfs check --repair the disk offline?
>> Yep.
>>
>> You would want the latest btrfs-progs though.
> Groovy has 5.7. Would that be good enough? Otherwise will be difficult
> to build on/for live usb image.
For your particular case, the fix are already in btrfs-progs v5.4.
Although newer is always better, just in case you have extent item
generation corruption, you may want v5.4.1.
So your v5.7 should be good enough.
Thanks,
Qu
>>
>> THanks,
>> Qu
>>>> Thanks,
>>>> Qu
>>>>
>>>>> Linux = 5.6.0-1032-oem
>>>>>
>>>>> Thanks,
>>>>> Ferry
>>>>>
>>>>> Op 05-11-2020 om 08:19 schreef Qu Wenruo:
>>>>>> On 2020/11/5 下午3:01, Tyler Richmond wrote:
>>>>>>> Qu,
>>>>>>>
>>>>>>> I'm wondering, was a fix for this ever implemented?
>>>>>> Already implemented the --repair ability in latest btrfs-progs.
>>>>>>
>>>>>>> I recently added a
>>>>>>> new drive to expand the array, and during the rebalance it dropped
>>>>>>> itself back to a read only filesystem. I suspect it's related to the
>>>>>>> issues discussed earlier in this thread. Is there anything I can
>>>>>>> do to
>>>>>>> complete the balance? The error that caused it to drop to read
>>>>>>> only is
>>>>>>> here: https://pastebin.com/GGYVMaiG
>>>>>> Yep, the same cause.
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>> Thanks!
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 25, 2020 at 9:43 AM Tyler Richmond
>>>>>>> <t.d.richmond@gmail.com> wrote:
>>>>>>>> Great, glad we got somewhere! I'll look forward to the fix!
>>>>>>>>
>>>>>>>> On Tue, Aug 25, 2020 at 9:38 AM Qu Wenruo <quwenruo.btrfs@gmx.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> On 2020/8/25 下午9:30, Tyler Richmond wrote:
>>>>>>>>>> Qu,
>>>>>>>>>>
>>>>>>>>>> The dump of the block is:
>>>>>>>>>>
>>>>>>>>>> https://pastebin.com/ran85JJv
>>>>>>>>>>
>>>>>>>>>> I've also completed the btrfs-image, but it's almost 50gb.
>>>>>>>>>> What's the
>>>>>>>>>> best way to get it to you? Also, does it work with -ss or are the
>>>>>>>>>> original filenames important?
>>>>>>>>> 50G is too big for me to even receive.
>>>>>>>>>
>>>>>>>>> But your dump shows the problem!
>>>>>>>>>
>>>>>>>>> It's not inode generation, but inode transid, which would affect
>>>>>>>>> send.
>>>>>>>>>
>>>>>>>>> This is not even checked in btrfs-progs, thus no wonder why it
>>>>>>>>> doesn't
>>>>>>>>> detect them.
>>>>>>>>>
>>>>>>>>> And copy-pasted kernel message shares the same "generation" word,
>>>>>>>>> not
>>>>>>>>> using proper transid to show the problem.
>>>>>>>>>
>>>>>>>>> Your dump really saved the day!
>>>>>>>>>
>>>>>>>>> The fix for kernel and btrfs-progs would come in next few days.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Qu
>>>>>>>>>> Thanks again!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 25, 2020 at 2:37 AM Qu Wenruo
>>>>>>>>>> <quwenruo.btrfs@gmx.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> On 2020/8/25 下午1:25, Tyler Richmond wrote:
>>>>>>>>>>>> Qu,
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, it's btrfs-progs 5.7. Here is the result of the lowmem
>>>>>>>>>>>> check:
>>>>>>>>>>>>
>>>>>>>>>>>> https://pastebin.com/8Tzx23EX
>>>>>>>>>>> That doesn't detect any inode generation problem at all,
>>>>>>>>>>> which is
>>>>>>>>>>> not a
>>>>>>>>>>> good sign.
>>>>>>>>>>>
>>>>>>>>>>> Would you also pvode the dump for the offending block?
>>>>>>>>>>>
>>>>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>>>> generation:
>>>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>>>
>>>>>>>>>>> For this case, would you please provide the tree dump of
>>>>>>>>>>> "203510940835840" ?
>>>>>>>>>>>
>>>>>>>>>>> # btrfs ins dump-tree -b 203510940835840 <device>
>>>>>>>>>>>
>>>>>>>>>>> And, since btrfs-image can't dump with regular extent tree, the
>>>>>>>>>>> "-w"
>>>>>>>>>>> dump would also help.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Qu
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Aug 24, 2020 at 4:26 AM Qu Wenruo
>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 2020/8/24 上午10:47, Tyler Richmond wrote:
>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Finally finished another repair and captured the output.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://pastebin.com/ffcbwvd8
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Does that show you what you need? Or should I still do one in
>>>>>>>>>>>>>> lowmem mode?
>>>>>>>>>>>>> Lowmem mode (no need for --repair) is recommended since
>>>>>>>>>>>>> original mode
>>>>>>>>>>>>> doesn't detect the inode generation problem.
>>>>>>>>>>>>>
>>>>>>>>>>>>> And it's already btrfs-progs v5.7 right?
>>>>>>>>>>>>>
>>>>>>>>>>>>> THanks,
>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>> Thanks for your help!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo
>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 2020/8/23 上午10:49, Tyler Richmond wrote:
>>>>>>>>>>>>>>>> Well, I can guarantee that I didn't create this fs before
>>>>>>>>>>>>>>>> 2015 (just
>>>>>>>>>>>>>>>> checked the order confirmation from when I bought the
>>>>>>>>>>>>>>>> server), but I
>>>>>>>>>>>>>>>> may have just used whatever was in the Ubuntu package
>>>>>>>>>>>>>>>> manager at the
>>>>>>>>>>>>>>>> time. So maybe I don't have a v0 ref?
>>>>>>>>>>>>>>> Then btrfs-image shouldn't report that.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There is an item smaller than any valid btrfs item, normally
>>>>>>>>>>>>>>> it means
>>>>>>>>>>>>>>> it's a v0 ref.
>>>>>>>>>>>>>>> If not, then it could be a bigger problem.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Could you please provide the full btrfs-check output?
>>>>>>>>>>>>>>> Also, if possible result from "btrfs check --mode=lowmem"
>>>>>>>>>>>>>>> would also help.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Also, if you really go "--repair", then the full output
>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>> also be
>>>>>>>>>>>>>>> needed to determine what's going wrong.
>>>>>>>>>>>>>>> There is a report about "btrfs check --repair" didn't repair
>>>>>>>>>>>>>>> the inode
>>>>>>>>>>>>>>> generation, if that's the case we must have a bug then.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo
>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
>>>>>>>>>>>>>>>>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>> Is my best bet just to downgrade the kernel and then try
>>>>>>>>>>>>>>>>>>> to delete the
>>>>>>>>>>>>>>>>>>> broken files? Or should I rebuild from scratch? Just
>>>>>>>>>>>>>>>>>>> don't know
>>>>>>>>>>>>>>>>>>> whether it's worth the time to try and figure this
>>>>>>>>>>>>>>>>>>> out or
>>>>>>>>>>>>>>>>>>> if the
>>>>>>>>>>>>>>>>>>> problems stem from the FS being too old and it's beyond
>>>>>>>>>>>>>>>>>>> trying to
>>>>>>>>>>>>>>>>>>> repair.
>>>>>>>>>>>>>>>>>> All invalid inode generations, should be able to be
>>>>>>>>>>>>>>>>>> repaired by latest
>>>>>>>>>>>>>>>>>> btrfs-check.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> If not, please provide the btrfs-image dump for us to
>>>>>>>>>>>>>>>>>> determine what's
>>>>>>>>>>>>>>>>>> going wrong.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond
>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> I didn't check dmesg during the btrfs check, but that
>>>>>>>>>>>>>>>>>>>> was the only
>>>>>>>>>>>>>>>>>>>> output during the rm -f before it was forced
>>>>>>>>>>>>>>>>>>>> readonly. I
>>>>>>>>>>>>>>>>>>>> just checked
>>>>>>>>>>>>>>>>>>>> dmesg for inode generation values, and there are a lot
>>>>>>>>>>>>>>>>>>>> of them.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> https://pastebin.com/stZdN0ta
>>>>>>>>>>>>>>>>>>>> The dmesg output had 990 lines containing inode
>>>>>>>>>>>>>>>>>>>> generation.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> However, these were at least later. I tried to do a
>>>>>>>>>>>>>>>>>>>> btrfs balance
>>>>>>>>>>>>>>>>>>>> -mconvert raid1 and it failed with an I/O error.
>>>>>>>>>>>>>>>>>>>> That is
>>>>>>>>>>>>>>>>>>>> probably what
>>>>>>>>>>>>>>>>>>>> generated these specific errors, but maybe they were
>>>>>>>>>>>>>>>>>>>> also happening
>>>>>>>>>>>>>>>>>>>> during the btrfs repair.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway
>>>>>>>>>>>>>>>>>>>> with:
>>>>>>>>>>>>>>>>>>>> ERROR: either extent tree is corrupted or deprecated
>>>>>>>>>>>>>>>>>>>> extent ref format
>>>>>>>>>>>>>>>>>>>> ERROR: create failed: -5
>>>>>>>>>>>>>>>>> Oh, forgot this part.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This means you have v0 ref?!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Then the fs is too old, no progs/kernel support after all.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In that case, please rollback to the last working kernel
>>>>>>>>>>>>>>>>> and copy your data.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In fact, that v0 ref should only be in the code base for
>>>>>>>>>>>>>>>>> several weeks
>>>>>>>>>>>>>>>>> before 2010, thus it's really too old.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The good news is, with tree-checker, we should never
>>>>>>>>>>>>>>>>> experience such
>>>>>>>>>>>>>>>>> too-old-to-be-usable problem (at least I hope so)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Sorry to resurrect this thread, but I just ran into
>>>>>>>>>>>>>>>>>>>>>> something that I
>>>>>>>>>>>>>>>>>>>>>> can't really just ignore. I've found a folder that is
>>>>>>>>>>>>>>>>>>>>>> full of files
>>>>>>>>>>>>>>>>>>>>>> which I guess have been broken somehow. I found a
>>>>>>>>>>>>>>>>>>>>>> backup and restored
>>>>>>>>>>>>>>>>>>>>>> them, but I want to delete this folder of broken
>>>>>>>>>>>>>>>>>>>>>> files. But whenever I
>>>>>>>>>>>>>>>>>>>>>> try, the fs is forced into readonly mode again. I
>>>>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>>> finished another
>>>>>>>>>>>>>>>>>>>>>> btrfs check --repair but it didn't fix the problem.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> https://pastebin.com/eTV3s3fr
>>>>>>>>>>>>>>>>>>>>> Is that the full output?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> No inode generation bugs?
>>>>>>>>>>>>>>>>>>>>>> I'm already on btrfs-progs v5.7. Any new
>>>>>>>>>>>>>>>>>>>>>> suggestions?
>>>>>>>>>>>>>>>>>>>>> Strange.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The detection and repair should have been merged into
>>>>>>>>>>>>>>>>>>>>> v5.5.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> If your fs is small enough, would you please provide
>>>>>>>>>>>>>>>>>>>>> the "btrfs-image
>>>>>>>>>>>>>>>>>>>>> -c9" dump?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> It would contain the filenames and directories names,
>>>>>>>>>>>>>>>>>>>>> but doesn't
>>>>>>>>>>>>>>>>>>>>> contain file contents.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond
>>>>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 5.6.1 also failed the same way. Here's the
>>>>>>>>>>>>>>>>>>>>>> usage
>>>>>>>>>>>>>>>>>>>>>> output. This is the
>>>>>>>>>>>>>>>>>>>>>> part where you see I've been using RAID5 haha
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> WARNING: RAID56 detected, not implemented
>>>>>>>>>>>>>>>>>>>>>> Overall:
>>>>>>>>>>>>>>>>>>>>>> Device size: 60.03TiB
>>>>>>>>>>>>>>>>>>>>>> Device allocated: 98.06GiB
>>>>>>>>>>>>>>>>>>>>>> Device unallocated: 59.93TiB
>>>>>>>>>>>>>>>>>>>>>> Device missing: 0.00B
>>>>>>>>>>>>>>>>>>>>>> Used: 92.56GiB
>>>>>>>>>>>>>>>>>>>>>> Free (estimated): 0.00B
>>>>>>>>>>>>>>>>>>>>>> (min: 8.00EiB)
>>>>>>>>>>>>>>>>>>>>>> Data ratio: 0.00
>>>>>>>>>>>>>>>>>>>>>> Metadata ratio: 2.00
>>>>>>>>>>>>>>>>>>>>>> Global reserve: 512.00MiB
>>>>>>>>>>>>>>>>>>>>>> (used: 0.00B)
>>>>>>>>>>>>>>>>>>>>>> Multiple profiles: no
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB
>>>>>>>>>>>>>>>>>>>>>> (99.42%)
>>>>>>>>>>>>>>>>>>>>>> /dev/sdh 8.07TiB
>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 8.07TiB
>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 8.07TiB
>>>>>>>>>>>>>>>>>>>>>> /dev/sdd 8.07TiB
>>>>>>>>>>>>>>>>>>>>>> /dev/sdc 8.07TiB
>>>>>>>>>>>>>>>>>>>>>> /dev/sde 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB
>>>>>>>>>>>>>>>>>>>>>> (94.44%)
>>>>>>>>>>>>>>>>>>>>>> /dev/sdh 34.00GiB
>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 32.00GiB
>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 32.00GiB
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB
>>>>>>>>>>>>>>>>>>>>>> (6.87%)
>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 32.00MiB
>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 32.00MiB
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Unallocated:
>>>>>>>>>>>>>>>>>>>>>> /dev/sdh 2.81TiB
>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 2.81TiB
>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 2.81TiB
>>>>>>>>>>>>>>>>>>>>>> /dev/sdd 1.03TiB
>>>>>>>>>>>>>>>>>>>>>> /dev/sdc 1.03TiB
>>>>>>>>>>>>>>>>>>>>>> /dev/sde 1.03TiB
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>>>>> > > If this is saying there's no extra
>>>>>>>>>>>>>>>>>>>>>> space for
>>>>>>>>>>>>>>>>>>>>>> metadata, is that why
>>>>>>>>>>>>>>>>>>>>>> > > adding more files often makes the
>>>>>>>>>>>>>>>>>>>>>> system hang
>>>>>>>>>>>>>>>>>>>>>> for 30-90s? Is there
>>>>>>>>>>>>>>>>>>>>>> > > anything I should do about that?
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > I'm not sure about the hang though.
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > It would be nice to give more info to
>>>>>>>>>>>>>>>>>>>>>> diagnosis.
>>>>>>>>>>>>>>>>>>>>>> > The output of 'btrfs fi usage' is useful for
>>>>>>>>>>>>>>>>>>>>>> space usage problem.
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > But the common idea is, to keep at 1~2 Gi
>>>>>>>>>>>>>>>>>>>>>> unallocated (not avaiable
>>>>>>>>>>>>>>>>>>>>>> > space in vanilla df command) space for
>>>>>>>>>>>>>>>>>>>>>> btrfs.
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>>>>> > Qu
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>>>>> > > Thank you so much for all of your help. I
>>>>>>>>>>>>>>>>>>>>>> love how flexible BTRFS is
>>>>>>>>>>>>>>>>>>>>>> > > but when things go wrong it's very hard
>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>> me to troubleshoot.
>>>>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> > >>> Something went wrong:
>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>> > >>> Reinitialize checksum tree
>>>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1`
>>>>>>>>>>>>>>>>>>>>>> triggered, value 1
>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
>>>>>>>>>>>>>>>>>>>>>> > >>> Aborted
>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>> > >> This means no space for extra metadata...
>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>> > >> Anyway the csum tree problem shouldn't
>>>>>>>>>>>>>>>>>>>>>> be a
>>>>>>>>>>>>>>>>>>>>>> big thing, you
>>>>>>>>>>>>>>>>>>>>>> could leave
>>>>>>>>>>>>>>>>>>>>>> > >> it and call it a day.
>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>> > >> BTW, as long as btrfs check reports no
>>>>>>>>>>>>>>>>>>>>>> extra
>>>>>>>>>>>>>>>>>>>>>> problem for the inode
>>>>>>>>>>>>>>>>>>>>>> > >> generation, it should be pretty safe
>>>>>>>>>>>>>>>>>>>>>> to use
>>>>>>>>>>>>>>>>>>>>>> the fs.
>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>> > >> Thanks,
>>>>>>>>>>>>>>>>>>>>>> > >> Qu
>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>> > >>> I just noticed I have btrfs-progs 5.6
>>>>>>>>>>>>>>>>>>>>>> installed and 5.6.1 is
>>>>>>>>>>>>>>>>>>>>>> > >>> available. I'll let that try overnight?
>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> > >>>>> Thank you for helping. The end
>>>>>>>>>>>>>>>>>>>>>> result of
>>>>>>>>>>>>>>>>>>>>>> the scan was:
>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>> [1/7] checking root items
>>>>>>>>>>>>>>>>>>>>>> > >>>>> [2/7] checking extents
>>>>>>>>>>>>>>>>>>>>>> > >>>>> [3/7] checking free space cache
>>>>>>>>>>>>>>>>>>>>>> > >>>>> [4/7] checking fs roots
>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>> Good news is, your fs is still mostly
>>>>>>>>>>>>>>>>>>>>>> fine.
>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>> [5/7] checking only csums items
>>>>>>>>>>>>>>>>>>>>>> (without
>>>>>>>>>>>>>>>>>>>>>> verifying data)
>>>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>>>>> 0-69632
>>>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 0-69632 but there
>>>>>>>>>>>>>>>>>>>>>> is no
>>>>>>>>>>>>>>>>>>>>>> extent record
>>>>>>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>>>>> 946692096-946827264
>>>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 946692096-946827264
>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>> there is no extent
>>>>>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>>>>> 946831360-947912704
>>>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 946831360-947912704
>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>> there is no extent
>>>>>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>>>>>> > >>>>> ERROR: errors found in csum tree
>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>> Only extent tree is corrupted.
>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>> Normally btrfs check --init-csum-tree
>>>>>>>>>>>>>>>>>>>>>> should be able to
>>>>>>>>>>>>>>>>>>>>>> handle it.
>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>> But still, please be sure you're
>>>>>>>>>>>>>>>>>>>>>> using the
>>>>>>>>>>>>>>>>>>>>>> latest btrfs-progs
>>>>>>>>>>>>>>>>>>>>>> to fix it.
>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> > >>>> Qu
>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>> [6/7] checking root refs
>>>>>>>>>>>>>>>>>>>>>> > >>>>> [7/7] checking quota groups skipped
>>>>>>>>>>>>>>>>>>>>>> (not
>>>>>>>>>>>>>>>>>>>>>> enabled on this FS)
>>>>>>>>>>>>>>>>>>>>>> > >>>>> found 44157956026368 bytes used,
>>>>>>>>>>>>>>>>>>>>>> error(s)
>>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>>> > >>>>> total csum bytes: 42038602716
>>>>>>>>>>>>>>>>>>>>>> > >>>>> total tree bytes: 49688616960
>>>>>>>>>>>>>>>>>>>>>> > >>>>> total fs tree bytes: 1256427520
>>>>>>>>>>>>>>>>>>>>>> > >>>>> total extent tree bytes: 1709105152
>>>>>>>>>>>>>>>>>>>>>> > >>>>> btree space waste bytes: 3172727316
>>>>>>>>>>>>>>>>>>>>>> > >>>>> file data blocks allocated:
>>>>>>>>>>>>>>>>>>>>>> 261625653436416
>>>>>>>>>>>>>>>>>>>>>> > >>>>> referenced 47477768499200
>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>> What do I need to do to fix all of
>>>>>>>>>>>>>>>>>>>>>> this?
>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu
>>>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Well, the repair doesn't look
>>>>>>>>>>>>>>>>>>>>>> terribly
>>>>>>>>>>>>>>>>>>>>>> successful.
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>> This means there are more
>>>>>>>>>>>>>>>>>>>>>> problems, not
>>>>>>>>>>>>>>>>>>>>>> only the hash name
>>>>>>>>>>>>>>>>>>>>>> mismatch.
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>> This means the fs is already
>>>>>>>>>>>>>>>>>>>>>> corrupted,
>>>>>>>>>>>>>>>>>>>>>> the name hash is
>>>>>>>>>>>>>>>>>>>>>> just one
>>>>>>>>>>>>>>>>>>>>>> > >>>>>> unrelated symptom.
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>> The only good news is, btrfs-progs
>>>>>>>>>>>>>>>>>>>>>> abort
>>>>>>>>>>>>>>>>>>>>>> the transaction,
>>>>>>>>>>>>>>>>>>>>>> thus no
>>>>>>>>>>>>>>>>>>>>>> > >>>>>> further damage to the fs.
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Please run a plain btrfs-check to
>>>>>>>>>>>>>>>>>>>>>> show
>>>>>>>>>>>>>>>>>>>>>> what's the problem
>>>>>>>>>>>>>>>>>>>>>> first.
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Qu
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: failed to zero log tree: -17
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: attempt to start transaction
>>>>>>>>>>>>>>>>>>>>>> over already running one
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: reserved space leaked,
>>>>>>>>>>>>>>>>>>>>>> flag=0x4 bytes_reserved=4096
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted
>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted
>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted
>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted
>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> What is going on?
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler
>>>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> Chris, I had used the correct
>>>>>>>>>>>>>>>>>>>>>> mountpoint in the command.
>>>>>>>>>>>>>>>>>>>>>> I just edited
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> it in the email to be
>>>>>>>>>>>>>>>>>>>>>> /mountpoint for
>>>>>>>>>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers
>>>>>>>>>>>>>>>>>>>>>> crossed!
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu
>>>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler
>>>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> I looked up this error and it
>>>>>>>>>>>>>>>>>>>>>> basically says ask a
>>>>>>>>>>>>>>>>>>>>>> developer to
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> determine if it's a false
>>>>>>>>>>>>>>>>>>>>>> error or
>>>>>>>>>>>>>>>>>>>>>> not. I just started
>>>>>>>>>>>>>>>>>>>>>> getting some
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> slow response times, and
>>>>>>>>>>>>>>>>>>>>>> looked at
>>>>>>>>>>>>>>>>>>>>>> the dmesg log to
>>>>>>>>>>>>>>>>>>>>>> find a ton of
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> these errors.
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical
>>>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492
>>>>>>>>>>>>>>>>>>>>>> expect [0,
>>>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error
>>>>>>>>>>>>>>>>>>>>>> (device
>>>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption
>>>>>>>>>>>>>>>>>>>>>> detected
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical
>>>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492
>>>>>>>>>>>>>>>>>>>>>> expect [0,
>>>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error
>>>>>>>>>>>>>>>>>>>>>> (device
>>>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption
>>>>>>>>>>>>>>>>>>>>>> detected
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical
>>>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492
>>>>>>>>>>>>>>>>>>>>>> expect [0,
>>>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error
>>>>>>>>>>>>>>>>>>>>>> (device
>>>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption
>>>>>>>>>>>>>>>>>>>>>> detected
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> btrfs device stats, however,
>>>>>>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>>>>> show any errors.
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Is there anything I should do
>>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>> this, or should I
>>>>>>>>>>>>>>>>>>>>>> just continue
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> using my array as normal?
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> This is caused by older kernel
>>>>>>>>>>>>>>>>>>>>>> underflow inode generation.
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it,
>>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>>> btrfs check --repair.
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Or you can go safer, by manually
>>>>>>>>>>>>>>>>>>>>>> locating the inode
>>>>>>>>>>>>>>>>>>>>>> using its inode
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> number (1311670), and copy it
>>>>>>>>>>>>>>>>>>>>>> to some
>>>>>>>>>>>>>>>>>>>>>> new location using
>>>>>>>>>>>>>>>>>>>>>> previous
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> working kernel, then delete the
>>>>>>>>>>>>>>>>>>>>>> old
>>>>>>>>>>>>>>>>>>>>>> file, copy the new
>>>>>>>>>>>>>>>>>>>>>> one back to fix it.
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-05 23:40 ` Qu Wenruo
@ 2020-11-06 10:09 ` Ferry Toth
2020-11-06 10:24 ` Qu Wenruo
0 siblings, 1 reply; 56+ messages in thread
From: Ferry Toth @ 2020-11-06 10:09 UTC (permalink / raw)
To: Qu Wenruo, Qu Wenruo, Tyler Richmond; +Cc: Btrfs BTRFS
Hi Qu
Op 06-11-2020 om 00:40 schreef Qu Wenruo:
>
> On 2020/11/6 上午7:37, Ferry Toth wrote:
>> Hi
>>
>> Op 06-11-2020 om 00:32 schreef Qu Wenruo:
>>> On 2020/11/6 上午7:12, Ferry Toth wrote:
>>>> Hi,
>>>>
>>>> Op 06-11-2020 om 00:00 schreef Qu Wenruo:
>>>>> On 2020/11/6 上午4:08, Ferry Toth wrote:
>>>>>> I am in a similar spot, during updating my distro (Kubuntu), I am
>>>>>> unable
>>>>>> to update a certain package. I know which file it is:
>>>>>>
>>>>>> ~$ ls -l /usr/share/doc/libatk1.0-data
>>>>>> ls: kan geen toegang krijgen tot '/usr/share/doc/libatk1.0-data':
>>>>>> Invoer-/uitvoerfout
>>>>>>
>>>>>> This creates the following in journal:
>>>>>>
>>>>>> kernel: BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>> block=1169152675840 slot=1 ino=915987, invalid inode generation: has
>>>>>> 18446744073709551492 expect [0, 5851353]
>>>>>> kernel: BTRFS error (device sda2): block=1169152675840 read time tree
>>>>>> block corruption detected
>>>>>>
>>>>>> Now, the problem: this file is on my rootfs, which is mounted. apt
>>>>>> (distribution updated) installed all packages but can't continue
>>>>>> configuring, because libatk is a dependancy. I can't delete the file
>>>>>> because of the I/O error. And btrfs check complains (I tried
>>>>>> running RO)
>>>>>> because the file system is mounted.
>>>>>>
>>>>>> But, on the sunny side, the file system is not RO.
>>>>>>
>>>>>> Is there any way to forcefully remove the file? Or do you have a
>>>>>> recommendation how to proceed?
>>>>> Newer kernel will reject to even read the item, thus will not be
>>>>> able to
>>>>> remove it.
>>>> That's already the case. (input / output error)
>>>>> I guess you have to use some distro ISO to fix the fs.
>>>> And then? btrfs check --repair the disk offline?
>>> Yep.
>>>
>>> You would want the latest btrfs-progs though.
>> Groovy has 5.7. Would that be good enough? Otherwise will be difficult
>> to build on/for live usb image.
> For your particular case, the fix are already in btrfs-progs v5.4.
>
> Although newer is always better, just in case you have extent item
> generation corruption, you may want v5.4.1.
>
> So your v5.7 should be good enough.
>
> Thanks,
> Qu
I made a live usb and performed:
btrfs check --repair /dev/sda2
It found errors and fixed them. However, it did not fix the corrupt leaf. The file is actually a directory:
~$ stat /usr/share/doc/libatk1.0-data
stat: cannot statx '/usr/share/doc/libatk1.0-data': Invoer-/uitvoerfout
in journal:
BTRFS critical (device sda2): corrupt leaf: root=294 block=1169152675840
slot=1 ino=915987, invalid inode generation: has 18446744073709551492
expect [0, 5852829]
BTRFS error (device sda2): block=1169152675840 read time tree block
corruption detected
So how do I repair this? Am I doing something wrong?
>>> THanks,
>>> Qu
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>>> Linux = 5.6.0-1032-oem
>>>>>>
>>>>>> Thanks,
>>>>>> Ferry
>>>>>>
>>>>>> Op 05-11-2020 om 08:19 schreef Qu Wenruo:
>>>>>>> On 2020/11/5 下午3:01, Tyler Richmond wrote:
>>>>>>>> Qu,
>>>>>>>>
>>>>>>>> I'm wondering, was a fix for this ever implemented?
>>>>>>> Already implemented the --repair ability in latest btrfs-progs.
>>>>>>>
>>>>>>>> I recently added a
>>>>>>>> new drive to expand the array, and during the rebalance it dropped
>>>>>>>> itself back to a read only filesystem. I suspect it's related to the
>>>>>>>> issues discussed earlier in this thread. Is there anything I can
>>>>>>>> do to
>>>>>>>> complete the balance? The error that caused it to drop to read
>>>>>>>> only is
>>>>>>>> here: https://pastebin.com/GGYVMaiG
>>>>>>> Yep, the same cause.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Qu
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Aug 25, 2020 at 9:43 AM Tyler Richmond
>>>>>>>> <t.d.richmond@gmail.com> wrote:
>>>>>>>>> Great, glad we got somewhere! I'll look forward to the fix!
>>>>>>>>>
>>>>>>>>> On Tue, Aug 25, 2020 at 9:38 AM Qu Wenruo <quwenruo.btrfs@gmx.com>
>>>>>>>>> wrote:
>>>>>>>>>> On 2020/8/25 下午9:30, Tyler Richmond wrote:
>>>>>>>>>>> Qu,
>>>>>>>>>>>
>>>>>>>>>>> The dump of the block is:
>>>>>>>>>>>
>>>>>>>>>>> https://pastebin.com/ran85JJv
>>>>>>>>>>>
>>>>>>>>>>> I've also completed the btrfs-image, but it's almost 50gb.
>>>>>>>>>>> What's the
>>>>>>>>>>> best way to get it to you? Also, does it work with -ss or are the
>>>>>>>>>>> original filenames important?
>>>>>>>>>> 50G is too big for me to even receive.
>>>>>>>>>>
>>>>>>>>>> But your dump shows the problem!
>>>>>>>>>>
>>>>>>>>>> It's not inode generation, but inode transid, which would affect
>>>>>>>>>> send.
>>>>>>>>>>
>>>>>>>>>> This is not even checked in btrfs-progs, thus no wonder why it
>>>>>>>>>> doesn't
>>>>>>>>>> detect them.
>>>>>>>>>>
>>>>>>>>>> And copy-pasted kernel message shares the same "generation" word,
>>>>>>>>>> not
>>>>>>>>>> using proper transid to show the problem.
>>>>>>>>>>
>>>>>>>>>> Your dump really saved the day!
>>>>>>>>>>
>>>>>>>>>> The fix for kernel and btrfs-progs would come in next few days.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Qu
>>>>>>>>>>> Thanks again!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 25, 2020 at 2:37 AM Qu Wenruo
>>>>>>>>>>> <quwenruo.btrfs@gmx.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> On 2020/8/25 下午1:25, Tyler Richmond wrote:
>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, it's btrfs-progs 5.7. Here is the result of the lowmem
>>>>>>>>>>>>> check:
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://pastebin.com/8Tzx23EX
>>>>>>>>>>>> That doesn't detect any inode generation problem at all,
>>>>>>>>>>>> which is
>>>>>>>>>>>> not a
>>>>>>>>>>>> good sign.
>>>>>>>>>>>>
>>>>>>>>>>>> Would you also pvode the dump for the offending block?
>>>>>>>>>>>>
>>>>>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>>>>> generation:
>>>>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>>>>
>>>>>>>>>>>> For this case, would you please provide the tree dump of
>>>>>>>>>>>> "203510940835840" ?
>>>>>>>>>>>>
>>>>>>>>>>>> # btrfs ins dump-tree -b 203510940835840 <device>
>>>>>>>>>>>>
>>>>>>>>>>>> And, since btrfs-image can't dump with regular extent tree, the
>>>>>>>>>>>> "-w"
>>>>>>>>>>>> dump would also help.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Qu
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Aug 24, 2020 at 4:26 AM Qu Wenruo
>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>> On 2020/8/24 上午10:47, Tyler Richmond wrote:
>>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Finally finished another repair and captured the output.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://pastebin.com/ffcbwvd8
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Does that show you what you need? Or should I still do one in
>>>>>>>>>>>>>>> lowmem mode?
>>>>>>>>>>>>>> Lowmem mode (no need for --repair) is recommended since
>>>>>>>>>>>>>> original mode
>>>>>>>>>>>>>> doesn't detect the inode generation problem.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And it's already btrfs-progs v5.7 right?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> THanks,
>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>> Thanks for your help!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo
>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>> On 2020/8/23 上午10:49, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>> Well, I can guarantee that I didn't create this fs before
>>>>>>>>>>>>>>>>> 2015 (just
>>>>>>>>>>>>>>>>> checked the order confirmation from when I bought the
>>>>>>>>>>>>>>>>> server), but I
>>>>>>>>>>>>>>>>> may have just used whatever was in the Ubuntu package
>>>>>>>>>>>>>>>>> manager at the
>>>>>>>>>>>>>>>>> time. So maybe I don't have a v0 ref?
>>>>>>>>>>>>>>>> Then btrfs-image shouldn't report that.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There is an item smaller than any valid btrfs item, normally
>>>>>>>>>>>>>>>> it means
>>>>>>>>>>>>>>>> it's a v0 ref.
>>>>>>>>>>>>>>>> If not, then it could be a bigger problem.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Could you please provide the full btrfs-check output?
>>>>>>>>>>>>>>>> Also, if possible result from "btrfs check --mode=lowmem"
>>>>>>>>>>>>>>>> would also help.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also, if you really go "--repair", then the full output
>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>> also be
>>>>>>>>>>>>>>>> needed to determine what's going wrong.
>>>>>>>>>>>>>>>> There is a report about "btrfs check --repair" didn't repair
>>>>>>>>>>>>>>>> the inode
>>>>>>>>>>>>>>>> generation, if that's the case we must have a bug then.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo
>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
>>>>>>>>>>>>>>>>>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>>> Is my best bet just to downgrade the kernel and then try
>>>>>>>>>>>>>>>>>>>> to delete the
>>>>>>>>>>>>>>>>>>>> broken files? Or should I rebuild from scratch? Just
>>>>>>>>>>>>>>>>>>>> don't know
>>>>>>>>>>>>>>>>>>>> whether it's worth the time to try and figure this
>>>>>>>>>>>>>>>>>>>> out or
>>>>>>>>>>>>>>>>>>>> if the
>>>>>>>>>>>>>>>>>>>> problems stem from the FS being too old and it's beyond
>>>>>>>>>>>>>>>>>>>> trying to
>>>>>>>>>>>>>>>>>>>> repair.
>>>>>>>>>>>>>>>>>>> All invalid inode generations, should be able to be
>>>>>>>>>>>>>>>>>>> repaired by latest
>>>>>>>>>>>>>>>>>>> btrfs-check.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> If not, please provide the btrfs-image dump for us to
>>>>>>>>>>>>>>>>>>> determine what's
>>>>>>>>>>>>>>>>>>> going wrong.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond
>>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> I didn't check dmesg during the btrfs check, but that
>>>>>>>>>>>>>>>>>>>>> was the only
>>>>>>>>>>>>>>>>>>>>> output during the rm -f before it was forced
>>>>>>>>>>>>>>>>>>>>> readonly. I
>>>>>>>>>>>>>>>>>>>>> just checked
>>>>>>>>>>>>>>>>>>>>> dmesg for inode generation values, and there are a lot
>>>>>>>>>>>>>>>>>>>>> of them.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> https://pastebin.com/stZdN0ta
>>>>>>>>>>>>>>>>>>>>> The dmesg output had 990 lines containing inode
>>>>>>>>>>>>>>>>>>>>> generation.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> However, these were at least later. I tried to do a
>>>>>>>>>>>>>>>>>>>>> btrfs balance
>>>>>>>>>>>>>>>>>>>>> -mconvert raid1 and it failed with an I/O error.
>>>>>>>>>>>>>>>>>>>>> That is
>>>>>>>>>>>>>>>>>>>>> probably what
>>>>>>>>>>>>>>>>>>>>> generated these specific errors, but maybe they were
>>>>>>>>>>>>>>>>>>>>> also happening
>>>>>>>>>>>>>>>>>>>>> during the btrfs repair.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway
>>>>>>>>>>>>>>>>>>>>> with:
>>>>>>>>>>>>>>>>>>>>> ERROR: either extent tree is corrupted or deprecated
>>>>>>>>>>>>>>>>>>>>> extent ref format
>>>>>>>>>>>>>>>>>>>>> ERROR: create failed: -5
>>>>>>>>>>>>>>>>>> Oh, forgot this part.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This means you have v0 ref?!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Then the fs is too old, no progs/kernel support after all.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> In that case, please rollback to the last working kernel
>>>>>>>>>>>>>>>>>> and copy your data.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> In fact, that v0 ref should only be in the code base for
>>>>>>>>>>>>>>>>>> several weeks
>>>>>>>>>>>>>>>>>> before 2010, thus it's really too old.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The good news is, with tree-checker, we should never
>>>>>>>>>>>>>>>>>> experience such
>>>>>>>>>>>>>>>>>> too-old-to-be-usable problem (at least I hope so)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Sorry to resurrect this thread, but I just ran into
>>>>>>>>>>>>>>>>>>>>>>> something that I
>>>>>>>>>>>>>>>>>>>>>>> can't really just ignore. I've found a folder that is
>>>>>>>>>>>>>>>>>>>>>>> full of files
>>>>>>>>>>>>>>>>>>>>>>> which I guess have been broken somehow. I found a
>>>>>>>>>>>>>>>>>>>>>>> backup and restored
>>>>>>>>>>>>>>>>>>>>>>> them, but I want to delete this folder of broken
>>>>>>>>>>>>>>>>>>>>>>> files. But whenever I
>>>>>>>>>>>>>>>>>>>>>>> try, the fs is forced into readonly mode again. I
>>>>>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>>>> finished another
>>>>>>>>>>>>>>>>>>>>>>> btrfs check --repair but it didn't fix the problem.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> https://pastebin.com/eTV3s3fr
>>>>>>>>>>>>>>>>>>>>>> Is that the full output?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> No inode generation bugs?
>>>>>>>>>>>>>>>>>>>>>>> I'm already on btrfs-progs v5.7. Any new
>>>>>>>>>>>>>>>>>>>>>>> suggestions?
>>>>>>>>>>>>>>>>>>>>>> Strange.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The detection and repair should have been merged into
>>>>>>>>>>>>>>>>>>>>>> v5.5.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> If your fs is small enough, would you please provide
>>>>>>>>>>>>>>>>>>>>>> the "btrfs-image
>>>>>>>>>>>>>>>>>>>>>> -c9" dump?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> It would contain the filenames and directories names,
>>>>>>>>>>>>>>>>>>>>>> but doesn't
>>>>>>>>>>>>>>>>>>>>>> contain file contents.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond
>>>>>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 5.6.1 also failed the same way. Here's the
>>>>>>>>>>>>>>>>>>>>>>> usage
>>>>>>>>>>>>>>>>>>>>>>> output. This is the
>>>>>>>>>>>>>>>>>>>>>>> part where you see I've been using RAID5 haha
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> WARNING: RAID56 detected, not implemented
>>>>>>>>>>>>>>>>>>>>>>> Overall:
>>>>>>>>>>>>>>>>>>>>>>> Device size: 60.03TiB
>>>>>>>>>>>>>>>>>>>>>>> Device allocated: 98.06GiB
>>>>>>>>>>>>>>>>>>>>>>> Device unallocated: 59.93TiB
>>>>>>>>>>>>>>>>>>>>>>> Device missing: 0.00B
>>>>>>>>>>>>>>>>>>>>>>> Used: 92.56GiB
>>>>>>>>>>>>>>>>>>>>>>> Free (estimated): 0.00B
>>>>>>>>>>>>>>>>>>>>>>> (min: 8.00EiB)
>>>>>>>>>>>>>>>>>>>>>>> Data ratio: 0.00
>>>>>>>>>>>>>>>>>>>>>>> Metadata ratio: 2.00
>>>>>>>>>>>>>>>>>>>>>>> Global reserve: 512.00MiB
>>>>>>>>>>>>>>>>>>>>>>> (used: 0.00B)
>>>>>>>>>>>>>>>>>>>>>>> Multiple profiles: no
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB
>>>>>>>>>>>>>>>>>>>>>>> (99.42%)
>>>>>>>>>>>>>>>>>>>>>>> /dev/sdh 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>> /dev/sdd 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>> /dev/sdc 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>> /dev/sde 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Metadata,RAID1: Size:49.00GiB, Used:46.28GiB
>>>>>>>>>>>>>>>>>>>>>>> (94.44%)
>>>>>>>>>>>>>>>>>>>>>>> /dev/sdh 34.00GiB
>>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 32.00GiB
>>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 32.00GiB
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB
>>>>>>>>>>>>>>>>>>>>>>> (6.87%)
>>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 32.00MiB
>>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 32.00MiB
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Unallocated:
>>>>>>>>>>>>>>>>>>>>>>> /dev/sdh 2.81TiB
>>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 2.81TiB
>>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 2.81TiB
>>>>>>>>>>>>>>>>>>>>>>> /dev/sdd 1.03TiB
>>>>>>>>>>>>>>>>>>>>>>> /dev/sdc 1.03TiB
>>>>>>>>>>>>>>>>>>>>>>> /dev/sde 1.03TiB
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>>>>>> > > If this is saying there's no extra
>>>>>>>>>>>>>>>>>>>>>>> space for
>>>>>>>>>>>>>>>>>>>>>>> metadata, is that why
>>>>>>>>>>>>>>>>>>>>>>> > > adding more files often makes the
>>>>>>>>>>>>>>>>>>>>>>> system hang
>>>>>>>>>>>>>>>>>>>>>>> for 30-90s? Is there
>>>>>>>>>>>>>>>>>>>>>>> > > anything I should do about that?
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > I'm not sure about the hang though.
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > It would be nice to give more info to
>>>>>>>>>>>>>>>>>>>>>>> diagnosis.
>>>>>>>>>>>>>>>>>>>>>>> > The output of 'btrfs fi usage' is useful for
>>>>>>>>>>>>>>>>>>>>>>> space usage problem.
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > But the common idea is, to keep at 1~2 Gi
>>>>>>>>>>>>>>>>>>>>>>> unallocated (not avaiable
>>>>>>>>>>>>>>>>>>>>>>> > space in vanilla df command) space for
>>>>>>>>>>>>>>>>>>>>>>> btrfs.
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>>>>>> > Qu
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>>>>>> > > Thank you so much for all of your help. I
>>>>>>>>>>>>>>>>>>>>>>> love how flexible BTRFS is
>>>>>>>>>>>>>>>>>>>>>>> > > but when things go wrong it's very hard
>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> me to troubleshoot.
>>>>>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> > >>> Something went wrong:
>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>> > >>> Reinitialize checksum tree
>>>>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1`
>>>>>>>>>>>>>>>>>>>>>>> triggered, value 1
>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
>>>>>>>>>>>>>>>>>>>>>>> > >>> Aborted
>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>> > >> This means no space for extra metadata...
>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>> > >> Anyway the csum tree problem shouldn't
>>>>>>>>>>>>>>>>>>>>>>> be a
>>>>>>>>>>>>>>>>>>>>>>> big thing, you
>>>>>>>>>>>>>>>>>>>>>>> could leave
>>>>>>>>>>>>>>>>>>>>>>> > >> it and call it a day.
>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>> > >> BTW, as long as btrfs check reports no
>>>>>>>>>>>>>>>>>>>>>>> extra
>>>>>>>>>>>>>>>>>>>>>>> problem for the inode
>>>>>>>>>>>>>>>>>>>>>>> > >> generation, it should be pretty safe
>>>>>>>>>>>>>>>>>>>>>>> to use
>>>>>>>>>>>>>>>>>>>>>>> the fs.
>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>> > >> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> > >> Qu
>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>> > >>> I just noticed I have btrfs-progs 5.6
>>>>>>>>>>>>>>>>>>>>>>> installed and 5.6.1 is
>>>>>>>>>>>>>>>>>>>>>>> > >>> available. I'll let that try overnight?
>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu Wenruo
>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler Richmond
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> Thank you for helping. The end
>>>>>>>>>>>>>>>>>>>>>>> result of
>>>>>>>>>>>>>>>>>>>>>>> the scan was:
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [1/7] checking root items
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [2/7] checking extents
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [3/7] checking free space cache
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [4/7] checking fs roots
>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>> Good news is, your fs is still mostly
>>>>>>>>>>>>>>>>>>>>>>> fine.
>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [5/7] checking only csums items
>>>>>>>>>>>>>>>>>>>>>>> (without
>>>>>>>>>>>>>>>>>>>>>>> verifying data)
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>>>>>> 0-69632
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 0-69632 but there
>>>>>>>>>>>>>>>>>>>>>>> is no
>>>>>>>>>>>>>>>>>>>>>>> extent record
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>>>>>> 946692096-946827264
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 946692096-946827264
>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>> there is no extent
>>>>>>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>>>>>> 946831360-947912704
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 946831360-947912704
>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>> there is no extent
>>>>>>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> ERROR: errors found in csum tree
>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>> Only extent tree is corrupted.
>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>> Normally btrfs check --init-csum-tree
>>>>>>>>>>>>>>>>>>>>>>> should be able to
>>>>>>>>>>>>>>>>>>>>>>> handle it.
>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>> But still, please be sure you're
>>>>>>>>>>>>>>>>>>>>>>> using the
>>>>>>>>>>>>>>>>>>>>>>> latest btrfs-progs
>>>>>>>>>>>>>>>>>>>>>>> to fix it.
>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> > >>>> Qu
>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [6/7] checking root refs
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [7/7] checking quota groups skipped
>>>>>>>>>>>>>>>>>>>>>>> (not
>>>>>>>>>>>>>>>>>>>>>>> enabled on this FS)
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> found 44157956026368 bytes used,
>>>>>>>>>>>>>>>>>>>>>>> error(s)
>>>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> total csum bytes: 42038602716
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> total tree bytes: 49688616960
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> total fs tree bytes: 1256427520
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> total extent tree bytes: 1709105152
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> btree space waste bytes: 3172727316
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> file data blocks allocated:
>>>>>>>>>>>>>>>>>>>>>>> 261625653436416
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> referenced 47477768499200
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> What do I need to do to fix all of
>>>>>>>>>>>>>>>>>>>>>>> this?
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu
>>>>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler Richmond
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Well, the repair doesn't look
>>>>>>>>>>>>>>>>>>>>>>> terribly
>>>>>>>>>>>>>>>>>>>>>>> successful.
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> This means there are more
>>>>>>>>>>>>>>>>>>>>>>> problems, not
>>>>>>>>>>>>>>>>>>>>>>> only the hash name
>>>>>>>>>>>>>>>>>>>>>>> mismatch.
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> This means the fs is already
>>>>>>>>>>>>>>>>>>>>>>> corrupted,
>>>>>>>>>>>>>>>>>>>>>>> the name hash is
>>>>>>>>>>>>>>>>>>>>>>> just one
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> unrelated symptom.
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> The only good news is, btrfs-progs
>>>>>>>>>>>>>>>>>>>>>>> abort
>>>>>>>>>>>>>>>>>>>>>>> the transaction,
>>>>>>>>>>>>>>>>>>>>>>> thus no
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> further damage to the fs.
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Please run a plain btrfs-check to
>>>>>>>>>>>>>>>>>>>>>>> show
>>>>>>>>>>>>>>>>>>>>>>> what's the problem
>>>>>>>>>>>>>>>>>>>>>>> first.
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Qu
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: failed to zero log tree: -17
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: attempt to start transaction
>>>>>>>>>>>>>>>>>>>>>>> over already running one
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: reserved space leaked,
>>>>>>>>>>>>>>>>>>>>>>> flag=0x4 bytes_reserved=4096
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted
>>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted
>>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted
>>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted
>>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> What is going on?
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM Tyler
>>>>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> Chris, I had used the correct
>>>>>>>>>>>>>>>>>>>>>>> mountpoint in the command.
>>>>>>>>>>>>>>>>>>>>>>> I just edited
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> it in the email to be
>>>>>>>>>>>>>>>>>>>>>>> /mountpoint for
>>>>>>>>>>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers
>>>>>>>>>>>>>>>>>>>>>>> crossed!
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13 PM Qu
>>>>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler
>>>>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> I looked up this error and it
>>>>>>>>>>>>>>>>>>>>>>> basically says ask a
>>>>>>>>>>>>>>>>>>>>>>> developer to
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> determine if it's a false
>>>>>>>>>>>>>>>>>>>>>>> error or
>>>>>>>>>>>>>>>>>>>>>>> not. I just started
>>>>>>>>>>>>>>>>>>>>>>> getting some
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> slow response times, and
>>>>>>>>>>>>>>>>>>>>>>> looked at
>>>>>>>>>>>>>>>>>>>>>>> the dmesg log to
>>>>>>>>>>>>>>>>>>>>>>> find a ton of
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> these errors.
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical
>>>>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492
>>>>>>>>>>>>>>>>>>>>>>> expect [0,
>>>>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error
>>>>>>>>>>>>>>>>>>>>>>> (device
>>>>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption
>>>>>>>>>>>>>>>>>>>>>>> detected
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical
>>>>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492
>>>>>>>>>>>>>>>>>>>>>>> expect [0,
>>>>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error
>>>>>>>>>>>>>>>>>>>>>>> (device
>>>>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption
>>>>>>>>>>>>>>>>>>>>>>> detected
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical
>>>>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492
>>>>>>>>>>>>>>>>>>>>>>> expect [0,
>>>>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error
>>>>>>>>>>>>>>>>>>>>>>> (device
>>>>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption
>>>>>>>>>>>>>>>>>>>>>>> detected
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> btrfs device stats, however,
>>>>>>>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>>>>>> show any errors.
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Is there anything I should do
>>>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>> this, or should I
>>>>>>>>>>>>>>>>>>>>>>> just continue
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> using my array as normal?
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> This is caused by older kernel
>>>>>>>>>>>>>>>>>>>>>>> underflow inode generation.
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it,
>>>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>>>> btrfs check --repair.
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Or you can go safer, by manually
>>>>>>>>>>>>>>>>>>>>>>> locating the inode
>>>>>>>>>>>>>>>>>>>>>>> using its inode
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> number (1311670), and copy it
>>>>>>>>>>>>>>>>>>>>>>> to some
>>>>>>>>>>>>>>>>>>>>>>> new location using
>>>>>>>>>>>>>>>>>>>>>>> previous
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> working kernel, then delete the
>>>>>>>>>>>>>>>>>>>>>>> old
>>>>>>>>>>>>>>>>>>>>>>> file, copy the new
>>>>>>>>>>>>>>>>>>>>>>> one back to fix it.
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-06 10:09 ` Ferry Toth
@ 2020-11-06 10:24 ` Qu Wenruo
2020-11-06 10:27 ` Qu Wenruo
2020-11-06 10:30 ` Ferry Toth
0 siblings, 2 replies; 56+ messages in thread
From: Qu Wenruo @ 2020-11-06 10:24 UTC (permalink / raw)
To: Ferry Toth, Qu Wenruo, Tyler Richmond; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 49574 bytes --]
On 2020/11/6 下午6:09, Ferry Toth wrote:
> Hi Qu
>
> Op 06-11-2020 om 00:40 schreef Qu Wenruo:
>>
>> On 2020/11/6 上午7:37, Ferry Toth wrote:
>>> Hi
>>>
>>> Op 06-11-2020 om 00:32 schreef Qu Wenruo:
>>>> On 2020/11/6 上午7:12, Ferry Toth wrote:
>>>>> Hi,
>>>>>
>>>>> Op 06-11-2020 om 00:00 schreef Qu Wenruo:
>>>>>> On 2020/11/6 上午4:08, Ferry Toth wrote:
>>>>>>> I am in a similar spot, during updating my distro (Kubuntu), I am
>>>>>>> unable
>>>>>>> to update a certain package. I know which file it is:
>>>>>>>
>>>>>>> ~$ ls -l /usr/share/doc/libatk1.0-data
>>>>>>> ls: kan geen toegang krijgen tot '/usr/share/doc/libatk1.0-data':
>>>>>>> Invoer-/uitvoerfout
>>>>>>>
>>>>>>> This creates the following in journal:
>>>>>>>
>>>>>>> kernel: BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>>> block=1169152675840 slot=1 ino=915987, invalid inode generation: has
>>>>>>> 18446744073709551492 expect [0, 5851353]
>>>>>>> kernel: BTRFS error (device sda2): block=1169152675840 read time
>>>>>>> tree
>>>>>>> block corruption detected
>>>>>>>
>>>>>>> Now, the problem: this file is on my rootfs, which is mounted. apt
>>>>>>> (distribution updated) installed all packages but can't continue
>>>>>>> configuring, because libatk is a dependancy. I can't delete the file
>>>>>>> because of the I/O error. And btrfs check complains (I tried
>>>>>>> running RO)
>>>>>>> because the file system is mounted.
>>>>>>>
>>>>>>> But, on the sunny side, the file system is not RO.
>>>>>>>
>>>>>>> Is there any way to forcefully remove the file? Or do you have a
>>>>>>> recommendation how to proceed?
>>>>>> Newer kernel will reject to even read the item, thus will not be
>>>>>> able to
>>>>>> remove it.
>>>>> That's already the case. (input / output error)
>>>>>> I guess you have to use some distro ISO to fix the fs.
>>>>> And then? btrfs check --repair the disk offline?
>>>> Yep.
>>>>
>>>> You would want the latest btrfs-progs though.
>>> Groovy has 5.7. Would that be good enough? Otherwise will be difficult
>>> to build on/for live usb image.
>> For your particular case, the fix are already in btrfs-progs v5.4.
>>
>> Although newer is always better, just in case you have extent item
>> generation corruption, you may want v5.4.1.
>>
>> So your v5.7 should be good enough.
>>
>> Thanks,
>> Qu
>
> I made a live usb and performed:
>
> btrfs check --repair /dev/sda2
>
> It found errors and fixed them. However, it did not fix the corrupt
> leaf. The file is actually a directory:
>
> ~$ stat /usr/share/doc/libatk1.0-data
> stat: cannot statx '/usr/share/doc/libatk1.0-data': Invoer-/uitvoerfout
>
> in journal:
>
> BTRFS critical (device sda2): corrupt leaf: root=294 block=1169152675840
> slot=1 ino=915987, invalid inode generation: has 18446744073709551492
> expect [0, 5852829]
> BTRFS error (device sda2): block=1169152675840 read time tree block
> corruption detected
>
> So how do I repair this? Am I doing something wrong?
Please provide the following dump:
btrfs ins dump-tree -b 1169152675840 /dev/sda2
Feel free to remove the filenames in the dump.
And 'btrfs check /dev/sda2' output after the repair.
As a workaround, you can use older kernel (v5.2 at most) to temporary
ignore the problem.
Thanks,
Qu
>
>>>> THanks,
>>>> Qu
>>>>>> Thanks,
>>>>>> Qu
>>>>>>
>>>>>>> Linux = 5.6.0-1032-oem
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Ferry
>>>>>>>
>>>>>>> Op 05-11-2020 om 08:19 schreef Qu Wenruo:
>>>>>>>> On 2020/11/5 下午3:01, Tyler Richmond wrote:
>>>>>>>>> Qu,
>>>>>>>>>
>>>>>>>>> I'm wondering, was a fix for this ever implemented?
>>>>>>>> Already implemented the --repair ability in latest btrfs-progs.
>>>>>>>>
>>>>>>>>> I recently added a
>>>>>>>>> new drive to expand the array, and during the rebalance it dropped
>>>>>>>>> itself back to a read only filesystem. I suspect it's related
>>>>>>>>> to the
>>>>>>>>> issues discussed earlier in this thread. Is there anything I can
>>>>>>>>> do to
>>>>>>>>> complete the balance? The error that caused it to drop to read
>>>>>>>>> only is
>>>>>>>>> here: https://pastebin.com/GGYVMaiG
>>>>>>>> Yep, the same cause.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Qu
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Aug 25, 2020 at 9:43 AM Tyler Richmond
>>>>>>>>> <t.d.richmond@gmail.com> wrote:
>>>>>>>>>> Great, glad we got somewhere! I'll look forward to the fix!
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 25, 2020 at 9:38 AM Qu Wenruo
>>>>>>>>>> <quwenruo.btrfs@gmx.com>
>>>>>>>>>> wrote:
>>>>>>>>>>> On 2020/8/25 下午9:30, Tyler Richmond wrote:
>>>>>>>>>>>> Qu,
>>>>>>>>>>>>
>>>>>>>>>>>> The dump of the block is:
>>>>>>>>>>>>
>>>>>>>>>>>> https://pastebin.com/ran85JJv
>>>>>>>>>>>>
>>>>>>>>>>>> I've also completed the btrfs-image, but it's almost 50gb.
>>>>>>>>>>>> What's the
>>>>>>>>>>>> best way to get it to you? Also, does it work with -ss or
>>>>>>>>>>>> are the
>>>>>>>>>>>> original filenames important?
>>>>>>>>>>> 50G is too big for me to even receive.
>>>>>>>>>>>
>>>>>>>>>>> But your dump shows the problem!
>>>>>>>>>>>
>>>>>>>>>>> It's not inode generation, but inode transid, which would affect
>>>>>>>>>>> send.
>>>>>>>>>>>
>>>>>>>>>>> This is not even checked in btrfs-progs, thus no wonder why it
>>>>>>>>>>> doesn't
>>>>>>>>>>> detect them.
>>>>>>>>>>>
>>>>>>>>>>> And copy-pasted kernel message shares the same "generation"
>>>>>>>>>>> word,
>>>>>>>>>>> not
>>>>>>>>>>> using proper transid to show the problem.
>>>>>>>>>>>
>>>>>>>>>>> Your dump really saved the day!
>>>>>>>>>>>
>>>>>>>>>>> The fix for kernel and btrfs-progs would come in next few days.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Qu
>>>>>>>>>>>> Thanks again!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Aug 25, 2020 at 2:37 AM Qu Wenruo
>>>>>>>>>>>> <quwenruo.btrfs@gmx.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> On 2020/8/25 下午1:25, Tyler Richmond wrote:
>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, it's btrfs-progs 5.7. Here is the result of the lowmem
>>>>>>>>>>>>>> check:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://pastebin.com/8Tzx23EX
>>>>>>>>>>>>> That doesn't detect any inode generation problem at all,
>>>>>>>>>>>>> which is
>>>>>>>>>>>>> not a
>>>>>>>>>>>>> good sign.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Would you also pvode the dump for the offending block?
>>>>>>>>>>>>>
>>>>>>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>>>>>
>>>>>>>>>>>>> For this case, would you please provide the tree dump of
>>>>>>>>>>>>> "203510940835840" ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> # btrfs ins dump-tree -b 203510940835840 <device>
>>>>>>>>>>>>>
>>>>>>>>>>>>> And, since btrfs-image can't dump with regular extent tree,
>>>>>>>>>>>>> the
>>>>>>>>>>>>> "-w"
>>>>>>>>>>>>> dump would also help.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Aug 24, 2020 at 4:26 AM Qu Wenruo
>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>> On 2020/8/24 上午10:47, Tyler Richmond wrote:
>>>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Finally finished another repair and captured the output.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> https://pastebin.com/ffcbwvd8
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Does that show you what you need? Or should I still do
>>>>>>>>>>>>>>>> one in
>>>>>>>>>>>>>>>> lowmem mode?
>>>>>>>>>>>>>>> Lowmem mode (no need for --repair) is recommended since
>>>>>>>>>>>>>>> original mode
>>>>>>>>>>>>>>> doesn't detect the inode generation problem.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And it's already btrfs-progs v5.7 right?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> THanks,
>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>> Thanks for your help!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo
>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>> On 2020/8/23 上午10:49, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>> Well, I can guarantee that I didn't create this fs before
>>>>>>>>>>>>>>>>>> 2015 (just
>>>>>>>>>>>>>>>>>> checked the order confirmation from when I bought the
>>>>>>>>>>>>>>>>>> server), but I
>>>>>>>>>>>>>>>>>> may have just used whatever was in the Ubuntu package
>>>>>>>>>>>>>>>>>> manager at the
>>>>>>>>>>>>>>>>>> time. So maybe I don't have a v0 ref?
>>>>>>>>>>>>>>>>> Then btrfs-image shouldn't report that.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> There is an item smaller than any valid btrfs item,
>>>>>>>>>>>>>>>>> normally
>>>>>>>>>>>>>>>>> it means
>>>>>>>>>>>>>>>>> it's a v0 ref.
>>>>>>>>>>>>>>>>> If not, then it could be a bigger problem.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Could you please provide the full btrfs-check output?
>>>>>>>>>>>>>>>>> Also, if possible result from "btrfs check --mode=lowmem"
>>>>>>>>>>>>>>>>> would also help.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Also, if you really go "--repair", then the full output
>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>> also be
>>>>>>>>>>>>>>>>> needed to determine what's going wrong.
>>>>>>>>>>>>>>>>> There is a report about "btrfs check --repair" didn't
>>>>>>>>>>>>>>>>> repair
>>>>>>>>>>>>>>>>> the inode
>>>>>>>>>>>>>>>>> generation, if that's the case we must have a bug then.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo
>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>>>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
>>>>>>>>>>>>>>>>>>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>>>> Is my best bet just to downgrade the kernel and
>>>>>>>>>>>>>>>>>>>>> then try
>>>>>>>>>>>>>>>>>>>>> to delete the
>>>>>>>>>>>>>>>>>>>>> broken files? Or should I rebuild from scratch? Just
>>>>>>>>>>>>>>>>>>>>> don't know
>>>>>>>>>>>>>>>>>>>>> whether it's worth the time to try and figure this
>>>>>>>>>>>>>>>>>>>>> out or
>>>>>>>>>>>>>>>>>>>>> if the
>>>>>>>>>>>>>>>>>>>>> problems stem from the FS being too old and it's
>>>>>>>>>>>>>>>>>>>>> beyond
>>>>>>>>>>>>>>>>>>>>> trying to
>>>>>>>>>>>>>>>>>>>>> repair.
>>>>>>>>>>>>>>>>>>>> All invalid inode generations, should be able to be
>>>>>>>>>>>>>>>>>>>> repaired by latest
>>>>>>>>>>>>>>>>>>>> btrfs-check.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> If not, please provide the btrfs-image dump for us to
>>>>>>>>>>>>>>>>>>>> determine what's
>>>>>>>>>>>>>>>>>>>> going wrong.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond
>>>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> I didn't check dmesg during the btrfs check, but that
>>>>>>>>>>>>>>>>>>>>>> was the only
>>>>>>>>>>>>>>>>>>>>>> output during the rm -f before it was forced
>>>>>>>>>>>>>>>>>>>>>> readonly. I
>>>>>>>>>>>>>>>>>>>>>> just checked
>>>>>>>>>>>>>>>>>>>>>> dmesg for inode generation values, and there are a
>>>>>>>>>>>>>>>>>>>>>> lot
>>>>>>>>>>>>>>>>>>>>>> of them.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> https://pastebin.com/stZdN0ta
>>>>>>>>>>>>>>>>>>>>>> The dmesg output had 990 lines containing inode
>>>>>>>>>>>>>>>>>>>>>> generation.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> However, these were at least later. I tried to do a
>>>>>>>>>>>>>>>>>>>>>> btrfs balance
>>>>>>>>>>>>>>>>>>>>>> -mconvert raid1 and it failed with an I/O error.
>>>>>>>>>>>>>>>>>>>>>> That is
>>>>>>>>>>>>>>>>>>>>>> probably what
>>>>>>>>>>>>>>>>>>>>>> generated these specific errors, but maybe they were
>>>>>>>>>>>>>>>>>>>>>> also happening
>>>>>>>>>>>>>>>>>>>>>> during the btrfs repair.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway
>>>>>>>>>>>>>>>>>>>>>> with:
>>>>>>>>>>>>>>>>>>>>>> ERROR: either extent tree is corrupted or deprecated
>>>>>>>>>>>>>>>>>>>>>> extent ref format
>>>>>>>>>>>>>>>>>>>>>> ERROR: create failed: -5
>>>>>>>>>>>>>>>>>>> Oh, forgot this part.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This means you have v0 ref?!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Then the fs is too old, no progs/kernel support after
>>>>>>>>>>>>>>>>>>> all.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> In that case, please rollback to the last working kernel
>>>>>>>>>>>>>>>>>>> and copy your data.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> In fact, that v0 ref should only be in the code base for
>>>>>>>>>>>>>>>>>>> several weeks
>>>>>>>>>>>>>>>>>>> before 2010, thus it's really too old.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The good news is, with tree-checker, we should never
>>>>>>>>>>>>>>>>>>> experience such
>>>>>>>>>>>>>>>>>>> too-old-to-be-usable problem (at least I hope so)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Sorry to resurrect this thread, but I just ran into
>>>>>>>>>>>>>>>>>>>>>>>> something that I
>>>>>>>>>>>>>>>>>>>>>>>> can't really just ignore. I've found a folder
>>>>>>>>>>>>>>>>>>>>>>>> that is
>>>>>>>>>>>>>>>>>>>>>>>> full of files
>>>>>>>>>>>>>>>>>>>>>>>> which I guess have been broken somehow. I found a
>>>>>>>>>>>>>>>>>>>>>>>> backup and restored
>>>>>>>>>>>>>>>>>>>>>>>> them, but I want to delete this folder of broken
>>>>>>>>>>>>>>>>>>>>>>>> files. But whenever I
>>>>>>>>>>>>>>>>>>>>>>>> try, the fs is forced into readonly mode again. I
>>>>>>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>>>>> finished another
>>>>>>>>>>>>>>>>>>>>>>>> btrfs check --repair but it didn't fix the problem.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> https://pastebin.com/eTV3s3fr
>>>>>>>>>>>>>>>>>>>>>>> Is that the full output?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> No inode generation bugs?
>>>>>>>>>>>>>>>>>>>>>>>> I'm already on btrfs-progs v5.7. Any new
>>>>>>>>>>>>>>>>>>>>>>>> suggestions?
>>>>>>>>>>>>>>>>>>>>>>> Strange.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> The detection and repair should have been merged
>>>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>> v5.5.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> If your fs is small enough, would you please provide
>>>>>>>>>>>>>>>>>>>>>>> the "btrfs-image
>>>>>>>>>>>>>>>>>>>>>>> -c9" dump?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> It would contain the filenames and directories
>>>>>>>>>>>>>>>>>>>>>>> names,
>>>>>>>>>>>>>>>>>>>>>>> but doesn't
>>>>>>>>>>>>>>>>>>>>>>> contain file contents.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond
>>>>>>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 5.6.1 also failed the same way. Here's the
>>>>>>>>>>>>>>>>>>>>>>>> usage
>>>>>>>>>>>>>>>>>>>>>>>> output. This is the
>>>>>>>>>>>>>>>>>>>>>>>> part where you see I've been using RAID5
>>>>>>>>>>>>>>>>>>>>>>>> haha
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> WARNING: RAID56 detected, not implemented
>>>>>>>>>>>>>>>>>>>>>>>> Overall:
>>>>>>>>>>>>>>>>>>>>>>>> Device size: 60.03TiB
>>>>>>>>>>>>>>>>>>>>>>>> Device allocated: 98.06GiB
>>>>>>>>>>>>>>>>>>>>>>>> Device unallocated: 59.93TiB
>>>>>>>>>>>>>>>>>>>>>>>> Device missing: 0.00B
>>>>>>>>>>>>>>>>>>>>>>>> Used: 92.56GiB
>>>>>>>>>>>>>>>>>>>>>>>> Free (estimated): 0.00B
>>>>>>>>>>>>>>>>>>>>>>>> (min: 8.00EiB)
>>>>>>>>>>>>>>>>>>>>>>>> Data ratio: 0.00
>>>>>>>>>>>>>>>>>>>>>>>> Metadata ratio: 2.00
>>>>>>>>>>>>>>>>>>>>>>>> Global reserve: 512.00MiB
>>>>>>>>>>>>>>>>>>>>>>>> (used: 0.00B)
>>>>>>>>>>>>>>>>>>>>>>>> Multiple profiles: no
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB
>>>>>>>>>>>>>>>>>>>>>>>> (99.42%)
>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdh 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdd 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdc 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>> /dev/sde 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Metadata,RAID1: Size:49.00GiB,
>>>>>>>>>>>>>>>>>>>>>>>> Used:46.28GiB
>>>>>>>>>>>>>>>>>>>>>>>> (94.44%)
>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdh 34.00GiB
>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 32.00GiB
>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 32.00GiB
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB
>>>>>>>>>>>>>>>>>>>>>>>> (6.87%)
>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 32.00MiB
>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 32.00MiB
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Unallocated:
>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdh 2.81TiB
>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 2.81TiB
>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 2.81TiB
>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdd 1.03TiB
>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdc 1.03TiB
>>>>>>>>>>>>>>>>>>>>>>>> /dev/sde 1.03TiB
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> > > If this is saying there's no extra
>>>>>>>>>>>>>>>>>>>>>>>> space for
>>>>>>>>>>>>>>>>>>>>>>>> metadata, is that why
>>>>>>>>>>>>>>>>>>>>>>>> > > adding more files often makes the
>>>>>>>>>>>>>>>>>>>>>>>> system hang
>>>>>>>>>>>>>>>>>>>>>>>> for 30-90s? Is there
>>>>>>>>>>>>>>>>>>>>>>>> > > anything I should do about that?
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > I'm not sure about the hang though.
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > It would be nice to give more info to
>>>>>>>>>>>>>>>>>>>>>>>> diagnosis.
>>>>>>>>>>>>>>>>>>>>>>>> > The output of 'btrfs fi usage' is
>>>>>>>>>>>>>>>>>>>>>>>> useful for
>>>>>>>>>>>>>>>>>>>>>>>> space usage problem.
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > But the common idea is, to keep at 1~2 Gi
>>>>>>>>>>>>>>>>>>>>>>>> unallocated (not avaiable
>>>>>>>>>>>>>>>>>>>>>>>> > space in vanilla df command) space for
>>>>>>>>>>>>>>>>>>>>>>>> btrfs.
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> > Qu
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>>>>>>> > > Thank you so much for all of your
>>>>>>>>>>>>>>>>>>>>>>>> help. I
>>>>>>>>>>>>>>>>>>>>>>>> love how flexible BTRFS is
>>>>>>>>>>>>>>>>>>>>>>>> > > but when things go wrong it's very hard
>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>> me to troubleshoot.
>>>>>>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu
>>>>>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> > >>> Something went wrong:
>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>> Reinitialize checksum tree
>>>>>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1`
>>>>>>>>>>>>>>>>>>>>>>>> triggered, value 1
>>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
>>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
>>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
>>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
>>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
>>>>>>>>>>>>>>>>>>>>>>>> > >>> Aborted
>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>> > >> This means no space for extra
>>>>>>>>>>>>>>>>>>>>>>>> metadata...
>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>> > >> Anyway the csum tree problem shouldn't
>>>>>>>>>>>>>>>>>>>>>>>> be a
>>>>>>>>>>>>>>>>>>>>>>>> big thing, you
>>>>>>>>>>>>>>>>>>>>>>>> could leave
>>>>>>>>>>>>>>>>>>>>>>>> > >> it and call it a day.
>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>> > >> BTW, as long as btrfs check reports no
>>>>>>>>>>>>>>>>>>>>>>>> extra
>>>>>>>>>>>>>>>>>>>>>>>> problem for the inode
>>>>>>>>>>>>>>>>>>>>>>>> > >> generation, it should be pretty safe
>>>>>>>>>>>>>>>>>>>>>>>> to use
>>>>>>>>>>>>>>>>>>>>>>>> the fs.
>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>> > >> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> > >> Qu
>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>> I just noticed I have btrfs-progs 5.6
>>>>>>>>>>>>>>>>>>>>>>>> installed and 5.6.1 is
>>>>>>>>>>>>>>>>>>>>>>>> > >>> available. I'll let that try
>>>>>>>>>>>>>>>>>>>>>>>> overnight?
>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu
>>>>>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler
>>>>>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> Thank you for helping. The end
>>>>>>>>>>>>>>>>>>>>>>>> result of
>>>>>>>>>>>>>>>>>>>>>>>> the scan was:
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [1/7] checking root items
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [2/7] checking extents
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [3/7] checking free space cache
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [4/7] checking fs roots
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Good news is, your fs is still
>>>>>>>>>>>>>>>>>>>>>>>> mostly
>>>>>>>>>>>>>>>>>>>>>>>> fine.
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [5/7] checking only csums items
>>>>>>>>>>>>>>>>>>>>>>>> (without
>>>>>>>>>>>>>>>>>>>>>>>> verifying data)
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>>>>>>> 0-69632
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 0-69632 but there
>>>>>>>>>>>>>>>>>>>>>>>> is no
>>>>>>>>>>>>>>>>>>>>>>>> extent record
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>>>>>>> 946692096-946827264
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 946692096-946827264
>>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>> there is no extent
>>>>>>>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>>>>>>> 946831360-947912704
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 946831360-947912704
>>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>> there is no extent
>>>>>>>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> ERROR: errors found in csum tree
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Only extent tree is corrupted.
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Normally btrfs check
>>>>>>>>>>>>>>>>>>>>>>>> --init-csum-tree
>>>>>>>>>>>>>>>>>>>>>>>> should be able to
>>>>>>>>>>>>>>>>>>>>>>>> handle it.
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>> But still, please be sure you're
>>>>>>>>>>>>>>>>>>>>>>>> using the
>>>>>>>>>>>>>>>>>>>>>>>> latest btrfs-progs
>>>>>>>>>>>>>>>>>>>>>>>> to fix it.
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Qu
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [6/7] checking root refs
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [7/7] checking quota groups skipped
>>>>>>>>>>>>>>>>>>>>>>>> (not
>>>>>>>>>>>>>>>>>>>>>>>> enabled on this FS)
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> found 44157956026368 bytes used,
>>>>>>>>>>>>>>>>>>>>>>>> error(s)
>>>>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> total csum bytes: 42038602716
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> total tree bytes: 49688616960
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> total fs tree bytes: 1256427520
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> total extent tree bytes: 1709105152
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> btree space waste bytes: 3172727316
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> file data blocks allocated:
>>>>>>>>>>>>>>>>>>>>>>>> 261625653436416
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> referenced 47477768499200
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> What do I need to do to fix all of
>>>>>>>>>>>>>>>>>>>>>>>> this?
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu
>>>>>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler
>>>>>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Well, the repair doesn't look
>>>>>>>>>>>>>>>>>>>>>>>> terribly
>>>>>>>>>>>>>>>>>>>>>>>> successful.
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> This means there are more
>>>>>>>>>>>>>>>>>>>>>>>> problems, not
>>>>>>>>>>>>>>>>>>>>>>>> only the hash name
>>>>>>>>>>>>>>>>>>>>>>>> mismatch.
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> This means the fs is already
>>>>>>>>>>>>>>>>>>>>>>>> corrupted,
>>>>>>>>>>>>>>>>>>>>>>>> the name hash is
>>>>>>>>>>>>>>>>>>>>>>>> just one
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> unrelated symptom.
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> The only good news is, btrfs-progs
>>>>>>>>>>>>>>>>>>>>>>>> abort
>>>>>>>>>>>>>>>>>>>>>>>> the transaction,
>>>>>>>>>>>>>>>>>>>>>>>> thus no
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> further damage to the fs.
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Please run a plain btrfs-check to
>>>>>>>>>>>>>>>>>>>>>>>> show
>>>>>>>>>>>>>>>>>>>>>>>> what's the problem
>>>>>>>>>>>>>>>>>>>>>>>> first.
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Qu
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: failed to zero log
>>>>>>>>>>>>>>>>>>>>>>>> tree: -17
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: attempt to start
>>>>>>>>>>>>>>>>>>>>>>>> transaction
>>>>>>>>>>>>>>>>>>>>>>>> over already running one
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: reserved space leaked,
>>>>>>>>>>>>>>>>>>>>>>>> flag=0x4 bytes_reserved=4096
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted
>>>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted
>>>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted
>>>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted
>>>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> What is going on?
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM
>>>>>>>>>>>>>>>>>>>>>>>> Tyler
>>>>>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> Chris, I had used the correct
>>>>>>>>>>>>>>>>>>>>>>>> mountpoint in the command.
>>>>>>>>>>>>>>>>>>>>>>>> I just edited
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> it in the email to be
>>>>>>>>>>>>>>>>>>>>>>>> /mountpoint for
>>>>>>>>>>>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers
>>>>>>>>>>>>>>>>>>>>>>>> crossed!
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13
>>>>>>>>>>>>>>>>>>>>>>>> PM Qu
>>>>>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler
>>>>>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> I looked up this error and it
>>>>>>>>>>>>>>>>>>>>>>>> basically says ask a
>>>>>>>>>>>>>>>>>>>>>>>> developer to
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> determine if it's a false
>>>>>>>>>>>>>>>>>>>>>>>> error or
>>>>>>>>>>>>>>>>>>>>>>>> not. I just started
>>>>>>>>>>>>>>>>>>>>>>>> getting some
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> slow response times, and
>>>>>>>>>>>>>>>>>>>>>>>> looked at
>>>>>>>>>>>>>>>>>>>>>>>> the dmesg log to
>>>>>>>>>>>>>>>>>>>>>>>> find a ton of
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> these errors.
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical
>>>>>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492
>>>>>>>>>>>>>>>>>>>>>>>> expect [0,
>>>>>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error
>>>>>>>>>>>>>>>>>>>>>>>> (device
>>>>>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption
>>>>>>>>>>>>>>>>>>>>>>>> detected
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical
>>>>>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492
>>>>>>>>>>>>>>>>>>>>>>>> expect [0,
>>>>>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error
>>>>>>>>>>>>>>>>>>>>>>>> (device
>>>>>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption
>>>>>>>>>>>>>>>>>>>>>>>> detected
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical
>>>>>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492
>>>>>>>>>>>>>>>>>>>>>>>> expect [0,
>>>>>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error
>>>>>>>>>>>>>>>>>>>>>>>> (device
>>>>>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption
>>>>>>>>>>>>>>>>>>>>>>>> detected
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> btrfs device stats, however,
>>>>>>>>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>>>>>>> show any errors.
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Is there anything I should do
>>>>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>> this, or should I
>>>>>>>>>>>>>>>>>>>>>>>> just continue
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> using my array as normal?
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> This is caused by older kernel
>>>>>>>>>>>>>>>>>>>>>>>> underflow inode generation.
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it,
>>>>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>>>>> btrfs check --repair.
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Or you can go safer, by
>>>>>>>>>>>>>>>>>>>>>>>> manually
>>>>>>>>>>>>>>>>>>>>>>>> locating the inode
>>>>>>>>>>>>>>>>>>>>>>>> using its inode
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> number (1311670), and copy it
>>>>>>>>>>>>>>>>>>>>>>>> to some
>>>>>>>>>>>>>>>>>>>>>>>> new location using
>>>>>>>>>>>>>>>>>>>>>>>> previous
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> working kernel, then delete the
>>>>>>>>>>>>>>>>>>>>>>>> old
>>>>>>>>>>>>>>>>>>>>>>>> file, copy the new
>>>>>>>>>>>>>>>>>>>>>>>> one back to fix it.
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-06 10:24 ` Qu Wenruo
@ 2020-11-06 10:27 ` Qu Wenruo
2020-11-06 10:32 ` Ferry Toth
2020-11-06 10:30 ` Ferry Toth
1 sibling, 1 reply; 56+ messages in thread
From: Qu Wenruo @ 2020-11-06 10:27 UTC (permalink / raw)
To: Ferry Toth, Qu Wenruo, Tyler Richmond; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 1303 bytes --]
>> BTRFS critical (device sda2): corrupt leaf: root=294 block=1169152675840
>> slot=1 ino=915987, invalid inode generation: has 18446744073709551492
>> expect [0, 5852829]
>> BTRFS error (device sda2): block=1169152675840 read time tree block
>> corruption detected
>>
>> So how do I repair this? Am I doing something wrong?
>
> Please provide the following dump:
> btrfs ins dump-tree -b 1169152675840 /dev/sda2
>
> Feel free to remove the filenames in the dump.
>
> And 'btrfs check /dev/sda2' output after the repair.
>
> As a workaround, you can use older kernel (v5.2 at most) to temporary
> ignore the problem.
>
> Thanks,
> Qu
>
>>
>>>>> THanks,
>>>>> Qu
>>>>>>> Thanks,
>>>>>>> Qu
>>>>>>>
>>>>>>>> Linux = 5.6.0-1032-oem
Wait, it's just v5.6??
Then that means, the error message can be wrong. Both transid and inode
generation error could be output as "inode generation error".
And to repair inode transid, the repair ability is not yet merged into
upstream btrfs-progs.
You can use the out-of-tree branch to repair it:
https://github.com/adam900710/btrfs-progs/tree/inode_transid
I'm afraid you have to build the btrfs-progs in liveUSB environment to
repair it...
I need to re-push the branch to make it into upstream.
Thanks,
Qu
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-06 10:24 ` Qu Wenruo
2020-11-06 10:27 ` Qu Wenruo
@ 2020-11-06 10:30 ` Ferry Toth
2020-11-06 10:32 ` Qu Wenruo
2020-11-06 11:28 ` Ferry Toth
1 sibling, 2 replies; 56+ messages in thread
From: Ferry Toth @ 2020-11-06 10:30 UTC (permalink / raw)
To: Qu Wenruo, Qu Wenruo, Tyler Richmond; +Cc: Btrfs BTRFS
Hi
Op 06-11-2020 om 11:24 schreef Qu Wenruo:
>
> On 2020/11/6 下午6:09, Ferry Toth wrote:
>> Hi Qu
>>
>> Op 06-11-2020 om 00:40 schreef Qu Wenruo:
>>> On 2020/11/6 上午7:37, Ferry Toth wrote:
>>>> Hi
>>>>
>>>> Op 06-11-2020 om 00:32 schreef Qu Wenruo:
>>>>> On 2020/11/6 上午7:12, Ferry Toth wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Op 06-11-2020 om 00:00 schreef Qu Wenruo:
>>>>>>> On 2020/11/6 上午4:08, Ferry Toth wrote:
>>>>>>>> I am in a similar spot, during updating my distro (Kubuntu), I am
>>>>>>>> unable
>>>>>>>> to update a certain package. I know which file it is:
>>>>>>>>
>>>>>>>> ~$ ls -l /usr/share/doc/libatk1.0-data
>>>>>>>> ls: kan geen toegang krijgen tot '/usr/share/doc/libatk1.0-data':
>>>>>>>> Invoer-/uitvoerfout
>>>>>>>>
>>>>>>>> This creates the following in journal:
>>>>>>>>
>>>>>>>> kernel: BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>>>> block=1169152675840 slot=1 ino=915987, invalid inode generation: has
>>>>>>>> 18446744073709551492 expect [0, 5851353]
>>>>>>>> kernel: BTRFS error (device sda2): block=1169152675840 read time
>>>>>>>> tree
>>>>>>>> block corruption detected
>>>>>>>>
>>>>>>>> Now, the problem: this file is on my rootfs, which is mounted. apt
>>>>>>>> (distribution updated) installed all packages but can't continue
>>>>>>>> configuring, because libatk is a dependancy. I can't delete the file
>>>>>>>> because of the I/O error. And btrfs check complains (I tried
>>>>>>>> running RO)
>>>>>>>> because the file system is mounted.
>>>>>>>>
>>>>>>>> But, on the sunny side, the file system is not RO.
>>>>>>>>
>>>>>>>> Is there any way to forcefully remove the file? Or do you have a
>>>>>>>> recommendation how to proceed?
>>>>>>> Newer kernel will reject to even read the item, thus will not be
>>>>>>> able to
>>>>>>> remove it.
>>>>>> That's already the case. (input / output error)
>>>>>>> I guess you have to use some distro ISO to fix the fs.
>>>>>> And then? btrfs check --repair the disk offline?
>>>>> Yep.
>>>>>
>>>>> You would want the latest btrfs-progs though.
>>>> Groovy has 5.7. Would that be good enough? Otherwise will be difficult
>>>> to build on/for live usb image.
>>> For your particular case, the fix are already in btrfs-progs v5.4.
>>>
>>> Although newer is always better, just in case you have extent item
>>> generation corruption, you may want v5.4.1.
>>>
>>> So your v5.7 should be good enough.
>>>
>>> Thanks,
>>> Qu
>> I made a live usb and performed:
>>
>> btrfs check --repair /dev/sda2
>>
>> It found errors and fixed them. However, it did not fix the corrupt
>> leaf. The file is actually a directory:
>>
>> ~$ stat /usr/share/doc/libatk1.0-data
>> stat: cannot statx '/usr/share/doc/libatk1.0-data': Invoer-/uitvoerfout
>>
>> in journal:
>>
>> BTRFS critical (device sda2): corrupt leaf: root=294 block=1169152675840
>> slot=1 ino=915987, invalid inode generation: has 18446744073709551492
>> expect [0, 5852829]
>> BTRFS error (device sda2): block=1169152675840 read time tree block
>> corruption detected
>>
>> So how do I repair this? Am I doing something wrong?
> Please provide the following dump:
> btrfs ins dump-tree -b 1169152675840 /dev/sda2
>
> Feel free to remove the filenames in the dump.
sudo btrfs ins dump-tree -b 1169152675840 /dev/sda2
btrfs-progs v5.3-rc1
leaf 1169152675840 items 36 free space 966 generation 5431733 owner 294
leaf 1169152675840 flags 0x1(WRITTEN) backref revision 1
fs uuid 27155120-9ef8-47fb-b248-eaac2b7c8375
chunk uuid 5704f1ba-08fd-4f6b-9117-0e080b4e9ef0
item 0 key (915986 DIR_INDEX 2) itemoff 3957 itemsize 38
location key (915987 INODE_ITEM 0) type FILE
transid 7782235549259005952 data_len 0 name_len 8
name: smb.conf
item 1 key (915987 INODE_ITEM 0) itemoff 3797 itemsize 160
generation 1 transid 18446744073709551492 size 12464
nbytes 16384
block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
sequence 0 flags 0x0(none)
atime 1350489744.0 (2012-10-17 18:02:24)
ctime 1353328654.0 (2012-11-19 13:37:34)
mtime 1350489744.0 (2012-10-17 18:02:24)
otime 0.0 (1970-01-01 01:00:00)
item 2 key (915987 INODE_REF 915986) itemoff 3779 itemsize 18
index 2 namelen 8 name: smb.conf
item 3 key (915987 EXTENT_DATA 0) itemoff 3726 itemsize 53
generation 18 type 1 (regular)
extent data disk byte 1110664871936 nr 16384
extent data offset 0 nr 16384 ram 16384
extent compression 0 (none)
item 4 key (915989 INODE_ITEM 0) itemoff 3566 itemsize 160
generation 1 transid 5431733 size 56 nbytes 0
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 160 flags 0x0(none)
atime 1571487761.716511096 (2019-10-19 14:22:41)
ctime 1588235951.582838139 (2020-04-30 10:39:11)
mtime 1588235951.582838139 (2020-04-30 10:39:11)
otime 0.0 (1970-01-01 01:00:00)
item 5 key (915989 INODE_REF 659081) itemoff 3546 itemsize 20
index 1101 namelen 10 name: libassuan0
item 6 key (915989 DIR_ITEM 653215628) itemoff 3497 itemsize 49
location key (72402032 INODE_ITEM 0) type FILE
transid 5431729 data_len 0 name_len 19
name: changelog.Debian.gz
item 7 key (915989 DIR_ITEM 1600214284) itemoff 3458 itemsize 39
location key (72402033 INODE_ITEM 0) type FILE
transid 5431729 data_len 0 name_len 9
name: copyright
item 8 key (915989 DIR_INDEX 62) itemoff 3409 itemsize 49
location key (72402032 INODE_ITEM 0) type FILE
transid 5431729 data_len 0 name_len 19
name: changelog.Debian.gz
item 9 key (915989 DIR_INDEX 63) itemoff 3370 itemsize 39
location key (72402033 INODE_ITEM 0) type FILE
transid 5431729 data_len 0 name_len 9
name: copyright
item 10 key (915990 INODE_ITEM 0) itemoff 3210 itemsize 160
generation 1 transid 4933124 size 56 nbytes 0
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 96 flags 0x0(none)
atime 1571487761.720511096 (2019-10-19 14:22:41)
ctime 1477470910.249847328 (2016-10-26 10:35:10)
mtime 1477470910.249847328 (2016-10-26 10:35:10)
otime 0.0 (1970-01-01 01:00:00)
item 11 key (915990 INODE_REF 659081) itemoff 3189 itemsize 21
index 1343 namelen 11 name: libasyncns0
item 12 key (915990 DIR_ITEM 653215628) itemoff 3140 itemsize 49
location key (25762857 INODE_ITEM 0) type FILE
transid 2068663 data_len 0 name_len 19
name: changelog.Debian.gz
item 13 key (915990 DIR_ITEM 1600214284) itemoff 3101 itemsize 39
location key (25762858 INODE_ITEM 0) type FILE
transid 2068663 data_len 0 name_len 9
name: copyright
item 14 key (915990 DIR_INDEX 38) itemoff 3052 itemsize 49
location key (25762857 INODE_ITEM 0) type FILE
transid 2068663 data_len 0 name_len 19
name: changelog.Debian.gz
item 15 key (915990 DIR_INDEX 39) itemoff 3013 itemsize 39
location key (25762858 INODE_ITEM 0) type FILE
transid 2068663 data_len 0 name_len 9
name: copyright
item 16 key (915991 INODE_ITEM 0) itemoff 2853 itemsize 160
generation 1 transid 4933124 size 68 nbytes 0
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 48 flags 0x0(none)
atime 1571487761.720511096 (2019-10-19 14:22:41)
ctime 1540813807.330187853 (2018-10-29 12:50:07)
mtime 1540813807.330187853 (2018-10-29 12:50:07)
otime 0.0 (1970-01-01 01:00:00)
item 17 key (915991 INODE_REF 659081) itemoff 2831 itemsize 22
index 1545 namelen 12 name: libatasmart4
item 18 key (915991 DIR_ITEM 653215628) itemoff 2782 itemsize 49
location key (52273681 INODE_ITEM 0) type FILE
transid 4036682 data_len 0 name_len 19
name: changelog.Debian.gz
item 19 key (915991 DIR_ITEM 1600214284) itemoff 2743 itemsize 39
location key (52273682 INODE_ITEM 0) type FILE
transid 4036682 data_len 0 name_len 9
name: copyright
item 20 key (915991 DIR_ITEM 3650993379) itemoff 2707 itemsize 36
location key (52273680 INODE_ITEM 0) type FILE
transid 4036682 data_len 0 name_len 6
name: README
item 21 key (915991 DIR_INDEX 20) itemoff 2671 itemsize 36
location key (52273680 INODE_ITEM 0) type FILE
transid 4036682 data_len 0 name_len 6
name: README
item 22 key (915991 DIR_INDEX 21) itemoff 2622 itemsize 49
location key (52273681 INODE_ITEM 0) type FILE
transid 4036682 data_len 0 name_len 19
name: changelog.Debian.gz
item 23 key (915991 DIR_INDEX 22) itemoff 2583 itemsize 39
location key (52273682 INODE_ITEM 0) type FILE
transid 4036682 data_len 0 name_len 9
name: copyright
item 24 key (915992 INODE_ITEM 0) itemoff 2423 itemsize 160
generation 1 transid 5431717 size 56 nbytes 0
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 611 flags 0x0(none)
atime 1571487761.728511097 (2019-10-19 14:22:41)
ctime 1588235951.295254395 (2020-04-30 10:39:11)
mtime 1588235951.295254395 (2020-04-30 10:39:11)
otime 0.0 (1970-01-01 01:00:00)
item 25 key (915992 INODE_REF 659081) itemoff 2402 itemsize 21
index 2141 namelen 11 name: libatk1.0-0
item 26 key (915992 DIR_ITEM 653215628) itemoff 2353 itemsize 49
location key (72401999 INODE_ITEM 0) type FILE
transid 5431713 data_len 0 name_len 19
name: changelog.Debian.gz
item 27 key (915992 DIR_ITEM 1600214284) itemoff 2314 itemsize 39
location key (72402000 INODE_ITEM 0) type FILE
transid 5431713 data_len 0 name_len 9
name: copyright
item 28 key (915992 DIR_INDEX 233) itemoff 2265 itemsize 49
location key (72401999 INODE_ITEM 0) type FILE
transid 5431713 data_len 0 name_len 19
name: changelog.Debian.gz
item 29 key (915992 DIR_INDEX 234) itemoff 2226 itemsize 39
location key (72402000 INODE_ITEM 0) type FILE
transid 5431713 data_len 0 name_len 9
name: copyright
item 30 key (915993 INODE_ITEM 0) itemoff 2066 itemsize 160
generation 1 transid 5431708 size 56 nbytes 0
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 315 flags 0x0(none)
atime 1571487761.720511096 (2019-10-19 14:22:41)
ctime 1588235951.95543462 (2020-04-30 10:39:11)
mtime 1588235951.95543462 (2020-04-30 10:39:11)
otime 0.0 (1970-01-01 01:00:00)
item 31 key (915993 INODE_REF 659081) itemoff 2042 itemsize 24
index 1639 namelen 14 name: libatk1.0-data
item 32 key (915993 DIR_ITEM 653215628) itemoff 1993 itemsize 49
location key (72401982 INODE_ITEM 0) type FILE
transid 5431704 data_len 0 name_len 19
name: changelog.Debian.gz
item 33 key (915993 DIR_ITEM 1600214284) itemoff 1954 itemsize 39
location key (72401983 INODE_ITEM 0) type FILE
transid 5431704 data_len 0 name_len 9
name: copyright
item 34 key (915993 DIR_INDEX 122) itemoff 1905 itemsize 49
location key (72401982 INODE_ITEM 0) type FILE
transid 5431704 data_len 0 name_len 19
name: changelog.Debian.gz
item 35 key (915993 DIR_INDEX 123) itemoff 1866 itemsize 39
location key (72401983 INODE_ITEM 0) type FILE
transid 5431704 data_len 0 name_len 9
name: copyright
> And 'btrfs check /dev/sda2' output after the repair.
There were no errors after the repair.
> As a workaround, you can use older kernel (v5.2 at most) to temporary
> ignore the problem.
My collegue had the brilliant idea to move /usr/share/doc to
/usr/share/dog, then cp back dog to doc.
So at least I was now able to finish the dist-upgrade.
Now question remains how to clean up /usr/share/dog
> Thanks,
> Qu
>
>>>>> THanks,
>>>>> Qu
>>>>>>> Thanks,
>>>>>>> Qu
>>>>>>>
>>>>>>>> Linux = 5.6.0-1032-oem
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Ferry
>>>>>>>>
>>>>>>>> Op 05-11-2020 om 08:19 schreef Qu Wenruo:
>>>>>>>>> On 2020/11/5 下午3:01, Tyler Richmond wrote:
>>>>>>>>>> Qu,
>>>>>>>>>>
>>>>>>>>>> I'm wondering, was a fix for this ever implemented?
>>>>>>>>> Already implemented the --repair ability in latest btrfs-progs.
>>>>>>>>>
>>>>>>>>>> I recently added a
>>>>>>>>>> new drive to expand the array, and during the rebalance it dropped
>>>>>>>>>> itself back to a read only filesystem. I suspect it's related
>>>>>>>>>> to the
>>>>>>>>>> issues discussed earlier in this thread. Is there anything I can
>>>>>>>>>> do to
>>>>>>>>>> complete the balance? The error that caused it to drop to read
>>>>>>>>>> only is
>>>>>>>>>> here: https://pastebin.com/GGYVMaiG
>>>>>>>>> Yep, the same cause.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Qu
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 25, 2020 at 9:43 AM Tyler Richmond
>>>>>>>>>> <t.d.richmond@gmail.com> wrote:
>>>>>>>>>>> Great, glad we got somewhere! I'll look forward to the fix!
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 25, 2020 at 9:38 AM Qu Wenruo
>>>>>>>>>>> <quwenruo.btrfs@gmx.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> On 2020/8/25 下午9:30, Tyler Richmond wrote:
>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>
>>>>>>>>>>>>> The dump of the block is:
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://pastebin.com/ran85JJv
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've also completed the btrfs-image, but it's almost 50gb.
>>>>>>>>>>>>> What's the
>>>>>>>>>>>>> best way to get it to you? Also, does it work with -ss or
>>>>>>>>>>>>> are the
>>>>>>>>>>>>> original filenames important?
>>>>>>>>>>>> 50G is too big for me to even receive.
>>>>>>>>>>>>
>>>>>>>>>>>> But your dump shows the problem!
>>>>>>>>>>>>
>>>>>>>>>>>> It's not inode generation, but inode transid, which would affect
>>>>>>>>>>>> send.
>>>>>>>>>>>>
>>>>>>>>>>>> This is not even checked in btrfs-progs, thus no wonder why it
>>>>>>>>>>>> doesn't
>>>>>>>>>>>> detect them.
>>>>>>>>>>>>
>>>>>>>>>>>> And copy-pasted kernel message shares the same "generation"
>>>>>>>>>>>> word,
>>>>>>>>>>>> not
>>>>>>>>>>>> using proper transid to show the problem.
>>>>>>>>>>>>
>>>>>>>>>>>> Your dump really saved the day!
>>>>>>>>>>>>
>>>>>>>>>>>> The fix for kernel and btrfs-progs would come in next few days.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Qu
>>>>>>>>>>>>> Thanks again!
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 25, 2020 at 2:37 AM Qu Wenruo
>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> On 2020/8/25 下午1:25, Tyler Richmond wrote:
>>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes, it's btrfs-progs 5.7. Here is the result of the lowmem
>>>>>>>>>>>>>>> check:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://pastebin.com/8Tzx23EX
>>>>>>>>>>>>>> That doesn't detect any inode generation problem at all,
>>>>>>>>>>>>>> which is
>>>>>>>>>>>>>> not a
>>>>>>>>>>>>>> good sign.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Would you also pvode the dump for the offending block?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For this case, would you please provide the tree dump of
>>>>>>>>>>>>>> "203510940835840" ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> # btrfs ins dump-tree -b 203510940835840 <device>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And, since btrfs-image can't dump with regular extent tree,
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> "-w"
>>>>>>>>>>>>>> dump would also help.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Aug 24, 2020 at 4:26 AM Qu Wenruo
>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>> On 2020/8/24 上午10:47, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Finally finished another repair and captured the output.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> https://pastebin.com/ffcbwvd8
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Does that show you what you need? Or should I still do
>>>>>>>>>>>>>>>>> one in
>>>>>>>>>>>>>>>>> lowmem mode?
>>>>>>>>>>>>>>>> Lowmem mode (no need for --repair) is recommended since
>>>>>>>>>>>>>>>> original mode
>>>>>>>>>>>>>>>> doesn't detect the inode generation problem.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> And it's already btrfs-progs v5.7 right?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> THanks,
>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>> Thanks for your help!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo
>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>>> On 2020/8/23 上午10:49, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>> Well, I can guarantee that I didn't create this fs before
>>>>>>>>>>>>>>>>>>> 2015 (just
>>>>>>>>>>>>>>>>>>> checked the order confirmation from when I bought the
>>>>>>>>>>>>>>>>>>> server), but I
>>>>>>>>>>>>>>>>>>> may have just used whatever was in the Ubuntu package
>>>>>>>>>>>>>>>>>>> manager at the
>>>>>>>>>>>>>>>>>>> time. So maybe I don't have a v0 ref?
>>>>>>>>>>>>>>>>>> Then btrfs-image shouldn't report that.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> There is an item smaller than any valid btrfs item,
>>>>>>>>>>>>>>>>>> normally
>>>>>>>>>>>>>>>>>> it means
>>>>>>>>>>>>>>>>>> it's a v0 ref.
>>>>>>>>>>>>>>>>>> If not, then it could be a bigger problem.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Could you please provide the full btrfs-check output?
>>>>>>>>>>>>>>>>>> Also, if possible result from "btrfs check --mode=lowmem"
>>>>>>>>>>>>>>>>>> would also help.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Also, if you really go "--repair", then the full output
>>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>> also be
>>>>>>>>>>>>>>>>>> needed to determine what's going wrong.
>>>>>>>>>>>>>>>>>> There is a report about "btrfs check --repair" didn't
>>>>>>>>>>>>>>>>>> repair
>>>>>>>>>>>>>>>>>> the inode
>>>>>>>>>>>>>>>>>> generation, if that's the case we must have a bug then.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo
>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>>>>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
>>>>>>>>>>>>>>>>>>>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>>>>> Is my best bet just to downgrade the kernel and
>>>>>>>>>>>>>>>>>>>>>> then try
>>>>>>>>>>>>>>>>>>>>>> to delete the
>>>>>>>>>>>>>>>>>>>>>> broken files? Or should I rebuild from scratch? Just
>>>>>>>>>>>>>>>>>>>>>> don't know
>>>>>>>>>>>>>>>>>>>>>> whether it's worth the time to try and figure this
>>>>>>>>>>>>>>>>>>>>>> out or
>>>>>>>>>>>>>>>>>>>>>> if the
>>>>>>>>>>>>>>>>>>>>>> problems stem from the FS being too old and it's
>>>>>>>>>>>>>>>>>>>>>> beyond
>>>>>>>>>>>>>>>>>>>>>> trying to
>>>>>>>>>>>>>>>>>>>>>> repair.
>>>>>>>>>>>>>>>>>>>>> All invalid inode generations, should be able to be
>>>>>>>>>>>>>>>>>>>>> repaired by latest
>>>>>>>>>>>>>>>>>>>>> btrfs-check.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> If not, please provide the btrfs-image dump for us to
>>>>>>>>>>>>>>>>>>>>> determine what's
>>>>>>>>>>>>>>>>>>>>> going wrong.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond
>>>>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> I didn't check dmesg during the btrfs check, but that
>>>>>>>>>>>>>>>>>>>>>>> was the only
>>>>>>>>>>>>>>>>>>>>>>> output during the rm -f before it was forced
>>>>>>>>>>>>>>>>>>>>>>> readonly. I
>>>>>>>>>>>>>>>>>>>>>>> just checked
>>>>>>>>>>>>>>>>>>>>>>> dmesg for inode generation values, and there are a
>>>>>>>>>>>>>>>>>>>>>>> lot
>>>>>>>>>>>>>>>>>>>>>>> of them.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> https://pastebin.com/stZdN0ta
>>>>>>>>>>>>>>>>>>>>>>> The dmesg output had 990 lines containing inode
>>>>>>>>>>>>>>>>>>>>>>> generation.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> However, these were at least later. I tried to do a
>>>>>>>>>>>>>>>>>>>>>>> btrfs balance
>>>>>>>>>>>>>>>>>>>>>>> -mconvert raid1 and it failed with an I/O error.
>>>>>>>>>>>>>>>>>>>>>>> That is
>>>>>>>>>>>>>>>>>>>>>>> probably what
>>>>>>>>>>>>>>>>>>>>>>> generated these specific errors, but maybe they were
>>>>>>>>>>>>>>>>>>>>>>> also happening
>>>>>>>>>>>>>>>>>>>>>>> during the btrfs repair.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> The FS is ~45TB, but the btrfs-image -c9 failed anway
>>>>>>>>>>>>>>>>>>>>>>> with:
>>>>>>>>>>>>>>>>>>>>>>> ERROR: either extent tree is corrupted or deprecated
>>>>>>>>>>>>>>>>>>>>>>> extent ref format
>>>>>>>>>>>>>>>>>>>>>>> ERROR: create failed: -5
>>>>>>>>>>>>>>>>>>>> Oh, forgot this part.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This means you have v0 ref?!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Then the fs is too old, no progs/kernel support after
>>>>>>>>>>>>>>>>>>>> all.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> In that case, please rollback to the last working kernel
>>>>>>>>>>>>>>>>>>>> and copy your data.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> In fact, that v0 ref should only be in the code base for
>>>>>>>>>>>>>>>>>>>> several weeks
>>>>>>>>>>>>>>>>>>>> before 2010, thus it's really too old.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The good news is, with tree-checker, we should never
>>>>>>>>>>>>>>>>>>>> experience such
>>>>>>>>>>>>>>>>>>>> too-old-to-be-usable problem (at least I hope so)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Sorry to resurrect this thread, but I just ran into
>>>>>>>>>>>>>>>>>>>>>>>>> something that I
>>>>>>>>>>>>>>>>>>>>>>>>> can't really just ignore. I've found a folder
>>>>>>>>>>>>>>>>>>>>>>>>> that is
>>>>>>>>>>>>>>>>>>>>>>>>> full of files
>>>>>>>>>>>>>>>>>>>>>>>>> which I guess have been broken somehow. I found a
>>>>>>>>>>>>>>>>>>>>>>>>> backup and restored
>>>>>>>>>>>>>>>>>>>>>>>>> them, but I want to delete this folder of broken
>>>>>>>>>>>>>>>>>>>>>>>>> files. But whenever I
>>>>>>>>>>>>>>>>>>>>>>>>> try, the fs is forced into readonly mode again. I
>>>>>>>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>>>>>> finished another
>>>>>>>>>>>>>>>>>>>>>>>>> btrfs check --repair but it didn't fix the problem.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> https://pastebin.com/eTV3s3fr
>>>>>>>>>>>>>>>>>>>>>>>> Is that the full output?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> No inode generation bugs?
>>>>>>>>>>>>>>>>>>>>>>>>> I'm already on btrfs-progs v5.7. Any new
>>>>>>>>>>>>>>>>>>>>>>>>> suggestions?
>>>>>>>>>>>>>>>>>>>>>>>> Strange.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> The detection and repair should have been merged
>>>>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>> v5.5.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> If your fs is small enough, would you please provide
>>>>>>>>>>>>>>>>>>>>>>>> the "btrfs-image
>>>>>>>>>>>>>>>>>>>>>>>> -c9" dump?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> It would contain the filenames and directories
>>>>>>>>>>>>>>>>>>>>>>>> names,
>>>>>>>>>>>>>>>>>>>>>>>> but doesn't
>>>>>>>>>>>>>>>>>>>>>>>> contain file contents.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond
>>>>>>>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 5.6.1 also failed the same way. Here's the
>>>>>>>>>>>>>>>>>>>>>>>>> usage
>>>>>>>>>>>>>>>>>>>>>>>>> output. This is the
>>>>>>>>>>>>>>>>>>>>>>>>> part where you see I've been using RAID5
>>>>>>>>>>>>>>>>>>>>>>>>> haha
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> WARNING: RAID56 detected, not implemented
>>>>>>>>>>>>>>>>>>>>>>>>> Overall:
>>>>>>>>>>>>>>>>>>>>>>>>> Device size: 60.03TiB
>>>>>>>>>>>>>>>>>>>>>>>>> Device allocated: 98.06GiB
>>>>>>>>>>>>>>>>>>>>>>>>> Device unallocated: 59.93TiB
>>>>>>>>>>>>>>>>>>>>>>>>> Device missing: 0.00B
>>>>>>>>>>>>>>>>>>>>>>>>> Used: 92.56GiB
>>>>>>>>>>>>>>>>>>>>>>>>> Free (estimated): 0.00B
>>>>>>>>>>>>>>>>>>>>>>>>> (min: 8.00EiB)
>>>>>>>>>>>>>>>>>>>>>>>>> Data ratio: 0.00
>>>>>>>>>>>>>>>>>>>>>>>>> Metadata ratio: 2.00
>>>>>>>>>>>>>>>>>>>>>>>>> Global reserve: 512.00MiB
>>>>>>>>>>>>>>>>>>>>>>>>> (used: 0.00B)
>>>>>>>>>>>>>>>>>>>>>>>>> Multiple profiles: no
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB
>>>>>>>>>>>>>>>>>>>>>>>>> (99.42%)
>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdh 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdd 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdc 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sde 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Metadata,RAID1: Size:49.00GiB,
>>>>>>>>>>>>>>>>>>>>>>>>> Used:46.28GiB
>>>>>>>>>>>>>>>>>>>>>>>>> (94.44%)
>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdh 34.00GiB
>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 32.00GiB
>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 32.00GiB
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB
>>>>>>>>>>>>>>>>>>>>>>>>> (6.87%)
>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 32.00MiB
>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 32.00MiB
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Unallocated:
>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdh 2.81TiB
>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 2.81TiB
>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 2.81TiB
>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdd 1.03TiB
>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdc 1.03TiB
>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sde 1.03TiB
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> > > If this is saying there's no extra
>>>>>>>>>>>>>>>>>>>>>>>>> space for
>>>>>>>>>>>>>>>>>>>>>>>>> metadata, is that why
>>>>>>>>>>>>>>>>>>>>>>>>> > > adding more files often makes the
>>>>>>>>>>>>>>>>>>>>>>>>> system hang
>>>>>>>>>>>>>>>>>>>>>>>>> for 30-90s? Is there
>>>>>>>>>>>>>>>>>>>>>>>>> > > anything I should do about that?
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > I'm not sure about the hang though.
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > It would be nice to give more info to
>>>>>>>>>>>>>>>>>>>>>>>>> diagnosis.
>>>>>>>>>>>>>>>>>>>>>>>>> > The output of 'btrfs fi usage' is
>>>>>>>>>>>>>>>>>>>>>>>>> useful for
>>>>>>>>>>>>>>>>>>>>>>>>> space usage problem.
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > But the common idea is, to keep at 1~2 Gi
>>>>>>>>>>>>>>>>>>>>>>>>> unallocated (not avaiable
>>>>>>>>>>>>>>>>>>>>>>>>> > space in vanilla df command) space for
>>>>>>>>>>>>>>>>>>>>>>>>> btrfs.
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> > Qu
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>>>>>>>> > > Thank you so much for all of your
>>>>>>>>>>>>>>>>>>>>>>>>> help. I
>>>>>>>>>>>>>>>>>>>>>>>>> love how flexible BTRFS is
>>>>>>>>>>>>>>>>>>>>>>>>> > > but when things go wrong it's very hard
>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>> me to troubleshoot.
>>>>>>>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu
>>>>>>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>>> > >> On 2020/5/8 下午12:23, Tyler Richmond
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> > >>> Something went wrong:
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>> Reinitialize checksum tree
>>>>>>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>>>>>>> > >>> ctree.c:2272: split_leaf: BUG_ON `1`
>>>>>>>>>>>>>>>>>>>>>>>>> triggered, value 1
>>>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
>>>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
>>>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
>>>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
>>>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(_start+0x2e)[0x55a933a9fa0e]
>>>>>>>>>>>>>>>>>>>>>>>>> > >>> Aborted
>>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>>> > >> This means no space for extra
>>>>>>>>>>>>>>>>>>>>>>>>> metadata...
>>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>>> > >> Anyway the csum tree problem shouldn't
>>>>>>>>>>>>>>>>>>>>>>>>> be a
>>>>>>>>>>>>>>>>>>>>>>>>> big thing, you
>>>>>>>>>>>>>>>>>>>>>>>>> could leave
>>>>>>>>>>>>>>>>>>>>>>>>> > >> it and call it a day.
>>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>>> > >> BTW, as long as btrfs check reports no
>>>>>>>>>>>>>>>>>>>>>>>>> extra
>>>>>>>>>>>>>>>>>>>>>>>>> problem for the inode
>>>>>>>>>>>>>>>>>>>>>>>>> > >> generation, it should be pretty safe
>>>>>>>>>>>>>>>>>>>>>>>>> to use
>>>>>>>>>>>>>>>>>>>>>>>>> the fs.
>>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>>> > >> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> > >> Qu
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>> I just noticed I have btrfs-progs 5.6
>>>>>>>>>>>>>>>>>>>>>>>>> installed and 5.6.1 is
>>>>>>>>>>>>>>>>>>>>>>>>> > >>> available. I'll let that try
>>>>>>>>>>>>>>>>>>>>>>>>> overnight?
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu
>>>>>>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler
>>>>>>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> Thank you for helping. The end
>>>>>>>>>>>>>>>>>>>>>>>>> result of
>>>>>>>>>>>>>>>>>>>>>>>>> the scan was:
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [1/7] checking root items
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [2/7] checking extents
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [3/7] checking free space cache
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [4/7] checking fs roots
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Good news is, your fs is still
>>>>>>>>>>>>>>>>>>>>>>>>> mostly
>>>>>>>>>>>>>>>>>>>>>>>>> fine.
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [5/7] checking only csums items
>>>>>>>>>>>>>>>>>>>>>>>>> (without
>>>>>>>>>>>>>>>>>>>>>>>>> verifying data)
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>>>>>>>> 0-69632
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 0-69632 but there
>>>>>>>>>>>>>>>>>>>>>>>>> is no
>>>>>>>>>>>>>>>>>>>>>>>>> extent record
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>>>>>>>> 946692096-946827264
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 946692096-946827264
>>>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>>> there is no extent
>>>>>>>>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for csum range
>>>>>>>>>>>>>>>>>>>>>>>>> 946831360-947912704
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 946831360-947912704
>>>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>>> there is no extent
>>>>>>>>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> ERROR: errors found in csum tree
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Only extent tree is corrupted.
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Normally btrfs check
>>>>>>>>>>>>>>>>>>>>>>>>> --init-csum-tree
>>>>>>>>>>>>>>>>>>>>>>>>> should be able to
>>>>>>>>>>>>>>>>>>>>>>>>> handle it.
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> But still, please be sure you're
>>>>>>>>>>>>>>>>>>>>>>>>> using the
>>>>>>>>>>>>>>>>>>>>>>>>> latest btrfs-progs
>>>>>>>>>>>>>>>>>>>>>>>>> to fix it.
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Qu
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [6/7] checking root refs
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [7/7] checking quota groups skipped
>>>>>>>>>>>>>>>>>>>>>>>>> (not
>>>>>>>>>>>>>>>>>>>>>>>>> enabled on this FS)
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> found 44157956026368 bytes used,
>>>>>>>>>>>>>>>>>>>>>>>>> error(s)
>>>>>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> total csum bytes: 42038602716
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> total tree bytes: 49688616960
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> total fs tree bytes: 1256427520
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> total extent tree bytes: 1709105152
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> btree space waste bytes: 3172727316
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> file data blocks allocated:
>>>>>>>>>>>>>>>>>>>>>>>>> 261625653436416
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> referenced 47477768499200
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> What do I need to do to fix all of
>>>>>>>>>>>>>>>>>>>>>>>>> this?
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52 AM Qu
>>>>>>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler
>>>>>>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Well, the repair doesn't look
>>>>>>>>>>>>>>>>>>>>>>>>> terribly
>>>>>>>>>>>>>>>>>>>>>>>>> successful.
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> This means there are more
>>>>>>>>>>>>>>>>>>>>>>>>> problems, not
>>>>>>>>>>>>>>>>>>>>>>>>> only the hash name
>>>>>>>>>>>>>>>>>>>>>>>>> mismatch.
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> This means the fs is already
>>>>>>>>>>>>>>>>>>>>>>>>> corrupted,
>>>>>>>>>>>>>>>>>>>>>>>>> the name hash is
>>>>>>>>>>>>>>>>>>>>>>>>> just one
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> unrelated symptom.
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> The only good news is, btrfs-progs
>>>>>>>>>>>>>>>>>>>>>>>>> abort
>>>>>>>>>>>>>>>>>>>>>>>>> the transaction,
>>>>>>>>>>>>>>>>>>>>>>>>> thus no
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> further damage to the fs.
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Please run a plain btrfs-check to
>>>>>>>>>>>>>>>>>>>>>>>>> show
>>>>>>>>>>>>>>>>>>>>>>>>> what's the problem
>>>>>>>>>>>>>>>>>>>>>>>>> first.
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Qu
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify failed on
>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted: parent
>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: failed to zero log
>>>>>>>>>>>>>>>>>>>>>>>>> tree: -17
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: attempt to start
>>>>>>>>>>>>>>>>>>>>>>>>> transaction
>>>>>>>>>>>>>>>>>>>>>>>>> over already running one
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: reserved space leaked,
>>>>>>>>>>>>>>>>>>>>>>>>> flag=0x4 bytes_reserved=4096
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted
>>>>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted
>>>>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted
>>>>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak (aborted
>>>>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> What is going on?
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> On Wed, May 6, 2020 at 9:30 PM
>>>>>>>>>>>>>>>>>>>>>>>>> Tyler
>>>>>>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> Chris, I had used the correct
>>>>>>>>>>>>>>>>>>>>>>>>> mountpoint in the command.
>>>>>>>>>>>>>>>>>>>>>>>>> I just edited
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> it in the email to be
>>>>>>>>>>>>>>>>>>>>>>>>> /mountpoint for
>>>>>>>>>>>>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> Qu, I'll try the repair. Fingers
>>>>>>>>>>>>>>>>>>>>>>>>> crossed!
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13
>>>>>>>>>>>>>>>>>>>>>>>>> PM Qu
>>>>>>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler
>>>>>>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> I looked up this error and it
>>>>>>>>>>>>>>>>>>>>>>>>> basically says ask a
>>>>>>>>>>>>>>>>>>>>>>>>> developer to
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> determine if it's a false
>>>>>>>>>>>>>>>>>>>>>>>>> error or
>>>>>>>>>>>>>>>>>>>>>>>>> not. I just started
>>>>>>>>>>>>>>>>>>>>>>>>> getting some
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> slow response times, and
>>>>>>>>>>>>>>>>>>>>>>>>> looked at
>>>>>>>>>>>>>>>>>>>>>>>>> the dmesg log to
>>>>>>>>>>>>>>>>>>>>>>>>> find a ton of
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> these errors.
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS critical
>>>>>>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492
>>>>>>>>>>>>>>>>>>>>>>>>> expect [0,
>>>>>>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS error
>>>>>>>>>>>>>>>>>>>>>>>>> (device
>>>>>>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption
>>>>>>>>>>>>>>>>>>>>>>>>> detected
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS critical
>>>>>>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492
>>>>>>>>>>>>>>>>>>>>>>>>> expect [0,
>>>>>>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS error
>>>>>>>>>>>>>>>>>>>>>>>>> (device
>>>>>>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption
>>>>>>>>>>>>>>>>>>>>>>>>> detected
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS critical
>>>>>>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840 slot=4
>>>>>>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492
>>>>>>>>>>>>>>>>>>>>>>>>> expect [0,
>>>>>>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS error
>>>>>>>>>>>>>>>>>>>>>>>>> (device
>>>>>>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption
>>>>>>>>>>>>>>>>>>>>>>>>> detected
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> btrfs device stats, however,
>>>>>>>>>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>>>>>>>> show any errors.
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Is there anything I should do
>>>>>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>>> this, or should I
>>>>>>>>>>>>>>>>>>>>>>>>> just continue
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> using my array as normal?
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> This is caused by older kernel
>>>>>>>>>>>>>>>>>>>>>>>>> underflow inode generation.
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Latest btrfs-progs can fix it,
>>>>>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>>>>>> btrfs check --repair.
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Or you can go safer, by
>>>>>>>>>>>>>>>>>>>>>>>>> manually
>>>>>>>>>>>>>>>>>>>>>>>>> locating the inode
>>>>>>>>>>>>>>>>>>>>>>>>> using its inode
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> number (1311670), and copy it
>>>>>>>>>>>>>>>>>>>>>>>>> to some
>>>>>>>>>>>>>>>>>>>>>>>>> new location using
>>>>>>>>>>>>>>>>>>>>>>>>> previous
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> working kernel, then delete the
>>>>>>>>>>>>>>>>>>>>>>>>> old
>>>>>>>>>>>>>>>>>>>>>>>>> file, copy the new
>>>>>>>>>>>>>>>>>>>>>>>>> one back to fix it.
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-06 10:27 ` Qu Wenruo
@ 2020-11-06 10:32 ` Ferry Toth
0 siblings, 0 replies; 56+ messages in thread
From: Ferry Toth @ 2020-11-06 10:32 UTC (permalink / raw)
To: Qu Wenruo, Qu Wenruo, Tyler Richmond; +Cc: Btrfs BTRFS
Op 06-11-2020 om 11:27 schreef Qu Wenruo:
>>> BTRFS critical (device sda2): corrupt leaf: root=294 block=1169152675840
>>> slot=1 ino=915987, invalid inode generation: has 18446744073709551492
>>> expect [0, 5852829]
>>> BTRFS error (device sda2): block=1169152675840 read time tree block
>>> corruption detected
>>>
>>> So how do I repair this? Am I doing something wrong?
>> Please provide the following dump:
>> btrfs ins dump-tree -b 1169152675840 /dev/sda2
>>
>> Feel free to remove the filenames in the dump.
>>
>> And 'btrfs check /dev/sda2' output after the repair.
>>
>> As a workaround, you can use older kernel (v5.2 at most) to temporary
>> ignore the problem.
>>
>> Thanks,
>> Qu
>>
>>>>>> THanks,
>>>>>> Qu
>>>>>>>> Thanks,
>>>>>>>> Qu
>>>>>>>>
>>>>>>>>> Linux = 5.6.0-1032-oem
> Wait, it's just v5.6??
I did repair with Kubuntu groovy live usb.
That has 5.8 kernel + 5.7 btrfs-progs.
> Then that means, the error message can be wrong. Both transid and inode
> generation error could be output as "inode generation error".
>
> And to repair inode transid, the repair ability is not yet merged into
> upstream btrfs-progs.
>
> You can use the out-of-tree branch to repair it:
> https://github.com/adam900710/btrfs-progs/tree/inode_transid
>
> I'm afraid you have to build the btrfs-progs in liveUSB environment to
> repair it...
>
> I need to re-push the branch to make it into upstream.
>
> Thanks,
> Qu
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-06 10:30 ` Ferry Toth
@ 2020-11-06 10:32 ` Qu Wenruo
2020-11-07 11:18 ` Ferry Toth
2020-11-06 11:28 ` Ferry Toth
1 sibling, 1 reply; 56+ messages in thread
From: Qu Wenruo @ 2020-11-06 10:32 UTC (permalink / raw)
To: Ferry Toth, Qu Wenruo, Tyler Richmond; +Cc: Btrfs BTRFS
On 2020/11/6 下午6:30, Ferry Toth wrote:
> Hi
>
> Op 06-11-2020 om 11:24 schreef Qu Wenruo:
>>
>> On 2020/11/6 下午6:09, Ferry Toth wrote:
>>> Hi Qu
>>>
>>> Op 06-11-2020 om 00:40 schreef Qu Wenruo:
>>>> On 2020/11/6 上午7:37, Ferry Toth wrote:
>>>>> Hi
>>>>>
>>>>> Op 06-11-2020 om 00:32 schreef Qu Wenruo:
>>>>>> On 2020/11/6 上午7:12, Ferry Toth wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Op 06-11-2020 om 00:00 schreef Qu Wenruo:
>>>>>>>> On 2020/11/6 上午4:08, Ferry Toth wrote:
>>>>>>>>> I am in a similar spot, during updating my distro (Kubuntu), I am
>>>>>>>>> unable
>>>>>>>>> to update a certain package. I know which file it is:
>>>>>>>>>
>>>>>>>>> ~$ ls -l /usr/share/doc/libatk1.0-data
>>>>>>>>> ls: kan geen toegang krijgen tot '/usr/share/doc/libatk1.0-data':
>>>>>>>>> Invoer-/uitvoerfout
>>>>>>>>>
>>>>>>>>> This creates the following in journal:
>>>>>>>>>
>>>>>>>>> kernel: BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>>>>> block=1169152675840 slot=1 ino=915987, invalid inode
>>>>>>>>> generation: has
>>>>>>>>> 18446744073709551492 expect [0, 5851353]
>>>>>>>>> kernel: BTRFS error (device sda2): block=1169152675840 read time
>>>>>>>>> tree
>>>>>>>>> block corruption detected
>>>>>>>>>
>>>>>>>>> Now, the problem: this file is on my rootfs, which is mounted. apt
>>>>>>>>> (distribution updated) installed all packages but can't continue
>>>>>>>>> configuring, because libatk is a dependancy. I can't delete the
>>>>>>>>> file
>>>>>>>>> because of the I/O error. And btrfs check complains (I tried
>>>>>>>>> running RO)
>>>>>>>>> because the file system is mounted.
>>>>>>>>>
>>>>>>>>> But, on the sunny side, the file system is not RO.
>>>>>>>>>
>>>>>>>>> Is there any way to forcefully remove the file? Or do you have a
>>>>>>>>> recommendation how to proceed?
>>>>>>>> Newer kernel will reject to even read the item, thus will not be
>>>>>>>> able to
>>>>>>>> remove it.
>>>>>>> That's already the case. (input / output error)
>>>>>>>> I guess you have to use some distro ISO to fix the fs.
>>>>>>> And then? btrfs check --repair the disk offline?
>>>>>> Yep.
>>>>>>
>>>>>> You would want the latest btrfs-progs though.
>>>>> Groovy has 5.7. Would that be good enough? Otherwise will be difficult
>>>>> to build on/for live usb image.
>>>> For your particular case, the fix are already in btrfs-progs v5.4.
>>>>
>>>> Although newer is always better, just in case you have extent item
>>>> generation corruption, you may want v5.4.1.
>>>>
>>>> So your v5.7 should be good enough.
>>>>
>>>> Thanks,
>>>> Qu
>>> I made a live usb and performed:
>>>
>>> btrfs check --repair /dev/sda2
>>>
>>> It found errors and fixed them. However, it did not fix the corrupt
>>> leaf. The file is actually a directory:
>>>
>>> ~$ stat /usr/share/doc/libatk1.0-data
>>> stat: cannot statx '/usr/share/doc/libatk1.0-data': Invoer-/uitvoerfout
>>>
>>> in journal:
>>>
>>> BTRFS critical (device sda2): corrupt leaf: root=294 block=1169152675840
>>> slot=1 ino=915987, invalid inode generation: has 18446744073709551492
>>> expect [0, 5852829]
>>> BTRFS error (device sda2): block=1169152675840 read time tree block
>>> corruption detected
>>>
>>> So how do I repair this? Am I doing something wrong?
>> Please provide the following dump:
>> btrfs ins dump-tree -b 1169152675840 /dev/sda2
>>
>> Feel free to remove the filenames in the dump.
> sudo btrfs ins dump-tree -b 1169152675840 /dev/sda2
> btrfs-progs v5.3-rc1
> leaf 1169152675840 items 36 free space 966 generation 5431733 owner 294
> leaf 1169152675840 flags 0x1(WRITTEN) backref revision 1
> fs uuid 27155120-9ef8-47fb-b248-eaac2b7c8375
> chunk uuid 5704f1ba-08fd-4f6b-9117-0e080b4e9ef0
> item 0 key (915986 DIR_INDEX 2) itemoff 3957 itemsize 38
> location key (915987 INODE_ITEM 0) type FILE
> transid 7782235549259005952 data_len 0 name_len 8
> name: smb.conf
> item 1 key (915987 INODE_ITEM 0) itemoff 3797 itemsize 160
> generation 1 transid 18446744073709551492 size 12464
> nbytes 16384
Yeah, corrupted transid.
The v5.6 kernel doesn't get the fix backported...
Now you have to use either the out-of-tree branch, or David's devel
branch to build a btrfs-progs which is able to repair the transid error.
Thanks,
Qu
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-06 10:30 ` Ferry Toth
2020-11-06 10:32 ` Qu Wenruo
@ 2020-11-06 11:28 ` Ferry Toth
1 sibling, 0 replies; 56+ messages in thread
From: Ferry Toth @ 2020-11-06 11:28 UTC (permalink / raw)
To: Qu Wenruo, Qu Wenruo, Tyler Richmond; +Cc: Btrfs BTRFS
Hi
Op 06-11-2020 om 11:30 schreef Ferry Toth:
> Hi
>
> Op 06-11-2020 om 11:24 schreef Qu Wenruo:
>>
>> On 2020/11/6 下午6:09, Ferry Toth wrote:
>>> Hi Qu
>>>
>>> Op 06-11-2020 om 00:40 schreef Qu Wenruo:
>>>> On 2020/11/6 上午7:37, Ferry Toth wrote:
>>>>> Hi
>>>>>
>>>>> Op 06-11-2020 om 00:32 schreef Qu Wenruo:
>>>>>> On 2020/11/6 上午7:12, Ferry Toth wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Op 06-11-2020 om 00:00 schreef Qu Wenruo:
>>>>>>>> On 2020/11/6 上午4:08, Ferry Toth wrote:
>>>>>>>>> I am in a similar spot, during updating my distro (Kubuntu), I am
>>>>>>>>> unable
>>>>>>>>> to update a certain package. I know which file it is:
>>>>>>>>>
>>>>>>>>> ~$ ls -l /usr/share/doc/libatk1.0-data
>>>>>>>>> ls: kan geen toegang krijgen tot '/usr/share/doc/libatk1.0-data':
>>>>>>>>> Invoer-/uitvoerfout
>>>>>>>>>
>>>>>>>>> This creates the following in journal:
>>>>>>>>>
>>>>>>>>> kernel: BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>>>>> block=1169152675840 slot=1 ino=915987, invalid inode
>>>>>>>>> generation: has
>>>>>>>>> 18446744073709551492 expect [0, 5851353]
>>>>>>>>> kernel: BTRFS error (device sda2): block=1169152675840 read time
>>>>>>>>> tree
>>>>>>>>> block corruption detected
>>>>>>>>>
>>>>>>>>> Now, the problem: this file is on my rootfs, which is mounted.
>>>>>>>>> apt
>>>>>>>>> (distribution updated) installed all packages but can't continue
>>>>>>>>> configuring, because libatk is a dependancy. I can't delete
>>>>>>>>> the file
>>>>>>>>> because of the I/O error. And btrfs check complains (I tried
>>>>>>>>> running RO)
>>>>>>>>> because the file system is mounted.
>>>>>>>>>
>>>>>>>>> But, on the sunny side, the file system is not RO.
>>>>>>>>>
>>>>>>>>> Is there any way to forcefully remove the file? Or do you have a
>>>>>>>>> recommendation how to proceed?
>>>>>>>> Newer kernel will reject to even read the item, thus will not be
>>>>>>>> able to
>>>>>>>> remove it.
>>>>>>> That's already the case. (input / output error)
>>>>>>>> I guess you have to use some distro ISO to fix the fs.
>>>>>>> And then? btrfs check --repair the disk offline?
>>>>>> Yep.
>>>>>>
>>>>>> You would want the latest btrfs-progs though.
>>>>> Groovy has 5.7. Would that be good enough? Otherwise will be
>>>>> difficult
>>>>> to build on/for live usb image.
>>>> For your particular case, the fix are already in btrfs-progs v5.4.
>>>>
>>>> Although newer is always better, just in case you have extent item
>>>> generation corruption, you may want v5.4.1.
>>>>
>>>> So your v5.7 should be good enough.
>>>>
>>>> Thanks,
>>>> Qu
>>> I made a live usb and performed:
>>>
>>> btrfs check --repair /dev/sda2
>>>
>>> It found errors and fixed them. However, it did not fix the corrupt
>>> leaf. The file is actually a directory:
>>>
>>> ~$ stat /usr/share/doc/libatk1.0-data
>>> stat: cannot statx '/usr/share/doc/libatk1.0-data': Invoer-/uitvoerfout
>>>
>>> in journal:
>>>
>>> BTRFS critical (device sda2): corrupt leaf: root=294
>>> block=1169152675840
>>> slot=1 ino=915987, invalid inode generation: has 18446744073709551492
>>> expect [0, 5852829]
>>> BTRFS error (device sda2): block=1169152675840 read time tree block
>>> corruption detected
>>>
>>> So how do I repair this? Am I doing something wrong?
>> Please provide the following dump:
>> btrfs ins dump-tree -b 1169152675840 /dev/sda2
>>
>> Feel free to remove the filenames in the dump.
> sudo btrfs ins dump-tree -b 1169152675840 /dev/sda2
> btrfs-progs v5.3-rc1
>
I see, now I am booted from the rootfs and btrfs-prog I built last time
from source is in /usr/local/bin. So let me repeat with Groovy version.
~$ sudo /bin/btrfs ins dump-tree -b 1169152675840 /dev/sda2
btrfs-progs v5.7
leaf 1169152675840 items 36 free space 966 generation 5431733 owner 294
leaf 1169152675840 flags 0x1(WRITTEN) backref revision 1
fs uuid 27155120-9ef8-47fb-b248-eaac2b7c8375
chunk uuid 5704f1ba-08fd-4f6b-9117-0e080b4e9ef0
item 0 key (915986 DIR_INDEX 2) itemoff 3957 itemsize 38
location key (915987 INODE_ITEM 0) type FILE
transid 7782235549259005952 data_len 0 name_len 8
name: smb.conf
item 1 key (915987 INODE_ITEM 0) itemoff 3797 itemsize 160
generation 1 transid 18446744073709551492 size 12464
nbytes 16384
block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
sequence 0 flags 0x0(none)
atime 1350489744.0 (2012-10-17 18:02:24)
ctime 1353328654.0 (2012-11-19 13:37:34)
mtime 1350489744.0 (2012-10-17 18:02:24)
otime 0.0 (1970-01-01 01:00:00)
item 2 key (915987 INODE_REF 915986) itemoff 3779 itemsize 18
index 2 namelen 8 name: smb.conf
item 3 key (915987 EXTENT_DATA 0) itemoff 3726 itemsize 53
generation 18 type 1 (regular)
extent data disk byte 1110664871936 nr 16384
extent data offset 0 nr 16384 ram 16384
extent compression 0 (none)
item 4 key (915989 INODE_ITEM 0) itemoff 3566 itemsize 160
generation 1 transid 5431733 size 56 nbytes 0
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 160 flags 0x0(none)
atime 1571487761.716511096 (2019-10-19 14:22:41)
ctime 1588235951.582838139 (2020-04-30 10:39:11)
mtime 1588235951.582838139 (2020-04-30 10:39:11)
otime 0.0 (1970-01-01 01:00:00)
item 5 key (915989 INODE_REF 659081) itemoff 3546 itemsize 20
index 1101 namelen 10 name: libassuan0
item 6 key (915989 DIR_ITEM 653215628) itemoff 3497 itemsize 49
location key (72402032 INODE_ITEM 0) type FILE
transid 5431729 data_len 0 name_len 19
name: changelog.Debian.gz
item 7 key (915989 DIR_ITEM 1600214284) itemoff 3458 itemsize 39
location key (72402033 INODE_ITEM 0) type FILE
transid 5431729 data_len 0 name_len 9
name: copyright
item 8 key (915989 DIR_INDEX 62) itemoff 3409 itemsize 49
location key (72402032 INODE_ITEM 0) type FILE
transid 5431729 data_len 0 name_len 19
name: changelog.Debian.gz
item 9 key (915989 DIR_INDEX 63) itemoff 3370 itemsize 39
location key (72402033 INODE_ITEM 0) type FILE
transid 5431729 data_len 0 name_len 9
name: copyright
item 10 key (915990 INODE_ITEM 0) itemoff 3210 itemsize 160
generation 1 transid 4933124 size 56 nbytes 0
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 96 flags 0x0(none)
atime 1571487761.720511096 (2019-10-19 14:22:41)
ctime 1477470910.249847328 (2016-10-26 10:35:10)
mtime 1477470910.249847328 (2016-10-26 10:35:10)
otime 0.0 (1970-01-01 01:00:00)
item 11 key (915990 INODE_REF 659081) itemoff 3189 itemsize 21
index 1343 namelen 11 name: libasyncns0
item 12 key (915990 DIR_ITEM 653215628) itemoff 3140 itemsize 49
location key (25762857 INODE_ITEM 0) type FILE
transid 2068663 data_len 0 name_len 19
name: changelog.Debian.gz
item 13 key (915990 DIR_ITEM 1600214284) itemoff 3101 itemsize 39
location key (25762858 INODE_ITEM 0) type FILE
transid 2068663 data_len 0 name_len 9
name: copyright
item 14 key (915990 DIR_INDEX 38) itemoff 3052 itemsize 49
location key (25762857 INODE_ITEM 0) type FILE
transid 2068663 data_len 0 name_len 19
name: changelog.Debian.gz
item 15 key (915990 DIR_INDEX 39) itemoff 3013 itemsize 39
location key (25762858 INODE_ITEM 0) type FILE
transid 2068663 data_len 0 name_len 9
name: copyright
item 16 key (915991 INODE_ITEM 0) itemoff 2853 itemsize 160
generation 1 transid 4933124 size 68 nbytes 0
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 48 flags 0x0(none)
atime 1571487761.720511096 (2019-10-19 14:22:41)
ctime 1540813807.330187853 (2018-10-29 12:50:07)
mtime 1540813807.330187853 (2018-10-29 12:50:07)
otime 0.0 (1970-01-01 01:00:00)
item 17 key (915991 INODE_REF 659081) itemoff 2831 itemsize 22
index 1545 namelen 12 name: libatasmart4
item 18 key (915991 DIR_ITEM 653215628) itemoff 2782 itemsize 49
location key (52273681 INODE_ITEM 0) type FILE
transid 4036682 data_len 0 name_len 19
name: changelog.Debian.gz
item 19 key (915991 DIR_ITEM 1600214284) itemoff 2743 itemsize 39
location key (52273682 INODE_ITEM 0) type FILE
transid 4036682 data_len 0 name_len 9
name: copyright
item 20 key (915991 DIR_ITEM 3650993379) itemoff 2707 itemsize 36
location key (52273680 INODE_ITEM 0) type FILE
transid 4036682 data_len 0 name_len 6
name: README
item 21 key (915991 DIR_INDEX 20) itemoff 2671 itemsize 36
location key (52273680 INODE_ITEM 0) type FILE
transid 4036682 data_len 0 name_len 6
name: README
item 22 key (915991 DIR_INDEX 21) itemoff 2622 itemsize 49
location key (52273681 INODE_ITEM 0) type FILE
transid 4036682 data_len 0 name_len 19
name: changelog.Debian.gz
item 23 key (915991 DIR_INDEX 22) itemoff 2583 itemsize 39
location key (52273682 INODE_ITEM 0) type FILE
transid 4036682 data_len 0 name_len 9
name: copyright
item 24 key (915992 INODE_ITEM 0) itemoff 2423 itemsize 160
generation 1 transid 5431717 size 56 nbytes 0
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 611 flags 0x0(none)
atime 1571487761.728511097 (2019-10-19 14:22:41)
ctime 1588235951.295254395 (2020-04-30 10:39:11)
mtime 1588235951.295254395 (2020-04-30 10:39:11)
otime 0.0 (1970-01-01 01:00:00)
item 25 key (915992 INODE_REF 659081) itemoff 2402 itemsize 21
index 2141 namelen 11 name: libatk1.0-0
item 26 key (915992 DIR_ITEM 653215628) itemoff 2353 itemsize 49
location key (72401999 INODE_ITEM 0) type FILE
transid 5431713 data_len 0 name_len 19
name: changelog.Debian.gz
item 27 key (915992 DIR_ITEM 1600214284) itemoff 2314 itemsize 39
location key (72402000 INODE_ITEM 0) type FILE
transid 5431713 data_len 0 name_len 9
name: copyright
item 28 key (915992 DIR_INDEX 233) itemoff 2265 itemsize 49
location key (72401999 INODE_ITEM 0) type FILE
transid 5431713 data_len 0 name_len 19
name: changelog.Debian.gz
item 29 key (915992 DIR_INDEX 234) itemoff 2226 itemsize 39
location key (72402000 INODE_ITEM 0) type FILE
transid 5431713 data_len 0 name_len 9
name: copyright
item 30 key (915993 INODE_ITEM 0) itemoff 2066 itemsize 160
generation 1 transid 5431708 size 56 nbytes 0
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 315 flags 0x0(none)
atime 1571487761.720511096 (2019-10-19 14:22:41)
ctime 1588235951.95543462 (2020-04-30 10:39:11)
mtime 1588235951.95543462 (2020-04-30 10:39:11)
otime 0.0 (1970-01-01 01:00:00)
item 31 key (915993 INODE_REF 659081) itemoff 2042 itemsize 24
index 1639 namelen 14 name: libatk1.0-data
item 32 key (915993 DIR_ITEM 653215628) itemoff 1993 itemsize 49
location key (72401982 INODE_ITEM 0) type FILE
transid 5431704 data_len 0 name_len 19
name: changelog.Debian.gz
item 33 key (915993 DIR_ITEM 1600214284) itemoff 1954 itemsize 39
location key (72401983 INODE_ITEM 0) type FILE
transid 5431704 data_len 0 name_len 9
name: copyright
item 34 key (915993 DIR_INDEX 122) itemoff 1905 itemsize 49
location key (72401982 INODE_ITEM 0) type FILE
transid 5431704 data_len 0 name_len 19
name: changelog.Debian.gz
item 35 key (915993 DIR_INDEX 123) itemoff 1866 itemsize 39
location key (72401983 INODE_ITEM 0) type FILE
transid 5431704 data_len 0 name_len 9
name: copyright
>> And 'btrfs check /dev/sda2' output after the repair.
> There were no errors after the repair.
>> As a workaround, you can use older kernel (v5.2 at most) to temporary
>> ignore the problem.
>
> My collegue had the brilliant idea to move /usr/share/doc to
> /usr/share/dog, then cp back dog to doc.
>
> So at least I was now able to finish the dist-upgrade.
>
> Now question remains how to clean up /usr/share/dog
>
>> Thanks,
>> Qu
>>
>>>>>> THanks,
>>>>>> Qu
>>>>>>>> Thanks,
>>>>>>>> Qu
>>>>>>>>
>>>>>>>>> Linux = 5.6.0-1032-oem
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Ferry
>>>>>>>>>
>>>>>>>>> Op 05-11-2020 om 08:19 schreef Qu Wenruo:
>>>>>>>>>> On 2020/11/5 下午3:01, Tyler Richmond wrote:
>>>>>>>>>>> Qu,
>>>>>>>>>>>
>>>>>>>>>>> I'm wondering, was a fix for this ever implemented?
>>>>>>>>>> Already implemented the --repair ability in latest btrfs-progs.
>>>>>>>>>>
>>>>>>>>>>> I recently added a
>>>>>>>>>>> new drive to expand the array, and during the rebalance it
>>>>>>>>>>> dropped
>>>>>>>>>>> itself back to a read only filesystem. I suspect it's related
>>>>>>>>>>> to the
>>>>>>>>>>> issues discussed earlier in this thread. Is there anything I
>>>>>>>>>>> can
>>>>>>>>>>> do to
>>>>>>>>>>> complete the balance? The error that caused it to drop to read
>>>>>>>>>>> only is
>>>>>>>>>>> here: https://pastebin.com/GGYVMaiG
>>>>>>>>>> Yep, the same cause.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Qu
>>>>>>>>>>> Thanks!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 25, 2020 at 9:43 AM Tyler Richmond
>>>>>>>>>>> <t.d.richmond@gmail.com> wrote:
>>>>>>>>>>>> Great, glad we got somewhere! I'll look forward to the fix!
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Aug 25, 2020 at 9:38 AM Qu Wenruo
>>>>>>>>>>>> <quwenruo.btrfs@gmx.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> On 2020/8/25 下午9:30, Tyler Richmond wrote:
>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The dump of the block is:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://pastebin.com/ran85JJv
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've also completed the btrfs-image, but it's almost 50gb.
>>>>>>>>>>>>>> What's the
>>>>>>>>>>>>>> best way to get it to you? Also, does it work with -ss or
>>>>>>>>>>>>>> are the
>>>>>>>>>>>>>> original filenames important?
>>>>>>>>>>>>> 50G is too big for me to even receive.
>>>>>>>>>>>>>
>>>>>>>>>>>>> But your dump shows the problem!
>>>>>>>>>>>>>
>>>>>>>>>>>>> It's not inode generation, but inode transid, which would
>>>>>>>>>>>>> affect
>>>>>>>>>>>>> send.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is not even checked in btrfs-progs, thus no wonder
>>>>>>>>>>>>> why it
>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>> detect them.
>>>>>>>>>>>>>
>>>>>>>>>>>>> And copy-pasted kernel message shares the same "generation"
>>>>>>>>>>>>> word,
>>>>>>>>>>>>> not
>>>>>>>>>>>>> using proper transid to show the problem.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Your dump really saved the day!
>>>>>>>>>>>>>
>>>>>>>>>>>>> The fix for kernel and btrfs-progs would come in next few
>>>>>>>>>>>>> days.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>> Thanks again!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Aug 25, 2020 at 2:37 AM Qu Wenruo
>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> On 2020/8/25 下午1:25, Tyler Richmond wrote:
>>>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes, it's btrfs-progs 5.7. Here is the result of the
>>>>>>>>>>>>>>>> lowmem
>>>>>>>>>>>>>>>> check:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> https://pastebin.com/8Tzx23EX
>>>>>>>>>>>>>>> That doesn't detect any inode generation problem at all,
>>>>>>>>>>>>>>> which is
>>>>>>>>>>>>>>> not a
>>>>>>>>>>>>>>> good sign.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Would you also pvode the dump for the offending block?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode
>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>> has 18446744073709551492 expect [0, 6875827]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For this case, would you please provide the tree dump of
>>>>>>>>>>>>>>> "203510940835840" ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> # btrfs ins dump-tree -b 203510940835840 <device>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And, since btrfs-image can't dump with regular extent tree,
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> "-w"
>>>>>>>>>>>>>>> dump would also help.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Aug 24, 2020 at 4:26 AM Qu Wenruo
>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>> On 2020/8/24 上午10:47, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Finally finished another repair and captured the output.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> https://pastebin.com/ffcbwvd8
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Does that show you what you need? Or should I still do
>>>>>>>>>>>>>>>>>> one in
>>>>>>>>>>>>>>>>>> lowmem mode?
>>>>>>>>>>>>>>>>> Lowmem mode (no need for --repair) is recommended since
>>>>>>>>>>>>>>>>> original mode
>>>>>>>>>>>>>>>>> doesn't detect the inode generation problem.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> And it's already btrfs-progs v5.7 right?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> THanks,
>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>> Thanks for your help!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Aug 23, 2020 at 12:28 AM Qu Wenruo
>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>>>> On 2020/8/23 上午10:49, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>>> Well, I can guarantee that I didn't create this fs
>>>>>>>>>>>>>>>>>>>> before
>>>>>>>>>>>>>>>>>>>> 2015 (just
>>>>>>>>>>>>>>>>>>>> checked the order confirmation from when I bought the
>>>>>>>>>>>>>>>>>>>> server), but I
>>>>>>>>>>>>>>>>>>>> may have just used whatever was in the Ubuntu package
>>>>>>>>>>>>>>>>>>>> manager at the
>>>>>>>>>>>>>>>>>>>> time. So maybe I don't have a v0 ref?
>>>>>>>>>>>>>>>>>>> Then btrfs-image shouldn't report that.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> There is an item smaller than any valid btrfs item,
>>>>>>>>>>>>>>>>>>> normally
>>>>>>>>>>>>>>>>>>> it means
>>>>>>>>>>>>>>>>>>> it's a v0 ref.
>>>>>>>>>>>>>>>>>>> If not, then it could be a bigger problem.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Could you please provide the full btrfs-check output?
>>>>>>>>>>>>>>>>>>> Also, if possible result from "btrfs check
>>>>>>>>>>>>>>>>>>> --mode=lowmem"
>>>>>>>>>>>>>>>>>>> would also help.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Also, if you really go "--repair", then the full output
>>>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>> also be
>>>>>>>>>>>>>>>>>>> needed to determine what's going wrong.
>>>>>>>>>>>>>>>>>>> There is a report about "btrfs check --repair" didn't
>>>>>>>>>>>>>>>>>>> repair
>>>>>>>>>>>>>>>>>>> the inode
>>>>>>>>>>>>>>>>>>> generation, if that's the case we must have a bug then.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>> On Sat, Aug 22, 2020 at 10:31 PM Qu Wenruo
>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>>>>>> On 2020/8/23 上午9:51, Qu Wenruo wrote:
>>>>>>>>>>>>>>>>>>>>>> On 2020/8/23 上午9:15, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>>>>>> Is my best bet just to downgrade the kernel and
>>>>>>>>>>>>>>>>>>>>>>> then try
>>>>>>>>>>>>>>>>>>>>>>> to delete the
>>>>>>>>>>>>>>>>>>>>>>> broken files? Or should I rebuild from scratch?
>>>>>>>>>>>>>>>>>>>>>>> Just
>>>>>>>>>>>>>>>>>>>>>>> don't know
>>>>>>>>>>>>>>>>>>>>>>> whether it's worth the time to try and figure this
>>>>>>>>>>>>>>>>>>>>>>> out or
>>>>>>>>>>>>>>>>>>>>>>> if the
>>>>>>>>>>>>>>>>>>>>>>> problems stem from the FS being too old and it's
>>>>>>>>>>>>>>>>>>>>>>> beyond
>>>>>>>>>>>>>>>>>>>>>>> trying to
>>>>>>>>>>>>>>>>>>>>>>> repair.
>>>>>>>>>>>>>>>>>>>>>> All invalid inode generations, should be able to be
>>>>>>>>>>>>>>>>>>>>>> repaired by latest
>>>>>>>>>>>>>>>>>>>>>> btrfs-check.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> If not, please provide the btrfs-image dump for
>>>>>>>>>>>>>>>>>>>>>> us to
>>>>>>>>>>>>>>>>>>>>>> determine what's
>>>>>>>>>>>>>>>>>>>>>> going wrong.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 8:18 AM Tyler Richmond
>>>>>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> I didn't check dmesg during the btrfs check,
>>>>>>>>>>>>>>>>>>>>>>>> but that
>>>>>>>>>>>>>>>>>>>>>>>> was the only
>>>>>>>>>>>>>>>>>>>>>>>> output during the rm -f before it was forced
>>>>>>>>>>>>>>>>>>>>>>>> readonly. I
>>>>>>>>>>>>>>>>>>>>>>>> just checked
>>>>>>>>>>>>>>>>>>>>>>>> dmesg for inode generation values, and there are a
>>>>>>>>>>>>>>>>>>>>>>>> lot
>>>>>>>>>>>>>>>>>>>>>>>> of them.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> https://pastebin.com/stZdN0ta
>>>>>>>>>>>>>>>>>>>>>>>> The dmesg output had 990 lines containing inode
>>>>>>>>>>>>>>>>>>>>>>>> generation.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> However, these were at least later. I tried to
>>>>>>>>>>>>>>>>>>>>>>>> do a
>>>>>>>>>>>>>>>>>>>>>>>> btrfs balance
>>>>>>>>>>>>>>>>>>>>>>>> -mconvert raid1 and it failed with an I/O error.
>>>>>>>>>>>>>>>>>>>>>>>> That is
>>>>>>>>>>>>>>>>>>>>>>>> probably what
>>>>>>>>>>>>>>>>>>>>>>>> generated these specific errors, but maybe they
>>>>>>>>>>>>>>>>>>>>>>>> were
>>>>>>>>>>>>>>>>>>>>>>>> also happening
>>>>>>>>>>>>>>>>>>>>>>>> during the btrfs repair.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> The FS is ~45TB, but the btrfs-image -c9 failed
>>>>>>>>>>>>>>>>>>>>>>>> anway
>>>>>>>>>>>>>>>>>>>>>>>> with:
>>>>>>>>>>>>>>>>>>>>>>>> ERROR: either extent tree is corrupted or
>>>>>>>>>>>>>>>>>>>>>>>> deprecated
>>>>>>>>>>>>>>>>>>>>>>>> extent ref format
>>>>>>>>>>>>>>>>>>>>>>>> ERROR: create failed: -5
>>>>>>>>>>>>>>>>>>>>> Oh, forgot this part.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This means you have v0 ref?!
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Then the fs is too old, no progs/kernel support after
>>>>>>>>>>>>>>>>>>>>> all.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> In that case, please rollback to the last working
>>>>>>>>>>>>>>>>>>>>> kernel
>>>>>>>>>>>>>>>>>>>>> and copy your data.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> In fact, that v0 ref should only be in the code
>>>>>>>>>>>>>>>>>>>>> base for
>>>>>>>>>>>>>>>>>>>>> several weeks
>>>>>>>>>>>>>>>>>>>>> before 2010, thus it's really too old.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The good news is, with tree-checker, we should never
>>>>>>>>>>>>>>>>>>>>> experience such
>>>>>>>>>>>>>>>>>>>>> too-old-to-be-usable problem (at least I hope so)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 18, 2020 at 2:07 AM Qu Wenruo
>>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> On 2020/8/18 上午11:35, Tyler Richmond wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> Qu,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry to resurrect this thread, but I just
>>>>>>>>>>>>>>>>>>>>>>>>>> ran into
>>>>>>>>>>>>>>>>>>>>>>>>>> something that I
>>>>>>>>>>>>>>>>>>>>>>>>>> can't really just ignore. I've found a folder
>>>>>>>>>>>>>>>>>>>>>>>>>> that is
>>>>>>>>>>>>>>>>>>>>>>>>>> full of files
>>>>>>>>>>>>>>>>>>>>>>>>>> which I guess have been broken somehow. I
>>>>>>>>>>>>>>>>>>>>>>>>>> found a
>>>>>>>>>>>>>>>>>>>>>>>>>> backup and restored
>>>>>>>>>>>>>>>>>>>>>>>>>> them, but I want to delete this folder of broken
>>>>>>>>>>>>>>>>>>>>>>>>>> files. But whenever I
>>>>>>>>>>>>>>>>>>>>>>>>>> try, the fs is forced into readonly mode
>>>>>>>>>>>>>>>>>>>>>>>>>> again. I
>>>>>>>>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>>>>>>> finished another
>>>>>>>>>>>>>>>>>>>>>>>>>> btrfs check --repair but it didn't fix the
>>>>>>>>>>>>>>>>>>>>>>>>>> problem.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> https://pastebin.com/eTV3s3fr
>>>>>>>>>>>>>>>>>>>>>>>>> Is that the full output?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> No inode generation bugs?
>>>>>>>>>>>>>>>>>>>>>>>>>> I'm already on btrfs-progs v5.7. Any new
>>>>>>>>>>>>>>>>>>>>>>>>>> suggestions?
>>>>>>>>>>>>>>>>>>>>>>>>> Strange.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> The detection and repair should have been merged
>>>>>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>>> v5.5.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> If your fs is small enough, would you please
>>>>>>>>>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>>>>>>>>>> the "btrfs-image
>>>>>>>>>>>>>>>>>>>>>>>>> -c9" dump?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> It would contain the filenames and directories
>>>>>>>>>>>>>>>>>>>>>>>>> names,
>>>>>>>>>>>>>>>>>>>>>>>>> but doesn't
>>>>>>>>>>>>>>>>>>>>>>>>> contain file contents.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 9:52 AM Tyler Richmond
>>>>>>>>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 5.6.1 also failed the same way.
>>>>>>>>>>>>>>>>>>>>>>>>>> Here's the
>>>>>>>>>>>>>>>>>>>>>>>>>> usage
>>>>>>>>>>>>>>>>>>>>>>>>>> output. This is the
>>>>>>>>>>>>>>>>>>>>>>>>>> part where you see I've been using
>>>>>>>>>>>>>>>>>>>>>>>>>> RAID5
>>>>>>>>>>>>>>>>>>>>>>>>>> haha
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> WARNING: RAID56 detected, not implemented
>>>>>>>>>>>>>>>>>>>>>>>>>> Overall:
>>>>>>>>>>>>>>>>>>>>>>>>>> Device size: 60.03TiB
>>>>>>>>>>>>>>>>>>>>>>>>>> Device allocated: 98.06GiB
>>>>>>>>>>>>>>>>>>>>>>>>>> Device unallocated: 59.93TiB
>>>>>>>>>>>>>>>>>>>>>>>>>> Device missing: 0.00B
>>>>>>>>>>>>>>>>>>>>>>>>>> Used: 92.56GiB
>>>>>>>>>>>>>>>>>>>>>>>>>> Free (estimated): 0.00B
>>>>>>>>>>>>>>>>>>>>>>>>>> (min: 8.00EiB)
>>>>>>>>>>>>>>>>>>>>>>>>>> Data ratio: 0.00
>>>>>>>>>>>>>>>>>>>>>>>>>> Metadata ratio: 2.00
>>>>>>>>>>>>>>>>>>>>>>>>>> Global reserve: 512.00MiB
>>>>>>>>>>>>>>>>>>>>>>>>>> (used: 0.00B)
>>>>>>>>>>>>>>>>>>>>>>>>>> Multiple profiles: no
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Data,RAID5: Size:40.35TiB, Used:40.12TiB
>>>>>>>>>>>>>>>>>>>>>>>>>> (99.42%)
>>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdh 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdd 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdc 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sde 8.07TiB
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Metadata,RAID1: Size:49.00GiB,
>>>>>>>>>>>>>>>>>>>>>>>>>> Used:46.28GiB
>>>>>>>>>>>>>>>>>>>>>>>>>> (94.44%)
>>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdh 34.00GiB
>>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 32.00GiB
>>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 32.00GiB
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> System,RAID1: Size:32.00MiB, Used:2.20MiB
>>>>>>>>>>>>>>>>>>>>>>>>>> (6.87%)
>>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 32.00MiB
>>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 32.00MiB
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Unallocated:
>>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdh 2.81TiB
>>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdf 2.81TiB
>>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdg 2.81TiB
>>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdd 1.03TiB
>>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sdc 1.03TiB
>>>>>>>>>>>>>>>>>>>>>>>>>> /dev/sde 1.03TiB
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 8, 2020 at 1:47 AM Qu
>>>>>>>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > On 2020/5/8 下午1:12, Tyler Richmond
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> > > If this is saying there's no extra
>>>>>>>>>>>>>>>>>>>>>>>>>> space for
>>>>>>>>>>>>>>>>>>>>>>>>>> metadata, is that why
>>>>>>>>>>>>>>>>>>>>>>>>>> > > adding more files often makes the
>>>>>>>>>>>>>>>>>>>>>>>>>> system hang
>>>>>>>>>>>>>>>>>>>>>>>>>> for 30-90s? Is there
>>>>>>>>>>>>>>>>>>>>>>>>>> > > anything I should do about that?
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > I'm not sure about the hang though.
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > It would be nice to give more info to
>>>>>>>>>>>>>>>>>>>>>>>>>> diagnosis.
>>>>>>>>>>>>>>>>>>>>>>>>>> > The output of 'btrfs fi usage' is
>>>>>>>>>>>>>>>>>>>>>>>>>> useful for
>>>>>>>>>>>>>>>>>>>>>>>>>> space usage problem.
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > But the common idea is, to keep at
>>>>>>>>>>>>>>>>>>>>>>>>>> 1~2 Gi
>>>>>>>>>>>>>>>>>>>>>>>>>> unallocated (not avaiable
>>>>>>>>>>>>>>>>>>>>>>>>>> > space in vanilla df command) space
>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>> btrfs.
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> > Qu
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>>>>>>>>> > > Thank you so much for all of your
>>>>>>>>>>>>>>>>>>>>>>>>>> help. I
>>>>>>>>>>>>>>>>>>>>>>>>>> love how flexible BTRFS is
>>>>>>>>>>>>>>>>>>>>>>>>>> > > but when things go wrong it's
>>>>>>>>>>>>>>>>>>>>>>>>>> very hard
>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>> me to troubleshoot.
>>>>>>>>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>>>>>>>>> > > On Fri, May 8, 2020 at 1:07 AM Qu
>>>>>>>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >> On 2020/5/8 下午12:23, Tyler
>>>>>>>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>> Something went wrong:
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>> Reinitialize checksum tree
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>> Unable to find block group for 0
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>> ctree.c:2272: split_leaf:
>>>>>>>>>>>>>>>>>>>>>>>>>> BUG_ON `1`
>>>>>>>>>>>>>>>>>>>>>>>>>> triggered, value 1
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x6dd94)[0x55a933af7d94]
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x71b94)[0x55a933afbb94]
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>>>> btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8]
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>>>> btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09]
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x360b2)[0x55a933ac00b2]
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(+0x46a3e)[0x55a933ad0a3e]
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>> btrfs(main+0x98)[0x55a933a9fe88]
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3]
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>>>> btrfs(_start+0x2e)[0x55a933a9fa0e]
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>> Aborted
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >> This means no space for extra
>>>>>>>>>>>>>>>>>>>>>>>>>> metadata...
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >> Anyway the csum tree problem
>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>>>>>> be a
>>>>>>>>>>>>>>>>>>>>>>>>>> big thing, you
>>>>>>>>>>>>>>>>>>>>>>>>>> could leave
>>>>>>>>>>>>>>>>>>>>>>>>>> > >> it and call it a day.
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >> BTW, as long as btrfs check
>>>>>>>>>>>>>>>>>>>>>>>>>> reports no
>>>>>>>>>>>>>>>>>>>>>>>>>> extra
>>>>>>>>>>>>>>>>>>>>>>>>>> problem for the inode
>>>>>>>>>>>>>>>>>>>>>>>>>> > >> generation, it should be pretty
>>>>>>>>>>>>>>>>>>>>>>>>>> safe
>>>>>>>>>>>>>>>>>>>>>>>>>> to use
>>>>>>>>>>>>>>>>>>>>>>>>>> the fs.
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> > >> Qu
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>> I just noticed I have
>>>>>>>>>>>>>>>>>>>>>>>>>> btrfs-progs 5.6
>>>>>>>>>>>>>>>>>>>>>>>>>> installed and 5.6.1 is
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>> available. I'll let that try
>>>>>>>>>>>>>>>>>>>>>>>>>> overnight?
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>> On Thu, May 7, 2020 at 8:11 PM Qu
>>>>>>>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> On 2020/5/7 下午11:52, Tyler
>>>>>>>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> Thank you for helping. The end
>>>>>>>>>>>>>>>>>>>>>>>>>> result of
>>>>>>>>>>>>>>>>>>>>>>>>>> the scan was:
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [1/7] checking root items
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [2/7] checking extents
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [3/7] checking free space cache
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [4/7] checking fs roots
>>>>>>>>>>>>>>>>>>>>>>>>>> �� > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Good news is, your fs is still
>>>>>>>>>>>>>>>>>>>>>>>>>> mostly
>>>>>>>>>>>>>>>>>>>>>>>>>> fine.
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [5/7] checking only csums items
>>>>>>>>>>>>>>>>>>>>>>>>>> (without
>>>>>>>>>>>>>>>>>>>>>>>>>> verifying data)
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for
>>>>>>>>>>>>>>>>>>>>>>>>>> csum range
>>>>>>>>>>>>>>>>>>>>>>>>>> 0-69632
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for 0-69632 but
>>>>>>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>>>>> is no
>>>>>>>>>>>>>>>>>>>>>>>>>> extent record
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> ...
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for
>>>>>>>>>>>>>>>>>>>>>>>>>> csum range
>>>>>>>>>>>>>>>>>>>>>>>>>> 946692096-946827264
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for
>>>>>>>>>>>>>>>>>>>>>>>>>> 946692096-946827264
>>>>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>>>> there is no extent
>>>>>>>>>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> there are no extents for
>>>>>>>>>>>>>>>>>>>>>>>>>> csum range
>>>>>>>>>>>>>>>>>>>>>>>>>> 946831360-947912704
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> csum exists for
>>>>>>>>>>>>>>>>>>>>>>>>>> 946831360-947912704
>>>>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>>>> there is no extent
>>>>>>>>>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> ERROR: errors found in csum
>>>>>>>>>>>>>>>>>>>>>>>>>> tree
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Only extent tree is corrupted.
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Normally btrfs check
>>>>>>>>>>>>>>>>>>>>>>>>>> --init-csum-tree
>>>>>>>>>>>>>>>>>>>>>>>>>> should be able to
>>>>>>>>>>>>>>>>>>>>>>>>>> handle it.
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> But still, please be sure you're
>>>>>>>>>>>>>>>>>>>>>>>>>> using the
>>>>>>>>>>>>>>>>>>>>>>>>>> latest btrfs-progs
>>>>>>>>>>>>>>>>>>>>>>>>>> to fix it.
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Qu
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [6/7] checking root refs
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> [7/7] checking quota groups
>>>>>>>>>>>>>>>>>>>>>>>>>> skipped
>>>>>>>>>>>>>>>>>>>>>>>>>> (not
>>>>>>>>>>>>>>>>>>>>>>>>>> enabled on this FS)
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> found 44157956026368 bytes
>>>>>>>>>>>>>>>>>>>>>>>>>> used,
>>>>>>>>>>>>>>>>>>>>>>>>>> error(s)
>>>>>>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> total csum bytes: 42038602716
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> total tree bytes: 49688616960
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> total fs tree bytes: 1256427520
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> total extent tree bytes:
>>>>>>>>>>>>>>>>>>>>>>>>>> 1709105152
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> btree space waste bytes:
>>>>>>>>>>>>>>>>>>>>>>>>>> 3172727316
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> file data blocks allocated:
>>>>>>>>>>>>>>>>>>>>>>>>>> 261625653436416
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> referenced 47477768499200
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> What do I need to do to fix
>>>>>>>>>>>>>>>>>>>>>>>>>> all of
>>>>>>>>>>>>>>>>>>>>>>>>>> this?
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>> On Thu, May 7, 2020 at 1:52
>>>>>>>>>>>>>>>>>>>>>>>>>> AM Qu
>>>>>>>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> On 2020/5/7 下午1:43, Tyler
>>>>>>>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Well, the repair doesn't look
>>>>>>>>>>>>>>>>>>>>>>>>>> terribly
>>>>>>>>>>>>>>>>>>>>>>>>>> successful.
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify
>>>>>>>>>>>>>>>>>>>>>>>>>> failed on
>>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify
>>>>>>>>>>>>>>>>>>>>>>>>>> failed on
>>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify
>>>>>>>>>>>>>>>>>>>>>>>>>> failed on
>>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted:
>>>>>>>>>>>>>>>>>>>>>>>>>> parent
>>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> This means there are more
>>>>>>>>>>>>>>>>>>>>>>>>>> problems, not
>>>>>>>>>>>>>>>>>>>>>>>>>> only the hash name
>>>>>>>>>>>>>>>>>>>>>>>>>> mismatch.
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> This means the fs is already
>>>>>>>>>>>>>>>>>>>>>>>>>> corrupted,
>>>>>>>>>>>>>>>>>>>>>>>>>> the name hash is
>>>>>>>>>>>>>>>>>>>>>>>>>> just one
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> unrelated symptom.
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> The only good news is,
>>>>>>>>>>>>>>>>>>>>>>>>>> btrfs-progs
>>>>>>>>>>>>>>>>>>>>>>>>>> abort
>>>>>>>>>>>>>>>>>>>>>>>>>> the transaction,
>>>>>>>>>>>>>>>>>>>>>>>>>> thus no
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> further damage to the fs.
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Please run a plain
>>>>>>>>>>>>>>>>>>>>>>>>>> btrfs-check to
>>>>>>>>>>>>>>>>>>>>>>>>>> show
>>>>>>>>>>>>>>>>>>>>>>>>>> what's the problem
>>>>>>>>>>>>>>>>>>>>>>>>>> first.
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Qu
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify
>>>>>>>>>>>>>>>>>>>>>>>>>> failed on
>>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted:
>>>>>>>>>>>>>>>>>>>>>>>>>> parent
>>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify
>>>>>>>>>>>>>>>>>>>>>>>>>> failed on
>>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted:
>>>>>>>>>>>>>>>>>>>>>>>>>> parent
>>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify
>>>>>>>>>>>>>>>>>>>>>>>>>> failed on
>>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted:
>>>>>>>>>>>>>>>>>>>>>>>>>> parent
>>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify
>>>>>>>>>>>>>>>>>>>>>>>>>> failed on
>>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted:
>>>>>>>>>>>>>>>>>>>>>>>>>> parent
>>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify
>>>>>>>>>>>>>>>>>>>>>>>>>> failed on
>>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted:
>>>>>>>>>>>>>>>>>>>>>>>>>> parent
>>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify
>>>>>>>>>>>>>>>>>>>>>>>>>> failed on
>>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted:
>>>>>>>>>>>>>>>>>>>>>>>>>> parent
>>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify
>>>>>>>>>>>>>>>>>>>>>>>>>> failed on
>>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted:
>>>>>>>>>>>>>>>>>>>>>>>>>> parent
>>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify
>>>>>>>>>>>>>>>>>>>>>>>>>> failed on
>>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted:
>>>>>>>>>>>>>>>>>>>>>>>>>> parent
>>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify
>>>>>>>>>>>>>>>>>>>>>>>>>> failed on
>>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted:
>>>>>>>>>>>>>>>>>>>>>>>>>> parent
>>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent transid verify
>>>>>>>>>>>>>>>>>>>>>>>>>> failed on
>>>>>>>>>>>>>>>>>>>>>>>>>> 218620880703488 wanted
>>>>>>>>>>>>>>>>>>>>>>>>>> 6875841 found 6876224
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Ignoring transid failure
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: child eb corrupted:
>>>>>>>>>>>>>>>>>>>>>>>>>> parent
>>>>>>>>>>>>>>>>>>>>>>>>>> bytenr=225049956061184
>>>>>>>>>>>>>>>>>>>>>>>>>> item=84
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> parent level=1
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> child level=4
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: failed to zero log
>>>>>>>>>>>>>>>>>>>>>>>>>> tree: -17
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> ERROR: attempt to start
>>>>>>>>>>>>>>>>>>>>>>>>>> transaction
>>>>>>>>>>>>>>>>>>>>>>>>>> over already running one
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: reserved space
>>>>>>>>>>>>>>>>>>>>>>>>>> leaked,
>>>>>>>>>>>>>>>>>>>>>>>>>> flag=0x4 bytes_reserved=4096
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak
>>>>>>>>>>>>>>>>>>>>>>>>>> (aborted
>>>>>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>>>>> 225049066086400 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak
>>>>>>>>>>>>>>>>>>>>>>>>>> (aborted
>>>>>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>>>>> 225049066094592 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak
>>>>>>>>>>>>>>>>>>>>>>>>>> (aborted
>>>>>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>>>>> 225049066102784 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> extent buffer leak: start
>>>>>>>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> WARNING: dirty eb leak
>>>>>>>>>>>>>>>>>>>>>>>>>> (aborted
>>>>>>>>>>>>>>>>>>>>>>>>>> trans):
>>>>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>>>>> 225049066131456 len 4096
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> What is going on?
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> On Wed, May 6, 2020 at
>>>>>>>>>>>>>>>>>>>>>>>>>> 9:30 PM
>>>>>>>>>>>>>>>>>>>>>>>>>> Tyler
>>>>>>>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>>>>>>>> <t.d.richmond@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>> <mailto:t.d.richmond@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> Chris, I had used the
>>>>>>>>>>>>>>>>>>>>>>>>>> correct
>>>>>>>>>>>>>>>>>>>>>>>>>> mountpoint in the command.
>>>>>>>>>>>>>>>>>>>>>>>>>> I just edited
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> it in the email to be
>>>>>>>>>>>>>>>>>>>>>>>>>> /mountpoint for
>>>>>>>>>>>>>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> Qu, I'll try the repair.
>>>>>>>>>>>>>>>>>>>>>>>>>> Fingers
>>>>>>>>>>>>>>>>>>>>>>>>>> crossed!
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> On Wed, May 6, 2020 at 9:13
>>>>>>>>>>>>>>>>>>>>>>>>>> PM Qu
>>>>>>>>>>>>>>>>>>>>>>>>>> Wenruo
>>>>>>>>>>>>>>>>>>>>>>>>>> <quwenruo.btrfs@gmx.com
>>>>>>>>>>>>>>>>>>>>>>>>>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> On 2020/5/7 上午5:54, Tyler
>>>>>>>>>>>>>>>>>>>>>>>>>> Richmond
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> I looked up this error
>>>>>>>>>>>>>>>>>>>>>>>>>> and it
>>>>>>>>>>>>>>>>>>>>>>>>>> basically says ask a
>>>>>>>>>>>>>>>>>>>>>>>>>> developer to
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> determine if it's a false
>>>>>>>>>>>>>>>>>>>>>>>>>> error or
>>>>>>>>>>>>>>>>>>>>>>>>>> not. I just started
>>>>>>>>>>>>>>>>>>>>>>>>>> getting some
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> slow response times, and
>>>>>>>>>>>>>>>>>>>>>>>>>> looked at
>>>>>>>>>>>>>>>>>>>>>>>>>> the dmesg log to
>>>>>>>>>>>>>>>>>>>>>>>>>> find a ton of
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> these errors.
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.446299] BTRFS
>>>>>>>>>>>>>>>>>>>>>>>>>> critical
>>>>>>>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840
>>>>>>>>>>>>>>>>>>>>>>>>>> slot=4
>>>>>>>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492
>>>>>>>>>>>>>>>>>>>>>>>>>> expect [0,
>>>>>>>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.449823] BTRFS
>>>>>>>>>>>>>>>>>>>>>>>>>> error
>>>>>>>>>>>>>>>>>>>>>>>>>> (device
>>>>>>>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption
>>>>>>>>>>>>>>>>>>>>>>>>>> detected
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.459238] BTRFS
>>>>>>>>>>>>>>>>>>>>>>>>>> critical
>>>>>>>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840
>>>>>>>>>>>>>>>>>>>>>>>>>> slot=4
>>>>>>>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492
>>>>>>>>>>>>>>>>>>>>>>>>>> expect [0,
>>>>>>>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.462773] BTRFS
>>>>>>>>>>>>>>>>>>>>>>>>>> error
>>>>>>>>>>>>>>>>>>>>>>>>>> (device
>>>>>>>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption
>>>>>>>>>>>>>>>>>>>>>>>>>> detected
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.464711] BTRFS
>>>>>>>>>>>>>>>>>>>>>>>>>> critical
>>>>>>>>>>>>>>>>>>>>>>>>>> (device sdh): corrupt
>>>>>>>>>>>>>>>>>>>>>>>>>> leaf: root=5
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> block=203510940835840
>>>>>>>>>>>>>>>>>>>>>>>>>> slot=4
>>>>>>>>>>>>>>>>>>>>>>>>>> ino=1311670, invalid inode
>>>>>>>>>>>>>>>>>>>>>>>>>> generation:
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> has 18446744073709551492
>>>>>>>>>>>>>>>>>>>>>>>>>> expect [0,
>>>>>>>>>>>>>>>>>>>>>>>>>> 6875827]
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> [192088.468457] BTRFS
>>>>>>>>>>>>>>>>>>>>>>>>>> error
>>>>>>>>>>>>>>>>>>>>>>>>>> (device
>>>>>>>>>>>>>>>>>>>>>>>>>> sdh):
>>>>>>>>>>>>>>>>>>>>>>>>>> block=203510940835840 read
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> time tree block corruption
>>>>>>>>>>>>>>>>>>>>>>>>>> detected
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> btrfs device stats,
>>>>>>>>>>>>>>>>>>>>>>>>>> however,
>>>>>>>>>>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>>>>>>>>> show any errors.
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Is there anything I
>>>>>>>>>>>>>>>>>>>>>>>>>> should do
>>>>>>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>>>> this, or should I
>>>>>>>>>>>>>>>>>>>>>>>>>> just continue
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> using my array as normal?
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> This is caused by older
>>>>>>>>>>>>>>>>>>>>>>>>>> kernel
>>>>>>>>>>>>>>>>>>>>>>>>>> underflow inode generation.
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Latest btrfs-progs can
>>>>>>>>>>>>>>>>>>>>>>>>>> fix it,
>>>>>>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>>>>>>> btrfs check --repair.
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Or you can go safer, by
>>>>>>>>>>>>>>>>>>>>>>>>>> manually
>>>>>>>>>>>>>>>>>>>>>>>>>> locating the inode
>>>>>>>>>>>>>>>>>>>>>>>>>> using its inode
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> number (1311670), and
>>>>>>>>>>>>>>>>>>>>>>>>>> copy it
>>>>>>>>>>>>>>>>>>>>>>>>>> to some
>>>>>>>>>>>>>>>>>>>>>>>>>> new location using
>>>>>>>>>>>>>>>>>>>>>>>>>> previous
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> working kernel, then
>>>>>>>>>>>>>>>>>>>>>>>>>> delete the
>>>>>>>>>>>>>>>>>>>>>>>>>> old
>>>>>>>>>>>>>>>>>>>>>>>>>> file, copy the new
>>>>>>>>>>>>>>>>>>>>>>>>>> one back to fix it.
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>> Qu
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-06 10:32 ` Qu Wenruo
@ 2020-11-07 11:18 ` Ferry Toth
2020-11-07 11:35 ` Qu Wenruo
0 siblings, 1 reply; 56+ messages in thread
From: Ferry Toth @ 2020-11-07 11:18 UTC (permalink / raw)
To: Qu Wenruo, Qu Wenruo, Tyler Richmond; +Cc: Btrfs BTRFS
Op 06-11-2020 om 11:32 schreef Qu Wenruo:
>
>
> On 2020/11/6 下午6:30, Ferry Toth wrote:
>> Hi
>>
>> Op 06-11-2020 om 11:24 schreef Qu Wenruo:
>>>
>>> On 2020/11/6 下午6:09, Ferry Toth wrote:
>>>> Hi Qu
>>>>
>>>> Op 06-11-2020 om 00:40 schreef Qu Wenruo:
>>>>> On 2020/11/6 上午7:37, Ferry Toth wrote:
>>>>>> Hi
>>>>>>
>>>>>> Op 06-11-2020 om 00:32 schreef Qu Wenruo:
>>>>>>> On 2020/11/6 上午7:12, Ferry Toth wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Op 06-11-2020 om 00:00 schreef Qu Wenruo:
>>>>>>>>> On 2020/11/6 上午4:08, Ferry Toth wrote:
>>>>>>>>>> I am in a similar spot, during updating my distro (Kubuntu), I am
>>>>>>>>>> unable
>>>>>>>>>> to update a certain package. I know which file it is:
>>>>>>>>>>
>>>>>>>>>> ~$ ls -l /usr/share/doc/libatk1.0-data
>>>>>>>>>> ls: kan geen toegang krijgen tot '/usr/share/doc/libatk1.0-data':
>>>>>>>>>> Invoer-/uitvoerfout
>>>>>>>>>>
>>>>>>>>>> This creates the following in journal:
>>>>>>>>>>
>>>>>>>>>> kernel: BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>>>>>> block=1169152675840 slot=1 ino=915987, invalid inode
>>>>>>>>>> generation: has
>>>>>>>>>> 18446744073709551492 expect [0, 5851353]
>>>>>>>>>> kernel: BTRFS error (device sda2): block=1169152675840 read time
>>>>>>>>>> tree
>>>>>>>>>> block corruption detected
>>>>>>>>>>
>>>>>>>>>> Now, the problem: this file is on my rootfs, which is mounted. apt
>>>>>>>>>> (distribution updated) installed all packages but can't continue
>>>>>>>>>> configuring, because libatk is a dependancy. I can't delete the
>>>>>>>>>> file
>>>>>>>>>> because of the I/O error. And btrfs check complains (I tried
>>>>>>>>>> running RO)
>>>>>>>>>> because the file system is mounted.
>>>>>>>>>>
>>>>>>>>>> But, on the sunny side, the file system is not RO.
>>>>>>>>>>
>>>>>>>>>> Is there any way to forcefully remove the file? Or do you have a
>>>>>>>>>> recommendation how to proceed?
>>>>>>>>> Newer kernel will reject to even read the item, thus will not be
>>>>>>>>> able to
>>>>>>>>> remove it.
>>>>>>>> That's already the case. (input / output error)
>>>>>>>>> I guess you have to use some distro ISO to fix the fs.
>>>>>>>> And then? btrfs check --repair the disk offline?
>>>>>>> Yep.
>>>>>>>
>>>>>>> You would want the latest btrfs-progs though.
>>>>>> Groovy has 5.7. Would that be good enough? Otherwise will be difficult
>>>>>> to build on/for live usb image.
>>>>> For your particular case, the fix are already in btrfs-progs v5.4.
>>>>>
>>>>> Although newer is always better, just in case you have extent item
>>>>> generation corruption, you may want v5.4.1.
>>>>>
>>>>> So your v5.7 should be good enough.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>> I made a live usb and performed:
>>>>
>>>> btrfs check --repair /dev/sda2
>>>>
>>>> It found errors and fixed them. However, it did not fix the corrupt
>>>> leaf. The file is actually a directory:
>>>>
>>>> ~$ stat /usr/share/doc/libatk1.0-data
>>>> stat: cannot statx '/usr/share/doc/libatk1.0-data': Invoer-/uitvoerfout
>>>>
>>>> in journal:
>>>>
>>>> BTRFS critical (device sda2): corrupt leaf: root=294 block=1169152675840
>>>> slot=1 ino=915987, invalid inode generation: has 18446744073709551492
>>>> expect [0, 5852829]
>>>> BTRFS error (device sda2): block=1169152675840 read time tree block
>>>> corruption detected
>>>>
>>>> So how do I repair this? Am I doing something wrong?
>>> Please provide the following dump:
>>> btrfs ins dump-tree -b 1169152675840 /dev/sda2
>>>
>>> Feel free to remove the filenames in the dump.
>> sudo btrfs ins dump-tree -b 1169152675840 /dev/sda2
>> btrfs-progs v5.3-rc1
>> leaf 1169152675840 items 36 free space 966 generation 5431733 owner 294
>> leaf 1169152675840 flags 0x1(WRITTEN) backref revision 1
>> fs uuid 27155120-9ef8-47fb-b248-eaac2b7c8375
>> chunk uuid 5704f1ba-08fd-4f6b-9117-0e080b4e9ef0
>> item 0 key (915986 DIR_INDEX 2) itemoff 3957 itemsize 38
>> location key (915987 INODE_ITEM 0) type FILE
>> transid 7782235549259005952 data_len 0 name_len 8
>> name: smb.conf
>> item 1 key (915987 INODE_ITEM 0) itemoff 3797 itemsize 160
>> generation 1 transid 18446744073709551492 size 12464
>> nbytes 16384
> Yeah, corrupted transid.
>
> The v5.6 kernel doesn't get the fix backported...
>
> Now you have to use either the out-of-tree branch, or David's devel
> branch to build a btrfs-progs which is able to repair the transid error.
>
> Thanks,
> Qu
>
Just be to be clear, I tried to repair with the Kubuntu Groovy Live usb,
which has linux 5.8 and btrfs-progs 5.7.
I didn't fix the above transid, above was taken after booting normally
again (linux 5.8), unfortunately with btrfs-progs v5.3-rc1 (that I built
a year ago). See the other post for the result with btrfs-progs 5.7.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-07 11:18 ` Ferry Toth
@ 2020-11-07 11:35 ` Qu Wenruo
2020-11-07 13:19 ` Ferry Toth
0 siblings, 1 reply; 56+ messages in thread
From: Qu Wenruo @ 2020-11-07 11:35 UTC (permalink / raw)
To: Ferry Toth, Qu Wenruo, Tyler Richmond; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 5370 bytes --]
On 2020/11/7 下午7:18, Ferry Toth wrote:
> Op 06-11-2020 om 11:32 schreef Qu Wenruo:
>>
>>
>> On 2020/11/6 下午6:30, Ferry Toth wrote:
>>> Hi
>>>
>>> Op 06-11-2020 om 11:24 schreef Qu Wenruo:
>>>>
>>>> On 2020/11/6 下午6:09, Ferry Toth wrote:
>>>>> Hi Qu
>>>>>
>>>>> Op 06-11-2020 om 00:40 schreef Qu Wenruo:
>>>>>> On 2020/11/6 上午7:37, Ferry Toth wrote:
>>>>>>> Hi
>>>>>>>
>>>>>>> Op 06-11-2020 om 00:32 schreef Qu Wenruo:
>>>>>>>> On 2020/11/6 上午7:12, Ferry Toth wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Op 06-11-2020 om 00:00 schreef Qu Wenruo:
>>>>>>>>>> On 2020/11/6 上午4:08, Ferry Toth wrote:
>>>>>>>>>>> I am in a similar spot, during updating my distro (Kubuntu),
>>>>>>>>>>> I am
>>>>>>>>>>> unable
>>>>>>>>>>> to update a certain package. I know which file it is:
>>>>>>>>>>>
>>>>>>>>>>> ~$ ls -l /usr/share/doc/libatk1.0-data
>>>>>>>>>>> ls: kan geen toegang krijgen tot
>>>>>>>>>>> '/usr/share/doc/libatk1.0-data':
>>>>>>>>>>> Invoer-/uitvoerfout
>>>>>>>>>>>
>>>>>>>>>>> This creates the following in journal:
>>>>>>>>>>>
>>>>>>>>>>> kernel: BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>>>>>>> block=1169152675840 slot=1 ino=915987, invalid inode
>>>>>>>>>>> generation: has
>>>>>>>>>>> 18446744073709551492 expect [0, 5851353]
>>>>>>>>>>> kernel: BTRFS error (device sda2): block=1169152675840 read time
>>>>>>>>>>> tree
>>>>>>>>>>> block corruption detected
>>>>>>>>>>>
>>>>>>>>>>> Now, the problem: this file is on my rootfs, which is
>>>>>>>>>>> mounted. apt
>>>>>>>>>>> (distribution updated) installed all packages but can't continue
>>>>>>>>>>> configuring, because libatk is a dependancy. I can't delete the
>>>>>>>>>>> file
>>>>>>>>>>> because of the I/O error. And btrfs check complains (I tried
>>>>>>>>>>> running RO)
>>>>>>>>>>> because the file system is mounted.
>>>>>>>>>>>
>>>>>>>>>>> But, on the sunny side, the file system is not RO.
>>>>>>>>>>>
>>>>>>>>>>> Is there any way to forcefully remove the file? Or do you have a
>>>>>>>>>>> recommendation how to proceed?
>>>>>>>>>> Newer kernel will reject to even read the item, thus will not be
>>>>>>>>>> able to
>>>>>>>>>> remove it.
>>>>>>>>> That's already the case. (input / output error)
>>>>>>>>>> I guess you have to use some distro ISO to fix the fs.
>>>>>>>>> And then? btrfs check --repair the disk offline?
>>>>>>>> Yep.
>>>>>>>>
>>>>>>>> You would want the latest btrfs-progs though.
>>>>>>> Groovy has 5.7. Would that be good enough? Otherwise will be
>>>>>>> difficult
>>>>>>> to build on/for live usb image.
>>>>>> For your particular case, the fix are already in btrfs-progs v5.4.
>>>>>>
>>>>>> Although newer is always better, just in case you have extent item
>>>>>> generation corruption, you may want v5.4.1.
>>>>>>
>>>>>> So your v5.7 should be good enough.
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>> I made a live usb and performed:
>>>>>
>>>>> btrfs check --repair /dev/sda2
>>>>>
>>>>> It found errors and fixed them. However, it did not fix the corrupt
>>>>> leaf. The file is actually a directory:
>>>>>
>>>>> ~$ stat /usr/share/doc/libatk1.0-data
>>>>> stat: cannot statx '/usr/share/doc/libatk1.0-data':
>>>>> Invoer-/uitvoerfout
>>>>>
>>>>> in journal:
>>>>>
>>>>> BTRFS critical (device sda2): corrupt leaf: root=294
>>>>> block=1169152675840
>>>>> slot=1 ino=915987, invalid inode generation: has 18446744073709551492
>>>>> expect [0, 5852829]
>>>>> BTRFS error (device sda2): block=1169152675840 read time tree block
>>>>> corruption detected
>>>>>
>>>>> So how do I repair this? Am I doing something wrong?
>>>> Please provide the following dump:
>>>> btrfs ins dump-tree -b 1169152675840 /dev/sda2
>>>>
>>>> Feel free to remove the filenames in the dump.
>>> sudo btrfs ins dump-tree -b 1169152675840 /dev/sda2
>>> btrfs-progs v5.3-rc1
>>> leaf 1169152675840 items 36 free space 966 generation 5431733 owner 294
>>> leaf 1169152675840 flags 0x1(WRITTEN) backref revision 1
>>> fs uuid 27155120-9ef8-47fb-b248-eaac2b7c8375
>>> chunk uuid 5704f1ba-08fd-4f6b-9117-0e080b4e9ef0
>>> item 0 key (915986 DIR_INDEX 2) itemoff 3957 itemsize 38
>>> location key (915987 INODE_ITEM 0) type FILE
>>> transid 7782235549259005952 data_len 0 name_len 8
>>> name: smb.conf
>>> item 1 key (915987 INODE_ITEM 0) itemoff 3797 itemsize 160
>>> generation 1 transid 18446744073709551492 size 12464
>>> nbytes 16384
>> Yeah, corrupted transid.
>>
>> The v5.6 kernel doesn't get the fix backported...
>>
>> Now you have to use either the out-of-tree branch, or David's devel
>> branch to build a btrfs-progs which is able to repair the transid error.
>>
>> Thanks,
>> Qu
>>
> Just be to be clear, I tried to repair with the Kubuntu Groovy Live usb,
> which has linux 5.8 and btrfs-progs 5.7.
>
> I didn't fix the above transid, above was taken after booting normally
> again (linux 5.8), unfortunately with btrfs-progs v5.3-rc1 (that I built
> a year ago). See the other post for the result with btrfs-progs 5.7.
>
>
As I said already, you need either the devel branch to do the fix.
Current release btrfs-progs hasn't the repair ability merged.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-07 11:35 ` Qu Wenruo
@ 2020-11-07 13:19 ` Ferry Toth
2020-11-07 13:28 ` Qu Wenruo
0 siblings, 1 reply; 56+ messages in thread
From: Ferry Toth @ 2020-11-07 13:19 UTC (permalink / raw)
To: Qu Wenruo, Qu Wenruo, Tyler Richmond; +Cc: Btrfs BTRFS
Hi
Op 07-11-2020 om 12:35 schreef Qu Wenruo:
>
> On 2020/11/7 下午7:18, Ferry Toth wrote:
>> Op 06-11-2020 om 11:32 schreef Qu Wenruo:
>>>
>>> On 2020/11/6 下午6:30, Ferry Toth wrote:
>>>> Hi
>>>>
>>>> Op 06-11-2020 om 11:24 schreef Qu Wenruo:
>>>>> On 2020/11/6 下午6:09, Ferry Toth wrote:
>>>>>> Hi Qu
>>>>>>
>>>>>> Op 06-11-2020 om 00:40 schreef Qu Wenruo:
>>>>>>> On 2020/11/6 上午7:37, Ferry Toth wrote:
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> Op 06-11-2020 om 00:32 schreef Qu Wenruo:
>>>>>>>>> On 2020/11/6 上午7:12, Ferry Toth wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Op 06-11-2020 om 00:00 schreef Qu Wenruo:
>>>>>>>>>>> On 2020/11/6 上午4:08, Ferry Toth wrote:
>>>>>>>>>>>> I am in a similar spot, during updating my distro (Kubuntu),
>>>>>>>>>>>> I am
>>>>>>>>>>>> unable
>>>>>>>>>>>> to update a certain package. I know which file it is:
>>>>>>>>>>>>
>>>>>>>>>>>> ~$ ls -l /usr/share/doc/libatk1.0-data
>>>>>>>>>>>> ls: kan geen toegang krijgen tot
>>>>>>>>>>>> '/usr/share/doc/libatk1.0-data':
>>>>>>>>>>>> Invoer-/uitvoerfout
>>>>>>>>>>>>
>>>>>>>>>>>> This creates the following in journal:
>>>>>>>>>>>>
>>>>>>>>>>>> kernel: BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>>>>>>>> block=1169152675840 slot=1 ino=915987, invalid inode
>>>>>>>>>>>> generation: has
>>>>>>>>>>>> 18446744073709551492 expect [0, 5851353]
>>>>>>>>>>>> kernel: BTRFS error (device sda2): block=1169152675840 read time
>>>>>>>>>>>> tree
>>>>>>>>>>>> block corruption detected
>>>>>>>>>>>>
>>>>>>>>>>>> Now, the problem: this file is on my rootfs, which is
>>>>>>>>>>>> mounted. apt
>>>>>>>>>>>> (distribution updated) installed all packages but can't continue
>>>>>>>>>>>> configuring, because libatk is a dependancy. I can't delete the
>>>>>>>>>>>> file
>>>>>>>>>>>> because of the I/O error. And btrfs check complains (I tried
>>>>>>>>>>>> running RO)
>>>>>>>>>>>> because the file system is mounted.
>>>>>>>>>>>>
>>>>>>>>>>>> But, on the sunny side, the file system is not RO.
>>>>>>>>>>>>
>>>>>>>>>>>> Is there any way to forcefully remove the file? Or do you have a
>>>>>>>>>>>> recommendation how to proceed?
>>>>>>>>>>> Newer kernel will reject to even read the item, thus will not be
>>>>>>>>>>> able to
>>>>>>>>>>> remove it.
>>>>>>>>>> That's already the case. (input / output error)
>>>>>>>>>>> I guess you have to use some distro ISO to fix the fs.
>>>>>>>>>> And then? btrfs check --repair the disk offline?
>>>>>>>>> Yep.
>>>>>>>>>
>>>>>>>>> You would want the latest btrfs-progs though.
>>>>>>>> Groovy has 5.7. Would that be good enough? Otherwise will be
>>>>>>>> difficult
>>>>>>>> to build on/for live usb image.
>>>>>>> For your particular case, the fix are already in btrfs-progs v5.4.
>>>>>>>
>>>>>>> Although newer is always better, just in case you have extent item
>>>>>>> generation corruption, you may want v5.4.1.
>>>>>>>
>>>>>>> So your v5.7 should be good enough.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Qu
>>>>>> I made a live usb and performed:
>>>>>>
>>>>>> btrfs check --repair /dev/sda2
>>>>>>
>>>>>> It found errors and fixed them. However, it did not fix the corrupt
>>>>>> leaf. The file is actually a directory:
>>>>>>
>>>>>> ~$ stat /usr/share/doc/libatk1.0-data
>>>>>> stat: cannot statx '/usr/share/doc/libatk1.0-data':
>>>>>> Invoer-/uitvoerfout
>>>>>>
>>>>>> in journal:
>>>>>>
>>>>>> BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>> block=1169152675840
>>>>>> slot=1 ino=915987, invalid inode generation: has 18446744073709551492
>>>>>> expect [0, 5852829]
>>>>>> BTRFS error (device sda2): block=1169152675840 read time tree block
>>>>>> corruption detected
>>>>>>
>>>>>> So how do I repair this? Am I doing something wrong?
>>>>> Please provide the following dump:
>>>>> btrfs ins dump-tree -b 1169152675840 /dev/sda2
>>>>>
>>>>> Feel free to remove the filenames in the dump.
>>>> sudo btrfs ins dump-tree -b 1169152675840 /dev/sda2
>>>> btrfs-progs v5.3-rc1
>>>> leaf 1169152675840 items 36 free space 966 generation 5431733 owner 294
>>>> leaf 1169152675840 flags 0x1(WRITTEN) backref revision 1
>>>> fs uuid 27155120-9ef8-47fb-b248-eaac2b7c8375
>>>> chunk uuid 5704f1ba-08fd-4f6b-9117-0e080b4e9ef0
>>>> item 0 key (915986 DIR_INDEX 2) itemoff 3957 itemsize 38
>>>> location key (915987 INODE_ITEM 0) type FILE
>>>> transid 7782235549259005952 data_len 0 name_len 8
>>>> name: smb.conf
>>>> item 1 key (915987 INODE_ITEM 0) itemoff 3797 itemsize 160
>>>> generation 1 transid 18446744073709551492 size 12464
>>>> nbytes 16384
>>> Yeah, corrupted transid.
>>>
>>> The v5.6 kernel doesn't get the fix backported...
>>>
>>> Now you have to use either the out-of-tree branch, or David's devel
>>> branch to build a btrfs-progs which is able to repair the transid error.
>>>
>>> Thanks,
>>> Qu
>>>
>> Just be to be clear, I tried to repair with the Kubuntu Groovy Live usb,
>> which has linux 5.8 and btrfs-progs 5.7.
>>
>> I didn't fix the above transid, above was taken after booting normally
>> again (linux 5.8), unfortunately with btrfs-progs v5.3-rc1 (that I built
>> a year ago). See the other post for the result with btrfs-progs 5.7.
>>
>>
> As I said already, you need either the devel branch to do the fix.
> Current release btrfs-progs hasn't the repair ability merged.
>
Ah, I understood wrong. I thought 5.7 was enough.
So, I need to build the latest and greatest and install on live usb. Or
I need to wait for a future live usb with this incorporated.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-07 13:19 ` Ferry Toth
@ 2020-11-07 13:28 ` Qu Wenruo
2020-11-07 19:50 ` Ferry Toth
2020-11-16 10:41 ` Ferry Toth
0 siblings, 2 replies; 56+ messages in thread
From: Qu Wenruo @ 2020-11-07 13:28 UTC (permalink / raw)
To: Ferry Toth, Qu Wenruo, Tyler Richmond; +Cc: Btrfs BTRFS
On 2020/11/7 下午9:19, Ferry Toth wrote:
> Hi
>
> Op 07-11-2020 om 12:35 schreef Qu Wenruo:
>>
>> On 2020/11/7 下午7:18, Ferry Toth wrote:
>>> Op 06-11-2020 om 11:32 schreef Qu Wenruo:
>>>>
>>>> On 2020/11/6 下午6:30, Ferry Toth wrote:
>>>>> Hi
>>>>>
>>>>> Op 06-11-2020 om 11:24 schreef Qu Wenruo:
>>>>>> On 2020/11/6 下午6:09, Ferry Toth wrote:
>>>>>>> Hi Qu
>>>>>>>
>>>>>>> Op 06-11-2020 om 00:40 schreef Qu Wenruo:
>>>>>>>> On 2020/11/6 上午7:37, Ferry Toth wrote:
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> Op 06-11-2020 om 00:32 schreef Qu Wenruo:
>>>>>>>>>> On 2020/11/6 上午7:12, Ferry Toth wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Op 06-11-2020 om 00:00 schreef Qu Wenruo:
>>>>>>>>>>>> On 2020/11/6 上午4:08, Ferry Toth wrote:
>>>>>>>>>>>>> I am in a similar spot, during updating my distro (Kubuntu),
>>>>>>>>>>>>> I am
>>>>>>>>>>>>> unable
>>>>>>>>>>>>> to update a certain package. I know which file it is:
>>>>>>>>>>>>>
>>>>>>>>>>>>> ~$ ls -l /usr/share/doc/libatk1.0-data
>>>>>>>>>>>>> ls: kan geen toegang krijgen tot
>>>>>>>>>>>>> '/usr/share/doc/libatk1.0-data':
>>>>>>>>>>>>> Invoer-/uitvoerfout
>>>>>>>>>>>>>
>>>>>>>>>>>>> This creates the following in journal:
>>>>>>>>>>>>>
>>>>>>>>>>>>> kernel: BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>>>>>>>>> block=1169152675840 slot=1 ino=915987, invalid inode
>>>>>>>>>>>>> generation: has
>>>>>>>>>>>>> 18446744073709551492 expect [0, 5851353]
>>>>>>>>>>>>> kernel: BTRFS error (device sda2): block=1169152675840 read
>>>>>>>>>>>>> time
>>>>>>>>>>>>> tree
>>>>>>>>>>>>> block corruption detected
>>>>>>>>>>>>>
>>>>>>>>>>>>> Now, the problem: this file is on my rootfs, which is
>>>>>>>>>>>>> mounted. apt
>>>>>>>>>>>>> (distribution updated) installed all packages but can't
>>>>>>>>>>>>> continue
>>>>>>>>>>>>> configuring, because libatk is a dependancy. I can't delete
>>>>>>>>>>>>> the
>>>>>>>>>>>>> file
>>>>>>>>>>>>> because of the I/O error. And btrfs check complains (I tried
>>>>>>>>>>>>> running RO)
>>>>>>>>>>>>> because the file system is mounted.
>>>>>>>>>>>>>
>>>>>>>>>>>>> But, on the sunny side, the file system is not RO.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is there any way to forcefully remove the file? Or do you
>>>>>>>>>>>>> have a
>>>>>>>>>>>>> recommendation how to proceed?
>>>>>>>>>>>> Newer kernel will reject to even read the item, thus will
>>>>>>>>>>>> not be
>>>>>>>>>>>> able to
>>>>>>>>>>>> remove it.
>>>>>>>>>>> That's already the case. (input / output error)
>>>>>>>>>>>> I guess you have to use some distro ISO to fix the fs.
>>>>>>>>>>> And then? btrfs check --repair the disk offline?
>>>>>>>>>> Yep.
>>>>>>>>>>
>>>>>>>>>> You would want the latest btrfs-progs though.
>>>>>>>>> Groovy has 5.7. Would that be good enough? Otherwise will be
>>>>>>>>> difficult
>>>>>>>>> to build on/for live usb image.
>>>>>>>> For your particular case, the fix are already in btrfs-progs v5.4.
>>>>>>>>
>>>>>>>> Although newer is always better, just in case you have extent item
>>>>>>>> generation corruption, you may want v5.4.1.
>>>>>>>>
>>>>>>>> So your v5.7 should be good enough.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Qu
>>>>>>> I made a live usb and performed:
>>>>>>>
>>>>>>> btrfs check --repair /dev/sda2
>>>>>>>
>>>>>>> It found errors and fixed them. However, it did not fix the corrupt
>>>>>>> leaf. The file is actually a directory:
>>>>>>>
>>>>>>> ~$ stat /usr/share/doc/libatk1.0-data
>>>>>>> stat: cannot statx '/usr/share/doc/libatk1.0-data':
>>>>>>> Invoer-/uitvoerfout
>>>>>>>
>>>>>>> in journal:
>>>>>>>
>>>>>>> BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>>> block=1169152675840
>>>>>>> slot=1 ino=915987, invalid inode generation: has
>>>>>>> 18446744073709551492
>>>>>>> expect [0, 5852829]
>>>>>>> BTRFS error (device sda2): block=1169152675840 read time tree block
>>>>>>> corruption detected
>>>>>>>
>>>>>>> So how do I repair this? Am I doing something wrong?
>>>>>> Please provide the following dump:
>>>>>> btrfs ins dump-tree -b 1169152675840 /dev/sda2
>>>>>>
>>>>>> Feel free to remove the filenames in the dump.
>>>>> sudo btrfs ins dump-tree -b 1169152675840 /dev/sda2
>>>>> btrfs-progs v5.3-rc1
>>>>> leaf 1169152675840 items 36 free space 966 generation 5431733 owner
>>>>> 294
>>>>> leaf 1169152675840 flags 0x1(WRITTEN) backref revision 1
>>>>> fs uuid 27155120-9ef8-47fb-b248-eaac2b7c8375
>>>>> chunk uuid 5704f1ba-08fd-4f6b-9117-0e080b4e9ef0
>>>>> item 0 key (915986 DIR_INDEX 2) itemoff 3957 itemsize 38
>>>>> location key (915987 INODE_ITEM 0) type FILE
>>>>> transid 7782235549259005952 data_len 0 name_len 8
>>>>> name: smb.conf
>>>>> item 1 key (915987 INODE_ITEM 0) itemoff 3797 itemsize 160
>>>>> generation 1 transid 18446744073709551492 size 12464
>>>>> nbytes 16384
>>>> Yeah, corrupted transid.
>>>>
>>>> The v5.6 kernel doesn't get the fix backported...
>>>>
>>>> Now you have to use either the out-of-tree branch, or David's devel
>>>> branch to build a btrfs-progs which is able to repair the transid
>>>> error.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>> Just be to be clear, I tried to repair with the Kubuntu Groovy Live usb,
>>> which has linux 5.8 and btrfs-progs 5.7.
>>>
>>> I didn't fix the above transid, above was taken after booting normally
>>> again (linux 5.8), unfortunately with btrfs-progs v5.3-rc1 (that I built
>>> a year ago). See the other post for the result with btrfs-progs 5.7.
>>>
>>>
>> As I said already, you need either the devel branch to do the fix.
>> Current release btrfs-progs hasn't the repair ability merged.
>>
> Ah, I understood wrong. I thought 5.7 was enough.
>
> So, I need to build the latest and greatest and install on live usb. Or
> I need to wait for a future live usb with this incorporated.
>
It's strongly recommended to build right now.
As you may already find, the btrfs-progs release cycle is normally
behind schedule, thus you may need to wait way longer than you thought...
BTW, after repairing, it's recommended to use newer enought (v5.8 should
be enough) kernel to mount the repaired fs, to verify the transid error
get fixed.
Thanks,
Qu
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
@ 2020-11-07 19:50 ` Ferry Toth
0 siblings, 0 replies; 56+ messages in thread
From: Ferry Toth @ 2020-11-07 19:50 UTC (permalink / raw)
To: Qu Wenruo, Qu Wenruo, Tyler Richmond; +Cc: Btrfs BTRFS
Hi,
Op 07-11-2020 om 14:28 schreef Qu Wenruo:
>
>
> On 2020/11/7 下午9:19, Ferry Toth wrote:
>> Hi
>>
>> Op 07-11-2020 om 12:35 schreef Qu Wenruo:
>>>
>>> On 2020/11/7 下午7:18, Ferry Toth wrote:
>>>> Op 06-11-2020 om 11:32 schreef Qu Wenruo:
>>>>>
>>>>> On 2020/11/6 下午6:30, Ferry Toth wrote:
>>>>>> Hi
>>>>>>
>>>>>> Op 06-11-2020 om 11:24 schreef Qu Wenruo:
>>>>>>> On 2020/11/6 下午6:09, Ferry Toth wrote:
>>>>>>>> Hi Qu
>>>>>>>>
>>>>>>>> Op 06-11-2020 om 00:40 schreef Qu Wenruo:
>>>>>>>>> On 2020/11/6 上午7:37, Ferry Toth wrote:
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> Op 06-11-2020 om 00:32 schreef Qu Wenruo:
>>>>>>>>>>> On 2020/11/6 上午7:12, Ferry Toth wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Op 06-11-2020 om 00:00 schreef Qu Wenruo:
>>>>>>>>>>>>> On 2020/11/6 上午4:08, Ferry Toth wrote:
>>>>>>>>>>>>>> I am in a similar spot, during updating my distro (Kubuntu),
>>>>>>>>>>>>>> I am
>>>>>>>>>>>>>> unable
>>>>>>>>>>>>>> to update a certain package. I know which file it is:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ~$ ls -l /usr/share/doc/libatk1.0-data
>>>>>>>>>>>>>> ls: kan geen toegang krijgen tot
>>>>>>>>>>>>>> '/usr/share/doc/libatk1.0-data':
>>>>>>>>>>>>>> Invoer-/uitvoerfout
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This creates the following in journal:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> kernel: BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>>>>>>>>>> block=1169152675840 slot=1 ino=915987, invalid inode
>>>>>>>>>>>>>> generation: has
>>>>>>>>>>>>>> 18446744073709551492 expect [0, 5851353]
>>>>>>>>>>>>>> kernel: BTRFS error (device sda2): block=1169152675840 read
>>>>>>>>>>>>>> time
>>>>>>>>>>>>>> tree
>>>>>>>>>>>>>> block corruption detected
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Now, the problem: this file is on my rootfs, which is
>>>>>>>>>>>>>> mounted. apt
>>>>>>>>>>>>>> (distribution updated) installed all packages but can't
>>>>>>>>>>>>>> continue
>>>>>>>>>>>>>> configuring, because libatk is a dependancy. I can't delete
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> file
>>>>>>>>>>>>>> because of the I/O error. And btrfs check complains (I tried
>>>>>>>>>>>>>> running RO)
>>>>>>>>>>>>>> because the file system is mounted.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But, on the sunny side, the file system is not RO.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is there any way to forcefully remove the file? Or do you
>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>> recommendation how to proceed?
>>>>>>>>>>>>> Newer kernel will reject to even read the item, thus will
>>>>>>>>>>>>> not be
>>>>>>>>>>>>> able to
>>>>>>>>>>>>> remove it.
>>>>>>>>>>>> That's already the case. (input / output error)
>>>>>>>>>>>>> I guess you have to use some distro ISO to fix the fs.
>>>>>>>>>>>> And then? btrfs check --repair the disk offline?
>>>>>>>>>>> Yep.
>>>>>>>>>>>
>>>>>>>>>>> You would want the latest btrfs-progs though.
>>>>>>>>>> Groovy has 5.7. Would that be good enough? Otherwise will be
>>>>>>>>>> difficult
>>>>>>>>>> to build on/for live usb image.
>>>>>>>>> For your particular case, the fix are already in btrfs-progs v5.4.
>>>>>>>>>
>>>>>>>>> Although newer is always better, just in case you have extent item
>>>>>>>>> generation corruption, you may want v5.4.1.
>>>>>>>>>
>>>>>>>>> So your v5.7 should be good enough.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Qu
>>>>>>>> I made a live usb and performed:
>>>>>>>>
>>>>>>>> btrfs check --repair /dev/sda2
>>>>>>>>
>>>>>>>> It found errors and fixed them. However, it did not fix the corrupt
>>>>>>>> leaf. The file is actually a directory:
>>>>>>>>
>>>>>>>> ~$ stat /usr/share/doc/libatk1.0-data
>>>>>>>> stat: cannot statx '/usr/share/doc/libatk1.0-data':
>>>>>>>> Invoer-/uitvoerfout
>>>>>>>>
>>>>>>>> in journal:
>>>>>>>>
>>>>>>>> BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>>>> block=1169152675840
>>>>>>>> slot=1 ino=915987, invalid inode generation: has
>>>>>>>> 18446744073709551492
>>>>>>>> expect [0, 5852829]
>>>>>>>> BTRFS error (device sda2): block=1169152675840 read time tree block
>>>>>>>> corruption detected
>>>>>>>>
>>>>>>>> So how do I repair this? Am I doing something wrong?
>>>>>>> Please provide the following dump:
>>>>>>> btrfs ins dump-tree -b 1169152675840 /dev/sda2
>>>>>>>
>>>>>>> Feel free to remove the filenames in the dump.
>>>>>> sudo btrfs ins dump-tree -b 1169152675840 /dev/sda2
>>>>>> btrfs-progs v5.3-rc1
>>>>>> leaf 1169152675840 items 36 free space 966 generation 5431733 owner
>>>>>> 294
>>>>>> leaf 1169152675840 flags 0x1(WRITTEN) backref revision 1
>>>>>> fs uuid 27155120-9ef8-47fb-b248-eaac2b7c8375
>>>>>> chunk uuid 5704f1ba-08fd-4f6b-9117-0e080b4e9ef0
>>>>>> item 0 key (915986 DIR_INDEX 2) itemoff 3957 itemsize 38
>>>>>> location key (915987 INODE_ITEM 0) type FILE
>>>>>> transid 7782235549259005952 data_len 0 name_len 8
>>>>>> name: smb.conf
>>>>>> item 1 key (915987 INODE_ITEM 0) itemoff 3797 itemsize 160
>>>>>> generation 1 transid 18446744073709551492 size 12464
>>>>>> nbytes 16384
>>>>> Yeah, corrupted transid.
>>>>>
>>>>> The v5.6 kernel doesn't get the fix backported...
>>>>>
>>>>> Now you have to use either the out-of-tree branch, or David's devel
>>>>> branch to build a btrfs-progs which is able to repair the transid
>>>>> error.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>> Just be to be clear, I tried to repair with the Kubuntu Groovy Live usb,
>>>> which has linux 5.8 and btrfs-progs 5.7.
>>>>
>>>> I didn't fix the above transid, above was taken after booting normally
>>>> again (linux 5.8), unfortunately with btrfs-progs v5.3-rc1 (that I built
>>>> a year ago). See the other post for the result with btrfs-progs 5.7.
>>>>
>>>>
>>> As I said already, you need either the devel branch to do the fix.
>>> Current release btrfs-progs hasn't the repair ability merged.
>>>
>> Ah, I understood wrong. I thought 5.7 was enough.
>>
>> So, I need to build the latest and greatest and install on live usb. Or
>> I need to wait for a future live usb with this incorporated.
>>
> It's strongly recommended to build right now.
>
> As you may already find, the btrfs-progs release cycle is normally
> behind schedule, thus you may need to wait way longer than you thought...
>
> BTW, after repairing, it's recommended to use newer enought (v5.8 should
> be enough) kernel to mount the repaired fs, to verify the transid error
> get fixed.
>
> Thanks,
> Qu
Thanks for the good advice. I found btrfs-progs 5.9 is in debian, and
quick test shows it doesn't do damage to install the debian package on
ubuntu.
This means that it is relatively easy to take the ubuntu live usb,
download the debian 5.9 package and test it.
But do I understand correctly you recommend to try to repair using the
development branch? That doesn't sound very safe :-)
I will report back here the results on monday.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
@ 2020-11-07 19:50 ` Ferry Toth
0 siblings, 0 replies; 56+ messages in thread
From: Ferry Toth @ 2020-11-07 19:50 UTC (permalink / raw)
To: linux-btrfs; +Cc: Btrfs BTRFS
Hi,
Op 07-11-2020 om 14:28 schreef Qu Wenruo:
>
>
> On 2020/11/7 下午9:19, Ferry Toth wrote:
>> Hi
>>
>> Op 07-11-2020 om 12:35 schreef Qu Wenruo:
>>>
>>> On 2020/11/7 下午7:18, Ferry Toth wrote:
>>>> Op 06-11-2020 om 11:32 schreef Qu Wenruo:
>>>>>
>>>>> On 2020/11/6 下午6:30, Ferry Toth wrote:
>>>>>> Hi
>>>>>>
>>>>>> Op 06-11-2020 om 11:24 schreef Qu Wenruo:
>>>>>>> On 2020/11/6 下午6:09, Ferry Toth wrote:
>>>>>>>> Hi Qu
>>>>>>>>
>>>>>>>> Op 06-11-2020 om 00:40 schreef Qu Wenruo:
>>>>>>>>> On 2020/11/6 上午7:37, Ferry Toth wrote:
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> Op 06-11-2020 om 00:32 schreef Qu Wenruo:
>>>>>>>>>>> On 2020/11/6 上午7:12, Ferry Toth wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Op 06-11-2020 om 00:00 schreef Qu Wenruo:
>>>>>>>>>>>>> On 2020/11/6 上午4:08, Ferry Toth wrote:
>>>>>>>>>>>>>> I am in a similar spot, during updating my distro (Kubuntu),
>>>>>>>>>>>>>> I am
>>>>>>>>>>>>>> unable
>>>>>>>>>>>>>> to update a certain package. I know which file it is:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ~$ ls -l /usr/share/doc/libatk1.0-data
>>>>>>>>>>>>>> ls: kan geen toegang krijgen tot
>>>>>>>>>>>>>> '/usr/share/doc/libatk1.0-data':
>>>>>>>>>>>>>> Invoer-/uitvoerfout
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This creates the following in journal:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> kernel: BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>>>>>>>>>> block=1169152675840 slot=1 ino=915987, invalid inode
>>>>>>>>>>>>>> generation: has
>>>>>>>>>>>>>> 18446744073709551492 expect [0, 5851353]
>>>>>>>>>>>>>> kernel: BTRFS error (device sda2): block=1169152675840 read
>>>>>>>>>>>>>> time
>>>>>>>>>>>>>> tree
>>>>>>>>>>>>>> block corruption detected
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Now, the problem: this file is on my rootfs, which is
>>>>>>>>>>>>>> mounted. apt
>>>>>>>>>>>>>> (distribution updated) installed all packages but can't
>>>>>>>>>>>>>> continue
>>>>>>>>>>>>>> configuring, because libatk is a dependancy. I can't delete
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> file
>>>>>>>>>>>>>> because of the I/O error. And btrfs check complains (I tried
>>>>>>>>>>>>>> running RO)
>>>>>>>>>>>>>> because the file system is mounted.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But, on the sunny side, the file system is not RO.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is there any way to forcefully remove the file? Or do you
>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>> recommendation how to proceed?
>>>>>>>>>>>>> Newer kernel will reject to even read the item, thus will
>>>>>>>>>>>>> not be
>>>>>>>>>>>>> able to
>>>>>>>>>>>>> remove it.
>>>>>>>>>>>> That's already the case. (input / output error)
>>>>>>>>>>>>> I guess you have to use some distro ISO to fix the fs.
>>>>>>>>>>>> And then? btrfs check --repair the disk offline?
>>>>>>>>>>> Yep.
>>>>>>>>>>>
>>>>>>>>>>> You would want the latest btrfs-progs though.
>>>>>>>>>> Groovy has 5.7. Would that be good enough? Otherwise will be
>>>>>>>>>> difficult
>>>>>>>>>> to build on/for live usb image.
>>>>>>>>> For your particular case, the fix are already in btrfs-progs v5.4.
>>>>>>>>>
>>>>>>>>> Although newer is always better, just in case you have extent item
>>>>>>>>> generation corruption, you may want v5.4.1.
>>>>>>>>>
>>>>>>>>> So your v5.7 should be good enough.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Qu
>>>>>>>> I made a live usb and performed:
>>>>>>>>
>>>>>>>> btrfs check --repair /dev/sda2
>>>>>>>>
>>>>>>>> It found errors and fixed them. However, it did not fix the corrupt
>>>>>>>> leaf. The file is actually a directory:
>>>>>>>>
>>>>>>>> ~$ stat /usr/share/doc/libatk1.0-data
>>>>>>>> stat: cannot statx '/usr/share/doc/libatk1.0-data':
>>>>>>>> Invoer-/uitvoerfout
>>>>>>>>
>>>>>>>> in journal:
>>>>>>>>
>>>>>>>> BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>>>> block=1169152675840
>>>>>>>> slot=1 ino=915987, invalid inode generation: has
>>>>>>>> 18446744073709551492
>>>>>>>> expect [0, 5852829]
>>>>>>>> BTRFS error (device sda2): block=1169152675840 read time tree block
>>>>>>>> corruption detected
>>>>>>>>
>>>>>>>> So how do I repair this? Am I doing something wrong?
>>>>>>> Please provide the following dump:
>>>>>>> btrfs ins dump-tree -b 1169152675840 /dev/sda2
>>>>>>>
>>>>>>> Feel free to remove the filenames in the dump.
>>>>>> sudo btrfs ins dump-tree -b 1169152675840 /dev/sda2
>>>>>> btrfs-progs v5.3-rc1
>>>>>> leaf 1169152675840 items 36 free space 966 generation 5431733 owner
>>>>>> 294
>>>>>> leaf 1169152675840 flags 0x1(WRITTEN) backref revision 1
>>>>>> fs uuid 27155120-9ef8-47fb-b248-eaac2b7c8375
>>>>>> chunk uuid 5704f1ba-08fd-4f6b-9117-0e080b4e9ef0
>>>>>> item 0 key (915986 DIR_INDEX 2) itemoff 3957 itemsize 38
>>>>>> location key (915987 INODE_ITEM 0) type FILE
>>>>>> transid 7782235549259005952 data_len 0 name_len 8
>>>>>> name: smb.conf
>>>>>> item 1 key (915987 INODE_ITEM 0) itemoff 3797 itemsize 160
>>>>>> generation 1 transid 18446744073709551492 size 12464
>>>>>> nbytes 16384
>>>>> Yeah, corrupted transid.
>>>>>
>>>>> The v5.6 kernel doesn't get the fix backported...
>>>>>
>>>>> Now you have to use either the out-of-tree branch, or David's devel
>>>>> branch to build a btrfs-progs which is able to repair the transid
>>>>> error.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>> Just be to be clear, I tried to repair with the Kubuntu Groovy Live usb,
>>>> which has linux 5.8 and btrfs-progs 5.7.
>>>>
>>>> I didn't fix the above transid, above was taken after booting normally
>>>> again (linux 5.8), unfortunately with btrfs-progs v5.3-rc1 (that I built
>>>> a year ago). See the other post for the result with btrfs-progs 5.7.
>>>>
>>>>
>>> As I said already, you need either the devel branch to do the fix.
>>> Current release btrfs-progs hasn't the repair ability merged.
>>>
>> Ah, I understood wrong. I thought 5.7 was enough.
>>
>> So, I need to build the latest and greatest and install on live usb. Or
>> I need to wait for a future live usb with this incorporated.
>>
> It's strongly recommended to build right now.
>
> As you may already find, the btrfs-progs release cycle is normally
> behind schedule, thus you may need to wait way longer than you thought...
>
> BTW, after repairing, it's recommended to use newer enought (v5.8 should
> be enough) kernel to mount the repaired fs, to verify the transid error
> get fixed.
>
> Thanks,
> Qu
Thanks for the good advice. I found btrfs-progs 5.9 is in debian, and
quick test shows it doesn't do damage to install the debian package on
ubuntu.
This means that it is relatively easy to take the ubuntu live usb,
download the debian 5.9 package and test it.
But do I understand correctly you recommend to try to repair using the
development branch? That doesn't sound very safe :-)
I will report back here the results on monday.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-07 13:28 ` Qu Wenruo
2020-11-07 19:50 ` Ferry Toth
@ 2020-11-16 10:41 ` Ferry Toth
2020-11-16 10:52 ` Andrei Borzenkov
1 sibling, 1 reply; 56+ messages in thread
From: Ferry Toth @ 2020-11-16 10:41 UTC (permalink / raw)
To: Qu Wenruo, Qu Wenruo, Tyler Richmond; +Cc: Btrfs BTRFS
Hi,
Op 07-11-2020 om 14:28 schreef Qu Wenruo:
> On 2020/11/7 下午9:19, Ferry Toth wrote:
>> Hi
>>
>> Op 07-11-2020 om 12:35 schreef Qu Wenruo:
>>> On 2020/11/7 下午7:18, Ferry Toth wrote:
>>>> Op 06-11-2020 om 11:32 schreef Qu Wenruo:
>>>>> On 2020/11/6 下午6:30, Ferry Toth wrote:
>>>>>> Hi
>>>>>>
>>>>>> Op 06-11-2020 om 11:24 schreef Qu Wenruo:
>>>>>>> On 2020/11/6 下午6:09, Ferry Toth wrote:
>>>>>>>> Hi Qu
>>>>>>>>
>>>>>>>> Op 06-11-2020 om 00:40 schreef Qu Wenruo:
>>>>>>>>> On 2020/11/6 上午7:37, Ferry Toth wrote:
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> Op 06-11-2020 om 00:32 schreef Qu Wenruo:
>>>>>>>>>>> On 2020/11/6 上午7:12, Ferry Toth wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Op 06-11-2020 om 00:00 schreef Qu Wenruo:
>>>>>>>>>>>>> On 2020/11/6 上午4:08, Ferry Toth wrote:
>>>>>>>>>>>>>> I am in a similar spot, during updating my distro (Kubuntu),
>>>>>>>>>>>>>> I am
>>>>>>>>>>>>>> unable
>>>>>>>>>>>>>> to update a certain package. I know which file it is:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ~$ ls -l /usr/share/doc/libatk1.0-data
>>>>>>>>>>>>>> ls: kan geen toegang krijgen tot
>>>>>>>>>>>>>> '/usr/share/doc/libatk1.0-data':
>>>>>>>>>>>>>> Invoer-/uitvoerfout
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This creates the following in journal:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> kernel: BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>>>>>>>>>> block=1169152675840 slot=1 ino=915987, invalid inode
>>>>>>>>>>>>>> generation: has
>>>>>>>>>>>>>> 18446744073709551492 expect [0, 5851353]
>>>>>>>>>>>>>> kernel: BTRFS error (device sda2): block=1169152675840 read
>>>>>>>>>>>>>> time
>>>>>>>>>>>>>> tree
>>>>>>>>>>>>>> block corruption detected
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Now, the problem: this file is on my rootfs, which is
>>>>>>>>>>>>>> mounted. apt
>>>>>>>>>>>>>> (distribution updated) installed all packages but can't
>>>>>>>>>>>>>> continue
>>>>>>>>>>>>>> configuring, because libatk is a dependancy. I can't delete
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> file
>>>>>>>>>>>>>> because of the I/O error. And btrfs check complains (I tried
>>>>>>>>>>>>>> running RO)
>>>>>>>>>>>>>> because the file system is mounted.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But, on the sunny side, the file system is not RO.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is there any way to forcefully remove the file? Or do you
>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>> recommendation how to proceed?
>>>>>>>>>>>>> Newer kernel will reject to even read the item, thus will
>>>>>>>>>>>>> not be
>>>>>>>>>>>>> able to
>>>>>>>>>>>>> remove it.
>>>>>>>>>>>> That's already the case. (input / output error)
>>>>>>>>>>>>> I guess you have to use some distro ISO to fix the fs.
>>>>>>>>>>>> And then? btrfs check --repair the disk offline?
>>>>>>>>>>> Yep.
>>>>>>>>>>>
>>>>>>>>>>> You would want the latest btrfs-progs though.
>>>>>>>>>> Groovy has 5.7. Would that be good enough? Otherwise will be
>>>>>>>>>> difficult
>>>>>>>>>> to build on/for live usb image.
>>>>>>>>> For your particular case, the fix are already in btrfs-progs v5.4.
>>>>>>>>>
>>>>>>>>> Although newer is always better, just in case you have extent item
>>>>>>>>> generation corruption, you may want v5.4.1.
>>>>>>>>>
>>>>>>>>> So your v5.7 should be good enough.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Qu
>>>>>>>> I made a live usb and performed:
>>>>>>>>
>>>>>>>> btrfs check --repair /dev/sda2
>>>>>>>>
>>>>>>>> It found errors and fixed them. However, it did not fix the corrupt
>>>>>>>> leaf. The file is actually a directory:
>>>>>>>>
>>>>>>>> ~$ stat /usr/share/doc/libatk1.0-data
>>>>>>>> stat: cannot statx '/usr/share/doc/libatk1.0-data':
>>>>>>>> Invoer-/uitvoerfout
>>>>>>>>
>>>>>>>> in journal:
>>>>>>>>
>>>>>>>> BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>>>> block=1169152675840
>>>>>>>> slot=1 ino=915987, invalid inode generation: has
>>>>>>>> 18446744073709551492
>>>>>>>> expect [0, 5852829]
>>>>>>>> BTRFS error (device sda2): block=1169152675840 read time tree block
>>>>>>>> corruption detected
>>>>>>>>
>>>>>>>> So how do I repair this? Am I doing something wrong?
>>>>>>> Please provide the following dump:
>>>>>>> btrfs ins dump-tree -b 1169152675840 /dev/sda2
>>>>>>>
>>>>>>> Feel free to remove the filenames in the dump.
>>>>>> sudo btrfs ins dump-tree -b 1169152675840 /dev/sda2
>>>>>> btrfs-progs v5.3-rc1
>>>>>> leaf 1169152675840 items 36 free space 966 generation 5431733 owner
>>>>>> 294
>>>>>> leaf 1169152675840 flags 0x1(WRITTEN) backref revision 1
>>>>>> fs uuid 27155120-9ef8-47fb-b248-eaac2b7c8375
>>>>>> chunk uuid 5704f1ba-08fd-4f6b-9117-0e080b4e9ef0
>>>>>> item 0 key (915986 DIR_INDEX 2) itemoff 3957 itemsize 38
>>>>>> location key (915987 INODE_ITEM 0) type FILE
>>>>>> transid 7782235549259005952 data_len 0 name_len 8
>>>>>> name: smb.conf
>>>>>> item 1 key (915987 INODE_ITEM 0) itemoff 3797 itemsize 160
>>>>>> generation 1 transid 18446744073709551492 size 12464
>>>>>> nbytes 16384
>>>>> Yeah, corrupted transid.
>>>>>
>>>>> The v5.6 kernel doesn't get the fix backported...
>>>>>
>>>>> Now you have to use either the out-of-tree branch, or David's devel
>>>>> branch to build a btrfs-progs which is able to repair the transid
>>>>> error.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>> Just be to be clear, I tried to repair with the Kubuntu Groovy Live usb,
>>>> which has linux 5.8 and btrfs-progs 5.7.
>>>>
>>>> I didn't fix the above transid, above was taken after booting normally
>>>> again (linux 5.8), unfortunately with btrfs-progs v5.3-rc1 (that I built
>>>> a year ago). See the other post for the result with btrfs-progs 5.7.
>>>>
>>>>
>>> As I said already, you need either the devel branch to do the fix.
>>> Current release btrfs-progs hasn't the repair ability merged.
>>>
>> Ah, I understood wrong. I thought 5.7 was enough.
>>
>> So, I need to build the latest and greatest and install on live usb. Or
>> I need to wait for a future live usb with this incorporated.
>>
> It's strongly recommended to build right now.
>
> As you may already find, the btrfs-progs release cycle is normally
> behind schedule, thus you may need to wait way longer than you thought...
>
> BTW, after repairing, it's recommended to use newer enought (v5.8 should
> be enough) kernel to mount the repaired fs, to verify the transid error
> get fixed.
>
> Thanks,
> Qu
>
Just in case others stumble upon the same situation here is a brief
summary of the problem and what I did to fix it:
- On upgrading from Kubuntu 20.04 to 20.10 certain package could not be
upgraded because they failed to stat. Eventually the upgrade aborts with
some packages not configured and some dependencies not installed.
- After rebooting this is still the case. Fortunately the failing files
were under /usr/share/doc and the dependencies not essential to boot. It
was possible to rename /usr/share/doc to /usr/share/dog then cp that to
/usr/share/doc. That was sufficient to continue configuration and finish
the upgrade.
- rm -rf /usr/share/dog of course still fails on the broken directories
- Kubuntu 20.10 kernel is linux 5.8 but btrfs-prog is v5.7 and did not
fix the problem
- Built btrfs-progs v5.9 from
git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git and
booting Kubuntu 20.10 from USB stick then running btrfs-progs v5.9 fixes
it with below log.
- Thanks Qu for the help!
Ferry
log
---
kubuntu@kubuntu:~$ sudo ./btrfs check /dev/sda2
Opening filesystem to check...
Checking filesystem on /dev/sda2
UUID: 27155120-9ef8-47fb-b248-eaac2b7c8375
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
root 294 inode 915975 errors 100000, invalid inode generation or transid
root 294 inode 915976 errors 100000, invalid inode generation or transid
root 294 inode 915977 errors 100000, invalid inode generation or transid
root 294 inode 915978 errors 100000, invalid inode generation or transid
root 294 inode 915979 errors 100000, invalid inode generation or transid
root 294 inode 915980 errors 100000, invalid inode generation or transid
root 294 inode 915982 errors 100000, invalid inode generation or transid
root 294 inode 915984 errors 100000, invalid inode generation or transid
root 294 inode 915985 errors 100000, invalid inode generation or transid
root 294 inode 915987 errors 100000, invalid inode generation or transid
root 294 inode 1962496 errors 100000, invalid inode generation or transid
root 294 inode 1962499 errors 100000, invalid inode generation or transid
root 294 inode 1962500 errors 100000, invalid inode generation or transid
root 294 inode 1962501 errors 100000, invalid inode generation or transid
root 294 inode 1962502 errors 100000, invalid inode generation or transid
root 294 inode 1962503 errors 100000, invalid inode generation or transid
root 294 inode 1962504 errors 100000, invalid inode generation or transid
root 294 inode 1962505 errors 100000, invalid inode generation or transid
root 294 inode 1962506 errors 100000, invalid inode generation or transid
root 294 inode 1962507 errors 100000, invalid inode generation or transid
root 294 inode 1962508 errors 100000, invalid inode generation or transid
root 294 inode 1962509 errors 100000, invalid inode generation or transid
root 294 inode 1962510 errors 100000, invalid inode generation or transid
root 294 inode 1962511 errors 100000, invalid inode generation or transid
root 294 inode 1962512 errors 100000, invalid inode generation or transid
root 294 inode 1962513 errors 100000, invalid inode generation or transid
root 294 inode 1962514 errors 100000, invalid inode generation or transid
root 294 inode 1962515 errors 100000, invalid inode generation or transid
root 294 inode 1962516 errors 100000, invalid inode generation or transid
root 294 inode 1962517 errors 100000, invalid inode generation or transid
root 294 inode 1962518 errors 100000, invalid inode generation or transid
root 294 inode 1962519 errors 100000, invalid inode generation or transid
root 294 inode 1962520 errors 100000, invalid inode generation or transid
root 294 inode 1962521 errors 100000, invalid inode generation or transid
root 294 inode 1962522 errors 100000, invalid inode generation or transid
root 294 inode 1962523 errors 100000, invalid inode generation or transid
root 294 inode 1962524 errors 100000, invalid inode generation or transid
root 294 inode 1962525 errors 100000, invalid inode generation or transid
root 294 inode 1962526 errors 100000, invalid inode generation or transid
root 294 inode 1962527 errors 100000, invalid inode generation or transid
root 294 inode 1962528 errors 100000, invalid inode generation or transid
root 294 inode 1962529 errors 100000, invalid inode generation or transid
root 294 inode 1962530 errors 100000, invalid inode generation or transid
root 294 inode 1962531 errors 100000, invalid inode generation or transid
root 294 inode 1962532 errors 100000, invalid inode generation or transid
root 294 inode 1962533 errors 100000, invalid inode generation or transid
root 294 inode 1962534 errors 100000, invalid inode generation or transid
root 294 inode 1962535 errors 100000, invalid inode generation or transid
root 294 inode 1962536 errors 100000, invalid inode generation or transid
root 294 inode 1962537 errors 100000, invalid inode generation or transid
root 294 inode 1962538 errors 100000, invalid inode generation or transid
root 294 inode 1962539 errors 100000, invalid inode generation or transid
root 294 inode 1962540 errors 100000, invalid inode generation or transid
root 294 inode 1962541 errors 100000, invalid inode generation or transid
root 294 inode 1962542 errors 100000, invalid inode generation or transid
root 294 inode 1962543 errors 100000, invalid inode generation or transid
root 294 inode 1962544 errors 100000, invalid inode generation or transid
root 294 inode 1962545 errors 100000, invalid inode generation or transid
root 294 inode 1962546 errors 100000, invalid inode generation or transid
root 294 inode 1962547 errors 100000, invalid inode generation or transid
root 294 inode 1962548 errors 100000, invalid inode generation or transid
root 294 inode 1962549 errors 100000, invalid inode generation or transid
root 294 inode 1962550 errors 100000, invalid inode generation or transid
root 294 inode 1962551 errors 100000, invalid inode generation or transid
root 294 inode 1962552 errors 100000, invalid inode generation or transid
root 294 inode 1962553 errors 100000, invalid inode generation or transid
root 294 inode 1962554 errors 100000, invalid inode generation or transid
root 294 inode 1962555 errors 100000, invalid inode generation or transid
root 294 inode 1962556 errors 100000, invalid inode generation or transid
root 294 inode 1962557 errors 100000, invalid inode generation or transid
root 294 inode 1962558 errors 100000, invalid inode generation or transid
root 294 inode 1962559 errors 100000, invalid inode generation or transid
root 294 inode 1962560 errors 100000, invalid inode generation or transid
root 294 inode 1962561 errors 100000, invalid inode generation or transid
root 294 inode 1962562 errors 100000, invalid inode generation or transid
root 294 inode 1962563 errors 100000, invalid inode generation or transid
root 294 inode 1962564 errors 100000, invalid inode generation or transid
root 294 inode 1962565 errors 100000, invalid inode generation or transid
root 294 inode 1962566 errors 100000, invalid inode generation or transid
root 294 inode 1962567 errors 100000, invalid inode generation or transid
root 294 inode 1962568 errors 100000, invalid inode generation or transid
root 294 inode 1962569 errors 100000, invalid inode generation or transid
root 294 inode 1962570 errors 100000, invalid inode generation or transid
root 294 inode 1962571 errors 100000, invalid inode generation or transid
root 294 inode 1962572 errors 100000, invalid inode generation or transid
root 294 inode 1962573 errors 100000, invalid inode generation or transid
root 294 inode 1962574 errors 100000, invalid inode generation or transid
root 294 inode 1962575 errors 100000, invalid inode generation or transid
root 294 inode 1962576 errors 100000, invalid inode generation or transid
root 294 inode 1962577 errors 100000, invalid inode generation or transid
root 294 inode 1962578 errors 100000, invalid inode generation or transid
root 294 inode 1962579 errors 100000, invalid inode generation or transid
root 294 inode 1962580 errors 100000, invalid inode generation or transid
root 294 inode 1962581 errors 100000, invalid inode generation or transid
root 294 inode 1962582 errors 100000, invalid inode generation or transid
root 294 inode 1962583 errors 100000, invalid inode generation or transid
root 294 inode 1962584 errors 100000, invalid inode generation or transid
root 294 inode 1962585 errors 100000, invalid inode generation or transid
root 294 inode 1962586 errors 100000, invalid inode generation or transid
ERROR: errors found in fs roots
found 444725678080 bytes used, error(s) found
total csum bytes: 272354940
total tree bytes: 1862316032
total fs tree bytes: 1250484224
total extent tree bytes: 277397504
btree space waste bytes: 401028956
file data blocks allocated: 16021691621376
referenced 394733535232
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-16 10:41 ` Ferry Toth
@ 2020-11-16 10:52 ` Andrei Borzenkov
2020-11-16 10:57 ` Ferry Toth
0 siblings, 1 reply; 56+ messages in thread
From: Andrei Borzenkov @ 2020-11-16 10:52 UTC (permalink / raw)
To: Ferry Toth; +Cc: Qu Wenruo, Qu Wenruo, Tyler Richmond, Btrfs BTRFS
On Mon, Nov 16, 2020 at 1:44 PM Ferry Toth <fntoth@gmail.com> wrote:
>
> - Built btrfs-progs v5.9 from
> git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git and
> booting Kubuntu 20.10 from USB stick then running btrfs-progs v5.9 fixes
> it with below log.
>
What do you mean "fixes"? "btrfs check" is read-only by default so
your invocation could not fix anything. Was it the only command you
invoked?
> - Thanks Qu for the help!
>
> Ferry
>
> log
>
> ---
>
> kubuntu@kubuntu:~$ sudo ./btrfs check /dev/sda2
> Opening filesystem to check...
> Checking filesystem on /dev/sda2
> UUID: 27155120-9ef8-47fb-b248-eaac2b7c8375
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> [4/7] checking fs roots
> root 294 inode 915975 errors 100000, invalid inode generation or transid
> root 294 inode 915976 errors 100000, invalid inode generation or transid
> root 294 inode 915977 errors 100000, invalid inode generation or transid
> root 294 inode 915978 errors 100000, invalid inode generation or transid
> root 294 inode 915979 errors 100000, invalid inode generation or transid
> root 294 inode 915980 errors 100000, invalid inode generation or transid
> root 294 inode 915982 errors 100000, invalid inode generation or transid
> root 294 inode 915984 errors 100000, invalid inode generation or transid
> root 294 inode 915985 errors 100000, invalid inode generation or transid
> root 294 inode 915987 errors 100000, invalid inode generation or transid
> root 294 inode 1962496 errors 100000, invalid inode generation or transid
> root 294 inode 1962499 errors 100000, invalid inode generation or transid
> root 294 inode 1962500 errors 100000, invalid inode generation or transid
> root 294 inode 1962501 errors 100000, invalid inode generation or transid
> root 294 inode 1962502 errors 100000, invalid inode generation or transid
> root 294 inode 1962503 errors 100000, invalid inode generation or transid
> root 294 inode 1962504 errors 100000, invalid inode generation or transid
> root 294 inode 1962505 errors 100000, invalid inode generation or transid
> root 294 inode 1962506 errors 100000, invalid inode generation or transid
> root 294 inode 1962507 errors 100000, invalid inode generation or transid
> root 294 inode 1962508 errors 100000, invalid inode generation or transid
> root 294 inode 1962509 errors 100000, invalid inode generation or transid
> root 294 inode 1962510 errors 100000, invalid inode generation or transid
> root 294 inode 1962511 errors 100000, invalid inode generation or transid
> root 294 inode 1962512 errors 100000, invalid inode generation or transid
> root 294 inode 1962513 errors 100000, invalid inode generation or transid
> root 294 inode 1962514 errors 100000, invalid inode generation or transid
> root 294 inode 1962515 errors 100000, invalid inode generation or transid
> root 294 inode 1962516 errors 100000, invalid inode generation or transid
> root 294 inode 1962517 errors 100000, invalid inode generation or transid
> root 294 inode 1962518 errors 100000, invalid inode generation or transid
> root 294 inode 1962519 errors 100000, invalid inode generation or transid
> root 294 inode 1962520 errors 100000, invalid inode generation or transid
> root 294 inode 1962521 errors 100000, invalid inode generation or transid
> root 294 inode 1962522 errors 100000, invalid inode generation or transid
> root 294 inode 1962523 errors 100000, invalid inode generation or transid
> root 294 inode 1962524 errors 100000, invalid inode generation or transid
> root 294 inode 1962525 errors 100000, invalid inode generation or transid
> root 294 inode 1962526 errors 100000, invalid inode generation or transid
> root 294 inode 1962527 errors 100000, invalid inode generation or transid
> root 294 inode 1962528 errors 100000, invalid inode generation or transid
> root 294 inode 1962529 errors 100000, invalid inode generation or transid
> root 294 inode 1962530 errors 100000, invalid inode generation or transid
> root 294 inode 1962531 errors 100000, invalid inode generation or transid
> root 294 inode 1962532 errors 100000, invalid inode generation or transid
> root 294 inode 1962533 errors 100000, invalid inode generation or transid
> root 294 inode 1962534 errors 100000, invalid inode generation or transid
> root 294 inode 1962535 errors 100000, invalid inode generation or transid
> root 294 inode 1962536 errors 100000, invalid inode generation or transid
> root 294 inode 1962537 errors 100000, invalid inode generation or transid
> root 294 inode 1962538 errors 100000, invalid inode generation or transid
> root 294 inode 1962539 errors 100000, invalid inode generation or transid
> root 294 inode 1962540 errors 100000, invalid inode generation or transid
> root 294 inode 1962541 errors 100000, invalid inode generation or transid
> root 294 inode 1962542 errors 100000, invalid inode generation or transid
> root 294 inode 1962543 errors 100000, invalid inode generation or transid
> root 294 inode 1962544 errors 100000, invalid inode generation or transid
> root 294 inode 1962545 errors 100000, invalid inode generation or transid
> root 294 inode 1962546 errors 100000, invalid inode generation or transid
> root 294 inode 1962547 errors 100000, invalid inode generation or transid
> root 294 inode 1962548 errors 100000, invalid inode generation or transid
> root 294 inode 1962549 errors 100000, invalid inode generation or transid
> root 294 inode 1962550 errors 100000, invalid inode generation or transid
> root 294 inode 1962551 errors 100000, invalid inode generation or transid
> root 294 inode 1962552 errors 100000, invalid inode generation or transid
> root 294 inode 1962553 errors 100000, invalid inode generation or transid
> root 294 inode 1962554 errors 100000, invalid inode generation or transid
> root 294 inode 1962555 errors 100000, invalid inode generation or transid
> root 294 inode 1962556 errors 100000, invalid inode generation or transid
> root 294 inode 1962557 errors 100000, invalid inode generation or transid
> root 294 inode 1962558 errors 100000, invalid inode generation or transid
> root 294 inode 1962559 errors 100000, invalid inode generation or transid
> root 294 inode 1962560 errors 100000, invalid inode generation or transid
> root 294 inode 1962561 errors 100000, invalid inode generation or transid
> root 294 inode 1962562 errors 100000, invalid inode generation or transid
> root 294 inode 1962563 errors 100000, invalid inode generation or transid
> root 294 inode 1962564 errors 100000, invalid inode generation or transid
> root 294 inode 1962565 errors 100000, invalid inode generation or transid
> root 294 inode 1962566 errors 100000, invalid inode generation or transid
> root 294 inode 1962567 errors 100000, invalid inode generation or transid
> root 294 inode 1962568 errors 100000, invalid inode generation or transid
> root 294 inode 1962569 errors 100000, invalid inode generation or transid
> root 294 inode 1962570 errors 100000, invalid inode generation or transid
> root 294 inode 1962571 errors 100000, invalid inode generation or transid
> root 294 inode 1962572 errors 100000, invalid inode generation or transid
> root 294 inode 1962573 errors 100000, invalid inode generation or transid
> root 294 inode 1962574 errors 100000, invalid inode generation or transid
> root 294 inode 1962575 errors 100000, invalid inode generation or transid
> root 294 inode 1962576 errors 100000, invalid inode generation or transid
> root 294 inode 1962577 errors 100000, invalid inode generation or transid
> root 294 inode 1962578 errors 100000, invalid inode generation or transid
> root 294 inode 1962579 errors 100000, invalid inode generation or transid
> root 294 inode 1962580 errors 100000, invalid inode generation or transid
> root 294 inode 1962581 errors 100000, invalid inode generation or transid
> root 294 inode 1962582 errors 100000, invalid inode generation or transid
> root 294 inode 1962583 errors 100000, invalid inode generation or transid
> root 294 inode 1962584 errors 100000, invalid inode generation or transid
> root 294 inode 1962585 errors 100000, invalid inode generation or transid
> root 294 inode 1962586 errors 100000, invalid inode generation or transid
> ERROR: errors found in fs roots
> found 444725678080 bytes used, error(s) found
> total csum bytes: 272354940
> total tree bytes: 1862316032
> total fs tree bytes: 1250484224
> total extent tree bytes: 277397504
> btree space waste bytes: 401028956
> file data blocks allocated: 16021691621376
> referenced 394733535232
>
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-16 10:52 ` Andrei Borzenkov
@ 2020-11-16 10:57 ` Ferry Toth
2020-11-16 16:35 ` Tyler Richmond
0 siblings, 1 reply; 56+ messages in thread
From: Ferry Toth @ 2020-11-16 10:57 UTC (permalink / raw)
To: Andrei Borzenkov; +Cc: Qu Wenruo, Qu Wenruo, Tyler Richmond, Btrfs BTRFS
Hi,
Op 16-11-2020 om 11:52 schreef Andrei Borzenkov:
> On Mon, Nov 16, 2020 at 1:44 PM Ferry Toth <fntoth@gmail.com> wrote:
>> - Built btrfs-progs v5.9 from
>> git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git and
>> booting Kubuntu 20.10 from USB stick then running btrfs-progs v5.9 fixes
>> it with below log.
>>
> What do you mean "fixes"? "btrfs check" is read-only by default so
> your invocation could not fix anything. Was it the only command you
> invoked?
Yeah, immediately after that command I ran
sudo ./btrfs check --repair /dev/sda2
to really repair, and then I could clean up:
rm -rf /usr/share/dog
>> - Thanks Qu for the help!
>>
>> Ferry
>>
>> log
>>
>> ---
>>
>> kubuntu@kubuntu:~$ sudo ./btrfs check /dev/sda2
>> Opening filesystem to check...
>> Checking filesystem on /dev/sda2
>> UUID: 27155120-9ef8-47fb-b248-eaac2b7c8375
>> [1/7] checking root items
>> [2/7] checking extents
>> [3/7] checking free space cache
>> [4/7] checking fs roots
>> root 294 inode 915975 errors 100000, invalid inode generation or transid
>> root 294 inode 915976 errors 100000, invalid inode generation or transid
>> root 294 inode 915977 errors 100000, invalid inode generation or transid
>> root 294 inode 915978 errors 100000, invalid inode generation or transid
>> root 294 inode 915979 errors 100000, invalid inode generation or transid
>> root 294 inode 915980 errors 100000, invalid inode generation or transid
>> root 294 inode 915982 errors 100000, invalid inode generation or transid
>> root 294 inode 915984 errors 100000, invalid inode generation or transid
>> root 294 inode 915985 errors 100000, invalid inode generation or transid
>> root 294 inode 915987 errors 100000, invalid inode generation or transid
>> root 294 inode 1962496 errors 100000, invalid inode generation or transid
>> root 294 inode 1962499 errors 100000, invalid inode generation or transid
>> root 294 inode 1962500 errors 100000, invalid inode generation or transid
>> root 294 inode 1962501 errors 100000, invalid inode generation or transid
>> root 294 inode 1962502 errors 100000, invalid inode generation or transid
>> root 294 inode 1962503 errors 100000, invalid inode generation or transid
>> root 294 inode 1962504 errors 100000, invalid inode generation or transid
>> root 294 inode 1962505 errors 100000, invalid inode generation or transid
>> root 294 inode 1962506 errors 100000, invalid inode generation or transid
>> root 294 inode 1962507 errors 100000, invalid inode generation or transid
>> root 294 inode 1962508 errors 100000, invalid inode generation or transid
>> root 294 inode 1962509 errors 100000, invalid inode generation or transid
>> root 294 inode 1962510 errors 100000, invalid inode generation or transid
>> root 294 inode 1962511 errors 100000, invalid inode generation or transid
>> root 294 inode 1962512 errors 100000, invalid inode generation or transid
>> root 294 inode 1962513 errors 100000, invalid inode generation or transid
>> root 294 inode 1962514 errors 100000, invalid inode generation or transid
>> root 294 inode 1962515 errors 100000, invalid inode generation or transid
>> root 294 inode 1962516 errors 100000, invalid inode generation or transid
>> root 294 inode 1962517 errors 100000, invalid inode generation or transid
>> root 294 inode 1962518 errors 100000, invalid inode generation or transid
>> root 294 inode 1962519 errors 100000, invalid inode generation or transid
>> root 294 inode 1962520 errors 100000, invalid inode generation or transid
>> root 294 inode 1962521 errors 100000, invalid inode generation or transid
>> root 294 inode 1962522 errors 100000, invalid inode generation or transid
>> root 294 inode 1962523 errors 100000, invalid inode generation or transid
>> root 294 inode 1962524 errors 100000, invalid inode generation or transid
>> root 294 inode 1962525 errors 100000, invalid inode generation or transid
>> root 294 inode 1962526 errors 100000, invalid inode generation or transid
>> root 294 inode 1962527 errors 100000, invalid inode generation or transid
>> root 294 inode 1962528 errors 100000, invalid inode generation or transid
>> root 294 inode 1962529 errors 100000, invalid inode generation or transid
>> root 294 inode 1962530 errors 100000, invalid inode generation or transid
>> root 294 inode 1962531 errors 100000, invalid inode generation or transid
>> root 294 inode 1962532 errors 100000, invalid inode generation or transid
>> root 294 inode 1962533 errors 100000, invalid inode generation or transid
>> root 294 inode 1962534 errors 100000, invalid inode generation or transid
>> root 294 inode 1962535 errors 100000, invalid inode generation or transid
>> root 294 inode 1962536 errors 100000, invalid inode generation or transid
>> root 294 inode 1962537 errors 100000, invalid inode generation or transid
>> root 294 inode 1962538 errors 100000, invalid inode generation or transid
>> root 294 inode 1962539 errors 100000, invalid inode generation or transid
>> root 294 inode 1962540 errors 100000, invalid inode generation or transid
>> root 294 inode 1962541 errors 100000, invalid inode generation or transid
>> root 294 inode 1962542 errors 100000, invalid inode generation or transid
>> root 294 inode 1962543 errors 100000, invalid inode generation or transid
>> root 294 inode 1962544 errors 100000, invalid inode generation or transid
>> root 294 inode 1962545 errors 100000, invalid inode generation or transid
>> root 294 inode 1962546 errors 100000, invalid inode generation or transid
>> root 294 inode 1962547 errors 100000, invalid inode generation or transid
>> root 294 inode 1962548 errors 100000, invalid inode generation or transid
>> root 294 inode 1962549 errors 100000, invalid inode generation or transid
>> root 294 inode 1962550 errors 100000, invalid inode generation or transid
>> root 294 inode 1962551 errors 100000, invalid inode generation or transid
>> root 294 inode 1962552 errors 100000, invalid inode generation or transid
>> root 294 inode 1962553 errors 100000, invalid inode generation or transid
>> root 294 inode 1962554 errors 100000, invalid inode generation or transid
>> root 294 inode 1962555 errors 100000, invalid inode generation or transid
>> root 294 inode 1962556 errors 100000, invalid inode generation or transid
>> root 294 inode 1962557 errors 100000, invalid inode generation or transid
>> root 294 inode 1962558 errors 100000, invalid inode generation or transid
>> root 294 inode 1962559 errors 100000, invalid inode generation or transid
>> root 294 inode 1962560 errors 100000, invalid inode generation or transid
>> root 294 inode 1962561 errors 100000, invalid inode generation or transid
>> root 294 inode 1962562 errors 100000, invalid inode generation or transid
>> root 294 inode 1962563 errors 100000, invalid inode generation or transid
>> root 294 inode 1962564 errors 100000, invalid inode generation or transid
>> root 294 inode 1962565 errors 100000, invalid inode generation or transid
>> root 294 inode 1962566 errors 100000, invalid inode generation or transid
>> root 294 inode 1962567 errors 100000, invalid inode generation or transid
>> root 294 inode 1962568 errors 100000, invalid inode generation or transid
>> root 294 inode 1962569 errors 100000, invalid inode generation or transid
>> root 294 inode 1962570 errors 100000, invalid inode generation or transid
>> root 294 inode 1962571 errors 100000, invalid inode generation or transid
>> root 294 inode 1962572 errors 100000, invalid inode generation or transid
>> root 294 inode 1962573 errors 100000, invalid inode generation or transid
>> root 294 inode 1962574 errors 100000, invalid inode generation or transid
>> root 294 inode 1962575 errors 100000, invalid inode generation or transid
>> root 294 inode 1962576 errors 100000, invalid inode generation or transid
>> root 294 inode 1962577 errors 100000, invalid inode generation or transid
>> root 294 inode 1962578 errors 100000, invalid inode generation or transid
>> root 294 inode 1962579 errors 100000, invalid inode generation or transid
>> root 294 inode 1962580 errors 100000, invalid inode generation or transid
>> root 294 inode 1962581 errors 100000, invalid inode generation or transid
>> root 294 inode 1962582 errors 100000, invalid inode generation or transid
>> root 294 inode 1962583 errors 100000, invalid inode generation or transid
>> root 294 inode 1962584 errors 100000, invalid inode generation or transid
>> root 294 inode 1962585 errors 100000, invalid inode generation or transid
>> root 294 inode 1962586 errors 100000, invalid inode generation or transid
>> ERROR: errors found in fs roots
>> found 444725678080 bytes used, error(s) found
>> total csum bytes: 272354940
>> total tree bytes: 1862316032
>> total fs tree bytes: 1250484224
>> total extent tree bytes: 277397504
>> btree space waste bytes: 401028956
>> file data blocks allocated: 16021691621376
>> referenced 394733535232
>>
>>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: Fwd: Read time tree block corruption detected
2020-11-16 10:57 ` Ferry Toth
@ 2020-11-16 16:35 ` Tyler Richmond
0 siblings, 0 replies; 56+ messages in thread
From: Tyler Richmond @ 2020-11-16 16:35 UTC (permalink / raw)
To: Ferry Toth, Btrfs BTRFS; +Cc: Qu Wenruo, Qu Wenruo
Qu,
I was finally able to complete the repair on my system. It took over a
week, but I think I may still have issues. The end of the repair is
below. Looks like there are still csum errors but no extent record?
https://pastebin.com/mMNAyUUa
After that, I tried to continue the rebalance of my array, but it
failed with a -22 status:
https://pastebin.com/B35UmVV3
Is this failure still related to more errors in my filesystem and
worth pursuing in this thread? If not, I can create a new one.
Thank you for all of your help!
-Tyler
On Mon, Nov 16, 2020 at 5:57 AM Ferry Toth <fntoth@gmail.com> wrote:
>
> Hi,
>
> Op 16-11-2020 om 11:52 schreef Andrei Borzenkov:
> > On Mon, Nov 16, 2020 at 1:44 PM Ferry Toth <fntoth@gmail.com> wrote:
> >> - Built btrfs-progs v5.9 from
> >> git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git and
> >> booting Kubuntu 20.10 from USB stick then running btrfs-progs v5.9 fixes
> >> it with below log.
> >>
> > What do you mean "fixes"? "btrfs check" is read-only by default so
> > your invocation could not fix anything. Was it the only command you
> > invoked?
> Yeah, immediately after that command I ran
>
> sudo ./btrfs check --repair /dev/sda2
>
> to really repair, and then I could clean up:
>
> rm -rf /usr/share/dog
>
> >> - Thanks Qu for the help!
> >>
> >> Ferry
> >>
> >> log
> >>
> >> ---
> >>
> >> kubuntu@kubuntu:~$ sudo ./btrfs check /dev/sda2
> >> Opening filesystem to check...
> >> Checking filesystem on /dev/sda2
> >> UUID: 27155120-9ef8-47fb-b248-eaac2b7c8375
> >> [1/7] checking root items
> >> [2/7] checking extents
> >> [3/7] checking free space cache
> >> [4/7] checking fs roots
> >> root 294 inode 915975 errors 100000, invalid inode generation or transid
> >> root 294 inode 915976 errors 100000, invalid inode generation or transid
> >> root 294 inode 915977 errors 100000, invalid inode generation or transid
> >> root 294 inode 915978 errors 100000, invalid inode generation or transid
> >> root 294 inode 915979 errors 100000, invalid inode generation or transid
> >> root 294 inode 915980 errors 100000, invalid inode generation or transid
> >> root 294 inode 915982 errors 100000, invalid inode generation or transid
> >> root 294 inode 915984 errors 100000, invalid inode generation or transid
> >> root 294 inode 915985 errors 100000, invalid inode generation or transid
> >> root 294 inode 915987 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962496 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962499 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962500 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962501 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962502 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962503 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962504 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962505 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962506 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962507 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962508 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962509 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962510 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962511 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962512 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962513 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962514 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962515 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962516 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962517 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962518 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962519 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962520 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962521 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962522 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962523 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962524 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962525 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962526 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962527 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962528 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962529 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962530 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962531 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962532 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962533 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962534 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962535 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962536 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962537 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962538 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962539 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962540 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962541 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962542 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962543 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962544 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962545 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962546 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962547 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962548 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962549 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962550 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962551 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962552 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962553 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962554 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962555 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962556 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962557 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962558 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962559 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962560 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962561 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962562 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962563 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962564 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962565 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962566 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962567 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962568 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962569 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962570 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962571 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962572 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962573 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962574 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962575 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962576 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962577 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962578 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962579 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962580 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962581 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962582 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962583 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962584 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962585 errors 100000, invalid inode generation or transid
> >> root 294 inode 1962586 errors 100000, invalid inode generation or transid
> >> ERROR: errors found in fs roots
> >> found 444725678080 bytes used, error(s) found
> >> total csum bytes: 272354940
> >> total tree bytes: 1862316032
> >> total fs tree bytes: 1250484224
> >> total extent tree bytes: 277397504
> >> btree space waste bytes: 401028956
> >> file data blocks allocated: 16021691621376
> >> referenced 394733535232
> >>
> >>
^ permalink raw reply [flat|nested] 56+ messages in thread
end of thread, other threads:[~2020-11-16 16:35 UTC | newest]
Thread overview: 56+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <CAJheHN0FUe-ijMco1ZOc6iKF2zbPocOw+iiVNeTT1r-JuXOJww@mail.gmail.com>
2020-05-06 21:54 ` Fwd: Read time tree block corruption detected Tyler Richmond
2020-05-06 23:55 ` Chris Murphy
2020-05-07 0:51 ` Tyler Richmond
2020-05-07 1:06 ` Chris Murphy
2020-05-07 1:13 ` Fwd: " Qu Wenruo
2020-05-07 1:30 ` Tyler Richmond
2020-05-07 5:43 ` Tyler Richmond
2020-05-07 5:52 ` Qu Wenruo
2020-05-07 15:52 ` Tyler Richmond
2020-05-08 0:11 ` Qu Wenruo
2020-05-08 4:23 ` Tyler Richmond
2020-05-08 5:07 ` Qu Wenruo
2020-05-08 5:12 ` Tyler Richmond
2020-05-08 5:47 ` Qu Wenruo
2020-05-08 13:52 ` Tyler Richmond
2020-08-18 3:36 ` Tyler Richmond
[not found] ` <CAJheHN3qwDAGY=z14zfO4LBrxNJZZ_rvAMsWLwe-k+4+t3zLog@mail.gmail.com>
2020-08-18 6:07 ` Qu Wenruo
2020-08-18 12:18 ` Tyler Richmond
2020-08-23 1:15 ` Tyler Richmond
2020-08-23 1:51 ` Qu Wenruo
2020-08-23 2:31 ` Qu Wenruo
2020-08-23 2:49 ` Tyler Richmond
2020-08-23 4:28 ` Qu Wenruo
2020-08-24 2:47 ` Tyler Richmond
2020-08-24 8:26 ` Qu Wenruo
2020-08-25 5:25 ` Tyler Richmond
2020-08-25 6:37 ` Qu Wenruo
2020-08-25 13:30 ` Tyler Richmond
2020-08-25 13:38 ` Qu Wenruo
2020-08-25 13:43 ` Tyler Richmond
2020-11-05 7:01 ` Tyler Richmond
2020-11-05 7:19 ` Qu Wenruo
2020-11-05 20:08 ` Ferry Toth
2020-11-05 23:00 ` Qu Wenruo
2020-11-05 23:12 ` Ferry Toth
2020-11-05 23:32 ` Qu Wenruo
2020-11-05 23:37 ` Ferry Toth
2020-11-05 23:40 ` Qu Wenruo
2020-11-06 10:09 ` Ferry Toth
2020-11-06 10:24 ` Qu Wenruo
2020-11-06 10:27 ` Qu Wenruo
2020-11-06 10:32 ` Ferry Toth
2020-11-06 10:30 ` Ferry Toth
2020-11-06 10:32 ` Qu Wenruo
2020-11-07 11:18 ` Ferry Toth
2020-11-07 11:35 ` Qu Wenruo
2020-11-07 13:19 ` Ferry Toth
2020-11-07 13:28 ` Qu Wenruo
2020-11-07 19:50 ` Ferry Toth
2020-11-07 19:50 ` Ferry Toth
2020-11-16 10:41 ` Ferry Toth
2020-11-16 10:52 ` Andrei Borzenkov
2020-11-16 10:57 ` Ferry Toth
2020-11-16 16:35 ` Tyler Richmond
2020-11-06 11:28 ` Ferry Toth
2020-08-23 2:32 ` Tyler Richmond
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.