linux-btrfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* read time tree block corruption detected
@ 2019-12-29 20:43 Patrick Erley
  2019-12-29 22:07 ` Chris Murphy
  2019-12-30  0:46 ` Qu Wenruo
  0 siblings, 2 replies; 75+ messages in thread
From: Patrick Erley @ 2019-12-29 20:43 UTC (permalink / raw)
  To: linux-btrfs

On a system where I've been tinkering with linux-next for years, my /
has got some errors.  When migrating from 5.1 to 5.2, I saw these
errors, but just ignored them and went back to 5.1, and continued my
tinkering.  Over the holidays, I decided to try to upgrade the kernel,
saw the errors again, and decided to look into them, finding the
tree-checker page on the kernel docs, and am writing this e-mail.

My / does not contain anything sensitive or important, as /home and
/usr/src are both subvolumes on a separate larger drive.

btrfs fi show:
Label: none  uuid: 815266d6-a8b9-4f63-a593-02fde178263f
Total devices 2 FS bytes used 93.81GiB
devid    1 size 115.12GiB used 115.11GiB path /dev/nvme0n1p2
devid    3 size 115.12GiB used 115.11GiB path /dev/sda3

Label: none  uuid: 4bd97711-b63c-40cb-bfa5-aa9c2868cf98
Total devices 1 FS bytes used 536.48GiB
devid    1 size 834.63GiB used 833.59GiB path /dev/sda5

Booting a more recent kernel, I get spammed with:
[    8.243899] BTRFS critical (device nvme0n1p2): corrupt leaf: root=5
block=303629811712 slot=30 ino=5380870, invalid inode generation: has
13221446351398931016 expect [0, 2997884]
[    8.243902] BTRFS error (device nvme0n1p2): block=303629811712 read
time tree block corruption detected

There are 6 corrupted inodes:
cat dmesg.foo  | grep "BTRFS critical" | sed -re
's:.*block=([0-9]*).*ino=([0-9]+).*:\1 \2:' | sort | uniq
303629811712 5380870
303712501760 3277548
303861395456 5909140
304079065088 2228479
304573444096 3539224
305039556608 1442149

and they all have the same value for the inode generation.  Before I
reboot into a livecd and btrfs check --repair, is there anything
interesting that a snapshot of the state would show?  I have 800gb
unpartitioned on the nvme, so backing up before is already in the
plans, and I could just as easily grab an image of the partitions
while I'm at it.

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-29 20:43 read time tree block corruption detected Patrick Erley
@ 2019-12-29 22:07 ` Chris Murphy
  2019-12-29 22:27   ` Patrick Erley
  2019-12-30  0:46 ` Qu Wenruo
  1 sibling, 1 reply; 75+ messages in thread
From: Chris Murphy @ 2019-12-29 22:07 UTC (permalink / raw)
  To: Patrick Erley; +Cc: Btrfs BTRFS

On Sun, Dec 29, 2019 at 1:44 PM Patrick Erley <pat-lkml@erley.org> wrote:
>
> On a system where I've been tinkering with linux-next for years, my /
> has got some errors.  When migrating from 5.1 to 5.2, I saw these
> errors, but just ignored them and went back to 5.1, and continued my
> tinkering.  Over the holidays, I decided to try to upgrade the kernel,
> saw the errors again, and decided to look into them, finding the
> tree-checker page on the kernel docs, and am writing this e-mail.
>
> My / does not contain anything sensitive or important, as /home and
> /usr/src are both subvolumes on a separate larger drive.
>
> btrfs fi show:
> Label: none  uuid: 815266d6-a8b9-4f63-a593-02fde178263f
> Total devices 2 FS bytes used 93.81GiB
> devid    1 size 115.12GiB used 115.11GiB path /dev/nvme0n1p2
> devid    3 size 115.12GiB used 115.11GiB path /dev/sda3
>
> Label: none  uuid: 4bd97711-b63c-40cb-bfa5-aa9c2868cf98
> Total devices 1 FS bytes used 536.48GiB
> devid    1 size 834.63GiB used 833.59GiB path /dev/sda5
>
> Booting a more recent kernel, I get spammed with:
> [    8.243899] BTRFS critical (device nvme0n1p2): corrupt leaf: root=5
> block=303629811712 slot=30 ino=5380870, invalid inode generation: has
> 13221446351398931016 expect [0, 2997884]
> [    8.243902] BTRFS error (device nvme0n1p2): block=303629811712 read
> time tree block corruption detected
>
> There are 6 corrupted inodes:
> cat dmesg.foo  | grep "BTRFS critical" | sed -re
> 's:.*block=([0-9]*).*ino=([0-9]+).*:\1 \2:' | sort | uniq
> 303629811712 5380870
> 303712501760 3277548
> 303861395456 5909140
> 304079065088 2228479
> 304573444096 3539224
> 305039556608 1442149
>
> and they all have the same value for the inode generation.  Before I
> reboot into a livecd and btrfs check --repair, is there anything
> interesting that a snapshot of the state would show?  I have 800gb
> unpartitioned on the nvme, so backing up before is already in the
> plans, and I could just as easily grab an image of the partitions
> while I'm at it.

I'm not certain whether btrfs check can fix these kinds of errors yet.
Can you use btrfs-progs v5.4 and just do a 'btrfs check' and also
again with 'btrfs check --mode=lowmem' ? I'm curious if either mode
sees the same problem the kernel tree checker sees.

-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-29 22:07 ` Chris Murphy
@ 2019-12-29 22:27   ` Patrick Erley
  2019-12-29 22:32     ` Chris Murphy
  0 siblings, 1 reply; 75+ messages in thread
From: Patrick Erley @ 2019-12-29 22:27 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Btrfs BTRFS

Should I --force it while mounted, or run the checks from RO mount status?

On Sun, Dec 29, 2019 at 2:07 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sun, Dec 29, 2019 at 1:44 PM Patrick Erley <pat-lkml@erley.org> wrote:
> >
> > On a system where I've been tinkering with linux-next for years, my /
> > has got some errors.  When migrating from 5.1 to 5.2, I saw these
> > errors, but just ignored them and went back to 5.1, and continued my
> > tinkering.  Over the holidays, I decided to try to upgrade the kernel,
> > saw the errors again, and decided to look into them, finding the
> > tree-checker page on the kernel docs, and am writing this e-mail.
> >
> > My / does not contain anything sensitive or important, as /home and
> > /usr/src are both subvolumes on a separate larger drive.
> >
> > btrfs fi show:
> > Label: none  uuid: 815266d6-a8b9-4f63-a593-02fde178263f
> > Total devices 2 FS bytes used 93.81GiB
> > devid    1 size 115.12GiB used 115.11GiB path /dev/nvme0n1p2
> > devid    3 size 115.12GiB used 115.11GiB path /dev/sda3
> >
> > Label: none  uuid: 4bd97711-b63c-40cb-bfa5-aa9c2868cf98
> > Total devices 1 FS bytes used 536.48GiB
> > devid    1 size 834.63GiB used 833.59GiB path /dev/sda5
> >
> > Booting a more recent kernel, I get spammed with:
> > [    8.243899] BTRFS critical (device nvme0n1p2): corrupt leaf: root=5
> > block=303629811712 slot=30 ino=5380870, invalid inode generation: has
> > 13221446351398931016 expect [0, 2997884]
> > [    8.243902] BTRFS error (device nvme0n1p2): block=303629811712 read
> > time tree block corruption detected
> >
> > There are 6 corrupted inodes:
> > cat dmesg.foo  | grep "BTRFS critical" | sed -re
> > 's:.*block=([0-9]*).*ino=([0-9]+).*:\1 \2:' | sort | uniq
> > 303629811712 5380870
> > 303712501760 3277548
> > 303861395456 5909140
> > 304079065088 2228479
> > 304573444096 3539224
> > 305039556608 1442149
> >
> > and they all have the same value for the inode generation.  Before I
> > reboot into a livecd and btrfs check --repair, is there anything
> > interesting that a snapshot of the state would show?  I have 800gb
> > unpartitioned on the nvme, so backing up before is already in the
> > plans, and I could just as easily grab an image of the partitions
> > while I'm at it.
>
> I'm not certain whether btrfs check can fix these kinds of errors yet.
> Can you use btrfs-progs v5.4 and just do a 'btrfs check' and also
> again with 'btrfs check --mode=lowmem' ? I'm curious if either mode
> sees the same problem the kernel tree checker sees.
>
> --
> Chris Murphy

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-29 22:27   ` Patrick Erley
@ 2019-12-29 22:32     ` Chris Murphy
  2019-12-29 22:36       ` Patrick Erley
  0 siblings, 1 reply; 75+ messages in thread
From: Chris Murphy @ 2019-12-29 22:32 UTC (permalink / raw)
  To: Patrick Erley; +Cc: Chris Murphy, Btrfs BTRFS

On Sun, Dec 29, 2019 at 3:28 PM Patrick Erley <pat-lkml@erley.org> wrote:
>
> Should I --force it while mounted, or run the checks from RO mount status?

Check isn't reliable when the volume is mounted rw so I personally
wouldn't bother trying.


-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-29 22:32     ` Chris Murphy
@ 2019-12-29 22:36       ` Patrick Erley
  2019-12-29 23:11         ` Chris Murphy
  0 siblings, 1 reply; 75+ messages in thread
From: Patrick Erley @ 2019-12-29 22:36 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Btrfs BTRFS

K.  Gotta roll a new initrd with btrfs 5.4, then will reboot into it
and grab logs of btrfs check (both iterations, against each of my
volumes).  Probably be a few hours, so don't expect another follow up
until tomorrow.  Thanks for chiming in.

Would it be worth creating an image of the drive while I'm in single
user mode for faster analysis/iteration?

On Sun, Dec 29, 2019 at 2:32 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sun, Dec 29, 2019 at 3:28 PM Patrick Erley <pat-lkml@erley.org> wrote:
> >
> > Should I --force it while mounted, or run the checks from RO mount status?
>
> Check isn't reliable when the volume is mounted rw so I personally
> wouldn't bother trying.
>
>
> --
> Chris Murphy

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-29 22:36       ` Patrick Erley
@ 2019-12-29 23:11         ` Chris Murphy
  2019-12-29 23:19           ` Patrick Erley
  0 siblings, 1 reply; 75+ messages in thread
From: Chris Murphy @ 2019-12-29 23:11 UTC (permalink / raw)
  To: Patrick Erley; +Cc: Chris Murphy, Btrfs BTRFS

On Sun, Dec 29, 2019 at 3:36 PM Patrick Erley <pat-lkml@erley.org> wrote:
>
> K.  Gotta roll a new initrd with btrfs 5.4, then will reboot into it
> and grab logs of btrfs check (both iterations, against each of my
> volumes).  Probably be a few hours, so don't expect another follow up
> until tomorrow.  Thanks for chiming in.
>
> Would it be worth creating an image of the drive while I'm in single
> user mode for faster analysis/iteration?

Depends on the developer who replies. These days they usually just ask
for selected dump-tree output. The image might be useful if you can't
wait and want to blow away the file system and start over sooner than
later.



-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-29 23:11         ` Chris Murphy
@ 2019-12-29 23:19           ` Patrick Erley
  2019-12-29 23:24             ` Chris Murphy
  0 siblings, 1 reply; 75+ messages in thread
From: Patrick Erley @ 2019-12-29 23:19 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Btrfs BTRFS

Nah, this 'corruption' hasn't affected anything other than not being
able to go to a newer kernel.  The system doesn't do anything
important.  I'll grab the logs, and clone an image of the drive so I
don't need to go single user RO mode to get dumps for devs when follow
up requests come in.

On Sun, Dec 29, 2019 at 3:11 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sun, Dec 29, 2019 at 3:36 PM Patrick Erley <pat-lkml@erley.org> wrote:
> >
> > K.  Gotta roll a new initrd with btrfs 5.4, then will reboot into it
> > and grab logs of btrfs check (both iterations, against each of my
> > volumes).  Probably be a few hours, so don't expect another follow up
> > until tomorrow.  Thanks for chiming in.
> >
> > Would it be worth creating an image of the drive while I'm in single
> > user mode for faster analysis/iteration?
>
> Depends on the developer who replies. These days they usually just ask
> for selected dump-tree output. The image might be useful if you can't
> wait and want to blow away the file system and start over sooner than
> later.
>
>
>
> --
> Chris Murphy

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-29 23:19           ` Patrick Erley
@ 2019-12-29 23:24             ` Chris Murphy
  2019-12-29 23:26               ` Patrick Erley
  0 siblings, 1 reply; 75+ messages in thread
From: Chris Murphy @ 2019-12-29 23:24 UTC (permalink / raw)
  To: Patrick Erley; +Cc: Chris Murphy, Btrfs BTRFS

On Sun, Dec 29, 2019 at 4:19 PM Patrick Erley <pat-lkml@erley.org> wrote:
>
> Nah, this 'corruption' hasn't affected anything other than not being
> able to go to a newer kernel.  The system doesn't do anything
> important.  I'll grab the logs, and clone an image of the drive so I
> don't need to go single user RO mode to get dumps for devs when follow
> up requests come in.

A clone of any rw file system is likewise unreliable, for the same
reasons that a forced btrfs check on a rw file system is unreliable.
So unreliable it's a waste of time for you and the developer, so I
wouldn't even bother with it.


-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-29 23:24             ` Chris Murphy
@ 2019-12-29 23:26               ` Patrick Erley
  0 siblings, 0 replies; 75+ messages in thread
From: Patrick Erley @ 2019-12-29 23:26 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Btrfs BTRFS

No, I mean, I'm going to reboot single user RO, run the btrfs checks,
catch the logs, and dump a copy of the partition, so follow ups don't
require RO reboots.

On Sun, Dec 29, 2019 at 3:25 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sun, Dec 29, 2019 at 4:19 PM Patrick Erley <pat-lkml@erley.org> wrote:
> >
> > Nah, this 'corruption' hasn't affected anything other than not being
> > able to go to a newer kernel.  The system doesn't do anything
> > important.  I'll grab the logs, and clone an image of the drive so I
> > don't need to go single user RO mode to get dumps for devs when follow
> > up requests come in.
>
> A clone of any rw file system is likewise unreliable, for the same
> reasons that a forced btrfs check on a rw file system is unreliable.
> So unreliable it's a waste of time for you and the developer, so I
> wouldn't even bother with it.
>
>
> --
> Chris Murphy

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-29 20:43 read time tree block corruption detected Patrick Erley
  2019-12-29 22:07 ` Chris Murphy
@ 2019-12-30  0:46 ` Qu Wenruo
  2019-12-30  5:36   ` Patrick Erley
  1 sibling, 1 reply; 75+ messages in thread
From: Qu Wenruo @ 2019-12-30  0:46 UTC (permalink / raw)
  To: Patrick Erley, linux-btrfs


[-- Attachment #1.1: Type: text/plain, Size: 2331 bytes --]



On 2019/12/30 上午4:43, Patrick Erley wrote:
> On a system where I've been tinkering with linux-next for years, my /
> has got some errors.  When migrating from 5.1 to 5.2, I saw these
> errors, but just ignored them and went back to 5.1, and continued my
> tinkering.  Over the holidays, I decided to try to upgrade the kernel,
> saw the errors again, and decided to look into them, finding the
> tree-checker page on the kernel docs, and am writing this e-mail.
> 
> My / does not contain anything sensitive or important, as /home and
> /usr/src are both subvolumes on a separate larger drive.
> 
> btrfs fi show:
> Label: none  uuid: 815266d6-a8b9-4f63-a593-02fde178263f
> Total devices 2 FS bytes used 93.81GiB
> devid    1 size 115.12GiB used 115.11GiB path /dev/nvme0n1p2
> devid    3 size 115.12GiB used 115.11GiB path /dev/sda3
> 
> Label: none  uuid: 4bd97711-b63c-40cb-bfa5-aa9c2868cf98
> Total devices 1 FS bytes used 536.48GiB
> devid    1 size 834.63GiB used 833.59GiB path /dev/sda5
> 
> Booting a more recent kernel, I get spammed with:
> [    8.243899] BTRFS critical (device nvme0n1p2): corrupt leaf: root=5
> block=303629811712 slot=30 ino=5380870, invalid inode generation: has
> 13221446351398931016 expect [0, 2997884]

Inode generation repair is introduced in v5.4. So feel free to use
`btrfs check --repair` to repair such problems.

But please run a `btrfs check` without --repair and paste the output,
just in case there are extra problems which may screw up repair.

Thanks,
Qu

> [    8.243902] BTRFS error (device nvme0n1p2): block=303629811712 read
> time tree block corruption detected
> 
> There are 6 corrupted inodes:
> cat dmesg.foo  | grep "BTRFS critical" | sed -re
> 's:.*block=([0-9]*).*ino=([0-9]+).*:\1 \2:' | sort | uniq
> 303629811712 5380870
> 303712501760 3277548
> 303861395456 5909140
> 304079065088 2228479
> 304573444096 3539224
> 305039556608 1442149
> 
> and they all have the same value for the inode generation.  Before I
> reboot into a livecd and btrfs check --repair, is there anything
> interesting that a snapshot of the state would show?  I have 800gb
> unpartitioned on the nvme, so backing up before is already in the
> plans, and I could just as easily grab an image of the partitions
> while I'm at it.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-30  0:46 ` Qu Wenruo
@ 2019-12-30  5:36   ` Patrick Erley
  2019-12-30  5:43     ` Qu Wenruo
  0 siblings, 1 reply; 75+ messages in thread
From: Patrick Erley @ 2019-12-30  5:36 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Btrfs BTRFS

(ugh, just realized gmail does top replies.  Sorry... will try to
figure out how to make gsuite behave like a sane mail client before my
next reply):

here's btrfs check /dev/nvme0n1p2 (sda3, which is a mirror of it, has
exactly the same output)

[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
Opening filesystem to check...
Checking filesystem on /dev/nvme0n1p2
UUID: 815266d6-a8b9-4f63-a593-02fde178263f
found 89383137280 bytes used, no error found
total csum bytes: 85617340
total tree bytes: 1670774784
total fs tree bytes: 1451180032
total extent tree bytes: 107905024
btree space waste bytes: 413362851
file data blocks allocated: 90769887232
 referenced 88836960256

And here's the 'lowmen' variant:

[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
ERROR: root 5 EXTENT_DATA[266 1409024] csum missing, have: 0, expected: 65536
ERROR: root 5 INODE[266] nbytes 1437696 not equal to extent_size 1503232
ERROR: root 5 EXTENT_DATA[4731008 4096] csum missing, have: 0, expected: 2093056
ERROR: root 5 EXTENT_DATA[4731008 2101248] csum missing, have: 0, expected: 8192
ERROR: root 5 INODE[4731008] nbytes 73728 not equal to extent_size 2174976
ERROR: root 5 EXTENT_DATA[4731012 4096] csum missing, have: 0, expected: 8192
ERROR: root 5 INODE[4731012] nbytes 8192 not equal to extent_size 16384
ERROR: root 5 EXTENT_DATA[6563647 8192] csum missing, have: 0, expected: 4096
ERROR: root 5 INODE[6563647] nbytes 12288 not equal to extent_size 16384
ERROR: root 5 EXTENT_DATA[7090739 131072] csum missing, have: 0, expected: 24576
ERROR: root 5 INODE[7090739] nbytes 139264 not equal to extent_size 163840
ERROR: errors found in fs roots
Opening filesystem to check...
Checking filesystem on /dev/nvme0n1p2
UUID: 815266d6-a8b9-4f63-a593-02fde178263f
found 89383137280 bytes used, error(s) found
total csum bytes: 85617340
total tree bytes: 1670774784
total fs tree bytes: 1451180032
total extent tree bytes: 107905024
btree space waste bytes: 413362851
file data blocks allocated: 90769887232
 referenced 88836960256

On Sun, Dec 29, 2019 at 4:46 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2019/12/30 上午4:43, Patrick Erley wrote:
> > On a system where I've been tinkering with linux-next for years, my /
> > has got some errors.  When migrating from 5.1 to 5.2, I saw these
> > errors, but just ignored them and went back to 5.1, and continued my
> > tinkering.  Over the holidays, I decided to try to upgrade the kernel,
> > saw the errors again, and decided to look into them, finding the
> > tree-checker page on the kernel docs, and am writing this e-mail.
> >
> > My / does not contain anything sensitive or important, as /home and
> > /usr/src are both subvolumes on a separate larger drive.
> >
> > btrfs fi show:
> > Label: none  uuid: 815266d6-a8b9-4f63-a593-02fde178263f
> > Total devices 2 FS bytes used 93.81GiB
> > devid    1 size 115.12GiB used 115.11GiB path /dev/nvme0n1p2
> > devid    3 size 115.12GiB used 115.11GiB path /dev/sda3
> >
> > Label: none  uuid: 4bd97711-b63c-40cb-bfa5-aa9c2868cf98
> > Total devices 1 FS bytes used 536.48GiB
> > devid    1 size 834.63GiB used 833.59GiB path /dev/sda5
> >
> > Booting a more recent kernel, I get spammed with:
> > [    8.243899] BTRFS critical (device nvme0n1p2): corrupt leaf: root=5
> > block=303629811712 slot=30 ino=5380870, invalid inode generation: has
> > 13221446351398931016 expect [0, 2997884]
>
> Inode generation repair is introduced in v5.4. So feel free to use
> `btrfs check --repair` to repair such problems.
>
> But please run a `btrfs check` without --repair and paste the output,
> just in case there are extra problems which may screw up repair.
>
> Thanks,
> Qu
>
> > [    8.243902] BTRFS error (device nvme0n1p2): block=303629811712 read
> > time tree block corruption detected
> >
> > There are 6 corrupted inodes:
> > cat dmesg.foo  | grep "BTRFS critical" | sed -re
> > 's:.*block=([0-9]*).*ino=([0-9]+).*:\1 \2:' | sort | uniq
> > 303629811712 5380870
> > 303712501760 3277548
> > 303861395456 5909140
> > 304079065088 2228479
> > 304573444096 3539224
> > 305039556608 1442149
> >
> > and they all have the same value for the inode generation.  Before I
> > reboot into a livecd and btrfs check --repair, is there anything
> > interesting that a snapshot of the state would show?  I have 800gb
> > unpartitioned on the nvme, so backing up before is already in the
> > plans, and I could just as easily grab an image of the partitions
> > while I'm at it.
> >
>

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-30  5:36   ` Patrick Erley
@ 2019-12-30  5:43     ` Qu Wenruo
  2019-12-30  5:47       ` Patrick Erley
  0 siblings, 1 reply; 75+ messages in thread
From: Qu Wenruo @ 2019-12-30  5:43 UTC (permalink / raw)
  To: Patrick Erley; +Cc: Btrfs BTRFS


[-- Attachment #1.1: Type: text/plain, Size: 5560 bytes --]



On 2019/12/30 下午1:36, Patrick Erley wrote:
> (ugh, just realized gmail does top replies.  Sorry... will try to
> figure out how to make gsuite behave like a sane mail client before my
> next reply):
> 
> here's btrfs check /dev/nvme0n1p2 (sda3, which is a mirror of it, has
> exactly the same output)
> 
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> Opening filesystem to check...
> Checking filesystem on /dev/nvme0n1p2
> UUID: 815266d6-a8b9-4f63-a593-02fde178263f
> found 89383137280 bytes used, no error found
> total csum bytes: 85617340
> total tree bytes: 1670774784
> total fs tree bytes: 1451180032
> total extent tree bytes: 107905024
> btree space waste bytes: 413362851
> file data blocks allocated: 90769887232
>  referenced 88836960256

It looks too good to be true, is the btrfs-progs v5.4? IIRC in v5.4 we
should report inodes generation problems.

But anyway, your fs looks pretty OK to have a --repair run on it.
Please note that, only v5.4 has inode generation repair, older ones
can't really repair such problem.

> 
> And here's the 'lowmen' variant:
> 
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> [4/7] checking fs roots
> ERROR: root 5 EXTENT_DATA[266 1409024] csum missing, have: 0, expected: 65536
> ERROR: root 5 INODE[266] nbytes 1437696 not equal to extent_size 1503232
> ERROR: root 5 EXTENT_DATA[4731008 4096] csum missing, have: 0, expected: 2093056
> ERROR: root 5 EXTENT_DATA[4731008 2101248] csum missing, have: 0, expected: 8192
> ERROR: root 5 INODE[4731008] nbytes 73728 not equal to extent_size 2174976
> ERROR: root 5 EXTENT_DATA[4731012 4096] csum missing, have: 0, expected: 8192
> ERROR: root 5 INODE[4731012] nbytes 8192 not equal to extent_size 16384
> ERROR: root 5 EXTENT_DATA[6563647 8192] csum missing, have: 0, expected: 4096
> ERROR: root 5 INODE[6563647] nbytes 12288 not equal to extent_size 16384
> ERROR: root 5 EXTENT_DATA[7090739 131072] csum missing, have: 0, expected: 24576
> ERROR: root 5 INODE[7090739] nbytes 139264 not equal to extent_size 163840

Those are all minor problems, and still no generation problem reported.
All these minor problem can also be repaired, or they are just false
reported in older lowmem mode.

It should be pretty OK to have a v5.4 repair run.

Thanks,
Qu

> ERROR: errors found in fs roots
> Opening filesystem to check...
> Checking filesystem on /dev/nvme0n1p2
> UUID: 815266d6-a8b9-4f63-a593-02fde178263f
> found 89383137280 bytes used, error(s) found
> total csum bytes: 85617340
> total tree bytes: 1670774784
> total fs tree bytes: 1451180032
> total extent tree bytes: 107905024
> btree space waste bytes: 413362851
> file data blocks allocated: 90769887232
>  referenced 88836960256
> 
> On Sun, Dec 29, 2019 at 4:46 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2019/12/30 上午4:43, Patrick Erley wrote:
>>> On a system where I've been tinkering with linux-next for years, my /
>>> has got some errors.  When migrating from 5.1 to 5.2, I saw these
>>> errors, but just ignored them and went back to 5.1, and continued my
>>> tinkering.  Over the holidays, I decided to try to upgrade the kernel,
>>> saw the errors again, and decided to look into them, finding the
>>> tree-checker page on the kernel docs, and am writing this e-mail.
>>>
>>> My / does not contain anything sensitive or important, as /home and
>>> /usr/src are both subvolumes on a separate larger drive.
>>>
>>> btrfs fi show:
>>> Label: none  uuid: 815266d6-a8b9-4f63-a593-02fde178263f
>>> Total devices 2 FS bytes used 93.81GiB
>>> devid    1 size 115.12GiB used 115.11GiB path /dev/nvme0n1p2
>>> devid    3 size 115.12GiB used 115.11GiB path /dev/sda3
>>>
>>> Label: none  uuid: 4bd97711-b63c-40cb-bfa5-aa9c2868cf98
>>> Total devices 1 FS bytes used 536.48GiB
>>> devid    1 size 834.63GiB used 833.59GiB path /dev/sda5
>>>
>>> Booting a more recent kernel, I get spammed with:
>>> [    8.243899] BTRFS critical (device nvme0n1p2): corrupt leaf: root=5
>>> block=303629811712 slot=30 ino=5380870, invalid inode generation: has
>>> 13221446351398931016 expect [0, 2997884]
>>
>> Inode generation repair is introduced in v5.4. So feel free to use
>> `btrfs check --repair` to repair such problems.
>>
>> But please run a `btrfs check` without --repair and paste the output,
>> just in case there are extra problems which may screw up repair.
>>
>> Thanks,
>> Qu
>>
>>> [    8.243902] BTRFS error (device nvme0n1p2): block=303629811712 read
>>> time tree block corruption detected
>>>
>>> There are 6 corrupted inodes:
>>> cat dmesg.foo  | grep "BTRFS critical" | sed -re
>>> 's:.*block=([0-9]*).*ino=([0-9]+).*:\1 \2:' | sort | uniq
>>> 303629811712 5380870
>>> 303712501760 3277548
>>> 303861395456 5909140
>>> 304079065088 2228479
>>> 304573444096 3539224
>>> 305039556608 1442149
>>>
>>> and they all have the same value for the inode generation.  Before I
>>> reboot into a livecd and btrfs check --repair, is there anything
>>> interesting that a snapshot of the state would show?  I have 800gb
>>> unpartitioned on the nvme, so backing up before is already in the
>>> plans, and I could just as easily grab an image of the partitions
>>> while I'm at it.
>>>
>>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-30  5:43     ` Qu Wenruo
@ 2019-12-30  5:47       ` Patrick Erley
  2019-12-30  5:50         ` Patrick Erley
  0 siblings, 1 reply; 75+ messages in thread
From: Patrick Erley @ 2019-12-30  5:47 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Btrfs BTRFS

On Sun, Dec 29, 2019 at 9:43 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2019/12/30 下午1:36, Patrick Erley wrote:
> > (ugh, just realized gmail does top replies.  Sorry... will try to
> > figure out how to make gsuite behave like a sane mail client before my
> > next reply):
> >
> > here's btrfs check /dev/nvme0n1p2 (sda3, which is a mirror of it, has
> > exactly the same output)
> >
> > [1/7] checking root items
> > [2/7] checking extents
> > [3/7] checking free space cache
> > [4/7] checking fs roots
> > [5/7] checking only csums items (without verifying data)
> > [6/7] checking root refs
> > [7/7] checking quota groups skipped (not enabled on this FS)
> > Opening filesystem to check...
> > Checking filesystem on /dev/nvme0n1p2
> > UUID: 815266d6-a8b9-4f63-a593-02fde178263f
> > found 89383137280 bytes used, no error found
> > total csum bytes: 85617340
> > total tree bytes: 1670774784
> > total fs tree bytes: 1451180032
> > total extent tree bytes: 107905024
> > btree space waste bytes: 413362851
> > file data blocks allocated: 90769887232
> >  referenced 88836960256
>
> It looks too good to be true, is the btrfs-progs v5.4? IIRC in v5.4 we
> should report inodes generation problems.

Hurray Bottom Reply?

/usr/src/initramfs/bin $ ./btrfs.static --version
btrfs-progs v5.4

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-30  5:47       ` Patrick Erley
@ 2019-12-30  5:50         ` Patrick Erley
  2019-12-30  5:58           ` Qu Wenruo
  0 siblings, 1 reply; 75+ messages in thread
From: Patrick Erley @ 2019-12-30  5:50 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Btrfs BTRFS

On Sun, Dec 29, 2019 at 9:47 PM Patrick Erley <pat-lkml@erley.org> wrote:
>
> On Sun, Dec 29, 2019 at 9:43 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >
> >
> >
> > On 2019/12/30 下午1:36, Patrick Erley wrote:
> > > (ugh, just realized gmail does top replies.  Sorry... will try to
> > > figure out how to make gsuite behave like a sane mail client before my
> > > next reply):
> > >
> > > here's btrfs check /dev/nvme0n1p2 (sda3, which is a mirror of it, has
> > > exactly the same output)
> > >
> > > [1/7] checking root items
> > > [2/7] checking extents
> > > [3/7] checking free space cache
> > > [4/7] checking fs roots
> > > [5/7] checking only csums items (without verifying data)
> > > [6/7] checking root refs
> > > [7/7] checking quota groups skipped (not enabled on this FS)
> > > Opening filesystem to check...
> > > Checking filesystem on /dev/nvme0n1p2
> > > UUID: 815266d6-a8b9-4f63-a593-02fde178263f
> > > found 89383137280 bytes used, no error found
> > > total csum bytes: 85617340
> > > total tree bytes: 1670774784
> > > total fs tree bytes: 1451180032
> > > total extent tree bytes: 107905024
> > > btree space waste bytes: 413362851
> > > file data blocks allocated: 90769887232
> > >  referenced 88836960256
> >
> > It looks too good to be true, is the btrfs-progs v5.4? IIRC in v5.4 we
> > should report inodes generation problems.
>
> Hurray Bottom Reply?
>
> /usr/src/initramfs/bin $ ./btrfs.static --version
> btrfs-progs v5.4

Dumb question, did I need to do that while booting a post 5.1 kernel?
I ran these while not having the filesystem mounted, but against
kernel 5.1.  I can easily repeat against 5.4.

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-30  5:50         ` Patrick Erley
@ 2019-12-30  5:58           ` Qu Wenruo
  2019-12-30  6:07             ` Patrick Erley
  0 siblings, 1 reply; 75+ messages in thread
From: Qu Wenruo @ 2019-12-30  5:58 UTC (permalink / raw)
  To: Patrick Erley; +Cc: Btrfs BTRFS


[-- Attachment #1.1: Type: text/plain, Size: 2950 bytes --]



On 2019/12/30 下午1:50, Patrick Erley wrote:
> On Sun, Dec 29, 2019 at 9:47 PM Patrick Erley <pat-lkml@erley.org> wrote:
>>
>> On Sun, Dec 29, 2019 at 9:43 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>
>>>
>>>
>>> On 2019/12/30 下午1:36, Patrick Erley wrote:
>>>> (ugh, just realized gmail does top replies.  Sorry... will try to
>>>> figure out how to make gsuite behave like a sane mail client before my
>>>> next reply):
>>>>
>>>> here's btrfs check /dev/nvme0n1p2 (sda3, which is a mirror of it, has
>>>> exactly the same output)
>>>>
>>>> [1/7] checking root items
>>>> [2/7] checking extents
>>>> [3/7] checking free space cache
>>>> [4/7] checking fs roots
>>>> [5/7] checking only csums items (without verifying data)
>>>> [6/7] checking root refs
>>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>>> Opening filesystem to check...
>>>> Checking filesystem on /dev/nvme0n1p2
>>>> UUID: 815266d6-a8b9-4f63-a593-02fde178263f
>>>> found 89383137280 bytes used, no error found
>>>> total csum bytes: 85617340
>>>> total tree bytes: 1670774784
>>>> total fs tree bytes: 1451180032
>>>> total extent tree bytes: 107905024
>>>> btree space waste bytes: 413362851
>>>> file data blocks allocated: 90769887232
>>>>  referenced 88836960256
>>>
>>> It looks too good to be true, is the btrfs-progs v5.4? IIRC in v5.4 we
>>> should report inodes generation problems.
>>
>> Hurray Bottom Reply?
>>
>> /usr/src/initramfs/bin $ ./btrfs.static --version
>> btrfs-progs v5.4

This is strange.


6084adam|thinkpad|~$ btrfs check --mode=lowmem test.img
Opening filesystem to check...
Checking filesystem on test.img
UUID: c6c6ddd2-01c1-47fc-b699-cacfae9d4bfd
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
cache and super generation don't match, space cache will be invalidated
[4/7] checking fs roots
ERROR: invalid inode generation for ino 257, have 8858344568388091671
expect [0, 9)
ERROR: errors found in fs roots
found 131072 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 131072
total fs tree bytes: 32768
total extent tree bytes: 16384
btree space waste bytes: 123409
file data blocks allocated: 0
 referenced 0
6085adam|thinkpad|~$ btrfs --version
btrfs-progs v5.4

As expected, v5.4 should detect such problem without problem.

Would you please provide extra tree dump to help us to determine what
makes btrfs check unable to detect such problems?

# btrfs ins dump-tree -b 303629811712 /dev/dm-1

> 
> Dumb question, did I need to do that while booting a post 5.1 kernel?
> I ran these while not having the filesystem mounted, but against
> kernel 5.1.  I can easily repeat against 5.4.

Kernel version doesn't affect at all. All the detection and repair
should be able to be done by btrfs-progs.

But it would be better to use v5.4 to quick verify the fix is working.

Thanks,
Qu


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-30  5:58           ` Qu Wenruo
@ 2019-12-30  6:07             ` Patrick Erley
  2019-12-30  6:09               ` Qu Wenruo
  0 siblings, 1 reply; 75+ messages in thread
From: Patrick Erley @ 2019-12-30  6:07 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Btrfs BTRFS

On Sun, Dec 29, 2019 at 9:58 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2019/12/30 下午1:50, Patrick Erley wrote:
> > On Sun, Dec 29, 2019 at 9:47 PM Patrick Erley <pat-lkml@erley.org> wrote:
> >>
> >> On Sun, Dec 29, 2019 at 9:43 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>
> >>>
> >>>
> >>> On 2019/12/30 下午1:36, Patrick Erley wrote:
> >>>> (ugh, just realized gmail does top replies.  Sorry... will try to
> >>>> figure out how to make gsuite behave like a sane mail client before my
> >>>> next reply):
> >>>>
> >>>> here's btrfs check /dev/nvme0n1p2 (sda3, which is a mirror of it, has
> >>>> exactly the same output)
> >>>>
> >>>> [1/7] checking root items
> >>>> [2/7] checking extents
> >>>> [3/7] checking free space cache
> >>>> [4/7] checking fs roots
> >>>> [5/7] checking only csums items (without verifying data)
> >>>> [6/7] checking root refs
> >>>> [7/7] checking quota groups skipped (not enabled on this FS)
> >>>> Opening filesystem to check...
> >>>> Checking filesystem on /dev/nvme0n1p2
> >>>> UUID: 815266d6-a8b9-4f63-a593-02fde178263f
> >>>> found 89383137280 bytes used, no error found
> >>>> total csum bytes: 85617340
> >>>> total tree bytes: 1670774784
> >>>> total fs tree bytes: 1451180032
> >>>> total extent tree bytes: 107905024
> >>>> btree space waste bytes: 413362851
> >>>> file data blocks allocated: 90769887232
> >>>>  referenced 88836960256
> >>>
> >>> It looks too good to be true, is the btrfs-progs v5.4? IIRC in v5.4 we
> >>> should report inodes generation problems.
> >>
> >> Hurray Bottom Reply?
> >>
> >> /usr/src/initramfs/bin $ ./btrfs.static --version
> >> btrfs-progs v5.4
>
> This is strange.
>
>
> 6084adam|thinkpad|~$ btrfs check --mode=lowmem test.img
> Opening filesystem to check...
> Checking filesystem on test.img
> UUID: c6c6ddd2-01c1-47fc-b699-cacfae9d4bfd
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> cache and super generation don't match, space cache will be invalidated
> [4/7] checking fs roots
> ERROR: invalid inode generation for ino 257, have 8858344568388091671
> expect [0, 9)
> ERROR: errors found in fs roots
> found 131072 bytes used, error(s) found
> total csum bytes: 0
> total tree bytes: 131072
> total fs tree bytes: 32768
> total extent tree bytes: 16384
> btree space waste bytes: 123409
> file data blocks allocated: 0
>  referenced 0
> 6085adam|thinkpad|~$ btrfs --version
> btrfs-progs v5.4
>
> As expected, v5.4 should detect such problem without problem.
>
> Would you please provide extra tree dump to help us to determine what
> makes btrfs check unable to detect such problems?
>
> # btrfs ins dump-tree -b 303629811712 /dev/dm-1
anvil ~ # btrfs ins dump-tree -b 303629811712 /dev/nvme0n1p2
btrfs-progs v5.4
Invalid mapping for 303629811712-303629815808, got 476092633088-477166374912
Couldn't map the block 303629811712
Couldn't map the block 303629811712
bad tree block 303629811712, bytenr mismatch, want=303629811712, have=0
ERROR: failed to read tree block 303629811712

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-30  6:07             ` Patrick Erley
@ 2019-12-30  6:09               ` Qu Wenruo
  2019-12-30  8:14                 ` Patrick Erley
  0 siblings, 1 reply; 75+ messages in thread
From: Qu Wenruo @ 2019-12-30  6:09 UTC (permalink / raw)
  To: Patrick Erley; +Cc: Btrfs BTRFS


[-- Attachment #1.1: Type: text/plain, Size: 3490 bytes --]



On 2019/12/30 下午2:07, Patrick Erley wrote:
> On Sun, Dec 29, 2019 at 9:58 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2019/12/30 下午1:50, Patrick Erley wrote:
>>> On Sun, Dec 29, 2019 at 9:47 PM Patrick Erley <pat-lkml@erley.org> wrote:
>>>>
>>>> On Sun, Dec 29, 2019 at 9:43 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 2019/12/30 下午1:36, Patrick Erley wrote:
>>>>>> (ugh, just realized gmail does top replies.  Sorry... will try to
>>>>>> figure out how to make gsuite behave like a sane mail client before my
>>>>>> next reply):
>>>>>>
>>>>>> here's btrfs check /dev/nvme0n1p2 (sda3, which is a mirror of it, has
>>>>>> exactly the same output)
>>>>>>
>>>>>> [1/7] checking root items
>>>>>> [2/7] checking extents
>>>>>> [3/7] checking free space cache
>>>>>> [4/7] checking fs roots
>>>>>> [5/7] checking only csums items (without verifying data)
>>>>>> [6/7] checking root refs
>>>>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>>>>> Opening filesystem to check...
>>>>>> Checking filesystem on /dev/nvme0n1p2
>>>>>> UUID: 815266d6-a8b9-4f63-a593-02fde178263f
>>>>>> found 89383137280 bytes used, no error found
>>>>>> total csum bytes: 85617340
>>>>>> total tree bytes: 1670774784
>>>>>> total fs tree bytes: 1451180032
>>>>>> total extent tree bytes: 107905024
>>>>>> btree space waste bytes: 413362851
>>>>>> file data blocks allocated: 90769887232
>>>>>>  referenced 88836960256
>>>>>
>>>>> It looks too good to be true, is the btrfs-progs v5.4? IIRC in v5.4 we
>>>>> should report inodes generation problems.
>>>>
>>>> Hurray Bottom Reply?
>>>>
>>>> /usr/src/initramfs/bin $ ./btrfs.static --version
>>>> btrfs-progs v5.4
>>
>> This is strange.
>>
>>
>> 6084adam|thinkpad|~$ btrfs check --mode=lowmem test.img
>> Opening filesystem to check...
>> Checking filesystem on test.img
>> UUID: c6c6ddd2-01c1-47fc-b699-cacfae9d4bfd
>> [1/7] checking root items
>> [2/7] checking extents
>> [3/7] checking free space cache
>> cache and super generation don't match, space cache will be invalidated
>> [4/7] checking fs roots
>> ERROR: invalid inode generation for ino 257, have 8858344568388091671
>> expect [0, 9)
>> ERROR: errors found in fs roots
>> found 131072 bytes used, error(s) found
>> total csum bytes: 0
>> total tree bytes: 131072
>> total fs tree bytes: 32768
>> total extent tree bytes: 16384
>> btree space waste bytes: 123409
>> file data blocks allocated: 0
>>  referenced 0
>> 6085adam|thinkpad|~$ btrfs --version
>> btrfs-progs v5.4
>>
>> As expected, v5.4 should detect such problem without problem.
>>
>> Would you please provide extra tree dump to help us to determine what
>> makes btrfs check unable to detect such problems?
>>
>> # btrfs ins dump-tree -b 303629811712 /dev/dm-1
> anvil ~ # btrfs ins dump-tree -b 303629811712 /dev/nvme0n1p2
> btrfs-progs v5.4
> Invalid mapping for 303629811712-303629815808, got 476092633088-477166374912
> Couldn't map the block 303629811712
> Couldn't map the block 303629811712
> bad tree block 303629811712, bytenr mismatch, want=303629811712, have=0
> ERROR: failed to read tree block 303629811712
> 
The bytenr is different from your initial report.

Anyway, you can try mount with v5.4, and use the bytenr from the dmesg.

Then please provide both dmeg (including the "corrupted leaf" line), and
the `btrfs ins dump-tree -b` output.

Thanks,
Qu


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-30  6:09               ` Qu Wenruo
@ 2019-12-30  8:14                 ` Patrick Erley
  2019-12-30  8:54                   ` Qu Wenruo
  0 siblings, 1 reply; 75+ messages in thread
From: Patrick Erley @ 2019-12-30  8:14 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Btrfs BTRFS

On Mon, Dec 30, 2019 at 6:09 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2019/12/30 下午2:07, Patrick Erley wrote:
> > On Sun, Dec 29, 2019 at 9:58 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>
> >>
> >>
> >> On 2019/12/30 下午1:50, Patrick Erley wrote:
> >>> On Sun, Dec 29, 2019 at 9:47 PM Patrick Erley <pat-lkml@erley.org> wrote:
> >>>>
> >>>> On Sun, Dec 29, 2019 at 9:43 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 2019/12/30 下午1:36, Patrick Erley wrote:
> >>>>>> (ugh, just realized gmail does top replies.  Sorry... will try to
> >>>>>> figure out how to make gsuite behave like a sane mail client before my
> >>>>>> next reply):
> >>>>>>
> >>>>>> here's btrfs check /dev/nvme0n1p2 (sda3, which is a mirror of it, has
> >>>>>> exactly the same output)
> >>>>>>
> >>>>>> [1/7] checking root items
> >>>>>> [2/7] checking extents
> >>>>>> [3/7] checking free space cache
> >>>>>> [4/7] checking fs roots
> >>>>>> [5/7] checking only csums items (without verifying data)
> >>>>>> [6/7] checking root refs
> >>>>>> [7/7] checking quota groups skipped (not enabled on this FS)
> >>>>>> Opening filesystem to check...
> >>>>>> Checking filesystem on /dev/nvme0n1p2
> >>>>>> UUID: 815266d6-a8b9-4f63-a593-02fde178263f
> >>>>>> found 89383137280 bytes used, no error found
> >>>>>> total csum bytes: 85617340
> >>>>>> total tree bytes: 1670774784
> >>>>>> total fs tree bytes: 1451180032
> >>>>>> total extent tree bytes: 107905024
> >>>>>> btree space waste bytes: 413362851
> >>>>>> file data blocks allocated: 90769887232
> >>>>>>  referenced 88836960256
> >>>>>
> >>>>> It looks too good to be true, is the btrfs-progs v5.4? IIRC in v5.4 we
> >>>>> should report inodes generation problems.
> >>>>
> >>>> Hurray Bottom Reply?
> >>>>
> >>>> /usr/src/initramfs/bin $ ./btrfs.static --version
> >>>> btrfs-progs v5.4
> >>
> >> This is strange.
> >>
> >>
> >> 6084adam|thinkpad|~$ btrfs check --mode=lowmem test.img
> >> Opening filesystem to check...
> >> Checking filesystem on test.img
> >> UUID: c6c6ddd2-01c1-47fc-b699-cacfae9d4bfd
> >> [1/7] checking root items
> >> [2/7] checking extents
> >> [3/7] checking free space cache
> >> cache and super generation don't match, space cache will be invalidated
> >> [4/7] checking fs roots
> >> ERROR: invalid inode generation for ino 257, have 8858344568388091671
> >> expect [0, 9)
> >> ERROR: errors found in fs roots
> >> found 131072 bytes used, error(s) found
> >> total csum bytes: 0
> >> total tree bytes: 131072
> >> total fs tree bytes: 32768
> >> total extent tree bytes: 16384
> >> btree space waste bytes: 123409
> >> file data blocks allocated: 0
> >>  referenced 0
> >> 6085adam|thinkpad|~$ btrfs --version
> >> btrfs-progs v5.4
> >>
> >> As expected, v5.4 should detect such problem without problem.
> >>
> >> Would you please provide extra tree dump to help us to determine what
> >> makes btrfs check unable to detect such problems?
> >>
> >> # btrfs ins dump-tree -b 303629811712 /dev/dm-1
> > anvil ~ # btrfs ins dump-tree -b 303629811712 /dev/nvme0n1p2
> > btrfs-progs v5.4
> > Invalid mapping for 303629811712-303629815808, got 476092633088-477166374912
> > Couldn't map the block 303629811712
> > Couldn't map the block 303629811712
> > bad tree block 303629811712, bytenr mismatch, want=303629811712, have=0
> > ERROR: failed to read tree block 303629811712
> >
> The bytenr is different from your initial report.
>
> Anyway, you can try mount with v5.4, and use the bytenr from the dmesg.
>
> Then please provide both dmeg (including the "corrupted leaf" line), and
> the `btrfs ins dump-tree -b` output.
>
> Thanks,
> Qu
>

Good news and bad news.  Running btrfs check --repair fixed the errors
listed in $subject, but seems to have fully hosed te filesystem.  I'm
now getting:

./btrfs.static check /dev/nvme0n1p2
Opening filesystem to check...
Checking filesystem on /dev/nvme0n1p2
UUID: 815266d6-a8b9-4f63-a593-02fde178263f
[1/7] checking root items
parent transid verify failed on 499774480384 wanted 3323349 found 3323340
parent transid verify failed on 499774386176 wanted 3323349 found 3323340
parent transid verify failed on 499774394368 wanted 3323349 found 3323340
parent transid verify failed on 499774398464 wanted 3323349 found 3323340
parent transid verify failed on 499774406656 wanted 3323349 found 3323340
parent transid verify failed on 499774423040 wanted 3323349 found 3323340
parent transid verify failed on 499774427136 wanted 3323349 found 3323340
parent transid verify failed on 499774439424 wanted 3323349 found 3323340
parent transid verify failed on 499774324736 wanted 3323349 found 3323340
parent transid verify failed on 499774513152 wanted 3323349 found 3323340
parent transid verify failed on 499774341120 wanted 3323349 found 3323340
[2/7] checking extents
leaf parent key incorrect 499774554112
bad block 499774554112
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
there is no free space entry for 499922907136-499952799744
there is no free space entry for 499922907136-500822249472
cache appears valid but isn't 499748507648
[4/7] checking fs roots
Wrong key of child node/leaf, wanted: (1442047, 1, 0), have:
(523365974016, 168, 16384)
root 5 inode 1442047 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name etc filetype 2
errors 6, no dir index, no inode ref
root 5 inode 1835263 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name bin filetype 2
errors 6, no dir index, no inode ref
root 5 inode 1966335 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 4 name proc filetype 2
errors 6, no dir index, no inode ref
root 5 inode 2097407 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 5 name lib32 filetype 2
errors 6, no dir index, no inode ref
root 5 inode 2228479 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name sys filetype 2
errors 6, no dir index, no inode ref
root 5 inode 2359551 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 5 name lib64 filetype 2
errors 6, no dir index, no inode ref
root 5 inode 3145983 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name dev filetype 2
errors 6, no dir index, no inode ref
root 5 inode 3277055 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name opt filetype 2
errors 6, no dir index, no inode ref
root 5 inode 3408127 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name mnt filetype 2
errors 6, no dir index, no inode ref
root 5 inode 3539199 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name usr filetype 2
errors 6, no dir index, no inode ref
root 5 inode 3670271 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 5 name media filetype 2
errors 6, no dir index, no inode ref
root 5 inode 4587775 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 4 name sbin filetype 2
errors 6, no dir index, no inode ref
root 5 inode 4980991 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 4 name etc2 filetype 2
errors 6, no dir index, no inode ref
root 5 inode 4984678 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name run filetype 2
errors 6, no dir index, no inode ref
root 5 inode 5505279 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 7 name exports filetype 2
errors 6, no dir index, no inode ref
root 5 inode 5767423 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 5 name debug filetype 2
errors 6, no dir index, no inode ref
root 5 inode 5898495 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 4 name etc3 filetype 2
errors 6, no dir index, no inode ref
root 5 inode 6029567 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 4 name root filetype 2
errors 6, no dir index, no inode ref
root 5 inode 6431345 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 4 name boot filetype 2
errors 6, no dir index, no inode ref
root 5 inode 6553855 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name var filetype 2
errors 6, no dir index, no inode ref
root 5 inode 7602431 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name tmp filetype 2
errors 6, no dir index, no inode ref
root 5 inode 9133291 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 13 name .pulse-cookie
filetype 1 errors 6, no dir index, no inode ref
root 5 inode 9133292 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 6 name .pulse filetype 2
errors 6, no dir index, no inode ref
root 5 inode 9980485 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 9 name bootchart filetype 2
errors 6, no dir index, no inode ref
root 5 inode 10254180 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 4 name temp filetype 2
errors 6, no dir index, no inode ref
root 5 inode 11827458 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 8 name tftpboot filetype 2
errors 6, no dir index, no inode ref
root 5 inode 14435353 errors 2001, no inode item, link count wrong
    unresolved ref dir 131327 index 0 namelen 4 name s0be filetype 2
errors 6, no dir index, no inode ref
root 5 inode 14646004 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 4 name s0be filetype 2
errors 6, no dir index, no inode ref
root 5 inode 14910590 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 13 name .bash_history
filetype 1 errors 6, no dir index, no inode ref
root 5 inode 23438027 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 8 name mknod.sh filetype 1
errors 6, no dir index, no inode ref
root 5 inode 25548332 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 8 name ctrl_dir filetype 1
errors 6, no dir index, no inode ref
root 5 inode 25548333 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 5 name state filetype 1
errors 6, no dir index, no inode ref
root 5 inode 26407893 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 6 name e2fsck filetype 1
errors 6, no dir index, no inode ref
root 5 inode 36767707 errors 2001, no inode item, link count wrong
    unresolved ref dir 924630 index 0 namelen 31 name
lib_mysqludf_log-warnings.patch filetype 1 errors 6, no dir index, no
inode ref
root 5 inode 36767712 errors 2001, no inode item, link count wrong
    unresolved ref dir 924632 index 0 namelen 32 name
lib_mysqludf_stem-mysql_m4.patch filetype 1 errors 6, no dir index, no
inode ref
root 5 inode 36767838 errors 2001, no inode item, link count wrong
    unresolved ref dir 924633 index 0 namelen 33 name
mysql-udf-base64-signedness.patch filetype 1 errors 6, no dir index,
no inode ref
root 5 inode 36767839 errors 2001, no inode item, link count wrong
    unresolved ref dir 924633 index 0 namelen 20 name
mysql-udf-base64.sql filetype 1 errors 6, no dir index, no inode ref
root 5 inode 36767840 errors 2001, no inode item, link count wrong
    unresolved ref dir 924634 index 0 namelen 27 name
mysql-udf-http-stdlib.patch filetype 1 errors 6, no dir index, no
inode ref
root 5 inode 36767843 errors 2001, no inode item, link count wrong
    unresolved ref dir 924635 index 0 namelen 29 name
mysql-udf-ipv6-warnings.patch filetype 1 errors 6, no dir index, no
inode ref
root 5 inode 37030378 errors 2001, no inode item, link count wrong
    unresolved ref dir 924616 index 0 namelen 12 name metadata.xml
filetype 1 errors 6, no dir index, no inode ref
root 5 inode 37030384 errors 2001, no inode item, link count wrong
    unresolved ref dir 924619 index 0 namelen 12 name metadata.xml
filetype 1 errors 6, no dir index, no inode ref
root 5 inode 37030490 errors 2001, no inode item, link count wrong
    unresolved ref dir 924625 index 0 namelen 12 name metadata.xml
filetype 1 errors 6, no dir index, no inode ref
root 5 inode 37030492 errors 2001, no inode item, link count wrong
    unresolved ref dir 924626 index 0 namelen 12 name metadata.xml
filetype 1 errors 6, no dir index, no inode ref
root 5 inode 37030497 errors 2001, no inode item, link count wrong
    unresolved ref dir 924628 index 0 namelen 12 name metadata.xml
filetype 1 errors 6, no dir index, no inode ref
root 5 inode 39198776 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 8 name bootback filetype 2
errors 6, no dir index, no inode ref
root 5 inode 51520157 errors 2001, no inode item, link count wrong
    unresolved ref dir 924616 index 0 namelen 29 name
lib_mysqludf_log-0.0.2.ebuild filetype 1 errors 6, no dir index, no
inode ref
root 5 inode 51520163 errors 2001, no inode item, link count wrong
    unresolved ref dir 924619 index 0 namelen 30 name
lib_mysqludf_stem-0.9.1.ebuild filetype 1 errors 6, no dir index, no
inode ref
root 5 inode 51520278 errors 2001, no inode item, link count wrong
    unresolved ref dir 924625 index 0 namelen 32 name
mysql-udf-base64-20010618.ebuild filetype 1 errors 6, no dir index, no
inode ref
root 5 inode 51520280 errors 2001, no inode item, link count wrong
    unresolved ref dir 924626 index 0 namelen 25 name
mysql-udf-http-1.0.ebuild filetype 1 errors 6, no dir index, no inode
ref
root 5 inode 51520284 errors 2001, no inode item, link count wrong
    unresolved ref dir 924628 index 0 namelen 25 name
mysql-udf-ipv6-0.4.ebuild filetype 1 errors 6, no dir index, no inode
ref
root 5 inode 58792107 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name foo filetype 1
errors 6, no dir index, no inode ref
root 5 inode 63681251 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 4 name xpra filetype 2
errors 6, no dir index, no inode ref
root 5 inode 66715793 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 8 name .viminfo filetype 1
errors 6, no dir index, no inode ref
root 5 inode 71469207 errors 2001, no inode item, link count wrong
    unresolved ref dir 924616 index 0 namelen 8 name Manifest filetype
1 errors 6, no dir index, no inode ref
root 5 inode 71469208 errors 2001, no inode item, link count wrong
    unresolved ref dir 924619 index 0 namelen 8 name Manifest filetype
1 errors 6, no dir index, no inode ref
root 5 inode 71469238 errors 2001, no inode item, link count wrong
    unresolved ref dir 924625 index 0 namelen 8 name Manifest filetype
1 errors 6, no dir index, no inode ref
root 5 inode 71469239 errors 2001, no inode item, link count wrong
    unresolved ref dir 924626 index 0 namelen 8 name Manifest filetype
1 errors 6, no dir index, no inode ref
root 5 inode 71469240 errors 2001, no inode item, link count wrong
    unresolved ref dir 924628 index 0 namelen 8 name Manifest filetype
1 errors 6, no dir index, no inode ref
root 5 inode 75334150 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 7 name lib.new filetype 2
errors 6, no dir index, no inode ref
root 5 inode 77351088 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name lib filetype 7
errors 6, no dir index, no inode ref
ERROR: errors found in fs roots
found 951992320 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 3895296
total fs tree bytes: 0
total extent tree bytes: 3813376
btree space waste bytes: 1560917
file data blocks allocated: 27983872
 referenced 27983872

Should I also paste in the repair log?

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-30  8:14                 ` Patrick Erley
@ 2019-12-30  8:54                   ` Qu Wenruo
  2019-12-30  9:01                     ` Patrick Erley
  0 siblings, 1 reply; 75+ messages in thread
From: Qu Wenruo @ 2019-12-30  8:54 UTC (permalink / raw)
  To: Patrick Erley; +Cc: Btrfs BTRFS


[-- Attachment #1.1: Type: text/plain, Size: 17174 bytes --]



On 2019/12/30 下午4:14, Patrick Erley wrote:
> On Mon, Dec 30, 2019 at 6:09 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2019/12/30 下午2:07, Patrick Erley wrote:
>>> On Sun, Dec 29, 2019 at 9:58 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>
>>>>
>>>>
>>>> On 2019/12/30 下午1:50, Patrick Erley wrote:
>>>>> On Sun, Dec 29, 2019 at 9:47 PM Patrick Erley <pat-lkml@erley.org> wrote:
>>>>>>
>>>>>> On Sun, Dec 29, 2019 at 9:43 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 2019/12/30 下午1:36, Patrick Erley wrote:
>>>>>>>> (ugh, just realized gmail does top replies.  Sorry... will try to
>>>>>>>> figure out how to make gsuite behave like a sane mail client before my
>>>>>>>> next reply):
>>>>>>>>
>>>>>>>> here's btrfs check /dev/nvme0n1p2 (sda3, which is a mirror of it, has
>>>>>>>> exactly the same output)
>>>>>>>>
>>>>>>>> [1/7] checking root items
>>>>>>>> [2/7] checking extents
>>>>>>>> [3/7] checking free space cache
>>>>>>>> [4/7] checking fs roots
>>>>>>>> [5/7] checking only csums items (without verifying data)
>>>>>>>> [6/7] checking root refs
>>>>>>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>>>>>>> Opening filesystem to check...
>>>>>>>> Checking filesystem on /dev/nvme0n1p2
>>>>>>>> UUID: 815266d6-a8b9-4f63-a593-02fde178263f
>>>>>>>> found 89383137280 bytes used, no error found
>>>>>>>> total csum bytes: 85617340
>>>>>>>> total tree bytes: 1670774784
>>>>>>>> total fs tree bytes: 1451180032
>>>>>>>> total extent tree bytes: 107905024
>>>>>>>> btree space waste bytes: 413362851
>>>>>>>> file data blocks allocated: 90769887232
>>>>>>>>  referenced 88836960256
>>>>>>>
>>>>>>> It looks too good to be true, is the btrfs-progs v5.4? IIRC in v5.4 we
>>>>>>> should report inodes generation problems.
>>>>>>
>>>>>> Hurray Bottom Reply?
>>>>>>
>>>>>> /usr/src/initramfs/bin $ ./btrfs.static --version
>>>>>> btrfs-progs v5.4
>>>>
>>>> This is strange.
>>>>
>>>>
>>>> 6084adam|thinkpad|~$ btrfs check --mode=lowmem test.img
>>>> Opening filesystem to check...
>>>> Checking filesystem on test.img
>>>> UUID: c6c6ddd2-01c1-47fc-b699-cacfae9d4bfd
>>>> [1/7] checking root items
>>>> [2/7] checking extents
>>>> [3/7] checking free space cache
>>>> cache and super generation don't match, space cache will be invalidated
>>>> [4/7] checking fs roots
>>>> ERROR: invalid inode generation for ino 257, have 8858344568388091671
>>>> expect [0, 9)
>>>> ERROR: errors found in fs roots
>>>> found 131072 bytes used, error(s) found
>>>> total csum bytes: 0
>>>> total tree bytes: 131072
>>>> total fs tree bytes: 32768
>>>> total extent tree bytes: 16384
>>>> btree space waste bytes: 123409
>>>> file data blocks allocated: 0
>>>>  referenced 0
>>>> 6085adam|thinkpad|~$ btrfs --version
>>>> btrfs-progs v5.4
>>>>
>>>> As expected, v5.4 should detect such problem without problem.
>>>>
>>>> Would you please provide extra tree dump to help us to determine what
>>>> makes btrfs check unable to detect such problems?
>>>>
>>>> # btrfs ins dump-tree -b 303629811712 /dev/dm-1
>>> anvil ~ # btrfs ins dump-tree -b 303629811712 /dev/nvme0n1p2
>>> btrfs-progs v5.4
>>> Invalid mapping for 303629811712-303629815808, got 476092633088-477166374912
>>> Couldn't map the block 303629811712
>>> Couldn't map the block 303629811712
>>> bad tree block 303629811712, bytenr mismatch, want=303629811712, have=0
>>> ERROR: failed to read tree block 303629811712
>>>
>> The bytenr is different from your initial report.
>>
>> Anyway, you can try mount with v5.4, and use the bytenr from the dmesg.
>>
>> Then please provide both dmeg (including the "corrupted leaf" line), and
>> the `btrfs ins dump-tree -b` output.
>>
>> Thanks,
>> Qu
>>
> 
> Good news and bad news.  Running btrfs check --repair fixed the errors
> listed in $subject, but seems to have fully hosed te filesystem.  I'm
> now getting:
> 
> ./btrfs.static check /dev/nvme0n1p2
> Opening filesystem to check...
> Checking filesystem on /dev/nvme0n1p2
> UUID: 815266d6-a8b9-4f63-a593-02fde178263f
> [1/7] checking root items
> parent transid verify failed on 499774480384 wanted 3323349 found 3323340
> parent transid verify failed on 499774386176 wanted 3323349 found 3323340
> parent transid verify failed on 499774394368 wanted 3323349 found 3323340
> parent transid verify failed on 499774398464 wanted 3323349 found 3323340
> parent transid verify failed on 499774406656 wanted 3323349 found 3323340
> parent transid verify failed on 499774423040 wanted 3323349 found 3323340
> parent transid verify failed on 499774427136 wanted 3323349 found 3323340
> parent transid verify failed on 499774439424 wanted 3323349 found 3323340
> parent transid verify failed on 499774324736 wanted 3323349 found 3323340
> parent transid verify failed on 499774513152 wanted 3323349 found 3323340
> parent transid verify failed on 499774341120 wanted 3323349 found 3323340
> [2/7] checking extents
> leaf parent key incorrect 499774554112
> bad block 499774554112
> ERROR: errors found in extent allocation tree or chunk allocation
> [3/7] checking free space cache
> there is no free space entry for 499922907136-499952799744
> there is no free space entry for 499922907136-500822249472
> cache appears valid but isn't 499748507648
> [4/7] checking fs roots
> Wrong key of child node/leaf, wanted: (1442047, 1, 0), have:
> (523365974016, 168, 16384)
> root 5 inode 1442047 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name etc filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 1835263 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name bin filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 1966335 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 4 name proc filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 2097407 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 5 name lib32 filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 2228479 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name sys filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 2359551 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 5 name lib64 filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 3145983 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name dev filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 3277055 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name opt filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 3408127 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name mnt filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 3539199 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name usr filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 3670271 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 5 name media filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 4587775 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 4 name sbin filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 4980991 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 4 name etc2 filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 4984678 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name run filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 5505279 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 7 name exports filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 5767423 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 5 name debug filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 5898495 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 4 name etc3 filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 6029567 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 4 name root filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 6431345 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 4 name boot filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 6553855 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name var filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 7602431 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name tmp filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 9133291 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 13 name .pulse-cookie
> filetype 1 errors 6, no dir index, no inode ref
> root 5 inode 9133292 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 6 name .pulse filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 9980485 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 9 name bootchart filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 10254180 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 4 name temp filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 11827458 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 8 name tftpboot filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 14435353 errors 2001, no inode item, link count wrong
>     unresolved ref dir 131327 index 0 namelen 4 name s0be filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 14646004 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 4 name s0be filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 14910590 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 13 name .bash_history
> filetype 1 errors 6, no dir index, no inode ref
> root 5 inode 23438027 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 8 name mknod.sh filetype 1
> errors 6, no dir index, no inode ref
> root 5 inode 25548332 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 8 name ctrl_dir filetype 1
> errors 6, no dir index, no inode ref
> root 5 inode 25548333 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 5 name state filetype 1
> errors 6, no dir index, no inode ref
> root 5 inode 26407893 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 6 name e2fsck filetype 1
> errors 6, no dir index, no inode ref
> root 5 inode 36767707 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924630 index 0 namelen 31 name
> lib_mysqludf_log-warnings.patch filetype 1 errors 6, no dir index, no
> inode ref
> root 5 inode 36767712 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924632 index 0 namelen 32 name
> lib_mysqludf_stem-mysql_m4.patch filetype 1 errors 6, no dir index, no
> inode ref
> root 5 inode 36767838 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924633 index 0 namelen 33 name
> mysql-udf-base64-signedness.patch filetype 1 errors 6, no dir index,
> no inode ref
> root 5 inode 36767839 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924633 index 0 namelen 20 name
> mysql-udf-base64.sql filetype 1 errors 6, no dir index, no inode ref
> root 5 inode 36767840 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924634 index 0 namelen 27 name
> mysql-udf-http-stdlib.patch filetype 1 errors 6, no dir index, no
> inode ref
> root 5 inode 36767843 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924635 index 0 namelen 29 name
> mysql-udf-ipv6-warnings.patch filetype 1 errors 6, no dir index, no
> inode ref
> root 5 inode 37030378 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924616 index 0 namelen 12 name metadata.xml
> filetype 1 errors 6, no dir index, no inode ref
> root 5 inode 37030384 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924619 index 0 namelen 12 name metadata.xml
> filetype 1 errors 6, no dir index, no inode ref
> root 5 inode 37030490 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924625 index 0 namelen 12 name metadata.xml
> filetype 1 errors 6, no dir index, no inode ref
> root 5 inode 37030492 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924626 index 0 namelen 12 name metadata.xml
> filetype 1 errors 6, no dir index, no inode ref
> root 5 inode 37030497 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924628 index 0 namelen 12 name metadata.xml
> filetype 1 errors 6, no dir index, no inode ref
> root 5 inode 39198776 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 8 name bootback filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 51520157 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924616 index 0 namelen 29 name
> lib_mysqludf_log-0.0.2.ebuild filetype 1 errors 6, no dir index, no
> inode ref
> root 5 inode 51520163 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924619 index 0 namelen 30 name
> lib_mysqludf_stem-0.9.1.ebuild filetype 1 errors 6, no dir index, no
> inode ref
> root 5 inode 51520278 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924625 index 0 namelen 32 name
> mysql-udf-base64-20010618.ebuild filetype 1 errors 6, no dir index, no
> inode ref
> root 5 inode 51520280 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924626 index 0 namelen 25 name
> mysql-udf-http-1.0.ebuild filetype 1 errors 6, no dir index, no inode
> ref
> root 5 inode 51520284 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924628 index 0 namelen 25 name
> mysql-udf-ipv6-0.4.ebuild filetype 1 errors 6, no dir index, no inode
> ref
> root 5 inode 58792107 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name foo filetype 1
> errors 6, no dir index, no inode ref
> root 5 inode 63681251 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 4 name xpra filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 66715793 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 8 name .viminfo filetype 1
> errors 6, no dir index, no inode ref
> root 5 inode 71469207 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924616 index 0 namelen 8 name Manifest filetype
> 1 errors 6, no dir index, no inode ref
> root 5 inode 71469208 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924619 index 0 namelen 8 name Manifest filetype
> 1 errors 6, no dir index, no inode ref
> root 5 inode 71469238 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924625 index 0 namelen 8 name Manifest filetype
> 1 errors 6, no dir index, no inode ref
> root 5 inode 71469239 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924626 index 0 namelen 8 name Manifest filetype
> 1 errors 6, no dir index, no inode ref
> root 5 inode 71469240 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924628 index 0 namelen 8 name Manifest filetype
> 1 errors 6, no dir index, no inode ref
> root 5 inode 75334150 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 7 name lib.new filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 77351088 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name lib filetype 7
> errors 6, no dir index, no inode ref
> ERROR: errors found in fs roots
> found 951992320 bytes used, error(s) found
> total csum bytes: 0
> total tree bytes: 3895296
> total fs tree bytes: 0
> total extent tree bytes: 3813376
> btree space waste bytes: 1560917
> file data blocks allocated: 27983872
>  referenced 27983872
> 
> Should I also paste in the repair log?
> 
Yes please.

This sounds very strange, especially for the transid mismatch part.

Thanks,
Qu


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-30  8:54                   ` Qu Wenruo
@ 2019-12-30  9:01                     ` Patrick Erley
  2019-12-30  9:09                       ` Qu Wenruo
  0 siblings, 1 reply; 75+ messages in thread
From: Patrick Erley @ 2019-12-30  9:01 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Btrfs BTRFS

On Mon, Dec 30, 2019 at 8:54 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> On 2019/12/30 下午4:14, Patrick Erley wrote:
> > On Mon, Dec 30, 2019 at 6:09 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >
> > Should I also paste in the repair log?
> >
> Yes please.
>
> This sounds very strange, especially for the transid mismatch part.
>
> Thanks,
> Qu
>



enabling repair mode
WARNING:

    Do not use --repair unless you are advised to do so by a developer
    or an experienced user, and then only after having accepted that no
    fsck can successfully repair all types of filesystem corruption. Eg.
    some software or hardware bugs can fatally damage a volume.
    The operation will start in 10 seconds.
    Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1parent transid verify failed on 499774464000
wanted 3323349 found 3323340
parent transid verify failed on 499774521344 wanted 3323349 found 3323340
parent transid verify failed on 499774529536 wanted 3323349 found 3323340
[1/7] checking root items
parent transid verify failed on 499774373888 wanted 3323349 found 3323340
parent transid verify failed on 499774480384 wanted 3323349 found 3323340
parent transid verify failed on 499774386176 wanted 3323349 found 3323340
parent transid verify failed on 499774390272 wanted 3323349 found 3323340
parent transid verify failed on 499774394368 wanted 3323349 found 3323340
parent transid verify failed on 499774398464 wanted 3323349 found 3323340
parent transid verify failed on 499774402560 wanted 3323349 found 3323340
parent transid verify failed on 499774406656 wanted 3323349 found 3323340
parent transid verify failed on 499774423040 wanted 3323349 found 3323340
parent transid verify failed on 499774427136 wanted 3323349 found 3323340
parent transid verify failed on 499774435328 wanted 3323349 found 3323340
parent transid verify failed on 499774439424 wanted 3323349 found 3323340
parent transid verify failed on 499774320640 wanted 3323349 found 3323340
parent transid verify failed on 499774324736 wanted 3323349 found 3323340
parent transid verify failed on 499774443520 wanted 3323349 found 3323340
parent transid verify failed on 499774447616 wanted 3323349 found 3323340
parent transid verify failed on 499774504960 wanted 3323349 found 3323340
parent transid verify failed on 499774513152 wanted 3323349 found 3323340
parent transid verify failed on 499774468096 wanted 3323349 found 3323340
parent transid verify failed on 499774337024 wanted 3323349 found 3323340
parent transid verify failed on 499774341120 wanted 3323349 found 3323340
parent transid verify failed on 499774517248 wanted 3323349 found 3323340
parent transid verify failed on 499774558208 wanted 3323349 found 3323340
Fixed 0 roots.
[2/7] checking extents
extent back ref already exists for 499952799744 parent 0 root 2
extent back ref already exists for 499952803840 parent 0 root 2
extent back ref already exists for 499952807936 parent 0 root 2
extent back ref already exists for 499952812032 parent 0 root 2
extent back ref already exists for 499952816128 parent 0 root 2
extent back ref already exists for 499952824320 parent 0 root 5
extent back ref already exists for 499952832512 parent 0 root 2
extent back ref already exists for 499952852992 parent 0 root 2
extent back ref already exists for 499952857088 parent 0 root 2
extent back ref already exists for 499952861184 parent 0 root 2
extent back ref already exists for 499952877568 parent 0 root 2
extent back ref already exists for 499952885760 parent 0 root 2
extent back ref already exists for 499952889856 parent 0 root 2
extent back ref already exists for 499952898048 parent 0 root 2
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499774554112
leaf parent key incorrect 499773358080
bad block 499773358080
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
parent transid verify failed on 499774283776 wanted 3323349 found 3323340
Wrong key of child node/leaf, wanted: (1442047, 1, 0), have:
(523365974016, 168, 16384)
ERROR: child eb corrupted: parent bytenr=499750891520 item=2 parent
level=2 child level=0
root 5 inode 1442047 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name etc filetype 2
errors 6, no dir index, no inode ref
root 5 inode 1835263 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name bin filetype 2
errors 6, no dir index, no inode ref
root 5 inode 1966335 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 4 name proc filetype 2
errors 6, no dir index, no inode ref
root 5 inode 2097407 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 5 name lib32 filetype 2
errors 6, no dir index, no inode ref
root 5 inode 2228479 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name sys filetype 2
errors 6, no dir index, no inode ref
root 5 inode 2359551 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 5 name lib64 filetype 2
errors 6, no dir index, no inode ref
root 5 inode 3145983 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name dev filetype 2
errors 6, no dir index, no inode ref
root 5 inode 3277055 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name opt filetype 2
errors 6, no dir index, no inode ref
root 5 inode 3408127 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name mnt filetype 2
errors 6, no dir index, no inode ref
root 5 inode 3539199 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name usr filetype 2
errors 6, no dir index, no inode ref
root 5 inode 3670271 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 5 name media filetype 2
errors 6, no dir index, no inode ref
root 5 inode 4587775 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 4 name sbin filetype 2
errors 6, no dir index, no inode ref
root 5 inode 4980991 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 4 name etc2 filetype 2
errors 6, no dir index, no inode ref
root 5 inode 4984678 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name run filetype 2
errors 6, no dir index, no inode ref
root 5 inode 5505279 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 7 name exports filetype 2
errors 6, no dir index, no inode ref
root 5 inode 5767423 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 5 name debug filetype 2
errors 6, no dir index, no inode ref
root 5 inode 5898495 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 4 name etc3 filetype 2
errors 6, no dir index, no inode ref
root 5 inode 6029567 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 4 name root filetype 2
errors 6, no dir index, no inode ref
root 5 inode 6431345 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 4 name boot filetype 2
errors 6, no dir index, no inode ref
ERROR: child eb corrupted: parent bytenr=499762622464 item=37 parent
level=2 child level=0
root 5 inode 6553855 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name var filetype 2
errors 6, no dir index, no inode ref
root 5 inode 7602431 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name tmp filetype 2
errors 6, no dir index, no inode ref
ERROR: child eb corrupted: parent bytenr=499752206336 item=14 parent
level=3 child level=0
root 5 inode 9133291 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 13 name .pulse-cookie
filetype 1 errors 6, no dir index, no inode ref
ERROR: child eb corrupted: parent bytenr=499748732928 item=14 parent
level=3 child level=0
root 5 inode 9133292 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 6 name .pulse filetype 2
errors 6, no dir index, no inode ref
ERROR: child eb corrupted: parent bytenr=499752206336 item=14 parent
level=3 child level=0
root 5 inode 9980485 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 9 name bootchart filetype 2
errors 6, no dir index, no inode ref
root 5 inode 10254180 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 4 name temp filetype 2
errors 6, no dir index, no inode ref
root 5 inode 11827458 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 8 name tftpboot filetype 2
errors 6, no dir index, no inode ref
ERROR: child eb corrupted: parent bytenr=499748732928 item=17 parent
level=3 child level=0
root 5 inode 14435353 errors 2001, no inode item, link count wrong
    unresolved ref dir 131327 index 0 namelen 4 name s0be filetype 2
errors 6, no dir index, no inode ref
ERROR: child eb corrupted: parent bytenr=499752206336 item=17 parent
level=3 child level=0
root 5 inode 14646004 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 4 name s0be filetype 2
errors 6, no dir index, no inode ref
ERROR: child eb corrupted: parent bytenr=499748732928 item=17 parent
level=3 child level=0
root 5 inode 14910590 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 13 name .bash_history
filetype 1 errors 6, no dir index, no inode ref
root 5 inode 23438027 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 8 name mknod.sh filetype 1
errors 6, no dir index, no inode ref
root 5 inode 25548332 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 8 name ctrl_dir filetype 1
errors 6, no dir index, no inode ref
root 5 inode 25548333 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 5 name state filetype 1
errors 6, no dir index, no inode ref
root 5 inode 26407893 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 6 name e2fsck filetype 1
errors 6, no dir index, no inode ref
root 5 inode 36767707 errors 2001, no inode item, link count wrong
    unresolved ref dir 924630 index 0 namelen 31 name
lib_mysqludf_log-warnings.patch filetype 1 errors 6, no dir index, no
inode ref
root 5 inode 36767712 errors 2001, no inode item, link count wrong
    unresolved ref dir 924632 index 0 namelen 32 name
lib_mysqludf_stem-mysql_m4.patch filetype 1 errors 6, no dir index, no
inode ref
root 5 inode 36767838 errors 2001, no inode item, link count wrong
    unresolved ref dir 924633 index 0 namelen 33 name
mysql-udf-base64-signedness.patch filetype 1 errors 6, no dir index,
no inode ref
root 5 inode 36767839 errors 2001, no inode item, link count wrong
    unresolved ref dir 924633 index 0 namelen 20 name
mysql-udf-base64.sql filetype 1 errors 6, no dir index, no inode ref
root 5 inode 36767840 errors 2001, no inode item, link count wrong
    unresolved ref dir 924634 index 0 namelen 27 name
mysql-udf-http-stdlib.patch filetype 1 errors 6, no dir index, no
inode ref
root 5 inode 36767843 errors 2001, no inode item, link count wrong
    unresolved ref dir 924635 index 0 namelen 29 name
mysql-udf-ipv6-warnings.patch filetype 1 errors 6, no dir index, no
inode ref
root 5 inode 37030378 errors 2001, no inode item, link count wrong
    unresolved ref dir 924616 index 0 namelen 12 name metadata.xml
filetype 1 errors 6, no dir index, no inode ref
root 5 inode 37030384 errors 2001, no inode item, link count wrong
    unresolved ref dir 924619 index 0 namelen 12 name metadata.xml
filetype 1 errors 6, no dir index, no inode ref
root 5 inode 37030490 errors 2001, no inode item, link count wrong
    unresolved ref dir 924625 index 0 namelen 12 name metadata.xml
filetype 1 errors 6, no dir index, no inode ref
root 5 inode 37030492 errors 2001, no inode item, link count wrong
    unresolved ref dir 924626 index 0 namelen 12 name metadata.xml
filetype 1 errors 6, no dir index, no inode ref
root 5 inode 37030497 errors 2001, no inode item, link count wrong
    unresolved ref dir 924628 index 0 namelen 12 name metadata.xml
filetype 1 errors 6, no dir index, no inode ref
root 5 inode 39198776 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 8 name bootback filetype 2
errors 6, no dir index, no inode ref
root 5 inode 51520157 errors 2001, no inode item, link count wrong
    unresolved ref dir 924616 index 0 namelen 29 name
lib_mysqludf_log-0.0.2.ebuild filetype 1 errors 6, no dir index, no
inode ref
root 5 inode 51520163 errors 2001, no inode item, link count wrong
    unresolved ref dir 924619 index 0 namelen 30 name
lib_mysqludf_stem-0.9.1.ebuild filetype 1 errors 6, no dir index, no
inode ref
root 5 inode 51520278 errors 2001, no inode item, link count wrong
    unresolved ref dir 924625 index 0 namelen 32 name
mysql-udf-base64-20010618.ebuild filetype 1 errors 6, no dir index, no
inode ref
root 5 inode 51520280 errors 2001, no inode item, link count wrong
    unresolved ref dir 924626 index 0 namelen 25 name
mysql-udf-http-1.0.ebuild filetype 1 errors 6, no dir index, no inode
ref
root 5 inode 51520284 errors 2001, no inode item, link count wrong
    unresolved ref dir 924628 index 0 namelen 25 name
mysql-udf-ipv6-0.4.ebuild filetype 1 errors 6, no dir index, no inode
ref
root 5 inode 58792107 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name foo filetype 1
errors 6, no dir index, no inode ref
root 5 inode 63681251 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 4 name xpra filetype 2
errors 6, no dir index, no inode ref
root 5 inode 66715793 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 8 name .viminfo filetype 1
errors 6, no dir index, no inode ref
root 5 inode 71469207 errors 2001, no inode item, link count wrong
    unresolved ref dir 924616 index 0 namelen 8 name Manifest filetype
1 errors 6, no dir index, no inode ref
root 5 inode 71469208 errors 2001, no inode item, link count wrong
    unresolved ref dir 924619 index 0 namelen 8 name Manifest filetype
1 errors 6, no dir index, no inode ref
root 5 inode 71469238 errors 2001, no inode item, link count wrong
    unresolved ref dir 924625 index 0 namelen 8 name Manifest filetype
1 errors 6, no dir index, no inode ref
root 5 inode 71469239 errors 2001, no inode item, link count wrong
    unresolved ref dir 924626 index 0 namelen 8 name Manifest filetype
1 errors 6, no dir index, no inode ref
root 5 inode 71469240 errors 2001, no inode item, link count wrong
    unresolved ref dir 924628 index 0 namelen 8 name Manifest filetype
1 errors 6, no dir index, no inode ref
ERROR: child eb corrupted: parent bytenr=499748732928 item=65 parent
level=3 child level=1
root 5 inode 75334150 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 7 name lib.new filetype 2
errors 6, no dir index, no inode ref
parent transid verify failed on 499774287872 wanted 3323349 found 3323340
parent transid verify failed on 499774291968 wanted 3323349 found 3323340
parent transid verify failed on 499774296064 wanted 3323349 found 3323340
root 5 inode 77351088 errors 2001, no inode item, link count wrong
    unresolved ref dir 256 index 0 namelen 3 name lib filetype 7
errors 6, no dir index, no inode ref
ERROR: errors found in fs roots

Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/sda3
UUID: 815266d6-a8b9-4f63-a593-02fde178263f
cache and super generation don't match, space cache will be invalidated
reset isize for dir 256 root 5
reset isize for dir 131327 root 5
reset isize for dir 924616 root 5
Moving file 'lib_mysqludf_log' to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 924616
reset isize for dir 924619 root 5
Moving file 'lib_mysqludf_stem' to 'lost+found' dir since it has no
valid backref
Fixed the nlink of inode 924619
reset isize for dir 924625 root 5
Moving file 'mysql-udf-base64' to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 924625
reset isize for dir 924626 root 5
Moving file 'mysql-udf-http' to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 924626
reset isize for dir 924628 root 5
Moving file 'mysql-udf-ipv6' to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 924628
reset isize for dir 924630 root 5
reset isize for dir 924632 root 5
reset isize for dir 924633 root 5
reset isize for dir 924634 root 5
reset isize for dir 924635 root 5
Trying to rebuild inode:1442047
Trying to rebuild inode:1835263
Trying to rebuild inode:1966335
Trying to rebuild inode:2097407
Trying to rebuild inode:2228479
Trying to rebuild inode:2359551
Trying to rebuild inode:3145983
Trying to rebuild inode:3277055
Trying to rebuild inode:3408127
Trying to rebuild inode:3539199
Trying to rebuild inode:3670271
Trying to rebuild inode:4587775
Trying to rebuild inode:4980991
Trying to rebuild inode:4984678
Trying to rebuild inode:5505279
Trying to rebuild inode:5767423
Trying to rebuild inode:5898495
Trying to rebuild inode:6029567
Trying to rebuild inode:6431345
Trying to rebuild inode:6553855
Trying to rebuild inode:7602431
Trying to rebuild inode:9133291
Trying to rebuild inode:9133292
Trying to rebuild inode:9980485
Trying to rebuild inode:10254180
Trying to rebuild inode:11827458
Trying to rebuild inode:14435353
Trying to rebuild inode:14646004
Trying to rebuild inode:14910590
Trying to rebuild inode:23438027
Trying to rebuild inode:25548332
Trying to rebuild inode:25548333
Trying to rebuild inode:26407893
Trying to rebuild inode:36767707
Trying to rebuild inode:36767712
Trying to rebuild inode:36767838
Trying to rebuild inode:36767839
Trying to rebuild inode:36767840
Trying to rebuild inode:36767843
Trying to rebuild inode:37030378
Trying to rebuild inode:37030384
Trying to rebuild inode:37030490
Trying to rebuild inode:37030492
Trying to rebuild inode:37030497
Trying to rebuild inode:39198776
Trying to rebuild inode:51520157
Trying to rebuild inode:51520163
Trying to rebuild inode:51520278
Trying to rebuild inode:51520280
Trying to rebuild inode:51520284
Trying to rebuild inode:58792107
Trying to rebuild inode:63681251
Trying to rebuild inode:66715793
Trying to rebuild inode:71469207
Trying to rebuild inode:71469208
Trying to rebuild inode:71469238
Trying to rebuild inode:71469239
Trying to rebuild inode:71469240
Trying to rebuild inode:75334150
Trying to rebuild inode:77351088
found 89594781696 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 108093440
total fs tree bytes: 53248
total extent tree bytes: 107958272
btree space waste bytes: 36265970
file data blocks allocated: 27983872
 referenced 27983872

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-30  9:01                     ` Patrick Erley
@ 2019-12-30  9:09                       ` Qu Wenruo
  2019-12-30  9:21                         ` Patrick Erley
  0 siblings, 1 reply; 75+ messages in thread
From: Qu Wenruo @ 2019-12-30  9:09 UTC (permalink / raw)
  To: Patrick Erley; +Cc: Btrfs BTRFS


[-- Attachment #1.1: Type: text/plain, Size: 21506 bytes --]



On 2019/12/30 下午5:01, Patrick Erley wrote:
> On Mon, Dec 30, 2019 at 8:54 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>> On 2019/12/30 下午4:14, Patrick Erley wrote:
>>> On Mon, Dec 30, 2019 at 6:09 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>
>>> Should I also paste in the repair log?
>>>
>> Yes please.
>>
>> This sounds very strange, especially for the transid mismatch part.
>>
>> Thanks,
>> Qu
>>
> 
> 
> 
> enabling repair mode
> WARNING:
> 
>     Do not use --repair unless you are advised to do so by a developer
>     or an experienced user, and then only after having accepted that no
>     fsck can successfully repair all types of filesystem corruption. Eg.
>     some software or hardware bugs can fatally damage a volume.
>     The operation will start in 10 seconds.
>     Use Ctrl-C to stop it.
> 10 9 8 7 6 5 4 3 2 1parent transid verify failed on 499774464000
> wanted 3323349 found 3323340
> parent transid verify failed on 499774521344 wanted 3323349 found 3323340
> parent transid verify failed on 499774529536 wanted 3323349 found 3323340

This message is from open_ctree(), which means the fs is already corrupted.

Would you like to provide the history between last good btrfs check run
and --repair run?

Thanks,
Qu

> [1/7] checking root items
> parent transid verify failed on 499774373888 wanted 3323349 found 3323340
> parent transid verify failed on 499774480384 wanted 3323349 found 3323340
> parent transid verify failed on 499774386176 wanted 3323349 found 3323340
> parent transid verify failed on 499774390272 wanted 3323349 found 3323340
> parent transid verify failed on 499774394368 wanted 3323349 found 3323340
> parent transid verify failed on 499774398464 wanted 3323349 found 3323340
> parent transid verify failed on 499774402560 wanted 3323349 found 3323340
> parent transid verify failed on 499774406656 wanted 3323349 found 3323340
> parent transid verify failed on 499774423040 wanted 3323349 found 3323340
> parent transid verify failed on 499774427136 wanted 3323349 found 3323340
> parent transid verify failed on 499774435328 wanted 3323349 found 3323340
> parent transid verify failed on 499774439424 wanted 3323349 found 3323340
> parent transid verify failed on 499774320640 wanted 3323349 found 3323340
> parent transid verify failed on 499774324736 wanted 3323349 found 3323340
> parent transid verify failed on 499774443520 wanted 3323349 found 3323340
> parent transid verify failed on 499774447616 wanted 3323349 found 3323340
> parent transid verify failed on 499774504960 wanted 3323349 found 3323340
> parent transid verify failed on 499774513152 wanted 3323349 found 3323340
> parent transid verify failed on 499774468096 wanted 3323349 found 3323340
> parent transid verify failed on 499774337024 wanted 3323349 found 3323340
> parent transid verify failed on 499774341120 wanted 3323349 found 3323340
> parent transid verify failed on 499774517248 wanted 3323349 found 3323340
> parent transid verify failed on 499774558208 wanted 3323349 found 3323340
> Fixed 0 roots.
> [2/7] checking extents
> extent back ref already exists for 499952799744 parent 0 root 2
> extent back ref already exists for 499952803840 parent 0 root 2
> extent back ref already exists for 499952807936 parent 0 root 2
> extent back ref already exists for 499952812032 parent 0 root 2
> extent back ref already exists for 499952816128 parent 0 root 2
> extent back ref already exists for 499952824320 parent 0 root 5
> extent back ref already exists for 499952832512 parent 0 root 2
> extent back ref already exists for 499952852992 parent 0 root 2
> extent back ref already exists for 499952857088 parent 0 root 2
> extent back ref already exists for 499952861184 parent 0 root 2
> extent back ref already exists for 499952877568 parent 0 root 2
> extent back ref already exists for 499952885760 parent 0 root 2
> extent back ref already exists for 499952889856 parent 0 root 2
> extent back ref already exists for 499952898048 parent 0 root 2
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499774554112
> leaf parent key incorrect 499773358080
> bad block 499773358080
> ERROR: errors found in extent allocation tree or chunk allocation
> [3/7] checking free space cache
> [4/7] checking fs roots
> parent transid verify failed on 499774283776 wanted 3323349 found 3323340
> Wrong key of child node/leaf, wanted: (1442047, 1, 0), have:
> (523365974016, 168, 16384)
> ERROR: child eb corrupted: parent bytenr=499750891520 item=2 parent
> level=2 child level=0
> root 5 inode 1442047 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name etc filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 1835263 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name bin filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 1966335 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 4 name proc filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 2097407 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 5 name lib32 filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 2228479 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name sys filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 2359551 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 5 name lib64 filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 3145983 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name dev filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 3277055 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name opt filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 3408127 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name mnt filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 3539199 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name usr filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 3670271 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 5 name media filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 4587775 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 4 name sbin filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 4980991 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 4 name etc2 filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 4984678 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name run filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 5505279 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 7 name exports filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 5767423 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 5 name debug filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 5898495 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 4 name etc3 filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 6029567 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 4 name root filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 6431345 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 4 name boot filetype 2
> errors 6, no dir index, no inode ref
> ERROR: child eb corrupted: parent bytenr=499762622464 item=37 parent
> level=2 child level=0
> root 5 inode 6553855 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name var filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 7602431 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name tmp filetype 2
> errors 6, no dir index, no inode ref
> ERROR: child eb corrupted: parent bytenr=499752206336 item=14 parent
> level=3 child level=0
> root 5 inode 9133291 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 13 name .pulse-cookie
> filetype 1 errors 6, no dir index, no inode ref
> ERROR: child eb corrupted: parent bytenr=499748732928 item=14 parent
> level=3 child level=0
> root 5 inode 9133292 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 6 name .pulse filetype 2
> errors 6, no dir index, no inode ref
> ERROR: child eb corrupted: parent bytenr=499752206336 item=14 parent
> level=3 child level=0
> root 5 inode 9980485 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 9 name bootchart filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 10254180 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 4 name temp filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 11827458 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 8 name tftpboot filetype 2
> errors 6, no dir index, no inode ref
> ERROR: child eb corrupted: parent bytenr=499748732928 item=17 parent
> level=3 child level=0
> root 5 inode 14435353 errors 2001, no inode item, link count wrong
>     unresolved ref dir 131327 index 0 namelen 4 name s0be filetype 2
> errors 6, no dir index, no inode ref
> ERROR: child eb corrupted: parent bytenr=499752206336 item=17 parent
> level=3 child level=0
> root 5 inode 14646004 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 4 name s0be filetype 2
> errors 6, no dir index, no inode ref
> ERROR: child eb corrupted: parent bytenr=499748732928 item=17 parent
> level=3 child level=0
> root 5 inode 14910590 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 13 name .bash_history
> filetype 1 errors 6, no dir index, no inode ref
> root 5 inode 23438027 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 8 name mknod.sh filetype 1
> errors 6, no dir index, no inode ref
> root 5 inode 25548332 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 8 name ctrl_dir filetype 1
> errors 6, no dir index, no inode ref
> root 5 inode 25548333 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 5 name state filetype 1
> errors 6, no dir index, no inode ref
> root 5 inode 26407893 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 6 name e2fsck filetype 1
> errors 6, no dir index, no inode ref
> root 5 inode 36767707 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924630 index 0 namelen 31 name
> lib_mysqludf_log-warnings.patch filetype 1 errors 6, no dir index, no
> inode ref
> root 5 inode 36767712 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924632 index 0 namelen 32 name
> lib_mysqludf_stem-mysql_m4.patch filetype 1 errors 6, no dir index, no
> inode ref
> root 5 inode 36767838 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924633 index 0 namelen 33 name
> mysql-udf-base64-signedness.patch filetype 1 errors 6, no dir index,
> no inode ref
> root 5 inode 36767839 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924633 index 0 namelen 20 name
> mysql-udf-base64.sql filetype 1 errors 6, no dir index, no inode ref
> root 5 inode 36767840 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924634 index 0 namelen 27 name
> mysql-udf-http-stdlib.patch filetype 1 errors 6, no dir index, no
> inode ref
> root 5 inode 36767843 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924635 index 0 namelen 29 name
> mysql-udf-ipv6-warnings.patch filetype 1 errors 6, no dir index, no
> inode ref
> root 5 inode 37030378 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924616 index 0 namelen 12 name metadata.xml
> filetype 1 errors 6, no dir index, no inode ref
> root 5 inode 37030384 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924619 index 0 namelen 12 name metadata.xml
> filetype 1 errors 6, no dir index, no inode ref
> root 5 inode 37030490 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924625 index 0 namelen 12 name metadata.xml
> filetype 1 errors 6, no dir index, no inode ref
> root 5 inode 37030492 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924626 index 0 namelen 12 name metadata.xml
> filetype 1 errors 6, no dir index, no inode ref
> root 5 inode 37030497 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924628 index 0 namelen 12 name metadata.xml
> filetype 1 errors 6, no dir index, no inode ref
> root 5 inode 39198776 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 8 name bootback filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 51520157 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924616 index 0 namelen 29 name
> lib_mysqludf_log-0.0.2.ebuild filetype 1 errors 6, no dir index, no
> inode ref
> root 5 inode 51520163 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924619 index 0 namelen 30 name
> lib_mysqludf_stem-0.9.1.ebuild filetype 1 errors 6, no dir index, no
> inode ref
> root 5 inode 51520278 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924625 index 0 namelen 32 name
> mysql-udf-base64-20010618.ebuild filetype 1 errors 6, no dir index, no
> inode ref
> root 5 inode 51520280 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924626 index 0 namelen 25 name
> mysql-udf-http-1.0.ebuild filetype 1 errors 6, no dir index, no inode
> ref
> root 5 inode 51520284 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924628 index 0 namelen 25 name
> mysql-udf-ipv6-0.4.ebuild filetype 1 errors 6, no dir index, no inode
> ref
> root 5 inode 58792107 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name foo filetype 1
> errors 6, no dir index, no inode ref
> root 5 inode 63681251 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 4 name xpra filetype 2
> errors 6, no dir index, no inode ref
> root 5 inode 66715793 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 8 name .viminfo filetype 1
> errors 6, no dir index, no inode ref
> root 5 inode 71469207 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924616 index 0 namelen 8 name Manifest filetype
> 1 errors 6, no dir index, no inode ref
> root 5 inode 71469208 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924619 index 0 namelen 8 name Manifest filetype
> 1 errors 6, no dir index, no inode ref
> root 5 inode 71469238 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924625 index 0 namelen 8 name Manifest filetype
> 1 errors 6, no dir index, no inode ref
> root 5 inode 71469239 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924626 index 0 namelen 8 name Manifest filetype
> 1 errors 6, no dir index, no inode ref
> root 5 inode 71469240 errors 2001, no inode item, link count wrong
>     unresolved ref dir 924628 index 0 namelen 8 name Manifest filetype
> 1 errors 6, no dir index, no inode ref
> ERROR: child eb corrupted: parent bytenr=499748732928 item=65 parent
> level=3 child level=1
> root 5 inode 75334150 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 7 name lib.new filetype 2
> errors 6, no dir index, no inode ref
> parent transid verify failed on 499774287872 wanted 3323349 found 3323340
> parent transid verify failed on 499774291968 wanted 3323349 found 3323340
> parent transid verify failed on 499774296064 wanted 3323349 found 3323340
> root 5 inode 77351088 errors 2001, no inode item, link count wrong
>     unresolved ref dir 256 index 0 namelen 3 name lib filetype 7
> errors 6, no dir index, no inode ref
> ERROR: errors found in fs roots
> 
> Starting repair.
> Opening filesystem to check...
> Checking filesystem on /dev/sda3
> UUID: 815266d6-a8b9-4f63-a593-02fde178263f
> cache and super generation don't match, space cache will be invalidated
> reset isize for dir 256 root 5
> reset isize for dir 131327 root 5
> reset isize for dir 924616 root 5
> Moving file 'lib_mysqludf_log' to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 924616
> reset isize for dir 924619 root 5
> Moving file 'lib_mysqludf_stem' to 'lost+found' dir since it has no
> valid backref
> Fixed the nlink of inode 924619
> reset isize for dir 924625 root 5
> Moving file 'mysql-udf-base64' to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 924625
> reset isize for dir 924626 root 5
> Moving file 'mysql-udf-http' to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 924626
> reset isize for dir 924628 root 5
> Moving file 'mysql-udf-ipv6' to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 924628
> reset isize for dir 924630 root 5
> reset isize for dir 924632 root 5
> reset isize for dir 924633 root 5
> reset isize for dir 924634 root 5
> reset isize for dir 924635 root 5
> Trying to rebuild inode:1442047
> Trying to rebuild inode:1835263
> Trying to rebuild inode:1966335
> Trying to rebuild inode:2097407
> Trying to rebuild inode:2228479
> Trying to rebuild inode:2359551
> Trying to rebuild inode:3145983
> Trying to rebuild inode:3277055
> Trying to rebuild inode:3408127
> Trying to rebuild inode:3539199
> Trying to rebuild inode:3670271
> Trying to rebuild inode:4587775
> Trying to rebuild inode:4980991
> Trying to rebuild inode:4984678
> Trying to rebuild inode:5505279
> Trying to rebuild inode:5767423
> Trying to rebuild inode:5898495
> Trying to rebuild inode:6029567
> Trying to rebuild inode:6431345
> Trying to rebuild inode:6553855
> Trying to rebuild inode:7602431
> Trying to rebuild inode:9133291
> Trying to rebuild inode:9133292
> Trying to rebuild inode:9980485
> Trying to rebuild inode:10254180
> Trying to rebuild inode:11827458
> Trying to rebuild inode:14435353
> Trying to rebuild inode:14646004
> Trying to rebuild inode:14910590
> Trying to rebuild inode:23438027
> Trying to rebuild inode:25548332
> Trying to rebuild inode:25548333
> Trying to rebuild inode:26407893
> Trying to rebuild inode:36767707
> Trying to rebuild inode:36767712
> Trying to rebuild inode:36767838
> Trying to rebuild inode:36767839
> Trying to rebuild inode:36767840
> Trying to rebuild inode:36767843
> Trying to rebuild inode:37030378
> Trying to rebuild inode:37030384
> Trying to rebuild inode:37030490
> Trying to rebuild inode:37030492
> Trying to rebuild inode:37030497
> Trying to rebuild inode:39198776
> Trying to rebuild inode:51520157
> Trying to rebuild inode:51520163
> Trying to rebuild inode:51520278
> Trying to rebuild inode:51520280
> Trying to rebuild inode:51520284
> Trying to rebuild inode:58792107
> Trying to rebuild inode:63681251
> Trying to rebuild inode:66715793
> Trying to rebuild inode:71469207
> Trying to rebuild inode:71469208
> Trying to rebuild inode:71469238
> Trying to rebuild inode:71469239
> Trying to rebuild inode:71469240
> Trying to rebuild inode:75334150
> Trying to rebuild inode:77351088
> found 89594781696 bytes used, error(s) found
> total csum bytes: 0
> total tree bytes: 108093440
> total fs tree bytes: 53248
> total extent tree bytes: 107958272
> btree space waste bytes: 36265970
> file data blocks allocated: 27983872
>  referenced 27983872
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-30  9:09                       ` Qu Wenruo
@ 2019-12-30  9:21                         ` Patrick Erley
  2019-12-30  9:27                           ` Qu Wenruo
  0 siblings, 1 reply; 75+ messages in thread
From: Patrick Erley @ 2019-12-30  9:21 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Btrfs BTRFS

On Mon, Dec 30, 2019 at 9:09 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2019/12/30 下午5:01, Patrick Erley wrote:
> > On Mon, Dec 30, 2019 at 8:54 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >> On 2019/12/30 下午4:14, Patrick Erley wrote:
> >>> On Mon, Dec 30, 2019 at 6:09 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>
> >>> Should I also paste in the repair log?
> >>>
> >> Yes please.
> >>
> >> This sounds very strange, especially for the transid mismatch part.
> >>
> >> Thanks,
> >> Qu
> >>
> >
> >
> >
> > enabling repair mode
> > WARNING:
> >
> >     Do not use --repair unless you are advised to do so by a developer
> >     or an experienced user, and then only after having accepted that no
> >     fsck can successfully repair all types of filesystem corruption. Eg.
> >     some software or hardware bugs can fatally damage a volume.
> >     The operation will start in 10 seconds.
> >     Use Ctrl-C to stop it.
> > 10 9 8 7 6 5 4 3 2 1parent transid verify failed on 499774464000
> > wanted 3323349 found 3323340
> > parent transid verify failed on 499774521344 wanted 3323349 found 3323340
> > parent transid verify failed on 499774529536 wanted 3323349 found 3323340
>
> This message is from open_ctree(), which means the fs is already corrupted.
>
> Would you like to provide the history between last good btrfs check run
> and --repair run?
>
> Thanks,
> Qu

In theory, all I did was boot back into 5.1 and continued using the
system.  After you said I should go ahead and try to --repair, I
rebooted into initramfs and ran the repair, then continued
booting(which failed spectacularly, due to almost all of / being
missing).  I then rebooted back into initramfs to assess what was
going on, and made a liveusb (from which I'm sending this on that
system).  Some 'background' on the FS: It was migrated from ext4 ~7?
years ago, and has been moved between multiple discs and systems using
dd.  Interesting point: The only files/folders that still exist in /
were created after I migrated the filesystem.  If I can get /etc and
maybe /var back, I'm golden (there are a few bits in each I don't
include in my hot backups, so will have to go to offline storage to
fetch them).

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-30  9:21                         ` Patrick Erley
@ 2019-12-30  9:27                           ` Qu Wenruo
  2019-12-30 10:06                             ` Patrick Erley
  0 siblings, 1 reply; 75+ messages in thread
From: Qu Wenruo @ 2019-12-30  9:27 UTC (permalink / raw)
  To: Patrick Erley; +Cc: Btrfs BTRFS


[-- Attachment #1.1: Type: text/plain, Size: 2888 bytes --]



On 2019/12/30 下午5:21, Patrick Erley wrote:
> On Mon, Dec 30, 2019 at 9:09 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2019/12/30 下午5:01, Patrick Erley wrote:
>>> On Mon, Dec 30, 2019 at 8:54 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>> On 2019/12/30 下午4:14, Patrick Erley wrote:
>>>>> On Mon, Dec 30, 2019 at 6:09 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>
>>>>> Should I also paste in the repair log?
>>>>>
>>>> Yes please.
>>>>
>>>> This sounds very strange, especially for the transid mismatch part.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>
>>>
>>>
>>> enabling repair mode
>>> WARNING:
>>>
>>>     Do not use --repair unless you are advised to do so by a developer
>>>     or an experienced user, and then only after having accepted that no
>>>     fsck can successfully repair all types of filesystem corruption. Eg.
>>>     some software or hardware bugs can fatally damage a volume.
>>>     The operation will start in 10 seconds.
>>>     Use Ctrl-C to stop it.
>>> 10 9 8 7 6 5 4 3 2 1parent transid verify failed on 499774464000
>>> wanted 3323349 found 3323340
>>> parent transid verify failed on 499774521344 wanted 3323349 found 3323340
>>> parent transid verify failed on 499774529536 wanted 3323349 found 3323340
>>
>> This message is from open_ctree(), which means the fs is already corrupted.
>>
>> Would you like to provide the history between last good btrfs check run
>> and --repair run?
>>
>> Thanks,
>> Qu
> 
> In theory, all I did was boot back into 5.1 and continued using the
> system. 

If that's the only thing before --repair (if there is only one repair
run, and output is exactly what you pasted), then I guess something
didn't go right in that 5.1 run?

Is that pasted output from the first --repair run?

If there is another run before the pasted output, then it could be
previous --repair.

Either way, I'm very sorry for the data loss...

> After you said I should go ahead and try to --repair, I
> rebooted into initramfs and ran the repair, then continued
> booting(which failed spectacularly, due to almost all of / being
> missing).  I then rebooted back into initramfs to assess what was
> going on, and made a liveusb (from which I'm sending this on that
> system).  Some 'background' on the FS: It was migrated from ext4 ~7?
> years ago, and has been moved between multiple discs and systems using
> dd.  Interesting point: The only files/folders that still exist in /
> were created after I migrated the filesystem.  If I can get /etc and
> maybe /var back, I'm golden (there are a few bits in each I don't
> include in my hot backups, so will have to go to offline storage to
> fetch them).

I'm afraid the only chance we left is btrfs-restore.

And normally for transid error, the chance is pretty low then.

Thanks,
Qu

> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2019-12-30  9:27                           ` Qu Wenruo
@ 2019-12-30 10:06                             ` Patrick Erley
  0 siblings, 0 replies; 75+ messages in thread
From: Patrick Erley @ 2019-12-30 10:06 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Btrfs BTRFS

That sucks :-( The only reason I had hope for /etc and /var was they
show up in btrfs check as not having a parent and i figured I could
reparent them onto the new / and have a chance of extracting something
useful from them.  I guess I'll plan on grabbing images of the FS
tomorrow and starting a full rebuild.  Hopefully someone will chime in
with a way to get /etc/ and /var/  before I reach the point I need
them.

On Mon, Dec 30, 2019 at 9:27 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2019/12/30 下午5:21, Patrick Erley wrote:
> > On Mon, Dec 30, 2019 at 9:09 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>
> >>
> >>
> >> On 2019/12/30 下午5:01, Patrick Erley wrote:
> >>> On Mon, Dec 30, 2019 at 8:54 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>> On 2019/12/30 下午4:14, Patrick Erley wrote:
> >>>>> On Mon, Dec 30, 2019 at 6:09 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>>
> >>>>> Should I also paste in the repair log?
> >>>>>
> >>>> Yes please.
> >>>>
> >>>> This sounds very strange, especially for the transid mismatch part.
> >>>>
> >>>> Thanks,
> >>>> Qu
> >>>>
> >>>
> >>>
> >>>
> >>> enabling repair mode
> >>> WARNING:
> >>>
> >>>     Do not use --repair unless you are advised to do so by a developer
> >>>     or an experienced user, and then only after having accepted that no
> >>>     fsck can successfully repair all types of filesystem corruption. Eg.
> >>>     some software or hardware bugs can fatally damage a volume.
> >>>     The operation will start in 10 seconds.
> >>>     Use Ctrl-C to stop it.
> >>> 10 9 8 7 6 5 4 3 2 1parent transid verify failed on 499774464000
> >>> wanted 3323349 found 3323340
> >>> parent transid verify failed on 499774521344 wanted 3323349 found 3323340
> >>> parent transid verify failed on 499774529536 wanted 3323349 found 3323340
> >>
> >> This message is from open_ctree(), which means the fs is already corrupted.
> >>
> >> Would you like to provide the history between last good btrfs check run
> >> and --repair run?
> >>
> >> Thanks,
> >> Qu
> >
> > In theory, all I did was boot back into 5.1 and continued using the
> > system.
>
> If that's the only thing before --repair (if there is only one repair
> run, and output is exactly what you pasted), then I guess something
> didn't go right in that 5.1 run?
>
> Is that pasted output from the first --repair run?
>
> If there is another run before the pasted output, then it could be
> previous --repair.
>
> Either way, I'm very sorry for the data loss...
>
> > After you said I should go ahead and try to --repair, I
> > rebooted into initramfs and ran the repair, then continued
> > booting(which failed spectacularly, due to almost all of / being
> > missing).  I then rebooted back into initramfs to assess what was
> > going on, and made a liveusb (from which I'm sending this on that
> > system).  Some 'background' on the FS: It was migrated from ext4 ~7?
> > years ago, and has been moved between multiple discs and systems using
> > dd.  Interesting point: The only files/folders that still exist in /
> > were created after I migrated the filesystem.  If I can get /etc and
> > maybe /var back, I'm golden (there are a few bits in each I don't
> > include in my hot backups, so will have to go to offline storage to
> > fetch them).
>
> I'm afraid the only chance we left is btrfs-restore.
>
> And normally for transid error, the chance is pretty low then.
>
> Thanks,
> Qu
>
> >
>

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2021-11-23  2:42       ` x8062
@ 2021-11-23  5:56         ` Nikolay Borisov
  0 siblings, 0 replies; 75+ messages in thread
From: Nikolay Borisov @ 2021-11-23  5:56 UTC (permalink / raw)
  To: x8062; +Cc: linux-btrfs



On 23.11.21 г. 4:42, x8062 wrote:
> At 2021-11-22 18:36:41, "Nikolay Borisov" <nborisov@suse.com> wrote:
>>
>>
>> On 22.11.21 г. 12:07, x8062 wrote:
>>> At 2021-11-22 15:24:38, "Nikolay Borisov" <nborisov@suse.com> wrote:
>>>>
>>>>
>>>> On 22.11.21 г. 7:26, x8062 wrote:
>>>>> Hello,
>>>>>  I got periodic warns in my linux console. in dmesg it is the following pasted text.
>>>>> At https://btrfs.wiki.kernel.org/index.php/Tree-checker I learned it may be a error, so i send the message. Hopefully it could help, Thanks in advance!
>>>>>
>>>>> [  513.900852] BTRFS critical (device sdb3): corrupt leaf: root=381 block=71928348672 slot=74 ino=2394634 file_offset=0, invalid ram_bytes for uncompressed inline extent, have 393 expect 131465
>>>>>
>>>>
>>>>
>>>> You have faulty ram, since 393 has the 17th bit set to 0 whilst has it
>>>> set to 1. So your ram is clearly corrupting bits. I advise you run a
>>>> memtest tool and look for possibly changing the faulty ram module.
>>>
>>> Thank you, can't believe the ram is not so stable.  I'll run a memtest later.
>>
>> Actually according to the output this is a read-time corruption. THis
>> means the corrupted data has already been written to disk, likely by an
>> older kernel that didn't have the tree cherk code. So running a memcheck
>> is still useful to prevent future corruption.
>>
>> As far as the corrupted files goes - well its data is corrupted. It can
>> technically be fixed, but you'd have to do it yourself. Or alternatively
>> go back on an older kernel i.e pre- 5.11 and try to copy that particular
>> file (inode 2394634).
>>
>>>
> I find some problems here.  I use the command "find . -inum 2394634" in the btrfs root dir,  but nothing printed.
> does "root=381" means the subvolume ID=381? but now I don't have such subvolume. I deleted some of the

Yes root is the id of the subvolume, if you have deleted it then the
corrupted inode should also be gone.


> subvolumes a few days ago. this is the current subvolume list(some of the dir name shortened)
> sudo btrfs subvol list .
> ID 263 gen 111732 top level 5 path 8007/a
> ID 354 gen 111729 top level 5 path 8007/b
> ID 622 gen 111757 top level 5 path f015
> ID 1174 gen 111758 top level 5 path cc
> ID 1326 gen 111757 top level 5 path 8007/c
> ID 1781 gen 111740 top level 5 path ip
> ID 1782 gen 111758 top level 5 path og
> ID 1856 gen 111586 top level 1782 path og/OG/DB/server
> ID 1858 gen 111757 top level 622 path V6/db
> ID 1875 gen 111742 top level 5 path sdk
> ID 1918 gen 111742 top level 5 path 8015
> ID 1942 gen 111745 top level 5 path ip6
> ID 2007 gen 111758 top level 5 path mnew
> ID 2114 gen 111751 top level 5 path dd
> ID 2116 gen 111760 top level 2117 path ds/trunk/20200616
> ID 2117 gen 111758 top level 5 path ds
> ID 2118 gen 111758 top level 2114 path dd/trunk/sourcecode
> ID 2119 gen 111751 top level 2114 path dd/b/1103
> ID 2120 gen 111761 top level 5 path tt
> 

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2021-11-22 10:07   ` x8062
@ 2021-11-22 10:36     ` Nikolay Borisov
  2021-11-23  2:42       ` x8062
  0 siblings, 1 reply; 75+ messages in thread
From: Nikolay Borisov @ 2021-11-22 10:36 UTC (permalink / raw)
  To: x8062; +Cc: linux-btrfs



On 22.11.21 г. 12:07, x8062 wrote:
> At 2021-11-22 15:24:38, "Nikolay Borisov" <nborisov@suse.com> wrote:
>>
>>
>> On 22.11.21 г. 7:26, x8062 wrote:
>>> Hello,
>>>  I got periodic warns in my linux console. in dmesg it is the following pasted text.
>>> At https://btrfs.wiki.kernel.org/index.php/Tree-checker I learned it may be a error, so i send the message. Hopefully it could help, Thanks in advance!
>>>
>>> [  513.900852] BTRFS critical (device sdb3): corrupt leaf: root=381 block=71928348672 slot=74 ino=2394634 file_offset=0, invalid ram_bytes for uncompressed inline extent, have 393 expect 131465
>>>
>>
>>
>> You have faulty ram, since 393 has the 17th bit set to 0 whilst has it
>> set to 1. So your ram is clearly corrupting bits. I advise you run a
>> memtest tool and look for possibly changing the faulty ram module.
> 
> Thank you, can't believe the ram is not so stable.  I'll run a memtest later.

Actually according to the output this is a read-time corruption. THis
means the corrupted data has already been written to disk, likely by an
older kernel that didn't have the tree cherk code. So running a memcheck
is still useful to prevent future corruption.

As far as the corrupted files goes - well its data is corrupted. It can
technically be fixed, but you'd have to do it yourself. Or alternatively
go back on an older kernel i.e pre- 5.11 and try to copy that particular
file (inode 2394634).

> 

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2021-11-22  5:26 read " x8062
@ 2021-11-22  7:24 ` Nikolay Borisov
  2021-11-22 10:07   ` x8062
  0 siblings, 1 reply; 75+ messages in thread
From: Nikolay Borisov @ 2021-11-22  7:24 UTC (permalink / raw)
  To: x8062, linux-btrfs



On 22.11.21 г. 7:26, x8062 wrote:
> Hello,
>  I got periodic warns in my linux console. in dmesg it is the following pasted text.
> At https://btrfs.wiki.kernel.org/index.php/Tree-checker I learned it may be a error, so i send the message. Hopefully it could help, Thanks in advance!
> 
> [  513.900852] BTRFS critical (device sdb3): corrupt leaf: root=381 block=71928348672 slot=74 ino=2394634 file_offset=0, invalid ram_bytes for uncompressed inline extent, have 393 expect 131465
> 


You have faulty ram, since 393 has the 17th bit set to 0 whilst has it
set to 1. So your ram is clearly corrupting bits. I advise you run a
memtest tool and look for possibly changing the faulty ram module.

^ permalink raw reply	[flat|nested] 75+ messages in thread

* read time tree block corruption detected
@ 2021-11-22  5:26 x8062
  2021-11-22  7:24 ` Nikolay Borisov
  0 siblings, 1 reply; 75+ messages in thread
From: x8062 @ 2021-11-22  5:26 UTC (permalink / raw)
  To: linux-btrfs

Hello,
 I got periodic warns in my linux console. in dmesg it is the following pasted text.
At https://btrfs.wiki.kernel.org/index.php/Tree-checker I learned it may be a error, so i send the message. Hopefully it could help, Thanks in advance!

[  513.900852] BTRFS critical (device sdb3): corrupt leaf: root=381 block=71928348672 slot=74 ino=2394634 file_offset=0, invalid ram_bytes for uncompressed inline extent, have 393 expect 131465

[  513.900869] BTRFS error (device sdb3): block=71928348672 read time tree block corruption detected
[  513.901534] BTRFS critical (device sdb3): corrupt leaf: root=381 block=71928348672 slot=74 ino=2394634 file_offset=0, invalid ram_bytes for uncompressed inline extent, have 393 expect 131465
[  513.901550] BTRFS error (device sdb3): block=71928348672 read time tree block corruption detected
[  513.912414] BTRFS critical (device sdb3): corrupt leaf: root=381 block=71928348672 slot=74 ino=2394634 file_offset=0, invalid ram_bytes for uncompressed inline extent, have 393 expect 131465
[  513.912433] BTRFS error (device sdb3): block=71928348672 read time tree block corruption detected
[  611.835932] BTRFS critical (device sdb3): corrupt leaf: root=381 block=71928348672 slot=74 ino=2394634 file_offset=0, invalid ram_bytes for uncompressed inline extent, have 393 expect 131465
[  611.835950] BTRFS error (device sdb3): block=71928348672 read time tree block corruption detected
[  611.836594] BTRFS critical (device sdb3): corrupt leaf: root=381 block=71928348672 slot=74 ino=2394634 file_offset=0, invalid ram_bytes for uncompressed inline extent, have 393 expect 131465
[  611.836603] BTRFS error (device sdb3): block=71928348672 read time tree block corruption detected
[  611.847484] BTRFS critical (device sdb3): corrupt leaf: root=381 block=71928348672 slot=74 ino=2394634 file_offset=0, invalid ram_bytes for uncompressed inline extent, have 393 expect 131465
[  611.847500] BTRFS error (device sdb3): block=71928348672 read time tree block corruption detected
[  708.354174] BTRFS critical (device sdb3): corrupt leaf: root=381 block=71928348672 slot=74 ino=2394634 file_offset=0, invalid ram_bytes for uncompressed inline extent, have 393 expect 131465
[  708.354186] BTRFS error (device sdb3): block=71928348672 read time tree block corruption detected
[  708.354833] BTRFS critical (device sdb3): corrupt leaf: root=381 block=71928348672 slot=74 ino=2394634 file_offset=0, invalid ram_bytes for uncompressed inline extent, have 393 expect 131465
[  708.354841] BTRFS error (device sdb3): block=71928348672 read time tree block corruption detected
[  708.357409] BTRFS critical (device sdb3): corrupt leaf: root=381 block=71928348672 slot=74 ino=2394634 file_offset=0, invalid ram_bytes for uncompressed inline extent, have 393 expect 131465
[  708.357420] BTRFS error (device sdb3): block=71928348672 read time tree block corruption detected

BTW: my mount option is (rw,relatime,space_cache,commit=90,subvolid=5,subvol=/), I'm using btrfs upon /dev/sdb3 and /dev/sda4, with btrfs raid0 support.
$ uname -a
Linux XXXP 5.15.3-arch1-1 #1 SMP PREEMPT Thu, 18 Nov 2021 22:23:08 +0000 x86_64 GNU/Linux
$ pacman -Qi btrfs-progs
Name            : btrfs-progs
Version         : 5.15-1
Description     : Btrfs filesystem utilities
Architecture    : x86_64
URL             : https://btrfs.wiki.kernel.org
Licenses        : GPL2
Groups          : None
Provides        : btrfs-progs-unstable
Depends On      : glibc  util-linux-libs  lzo  zlib  zstd  libgcrypt
Optional Deps   : python: libbtrfsutil python bindings [installed]
                  e2fsprogs: btrfs-convert [installed]
                  reiserfsprogs: btrfs-convert [installed]
Required By     : None
Optional For    : None
Conflicts With  : btrfs-progs-unstable
Replaces        : btrfs-progs-unstable
Installed Size  : 4.80 MiB
Packager        : Tobias Powalowski <tpowa@archlinux.org>
Build Date      : Sun 07 Nov 2021 02:56:20 AM CST
Install Date    : Tue 09 Nov 2021 01:10:11 PM CST
Install Reason  : Explicitly installed
Install Script  : Yes
Validated By    : Signature


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: Read time tree block corruption detected
       [not found]                             ` <162660684635.8.12423097770824212671.10223516@simplelogin.co>
@ 2021-07-18 12:16                               ` pepperpoint
  0 siblings, 0 replies; 75+ messages in thread
From: pepperpoint @ 2021-07-18 12:16 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Qu Wenruo, linux-btrfs

Hi Qu,

I run the command and returns no output.
Thank you for helping me solve the problem!

Regards,
Lester

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Sunday, July 18th, 2021 at 7:14 PM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:

> On 2021/7/18 下午6:34, pepperpoint@mb.ardentcoding.com wrote:
>
> > Hi Qu,
> >
> > I boot an ISO with Linux 5.1.15, mount the filesystem, wait for a while and restart.
> >
> > This command did not output anything when I boot the system and run it:
> >
> > btrfs ins dump-tree -t extent /dev/dm-0 | grep 174113599488 -A 3
> >
> > Checking the logs, I do not see the error anymore from the time I boot the system. I also ran btrfs scrub just in case and detected no errors.
> >
> > I think the filesystem is now fixed?
>
> Yep, as expected.
>
> If you want to be extra extra sure, you can try another command (no need
>
> to unmount the fs):
>
> btrfs ins dump-tree -t root /dev/dm-0 | grep "(363 ROOT"
> ========================================================
>
> It should has no output.
>
> I'll later submit a patch to enhance btrfs-progs to detect such problem.
>
> Thanks,
>
> Qu
>
> > Regards,
> >
> > Lester
> >
> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> >
> > On Sunday, July 18th, 2021 at 5:32 PM, Qu Wenruo quwenruo.btrfs@gmx.com wrote:
> >
> > > On 2021/7/18 下午4:46, pepperpoint@mb.ardentcoding.com wrote:
> > >
> > > > Hi Qu,
> > > >
> > > > When I find which directory some of the filenames are located, they are under /var/lib.
> > > >
> > > > I had subvolume for /var which was created probably around 2018.
> > >
> > > Then it's possible by somehow we allowed that hardlink to directory.
> > >
> > > Not sure if it's a bug in VFS layer or in btrfs itself.
> > >
> > > But around 2019 (aka, v5.2 kernel), that check for refs of directory is
> > >
> > > introduced and at the same time, write-time tree checker is introduced.
> > >
> > > This means if the bug happens after v5.2 kernel, it will be rejected
> > >
> > > before submitting to disk.
> > >
> > > So the problem definitely happens before the install of v5.2 kernel.
> > >
> > > > I don't remember how I created this but I probably use rsync to copy the files from existing /var
> > > >
> > > > or created a snapshot of root and delete other files that is not under /var.
> > >
> > > But that's still pretty weird.
> > >
> > > > Around June, I tried to move the filesystem to another partition through btrfs device add and btrfs device remove
> > > >
> > > > but failed due to that error and was advised to use btrfs replace instead.
> > > >
> > > > Then at the beginning of this month, I reorganized it merging most of the /var content back to root
> > > >
> > > > and created subvolume for /var/lib/mysql and /var/lib/mongodb. I encountered an error when
> > > >
> > > > I copy some of the files through cp --reflink but I failed for /var/lib/mysql so I created a snapshot
> > > >
> > > > from /var and remove the extra files.
> > > >
> > > > This is also the time I saw the errors in the log. Before that, the errors was not in the log.
> > >
> > > At least, we should prevent such problem from reaching disk.
> > >
> > > If you reverted to older LTS kernel, using Arch Linux Archive, it would
> > >
> > > be possible to continue deleting the subvolume and solve the problem.
> > >
> > > After the root 363 get fully deleted, you can verify that tree block get
> > >
> > > deleted by the following command:
> > >
> > > btrfs ins dump-tree -t extent <device> | grep 174113599488 -A 3
> > > ===============================================================
> > >
> > > Which should show no output.
> > >
> > > Thanks,
> > >
> > > Qu
> > >
> > > > Regards,
> > > >
> > > > Lester
> > > >
> > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > >
> > > > On Sunday, July 18th, 2021 at 3:27 PM, Qu Wenruo quwenruo.btrfs@gmx.com wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > BTW, it's really important for us to know how the directory is hardlinked.
> > > > >
> > > > > Thus I salvaged the filenames found in the half-dropped root 363.
> > > > >
> > > > > Since it may contain confidential info, I send the filename list to you
> > > > >
> > > > > off-list.
> > > > >
> > > > > If you can remind what the root 363 is used for, and any possible
> > > > >
> > > > > operations which may be involved in that subvolume, it's better to reply
> > > > >
> > > > > it to the mail list so that we can get some clue on the root cause.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Qu
> > > > >
> > > > > On 2021/7/18 下午3:15, Qu Wenruo wrote:
> > > > >
> > > > > > On 2021/7/18 下午1:26, pepperpoint@mb.ardentcoding.com wrote:
> > > > > >
> > > > > > > Hi Qu,
> > > > > > >
> > > > > > > May I know if there are any leads on this? What should I do for now?
> > > > > >
> > > > > > Sorry for the late reply.
> > > > > >
> > > > > > With the image dump, it's much easier to find what's going wrong.
> > > > > >
> > > > > > -   About root 363
> > > > > >
> > > > > >     It's an orphan root, thus user can't access it directly.
> > > > > >
> > > > > >
> > > > > > Furthermore, it's being dropped, thus "btrfs ins dump-tree -t 363"
> > > > > >
> > > > > > reports transid error, as part of the tree has already been dropped,
> > > > > >
> > > > > > and this is expected.
> > > > > >
> > > > > > So far your fs is still OK, except that reported error.
> > > > > >
> > > > > > -   About the offending tree block
> > > > > >
> > > > > >     The offending tree block only belongs to the delete subvolume 363,
> > > > > >
> > > > > >     thus it should be delete soon.
> > > > > >
> > > > > >
> > > > > > But unfortunately due to the corrupted content, it's unable to be
> > > > > >
> > > > > > deleted.
> > > > > >
> > > > > > For now, if you can re-compile btrfs module, we can workaround the
> > > > > >
> > > > > > problem by temporarily disable read-time tree-checker so that the
> > > > > >
> > > > > > deletion can continue, and after the root 363 get fully deleted, the
> > > > > >
> > > > > > problem should be gone.
> > > > > >
> > > > > > Or you can use older kernel, any kernel <= v5.1 should not have the
> > > > > >
> > > > > > enhanced check, thus can continue with the subvolume deletion.
> > > > > >
> > > > > > If you want to go through the re-compile path, the needed diff is
> > > > > >
> > > > > > attached
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Qu
> > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > Lester
> > > > > > >
> > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > > > >
> > > > > > > On Saturday, July 17th, 2021 at 8:51 PM,
> > > > > > >
> > > > > > > pepperpoint@mb.ardentcoding.com wrote:
> > > > > > >
> > > > > > > > Hi Qu,
> > > > > > > >
> > > > > > > > I run btrfs ins dump-tree -t 363 unmounted but the same error
> > > > > > > >
> > > > > > > > appears. Rerunning btrfs check does not show any error.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Lester
> > > > > > > >
> > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > > > > >
> > > > > > > > On Saturday, July 17th, 2021 at 6:48 PM, Qu Wenruo -
> > > > > > > >
> > > > > > > > quwenruo.btrfs@gmx.com wrote:
> > > > > > > >
> > > > > > > > > On 2021/7/17 下午6:34, pepperpoint@mb.ardentcoding.com wrote:
> > > > > > > > >
> > > > > > > > > > Hi Qu,
> > > > > > > > > >
> > > > > > > > > > Unfortunately I cannot find subvolume 363
> > > > > > > > > >
> > > > > > > > > > btrfs subvolume list /run/media/root
> > > > > > > > > > ====================================
> > > > > > > > > >
> > > > > > > > > > ID 361 gen 1814826 top level 584 path @/live/snapshot
> > > > > > > > > >
> > > > > > > > > > ID 364 gen 1814414 top level 5 path @vtmp/live/snapshot
> > > > > > > > > >
> > > > > > > > > > ID 369 gen 1814414 top level 5 path @vlmachines/live/snapshot
> > > > > > > > > >
> > > > > > > > > > ID 493 gen 1814414 top level 5 path @vlportables/live/snapshot
> > > > > > > > > >
> > > > > > > > > > ID 579 gen 1814828 top level 5 path @vlog/live/snapshot
> > > > > > > > > >
> > > > > > > > > > ID 580 gen 1814414 top level 5 path @vcache/live/snapshot
> > > > > > > > > >
> > > > > > > > > > ID 581 gen 1814414 top level 5 path @vlmongodb/live/snapshot
> > > > > > > > > >
> > > > > > > > > > ID 582 gen 1814414 top level 5 path @vlmysql/live/snapshot
> > > > > > > > > >
> > > > > > > > > > ID 583 gen 1814414 top level 5 path @vspool/live/snapshot
> > > > > > > > > >
> > > > > > > > > > ID 584 gen 1814414 top level 5 path @
> > > > > > > > > >
> > > > > > > > > > ID 598 gen 1813420 top level 584 path @/4/snapshot
> > > > > > > > >
> > > > > > > > > Maybe 363 is some subvolume get deleted and later snapshot of it still
> > > > > > > > >
> > > > > > > > > exists.
> > > > > > > > >
> > > > > > > > > This will be harder to debug.
> > > > > > > > >
> > > > > > > > > Can you take a btrfs-image dump of your filesystem? (needs to be taken
> > > > > > > > >
> > > > > > > > > with the fs unmounted).
> > > > > > > > >
> > > > > > > > > The dumped image will contain your metadata, including file names and
> > > > > > > > >
> > > > > > > > > directory structures, but no data inside those files.
> > > > > > > > >
> > > > > > > > > Although btrfs-image has "-s" option to mask the filenames, but
> > > > > > > > >
> > > > > > > > > considering the filename in this case is useful to locate the inode, I
> > > > > > > > >
> > > > > > > > > guess it's better to take the image without any "-s" option.
> > > > > > > > >
> > > > > > > > > > btrfs ins dump-tree -t 363 /dev/dm-0 | grep -A 5 "(286 "
> > > > > > > > > > ========================================================
> > > > > > > > > >
> > > > > > > > > > parent transid verify failed on 174170742784 wanted 1789655 found
> > > > > > > > > >
> > > > > > > > > > 1812621
> > > > > > > > > >
> > > > > > > > > > parent transid verify failed on 174170742784 wanted 1789655 found
> > > > > > > > > >
> > > > > > > > > > 1812621
> > > > > > > > > >
> > > > > > > > > > parent transid verify failed on 174170742784 wanted 1789655 found
> > > > > > > > > >
> > > > > > > > > > 1812621
> > > > > > > > > >
> > > > > > > > > > Ignoring transid failure
> > > > > > > > > >
> > > > > > > > > > ERROR: child eb corrupted: parent bytenr=174170738688 item=0 parent
> > > > > > > > > >
> > > > > > > > > > level=2 child bytenr=174170742784 child level=0
> > > > > > > > >
> > > > > > > > > This transid mismatch may be a problem when running dump-tree on
> > > > > > > > >
> > > > > > > > > mounted
> > > > > > > > >
> > > > > > > > > fs, since you mentioned btrfs check reported no error, there shouldn't
> > > > > > > > >
> > > > > > > > > be a transid mismatch error.
> > > > > > > > >
> > > > > > > > > Anyway, if you can upload the btrfs-image dump, it would be much easier
> > > > > > > > >
> > > > > > > > > for us to debug and find out what's really going.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Qu
> > > > > > > > >
> > > > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > > > > > > >
> > > > > > > > > > On Saturday, July 17th, 2021 at 6:12 PM, Qu Wenruo wqu@suse.com wrote:
> > > > > > > > > >
> > > > > > > > > > > On 2021/7/17 下午4:57, pepperpoint@mb.ardentcoding.com wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Qu,
> > > > > > > > > > > >
> > > > > > > > > > > > I don't know how the directory was created but last month, I used
> > > > > > > > > > > >
> > > > > > > > > > > > btrfs device add and btrfs device remove to move the filesystem
> > > > > > > > > > > >
> > > > > > > > > > > > from one partition to another. It failed because of the same
> > > > > > > > > > > >
> > > > > > > > > > > > error and was advised to use btrfs replace instead. I don't know
> > > > > > > > > > > >
> > > > > > > > > > > > if the error also happened before I move the file system as I
> > > > > > > > > > > >
> > > > > > > > > > > > don't have any previous logs.
> > > > > > > > > > >
> > > > > > > > > > > It definitely happens before you moving the fs.
> > > > > > > > > > >
> > > > > > > > > > > As regular dev replacing/add/move only relocates the metadata, but
> > > > > > > > > > >
> > > > > > > > > > > not
> > > > > > > > > > >
> > > > > > > > > > > touching the fs trees.
> > > > > > > > > > >
> > > > > > > > > > > > Here is the result when I search for the inodes you mentioned if
> > > > > > > > > > > >
> > > > > > > > > > > > it helps:
> > > > > > > > > > > >
> > > > > > > > > > > > find /run/media/root -inum 260 -exec ls -ldi {} \;
> > > > > > > > > > > > ==================================================
> > > > > > > > > > > >
> > > > > > > > > > > > 260 -rw-r--r-- 1 root root 36864 Jun 25 06:22
> > > > > > > > > > > >
> > > > > > > > > > > > /run/media/root/@vcache/live/snapshot/app-info/cache/en_US.cache
> > > > > > > > > > > >
> > > > > > > > > > > > 260 drwx------ 1 mongodb mongodb 136 Sep 12 2020
> > > > > > > > > > > >
> > > > > > > > > > > > /run/media/root/@vlmongodb/live/snapshot/diagnostic.data
> > > > > > > > > > > >
> > > > > > > > > > > > 260 -rw-rw---- 1 mysql mysql 50331648 Sep 13 2015
> > > > > > > > > > > >
> > > > > > > > > > > > /run/media/root/@vlmysql/live/snapshot/ib_logfile0
> > > > > > > > > > > >
> > > > > > > > > > > > 260 -rw-r----- 1 root lp 8641 Mar 5 2014
> > > > > > > > > > > >
> > > > > > > > > > > > /run/media/root/@vspool/live/snapshot/cups/d00001-001
> > > > > > > > > > > >
> > > > > > > > > > > > 260 dr-xr-xr-x 1 root root 0 Sep 13 2013
> > > > > > > > > > > >
> > > > > > > > > > > > /run/media/root/@/live/snapshot/sys
> > > > > > > > > > > >
> > > > > > > > > > > > 260 dr-xr-xr-x 1 root root 0 Sep 13 2013
> > > > > > > > > > > >
> > > > > > > > > > > > /run/media/root/@/4/snapshot/sys
> > > > > > > > > > >
> > > > > > > > > > > Since btrfs can have the same inode number inside different
> > > > > > > > > > >
> > > > > > > > > > > subvolumes,
> > > > > > > > > > >
> > > > > > > > > > > you may want to limit the search inside subvolume 363.
> > > > > > > > > > >
> > > > > > > > > > > "-mount" option of find can do that, you only need to locate
> > > > > > > > > > >
> > > > > > > > > > > subvolume
> > > > > > > > > > >
> > > > > > > > > > > 363 by "btrfs subv list".
> > > > > > > > > > >
> > > > > > > > > > > But from these output I guess above two "sys" directory are more
> > > > > > > > > > >
> > > > > > > > > > > possible.
> > > > > > > > > > >
> > > > > > > > > > > Is there any directory named "blaklight" inside those directory?
> > > > > > > > > > >
> > > > > > > > > > > > find /run/media/root -inum 286 -exec ls -ldi {} \;
> > > > > > > > > > > > ==================================================
> > > > > > > > > > > >
> > > > > > > > > > > > 286 -rw-r--r-- 1 root root 96 Aug 16 2015
> > > > > > > > > > > >
> > > > > > > > > > > > /run/media/root/@vcache/live/snapshot/fontconfig/4b172ca7f111e3cffadc3636415fead9-le64.cache-4
> > > > > > > > > > > >
> > > > > > > > > > > > 286 -rw-rw---- 1 mysql mysql 4096 Sep 15 2013
> > > > > > > > > > > >
> > > > > > > > > > > > /run/media/root/@vlmysql/live/snapshot/mysql/columns_priv.MYI
> > > > > > > > > > > >
> > > > > > > > > > > > 286 -rw-r-----+ 1 root systemd-journal 16777216 Jul 4 01:14
> > > > > > > > > > > >
> > > > > > > > > > > > /run/media/root/@vlog/live/snapshot/journal/5098dd7845ae46d3ba1826c68a809a7c/user-1000@fbd9f65d0ea349f6b996716280e6c4dd-00000000002314c5-0005c5cb84a3a438.journal
> > > > > > > > > > >
> > > > > > > > > > > This is interesting, it means the inode 286 is not accessible.
> > > > > > > > > > >
> > > > > > > > > > > It can be some orphan inode, but would you dump subvolume 363 then
> > > > > > > > > > >
> > > > > > > > > > > try
> > > > > > > > > > >
> > > > > > > > > > > to locate the inode 286?
> > > > > > > > > > >
> > > > > > > > > > > One example command would be:
> > > > > > > > > > >
> > > > > > > > > > > btrfs ins dump-tree -t 363 <dev> | grep -A 5 "(286 "
> > > > > > > > > > > ====================================================
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Qu
> > > > > > > > > > >
> > > > > > > > > > > > Directories with pattern /root/@<dir>/live/snapshot/ are
> > > > > > > > > > > >
> > > > > > > > > > > > subvolumes and directories with pattern
> > > > > > > > > > > >
> > > > > > > > > > > > /root/@<dir>/<num>/snapshot/ are snapshots of live.
> > > > > > > > > > > >
> > > > > > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > > > > > > > > >
> > > > > > > > > > > > On Saturday, July 17th, 2021 at 4:14 PM, Qu Wenruo
> > > > > > > > > > > >
> > > > > > > > > > > > quwenruo.btrfs@gmx.com wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > On 2021/7/17 下午3:51, pepperpoint@mb.ardentcoding.com wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Qu,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Please see below for the dump.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > btrfs-progs v5.12.1
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > leaf 174113599488 items 18 free space 2008 generation 1330906
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > owner 363
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > leaf 174113599488 flags 0x1(WRITTEN) backref revision 1
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > fs uuid a7d327c4-8594-4116-a6f8-8aa2a4162063
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > chunk uuid f885f49e-14a0-4c80-9c12-c2302b9a0229
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > item 0 key (5471 INODE_ITEM 0) itemoff 3835 itemsize 160
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > generation 2063 transid 27726 size 40 nbytes 40
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > sequence 1501 flags 0x0(none)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > atime 1386484844.468769570 (2013-12-08 14:40:44)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ctime 1386484844.468769570 (2013-12-08 14:40:44)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > mtime 1386484844.468769570 (2013-12-08 14:40:44)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > item 1 key (5471 INODE_REF 4399) itemoff 3824 itemsize 11
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > index 12 namelen 1 name: 8
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > item 2 key (5471 EXTENT_DATA 0) itemoff 3763 itemsize 61
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > generation 27726 type 0 (inline)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > inline extent data size 40 ram_bytes 40 compression 0 (none)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > item 3 key (5645 INODE_ITEM 0) itemoff 3603 itemsize 160
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > generation 2542 transid 61261 size 40 nbytes 40
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > sequence 24769 flags 0x0(none)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > atime 1394335806.351857522 (2014-03-09 11:30:06)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ctime 1394335827.344389955 (2014-03-09 11:30:27)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > mtime 1394335827.344389955 (2014-03-09 11:30:27)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > item 4 key (5645 INODE_REF 4399) itemoff 3592 itemsize 11
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > index 13 namelen 1 name: 7
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > item 5 key (5645 EXTENT_DATA 0) itemoff 3531 itemsize 61
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > generation 61261 type 0 (inline)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > inline extent data size 40 ram_bytes 40 compression 0 (none)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > item 6 key (7222 INODE_ITEM 0) itemoff 3371 itemsize 160
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > generation 5754 transid 5767 size 307 nbytes 307
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > sequence 7 flags 0x0(none)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > atime 1379834835.428558020 (2013-09-22 15:27:15)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ctime 1379834835.428558020 (2013-09-22 15:27:15)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > mtime 1379834835.428558020 (2013-09-22 15:27:15)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > item 7 key (7222 INODE_REF 287) itemoff 3344 itemsize 27
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > index 6 namelen 17 name: dhcpcd-eth0.lease
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > item 8 key (7222 EXTENT_DATA 0) itemoff 3016 itemsize 328
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > generation 5767 type 0 (inline)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > inline extent data size 307 ram_bytes 307 compression 0 (none)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > item 9 key (7415 INODE_ITEM 0) itemoff 2856 itemsize 160
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > generation 5904 transid 1330906 size 180 nbytes 0
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > block group 0 mode 40755 links 2 uid 0 gid 0 rdev 0
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > sequence 177 flags 0x0(none)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > atime 1483277713.141980592 (2017-01-01 21:35:13)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ctime 1563162901.234656246 (2019-07-15 11:55:01)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > mtime 1406534032.158605559 (2014-07-28 15:53:52)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > > > > > > > > >
> > > > > > > > > > > > > This inode is indeed a directory.
> > > > > > > > > > > > >
> > > > > > > > > > > > > But it has two hard links, which is definitely something
> > > > > > > > > > > > >
> > > > > > > > > > > > > unexpected.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Under Linux we shouldn't have any hardlink for directory, as it
> > > > > > > > > > > > >
> > > > > > > > > > > > > would
> > > > > > > > > > > > >
> > > > > > > > > > > > > easily lead to loops.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > item 10 key (7415 INODE_REF 260) itemoff 2837 itemsize 19
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > index 28 namelen 9 name: backlight
> > > > > > > > > > > > >
> > > > > > > > > > > > > Its parent inode is 260 in the same root, with the name backlight.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > item 11 key (7415 INODE_REF 286) itemoff 2818 itemsize 19
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > index 3 namelen 9 name: backlight
> > > > > > > > > > > > >
> > > > > > > > > > > > > Another hardlink in inode 286, which is definitely a regular thing.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Btrfs-progs lacks the ability to detect such problem, we need to
> > > > > > > > > > > > >
> > > > > > > > > > > > > enhance
> > > > > > > > > > > > >
> > > > > > > > > > > > > it first.
> > > > > > > > > > > > >
> > > > > > > > > > > > > But do you have any idea how this directory get created?
> > > > > > > > > > > > >
> > > > > > > > > > > > > It looks like the content of sysfs.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Qu
> > > > > > > > > > > > >
> > > > > > > > > > > > > > item 12 key (7415 DIR_ITEM 3128336373) itemoff 2746 itemsize 72
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > location key (120417 INODE_ITEM 0) type FILE
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > transid 117279 data_len 0 name_len 42
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > name: pci-0000:00:02.0:backlight:intel_backlight
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > item 13 key (7415 DIR_ITEM 3218198317) itemoff 2705 itemsize 41
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > location key (7487 INODE_ITEM 0) type FILE
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > transid 5992 data_len 0 name_len 11
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > name: acpi_video0
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > item 14 key (7415 DIR_ITEM 3582254411) itemoff 2638 itemsize 67
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > location key (55325 INODE_ITEM 0) type FILE
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > transid 63351 data_len 0 name_len 37
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > name: platform-VPC2004:00:backlight:ideapad
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > item 15 key (7415 DIR_INDEX 2) itemoff 2597 itemsize 41
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > location key (7487 INODE_ITEM 0) type FILE
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > transid 5992 data_len 0 name_len 11
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > name: acpi_video0
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > item 16 key (7415 DIR_INDEX 4) itemoff 2530 itemsize 67
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > location key (55325 INODE_ITEM 0) type FILE
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > transid 63351 data_len 0 name_len 37
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > name: platform-VPC2004:00:backlight:ideapad
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > item 17 key (7415 DIR_INDEX 5) itemoff 2458 itemsize 72
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > location key (120417 INODE_ITEM 0) type FILE
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > transid 117279 data_len 0 name_len 42
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > name: pci-0000:00:02.0:backlight:intel_backlight
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Saturday, July 17th, 2021 at 3:05 PM, Qu Wenruo
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > quwenruo.btrfs@gmx.com wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On 2021/7/17 上午9:45, pepperpoint@mb.ardentcoding.com wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hello,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I see this message on dmesg:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > [ 2452.256756] BTRFS critical (device dm-0): corrupt leaf:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > root=363 block=174113599488 slot=9 ino=7415, invalid nlink:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > has 2 expect no more than 1 for dir
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > [ 2452.256776] BTRFS error (device dm-0): block=174113599488
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > read time tree block corruption detected
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Please provide the following dump:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > btrfs ins dump-tree -b 174113599488 /dev/dm-0
> > > > > > > > > > > > > > > =============================================
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Qu
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > When I run btrfs scrub and btrfs check, no error was detected.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I am running Linux 5.12.15-arch1-1 and btrfs-progs v5.12.1
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > How should I fix this error?

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: Read time tree block corruption detected
       [not found]                           ` <162660447733.8.9782212603273165785.10222524@mb.ardentcoding.com>
@ 2021-07-18 11:13                             ` Qu Wenruo
       [not found]                             ` <162660684635.8.12423097770824212671.10223516@simplelogin.co>
  1 sibling, 0 replies; 75+ messages in thread
From: Qu Wenruo @ 2021-07-18 11:13 UTC (permalink / raw)
  To: pepperpoint; +Cc: Qu Wenruo, linux-btrfs



On 2021/7/18 下午6:34, pepperpoint@mb.ardentcoding.com wrote:
> Hi Qu,
>
> I boot an ISO with Linux 5.1.15, mount the filesystem, wait for a while and restart.
>
> This command did not output anything when I boot the system and run it:
> btrfs ins dump-tree -t extent /dev/dm-0 | grep 174113599488 -A 3
>
> Checking the logs, I do not see the error anymore from the time I boot the system. I also ran btrfs scrub just in case and detected no errors.
>
> I think the filesystem is now fixed?

Yep, as expected.

If you want to be extra extra sure, you can try another command (no need
to unmount the fs):

# btrfs ins dump-tree -t root /dev/dm-0 | grep "(363 ROOT"

It should has no output.

I'll later submit a patch to enhance btrfs-progs to detect such problem.

Thanks,
Qu
>
> Regards,
> Lester
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>
> On Sunday, July 18th, 2021 at 5:32 PM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>> On 2021/7/18 下午4:46, pepperpoint@mb.ardentcoding.com wrote:
>>
>>> Hi Qu,
>>>
>>> When I find which directory some of the filenames are located, they are under /var/lib.
>>>
>>> I had subvolume for /var which was created probably around 2018.
>>
>> Then it's possible by somehow we allowed that hardlink to directory.
>>
>> Not sure if it's a bug in VFS layer or in btrfs itself.
>>
>> But around 2019 (aka, v5.2 kernel), that check for refs of directory is
>>
>> introduced and at the same time, write-time tree checker is introduced.
>>
>> This means if the bug happens after v5.2 kernel, it will be rejected
>>
>> before submitting to disk.
>>
>> So the problem definitely happens before the install of v5.2 kernel.
>>
>>> I don't remember how I created this but I probably use rsync to copy the files from existing /var
>>>
>>> or created a snapshot of root and delete other files that is not under /var.
>>
>> But that's still pretty weird.
>>
>>> Around June, I tried to move the filesystem to another partition through btrfs device add and btrfs device remove
>>>
>>> but failed due to that error and was advised to use btrfs replace instead.
>>>
>>> Then at the beginning of this month, I reorganized it merging most of the /var content back to root
>>>
>>> and created subvolume for /var/lib/mysql and /var/lib/mongodb. I encountered an error when
>>>
>>> I copy some of the files through cp --reflink but I failed for /var/lib/mysql so I created a snapshot
>>>
>>> from /var and remove the extra files.
>>>
>>> This is also the time I saw the errors in the log. Before that, the errors was not in the log.
>>
>> At least, we should prevent such problem from reaching disk.
>>
>> If you reverted to older LTS kernel, using Arch Linux Archive, it would
>>
>> be possible to continue deleting the subvolume and solve the problem.
>>
>> After the root 363 get fully deleted, you can verify that tree block get
>>
>> deleted by the following command:
>>
>> btrfs ins dump-tree -t extent <device> | grep 174113599488 -A 3
>> ===============================================================
>>
>> Which should show no output.
>>
>> Thanks,
>>
>> Qu
>>
>>> Regards,
>>>
>>> Lester
>>>
>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>
>>> On Sunday, July 18th, 2021 at 3:27 PM, Qu Wenruo quwenruo.btrfs@gmx.com wrote:
>>>
>>>> Hi,
>>>>
>>>> BTW, it's really important for us to know how the directory is hardlinked.
>>>>
>>>> Thus I salvaged the filenames found in the half-dropped root 363.
>>>>
>>>> Since it may contain confidential info, I send the filename list to you
>>>>
>>>> off-list.
>>>>
>>>> If you can remind what the root 363 is used for, and any possible
>>>>
>>>> operations which may be involved in that subvolume, it's better to reply
>>>>
>>>> it to the mail list so that we can get some clue on the root cause.
>>>>
>>>> Thanks,
>>>>
>>>> Qu
>>>>
>>>> On 2021/7/18 下午3:15, Qu Wenruo wrote:
>>>>
>>>>> On 2021/7/18 下午1:26, pepperpoint@mb.ardentcoding.com wrote:
>>>>>
>>>>>> Hi Qu,
>>>>>>
>>>>>> May I know if there are any leads on this? What should I do for now?
>>>>>
>>>>> Sorry for the late reply.
>>>>>
>>>>> With the image dump, it's much easier to find what's going wrong.
>>>>>
>>>>> -   About root 363
>>>>>
>>>>>      It's an orphan root, thus user can't access it directly.
>>>>>
>>>>>
>>>>> Furthermore, it's being dropped, thus "btrfs ins dump-tree -t 363"
>>>>>
>>>>> reports transid error, as part of the tree has already been dropped,
>>>>>
>>>>> and this is expected.
>>>>>
>>>>> So far your fs is still OK, except that reported error.
>>>>>
>>>>> -   About the offending tree block
>>>>>
>>>>>      The offending tree block only belongs to the delete subvolume 363,
>>>>>
>>>>>      thus it should be delete soon.
>>>>>
>>>>>
>>>>> But unfortunately due to the corrupted content, it's unable to be
>>>>>
>>>>> deleted.
>>>>>
>>>>> For now, if you can re-compile btrfs module, we can workaround the
>>>>>
>>>>> problem by temporarily disable read-time tree-checker so that the
>>>>>
>>>>> deletion can continue, and after the root 363 get fully deleted, the
>>>>>
>>>>> problem should be gone.
>>>>>
>>>>> Or you can use older kernel, any kernel <= v5.1 should not have the
>>>>>
>>>>> enhanced check, thus can continue with the subvolume deletion.
>>>>>
>>>>> If you want to go through the re-compile path, the needed diff is
>>>>>
>>>>> attached
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Qu
>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Lester
>>>>>>
>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>
>>>>>> On Saturday, July 17th, 2021 at 8:51 PM,
>>>>>>
>>>>>> pepperpoint@mb.ardentcoding.com wrote:
>>>>>>
>>>>>>> Hi Qu,
>>>>>>>
>>>>>>> I run btrfs ins dump-tree -t 363 unmounted but the same error
>>>>>>>
>>>>>>> appears. Rerunning btrfs check does not show any error.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Lester
>>>>>>>
>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>
>>>>>>> On Saturday, July 17th, 2021 at 6:48 PM, Qu Wenruo -
>>>>>>>
>>>>>>> quwenruo.btrfs@gmx.com wrote:
>>>>>>>
>>>>>>>> On 2021/7/17 下午6:34, pepperpoint@mb.ardentcoding.com wrote:
>>>>>>>>
>>>>>>>>> Hi Qu,
>>>>>>>>>
>>>>>>>>> Unfortunately I cannot find subvolume 363
>>>>>>>>>
>>>>>>>>> btrfs subvolume list /run/media/root
>>>>>>>>> ====================================
>>>>>>>>>
>>>>>>>>> ID 361 gen 1814826 top level 584 path @/live/snapshot
>>>>>>>>>
>>>>>>>>> ID 364 gen 1814414 top level 5 path @vtmp/live/snapshot
>>>>>>>>>
>>>>>>>>> ID 369 gen 1814414 top level 5 path @vlmachines/live/snapshot
>>>>>>>>>
>>>>>>>>> ID 493 gen 1814414 top level 5 path @vlportables/live/snapshot
>>>>>>>>>
>>>>>>>>> ID 579 gen 1814828 top level 5 path @vlog/live/snapshot
>>>>>>>>>
>>>>>>>>> ID 580 gen 1814414 top level 5 path @vcache/live/snapshot
>>>>>>>>>
>>>>>>>>> ID 581 gen 1814414 top level 5 path @vlmongodb/live/snapshot
>>>>>>>>>
>>>>>>>>> ID 582 gen 1814414 top level 5 path @vlmysql/live/snapshot
>>>>>>>>>
>>>>>>>>> ID 583 gen 1814414 top level 5 path @vspool/live/snapshot
>>>>>>>>>
>>>>>>>>> ID 584 gen 1814414 top level 5 path @
>>>>>>>>>
>>>>>>>>> ID 598 gen 1813420 top level 584 path @/4/snapshot
>>>>>>>>
>>>>>>>> Maybe 363 is some subvolume get deleted and later snapshot of it still
>>>>>>>>
>>>>>>>> exists.
>>>>>>>>
>>>>>>>> This will be harder to debug.
>>>>>>>>
>>>>>>>> Can you take a btrfs-image dump of your filesystem? (needs to be taken
>>>>>>>>
>>>>>>>> with the fs unmounted).
>>>>>>>>
>>>>>>>> The dumped image will contain your metadata, including file names and
>>>>>>>>
>>>>>>>> directory structures, but no data inside those files.
>>>>>>>>
>>>>>>>> Although btrfs-image has "-s" option to mask the filenames, but
>>>>>>>>
>>>>>>>> considering the filename in this case is useful to locate the inode, I
>>>>>>>>
>>>>>>>> guess it's better to take the image without any "-s" option.
>>>>>>>>
>>>>>>>>> btrfs ins dump-tree -t 363 /dev/dm-0 | grep -A 5 "(286 "
>>>>>>>>> ========================================================
>>>>>>>>>
>>>>>>>>> parent transid verify failed on 174170742784 wanted 1789655 found
>>>>>>>>>
>>>>>>>>> 1812621
>>>>>>>>>
>>>>>>>>> parent transid verify failed on 174170742784 wanted 1789655 found
>>>>>>>>>
>>>>>>>>> 1812621
>>>>>>>>>
>>>>>>>>> parent transid verify failed on 174170742784 wanted 1789655 found
>>>>>>>>>
>>>>>>>>> 1812621
>>>>>>>>>
>>>>>>>>> Ignoring transid failure
>>>>>>>>>
>>>>>>>>> ERROR: child eb corrupted: parent bytenr=174170738688 item=0 parent
>>>>>>>>>
>>>>>>>>> level=2 child bytenr=174170742784 child level=0
>>>>>>>>
>>>>>>>> This transid mismatch may be a problem when running dump-tree on
>>>>>>>>
>>>>>>>> mounted
>>>>>>>>
>>>>>>>> fs, since you mentioned btrfs check reported no error, there shouldn't
>>>>>>>>
>>>>>>>> be a transid mismatch error.
>>>>>>>>
>>>>>>>> Anyway, if you can upload the btrfs-image dump, it would be much easier
>>>>>>>>
>>>>>>>> for us to debug and find out what's really going.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Qu
>>>>>>>>
>>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>>>
>>>>>>>>> On Saturday, July 17th, 2021 at 6:12 PM, Qu Wenruo wqu@suse.com wrote:
>>>>>>>>>
>>>>>>>>>> On 2021/7/17 下午4:57, pepperpoint@mb.ardentcoding.com wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Qu,
>>>>>>>>>>>
>>>>>>>>>>> I don't know how the directory was created but last month, I used
>>>>>>>>>>>
>>>>>>>>>>> btrfs device add and btrfs device remove to move the filesystem
>>>>>>>>>>>
>>>>>>>>>>> from one partition to another. It failed because of the same
>>>>>>>>>>>
>>>>>>>>>>> error and was advised to use btrfs replace instead. I don't know
>>>>>>>>>>>
>>>>>>>>>>> if the error also happened before I move the file system as I
>>>>>>>>>>>
>>>>>>>>>>> don't have any previous logs.
>>>>>>>>>>
>>>>>>>>>> It definitely happens before you moving the fs.
>>>>>>>>>>
>>>>>>>>>> As regular dev replacing/add/move only relocates the metadata, but
>>>>>>>>>>
>>>>>>>>>> not
>>>>>>>>>>
>>>>>>>>>> touching the fs trees.
>>>>>>>>>>
>>>>>>>>>>> Here is the result when I search for the inodes you mentioned if
>>>>>>>>>>>
>>>>>>>>>>> it helps:
>>>>>>>>>>>
>>>>>>>>>>> find /run/media/root -inum 260 -exec ls -ldi {} \;
>>>>>>>>>>> ==================================================
>>>>>>>>>>>
>>>>>>>>>>> 260 -rw-r--r-- 1 root root 36864 Jun 25 06:22
>>>>>>>>>>>
>>>>>>>>>>> /run/media/root/@vcache/live/snapshot/app-info/cache/en_US.cache
>>>>>>>>>>>
>>>>>>>>>>> 260 drwx------ 1 mongodb mongodb 136 Sep 12 2020
>>>>>>>>>>>
>>>>>>>>>>> /run/media/root/@vlmongodb/live/snapshot/diagnostic.data
>>>>>>>>>>>
>>>>>>>>>>> 260 -rw-rw---- 1 mysql mysql 50331648 Sep 13 2015
>>>>>>>>>>>
>>>>>>>>>>> /run/media/root/@vlmysql/live/snapshot/ib_logfile0
>>>>>>>>>>>
>>>>>>>>>>> 260 -rw-r----- 1 root lp 8641 Mar 5 2014
>>>>>>>>>>>
>>>>>>>>>>> /run/media/root/@vspool/live/snapshot/cups/d00001-001
>>>>>>>>>>>
>>>>>>>>>>> 260 dr-xr-xr-x 1 root root 0 Sep 13 2013
>>>>>>>>>>>
>>>>>>>>>>> /run/media/root/@/live/snapshot/sys
>>>>>>>>>>>
>>>>>>>>>>> 260 dr-xr-xr-x 1 root root 0 Sep 13 2013
>>>>>>>>>>>
>>>>>>>>>>> /run/media/root/@/4/snapshot/sys
>>>>>>>>>>
>>>>>>>>>> Since btrfs can have the same inode number inside different
>>>>>>>>>>
>>>>>>>>>> subvolumes,
>>>>>>>>>>
>>>>>>>>>> you may want to limit the search inside subvolume 363.
>>>>>>>>>>
>>>>>>>>>> "-mount" option of find can do that, you only need to locate
>>>>>>>>>>
>>>>>>>>>> subvolume
>>>>>>>>>>
>>>>>>>>>> 363 by "btrfs subv list".
>>>>>>>>>>
>>>>>>>>>> But from these output I guess above two "sys" directory are more
>>>>>>>>>>
>>>>>>>>>> possible.
>>>>>>>>>>
>>>>>>>>>> Is there any directory named "blaklight" inside those directory?
>>>>>>>>>>
>>>>>>>>>>> find /run/media/root -inum 286 -exec ls -ldi {} \;
>>>>>>>>>>> ==================================================
>>>>>>>>>>>
>>>>>>>>>>> 286 -rw-r--r-- 1 root root 96 Aug 16 2015
>>>>>>>>>>>
>>>>>>>>>>> /run/media/root/@vcache/live/snapshot/fontconfig/4b172ca7f111e3cffadc3636415fead9-le64.cache-4
>>>>>>>>>>>
>>>>>>>>>>> 286 -rw-rw---- 1 mysql mysql 4096 Sep 15 2013
>>>>>>>>>>>
>>>>>>>>>>> /run/media/root/@vlmysql/live/snapshot/mysql/columns_priv.MYI
>>>>>>>>>>>
>>>>>>>>>>> 286 -rw-r-----+ 1 root systemd-journal 16777216 Jul 4 01:14
>>>>>>>>>>>
>>>>>>>>>>> /run/media/root/@vlog/live/snapshot/journal/5098dd7845ae46d3ba1826c68a809a7c/user-1000@fbd9f65d0ea349f6b996716280e6c4dd-00000000002314c5-0005c5cb84a3a438.journal
>>>>>>>>>>
>>>>>>>>>> This is interesting, it means the inode 286 is not accessible.
>>>>>>>>>>
>>>>>>>>>> It can be some orphan inode, but would you dump subvolume 363 then
>>>>>>>>>>
>>>>>>>>>> try
>>>>>>>>>>
>>>>>>>>>> to locate the inode 286?
>>>>>>>>>>
>>>>>>>>>> One example command would be:
>>>>>>>>>>
>>>>>>>>>> btrfs ins dump-tree -t 363 <dev> | grep -A 5 "(286 "
>>>>>>>>>> ====================================================
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Qu
>>>>>>>>>>
>>>>>>>>>>> Directories with pattern /root/@<dir>/live/snapshot/ are
>>>>>>>>>>>
>>>>>>>>>>> subvolumes and directories with pattern
>>>>>>>>>>>
>>>>>>>>>>> /root/@<dir>/<num>/snapshot/ are snapshots of live.
>>>>>>>>>>>
>>>>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>>>>>
>>>>>>>>>>> On Saturday, July 17th, 2021 at 4:14 PM, Qu Wenruo
>>>>>>>>>>>
>>>>>>>>>>> quwenruo.btrfs@gmx.com wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On 2021/7/17 下午3:51, pepperpoint@mb.ardentcoding.com wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Qu,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please see below for the dump.
>>>>>>>>>>>>>
>>>>>>>>>>>>> btrfs-progs v5.12.1
>>>>>>>>>>>>>
>>>>>>>>>>>>> leaf 174113599488 items 18 free space 2008 generation 1330906
>>>>>>>>>>>>>
>>>>>>>>>>>>> owner 363
>>>>>>>>>>>>>
>>>>>>>>>>>>> leaf 174113599488 flags 0x1(WRITTEN) backref revision 1
>>>>>>>>>>>>>
>>>>>>>>>>>>> fs uuid a7d327c4-8594-4116-a6f8-8aa2a4162063
>>>>>>>>>>>>>
>>>>>>>>>>>>> chunk uuid f885f49e-14a0-4c80-9c12-c2302b9a0229
>>>>>>>>>>>>>
>>>>>>>>>>>>> item 0 key (5471 INODE_ITEM 0) itemoff 3835 itemsize 160
>>>>>>>>>>>>>
>>>>>>>>>>>>> generation 2063 transid 27726 size 40 nbytes 40
>>>>>>>>>>>>>
>>>>>>>>>>>>> block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
>>>>>>>>>>>>>
>>>>>>>>>>>>> sequence 1501 flags 0x0(none)
>>>>>>>>>>>>>
>>>>>>>>>>>>> atime 1386484844.468769570 (2013-12-08 14:40:44)
>>>>>>>>>>>>>
>>>>>>>>>>>>> ctime 1386484844.468769570 (2013-12-08 14:40:44)
>>>>>>>>>>>>>
>>>>>>>>>>>>> mtime 1386484844.468769570 (2013-12-08 14:40:44)
>>>>>>>>>>>>>
>>>>>>>>>>>>> otime 0.0 (1970-01-01 08:00:00)
>>>>>>>>>>>>>
>>>>>>>>>>>>> item 1 key (5471 INODE_REF 4399) itemoff 3824 itemsize 11
>>>>>>>>>>>>>
>>>>>>>>>>>>> index 12 namelen 1 name: 8
>>>>>>>>>>>>>
>>>>>>>>>>>>> item 2 key (5471 EXTENT_DATA 0) itemoff 3763 itemsize 61
>>>>>>>>>>>>>
>>>>>>>>>>>>> generation 27726 type 0 (inline)
>>>>>>>>>>>>>
>>>>>>>>>>>>> inline extent data size 40 ram_bytes 40 compression 0 (none)
>>>>>>>>>>>>>
>>>>>>>>>>>>> item 3 key (5645 INODE_ITEM 0) itemoff 3603 itemsize 160
>>>>>>>>>>>>>
>>>>>>>>>>>>> generation 2542 transid 61261 size 40 nbytes 40
>>>>>>>>>>>>>
>>>>>>>>>>>>> block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
>>>>>>>>>>>>>
>>>>>>>>>>>>> sequence 24769 flags 0x0(none)
>>>>>>>>>>>>>
>>>>>>>>>>>>> atime 1394335806.351857522 (2014-03-09 11:30:06)
>>>>>>>>>>>>>
>>>>>>>>>>>>> ctime 1394335827.344389955 (2014-03-09 11:30:27)
>>>>>>>>>>>>>
>>>>>>>>>>>>> mtime 1394335827.344389955 (2014-03-09 11:30:27)
>>>>>>>>>>>>>
>>>>>>>>>>>>> otime 0.0 (1970-01-01 08:00:00)
>>>>>>>>>>>>>
>>>>>>>>>>>>> item 4 key (5645 INODE_REF 4399) itemoff 3592 itemsize 11
>>>>>>>>>>>>>
>>>>>>>>>>>>> index 13 namelen 1 name: 7
>>>>>>>>>>>>>
>>>>>>>>>>>>> item 5 key (5645 EXTENT_DATA 0) itemoff 3531 itemsize 61
>>>>>>>>>>>>>
>>>>>>>>>>>>> generation 61261 type 0 (inline)
>>>>>>>>>>>>>
>>>>>>>>>>>>> inline extent data size 40 ram_bytes 40 compression 0 (none)
>>>>>>>>>>>>>
>>>>>>>>>>>>> item 6 key (7222 INODE_ITEM 0) itemoff 3371 itemsize 160
>>>>>>>>>>>>>
>>>>>>>>>>>>> generation 5754 transid 5767 size 307 nbytes 307
>>>>>>>>>>>>>
>>>>>>>>>>>>> block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
>>>>>>>>>>>>>
>>>>>>>>>>>>> sequence 7 flags 0x0(none)
>>>>>>>>>>>>>
>>>>>>>>>>>>> atime 1379834835.428558020 (2013-09-22 15:27:15)
>>>>>>>>>>>>>
>>>>>>>>>>>>> ctime 1379834835.428558020 (2013-09-22 15:27:15)
>>>>>>>>>>>>>
>>>>>>>>>>>>> mtime 1379834835.428558020 (2013-09-22 15:27:15)
>>>>>>>>>>>>>
>>>>>>>>>>>>> otime 0.0 (1970-01-01 08:00:00)
>>>>>>>>>>>>>
>>>>>>>>>>>>> item 7 key (7222 INODE_REF 287) itemoff 3344 itemsize 27
>>>>>>>>>>>>>
>>>>>>>>>>>>> index 6 namelen 17 name: dhcpcd-eth0.lease
>>>>>>>>>>>>>
>>>>>>>>>>>>> item 8 key (7222 EXTENT_DATA 0) itemoff 3016 itemsize 328
>>>>>>>>>>>>>
>>>>>>>>>>>>> generation 5767 type 0 (inline)
>>>>>>>>>>>>>
>>>>>>>>>>>>> inline extent data size 307 ram_bytes 307 compression 0 (none)
>>>>>>>>>>>>>
>>>>>>>>>>>>> item 9 key (7415 INODE_ITEM 0) itemoff 2856 itemsize 160
>>>>>>>>>>>>>
>>>>>>>>>>>>> generation 5904 transid 1330906 size 180 nbytes 0
>>>>>>>>>>>>>
>>>>>>>>>>>>> block group 0 mode 40755 links 2 uid 0 gid 0 rdev 0
>>>>>>>>>>>>>
>>>>>>>>>>>>> sequence 177 flags 0x0(none)
>>>>>>>>>>>>>
>>>>>>>>>>>>> atime 1483277713.141980592 (2017-01-01 21:35:13)
>>>>>>>>>>>>>
>>>>>>>>>>>>> ctime 1563162901.234656246 (2019-07-15 11:55:01)
>>>>>>>>>>>>>
>>>>>>>>>>>>> mtime 1406534032.158605559 (2014-07-28 15:53:52)
>>>>>>>>>>>>>
>>>>>>>>>>>>> otime 0.0 (1970-01-01 08:00:00)
>>>>>>>>>>>>
>>>>>>>>>>>> This inode is indeed a directory.
>>>>>>>>>>>>
>>>>>>>>>>>> But it has two hard links, which is definitely something
>>>>>>>>>>>>
>>>>>>>>>>>> unexpected.
>>>>>>>>>>>>
>>>>>>>>>>>> Under Linux we shouldn't have any hardlink for directory, as it
>>>>>>>>>>>>
>>>>>>>>>>>> would
>>>>>>>>>>>>
>>>>>>>>>>>> easily lead to loops.
>>>>>>>>>>>>
>>>>>>>>>>>>> item 10 key (7415 INODE_REF 260) itemoff 2837 itemsize 19
>>>>>>>>>>>>>
>>>>>>>>>>>>> index 28 namelen 9 name: backlight
>>>>>>>>>>>>
>>>>>>>>>>>> Its parent inode is 260 in the same root, with the name backlight.
>>>>>>>>>>>>
>>>>>>>>>>>>> item 11 key (7415 INODE_REF 286) itemoff 2818 itemsize 19
>>>>>>>>>>>>>
>>>>>>>>>>>>> index 3 namelen 9 name: backlight
>>>>>>>>>>>>
>>>>>>>>>>>> Another hardlink in inode 286, which is definitely a regular thing.
>>>>>>>>>>>>
>>>>>>>>>>>> Btrfs-progs lacks the ability to detect such problem, we need to
>>>>>>>>>>>>
>>>>>>>>>>>> enhance
>>>>>>>>>>>>
>>>>>>>>>>>> it first.
>>>>>>>>>>>>
>>>>>>>>>>>> But do you have any idea how this directory get created?
>>>>>>>>>>>>
>>>>>>>>>>>> It looks like the content of sysfs.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> Qu
>>>>>>>>>>>>
>>>>>>>>>>>>> item 12 key (7415 DIR_ITEM 3128336373) itemoff 2746 itemsize 72
>>>>>>>>>>>>>
>>>>>>>>>>>>> location key (120417 INODE_ITEM 0) type FILE
>>>>>>>>>>>>>
>>>>>>>>>>>>> transid 117279 data_len 0 name_len 42
>>>>>>>>>>>>>
>>>>>>>>>>>>> name: pci-0000:00:02.0:backlight:intel_backlight
>>>>>>>>>>>>>
>>>>>>>>>>>>> item 13 key (7415 DIR_ITEM 3218198317) itemoff 2705 itemsize 41
>>>>>>>>>>>>>
>>>>>>>>>>>>> location key (7487 INODE_ITEM 0) type FILE
>>>>>>>>>>>>>
>>>>>>>>>>>>> transid 5992 data_len 0 name_len 11
>>>>>>>>>>>>>
>>>>>>>>>>>>> name: acpi_video0
>>>>>>>>>>>>>
>>>>>>>>>>>>> item 14 key (7415 DIR_ITEM 3582254411) itemoff 2638 itemsize 67
>>>>>>>>>>>>>
>>>>>>>>>>>>> location key (55325 INODE_ITEM 0) type FILE
>>>>>>>>>>>>>
>>>>>>>>>>>>> transid 63351 data_len 0 name_len 37
>>>>>>>>>>>>>
>>>>>>>>>>>>> name: platform-VPC2004:00:backlight:ideapad
>>>>>>>>>>>>>
>>>>>>>>>>>>> item 15 key (7415 DIR_INDEX 2) itemoff 2597 itemsize 41
>>>>>>>>>>>>>
>>>>>>>>>>>>> location key (7487 INODE_ITEM 0) type FILE
>>>>>>>>>>>>>
>>>>>>>>>>>>> transid 5992 data_len 0 name_len 11
>>>>>>>>>>>>>
>>>>>>>>>>>>> name: acpi_video0
>>>>>>>>>>>>>
>>>>>>>>>>>>> item 16 key (7415 DIR_INDEX 4) itemoff 2530 itemsize 67
>>>>>>>>>>>>>
>>>>>>>>>>>>> location key (55325 INODE_ITEM 0) type FILE
>>>>>>>>>>>>>
>>>>>>>>>>>>> transid 63351 data_len 0 name_len 37
>>>>>>>>>>>>>
>>>>>>>>>>>>> name: platform-VPC2004:00:backlight:ideapad
>>>>>>>>>>>>>
>>>>>>>>>>>>> item 17 key (7415 DIR_INDEX 5) itemoff 2458 itemsize 72
>>>>>>>>>>>>>
>>>>>>>>>>>>> location key (120417 INODE_ITEM 0) type FILE
>>>>>>>>>>>>>
>>>>>>>>>>>>> transid 117279 data_len 0 name_len 42
>>>>>>>>>>>>>
>>>>>>>>>>>>> name: pci-0000:00:02.0:backlight:intel_backlight
>>>>>>>>>>>>>
>>>>>>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Saturday, July 17th, 2021 at 3:05 PM, Qu Wenruo
>>>>>>>>>>>>>
>>>>>>>>>>>>> quwenruo.btrfs@gmx.com wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 2021/7/17 上午9:45, pepperpoint@mb.ardentcoding.com wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I see this message on dmesg:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [ 2452.256756] BTRFS critical (device dm-0): corrupt leaf:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> root=363 block=174113599488 slot=9 ino=7415, invalid nlink:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> has 2 expect no more than 1 for dir
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [ 2452.256776] BTRFS error (device dm-0): block=174113599488
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> read time tree block corruption detected
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please provide the following dump:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> btrfs ins dump-tree -b 174113599488 /dev/dm-0
>>>>>>>>>>>>>> =============================================
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Qu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When I run btrfs scrub and btrfs check, no error was detected.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am running Linux 5.12.15-arch1-1 and btrfs-progs v5.12.1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> How should I fix this error?

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: Read time tree block corruption detected
       [not found]                         ` <162660074747.7.3300740266059717894.10221270@simplelogin.co>
@ 2021-07-18 10:34                           ` pepperpoint
       [not found]                           ` <162660447733.8.9782212603273165785.10222524@mb.ardentcoding.com>
  1 sibling, 0 replies; 75+ messages in thread
From: pepperpoint @ 2021-07-18 10:34 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Qu Wenruo, linux-btrfs

Hi Qu,

I boot an ISO with Linux 5.1.15, mount the filesystem, wait for a while and restart.

This command did not output anything when I boot the system and run it:
btrfs ins dump-tree -t extent /dev/dm-0 | grep 174113599488 -A 3

Checking the logs, I do not see the error anymore from the time I boot the system. I also ran btrfs scrub just in case and detected no errors.

I think the filesystem is now fixed?

Regards,
Lester

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Sunday, July 18th, 2021 at 5:32 PM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:

> On 2021/7/18 下午4:46, pepperpoint@mb.ardentcoding.com wrote:
>
> > Hi Qu,
> >
> > When I find which directory some of the filenames are located, they are under /var/lib.
> >
> > I had subvolume for /var which was created probably around 2018.
>
> Then it's possible by somehow we allowed that hardlink to directory.
>
> Not sure if it's a bug in VFS layer or in btrfs itself.
>
> But around 2019 (aka, v5.2 kernel), that check for refs of directory is
>
> introduced and at the same time, write-time tree checker is introduced.
>
> This means if the bug happens after v5.2 kernel, it will be rejected
>
> before submitting to disk.
>
> So the problem definitely happens before the install of v5.2 kernel.
>
> > I don't remember how I created this but I probably use rsync to copy the files from existing /var
> >
> > or created a snapshot of root and delete other files that is not under /var.
>
> But that's still pretty weird.
>
> > Around June, I tried to move the filesystem to another partition through btrfs device add and btrfs device remove
> >
> > but failed due to that error and was advised to use btrfs replace instead.
> >
> > Then at the beginning of this month, I reorganized it merging most of the /var content back to root
> >
> > and created subvolume for /var/lib/mysql and /var/lib/mongodb. I encountered an error when
> >
> > I copy some of the files through cp --reflink but I failed for /var/lib/mysql so I created a snapshot
> >
> > from /var and remove the extra files.
> >
> > This is also the time I saw the errors in the log. Before that, the errors was not in the log.
>
> At least, we should prevent such problem from reaching disk.
>
> If you reverted to older LTS kernel, using Arch Linux Archive, it would
>
> be possible to continue deleting the subvolume and solve the problem.
>
> After the root 363 get fully deleted, you can verify that tree block get
>
> deleted by the following command:
>
> btrfs ins dump-tree -t extent <device> | grep 174113599488 -A 3
> ===============================================================
>
> Which should show no output.
>
> Thanks,
>
> Qu
>
> > Regards,
> >
> > Lester
> >
> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> >
> > On Sunday, July 18th, 2021 at 3:27 PM, Qu Wenruo quwenruo.btrfs@gmx.com wrote:
> >
> > > Hi,
> > >
> > > BTW, it's really important for us to know how the directory is hardlinked.
> > >
> > > Thus I salvaged the filenames found in the half-dropped root 363.
> > >
> > > Since it may contain confidential info, I send the filename list to you
> > >
> > > off-list.
> > >
> > > If you can remind what the root 363 is used for, and any possible
> > >
> > > operations which may be involved in that subvolume, it's better to reply
> > >
> > > it to the mail list so that we can get some clue on the root cause.
> > >
> > > Thanks,
> > >
> > > Qu
> > >
> > > On 2021/7/18 下午3:15, Qu Wenruo wrote:
> > >
> > > > On 2021/7/18 下午1:26, pepperpoint@mb.ardentcoding.com wrote:
> > > >
> > > > > Hi Qu,
> > > > >
> > > > > May I know if there are any leads on this? What should I do for now?
> > > >
> > > > Sorry for the late reply.
> > > >
> > > > With the image dump, it's much easier to find what's going wrong.
> > > >
> > > > -   About root 363
> > > >
> > > >     It's an orphan root, thus user can't access it directly.
> > > >
> > > >
> > > > Furthermore, it's being dropped, thus "btrfs ins dump-tree -t 363"
> > > >
> > > > reports transid error, as part of the tree has already been dropped,
> > > >
> > > > and this is expected.
> > > >
> > > > So far your fs is still OK, except that reported error.
> > > >
> > > > -   About the offending tree block
> > > >
> > > >     The offending tree block only belongs to the delete subvolume 363,
> > > >
> > > >     thus it should be delete soon.
> > > >
> > > >
> > > > But unfortunately due to the corrupted content, it's unable to be
> > > >
> > > > deleted.
> > > >
> > > > For now, if you can re-compile btrfs module, we can workaround the
> > > >
> > > > problem by temporarily disable read-time tree-checker so that the
> > > >
> > > > deletion can continue, and after the root 363 get fully deleted, the
> > > >
> > > > problem should be gone.
> > > >
> > > > Or you can use older kernel, any kernel <= v5.1 should not have the
> > > >
> > > > enhanced check, thus can continue with the subvolume deletion.
> > > >
> > > > If you want to go through the re-compile path, the needed diff is
> > > >
> > > > attached
> > > >
> > > > Thanks,
> > > >
> > > > Qu
> > > >
> > > > > Regards,
> > > > >
> > > > > Lester
> > > > >
> > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > >
> > > > > On Saturday, July 17th, 2021 at 8:51 PM,
> > > > >
> > > > > pepperpoint@mb.ardentcoding.com wrote:
> > > > >
> > > > > > Hi Qu,
> > > > > >
> > > > > > I run btrfs ins dump-tree -t 363 unmounted but the same error
> > > > > >
> > > > > > appears. Rerunning btrfs check does not show any error.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Lester
> > > > > >
> > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > > >
> > > > > > On Saturday, July 17th, 2021 at 6:48 PM, Qu Wenruo -
> > > > > >
> > > > > > quwenruo.btrfs@gmx.com wrote:
> > > > > >
> > > > > > > On 2021/7/17 下午6:34, pepperpoint@mb.ardentcoding.com wrote:
> > > > > > >
> > > > > > > > Hi Qu,
> > > > > > > >
> > > > > > > > Unfortunately I cannot find subvolume 363
> > > > > > > >
> > > > > > > > btrfs subvolume list /run/media/root
> > > > > > > > ====================================
> > > > > > > >
> > > > > > > > ID 361 gen 1814826 top level 584 path @/live/snapshot
> > > > > > > >
> > > > > > > > ID 364 gen 1814414 top level 5 path @vtmp/live/snapshot
> > > > > > > >
> > > > > > > > ID 369 gen 1814414 top level 5 path @vlmachines/live/snapshot
> > > > > > > >
> > > > > > > > ID 493 gen 1814414 top level 5 path @vlportables/live/snapshot
> > > > > > > >
> > > > > > > > ID 579 gen 1814828 top level 5 path @vlog/live/snapshot
> > > > > > > >
> > > > > > > > ID 580 gen 1814414 top level 5 path @vcache/live/snapshot
> > > > > > > >
> > > > > > > > ID 581 gen 1814414 top level 5 path @vlmongodb/live/snapshot
> > > > > > > >
> > > > > > > > ID 582 gen 1814414 top level 5 path @vlmysql/live/snapshot
> > > > > > > >
> > > > > > > > ID 583 gen 1814414 top level 5 path @vspool/live/snapshot
> > > > > > > >
> > > > > > > > ID 584 gen 1814414 top level 5 path @
> > > > > > > >
> > > > > > > > ID 598 gen 1813420 top level 584 path @/4/snapshot
> > > > > > >
> > > > > > > Maybe 363 is some subvolume get deleted and later snapshot of it still
> > > > > > >
> > > > > > > exists.
> > > > > > >
> > > > > > > This will be harder to debug.
> > > > > > >
> > > > > > > Can you take a btrfs-image dump of your filesystem? (needs to be taken
> > > > > > >
> > > > > > > with the fs unmounted).
> > > > > > >
> > > > > > > The dumped image will contain your metadata, including file names and
> > > > > > >
> > > > > > > directory structures, but no data inside those files.
> > > > > > >
> > > > > > > Although btrfs-image has "-s" option to mask the filenames, but
> > > > > > >
> > > > > > > considering the filename in this case is useful to locate the inode, I
> > > > > > >
> > > > > > > guess it's better to take the image without any "-s" option.
> > > > > > >
> > > > > > > > btrfs ins dump-tree -t 363 /dev/dm-0 | grep -A 5 "(286 "
> > > > > > > > ========================================================
> > > > > > > >
> > > > > > > > parent transid verify failed on 174170742784 wanted 1789655 found
> > > > > > > >
> > > > > > > > 1812621
> > > > > > > >
> > > > > > > > parent transid verify failed on 174170742784 wanted 1789655 found
> > > > > > > >
> > > > > > > > 1812621
> > > > > > > >
> > > > > > > > parent transid verify failed on 174170742784 wanted 1789655 found
> > > > > > > >
> > > > > > > > 1812621
> > > > > > > >
> > > > > > > > Ignoring transid failure
> > > > > > > >
> > > > > > > > ERROR: child eb corrupted: parent bytenr=174170738688 item=0 parent
> > > > > > > >
> > > > > > > > level=2 child bytenr=174170742784 child level=0
> > > > > > >
> > > > > > > This transid mismatch may be a problem when running dump-tree on
> > > > > > >
> > > > > > > mounted
> > > > > > >
> > > > > > > fs, since you mentioned btrfs check reported no error, there shouldn't
> > > > > > >
> > > > > > > be a transid mismatch error.
> > > > > > >
> > > > > > > Anyway, if you can upload the btrfs-image dump, it would be much easier
> > > > > > >
> > > > > > > for us to debug and find out what's really going.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Qu
> > > > > > >
> > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > > > > >
> > > > > > > > On Saturday, July 17th, 2021 at 6:12 PM, Qu Wenruo wqu@suse.com wrote:
> > > > > > > >
> > > > > > > > > On 2021/7/17 下午4:57, pepperpoint@mb.ardentcoding.com wrote:
> > > > > > > > >
> > > > > > > > > > Hi Qu,
> > > > > > > > > >
> > > > > > > > > > I don't know how the directory was created but last month, I used
> > > > > > > > > >
> > > > > > > > > > btrfs device add and btrfs device remove to move the filesystem
> > > > > > > > > >
> > > > > > > > > > from one partition to another. It failed because of the same
> > > > > > > > > >
> > > > > > > > > > error and was advised to use btrfs replace instead. I don't know
> > > > > > > > > >
> > > > > > > > > > if the error also happened before I move the file system as I
> > > > > > > > > >
> > > > > > > > > > don't have any previous logs.
> > > > > > > > >
> > > > > > > > > It definitely happens before you moving the fs.
> > > > > > > > >
> > > > > > > > > As regular dev replacing/add/move only relocates the metadata, but
> > > > > > > > >
> > > > > > > > > not
> > > > > > > > >
> > > > > > > > > touching the fs trees.
> > > > > > > > >
> > > > > > > > > > Here is the result when I search for the inodes you mentioned if
> > > > > > > > > >
> > > > > > > > > > it helps:
> > > > > > > > > >
> > > > > > > > > > find /run/media/root -inum 260 -exec ls -ldi {} \;
> > > > > > > > > > ==================================================
> > > > > > > > > >
> > > > > > > > > > 260 -rw-r--r-- 1 root root 36864 Jun 25 06:22
> > > > > > > > > >
> > > > > > > > > > /run/media/root/@vcache/live/snapshot/app-info/cache/en_US.cache
> > > > > > > > > >
> > > > > > > > > > 260 drwx------ 1 mongodb mongodb 136 Sep 12 2020
> > > > > > > > > >
> > > > > > > > > > /run/media/root/@vlmongodb/live/snapshot/diagnostic.data
> > > > > > > > > >
> > > > > > > > > > 260 -rw-rw---- 1 mysql mysql 50331648 Sep 13 2015
> > > > > > > > > >
> > > > > > > > > > /run/media/root/@vlmysql/live/snapshot/ib_logfile0
> > > > > > > > > >
> > > > > > > > > > 260 -rw-r----- 1 root lp 8641 Mar 5 2014
> > > > > > > > > >
> > > > > > > > > > /run/media/root/@vspool/live/snapshot/cups/d00001-001
> > > > > > > > > >
> > > > > > > > > > 260 dr-xr-xr-x 1 root root 0 Sep 13 2013
> > > > > > > > > >
> > > > > > > > > > /run/media/root/@/live/snapshot/sys
> > > > > > > > > >
> > > > > > > > > > 260 dr-xr-xr-x 1 root root 0 Sep 13 2013
> > > > > > > > > >
> > > > > > > > > > /run/media/root/@/4/snapshot/sys
> > > > > > > > >
> > > > > > > > > Since btrfs can have the same inode number inside different
> > > > > > > > >
> > > > > > > > > subvolumes,
> > > > > > > > >
> > > > > > > > > you may want to limit the search inside subvolume 363.
> > > > > > > > >
> > > > > > > > > "-mount" option of find can do that, you only need to locate
> > > > > > > > >
> > > > > > > > > subvolume
> > > > > > > > >
> > > > > > > > > 363 by "btrfs subv list".
> > > > > > > > >
> > > > > > > > > But from these output I guess above two "sys" directory are more
> > > > > > > > >
> > > > > > > > > possible.
> > > > > > > > >
> > > > > > > > > Is there any directory named "blaklight" inside those directory?
> > > > > > > > >
> > > > > > > > > > find /run/media/root -inum 286 -exec ls -ldi {} \;
> > > > > > > > > > ==================================================
> > > > > > > > > >
> > > > > > > > > > 286 -rw-r--r-- 1 root root 96 Aug 16 2015
> > > > > > > > > >
> > > > > > > > > > /run/media/root/@vcache/live/snapshot/fontconfig/4b172ca7f111e3cffadc3636415fead9-le64.cache-4
> > > > > > > > > >
> > > > > > > > > > 286 -rw-rw---- 1 mysql mysql 4096 Sep 15 2013
> > > > > > > > > >
> > > > > > > > > > /run/media/root/@vlmysql/live/snapshot/mysql/columns_priv.MYI
> > > > > > > > > >
> > > > > > > > > > 286 -rw-r-----+ 1 root systemd-journal 16777216 Jul 4 01:14
> > > > > > > > > >
> > > > > > > > > > /run/media/root/@vlog/live/snapshot/journal/5098dd7845ae46d3ba1826c68a809a7c/user-1000@fbd9f65d0ea349f6b996716280e6c4dd-00000000002314c5-0005c5cb84a3a438.journal
> > > > > > > > >
> > > > > > > > > This is interesting, it means the inode 286 is not accessible.
> > > > > > > > >
> > > > > > > > > It can be some orphan inode, but would you dump subvolume 363 then
> > > > > > > > >
> > > > > > > > > try
> > > > > > > > >
> > > > > > > > > to locate the inode 286?
> > > > > > > > >
> > > > > > > > > One example command would be:
> > > > > > > > >
> > > > > > > > > btrfs ins dump-tree -t 363 <dev> | grep -A 5 "(286 "
> > > > > > > > > ====================================================
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Qu
> > > > > > > > >
> > > > > > > > > > Directories with pattern /root/@<dir>/live/snapshot/ are
> > > > > > > > > >
> > > > > > > > > > subvolumes and directories with pattern
> > > > > > > > > >
> > > > > > > > > > /root/@<dir>/<num>/snapshot/ are snapshots of live.
> > > > > > > > > >
> > > > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > > > > > > >
> > > > > > > > > > On Saturday, July 17th, 2021 at 4:14 PM, Qu Wenruo
> > > > > > > > > >
> > > > > > > > > > quwenruo.btrfs@gmx.com wrote:
> > > > > > > > > >
> > > > > > > > > > > On 2021/7/17 下午3:51, pepperpoint@mb.ardentcoding.com wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Qu,
> > > > > > > > > > > >
> > > > > > > > > > > > Please see below for the dump.
> > > > > > > > > > > >
> > > > > > > > > > > > btrfs-progs v5.12.1
> > > > > > > > > > > >
> > > > > > > > > > > > leaf 174113599488 items 18 free space 2008 generation 1330906
> > > > > > > > > > > >
> > > > > > > > > > > > owner 363
> > > > > > > > > > > >
> > > > > > > > > > > > leaf 174113599488 flags 0x1(WRITTEN) backref revision 1
> > > > > > > > > > > >
> > > > > > > > > > > > fs uuid a7d327c4-8594-4116-a6f8-8aa2a4162063
> > > > > > > > > > > >
> > > > > > > > > > > > chunk uuid f885f49e-14a0-4c80-9c12-c2302b9a0229
> > > > > > > > > > > >
> > > > > > > > > > > > item 0 key (5471 INODE_ITEM 0) itemoff 3835 itemsize 160
> > > > > > > > > > > >
> > > > > > > > > > > > generation 2063 transid 27726 size 40 nbytes 40
> > > > > > > > > > > >
> > > > > > > > > > > > block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
> > > > > > > > > > > >
> > > > > > > > > > > > sequence 1501 flags 0x0(none)
> > > > > > > > > > > >
> > > > > > > > > > > > atime 1386484844.468769570 (2013-12-08 14:40:44)
> > > > > > > > > > > >
> > > > > > > > > > > > ctime 1386484844.468769570 (2013-12-08 14:40:44)
> > > > > > > > > > > >
> > > > > > > > > > > > mtime 1386484844.468769570 (2013-12-08 14:40:44)
> > > > > > > > > > > >
> > > > > > > > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > > > > > > > >
> > > > > > > > > > > > item 1 key (5471 INODE_REF 4399) itemoff 3824 itemsize 11
> > > > > > > > > > > >
> > > > > > > > > > > > index 12 namelen 1 name: 8
> > > > > > > > > > > >
> > > > > > > > > > > > item 2 key (5471 EXTENT_DATA 0) itemoff 3763 itemsize 61
> > > > > > > > > > > >
> > > > > > > > > > > > generation 27726 type 0 (inline)
> > > > > > > > > > > >
> > > > > > > > > > > > inline extent data size 40 ram_bytes 40 compression 0 (none)
> > > > > > > > > > > >
> > > > > > > > > > > > item 3 key (5645 INODE_ITEM 0) itemoff 3603 itemsize 160
> > > > > > > > > > > >
> > > > > > > > > > > > generation 2542 transid 61261 size 40 nbytes 40
> > > > > > > > > > > >
> > > > > > > > > > > > block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
> > > > > > > > > > > >
> > > > > > > > > > > > sequence 24769 flags 0x0(none)
> > > > > > > > > > > >
> > > > > > > > > > > > atime 1394335806.351857522 (2014-03-09 11:30:06)
> > > > > > > > > > > >
> > > > > > > > > > > > ctime 1394335827.344389955 (2014-03-09 11:30:27)
> > > > > > > > > > > >
> > > > > > > > > > > > mtime 1394335827.344389955 (2014-03-09 11:30:27)
> > > > > > > > > > > >
> > > > > > > > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > > > > > > > >
> > > > > > > > > > > > item 4 key (5645 INODE_REF 4399) itemoff 3592 itemsize 11
> > > > > > > > > > > >
> > > > > > > > > > > > index 13 namelen 1 name: 7
> > > > > > > > > > > >
> > > > > > > > > > > > item 5 key (5645 EXTENT_DATA 0) itemoff 3531 itemsize 61
> > > > > > > > > > > >
> > > > > > > > > > > > generation 61261 type 0 (inline)
> > > > > > > > > > > >
> > > > > > > > > > > > inline extent data size 40 ram_bytes 40 compression 0 (none)
> > > > > > > > > > > >
> > > > > > > > > > > > item 6 key (7222 INODE_ITEM 0) itemoff 3371 itemsize 160
> > > > > > > > > > > >
> > > > > > > > > > > > generation 5754 transid 5767 size 307 nbytes 307
> > > > > > > > > > > >
> > > > > > > > > > > > block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
> > > > > > > > > > > >
> > > > > > > > > > > > sequence 7 flags 0x0(none)
> > > > > > > > > > > >
> > > > > > > > > > > > atime 1379834835.428558020 (2013-09-22 15:27:15)
> > > > > > > > > > > >
> > > > > > > > > > > > ctime 1379834835.428558020 (2013-09-22 15:27:15)
> > > > > > > > > > > >
> > > > > > > > > > > > mtime 1379834835.428558020 (2013-09-22 15:27:15)
> > > > > > > > > > > >
> > > > > > > > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > > > > > > > >
> > > > > > > > > > > > item 7 key (7222 INODE_REF 287) itemoff 3344 itemsize 27
> > > > > > > > > > > >
> > > > > > > > > > > > index 6 namelen 17 name: dhcpcd-eth0.lease
> > > > > > > > > > > >
> > > > > > > > > > > > item 8 key (7222 EXTENT_DATA 0) itemoff 3016 itemsize 328
> > > > > > > > > > > >
> > > > > > > > > > > > generation 5767 type 0 (inline)
> > > > > > > > > > > >
> > > > > > > > > > > > inline extent data size 307 ram_bytes 307 compression 0 (none)
> > > > > > > > > > > >
> > > > > > > > > > > > item 9 key (7415 INODE_ITEM 0) itemoff 2856 itemsize 160
> > > > > > > > > > > >
> > > > > > > > > > > > generation 5904 transid 1330906 size 180 nbytes 0
> > > > > > > > > > > >
> > > > > > > > > > > > block group 0 mode 40755 links 2 uid 0 gid 0 rdev 0
> > > > > > > > > > > >
> > > > > > > > > > > > sequence 177 flags 0x0(none)
> > > > > > > > > > > >
> > > > > > > > > > > > atime 1483277713.141980592 (2017-01-01 21:35:13)
> > > > > > > > > > > >
> > > > > > > > > > > > ctime 1563162901.234656246 (2019-07-15 11:55:01)
> > > > > > > > > > > >
> > > > > > > > > > > > mtime 1406534032.158605559 (2014-07-28 15:53:52)
> > > > > > > > > > > >
> > > > > > > > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > > > > > > >
> > > > > > > > > > > This inode is indeed a directory.
> > > > > > > > > > >
> > > > > > > > > > > But it has two hard links, which is definitely something
> > > > > > > > > > >
> > > > > > > > > > > unexpected.
> > > > > > > > > > >
> > > > > > > > > > > Under Linux we shouldn't have any hardlink for directory, as it
> > > > > > > > > > >
> > > > > > > > > > > would
> > > > > > > > > > >
> > > > > > > > > > > easily lead to loops.
> > > > > > > > > > >
> > > > > > > > > > > > item 10 key (7415 INODE_REF 260) itemoff 2837 itemsize 19
> > > > > > > > > > > >
> > > > > > > > > > > > index 28 namelen 9 name: backlight
> > > > > > > > > > >
> > > > > > > > > > > Its parent inode is 260 in the same root, with the name backlight.
> > > > > > > > > > >
> > > > > > > > > > > > item 11 key (7415 INODE_REF 286) itemoff 2818 itemsize 19
> > > > > > > > > > > >
> > > > > > > > > > > > index 3 namelen 9 name: backlight
> > > > > > > > > > >
> > > > > > > > > > > Another hardlink in inode 286, which is definitely a regular thing.
> > > > > > > > > > >
> > > > > > > > > > > Btrfs-progs lacks the ability to detect such problem, we need to
> > > > > > > > > > >
> > > > > > > > > > > enhance
> > > > > > > > > > >
> > > > > > > > > > > it first.
> > > > > > > > > > >
> > > > > > > > > > > But do you have any idea how this directory get created?
> > > > > > > > > > >
> > > > > > > > > > > It looks like the content of sysfs.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Qu
> > > > > > > > > > >
> > > > > > > > > > > > item 12 key (7415 DIR_ITEM 3128336373) itemoff 2746 itemsize 72
> > > > > > > > > > > >
> > > > > > > > > > > > location key (120417 INODE_ITEM 0) type FILE
> > > > > > > > > > > >
> > > > > > > > > > > > transid 117279 data_len 0 name_len 42
> > > > > > > > > > > >
> > > > > > > > > > > > name: pci-0000:00:02.0:backlight:intel_backlight
> > > > > > > > > > > >
> > > > > > > > > > > > item 13 key (7415 DIR_ITEM 3218198317) itemoff 2705 itemsize 41
> > > > > > > > > > > >
> > > > > > > > > > > > location key (7487 INODE_ITEM 0) type FILE
> > > > > > > > > > > >
> > > > > > > > > > > > transid 5992 data_len 0 name_len 11
> > > > > > > > > > > >
> > > > > > > > > > > > name: acpi_video0
> > > > > > > > > > > >
> > > > > > > > > > > > item 14 key (7415 DIR_ITEM 3582254411) itemoff 2638 itemsize 67
> > > > > > > > > > > >
> > > > > > > > > > > > location key (55325 INODE_ITEM 0) type FILE
> > > > > > > > > > > >
> > > > > > > > > > > > transid 63351 data_len 0 name_len 37
> > > > > > > > > > > >
> > > > > > > > > > > > name: platform-VPC2004:00:backlight:ideapad
> > > > > > > > > > > >
> > > > > > > > > > > > item 15 key (7415 DIR_INDEX 2) itemoff 2597 itemsize 41
> > > > > > > > > > > >
> > > > > > > > > > > > location key (7487 INODE_ITEM 0) type FILE
> > > > > > > > > > > >
> > > > > > > > > > > > transid 5992 data_len 0 name_len 11
> > > > > > > > > > > >
> > > > > > > > > > > > name: acpi_video0
> > > > > > > > > > > >
> > > > > > > > > > > > item 16 key (7415 DIR_INDEX 4) itemoff 2530 itemsize 67
> > > > > > > > > > > >
> > > > > > > > > > > > location key (55325 INODE_ITEM 0) type FILE
> > > > > > > > > > > >
> > > > > > > > > > > > transid 63351 data_len 0 name_len 37
> > > > > > > > > > > >
> > > > > > > > > > > > name: platform-VPC2004:00:backlight:ideapad
> > > > > > > > > > > >
> > > > > > > > > > > > item 17 key (7415 DIR_INDEX 5) itemoff 2458 itemsize 72
> > > > > > > > > > > >
> > > > > > > > > > > > location key (120417 INODE_ITEM 0) type FILE
> > > > > > > > > > > >
> > > > > > > > > > > > transid 117279 data_len 0 name_len 42
> > > > > > > > > > > >
> > > > > > > > > > > > name: pci-0000:00:02.0:backlight:intel_backlight
> > > > > > > > > > > >
> > > > > > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > > > > > > > > >
> > > > > > > > > > > > On Saturday, July 17th, 2021 at 3:05 PM, Qu Wenruo
> > > > > > > > > > > >
> > > > > > > > > > > > quwenruo.btrfs@gmx.com wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > On 2021/7/17 上午9:45, pepperpoint@mb.ardentcoding.com wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hello,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I see this message on dmesg:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [ 2452.256756] BTRFS critical (device dm-0): corrupt leaf:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > root=363 block=174113599488 slot=9 ino=7415, invalid nlink:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > has 2 expect no more than 1 for dir
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [ 2452.256776] BTRFS error (device dm-0): block=174113599488
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > read time tree block corruption detected
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please provide the following dump:
> > > > > > > > > > > > >
> > > > > > > > > > > > > btrfs ins dump-tree -b 174113599488 /dev/dm-0
> > > > > > > > > > > > > =============================================
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Qu
> > > > > > > > > > > > >
> > > > > > > > > > > > > > When I run btrfs scrub and btrfs check, no error was detected.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I am running Linux 5.12.15-arch1-1 and btrfs-progs v5.12.1
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > How should I fix this error?

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: Read time tree block corruption detected
       [not found]                       ` <162659797656.6.14385932343235367381.10220447@mb.ardentcoding.com>
@ 2021-07-18  9:32                         ` Qu Wenruo
       [not found]                         ` <162660074747.7.3300740266059717894.10221270@simplelogin.co>
  1 sibling, 0 replies; 75+ messages in thread
From: Qu Wenruo @ 2021-07-18  9:32 UTC (permalink / raw)
  To: pepperpoint; +Cc: Qu Wenruo, linux-btrfs



On 2021/7/18 下午4:46, pepperpoint@mb.ardentcoding.com wrote:
> Hi Qu,
>
> When I find which directory some of the filenames are located, they are under /var/lib.
> I had subvolume for /var which was created probably around 2018.

Then it's possible by somehow we allowed that hardlink to directory.

Not sure if it's a bug in VFS layer or in btrfs itself.

But around 2019 (aka, v5.2 kernel), that check for refs of directory is
introduced and at the same time, write-time tree checker is introduced.

This means if the bug happens after v5.2 kernel, it will be rejected
before submitting to disk.

So the problem definitely happens before the install of v5.2 kernel.

>
> I don't remember how I created this but I probably use rsync to copy the files from existing /var
> or created a snapshot of root and delete other files that is not under /var.

But that's still pretty weird.

>
> Around June, I tried to move the filesystem to another partition through btrfs device add and btrfs device remove
> but failed due to that error and was advised to use btrfs replace instead.
>
> Then at the beginning of this month, I reorganized it merging most of the /var content back to root
> and created subvolume for /var/lib/mysql and /var/lib/mongodb. I encountered an error when
> I copy some of the files through cp --reflink but I failed for /var/lib/mysql so I created a snapshot
> from /var and remove the extra files.
>
> This is also the time I saw the errors in the log. Before that, the errors was not in the log.

At least, we should prevent such problem from reaching disk.

If you reverted to older LTS kernel, using Arch Linux Archive, it would
be possible to continue deleting the subvolume and solve the problem.

After the root 363 get fully deleted, you can verify that tree block get
deleted by the following command:

# btrfs ins dump-tree -t extent <device> | grep 174113599488 -A 3

Which should show no output.

Thanks,
Qu

>
> Regards,
> Lester
>
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>
> On Sunday, July 18th, 2021 at 3:27 PM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>> Hi,
>>
>> BTW, it's really important for us to know how the directory is hardlinked.
>>
>> Thus I salvaged the filenames found in the half-dropped root 363.
>>
>> Since it may contain confidential info, I send the filename list to you
>>
>> off-list.
>>
>> If you can remind what the root 363 is used for, and any possible
>>
>> operations which may be involved in that subvolume, it's better to reply
>>
>> it to the mail list so that we can get some clue on the root cause.
>>
>> Thanks,
>>
>> Qu
>>
>> On 2021/7/18 下午3:15, Qu Wenruo wrote:
>>
>>> On 2021/7/18 下午1:26, pepperpoint@mb.ardentcoding.com wrote:
>>>
>>>> Hi Qu,
>>>>
>>>> May I know if there are any leads on this? What should I do for now?
>>>
>>> Sorry for the late reply.
>>>
>>> With the image dump, it's much easier to find what's going wrong.
>>>
>>> -   About root 363
>>>
>>>        It's an orphan root, thus user can't access it directly.
>>>
>>> Furthermore, it's being dropped, thus "btrfs ins dump-tree -t 363"
>>>
>>>    reports transid error, as part of the tree has already been dropped,
>>>
>>>    and this is expected.
>>>
>>> So far your fs is still OK, except that reported error.
>>>
>>> -   About the offending tree block
>>>
>>>        The offending tree block only belongs to the delete subvolume 363,
>>>
>>>        thus it should be delete soon.
>>>
>>> But unfortunately due to the corrupted content, it's unable to be
>>>
>>>    deleted.
>>>
>>> For now, if you can re-compile btrfs module, we can workaround the
>>>
>>>    problem by temporarily disable read-time tree-checker so that the
>>>
>>>    deletion can continue, and after the root 363 get fully deleted, the
>>>
>>>    problem should be gone.
>>>
>>> Or you can use older kernel, any kernel <= v5.1 should not have the
>>>
>>>    enhanced check, thus can continue with the subvolume deletion.
>>>
>>> If you want to go through the re-compile path, the needed diff is
>>>
>>>    attached
>>>
>>> Thanks,
>>>
>>> Qu
>>>
>>>> Regards,
>>>>
>>>> Lester
>>>>
>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>
>>>> On Saturday, July 17th, 2021 at 8:51 PM,
>>>>
>>>> pepperpoint@mb.ardentcoding.com wrote:
>>>>
>>>>> Hi Qu,
>>>>>
>>>>> I run btrfs ins dump-tree -t 363 unmounted but the same error
>>>>>
>>>>> appears. Rerunning btrfs check does not show any error.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Lester
>>>>>
>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>
>>>>> On Saturday, July 17th, 2021 at 6:48 PM, Qu Wenruo -
>>>>>
>>>>> quwenruo.btrfs@gmx.com wrote:
>>>>>
>>>>>> On 2021/7/17 下午6:34, pepperpoint@mb.ardentcoding.com wrote:
>>>>>>
>>>>>>> Hi Qu,
>>>>>>>
>>>>>>> Unfortunately I cannot find subvolume 363
>>>>>>>
>>>>>>> btrfs subvolume list /run/media/root
>>>>>>> ====================================
>>>>>>>
>>>>>>> ID 361 gen 1814826 top level 584 path @/live/snapshot
>>>>>>>
>>>>>>> ID 364 gen 1814414 top level 5 path @vtmp/live/snapshot
>>>>>>>
>>>>>>> ID 369 gen 1814414 top level 5 path @vlmachines/live/snapshot
>>>>>>>
>>>>>>> ID 493 gen 1814414 top level 5 path @vlportables/live/snapshot
>>>>>>>
>>>>>>> ID 579 gen 1814828 top level 5 path @vlog/live/snapshot
>>>>>>>
>>>>>>> ID 580 gen 1814414 top level 5 path @vcache/live/snapshot
>>>>>>>
>>>>>>> ID 581 gen 1814414 top level 5 path @vlmongodb/live/snapshot
>>>>>>>
>>>>>>> ID 582 gen 1814414 top level 5 path @vlmysql/live/snapshot
>>>>>>>
>>>>>>> ID 583 gen 1814414 top level 5 path @vspool/live/snapshot
>>>>>>>
>>>>>>> ID 584 gen 1814414 top level 5 path @
>>>>>>>
>>>>>>> ID 598 gen 1813420 top level 584 path @/4/snapshot
>>>>>>
>>>>>> Maybe 363 is some subvolume get deleted and later snapshot of it still
>>>>>>
>>>>>> exists.
>>>>>>
>>>>>> This will be harder to debug.
>>>>>>
>>>>>> Can you take a btrfs-image dump of your filesystem? (needs to be taken
>>>>>>
>>>>>> with the fs unmounted).
>>>>>>
>>>>>> The dumped image will contain your metadata, including file names and
>>>>>>
>>>>>> directory structures, but no data inside those files.
>>>>>>
>>>>>> Although btrfs-image has "-s" option to mask the filenames, but
>>>>>>
>>>>>> considering the filename in this case is useful to locate the inode, I
>>>>>>
>>>>>> guess it's better to take the image without any "-s" option.
>>>>>>
>>>>>>> btrfs ins dump-tree -t 363 /dev/dm-0 | grep -A 5 "(286 "
>>>>>>> ========================================================
>>>>>>>
>>>>>>> parent transid verify failed on 174170742784 wanted 1789655 found
>>>>>>>
>>>>>>> 1812621
>>>>>>>
>>>>>>> parent transid verify failed on 174170742784 wanted 1789655 found
>>>>>>>
>>>>>>> 1812621
>>>>>>>
>>>>>>> parent transid verify failed on 174170742784 wanted 1789655 found
>>>>>>>
>>>>>>> 1812621
>>>>>>>
>>>>>>> Ignoring transid failure
>>>>>>>
>>>>>>> ERROR: child eb corrupted: parent bytenr=174170738688 item=0 parent
>>>>>>>
>>>>>>> level=2 child bytenr=174170742784 child level=0
>>>>>>
>>>>>> This transid mismatch may be a problem when running dump-tree on
>>>>>>
>>>>>> mounted
>>>>>>
>>>>>> fs, since you mentioned btrfs check reported no error, there shouldn't
>>>>>>
>>>>>> be a transid mismatch error.
>>>>>>
>>>>>> Anyway, if you can upload the btrfs-image dump, it would be much easier
>>>>>>
>>>>>> for us to debug and find out what's really going.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Qu
>>>>>>
>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>
>>>>>>> On Saturday, July 17th, 2021 at 6:12 PM, Qu Wenruo wqu@suse.com wrote:
>>>>>>>
>>>>>>>> On 2021/7/17 下午4:57, pepperpoint@mb.ardentcoding.com wrote:
>>>>>>>>
>>>>>>>>> Hi Qu,
>>>>>>>>>
>>>>>>>>> I don't know how the directory was created but last month, I used
>>>>>>>>>
>>>>>>>>> btrfs device add and btrfs device remove to move the filesystem
>>>>>>>>>
>>>>>>>>> from one partition to another. It failed because of the same
>>>>>>>>>
>>>>>>>>> error and was advised to use btrfs replace instead. I don't know
>>>>>>>>>
>>>>>>>>> if the error also happened before I move the file system as I
>>>>>>>>>
>>>>>>>>> don't have any previous logs.
>>>>>>>>
>>>>>>>> It definitely happens before you moving the fs.
>>>>>>>>
>>>>>>>> As regular dev replacing/add/move only relocates the metadata, but
>>>>>>>>
>>>>>>>> not
>>>>>>>>
>>>>>>>> touching the fs trees.
>>>>>>>>
>>>>>>>>> Here is the result when I search for the inodes you mentioned if
>>>>>>>>>
>>>>>>>>> it helps:
>>>>>>>>>
>>>>>>>>> find /run/media/root -inum 260 -exec ls -ldi {} \;
>>>>>>>>> ==================================================
>>>>>>>>>
>>>>>>>>> 260 -rw-r--r-- 1 root root 36864 Jun 25 06:22
>>>>>>>>>
>>>>>>>>> /run/media/root/@vcache/live/snapshot/app-info/cache/en_US.cache
>>>>>>>>>
>>>>>>>>> 260 drwx------ 1 mongodb mongodb 136 Sep 12 2020
>>>>>>>>>
>>>>>>>>> /run/media/root/@vlmongodb/live/snapshot/diagnostic.data
>>>>>>>>>
>>>>>>>>> 260 -rw-rw---- 1 mysql mysql 50331648 Sep 13 2015
>>>>>>>>>
>>>>>>>>> /run/media/root/@vlmysql/live/snapshot/ib_logfile0
>>>>>>>>>
>>>>>>>>> 260 -rw-r----- 1 root lp 8641 Mar 5 2014
>>>>>>>>>
>>>>>>>>> /run/media/root/@vspool/live/snapshot/cups/d00001-001
>>>>>>>>>
>>>>>>>>> 260 dr-xr-xr-x 1 root root 0 Sep 13 2013
>>>>>>>>>
>>>>>>>>> /run/media/root/@/live/snapshot/sys
>>>>>>>>>
>>>>>>>>> 260 dr-xr-xr-x 1 root root 0 Sep 13 2013
>>>>>>>>>
>>>>>>>>> /run/media/root/@/4/snapshot/sys
>>>>>>>>
>>>>>>>> Since btrfs can have the same inode number inside different
>>>>>>>>
>>>>>>>> subvolumes,
>>>>>>>>
>>>>>>>> you may want to limit the search inside subvolume 363.
>>>>>>>>
>>>>>>>> "-mount" option of find can do that, you only need to locate
>>>>>>>>
>>>>>>>> subvolume
>>>>>>>>
>>>>>>>> 363 by "btrfs subv list".
>>>>>>>>
>>>>>>>> But from these output I guess above two "sys" directory are more
>>>>>>>>
>>>>>>>> possible.
>>>>>>>>
>>>>>>>> Is there any directory named "blaklight" inside those directory?
>>>>>>>>
>>>>>>>>> find /run/media/root -inum 286 -exec ls -ldi {} \;
>>>>>>>>> ==================================================
>>>>>>>>>
>>>>>>>>> 286 -rw-r--r-- 1 root root 96 Aug 16 2015
>>>>>>>>>
>>>>>>>>> /run/media/root/@vcache/live/snapshot/fontconfig/4b172ca7f111e3cffadc3636415fead9-le64.cache-4
>>>>>>>>>
>>>>>>>>> 286 -rw-rw---- 1 mysql mysql 4096 Sep 15 2013
>>>>>>>>>
>>>>>>>>> /run/media/root/@vlmysql/live/snapshot/mysql/columns_priv.MYI
>>>>>>>>>
>>>>>>>>> 286 -rw-r-----+ 1 root systemd-journal 16777216 Jul 4 01:14
>>>>>>>>>
>>>>>>>>> /run/media/root/@vlog/live/snapshot/journal/5098dd7845ae46d3ba1826c68a809a7c/user-1000@fbd9f65d0ea349f6b996716280e6c4dd-00000000002314c5-0005c5cb84a3a438.journal
>>>>>>>>
>>>>>>>> This is interesting, it means the inode 286 is not accessible.
>>>>>>>>
>>>>>>>> It can be some orphan inode, but would you dump subvolume 363 then
>>>>>>>>
>>>>>>>> try
>>>>>>>>
>>>>>>>> to locate the inode 286?
>>>>>>>>
>>>>>>>> One example command would be:
>>>>>>>>
>>>>>>>> btrfs ins dump-tree -t 363 <dev> | grep -A 5 "(286 "
>>>>>>>> ====================================================
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Qu
>>>>>>>>
>>>>>>>>> Directories with pattern /root/@<dir>/live/snapshot/ are
>>>>>>>>>
>>>>>>>>> subvolumes and directories with pattern
>>>>>>>>>
>>>>>>>>> /root/@<dir>/<num>/snapshot/ are snapshots of live.
>>>>>>>>>
>>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>>>
>>>>>>>>> On Saturday, July 17th, 2021 at 4:14 PM, Qu Wenruo
>>>>>>>>>
>>>>>>>>> quwenruo.btrfs@gmx.com wrote:
>>>>>>>>>
>>>>>>>>>> On 2021/7/17 下午3:51, pepperpoint@mb.ardentcoding.com wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Qu,
>>>>>>>>>>>
>>>>>>>>>>> Please see below for the dump.
>>>>>>>>>>>
>>>>>>>>>>> btrfs-progs v5.12.1
>>>>>>>>>>>
>>>>>>>>>>> leaf 174113599488 items 18 free space 2008 generation 1330906
>>>>>>>>>>>
>>>>>>>>>>> owner 363
>>>>>>>>>>>
>>>>>>>>>>> leaf 174113599488 flags 0x1(WRITTEN) backref revision 1
>>>>>>>>>>>
>>>>>>>>>>> fs uuid a7d327c4-8594-4116-a6f8-8aa2a4162063
>>>>>>>>>>>
>>>>>>>>>>> chunk uuid f885f49e-14a0-4c80-9c12-c2302b9a0229
>>>>>>>>>>>
>>>>>>>>>>> item 0 key (5471 INODE_ITEM 0) itemoff 3835 itemsize 160
>>>>>>>>>>>
>>>>>>>>>>> generation 2063 transid 27726 size 40 nbytes 40
>>>>>>>>>>>
>>>>>>>>>>> block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
>>>>>>>>>>>
>>>>>>>>>>> sequence 1501 flags 0x0(none)
>>>>>>>>>>>
>>>>>>>>>>> atime 1386484844.468769570 (2013-12-08 14:40:44)
>>>>>>>>>>>
>>>>>>>>>>> ctime 1386484844.468769570 (2013-12-08 14:40:44)
>>>>>>>>>>>
>>>>>>>>>>> mtime 1386484844.468769570 (2013-12-08 14:40:44)
>>>>>>>>>>>
>>>>>>>>>>> otime 0.0 (1970-01-01 08:00:00)
>>>>>>>>>>>
>>>>>>>>>>> item 1 key (5471 INODE_REF 4399) itemoff 3824 itemsize 11
>>>>>>>>>>>
>>>>>>>>>>> index 12 namelen 1 name: 8
>>>>>>>>>>>
>>>>>>>>>>> item 2 key (5471 EXTENT_DATA 0) itemoff 3763 itemsize 61
>>>>>>>>>>>
>>>>>>>>>>> generation 27726 type 0 (inline)
>>>>>>>>>>>
>>>>>>>>>>> inline extent data size 40 ram_bytes 40 compression 0 (none)
>>>>>>>>>>>
>>>>>>>>>>> item 3 key (5645 INODE_ITEM 0) itemoff 3603 itemsize 160
>>>>>>>>>>>
>>>>>>>>>>> generation 2542 transid 61261 size 40 nbytes 40
>>>>>>>>>>>
>>>>>>>>>>> block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
>>>>>>>>>>>
>>>>>>>>>>> sequence 24769 flags 0x0(none)
>>>>>>>>>>>
>>>>>>>>>>> atime 1394335806.351857522 (2014-03-09 11:30:06)
>>>>>>>>>>>
>>>>>>>>>>> ctime 1394335827.344389955 (2014-03-09 11:30:27)
>>>>>>>>>>>
>>>>>>>>>>> mtime 1394335827.344389955 (2014-03-09 11:30:27)
>>>>>>>>>>>
>>>>>>>>>>> otime 0.0 (1970-01-01 08:00:00)
>>>>>>>>>>>
>>>>>>>>>>> item 4 key (5645 INODE_REF 4399) itemoff 3592 itemsize 11
>>>>>>>>>>>
>>>>>>>>>>> index 13 namelen 1 name: 7
>>>>>>>>>>>
>>>>>>>>>>> item 5 key (5645 EXTENT_DATA 0) itemoff 3531 itemsize 61
>>>>>>>>>>>
>>>>>>>>>>> generation 61261 type 0 (inline)
>>>>>>>>>>>
>>>>>>>>>>> inline extent data size 40 ram_bytes 40 compression 0 (none)
>>>>>>>>>>>
>>>>>>>>>>> item 6 key (7222 INODE_ITEM 0) itemoff 3371 itemsize 160
>>>>>>>>>>>
>>>>>>>>>>> generation 5754 transid 5767 size 307 nbytes 307
>>>>>>>>>>>
>>>>>>>>>>> block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
>>>>>>>>>>>
>>>>>>>>>>> sequence 7 flags 0x0(none)
>>>>>>>>>>>
>>>>>>>>>>> atime 1379834835.428558020 (2013-09-22 15:27:15)
>>>>>>>>>>>
>>>>>>>>>>> ctime 1379834835.428558020 (2013-09-22 15:27:15)
>>>>>>>>>>>
>>>>>>>>>>> mtime 1379834835.428558020 (2013-09-22 15:27:15)
>>>>>>>>>>>
>>>>>>>>>>> otime 0.0 (1970-01-01 08:00:00)
>>>>>>>>>>>
>>>>>>>>>>> item 7 key (7222 INODE_REF 287) itemoff 3344 itemsize 27
>>>>>>>>>>>
>>>>>>>>>>> index 6 namelen 17 name: dhcpcd-eth0.lease
>>>>>>>>>>>
>>>>>>>>>>> item 8 key (7222 EXTENT_DATA 0) itemoff 3016 itemsize 328
>>>>>>>>>>>
>>>>>>>>>>> generation 5767 type 0 (inline)
>>>>>>>>>>>
>>>>>>>>>>> inline extent data size 307 ram_bytes 307 compression 0 (none)
>>>>>>>>>>>
>>>>>>>>>>> item 9 key (7415 INODE_ITEM 0) itemoff 2856 itemsize 160
>>>>>>>>>>>
>>>>>>>>>>> generation 5904 transid 1330906 size 180 nbytes 0
>>>>>>>>>>>
>>>>>>>>>>> block group 0 mode 40755 links 2 uid 0 gid 0 rdev 0
>>>>>>>>>>>
>>>>>>>>>>> sequence 177 flags 0x0(none)
>>>>>>>>>>>
>>>>>>>>>>> atime 1483277713.141980592 (2017-01-01 21:35:13)
>>>>>>>>>>>
>>>>>>>>>>> ctime 1563162901.234656246 (2019-07-15 11:55:01)
>>>>>>>>>>>
>>>>>>>>>>> mtime 1406534032.158605559 (2014-07-28 15:53:52)
>>>>>>>>>>>
>>>>>>>>>>> otime 0.0 (1970-01-01 08:00:00)
>>>>>>>>>>
>>>>>>>>>> This inode is indeed a directory.
>>>>>>>>>>
>>>>>>>>>> But it has two hard links, which is definitely something
>>>>>>>>>>
>>>>>>>>>> unexpected.
>>>>>>>>>>
>>>>>>>>>> Under Linux we shouldn't have any hardlink for directory, as it
>>>>>>>>>>
>>>>>>>>>> would
>>>>>>>>>>
>>>>>>>>>> easily lead to loops.
>>>>>>>>>>
>>>>>>>>>>> item 10 key (7415 INODE_REF 260) itemoff 2837 itemsize 19
>>>>>>>>>>>
>>>>>>>>>>> index 28 namelen 9 name: backlight
>>>>>>>>>>
>>>>>>>>>> Its parent inode is 260 in the same root, with the name backlight.
>>>>>>>>>>
>>>>>>>>>>> item 11 key (7415 INODE_REF 286) itemoff 2818 itemsize 19
>>>>>>>>>>>
>>>>>>>>>>> index 3 namelen 9 name: backlight
>>>>>>>>>>
>>>>>>>>>> Another hardlink in inode 286, which is definitely a regular thing.
>>>>>>>>>>
>>>>>>>>>> Btrfs-progs lacks the ability to detect such problem, we need to
>>>>>>>>>>
>>>>>>>>>> enhance
>>>>>>>>>>
>>>>>>>>>> it first.
>>>>>>>>>>
>>>>>>>>>> But do you have any idea how this directory get created?
>>>>>>>>>>
>>>>>>>>>> It looks like the content of sysfs.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Qu
>>>>>>>>>>
>>>>>>>>>>> item 12 key (7415 DIR_ITEM 3128336373) itemoff 2746 itemsize 72
>>>>>>>>>>>
>>>>>>>>>>> location key (120417 INODE_ITEM 0) type FILE
>>>>>>>>>>>
>>>>>>>>>>> transid 117279 data_len 0 name_len 42
>>>>>>>>>>>
>>>>>>>>>>> name: pci-0000:00:02.0:backlight:intel_backlight
>>>>>>>>>>>
>>>>>>>>>>> item 13 key (7415 DIR_ITEM 3218198317) itemoff 2705 itemsize 41
>>>>>>>>>>>
>>>>>>>>>>> location key (7487 INODE_ITEM 0) type FILE
>>>>>>>>>>>
>>>>>>>>>>> transid 5992 data_len 0 name_len 11
>>>>>>>>>>>
>>>>>>>>>>> name: acpi_video0
>>>>>>>>>>>
>>>>>>>>>>> item 14 key (7415 DIR_ITEM 3582254411) itemoff 2638 itemsize 67
>>>>>>>>>>>
>>>>>>>>>>> location key (55325 INODE_ITEM 0) type FILE
>>>>>>>>>>>
>>>>>>>>>>> transid 63351 data_len 0 name_len 37
>>>>>>>>>>>
>>>>>>>>>>> name: platform-VPC2004:00:backlight:ideapad
>>>>>>>>>>>
>>>>>>>>>>> item 15 key (7415 DIR_INDEX 2) itemoff 2597 itemsize 41
>>>>>>>>>>>
>>>>>>>>>>> location key (7487 INODE_ITEM 0) type FILE
>>>>>>>>>>>
>>>>>>>>>>> transid 5992 data_len 0 name_len 11
>>>>>>>>>>>
>>>>>>>>>>> name: acpi_video0
>>>>>>>>>>>
>>>>>>>>>>> item 16 key (7415 DIR_INDEX 4) itemoff 2530 itemsize 67
>>>>>>>>>>>
>>>>>>>>>>> location key (55325 INODE_ITEM 0) type FILE
>>>>>>>>>>>
>>>>>>>>>>> transid 63351 data_len 0 name_len 37
>>>>>>>>>>>
>>>>>>>>>>> name: platform-VPC2004:00:backlight:ideapad
>>>>>>>>>>>
>>>>>>>>>>> item 17 key (7415 DIR_INDEX 5) itemoff 2458 itemsize 72
>>>>>>>>>>>
>>>>>>>>>>> location key (120417 INODE_ITEM 0) type FILE
>>>>>>>>>>>
>>>>>>>>>>> transid 117279 data_len 0 name_len 42
>>>>>>>>>>>
>>>>>>>>>>> name: pci-0000:00:02.0:backlight:intel_backlight
>>>>>>>>>>>
>>>>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>>>>>
>>>>>>>>>>> On Saturday, July 17th, 2021 at 3:05 PM, Qu Wenruo
>>>>>>>>>>>
>>>>>>>>>>> quwenruo.btrfs@gmx.com wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On 2021/7/17 上午9:45, pepperpoint@mb.ardentcoding.com wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I see this message on dmesg:
>>>>>>>>>>>>>
>>>>>>>>>>>>> [ 2452.256756] BTRFS critical (device dm-0): corrupt leaf:
>>>>>>>>>>>>>
>>>>>>>>>>>>> root=363 block=174113599488 slot=9 ino=7415, invalid nlink:
>>>>>>>>>>>>>
>>>>>>>>>>>>> has 2 expect no more than 1 for dir
>>>>>>>>>>>>>
>>>>>>>>>>>>> [ 2452.256776] BTRFS error (device dm-0): block=174113599488
>>>>>>>>>>>>>
>>>>>>>>>>>>> read time tree block corruption detected
>>>>>>>>>>>>
>>>>>>>>>>>> Please provide the following dump:
>>>>>>>>>>>>
>>>>>>>>>>>> btrfs ins dump-tree -b 174113599488 /dev/dm-0
>>>>>>>>>>>> =============================================
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> Qu
>>>>>>>>>>>>
>>>>>>>>>>>>> When I run btrfs scrub and btrfs check, no error was detected.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am running Linux 5.12.15-arch1-1 and btrfs-progs v5.12.1
>>>>>>>>>>>>>
>>>>>>>>>>>>> How should I fix this error?

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: Read time tree block corruption detected
       [not found]                     ` <162659327011.8.12718249358254503695.10218325@simplelogin.co>
@ 2021-07-18  8:46                       ` pepperpoint
       [not found]                       ` <162659797656.6.14385932343235367381.10220447@mb.ardentcoding.com>
  1 sibling, 0 replies; 75+ messages in thread
From: pepperpoint @ 2021-07-18  8:46 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Qu Wenruo, linux-btrfs

Hi Qu,

When I find which directory some of the filenames are located, they are under /var/lib.
I had subvolume for /var which was created probably around 2018.

I don't remember how I created this but I probably use rsync to copy the files from existing /var
or created a snapshot of root and delete other files that is not under /var.

Around June, I tried to move the filesystem to another partition through btrfs device add and btrfs device remove
but failed due to that error and was advised to use btrfs replace instead.

Then at the beginning of this month, I reorganized it merging most of the /var content back to root
and created subvolume for /var/lib/mysql and /var/lib/mongodb. I encountered an error when
I copy some of the files through cp --reflink but I failed for /var/lib/mysql so I created a snapshot
from /var and remove the extra files.

This is also the time I saw the errors in the log. Before that, the errors was not in the log.

Regards,
Lester


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Sunday, July 18th, 2021 at 3:27 PM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:

> Hi,
>
> BTW, it's really important for us to know how the directory is hardlinked.
>
> Thus I salvaged the filenames found in the half-dropped root 363.
>
> Since it may contain confidential info, I send the filename list to you
>
> off-list.
>
> If you can remind what the root 363 is used for, and any possible
>
> operations which may be involved in that subvolume, it's better to reply
>
> it to the mail list so that we can get some clue on the root cause.
>
> Thanks,
>
> Qu
>
> On 2021/7/18 下午3:15, Qu Wenruo wrote:
>
> > On 2021/7/18 下午1:26, pepperpoint@mb.ardentcoding.com wrote:
> >
> > > Hi Qu,
> > >
> > > May I know if there are any leads on this? What should I do for now?
> >
> > Sorry for the late reply.
> >
> > With the image dump, it's much easier to find what's going wrong.
> >
> > -   About root 363
> >
> >       It's an orphan root, thus user can't access it directly.
> >
> > Furthermore, it's being dropped, thus "btrfs ins dump-tree -t 363"
> >
> >   reports transid error, as part of the tree has already been dropped,
> >
> >   and this is expected.
> >
> > So far your fs is still OK, except that reported error.
> >
> > -   About the offending tree block
> >
> >       The offending tree block only belongs to the delete subvolume 363,
> >
> >       thus it should be delete soon.
> >
> > But unfortunately due to the corrupted content, it's unable to be
> >
> >   deleted.
> >
> > For now, if you can re-compile btrfs module, we can workaround the
> >
> >   problem by temporarily disable read-time tree-checker so that the
> >
> >   deletion can continue, and after the root 363 get fully deleted, the
> >
> >   problem should be gone.
> >
> > Or you can use older kernel, any kernel <= v5.1 should not have the
> >
> >   enhanced check, thus can continue with the subvolume deletion.
> >
> > If you want to go through the re-compile path, the needed diff is
> >
> >   attached
> >
> > Thanks,
> >
> > Qu
> >
> > > Regards,
> > >
> > > Lester
> > >
> > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > >
> > > On Saturday, July 17th, 2021 at 8:51 PM,
> > >
> > > pepperpoint@mb.ardentcoding.com wrote:
> > >
> > > > Hi Qu,
> > > >
> > > > I run btrfs ins dump-tree -t 363 unmounted but the same error
> > > >
> > > > appears. Rerunning btrfs check does not show any error.
> > > >
> > > > Regards,
> > > >
> > > > Lester
> > > >
> > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > >
> > > > On Saturday, July 17th, 2021 at 6:48 PM, Qu Wenruo -
> > > >
> > > > quwenruo.btrfs@gmx.com wrote:
> > > >
> > > > > On 2021/7/17 下午6:34, pepperpoint@mb.ardentcoding.com wrote:
> > > > >
> > > > > > Hi Qu,
> > > > > >
> > > > > > Unfortunately I cannot find subvolume 363
> > > > > >
> > > > > > btrfs subvolume list /run/media/root
> > > > > > ====================================
> > > > > >
> > > > > > ID 361 gen 1814826 top level 584 path @/live/snapshot
> > > > > >
> > > > > > ID 364 gen 1814414 top level 5 path @vtmp/live/snapshot
> > > > > >
> > > > > > ID 369 gen 1814414 top level 5 path @vlmachines/live/snapshot
> > > > > >
> > > > > > ID 493 gen 1814414 top level 5 path @vlportables/live/snapshot
> > > > > >
> > > > > > ID 579 gen 1814828 top level 5 path @vlog/live/snapshot
> > > > > >
> > > > > > ID 580 gen 1814414 top level 5 path @vcache/live/snapshot
> > > > > >
> > > > > > ID 581 gen 1814414 top level 5 path @vlmongodb/live/snapshot
> > > > > >
> > > > > > ID 582 gen 1814414 top level 5 path @vlmysql/live/snapshot
> > > > > >
> > > > > > ID 583 gen 1814414 top level 5 path @vspool/live/snapshot
> > > > > >
> > > > > > ID 584 gen 1814414 top level 5 path @
> > > > > >
> > > > > > ID 598 gen 1813420 top level 584 path @/4/snapshot
> > > > >
> > > > > Maybe 363 is some subvolume get deleted and later snapshot of it still
> > > > >
> > > > > exists.
> > > > >
> > > > > This will be harder to debug.
> > > > >
> > > > > Can you take a btrfs-image dump of your filesystem? (needs to be taken
> > > > >
> > > > > with the fs unmounted).
> > > > >
> > > > > The dumped image will contain your metadata, including file names and
> > > > >
> > > > > directory structures, but no data inside those files.
> > > > >
> > > > > Although btrfs-image has "-s" option to mask the filenames, but
> > > > >
> > > > > considering the filename in this case is useful to locate the inode, I
> > > > >
> > > > > guess it's better to take the image without any "-s" option.
> > > > >
> > > > > > btrfs ins dump-tree -t 363 /dev/dm-0 | grep -A 5 "(286 "
> > > > > > ========================================================
> > > > > >
> > > > > > parent transid verify failed on 174170742784 wanted 1789655 found
> > > > > >
> > > > > > 1812621
> > > > > >
> > > > > > parent transid verify failed on 174170742784 wanted 1789655 found
> > > > > >
> > > > > > 1812621
> > > > > >
> > > > > > parent transid verify failed on 174170742784 wanted 1789655 found
> > > > > >
> > > > > > 1812621
> > > > > >
> > > > > > Ignoring transid failure
> > > > > >
> > > > > > ERROR: child eb corrupted: parent bytenr=174170738688 item=0 parent
> > > > > >
> > > > > > level=2 child bytenr=174170742784 child level=0
> > > > >
> > > > > This transid mismatch may be a problem when running dump-tree on
> > > > >
> > > > > mounted
> > > > >
> > > > > fs, since you mentioned btrfs check reported no error, there shouldn't
> > > > >
> > > > > be a transid mismatch error.
> > > > >
> > > > > Anyway, if you can upload the btrfs-image dump, it would be much easier
> > > > >
> > > > > for us to debug and find out what's really going.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Qu
> > > > >
> > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > > >
> > > > > > On Saturday, July 17th, 2021 at 6:12 PM, Qu Wenruo wqu@suse.com wrote:
> > > > > >
> > > > > > > On 2021/7/17 下午4:57, pepperpoint@mb.ardentcoding.com wrote:
> > > > > > >
> > > > > > > > Hi Qu,
> > > > > > > >
> > > > > > > > I don't know how the directory was created but last month, I used
> > > > > > > >
> > > > > > > > btrfs device add and btrfs device remove to move the filesystem
> > > > > > > >
> > > > > > > > from one partition to another. It failed because of the same
> > > > > > > >
> > > > > > > > error and was advised to use btrfs replace instead. I don't know
> > > > > > > >
> > > > > > > > if the error also happened before I move the file system as I
> > > > > > > >
> > > > > > > > don't have any previous logs.
> > > > > > >
> > > > > > > It definitely happens before you moving the fs.
> > > > > > >
> > > > > > > As regular dev replacing/add/move only relocates the metadata, but
> > > > > > >
> > > > > > > not
> > > > > > >
> > > > > > > touching the fs trees.
> > > > > > >
> > > > > > > > Here is the result when I search for the inodes you mentioned if
> > > > > > > >
> > > > > > > > it helps:
> > > > > > > >
> > > > > > > > find /run/media/root -inum 260 -exec ls -ldi {} \;
> > > > > > > > ==================================================
> > > > > > > >
> > > > > > > > 260 -rw-r--r-- 1 root root 36864 Jun 25 06:22
> > > > > > > >
> > > > > > > > /run/media/root/@vcache/live/snapshot/app-info/cache/en_US.cache
> > > > > > > >
> > > > > > > > 260 drwx------ 1 mongodb mongodb 136 Sep 12 2020
> > > > > > > >
> > > > > > > > /run/media/root/@vlmongodb/live/snapshot/diagnostic.data
> > > > > > > >
> > > > > > > > 260 -rw-rw---- 1 mysql mysql 50331648 Sep 13 2015
> > > > > > > >
> > > > > > > > /run/media/root/@vlmysql/live/snapshot/ib_logfile0
> > > > > > > >
> > > > > > > > 260 -rw-r----- 1 root lp 8641 Mar 5 2014
> > > > > > > >
> > > > > > > > /run/media/root/@vspool/live/snapshot/cups/d00001-001
> > > > > > > >
> > > > > > > > 260 dr-xr-xr-x 1 root root 0 Sep 13 2013
> > > > > > > >
> > > > > > > > /run/media/root/@/live/snapshot/sys
> > > > > > > >
> > > > > > > > 260 dr-xr-xr-x 1 root root 0 Sep 13 2013
> > > > > > > >
> > > > > > > > /run/media/root/@/4/snapshot/sys
> > > > > > >
> > > > > > > Since btrfs can have the same inode number inside different
> > > > > > >
> > > > > > > subvolumes,
> > > > > > >
> > > > > > > you may want to limit the search inside subvolume 363.
> > > > > > >
> > > > > > > "-mount" option of find can do that, you only need to locate
> > > > > > >
> > > > > > > subvolume
> > > > > > >
> > > > > > > 363 by "btrfs subv list".
> > > > > > >
> > > > > > > But from these output I guess above two "sys" directory are more
> > > > > > >
> > > > > > > possible.
> > > > > > >
> > > > > > > Is there any directory named "blaklight" inside those directory?
> > > > > > >
> > > > > > > > find /run/media/root -inum 286 -exec ls -ldi {} \;
> > > > > > > > ==================================================
> > > > > > > >
> > > > > > > > 286 -rw-r--r-- 1 root root 96 Aug 16 2015
> > > > > > > >
> > > > > > > > /run/media/root/@vcache/live/snapshot/fontconfig/4b172ca7f111e3cffadc3636415fead9-le64.cache-4
> > > > > > > >
> > > > > > > > 286 -rw-rw---- 1 mysql mysql 4096 Sep 15 2013
> > > > > > > >
> > > > > > > > /run/media/root/@vlmysql/live/snapshot/mysql/columns_priv.MYI
> > > > > > > >
> > > > > > > > 286 -rw-r-----+ 1 root systemd-journal 16777216 Jul 4 01:14
> > > > > > > >
> > > > > > > > /run/media/root/@vlog/live/snapshot/journal/5098dd7845ae46d3ba1826c68a809a7c/user-1000@fbd9f65d0ea349f6b996716280e6c4dd-00000000002314c5-0005c5cb84a3a438.journal
> > > > > > >
> > > > > > > This is interesting, it means the inode 286 is not accessible.
> > > > > > >
> > > > > > > It can be some orphan inode, but would you dump subvolume 363 then
> > > > > > >
> > > > > > > try
> > > > > > >
> > > > > > > to locate the inode 286?
> > > > > > >
> > > > > > > One example command would be:
> > > > > > >
> > > > > > > btrfs ins dump-tree -t 363 <dev> | grep -A 5 "(286 "
> > > > > > > ====================================================
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Qu
> > > > > > >
> > > > > > > > Directories with pattern /root/@<dir>/live/snapshot/ are
> > > > > > > >
> > > > > > > > subvolumes and directories with pattern
> > > > > > > >
> > > > > > > > /root/@<dir>/<num>/snapshot/ are snapshots of live.
> > > > > > > >
> > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > > > > >
> > > > > > > > On Saturday, July 17th, 2021 at 4:14 PM, Qu Wenruo
> > > > > > > >
> > > > > > > > quwenruo.btrfs@gmx.com wrote:
> > > > > > > >
> > > > > > > > > On 2021/7/17 下午3:51, pepperpoint@mb.ardentcoding.com wrote:
> > > > > > > > >
> > > > > > > > > > Hi Qu,
> > > > > > > > > >
> > > > > > > > > > Please see below for the dump.
> > > > > > > > > >
> > > > > > > > > > btrfs-progs v5.12.1
> > > > > > > > > >
> > > > > > > > > > leaf 174113599488 items 18 free space 2008 generation 1330906
> > > > > > > > > >
> > > > > > > > > > owner 363
> > > > > > > > > >
> > > > > > > > > > leaf 174113599488 flags 0x1(WRITTEN) backref revision 1
> > > > > > > > > >
> > > > > > > > > > fs uuid a7d327c4-8594-4116-a6f8-8aa2a4162063
> > > > > > > > > >
> > > > > > > > > > chunk uuid f885f49e-14a0-4c80-9c12-c2302b9a0229
> > > > > > > > > >
> > > > > > > > > > item 0 key (5471 INODE_ITEM 0) itemoff 3835 itemsize 160
> > > > > > > > > >
> > > > > > > > > > generation 2063 transid 27726 size 40 nbytes 40
> > > > > > > > > >
> > > > > > > > > > block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
> > > > > > > > > >
> > > > > > > > > > sequence 1501 flags 0x0(none)
> > > > > > > > > >
> > > > > > > > > > atime 1386484844.468769570 (2013-12-08 14:40:44)
> > > > > > > > > >
> > > > > > > > > > ctime 1386484844.468769570 (2013-12-08 14:40:44)
> > > > > > > > > >
> > > > > > > > > > mtime 1386484844.468769570 (2013-12-08 14:40:44)
> > > > > > > > > >
> > > > > > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > > > > > >
> > > > > > > > > > item 1 key (5471 INODE_REF 4399) itemoff 3824 itemsize 11
> > > > > > > > > >
> > > > > > > > > > index 12 namelen 1 name: 8
> > > > > > > > > >
> > > > > > > > > > item 2 key (5471 EXTENT_DATA 0) itemoff 3763 itemsize 61
> > > > > > > > > >
> > > > > > > > > > generation 27726 type 0 (inline)
> > > > > > > > > >
> > > > > > > > > > inline extent data size 40 ram_bytes 40 compression 0 (none)
> > > > > > > > > >
> > > > > > > > > > item 3 key (5645 INODE_ITEM 0) itemoff 3603 itemsize 160
> > > > > > > > > >
> > > > > > > > > > generation 2542 transid 61261 size 40 nbytes 40
> > > > > > > > > >
> > > > > > > > > > block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
> > > > > > > > > >
> > > > > > > > > > sequence 24769 flags 0x0(none)
> > > > > > > > > >
> > > > > > > > > > atime 1394335806.351857522 (2014-03-09 11:30:06)
> > > > > > > > > >
> > > > > > > > > > ctime 1394335827.344389955 (2014-03-09 11:30:27)
> > > > > > > > > >
> > > > > > > > > > mtime 1394335827.344389955 (2014-03-09 11:30:27)
> > > > > > > > > >
> > > > > > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > > > > > >
> > > > > > > > > > item 4 key (5645 INODE_REF 4399) itemoff 3592 itemsize 11
> > > > > > > > > >
> > > > > > > > > > index 13 namelen 1 name: 7
> > > > > > > > > >
> > > > > > > > > > item 5 key (5645 EXTENT_DATA 0) itemoff 3531 itemsize 61
> > > > > > > > > >
> > > > > > > > > > generation 61261 type 0 (inline)
> > > > > > > > > >
> > > > > > > > > > inline extent data size 40 ram_bytes 40 compression 0 (none)
> > > > > > > > > >
> > > > > > > > > > item 6 key (7222 INODE_ITEM 0) itemoff 3371 itemsize 160
> > > > > > > > > >
> > > > > > > > > > generation 5754 transid 5767 size 307 nbytes 307
> > > > > > > > > >
> > > > > > > > > > block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
> > > > > > > > > >
> > > > > > > > > > sequence 7 flags 0x0(none)
> > > > > > > > > >
> > > > > > > > > > atime 1379834835.428558020 (2013-09-22 15:27:15)
> > > > > > > > > >
> > > > > > > > > > ctime 1379834835.428558020 (2013-09-22 15:27:15)
> > > > > > > > > >
> > > > > > > > > > mtime 1379834835.428558020 (2013-09-22 15:27:15)
> > > > > > > > > >
> > > > > > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > > > > > >
> > > > > > > > > > item 7 key (7222 INODE_REF 287) itemoff 3344 itemsize 27
> > > > > > > > > >
> > > > > > > > > > index 6 namelen 17 name: dhcpcd-eth0.lease
> > > > > > > > > >
> > > > > > > > > > item 8 key (7222 EXTENT_DATA 0) itemoff 3016 itemsize 328
> > > > > > > > > >
> > > > > > > > > > generation 5767 type 0 (inline)
> > > > > > > > > >
> > > > > > > > > > inline extent data size 307 ram_bytes 307 compression 0 (none)
> > > > > > > > > >
> > > > > > > > > > item 9 key (7415 INODE_ITEM 0) itemoff 2856 itemsize 160
> > > > > > > > > >
> > > > > > > > > > generation 5904 transid 1330906 size 180 nbytes 0
> > > > > > > > > >
> > > > > > > > > > block group 0 mode 40755 links 2 uid 0 gid 0 rdev 0
> > > > > > > > > >
> > > > > > > > > > sequence 177 flags 0x0(none)
> > > > > > > > > >
> > > > > > > > > > atime 1483277713.141980592 (2017-01-01 21:35:13)
> > > > > > > > > >
> > > > > > > > > > ctime 1563162901.234656246 (2019-07-15 11:55:01)
> > > > > > > > > >
> > > > > > > > > > mtime 1406534032.158605559 (2014-07-28 15:53:52)
> > > > > > > > > >
> > > > > > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > > > > >
> > > > > > > > > This inode is indeed a directory.
> > > > > > > > >
> > > > > > > > > But it has two hard links, which is definitely something
> > > > > > > > >
> > > > > > > > > unexpected.
> > > > > > > > >
> > > > > > > > > Under Linux we shouldn't have any hardlink for directory, as it
> > > > > > > > >
> > > > > > > > > would
> > > > > > > > >
> > > > > > > > > easily lead to loops.
> > > > > > > > >
> > > > > > > > > > item 10 key (7415 INODE_REF 260) itemoff 2837 itemsize 19
> > > > > > > > > >
> > > > > > > > > > index 28 namelen 9 name: backlight
> > > > > > > > >
> > > > > > > > > Its parent inode is 260 in the same root, with the name backlight.
> > > > > > > > >
> > > > > > > > > > item 11 key (7415 INODE_REF 286) itemoff 2818 itemsize 19
> > > > > > > > > >
> > > > > > > > > > index 3 namelen 9 name: backlight
> > > > > > > > >
> > > > > > > > > Another hardlink in inode 286, which is definitely a regular thing.
> > > > > > > > >
> > > > > > > > > Btrfs-progs lacks the ability to detect such problem, we need to
> > > > > > > > >
> > > > > > > > > enhance
> > > > > > > > >
> > > > > > > > > it first.
> > > > > > > > >
> > > > > > > > > But do you have any idea how this directory get created?
> > > > > > > > >
> > > > > > > > > It looks like the content of sysfs.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Qu
> > > > > > > > >
> > > > > > > > > > item 12 key (7415 DIR_ITEM 3128336373) itemoff 2746 itemsize 72
> > > > > > > > > >
> > > > > > > > > > location key (120417 INODE_ITEM 0) type FILE
> > > > > > > > > >
> > > > > > > > > > transid 117279 data_len 0 name_len 42
> > > > > > > > > >
> > > > > > > > > > name: pci-0000:00:02.0:backlight:intel_backlight
> > > > > > > > > >
> > > > > > > > > > item 13 key (7415 DIR_ITEM 3218198317) itemoff 2705 itemsize 41
> > > > > > > > > >
> > > > > > > > > > location key (7487 INODE_ITEM 0) type FILE
> > > > > > > > > >
> > > > > > > > > > transid 5992 data_len 0 name_len 11
> > > > > > > > > >
> > > > > > > > > > name: acpi_video0
> > > > > > > > > >
> > > > > > > > > > item 14 key (7415 DIR_ITEM 3582254411) itemoff 2638 itemsize 67
> > > > > > > > > >
> > > > > > > > > > location key (55325 INODE_ITEM 0) type FILE
> > > > > > > > > >
> > > > > > > > > > transid 63351 data_len 0 name_len 37
> > > > > > > > > >
> > > > > > > > > > name: platform-VPC2004:00:backlight:ideapad
> > > > > > > > > >
> > > > > > > > > > item 15 key (7415 DIR_INDEX 2) itemoff 2597 itemsize 41
> > > > > > > > > >
> > > > > > > > > > location key (7487 INODE_ITEM 0) type FILE
> > > > > > > > > >
> > > > > > > > > > transid 5992 data_len 0 name_len 11
> > > > > > > > > >
> > > > > > > > > > name: acpi_video0
> > > > > > > > > >
> > > > > > > > > > item 16 key (7415 DIR_INDEX 4) itemoff 2530 itemsize 67
> > > > > > > > > >
> > > > > > > > > > location key (55325 INODE_ITEM 0) type FILE
> > > > > > > > > >
> > > > > > > > > > transid 63351 data_len 0 name_len 37
> > > > > > > > > >
> > > > > > > > > > name: platform-VPC2004:00:backlight:ideapad
> > > > > > > > > >
> > > > > > > > > > item 17 key (7415 DIR_INDEX 5) itemoff 2458 itemsize 72
> > > > > > > > > >
> > > > > > > > > > location key (120417 INODE_ITEM 0) type FILE
> > > > > > > > > >
> > > > > > > > > > transid 117279 data_len 0 name_len 42
> > > > > > > > > >
> > > > > > > > > > name: pci-0000:00:02.0:backlight:intel_backlight
> > > > > > > > > >
> > > > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > > > > > > >
> > > > > > > > > > On Saturday, July 17th, 2021 at 3:05 PM, Qu Wenruo
> > > > > > > > > >
> > > > > > > > > > quwenruo.btrfs@gmx.com wrote:
> > > > > > > > > >
> > > > > > > > > > > On 2021/7/17 上午9:45, pepperpoint@mb.ardentcoding.com wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hello,
> > > > > > > > > > > >
> > > > > > > > > > > > I see this message on dmesg:
> > > > > > > > > > > >
> > > > > > > > > > > > [ 2452.256756] BTRFS critical (device dm-0): corrupt leaf:
> > > > > > > > > > > >
> > > > > > > > > > > > root=363 block=174113599488 slot=9 ino=7415, invalid nlink:
> > > > > > > > > > > >
> > > > > > > > > > > > has 2 expect no more than 1 for dir
> > > > > > > > > > > >
> > > > > > > > > > > > [ 2452.256776] BTRFS error (device dm-0): block=174113599488
> > > > > > > > > > > >
> > > > > > > > > > > > read time tree block corruption detected
> > > > > > > > > > >
> > > > > > > > > > > Please provide the following dump:
> > > > > > > > > > >
> > > > > > > > > > > btrfs ins dump-tree -b 174113599488 /dev/dm-0
> > > > > > > > > > > =============================================
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Qu
> > > > > > > > > > >
> > > > > > > > > > > > When I run btrfs scrub and btrfs check, no error was detected.
> > > > > > > > > > > >
> > > > > > > > > > > > I am running Linux 5.12.15-arch1-1 and btrfs-progs v5.12.1
> > > > > > > > > > > >
> > > > > > > > > > > > How should I fix this error?

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: Read time tree block corruption detected
  2021-07-18  5:26                 ` pepperpoint
@ 2021-07-18  7:15                   ` Qu Wenruo
       [not found]                     ` <162659327011.8.12718249358254503695.10218325@simplelogin.co>
  0 siblings, 1 reply; 75+ messages in thread
From: Qu Wenruo @ 2021-07-18  7:15 UTC (permalink / raw)
  To: pepperpoint; +Cc: Qu Wenruo, linux-btrfs

[-- Attachment #1: Type: text/plain, Size: 15135 bytes --]



On 2021/7/18 下午1:26, pepperpoint@mb.ardentcoding.com wrote:
> Hi Qu,
>
> May I know if there are any leads on this? What should I do for now?

Sorry for the late reply.

With the image dump, it's much easier to find what's going wrong.

- About root 363
   It's an orphan root, thus user can't access it directly.

   Furthermore, it's being dropped, thus "btrfs ins dump-tree -t 363"
   reports transid error, as part of the tree has already been dropped,
   and this is expected.

   So far your fs is still OK, except that reported error.

- About the offending tree block
   The offending tree block only belongs to the delete subvolume 363,
   thus it should be delete soon.

   But unfortunately due to the corrupted content, it's unable to be
   deleted.

   For now, if you can re-compile btrfs module, we can workaround the
   problem by temporarily disable read-time tree-checker so that the
   deletion can continue, and after the root 363 get fully deleted, the
   problem should be gone.

   Or you can use older kernel, any kernel <= v5.1 should not have the
   enhanced check, thus can continue with the subvolume deletion.

   If you want to go through the re-compile path, the needed diff is
   attached

Thanks,
Qu
>
> Regards,
> Lester
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>
> On Saturday, July 17th, 2021 at 8:51 PM, pepperpoint@mb.ardentcoding.com wrote:
>
>> Hi Qu,
>>
>> I run btrfs ins dump-tree -t 363 unmounted but the same error appears. Rerunning btrfs check does not show any error.
>>
>> Regards,
>>
>> Lester
>>
>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>
>> On Saturday, July 17th, 2021 at 6:48 PM, Qu Wenruo - quwenruo.btrfs@gmx.com wrote:
>>
>>> On 2021/7/17 下午6:34, pepperpoint@mb.ardentcoding.com wrote:
>>>
>>>> Hi Qu,
>>>>
>>>> Unfortunately I cannot find subvolume 363
>>>>
>>>> btrfs subvolume list /run/media/root
>>>> ====================================
>>>>
>>>> ID 361 gen 1814826 top level 584 path @/live/snapshot
>>>>
>>>> ID 364 gen 1814414 top level 5 path @vtmp/live/snapshot
>>>>
>>>> ID 369 gen 1814414 top level 5 path @vlmachines/live/snapshot
>>>>
>>>> ID 493 gen 1814414 top level 5 path @vlportables/live/snapshot
>>>>
>>>> ID 579 gen 1814828 top level 5 path @vlog/live/snapshot
>>>>
>>>> ID 580 gen 1814414 top level 5 path @vcache/live/snapshot
>>>>
>>>> ID 581 gen 1814414 top level 5 path @vlmongodb/live/snapshot
>>>>
>>>> ID 582 gen 1814414 top level 5 path @vlmysql/live/snapshot
>>>>
>>>> ID 583 gen 1814414 top level 5 path @vspool/live/snapshot
>>>>
>>>> ID 584 gen 1814414 top level 5 path @
>>>>
>>>> ID 598 gen 1813420 top level 584 path @/4/snapshot
>>>
>>> Maybe 363 is some subvolume get deleted and later snapshot of it still
>>>
>>> exists.
>>>
>>> This will be harder to debug.
>>>
>>> Can you take a btrfs-image dump of your filesystem? (needs to be taken
>>>
>>> with the fs unmounted).
>>>
>>> The dumped image will contain your metadata, including file names and
>>>
>>> directory structures, but no data inside those files.
>>>
>>> Although btrfs-image has "-s" option to mask the filenames, but
>>>
>>> considering the filename in this case is useful to locate the inode, I
>>>
>>> guess it's better to take the image without any "-s" option.
>>>
>>>> btrfs ins dump-tree -t 363 /dev/dm-0 | grep -A 5 "(286 "
>>>> ========================================================
>>>>
>>>> parent transid verify failed on 174170742784 wanted 1789655 found 1812621
>>>>
>>>> parent transid verify failed on 174170742784 wanted 1789655 found 1812621
>>>>
>>>> parent transid verify failed on 174170742784 wanted 1789655 found 1812621
>>>>
>>>> Ignoring transid failure
>>>>
>>>> ERROR: child eb corrupted: parent bytenr=174170738688 item=0 parent level=2 child bytenr=174170742784 child level=0
>>>
>>> This transid mismatch may be a problem when running dump-tree on mounted
>>>
>>> fs, since you mentioned btrfs check reported no error, there shouldn't
>>>
>>> be a transid mismatch error.
>>>
>>> Anyway, if you can upload the btrfs-image dump, it would be much easier
>>>
>>> for us to debug and find out what's really going.
>>>
>>> Thanks,
>>>
>>> Qu
>>>
>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>
>>>> On Saturday, July 17th, 2021 at 6:12 PM, Qu Wenruo wqu@suse.com wrote:
>>>>
>>>>> On 2021/7/17 下午4:57, pepperpoint@mb.ardentcoding.com wrote:
>>>>>
>>>>>> Hi Qu,
>>>>>>
>>>>>> I don't know how the directory was created but last month, I used btrfs device add and btrfs device remove to move the filesystem from one partition to another. It failed because of the same error and was advised to use btrfs replace instead. I don't know if the error also happened before I move the file system as I don't have any previous logs.
>>>>>
>>>>> It definitely happens before you moving the fs.
>>>>>
>>>>> As regular dev replacing/add/move only relocates the metadata, but not
>>>>>
>>>>> touching the fs trees.
>>>>>
>>>>>> Here is the result when I search for the inodes you mentioned if it helps:
>>>>>>
>>>>>> find /run/media/root -inum 260 -exec ls -ldi {} \;
>>>>>> ==================================================
>>>>>>
>>>>>> 260 -rw-r--r-- 1 root root 36864 Jun 25 06:22 /run/media/root/@vcache/live/snapshot/app-info/cache/en_US.cache
>>>>>>
>>>>>> 260 drwx------ 1 mongodb mongodb 136 Sep 12 2020 /run/media/root/@vlmongodb/live/snapshot/diagnostic.data
>>>>>>
>>>>>> 260 -rw-rw---- 1 mysql mysql 50331648 Sep 13 2015 /run/media/root/@vlmysql/live/snapshot/ib_logfile0
>>>>>>
>>>>>> 260 -rw-r----- 1 root lp 8641 Mar 5 2014 /run/media/root/@vspool/live/snapshot/cups/d00001-001
>>>>>>
>>>>>> 260 dr-xr-xr-x 1 root root 0 Sep 13 2013 /run/media/root/@/live/snapshot/sys
>>>>>>
>>>>>> 260 dr-xr-xr-x 1 root root 0 Sep 13 2013 /run/media/root/@/4/snapshot/sys
>>>>>
>>>>> Since btrfs can have the same inode number inside different subvolumes,
>>>>>
>>>>> you may want to limit the search inside subvolume 363.
>>>>>
>>>>> "-mount" option of find can do that, you only need to locate subvolume
>>>>>
>>>>> 363 by "btrfs subv list".
>>>>>
>>>>> But from these output I guess above two "sys" directory are more possible.
>>>>>
>>>>> Is there any directory named "blaklight" inside those directory?
>>>>>
>>>>>> find /run/media/root -inum 286 -exec ls -ldi {} \;
>>>>>> ==================================================
>>>>>>
>>>>>> 286 -rw-r--r-- 1 root root 96 Aug 16 2015 /run/media/root/@vcache/live/snapshot/fontconfig/4b172ca7f111e3cffadc3636415fead9-le64.cache-4
>>>>>>
>>>>>> 286 -rw-rw---- 1 mysql mysql 4096 Sep 15 2013 /run/media/root/@vlmysql/live/snapshot/mysql/columns_priv.MYI
>>>>>>
>>>>>> 286 -rw-r-----+ 1 root systemd-journal 16777216 Jul 4 01:14 /run/media/root/@vlog/live/snapshot/journal/5098dd7845ae46d3ba1826c68a809a7c/user-1000@fbd9f65d0ea349f6b996716280e6c4dd-00000000002314c5-0005c5cb84a3a438.journal
>>>>>
>>>>> This is interesting, it means the inode 286 is not accessible.
>>>>>
>>>>> It can be some orphan inode, but would you dump subvolume 363 then try
>>>>>
>>>>> to locate the inode 286?
>>>>>
>>>>> One example command would be:
>>>>>
>>>>> btrfs ins dump-tree -t 363 <dev> | grep -A 5 "(286 "
>>>>> ====================================================
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Qu
>>>>>
>>>>>> Directories with pattern /root/@<dir>/live/snapshot/ are subvolumes and directories with pattern /root/@<dir>/<num>/snapshot/ are snapshots of live.
>>>>>>
>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>
>>>>>> On Saturday, July 17th, 2021 at 4:14 PM, Qu Wenruo quwenruo.btrfs@gmx.com wrote:
>>>>>>
>>>>>>> On 2021/7/17 下午3:51, pepperpoint@mb.ardentcoding.com wrote:
>>>>>>>
>>>>>>>> Hi Qu,
>>>>>>>>
>>>>>>>> Please see below for the dump.
>>>>>>>>
>>>>>>>> btrfs-progs v5.12.1
>>>>>>>>
>>>>>>>> leaf 174113599488 items 18 free space 2008 generation 1330906 owner 363
>>>>>>>>
>>>>>>>> leaf 174113599488 flags 0x1(WRITTEN) backref revision 1
>>>>>>>>
>>>>>>>> fs uuid a7d327c4-8594-4116-a6f8-8aa2a4162063
>>>>>>>>
>>>>>>>> chunk uuid f885f49e-14a0-4c80-9c12-c2302b9a0229
>>>>>>>>
>>>>>>>> item 0 key (5471 INODE_ITEM 0) itemoff 3835 itemsize 160
>>>>>>>>
>>>>>>>> generation 2063 transid 27726 size 40 nbytes 40
>>>>>>>>
>>>>>>>> block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
>>>>>>>>
>>>>>>>> sequence 1501 flags 0x0(none)
>>>>>>>>
>>>>>>>> atime 1386484844.468769570 (2013-12-08 14:40:44)
>>>>>>>>
>>>>>>>> ctime 1386484844.468769570 (2013-12-08 14:40:44)
>>>>>>>>
>>>>>>>> mtime 1386484844.468769570 (2013-12-08 14:40:44)
>>>>>>>>
>>>>>>>> otime 0.0 (1970-01-01 08:00:00)
>>>>>>>>
>>>>>>>> item 1 key (5471 INODE_REF 4399) itemoff 3824 itemsize 11
>>>>>>>>
>>>>>>>> index 12 namelen 1 name: 8
>>>>>>>>
>>>>>>>> item 2 key (5471 EXTENT_DATA 0) itemoff 3763 itemsize 61
>>>>>>>>
>>>>>>>> generation 27726 type 0 (inline)
>>>>>>>>
>>>>>>>> inline extent data size 40 ram_bytes 40 compression 0 (none)
>>>>>>>>
>>>>>>>> item 3 key (5645 INODE_ITEM 0) itemoff 3603 itemsize 160
>>>>>>>>
>>>>>>>> generation 2542 transid 61261 size 40 nbytes 40
>>>>>>>>
>>>>>>>> block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
>>>>>>>>
>>>>>>>> sequence 24769 flags 0x0(none)
>>>>>>>>
>>>>>>>> atime 1394335806.351857522 (2014-03-09 11:30:06)
>>>>>>>>
>>>>>>>> ctime 1394335827.344389955 (2014-03-09 11:30:27)
>>>>>>>>
>>>>>>>> mtime 1394335827.344389955 (2014-03-09 11:30:27)
>>>>>>>>
>>>>>>>> otime 0.0 (1970-01-01 08:00:00)
>>>>>>>>
>>>>>>>> item 4 key (5645 INODE_REF 4399) itemoff 3592 itemsize 11
>>>>>>>>
>>>>>>>> index 13 namelen 1 name: 7
>>>>>>>>
>>>>>>>> item 5 key (5645 EXTENT_DATA 0) itemoff 3531 itemsize 61
>>>>>>>>
>>>>>>>> generation 61261 type 0 (inline)
>>>>>>>>
>>>>>>>> inline extent data size 40 ram_bytes 40 compression 0 (none)
>>>>>>>>
>>>>>>>> item 6 key (7222 INODE_ITEM 0) itemoff 3371 itemsize 160
>>>>>>>>
>>>>>>>> generation 5754 transid 5767 size 307 nbytes 307
>>>>>>>>
>>>>>>>> block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
>>>>>>>>
>>>>>>>> sequence 7 flags 0x0(none)
>>>>>>>>
>>>>>>>> atime 1379834835.428558020 (2013-09-22 15:27:15)
>>>>>>>>
>>>>>>>> ctime 1379834835.428558020 (2013-09-22 15:27:15)
>>>>>>>>
>>>>>>>> mtime 1379834835.428558020 (2013-09-22 15:27:15)
>>>>>>>>
>>>>>>>> otime 0.0 (1970-01-01 08:00:00)
>>>>>>>>
>>>>>>>> item 7 key (7222 INODE_REF 287) itemoff 3344 itemsize 27
>>>>>>>>
>>>>>>>> index 6 namelen 17 name: dhcpcd-eth0.lease
>>>>>>>>
>>>>>>>> item 8 key (7222 EXTENT_DATA 0) itemoff 3016 itemsize 328
>>>>>>>>
>>>>>>>> generation 5767 type 0 (inline)
>>>>>>>>
>>>>>>>> inline extent data size 307 ram_bytes 307 compression 0 (none)
>>>>>>>>
>>>>>>>> item 9 key (7415 INODE_ITEM 0) itemoff 2856 itemsize 160
>>>>>>>>
>>>>>>>> generation 5904 transid 1330906 size 180 nbytes 0
>>>>>>>>
>>>>>>>> block group 0 mode 40755 links 2 uid 0 gid 0 rdev 0
>>>>>>>>
>>>>>>>> sequence 177 flags 0x0(none)
>>>>>>>>
>>>>>>>> atime 1483277713.141980592 (2017-01-01 21:35:13)
>>>>>>>>
>>>>>>>> ctime 1563162901.234656246 (2019-07-15 11:55:01)
>>>>>>>>
>>>>>>>> mtime 1406534032.158605559 (2014-07-28 15:53:52)
>>>>>>>>
>>>>>>>> otime 0.0 (1970-01-01 08:00:00)
>>>>>>>
>>>>>>> This inode is indeed a directory.
>>>>>>>
>>>>>>> But it has two hard links, which is definitely something unexpected.
>>>>>>>
>>>>>>> Under Linux we shouldn't have any hardlink for directory, as it would
>>>>>>>
>>>>>>> easily lead to loops.
>>>>>>>
>>>>>>>> item 10 key (7415 INODE_REF 260) itemoff 2837 itemsize 19
>>>>>>>>
>>>>>>>> index 28 namelen 9 name: backlight
>>>>>>>
>>>>>>> Its parent inode is 260 in the same root, with the name backlight.
>>>>>>>
>>>>>>>> item 11 key (7415 INODE_REF 286) itemoff 2818 itemsize 19
>>>>>>>>
>>>>>>>> index 3 namelen 9 name: backlight
>>>>>>>
>>>>>>> Another hardlink in inode 286, which is definitely a regular thing.
>>>>>>>
>>>>>>> Btrfs-progs lacks the ability to detect such problem, we need to enhance
>>>>>>>
>>>>>>> it first.
>>>>>>>
>>>>>>> But do you have any idea how this directory get created?
>>>>>>>
>>>>>>> It looks like the content of sysfs.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Qu
>>>>>>>
>>>>>>>> item 12 key (7415 DIR_ITEM 3128336373) itemoff 2746 itemsize 72
>>>>>>>>
>>>>>>>> location key (120417 INODE_ITEM 0) type FILE
>>>>>>>>
>>>>>>>> transid 117279 data_len 0 name_len 42
>>>>>>>>
>>>>>>>> name: pci-0000:00:02.0:backlight:intel_backlight
>>>>>>>>
>>>>>>>> item 13 key (7415 DIR_ITEM 3218198317) itemoff 2705 itemsize 41
>>>>>>>>
>>>>>>>> location key (7487 INODE_ITEM 0) type FILE
>>>>>>>>
>>>>>>>> transid 5992 data_len 0 name_len 11
>>>>>>>>
>>>>>>>> name: acpi_video0
>>>>>>>>
>>>>>>>> item 14 key (7415 DIR_ITEM 3582254411) itemoff 2638 itemsize 67
>>>>>>>>
>>>>>>>> location key (55325 INODE_ITEM 0) type FILE
>>>>>>>>
>>>>>>>> transid 63351 data_len 0 name_len 37
>>>>>>>>
>>>>>>>> name: platform-VPC2004:00:backlight:ideapad
>>>>>>>>
>>>>>>>> item 15 key (7415 DIR_INDEX 2) itemoff 2597 itemsize 41
>>>>>>>>
>>>>>>>> location key (7487 INODE_ITEM 0) type FILE
>>>>>>>>
>>>>>>>> transid 5992 data_len 0 name_len 11
>>>>>>>>
>>>>>>>> name: acpi_video0
>>>>>>>>
>>>>>>>> item 16 key (7415 DIR_INDEX 4) itemoff 2530 itemsize 67
>>>>>>>>
>>>>>>>> location key (55325 INODE_ITEM 0) type FILE
>>>>>>>>
>>>>>>>> transid 63351 data_len 0 name_len 37
>>>>>>>>
>>>>>>>> name: platform-VPC2004:00:backlight:ideapad
>>>>>>>>
>>>>>>>> item 17 key (7415 DIR_INDEX 5) itemoff 2458 itemsize 72
>>>>>>>>
>>>>>>>> location key (120417 INODE_ITEM 0) type FILE
>>>>>>>>
>>>>>>>> transid 117279 data_len 0 name_len 42
>>>>>>>>
>>>>>>>> name: pci-0000:00:02.0:backlight:intel_backlight
>>>>>>>>
>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>>
>>>>>>>> On Saturday, July 17th, 2021 at 3:05 PM, Qu Wenruo quwenruo.btrfs@gmx.com wrote:
>>>>>>>>
>>>>>>>>> On 2021/7/17 上午9:45, pepperpoint@mb.ardentcoding.com wrote:
>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I see this message on dmesg:
>>>>>>>>>>
>>>>>>>>>> [ 2452.256756] BTRFS critical (device dm-0): corrupt leaf: root=363 block=174113599488 slot=9 ino=7415, invalid nlink: has 2 expect no more than 1 for dir
>>>>>>>>>>
>>>>>>>>>> [ 2452.256776] BTRFS error (device dm-0): block=174113599488 read time tree block corruption detected
>>>>>>>>>
>>>>>>>>> Please provide the following dump:
>>>>>>>>>
>>>>>>>>> btrfs ins dump-tree -b 174113599488 /dev/dm-0
>>>>>>>>> =============================================
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Qu
>>>>>>>>>
>>>>>>>>>> When I run btrfs scrub and btrfs check, no error was detected.
>>>>>>>>>>
>>>>>>>>>> I am running Linux 5.12.15-arch1-1 and btrfs-progs v5.12.1
>>>>>>>>>>
>>>>>>>>>> How should I fix this error?

[-- Attachment #2: diff --]
[-- Type: text/plain, Size: 660 bytes --]

diff --git a/fs/btrfs/tree-checker.c b/fs/btrfs/tree-checker.c
index a8b2e0d2c025..99b4309d6db6 100644
--- a/fs/btrfs/tree-checker.c
+++ b/fs/btrfs/tree-checker.c
@@ -1048,12 +1048,14 @@ static int check_inode_item(struct extent_buffer *leaf,
 			return -EUCLEAN;
 		}
 	}
+	/*
 	if (unlikely(S_ISDIR(mode) && btrfs_inode_nlink(leaf, iitem) > 1)) {
 		inode_item_err(leaf, slot,
 		       "invalid nlink: has %u expect no more than 1 for dir",
 			btrfs_inode_nlink(leaf, iitem));
 		return -EUCLEAN;
 	}
+	*/
 	if (unlikely(btrfs_inode_flags(leaf, iitem) & ~BTRFS_INODE_FLAG_MASK)) {
 		inode_item_err(leaf, slot,
 			       "unknown flags detected: 0x%llx",

^ permalink raw reply related	[flat|nested] 75+ messages in thread

* Re: Read time tree block corruption detected
       [not found]               ` <CQeY09U34m7SrT6nTAlkSQrbLJtmyKF1tDfuGDtKUkwJqujMI_nZU4MpGiU4F_Q1U3Lk1aWD1mFCT5SlsOsOcILWECflIw7EhVQTQpy_1Ts=@email.ardentcoding.com>
@ 2021-07-18  5:26                 ` pepperpoint
  2021-07-18  7:15                   ` Qu Wenruo
  0 siblings, 1 reply; 75+ messages in thread
From: pepperpoint @ 2021-07-18  5:26 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Qu Wenruo, linux-btrfs

Hi Qu,

May I know if there are any leads on this? What should I do for now?

Regards,
Lester

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Saturday, July 17th, 2021 at 8:51 PM, pepperpoint@mb.ardentcoding.com wrote:

> Hi Qu,
>
> I run btrfs ins dump-tree -t 363 unmounted but the same error appears. Rerunning btrfs check does not show any error.
>
> Regards,
>
> Lester
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>
> On Saturday, July 17th, 2021 at 6:48 PM, Qu Wenruo - quwenruo.btrfs@gmx.com wrote:
>
> > On 2021/7/17 下午6:34, pepperpoint@mb.ardentcoding.com wrote:
> >
> > > Hi Qu,
> > >
> > > Unfortunately I cannot find subvolume 363
> > >
> > > btrfs subvolume list /run/media/root
> > > ====================================
> > >
> > > ID 361 gen 1814826 top level 584 path @/live/snapshot
> > >
> > > ID 364 gen 1814414 top level 5 path @vtmp/live/snapshot
> > >
> > > ID 369 gen 1814414 top level 5 path @vlmachines/live/snapshot
> > >
> > > ID 493 gen 1814414 top level 5 path @vlportables/live/snapshot
> > >
> > > ID 579 gen 1814828 top level 5 path @vlog/live/snapshot
> > >
> > > ID 580 gen 1814414 top level 5 path @vcache/live/snapshot
> > >
> > > ID 581 gen 1814414 top level 5 path @vlmongodb/live/snapshot
> > >
> > > ID 582 gen 1814414 top level 5 path @vlmysql/live/snapshot
> > >
> > > ID 583 gen 1814414 top level 5 path @vspool/live/snapshot
> > >
> > > ID 584 gen 1814414 top level 5 path @
> > >
> > > ID 598 gen 1813420 top level 584 path @/4/snapshot
> >
> > Maybe 363 is some subvolume get deleted and later snapshot of it still
> >
> > exists.
> >
> > This will be harder to debug.
> >
> > Can you take a btrfs-image dump of your filesystem? (needs to be taken
> >
> > with the fs unmounted).
> >
> > The dumped image will contain your metadata, including file names and
> >
> > directory structures, but no data inside those files.
> >
> > Although btrfs-image has "-s" option to mask the filenames, but
> >
> > considering the filename in this case is useful to locate the inode, I
> >
> > guess it's better to take the image without any "-s" option.
> >
> > > btrfs ins dump-tree -t 363 /dev/dm-0 | grep -A 5 "(286 "
> > > ========================================================
> > >
> > > parent transid verify failed on 174170742784 wanted 1789655 found 1812621
> > >
> > > parent transid verify failed on 174170742784 wanted 1789655 found 1812621
> > >
> > > parent transid verify failed on 174170742784 wanted 1789655 found 1812621
> > >
> > > Ignoring transid failure
> > >
> > > ERROR: child eb corrupted: parent bytenr=174170738688 item=0 parent level=2 child bytenr=174170742784 child level=0
> >
> > This transid mismatch may be a problem when running dump-tree on mounted
> >
> > fs, since you mentioned btrfs check reported no error, there shouldn't
> >
> > be a transid mismatch error.
> >
> > Anyway, if you can upload the btrfs-image dump, it would be much easier
> >
> > for us to debug and find out what's really going.
> >
> > Thanks,
> >
> > Qu
> >
> > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > >
> > > On Saturday, July 17th, 2021 at 6:12 PM, Qu Wenruo wqu@suse.com wrote:
> > >
> > > > On 2021/7/17 下午4:57, pepperpoint@mb.ardentcoding.com wrote:
> > > >
> > > > > Hi Qu,
> > > > >
> > > > > I don't know how the directory was created but last month, I used btrfs device add and btrfs device remove to move the filesystem from one partition to another. It failed because of the same error and was advised to use btrfs replace instead. I don't know if the error also happened before I move the file system as I don't have any previous logs.
> > > >
> > > > It definitely happens before you moving the fs.
> > > >
> > > > As regular dev replacing/add/move only relocates the metadata, but not
> > > >
> > > > touching the fs trees.
> > > >
> > > > > Here is the result when I search for the inodes you mentioned if it helps:
> > > > >
> > > > > find /run/media/root -inum 260 -exec ls -ldi {} \;
> > > > > ==================================================
> > > > >
> > > > > 260 -rw-r--r-- 1 root root 36864 Jun 25 06:22 /run/media/root/@vcache/live/snapshot/app-info/cache/en_US.cache
> > > > >
> > > > > 260 drwx------ 1 mongodb mongodb 136 Sep 12 2020 /run/media/root/@vlmongodb/live/snapshot/diagnostic.data
> > > > >
> > > > > 260 -rw-rw---- 1 mysql mysql 50331648 Sep 13 2015 /run/media/root/@vlmysql/live/snapshot/ib_logfile0
> > > > >
> > > > > 260 -rw-r----- 1 root lp 8641 Mar 5 2014 /run/media/root/@vspool/live/snapshot/cups/d00001-001
> > > > >
> > > > > 260 dr-xr-xr-x 1 root root 0 Sep 13 2013 /run/media/root/@/live/snapshot/sys
> > > > >
> > > > > 260 dr-xr-xr-x 1 root root 0 Sep 13 2013 /run/media/root/@/4/snapshot/sys
> > > >
> > > > Since btrfs can have the same inode number inside different subvolumes,
> > > >
> > > > you may want to limit the search inside subvolume 363.
> > > >
> > > > "-mount" option of find can do that, you only need to locate subvolume
> > > >
> > > > 363 by "btrfs subv list".
> > > >
> > > > But from these output I guess above two "sys" directory are more possible.
> > > >
> > > > Is there any directory named "blaklight" inside those directory?
> > > >
> > > > > find /run/media/root -inum 286 -exec ls -ldi {} \;
> > > > > ==================================================
> > > > >
> > > > > 286 -rw-r--r-- 1 root root 96 Aug 16 2015 /run/media/root/@vcache/live/snapshot/fontconfig/4b172ca7f111e3cffadc3636415fead9-le64.cache-4
> > > > >
> > > > > 286 -rw-rw---- 1 mysql mysql 4096 Sep 15 2013 /run/media/root/@vlmysql/live/snapshot/mysql/columns_priv.MYI
> > > > >
> > > > > 286 -rw-r-----+ 1 root systemd-journal 16777216 Jul 4 01:14 /run/media/root/@vlog/live/snapshot/journal/5098dd7845ae46d3ba1826c68a809a7c/user-1000@fbd9f65d0ea349f6b996716280e6c4dd-00000000002314c5-0005c5cb84a3a438.journal
> > > >
> > > > This is interesting, it means the inode 286 is not accessible.
> > > >
> > > > It can be some orphan inode, but would you dump subvolume 363 then try
> > > >
> > > > to locate the inode 286?
> > > >
> > > > One example command would be:
> > > >
> > > > btrfs ins dump-tree -t 363 <dev> | grep -A 5 "(286 "
> > > > ====================================================
> > > >
> > > > Thanks,
> > > >
> > > > Qu
> > > >
> > > > > Directories with pattern /root/@<dir>/live/snapshot/ are subvolumes and directories with pattern /root/@<dir>/<num>/snapshot/ are snapshots of live.
> > > > >
> > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > >
> > > > > On Saturday, July 17th, 2021 at 4:14 PM, Qu Wenruo quwenruo.btrfs@gmx.com wrote:
> > > > >
> > > > > > On 2021/7/17 下午3:51, pepperpoint@mb.ardentcoding.com wrote:
> > > > > >
> > > > > > > Hi Qu,
> > > > > > >
> > > > > > > Please see below for the dump.
> > > > > > >
> > > > > > > btrfs-progs v5.12.1
> > > > > > >
> > > > > > > leaf 174113599488 items 18 free space 2008 generation 1330906 owner 363
> > > > > > >
> > > > > > > leaf 174113599488 flags 0x1(WRITTEN) backref revision 1
> > > > > > >
> > > > > > > fs uuid a7d327c4-8594-4116-a6f8-8aa2a4162063
> > > > > > >
> > > > > > > chunk uuid f885f49e-14a0-4c80-9c12-c2302b9a0229
> > > > > > >
> > > > > > > item 0 key (5471 INODE_ITEM 0) itemoff 3835 itemsize 160
> > > > > > >
> > > > > > > generation 2063 transid 27726 size 40 nbytes 40
> > > > > > >
> > > > > > > block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
> > > > > > >
> > > > > > > sequence 1501 flags 0x0(none)
> > > > > > >
> > > > > > > atime 1386484844.468769570 (2013-12-08 14:40:44)
> > > > > > >
> > > > > > > ctime 1386484844.468769570 (2013-12-08 14:40:44)
> > > > > > >
> > > > > > > mtime 1386484844.468769570 (2013-12-08 14:40:44)
> > > > > > >
> > > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > > >
> > > > > > > item 1 key (5471 INODE_REF 4399) itemoff 3824 itemsize 11
> > > > > > >
> > > > > > > index 12 namelen 1 name: 8
> > > > > > >
> > > > > > > item 2 key (5471 EXTENT_DATA 0) itemoff 3763 itemsize 61
> > > > > > >
> > > > > > > generation 27726 type 0 (inline)
> > > > > > >
> > > > > > > inline extent data size 40 ram_bytes 40 compression 0 (none)
> > > > > > >
> > > > > > > item 3 key (5645 INODE_ITEM 0) itemoff 3603 itemsize 160
> > > > > > >
> > > > > > > generation 2542 transid 61261 size 40 nbytes 40
> > > > > > >
> > > > > > > block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
> > > > > > >
> > > > > > > sequence 24769 flags 0x0(none)
> > > > > > >
> > > > > > > atime 1394335806.351857522 (2014-03-09 11:30:06)
> > > > > > >
> > > > > > > ctime 1394335827.344389955 (2014-03-09 11:30:27)
> > > > > > >
> > > > > > > mtime 1394335827.344389955 (2014-03-09 11:30:27)
> > > > > > >
> > > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > > >
> > > > > > > item 4 key (5645 INODE_REF 4399) itemoff 3592 itemsize 11
> > > > > > >
> > > > > > > index 13 namelen 1 name: 7
> > > > > > >
> > > > > > > item 5 key (5645 EXTENT_DATA 0) itemoff 3531 itemsize 61
> > > > > > >
> > > > > > > generation 61261 type 0 (inline)
> > > > > > >
> > > > > > > inline extent data size 40 ram_bytes 40 compression 0 (none)
> > > > > > >
> > > > > > > item 6 key (7222 INODE_ITEM 0) itemoff 3371 itemsize 160
> > > > > > >
> > > > > > > generation 5754 transid 5767 size 307 nbytes 307
> > > > > > >
> > > > > > > block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
> > > > > > >
> > > > > > > sequence 7 flags 0x0(none)
> > > > > > >
> > > > > > > atime 1379834835.428558020 (2013-09-22 15:27:15)
> > > > > > >
> > > > > > > ctime 1379834835.428558020 (2013-09-22 15:27:15)
> > > > > > >
> > > > > > > mtime 1379834835.428558020 (2013-09-22 15:27:15)
> > > > > > >
> > > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > > >
> > > > > > > item 7 key (7222 INODE_REF 287) itemoff 3344 itemsize 27
> > > > > > >
> > > > > > > index 6 namelen 17 name: dhcpcd-eth0.lease
> > > > > > >
> > > > > > > item 8 key (7222 EXTENT_DATA 0) itemoff 3016 itemsize 328
> > > > > > >
> > > > > > > generation 5767 type 0 (inline)
> > > > > > >
> > > > > > > inline extent data size 307 ram_bytes 307 compression 0 (none)
> > > > > > >
> > > > > > > item 9 key (7415 INODE_ITEM 0) itemoff 2856 itemsize 160
> > > > > > >
> > > > > > > generation 5904 transid 1330906 size 180 nbytes 0
> > > > > > >
> > > > > > > block group 0 mode 40755 links 2 uid 0 gid 0 rdev 0
> > > > > > >
> > > > > > > sequence 177 flags 0x0(none)
> > > > > > >
> > > > > > > atime 1483277713.141980592 (2017-01-01 21:35:13)
> > > > > > >
> > > > > > > ctime 1563162901.234656246 (2019-07-15 11:55:01)
> > > > > > >
> > > > > > > mtime 1406534032.158605559 (2014-07-28 15:53:52)
> > > > > > >
> > > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > >
> > > > > > This inode is indeed a directory.
> > > > > >
> > > > > > But it has two hard links, which is definitely something unexpected.
> > > > > >
> > > > > > Under Linux we shouldn't have any hardlink for directory, as it would
> > > > > >
> > > > > > easily lead to loops.
> > > > > >
> > > > > > > item 10 key (7415 INODE_REF 260) itemoff 2837 itemsize 19
> > > > > > >
> > > > > > > index 28 namelen 9 name: backlight
> > > > > >
> > > > > > Its parent inode is 260 in the same root, with the name backlight.
> > > > > >
> > > > > > > item 11 key (7415 INODE_REF 286) itemoff 2818 itemsize 19
> > > > > > >
> > > > > > > index 3 namelen 9 name: backlight
> > > > > >
> > > > > > Another hardlink in inode 286, which is definitely a regular thing.
> > > > > >
> > > > > > Btrfs-progs lacks the ability to detect such problem, we need to enhance
> > > > > >
> > > > > > it first.
> > > > > >
> > > > > > But do you have any idea how this directory get created?
> > > > > >
> > > > > > It looks like the content of sysfs.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Qu
> > > > > >
> > > > > > > item 12 key (7415 DIR_ITEM 3128336373) itemoff 2746 itemsize 72
> > > > > > >
> > > > > > > location key (120417 INODE_ITEM 0) type FILE
> > > > > > >
> > > > > > > transid 117279 data_len 0 name_len 42
> > > > > > >
> > > > > > > name: pci-0000:00:02.0:backlight:intel_backlight
> > > > > > >
> > > > > > > item 13 key (7415 DIR_ITEM 3218198317) itemoff 2705 itemsize 41
> > > > > > >
> > > > > > > location key (7487 INODE_ITEM 0) type FILE
> > > > > > >
> > > > > > > transid 5992 data_len 0 name_len 11
> > > > > > >
> > > > > > > name: acpi_video0
> > > > > > >
> > > > > > > item 14 key (7415 DIR_ITEM 3582254411) itemoff 2638 itemsize 67
> > > > > > >
> > > > > > > location key (55325 INODE_ITEM 0) type FILE
> > > > > > >
> > > > > > > transid 63351 data_len 0 name_len 37
> > > > > > >
> > > > > > > name: platform-VPC2004:00:backlight:ideapad
> > > > > > >
> > > > > > > item 15 key (7415 DIR_INDEX 2) itemoff 2597 itemsize 41
> > > > > > >
> > > > > > > location key (7487 INODE_ITEM 0) type FILE
> > > > > > >
> > > > > > > transid 5992 data_len 0 name_len 11
> > > > > > >
> > > > > > > name: acpi_video0
> > > > > > >
> > > > > > > item 16 key (7415 DIR_INDEX 4) itemoff 2530 itemsize 67
> > > > > > >
> > > > > > > location key (55325 INODE_ITEM 0) type FILE
> > > > > > >
> > > > > > > transid 63351 data_len 0 name_len 37
> > > > > > >
> > > > > > > name: platform-VPC2004:00:backlight:ideapad
> > > > > > >
> > > > > > > item 17 key (7415 DIR_INDEX 5) itemoff 2458 itemsize 72
> > > > > > >
> > > > > > > location key (120417 INODE_ITEM 0) type FILE
> > > > > > >
> > > > > > > transid 117279 data_len 0 name_len 42
> > > > > > >
> > > > > > > name: pci-0000:00:02.0:backlight:intel_backlight
> > > > > > >
> > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > > > >
> > > > > > > On Saturday, July 17th, 2021 at 3:05 PM, Qu Wenruo quwenruo.btrfs@gmx.com wrote:
> > > > > > >
> > > > > > > > On 2021/7/17 上午9:45, pepperpoint@mb.ardentcoding.com wrote:
> > > > > > > >
> > > > > > > > > Hello,
> > > > > > > > >
> > > > > > > > > I see this message on dmesg:
> > > > > > > > >
> > > > > > > > > [ 2452.256756] BTRFS critical (device dm-0): corrupt leaf: root=363 block=174113599488 slot=9 ino=7415, invalid nlink: has 2 expect no more than 1 for dir
> > > > > > > > >
> > > > > > > > > [ 2452.256776] BTRFS error (device dm-0): block=174113599488 read time tree block corruption detected
> > > > > > > >
> > > > > > > > Please provide the following dump:
> > > > > > > >
> > > > > > > > btrfs ins dump-tree -b 174113599488 /dev/dm-0
> > > > > > > > =============================================
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Qu
> > > > > > > >
> > > > > > > > > When I run btrfs scrub and btrfs check, no error was detected.
> > > > > > > > >
> > > > > > > > > I am running Linux 5.12.15-arch1-1 and btrfs-progs v5.12.1
> > > > > > > > >
> > > > > > > > > How should I fix this error?

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: Read time tree block corruption detected
       [not found]             ` <162651892663.6.17938009695497100557.10189169@simplelogin.co>
@ 2021-07-17 12:51               ` pepperpoint
       [not found]               ` <CQeY09U34m7SrT6nTAlkSQrbLJtmyKF1tDfuGDtKUkwJqujMI_nZU4MpGiU4F_Q1U3Lk1aWD1mFCT5SlsOsOcILWECflIw7EhVQTQpy_1Ts=@email.ardentcoding.com>
  1 sibling, 0 replies; 75+ messages in thread
From: pepperpoint @ 2021-07-17 12:51 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Qu Wenruo, linux-btrfs

Hi Qu,

I run btrfs ins dump-tree -t 363 unmounted but the same error appears. Rerunning btrfs check does not show any error.

Regards,
Lester

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Saturday, July 17th, 2021 at 6:48 PM, Qu Wenruo -  <quwenruo.btrfs@gmx.com> wrote:

> On 2021/7/17 下午6:34, pepperpoint@mb.ardentcoding.com wrote:
>
> > Hi Qu,
> >
> > Unfortunately I cannot find subvolume 363
> >
> > btrfs subvolume list /run/media/root
> > ====================================
> >
> > ID 361 gen 1814826 top level 584 path @/live/snapshot
> >
> > ID 364 gen 1814414 top level 5 path @vtmp/live/snapshot
> >
> > ID 369 gen 1814414 top level 5 path @vlmachines/live/snapshot
> >
> > ID 493 gen 1814414 top level 5 path @vlportables/live/snapshot
> >
> > ID 579 gen 1814828 top level 5 path @vlog/live/snapshot
> >
> > ID 580 gen 1814414 top level 5 path @vcache/live/snapshot
> >
> > ID 581 gen 1814414 top level 5 path @vlmongodb/live/snapshot
> >
> > ID 582 gen 1814414 top level 5 path @vlmysql/live/snapshot
> >
> > ID 583 gen 1814414 top level 5 path @vspool/live/snapshot
> >
> > ID 584 gen 1814414 top level 5 path @
> >
> > ID 598 gen 1813420 top level 584 path @/4/snapshot
>
> Maybe 363 is some subvolume get deleted and later snapshot of it still
>
> exists.
>
> This will be harder to debug.
>
> Can you take a btrfs-image dump of your filesystem? (needs to be taken
>
> with the fs unmounted).
>
> The dumped image will contain your metadata, including file names and
>
> directory structures, but no data inside those files.
>
> Although btrfs-image has "-s" option to mask the filenames, but
>
> considering the filename in this case is useful to locate the inode, I
>
> guess it's better to take the image without any "-s" option.
>
> > btrfs ins dump-tree -t 363 /dev/dm-0 | grep -A 5 "(286 "
> > ========================================================
> >
> > parent transid verify failed on 174170742784 wanted 1789655 found 1812621
> >
> > parent transid verify failed on 174170742784 wanted 1789655 found 1812621
> >
> > parent transid verify failed on 174170742784 wanted 1789655 found 1812621
> >
> > Ignoring transid failure
> >
> > ERROR: child eb corrupted: parent bytenr=174170738688 item=0 parent level=2 child bytenr=174170742784 child level=0
>
> This transid mismatch may be a problem when running dump-tree on mounted
>
> fs, since you mentioned btrfs check reported no error, there shouldn't
>
> be a transid mismatch error.
>
> Anyway, if you can upload the btrfs-image dump, it would be much easier
>
> for us to debug and find out what's really going.
>
> Thanks,
>
> Qu
>
> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> >
> > On Saturday, July 17th, 2021 at 6:12 PM, Qu Wenruo wqu@suse.com wrote:
> >
> > > On 2021/7/17 下午4:57, pepperpoint@mb.ardentcoding.com wrote:
> > >
> > > > Hi Qu,
> > > >
> > > > I don't know how the directory was created but last month, I used btrfs device add and btrfs device remove to move the filesystem from one partition to another. It failed because of the same error and was advised to use btrfs replace instead. I don't know if the error also happened before I move the file system as I don't have any previous logs.
> > >
> > > It definitely happens before you moving the fs.
> > >
> > > As regular dev replacing/add/move only relocates the metadata, but not
> > >
> > > touching the fs trees.
> > >
> > > > Here is the result when I search for the inodes you mentioned if it helps:
> > > >
> > > > find /run/media/root -inum 260 -exec ls -ldi {} \;
> > > > ==================================================
> > > >
> > > > 260 -rw-r--r-- 1 root root 36864 Jun 25 06:22 /run/media/root/@vcache/live/snapshot/app-info/cache/en_US.cache
> > > >
> > > > 260 drwx------ 1 mongodb mongodb 136 Sep 12 2020 /run/media/root/@vlmongodb/live/snapshot/diagnostic.data
> > > >
> > > > 260 -rw-rw---- 1 mysql mysql 50331648 Sep 13 2015 /run/media/root/@vlmysql/live/snapshot/ib_logfile0
> > > >
> > > > 260 -rw-r----- 1 root lp 8641 Mar 5 2014 /run/media/root/@vspool/live/snapshot/cups/d00001-001
> > > >
> > > > 260 dr-xr-xr-x 1 root root 0 Sep 13 2013 /run/media/root/@/live/snapshot/sys
> > > >
> > > > 260 dr-xr-xr-x 1 root root 0 Sep 13 2013 /run/media/root/@/4/snapshot/sys
> > >
> > > Since btrfs can have the same inode number inside different subvolumes,
> > >
> > > you may want to limit the search inside subvolume 363.
> > >
> > > "-mount" option of find can do that, you only need to locate subvolume
> > >
> > > 363 by "btrfs subv list".
> > >
> > > But from these output I guess above two "sys" directory are more possible.
> > >
> > > Is there any directory named "blaklight" inside those directory?
> > >
> > > > find /run/media/root -inum 286 -exec ls -ldi {} \;
> > > > ==================================================
> > > >
> > > > 286 -rw-r--r-- 1 root root 96 Aug 16 2015 /run/media/root/@vcache/live/snapshot/fontconfig/4b172ca7f111e3cffadc3636415fead9-le64.cache-4
> > > >
> > > > 286 -rw-rw---- 1 mysql mysql 4096 Sep 15 2013 /run/media/root/@vlmysql/live/snapshot/mysql/columns_priv.MYI
> > > >
> > > > 286 -rw-r-----+ 1 root systemd-journal 16777216 Jul 4 01:14 /run/media/root/@vlog/live/snapshot/journal/5098dd7845ae46d3ba1826c68a809a7c/user-1000@fbd9f65d0ea349f6b996716280e6c4dd-00000000002314c5-0005c5cb84a3a438.journal
> > >
> > > This is interesting, it means the inode 286 is not accessible.
> > >
> > > It can be some orphan inode, but would you dump subvolume 363 then try
> > >
> > > to locate the inode 286?
> > >
> > > One example command would be:
> > >
> > > btrfs ins dump-tree -t 363 <dev> | grep -A 5 "(286 "
> > > ====================================================
> > >
> > > Thanks,
> > >
> > > Qu
> > >
> > > > Directories with pattern /root/@<dir>/live/snapshot/ are subvolumes and directories with pattern /root/@<dir>/<num>/snapshot/ are snapshots of live.
> > > >
> > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > >
> > > > On Saturday, July 17th, 2021 at 4:14 PM, Qu Wenruo quwenruo.btrfs@gmx.com wrote:
> > > >
> > > > > On 2021/7/17 下午3:51, pepperpoint@mb.ardentcoding.com wrote:
> > > > >
> > > > > > Hi Qu,
> > > > > >
> > > > > > Please see below for the dump.
> > > > > >
> > > > > > btrfs-progs v5.12.1
> > > > > >
> > > > > > leaf 174113599488 items 18 free space 2008 generation 1330906 owner 363
> > > > > >
> > > > > > leaf 174113599488 flags 0x1(WRITTEN) backref revision 1
> > > > > >
> > > > > > fs uuid a7d327c4-8594-4116-a6f8-8aa2a4162063
> > > > > >
> > > > > > chunk uuid f885f49e-14a0-4c80-9c12-c2302b9a0229
> > > > > >
> > > > > > item 0 key (5471 INODE_ITEM 0) itemoff 3835 itemsize 160
> > > > > >
> > > > > > generation 2063 transid 27726 size 40 nbytes 40
> > > > > >
> > > > > > block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
> > > > > >
> > > > > > sequence 1501 flags 0x0(none)
> > > > > >
> > > > > > atime 1386484844.468769570 (2013-12-08 14:40:44)
> > > > > >
> > > > > > ctime 1386484844.468769570 (2013-12-08 14:40:44)
> > > > > >
> > > > > > mtime 1386484844.468769570 (2013-12-08 14:40:44)
> > > > > >
> > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > >
> > > > > > item 1 key (5471 INODE_REF 4399) itemoff 3824 itemsize 11
> > > > > >
> > > > > > index 12 namelen 1 name: 8
> > > > > >
> > > > > > item 2 key (5471 EXTENT_DATA 0) itemoff 3763 itemsize 61
> > > > > >
> > > > > > generation 27726 type 0 (inline)
> > > > > >
> > > > > > inline extent data size 40 ram_bytes 40 compression 0 (none)
> > > > > >
> > > > > > item 3 key (5645 INODE_ITEM 0) itemoff 3603 itemsize 160
> > > > > >
> > > > > > generation 2542 transid 61261 size 40 nbytes 40
> > > > > >
> > > > > > block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
> > > > > >
> > > > > > sequence 24769 flags 0x0(none)
> > > > > >
> > > > > > atime 1394335806.351857522 (2014-03-09 11:30:06)
> > > > > >
> > > > > > ctime 1394335827.344389955 (2014-03-09 11:30:27)
> > > > > >
> > > > > > mtime 1394335827.344389955 (2014-03-09 11:30:27)
> > > > > >
> > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > >
> > > > > > item 4 key (5645 INODE_REF 4399) itemoff 3592 itemsize 11
> > > > > >
> > > > > > index 13 namelen 1 name: 7
> > > > > >
> > > > > > item 5 key (5645 EXTENT_DATA 0) itemoff 3531 itemsize 61
> > > > > >
> > > > > > generation 61261 type 0 (inline)
> > > > > >
> > > > > > inline extent data size 40 ram_bytes 40 compression 0 (none)
> > > > > >
> > > > > > item 6 key (7222 INODE_ITEM 0) itemoff 3371 itemsize 160
> > > > > >
> > > > > > generation 5754 transid 5767 size 307 nbytes 307
> > > > > >
> > > > > > block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
> > > > > >
> > > > > > sequence 7 flags 0x0(none)
> > > > > >
> > > > > > atime 1379834835.428558020 (2013-09-22 15:27:15)
> > > > > >
> > > > > > ctime 1379834835.428558020 (2013-09-22 15:27:15)
> > > > > >
> > > > > > mtime 1379834835.428558020 (2013-09-22 15:27:15)
> > > > > >
> > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > > >
> > > > > > item 7 key (7222 INODE_REF 287) itemoff 3344 itemsize 27
> > > > > >
> > > > > > index 6 namelen 17 name: dhcpcd-eth0.lease
> > > > > >
> > > > > > item 8 key (7222 EXTENT_DATA 0) itemoff 3016 itemsize 328
> > > > > >
> > > > > > generation 5767 type 0 (inline)
> > > > > >
> > > > > > inline extent data size 307 ram_bytes 307 compression 0 (none)
> > > > > >
> > > > > > item 9 key (7415 INODE_ITEM 0) itemoff 2856 itemsize 160
> > > > > >
> > > > > > generation 5904 transid 1330906 size 180 nbytes 0
> > > > > >
> > > > > > block group 0 mode 40755 links 2 uid 0 gid 0 rdev 0
> > > > > >
> > > > > > sequence 177 flags 0x0(none)
> > > > > >
> > > > > > atime 1483277713.141980592 (2017-01-01 21:35:13)
> > > > > >
> > > > > > ctime 1563162901.234656246 (2019-07-15 11:55:01)
> > > > > >
> > > > > > mtime 1406534032.158605559 (2014-07-28 15:53:52)
> > > > > >
> > > > > > otime 0.0 (1970-01-01 08:00:00)
> > > > >
> > > > > This inode is indeed a directory.
> > > > >
> > > > > But it has two hard links, which is definitely something unexpected.
> > > > >
> > > > > Under Linux we shouldn't have any hardlink for directory, as it would
> > > > >
> > > > > easily lead to loops.
> > > > >
> > > > > > item 10 key (7415 INODE_REF 260) itemoff 2837 itemsize 19
> > > > > >
> > > > > > index 28 namelen 9 name: backlight
> > > > >
> > > > > Its parent inode is 260 in the same root, with the name backlight.
> > > > >
> > > > > > item 11 key (7415 INODE_REF 286) itemoff 2818 itemsize 19
> > > > > >
> > > > > > index 3 namelen 9 name: backlight
> > > > >
> > > > > Another hardlink in inode 286, which is definitely a regular thing.
> > > > >
> > > > > Btrfs-progs lacks the ability to detect such problem, we need to enhance
> > > > >
> > > > > it first.
> > > > >
> > > > > But do you have any idea how this directory get created?
> > > > >
> > > > > It looks like the content of sysfs.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Qu
> > > > >
> > > > > > item 12 key (7415 DIR_ITEM 3128336373) itemoff 2746 itemsize 72
> > > > > >
> > > > > > location key (120417 INODE_ITEM 0) type FILE
> > > > > >
> > > > > > transid 117279 data_len 0 name_len 42
> > > > > >
> > > > > > name: pci-0000:00:02.0:backlight:intel_backlight
> > > > > >
> > > > > > item 13 key (7415 DIR_ITEM 3218198317) itemoff 2705 itemsize 41
> > > > > >
> > > > > > location key (7487 INODE_ITEM 0) type FILE
> > > > > >
> > > > > > transid 5992 data_len 0 name_len 11
> > > > > >
> > > > > > name: acpi_video0
> > > > > >
> > > > > > item 14 key (7415 DIR_ITEM 3582254411) itemoff 2638 itemsize 67
> > > > > >
> > > > > > location key (55325 INODE_ITEM 0) type FILE
> > > > > >
> > > > > > transid 63351 data_len 0 name_len 37
> > > > > >
> > > > > > name: platform-VPC2004:00:backlight:ideapad
> > > > > >
> > > > > > item 15 key (7415 DIR_INDEX 2) itemoff 2597 itemsize 41
> > > > > >
> > > > > > location key (7487 INODE_ITEM 0) type FILE
> > > > > >
> > > > > > transid 5992 data_len 0 name_len 11
> > > > > >
> > > > > > name: acpi_video0
> > > > > >
> > > > > > item 16 key (7415 DIR_INDEX 4) itemoff 2530 itemsize 67
> > > > > >
> > > > > > location key (55325 INODE_ITEM 0) type FILE
> > > > > >
> > > > > > transid 63351 data_len 0 name_len 37
> > > > > >
> > > > > > name: platform-VPC2004:00:backlight:ideapad
> > > > > >
> > > > > > item 17 key (7415 DIR_INDEX 5) itemoff 2458 itemsize 72
> > > > > >
> > > > > > location key (120417 INODE_ITEM 0) type FILE
> > > > > >
> > > > > > transid 117279 data_len 0 name_len 42
> > > > > >
> > > > > > name: pci-0000:00:02.0:backlight:intel_backlight
> > > > > >
> > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > > >
> > > > > > On Saturday, July 17th, 2021 at 3:05 PM, Qu Wenruo quwenruo.btrfs@gmx.com wrote:
> > > > > >
> > > > > > > On 2021/7/17 上午9:45, pepperpoint@mb.ardentcoding.com wrote:
> > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > I see this message on dmesg:
> > > > > > > >
> > > > > > > > [ 2452.256756] BTRFS critical (device dm-0): corrupt leaf: root=363 block=174113599488 slot=9 ino=7415, invalid nlink: has 2 expect no more than 1 for dir
> > > > > > > >
> > > > > > > > [ 2452.256776] BTRFS error (device dm-0): block=174113599488 read time tree block corruption detected
> > > > > > >
> > > > > > > Please provide the following dump:
> > > > > > >
> > > > > > > btrfs ins dump-tree -b 174113599488 /dev/dm-0
> > > > > > > =============================================
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Qu
> > > > > > >
> > > > > > > > When I run btrfs scrub and btrfs check, no error was detected.
> > > > > > > >
> > > > > > > > I am running Linux 5.12.15-arch1-1 and btrfs-progs v5.12.1
> > > > > > > >
> > > > > > > > How should I fix this error?

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: Read time tree block corruption detected
       [not found]           ` <162651809235.7.7061042874033963922.10188873@mb.ardentcoding.com>
@ 2021-07-17 10:48             ` Qu Wenruo
       [not found]             ` <162651892663.6.17938009695497100557.10189169@simplelogin.co>
  1 sibling, 0 replies; 75+ messages in thread
From: Qu Wenruo @ 2021-07-17 10:48 UTC (permalink / raw)
  To: pepperpoint, Qu Wenruo; +Cc: linux-btrfs



On 2021/7/17 下午6:34, pepperpoint@mb.ardentcoding.com wrote:
> Hi Qu,
>
> Unfortunately I cannot find subvolume 363
> # btrfs subvolume list /run/media/root
> ID 361 gen 1814826 top level 584 path @/live/snapshot
> ID 364 gen 1814414 top level 5 path @vtmp/live/snapshot
> ID 369 gen 1814414 top level 5 path @vlmachines/live/snapshot
> ID 493 gen 1814414 top level 5 path @vlportables/live/snapshot
> ID 579 gen 1814828 top level 5 path @vlog/live/snapshot
> ID 580 gen 1814414 top level 5 path @vcache/live/snapshot
> ID 581 gen 1814414 top level 5 path @vlmongodb/live/snapshot
> ID 582 gen 1814414 top level 5 path @vlmysql/live/snapshot
> ID 583 gen 1814414 top level 5 path @vspool/live/snapshot
> ID 584 gen 1814414 top level 5 path @
> ID 598 gen 1813420 top level 584 path @/4/snapshot

Maybe 363 is some subvolume get deleted and later snapshot of it still
exists.

This will be harder to debug.

Can you take a btrfs-image dump of your filesystem? (needs to be taken
with the fs unmounted).

The dumped image will contain your metadata, including file names and
directory structures, but no data inside those files.

Although btrfs-image has "-s" option to mask the filenames, but
considering the filename in this case is useful to locate the inode, I
guess it's better to take the image without any "-s" option.
>
> # btrfs ins dump-tree -t 363 /dev/dm-0 | grep -A 5 "(286 "
> parent transid verify failed on 174170742784 wanted 1789655 found 1812621
> parent transid verify failed on 174170742784 wanted 1789655 found 1812621
> parent transid verify failed on 174170742784 wanted 1789655 found 1812621
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=174170738688 item=0 parent level=2 child bytenr=174170742784 child level=0

This transid mismatch may be a problem when running dump-tree on mounted
fs, since you mentioned btrfs check reported no error, there shouldn't
be a transid mismatch error.

Anyway, if you can upload the btrfs-image dump, it would be much easier
for us to debug and find out what's really going.

Thanks,
Qu

>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>
> On Saturday, July 17th, 2021 at 6:12 PM, Qu Wenruo <wqu@suse.com> wrote:
>
>> On 2021/7/17 下午4:57, pepperpoint@mb.ardentcoding.com wrote:
>>
>>> Hi Qu,
>>>
>>> I don't know how the directory was created but last month, I used btrfs device add and btrfs device remove to move the filesystem from one partition to another. It failed because of the same error and was advised to use btrfs replace instead. I don't know if the error also happened before I move the file system as I don't have any previous logs.
>>
>> It definitely happens before you moving the fs.
>>
>> As regular dev replacing/add/move only relocates the metadata, but not
>>
>> touching the fs trees.
>>
>>> Here is the result when I search for the inodes you mentioned if it helps:
>>>
>>> find /run/media/root -inum 260 -exec ls -ldi {} \;
>>> ==================================================
>>>
>>> 260 -rw-r--r-- 1 root root 36864 Jun 25 06:22 /run/media/root/@vcache/live/snapshot/app-info/cache/en_US.cache
>>>
>>> 260 drwx------ 1 mongodb mongodb 136 Sep 12 2020 /run/media/root/@vlmongodb/live/snapshot/diagnostic.data
>>>
>>> 260 -rw-rw---- 1 mysql mysql 50331648 Sep 13 2015 /run/media/root/@vlmysql/live/snapshot/ib_logfile0
>>>
>>> 260 -rw-r----- 1 root lp 8641 Mar 5 2014 /run/media/root/@vspool/live/snapshot/cups/d00001-001
>>>
>>> 260 dr-xr-xr-x 1 root root 0 Sep 13 2013 /run/media/root/@/live/snapshot/sys
>>>
>>> 260 dr-xr-xr-x 1 root root 0 Sep 13 2013 /run/media/root/@/4/snapshot/sys
>>
>> Since btrfs can have the same inode number inside different subvolumes,
>>
>> you may want to limit the search inside subvolume 363.
>>
>> "-mount" option of find can do that, you only need to locate subvolume
>>
>> 363 by "btrfs subv list".
>>
>> But from these output I guess above two "sys" directory are more possible.
>>
>> Is there any directory named "blaklight" inside those directory?
>>
>>> find /run/media/root -inum 286 -exec ls -ldi {} \;
>>> ==================================================
>>>
>>> 286 -rw-r--r-- 1 root root 96 Aug 16 2015 /run/media/root/@vcache/live/snapshot/fontconfig/4b172ca7f111e3cffadc3636415fead9-le64.cache-4
>>>
>>> 286 -rw-rw---- 1 mysql mysql 4096 Sep 15 2013 /run/media/root/@vlmysql/live/snapshot/mysql/columns_priv.MYI
>>>
>>> 286 -rw-r-----+ 1 root systemd-journal 16777216 Jul 4 01:14 /run/media/root/@vlog/live/snapshot/journal/5098dd7845ae46d3ba1826c68a809a7c/user-1000@fbd9f65d0ea349f6b996716280e6c4dd-00000000002314c5-0005c5cb84a3a438.journal
>>
>> This is interesting, it means the inode 286 is not accessible.
>>
>> It can be some orphan inode, but would you dump subvolume 363 then try
>>
>> to locate the inode 286?
>>
>> One example command would be:
>>
>> btrfs ins dump-tree -t 363 <dev> | grep -A 5 "(286 "
>> ====================================================
>>
>> Thanks,
>>
>> Qu
>>
>>> Directories with pattern /root/@<dir>/live/snapshot/ are subvolumes and directories with pattern /root/@<dir>/<num>/snapshot/ are snapshots of live.
>>>
>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>
>>> On Saturday, July 17th, 2021 at 4:14 PM, Qu Wenruo quwenruo.btrfs@gmx.com wrote:
>>>
>>>> On 2021/7/17 下午3:51, pepperpoint@mb.ardentcoding.com wrote:
>>>>
>>>>> Hi Qu,
>>>>>
>>>>> Please see below for the dump.
>>>>>
>>>>> btrfs-progs v5.12.1
>>>>>
>>>>> leaf 174113599488 items 18 free space 2008 generation 1330906 owner 363
>>>>>
>>>>> leaf 174113599488 flags 0x1(WRITTEN) backref revision 1
>>>>>
>>>>> fs uuid a7d327c4-8594-4116-a6f8-8aa2a4162063
>>>>>
>>>>> chunk uuid f885f49e-14a0-4c80-9c12-c2302b9a0229
>>>>>
>>>>> item 0 key (5471 INODE_ITEM 0) itemoff 3835 itemsize 160
>>>>>
>>>>> generation 2063 transid 27726 size 40 nbytes 40
>>>>>
>>>>> block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
>>>>>
>>>>> sequence 1501 flags 0x0(none)
>>>>>
>>>>> atime 1386484844.468769570 (2013-12-08 14:40:44)
>>>>>
>>>>> ctime 1386484844.468769570 (2013-12-08 14:40:44)
>>>>>
>>>>> mtime 1386484844.468769570 (2013-12-08 14:40:44)
>>>>>
>>>>> otime 0.0 (1970-01-01 08:00:00)
>>>>>
>>>>> item 1 key (5471 INODE_REF 4399) itemoff 3824 itemsize 11
>>>>>
>>>>> index 12 namelen 1 name: 8
>>>>>
>>>>> item 2 key (5471 EXTENT_DATA 0) itemoff 3763 itemsize 61
>>>>>
>>>>> generation 27726 type 0 (inline)
>>>>>
>>>>> inline extent data size 40 ram_bytes 40 compression 0 (none)
>>>>>
>>>>> item 3 key (5645 INODE_ITEM 0) itemoff 3603 itemsize 160
>>>>>
>>>>> generation 2542 transid 61261 size 40 nbytes 40
>>>>>
>>>>> block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
>>>>>
>>>>> sequence 24769 flags 0x0(none)
>>>>>
>>>>> atime 1394335806.351857522 (2014-03-09 11:30:06)
>>>>>
>>>>> ctime 1394335827.344389955 (2014-03-09 11:30:27)
>>>>>
>>>>> mtime 1394335827.344389955 (2014-03-09 11:30:27)
>>>>>
>>>>> otime 0.0 (1970-01-01 08:00:00)
>>>>>
>>>>> item 4 key (5645 INODE_REF 4399) itemoff 3592 itemsize 11
>>>>>
>>>>> index 13 namelen 1 name: 7
>>>>>
>>>>> item 5 key (5645 EXTENT_DATA 0) itemoff 3531 itemsize 61
>>>>>
>>>>> generation 61261 type 0 (inline)
>>>>>
>>>>> inline extent data size 40 ram_bytes 40 compression 0 (none)
>>>>>
>>>>> item 6 key (7222 INODE_ITEM 0) itemoff 3371 itemsize 160
>>>>>
>>>>> generation 5754 transid 5767 size 307 nbytes 307
>>>>>
>>>>> block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
>>>>>
>>>>> sequence 7 flags 0x0(none)
>>>>>
>>>>> atime 1379834835.428558020 (2013-09-22 15:27:15)
>>>>>
>>>>> ctime 1379834835.428558020 (2013-09-22 15:27:15)
>>>>>
>>>>> mtime 1379834835.428558020 (2013-09-22 15:27:15)
>>>>>
>>>>> otime 0.0 (1970-01-01 08:00:00)
>>>>>
>>>>> item 7 key (7222 INODE_REF 287) itemoff 3344 itemsize 27
>>>>>
>>>>> index 6 namelen 17 name: dhcpcd-eth0.lease
>>>>>
>>>>> item 8 key (7222 EXTENT_DATA 0) itemoff 3016 itemsize 328
>>>>>
>>>>> generation 5767 type 0 (inline)
>>>>>
>>>>> inline extent data size 307 ram_bytes 307 compression 0 (none)
>>>>>
>>>>> item 9 key (7415 INODE_ITEM 0) itemoff 2856 itemsize 160
>>>>>
>>>>> generation 5904 transid 1330906 size 180 nbytes 0
>>>>>
>>>>> block group 0 mode 40755 links 2 uid 0 gid 0 rdev 0
>>>>>
>>>>> sequence 177 flags 0x0(none)
>>>>>
>>>>> atime 1483277713.141980592 (2017-01-01 21:35:13)
>>>>>
>>>>> ctime 1563162901.234656246 (2019-07-15 11:55:01)
>>>>>
>>>>> mtime 1406534032.158605559 (2014-07-28 15:53:52)
>>>>>
>>>>> otime 0.0 (1970-01-01 08:00:00)
>>>>
>>>> This inode is indeed a directory.
>>>>
>>>> But it has two hard links, which is definitely something unexpected.
>>>>
>>>> Under Linux we shouldn't have any hardlink for directory, as it would
>>>>
>>>> easily lead to loops.
>>>>
>>>>> item 10 key (7415 INODE_REF 260) itemoff 2837 itemsize 19
>>>>>
>>>>> index 28 namelen 9 name: backlight
>>>>
>>>> Its parent inode is 260 in the same root, with the name backlight.
>>>>
>>>>> item 11 key (7415 INODE_REF 286) itemoff 2818 itemsize 19
>>>>>
>>>>> index 3 namelen 9 name: backlight
>>>>
>>>> Another hardlink in inode 286, which is definitely a regular thing.
>>>>
>>>> Btrfs-progs lacks the ability to detect such problem, we need to enhance
>>>>
>>>> it first.
>>>>
>>>> But do you have any idea how this directory get created?
>>>>
>>>> It looks like the content of sysfs.
>>>>
>>>> Thanks,
>>>>
>>>> Qu
>>>>
>>>>> item 12 key (7415 DIR_ITEM 3128336373) itemoff 2746 itemsize 72
>>>>>
>>>>> location key (120417 INODE_ITEM 0) type FILE
>>>>>
>>>>> transid 117279 data_len 0 name_len 42
>>>>>
>>>>> name: pci-0000:00:02.0:backlight:intel_backlight
>>>>>
>>>>> item 13 key (7415 DIR_ITEM 3218198317) itemoff 2705 itemsize 41
>>>>>
>>>>> location key (7487 INODE_ITEM 0) type FILE
>>>>>
>>>>> transid 5992 data_len 0 name_len 11
>>>>>
>>>>> name: acpi_video0
>>>>>
>>>>> item 14 key (7415 DIR_ITEM 3582254411) itemoff 2638 itemsize 67
>>>>>
>>>>> location key (55325 INODE_ITEM 0) type FILE
>>>>>
>>>>> transid 63351 data_len 0 name_len 37
>>>>>
>>>>> name: platform-VPC2004:00:backlight:ideapad
>>>>>
>>>>> item 15 key (7415 DIR_INDEX 2) itemoff 2597 itemsize 41
>>>>>
>>>>> location key (7487 INODE_ITEM 0) type FILE
>>>>>
>>>>> transid 5992 data_len 0 name_len 11
>>>>>
>>>>> name: acpi_video0
>>>>>
>>>>> item 16 key (7415 DIR_INDEX 4) itemoff 2530 itemsize 67
>>>>>
>>>>> location key (55325 INODE_ITEM 0) type FILE
>>>>>
>>>>> transid 63351 data_len 0 name_len 37
>>>>>
>>>>> name: platform-VPC2004:00:backlight:ideapad
>>>>>
>>>>> item 17 key (7415 DIR_INDEX 5) itemoff 2458 itemsize 72
>>>>>
>>>>> location key (120417 INODE_ITEM 0) type FILE
>>>>>
>>>>> transid 117279 data_len 0 name_len 42
>>>>>
>>>>> name: pci-0000:00:02.0:backlight:intel_backlight
>>>>>
>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>
>>>>> On Saturday, July 17th, 2021 at 3:05 PM, Qu Wenruo quwenruo.btrfs@gmx.com wrote:
>>>>>
>>>>>> On 2021/7/17 上午9:45, pepperpoint@mb.ardentcoding.com wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I see this message on dmesg:
>>>>>>>
>>>>>>> [ 2452.256756] BTRFS critical (device dm-0): corrupt leaf: root=363 block=174113599488 slot=9 ino=7415, invalid nlink: has 2 expect no more than 1 for dir
>>>>>>>
>>>>>>> [ 2452.256776] BTRFS error (device dm-0): block=174113599488 read time tree block corruption detected
>>>>>>
>>>>>> Please provide the following dump:
>>>>>>
>>>>>> btrfs ins dump-tree -b 174113599488 /dev/dm-0
>>>>>> =============================================
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Qu
>>>>>>
>>>>>>> When I run btrfs scrub and btrfs check, no error was detected.
>>>>>>>
>>>>>>> I am running Linux 5.12.15-arch1-1 and btrfs-progs v5.12.1
>>>>>>>
>>>>>>> How should I fix this error?

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: Read time tree block corruption detected
       [not found]         ` <162651674065.6.7912816287233908759.10188327@simplelogin.co>
@ 2021-07-17 10:34           ` pepperpoint
       [not found]           ` <162651809235.7.7061042874033963922.10188873@mb.ardentcoding.com>
  1 sibling, 0 replies; 75+ messages in thread
From: pepperpoint @ 2021-07-17 10:34 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Qu Wenruo, linux-btrfs

Hi Qu,

Unfortunately I cannot find subvolume 363
# btrfs subvolume list /run/media/root
ID 361 gen 1814826 top level 584 path @/live/snapshot
ID 364 gen 1814414 top level 5 path @vtmp/live/snapshot
ID 369 gen 1814414 top level 5 path @vlmachines/live/snapshot
ID 493 gen 1814414 top level 5 path @vlportables/live/snapshot
ID 579 gen 1814828 top level 5 path @vlog/live/snapshot
ID 580 gen 1814414 top level 5 path @vcache/live/snapshot
ID 581 gen 1814414 top level 5 path @vlmongodb/live/snapshot
ID 582 gen 1814414 top level 5 path @vlmysql/live/snapshot
ID 583 gen 1814414 top level 5 path @vspool/live/snapshot
ID 584 gen 1814414 top level 5 path @
ID 598 gen 1813420 top level 584 path @/4/snapshot

# btrfs ins dump-tree -t 363 /dev/dm-0 | grep -A 5 "(286 "
parent transid verify failed on 174170742784 wanted 1789655 found 1812621
parent transid verify failed on 174170742784 wanted 1789655 found 1812621
parent transid verify failed on 174170742784 wanted 1789655 found 1812621
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=174170738688 item=0 parent level=2 child bytenr=174170742784 child level=0

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Saturday, July 17th, 2021 at 6:12 PM, Qu Wenruo <wqu@suse.com> wrote:

> On 2021/7/17 下午4:57, pepperpoint@mb.ardentcoding.com wrote:
>
> > Hi Qu,
> >
> > I don't know how the directory was created but last month, I used btrfs device add and btrfs device remove to move the filesystem from one partition to another. It failed because of the same error and was advised to use btrfs replace instead. I don't know if the error also happened before I move the file system as I don't have any previous logs.
>
> It definitely happens before you moving the fs.
>
> As regular dev replacing/add/move only relocates the metadata, but not
>
> touching the fs trees.
>
> > Here is the result when I search for the inodes you mentioned if it helps:
> >
> > find /run/media/root -inum 260 -exec ls -ldi {} \;
> > ==================================================
> >
> > 260 -rw-r--r-- 1 root root 36864 Jun 25 06:22 /run/media/root/@vcache/live/snapshot/app-info/cache/en_US.cache
> >
> > 260 drwx------ 1 mongodb mongodb 136 Sep 12 2020 /run/media/root/@vlmongodb/live/snapshot/diagnostic.data
> >
> > 260 -rw-rw---- 1 mysql mysql 50331648 Sep 13 2015 /run/media/root/@vlmysql/live/snapshot/ib_logfile0
> >
> > 260 -rw-r----- 1 root lp 8641 Mar 5 2014 /run/media/root/@vspool/live/snapshot/cups/d00001-001
> >
> > 260 dr-xr-xr-x 1 root root 0 Sep 13 2013 /run/media/root/@/live/snapshot/sys
> >
> > 260 dr-xr-xr-x 1 root root 0 Sep 13 2013 /run/media/root/@/4/snapshot/sys
>
> Since btrfs can have the same inode number inside different subvolumes,
>
> you may want to limit the search inside subvolume 363.
>
> "-mount" option of find can do that, you only need to locate subvolume
>
> 363 by "btrfs subv list".
>
> But from these output I guess above two "sys" directory are more possible.
>
> Is there any directory named "blaklight" inside those directory?
>
> > find /run/media/root -inum 286 -exec ls -ldi {} \;
> > ==================================================
> >
> > 286 -rw-r--r-- 1 root root 96 Aug 16 2015 /run/media/root/@vcache/live/snapshot/fontconfig/4b172ca7f111e3cffadc3636415fead9-le64.cache-4
> >
> > 286 -rw-rw---- 1 mysql mysql 4096 Sep 15 2013 /run/media/root/@vlmysql/live/snapshot/mysql/columns_priv.MYI
> >
> > 286 -rw-r-----+ 1 root systemd-journal 16777216 Jul 4 01:14 /run/media/root/@vlog/live/snapshot/journal/5098dd7845ae46d3ba1826c68a809a7c/user-1000@fbd9f65d0ea349f6b996716280e6c4dd-00000000002314c5-0005c5cb84a3a438.journal
>
> This is interesting, it means the inode 286 is not accessible.
>
> It can be some orphan inode, but would you dump subvolume 363 then try
>
> to locate the inode 286?
>
> One example command would be:
>
> btrfs ins dump-tree -t 363 <dev> | grep -A 5 "(286 "
> ====================================================
>
> Thanks,
>
> Qu
>
> > Directories with pattern /root/@<dir>/live/snapshot/ are subvolumes and directories with pattern /root/@<dir>/<num>/snapshot/ are snapshots of live.
> >
> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> >
> > On Saturday, July 17th, 2021 at 4:14 PM, Qu Wenruo quwenruo.btrfs@gmx.com wrote:
> >
> > > On 2021/7/17 下午3:51, pepperpoint@mb.ardentcoding.com wrote:
> > >
> > > > Hi Qu,
> > > >
> > > > Please see below for the dump.
> > > >
> > > > btrfs-progs v5.12.1
> > > >
> > > > leaf 174113599488 items 18 free space 2008 generation 1330906 owner 363
> > > >
> > > > leaf 174113599488 flags 0x1(WRITTEN) backref revision 1
> > > >
> > > > fs uuid a7d327c4-8594-4116-a6f8-8aa2a4162063
> > > >
> > > > chunk uuid f885f49e-14a0-4c80-9c12-c2302b9a0229
> > > >
> > > > item 0 key (5471 INODE_ITEM 0) itemoff 3835 itemsize 160
> > > >
> > > > generation 2063 transid 27726 size 40 nbytes 40
> > > >
> > > > block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
> > > >
> > > > sequence 1501 flags 0x0(none)
> > > >
> > > > atime 1386484844.468769570 (2013-12-08 14:40:44)
> > > >
> > > > ctime 1386484844.468769570 (2013-12-08 14:40:44)
> > > >
> > > > mtime 1386484844.468769570 (2013-12-08 14:40:44)
> > > >
> > > > otime 0.0 (1970-01-01 08:00:00)
> > > >
> > > > item 1 key (5471 INODE_REF 4399) itemoff 3824 itemsize 11
> > > >
> > > > index 12 namelen 1 name: 8
> > > >
> > > > item 2 key (5471 EXTENT_DATA 0) itemoff 3763 itemsize 61
> > > >
> > > > generation 27726 type 0 (inline)
> > > >
> > > > inline extent data size 40 ram_bytes 40 compression 0 (none)
> > > >
> > > > item 3 key (5645 INODE_ITEM 0) itemoff 3603 itemsize 160
> > > >
> > > > generation 2542 transid 61261 size 40 nbytes 40
> > > >
> > > > block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
> > > >
> > > > sequence 24769 flags 0x0(none)
> > > >
> > > > atime 1394335806.351857522 (2014-03-09 11:30:06)
> > > >
> > > > ctime 1394335827.344389955 (2014-03-09 11:30:27)
> > > >
> > > > mtime 1394335827.344389955 (2014-03-09 11:30:27)
> > > >
> > > > otime 0.0 (1970-01-01 08:00:00)
> > > >
> > > > item 4 key (5645 INODE_REF 4399) itemoff 3592 itemsize 11
> > > >
> > > > index 13 namelen 1 name: 7
> > > >
> > > > item 5 key (5645 EXTENT_DATA 0) itemoff 3531 itemsize 61
> > > >
> > > > generation 61261 type 0 (inline)
> > > >
> > > > inline extent data size 40 ram_bytes 40 compression 0 (none)
> > > >
> > > > item 6 key (7222 INODE_ITEM 0) itemoff 3371 itemsize 160
> > > >
> > > > generation 5754 transid 5767 size 307 nbytes 307
> > > >
> > > > block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
> > > >
> > > > sequence 7 flags 0x0(none)
> > > >
> > > > atime 1379834835.428558020 (2013-09-22 15:27:15)
> > > >
> > > > ctime 1379834835.428558020 (2013-09-22 15:27:15)
> > > >
> > > > mtime 1379834835.428558020 (2013-09-22 15:27:15)
> > > >
> > > > otime 0.0 (1970-01-01 08:00:00)
> > > >
> > > > item 7 key (7222 INODE_REF 287) itemoff 3344 itemsize 27
> > > >
> > > > index 6 namelen 17 name: dhcpcd-eth0.lease
> > > >
> > > > item 8 key (7222 EXTENT_DATA 0) itemoff 3016 itemsize 328
> > > >
> > > > generation 5767 type 0 (inline)
> > > >
> > > > inline extent data size 307 ram_bytes 307 compression 0 (none)
> > > >
> > > > item 9 key (7415 INODE_ITEM 0) itemoff 2856 itemsize 160
> > > >
> > > > generation 5904 transid 1330906 size 180 nbytes 0
> > > >
> > > > block group 0 mode 40755 links 2 uid 0 gid 0 rdev 0
> > > >
> > > > sequence 177 flags 0x0(none)
> > > >
> > > > atime 1483277713.141980592 (2017-01-01 21:35:13)
> > > >
> > > > ctime 1563162901.234656246 (2019-07-15 11:55:01)
> > > >
> > > > mtime 1406534032.158605559 (2014-07-28 15:53:52)
> > > >
> > > > otime 0.0 (1970-01-01 08:00:00)
> > >
> > > This inode is indeed a directory.
> > >
> > > But it has two hard links, which is definitely something unexpected.
> > >
> > > Under Linux we shouldn't have any hardlink for directory, as it would
> > >
> > > easily lead to loops.
> > >
> > > > item 10 key (7415 INODE_REF 260) itemoff 2837 itemsize 19
> > > >
> > > > index 28 namelen 9 name: backlight
> > >
> > > Its parent inode is 260 in the same root, with the name backlight.
> > >
> > > > item 11 key (7415 INODE_REF 286) itemoff 2818 itemsize 19
> > > >
> > > > index 3 namelen 9 name: backlight
> > >
> > > Another hardlink in inode 286, which is definitely a regular thing.
> > >
> > > Btrfs-progs lacks the ability to detect such problem, we need to enhance
> > >
> > > it first.
> > >
> > > But do you have any idea how this directory get created?
> > >
> > > It looks like the content of sysfs.
> > >
> > > Thanks,
> > >
> > > Qu
> > >
> > > > item 12 key (7415 DIR_ITEM 3128336373) itemoff 2746 itemsize 72
> > > >
> > > > location key (120417 INODE_ITEM 0) type FILE
> > > >
> > > > transid 117279 data_len 0 name_len 42
> > > >
> > > > name: pci-0000:00:02.0:backlight:intel_backlight
> > > >
> > > > item 13 key (7415 DIR_ITEM 3218198317) itemoff 2705 itemsize 41
> > > >
> > > > location key (7487 INODE_ITEM 0) type FILE
> > > >
> > > > transid 5992 data_len 0 name_len 11
> > > >
> > > > name: acpi_video0
> > > >
> > > > item 14 key (7415 DIR_ITEM 3582254411) itemoff 2638 itemsize 67
> > > >
> > > > location key (55325 INODE_ITEM 0) type FILE
> > > >
> > > > transid 63351 data_len 0 name_len 37
> > > >
> > > > name: platform-VPC2004:00:backlight:ideapad
> > > >
> > > > item 15 key (7415 DIR_INDEX 2) itemoff 2597 itemsize 41
> > > >
> > > > location key (7487 INODE_ITEM 0) type FILE
> > > >
> > > > transid 5992 data_len 0 name_len 11
> > > >
> > > > name: acpi_video0
> > > >
> > > > item 16 key (7415 DIR_INDEX 4) itemoff 2530 itemsize 67
> > > >
> > > > location key (55325 INODE_ITEM 0) type FILE
> > > >
> > > > transid 63351 data_len 0 name_len 37
> > > >
> > > > name: platform-VPC2004:00:backlight:ideapad
> > > >
> > > > item 17 key (7415 DIR_INDEX 5) itemoff 2458 itemsize 72
> > > >
> > > > location key (120417 INODE_ITEM 0) type FILE
> > > >
> > > > transid 117279 data_len 0 name_len 42
> > > >
> > > > name: pci-0000:00:02.0:backlight:intel_backlight
> > > >
> > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > >
> > > > On Saturday, July 17th, 2021 at 3:05 PM, Qu Wenruo quwenruo.btrfs@gmx.com wrote:
> > > >
> > > > > On 2021/7/17 上午9:45, pepperpoint@mb.ardentcoding.com wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I see this message on dmesg:
> > > > > >
> > > > > > [ 2452.256756] BTRFS critical (device dm-0): corrupt leaf: root=363 block=174113599488 slot=9 ino=7415, invalid nlink: has 2 expect no more than 1 for dir
> > > > > >
> > > > > > [ 2452.256776] BTRFS error (device dm-0): block=174113599488 read time tree block corruption detected
> > > > >
> > > > > Please provide the following dump:
> > > > >
> > > > > btrfs ins dump-tree -b 174113599488 /dev/dm-0
> > > > > =============================================
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Qu
> > > > >
> > > > > > When I run btrfs scrub and btrfs check, no error was detected.
> > > > > >
> > > > > > I am running Linux 5.12.15-arch1-1 and btrfs-progs v5.12.1
> > > > > >
> > > > > > How should I fix this error?

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: Read time tree block corruption detected
  2021-07-17  8:57       ` pepperpoint
@ 2021-07-17 10:12         ` Qu Wenruo
       [not found]         ` <162651674065.6.7912816287233908759.10188327@simplelogin.co>
  1 sibling, 0 replies; 75+ messages in thread
From: Qu Wenruo @ 2021-07-17 10:12 UTC (permalink / raw)
  To: pepperpoint, Qu Wenruo; +Cc: linux-btrfs



On 2021/7/17 下午4:57, pepperpoint@mb.ardentcoding.com wrote:
> Hi Qu,
> 
> I don't know how the directory was created but last month, I used btrfs device add and btrfs device remove to move the filesystem from one partition to another. It failed because of the same error and was advised to use btrfs replace instead. I don't know if the error also happened before I move the file system as I don't have any previous logs.

It definitely happens before you moving the fs.

As regular dev replacing/add/move only relocates the metadata, but not 
touching the fs trees.

> 
> Here is the result when I search for the inodes you mentioned if it helps:
> # find /run/media/root -inum 260 -exec ls -ldi {} \;
> 260 -rw-r--r-- 1 root root 36864 Jun 25 06:22 /run/media/root/@vcache/live/snapshot/app-info/cache/en_US.cache
> 260 drwx------ 1 mongodb mongodb 136 Sep 12  2020 /run/media/root/@vlmongodb/live/snapshot/diagnostic.data
> 260 -rw-rw---- 1 mysql mysql 50331648 Sep 13  2015 /run/media/root/@vlmysql/live/snapshot/ib_logfile0
> 260 -rw-r----- 1 root lp 8641 Mar  5  2014 /run/media/root/@vspool/live/snapshot/cups/d00001-001
> 260 dr-xr-xr-x 1 root root 0 Sep 13  2013 /run/media/root/@/live/snapshot/sys
> 260 dr-xr-xr-x 1 root root 0 Sep 13  2013 /run/media/root/@/4/snapshot/sys

Since btrfs can have the same inode number inside different subvolumes, 
you may want to limit the search inside subvolume 363.

"-mount" option of find can do that, you only need to locate subvolume 
363 by "btrfs subv list".


But from these output I guess above two "sys" directory are more possible.

Is there any directory named "blaklight" inside those directory?

> 
> # find /run/media/root -inum 286 -exec ls -ldi {} \;
> 286 -rw-r--r-- 1 root root 96 Aug 16  2015 /run/media/root/@vcache/live/snapshot/fontconfig/4b172ca7f111e3cffadc3636415fead9-le64.cache-4
> 286 -rw-rw---- 1 mysql mysql 4096 Sep 15  2013 /run/media/root/@vlmysql/live/snapshot/mysql/columns_priv.MYI
> 286 -rw-r-----+ 1 root systemd-journal 16777216 Jul  4 01:14 /run/media/root/@vlog/live/snapshot/journal/5098dd7845ae46d3ba1826c68a809a7c/user-1000@fbd9f65d0ea349f6b996716280e6c4dd-00000000002314c5-0005c5cb84a3a438.journal

This is interesting, it means the inode 286 is not accessible.

It can be some orphan inode, but would you dump subvolume 363 then try 
to locate the inode 286?
One example command would be:

# btrfs ins dump-tree -t 363 <dev> | grep -A 5 "(286 "

Thanks,
Qu


> 
> Directories with pattern /root/@<dir>/live/snapshot/ are subvolumes and directories with pattern /root/@<dir>/<num>/snapshot/ are snapshots of live.
> 
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> 
> On Saturday, July 17th, 2021 at 4:14 PM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> 
>> On 2021/7/17 下午3:51, pepperpoint@mb.ardentcoding.com wrote:
>>
>>> Hi Qu,
>>>
>>> Please see below for the dump.
>>>
>>> btrfs-progs v5.12.1
>>>
>>> leaf 174113599488 items 18 free space 2008 generation 1330906 owner 363
>>>
>>> leaf 174113599488 flags 0x1(WRITTEN) backref revision 1
>>>
>>> fs uuid a7d327c4-8594-4116-a6f8-8aa2a4162063
>>>
>>> chunk uuid f885f49e-14a0-4c80-9c12-c2302b9a0229
>>>
>>> item 0 key (5471 INODE_ITEM 0) itemoff 3835 itemsize 160
>>>
>>> generation 2063 transid 27726 size 40 nbytes 40
>>>
>>> block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
>>>
>>> sequence 1501 flags 0x0(none)
>>>
>>> atime 1386484844.468769570 (2013-12-08 14:40:44)
>>>
>>> ctime 1386484844.468769570 (2013-12-08 14:40:44)
>>>
>>> mtime 1386484844.468769570 (2013-12-08 14:40:44)
>>>
>>> otime 0.0 (1970-01-01 08:00:00)
>>>
>>> item 1 key (5471 INODE_REF 4399) itemoff 3824 itemsize 11
>>>
>>> index 12 namelen 1 name: 8
>>>
>>> item 2 key (5471 EXTENT_DATA 0) itemoff 3763 itemsize 61
>>>
>>> generation 27726 type 0 (inline)
>>>
>>> inline extent data size 40 ram_bytes 40 compression 0 (none)
>>>
>>> item 3 key (5645 INODE_ITEM 0) itemoff 3603 itemsize 160
>>>
>>> generation 2542 transid 61261 size 40 nbytes 40
>>>
>>> block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
>>>
>>> sequence 24769 flags 0x0(none)
>>>
>>> atime 1394335806.351857522 (2014-03-09 11:30:06)
>>>
>>> ctime 1394335827.344389955 (2014-03-09 11:30:27)
>>>
>>> mtime 1394335827.344389955 (2014-03-09 11:30:27)
>>>
>>> otime 0.0 (1970-01-01 08:00:00)
>>>
>>> item 4 key (5645 INODE_REF 4399) itemoff 3592 itemsize 11
>>>
>>> index 13 namelen 1 name: 7
>>>
>>> item 5 key (5645 EXTENT_DATA 0) itemoff 3531 itemsize 61
>>>
>>> generation 61261 type 0 (inline)
>>>
>>> inline extent data size 40 ram_bytes 40 compression 0 (none)
>>>
>>> item 6 key (7222 INODE_ITEM 0) itemoff 3371 itemsize 160
>>>
>>> generation 5754 transid 5767 size 307 nbytes 307
>>>
>>> block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
>>>
>>> sequence 7 flags 0x0(none)
>>>
>>> atime 1379834835.428558020 (2013-09-22 15:27:15)
>>>
>>> ctime 1379834835.428558020 (2013-09-22 15:27:15)
>>>
>>> mtime 1379834835.428558020 (2013-09-22 15:27:15)
>>>
>>> otime 0.0 (1970-01-01 08:00:00)
>>>
>>> item 7 key (7222 INODE_REF 287) itemoff 3344 itemsize 27
>>>
>>> index 6 namelen 17 name: dhcpcd-eth0.lease
>>>
>>> item 8 key (7222 EXTENT_DATA 0) itemoff 3016 itemsize 328
>>>
>>> generation 5767 type 0 (inline)
>>>
>>> inline extent data size 307 ram_bytes 307 compression 0 (none)
>>>
>>> item 9 key (7415 INODE_ITEM 0) itemoff 2856 itemsize 160
>>>
>>> generation 5904 transid 1330906 size 180 nbytes 0
>>>
>>> block group 0 mode 40755 links 2 uid 0 gid 0 rdev 0
>>>
>>> sequence 177 flags 0x0(none)
>>>
>>> atime 1483277713.141980592 (2017-01-01 21:35:13)
>>>
>>> ctime 1563162901.234656246 (2019-07-15 11:55:01)
>>>
>>> mtime 1406534032.158605559 (2014-07-28 15:53:52)
>>>
>>> otime 0.0 (1970-01-01 08:00:00)
>>
>> This inode is indeed a directory.
>>
>> But it has two hard links, which is definitely something unexpected.
>>
>> Under Linux we shouldn't have any hardlink for directory, as it would
>>
>> easily lead to loops.
>>
>>> item 10 key (7415 INODE_REF 260) itemoff 2837 itemsize 19
>>>
>>> index 28 namelen 9 name: backlight
>>
>> Its parent inode is 260 in the same root, with the name backlight.
>>
>>> item 11 key (7415 INODE_REF 286) itemoff 2818 itemsize 19
>>>
>>> index 3 namelen 9 name: backlight
>>
>> Another hardlink in inode 286, which is definitely a regular thing.
>>
>> Btrfs-progs lacks the ability to detect such problem, we need to enhance
>>
>> it first.
>>
>> But do you have any idea how this directory get created?
>>
>> It looks like the content of sysfs.
>>
>> Thanks,
>>
>> Qu
>>
>>> item 12 key (7415 DIR_ITEM 3128336373) itemoff 2746 itemsize 72
>>>
>>> location key (120417 INODE_ITEM 0) type FILE
>>>
>>> transid 117279 data_len 0 name_len 42
>>>
>>> name: pci-0000:00:02.0:backlight:intel_backlight
>>>
>>> item 13 key (7415 DIR_ITEM 3218198317) itemoff 2705 itemsize 41
>>>
>>> location key (7487 INODE_ITEM 0) type FILE
>>>
>>> transid 5992 data_len 0 name_len 11
>>>
>>> name: acpi_video0
>>>
>>> item 14 key (7415 DIR_ITEM 3582254411) itemoff 2638 itemsize 67
>>>
>>> location key (55325 INODE_ITEM 0) type FILE
>>>
>>> transid 63351 data_len 0 name_len 37
>>>
>>> name: platform-VPC2004:00:backlight:ideapad
>>>
>>> item 15 key (7415 DIR_INDEX 2) itemoff 2597 itemsize 41
>>>
>>> location key (7487 INODE_ITEM 0) type FILE
>>>
>>> transid 5992 data_len 0 name_len 11
>>>
>>> name: acpi_video0
>>>
>>> item 16 key (7415 DIR_INDEX 4) itemoff 2530 itemsize 67
>>>
>>> location key (55325 INODE_ITEM 0) type FILE
>>>
>>> transid 63351 data_len 0 name_len 37
>>>
>>> name: platform-VPC2004:00:backlight:ideapad
>>>
>>> item 17 key (7415 DIR_INDEX 5) itemoff 2458 itemsize 72
>>>
>>> location key (120417 INODE_ITEM 0) type FILE
>>>
>>> transid 117279 data_len 0 name_len 42
>>>
>>> name: pci-0000:00:02.0:backlight:intel_backlight
>>>
>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>
>>> On Saturday, July 17th, 2021 at 3:05 PM, Qu Wenruo quwenruo.btrfs@gmx.com wrote:
>>>
>>>> On 2021/7/17 上午9:45, pepperpoint@mb.ardentcoding.com wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I see this message on dmesg:
>>>>>
>>>>> [ 2452.256756] BTRFS critical (device dm-0): corrupt leaf: root=363 block=174113599488 slot=9 ino=7415, invalid nlink: has 2 expect no more than 1 for dir
>>>>>
>>>>> [ 2452.256776] BTRFS error (device dm-0): block=174113599488 read time tree block corruption detected
>>>>
>>>> Please provide the following dump:
>>>>
>>>> btrfs ins dump-tree -b 174113599488 /dev/dm-0
>>>> =============================================
>>>>
>>>> Thanks,
>>>>
>>>> Qu
>>>>
>>>>> When I run btrfs scrub and btrfs check, no error was detected.
>>>>>
>>>>> I am running Linux 5.12.15-arch1-1 and btrfs-progs v5.12.1
>>>>>
>>>>> How should I fix this error?
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: Read time tree block corruption detected
       [not found]     ` <162650966150.7.11743767259405124657.10185986@simplelogin.co>
@ 2021-07-17  8:57       ` pepperpoint
  2021-07-17 10:12         ` Qu Wenruo
       [not found]         ` <162651674065.6.7912816287233908759.10188327@simplelogin.co>
  0 siblings, 2 replies; 75+ messages in thread
From: pepperpoint @ 2021-07-17  8:57 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: linux-btrfs

Hi Qu,

I don't know how the directory was created but last month, I used btrfs device add and btrfs device remove to move the filesystem from one partition to another. It failed because of the same error and was advised to use btrfs replace instead. I don't know if the error also happened before I move the file system as I don't have any previous logs.

Here is the result when I search for the inodes you mentioned if it helps:
# find /run/media/root -inum 260 -exec ls -ldi {} \;
260 -rw-r--r-- 1 root root 36864 Jun 25 06:22 /run/media/root/@vcache/live/snapshot/app-info/cache/en_US.cache
260 drwx------ 1 mongodb mongodb 136 Sep 12  2020 /run/media/root/@vlmongodb/live/snapshot/diagnostic.data
260 -rw-rw---- 1 mysql mysql 50331648 Sep 13  2015 /run/media/root/@vlmysql/live/snapshot/ib_logfile0
260 -rw-r----- 1 root lp 8641 Mar  5  2014 /run/media/root/@vspool/live/snapshot/cups/d00001-001
260 dr-xr-xr-x 1 root root 0 Sep 13  2013 /run/media/root/@/live/snapshot/sys
260 dr-xr-xr-x 1 root root 0 Sep 13  2013 /run/media/root/@/4/snapshot/sys

# find /run/media/root -inum 286 -exec ls -ldi {} \;
286 -rw-r--r-- 1 root root 96 Aug 16  2015 /run/media/root/@vcache/live/snapshot/fontconfig/4b172ca7f111e3cffadc3636415fead9-le64.cache-4
286 -rw-rw---- 1 mysql mysql 4096 Sep 15  2013 /run/media/root/@vlmysql/live/snapshot/mysql/columns_priv.MYI
286 -rw-r-----+ 1 root systemd-journal 16777216 Jul  4 01:14 /run/media/root/@vlog/live/snapshot/journal/5098dd7845ae46d3ba1826c68a809a7c/user-1000@fbd9f65d0ea349f6b996716280e6c4dd-00000000002314c5-0005c5cb84a3a438.journal

Directories with pattern /root/@<dir>/live/snapshot/ are subvolumes and directories with pattern /root/@<dir>/<num>/snapshot/ are snapshots of live.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Saturday, July 17th, 2021 at 4:14 PM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:

> On 2021/7/17 下午3:51, pepperpoint@mb.ardentcoding.com wrote:
>
> > Hi Qu,
> >
> > Please see below for the dump.
> >
> > btrfs-progs v5.12.1
> >
> > leaf 174113599488 items 18 free space 2008 generation 1330906 owner 363
> >
> > leaf 174113599488 flags 0x1(WRITTEN) backref revision 1
> >
> > fs uuid a7d327c4-8594-4116-a6f8-8aa2a4162063
> >
> > chunk uuid f885f49e-14a0-4c80-9c12-c2302b9a0229
> >
> > item 0 key (5471 INODE_ITEM 0) itemoff 3835 itemsize 160
> >
> > generation 2063 transid 27726 size 40 nbytes 40
> >
> > block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
> >
> > sequence 1501 flags 0x0(none)
> >
> > atime 1386484844.468769570 (2013-12-08 14:40:44)
> >
> > ctime 1386484844.468769570 (2013-12-08 14:40:44)
> >
> > mtime 1386484844.468769570 (2013-12-08 14:40:44)
> >
> > otime 0.0 (1970-01-01 08:00:00)
> >
> > item 1 key (5471 INODE_REF 4399) itemoff 3824 itemsize 11
> >
> > index 12 namelen 1 name: 8
> >
> > item 2 key (5471 EXTENT_DATA 0) itemoff 3763 itemsize 61
> >
> > generation 27726 type 0 (inline)
> >
> > inline extent data size 40 ram_bytes 40 compression 0 (none)
> >
> > item 3 key (5645 INODE_ITEM 0) itemoff 3603 itemsize 160
> >
> > generation 2542 transid 61261 size 40 nbytes 40
> >
> > block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
> >
> > sequence 24769 flags 0x0(none)
> >
> > atime 1394335806.351857522 (2014-03-09 11:30:06)
> >
> > ctime 1394335827.344389955 (2014-03-09 11:30:27)
> >
> > mtime 1394335827.344389955 (2014-03-09 11:30:27)
> >
> > otime 0.0 (1970-01-01 08:00:00)
> >
> > item 4 key (5645 INODE_REF 4399) itemoff 3592 itemsize 11
> >
> > index 13 namelen 1 name: 7
> >
> > item 5 key (5645 EXTENT_DATA 0) itemoff 3531 itemsize 61
> >
> > generation 61261 type 0 (inline)
> >
> > inline extent data size 40 ram_bytes 40 compression 0 (none)
> >
> > item 6 key (7222 INODE_ITEM 0) itemoff 3371 itemsize 160
> >
> > generation 5754 transid 5767 size 307 nbytes 307
> >
> > block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
> >
> > sequence 7 flags 0x0(none)
> >
> > atime 1379834835.428558020 (2013-09-22 15:27:15)
> >
> > ctime 1379834835.428558020 (2013-09-22 15:27:15)
> >
> > mtime 1379834835.428558020 (2013-09-22 15:27:15)
> >
> > otime 0.0 (1970-01-01 08:00:00)
> >
> > item 7 key (7222 INODE_REF 287) itemoff 3344 itemsize 27
> >
> > index 6 namelen 17 name: dhcpcd-eth0.lease
> >
> > item 8 key (7222 EXTENT_DATA 0) itemoff 3016 itemsize 328
> >
> > generation 5767 type 0 (inline)
> >
> > inline extent data size 307 ram_bytes 307 compression 0 (none)
> >
> > item 9 key (7415 INODE_ITEM 0) itemoff 2856 itemsize 160
> >
> > generation 5904 transid 1330906 size 180 nbytes 0
> >
> > block group 0 mode 40755 links 2 uid 0 gid 0 rdev 0
> >
> > sequence 177 flags 0x0(none)
> >
> > atime 1483277713.141980592 (2017-01-01 21:35:13)
> >
> > ctime 1563162901.234656246 (2019-07-15 11:55:01)
> >
> > mtime 1406534032.158605559 (2014-07-28 15:53:52)
> >
> > otime 0.0 (1970-01-01 08:00:00)
>
> This inode is indeed a directory.
>
> But it has two hard links, which is definitely something unexpected.
>
> Under Linux we shouldn't have any hardlink for directory, as it would
>
> easily lead to loops.
>
> > item 10 key (7415 INODE_REF 260) itemoff 2837 itemsize 19
> >
> > index 28 namelen 9 name: backlight
>
> Its parent inode is 260 in the same root, with the name backlight.
>
> > item 11 key (7415 INODE_REF 286) itemoff 2818 itemsize 19
> >
> > index 3 namelen 9 name: backlight
>
> Another hardlink in inode 286, which is definitely a regular thing.
>
> Btrfs-progs lacks the ability to detect such problem, we need to enhance
>
> it first.
>
> But do you have any idea how this directory get created?
>
> It looks like the content of sysfs.
>
> Thanks,
>
> Qu
>
> > item 12 key (7415 DIR_ITEM 3128336373) itemoff 2746 itemsize 72
> >
> > location key (120417 INODE_ITEM 0) type FILE
> >
> > transid 117279 data_len 0 name_len 42
> >
> > name: pci-0000:00:02.0:backlight:intel_backlight
> >
> > item 13 key (7415 DIR_ITEM 3218198317) itemoff 2705 itemsize 41
> >
> > location key (7487 INODE_ITEM 0) type FILE
> >
> > transid 5992 data_len 0 name_len 11
> >
> > name: acpi_video0
> >
> > item 14 key (7415 DIR_ITEM 3582254411) itemoff 2638 itemsize 67
> >
> > location key (55325 INODE_ITEM 0) type FILE
> >
> > transid 63351 data_len 0 name_len 37
> >
> > name: platform-VPC2004:00:backlight:ideapad
> >
> > item 15 key (7415 DIR_INDEX 2) itemoff 2597 itemsize 41
> >
> > location key (7487 INODE_ITEM 0) type FILE
> >
> > transid 5992 data_len 0 name_len 11
> >
> > name: acpi_video0
> >
> > item 16 key (7415 DIR_INDEX 4) itemoff 2530 itemsize 67
> >
> > location key (55325 INODE_ITEM 0) type FILE
> >
> > transid 63351 data_len 0 name_len 37
> >
> > name: platform-VPC2004:00:backlight:ideapad
> >
> > item 17 key (7415 DIR_INDEX 5) itemoff 2458 itemsize 72
> >
> > location key (120417 INODE_ITEM 0) type FILE
> >
> > transid 117279 data_len 0 name_len 42
> >
> > name: pci-0000:00:02.0:backlight:intel_backlight
> >
> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> >
> > On Saturday, July 17th, 2021 at 3:05 PM, Qu Wenruo quwenruo.btrfs@gmx.com wrote:
> >
> > > On 2021/7/17 上午9:45, pepperpoint@mb.ardentcoding.com wrote:
> > >
> > > > Hello,
> > > >
> > > > I see this message on dmesg:
> > > >
> > > > [ 2452.256756] BTRFS critical (device dm-0): corrupt leaf: root=363 block=174113599488 slot=9 ino=7415, invalid nlink: has 2 expect no more than 1 for dir
> > > >
> > > > [ 2452.256776] BTRFS error (device dm-0): block=174113599488 read time tree block corruption detected
> > >
> > > Please provide the following dump:
> > >
> > > btrfs ins dump-tree -b 174113599488 /dev/dm-0
> > > =============================================
> > >
> > > Thanks,
> > >
> > > Qu
> > >
> > > > When I run btrfs scrub and btrfs check, no error was detected.
> > > >
> > > > I am running Linux 5.12.15-arch1-1 and btrfs-progs v5.12.1
> > > >
> > > > How should I fix this error?

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: Read time tree block corruption detected
       [not found]   ` <162650826457.7.1050455337652772013.10184548@mb.ardentcoding.com>
@ 2021-07-17  8:14     ` Qu Wenruo
       [not found]     ` <162650966150.7.11743767259405124657.10185986@simplelogin.co>
  1 sibling, 0 replies; 75+ messages in thread
From: Qu Wenruo @ 2021-07-17  8:14 UTC (permalink / raw)
  To: pepperpoint; +Cc: linux-btrfs



On 2021/7/17 下午3:51, pepperpoint@mb.ardentcoding.com wrote:
> Hi Qu,
>
> Please see below for the dump.
>
> btrfs-progs v5.12.1
> leaf 174113599488 items 18 free space 2008 generation 1330906 owner 363
> leaf 174113599488 flags 0x1(WRITTEN) backref revision 1
> fs uuid a7d327c4-8594-4116-a6f8-8aa2a4162063
> chunk uuid f885f49e-14a0-4c80-9c12-c2302b9a0229
> 	item 0 key (5471 INODE_ITEM 0) itemoff 3835 itemsize 160
> 		generation 2063 transid 27726 size 40 nbytes 40
> 		block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
> 		sequence 1501 flags 0x0(none)
> 		atime 1386484844.468769570 (2013-12-08 14:40:44)
> 		ctime 1386484844.468769570 (2013-12-08 14:40:44)
> 		mtime 1386484844.468769570 (2013-12-08 14:40:44)
> 		otime 0.0 (1970-01-01 08:00:00)
> 	item 1 key (5471 INODE_REF 4399) itemoff 3824 itemsize 11
> 		index 12 namelen 1 name: 8
> 	item 2 key (5471 EXTENT_DATA 0) itemoff 3763 itemsize 61
> 		generation 27726 type 0 (inline)
> 		inline extent data size 40 ram_bytes 40 compression 0 (none)
> 	item 3 key (5645 INODE_ITEM 0) itemoff 3603 itemsize 160
> 		generation 2542 transid 61261 size 40 nbytes 40
> 		block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
> 		sequence 24769 flags 0x0(none)
> 		atime 1394335806.351857522 (2014-03-09 11:30:06)
> 		ctime 1394335827.344389955 (2014-03-09 11:30:27)
> 		mtime 1394335827.344389955 (2014-03-09 11:30:27)
> 		otime 0.0 (1970-01-01 08:00:00)
> 	item 4 key (5645 INODE_REF 4399) itemoff 3592 itemsize 11
> 		index 13 namelen 1 name: 7
> 	item 5 key (5645 EXTENT_DATA 0) itemoff 3531 itemsize 61
> 		generation 61261 type 0 (inline)
> 		inline extent data size 40 ram_bytes 40 compression 0 (none)
> 	item 6 key (7222 INODE_ITEM 0) itemoff 3371 itemsize 160
> 		generation 5754 transid 5767 size 307 nbytes 307
> 		block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
> 		sequence 7 flags 0x0(none)
> 		atime 1379834835.428558020 (2013-09-22 15:27:15)
> 		ctime 1379834835.428558020 (2013-09-22 15:27:15)
> 		mtime 1379834835.428558020 (2013-09-22 15:27:15)
> 		otime 0.0 (1970-01-01 08:00:00)
> 	item 7 key (7222 INODE_REF 287) itemoff 3344 itemsize 27
> 		index 6 namelen 17 name: dhcpcd-eth0.lease
> 	item 8 key (7222 EXTENT_DATA 0) itemoff 3016 itemsize 328
> 		generation 5767 type 0 (inline)
> 		inline extent data size 307 ram_bytes 307 compression 0 (none)
> 	item 9 key (7415 INODE_ITEM 0) itemoff 2856 itemsize 160
> 		generation 5904 transid 1330906 size 180 nbytes 0
> 		block group 0 mode 40755 links 2 uid 0 gid 0 rdev 0
> 		sequence 177 flags 0x0(none)
> 		atime 1483277713.141980592 (2017-01-01 21:35:13)
> 		ctime 1563162901.234656246 (2019-07-15 11:55:01)
> 		mtime 1406534032.158605559 (2014-07-28 15:53:52)
> 		otime 0.0 (1970-01-01 08:00:00)

This inode is indeed a directory.

But it has two hard links, which is definitely something unexpected.

Under Linux we shouldn't have any hardlink for directory, as it would
easily lead to loops.

> 	item 10 key (7415 INODE_REF 260) itemoff 2837 itemsize 19
> 		index 28 namelen 9 name: backlight

Its parent inode is 260 in the same root, with the name backlight.

> 	item 11 key (7415 INODE_REF 286) itemoff 2818 itemsize 19
> 		index 3 namelen 9 name: backlight

Another hardlink in inode 286, which is definitely a regular thing.

Btrfs-progs lacks the ability to detect such problem, we need to enhance
it first.

But do you have any idea how this directory get created?
It looks like the content of sysfs.

Thanks,
Qu

> 	item 12 key (7415 DIR_ITEM 3128336373) itemoff 2746 itemsize 72
> 		location key (120417 INODE_ITEM 0) type FILE
> 		transid 117279 data_len 0 name_len 42
> 		name: pci-0000:00:02.0:backlight:intel_backlight
> 	item 13 key (7415 DIR_ITEM 3218198317) itemoff 2705 itemsize 41
> 		location key (7487 INODE_ITEM 0) type FILE
> 		transid 5992 data_len 0 name_len 11
> 		name: acpi_video0
> 	item 14 key (7415 DIR_ITEM 3582254411) itemoff 2638 itemsize 67
> 		location key (55325 INODE_ITEM 0) type FILE
> 		transid 63351 data_len 0 name_len 37
> 		name: platform-VPC2004:00:backlight:ideapad
> 	item 15 key (7415 DIR_INDEX 2) itemoff 2597 itemsize 41
> 		location key (7487 INODE_ITEM 0) type FILE
> 		transid 5992 data_len 0 name_len 11
> 		name: acpi_video0
> 	item 16 key (7415 DIR_INDEX 4) itemoff 2530 itemsize 67
> 		location key (55325 INODE_ITEM 0) type FILE
> 		transid 63351 data_len 0 name_len 37
> 		name: platform-VPC2004:00:backlight:ideapad
> 	item 17 key (7415 DIR_INDEX 5) itemoff 2458 itemsize 72
> 		location key (120417 INODE_ITEM 0) type FILE
> 		transid 117279 data_len 0 name_len 42
> 		name: pci-0000:00:02.0:backlight:intel_backlight
>
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>
> On Saturday, July 17th, 2021 at 3:05 PM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>> On 2021/7/17 上午9:45, pepperpoint@mb.ardentcoding.com wrote:
>>
>>> Hello,
>>>
>>> I see this message on dmesg:
>>>
>>> [ 2452.256756] BTRFS critical (device dm-0): corrupt leaf: root=363 block=174113599488 slot=9 ino=7415, invalid nlink: has 2 expect no more than 1 for dir
>>>
>>> [ 2452.256776] BTRFS error (device dm-0): block=174113599488 read time tree block corruption detected
>>
>> Please provide the following dump:
>>
>> btrfs ins dump-tree -b 174113599488 /dev/dm-0
>> =============================================
>>
>> Thanks,
>>
>> Qu
>>
>>> When I run btrfs scrub and btrfs check, no error was detected.
>>>
>>> I am running Linux 5.12.15-arch1-1 and btrfs-progs v5.12.1
>>>
>>> How should I fix this error?

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: Read time tree block corruption detected
       [not found] ` <162650555086.7.16811903270475408953.10183708@simplelogin.co>
@ 2021-07-17  7:51   ` pepperpoint
       [not found]   ` <162650826457.7.1050455337652772013.10184548@mb.ardentcoding.com>
  1 sibling, 0 replies; 75+ messages in thread
From: pepperpoint @ 2021-07-17  7:51 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: linux-btrfs

Hi Qu,

Please see below for the dump.

btrfs-progs v5.12.1
leaf 174113599488 items 18 free space 2008 generation 1330906 owner 363
leaf 174113599488 flags 0x1(WRITTEN) backref revision 1
fs uuid a7d327c4-8594-4116-a6f8-8aa2a4162063
chunk uuid f885f49e-14a0-4c80-9c12-c2302b9a0229
	item 0 key (5471 INODE_ITEM 0) itemoff 3835 itemsize 160
		generation 2063 transid 27726 size 40 nbytes 40
		block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
		sequence 1501 flags 0x0(none)
		atime 1386484844.468769570 (2013-12-08 14:40:44)
		ctime 1386484844.468769570 (2013-12-08 14:40:44)
		mtime 1386484844.468769570 (2013-12-08 14:40:44)
		otime 0.0 (1970-01-01 08:00:00)
	item 1 key (5471 INODE_REF 4399) itemoff 3824 itemsize 11
		index 12 namelen 1 name: 8
	item 2 key (5471 EXTENT_DATA 0) itemoff 3763 itemsize 61
		generation 27726 type 0 (inline)
		inline extent data size 40 ram_bytes 40 compression 0 (none)
	item 3 key (5645 INODE_ITEM 0) itemoff 3603 itemsize 160
		generation 2542 transid 61261 size 40 nbytes 40
		block group 0 mode 100600 links 1 uid 0 gid 100 rdev 0
		sequence 24769 flags 0x0(none)
		atime 1394335806.351857522 (2014-03-09 11:30:06)
		ctime 1394335827.344389955 (2014-03-09 11:30:27)
		mtime 1394335827.344389955 (2014-03-09 11:30:27)
		otime 0.0 (1970-01-01 08:00:00)
	item 4 key (5645 INODE_REF 4399) itemoff 3592 itemsize 11
		index 13 namelen 1 name: 7
	item 5 key (5645 EXTENT_DATA 0) itemoff 3531 itemsize 61
		generation 61261 type 0 (inline)
		inline extent data size 40 ram_bytes 40 compression 0 (none)
	item 6 key (7222 INODE_ITEM 0) itemoff 3371 itemsize 160
		generation 5754 transid 5767 size 307 nbytes 307
		block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
		sequence 7 flags 0x0(none)
		atime 1379834835.428558020 (2013-09-22 15:27:15)
		ctime 1379834835.428558020 (2013-09-22 15:27:15)
		mtime 1379834835.428558020 (2013-09-22 15:27:15)
		otime 0.0 (1970-01-01 08:00:00)
	item 7 key (7222 INODE_REF 287) itemoff 3344 itemsize 27
		index 6 namelen 17 name: dhcpcd-eth0.lease
	item 8 key (7222 EXTENT_DATA 0) itemoff 3016 itemsize 328
		generation 5767 type 0 (inline)
		inline extent data size 307 ram_bytes 307 compression 0 (none)
	item 9 key (7415 INODE_ITEM 0) itemoff 2856 itemsize 160
		generation 5904 transid 1330906 size 180 nbytes 0
		block group 0 mode 40755 links 2 uid 0 gid 0 rdev 0
		sequence 177 flags 0x0(none)
		atime 1483277713.141980592 (2017-01-01 21:35:13)
		ctime 1563162901.234656246 (2019-07-15 11:55:01)
		mtime 1406534032.158605559 (2014-07-28 15:53:52)
		otime 0.0 (1970-01-01 08:00:00)
	item 10 key (7415 INODE_REF 260) itemoff 2837 itemsize 19
		index 28 namelen 9 name: backlight
	item 11 key (7415 INODE_REF 286) itemoff 2818 itemsize 19
		index 3 namelen 9 name: backlight
	item 12 key (7415 DIR_ITEM 3128336373) itemoff 2746 itemsize 72
		location key (120417 INODE_ITEM 0) type FILE
		transid 117279 data_len 0 name_len 42
		name: pci-0000:00:02.0:backlight:intel_backlight
	item 13 key (7415 DIR_ITEM 3218198317) itemoff 2705 itemsize 41
		location key (7487 INODE_ITEM 0) type FILE
		transid 5992 data_len 0 name_len 11
		name: acpi_video0
	item 14 key (7415 DIR_ITEM 3582254411) itemoff 2638 itemsize 67
		location key (55325 INODE_ITEM 0) type FILE
		transid 63351 data_len 0 name_len 37
		name: platform-VPC2004:00:backlight:ideapad
	item 15 key (7415 DIR_INDEX 2) itemoff 2597 itemsize 41
		location key (7487 INODE_ITEM 0) type FILE
		transid 5992 data_len 0 name_len 11
		name: acpi_video0
	item 16 key (7415 DIR_INDEX 4) itemoff 2530 itemsize 67
		location key (55325 INODE_ITEM 0) type FILE
		transid 63351 data_len 0 name_len 37
		name: platform-VPC2004:00:backlight:ideapad
	item 17 key (7415 DIR_INDEX 5) itemoff 2458 itemsize 72
		location key (120417 INODE_ITEM 0) type FILE
		transid 117279 data_len 0 name_len 42
		name: pci-0000:00:02.0:backlight:intel_backlight


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Saturday, July 17th, 2021 at 3:05 PM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:

> On 2021/7/17 上午9:45, pepperpoint@mb.ardentcoding.com wrote:
>
> > Hello,
> >
> > I see this message on dmesg:
> >
> > [ 2452.256756] BTRFS critical (device dm-0): corrupt leaf: root=363 block=174113599488 slot=9 ino=7415, invalid nlink: has 2 expect no more than 1 for dir
> >
> > [ 2452.256776] BTRFS error (device dm-0): block=174113599488 read time tree block corruption detected
>
> Please provide the following dump:
>
> btrfs ins dump-tree -b 174113599488 /dev/dm-0
> =============================================
>
> Thanks,
>
> Qu
>
> > When I run btrfs scrub and btrfs check, no error was detected.
> >
> > I am running Linux 5.12.15-arch1-1 and btrfs-progs v5.12.1
> >
> > How should I fix this error?

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: Read time tree block corruption detected
  2021-07-17  1:45 Read " pepperpoint
@ 2021-07-17  7:05 ` Qu Wenruo
       [not found] ` <162650555086.7.16811903270475408953.10183708@simplelogin.co>
  1 sibling, 0 replies; 75+ messages in thread
From: Qu Wenruo @ 2021-07-17  7:05 UTC (permalink / raw)
  To: pepperpoint, linux-btrfs



On 2021/7/17 上午9:45, pepperpoint@mb.ardentcoding.com wrote:
> Hello,
>
> I see this message on dmesg:
>
> [ 2452.256756] BTRFS critical (device dm-0): corrupt leaf: root=363 block=174113599488 slot=9 ino=7415, invalid nlink: has 2 expect no more than 1 for dir
> [ 2452.256776] BTRFS error (device dm-0): block=174113599488 read time tree block corruption detected

Please provide the following dump:

# btrfs ins dump-tree -b 174113599488 /dev/dm-0

Thanks,
Qu
>
> When I run btrfs scrub and btrfs check, no error was detected.
> I am running Linux 5.12.15-arch1-1 and btrfs-progs v5.12.1
>
> How should I fix this error?
>

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Read time tree block corruption detected
@ 2021-07-17  1:45 pepperpoint
  2021-07-17  7:05 ` Qu Wenruo
       [not found] ` <162650555086.7.16811903270475408953.10183708@simplelogin.co>
  0 siblings, 2 replies; 75+ messages in thread
From: pepperpoint @ 2021-07-17  1:45 UTC (permalink / raw)
  To: linux-btrfs

Hello,

I see this message on dmesg:

[ 2452.256756] BTRFS critical (device dm-0): corrupt leaf: root=363 block=174113599488 slot=9 ino=7415, invalid nlink: has 2 expect no more than 1 for dir
[ 2452.256776] BTRFS error (device dm-0): block=174113599488 read time tree block corruption detected

When I run btrfs scrub and btrfs check, no error was detected.
I am running Linux 5.12.15-arch1-1 and btrfs-progs v5.12.1

How should I fix this error?


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2021-05-26 23:03                     ` Gervais, Francois
@ 2021-05-26 23:25                       ` Qu Wenruo
  0 siblings, 0 replies; 75+ messages in thread
From: Qu Wenruo @ 2021-05-26 23:25 UTC (permalink / raw)
  To: Gervais, Francois, linux-btrfs, Filipe Manana



On 2021/5/27 上午7:03, Gervais, Francois wrote:
> I had to re-send this message as I forgot to send it as plain text earlier today. I apologize if some people receive it twice.
>
>>>> - Anything you would recommend as of configuration of the device?
>>>
>>> You can test using dm-logwrites, which really logs every writes and
>>> replay it.
>>> By using dm-logwrites, you can emulate powerloss for each write operation.
>
> I'm in the process of putting together a test scenario using dm-log-writes on our
> device but I have a few questions.
>
> 1. What I'm thinking of doing is do the firmware operation that the device was
> doing when we got the corruption but this time using dm-log-writes and then
> replay the log entry per entry.
>
> Will tree-checker catch to issue just by replaying or do I need to do an additional
> step for every log entry replay? (btrfs check? btrfs scrub?)

Normally we only care about the filesystem consistency at FUA/FLUSH.

Doing extra check at each replay won't hurt, but it would be very slow
though.

And for the tool to catch the problem, I'm afraid you have to use both
kernel tree-checker (aka mount it), and btrfs-check to catch all problems.

Tree-check can check all involved metadata, while btrfs-check doesn't
check log tree.

>
> 2. From what I understand, our corruption of the log-tree can either be software
> which would have requested a corrupted entry to be written to disk or hardware
> which would have corrupted the entry when trying to write it to disk.

I don't believe it's hardware, one point to notice is, all tree block
written back to disk has checksum.

If it's hardware problem leading to some data corruption on-disk, its
csum should not match at all.

Thus I believe this bug is mostly from btrfs.

Filipe is doing a lot of fixes related to log tree, and some of them are
related to item size overflow.
But if you can reproduce the problem reliably, then it would be a great
help to pin down the cause.

>
> Debugging with dm-log-writes would not catch a corruption by the hardware if due
> to a one-off glitch or something right?

Although I don't believe it's hardware problem, but just to be clear,
no, it won't detect any hardware problem.

Dm-log-writes is only to verify the filesystem layer is doing correct work.

Thanks,
Qu

>
>>>>     - Should we run a newer kernel than our current v5.4?
>>>
>>> Definitely. In fact my fuzzy memory points me to some fix, but I can't
>>> remember exactly which fix.
>>>
>>>>     - Any debug you think would be useful to enable or add?
>>>>
>>>
>>> Tree-checker, which is already enabled by default (in fact no way to
>>> disable) in newer kernels.
>>>
>>> Thanks,
>>> Qu
>>
>> Thank you very much,
>>
>> I will report here if/when we get more information on this issue.

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2021-04-22 14:26                   ` Gervais, Francois
@ 2021-05-26 23:03                     ` Gervais, Francois
  2021-05-26 23:25                       ` Qu Wenruo
  0 siblings, 1 reply; 75+ messages in thread
From: Gervais, Francois @ 2021-05-26 23:03 UTC (permalink / raw)
  To: Qu Wenruo, linux-btrfs, Filipe Manana

I had to re-send this message as I forgot to send it as plain text earlier today. I apologize if some people receive it twice.

>>> - Anything you would recommend as of configuration of the device?
>>
>> You can test using dm-logwrites, which really logs every writes and
>> replay it.
>> By using dm-logwrites, you can emulate powerloss for each write operation.

I'm in the process of putting together a test scenario using dm-log-writes on our
device but I have a few questions.

1. What I'm thinking of doing is do the firmware operation that the device was
doing when we got the corruption but this time using dm-log-writes and then
replay the log entry per entry.

Will tree-checker catch to issue just by replaying or do I need to do an additional
step for every log entry replay? (btrfs check? btrfs scrub?)

2. From what I understand, our corruption of the log-tree can either be software
which would have requested a corrupted entry to be written to disk or hardware
which would have corrupted the entry when trying to write it to disk.

Debugging with dm-log-writes would not catch a corruption by the hardware if due
to a one-off glitch or something right?

>>>    - Should we run a newer kernel than our current v5.4?
>>
>> Definitely. In fact my fuzzy memory points me to some fix, but I can't
>> remember exactly which fix.
>>
>>>    - Any debug you think would be useful to enable or add?
>>>
>>
>> Tree-checker, which is already enabled by default (in fact no way to
>> disable) in newer kernels.
>>
>> Thanks,
>> Qu
>
> Thank you very much,
>
> I will report here if/when we get more information on this issue.

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2021-04-21 23:01                 ` Qu Wenruo
@ 2021-04-22 14:26                   ` Gervais, Francois
  2021-05-26 23:03                     ` Gervais, Francois
  0 siblings, 1 reply; 75+ messages in thread
From: Gervais, Francois @ 2021-04-22 14:26 UTC (permalink / raw)
  To: Qu Wenruo, linux-btrfs, Filipe Manana

>> Alright, for the next step, I feel the best is that we try to reproduce and
>> get more information as of the events that caused state.
>>
>> A few questions ,if you want, before we start.
>>
>> - Anything you would recommend as of configuration of the device?
> 
> You can test using dm-logwrites, which really logs every writes and
> replay it.
> By using dm-logwrites, you can emulate powerloss for each write operation.
> 
>>    - Should we run a newer kernel than our current v5.4?
> 
> Definitely. In fact my fuzzy memory points me to some fix, but I can't
> remember exactly which fix.
> 
>>    - Any debug you think would be useful to enable or add?
>>
> 
> Tree-checker, which is already enabled by default (in fact no way to
> disable) in newer kernels.
> 
> Thanks,
> Qu

Thank you very much,

I will report here if/when we get more information on this issue.

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2021-04-21 14:17               ` Gervais, Francois
@ 2021-04-21 23:01                 ` Qu Wenruo
  2021-04-22 14:26                   ` Gervais, Francois
  0 siblings, 1 reply; 75+ messages in thread
From: Qu Wenruo @ 2021-04-21 23:01 UTC (permalink / raw)
  To: Gervais, Francois, linux-btrfs, Filipe Manana



On 2021/4/21 下午10:17, Gervais, Francois wrote:
>>> Would detecting it at runtime with a newer kernel have helped in any way with
>>> the corruption?
>>
>> Yes, newer kernel will reject the write, so such damaged metadata won't
>> reach disk.
>>
>> But that's just more graceful than corrupted fs.
>> It will still cause error like transaction aborted.
>>
>> [...]
>>
>>> Could power loss be the cause of this issue?
>>
>> It shouldn't.
>> The log tree can only be exposed by power loss, but it's not designed to
>> have such corrupted data on-disk.
>>
>> This normally means some code is wrong when generating log tree.
>
> Alright, for the next step, I feel the best is that we try to reproduce and
> get more information as of the events that caused state.
>
> A few questions ,if you want, before we start.
>
> - Anything you would recommend as of configuration of the device?

You can test using dm-logwrites, which really logs every writes and
replay it.
By using dm-logwrites, you can emulate powerloss for each write operation.

>    - Should we run a newer kernel than our current v5.4?

Definitely. In fact my fuzzy memory points me to some fix, but I can't
remember exactly which fix.

>    - Any debug you think would be useful to enable or add?
>

Tree-checker, which is already enabled by default (in fact no way to
disable) in newer kernels.

Thanks,
Qu

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2021-04-20 23:47             ` Qu Wenruo
@ 2021-04-21 14:17               ` Gervais, Francois
  2021-04-21 23:01                 ` Qu Wenruo
  0 siblings, 1 reply; 75+ messages in thread
From: Gervais, Francois @ 2021-04-21 14:17 UTC (permalink / raw)
  To: Qu Wenruo, linux-btrfs, Filipe Manana

>> Would detecting it at runtime with a newer kernel have helped in any way with
>> the corruption?
> 
> Yes, newer kernel will reject the write, so such damaged metadata won't 
> reach disk.
> 
> But that's just more graceful than corrupted fs.
> It will still cause error like transaction aborted.
> 
> [...]
> 
>> Could power loss be the cause of this issue?
> 
> It shouldn't.
> The log tree can only be exposed by power loss, but it's not designed to 
> have such corrupted data on-disk.
> 
> This normally means some code is wrong when generating log tree.

Alright, for the next step, I feel the best is that we try to reproduce and
get more information as of the events that caused state.

A few questions ,if you want, before we start.

- Anything you would recommend as of configuration of the device?
  - Should we run a newer kernel than our current v5.4?
  - Any debug you think would be useful to enable or add?

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2021-04-20 14:19           ` Gervais, Francois
@ 2021-04-20 23:47             ` Qu Wenruo
  2021-04-21 14:17               ` Gervais, Francois
  0 siblings, 1 reply; 75+ messages in thread
From: Qu Wenruo @ 2021-04-20 23:47 UTC (permalink / raw)
  To: Gervais, Francois, linux-btrfs, Filipe Manana



On 2021/4/20 下午10:19, Gervais, Francois wrote:
>> On 2021/4/19 下午10:56, Gervais, Francois wrote:
>>>> My bad, wrong number.
>>>>
>>>> The correct number command is:
>>>> # btrfs ins dump-tree -b 790151168 /dev/loop0p3
>>>
>>>
>>> root@debug:~# btrfs ins dump-tree -b 790151168 /dev/loop0p3
>>> btrfs-progs v5.7
>> [...]
>>>          item 4 key (5007 INODE_ITEM 0) itemoff 15760 itemsize 160
>>>                  generation 294 transid 219603 size 0 nbytes 18446462598731726987
>>
>> The nbytes looks very strange.
>>
>> It's 0x0xfffeffffffef008b, which definitely looks aweful for an empty inode.
>>
>>>                  block group 0 mode 100600 links 1 uid 1000 gid 1000 rdev 0
>>>                  sequence 476091 flags 0x0(none)
>>>                  atime 1610373772.750632843 (2021-01-11 14:02:52)
>>>                  ctime 1617477826.205928110 (2021-04-03 19:23:46)
>>>                  mtime 1617477826.205928110 (2021-04-03 19:23:46)
>>>                  otime 0.0 (1970-01-01 00:00:00)
>>>          item 5 key (5007 INODE_REF 4727) itemoff 15732 itemsize 28
>>>                  index 0 namelen 0 name:
>>>                  index 0 namelen 0 name:
>>>                  index 0 namelen 294 name:
>>
>> Definitely corrupted. I'm afraid tree-checker is correct.
>>
>> The log tree is corrupted.
>> And the check to detect such corrupted inode ref is only introduced in
>> v5.5 kernel, no wonder v5.4 kernel didn't catch it at runtime.
> 
> Would detecting it at runtime with a newer kernel have helped in any way with
> the corruption?

Yes, newer kernel will reject the write, so such damaged metadata won't 
reach disk.

But that's just more graceful than corrupted fs.
It will still cause error like transaction aborted.

> 
>>
>> I don't have any idea why this could happen, as it doesn't look like an
>> obvious memory flip.
> 
> The test engineer says that the last thing he did was remove power from the
> device.
> 
> Could power loss be the cause of this issue?

It shouldn't.
The log tree can only be exposed by power loss, but it's not designed to 
have such corrupted data on-disk.

This normally means some code is wrong when generating log tree.

Thanks,
Qu

> 
>>
>> Maybe Filipe could have some clue on this?
>>
>> Thanks,
>> Qu
>>
>>>          item 6 key (5041 INODE_ITEM 0) itemoff 15572 itemsize 160
>>>                  generation 295 transid 219603 size 4096 nbytes 4096
>>>                  block group 0 mode 100600 links 1 uid 1000 gid 1000 rdev 0
>>>                  sequence 321954 flags 0x0(none)
>>>                  atime 1610373832.763235044 (2021-01-11 14:03:52)
>>>                  ctime 1617477815.541863825 (2021-04-03 19:23:35)
>>>                  mtime 1617477815.541863825 (2021-04-03 19:23:35)
>>>                  otime 0.0 (1970-01-01 00:00:00)
>>>          item 7 key (5041 INODE_REF 4727) itemoff 15544 itemsize 28
>>>                  index 12 namelen 18 name: health_metrics.txt
>>>          item 8 key (5041 EXTENT_DATA 0) itemoff 15491 itemsize 53
>>>                  generation 219603 type 1 (regular)
>>>                  extent data disk byte 12746752 nr 4096
>>>                  extent data offset 0 nr 4096 ram 4096
>>>                  extent compression 0 (none)
>>>          item 9 key (EXTENT_CSUM EXTENT_CSUM 12746752) itemoff 15487 itemsize 4
>>>                  range start 12746752 end 12750848 length 4096
>> >

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2021-04-20  1:34         ` Qu Wenruo
@ 2021-04-20 14:19           ` Gervais, Francois
  2021-04-20 23:47             ` Qu Wenruo
  0 siblings, 1 reply; 75+ messages in thread
From: Gervais, Francois @ 2021-04-20 14:19 UTC (permalink / raw)
  To: Qu Wenruo, linux-btrfs, Filipe Manana

> On 2021/4/19 下午10:56, Gervais, Francois wrote:
>>> My bad, wrong number.
>>>
>>> The correct number command is:
>>> # btrfs ins dump-tree -b 790151168 /dev/loop0p3
>>
>>
>> root@debug:~# btrfs ins dump-tree -b 790151168 /dev/loop0p3
>> btrfs-progs v5.7
> [...]
>>        item 4 key (5007 INODE_ITEM 0) itemoff 15760 itemsize 160
>>                generation 294 transid 219603 size 0 nbytes 18446462598731726987
> 
> The nbytes looks very strange.
> 
> It's 0x0xfffeffffffef008b, which definitely looks aweful for an empty inode.
> 
>>                block group 0 mode 100600 links 1 uid 1000 gid 1000 rdev 0
>>                sequence 476091 flags 0x0(none)
>>                atime 1610373772.750632843 (2021-01-11 14:02:52)
>>                ctime 1617477826.205928110 (2021-04-03 19:23:46)
>>                mtime 1617477826.205928110 (2021-04-03 19:23:46)
>>                otime 0.0 (1970-01-01 00:00:00)
>>        item 5 key (5007 INODE_REF 4727) itemoff 15732 itemsize 28
>>                index 0 namelen 0 name:
>>                index 0 namelen 0 name:
>>                index 0 namelen 294 name:
> 
> Definitely corrupted. I'm afraid tree-checker is correct.
> 
> The log tree is corrupted.
> And the check to detect such corrupted inode ref is only introduced in
> v5.5 kernel, no wonder v5.4 kernel didn't catch it at runtime.

Would detecting it at runtime with a newer kernel have helped in any way with
the corruption?

> 
> I don't have any idea why this could happen, as it doesn't look like an
> obvious memory flip.

The test engineer says that the last thing he did was remove power from the
device.

Could power loss be the cause of this issue?

> 
> Maybe Filipe could have some clue on this?
> 
> Thanks,
> Qu
> 
>>        item 6 key (5041 INODE_ITEM 0) itemoff 15572 itemsize 160
>>                generation 295 transid 219603 size 4096 nbytes 4096
>>                block group 0 mode 100600 links 1 uid 1000 gid 1000 rdev 0
>>                sequence 321954 flags 0x0(none)
>>                atime 1610373832.763235044 (2021-01-11 14:03:52)
>>                ctime 1617477815.541863825 (2021-04-03 19:23:35)
>>                mtime 1617477815.541863825 (2021-04-03 19:23:35)
>>                otime 0.0 (1970-01-01 00:00:00)
>>        item 7 key (5041 INODE_REF 4727) itemoff 15544 itemsize 28
>>                index 12 namelen 18 name: health_metrics.txt
>>        item 8 key (5041 EXTENT_DATA 0) itemoff 15491 itemsize 53
>>                generation 219603 type 1 (regular)
>>                extent data disk byte 12746752 nr 4096
>>                extent data offset 0 nr 4096 ram 4096
>>                extent compression 0 (none)
>>        item 9 key (EXTENT_CSUM EXTENT_CSUM 12746752) itemoff 15487 itemsize 4
>>                range start 12746752 end 12750848 length 4096
>>

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2021-04-19 14:56       ` Gervais, Francois
@ 2021-04-20  1:34         ` Qu Wenruo
  2021-04-20 14:19           ` Gervais, Francois
  0 siblings, 1 reply; 75+ messages in thread
From: Qu Wenruo @ 2021-04-20  1:34 UTC (permalink / raw)
  To: Gervais, Francois, linux-btrfs, Filipe Manana



On 2021/4/19 下午10:56, Gervais, Francois wrote:
>> My bad, wrong number.
>>
>> The correct number command is:
>> # btrfs ins dump-tree -b 790151168 /dev/loop0p3
>
>
> root@debug:~# btrfs ins dump-tree -b 790151168 /dev/loop0p3
> btrfs-progs v5.7
[...]
> 	item 4 key (5007 INODE_ITEM 0) itemoff 15760 itemsize 160
> 		generation 294 transid 219603 size 0 nbytes 18446462598731726987

The nbytes looks very strange.

It's 0x0xfffeffffffef008b, which definitely looks aweful for an empty inode.

> 		block group 0 mode 100600 links 1 uid 1000 gid 1000 rdev 0
> 		sequence 476091 flags 0x0(none)
> 		atime 1610373772.750632843 (2021-01-11 14:02:52)
> 		ctime 1617477826.205928110 (2021-04-03 19:23:46)
> 		mtime 1617477826.205928110 (2021-04-03 19:23:46)
> 		otime 0.0 (1970-01-01 00:00:00)
> 	item 5 key (5007 INODE_REF 4727) itemoff 15732 itemsize 28
> 		index 0 namelen 0 name:
> 		index 0 namelen 0 name:
> 		index 0 namelen 294 name:

Definitely corrupted. I'm afraid tree-checker is correct.

The log tree is corrupted.
And the check to detect such corrupted inode ref is only introduced in
v5.5 kernel, no wonder v5.4 kernel didn't catch it at runtime.

I don't have any idea why this could happen, as it doesn't look like an
obvious memory flip.

Maybe Filipe could have some clue on this?

Thanks,
Qu

> 	item 6 key (5041 INODE_ITEM 0) itemoff 15572 itemsize 160
> 		generation 295 transid 219603 size 4096 nbytes 4096
> 		block group 0 mode 100600 links 1 uid 1000 gid 1000 rdev 0
> 		sequence 321954 flags 0x0(none)
> 		atime 1610373832.763235044 (2021-01-11 14:03:52)
> 		ctime 1617477815.541863825 (2021-04-03 19:23:35)
> 		mtime 1617477815.541863825 (2021-04-03 19:23:35)
> 		otime 0.0 (1970-01-01 00:00:00)
> 	item 7 key (5041 INODE_REF 4727) itemoff 15544 itemsize 28
> 		index 12 namelen 18 name: health_metrics.txt
> 	item 8 key (5041 EXTENT_DATA 0) itemoff 15491 itemsize 53
> 		generation 219603 type 1 (regular)
> 		extent data disk byte 12746752 nr 4096
> 		extent data offset 0 nr 4096 ram 4096
> 		extent compression 0 (none)
> 	item 9 key (EXTENT_CSUM EXTENT_CSUM 12746752) itemoff 15487 itemsize 4
> 		range start 12746752 end 12750848 length 4096
>

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2021-04-19 13:33     ` Qu Wenruo
@ 2021-04-19 14:56       ` Gervais, Francois
  2021-04-20  1:34         ` Qu Wenruo
  0 siblings, 1 reply; 75+ messages in thread
From: Gervais, Francois @ 2021-04-19 14:56 UTC (permalink / raw)
  To: Qu Wenruo, linux-btrfs

> My bad, wrong number.
>
> The correct number command is:
> # btrfs ins dump-tree -b 790151168 /dev/loop0p3


root@debug:~# btrfs ins dump-tree -b 790151168 /dev/loop0p3
btrfs-progs v5.7 
leaf 790151168 items 10 free space 15237 generation 219603 owner TREE_LOG
leaf 790151168 flags 0x1(WRITTEN) backref revision 1
fs uuid 29d53427-f943-43ad-a99e-ac695d225d0b
chunk uuid 04c4bf25-55ac-487e-97a3-fbdc84961b4a
	item 0 key (4614 INODE_ITEM 0) itemoff 16123 itemsize 160
		generation 282 transid 219603 size 0 nbytes 0
		block group 0 mode 100600 links 1 uid 1000 gid 1000 rdev 0
		sequence 1345948 flags 0x0(none)
		atime 1610373764.218465480 (2021-01-11 14:02:44)
		ctime 1617477830.389953334 (2021-04-03 19:23:50)
		mtime 1617477830.389953334 (2021-04-03 19:23:50)
		otime 606208.1 (1970-01-08 00:23:28)
	item 1 key (4614 INODE_REF 1020) itemoff 16101 itemsize 22
		index 1217 namelen 12 name: brokerStatus
	item 2 key (4996 INODE_ITEM 0) itemoff 15941 itemsize 160
		generation 290 transid 219603 size 0 nbytes 0
		block group 0 mode 100600 links 1 uid 1000 gid 1000 rdev 0
		sequence 4801736 flags 0x0(none)
		atime 1617304887.496533028 (2021-04-01 19:21:27)
		ctime 1617477830.681955095 (2021-04-03 19:23:50)
		mtime 1617477830.681955095 (2021-04-03 19:23:50)
		otime 0.0 (1970-01-01 00:00:00)
	item 3 key (4996 INODE_REF 4715) itemoff 15920 itemsize 21
		index 9 namelen 11 name: scodes.conf
	item 4 key (5007 INODE_ITEM 0) itemoff 15760 itemsize 160
		generation 294 transid 219603 size 0 nbytes 18446462598731726987
		block group 0 mode 100600 links 1 uid 1000 gid 1000 rdev 0
		sequence 476091 flags 0x0(none)
		atime 1610373772.750632843 (2021-01-11 14:02:52)
		ctime 1617477826.205928110 (2021-04-03 19:23:46)
		mtime 1617477826.205928110 (2021-04-03 19:23:46)
		otime 0.0 (1970-01-01 00:00:00)
	item 5 key (5007 INODE_REF 4727) itemoff 15732 itemsize 28
		index 0 namelen 0 name: 
		index 0 namelen 0 name: 
		index 0 namelen 294 name: 
	item 6 key (5041 INODE_ITEM 0) itemoff 15572 itemsize 160
		generation 295 transid 219603 size 4096 nbytes 4096
		block group 0 mode 100600 links 1 uid 1000 gid 1000 rdev 0
		sequence 321954 flags 0x0(none)
		atime 1610373832.763235044 (2021-01-11 14:03:52)
		ctime 1617477815.541863825 (2021-04-03 19:23:35)
		mtime 1617477815.541863825 (2021-04-03 19:23:35)
		otime 0.0 (1970-01-01 00:00:00)
	item 7 key (5041 INODE_REF 4727) itemoff 15544 itemsize 28
		index 12 namelen 18 name: health_metrics.txt
	item 8 key (5041 EXTENT_DATA 0) itemoff 15491 itemsize 53
		generation 219603 type 1 (regular)
		extent data disk byte 12746752 nr 4096
		extent data offset 0 nr 4096 ram 4096
		extent compression 0 (none)
	item 9 key (EXTENT_CSUM EXTENT_CSUM 12746752) itemoff 15487 itemsize 4
		range start 12746752 end 12750848 length 4096

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2021-04-19 13:20   ` Gervais, Francois
@ 2021-04-19 13:33     ` Qu Wenruo
  2021-04-19 14:56       ` Gervais, Francois
  0 siblings, 1 reply; 75+ messages in thread
From: Qu Wenruo @ 2021-04-19 13:33 UTC (permalink / raw)
  To: Gervais, Francois, linux-btrfs



On 2021/4/19 下午9:20, Gervais, Francois wrote:
>> Please provide the following dump:
>>    #btrfs ins dump-tree -b 18446744073709551610 /dev/loop0p3
>>
>> I'm wondering why write-time tree-check didn't catch it.
>>
>> Thanks,
>> Qu
>
> I get:
>
> root@debug:~# btrfs ins dump-tree -b 18446744073709551610 /dev/loop0p3
> btrfs-progs v5.7
> ERROR: tree block bytenr 18446744073709551610 is not aligned to sectorsize 4096

My bad, wrong number.

The correct number command is:
# btrfs ins dump-tree -b 790151168 /dev/loop0p3

Thanks,
Qu
>
> We have an unusual partition table due to an hardware (cpu) requirement.
> This might be the source of this error?
>
> Disk /dev/loop0: 40763392 sectors, 19.4 GiB
> Sector size (logical/physical): 512/512 bytes
> Disk identifier (GUID): A18E4543-634A-4E8C-B55D-DA1E217C4D98
> Partition table holds up to 24 entries
> Main partition table begins at sector 2 and ends at sector 7
> First usable sector is 8, last usable sector is 40763384
> Partitions will be aligned on 8-sector boundaries
> Total free space is 0 sectors (0 bytes)
>
> Number  Start (sector)    End (sector)  Size       Code  Name
>     1               8           32775   16.0 MiB    8300
>     2           32776          237575   100.0 MiB   8300
>     3          237576        40763384   19.3 GiB    8300
>
>
>
>

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2021-04-17  1:01 ` Qu Wenruo
@ 2021-04-19 13:20   ` Gervais, Francois
  2021-04-19 13:33     ` Qu Wenruo
  0 siblings, 1 reply; 75+ messages in thread
From: Gervais, Francois @ 2021-04-19 13:20 UTC (permalink / raw)
  To: Qu Wenruo, linux-btrfs

> Please provide the following dump:
>   #btrfs ins dump-tree -b 18446744073709551610 /dev/loop0p3
>
> I'm wondering why write-time tree-check didn't catch it.
>
> Thanks,
> Qu

I get:

root@debug:~# btrfs ins dump-tree -b 18446744073709551610 /dev/loop0p3
btrfs-progs v5.7 
ERROR: tree block bytenr 18446744073709551610 is not aligned to sectorsize 4096

We have an unusual partition table due to an hardware (cpu) requirement.
This might be the source of this error?

Disk /dev/loop0: 40763392 sectors, 19.4 GiB
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): A18E4543-634A-4E8C-B55D-DA1E217C4D98
Partition table holds up to 24 entries
Main partition table begins at sector 2 and ends at sector 7
First usable sector is 8, last usable sector is 40763384
Partitions will be aligned on 8-sector boundaries
Total free space is 0 sectors (0 bytes)

Number  Start (sector)    End (sector)  Size       Code  Name
   1               8           32775   16.0 MiB    8300  
   2           32776          237575   100.0 MiB   8300  
   3          237576        40763384   19.3 GiB    8300  





^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2021-04-16 19:35 read " Gervais, Francois
@ 2021-04-17  1:01 ` Qu Wenruo
  2021-04-19 13:20   ` Gervais, Francois
  0 siblings, 1 reply; 75+ messages in thread
From: Qu Wenruo @ 2021-04-17  1:01 UTC (permalink / raw)
  To: Gervais, Francois, linux-btrfs



On 2021/4/17 上午3:35, Gervais, Francois wrote:
> We are using btrfs on one of our embedded devices and we got filesystem corruption on one of them.
>
> This product undergo a lot of tests on our side and apparently it's the first it happened so it seems to be a pretty rare occurrence. However we still want to get to the bottom of this to ensure it doesn't happen in the future.
>
> Some background:
> - The corruption happened on kernel v5.4.72.
> - On the debug device I'm on master (v5.12.0-rc7) hoping it might help to have all the latest patches.
>
> Here what kernel v5.12.0-rc7 tells me when trying to mount the partition:
>
> Apr 16 19:31:45 buildroot kernel: BTRFS info (device loop0p3): disk space caching is enabled
> Apr 16 19:31:45 buildroot kernel: BTRFS info (device loop0p3): has skinny extents
> Apr 16 19:31:45 buildroot kernel: BTRFS info (device loop0p3): start tree-log replay
> Apr 16 19:31:45 buildroot kernel: BTRFS critical (device loop0p3): corrupt leaf: root=18446744073709551610 block=790151168 slot=5 ino=5007, inode ref overflow, ptr 15853 end 15861 namelen 294

Please provide the following dump:
  #btrfs ins dump-tree -b 18446744073709551610 /dev/loop0p3

I'm wondering why write-time tree-check didn't catch it.

Thanks,
Qu
> Apr 16 19:31:45 buildroot kernel: BTRFS error (device loop0p3): block=790151168 read time tree block corruption detected
> Apr 16 19:31:45 buildroot kernel: BTRFS critical (device loop0p3): corrupt leaf: root=18446744073709551610 block=790151168 slot=5 ino=5007, inode ref overflow, ptr 15853 end 15861 namelen 294
> Apr 16 19:31:45 buildroot kernel: BTRFS error (device loop0p3): block=790151168 read time tree block corruption detected
> Apr 16 19:31:45 buildroot kernel: BTRFS: error (device loop0p3) in btrfs_recover_log_trees:6246: errno=-5 IO failure (Couldn't read tree log root.)
> Apr 16 19:31:45 buildroot kernel: BTRFS: error (device loop0p3) in btrfs_replay_log:2341: errno=-5 IO failure (Failed to recover log tree)
> Apr 16 19:31:45 buildroot e512c123daaa[468]: mount: /root/mnt: can't read superblock on /dev/loop0p3.
> Apr 16 19:31:45 buildroot kernel: BTRFS error (device loop0p3): open_ctree failed: -5
>
> Any suggestions?
>

^ permalink raw reply	[flat|nested] 75+ messages in thread

* read time tree block corruption detected
@ 2021-04-16 19:35 Gervais, Francois
  2021-04-17  1:01 ` Qu Wenruo
  0 siblings, 1 reply; 75+ messages in thread
From: Gervais, Francois @ 2021-04-16 19:35 UTC (permalink / raw)
  To: linux-btrfs

We are using btrfs on one of our embedded devices and we got filesystem corruption on one of them.

This product undergo a lot of tests on our side and apparently it's the first it happened so it seems to be a pretty rare occurrence. However we still want to get to the bottom of this to ensure it doesn't happen in the future.

Some background:
- The corruption happened on kernel v5.4.72.
- On the debug device I'm on master (v5.12.0-rc7) hoping it might help to have all the latest patches.

Here what kernel v5.12.0-rc7 tells me when trying to mount the partition:

Apr 16 19:31:45 buildroot kernel: BTRFS info (device loop0p3): disk space caching is enabled
Apr 16 19:31:45 buildroot kernel: BTRFS info (device loop0p3): has skinny extents
Apr 16 19:31:45 buildroot kernel: BTRFS info (device loop0p3): start tree-log replay
Apr 16 19:31:45 buildroot kernel: BTRFS critical (device loop0p3): corrupt leaf: root=18446744073709551610 block=790151168 slot=5 ino=5007, inode ref overflow, ptr 15853 end 15861 namelen 294
Apr 16 19:31:45 buildroot kernel: BTRFS error (device loop0p3): block=790151168 read time tree block corruption detected
Apr 16 19:31:45 buildroot kernel: BTRFS critical (device loop0p3): corrupt leaf: root=18446744073709551610 block=790151168 slot=5 ino=5007, inode ref overflow, ptr 15853 end 15861 namelen 294
Apr 16 19:31:45 buildroot kernel: BTRFS error (device loop0p3): block=790151168 read time tree block corruption detected
Apr 16 19:31:45 buildroot kernel: BTRFS: error (device loop0p3) in btrfs_recover_log_trees:6246: errno=-5 IO failure (Couldn't read tree log root.)
Apr 16 19:31:45 buildroot kernel: BTRFS: error (device loop0p3) in btrfs_replay_log:2341: errno=-5 IO failure (Failed to recover log tree)
Apr 16 19:31:45 buildroot e512c123daaa[468]: mount: /root/mnt: can't read superblock on /dev/loop0p3.
Apr 16 19:31:45 buildroot kernel: BTRFS error (device loop0p3): open_ctree failed: -5

Any suggestions?

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: Read time tree block corruption detected
  2020-05-07  0:51     ` Tyler Richmond
@ 2020-05-07  1:06       ` Chris Murphy
  0 siblings, 0 replies; 75+ messages in thread
From: Chris Murphy @ 2020-05-07  1:06 UTC (permalink / raw)
  To: Tyler Richmond; +Cc: Chris Murphy, Btrfs BTRFS

On Wed, May 6, 2020 at 6:51 PM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>
> Kernel version is 5.4.0-28. Just upgraded to Ubuntu 20.04.
> btrfs-progs is version 5.6
>
> That inode search returned an error:
> ERROR: ino paths ioctl: Input/output error
>
> I don't use any subvolumes.
>
> The next command also resulted in an error. Replaced the actual
> mountpoint in the error with /mountpoint.
>
> ERROR: not a block device or regular file: /mountpoint/
> ERROR: device scan /mountpoint/: Input/output error
>
> Something definitely doesn't seem right.

You need to substitute the /mountpoint with the actual mountpoint of
the file system, be it / or /home or /mnt

-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: Read time tree block corruption detected
  2020-05-06 23:55   ` Chris Murphy
@ 2020-05-07  0:51     ` Tyler Richmond
  2020-05-07  1:06       ` Chris Murphy
  0 siblings, 1 reply; 75+ messages in thread
From: Tyler Richmond @ 2020-05-07  0:51 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Btrfs BTRFS

Kernel version is 5.4.0-28. Just upgraded to Ubuntu 20.04.
btrfs-progs is version 5.6

That inode search returned an error:
ERROR: ino paths ioctl: Input/output error

I don't use any subvolumes.

The next command also resulted in an error. Replaced the actual
mountpoint in the error with /mountpoint.

ERROR: not a block device or regular file: /mountpoint/
ERROR: device scan /mountpoint/: Input/output error

Something definitely doesn't seem right.

On Wed, May 6, 2020 at 7:55 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Wed, May 6, 2020 at 3:54 PM Tyler Richmond <t.d.richmond@gmail.com> wrote:
> >
> > Hello,
> >
> > I looked up this error and it basically says ask a developer to
> > determine if it's a false error or not. I just started getting some
> > slow response times, and looked at the dmesg log to find a ton of
> > these errors.
> >
> > [192088.446299] BTRFS critical (device sdh): corrupt leaf: root=5
> > block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> > has 18446744073709551492 expect [0, 6875827]
> > [192088.449823] BTRFS error (device sdh): block=203510940835840 read
> > time tree block corruption detected
> > [192088.459238] BTRFS critical (device sdh): corrupt leaf: root=5
> > block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> > has 18446744073709551492 expect [0, 6875827]
> > [192088.462773] BTRFS error (device sdh): block=203510940835840 read
> > time tree block corruption detected
> > [192088.464711] BTRFS critical (device sdh): corrupt leaf: root=5
> > block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> > has 18446744073709551492 expect [0, 6875827]
> > [192088.468457] BTRFS error (device sdh): block=203510940835840 read
> > time tree block corruption detected
> >
> > btrfs device stats, however, doesn't show any errors.
> >
> > Is there anything I should do about this, or should I just continue
> > using my array as normal?
>
> What kernel version? This looks like relatively recent kernel
> reporting format. Can you search for inode 1311670? e.g.
>
> $ sudo btrfs insp ino -v 1311670 /mountpoint
>
> Note that each subvolume has its own set of inodes. You need to point
> the command to the correct subvolume. In this case it's root=5 which
> is the default/top-level. As long as you haven't changed the default
> subvolume, and you've mounted the file system without subvol or
> subvolid option, it should point to the correct file that's affected
> by this. And also maybe useful:
>
> $ sudo btrfs insp dump-t -b 203510940835840 /mountpoint
>
>
>
> --
> Chris Murphy

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: Read time tree block corruption detected
  2020-05-06 21:54 ` Fwd: Read " Tyler Richmond
@ 2020-05-06 23:55   ` Chris Murphy
  2020-05-07  0:51     ` Tyler Richmond
  0 siblings, 1 reply; 75+ messages in thread
From: Chris Murphy @ 2020-05-06 23:55 UTC (permalink / raw)
  To: Tyler Richmond; +Cc: Btrfs BTRFS

On Wed, May 6, 2020 at 3:54 PM Tyler Richmond <t.d.richmond@gmail.com> wrote:
>
> Hello,
>
> I looked up this error and it basically says ask a developer to
> determine if it's a false error or not. I just started getting some
> slow response times, and looked at the dmesg log to find a ton of
> these errors.
>
> [192088.446299] BTRFS critical (device sdh): corrupt leaf: root=5
> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> has 18446744073709551492 expect [0, 6875827]
> [192088.449823] BTRFS error (device sdh): block=203510940835840 read
> time tree block corruption detected
> [192088.459238] BTRFS critical (device sdh): corrupt leaf: root=5
> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> has 18446744073709551492 expect [0, 6875827]
> [192088.462773] BTRFS error (device sdh): block=203510940835840 read
> time tree block corruption detected
> [192088.464711] BTRFS critical (device sdh): corrupt leaf: root=5
> block=203510940835840 slot=4 ino=1311670, invalid inode generation:
> has 18446744073709551492 expect [0, 6875827]
> [192088.468457] BTRFS error (device sdh): block=203510940835840 read
> time tree block corruption detected
>
> btrfs device stats, however, doesn't show any errors.
>
> Is there anything I should do about this, or should I just continue
> using my array as normal?

What kernel version? This looks like relatively recent kernel
reporting format. Can you search for inode 1311670? e.g.

$ sudo btrfs insp ino -v 1311670 /mountpoint

Note that each subvolume has its own set of inodes. You need to point
the command to the correct subvolume. In this case it's root=5 which
is the default/top-level. As long as you haven't changed the default
subvolume, and you've mounted the file system without subvol or
subvolid option, it should point to the correct file that's affected
by this. And also maybe useful:

$ sudo btrfs insp dump-t -b 203510940835840 /mountpoint



-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2020-02-13 14:01   ` Qu Wenruo
@ 2020-02-15 15:34     ` Samir Benmendil
  0 siblings, 0 replies; 75+ messages in thread
From: Samir Benmendil @ 2020-02-15 15:34 UTC (permalink / raw)
  To: linux-btrfs

[-- Attachment #1: Type: text/plain, Size: 2176 bytes --]

On Feb 13, 2020 at 22:01, Qu Wenruo wrote:
> On 2020/2/13 上午8:26, Qu Wenruo wrote:
>> On 2020/2/13 上午5:58, Samir Benmendil wrote:
>>> I've been getting the following "BTRFS errors" for a while now, the 
>>> wiki [0] advises to report such occurrences to this list.
>> 
>> Please provide the following dump:
>> 
>> # btrfs ins dump-tree -b 194756837376 /dev/sda2
>> # btrfs ins dump-tree -b 194347958272 /dev/sda2
>>> 
>>> BTRFS critical (device sda2): corrupt leaf: root=466 block=194756837376
>>> slot=72 ino=1359622 file_offset=475136, extent end overflow, have file
>>> offset 475136 extent num bytes 18446744073709486080
> 
> 	item 72 key (1359622 EXTENT_DATA 475136) itemoff 11140 itemsize 53
> 		generation 954719 type 1 (regular)
> 		extent data disk byte 0 nr 0
> 		extent data offset 0 nr 18446744073709486080 ram 18446744073709486080
> 		extent compression 0 (none)
> 
> Also obvious underflow.
> 
>>> BTRFS critical (device sda2): corrupt leaf: root=466 block=194347958272
>>> slot=131 ino=1357455 file_offset=1044480, extent end overflow, have file
>>> offset 1044480 extent num bytes 18446744073708908544
> 
> 	item 131 key (1357455 EXTENT_DATA 1044480) itemoff 6497 itemsize 53
> 		generation 952602 type 1 (regular)
> 		extent data disk byte 0 nr 0
> 		extent data offset 0 nr 18446744073708908544 ram 18446744073708908544
> 		extent compression 0 (none)
> 
> As you can see, 18446744073708908544 = -653072, which means it overflows.
> 
> Both look like a bug in older kernels.
> 
> Since currently btrfsprogs can't detect nor fix it yet, the only way is
> to delete the offending files.
> 
> You can use the inode number 1357455, and root id 466 to locate the
> offending files, and delete it using older kernels.
> (root id is your subvolume id, which is shown in `btrfs subv list`.
> inode number can be passed to `find` command using `-inum` option)

I deleted the offending file for inode 1357455 using kernel 4.4.165.

However `find` did not return any results for inode 1359622. The newer 
kernel doesn't seem to complain about that anymore though.

Thank you for you help Qu.

Regards
Samir

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2020-02-13  0:26 ` Qu Wenruo
  2020-02-13 13:04   ` Samir Benmendil
@ 2020-02-13 14:01   ` Qu Wenruo
  2020-02-15 15:34     ` Samir Benmendil
  1 sibling, 1 reply; 75+ messages in thread
From: Qu Wenruo @ 2020-02-13 14:01 UTC (permalink / raw)
  To: Samir Benmendil, linux-btrfs


[-- Attachment #1.1: Type: text/plain, Size: 2317 bytes --]



On 2020/2/13 上午8:26, Qu Wenruo wrote:
> 
> 
> On 2020/2/13 上午5:58, Samir Benmendil wrote:
>> Hello,
>>
>> I've been getting the following "BTRFS errors" for a while now, the wiki
>> [0] advises to report such occurrences to this list.
> 
> Please provide the following dump:
> 
> # btrfs ins dump-tree -b 194756837376 /dev/sda2
> # btrfs ins dump-tree -b 194347958272 /dev/sda2
> 
> Thanks,
> Qu
>>
>> BTRFS critical (device sda2): corrupt leaf: root=466 block=194756837376
>> slot=72 ino=1359622 file_offset=475136, extent end overflow, have file
>> offset 475136 extent num bytes 18446744073709486080

	item 72 key (1359622 EXTENT_DATA 475136) itemoff 11140 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 0 nr 0
		extent data offset 0 nr 18446744073709486080 ram 18446744073709486080
		extent compression 0 (none)

Also obvious underflow.

>> BTRFS critical (device sda2): corrupt leaf: root=466 block=194347958272
>> slot=131 ino=1357455 file_offset=1044480, extent end overflow, have file
>> offset 1044480 extent num bytes 18446744073708908544

	item 131 key (1357455 EXTENT_DATA 1044480) itemoff 6497 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 0 nr 0
		extent data offset 0 nr 18446744073708908544 ram 18446744073708908544
		extent compression 0 (none)

As you can see, 18446744073708908544 = -653072, which means it overflows.

Both look like a bug in older kernels.

Since currently btrfsprogs can't detect nor fix it yet, the only way is
to delete the offending files.

You can use the inode number 1357455, and root id 466 to locate the
offending files, and delete it using older kernels.
(root id is your subvolume id, which is shown in `btrfs subv list`.
 inode number can be passed to `find` command using `-inum` option)

The btrfs-progs fix will come soon.

Thanks,
Qu

>> BTRFS error (device sda2): block=194347958272 read time tree block
>> corruption detected
>>
>> I can reproduce these errors consistently by running `updatedb`, I
>> suppose some tree block in one of the file it reads is corrupted.
>>
>> Thanks in advance for your help,
>> Regards,
>> Samir
>>
>> [0]
>> https://btrfs.wiki.kernel.org/index.php/Tree-checker#How_to_handle_such_error
>>
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2020-02-13  0:26 ` Qu Wenruo
@ 2020-02-13 13:04   ` Samir Benmendil
  2020-02-13 14:01   ` Qu Wenruo
  1 sibling, 0 replies; 75+ messages in thread
From: Samir Benmendil @ 2020-02-13 13:04 UTC (permalink / raw)
  To: linux-btrfs


[-- Attachment #1.1: Type: text/plain, Size: 0 bytes --]



[-- Attachment #1.2: btrfs-dump-tree-194347958272.txt --]
[-- Type: text/plain, Size: 40268 bytes --]

btrfs-progs v5.2.2 
leaf 194347958272 items 171 free space 653 generation 954719 owner 466
leaf 194347958272 flags 0x1(WRITTEN) backref revision 1
fs uuid b4947c1b-447f-4521-841e-5882003e3721
chunk uuid 7e21c972-dd41-447e-97e3-867eb86fa440
	item 0 key (1357418 EXTENT_DATA 335872) itemoff 16230 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66214678528 nr 45056
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 1 key (1357418 EXTENT_DATA 466944) itemoff 16177 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66214723584 nr 45056
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 2 key (1357418 EXTENT_DATA 598016) itemoff 16124 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66214768640 nr 40960
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 3 key (1357418 EXTENT_DATA 729088) itemoff 16071 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66214809600 nr 36864
		extent data offset 0 nr 106496 ram 106496
		extent compression 2 (lzo)
	item 4 key (1357419 INODE_ITEM 0) itemoff 15911 itemsize 160
		generation 952602 transid 952602 size 12489 nbytes 16384
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x0(none)
		atime 1548109731.824211431 (2019-01-21 22:28:51)
		ctime 1548109731.824211431 (2019-01-21 22:28:51)
		mtime 1548109731.824211431 (2019-01-21 22:28:51)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 5 key (1357419 INODE_REF 1355322) itemoff 15891 itemsize 20
		index 962 namelen 10 name: cdc_eem.ko
	item 6 key (1357419 EXTENT_DATA 0) itemoff 15838 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66214846464 nr 8192
		extent data offset 0 nr 16384 ram 16384
		extent compression 2 (lzo)
	item 7 key (1357420 INODE_ITEM 0) itemoff 15678 itemsize 160
		generation 952602 transid 952602 size 51569 nbytes 53248
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109731.834211502 (2019-01-21 22:28:51)
		ctime 1548109731.834211502 (2019-01-21 22:28:51)
		mtime 1548109731.834211502 (2019-01-21 22:28:51)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 8 key (1357420 INODE_REF 1355322) itemoff 15659 itemsize 19
		index 963 namelen 9 name: sunhme.ko
	item 9 key (1357420 EXTENT_DATA 0) itemoff 15606 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66214854656 nr 24576
		extent data offset 0 nr 53248 ram 53248
		extent compression 2 (lzo)
	item 10 key (1357421 INODE_ITEM 0) itemoff 15446 itemsize 160
		generation 952602 transid 952602 size 29033 nbytes 32768
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109731.844211572 (2019-01-21 22:28:51)
		ctime 1548109731.844211572 (2019-01-21 22:28:51)
		mtime 1548109731.844211572 (2019-01-21 22:28:51)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 11 key (1357421 INODE_REF 1355322) itemoff 15427 itemsize 19
		index 964 namelen 9 name: vringh.ko
	item 12 key (1357421 EXTENT_DATA 0) itemoff 15374 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66214879232 nr 12288
		extent data offset 0 nr 32768 ram 32768
		extent compression 2 (lzo)
	item 13 key (1357422 INODE_ITEM 0) itemoff 15214 itemsize 160
		generation 952602 transid 952602 size 25825 nbytes 28672
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109731.854211646 (2019-01-21 22:28:51)
		ctime 1548109731.854211646 (2019-01-21 22:28:51)
		mtime 1548109731.854211646 (2019-01-21 22:28:51)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 14 key (1357422 INODE_REF 1355322) itemoff 15190 itemsize 24
		index 965 namelen 14 name: wlcore_sdio.ko
	item 15 key (1357422 EXTENT_DATA 0) itemoff 15137 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66214891520 nr 12288
		extent data offset 0 nr 28672 ram 28672
		extent compression 2 (lzo)
	item 16 key (1357423 INODE_ITEM 0) itemoff 14977 itemsize 160
		generation 952602 transid 952602 size 19273 nbytes 20480
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109731.864211716 (2019-01-21 22:28:51)
		ctime 1548109731.864211716 (2019-01-21 22:28:51)
		mtime 1548109731.864211716 (2019-01-21 22:28:51)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 17 key (1357423 INODE_REF 1355322) itemoff 14953 itemsize 24
		index 966 namelen 14 name: wl1251_sdio.ko
	item 18 key (1357423 EXTENT_DATA 0) itemoff 14900 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66214903808 nr 8192
		extent data offset 0 nr 20480 ram 20480
		extent compression 2 (lzo)
	item 19 key (1357424 INODE_ITEM 0) itemoff 14740 itemsize 160
		generation 952602 transid 952602 size 94969 nbytes 98304
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109731.874211787 (2019-01-21 22:28:51)
		ctime 1548109731.884211857 (2019-01-21 22:28:51)
		mtime 1548109731.884211857 (2019-01-21 22:28:51)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 20 key (1357424 INODE_REF 1355322) itemoff 14715 itemsize 25
		index 967 namelen 15 name: at76c50x-usb.ko
	item 21 key (1357424 EXTENT_DATA 0) itemoff 14662 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66214912000 nr 40960
		extent data offset 0 nr 98304 ram 98304
		extent compression 2 (lzo)
	item 22 key (1357425 INODE_ITEM 0) itemoff 14502 itemsize 160
		generation 952602 transid 952602 size 192233 nbytes 192512
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109731.894211931 (2019-01-21 22:28:51)
		ctime 1548109731.904212001 (2019-01-21 22:28:51)
		mtime 1548109731.904212001 (2019-01-21 22:28:51)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 23 key (1357425 INODE_REF 1355322) itemoff 14480 itemsize 22
		index 968 namelen 12 name: ath9k_htc.ko
	item 24 key (1357425 EXTENT_DATA 0) itemoff 14427 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66214952960 nr 57344
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 25 key (1357425 EXTENT_DATA 131072) itemoff 14374 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215010304 nr 20480
		extent data offset 0 nr 61440 ram 61440
		extent compression 2 (lzo)
	item 26 key (1357426 INODE_ITEM 0) itemoff 14214 itemsize 160
		generation 952602 transid 952602 size 7633 nbytes 8192
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109731.914212071 (2019-01-21 22:28:51)
		ctime 1548109731.914212071 (2019-01-21 22:28:51)
		mtime 1548109731.914212071 (2019-01-21 22:28:51)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 27 key (1357426 INODE_REF 1355322) itemoff 14181 itemsize 33
		index 969 namelen 23 name: team_mode_roundrobin.ko
	item 28 key (1357426 EXTENT_DATA 0) itemoff 14128 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215030784 nr 4096
		extent data offset 0 nr 8192 ram 8192
		extent compression 2 (lzo)
	item 29 key (1357427 INODE_ITEM 0) itemoff 13968 itemsize 160
		generation 952602 transid 952602 size 9633 nbytes 12288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109731.924212145 (2019-01-21 22:28:51)
		ctime 1548109731.924212145 (2019-01-21 22:28:51)
		mtime 1548109731.924212145 (2019-01-21 22:28:51)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 30 key (1357427 INODE_REF 1355322) itemoff 13948 itemsize 20
		index 970 namelen 10 name: dp83867.ko
	item 31 key (1357427 EXTENT_DATA 0) itemoff 13895 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215034880 nr 8192
		extent data offset 0 nr 12288 ram 12288
		extent compression 2 (lzo)
	item 32 key (1357428 INODE_ITEM 0) itemoff 13735 itemsize 160
		generation 952602 transid 952602 size 5377 nbytes 8192
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109731.934212215 (2019-01-21 22:28:51)
		ctime 1548109731.934212215 (2019-01-21 22:28:51)
		mtime 1548109731.934212215 (2019-01-21 22:28:51)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 33 key (1357428 INODE_REF 1355322) itemoff 13718 itemsize 17
		index 971 namelen 7 name: crc8.ko
	item 34 key (1357428 EXTENT_DATA 0) itemoff 13665 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215043072 nr 4096
		extent data offset 0 nr 8192 ram 8192
		extent compression 2 (lzo)
	item 35 key (1357429 INODE_ITEM 0) itemoff 13505 itemsize 160
		generation 952602 transid 952602 size 299289 nbytes 278528
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109731.944212286 (2019-01-21 22:28:51)
		ctime 1548109731.964212430 (2019-01-21 22:28:51)
		mtime 1548109731.964212430 (2019-01-21 22:28:51)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 36 key (1357429 INODE_REF 1355322) itemoff 13483 itemsize 22
		index 972 namelen 12 name: mv88e6xxx.ko
	item 37 key (1357429 EXTENT_DATA 0) itemoff 13430 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215047168 nr 32768
		extent data offset 0 nr 73728 ram 73728
		extent compression 2 (lzo)
	item 38 key (1357429 EXTENT_DATA 73728) itemoff 13377 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 0 nr 0
		extent data offset 0 nr 16384 ram 16384
		extent compression 0 (none)
	item 39 key (1357429 EXTENT_DATA 90112) itemoff 13324 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215079936 nr 4096
		extent data offset 0 nr 8192 ram 8192
		extent compression 2 (lzo)
	item 40 key (1357429 EXTENT_DATA 98304) itemoff 13271 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 0 nr 0
		extent data offset 0 nr 8192 ram 8192
		extent compression 0 (none)
	item 41 key (1357429 EXTENT_DATA 106496) itemoff 13218 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215084032 nr 45056
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 42 key (1357429 EXTENT_DATA 237568) itemoff 13165 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215129088 nr 20480
		extent data offset 0 nr 65536 ram 65536
		extent compression 2 (lzo)
	item 43 key (1357430 INODE_ITEM 0) itemoff 13005 itemsize 160
		generation 952602 transid 952602 size 13905 nbytes 16384
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109731.964212430 (2019-01-21 22:28:51)
		ctime 1548109731.974212500 (2019-01-21 22:28:51)
		mtime 1548109731.974212500 (2019-01-21 22:28:51)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 44 key (1357430 INODE_REF 1355322) itemoff 12981 itemsize 24
		index 973 namelen 14 name: orinoco_tmd.ko
	item 45 key (1357430 EXTENT_DATA 0) itemoff 12928 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215149568 nr 8192
		extent data offset 0 nr 16384 ram 16384
		extent compression 2 (lzo)
	item 46 key (1357431 INODE_ITEM 0) itemoff 12768 itemsize 160
		generation 952602 transid 952602 size 98073 nbytes 98304
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109731.974212500 (2019-01-21 22:28:51)
		ctime 1548109731.984212571 (2019-01-21 22:28:51)
		mtime 1548109731.984212571 (2019-01-21 22:28:51)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 47 key (1357431 INODE_REF 1355322) itemoff 12745 itemsize 23
		index 974 namelen 13 name: virtio_net.ko
	item 48 key (1357431 EXTENT_DATA 0) itemoff 12692 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215157760 nr 45056
		extent data offset 0 nr 98304 ram 98304
		extent compression 2 (lzo)
	item 49 key (1357432 INODE_ITEM 0) itemoff 12532 itemsize 160
		generation 952602 transid 952602 size 12929 nbytes 16384
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109731.994212641 (2019-01-21 22:28:51)
		ctime 1548109731.994212641 (2019-01-21 22:28:51)
		mtime 1548109731.994212641 (2019-01-21 22:28:51)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 50 key (1357432 INODE_REF 1355322) itemoff 12512 itemsize 20
		index 975 namelen 10 name: int51x1.ko
	item 51 key (1357432 EXTENT_DATA 0) itemoff 12459 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215202816 nr 8192
		extent data offset 0 nr 16384 ram 16384
		extent compression 2 (lzo)
	item 52 key (1357433 INODE_ITEM 0) itemoff 12299 itemsize 160
		generation 952602 transid 952602 size 7425 nbytes 8192
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.4212715 (2019-01-21 22:28:52)
		ctime 1548109732.4212715 (2019-01-21 22:28:52)
		mtime 1548109732.4212715 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 53 key (1357433 INODE_REF 1355322) itemoff 12281 itemsize 18
		index 976 namelen 8 name: qsemi.ko
	item 54 key (1357433 EXTENT_DATA 0) itemoff 12228 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215211008 nr 4096
		extent data offset 0 nr 8192 ram 8192
		extent compression 2 (lzo)
	item 55 key (1357434 INODE_ITEM 0) itemoff 12068 itemsize 160
		generation 952602 transid 952602 size 60105 nbytes 61440
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.14212785 (2019-01-21 22:28:52)
		ctime 1548109732.14212785 (2019-01-21 22:28:52)
		mtime 1548109732.14212785 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 56 key (1357434 INODE_REF 1355322) itemoff 12044 itemsize 24
		index 977 namelen 14 name: orinoco_usb.ko
	item 57 key (1357434 EXTENT_DATA 0) itemoff 11991 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215215104 nr 24576
		extent data offset 0 nr 61440 ram 61440
		extent compression 2 (lzo)
	item 58 key (1357435 INODE_ITEM 0) itemoff 11831 itemsize 160
		generation 952602 transid 952602 size 12161 nbytes 12288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.24212855 (2019-01-21 22:28:52)
		ctime 1548109732.34212929 (2019-01-21 22:28:52)
		mtime 1548109732.34212929 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 59 key (1357435 INODE_REF 1355322) itemoff 11810 itemsize 21
		index 978 namelen 11 name: aquantia.ko
	item 60 key (1357435 EXTENT_DATA 0) itemoff 11757 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215239680 nr 4096
		extent data offset 0 nr 12288 ram 12288
		extent compression 2 (lzo)
	item 61 key (1357436 INODE_ITEM 0) itemoff 11597 itemsize 160
		generation 952602 transid 952602 size 75089 nbytes 77824
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.34212929 (2019-01-21 22:28:52)
		ctime 1548109732.44212999 (2019-01-21 22:28:52)
		mtime 1548109732.44212999 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 62 key (1357436 INODE_REF 1355322) itemoff 11578 itemsize 19
		index 979 namelen 9 name: macsec.ko
	item 63 key (1357436 EXTENT_DATA 0) itemoff 11525 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215243776 nr 32768
		extent data offset 0 nr 77824 ram 77824
		extent compression 2 (lzo)
	item 64 key (1357437 INODE_ITEM 0) itemoff 11365 itemsize 160
		generation 952602 transid 952602 size 77249 nbytes 77824
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.54213070 (2019-01-21 22:28:52)
		ctime 1548109732.64213143 (2019-01-21 22:28:52)
		mtime 1548109732.64213143 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 65 key (1357437 INODE_REF 1355322) itemoff 11349 itemsize 16
		index 980 namelen 6 name: jme.ko
	item 66 key (1357437 EXTENT_DATA 0) itemoff 11296 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215276544 nr 36864
		extent data offset 0 nr 77824 ram 77824
		extent compression 2 (lzo)
	item 67 key (1357438 INODE_ITEM 0) itemoff 11136 itemsize 160
		generation 952602 transid 952602 size 28729 nbytes 32768
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.74213214 (2019-01-21 22:28:52)
		ctime 1548109732.74213214 (2019-01-21 22:28:52)
		mtime 1548109732.74213214 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 68 key (1357438 INODE_REF 1355322) itemoff 11114 itemsize 22
		index 981 namelen 12 name: xircom_cb.ko
	item 69 key (1357438 EXTENT_DATA 0) itemoff 11061 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215313408 nr 12288
		extent data offset 0 nr 32768 ram 32768
		extent compression 2 (lzo)
	item 70 key (1357439 INODE_ITEM 0) itemoff 10901 itemsize 160
		generation 952602 transid 952602 size 7761 nbytes 8192
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.84213284 (2019-01-21 22:28:52)
		ctime 1548109732.84213284 (2019-01-21 22:28:52)
		mtime 1548109732.84213284 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 71 key (1357439 INODE_REF 1355322) itemoff 10872 itemsize 29
		index 982 namelen 19 name: team_mode_random.ko
	item 72 key (1357439 EXTENT_DATA 0) itemoff 10819 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215325696 nr 4096
		extent data offset 0 nr 8192 ram 8192
		extent compression 2 (lzo)
	item 73 key (1357440 INODE_ITEM 0) itemoff 10659 itemsize 160
		generation 952602 transid 952602 size 10513 nbytes 12288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.94213357 (2019-01-21 22:28:52)
		ctime 1548109732.94213357 (2019-01-21 22:28:52)
		mtime 1548109732.94213357 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 74 key (1357440 INODE_REF 1355322) itemoff 10637 itemsize 22
		index 983 namelen 12 name: mdio-gpio.ko
	item 75 key (1357440 EXTENT_DATA 0) itemoff 10584 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215329792 nr 8192
		extent data offset 0 nr 12288 ram 12288
		extent compression 2 (lzo)
	item 76 key (1357441 INODE_ITEM 0) itemoff 10424 itemsize 160
		generation 952602 transid 952602 size 23697 nbytes 24576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.94213357 (2019-01-21 22:28:52)
		ctime 1548109732.104213428 (2019-01-21 22:28:52)
		mtime 1548109732.104213428 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 77 key (1357441 INODE_REF 1355322) itemoff 10402 itemsize 22
		index 984 namelen 12 name: rt2800pci.ko
	item 78 key (1357441 EXTENT_DATA 0) itemoff 10349 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215337984 nr 12288
		extent data offset 0 nr 24576 ram 24576
		extent compression 2 (lzo)
	item 79 key (1357442 INODE_ITEM 0) itemoff 10189 itemsize 160
		generation 952602 transid 952602 size 12953 nbytes 16384
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.104213428 (2019-01-21 22:28:52)
		ctime 1548109732.114213498 (2019-01-21 22:28:52)
		mtime 1548109732.114213498 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 80 key (1357442 INODE_REF 1355322) itemoff 10166 itemsize 23
		index 985 namelen 13 name: rt2x00mmio.ko
	item 81 key (1357442 EXTENT_DATA 0) itemoff 10113 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215350272 nr 8192
		extent data offset 0 nr 16384 ram 16384
		extent compression 2 (lzo)
	item 82 key (1357443 INODE_ITEM 0) itemoff 9953 itemsize 160
		generation 952602 transid 952602 size 9593 nbytes 12288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.114213498 (2019-01-21 22:28:52)
		ctime 1548109732.124213571 (2019-01-21 22:28:52)
		mtime 1548109732.124213571 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 83 key (1357443 INODE_REF 1355322) itemoff 9935 itemsize 18
		index 986 namelen 8 name: pppox.ko
	item 84 key (1357443 EXTENT_DATA 0) itemoff 9882 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215358464 nr 8192
		extent data offset 0 nr 12288 ram 12288
		extent compression 2 (lzo)
	item 85 key (1357444 INODE_ITEM 0) itemoff 9722 itemsize 160
		generation 952602 transid 952602 size 24265 nbytes 24576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.124213571 (2019-01-21 22:28:52)
		ctime 1548109732.134213642 (2019-01-21 22:28:52)
		mtime 1548109732.134213642 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 86 key (1357444 INODE_REF 1355322) itemoff 9700 itemsize 22
		index 987 namelen 12 name: ppp_async.ko
	item 87 key (1357444 EXTENT_DATA 0) itemoff 9647 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215366656 nr 12288
		extent data offset 0 nr 24576 ram 24576
		extent compression 2 (lzo)
	item 88 key (1357445 INODE_ITEM 0) itemoff 9487 itemsize 160
		generation 952602 transid 952602 size 102257 nbytes 102400
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.134213642 (2019-01-21 22:28:52)
		ctime 1548109732.144213712 (2019-01-21 22:28:52)
		mtime 1548109732.144213712 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 89 key (1357445 INODE_REF 1355322) itemoff 9467 itemsize 20
		index 988 namelen 10 name: lan743x.ko
	item 90 key (1357445 EXTENT_DATA 0) itemoff 9414 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215378944 nr 45056
		extent data offset 0 nr 102400 ram 102400
		extent compression 2 (lzo)
	item 91 key (1357446 INODE_ITEM 0) itemoff 9254 itemsize 160
		generation 952602 transid 952602 size 288177 nbytes 282624
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.154213785 (2019-01-21 22:28:52)
		ctime 1548109732.184213999 (2019-01-21 22:28:52)
		mtime 1548109732.184213999 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 92 key (1357446 INODE_REF 1355322) itemoff 9235 itemsize 19
		index 989 namelen 9 name: be2net.ko
	item 93 key (1357446 EXTENT_DATA 0) itemoff 9182 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215424000 nr 73728
		extent data offset 0 nr 106496 ram 106496
		extent compression 2 (lzo)
	item 94 key (1357446 EXTENT_DATA 106496) itemoff 9129 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 0 nr 0
		extent data offset 0 nr 8192 ram 8192
		extent compression 0 (none)
	item 95 key (1357446 EXTENT_DATA 114688) itemoff 9076 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215497728 nr 45056
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 96 key (1357446 EXTENT_DATA 245760) itemoff 9023 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215542784 nr 16384
		extent data offset 0 nr 45056 ram 45056
		extent compression 2 (lzo)
	item 97 key (1357447 INODE_ITEM 0) itemoff 8863 itemsize 160
		generation 952602 transid 952602 size 311681 nbytes 315392
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.184213999 (2019-01-21 22:28:52)
		ctime 1548109732.214214214 (2019-01-21 22:28:52)
		mtime 1548109732.214214214 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 98 key (1357447 INODE_REF 1355322) itemoff 8847 itemsize 16
		index 990 namelen 6 name: bna.ko
	item 99 key (1357447 EXTENT_DATA 0) itemoff 8794 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215559168 nr 69632
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 100 key (1357447 EXTENT_DATA 131072) itemoff 8741 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215628800 nr 49152
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 101 key (1357447 EXTENT_DATA 262144) itemoff 8688 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215677952 nr 16384
		extent data offset 0 nr 53248 ram 53248
		extent compression 2 (lzo)
	item 102 key (1357448 INODE_ITEM 0) itemoff 8528 itemsize 160
		generation 952602 transid 952602 size 15081 nbytes 16384
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.224214284 (2019-01-21 22:28:52)
		ctime 1548109732.234214355 (2019-01-21 22:28:52)
		mtime 1548109732.234214355 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 103 key (1357448 INODE_REF 1355322) itemoff 8507 itemsize 21
		index 991 namelen 11 name: b53_mmap.ko
	item 104 key (1357448 EXTENT_DATA 0) itemoff 8454 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215694336 nr 8192
		extent data offset 0 nr 16384 ram 16384
		extent compression 2 (lzo)
	item 105 key (1357449 INODE_ITEM 0) itemoff 8294 itemsize 160
		generation 952602 transid 952602 size 18529 nbytes 20480
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.234214355 (2019-01-21 22:28:52)
		ctime 1548109732.244214428 (2019-01-21 22:28:52)
		mtime 1548109732.244214428 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 106 key (1357449 INODE_REF 1355322) itemoff 8269 itemsize 25
		index 992 namelen 15 name: i2c-algo-bit.ko
	item 107 key (1357449 EXTENT_DATA 0) itemoff 8216 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215702528 nr 8192
		extent data offset 0 nr 20480 ram 20480
		extent compression 2 (lzo)
	item 108 key (1357450 INODE_ITEM 0) itemoff 8056 itemsize 160
		generation 952602 transid 952602 size 63785 nbytes 65536
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.244214428 (2019-01-21 22:28:52)
		ctime 1548109732.254214499 (2019-01-21 22:28:52)
		mtime 1548109732.254214499 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 109 key (1357450 INODE_REF 1355322) itemoff 8034 itemsize 22
		index 993 namelen 12 name: at86rf230.ko
	item 110 key (1357450 EXTENT_DATA 0) itemoff 7981 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215710720 nr 24576
		extent data offset 0 nr 65536 ram 65536
		extent compression 2 (lzo)
	item 111 key (1357451 INODE_ITEM 0) itemoff 7821 itemsize 160
		generation 952602 transid 952602 size 44881 nbytes 45056
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.264214569 (2019-01-21 22:28:52)
		ctime 1548109732.264214569 (2019-01-21 22:28:52)
		mtime 1548109732.264214569 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 112 key (1357451 INODE_REF 1355322) itemoff 7802 itemsize 19
		index 994 namelen 9 name: tehuti.ko
	item 113 key (1357451 EXTENT_DATA 0) itemoff 7749 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215735296 nr 24576
		extent data offset 0 nr 45056 ram 45056
		extent compression 2 (lzo)
	item 114 key (1357452 INODE_ITEM 0) itemoff 7589 itemsize 160
		generation 952602 transid 952602 size 14489 nbytes 16384
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.274214642 (2019-01-21 22:28:52)
		ctime 1548109732.274214642 (2019-01-21 22:28:52)
		mtime 1548109732.274214642 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 115 key (1357452 INODE_REF 1355322) itemoff 7571 itemsize 18
		index 995 namelen 8 name: vxcan.ko
	item 116 key (1357452 EXTENT_DATA 0) itemoff 7518 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215759872 nr 8192
		extent data offset 0 nr 16384 ram 16384
		extent compression 2 (lzo)
	item 117 key (1357453 INODE_ITEM 0) itemoff 7358 itemsize 160
		generation 952602 transid 952602 size 71553 nbytes 73728
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.284214713 (2019-01-21 22:28:52)
		ctime 1548109732.284214713 (2019-01-21 22:28:52)
		mtime 1548109732.284214713 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 118 key (1357453 INODE_REF 1355322) itemoff 7336 itemsize 22
		index 996 namelen 12 name: rt2800usb.ko
	item 119 key (1357453 EXTENT_DATA 0) itemoff 7283 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215768064 nr 24576
		extent data offset 0 nr 73728 ram 73728
		extent compression 2 (lzo)
	item 120 key (1357454 INODE_ITEM 0) itemoff 7123 itemsize 160
		generation 952602 transid 952602 size 73681 nbytes 73728
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.294214783 (2019-01-21 22:28:52)
		ctime 1548109732.304214856 (2019-01-21 22:28:52)
		mtime 1548109732.304214856 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 121 key (1357454 INODE_REF 1355322) itemoff 7105 itemsize 18
		index 997 namelen 8 name: atmel.ko
	item 122 key (1357454 EXTENT_DATA 0) itemoff 7052 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215792640 nr 36864
		extent data offset 0 nr 73728 ram 73728
		extent compression 2 (lzo)
	item 123 key (1357455 INODE_ITEM 0) itemoff 6892 itemsize 160
		generation 952602 transid 952602 size 1040537 nbytes 507904
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548109732.314214927 (2019-01-21 22:28:52)
		ctime 1548109732.394215499 (2019-01-21 22:28:52)
		mtime 1548109732.394215499 (2019-01-21 22:28:52)
		otime 16107652860835337076.2725157735 (-1108403787-01-21 01:43:05)
	item 124 key (1357455 INODE_REF 1355322) itemoff 6868 itemsize 24
		index 998 namelen 14 name: ath10k_core.ko
	item 125 key (1357455 EXTENT_DATA 0) itemoff 6815 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215829504 nr 86016
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 126 key (1357455 EXTENT_DATA 131072) itemoff 6762 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215915520 nr 77824
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 127 key (1357455 EXTENT_DATA 262144) itemoff 6709 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66215993344 nr 45056
		extent data offset 0 nr 114688 ram 114688
		extent compression 2 (lzo)
	item 128 key (1357455 EXTENT_DATA 376832) itemoff 6656 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 0 nr 0
		extent data offset 0 nr 24576 ram 667648
		extent compression 0 (none)
	item 129 key (1357455 EXTENT_DATA 401408) itemoff 6603 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 66216038400 nr 36864
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 130 key (1357455 EXTENT_DATA 532480) itemoff 6550 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 0 nr 0
		extent data offset 155648 nr 512000 ram 667648
		extent compression 0 (none)
	item 131 key (1357455 EXTENT_DATA 1044480) itemoff 6497 itemsize 53
		generation 952602 type 1 (regular)
		extent data disk byte 0 nr 0
		extent data offset 0 nr 18446744073708908544 ram 18446744073708908544
		extent compression 0 (none)
	item 132 key (1357456 INODE_ITEM 0) itemoff 6337 itemsize 160
		generation 954716 transid 954716 size 8 nbytes 0
		block group 0 mode 40700 links 1 uid 0 gid 0 rdev 0
		sequence 1 flags 0x0(none)
		atime 1548184106.645477005 (2019-01-22 19:08:26)
		ctime 1548184106.665476628 (2019-01-22 19:08:26)
		mtime 1548184106.665476628 (2019-01-22 19:08:26)
		otime 1548184106.645477005 (2019-01-22 19:08:26)
	item 133 key (1357456 INODE_REF 1707) itemoff 6310 itemsize 27
		index 6 namelen 17 name: mkinitcpio.c5mBu2
	item 134 key (1357456 DIR_ITEM 4214885080) itemoff 6276 itemsize 34
		location key (1357457 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 4
		name: root
	item 135 key (1357456 DIR_INDEX 2) itemoff 6242 itemsize 34
		location key (1357457 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 4
		name: root
	item 136 key (1357457 INODE_ITEM 0) itemoff 6082 itemsize 160
		generation 954716 transid 954719 size 190 nbytes 0
		block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
		sequence 19 flags 0x0(none)
		atime 1548184106.665476628 (2019-01-22 19:08:26)
		ctime 1548185863.337550430 (2019-01-22 19:37:43)
		mtime 1548185863.337550430 (2019-01-22 19:37:43)
		otime 1548184106.665476628 (2019-01-22 19:08:26)
	item 137 key (1357457 INODE_REF 1357456) itemoff 6068 itemsize 14
		index 2 namelen 4 name: root
	item 138 key (1357457 DIR_ITEM 145260132) itemoff 6035 itemsize 33
		location key (1357461 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 3
		name: dev
	item 139 key (1357457 DIR_ITEM 217684952) itemoff 6002 itemsize 33
		location key (1357462 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 3
		name: run
	item 140 key (1357457 DIR_ITEM 533965061) itemoff 5958 itemsize 44
		location key (1357649 INODE_ITEM 0) type FILE
		transid 954716 data_len 0 name_len 14
		name: init_functions
	item 141 key (1357457 DIR_ITEM 883143349) itemoff 5925 itemsize 33
		location key (1357470 INODE_ITEM 0) type SYMLINK
		transid 954716 data_len 0 name_len 3
		name: lib
	item 142 key (1357457 DIR_ITEM 935466660) itemoff 5891 itemsize 34
		location key (1357474 INODE_ITEM 0) type SYMLINK
		transid 954716 data_len 0 name_len 4
		name: sbin
	item 143 key (1357457 DIR_ITEM 1253479030) itemoff 5857 itemsize 34
		location key (1357650 INODE_ITEM 0) type FILE
		transid 954716 data_len 0 name_len 4
		name: init
	item 144 key (1357457 DIR_ITEM 1289463356) itemoff 5824 itemsize 33
		location key (1357466 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 3
		name: usr
	item 145 key (1357457 DIR_ITEM 1802746451) itemoff 5789 itemsize 35
		location key (1357479 INODE_ITEM 0) type SYMLINK
		transid 954716 data_len 0 name_len 5
		name: lib64
	item 146 key (1357457 DIR_ITEM 2063292483) itemoff 5748 itemsize 41
		location key (1358618 INODE_ITEM 0) type FILE
		transid 954719 data_len 0 name_len 11
		name: buildconfig
	item 147 key (1357457 DIR_ITEM 2415965623) itemoff 5715 itemsize 33
		location key (1357473 INODE_ITEM 0) type SYMLINK
		transid 954716 data_len 0 name_len 3
		name: bin
	item 148 key (1357457 DIR_ITEM 2753648287) itemoff 5682 itemsize 33
		location key (1357465 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 3
		name: etc
	item 149 key (1357457 DIR_ITEM 2879401604) itemoff 5645 itemsize 37
		location key (1357480 INODE_ITEM 0) type FILE
		transid 954716 data_len 0 name_len 7
		name: VERSION
	item 150 key (1357457 DIR_ITEM 3145042590) itemoff 5609 itemsize 36
		location key (1358617 INODE_ITEM 0) type FILE
		transid 954719 data_len 0 name_len 6
		name: config
	item 151 key (1357457 DIR_ITEM 3195381536) itemoff 5576 itemsize 33
		location key (1357464 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 3
		name: var
	item 152 key (1357457 DIR_ITEM 3284084670) itemoff 5543 itemsize 33
		location key (1357463 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 3
		name: tmp
	item 153 key (1357457 DIR_ITEM 3578798206) itemoff 5510 itemsize 33
		location key (1357460 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 3
		name: sys
	item 154 key (1357457 DIR_ITEM 3655494385) itemoff 5475 itemsize 35
		location key (1357685 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 5
		name: hooks
	item 155 key (1357457 DIR_ITEM 3750368475) itemoff 5437 itemsize 38
		location key (1357458 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 8
		name: new_root
	item 156 key (1357457 DIR_ITEM 4087742454) itemoff 5403 itemsize 34
		location key (1357459 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 4
		name: proc
	item 157 key (1357457 DIR_INDEX 2) itemoff 5365 itemsize 38
		location key (1357458 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 8
		name: new_root
	item 158 key (1357457 DIR_INDEX 3) itemoff 5331 itemsize 34
		location key (1357459 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 4
		name: proc
	item 159 key (1357457 DIR_INDEX 4) itemoff 5298 itemsize 33
		location key (1357460 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 3
		name: sys
	item 160 key (1357457 DIR_INDEX 5) itemoff 5265 itemsize 33
		location key (1357461 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 3
		name: dev
	item 161 key (1357457 DIR_INDEX 6) itemoff 5232 itemsize 33
		location key (1357462 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 3
		name: run
	item 162 key (1357457 DIR_INDEX 7) itemoff 5199 itemsize 33
		location key (1357463 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 3
		name: tmp
	item 163 key (1357457 DIR_INDEX 8) itemoff 5166 itemsize 33
		location key (1357464 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 3
		name: var
	item 164 key (1357457 DIR_INDEX 9) itemoff 5133 itemsize 33
		location key (1357465 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 3
		name: etc
	item 165 key (1357457 DIR_INDEX 10) itemoff 5100 itemsize 33
		location key (1357466 INODE_ITEM 0) type DIR
		transid 954716 data_len 0 name_len 3
		name: usr
	item 166 key (1357457 DIR_INDEX 11) itemoff 5067 itemsize 33
		location key (1357470 INODE_ITEM 0) type SYMLINK
		transid 954716 data_len 0 name_len 3
		name: lib
	item 167 key (1357457 DIR_INDEX 12) itemoff 5034 itemsize 33
		location key (1357473 INODE_ITEM 0) type SYMLINK
		transid 954716 data_len 0 name_len 3
		name: bin
	item 168 key (1357457 DIR_INDEX 13) itemoff 5000 itemsize 34
		location key (1357474 INODE_ITEM 0) type SYMLINK
		transid 954716 data_len 0 name_len 4
		name: sbin
	item 169 key (1357457 DIR_INDEX 14) itemoff 4965 itemsize 35
		location key (1357479 INODE_ITEM 0) type SYMLINK
		transid 954716 data_len 0 name_len 5
		name: lib64
	item 170 key (1357457 DIR_INDEX 15) itemoff 4928 itemsize 37
		location key (1357480 INODE_ITEM 0) type FILE
		transid 954716 data_len 0 name_len 7
		name: VERSION

[-- Attachment #1.3: btrfs-dump-tree-194756837376.txt --]
[-- Type: text/plain, Size: 33616 bytes --]

btrfs-progs v5.2.2 
leaf 194756837376 items 142 free space 2196 generation 1635349 owner 466
leaf 194756837376 flags 0x1(WRITTEN) backref revision 1
fs uuid b4947c1b-447f-4521-841e-5882003e3721
chunk uuid 7e21c972-dd41-447e-97e3-867eb86fa440
	item 0 key (1359605 EXTENT_DATA 0) itemoff 16230 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66847625216 nr 12288
		extent data offset 0 nr 24576 ram 24576
		extent compression 2 (lzo)
	item 1 key (1359606 INODE_ITEM 0) itemoff 16070 itemsize 160
		generation 954719 transid 954719 size 102257 nbytes 102400
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548184228.843170401 (2019-01-22 19:10:28)
		ctime 1548184228.853170212 (2019-01-22 19:10:28)
		mtime 1548184228.853170212 (2019-01-22 19:10:28)
		otime 16107652860835337076.2725157735 (-1108403787-01-22 01:43:05)
	item 2 key (1359606 INODE_REF 1357483) itemoff 16050 itemsize 20
		index 988 namelen 10 name: lan743x.ko
	item 3 key (1359606 EXTENT_DATA 0) itemoff 15997 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66847637504 nr 45056
		extent data offset 0 nr 102400 ram 102400
		extent compression 2 (lzo)
	item 4 key (1359607 INODE_ITEM 0) itemoff 15837 itemsize 160
		generation 954719 transid 954719 size 288177 nbytes 282624
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x0(none)
		atime 1548184228.863170023 (2019-01-22 19:10:28)
		ctime 1548184228.883169645 (2019-01-22 19:10:28)
		mtime 1548184228.883169645 (2019-01-22 19:10:28)
		otime 16107652860835337076.2725157735 (-1108403787-01-22 01:43:05)
	item 5 key (1359607 INODE_REF 1357483) itemoff 15818 itemsize 19
		index 989 namelen 9 name: be2net.ko
	item 6 key (1359607 EXTENT_DATA 0) itemoff 15765 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66847682560 nr 73728
		extent data offset 0 nr 106496 ram 106496
		extent compression 2 (lzo)
	item 7 key (1359607 EXTENT_DATA 106496) itemoff 15712 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 0 nr 0
		extent data offset 0 nr 8192 ram 8192
		extent compression 0 (none)
	item 8 key (1359607 EXTENT_DATA 114688) itemoff 15659 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66847756288 nr 45056
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 9 key (1359607 EXTENT_DATA 245760) itemoff 15606 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66847801344 nr 16384
		extent data offset 0 nr 45056 ram 45056
		extent compression 2 (lzo)
	item 10 key (1359608 INODE_ITEM 0) itemoff 15446 itemsize 160
		generation 954719 transid 954719 size 311681 nbytes 315392
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x0(none)
		atime 1548184228.893169459 (2019-01-22 19:10:28)
		ctime 1548184228.923168892 (2019-01-22 19:10:28)
		mtime 1548184228.923168892 (2019-01-22 19:10:28)
		otime 16107652860835337076.2725157735 (-1108403787-01-22 01:43:05)
	item 11 key (1359608 INODE_REF 1357483) itemoff 15430 itemsize 16
		index 990 namelen 6 name: bna.ko
	item 12 key (1359608 EXTENT_DATA 0) itemoff 15377 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66847817728 nr 69632
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 13 key (1359608 EXTENT_DATA 131072) itemoff 15324 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66847887360 nr 49152
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 14 key (1359608 EXTENT_DATA 262144) itemoff 15271 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66847936512 nr 16384
		extent data offset 0 nr 53248 ram 53248
		extent compression 2 (lzo)
	item 15 key (1359609 INODE_ITEM 0) itemoff 15111 itemsize 160
		generation 954719 transid 954719 size 15081 nbytes 16384
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548184228.933168703 (2019-01-22 19:10:28)
		ctime 1548184228.933168703 (2019-01-22 19:10:28)
		mtime 1548184228.933168703 (2019-01-22 19:10:28)
		otime 16107652860835337076.2725157735 (-1108403787-01-22 01:43:05)
	item 16 key (1359609 INODE_REF 1357483) itemoff 15090 itemsize 21
		index 991 namelen 11 name: b53_mmap.ko
	item 17 key (1359609 EXTENT_DATA 0) itemoff 15037 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66847952896 nr 8192
		extent data offset 0 nr 16384 ram 16384
		extent compression 2 (lzo)
	item 18 key (1359610 INODE_ITEM 0) itemoff 14877 itemsize 160
		generation 954719 transid 954719 size 18529 nbytes 20480
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548184228.933168703 (2019-01-22 19:10:28)
		ctime 1548184228.943168516 (2019-01-22 19:10:28)
		mtime 1548184228.943168516 (2019-01-22 19:10:28)
		otime 16107652860835337076.2725157735 (-1108403787-01-22 01:43:05)
	item 19 key (1359610 INODE_REF 1357483) itemoff 14852 itemsize 25
		index 992 namelen 15 name: i2c-algo-bit.ko
	item 20 key (1359610 EXTENT_DATA 0) itemoff 14799 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66847961088 nr 8192
		extent data offset 0 nr 20480 ram 20480
		extent compression 2 (lzo)
	item 21 key (1359611 INODE_ITEM 0) itemoff 14639 itemsize 160
		generation 954719 transid 954719 size 63785 nbytes 65536
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548184228.943168516 (2019-01-22 19:10:28)
		ctime 1548184228.953168328 (2019-01-22 19:10:28)
		mtime 1548184228.953168328 (2019-01-22 19:10:28)
		otime 16107652860835337076.2725157735 (-1108403787-01-22 01:43:05)
	item 22 key (1359611 INODE_REF 1357483) itemoff 14617 itemsize 22
		index 993 namelen 12 name: at86rf230.ko
	item 23 key (1359611 EXTENT_DATA 0) itemoff 14564 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66847969280 nr 24576
		extent data offset 0 nr 65536 ram 65536
		extent compression 2 (lzo)
	item 24 key (1359612 INODE_ITEM 0) itemoff 14404 itemsize 160
		generation 954719 transid 954719 size 44881 nbytes 45056
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548184228.963168139 (2019-01-22 19:10:28)
		ctime 1548184228.963168139 (2019-01-22 19:10:28)
		mtime 1548184228.963168139 (2019-01-22 19:10:28)
		otime 16107652860835337076.2725157735 (-1108403787-01-22 01:43:05)
	item 25 key (1359612 INODE_REF 1357483) itemoff 14385 itemsize 19
		index 994 namelen 9 name: tehuti.ko
	item 26 key (1359612 EXTENT_DATA 0) itemoff 14332 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66847993856 nr 24576
		extent data offset 0 nr 45056 ram 45056
		extent compression 2 (lzo)
	item 27 key (1359613 INODE_ITEM 0) itemoff 14172 itemsize 160
		generation 954719 transid 954719 size 14489 nbytes 16384
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548184228.973167950 (2019-01-22 19:10:28)
		ctime 1548184228.973167950 (2019-01-22 19:10:28)
		mtime 1548184228.973167950 (2019-01-22 19:10:28)
		otime 16107652860835337076.2725157735 (-1108403787-01-22 01:43:05)
	item 28 key (1359613 INODE_REF 1357483) itemoff 14154 itemsize 18
		index 995 namelen 8 name: vxcan.ko
	item 29 key (1359613 EXTENT_DATA 0) itemoff 14101 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848018432 nr 8192
		extent data offset 0 nr 16384 ram 16384
		extent compression 2 (lzo)
	item 30 key (1359614 INODE_ITEM 0) itemoff 13941 itemsize 160
		generation 954719 transid 954719 size 71553 nbytes 73728
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548184228.983167760 (2019-01-22 19:10:28)
		ctime 1548184228.983167760 (2019-01-22 19:10:28)
		mtime 1548184228.983167760 (2019-01-22 19:10:28)
		otime 16107652860835337076.2725157735 (-1108403787-01-22 01:43:05)
	item 31 key (1359614 INODE_REF 1357483) itemoff 13919 itemsize 22
		index 996 namelen 12 name: rt2800usb.ko
	item 32 key (1359614 EXTENT_DATA 0) itemoff 13866 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848026624 nr 24576
		extent data offset 0 nr 73728 ram 73728
		extent compression 2 (lzo)
	item 33 key (1359615 INODE_ITEM 0) itemoff 13706 itemsize 160
		generation 954719 transid 954719 size 73681 nbytes 73728
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548184228.993167574 (2019-01-22 19:10:28)
		ctime 1548184229.3167385 (2019-01-22 19:10:29)
		mtime 1548184229.3167385 (2019-01-22 19:10:29)
		otime 16107652860835337076.2725157735 (-1108403787-01-22 01:43:05)
	item 34 key (1359615 INODE_REF 1357483) itemoff 13688 itemsize 18
		index 997 namelen 8 name: atmel.ko
	item 35 key (1359615 EXTENT_DATA 0) itemoff 13635 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848051200 nr 36864
		extent data offset 0 nr 73728 ram 73728
		extent compression 2 (lzo)
	item 36 key (1359616 INODE_ITEM 0) itemoff 13475 itemsize 160
		generation 954719 transid 954719 size 1040537 nbytes 1019904
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548184229.13167197 (2019-01-22 19:10:29)
		ctime 1548184229.83165879 (2019-01-22 19:10:29)
		mtime 1548184229.83165879 (2019-01-22 19:10:29)
		otime 16107652860835337076.2725157735 (-1108403787-01-22 01:43:05)
	item 37 key (1359616 INODE_REF 1357483) itemoff 13451 itemsize 24
		index 998 namelen 14 name: ath10k_core.ko
	item 38 key (1359616 EXTENT_DATA 0) itemoff 13398 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848088064 nr 86016
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 39 key (1359616 EXTENT_DATA 131072) itemoff 13345 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848174080 nr 77824
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 40 key (1359616 EXTENT_DATA 262144) itemoff 13292 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848251904 nr 45056
		extent data offset 0 nr 114688 ram 114688
		extent compression 2 (lzo)
	item 41 key (1359616 EXTENT_DATA 376832) itemoff 13239 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 0 nr 0
		extent data offset 0 nr 24576 ram 24576
		extent compression 0 (none)
	item 42 key (1359616 EXTENT_DATA 401408) itemoff 13186 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848296960 nr 36864
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 43 key (1359616 EXTENT_DATA 532480) itemoff 13133 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848333824 nr 49152
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 44 key (1359616 EXTENT_DATA 663552) itemoff 13080 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848382976 nr 45056
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 45 key (1359616 EXTENT_DATA 794624) itemoff 13027 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848428032 nr 40960
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 46 key (1359616 EXTENT_DATA 925696) itemoff 12974 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848468992 nr 40960
		extent data offset 0 nr 118784 ram 118784
		extent compression 2 (lzo)
	item 47 key (1359617 INODE_ITEM 0) itemoff 12814 itemsize 160
		generation 954719 transid 954719 size 28537 nbytes 28672
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548184229.93165690 (2019-01-22 19:10:29)
		ctime 1548184229.93165690 (2019-01-22 19:10:29)
		mtime 1548184229.93165690 (2019-01-22 19:10:29)
		otime 16107652860835337076.2725157735 (-1108403787-01-22 01:43:05)
	item 48 key (1359617 INODE_REF 1357483) itemoff 12795 itemsize 19
		index 999 namelen 9 name: dm9601.ko
	item 49 key (1359617 EXTENT_DATA 0) itemoff 12742 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848509952 nr 12288
		extent data offset 0 nr 28672 ram 28672
		extent compression 2 (lzo)
	item 50 key (1359618 INODE_ITEM 0) itemoff 12582 itemsize 160
		generation 954719 transid 954719 size 36329 nbytes 36864
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548184229.103165501 (2019-01-22 19:10:29)
		ctime 1548184229.113165312 (2019-01-22 19:10:29)
		mtime 1548184229.113165312 (2019-01-22 19:10:29)
		otime 16107652860835337076.2725157735 (-1108403787-01-22 01:43:05)
	item 51 key (1359618 INODE_REF 1357483) itemoff 12559 itemsize 23
		index 1000 namelen 13 name: netconsole.ko
	item 52 key (1359618 EXTENT_DATA 0) itemoff 12506 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848522240 nr 16384
		extent data offset 0 nr 36864 ram 36864
		extent compression 2 (lzo)
	item 53 key (1359619 INODE_ITEM 0) itemoff 12346 itemsize 160
		generation 954719 transid 954719 size 7377 nbytes 8192
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548184229.113165312 (2019-01-22 19:10:29)
		ctime 1548184229.113165312 (2019-01-22 19:10:29)
		mtime 1548184229.113165312 (2019-01-22 19:10:29)
		otime 16107652860835337076.2725157735 (-1108403787-01-22 01:43:05)
	item 54 key (1359619 INODE_REF 1357483) itemoff 12319 itemsize 27
		index 1001 namelen 17 name: ip6_udp_tunnel.ko
	item 55 key (1359619 EXTENT_DATA 0) itemoff 12266 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848538624 nr 4096
		extent data offset 0 nr 8192 ram 8192
		extent compression 2 (lzo)
	item 56 key (1359620 INODE_ITEM 0) itemoff 12106 itemsize 160
		generation 954719 transid 954719 size 140689 nbytes 143360
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548184229.123165123 (2019-01-22 19:10:29)
		ctime 1548184229.133164936 (2019-01-22 19:10:29)
		mtime 1548184229.133164936 (2019-01-22 19:10:29)
		otime 16107652860835337076.2725157735 (-1108403787-01-22 01:43:05)
	item 57 key (1359620 INODE_REF 1357483) itemoff 12088 itemsize 18
		index 1002 namelen 8 name: tulip.ko
	item 58 key (1359620 EXTENT_DATA 0) itemoff 12035 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848542720 nr 57344
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 59 key (1359620 EXTENT_DATA 131072) itemoff 11982 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848600064 nr 4096
		extent data offset 0 nr 12288 ram 12288
		extent compression 2 (lzo)
	item 60 key (1359621 INODE_ITEM 0) itemoff 11822 itemsize 160
		generation 954719 transid 954719 size 11705 nbytes 12288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x0(none)
		atime 1548184229.143164748 (2019-01-22 19:10:29)
		ctime 1548184229.143164748 (2019-01-22 19:10:29)
		mtime 1548184229.143164748 (2019-01-22 19:10:29)
		otime 16107652860835337076.2725157735 (-1108403787-01-22 01:43:05)
	item 61 key (1359621 INODE_REF 1357483) itemoff 11799 itemsize 23
		index 1003 namelen 13 name: kvaser_pci.ko
	item 62 key (1359621 EXTENT_DATA 0) itemoff 11746 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848604160 nr 8192
		extent data offset 0 nr 12288 ram 12288
		extent compression 2 (lzo)
	item 63 key (1359622 INODE_ITEM 0) itemoff 11586 itemsize 160
		generation 954719 transid 954719 size 475136 nbytes 360448
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x0(none)
		atime 1548184229.153164559 (2019-01-22 19:10:29)
		ctime 1548184229.233163052 (2019-01-22 19:10:29)
		mtime 1548184229.233163052 (2019-01-22 19:10:29)
		otime 16107652860835337076.2725157735 (-1108403787-01-22 01:43:05)
	item 64 key (1359622 INODE_REF 1357483) itemoff 11564 itemsize 22
		index 1004 namelen 12 name: bluetooth.ko
	item 65 key (1359622 EXTENT_DATA 0) itemoff 11511 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848624640 nr 86016
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 66 key (1359622 EXTENT_DATA 131072) itemoff 11458 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848710656 nr 94208
		extent data offset 0 nr 131072 ram 131072
		extent compression 2 (lzo)
	item 67 key (1359622 EXTENT_DATA 262144) itemoff 11405 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 0 nr 0
		extent data offset 0 nr 81920 ram 81920
		extent compression 0 (none)
	item 68 key (1359622 EXTENT_DATA 344064) itemoff 11352 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 0 nr 0
		extent data offset 0 nr 8192 ram 40960
		extent compression 0 (none)
	item 69 key (1359622 EXTENT_DATA 352256) itemoff 11299 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848612352 nr 12288
		extent data offset 0 nr 32768 ram 32768
		extent compression 2 (lzo)
	item 70 key (1359622 EXTENT_DATA 385024) itemoff 11246 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 0 nr 0
		extent data offset 0 nr 24576 ram 90112
		extent compression 0 (none)
	item 71 key (1359622 EXTENT_DATA 409600) itemoff 11193 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 66848845824 nr 12288
		extent data offset 0 nr 65536 ram 65536
		extent compression 2 (lzo)
	item 72 key (1359622 EXTENT_DATA 475136) itemoff 11140 itemsize 53
		generation 954719 type 1 (regular)
		extent data disk byte 0 nr 0
		extent data offset 0 nr 18446744073709486080 ram 18446744073709486080
		extent compression 0 (none)
	item 73 key (1359629 INODE_ITEM 0) itemoff 10980 itemsize 160
		generation 954770 transid 1635337 size 1712797 nbytes 1716224
		block group 0 mode 100644 links 1 uid 1000 gid 1000 rdev 0
		sequence 7 flags 0x0(none)
		atime 1572204821.932798461 (2019-10-27 19:33:41)
		ctime 1557856996.368343436 (2019-05-14 19:03:16)
		mtime 1557856996.368343436 (2019-05-14 19:03:16)
		otime 1548187263.115364254 (2019-01-22 20:01:03)
	item 74 key (1359629 INODE_REF 197739) itemoff 10962 itemsize 18
		index 19 namelen 8 name: database
	item 75 key (1359629 EXTENT_DATA 0) itemoff 10909 itemsize 53
		generation 1171531 type 1 (regular)
		extent data disk byte 91207770112 nr 1716224
		extent data offset 0 nr 1716224 ram 1716224
		extent compression 0 (none)
	item 76 key (1359651 INODE_ITEM 0) itemoff 10749 itemsize 160
		generation 954773 transid 1635349 size 64496 nbytes 65536
		block group 0 mode 100644 links 1 uid 1000 gid 1000 rdev 0
		sequence 227 flags 0x0(none)
		atime 1572205149.925702853 (2019-10-27 19:39:09)
		ctime 1572205076.417291893 (2019-10-27 19:37:56)
		mtime 1572205076.417291893 (2019-10-27 19:37:56)
		otime 1548187349.194062509 (2019-01-22 20:02:29)
	item 77 key (1359651 INODE_REF 197739) itemoff 10734 itemsize 15
		index 20 namelen 5 name: state
	item 78 key (1359651 EXTENT_DATA 0) itemoff 10681 itemsize 53
		generation 1635347 type 1 (regular)
		extent data disk byte 81073704960 nr 24576
		extent data offset 0 nr 65536 ram 65536
		extent compression 2 (lzo)
	item 79 key (1359666 INODE_ITEM 0) itemoff 10521 itemsize 160
		generation 954795 transid 1635337 size 32 nbytes 0
		block group 0 mode 40700 links 1 uid 1000 gid 1000 rdev 0
		sequence 1 flags 0x0(none)
		atime 1572204828.962646209 (2019-10-27 19:33:48)
		ctime 1548188187.161114833 (2019-01-22 20:16:27)
		mtime 1548188187.161114833 (2019-01-22 20:16:27)
		otime 1548188187.161114833 (2019-01-22 20:16:27)
	item 80 key (1359666 INODE_REF 223844) itemoff 10479 itemsize 42
		index 8 namelen 32 name: ca7c949c26f976e0f53c14399c2ef02e
	item 81 key (1359666 DIR_ITEM 2051725476) itemoff 10433 itemsize 46
		location key (1359667 INODE_ITEM 0) type DIR
		transid 954795 data_len 0 name_len 16
		name: 0255e75cd17e3ecd
	item 82 key (1359666 DIR_INDEX 2) itemoff 10387 itemsize 46
		location key (1359667 INODE_ITEM 0) type DIR
		transid 954795 data_len 0 name_len 16
		name: 0255e75cd17e3ecd
	item 83 key (1359667 INODE_ITEM 0) itemoff 10227 itemsize 160
		generation 954795 transid 1635337 size 240 nbytes 0
		block group 0 mode 40700 links 1 uid 1000 gid 1000 rdev 0
		sequence 6 flags 0x0(none)
		atime 1572204828.962646209 (2019-10-27 19:33:48)
		ctime 1554329459.806022860 (2019-04-03 23:10:59)
		mtime 1554329459.806022860 (2019-04-03 23:10:59)
		otime 1548188187.161114833 (2019-01-22 20:16:27)
	item 84 key (1359667 INODE_REF 1359666) itemoff 10201 itemsize 26
		index 2 namelen 16 name: 0255e75cd17e3ecd
	item 85 key (1359667 DIR_ITEM 148659992) itemoff 10151 itemsize 50
		location key (1359669 INODE_ITEM 0) type FILE
		transid 954795 data_len 0 name_len 20
		name: 9334bbb63cd97d82.bin
	item 86 key (1359667 DIR_ITEM 2133379428) itemoff 10101 itemsize 50
		location key (1359668 INODE_ITEM 0) type FILE
		transid 954795 data_len 0 name_len 20
		name: 9334bbb63cd97d82.toc
	item 87 key (1359667 DIR_ITEM 2644085418) itemoff 10051 itemsize 50
		location key (1453872 INODE_ITEM 0) type FILE
		transid 1075528 data_len 0 name_len 20
		name: 2d1d750c26fe5f15.toc
	item 88 key (1359667 DIR_ITEM 2777212166) itemoff 10001 itemsize 50
		location key (1448494 INODE_ITEM 0) type FILE
		transid 1072132 data_len 0 name_len 20
		name: 2d1d750c26fe5f14.toc
	item 89 key (1359667 DIR_ITEM 3531371386) itemoff 9951 itemsize 50
		location key (1448495 INODE_ITEM 0) type FILE
		transid 1072132 data_len 0 name_len 20
		name: 2d1d750c26fe5f14.bin
	item 90 key (1359667 DIR_ITEM 3933019350) itemoff 9901 itemsize 50
		location key (1453873 INODE_ITEM 0) type FILE
		transid 1075528 data_len 0 name_len 20
		name: 2d1d750c26fe5f15.bin
	item 91 key (1359667 DIR_INDEX 2) itemoff 9851 itemsize 50
		location key (1359668 INODE_ITEM 0) type FILE
		transid 954795 data_len 0 name_len 20
		name: 9334bbb63cd97d82.toc
	item 92 key (1359667 DIR_INDEX 3) itemoff 9801 itemsize 50
		location key (1359669 INODE_ITEM 0) type FILE
		transid 954795 data_len 0 name_len 20
		name: 9334bbb63cd97d82.bin
	item 93 key (1359667 DIR_INDEX 4) itemoff 9751 itemsize 50
		location key (1448494 INODE_ITEM 0) type FILE
		transid 1072132 data_len 0 name_len 20
		name: 2d1d750c26fe5f14.toc
	item 94 key (1359667 DIR_INDEX 5) itemoff 9701 itemsize 50
		location key (1448495 INODE_ITEM 0) type FILE
		transid 1072132 data_len 0 name_len 20
		name: 2d1d750c26fe5f14.bin
	item 95 key (1359667 DIR_INDEX 6) itemoff 9651 itemsize 50
		location key (1453872 INODE_ITEM 0) type FILE
		transid 1075528 data_len 0 name_len 20
		name: 2d1d750c26fe5f15.toc
	item 96 key (1359667 DIR_INDEX 7) itemoff 9601 itemsize 50
		location key (1453873 INODE_ITEM 0) type FILE
		transid 1075528 data_len 0 name_len 20
		name: 2d1d750c26fe5f15.bin
	item 97 key (1359668 INODE_ITEM 0) itemoff 9441 itemsize 160
		generation 954795 transid 1054781 size 564 nbytes 564
		block group 0 mode 100644 links 1 uid 1000 gid 1000 rdev 0
		sequence 0 flags 0x0(none)
		atime 1553282287.623462056 (2019-03-22 19:18:07)
		ctime 1551909714.45619774 (2019-03-06 22:01:54)
		mtime 1551909714.45619774 (2019-03-06 22:01:54)
		otime 1548188187.161114833 (2019-01-22 20:16:27)
	item 98 key (1359668 INODE_REF 1359667) itemoff 9411 itemsize 30
		index 2 namelen 20 name: 9334bbb63cd97d82.toc
	item 99 key (1359668 EXTENT_DATA 0) itemoff 8909 itemsize 502
		generation 1028065 type 0 (inline)
		inline extent data size 481 ram_bytes 564 compression 2 (lzo)
	item 100 key (1359669 INODE_ITEM 0) itemoff 8749 itemsize 160
		generation 954795 transid 1054781 size 10001 nbytes 12288
		block group 0 mode 100644 links 1 uid 1000 gid 1000 rdev 0
		sequence 0 flags 0x0(none)
		atime 1553282287.633461873 (2019-03-22 19:18:07)
		ctime 1551909714.45619774 (2019-03-06 22:01:54)
		mtime 1551909714.45619774 (2019-03-06 22:01:54)
		otime 1548188187.161114833 (2019-01-22 20:16:27)
	item 101 key (1359669 INODE_REF 1359667) itemoff 8719 itemsize 30
		index 3 namelen 20 name: 9334bbb63cd97d82.bin
	item 102 key (1359669 EXTENT_DATA 0) itemoff 8666 itemsize 53
		generation 976418 type 1 (regular)
		extent data disk byte 1147408384 nr 4096
		extent data offset 0 nr 4096 ram 8192
		extent compression 2 (lzo)
	item 103 key (1359669 EXTENT_DATA 4096) itemoff 8613 itemsize 53
		generation 1028064 type 1 (regular)
		extent data disk byte 58874425344 nr 4096
		extent data offset 0 nr 4096 ram 8192
		extent compression 2 (lzo)
	item 104 key (1359669 EXTENT_DATA 8192) itemoff 8560 itemsize 53
		generation 1028065 type 1 (regular)
		extent data disk byte 63909822464 nr 4096
		extent data offset 0 nr 4096 ram 4096
		extent compression 0 (none)
	item 105 key (1359746 INODE_ITEM 0) itemoff 8400 itemsize 160
		generation 954996 transid 954996 size 42396 nbytes 45056
		block group 0 mode 100644 links 1 uid 1000 gid 1000 rdev 0
		sequence 53 flags 0x0(none)
		atime 1548197385.203646006 (2019-01-22 22:49:45)
		ctime 1548197385.203646006 (2019-01-22 22:49:45)
		mtime 1548197385.203646006 (2019-01-22 22:49:45)
		otime 1548197385.203646006 (2019-01-22 22:49:45)
	item 106 key (1359746 INODE_REF 110364) itemoff 8378 itemsize 22
		index 11204 namelen 12 name: bd5a793e.jpg
	item 107 key (1359746 EXTENT_DATA 0) itemoff 8325 itemsize 53
		generation 954996 type 1 (regular)
		extent data disk byte 4541730816 nr 45056
		extent data offset 0 nr 45056 ram 45056
		extent compression 0 (none)
	item 108 key (1359802 INODE_ITEM 0) itemoff 8165 itemsize 160
		generation 955004 transid 1072255 size 136 nbytes 0
		block group 0 mode 40755 links 1 uid 1000 gid 1000 rdev 0
		sequence 12 flags 0x0(none)
		atime 1554158333.766861801 (2019-04-01 23:38:53)
		ctime 1548197604.229726284 (2019-01-22 22:53:24)
		mtime 1548197604.229726284 (2019-01-22 22:53:24)
		otime 1548197597.639843733 (2019-01-22 22:53:17)
	item 109 key (1359802 INODE_REF 188692) itemoff 8151 itemsize 14
		index 27 namelen 4 name: temp
	item 110 key (1359802 DIR_ITEM 3060314850) itemoff 8087 itemsize 64
		location key (1359810 INODE_ITEM 0) type DIR
		transid 955004 data_len 0 name_len 34
		name: La_casa_de_papel_s1_WEBRip_eng_zip
	item 111 key (1359802 DIR_ITEM 4274709003) itemoff 8023 itemsize 64
		location key (1359809 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 34
		name: La_casa_de_papel_s1_WEBRip_eng.zip
	item 112 key (1359802 DIR_INDEX 2) itemoff 7959 itemsize 64
		location key (1359809 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 34
		name: La_casa_de_papel_s1_WEBRip_eng.zip
	item 113 key (1359802 DIR_INDEX 3) itemoff 7895 itemsize 64
		location key (1359810 INODE_ITEM 0) type DIR
		transid 955004 data_len 0 name_len 34
		name: La_casa_de_papel_s1_WEBRip_eng_zip
	item 114 key (1359809 INODE_ITEM 0) itemoff 7735 itemsize 160
		generation 955004 transid 955004 size 223267 nbytes 225280
		block group 0 mode 100644 links 1 uid 1000 gid 1000 rdev 0
		sequence 9 flags 0x0(none)
		atime 1548197604.59729311 (2019-01-22 22:53:24)
		ctime 1548197604.199726821 (2019-01-22 22:53:24)
		mtime 1548197604.199726821 (2019-01-22 22:53:24)
		otime 1548197604.59729311 (2019-01-22 22:53:24)
	item 115 key (1359809 INODE_REF 1359802) itemoff 7691 itemsize 44
		index 2 namelen 34 name: La_casa_de_papel_s1_WEBRip_eng.zip
	item 116 key (1359809 EXTENT_DATA 0) itemoff 7638 itemsize 53
		generation 955004 type 1 (regular)
		extent data disk byte 61471092736 nr 225280
		extent data offset 0 nr 225280 ram 225280
		extent compression 0 (none)
	item 117 key (1359810 INODE_ITEM 0) itemoff 7478 itemsize 160
		generation 955004 transid 1072255 size 60 nbytes 0
		block group 0 mode 40755 links 1 uid 1000 gid 1000 rdev 0
		sequence 29 flags 0x0(none)
		atime 1554158333.766861801 (2019-04-01 23:38:53)
		ctime 1548197604.229726284 (2019-01-22 22:53:24)
		mtime 1548197604.229726284 (2019-01-22 22:53:24)
		otime 1548197604.229726284 (2019-01-22 22:53:24)
	item 118 key (1359810 INODE_REF 1359802) itemoff 7434 itemsize 44
		index 3 namelen 34 name: La_casa_de_papel_s1_WEBRip_eng_zip
	item 119 key (1359810 DIR_ITEM 3732099734) itemoff 7374 itemsize 60
		location key (1359811 INODE_ITEM 0) type DIR
		transid 955004 data_len 0 name_len 30
		name: La casa de papel s1 WEBRip eng
	item 120 key (1359810 DIR_INDEX 2) itemoff 7314 itemsize 60
		location key (1359811 INODE_ITEM 0) type DIR
		transid 955004 data_len 0 name_len 30
		name: La casa de papel s1 WEBRip eng
	item 121 key (1359811 INODE_ITEM 0) itemoff 7154 itemsize 160
		generation 955004 transid 1072255 size 1092 nbytes 0
		block group 0 mode 40755 links 1 uid 1000 gid 1000 rdev 0
		sequence 38 flags 0x0(none)
		atime 1554158333.766861801 (2019-04-01 23:38:53)
		ctime 1548197604.299725037 (2019-01-22 22:53:24)
		mtime 1548197604.299725037 (2019-01-22 22:53:24)
		otime 1548197604.229726284 (2019-01-22 22:53:24)
	item 122 key (1359811 INODE_REF 1359810) itemoff 7114 itemsize 40
		index 2 namelen 30 name: La casa de papel s1 WEBRip eng
	item 123 key (1359811 DIR_ITEM 329641938) itemoff 7042 itemsize 72
		location key (1359818 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E07.WEBRip.Netflix.srt
	item 124 key (1359811 DIR_ITEM 534648133) itemoff 6970 itemsize 72
		location key (1359820 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E09.WEBRip.Netflix.srt
	item 125 key (1359811 DIR_ITEM 619059311) itemoff 6898 itemsize 72
		location key (1359817 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E06.WEBRip.Netflix.srt
	item 126 key (1359811 DIR_ITEM 681435896) itemoff 6826 itemsize 72
		location key (1359819 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E08.WEBRip.Netflix.srt
	item 127 key (1359811 DIR_ITEM 1247934229) itemoff 6754 itemsize 72
		location key (1359815 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E04.WEBRip.Netflix.srt
	item 128 key (1359811 DIR_ITEM 2099380392) itemoff 6682 itemsize 72
		location key (1359816 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E05.WEBRip.Netflix.srt
	item 129 key (1359811 DIR_ITEM 2590266380) itemoff 6610 itemsize 72
		location key (1359821 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E10.WEBRip.Netflix.srt
	item 130 key (1359811 DIR_ITEM 2687360604) itemoff 6538 itemsize 72
		location key (1359812 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E01.WEBRip.Netflix.srt
	item 131 key (1359811 DIR_ITEM 2905056177) itemoff 6466 itemsize 72
		location key (1359822 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E11.WEBRip.Netflix.srt
	item 132 key (1359811 DIR_ITEM 3282062539) itemoff 6394 itemsize 72
		location key (1359824 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E13.WEBRip.Netflix.srt
	item 133 key (1359811 DIR_ITEM 3467260198) itemoff 6322 itemsize 72
		location key (1359814 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E03.WEBRip.Netflix.srt
	item 134 key (1359811 DIR_ITEM 4108565366) itemoff 6250 itemsize 72
		location key (1359823 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E12.WEBRip.Netflix.srt
	item 135 key (1359811 DIR_ITEM 4192847515) itemoff 6178 itemsize 72
		location key (1359813 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E02.WEBRip.Netflix.srt
	item 136 key (1359811 DIR_INDEX 2) itemoff 6106 itemsize 72
		location key (1359812 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E01.WEBRip.Netflix.srt
	item 137 key (1359811 DIR_INDEX 3) itemoff 6034 itemsize 72
		location key (1359813 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E02.WEBRip.Netflix.srt
	item 138 key (1359811 DIR_INDEX 4) itemoff 5962 itemsize 72
		location key (1359814 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E03.WEBRip.Netflix.srt
	item 139 key (1359811 DIR_INDEX 5) itemoff 5890 itemsize 72
		location key (1359815 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E04.WEBRip.Netflix.srt
	item 140 key (1359811 DIR_INDEX 6) itemoff 5818 itemsize 72
		location key (1359816 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E05.WEBRip.Netflix.srt
	item 141 key (1359811 DIR_INDEX 7) itemoff 5746 itemsize 72
		location key (1359817 INODE_ITEM 0) type FILE
		transid 955004 data_len 0 name_len 42
		name: La.casa.de.papel.S01E06.WEBRip.Netflix.srt

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2020-02-12 21:58 Samir Benmendil
@ 2020-02-13  0:26 ` Qu Wenruo
  2020-02-13 13:04   ` Samir Benmendil
  2020-02-13 14:01   ` Qu Wenruo
  0 siblings, 2 replies; 75+ messages in thread
From: Qu Wenruo @ 2020-02-13  0:26 UTC (permalink / raw)
  To: Samir Benmendil, linux-btrfs


[-- Attachment #1.1: Type: text/plain, Size: 1552 bytes --]



On 2020/2/13 上午5:58, Samir Benmendil wrote:
> Hello,
> 
> I've been getting the following "BTRFS errors" for a while now, the wiki
> [0] advises to report such occurrences to this list.

Please provide the following dump:

# btrfs ins dump-tree -b 194756837376 /dev/sda2
# btrfs ins dump-tree -b 194347958272 /dev/sda2

Thanks,
Qu
> 
> BTRFS critical (device sda2): corrupt leaf: root=466 block=194756837376
> slot=72 ino=1359622 file_offset=475136, extent end overflow, have file
> offset 475136 extent num bytes 18446744073709486080
> BTRFS error (device sda2): block=194756837376 read time tree block
> corruption detected
> BTRFS critical (device sda2): corrupt leaf: root=466 block=194756837376
> slot=72 ino=1359622 file_offset=475136, extent end overflow, have file
> offset 475136 extent num bytes 18446744073709486080
> BTRFS error (device sda2): block=194756837376 read time tree block
> corruption detected
> BTRFS critical (device sda2): corrupt leaf: root=466 block=194347958272
> slot=131 ino=1357455 file_offset=1044480, extent end overflow, have file
> offset 1044480 extent num bytes 18446744073708908544
> BTRFS error (device sda2): block=194347958272 read time tree block
> corruption detected
> 
> I can reproduce these errors consistently by running `updatedb`, I
> suppose some tree block in one of the file it reads is corrupted.
> 
> Thanks in advance for your help,
> Regards,
> Samir
> 
> [0]
> https://btrfs.wiki.kernel.org/index.php/Tree-checker#How_to_handle_such_error
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* read time tree block corruption detected
@ 2020-02-12 21:58 Samir Benmendil
  2020-02-13  0:26 ` Qu Wenruo
  0 siblings, 1 reply; 75+ messages in thread
From: Samir Benmendil @ 2020-02-12 21:58 UTC (permalink / raw)
  To: linux-btrfs

[-- Attachment #1: Type: text/plain, Size: 1272 bytes --]

Hello,

I've been getting the following "BTRFS errors" for a while now, the wiki 
[0] advises to report such occurrences to this list.

BTRFS critical (device sda2): corrupt leaf: root=466 block=194756837376 slot=72 ino=1359622 file_offset=475136, extent end overflow, have file offset 475136 extent num bytes 18446744073709486080
BTRFS error (device sda2): block=194756837376 read time tree block corruption detected
BTRFS critical (device sda2): corrupt leaf: root=466 block=194756837376 slot=72 ino=1359622 file_offset=475136, extent end overflow, have file offset 475136 extent num bytes 18446744073709486080
BTRFS error (device sda2): block=194756837376 read time tree block corruption detected
BTRFS critical (device sda2): corrupt leaf: root=466 block=194347958272 slot=131 ino=1357455 file_offset=1044480, extent end overflow, have file offset 1044480 extent num bytes 18446744073708908544
BTRFS error (device sda2): block=194347958272 read time tree block corruption detected

I can reproduce these errors consistently by running `updatedb`, I 
suppose some tree block in one of the file it reads is corrupted.

Thanks in advance for your help,
Regards,
Samir

[0] https://btrfs.wiki.kernel.org/index.php/Tree-checker#How_to_handle_such_error

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2020-01-17  9:10               ` Nikolay Borisov
@ 2020-01-17 12:04                 ` Peter Luladjiev
  0 siblings, 0 replies; 75+ messages in thread
From: Peter Luladjiev @ 2020-01-17 12:04 UTC (permalink / raw)
  To: Nikolay Borisov; +Cc: linux-btrfs

Thank you very much.
Everything seems fine now.

On Fri, 17 Jan 2020 at 11:10, Nikolay Borisov <nborisov@suse.com> wrote:
>
>
>
> On 17.01.20 г. 10:22 ч., Peter Luladjiev wrote:
> > btrfs-progs v5.4
>
> As Qu mentioned - with the rootfs unmounted you can run btrfs check
> --repair and it should repair that one broken backref.
>
> >
> > On Fri, 17 Jan 2020 at 10:14, Nikolay Borisov <nborisov@suse.com> wrote:
> >>
> >>
> >>
> >> On 17.01.20 г. 9:54 ч., Peter Luladjiev wrote:
> >>> Should I run with repair flag?
> >>>
> >>
> >>
> >> Which version of progs do you have ? Because it seems original mode
> >> (Default) finds the corrupted backref but lowmem doesn't ?
> >>
> >>
> >>

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2020-01-17  8:22             ` Peter Luladjiev
@ 2020-01-17  9:10               ` Nikolay Borisov
  2020-01-17 12:04                 ` Peter Luladjiev
  0 siblings, 1 reply; 75+ messages in thread
From: Nikolay Borisov @ 2020-01-17  9:10 UTC (permalink / raw)
  To: Peter Luladjiev; +Cc: linux-btrfs



On 17.01.20 г. 10:22 ч., Peter Luladjiev wrote:
> btrfs-progs v5.4

As Qu mentioned - with the rootfs unmounted you can run btrfs check
--repair and it should repair that one broken backref.

> 
> On Fri, 17 Jan 2020 at 10:14, Nikolay Borisov <nborisov@suse.com> wrote:
>>
>>
>>
>> On 17.01.20 г. 9:54 ч., Peter Luladjiev wrote:
>>> Should I run with repair flag?
>>>
>>
>>
>> Which version of progs do you have ? Because it seems original mode
>> (Default) finds the corrupted backref but lowmem doesn't ?
>>
>>
>>

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2020-01-17  8:14           ` Nikolay Borisov
@ 2020-01-17  8:22             ` Peter Luladjiev
  2020-01-17  9:10               ` Nikolay Borisov
  0 siblings, 1 reply; 75+ messages in thread
From: Peter Luladjiev @ 2020-01-17  8:22 UTC (permalink / raw)
  To: Nikolay Borisov; +Cc: linux-btrfs

btrfs-progs v5.4

On Fri, 17 Jan 2020 at 10:14, Nikolay Borisov <nborisov@suse.com> wrote:
>
>
>
> On 17.01.20 г. 9:54 ч., Peter Luladjiev wrote:
> > Should I run with repair flag?
> >
>
>
> Which version of progs do you have ? Because it seems original mode
> (Default) finds the corrupted backref but lowmem doesn't ?
>
>
>

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2020-01-17  7:54         ` Peter Luladjiev
  2020-01-17  7:59           ` Qu Wenruo
@ 2020-01-17  8:14           ` Nikolay Borisov
  2020-01-17  8:22             ` Peter Luladjiev
  1 sibling, 1 reply; 75+ messages in thread
From: Nikolay Borisov @ 2020-01-17  8:14 UTC (permalink / raw)
  To: Peter Luladjiev; +Cc: linux-btrfs



On 17.01.20 г. 9:54 ч., Peter Luladjiev wrote:
> Should I run with repair flag?
> 


Which version of progs do you have ? Because it seems original mode
(Default) finds the corrupted backref but lowmem doesn't ?




^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2020-01-17  7:54         ` Peter Luladjiev
@ 2020-01-17  7:59           ` Qu Wenruo
  2020-01-17  8:14           ` Nikolay Borisov
  1 sibling, 0 replies; 75+ messages in thread
From: Qu Wenruo @ 2020-01-17  7:59 UTC (permalink / raw)
  To: Peter Luladjiev, Nikolay Borisov; +Cc: linux-btrfs



On 2020/1/17 下午3:54, Peter Luladjiev wrote:
> Should I run with repair flag?

Please only run --repair with the fs unmounted.

Thanks,
Qu

>
> On Fri, 17 Jan 2020 at 09:51, Peter Luladjiev <luladjiev@gmail.com> wrote:
>>
>> Here is the output:
>>
>> btrfs check --force --mode lowmem /dev/mapper/system-root
>>
>> Opening filesystem to check...
>> WARNING: filesystem mounted, continuing because of --force
>> Checking filesystem on /dev/mapper/system-root
>> UUID: 9639a3e6-cd08-4270-b4d1-d2946d2b8d2e
>> [1/7] checking root items
>> [2/7] checking extents
>> [3/7] checking free space cache
>> btrfs: space cache generation (539645) does not match inode (539641)
>> failed to load free space cache for block group 22020096
>> btrfs: space cache generation (539645) does not match inode (539641)
>> failed to load free space cache for block group 1095761920
>> btrfs: space cache generation (539643) does not match inode (539640)
>> failed to load free space cache for block group 102161711104
>> [4/7] checking fs roots
>> [5/7] checking only csums items (without verifying data)
>> [6/7] checking root refs done with fs roots in lowmem mode, skipping
>> [7/7] checking quota groups skipped (not enabled on this FS)
>> found 53501751296 bytes used, no error found
>> total csum bytes: 43476196
>> total tree bytes: 1552203776
>> total fs tree bytes: 1422196736
>> total extent tree bytes: 70172672
>> btree space waste bytes: 276902557
>> file data blocks allocated: 331882188800
>>  referenced 105424904192
>>
>> On Fri, 17 Jan 2020 at 09:34, Nikolay Borisov <nborisov@suse.com> wrote:
>>>
>>>
>>>
>>> On 16.01.20 г. 18:53 ч., Peter Luladjiev wrote:
>>>> Hello, thanks for helping, here is the output:
>>>>
>>>> btrfs check --force /dev/mapper/system-root
>>>>
>>>> Opening filesystem to check...
>>>> WARNING: filesystem mounted, continuing because of --force
>>>> Checking filesystem on /dev/mapper/system-root
>>>> UUID: 9639a3e6-cd08-4270-b4d1-d2946d2b8d2e
>>>> [1/7] checking root items
>>>> [2/7] checking extents
>>>> ref mismatch on [1497018368 4096] extent item 72057183177116417, found 1
>>>> incorrect local backref count on 1497022464 parent 51611369472 owner 0
>>>> offset 0 found 1 wanted 3087007745 back 0x564582174c70
>>>> backpointer mismatch on [1497022464 4096]
>>>> ERROR: errors found in extent allocation tree or chunk allocation
>>>> [3/7] checking free space cache
>>>> [4/7] checking fs roots
>>>> [5/7] checking only csums items (without verifying data)
>>>> [6/7] checking root refs
>>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>>> found 53532647424 bytes used, error(s) found
>>>> total csum bytes: 43476204
>>>> total tree bytes: 1551368192
>>>> total fs tree bytes: 1421295616
>>>> total extent tree bytes: 70238208
>>>> btree space waste bytes: 276638054
>>>> file data blocks allocated: 331679563776
>>>>  referenced 105423634432
>>>>
>>>
>>>
>>> Right so it seems this the only error. Just to be sure run btrfs check
>>> --mode lowmem  since it provides more readable output. If this is the
>>> only error in the filesystem then btrfs check --repair --mode lowmem
>>> should be able to fix it.

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2020-01-17  7:51       ` Peter Luladjiev
@ 2020-01-17  7:54         ` Peter Luladjiev
  2020-01-17  7:59           ` Qu Wenruo
  2020-01-17  8:14           ` Nikolay Borisov
  0 siblings, 2 replies; 75+ messages in thread
From: Peter Luladjiev @ 2020-01-17  7:54 UTC (permalink / raw)
  To: Nikolay Borisov; +Cc: linux-btrfs

Should I run with repair flag?

On Fri, 17 Jan 2020 at 09:51, Peter Luladjiev <luladjiev@gmail.com> wrote:
>
> Here is the output:
>
> btrfs check --force --mode lowmem /dev/mapper/system-root
>
> Opening filesystem to check...
> WARNING: filesystem mounted, continuing because of --force
> Checking filesystem on /dev/mapper/system-root
> UUID: 9639a3e6-cd08-4270-b4d1-d2946d2b8d2e
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> btrfs: space cache generation (539645) does not match inode (539641)
> failed to load free space cache for block group 22020096
> btrfs: space cache generation (539645) does not match inode (539641)
> failed to load free space cache for block group 1095761920
> btrfs: space cache generation (539643) does not match inode (539640)
> failed to load free space cache for block group 102161711104
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs done with fs roots in lowmem mode, skipping
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 53501751296 bytes used, no error found
> total csum bytes: 43476196
> total tree bytes: 1552203776
> total fs tree bytes: 1422196736
> total extent tree bytes: 70172672
> btree space waste bytes: 276902557
> file data blocks allocated: 331882188800
>  referenced 105424904192
>
> On Fri, 17 Jan 2020 at 09:34, Nikolay Borisov <nborisov@suse.com> wrote:
> >
> >
> >
> > On 16.01.20 г. 18:53 ч., Peter Luladjiev wrote:
> > > Hello, thanks for helping, here is the output:
> > >
> > > btrfs check --force /dev/mapper/system-root
> > >
> > > Opening filesystem to check...
> > > WARNING: filesystem mounted, continuing because of --force
> > > Checking filesystem on /dev/mapper/system-root
> > > UUID: 9639a3e6-cd08-4270-b4d1-d2946d2b8d2e
> > > [1/7] checking root items
> > > [2/7] checking extents
> > > ref mismatch on [1497018368 4096] extent item 72057183177116417, found 1
> > > incorrect local backref count on 1497022464 parent 51611369472 owner 0
> > > offset 0 found 1 wanted 3087007745 back 0x564582174c70
> > > backpointer mismatch on [1497022464 4096]
> > > ERROR: errors found in extent allocation tree or chunk allocation
> > > [3/7] checking free space cache
> > > [4/7] checking fs roots
> > > [5/7] checking only csums items (without verifying data)
> > > [6/7] checking root refs
> > > [7/7] checking quota groups skipped (not enabled on this FS)
> > > found 53532647424 bytes used, error(s) found
> > > total csum bytes: 43476204
> > > total tree bytes: 1551368192
> > > total fs tree bytes: 1421295616
> > > total extent tree bytes: 70238208
> > > btree space waste bytes: 276638054
> > > file data blocks allocated: 331679563776
> > >  referenced 105423634432
> > >
> >
> >
> > Right so it seems this the only error. Just to be sure run btrfs check
> > --mode lowmem  since it provides more readable output. If this is the
> > only error in the filesystem then btrfs check --repair --mode lowmem
> > should be able to fix it.

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2020-01-17  7:34     ` Nikolay Borisov
@ 2020-01-17  7:51       ` Peter Luladjiev
  2020-01-17  7:54         ` Peter Luladjiev
  0 siblings, 1 reply; 75+ messages in thread
From: Peter Luladjiev @ 2020-01-17  7:51 UTC (permalink / raw)
  To: Nikolay Borisov; +Cc: linux-btrfs

Here is the output:

btrfs check --force --mode lowmem /dev/mapper/system-root

Opening filesystem to check...
WARNING: filesystem mounted, continuing because of --force
Checking filesystem on /dev/mapper/system-root
UUID: 9639a3e6-cd08-4270-b4d1-d2946d2b8d2e
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
btrfs: space cache generation (539645) does not match inode (539641)
failed to load free space cache for block group 22020096
btrfs: space cache generation (539645) does not match inode (539641)
failed to load free space cache for block group 1095761920
btrfs: space cache generation (539643) does not match inode (539640)
failed to load free space cache for block group 102161711104
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs done with fs roots in lowmem mode, skipping
[7/7] checking quota groups skipped (not enabled on this FS)
found 53501751296 bytes used, no error found
total csum bytes: 43476196
total tree bytes: 1552203776
total fs tree bytes: 1422196736
total extent tree bytes: 70172672
btree space waste bytes: 276902557
file data blocks allocated: 331882188800
 referenced 105424904192

On Fri, 17 Jan 2020 at 09:34, Nikolay Borisov <nborisov@suse.com> wrote:
>
>
>
> On 16.01.20 г. 18:53 ч., Peter Luladjiev wrote:
> > Hello, thanks for helping, here is the output:
> >
> > btrfs check --force /dev/mapper/system-root
> >
> > Opening filesystem to check...
> > WARNING: filesystem mounted, continuing because of --force
> > Checking filesystem on /dev/mapper/system-root
> > UUID: 9639a3e6-cd08-4270-b4d1-d2946d2b8d2e
> > [1/7] checking root items
> > [2/7] checking extents
> > ref mismatch on [1497018368 4096] extent item 72057183177116417, found 1
> > incorrect local backref count on 1497022464 parent 51611369472 owner 0
> > offset 0 found 1 wanted 3087007745 back 0x564582174c70
> > backpointer mismatch on [1497022464 4096]
> > ERROR: errors found in extent allocation tree or chunk allocation
> > [3/7] checking free space cache
> > [4/7] checking fs roots
> > [5/7] checking only csums items (without verifying data)
> > [6/7] checking root refs
> > [7/7] checking quota groups skipped (not enabled on this FS)
> > found 53532647424 bytes used, error(s) found
> > total csum bytes: 43476204
> > total tree bytes: 1551368192
> > total fs tree bytes: 1421295616
> > total extent tree bytes: 70238208
> > btree space waste bytes: 276638054
> > file data blocks allocated: 331679563776
> >  referenced 105423634432
> >
>
>
> Right so it seems this the only error. Just to be sure run btrfs check
> --mode lowmem  since it provides more readable output. If this is the
> only error in the filesystem then btrfs check --repair --mode lowmem
> should be able to fix it.

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
       [not found]   ` <CA+ZCqs5=N5Hdf3NxZAmPCnA8wbcJPrcH8zM-fRbt-w8tL+TjUQ@mail.gmail.com>
@ 2020-01-17  7:34     ` Nikolay Borisov
  2020-01-17  7:51       ` Peter Luladjiev
  0 siblings, 1 reply; 75+ messages in thread
From: Nikolay Borisov @ 2020-01-17  7:34 UTC (permalink / raw)
  To: Peter Luladjiev; +Cc: linux-btrfs



On 16.01.20 г. 18:53 ч., Peter Luladjiev wrote:
> Hello, thanks for helping, here is the output:
> 
> btrfs check --force /dev/mapper/system-root
> 
> Opening filesystem to check...
> WARNING: filesystem mounted, continuing because of --force
> Checking filesystem on /dev/mapper/system-root
> UUID: 9639a3e6-cd08-4270-b4d1-d2946d2b8d2e
> [1/7] checking root items
> [2/7] checking extents
> ref mismatch on [1497018368 4096] extent item 72057183177116417, found 1
> incorrect local backref count on 1497022464 parent 51611369472 owner 0
> offset 0 found 1 wanted 3087007745 back 0x564582174c70
> backpointer mismatch on [1497022464 4096]
> ERROR: errors found in extent allocation tree or chunk allocation
> [3/7] checking free space cache
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 53532647424 bytes used, error(s) found
> total csum bytes: 43476204
> total tree bytes: 1551368192
> total fs tree bytes: 1421295616
> total extent tree bytes: 70238208
> btree space waste bytes: 276638054
> file data blocks allocated: 331679563776
>  referenced 105423634432
> 


Right so it seems this the only error. Just to be sure run btrfs check
--mode lowmem  since it provides more readable output. If this is the
only error in the filesystem then btrfs check --repair --mode lowmem
should be able to fix it.

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: read time tree block corruption detected
  2020-01-16 13:40 Peter Luladjiev
@ 2020-01-16 16:12 ` Nikolay Borisov
       [not found]   ` <CA+ZCqs5=N5Hdf3NxZAmPCnA8wbcJPrcH8zM-fRbt-w8tL+TjUQ@mail.gmail.com>
  0 siblings, 1 reply; 75+ messages in thread
From: Nikolay Borisov @ 2020-01-16 16:12 UTC (permalink / raw)
  To: Peter Luladjiev, linux-btrfs



On 16.01.20 г. 15:40 ч., Peter Luladjiev wrote:
> Hello,
> 
> I'm having a "read time tree block corruption detected" error after
> upgrading to kernel 5.4, reverting to kernel 5.3 fixes the problem.
> I'm using openSuse Tumbleweed, btrfs-progs v5.4

The reason is that the read-time validator got added in v5.4. 
Looking at the provided dump: 


item 70 key (1497006080 EXTENT_ITEM 4096) itemoff 11757 itemsize 37
		refs 1 gen 17889 flags DATA
		shared data backref parent 51611369472 count 1
	item 71 key (1497010176 EXTENT_ITEM 4096) itemoff 11720 itemsize 37
		refs 1 gen 17890 flags DATA
		shared data backref parent 51611369472 count 1
	item 72 key (1497014272 EXTENT_ITEM 4096) itemoff 11683 itemsize 37
		refs 1 gen 17892 flags DATA
		shared data backref parent 51611369472 count 1
	item 73 key (1497018368 EXTENT_ITEM 4096) itemoff 11646 itemsize 37
		refs 72057183177116417 gen 17894 flags DATA
		shared data backref parent 51611369472 count 1
	item 74 key (1497022464 EXTENT_ITEM 4096) itemoff 11609 itemsize 37
		refs 1 gen 17896 flags DATA
		shared data backref parent 51611369472 count 3087007745


Clearly the problematic extent has an abnormal count: 

3087007745, in binary this makes: 
10111000000000000000000000000001

so the topmost 5 bits (apart from 1) has been set to 1 whereas 
I'd assume  only the LSB (1) should be set, judging based on the the rest
of the extents that have 51611369472 as their parent. 

Can you run btrfs check on /dev/mapper/system-root and post the log here. 
Note this will be a read-only operation to you don't risk breaking
 anything. You might also have to set the --force option since this looks 
like to be your root fs which is always mounted (barring working from a rescue
disk). 

^ permalink raw reply	[flat|nested] 75+ messages in thread

* read time tree block corruption detected
@ 2020-01-16 13:40 Peter Luladjiev
  2020-01-16 16:12 ` Nikolay Borisov
  0 siblings, 1 reply; 75+ messages in thread
From: Peter Luladjiev @ 2020-01-16 13:40 UTC (permalink / raw)
  To: linux-btrfs

[-- Attachment #1: Type: text/plain, Size: 1175 bytes --]

Hello,

I'm having a "read time tree block corruption detected" error after
upgrading to kernel 5.4, reverting to kernel 5.3 fixes the problem.
I'm using openSuse Tumbleweed, btrfs-progs v5.4

Here is the dmesg output:

[   57.817776] BTRFS critical (device dm-1): corrupt leaf:
block=52197507072 slot=74 extent bytenr=1497022464 len=4096 invalid
extent refs, have 1 expect >= inline 3087007745
[   57.817778] BTRFS error (device dm-1): block=52197507072 read time
tree block corruption detected
[   57.817783] BTRFS: error (device dm-1) in
btrfs_run_delayed_refs:2188: errno=-5 IO failure
[   57.817785] BTRFS info (device dm-1): forced readonly

btrfs fi show

Label: none  uuid: 9639a3e6-cd08-4270-b4d1-d2946d2b8d2e
        Total devices 1 FS bytes used 49.85GiB
        devid    1 size 70.00GiB used 54.04GiB path /dev/mapper/system-root

btrfs fi df /

Data, single: total=52.01GiB, used=48.40GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=2.00GiB, used=1.44GiB
GlobalReserve, single: total=123.92MiB, used=0.00B

I'm attaching the output of btrfs ins dump-tree -b 52197507072
/dev/mapper/system-root

What are my options?

Thanks in advance!

[-- Attachment #2: dump-tree.txt --]
[-- Type: text/plain, Size: 27666 bytes --]

btrfs-progs v5.4 
leaf 52197507072 items 139 free space 4636 generation 538071 owner EXTENT_TREE
leaf 52197507072 flags 0x1(WRITTEN) backref revision 1
fs uuid 9639a3e6-cd08-4270-b4d1-d2946d2b8d2e
chunk uuid 004aeeba-bb0d-4915-b7c1-c15fb4745c4f
	item 0 key (1495171072 EXTENT_ITEM 8192) itemoff 16181 itemsize 102
		refs 6 gen 500598 flags DATA
		shared data backref parent 51538460672 count 1
		shared data backref parent 988725248 count 1
		shared data backref parent 897351680 count 1
		shared data backref parent 821608448 count 1
		shared data backref parent 587972608 count 1
		shared data backref parent 508870656 count 1
	item 1 key (1495179264 EXTENT_ITEM 4096) itemoff 16092 itemsize 89
		refs 5 gen 403758 flags DATA
		shared data backref parent 52216168448 count 1
		shared data backref parent 52140294144 count 1
		shared data backref parent 51774226432 count 1
		shared data backref parent 980254720 count 1
		shared data backref parent 218169344 count 1
	item 2 key (1495183360 EXTENT_ITEM 8192) itemoff 16039 itemsize 53
		refs 1 gen 292437 flags DATA
		extent data backref root 260 objectid 698805 offset 0 count 1
	item 3 key (1495191552 EXTENT_ITEM 8192) itemoff 15921 itemsize 118
		refs 6 gen 113455 flags DATA
		extent data backref root 1338 objectid 2699311 offset 0 count 1
		shared data backref parent 52132085760 count 1
		shared data backref parent 51829489664 count 1
		shared data backref parent 51288211456 count 1
		shared data backref parent 505987072 count 1
		shared data backref parent 180977664 count 1
	item 4 key (1495199744 EXTENT_ITEM 16384) itemoff 15816 itemsize 105
		refs 5 gen 501446 flags DATA
		extent data backref root 1338 objectid 4646245 offset 0 count 1
		shared data backref parent 51833847808 count 1
		shared data backref parent 51359678464 count 1
		shared data backref parent 936361984 count 1
		shared data backref parent 415236096 count 1
	item 5 key (1495216128 EXTENT_ITEM 12288) itemoff 15711 itemsize 105
		refs 5 gen 501446 flags DATA
		extent data backref root 1338 objectid 4646255 offset 0 count 1
		shared data backref parent 51833798656 count 1
		shared data backref parent 51359760384 count 1
		shared data backref parent 936394752 count 1
		shared data backref parent 415219712 count 1
	item 6 key (1495228416 EXTENT_ITEM 8192) itemoff 15658 itemsize 53
		refs 1 gen 292032 flags DATA
		extent data backref root 258 objectid 1754582 offset 0 count 1
	item 7 key (1495236608 EXTENT_ITEM 4096) itemoff 15621 itemsize 37
		refs 1 gen 501446 flags DATA
		shared data backref parent 51376160768 count 1
	item 8 key (1495240704 EXTENT_ITEM 12288) itemoff 15503 itemsize 118
		refs 6 gen 113455 flags DATA
		extent data backref root 1338 objectid 2699310 offset 0 count 1
		shared data backref parent 52132085760 count 1
		shared data backref parent 51829489664 count 1
		shared data backref parent 51288211456 count 1
		shared data backref parent 505987072 count 1
		shared data backref parent 180977664 count 1
	item 9 key (1495252992 EXTENT_ITEM 20480) itemoff 15385 itemsize 118
		refs 6 gen 113455 flags DATA
		extent data backref root 1338 objectid 2699312 offset 0 count 1
		shared data backref parent 52132085760 count 1
		shared data backref parent 51829489664 count 1
		shared data backref parent 51288211456 count 1
		shared data backref parent 505987072 count 1
		shared data backref parent 180977664 count 1
	item 10 key (1495273472 EXTENT_ITEM 8192) itemoff 15267 itemsize 118
		refs 6 gen 113455 flags DATA
		extent data backref root 1338 objectid 2699313 offset 0 count 1
		shared data backref parent 52132085760 count 1
		shared data backref parent 51829489664 count 1
		shared data backref parent 51288211456 count 1
		shared data backref parent 505987072 count 1
		shared data backref parent 180977664 count 1
	item 11 key (1495281664 EXTENT_ITEM 8192) itemoff 15214 itemsize 53
		refs 1 gen 521727 flags DATA
		extent data backref root 258 objectid 2177295 offset 0 count 1
	item 12 key (1495289856 EXTENT_ITEM 77824) itemoff 15177 itemsize 37
		refs 1 gen 501446 flags DATA
		shared data backref parent 51300941824 count 1
	item 13 key (1495367680 EXTENT_ITEM 4096) itemoff 15124 itemsize 53
		refs 1 gen 288275 flags DATA
		extent data backref root 260 objectid 680046 offset 0 count 1
	item 14 key (1495371776 EXTENT_ITEM 4096) itemoff 15071 itemsize 53
		refs 1 gen 288214 flags DATA
		extent data backref root 260 objectid 679425 offset 8192 count 1
	item 15 key (1495375872 EXTENT_ITEM 24576) itemoff 15018 itemsize 53
		refs 1 gen 288035 flags DATA
		extent data backref root 260 objectid 678605 offset 0 count 1
	item 16 key (1495400448 EXTENT_ITEM 20480) itemoff 14965 itemsize 53
		refs 1 gen 288035 flags DATA
		extent data backref root 260 objectid 678606 offset 0 count 1
	item 17 key (1495420928 EXTENT_ITEM 24576) itemoff 14912 itemsize 53
		refs 1 gen 288035 flags DATA
		extent data backref root 260 objectid 678607 offset 0 count 1
	item 18 key (1495445504 EXTENT_ITEM 118784) itemoff 14859 itemsize 53
		refs 5 gen 288212 flags DATA
		extent data backref root 260 objectid 679426 offset 0 count 5
	item 19 key (1495564288 EXTENT_ITEM 65536) itemoff 14806 itemsize 53
		refs 1 gen 288212 flags DATA
		extent data backref root 260 objectid 679425 offset 0 count 1
	item 20 key (1495629824 EXTENT_ITEM 20480) itemoff 14753 itemsize 53
		refs 1 gen 288213 flags DATA
		extent data backref root 260 objectid 679460 offset 0 count 1
	item 21 key (1495650304 EXTENT_ITEM 4096) itemoff 14700 itemsize 53
		refs 1 gen 288275 flags DATA
		extent data backref root 260 objectid 680098 offset 0 count 1
	item 22 key (1495654400 EXTENT_ITEM 36864) itemoff 14663 itemsize 37
		refs 1 gen 400951 flags DATA
		shared data backref parent 51916881920 count 1
	item 23 key (1495691264 EXTENT_ITEM 12288) itemoff 14558 itemsize 105
		refs 5 gen 501446 flags DATA
		extent data backref root 1338 objectid 4646266 offset 0 count 1
		shared data backref parent 51833798656 count 1
		shared data backref parent 51359760384 count 1
		shared data backref parent 936394752 count 1
		shared data backref parent 415219712 count 1
	item 24 key (1495703552 EXTENT_ITEM 12288) itemoff 14521 itemsize 37
		refs 1 gen 360665 flags DATA
		shared data backref parent 51275005952 count 1
	item 25 key (1495715840 EXTENT_ITEM 8192) itemoff 14468 itemsize 53
		refs 1 gen 422702 flags DATA
		extent data backref root 260 objectid 919536 offset 12288 count 1
	item 26 key (1495724032 EXTENT_ITEM 4096) itemoff 14431 itemsize 37
		refs 1 gen 501446 flags DATA
		shared data backref parent 51376160768 count 1
	item 27 key (1495728128 EXTENT_ITEM 225280) itemoff 14378 itemsize 53
		refs 1 gen 108502 flags DATA
		extent data backref root 260 objectid 530211 offset 0 count 1
	item 28 key (1495953408 EXTENT_ITEM 36864) itemoff 14328 itemsize 50
		refs 2 gen 270767 flags DATA
		shared data backref parent 51854098432 count 1
		shared data backref parent 218349568 count 1
	item 29 key (1495990272 EXTENT_ITEM 8192) itemoff 14275 itemsize 53
		refs 1 gen 422705 flags DATA
		extent data backref root 260 objectid 919420 offset 0 count 1
	item 30 key (1495998464 EXTENT_ITEM 12288) itemoff 14222 itemsize 53
		refs 1 gen 472554 flags DATA
		extent data backref root 260 objectid 974375 offset 0 count 1
	item 31 key (1496010752 EXTENT_ITEM 4096) itemoff 14169 itemsize 53
		refs 1 gen 472554 flags DATA
		extent data backref root 260 objectid 974414 offset 0 count 1
	item 32 key (1496014848 EXTENT_ITEM 8192) itemoff 14116 itemsize 53
		refs 1 gen 292437 flags DATA
		extent data backref root 260 objectid 698839 offset 0 count 1
	item 33 key (1496023040 EXTENT_ITEM 24576) itemoff 14079 itemsize 37
		refs 1 gen 450713 flags DATA
		shared data backref parent 51439534080 count 1
	item 34 key (1496047616 EXTENT_ITEM 8192) itemoff 14042 itemsize 37
		refs 1 gen 450713 flags DATA
		shared data backref parent 51920568320 count 1
	item 35 key (1496055808 EXTENT_ITEM 16384) itemoff 13937 itemsize 105
		refs 5 gen 501446 flags DATA
		extent data backref root 1338 objectid 4646272 offset 0 count 1
		shared data backref parent 51833798656 count 1
		shared data backref parent 51359760384 count 1
		shared data backref parent 936394752 count 1
		shared data backref parent 415219712 count 1
	item 36 key (1496072192 EXTENT_ITEM 12288) itemoff 13835 itemsize 102
		refs 6 gen 500598 flags DATA
		shared data backref parent 51538460672 count 1
		shared data backref parent 988725248 count 1
		shared data backref parent 897351680 count 1
		shared data backref parent 821608448 count 1
		shared data backref parent 587972608 count 1
		shared data backref parent 508870656 count 1
	item 37 key (1496084480 EXTENT_ITEM 24576) itemoff 13782 itemsize 53
		refs 1 gen 131540 flags DATA
		extent data backref root 260 objectid 567818 offset 0 count 1
	item 38 key (1496109056 EXTENT_ITEM 94208) itemoff 13745 itemsize 37
		refs 1 gen 501446 flags DATA
		shared data backref parent 51300515840 count 1
	item 39 key (1496203264 EXTENT_ITEM 8192) itemoff 13708 itemsize 37
		refs 1 gen 501446 flags DATA
		shared data backref parent 222543872 count 1
	item 40 key (1496211456 EXTENT_ITEM 8192) itemoff 13655 itemsize 53
		refs 1 gen 292437 flags DATA
		extent data backref root 260 objectid 698847 offset 0 count 1
	item 41 key (1496219648 EXTENT_ITEM 20480) itemoff 13550 itemsize 105
		refs 5 gen 501446 flags DATA
		extent data backref root 1338 objectid 4646246 offset 0 count 1
		shared data backref parent 51833847808 count 1
		shared data backref parent 51359678464 count 1
		shared data backref parent 936361984 count 1
		shared data backref parent 415236096 count 1
	item 42 key (1496240128 EXTENT_ITEM 4096) itemoff 13513 itemsize 37
		refs 1 gen 501446 flags DATA
		shared data backref parent 51376160768 count 1
	item 43 key (1496244224 EXTENT_ITEM 8192) itemoff 13476 itemsize 37
		refs 1 gen 392397 flags DATA
		shared data backref parent 1076494336 count 1
	item 44 key (1496252416 EXTENT_ITEM 208896) itemoff 13423 itemsize 53
		refs 1 gen 292028 flags DATA
		extent data backref root 258 objectid 1752577 offset 0 count 1
	item 45 key (1496461312 EXTENT_ITEM 4096) itemoff 13370 itemsize 53
		refs 1 gen 473090 flags DATA
		extent data backref root 260 objectid 984975 offset 0 count 1
	item 46 key (1496465408 EXTENT_ITEM 8192) itemoff 13317 itemsize 53
		refs 1 gen 473090 flags DATA
		extent data backref root 260 objectid 984976 offset 0 count 1
	item 47 key (1496473600 EXTENT_ITEM 4096) itemoff 13280 itemsize 37
		refs 1 gen 392397 flags DATA
		shared data backref parent 1077231616 count 1
	item 48 key (1496477696 EXTENT_ITEM 28672) itemoff 13175 itemsize 105
		refs 5 gen 501446 flags DATA
		extent data backref root 1338 objectid 4646256 offset 0 count 1
		shared data backref parent 51833798656 count 1
		shared data backref parent 51359760384 count 1
		shared data backref parent 936394752 count 1
		shared data backref parent 415219712 count 1
	item 49 key (1496506368 EXTENT_ITEM 8192) itemoff 13122 itemsize 53
		refs 1 gen 292437 flags DATA
		extent data backref root 260 objectid 698849 offset 0 count 1
	item 50 key (1496514560 EXTENT_ITEM 8192) itemoff 13085 itemsize 37
		refs 1 gen 435902 flags DATA
		shared data backref parent 52115750912 count 1
	item 51 key (1496522752 EXTENT_ITEM 8192) itemoff 13048 itemsize 37
		refs 1 gen 435902 flags DATA
		shared data backref parent 52115750912 count 1
	item 52 key (1496530944 EXTENT_ITEM 86016) itemoff 13011 itemsize 37
		refs 1 gen 501446 flags DATA
		shared data backref parent 51300515840 count 1
	item 53 key (1496616960 EXTENT_ITEM 8192) itemoff 12974 itemsize 37
		refs 1 gen 501446 flags DATA
		shared data backref parent 222543872 count 1
	item 54 key (1496625152 EXTENT_ITEM 12288) itemoff 12739 itemsize 235
		refs 15 gen 185300 flags DATA
		extent data backref root 1338 objectid 3452741 offset 0 count 1
		shared data backref parent 52073168896 count 1
		shared data backref parent 52042973184 count 1
		shared data backref parent 51914752000 count 1
		shared data backref parent 51798016000 count 1
		shared data backref parent 51570212864 count 1
		shared data backref parent 51487080448 count 1
		shared data backref parent 51324616704 count 1
		shared data backref parent 51299778560 count 1
		shared data backref parent 997015552 count 1
		shared data backref parent 904347648 count 1
		shared data backref parent 550174720 count 1
		shared data backref parent 388661248 count 1
		shared data backref parent 223444992 count 1
		shared data backref parent 103546880 count 1
	item 55 key (1496637440 EXTENT_ITEM 143360) itemoff 12686 itemsize 53
		refs 1 gen 527818 flags DATA
		extent data backref root 258 objectid 2242165 offset 0 count 1
	item 56 key (1496780800 EXTENT_ITEM 12288) itemoff 12633 itemsize 53
		refs 1 gen 537609 flags DATA
		extent data backref root 1338 objectid 4788704 offset 0 count 1
	item 57 key (1496793088 EXTENT_ITEM 4096) itemoff 12583 itemsize 50
		refs 2 gen 435905 flags DATA
		shared data backref parent 52144455680 count 1
		shared data backref parent 51798720512 count 1
	item 58 key (1496797184 EXTENT_ITEM 53248) itemoff 12530 itemsize 53
		refs 1 gen 292030 flags DATA
		extent data backref root 258 objectid 1753023 offset 0 count 1
	item 59 key (1496850432 EXTENT_ITEM 8192) itemoff 12477 itemsize 53
		refs 1 gen 473090 flags DATA
		extent data backref root 260 objectid 984977 offset 0 count 1
	item 60 key (1496858624 EXTENT_ITEM 53248) itemoff 12440 itemsize 37
		refs 1 gen 400951 flags DATA
		shared data backref parent 70762496 count 1
	item 61 key (1496911872 EXTENT_ITEM 8192) itemoff 12387 itemsize 53
		refs 1 gen 537609 flags DATA
		extent data backref root 1338 objectid 4788723 offset 0 count 1
	item 62 key (1496920064 EXTENT_ITEM 4096) itemoff 12350 itemsize 37
		refs 1 gen 400938 flags DATA
		shared data backref parent 51236257792 count 1
	item 63 key (1496924160 EXTENT_ITEM 8192) itemoff 12300 itemsize 50
		refs 2 gen 138497 flags DATA
		shared data backref parent 51306774528 count 1
		shared data backref parent 1033256960 count 1
	item 64 key (1496932352 EXTENT_ITEM 8192) itemoff 12247 itemsize 53
		refs 1 gen 392402 flags DATA
		extent data backref root 258 objectid 4159 offset 1048576 count 1
	item 65 key (1496940544 EXTENT_ITEM 4096) itemoff 12194 itemsize 53
		refs 1 gen 394519 flags DATA
		extent data backref root 258 objectid 46599 offset 742244352 count 1
	item 66 key (1496944640 EXTENT_ITEM 8192) itemoff 12141 itemsize 53
		refs 1 gen 423224 flags DATA
		extent data backref root 260 objectid 920886 offset 0 count 1
	item 67 key (1496952832 EXTENT_ITEM 36864) itemoff 11900 itemsize 241
		refs 13 gen 400951 flags DATA
		extent data backref root 1338 objectid 4284886 offset 0 count 1
		extent data backref root 1310 objectid 4284886 offset 0 count 1
		extent data backref root 1312 objectid 4284886 offset 0 count 1
		shared data backref parent 52220051456 count 1
		shared data backref parent 51942096896 count 1
		shared data backref parent 51859947520 count 1
		shared data backref parent 51690700800 count 1
		shared data backref parent 51331710976 count 1
		shared data backref parent 386252800 count 1
		shared data backref parent 304005120 count 1
		shared data backref parent 247447552 count 1
		shared data backref parent 71696384 count 1
		shared data backref parent 70762496 count 1
	item 68 key (1496989696 EXTENT_ITEM 12288) itemoff 11847 itemsize 53
		refs 1 gen 292549 flags DATA
		extent data backref root 260 objectid 701302 offset 0 count 1
	item 69 key (1497001984 EXTENT_ITEM 4096) itemoff 11794 itemsize 53
		refs 1 gen 292555 flags DATA
		extent data backref root 260 objectid 701342 offset 0 count 1
	item 70 key (1497006080 EXTENT_ITEM 4096) itemoff 11757 itemsize 37
		refs 1 gen 17889 flags DATA
		shared data backref parent 51611369472 count 1
	item 71 key (1497010176 EXTENT_ITEM 4096) itemoff 11720 itemsize 37
		refs 1 gen 17890 flags DATA
		shared data backref parent 51611369472 count 1
	item 72 key (1497014272 EXTENT_ITEM 4096) itemoff 11683 itemsize 37
		refs 1 gen 17892 flags DATA
		shared data backref parent 51611369472 count 1
	item 73 key (1497018368 EXTENT_ITEM 4096) itemoff 11646 itemsize 37
		refs 72057183177116417 gen 17894 flags DATA
		shared data backref parent 51611369472 count 1
	item 74 key (1497022464 EXTENT_ITEM 4096) itemoff 11609 itemsize 37
		refs 1 gen 17896 flags DATA
		shared data backref parent 51611369472 count 3087007745
	item 75 key (1497026560 EXTENT_ITEM 32768) itemoff 11572 itemsize 37
		refs 1 gen 450713 flags DATA
		shared data backref parent 51439534080 count 1
	item 76 key (1497059328 EXTENT_ITEM 4096) itemoff 11535 itemsize 37
		refs 1 gen 450713 flags DATA
		shared data backref parent 51455557632 count 1
	item 77 key (1497063424 EXTENT_ITEM 229376) itemoff 11482 itemsize 53
		refs 1 gen 58759 flags DATA
		extent data backref root 260 objectid 278424 offset 0 count 1
	item 78 key (1497292800 EXTENT_ITEM 8192) itemoff 11380 itemsize 102
		refs 6 gen 500598 flags DATA
		shared data backref parent 51538460672 count 1
		shared data backref parent 988725248 count 1
		shared data backref parent 897351680 count 1
		shared data backref parent 821608448 count 1
		shared data backref parent 587972608 count 1
		shared data backref parent 508870656 count 1
	item 79 key (1497300992 EXTENT_ITEM 4096) itemoff 11291 itemsize 89
		refs 5 gen 403758 flags DATA
		shared data backref parent 52216168448 count 1
		shared data backref parent 52140294144 count 1
		shared data backref parent 51774226432 count 1
		shared data backref parent 980254720 count 1
		shared data backref parent 218169344 count 1
	item 80 key (1497305088 EXTENT_ITEM 147456) itemoff 11238 itemsize 53
		refs 1 gen 111262 flags DATA
		extent data backref root 258 objectid 994234 offset 0 count 1
	item 81 key (1497452544 EXTENT_ITEM 61440) itemoff 11120 itemsize 118
		refs 6 gen 400951 flags DATA
		extent data backref root 1338 objectid 4284778 offset 0 count 1
		shared data backref parent 52106543104 count 1
		shared data backref parent 277561344 count 1
		shared data backref parent 252542976 count 1
		shared data backref parent 169377792 count 1
		shared data backref parent 28966912 count 1
	item 82 key (1497513984 EXTENT_ITEM 4096) itemoff 11083 itemsize 37
		refs 1 gen 501446 flags DATA
		shared data backref parent 51376160768 count 1
	item 83 key (1497518080 EXTENT_ITEM 12288) itemoff 11046 itemsize 37
		refs 1 gen 400938 flags DATA
		shared data backref parent 51233325056 count 1
	item 84 key (1497530368 EXTENT_ITEM 8192) itemoff 10993 itemsize 53
		refs 1 gen 292437 flags DATA
		extent data backref root 260 objectid 698850 offset 0 count 1
	item 85 key (1497538560 EXTENT_ITEM 24576) itemoff 10888 itemsize 105
		refs 5 gen 501446 flags DATA
		extent data backref root 1338 objectid 4646264 offset 0 count 1
		shared data backref parent 51833798656 count 1
		shared data backref parent 51359760384 count 1
		shared data backref parent 936394752 count 1
		shared data backref parent 415219712 count 1
	item 86 key (1497563136 EXTENT_ITEM 4096) itemoff 10851 itemsize 37
		refs 1 gen 501446 flags DATA
		shared data backref parent 51376160768 count 1
	item 87 key (1497567232 EXTENT_ITEM 4096) itemoff 10798 itemsize 53
		refs 1 gen 292555 flags DATA
		extent data backref root 260 objectid 701344 offset 0 count 1
	item 88 key (1497571328 EXTENT_ITEM 4096) itemoff 10761 itemsize 37
		refs 1 gen 501446 flags DATA
		shared data backref parent 51376209920 count 1
	item 89 key (1497575424 EXTENT_ITEM 16384) itemoff 10708 itemsize 53
		refs 1 gen 116513 flags DATA
		extent data backref root 258 objectid 932671 offset 4128768 count 1
	item 90 key (1497591808 EXTENT_ITEM 4096) itemoff 10655 itemsize 53
		refs 1 gen 292555 flags DATA
		extent data backref root 260 objectid 701345 offset 0 count 1
	item 91 key (1497595904 EXTENT_ITEM 4096) itemoff 10602 itemsize 53
		refs 1 gen 15796 flags DATA
		extent data backref root 260 objectid 70213 offset 0 count 1
	item 92 key (1497600000 EXTENT_ITEM 8192) itemoff 10549 itemsize 53
		refs 1 gen 15911 flags DATA
		extent data backref root 260 objectid 73333 offset 12288 count 1
	item 93 key (1497608192 EXTENT_ITEM 4096) itemoff 10496 itemsize 53
		refs 1 gen 15796 flags DATA
		extent data backref root 260 objectid 70218 offset 0 count 1
	item 94 key (1497612288 EXTENT_ITEM 12288) itemoff 10443 itemsize 53
		refs 1 gen 15911 flags DATA
		extent data backref root 260 objectid 73263 offset 139264 count 1
	item 95 key (1497624576 EXTENT_ITEM 16384) itemoff 10390 itemsize 53
		refs 1 gen 15796 flags DATA
		extent data backref root 260 objectid 70208 offset 0 count 1
	item 96 key (1497640960 EXTENT_ITEM 8192) itemoff 10337 itemsize 53
		refs 1 gen 15796 flags DATA
		extent data backref root 260 objectid 70214 offset 0 count 1
	item 97 key (1497649152 EXTENT_ITEM 8192) itemoff 10284 itemsize 53
		refs 1 gen 15796 flags DATA
		extent data backref root 260 objectid 70215 offset 0 count 1
	item 98 key (1497657344 EXTENT_ITEM 8192) itemoff 10231 itemsize 53
		refs 1 gen 15796 flags DATA
		extent data backref root 260 objectid 70216 offset 0 count 1
	item 99 key (1497665536 EXTENT_ITEM 8192) itemoff 10178 itemsize 53
		refs 1 gen 15796 flags DATA
		extent data backref root 260 objectid 70217 offset 0 count 1
	item 100 key (1497673728 EXTENT_ITEM 57344) itemoff 10125 itemsize 53
		refs 1 gen 15796 flags DATA
		extent data backref root 260 objectid 70220 offset 0 count 1
	item 101 key (1497731072 EXTENT_ITEM 8192) itemoff 10072 itemsize 53
		refs 1 gen 15796 flags DATA
		extent data backref root 260 objectid 70223 offset 0 count 1
	item 102 key (1497739264 EXTENT_ITEM 8192) itemoff 10019 itemsize 53
		refs 1 gen 15796 flags DATA
		extent data backref root 260 objectid 70224 offset 0 count 1
	item 103 key (1497747456 EXTENT_ITEM 8192) itemoff 9966 itemsize 53
		refs 1 gen 15796 flags DATA
		extent data backref root 260 objectid 70227 offset 0 count 1
	item 104 key (1497755648 EXTENT_ITEM 24576) itemoff 9913 itemsize 53
		refs 1 gen 15796 flags DATA
		extent data backref root 260 objectid 70228 offset 0 count 1
	item 105 key (1497780224 EXTENT_ITEM 24576) itemoff 9860 itemsize 53
		refs 1 gen 15796 flags DATA
		extent data backref root 260 objectid 70229 offset 0 count 1
	item 106 key (1497804800 EXTENT_ITEM 12288) itemoff 9807 itemsize 53
		refs 1 gen 15914 flags DATA
		extent data backref root 260 objectid 73718 offset 0 count 1
	item 107 key (1497817088 EXTENT_ITEM 4096) itemoff 9754 itemsize 53
		refs 1 gen 15918 flags DATA
		extent data backref root 260 objectid 73703 offset 45056 count 1
	item 108 key (1497821184 EXTENT_ITEM 8192) itemoff 9701 itemsize 53
		refs 1 gen 15796 flags DATA
		extent data backref root 260 objectid 69911 offset 12288 count 1
	item 109 key (1497829376 EXTENT_ITEM 12288) itemoff 9648 itemsize 53
		refs 1 gen 15910 flags DATA
		extent data backref root 260 objectid 73263 offset 57344 count 1
	item 110 key (1497841664 EXTENT_ITEM 12288) itemoff 9595 itemsize 53
		refs 1 gen 15911 flags DATA
		extent data backref root 260 objectid 73259 offset 0 count 1
	item 111 key (1497853952 EXTENT_ITEM 4096) itemoff 9542 itemsize 53
		refs 1 gen 15911 flags DATA
		extent data backref root 260 objectid 73259 offset 122880 count 1
	item 112 key (1497858048 EXTENT_ITEM 8192) itemoff 9489 itemsize 53
		refs 1 gen 15910 flags DATA
		extent data backref root 260 objectid 73332 offset 0 count 1
	item 113 key (1497866240 EXTENT_ITEM 8192) itemoff 9436 itemsize 53
		refs 1 gen 15910 flags DATA
		extent data backref root 260 objectid 73349 offset 0 count 1
	item 114 key (1497874432 EXTENT_ITEM 12288) itemoff 9383 itemsize 53
		refs 1 gen 15910 flags DATA
		extent data backref root 260 objectid 73350 offset 0 count 1
	item 115 key (1497886720 EXTENT_ITEM 8192) itemoff 9330 itemsize 53
		refs 1 gen 15910 flags DATA
		extent data backref root 260 objectid 73351 offset 0 count 1
	item 116 key (1497894912 EXTENT_ITEM 126976) itemoff 9277 itemsize 53
		refs 1 gen 15854 flags DATA
		extent data backref root 260 objectid 70597 offset 0 count 1
	item 117 key (1498021888 EXTENT_ITEM 24576) itemoff 9224 itemsize 53
		refs 1 gen 15911 flags DATA
		extent data backref root 260 objectid 73263 offset 106496 count 1
	item 118 key (1498046464 EXTENT_ITEM 24576) itemoff 9171 itemsize 53
		refs 1 gen 15911 flags DATA
		extent data backref root 260 objectid 73672 offset 0 count 1
	item 119 key (1498071040 EXTENT_ITEM 8192) itemoff 9118 itemsize 53
		refs 1 gen 422730 flags DATA
		extent data backref root 260 objectid 920176 offset 0 count 1
	item 120 key (1498079232 EXTENT_ITEM 131072) itemoff 9065 itemsize 53
		refs 1 gen 15910 flags DATA
		extent data backref root 260 objectid 73342 offset 0 count 1
	item 121 key (1498210304 EXTENT_ITEM 8192) itemoff 9012 itemsize 53
		refs 1 gen 15910 flags DATA
		extent data backref root 260 objectid 73352 offset 0 count 1
	item 122 key (1498218496 EXTENT_ITEM 4096) itemoff 8959 itemsize 53
		refs 1 gen 15911 flags DATA
		extent data backref root 260 objectid 73263 offset 0 count 1
	item 123 key (1498222592 EXTENT_ITEM 4096) itemoff 8906 itemsize 53
		refs 1 gen 15918 flags DATA
		extent data backref root 260 objectid 73703 offset 57344 count 1
	item 124 key (1498226688 EXTENT_ITEM 8192) itemoff 8853 itemsize 53
		refs 1 gen 15910 flags DATA
		extent data backref root 260 objectid 73273 offset 0 count 1
	item 125 key (1498234880 EXTENT_ITEM 8192) itemoff 8800 itemsize 53
		refs 1 gen 15911 flags DATA
		extent data backref root 260 objectid 73263 offset 32768 count 1
	item 126 key (1498243072 EXTENT_ITEM 8192) itemoff 8747 itemsize 53
		refs 1 gen 15916 flags DATA
		extent data backref root 260 objectid 73777 offset 12288 count 1
	item 127 key (1498251264 EXTENT_ITEM 12288) itemoff 8694 itemsize 53
		refs 1 gen 15910 flags DATA
		extent data backref root 260 objectid 73272 offset 12288 count 1
	item 128 key (1498263552 EXTENT_ITEM 12288) itemoff 8641 itemsize 53
		refs 1 gen 15914 flags DATA
		extent data backref root 260 objectid 73726 offset 0 count 1
	item 129 key (1498275840 EXTENT_ITEM 24576) itemoff 8588 itemsize 53
		refs 1 gen 15910 flags DATA
		extent data backref root 260 objectid 73272 offset 65536 count 1
	item 130 key (1498300416 EXTENT_ITEM 16384) itemoff 8535 itemsize 53
		refs 1 gen 15910 flags DATA
		extent data backref root 260 objectid 73278 offset 4096 count 1
	item 131 key (1498316800 EXTENT_ITEM 20480) itemoff 8482 itemsize 53
		refs 1 gen 15914 flags DATA
		extent data backref root 260 objectid 73697 offset 0 count 1
	item 132 key (1498337280 EXTENT_ITEM 20480) itemoff 8429 itemsize 53
		refs 1 gen 15914 flags DATA
		extent data backref root 260 objectid 73733 offset 0 count 1
	item 133 key (1498357760 EXTENT_ITEM 20480) itemoff 8376 itemsize 53
		refs 1 gen 15914 flags DATA
		extent data backref root 260 objectid 73735 offset 0 count 1
	item 134 key (1498378240 EXTENT_ITEM 12288) itemoff 8323 itemsize 53
		refs 1 gen 15918 flags DATA
		extent data backref root 260 objectid 73703 offset 0 count 1
	item 135 key (1498390528 EXTENT_ITEM 4096) itemoff 8270 itemsize 53
		refs 1 gen 15918 flags DATA
		extent data backref root 260 objectid 73703 offset 69632 count 1
	item 136 key (1498394624 EXTENT_ITEM 24576) itemoff 8217 itemsize 53
		refs 1 gen 15912 flags DATA
		extent data backref root 260 objectid 73694 offset 0 count 1
	item 137 key (1498419200 EXTENT_ITEM 28672) itemoff 8164 itemsize 53
		refs 1 gen 15912 flags DATA
		extent data backref root 260 objectid 73695 offset 0 count 1
	item 138 key (1498447872 EXTENT_ITEM 20480) itemoff 8111 itemsize 53
		refs 1 gen 15916 flags DATA
		extent data backref root 260 objectid 73764 offset 0 count 1

^ permalink raw reply	[flat|nested] 75+ messages in thread

end of thread, other threads:[~2021-11-23  5:56 UTC | newest]

Thread overview: 75+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-12-29 20:43 read time tree block corruption detected Patrick Erley
2019-12-29 22:07 ` Chris Murphy
2019-12-29 22:27   ` Patrick Erley
2019-12-29 22:32     ` Chris Murphy
2019-12-29 22:36       ` Patrick Erley
2019-12-29 23:11         ` Chris Murphy
2019-12-29 23:19           ` Patrick Erley
2019-12-29 23:24             ` Chris Murphy
2019-12-29 23:26               ` Patrick Erley
2019-12-30  0:46 ` Qu Wenruo
2019-12-30  5:36   ` Patrick Erley
2019-12-30  5:43     ` Qu Wenruo
2019-12-30  5:47       ` Patrick Erley
2019-12-30  5:50         ` Patrick Erley
2019-12-30  5:58           ` Qu Wenruo
2019-12-30  6:07             ` Patrick Erley
2019-12-30  6:09               ` Qu Wenruo
2019-12-30  8:14                 ` Patrick Erley
2019-12-30  8:54                   ` Qu Wenruo
2019-12-30  9:01                     ` Patrick Erley
2019-12-30  9:09                       ` Qu Wenruo
2019-12-30  9:21                         ` Patrick Erley
2019-12-30  9:27                           ` Qu Wenruo
2019-12-30 10:06                             ` Patrick Erley
2020-01-16 13:40 Peter Luladjiev
2020-01-16 16:12 ` Nikolay Borisov
     [not found]   ` <CA+ZCqs5=N5Hdf3NxZAmPCnA8wbcJPrcH8zM-fRbt-w8tL+TjUQ@mail.gmail.com>
2020-01-17  7:34     ` Nikolay Borisov
2020-01-17  7:51       ` Peter Luladjiev
2020-01-17  7:54         ` Peter Luladjiev
2020-01-17  7:59           ` Qu Wenruo
2020-01-17  8:14           ` Nikolay Borisov
2020-01-17  8:22             ` Peter Luladjiev
2020-01-17  9:10               ` Nikolay Borisov
2020-01-17 12:04                 ` Peter Luladjiev
2020-02-12 21:58 Samir Benmendil
2020-02-13  0:26 ` Qu Wenruo
2020-02-13 13:04   ` Samir Benmendil
2020-02-13 14:01   ` Qu Wenruo
2020-02-15 15:34     ` Samir Benmendil
     [not found] <CAJheHN0FUe-ijMco1ZOc6iKF2zbPocOw+iiVNeTT1r-JuXOJww@mail.gmail.com>
2020-05-06 21:54 ` Fwd: Read " Tyler Richmond
2020-05-06 23:55   ` Chris Murphy
2020-05-07  0:51     ` Tyler Richmond
2020-05-07  1:06       ` Chris Murphy
2021-04-16 19:35 read " Gervais, Francois
2021-04-17  1:01 ` Qu Wenruo
2021-04-19 13:20   ` Gervais, Francois
2021-04-19 13:33     ` Qu Wenruo
2021-04-19 14:56       ` Gervais, Francois
2021-04-20  1:34         ` Qu Wenruo
2021-04-20 14:19           ` Gervais, Francois
2021-04-20 23:47             ` Qu Wenruo
2021-04-21 14:17               ` Gervais, Francois
2021-04-21 23:01                 ` Qu Wenruo
2021-04-22 14:26                   ` Gervais, Francois
2021-05-26 23:03                     ` Gervais, Francois
2021-05-26 23:25                       ` Qu Wenruo
2021-07-17  1:45 Read " pepperpoint
2021-07-17  7:05 ` Qu Wenruo
     [not found] ` <162650555086.7.16811903270475408953.10183708@simplelogin.co>
2021-07-17  7:51   ` pepperpoint
     [not found]   ` <162650826457.7.1050455337652772013.10184548@mb.ardentcoding.com>
2021-07-17  8:14     ` Qu Wenruo
     [not found]     ` <162650966150.7.11743767259405124657.10185986@simplelogin.co>
2021-07-17  8:57       ` pepperpoint
2021-07-17 10:12         ` Qu Wenruo
     [not found]         ` <162651674065.6.7912816287233908759.10188327@simplelogin.co>
2021-07-17 10:34           ` pepperpoint
     [not found]           ` <162651809235.7.7061042874033963922.10188873@mb.ardentcoding.com>
2021-07-17 10:48             ` Qu Wenruo
     [not found]             ` <162651892663.6.17938009695497100557.10189169@simplelogin.co>
2021-07-17 12:51               ` pepperpoint
     [not found]               ` <CQeY09U34m7SrT6nTAlkSQrbLJtmyKF1tDfuGDtKUkwJqujMI_nZU4MpGiU4F_Q1U3Lk1aWD1mFCT5SlsOsOcILWECflIw7EhVQTQpy_1Ts=@email.ardentcoding.com>
2021-07-18  5:26                 ` pepperpoint
2021-07-18  7:15                   ` Qu Wenruo
     [not found]                     ` <162659327011.8.12718249358254503695.10218325@simplelogin.co>
2021-07-18  8:46                       ` pepperpoint
     [not found]                       ` <162659797656.6.14385932343235367381.10220447@mb.ardentcoding.com>
2021-07-18  9:32                         ` Qu Wenruo
     [not found]                         ` <162660074747.7.3300740266059717894.10221270@simplelogin.co>
2021-07-18 10:34                           ` pepperpoint
     [not found]                           ` <162660447733.8.9782212603273165785.10222524@mb.ardentcoding.com>
2021-07-18 11:13                             ` Qu Wenruo
     [not found]                             ` <162660684635.8.12423097770824212671.10223516@simplelogin.co>
2021-07-18 12:16                               ` pepperpoint
2021-11-22  5:26 read " x8062
2021-11-22  7:24 ` Nikolay Borisov
2021-11-22 10:07   ` x8062
2021-11-22 10:36     ` Nikolay Borisov
2021-11-23  2:42       ` x8062
2021-11-23  5:56         ` Nikolay Borisov

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).