BTRFS Raid5 error during Scrub.

BTRFS Raid5 error during Scrub.

[Robert Krig]

Hi guys. First off, I've got backups so no worries there. I'm just
trying to understand what's happening and which files are affected.
I've got a scrub running and the kernel dmesg buffer spit out the
following:

BTRFS warning (device sda): checksum/header error at logical
48781340082176 on dev /dev/sdb, physical 5651115966464: metadata leaf
(level 0) in tree 7
BTRFS warning (device sda): checksum/header error at logical
48781340082176 on dev /dev/sdb, physical 5651115966464: metadata leaf
(level 0) in tree 7
BTRFS error (device sda): bdev /dev/sdb errs: wr 0, rd 0, flush 0,
corrupt 0, gen 1
BTRFS error (device sda): unable to fixup (regular) error at logical
48781340082176 on dev /dev/sdb
BTRFS warning (device sda): checksum/header error at logical
48781340082176 on dev /dev/sdc, physical 5651115966464: metadata leaf
(level 0) in tree 7
BTRFS warning (device sda): checksum/header error at logical
48781340082176 on dev /dev/sdc, physical 5651115966464: metadata leaf
(level 0) in tree 7
BTRFS error (device sda): bdev /dev/sdc errs: wr 0, rd 0, flush 0,
corrupt 0, gen 1
BTRFS error (device sda): unable to fixup (regular) error at logical
48781340082176 on dev /dev/sdc

Is there any way I can find out which files are affected so that I can
just restore them from a backup? 

I'm running Debian Buster with Kernel 5.2.
Btrfs-progs v4.20.1


Let me know if you need any further info.

Re: BTRFS Raid5 error during Scrub.

[Nikolay Borisov]

On 30.09.19 г. 0:38 ч., Robert Krig wrote:
> Hi guys. First off, I've got backups so no worries there. I'm just
> trying to understand what's happening and which files are affected.
> I've got a scrub running and the kernel dmesg buffer spit out the
> following:
> 
> BTRFS warning (device sda): checksum/header error at logical
> 48781340082176 on dev /dev/sdb, physical 5651115966464: metadata leaf
> (level 0) in tree 7

This indicates you have corrupted checksum tree blocks. Concretely the
checksum in the header does not match the data in the block.

> BTRFS warning (device sda): checksum/header error at logical
> 48781340082176 on dev /dev/sdb, physical 5651115966464: metadata leaf
> (level 0) in tree 7
> BTRFS error (device sda): bdev /dev/sdb errs: wr 0, rd 0, flush 0,
> corrupt 0, gen 1
> BTRFS error (device sda): unable to fixup (regular) error at logical
> 48781340082176 on dev /dev/sdb
> BTRFS warning (device sda): checksum/header error at logical
> 48781340082176 on dev /dev/sdc, physical 5651115966464: metadata leaf
> (level 0) in tree 7
> BTRFS warning (device sda): checksum/header error at logical
> 48781340082176 on dev /dev/sdc, physical 5651115966464: metadata leaf
> (level 0) in tree 7
> BTRFS error (device sda): bdev /dev/sdc errs: wr 0, rd 0, flush 0,
> corrupt 0, gen 1
> BTRFS error (device sda): unable to fixup (regular) error at logical
> 48781340082176 on dev /dev/sdc
> 
> Is there any way I can find out which files are affected so that I can
> just restore them from a backup? 
> 
> I'm running Debian Buster with Kernel 5.2.

There was a known corruption issue in 5.2 kernel up to v5.2.15. However
it could result in partial writes of transaction data whereas your
report seems to indicate data has been written corrupted on-disk.

As a first step run :

btrfs check repair --readonly

(this is a read only command so it cannot do further damage) and post
the log to see what else btrfs check finds.

> Btrfs-progs v4.20.1
> 
> 
> Let me know if you need any further info.
> 
> 

Re: BTRFS Raid5 error during Scrub.

[Robert Krig]

I've upgraded to btrfs-progs v5.2.1
Here is the output from btrfs check -p --readonly /dev/sda


Opening filesystem to check...
Checking filesystem on /dev/sda
UUID: f7573191-664f-4540-a830-71ad654d9301
[1/7] checking root items                      (0:01:17 elapsed,
5138533 items checked)
parent transid verify failed on 48781340082176 wanted 109181 found
109008items checked)
parent transid verify failed on 48781340082176 wanted 109181 found
109008
parent transid verify failed on 48781340082176 wanted 109181 found
109008
Ignoring transid failure
leaf parent key incorrect 48781340082176
bad block 48781340082176
[2/7] checking extents                         (0:03:22 elapsed,
1143429 items checked)
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache                (0:05:10 elapsed, 7236
items checked)
parent transid verify failed on 48781340082176 wanted 109181 found
109008ems checked)
Ignoring transid failure
root 15197 inode 81781 errors 1000, some csum missing48 elapsed, 33952
items checked)
[4/7] checking fs roots                        (0:42:53 elapsed, 34145
items checked)
ERROR: errors found in fs roots
found 22975533985792 bytes used, error(s) found
total csum bytes: 16806711120
total tree bytes: 18733842432
total fs tree bytes: 130121728
total extent tree bytes: 466305024
btree space waste bytes: 1100711497
file data blocks allocated: 3891333279744
 referenced 1669470507008



Am Montag, den 30.09.2019, 09:01 +0300 schrieb Nikolay Borisov:
> 
> On 30.09.19 г. 0:38 ч., Robert Krig wrote:
> > Hi guys. First off, I've got backups so no worries there. I'm just
> > trying to understand what's happening and which files are affected.
> > I've got a scrub running and the kernel dmesg buffer spit out the
> > following:
> > 
> > BTRFS warning (device sda): checksum/header error at logical
> > 48781340082176 on dev /dev/sdb, physical 5651115966464: metadata
> > leaf
> > (level 0) in tree 7
> 
> This indicates you have corrupted checksum tree blocks. Concretely
> the
> checksum in the header does not match the data in the block.
> 
> > BTRFS warning (device sda): checksum/header error at logical
> > 48781340082176 on dev /dev/sdb, physical 5651115966464: metadata
> > leaf
> > (level 0) in tree 7
> > BTRFS error (device sda): bdev /dev/sdb errs: wr 0, rd 0, flush 0,
> > corrupt 0, gen 1
> > BTRFS error (device sda): unable to fixup (regular) error at
> > logical
> > 48781340082176 on dev /dev/sdb
> > BTRFS warning (device sda): checksum/header error at logical
> > 48781340082176 on dev /dev/sdc, physical 5651115966464: metadata
> > leaf
> > (level 0) in tree 7
> > BTRFS warning (device sda): checksum/header error at logical
> > 48781340082176 on dev /dev/sdc, physical 5651115966464: metadata
> > leaf
> > (level 0) in tree 7
> > BTRFS error (device sda): bdev /dev/sdc errs: wr 0, rd 0, flush 0,
> > corrupt 0, gen 1
> > BTRFS error (device sda): unable to fixup (regular) error at
> > logical
> > 48781340082176 on dev /dev/sdc
> > 
> > Is there any way I can find out which files are affected so that I
> > can
> > just restore them from a backup? 
> > 
> > I'm running Debian Buster with Kernel 5.2.
> 
> There was a known corruption issue in 5.2 kernel up to v5.2.15.
> However
> it could result in partial writes of transaction data whereas your
> report seems to indicate data has been written corrupted on-disk.
> 
> As a first step run :
> 
> btrfs check repair --readonly
> 
> (this is a read only command so it cannot do further damage) and post
> the log to see what else btrfs check finds.
> 
> > Btrfs-progs v4.20.1
> > 
> > 
> > Let me know if you need any further info.
> > 
> > 

Re: [BTRFS Raid5 error during Scrub.

[Graham Cobb]

On 29/09/2019 22:38, Robert Krig wrote:
> I'm running Debian Buster with Kernel 5.2.
> Btrfs-progs v4.20.1

I am running Debian testing (bullseye) and have chosen not to install
the 5.2 kernel yet because the version of it in bullseye
(linux-image-5.2.0-2-amd64) is based on 5.2.9 and (as far as I can tell)
does not contain the BTRFS corruption fix.

I do not know which version of the 5.2 kernel is in buster but you may
want to check that it contains a backport of the BTRFS fix or downgrade
to the 4.19 kernel until you can be sure.

I note that linux-image-5.2.0-3-amd64 is in unstable and is based on
5.2.17 so should have the fix. I presume it will make its way to testing
soon.

If anyone can confirm which versions of the Debian kernel package the
5.2 corruption fixes are in, it would be helpful.

Re: BTRFS Raid5 error during Scrub.

[Chris Murphy]

On Mon, Sep 30, 2019 at 3:37 AM Robert Krig
<robert.krig@render-wahnsinn.de> wrote:
>
> I've upgraded to btrfs-progs v5.2.1
> Here is the output from btrfs check -p --readonly /dev/sda
>
>
> Opening filesystem to check...
> Checking filesystem on /dev/sda
> UUID: f7573191-664f-4540-a830-71ad654d9301
> [1/7] checking root items                      (0:01:17 elapsed,
> 5138533 items checked)
> parent transid verify failed on 48781340082176 wanted 109181 found
> 109008items checked)
> parent transid verify failed on 48781340082176 wanted 109181 found
> 109008
> parent transid verify failed on 48781340082176 wanted 109181 found
> 109008
> Ignoring transid failure
> leaf parent key incorrect 48781340082176
> bad block 48781340082176
> [2/7] checking extents                         (0:03:22 elapsed,
> 1143429 items checked)
> ERROR: errors found in extent allocation tree or chunk allocation
> [3/7] checking free space cache                (0:05:10 elapsed, 7236
> items checked)
> parent transid verify failed on 48781340082176 wanted 109181 found
> 109008ems checked)
> Ignoring transid failure
> root 15197 inode 81781 errors 1000, some csum missing48 elapsed, 33952
> items checked)
> [4/7] checking fs roots                        (0:42:53 elapsed, 34145
> items checked)
> ERROR: errors found in fs roots
> found 22975533985792 bytes used, error(s) found
> total csum bytes: 16806711120
> total tree bytes: 18733842432
> total fs tree bytes: 130121728
> total extent tree bytes: 466305024
> btree space waste bytes: 1100711497
> file data blocks allocated: 3891333279744
>  referenced 1669470507008


What do you get for
# btrfs insp dump-t -b 48781340082176 /dev/

It's possible there will be filenames, it's OK to sanitize them by
just deleting the names from the output before posting it.



-- 
Chris Murphy

Re: BTRFS Raid5 error during Scrub.

[Robert Krig]

Here's the output of btrfs insp dump-t -b 48781340082176 /dev/sda

Since /dev/sda is just one device from my RAID5, I'm guessing the
command doesn't need to be run separately for each device member of my
BTRFS Raid5 setup.

http://paste.debian.net/1103596/


Am Dienstag, den 01.10.2019, 12:10 -0600 schrieb Chris Murphy:
> On Mon, Sep 30, 2019 at 3:37 AM Robert Krig
> <robert.krig@render-wahnsinn.de> wrote:
> > I've upgraded to btrfs-progs v5.2.1
> > Here is the output from btrfs check -p --readonly /dev/sda
> > 
> > 
> > Opening filesystem to check...
> > Checking filesystem on /dev/sda
> > UUID: f7573191-664f-4540-a830-71ad654d9301
> > [1/7] checking root items                      (0:01:17 elapsed,
> > 5138533 items checked)
> > parent transid verify failed on 48781340082176 wanted 109181 found
> > 109008items checked)
> > parent transid verify failed on 48781340082176 wanted 109181 found
> > 109008
> > parent transid verify failed on 48781340082176 wanted 109181 found
> > 109008
> > Ignoring transid failure
> > leaf parent key incorrect 48781340082176
> > bad block 48781340082176
> > [2/7] checking extents                         (0:03:22 elapsed,
> > 1143429 items checked)
> > ERROR: errors found in extent allocation tree or chunk allocation
> > [3/7] checking free space cache                (0:05:10 elapsed,
> > 7236
> > items checked)
> > parent transid verify failed on 48781340082176 wanted 109181 found
> > 109008ems checked)
> > Ignoring transid failure
> > root 15197 inode 81781 errors 1000, some csum missing48 elapsed,
> > 33952
> > items checked)
> > [4/7] checking fs roots                        (0:42:53 elapsed,
> > 34145
> > items checked)
> > ERROR: errors found in fs roots
> > found 22975533985792 bytes used, error(s) found
> > total csum bytes: 16806711120
> > total tree bytes: 18733842432
> > total fs tree bytes: 130121728
> > total extent tree bytes: 466305024
> > btree space waste bytes: 1100711497
> > file data blocks allocated: 3891333279744
> >  referenced 1669470507008
> 
> What do you get for
> # btrfs insp dump-t -b 48781340082176 /dev/
> 
> It's possible there will be filenames, it's OK to sanitize them by
> just deleting the names from the output before posting it.
> 
> 
> 

Re: BTRFS Raid5 error during Scrub.

[Robert Krig]

By the way, how serious is the error I've encountered?
I've run a second scrub in the meantime, it aborted when it came close
to the end, just like the first time. 
If the files that are corrupt have been deleted is this error going to
go away?



On Mi, 2019-10-02 at 12:17 +0200, Robert Krig wrote:
> Here's the output of btrfs insp dump-t -b 48781340082176 /dev/sda
> 
> Since /dev/sda is just one device from my RAID5, I'm guessing the
> command doesn't need to be run separately for each device member of
> my
> BTRFS Raid5 setup.
> 
> http://paste.debian.net/1103596/
> 
> 
> Am Dienstag, den 01.10.2019, 12:10 -0600 schrieb Chris Murphy:
> > On Mon, Sep 30, 2019 at 3:37 AM Robert Krig
> > <robert.krig@render-wahnsinn.de> wrote:
> > > I've upgraded to btrfs-progs v5.2.1
> > > Here is the output from btrfs check -p --readonly /dev/sda
> > > 
> > > 
> > > Opening filesystem to check...
> > > Checking filesystem on /dev/sda
> > > UUID: f7573191-664f-4540-a830-71ad654d9301
> > > [1/7] checking root items                      (0:01:17 elapsed,
> > > 5138533 items checked)
> > > parent transid verify failed on 48781340082176 wanted 109181
> > > found
> > > 109008items checked)
> > > parent transid verify failed on 48781340082176 wanted 109181
> > > found
> > > 109008
> > > parent transid verify failed on 48781340082176 wanted 109181
> > > found
> > > 109008
> > > Ignoring transid failure
> > > leaf parent key incorrect 48781340082176
> > > bad block 48781340082176
> > > [2/7] checking extents                         (0:03:22 elapsed,
> > > 1143429 items checked)
> > > ERROR: errors found in extent allocation tree or chunk allocation
> > > [3/7] checking free space cache                (0:05:10 elapsed,
> > > 7236
> > > items checked)
> > > parent transid verify failed on 48781340082176 wanted 109181
> > > found
> > > 109008ems checked)
> > > Ignoring transid failure
> > > root 15197 inode 81781 errors 1000, some csum missing48 elapsed,
> > > 33952
> > > items checked)
> > > [4/7] checking fs roots                        (0:42:53 elapsed,
> > > 34145
> > > items checked)
> > > ERROR: errors found in fs roots
> > > found 22975533985792 bytes used, error(s) found
> > > total csum bytes: 16806711120
> > > total tree bytes: 18733842432
> > > total fs tree bytes: 130121728
> > > total extent tree bytes: 466305024
> > > btree space waste bytes: 1100711497
> > > file data blocks allocated: 3891333279744
> > >  referenced 1669470507008
> > 
> > What do you get for
> > # btrfs insp dump-t -b 48781340082176 /dev/
> > 
> > It's possible there will be filenames, it's OK to sanitize them by
> > just deleting the names from the output before posting it.
> > 
> > 
> > 

Re: BTRFS Raid5 error during Scrub.

[Chris Murphy]

On Thu, Oct 3, 2019 at 6:18 AM Robert Krig
<robert.krig@render-wahnsinn.de> wrote:
>
> By the way, how serious is the error I've encountered?
> I've run a second scrub in the meantime, it aborted when it came close
> to the end, just like the first time.
> If the files that are corrupt have been deleted is this error going to
> go away?

Maybe.


> > > > Opening filesystem to check...
> > > > Checking filesystem on /dev/sda
> > > > UUID: f7573191-664f-4540-a830-71ad654d9301
> > > > [1/7] checking root items                      (0:01:17 elapsed,
> > > > 5138533 items checked)
> > > > parent transid verify failed on 48781340082176 wanted 109181
> > > > found
> > > > 109008items checked)
> > > > parent transid verify failed on 48781340082176 wanted 109181
> > > > found
> > > > 109008
> > > > parent transid verify failed on 48781340082176 wanted 109181
> > > > found
> > > > 109008

These look suspiciously like the 5.2 regression:
https://lore.kernel.org/linux-btrfs/20190911145542.1125-1-fdmanana@kernel.org/T/#u

You should either revert to a 5.1 kernel, or use 5.2.15+.

As far as I'm aware it's not possible to fix this kind of corruption,
so I suggest refreshing your backups while you can still mount this
file system, and prepare to create it from scratch.


> > > > Ignoring transid failure
> > > > leaf parent key incorrect 48781340082176
> > > > bad block 48781340082176
> > > > [2/7] checking extents                         (0:03:22 elapsed,
> > > > 1143429 items checked)
> > > > ERROR: errors found in extent allocation tree or chunk allocation

That's usually not a good sign.



> > > > [3/7] checking free space cache                (0:05:10 elapsed,
> > > > 7236
> > > > items checked)
> > > > parent transid verify failed on 48781340082176 wanted 109181
> > > > found
> > > > 109008ems checked)
> > > > Ignoring transid failure
> > > > root 15197 inode 81781 errors 1000, some csum missing48 elapsed,

That's inode 81781 in the subvolume with ID 15197. I'm not sure what
error 1000 is, but btrfs check is a bit fussy when it enounters files
that are marked +C (nocow) but have been compressed. This used to be
possible with older kernels when nocow files were defragmented while
the file system is mounted with compression enabled. If that sounds
like your use case, that might be what's going on here, and it's
actually a benign message. It's normal for nocow files to be missing
csums. To confirm you can use 'find /pathtosubvol/ -inum 81781' to
find the file, then lsattr it and see if +C is set.

You have a few options but the first thing is to refresh backups and
prepare to lose this file system:

a. bail now, and just create a new Btrfs from scratch and restore from backup
b. try 'btrfs check --repair' to see if the transid problems are fixed; if not
c. try 'btrfs check --repair --init-extent-tree' there's a good chance
this fails and makes things worse but probably faster to try than
restoring from backup

-- 
Chris Murphy

Re: BTRFS Raid5 error during Scrub.

[Robert Krig]

Thank you all for your help so far.

I'm doing backups at the moment. My other Server is a ZFS system. I
think what I'm going to do, is to have this system, that's currently
BTRFS RAID5 migrated to using ZFS and once that's done, migrate my
backup system to BTRFS RAID5. That way I can still experiment with
BTRFS Raid5, but starting from scratch is not so problematic, since
it's a backup system and not being actively used.

I'm still going to try the steps you mentioned with btrfs check --
repair and see what that's going to spit out.


Am Donnerstag, den 03.10.2019, 14:18 -0600 schrieb Chris Murphy:
> On Thu, Oct 3, 2019 at 6:18 AM Robert Krig
> <robert.krig@render-wahnsinn.de> wrote:
> > By the way, how serious is the error I've encountered?
> > I've run a second scrub in the meantime, it aborted when it came
> > close
> > to the end, just like the first time.
> > If the files that are corrupt have been deleted is this error going
> > to
> > go away?
> 
> Maybe.
> 
> 
> > > > > Opening filesystem to check...
> > > > > Checking filesystem on /dev/sda
> > > > > UUID: f7573191-664f-4540-a830-71ad654d9301
> > > > > [1/7] checking root items                      (0:01:17
> > > > > elapsed,
> > > > > 5138533 items checked)
> > > > > parent transid verify failed on 48781340082176 wanted 109181
> > > > > found
> > > > > 109008items checked)
> > > > > parent transid verify failed on 48781340082176 wanted 109181
> > > > > found
> > > > > 109008
> > > > > parent transid verify failed on 48781340082176 wanted 109181
> > > > > found
> > > > > 109008
> 
> These look suspiciously like the 5.2 regression:
> https://lore.kernel.org/linux-btrfs/20190911145542.1125-1-fdmanana@kernel.org/T/#u
> 
> You should either revert to a 5.1 kernel, or use 5.2.15+.
> 
> As far as I'm aware it's not possible to fix this kind of corruption,
> so I suggest refreshing your backups while you can still mount this
> file system, and prepare to create it from scratch.
> 
> 
> > > > > Ignoring transid failure
> > > > > leaf parent key incorrect 48781340082176
> > > > > bad block 48781340082176
> > > > > [2/7] checking extents                         (0:03:22
> > > > > elapsed,
> > > > > 1143429 items checked)
> > > > > ERROR: errors found in extent allocation tree or chunk
> > > > > allocation
> 
> That's usually not a good sign.
> 
> 
> 
> > > > > [3/7] checking free space cache                (0:05:10
> > > > > elapsed,
> > > > > 7236
> > > > > items checked)
> > > > > parent transid verify failed on 48781340082176 wanted 109181
> > > > > found
> > > > > 109008ems checked)
> > > > > Ignoring transid failure
> > > > > root 15197 inode 81781 errors 1000, some csum missing48
> > > > > elapsed,
> 
> That's inode 81781 in the subvolume with ID 15197. I'm not sure what
> error 1000 is, but btrfs check is a bit fussy when it enounters files
> that are marked +C (nocow) but have been compressed. This used to be
> possible with older kernels when nocow files were defragmented while
> the file system is mounted with compression enabled. If that sounds
> like your use case, that might be what's going on here, and it's
> actually a benign message. It's normal for nocow files to be missing
> csums. To confirm you can use 'find /pathtosubvol/ -inum 81781' to
> find the file, then lsattr it and see if +C is set.
> 
> You have a few options but the first thing is to refresh backups and
> prepare to lose this file system:
> 
> a. bail now, and just create a new Btrfs from scratch and restore
> from backup
> b. try 'btrfs check --repair' to see if the transid problems are
> fixed; if not
> c. try 'btrfs check --repair --init-extent-tree' there's a good
> chance
> this fails and makes things worse but probably faster to try than
> restoring from backup
>