Re: btrfs errors

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

[-- Attachment #1.1: Type: text/plain, Size: 8548 bytes --]



On 2019/8/13 下午9:40, Konstantin V. Gavrilenko wrote:
> 
> Hi Qu,
> 
> thanks for the quick response. so I've booted into the latest Archiso (kernel 5.2.xx, btrfs 5.2.xx) and rerun the # btrfs check  and # btrfs scrub. The btrfs check output is rather large and is included as an attachment to this message.
> 
> It seems that the problem lies in fs_roots.
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> [4/7] checking fs roots 
> root 257 inode 6360900 errors 1000, some csum missing 
> root 258 inode 364233 errors 1040, bad file extent, some csum missing
> root 258 inode 364234 errors 1040, bad file extent, some csum missing
> ....
> root 258 inode 5074178 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 4096
> root 258 inode 5386921 errors 1040, bad file extent, some csum missing
> ...
> ERROR: errors found in fs roots
> 
> 
> Opening filesystem to check...
> Checking filesystem on /dev/kubuntu-vg/lv_root
> UUID: 7b19fb5e-4e16-4b62-b803-f59364fd61a2
> cache and super generation don't match, space cache will be invalidated
> found 335628447749 bytes used, error(s) found
> total csum bytes: 292272496
> total tree bytes: 5118279680
> total fs tree bytes: 4523163648
> total extent tree bytes: 244498432
> btree space waste bytes: 1186351226
> file data blocks allocated: 536719597568
>  referenced 630181720064

The result indeeds shows some *minor* problem.

One is "bad file extent" normally related to some extent too large for
compressed inlined extent.

Considering you're using lzo compression, it looks like some older
kernel behavior which is no longer considered sane nowadays.

You don't need to panic (never run btrfs check --repair yet), as your fs
is mostly fine, no data loss or whatever.

At least in recent kernel releases, you won't hit a problem.
> 
> 
> so I've mounted the FS and run scrub, which resulted in "ALL OK" again.
> UUID:             7b19fb5e-4e16-4b62-b803-f59364fd61a2
> Scrub started:    Tue Aug 13 13:01:26 2019
> Status:           finished
> Duration:         0:02:44
> Total to scrub:   312.59GiB
> Rate:             1.91GiB/s
> Error summary:    no errors found
> 
> 
> I have backed up all the important data on the external disc just now, and no errors in the dmesg were reported, so I assume the data is OK.
> I also have snapshots of this system stored on the external disc dating back to Apr'19.

Currently it looks like false alert.

But to be sure, please do me a favor by running lowmem mode check, which
should output more useful info other than "bad file extent".

# btrfs check --mode=lowmem <dev>

It may take a longer time to finish. But should be more useful.

> So I guess the two important questions are 
> - is it possible to reliable recover FS, or at least find out which files were affected  at the reported inode location.

If it's a false alert, you don't need to recover anything.

If it's too strict btrfs check, and you want to follow latest btrfs
*too sane* behavior, you can just try copy the old data to a newly
created btrfs.

Thanks,
Qu

> - is it possible to # btrfs check <snapshot> without copying it back on the main disk. maybe loopdevice?
> 
> 
> thanks,
> Konstantin
> 
> 
> ----- Original Message -----
> From: "Qu Wenruo" <quwenruo.btrfs@gmx.com>
> To: "Konstantin V. Gavrilenko" <k.gavrilenko@arhont.com>, "linux-btrfs" <linux-btrfs@vger.kernel.org>
> Sent: Tuesday, 13 August, 2019 12:55:47 PM
> Subject: Re: btrfs errors
> 
> 
> 
> On 2019/8/13 下午5:48, Konstantin V. Gavrilenko wrote:
>> Hi list
>>
>> I have run the btrfs check, and that reported multiple errors on the FS.
>>
>> # btrfs check --readonly --force /dev/kubuntu-vg/lv_root
>> <SKIP>
> 
> Please don't skip the output, especially for btrfs check.
> 
> The first tree btrfs check checks is extent tree, if we have anything
> wrong in extent tree, it's way serious than the later part.
> 
> And I understand you want to check your root fs, thus you have to use
> --force, but I'd recommend to go whatever distro you like, use its
> liveCD/USB to check your root fs.
> 
> It looks like that since your fs is still mounted, the data structure
> changed during the btrfs check run, it's possible to cause false alert.
> 
>> root 9214 inode 6850330 errors 2001, no inode item, link count wrong
>>         unresolved ref dir 266982 index 22459 namelen 36 name 9621041045a17a475428a26fcfb5982f.png filetype 1 errors 6, no dir index, no inode ref
>>         unresolved ref dir 226516 index 9 namelen 28 name GYTSPMxjwCVp8kXB7+j91O8kcq4= filetype 1 errors 4, no inode ref
>> root 9214 inode 6877070 errors 2001, no inode item, link count wrong
>>         unresolved ref dir 226516 index 11 namelen 28 name VSqlYzl4pFqJpvC3GA9bQ0mZK8A= filetype 1 errors 4, no inode ref
>> root 9214 inode 6878054 errors 2001, no inode item, link count wrong
>>         unresolved ref dir 266982 index 22460 namelen 36 name 52e74e9d2b6f598038486f90f8f911c4.png filetype 1 errors 4, no inode ref
>> root 9214 inode 6888414 errors 2001, no inode item, link count wrong
>>         unresolved ref dir 226391 index 122475 namelen 14 name the-real-index filetype 1 errors 4, no inode ref
>> root 9214 inode 6889202 errors 100, file extent discount
>> Found file extent holes:
>>         start: 0, len: 4096
>> root 9214 inode 6889203 errors 100, file extent discount
>> Found file extent holes:
>>         start: 0, len: 4096
>> ERROR: errors found in fs roots
>> found 334531551237 bytes used, error(s) found
>> total csum bytes: 291555464
>> total tree bytes: 1004404736
>> total fs tree bytes: 411713536
>> total extent tree bytes: 242974720
>> btree space waste bytes: 186523621
>> file data blocks allocated: 36730163200
>>  referenced 42646511616
>>
>>
>> However, scrub and badblock find no errors.
>>
>> # btrfs scrub status /mnt/
>> scrub status for 7b19fb5e-4e16-4b62-b803-f59364fd61a2
>>         scrub started at Tue Aug 13 07:31:38 2019 and finished after 00:02:47
>>         total bytes scrubbed: 311.15GiB with 0 errors
> 
> Scrub only checks checksum, doesn't care the content.
> (Kernel newer than v5.0 will care the content, but doesn't do full
> cross-check, unlike btrfs-check)
> 
>>
>> # badblocks -sv /dev/kubuntu-vg/lv_root 
>> Checking blocks 0 to 352133119
>> Checking for bad blocks (read-only test):  done                                                 
>> Pass completed, 0 bad blocks found. (0/0/0 errors)
>>
>> # btrfs dev stats /dev/kubuntu-vg/lv_root                                                                                                                                                       
>> [/dev/mapper/kubuntu--vg-lv_root].write_io_errs    0
>> [/dev/mapper/kubuntu--vg-lv_root].read_io_errs     0
>> [/dev/mapper/kubuntu--vg-lv_root].flush_io_errs    0
>> [/dev/mapper/kubuntu--vg-lv_root].corruption_errs  0
>> [/dev/mapper/kubuntu--vg-lv_root].generation_errs  0
>>
>>
>>
>> FS mount fine, and is operational.
>> would you recommend executing the btrfs check --repair option or is there something else that I can try.
> 
> Don't do anything stupid yet.
> Just go LiveCD/USB and check again.
> 
>>
>> #  uname -a                                                                                                                                                                                         Linux xps 5.1.16-050116-generic #201907031232 SMP Wed Jul 3 12:35:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> 
> Since v5.2 introduced a lot of new restrict check, I'd recommend to go
> mount with latest Archiso, btrfs-check first, if no problem, mount and
> scrub again just in case.
> 
>> # btrfs --version                                                                                                                                                                               
>> btrfs-progs v4.15.1
> 
> Big nope. You won't really want to run btrfs check --repair on such old
> and buggy progs. Unless recent releases (5.2?) btrfs-progs has a bug
> that transaction is not committed correctly, thus if something wrong
> happened like BUG_ON() or transaction aborted, the fs can easily be
> screwed up.
> 
> Thanks,
> Qu
> 
>>
>>
>> mount options
>> on / type btrfs (rw,relatime,compress-force=lzo,ssd,discard,space_cache=v2,autodefrag,subvol=/@)
>>
>> thanks,
>> Konstantin
>>
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]