Re: Fwd: Read time tree block corruption detected

[Qu Wenruo <wqu@suse.com>]

On 2020/11/7 下午9:19, Ferry Toth wrote:
> Hi
> 
> Op 07-11-2020 om 12:35 schreef Qu Wenruo:
>>
>> On 2020/11/7 下午7:18, Ferry Toth wrote:
>>> Op 06-11-2020 om 11:32 schreef Qu Wenruo:
>>>>
>>>> On 2020/11/6 下午6:30, Ferry Toth wrote:
>>>>> Hi
>>>>>
>>>>> Op 06-11-2020 om 11:24 schreef Qu Wenruo:
>>>>>> On 2020/11/6 下午6:09, Ferry Toth wrote:
>>>>>>> Hi Qu
>>>>>>>
>>>>>>> Op 06-11-2020 om 00:40 schreef Qu Wenruo:
>>>>>>>> On 2020/11/6 上午7:37, Ferry Toth wrote:
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> Op 06-11-2020 om 00:32 schreef Qu Wenruo:
>>>>>>>>>> On 2020/11/6 上午7:12, Ferry Toth wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Op 06-11-2020 om 00:00 schreef Qu Wenruo:
>>>>>>>>>>>> On 2020/11/6 上午4:08, Ferry Toth wrote:
>>>>>>>>>>>>> I am in a similar spot, during updating my distro (Kubuntu),
>>>>>>>>>>>>> I am
>>>>>>>>>>>>> unable
>>>>>>>>>>>>> to update a certain package. I know which file it is:
>>>>>>>>>>>>>
>>>>>>>>>>>>> ~$ ls -l /usr/share/doc/libatk1.0-data
>>>>>>>>>>>>> ls: kan geen toegang krijgen tot
>>>>>>>>>>>>> '/usr/share/doc/libatk1.0-data':
>>>>>>>>>>>>> Invoer-/uitvoerfout
>>>>>>>>>>>>>
>>>>>>>>>>>>> This creates the following in journal:
>>>>>>>>>>>>>
>>>>>>>>>>>>> kernel: BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>>>>>>>>> block=1169152675840 slot=1 ino=915987, invalid inode
>>>>>>>>>>>>> generation: has
>>>>>>>>>>>>> 18446744073709551492 expect [0, 5851353]
>>>>>>>>>>>>> kernel: BTRFS error (device sda2): block=1169152675840 read
>>>>>>>>>>>>> time
>>>>>>>>>>>>> tree
>>>>>>>>>>>>> block corruption detected
>>>>>>>>>>>>>
>>>>>>>>>>>>> Now, the problem: this file is on my rootfs, which is
>>>>>>>>>>>>> mounted. apt
>>>>>>>>>>>>> (distribution updated) installed all packages but can't
>>>>>>>>>>>>> continue
>>>>>>>>>>>>> configuring, because libatk is a dependancy. I can't delete
>>>>>>>>>>>>> the
>>>>>>>>>>>>> file
>>>>>>>>>>>>> because of the I/O error. And btrfs check complains (I tried
>>>>>>>>>>>>> running RO)
>>>>>>>>>>>>> because the file system is mounted.
>>>>>>>>>>>>>
>>>>>>>>>>>>> But, on the sunny side, the file system is not RO.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is there any way to forcefully remove the file? Or do you
>>>>>>>>>>>>> have a
>>>>>>>>>>>>> recommendation how to proceed?
>>>>>>>>>>>> Newer kernel will reject to even read the item, thus will
>>>>>>>>>>>> not be
>>>>>>>>>>>> able to
>>>>>>>>>>>> remove it.
>>>>>>>>>>> That's already the case. (input / output error)
>>>>>>>>>>>> I guess you have to use some distro ISO to fix the fs.
>>>>>>>>>>> And then? btrfs check --repair the disk offline?
>>>>>>>>>> Yep.
>>>>>>>>>>
>>>>>>>>>> You would want the latest btrfs-progs though.
>>>>>>>>> Groovy has 5.7. Would that be good enough? Otherwise will be
>>>>>>>>> difficult
>>>>>>>>> to build on/for live usb image.
>>>>>>>> For your particular case, the fix are already in btrfs-progs v5.4.
>>>>>>>>
>>>>>>>> Although newer is always better, just in case you have extent item
>>>>>>>> generation corruption, you may want v5.4.1.
>>>>>>>>
>>>>>>>> So your v5.7 should be good enough.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Qu
>>>>>>> I made a live usb and performed:
>>>>>>>
>>>>>>> btrfs check --repair /dev/sda2
>>>>>>>
>>>>>>> It found errors and fixed them. However, it did not fix the corrupt
>>>>>>> leaf. The file is actually a directory:
>>>>>>>
>>>>>>> ~$ stat /usr/share/doc/libatk1.0-data
>>>>>>> stat: cannot statx '/usr/share/doc/libatk1.0-data':
>>>>>>> Invoer-/uitvoerfout
>>>>>>>
>>>>>>> in journal:
>>>>>>>
>>>>>>> BTRFS critical (device sda2): corrupt leaf: root=294
>>>>>>> block=1169152675840
>>>>>>> slot=1 ino=915987, invalid inode generation: has
>>>>>>> 18446744073709551492
>>>>>>> expect [0, 5852829]
>>>>>>> BTRFS error (device sda2): block=1169152675840 read time tree block
>>>>>>> corruption detected
>>>>>>>
>>>>>>> So how do I repair this? Am I doing something wrong?
>>>>>> Please provide the following dump:
>>>>>> btrfs ins dump-tree -b 1169152675840 /dev/sda2
>>>>>>
>>>>>> Feel free to remove the filenames in the dump.
>>>>> sudo btrfs ins dump-tree -b 1169152675840 /dev/sda2
>>>>> btrfs-progs v5.3-rc1
>>>>> leaf 1169152675840 items 36 free space 966 generation 5431733 owner
>>>>> 294
>>>>> leaf 1169152675840 flags 0x1(WRITTEN) backref revision 1
>>>>> fs uuid 27155120-9ef8-47fb-b248-eaac2b7c8375
>>>>> chunk uuid 5704f1ba-08fd-4f6b-9117-0e080b4e9ef0
>>>>>           item 0 key (915986 DIR_INDEX 2) itemoff 3957 itemsize 38
>>>>>                   location key (915987 INODE_ITEM 0) type FILE
>>>>>                   transid 7782235549259005952 data_len 0 name_len 8
>>>>>                   name: smb.conf
>>>>>           item 1 key (915987 INODE_ITEM 0) itemoff 3797 itemsize 160
>>>>>                   generation 1 transid 18446744073709551492 size 12464
>>>>> nbytes 16384
>>>> Yeah, corrupted transid.
>>>>
>>>> The v5.6 kernel doesn't get the fix backported...
>>>>
>>>> Now you have to use either the out-of-tree branch, or David's devel
>>>> branch to build a btrfs-progs which is able to repair the transid
>>>> error.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>> Just be to be clear, I tried to repair with the Kubuntu Groovy Live usb,
>>> which has linux 5.8 and btrfs-progs 5.7.
>>>
>>> I didn't fix the above transid, above was taken after booting normally
>>> again (linux 5.8), unfortunately with btrfs-progs v5.3-rc1 (that I built
>>> a year ago). See the other post for the result with btrfs-progs 5.7.
>>>
>>>
>> As I said already, you need either the devel branch to do the fix.
>> Current release btrfs-progs hasn't the repair ability merged.
>>
> Ah, I understood wrong. I thought 5.7 was enough.
> 
> So, I need to build the latest and greatest and install on live usb. Or
> I need to wait for a future live usb with this incorporated.
> 
It's strongly recommended to build right now.

As you may already find, the btrfs-progs release cycle is normally
behind schedule, thus you may need to wait way longer than you thought...

BTW, after repairing, it's recommended to use newer enought (v5.8 should
be enough) kernel to mount the repaired fs, to verify the transid error
get fixed.

Thanks,
Qu