linux-btrfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Alexander Wetzel <alexander.wetzel@web.de>
To: Qu Wenruo <quwenruo.btrfs@gmx.com>,
	linux-btrfs@vger.kernel.org, wqu@suse.com
Subject: Re: [BUG] BTRFS critical corrupt leaf - bisected to 496245cac57e
Date: Sun, 14 Jul 2019 14:07:35 +0200	[thread overview]
Message-ID: <e2776741-3554-e9a6-440a-4bfe8a839cbb@web.de> (raw)
In-Reply-To: <6e764f38-a8dd-19e2-e885-3d7561479681@gmx.com>

Am 14.07.19 um 11:49 schrieb Qu Wenruo:
> 
> 
> On 2019/7/14 下午5:25, Alexander Wetzel wrote:
>>
>>>>
>>>> filtering for btrfs and removing duplicate lines just shows three uniq
>>>> error messages:
>>>>    BTRFS critical (device vda3): corrupt leaf: root=300 block=8645398528
>>>> slot=4 ino=259223, invalid inode generation: has 139737289170944 expect
>>>> [0, 1425224]
>>>>    BTRFS critical (device vda3): corrupt leaf: root=300 block=8645398528
>>>> slot=4 ino=259223, invalid inode generation: has 139737289170944 expect
>>>> [0, 1425225]
>>>>    BTRFS critical (device vda3): corrupt leaf: root=300 block=8645398528
>>>> slot=4 ino=259223, invalid inode generation: has 139737289170944 expect
>>>> [0, 1425227]
>>>
>>> The generation number is 0x7f171f7ba000, I see no reason why it would
>>> make any sense.
>>>
>>> I see no problem rejecting obviously corrupted item.
>>>
>>> The problem is:
>>> - Is that corrupted item?
>>>     At least to me, it looks corrupted just from the dmesg.
>>>
>>> - How and when this happens
>>>     Obviously happened on some older kernel.
>>>     V5.2 will report such problem before writing corrupted data back to
>>>     disk, at least prevent such problem from happening.
>>
>> It's probably useless information at that point, but the FS was created
>> with a boot image from Debian 8 around Dec 1st 2016 by migrating an also
>> freshly created ext4 filesystem to btrfs.
> 
> Migrated image could has something unexpected, but according to the
> owner id, it's definitely not the converted subvolume. But newly created
> subvolume/snapshot.
> 

Yes, I'm using snapshots and I've added subvolumes after the migration. 
Basically I moved everything from root into a subvolume. But not sure 
which procedure I used back then... I think I created a snapshot of old 
root and then deleted all files/dirs in root with the exception of the 
snapshot. Now I would have said that I did that under the 4.7.10 kernel, 
but it's long ago and there may have been some complications.

>> I'm pretty sure the migration failed with the newer gentoo kernel
>> intended for operation - which was sys-kernel/hardened-sources-4.7.10 -
>> and a used the Debian boot image for that. (I can piece together all
>> kernel versions used from wtmp, but the Debian boot kernel would be
>> "guess only".)
>>
>> The time stamps like "2016-12-01 21:51:27" in the dump below are
>> matching very well to the time I was setting up the system based on the
>> few remaining log evidence I have.
> 
> I just did a quick grep and blame for inode transid related code.
> The latest direct modification to inode_transid is 6e17d30bfaf4 ("Btrfs:
> fill ->last_trans for delayed inode in btrfs_fill_inode."), which is
> upstreamed in v4.1.
> 
> Furthermore, at that time, we don't have good enough practice for
> backport, thus that commit lacks fixes tag, and won't be backported to
> most stable branches.
> I don't believe Debian backport team would pick this into their kernels,
> so if the fs is modified by kernel older than v4.1, then that may be the
> cause.

Sounds plausible, the Debian 8 boot image used for migration probably 
was using a 3.16 kernel. So this version did touch the FS and this may 
be some left overs.
Especially since I have multiple very similar VMs which are working 
fine. (The broken one is the only one hosted in a VPS, forcing me to use 
whatever the provider had at the time to jump start the system. All the 
other VMs - most of them older by some years - are working with 5.2.)

Long story short: I guess we can assume it was something I did on the FS 
while running a 3.16 kernel and the new checks are now complaining about.

> 
>>
>>> Please provide the following dump:
>>>    # btrfs ins dump-tree -b 8645398528 /dev/vda3
>>>
>>
>> xar /home/alex # btrfs ins dump-tree -b 8645398528 /dev/vda3
>> btrfs-progs v4.19
>> leaf 8645398528 items 48 free space 509 generation 1425074 owner 300
>> leaf 8645398528 flags 0x1(WRITTEN) backref revision 1
>> fs uuid 668c885e-50b9-41d0-a3ce-b653a4d3f87a
>> chunk uuid 54c6809b-e261-423f-b4a1-362304e887bd
>>          item 0 key (259222 DIR_ITEM 2504220146) itemoff 3960 itemsize 35
>>                  location key (259223 INODE_ITEM 0) type FILE
>>                  transid 8119256875011 data_len 0 name_len 5
>>                  name: .keep
> 
> If we're checking DIR_ITEM/DIR_INDEX transid, it kernel should fail even
> easier.
> 
> Those transid makes no sense at all.
> 
>>          item 1 key (259222 DIR_INDEX 2) itemoff 3925 itemsize 35
>>                  location key (259223 INODE_ITEM 0) type FILE
>>                  transid 8119256875011 data_len 0 name_len 5
>>                  name: .keep
>>          item 2 key (259222 DIR_INDEX 3) itemoff 3888 itemsize 37
>>                  location key (258830 INODE_ITEM 0) type DIR
>>                  transid 2673440063491 data_len 0 name_len 7
>>                  name: portage
>>          item 3 key (259222 DIR_INDEX 4) itemoff 3851 itemsize 37
>>                  location key (3632036 INODE_ITEM 0) type DIR
>>                  transid 169620 data_len 0 name_len 7
>>                  name: binpkgs
>>          item 4 key (259223 INODE_ITEM 0) itemoff 3691 itemsize 160
>>                  generation 1 transid 139737289170944 size 0 nbytes 0
>>                  block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
>>                  sequence 139737289225400 flags 0x0(none)
> 
> Either the reported transid makes sense.
> 
>>                  atime 1480625487.0 (2016-12-01 21:51:27)
>>                  ctime 1480625487.0 (2016-12-01 21:51:27)
>>                  mtime 1480015482.0 (2016-11-24 20:24:42)
>>                  otime 0.0 (1970-01-01 01:00:00)
>>          item 5 key (259223 INODE_REF 259222) itemoff 3676 itemsize 15
>>                  index 2 namelen 5 name: .keep
>>          item 6 key (259224 INODE_ITEM 0) itemoff 3516 itemsize 160
>>                  generation 1 transid 1733 size 4 nbytes 5
> 
> This transid shold be correct.
> 
> According to the leaf geneartion, any transid larger than 1425074 should
> be incorrect.
> 
> So, the are a lot of transid error, not limited to the reported item 4.
> There may be so many transid error that most of your tree blocks may get
> modified to update the transid.
> 
> To fix this, I believe it's possible to reset all these inodes' transid
> to leaf transid, but I'm not 100% sure if such fix won't affect things
> like send.
> 
> 
> I totally understand that the solution I'm going to provide sounds
> aweful, but I'd recommend to use a newer enough kernel but without that
> check, to copy all the data to another btrfs fs.
>  > It could be more safe than waiting for a btrfs check to repair it.

No problem for me. This report here was created for science only:-)
I just wanted to your attention prior to destroying the broken FS and 
shredding potential useful data useful to track down what went wrong.
With that now concluded I'll just do that!

But maybe one additional remark: The snapshots transferred via btrfs 
send/receive to another PC are working fine on a system using a 5.2 
kernel. Since the "moved" subvolume also does not have the block 
8645398528 I assume I don't really have to copy the files but restoring 
the subvolume with btrfs receive on a new btrfs image will also get rid 
of the errors.

Thanks for your time, the incredible fast feedback and your help,

Alexander

  reply	other threads:[~2019-07-14 15:45 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-07-13 20:48 [BUG] BTRFS critical corrupt leaf - bisected to 496245cac57e Alexander Wetzel
2019-07-14  1:30 ` Qu Wenruo
2019-07-14  9:25   ` Alexander Wetzel
2019-07-14  9:49     ` Qu Wenruo
2019-07-14 12:07       ` Alexander Wetzel [this message]
2019-07-14 12:51         ` Qu Wenruo
2019-07-14 15:40       ` Chris Murphy
2019-07-15  1:07         ` Qu Wenruo
2019-07-29 12:46 ` Swâmi Petaramesh
2019-07-29 13:33   ` Qu Wenruo
2019-07-30  4:56     ` Qu Wenruo
2019-07-30  6:44       ` Swâmi Petaramesh
2019-07-30  7:21         ` Qu Wenruo
2019-07-30  8:02           ` Swâmi Petaramesh
2019-07-30 13:57           ` Swâmi Petaramesh
2019-07-30 14:16             ` Qu Wenruo

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=e2776741-3554-e9a6-440a-4bfe8a839cbb@web.de \
    --to=alexander.wetzel@web.de \
    --cc=linux-btrfs@vger.kernel.org \
    --cc=quwenruo.btrfs@gmx.com \
    --cc=wqu@suse.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).