Re: How to Fix 'Error: could not find extent items for root 257'?

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

[-- Attachment #1.1: Type: text/plain, Size: 4373 bytes --]



On 2020/2/6 下午12:13, Chiung-Ming Huang wrote:
> Qu Wenruo <quwenruo.btrfs@gmx.com> 於 2020年2月6日 週四 上午9:13寫道：
>> Please keep in mind that, if you post dmesg, the first time such error
>> happens is the most important.
>> Not something after you modified the fs by btrfs check --repair.
> 
> Thanks for your advice. I'll keep in my mind. :)
> 
> 
>>>
>>> Feb  3 15:38:24 rescue kernel: [ 8731.172674] BTRFS critical (device
>>> bcache2): corrupt leaf: root=23146 block=19498503094272 slot=8,
>>> invalid key objectid: has 18446744073709551606 expect 6 or [256,
>>> 18446744073709551360] or 18446744073709551604
>>
>> This is message is even earlier than your initial report, and it's more
>> important.
>> This means you have a bad inode item with objectid EXTENT_CSUM_OBJECTID.
>>
>> This is a bigger problem.
> 
> It sounds bad. Is it possible to save the data or part of them?

Metadata is already screwed up.

Data maybe partly saved for btrfs-restore or if you can mount it read-only.

> 
> 
>> Are you sure that is the very first error message you hit?
> 
> My .bash_history doesn't show timestamp so I'm not really sure which
> critical/error
> message is exactly right after the first `btrfs check --repair`. I
> tried to make log file
> smaller and excerpted only btrfs messages before the first critical
> message in the
> attachment. I'm not so familiar with mailing list. Could you see `btrfs_.log`?

Got the attachment.

The first strange part is, I see several mount failure with is caused by
4 or more devices missing.

Then it mounted with devid1 missing.

After reboot, you got the the full fs mounted without any missing.
So far so good, but I'm not sure how degraded mount affects here.

Soon after that, there is already problem showing some degraded mount is
causing problem, where num_devices doesn't match.

Further more, around 14:16 Feb 3, there are metadata transid mismatch,
which means some metadata is already way older.

At that point, btrfs can still try to read from the other copy, thus
it's not a big problem yet.

But that's already poisoning your fs, reducing the stability
step-by-step. It's the RAID1 of btrfs barely saved your fs.
The normal way to handle it is, trigger a full fs scrub to
resilver/resync all RAID1 copies.

And finally, you hit the last stage, where around 15:38 btrfs can't
repair the metadata mismatch caused by multiple brain-split RAID1
situation, causing tons of transid error where btrfs can't fix.


So from the full dmesg, it looks like the abuse of degraded is causing
the problem.

This shows one shortcoming of current btrfs RAID implementation, it
doesn't do automatic re-silver. Unlike mdraid which will do re-silver
before it can be accessed.
Btrfs doesn't have a record of which blocks are written before some
device go missing.

Thus degraded for btrfs should really be considered as a last-resort
method. And manual scrub after all devices go back online is really
recommended.

Thanks,
Qu
> 
> 
>>> rescue ~$ btrfs check /dev/bcache4
>>> Opening filesystem to check...
>>> Checking filesystem on /dev/bcache4
>>> UUID: 0b79cf54-c424-40ed-adca-bd66b38ad57a
>>> [1/7] checking root items
>>> Error: could not find extent items for root 257
>>> ERROR: failed to repair root items: No such file or directory
>>
>> This part is from a special repair for a regression in 3.17.
>>
>> I guess we should not enable it by default.
>> That will be another patch for btrfs-progs.
> 
> Is this patch safe for saving my btrfs? If it is, I can build btrfs-progs.

Here is the diff, should be pretty safe:
diff --git a/check/main.c b/check/main.c
index 7db65150048b..bcde157c415d 100644
--- a/check/main.c
+++ b/check/main.c
@@ -10373,7 +10373,8 @@ static int cmd_check(const struct cmd_struct
*cmd, int argc, char **argv)
                        ctx.tp = TASK_ROOT_ITEMS;
                        task_start(ctx.info, &ctx.start_time,
&ctx.item_count);
                }
-               ret = repair_root_items(info);
+               if (repair)
+                       ret = repair_root_items(info);
                task_stop(ctx.info);
                if (ret < 0) {
                        err = !!ret;


Thanks,
Qu
> 
> 
> Regards,
> Chiung-Ming Huang
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]