From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48635C433DB for ; Mon, 22 Mar 2021 03:14:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 182C361944 for ; Mon, 22 Mar 2021 03:14:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229613AbhCVDOA (ORCPT ); Sun, 21 Mar 2021 23:14:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44866 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229865AbhCVDNZ (ORCPT ); Sun, 21 Mar 2021 23:13:25 -0400 Received: from mail-qv1-xf33.google.com (mail-qv1-xf33.google.com [IPv6:2607:f8b0:4864:20::f33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AFFF2C061574 for ; Sun, 21 Mar 2021 20:13:25 -0700 (PDT) Received: by mail-qv1-xf33.google.com with SMTP id x16so8026180qvk.3 for ; Sun, 21 Mar 2021 20:13:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=BhMRC18s+X6rAfh3tD+Uc3d5M3rLDXyPbfW2VVktxvg=; b=ookW0j9yLek1SDCfi7TbPrYpyKu47LMGUVyDSeYLd9TyhwrtQJNRhNivzBvmZq2eCJ z4EiqVUQJqyjvv5/ZPJAcTfsMf7Y0Z4rLz+GC1h9zlIPHfLjJZPaPmioeSwqHMKaQ/Yf 2M8eYUM8aho8EkmRk+w5MKA8G1OMNREdugIW23g54wDyqlIE0RRo4IcBP4tiRdw8jsG4 oNXTp8FesghdYlOSgGsJE4PJHOOB9yPP+T3gdhCT1WRQdrmNwRSd1nGK5vmuM7YCuDbH cFQoge0+sTy61+btyQLRf+I1e31MVX8Czea46uOL34R5pqO5SWJzk4MwmmZWefqIO+Bw 9YNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=BhMRC18s+X6rAfh3tD+Uc3d5M3rLDXyPbfW2VVktxvg=; b=dncwcMes2/rj3UFf4njjT/+nRWsww+IjSPVq6f1yMEU7uK+yg+Gh3JNTdj/fUfQMHm Tv0PItYLKYyBh7vqba6yUdHDuLpEUOMU1ZRWHMJutp/NtLAHW6mytHAm4REf6SGuhjAO +Igx/jloP0uuDoILihJVZWToDNTxrYdCs/VqESHtPbtulMmpdHf9nSDv6oC/XmA+i3Fc RQCaUDVbbAka/ng51p4WbNacZ9X+11NVEoXJermkSL8TMALe6tPCcrOC51i3GhzKgRp4 e7jKB+5RqK1Ufa6GJHIbbnG3B7IJ7jCflvrTWjPgjGsFQdrxcDty4xnG84pXmZn49PyS 0Kow== X-Gm-Message-State: AOAM533Z9JrvABqqzEbpawJLdvv7d+rIQobL93TP/gAlxb4HQ+8Z9MYj aFr7qo/djJdenFxIQnmTnljZnk2gljoaaj8gQ9IFu0gEQg1+DpRS X-Google-Smtp-Source: ABdhPJw3YgirrNAv6XZ57dJKuobFSR4Uxg0BgiOTFvzk7pD5cFqR+f05I19TItvao2obJ6403AD3IM1Fde8jlj/Inos= X-Received: by 2002:a05:6214:1051:: with SMTP id l17mr19563263qvr.49.1616382804603; Sun, 21 Mar 2021 20:13:24 -0700 (PDT) MIME-Version: 1.0 References: <20210317012054.238334-1-davispuh@gmail.com> <20a5d997-740a-ca57-8cbc-b88c1e34c8fc@gmx.com> <01129192-1b93-2a93-2edd-f29f544fe340@gmx.com> <7db8f3ca-785b-e985-99eb-474aba82281f@gmx.com> In-Reply-To: <7db8f3ca-785b-e985-99eb-474aba82281f@gmx.com> From: =?UTF-8?B?RMSBdmlzIE1vc8SBbnM=?= Date: Mon, 22 Mar 2021 05:13:13 +0200 Message-ID: Subject: Re: [RFC] btrfs: Allow read-only mount with corrupted extent tree To: Qu Wenruo Cc: Btrfs BTRFS , clm@fb.com, Josef Bacik , dsterba@suse.com, Zygo Blaxell Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org pirmd., 2021. g. 22. marts, plkst. 02:25 =E2=80=94 lietot=C4=81js Qu Wenruo () rakst=C4=ABja: > > > > On 2021/3/22 =E4=B8=8A=E5=8D=885:54, D=C4=81vis Mos=C4=81ns wrote: > > sestd., 2021. g. 20. marts, plkst. 02:34 =E2=80=94 lietot=C4=81js Qu We= nruo > > () rakst=C4=ABja: > >> > >> > >> > >> On 2021/3/19 =E4=B8=8B=E5=8D=8811:34, D=C4=81vis Mos=C4=81ns wrote: > >>> ceturtd., 2021. g. 18. marts, plkst. 01:49 =E2=80=94 lietot=C4=81js Q= u Wenruo > >>> () rakst=C4=ABja: > >>>> > >>>> > >>>> > >>>> On 2021/3/18 =E4=B8=8A=E5=8D=885:03, D=C4=81vis Mos=C4=81ns wrote: > >>>>> tre=C5=A1d., 2021. g. 17. marts, plkst. 12:28 =E2=80=94 lietot=C4= =81js Qu Wenruo > >>>>> () rakst=C4=ABja: > >>>>>> > >>>>>> > >>>>>> > >>>>>> On 2021/3/17 =E4=B8=8A=E5=8D=889:29, D=C4=81vis Mos=C4=81ns wrote: > >>>>>>> tre=C5=A1d., 2021. g. 17. marts, plkst. 03:18 =E2=80=94 lietot=C4= =81js D=C4=81vis Mos=C4=81ns > >>>>>>> () rakst=C4=ABja: > >>>>>>>> > >>>>>>>> Currently if there's any corruption at all in extent tree > >>>>>>>> (eg. even single bit) then mounting will fail with: > >>>>>>>> "failed to read block groups: -5" (-EIO) > >>>>>>>> It happens because we immediately abort on first error when > >>>>>>>> searching in extent tree for block groups. > >>>>>>>> > >>>>>>>> Now with this patch if `ignorebadroots` option is specified > >>>>>>>> then we handle such case and continue by removing already > >>>>>>>> created block groups and creating dummy block groups. > >>>>>>>> > >>>>>>>> Signed-off-by: D=C4=81vis Mos=C4=81ns > >>>>>>>> --- > >>>>>>>> fs/btrfs/block-group.c | 14 ++++++++++++++ > >>>>>>>> fs/btrfs/disk-io.c | 4 ++-- > >>>>>>>> fs/btrfs/disk-io.h | 2 ++ > >>>>>>>> 3 files changed, 18 insertions(+), 2 deletions(-) > >>>>>>>> > >>>>>>>> diff --git a/fs/btrfs/block-group.c b/fs/btrfs/block-group.c > >>>>>>>> index 48ebc106a606..827a977614b3 100644 > >>>>>>>> --- a/fs/btrfs/block-group.c > >>>>>>>> +++ b/fs/btrfs/block-group.c > >>>>>>>> @@ -2048,6 +2048,20 @@ int btrfs_read_block_groups(struct btrfs_= fs_info *info) > >>>>>>>> ret =3D check_chunk_block_group_mappings(info); > >>>>>>>> error: > >>>>>>>> btrfs_free_path(path); > >>>>>>>> + > >>>>>>>> + if (ret =3D=3D -EIO && btrfs_test_opt(info, IGNOREBADROO= TS)) { > >>>>>>>> + btrfs_put_block_group_cache(info); > >>>>>>>> + btrfs_stop_all_workers(info); > >>>>>>>> + btrfs_free_block_groups(info); > >>>>>>>> + ret =3D btrfs_init_workqueues(info, NULL); > >>>>>>>> + if (ret) > >>>>>>>> + return ret; > >>>>>>>> + ret =3D btrfs_init_space_info(info); > >>>>>>>> + if (ret) > >>>>>>>> + return ret; > >>>>>>>> + return fill_dummy_bgs(info); > >>>>>> > >>>>>> When we hit bad things in extent tree, we should ensure we're moun= ting > >>>>>> the fs RO, or we can't continue. > >>>>>> > >>>>>> And we should also refuse to mount back to RW if we hit such case,= so > >>>>>> that we don't need anything complex, just ignore the whole extent = tree > >>>>>> and create the dummy block groups. > >>>>>> > >>>>> > >>>>> That's what we're doing here, `ignorebadroots` implies RO mount and > >>>>> without specifying it doesn't mount at all. > >>>>> > >>>>>>> > >>>>>>> This isn't that nice, but I don't really know how to properly cle= an up > >>>>>>> everything related to already created block groups so this was ea= siest > >>>>>>> way. It seems to work fine. > >>>>>>> But looks like need to do something about replay log aswell becau= se if > >>>>>>> it's not disabled then it fails with: > >>>>>>> > >>>>>>> [ 1397.246869] BTRFS info (device sde): start tree-log replay > >>>>>>> [ 1398.218685] BTRFS warning (device sde): sde checksum verify fa= iled > >>>>>>> on 21057127661568 wanted 0xd1506ed9 found 0x22ab750a level 0 > >>>>>>> [ 1398.218803] BTRFS warning (device sde): sde checksum verify fa= iled > >>>>>>> on 21057127661568 wanted 0xd1506ed9 found 0x7dd54bb9 level 0 > >>>>>>> [ 1398.218813] BTRFS: error (device sde) in __btrfs_free_extent:3= 054: > >>>>>>> errno=3D-5 IO failure > >>>>>>> [ 1398.218828] BTRFS: error (device sde) in > >>>>>>> btrfs_run_delayed_refs:2124: errno=3D-5 IO failure > >>>>>>> [ 1398.219002] BTRFS: error (device sde) in btrfs_replay_log:2254= : > >>>>>>> errno=3D-5 IO failure (Failed to recover log tree) > >>>>>>> [ 1398.229048] BTRFS error (device sde): open_ctree failed > >>>>>> > >>>>>> This is because we shouldn't allow to do anything write to the fs = if we > >>>>>> have anything wrong in extent tree. > >>>>>> > >>>>> > >>>>> This is happening when mounting read-only. My assumption is that it > >>>>> only tries to replay in memory without writing anything to disk. > >>>>> > >>>> > >>>> We lacks the check on log tree. > >>>> > >>>> Normally for such forced RO mount, log replay is not allowed. > >>>> > >>>> We should output a warning to prompt user to use nologreplay, and re= ject > >>>> the mount. > >>>> > >>> > >>> I'm not familiar with log replay but couldn't there be something > >>> useful (ignoring ref counts) that would still be worth replaying in > >>> memory? > >>> > >> Log replay means metadata write. > >> > >> Any write needs a valid extent tree to find out free space for new > >> metadata/data. > >> > >> So no, we can't do anything but completely ignoring the log. > >> > > > > I see, updated patch. But even then it seems it could be possible to > > add new ramdisk and make allocations there (eg. create new extent tree > > there) thus allowing replay. > > The problem here is, since the extent tree is corrupted, we won't know > which range has metadata already. > While metadata CoW, just like its name, needs to CoW, which means it > can't writeback (even just in memory) to anywhere we have metadata. > > The worst case is, we choose a bytenr for the new metadata to be (in > memory), but it turns out later read needs to read metadata from the > exactly same location. > The idea is if we add new disk then we would put it after last bytenr (which isn't mapped to any existing disks) thus there wouldn't be any overlap. > > BTW, I'm curious what's your test cases? As it seems you're using > log-replay but if we hit anything wrong for the replayed data, it means > btrfs kernel module has something wrong. > Did you add extra corruption for the replayed data, or it's some bug > unexposed? Basically I've a corrupted btrfs due to HBA card fault and before I nuke it I want to copy as much usable data as possible. So I was thinking if whatever is in replay log could be restored. The replay tree log itself is perfectly fine with valid checksum and there isn't any issues regarding that. I looked at it with `btrfs inspect dump-tree` and saw that there isn't anything important so it's fine ignoring it.