From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D3B3C433EF for ; Wed, 1 Jun 2022 00:25:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344517AbiFAAZz (ORCPT ); Tue, 31 May 2022 20:25:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38494 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235844AbiFAAZy (ORCPT ); Tue, 31 May 2022 20:25:54 -0400 Received: from mail1.merlins.org (magic.merlins.org [209.81.13.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDF7A5BE63 for ; Tue, 31 May 2022 17:25:52 -0700 (PDT) Received: from merlin by mail1.merlins.org with local (Exim 4.94.2 #2) id 1nwCBM-0005sj-9Z by authid ; Tue, 31 May 2022 17:25:52 -0700 Date: Tue, 31 May 2022 17:25:52 -0700 From: Marc MERLIN To: Josef Bacik Cc: linux-btrfs Subject: Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) Message-ID: <20220601002552.GD22722@merlins.org> References: <20220529200415.GI24951@merlins.org> <20220530003701.GJ24951@merlins.org> <20220530191834.GK24951@merlins.org> <20220531011224.GA1745079@merlins.org> <20220531224951.GC22722@merlins.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Sysadmin: BOFH X-URL: http://marc.merlins.org/ User-Agent: Mutt/1.10.1 (2018-07-13) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: marc@merlins.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org On Tue, May 31, 2022 at 08:14:27PM -0400, Josef Bacik wrote: > Wtf, we're clearly writing the chunk root properly because I have to > re-open it to recover the tree root, and that's where it fails, but > then the chunk restore can't open the root, despite it being correctly > read in the tree recover. I've pushed new code, try tree-recover and > then recover-chunks again and capture the output please. Thanks, gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match ERROR: cannot read chunk root 21135360 WTF??? ERROR: open ctree failed, try btrfs rescue tree-recover Recover chunks tree failed gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ret is 0 offset 20971520 len 8388608 ret is -2 offset 20971520 len 8388608 checking block 22495232 generation 1572124 fs info generation 2582703 trying bytenr 22495232 got 1 blocks 0 bad checking block 22462464 generation 1479229 fs info generation 2582703 trying bytenr 22462464 got 1 blocks 0 bad checking block 22528000 generation 1572115 fs info generation 2582703 trying bytenr 22528000 got 1 blocks 0 bad checking block 22446080 generation 1571791 fs info generation 2582703 trying bytenr 22446080 got 1 blocks 0 bad checking block 22544384 generation 1556078 fs info generation 2582703 trying bytenr 22544384 got 1 blocks 0 bad checking block 22511616 generation 1555799 fs info generation 2582703 trying bytenr 22511616 got 1 blocks 0 bad checking block 22577152 generation 1586277 fs info generation 2582703 trying bytenr 22577152 got 1 blocks 0 bad checking block 22478848 generation 1561557 fs info generation 2582703 trying bytenr 22478848 got 1 blocks 0 bad checking block 22593536 generation 1590219 fs info generation 2582703 trying bytenr 22593536 got 1 blocks 0 bad checking block 22609920 generation 1551635 fs info generation 2582703 trying bytenr 22609920 got 1 blocks 0 bad checking block 22560768 generation 1590217 fs info generation 2582703 trying bytenr 22560768 got 1 blocks 0 bad No mapping for 15645202989056-15645203005440 Couldn't map the block 15645202989056 Couldn't map the block 15645202989056 No mapping for 15645202907136-15645202923520 Couldn't map the block 15645202907136 Couldn't map the block 15645202907136 No mapping for 15645202989056-15645203005440 Couldn't map the block 15645202989056 Couldn't map the block 15645202989056 No mapping for 15645202989056-15645203005440 Couldn't map the block 15645202989056 Couldn't map the block 15645202989056 No mapping for 15645202907136-15645202923520 Couldn't map the block 15645202907136 Couldn't map the block 15645202907136 none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ERROR: Couldn't find a valid root block for 1, we're going to clear it and hope for the best Tree recover failed gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match ERROR: cannot read chunk root 21135360 WTF??? ERROR: open ctree failed, try btrfs rescue tree-recover Recover chunks tree failed -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/