From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF146C433FE for ; Sun, 8 May 2022 19:46:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231694AbiEHTty (ORCPT ); Sun, 8 May 2022 15:49:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38510 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230500AbiEHTtu (ORCPT ); Sun, 8 May 2022 15:49:50 -0400 Received: from mail1.merlins.org (magic.merlins.org [209.81.13.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3484FBF42 for ; Sun, 8 May 2022 12:45:59 -0700 (PDT) Received: from c-24-5-124-255.hsd1.ca.comcast.net ([24.5.124.255]:58460 helo=sauron.svh.merlins.org) by mail1.merlins.org with esmtpsa (Cipher TLS1.3:ECDHE_SECP256R1__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.94.2 #2) id 1nnmqr-0002L1-Tw by authid with srv_auth_plain; Sun, 08 May 2022 12:45:57 -0700 Received: from merlin by sauron.svh.merlins.org with local (Exim 4.92) (envelope-from ) id 1nnmqr-00EQ9N-N0; Sun, 08 May 2022 12:45:57 -0700 Date: Sun, 8 May 2022 12:45:57 -0700 From: Marc MERLIN To: Josef Bacik Cc: linux-btrfs Subject: Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) Message-ID: <20220508194557.GP12542@merlins.org> References: <20220503172425.GA12542@merlins.org> <20220505150821.GB1020265@merlins.org> <20220506031910.GH12542@merlins.org> <20220507153921.GG1020265@merlins.org> <20220507193628.GO12542@merlins.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220507193628.GO12542@merlins.org> X-Sysadmin: BOFH X-URL: http://marc.merlins.org/ X-SA-Exim-Connect-IP: 24.5.124.255 X-SA-Exim-Mail-From: marc@merlins.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org On Sat, May 07, 2022 at 12:36:28PM -0700, Marc MERLIN wrote: > On Sat, May 07, 2022 at 02:58:38PM -0400, Josef Bacik wrote: > > Did any of the previous ones succeed? I hope so and we just have one > > misbehaving thing. I've pushed more debugging, maybe it's a large > > file that has a lot of broken extents, in either case it'll tell us > > what's going on so I can narrow down the problem. Thanks, > > Ok, the debugging helps, now I can see that it's deleting different > blocks for the same filename. > Before I couldn't quite tell if it was making progress, but now I see > it's deleting new locations. > Looks like I have 100s of deletions coming up, thanks for automating > this. > > Will report back, it looks like it may run for a few hours > > Afer that just do a > check --repair > or some other command options? Ok, so I got 33GB of output, so glad the tool is automated and the tool died eventually. I'll re-run under gdb, at least it did a lot of work. searching 165298 for bad extents processed 108707840 of 108756992 possible bytes, 99% Found an extent we don't have a block group for in the file inode ref info failed??? History/Clubbing/20220318_Pure_Markus_Schulz.mp4 Deleting [76600, 108, 111673344] root 13576824389632 path top 13576824389632 top slot 24 leaf 11821927333888 slot 52 searching 165298 for bad extents processed 108707840 of 108756992 possible bytes, 99% Found an extent we don't have a block group for in the file inode ref info failed??? History/Clubbing/20220318_Pure_Markus_Schulz.mp4 Deleting [76600, 108, 228950016] root 13576824406016 path top 13576824406016 top slot 24 leaf 11821927350272 slot 52 searching 165298 for bad extents processed 108756992 of 108756992 possible bytes, 100% searching 165299 for bad extents processed 75792384 of 75792384 possible bytes, 100% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3 Floating point exception I'll re-run and hopefully it will finish without dying or I'll get a backtrace. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08