From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24129C433F5 for ; Wed, 27 Apr 2022 21:20:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233073AbiD0VXm (ORCPT ); Wed, 27 Apr 2022 17:23:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51226 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231338AbiD0VXi (ORCPT ); Wed, 27 Apr 2022 17:23:38 -0400 Received: from mail1.merlins.org (magic.merlins.org [209.81.13.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5EF996E8D0 for ; Wed, 27 Apr 2022 14:20:24 -0700 (PDT) Received: from c-24-5-124-255.hsd1.ca.comcast.net ([24.5.124.255]:58286 helo=sauron.svh.merlins.org) by mail1.merlins.org with esmtpsa (Cipher TLS1.3:ECDHE_SECP256R1__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.94.2 #2) id 1njp5D-0007Qt-Oq by authid with srv_auth_plain; Wed, 27 Apr 2022 14:20:23 -0700 Received: from merlin by sauron.svh.merlins.org with local (Exim 4.92) (envelope-from ) id 1njp5D-006gOs-J0; Wed, 27 Apr 2022 14:20:23 -0700 Date: Wed, 27 Apr 2022 14:20:23 -0700 From: Marc MERLIN To: Josef Bacik Cc: linux-btrfs Subject: Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) Message-ID: <20220427212023.GW12542@merlins.org> References: <20220427035451.GM29107@merlins.org> <20220427163423.GN29107@merlins.org> <20220427182440.GO12542@merlins.org> <20220427210246.GV12542@merlins.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Sysadmin: BOFH X-URL: http://marc.merlins.org/ X-SA-Exim-Connect-IP: 24.5.124.255 X-SA-Exim-Mail-From: marc@merlins.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org On Wed, Apr 27, 2022 at 05:11:30PM -0400, Josef Bacik wrote: > > inserting block group 15838291689472 > > inserting block group 15839365431296 > > inserting block group 15840439173120 > > inserting block group 15842586656768 > > processed 1556480 of 0 possible bytes > > processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0] > > doing an insert that overlaps our bytenr 7750833627136 262144 > > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths > > Failed to find [7750833868800, 168, 262144] > > > > Of course it doesn't work for you, I pushed some debug stuff. Thanks, (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". FS_INFO IS 0x55555564cbc0 JOSEF: root 9 Couldn't find the last root for 8 checksum verify failed on 58720256 wanted 0x0d38525a found 0xb3a707fa FS_INFO AFTER IS 0x55555564cbc0 Walking all our trees and pinning down the currently accessible blocks (...) inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0] elem_cnt 0 elem_missed 0 ret -2 doing an insert that overlaps our bytenr 7750833627136 262144 processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths elem_cnt 0 elem_missed 0 ret -2 Failed to find [7750833868800, 168, 262144] Program received signal SIGSEGV, Segmentation fault. rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffdce0, comp=comp@entry=0x55555559ae9a , next_ret=next_ret@entry=0x7fffffffdcf8) at common/rbtree-utils.c:48 48 struct rb_node *n = root->rb_node; (gdb) bt #0 rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffdce0, comp=comp@entry=0x55555559ae9a , next_ret=next_ret@entry=0x7fffffffdcf8) at common/rbtree-utils.c:48 #1 0x000055555559b09e in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=100564992) at common/extent-cache.c:179 #2 0x0000555555584c2c in set_extent_bits (tree=0x100000060, start=100564992, end=100564991, bits=bits@entry=1) at kernel-shared/extent_io.c:380 #3 0x0000555555584e5f in set_extent_dirty (tree=, start=, end=) at kernel-shared/extent_io.c:486 #4 0x0000555555585842 in set_extent_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/extent_io.c:976 #5 0x000055555557bc10 in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/disk-io.c:2224 #6 0x000055555557ee39 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998755744, parent=0, iref=0xffffffffffffffe3, path=0x555555b8cda0, root=) at kernel-shared/extent-tree.c:1085 #7 insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998755744, parent=0, num_bytes=, bytenr=, path=0x555555b8cda0, root=, trans=) at kernel-shared/extent-tree.c:1197 #8 btrfs_inc_extent_ref (trans=trans@entry=0x555558a8d6a0, root=root@entry=0x55555564cde0, bytenr=, num_bytes=, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0) at kernel-shared/extent-tree.c:1262 #9 0x00005555555dfd17 in process_eb (trans=trans@entry=0x555558a8d6a0, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559d00f00, current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:557 #10 0x00005555555dfe70 in process_eb (trans=trans@entry=0x555558a8d6a0, root=root@entry=0x55555564cde0, eb=0x555555944100, current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:632 #11 0x00005555555e00db in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:703 #12 0x00005555555e03e5 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe6c5 "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:839 #13 0x00005555555d7a08 in cmd_rescue_init_extent_tree (cmd=, argc=, argv=) at cmds/rescue.c:65 #14 0x000055555556c17b in cmd_execute (argv=0x7fffffffe3c8, argc=2, cmd=0x555555642d40 ) at cmds/commands.h:125 #15 handle_command_group (cmd=, argc=2, argv=0x7fffffffe3c8) at btrfs.c:152 #16 0x000055555556c275 in cmd_execute (argv=0x7fffffffe3c0, argc=3, cmd=0x555555643cc0 ) at cmds/commands.h:125 #17 main (argc=3, argv=0x7fffffffe3c0) at btrfs.c:405 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08