From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63C2BC6FA83 for ; Fri, 2 Sep 2022 18:52:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230049AbiIBSw3 (ORCPT ); Fri, 2 Sep 2022 14:52:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40802 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229968AbiIBSw2 (ORCPT ); Fri, 2 Sep 2022 14:52:28 -0400 Received: from mail-io1-xd32.google.com (mail-io1-xd32.google.com [IPv6:2607:f8b0:4864:20::d32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 28EF3E1ABB for ; Fri, 2 Sep 2022 11:52:26 -0700 (PDT) Received: by mail-io1-xd32.google.com with SMTP id p187so2369607iod.8 for ; Fri, 02 Sep 2022 11:52:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date; bh=9iGINOu3ArsvLN2VwRpyclb8al5c4172ORJdrqAFid4=; b=c/ke+nkzJ5gEsFRFAkww5/w6/Hb2M7pjmol9j7HdlWZqUg9XEJfGhexm+jsPbJQhYh HCpF7V0qp8gdYiwvRGZuXBxyr3tAtWNgX5aP9gqy3fUxfOpZHansHQkR5mmwkjJA6KXv KLM3EIbhaaPuRS7U7L2S6P2TloZIiB5hTomeQvWXZrKenOVWDA9CQfd+XZCLmjk+SrGs KzqlFIkGSFL4J/DwZUuvrxErCAbO3pOEOVW7Bl2qr18CmoGb71uBaxZb1iK9svAsQ4mo DNSCGSmDTxzWw9WlgCikyFBqpIbMxK5KmdcGVtwzvnmkQwCtZ7KgjUOP86OKWQLNkd+a 4QwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date; bh=9iGINOu3ArsvLN2VwRpyclb8al5c4172ORJdrqAFid4=; b=TWesGaGq+wMS6dQcMpGmzWcrikFasHNzK/8f/GPpFwi/5j387oHDtBL2d+bPUlgkN+ XOoitI9PXzF4tn8oWlMpYGEz/3khG70V/ORAYftt9+iVYaK+a4sAKQVyFMxwP917KBVH QXliOnhfKPfeAZH41PY11wGlsosNgDmD0Z+AW/c0iZkSvpK1gioG1iaAtpl2WxUEjfkj Q3x2owt196IWptctuMTw9o1/rHr7t01m395Y29Mqlqb4muUZqTWZ41z8yYxoNBV1HRrN 22juxwfokIMYSUd+ff6geGbtJA1+vwAtSbMMb4UnU1rJmG/nbRTO6gIYr2qMfcgVxdmH UBKg== X-Gm-Message-State: ACgBeo3Gm2OI5G98StZ21Vy1kQdwgazvIhvp7YhRFrc/gTC7wa/EaM3J pMkINDWFrIJEUjXSLs6imQu+FSC52n7ncWJbn7ZSjGA= X-Google-Smtp-Source: AA6agR4qoBijhLdYJJqQpYkRQ1JpaJ0JW1GyImItmDXBKqUHpfLbzLzwZ4qit27UbLIf0TSSTHaAtU+FDQlnwHGvg6g= X-Received: by 2002:a05:6602:2c02:b0:690:b560:7fae with SMTP id w2-20020a0566022c0200b00690b5607faemr2946368iov.169.1662144746191; Fri, 02 Sep 2022 11:52:26 -0700 (PDT) MIME-Version: 1.0 References: <70e2ae22-bbba-77a4-c9bc-4c02752f4cb7@youngman.org.uk> <4a414fc6-2666-302f-8d3d-08eb7a2986fc@turmel.org> <25355.47062.897268.3355@quad.stoffel.home> <25355.50871.743993.605394@quad.stoffel.home> <25357.13191.843087.630097@quad.stoffel.home> <1d978f6c-e1cc-e928-efc5-11ff167938b1@eyal.emu.id.au> <8e994200-146e-61ce-bb4a-f7f111f47b10@youngman.org.uk> <25359.50842.604856.467479@quad.stoffel.home> In-Reply-To: From: Peter Sanders Date: Fri, 2 Sep 2022 14:52:15 -0400 Message-ID: Subject: Re: RAID 6, 6 device array - all devices lost superblock To: John Stoffel Cc: Wols Lists , Eyal Lebedinsky , linux-raid@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org Question on restarting from scratch... How to reset to the starting point? dmsetup, both for remove and create of the overlay seems to be hanging. On Fri, Sep 2, 2022 at 10:56 AM Peter Sanders wrote: > > contents of /proc/mdstat > > root@superior:/mnt/backup# cat /proc/mdstat > Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] > [raid4] [raid10] > unused devices: > root@superior:/mnt/backup# > > > > Here are the steps I ran (minus some mounting other devices and > looking around for mdadm tracks on the old os disk) > > 410 DEVICES=3D$(cat /proc/partitions | parallel --tagstring {5} > --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' > echo /dev/{1}) > 411 apt install parallel > 412 DEVICES=3D$(cat /proc/partitions | parallel --tagstring {5} > --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' > echo /dev/{1}) > 413 echo $DEVICES > 414 cat /proc/partitions > 415 DEVICES=3D/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg > 416 DEVICES=3D"/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg" > 417 echo $DEVICES > 418 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 > {#}' ::: $DEVICES > 419 ls /dev/loop* > 420 dc > 421 cd /mnt/backup/ > 422 ls > 423 parallel truncate -s300G overlay-{/} ::: $DEVICES > 424 ls > 425 ls -la > 426 df -h > 427 parallel 'size=3D$(blockdev --getsize {}); loop=3D$(losetup -f > --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup > create {/}' ::: $DEVICES > 428 ls /dev/mapper/ > 429 OVERLAYS=3D$(parallel echo /dev/mapper/{/} ::: $DEVICES) > 430 echo $OVERLAYS > 431 dmsetup status > 432 mdadm --assemble --force /dev/md1 $OVERLAYS > 433 history > 434 dmsetup status > 435 echo $OVERLAYS > 436 mdadm --assemble --force /dev/md0 $OVERLAYS > 437 cat /proc/partitions > 438 mkdir /mnt/oldroot > << look for inird mdadm files >> > 484 echo $OVERLAYS > 485 mdadm --create /dev/md0 --level=3Draid6 -n 6 /dev/mapper/sdb > /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf > /dev/mapper/sdg > << cancelled out of 485, review instructions... >> > 486 mdadm --create /dev/md0 --level=3Draid6 -n 6 /dev/mapper/sdb > /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf > /dev/mapper/sdg > 487 fsck -n /dev/md0 > 488 mdadm --stop /dev/md0 > 489 echo $DEVICES > 490 parallel 'dmsetup remove {/}; rm overlay-{/}' ::: $DEVICES > 491 dmsetup status > 492 ls > 493 rm overlay-* > 494 ls > 495 parallel losetup -d ::: /dev/loop[0-9]* > 496 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 > {#}' ::: $DEVICES > 497 parallel truncate -s300G overlay-{/} ::: $DEVICES > 498 parallel 'size=3D$(blockdev --getsize {}); loop=3D$(losetup -f > --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup > create {/}' ::: $DEVICES > 499 dmsetup status > 500 /sbin/reboot > 501 history > 502 dmsetup status > 503 mount > 504 cat /proc/partitions > 505 nano /etc/fstab > 506 mount /mnt/backup/ > 507 ls /mnt/backup/ > 508 rm /mnt/backup/ > 509 rm /mnt/backup/overlay-sd* > 510 emacs setupOverlay & > 511 ps auxww | grep emacs > 512 kill 65017 > 513 ls /dev/loo* > 514 DEVICES=3D'/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg' > 515 echo $DEVICES > 516 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b > 7 {#}' ::: $DEVICES > 517 ls /dev/loo* > 518 parallel truncate -s4000G overlay-{/} ::: $DEVICES > 519 ls > 520 rm overlay-sd* > 521 cd /mnt/bak > 522 cd /mnt/backup/ > 523 ls > 524 parallel truncate -s4000G overlay-{/} ::: $DEVICES > 525 ls -la > 526 blockdev --getsize /dev/sdb > 527 man losetup > 528 man losetup > 529 parallel 'size=3D$(blockdev --getsize {}); loop=3D$(losetup -f > --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup > create {/}' ::: $DEVICES > 530 dmsetup status > 531 history | grep mdadm > 532 history > 533 dmsetup status > 534 history | grep dmsetup > 535 dmsetup status > 536 dmsetup remove sdg > 537 dmsetup ls --tree > 538 lsof > 539 dmsetup ls --tre > 540 dmsetup ls --tree > 541 lsof | grep -i sdg > 542 lsof | grep -i sdf > 543 history |grep dmsetup | less > 544 dmsetup status > 545 history > ~plsander/Documents/raidIssues/joblog > > On Wed, Aug 31, 2022 at 4:37 PM John Stoffel wrote: > > > > >>>>> "Peter" =3D=3D Peter Sanders writes: > > > > > encountering a puzzling situation. > > > dmsetup is failing to return. > > > > I don't think you need to use dmsetup in your case, but can you post > > *all* the commands you ran before you got to this point, and the > > output of > > > > cat /proc/mdstat > > > > as well? Thinking on this some more, you might need to actually also > > add: > > > > --assume-clean > > > > to the 'mdadm create ....' string, since you don't want it to zero the > > array or anything. > > > > Sorry for not remembering this at the time! > > > > So if you can, please just start over from scratch, showing the setup > > of the loop devices, the overlayfs setup, and the building the RAID6 > > array, along with the cat /proc/mdstat after you do the initial build. > > > > John > > > > P.S. For those who hated my email citing tool, I pulled it out for > > now. Only citing with > now. :-) > > > > > root@superior:/mnt/backup# dmsetup status > > > sdg: 0 5860533168 snapshot 16/8388608000 16 > > > sdf: 0 5860533168 snapshot 16/8388608000 16 > > > sde: 0 5860533168 snapshot 16/8388608000 16 > > > sdd: 0 5860533168 snapshot 16/8388608000 16 > > > sdc: 0 5860533168 snapshot 16/8388608000 16 > > > sdb: 0 5860533168 snapshot 16/8388608000 16 > > > > > dmsetup remove sdg runs for hours. > > > Canceled it, ran dmsetup ls --tree and find that sdg is not present i= n the list. > > > > > dmsetup status shows: > > > sdf: 0 5860533168 snapshot 16/8388608000 16 > > > sde: 0 5860533168 snapshot 16/8388608000 16 > > > sdd: 0 5860533168 snapshot 16/8388608000 16 > > > sdc: 0 5860533168 snapshot 16/8388608000 16 > > > sdb: 0 5860533168 snapshot 16/8388608000 16 > > > > > dmsetup ls --tree > > > root@superior:/mnt/backup# dmsetup ls --tree > > > sdf (253:3) > > > =E2=94=9C=E2=94=80 (7:3) > > > =E2=94=94=E2=94=80 (8:80) > > > sde (253:1) > > > =E2=94=9C=E2=94=80 (7:1) > > > =E2=94=94=E2=94=80 (8:64) > > > sdd (253:2) > > > =E2=94=9C=E2=94=80 (7:2) > > > =E2=94=94=E2=94=80 (8:48) > > > sdc (253:0) > > > =E2=94=9C=E2=94=80 (7:0) > > > =E2=94=94=E2=94=80 (8:32) > > > sdb (253:5) > > > =E2=94=9C=E2=94=80 (7:5) > > > =E2=94=94=E2=94=80 (8:16) > > > > > any suggestions? > > > > > > > > > On Tue, Aug 30, 2022 at 2:03 PM Wols Lists = wrote: > > >> > > >> On 30/08/2022 14:27, Peter Sanders wrote: > > >> > > > >> > And the victory conditions would be a mountable file system that p= asses a fsck? > > >> > > >> Yes. Just make sure you delve through the file system a bit and sati= sfy > > >> yourself it looks good, too ... > > >> > > >> Cheers, > > >> Wol