All of lore.kernel.org
 help / color / mirror / Atom feed
* overlays on dd images of 4T drives
@ 2021-11-14  2:29 David T-G
  2021-11-14  8:47 ` Roman Mamedov
  2021-11-15  2:51 ` David T-G
  0 siblings, 2 replies; 6+ messages in thread
From: David T-G @ 2021-11-14  2:29 UTC (permalink / raw)
  To: Linux RAID

Hi, all --

Crossing my fingers that this gets through ...  My last few messages have
not come back to me on the list.

After a first attempt to assemble the two working and one wonky drives of
my 4-disk array and then watching reads hang forever, I have now used dd
to pull an image of each RAID partition

  davidtg@gezebel:~> sudo fdisk -l /dev/sda
  Disk /dev/sda: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
  Disk model: ST4000DM000-1F21
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 4096 bytes
  I/O size (minimum/optimal): 4096 bytes / 4096 bytes
  Disklabel type: gpt
  Disk identifier: 482CA0F0-663B-409D-A4DF-4A4B248E6D23

  Device          Start        End    Sectors   Size Type
  /dev/sda1        2048 7813774990 7813772943   3.7T Linux RAID
  /dev/sda2  7813775360 7814037134     261775 127.8M Linux filesystem

  diskfarm:/mnt/10Traid50md/tmp # ls -goh 4T*
  -rw-r--r-- 1 4.0T Nov 12 02:05 '4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror'
  -rw-r--r-- 1 4.0T Nov 12 02:05 '4Tsdb1.5YD9.dd-bs=256M-conv=sparse,noerror'
  -rw-r--r-- 1 4.0T Nov 12 02:05 '4Tsdc1.5ZY3.dd-bs=256M-conv=sparse,noerror'
  diskfarm:/mnt/10Traid50md/tmp # file 4Tsda1.EYNA.dd-bs\=256M-conv\=sparse\,noerror
  4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror: Linux Software RAID version 1.2 (1) UUID=ca7008ef:90693dae:6c231ad7: 8b3f92d name=diskfarm:0 level=5 disks=4

so that I can use overlays to assemble the device and replay the XFS
journal to get up and running again to finally be able to pull off this
data.

After going back and forth across the deprecated wiki pages, I'm working
through Irreversible Failure Recovery

  https://raid.wiki.kernel.org/index.php/Irreversible_mdadm_failure_recovery

and have created

  diskfarm:/mnt/10Traid50md/tmp # ls -goh overlay-sd*
  -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sda1
  -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sdb1
  -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sdc1
  diskfarm:/mnt/10Traid50md/tmp # losetup -a
  /dev/loop1: [66326]:1059 (/mnt/10Traid50md/tmp/overlay-sdb1)
  /dev/loop2: [66326]:1060 (/mnt/10Traid50md/tmp/overlay-sdc1)
  /dev/loop0: [66326]:1058 (/mnt/10Traid50md/tmp/overlay-sda1)

my overlay files and loopback devices.  But ...

It seems that blockdev does not like

  diskfarm:/mnt/10Traid50md/tmp # blockdev --getsize ./overlay-sda1
  blockdev: ioctl error on BLKGETSIZE: Inappropriate ioctl for device

my files and so it won't give me the size that I need for dmsetup in the
next step.

How can I use these partition images as source for faking up my RAID
array to recover the contents?  I seem to have gotten myself stuck :-/


TIA & HAND

:-D
-- 
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt


^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: overlays on dd images of 4T drives
  2021-11-14  2:29 overlays on dd images of 4T drives David T-G
@ 2021-11-14  8:47 ` Roman Mamedov
  2021-11-15  2:51 ` David T-G
  1 sibling, 0 replies; 6+ messages in thread
From: Roman Mamedov @ 2021-11-14  8:47 UTC (permalink / raw)
  To: David T-G; +Cc: Linux RAID

On Sun, 14 Nov 2021 02:29:24 +0000
David T-G <davidtg+robot@justpickone.org> wrote:

>   diskfarm:/mnt/10Traid50md/tmp # ls -goh overlay-sd*
>   -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sda1
>   -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sdb1
>   -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sdc1
>   diskfarm:/mnt/10Traid50md/tmp # losetup -a
>   /dev/loop1: [66326]:1059 (/mnt/10Traid50md/tmp/overlay-sdb1)
>   /dev/loop2: [66326]:1060 (/mnt/10Traid50md/tmp/overlay-sdc1)
>   /dev/loop0: [66326]:1058 (/mnt/10Traid50md/tmp/overlay-sda1)
> 
> my overlay files and loopback devices.  But ...
> 
> It seems that blockdev does not like
> 
>   diskfarm:/mnt/10Traid50md/tmp # blockdev --getsize ./overlay-sda1
>   blockdev: ioctl error on BLKGETSIZE: Inappropriate ioctl for device
> 
> my files

It must be run on block devices, not files, so try: 

  blockdev --getsize /dev/loop0

-- 
With respect,
Roman

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: overlays on dd images of 4T drives
  2021-11-14  2:29 overlays on dd images of 4T drives David T-G
  2021-11-14  8:47 ` Roman Mamedov
@ 2021-11-15  2:51 ` David T-G
  2021-11-15 19:32   ` Phillip Susi
  1 sibling, 1 reply; 6+ messages in thread
From: David T-G @ 2021-11-15  2:51 UTC (permalink / raw)
  To: Linux RAID

Hi, all --

...and then David T-G said...
% 
% Crossing my fingers that this gets through ...  My last few messages have
% not come back to me on the list.

It looks like anything sent from my server is silently dropped.  I've
asked majordomo for help; wish me luck.  Meanwhile, I have to send these
through an alternate server :-/

This is even more fun because I'm only home every three days or so to be
able to poke at this :-(  Dang!


% 
...
% to pull an image of each RAID partition
% 
%   davidtg@gezebel:~> sudo fdisk -l /dev/sda
%   Disk /dev/sda: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
%   Disk model: ST4000DM000-1F21
...
%   diskfarm:/mnt/10Traid50md/tmp # ls -goh 4T*
%   -rw-r--r-- 1 4.0T Nov 12 02:05 '4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror'
%   -rw-r--r-- 1 4.0T Nov 12 02:05 '4Tsdb1.5YD9.dd-bs=256M-conv=sparse,noerror'
%   -rw-r--r-- 1 4.0T Nov 12 02:05 '4Tsdc1.5ZY3.dd-bs=256M-conv=sparse,noerror'
%   diskfarm:/mnt/10Traid50md/tmp # file 4Tsda1.EYNA.dd-bs\=256M-conv\=sparse\,noerror
%   4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror: Linux Software RAID version 1.2 (1) UUID=ca7008ef:90693dae:6c231ad7: 8b3f92d name=diskfarm:0 level=5 disks=4
% 
% so that I can use overlays to assemble the device and replay the XFS
...
% 
% and have created
% 
%   diskfarm:/mnt/10Traid50md/tmp # ls -goh overlay-sd*
%   -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sda1
%   -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sdb1
%   -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sdc1
%   diskfarm:/mnt/10Traid50md/tmp # losetup -a
%   /dev/loop1: [66326]:1059 (/mnt/10Traid50md/tmp/overlay-sdb1)
%   /dev/loop2: [66326]:1060 (/mnt/10Traid50md/tmp/overlay-sdc1)
%   /dev/loop0: [66326]:1058 (/mnt/10Traid50md/tmp/overlay-sda1)
% 
% my overlay files and loopback devices.  But ...
% 
% It seems that blockdev does not like
[snip]

Thanks to Roman for pointing out that I should look at the loopback
devices rather than the overlay files.  That gets me on my way again.
But ... now I have problems with the device mapper:

  diskfarm:/mnt/10Traid50md/tmp # echo $size
  8388608000
  diskfarm:/mnt/10Traid50md/tmp # echo "0 $size snapshot overlay-sda1 /dev/loop0 P 8" | dmsetup create ov-a
  device-mapper: reload ioctl on ov-a  failed: No such device
  Command failed.

ARRRGH!

OK, so I'll go to my workstation and poke at the actual devices, with
great trepidation, from there.

  davidtg@gezebel:/mnt/data/tmp/4Traid> ls -goh
  total 0
  -rw-r--r-- 1 4.0T Nov 15 02:00 overlay-sda1
  -rw-r--r-- 1 4.0T Nov 15 02:00 overlay-sdb1
  -rw-r--r-- 1 4.0T Nov 15 02:00 overlay-sdc1

  davidtg@gezebel:/mnt/data/tmp/4Traid> parallel 'size=$(sudo blockdev --getsize {}); loop=$(sudo losetup -f --show -- overlay-{/}) ; echo $loop' ::: $DEVICES
  /dev/loop0
  /dev/loop2
  /dev/loop1

  davidtg@gezebel:/mnt/data/tmp/4Traid> sudo losetup -a
  /dev/loop1: [66307]:624372298 (/mnt/data/tmp/4Traid/overlay-sdc1)
  /dev/loop2: [66307]:624372297 (/mnt/data/tmp/4Traid/overlay-sdb1)
  /dev/loop0: [66307]:624372296 (/mnt/data/tmp/4Traid/overlay-sda1)
  ### why are these out of order?!? *sigh*

Now, however, when I try to use the dev mapper to create the snapshot
device I get

  davidtg@gezebel:/mnt/data/tmp/4Traid> echo 0 `sudo blockdev --getsize /dev/sda1` snapshot /dev/sda1 /dev/loop0 P 8 | sudo dmsetup create sda1
  device-mapper: reload ioctl on sda1  failed: Device or resource busy
  Command failed.

an error.  It's simply plugged in!  What could have it busy?  Or is this
a device mapper problem on my OpenSuSE LEAP 15.2 system?

I think we need the device mapper to create the snapshot dev to be able
to write changes not to the original disks, so we're stuck there.

Soooo ...  Let's go back to diskfarm and, throwing caution to the winds,
attempt to assemble the image files copied over:

  diskfarm:/mnt/10Traid50md/tmp # mdadm --assemble --force /dev/md4 4Tsda1* 4Tsda2* 4Tsda3*
  mdadm: 4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror is not a block device.
  mdadm: 4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror has no superblock - assembly aborted

OK, yeah, we could predict that.  But it gets even more fun:

  diskfarm:/mnt/10Traid50md/tmp # mdadm --assemble --force /dev/md4 /dev/loop{0,1,2}
  mdadm: no recogniseable superblock on /dev/loop0
  mdadm: /dev/loop0 has no superblock - assembly aborted

What?!?  Where is my superblock?  This is a bit-for-bit copy of the
partition itself.

Time to fall back for more help from the RAID gods *sigh*  Any further
recommendations?


TIA again

:-D
-- 
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt


^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: overlays on dd images of 4T drives
  2021-11-15  2:51 ` David T-G
@ 2021-11-15 19:32   ` Phillip Susi
  2021-11-16  2:01     ` David T-G
  0 siblings, 1 reply; 6+ messages in thread
From: Phillip Susi @ 2021-11-15 19:32 UTC (permalink / raw)
  To: David T-G; +Cc: Linux RAID


David T-G <davidtg+robot@justpickone.org> writes:

>   davidtg@gezebel:/mnt/data/tmp/4Traid> parallel 'size=$(sudo blockdev
> --getsize {}); loop=$(sudo losetup -f --show -- overlay-{/}) ; echo
> $loop' ::: $DEVICES
>   /dev/loop0
>   /dev/loop2
>   /dev/loop1
>
>   davidtg@gezebel:/mnt/data/tmp/4Traid> sudo losetup -a
>   /dev/loop1: [66307]:624372298 (/mnt/data/tmp/4Traid/overlay-sdc1)
>   /dev/loop2: [66307]:624372297 (/mnt/data/tmp/4Traid/overlay-sdb1)
>   /dev/loop0: [66307]:624372296 (/mnt/data/tmp/4Traid/overlay-sda1)
>   ### why are these out of order?!? *sigh*

Because you ran the command to create them with parallel.  Don't do that.

> OK, yeah, we could predict that.  But it gets even more fun:
>
>   diskfarm:/mnt/10Traid50md/tmp # mdadm --assemble --force /dev/md4 /dev/loop{0,1,2}
>   mdadm: no recogniseable superblock on /dev/loop0
>   mdadm: /dev/loop0 has no superblock - assembly aborted
>
> What?!?  Where is my superblock?  This is a bit-for-bit copy of the
> partition itself.
>
> Time to fall back for more help from the RAID gods *sigh*  Any further
> recommendations?

Run mdadm -E /dev/loop{0,1,2}


^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: overlays on dd images of 4T drives
  2021-11-15 19:32   ` Phillip Susi
@ 2021-11-16  2:01     ` David T-G
  2021-11-16  2:07       ` can't loop the image files and assemble (was "Re: overlays on dd images of 4T drives") David T-G
  0 siblings, 1 reply; 6+ messages in thread
From: David T-G @ 2021-11-16  2:01 UTC (permalink / raw)
  To: Linux RAID

Phillip, et al --

...and then Phillip Susi said...
% 
% David T-G <davidtg+robot@justpickone.org> writes:
% 
% >   davidtg@gezebel:/mnt/data/tmp/4Traid> parallel 'size=$(sudo blockdev
% > --getsize {}); loop=$(sudo losetup -f --show -- overlay-{/}) ; echo
% > $loop' ::: $DEVICES
...
% >   davidtg@gezebel:/mnt/data/tmp/4Traid> sudo losetup -a
...
% >   ### why are these out of order?!? *sigh*
% 
% Because you ran the command to create them with parallel.  Don't do that.

Ah.  Well, yeah; I'd like to not, not least because I don't have parallel
on all machines and I don't know its grammar enough to know what it's
really doing.  But I also know that sometimes I should just shut up and
follow the documentation :-)

And I would figure that parallel would process them in order since they
should all take the same amount of time to run!  But that's MUTEX play
for ya.


% 
% > OK, yeah, we could predict that.  But it gets even more fun:
...
% > What?!?  Where is my superblock?  This is a bit-for-bit copy of the
% > partition itself.
% >
% > Time to fall back for more help from the RAID gods *sigh*  Any further
% > recommendations?
% 
% Run mdadm -E /dev/loop{0,1,2}

Well, that appears to agree:

  diskfarm:/mnt/10Traid50md/tmp # mdadm -E /dev/loop{0,1,2}
  mdadm: No md superblock detected on /dev/loop0.
  mdadm: No md superblock detected on /dev/loop1.
  mdadm: No md superblock detected on /dev/loop2.
  diskfarm:/mnt/10Traid50md/tmp # losetup -a
  /dev/loop1: [66326]:1059 (/mnt/10Traid50md/tmp/overlay-sdb1)
  /dev/loop2: [66326]:1060 (/mnt/10Traid50md/tmp/overlay-sdc1)
  /dev/loop0: [66326]:1058 (/mnt/10Traid50md/tmp/overlay-sda1)

But I'm confused, all right:

  diskfarm:/mnt/10Traid50md/tmp # mdadm -E 4Tsda1.EYNA.dd-bs\=256M-conv\=sparse\,noerror
  4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror:
	    Magic : a92b4efc
	  Version : 1.2
      Feature Map : 0x0
       Array UUID : ca7008ef:90693dae:6c231ad7:08b3f92d
	     Name : diskfarm:0  (local to host diskfarm)
    Creation Time : Mon Feb  6 05:56:35 2017
       Raid Level : raid5
     Raid Devices : 4

   Avail Dev Size : 7813510799 sectors (3.64 TiB 4.00 TB)
       Array Size : 11720265216 KiB (10.92 TiB 12.00 TB)
    Used Dev Size : 7813510144 sectors (3.64 TiB 4.00 TB)
      Data Offset : 262144 sectors
     Super Offset : 8 sectors
     Unused Space : before=262064 sectors, after=574835712 sectors
	    State : clean
      Device UUID : f05a143b:50c9b024:36714b9a:44b6a159

      Update Time : Sun Nov  7 01:00:00 2021
	 Checksum : 4adb8b59 - correct
	   Events : 128116

	   Layout : left-symmetric
       Chunk Size : 512K

     Device Role : Active device 3
     Array State : A.AA ('A' == active, '.' == missing, 'R' == replacing)

OH!!!  Wait.  The loopback device points to the overlay file, which is
sparse.  No wonder it has no superblock.  When I was here on this host
after having lots of device mapper issues, I tried to assemble the files
and got nowhere.

Can I create loopback devices pointing to the image files, which will
give me raw handles, and then assemble that?

Or am I stuck with struggling with the mapper on my workstation with the
actual disks in it and wondering why sda1

  davidtg@gezebel:/mnt/data/tmp/4Traid> sudo losetup -a
  /dev/loop1: [66307]:624372298 (/mnt/data/tmp/4Traid/overlay-sdc1)
  /dev/loop2: [66307]:624372297 (/mnt/data/tmp/4Traid/overlay-sdb1)
  /dev/loop0: [66307]:624372296 (/mnt/data/tmp/4Traid/overlay-sda1)
  davidtg@gezebel:/mnt/data/tmp/4Traid> echo 0 `sudo blockdev --getsize
  /dev/sda1` snapshot /dev/sda1 /dev/loop0 P 8 | sudo dmsetup create sda1
  device-mapper: reload ioctl on sda1  failed: Device or resource busy
  Command failed.

is busy?


Thanks again & HANN

:-D
-- 
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt


^ permalink raw reply	[flat|nested] 6+ messages in thread
* can't loop the image files and assemble (was "Re: overlays on dd images of 4T drives")
  2021-11-16  2:01     ` David T-G
@ 2021-11-16  2:07       ` David T-G
  0 siblings, 0 replies; 6+ messages in thread
From: David T-G @ 2021-11-16  2:07 UTC (permalink / raw)
  To: Linux RAID

Hi again, all --

...and then David T-G said...
% 
...
% 
% Can I create loopback devices pointing to the image files, which will
% give me raw handles, and then assemble that?
[snip]

Apparently that's a "no":

  diskfarm:/mnt/10Traid50md/tmp # losetup -r /dev/loop10 `pwd`/4Tsda1*
  diskfarm:/mnt/10Traid50md/tmp # losetup -r /dev/loop11 `pwd`/4Tsdb1*
  diskfarm:/mnt/10Traid50md/tmp # losetup -r /dev/loop12 `pwd`/4Tsdc1*

  diskfarm:/mnt/10Traid50md/tmp # losetup -a | grep 4T
  /dev/loop11: [66326]:1054 (/mnt/10Traid50md/tmp/4Tsdb1.5YD9.dd-bs=256M-conv=sparse,noerror)
  /dev/loop12: [66326]:1055 (/mnt/10Traid50md/tmp/4Tsdc1.5ZY3.dd-bs=256M-conv=sparse,noerror)
  /dev/loop10: [66326]:1028 (/mnt/10Traid50md/tmp/4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror)
  # hey ... i did those manually and in order! *sigh*

  diskfarm:/mnt/10Traid50md/tmp # mdadm --assemble /dev/md4 /dev/loop1{0,1,2}
  mdadm: /dev/loop10 is busy - skipping
  mdadm: /dev/loop11 is busy - skipping
  mdadm: /dev/loop12 is busy - skipping

Scratch that idea ...


:-D
-- 
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt


^ permalink raw reply	[flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-11-16  2:10 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-14  2:29 overlays on dd images of 4T drives David T-G
2021-11-14  8:47 ` Roman Mamedov
2021-11-15  2:51 ` David T-G
2021-11-15 19:32   ` Phillip Susi
2021-11-16  2:01     ` David T-G
2021-11-16  2:07       ` can't loop the image files and assemble (was "Re: overlays on dd images of 4T drives") David T-G

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.