* overlays on dd images of 4T drives
@ 2021-11-14 2:29 David T-G
2021-11-14 8:47 ` Roman Mamedov
2021-11-15 2:51 ` David T-G
0 siblings, 2 replies; 6+ messages in thread
From: David T-G @ 2021-11-14 2:29 UTC (permalink / raw)
To: Linux RAID
Hi, all --
Crossing my fingers that this gets through ... My last few messages have
not come back to me on the list.
After a first attempt to assemble the two working and one wonky drives of
my 4-disk array and then watching reads hang forever, I have now used dd
to pull an image of each RAID partition
davidtg@gezebel:~> sudo fdisk -l /dev/sda
Disk /dev/sda: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: ST4000DM000-1F21
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 482CA0F0-663B-409D-A4DF-4A4B248E6D23
Device Start End Sectors Size Type
/dev/sda1 2048 7813774990 7813772943 3.7T Linux RAID
/dev/sda2 7813775360 7814037134 261775 127.8M Linux filesystem
diskfarm:/mnt/10Traid50md/tmp # ls -goh 4T*
-rw-r--r-- 1 4.0T Nov 12 02:05 '4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror'
-rw-r--r-- 1 4.0T Nov 12 02:05 '4Tsdb1.5YD9.dd-bs=256M-conv=sparse,noerror'
-rw-r--r-- 1 4.0T Nov 12 02:05 '4Tsdc1.5ZY3.dd-bs=256M-conv=sparse,noerror'
diskfarm:/mnt/10Traid50md/tmp # file 4Tsda1.EYNA.dd-bs\=256M-conv\=sparse\,noerror
4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror: Linux Software RAID version 1.2 (1) UUID=ca7008ef:90693dae:6c231ad7: 8b3f92d name=diskfarm:0 level=5 disks=4
so that I can use overlays to assemble the device and replay the XFS
journal to get up and running again to finally be able to pull off this
data.
After going back and forth across the deprecated wiki pages, I'm working
through Irreversible Failure Recovery
https://raid.wiki.kernel.org/index.php/Irreversible_mdadm_failure_recovery
and have created
diskfarm:/mnt/10Traid50md/tmp # ls -goh overlay-sd*
-rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sda1
-rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sdb1
-rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sdc1
diskfarm:/mnt/10Traid50md/tmp # losetup -a
/dev/loop1: [66326]:1059 (/mnt/10Traid50md/tmp/overlay-sdb1)
/dev/loop2: [66326]:1060 (/mnt/10Traid50md/tmp/overlay-sdc1)
/dev/loop0: [66326]:1058 (/mnt/10Traid50md/tmp/overlay-sda1)
my overlay files and loopback devices. But ...
It seems that blockdev does not like
diskfarm:/mnt/10Traid50md/tmp # blockdev --getsize ./overlay-sda1
blockdev: ioctl error on BLKGETSIZE: Inappropriate ioctl for device
my files and so it won't give me the size that I need for dmsetup in the
next step.
How can I use these partition images as source for faking up my RAID
array to recover the contents? I seem to have gotten myself stuck :-/
TIA & HAND
:-D
--
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: overlays on dd images of 4T drives
2021-11-14 2:29 overlays on dd images of 4T drives David T-G
@ 2021-11-14 8:47 ` Roman Mamedov
2021-11-15 2:51 ` David T-G
1 sibling, 0 replies; 6+ messages in thread
From: Roman Mamedov @ 2021-11-14 8:47 UTC (permalink / raw)
To: David T-G; +Cc: Linux RAID
On Sun, 14 Nov 2021 02:29:24 +0000
David T-G <davidtg+robot@justpickone.org> wrote:
> diskfarm:/mnt/10Traid50md/tmp # ls -goh overlay-sd*
> -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sda1
> -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sdb1
> -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sdc1
> diskfarm:/mnt/10Traid50md/tmp # losetup -a
> /dev/loop1: [66326]:1059 (/mnt/10Traid50md/tmp/overlay-sdb1)
> /dev/loop2: [66326]:1060 (/mnt/10Traid50md/tmp/overlay-sdc1)
> /dev/loop0: [66326]:1058 (/mnt/10Traid50md/tmp/overlay-sda1)
>
> my overlay files and loopback devices. But ...
>
> It seems that blockdev does not like
>
> diskfarm:/mnt/10Traid50md/tmp # blockdev --getsize ./overlay-sda1
> blockdev: ioctl error on BLKGETSIZE: Inappropriate ioctl for device
>
> my files
It must be run on block devices, not files, so try:
blockdev --getsize /dev/loop0
--
With respect,
Roman
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: overlays on dd images of 4T drives
2021-11-14 2:29 overlays on dd images of 4T drives David T-G
2021-11-14 8:47 ` Roman Mamedov
@ 2021-11-15 2:51 ` David T-G
2021-11-15 19:32 ` Phillip Susi
1 sibling, 1 reply; 6+ messages in thread
From: David T-G @ 2021-11-15 2:51 UTC (permalink / raw)
To: Linux RAID
Hi, all --
...and then David T-G said...
%
% Crossing my fingers that this gets through ... My last few messages have
% not come back to me on the list.
It looks like anything sent from my server is silently dropped. I've
asked majordomo for help; wish me luck. Meanwhile, I have to send these
through an alternate server :-/
This is even more fun because I'm only home every three days or so to be
able to poke at this :-( Dang!
%
...
% to pull an image of each RAID partition
%
% davidtg@gezebel:~> sudo fdisk -l /dev/sda
% Disk /dev/sda: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
% Disk model: ST4000DM000-1F21
...
% diskfarm:/mnt/10Traid50md/tmp # ls -goh 4T*
% -rw-r--r-- 1 4.0T Nov 12 02:05 '4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror'
% -rw-r--r-- 1 4.0T Nov 12 02:05 '4Tsdb1.5YD9.dd-bs=256M-conv=sparse,noerror'
% -rw-r--r-- 1 4.0T Nov 12 02:05 '4Tsdc1.5ZY3.dd-bs=256M-conv=sparse,noerror'
% diskfarm:/mnt/10Traid50md/tmp # file 4Tsda1.EYNA.dd-bs\=256M-conv\=sparse\,noerror
% 4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror: Linux Software RAID version 1.2 (1) UUID=ca7008ef:90693dae:6c231ad7: 8b3f92d name=diskfarm:0 level=5 disks=4
%
% so that I can use overlays to assemble the device and replay the XFS
...
%
% and have created
%
% diskfarm:/mnt/10Traid50md/tmp # ls -goh overlay-sd*
% -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sda1
% -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sdb1
% -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sdc1
% diskfarm:/mnt/10Traid50md/tmp # losetup -a
% /dev/loop1: [66326]:1059 (/mnt/10Traid50md/tmp/overlay-sdb1)
% /dev/loop2: [66326]:1060 (/mnt/10Traid50md/tmp/overlay-sdc1)
% /dev/loop0: [66326]:1058 (/mnt/10Traid50md/tmp/overlay-sda1)
%
% my overlay files and loopback devices. But ...
%
% It seems that blockdev does not like
[snip]
Thanks to Roman for pointing out that I should look at the loopback
devices rather than the overlay files. That gets me on my way again.
But ... now I have problems with the device mapper:
diskfarm:/mnt/10Traid50md/tmp # echo $size
8388608000
diskfarm:/mnt/10Traid50md/tmp # echo "0 $size snapshot overlay-sda1 /dev/loop0 P 8" | dmsetup create ov-a
device-mapper: reload ioctl on ov-a failed: No such device
Command failed.
ARRRGH!
OK, so I'll go to my workstation and poke at the actual devices, with
great trepidation, from there.
davidtg@gezebel:/mnt/data/tmp/4Traid> ls -goh
total 0
-rw-r--r-- 1 4.0T Nov 15 02:00 overlay-sda1
-rw-r--r-- 1 4.0T Nov 15 02:00 overlay-sdb1
-rw-r--r-- 1 4.0T Nov 15 02:00 overlay-sdc1
davidtg@gezebel:/mnt/data/tmp/4Traid> parallel 'size=$(sudo blockdev --getsize {}); loop=$(sudo losetup -f --show -- overlay-{/}) ; echo $loop' ::: $DEVICES
/dev/loop0
/dev/loop2
/dev/loop1
davidtg@gezebel:/mnt/data/tmp/4Traid> sudo losetup -a
/dev/loop1: [66307]:624372298 (/mnt/data/tmp/4Traid/overlay-sdc1)
/dev/loop2: [66307]:624372297 (/mnt/data/tmp/4Traid/overlay-sdb1)
/dev/loop0: [66307]:624372296 (/mnt/data/tmp/4Traid/overlay-sda1)
### why are these out of order?!? *sigh*
Now, however, when I try to use the dev mapper to create the snapshot
device I get
davidtg@gezebel:/mnt/data/tmp/4Traid> echo 0 `sudo blockdev --getsize /dev/sda1` snapshot /dev/sda1 /dev/loop0 P 8 | sudo dmsetup create sda1
device-mapper: reload ioctl on sda1 failed: Device or resource busy
Command failed.
an error. It's simply plugged in! What could have it busy? Or is this
a device mapper problem on my OpenSuSE LEAP 15.2 system?
I think we need the device mapper to create the snapshot dev to be able
to write changes not to the original disks, so we're stuck there.
Soooo ... Let's go back to diskfarm and, throwing caution to the winds,
attempt to assemble the image files copied over:
diskfarm:/mnt/10Traid50md/tmp # mdadm --assemble --force /dev/md4 4Tsda1* 4Tsda2* 4Tsda3*
mdadm: 4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror is not a block device.
mdadm: 4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror has no superblock - assembly aborted
OK, yeah, we could predict that. But it gets even more fun:
diskfarm:/mnt/10Traid50md/tmp # mdadm --assemble --force /dev/md4 /dev/loop{0,1,2}
mdadm: no recogniseable superblock on /dev/loop0
mdadm: /dev/loop0 has no superblock - assembly aborted
What?!? Where is my superblock? This is a bit-for-bit copy of the
partition itself.
Time to fall back for more help from the RAID gods *sigh* Any further
recommendations?
TIA again
:-D
--
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: overlays on dd images of 4T drives
2021-11-15 2:51 ` David T-G
@ 2021-11-15 19:32 ` Phillip Susi
2021-11-16 2:01 ` David T-G
0 siblings, 1 reply; 6+ messages in thread
From: Phillip Susi @ 2021-11-15 19:32 UTC (permalink / raw)
To: David T-G; +Cc: Linux RAID
David T-G <davidtg+robot@justpickone.org> writes:
> davidtg@gezebel:/mnt/data/tmp/4Traid> parallel 'size=$(sudo blockdev
> --getsize {}); loop=$(sudo losetup -f --show -- overlay-{/}) ; echo
> $loop' ::: $DEVICES
> /dev/loop0
> /dev/loop2
> /dev/loop1
>
> davidtg@gezebel:/mnt/data/tmp/4Traid> sudo losetup -a
> /dev/loop1: [66307]:624372298 (/mnt/data/tmp/4Traid/overlay-sdc1)
> /dev/loop2: [66307]:624372297 (/mnt/data/tmp/4Traid/overlay-sdb1)
> /dev/loop0: [66307]:624372296 (/mnt/data/tmp/4Traid/overlay-sda1)
> ### why are these out of order?!? *sigh*
Because you ran the command to create them with parallel. Don't do that.
> OK, yeah, we could predict that. But it gets even more fun:
>
> diskfarm:/mnt/10Traid50md/tmp # mdadm --assemble --force /dev/md4 /dev/loop{0,1,2}
> mdadm: no recogniseable superblock on /dev/loop0
> mdadm: /dev/loop0 has no superblock - assembly aborted
>
> What?!? Where is my superblock? This is a bit-for-bit copy of the
> partition itself.
>
> Time to fall back for more help from the RAID gods *sigh* Any further
> recommendations?
Run mdadm -E /dev/loop{0,1,2}
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: overlays on dd images of 4T drives
2021-11-15 19:32 ` Phillip Susi
@ 2021-11-16 2:01 ` David T-G
2021-11-16 2:07 ` can't loop the image files and assemble (was "Re: overlays on dd images of 4T drives") David T-G
0 siblings, 1 reply; 6+ messages in thread
From: David T-G @ 2021-11-16 2:01 UTC (permalink / raw)
To: Linux RAID
Phillip, et al --
...and then Phillip Susi said...
%
% David T-G <davidtg+robot@justpickone.org> writes:
%
% > davidtg@gezebel:/mnt/data/tmp/4Traid> parallel 'size=$(sudo blockdev
% > --getsize {}); loop=$(sudo losetup -f --show -- overlay-{/}) ; echo
% > $loop' ::: $DEVICES
...
% > davidtg@gezebel:/mnt/data/tmp/4Traid> sudo losetup -a
...
% > ### why are these out of order?!? *sigh*
%
% Because you ran the command to create them with parallel. Don't do that.
Ah. Well, yeah; I'd like to not, not least because I don't have parallel
on all machines and I don't know its grammar enough to know what it's
really doing. But I also know that sometimes I should just shut up and
follow the documentation :-)
And I would figure that parallel would process them in order since they
should all take the same amount of time to run! But that's MUTEX play
for ya.
%
% > OK, yeah, we could predict that. But it gets even more fun:
...
% > What?!? Where is my superblock? This is a bit-for-bit copy of the
% > partition itself.
% >
% > Time to fall back for more help from the RAID gods *sigh* Any further
% > recommendations?
%
% Run mdadm -E /dev/loop{0,1,2}
Well, that appears to agree:
diskfarm:/mnt/10Traid50md/tmp # mdadm -E /dev/loop{0,1,2}
mdadm: No md superblock detected on /dev/loop0.
mdadm: No md superblock detected on /dev/loop1.
mdadm: No md superblock detected on /dev/loop2.
diskfarm:/mnt/10Traid50md/tmp # losetup -a
/dev/loop1: [66326]:1059 (/mnt/10Traid50md/tmp/overlay-sdb1)
/dev/loop2: [66326]:1060 (/mnt/10Traid50md/tmp/overlay-sdc1)
/dev/loop0: [66326]:1058 (/mnt/10Traid50md/tmp/overlay-sda1)
But I'm confused, all right:
diskfarm:/mnt/10Traid50md/tmp # mdadm -E 4Tsda1.EYNA.dd-bs\=256M-conv\=sparse\,noerror
4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ca7008ef:90693dae:6c231ad7:08b3f92d
Name : diskfarm:0 (local to host diskfarm)
Creation Time : Mon Feb 6 05:56:35 2017
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 7813510799 sectors (3.64 TiB 4.00 TB)
Array Size : 11720265216 KiB (10.92 TiB 12.00 TB)
Used Dev Size : 7813510144 sectors (3.64 TiB 4.00 TB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=574835712 sectors
State : clean
Device UUID : f05a143b:50c9b024:36714b9a:44b6a159
Update Time : Sun Nov 7 01:00:00 2021
Checksum : 4adb8b59 - correct
Events : 128116
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : A.AA ('A' == active, '.' == missing, 'R' == replacing)
OH!!! Wait. The loopback device points to the overlay file, which is
sparse. No wonder it has no superblock. When I was here on this host
after having lots of device mapper issues, I tried to assemble the files
and got nowhere.
Can I create loopback devices pointing to the image files, which will
give me raw handles, and then assemble that?
Or am I stuck with struggling with the mapper on my workstation with the
actual disks in it and wondering why sda1
davidtg@gezebel:/mnt/data/tmp/4Traid> sudo losetup -a
/dev/loop1: [66307]:624372298 (/mnt/data/tmp/4Traid/overlay-sdc1)
/dev/loop2: [66307]:624372297 (/mnt/data/tmp/4Traid/overlay-sdb1)
/dev/loop0: [66307]:624372296 (/mnt/data/tmp/4Traid/overlay-sda1)
davidtg@gezebel:/mnt/data/tmp/4Traid> echo 0 `sudo blockdev --getsize
/dev/sda1` snapshot /dev/sda1 /dev/loop0 P 8 | sudo dmsetup create sda1
device-mapper: reload ioctl on sda1 failed: Device or resource busy
Command failed.
is busy?
Thanks again & HANN
:-D
--
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt
^ permalink raw reply [flat|nested] 6+ messages in thread* can't loop the image files and assemble (was "Re: overlays on dd images of 4T drives")
2021-11-16 2:01 ` David T-G
@ 2021-11-16 2:07 ` David T-G
0 siblings, 0 replies; 6+ messages in thread
From: David T-G @ 2021-11-16 2:07 UTC (permalink / raw)
To: Linux RAID
Hi again, all --
...and then David T-G said...
%
...
%
% Can I create loopback devices pointing to the image files, which will
% give me raw handles, and then assemble that?
[snip]
Apparently that's a "no":
diskfarm:/mnt/10Traid50md/tmp # losetup -r /dev/loop10 `pwd`/4Tsda1*
diskfarm:/mnt/10Traid50md/tmp # losetup -r /dev/loop11 `pwd`/4Tsdb1*
diskfarm:/mnt/10Traid50md/tmp # losetup -r /dev/loop12 `pwd`/4Tsdc1*
diskfarm:/mnt/10Traid50md/tmp # losetup -a | grep 4T
/dev/loop11: [66326]:1054 (/mnt/10Traid50md/tmp/4Tsdb1.5YD9.dd-bs=256M-conv=sparse,noerror)
/dev/loop12: [66326]:1055 (/mnt/10Traid50md/tmp/4Tsdc1.5ZY3.dd-bs=256M-conv=sparse,noerror)
/dev/loop10: [66326]:1028 (/mnt/10Traid50md/tmp/4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror)
# hey ... i did those manually and in order! *sigh*
diskfarm:/mnt/10Traid50md/tmp # mdadm --assemble /dev/md4 /dev/loop1{0,1,2}
mdadm: /dev/loop10 is busy - skipping
mdadm: /dev/loop11 is busy - skipping
mdadm: /dev/loop12 is busy - skipping
Scratch that idea ...
:-D
--
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-11-16 2:10 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-14 2:29 overlays on dd images of 4T drives David T-G
2021-11-14 8:47 ` Roman Mamedov
2021-11-15 2:51 ` David T-G
2021-11-15 19:32 ` Phillip Susi
2021-11-16 2:01 ` David T-G
2021-11-16 2:07 ` can't loop the image files and assemble (was "Re: overlays on dd images of 4T drives") David T-G
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.