From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mimecast-mx02.redhat.com (mimecast04.extmail.prod.ext.rdu2.redhat.com [10.11.55.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5638120267F2 for ; Tue, 31 Mar 2020 04:56:26 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 136A6101A55A for ; Tue, 31 Mar 2020 04:56:26 +0000 (UTC) Received: by mail-qt1-f180.google.com with SMTP id t9so17265132qto.9 for ; Mon, 30 Mar 2020 21:56:22 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Andrew Falgout Date: Mon, 30 Mar 2020 23:56:10 -0500 Message-ID: Content-Type: multipart/alternative; boundary="0000000000009970cc05a21f63da" Subject: Re: [linux-lvm] lvm raid5 : drives all present but vg/lvm will not assemble Reply-To: LVM general discussion and development List-Id: LVM general discussion and development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , List-Id: To: LVM general discussion and development --0000000000009970cc05a21f63da Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Anyone have any ideas how I can salvage this array? ./drae On Tue, Mar 24, 2020 at 11:09 AM Andrew Falgout wrote: > The disks are seen, the volume groups are seen. When I try to activate > the VG I get this: > > vgchange -ay vg1 > device-mapper: reload ioctl on (253:19) failed: Input/output error > 0 logical volume(s) in volume group "vg1" now active > > I executed 'vgchange -ay vg1 -vvvv -dddd' and this is the only time an > error was thrown. > 20:53:16.552602 vgchange[10795] device_mapper/libdm-deptree.c:2921 Addin= g > target to (253:19): 0 31256068096 raid raid5_ls 3 128 region_size 32768 3 > 253:13 253:14 253:15 253:16 253:17 253:18 > 20:53:16.552609 vgchange[10795] device_mapper/ioctl/libdm-iface.c:1853 d= m > table (253:19) [ opencount flush ] [16384] (*1) > 20:53:16.552619 vgchange[10795] device_mapper/ioctl/libdm-iface.c:1853 d= m > reload (253:19) [ noopencount flush ] [16384] (*1) > 20:53:16.572481 vgchange[10795] device_mapper/ioctl/libdm-iface.c:1903 > device-mapper: reload ioctl on (253:19) failed: Input/output error > > I've uploaded two very verbose and debug ridden. > https://pastebin.com/bw5eQBa8 > https://pastebin.com/qV5yft05 > > Ignore the naming. It's not a gluster. I was planning on making two and > mirroring them in a gluster. > > ./drae > > > On Mon, Mar 23, 2020 at 5:14 AM Bernd Eckenfels > wrote: > >> Do you see any dmesg kernel errors when you try to activate the LVs? >> >> Gruss >> Bernd >> >> >> -- >> http://bernd.eckenfels.net >> ------------------------------ >> *Von:* linux-lvm-bounces@redhat.com im >> Auftrag von Andrew Falgout >> *Gesendet:* Saturday, March 21, 2020 4:22:04 AM >> *An:* linux-lvm@redhat.com >> *Betreff:* [linux-lvm] lvm raid5 : drives all present but vg/lvm will >> not assemble >> >> >> This started on a Raspberry PI 4 running raspbian. I moved the disks to >> my Fedora 31 system, that is running the latest updates and kernel. Whe= n I >> had the same issues there I knew it wasn't raspbian. >> >> I've reached the end of my rope on this. The disks are there, all three >> are accounted for, and the LVM data on them can be seen. But it refuses= to >> activate stating I/O errors. >> >> [root@hypervisor01 ~]# pvs >> PV VG Fmt Attr PSize PFree >> /dev/sda1 local_storage01 lvm2 a-- <931.51g 0 >> /dev/sdb1 local_storage01 lvm2 a-- <931.51g 0 >> /dev/sdc1 local_storage01 lvm2 a-- <931.51g 0 >> /dev/sdd1 local_storage01 lvm2 a-- <931.51g 0 >> /dev/sde1 local_storage01 lvm2 a-- <931.51g 0 >> /dev/sdf1 local_storage01 lvm2 a-- <931.51g <931.51g >> /dev/sdg1 local_storage01 lvm2 a-- <931.51g <931.51g >> /dev/sdh1 local_storage01 lvm2 a-- <931.51g <931.51g >> /dev/sdi3 fedora_hypervisor lvm2 a-- 27.33g <9.44g >> /dev/sdk1 vg1 lvm2 a-- <7.28t 0 >> /dev/sdl1 vg1 lvm2 a-- <7.28t 0 >> /dev/sdm1 vg1 lvm2 a-- <7.28t 0 >> [root@hypervisor01 ~]# vgs >> VG #PV #LV #SN Attr VSize VFree >> fedora_hypervisor 1 2 0 wz--n- 27.33g <9.44g >> local_storage01 8 1 0 wz--n- <7.28t <2.73t >> vg1 3 1 0 wz--n- 21.83t 0 >> [root@hypervisor01 ~]# lvs >> LV VG Attr LSize Pool Origin Data% Meta% >> Move Log Cpy%Sync Convert >> root fedora_hypervisor -wi-ao---- 15.00g >> >> swap fedora_hypervisor -wi-ao---- 2.89g >> >> libvirt local_storage01 rwi-aor--- <2.73t >> 100.00 >> gluster02 vg1 Rwi---r--- 14.55t >> >> >> The one in question is the vg1/gluster02 lvm group. >> >> I try to activate the VG: >> [root@hypervisor01 ~]# vgchange -ay vg1 >> device-mapper: reload ioctl on (253:19) failed: Input/output error >> 0 logical volume(s) in volume group "vg1" now active >> >> I've got the debugging output from : >> vgchange -ay vg1 -vvvv -dddd >> lvchange -ay --partial vg1/gluster02 -vvvv -dddd >> >> Just not sure where I should dump the data for people to look at. Is >> there a way to tell the md system to ignore the metadata since there was= n't >> an actual disk failure, and rebuild the metadata off what is in the lvm? >> Or can I even get the LV to mount, so I can pull the data off. >> >> Any help is appreciated. If I can save the data great. I'm tossing thi= s >> to the community to see if anyone else has an idea of what I can do. >> ./digitalw00t >> _______________________________________________ >> linux-lvm mailing list >> linux-lvm@redhat.com >> https://www.redhat.com/mailman/listinfo/linux-lvm >> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ > > --0000000000009970cc05a21f63da Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Anyone have any ideas how I can salvage this array?
./drae


On Tue, Mar 24, 2020 at= 11:09 AM Andrew Falgout <digit= alw00t@gmail.com> wrote:
The disks are seen, the volume groups are = seen.=C2=A0 When I try to activate the VG I get this:

vgchange -ay = vg1
=C2=A0 device-mapper: reload ioctl on =C2=A0(253:19) failed: Input/o= utput error
=C2=A0 0 logical volume(s) in volume group "vg1" n= ow active

I executed 'vgchange -ay vg1 -vvvv -dddd&#= 39; and this is the only time an error was thrown.
20:53:16.552602 vgcha= nge[10795] device_mapper/libdm-deptree.c:2921 =C2=A0Adding target to (253:1= 9): 0 31256068096 raid raid5_ls 3 128 region_size 32768 3 253:13 253:14 253= :15 253:16 253:17 253:18
20:53:16.552609 vgchange[10795] device_mapper/i= octl/libdm-iface.c:1853 =C2=A0dm table =C2=A0 (253:19) [ opencount flush ] = =C2=A0 [16384] (*1)
20:53:16.552619 vgchange[10795] device_mapper/ioctl/= libdm-iface.c:1853 =C2=A0dm reload =C2=A0 (253:19) [ noopencount flush ] = =C2=A0 [16384] (*1)
20:53:16.572481 vgchange[10795] device_mapper/ioctl/= libdm-iface.c:1903 =C2=A0device-mapper: reload ioctl on =C2=A0(253:19) fail= ed: Input/output error

I've uploaded two v= ery verbose and debug ridden.

Ignore the naming.=C2=A0 It'= ;s not a gluster.=C2=A0 I was planning on making two and mirroring them in = a gluster.=C2=A0

./drae<= /div>


On Mon, Mar 23, 2020 at 5:14 AM Bernd Eckenfels = <ecki@zusamm= enkunft.net> wrote:
Do you see any dmesg kernel errors when you tr= y to activate the LVs?

Gruss
Bernd

Von: linux-lvm-bounces@redhat.com <linux-lvm-bounces@redha= t.com> im Auftrag von Andrew Falgout <digitalw00t@gmail.com>
Gesendet: Saturday, March 21, 2020 4:22:04 AM
An: linux-= lvm@redhat.com <linux-lvm@redhat.com>
Betreff: [linux-lvm] lvm raid5 : drives all present but vg/lvm will = not assemble
=C2=A0

This started on a Raspberry PI 4 running raspbian.=C2=A0 I moved the d= isks to my Fedora 31 system, that is running the latest updates and kernel.= =C2=A0 When I had the same issues there I knew it wasn't raspbian.

I've reached the end of my rope on this. The disks are there, all = three are accounted for, and the LVM data on them can be seen.=C2=A0 But it= refuses to activate stating I/O errors.

[root@hypervisor01 ~]# pvs
=C2=A0 PV =C2=A0 =C2=A0 =C2=A0 =C2=A0 VG =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0Fmt =C2=A0Attr PSize =C2=A0 =C2=A0PFree =C2=A0
=C2=A0 /dev/sda1 =C2=A0local_storage01 =C2=A0 lvm2 a-- =C2=A0<931.51g = =C2=A0 =C2=A0 =C2=A0 0
=C2=A0 /dev/sdb1 =C2=A0local_storage01 =C2=A0 lvm2 a-- =C2=A0<931.51g = =C2=A0 =C2=A0 =C2=A0 0
=C2=A0 /dev/sdc1 =C2=A0local_storage01 =C2=A0 lvm2 a-- =C2=A0<931.51g = =C2=A0 =C2=A0 =C2=A0 0
=C2=A0 /dev/sdd1 =C2=A0local_storage01 =C2=A0 lvm2 a-- =C2=A0<931.51g = =C2=A0 =C2=A0 =C2=A0 0
=C2=A0 /dev/sde1 =C2=A0local_storage01 =C2=A0 lvm2 a-- =C2=A0<931.51g = =C2=A0 =C2=A0 =C2=A0 0
=C2=A0 /dev/sdf1 =C2=A0local_storage01 =C2=A0 lvm2 a-- =C2=A0<931.51g &l= t;931.51g
=C2=A0 /dev/sdg1 =C2=A0local_storage01 =C2=A0 lvm2 a-- =C2=A0<931.51g &l= t;931.51g
=C2=A0 /dev/sdh1 =C2=A0local_storage01 =C2=A0 lvm2 a-- =C2=A0<931.51g &l= t;931.51g
=C2=A0 /dev/sdi3 =C2=A0fedora_hypervisor lvm2 a-- =C2=A0 =C2=A027.33g =C2= =A0 <9.44g
=C2=A0 /dev/sdk1 =C2=A0vg1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= lvm2 a-- =C2=A0 =C2=A0<7.28t =C2=A0 =C2=A0 =C2=A0 0
=C2=A0 /dev/sdl1 =C2=A0vg1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= lvm2 a-- =C2=A0 =C2=A0<7.28t =C2=A0 =C2=A0 =C2=A0 0
=C2=A0 /dev/sdm1 =C2=A0vg1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= lvm2 a-- =C2=A0 =C2=A0<7.28t =C2=A0 =C2=A0 =C2=A0 0
[root@hypervisor01 ~]# vgs
=C2=A0 VG =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0#PV #LV #S= N Attr =C2=A0 VSize =C2=A0VFree
=C2=A0 fedora_hypervisor =C2=A0 1 =C2=A0 2 =C2=A0 0 wz--n- 27.33g <9.44g=
=C2=A0 local_storage01 =C2=A0 =C2=A0 8 =C2=A0 1 =C2=A0 0 wz--n- <7.28t &= lt;2.73t
=C2=A0 vg1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 3 =C2=A0= 1 =C2=A0 0 wz--n- 21.83t =C2=A0 =C2=A0 0
[root@hypervisor01 ~]# lvs
=C2=A0 LV =C2=A0 =C2=A0 =C2=A0 =C2=A0VG =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0Attr =C2=A0 =C2=A0 =C2=A0 LSize =C2=A0Pool Origin Data%= =C2=A0Meta% =C2=A0Move Log Cpy%Sync Convert
=C2=A0 root =C2=A0 =C2=A0 =C2=A0fedora_hypervisor -wi-ao---- 15.00g =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0
=C2=A0 swap =C2=A0 =C2=A0 =C2=A0fedora_hypervisor -wi-ao---- =C2=A02.89g = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
=C2=A0 libvirt =C2=A0 local_storage01 =C2=A0 rwi-aor--- <2.73t =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0100.00 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0
=C2=A0 gluster02 vg1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Rwi--= -r--- 14.55t=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0

The one in question is the vg1/gluster02 lvm group.=C2=A0=C2=A0

I try to activate the VG:
[root@hypervisor01 ~]# vgchange -ay vg1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<= br> =C2=A0 device-mapper: reload ioctl on =C2=A0(253:19) failed: Input/output e= rror
=C2=A0 0 logical volume(s) in volume group "vg1" now active

I've got the debugging output from :
vgchange -ay vg1 -vvvv -dddd
lvchange -ay --partial vg1/gluster02 -vvvv -dddd

Just not sure where I should dump the data for people to look at.=C2=A0 Is = there a way to tell the md system to ignore the metadata since there wasn&#= 39;t an actual disk failure, and rebuild the metadata off what is in the lv= m?=C2=A0 Or can I even get the LV to mount, so I can pull the data off.

Any help is appreciated.=C2=A0 If I can save the data great.=C2=A0 I'm = tossing this to the community to see if anyone else has an idea of what I c= an do.
./digitalw00t
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.= com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
--0000000000009970cc05a21f63da--