From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mimecast-mx02.redhat.com (mimecast06.extmail.prod.ext.rdu2.redhat.com [10.11.55.22]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5B21912455D for ; Mon, 23 Mar 2020 07:46:12 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 986A4185A78E for ; Mon, 23 Mar 2020 07:46:12 +0000 (UTC) Received: by mail-wr1-f48.google.com with SMTP id j17so12213421wru.13 for ; Mon, 23 Mar 2020 00:46:08 -0700 (PDT) Received: from AM6PR03MB4389.eurprd03.prod.outlook.com ([2603:1026:c03:280f::5]) by smtp.gmail.com with ESMTPSA id k3sm20512931wmf.16.2020.03.23.00.46.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 23 Mar 2020 00:46:06 -0700 (PDT) From: Bernd Eckenfels Date: Mon, 23 Mar 2020 07:46:05 +0000 Message-ID: References: In-Reply-To: Content-Language: de-DE MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="_000_AM6PR03MB4389713EAF839D54B4A6D95CFFF00AM6PR03MB4389eurp_" Subject: Re: [linux-lvm] lvm raid5 : drives all present but vg/lvm will not assemble Reply-To: LVM general discussion and development List-Id: LVM general discussion and development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , List-Id: To: LVM general discussion and development --_000_AM6PR03MB4389713EAF839D54B4A6D95CFFF00AM6PR03MB4389eurp_ Content-Type: text/plain; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable Do you see any dmesg kernel errors when you try to activate the LVs? Gruss Bernd -- http://bernd.eckenfels.net ________________________________ Von: linux-lvm-bounces@redhat.com im Auftrag= von Andrew Falgout Gesendet: Saturday, March 21, 2020 4:22:04 AM An: linux-lvm@redhat.com Betreff: [linux-lvm] lvm raid5 : drives all present but vg/lvm will not ass= emble This started on a Raspberry PI 4 running raspbian. I moved the disks to my= Fedora 31 system, that is running the latest updates and kernel. When I h= ad the same issues there I knew it wasn't raspbian. I've reached the end of my rope on this. The disks are there, all three are= accounted for, and the LVM data on them can be seen. But it refuses to ac= tivate stating I/O errors. [root@hypervisor01 ~]# pvs PV VG Fmt Attr PSize PFree /dev/sda1 local_storage01 lvm2 a-- <931.51g 0 /dev/sdb1 local_storage01 lvm2 a-- <931.51g 0 /dev/sdc1 local_storage01 lvm2 a-- <931.51g 0 /dev/sdd1 local_storage01 lvm2 a-- <931.51g 0 /dev/sde1 local_storage01 lvm2 a-- <931.51g 0 /dev/sdf1 local_storage01 lvm2 a-- <931.51g <931.51g /dev/sdg1 local_storage01 lvm2 a-- <931.51g <931.51g /dev/sdh1 local_storage01 lvm2 a-- <931.51g <931.51g /dev/sdi3 fedora_hypervisor lvm2 a-- 27.33g <9.44g /dev/sdk1 vg1 lvm2 a-- <7.28t 0 /dev/sdl1 vg1 lvm2 a-- <7.28t 0 /dev/sdm1 vg1 lvm2 a-- <7.28t 0 [root@hypervisor01 ~]# vgs VG #PV #LV #SN Attr VSize VFree fedora_hypervisor 1 2 0 wz--n- 27.33g <9.44g local_storage01 8 1 0 wz--n- <7.28t <2.73t vg1 3 1 0 wz--n- 21.83t 0 [root@hypervisor01 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% M= ove Log Cpy%Sync Convert root fedora_hypervisor -wi-ao---- 15.00g swap fedora_hypervisor -wi-ao---- 2.89g libvirt local_storage01 rwi-aor--- <2.73t = 100.00 gluster02 vg1 Rwi---r--- 14.55t The one in question is the vg1/gluster02 lvm group. I try to activate the VG: [root@hypervisor01 ~]# vgchange -ay vg1 device-mapper: reload ioctl on (253:19) failed: Input/output error 0 logical volume(s) in volume group "vg1" now active I've got the debugging output from : vgchange -ay vg1 -vvvv -dddd lvchange -ay --partial vg1/gluster02 -vvvv -dddd Just not sure where I should dump the data for people to look at. Is there= a way to tell the md system to ignore the metadata since there wasn't an a= ctual disk failure, and rebuild the metadata off what is in the lvm? Or ca= n I even get the LV to mount, so I can pull the data off. Any help is appreciated. If I can save the data great. I'm tossing this t= o the community to see if anyone else has an idea of what I can do. ./digitalw00t --_000_AM6PR03MB4389713EAF839D54B4A6D95CFFF00AM6PR03MB4389eurp_ Content-Type: text/html; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable
Do you see any dmesg kernel errors when you tr= y to activate the LVs?

Gruss
Bernd


--
http://bernd.eckenfels.net
Von: linux-lvm-bounces@redh= at.com <linux-lvm-bounces@redhat.com> im Auftrag von Andrew Falgout &= lt;digitalw00t@gmail.com>
Gesendet: Saturday, March 21, 2020 4:22:04 AM
An: linux-lvm@redhat.com <linux-lvm@redhat.com>
Betreff: [linux-lvm] lvm raid5 : drives all present but vg/lvm will = not assemble
 

This started on a Raspberry PI 4 running raspbian.  I moved the d= isks to my Fedora 31 system, that is running the latest updates and kernel.=   When I had the same issues there I knew it wasn't raspbian.

I've reached the end of my rope on this. The disks are there, all thre= e are accounted for, and the LVM data on them can be seen.  But it ref= uses to activate stating I/O errors.

[root@hypervisor01 ~]# pvs
  PV         VG          =      Fmt  Attr PSize    PFree  
  /dev/sda1  local_storage01   lvm2 a--  <931.51g &n= bsp;     0
  /dev/sdb1  local_storage01   lvm2 a--  <931.51g &n= bsp;     0
  /dev/sdc1  local_storage01   lvm2 a--  <931.51g &n= bsp;     0
  /dev/sdd1  local_storage01   lvm2 a--  <931.51g &n= bsp;     0
  /dev/sde1  local_storage01   lvm2 a--  <931.51g &n= bsp;     0
  /dev/sdf1  local_storage01   lvm2 a--  <931.51g &l= t;931.51g
  /dev/sdg1  local_storage01   lvm2 a--  <931.51g &l= t;931.51g
  /dev/sdh1  local_storage01   lvm2 a--  <931.51g &l= t;931.51g
  /dev/sdi3  fedora_hypervisor lvm2 a--    27.33g  = ; <9.44g
  /dev/sdk1  vg1              = lvm2 a--    <7.28t       0
  /dev/sdl1  vg1              = lvm2 a--    <7.28t       0
  /dev/sdm1  vg1              = lvm2 a--    <7.28t       0
[root@hypervisor01 ~]# vgs
  VG                #PV #LV #S= N Attr   VSize  VFree
  fedora_hypervisor   1   2   0 wz--n- 27.33g <9.44g=
  local_storage01     8   1   0 wz--n- <7.28t &= lt;2.73t
  vg1                 3  = 1   0 wz--n- 21.83t     0
[root@hypervisor01 ~]# lvs
  LV        VG           =      Attr       LSize  Pool Origin Data%=  Meta%  Move Log Cpy%Sync Convert
  root      fedora_hypervisor -wi-ao---- 15.00g   =                      = ;                     &nb= sp;      
  swap      fedora_hypervisor -wi-ao----  2.89g &n= bsp;                     =                      = ;        
  libvirt   local_storage01   rwi-aor--- <2.73t   &n= bsp;                     =            100.00        =  
  gluster02 vg1               Rwi--= -r--- 14.55t                  =                      = ;             

The one in question is the vg1/gluster02 lvm group.  

I try to activate the VG:
[root@hypervisor01 ~]# vgchange -ay vg1          <= br>   device-mapper: reload ioctl on  (253:19) failed: Input/output e= rror
  0 logical volume(s) in volume group "vg1" now active

I've got the debugging output from :
vgchange -ay vg1 -vvvv -dddd
lvchange -ay --partial vg1/gluster02 -vvvv -dddd

Just not sure where I should dump the data for people to look at.  Is = there a way to tell the md system to ignore the metadata since there wasn't= an actual disk failure, and rebuild the metadata off what is in the lvm?&n= bsp; Or can I even get the LV to mount, so I can pull the data off.

Any help is appreciated.  If I can save the data great.  I'm toss= ing this to the community to see if anyone else has an idea of what I can d= o.
./digitalw00t
--_000_AM6PR03MB4389713EAF839D54B4A6D95CFFF00AM6PR03MB4389eurp_--