Re: [linux-lvm] LVM RAID behavior after losing physical disk

["John Stoffel" <john@stoffel.org>]

Andrei> I apologize for replying to my own message, I was subscribed in the
Andrei> digest mode...

No problem

Andrei> I've been hitting my head against this one for a while
Andrei> now. Originally discovered this on Ubuntu 20.04, but I'm
Andrei> seeing the same issue with RHEL 8.5. The loss of a single disk
Andrei> leaves the RAID in "partial" mode, when it should be
Andrei> "degraded".

Ouch, this isn't good.  But why aren't you using MD RAID on top of the
disks (partitioned ideally in my book) and then turn that MD device
into a PV in a VG, and then make your LVs in there?  

Andrei> I've tried to explicitly specify the number of stripes, but it
Andrei> did not make a difference. After adding the missing disk back,
Andrei> the array is healthy again. Please see below.

I'm wondering if there's something setup in the defaults for
/etc/lvm.conf which makes a degraded array fail, instead of coming up
degraded.

But honestly, if you're looking for disk level redundancy, then I'd
stronly recommend you use disks -> MD RAID6 -> LVM - > filesystem  for
your data.

It's reliable, durable and well understood.

I know there's an attraction to runnning LVs and RAID all together,
since that should be easier to manage, right?  But I think not.

Have you tried to activate the LV using:

   lvchange -ay --activationmode degraded LV
   
as a test?  What does it say?  I'm looking at the lvmraid man page for
this suggestion.


Andrei> # cat /etc/redhat-release
Andrei> Red Hat Enterprise Linux release 8.5 (Ootpa)
Andrei> # lvm version
Andrei>   LVM version:     2.03.12(2)-RHEL8 (2021-05-19)
Andrei>   Library version: 1.02.177-RHEL8 (2021-05-19)
Andrei>   Driver version:  4.43.0

Andrei> # lsblk
Andrei> NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
Andrei> sda             8:0    0   50G  0 disk
Andrei> ├─sda1          8:1    0    1G  0 part /boot
Andrei> └─sda2          8:2    0   49G  0 part
Andrei>   ├─rhel-root 253:0    0   44G  0 lvm  /
Andrei>   └─rhel-swap 253:1    0    5G  0 lvm  [SWAP]
Andrei> sdb             8:16   0   70G  0 disk
Andrei> sdc             8:32   0  100G  0 disk
Andrei> sdd             8:48   0  100G  0 disk
Andrei> sde             8:64   0  100G  0 disk
Andrei> sdf             8:80   0  100G  0 disk
Andrei> sdg             8:96   0  100G  0 disk

Andrei> # pvcreate /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg
Andrei> # vgcreate pool_vg /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg

Andrei> # lvcreate -l +100%FREE -n pool_lv --type raid6 --stripes 3
Andrei> --stripesize 1 pool_vg
Andrei>   Invalid stripe size 1.00 KiB.
Andrei>   Run `lvcreate --help' for more information.

Andrei> # lvcreate -l +100%FREE -n pool_lv --type raid6 --stripes 3
Andrei> --stripesize 4 pool_vg
Andrei>   Logical volume "pool_lv" created.

Andrei> # mkfs.xfs /dev/pool_vg/pool_lv
Andrei> # echo "/dev/mapper/pool_vg-pool_lv /mnt xfs
Andrei> defaults,x-systemd.mount-timeout=30 0 0" >> /etc/fstab
Andrei> # mount -a
Andrei> # touch /mnt/test

Andrei> Note the RAID is correctly striped across all 5 disks:

Andrei> # lvs -a -o name,lv_attr,copy_percent,health_status,devices pool_vg
Andrei>   LV                 Attr       Cpy%Sync Health          Devices
Andrei>   pool_lv            rwi-aor--- 100.00
Andrei> pool_lv_rimage_0(0),pool_lv_rimage_1(0),pool_lv_rimage_2(0),pool_lv_rimage_3(0),pool_lv_rimage_4(0)
Andrei>   [pool_lv_rimage_0] iwi-aor---                          /dev/sdc(1)
Andrei>   [pool_lv_rimage_1] iwi-aor---                          /dev/sdd(1)
Andrei>   [pool_lv_rimage_2] iwi-aor---                          /dev/sde(1)
Andrei>   [pool_lv_rimage_3] iwi-aor---                          /dev/sdf(1)
Andrei>   [pool_lv_rimage_4] iwi-aor---                          /dev/sdg(1)
Andrei>   [pool_lv_rmeta_0]  ewi-aor---                          /dev/sdc(0)
Andrei>   [pool_lv_rmeta_1]  ewi-aor---                          /dev/sdd(0)
Andrei>   [pool_lv_rmeta_2]  ewi-aor---                          /dev/sde(0)
Andrei>   [pool_lv_rmeta_3]  ewi-aor---                          /dev/sdf(0)
Andrei>   [pool_lv_rmeta_4]  ewi-aor---                          /dev/sdg(0)

Andrei> After shutting down the OS and removing a disk, reboot drops the
Andrei> system into a single user mode because it cannot mount /mnt! The RAID
Andrei> is now in "partial" mode, when it must be just "degraded".

Andrei> # lvs -a -o name,lv_attr,copy_percent,health_status,devices pool_vg
Andrei>   WARNING: Couldn't find device with uuid
Andrei> d5y3gp-taRv-2YMa-3mR0-94ZZ-72Od-IKF8Co.
Andrei>   WARNING: VG pool_vg is missing PV
Andrei> d5y3gp-taRv-2YMa-3mR0-94ZZ-72Od-IKF8Co (last written to /dev/sdc).
Andrei>   LV                 Attr       Cpy%Sync Health          Devices
Andrei>   pool_lv            rwi---r-p-          partial
Andrei> pool_lv_rimage_0(0),pool_lv_rimage_1(0),pool_lv_rimage_2(0),pool_lv_rimage_3(0),pool_lv_rimage_4(0)
Andrei>   [pool_lv_rimage_0] Iwi---r-p-          partial         [unknown](1)
Andrei>   [pool_lv_rimage_1] Iwi---r---                          /dev/sdc(1)
Andrei>   [pool_lv_rimage_2] Iwi---r---                          /dev/sdd(1)
Andrei>   [pool_lv_rimage_3] Iwi---r---                          /dev/sde(1)
Andrei>   [pool_lv_rimage_4] Iwi---r---                          /dev/sdf(1)
Andrei>   [pool_lv_rmeta_0]  ewi---r-p-          partial         [unknown](0)
Andrei>   [pool_lv_rmeta_1]  ewi---r---                          /dev/sdc(0)
Andrei>   [pool_lv_rmeta_2]  ewi---r---                          /dev/sdd(0)
Andrei>   [pool_lv_rmeta_3]  ewi---r---                          /dev/sde(0)
Andrei>   [pool_lv_rmeta_4]  ewi---r---                          /dev/sdf(0)

Andrei> After adding the missing disk back, the system boots correctly and
Andrei> there are no issues with the RAID:

Andrei> # lvs -a -o name,lv_attr,copy_percent,health_status,devices pool_vg
Andrei>   LV                 Attr       Cpy%Sync Health          Devices
Andrei>   pool_lv            rwi-a-r--- 100.00
Andrei> pool_lv_rimage_0(0),pool_lv_rimage_1(0),pool_lv_rimage_2(0),pool_lv_rimage_3(0),pool_lv_rimage_4(0)
Andrei>   [pool_lv_rimage_0] iwi-aor---                          /dev/sdc(1)
Andrei>   [pool_lv_rimage_1] iwi-aor---                          /dev/sdd(1)
Andrei>   [pool_lv_rimage_2] iwi-aor---                          /dev/sde(1)
Andrei>   [pool_lv_rimage_3] iwi-aor---                          /dev/sdf(1)
Andrei>   [pool_lv_rimage_4] iwi-aor---                          /dev/sdg(1)
Andrei>   [pool_lv_rmeta_0]  ewi-aor---                          /dev/sdc(0)
Andrei>   [pool_lv_rmeta_1]  ewi-aor---                          /dev/sdd(0)
Andrei>   [pool_lv_rmeta_2]  ewi-aor---                          /dev/sde(0)
Andrei>   [pool_lv_rmeta_3]  ewi-aor---                          /dev/sdf(0)
Andrei>   [pool_lv_rmeta_4]  ewi-aor---                          /dev/sdg(0)


Andrei> On Tue, Jan 25, 2022 at 11:44 AM Andrei Rodionov
Andrei> <andrei.rodionov@gmail.com> wrote:
>> 
>> Hello,
>> 
>> I've provisioned an LVM RAID 6 across 5 physical disks. I'm trying to understand the RAID behavior after injecting the failure - removing physical disk /dev/sdc.


Andrei> _______________________________________________
Andrei> linux-lvm mailing list
Andrei> linux-lvm@redhat.com
Andrei> https://listman.redhat.com/mailman/listinfo/linux-lvm
Andrei> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/


_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/