* [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks
@ 2022-04-27 17:11 Alex Lieflander
2022-05-02 0:36 ` John Stoffel
0 siblings, 1 reply; 9+ messages in thread
From: Alex Lieflander @ 2022-04-27 17:11 UTC (permalink / raw)
To: linux-lvm
Hello,
Thank you for your continued work on LVM(2).
I have 4 disks that I’d really like to put into a RAID6. I know about RAID10, but it wouldn’t work well for me for several reasons. Buying another disk would also be a waste of money because I don’t need 3-disks-worth of usable capacity.
I know there was a question regarding this a few years ago, and the consensus was to not natively support that configuration. I can respect that (although I would urge you to reconsider), but I’d still like to do it on my machine.
So far I’ve tried removing the restrictions from the source code and recompiling (I don’t know C, but I’m familiar with general code syntax). I’ve slowly gotten further in the lvconvert process, but there seems to be many similar checks throughout the code.
I’m hoping someone could point me in the right direction towards achieving this goal. If I successfully bypass the user-space tool restrictions, will the rest of LVM likely work with my desired config? Would you be willing to allow the --force option to bypass the restrictions that are not strictly necessary, even at the expense of expected stability? Is there anything else you could suggest?
Thanks,
Alex Lieflander
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks
2022-04-27 17:11 [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks Alex Lieflander
@ 2022-05-02 0:36 ` John Stoffel
2022-05-02 19:22 ` Alex Lieflander
0 siblings, 1 reply; 9+ messages in thread
From: John Stoffel @ 2022-05-02 0:36 UTC (permalink / raw)
To: LVM general discussion and development
Alex> I have 4 disks that I’d really like to put into a RAID6. I know
Alex> about RAID10, but it wouldn’t work well for me for several
Alex> reasons.
Can you explain those reasons? In general, RAID10 gives you only 50%
capacity, but much improved performance over RAID5/6 in terms of
read/write performance.
But if you want to be able to handle the failure of any two disks in
your RAID6, then I can understand your decision.
Alex> Buying another disk would also be a waste of money because I
Alex> don’t need 3-disks-worth of usable capacity.
That's fair.
Alex> I know there was a question regarding this a few years ago, and
Alex> the consensus was to not natively support that configuration. I
Alex> can respect that (although I would urge you to reconsider), but
Alex> I’d still like to do it on my machine.
I would instead build your RAID6 using MD, and then layer LVM on top
of it. It works, it's solid and it runs really well.
Alex> So far I’ve tried removing the restrictions from the source code
Alex> and recompiling (I don’t know C, but I’m familiar with general
Alex> code syntax). I’ve slowly gotten further in the lvconvert
Alex> process, but there seems to be many similar checks throughout
Alex> the code.
If you don't know the code, then you're not going to get working RAID6
up and running any time soon.
Alex> I’m hoping someone could point me in the right direction towards
Alex> achieving this goal. If I successfully bypass the user-space
Alex> tool restrictions, will the rest of LVM likely work with my
Alex> desired config? Would you be willing to allow the --force option
Alex> to bypass the restrictions that are not strictly necessary, even
Alex> at the expense of expected stability? Is there anything else you
Alex> could suggest?
I really can only suggest you setup RAID6 using the MD raid tools
(mdadm) and then create your LVM PVs, VGs and LVs on top of that. It
really works well.
Yes, you now need to have another tool to manage another layer, but
since the MD system is well tested, reliable and just works, then I
would go with it as the base.
If you can copy all your data onto a single disk, then you could
simple create a RAID5 on your other three disks, create a PV on the
/dev/md0, then add the PV into your VG and do a lvmove to copy your
data onto the RAID5, all without downtime.
Once it's moved over, you remove the final disk, then grow the MD
array from RAID5 to RAID6, again online.
The details aren't hard, but you do have to be careful.
John
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks
2022-05-02 0:36 ` John Stoffel
@ 2022-05-02 19:22 ` Alex Lieflander
2022-05-02 21:54 ` John Stoffel
0 siblings, 1 reply; 9+ messages in thread
From: Alex Lieflander @ 2022-05-02 19:22 UTC (permalink / raw)
To: LVM general discussion and development
Thanks for the response.
John> Alex> I have 4 disks that I’d really like to put into a RAID6. I know about RAID10, but it wouldn’t work well for me for several reasons.
John> Can you explain those reasons? In general, RAID10 gives you only 50% capacity, but much improved performance over RAID5/6 in terms of read/write performance.
John> But if you want to be able to handle the failure of any two disks in your RAID6, then I can understand your decision.
1. Resilience to any two disk failures/inconsistencies
2. Ability to safely and easily add single disks in the future† (Having 5 disks with my desired resiliency would require RAID6)
3. Ability to safely and easily switch between 1-disk-resiliency and 2-disk-resiliency in the future† (Going from a clean raid6_ls_6 to a clean raid5_ls is extremely easy by comparison)
†All conversions need to be online. With RAID10 I’d need to go from RAID10 -> RAID0 -> RAID5 (-> RAID6). This process is both lengthy and vulnerable to any single disk failure during part of the conversion.
John> Alex> Buying another disk would also be a waste of money because I don’t need 3-disks-worth of usable capacity.
John> That's fair.
John> Alex> I know there was a question regarding this a few years ago, and the consensus was to not natively support that configuration. I can respect that (although I would urge you to reconsider), but I’d still like to do it on my machine.
John> I would instead build your RAID6 using MD, and then layer LVM on top of it. It works, it's solid and it runs really well.
John> Alex> So far I’ve tried removing the restrictions from the source code and recompiling (I don’t know C, but I’m familiar with general code syntax). I’ve slowly gotten further in the lvconvert process, but there seems to be many similar checks throughout the code.
John> If you don't know the code, then you're not going to get working RAID6 up and running any time soon.
John> Alex> I’m hoping someone could point me in the right direction towards achieving this goal. If I successfully bypass the user-space tool restrictions, will the rest of LVM likely work with my desired config? Would you be willing to allow the --force option to bypass the restrictions that are not strictly necessary, even at the expense of expected stability? Is there anything else you could suggest?
John> I really can only suggest you setup RAID6 using the MD raid tools (mdadm) and then create your LVM PVs, VGs and LVs on top of that. It really works well.
John> Yes, you now need to have another tool to manage another layer, but since the MD system is well tested, reliable and just works, then I would go with it as the base.
I actually used to use MDADM with LVM on top. I switched to pure LVM for simplicity and per-disk host-managed integrity-checking. I don’t know if MDADM has since gained the ability to correct single-disk inconsistencies, but without per-disk integrity-checking it would be technically impossible to do this if 1 disk had already failed.
Alex Lieflander
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks
2022-05-02 19:22 ` Alex Lieflander
@ 2022-05-02 21:54 ` John Stoffel
2022-05-07 3:33 ` Alex Lieflander
0 siblings, 1 reply; 9+ messages in thread
From: John Stoffel @ 2022-05-02 21:54 UTC (permalink / raw)
To: LVM general discussion and development
>>>>> "Alex" == Alex Lieflander <atlief@icloud.com> writes:
Alex> I actually used to use MDADM with LVM on top. I switched to pure
Alex> LVM for simplicity and per-disk host-managed
Alex> integrity-checking. I don’t know if MDADM has since gained the
Alex> ability to correct single-disk inconsistencies, but without
Alex> per-disk integrity-checking it would be technically impossible
Alex> to do this if 1 disk had already failed.
Can you tell me what you mean by "per-disk host-managed
integrity-checking"? Are you running dm-integrity in your setup? Can
you post the output of:
lsblk
so we understand better what you have? But in any case, good luck
with getting RAID6 solid and working as a pure LVM setup. I don't
think you're going to make it happen any time soon, and I honestly
prefer to have seperate layers for my setup so that each layer does
it's own think and does it well.
John
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks
2022-05-02 21:54 ` John Stoffel
@ 2022-05-07 3:33 ` Alex Lieflander
2022-05-07 20:41 ` Stuart D Gathman
0 siblings, 1 reply; 9+ messages in thread
From: Alex Lieflander @ 2022-05-07 3:33 UTC (permalink / raw)
To: LVM general discussion and development
[-- Attachment #1: Type: text/plain, Size: 1021 bytes --]
John> Can you tell me what you mean by "per-disk host-managed integrity-checking"? Are you running dm-integrity in your setup?
Yes, that’s what I meant. I know that I could manually use IntegritySetup for each RAID member, but that seems very unappealing.
John> Can you post the output of:
John> lsblk
John> so we understand better what you have?
Please see the attached text files, although it’s a bit convoluted. “Postulate” devices are part of the RAID that I’d like to transform. If anyone has any recommendations for my setup, I’d be interested to hear them. Maybe I’ll switch to LVMCache someday. :P
John> But in any case, good luck with getting RAID6 solid and working as a pure LVM setup. I don’t think you're going to make it happen any time soon, and I honestly prefer to have seperate layers for my setup so that each layer does it's own think and does it well.
Thanks. I really don’t want to give up the DM-Integrity management. Less complexity is just a bonus.
Alex
[-- Attachment #2: lsblk.txt --]
[-- Type: text/plain, Size: 10514 bytes --]
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 1.8T 0 disk
└─sda4 8:4 0 1.8T 0 part
└─Postulate_Double-1 254:2 0 3.6T 0 lvm
sdb 8:16 0 3.6T 0 disk
└─sdb4 8:20 0 3.6T 0 part
├─Postulate_Outer-Primary_rmeta_0 254:5 0 1M 0 lvm
│ └─Postulate_Outer-Primary 254:25 0 7.2T 0 lvm
│ └─bcache1 253:128 0 7.2T 0 disk
│ └─main_crypt_5 254:26 0 7.2T 0 crypt
│ ├─Postulate_Inner-Primary 254:27 0 5T 0 lvm /
│ └─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP]
├─Postulate_Outer-Primary_rimage_0_imeta 254:6 0 29.6G 0 lvm
│ └─Postulate_Outer-Primary_rimage_0 254:10 0 3.6T 0 lvm
│ └─Postulate_Outer-Primary 254:25 0 7.2T 0 lvm
│ └─bcache1 253:128 0 7.2T 0 disk
│ └─main_crypt_5 254:26 0 7.2T 0 crypt
│ ├─Postulate_Inner-Primary 254:27 0 5T 0 lvm /
│ └─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP]
└─Postulate_Outer-Primary_rimage_0_iorig 254:9 0 3.6T 0 lvm
└─Postulate_Outer-Primary_rimage_0 254:10 0 3.6T 0 lvm
└─Postulate_Outer-Primary 254:25 0 7.2T 0 lvm
└─bcache1 253:128 0 7.2T 0 disk
└─main_crypt_5 254:26 0 7.2T 0 crypt
├─Postulate_Inner-Primary 254:27 0 5T 0 lvm /
└─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP]
sdc 8:32 0 3.6T 0 disk
├─sdc1 8:33 0 10M 0 part
├─sdc2 8:34 0 200M 0 part
├─sdc3 8:35 0 809M 0 part
└─sdc4 8:36 0 3.6T 0 part
├─Postulate_Outer-Primary_rmeta_2 254:20 0 1M 0 lvm
│ └─Postulate_Outer-Primary 254:25 0 7.2T 0 lvm
│ └─bcache1 253:128 0 7.2T 0 disk
│ └─main_crypt_5 254:26 0 7.2T 0 crypt
│ ├─Postulate_Inner-Primary 254:27 0 5T 0 lvm /
│ └─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP]
├─Postulate_Outer-Primary_rimage_2_imeta 254:21 0 29.6G 0 lvm
│ └─Postulate_Outer-Primary_rimage_2 254:23 0 3.6T 0 lvm
│ └─Postulate_Outer-Primary 254:25 0 7.2T 0 lvm
│ └─bcache1 253:128 0 7.2T 0 disk
│ └─main_crypt_5 254:26 0 7.2T 0 crypt
│ ├─Postulate_Inner-Primary 254:27 0 5T 0 lvm /
│ └─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP]
└─Postulate_Outer-Primary_rimage_2_iorig 254:22 0 3.6T 0 lvm
└─Postulate_Outer-Primary_rimage_2 254:23 0 3.6T 0 lvm
└─Postulate_Outer-Primary 254:25 0 7.2T 0 lvm
└─bcache1 253:128 0 7.2T 0 disk
└─main_crypt_5 254:26 0 7.2T 0 crypt
├─Postulate_Inner-Primary 254:27 0 5T 0 lvm /
└─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP]
sdd 8:48 0 3.6T 0 disk
└─sdd4 8:52 0 3.6T 0 part
├─Postulate_Outer-Primary_rmeta_1 254:12 0 1M 0 lvm
│ └─Postulate_Outer-Primary 254:25 0 7.2T 0 lvm
│ └─bcache1 253:128 0 7.2T 0 disk
│ └─main_crypt_5 254:26 0 7.2T 0 crypt
│ ├─Postulate_Inner-Primary 254:27 0 5T 0 lvm /
│ └─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP]
├─Postulate_Outer-Primary_rimage_1_imeta 254:13 0 29.6G 0 lvm
│ └─Postulate_Outer-Primary_rimage_1 254:15 0 3.6T 0 lvm
│ └─Postulate_Outer-Primary 254:25 0 7.2T 0 lvm
│ └─bcache1 253:128 0 7.2T 0 disk
│ └─main_crypt_5 254:26 0 7.2T 0 crypt
│ ├─Postulate_Inner-Primary 254:27 0 5T 0 lvm /
│ └─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP]
└─Postulate_Outer-Primary_rimage_1_iorig 254:14 0 3.6T 0 lvm
└─Postulate_Outer-Primary_rimage_1 254:15 0 3.6T 0 lvm
└─Postulate_Outer-Primary 254:25 0 7.2T 0 lvm
└─bcache1 253:128 0 7.2T 0 disk
└─main_crypt_5 254:26 0 7.2T 0 crypt
├─Postulate_Inner-Primary 254:27 0 5T 0 lvm /
└─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP]
sde 8:64 0 4.5T 0 disk
└─sde4 8:68 0 4.5T 0 part
├─Axiom_Outer-Primary_rmeta_0 254:4 0 4M 0 lvm
│ └─Axiom_Outer-Primary 254:24 0 4.5T 0 lvm
│ └─bcache0 253:0 0 4.5T 0 disk
│ └─main_crypt_2 254:29 0 4.5T 0 crypt
│ └─Axiom_Inner-New 254:30 0 3.6T 0 lvm
├─Axiom_Outer-Primary_rimage_0_imeta 254:7 0 36.8G 0 lvm
│ └─Axiom_Outer-Primary_rimage_0 254:11 0 4.5T 0 lvm
│ └─Axiom_Outer-Primary 254:24 0 4.5T 0 lvm
│ └─bcache0 253:0 0 4.5T 0 disk
│ └─main_crypt_2 254:29 0 4.5T 0 crypt
│ └─Axiom_Inner-New 254:30 0 3.6T 0 lvm
└─Axiom_Outer-Primary_rimage_0_iorig 254:8 0 4.5T 0 lvm
└─Axiom_Outer-Primary_rimage_0 254:11 0 4.5T 0 lvm
└─Axiom_Outer-Primary 254:24 0 4.5T 0 lvm
└─bcache0 253:0 0 4.5T 0 disk
└─main_crypt_2 254:29 0 4.5T 0 crypt
└─Axiom_Inner-New 254:30 0 3.6T 0 lvm
sdf 8:80 0 931.5G 0 disk
└─sdf4 8:84 0 930.5G 0 part
sdg 8:96 0 931.5G 0 disk
└─sdg4 8:100 0 930.5G 0 part
sdh 8:112 0 5.5T 0 disk
└─sdh4 8:116 0 5.5T 0 part
├─Postulate_Backup-Primary 254:0 0 5T 0 lvm
│ └─backup_crypt_3 254:31 0 5T 0 crypt /mnt/bk/3
└─Postulate_Backup-Old 254:1 0 128G 0 lvm
sdi 8:128 0 1.8T 0 disk
└─sdi4 8:132 0 1.8T 0 part
└─Postulate_Double-1 254:2 0 3.6T 0 lvm
sdj 8:144 0 4.5T 0 disk
└─sdj4 8:148 0 4.5T 0 part
├─Axiom_Outer-Primary_rmeta_1 254:16 0 4M 0 lvm
│ └─Axiom_Outer-Primary 254:24 0 4.5T 0 lvm
│ └─bcache0 253:0 0 4.5T 0 disk
│ └─main_crypt_2 254:29 0 4.5T 0 crypt
│ └─Axiom_Inner-New 254:30 0 3.6T 0 lvm
├─Axiom_Outer-Primary_rimage_1_imeta 254:17 0 36.8G 0 lvm
│ └─Axiom_Outer-Primary_rimage_1 254:19 0 4.5T 0 lvm
│ └─Axiom_Outer-Primary 254:24 0 4.5T 0 lvm
│ └─bcache0 253:0 0 4.5T 0 disk
│ └─main_crypt_2 254:29 0 4.5T 0 crypt
│ └─Axiom_Inner-New 254:30 0 3.6T 0 lvm
└─Axiom_Outer-Primary_rimage_1_iorig 254:18 0 4.5T 0 lvm
└─Axiom_Outer-Primary_rimage_1 254:19 0 4.5T 0 lvm
└─Axiom_Outer-Primary 254:24 0 4.5T 0 lvm
└─bcache0 253:0 0 4.5T 0 disk
└─main_crypt_2 254:29 0 4.5T 0 crypt
└─Axiom_Inner-New 254:30 0 3.6T 0 lvm
sdk 8:160 0 223.6G 0 disk
└─sdk4 8:164 0 189.6G 0 part
nvme0n1 259:0 0 931.5G 0 disk
├─nvme0n1p2 259:1 0 200M 0 part /boot/efi
├─nvme0n1p3 259:2 0 814M 0 part /boot
├─nvme0n1p4 259:3 0 832G 0 part
│ └─Postulate_Outer-Cache 254:3 0 800G 0 lvm
│ └─bcache1 253:128 0 7.2T 0 disk
│ └─main_crypt_5 254:26 0 7.2T 0 crypt
│ ├─Postulate_Inner-Primary 254:27 0 5T 0 lvm /
│ └─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP]
└─nvme0n1p8 259:4 0 54G 0 part
└─recovery_crypt 254:32 0 54G 0 crypt /mnt/recovery
[-- Attachment #3: lsblk_m.txt --]
[-- Type: text/plain, Size: 6293 bytes --]
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 1.8T 0 disk
┌┈▶ └─sda4 8:4 0 1.8T 0 part
┆ sdb 8:16 0 3.6T 0 disk
┆ └─sdb4 8:20 0 3.6T 0 part
┌┈▶┆ ├─Postulate_Outer-Primary_rmeta_0 254:5 0 1M 0 lvm
┌┈▶┆ ┆ ├─Postulate_Outer-Primary_rimage_0_imeta 254:6 0 29.6G 0 lvm
└┬▶┆ ┆ └─Postulate_Outer-Primary_rimage_0_iorig 254:9 0 3.6T 0 lvm
└┈├┈▶┆┈┈┈Postulate_Outer-Primary_rimage_0 254:10 0 3.6T 0 lvm
┆ ┆ sdc 8:32 0 3.6T 0 disk
┆ ┆ ├─sdc1 8:33 0 10M 0 part
┆ ┆ ├─sdc2 8:34 0 200M 0 part
┆ ┆ ├─sdc3 8:35 0 809M 0 part
┆ ┆ └─sdc4 8:36 0 3.6T 0 part
├┈▶┆ ├─Postulate_Outer-Primary_rmeta_2 254:20 0 1M 0 lvm
┌┈▶┆ ┆ ├─Postulate_Outer-Primary_rimage_2_imeta 254:21 0 29.6G 0 lvm
└┬▶┆ ┆ └─Postulate_Outer-Primary_rimage_2_iorig 254:22 0 3.6T 0 lvm
└┈├┈▶┆┈┈┈Postulate_Outer-Primary_rimage_2 254:23 0 3.6T 0 lvm
┆ ┆ sdd 8:48 0 3.6T 0 disk
┆ ┆ └─sdd4 8:52 0 3.6T 0 part
├┈▶┆ ├─Postulate_Outer-Primary_rmeta_1 254:12 0 1M 0 lvm
┌┈▶┆ ┆ ├─Postulate_Outer-Primary_rimage_1_imeta 254:13 0 29.6G 0 lvm
└┬▶┆ ┆ └─Postulate_Outer-Primary_rimage_1_iorig 254:14 0 3.6T 0 lvm
└┈└┬▶┆┈┈┈Postulate_Outer-Primary_rimage_1 254:15 0 3.6T 0 lvm
┌┈▶ └┈┆┈┈┈Postulate_Outer-Primary 254:25 0 7.2T 0 lvm
┆ ┆ sde 8:64 0 4.5T 0 disk
┆ ┆ └─sde4 8:68 0 4.5T 0 part
┆ ┌┈▶┆ ├─Axiom_Outer-Primary_rmeta_0 254:4 0 4M 0 lvm
┌┈▶┆ ┆ ┆ ├─Axiom_Outer-Primary_rimage_0_imeta 254:7 0 36.8G 0 lvm
└┬▶┆ ┆ ┆ └─Axiom_Outer-Primary_rimage_0_iorig 254:8 0 4.5T 0 lvm
└┈┆┈┈├┈▶┆┈┈┈Axiom_Outer-Primary_rimage_0 254:11 0 4.5T 0 lvm
┆ ┆ ┆ sdf 8:80 0 931.5G 0 disk
┆ ┆ ┆ └─sdf4 8:84 0 930.5G 0 part
┆ ┆ ┆ sdg 8:96 0 931.5G 0 disk
┆ ┆ ┆ └─sdg4 8:100 0 930.5G 0 part
┆ ┆ ┆ sdh 8:112 0 5.5T 0 disk
┆ ┆ ┆ └─sdh4 8:116 0 5.5T 0 part
┆ ┆ ┆ ├─Postulate_Backup-Primary 254:0 0 5T 0 lvm
┆ ┆ ┆ │ └─backup_crypt_3 254:31 0 5T 0 crypt /mnt/bk/3
┆ ┆ ┆ └─Postulate_Backup-Old 254:1 0 128G 0 lvm
┆ ┆ ┆ sdi 8:128 0 1.8T 0 disk
┆ ┆ └┬▶ └─sdi4 8:132 0 1.8T 0 part
┆ ┆ └┈┈Postulate_Double-1 254:2 0 3.6T 0 lvm
┆ ┆ sdj 8:144 0 4.5T 0 disk
┆ ┆ └─sdj4 8:148 0 4.5T 0 part
┆ ├┈▶ ├─Axiom_Outer-Primary_rmeta_1 254:16 0 4M 0 lvm
┆ ┆ ┌┈▶ ├─Axiom_Outer-Primary_rimage_1_imeta 254:17 0 36.8G 0 lvm
┆ ┆ └┬▶ └─Axiom_Outer-Primary_rimage_1_iorig 254:18 0 4.5T 0 lvm
┆ └┬▶ └┈┈Axiom_Outer-Primary_rimage_1 254:19 0 4.5T 0 lvm
┆ └┈┈┈┈┈Axiom_Outer-Primary 254:24 0 4.5T 0 lvm
┆ └─bcache0 253:0 0 4.5T 0 disk
┆ └─main_crypt_2 254:29 0 4.5T 0 crypt
┆ └─Axiom_Inner-New 254:30 0 3.6T 0 lvm
┆ sdk 8:160 0 223.6G 0 disk
┆ └─sdk4 8:164 0 189.6G 0 part
┆ nvme0n1 259:0 0 931.5G 0 disk
┆ ├─nvme0n1p2 259:1 0 200M 0 part /boot/efi
┆ ├─nvme0n1p3 259:2 0 814M 0 part /boot
┆ ├─nvme0n1p4 259:3 0 832G 0 part
└┬▶ │ └─Postulate_Outer-Cache 254:3 0 800G 0 lvm
┆ └─nvme0n1p8 259:4 0 54G 0 part
┆ └─recovery_crypt 254:32 0 54G 0 crypt /mnt/recovery
└┈┈┈┈┈┈┈┈bcache1 253:128 0 7.2T 0 disk
└─main_crypt_5 254:26 0 7.2T 0 crypt
├─Postulate_Inner-Primary 254:27 0 5T 0 lvm /
└─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP]
[-- Attachment #4: Type: text/plain, Size: 202 bytes --]
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks
2022-05-07 3:33 ` Alex Lieflander
@ 2022-05-07 20:41 ` Stuart D Gathman
2022-05-07 23:14 ` Alex Lieflander
0 siblings, 1 reply; 9+ messages in thread
From: Stuart D Gathman @ 2022-05-07 20:41 UTC (permalink / raw)
To: LVM general discussion and development
[-- Attachment #1: Type: text/plain, Size: 552 bytes --]
On Fri, 6 May 2022, Alex Lieflander wrote:
> Thanks. I really don’t want to give up the DM-Integrity management. Less complexity is just a bonus.
What are you trying to get out of RAID6? If redundancy and integrity
are already managed at another layer, then just use RAID0 for striping.
I like to use RAID10 for mirror + striping, but I understand parity disks
give redundancy without halving capacity. Parity means RMW cycles of
largish blocks, whereas straight mirroring (RAID1, RAID10) can write
single sectors without a RMW cycle.
[-- Attachment #2: Type: text/plain, Size: 202 bytes --]
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks
2022-05-07 20:41 ` Stuart D Gathman
@ 2022-05-07 23:14 ` Alex Lieflander
2022-05-08 13:25 ` Stuart D Gathman
2022-05-09 0:18 ` John Stoffel
0 siblings, 2 replies; 9+ messages in thread
From: Alex Lieflander @ 2022-05-07 23:14 UTC (permalink / raw)
To: LVM general discussion and development
> On May 7, 2022, at 4:41 PM, Stuart D Gathman wrote:
>
>> On Fri, 6 May 2022, Alex Lieflander wrote:
>>
>> Thanks. I really don’t want to give up the DM-Integrity management. Less complexity is just a bonus.
>
> What are you trying to get out of RAID6? If redundancy and integrity
> are already managed at another layer, then just use RAID0 for striping.
>
> I like to use RAID10 for mirror + striping, but I understand parity disks give redundancy without halving capacity. Parity means RMW cycles of
> largish blocks, whereas straight mirroring (RAID1, RAID10) can write
> single sectors without a RMW cycle.
I don’t trust the hardware I’m running on very much, but it’s all I have to work with at the moment; it’s important that the array is resilient to *any* (and multiple) single chunk corruptions because such corruptions are likely to happen in the future.
For the last several months I’ve periodically been seeing (DM-Integrity) checksum mismatch warnings at various locations on all of my disks. I stopped using a few SATA ports that were explicitly throwing SATA errors, but I suspect that the remaining connections are unpredictably (albeit infrequently) corrupting data in ways that are more difficult to detect.
I’ve tried to “check” and “repair” my array on multiple kernel versions and live recovery USB sticks, but the “check" always seems to freeze and all subsequent IO to the array hangs until reboot; at the moment, a chunk is only ever made consistent when its data is overwritten, so it needs to survive periodic, random corruption for as long as possible.
I also have a disk that infrequently fails to read from a particular area, but the rest of the disk is fine. I wouldn’t trust that disk with valuable data, but it seems like a perfect candidate to hold additional parity (raid6_ls_6) that I hopefully never need.
Alex
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks
2022-05-07 23:14 ` Alex Lieflander
@ 2022-05-08 13:25 ` Stuart D Gathman
2022-05-09 0:18 ` John Stoffel
1 sibling, 0 replies; 9+ messages in thread
From: Stuart D Gathman @ 2022-05-08 13:25 UTC (permalink / raw)
To: LVM general discussion and development
[-- Attachment #1: Type: text/plain, Size: 947 bytes --]
On Sat, 7 May 2022, Alex Lieflander wrote:
> I don’t trust the hardware I’m running on very much, but it’s all I have to work with at the moment; it’s important that the array is resilient to *any* (and multiple) single chunk corruptions because such corruptions are likely to happen in the future.
>
> For the last several months I’ve periodically been seeing (DM-Integrity) checksum mismatch warnings at various locations on all of my disks. I stopped using a few SATA ports that were explicitly throwing SATA errors, but I suspect that the remaining connections are unpredictably (albeit infrequently) corrupting data in ways that are more difficult to detect.
Sounds like a *great* test bed for software data integrity tools. Don't
throw that system away when you get a more reliable one!
That sounds like a situation that btrfs with multiple copies could
handle. Use a beefier checksum than the default crc-32 also.
[-- Attachment #2: Type: text/plain, Size: 202 bytes --]
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks
2022-05-07 23:14 ` Alex Lieflander
2022-05-08 13:25 ` Stuart D Gathman
@ 2022-05-09 0:18 ` John Stoffel
1 sibling, 0 replies; 9+ messages in thread
From: John Stoffel @ 2022-05-09 0:18 UTC (permalink / raw)
To: LVM general discussion and development
>>>>> "Alex" == Alex Lieflander <atlief@icloud.com> writes:
>> On May 7, 2022, at 4:41 PM, Stuart D Gathman wrote:
>>
>>> On Fri, 6 May 2022, Alex Lieflander wrote:
>>>
>>> Thanks. I really don’t want to give up the DM-Integrity management. Less complexity is just a bonus.
>>
>> What are you trying to get out of RAID6? If redundancy and integrity
>> are already managed at another layer, then just use RAID0 for striping.
>>
>> I like to use RAID10 for mirror + striping, but I understand parity disks give redundancy without halving capacity. Parity means RMW cycles of
>> largish blocks, whereas straight mirroring (RAID1, RAID10) can write
>> single sectors without a RMW cycle.
Alex> I don’t trust the hardware I’m running on very much, but it’s
Alex> all I have to work with at the moment; it’s important that the
Alex> array is resilient to *any* (and multiple) single chunk
Alex> corruptions because such corruptions are likely to happen in the
Alex> future.
Ouch! I hope you have good backups somewhere, because I suspect
you're doing to suffer a complete failure at some point.
Alex> For the last several months I’ve periodically been seeing
Alex> (DM-Integrity) checksum mismatch warnings at various locations
Alex> on all of my disks. I stopped using a few SATA ports that were
Alex> explicitly throwing SATA errors, but I suspect that the
Alex> remaining connections are unpredictably (albeit infrequently)
Alex> corrupting data in ways that are more difficult to detect.
This is interesting. And worrisome, because I would not expect moving
from one SATA port to another to cure problems, unless it was A)
moving to a different controller, or B) you changed/reseated the SATA
cable.
But I also wonder about your power supply and what it's rated for.
You might just be hitting the ragged edge of what it can supply, and
so you're running into problems with voltage dropping just enough to
make things slightly flaky.
Alex> I’ve tried to “check” and “repair” my array on multiple kernel
Alex> versions and live recovery USB sticks, but the “check" always
Alex> seems to freeze and all subsequent IO to the array hangs until
Alex> reboot; at the moment, a chunk is only ever made consistent when
Alex> its data is overwritten, so it needs to survive periodic, random
Alex> corruption for as long as possible.
This is also a warning to my that maybe you have power supply issues.
Can you give a summary of your hardware configuration and model
numbers? If you're running a smallish power supply, maybe look for a
replacement which can get you more power. Go from a 430W one to 600W,
or 500W to 750W and see if that makes a difference.
Looking at your data from before, I see you have 12 disks on the
system, 11 spinning disks and one nvme device. So I *really* suspect
you have an overloaded power supply.
Are you also using a disk controller? And which version of linux?
Alex> I also have a disk that infrequently fails to read from a
Alex> particular area, but the rest of the disk is fine. I wouldn’t
Alex> trust that disk with valuable data, but it seems like a perfect
Alex> candidate to hold additional parity (raid6_ls_6) that I
Alex> hopefully never need.
This is not how RAID6 parity works. The entire disk (or partition) is
used to write data and/or parity. It's RAID4 which dedicates a single
disk to parity duties. So thinking that a known flaky disk will be ok
for just parity use isn't really a good idea.
I'd also look at the output of 'smartctl --all /dev/sd<letter>' for
all your disks and see what the numbers say. But honestly, it sounds
like you have some serious hardware issues which you're trying to
paper over with DM-Integrity and RAID5. And I suspect it will all end
in tears sooner or later.
You do have backups of your data, right? Even onto a single new 10tb
disk that's now connected to the system all the time?
Good luck,
John
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2022-05-09 0:19 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-04-27 17:11 [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks Alex Lieflander
2022-05-02 0:36 ` John Stoffel
2022-05-02 19:22 ` Alex Lieflander
2022-05-02 21:54 ` John Stoffel
2022-05-07 3:33 ` Alex Lieflander
2022-05-07 20:41 ` Stuart D Gathman
2022-05-07 23:14 ` Alex Lieflander
2022-05-08 13:25 ` Stuart D Gathman
2022-05-09 0:18 ` John Stoffel
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).