* [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks @ 2022-04-27 17:11 Alex Lieflander 2022-05-02 0:36 ` John Stoffel 0 siblings, 1 reply; 9+ messages in thread From: Alex Lieflander @ 2022-04-27 17:11 UTC (permalink / raw) To: linux-lvm Hello, Thank you for your continued work on LVM(2). I have 4 disks that I’d really like to put into a RAID6. I know about RAID10, but it wouldn’t work well for me for several reasons. Buying another disk would also be a waste of money because I don’t need 3-disks-worth of usable capacity. I know there was a question regarding this a few years ago, and the consensus was to not natively support that configuration. I can respect that (although I would urge you to reconsider), but I’d still like to do it on my machine. So far I’ve tried removing the restrictions from the source code and recompiling (I don’t know C, but I’m familiar with general code syntax). I’ve slowly gotten further in the lvconvert process, but there seems to be many similar checks throughout the code. I’m hoping someone could point me in the right direction towards achieving this goal. If I successfully bypass the user-space tool restrictions, will the rest of LVM likely work with my desired config? Would you be willing to allow the --force option to bypass the restrictions that are not strictly necessary, even at the expense of expected stability? Is there anything else you could suggest? Thanks, Alex Lieflander _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks 2022-04-27 17:11 [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks Alex Lieflander @ 2022-05-02 0:36 ` John Stoffel 2022-05-02 19:22 ` Alex Lieflander 0 siblings, 1 reply; 9+ messages in thread From: John Stoffel @ 2022-05-02 0:36 UTC (permalink / raw) To: LVM general discussion and development Alex> I have 4 disks that I’d really like to put into a RAID6. I know Alex> about RAID10, but it wouldn’t work well for me for several Alex> reasons. Can you explain those reasons? In general, RAID10 gives you only 50% capacity, but much improved performance over RAID5/6 in terms of read/write performance. But if you want to be able to handle the failure of any two disks in your RAID6, then I can understand your decision. Alex> Buying another disk would also be a waste of money because I Alex> don’t need 3-disks-worth of usable capacity. That's fair. Alex> I know there was a question regarding this a few years ago, and Alex> the consensus was to not natively support that configuration. I Alex> can respect that (although I would urge you to reconsider), but Alex> I’d still like to do it on my machine. I would instead build your RAID6 using MD, and then layer LVM on top of it. It works, it's solid and it runs really well. Alex> So far I’ve tried removing the restrictions from the source code Alex> and recompiling (I don’t know C, but I’m familiar with general Alex> code syntax). I’ve slowly gotten further in the lvconvert Alex> process, but there seems to be many similar checks throughout Alex> the code. If you don't know the code, then you're not going to get working RAID6 up and running any time soon. Alex> I’m hoping someone could point me in the right direction towards Alex> achieving this goal. If I successfully bypass the user-space Alex> tool restrictions, will the rest of LVM likely work with my Alex> desired config? Would you be willing to allow the --force option Alex> to bypass the restrictions that are not strictly necessary, even Alex> at the expense of expected stability? Is there anything else you Alex> could suggest? I really can only suggest you setup RAID6 using the MD raid tools (mdadm) and then create your LVM PVs, VGs and LVs on top of that. It really works well. Yes, you now need to have another tool to manage another layer, but since the MD system is well tested, reliable and just works, then I would go with it as the base. If you can copy all your data onto a single disk, then you could simple create a RAID5 on your other three disks, create a PV on the /dev/md0, then add the PV into your VG and do a lvmove to copy your data onto the RAID5, all without downtime. Once it's moved over, you remove the final disk, then grow the MD array from RAID5 to RAID6, again online. The details aren't hard, but you do have to be careful. John _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks 2022-05-02 0:36 ` John Stoffel @ 2022-05-02 19:22 ` Alex Lieflander 2022-05-02 21:54 ` John Stoffel 0 siblings, 1 reply; 9+ messages in thread From: Alex Lieflander @ 2022-05-02 19:22 UTC (permalink / raw) To: LVM general discussion and development Thanks for the response. John> Alex> I have 4 disks that I’d really like to put into a RAID6. I know about RAID10, but it wouldn’t work well for me for several reasons. John> Can you explain those reasons? In general, RAID10 gives you only 50% capacity, but much improved performance over RAID5/6 in terms of read/write performance. John> But if you want to be able to handle the failure of any two disks in your RAID6, then I can understand your decision. 1. Resilience to any two disk failures/inconsistencies 2. Ability to safely and easily add single disks in the future† (Having 5 disks with my desired resiliency would require RAID6) 3. Ability to safely and easily switch between 1-disk-resiliency and 2-disk-resiliency in the future† (Going from a clean raid6_ls_6 to a clean raid5_ls is extremely easy by comparison) †All conversions need to be online. With RAID10 I’d need to go from RAID10 -> RAID0 -> RAID5 (-> RAID6). This process is both lengthy and vulnerable to any single disk failure during part of the conversion. John> Alex> Buying another disk would also be a waste of money because I don’t need 3-disks-worth of usable capacity. John> That's fair. John> Alex> I know there was a question regarding this a few years ago, and the consensus was to not natively support that configuration. I can respect that (although I would urge you to reconsider), but I’d still like to do it on my machine. John> I would instead build your RAID6 using MD, and then layer LVM on top of it. It works, it's solid and it runs really well. John> Alex> So far I’ve tried removing the restrictions from the source code and recompiling (I don’t know C, but I’m familiar with general code syntax). I’ve slowly gotten further in the lvconvert process, but there seems to be many similar checks throughout the code. John> If you don't know the code, then you're not going to get working RAID6 up and running any time soon. John> Alex> I’m hoping someone could point me in the right direction towards achieving this goal. If I successfully bypass the user-space tool restrictions, will the rest of LVM likely work with my desired config? Would you be willing to allow the --force option to bypass the restrictions that are not strictly necessary, even at the expense of expected stability? Is there anything else you could suggest? John> I really can only suggest you setup RAID6 using the MD raid tools (mdadm) and then create your LVM PVs, VGs and LVs on top of that. It really works well. John> Yes, you now need to have another tool to manage another layer, but since the MD system is well tested, reliable and just works, then I would go with it as the base. I actually used to use MDADM with LVM on top. I switched to pure LVM for simplicity and per-disk host-managed integrity-checking. I don’t know if MDADM has since gained the ability to correct single-disk inconsistencies, but without per-disk integrity-checking it would be technically impossible to do this if 1 disk had already failed. Alex Lieflander _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks 2022-05-02 19:22 ` Alex Lieflander @ 2022-05-02 21:54 ` John Stoffel 2022-05-07 3:33 ` Alex Lieflander 0 siblings, 1 reply; 9+ messages in thread From: John Stoffel @ 2022-05-02 21:54 UTC (permalink / raw) To: LVM general discussion and development >>>>> "Alex" == Alex Lieflander <atlief@icloud.com> writes: Alex> I actually used to use MDADM with LVM on top. I switched to pure Alex> LVM for simplicity and per-disk host-managed Alex> integrity-checking. I don’t know if MDADM has since gained the Alex> ability to correct single-disk inconsistencies, but without Alex> per-disk integrity-checking it would be technically impossible Alex> to do this if 1 disk had already failed. Can you tell me what you mean by "per-disk host-managed integrity-checking"? Are you running dm-integrity in your setup? Can you post the output of: lsblk so we understand better what you have? But in any case, good luck with getting RAID6 solid and working as a pure LVM setup. I don't think you're going to make it happen any time soon, and I honestly prefer to have seperate layers for my setup so that each layer does it's own think and does it well. John _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks 2022-05-02 21:54 ` John Stoffel @ 2022-05-07 3:33 ` Alex Lieflander 2022-05-07 20:41 ` Stuart D Gathman 0 siblings, 1 reply; 9+ messages in thread From: Alex Lieflander @ 2022-05-07 3:33 UTC (permalink / raw) To: LVM general discussion and development [-- Attachment #1: Type: text/plain, Size: 1021 bytes --] John> Can you tell me what you mean by "per-disk host-managed integrity-checking"? Are you running dm-integrity in your setup? Yes, that’s what I meant. I know that I could manually use IntegritySetup for each RAID member, but that seems very unappealing. John> Can you post the output of: John> lsblk John> so we understand better what you have? Please see the attached text files, although it’s a bit convoluted. “Postulate” devices are part of the RAID that I’d like to transform. If anyone has any recommendations for my setup, I’d be interested to hear them. Maybe I’ll switch to LVMCache someday. :P John> But in any case, good luck with getting RAID6 solid and working as a pure LVM setup. I don’t think you're going to make it happen any time soon, and I honestly prefer to have seperate layers for my setup so that each layer does it's own think and does it well. Thanks. I really don’t want to give up the DM-Integrity management. Less complexity is just a bonus. Alex [-- Attachment #2: lsblk.txt --] [-- Type: text/plain, Size: 10514 bytes --] NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 1.8T 0 disk └─sda4 8:4 0 1.8T 0 part └─Postulate_Double-1 254:2 0 3.6T 0 lvm sdb 8:16 0 3.6T 0 disk └─sdb4 8:20 0 3.6T 0 part ├─Postulate_Outer-Primary_rmeta_0 254:5 0 1M 0 lvm │ └─Postulate_Outer-Primary 254:25 0 7.2T 0 lvm │ └─bcache1 253:128 0 7.2T 0 disk │ └─main_crypt_5 254:26 0 7.2T 0 crypt │ ├─Postulate_Inner-Primary 254:27 0 5T 0 lvm / │ └─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP] ├─Postulate_Outer-Primary_rimage_0_imeta 254:6 0 29.6G 0 lvm │ └─Postulate_Outer-Primary_rimage_0 254:10 0 3.6T 0 lvm │ └─Postulate_Outer-Primary 254:25 0 7.2T 0 lvm │ └─bcache1 253:128 0 7.2T 0 disk │ └─main_crypt_5 254:26 0 7.2T 0 crypt │ ├─Postulate_Inner-Primary 254:27 0 5T 0 lvm / │ └─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP] └─Postulate_Outer-Primary_rimage_0_iorig 254:9 0 3.6T 0 lvm └─Postulate_Outer-Primary_rimage_0 254:10 0 3.6T 0 lvm └─Postulate_Outer-Primary 254:25 0 7.2T 0 lvm └─bcache1 253:128 0 7.2T 0 disk └─main_crypt_5 254:26 0 7.2T 0 crypt ├─Postulate_Inner-Primary 254:27 0 5T 0 lvm / └─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP] sdc 8:32 0 3.6T 0 disk ├─sdc1 8:33 0 10M 0 part ├─sdc2 8:34 0 200M 0 part ├─sdc3 8:35 0 809M 0 part └─sdc4 8:36 0 3.6T 0 part ├─Postulate_Outer-Primary_rmeta_2 254:20 0 1M 0 lvm │ └─Postulate_Outer-Primary 254:25 0 7.2T 0 lvm │ └─bcache1 253:128 0 7.2T 0 disk │ └─main_crypt_5 254:26 0 7.2T 0 crypt │ ├─Postulate_Inner-Primary 254:27 0 5T 0 lvm / │ └─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP] ├─Postulate_Outer-Primary_rimage_2_imeta 254:21 0 29.6G 0 lvm │ └─Postulate_Outer-Primary_rimage_2 254:23 0 3.6T 0 lvm │ └─Postulate_Outer-Primary 254:25 0 7.2T 0 lvm │ └─bcache1 253:128 0 7.2T 0 disk │ └─main_crypt_5 254:26 0 7.2T 0 crypt │ ├─Postulate_Inner-Primary 254:27 0 5T 0 lvm / │ └─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP] └─Postulate_Outer-Primary_rimage_2_iorig 254:22 0 3.6T 0 lvm └─Postulate_Outer-Primary_rimage_2 254:23 0 3.6T 0 lvm └─Postulate_Outer-Primary 254:25 0 7.2T 0 lvm └─bcache1 253:128 0 7.2T 0 disk └─main_crypt_5 254:26 0 7.2T 0 crypt ├─Postulate_Inner-Primary 254:27 0 5T 0 lvm / └─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP] sdd 8:48 0 3.6T 0 disk └─sdd4 8:52 0 3.6T 0 part ├─Postulate_Outer-Primary_rmeta_1 254:12 0 1M 0 lvm │ └─Postulate_Outer-Primary 254:25 0 7.2T 0 lvm │ └─bcache1 253:128 0 7.2T 0 disk │ └─main_crypt_5 254:26 0 7.2T 0 crypt │ ├─Postulate_Inner-Primary 254:27 0 5T 0 lvm / │ └─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP] ├─Postulate_Outer-Primary_rimage_1_imeta 254:13 0 29.6G 0 lvm │ └─Postulate_Outer-Primary_rimage_1 254:15 0 3.6T 0 lvm │ └─Postulate_Outer-Primary 254:25 0 7.2T 0 lvm │ └─bcache1 253:128 0 7.2T 0 disk │ └─main_crypt_5 254:26 0 7.2T 0 crypt │ ├─Postulate_Inner-Primary 254:27 0 5T 0 lvm / │ └─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP] └─Postulate_Outer-Primary_rimage_1_iorig 254:14 0 3.6T 0 lvm └─Postulate_Outer-Primary_rimage_1 254:15 0 3.6T 0 lvm └─Postulate_Outer-Primary 254:25 0 7.2T 0 lvm └─bcache1 253:128 0 7.2T 0 disk └─main_crypt_5 254:26 0 7.2T 0 crypt ├─Postulate_Inner-Primary 254:27 0 5T 0 lvm / └─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP] sde 8:64 0 4.5T 0 disk └─sde4 8:68 0 4.5T 0 part ├─Axiom_Outer-Primary_rmeta_0 254:4 0 4M 0 lvm │ └─Axiom_Outer-Primary 254:24 0 4.5T 0 lvm │ └─bcache0 253:0 0 4.5T 0 disk │ └─main_crypt_2 254:29 0 4.5T 0 crypt │ └─Axiom_Inner-New 254:30 0 3.6T 0 lvm ├─Axiom_Outer-Primary_rimage_0_imeta 254:7 0 36.8G 0 lvm │ └─Axiom_Outer-Primary_rimage_0 254:11 0 4.5T 0 lvm │ └─Axiom_Outer-Primary 254:24 0 4.5T 0 lvm │ └─bcache0 253:0 0 4.5T 0 disk │ └─main_crypt_2 254:29 0 4.5T 0 crypt │ └─Axiom_Inner-New 254:30 0 3.6T 0 lvm └─Axiom_Outer-Primary_rimage_0_iorig 254:8 0 4.5T 0 lvm └─Axiom_Outer-Primary_rimage_0 254:11 0 4.5T 0 lvm └─Axiom_Outer-Primary 254:24 0 4.5T 0 lvm └─bcache0 253:0 0 4.5T 0 disk └─main_crypt_2 254:29 0 4.5T 0 crypt └─Axiom_Inner-New 254:30 0 3.6T 0 lvm sdf 8:80 0 931.5G 0 disk └─sdf4 8:84 0 930.5G 0 part sdg 8:96 0 931.5G 0 disk └─sdg4 8:100 0 930.5G 0 part sdh 8:112 0 5.5T 0 disk └─sdh4 8:116 0 5.5T 0 part ├─Postulate_Backup-Primary 254:0 0 5T 0 lvm │ └─backup_crypt_3 254:31 0 5T 0 crypt /mnt/bk/3 └─Postulate_Backup-Old 254:1 0 128G 0 lvm sdi 8:128 0 1.8T 0 disk └─sdi4 8:132 0 1.8T 0 part └─Postulate_Double-1 254:2 0 3.6T 0 lvm sdj 8:144 0 4.5T 0 disk └─sdj4 8:148 0 4.5T 0 part ├─Axiom_Outer-Primary_rmeta_1 254:16 0 4M 0 lvm │ └─Axiom_Outer-Primary 254:24 0 4.5T 0 lvm │ └─bcache0 253:0 0 4.5T 0 disk │ └─main_crypt_2 254:29 0 4.5T 0 crypt │ └─Axiom_Inner-New 254:30 0 3.6T 0 lvm ├─Axiom_Outer-Primary_rimage_1_imeta 254:17 0 36.8G 0 lvm │ └─Axiom_Outer-Primary_rimage_1 254:19 0 4.5T 0 lvm │ └─Axiom_Outer-Primary 254:24 0 4.5T 0 lvm │ └─bcache0 253:0 0 4.5T 0 disk │ └─main_crypt_2 254:29 0 4.5T 0 crypt │ └─Axiom_Inner-New 254:30 0 3.6T 0 lvm └─Axiom_Outer-Primary_rimage_1_iorig 254:18 0 4.5T 0 lvm └─Axiom_Outer-Primary_rimage_1 254:19 0 4.5T 0 lvm └─Axiom_Outer-Primary 254:24 0 4.5T 0 lvm └─bcache0 253:0 0 4.5T 0 disk └─main_crypt_2 254:29 0 4.5T 0 crypt └─Axiom_Inner-New 254:30 0 3.6T 0 lvm sdk 8:160 0 223.6G 0 disk └─sdk4 8:164 0 189.6G 0 part nvme0n1 259:0 0 931.5G 0 disk ├─nvme0n1p2 259:1 0 200M 0 part /boot/efi ├─nvme0n1p3 259:2 0 814M 0 part /boot ├─nvme0n1p4 259:3 0 832G 0 part │ └─Postulate_Outer-Cache 254:3 0 800G 0 lvm │ └─bcache1 253:128 0 7.2T 0 disk │ └─main_crypt_5 254:26 0 7.2T 0 crypt │ ├─Postulate_Inner-Primary 254:27 0 5T 0 lvm / │ └─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP] └─nvme0n1p8 259:4 0 54G 0 part └─recovery_crypt 254:32 0 54G 0 crypt /mnt/recovery [-- Attachment #3: lsblk_m.txt --] [-- Type: text/plain, Size: 6293 bytes --] NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 1.8T 0 disk ┌┈▶ └─sda4 8:4 0 1.8T 0 part ┆ sdb 8:16 0 3.6T 0 disk ┆ └─sdb4 8:20 0 3.6T 0 part ┌┈▶┆ ├─Postulate_Outer-Primary_rmeta_0 254:5 0 1M 0 lvm ┌┈▶┆ ┆ ├─Postulate_Outer-Primary_rimage_0_imeta 254:6 0 29.6G 0 lvm └┬▶┆ ┆ └─Postulate_Outer-Primary_rimage_0_iorig 254:9 0 3.6T 0 lvm └┈├┈▶┆┈┈┈Postulate_Outer-Primary_rimage_0 254:10 0 3.6T 0 lvm ┆ ┆ sdc 8:32 0 3.6T 0 disk ┆ ┆ ├─sdc1 8:33 0 10M 0 part ┆ ┆ ├─sdc2 8:34 0 200M 0 part ┆ ┆ ├─sdc3 8:35 0 809M 0 part ┆ ┆ └─sdc4 8:36 0 3.6T 0 part ├┈▶┆ ├─Postulate_Outer-Primary_rmeta_2 254:20 0 1M 0 lvm ┌┈▶┆ ┆ ├─Postulate_Outer-Primary_rimage_2_imeta 254:21 0 29.6G 0 lvm └┬▶┆ ┆ └─Postulate_Outer-Primary_rimage_2_iorig 254:22 0 3.6T 0 lvm └┈├┈▶┆┈┈┈Postulate_Outer-Primary_rimage_2 254:23 0 3.6T 0 lvm ┆ ┆ sdd 8:48 0 3.6T 0 disk ┆ ┆ └─sdd4 8:52 0 3.6T 0 part ├┈▶┆ ├─Postulate_Outer-Primary_rmeta_1 254:12 0 1M 0 lvm ┌┈▶┆ ┆ ├─Postulate_Outer-Primary_rimage_1_imeta 254:13 0 29.6G 0 lvm └┬▶┆ ┆ └─Postulate_Outer-Primary_rimage_1_iorig 254:14 0 3.6T 0 lvm └┈└┬▶┆┈┈┈Postulate_Outer-Primary_rimage_1 254:15 0 3.6T 0 lvm ┌┈▶ └┈┆┈┈┈Postulate_Outer-Primary 254:25 0 7.2T 0 lvm ┆ ┆ sde 8:64 0 4.5T 0 disk ┆ ┆ └─sde4 8:68 0 4.5T 0 part ┆ ┌┈▶┆ ├─Axiom_Outer-Primary_rmeta_0 254:4 0 4M 0 lvm ┌┈▶┆ ┆ ┆ ├─Axiom_Outer-Primary_rimage_0_imeta 254:7 0 36.8G 0 lvm └┬▶┆ ┆ ┆ └─Axiom_Outer-Primary_rimage_0_iorig 254:8 0 4.5T 0 lvm └┈┆┈┈├┈▶┆┈┈┈Axiom_Outer-Primary_rimage_0 254:11 0 4.5T 0 lvm ┆ ┆ ┆ sdf 8:80 0 931.5G 0 disk ┆ ┆ ┆ └─sdf4 8:84 0 930.5G 0 part ┆ ┆ ┆ sdg 8:96 0 931.5G 0 disk ┆ ┆ ┆ └─sdg4 8:100 0 930.5G 0 part ┆ ┆ ┆ sdh 8:112 0 5.5T 0 disk ┆ ┆ ┆ └─sdh4 8:116 0 5.5T 0 part ┆ ┆ ┆ ├─Postulate_Backup-Primary 254:0 0 5T 0 lvm ┆ ┆ ┆ │ └─backup_crypt_3 254:31 0 5T 0 crypt /mnt/bk/3 ┆ ┆ ┆ └─Postulate_Backup-Old 254:1 0 128G 0 lvm ┆ ┆ ┆ sdi 8:128 0 1.8T 0 disk ┆ ┆ └┬▶ └─sdi4 8:132 0 1.8T 0 part ┆ ┆ └┈┈Postulate_Double-1 254:2 0 3.6T 0 lvm ┆ ┆ sdj 8:144 0 4.5T 0 disk ┆ ┆ └─sdj4 8:148 0 4.5T 0 part ┆ ├┈▶ ├─Axiom_Outer-Primary_rmeta_1 254:16 0 4M 0 lvm ┆ ┆ ┌┈▶ ├─Axiom_Outer-Primary_rimage_1_imeta 254:17 0 36.8G 0 lvm ┆ ┆ └┬▶ └─Axiom_Outer-Primary_rimage_1_iorig 254:18 0 4.5T 0 lvm ┆ └┬▶ └┈┈Axiom_Outer-Primary_rimage_1 254:19 0 4.5T 0 lvm ┆ └┈┈┈┈┈Axiom_Outer-Primary 254:24 0 4.5T 0 lvm ┆ └─bcache0 253:0 0 4.5T 0 disk ┆ └─main_crypt_2 254:29 0 4.5T 0 crypt ┆ └─Axiom_Inner-New 254:30 0 3.6T 0 lvm ┆ sdk 8:160 0 223.6G 0 disk ┆ └─sdk4 8:164 0 189.6G 0 part ┆ nvme0n1 259:0 0 931.5G 0 disk ┆ ├─nvme0n1p2 259:1 0 200M 0 part /boot/efi ┆ ├─nvme0n1p3 259:2 0 814M 0 part /boot ┆ ├─nvme0n1p4 259:3 0 832G 0 part └┬▶ │ └─Postulate_Outer-Cache 254:3 0 800G 0 lvm ┆ └─nvme0n1p8 259:4 0 54G 0 part ┆ └─recovery_crypt 254:32 0 54G 0 crypt /mnt/recovery └┈┈┈┈┈┈┈┈bcache1 253:128 0 7.2T 0 disk └─main_crypt_5 254:26 0 7.2T 0 crypt ├─Postulate_Inner-Primary 254:27 0 5T 0 lvm / └─Postulate_Inner-Swap 254:28 0 24G 0 lvm [SWAP] [-- Attachment #4: Type: text/plain, Size: 202 bytes --] _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks 2022-05-07 3:33 ` Alex Lieflander @ 2022-05-07 20:41 ` Stuart D Gathman 2022-05-07 23:14 ` Alex Lieflander 0 siblings, 1 reply; 9+ messages in thread From: Stuart D Gathman @ 2022-05-07 20:41 UTC (permalink / raw) To: LVM general discussion and development [-- Attachment #1: Type: text/plain, Size: 552 bytes --] On Fri, 6 May 2022, Alex Lieflander wrote: > Thanks. I really don’t want to give up the DM-Integrity management. Less complexity is just a bonus. What are you trying to get out of RAID6? If redundancy and integrity are already managed at another layer, then just use RAID0 for striping. I like to use RAID10 for mirror + striping, but I understand parity disks give redundancy without halving capacity. Parity means RMW cycles of largish blocks, whereas straight mirroring (RAID1, RAID10) can write single sectors without a RMW cycle. [-- Attachment #2: Type: text/plain, Size: 202 bytes --] _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks 2022-05-07 20:41 ` Stuart D Gathman @ 2022-05-07 23:14 ` Alex Lieflander 2022-05-08 13:25 ` Stuart D Gathman 2022-05-09 0:18 ` John Stoffel 0 siblings, 2 replies; 9+ messages in thread From: Alex Lieflander @ 2022-05-07 23:14 UTC (permalink / raw) To: LVM general discussion and development > On May 7, 2022, at 4:41 PM, Stuart D Gathman wrote: > >> On Fri, 6 May 2022, Alex Lieflander wrote: >> >> Thanks. I really don’t want to give up the DM-Integrity management. Less complexity is just a bonus. > > What are you trying to get out of RAID6? If redundancy and integrity > are already managed at another layer, then just use RAID0 for striping. > > I like to use RAID10 for mirror + striping, but I understand parity disks give redundancy without halving capacity. Parity means RMW cycles of > largish blocks, whereas straight mirroring (RAID1, RAID10) can write > single sectors without a RMW cycle. I don’t trust the hardware I’m running on very much, but it’s all I have to work with at the moment; it’s important that the array is resilient to *any* (and multiple) single chunk corruptions because such corruptions are likely to happen in the future. For the last several months I’ve periodically been seeing (DM-Integrity) checksum mismatch warnings at various locations on all of my disks. I stopped using a few SATA ports that were explicitly throwing SATA errors, but I suspect that the remaining connections are unpredictably (albeit infrequently) corrupting data in ways that are more difficult to detect. I’ve tried to “check” and “repair” my array on multiple kernel versions and live recovery USB sticks, but the “check" always seems to freeze and all subsequent IO to the array hangs until reboot; at the moment, a chunk is only ever made consistent when its data is overwritten, so it needs to survive periodic, random corruption for as long as possible. I also have a disk that infrequently fails to read from a particular area, but the rest of the disk is fine. I wouldn’t trust that disk with valuable data, but it seems like a perfect candidate to hold additional parity (raid6_ls_6) that I hopefully never need. Alex _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks 2022-05-07 23:14 ` Alex Lieflander @ 2022-05-08 13:25 ` Stuart D Gathman 2022-05-09 0:18 ` John Stoffel 1 sibling, 0 replies; 9+ messages in thread From: Stuart D Gathman @ 2022-05-08 13:25 UTC (permalink / raw) To: LVM general discussion and development [-- Attachment #1: Type: text/plain, Size: 947 bytes --] On Sat, 7 May 2022, Alex Lieflander wrote: > I don’t trust the hardware I’m running on very much, but it’s all I have to work with at the moment; it’s important that the array is resilient to *any* (and multiple) single chunk corruptions because such corruptions are likely to happen in the future. > > For the last several months I’ve periodically been seeing (DM-Integrity) checksum mismatch warnings at various locations on all of my disks. I stopped using a few SATA ports that were explicitly throwing SATA errors, but I suspect that the remaining connections are unpredictably (albeit infrequently) corrupting data in ways that are more difficult to detect. Sounds like a *great* test bed for software data integrity tools. Don't throw that system away when you get a more reliable one! That sounds like a situation that btrfs with multiple copies could handle. Use a beefier checksum than the default crc-32 also. [-- Attachment #2: Type: text/plain, Size: 202 bytes --] _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks 2022-05-07 23:14 ` Alex Lieflander 2022-05-08 13:25 ` Stuart D Gathman @ 2022-05-09 0:18 ` John Stoffel 1 sibling, 0 replies; 9+ messages in thread From: John Stoffel @ 2022-05-09 0:18 UTC (permalink / raw) To: LVM general discussion and development >>>>> "Alex" == Alex Lieflander <atlief@icloud.com> writes: >> On May 7, 2022, at 4:41 PM, Stuart D Gathman wrote: >> >>> On Fri, 6 May 2022, Alex Lieflander wrote: >>> >>> Thanks. I really don’t want to give up the DM-Integrity management. Less complexity is just a bonus. >> >> What are you trying to get out of RAID6? If redundancy and integrity >> are already managed at another layer, then just use RAID0 for striping. >> >> I like to use RAID10 for mirror + striping, but I understand parity disks give redundancy without halving capacity. Parity means RMW cycles of >> largish blocks, whereas straight mirroring (RAID1, RAID10) can write >> single sectors without a RMW cycle. Alex> I don’t trust the hardware I’m running on very much, but it’s Alex> all I have to work with at the moment; it’s important that the Alex> array is resilient to *any* (and multiple) single chunk Alex> corruptions because such corruptions are likely to happen in the Alex> future. Ouch! I hope you have good backups somewhere, because I suspect you're doing to suffer a complete failure at some point. Alex> For the last several months I’ve periodically been seeing Alex> (DM-Integrity) checksum mismatch warnings at various locations Alex> on all of my disks. I stopped using a few SATA ports that were Alex> explicitly throwing SATA errors, but I suspect that the Alex> remaining connections are unpredictably (albeit infrequently) Alex> corrupting data in ways that are more difficult to detect. This is interesting. And worrisome, because I would not expect moving from one SATA port to another to cure problems, unless it was A) moving to a different controller, or B) you changed/reseated the SATA cable. But I also wonder about your power supply and what it's rated for. You might just be hitting the ragged edge of what it can supply, and so you're running into problems with voltage dropping just enough to make things slightly flaky. Alex> I’ve tried to “check” and “repair” my array on multiple kernel Alex> versions and live recovery USB sticks, but the “check" always Alex> seems to freeze and all subsequent IO to the array hangs until Alex> reboot; at the moment, a chunk is only ever made consistent when Alex> its data is overwritten, so it needs to survive periodic, random Alex> corruption for as long as possible. This is also a warning to my that maybe you have power supply issues. Can you give a summary of your hardware configuration and model numbers? If you're running a smallish power supply, maybe look for a replacement which can get you more power. Go from a 430W one to 600W, or 500W to 750W and see if that makes a difference. Looking at your data from before, I see you have 12 disks on the system, 11 spinning disks and one nvme device. So I *really* suspect you have an overloaded power supply. Are you also using a disk controller? And which version of linux? Alex> I also have a disk that infrequently fails to read from a Alex> particular area, but the rest of the disk is fine. I wouldn’t Alex> trust that disk with valuable data, but it seems like a perfect Alex> candidate to hold additional parity (raid6_ls_6) that I Alex> hopefully never need. This is not how RAID6 parity works. The entire disk (or partition) is used to write data and/or parity. It's RAID4 which dedicates a single disk to parity duties. So thinking that a known flaky disk will be ok for just parity use isn't really a good idea. I'd also look at the output of 'smartctl --all /dev/sd<letter>' for all your disks and see what the numbers say. But honestly, it sounds like you have some serious hardware issues which you're trying to paper over with DM-Integrity and RAID5. And I suspect it will all end in tears sooner or later. You do have backups of your data, right? Even onto a single new 10tb disk that's now connected to the system all the time? Good luck, John _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2022-05-09 0:19 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-04-27 17:11 [linux-lvm] Bypassing LVM Restrictions - RAID6 With Less Than 5 Disks Alex Lieflander 2022-05-02 0:36 ` John Stoffel 2022-05-02 19:22 ` Alex Lieflander 2022-05-02 21:54 ` John Stoffel 2022-05-07 3:33 ` Alex Lieflander 2022-05-07 20:41 ` Stuart D Gathman 2022-05-07 23:14 ` Alex Lieflander 2022-05-08 13:25 ` Stuart D Gathman 2022-05-09 0:18 ` John Stoffel
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).