Re: Suggestions for building new 44TB Raid5 array

[ronnie sahlberg <ronniesahlberg@gmail.com>]

On Sat, 11 Jun 2022 at 17:16, Marc MERLIN <marc@merlins.org> wrote:
>
> so, my apologies to all for the thread of death that is hopefully going
> to be over soon. I still want to help Josef fix the tools though,
> hopefully we'll get that filesystem back to a mountable state.
>
> That said, it's been over 2 months now, and I do need to get this
> filesystem back up from backup, so I ended up buying new drives (5x
> 11TiB in raid5).
>
> Given the pretty massive corruption that happened in ways that I still
> can't explain, I'll make sure to turn off all the drive write caches
> but I think I'm not sure I want to trust bcache anymore even though
> I had it in writethrough mode.
>
> Here's the Email from March, questions still apply:
>
> Kernel will be 5.16. Filesystem will be 24TB and contain mostly bigger
> files (100MB to 10GB).
>
> 1) mdadm --create /dev/md7 --level=5 --consistency-policy=ppl --raid-devices=5 /dev/sd[abdef]1 --chunk=256 --bitmap=internal
> 2) echo 0fb96f02-d8da-45ce-aba7-070a1a8420e3 >  /sys/block/bcache64/bcache/attach
>    gargamel:/dev# cat /sys/block/md7/bcache/cache_mode
>    [writethrough] writeback writearound none
> 3) cryptsetup luksFormat --align-payload=2048 -s 256 -c aes-xts-plain64  /dev/bcache64
> 4) cryptsetup luksOpen /dev/bcache64 dshelf1
> 5) mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1
>
> Any other btrfs options I should set for format to improve reliability
> first and performance second?
> I'm told I should use space_cache=v2, is it default now with btrfs-progs 5.10.1-2 ?
>
> As for bcache, I'm really thinking about droppping it, unless I'm told
> it should be safe to use.
>
> Thanks,
> Marc

My needs are much more basic.
I have a LOT of large files. ISO images and QEMU disk images. I also
have hundreds of thousands of photos.

I used different multi-disk solutions but found them all too fragile
so now, last 8 years, I have used a setup that is basically
5 disks each with their own EXT4 filesystem ontop of LUKS and two
additional drives to have 2 disk parity in snapraid.
Now, snapraid does not do in-line raid updates so I carefully manage
how I handle the data.
Audio, photos and QEMU base images are immutable so these files are
not a problem.
For VM images I have for each machine an immutable 'base' image that
snapraid takes care of and I have live images based on that that are
not
handled by snapraid.  (qemu-img create -b base.img cow.img)
If a live image goes corrupt due to a poweroutage or similar I just
re-create it ontop of the latest archived base image.
As I often do stuff to the VM images that cause kernel panics, this is
a very convenient way to restore them quickly and with little effort
to a known good state.

If one disk has a catastrophic failure, I only lose the files on that
particular disk and just have to restore them but nothing else.
Now I do export them as 5 different filesystems/shares  but that is
just because I am too lazy to set up some kind of "merge fs".

If your use case is mostly-read and mostly-archive this might work for
you too and it is VERY reliable.
Ease of mind knowing that if a single disk dies I do not have a total
dataloss scenario. If the Windows VM disk dies, I just restore those
images while all the other disks are still online.


It is simple, primitive and 1980 type of technology but it works. And
it is reliable.

> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/