New setup: partitions or raw devices

New setup: partitions or raw devices

[Gandalf Corvotempesta]

Hi to all
I have to setup some new servers, which is the best practise, currently?
Should I use raw devices or partitions ? (one huge partition for mdadm, and then
LVM on top of it)

Any advantage/drawbacks for both configurations ?

Re: New setup: partitions or raw devices

[Reindl Harald]

Am 29.11.2017 um 17:22 schrieb Gandalf Corvotempesta:
> Hi to all
> I have to setup some new servers, which is the best practise, currently?
> Should I use raw devices or partitions ? (one huge partition for mdadm, and then
> LVM on top of it)
> 
> Any advantage/drawbacks for both configurations?

i would make partitions for 2 reasons:

* leave some space free in case a replacement disk has
   a slighty different usebale size - some SSD's do
   this instead overprovisioning for wear leveling

* whenever you decide you replace it against SSD's
   free space will enhance the overall lifetime

Re: New setup: partitions or raw devices

[Phil Turmel]

Hi Gandalf,

On 11/29/2017 11:44 AM, Reindl Harald wrote:

> Am 29.11.2017 um 17:22 schrieb Gandalf Corvotempesta:
>> Hi to all
>> I have to setup some new servers, which is the best practise, currently?
>> Should I use raw devices or partitions ? (one huge partition for
>> mdadm, and then
>> LVM on top of it)
>>
>> Any advantage/drawbacks for both configurations?
> 
> i would make partitions for 2 reasons:
> 
> * leave some space free in case a replacement disk has
>   a slighty different usebale size - some SSD's do
>   this instead overprovisioning for wear leveling
> 
> * whenever you decide you replace it against SSD's
>   free space will enhance the overall lifetime

These are good reasons.  I have also seen reports of NAS devices/distros
unconditionally partitioning devices that don't have one.  Although I
have in the past used raw devices with mdadm, I don't plan to do so with
any future systems.

Phil

Re: New setup: partitions or raw devices

[Gandalf Corvotempesta]

2017-11-29 17:44 GMT+01:00 Reindl Harald <h.reindl@thelounge.net>:
> * leave some space free in case a replacement disk has
>   a slighty different usebale size - some SSD's do
>   this instead overprovisioning for wear leveling

Good reason.
How much space would you reserve ? 100MB? 1GB ?

Re: New setup: partitions or raw devices

[Gandalf Corvotempesta]

2017-11-29 17:52 GMT+01:00 Phil Turmel <philip@turmel.org>:
> These are good reasons.  I have also seen reports of NAS devices/distros
> unconditionally partitioning devices that don't have one.  Although I
> have in the past used raw devices with mdadm, I don't plan to do so with
> any future systems.

I always used partitioning.
But last time i've tried to configure mdadm to automatically replace a
disk when replaced:
https://linux.die.net/man/5/mdadm.conf
"POLICY" section, tried all but the best candidate seems to be "spare-same-slot"

I was never able to make that working. I've tried everything: insert a
brand new disk, insert an already partitioned disks and so on.
I always though that the main issue was due to disk partitioning,
mdadm won't be able to re-add a brand new disks if raid is made
with paritions (the new disks doesn't have any partition)

Any idea ?

Re: New setup: partitions or raw devices

[Phil Turmel]

On 11/29/2017 12:42 PM, Gandalf Corvotempesta wrote:
> 2017-11-29 17:52 GMT+01:00 Phil Turmel <philip@turmel.org>:
>> These are good reasons.  I have also seen reports of NAS devices/distros
>> unconditionally partitioning devices that don't have one.  Although I
>> have in the past used raw devices with mdadm, I don't plan to do so with
>> any future systems.
> 
> I always used partitioning.
> But last time i've tried to configure mdadm to automatically replace a
> disk when replaced:
> https://linux.die.net/man/5/mdadm.conf
> "POLICY" section, tried all but the best candidate seems to be "spare-same-slot"
> 
> I was never able to make that working. I've tried everything: insert a
> brand new disk, insert an already partitioned disks and so on.
> I always though that the main issue was due to disk partitioning,
> mdadm won't be able to re-add a brand new disks if raid is made
> with paritions (the new disks doesn't have any partition)
> 
> Any idea ?

I don't let mdadm do any such automatic allocation of new devices.  If
you have to manually put the device in the server, its only a few
seconds more to note which device it is in dmesg, and add it to the
array.  (And partition it, if not already.)

Phil

Re: New setup: partitions or raw devices

[Reindl Harald]

Am 29.11.2017 um 18:38 schrieb Gandalf Corvotempesta:
> 2017-11-29 17:44 GMT+01:00 Reindl Harald <h.reindl@thelounge.net>:
>> * leave some space free in case a replacement disk has
>>    a slighty different usebale size - some SSD's do
>>    this instead overprovisioning for wear leveling
> 
> Good reason.
> How much space would you reserve ? 100MB? 1GB ?

if i would be in the position without create arrays from scratch these 
das i would leave 2-5% of the disks for overprovisioning instead only 20 
MB as i did in 2011 because i am about to replace with SSD which i did 
not consider that days

Re: New setup: partitions or raw devices

[Gandalf Corvotempesta]

2-5% ? With current hdd size (ie, 8TB), 2% is about 160GB.
I don't have disks with that size, but only 2 or 4TB. Are hundreds of
GB needed ? Why not 100MB ?

2017-11-29 19:28 GMT+01:00 Reindl Harald <h.reindl@thelounge.net>:
>
>
> Am 29.11.2017 um 18:38 schrieb Gandalf Corvotempesta:
>>
>> 2017-11-29 17:44 GMT+01:00 Reindl Harald <h.reindl@thelounge.net>:
>>>
>>> * leave some space free in case a replacement disk has
>>>    a slighty different usebale size - some SSD's do
>>>    this instead overprovisioning for wear leveling
>>
>>
>> Good reason.
>> How much space would you reserve ? 100MB? 1GB ?
>
>
> if i would be in the position without create arrays from scratch these das i
> would leave 2-5% of the disks for overprovisioning instead only 20 MB as i
> did in 2011 because i am about to replace with SSD which i did not consider
> that days

Fwd: New setup: partitions or raw devices

[Gandalf Corvotempesta]

repost due to previous html mail


---------- Forwarded message ----------
From: Gandalf Corvotempesta <gandalf.corvotempesta@gmail.com>
Date: 2017-11-29 19:09 GMT+01:00
Subject: Re: New setup: partitions or raw devices
To: Phil Turmel <philip@turmel.org>
Cc: Reindl Harald <h.reindl@thelounge.net>, Linux RAID Mailing List
<linux-raid@vger.kernel.org>


Sure, but this require a sysadmin logged on the system

If I'm far away, nobody would be able to change a disk because I'm the
only with root privilege

Re: New setup: partitions or raw devices

[Reindl Harald]

Am 29.11.2017 um 20:51 schrieb Gandalf Corvotempesta:
> 2-5% ? With current hdd size (ie, 8TB), 2% is about 160GB.
> I don't have disks with that size, but only 2 or 4TB. Are hundreds of
> GB needed ? Why not 100MB ?

in case of 2 TB disks it's 40 GB per disk

because 100 MB IMHO don't help that much for SSD overprovisioning and 
given how it may extend the lifetime versus the price and you don't have 
your disks typically filled completly

in a 4 drive RAID10 it's 80 GB missing useable disk space for 
overprovisioning out of around 3.6 TB (1000 versus 1024) while you have 
a lightning fast array which should last as long as possible without 
replace terrible expensive drives

why not RAID5/6? besides 
https://www.askdbmgt.com/why-raid5-should-be-avoided-at-all-costs.html 
the parity data are additional writes wearing out the drives

> 2017-11-29 19:28 GMT+01:00 Reindl Harald <h.reindl@thelounge.net>:
>>
>>
>> Am 29.11.2017 um 18:38 schrieb Gandalf Corvotempesta:
>>>
>>> 2017-11-29 17:44 GMT+01:00 Reindl Harald <h.reindl@thelounge.net>:
>>>>
>>>> * leave some space free in case a replacement disk has
>>>>     a slighty different usebale size - some SSD's do
>>>>     this instead overprovisioning for wear leveling
>>>
>>>
>>> Good reason.
>>> How much space would you reserve ? 100MB? 1GB ?
>>
>>
>> if i would be in the position without create arrays from scratch these das i
>> would leave 2-5% of the disks for overprovisioning instead only 20 MB as i
>> did in 2011 because i am about to replace with SSD which i did not consider
>> that days

Re: New setup: partitions or raw devices

[Gandalf Corvotempesta]

2017-11-29 21:02 GMT+01:00 Reindl Harald <h.reindl@thelounge.net>:
> in case of 2 TB disks it's 40 GB per disk

Yes

> because 100 MB IMHO don't help that much for SSD overprovisioning and given
> how it may extend the lifetime versus the price and you don't have your
> disks typically filled completly

I'm talking about multi-tera disks, I don't think i'll ever (in the
near future) replace them
with SSDs....

> in a 4 drive RAID10 it's 80 GB missing useable disk space for
> overprovisioning out of around 3.6 TB (1000 versus 1024) while you have a
> lightning fast array which should last as long as possible without replace
> terrible expensive drives

Yes, but you are talking about SSD...
Here I have many HDD....

> why not RAID5/6? besides
> https://www.askdbmgt.com/why-raid5-should-be-avoided-at-all-costs.html the
> parity data are additional writes wearing out the drives

I think exactly the opposite.
I will never ever run anything with redundancy level lower than 2.
I'm totally againt RAID10, after i've loose (4 times!) the whole RAID due to
double failure on the same mirror.

Now, I'm only using RAID-6 on very small arrays (6 disks maximum)
If I need more space, I'll create a RAID-60

My newer server with SSD are running with 4 SSD in RAID-6 (or RAID-Z2,
it depends)

Re: New setup: partitions or raw devices

[Chris Murphy]

On Wed, Nov 29, 2017 at 9:52 AM, Phil Turmel <philip@turmel.org> wrote:

> These are good reasons.  I have also seen reports of NAS devices/distros
> unconditionally partitioning devices that don't have one.  Although I
> have in the past used raw devices with mdadm, I don't plan to do so with
> any future systems.

I'm in this same boat. Historically I've used raw devices, no
partitioning, but I'm going to GPT partition everything from here on
out, even if it's just a single partition.

a. It makes it unambiguous this drive has had some purpose, it's not a
blank slate.
b. GPT is unambiguous, checksummed, has redundancy, and a user
definable partition name 72 bytes UTF-16E to make it even more
unambiguous
c. The MBR had too few type codes, leading to a lot of ambiguity as to
a partition's contents, so on Linux the idea was for libblkid to have
a thorough understanding of (most) every conceivable volume format
signature. But GPT solves this in a more standard way, and the
explosion of undefined volume formats and binary blobs installed on
partitions has made keeping up with their identification in blkid
increasingly challenging.

So anyway... GPT is good. And when LUKS2 gets a little farther along
the torture testing I'll move to that as well, as it too has redundant
metadata.

-- 
Chris Murphy

Re: New setup: partitions or raw devices

[Reindl Harald]

Am 29.11.2017 um 23:02 schrieb Gandalf Corvotempesta:
> 2017-11-29 21:02 GMT+01:00 Reindl Harald <h.reindl@thelounge.net>:
>> in case of 2 TB disks it's 40 GB per disk
> 
> Yes
> 
>> because 100 MB IMHO don't help that much for SSD overprovisioning and given
>> how it may extend the lifetime versus the price and you don't have your
>> disks typically filled completly
> 
> I'm talking about multi-tera disks, I don't think i'll ever (in the
> near future) replace them
> with SSDs....

i thought the same in 2011 and now i regret it *because* i change the 2 
TB disks against SSD's (while i would sell my sould for RAID10 
supporting the RAID1 writemoostly feature to only have writes on the 
current 2 remaining HDD and reads de-facto from the SSD stripe-part)

[harry@srv-rhsoft:~]$ df
Dateisystem    Typ  Größe Benutzt Verf. Verw% Eingehängt auf
/dev/md1       ext4   29G    6,8G   22G   24% /
/dev/md0       ext4  485M     35M  446M    8% /boot
/dev/md2       ext4  3,6T    1,9T  1,8T   53% /mnt/data

[0:0:0:0]    disk    ATA      Samsung SSD 850  2B6Q  /dev/sda
[1:0:0:0]    disk    ATA      Samsung SSD 850  2B6Q  /dev/sdb
[2:0:0:0]    disk    ATA      WDC WD2003FYYS-0 1D01  /dev/sdc
[3:0:0:0]    disk    ATA      ST2000DX002-2DV1 CC41  /dev/sdd

Re: New setup: partitions or raw devices

[Gandalf Corvotempesta]

2017-11-29 23:10 GMT+01:00 Chris Murphy <lists@colorremedies.com>:
> a. It makes it unambiguous this drive has had some purpose, it's not a
> blank slate.
> b. GPT is unambiguous, checksummed, has redundancy, and a user
> definable partition name 72 bytes UTF-16E to make it even more
> unambiguous
> c. The MBR had too few type codes, leading to a lot of ambiguity as to
> a partition's contents, so on Linux the idea was for libblkid to have
> a thorough understanding of (most) every conceivable volume format
> signature. But GPT solves this in a more standard way, and the
> explosion of undefined volume formats and binary blobs installed on
> partitions has made keeping up with their identification in blkid
> increasingly challenging.

So, if you where me, how many gpt partition and which types would you make ?

Re: New setup: partitions or raw devices

[Chris Murphy]

An exception to the single partition "rule" is with UDF. In order to
get a fully cross platform supported UDF volume, it needs to own the
entire physical block device. It's quite a bit more sane to use as a
cross platform volume format than FAT32 or exFAT, for sticks or
drives.


Chris Murphy

Re: New setup: partitions or raw devices

[Wol's lists]

On 29/11/17 20:02, Reindl Harald wrote:
> why not RAID5/6? besides 
> https://www.askdbmgt.com/why-raid5-should-be-avoided-at-all-costs.html 
> the parity data are additional writes wearing out the drives

So, if I have a four-drive raid 5, for every 3 blocks of data I write I 
write 1 parity block. But with raid 1 or 10, for every 3 blocks of data 
I write, I write *3* "parity" blocks!

What was that about "the additional writes wearing out the drives" then?

(Yes, I get the write amplification thing - but if you are writing a lot 
of data, then raid 5 needs far *fewer* writes.)

Cheers,
Wol

Re: New setup: partitions or raw devices

[Gandalf Corvotempesta]

2017-11-29 23:10 GMT+01:00 Reindl Harald <h.reindl@thelounge.net>:
> [0:0:0:0]    disk    ATA      Samsung SSD 850  2B6Q  /dev/sda
> [1:0:0:0]    disk    ATA      Samsung SSD 850  2B6Q  /dev/sdb

these are consumer SSDs.
Even the "PRO" is consumer and a 2TB is priced about 800 USD

Look at one of Intel DC SSD, the S3610 1.6TB is about 1500 USD. Twice
the price, 40% less capacity

Re: New setup: partitions or raw devices

[Chris Murphy]

On Wed, Nov 29, 2017 at 3:14 PM, Gandalf Corvotempesta
<gandalf.corvotempesta@gmail.com> wrote:

>
> So, if you where me, how many gpt partition and which types would you make ?

What's the complete use case scenario? I can't really tell from the first post.

What I do is use gdisk type code 8300 (generic linux partition), and
then format that partition as LUKS1. And then I open the LUKS device
and make it a PV, create VG, and do that on every physical device.

Now you can make however many LV's you want, any size you want, in
arbitrary order, with whatever raid level you want. LVM's
raid0,1,10,5,6 use the md driver in the kernel, same as mdadm, but
it's LVM metadata which means everything is managed with LVM (disk
replacement and so forth). And there are still features mdadm has that
LVM raid does not implement. So you it's not a question of which is
better overall, it's a question of which one fits your use case best,
and the tools with which you're most familiar.

But it's pretty neat to be able to have one big VG, and to be able to
arbitrarily create LV's that themselves have the raid type. It's a lot
easier to manage in the case where you need different levels of
redundancy but you're not completely certain what the utilization of
those volumes will be, you can leave some unallocated space in the VG
"pool" so you can increase the size of any LV on demand whenever you
want.

If you're more comfortable with a conventional approach: GPT > LUKS >
mdadm > LVM > file system is just fine. I pretty much always start out
with LUKS for data drives because it's the only way to be certain
you're not leaking data should the drive need to be repurposed or
returned under warranty and won't spin up. It's also too easy to just
luksErase or luksFormat to obliterate everything on the drive, rather
than have to do a complete tear down of each layer's signature, and
you really shouldn't leave stale storage stack layers behind when
repurposing drives, it can cause confusion later.

-- 
Chris Murphy

Re: New setup: partitions or raw devices

[Reindl Harald]

Am 29.11.2017 um 23:20 schrieb Wol's lists:
> On 29/11/17 20:02, Reindl Harald wrote:
>> why not RAID5/6? besides 
>> https://www.askdbmgt.com/why-raid5-should-be-avoided-at-all-costs.html 
>> the parity data are additional writes wearing out the drives
> 
> So, if I have a four-drive raid 5, for every 3 blocks of data I write I 
> write 1 parity block. But with raid 1 or 10, for every 3 blocks of data 
> I write, I write *3* "parity" blocks!
> 
> What was that about "the additional writes wearing out the drives" then?
> 
> (Yes, I get the write amplification thing - but if you are writing a lot 
> of data, then raid 5 needs far *fewer* writes.)

RAID10 has a lot of other benefits:

* better performance at all
* lower CPU usage
* only 1 disk is affected with reads at rebuild
* you can clone a whole phyiscal machine easily by just move
   half of the drives to a new machine and start rebuild on
   both - in the past years i cloned multiple machines that
   way which are after that in sync with "synch-machine.sh push/get"

especially the clone thing saves a lot of time when you collected over 
time machines with all sorts of use-cases and storage sizes and you just 
need to consider which one has the most similar setip to the new one

before i drive home from the office "synch-machine.sh push && power off" 
and when i arrive at the office -> CTL+ALT+F" -> Login as root -> 
"synch-machine.sh get", CTRL+ALT+F1 -> grapical login

that way of work is BTW a great backup for human mistakes :-)

Re: New setup: partitions or raw devices

[Reindl Harald]

Am 29.11.2017 um 23:25 schrieb Gandalf Corvotempesta:
> 2017-11-29 23:10 GMT+01:00 Reindl Harald <h.reindl@thelounge.net>:
>> [0:0:0:0]    disk    ATA      Samsung SSD 850  2B6Q  /dev/sda
>> [1:0:0:0]    disk    ATA      Samsung SSD 850  2B6Q  /dev/sdb
> 
> these are consumer SSDs

so what

> Even the "PRO" is consumer and a 2TB is priced about 800 USD

in doubt - so what too - nothing you buy every day and in case of a 
RAID10 i put out the drives holding the OS and data and move them to the 
next machine, change the MAC of the internal NIC in the config and i am 
done (the other network cards and the wireless card are move too)

> Look at one of Intel DC SSD, the S3610 1.6TB is about 1500 USD. Twice
> the price, 40% less capacity

and do you really need them?
for what usecase?

performance?

well, the RAID10 versus RAID1 with two nearly double priced achieves the 
same or at least nothing different you could notice in real world usage 
and if one dies that's what RAID is for (the I is for inexpensive) and a 
few years later you get a "consumer SSD" with the same size probably 
much cheaper while a ton of tests prove that the consumer SSD's suck a 
lot of writes over a long time too

Re: New setup: partitions or raw devices

[Nix]

On 29 Nov 2017, Reindl Harald said:

> Am 29.11.2017 um 23:25 schrieb Gandalf Corvotempesta:
>> Look at one of Intel DC SSD, the S3610 1.6TB is about 1500 USD. Twice
>> the price, 40% less capacity
>
> and do you really need them?
> for what usecase?

Not bricking or corrupting themselves when the power goes out.

Intel DC SSDs are the only SSDs I have *ever* heard of surviving such
tests.

(btw, if you really are overprovisioning SSDS by hand, 2--5% is not
remotely enough. 10% is not really enough. I'd go for 15--20%.

-- 
NULL && (void)

Re: New setup: partitions or raw devices

[Nix]

On 29 Nov 2017, Reindl Harald said:

> Am 29.11.2017 um 23:20 schrieb Wol's lists:
>> On 29/11/17 20:02, Reindl Harald wrote:
>>> why not RAID5/6? besides https://www.askdbmgt.com/why-raid5-should-be-avoided-at-all-costs.html the parity data are additional
>>> writes wearing out the drives
>>
>> So, if I have a four-drive raid 5, for every 3 blocks of data I write I write 1 parity block. But with raid 1 or 10, for every 3
>> blocks of data I write, I write *3* "parity" blocks!
>>
>> What was that about "the additional writes wearing out the drives" then?
>>
>> (Yes, I get the write amplification thing - but if you are writing a lot of data, then raid 5 needs far *fewer* writes.)
>
> RAID10 has a lot of other benefits:

That's not actually answering the question that was asked, y'know. If
you're against RAID 5 because the parity writes wear the drives out, you
should be much more strongly against RAID 10 for the same reason.

-- 
NULL && (void)

Re: New setup: partitions or raw devices

[Reindl Harald]

Am 01.12.2017 um 17:19 schrieb Nix:
> That's not actually answering the question that was asked, y'know. If
> you're against RAID 5 because the parity writes wear the drives out, you
> should be much more strongly against RAID 10 for the same reason

RAID10 is simple mirroring of stripes

Re: New setup: partitions or raw devices

[Wols Lists]

On 01/12/17 16:27, Reindl Harald wrote:
> 
> 
> Am 01.12.2017 um 17:19 schrieb Nix:
>> That's not actually answering the question that was asked, y'know. If
>> you're against RAID 5 because the parity writes wear the drives out, you
>> should be much more strongly against RAID 10 for the same reason
> 
> RAID10 is simple mirroring of stripes
> 
And?

The more I think about it, the more I come to the conclusion that
raid-10 is a bad idea for (a) minimising writes (and wear), and (b) for
safeguarding your data.

Yes it does have advantages, and yes I plan to put a raid-10 array on my
new system, but if reducing wear or protecting data are your priorities,
raid-10 is the wrong choice.

Cheers,
Wol

Re: New setup: partitions or raw devices

[Nix]

On 1 Dec 2017, Wols Lists stated:
> Yes it does have advantages, and yes I plan to put a raid-10 array on my
> new system, but if reducing wear or protecting data are your priorities,
> raid-10 is the wrong choice.

It depends on the access patterns. It's no worse at safeguarding data
than RAID-5 and can be better (some, but not all, combinations of
multiple disk failures will lead to no data loss, something which would
otherwise require RAID-6). It's faster at reads and much faster at
random writes. The cost: possibly quite a lot more spinning rust for the
amount of available storage. (For me, power-consumption considerations
led me to stick with RAID, though I've gone to RAID-6.)

-- 
NULL && (void)

Re: New setup: partitions or raw devices

[Reindl Harald]

Am 01.12.2017 um 18:18 schrieb Wols Lists:
> On 01/12/17 16:27, Reindl Harald wrote:
>>
>>
>> Am 01.12.2017 um 17:19 schrieb Nix:
>>> That's not actually answering the question that was asked, y'know. If
>>> you're against RAID 5 because the parity writes wear the drives out, you
>>> should be much more strongly against RAID 10 for the same reason
>>
>> RAID10 is simple mirroring of stripes
>>
> And?

why should mirroring on a different disk wear more?

> The more I think about it, the more I come to the conclusion that
> raid-10 is a bad idea for (a) minimising writes (and wear), and (b) for
> safeguarding your data.

why?

> Yes it does have advantages, and yes I plan to put a raid-10 array on my
> new system, but if reducing wear or protecting data are your priorities,
> raid-10 is the wrong choice
wrong!

rebuilds are much faster then with RAID5 or even RAID6 which is worst 
and so the degraded timeslot is smaller, a RAID10 *can* survive two 
faile disks while a RAID5 is for sure dead

RAID10 is practically a RAID1 with the performance enhancement of RAID0 
and if the linux kernel would support "writemostly" on RAID10 as it does 
with RAID1 it would be the perfect choice for a hybrid RAID for 
workloads which are mostly read-bound and so combine all worlds (RAID0 
at read from only the two SSD stripes, redundancy like RAID1 and only 
the need of 2 SSD's and two HDD's for redundancy)

RAID5 has *zero* benefits against RAID10 except costs while RAID6 is 
terrible for performance and wear out of disks

Re: New setup: partitions or raw devices

[Gandalf Corvotempesta]

Il 01/12/2017 17:18, Nix ha scritto:
>
> Not bricking or corrupting themselves when the power goes out.
>
> Intel DC SSDs are the only SSDs I have *ever* heard of surviving such
> tests.

Exactly.
And additionally, enterprise SSDs are far more reliable than consumer SSDs,
in a RAID-1, where both disks will get the same identical write pattern, 
the risk
for a double failure at the same time is high.

And if you have to proactively replace an SSD before it fails,
then your are not using "inexpesive" disks anymore.

Re: New setup: partitions or raw devices

[Gandalf Corvotempesta]

Il 02/12/2017 00:44, Reindl Harald ha scritto:
> a RAID10 can survive two faile disks while a RAID5 is for sure dead
Absolutely not.
This is a common misconception.
RAID-10 can survive two failed disks IF AND ONLY IF these disks are on 
different mirrors.

I had multiple (more than 4) RAID-10s totally lost due to double failure 
in the same mirror.
Each disk in a mirror will get the same write pattern, thus, you'll risk 
a double failure for any
firmware bugs or similiar. With SSDs this risk is higher, because you 
wear out both members
in the same way at the same time.
If the controller kicks out two adiacent disks, you'll loose everything
(this is happened to me: fully synced RAID10: disk0 was kicked out by 
the controller, RAID
survived and after a couple of minutes, disk0 was automatically 
reactivated and started a rebuild.
during the rebuild, disk1 was kicked out. RAID lost: disk0 out-of-sync 
and disk1 kicked out)

RAID-6 is *much* more safe than RAID-1/RAID-10 as it can survive ANY TWO 
disks failure,
you will loose data on the third failure.

Re: New setup: partitions or raw devices

[Nix]

On 1 Dec 2017, Reindl Harald said:

> RAID5 has *zero* benefits against RAID10 except costs while RAID6 is
> terrible for performance and wear out of disks

You still haven't explained why RAID-6 wears out disks more than RAID-10
does.

RAID-5 has one huge benefit over RAID-10: more accessible storage for a
given number of disks, and, given that power is not free, that means
lower running costs as well, as well as lower noise, lower vibration,
lower maintenance costs (since having more disks does mean you have to
replace disks more often, even if the chance of losing data when one
fails is reduced by RAID).

If RAID-5 and RAID-6 had no benefits at all over RAID-10 it is unlikely
they would still be in wide use. They are, even for new installations,
because they truly do offer benefits for some use cases. They may not
for yours, but that doesn't mean they don't for anyone.

Re: New setup: partitions or raw devices

[Reindl Harald]

Am 02.12.2017 um 14:14 schrieb Gandalf Corvotempesta:
> Il 02/12/2017 00:44, Reindl Harald ha scritto:
>> a RAID10 can survive two faile disks while a RAID5 is for sure dead
> Absolutely not.
> This is a common misconception.
> RAID-10 can survive two failed disks IF AND ONLY IF these disks are on 
> different mirrors.

i know that - thats the difference between "will" and "can"

> I had multiple (more than 4) RAID-10s totally lost due to double failure 
> in the same mirror.
> Each disk in a mirror will get the same write pattern, thus, you'll risk 
> a double failure for any
> firmware bugs or similiar. With SSDs this risk is higher, because you 
> wear out both members
> in the same way at the same time

when you are not an idiot the mirror is not the same disk or not the 
same age, guess why i replaced first two of the HDD's with SSD's and 
somewhere next year i plan to replace the remaining two too

and for 4 disk RAID10 with HDD's i use typicaly 4 different disks at all 
from as many vendors as possible (the machine in the office is from a 
time where you had 4 real vendors)

Re: New setup: partitions or raw devices

[Reindl Harald]

Am 02.12.2017 um 14:19 schrieb Nix:
> On 1 Dec 2017, Reindl Harald said:
> 
>> RAID5 has *zero* benefits against RAID10 except costs while RAID6 is
>> terrible for performance and wear out of disks
> 
> You still haven't explained why RAID-6 wears out disks more than RAID-10
> does.

what about common sense?
double parity anyone?

a write goes at least to 3 disks of the array instead two in case of a 4 
meber RAID10

> RAID-5 has one huge benefit over RAID-10: more accessible storage for a
> given number of disks, and, given that power is not free, that means
> lower running costs as well, as well as lower noise, lower vibration,
> lower maintenance costs (since having more disks does mean you have to
> replace disks more often, even if the chance of losing data when one
> fails is reduced by RAID).

the more accessible storage is the only real one

> If RAID-5 and RAID-6 had no benefits at all over RAID-10 it is unlikely
> they would still be in wide use. They are, even for new installations,
> because they truly do offer benefits for some use cases. They may not
> for yours, but that doesn't mean they don't for anyone

yes, when you need far more than 8 TB...

Re: New setup: partitions or raw devices

[Phil Turmel]

Hi Gandalf,

You and Reindl are both wrong. (-:

On 12/02/2017 08:14 AM, Gandalf Corvotempesta wrote:
> Il 02/12/2017 00:44, Reindl Harald ha scritto:
>> a RAID10 can survive two faile disks while a RAID5 is for sure dead
> Absolutely not.
> This is a common misconception.
> RAID-10 can survive two failed disks IF AND ONLY IF these disks are on
> different mirrors.

You (Gandalf) are ignoring the ability of raid10 to use multiple copies
across device counts that aren't multiples of the number of copies.  A
raid10,n3 array on five devices provides a total capacity of 1-2/3 the
capacity of one member, but can fail any two members just like a raid6.

It has blistering fast read performance, especially for parallel loads.
It has the same write performance as as similarly capacity raid0.  It
has terrible write amplification on SSDs, but is very low CPU
utilization (no parity computation and never any Read-Modify-Write cycles).

Raid10 has an valuable role to play in high performance systems, and I
use LVM on top of raid10,n3 for my own home directories, general purpose
user partitions, and the root LVs for performance-critical VMs.  I use
LVM on top of raid6 for media files, large capacity databases, and
not-so-critical VMs.

> RAID-6 is *much* more safe than RAID-1/RAID-10 as it can survive ANY TWO
> disks failure,
> you will loose data on the third failure.

You (Reindl) are crazy to argue about can-vs-will on a raid10,n2 layout.
 You're crazy to rely on any single mirror layout for data you care
about.  It is a disaster waiting to happen (during rebuild) in exactly
the same way raid5 is vulnerable during rebuild.  But raid10,n3 is just
as safe as raid6.

Phil

Re: New setup: partitions or raw devices

[Reindl Harald]

Am 02.12.2017 um 18:12 schrieb Phil Turmel:
>> RAID-6 is *much* more safe than RAID-1/RAID-10 as it can survive ANY TWO
>> disks failure,
>> you will loose data on the third failure.
> 
> You (Reindl) are crazy to argue about can-vs-will on a raid10,n2 layout.
> You're crazy to rely on any single mirror layout for data you care
> about.  It is a disaster waiting to happen (during rebuild) in exactly
> the same way raid5 is vulnerable during rebuild.  But raid10,n3 is just
> as safe as raid6

besides that this above is not a quote of me nor makes this quote sense

no, i take all the benefits RAID10 has *anyways* and when you have the 
mirrors on different drives.... but that's only *one* minor point for RAID10

probability is always a valid point when you have to make technical 
decisions be it how long needs my UPS typically give me power based on 
typical power outages, and yes the first one could be longer as anyone 
you have ever seen and you still lost

i explained a lot of other benefits of RAID10 so i have no idea why you 
hang on one nuance - the most importatnt is that i can clone a fukcing 
machine by just put half of the drives in a differnt one and start 
rebuild on both machines after that

the whole porblem of that thread in fact his dumb "but you are talking 
about SSD" - yes damned - i talk about how i would setup a machine *now* 
and how i would have done it many years ago with the knowledge of today 
while the 40 GB overprovisioning won't matter for storage size but be 
good for the disks (and no i don't need another one whining that it#s 
not enough because that's nonsense - there is no definite enough - no 
overprovisioning won't make it better at all)

Re: New setup: partitions or raw devices

[Nix]

On 2 Dec 2017, Reindl Harald outgrape:

> Am 02.12.2017 um 14:19 schrieb Nix:
>> If RAID-5 and RAID-6 had no benefits at all over RAID-10 it is unlikely
>> they would still be in wide use. They are, even for new installations,
>> because they truly do offer benefits for some use cases. They may not
>> for yours, but that doesn't mean they don't for anyone
>
> yes, when you need far more than 8 TB...

I can't figure out whether you're castigating people who need more than
8TiB, castigating people who need less, applauding people who need more
than 8TiB, applauding people who need less, saying that people who need
more than 8TiB should use RAID 10, saying that people who need more
should use RAID 6, or *what*. All I can tell is that you want to be as
unpleasant as possible while saying it. (I suspect this ambiguity is
intentional, so that no matter what people assume you meant, you can
flame them for not interpreting it the other way.)

But then that's not surprising because you've morphed again to try to
escape from people's killfiles. Back you go into mine, *yet again*.

-- 
NULL && (void)

Re: New setup: partitions or raw devices

[Reindl Harald]

Am 02.12.2017 um 21:35 schrieb Nix:
> On 2 Dec 2017, Reindl Harald outgrape:
> 
>> Am 02.12.2017 um 14:19 schrieb Nix:
>>> If RAID-5 and RAID-6 had no benefits at all over RAID-10 it is unlikely
>>> they would still be in wide use. They are, even for new installations,
>>> because they truly do offer benefits for some use cases. They may not
>>> for yours, but that doesn't mean they don't for anyone
>>
>> yes, when you need far more than 8 TB...
> 
> I can't figure out whether you're castigating people who need more than
> 8TiB, castigating people who need less, applauding people who need more
> than 8TiB, applauding people who need less, saying that people who need
> more than 8TiB should use RAID 10, saying that people who need more
> should use RAID 6, or *what*. All I can tell is that you want to be as
> unpleasant as possible while saying it. (I suspect this ambiguity is
> intentional, so that no matter what people assume you meant, you can
> flame them for not interpreting it the other way.)

i don't castigating anybody - it's simply always about price, size, 
performance and redundancy level and since disk sizes and their prices 
as well as technology changes there is no definite answer - but 
*currently* up to 8 TB storage is doable with *any* raid level without 
spend many thousands of money

for storgae sizes up to 8 TB *these days* you get away with 4 disks 
which are not that expensive, if you need 100 TB the overhead of needed 
disks will simply kill you in context of price

and for performance you have practically the same "problems" to solve, 
small-to-middle arrays with SSD and RAID10 are possible, for middle 
large arrays you can reach similar performance with RAID10 and HHD by 
just use more drives

> But then that's not surprising because you've morphed again to try to
> escape from people's killfiles. Back you go into mine, *yet again*

i could not care less about you - seriously