raid10 redundancy

raid10 redundancy

[d tbsky]

Hi:
   I am curious about raid10 redundancy when created with disk numbers below:

2 disks => 1 disk tolerance
3 disks = > 1 disk  tolerance
4 disks =>  possible 2 disks?  or still only 1 disk ?

how about 5/6 disks with raid10?
thanks a lot for confirmation!!

Re: raid10 redundancy

[Xiao Ni]

Hi

It depends on which layout do you use and the copies you specify. There 
is a detailed description in `man md 3`

On 05/06/2021 01:09 PM, d tbsky wrote:
> Hi:
>     I am curious about raid10 redundancy when created with disk numbers below:
>
> 2 disks => 1 disk tolerance
> 3 disks = > 1 disk  tolerance
> 4 disks =>  possible 2 disks?  or still only 1 disk ?
>
> how about 5/6 disks with raid10?
> thanks a lot for confirmation!!
>

Re: raid10 redundancy

[d tbsky]

Xiao Ni <xni@redhat.com>
>
> Hi
>
> It depends on which layout do you use and the copies you specify. There
> is a detailed description in `man md 3`

Thanks a lot for the hint. After studying the pattern I think n2 and
o2 could survive if losing two correct disks with 4 or 5 disks. but f2
will be dead. is my understanding correct?

will n2 and o2 be promised with the same pattern even if I grow the
disk numbers, like 3->4, or 4->5?

if losing two disks will madam find out the raid can be rebuilded safely or not?

I should try this myself, but I hope I can get  some knowledge before trying...

Thanks again for any help.

Re: raid10 redundancy

[Peter Grandi]

>>> On Thu, 6 May 2021 13:09:53 +0800, d tbsky <tbskyd@gmail.com> said:

> Hi: I am curious about raid10 redundancy when created with
> disk numbers below: [...]

This looks like a "homework" question... There are a lot of
pages on the WWW that explain that in summary and detail, and
the shortest is: RAID10 only loses data if all devices that
compose one of mirror set become impaired at the same block
addresses.

More in detail I have written some posts that explain the
tradeoffs involved, they seem to be widely "misunderestimated":

https://sabi.co.uk/blog/13-one.html?130117#130117
"Comparing the resilience of RAID6 and RAID 10"

http://www.sabi.co.uk/blog/12-two.html?120218#120218
"When double parity may make some sense"

Re: raid10 redundancy

[Reindl Harald]

Am 06.05.21 um 11:57 schrieb d tbsky:
> if losing two disks will madam find out the raid can be rebuilded safely or not?

it's pretty simple

* if you lose the wrong disks all is gone
* if you lsoe the right disks no problem

RAID10 with 4 disk is prcatically a stripe over the data which is 
mirrored, wehn you lose two disks containg both mirrors of one of the 
stripes your data are done

pretty easy to test  - pull out two disks and move them to another 
machine, if you take the right ones you have on both a degraded RAID, 
can add two blank disks and resync it

that's how i clone machines for years

Re: raid10 redundancy

[Phillip Susi]

d tbsky writes:

> Xiao Ni <xni@redhat.com>
>>
>> Hi
>>
>> It depends on which layout do you use and the copies you specify. There
>> is a detailed description in `man md 3`

No, it only depends on the number of copies.  They layout just effects
the performance.

> Thanks a lot for the hint. After studying the pattern I think n2 and
> o2 could survive if losing two correct disks with 4 or 5 disks. but f2
> will be dead. is my understanding correct?

No; 2 copies means you can lose one disk and still have the other copy.
Where those copies are stored doesn't matter for redundancy, only performance.

Re: raid10 redundancy

[d tbsky]

Adam Goryachev <mailinglists@websitemanagers.com.au>
>
> I guess it depends on your definition of raid 10... In my experience it means one or more raid 1 arrays combine with raid 0, so if each raid 1 arrays had 2 members, then it is either 2, 4, 6, etc for the total number of drives.

indeed. What I want to use is linux raid10 which can be used on
2,3,4,5, etc of disk drives. so it is unlike hardware raid 10.

Re: raid10 redundancy

[d tbsky]

Xiao Ni <xni@redhat.com>
> > It depends on which layout do you use and the copies you specify. There
> > is a detailed description in `man md 3`
>
> Thanks a lot for the hint. After studying the pattern I think n2 and
> o2 could survive if losing two correct disks with 4 or 5 disks. but f2
> will be dead. is my understanding correct?

it seems not correct. I check the pattern again. I found all
layout(n2,o2,f2) tend to put second copy to neighbor disk.

so if I link all disks as a circle ring, I will be safe if there are
no two disks nearby are dead.
in other words: if a disk is dead, but the two neighbor disks around
it are alive, then I will be safe.
hope this time the logic is correct....


.

Re: raid10 redundancy

[d tbsky]

Reindl Harald <h.reindl@thelounge.net>
> Am 06.05.21 um 11:57 schrieb d tbsky:
> > if losing two disks will madam find out the raid can be rebuilded safely or not?
>
> it's pretty simple
>
> * if you lose the wrong disks all is gone
> * if you lsoe the right disks no problem

 that's seems reasonable. I will give it a try.

Re: raid10 redundancy

[d tbsky]

Peter Grandi <pg@mdraid.list.sabi.co.uk>
> This looks like a "homework" question... There are a lot of
> pages on the WWW that explain that in summary and detail, and
> the shortest is: RAID10 only loses data if all devices that
> compose one of mirror set become impaired at the same block
> addresses.

 yes it is like a "homework" question. but there is not many
information for linux raid10.
like someone who test raid10,f2  and said it is fast, but others said
raid1 is faster with fio under multiple sessions.
so I always just use raid1 when I have only two disks.
this time I have a need that the two disk linux raid1 may need to grow
to 3 or 4 disks in the future. and I found:

1. raid1 can not reshape to raid 10
2. raid10,f2 can not reshape
3. someone report at the list that raid 10,o2 reshape is broken.

the above seems leave me the only option is raid10,n2

Re: raid10 redundancy

[d tbsky]

Phillip Susi <phill@thesusis.net>
> No, it only depends on the number of copies.  They layout just effects
> the performance.

I thought someone test the performance of two ssd, raid-1 outperforms
all the layout. so maybe under ssd it's not important?

> No; 2 copies means you can lose one disk and still have the other copy.
> Where those copies are stored doesn't matter for redundancy, only performance.

 I am trying to figure out how to lose two disks without losing two
copies. in history there are time bombs in ssd (like hp ssd made by
samsung) caused by buggy firmware.
if I put the orders correct, it will be safe even the same twin ssd
died together.

Re: raid10 redundancy

[d tbsky]

Adam Goryachev <mailinglists@websitemanagers.com.au>
>
> Sounds like you want raid 6, you can lose any two drives....
> Regards
> Adam

I was using some ssd raid6. the performance and write amplification
are not so good.
Maybe it needs more tuning. but this time I want to try ssd raid10
since there will be no more four ssd disks.

Re: raid10 redundancy

[Reindl Harald]

Am 07.05.21 um 03:47 schrieb d tbsky:
> Phillip Susi <phill@thesusis.net>
>> No, it only depends on the number of copies.  They layout just effects
>> the performance.
> 
> I thought someone test the performance of two ssd, raid-1 outperforms
> all the layout. so maybe under ssd it's not important?
> 
>> No; 2 copies means you can lose one disk and still have the other copy.
>> Where those copies are stored doesn't matter for redundancy, only performance.
> 
>   I am trying to figure out how to lose two disks without losing two
> copies. in history there are time bombs in ssd (like hp ssd made by
> samsung) caused by buggy firmware.
> if I put the orders correct, it will be safe even the same twin ssd
> died together.

forget what you try to figure out, you can't control which two disks 
fail at the same time, period

you can minimize the risk by using different disk types and/or 
distribute their age (i am doing that currently by replacing all disks 
on a backup server with two weeks between)

in case of a frimware timebomb you have no chance in case of identical 
disks and when you seek a soluton whuch survives two failures then for 
the sake of god use one which is called RAID6 and not RAID10

that you can lose two disk in a RAID10 with luck is exactly that: LUCK - 
period

Re: raid10 redundancy

[Wols Lists]

On 07/05/21 02:12, d tbsky wrote:
> Adam Goryachev <mailinglists@websitemanagers.com.au>
>>
>> I guess it depends on your definition of raid 10... In my experience it means one or more raid 1 arrays combine with raid 0, so if each raid 1 arrays had 2 members, then it is either 2, 4, 6, etc for the total number of drives.
> 
> indeed. What I want to use is linux raid10 which can be used on
> 2,3,4,5, etc of disk drives. so it is unlike hardware raid 10.
> 
If you're worried about losing two drives, okay it's more disk space,
but add the third drive and go for three copies. Then adding the fourth
drive will give you extra space. Not the best but heigh ho.

Or make sure you've got a spare drive configured, so if one drive fails
the array will rebuild immediately, and your window of danger is minimised.

Cheers,
Wol

Re: raid10 redundancy

[Andy Smith]

Hello,

On Fri, May 07, 2021 at 09:47:39AM +0800, d tbsky wrote:
> I thought someone test the performance of two ssd, raid-1 outperforms
> all the layout. so maybe under ssd it's not important?

If you're referring to this, which I wrote:

    http://strugglers.net/~andy/blog/2019/06/01/why-linux-raid-10-sometimes-performs-worse-than-raid-1/

then it only matters when the devices have dramatically different
performance. In that case is was a SATA SSD and an NVMe, but
probably you could see the same with a rotational HDD and a SATA SSD.

Also, it was a bug (or rather a missing feature). RAID-10 was
missing the ability to choose to read from the least loaded device,
so it's the difference between getting 50% of your reads from the
much slower device compared to hardly any of them.

And Guoqing Jiang fixed it 2 years ago.

Cheers,
Andy

Re: raid10 redundancy

[d tbsky]

Andy Smith <andy@strugglers.net>
> On Fri, May 07, 2021 at 09:47:39AM +0800, d tbsky wrote:
> > I thought someone test the performance of two ssd, raid-1 outperforms
> > all the layout. so maybe under ssd it's not important?
>
> If you're referring to this, which I wrote:
>
>     http://strugglers.net/~andy/blog/2019/06/01/why-linux-raid-10-sometimes-performs-worse-than-raid-1/
>
> then it only matters when the devices have dramatically different
> performance. In that case is was a SATA SSD and an NVMe, but
> probably you could see the same with a rotational HDD and a SATA SSD.
> Also, it was a bug (or rather a missing feature). RAID-10 was
> missing the ability to choose to read from the least loaded device,
> so it's the difference between getting 50% of your reads from the
> much slower device compared to hardly any of them.
> And Guoqing Jiang fixed it 2 years ago.

sorry I didn't find that comprehensive report before. what I saw is
that raid10 and raid1 performance are similar and raid1 is a little
faster.
so I just use raid1 at two disks conditions these years. like the
discussion here
https://www.reddit.com/r/homelab/comments/4pfonh/2_disk_ssd_raid_raid_1_or_10/
I don't know if the situation is the same now. I will try to do my
testing. but I think in theory they are similar under multiple
process.

Re: raid10 redundancy

[Andy Smith]

Hello,

On Sat, May 08, 2021 at 09:54:03AM +0800, d tbsky wrote:
> Andy Smith <andy@strugglers.net>
> > If you're referring to this, which I wrote:
> >
> >     http://strugglers.net/~andy/blog/2019/06/01/why-linux-raid-10-sometimes-performs-worse-than-raid-1/

[…]

> sorry I didn't find that comprehensive report before.

Okay, so that wasn't what you were thinking of then.

I haven't got anything published to back up the assertion but I
haven't really noticed very much performance difference between
RAID-10 and RAID-1 on non-rotational storage since the above fix.
Most of my storage is non-rotational these days.

> what I saw is that raid10 and raid1 performance are similar and
> raid1 is a little faster.

I haven't got anything published to back up the assertion but I
haven't really noticed very much performance difference between
RAID-10 and RAID-1 on non-rotational storage since the above fix.
Most of my storage is non-rotational these days.

That does assume a load that isn't single-threaded, since a single
thread is only ever going to read from one half of an md RAID-1. It
doesn't stripe.

> so I just use raid1 at two disks conditions these years. like the
> discussion here
> https://www.reddit.com/r/homelab/comments/4pfonh/2_disk_ssd_raid_raid_1_or_10/

I must admit that as most of my storage has shifted from HDD to SSD
I've shifted away from md RAID-10, which I used to use even when
there were only 2 devices. With HDDs I felt (and measured) the
increased performance.

But with SSDs these days I tend to just use RAID-1 pairs and
concatenate them in LVM (which I am using anyway) afterwards. Mainly
just because it's much simpler and the performance is good enough.

If you need to eke out the most performance this is maybe not the
way. Certainly not the way if you need better redundancy (lose any
two devices etc). Many concerns, performance only one of them…

> I don't know if the situation is the same now. I will try to do my
> testing. but I think in theory they are similar under multiple
> process.

I think so but it's always good to see a recent test with numbers!

Cheers,
Andy

Re: raid10 redundancy

[keld]

On Fri, May 07, 2021 at 09:28:35AM +0800, d tbsky wrote:
> Reindl Harald <h.reindl@thelounge.net>
> > Am 06.05.21 um 11:57 schrieb d tbsky:
> > > if losing two disks will madam find out the raid can be rebuilded safely or not?
> >
> > it's pretty simple
> >
> > * if you lose the wrong disks all is gone
> > * if you lsoe the right disks no problem
> 
>  that's seems reasonable. I will give it a try.


My understanding is that raid10 - all layouts - are really raid0 and then raid 1 on top.
If you loose two disks you loose the whole raid in 2 out of 3 cases in a 4 disk setup.
If it was raid0 over a raid1 you would only lose the whole raid in one out of 3.

               raid1
           raid0  raid0      
           d1 d2  d3 d4


gone d1+d2  survives d3+d4
gone d1+d3  dead
gone d1+d4  dead


should be.

            raid0
         raid1  raid1
         d1 d2  d3 d4

gone d1+d2  dead
gone d1+d3  survives
gone d1+d4  survives

keld

Re: raid10 redundancy

[Phillip Susi]

keld@keldix.com writes:

> My understanding is that raid10 - all layouts - are really raid0 and
> then raid 1 on top.

Naieve implementations work that way, and this is also why they require
a an even number of disks with 4 being the minimum.  Linux raid10 is not
naieve and can operate with any number of disks >= 2.

> If you loose two disks you loose the whole raid in 2 out of 3 cases in a 4 disk setup.
> If it was raid0 over a raid1 you would only lose the whole raid in one out of 3.
>
>                raid1
>            raid0  raid0      
>            d1 d2  d3 d4
>
>
> gone d1+d2  survives d3+d4
> gone d1+d3  dead
> gone d1+d4  dead

Only for the naieve implementation that considers the whole raid0 dead
as soon as one drive fails.  Linux raid10 does not have this problem
because it knows that if d1 and d3 fail, you still have a copy of all of
the required data on d2 and d4.

The bottom line is that the exact layout doesn't really matter.  You
have two copies of all data and so you can be sure that the array can
survie any one failure.  An additional failure has a 1/3 chance of
killing the whole array ( for a 4 drive array ).  Of course, you could
say, have an 8 disk array with 3 copies, then it can survive any 2
failures, and has a 5/6 chance of surviving another failure.  On the
other hand, you only get 1/3rd of the usable capacity of those 8 disks.

Re: raid10 redundancy

[Phillip Susi]

d tbsky writes:

> I thought someone test the performance of two ssd, raid-1 outperforms
> all the layout. so maybe under ssd it's not important?

A two disk raid10 in the near layout should be identical to a raid1 and
have exactly the same performance.  The benefits from the offset and far
layouts obviosuly should not help with SSDs since they have no seek penalty.

>  I am trying to figure out how to lose two disks without losing two
> copies. in history there are time bombs in ssd (like hp ssd made by
> samsung) caused by buggy firmware.
> if I put the orders correct, it will be safe even the same twin ssd
> died together.

You don't.  You don't get to pick which of the remaining disks the
second failure will take.  In my experience, the real problem with buggy
SSD firmware is that it silently corrupts data by doing nasty things
like changing the contents of an LBA to something that had been there
prior to the last completed write, or worse yet, conents of another LBA
completely.  I had an OCZ SSD that started throwing ext4 errors and fsck
complained that an inode table was corrupt and when I looked into it
with debugfs, I found the contents of some text file where the inode
table should have been, which clearly had never been written to that
LBA.

Re: raid10 redundancy

[Phil Turmel]

On 5/7/21 5:46 AM, Wols Lists wrote:
> On 07/05/21 02:12, d tbsky wrote:
>> Adam Goryachev <mailinglists@websitemanagers.com.au>
>>>
>>> I guess it depends on your definition of raid 10... In my experience it means one or more raid 1 arrays combine with raid 0, so if each raid 1 arrays had 2 members, then it is either 2, 4, 6, etc for the total number of drives.
>>
>> indeed. What I want to use is linux raid10 which can be used on
>> 2,3,4,5, etc of disk drives. so it is unlike hardware raid 10.
>>
> If you're worried about losing two drives, okay it's more disk space,
> but add the third drive and go for three copies. Then adding the fourth
> drive will give you extra space. Not the best but heigh ho.
> 
> Or make sure you've got a spare drive configured, so if one drive fails
> the array will rebuild immediately, and your window of danger is minimised.
> 
> Cheers,
> Wol
> 

I do this for my medium-speed read-mostly tasks.  Raid10,n3 across 4 or 
5 disks gives me redundancy comparable to raid6 (lose any two) without 
the CPU load of parity and syndrome calculations.

Phil

Re: raid10 redundancy

[Roy Sigurd Karlsbakk]

> Hi:
>   I am curious about raid10 redundancy when created with disk numbers below:
> 
> 2 disks => 1 disk tolerance
> 3 disks = > 1 disk  tolerance
> 4 disks =>  possible 2 disks?  or still only 1 disk ?
> 
> how about 5/6 disks with raid10?
> thanks a lot for confirmation!!

Basically, the reason to use raid10 over raid6 is to increase performance. This is particularly important regarding rebuild times. If you have a huge raid-6 array with large drives, it'll take a long time to rebuild it after a disk fails. With raid10, this is far lower, since you don't need to rewrite and compute so much. Personally, I'd choose raid6 over raid10 in most setups unless I need I lot of IOPS, where RAID6 isn't that good. RAID5 is also ok, if you don't have too much drives, but if you get a double disk failure, well, you're fucked.

PS: Always make sure to have a good backup. RAID isn't backup, it's just redundancy and won't help you if your house burns down or a filesystem messes up after a PSU problem or similar. 

Vennlig hilsen

roy
-- 
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.

Re: raid10 redundancy

[David T-G]

Phillip, et al --

...and then Phillip Susi said...
% 
% keld@keldix.com writes:
% 
% > My understanding is that raid10 - all layouts - are really raid0 and
% > then raid 1 on top.
% 
% Naieve implementations work that way, and this is also why they require
% a an even number of disks with 4 being the minimum.  Linux raid10 is not
% naieve and can operate with any number of disks >= 2.
[snip]

I've been chewing on this for a few days and I am STILL confused.  Please
help! :-)

RAID10 is striping AND mirroring (leaving out for the moment the distinction
of striping mirrors vs mirroring stripes).  How can one have both with only
two disks??  You either stripe the two disks together for more storage or
mirror them for redundant storage, but I just don't see how one could do both.


Thanks in advance :-)

:-D
-- 
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt

Re: raid10 redundancy

[Reindl Harald]

Am 12.05.21 um 19:22 schrieb David T-G:
> Phillip, et al --
> 
> ...and then Phillip Susi said...
> %
> % keld@keldix.com writes:
> %
> % > My understanding is that raid10 - all layouts - are really raid0 and
> % > then raid 1 on top.
> %
> % Naieve implementations work that way, and this is also why they require
> % a an even number of disks with 4 being the minimum.  Linux raid10 is not
> % naieve and can operate with any number of disks >= 2.
> [snip]
> 
> I've been chewing on this for a few days and I am STILL confused.  Please
> help! :-)
> 
> RAID10 is striping AND mirroring (leaving out for the moment the distinction
> of striping mirrors vs mirroring stripes).  How can one have both with only
> two disks??  You either stripe the two disks together for more storage or
> mirror them for redundant storage, but I just don't see how one could do both

by wasting even more space

the point of mirroring is that there are two phyiscal drives with each 
stripe

you can even have 20 partitions on the disks and make a big RAID10 out 
of them - not that it makes sense but you can

the point of such setups is typically temporary: at the moment only two 
disks are here but i want a RAID10 at the end of the day

for such setups often you even start with a degraded RAID and later add 
drives

Re: raid10 redundancy

[David T-G]

Phil, et al --

...and then Phil Turmel said...
% 
% I do this for my medium-speed read-mostly tasks.  Raid10,n3 across 4
% or 5 disks gives me redundancy comparable to raid6 (lose any two)
% without the CPU load of parity and syndrome calculations.

I've been reading and I still need to catch up on the notation, but how
much space do you get in the end?

I'm hoping to grow our disk farm and end up with 8+ disks.  I'm more than
a bit nervous about RAID5 across a bunch of 6T (or bigger) disks, so I've
been thinking of RAID6.  That would give me 6x6 = 36T plus two parity.

Putting 8 disks in RAID10 should give me 6x4 = 24T with mirroring.
That's a pretty hefty space penalty :-(  But ...

How does RAID10 across 5 disks as above 1) work and 2) work out?  If you
had 8 disks with a huge need for space, how would y'all lay out everything?


% 
% Phil


Thanks in advance :-)

:-D
-- 
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt

Re: raid10 redundancy

[David T-G]

Reindl, et al --

...and then Reindl Harald said...
% 
% Am 12.05.21 um 19:22 schrieb David T-G:
% >
% >...and then Phillip Susi said...
% >%
% >% Naieve implementations work that way, and this is also why they require
% >% a an even number of disks with 4 being the minimum.  Linux raid10 is not
% >% naieve and can operate with any number of disks >= 2.
...
% >
% >RAID10 is striping AND mirroring (leaving out for the moment the distinction
...
% >mirror them for redundant storage, but I just don't see how one could do both
% 
% by wasting even more space

Well, that's an interesting approach :-)


% 
% the point of mirroring is that there are two phyiscal drives with
% each stripe

Right.


% 
% you can even have 20 partitions on the disks and make a big RAID10
% out of them - not that it makes sense but you can

Hmmmm...  OK.  Yes, that makes sense (logically, anyway!).


% 
% the point of such setups is typically temporary: at the moment only
% two disks are here but i want a RAID10 at the end of the day
% 
% for such setups often you even start with a degraded RAID and later
% add drives

Aha!  NOW I start to see it.  So you set up two mirrors each with only
one disk, and then you stripe them, and you get the full size before
later adding the other side of each mirror.  Right?

So it's not an end goal configuration :-)


Thanks again & HAND

:-D
-- 
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt

Re: raid10 redundancy

[Phil Turmel]

On 5/12/21 1:27 PM, David T-G wrote:
> Phil, et al --
> 
> ...and then Phil Turmel said...
> %
> % I do this for my medium-speed read-mostly tasks.  Raid10,n3 across 4
> % or 5 disks gives me redundancy comparable to raid6 (lose any two)
> % without the CPU load of parity and syndrome calculations.
> 
> I've been reading and I still need to catch up on the notation, but how
> much space do you get in the end?

One third of the total, since I'm using n3.

> I'm hoping to grow our disk farm and end up with 8+ disks.  I'm more than
> a bit nervous about RAID5 across a bunch of 6T (or bigger) disks, so I've
> been thinking of RAID6.  That would give me 6x6 = 36T plus two parity.
> 
> Putting 8 disks in RAID10 should give me 6x4 = 24T with mirroring.
> That's a pretty hefty space penalty :-(  But ...

I wouldn't use only two copies, as you cannot lose any two.  With three 
copies, 8x6T/3 = 16T usable space.  Heftier space penalty, but necessary 
to have confidence surviving a rebuild after a disk replacement.

> How does RAID10 across 5 disks as above 1) work and 2) work out? 

1)  Linux raid 10 does not layer raid 1 on top of raid 0, but implements 
"n" copies of each chunk striped across all devices.  Which is why the 
number of devices doesn't have to be a multiple of "n".

>  If you
> had 8 disks with a huge need for space, how would y'all lay out everything?

raid6.  I pretty much always layer LVM on top of mdraid.

> Thanks in advance :-)
> 
> :-D
> 

Phil

Re: raid10 redundancy

[Andy Smith]

Hello,

On Wed, May 12, 2021 at 01:22:42PM -0400, David T-G wrote:
> RAID10 is striping AND mirroring (leaving out for the moment the distinction
> of striping mirrors vs mirroring stripes).  How can one have both with only
> two disks??  You either stripe the two disks together for more storage or
> mirror them for redundant storage, but I just don't see how one could do both.

Linux RAID-1 is a mirror - all devices are identical. A reading
thread picks a device and reads from it. You get up to 1 device
worth of IOPS.

Linux RAID-10 splits the data into chunks and stripes them across
the devices with as many copies as desired (default 2). So a single
threaded reader can get up to n devices worth of IOPS.

When n is 2 and there's only 2 identical devices and the load is
multi-thread/process then you may not see much difference. Measure
it and let us know!

Cheers,
Andy

Re: raid10 redundancy

[Phillip Susi]

Roy Sigurd Karlsbakk writes:

> Basically, the reason to use raid10 over raid6 is to increase
> performance. This is particularly important regarding rebuild
> times. If you have a huge raid-6 array with large drives, it'll take a
> long time to rebuild it after a disk fails. With raid10, this is far
> lower, since you don't need to rewrite and compute so
> much. Personally, I'd choose raid6 over raid10 in most setups unless I

How do you figure that?  Sure, raid6 is going to use more CPU time but
that isn't going to be a bottleneck unless you are using some blazing
fast NVME drives.  Certainly with HDD the rebuild time is simply how
long it takes to write all of the data to the new disk, so it's going to
be the same either way.

Re: raid10 redundancy

[Phillip Susi]

David T-G writes:

> RAID10 is striping AND mirroring (leaving out for the moment the distinction
> of striping mirrors vs mirroring stripes).  How can one have both with only
> two disks??  You either stripe the two disks together for more storage or
> mirror them for redundant storage, but I just don't see how one could do both.

In the default near layout, you don't get the striping with only two
disks; it is identical to raid1.  With the offset and far layouts
though, you do get the striping and the mirroring by mirroring each
stripe elsewhere on the disk ( one stripe down in the case of offset,
and in the second half of the disk in the case of far ).

With 3 disks, even the near layout benefits from striping.

Re: raid10 redundancy

[Andy Smith]

Hello,

On Thu, May 13, 2021 at 11:46:42AM -0400, Phillip Susi wrote:
> 
> David T-G writes:
> 
> > RAID10 is striping AND mirroring (leaving out for the moment the distinction
> > of striping mirrors vs mirroring stripes).  How can one have both with only
> > two disks??  You either stripe the two disks together for more storage or
> > mirror them for redundant storage, but I just don't see how one could do both.
> 
> In the default near layout, you don't get the striping with only two
> disks; it is identical to raid1.

While the *layout* would be identical to RAID-1 in this case, there
is the difference that a single threaded read will come from both
devices with RAID-10, right?

Cheers,
Andy

Re: raid10 redundancy

[Phil Turmel]

On 5/13/21 11:38 AM, Phillip Susi wrote:
> 
> Roy Sigurd Karlsbakk writes:
> 
>> Basically, the reason to use raid10 over raid6 is to increase
>> performance. This is particularly important regarding rebuild
>> times. If you have a huge raid-6 array with large drives, it'll take a
>> long time to rebuild it after a disk fails. With raid10, this is far
>> lower, since you don't need to rewrite and compute so
>> much. Personally, I'd choose raid6 over raid10 in most setups unless I
> 
> How do you figure that?  Sure, raid6 is going to use more CPU time but
> that isn't going to be a bottleneck unless you are using some blazing
> fast NVME drives.  Certainly with HDD the rebuild time is simply how
> long it takes to write all of the data to the new disk, so it's going to
> be the same either way.
> 

No, rebuild isn't just writing to the new disk.  You have to read other 
disks to get the data to write.  In raid6, you must read at least n-2 
drives, compute, then write.  In raid10, you just read the other drive 
(or one of the other drives when copies>2), then write.

Phil

Re: raid10 redundancy

[Phillip Susi]

Andy Smith writes:

> While the *layout* would be identical to RAID-1 in this case, there
> is the difference that a single threaded read will come from both
> devices with RAID-10, right?

No, since the data is not striped, you would get *worse* performance if
you tried to do that.

Re: raid10 redundancy

[Phillip Susi]

Phil Turmel writes:

> No, rebuild isn't just writing to the new disk.  You have to read other 
> disks to get the data to write.  In raid6, you must read at least n-2 
> drives, compute, then write.  In raid10, you just read the other drive 
> (or one of the other drives when copies>2), then write.

Yes, but the data is striped across those multiple disks, so reading
them in parallel takes no more time.  At least unless you have a
memory/bus bottleneck rather than a disk bottleneck.  so again, you're
probably right if you are using blazing fast NVME drives, but not for
conventional HDD.

Re: raid10 redundancy

[Andy Smith]

Hi Phillip,

On Fri, May 14, 2021 at 10:28:52AM -0400, Phillip Susi wrote:
> Andy Smith writes:
> > While the *layout* would be identical to RAID-1 in this case, there
> > is the difference that a single threaded read will come from both
> > devices with RAID-10, right?
> 
> No, since the data is not striped, you would get *worse* performance if
> you tried to do that.

Are you absolutely sure about this? Previous posts from Neil and
others seem to contradict you:

    https://www.spinics.net/lists/raid/msg47219.html
    https://www.spinics.net/lists/raid/msg47240.html

and what I have tested and observed in the past backs up the theory
that a single threaded read against a Linux RAID-10 can use n
devices whereas a single-threaded read against Linux RAID-1 will
only use 1 device.

Cheers,
Andy

Re: raid10 redundancy

[Roy Sigurd Karlsbakk]

> Phil Turmel writes:
> 
>> No, rebuild isn't just writing to the new disk.  You have to read other
>> disks to get the data to write.  In raid6, you must read at least n-2
>> drives, compute, then write.  In raid10, you just read the other drive
>> (or one of the other drives when copies>2), then write.
> 
> Yes, but the data is striped across those multiple disks, so reading
> them in parallel takes no more time.  At least unless you have a
> memory/bus bottleneck rather than a disk bottleneck.  so again, you're
> probably right if you are using blazing fast NVME drives, but not for
> conventional HDD.

RAID10 is like RAID1+0, only a bit more fancy. That means it's basically striping across mirrors. It's *not* like RAID0+1, which is the other way, when you mirror two RAID0 sets. So when a drive dies in a RAID10, you'll have to read from one or two other drives, depending on redundancy and the number of drives (odd or even).

Vennlig hilsen

roy
-- 
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.

Re: raid10 redundancy

[Brad Campbell]

On 14/5/21 10:37 pm, Andy Smith wrote:
> Hi Phillip,
> 
> On Fri, May 14, 2021 at 10:28:52AM -0400, Phillip Susi wrote:
>> Andy Smith writes:
>>> While the *layout* would be identical to RAID-1 in this case, there
>>> is the difference that a single threaded read will come from both
>>> devices with RAID-10, right?
>>
>> No, since the data is not striped, you would get *worse* performance if
>> you tried to do that.
> 
> Are you absolutely sure about this? Previous posts from Neil and
> others seem to contradict you:
> 

No they don't. If you read those posts closely you'll see that they are using RAID10 in a striped configuration (far). So not at all like a RAID1.

Picture it this way.
You have two disks. You partition each disk in half and create a RAID0 across the first partition on each disk.
You now have a RAID0 which has the same capacity as a single disk but twice the read performance.
You have no redundancy.
You now create another RAID0 across the second partition on each disk, but you do this such that the layout is alternated making sure there is a copy of every chunk on each disk.
You then RAID1 these 2 RAID0 partitions.

Ordinarily this wouldn't provide any redundancy because a dead disk would leave you with 2 dead RAID0, however the linux RAID10 implementation knows that all the chunks are available on a single drive, they're just spread across both halves.

So they are still striped and provide the performance of a striped array, moreso because when reading the drives are effectively "short-stroked" where all data is available on the first half of each disk.

The layout is *completely* different from a RAID-1 unless you use the "near" layout, in which case it'll be pretty much the same with the same performance limitations.

This gives a better explanation that my words :
http://www.ilsistemista.net/index.php/linux-a-unix/35-linux-software-raid-10-layouts-performance-near-far-and-offset-benchmark-analysis.html?start=1

Regards,
Brad

Re: raid10 redundancy

[Phillip Susi]

Roy Sigurd Karlsbakk writes:

> RAID10 is like RAID1+0, only a bit more fancy. That means it's
> basically striping across mirrors. It's *not* like RAID0+1, which is
> the other way, when you mirror two RAID0 sets. So when a drive dies in
> a RAID10, you'll have to read from one or two other drives, depending
> on redundancy and the number of drives (odd or even).

Yes... what does that have to do with what I said?  My point was that as
long as you are IO bound, it doesn't make much difference between having
to read all of the disks in the stripe for a raid6 and having to read
some number that is possibly less than that for a raid10.  They both
take about the same amount of time as just writing the data to the new
disk.

Re: raid10 redundancy

[Wols Lists]

On 17/05/21 21:50, Phillip Susi wrote:
> 
> Roy Sigurd Karlsbakk writes:
> 
>> RAID10 is like RAID1+0, only a bit more fancy. That means it's
>> basically striping across mirrors. It's *not* like RAID0+1, which is
>> the other way, when you mirror two RAID0 sets. So when a drive dies in
>> a RAID10, you'll have to read from one or two other drives, depending
>> on redundancy and the number of drives (odd or even).
> 
> Yes... what does that have to do with what I said?  My point was that as
> long as you are IO bound, it doesn't make much difference between having
> to read all of the disks in the stripe for a raid6 and having to read
> some number that is possibly less than that for a raid10.  They both
> take about the same amount of time as just writing the data to the new
> disk.
> 
Possibly less? Or DEFINITELY less!

When rebuilding a mirror (of any sort), one block written requires ONE
block read. When rebuilding a parity array, one block written requires
one STRIPE read.

That's a hell of a lot more load on the machine. And when faced with a
production machine that needs to work (as opposed to a hobbyist machine
which can dedicate itself solely to a rebuild), you have the two
conflicting requirements that you need to finish the rebuild as quickly
as possible for data safety, but you also need the computer to do real
work. Minimising disk i/o is *crucial*.

This general attitude of "oh the computer can do everything, we don't
need to be efficient" is great - until it isn't. If it takes longer to
do the job than the time available, then you're in trouble ... been
there done that ...

Cheers,
Wol

Re: raid10 redundancy

[Phil Turmel]

On 5/17/21 6:21 PM, Wols Lists wrote:
> On 17/05/21 21:50, Phillip Susi wrote:
>>
>> Roy Sigurd Karlsbakk writes:
>>
>>> RAID10 is like RAID1+0, only a bit more fancy. That means it's
>>> basically striping across mirrors. It's *not* like RAID0+1, which is
>>> the other way, when you mirror two RAID0 sets. So when a drive dies in
>>> a RAID10, you'll have to read from one or two other drives, depending
>>> on redundancy and the number of drives (odd or even).
>>
>> Yes... what does that have to do with what I said?  My point was that as
>> long as you are IO bound, it doesn't make much difference between having
>> to read all of the disks in the stripe for a raid6 and having to read
>> some number that is possibly less than that for a raid10.  They both
>> take about the same amount of time as just writing the data to the new
>> disk.
>>
> Possibly less? Or DEFINITELY less!
> 
> When rebuilding a mirror (of any sort), one block written requires ONE
> block read. When rebuilding a parity array, one block written requires
> one STRIPE read.
> 
> That's a hell of a lot more load on the machine. And when faced with a
> production machine that needs to work (as opposed to a hobbyist machine
> which can dedicate itself solely to a rebuild), you have the two
> conflicting requirements that you need to finish the rebuild as quickly
> as possible for data safety, but you also need the computer to do real
> work. Minimising disk i/o is *crucial*.
> 
> This general attitude of "oh the computer can do everything, we don't
> need to be efficient" is great - until it isn't. If it takes longer to
> do the job than the time available, then you're in trouble ... been
> there done that ...
> 
> Cheers,
> Wol
> 

And don't forget the CPU load computing parity and syndrome.  Which can 
be significant even if the machine isn't loaded down with real work.

Re: raid10 redundancy

[Phillip Susi]

Wols Lists writes:

> When rebuilding a mirror (of any sort), one block written requires ONE
> block read. When rebuilding a parity array, one block written requires
> one STRIPE read.

Again, we're in agreement here.  What you keep ignoring is the fact that
both of these take the same amount of time, provided that you are IO bound.

> That's a hell of a lot more load on the machine. And when faced with a
> production machine that needs to work (as opposed to a hobbyist machine
> which can dedicate itself solely to a rebuild), you have the two
> conflicting requirements that you need to finish the rebuild as quickly
> as possible for data safety, but you also need the computer to do real
> work. Minimising disk i/o is *crucial*.

That is true, it does put less load on the machine giving it more time
to perform other tasks, but your original argument was that it is more
likely to fail during a rebuild.  I suppose if you take the two together
and assume the machine is busy servicing other tasks while doing the
rebuild, then both are probably going to slow down somewhat leading to
the rebuild taking a little longer, but I have a hard time believing
that it is going to be 2-3 times longer, or that it is really very
likely of having a second failure in that time.

Re: raid10 redundancy

[Reindl Harald]

Am 18.05.21 um 18:05 schrieb Phillip Susi:
> 
> Wols Lists writes:
> 
>> When rebuilding a mirror (of any sort), one block written requires ONE
>> block read. When rebuilding a parity array, one block written requires
>> one STRIPE read.
> 
> Again, we're in agreement here.  What you keep ignoring is the fact that
> both of these take the same amount of time, provided that you are IO bound.
> 
>> That's a hell of a lot more load on the machine. And when faced with a
>> production machine that needs to work (as opposed to a hobbyist machine
>> which can dedicate itself solely to a rebuild), you have the two
>> conflicting requirements that you need to finish the rebuild as quickly
>> as possible for data safety, but you also need the computer to do real
>> work. Minimising disk i/o is *crucial*.
> 
> That is true, it does put less load on the machine giving it more time
> to perform other tasks, but your original argument was that it is more
> likely to fail during a rebuild.  I suppose if you take the two together
> and assume the machine is busy servicing other tasks while doing the
> rebuild, then both are probably going to slow down somewhat leading to
> the rebuild taking a little longer, but I have a hard time believing
> that it is going to be 2-3 times longer, or that it is really very
> likely of having a second failure in that time.

it's common sense that additional load on drives which have the same 
history makes a failure one one of them more likely

Re: raid10 redundancy

[Phillip Susi]

Reindl Harald writes:

> it's common sense that additional load on drives which have the same 
> history makes a failure one one of them more likely

"It's common sense" = the logical fallacy of hand waving.  Show me
statistical evidence.  I have had lightly loaded drives die in under 2
years and heavily loaded ones last 10 years.  I have replaced failed
drives in a raid and the other drives with essentially the same wear on
them lasted for years without another failure.  There does not appear to
be a strong correlation usage and drive failure.  Certainly not one that
is so strong that you can claim with a straight face that after the
first failure, a second one can be expected within X IOPS, and the IOPS
needed to rebuild the array are a significant fraction of X.

Re: raid10 redundancy

[Reindl Harald]

Am 18.05.21 um 20:51 schrieb Phillip Susi:
> 
> Reindl Harald writes:
> 
>> it's common sense that additional load on drives which have the same
>> history makes a failure one one of them more likely
> 
> "It's common sense" = the logical fallacy of hand waving.  Show me
> statistical evidence.  I have had lightly loaded drives die in under 2
> years and heavily loaded ones last 10 years.  I have replaced failed
> drives in a raid and the other drives with essentially the same wear on
> them lasted for years without another failure.  There does not appear to
> be a strong correlation usage and drive failure.  Certainly not one that
> is so strong that you can claim with a straight face that after the
> first failure, a second one can be expected within X IOPS, and the IOPS
> needed to rebuild the array are a significant fraction of X

do what you want - others like to be better safe then sorry especially 
when there is no longer redundancy and you don't surive any error until 
the rebuild is finished

and yes i replaced last week a 365/24 for years running Seagate *desktop 
drive* in a RAID10 with 50k power up hours but that don't imply that you 
can expect that

Re: raid10 redundancy

[antlists]

On 18/05/2021 17:05, Phillip Susi wrote:
> 
> Wols Lists writes:
> 
>> When rebuilding a mirror (of any sort), one block written requires ONE
>> block read. When rebuilding a parity array, one block written requires
>> one STRIPE read.
> 
> Again, we're in agreement here.  What you keep ignoring is the fact that
> both of these take the same amount of time, provided that you are IO bound.
> 
And if you've got spinning rust, that's unlikely to be true. I can't 
speak for SATA, but on PATA I've personally experienced the exact 
opposite. Doubling the load on the interface absolutely DEMOLISHED 
throughput, turning what should have been a five-minute job into a 
several-hours job.

And if you've got many drives in your stripe, who's to say that won't 
overwhelm the i/o bandwidth. Your reads could be 50% or less of full 
speed, because there isn't the back end capacity to pass them on.

Cheers,
Wol

Re: raid10 redundancy

[Adam Goryachev]

On 19/5/21 09:48, antlists wrote:
> On 18/05/2021 17:05, Phillip Susi wrote:
>>
>> Wols Lists writes:
>>
>>> When rebuilding a mirror (of any sort), one block written requires ONE
>>> block read. When rebuilding a parity array, one block written requires
>>> one STRIPE read.
>>
>> Again, we're in agreement here.  What you keep ignoring is the fact that
>> both of these take the same amount of time, provided that you are IO 
>> bound.
>>
> And if you've got spinning rust, that's unlikely to be true. I can't 
> speak for SATA, but on PATA I've personally experienced the exact 
> opposite. Doubling the load on the interface absolutely DEMOLISHED 
> throughput, turning what should have been a five-minute job into a 
> several-hours job.
>
> And if you've got many drives in your stripe, who's to say that won't 
> overwhelm the i/o bandwidth. Your reads could be 50% or less of full 
> speed, because there isn't the back end capacity to pass them on.
>
> Cheers,
> Wol

Jumping into this one late, but I thought the main risk was related to 
the fact that for every read there is a chance the device will fail to 
read the data successfully, and so the more data you need to read in 
order to restore redundancy, the greater the risk of not being able to 
regain redundancy.

So, assuming all drives are of equal capacity, you will need to read 
less data to recover a RAID10 than you would to recover a RAID5/6, thus 
a RAID10 has a better chance to recover.

1) speed of recovery (quicker to read 1 x drive capacity instead of n x 
drive capacity even if all in parallel) (unless you can sustain full 
read speed on all devices concurrently I guess).

2) equal load on the single "mirror" device from RAID10 compared to load 
on ALL devices in the RAID5/6.

3) lower impact to operational status (ie, real work load can continue 
without impact on all reads/writes not involving the small part of the 
array being recovered, equal impact for read/write that does involve 
this part of the array).

Right?

Regards,
Adam

Re: raid10 redundancy

[Phillip Susi]

Adam Goryachev writes:

> Jumping into this one late, but I thought the main risk was related to 
> the fact that for every read there is a chance the device will fail to 
> read the data successfully, and so the more data you need to read in 
> order to restore redundancy, the greater the risk of not being able to 
> regain redundancy.

Also the assumption that the drives tend to fail after about the same
number of reads, and since all of the drives in the array have had about
the same number of reads, by the time you get the first failure, a
second likely is not far behind.

Both of these assumptions are about as flawed as the mistaken belief
that many have that based on the bit error rates published by drive
manufacturers, that if you read the entire multi TB drive, odds are
quite good that you will get an uncorrectable error.  I've tried it
many times and it doesn't work that way.

Re: raid10 redundancy

[Reindl Harald]

Am 19.05.21 um 15:02 schrieb Phillip Susi:
> 
> Adam Goryachev writes:
> 
>> Jumping into this one late, but I thought the main risk was related to
>> the fact that for every read there is a chance the device will fail to
>> read the data successfully, and so the more data you need to read in
>> order to restore redundancy, the greater the risk of not being able to
>> regain redundancy.
> 
> Also the assumption that the drives tend to fail after about the same
> number of reads, and since all of the drives in the array have had about
> the same number of reads, by the time you get the first failure, a
> second likely is not far behind.
> 
> Both of these assumptions are about as flawed as the mistaken belief
> that many have that based on the bit error rates published by drive
> manufacturers, that if you read the entire multi TB drive, odds are
> quite good that you will get an uncorrectable error.  I've tried it
> many times and it doesn't work that way.

frankly that discussion is idiotic - there is nothing like a fixed 
number of whatever load until something fails

but it's fact the *minimize whatever load* in doubt is a good thing - no 
matter what type of devices

Re: raid10 redundancy

[Adam Goryachev]

On 19/5/21 23:02, Phillip Susi wrote:
> Adam Goryachev writes:
>
>> Jumping into this one late, but I thought the main risk was related to
>> the fact that for every read there is a chance the device will fail to
>> read the data successfully, and so the more data you need to read in
>> order to restore redundancy, the greater the risk of not being able to
>> regain redundancy.
> Also the assumption that the drives tend to fail after about the same
> number of reads, and since all of the drives in the array have had about
> the same number of reads, by the time you get the first failure, a
> second likely is not far behind.
>
> Both of these assumptions are about as flawed as the mistaken belief
> that many have that based on the bit error rates published by drive
> manufacturers, that if you read the entire multi TB drive, odds are
> quite good that you will get an uncorrectable error.  I've tried it
> many times and it doesn't work that way.

Except that is not what I said. I said the risk is increased for each 
read required. I didn't say that you *will* experience a read failure. 
It's all about reducing risks without increasing costs to the point that 
there is no benefit gained. Your costs and benefits will differ from the 
next person, so there is no single answer that matches for everyone. 
Some people will say they need a minimum of triple mirror RAID10, others 
will be fine with RAID5 or even RAID0.

It sounds like you are trying to say that regardless of how many reads 
are required you will never experience a read failure? That doesn't seem 
to match what the manufacturers are saying.

Regards,
Adam

Re: raid10 redundancy

[antlists]

On 20/05/2021 12:12, Roger Heflin wrote:
> The read failures exist on the platter for the most part. Reading the 
> platters or not does not change the fact that sectors are already bad. 
> raid6 reading more sectors just means you have a higher risk of finding 
> the already corrupted sector.

I think you're wrong there ... glitches in the electronics also seem to 
be not uncommon. But here, the read normally fails, and if the OS 
retries, it succeeds without trouble.

But as I look at it, if the manufacturers specify "less than 1 error per 
X bytes read", even if X in practice is much larger than X in the spec, 
you should plan to handle what the spec says.

Cheers,
Wol