Migrating a RAID 5 from 4x2TB to 3x6TB ?

Migrating a RAID 5 from 4x2TB to 3x6TB ?

[Pierre Wieser]

Hi all,

I currently have an almost full RAID 5 built with 4 x 2 TB disks.
I wonder if it would be possible to migrate it to a bigger RAID 5
with 3 x 6TB new disks.

I've imagined something like that :
- successively fail, remove a 2TB disk, add a 4TB disk, wait for end of recovery on three 2TB disks
- at the end of this first phase, I have the same ~6TB RAID 5 clean group with 3 x 4TB + 1 x 2TB disks
- declare the last 2 TB disk faulty and remove it
- the RAID 5 group state goes to clean, degraded
- grow the RAID 5 group with --size=max option
- grow the RAID 5 group with --array-size=~12TB option
- last, grow the RAID 5 group with --raid-devices=3 and --backup-file=... options.

And I have tested it on a small test RAID 5 group.
As expected, this last command makes the RAID 5 group begins a reshaping operation.
But this one keeps stucked at zero.

So I have several questions :

- is it even theorically possible to grow a RAID 5 while decreasing the number of disks ?
- do you think the sequence i've imagined is correct ?
- why the reshaping operation does it stuck at zero ?

Any help or hint would be greatly appreciated :)
Thanks
Regards
Pierre

Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?

[Wols Lists]

On 08/06/15 20:28, Pierre Wieser wrote:
> Hi all,
> 
> I currently have an almost full RAID 5 built with 4 x 2 TB disks.
> I wonder if it would be possible to migrate it to a bigger RAID 5
> with 3 x 6TB new disks.
> 
Do you have a spare (I presume SATA) disk port?

> I've imagined something like that :
> - successively fail, remove a 2TB disk, add a 4TB disk, wait for end of recovery on three 2TB disks

If you've got a spare port, the newer mdadm's have, I believe, a "clone
and replace" option. Much better than failing then rebuilding.

If not, is it worth getting a SATA expansion board? If you've not got a
specialist mobo, surely a board is only going to cost a tenner or so,
and quality isn't *that* important seeing as it's only a temporary measure.

> - at the end of this first phase, I have the same ~6TB RAID 5 clean group with 3 x 4TB + 1 x 2TB disks
> - declare the last 2 TB disk faulty and remove it
> - the RAID 5 group state goes to clean, degraded
> - grow the RAID 5 group with --size=max option
> - grow the RAID 5 group with --array-size=~12TB option
> - last, grow the RAID 5 group with --raid-devices=3 and --backup-file=... options.
> 
> And I have tested it on a small test RAID 5 group.
> As expected, this last command makes the RAID 5 group begins a reshaping operation.
> But this one keeps stucked at zero.
> 
> So I have several questions :
> 
Don't think I've answered any of them, but I might have raised new ones.
I just hope the tip saves you a bit of time.

Cheers,
Wol

Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?

[Can Jeuleers]

On 08/06/15 21:28, Pierre Wieser wrote:
> Hi all,
> 
> I currently have an almost full RAID 5 built with 4 x 2 TB disks.
> I wonder if it would be possible to migrate it to a bigger RAID 5
> with 3 x 6TB new disks.

I'd recommend against it:

https://en.wikipedia.org/wiki/RAID#Unrecoverable_read_errors_during_rebuild

Jan

Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?

[Pierre Wieser]

----- Original Message -----
> I assume the motherboard you're using only has 4 sata ports?

Yes, the motherboard has 4 sata ports, and there is already a small external
sata controller. So my actual raid device has 5 disks (4+S). For the discussion,
I just omitted the spare device ;)

> On 8 June 2015 at 20:28, Pierre Wieser <pwieser@trychlos.org> wrote:
> > Hi all,
> >
> > I currently have an almost full RAID 5 built with 4 x 2 TB disks.
> > I wonder if it would be possible to migrate it to a bigger RAID 5
> > with 3 x 6TB new disks.

Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?

[Pierre Wieser]

----- Original Message -----
> On 08/06/15 20:28, Pierre Wieser wrote:
> > Hi all,
> > 
> > I currently have an almost full RAID 5 built with 4 x 2 TB disks.
> > I wonder if it would be possible to migrate it to a bigger RAID 5
> > with 3 x 6TB new disks.
> > 
> Do you have a spare (I presume SATA) disk port?

I am away from home this evening, but, no, I do not "see" any free sata port.

> > I've imagined something like that :
> > - successively fail, remove a 2TB disk, add a 4TB disk, wait for end of
> > recovery on three 2TB disks
> 
> If you've got a spare port, the newer mdadm's have, I believe, a "clone
> and replace" option. Much better than failing then rebuilding.

I was not conscious of this option. I understand that it is better because
this prevents me to pass through a "clean, degraded" state ? As I expect that
the clone time will be identical the a recovery one ?

> If not, is it worth getting a SATA expansion board? If you've not got a
> specialist mobo, surely a board is only going to cost a tenner or so,
> and quality isn't *that* important seeing as it's only a temporary measure.

Yes, you're right: it might be worth to purchase another sata controller,
as I have PCI-e free ports, enough alimentation. Maybe a small issue at
disk enclosure level. To be checked... 

> > - at the end of this first phase, I have the same ~6TB RAID 5 clean group
> > with 3 x 4TB + 1 x 2TB disks
> > - declare the last 2 TB disk faulty and remove it
> > - the RAID 5 group state goes to clean, degraded
> > - grow the RAID 5 group with --size=max option
> > - grow the RAID 5 group with --array-size=~12TB option
> > - last, grow the RAID 5 group with --raid-devices=3 and --backup-file=...
> > options.
> > 
> > And I have tested it on a small test RAID 5 group.
> > As expected, this last command makes the RAID 5 group begins a reshaping
> > operation.
> > But this one keeps stucked at zero.
> > 
> > So I have several questions :
> > 
> Don't think I've answered any of them, but I might have raised new ones.
> I just hope the tip saves you a bit of time.

Surely you have opened new pistes. Thanks.

> Cheers,
> Wol
> 
> 

Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?

[Pierre Wieser]

---- Original Message -----
> On 08/06/15 21:28, Pierre Wieser wrote:
> > Hi all,
> > 
> > I currently have an almost full RAID 5 built with 4 x 2 TB disks.
> > I wonder if it would be possible to migrate it to a bigger RAID 5
> > with 3 x 6TB new disks.
> 
> I'd recommend against it:
> 
> https://en.wikipedia.org/wiki/RAID#Unrecoverable_read_errors_during_rebuild

Oop's! I was not conscious at all of this issue. It happens that I am currenly
living dangerously as I have another RAID 5 11,5 TB device :( I had already
seen various administration issues when the size increases to this level, but
I tought this was only an issue regarding the volumes organization (not tought
deeply enough, obviously)....

Starting from your link, and searching a bit, I understand now that the 10TB
is a maximal limit for desktop-grade disks (regarding the URE at least). And
thus for any element of a RAID device which needs to be scanned at recovery
time.
Apart from my poor english, would you say I'm right with this ?

Does linux-raid have any recommandation(s) when managing more than 10TB of data ?

I may imagine:
- several smaller RAID 5 devices
- would RAID10 be a valuable solution in your opinion ?

(as a precision, all my servers have been migrated to CentOS 7.1)

Nonetheless, I thank you very much for the link, which may prevent me to
lose a big bunch of data !

> Jan
> 
Pierre

Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?

[Wols Lists]

On 09/06/15 06:23, Can Jeuleers wrote:
> On 08/06/15 21:28, Pierre Wieser wrote:
>> Hi all,
>>
>> I currently have an almost full RAID 5 built with 4 x 2 TB disks.
>> I wonder if it would be possible to migrate it to a bigger RAID 5
>> with 3 x 6TB new disks.
> 
> I'd recommend against it:
> 
> https://en.wikipedia.org/wiki/RAID#Unrecoverable_read_errors_during_rebuild
> 
> Jan
> 
Please expand! Having read the article, it doesn't seem to say anything
more than what is repeated time and time on this list - MAKE SURE YOUR
DRIVES ARE DECENT RAID DRIVES.

If you have ERC, then the odd "soft" read error doesn't matter. If you
don't have ERC, then your data is at risk when you replace a drive, and
it doesn't matter how big your drives are, it's the array size that matters.

Cheers,
Wol

Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?

[Wols Lists]

On 09/06/15 19:46, Pierre Wieser wrote:
> ---- Original Message -----
>> On 08/06/15 21:28, Pierre Wieser wrote:
>>> Hi all,
>>>
>>> I currently have an almost full RAID 5 built with 4 x 2 TB disks.
>>> I wonder if it would be possible to migrate it to a bigger RAID 5
>>> with 3 x 6TB new disks.
>>
>> I'd recommend against it:
>>
>> https://en.wikipedia.org/wiki/RAID#Unrecoverable_read_errors_during_rebuild
> 
> Oop's! I was not conscious at all of this issue. It happens that I am currenly
> living dangerously as I have another RAID 5 11,5 TB device :( I had already
> seen various administration issues when the size increases to this level, but
> I tought this was only an issue regarding the volumes organization (not tought
> deeply enough, obviously)....
> 
> Starting from your link, and searching a bit, I understand now that the 10TB
> is a maximal limit for desktop-grade disks (regarding the URE at least). And
> thus for any element of a RAID device which needs to be scanned at recovery
> time.
> Apart from my poor english, would you say I'm right with this ?

I don't think so. You may be lucky, and your drives are better than
average. You may be unlucky, and your drives are worse than average.

DON'T USE DESKTOP GRADE DISKS EXCEPT IN RAID 1. That said, I'm using
Seagate Barracudas which I was planning to upgrade to raid 5 - not a
good idea :-(
> 
> Does linux-raid have any recommandation(s) when managing more than 10TB of data ?
> 
> I may imagine:
> - several smaller RAID 5 devices
> - would RAID10 be a valuable solution in your opinion ?

Read the list archive. There's a bunch of stuff about how to mitigate
the problem - mostly by increasing the raid timeout (the problem is,
basically, that the raid software returns with an error before the disk
times out - increase the raid timeout and it will detect the disk error
and retry).
> 
> (as a precision, all my servers have been migrated to CentOS 7.1)
> 
> Nonetheless, I thank you very much for the link, which may prevent me to
> lose a big bunch of data !
> 
"man smartctl" is your friend :-)

You'll have to read up, but try

"smartctl -i /dev/sdx"

That'll tell you if smart is turned on - if it isn't, turn it on.

"smartctl -s on /dev/sdx"

Then do

"smartctl -x /dev/sdx"

and look for anything about ERC or Error Recovery Control. From my
Barracudas I get

SCT Data Table command not supported

SCT Error Recovery Control command not supported

Device Statistics (GP Log 0x04) not supported

OOPS!!! These drives are NOT NOT NOT suitable for raid :-( Everything
will be fine if I increase the raid timeout, but given that the typical
drive timeout is two minutes, the raid timeout needs to be longer than
that which means if I have any soft errors, the rebuild will be horribly
slow.

Looks like my next drives will be WD Reds, they're not much more expensive.

WARNING: If you have to enable smartctl, it's supposed to survive a
cold-boot, but it doesn't look like it has on my drives, and it's
reported a lot of drives don't. You need to make sure you have a boot
script that forces it on, and forces ERC on or sets the timeout.


All that said, as you can see, desktop drives are fine for raid IF
repeat IF you take the necessary precautions. They're probably fine on a
desktop :-)

Cheers,
Wol

Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?

[Another Sillyname]

If you can try to pick up a cheap IBM M1015 Raid controller that you
can configure as JBOD (look to pay no more then £50 or $75).

Flash it to the standard LSI firmware (google M1015 LSI
firmware...there's plenty of forum pages on this).

Add your new drives to this controller, build new raid on this new
controller......copy dataset from old array.

I personally would NEVER suggest doing a migrate of drives or data
sizes on an active live dataset...if anything goes wrong you could
easily kill your data and have no backup.



On 9 June 2015 at 19:46, Wols Lists <antlists@youngman.org.uk> wrote:
> On 09/06/15 06:23, Can Jeuleers wrote:
>> On 08/06/15 21:28, Pierre Wieser wrote:
>>> Hi all,
>>>
>>> I currently have an almost full RAID 5 built with 4 x 2 TB disks.
>>> I wonder if it would be possible to migrate it to a bigger RAID 5
>>> with 3 x 6TB new disks.
>>
>> I'd recommend against it:
>>
>> https://en.wikipedia.org/wiki/RAID#Unrecoverable_read_errors_during_rebuild
>>
>> Jan
>>
> Please expand! Having read the article, it doesn't seem to say anything
> more than what is repeated time and time on this list - MAKE SURE YOUR
> DRIVES ARE DECENT RAID DRIVES.
>
> If you have ERC, then the odd "soft" read error doesn't matter. If you
> don't have ERC, then your data is at risk when you replace a drive, and
> it doesn't matter how big your drives are, it's the array size that matters.
>
> Cheers,
> Wol
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 840 bytes --]

On Tue, 9 Jun 2015 20:46:12 +0200 (CEST)
Pierre Wieser <pwieser@trychlos.org> wrote:

> Does linux-raid have any recommandation(s) when managing more than 10TB of data ?
> 
> I may imagine:
> - several smaller RAID 5 devices
> - would RAID10 be a valuable solution in your opinion ?

The solution is called RAID6.

And don't go for overly large drives, e.g. 6x2TB is better than 4x3TB.

1) smaller drives might use more proven technology, require less precision, so
might be more reliable (at least according to hearsay and urban legends);

2) you need a considerable array member count anyway, to justify "losing" two
drives for parity in RAID6.

"But my enclosure only fits 4 drives". "But I don't have enough SATA ports".

Get a bigger enclosure. Get a second/third SATA controller. :)

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?

[Can Jeuleers]

On 09/06/15 20:46, Wols Lists wrote:
> Please expand! Having read the article, it doesn't seem to say anything
> more than what is repeated time and time on this list - MAKE SURE YOUR
> DRIVES ARE DECENT RAID DRIVES.

Large RAID5 arrays are a bad idea because of the probability of
unrecoverable read errors occurring during a rebuild, which increases
with the size of the array.

Decent RAID drives will have better reliability than indecent-ones
(haha), but in absolute terms their URE rates are still going to
increase with array size.

> If you have ERC, then the odd "soft" read error doesn't matter. If you
> don't have ERC, then your data is at risk when you replace a drive, and
> it doesn't matter how big your drives are, it's the array size that matters.

Indeed. Pierre was proposing to further increase the size of his RAID5
array, and I was advising him against it because of the above.

Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?

[Pierre Wieser]

Hi all,

> I currently have an almost full RAID 5 built with 4 x 2 TB disks.
> I wonder if it would be possible to migrate it to a bigger RAID 5
> with 3 x 6TB new disks.

Due to all suggestions, and I'd wish another time thank all contributions,
I've spent some hours reading the mailing list archives, surfing the web
with more appropiate keywords, and so on..

So, here what I now plan to do:

First, I hace cancelled my order for the new 6TB desktop-grade disks, 
replacing it with 4TB WD RedPro, and one 6TB desktop-grade (see below its use)

As the full RAID5 array I planned to migrate is already my backup system,
I cannot rely on a restore :(. So the first thing is to rsync the current
array to the directly attached 6TB disk. I don't thing I have a free SATA 
port on my motherboard, but at worst I will be able to use the one currently
used for the DVD drive.

I've chosen to build new RAID10 arrays.
I've moved away the RAID6 suggestion due to its known bad write performance,
and also because I'm willing/able to put a bit more money to get better perfs.

The 4x4TB new disks will be partitioned as:
- 512MB to be a RAID1 array mounted as /boot
- 8GB to be a RAID10 array used as swap
- two 25 GB parts to be two RAID10 arrays used as root filesystem
  (plus place for an alternate when upgrading the OS)
- the rest of the disk will be splitted in four equal parts (about 930 MB 
I think), each of which being member of a separate data RAID10 array.

I am conscious that this seems as a waste of space, and especially for the
/boot partition. But this scheme will let me:
a) have banalized disks: all disks have same rules, are partitioned identically
b) replace my system disk which is not part of any RAID system as of today,
thus gaining actually both a SATA port for the RAID systems and more security
for the boot and root filesystems
c) also because I use to use LVM on top of RAID to get advantages of its
flexibility (so several PVs which may or may not be aggregated later)

Other suggestions include the use of smartctl tool. I've checked that the 
daemon was already running. But I didn't use the '-x' option that I understand
is hardly an option !

I plan to build these RAID devices out of CentOS 7 standard install process
(I'm currently downloading a CentOS Live iso), thus presenting to the install
some predefined partitions.

I expect about 5-10 days to get these orders delivered. So more news at this time :)

Thank you all for your help. I keep reading the list that I discovered for 
the occasion...

Regards
Pierre

Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?

[Wilson, Jonathan]

On Wed, 2015-06-10 at 21:37 +0200, Pierre Wieser wrote:
> Hi all,
> 
> > I currently have an almost full RAID 5 built with 4 x 2 TB disks.
> > I wonder if it would be possible to migrate it to a bigger RAID 5
> > with 3 x 6TB new disks.
> 
> Due to all suggestions, and I'd wish another time thank all contributions,
> I've spent some hours reading the mailing list archives, surfing the web
> with more appropiate keywords, and so on..
> 
> So, here what I now plan to do:
> 
> First, I hace cancelled my order for the new 6TB desktop-grade disks, 
> replacing it with 4TB WD RedPro, and one 6TB desktop-grade (see below its use)
> 
> As the full RAID5 array I planned to migrate is already my backup system,
> I cannot rely on a restore :(. So the first thing is to rsync the current
> array to the directly attached 6TB disk. I don't thing I have a free SATA 
> port on my motherboard, but at worst I will be able to use the one currently
> used for the DVD drive.
> 
> I've chosen to build new RAID10 arrays.
> I've moved away the RAID6 suggestion due to its known bad write performance,
> and also because I'm willing/able to put a bit more money to get better perfs.

First I am not an expert, the following is based on multiple web sites
so is kind of cobbled together and using your setup but based on my
system as it stands now.

I'm going to make some assumptions here... 
1) the motherboard can see and boot from 4TB drives instead of only
seeing 750G (approx) in the bios, if not you will need a smaller disk
for the boot/os. 
2) this will be a "bios" install/boot or UEFI with CSM to simulate a
bios install/boot
3) these will be the only disks in the system. 


> 
> The 4x4TB new disks will be partitioned as:

As this will be a clean install, make a 1M partition flagged as "bios
boot" (EF02 in gdisk) this will allow grub2 to install into the member
(as normal) and its larger next stage loader & raid "drivers/ability" to
be installed into the "bios boot" partition, do this for all 4 disks.
(see #a later)

> - 512MB to be a RAID1 array mounted as /boot

Of the 4 drives 512M partitions, create a 4 way raid1 for /boot
(grub2/config and the kernels & initiramfs will live in here) (see #b
later)

> - 8GB to be a RAID10 array used as swap

On the 4 disks, create 17G partitions then create a 4 disk raid10 far2
array with 64K chunk. This will give you a swap file of 34G in size
(well over provisioned, but doesn't hurt or impact performance). As its
likely swap access will be in small random amounts this means the disk
write size is not overly large, no point in writing/reading 512K chunks
(the current default) for a 4K page swap/memory access; raid10 is fast;
far2 from what I've read also improves the speed of read/writes in some
tests (I don't know why or if the tests I've seen mentioned on the web
are accurate for the type of access swap will cause but on my setup I
can get a dd speed of 582M read and 215M write from drives with a single
device speed of about 80-100M as a rough and ready speed test).

> - two 25 GB parts to be two RAID10 arrays used as root filesystem
>   (plus place for an alternate when upgrading the OS)

25G partition on all 4 disks in to a single raid10 far2 (default 512K
chunk) = 50G for "root"

Duplicate above for a second "root/install" (this might be useful for #b
later also)

> - the rest of the disk will be splitted in four equal parts (about 930 MB 
> I think), each of which being member of a separate data RAID10 array.

I would not bother creating 4 smaller partitions on each disk, nothing
will be gained except more complexity and may even reduce speeds due to
increasing seeks when data doesn't reside exactly on one raid group. LVM
can still sit on the top for flexibility later. You could also go for a
4 disk raid6 (which I have) which would give you the same amount of
storage space on creation but would then mean 1 extra disk=1 extra disks
worth of space, not half,as you add more. (I'm not sure about R/W
speeds, also while I think it can - I'm not sure if mdadm --grow works
on raid10)

> 
> I am conscious that this seems as a waste of space, and especially for the
> /boot partition. But this scheme will let me:
> a) have banalized disks: all disks have same rules, are partitioned identically

I have found with GPT/raid etc. that as time has gone on I have created
partitions with the same "number" as the md/X numbering, while not
needed it does mean I know "/dev/md/3" is made up of /dev/sd[a-d]3 so if
at some future point I add more disks and create a new array I do it by
creating partition number(s) "4" and array /dev/md/4 instead of having a
bunch of partition "1"s with a multitude of differing number mdadm
arrays which gives my brain a kick to remind me that "no you can't
delete that partition because it doesn't match the array number you are
doing stuff with".

> b) replace my system disk which is not part of any RAID system as of today,
> thus gaining actually both a SATA port for the RAID systems and more security
> for the boot and root filesystems

See my assumption 1, on my old P45DE core2/quad system linux can happily
see big drives (over 2TB I think is the limit) and use all the space as
one large partition or further divided, but the bios could only see a
smaller 750G amount so could not boot from my 3TB drives so while I did
all the partitioning mentioned in my replies (ready for when I upgraded
to newer hardware, which I have done) I needed a 1TB disk to hold the
"bios boot," "/boot," and "root" to be able to then see the larger
drives. (actually strictly speaking you could probably get away with
just "bios boot" and "/boot" on the smaller disk, and have /root on the
larger ones as grub2 loads the kernel file and initramfs from /boot...
I'm not 100% sure but I think grub2 can also see and understand larger
disks, so you might be able to install grub2 to the small disk (or flash
drive), which the bios can then boot from, which can then load the
kernel from the large disk's /boot raid.)

> c) also because I use to use LVM on top of RAID to get advantages of its
> flexibility (so several PVs which may or may not be aggregated later)
> Other suggestions include the use of smartctl tool. I've checked that the 
> daemon was already running. But I didn't use the '-x' option that I understand
> is hardly an option !
> 
> I plan to build these RAID devices out of CentOS 7 standard install process
> (I'm currently downloading a CentOS Live iso), thus presenting to the install
> some predefined partitions.
> 
> I expect about 5-10 days to get these orders delivered. So more news at this time :)
> 
> Thank you all for your help. I keep reading the list that I discovered for 
> the occasion...

(#a) After installing to sda and booting etc, you than then install grub
on to sd[b,c,d]. This means that should you lose sda, you can boot from
any of the remaining disks without having to worry about getting a
"live" cd or some such method of recovering the system. 

(#b) Should you upgrade to a UEFI motherboard and/or disable CSM remove
the array on the 4 disks (512M partitions), mark them as EF00 (EFI
System) in gdisk, format them all as fat32, install the boot loader
& /boot to disk sda2 to get a working system, replicate sda2 into
sd[b,c,d]2, to allow recovery should sda fail, and use efibootmgr to add
boot entries to NVRAM for disks b2,c2,d2. (I think grub install to each
disk under uefi should also add boot entries to the uefi NVRAM but UEFI
is much more of a pain than "bios" with stupid things such as forgetting
its entries if a disk is removed/replaced, so efibootmgr is a tool to
get used to)



> 
> Regards
> Pierre

Jon.

(all the above is based on my experience/hassles as an "end user/self
learner" and various web searches and posts on this list, so may be
totally different advice from what a systems administrator would give
for a work server set up with way more experience and knowledge of just
what works best, and why, especially system/raid performance which is an
art that an end user doesn't really have to worry about as "its fast
enough/it works ok" usually suffices.)

> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?

[Wilson, Jonathan]

On Tue, 2015-06-09 at 19:46 +0100, Wols Lists wrote:
> On 09/06/15 06:23, Can Jeuleers wrote:
> > On 08/06/15 21:28, Pierre Wieser wrote:
> >> Hi all,
> >>
> >> I currently have an almost full RAID 5 built with 4 x 2 TB disks.
> >> I wonder if it would be possible to migrate it to a bigger RAID 5
> >> with 3 x 6TB new disks.
> > 
> > I'd recommend against it:
> > 
> > https://en.wikipedia.org/wiki/RAID#Unrecoverable_read_errors_during_rebuild
> > 
> > Jan
> > 
> Please expand! Having read the article, it doesn't seem to say anything
> more than what is repeated time and time on this list - MAKE SURE YOUR
> DRIVES ARE DECENT RAID DRIVES.
> 
> If you have ERC, then the odd "soft" read error doesn't matter. If you
> don't have ERC, then your data is at risk when you replace a drive, and
> it doesn't matter how big your drives are, it's the array size that matters.

TLER doesn't actually affect the raid or its integrity compared to
non-tler drives (well strictly it _might_ as drives with tler might have
better lifespans, might have longer warranties (which suggests better
life), might have better URE rates, etc.) but the difference to how
mdadm handles things is actually down to the way the block device layer
handles things.

From what I can tell, with TLER the disk just gives up and reports an
error very quickly, this is then passed up the stack to the raid layer
which then tries to resolve the problem using various methods... a TLER
"error" does not mean the device is kicked, only if mdadm can't resolve
the problem does the device get booted. (I think it tries to recover the
data then tries to write recovered data back to the device, only if this
fails does the disk get booted)

Without TLER the disk tries to sort its own problems out instead of
reporting an error, this might take a long time, it might try to resolve
the problem forever in one long endless loop. The block layer (sdX)
knows it asked for something to happen, it gets bored and decides its
taken to long for the disk to return data so it decides that the disk no
longer exists, it (the device block layer as far as I can tell) kicks
the disk then passes on a message to mdadm that the disk is down for the
count and has been booted from the system.

I don't know who sets the block layer time out, or if it varies
depending on if the disk is a "file system" or is "a raid member" but
someone decided that after a few seconds the device should disappear/be
marked as bad within the system to prevent the raid from stalling, or as
a "normal" disk/file system various types of errors up to and including
a complete crash.

By setting the time out in the block layer /sys/block/sdX/device/timeout
to a high(ish) value the raid will stall (not a problem for most end
users, big no-no for a high end data server with 100's of users relying
on quick responses) or "hang" on a "normal" disk producing a frozen
screen or what not to the end user... while a pain, better than a disk
fail especially if eventually the disk internally manages to sort the
problem and give valid data back instead of the system crashing.

I set the block time out to 180 seconds on all disks (3 mins) for disks
with TLER enabled they will still give up and send an error message up
the stack in less than 7 seconds, for my other "green" drives with no
TLER they will try their best to recover and if not will eventually pass
the error up to the block layer or after 3 mins the block layer will
report they timed out to either mdadm or to the file system.

Unlike with mdadm and the block device which can be tuned, a hardware
raid will give up on the drive after 7 seconds and kick it (which is why
you should only use raid/TLER drives in a HW raid); at least with mdadm,
specifically the block device layer, depending on the type of drive and
how much internal (to the disk) error recovery is performed and how
important response times are you can use any old disk with mdadm raid
with no problems. 
It should also be noted that the same issue would happen without raid, a
pause/hang or a drive marked as failed and/or the system crashing if the
block layer gives up or an error message passed up to the file system if
the disk has TLER and is used in a non raid way... how the file system
handles it is up to the file system.

> 
> Cheers,
> Wol
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?

[Wols Lists]

On 15/06/15 11:46, Wilson, Jonathan wrote:
> On the 4 disks, create 17G partitions then create a 4 disk raid10 far2
> array with 64K chunk. This will give you a swap file of 34G in size
> (well over provisioned, but doesn't hurt or impact performance). As its
> likely swap access will be in small random amounts this means the disk
> write size is not overly large, no point in writing/reading 512K chunks
> (the current default) for a 4K page swap/memory access; raid10 is fast;
> far2 from what I've read also improves the speed of read/writes in some
> tests (I don't know why or if the tests I've seen mentioned on the web
> are accurate for the type of access swap will cause but on my setup I
> can get a dd speed of 582M read and 215M write from drives with a single
> device speed of about 80-100M as a rough and ready speed test).

Bear in mind that linux will all by itself do a raid-0 on your swap
partitions if you ask it to.

I *always* size my swap partitions at twice mobo max ram. If you read
the release notes for linux 2.4.early, you'll notice Linus says "if you
have swap, it MUST be twice ram or more" - that was a kernel panic if
you ignored it ! Given that people have been saying that rule was
obsolete since before linux was born, and that while a lot of water has
passed under the bridge since then I've seen and heard nothing to tell
me that the fundamentals have changed ... so because my mobo maxes out
at 16MB ram, all my disks have a 32GB swap partition each.

If you want to raid10 that lot, fine, I just set equal priority on all
my swap partitions, and linux will raid-0 it for me.

The other thing to bear in mind, it's all very well doing speed tests on
your drive, but if you're hammering swap and backing store at the same
time, your speeds are going to plummet as your drive starts seeking all
over the shop ... mind you, with a decent amount of ram you probably
won't need swap at all.

Cheers,
Wol

Cheers,
Wol