Linux raid wiki - setting up a system - advice wanted :-)

Linux raid wiki - setting up a system - advice wanted :-)

[Wols Lists]

This is a great way for learning lots about raid :-)

I'm planning a section on setting up a new system, and I need to know
what will happen if you give entire drives to mdadm.

Does it leave the first 2 megs empty? Basically, what I'm asking is if I do

mdadm --create /dev/md/bigarray -add /dev/sda /dev/sdb /dev/sdc

(note I am passing the entire drive, not the first partition) and then I
install grub on those drives, will I trash the array?

Cheers,
Wol

Re: Linux raid wiki - setting up a system - advice wanted :-)

[Francisco Parada]

Hi Wols,

Based on my own experience, you can do it without trashing the array.
However, I should note, that I have never done it this way to an array
that I was booting from.  But, as long as you've set up GPT to account
for the 2MB boundary.  If you use "parted" or the graphical
equivalent, "gparted", you can account for that in newer drives above
the 2TerraByte capacity anyway with 1MB instead of 2MB.  So if you add
a little extra padding, say 3MB (2MB for your grub, 1MB for a blank
section that all drives above 2TB require), you should be in good
shape.

Perhaps someone else can chime in also, to confirm.

On Sun, Sep 25, 2016 at 5:16 PM, Wols Lists <antlists@youngman.org.uk> wrote:
> This is a great way for learning lots about raid :-)
>
> I'm planning a section on setting up a new system, and I need to know
> what will happen if you give entire drives to mdadm.
>
> Does it leave the first 2 megs empty? Basically, what I'm asking is if I do
>
> mdadm --create /dev/md/bigarray -add /dev/sda /dev/sdb /dev/sdc
>
> (note I am passing the entire drive, not the first partition) and then I
> install grub on those drives, will I trash the array?
>
> Cheers,
> Wol
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux raid wiki - setting up a system - advice wanted :-)

[Andreas Klauer]

On Sun, Sep 25, 2016 at 10:16:24PM +0100, Wols Lists wrote:
> I need to know what will happen if you give entire drives to mdadm.

Installers will pick unpartitioned disks first. Forget just trashing 
the metadata, easy to accidentally write across the entire disk.

This is what it looks like when installing Windows: http://imgur.com/a/GtcR2 

Same can happen with Linux installers. Unpartitioned disks are just unusual. 

Not sure why this is a thing anyway. There's no downside to partitions. 
Adds a safety margin, is yet another place that has metadata (with GPT 
you can use mdnumber-role as partition name / partlabel), doesn't harm 
performance in any way...

People panic too much about partition alignment? But alignment is something 
you need to provide through all layers, all the way down to the filesystem, 
not just partitions. Besides, MiB alignment has been standard for years now, 
so this shouldn't be a problem.

The only other obscure issue with partition tables I can think of is 
enclosures for USB-HDD that emulate the wrong sector size (4K vs 512) 
and unfortunately GPT still depends on the sector size; and Linux is 
not flexible/smart enough to support alien sector size GPT partitions.

So if you switch HDD enclosures you might be forced to recreate 
the partition table before you can access your data.

Regards
Andreas Klauer

Re: Linux raid wiki - setting up a system - advice wanted :-)

[Adam Goryachev]

On 26/09/16 12:16, Andreas Klauer wrote:
> On Sun, Sep 25, 2016 at 10:16:24PM +0100, Wols Lists wrote:
>> I need to know what will happen if you give entire drives to mdadm.
> Installers will pick unpartitioned disks first. Forget just trashing
> the metadata, easy to accidentally write across the entire disk.
>
> This is what it looks like when installing Windows: http://imgur.com/a/GtcR2
>
> Same can happen with Linux installers. Unpartitioned disks are just unusual.
>
> Not sure why this is a thing anyway. There's no downside to partitions.
> Adds a safety margin, is yet another place that has metadata (with GPT
> you can use mdnumber-role as partition name / partlabel), doesn't harm
> performance in any way...
>
> People panic too much about partition alignment? But alignment is something
> you need to provide through all layers, all the way down to the filesystem,
> not just partitions. Besides, MiB alignment has been standard for years now,
> so this shouldn't be a problem.
>
> The only other obscure issue with partition tables I can think of is
> enclosures for USB-HDD that emulate the wrong sector size (4K vs 512)
> and unfortunately GPT still depends on the sector size; and Linux is
> not flexible/smart enough to support alien sector size GPT partitions.
>
> So if you switch HDD enclosures you might be forced to recreate
> the partition table before you can access your data.
>
Personally, I agree, avoiding a partition table has almost zero benefit. 
Having a partition table can help massively (ie, clearly identifies the 
drive as in-use, shows the content of the drive/partition (RAID), etc....
I would think using a USB interfaced drive in a raid array is hopefully 
not common, and changing the enclosure should be even less common, 
though perhaps likely when dealing with failures.... Can you comment on 
the behaviour of removing the drive from the enclosure and direct 
connecting it? What is the worst case scenario here?
When you say forced to recreate the partition table, I assume in the 
majority of cases it is just delete and re-create using 100% of 
available space, or is there some other difference (eg, the gap at the 
beginning of the drive that might require some searching for the right 
value)?

Regards,
Adam



-- 
Adam Goryachev Website Managers www.websitemanagers.com.au

Re: Linux raid wiki - setting up a system - advice wanted :-)

[Wols Lists]

On 26/09/16 03:16, Andreas Klauer wrote:
> On Sun, Sep 25, 2016 at 10:16:24PM +0100, Wols Lists wrote:
>> > I need to know what will happen if you give entire drives to mdadm.
> Installers will pick unpartitioned disks first. Forget just trashing 
> the metadata, easy to accidentally write across the entire disk.
> 
> This is what it looks like when installing Windows: http://imgur.com/a/GtcR2 
> 
> Same can happen with Linux installers. Unpartitioned disks are just unusual. 
> 
> Not sure why this is a thing anyway. There's no downside to partitions. 
> Adds a safety margin, is yet another place that has metadata (with GPT 
> you can use mdnumber-role as partition name / partlabel), doesn't harm 
> performance in any way...

Actually, there IS a downside, which is what I'm getting at.

Bare metal -> partitions -> raid -> lvm -> partions ...

I'm a DB guy by trade. I hate relational DBs with a vengeance - because
they are necessarily complex thanks to relational theory but because
they are also so totally UNnecessary if people weren't wedded to the
(totally impractical in the real world) maths!

I know what I'm doing here. I'm very bright. And I'm trying to work out
how to explain myself without leaving your "bear of very little brain"
in charge of sys-adminning a server scratching his head in confusion
trying to work out what goes where.

As far as linux is concerned, a block device is a block device. But the
poor sysadmin has got to get his head round what goes where, and my
experience with DBs tells me that most people probably aren't as bright
as us ...

At the end of the day, I don't want to recommend anything. I simply want
to know - is it *possible*. Unfortunately, I don't have the hardware to
try it myself :-( The setup I want to know is

Bare metal -> raid [-> lvm] -> /

Is there any room on the disk to install grub?

(Note that - and I know you shouldn't believe everything you read on the
internet - apparently Neil Brown prefers passing the entire
unpartitioned disk to raid ...)

Cheers,
Wol

Re: Linux raid wiki - setting up a system - advice wanted :-)

[keld]

Hi

If you do not have a partition table, you cannot  have different partitions
on the disks.

It is in many cases a good idea to have different types of raids,
and partitions for different purposes, and this is where MD RAID has some
advantages over HW RAID. For instance you want a /boot a /root a Swap and one or more
data partitions.

And then different RAID types suits the different purposes, like RAID1, RAID10 and RAID5.

Best regards
Keld

On Sun, Sep 25, 2016 at 08:59:27PM -0400, Francisco Parada wrote:
> Hi Wols,
> 
> Based on my own experience, you can do it without trashing the array.
> However, I should note, that I have never done it this way to an array
> that I was booting from.  But, as long as you've set up GPT to account
> for the 2MB boundary.  If you use "parted" or the graphical
> equivalent, "gparted", you can account for that in newer drives above
> the 2TerraByte capacity anyway with 1MB instead of 2MB.  So if you add
> a little extra padding, say 3MB (2MB for your grub, 1MB for a blank
> section that all drives above 2TB require), you should be in good
> shape.
> 
> Perhaps someone else can chime in also, to confirm.
> 
> On Sun, Sep 25, 2016 at 5:16 PM, Wols Lists <antlists@youngman.org.uk> wrote:
> > This is a great way for learning lots about raid :-)
> >
> > I'm planning a section on setting up a new system, and I need to know
> > what will happen if you give entire drives to mdadm.
> >
> > Does it leave the first 2 megs empty? Basically, what I'm asking is if I do
> >
> > mdadm --create /dev/md/bigarray -add /dev/sda /dev/sdb /dev/sdc
> >
> > (note I am passing the entire drive, not the first partition) and then I
> > install grub on those drives, will I trash the array?
> >
> > Cheers,
> > Wol
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux raid wiki - setting up a system - advice wanted :-)

[keld]

On Sun, Sep 25, 2016 at 10:16:24PM +0100, Wols Lists wrote:
> This is a great way for learning lots about raid :-)
> 
> I'm planning a section on setting up a new system, and I need to know
> what will happen if you give entire drives to mdadm.
> 
> Does it leave the first 2 megs empty? Basically, what I'm asking is if I do
> 
> mdadm --create /dev/md/bigarray -add /dev/sda /dev/sdb /dev/sdc
> 
> (note I am passing the entire drive, not the first partition) and then I
> install grub on those drives, will I trash the array?

There is already a section in the wiki:

https://raid.wiki.kernel.org/index.php/Preventing_against_a_failing_disk

Which should be updated for 2TB+ disks.

best regards
Keld

Re: Linux raid wiki - setting up a system - advice wanted :-)

[Phil Turmel]

On 09/26/2016 02:50 AM, Wols Lists wrote:

> Bare metal -> raid [-> lvm] -> /
> 
> Is there any room on the disk to install grub?

No.

> (Note that - and I know you shouldn't believe everything you read on the
> internet - apparently Neil Brown prefers passing the entire
> unpartitioned disk to raid ...)

I'm with Neil for my large arrays.  I partition a pair of SSDs for UEFI
boot and a raid mirror holding an LVM volume group for the OS.  All
other drives are unpartitioned, given entirely to a raid6 w/ a small
chunk size (16k lately).  A separate LVM volume group on top of that.

( I no longer use any bootloader, either, as UEFI will boot a kernel
directly that's been built with EFI_STUB and a nested initramfs. )

Phil

Re: Linux raid wiki - setting up a system - advice wanted :-)

[Wols Lists]

On 26/09/16 15:13, Phil Turmel wrote:
> On 09/26/2016 02:50 AM, Wols Lists wrote:
> 
>> Bare metal -> raid [-> lvm] -> /
>>
>> Is there any room on the disk to install grub?
> 
> No.

Ten out of ten. Thanks.

(Hint to examinees - please answer the question on the paper, not the
question you want to answer :-)
> 
>> (Note that - and I know you shouldn't believe everything you read on the
>> internet - apparently Neil Brown prefers passing the entire
>> unpartitioned disk to raid ...)
> 
> I'm with Neil for my large arrays.  I partition a pair of SSDs for UEFI
> boot and a raid mirror holding an LVM volume group for the OS.  All
> other drives are unpartitioned, given entirely to a raid6 w/ a small
> chunk size (16k lately).  A separate LVM volume group on top of that.
> 
> ( I no longer use any bootloader, either, as UEFI will boot a kernel
> directly that's been built with EFI_STUB and a nested initramfs. )
> 
That's good to know. I don't have any experience with UEFI as yet, so I
can document that as a hint to others.

Cheers,
Wol