Software RAID and Fakeraid

Software RAID and Fakeraid

[John Sheu]

[-- Attachment #1: Type: text/plain, Size: 1445 bytes --]

What's the preferred way to differentiate BIOS fakeraid from regular
software mdraid?

I ask this as I'm booting with GRUB2 off a system that has one of those
Intel fakeraid chipsets.  As of a few months ago, the mdadm package has
supported these fakeraid setups, so the RAID array comes up as a /dev/md###
device.  This is unfortunate, as GRUB2 assumes that any device of the type
/dev/md### must be a pure software RAID device, and in
util/grub-setup.c:939, tries to install itself to the RAID members
individually:

if (0 && dest_dev[0] == 'm' && dest_dev[1] == 'd'
    && ((dest_dev[2] >= '0' && dest_dev[2] <= '9') || dest_dev[2] == '/'))
  {
    char **devicelist;
    int i;

    devicelist = grub_util_raid_getmembers (dest_dev);

    for (i = 0; devicelist[i]; i++)
      {
        setup (arguments.dir ? : DEFAULT_DIRECTORY,
               arguments.boot_file ? : DEFAULT_BOOT_FILE,
               arguments.core_file ? : DEFAULT_CORE_FILE,
               root_dev, grub_util_get_grub_dev (devicelist[i]), 1,
               arguments.force, arguments.fs_probe);
      }
  }

For a fakeraid setup, however, the BIOS presents the entire device as
"regular" int13 device, so GRUB2 really should be installing it to the
entire /dev/md### device, not the individual members.

So what's the preferred way to differentiate BIOS fakeraid?  Is there some
ioctl that would make this easier than having to parse /proc/mdstat?

Thanks,
-John Sheu

[-- Attachment #2: Type: text/html, Size: 1639 bytes --]

Re: Software RAID and Fakeraid

[Phillip Susi]

On 11/25/2010 5:26 AM, John Sheu wrote:
> What's the preferred way to differentiate BIOS fakeraid from regular
> software mdraid?

The only way I know of is detecting that it is a dmraid device as
opposed to md, which is why grub does it that way.  This worked well in
the past when each tool exclusively handled one type of raid.

> I ask this as I'm booting with GRUB2 off a system that has one of those
> Intel fakeraid chipsets.  As of a few months ago, the mdadm package has
> supported these fakeraid setups, so the RAID array comes up as a /dev/md###
> device.  This is unfortunate, as GRUB2 assumes that any device of the type
> /dev/md### must be a pure software RAID device, and in
> util/grub-setup.c:939, tries to install itself to the RAID members
> individually:

For grub to support fakeraids activated by the md driver, it needs some
way to find out that it is actually a fake raid, and not a software
raid.  Adding linux-raid to Cc list to see if they can suggest a way of
doing that.

Re: Software RAID and Fakeraid

[Phillip Susi]

On 11/25/2010 5:26 AM, John Sheu wrote:
> What's the preferred way to differentiate BIOS fakeraid from regular
> software mdraid?

The only way I know of is detecting that it is a dmraid device as
opposed to md, which is why grub does it that way.  This worked well in
the past when each tool exclusively handled one type of raid.

> I ask this as I'm booting with GRUB2 off a system that has one of those
> Intel fakeraid chipsets.  As of a few months ago, the mdadm package has
> supported these fakeraid setups, so the RAID array comes up as a /dev/md###
> device.  This is unfortunate, as GRUB2 assumes that any device of the type
> /dev/md### must be a pure software RAID device, and in
> util/grub-setup.c:939, tries to install itself to the RAID members
> individually:

For grub to support fakeraids activated by the md driver, it needs some
way to find out that it is actually a fake raid, and not a software
raid.  Adding linux-raid to Cc list to see if they can suggest a way of
doing that.

Re: Software RAID and Fakeraid

[Neil Brown]

On Tue, 30 Nov 2010 14:54:40 -0500 Phillip Susi <psusi@cfl.rr.com> wrote:

> On 11/25/2010 5:26 AM, John Sheu wrote:
> > What's the preferred way to differentiate BIOS fakeraid from regular
> > software mdraid?
> 
> The only way I know of is detecting that it is a dmraid device as
> opposed to md, which is why grub does it that way.  This worked well in
> the past when each tool exclusively handled one type of raid.
> 
> > I ask this as I'm booting with GRUB2 off a system that has one of those
> > Intel fakeraid chipsets.  As of a few months ago, the mdadm package has
> > supported these fakeraid setups, so the RAID array comes up as a /dev/md###
> > device.  This is unfortunate, as GRUB2 assumes that any device of the type
> > /dev/md### must be a pure software RAID device, and in
> > util/grub-setup.c:939, tries to install itself to the RAID members
> > individually:
> 
> For grub to support fakeraids activated by the md driver, it needs some
> way to find out that it is actually a fake raid, and not a software
> raid.  Adding linux-raid to Cc list to see if they can suggest a way of
> doing that.

My feeling is that grub just needs to be a bit more careful.

If the members of the md array are partitions, then installing itself in the
boot blocks of the devices holding those partitions always makes sense.

If the members of the md array are whole devices, then installing grub in
those devices might make sense depending on specific details of the
metadata.  The default should be that it doesn't make sense, but specific
cases do.
e.g. if the metadata (/sys/block/mdX/md/metadata_version) is 0.90 or 1.0, and
the array is RAID1, then grub should install itself in the *array*, not in
the devices.
If the metadata is 1.1, then grub cannot install itself
If the metadata is 1.2, then grub can install itself at the start
If the metadata is external:imsm then (I think) grub should install itself in
the array ... though there are some complexities there.

I often wonder why people who add knowledge of md to grub etc don't at least
let me know what they are doing in case I can see something obviously wrong
with their approach..

NeilBrown

Re: Software RAID and Fakeraid

[Neil Brown]

On Tue, 30 Nov 2010 14:54:40 -0500 Phillip Susi <psusi@cfl.rr.com> wrote:

> On 11/25/2010 5:26 AM, John Sheu wrote:
> > What's the preferred way to differentiate BIOS fakeraid from regular
> > software mdraid?
> 
> The only way I know of is detecting that it is a dmraid device as
> opposed to md, which is why grub does it that way.  This worked well in
> the past when each tool exclusively handled one type of raid.
> 
> > I ask this as I'm booting with GRUB2 off a system that has one of those
> > Intel fakeraid chipsets.  As of a few months ago, the mdadm package has
> > supported these fakeraid setups, so the RAID array comes up as a /dev/md###
> > device.  This is unfortunate, as GRUB2 assumes that any device of the type
> > /dev/md### must be a pure software RAID device, and in
> > util/grub-setup.c:939, tries to install itself to the RAID members
> > individually:
> 
> For grub to support fakeraids activated by the md driver, it needs some
> way to find out that it is actually a fake raid, and not a software
> raid.  Adding linux-raid to Cc list to see if they can suggest a way of
> doing that.

My feeling is that grub just needs to be a bit more careful.

If the members of the md array are partitions, then installing itself in the
boot blocks of the devices holding those partitions always makes sense.

If the members of the md array are whole devices, then installing grub in
those devices might make sense depending on specific details of the
metadata.  The default should be that it doesn't make sense, but specific
cases do.
e.g. if the metadata (/sys/block/mdX/md/metadata_version) is 0.90 or 1.0, and
the array is RAID1, then grub should install itself in the *array*, not in
the devices.
If the metadata is 1.1, then grub cannot install itself
If the metadata is 1.2, then grub can install itself at the start
If the metadata is external:imsm then (I think) grub should install itself in
the array ... though there are some complexities there.

I often wonder why people who add knowledge of md to grub etc don't at least
let me know what they are doing in case I can see something obviously wrong
with their approach..

NeilBrown

Re: Software RAID and Fakeraid

[Phillip Susi]

On 11/30/2010 5:25 PM, Neil Brown wrote:
> My feeling is that grub just needs to be a bit more careful.
> 
> If the members of the md array are partitions, then installing itself in the
> boot blocks of the devices holding those partitions always makes sense.
>
> If the members of the md array are whole devices, then installing grub in
> those devices might make sense depending on specific details of the
> metadata.  The default should be that it doesn't make sense, but specific
> cases do.
> e.g. if the metadata (/sys/block/mdX/md/metadata_version) is 0.90 or 1.0, and
> the array is RAID1, then grub should install itself in the *array*, not in
> the devices.

I don't think that is quite right.  For software raid, you can't
actually install to the array per se, since the bios does not know about
it; it only knows about the individual disks.  Therefore, grub needs to
be installed to the individual disk(s), and preferably on each member of
a raid 1 so you can still boot with a failed disk.  To do this, it needs
the embed area to place the core image into, which doesn't exist if the
array uses the whole disk instead of a partition in it.

In the case of fakeraid, the bios does know about it, so grub can and
does install itself into the array, but since this won't work with true
mdadm soft raid using the raw disks, grub needs to be able to tell the
difference.  Only seeing the members of the array are raw disks instead
of partitions is not enough information.

Re: Software RAID and Fakeraid

[Phillip Susi]

On 11/30/2010 5:25 PM, Neil Brown wrote:
> My feeling is that grub just needs to be a bit more careful.
> 
> If the members of the md array are partitions, then installing itself in the
> boot blocks of the devices holding those partitions always makes sense.
>
> If the members of the md array are whole devices, then installing grub in
> those devices might make sense depending on specific details of the
> metadata.  The default should be that it doesn't make sense, but specific
> cases do.
> e.g. if the metadata (/sys/block/mdX/md/metadata_version) is 0.90 or 1.0, and
> the array is RAID1, then grub should install itself in the *array*, not in
> the devices.

I don't think that is quite right.  For software raid, you can't
actually install to the array per se, since the bios does not know about
it; it only knows about the individual disks.  Therefore, grub needs to
be installed to the individual disk(s), and preferably on each member of
a raid 1 so you can still boot with a failed disk.  To do this, it needs
the embed area to place the core image into, which doesn't exist if the
array uses the whole disk instead of a partition in it.

In the case of fakeraid, the bios does know about it, so grub can and
does install itself into the array, but since this won't work with true
mdadm soft raid using the raw disks, grub needs to be able to tell the
difference.  Only seeing the members of the array are raw disks instead
of partitions is not enough information.

Re: Software RAID and Fakeraid

[Neil Brown]

On Thu, 02 Dec 2010 17:13:42 -0500 Phillip Susi <psusi@cfl.rr.com> wrote:

> On 11/30/2010 5:25 PM, Neil Brown wrote:
> > My feeling is that grub just needs to be a bit more careful.
> > 
> > If the members of the md array are partitions, then installing itself in the
> > boot blocks of the devices holding those partitions always makes sense.
> >
> > If the members of the md array are whole devices, then installing grub in
> > those devices might make sense depending on specific details of the
> > metadata.  The default should be that it doesn't make sense, but specific
> > cases do.
> > e.g. if the metadata (/sys/block/mdX/md/metadata_version) is 0.90 or 1.0, and
> > the array is RAID1, then grub should install itself in the *array*, not in
> > the devices.
> 
> I don't think that is quite right.  For software raid, you can't
> actually install to the array per se, since the bios does not know about
> it; it only knows about the individual disks.  Therefore, grub needs to
> be installed to the individual disk(s), and preferably on each member of
> a raid 1 so you can still boot with a failed disk.  To do this, it needs
> the embed area to place the core image into, which doesn't exist if the
> array uses the whole disk instead of a partition in it.

If the array uses 0.90 or 1.0 metadata and comprises whole-disks (not
partitions), and if the array is RAID1, then each device (except for the very
end) contains exactly the same data as the whole array.
If you install grub to the array, then it will be installed onto all of the
(active) devices in the array.  And that is certainly the easiest way to
write to all device.

It won't write to 'spares', so if you want to be able to boot from spares as
well .... but I'm not sure that makes sense anyway.

> 
> In the case of fakeraid, the bios does know about it, so grub can and
> does install itself into the array, but since this won't work with true
> mdadm soft raid using the raw disks, grub needs to be able to tell the
> difference.  Only seeing the members of the array are raw disks instead
> of partitions is not enough information.

Completely agree.  As I said, there are only some cases where you can boot
from an array which uses whole-disks.
One case if in the bios understands the array, such as Intel bios's with IMSM
metadata, or possibly some bioses with DDF metadata.
Another case is RAID1 which starts at the beginning of the device, where the
bios doesn't need to know about the RAID.

NeilBrown

Re: Software RAID and Fakeraid

[Neil Brown]

On Thu, 02 Dec 2010 17:13:42 -0500 Phillip Susi <psusi@cfl.rr.com> wrote:

> On 11/30/2010 5:25 PM, Neil Brown wrote:
> > My feeling is that grub just needs to be a bit more careful.
> > 
> > If the members of the md array are partitions, then installing itself in the
> > boot blocks of the devices holding those partitions always makes sense.
> >
> > If the members of the md array are whole devices, then installing grub in
> > those devices might make sense depending on specific details of the
> > metadata.  The default should be that it doesn't make sense, but specific
> > cases do.
> > e.g. if the metadata (/sys/block/mdX/md/metadata_version) is 0.90 or 1.0, and
> > the array is RAID1, then grub should install itself in the *array*, not in
> > the devices.
> 
> I don't think that is quite right.  For software raid, you can't
> actually install to the array per se, since the bios does not know about
> it; it only knows about the individual disks.  Therefore, grub needs to
> be installed to the individual disk(s), and preferably on each member of
> a raid 1 so you can still boot with a failed disk.  To do this, it needs
> the embed area to place the core image into, which doesn't exist if the
> array uses the whole disk instead of a partition in it.

If the array uses 0.90 or 1.0 metadata and comprises whole-disks (not
partitions), and if the array is RAID1, then each device (except for the very
end) contains exactly the same data as the whole array.
If you install grub to the array, then it will be installed onto all of the
(active) devices in the array.  And that is certainly the easiest way to
write to all device.

It won't write to 'spares', so if you want to be able to boot from spares as
well .... but I'm not sure that makes sense anyway.

> 
> In the case of fakeraid, the bios does know about it, so grub can and
> does install itself into the array, but since this won't work with true
> mdadm soft raid using the raw disks, grub needs to be able to tell the
> difference.  Only seeing the members of the array are raw disks instead
> of partitions is not enough information.

Completely agree.  As I said, there are only some cases where you can boot
from an array which uses whole-disks.
One case if in the bios understands the array, such as Intel bios's with IMSM
metadata, or possibly some bioses with DDF metadata.
Another case is RAID1 which starts at the beginning of the device, where the
bios doesn't need to know about the RAID.

NeilBrown

Re: Software RAID and Fakeraid

[Phillip Susi]

On 12/02/2010 08:36 PM, Neil Brown wrote:
> If the array uses 0.90 or 1.0 metadata and comprises whole-disks (not
> partitions), and if the array is RAID1, then each device (except for the very
> end) contains exactly the same data as the whole array.
> If you install grub to the array, then it will be installed onto all of the
> (active) devices in the array.  And that is certainly the easiest way to
> write to all device.
>
> It won't write to 'spares', so if you want to be able to boot from spares as
> well .... but I'm not sure that makes sense anyway.

Yes, for a raid1 with no spares, installing to the array is equivalent 
to installing to each individual disk, but it helps avoid confusion to 
ignore this fact and remain thinking in terms of the physical disks, at 
least as they appear to the bios.

> Completely agree.  As I said, there are only some cases where you can boot
> from an array which uses whole-disks.
> One case if in the bios understands the array, such as Intel bios's with IMSM
> metadata, or possibly some bioses with DDF metadata.
> Another case is RAID1 which starts at the beginning of the device, where the
> bios doesn't need to know about the RAID.

So how do we tell the difference?  Right now grub uses the rule of 
dmraid = bios aware, so install to the raid device, and mdadm = software 
raid, so install to the component devices individually.  You have noted 
that in same cases both methods will produce the same results, but grub 
needs to be certain that whichever method it chooses will work, whether 
or not either one will.  To do this, it needs to install to the raid 
device if and only if it is a bios recognized fakeraid.

Re: Software RAID and Fakeraid

[Neil Brown]

On Thu, 02 Dec 2010 22:15:55 -0500 Phillip Susi <psusi@cfl.rr.com> wrote:

> On 12/02/2010 08:36 PM, Neil Brown wrote:
> > If the array uses 0.90 or 1.0 metadata and comprises whole-disks (not
> > partitions), and if the array is RAID1, then each device (except for the very
> > end) contains exactly the same data as the whole array.
> > If you install grub to the array, then it will be installed onto all of the
> > (active) devices in the array.  And that is certainly the easiest way to
> > write to all device.
> >
> > It won't write to 'spares', so if you want to be able to boot from spares as
> > well .... but I'm not sure that makes sense anyway.
> 
> Yes, for a raid1 with no spares, installing to the array is equivalent 
> to installing to each individual disk, but it helps avoid confusion to 
> ignore this fact and remain thinking in terms of the physical disks, at 
> least as they appear to the bios.
> 
> > Completely agree.  As I said, there are only some cases where you can boot
> > from an array which uses whole-disks.
> > One case if in the bios understands the array, such as Intel bios's with IMSM
> > metadata, or possibly some bioses with DDF metadata.
> > Another case is RAID1 which starts at the beginning of the device, where the
> > bios doesn't need to know about the RAID.
> 
> So how do we tell the difference?  Right now grub uses the rule of 
> dmraid = bios aware, so install to the raid device, and mdadm = software 
> raid, so install to the component devices individually.  You have noted 
> that in same cases both methods will produce the same results, but grub 
> needs to be certain that whichever method it chooses will work, whether 
> or not either one will.  To do this, it needs to install to the raid 
> device if and only if it is a bios recognized fakeraid.

Could you install to both??  Maybe not...

What exactly needs to be installed?  My understanding is that you need to
install a boot block which cal load a larger chunk from some fixed location,
and the larger chunk understands raid and filesystems and everything and can
figure out what to do.
Where do you install that larger chunk??

I'd be happy to get mdadm to provide useful information for grub if I knew
what information was needed, and knew that grub would use it.
It would probably be an extension to "mdadm --detail --export".
It could:
 - list devices on which it was safe to install a boot block, which might be
   the array, or might be components, or might be nothing
 - list and offset/size on each device where there is some spare space for
   the second stage (or is that 'stage1.5'??) could be stored safely.
 - report an offset for mapping from 'the array' to 'individual device' when
   that makes sense (i.e. raid1 and maybe linear).
 - anything else required.

I'd really rather if grub didn't assume that 'md' meant something specific,
but rather asked mdadm what it should do with a given 'md' device.

So if you (or someone) can tell me what grub needs and how it would use it,
and undertakes to teach grub to ask mdadm and make use of the answers, and to
continue the dialogue if it turns out that our first design approach doesn't
quite work right,  then I will very happily get the required functionality
into mdadm very promptly.

Thanks,
NeilBrown

Re: Software RAID and Fakeraid

[Neil Brown]

On Thu, 02 Dec 2010 22:15:55 -0500 Phillip Susi <psusi@cfl.rr.com> wrote:

> On 12/02/2010 08:36 PM, Neil Brown wrote:
> > If the array uses 0.90 or 1.0 metadata and comprises whole-disks (not
> > partitions), and if the array is RAID1, then each device (except for the very
> > end) contains exactly the same data as the whole array.
> > If you install grub to the array, then it will be installed onto all of the
> > (active) devices in the array.  And that is certainly the easiest way to
> > write to all device.
> >
> > It won't write to 'spares', so if you want to be able to boot from spares as
> > well .... but I'm not sure that makes sense anyway.
> 
> Yes, for a raid1 with no spares, installing to the array is equivalent 
> to installing to each individual disk, but it helps avoid confusion to 
> ignore this fact and remain thinking in terms of the physical disks, at 
> least as they appear to the bios.
> 
> > Completely agree.  As I said, there are only some cases where you can boot
> > from an array which uses whole-disks.
> > One case if in the bios understands the array, such as Intel bios's with IMSM
> > metadata, or possibly some bioses with DDF metadata.
> > Another case is RAID1 which starts at the beginning of the device, where the
> > bios doesn't need to know about the RAID.
> 
> So how do we tell the difference?  Right now grub uses the rule of 
> dmraid = bios aware, so install to the raid device, and mdadm = software 
> raid, so install to the component devices individually.  You have noted 
> that in same cases both methods will produce the same results, but grub 
> needs to be certain that whichever method it chooses will work, whether 
> or not either one will.  To do this, it needs to install to the raid 
> device if and only if it is a bios recognized fakeraid.

Could you install to both??  Maybe not...

What exactly needs to be installed?  My understanding is that you need to
install a boot block which cal load a larger chunk from some fixed location,
and the larger chunk understands raid and filesystems and everything and can
figure out what to do.
Where do you install that larger chunk??

I'd be happy to get mdadm to provide useful information for grub if I knew
what information was needed, and knew that grub would use it.
It would probably be an extension to "mdadm --detail --export".
It could:
 - list devices on which it was safe to install a boot block, which might be
   the array, or might be components, or might be nothing
 - list and offset/size on each device where there is some spare space for
   the second stage (or is that 'stage1.5'??) could be stored safely.
 - report an offset for mapping from 'the array' to 'individual device' when
   that makes sense (i.e. raid1 and maybe linear).
 - anything else required.

I'd really rather if grub didn't assume that 'md' meant something specific,
but rather asked mdadm what it should do with a given 'md' device.

So if you (or someone) can tell me what grub needs and how it would use it,
and undertakes to teach grub to ask mdadm and make use of the answers, and to
continue the dialogue if it turns out that our first design approach doesn't
quite work right,  then I will very happily get the required functionality
into mdadm very promptly.

Thanks,
NeilBrown

Re: Software RAID and Fakeraid

[Phillip Susi]

On 12/8/2010 5:43 PM, Neil Brown wrote:
> Could you install to both??  Maybe not...

For a fakeraid raid1 you could do both, or either.  For a conventional
md raid1, or a fakeraid raid0 ( or raid5/6 ) you can not.

> What exactly needs to be installed?  My understanding is that you need to
> install a boot block which cal load a larger chunk from some fixed location,
> and the larger chunk understands raid and filesystems and everything and can
> figure out what to do.
> Where do you install that larger chunk??

The larger chunk is called the core image in grub terminology.  It is
generated during installation and written to /boot/grub/core.img and
contains the main kernel plus all disk, filesystem, and abstraction
modules required to access /boot/grub and load more modules and the
config file from there.  The MBR can be written with blocklists directly
pointing to that file, but this is inherently unreliable ( and not even
possible on some filesystems that do things like tail packing ) so it is
preferred to copy the core image to the embed area immediately following
the MBR, or to the BIOS boot partition if you are using GPT.

> I'd be happy to get mdadm to provide useful information for grub if I knew
> what information was needed, and knew that grub would use it.
> It would probably be an extension to "mdadm --detail --export".

What grub really needs to know is whether or not the array is accessible
via the bios int 13.  If it is not, then we need to figure out which
abstraction modules are needed to access the array ( md and possibly lvm
), build them into the core image, and install the mbr and core image to
the underlying physical disks.  If it is an int13 device, then the md
module is not required.

If mdadm --detail --export lists the metadata type being used, we can
probably use that to infer whether it is conventional md metadata, or
fakeraid metadata, and thus assume int13 access, but it might be better
to get a more explicit statement from mdadm about whether it is fakeraid
or some kind of pure software raid, whether it be conventional md or
otherwise.

Things get a little tricky with md using the whole disks instead of
partitions in them.  In that case, it is simply not possible to install
grub if using 1.1.  For 0.9 and 1.0, it is possible iff it is a
partitioned array and therefore, the primary component disk appears to
have a valid partition table and embed area and the stripe size is >=
32k ( except for raid1 ).  Even then it is only possible to install grub
to the primary disk, which obviously means you can not boot if that disk
fails, so this isn't a desirable setup.

I suppose it could be possible to install with 1.2, if mdadm can point
grub to a location it can install the core to, instead of the embed area
( since the md metadata is there ).  This would probably be the leftover
space at the end of the disk, that is not part of a stripe ( but still
needs to be >= 32k ).  This would even allow you to install to all
components and have redundant boot from each of them.

Re: Software RAID and Fakeraid

[Phillip Susi]

On 12/8/2010 5:43 PM, Neil Brown wrote:
> Could you install to both??  Maybe not...

For a fakeraid raid1 you could do both, or either.  For a conventional
md raid1, or a fakeraid raid0 ( or raid5/6 ) you can not.

> What exactly needs to be installed?  My understanding is that you need to
> install a boot block which cal load a larger chunk from some fixed location,
> and the larger chunk understands raid and filesystems and everything and can
> figure out what to do.
> Where do you install that larger chunk??

The larger chunk is called the core image in grub terminology.  It is
generated during installation and written to /boot/grub/core.img and
contains the main kernel plus all disk, filesystem, and abstraction
modules required to access /boot/grub and load more modules and the
config file from there.  The MBR can be written with blocklists directly
pointing to that file, but this is inherently unreliable ( and not even
possible on some filesystems that do things like tail packing ) so it is
preferred to copy the core image to the embed area immediately following
the MBR, or to the BIOS boot partition if you are using GPT.

> I'd be happy to get mdadm to provide useful information for grub if I knew
> what information was needed, and knew that grub would use it.
> It would probably be an extension to "mdadm --detail --export".

What grub really needs to know is whether or not the array is accessible
via the bios int 13.  If it is not, then we need to figure out which
abstraction modules are needed to access the array ( md and possibly lvm
), build them into the core image, and install the mbr and core image to
the underlying physical disks.  If it is an int13 device, then the md
module is not required.

If mdadm --detail --export lists the metadata type being used, we can
probably use that to infer whether it is conventional md metadata, or
fakeraid metadata, and thus assume int13 access, but it might be better
to get a more explicit statement from mdadm about whether it is fakeraid
or some kind of pure software raid, whether it be conventional md or
otherwise.

Things get a little tricky with md using the whole disks instead of
partitions in them.  In that case, it is simply not possible to install
grub if using 1.1.  For 0.9 and 1.0, it is possible iff it is a
partitioned array and therefore, the primary component disk appears to
have a valid partition table and embed area and the stripe size is >=
32k ( except for raid1 ).  Even then it is only possible to install grub
to the primary disk, which obviously means you can not boot if that disk
fails, so this isn't a desirable setup.

I suppose it could be possible to install with 1.2, if mdadm can point
grub to a location it can install the core to, instead of the embed area
( since the md metadata is there ).  This would probably be the leftover
space at the end of the disk, that is not part of a stripe ( but still
needs to be >= 32k ).  This would even allow you to install to all
components and have redundant boot from each of them.

Re: Software RAID and Fakeraid

[Phillip Susi]

I know you went on vacation over the new year, but it has been about two
months now so I figured I would try to revive this conversation.

Having collected my thoughts, this is what I propose:

I believe that right now, grub recognizes an md device that is built on
partitions, and installs to the underlying disks.  When the device is
built on the whole disks, it needs to divide into two categories:

1)  Native md metadata.  Grub should ask mdadm for a suitable location
to embed the core image on each component disk.  This can be satisfied
with a 32kb area at the end ( or maybe the beginning ) of the disk that
is not used by the array ( or the metadata ).  This will work for all
native formats except for 1.1, for the obvious reason that it is using
the boot block.

2)  Other metadata formats.  Either assume or explicitly be told by
mdadm that these formats are fakeraid and thus understood by the bios
int 13.  Grub installs to the array as a whole, just like it does today
when they are handled by dmraid.  If it is known that it is not bios
supported, then error out.  In the future this could be supported by
creating additional grub modules that understand those metadata formats.

One question that needs answered for method 1 is what should the
partition table look like?  Should grub, or maybe mdadm, create a
protective mbr when using the whole disk, similar to the one used with GPT?

Re: Software RAID and Fakeraid

[Lennart Sorensen]

On Mon, Jan 31, 2011 at 11:44:07AM -0500, Phillip Susi wrote:
> I know you went on vacation over the new year, but it has been about two
> months now so I figured I would try to revive this conversation.
> 
> Having collected my thoughts, this is what I propose:
> 
> I believe that right now, grub recognizes an md device that is built on
> partitions, and installs to the underlying disks.  When the device is
> built on the whole disks, it needs to divide into two categories:
> 
> 1)  Native md metadata.  Grub should ask mdadm for a suitable location
> to embed the core image on each component disk.  This can be satisfied
> with a 32kb area at the end ( or maybe the beginning ) of the disk that
> is not used by the array ( or the metadata ).  This will work for all
> native formats except for 1.1, for the obvious reason that it is using
> the boot block.
> 
> 2)  Other metadata formats.  Either assume or explicitly be told by
> mdadm that these formats are fakeraid and thus understood by the bios
> int 13.  Grub installs to the array as a whole, just like it does today
> when they are handled by dmraid.  If it is known that it is not bios
> supported, then error out.  In the future this could be supported by
> creating additional grub modules that understand those metadata formats.
> 
> One question that needs answered for method 1 is what should the
> partition table look like?  Should grub, or maybe mdadm, create a
> protective mbr when using the whole disk, similar to the one used with GPT?

I think in the case of BIOS raid, the bios will boot from the raid
device, so installing to the boot area of the raid device makes sense
(/dev/mapper/whatever or similar).

In the case of software md raid on partitions, the BIOS boots from the
boot area of the disks directly, so installing to all member disks makes
sense (/dev/sda, /dev/sdb, etc).

If you do software md raid on whole disks, I am not even sure if the
BIOS could boot from that, since there won't be a partition table, no
partition marked bootable (which some BIOSs requrie), sector 0 may not
even contain boot code.  Not sure there is anyway whole device software
raid makes sense for a bootable drive at all.  It makes sense for data
drives perhaps.  I just don't see this as a candidate for booting at all.

Now that's all assuming x86 BIOS based booting.  For other systems
supported by grub, things could be different perhaps.

-- 
Len Sorensen

Re: Software RAID and Fakeraid

[Phillip Susi]

On 1/31/2011 12:03 PM, Lennart Sorensen wrote:
> If you do software md raid on whole disks, I am not even sure if the
> BIOS could boot from that, since there won't be a partition table, no
> partition marked bootable (which some BIOSs requrie), sector 0 may not
> even contain boot code.  Not sure there is anyway whole device software
> raid makes sense for a bootable drive at all.  It makes sense for data
> drives perhaps.  I just don't see this as a candidate for booting at all.

That is why I asked what the partition table should look like.  When
grub is installed there will be one ( and boot code ), the question is
whether it should be empty, or have a protective partition that claims
the entire disk, like EFI does.

Normally with format 0.9, the raid array starts on sector 0, so the same
MBR appears both at the start of the first physical disk, and at the
start of the array.  This is not possible with 1.1 or 1.2, so where do
they start the array?  I must be at some point further away from sector
0, leaving room for a protective MBR on the component disks that would
be entirely separate from any MBR inside the array.  This leaves room
for grub to be installed on each component disk, rather than only the
first.  It also means that it is incompatible with formats 0.9, 1.1, and
possibly 1.0, depending on whether it includes the first sector in the
array like 0.9 does.

Re: Software RAID and Fakeraid

[Phillip Susi]

On 1/31/2011 12:03 PM, Lennart Sorensen wrote:
> If you do software md raid on whole disks, I am not even sure if the
> BIOS could boot from that, since there won't be a partition table, no
> partition marked bootable (which some BIOSs requrie), sector 0 may not
> even contain boot code.  Not sure there is anyway whole device software
> raid makes sense for a bootable drive at all.  It makes sense for data
> drives perhaps.  I just don't see this as a candidate for booting at all.

That is why I asked what the partition table should look like.  When
grub is installed there will be one ( and boot code ), the question is
whether it should be empty, or have a protective partition that claims
the entire disk, like EFI does.

Normally with format 0.9, the raid array starts on sector 0, so the same
MBR appears both at the start of the first physical disk, and at the
start of the array.  This is not possible with 1.1 or 1.2, so where do
they start the array?  I must be at some point further away from sector
0, leaving room for a protective MBR on the component disks that would
be entirely separate from any MBR inside the array.  This leaves room
for grub to be installed on each component disk, rather than only the
first.  It also means that it is incompatible with formats 0.9, 1.1, and
possibly 1.0, depending on whether it includes the first sector in the
array like 0.9 does.

Re: Software RAID and Fakeraid

[Lennart Sorensen]

On Mon, Jan 31, 2011 at 02:21:51PM -0500, Phillip Susi wrote:
> That is why I asked what the partition table should look like.  When
> grub is installed there will be one ( and boot code ), the question is
> whether it should be empty, or have a protective partition that claims
> the entire disk, like EFI does.
> 
> Normally with format 0.9, the raid array starts on sector 0, so the same
> MBR appears both at the start of the first physical disk, and at the
> start of the array.  This is not possible with 1.1 or 1.2, so where do
> they start the array?  I must be at some point further away from sector
> 0, leaving room for a protective MBR on the component disks that would
> be entirely separate from any MBR inside the array.  This leaves room
> for grub to be installed on each component disk, rather than only the
> first.  It also means that it is incompatible with formats 0.9, 1.1, and
> possibly 1.0, depending on whether it includes the first sector in the
> array like 0.9 does.

Well one of them stores the raid info at the end of the device, so that
one offers no help for the bootloader at all.  I think the other stores
it at 4KB in to the device if my memory serves me right.  Not sure if
the first 4KB are left empty for other uses or not.

It seems to me the simplest thing to do is to simply state in
documentation that grub can not be installed on md raid if there is no
partition table in use.  This doesn't mean people can't do it for purely
data oriented drives, but for the drives with the boot laoder there
are limitations.

-- 
Len Sorensen

Re: Software RAID and Fakeraid

[Phillip Susi]

On 01/31/2011 05:12 PM, Lennart Sorensen wrote:
> Well one of them stores the raid info at the end of the device, so that
> one offers no help for the bootloader at all.  I think the other stores

What do you mean help?

> it at 4KB in to the device if my memory serves me right.  Not sure if
> the first 4KB are left empty for other uses or not.

It is left empty afaik, which is the whole point of shifting it over to 4kb.

> It seems to me the simplest thing to do is to simply state in
> documentation that grub can not be installed on md raid if there is no
> partition table in use.  This doesn't mean people can't do it for purely
> data oriented drives, but for the drives with the boot laoder there
> are limitations.

Obviously doing nothing is the simplest thing one can do.  Instead I 
would like to do something and end up with a better, more functional 
system.  Hence my suggestion on WHAT to do and the existence of this 
entire thread.

Re: Software RAID and Fakeraid

[Phillip Susi]

On 01/31/2011 05:12 PM, Lennart Sorensen wrote:
> Well one of them stores the raid info at the end of the device, so that
> one offers no help for the bootloader at all.  I think the other stores

What do you mean help?

> it at 4KB in to the device if my memory serves me right.  Not sure if
> the first 4KB are left empty for other uses or not.

It is left empty afaik, which is the whole point of shifting it over to 4kb.

> It seems to me the simplest thing to do is to simply state in
> documentation that grub can not be installed on md raid if there is no
> partition table in use.  This doesn't mean people can't do it for purely
> data oriented drives, but for the drives with the boot laoder there
> are limitations.

Obviously doing nothing is the simplest thing one can do.  Instead I 
would like to do something and end up with a better, more functional 
system.  Hence my suggestion on WHAT to do and the existence of this 
entire thread.

Re: Software RAID and Fakeraid

[Michal Suchanek]

On 1 February 2011 02:31, Phillip Susi <psusi@cfl.rr.com> wrote:
> On 01/31/2011 05:12 PM, Lennart Sorensen wrote:
>>
>> Well one of them stores the raid info at the end of the device, so that
>> one offers no help for the bootloader at all.  I think the other stores
>
> What do you mean help?
>


I guess that the point of this thread is to determine under what
conditions and where Grub can be installed on a raid array.

Since Grub is somewhat bloated compared to other bootloaders it has
its specific problems. However, the point is that even bootloaders
that do fit into a single sector do need to know where that sector can
be placed, if anywhere, and it is currently not possible to know with
mdraid unless the bootloader reimplements the full mdadm logic itself.

To avoid that mdadm should at the very least provide the information
whether the raid is bios-bootable or not. Ideally it should also
provide a list of sectors unused by the raid format in question.
Failing that at least a document clearly summarizing this for various
raid format versions and mdadm reporting the version would be helpful.

Thanks

Michal
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Software RAID and Fakeraid

[Michal Suchanek]

On 1 February 2011 02:31, Phillip Susi <psusi@cfl.rr.com> wrote:
> On 01/31/2011 05:12 PM, Lennart Sorensen wrote:
>>
>> Well one of them stores the raid info at the end of the device, so that
>> one offers no help for the bootloader at all.  I think the other stores
>
> What do you mean help?
>


I guess that the point of this thread is to determine under what
conditions and where Grub can be installed on a raid array.

Since Grub is somewhat bloated compared to other bootloaders it has
its specific problems. However, the point is that even bootloaders
that do fit into a single sector do need to know where that sector can
be placed, if anywhere, and it is currently not possible to know with
mdraid unless the bootloader reimplements the full mdadm logic itself.

To avoid that mdadm should at the very least provide the information
whether the raid is bios-bootable or not. Ideally it should also
provide a list of sectors unused by the raid format in question.
Failing that at least a document clearly summarizing this for various
raid format versions and mdadm reporting the version would be helpful.

Thanks

Michal

Re: Software RAID and Fakeraid

[Lennart Sorensen]

On Mon, Jan 31, 2011 at 08:31:38PM -0500, Phillip Susi wrote:
> What do you mean help?

If the raid stores the raid info at the end, then the data starts at
sector 0.  So no space for a bootloader at all.

> It is left empty afaik, which is the whole point of shifting it over to 4kb.

Certainly makes sense.  Now is 4K enough for a boot loader?  Not sure.

> Obviously doing nothing is the simplest thing one can do.  Instead I  
> would like to do something and end up with a better, more functional  
> system.  Hence my suggestion on WHAT to do and the existence of this  
> entire thread.

I personally consider soft raid on raw devices so convluted that I
have never done it.  I would rather have something I know works with my
bootloader and other tools, than gain that extra 1MB (at most) that not
having partitions gives.  Also given many PCs won't boot from a drive
without a partition table, it isn't even an option then.

Really I think it comes down to only being able to handle the case
where there is a 4K gap, if you can fit grub in there in a format that
a system would be able to boot reading just sector 0 of the device.
The raid info at the start and raid info at the end both prevent you
from having any boot code in sector 0 so can't be supported at all.

-- 
Len Sorensen

Re: Software RAID and Fakeraid

[Lennart Sorensen]

On Mon, Jan 31, 2011 at 08:31:38PM -0500, Phillip Susi wrote:
> What do you mean help?

If the raid stores the raid info at the end, then the data starts at
sector 0.  So no space for a bootloader at all.

> It is left empty afaik, which is the whole point of shifting it over to 4kb.

Certainly makes sense.  Now is 4K enough for a boot loader?  Not sure.

> Obviously doing nothing is the simplest thing one can do.  Instead I  
> would like to do something and end up with a better, more functional  
> system.  Hence my suggestion on WHAT to do and the existence of this  
> entire thread.

I personally consider soft raid on raw devices so convluted that I
have never done it.  I would rather have something I know works with my
bootloader and other tools, than gain that extra 1MB (at most) that not
having partitions gives.  Also given many PCs won't boot from a drive
without a partition table, it isn't even an option then.

Really I think it comes down to only being able to handle the case
where there is a 4K gap, if you can fit grub in there in a format that
a system would be able to boot reading just sector 0 of the device.
The raid info at the start and raid info at the end both prevent you
from having any boot code in sector 0 so can't be supported at all.

-- 
Len Sorensen

Re: Software RAID and Fakeraid

[Phillip Susi]

On 02/01/2011 11:26 AM, Lennart Sorensen wrote:
> If the raid stores the raid info at the end, then the data starts at
> sector 0.  So no space for a bootloader at all.

I know that is how it works with 0.9, but are you sure it is for 1.0? 
If so, then for anything but raid-1 we will just have to try to install 
only to the first device if it has an MBR.

> Certainly makes sense.  Now is 4K enough for a boot loader?  Not sure.

It is enough for the MBR.  The core image will need to go elsewhere, 
hence the proposal to ask mdadm for a suitable location.

> I personally consider soft raid on raw devices so convluted that I
> have never done it.  I would rather have something I know works with my
> bootloader and other tools, than gain that extra 1MB (at most) that not
> having partitions gives.  Also given many PCs won't boot from a drive
> without a partition table, it isn't even an option then.

That is why I like the idea of 1.2 since you could still have a bootable 
MBR when using the whole disk.  Though now that you mention it, I can't 
think of a good reason to use the whole disk instead of a partition either.

Re: Software RAID and Fakeraid

[Phillip Susi]

On 02/01/2011 11:26 AM, Lennart Sorensen wrote:
> If the raid stores the raid info at the end, then the data starts at
> sector 0.  So no space for a bootloader at all.

I know that is how it works with 0.9, but are you sure it is for 1.0? 
If so, then for anything but raid-1 we will just have to try to install 
only to the first device if it has an MBR.

> Certainly makes sense.  Now is 4K enough for a boot loader?  Not sure.

It is enough for the MBR.  The core image will need to go elsewhere, 
hence the proposal to ask mdadm for a suitable location.

> I personally consider soft raid on raw devices so convluted that I
> have never done it.  I would rather have something I know works with my
> bootloader and other tools, than gain that extra 1MB (at most) that not
> having partitions gives.  Also given many PCs won't boot from a drive
> without a partition table, it isn't even an option then.

That is why I like the idea of 1.2 since you could still have a bootable 
MBR when using the whole disk.  Though now that you mention it, I can't 
think of a good reason to use the whole disk instead of a partition either.

Re: Software RAID and Fakeraid

[NeilBrown]

On Tue, 01 Feb 2011 19:08:09 -0500 Phillip Susi <psusi@cfl.rr.com> wrote:

> On 02/01/2011 11:26 AM, Lennart Sorensen wrote:
> > If the raid stores the raid info at the end, then the data starts at
> > sector 0.  So no space for a bootloader at all.
> 
> I know that is how it works with 0.9, but are you sure it is for 1.0? 
> If so, then for anything but raid-1 we will just have to try to install 
> only to the first device if it has an MBR.
> 
> > Certainly makes sense.  Now is 4K enough for a boot loader?  Not sure.
> 
> It is enough for the MBR.  The core image will need to go elsewhere, 
> hence the proposal to ask mdadm for a suitable location.
> 
> > I personally consider soft raid on raw devices so convluted that I
> > have never done it.  I would rather have something I know works with my
> > bootloader and other tools, than gain that extra 1MB (at most) that not
> > having partitions gives.  Also given many PCs won't boot from a drive
> > without a partition table, it isn't even an option then.
> 
> That is why I like the idea of 1.2 since you could still have a bootable 
> MBR when using the whole disk.  Though now that you mention it, I can't 
> think of a good reason to use the whole disk instead of a partition either.


It seems to me that a case analysis would be useful here.
Assuming that the area of interest is loading the grub core image when
/boot (or '/') is on an md device,

 0 If the md device is comprised entirely of partitions, then it is
   not involved in loading the core image at all
 
 otherwise:

 1 The md device could be addressable directly by the bios.  This applies
   to a RAID1 which starts at the start of the devices, or any RAID level
   which is explicitly understood by BIOS or an option ROM (such as Intel
   IMSM)

 2 The md device could leave the first block, and some other section of
   each device unused.  These can be used to store the boot block and
   the core image.
   This applies to 1.2 metadata stored on whole devices.  It could apply to
   1.0 (as the start address is configurable) but doesn't in practise.  The
   main reason to choose 1.0 is to have the array aligned with the start of
   the device.

 3 The md device does not permit booting.  This applies to 1.1 metadata
   and various other combinations other than those identified above.


There is a difficulty in case 2 as it is not clear who's responsibility it is
to write a partition table at the start of each device.
Presumably GRUB doesn't like to write partition tables unless one already
exists.
Currently mdadm doesn't write a partition table either.  Possibly it could,
but I would rather avoid that if possible.
Maybe once case 2 has been clearly identified, GRUB could consider that
sufficient permission to write a boot block and partition table even if no
partition table existed??

I imagine that the best way to distinguish between the cases would be to have
   mdadm --detail --export /dev/mdXXX

report something appropriate.  Maybe a setting for "MD_BOOTABLE"
e.g.

 MD_BOOTABLE=partitions    # case 0 - the array is comprised entirely of
                           # partitions
 MD_BOOTABLE=BIOS          # mdadm believes that the bios can an will read the
                           # the array directly - i.e. case 1
 MD_BOOTABLE=reserved      # Space at the start of each device is reserved
                           # for storing boot information.  In particular the
                           # first block (4K) is reserved plus some more.
 MD_BOOTABLE=no            # mdadm does not believe it is possible to boot
                           # from this array

In the 'reserved' case, mdadm would also report where the space is. e.g.

 MD_BOOT_SPACE="/dev/sda 8192 32768"

means that from byte offset 8192 there is 32768 bytes of available space.
I would need to make sure that mdadm kept that space available, so I would
need to know how much to reserve.  Maybe 32K.  Maybe 1M is safe?

However there is another complication.
I understand that the boot block sometimes lives at the start of the
partition instead of (or as well as) the start of the device.
I'm fairly syslinux does this - I don't know about GRUB.
So I really want to still report BIOS or 'reserved' or 'no' even when
partitions are in use.

So maybe I should scrap case 0 (MD_BOOTABLE=partitions), assume that the
boot-loader configurer can detect and understand partitions itself, and just
report the other 3 cases ignoring the details about partitions.

Would that be helpful?  Would it get used?  How could it be better?

Thanks,
NeilBrown

Re: Software RAID and Fakeraid

[NeilBrown]

On Tue, 01 Feb 2011 19:08:09 -0500 Phillip Susi <psusi@cfl.rr.com> wrote:

> On 02/01/2011 11:26 AM, Lennart Sorensen wrote:
> > If the raid stores the raid info at the end, then the data starts at
> > sector 0.  So no space for a bootloader at all.
> 
> I know that is how it works with 0.9, but are you sure it is for 1.0? 
> If so, then for anything but raid-1 we will just have to try to install 
> only to the first device if it has an MBR.
> 
> > Certainly makes sense.  Now is 4K enough for a boot loader?  Not sure.
> 
> It is enough for the MBR.  The core image will need to go elsewhere, 
> hence the proposal to ask mdadm for a suitable location.
> 
> > I personally consider soft raid on raw devices so convluted that I
> > have never done it.  I would rather have something I know works with my
> > bootloader and other tools, than gain that extra 1MB (at most) that not
> > having partitions gives.  Also given many PCs won't boot from a drive
> > without a partition table, it isn't even an option then.
> 
> That is why I like the idea of 1.2 since you could still have a bootable 
> MBR when using the whole disk.  Though now that you mention it, I can't 
> think of a good reason to use the whole disk instead of a partition either.


It seems to me that a case analysis would be useful here.
Assuming that the area of interest is loading the grub core image when
/boot (or '/') is on an md device,

 0 If the md device is comprised entirely of partitions, then it is
   not involved in loading the core image at all
 
 otherwise:

 1 The md device could be addressable directly by the bios.  This applies
   to a RAID1 which starts at the start of the devices, or any RAID level
   which is explicitly understood by BIOS or an option ROM (such as Intel
   IMSM)

 2 The md device could leave the first block, and some other section of
   each device unused.  These can be used to store the boot block and
   the core image.
   This applies to 1.2 metadata stored on whole devices.  It could apply to
   1.0 (as the start address is configurable) but doesn't in practise.  The
   main reason to choose 1.0 is to have the array aligned with the start of
   the device.

 3 The md device does not permit booting.  This applies to 1.1 metadata
   and various other combinations other than those identified above.


There is a difficulty in case 2 as it is not clear who's responsibility it is
to write a partition table at the start of each device.
Presumably GRUB doesn't like to write partition tables unless one already
exists.
Currently mdadm doesn't write a partition table either.  Possibly it could,
but I would rather avoid that if possible.
Maybe once case 2 has been clearly identified, GRUB could consider that
sufficient permission to write a boot block and partition table even if no
partition table existed??

I imagine that the best way to distinguish between the cases would be to have
   mdadm --detail --export /dev/mdXXX

report something appropriate.  Maybe a setting for "MD_BOOTABLE"
e.g.

 MD_BOOTABLE=partitions    # case 0 - the array is comprised entirely of
                           # partitions
 MD_BOOTABLE=BIOS          # mdadm believes that the bios can an will read the
                           # the array directly - i.e. case 1
 MD_BOOTABLE=reserved      # Space at the start of each device is reserved
                           # for storing boot information.  In particular the
                           # first block (4K) is reserved plus some more.
 MD_BOOTABLE=no            # mdadm does not believe it is possible to boot
                           # from this array

In the 'reserved' case, mdadm would also report where the space is. e.g.

 MD_BOOT_SPACE="/dev/sda 8192 32768"

means that from byte offset 8192 there is 32768 bytes of available space.
I would need to make sure that mdadm kept that space available, so I would
need to know how much to reserve.  Maybe 32K.  Maybe 1M is safe?

However there is another complication.
I understand that the boot block sometimes lives at the start of the
partition instead of (or as well as) the start of the device.
I'm fairly syslinux does this - I don't know about GRUB.
So I really want to still report BIOS or 'reserved' or 'no' even when
partitions are in use.

So maybe I should scrap case 0 (MD_BOOTABLE=partitions), assume that the
boot-loader configurer can detect and understand partitions itself, and just
report the other 3 cases ignoring the details about partitions.

Would that be helpful?  Would it get used?  How could it be better?

Thanks,
NeilBrown

Re: Software RAID and Fakeraid

[Phillip Susi]

On 2/1/2011 10:22 PM, NeilBrown wrote:
> There is a difficulty in case 2 as it is not clear who's responsibility it is
> to write a partition table at the start of each device.
> Presumably GRUB doesn't like to write partition tables unless one already
> exists.

Yes, it preserves the existing partition table and just modifies the
boot loader code in the MBR.

> Currently mdadm doesn't write a partition table either.  Possibly it could,
> but I would rather avoid that if possible.
> Maybe once case 2 has been clearly identified, GRUB could consider that
> sufficient permission to write a boot block and partition table even if no
> partition table existed??

Possibly, but that is the kind of thing I think should require an
explicit request.  Whether it is done by mdadm or not, I think that
someone should write a protective mbr.  Since mdadm is what is
effectively formatting the disk, it makes the most sense to me to do it
there, rather than in grub, which is just trying to install a boot
loader to an existing disk.  I suppose the OS installer could add the
MBR before asking mdadm to add its superblocks too.  What partition type
would be appropriate to use?

> In the 'reserved' case, mdadm would also report where the space is. e.g.
> 
>  MD_BOOT_SPACE="/dev/sda 8192 32768"
> 
> means that from byte offset 8192 there is 32768 bytes of available space.
> I would need to make sure that mdadm kept that space available, so I would
> need to know how much to reserve.  Maybe 32K.  Maybe 1M is safe?

Sounds good.  Grub is used to operating with 32k or less since that is
the historical amount of free space following the MBR.

> However there is another complication.
> I understand that the boot block sometimes lives at the start of the
> partition instead of (or as well as) the start of the device.
> I'm fairly syslinux does this - I don't know about GRUB.
> So I really want to still report BIOS or 'reserved' or 'no' even when
> partitions are in use.

Grub can be installed to the partition boot block, but it is strongly
discouraged since there is no gap to embed the core into, so the boot
block must use block lists to locate it.  This comes with all kinds of
headaches, including not being possible at all on some filesystems, and
frequently breaking on others.  Either way you still have to have boot
code in the MBR to go load the partition boot block, so I don't think
that changes anything with respect to this discussion.

> So maybe I should scrap case 0 (MD_BOOTABLE=partitions), assume that the
> boot-loader configurer can detect and understand partitions itself, and just
> report the other 3 cases ignoring the details about partitions.
> 
> Would that be helpful?  Would it get used?  How could it be better?

Yes, it sounds quite helpful.

Re: Software RAID and Fakeraid

[Phillip Susi]

On 2/1/2011 10:22 PM, NeilBrown wrote:
> There is a difficulty in case 2 as it is not clear who's responsibility it is
> to write a partition table at the start of each device.
> Presumably GRUB doesn't like to write partition tables unless one already
> exists.

Yes, it preserves the existing partition table and just modifies the
boot loader code in the MBR.

> Currently mdadm doesn't write a partition table either.  Possibly it could,
> but I would rather avoid that if possible.
> Maybe once case 2 has been clearly identified, GRUB could consider that
> sufficient permission to write a boot block and partition table even if no
> partition table existed??

Possibly, but that is the kind of thing I think should require an
explicit request.  Whether it is done by mdadm or not, I think that
someone should write a protective mbr.  Since mdadm is what is
effectively formatting the disk, it makes the most sense to me to do it
there, rather than in grub, which is just trying to install a boot
loader to an existing disk.  I suppose the OS installer could add the
MBR before asking mdadm to add its superblocks too.  What partition type
would be appropriate to use?

> In the 'reserved' case, mdadm would also report where the space is. e.g.
> 
>  MD_BOOT_SPACE="/dev/sda 8192 32768"
> 
> means that from byte offset 8192 there is 32768 bytes of available space.
> I would need to make sure that mdadm kept that space available, so I would
> need to know how much to reserve.  Maybe 32K.  Maybe 1M is safe?

Sounds good.  Grub is used to operating with 32k or less since that is
the historical amount of free space following the MBR.

> However there is another complication.
> I understand that the boot block sometimes lives at the start of the
> partition instead of (or as well as) the start of the device.
> I'm fairly syslinux does this - I don't know about GRUB.
> So I really want to still report BIOS or 'reserved' or 'no' even when
> partitions are in use.

Grub can be installed to the partition boot block, but it is strongly
discouraged since there is no gap to embed the core into, so the boot
block must use block lists to locate it.  This comes with all kinds of
headaches, including not being possible at all on some filesystems, and
frequently breaking on others.  Either way you still have to have boot
code in the MBR to go load the partition boot block, so I don't think
that changes anything with respect to this discussion.

> So maybe I should scrap case 0 (MD_BOOTABLE=partitions), assume that the
> boot-loader configurer can detect and understand partitions itself, and just
> report the other 3 cases ignoring the details about partitions.
> 
> Would that be helpful?  Would it get used?  How could it be better?

Yes, it sounds quite helpful.

Re: Software RAID and Fakeraid

[hansbkk]

On Tue, Feb 1, 2011 at 11:26 PM, Lennart Sorensen
<lsorense@csclub.uwaterloo.ca> wrote:
> I personally consider soft raid on raw devices so convluted that I
> have never done it.  I would rather have something I know works with my
> bootloader and other tools, than gain that extra 1MB (at most) that not
> having partitions gives.  Also given many PCs won't boot from a drive
> without a partition table, it isn't even an option then.

For others googling this later, another very good reason for *never*
RAID'ing raw block devices (ie always creating at least one partition
first) is that if you ever mistakenly boot into some flavors of
Windows (even from some optical discs, perhaps unknowingly left ina
drive), your disks will automatically get "helpfully" initialized, as
windoze thinks it's a brand new empty drive being offered up like a
virgin for sacrifice - **poof** there goes all your data.

Speaking from experience 8-(
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Software RAID and Fakeraid

[hansbkk]

On Tue, Feb 1, 2011 at 11:26 PM, Lennart Sorensen
<lsorense@csclub.uwaterloo.ca> wrote:
> I personally consider soft raid on raw devices so convluted that I
> have never done it.  I would rather have something I know works with my
> bootloader and other tools, than gain that extra 1MB (at most) that not
> having partitions gives.  Also given many PCs won't boot from a drive
> without a partition table, it isn't even an option then.

For others googling this later, another very good reason for *never*
RAID'ing raw block devices (ie always creating at least one partition
first) is that if you ever mistakenly boot into some flavors of
Windows (even from some optical discs, perhaps unknowingly left ina
drive), your disks will automatically get "helpfully" initialized, as
windoze thinks it's a brand new empty drive being offered up like a
virgin for sacrifice - **poof** there goes all your data.

Speaking from experience 8-(

RE: Software RAID and Fakeraid

[Leslie Rhorer]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of hansbkk@gmail.com
> Sent: Wednesday, February 02, 2011 10:09 AM
> To: linux-raid@vger.kernel.org
> Cc: grub-devel@gnu.org
> Subject: Re: Software RAID and Fakeraid
> 
> On Tue, Feb 1, 2011 at 11:26 PM, Lennart Sorensen
> <lsorense@csclub.uwaterloo.ca> wrote:
> > I personally consider soft raid on raw devices so convluted that I
> > have never done it.

	Convoluted?  It simply removes an unnecessary layer of management.
If anything, it is less convoluted, and one needn't worry about partition
types and their associated limitations.  I currently have 24 un-partitioned
drives in my two RAID systems.

> > I would rather have something I know works with my
> > bootloader and other tools, than gain that extra 1MB (at most) that not

	Modern bootloaders don't require partitions, either, IIRC, although
in fact I do partition my boot drives.

> > having partitions gives.  Also given many PCs won't boot from a drive
> > without a partition table, it isn't even an option then.

	Do you have an example?  I'm not aware of any.  In any case, as I
said, I do recommend partitioning the boot drive, at a minimum into a root
and a swap partition.  I also like to keep the /boot target separate, but
then /boot is tiny.
 
> For others googling this later, another very good reason for *never*
> RAID'ing raw block devices (ie always creating at least one partition
> first) is that if you ever mistakenly boot into some flavors of
> Windows (even from some optical discs, perhaps unknowingly left ina
> drive), your disks will automatically get "helpfully" initialized, as
> windoze thinks it's a brand new empty drive being offered up like a
> virgin for sacrifice - **poof** there goes all your data.
> 
> Speaking from experience 8-(

	This is only true with very old versions of Windows.  What's more,
the data certainly can be recovered.  None of my data drives are
partitioned.  There's really no point.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Software RAID and Fakeraid

[Leslie Rhorer]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of Neil Brown
> Sent: Tuesday, November 30, 2010 4:25 PM
> To: Phillip Susi
> Cc: The development of GNU GRUB; John Sheu; linux-raid@vger.kernel.org
> Subject: Re: Software RAID and Fakeraid
> 
> On Tue, 30 Nov 2010 14:54:40 -0500 Phillip Susi <psusi@cfl.rr.com> wrote:
> 
> > On 11/25/2010 5:26 AM, John Sheu wrote:
> > > What's the preferred way to differentiate BIOS fakeraid from regular
> > > software mdraid?
> >
> > The only way I know of is detecting that it is a dmraid device as
> > opposed to md, which is why grub does it that way.  This worked well in
> > the past when each tool exclusively handled one type of raid.
> >
> > > I ask this as I'm booting with GRUB2 off a system that has one of
> those
> > > Intel fakeraid chipsets.  As of a few months ago, the mdadm package
> has
> > > supported these fakeraid setups, so the RAID array comes up as a
> /dev/md###
> > > device.  This is unfortunate, as GRUB2 assumes that any device of the
> type
> > > /dev/md### must be a pure software RAID device, and in
> > > util/grub-setup.c:939, tries to install itself to the RAID members
> > > individually:
> >
> > For grub to support fakeraids activated by the md driver, it needs some
> > way to find out that it is actually a fake raid, and not a software
> > raid.  Adding linux-raid to Cc list to see if they can suggest a way of
> > doing that.
> 
> My feeling is that grub just needs to be a bit more careful.
> 
> If the members of the md array are partitions, then installing itself in
> the
> boot blocks of the devices holding those partitions always makes sense.
> 
> If the members of the md array are whole devices, then installing grub in
> those devices might make sense depending on specific details of the
> metadata.  The default should be that it doesn't make sense, but specific
> cases do.
> e.g. if the metadata (/sys/block/mdX/md/metadata_version) is 0.90 or 1.0,
> and
> the array is RAID1, then grub should install itself in the *array*, not in
> the devices.
> If the metadata is 1.1, then grub cannot install itself
> If the metadata is 1.2, then grub can install itself at the start
> If the metadata is external:imsm then (I think) grub should install itself
> in
> the array ... though there are some complexities there.
> 
> I often wonder why people who add knowledge of md to grub etc don't at
> least
> let me know what they are doing in case I can see something obviously
> wrong
> with their approach..

	I wonder why GRUB2 only supports 0.90 version superblocks on arrays
from which it can boot.  I wonder even more why they seem to keep it a deep,
dark secret.  I tore my hair out for days trying to figure out why my
upgraded Linux box would not boot.  Under legacy GRUB, it took some
fiddling, but 1.x RAID1 arrays were bootable.

Re: Software RAID and Fakeraid

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 3512 bytes --]

On 12/04/2010 05:34 AM, Leslie Rhorer wrote:
>
>   
>> -----Original Message-----
>> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
>> owner@vger.kernel.org] On Behalf Of Neil Brown
>> Sent: Tuesday, November 30, 2010 4:25 PM
>> To: Phillip Susi
>> Cc: The development of GNU GRUB; John Sheu; linux-raid@vger.kernel.org
>> Subject: Re: Software RAID and Fakeraid
>>
>> On Tue, 30 Nov 2010 14:54:40 -0500 Phillip Susi <psusi@cfl.rr.com> wrote:
>>
>>     
>>> On 11/25/2010 5:26 AM, John Sheu wrote:
>>>       
>>>> What's the preferred way to differentiate BIOS fakeraid from regular
>>>> software mdraid?
>>>>         
>>> The only way I know of is detecting that it is a dmraid device as
>>> opposed to md, which is why grub does it that way.  This worked well in
>>> the past when each tool exclusively handled one type of raid.
>>>
>>>       
>>>> I ask this as I'm booting with GRUB2 off a system that has one of
>>>>         
>> those
>>     
>>>> Intel fakeraid chipsets.  As of a few months ago, the mdadm package
>>>>         
>> has
>>     
>>>> supported these fakeraid setups, so the RAID array comes up as a
>>>>         
>> /dev/md###
>>     
>>>> device.  This is unfortunate, as GRUB2 assumes that any device of the
>>>>         
>> type
>>     
>>>> /dev/md### must be a pure software RAID device, and in
>>>> util/grub-setup.c:939, tries to install itself to the RAID members
>>>> individually:
>>>>         
>>> For grub to support fakeraids activated by the md driver, it needs some
>>> way to find out that it is actually a fake raid, and not a software
>>> raid.  Adding linux-raid to Cc list to see if they can suggest a way of
>>> doing that.
>>>       
>> My feeling is that grub just needs to be a bit more careful.
>>
>> If the members of the md array are partitions, then installing itself in
>> the
>> boot blocks of the devices holding those partitions always makes sense.
>>
>> If the members of the md array are whole devices, then installing grub in
>> those devices might make sense depending on specific details of the
>> metadata.  The default should be that it doesn't make sense, but specific
>> cases do.
>> e.g. if the metadata (/sys/block/mdX/md/metadata_version) is 0.90 or 1.0,
>> and
>> the array is RAID1, then grub should install itself in the *array*, not in
>> the devices.
>> If the metadata is 1.1, then grub cannot install itself
>> If the metadata is 1.2, then grub can install itself at the start
>> If the metadata is external:imsm then (I think) grub should install itself
>> in
>> the array ... though there are some complexities there.
>>
>> I often wonder why people who add knowledge of md to grub etc don't at
>> least
>> let me know what they are doing in case I can see something obviously
>> wrong
>> with their approach..
>>     
> 	I wonder why GRUB2 only supports 0.90 version superblocks on arrays
> from which it can boot.  I wonder even more why they seem to keep it a deep,
> dark secret.  I tore my hair out for days trying to figure out why my
> upgraded Linux box would not boot.  Under legacy GRUB, it took some
> fiddling, but 1.x RAID1 arrays were bootable.
>
>
>   
1.x metadata was added this summer. Please upgrade
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel
>
>   


-- 
Regards
Vladimir 'φ-coder/phcoder' Serbinenko



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 294 bytes --]

Re: Software RAID and Fakeraid

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1988 bytes --]

On 11/25/2010 11:26 AM, John Sheu wrote:
> What's the preferred way to differentiate BIOS fakeraid from regular
> software mdraid?
>
> I ask this as I'm booting with GRUB2 off a system that has one of
> those Intel fakeraid chipsets.  As of a few months ago, the mdadm
> package has supported these fakeraid setups, so the RAID array comes
> up as a /dev/md### device.  This is unfortunate, as GRUB2 assumes that
> any device of the type /dev/md### must be a pure software RAID device,
> and in util/grub-setup.c:939, tries to install itself to the RAID
> members individually:
>
Right now GRUB2 is unable to distinguish between 2 cases. However you
can inform it that you have scamraid by adding following line to device.map:
(hd0)  /dev/md0

> if (0 && dest_dev[0] == 'm' && dest_dev[1] == 'd'
>     && ((dest_dev[2] >= '0' && dest_dev[2] <= '9') || dest_dev[2] == '/'))
>   {
>     char **devicelist;
>     int i;
>
>     devicelist = grub_util_raid_getmembers (dest_dev);
>
>     for (i = 0; devicelist[i]; i++)
>       {
>         setup (arguments.dir ? : DEFAULT_DIRECTORY,
>                arguments.boot_file ? : DEFAULT_BOOT_FILE,
>                arguments.core_file ? : DEFAULT_CORE_FILE,
>                root_dev, grub_util_get_grub_dev (devicelist[i]), 1,
>                arguments.force, arguments.fs_probe);
>       }
>   }
>
> For a fakeraid setup, however, the BIOS presents the entire device as
> "regular" int13 device, so GRUB2 really should be installing it to the
> entire /dev/md### device, not the individual members.
>
> So what's the preferred way to differentiate BIOS fakeraid?  Is there
> some ioctl that would make this easier than having to parse /proc/mdstat?
>
> Thanks,
> -John Sheu
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel
>   


-- 
Regards
Vladimir 'φ-coder/phcoder' Serbinenko



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 294 bytes --]