start dm-multipath before mdadm raid

start dm-multipath before mdadm raid

[P. Remek]

Hello,

I am trying to create mdadm raid on top of dm-multipath devices.
Everything works but after reboot the mdadm array is assembled from
the original (not multipath devices) devices and the multipath devices
are not created at all.

Aparently it is because the mdadm software raid is started before the
dm-multipath creates the multipathed devices. Is there a way how to
make the dm-multipath start before the mdadm is started?

Regards,
Remek

Re: start dm-multipath before mdadm raid

[John Stoffel]

P> I am trying to create mdadm raid on top of dm-multipath devices.
P> Everything works but after reboot the mdadm array is assembled from
P> the original (not multipath devices) devices and the multipath
P> devices are not created at all.

Can you give more details about the OS, kernel version and
configuration of the setup?  

It sounds like you have some local disks and some remote disks that
you want to use in the RAID.  But it's not clear how you're setting
things up.

P> Aparently it is because the mdadm software raid is started before
P> the dm-multipath creates the multipathed devices. Is there a way
P> how to make the dm-multipath start before the mdadm is started?

Hmm... I think devicemapper should be starting pretty early.  Do you
have your initramfs setup properly?  You might need to setup things so
they are working properly, the do:

     update-grub2

to make sure the initramfs gets properly updated.  You might also want
to make sure that LVM is compiled into your kernel, but MDADM is a
module, which should force them to be started in the proper order.

But really, you need to provide details.  Can you setup a small test
MD RAID and show all the details?

John

Re: start dm-multipath before mdadm raid

[Phil Turmel]

On 12/10/2015 10:29 AM, P. Remek wrote:
> Hello,
> 
> I am trying to create mdadm raid on top of dm-multipath devices.
> Everything works but after reboot the mdadm array is assembled from
> the original (not multipath devices) devices and the multipath devices
> are not created at all.
> 
> Aparently it is because the mdadm software raid is started before the
> dm-multipath creates the multipathed devices. Is there a way how to
> make the dm-multipath start before the mdadm is started?

Order of device discovery is not guaranteed, but base devices will
almost always show up before multipath devices.  You have to filter out
the base devices from mdadm consideration:

Add a DEVICES statement to your mdadm.conf that only matches your
multipath device names, not the base names.  Then update your initramfs
so it applies to early boot as well.

Phil

Re: start dm-multipath before mdadm raid

[P. Remek]

>
> Order of device discovery is not guaranteed, but base devices will
> almost always show up before multipath devices.  You have to filter out
> the base devices from mdadm consideration:
>
> Add a DEVICES statement to your mdadm.conf that only matches your
> multipath device names, not the base names.  Then update your initramfs
> so it applies to early boot as well.
>


This is something which I actually already tried. I specified in my
/etc/mdadm/mdadm.conf following:

DEVICE /dev/mapper/hgst-ssd*

In my /etc/multipath.conf I instruct dm-multipath to create
/dev/mapper/hgst-ssd1 /dev/mapper/hgst-ssd2 /dev/mapper/hgst-ssd3


But the only result is that after this config, the md raid is not
started at all. My conclusion was that when md raid was starting, the
multipath devices did not yet exist so it did not start up the array.

Re: start dm-multipath before mdadm raid

[P. Remek]

> But really, you need to provide details.  Can you setup a small test
> MD RAID and show all the details?

Sure, thanks, I will provide all the required information tomorrow
when I am back in office

Remek

Re: start dm-multipath before mdadm raid

[Phil Turmel]

On 12/10/2015 05:44 PM, P. Remek wrote:
>>
>> Order of device discovery is not guaranteed, but base devices will
>> almost always show up before multipath devices.  You have to filter out
>> the base devices from mdadm consideration:
>>
>> Add a DEVICES statement to your mdadm.conf that only matches your
>> multipath device names, not the base names.  Then update your initramfs
>> so it applies to early boot as well.
>>
> 
> 
> This is something which I actually already tried. I specified in my
> /etc/mdadm/mdadm.conf following:
> 
> DEVICE /dev/mapper/hgst-ssd*
> 
> In my /etc/multipath.conf I instruct dm-multipath to create
> /dev/mapper/hgst-ssd1 /dev/mapper/hgst-ssd2 /dev/mapper/hgst-ssd3

You haven't said that you've updated your initramfs with this info.

> But the only result is that after this config, the md raid is not
> started at all. My conclusion was that when md raid was starting, the
> multipath devices did not yet exist so it did not start up the array.

That would be an initramfs bug, most likely.  Modern initramfs like
dracut use incremental assembly to start arrays and subsystems as
devices are started, regardless what order they show up.

Phil

Re: start dm-multipath before mdadm raid

[P. Remek]

Here is my config:

root@os-node1:~# uname  -a
Linux os-node1 3.16.0-48-generic #64~14.04.1-Ubuntu SMP Thu Aug 20
23:03:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux





root@os-node1:~# cat /etc/mdadm/mdadm.conf

DEVICE /dev/mapper/hgst-ssd*
CREATE owner=root group=disk mode=0660 auto=yes
HOMEHOST <system>
MAILADDR root
ARRAY /dev/md0 metadata=0.90 UUID=4472a74f:ca206897:3b82bcad:daeae9f9






root@os-node1:~# cat /etc/multipath.conf

blacklist {
       wwid 3600508b1001cbb0c5b5a2fe0a85e0053
       wwid 3600508b1001c33e8fe607dfd73043d03
       wwid 32020030102060804

}


multipaths {
       multipath {
               wwid                    35000cca04f1cbdfc
               alias                   hgst-ssd1
               path_grouping_policy    multibus
               path_selector           "round-robin 0"
               failback                2
               rr_weight               uniform
               no_path_retry           0
               rr_min_io               1000
       }

       multipath {
               wwid                    35000cca04f1cc050
               alias                   hgst-ssd2
               path_grouping_policy    multibus
               path_selector           "round-robin 0"
               failback                2
               rr_weight               uniform
               no_path_retry           0
               rr_min_io               1000
       }

       multipath {
               wwid                    35000cca04f1ce930
               alias                   hgst-ssd3
               path_grouping_policy    multibus
               path_selector           "round-robin 0"
               failback                2
               rr_weight               uniform
               no_path_retry           0
               rr_min_io               1000
       }

       multipath {
               wwid                    35000cca04f1ce1f4
               alias                   hgst-ssd4
               path_grouping_policy    multibus
               path_selector           "round-robin 0"
               failback                2
               rr_weight               uniform
               no_path_retry           0
               rr_min_io               1000
       }

       multipath {
               wwid                    35000cca04f1cbe18
               alias                   hgst-ssd5
               path_grouping_policy    multibus
               path_selector           "round-robin 0"
               failback                2
               rr_weight               uniform
               no_path_retry           0
               rr_min_io               1000
       }

       multipath {
               wwid                    35000cca04f1cdc58
               alias                   hgst-ssd6
               path_grouping_policy    multibus
               path_selector           "round-robin 0"
               failback                2
               rr_weight               uniform
               no_path_retry           0
               rr_min_io               1000



with this config I do:

root@os-node1:~# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-3.16.0-48-generic


Then I reboot the computer but the mdadm raid doesn't start up. The
multipathed devides does start up:


root@os-node1:~# multipath -ll
Error: : Inappropriate ioctl for device
cciss TUR  failed in CCISS_GETLUNINFO: Inappropriate ioctl for device
Error: : Inappropriate ioctl for device
cciss TUR  failed in CCISS_GETLUNINFO: Inappropriate ioctl for device
hgst-ssd6 (35000cca04f1cdc58) dm-5 HGST    ,HUSMM1680ASS200
size=745G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:2:0 sde   8:64   active ready  running
  `- 4:0:2:0 sdk   8:160  active ready  running
hgst-ssd5 (35000cca04f1cbe18) dm-7 HGST    ,HUSMM1680ASS200
size=745G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:4:0 sdg   8:96   active ready  running
  `- 4:0:4:0 sdm   8:192  active ready  running
hgst-ssd4 (35000cca04f1ce1f4) dm-3 HGST    ,HUSMM1680ASS200
size=745G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:0:0 sdc   8:32   active ready  running
  `- 4:0:0:0 sdi   8:128  active ready  running
hgst-ssd3 (35000cca04f1ce930) dm-4 HGST    ,HUSMM1680ASS200
size=745G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:1:0 sdd   8:48   active ready  running
  `- 4:0:1:0 sdj   8:144  active ready  running
hgst-ssd2 (35000cca04f1cc050) dm-8 HGST    ,HUSMM1680ASS200
size=745G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:5:0 sdh   8:112  active ready  running
  `- 4:0:5:0 sdn   8:208  active ready  running
hgst-ssd1 (35000cca04f1cbdfc) dm-6 HGST    ,HUSMM1680ASS200
size=745G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:3:0 sdf   8:80   active ready  running
  `- 4:0:3:0 sdl   8:176  active ready  running


Regards,
Remek


On Fri, Dec 11, 2015 at 3:04 PM, Phil Turmel <philip@turmel.org> wrote:
> On 12/10/2015 05:44 PM, P. Remek wrote:
>>>
>>> Order of device discovery is not guaranteed, but base devices will
>>> almost always show up before multipath devices.  You have to filter out
>>> the base devices from mdadm consideration:
>>>
>>> Add a DEVICES statement to your mdadm.conf that only matches your
>>> multipath device names, not the base names.  Then update your initramfs
>>> so it applies to early boot as well.
>>>
>>
>>
>> This is something which I actually already tried. I specified in my
>> /etc/mdadm/mdadm.conf following:
>>
>> DEVICE /dev/mapper/hgst-ssd*
>>
>> In my /etc/multipath.conf I instruct dm-multipath to create
>> /dev/mapper/hgst-ssd1 /dev/mapper/hgst-ssd2 /dev/mapper/hgst-ssd3
>
> You haven't said that you've updated your initramfs with this info.
>
>> But the only result is that after this config, the md raid is not
>> started at all. My conclusion was that when md raid was starting, the
>> multipath devices did not yet exist so it did not start up the array.
>
> That would be an initramfs bug, most likely.  Modern initramfs like
> dracut use incremental assembly to start arrays and subsystems as
> devices are started, regardless what order they show up.
>
> Phil
>

Re: start dm-multipath before mdadm raid

[Phil Turmel]

{Convention on kernel.org is to trim and either interleave the reply or
bottom post.  Please do.}

On 12/11/2015 11:01 AM, P. Remek wrote:
> Here is my config:

> root@os-node1:~# cat /etc/mdadm/mdadm.conf
> 
> DEVICE /dev/mapper/hgst-ssd*
> CREATE owner=root group=disk mode=0660 auto=yes
> HOMEHOST <system>
> MAILADDR root
> ARRAY /dev/md0 metadata=0.90 UUID=4472a74f:ca206897:3b82bcad:daeae9f9

It's likely that the udev rules that drive incremental assembly don't
have the mapper aliases at the moment they run.  Try:

DEVICE /dev/dm*

Phil

Re: start dm-multipath before mdadm raid

[P. Remek]

> It's likely that the udev rules that drive incremental assembly don't
> have the mapper aliases at the moment they run.  Try:
>
> DEVICE /dev/dm*
>

That crossed my mind too, but no, this doesn't help, the array still
doesn't show up

Re: start dm-multipath before mdadm raid

[Phil Turmel]

On 12/11/2015 11:32 AM, P. Remek wrote:
>> It's likely that the udev rules that drive incremental assembly don't
>> have the mapper aliases at the moment they run.  Try:
>>
>> DEVICE /dev/dm*
> 
> That crossed my mind too, but no, this doesn't help, the array still
> doesn't show up

Next step would be to run udevadm trigger while running udevadm monitor
in another session right after boot to see what events run and what
rules are executed.  That should help narrow this down.  Also compare
dmesg from boot with dmesg from udevadm trigger.

I'm not familiar enough with ubuntu to be much more help.

Re: start dm-multipath before mdadm raid

[P. Remek]

Not sure what to look for, the only relevant thing I see is:


>
> Next step would be to run udevadm trigger while running udevadm monitor
> in another session right after boot to see what events run and what
> rules are executed.  That should help narrow this down.  Also compare
> dmesg from boot with dmesg from udevadm trigger.
>
> I'm not familiar enough with ubuntu to be much more help.



UDEV  [5341.078565] change   /devices/virtual/block/dm-1 (block)
UDEV  [5341.079255] change   /devices/virtual/block/dm-2 (block)
UDEV  [5341.082457] change   /devices/virtual/block/dm-0 (block)
UDEV  [5341.085049] change   /devices/virtual/block/dm-6 (block)
UDEV  [5341.087382] change   /devices/virtual/block/dm-8 (block)
UDEV  [5341.088389] change   /devices/virtual/block/dm-7 (block)
UDEV  [5341.089766] change   /devices/virtual/block/dm-5 (block)
UDEV  [5341.090079] change   /devices/virtual/block/dm-4 (block)
UDEV  [5341.090105] change   /devices/virtual/block/dm-3 (block)

Re: start dm-multipath before mdadm raid

[P. Remek]

>
> But really, you need to provide details.  Can you setup a small test
> MD RAID and show all the details?
>

I've provided all the details in the other response in this thread.
What do you want me to set up and what other details do you need?

Regards,
Remek

Re: start dm-multipath before mdadm raid

[John Stoffel]

>>>>> "P" == P Remek <p.remek1@googlemail.com> writes:

>> It's likely that the udev rules that drive incremental assembly don't
>> have the mapper aliases at the moment they run.  Try:
>> 
>> DEVICE /dev/dm*
>> 

P> That crossed my mind too, but no, this doesn't help, the array still
P> doesn't show up

What distro are you based on?  Have you re-built your initramfs and
then looked inside it to check that it made the changes?

Do you have everything setup as a module or a mix?  That can certainly
change things.  It might make sense to re-build the kernel so that the
MDADM stuff is modules, but the rest is compiled in, or visa versa.

John

Re: start dm-multipath before mdadm raid

[P. Remek]

>
> Do you have everything setup as a module or a mix?  That can certainly
> change things.  It might make sense to re-build the kernel so that the
> MDADM stuff is modules, but the rest is compiled in, or visa versa.

Here is all the information I collected: http://pastebin.com/VF4Sfrq5


Regarding the testing setup, we can safely consider whole system as a
test, we will rebuild the system once we figure final config anyway.
So we can play around with the current raid with no risk of data loss.

Re: start dm-multipath before mdadm raid

[P. Remek]

Hello John, Phil,

have you managed to look at the information in the link below? I think
dm-multihost is as module.

Regards,
Remek

>
> Here is all the information I collected: http://pastebin.com/VF4Sfrq5
>
>
> Regarding the testing setup, we can safely consider whole system as a
> test, we will rebuild the system once we figure final config anyway.
> So we can play around with the current raid with no risk of data loss.

Re: start dm-multipath before mdadm raid

[Phil Turmel]

On 12/15/2015 10:00 AM, P. Remek wrote:
> Hello John, Phil,
> 
> have you managed to look at the information in the link below? I think
> dm-multihost is as module.

I skimmed it but didn't have any insight to add.  As I mentioned, I
don't run Ubuntu on bare metal or with any advanced disk setup.  (And I
have no systems with multipath at all.)  I expect that if the list has
the requisite knowledge, they'll pipe up.

Phil

Re: start dm-multipath before mdadm raid

[John Stoffel]

Sorry, haven't had a chance yet.  Maybe later today/tonight.

It might be a good thing to compile your own kernel, using the latest
Linux version (v4.4-rc5) and see how that works for you.

John
 

P> have you managed to look at the information in the link below? I think
P> dm-multihost is as module.

P> Regards,
P> Remek

>> 
>> Here is all the information I collected: http://pastebin.com/VF4Sfrq5
>> 
>> 
>> Regarding the testing setup, we can safely consider whole system as a
>> test, we will rebuild the system once we figure final config anyway.
>> So we can play around with the current raid with no risk of data loss.