RHEL 5.4 multipath doesn't handle virtio block devices

RHEL 5.4 multipath doesn't handle virtio block devices

[Martin Schwenke]

[Repost since subscribing - post from 2010-03-23 never made it...]

multipath in RHEL 5.4 doesn't handle virtio block devices.  It excludes
them because they use a sysfs bus that is not recognised.

Here's a patch that seems to fix this:

diff -r -u multipath-tools-0.4.7.rhel5.16.orig/libmultipath/discovery.c multipath-tools-0.4.7.rhel5.16/libmultipath/discovery.c
--- multipath-tools-0.4.7.rhel5.16.orig/libmultipath/discovery.c	2009-04-21 10:05:22.000000000 +1000
+++ multipath-tools-0.4.7.rhel5.16/libmultipath/discovery.c	2010-03-22 10:17:36.000000000 +1100
@@ -486,8 +486,6 @@
 		pp->bus = SYSFS_BUS_CCW;
 	else if (!strncmp(sdev->bus, "cciss", 5)) 
 		pp->bus = SYSFS_BUS_CCISS;
-	else
-		return 1;
 
 	sysfs_close_device(sdev);
 


The problem occurs because on line 490 of libmultipath/discovery.c
sysfs_get_bus() returns 1 (failure) if the bus type is not recognised.
The function should just fall through and return 0 (success), leaving
pp->bus set to the default value SYSFS_BUS_UNDEF.

This can be confirmed by noting that SYSFS_BUS_IDE is only ever used
once in the code (i.e. when pp->bus is set to SYSFS_BUS_IDE in
sysfs_get_bus()).  This shows that although there is no special case
code for SYSFS_BUS_IDE the rest of code works fine.  Similarly,
without a special case for SYSFS_BUS_UNDEF the rest of the code will
work fine.

To complicate the analysis very slightly, sysfs_pathinfo() does have a
special case for SYSFS_BUS_UNDEF.  However, it could be removed
because it simply returns 0 and this is the default return value of
sysfs_pathinfo()...

A quick scan of the current upstream multipath code in git seems to
suggest that it copes fine with unknown sysfs bus types and behaves as I
describe above.  In fact, the code no longer seems to have a special
case for SYSFS_BUS_IDE, which seems sensible.

I've tested the patch above on a KVM guest with virtio block devices
and it appears to work just fine.  Given that IDE (apparently) works
and the code has no special handling for it suggests that the change
should be backward compatible.  However, I don't have various pieces of
hardware handy to test with.

I'm happy to post a description of how to reproduce this if anyone is
interested.  It basically involves writing a unique ID at the beginning
each disk image, telling libvirt to create 2 devices pointing at each
image and then using a script for getuid_callout that reads the unique
IDs.  Hence, we're able to simulate multiple paths on virtio block
devices.

peace & happiness,
martin

Re: RHEL 5.4 multipath doesn't handle virtio block devices

[Kiyoshi Ueda]

Hi Martin,

On 2010/03/30 10:16 +0900, Martin Schwenke wrote:
> multipath in RHEL 5.4 doesn't handle virtio block devices.  It excludes
> them because they use a sysfs bus that is not recognised.
> 
> Here's a patch that seems to fix this:

Why do you want to use dm-multipath on virtio block devices?

Thanks,
Kiyoshi Ueda

Re: RHEL 5.4 multipath doesn't handle virtio block devices

[Martin Schwenke]

On Tue, 30 Mar 2010 14:07:02 +0900, Kiyoshi Ueda <k-ueda@ct.jp.nec.com>
wrote:

> Why do you want to use dm-multipath on virtio block devices?

For consistency rather than failover...  :-)

I'm using virtual machines to test a clustered NAS environment.  The
production setup expects to see dm devices from dm-multipath.  So I'm
using dm-multipath on virtio block devices to simulate use of
dm-multipath in a SAN environment.  

The RHEL 5.4 version of KVM doesn't contain a SCSI emulation, so the
only 2 choices for large numbers of block devices are virtio and
iSCSI.  I'm trying to keep my options open for both of these.

The fix for virtio block devices seems quite straightforward, so I
thought I would propose it.  :-)

peace & happiness,
martin

Re: RHEL 5.4 multipath doesn't handle virtio block devices

[Hannes Reinecke]

Martin Schwenke wrote:
> On Tue, 30 Mar 2010 14:07:02 +0900, Kiyoshi Ueda <k-ueda@ct.jp.nec.com>
> wrote:
> 
>> Why do you want to use dm-multipath on virtio block devices?
> 
> For consistency rather than failover...  :-)
> 
> I'm using virtual machines to test a clustered NAS environment.  The
> production setup expects to see dm devices from dm-multipath.  So I'm
> using dm-multipath on virtio block devices to simulate use of
> dm-multipath in a SAN environment.  
> 
megasas megasas megasas ...
I posted a megasas HBA emulation for KVM a while back.
Would solve your problem rather elegantly.
But then you'd have to recompile KVM, at which point you could include
the SCSI emulation as well ...

> The RHEL 5.4 version of KVM doesn't contain a SCSI emulation, so the
> only 2 choices for large numbers of block devices are virtio and
> iSCSI.  I'm trying to keep my options open for both of these.
> 
> The fix for virtio block devices seems quite straightforward, so I
> thought I would propose it.  :-)
> 
Problem is, though, that you don't have a unique ID for virtio devices.
EG scsi_id and friends don't work here.
And AFAIR there is no feature flag which would allow you do set one.

How do you distinguish between, say, 2 virtio devices across 2 paths?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

Re: RHEL 5.4 multipath doesn't handle virtio block devices

[Martin Schwenke]

On Tue, 30 Mar 2010 09:16:10 +0200, Hannes Reinecke <hare@suse.de>
wrote:

> megasas megasas megasas ...
> I posted a megasas HBA emulation for KVM a while back.
> Would solve your problem rather elegantly.
> But then you'd have to recompile KVM, at which point you could include
> the SCSI emulation as well ...

Right, that sounds good.  However, for reasons I won't go into, I need
to use the version of KVM that ships with RHEL 5.  So recompiling or
even using RPMs provided by a 3rd party isn't an easy option.  :-(

> Problem is, though, that you don't have a unique ID for virtio
> devices. EG scsi_id and friends don't work here.
> And AFAIR there is no feature flag which would allow you do set one.
> 
> How do you distinguish between, say, 2 virtio devices across 2 paths?

I write a unique identifier onto each image using dd and then as you
would expect, in the libvirt XML configuration, I make 2 devices point
to each image.

Then I use a custom getuid_callout setting, as follows:

  [root@strg001st068 ~]# cat /sbin/scsi_id_autocluster.sh
  #!/bin/sh
  # get scsi ID assuming disks setup by autocluster
  /bin/dd if=/dev/$1 bs=1 count=13 2> /dev/null | cat -v
  echo

The filesystem being used on those disks doesn't use the first
block...  ;-)

It works nicely...

peace & happiness,
martin