Re: [linux-lvm] Alternate Pathing?

Re: [linux-lvm] Alternate Pathing?

[Heinz Mauelshagen]

On Mon, Jul 24, 2000 at 09:25:19PM +0200, Sergey Vichik wrote:
> 
> Hi,
> 
> 
> It seems to be a litle tricky to detect via LVM which HBA has failed or
> disconnected, because the mapping to real disks occurs at lvm_map, and you
> cannot know whether this IO proccess have ended succesfully or not in
> lvm_map.
> 
> The only solutuion I see, is to pass all IO operations via write/read of LVM
> and detect failed HBA by retrying failed IO request, when eliminating every
> time one of HBAs from mapping till IO operation successfull .

That's roughly what i was thinking to do in the future.
But Martin K. Peter already mentioned that he needs to implement it anyway
and he wants to do it in the general block device layer.
This has the major advantage of beeing available for every block device
rather than only for LVM driven ones.

> Not really effective, ugly, may work.

Ugly only in terms of beeing less general.
It would work.
But the preferable place to implement it is in the request handler
which can take care of requeueing the request to the optional device (path).

Open question anyway is the administration interface.
In case of the existing non self identifying disk devices there's the
need of configuration of the alternate pathes.


> Any other idea ?


Basically Peter's.

Regards,
Heinz  --  the LVM guy

Re: [linux-lvm] Alternate Pathing?

[Jos Visser]

I think it will be a *big* challenge to implement this feature in the
generic block device layer. Alternate pathing depends on the driver
being able to decide that a device probed via a particular controller is
exactly the same device as one that it has already seen through a
controller that has been probed earlier. I think it will be hard to
decide on a unique identifier that works for all types of block devices
(serial number?)

The LVM can do it more easily because it can use the VGid/PVid 
combination in the private areas of the physical volumes to keep track 
of this. 

Furthermore, as an administrator I would like to have control over which
of the paths is the "primary link" and which the "alternate link".
Suppose I have 4 disks in a VG where all 4 disks are reachable through 
2 controllers:

           +-------------------------------------------+
	   |                                           |
	   |                                           |
	   |                                           |
	   |              ------      ------           |
	   |              | A  |      | B  |           |
	   +-------------------------------------------+
                            |           |
	                 /-----\     /-----\
			 |     |     |     |
			 |  1  |     |  2  |
			 +-----+     +-----+
                            |           |
	                 /-----\     /-----\
			 |     |     |     |
			 |  3  |     |  4  |
			 +-----+     +-----+
                            |           |
			    +-----------+


In this case I might want to use controller A as the primary controller
for disks 1 and 3, and controller B as the primary controller for disks
2 and 4. Given a limited SCSI queue depth such a setup would allow me to
have more outstanding I/O's at a given time. Since I can not imagine the
system being able to smartly decide this for me, I must be able to
specify this preference in some way.

As a reference: in HP's LVM, when I vgextend a volume group with a
block device that is already know through another controller, than the
LVM adds this new device's designation as the alternate link for this
physical device.

++Jos

And thus it came to pass that Heinz Mauelshagen wrote:
(on Mon, Jul 24, 2000 at 01:58:18PM -0500 to be exact)

> On Mon, Jul 24, 2000 at 09:25:19PM +0200, Sergey Vichik wrote:
> > 
> > Hi,
> > 
> > 
> > It seems to be a litle tricky to detect via LVM which HBA has failed or
> > disconnected, because the mapping to real disks occurs at lvm_map, and you
> > cannot know whether this IO proccess have ended succesfully or not in
> > lvm_map.
> > 
> > The only solutuion I see, is to pass all IO operations via write/read of LVM
> > and detect failed HBA by retrying failed IO request, when eliminating every
> > time one of HBAs from mapping till IO operation successfull .
> 
> That's roughly what i was thinking to do in the future.
> But Martin K. Peter already mentioned that he needs to implement it anyway
> and he wants to do it in the general block device layer.
> This has the major advantage of beeing available for every block device
> rather than only for LVM driven ones.
> 
> > Not really effective, ugly, may work.
> 
> Ugly only in terms of beeing less general.
> It would work.
> But the preferable place to implement it is in the request handler
> which can take care of requeueing the request to the optional device (path).
> 
> Open question anyway is the administration interface.
> In case of the existing non self identifying disk devices there's the
> need of configuration of the alternate pathes.
> 
> 
> > Any other idea ?
> 
> 
> Basically Peter's.
> 
> Regards,
> Heinz  --  the LVM guy

-- 
Sometimes the best way,
Is with an old cliche.

Re: [linux-lvm] Alternate Pathing?

[Martin K. Petersen]

>>>>> "Jos" == Jos Visser <josv@osp.nl> writes:

Jos> Alternate pathing depends on the driver being able to decide that
Jos> a device probed via a particular controller is exactly the same
Jos> device as one that it has already seen through a controller that
Jos> has been probed earlier. I think it will be hard to decide on a
Jos> unique identifier that works for all types of block devices
Jos> (serial number?)

Fortunately, (at least in the FC-AL case) most vendors are smart
enough to assign UUIDs to their devices.  SCSI inquiries also help.
Finally, ext2 and XFS also have UUIDs in their superblocks.  Hence we
have several ways to get hints about the physical setup.

It is my intention to feature a pluggable driver scheme depending on
the physical device in the other end.  Also, some devices require to
be poked to initiate failover/failback.

Worst case (i.e. with incredibly stupid hardware) people will have to
specify the paths manually.

-- 
Martin K. Petersen, Principal Linux Consultant, Linuxcare, Inc.
mkp@linuxcare.com, http://www.linuxcare.com/
Linuxcare. Support for the revolution.

Re: [linux-lvm] Alternate Pathing?

[gerhard.fuernkranz]

"Martin K. Petersen" wrote:

> Jos> Alternate pathing depends on the driver being able to decide that
> Jos> a device probed via a particular controller is exactly the same
> Jos> device as one that it has already seen through a controller that
> Jos> has been probed earlier. I think it will be hard to decide on a
> Jos> unique identifier that works for all types of block devices
> Jos> (serial number?)
> 
> Fortunately, (at least in the FC-AL case) most vendors are smart
> enough to assign UUIDs to their devices.  SCSI inquiries also help.
> Finally, ext2 and XFS also have UUIDs in their superblocks.  Hence we
> have several ways to get hints about the physical setup.

Using the UUIDs in the superblocks can cause confusion, if you e.g. have two
physical copies of a disk. You'll look at the the superblock and think there
are two paths, but in reality you have two different disks with the same
contents in the superblock.

SCSI and Fibre-Channel devices usually can be uniquely identified by the SCSI
serial number of the device, which can be retrieved with SCSI inquiry.
Unfortunately some devices report different serial numbers for the same
target/lun if you access the lun via a different SCSI/FC port. Some devices
encode the accessing SCSI/FC port number into a few bits of the serial number
so that only a truncated serial number could be used for identification and
again some others (e.g. EMC Symmetrix) can be configured whether they shall
report common serial numbers or not.

> It is my intention to feature a pluggable driver scheme depending on
> the physical device in the other end.

I agree. Despite the problems I mentioned above I still think that for a
majority of devices SCSI serial numbers will be sufficient for unique
identification.

For devices supporting only active/passive mode (see below), a special device
dependent driver is required anyway to perform the path failover in a device
dependent way. Thus such a driver could also do the device dependent LUN
identification if it should be required.

> Also, some devices require to
> be poked to initiate failover/failback.

Correct. Actually with multipathing we have to distinguish 2
strategies depending on the capability of the device:

1. Active - Active:
   Here the device can be accessed via the multiple access paths
   simultaneously. The multiple paths can also used to load
   balance the I/O requests to increase the overall throughput.

   But you have to be very careful. Not each dynamic
   multipath load balancing strategy will automatically increase
   the performance. Some strategies may even make the performance
   worse. Especially round robin strategies implemented in a layer
   that resides on the top of e.g. two disk drivers may be affected.
   Assume that such an MP driver distributes the requests which come
   down to the underlying two disk drivers: one request to the left
   and one to the right ... Although both underlying drivers implement
   an elevator strategy and keep a sorted list of request, the requests
   arriving at the disk (from both drivers simultaneously) are no
   longer sorted and may lead to unnecessary disk head movements,
   which are avoided if all requests are fed through a single disk
   driver only. In the worst case the 1st driver's elevator is going
   up while the 2nd driver's elevator goes down at the same time.
   Of course devices with a large write-back cache will suffer less
   than e.g. simple disks with a small cache or a write-through only
   cache. Furthermore everything will depend heavily on the
   application's disk access pattern.

2. Active - Passive:
   The device can be accessed via multiple paths, but not at the
   same time. Normally you access the device via path1 and
   path2 is idle, and only if path1 fails, then you switch
   the device to the 2nd path and do all futher I/O via path2.
   This switchover is usually done with some (usually proprietary)
   SCSI commands sent to the device.
   Here load balancing can only be achieved in a static way, but
   not dynamically. E.g. you use path1 as default path for Lun 1
   and use path2 as default path for another Lun 2. But whether this
   is possible or not will depend on the device.

My impression is, that (1) is rather supported by more expensive enterprise
storage subsystems, while the cheaper raid subsystems often only support method
(2) if they do support multipathing at all. But there is no general rule for
that. Method (1) is usually also usable for simple, (non-RAID) JBODs ("just a
bunch of disks", e.g. Fibre Channel disks), if you have a storage subsystem,
that provides you redundant, multipathed access to these disks.

> Worst case (i.e. with incredibly stupid hardware) people will have to
> specify the paths manually.

Yes.


Gerhard

Re: [linux-lvm] Alternate Pathing?

[Heinz Mauelshagen]

On Mon, Jul 24, 2000 at 07:58:03AM -0400, Martin K. Petersen wrote:
> >>>>> "Barry" == Barry Roomberg <broom@voicenet.com> writes:
> 
> Barry> Ohh LVM Gurus, I beseech thee: Does the Linux kernel,
> Barry> filesystem, device tree, or LVM have the concept of Alternate
> Barry> Pathing?
> 
> Barry> Does this concept exist anywhere in Linux?
> 
> It's on my todo list...

Is it on your general sort of block device todo list or will
you be helping to enhance the Linux LVM?

Heinz   -- the LVM guy

Re: [linux-lvm] Alternate Pathing?

[Martin K. Petersen]

>>>>> "Barry" == Barry Roomberg <broom@voicenet.com> writes:

Barry> Ohh LVM Gurus, I beseech thee: Does the Linux kernel,
Barry> filesystem, device tree, or LVM have the concept of Alternate
Barry> Pathing?

Barry> Does this concept exist anywhere in Linux?

It's on my todo list...

-- 
Martin K. Petersen, Principal Linux Consultant, Linuxcare, Inc.
mkp@linuxcare.com, http://www.linuxcare.com/
Linuxcare. Support for the revolution.

Re: [linux-lvm] Alternate Pathing?

[Paul Jakma]

On Sun, 23 Jul 2000, Barry Roomberg wrote:

> Does this concept exist anywhere in Linux?
> It seems the right place for it to exist
> would be at the disk side of the LVM.
> 
> I searched GOOGLE and the LVM list archive
> for mentions but did not come up with anything.
> 
> Any pointers?
> 

GFS was designed spefically with FC in mind. see
www.globalfilesystem.org

> Thanks
> Barry
> broom@nospam-voicenet.com
> 
> 

-- 
Paul Jakma	paul@clubi.ie
PGP5 key: http://www.clubi.ie/jakma/publickey.txt
-------------------------------------------
Fortune:
Freedom begins when you tell Mrs. Grundy to go fly a kite.

[linux-lvm] Alternate Pathing?

[Barry Roomberg]

Ohh LVM Gurus, I beseech thee:

Does the Linux kernel, filesystem, device tree,
or LVM have the concept of Alternate Pathing?

My situation is that one of the key features
I have in Solaris (which I want in Linux) is the 
ability to have multiple fibre channel devices 
(disks, RAID luns, etc) seen
via multiple cards in the system.  All are
shown simultaneously through the different paths.
If one of the paths should fail (slot, card, cable,
controller) than the device can be picked up
via the alternative path.

Does this concept exist anywhere in Linux?
It seems the right place for it to exist
would be at the disk side of the LVM.

I searched GOOGLE and the LVM list archive
for mentions but did not come up with anything.

Any pointers?

Thanks
Barry
broom@nospam-voicenet.com