RE: QLogic Linux failover/Load Balancing ER0000000020860

RE: QLogic Linux failover/Load Balancing ER0000000020860

[Austin Gonyou]

The latest driver does not load balance with STK D178 array either.

I've discovered why, but I'm not sure which direction to take with
helping out to get better closure/resolution with this.


The cause of the problem is that the QLogic driver doesn't to
transparent LUN masking it seems. The reason this is a problem, is that
when the LSI/StoragTEK controllers present their luns, AVT is enabled.
This causes LUN "ghosting" down each path from the storage to the HBAs.
This becomes a problem because when the Linux Driver is told to perform
load balancing via static bindings, the LUNs are now out of order.
(whether LUN ghosting is happening or not). 

This is where transparent LUN masking would solve this problem. 

Example:
HBA0 is told it's only allowed to address the even numbered volumes.
HBA1 is told it's only allowed to address the odd-numberd volumes. 

When this happens, the driver can see all the volumes, and names them
0:0:0:0-16(or whatever the last number is) for HBA0, and then
1:0:0:0-16(or whatever thelast number is). But, the OS is not shown
that, it sees the real LUN numbers as presented by the storage.
(0,2,4,6,8,10,etc) 

Linux is not allowed to address LUNs out of sequence, so searching for
further LUN numbers stops after 0, since 2 is the next one. 

Is there a way to resolve this, either at the driver level, IMHO the
place it *should* happen. At the storage level, the place that it could
also happen, or in the Kernel?

For the time being, I'm using a temporary load balanced setup for
performance reasons since we just extended our two primary loops from 1
tray each, to 3 and 4 trays. Please advise ASAP, as in this
configuration, we cannot fail-over. 

TIA
-- 
Austin Gonyou <austin@coremetrics.com>
Systems Architect
Coremetrics, Inc.
Cel: 512-698-7250

RE: QLogic Linux failover/Load Balancing ER0000000020860

[GrandMasterLee]

I was hoping someone on this list might have some ideas about this.
Please respond to me at this address or the Coremetrics one.  I'm not
sure exactly what approach to take on this. Any hints would be helpful.


TIA.

On Sat, 2002-10-05 at 19:09, Austin Gonyou wrote:
> The latest driver does not load balance with STK D178 array either.
> 
> I've discovered why, but I'm not sure which direction to take with
> helping out to get better closure/resolution with this.
> 
> 
> The cause of the problem is that the QLogic driver doesn't to
> transparent LUN masking it seems. The reason this is a problem, is that
> when the LSI/StoragTEK controllers present their luns, AVT is enabled.
> This causes LUN "ghosting" down each path from the storage to the HBAs.
> This becomes a problem because when the Linux Driver is told to perform
> load balancing via static bindings, the LUNs are now out of order.
> (whether LUN ghosting is happening or not). 
> 
....
> Linux is not allowed to address LUNs out of sequence, so searching for
> further LUN numbers stops after 0, since 2 is the next one. 
> 
> Is there a way to resolve this, either at the driver level, IMHO the
> place it *should* happen. At the storage level, the place that it could
> also happen, or in the Kernel?
> 
> For the time being, I'm using a temporary load balanced setup for
> performance reasons since we just extended our two primary loops from 1
> tray each, to 3 and 4 trays. Please advise ASAP, as in this
> configuration, we cannot fail-over. 
> 
> TIA
> -- 
> Austin Gonyou <austin@coremetrics.com>
> Systems Architect
> Coremetrics, Inc.
> Cel: 512-698-7250
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: QLogic Linux failover/Load Balancing ER0000000020860

[Lars Marowsky-Bree]

On 2002-10-05T19:09:26,
   Austin Gonyou <austin@coremetrics.com> said:

> Is there a way to resolve this, either at the driver level, IMHO the
> place it *should* happen. At the storage level, the place that it could
> also happen, or in the Kernel?

You can always use md multipathing; an extension to the 2.4 multipathing has
been implemented by Jens Axboe and yours truely and is available at
http://lars.marowsky-bree.de/dl/md-mp; we'll see how Neil takes it when he
returns from vacation ;-)

We'll also be shipping that patch as part of United Linux.

IBM also did an extension to the LVM1 code to support multipathing; I don't
have an URL handy right now, but Google will certainly help out.

For 2.5, this is still not fully hashed out, but I assume you are running 2.4
on a production system ;-)


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
Principal Squirrel
Research and Development, SuSE Linux AG
 
``Immortality is an adequate definition of high availability for me.''
	--- Gregory F. Pfister

Re: QLogic Linux failover/Load Balancing ER0000000020860

[jbradford]

> > Linux is not allowed to address LUNs out of sequence, so searching for
> > further LUN numbers stops after 0, since 2 is the next one. 

That's not true:

CONFIG_SCSI_REPORT_LUNS:

If you want to build with SCSI REPORT LUNS support i the kernel, say Y here.
The REPORT LUNS command is useful for devices (such as disk arrays) with large numbers of LUNs where the LUN values are not contiguous (sparse LUN).
REPORT LUNS scanning is done only for SCSI-3 devices.

> > Is there a way to resolve this, either at the driver level, IMHO the
> > place it *should* happen. At the storage level, the place that it could
> > also happen, or in the Kernel?

This is new in 2.5.x

John.

Re: QLogic Linux failover/Load Balancing ER0000000020860

[GrandMasterLee]

On Sun, 2002-10-06 at 05:19, jbradford@dial.pipex.com wrote:
> > > Linux is not allowed to address LUNs out of sequence, so searching for
> > > further LUN numbers stops after 0, since 2 is the next one. 
> 
> That's not true:
> 
> CONFIG_SCSI_REPORT_LUNS:
> 
> If you want to build with SCSI REPORT LUNS support i the kernel, say Y here.
> The REPORT LUNS command is useful for devices (such as disk arrays) with large numbers of LUNs where the LUN values are not contiguous (sparse LUN).
> REPORT LUNS scanning is done only for SCSI-3 devices.

I believe my kernel has that configured. I will look when I wake. It's
530 am now, and I've been setting up my volumes for a while now. Just
about time to go to sleep, then wake up and install oracle. :)

> > > Is there a way to resolve this, either at the driver level, IMHO the
> > > place it *should* happen. At the storage level, the place that it could
> > > also happen, or in the Kernel?
> 
> This is new in 2.5.x
> 


I see. ATM I'm using 2.4.19, but would like to get to 2.4.20, because of
the TG3 fixes. 

> John.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: QLogic Linux failover/Load Balancing ER0000000020860

[jbradford]

> > > > Linux is not allowed to address LUNs out of sequence, so searching for
> > > > further LUN numbers stops after 0, since 2 is the next one. 
> > 
> > That's not true:
> > 
> > CONFIG_SCSI_REPORT_LUNS:
> > 
> > If you want to build with SCSI REPORT LUNS support i the kernel, say Y
> > here.
> > The REPORT LUNS command is useful for devices (such as disk arrays) with
> > large numbers of LUNs where the LUN values are not contiguous (sparse LUN).
> > REPORT LUNS scanning is done only for SCSI-3 devices.
> 
> I believe my kernel has that configured. I will look when I wake. It's
> 530 am now, and I've been setting up my volumes for a while now. Just
> about time to go to sleep, then wake up and install oracle. :)

All night hacking sessions, are cool :-).

> > > > Is there a way to resolve this, either at the driver level, IMHO the
> > > > place it *should* happen. At the storage level, the place that it could
> > > > also happen, or in the Kernel?
> > 
> > This is new in 2.5.x
> > 
> 
> 
> I see. ATM I'm using 2.4.19, but would like to get to 2.4.20, because of
> the TG3 fixes. 

You were probably thinking of CONFIG_SCSI_MULTI_LUN above, then.  That causes the kernel to probe all LUNs instead of just LUN 0, which is the default due to a lot of broken devices to respond to all LUNs instead of just LUN 0.  The sparse LUN option is in addition to that in 2.5.x.

If this is for a live server, it might be easiest to hard code the LUNs you need it to probe in to 2.4.x for now, and wait until 2.6.x for proper support.

John.

Re: QLogic Linux failover/Load Balancing ER0000000020860

[Michael Clark]

>>I see. ATM I'm using 2.4.19, but would like to get to 2.4.20, because of
>>the TG3 fixes. 
> 
> 
> You were probably thinking of CONFIG_SCSI_MULTI_LUN above, then.  That causes the kernel to probe all LUNs instead of just LUN 0, which is the default due to a lot of broken devices to respond to all LUNs instead of just LUN 0.  The sparse LUN option is in addition to that in 2.5.x.
> 
> If this is for a live server, it might be easiest to hard code the LUNs you need it to probe in to 2.4.x for now, and wait until 2.6.x for proper support.

2.4 supports sparse lun scanning, although it is not enabled
dynamically and requires add a BLIST_SPARSELUN flag for your device
in the device_list array in drivers/scsi/scsi_scan.c

You just need to get the Vendor and Model info from /proc/scsi/scsi

I am using qlogic 2300s with sparse luns working fine on 2.4.18.

~mc

Re: QLogic Linux failover/Load Balancing ER0000000020860

[GrandMasterLee]

On Sun, 2002-10-06 at 07:26, Michael Clark wrote:
> >>I see. ATM I'm using 2.4.19, but would like to get to 2.4.20, because of
> >>the TG3 fixes. 
> > 
> > 
> > You were probably thinking of CONFIG_SCSI_MULTI_LUN above, then.  That causes the kernel to probe all LUNs instead of just LUN 0, which is the default due to a lot of broken devices to respond to all LUNs instead of just LUN 0.  The sparse LUN option is in addition to that in 2.5.x.
> > 
> > If this is for a live server, it might be easiest to hard code the LUNs you need it to probe in to 2.4.x for now, and wait until 2.6.x for proper support.
> 
> 2.4 supports sparse lun scanning, although it is not enabled
> dynamically and requires add a BLIST_SPARSELUN flag for your device
> in the device_list array in drivers/scsi/scsi_scan.c

I worked with Andrea to get the PV660F doing this, because it was not
working correctly, so I know of what you speak. I'll get that info from
/proc/scsi/scsi and then fix it up and see if that does it. 

> You just need to get the Vendor and Model info from /proc/scsi/scsi
> 
> I am using qlogic 2300s with sparse luns working fine on 2.4.18.

using the LB static bindings and *failover* still works?


> ~mc
> 

Re: QLogic Linux failover/Load Balancing ER0000000020860

[GrandMasterLee]

On Sun, 2002-10-06 at 14:40, GrandMasterLee wrote:
> On Sun, 2002-10-06 at 07:26, Michael Clark wrote:
> > >>I see. ATM I'm using 2.4.19, but would like to get to 2.4.20, because of
> > >>the TG3 fixes. 
> > > 
> > > 
> > > You were probably thinking of CONFIG_SCSI_MULTI_LUN above, then.  That causes the kernel to probe all LUNs instead of just LUN 0, which is the default due to a lot of broken devices to respond to all LUNs instead of just LUN 0.  The sparse LUN option is in addition to that in 2.5.x.
> > > 
> > > If this is for a live server, it might be easiest to hard code the LUNs you need it to probe in to 2.4.x for now, and wait until 2.6.x for proper support.
> > 
> > 2.4 supports sparse lun scanning, although it is not enabled
> > dynamically and requires add a BLIST_SPARSELUN flag for your device
> > in the device_list array in drivers/scsi/scsi_scan.c
> 
> I worked with Andrea to get the PV660F doing this, because it was not
> working correctly, so I know of what you speak. I'll get that info from
> /proc/scsi/scsi and then fix it up and see if that does it. 

I just reassigned all my LUNs to be a part of the same host
configuration on the storage(polling by HBAs and host, versus splitting
LUNs by HBA). I do get more than 1 LUN now, but only EVEN luns. I'll see
if I can identify why that is. 

> > You just need to get the Vendor and Model info from /proc/scsi/scsi
> > 
> > I am using qlogic 2300s with sparse luns working fine on 2.4.18.
> 
> using the LB static bindings and *failover* still works?
> 
> 
> > ~mc
> > 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: QLogic Linux failover/Load Balancing ER0000000020860

[GrandMasterLee]

On Sun, 2002-10-06 at 18:14, GrandMasterLee wrote:

> I just reassigned all my LUNs to be a part of the same host
> configuration on the storage(polling by HBAs and host, versus splitting
> LUNs by HBA). I do get more than 1 LUN now, but only EVEN luns. I'll see
> if I can identify why that is. 

After defining LSI in drivers/scsi/scsi_scan.c I can get half my luns,
but still not all. I'm not sure what else I need to do. I now can see
LUNs 0,2,4,6,8, etc but not 1,3,5,7,etc. I'm not sure what else to do,
but maybe now that I've done this, I can get information from QLogic
about what should be happening. Or does this still seem like a kernel
config issue? 

TIA
> > > You just need to get the Vendor and Model info from /proc/scsi/scsi
> > > 
> > > I am using qlogic 2300s with sparse luns working fine on 2.4.18.
> > 
> > using the LB static bindings and *failover* still works?

Re: QLogic Linux failover/Load Balancing ER0000000020860

[Michael Clark]

On 10/07/02 12:54, GrandMasterLee wrote:
> On Sun, 2002-10-06 at 18:14, GrandMasterLee wrote:
> 
> 
>>I just reassigned all my LUNs to be a part of the same host
>>configuration on the storage(polling by HBAs and host, versus splitting
>>LUNs by HBA). I do get more than 1 LUN now, but only EVEN luns. I'll see
>>if I can identify why that is. 
> 
> 
> After defining LSI in drivers/scsi/scsi_scan.c I can get half my luns,
> but still not all. I'm not sure what else I need to do. I now can see
> LUNs 0,2,4,6,8, etc but not 1,3,5,7,etc. I'm not sure what else to do,
> but maybe now that I've done this, I can get information from QLogic
> about what should be happening. Or does this still seem like a kernel
> config issue? 

So sparse lun scanning is working then - sounds like your missing luns
is a problem with your array configuration as the kernel is probing them
(if it is was creating the even ones) - means the qlogic driver must
not be able to see these luns. Not familiar with your array so can't
help any more - your array vendor would probably be the most help.

~mc

Re: QLogic Linux failover/Load Balancing ER0000000020860

[GrandMasterLee]

On Mon, 2002-10-07 at 00:15, Michael Clark wrote:

> So sparse lun scanning is working then - sounds like your missing luns
> is a problem with your array configuration as the kernel is probing them
> (if it is was creating the even ones) - means the qlogic driver must
> not be able to see these luns. Not familiar with your array so can't
> help any more - your array vendor would probably be the most help.
> 
> ~mc
> 

Seems so. The odd thing is, that the hba1, vs. hba0, is not listed as
seeing any LUNs just existing. I'll go back and re-check my config. My
DSL really sux ATM, so doing things through X sessions has been a bit
painful. 

--The GrandMaster

Re: QLogic Linux failover/Load Balancing ER0000000020860

[Austin Gonyou]

[-- Attachment #1: Type: text/plain, Size: 1085 bytes --]

On Sun, 2002-10-06 at 02:58, Lars Marowsky-Bree wrote:
> On 2002-10-05T19:09:26,
>    Austin Gonyou <austin@coremetrics.com> said:
> 
> > Is there a way to resolve this, either at the driver level, IMHO the
> > place it *should* happen. At the storage level, the place that it could
> > also happen, or in the Kernel?
> 
> You can always use md multipathing; an extension to the 2.4 multipathing has
> been implemented by Jens Axboe and yours truely and is available at
> http://lars.marowsky-bree.de/dl/md-mp; we'll see how Neil takes it when he
> returns from vacation ;-)
> 
> We'll also be shipping that patch as part of United Linux.

Is this for 2.4 or 2.5. Just FMI. 


> IBM also did an extension to the LVM1 code to support multipathing; I don't
> have an URL handy right now, but Google will certainly help out.
> 
> For 2.5, this is still not fully hashed out, but I assume you are running 2.4
> on a production system ;-)
> 
> 
> Sincerely,
>     Lars Marowsky-Brée <lmb@suse.de>

-- 
Austin Gonyou <austin@coremetrics.com>
Coremetrics, Inc.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 232 bytes --]

RE: QLogic Linux failover/Load Balancing ER0000000020860

[GrandMasterLee]

On Sun, 2002-10-06 at 10:45, James Bottomley wrote:
> > The reason this is a problem, is that when the LSI/StoragTEK
> > controllers present their luns, AVT is enabled.
> 
> Others have answered the kernel questions, but just a note that you really 
> don't want to do load balancing in this environment.

Hard-coding it to get load-balancing isn't terrible, and when the qlogic
driver works, the way I've got my controllers setup right now, it will
work just like powerpath, etc. There are no *ghosted* luns ATM, just the
*other luns*. Only when a failure occurrs, do I want them to be able to
be addressed down the alternate path. Since Target Reset is on, and the
driver is aware of all LUNs, it should allow the failover to occur. I'm
going to do some testing right now and see anyway, after adding
sparselun for the STK array. 

> the way AVT works is that a LUN is locked to a specific controller (although 
> it has a ghost on the alternate controller).  If you send an I/O packet to the 
> alternate controller, the controllers will immediately negotiate to transfer 
> the LUN across (AVT is Auto Volume Transfer).  It takes quite a while (in I/O 
> terms) for the LUN to transfer, so if you load balance to this array you'll 
> end up killing performance because most of the time will be spent oscillating 
> the LUN.
> 
> The way the setup was intended to work was for simple failover, where you only 
> use an alternate path if the primary fails.

Currently things are more hard-coded than I'd like, but we're redundant
in many ways anyway. I'm going to see if the sparseluns bits at least
lets it keep working, but then I'll see if it actually does keep the
luns in more AVT time, than in operational time. 

> In general, arrays that can gain performance from controller load balancing 
> tend to be extremely expensive (EMC being the one that springs immediately to 
> mind).
> 
> James
> 

RE: QLogic Linux failover/Load Balancing ER0000000020860

[James Bottomley]

> The reason this is a problem, is that when the LSI/StoragTEK
> controllers present their luns, AVT is enabled.

Others have answered the kernel questions, but just a note that you really 
don't want to do load balancing in this environment.

the way AVT works is that a LUN is locked to a specific controller (although 
it has a ghost on the alternate controller).  If you send an I/O packet to the 
alternate controller, the controllers will immediately negotiate to transfer 
the LUN across (AVT is Auto Volume Transfer).  It takes quite a while (in I/O 
terms) for the LUN to transfer, so if you load balance to this array you'll 
end up killing performance because most of the time will be spent oscillating 
the LUN.

The way the setup was intended to work was for simple failover, where you only 
use an alternate path if the primary fails.

In general, arrays that can gain performance from controller load balancing 
tend to be extremely expensive (EMC being the one that springs immediately to 
mind).

James