* [PATCH] scsi: break from queue depth adjusting loops when device found @ 2014-07-03 15:05 Stephen M. Cameron 2014-07-03 15:25 ` Douglas Gilbert 2014-07-03 17:11 ` Christoph Hellwig 0 siblings, 2 replies; 11+ messages in thread From: Stephen M. Cameron @ 2014-07-03 15:05 UTC (permalink / raw) To: =james.bottomley; +Cc: elliott, stephenmcameron, hch, linux-scsi From: Stephen M. Cameron <scameron@beardog.cce.hp.com> Don't loop through all the devices even after finding the one we're looking for Signed-off-by: Stephen M. Cameron <scameron@beardog.cce.hp.com> Reviewed-by: Robert Elliott <elliott@hp.com> --- drivers/scsi/scsi_error.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c index cbe38e5..db8a488 100644 --- a/drivers/scsi/scsi_error.c +++ b/drivers/scsi/scsi_error.c @@ -635,6 +635,7 @@ static void scsi_handle_queue_ramp_up(struct scsi_device *sdev) sht->change_queue_depth(tmp_sdev, tmp_sdev->queue_depth + 1, SCSI_QDEPTH_RAMP_UP); sdev->last_queue_ramp_up = jiffies; + break; } } @@ -657,6 +658,7 @@ static void scsi_handle_queue_full(struct scsi_device *sdev) */ sht->change_queue_depth(tmp_sdev, tmp_sdev->queue_depth - 1, SCSI_QDEPTH_QFULL); + break; } } ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH] scsi: break from queue depth adjusting loops when device found 2014-07-03 15:05 [PATCH] scsi: break from queue depth adjusting loops when device found Stephen M. Cameron @ 2014-07-03 15:25 ` Douglas Gilbert 2014-07-03 17:11 ` Christoph Hellwig 1 sibling, 0 replies; 11+ messages in thread From: Douglas Gilbert @ 2014-07-03 15:25 UTC (permalink / raw) To: Stephen M. Cameron, =james.bottomley Cc: elliott, stephenmcameron, hch, linux-scsi On 14-07-03 11:05 AM, Stephen M. Cameron wrote: > From: Stephen M. Cameron <scameron@beardog.cce.hp.com> > > Don't loop through all the devices even after > finding the one we're looking for > > Signed-off-by: Stephen M. Cameron <scameron@beardog.cce.hp.com> > Reviewed-by: Robert Elliott <elliott@hp.com> With my scsi_debug testing on Christoph's core-for-3.17 tree I can simulate infrequent TSF conditions using the every_nth mechanism. When one is triggered, all scsi devices (that scsi_debug is monitoring) get their queue_depth adjusted. So: Acked-by: Douglas Gilbert <dgilbert@interlog.com> > --- > drivers/scsi/scsi_error.c | 2 ++ > 1 files changed, 2 insertions(+), 0 deletions(-) > > diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c > index cbe38e5..db8a488 100644 > --- a/drivers/scsi/scsi_error.c > +++ b/drivers/scsi/scsi_error.c > @@ -635,6 +635,7 @@ static void scsi_handle_queue_ramp_up(struct scsi_device *sdev) > sht->change_queue_depth(tmp_sdev, tmp_sdev->queue_depth + 1, > SCSI_QDEPTH_RAMP_UP); > sdev->last_queue_ramp_up = jiffies; > + break; > } > } > > @@ -657,6 +658,7 @@ static void scsi_handle_queue_full(struct scsi_device *sdev) > */ > sht->change_queue_depth(tmp_sdev, tmp_sdev->queue_depth - 1, > SCSI_QDEPTH_QFULL); > + break; > } > } > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-scsi" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] scsi: break from queue depth adjusting loops when device found 2014-07-03 15:05 [PATCH] scsi: break from queue depth adjusting loops when device found Stephen M. Cameron 2014-07-03 15:25 ` Douglas Gilbert @ 2014-07-03 17:11 ` Christoph Hellwig 2014-07-03 17:21 ` Mike Christie 2014-07-04 10:53 ` Hannes Reinecke 1 sibling, 2 replies; 11+ messages in thread From: Christoph Hellwig @ 2014-07-03 17:11 UTC (permalink / raw) To: Stephen M. Cameron Cc: james.bottomley, elliott, stephenmcameron, linux-scsi, Vasu Dev, Mike Christie On Thu, Jul 03, 2014 at 10:05:57AM -0500, Stephen M. Cameron wrote: > From: Stephen M. Cameron <scameron@beardog.cce.hp.com> > > Don't loop through all the devices even after > finding the one we're looking for The comments in the code seem to indicate that we want to modify the queue depth for all LUNs on a given target. Ccing Mike and Vasu as they wrote this code. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] scsi: break from queue depth adjusting loops when device found 2014-07-03 17:11 ` Christoph Hellwig @ 2014-07-03 17:21 ` Mike Christie 2014-07-04 18:36 ` Douglas Gilbert 2014-07-04 10:53 ` Hannes Reinecke 1 sibling, 1 reply; 11+ messages in thread From: Mike Christie @ 2014-07-03 17:21 UTC (permalink / raw) To: Christoph Hellwig Cc: Stephen M. Cameron, james.bottomley, elliott, stephenmcameron, linux-scsi, Vasu Dev, James Smart On 07/03/2014 12:11 PM, Christoph Hellwig wrote: > On Thu, Jul 03, 2014 at 10:05:57AM -0500, Stephen M. Cameron wrote: >> From: Stephen M. Cameron <scameron@beardog.cce.hp.com> >> >> Don't loop through all the devices even after >> finding the one we're looking for > > The comments in the code seem to indicate that we want to modify > the queue depth for all LUNs on a given target. > > Ccing Mike and Vasu as they wrote this code. > Ccing James Smart for more details. We copied this behavior from lpfc's SAM_STAT_TASK_SET_FULL handling (look at around the 2.6.32 kernel for reference). ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] scsi: break from queue depth adjusting loops when device found 2014-07-03 17:21 ` Mike Christie @ 2014-07-04 18:36 ` Douglas Gilbert 0 siblings, 0 replies; 11+ messages in thread From: Douglas Gilbert @ 2014-07-04 18:36 UTC (permalink / raw) To: Mike Christie, Christoph Hellwig Cc: Stephen M. Cameron, james.bottomley, elliott, stephenmcameron, linux-scsi, Vasu Dev, James Smart On 14-07-03 01:21 PM, Mike Christie wrote: > On 07/03/2014 12:11 PM, Christoph Hellwig wrote: >> On Thu, Jul 03, 2014 at 10:05:57AM -0500, Stephen M. Cameron wrote: >>> From: Stephen M. Cameron <scameron@beardog.cce.hp.com> >>> >>> Don't loop through all the devices even after >>> finding the one we're looking for >> >> The comments in the code seem to indicate that we want to modify >> the queue depth for all LUNs on a given target. >> >> Ccing Mike and Vasu as they wrote this code. >> > > Ccing James Smart for more details. > > We copied this behavior from lpfc's SAM_STAT_TASK_SET_FULL handling > (look at around the 2.6.32 kernel for reference). Perhaps there should be a knob to control this, maybe within the scope of a host. I have one RAID controller that has a JBOD mode in which real disks are presented as LUs in the same target. Obviously the TSF (queue_type and other) capabilities of those real disks might differ. Doug Gilbert ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] scsi: break from queue depth adjusting loops when device found 2014-07-03 17:11 ` Christoph Hellwig 2014-07-03 17:21 ` Mike Christie @ 2014-07-04 10:53 ` Hannes Reinecke 2014-07-04 19:21 ` Elliott, Robert (Server Storage) 1 sibling, 1 reply; 11+ messages in thread From: Hannes Reinecke @ 2014-07-04 10:53 UTC (permalink / raw) To: Christoph Hellwig, Stephen M. Cameron Cc: james.bottomley, elliott, stephenmcameron, linux-scsi, Vasu Dev, Mike Christie On 07/03/2014 07:11 PM, Christoph Hellwig wrote: > On Thu, Jul 03, 2014 at 10:05:57AM -0500, Stephen M. Cameron wrote: >> From: Stephen M. Cameron <scameron@beardog.cce.hp.com> >> >> Don't loop through all the devices even after >> finding the one we're looking for > > The comments in the code seem to indicate that we want to modify > the queue depth for all LUNs on a given target. > > Ccing Mike and Vasu as they wrote this code. > Indeed, that was the idea. This piece of code tries to keep track of the remote port queue depth, which isn't represented at all. Thing is, each remote target has a target queue depth which can hold only so many outstanding SCSI requests. If that is full it'll return BUSY for _all_ LUNs served from that port. And the very _next_ command after the one which filled the target queue will get the BUSY status. So we need to decrease the queue depth on _all_ LUNs here to avoid starvation of individual devices. Hence I guess this is not the correct fix. Cheers, Hannes -- Dr. Hannes Reinecke zSeries & Storage hare@suse.de +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] scsi: break from queue depth adjusting loops when device found 2014-07-04 10:53 ` Hannes Reinecke @ 2014-07-04 19:21 ` Elliott, Robert (Server Storage) 2014-07-06 9:49 ` Christoph Hellwig 2014-07-26 15:36 ` Christoph Hellwig 0 siblings, 2 replies; 11+ messages in thread From: Elliott, Robert (Server Storage) @ 2014-07-04 19:21 UTC (permalink / raw) To: Hannes Reinecke, Christoph Hellwig, Stephen M. Cameron Cc: james.bottomley, stephenmcameron, linux-scsi, Vasu Dev, Mike Christie > -----Original Message----- > From: Hannes Reinecke [mailto:hare@suse.de] > Sent: Friday, 04 July, 2014 5:53 AM > To: Christoph Hellwig; Stephen M. Cameron > Cc: james.bottomley@parallels.com; Elliott, Robert (Server Storage); > stephenmcameron@gmail.com; linux-scsi@vger.kernel.org; Vasu Dev; Mike > Christie > Subject: Re: [PATCH] scsi: break from queue depth adjusting loops when device > found > > On 07/03/2014 07:11 PM, Christoph Hellwig wrote: > > On Thu, Jul 03, 2014 at 10:05:57AM -0500, Stephen M. Cameron wrote: > >> From: Stephen M. Cameron <scameron@beardog.cce.hp.com> > >> > >> Don't loop through all the devices even after > >> finding the one we're looking for > > > > The comments in the code seem to indicate that we want to modify > > the queue depth for all LUNs on a given target. > > > > Ccing Mike and Vasu as they wrote this code. > > > Indeed, that was the idea. > This piece of code tries to keep track of the remote port queue > depth, which isn't represented at all. > Thing is, each remote target has a target queue depth which can hold > only so many outstanding SCSI requests. If that is full it'll return > BUSY for _all_ LUNs served from that port. > And the very _next_ command after the one which filled the target > queue will get the BUSY status. > So we need to decrease the queue depth on _all_ LUNs here to avoid > starvation of individual devices. > > Hence I guess this is not the correct fix. Some SCSI target device designs work that way, but others don't. In the original SCSI Architecture Model (SAM-1; 1995): * BUSY meant "the logical unit is busy" * TASK SET FULL meant "the logical unit does not have enough resources" It didn't say "SCSI target port" or "SCSI target device", so they were not intended to be affected by other logical units. That was based on the idealized notion that each I_T_L nexus was independent, also assumed by the hierarchical LUN addressing scheme and the SCSI Controller Commands (SCC) standard. With that interpretation, the command identifier/tag (the Q in I_T_L_Q nexus) is independent for each I_T_L nexus - two logical units behind the same SCSI target port could use the same tag value at the same time for different commands. However, every SCSI transport protocol after parallel SCSI has chosen to share command tags across all logical units; passing the (usually) 8-byte LUN field in every IU/frame is not practical. So, that idealized I_T_L nexus concept has broken down. Implementations really have a mix of SCSI target device, SCSI target port, and logical unit resources. When we added the optional status qualifier to SAM-4 (2006), we added a SCOPE field indicating if the scope of the BUSY or TASK SET FULL is for: * just the logical unit; * all logical units in the SCSI target device accessible through the SCSI target port; or * all logical units in the SCSI target device. There is no VPD page field defined to report which scope(s) are implemented or describing the resource limits. Some designs are simple, others are complicated. If TASK SET FULL means a SCSI target port limit has been reached, then decrementing the limit on all the logical units will over-correct. If the limit is 256 and there are 16 LUNs, nominally 16 per LUN, and you send a 257th command, then the total will drop to 16x15=240, not 256. mpt3sas assigns separate target numbers for target ports it discovers, so the SCSI midlayer queue depth logic is more correct if those SCSI target devices are implementing a SCSI target port scope for BUSY and TASK SET FULL, but incorrect if they are implementing a logical unit scope. [ 2.711047] scsi 0:0:0:0: Direct-Access HP EO0400JDVFB HPD1 PQ: 0 ANSI: 6 [ 2.711746] scsi 0:0:1:0: Direct-Access HP EO0400JDVFB HPD1 PQ: 0 ANSI: 6 [ 2.712423] scsi 0:0:2:0: Direct-Access HP EO0400JDVFB HPD1 PQ: 0 ANSI: 6 hpsa presents multiple LUNs in one target, and a TASK SET FULL on one logical unit doesn't mean that you'll get it on the others, so the SCSI midlayer queue depth logic is incorrect: [ 10.790885] scsi 2:0:0:0: Direct-Access HP LOGICAL VOLUME 10.0 PQ: 0 ANSI: 5 [ 10.791095] scsi 2:0:0:1: Direct-Access HP LOGICAL VOLUME 10.0 PQ: 0 ANSI: 5 [ 10.791300] scsi 2:0:0:2: Direct-Access HP LOGICAL VOLUME 10.0 PQ: 0 ANSI: 5 The SCSI host template .cmd_per_lun value, which is used to set the default initial queue depth for each device (i.e., each LUN), implies the logical unit scope. Perhaps the SCSI midlayer should keep track of both SCSI target port and logical unit queue depths, parse the status qualifier if present, and let the host template advise on the policy to assume if the status qualifier is not present. Short of that, I think it's best to assume logical unit scope, which always the presumption in SCSI, for BUSY and TASK SET FULL. There is some code for a scsi_target structure that I don't understand and have just been ignoring: target_busy, target_blocked, etc. Does that represent the SCSI target port over multiple logical units, or does that relate to target-mode where the system is acting as the SCSI target and presenting logical units itself? --- Rob Elliott HP Server Storage ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] scsi: break from queue depth adjusting loops when device found 2014-07-04 19:21 ` Elliott, Robert (Server Storage) @ 2014-07-06 9:49 ` Christoph Hellwig 2014-07-26 15:36 ` Christoph Hellwig 1 sibling, 0 replies; 11+ messages in thread From: Christoph Hellwig @ 2014-07-06 9:49 UTC (permalink / raw) To: Elliott, Robert (Server Storage) Cc: Hannes Reinecke, Christoph Hellwig, Stephen M. Cameron, james.bottomley, stephenmcameron, linux-scsi, Vasu Dev, Mike Christie On Fri, Jul 04, 2014 at 07:21:52PM +0000, Elliott, Robert (Server Storage) wrote: > Perhaps the SCSI midlayer should keep track of both > SCSI target port and logical unit queue depths, parse > the status qualifier if present, and let the host > template advise on the policy to assume if the > status qualifier is not present. We already keep track of the target queue depth, although with the scsi-mq series this is not optional and only done for hosts that declare they have a per-target queue limit by setting ->can_queue in the scsi_target structurue. Only a small number of iSCSI offload HBAs does this. I need to look at the SAM scope defintions, but it doesn't sound too hard to implement. > There is some code for a scsi_target structure that I > don't understand and have just been ignoring: target_busy, > target_blocked, etc. Does that represent the SCSI target > port over multiple logical units, or does that relate to > target-mode where the system is acting as the SCSI target > and presenting logical units itself? It represents the target port, and is not related to target mode support at all. The target mode code uses completely different data structures than the initiator side mid layer. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] scsi: break from queue depth adjusting loops when device found 2014-07-04 19:21 ` Elliott, Robert (Server Storage) 2014-07-06 9:49 ` Christoph Hellwig @ 2014-07-26 15:36 ` Christoph Hellwig [not found] ` <CADzpL0Rn+x6N=v123EBej_iZhTwmjMYZ4deeTeRG7_sxgXU9PQ@mail.gmail.com> 1 sibling, 1 reply; 11+ messages in thread From: Christoph Hellwig @ 2014-07-26 15:36 UTC (permalink / raw) To: Elliott, Robert (Server Storage) Cc: Hannes Reinecke, Christoph Hellwig, Stephen M. Cameron, james.bottomley, stephenmcameron, linux-scsi, Vasu Dev, Mike Christie I just saw this patch showing up again in the hpsa series, so let's get the discussiong going on how to proceed again: I think we can't simply apply the current version as it breaks the intended semantics in the FC drivers that expect an array with target-wide ressources. So if we want to go ahead quickly we might just need two versions of it, one that does LUN-level ramp up/down and one that does target level. I'd also very much like to be able to use the SAM-4 indicators, but I fear that a lot of hardware might not be able to get this right. If anyone has experience with using those I'd love to hear about it. ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <CADzpL0Rn+x6N=v123EBej_iZhTwmjMYZ4deeTeRG7_sxgXU9PQ@mail.gmail.com>]
* Re: [PATCH] scsi: break from queue depth adjusting loops when device found [not found] ` <CADzpL0Rn+x6N=v123EBej_iZhTwmjMYZ4deeTeRG7_sxgXU9PQ@mail.gmail.com> @ 2014-07-26 16:17 ` Christoph Hellwig 2014-07-27 15:36 ` Hannes Reinecke 0 siblings, 1 reply; 11+ messages in thread From: Christoph Hellwig @ 2014-07-26 16:17 UTC (permalink / raw) To: Stephen Cameron Cc: Elliott, Robert (Server Storage), Hannes Reinecke, Stephen M. Cameron, james.bottomley, linux-scsi, Vasu Dev, Mike Christie On Sat, Jul 26, 2014 at 11:14:35AM -0500, Stephen Cameron wrote: > Hmm, I forgot that that patch was in there, I wasn't trying to keep pushing > it along. From the previous discussion, I got the impression I was simply > wrong, and that this patch wasn't needed, so I had meant to drop it, I just > forgot to actually drop it. It's more complicated - as Robert indicated you're tenically right, although in practice it's not what the existing users expect. If you have some cycles for it I'd love to at lest get the per-LUN and per-target ramp up/down in ASAP. We can then start looking into doing it even better based on the target response later on. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] scsi: break from queue depth adjusting loops when device found 2014-07-26 16:17 ` Christoph Hellwig @ 2014-07-27 15:36 ` Hannes Reinecke 0 siblings, 0 replies; 11+ messages in thread From: Hannes Reinecke @ 2014-07-27 15:36 UTC (permalink / raw) To: Christoph Hellwig, Stephen Cameron Cc: Elliott, Robert (Server Storage), Stephen M. Cameron, james.bottomley, linux-scsi, Vasu Dev, Mike Christie On 07/26/2014 06:17 PM, Christoph Hellwig wrote: > On Sat, Jul 26, 2014 at 11:14:35AM -0500, Stephen Cameron wrote: >> Hmm, I forgot that that patch was in there, I wasn't trying to keep pushing >> it along. From the previous discussion, I got the impression I was simply >> wrong, and that this patch wasn't needed, so I had meant to drop it, I just >> forgot to actually drop it. > > It's more complicated - as Robert indicated you're tenically right, although > in practice it's not what the existing users expect. If you have some > cycles for it I'd love to at lest get the per-LUN and per-target > ramp up/down in ASAP. We can then start looking into doing it even > better based on the target response later on. > The current implementation is based on the needs of the FC HBAs, which would need a way to throttle I/O to avoid flooding a target port. The reason it was done per target is (from what I can tell) due to a specific implementation from a large vendor which is using a per-target-port request queue. And more often than not defaulting to SCSI-2 conformance to be 'legacy compatible'. So no way one could use any shiny new commands. Having said that it has been quite some time since it's been implemented, and it _might_ be that things have changed and we can get away with a per-LUN throttling. As I doubt we'll get information about this from the various storage vendors (at least not from those we've got issues with even today) we probably have to bite the bullet here and test things out. But hey, it's worth a shot anyway. So, storage vendors, speak up! Cheers, Hannes -- Dr. Hannes Reinecke zSeries & Storage hare@suse.de +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2014-07-27 15:36 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-07-03 15:05 [PATCH] scsi: break from queue depth adjusting loops when device found Stephen M. Cameron 2014-07-03 15:25 ` Douglas Gilbert 2014-07-03 17:11 ` Christoph Hellwig 2014-07-03 17:21 ` Mike Christie 2014-07-04 18:36 ` Douglas Gilbert 2014-07-04 10:53 ` Hannes Reinecke 2014-07-04 19:21 ` Elliott, Robert (Server Storage) 2014-07-06 9:49 ` Christoph Hellwig 2014-07-26 15:36 ` Christoph Hellwig [not found] ` <CADzpL0Rn+x6N=v123EBej_iZhTwmjMYZ4deeTeRG7_sxgXU9PQ@mail.gmail.com> 2014-07-26 16:17 ` Christoph Hellwig 2014-07-27 15:36 ` Hannes Reinecke
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.