[Question] abort shared tags for SCSI drivers

[Question] abort shared tags for SCSI drivers

[Yufen Yu]

Hi, all

Shared tags is introduced to maintains a notion of fairness between
active users. This may be good for nvme with multiple namespace to
avoid starving some users. Right?

However, I don't understand why we introduce the shared tag for SCSI.
IMO, there are two concerns for scsi shared tag:

1) For now, 'shost->can_queue' is used as queue depth in block layer.
And all target drivers share tags on one host. Then, the max tags for
each target can get:

	depth = max((bt->sb.depth + users - 1) / users, 4U);

But, each target driver may have their own capacity of tags and queue depth.
Does shared tag limit target device bandwidth?

2) When add new target or remove device, it may need to freeze other device
to update hctx->flags of BLK_MQ_F_TAG_SHARED. That may hurt performance.

Recently we discuss abort hostwide shared tags for SCSI[0] and sharing tags
across hardware queues[1]. These discussion are abort shared tag. But, I
confuse whether shared tag across hardware queues can solve my concerns as mentioned.

I have not deeply understand for SCSI and please correct me if I got wrong.

[0] https://www.spinics.net/lists/linux-scsi/msg136131.html
[1] https://www.spinics.net/lists/linux-block/msg47504.html

Thanks,
Yufen

Re: [Question] abort shared tags for SCSI drivers

[Ming Lei]

On Thu, Jan 16, 2020 at 12:06:02PM +0800, Yufen Yu wrote:
> Hi, all
> 
> Shared tags is introduced to maintains a notion of fairness between
> active users. This may be good for nvme with multiple namespace to
> avoid starving some users. Right?

Actually nvme namespace is LUN of scsi world.

Shared tags isn't for maintaining fairness, it is just natural sw
implementation of scsi host's tags, since every scsi host shares
tags among all LUNs. If the SCSI host supports real MQ, the tags
is hw-queue wide, otherwise it is host wide.

> 
> However, I don't understand why we introduce the shared tag for SCSI.
> IMO, there are two concerns for scsi shared tag:
> 
> 1) For now, 'shost->can_queue' is used as queue depth in block layer.
> And all target drivers share tags on one host. Then, the max tags for
> each target can get:
> 
> 	depth = max((bt->sb.depth + users - 1) / users, 4U);
> 
> But, each target driver may have their own capacity of tags and queue depth.
> Does shared tag limit target device bandwidth?

No, if the 'target driver' means LUN, each LUN hasn't its independent
tags, maybe it has its own queue depth, which is often for maintaining
fairness among all active LUNs, not real queue depth. 

You may see the patches[1] which try to bypass per-LUN queue depth for SSD.

[1] https://lore.kernel.org/linux-block/20191118103117.978-1-ming.lei@redhat.com/

> 
> 2) When add new target or remove device, it may need to freeze other device
> to update hctx->flags of BLK_MQ_F_TAG_SHARED. That may hurt performance.

Add/removing device isn't a frequent event, so it shouldn't be a real
issue, or you have seen effect on real use case?

> 
> Recently we discuss abort hostwide shared tags for SCSI[0] and sharing tags
> across hardware queues[1]. These discussion are abort shared tag. But, I
> confuse whether shared tag across hardware queues can solve my concerns as mentioned.

Both [1] and [0] are for converting some single queue SCSI host into MQ
because these HBAs support multiple reply queue for completing request,
meantime they only have single tags(so they are SQ driver now). So far
not many such kind of hardwares(HPSA, hisi sas, megaraid_sas, ...).


Thanks, 
Ming

Re: [Question] about shared tags for SCSI drivers

[John Garry]

On 16/01/2020 09:03, Ming Lei wrote:

(fixed title)

> On Thu, Jan 16, 2020 at 12:06:02PM +0800, Yufen Yu wrote:
>> Hi, all
>>
>> Shared tags is introduced to maintains a notion of fairness between
>> active users. This may be good for nvme with multiple namespace to
>> avoid starving some users. Right?
> 
> Actually nvme namespace is LUN of scsi world.
> 
> Shared tags isn't for maintaining fairness, it is just natural sw
> implementation of scsi host's tags, since every scsi host shares
> tags among all LUNs. If the SCSI host supports real MQ, the tags
> is hw-queue wide, otherwise it is host wide.
> 
>>
>> However, I don't understand why we introduce the shared tag for SCSI.
>> IMO, there are two concerns for scsi shared tag:
>>
>> 1) For now, 'shost->can_queue' is used as queue depth in block layer.

Note that in scsi_alloc_sdev(), sdev default queue depth is set to 
shost->cmd_per_lun. Slightly different for ATA devices - see 
ata_scsi_dev_config().

There is then also the scsi_host_template.change_queue_depth() callback 
for SCSI hosts can use to set this.

>> And all target drivers share tags on one host. Then, the max tags for
>> each target can get:
>>
>> 	depth = max((bt->sb.depth + users - 1) / users, 4U);

The host HW is limited by the amount of simultaneous commands it can 
issue, regardless of where these tags are managed.

And you seem to be assuming in this equation that all users will own a 
subset of host tags, which is not true.

>>
>> But, each target driver may have their own capacity of tags and queue depth.
>> Does shared tag limit target device bandwidth?
> 
> No, if the 'target driver' means LUN, each LUN hasn't its independent
> tags, maybe it has its own queue depth, which is often for maintaining
> fairness among all active LUNs, not real queue depth.
> 
> You may see the patches[1] which try to bypass per-LUN queue depth for SSD.
> 
> [1] https://lore.kernel.org/linux-block/20191118103117.978-1-ming.lei@redhat.com/
> 
>>
>> 2) When add new target or remove device, it may need to freeze other device
>> to update hctx->flags of BLK_MQ_F_TAG_SHARED. That may hurt performance.
> 
> Add/removing device isn't a frequent event, so it shouldn't be a real
> issue, or you have seen effect on real use case?
> 
>>
>> Recently we discuss abort hostwide shared tags for SCSI[0] and sharing tags
>> across hardware queues[1]. These discussion are abort shared tag. But, I
>> confuse whether shared tag across hardware queues can solve my concerns as mentioned.
> 
> Both [1] and [0] are for converting some single queue SCSI host into MQ
> because these HBAs support multiple reply queue for completing request,
> meantime they only have single tags(so they are SQ driver now). So far
> not many such kind of hardwares(HPSA, hisi sas, megaraid_sas, ...).
> 
> 
> Thanks,
> Ming
> 

thanks Ming

> .
> 

Re: [Question] abort shared tags for SCSI drivers

[James Bottomley]

On Thu, 2020-01-16 at 17:03 +0800, Ming Lei wrote:
> On Thu, Jan 16, 2020 at 12:06:02PM +0800, Yufen Yu wrote:
> > Hi, all
> > 
> > Shared tags is introduced to maintains a notion of fairness between
> > active users. This may be good for nvme with multiple namespace to
> > avoid starving some users. Right?
> 
> Actually nvme namespace is LUN of scsi world.
> 
> Shared tags isn't for maintaining fairness, it is just natural sw
> implementation of scsi host's tags, since every scsi host shares
> tags among all LUNs. If the SCSI host supports real MQ, the tags
> is hw-queue wide, otherwise it is host wide.

From the standards point of view, this statement is incorrect, it's the
ITLQ that identifies the Nexus, so the same tag *may* be in use for
different commands on different LUNS according to standards.

However, some device drivers choose to use ITQ as the identifier
because it fits with the ITLQ standard and they can use Q as a per
target mailbox identifier.

I don't think this affects the analysis, but I just didn't want people
to think it was always true.  We have some older drivers that will
allocate the same tag for different LUNS and this was incredibly
prevalent in the SPI world where the tag was only  bits.

James

Re: [Question] about shared tags for SCSI drivers

[Yufen Yu]

Hi, ming

On 2020/1/16 17:03, Ming Lei wrote:
> On Thu, Jan 16, 2020 at 12:06:02PM +0800, Yufen Yu wrote:
>> Hi, all
>>
>> Shared tags is introduced to maintains a notion of fairness between
>> active users. This may be good for nvme with multiple namespace to
>> avoid starving some users. Right?
> 
> Actually nvme namespace is LUN of scsi world.
> 
> Shared tags isn't for maintaining fairness, it is just natural sw
> implementation of scsi host's tags, since every scsi host shares
> tags among all LUNs. If the SCSI host supports real MQ, the tags
> is hw-queue wide, otherwise it is host wide.
> 
>>
>> However, I don't understand why we introduce the shared tag for SCSI.
>> IMO, there are two concerns for scsi shared tag:
>>
>> 1) For now, 'shost->can_queue' is used as queue depth in block layer.
>> And all target drivers share tags on one host. Then, the max tags for
>> each target can get:
>>
>> 	depth = max((bt->sb.depth + users - 1) / users, 4U);
>>
>> But, each target driver may have their own capacity of tags and queue depth.
>> Does shared tag limit target device bandwidth?
> 
> No, if the 'target driver' means LUN, each LUN hasn't its independent
> tags, maybe it has its own queue depth, which is often for maintaining
> fairness among all active LUNs, not real queue depth.
> 
> You may see the patches[1] which try to bypass per-LUN queue depth for SSD.
> 
> [1] https://lore.kernel.org/linux-block/20191118103117.978-1-ming.lei@redhat.com/
> 
>>
>> 2) When add new target or remove device, it may need to freeze other device
>> to update hctx->flags of BLK_MQ_F_TAG_SHARED. That may hurt performance.
> 
> Add/removing device isn't a frequent event, so it shouldn't be a real
> issue, or you have seen effect on real use case?

Thanks a lot for your detailed explanation.

We found that removing scsi device will delay a long time (such as 6 * 30s)
for waiting the other device in the same host to complete all IOs, where
some IO retry multiple times. If our driver allowed more times to retry,
removing device will wait longer. That is not expected.

In fact, that is not problem before switching scsi blk-mq. All target
devices are independent when removing.

Thanks,
Yufen

Re: [Question] about shared tags for SCSI drivers

[Ming Lei]

On Fri, Jan 17, 2020 at 03:19:18PM +0800, Yufen Yu wrote:
> Hi, ming
> 
> On 2020/1/16 17:03, Ming Lei wrote:
> > On Thu, Jan 16, 2020 at 12:06:02PM +0800, Yufen Yu wrote:
> > > Hi, all
> > > 
> > > Shared tags is introduced to maintains a notion of fairness between
> > > active users. This may be good for nvme with multiple namespace to
> > > avoid starving some users. Right?
> > 
> > Actually nvme namespace is LUN of scsi world.
> > 
> > Shared tags isn't for maintaining fairness, it is just natural sw
> > implementation of scsi host's tags, since every scsi host shares
> > tags among all LUNs. If the SCSI host supports real MQ, the tags
> > is hw-queue wide, otherwise it is host wide.
> > 
> > > 
> > > However, I don't understand why we introduce the shared tag for SCSI.
> > > IMO, there are two concerns for scsi shared tag:
> > > 
> > > 1) For now, 'shost->can_queue' is used as queue depth in block layer.
> > > And all target drivers share tags on one host. Then, the max tags for
> > > each target can get:
> > > 
> > > 	depth = max((bt->sb.depth + users - 1) / users, 4U);
> > > 
> > > But, each target driver may have their own capacity of tags and queue depth.
> > > Does shared tag limit target device bandwidth?
> > 
> > No, if the 'target driver' means LUN, each LUN hasn't its independent
> > tags, maybe it has its own queue depth, which is often for maintaining
> > fairness among all active LUNs, not real queue depth.
> > 
> > You may see the patches[1] which try to bypass per-LUN queue depth for SSD.
> > 
> > [1] https://lore.kernel.org/linux-block/20191118103117.978-1-ming.lei@redhat.com/
> > 
> > > 
> > > 2) When add new target or remove device, it may need to freeze other device
> > > to update hctx->flags of BLK_MQ_F_TAG_SHARED. That may hurt performance.
> > 
> > Add/removing device isn't a frequent event, so it shouldn't be a real
> > issue, or you have seen effect on real use case?
> 
> Thanks a lot for your detailed explanation.
> 
> We found that removing scsi device will delay a long time (such as 6 * 30s)
> for waiting the other device in the same host to complete all IOs, where
> some IO retry multiple times. If our driver allowed more times to retry,
> removing device will wait longer. That is not expected.

I'd suggest you to figure out why IO timeout is triggered in your
device.

> 
> In fact, that is not problem before switching scsi blk-mq. All target
> devices are independent when removing.

Is there IO timeout triggered before switching to scsi-mq?

I guess it shouldn't be one issue if io timeout isn't triggered.

However, there is still something we can improve, such as,
start concurrent queue freeze in blk_mq_update_tag_set_depth().

Thanks,
Ming

Re: [Question] about shared tags for SCSI drivers

[Yufen Yu]

On 2020/1/17 18:16, Ming Lei wrote:
> On Fri, Jan 17, 2020 at 03:19:18PM +0800, Yufen Yu wrote:
>> Hi, ming
>>
>> On 2020/1/16 17:03, Ming Lei wrote:
>>> On Thu, Jan 16, 2020 at 12:06:02PM +0800, Yufen Yu wrote:
>>>> Hi, all
>>>>
>>>> Shared tags is introduced to maintains a notion of fairness between
>>>> active users. This may be good for nvme with multiple namespace to
>>>> avoid starving some users. Right?
>>>
>>> Actually nvme namespace is LUN of scsi world.
>>>
>>> Shared tags isn't for maintaining fairness, it is just natural sw
>>> implementation of scsi host's tags, since every scsi host shares
>>> tags among all LUNs. If the SCSI host supports real MQ, the tags
>>> is hw-queue wide, otherwise it is host wide.
>>>
>>>>
>>>> However, I don't understand why we introduce the shared tag for SCSI.
>>>> IMO, there are two concerns for scsi shared tag:
>>>>
>>>> 1) For now, 'shost->can_queue' is used as queue depth in block layer.
>>>> And all target drivers share tags on one host. Then, the max tags for
>>>> each target can get:
>>>>
>>>> 	depth = max((bt->sb.depth + users - 1) / users, 4U);
>>>>
>>>> But, each target driver may have their own capacity of tags and queue depth.
>>>> Does shared tag limit target device bandwidth?
>>>
>>> No, if the 'target driver' means LUN, each LUN hasn't its independent
>>> tags, maybe it has its own queue depth, which is often for maintaining
>>> fairness among all active LUNs, not real queue depth.
>>>
>>> You may see the patches[1] which try to bypass per-LUN queue depth for SSD.
>>>
>>> [1] https://lore.kernel.org/linux-block/20191118103117.978-1-ming.lei@redhat.com/
>>>
>>>>
>>>> 2) When add new target or remove device, it may need to freeze other device
>>>> to update hctx->flags of BLK_MQ_F_TAG_SHARED. That may hurt performance.
>>>
>>> Add/removing device isn't a frequent event, so it shouldn't be a real
>>> issue, or you have seen effect on real use case?
>>
>> Thanks a lot for your detailed explanation.
>>
>> We found that removing scsi device will delay a long time (such as 6 * 30s)
>> for waiting the other device in the same host to complete all IOs, where
>> some IO retry multiple times. If our driver allowed more times to retry,
>> removing device will wait longer. That is not expected.
> 
> I'd suggest you to figure out why IO timeout is triggered in your
> device.
> 

I agree with your suggestion. But we cannot prevent IO timeout and
retrying multiple times in device. Right? I think we should handle
gently even in that situation.

>>
>> In fact, that is not problem before switching scsi blk-mq. All target
>> devices are independent when removing.
> 
> Is there IO timeout triggered before switching to scsi-mq?
> 
> I guess it shouldn't be one issue if io timeout isn't triggered >
> However, there is still something we can improve, such as,
> start concurrent queue freeze in blk_mq_update_tag_set_depth().

Before switching scsi-mq, timeout have been triggered as well.
But there is no delay when remove device. And it would not need to
wait IOs in the other device to complete. So, I also think we may
need to improve the freeze for scsi-mq.

Thanks,
Yufen