Re: [PATCH 05/11] vhost scsi: use lio wq cmd submission helper

From: Mike Christie <michael.christie@oracle.com>
To: "Michael S. Tsirkin" <mst@redhat.com>
Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org,
	target-devel@vger.kernel.org, jasowang@redhat.com,
	stefanha@redhat.com, virtualization@lists.linux-foundation.org
Subject: Re: [PATCH 05/11] vhost scsi: use lio wq cmd submission helper
Date: Fri, 5 Feb 2021 12:04:18 -0600	[thread overview]
Message-ID: <e1ffb7ce-42bc-c20a-45f3-a63e8f904d32@oracle.com> (raw)
In-Reply-To: <26fe5be1-842d-d85b-c00a-69694bebf709@oracle.com>

On 2/5/21 11:38 AM, Mike Christie wrote:
> On 2/5/21 10:17 AM, Michael S. Tsirkin wrote:
>> On Thu, Feb 04, 2021 at 05:35:07AM -0600, Mike Christie wrote:
>>> @@ -1132,14 +1127,8 @@ vhost_scsi_handle_vq(struct vhost_scsi *vs, struct vhost_virtqueue *vq)
>>>  		 * vhost_scsi_queue_data_in() and vhost_scsi_queue_status()
>>>  		 */
>>>  		cmd->tvc_vq_desc = vc.head;
>>> -		/*
>>> -		 * Dispatch cmd descriptor for cmwq execution in process
>>> -		 * context provided by vhost_scsi_workqueue.  This also ensures
>>> -		 * cmd is executed on the same kworker CPU as this vhost
>>> -		 * thread to gain positive L2 cache locality effects.
>>> -		 */
>>> -		INIT_WORK(&cmd->work, vhost_scsi_submission_work);
>>> -		queue_work(vhost_scsi_workqueue, &cmd->work);
>>> +		target_queue_cmd_submit(tpg->tpg_nexus->tvn_se_sess,
>>> +					&cmd->tvc_se_cmd);
>>>  		ret = 0;
>>>  err:
>>>  		/*
>>
>> What about this aspect? Will things still stay on the same CPU
> Yes, if that is what it's configured to do.

Oh yeah, I wasn't sure if you were only asking about the code in this
patch or the combined patchset. The above chunk modifies the submission
code. Like I wrote below, there are no changes CPU use wise in that path.

Patch:

[PATCH 11/11] target, vhost-scsi: don't switch cpus on completion

modifies the completion path in LIO so we can complete the cmd on the
CPU that the layers below LIO were configured to complete on. The user
can then configure those layers to complete on the specific CPU it was
submitted on, just one that shares a cache, or what the layer below the
block layer completed it on.

> 
> On the submission path there is no change in behavior. target_queue_cmd_submit
> does queue_work_on so it executes the cmd on the same CPU in LIO. Once
> LIO passes it to the block layer then that layer does whatever is setup.
> 
> On the completion path the low level works the same. The low level
> driver goes by its ISRs/softirq/completion-thread settings, the block layer
> then goes by the queue settings like rq_affinity.
> 
> The change in behavior is that in LIO we will do what was configured
> in the layer below us instead of always trying to complete on the same
> CPU it was submitted on.
> 