From: Jason Wang <jasowang@redhat.com> To: Stefan Hajnoczi <stefanha@redhat.com>, Mike Christie <michael.christie@oracle.com> Cc: fam@euphon.net, linux-scsi@vger.kernel.org, mst@redhat.com, qemu-devel@nongnu.org, virtualization@lists.linux-foundation.org, target-devel@vger.kernel.org, pbonzini@redhat.com Subject: Re: [PATCH 00/10] vhost/qemu: thread per IO SCSI vq Date: Wed, 18 Nov 2020 13:17:50 +0800 [thread overview] Message-ID: <bba47572-bec9-794f-5a70-d7f016267022@redhat.com> (raw) In-Reply-To: <20201117164043.GS131917@stefanha-x1.localdomain> On 2020/11/18 上午12:40, Stefan Hajnoczi wrote: > On Thu, Nov 12, 2020 at 05:18:59PM -0600, Mike Christie wrote: >> The following kernel patches were made over Michael's vhost branch: >> >> https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git/log/?h=vhost >> >> and the vhost-scsi bug fix patchset: >> >> https://lore.kernel.org/linux-scsi/20201112170008.GB1555653@stefanha-x1.localdomain/T/#t >> >> And the qemu patch was made over the qemu master branch. >> >> vhost-scsi currently supports multiple queues with the num_queues >> setting, but we end up with a setup where the guest's scsi/block >> layer can do a queue per vCPU and the layers below vhost can do >> a queue per CPU. vhost-scsi will then do a num_queue virtqueues, >> but all IO gets set on and completed on a single vhost-scsi thread. >> After 2 - 4 vqs this becomes a bottleneck. >> >> This patchset allows us to create a worker thread per IO vq, so we >> can better utilize multiple CPUs with the multiple queues. It >> implments Jason's suggestion to create the initial worker like >> normal, then create the extra workers for IO vqs with the >> VHOST_SET_VRING_ENABLE ioctl command added in this patchset. > How does userspace find out the tids and set their CPU affinity? > > What is the meaning of the new VHOST_SET_VRING_ENABLE ioctl? It doesn't > really "enable" or "disable" the vq, requests are processed regardless. Actually I think it should do the real "enable/disable" that tries to follow the virtio spec. (E.g both PCI and MMIO have something similar). > > The purpose of the ioctl isn't clear to me because the kernel could > automatically create 1 thread per vq without a new ioctl. It's not necessarily to create or destroy kthread according to VRING_ENABLE but could be a hint. > On the other > hand, if userspace is supposed to control worker threads then a > different interface would be more powerful: > > struct vhost_vq_worker_info { > /* > * The pid of an existing vhost worker that this vq will be > * assigned to. When pid is 0 the virtqueue is assigned to the > * default vhost worker. When pid is -1 a new worker thread is > * created for this virtqueue. When pid is -2 the virtqueue's > * worker thread is unchanged. > * > * If a vhost worker no longer has any virtqueues assigned to it > * then it will terminate. > * > * The pid of the vhost worker is stored to this field when the > * ioctl completes successfully. Use pid -2 to query the current > * vhost worker pid. > */ > __kernel_pid_t pid; /* in/out */ > > /* The virtqueue index*/ > unsigned int vq_idx; /* in */ > }; > > ioctl(vhost_fd, VHOST_SET_VQ_WORKER, &info); This seems to leave the question to userspace which I'm not sure it's good since it tries to introduce another scheduling layer. Per vq worker seems be good enough to start with. Thanks > > Stefan
WARNING: multiple messages have this Message-ID (diff)
From: Jason Wang <jasowang@redhat.com> To: Stefan Hajnoczi <stefanha@redhat.com>, Mike Christie <michael.christie@oracle.com> Cc: fam@euphon.net, linux-scsi@vger.kernel.org, mst@redhat.com, qemu-devel@nongnu.org, virtualization@lists.linux-foundation.org, target-devel@vger.kernel.org, pbonzini@redhat.com Subject: Re: [PATCH 00/10] vhost/qemu: thread per IO SCSI vq Date: Wed, 18 Nov 2020 13:17:50 +0800 [thread overview] Message-ID: <bba47572-bec9-794f-5a70-d7f016267022@redhat.com> (raw) In-Reply-To: <20201117164043.GS131917@stefanha-x1.localdomain> On 2020/11/18 上午12:40, Stefan Hajnoczi wrote: > On Thu, Nov 12, 2020 at 05:18:59PM -0600, Mike Christie wrote: >> The following kernel patches were made over Michael's vhost branch: >> >> https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git/log/?h=vhost >> >> and the vhost-scsi bug fix patchset: >> >> https://lore.kernel.org/linux-scsi/20201112170008.GB1555653@stefanha-x1.localdomain/T/#t >> >> And the qemu patch was made over the qemu master branch. >> >> vhost-scsi currently supports multiple queues with the num_queues >> setting, but we end up with a setup where the guest's scsi/block >> layer can do a queue per vCPU and the layers below vhost can do >> a queue per CPU. vhost-scsi will then do a num_queue virtqueues, >> but all IO gets set on and completed on a single vhost-scsi thread. >> After 2 - 4 vqs this becomes a bottleneck. >> >> This patchset allows us to create a worker thread per IO vq, so we >> can better utilize multiple CPUs with the multiple queues. It >> implments Jason's suggestion to create the initial worker like >> normal, then create the extra workers for IO vqs with the >> VHOST_SET_VRING_ENABLE ioctl command added in this patchset. > How does userspace find out the tids and set their CPU affinity? > > What is the meaning of the new VHOST_SET_VRING_ENABLE ioctl? It doesn't > really "enable" or "disable" the vq, requests are processed regardless. Actually I think it should do the real "enable/disable" that tries to follow the virtio spec. (E.g both PCI and MMIO have something similar). > > The purpose of the ioctl isn't clear to me because the kernel could > automatically create 1 thread per vq without a new ioctl. It's not necessarily to create or destroy kthread according to VRING_ENABLE but could be a hint. > On the other > hand, if userspace is supposed to control worker threads then a > different interface would be more powerful: > > struct vhost_vq_worker_info { > /* > * The pid of an existing vhost worker that this vq will be > * assigned to. When pid is 0 the virtqueue is assigned to the > * default vhost worker. When pid is -1 a new worker thread is > * created for this virtqueue. When pid is -2 the virtqueue's > * worker thread is unchanged. > * > * If a vhost worker no longer has any virtqueues assigned to it > * then it will terminate. > * > * The pid of the vhost worker is stored to this field when the > * ioctl completes successfully. Use pid -2 to query the current > * vhost worker pid. > */ > __kernel_pid_t pid; /* in/out */ > > /* The virtqueue index*/ > unsigned int vq_idx; /* in */ > }; > > ioctl(vhost_fd, VHOST_SET_VQ_WORKER, &info); This seems to leave the question to userspace which I'm not sure it's good since it tries to introduce another scheduling layer. Per vq worker seems be good enough to start with. Thanks > > Stefan _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
next prev parent reply other threads:[~2020-11-18 5:18 UTC|newest] Thread overview: 114+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-11-12 23:18 [PATCH 00/10] vhost/qemu: thread per IO SCSI vq Mike Christie 2020-11-12 23:18 ` Mike Christie 2020-11-12 23:19 ` [PATCH 1/1] qemu vhost scsi: add VHOST_SET_VRING_ENABLE support Mike Christie 2020-11-12 23:19 ` Mike Christie 2020-11-17 11:53 ` Stefan Hajnoczi 2020-11-17 11:53 ` Stefan Hajnoczi 2020-11-17 11:53 ` Stefan Hajnoczi 2020-12-02 9:59 ` Michael S. Tsirkin 2020-12-02 9:59 ` Michael S. Tsirkin 2020-12-02 9:59 ` Michael S. Tsirkin 2020-12-02 16:05 ` Michael Christie 2020-12-02 16:05 ` Michael Christie 2020-11-12 23:19 ` [PATCH 01/10] vhost: remove work arg from vhost_work_flush Mike Christie 2020-11-12 23:19 ` Mike Christie 2020-11-17 13:04 ` Stefan Hajnoczi 2020-11-17 13:04 ` Stefan Hajnoczi 2020-11-17 13:04 ` Stefan Hajnoczi 2020-11-12 23:19 ` [PATCH 02/10] vhost scsi: remove extra flushes Mike Christie 2020-11-12 23:19 ` Mike Christie 2020-11-17 13:07 ` Stefan Hajnoczi 2020-11-17 13:07 ` Stefan Hajnoczi 2020-11-17 13:07 ` Stefan Hajnoczi 2020-11-12 23:19 ` [PATCH 03/10] vhost poll: fix coding style Mike Christie 2020-11-12 23:19 ` Mike Christie 2020-11-17 13:07 ` Stefan Hajnoczi 2020-11-17 13:07 ` Stefan Hajnoczi 2020-11-17 13:07 ` Stefan Hajnoczi 2020-11-12 23:19 ` [PATCH 04/10] vhost: support multiple worker threads Mike Christie 2020-11-12 23:19 ` Mike Christie 2020-11-12 23:19 ` [PATCH 05/10] vhost: poll support support multiple workers Mike Christie 2020-11-12 23:19 ` Mike Christie 2020-11-17 15:32 ` Stefan Hajnoczi 2020-11-17 15:32 ` Stefan Hajnoczi 2020-11-17 15:32 ` Stefan Hajnoczi 2020-11-12 23:19 ` [PATCH 06/10] vhost scsi: make SCSI cmd completion per vq Mike Christie 2020-11-12 23:19 ` Mike Christie 2020-11-17 16:04 ` Stefan Hajnoczi 2020-11-17 16:04 ` Stefan Hajnoczi 2020-11-17 16:04 ` Stefan Hajnoczi 2020-11-12 23:19 ` [PATCH 07/10] vhost, vhost-scsi: flush IO vqs then send TMF rsp Mike Christie 2020-11-12 23:19 ` Mike Christie 2020-11-17 16:05 ` Stefan Hajnoczi 2020-11-17 16:05 ` Stefan Hajnoczi 2020-11-17 16:05 ` Stefan Hajnoczi 2020-11-12 23:19 ` [PATCH 08/10] vhost: move msg_handler to new ops struct Mike Christie 2020-11-12 23:19 ` Mike Christie 2020-11-17 16:08 ` Stefan Hajnoczi 2020-11-17 16:08 ` Stefan Hajnoczi 2020-11-17 16:08 ` Stefan Hajnoczi 2020-11-12 23:19 ` [PATCH 09/10] vhost: add VHOST_SET_VRING_ENABLE support Mike Christie 2020-11-12 23:19 ` Mike Christie 2020-11-17 16:14 ` Stefan Hajnoczi 2020-11-17 16:14 ` Stefan Hajnoczi 2020-11-17 16:14 ` Stefan Hajnoczi 2020-11-12 23:19 ` [PATCH 10/10] vhost-scsi: create a woker per IO vq Mike Christie 2020-11-12 23:19 ` Mike Christie 2020-11-17 16:40 ` [PATCH 00/10] vhost/qemu: thread per IO SCSI vq Stefan Hajnoczi 2020-11-17 16:40 ` Stefan Hajnoczi 2020-11-17 16:40 ` Stefan Hajnoczi 2020-11-17 19:13 ` Mike Christie 2020-11-17 19:13 ` Mike Christie 2020-11-18 9:54 ` Michael S. Tsirkin 2020-11-18 9:54 ` Michael S. Tsirkin 2020-11-18 9:54 ` Michael S. Tsirkin 2020-11-19 14:00 ` Stefan Hajnoczi 2020-11-19 14:00 ` Stefan Hajnoczi 2020-11-19 14:00 ` Stefan Hajnoczi 2020-11-18 11:31 ` Stefan Hajnoczi 2020-11-18 11:31 ` Stefan Hajnoczi 2020-11-18 11:31 ` Stefan Hajnoczi 2020-11-19 14:46 ` Michael S. Tsirkin 2020-11-19 14:46 ` Michael S. Tsirkin 2020-11-19 14:46 ` Michael S. Tsirkin 2020-11-19 16:11 ` Mike Christie 2020-11-19 16:11 ` Mike Christie 2020-11-19 16:24 ` Stefan Hajnoczi 2020-11-19 16:24 ` Stefan Hajnoczi 2020-11-19 16:24 ` Stefan Hajnoczi 2020-11-19 16:43 ` Mike Christie 2020-11-19 16:43 ` Mike Christie 2020-11-19 17:08 ` Stefan Hajnoczi 2020-11-19 17:08 ` Stefan Hajnoczi 2020-11-19 17:08 ` Stefan Hajnoczi 2020-11-20 8:45 ` Stefan Hajnoczi 2020-11-20 8:45 ` Stefan Hajnoczi 2020-11-20 8:45 ` Stefan Hajnoczi 2020-11-20 12:31 ` Michael S. Tsirkin 2020-11-20 12:31 ` Michael S. Tsirkin 2020-11-20 12:31 ` Michael S. Tsirkin 2020-12-01 12:59 ` Stefan Hajnoczi 2020-12-01 12:59 ` Stefan Hajnoczi 2020-12-01 12:59 ` Stefan Hajnoczi 2020-12-01 13:45 ` Stefano Garzarella 2020-12-01 13:45 ` Stefano Garzarella 2020-12-01 13:45 ` Stefano Garzarella 2020-12-01 17:43 ` Stefan Hajnoczi 2020-12-01 17:43 ` Stefan Hajnoczi 2020-12-01 17:43 ` Stefan Hajnoczi 2020-12-02 10:35 ` Stefano Garzarella 2020-12-02 10:35 ` Stefano Garzarella 2020-12-02 10:35 ` Stefano Garzarella 2020-11-23 15:17 ` Stefano Garzarella 2020-11-23 15:17 ` Stefano Garzarella 2020-11-23 15:17 ` Stefano Garzarella 2020-11-18 5:17 ` Jason Wang [this message] 2020-11-18 5:17 ` Jason Wang 2020-11-18 6:57 ` Mike Christie 2020-11-18 7:19 ` Mike Christie 2020-11-18 7:54 ` Jason Wang 2020-11-18 7:54 ` Jason Wang 2020-11-18 20:06 ` Mike Christie 2020-11-19 4:35 ` Jason Wang 2020-11-19 4:35 ` Jason Wang 2020-11-19 15:49 ` Mike Christie
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=bba47572-bec9-794f-5a70-d7f016267022@redhat.com \ --to=jasowang@redhat.com \ --cc=fam@euphon.net \ --cc=linux-scsi@vger.kernel.org \ --cc=michael.christie@oracle.com \ --cc=mst@redhat.com \ --cc=pbonzini@redhat.com \ --cc=qemu-devel@nongnu.org \ --cc=stefanha@redhat.com \ --cc=target-devel@vger.kernel.org \ --cc=virtualization@lists.linux-foundation.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.