From: "Michael S. Tsirkin" <mst@redhat.com>
To: Mike Christie <michael.christie@oracle.com>
Cc: Stefan Hajnoczi <stefanha@redhat.com>,
qemu-devel@nongnu.org, fam@euphon.net,
linux-scsi@vger.kernel.org, target-devel@vger.kernel.org,
jasowang@redhat.com, pbonzini@redhat.com,
virtualization@lists.linux-foundation.org
Subject: Re: [PATCH 00/10] vhost/qemu: thread per IO SCSI vq
Date: Wed, 18 Nov 2020 04:54:07 -0500 [thread overview]
Message-ID: <20201118044620-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <b3343762-bb11-b750-46ec-43b5556f2b8e@oracle.com>
On Tue, Nov 17, 2020 at 01:13:14PM -0600, Mike Christie wrote:
> On 11/17/20 10:40 AM, Stefan Hajnoczi wrote:
> > On Thu, Nov 12, 2020 at 05:18:59PM -0600, Mike Christie wrote:
> >> The following kernel patches were made over Michael's vhost branch:
> >>
> >> https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git/log/?h=vhost
> >>
> >> and the vhost-scsi bug fix patchset:
> >>
> >> https://lore.kernel.org/linux-scsi/20201112170008.GB1555653@stefanha-x1.localdomain/T/#t
> >>
> >> And the qemu patch was made over the qemu master branch.
> >>
> >> vhost-scsi currently supports multiple queues with the num_queues
> >> setting, but we end up with a setup where the guest's scsi/block
> >> layer can do a queue per vCPU and the layers below vhost can do
> >> a queue per CPU. vhost-scsi will then do a num_queue virtqueues,
> >> but all IO gets set on and completed on a single vhost-scsi thread.
> >> After 2 - 4 vqs this becomes a bottleneck.
> >>
> >> This patchset allows us to create a worker thread per IO vq, so we
> >> can better utilize multiple CPUs with the multiple queues. It
> >> implments Jason's suggestion to create the initial worker like
> >> normal, then create the extra workers for IO vqs with the
> >> VHOST_SET_VRING_ENABLE ioctl command added in this patchset.
> >
> > How does userspace find out the tids and set their CPU affinity?
> >
>
> When we create the worker thread we add it to the device owner's cgroup,
> so we end up inheriting those settings like affinity.
>
> However, are you more asking about finer control like if the guest is
> doing mq, and the mq hw queue is bound to cpu0, it would perform
> better if we could bind vhost vq's worker thread to cpu0? I think the
> problem might is if you are in the cgroup then we can't set a specific
> threads CPU affinity to just one specific CPU. So you can either do
> cgroups or not.
Something we wanted to try for a while is to allow userspace
to create threads for us, then specify which vqs it processes.
That would address this set of concerns ...
--
MST
WARNING: multiple messages have this Message-ID (diff)
From: "Michael S. Tsirkin" <mst@redhat.com>
To: Mike Christie <michael.christie@oracle.com>
Cc: fam@euphon.net, linux-scsi@vger.kernel.org, jasowang@redhat.com,
qemu-devel@nongnu.org, virtualization@lists.linux-foundation.org,
target-devel@vger.kernel.org,
Stefan Hajnoczi <stefanha@redhat.com>,
pbonzini@redhat.com
Subject: Re: [PATCH 00/10] vhost/qemu: thread per IO SCSI vq
Date: Wed, 18 Nov 2020 04:54:07 -0500 [thread overview]
Message-ID: <20201118044620-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <b3343762-bb11-b750-46ec-43b5556f2b8e@oracle.com>
On Tue, Nov 17, 2020 at 01:13:14PM -0600, Mike Christie wrote:
> On 11/17/20 10:40 AM, Stefan Hajnoczi wrote:
> > On Thu, Nov 12, 2020 at 05:18:59PM -0600, Mike Christie wrote:
> >> The following kernel patches were made over Michael's vhost branch:
> >>
> >> https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git/log/?h=vhost
> >>
> >> and the vhost-scsi bug fix patchset:
> >>
> >> https://lore.kernel.org/linux-scsi/20201112170008.GB1555653@stefanha-x1.localdomain/T/#t
> >>
> >> And the qemu patch was made over the qemu master branch.
> >>
> >> vhost-scsi currently supports multiple queues with the num_queues
> >> setting, but we end up with a setup where the guest's scsi/block
> >> layer can do a queue per vCPU and the layers below vhost can do
> >> a queue per CPU. vhost-scsi will then do a num_queue virtqueues,
> >> but all IO gets set on and completed on a single vhost-scsi thread.
> >> After 2 - 4 vqs this becomes a bottleneck.
> >>
> >> This patchset allows us to create a worker thread per IO vq, so we
> >> can better utilize multiple CPUs with the multiple queues. It
> >> implments Jason's suggestion to create the initial worker like
> >> normal, then create the extra workers for IO vqs with the
> >> VHOST_SET_VRING_ENABLE ioctl command added in this patchset.
> >
> > How does userspace find out the tids and set their CPU affinity?
> >
>
> When we create the worker thread we add it to the device owner's cgroup,
> so we end up inheriting those settings like affinity.
>
> However, are you more asking about finer control like if the guest is
> doing mq, and the mq hw queue is bound to cpu0, it would perform
> better if we could bind vhost vq's worker thread to cpu0? I think the
> problem might is if you are in the cgroup then we can't set a specific
> threads CPU affinity to just one specific CPU. So you can either do
> cgroups or not.
Something we wanted to try for a while is to allow userspace
to create threads for us, then specify which vqs it processes.
That would address this set of concerns ...
--
MST
WARNING: multiple messages have this Message-ID (diff)
From: "Michael S. Tsirkin" <mst@redhat.com>
To: Mike Christie <michael.christie@oracle.com>
Cc: fam@euphon.net, linux-scsi@vger.kernel.org,
qemu-devel@nongnu.org, virtualization@lists.linux-foundation.org,
target-devel@vger.kernel.org,
Stefan Hajnoczi <stefanha@redhat.com>,
pbonzini@redhat.com
Subject: Re: [PATCH 00/10] vhost/qemu: thread per IO SCSI vq
Date: Wed, 18 Nov 2020 04:54:07 -0500 [thread overview]
Message-ID: <20201118044620-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <b3343762-bb11-b750-46ec-43b5556f2b8e@oracle.com>
On Tue, Nov 17, 2020 at 01:13:14PM -0600, Mike Christie wrote:
> On 11/17/20 10:40 AM, Stefan Hajnoczi wrote:
> > On Thu, Nov 12, 2020 at 05:18:59PM -0600, Mike Christie wrote:
> >> The following kernel patches were made over Michael's vhost branch:
> >>
> >> https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git/log/?h=vhost
> >>
> >> and the vhost-scsi bug fix patchset:
> >>
> >> https://lore.kernel.org/linux-scsi/20201112170008.GB1555653@stefanha-x1.localdomain/T/#t
> >>
> >> And the qemu patch was made over the qemu master branch.
> >>
> >> vhost-scsi currently supports multiple queues with the num_queues
> >> setting, but we end up with a setup where the guest's scsi/block
> >> layer can do a queue per vCPU and the layers below vhost can do
> >> a queue per CPU. vhost-scsi will then do a num_queue virtqueues,
> >> but all IO gets set on and completed on a single vhost-scsi thread.
> >> After 2 - 4 vqs this becomes a bottleneck.
> >>
> >> This patchset allows us to create a worker thread per IO vq, so we
> >> can better utilize multiple CPUs with the multiple queues. It
> >> implments Jason's suggestion to create the initial worker like
> >> normal, then create the extra workers for IO vqs with the
> >> VHOST_SET_VRING_ENABLE ioctl command added in this patchset.
> >
> > How does userspace find out the tids and set their CPU affinity?
> >
>
> When we create the worker thread we add it to the device owner's cgroup,
> so we end up inheriting those settings like affinity.
>
> However, are you more asking about finer control like if the guest is
> doing mq, and the mq hw queue is bound to cpu0, it would perform
> better if we could bind vhost vq's worker thread to cpu0? I think the
> problem might is if you are in the cgroup then we can't set a specific
> threads CPU affinity to just one specific CPU. So you can either do
> cgroups or not.
Something we wanted to try for a while is to allow userspace
to create threads for us, then specify which vqs it processes.
That would address this set of concerns ...
--
MST
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
next prev parent reply other threads:[~2020-11-18 9:54 UTC|newest]
Thread overview: 114+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-12 23:18 [PATCH 00/10] vhost/qemu: thread per IO SCSI vq Mike Christie
2020-11-12 23:18 ` Mike Christie
2020-11-12 23:19 ` [PATCH 1/1] qemu vhost scsi: add VHOST_SET_VRING_ENABLE support Mike Christie
2020-11-12 23:19 ` Mike Christie
2020-11-17 11:53 ` Stefan Hajnoczi
2020-11-17 11:53 ` Stefan Hajnoczi
2020-11-17 11:53 ` Stefan Hajnoczi
2020-12-02 9:59 ` Michael S. Tsirkin
2020-12-02 9:59 ` Michael S. Tsirkin
2020-12-02 9:59 ` Michael S. Tsirkin
2020-12-02 16:05 ` Michael Christie
2020-12-02 16:05 ` Michael Christie
2020-11-12 23:19 ` [PATCH 01/10] vhost: remove work arg from vhost_work_flush Mike Christie
2020-11-12 23:19 ` Mike Christie
2020-11-17 13:04 ` Stefan Hajnoczi
2020-11-17 13:04 ` Stefan Hajnoczi
2020-11-17 13:04 ` Stefan Hajnoczi
2020-11-12 23:19 ` [PATCH 02/10] vhost scsi: remove extra flushes Mike Christie
2020-11-12 23:19 ` Mike Christie
2020-11-17 13:07 ` Stefan Hajnoczi
2020-11-17 13:07 ` Stefan Hajnoczi
2020-11-17 13:07 ` Stefan Hajnoczi
2020-11-12 23:19 ` [PATCH 03/10] vhost poll: fix coding style Mike Christie
2020-11-12 23:19 ` Mike Christie
2020-11-17 13:07 ` Stefan Hajnoczi
2020-11-17 13:07 ` Stefan Hajnoczi
2020-11-17 13:07 ` Stefan Hajnoczi
2020-11-12 23:19 ` [PATCH 04/10] vhost: support multiple worker threads Mike Christie
2020-11-12 23:19 ` Mike Christie
2020-11-12 23:19 ` [PATCH 05/10] vhost: poll support support multiple workers Mike Christie
2020-11-12 23:19 ` Mike Christie
2020-11-17 15:32 ` Stefan Hajnoczi
2020-11-17 15:32 ` Stefan Hajnoczi
2020-11-17 15:32 ` Stefan Hajnoczi
2020-11-12 23:19 ` [PATCH 06/10] vhost scsi: make SCSI cmd completion per vq Mike Christie
2020-11-12 23:19 ` Mike Christie
2020-11-17 16:04 ` Stefan Hajnoczi
2020-11-17 16:04 ` Stefan Hajnoczi
2020-11-17 16:04 ` Stefan Hajnoczi
2020-11-12 23:19 ` [PATCH 07/10] vhost, vhost-scsi: flush IO vqs then send TMF rsp Mike Christie
2020-11-12 23:19 ` Mike Christie
2020-11-17 16:05 ` Stefan Hajnoczi
2020-11-17 16:05 ` Stefan Hajnoczi
2020-11-17 16:05 ` Stefan Hajnoczi
2020-11-12 23:19 ` [PATCH 08/10] vhost: move msg_handler to new ops struct Mike Christie
2020-11-12 23:19 ` Mike Christie
2020-11-17 16:08 ` Stefan Hajnoczi
2020-11-17 16:08 ` Stefan Hajnoczi
2020-11-17 16:08 ` Stefan Hajnoczi
2020-11-12 23:19 ` [PATCH 09/10] vhost: add VHOST_SET_VRING_ENABLE support Mike Christie
2020-11-12 23:19 ` Mike Christie
2020-11-17 16:14 ` Stefan Hajnoczi
2020-11-17 16:14 ` Stefan Hajnoczi
2020-11-17 16:14 ` Stefan Hajnoczi
2020-11-12 23:19 ` [PATCH 10/10] vhost-scsi: create a woker per IO vq Mike Christie
2020-11-12 23:19 ` Mike Christie
2020-11-17 16:40 ` [PATCH 00/10] vhost/qemu: thread per IO SCSI vq Stefan Hajnoczi
2020-11-17 16:40 ` Stefan Hajnoczi
2020-11-17 16:40 ` Stefan Hajnoczi
2020-11-17 19:13 ` Mike Christie
2020-11-17 19:13 ` Mike Christie
2020-11-18 9:54 ` Michael S. Tsirkin [this message]
2020-11-18 9:54 ` Michael S. Tsirkin
2020-11-18 9:54 ` Michael S. Tsirkin
2020-11-19 14:00 ` Stefan Hajnoczi
2020-11-19 14:00 ` Stefan Hajnoczi
2020-11-19 14:00 ` Stefan Hajnoczi
2020-11-18 11:31 ` Stefan Hajnoczi
2020-11-18 11:31 ` Stefan Hajnoczi
2020-11-18 11:31 ` Stefan Hajnoczi
2020-11-19 14:46 ` Michael S. Tsirkin
2020-11-19 14:46 ` Michael S. Tsirkin
2020-11-19 14:46 ` Michael S. Tsirkin
2020-11-19 16:11 ` Mike Christie
2020-11-19 16:11 ` Mike Christie
2020-11-19 16:24 ` Stefan Hajnoczi
2020-11-19 16:24 ` Stefan Hajnoczi
2020-11-19 16:24 ` Stefan Hajnoczi
2020-11-19 16:43 ` Mike Christie
2020-11-19 16:43 ` Mike Christie
2020-11-19 17:08 ` Stefan Hajnoczi
2020-11-19 17:08 ` Stefan Hajnoczi
2020-11-19 17:08 ` Stefan Hajnoczi
2020-11-20 8:45 ` Stefan Hajnoczi
2020-11-20 8:45 ` Stefan Hajnoczi
2020-11-20 8:45 ` Stefan Hajnoczi
2020-11-20 12:31 ` Michael S. Tsirkin
2020-11-20 12:31 ` Michael S. Tsirkin
2020-11-20 12:31 ` Michael S. Tsirkin
2020-12-01 12:59 ` Stefan Hajnoczi
2020-12-01 12:59 ` Stefan Hajnoczi
2020-12-01 12:59 ` Stefan Hajnoczi
2020-12-01 13:45 ` Stefano Garzarella
2020-12-01 13:45 ` Stefano Garzarella
2020-12-01 13:45 ` Stefano Garzarella
2020-12-01 17:43 ` Stefan Hajnoczi
2020-12-01 17:43 ` Stefan Hajnoczi
2020-12-01 17:43 ` Stefan Hajnoczi
2020-12-02 10:35 ` Stefano Garzarella
2020-12-02 10:35 ` Stefano Garzarella
2020-12-02 10:35 ` Stefano Garzarella
2020-11-23 15:17 ` Stefano Garzarella
2020-11-23 15:17 ` Stefano Garzarella
2020-11-23 15:17 ` Stefano Garzarella
2020-11-18 5:17 ` Jason Wang
2020-11-18 5:17 ` Jason Wang
2020-11-18 6:57 ` Mike Christie
2020-11-18 7:19 ` Mike Christie
2020-11-18 7:54 ` Jason Wang
2020-11-18 7:54 ` Jason Wang
2020-11-18 20:06 ` Mike Christie
2020-11-19 4:35 ` Jason Wang
2020-11-19 4:35 ` Jason Wang
2020-11-19 15:49 ` Mike Christie
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201118044620-mutt-send-email-mst@kernel.org \
--to=mst@redhat.com \
--cc=fam@euphon.net \
--cc=jasowang@redhat.com \
--cc=linux-scsi@vger.kernel.org \
--cc=michael.christie@oracle.com \
--cc=pbonzini@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=stefanha@redhat.com \
--cc=target-devel@vger.kernel.org \
--cc=virtualization@lists.linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.