From: Max Gurtovoy <mgurtovoy@nvidia.com>
To: Stefan Hajnoczi <stefanha@redhat.com>
Cc: <mst@redhat.com>, <virtualization@lists.linux-foundation.org>,
<kvm@vger.kernel.org>, <oren@nvidia.com>, <nitzanc@nvidia.com>,
<israelr@nvidia.com>, <hch@infradead.org>,
<linux-block@vger.kernel.org>, <axboe@kernel.dk>
Subject: Re: [PATCH 2/2] virtio-blk: set NUMA affinity for a tagset
Date: Mon, 27 Sep 2021 20:39:30 +0300 [thread overview]
Message-ID: <21295187-41c4-5fb6-21c3-28004eb7c5d8@nvidia.com> (raw)
In-Reply-To: <YVF8RBZSaJs9BScd@stefanha-x1.localdomain>
On 9/27/2021 11:09 AM, Stefan Hajnoczi wrote:
> On Sun, Sep 26, 2021 at 05:55:18PM +0300, Max Gurtovoy wrote:
>> To optimize performance, set the affinity of the block device tagset
>> according to the virtio device affinity.
>>
>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>> ---
>> drivers/block/virtio_blk.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
>> index 9b3bd083b411..1c68c3e0ebf9 100644
>> --- a/drivers/block/virtio_blk.c
>> +++ b/drivers/block/virtio_blk.c
>> @@ -774,7 +774,7 @@ static int virtblk_probe(struct virtio_device *vdev)
>> memset(&vblk->tag_set, 0, sizeof(vblk->tag_set));
>> vblk->tag_set.ops = &virtio_mq_ops;
>> vblk->tag_set.queue_depth = queue_depth;
>> - vblk->tag_set.numa_node = NUMA_NO_NODE;
>> + vblk->tag_set.numa_node = virtio_dev_to_node(vdev);
>> vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
>> vblk->tag_set.cmd_size =
>> sizeof(struct virtblk_req) +
> I implemented NUMA affinity in the past and could not demonstrate a
> performance improvement:
> https://lists.linuxfoundation.org/pipermail/virtualization/2020-June/048248.html
>
> The pathological case is when a guest with vNUMA has the virtio-blk-pci
> device on the "wrong" host NUMA node. Then memory accesses should cross
> NUMA nodes. Still, it didn't seem to matter.
I think the reason you didn't see any improvement is since you didn't
use the right device for the node query. See my patch 1/2.
I can try integrating these patches in my series and fix it.
BTW, we might not see a big improvement because of other bottlenecks but
this is known perf optimization we use often in block storage drivers.
>
> Please share your benchmark results. If you haven't collected data yet
> you could even combine our patches to see if it helps. Thanks!
>
> Stefan
next prev parent reply other threads:[~2021-09-27 17:40 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-26 14:55 [PATCH 1/2] virtio: introduce virtio_dev_to_node helper Max Gurtovoy
2021-09-26 14:55 ` [PATCH 2/2] virtio-blk: set NUMA affinity for a tagset Max Gurtovoy
2021-09-27 8:09 ` Stefan Hajnoczi
2021-09-27 17:39 ` Max Gurtovoy [this message]
2021-09-28 6:47 ` Stefan Hajnoczi
2021-09-29 15:07 ` Max Gurtovoy
2021-09-30 13:16 ` Stefan Hajnoczi
2021-09-30 13:24 ` Max Gurtovoy
2021-09-27 11:34 ` Leon Romanovsky
2021-09-27 17:25 ` Max Gurtovoy
2021-09-27 18:23 ` Leon Romanovsky
2021-09-28 15:59 ` Max Gurtovoy
2021-09-28 16:27 ` Leon Romanovsky
2021-09-28 23:28 ` Max Gurtovoy
2021-09-29 6:50 ` Leon Romanovsky
2021-09-29 9:48 ` Max Gurtovoy
2021-09-27 8:02 ` [PATCH 1/2] virtio: introduce virtio_dev_to_node helper Stefan Hajnoczi
2021-09-27 9:31 ` Michael S. Tsirkin
2021-09-28 16:14 ` Max Gurtovoy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=21295187-41c4-5fb6-21c3-28004eb7c5d8@nvidia.com \
--to=mgurtovoy@nvidia.com \
--cc=axboe@kernel.dk \
--cc=hch@infradead.org \
--cc=israelr@nvidia.com \
--cc=kvm@vger.kernel.org \
--cc=linux-block@vger.kernel.org \
--cc=mst@redhat.com \
--cc=nitzanc@nvidia.com \
--cc=oren@nvidia.com \
--cc=stefanha@redhat.com \
--cc=virtualization@lists.linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).