From: Jason Wang <jasowang@redhat.com>
To: Xie Yongji <xieyongji@bytedance.com>
Cc: mst@redhat.com, tglx@linutronix.de, hch@lst.de,
virtualization@lists.linux-foundation.org,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 06/11] vduse: Support automatic irq callback affinity
Date: Fri, 16 Dec 2022 13:30:12 +0800 [thread overview]
Message-ID: <CACGkMEuhYO3neFmxwiBp8C0QTaa+Mb13kken+RZ9QuruMct6tA@mail.gmail.com> (raw)
In-Reply-To: <20221205085846.741-1-xieyongji@bytedance.com>
On Mon, Dec 5, 2022 at 4:59 PM Xie Yongji <xieyongji@bytedance.com> wrote:
>
> This brings current interrupt affinity spreading mechanism
> to vduse device. We will make use of irq_create_affinity_masks()
> to create an irq callback affinity mask for each virtqueue of
> vduse device. Then we will choose the CPU which has the lowest
> number of interrupt allocated in the affinity mask to run the
> irq callback.
This seems a balance mechanism but it might not be the semantic of the
affinity or any reason we need to do this? I guess we should use at
least round-robin in this case.
>
> Signed-off-by: Xie Yongji <xieyongji@bytedance.com>
> ---
> drivers/vdpa/vdpa_user/vduse_dev.c | 50 ++++++++++++++++++++++++++++++
> 1 file changed, 50 insertions(+)
>
> diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
> index d126f3e32a20..90c2896039d9 100644
> --- a/drivers/vdpa/vdpa_user/vduse_dev.c
> +++ b/drivers/vdpa/vdpa_user/vduse_dev.c
> @@ -23,6 +23,7 @@
> #include <linux/nospec.h>
> #include <linux/vmalloc.h>
> #include <linux/sched/mm.h>
> +#include <linux/interrupt.h>
> #include <uapi/linux/vduse.h>
> #include <uapi/linux/vdpa.h>
> #include <uapi/linux/virtio_config.h>
> @@ -58,6 +59,8 @@ struct vduse_virtqueue {
> struct work_struct inject;
> struct work_struct kick;
> int irq_effective_cpu;
> + struct cpumask irq_affinity;
> + spinlock_t irq_affinity_lock;
Ok, I'd suggest to squash this into patch 5 to make it more easier to
be reviewed.
> };
>
> struct vduse_dev;
> @@ -123,6 +126,7 @@ struct vduse_control {
>
> static DEFINE_MUTEX(vduse_lock);
> static DEFINE_IDR(vduse_idr);
> +static DEFINE_PER_CPU(unsigned long, vduse_allocated_irq);
>
> static dev_t vduse_major;
> static struct class *vduse_class;
> @@ -710,6 +714,49 @@ static u32 vduse_vdpa_get_generation(struct vdpa_device *vdpa)
> return dev->generation;
> }
>
> +static void vduse_vq_update_effective_cpu(struct vduse_virtqueue *vq)
> +{
> + unsigned int cpu, best_cpu;
> + unsigned long allocated, allocated_min = UINT_MAX;
> +
> + spin_lock(&vq->irq_affinity_lock);
> +
> + best_cpu = vq->irq_effective_cpu;
> + if (best_cpu != -1)
> + per_cpu(vduse_allocated_irq, best_cpu) -= 1;
> +
> + for_each_cpu(cpu, &vq->irq_affinity) {
> + allocated = per_cpu(vduse_allocated_irq, cpu);
> + if (!cpu_online(cpu) || allocated >= allocated_min)
> + continue;
> +
> + best_cpu = cpu;
> + allocated_min = allocated;
> + }
> + vq->irq_effective_cpu = best_cpu;
> + per_cpu(vduse_allocated_irq, best_cpu) += 1;
> +
> + spin_unlock(&vq->irq_affinity_lock);
> +}
> +
> +static void vduse_vdpa_set_irq_affinity(struct vdpa_device *vdpa,
> + struct irq_affinity *desc)
> +{
> + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> + struct irq_affinity_desc *affd = NULL;
> + int i;
> +
> + affd = irq_create_affinity_masks(dev->vq_num, desc);
> + if (!affd)
Let's add a comment on the vdpa config ops to say set_irq_affinity()
is best effort.
Thanks
> + return;
> +
> + for (i = 0; i < dev->vq_num; i++) {
> + cpumask_copy(&dev->vqs[i]->irq_affinity, &affd[i].mask);
> + vduse_vq_update_effective_cpu(dev->vqs[i]);
> + }
> + kfree(affd);
> +}
> +
> static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
> unsigned int asid,
> struct vhost_iotlb *iotlb)
> @@ -760,6 +807,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
> .get_config = vduse_vdpa_get_config,
> .set_config = vduse_vdpa_set_config,
> .get_generation = vduse_vdpa_get_generation,
> + .set_irq_affinity = vduse_vdpa_set_irq_affinity,
> .reset = vduse_vdpa_reset,
> .set_map = vduse_vdpa_set_map,
> .free = vduse_vdpa_free,
> @@ -1380,6 +1428,8 @@ static int vduse_dev_init_vqs(struct vduse_dev *dev, u32 vq_align, u32 vq_num)
> INIT_WORK(&dev->vqs[i]->kick, vduse_vq_kick_work);
> spin_lock_init(&dev->vqs[i]->kick_lock);
> spin_lock_init(&dev->vqs[i]->irq_lock);
> + spin_lock_init(&dev->vqs[i]->irq_affinity_lock);
> + cpumask_setall(&dev->vqs[i]->irq_affinity);
> }
>
> return 0;
> --
> 2.20.1
>
next prev parent reply other threads:[~2022-12-16 5:31 UTC|newest]
Thread overview: 51+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-12-05 8:41 [PATCH v2 00/11] VDUSE: Improve performance Xie Yongji
2022-12-05 8:41 ` [PATCH v2 01/11] genirq/affinity:: Export irq_create_affinity_masks() Xie Yongji
2022-12-06 8:18 ` Christoph Hellwig
2022-12-06 8:40 ` Yongji Xie
2022-12-06 8:47 ` Christoph Hellwig
2022-12-06 9:28 ` Yongji Xie
2022-12-07 5:56 ` Jason Wang
2022-12-19 7:33 ` Michael S. Tsirkin
2022-12-19 9:36 ` Yongji Xie
2023-01-27 8:22 ` Michael S. Tsirkin
2023-01-30 11:53 ` Yongji Xie
2023-02-13 11:59 ` Michael S. Tsirkin
2023-02-13 14:50 ` Yongji Xie
2023-02-13 18:53 ` Thomas Gleixner
2023-02-14 2:38 ` Yongji Xie
2022-12-05 8:41 ` [PATCH v2 02/11] vdpa: Add set/get_vq_affinity callbacks in vdpa_config_ops Xie Yongji
2022-12-16 3:52 ` Jason Wang
2022-12-05 8:41 ` [PATCH v2 03/11] vdpa: Add set_irq_affinity callback " Xie Yongji
2022-12-16 3:58 ` Jason Wang
2022-12-19 4:38 ` Yongji Xie
2022-12-19 6:06 ` Jason Wang
2022-12-19 7:12 ` Yongji Xie
2022-12-20 6:31 ` Jason Wang
2022-12-20 10:14 ` Yongji Xie
2022-12-21 3:20 ` Jason Wang
2022-12-05 8:41 ` [PATCH v2 04/11] vduse: Refactor allocation for vduse virtqueues Xie Yongji
2022-12-16 3:59 ` Jason Wang
2022-12-05 8:41 ` [PATCH v2 05/11] vduse: Introduce bound workqueue for irq injection Xie Yongji
2022-12-16 4:02 ` Jason Wang
2022-12-19 5:04 ` Yongji Xie
2022-12-20 6:27 ` Jason Wang
2022-12-20 10:01 ` Yongji Xie
2022-12-21 3:19 ` Jason Wang
2022-12-05 8:58 ` [PATCH v2 06/11] vduse: Support automatic irq callback affinity Xie Yongji
2022-12-16 5:30 ` Jason Wang [this message]
2022-12-19 4:56 ` Yongji Xie
2022-12-20 6:32 ` Jason Wang
2022-12-20 8:21 ` Yongji Xie
2022-12-05 9:02 ` [PATCH v2 07/11] vduse: Support set/get_vq_affinity callbacks Xie Yongji
2022-12-05 9:02 ` [PATCH v2 08/11] vduse: Add sysfs interface for irq callback affinity Xie Yongji
2022-12-16 5:35 ` Jason Wang
2022-12-19 5:16 ` Yongji Xie
2022-12-20 6:29 ` Jason Wang
2022-12-20 8:24 ` Yongji Xie
2022-12-05 9:02 ` [PATCH v2 09/11] vduse: Add enable_irq_wq sysfs interface for virtqueues Xie Yongji
2022-12-16 5:43 ` Jason Wang
2022-12-19 6:39 ` Yongji Xie
2022-12-05 9:02 ` [PATCH v2 10/11] vduse: Delay iova domain creation Xie Yongji
2022-12-16 5:49 ` Jason Wang
2022-12-05 9:02 ` [PATCH v2 11/11] vduse: Support specifying bounce buffer size via sysfs Xie Yongji
2022-12-16 6:02 ` Jason Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CACGkMEuhYO3neFmxwiBp8C0QTaa+Mb13kken+RZ9QuruMct6tA@mail.gmail.com \
--to=jasowang@redhat.com \
--cc=hch@lst.de \
--cc=linux-kernel@vger.kernel.org \
--cc=mst@redhat.com \
--cc=tglx@linutronix.de \
--cc=virtualization@lists.linux-foundation.org \
--cc=xieyongji@bytedance.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).