From: "Michael S. Tsirkin" <mst@redhat.com> To: Jason Wang <jasowang@redhat.com> Cc: Halil Pasic <pasic@linux.ibm.com>, Cornelia Huck <cohuck@redhat.com>, linux-kernel <linux-kernel@vger.kernel.org>, virtualization <virtualization@lists.linux-foundation.org>, Stefano Garzarella <sgarzare@redhat.com>, eperezma <eperezma@redhat.com>, Cindy Lu <lulu@redhat.com>, Thomas Gleixner <tglx@linutronix.de>, Peter Zijlstra <peterz@infradead.org>, "Paul E. McKenney" <paulmck@kernel.org>, Marc Zyngier <maz@kernel.org> Subject: Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs() Date: Wed, 27 Apr 2022 11:15:33 -0400 [thread overview] Message-ID: <20220427111228-mutt-send-email-mst@kernel.org> (raw) In-Reply-To: <CACGkMEvfWA9vzVWU6yVdGNwueAAR9MhW8sFAAnGaRB7T+3GYeQ@mail.gmail.com> On Wed, Apr 27, 2022 at 03:57:57PM +0800, Jason Wang wrote: > On Wed, Apr 27, 2022 at 2:30 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Wed, Apr 27, 2022 at 11:53:25AM +0800, Jason Wang wrote: > > > On Tue, Apr 26, 2022 at 2:30 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > On Tue, Apr 26, 2022 at 12:07:39PM +0800, Jason Wang wrote: > > > > > On Tue, Apr 26, 2022 at 11:55 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > > > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote: > > > > > > > On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote: > > > > > > > > > > > > > > > > 在 2022/4/26 11:38, Michael S. Tsirkin 写道: > > > > > > > > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote: > > > > > > > > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote: > > > > > > > > > > > On Mon, 25 Apr 2022 09:59:55 -0400 > > > > > > > > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote: > > > > > > > > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote: > > > > > > > > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote: > > > > > > > > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the > > > > > > > > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the > > > > > > > > > > > > > > > synchronization is simply done via the airq_info's lock. For the > > > > > > > > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per > > > > > > > > > > > > > > > device spinlock for irq is introduced ans used in the synchronization > > > > > > > > > > > > > > > method. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de> > > > > > > > > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org> > > > > > > > > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org> > > > > > > > > > > > > > > > Cc: Marc Zyngier <maz@kernel.org> > > > > > > > > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com> > > > > > > > > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com> > > > > > > > > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com> > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is the only one that is giving me pause. Halil, Cornelia, > > > > > > > > > > > > > > should we be concerned about the performance impact here? > > > > > > > > > > > > > > Any chance it can be tested? > > > > > > > > > > > > > We can have a bunch of devices using the same airq structure, and the > > > > > > > > > > > > > sync cb creates a choke point, same as registering/unregistering. > > > > > > > > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment? > > > > > > > > > > > I'm not sure I understand the question. > > > > > > > > > > > > > > > > > > > > > > I do think we can have multiple CPUs that are executing some portion of > > > > > > > > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think? > > > > > > > > > > > > > > > > > > > > > > On the other hand we could also end up serializing synchronize_cbs() > > > > > > > > > > > calls for different devices if they happen to use the same airq_info. But > > > > > > > > > > > this probably was not your question > > > > > > > > > > > > > > > > > > > > I am less concerned about synchronize_cbs being slow and more about > > > > > > > > > > the slowdown in interrupt processing itself. > > > > > > > > > > > > > > > > > > > > > > this patch serializes them on a spinlock. > > > > > > > > > > > > > > > > > > > > > > > Those could then pile up on the newly introduced spinlock. > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > Halil > > > > > > > > > > Hmm yea ... not good. > > > > > > > > > Is there any other way to synchronize with all callbacks? > > > > > > > > > > > > > > > > > > > > > > > > Maybe using rwlock as airq handler? > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > rwlock is still a shared cacheline bouncing between CPUs and > > > > > > > a bunch of ordering instructions. > > > > > > > > > > Yes, but it should be faster than spinlocks anyhow. > > > > > > > > > > > > Maybe something per-cpu + some IPIs to run things on all CPUs instead? > > > > > > > > > > Is this something like a customized version of synchronzie_rcu_expedited()? > > > > > > > > With interrupts running in an RCU read size critical section? > > > > > > For vring_interrupt(), yes. > > > > > > > > > > Quite possibly that is also an option. > > > > This will need a bunch of documentation since this is not > > > > a standard use of RCU, > > > > > > According to Documentation/RCU/requirements.rst, it looks like a legal case: > > > > > > " > > > The Linux kernel has interrupts, and RCU read-side critical sections are > > > legal within interrupt handlers and within interrupt-disabled regions of > > > code, as are invocations of call_rcu(). > > > " > > > > My problem is it is not clear what data is protected by rcu here. > > Nothing is tagged with __rcu or uses rcu_dereference. > > It should be the vq->broken here. We can rcuify it. I don't exactly exactly see how - rcu_dereference works on pointers. Feel free to send a patch. > > > We need at least an ack from rcu maintainers that this is > > a valid use of rcu and not an undocumented side effect. > > Yes. > > Thanks > > > > > > > And as discussed, synchronize_rcu_expedited() is not friendly to real > > > time workload. I think we can simply > > > > > > 1) protect vring_interrupt() with rcu_read_lock() > > > 2) use synchronize_rcu() in synchronize_cbs for ccw > > > > > > And if we care about the long delay we can use per device srcu to reduce that? > > > > > > Thanks > > > > > > > and probably get a confirmation > > > > from RCU maintainers that whatever assumptions we make > > > > are guaranteed to hold down the road. > > > > > > > > > > > > > > > > ... and I think classic and device interrupts are different enough > > > > > > here ... > > > > > > > > > > Yes. > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > MST > > > > > > > > > > > >
WARNING: multiple messages have this Message-ID (diff)
From: "Michael S. Tsirkin" <mst@redhat.com> To: Jason Wang <jasowang@redhat.com> Cc: "Paul E. McKenney" <paulmck@kernel.org>, Cindy Lu <lulu@redhat.com>, Peter Zijlstra <peterz@infradead.org>, Marc Zyngier <maz@kernel.org>, Cornelia Huck <cohuck@redhat.com>, linux-kernel <linux-kernel@vger.kernel.org>, virtualization <virtualization@lists.linux-foundation.org>, Halil Pasic <pasic@linux.ibm.com>, eperezma <eperezma@redhat.com>, Thomas Gleixner <tglx@linutronix.de> Subject: Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs() Date: Wed, 27 Apr 2022 11:15:33 -0400 [thread overview] Message-ID: <20220427111228-mutt-send-email-mst@kernel.org> (raw) In-Reply-To: <CACGkMEvfWA9vzVWU6yVdGNwueAAR9MhW8sFAAnGaRB7T+3GYeQ@mail.gmail.com> On Wed, Apr 27, 2022 at 03:57:57PM +0800, Jason Wang wrote: > On Wed, Apr 27, 2022 at 2:30 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Wed, Apr 27, 2022 at 11:53:25AM +0800, Jason Wang wrote: > > > On Tue, Apr 26, 2022 at 2:30 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > On Tue, Apr 26, 2022 at 12:07:39PM +0800, Jason Wang wrote: > > > > > On Tue, Apr 26, 2022 at 11:55 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > > > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote: > > > > > > > On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote: > > > > > > > > > > > > > > > > 在 2022/4/26 11:38, Michael S. Tsirkin 写道: > > > > > > > > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote: > > > > > > > > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote: > > > > > > > > > > > On Mon, 25 Apr 2022 09:59:55 -0400 > > > > > > > > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote: > > > > > > > > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote: > > > > > > > > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote: > > > > > > > > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the > > > > > > > > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the > > > > > > > > > > > > > > > synchronization is simply done via the airq_info's lock. For the > > > > > > > > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per > > > > > > > > > > > > > > > device spinlock for irq is introduced ans used in the synchronization > > > > > > > > > > > > > > > method. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de> > > > > > > > > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org> > > > > > > > > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org> > > > > > > > > > > > > > > > Cc: Marc Zyngier <maz@kernel.org> > > > > > > > > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com> > > > > > > > > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com> > > > > > > > > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com> > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is the only one that is giving me pause. Halil, Cornelia, > > > > > > > > > > > > > > should we be concerned about the performance impact here? > > > > > > > > > > > > > > Any chance it can be tested? > > > > > > > > > > > > > We can have a bunch of devices using the same airq structure, and the > > > > > > > > > > > > > sync cb creates a choke point, same as registering/unregistering. > > > > > > > > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment? > > > > > > > > > > > I'm not sure I understand the question. > > > > > > > > > > > > > > > > > > > > > > I do think we can have multiple CPUs that are executing some portion of > > > > > > > > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think? > > > > > > > > > > > > > > > > > > > > > > On the other hand we could also end up serializing synchronize_cbs() > > > > > > > > > > > calls for different devices if they happen to use the same airq_info. But > > > > > > > > > > > this probably was not your question > > > > > > > > > > > > > > > > > > > > I am less concerned about synchronize_cbs being slow and more about > > > > > > > > > > the slowdown in interrupt processing itself. > > > > > > > > > > > > > > > > > > > > > > this patch serializes them on a spinlock. > > > > > > > > > > > > > > > > > > > > > > > Those could then pile up on the newly introduced spinlock. > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > Halil > > > > > > > > > > Hmm yea ... not good. > > > > > > > > > Is there any other way to synchronize with all callbacks? > > > > > > > > > > > > > > > > > > > > > > > > Maybe using rwlock as airq handler? > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > rwlock is still a shared cacheline bouncing between CPUs and > > > > > > > a bunch of ordering instructions. > > > > > > > > > > Yes, but it should be faster than spinlocks anyhow. > > > > > > > > > > > > Maybe something per-cpu + some IPIs to run things on all CPUs instead? > > > > > > > > > > Is this something like a customized version of synchronzie_rcu_expedited()? > > > > > > > > With interrupts running in an RCU read size critical section? > > > > > > For vring_interrupt(), yes. > > > > > > > > > > Quite possibly that is also an option. > > > > This will need a bunch of documentation since this is not > > > > a standard use of RCU, > > > > > > According to Documentation/RCU/requirements.rst, it looks like a legal case: > > > > > > " > > > The Linux kernel has interrupts, and RCU read-side critical sections are > > > legal within interrupt handlers and within interrupt-disabled regions of > > > code, as are invocations of call_rcu(). > > > " > > > > My problem is it is not clear what data is protected by rcu here. > > Nothing is tagged with __rcu or uses rcu_dereference. > > It should be the vq->broken here. We can rcuify it. I don't exactly exactly see how - rcu_dereference works on pointers. Feel free to send a patch. > > > We need at least an ack from rcu maintainers that this is > > a valid use of rcu and not an undocumented side effect. > > Yes. > > Thanks > > > > > > > And as discussed, synchronize_rcu_expedited() is not friendly to real > > > time workload. I think we can simply > > > > > > 1) protect vring_interrupt() with rcu_read_lock() > > > 2) use synchronize_rcu() in synchronize_cbs for ccw > > > > > > And if we care about the long delay we can use per device srcu to reduce that? > > > > > > Thanks > > > > > > > and probably get a confirmation > > > > from RCU maintainers that whatever assumptions we make > > > > are guaranteed to hold down the road. > > > > > > > > > > > > > > > > ... and I think classic and device interrupts are different enough > > > > > > here ... > > > > > > > > > > Yes. > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > MST > > > > > > > > > > > > _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
next prev parent reply other threads:[~2022-04-27 15:16 UTC|newest] Thread overview: 103+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-04-25 2:44 [PATCH V3 0/9] rework on the IRQ hardening of virtio Jason Wang 2022-04-25 2:44 ` Jason Wang 2022-04-25 2:44 ` [PATCH V3 1/9] virtio: use virtio_device_ready() in virtio_device_restore() Jason Wang 2022-04-25 2:44 ` Jason Wang 2022-04-28 9:07 ` Cornelia Huck 2022-04-28 9:07 ` Cornelia Huck 2022-04-25 2:44 ` [PATCH V3 2/9] virtio: use virtio_reset_device() when possible Jason Wang 2022-04-25 2:44 ` Jason Wang 2022-04-28 9:07 ` Cornelia Huck 2022-04-28 9:07 ` Cornelia Huck 2022-04-25 2:44 ` [PATCH V3 3/9] virtio: introduce config op to synchronize vring callbacks Jason Wang 2022-04-25 2:44 ` Jason Wang 2022-04-28 9:13 ` Cornelia Huck 2022-04-28 9:13 ` Cornelia Huck 2022-04-29 2:08 ` Jason Wang 2022-04-29 2:08 ` Jason Wang 2022-04-29 7:25 ` Cornelia Huck 2022-04-29 7:25 ` Cornelia Huck 2022-05-05 8:15 ` Jason Wang 2022-05-05 8:15 ` Jason Wang 2022-04-25 2:44 ` [PATCH V3 4/9] virtio-pci: implement synchronize_cbs() Jason Wang 2022-04-25 2:44 ` Jason Wang 2022-04-25 2:44 ` [PATCH V3 5/9] virtio-mmio: " Jason Wang 2022-04-25 2:44 ` Jason Wang 2022-04-25 2:44 ` [PATCH V3 6/9] virtio-ccw: " Jason Wang 2022-04-25 2:44 ` Jason Wang 2022-04-25 8:08 ` Michael S. Tsirkin 2022-04-25 8:08 ` Michael S. Tsirkin 2022-04-25 8:54 ` Cornelia Huck 2022-04-25 8:54 ` Cornelia Huck 2022-04-25 13:59 ` Michael S. Tsirkin 2022-04-25 13:59 ` Michael S. Tsirkin 2022-04-26 2:29 ` Halil Pasic 2022-04-26 2:29 ` Halil Pasic 2022-04-26 3:35 ` Michael S. Tsirkin 2022-04-26 3:35 ` Michael S. Tsirkin 2022-04-26 3:38 ` Michael S. Tsirkin 2022-04-26 3:38 ` Michael S. Tsirkin 2022-04-26 3:42 ` Jason Wang 2022-04-26 3:42 ` Jason Wang 2022-04-26 3:53 ` Michael S. Tsirkin 2022-04-26 3:53 ` Michael S. Tsirkin 2022-04-26 3:55 ` Michael S. Tsirkin 2022-04-26 3:55 ` Michael S. Tsirkin 2022-04-26 4:07 ` Jason Wang 2022-04-26 4:07 ` Jason Wang 2022-04-26 6:30 ` Michael S. Tsirkin 2022-04-26 6:30 ` Michael S. Tsirkin 2022-04-27 3:53 ` Jason Wang 2022-04-27 3:53 ` Jason Wang 2022-04-27 6:28 ` Michael S. Tsirkin 2022-04-27 6:28 ` Michael S. Tsirkin 2022-04-27 6:30 ` Michael S. Tsirkin 2022-04-27 6:30 ` Michael S. Tsirkin 2022-04-27 7:57 ` Jason Wang 2022-04-27 7:57 ` Jason Wang 2022-04-27 15:15 ` Michael S. Tsirkin [this message] 2022-04-27 15:15 ` Michael S. Tsirkin 2022-04-26 15:47 ` Cornelia Huck 2022-04-26 15:47 ` Cornelia Huck 2022-04-26 16:43 ` Michael S. Tsirkin 2022-04-26 16:43 ` Michael S. Tsirkin 2022-04-27 9:27 ` Cornelia Huck 2022-04-27 9:27 ` Cornelia Huck 2022-04-28 2:43 ` Halil Pasic 2022-04-28 2:43 ` Halil Pasic 2022-04-28 3:04 ` Jason Wang 2022-04-28 3:04 ` Jason Wang 2022-04-28 5:24 ` Michael S. Tsirkin 2022-04-28 5:24 ` Michael S. Tsirkin 2022-04-28 5:51 ` Jason Wang 2022-04-28 5:51 ` Jason Wang 2022-04-28 5:55 ` Michael S. Tsirkin 2022-04-28 5:55 ` Michael S. Tsirkin 2022-04-28 6:02 ` Jason Wang 2022-04-28 6:02 ` Jason Wang 2022-04-28 6:17 ` Michael S. Tsirkin 2022-04-28 6:17 ` Michael S. Tsirkin 2022-04-28 7:42 ` Cornelia Huck 2022-04-28 7:42 ` Cornelia Huck 2022-04-29 2:02 ` Jason Wang 2022-04-29 2:02 ` Jason Wang 2022-04-28 4:12 ` Michael S. Tsirkin 2022-04-28 4:12 ` Michael S. Tsirkin 2022-04-26 2:50 ` Halil Pasic 2022-04-26 2:50 ` Halil Pasic 2022-04-26 3:46 ` Jason Wang 2022-04-26 3:46 ` Jason Wang [not found] ` <20220426053856.3325-1-hdanton@sina.com> 2022-04-26 5:59 ` Jason Wang 2022-04-25 2:44 ` [PATCH V3 7/9] virtio: allow to unbreak virtqueue Jason Wang 2022-04-25 2:44 ` Jason Wang 2022-04-25 12:44 ` Cornelia Huck 2022-04-25 12:44 ` Cornelia Huck 2022-04-25 14:04 ` Michael S. Tsirkin 2022-04-25 14:04 ` Michael S. Tsirkin 2022-04-26 3:45 ` Jason Wang 2022-04-26 3:45 ` Jason Wang 2022-04-25 2:44 ` [PATCH V3 8/9] virtio: harden vring IRQ Jason Wang 2022-04-25 2:44 ` Jason Wang 2022-04-25 2:44 ` [PATCH V3 9/9] virtio: use WARN_ON() to warning illegal status value Jason Wang 2022-04-25 2:44 ` Jason Wang 2022-04-25 2:49 ` [PATCH V3 0/9] rework on the IRQ hardening of virtio Jason Wang 2022-04-25 2:49 ` Jason Wang
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20220427111228-mutt-send-email-mst@kernel.org \ --to=mst@redhat.com \ --cc=cohuck@redhat.com \ --cc=eperezma@redhat.com \ --cc=jasowang@redhat.com \ --cc=linux-kernel@vger.kernel.org \ --cc=lulu@redhat.com \ --cc=maz@kernel.org \ --cc=pasic@linux.ibm.com \ --cc=paulmck@kernel.org \ --cc=peterz@infradead.org \ --cc=sgarzare@redhat.com \ --cc=tglx@linutronix.de \ --cc=virtualization@lists.linux-foundation.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.