From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: cdupontd@redhat.com From: Christophe de Dinechin Message-ID: <55EF6822-2B1A-4564-ADF6-D06244DED0D8@redhat.com> Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: [virtio-dev] On doorbells (queue notifications) Date: Thu, 16 Jul 2020 16:31:47 +0200 In-Reply-To: <20200716141930.GA114428@stefanha-x1.localdomain> References: <87r1tdydpz.fsf@linaro.org> <20200715114855.GF18817@stefanha-x1.localdomain> <877dv4ykin.fsf@linaro.org> <20200715154732.GC47883@stefanha-x1.localdomain> <871rlcybni.fsf@linaro.org> <20200716100051.GC85868@stefanha-x1.localdomain> <20200716141930.GA114428@stefanha-x1.localdomain> Content-Type: multipart/alternative; boundary="Apple-Mail=_F0D894EB-17FA-4632-8E34-AA339BFC305F" To: Stefan Hajnoczi Cc: =?utf-8?Q?Alex_Benn=C3=A9e?= , virtio-dev@lists.oasis-open.org, Zha Bin , Jing Liu , Chao Peng , Cornelia Huck , Jan Kiszka , "Michael S. Tsirkin" List-ID: --Apple-Mail=_F0D894EB-17FA-4632-8E34-AA339BFC305F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On 16 Jul 2020, at 16:19, Stefan Hajnoczi wrote: >=20 > On Thu, Jul 16, 2020 at 01:25:37PM +0200, Christophe de Dinechin wrote: >>=20 >>=20 >>> On 16 Jul 2020, at 12:00, Stefan Hajnoczi wrote: >>>=20 >>> On Wed, Jul 15, 2020 at 05:40:33PM +0100, Alex Benn=C3=A9e wrote: >>>>=20 >>>> Stefan Hajnoczi writes: >>>>=20 >>>>> On Wed, Jul 15, 2020 at 02:29:04PM +0100, Alex Benn=C3=A9e wrote: >>>>>> Stefan Hajnoczi writes: >>>>>>> On Tue, Jul 14, 2020 at 10:43:36PM +0100, Alex Benn=C3=A9e wrote: >>>>>>>> Finally I'm curious if this is just a problem avoided by the s390 >>>>>>>> channel approach? Does the use of messages over a channel just avo= id the >>>>>>>> sort of bouncing back and forth that other hypervisors have to do = when >>>>>>>> emulating a device? >>>>>>>=20 >>>>>>> What does "bouncing back and forth" mean exactly? >>>>>>=20 >>>>>> Context switching between guest and hypervisor. >>>>>=20 >>>>> I have CCed Cornelia Huck, who can explain the lifecycle of an I/O >>>>> request on s390 channel I/O. >>>>=20 >>>> Thanks. >>>>=20 >>>> I was also wondering about the efficiency of doorbells/notifications t= he >>>> other way. AFAIUI for both PCI and MMIO only a single write is require= d >>>> to the notify flag which causes a trap to the hypervisor and the rest = of >>>> the processing. The hypervisor doesn't have the cost multiple exits to >>>> read the guest state although it obviously wants to be as efficient as >>>> possible passing the data back up to what ever is handling the backend >>>> of the device so it doesn't need to do multiple context switches. >>>>=20 >>>> Has there been any investigation into other mechanisms for notifying t= he >>>> hypervisor of an event - for example using a HYP call or similar >>>> mechanism? >>>>=20 >>>> My gut tells me this probably doesn't make any difference as a trap to >>>> the hypervisor is likely to cost the same either way because you still >>>> need to save the guest context before actioning something but it would >>>> be interesting to know if anyone has looked at it. Perhaps there is a >>>> benefit in partitioned systems where core running the guest can return >>>> straight away after initiating what it needs to internally in the >>>> hypervisor to pass the notification to something that can deal with it= ? >>>=20 >>> It's very architecture-specific. This is something Michael Tsirkin >>> looked in in the past. He found that MMIO and PIO perform differently o= n >>> x86. VIRTIO supports both so the device can be configured optimally. >>> There was an old discussion from 2013 here: >>> https://lkml.org/lkml/2013/4/4/299 = > >>>=20 >>> Without nested page tables MMIO was slower than PIO. But with nested >>> page tables it was faster. >>>=20 >>> Another option on x86 is using Model-Specific Registers (for hypercalls= ) >>> but this doesn't fit into the PCI device model. >>=20 >> (Warning: What I write below is based on experience with very different >> architectures, both CPU and hypervisor; your mileage may vary) >>=20 >> It looks to me like the discussion so far is mostly focused on a "synchr= onous" >> model where presumably the same CPU is switching context between >> guest and (host) device emulation. >>=20 >> However, I/O devices on real hardware are asynchronous by construction. >> They do their thing while the CPU processes stuff. So at least theoretic= ally, >> there is no reason to context switch on the same CPU. You could very wel= l >> have an I/O thread on some other CPU doing its thing. This allows to >> do something some of you may have heard me talk about, called >> "interrupt coalescing". >>=20 >> As Stefan noted, this is not always a win, as it may introduce latency. >> There are at least two cases where this latency really hurts: >>=20 >> 1. When the I/O thread is in some kind of deep sleep, e.g. because it >> was not active recently. Everything from cache to TLB may hit you here, >> but that normally happens when there isn't much I/O activity, so this ca= se >> in practice does not hurt that much, or rather it hurts in a case where >> don't really care. >>=20 >> 2. When the I/O thread is preempted, or not given enough cycles to do it= s >> stuff. This happens when the system is both CPU and I/O bound, and >> addressing that is mostly a scheduling issue. A CPU thread could hand-of= f >> to a specific I/O thread, reducing that case to the kind of context swit= ch >> Alex was mentioning, but I'm not sure how feasible it is to implement >> that on Linux / kvm. >>=20 >> In such cases, you have to pay for context switch. I'm not sure if that >> context switch is markedly more expensive than a "vmexit". On at least >> that alien architecture I was familiar with, there was little difference= between >> switching to "your" host CPU thread and switching to "another" host >> I/O thread. But then the context switch was all in software, so we had >> designed it that way. >>=20 >> So let's assume now that you run your device emulation fully in an I/O >> thread, which we will assume for simplicity sits mostly in host user-spa= ce, >> and your guest I/O code runs in a CPU thread, which we will assume >> sits mostly in guest user/kernel space. >>=20 >> It is possible to share two-way doorbells / IRQ queues on some memory >> page, very similar to a virtqueue. When you want to "doorbell" your devi= ce, >> you simply write to that page. The device threads picks it up by reading >> the same page, and posts I/O completions on the same page, with simple >> memory writes. >>=20 >> Consider this I/O exchange buffer as having (at least) a writer and read= er >> index for both doorbells and virtual interrupts. In the explanation >> below, I will call them "dwi", "dri", "iwi", "iri" for doorbell / interr= upt read >> and write index. (Note that as a key optimization, you really >> don't want dwi and dri to be in the same cache line, since different >> CPUs are going to read and write them) >>=20 >> You obviously still need to "kick" the I/O or CPU thread, and we are >> talking about an IPI here since you don't know which CPU that other >> thread is sitting on. But the interesting property is that you only need >> to do that when dwi=3D=3Ddri or iwi=3D=3Diri, because if not, the other = side >> has already been "kicked" and will keep working, i.e. incrementing >> dri or iri, until it reaches back that state. >>=20 >> The real "interrupt coalescing" trick can happen here. In some >> cases, you can decide to update your dwi or iwi without kicking, >> as long as you know that you will need to kick later. That requires >> some heavy cooperation from guest drivers, though, and is a >> second-order optimization. >>=20 >> With a scheme like this, you replace a systematic context switch >> for each device interrupt with a memory write and a "fire and forget" >> kick IPI that only happens when the system is not already busy >> processing I/Os, so that it can be eliminated when the system is >> most busy. With interrupt coalescing, you can send IPIs at a rate >> much lower than the actual I/O rate. >>=20 >> Not sure how difficult it is to adapt a scheme like this to the current >> state of qemu / kvm, but I'm pretty sure it works well if you implement >> it correctly ;-) >>=20 >>>=20 >>> A bigger issue than vmexit latency is device emulation thread wakeup >>> latency. There is a thread (QEMU, vhost-user, vhost, etc) monitoring th= e >>> ioeventfd but it may be descheduled. Its physical CPU may be in a low >>> power state. I ran a benchmark late last year with QEMU's AioContext >>> adaptive polling disabled so we can measure the wakeup latency: >>>=20 >>> CPU 0/KVM 26102 [000] 85626.737072: kvm:kvm_fast_mmio: >>> fast mmio at gpa 0xfde03000 >>> IO iothread1 26099 [001] 85626.737076: syscalls:sys_exit_ppoll: 0x1 >>> 4 microseconds ------^ >=20 > Hi Christophe, > QEMU/KVM does something similar to what you described. In the perf > output above the vmexit kvm_fast_mmio event occurs on physical CPU > "[000]". The IOThread wakes up on physical CPU "[001]". Physical CPU#0 > resumes guest execution immediately after marking the ioeventfd ready. > There is no context switch to the IOThread or a return from > ioctl(KVM_RUN) on CPU#0. Oh, that's good. But then the conclusion that the 4us delay limits us to 250kIOPS is incorrect, no? Is there anything that would prevent multiple I/O events (doorbell or interrupt) to be in flight at the same time? >=20 > The IOThread reads the eventfd. An eventfd is a counter that is reset to > 0 on read. Because it's a counter you get coalescing: if the guest > performs multiple MMIO writes the ioeventfd counter increases but the > IOThread only wakes up once and reads the ioeventfd. >=20 > VIRTIO itself also has a mechanism for suppressing notifications called > EVENT_IDX. It allows the driver to let the device know that it does not > require interrupts, and the device to let the driver know it does not > require virtqueue kicks. This reminds me a bit of the mitigation > mechanism you described. >=20 > Stefan --Apple-Mail=_F0D894EB-17FA-4632-8E34-AA339BFC305F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On 16 Jul 202= 0, at 16:19, Stefan Hajnoczi <stefanha@redhat.com> wrote:

On Thu, Jul 16, 2020 at 01:25:37PM +020= 0, Christophe de Dinechin wrote:


On 16 Jul 2020, at 12:00, Stefan Hajnoczi <stefanha@redhat.com> wrote:

On Wed, Jul 15, 2020 at 05:40:33PM +0100, Alex Ben= n=C3=A9e wrote:

Stefan Hajnoczi <stefanha@redhat.com> writes:

On Wed, Jul 15, 2020 at 02:29:04PM +0100, Alex= Benn=C3=A9e wrote:
Stef= an Hajnoczi <stefanha@= redhat.com> writes:
On Tue, Jul 14, 2020 at 10:43:36PM +0100, Alex Benn=C3=A9e wrote:
Finally I'm curious if this is= just a problem avoided by the s390
channel approach? Does th= e use of messages over a channel just avoid the
sort of bounc= ing back and forth that other hypervisors have to do when
emu= lating a device?

What does "bounc= ing back and forth" mean exactly?

Context switching between guest and hypervisor.

I have CCed Cornelia Huck, who can explain the lifecycle of= an I/O
request on s390 channel I/O.

Thanks.

I was also wondering= about the efficiency of doorbells/notifications the
other wa= y. AFAIUI for both PCI and MMIO only a single write is required
to the notify flag which causes a trap to the hypervisor and the rest of=
the processing. The hypervisor doesn't have the cost multipl= e exits to
read the guest state although it obviously wants t= o be as efficient as
possible passing the data back up to wha= t ever is handling the backend
of the device so it doesn't ne= ed to do multiple context switches.

Has there = been any investigation into other mechanisms for notifying the
hypervisor of an event - for example using a HYP call or similar
mechanism?

My gut tells me this probably= doesn't make any difference as a trap to
the hypervisor is l= ikely to cost the same either way because you still
need to s= ave the guest context before actioning something but it would
be interesting to know if anyone has looked at it. Perhaps there is a
benefit in partitioned systems where core running the guest can r= eturn
straight away after initiating what it needs to interna= lly in the
hypervisor to pass the notification to something t= hat can deal with it?

It's very a= rchitecture-specific. This is something Michael Tsirkin
looke= d in in the past. He found that MMIO and PIO perform differently on
x86. VIRTIO supports both so the device can be configured optimally.=
There was an old discussion from 2013 here:
https://lkml.org/lk= ml/2013/4/4/299 <<= a href=3D"https://lkml.org/lkml/2013/4/4/299" class=3D"">https://lkml.org/l= kml/2013/4/4/299>

Without nested page t= ables MMIO was slower than PIO. But with nested
page tables i= t was faster.

Another option on x86 is using M= odel-Specific Registers (for hypercalls)
but this doesn't fit= into the PCI device model.

(Warn= ing: What I write below is based on experience with very different
architectures, both CPU and hypervisor; your mileage may vary)

It looks to me like the discussion so far is mostly = focused on a "synchronous"
model where presumably the same CP= U is switching context between
guest and (host) device emulat= ion.

However, I/O devices on real hardware are= asynchronous by construction.
They do their thing while the = CPU processes stuff. So at least theoretically,
there is no r= eason to context switch on the same CPU. You could very well
= have an I/O thread on some other CPU doing its thing. This allows to
do something some of you may have heard me talk about, called
"interrupt coalescing".

As Stefan not= ed, this is not always a win, as it may introduce latency.
Th= ere are at least two cases where this latency really hurts:
<= br class=3D"">1. When the I/O thread is in some kind of deep sleep, e.g. be= cause it
was not active recently. Everything from cache to TL= B may hit you here,
but that normally happens when there isn'= t much I/O activity, so this case
in practice does not hurt t= hat much, or rather it hurts in a case where
don't really car= e.

2. When the I/O thread is preempted, or not= given enough cycles to do its
stuff. This happens when the s= ystem is both CPU and I/O bound, and
addressing that is mostl= y a scheduling issue. A CPU thread could hand-off
to a specif= ic I/O thread, reducing that case to the kind of context switch
Alex was mentioning, but I'm not sure how feasible it is to implementthat on Linux / kvm.

In such cases= , you have to pay for context switch. I'm not sure if that
co= ntext switch is markedly more expensive than a "vmexit". On at least
that alien architecture I was familiar with, there was little diffe= rence between
switching to "your" host CPU thread and switchi= ng to "another" host
I/O thread. But then the context switch = was all in software, so we had
designed it that way.

So let's assume now that you run your device emulation= fully in an I/O
thread, which we will assume for simplicity = sits mostly in host user-space,
and your guest I/O code runs = in a CPU thread, which we will assume
sits mostly in guest us= er/kernel space.

It is possible to share two-w= ay doorbells / IRQ queues on some memory
page, very similar t= o a virtqueue. When you want to "doorbell" your device,
you s= imply write to that page. The device threads picks it up by reading
the same page, and posts I/O completions on the same page, with simp= le
memory writes.

Consider this = I/O exchange buffer as having (at least) a writer and reader
= index for both doorbells and virtual interrupts. In the explanation
below, I will call them "dwi", "dri", "iwi", "iri" for doorbell / in= terrupt read
and write index. (Note that as a key optimizatio= n, you really
don't want dwi and dri to be in the same cache = line, since different
CPUs are going to read and write them)<= br class=3D"">
You obviously still need to "kick" the I/O or = CPU thread, and we are
talking about an IPI here since you do= n't know which CPU that other
thread is sitting on. But the i= nteresting property is that you only need
to do that when dwi= =3D=3Ddri or iwi=3D=3Diri, because if not, the other side
has= already been "kicked" and will keep working, i.e. incrementing
dri or iri, until it reaches back that state.

The real "interrupt coalescing" trick can happen here. In some
cases, you can decide to update your dwi or iwi without kicking,
as long as you know that you will need to kick later. That requir= es
some heavy cooperation from guest drivers, though, and is = a
second-order optimization.

Wit= h a scheme like this, you replace a systematic context switch
for each device interrupt with a memory write and a "fire and forget"
kick IPI that only happens when the system is not already busyprocessing I/Os, so that it can be eliminated when the system i= s
most busy. With interrupt coalescing, you can send IPIs at = a rate
much lower than the actual I/O rate.
Not sure how difficult it is to adapt a scheme like this to the= current
state of qemu / kvm, but I'm pretty sure it works we= ll if you implement
it correctly ;-)


A bigger issue th= an vmexit latency is device emulation thread wakeup
latency. = There is a thread (QEMU, vhost-user, vhost, etc) monitoring the
ioeventfd but it may be descheduled. Its physical CPU may be in a lowpower state. I ran a benchmark late last year with QEMU's AioCo= ntext
adaptive polling disabled so we can measure the wakeup = latency:

     CPU 0/K= VM 26102 [000] 85626.737072:       kvm:kvm_fa= st_mmio:
fast mmio at gpa 0xfde03000
 &nbs= p;IO iothread1 26099 [001] 85626.737076: syscalls:sys_exit_ppoll: 0x1
           =       4 microseconds ------^

Hi C= hristophe,
QEMU/KVM doe= s something similar to what you described. In the perf
output above the vmexit kvm_fast_mmio event= occurs on physical CPU
"[000]".  The IOThread wakes up on physical CPU "[001]". Physical CPU= #0
resumes guest execut= ion immediately after marking the ioeventfd ready.
There is no context switch to the IOThread or a= return from
ioctl(KVM_= RUN) on CPU#0.

Oh, that's good.

But then the conclusion that the 4us delay limits us to 250kIO= PS
is incorrect, no? Is there anything that would prevent multipl= e
I/O events (doorbell or interrupt) to be in flight at the same = time?


The IOThread reads the = eventfd. An eventfd is a counter that is reset to
0 on read. Because it's a counter you get coales= cing: if the guest
perf= orms multiple MMIO writes the ioeventfd counter increases but theIOThread only wakes up once and = reads the ioeventfd.

VIRTIO itself also has a mechanism for= suppressing notifications called
EVENT_IDX. It allows the driver to let the device know that it= does not
require inter= rupts, and the device to let the driver know it does not
require virtqueue kicks. This reminds me = a bit of the mitigation
mechanism you described.

Stefan
<= /div>
--Apple-Mail=_F0D894EB-17FA-4632-8E34-AA339BFC305F--