qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* Re: [PATCH v2 0/5] virtio mmio specification enhancement
@ 2020-07-30 20:15 Pincus, Josh
  2020-07-31 10:13 ` Stefan Hajnoczi
  2020-07-31 15:44 ` Alex Bennée
  0 siblings, 2 replies; 9+ messages in thread
From: Pincus, Josh @ 2020-07-30 20:15 UTC (permalink / raw)
  To: linux-kernel, zhabin; +Cc: virtio-dev, qemu-devel

[-- Attachment #1: Type: text/plain, Size: 265 bytes --]

Hi,

We were looking into a similar enhancement for the Virt I/O MMIO transport and came across this project.
This enhancement would be perfect for us.

Has there been any progress since Feb, 2020?  It looks like the effort might have stalled?

Thanks,
JP

[-- Attachment #2: Type: text/html, Size: 2226 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2 0/5] virtio mmio specification enhancement
  2020-07-30 20:15 [PATCH v2 0/5] virtio mmio specification enhancement Pincus, Josh
@ 2020-07-31 10:13 ` Stefan Hajnoczi
  2020-07-31 15:44 ` Alex Bennée
  1 sibling, 0 replies; 9+ messages in thread
From: Stefan Hajnoczi @ 2020-07-31 10:13 UTC (permalink / raw)
  To: Pincus, Josh
  Cc: virtio-dev, zhabin, Alex Bennée, linux-kernel, qemu-devel

[-- Attachment #1: Type: text/plain, Size: 391 bytes --]

On Thu, Jul 30, 2020 at 08:15:12PM +0000, Pincus, Josh wrote:
> We were looking into a similar enhancement for the Virt I/O MMIO transport and came across this project.
> This enhancement would be perfect for us.
> 
> Has there been any progress since Feb, 2020?  It looks like the effort might have stalled?

CCing Alex Bennee. I think he recently asked the same question.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2 0/5] virtio mmio specification enhancement
  2020-07-30 20:15 [PATCH v2 0/5] virtio mmio specification enhancement Pincus, Josh
  2020-07-31 10:13 ` Stefan Hajnoczi
@ 2020-07-31 15:44 ` Alex Bennée
  2020-08-03 16:19   ` Alex Bennée
  2020-08-03 23:31   ` Pincus, Josh
  1 sibling, 2 replies; 9+ messages in thread
From: Alex Bennée @ 2020-07-31 15:44 UTC (permalink / raw)
  To: Pincus, Josh; +Cc: virtio-dev, zhabin, linux-kernel, qemu-devel


Pincus, Josh <Josh.Pincus@windriver.com> writes:

> Hi,
>
>  
>
> We were looking into a similar enhancement for the Virt I/O MMIO transport and came across this project.
>
> This enhancement would be perfect for us.

So there is certainly an interest in optimising MMIO based virtio and
the current read/ack cycle adds additional round trip time for any trap
and emulate hypervisor. However I think there is some resistance to
making MMIO a re-implementation of what PCI already gives us for "free".

I believe the current questions that need to be addressed are:

  - Clear definitions in the spec on doorbells/notifications

    The current virtio spec uses different terms in some places so it
    would be nice to clarify the language and formalise what the
    standard expects from transports w.r.t the capabilities of
    notifications and doorbells.

  - Quantifying the memory foot-print difference between PCI/MMIO

    PCI gives a lot for free including a discovery and IRQ model already
    designed to handle MSI/MSI-X. There is a claim that this brings in a
    lot of bloat but I think there was some debate around the numbers.
    My rough initial experiment with a PCI and non-PCI build with
    otherwise identical VIRTIO configs results in the following:

    16:40:15 c.282% [alex@zen:~/l/l/builds] review/rpmb|… + ls -l arm64/vmlinux arm64.nopci/vmlinux
    -rwxr-xr-x 1 alex alex 83914728 Jul 31 16:39 arm64.nopci/vmlinux*
    -rwxr-xr-x 1 alex alex 86368080 Jul 31 16:33 arm64/vmlinux*

    which certainly implies there could be a fair amount of headroom for
    an MMIO version to implement some features. However I don't know if
    it's fully apples to apples as there maybe unneeded PCI bloat that a
    virtio-only kernel doesn't need.


What are the features you are most interested in?

> Has there been any progress since Feb, 2020?  It looks like the effort
> might have stalled?

I can't speak to the OP's but there is certainly interest from others
that are not the original posters.


-- 
Alex Bennée


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2 0/5] virtio mmio specification enhancement
  2020-07-31 15:44 ` Alex Bennée
@ 2020-08-03 16:19   ` Alex Bennée
  2020-08-03 23:31   ` Pincus, Josh
  1 sibling, 0 replies; 9+ messages in thread
From: Alex Bennée @ 2020-08-03 16:19 UTC (permalink / raw)
  To: Pincus, Josh; +Cc: virtio-dev, zhabin, linux-kernel, qemu-devel


Alex Bennée <alex.bennee@linaro.org> writes:

> Pincus, Josh <Josh.Pincus@windriver.com> writes:
>
>> Hi,
>>
>>  
>>
>> We were looking into a similar enhancement for the Virt I/O MMIO transport and came across this project.
>>
>> This enhancement would be perfect for us.
>
> So there is certainly an interest in optimising MMIO based virtio and
> the current read/ack cycle adds additional round trip time for any trap
> and emulate hypervisor. However I think there is some resistance to
> making MMIO a re-implementation of what PCI already gives us for "free".
<snip>
>
>   - Quantifying the memory foot-print difference between PCI/MMIO
>
>     PCI gives a lot for free including a discovery and IRQ model already
>     designed to handle MSI/MSI-X. There is a claim that this brings in a
>     lot of bloat but I think there was some debate around the numbers.
>     My rough initial experiment with a PCI and non-PCI build with
>     otherwise identical VIRTIO configs results in the following:
>
>     16:40:15 c.282% [alex@zen:~/l/l/builds] review/rpmb|… + ls -l arm64/vmlinux arm64.nopci/vmlinux
>     -rwxr-xr-x 1 alex alex 83914728 Jul 31 16:39 arm64.nopci/vmlinux*
>     -rwxr-xr-x 1 alex alex 86368080 Jul 31 16:33 arm64/vmlinux*
>
>     which certainly implies there could be a fair amount of headroom for
>     an MMIO version to implement some features. However I don't know if
>     it's fully apples to apples as there maybe unneeded PCI bloat that a
>     virtio-only kernel doesn't need.

Just following up after cutting the Xgene and ThunderX PCI bloat from
the kernel the margin is a little smaller:

  -rwxr-xr-x 1 alex alex 83914728 Jul 31 16:39 arm64.nopci/vmlinux*
  -rwxr-xr-x 1 alex alex 85639808 Aug  3 17:12 arm64/vmlinux*

-- 
Alex Bennée


^ permalink raw reply	[flat|nested] 9+ messages in thread

* RE: [PATCH v2 0/5] virtio mmio specification enhancement
  2020-07-31 15:44 ` Alex Bennée
  2020-08-03 16:19   ` Alex Bennée
@ 2020-08-03 23:31   ` Pincus, Josh
  1 sibling, 0 replies; 9+ messages in thread
From: Pincus, Josh @ 2020-08-03 23:31 UTC (permalink / raw)
  To: Alex Bennée; +Cc: virtio-dev, zhabin, linux-kernel, qemu-devel

Hi Alex,

Thank you for the reply.

Please see my inline response below.

-----Original Message-----
From: Alex Bennée <alex.bennee@linaro.org> 
Sent: Friday, July 31, 2020 8:45 AM
To: Pincus, Josh <Josh.Pincus@windriver.com>
Cc: linux-kernel@vger.kernel.org; zhabin@linux.alibaba.com; virtio-dev@lists.oasis-open.org; qemu-devel@nongnu.org
Subject: Re: [PATCH v2 0/5] virtio mmio specification enhancement


Pincus, Josh <Josh.Pincus@windriver.com> writes:

> Hi,
>
>  
>
> We were looking into a similar enhancement for the Virt I/O MMIO transport and came across this project.
>
> This enhancement would be perfect for us.

So there is certainly an interest in optimising MMIO based virtio and the current read/ack cycle adds additional round trip time for any trap and emulate hypervisor. However I think there is some resistance to making MMIO a re-implementation of what PCI already gives us for "free".

I believe the current questions that need to be addressed are:

  - Clear definitions in the spec on doorbells/notifications

    The current virtio spec uses different terms in some places so it
    would be nice to clarify the language and formalise what the
    standard expects from transports w.r.t the capabilities of
    notifications and doorbells.

[JP] The read/ack cycle not only adds to the round-trip time for any trap and emulate HYP, but it also precludes an environment where one might want to avoid emulation completely.  We're interested in using the MMIO transport combined with an augmented device node in the DTB to have device features, reserved memory for queues, and specific MSI interrupts per queue conveyed to the guest statically.  In this kind of restricted environment, negotiation for features might be completely disabled; you see what the device node describes and you either support those features or not.  Likewise, the standard list of state machine transitions for communicating driver and device state would be skipped.  A driver in a guest comes up, reads the device node info, uses the queues as described, and assigns the MSI vectors per queue and config-has-changed service.  When an interrupt comes in, there's no need to ack it beyond the normal way in which one conveys an EOI to hardware.  It also means that with one dedicated interrupt per queue we won't have to select the queue in question and test which one got updated.  In short, we are experimenting with getting rid of the emulation if we can.

  - Quantifying the memory foot-print difference between PCI/MMIO

    PCI gives a lot for free including a discovery and IRQ model already
    designed to handle MSI/MSI-X. There is a claim that this brings in a
    lot of bloat but I think there was some debate around the numbers.
    My rough initial experiment with a PCI and non-PCI build with
    otherwise identical VIRTIO configs results in the following:

    16:40:15 c.282% [alex@zen:~/l/l/builds] review/rpmb|… + ls -l arm64/vmlinux arm64.nopci/vmlinux
    -rwxr-xr-x 1 alex alex 83914728 Jul 31 16:39 arm64.nopci/vmlinux*
    -rwxr-xr-x 1 alex alex 86368080 Jul 31 16:33 arm64/vmlinux*

    which certainly implies there could be a fair amount of headroom for
    an MMIO version to implement some features. However I don't know if
    it's fully apples to apples as there maybe unneeded PCI bloat that a
    virtio-only kernel doesn't need.

[JP] Apropos of your subsequent email on this topic, the PCI bloat isn't terrible.  The major stumbling block in our case is that we would like to see if there's a restricted model in which the emulation can be removed completely.  Case in point: Virt I/O RPMsgs in OpenAMP only use the queues to transfer data back and forth.  (Unless I'm mistaken?)   We'd like to see if that model can be a bit more generalized so that other kinds of drivers can be constructed that similarly don't rely on emulation for handling interrupt read/ack, feature negotiation, queue selection, etc.  Memory is mapped into the guest for queues and R/O device registers, interrupts are assigned in the DTB for each queue, and features are, essentially, non-negotiable.  

What are the features you are most interested in?

[JP] See above. 😉 The restricted environment in question is for very simple applications that don't have any kind of PCI infrastructure and for virtual environments with no HYP or a very restricted HYP.  

> Has there been any progress since Feb, 2020?  It looks like the effort 
> might have stalled?

I can't speak to the OP's but there is certainly interest from others that are not the original posters.

[JP] Maybe we can restart the thread/discussion and see where it goes from here.

--
Alex Bennée

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2 0/5] virtio mmio specification enhancement
  2020-02-10 11:44 ` Michael S. Tsirkin
@ 2020-02-11 16:05   ` Chao Peng
  2020-02-11 10:57     ` Michael S. Tsirkin
  0 siblings, 1 reply; 9+ messages in thread
From: Chao Peng @ 2020-02-11 16:05 UTC (permalink / raw)
  To: Michael S. Tsirkin
  Cc: virtio-dev, Zha Bin, slp, jasowang, jing2.liu, linux-kernel,
	qemu-devel, gerry

On Mon, Feb 10, 2020 at 06:44:50AM -0500, Michael S. Tsirkin wrote:
> On Mon, Feb 10, 2020 at 05:05:16PM +0800, Zha Bin wrote:
> > We have compared the number of files and the lines of code between
> > virtio-mmio and virio-pci.
> > 
> > 				Virtio-PCI	    Virtio-MMIO	
> > 	number of files(Linux)	    161			1
> > 	lines of code(Linux)	    78237		538
> 
> 
> 
> Something's very wrong here. virtio PCI is 161 files?
> Are you counting the whole PCI subsystem?

Right, that is just a rough statistics. Surely enough, some drivers will
never get enabled in a typcial config.

> Sure enough:
> 
> $ find drivers/pci -name '*c' |wc -l
> 150

and plus:
$ find arch/x86/pci/ -name '*c' |wc -l
22

> 
> That's not reasonable, this includes a bunch of drivers that
> never run on a typical hypervisor.
> 
> MMIO is also not as small as you are trying to show:
> 
> $ cloc drivers/virtio/virtio_mmio.c include/uapi/linux/virtio_mmio.h
>        2 text files.
>        2 unique files.                              
>        0 files ignored.
> 
> github.com/AlDanial/cloc v 1.82  T=0.01 s (230.7 files/s, 106126.5 lines/s)
> -------------------------------------------------------------------------------
> Language                     files          blank        comment           code
> -------------------------------------------------------------------------------
> C                                1            144            100            535
> C/C++ Header                     1             39             66             36
> -------------------------------------------------------------------------------
> SUM:                             2            183            166            571
> -------------------------------------------------------------------------------
> 
> 
> I don't doubt MMIO is smaller than PCI. Of course that's because it has
> no features to speak of - just this patch already doubles it's size. If
> we keep doing that because we want the features then they will reach
> the same size in about 4 iterations.

Since current virtio-mmio size is small enough, so adding any notable
feature would easily double it. I have no objection that it may one day
reach the same level of PCI, but in this patch some are actually
generic changes and for MSI specific code we provide the option to
confige away.

Thanks,
Chao

> 
> 
> -- 
> MST


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2 0/5] virtio mmio specification enhancement
  2020-02-11 16:05   ` Chao Peng
@ 2020-02-11 10:57     ` Michael S. Tsirkin
  0 siblings, 0 replies; 9+ messages in thread
From: Michael S. Tsirkin @ 2020-02-11 10:57 UTC (permalink / raw)
  To: Chao Peng
  Cc: virtio-dev, Zha Bin, slp, jasowang, jing2.liu, linux-kernel,
	qemu-devel, gerry

On Tue, Feb 11, 2020 at 04:05:41PM +0000, Chao Peng wrote:
> On Mon, Feb 10, 2020 at 06:44:50AM -0500, Michael S. Tsirkin wrote:
> > On Mon, Feb 10, 2020 at 05:05:16PM +0800, Zha Bin wrote:
> > > We have compared the number of files and the lines of code between
> > > virtio-mmio and virio-pci.
> > > 
> > > 				Virtio-PCI	    Virtio-MMIO	
> > > 	number of files(Linux)	    161			1
> > > 	lines of code(Linux)	    78237		538
> > 
> > 
> > 
> > Something's very wrong here. virtio PCI is 161 files?
> > Are you counting the whole PCI subsystem?
> 
> Right, that is just a rough statistics.

Please try not to make them look so wrong then.
E.g. you don't include drivers/base/platform-msi.c for
mmio do you? Your patch brings a bunch of code in there.

> Surely enough, some drivers will
> never get enabled in a typcial config.
> 
> > Sure enough:
> > 
> > $ find drivers/pci -name '*c' |wc -l
> > 150
> 
> and plus:
> $ find arch/x86/pci/ -name '*c' |wc -l
> 22

But what's the point? This is code that is maintained by PCI core
people anyway.

> > 
> > That's not reasonable, this includes a bunch of drivers that
> > never run on a typical hypervisor.
> > 
> > MMIO is also not as small as you are trying to show:
> > 
> > $ cloc drivers/virtio/virtio_mmio.c include/uapi/linux/virtio_mmio.h
> >        2 text files.
> >        2 unique files.                              
> >        0 files ignored.
> > 
> > github.com/AlDanial/cloc v 1.82  T=0.01 s (230.7 files/s, 106126.5 lines/s)
> > -------------------------------------------------------------------------------
> > Language                     files          blank        comment           code
> > -------------------------------------------------------------------------------
> > C                                1            144            100            535
> > C/C++ Header                     1             39             66             36
> > -------------------------------------------------------------------------------
> > SUM:                             2            183            166            571
> > -------------------------------------------------------------------------------
> > 
> > 
> > I don't doubt MMIO is smaller than PCI. Of course that's because it has
> > no features to speak of - just this patch already doubles it's size. If
> > we keep doing that because we want the features then they will reach
> > the same size in about 4 iterations.
> 
> Since current virtio-mmio size is small enough, so adding any notable
> feature would easily double it.

But really unlike PCI this is just PV stuff that is not reused by
anyone. We end up maintaining all this by ourselves.

> I have no objection that it may one day
> reach the same level of PCI, but in this patch some are actually
> generic changes and for MSI specific code we provide the option to
> confige away.
> 
> Thanks,
> Chao

The option will make it fall down at runtime but
it does not actually seem to remove all of the overhead.



> > 
> > 
> > -- 
> > MST



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2 0/5] virtio mmio specification enhancement
  2020-02-10  9:05 Zha Bin
@ 2020-02-10 11:44 ` Michael S. Tsirkin
  2020-02-11 16:05   ` Chao Peng
  0 siblings, 1 reply; 9+ messages in thread
From: Michael S. Tsirkin @ 2020-02-10 11:44 UTC (permalink / raw)
  To: Zha Bin
  Cc: virtio-dev, slp, jasowang, jing2.liu, linux-kernel, qemu-devel,
	chao.p.peng, gerry

On Mon, Feb 10, 2020 at 05:05:16PM +0800, Zha Bin wrote:
> We have compared the number of files and the lines of code between
> virtio-mmio and virio-pci.
> 
> 				Virtio-PCI	    Virtio-MMIO	
> 	number of files(Linux)	    161			1
> 	lines of code(Linux)	    78237		538



Something's very wrong here. virtio PCI is 161 files?
Are you counting the whole PCI subsystem?
Sure enough:

$ find drivers/pci -name '*c' |wc -l
150

That's not reasonable, this includes a bunch of drivers that
never run on a typical hypervisor.

MMIO is also not as small as you are trying to show:

$ cloc drivers/virtio/virtio_mmio.c include/uapi/linux/virtio_mmio.h
       2 text files.
       2 unique files.                              
       0 files ignored.

github.com/AlDanial/cloc v 1.82  T=0.01 s (230.7 files/s, 106126.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                                1            144            100            535
C/C++ Header                     1             39             66             36
-------------------------------------------------------------------------------
SUM:                             2            183            166            571
-------------------------------------------------------------------------------


I don't doubt MMIO is smaller than PCI. Of course that's because it has
no features to speak of - just this patch already doubles it's size. If
we keep doing that because we want the features then they will reach
the same size in about 4 iterations.


-- 
MST



^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 0/5] virtio mmio specification enhancement
@ 2020-02-10  9:05 Zha Bin
  2020-02-10 11:44 ` Michael S. Tsirkin
  0 siblings, 1 reply; 9+ messages in thread
From: Zha Bin @ 2020-02-10  9:05 UTC (permalink / raw)
  To: linux-kernel
  Cc: virtio-dev, zhabin, slp, mst, jasowang, jing2.liu, qemu-devel,
	chao.p.peng, gerry

In cloud native environment, we need a lightweight and secure system. It
should benefit from the speed of containers and the security of VM, which
is classified as secure containers. The traditional solution of cloud VM
is Qemu. In fact we don't need to pay for the legacy devices. Currently,
more userspace VMMs, e.g. Qemu, Firecracker, Cloud Hypervisor and Alibaba
Cloud VMM which is called Dragonball, began to pay attention to a
lightweight solution.

The lightweight VMM is suitable to cloud native infrastructure which is
designed for creating secure sandbox to address the requirements of
multi-tenant. Meanwhile, with faster startup time and lower memory
overhead, it makes possible to launch thousands of microVMs on the same
machine. This VMM minimizes the emulation devices and uses virtio-mmio to
get a more lightweight transport layer. The virtio-mmio devices have less
code than virtio-pci, which can decrease boot time and increase deploy
density by customizing kernel such as setting pci=off. From another point
of view, the minimal device can reduce the attack surface.

We have compared the number of files and the lines of code between
virtio-mmio and virio-pci.

				Virtio-PCI	    Virtio-MMIO	
	number of files(Linux)	    161			1
	lines of code(Linux)	    78237		538
	number of files(Qemu)	    24			1
	lines of code(Qemu)	    8952		421

But the current standard virtio-mmio spec has some limitations which is
only support legacy interrupt and will cause performance penalties.

To address such limitation, we proposed to update virtio-mmio spec with
two new feature bits to support MSI interrupt and enhancing notification
mechanism[1], which can achieve the same performance as virtio-pci devices
with only around 600 lines of code.

Here are the performance gain of MSI interrupt in virtio-mmio. Each case is
repeated three times.

        netperf -t TCP_RR -H 192.168.1.36 -l 30 -- -r 32,1024

                Virtio-PCI    Virtio-MMIO   Virtio-MMIO(MSI)
        trans/s     9536        6939            9500
        trans/s     9734        7029            9749
        trans/s     9894        7095            9318

With the virtio spec proposal[1], other VMMs (e.g. Qemu) can also make use
of the new features to get a enhancing performance.

[1] https://lkml.org/lkml/2020/1/21/31

Change Log:
v1->v2
* Change version update to feature bit
* Add mask/unmask support
* Add two MSI sharing/non-sharing modes
* Create generic irq domain for all architectures

Liu Jiang (5):
  virtio-mmio: add notify feature for per-queue
  virtio-mmio: refactor common functionality
  virtio-mmio: create a generic MSI irq domain
  virtio-mmio: add MSI interrupt feature support
  x86: virtio-mmio: support virtio-mmio with MSI for x86

 arch/x86/kernel/apic/msi.c          |  11 +-
 drivers/base/platform-msi.c         |   4 +-
 drivers/virtio/Kconfig              |   9 +
 drivers/virtio/virtio_mmio.c        | 351 ++++++++++++++++++++++++++++++++----
 drivers/virtio/virtio_mmio_common.h |  39 ++++
 drivers/virtio/virtio_mmio_msi.h    | 175 ++++++++++++++++++
 include/linux/msi.h                 |   1 +
 include/uapi/linux/virtio_config.h  |  13 +-
 include/uapi/linux/virtio_mmio.h    |  31 ++++
 9 files changed, 596 insertions(+), 38 deletions(-)
 create mode 100644 drivers/virtio/virtio_mmio_common.h
 create mode 100644 drivers/virtio/virtio_mmio_msi.h

-- 
1.8.3.1



^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2020-08-03 23:47 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-30 20:15 [PATCH v2 0/5] virtio mmio specification enhancement Pincus, Josh
2020-07-31 10:13 ` Stefan Hajnoczi
2020-07-31 15:44 ` Alex Bennée
2020-08-03 16:19   ` Alex Bennée
2020-08-03 23:31   ` Pincus, Josh
  -- strict thread matches above, loose matches on Subject: below --
2020-02-10  9:05 Zha Bin
2020-02-10 11:44 ` Michael S. Tsirkin
2020-02-11 16:05   ` Chao Peng
2020-02-11 10:57     ` Michael S. Tsirkin

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).