From: Oleksandr <olekstysh@gmail.com>
To: "Alex Bennée" <alex.bennee@linaro.org>
Cc: xen-devel@lists.xenproject.org,
"Oleksandr Tyshchenko" <oleksandr_tyshchenko@epam.com>,
"Paul Durrant" <paul@xen.org>, "Jan Beulich" <jbeulich@suse.com>,
"Andrew Cooper" <andrew.cooper3@citrix.com>,
"Roger Pau Monné" <roger.pau@citrix.com>, "Wei Liu" <wl@xen.org>,
"Julien Grall" <julien.grall@arm.com>,
"George Dunlap" <george.dunlap@citrix.com>,
"Ian Jackson" <iwj@xenproject.org>,
"Julien Grall" <julien@xen.org>,
"Stefano Stabellini" <sstabellini@kernel.org>,
"Tim Deegan" <tim@xen.org>,
"Daniel De Graaf" <dgdegra@tycho.nsa.gov>,
"Volodymyr Babchuk" <Volodymyr_Babchuk@epam.com>,
"Jun Nakajima" <jun.nakajima@intel.com>,
"Kevin Tian" <kevin.tian@intel.com>,
"Anthony PERARD" <anthony.perard@citrix.com>,
"Bertrand Marquis" <bertrand.marquis@arm.com>,
"Wei Chen" <Wei.Chen@arm.com>, "Kaly Xin" <Kaly.Xin@arm.com>,
"Artem Mygaiev" <joculator@gmail.com>
Subject: Re: [PATCH V3 00/23] IOREQ feature (+ virtio-mmio) on Arm
Date: Tue, 1 Dec 2020 00:22:56 +0200 [thread overview]
Message-ID: <248003d5-87a2-a7b2-5b30-e94c2a49945b@gmail.com> (raw)
In-Reply-To: <87h7p6u860.fsf@linaro.org>
On 30.11.20 18:21, Alex Bennée wrote:
Hi Alex
[added missed subject title]
> Oleksandr Tyshchenko <olekstysh@gmail.com> writes:
>
>> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
>>
>>
>> Date: Sat, 28 Nov 2020 22:33:51 +0200
>> Subject: [PATCH V3 00/23] IOREQ feature (+ virtio-mmio) on Arm
>> MIME-Version: 1.0
>> Content-Type: text/plain; charset=UTF-8
>> Content-Transfer-Encoding: 8bit
>>
>> Hello all.
>>
>> The purpose of this patch series is to add IOREQ/DM support to Xen on Arm.
>> You can find an initial discussion at [1] and RFC/V1/V2 series at [2]/[3]/[4].
>> Xen on Arm requires some implementation to forward guest MMIO access to a device
>> model in order to implement virtio-mmio backend or even mediator outside of hypervisor.
>> As Xen on x86 already contains required support this series tries to make it common
>> and introduce Arm specific bits plus some new functionality. Patch series is based on
>> Julien's PoC "xen/arm: Add support for Guest IO forwarding to a device emulator".
>> Besides splitting existing IOREQ/DM support and introducing Arm side, the series
>> also includes virtio-mmio related changes (last 2 patches for toolstack)
>> for the reviewers to be able to see how the whole picture could look
>> like.
> Thanks for posting the latest version.
>
>> According to the initial discussion there are a few open questions/concerns
>> regarding security, performance in VirtIO solution:
>> 1. virtio-mmio vs virtio-pci, SPI vs MSI, different use-cases require different
>> transport...
> I think I'm repeating things here I've said in various ephemeral video
> chats over the last few weeks but I should probably put things down on
> the record.
>
> I think the original intention of the virtio framers is advanced
> features would build on virtio-pci because you get a bunch of things
> "for free" - notably enumeration and MSI support. There is assumption
> that by the time you add these features to virtio-mmio you end up
> re-creating your own less well tested version of virtio-pci. I've not
> been terribly convinced by the argument that the guest implementation of
> PCI presents a sufficiently large blob of code to make the simpler MMIO
> desirable. My attempts to build two virtio kernels (PCI/MMIO) with
> otherwise the same devices wasn't terribly conclusive either way.
>
> That said virtio-mmio still has life in it because the cloudy slimmed
> down guests moved to using it because the enumeration of PCI is a road
> block to their fast boot up requirements. I'm sure they would also
> appreciate a MSI implementation to reduce the overhead that handling
> notifications currently has on trap-and-emulate.
>
> AIUI for Xen the other downside to PCI is you would have to emulate it
> in the hypervisor which would be additional code at the most privileged
> level.
Thank you for putting things together here and valuable input. As for
me, the "virtio-mmio & MSI solution" as a performance improvement sounds
indeed
interesting. Flipping through the virtio-mmio links I found a discussion
regarding that [1].
I think this needs an additional investigation and experiments, however
I am not sure there is an existing infrastructure in Xen on Arm to do so.
Once we make some progress with the IOREQ series I would be able to
focus on enhancements which we would consider worthwhile.
>
>> 2. virtio backend is able to access all guest memory, some kind of protection
>> is needed: 'virtio-iommu in Xen' vs 'pre-shared-memory & memcpys in
>> guest'
> This is also an area of interest for Project Stratos and something we
> would like to be solved generally for all hypervisors. There is a good
> write up of some approaches that Jean Phillipe did on the stratos
> mailing list:
>
> From: Jean-Philippe Brucker <jean-philippe@linaro.org>
> Subject: Limited memory sharing investigation
> Message-ID: <20201002134336.GA2196245@myrica>
>
> I suspect there is a good argument for the simplicity of a combined
> virt queue but it is unlikely to be very performance orientated.
I will look at it.
>> 3. interface between toolstack and 'out-of-qemu' virtio backend, avoid using
>> Xenstore in virtio backend if possible.
> I wonder how much work it would be for a rust expert to make:
>
> https://github.com/slp/vhost-user-blk
>
> handle an IOREQ signalling pathway instead of the vhost-user/eventfd
> pathway? That would give a good indication on how "hypervisor blind"
> these daemons could be made.
>
> <snip>
>> Please note, build-test passed for the following modes:
>> 1. x86: CONFIG_HVM=y / CONFIG_IOREQ_SERVER=y (default)
>> 2. x86: #CONFIG_HVM is not set / #CONFIG_IOREQ_SERVER is not set
>> 3. Arm64: CONFIG_HVM=y / CONFIG_IOREQ_SERVER=y
> Forgive my relative newness to Xen, how do I convince the hypervisor to
> build with this on? I've tried variants of:
>
> make -j9 CROSS_COMPILE=aarch64-linux-gnu- XEN_TARGET_ARCH=arm64 menuconfig XEN_EXPERT=y [CONFIG_|XEN_|_]IOREQ_SERVER=y
CONFIG_IOREQ_SERVER is not protected by CONFIG_XEN_EXPERT. I mentioned
how to enable CONFIG_IOREQ_SERVER on Arm (since it is disabled by
default within this series) when
describing how test this series to Masami, but forgot to add here. Could
you apply the one-line patch [2] and rebuild. Sorry for inconvenience.
[1] https://lwn.net/Articles/812055/
[2]
https://github.com/otyshchenko1/xen/commit/b371bc9a3c954595bfce01bad244260364bbcd48
--
Regards,
Oleksandr Tyshchenko
next prev parent reply other threads:[~2020-11-30 22:23 UTC|newest]
Thread overview: 127+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-30 10:31 Oleksandr Tyshchenko
2020-11-30 10:31 ` [PATCH V3 01/23] x86/ioreq: Prepare IOREQ feature for making it common Oleksandr Tyshchenko
2020-12-01 11:03 ` Alex Bennée
2020-12-01 18:53 ` Oleksandr
2020-12-01 19:36 ` Alex Bennée
2020-12-02 8:00 ` Jan Beulich
2020-12-02 11:19 ` Oleksandr
2020-12-07 11:13 ` Jan Beulich
2020-12-07 15:27 ` Oleksandr
2020-12-07 16:29 ` Jan Beulich
2020-12-07 17:21 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 02/23] x86/ioreq: Add IOREQ_STATUS_* #define-s and update code for moving Oleksandr Tyshchenko
2020-12-01 11:07 ` Alex Bennée
2020-12-07 11:19 ` Jan Beulich
2020-12-07 15:37 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 03/23] x86/ioreq: Provide out-of-line wrapper for the handle_mmio() Oleksandr Tyshchenko
2020-12-07 11:27 ` Jan Beulich
2020-12-07 15:39 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 04/23] xen/ioreq: Make x86's IOREQ feature common Oleksandr Tyshchenko
2020-12-07 11:41 ` Jan Beulich
2020-12-07 19:43 ` Oleksandr
2020-12-08 9:21 ` Jan Beulich
2020-12-08 13:56 ` Oleksandr
2020-12-08 15:02 ` Jan Beulich
2020-12-08 17:24 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 05/23] xen/ioreq: Make x86's hvm_ioreq_needs_completion() common Oleksandr Tyshchenko
2020-12-07 11:47 ` Jan Beulich
2020-11-30 10:31 ` [PATCH V3 06/23] xen/ioreq: Make x86's hvm_mmio_first(last)_byte() common Oleksandr Tyshchenko
2020-12-07 11:48 ` Jan Beulich
2020-11-30 10:31 ` [PATCH V3 07/23] xen/ioreq: Make x86's hvm_ioreq_(page/vcpu/server) structs common Oleksandr Tyshchenko
2020-12-07 11:54 ` Jan Beulich
2020-11-30 10:31 ` [PATCH V3 08/23] xen/ioreq: Move x86's ioreq_server to struct domain Oleksandr Tyshchenko
2020-12-07 12:04 ` Jan Beulich
2020-12-07 12:12 ` Paul Durrant
2020-12-07 19:52 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 09/23] xen/dm: Make x86's DM feature common Oleksandr Tyshchenko
2020-12-07 12:08 ` Jan Beulich
2020-12-07 20:23 ` Oleksandr
2020-12-08 9:30 ` Jan Beulich
2020-12-08 14:54 ` Oleksandr
2021-01-07 14:38 ` Oleksandr
2021-01-07 15:01 ` Jan Beulich
2021-01-07 16:49 ` Oleksandr
2021-01-12 22:23 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 10/23] xen/mm: Make x86's XENMEM_resource_ioreq_server handling common Oleksandr Tyshchenko
2020-12-07 11:35 ` Jan Beulich
2020-12-07 12:11 ` Jan Beulich
2020-12-07 21:06 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 11/23] xen/ioreq: Move x86's io_completion/io_req fields to struct vcpu Oleksandr Tyshchenko
2020-12-07 12:32 ` Jan Beulich
2020-12-07 20:59 ` Oleksandr
2020-12-08 7:52 ` Paul Durrant
2020-12-08 9:35 ` Jan Beulich
2020-12-08 18:21 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 12/23] xen/ioreq: Remove "hvm" prefixes from involved function names Oleksandr Tyshchenko
2020-12-07 12:45 ` Jan Beulich
2020-12-07 20:28 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 13/23] xen/ioreq: Use guest_cmpxchg64() instead of cmpxchg() Oleksandr Tyshchenko
2020-12-09 21:32 ` Stefano Stabellini
2020-12-09 22:34 ` Oleksandr
2020-12-10 2:30 ` Stefano Stabellini
2020-11-30 10:31 ` [PATCH V3 14/23] arm/ioreq: Introduce arch specific bits for IOREQ/DM features Oleksandr Tyshchenko
2020-12-09 22:04 ` Stefano Stabellini
2020-12-09 22:49 ` Oleksandr
2020-12-10 2:30 ` Stefano Stabellini
2020-11-30 10:31 ` [PATCH V3 15/23] xen/arm: Stick around in leave_hypervisor_to_guest until I/O has completed Oleksandr Tyshchenko
2020-11-30 20:51 ` Volodymyr Babchuk
2020-12-01 12:46 ` Julien Grall
2020-12-09 23:18 ` Stefano Stabellini
2020-12-09 23:35 ` Stefano Stabellini
2020-12-09 23:47 ` Julien Grall
2020-12-10 2:30 ` Stefano Stabellini
2020-12-10 13:17 ` Julien Grall
2020-12-10 13:21 ` Oleksandr
2020-12-09 23:38 ` Julien Grall
2020-11-30 10:31 ` [PATCH V3 16/23] xen/mm: Handle properly reference in set_foreign_p2m_entry() on Arm Oleksandr Tyshchenko
2020-12-08 14:24 ` Jan Beulich
2020-12-08 16:41 ` Oleksandr
2020-12-09 23:49 ` Stefano Stabellini
2021-01-15 1:18 ` Stefano Stabellini
2020-11-30 10:31 ` [PATCH V3 17/23] xen/ioreq: Introduce domain_has_ioreq_server() Oleksandr Tyshchenko
2020-12-08 15:11 ` Jan Beulich
2020-12-08 15:33 ` Oleksandr
2020-12-08 16:56 ` Oleksandr
2020-12-08 19:43 ` Paul Durrant
2020-12-08 20:16 ` Oleksandr
2020-12-09 9:01 ` Paul Durrant
2020-12-09 18:58 ` Julien Grall
2020-12-09 21:05 ` Oleksandr
2020-12-09 20:36 ` Oleksandr
2020-12-10 8:38 ` Paul Durrant
2020-12-10 16:57 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 18/23] xen/dm: Introduce xendevicemodel_set_irq_level DM op Oleksandr Tyshchenko
2020-12-10 2:21 ` Stefano Stabellini
2020-12-10 12:58 ` Oleksandr
2020-12-10 13:38 ` Julien Grall
2020-11-30 10:31 ` [PATCH V3 19/23] xen/arm: io: Abstract sign-extension Oleksandr Tyshchenko
2020-11-30 21:03 ` Volodymyr Babchuk
2020-11-30 23:27 ` Oleksandr
2020-12-01 7:55 ` Jan Beulich
2020-12-01 10:30 ` Julien Grall
2020-12-01 10:42 ` Oleksandr
2020-12-01 12:13 ` Julien Grall
2020-12-01 12:24 ` Oleksandr
2020-12-01 12:28 ` Julien Grall
2020-12-01 10:49 ` Jan Beulich
2020-12-01 10:23 ` Julien Grall
2020-11-30 10:31 ` [PATCH V3 20/23] xen/ioreq: Make x86's send_invalidate_req() common Oleksandr Tyshchenko
2020-12-08 15:24 ` Jan Beulich
2020-12-08 16:49 ` Oleksandr
2020-12-09 8:21 ` Jan Beulich
2020-11-30 10:31 ` [PATCH V3 21/23] xen/arm: Add mapcache invalidation handling Oleksandr Tyshchenko
2020-12-10 2:30 ` Stefano Stabellini
2020-12-10 18:50 ` Julien Grall
2020-12-11 1:28 ` Stefano Stabellini
2020-12-11 11:21 ` Oleksandr
2020-12-11 19:07 ` Stefano Stabellini
2020-12-11 19:37 ` Julien Grall
2020-12-11 19:27 ` Julien Grall
2020-11-30 10:31 ` [PATCH V3 22/23] libxl: Introduce basic virtio-mmio support on Arm Oleksandr Tyshchenko
2020-11-30 10:31 ` [PATCH V3 23/23] [RFC] libxl: Add support for virtio-disk configuration Oleksandr Tyshchenko
2020-11-30 11:22 ` [PATCH V3 00/23] IOREQ feature (+ virtio-mmio) on Arm Oleksandr
2020-12-07 13:03 ` Wei Chen
2020-12-07 21:03 ` Oleksandr
2020-11-30 16:21 ` Alex Bennée
2020-11-30 22:22 ` Oleksandr [this message]
2020-12-29 15:32 ` Re: Roger Pau Monné
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=248003d5-87a2-a7b2-5b30-e94c2a49945b@gmail.com \
--to=olekstysh@gmail.com \
--cc=Kaly.Xin@arm.com \
--cc=Volodymyr_Babchuk@epam.com \
--cc=Wei.Chen@arm.com \
--cc=alex.bennee@linaro.org \
--cc=andrew.cooper3@citrix.com \
--cc=anthony.perard@citrix.com \
--cc=bertrand.marquis@arm.com \
--cc=dgdegra@tycho.nsa.gov \
--cc=george.dunlap@citrix.com \
--cc=iwj@xenproject.org \
--cc=jbeulich@suse.com \
--cc=joculator@gmail.com \
--cc=julien.grall@arm.com \
--cc=julien@xen.org \
--cc=jun.nakajima@intel.com \
--cc=kevin.tian@intel.com \
--cc=oleksandr_tyshchenko@epam.com \
--cc=paul@xen.org \
--cc=roger.pau@citrix.com \
--cc=sstabellini@kernel.org \
--cc=tim@xen.org \
--cc=wl@xen.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).