From: Oleksandr <olekstysh@gmail.com>
To: Julien Grall <julien@xen.org>, paul@xen.org
Cc: 'Jan Beulich' <jbeulich@suse.com>,
'Oleksandr Tyshchenko' <oleksandr_tyshchenko@epam.com>,
'Stefano Stabellini' <sstabellini@kernel.org>,
'Volodymyr Babchuk' <Volodymyr_Babchuk@epam.com>,
'Andrew Cooper' <andrew.cooper3@citrix.com>,
'George Dunlap' <george.dunlap@citrix.com>,
'Ian Jackson' <iwj@xenproject.org>, 'Wei Liu' <wl@xen.org>,
'Julien Grall' <julien.grall@arm.com>,
xen-devel@lists.xenproject.org
Subject: Re: [PATCH V3 17/23] xen/ioreq: Introduce domain_has_ioreq_server()
Date: Wed, 9 Dec 2020 23:05:02 +0200 [thread overview]
Message-ID: <dfe1f85a-6293-5c5e-ad33-4367f83a5c60@gmail.com> (raw)
In-Reply-To: <84d7238d-0ec1-acdd-6cea-db78aba6f3d7@xen.org>
On 09.12.20 20:58, Julien Grall wrote:
> Hi Oleksandr and Paul,
Hi Julien, Paul.
>
> Sorry for jumping late in the conversation.
>
> On 09/12/2020 09:01, Paul Durrant wrote:
>>> -----Original Message-----
>>> From: Oleksandr <olekstysh@gmail.com>
>>> Sent: 08 December 2020 20:17
>>> To: paul@xen.org
>>> Cc: 'Jan Beulich' <jbeulich@suse.com>; 'Oleksandr Tyshchenko'
>>> <oleksandr_tyshchenko@epam.com>;
>>> 'Stefano Stabellini' <sstabellini@kernel.org>; 'Julien Grall'
>>> <julien@xen.org>; 'Volodymyr Babchuk'
>>> <Volodymyr_Babchuk@epam.com>; 'Andrew Cooper'
>>> <andrew.cooper3@citrix.com>; 'George Dunlap'
>>> <george.dunlap@citrix.com>; 'Ian Jackson' <iwj@xenproject.org>; 'Wei
>>> Liu' <wl@xen.org>; 'Julien Grall'
>>> <julien.grall@arm.com>; xen-devel@lists.xenproject.org
>>> Subject: Re: [PATCH V3 17/23] xen/ioreq: Introduce
>>> domain_has_ioreq_server()
>>>
>>>
>>> On 08.12.20 21:43, Paul Durrant wrote:
>>>
>>> Hi Paul
>>>
>>>>> -----Original Message-----
>>>>> From: Oleksandr <olekstysh@gmail.com>
>>>>> Sent: 08 December 2020 16:57
>>>>> To: Paul Durrant <paul@xen.org>
>>>>> Cc: Jan Beulich <jbeulich@suse.com>; Oleksandr Tyshchenko
>>>>> <oleksandr_tyshchenko@epam.com>; Stefano
>>>>> Stabellini <sstabellini@kernel.org>; Julien Grall
>>>>> <julien@xen.org>; Volodymyr Babchuk
>>>>> <Volodymyr_Babchuk@epam.com>; Andrew Cooper
>>>>> <andrew.cooper3@citrix.com>; George Dunlap
>>>>> <george.dunlap@citrix.com>; Ian Jackson <iwj@xenproject.org>; Wei
>>>>> Liu <wl@xen.org>; Julien Grall
>>>>> <julien.grall@arm.com>; xen-devel@lists.xenproject.org
>>>>> Subject: Re: [PATCH V3 17/23] xen/ioreq: Introduce
>>>>> domain_has_ioreq_server()
>>>>>
>>>>>
>>>>> Hi Paul.
>>>>>
>>>>>
>>>>> On 08.12.20 17:33, Oleksandr wrote:
>>>>>> On 08.12.20 17:11, Jan Beulich wrote:
>>>>>>
>>>>>> Hi Jan
>>>>>>
>>>>>>> On 30.11.2020 11:31, Oleksandr Tyshchenko wrote:
>>>>>>>> --- a/xen/include/xen/ioreq.h
>>>>>>>> +++ b/xen/include/xen/ioreq.h
>>>>>>>> @@ -55,6 +55,20 @@ struct ioreq_server {
>>>>>>>> uint8_t bufioreq_handling;
>>>>>>>> };
>>>>>>>> +/*
>>>>>>>> + * This should only be used when d == current->domain and it's
>>>>>>>> not
>>>>>>>> paused,
>>>>>>> Is the "not paused" part really relevant here? Besides it being
>>>>>>> rare
>>>>>>> that the current domain would be paused (if so, it's in the process
>>>>>>> of having all its vCPU-s scheduled out), does this matter at all?do
>>>>>>> any extra actionsdo any extra actions
>>>>>> No, it isn't relevant, I will drop it.
>>>>>>
>>>>>>
>>>>>>> Apart from this the patch looks okay to me, but I'm not sure it
>>>>>>> addresses Paul's concerns. Iirc he had suggested to switch back to
>>>>>>> a list if doing a swipe over the entire array is too expensive in
>>>>>>> this specific case.
>>>>>> We would like to avoid to do any extra actions in
>>>>>> leave_hypervisor_to_guest() if possible.
>>>>>> But not only there, the logic whether we check/set
>>>>>> mapcache_invalidation variable could be avoided if a domain doesn't
>>>>>> use IOREQ server...
>>>>>
>>>>> Are you OK with this patch (common part of it)?
>>>> How much of a performance benefit is this? The array is small to
>>>> simply counting the non-NULL
>>> entries should be pretty quick.
>>> I didn't perform performance measurements on how much this call
>>> consumes.
>>> In our system we run three domains. The emulator is in DomD only, so I
>>> would like to avoid to call vcpu_ioreq_handle_completion() for every
>>> Dom0/DomU's vCPUs
>>> if there is no real need to do it.
>>
>> This is not relevant to the domain that the emulator is running in;
>> it's concerning the domains which the emulator is servicing. How many
>> of those are there?
>
> AFAICT, the maximum number of IOREQ servers is 8 today.
>
>>
>>> On Arm vcpu_ioreq_handle_completion()
>>> is called with IRQ enabled, so the call is accompanied with
>>> corresponding irq_enable/irq_disable.
>>> These unneeded actions could be avoided by using this simple one-line
>>> helper...
>>>
>>
>> The helper may be one line but there is more to the patch than that.
>> I still think you could just walk the array in the helper rather than
>> keeping a running occupancy count.
>
> Right, the concern here is this function will be called in an hotpath
> (everytime we are re-entering to the guest). At the difference of x86,
> the entry/exit code is really small, so any additional code will have
> an impact on the overall performance.
+1
>
>
> That said, the IOREQ code is a tech preview for Arm. So I would be
> fine going with Paul's approach until we have a better understanding
> on the performance of virtio/IOREQ.
I am fine with Paul's approach for now (I only need a confirmation that
I got it correctly).
>
>
> I am going to throw some more thoughts about the optimization here.
> The patch is focusing on performance impact when IOREQ is built-in and
> not used.
It is true, what I would to add here is the helper also avoids
unnecessary vcpu_ioreq_handle_completion() calls as well another
unnecessary action
(mapcache handling logic, although it is not a hotpath) in subsequent
patch when IOREQ is used.
> I think we can do further optimization (which may superseed this one).
>
> get_pending_vcpu() (called from handle_hvm_io_completion()) is overly
> expensive in particular if you have no I/O forwarded to an IOREQ
> server. Entry to the hypervisor can happen for many reasons
> (interrupts, system registers emulation, I/O emulation...) and the I/O
> forwarded should be a small subset.
>
> Ideally, handle_hvm_io_completion() should be a NOP (at max a few
> instructions) if there are nothing to do. Maybe we want to introduce a
> per-vCPU flag indicating if an I/O has been forwarded to an IOREQ server.
>
> This would also us to bypass most of the function if there is nothing
> to do.
>
> Any thoughts?
>
> In any case this is more a forward looking rather than a request for
> the current series. What matters to me is we have a functional (not
> necessarily optimized) version of IOREQ in Xen 4.15. This would be a
> great step towards using Virto on Xen.
Completely agree, current series is quite big) and if we will try to
make it perfect I am afraid, we won't have it even in Xen 4.16). As for
proposed optimization - I think it worth considering, I will mention
about it in the cover letter for the series among other possible things
such as buffered request, etc.
>
>
> Cheers,
>
--
Regards,
Oleksandr Tyshchenko
next prev parent reply other threads:[~2020-12-09 21:05 UTC|newest]
Thread overview: 127+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-30 10:31 Oleksandr Tyshchenko
2020-11-30 10:31 ` [PATCH V3 01/23] x86/ioreq: Prepare IOREQ feature for making it common Oleksandr Tyshchenko
2020-12-01 11:03 ` Alex Bennée
2020-12-01 18:53 ` Oleksandr
2020-12-01 19:36 ` Alex Bennée
2020-12-02 8:00 ` Jan Beulich
2020-12-02 11:19 ` Oleksandr
2020-12-07 11:13 ` Jan Beulich
2020-12-07 15:27 ` Oleksandr
2020-12-07 16:29 ` Jan Beulich
2020-12-07 17:21 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 02/23] x86/ioreq: Add IOREQ_STATUS_* #define-s and update code for moving Oleksandr Tyshchenko
2020-12-01 11:07 ` Alex Bennée
2020-12-07 11:19 ` Jan Beulich
2020-12-07 15:37 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 03/23] x86/ioreq: Provide out-of-line wrapper for the handle_mmio() Oleksandr Tyshchenko
2020-12-07 11:27 ` Jan Beulich
2020-12-07 15:39 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 04/23] xen/ioreq: Make x86's IOREQ feature common Oleksandr Tyshchenko
2020-12-07 11:41 ` Jan Beulich
2020-12-07 19:43 ` Oleksandr
2020-12-08 9:21 ` Jan Beulich
2020-12-08 13:56 ` Oleksandr
2020-12-08 15:02 ` Jan Beulich
2020-12-08 17:24 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 05/23] xen/ioreq: Make x86's hvm_ioreq_needs_completion() common Oleksandr Tyshchenko
2020-12-07 11:47 ` Jan Beulich
2020-11-30 10:31 ` [PATCH V3 06/23] xen/ioreq: Make x86's hvm_mmio_first(last)_byte() common Oleksandr Tyshchenko
2020-12-07 11:48 ` Jan Beulich
2020-11-30 10:31 ` [PATCH V3 07/23] xen/ioreq: Make x86's hvm_ioreq_(page/vcpu/server) structs common Oleksandr Tyshchenko
2020-12-07 11:54 ` Jan Beulich
2020-11-30 10:31 ` [PATCH V3 08/23] xen/ioreq: Move x86's ioreq_server to struct domain Oleksandr Tyshchenko
2020-12-07 12:04 ` Jan Beulich
2020-12-07 12:12 ` Paul Durrant
2020-12-07 19:52 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 09/23] xen/dm: Make x86's DM feature common Oleksandr Tyshchenko
2020-12-07 12:08 ` Jan Beulich
2020-12-07 20:23 ` Oleksandr
2020-12-08 9:30 ` Jan Beulich
2020-12-08 14:54 ` Oleksandr
2021-01-07 14:38 ` Oleksandr
2021-01-07 15:01 ` Jan Beulich
2021-01-07 16:49 ` Oleksandr
2021-01-12 22:23 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 10/23] xen/mm: Make x86's XENMEM_resource_ioreq_server handling common Oleksandr Tyshchenko
2020-12-07 11:35 ` Jan Beulich
2020-12-07 12:11 ` Jan Beulich
2020-12-07 21:06 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 11/23] xen/ioreq: Move x86's io_completion/io_req fields to struct vcpu Oleksandr Tyshchenko
2020-12-07 12:32 ` Jan Beulich
2020-12-07 20:59 ` Oleksandr
2020-12-08 7:52 ` Paul Durrant
2020-12-08 9:35 ` Jan Beulich
2020-12-08 18:21 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 12/23] xen/ioreq: Remove "hvm" prefixes from involved function names Oleksandr Tyshchenko
2020-12-07 12:45 ` Jan Beulich
2020-12-07 20:28 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 13/23] xen/ioreq: Use guest_cmpxchg64() instead of cmpxchg() Oleksandr Tyshchenko
2020-12-09 21:32 ` Stefano Stabellini
2020-12-09 22:34 ` Oleksandr
2020-12-10 2:30 ` Stefano Stabellini
2020-11-30 10:31 ` [PATCH V3 14/23] arm/ioreq: Introduce arch specific bits for IOREQ/DM features Oleksandr Tyshchenko
2020-12-09 22:04 ` Stefano Stabellini
2020-12-09 22:49 ` Oleksandr
2020-12-10 2:30 ` Stefano Stabellini
2020-11-30 10:31 ` [PATCH V3 15/23] xen/arm: Stick around in leave_hypervisor_to_guest until I/O has completed Oleksandr Tyshchenko
2020-11-30 20:51 ` Volodymyr Babchuk
2020-12-01 12:46 ` Julien Grall
2020-12-09 23:18 ` Stefano Stabellini
2020-12-09 23:35 ` Stefano Stabellini
2020-12-09 23:47 ` Julien Grall
2020-12-10 2:30 ` Stefano Stabellini
2020-12-10 13:17 ` Julien Grall
2020-12-10 13:21 ` Oleksandr
2020-12-09 23:38 ` Julien Grall
2020-11-30 10:31 ` [PATCH V3 16/23] xen/mm: Handle properly reference in set_foreign_p2m_entry() on Arm Oleksandr Tyshchenko
2020-12-08 14:24 ` Jan Beulich
2020-12-08 16:41 ` Oleksandr
2020-12-09 23:49 ` Stefano Stabellini
2021-01-15 1:18 ` Stefano Stabellini
2020-11-30 10:31 ` [PATCH V3 17/23] xen/ioreq: Introduce domain_has_ioreq_server() Oleksandr Tyshchenko
2020-12-08 15:11 ` Jan Beulich
2020-12-08 15:33 ` Oleksandr
2020-12-08 16:56 ` Oleksandr
2020-12-08 19:43 ` Paul Durrant
2020-12-08 20:16 ` Oleksandr
2020-12-09 9:01 ` Paul Durrant
2020-12-09 18:58 ` Julien Grall
2020-12-09 21:05 ` Oleksandr [this message]
2020-12-09 20:36 ` Oleksandr
2020-12-10 8:38 ` Paul Durrant
2020-12-10 16:57 ` Oleksandr
2020-11-30 10:31 ` [PATCH V3 18/23] xen/dm: Introduce xendevicemodel_set_irq_level DM op Oleksandr Tyshchenko
2020-12-10 2:21 ` Stefano Stabellini
2020-12-10 12:58 ` Oleksandr
2020-12-10 13:38 ` Julien Grall
2020-11-30 10:31 ` [PATCH V3 19/23] xen/arm: io: Abstract sign-extension Oleksandr Tyshchenko
2020-11-30 21:03 ` Volodymyr Babchuk
2020-11-30 23:27 ` Oleksandr
2020-12-01 7:55 ` Jan Beulich
2020-12-01 10:30 ` Julien Grall
2020-12-01 10:42 ` Oleksandr
2020-12-01 12:13 ` Julien Grall
2020-12-01 12:24 ` Oleksandr
2020-12-01 12:28 ` Julien Grall
2020-12-01 10:49 ` Jan Beulich
2020-12-01 10:23 ` Julien Grall
2020-11-30 10:31 ` [PATCH V3 20/23] xen/ioreq: Make x86's send_invalidate_req() common Oleksandr Tyshchenko
2020-12-08 15:24 ` Jan Beulich
2020-12-08 16:49 ` Oleksandr
2020-12-09 8:21 ` Jan Beulich
2020-11-30 10:31 ` [PATCH V3 21/23] xen/arm: Add mapcache invalidation handling Oleksandr Tyshchenko
2020-12-10 2:30 ` Stefano Stabellini
2020-12-10 18:50 ` Julien Grall
2020-12-11 1:28 ` Stefano Stabellini
2020-12-11 11:21 ` Oleksandr
2020-12-11 19:07 ` Stefano Stabellini
2020-12-11 19:37 ` Julien Grall
2020-12-11 19:27 ` Julien Grall
2020-11-30 10:31 ` [PATCH V3 22/23] libxl: Introduce basic virtio-mmio support on Arm Oleksandr Tyshchenko
2020-11-30 10:31 ` [PATCH V3 23/23] [RFC] libxl: Add support for virtio-disk configuration Oleksandr Tyshchenko
2020-11-30 11:22 ` [PATCH V3 00/23] IOREQ feature (+ virtio-mmio) on Arm Oleksandr
2020-12-07 13:03 ` Wei Chen
2020-12-07 21:03 ` Oleksandr
2020-11-30 16:21 ` Alex Bennée
2020-11-30 22:22 ` [PATCH V3 00/23] IOREQ feature (+ virtio-mmio) on Arm Oleksandr
2020-12-29 15:32 ` Roger Pau Monné
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=dfe1f85a-6293-5c5e-ad33-4367f83a5c60@gmail.com \
--to=olekstysh@gmail.com \
--cc=Volodymyr_Babchuk@epam.com \
--cc=andrew.cooper3@citrix.com \
--cc=george.dunlap@citrix.com \
--cc=iwj@xenproject.org \
--cc=jbeulich@suse.com \
--cc=julien.grall@arm.com \
--cc=julien@xen.org \
--cc=oleksandr_tyshchenko@epam.com \
--cc=paul@xen.org \
--cc=sstabellini@kernel.org \
--cc=wl@xen.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).