From: David Vrabel <david.vrabel@citrix.com>
To: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Stefano Stabellini <stefano.stabellini@eu.citrix.com>,
"xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>,
Feng Jin <joe.jin@oracle.com>,
Zhenzhong Duan <zhenzhong.duan@oracle.com>,
Yuval Shaia <yuval.shaia@oracle.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
Chien Yen <chien.yen@oracle.com>, Ingo Molnar <mingo@redhat.com>,
"H. Peter Anvin" <hpa@zytor.com>,
Thomas Gleixner <tglx@linutronix.de>
Subject: Re: [Xen-devel] [PATCH] xen: reuse the same pirq allocated when driver load first time
Date: Tue, 21 May 2013 11:07:10 +0100 [thread overview]
Message-ID: <519B474E.4000202@citrix.com> (raw)
In-Reply-To: <20130520203855.GA30616@phenom.dumpdata.com>
On 20/05/13 21:38, Konrad Rzeszutek Wilk wrote:
>> At this point I think that upstream option is to save the PIRQ value and re-use it.
>> Will post a patch for it.
>
> Here is the patch. It works for me when passing in a NIC driver.
>
>>From 509499568d1cdf1f2a3fb53773c991f4b063eb56 Mon Sep 17 00:00:00 2001
> From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> Date: Mon, 20 May 2013 16:08:16 -0400
> Subject: [PATCH] xen/pci: Track PVHVM PIRQs.
>
> The PIRQs that the hypervisor provides for the guest are a limited
> resource. They are acquired via PHYSDEVOP_get_free_pirq and in
> theory should be returned back to the hypervisor via PHYSDEVOP_unmap_pirq
> hypercall. Unfortunatly that is not the case.
>
> This means that if there is a PCI device that has been passed in
> the guest and does a loop of 'rmmod <driver>;modprobe <driver>"
> we end up exhausting all of the PIRQs that are available.
>
> For example (with kernel built as debug), we get this:
> 00:05.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
> [ 152.659396] e1000e 0000:00:05.0: xen: msi bound to pirq=53
> [ 152.665856] e1000e 0000:00:05.0: xen: msi --> pirq=53 --> irq=73
> .. snip
> [ 188.276835] e1000e 0000:00:05.0: xen: msi bound to pirq=51
> [ 188.283194] e1000e 0000:00:05.0: xen: msi --> pirq=51 --> irq=73
>
> .. and so on, until the pirq value is zero. This is an acute problem
> when many PCI devices with many MSI-X entries are passed in the guest.
>
> There is an alternative solution where we assume that on PCI
> initialization (so when user passes in the PCI device) QEMU will init
> the MSI and MSI-X entries to zero. Based on that assumptions and
> that the Linux MSI API will write the PIRQ value to the MSI/MSI-X
> (and used by QEMU through the life-cycle of the PCI device), we can
> also depend on that. That means if MSI (or MSI-X entries) are read back
> and are not 0, we can re-use that PIRQ value. However this patch
> guards against future changes in QEMU in case that assumption
> is incorrect.
>
> Reported-by: Zhenzhong Duan <zhenzhong.duan@oracle.com>
> CC: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
> drivers/xen/events.c | 124 +++++++++++++++++++++++++++++++++++++++++++++++++-
> 1 files changed, 122 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/xen/events.c b/drivers/xen/events.c
> index 6a6bbe4..8aae21a 100644
> --- a/drivers/xen/events.c
> +++ b/drivers/xen/events.c
> @@ -112,6 +112,27 @@ struct irq_info {
> #define PIRQ_NEEDS_EOI (1 << 0)
> #define PIRQ_SHAREABLE (1 << 1)
>
> +/*
> + * The PHYSDEVOP_get_free_pirq allocates a set of PIRQs for the guest and
> + * the PHYSDEVOP_unmap_pirq is suppose to return them to the hypervisor.
> + * Unfortunatly that is not the case and we exhaust all of the PIRQs that are
> + * allocated for the domain if a driver is loaded/unloaded in a loop.
> + * The pirq_info serves a cache of the allocated PIRQs so that we can reuse
> + * for drivers. Note, it is only used by the MSI, MSI-X routines.
> + */
Ick. Let's fix the bug in the hypervisor instead of hacking up the
kernel like this.
Looking at the hypervisor code I couldn't see anything obviously wrong.
I do note that Xen doesn't free the pirq until it has been unbound by
the guest. Xen will warn if the guest unmaps a pirq that is still bound
("domD: forcing unbind of pirq P"). Is this what is happening? If so,
that would suggest a bug in the guest rather than the hypervisor.
David
next prev parent reply other threads:[~2013-05-21 10:07 UTC|newest]
Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-05-08 8:18 [PATCH] xen: reuse the same pirq allocated when driver load first time Zhenzhong Duan
2013-05-10 18:53 ` Konrad Rzeszutek Wilk
2013-05-13 7:44 ` Zhenzhong Duan
2013-05-13 11:06 ` Stefano Stabellini
2013-05-13 14:07 ` Konrad Rzeszutek Wilk
2013-05-13 14:50 ` Stefano Stabellini
2013-05-13 16:17 ` Konrad Rzeszutek Wilk
2013-05-13 17:24 ` Stefano Stabellini
2013-05-13 18:20 ` Konrad Rzeszutek Wilk
2013-05-14 13:49 ` Stefano Stabellini
2013-05-14 14:20 ` Konrad Rzeszutek Wilk
2013-05-15 9:41 ` Stefano Stabellini
2013-05-15 14:18 ` Zhenzhong Duan
2013-05-17 2:22 ` Zhenzhong Duan
2013-05-20 10:24 ` Stefano Stabellini
2013-05-20 15:24 ` Konrad Rzeszutek Wilk
2013-05-20 17:57 ` Konrad Rzeszutek Wilk
2013-05-20 20:38 ` Konrad Rzeszutek Wilk
2013-05-21 10:07 ` David Vrabel [this message]
2013-05-21 13:40 ` [Xen-devel] " Konrad Rzeszutek Wilk
2013-05-21 16:51 ` Stefano Stabellini
2013-05-21 20:42 ` Konrad Rzeszutek Wilk
2013-05-21 21:50 ` Stefano Stabellini
2013-05-21 22:41 ` Konrad Rzeszutek Wilk
2013-05-22 9:37 ` Jan Beulich
2013-05-22 15:14 ` Konrad Rzeszutek Wilk
2013-05-22 15:25 ` Jan Beulich
2013-05-22 16:41 ` Konrad Rzeszutek Wilk
2013-05-23 6:31 ` Jan Beulich
2013-05-29 17:50 ` Stefano Stabellini
2013-05-30 17:48 ` Konrad Rzeszutek Wilk
[not found] ` <51AECC3A.7060803@oracle.com>
2013-06-05 12:50 ` Stefano Stabellini
2013-06-20 2:57 ` Zhenzhong Duan
2013-06-20 14:21 ` Stefano Stabellini
2013-06-24 7:19 ` Zhenzhong Duan
2013-06-24 17:18 ` Stefano Stabellini
2013-06-25 5:33 ` DuanZhenzhong
2013-06-25 17:51 ` Stefano Stabellini
2013-06-26 4:00 ` Zhenzhong Duan
2013-06-26 18:08 ` Stefano Stabellini
2013-06-27 4:01 ` Zhenzhong Duan
2013-06-27 11:52 ` Stefano Stabellini
2013-06-28 2:33 ` Zhenzhong Duan
2013-06-28 11:12 ` Stefano Stabellini
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=519B474E.4000202@citrix.com \
--to=david.vrabel@citrix.com \
--cc=chien.yen@oracle.com \
--cc=hpa@zytor.com \
--cc=joe.jin@oracle.com \
--cc=konrad.wilk@oracle.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=stefano.stabellini@eu.citrix.com \
--cc=tglx@linutronix.de \
--cc=xen-devel@lists.xensource.com \
--cc=yuval.shaia@oracle.com \
--cc=zhenzhong.duan@oracle.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).