From: Boris Ostrovsky <boris.ostrovsky@oracle.com>
To: Dan Streetman <ddstreet@ieee.org>,
Stefano Stabellini <sstabellini@kernel.org>,
jgross@suse.com, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Dan Streetman <dan.streetman@canonical.com>,
Bjorn Helgaas <bhelgaas@google.com>,
xen-devel@lists.xenproject.org,
linux-kernel <linux-kernel@vger.kernel.org>,
linux-pci@vger.kernel.org
Subject: Re: [PATCH] xen: do not re-use pirq number cached in pci device msi msg data
Date: Fri, 13 Jan 2017 16:49:29 -0500 [thread overview]
Message-ID: <20d6b3fa-5a3c-a1b7-6d56-b66a9cb90b69@oracle.com> (raw)
In-Reply-To: <20170113200751.20125-1-ddstreet@ieee.org>
On 01/13/2017 03:07 PM, Dan Streetman wrote:
> Revert the main part of commit:
> af42b8d12f8a ("xen: fix MSI setup and teardown for PV on HVM guests")
>
> That commit introduced reading the pci device's msi message data to see
> if a pirq was previously configured for the device's msi/msix, and re-use
> that pirq. At the time, that was the correct behavior. However, a
> later change to Qemu caused it to call into the Xen hypervisor to unmap
> all pirqs for a pci device, when the pci device disables its MSI/MSIX
> vectors; specifically the Qemu commit:
> c976437c7dba9c7444fb41df45468968aaa326ad
> ("qemu-xen: free all the pirqs for msi/msix when driver unload")
>
> Once Qemu added this pirq unmapping, it was no longer correct for the
> kernel to re-use the pirq number cached in the pci device msi message
> data. All Qemu releases since 2.1.0 contain the patch that unmaps the
> pirqs when the pci device disables its MSI/MSIX vectors.
>
> This bug is causing failures to initialize multiple NVMe controllers
> under Xen, because the NVMe driver sets up a single MSIX vector for
> each controller (concurrently), and then after using that to talk to
> the controller for some configuration data, it disables the single MSIX
> vector and re-configures all the MSIX vectors it needs. So the MSIX
> setup code tries to re-use the cached pirq from the first vector
> for each controller, but the hypervisor has already given away that
> pirq to another controller, and its initialization fails.
>
> This is discussed in more detail at:
> https://lists.xen.org/archives/html/xen-devel/2017-01/msg00447.html
>
> Fixes: af42b8d12f8a ("xen: fix MSI setup and teardown for PV on HVM guests")
> Signed-off-by: Dan Streetman <dan.streetman@canonical.com>
Reviewed-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
> ---
> arch/x86/pci/xen.c | 23 +++++++----------------
> 1 file changed, 7 insertions(+), 16 deletions(-)
>
> diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c
> index bedfab9..a00a6c0 100644
> --- a/arch/x86/pci/xen.c
> +++ b/arch/x86/pci/xen.c
> @@ -234,23 +234,14 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
> return 1;
>
> for_each_pci_msi_entry(msidesc, dev) {
> - __pci_read_msi_msg(msidesc, &msg);
> - pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) |
> - ((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff);
> - if (msg.data != XEN_PIRQ_MSI_DATA ||
> - xen_irq_from_pirq(pirq) < 0) {
> - pirq = xen_allocate_pirq_msi(dev, msidesc);
> - if (pirq < 0) {
> - irq = -ENODEV;
> - goto error;
> - }
> - xen_msi_compose_msg(dev, pirq, &msg);
> - __pci_write_msi_msg(msidesc, &msg);
> - dev_dbg(&dev->dev, "xen: msi bound to pirq=%d\n", pirq);
> - } else {
> - dev_dbg(&dev->dev,
> - "xen: msi already bound to pirq=%d\n", pirq);
> + pirq = xen_allocate_pirq_msi(dev, msidesc);
> + if (pirq < 0) {
> + irq = -ENODEV;
> + goto error;
> }
> + xen_msi_compose_msg(dev, pirq, &msg);
> + __pci_write_msi_msg(msidesc, &msg);
> + dev_dbg(&dev->dev, "xen: msi bound to pirq=%d\n", pirq);
> irq = xen_bind_pirq_msi_to_irq(dev, msidesc, pirq,
> (type == PCI_CAP_ID_MSI) ? nvec : 1,
> (type == PCI_CAP_ID_MSIX) ?
next prev parent reply other threads:[~2017-01-13 21:49 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-01-05 19:28 [PATCH] xen: do not re-use pirq number cached in pci device msi msg data Dan Streetman
2017-01-07 1:06 ` Konrad Rzeszutek Wilk
2017-01-09 14:59 ` Boris Ostrovsky
2017-01-09 15:42 ` [Xen-devel] " Dan Streetman
2017-01-09 15:59 ` Konrad Rzeszutek Wilk
2017-01-09 19:30 ` Stefano Stabellini
2017-01-10 15:57 ` Dan Streetman
2017-01-10 18:41 ` Dan Streetman
2017-01-10 19:03 ` Stefano Stabellini
2017-01-10 21:32 ` Dan Streetman
2017-01-10 23:28 ` Boris Ostrovsky
2017-01-11 1:25 ` Stefano Stabellini
2017-01-11 15:26 ` Dan Streetman
2017-01-11 18:46 ` Stefano Stabellini
2017-01-11 23:25 ` Dan Streetman
2017-01-13 18:31 ` Dan Streetman
2017-01-13 18:44 ` Stefano Stabellini
2017-01-13 20:00 ` Boris Ostrovsky
2017-01-13 20:07 ` Dan Streetman
2017-01-13 20:54 ` Stefano Stabellini
2017-01-13 21:49 ` Boris Ostrovsky [this message]
2017-01-13 22:30 ` Konrad Rzeszutek Wilk
2017-02-21 15:31 ` Dan Streetman
2017-02-21 15:45 ` Juergen Gross
2017-02-21 15:58 ` Boris Ostrovsky
2017-02-22 14:28 ` Bjorn Helgaas
2017-02-22 15:14 ` Boris Ostrovsky
2017-05-03 18:19 ` [Xen-devel] " David Woodhouse
2017-05-03 18:43 ` Boris Ostrovsky
2017-05-03 22:59 ` Stefano Stabellini
2017-05-03 23:06 ` Greg KH
2017-05-03 23:12 ` Stefano Stabellini
2017-05-03 23:19 ` Greg KH
2017-01-13 20:13 ` Dan Streetman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20d6b3fa-5a3c-a1b7-6d56-b66a9cb90b69@oracle.com \
--to=boris.ostrovsky@oracle.com \
--cc=bhelgaas@google.com \
--cc=dan.streetman@canonical.com \
--cc=ddstreet@ieee.org \
--cc=jgross@suse.com \
--cc=konrad.wilk@oracle.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=sstabellini@kernel.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).