From: David Gibson <david@gibson.dropbear.id.au>
To: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Alexey Kardashevskiy <aik@au1.ibm.com>,
Leon Romanovsky <leon@kernel.org>,
saeedm@mellanox.com, linux-rdma@vger.kernel.org,
netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
sbest@redhat.com, Jason Gunthorpe <jgg@ziepe.ca>,
alex.williamson@redhat.com, paulus@samba.org,
linux-pci@vger.kernel.org, bhelgaas@google.com,
ogerlitz@mellanox.com, linuxppc-dev@lists.ozlabs.org,
davem@davemloft.net, tariqt@mellanox.com
Subject: Re: [PATCH] PCI: Add no-D3 quirk for Mellanox ConnectX-[45]
Date: Wed, 9 Jan 2019 16:30:45 +1100 [thread overview]
Message-ID: <20190109053045.GE6682@umbus.fritz.box> (raw)
In-Reply-To: <012d24d58a542ed44c8af9f517f1bd61ab912037.camel@kernel.crashing.org>
[-- Attachment #1: Type: text/plain, Size: 2493 bytes --]
On Wed, Jan 09, 2019 at 04:09:02PM +1100, Benjamin Herrenschmidt wrote:
> On Mon, 2019-01-07 at 21:01 -0700, Jason Gunthorpe wrote:
> >
> > > In a very cryptic way that requires manual parsing using non-public
> > > docs sadly but yes. From the look of it, it's a completion timeout.
> > >
> > > Looks to me like we don't get a response to a config space access
> > > during the change of D state. I don't know if it's the write of the D3
> > > state itself or the read back though (it's probably detected on the
> > > read back or a subsequent read, but that doesn't tell me which specific
> > > one failed).
> >
> > If it is just one card doing it (again, check you have latest
> > firmware) I wonder if it is a sketchy PCI-E electrical link that is
> > causing a long re-training cycle? Can you tell if the PCI-E link is
> > permanently gone or does it eventually return?
>
> No, it's 100% reproducable on systems with that specific card model,
> not card instance, and maybe different systems/cards as well, I'll let
> David & Alexey comment further on that.
Well, it's 100% reproducable on a particular model of system
(garrison) with a particular model of card. I've had some suggestions
that it fails with some other systems card card models, but nothing
confirmed - the one other system model I've been able to try, which
also had a newer card model didn't reproduce the problem.
> > Does the card work in Gen 3 when it starts? Is there any indication of
> > PCI-E link errors?
>
> Nope.
>
> > Everytime or sometimes?
> >
> > POWER 8 firmware is good? If the link does eventually come back, is
> > the POWER8's D3 resumption timeout long enough?
> >
> > If this doesn't lead to an obvious conclusion you'll probably need to
> > connect to IBM's Mellanox support team to get more information from
> > the card side.
>
> We are IBM :-) So far, it seems to be that the card is doing something
> not quite right, but we don't know what. We might need to engage
> Mellanox themselves.
Possibly. On the other hand, I've had it reported that this is a
software regression at least with downstream red hat kernels. I
haven't yet been able to eliminate factors that might be confusing
that, or try to find a working version upstream.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
next prev parent reply other threads:[~2019-01-09 5:47 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-12-06 4:19 [PATCH] PCI: Add no-D3 quirk for Mellanox ConnectX-[45] David Gibson
2018-12-06 6:45 ` Leon Romanovsky
2018-12-11 2:31 ` David Gibson
2019-01-04 3:44 ` David Gibson
2019-01-05 17:51 ` Jason Gunthorpe
2019-01-05 22:43 ` Benjamin Herrenschmidt
2019-01-08 4:01 ` Jason Gunthorpe
2019-01-08 6:07 ` Leon Romanovsky
2019-01-09 5:09 ` Benjamin Herrenschmidt
2019-01-09 5:30 ` David Gibson [this message]
2019-01-09 6:32 ` Alexey Kardashevskiy
2019-01-09 7:25 ` Benjamin Herrenschmidt
2019-01-09 8:14 ` Alexey Kardashevskiy
2019-01-09 15:27 ` Jason Gunthorpe
2019-01-09 4:53 ` Alexey Kardashevskiy
2019-01-09 7:24 ` Benjamin Herrenschmidt
2019-01-09 8:20 ` Alexey Kardashevskiy
2018-12-11 14:01 ` Bjorn Helgaas
2018-12-12 0:22 ` David Gibson
2018-12-12 3:04 ` Bjorn Helgaas
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190109053045.GE6682@umbus.fritz.box \
--to=david@gibson.dropbear.id.au \
--cc=aik@au1.ibm.com \
--cc=alex.williamson@redhat.com \
--cc=benh@kernel.crashing.org \
--cc=bhelgaas@google.com \
--cc=davem@davemloft.net \
--cc=jgg@ziepe.ca \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=netdev@vger.kernel.org \
--cc=ogerlitz@mellanox.com \
--cc=paulus@samba.org \
--cc=saeedm@mellanox.com \
--cc=sbest@redhat.com \
--cc=tariqt@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).