From: Tal Gilboa <talgi@mellanox.com>
To: Bjorn Helgaas <helgaas@kernel.org>,
Alexandru Gagniuc <mr.nuke.me@gmail.com>
Cc: bhelgaas@google.com, alex_gagniuc@dellteam.com,
austin_bolen@dell.com, shyam_iyer@dell.com,
keith.busch@intel.com, linux-pci@vger.kernel.org,
linux-kernel@vger.kernel.org,
Jeff Kirsher <jeffrey.t.kirsher@intel.com>,
Ariel Elior <ariel.elior@cavium.com>,
Michael Chan <michael.chan@broadcom.com>,
Ganesh Goudar <ganeshgr@chelsio.com>,
Tariq Toukan <tariqt@mellanox.com>,
Jakub Kicinski <jakub.kicinski@netronome.com>,
Dave Airlie <airlied@gmail.com>,
Alex Deucher <alexander.deucher@amd.com>
Subject: Re: [PATCH v3] PCI: Check for PCIe downtraining conditions
Date: Wed, 18 Jul 2018 09:38:10 -0400 [thread overview]
Message-ID: <628346bc-b455-05f2-2051-35a807076ea0@mellanox.com> (raw)
In-Reply-To: <20180716211706.GB12391@bhelgaas-glaptop.roam.corp.google.com>
On 7/16/2018 5:17 PM, Bjorn Helgaas wrote:
> [+cc maintainers of drivers that already use pcie_print_link_status()
> and GPU folks]
>
> On Mon, Jun 04, 2018 at 10:55:21AM -0500, Alexandru Gagniuc wrote:
>> PCIe downtraining happens when both the device and PCIe port are
>> capable of a larger bus width or higher speed than negotiated.
>> Downtraining might be indicative of other problems in the system, and
>> identifying this from userspace is neither intuitive, nor straigh
>> forward.
>
> s/straigh/straight/
> In this context, I think "straightforward" should be closed up
> (without the space).
>
>> The easiest way to detect this is with pcie_print_link_status(),
>> since the bottleneck is usually the link that is downtrained. It's not
>> a perfect solution, but it works extremely well in most cases.
>
> This is an interesting idea. I have two concerns:
>
> Some drivers already do this on their own, and we probably don't want
> duplicate output for those devices. In most cases (ixgbe and mlx* are
> exceptions), the drivers do this unconditionally so we *could* remove
> it from the driver if we add it to the core. The dmesg order would
> change, and the message wouldn't be associated with the driver as it
> now is.
>
> Also, I think some of the GPU devices might come up at a lower speed,
> then download firmware, then reset the device so it comes up at a
> higher speed. I think this patch will make us complain about about
> the low initial speed, which might confuse users.
>
> So I'm not sure whether it's better to do this in the core for all
> devices, or if we should just add it to the high-performance drivers
> that really care.
>
>> Signed-off-by: Alexandru Gagniuc <mr.nuke.me@gmail.com>
>> ---
>>
>> Changes since v2:
>> - Check dev->is_virtfn flag
>>
>> Changes since v1:
>> - Use pcie_print_link_status() instead of reimplementing logic
>>
>> drivers/pci/probe.c | 22 ++++++++++++++++++++++
>> 1 file changed, 22 insertions(+)
>>
>> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
>> index ac91b6fd0bcd..a88ec8c25dd5 100644
>> --- a/drivers/pci/probe.c
>> +++ b/drivers/pci/probe.c
>> @@ -2146,6 +2146,25 @@ static struct pci_dev *pci_scan_device(struct pci_bus *bus, int devfn)
>> return dev;
>> }
>>
>> +static void pcie_check_upstream_link(struct pci_dev *dev)
>> +{
>> +
>> + if (!pci_is_pcie(dev))
>> + return;
>> +
>> + /* Look from the device up to avoid downstream ports with no devices. */
>> + if ((pci_pcie_type(dev) != PCI_EXP_TYPE_ENDPOINT) &&
>> + (pci_pcie_type(dev) != PCI_EXP_TYPE_LEG_END) &&
>> + (pci_pcie_type(dev) != PCI_EXP_TYPE_UPSTREAM))
>> + return;
>
> Do we care about Upstream Ports here? I suspect that ultimately we
> only care about the bandwidth to Endpoints, and if an Endpoint is
> constrained by a slow link farther up the tree,
> pcie_print_link_status() is supposed to identify that slow link.
>
> I would find this test easier to read as
>
> if (!(type == PCI_EXP_TYPE_ENDPOINT || type == PCI_EXP_TYPE_LEG_END))
> return;
>
> But maybe I'm the only one that finds the conjunction of inequalities
> hard to read. No big deal either way.
>
>> + /* Multi-function PCIe share the same link/status. */
>> + if ((PCI_FUNC(dev->devfn) != 0) || dev->is_virtfn)
>> + return;
>> +
>> + pcie_print_link_status(dev);
>> +}
Is this function called by default for every PCIe device? What about
VFs? We make an exception for them on our driver since a VF doesn't have
access to the needed information in order to provide a meaningful message.
>> +
>> static void pci_init_capabilities(struct pci_dev *dev)
>> {
>> /* Enhanced Allocation */
>> @@ -2181,6 +2200,9 @@ static void pci_init_capabilities(struct pci_dev *dev)
>> /* Advanced Error Reporting */
>> pci_aer_init(dev);
>>
>> + /* Check link and detect downtrain errors */
>> + pcie_check_upstream_link(dev);
>> +
>> if (pci_probe_reset_function(dev) == 0)
>> dev->reset_fn = 1;
>> }
>> --
>> 2.14.4
>>
next prev parent reply other threads:[~2018-07-18 13:38 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-06-04 15:55 [PATCH v3] PCI: Check for PCIe downtraining conditions Alexandru Gagniuc
2018-06-05 12:27 ` Andy Shevchenko
2018-06-05 13:04 ` Andy Shevchenko
2018-07-16 21:17 ` Bjorn Helgaas
2018-07-16 22:28 ` Alex_Gagniuc
2018-07-18 21:53 ` Bjorn Helgaas
2018-07-19 15:46 ` Alex G.
2018-07-23 20:01 ` [PATCH v2] PCI/AER: Do not clear AER bits if we don't own AER Alexandru Gagniuc
2018-07-25 1:24 ` kbuild test robot
2018-07-23 20:03 ` [PATCH v5] PCI: Check for PCIe downtraining conditions Alexandru Gagniuc
2018-07-23 21:01 ` Jakub Kicinski
2018-07-23 21:52 ` Tal Gilboa
2018-07-23 22:14 ` Jakub Kicinski
2018-07-23 23:59 ` Alex G.
2018-07-24 13:39 ` Tal Gilboa
2018-07-30 23:26 ` Alex_Gagniuc
2018-07-31 6:40 ` Tal Gilboa
2018-07-31 15:10 ` Alex G.
2018-08-05 7:05 ` Tal Gilboa
2018-08-06 18:39 ` Alex_Gagniuc
2018-08-06 19:46 ` Bjorn Helgaas
2018-08-06 23:25 ` [PATCH v6 1/9] " Alexandru Gagniuc
2018-08-06 23:25 ` [PATCH v6 2/9] bnx2x: Do not call pcie_print_link_status() Alexandru Gagniuc
2018-08-06 23:25 ` [PATCH v6 3/9] bnxt_en: " Alexandru Gagniuc
2018-08-06 23:25 ` [PATCH v6 4/9] cxgb4: " Alexandru Gagniuc
2018-08-06 23:25 ` [PATCH v6 5/9] fm10k: " Alexandru Gagniuc
2018-08-07 17:52 ` Jeff Kirsher
2018-08-06 23:25 ` [PATCH v6 6/9] ixgbe: " Alexandru Gagniuc
2018-08-07 17:51 ` Jeff Kirsher
2018-08-06 23:25 ` [PATCH v6 7/9] net/mlx4: " Alexandru Gagniuc
2018-08-08 6:10 ` Leon Romanovsky
2018-08-06 23:25 ` [PATCH v6 8/9] net/mlx5: " Alexandru Gagniuc
2018-08-08 6:08 ` Leon Romanovsky
2018-08-08 14:23 ` Tal Gilboa
2018-08-08 15:41 ` Leon Romanovsky
2018-08-08 15:56 ` Tal Gilboa
2018-08-08 16:33 ` Alex G.
2018-08-08 17:27 ` Leon Romanovsky
2018-08-09 14:02 ` Bjorn Helgaas
2018-08-06 23:25 ` [PATCH v6 9/9] nfp: " Alexandru Gagniuc
2018-08-07 19:44 ` [PATCH v6 1/9] PCI: Check for PCIe downtraining conditions David Miller
2018-08-07 21:41 ` Bjorn Helgaas
2018-07-18 13:38 ` Tal Gilboa [this message]
2018-07-19 15:49 ` [PATCH v3] " Alex G.
2018-07-23 5:21 ` Tal Gilboa
2018-07-23 17:01 ` Alex G.
2018-07-23 21:35 ` Tal Gilboa
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=628346bc-b455-05f2-2051-35a807076ea0@mellanox.com \
--to=talgi@mellanox.com \
--cc=airlied@gmail.com \
--cc=alex_gagniuc@dellteam.com \
--cc=alexander.deucher@amd.com \
--cc=ariel.elior@cavium.com \
--cc=austin_bolen@dell.com \
--cc=bhelgaas@google.com \
--cc=ganeshgr@chelsio.com \
--cc=helgaas@kernel.org \
--cc=jakub.kicinski@netronome.com \
--cc=jeffrey.t.kirsher@intel.com \
--cc=keith.busch@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=michael.chan@broadcom.com \
--cc=mr.nuke.me@gmail.com \
--cc=shyam_iyer@dell.com \
--cc=tariqt@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).