linux-pci.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: "™֟☻̭҇ Ѽ ҉ ®" <vtolkm@googlemail.com>
To: "Bjorn Helgaas" <helgaas@kernel.org>,
	"Toke Høiland-Jørgensen" <toke@redhat.com>
Cc: "Pali Rohár" <pali@kernel.org>,
	linux-pci@vger.kernel.org, linux-arm-kernel@lists.infradead.org,
	"Rob Herring" <robh@kernel.org>,
	"Ilias Apalodimas" <ilias.apalodimas@linaro.org>,
	"Marek Behún" <marek.behun@nic.cz>,
	"Thomas Petazzoni" <thomas.petazzoni@bootlin.com>,
	"Jason Cooper" <jason@lakedaemon.net>
Subject: Re: PCI trouble on mvebu (Turris Omnia)
Date: Thu, 29 Oct 2020 19:56:00 +0000	[thread overview]
Message-ID: <d674bfec-5045-d2f2-b4cc-e17a04b5e529@gmail.com> (raw)
In-Reply-To: <20201029193022.GA476048@bjorn-Precision-5520>


[-- Attachment #1.1.1: Type: text/plain, Size: 4820 bytes --]

On 29/10/2020 20:30, Bjorn Helgaas wrote:
> On Thu, Oct 29, 2020 at 12:12:21PM +0100, Toke Høiland-Jørgensen wrote:
>> Pali Rohár <pali@kernel.org> writes:
>>> I have been testing mainline kernel on Turris Omnia with two PCIe
>>> default cards (WLE200 and WLE900) and it worked fine. But I do not know
>>> if I had ASPM enabled or not.
>>>
>>> So it is working fine for you when CONFIG_PCIEASPM is disabled and whole
>>> issue is only when CONFIG_PCIEASPM is enabled?
>> Yup, exactly. And I'm also currently testing with the default WLE200/900
>> cards... I just tried sticking an MT76-based WiFi card into the third
>> PCI slot, and that doesn't come up either when I enable PCIEASPM.
> Huh.  So IIUC, the following cases all try to retrain the link and it
> fails to come up again:
>
>    - aardvark + WLE900VX (see commit 43fc679ced18)
>    - mvebu + WLE200
>    - mvebu + WLE900
>    - mvebu + MT76
>
> In all these cases, Linux was able to enumerate the NIC, which means
> the link was up when firmware handed it off.
>
> I think Linux decided the Common Clock Configuration was wrong, so it
> tried to fix it and retrain the link, and the link didn't come back
> up.
>
> I don't have "lspci -vv" output from all of them, but in vtolkm's
> case, the firmware handed off with:
>
>    00:02.0 Root Port to [bus 02]  SlotClk+ CommClk+
>    02:00.0 QCA986x/988x NIC       SlotClk+ CommClk-
>
> Per spec (PCIe r5, sec 7.5.3.7), SlotClk is HwInit and CommClk is RW
> and should power up as 0.  If I'm reading the implementation note
> correctly, if SlotClk is set on both ends of the link, software should
> set CommClk, so the config above *does* look wrong, and CommClk+ on
> the Root Port suggests that firmware set it.
>
> I think both the aardvark and mvebu systems probably use U-Boot.  I
> don't know U-Boot at all, but I don't see anything in it that touches
> Link Control.  I'm curious what happens if you put one of these cards
> in a PC.  If anybody tries it, please collect the "sudo lspci -vv" and
> dmesg output.
>
> We could quirk these NICs to avoid the retrain, but since aardvark and
> mvebu have no obvious connection and WLE200/WLE900 and MT76 have no
> obvious connection, I doubt there's a simple hardware defect that
> explains all these.
>
> Maybe we're doing something wrong in the retrain, but obviously the
> link came up in the first place.  AFAIK the only thing we're changing
> is the CommClk setting, and that looks legitimate per spec.
>
> Another experiment: build kernel without CONFIG_PCIEASPM, set $ROOT
> and $NIC appropriately, and try the following:
>
>    # Set $ROOT and $NIC (update to match your system):
>
>      # ROOT=00:02.0
>      # NIC=02:00.0
>
>    # Dump the Root Port and NIC Link registers:
>
>      # setpci -s$ROOT CAP_EXP+0xc.l              # Link Capabilities
>      # setpci -s$ROOT CAP_EXP+0x10.w             # Link Control
>      # setpci -s$ROOT CAP_EXP+0x12.w             # Link Status
>
>      # setpci -s$NIC  CAP_EXP+0xc.l              # Link Capabilities
>      # setpci -s$NIC  CAP_EXP+0x10.w             # Link Control
>      # setpci -s$NIC  CAP_EXP+0x12.w             # Link Status
>
>    # Retrain the link:
>
>      # setpci -s$ROOT CAP_EXP+0x10.w=0x0020      # Link Control Retrain Link
>      # sleep 1
>      # setpci -s$ROOT CAP_EXP+0x12.w             # Link Status
>      # setpci -s$NIC  CAP_EXP+0x12.w             # Link Status
>
>    # Set CommClk+ and retrain the link:
>
>      # setpci -s$NIC  CAP_EXP+0x10.w=0x0040      # Link Control Common Clock
>      # setpci -s$ROOT CAP_EXP+0x10.w=0x0040      # Link Control Common Clock
>      # setpci -s$ROOT CAP_EXP+0x10.w=0x0060      # Link Control RL + CC
>      # sleep 1
>      # setpci -s$ROOT CAP_EXP+0x12.w             # Link Status
>      # setpci -s$NIC  CAP_EXP+0x12.w             # Link Status

ROOT=00:02.0
NIC=02:00.0
setpci -s$ROOT CAP_EXP+0xc.l
0003ac12
setpci -s$ROOT CAP_EXP+0x10.w
0040
setpci -s$ROOT CAP_EXP+0x12.w
1011
setpci -s$NIC  CAP_EXP+0xc.l

00036c11
setpci -s$NIC  CAP_EXP+0x10.w
0000
setpci -s$NIC  CAP_EXP+0x12.w
1011
setpci -s$ROOT CAP_EXP+0x10.w=0x0020
sleep 1
setpci -s$ROOT CAP_EXP+0x12.w
1011
setpci -s$NIC  CAP_EXP+0x12.w
setpci: 0000:02:00.0: Instance #0 of Capability 0010 not found - there 
are no capabilities with that id.
setpci -s$NIC  CAP_EXP+0x10.w=0x0040
setpci: 0000:02:00.0: Instance #0 of Capability 0010 not found - there 
are no capabilities with that id.
setpci -s$ROOT CAP_EXP+0x10.w=0x0040
setpci -s$ROOT CAP_EXP+0x10.w=0x0060
sleep 1
setpci -s$ROOT CAP_EXP+0x12.w
1811
setpci -s$NIC  CAP_EXP+0x12.w
setpci: 0000:02:00.0: Instance #0 of Capability 0010 not found - there 
are no capabilities with that id.


[-- Attachment #1.1.2: OpenPGP_0x729CFF47A416598B.asc --]
[-- Type: application/pgp-keys, Size: 3163 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

  reply	other threads:[~2020-10-29 19:59 UTC|newest]

Thread overview: 62+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-27 15:43 Toke Høiland-Jørgensen
2020-10-27 17:20 ` Bjorn Helgaas
2020-10-27 17:44   ` ™֟☻̭҇ Ѽ ҉ ®
2020-10-27 18:59     ` Toke Høiland-Jørgensen
2020-10-27 20:20       ` Toke Høiland-Jørgensen
2020-10-27 21:22         ` ™֟☻̭҇ Ѽ ҉ ®
2020-10-27 21:31           ` Toke Høiland-Jørgensen
2020-10-27 22:01             ` ™֟☻̭҇ Ѽ ҉ ®
2020-10-27 22:12               ` Toke Høiland-Jørgensen
2020-10-27 18:56   ` Toke Høiland-Jørgensen
2020-10-28 13:36     ` Toke Høiland-Jørgensen
2020-10-28 14:42       ` Bjorn Helgaas
2020-10-28 15:08         ` Toke Høiland-Jørgensen
2020-10-28 16:40           ` ™֟☻̭҇ Ѽ ҉ ®
2020-10-28 23:16             ` Bjorn Helgaas
2020-10-29 10:09               ` Pali Rohár
2020-10-29 10:56                 ` ™֟☻̭҇ Ѽ ҉ ®
2020-10-29 11:12                 ` Toke Høiland-Jørgensen
2020-10-29 19:30                   ` Bjorn Helgaas
2020-10-29 19:56                     ` ™֟☻̭҇ Ѽ ҉ ® [this message]
2020-10-29 19:57                     ` Andrew Lunn
2020-10-29 21:55                       ` Thomas Petazzoni
2020-10-29 20:18                     ` Toke Høiland-Jørgensen
2020-10-29 22:09                       ` Toke Høiland-Jørgensen
2020-10-29 20:58                     ` Marek Behun
2020-10-30 10:08                       ` Pali Rohár
2020-10-30 10:45                         ` Marek Behun
2020-10-29 21:54                     ` Thomas Petazzoni
2020-10-29 23:15                       ` Toke Høiland-Jørgensen
2020-10-30  8:23                         ` Thomas Petazzoni
2020-10-30 10:15                         ` Pali Rohár
2020-10-29 10:41               ` Toke Høiland-Jørgensen
2020-10-29 11:18                 ` ™֟☻̭҇ Ѽ ҉ ®
2020-10-30 11:23               ` Pali Rohár
2020-10-30 13:02                 ` Toke Høiland-Jørgensen
2020-10-30 14:23                   ` Pali Rohár
2020-10-30 14:54                     ` ™֟☻̭҇ Ѽ ҉ ®
2020-10-31 12:49                       ` Toke Høiland-Jørgensen
2020-11-02 15:24                         ` Pali Rohár
2020-11-02 15:54                           ` Toke Høiland-Jørgensen
2020-11-02 16:18                             ` ™֟☻̭҇ Ѽ ҉ ®
2020-11-02 16:33                               ` Toke Høiland-Jørgensen
2021-03-15 19:58                             ` Pali Rohár
2021-03-16  9:25                               ` Pali Rohár
2021-03-18 22:43                                 ` Toke Høiland-Jørgensen
2021-03-18 23:16                                   ` Pali Rohár
2021-03-26 12:50                                     ` Pali Rohár
2021-03-26 15:25                                       ` Toke Høiland-Jørgensen
2021-03-26 15:34                                         ` Pali Rohár
2021-03-26 16:54                                           ` Toke Høiland-Jørgensen
2021-03-26 17:11                                             ` Pali Rohár
2021-03-26 17:51                                               ` Toke Høiland-Jørgensen
2021-03-29 17:09                                                 ` Pali Rohár
2021-03-31 14:02                                                   ` Toke Høiland-Jørgensen
2021-03-31 16:15                                                     ` Pali Rohár
2021-03-31 16:53                                                       ` Toke Høiland-Jørgensen
2020-10-29  1:21             ` Marek Behun
2020-10-29 15:12           ` Rob Herring
2020-10-27 18:03 ` Marek Behun
2020-10-27 19:00   ` Toke Høiland-Jørgensen
2020-10-27 20:19     ` Marek Behun
2020-10-27 20:49       ` Toke Høiland-Jørgensen

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d674bfec-5045-d2f2-b4cc-e17a04b5e529@gmail.com \
    --to=vtolkm@googlemail.com \
    --cc=helgaas@kernel.org \
    --cc=ilias.apalodimas@linaro.org \
    --cc=jason@lakedaemon.net \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=marek.behun@nic.cz \
    --cc=pali@kernel.org \
    --cc=robh@kernel.org \
    --cc=thomas.petazzoni@bootlin.com \
    --cc=toke@redhat.com \
    --cc=vtolkm@gmail.com \
    --subject='Re: PCI trouble on mvebu (Turris Omnia)' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).