From: Jason Gunthorpe <jgunthorpe@obsidianresearch.com> To: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: "Neil Greatorex" <neil@fatboyfat.co.uk>, "Willy Tarreau" <w@1wt.eu>, "Matthew Minter" <matthew_minter@xyratex.com>, "Gerlando Falauto" <gerlando.falauto@keymile.com>, linux-arm-kernel@lists.infradead.org, "Jason Cooper" <jason@lakedaemon.net>, "Gregory Clément" <gregory.clement@free-electrons.com>, "Ezequiel Garcia" <ezequiel.garcia@free-electrons.com>, "Andrew Lunn" <andrew@lunn.ch>, linux-pci@vger.kernel.org, "Tawfik Bayouk" <tawfik@marvell.com>, "Lior Amsalem" <alior@marvell.com> Subject: Re: Fixing PCIe issues on Armada XP Date: Tue, 22 Apr 2014 11:34:06 -0600 [thread overview] Message-ID: <20140422173406.GC23955@obsidianresearch.com> (raw) In-Reply-To: <20140418150244.4b3d5229@skate> On Fri, Apr 18, 2014 at 03:02:44PM +0200, Thomas Petazzoni wrote: > > The LinkUp bit after resume is also clearly a lie - most likely the > > PEX takes some time to detect the change in remote state to trigger a > > link down. After all, it was suspended while the remote was busy > > trying to recover. > > Ok, fair enough. However, Neil's patch (which is basically your patch > with longer delays) isn't working here, as I just reported in a > separate e-mail. > > So we don't have a solution right now. Do you have another proposal to > try ? Do you have time to see if we can isolate the difference on your system? > > The fundamental problem here is the clock driver. It should not be > > gating PEX clocks so naively. A PEX suspend needs to be sequenced to > > ensure the link is cleanly brought down before the PEX is put to > > sleep. That way it can cleanly and unambiguously be started up when it > > resumes. No risk of glitching/corrupting the far side with some kind > > of crap on the serial bus. > > > > In any event, the most important required patch here is one that fixes > > socid. It must not turn off the PEX clock. Then we can talk about how > > to fix pci-mvebu to work as a module... > I don't really understand this: all clocks are gated at boot time, so > regardless of mvebu-soc-id behavior, the PCIe driver should take care > of doing all the necessary initialization without making the > assumptions that the clocks were left turned on from the bootloader The current mvebu-soc-id makes it impossible to do a clean hand off of the boot loader PEX setup to the PEX driver. I think that is a problem. Certainly if we fix it non-modular PEX will start working reliably. Keep in mind the current driver cannot startup a PEX without bootloader support. It does not clear the set-at-reset Conf_TrainingDisable bit, and it doesn't wait for a link to come up after doing so. > time. This is needed if we want to support pci-mvebu as a module, so I > don't see why hacking mvebu-soc-id is going to solve this. I agree we should try to figure the modular case out, but it looks to me that suspending the PEX is a bigger job that just gating the clock.. The automatic gating of the PEX clocks should be more clever. If there are no DT elements that reference the clock it can be disabled, otherwise we should probably leave it enabled and rely on the PEX driver to do power management once it gets loaded. So broadly, my thinking is the following largely orthogonal items: 1) PEX's that are setup by the bootloader must be cleanly handed off to the driver. The clocks must never gate. 2) The driver should learn to turn on a PEX from either the Conf_TrainingDisable=1 state or the clock gated state 3) PEX clocks that are never used in DT can be safely shutoff, otherwise they must remain on 4) The PEX driver can learn to properly suspend the PEX for power savings, via the normal Linux power management stuff. What do you think? Jason
WARNING: multiple messages have this Message-ID (diff)
From: jgunthorpe@obsidianresearch.com (Jason Gunthorpe) To: linux-arm-kernel@lists.infradead.org Subject: Fixing PCIe issues on Armada XP Date: Tue, 22 Apr 2014 11:34:06 -0600 [thread overview] Message-ID: <20140422173406.GC23955@obsidianresearch.com> (raw) In-Reply-To: <20140418150244.4b3d5229@skate> On Fri, Apr 18, 2014 at 03:02:44PM +0200, Thomas Petazzoni wrote: > > The LinkUp bit after resume is also clearly a lie - most likely the > > PEX takes some time to detect the change in remote state to trigger a > > link down. After all, it was suspended while the remote was busy > > trying to recover. > > Ok, fair enough. However, Neil's patch (which is basically your patch > with longer delays) isn't working here, as I just reported in a > separate e-mail. > > So we don't have a solution right now. Do you have another proposal to > try ? Do you have time to see if we can isolate the difference on your system? > > The fundamental problem here is the clock driver. It should not be > > gating PEX clocks so naively. A PEX suspend needs to be sequenced to > > ensure the link is cleanly brought down before the PEX is put to > > sleep. That way it can cleanly and unambiguously be started up when it > > resumes. No risk of glitching/corrupting the far side with some kind > > of crap on the serial bus. > > > > In any event, the most important required patch here is one that fixes > > socid. It must not turn off the PEX clock. Then we can talk about how > > to fix pci-mvebu to work as a module... > I don't really understand this: all clocks are gated at boot time, so > regardless of mvebu-soc-id behavior, the PCIe driver should take care > of doing all the necessary initialization without making the > assumptions that the clocks were left turned on from the bootloader The current mvebu-soc-id makes it impossible to do a clean hand off of the boot loader PEX setup to the PEX driver. I think that is a problem. Certainly if we fix it non-modular PEX will start working reliably. Keep in mind the current driver cannot startup a PEX without bootloader support. It does not clear the set-at-reset Conf_TrainingDisable bit, and it doesn't wait for a link to come up after doing so. > time. This is needed if we want to support pci-mvebu as a module, so I > don't see why hacking mvebu-soc-id is going to solve this. I agree we should try to figure the modular case out, but it looks to me that suspending the PEX is a bigger job that just gating the clock.. The automatic gating of the PEX clocks should be more clever. If there are no DT elements that reference the clock it can be disabled, otherwise we should probably leave it enabled and rely on the PEX driver to do power management once it gets loaded. So broadly, my thinking is the following largely orthogonal items: 1) PEX's that are setup by the bootloader must be cleanly handed off to the driver. The clocks must never gate. 2) The driver should learn to turn on a PEX from either the Conf_TrainingDisable=1 state or the clock gated state 3) PEX clocks that are never used in DT can be safely shutoff, otherwise they must remain on 4) The PEX driver can learn to properly suspend the PEX for power savings, via the normal Linux power management stuff. What do you think? Jason
next prev parent reply other threads:[~2014-04-22 17:34 UTC|newest] Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top 2014-04-10 16:19 Fixing PCIe issues on Armada XP Thomas Petazzoni 2014-04-10 16:19 ` Thomas Petazzoni 2014-04-10 16:57 ` Jason Gunthorpe 2014-04-10 16:57 ` Jason Gunthorpe 2014-04-10 18:01 ` Thomas Petazzoni 2014-04-10 18:01 ` Thomas Petazzoni 2014-04-10 20:12 ` Jason Gunthorpe 2014-04-10 20:12 ` Jason Gunthorpe 2014-04-10 21:04 ` Thomas Petazzoni 2014-04-10 21:04 ` Thomas Petazzoni 2014-04-10 21:56 ` Neil Greatorex 2014-04-10 21:56 ` Neil Greatorex 2014-04-10 22:06 ` Jason Gunthorpe 2014-04-10 22:06 ` Jason Gunthorpe 2014-04-10 22:15 ` Neil Greatorex 2014-04-10 22:15 ` Neil Greatorex 2014-04-11 10:23 ` Thomas Petazzoni 2014-04-11 10:23 ` Thomas Petazzoni 2014-04-11 16:31 ` Jason Gunthorpe 2014-04-11 16:31 ` Jason Gunthorpe 2014-04-11 17:21 ` Matthew Minter 2014-04-11 17:21 ` Matthew Minter 2014-04-11 17:29 ` Jason Gunthorpe 2014-04-11 17:29 ` Jason Gunthorpe 2014-04-18 13:02 ` Thomas Petazzoni 2014-04-18 13:02 ` Thomas Petazzoni 2014-04-22 17:34 ` Jason Gunthorpe [this message] 2014-04-22 17:34 ` Jason Gunthorpe 2014-04-18 12:58 ` Thomas Petazzoni 2014-04-18 12:58 ` Thomas Petazzoni 2014-04-22 17:56 ` Jason Gunthorpe 2014-04-22 17:56 ` Jason Gunthorpe 2014-04-10 17:10 ` Willy Tarreau 2014-04-10 17:10 ` Willy Tarreau 2014-04-10 18:02 ` Thomas Petazzoni 2014-04-10 18:02 ` Thomas Petazzoni 2014-04-10 23:13 ` Willy Tarreau 2014-04-10 23:13 ` Willy Tarreau 2014-04-10 23:40 ` Jason Gunthorpe 2014-04-10 23:40 ` Jason Gunthorpe 2014-04-11 6:23 ` Willy Tarreau 2014-04-11 6:23 ` Willy Tarreau 2014-04-10 18:20 ` Neil Greatorex 2014-04-10 18:20 ` Neil Greatorex 2014-04-10 21:07 ` Thomas Petazzoni 2014-04-10 21:07 ` Thomas Petazzoni 2014-04-11 14:32 ` Thomas Petazzoni 2014-04-11 14:32 ` Thomas Petazzoni 2014-04-11 15:57 ` Neil Greatorex 2014-04-11 15:57 ` Neil Greatorex
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20140422173406.GC23955@obsidianresearch.com \ --to=jgunthorpe@obsidianresearch.com \ --cc=alior@marvell.com \ --cc=andrew@lunn.ch \ --cc=ezequiel.garcia@free-electrons.com \ --cc=gerlando.falauto@keymile.com \ --cc=gregory.clement@free-electrons.com \ --cc=jason@lakedaemon.net \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-pci@vger.kernel.org \ --cc=matthew_minter@xyratex.com \ --cc=neil@fatboyfat.co.uk \ --cc=tawfik@marvell.com \ --cc=thomas.petazzoni@free-electrons.com \ --cc=w@1wt.eu \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.