From: Thierry Reding <thierry.reding@gmail.com>
To: Manikanta Maddireddy <mmaddireddy@nvidia.com>
Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
bhelgaas@google.com, robh+dt@kernel.org, mark.rutland@arm.com,
jonathanh@nvidia.com, vidyas@nvidia.com,
linux-tegra@vger.kernel.org, linux-pci@vger.kernel.org,
devicetree@vger.kernel.org
Subject: Re: [PATCH V4 22/28] PCI: tegra: Access endpoint config only if PCIe link is up
Date: Mon, 17 Jun 2019 13:47:45 +0200 [thread overview]
Message-ID: <20190617114745.GL508@ulmo> (raw)
In-Reply-To: <a523a19c-fdfa-01f7-6f6d-2ca367a10a50@nvidia.com>
[-- Attachment #1: Type: text/plain, Size: 12403 bytes --]
On Mon, Jun 17, 2019 at 03:31:38PM +0530, Manikanta Maddireddy wrote:
>
>
> On 13-Jun-19 9:12 PM, Thierry Reding wrote:
> > On Thu, Jun 13, 2019 at 03:39:46PM +0100, Lorenzo Pieralisi wrote:
> >> On Mon, Jun 10, 2019 at 10:08:16AM +0530, Manikanta Maddireddy wrote:
> >>>
> >>> On 04-Jun-19 7:40 PM, Manikanta Maddireddy wrote:
> >>>> On 04-Jun-19 6:44 PM, Thierry Reding wrote:
> >>>>> On Thu, May 16, 2019 at 11:23:01AM +0530, Manikanta Maddireddy wrote:
> >>>>>> Few endpoints like Wi-Fi supports power on/off and to leverage that
> >>>>>> root port must support hot-plug and hot-unplug. Tegra PCIe doesn't
> >>>>>> support hot-plug and hot-unplug, however it supports endpoint power
> >>>>>> on/off feature as follows,
> >>>>>> - Power off sequence:
> >>>>>> - Transition of PCIe link to L2
> >>>>>> - Power off endpoint
> >>>>>> - Leave root port in power up state with the link in L2
> >>>>>> - Power on sequence:
> >>>>>> - Power on endpoint
> >>>>>> - Apply hot reset to get PCIe link up
> >>>>>>
> >>>>>> PCIe client driver stops accessing PCIe endpoint config and BAR registers
> >>>>>> after endpoint is powered off. However, software applications like x11
> >>>>>> server or lspci can access endpoint config registers in which case
> >>>>>> host controller raises "response decoding" errors. To avoid this scenario,
> >>>>>> add PCIe link up check in config read and write callback functions before
> >>>>>> accessing endpoint config registers.
> >>>>>>
> >>>>>> Signed-off-by: Manikanta Maddireddy <mmaddireddy@nvidia.com>
> >>>>>> ---
> >>>>>> V4: No change
> >>>>>>
> >>>>>> V3: Update the commit log with explanation for the need of this patch
> >>>>>>
> >>>>>> V2: Change tegra_pcie_link_status() to tegra_pcie_link_up()
> >>>>>>
> >>>>>> drivers/pci/controller/pci-tegra.c | 38 ++++++++++++++++++++++++++++++
> >>>>>> 1 file changed, 38 insertions(+)
> >>>>> This still doesn't look right to me conceptually. If somebody wants to
> >>>>> access the PCI devices after the kernel has powered them off, why can't
> >>>>> we just power the devices back on so that we allow userspace to properly
> >>>>> access the devices?
> >>>> 1. WiFi devices provides power-off feature for power saving in mobiles.
> >>>> When WiFi is turned off we shouldn't power on the HW back without user
> >>>> turning it back on.
> >>>> 2. When ever user process tries to access config space, it'll end up
> >>>> in these functions. We cannot have is_powered_on check in config read/write
> >>>> callbacks.
> >>>> 3. WiFi power on/off is device specific feature, we shouldn't handle it
> >>>> in PCI subsystem or host controller driver.
> >>>>
> >>>>> Or if that's not what we want, shouldn't we add something to the core
> >>>>> PCI infrastructure to let us deal with this? It seems like this is some
> >>>>> general problem that would apply to every PCI device and host bridge
> >>>>> driver. Having each driver implement this logic separately doesn't seem
> >>>>> like a good idea to me.
> >>>>>
> >>>>> Thierry
> >>>> This should be handled by hotplug feature, whenever endpoint is powered-off/
> >>>> removed from the slot, hot unplug event should take care of it. Unfortunately
> >>>> Tegra PCIe doesn't support hotplug feature.
> >>>>
> >>>> Manikanta
> >>> Hi Bjorn,
> >>>
> >>> I thought about your comment in
> >>> https://patchwork.ozlabs.org/patch/1084204/ again. What if I add link
> >>> up check in tegra_pcie_isr() and make "response decoding error" as
> >>> debug print? EP Config access will happen when link is down, but
> >>> "Response decoding error" print comes only if debug log is enabled.
> >>> This way we can avoid race issue in config accessors and we get prints
> >>> when debug logs are enabled.
> >> I still do not see what you are actually solving. This patch should
> >> be dropped.
> > The problem that Manikanta is trying to solve here occurs in this
> > situation (Manikanta, correct me if I've got this wrong): on some
> > setups, a WiFi module connected over PCI will toggle a power GPIO as
> > part of runtime suspend. This effectively causes the module to disappear
> > from the PCI bus (i.e. it can no longer be accessed until the power GPIO
> > is toggled again).
>
> GPIO is toggled as part of WiFi on/off, can be triggered from network manager UI.
>
> >
> > This is fine from a kernel point of view because the kernel keeps track
> > of what devices are suspended. However, userspace will occasionally try
> > to read the configuration space access of all devices, and since it
> > doesn't have any knowledge about the suspend state of these devices, it
> > doesn't know which ones to leave alone. I think this happens when the
> > X.Org server is running.
>
> This is fine from a kernel point of view because PCI client driver
> doesn't initiate any PCIe transaction until network interface
> is up during WiFi on.
>
> >
> > One thing that Manikanta and I had discussed was that perhaps the device
> > should be hot-unplugged when it goes into this low-power state. However,
> > we don't support hotplug on Tegra210 where this is needed, so we'd need
> > some sort of software-induced hot-unplug. However, this low power state
> > is entered when the WiFi interface is taken down (i.e. ip link set dev
> > <interface> down). If we were to remove the PCI device in that case, it
> > means that the interface goes away completely, which is completely
> > unexpected from a user's perspective. After all, taking a link down and
> > up may be something that scripts are doing all the time. They'd fall
> > over if after taking the interface down, the interface completely
> > disappears.
> >
> > It's also not entirely clear to me how we get the device back onto the
> > bus again after it is in low power. If we hot-unplug the device, then
> > the driver will be unbound. Presumably the driver is what's controlling
> > the power GPIO, so there won't be any entity that can be used to bring
> > the chip back to life. Unless we deal with that power GPIO elsewhere
> > (rfkill switch perhaps?).
>
> Correct, rfkill switch should handle the GPIO.
> Sequence will be,
> - WiFi ON
> - rfkill switch enables the WiFi GPIO
> - Tegra PCIe receives hot plug event
> - Tegra PCIe hot plug driver rescans PCI bus and enumerates the device
> - PCI client driver is probed, which will create network interface
> - WiFi OFF
> - rfkill switch disables the WiFi GPIO
> - Tegra PCIe receives hot unplug event
> - Tegra PCIe hot plug driver removes PCI devices under the bus
> - PCI client driver remove is executed, which will remove network interface
>
> We don't need current patch in this case because PCI device is not present
> in the PCI hierarchy, so there cannot be EP config access with link down.
> However Tegra doesn't support hot plug and unplug events. I am not sure
> if we have any software based hot plug event trigger.
>
> I will drop current patch and pursue if above sequence can be
> implemented for Tegra.
I just recalled that we have these messages in the kernel log:
# dmesg | grep tegra-pcie
[ 1.055761] tegra-pcie 1003000.pcie: 4x1, 1x1 configuration
[ 2.745764] tegra-pcie 1003000.pcie: 4x1, 1x1 configuration
[ 2.753073] tegra-pcie 1003000.pcie: probing port 0, using 4 lanes
[ 2.761334] tegra-pcie 1003000.pcie: Slot present pin change, signature: 00000008
[ 3.177607] tegra-pcie 1003000.pcie: link 0 down, retrying
[ 3.585605] tegra-pcie 1003000.pcie: link 0 down, retrying
[ 3.993606] tegra-pcie 1003000.pcie: link 0 down, retrying
[ 4.001214] tegra-pcie 1003000.pcie: link 0 down, ignoring
[ 4.006733] tegra-pcie 1003000.pcie: probing port 1, using 1 lanes
[ 4.015042] tegra-pcie 1003000.pcie: Slot present pin change, signature: 00000000
[ 4.031177] tegra-pcie 1003000.pcie: PCI host bridge to bus 0000:00
These "slot present pin change" message do look a lot like hotplug
related messages. Could we perhaps use those to our advantage for this
case? Do you see these when you run on the platform where WiFi is
enabled/disabled using rfkill?
Given that rfkill is completely decoupled from PCI, I don't see how we
would trigger any software-based hotplug mechanism. Perhaps one thing
that we could do is the equivalent of this:
# echo 1 > /sys/bus/pci/rescan
from some script that's perhaps tied to the rfkill somehow. I'm not sure
if that's possible, or generic enough.
Thierry
> > Perhaps one other way to deal with this would be to track the suspend
> > state of devices and then have the code that implements the PCI access
> > from userspace refuse accesses to devices that are asleep. I suppose
> > this is somewhat of an odd use-case because traditionally I guess PCI
> > devices never power down to a state where their configuration space can
> > no longer be accessed. At least that's what would explain why this has
> > never been an issue before. Or perhaps it has?
> >
> > The last resort would be to just never put the WiFi chip into that low
> > power mode, though I'm not exactly sure what that means for the power
> > consumption on the affected systems.
> >
> > Manikanta, can you fill in some of the blanks above?
> >
> > Thierry
> >>> Thierry,
> >>> Please share your inputs as well.
> >>>
> >>> Manikanta
> >>>
> >>>
> >>>>>> diff --git a/drivers/pci/controller/pci-tegra.c b/drivers/pci/controller/pci-tegra.c
> >>>>>> index d20c88a79e00..33f4dfab9e35 100644
> >>>>>> --- a/drivers/pci/controller/pci-tegra.c
> >>>>>> +++ b/drivers/pci/controller/pci-tegra.c
> >>>>>> @@ -428,6 +428,14 @@ static inline u32 pads_readl(struct tegra_pcie *pcie, unsigned long offset)
> >>>>>> return readl(pcie->pads + offset);
> >>>>>> }
> >>>>>>
> >>>>>> +static bool tegra_pcie_link_up(struct tegra_pcie_port *port)
> >>>>>> +{
> >>>>>> + u32 value;
> >>>>>> +
> >>>>>> + value = readl(port->base + RP_LINK_CONTROL_STATUS);
> >>>>>> + return !!(value & RP_LINK_CONTROL_STATUS_DL_LINK_ACTIVE);
> >>>>>> +}
> >>>>>> +
> >>>>>> /*
> >>>>>> * The configuration space mapping on Tegra is somewhat similar to the ECAM
> >>>>>> * defined by PCIe. However it deviates a bit in how the 4 bits for extended
> >>>>>> @@ -493,20 +501,50 @@ static void __iomem *tegra_pcie_map_bus(struct pci_bus *bus,
> >>>>>> static int tegra_pcie_config_read(struct pci_bus *bus, unsigned int devfn,
> >>>>>> int where, int size, u32 *value)
> >>>>>> {
> >>>>>> + struct tegra_pcie *pcie = bus->sysdata;
> >>>>>> + struct pci_dev *bridge;
> >>>>>> + struct tegra_pcie_port *port;
> >>>>>> +
> >>>>>> if (bus->number == 0)
> >>>>>> return pci_generic_config_read32(bus, devfn, where, size,
> >>>>>> value);
> >>>>>>
> >>>>>> + bridge = pcie_find_root_port(bus->self);
> >>>>>> +
> >>>>>> + list_for_each_entry(port, &pcie->ports, list)
> >>>>>> + if (port->index + 1 == PCI_SLOT(bridge->devfn))
> >>>>>> + break;
> >>>>>> +
> >>>>>> + /* If there is no link, then there is no device */
> >>>>>> + if (!tegra_pcie_link_up(port)) {
> >>>>>> + *value = 0xffffffff;
> >>>>>> + return PCIBIOS_DEVICE_NOT_FOUND;
> >>>>>> + }
> >>>>>> +
> >>>>>> return pci_generic_config_read(bus, devfn, where, size, value);
> >>>>>> }
> >>>>>>
> >>>>>> static int tegra_pcie_config_write(struct pci_bus *bus, unsigned int devfn,
> >>>>>> int where, int size, u32 value)
> >>>>>> {
> >>>>>> + struct tegra_pcie *pcie = bus->sysdata;
> >>>>>> + struct tegra_pcie_port *port;
> >>>>>> + struct pci_dev *bridge;
> >>>>>> +
> >>>>>> if (bus->number == 0)
> >>>>>> return pci_generic_config_write32(bus, devfn, where, size,
> >>>>>> value);
> >>>>>>
> >>>>>> + bridge = pcie_find_root_port(bus->self);
> >>>>>> +
> >>>>>> + list_for_each_entry(port, &pcie->ports, list)
> >>>>>> + if (port->index + 1 == PCI_SLOT(bridge->devfn))
> >>>>>> + break;
> >>>>>> +
> >>>>>> + /* If there is no link, then there is no device */
> >>>>>> + if (!tegra_pcie_link_up(port))
> >>>>>> + return PCIBIOS_DEVICE_NOT_FOUND;
> >>>>>> +
> >>>>>> return pci_generic_config_write(bus, devfn, where, size, value);
> >>>>>> }
> >>>>>>
> >>>>>> --
> >>>>>> 2.17.1
> >>>>>>
>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
next prev parent reply other threads:[~2019-06-17 11:47 UTC|newest]
Thread overview: 72+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-16 5:52 [PATCH V4 00/28] Enable Tegra PCIe root port features Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 01/28] soc/tegra: pmc: Export tegra_powergate_power_on() Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 02/28] PCI: tegra: Handle failure cases in tegra_pcie_power_on() Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 03/28] PCI: tegra: Rearrange Tegra PCIe driver functions Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 04/28] PCI: tegra: Mask AFI_INTR in runtime suspend Manikanta Maddireddy
2019-06-04 13:08 ` Thierry Reding
2019-05-16 5:52 ` [PATCH V4 05/28] PCI: tegra: Fix PCIe host power up sequence Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 06/28] PCI: tegra: Add PCIe Gen2 link speed support Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 07/28] PCI: tegra: Advertise PCIe Advanced Error Reporting (AER) capability Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 08/28] PCI: tegra: Program UPHY electrical settings for Tegra210 Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 09/28] PCI: tegra: Enable opportunistic UpdateFC and ACK Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 10/28] PCI: tegra: Disable AFI dynamic clock gating Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 11/28] PCI: tegra: Process pending DLL transactions before entering L1 or L2 Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 12/28] PCI: tegra: Enable PCIe xclk clock clamping Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 13/28] PCI: tegra: Increase the deskew retry time Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 14/28] PCI: tegra: Add SW fixup for RAW violations Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 15/28] PCI: tegra: Update flow control timer frequency in Tegra210 Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 16/28] PCI: tegra: Set target speed as Gen1 before starting LTSSM Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 17/28] PCI: tegra: Fix PLLE power down issue due to CLKREQ# signal Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 18/28] PCI: tegra: Program AFI_CACHE* registers only for Tegra20 Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 19/28] PCI: tegra: Change PRSNT_SENSE IRQ log to debug Manikanta Maddireddy
2019-05-16 5:52 ` [PATCH V4 20/28] PCI: tegra: Use legacy IRQ for port service drivers Manikanta Maddireddy
2019-05-20 20:37 ` Bjorn Helgaas
2019-05-21 9:07 ` Manikanta Maddireddy
2019-05-16 5:53 ` [PATCH V4 21/28] PCI: tegra: Add AFI_PEX2_CTRL reg offset as part of soc struct Manikanta Maddireddy
2019-05-16 5:53 ` [PATCH V4 22/28] PCI: tegra: Access endpoint config only if PCIe link is up Manikanta Maddireddy
2019-06-04 13:14 ` Thierry Reding
2019-06-04 14:10 ` Manikanta Maddireddy
2019-06-10 4:38 ` Manikanta Maddireddy
2019-06-13 14:39 ` Lorenzo Pieralisi
2019-06-13 15:42 ` Thierry Reding
2019-06-17 10:01 ` Manikanta Maddireddy
2019-06-17 11:47 ` Thierry Reding [this message]
2019-06-17 19:30 ` Bjorn Helgaas
2019-06-18 5:36 ` Manikanta Maddireddy
2019-06-18 10:49 ` Thierry Reding
2019-06-18 12:32 ` Johannes Berg
2019-06-18 13:40 ` Thierry Reding
2019-06-18 14:48 ` Johannes Berg
2019-06-19 13:38 ` Bjorn Helgaas
2019-06-19 13:40 ` Johannes Berg
2019-05-16 5:53 ` [PATCH V4 23/28] dt-bindings: pci: tegra: Document PCIe DPD pinctrl optional prop Manikanta Maddireddy
2019-05-16 5:53 ` [PATCH V4 24/28] arm64: tegra: Add PEX DPD states as pinctrl properties Manikanta Maddireddy
2019-05-16 5:53 ` [PATCH V4 25/28] PCI: tegra: Put PEX CLK & BIAS pads in DPD mode Manikanta Maddireddy
2019-05-16 5:53 ` [PATCH V4 26/28] PCI: Add DT binding for "reset-gpios" property Manikanta Maddireddy
2019-06-17 11:30 ` Thierry Reding
2019-06-17 11:38 ` Manikanta Maddireddy
2019-06-17 11:48 ` Thierry Reding
2019-05-16 5:53 ` [PATCH V4 27/28] PCI: tegra: Add support for GPIO based PERST# Manikanta Maddireddy
2019-06-04 13:22 ` Thierry Reding
2019-06-13 15:24 ` Lorenzo Pieralisi
2019-06-14 10:37 ` Manikanta Maddireddy
2019-06-14 14:32 ` Lorenzo Pieralisi
2019-06-14 14:38 ` Manikanta Maddireddy
2019-06-14 14:50 ` Lorenzo Pieralisi
2019-06-14 14:56 ` Manikanta Maddireddy
2019-06-14 15:23 ` Thierry Reding
2019-06-14 15:59 ` Lorenzo Pieralisi
2019-06-14 16:30 ` Manikanta Maddireddy
2019-06-14 16:53 ` Lorenzo Pieralisi
2019-06-14 17:23 ` Manikanta Maddireddy
2019-06-17 9:48 ` Lorenzo Pieralisi
2019-06-17 10:27 ` Manikanta Maddireddy
2019-06-17 10:39 ` Lorenzo Pieralisi
2019-06-17 11:29 ` Thierry Reding
2019-06-17 11:26 ` Thierry Reding
2019-05-16 5:53 ` [PATCH V4 28/28] PCI: tegra: Change link retry log level to debug Manikanta Maddireddy
2019-06-04 13:22 ` Thierry Reding
2019-05-16 13:12 ` [PATCH V4 00/28] Enable Tegra PCIe root port features Bjorn Helgaas
2019-05-17 8:38 ` Manikanta Maddireddy
2019-06-10 4:45 ` Manikanta Maddireddy
2019-06-10 17:33 ` Lorenzo Pieralisi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190617114745.GL508@ulmo \
--to=thierry.reding@gmail.com \
--cc=bhelgaas@google.com \
--cc=devicetree@vger.kernel.org \
--cc=jonathanh@nvidia.com \
--cc=linux-pci@vger.kernel.org \
--cc=linux-tegra@vger.kernel.org \
--cc=lorenzo.pieralisi@arm.com \
--cc=mark.rutland@arm.com \
--cc=mmaddireddy@nvidia.com \
--cc=robh+dt@kernel.org \
--cc=vidyas@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).