Linux-PCI Archive on lore.kernel.org
 help / color / Atom feed
From: Manikanta Maddireddy <mmaddireddy@nvidia.com>
To: Thierry Reding <thierry.reding@gmail.com>
Cc: <bhelgaas@google.com>, <robh+dt@kernel.org>,
	<mark.rutland@arm.com>, <jonathanh@nvidia.com>,
	<lorenzo.pieralisi@arm.com>, <vidyas@nvidia.com>,
	<linux-tegra@vger.kernel.org>, <linux-pci@vger.kernel.org>,
	<devicetree@vger.kernel.org>
Subject: Re: [PATCH 02/30] PCI: tegra: Fix PCIe host power up sequence
Date: Mon, 15 Apr 2019 23:44:16 +0530
Message-ID: <33f945e1-ca97-b8ab-4687-62c5a0e4f45c@nvidia.com> (raw)
In-Reply-To: <20190415143019.GE29254@ulmo>



On 15-Apr-19 8:00 PM, Thierry Reding wrote:
> On Mon, Apr 15, 2019 at 07:41:21PM +0530, Manikanta Maddireddy wrote:
>> On 15-Apr-19 4:31 PM, Thierry Reding wrote:
>>> On Thu, Apr 11, 2019 at 10:33:27PM +0530, Manikanta Maddireddy wrote:
>>>> PCIe host power up sequence involves programming AFI(AXI to FPCI bridge)
>>>> registers first and then PCIe registers. Otherwise AFI register settings
>>>> may not latch to PCIe IP.
>>>>
>>>> PCIe root port starts LTSSM as soon as PCIe xrst is deasserted.
>>>> So deassert PCIe xrst after programming PCIe registers.
>>>>
>>>> Modify PCIe power up sequence as follows,
>>>>   - Power ungate PCIe partition
>>>>   - Enable AFI clock
>>>>   - Deassert AFI reset
>>>>   - Program AFI registers
>>>>   - Enable PCIe clock
>>>>   - Deassert PCIe reset
>>>>   - Program PCIe registers
>>>>   - Deassert PCIe xrst to start LTSSM
>>>>
>>>> Signed-off-by: Manikanta Maddireddy <mmaddireddy@nvidia.com>
>>>> ---
>>>>  drivers/pci/controller/pci-tegra.c | 73 ++++++++++++++++++------------
>>>>  1 file changed, 43 insertions(+), 30 deletions(-)
>>>>
>>>> diff --git a/drivers/pci/controller/pci-tegra.c b/drivers/pci/controller/pci-tegra.c
>>>> index f4f53d092e00..0bf270bcea34 100644
>>>> --- a/drivers/pci/controller/pci-tegra.c
>>>> +++ b/drivers/pci/controller/pci-tegra.c
>>>> @@ -966,9 +966,6 @@ static int tegra_pcie_enable_controller(struct tegra_pcie *pcie)
>>>>  		}
>>>>  	}
>>>>  
>>>> -	/* take the PCIe interface module out of reset */
>>>> -	reset_control_deassert(pcie->pcie_xrst);
>>>> -
>>>>  	/* finally enable PCIe */
>>>>  	value = afi_readl(pcie, AFI_CONFIGURATION);
>>>>  	value |= AFI_CONFIGURATION_EN_FPCI;
>>>> @@ -997,8 +994,6 @@ static void tegra_pcie_disable_controller(struct tegra_pcie *pcie)
>>>>  {
>>>>  	int err;
>>>>  
>>>> -	reset_control_assert(pcie->pcie_xrst);
>>>> -
>>>>  	if (pcie->soc->program_uphy) {
>>>>  		err = tegra_pcie_phy_power_off(pcie);
>>>>  		if (err < 0)
>>>> @@ -1014,13 +1009,11 @@ static void tegra_pcie_power_off(struct tegra_pcie *pcie)
>>>>  	int err;
>>>>  
>>>>  	reset_control_assert(pcie->afi_rst);
>>>> -	reset_control_assert(pcie->pex_rst);
>>>>  
>>>>  	clk_disable_unprepare(pcie->pll_e);
>>>>  	if (soc->has_cml_clk)
>>>>  		clk_disable_unprepare(pcie->cml_clk);
>>>>  	clk_disable_unprepare(pcie->afi_clk);
>>>> -	clk_disable_unprepare(pcie->pex_clk);
>>>>  
>>>>  	if (!dev->pm_domain)
>>>>  		tegra_powergate_power_off(TEGRA_POWERGATE_PCIE);
>>>> @@ -1036,58 +1029,59 @@ static int tegra_pcie_power_on(struct tegra_pcie *pcie)
>>>>  	const struct tegra_pcie_soc *soc = pcie->soc;
>>>>  	int err;
>>>>  
>>>> -	reset_control_assert(pcie->pcie_xrst);
>>>> -	reset_control_assert(pcie->afi_rst);
>>>> -	reset_control_assert(pcie->pex_rst);
>>>> -
>>>> -	if (!dev->pm_domain)
>>>> -		tegra_powergate_power_off(TEGRA_POWERGATE_PCIE);
>>>> -
>>> This code was in place to ensure that PCIe was in a known good state
>>> before following the power up sequence below. You mentioned elsewhere
>>> that there's a regression on Cardhu after applying this series, so
>>> perhaps Cardhu relies on the above hunk?
>> No, Tegra30 and Tegra20 has legacy PHY which are dependent on PEX clk and rst.
>> PHY power on is done in tegra_pcie_enable_controller(), but in this patch I am
>> enabling PEX clk and rst after tegra_pcie_enable_controller(). This caused
>> regression on Cardhu.
>> I realized that sanity test is failing on Cardhu after publishing this series, I will
>> fix the issue in V2.
>> I believe you are talking about the bootloader(uboot) which can enable
>> PCIe partition and reset. To bring the PCIe into good state then we
>> have to take care of clocks as well. AFAIK clock_disable() is not added
>> because it maintains the refcount and any mismatch in the count
>> will thrown warning. I downstream kernel I see pmc driver itself taking
>> care of initial state and there after maintaining the state with refcount.
>> Since bootloader may or may not enable PCIe, Can we get the state fixed
>> in pmc driver instead of fixing it in PCIe driver?
> I don't think we can do that. The PMC driver only knows about which
> clocks and resets need to be controlled as part of which power partition
> if we use power domains. We don't do that on all platforms, so there is
> not enough information.
>
> Even if we had that information, we would still not be able to force the
> clock off because of the reference counting.
>
> Generally, though, the clock don't matter all that much for getting the
> hardware into a good state. All we really care about is that it is put
> into reset so that when we take it out again we start from scratch.
>
> If we can verify that we don't need this anymore, I'm fine with taking
> it out, though. Perhaps do it in a separate patch to make it easier to
> revert if it turns out to be necessary on some platform after all.
>
> Thierry
In one of the Nvidia internal bugs, HW team confirmed that clock matter because
PCIe IP has non re-settable flops. They asked me to use following sequence,
Power up case:
 - Ungate PCIe partition
 - Enable clock
 - Deassert reset
Power down case:
 - Assert reset
 - Disable clock
 - Power gate PCIe partition.
I agree that this clean up should be done in new patch. So when I make new patch set for it,
I will make sure that above sequence is followed to ensure that PCIe IP is in good state.
>
>>>>  	/* enable regulators */
>>>>  	err = regulator_bulk_enable(pcie->num_supplies, pcie->supplies);
>>>>  	if (err < 0)
>>>>  		dev_err(dev, "failed to enable regulators: %d\n", err);
>>>>  
>>>> -	if (dev->pm_domain) {
>>>> -		err = clk_prepare_enable(pcie->pex_clk);
>>>> +	if (!dev->pm_domain) {
>>>> +		err = tegra_powergate_power_on(TEGRA_POWERGATE_PCIE);
>>>>  		if (err) {
>>>> -			dev_err(dev, "failed to enable PEX clock: %d\n", err);
>>>> -			return err;
>>>> +			dev_err(dev, "power ungate failed: %d\n", err);
>>>> +			goto regulator_disable;
>>>>  		}
>>>> -		reset_control_deassert(pcie->pex_rst);
>>>> -	} else {
>>>> -		err = tegra_powergate_sequence_power_up(TEGRA_POWERGATE_PCIE,
>>>> -							pcie->pex_clk,
>>>> -							pcie->pex_rst);
>>>> +		err = tegra_powergate_remove_clamping(TEGRA_POWERGATE_PCIE);
>>>>  		if (err) {
>>>> -			dev_err(dev, "powerup sequence failed: %d\n", err);
>>>> -			return err;
>>>> +			dev_err(dev, "remove clamp failed: %d\n", err);
>>>> +			goto powergate;
>>>>  		}
>>>>  	}
>>>>  
>>>> -	reset_control_deassert(pcie->afi_rst);
>>>> -
>>>>  	err = clk_prepare_enable(pcie->afi_clk);
>>>>  	if (err < 0) {
>>>>  		dev_err(dev, "failed to enable AFI clock: %d\n", err);
>>>> -		return err;
>>>> +		goto powergate;
>>>>  	}
>>>>  
>>>>  	if (soc->has_cml_clk) {
>>>>  		err = clk_prepare_enable(pcie->cml_clk);
>>>>  		if (err < 0) {
>>>>  			dev_err(dev, "failed to enable CML clock: %d\n", err);
>>>> -			return err;
>>>> +			goto afi_clk_disable;
>>>>  		}
>>>>  	}
>>>>  
>>>>  	err = clk_prepare_enable(pcie->pll_e);
>>>>  	if (err < 0) {
>>>>  		dev_err(dev, "failed to enable PLLE clock: %d\n", err);
>>>> -		return err;
>>>> +		goto cml_clk_disable;
>>>>  	}
>>>>  
>>>> +	reset_control_deassert(pcie->afi_rst);
>>>> +
>>>>  	return 0;
>>>> +
>>>> +cml_clk_disable:
>>>> +	if (soc->has_cml_clk)
>>>> +		clk_disable_unprepare(pcie->cml_clk);
>>>> +afi_clk_disable:
>>>> +	clk_disable_unprepare(pcie->afi_clk);
>>>> +powergate:
>>>> +	if (!dev->pm_domain)
>>>> +		tegra_powergate_power_off(TEGRA_POWERGATE_PCIE);
>>>> +regulator_disable:
>>>> +	regulator_bulk_disable(pcie->num_supplies, pcie->supplies);
>>>> +	return err;
>>>>  }
>>>>  
>>>>  static int tegra_pcie_clocks_get(struct tegra_pcie *pcie)
>>>> @@ -2108,7 +2102,12 @@ static void tegra_pcie_enable_ports(struct tegra_pcie *pcie)
>>>>  			 port->index, port->lanes);
>>>>  
>>>>  		tegra_pcie_port_enable(port);
>>>> +	}
>>>> +
>>>> +	/* Start LTSSM from Tegra side */
>>>> +	reset_control_deassert(pcie->pcie_xrst);
>>>>  
>>>> +	list_for_each_entry_safe(port, tmp, &pcie->ports, list) {
>>>>  		if (tegra_pcie_port_check_link(port))
>>>>  			continue;
>>>>  
>>>> @@ -2123,6 +2122,8 @@ static void tegra_pcie_disable_ports(struct tegra_pcie *pcie)
>>>>  {
>>>>  	struct tegra_pcie_port *port, *tmp;
>>>>  
>>>> +	reset_control_assert(pcie->pcie_xrst);
>>>> +
>>>>  	list_for_each_entry_safe(port, tmp, &pcie->ports, list)
>>>>  		tegra_pcie_port_disable(port);
>>>>  }
>>>> @@ -2472,6 +2473,9 @@ static int __maybe_unused tegra_pcie_pm_suspend(struct device *dev)
>>>>  
>>>>  	tegra_pcie_disable_ports(pcie);
>>>>  
>>>> +	reset_control_assert(pcie->pex_rst);
>>>> +	clk_disable_unprepare(pcie->pex_clk);
>>>> +
>>>>  	if (IS_ENABLED(CONFIG_PCI_MSI))
>>>>  		tegra_pcie_disable_msi(pcie);
>>>>  
>>>> @@ -2501,10 +2505,19 @@ static int __maybe_unused tegra_pcie_pm_resume(struct device *dev)
>>>>  	if (IS_ENABLED(CONFIG_PCI_MSI))
>>>>  		tegra_pcie_enable_msi(pcie);
>>>>  
>>>> +	err = clk_prepare_enable(pcie->pex_clk);
>>>> +	if (err) {
>>>> +		dev_err(dev, "failed to enable PEX clock: %d\n", err);
>>>> +		goto disable_controller;
>>>> +	}
>>>> +	reset_control_deassert(pcie->pex_rst);
>>>> +
>>>>  	tegra_pcie_enable_ports(pcie);
>>>>  
>>>>  	return 0;
>>>>  
>>>> +disable_controller:
>>>> +	tegra_pcie_disable_controller(pcie);
>>>>  poweroff:
>>>>  	tegra_pcie_power_off(pcie);
>>>>  
>>> There's quite a bit going on in this patch in general and I find it hard
>>> to review because not all the changes seem related to what you described
>>> in the commit message.
>>>
>>> Can you perhaps try to split out the error cleanup changes into a
>>> separate patch where it makes sense? It seems to me like at least for
>>> tegra_pcie_power_on() we're currently missing all of the cleanup code.
>>> You could make that a preparatory patch that goes before this particular
>>> patch, which will hopefully make this patch easier to review.
>>>
>>> Thierry
>> Okay, I will prepare new patch for error handling and restrict this patch only for
>> sequence correction


  reply index

Thread overview: 106+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-04-11 17:03 [PATCH 00/30] Enable Tegra PCIe root port features Manikanta Maddireddy
2019-04-11 17:03 ` [PATCH 01/30] soc/tegra: pmc: Export tegra_powergate_power_on() Manikanta Maddireddy
2019-04-11 17:03 ` [PATCH 02/30] PCI: tegra: Fix PCIe host power up sequence Manikanta Maddireddy
2019-04-15 11:01   ` Thierry Reding
2019-04-15 14:11     ` Manikanta Maddireddy
2019-04-15 14:30       ` Thierry Reding
2019-04-15 18:14         ` Manikanta Maddireddy [this message]
2019-04-11 17:03 ` [PATCH 03/30] PCI: tegra: Move REFCLK pad settings out of phy_power_on() Manikanta Maddireddy
2019-04-15 11:06   ` Thierry Reding
2019-04-15 14:20     ` Manikanta Maddireddy
2019-04-11 17:03 ` [PATCH 04/30] PCI: tegra: Add PCIe Gen2 link speed support Manikanta Maddireddy
2019-04-15 11:21   ` Thierry Reding
2019-04-15 14:47     ` Manikanta Maddireddy
2019-04-15 15:36       ` Thierry Reding
2019-04-15 15:53         ` Manikanta Maddireddy
2019-04-11 17:03 ` [PATCH 05/30] PCI: tegra: Advertise PCIe Advanced Error Reporting (AER) capability Manikanta Maddireddy
2019-04-15 11:23   ` Thierry Reding
2019-04-15 14:49     ` Manikanta Maddireddy
2019-04-11 17:03 ` [PATCH 06/30] PCI: tegra: Program UPHY electrical settings for Tegra210 Manikanta Maddireddy
2019-04-15 11:29   ` Thierry Reding
2019-04-15 14:55     ` Manikanta Maddireddy
2019-04-15 15:38       ` Thierry Reding
2019-04-11 17:03 ` [PATCH 07/30] PCI: tegra: Enable opportunistic update FC and ACK Manikanta Maddireddy
2019-04-15 11:30   ` Thierry Reding
2019-04-11 17:03 ` [PATCH 08/30] PCI: tegra: Disable AFI dynamic clock gating Manikanta Maddireddy
2019-04-15 11:32   ` Thierry Reding
2019-04-11 17:03 ` [PATCH 09/30] PCI: tegra: Process pending DLL transactions before entering L1 or L2 Manikanta Maddireddy
2019-04-15 11:33   ` Thierry Reding
2019-04-11 17:03 ` [PATCH 10/30] PCI: tegra: Enable PCIe xclk clock clamping Manikanta Maddireddy
2019-04-15 11:37   ` Thierry Reding
2019-04-15 14:58     ` Manikanta Maddireddy
2019-04-11 17:03 ` [PATCH 11/30] PCI: tegra: Increase the deskew retry time Manikanta Maddireddy
2019-04-15 11:39   ` Thierry Reding
2019-04-15 14:58     ` Manikanta Maddireddy
2019-04-11 17:03 ` [PATCH 12/30] PCI: tegra: Add SW fixup for RAW violations Manikanta Maddireddy
2019-04-11 20:01   ` Bjorn Helgaas
2019-04-12  5:59     ` Manikanta Maddireddy
2019-04-15 11:41       ` Thierry Reding
2019-04-15 11:45   ` Thierry Reding
2019-04-15 15:02     ` Manikanta Maddireddy
2019-04-11 17:03 ` [PATCH 13/30] PCI: tegra: Update flow control threshold in Tegra210 Manikanta Maddireddy
2019-04-15 11:47   ` Thierry Reding
2019-04-15 15:05     ` Manikanta Maddireddy
2019-04-23  9:27       ` Manikanta Maddireddy
2019-04-11 17:03 ` [PATCH 14/30] PCI: tegra: Set target speed as Gen1 before link up Manikanta Maddireddy
2019-04-11 20:04   ` Bjorn Helgaas
2019-04-12  6:44     ` Manikanta Maddireddy
2019-04-12 14:35       ` Bjorn Helgaas
2019-04-15 10:43         ` Manikanta Maddireddy
2019-04-15 11:52   ` Thierry Reding
2019-04-15 15:12     ` Manikanta Maddireddy
2019-04-11 17:03 ` [PATCH 15/30] PCI: tegra: Fix PLLE powerdown issue due to CLKREQ# signal Manikanta Maddireddy
2019-04-15 13:17   ` Thierry Reding
2019-04-15 15:14     ` Manikanta Maddireddy
2019-04-11 17:03 ` [PATCH 16/30] PCI: tegra: Program AFI_CACHE* registers only for Tegra20 Manikanta Maddireddy
2019-04-15 13:20   ` Thierry Reding
2019-04-16 10:47     ` Manikanta Maddireddy
2019-04-16 16:11       ` Thierry Reding
2019-04-11 17:03 ` [PATCH 17/30] PCI: tegra: Use switch statements in tegra_pcie_isr() Manikanta Maddireddy
2019-04-15 13:25   ` Thierry Reding
2019-04-15 15:25     ` Manikanta Maddireddy
2019-04-11 17:03 ` [PATCH 18/30] PCI: tegra: Change PRSNT_SENSE irq log to debug Manikanta Maddireddy
2019-04-11 17:03 ` [PATCH 19/30] PCI: tegra: Use legacy irq for port service drivers Manikanta Maddireddy
2019-04-15 13:35   ` Thierry Reding
2019-04-11 17:03 ` [PATCH 20/30] PCI: tegra: Add AFI_PEX2_CTRL reg offset as part of soc struct Manikanta Maddireddy
2019-04-15 13:31   ` Thierry Reding
2019-04-11 17:03 ` [PATCH 21/30] PCI: tegra: Add "pci" type check before parsing child device tree node Manikanta Maddireddy
2019-04-15 13:37   ` Thierry Reding
2019-04-15 15:30     ` Manikanta Maddireddy
2019-04-15 15:42       ` Thierry Reding
2019-04-11 17:03 ` [PATCH 22/30] PCI: tegra: Access endpoint config only if PCIe link is up Manikanta Maddireddy
2019-04-11 20:15   ` Bjorn Helgaas
2019-04-12  7:00     ` Manikanta Maddireddy
2019-04-12 14:50       ` Bjorn Helgaas
2019-04-15 11:36         ` Manikanta Maddireddy
2019-04-15 13:45           ` Thierry Reding
2019-04-15 13:52             ` Thierry Reding
2019-04-15 14:04           ` Bjorn Helgaas
2019-04-15 15:43             ` Manikanta Maddireddy
2019-04-23 20:24               ` Bjorn Helgaas
2019-04-11 17:03 ` [PATCH 23/30] dt-bindings: pci: tegra: Document PCIe DPD pinctrl optional prop Manikanta Maddireddy
2019-04-15 14:07   ` Thierry Reding
2019-04-15 15:48     ` Manikanta Maddireddy
2019-04-11 17:03 ` [PATCH 24/30] arm64: tegra: Add PEX DPD states as pinctrl properties Manikanta Maddireddy
2019-04-11 17:03 ` [PATCH 25/30] PCI: tegra: Put PEX CLK & BIAS pads in DPD mode Manikanta Maddireddy
2019-04-15 14:11   ` Thierry Reding
2019-04-11 17:03 ` [PATCH 26/30] dt-bindings: pci: tegra: Document nvidia,plat-gpios optional prop Manikanta Maddireddy
2019-04-11 20:18   ` Bjorn Helgaas
2019-04-12  7:01     ` Manikanta Maddireddy
2019-04-15 14:16   ` Thierry Reding
2019-04-15 17:58     ` Manikanta Maddireddy
2019-04-16 15:34       ` Thierry Reding
2019-04-17 11:22         ` Manikanta Maddireddy
2019-04-17 15:19           ` Thierry Reding
2019-04-17 18:26             ` Manikanta Maddireddy
2019-04-11 17:03 ` [PATCH 27/30] PCI: tegra: Add support to configure platform GPIOs Manikanta Maddireddy
2019-04-11 17:03 ` [PATCH 28/30] dt-bindings: pci: tegra: Document nvidia,rst-gpio optional prop Manikanta Maddireddy
2019-04-15 14:20   ` Thierry Reding
2019-04-15 18:01     ` Manikanta Maddireddy
2019-04-29 18:33     ` Rob Herring
2019-04-11 17:03 ` [PATCH 29/30] PCI: tegra: Add support for GPIO based PCIe reset Manikanta Maddireddy
2019-04-15 14:20   ` Thierry Reding
2019-04-15 18:03     ` Manikanta Maddireddy
2019-04-11 17:03 ` [PATCH 30/30] PCI: tegra: Change link retry log level to INFO Manikanta Maddireddy
2019-04-15 14:23   ` Thierry Reding
2019-04-15 18:05     ` Manikanta Maddireddy

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=33f945e1-ca97-b8ab-4687-62c5a0e4f45c@nvidia.com \
    --to=mmaddireddy@nvidia.com \
    --cc=bhelgaas@google.com \
    --cc=devicetree@vger.kernel.org \
    --cc=jonathanh@nvidia.com \
    --cc=linux-pci@vger.kernel.org \
    --cc=linux-tegra@vger.kernel.org \
    --cc=lorenzo.pieralisi@arm.com \
    --cc=mark.rutland@arm.com \
    --cc=robh+dt@kernel.org \
    --cc=thierry.reding@gmail.com \
    --cc=vidyas@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-PCI Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-pci/0 linux-pci/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-pci linux-pci/ https://lore.kernel.org/linux-pci \
		linux-pci@vger.kernel.org linux-pci@archiver.kernel.org
	public-inbox-index linux-pci


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-pci


AGPL code for this site: git clone https://public-inbox.org/ public-inbox