linux-pci.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: "Pali Rohár" <pali@kernel.org>
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: "Marek Behún" <marek.behun@nic.cz>,
	linux-pci@vger.kernel.org, "Jason Cooper" <jason@lakedaemon.net>,
	"Andrew Lunn" <andrew@lunn.ch>,
	"Gregory Clement" <gregory.clement@bootlin.com>,
	"Sebastian Hesselbarth" <sebastian.hesselbarth@gmail.com>,
	"Thomas Petazzoni" <thomas.petazzoni@bootlin.com>,
	"Lorenzo Pieralisi" <lorenzo.pieralisi@arm.com>,
	"Andrew Murray" <amurray@thegoodpenguin.co.uk>,
	"Remi Pommarel" <repk@triplefau.lt>,
	"Tomasz Maciej Nowak" <tmn505@gmail.com>,
	Xogium <contact@xogium.me>, "Rob Herring" <robh@kernel.org>
Subject: Re: [PATCH v2 3/9] PCI: aardvark: improve link training
Date: Thu, 23 Apr 2020 20:56:27 +0200	[thread overview]
Message-ID: <20200423185627.dm2id6k7da7uvwen@pali> (raw)
In-Reply-To: <20200423183914.GA201745@google.com>

On Thursday 23 April 2020 13:39:14 Bjorn Helgaas wrote:
> [+cc Rob]
> 
> On Tue, Apr 21, 2020 at 01:16:55PM +0200, Marek Behún wrote:
> > Currently the aardvark driver trains link in PCIe gen2 mode. This may
> > cause some buggy gen 1 cards (such as Compex WLE900VX) to be unstable or
> > even not detected. Moreover when ASPM code tries to retrain link second
> > time, these cards may stop responding and link goes down. If gen1 is
> > used this does not happen.
> 
> Does this patch make the retrain done by ASPM reliable?

Yes, after this patch all my tested cards work fine. I tried to enable
ASPM for tested cards and there were no problem with link training
issued by ASPM kernel code.

So this patch makes link retrain done by ASPM kernel code reliable.

> If aardvark can't work reliably at gen2, you may need to just restrict it to gen1.

aardvark gen2 does not work reliable with WLE900VX gen1 card. It works
reliable with WLE200NX gen1 card and works also reliable with gen2
WLE1216V5-20 card.

Marek also tested aardvark gen2 with sata gen2 cards and it worked
reliable too.

So result is that aardvark gen2 works reliable with gen2 cards and does
not work reliable with gen1 cards.

Therefore we decided to set aardvark gen to match card gen and then
aardvark with tested cards work reliable.

Also more people on espressobin forums were manually patching kernel for
particular gen1 cards to set aardvark to gen1 module because they had
same symptoms as we are observing.

> > Unconditionally forcing gen1 is not a good solution since it may have
> > performance impact on gen2 cards.
> > 
> > To overcome this, read 'max-link-speed' property (as defined in PCI
> > device tree bindings) and use this as max gen mode. Then iteratively try
> > link training at this mode or lower until successful. After successful
> > link training choose final controlled gen based on Negotiated Link Speed
> > from Link Status register, which should match card speed.
> > 
> > Signed-off-by: Pali Rohár <pali@kernel.org>
> > Signed-off-by: Marek Behún <marek.behun@nic.cz>
> > ---
> >  drivers/pci/controller/pci-aardvark.c | 111 ++++++++++++++++++++------
> >  1 file changed, 86 insertions(+), 25 deletions(-)
> > 
> > diff --git a/drivers/pci/controller/pci-aardvark.c b/drivers/pci/controller/pci-aardvark.c
> > index 551d98174613..606bae1e7a88 100644
> > --- a/drivers/pci/controller/pci-aardvark.c
> > +++ b/drivers/pci/controller/pci-aardvark.c
> > @@ -40,6 +40,7 @@
> >  #define PCIE_CORE_LINK_CTRL_STAT_REG				0xd0
> >  #define     PCIE_CORE_LINK_L0S_ENTRY				BIT(0)
> >  #define     PCIE_CORE_LINK_TRAINING				BIT(5)
> > +#define     PCIE_CORE_LINK_SPEED_SHIFT				16
> >  #define     PCIE_CORE_LINK_WIDTH_SHIFT				20
> >  #define PCIE_CORE_ERR_CAPCTL_REG				0x118
> >  #define     PCIE_CORE_ERR_CAPCTL_ECRC_CHK_TX			BIT(5)
> > @@ -201,6 +202,7 @@ struct advk_pcie {
> >  	struct mutex msi_used_lock;
> >  	u16 msi_msg;
> >  	int root_bus_nr;
> > +	int link_gen;
> >  	struct pci_bridge_emul bridge;
> >  };
> >  
> > @@ -225,20 +227,16 @@ static int advk_pcie_link_up(struct advk_pcie *pcie)
> >  
> >  static int advk_pcie_wait_for_link(struct advk_pcie *pcie)
> >  {
> > -	struct device *dev = &pcie->pdev->dev;
> >  	int retries;
> >  
> >  	/* check if the link is up or not */
> >  	for (retries = 0; retries < LINK_WAIT_MAX_RETRIES; retries++) {
> > -		if (advk_pcie_link_up(pcie)) {
> > -			dev_info(dev, "link up\n");
> > +		if (advk_pcie_link_up(pcie))
> >  			return 0;
> > -		}
> >  
> >  		usleep_range(LINK_WAIT_USLEEP_MIN, LINK_WAIT_USLEEP_MAX);
> >  	}
> >  
> > -	dev_err(dev, "link never came up\n");
> >  	return -ETIMEDOUT;
> >  }
> >  
> > @@ -253,6 +251,84 @@ static void advk_pcie_wait_for_retrain(struct advk_pcie *pcie)
> >  	}
> >  }
> >  
> > +static int advk_pcie_train_at_gen(struct advk_pcie *pcie, int gen)
> > +{
> > +	int ret, neg_gen;
> > +	u32 reg;
> > +
> > +	/* Setup link speed */
> > +	reg = advk_readl(pcie, PCIE_CORE_CTRL0_REG);
> > +	reg &= ~PCIE_GEN_SEL_MSK;
> > +	if (gen == 2)
> > +		reg |= SPEED_GEN_2;
> > +	else
> > +		reg |= SPEED_GEN_1;
> > +	advk_writel(pcie, reg, PCIE_CORE_CTRL0_REG);
> > +
> > +	/*
> > +	 * Enable link training. This is not needed in every call to this
> > +	 * function, just once suffices, but it does not break anything either.
> > +	 */
> > +	reg = advk_readl(pcie, PCIE_CORE_CTRL0_REG);
> > +	reg |= LINK_TRAINING_EN;
> > +	advk_writel(pcie, reg, PCIE_CORE_CTRL0_REG);
> > +
> > +	/*
> > +	 * Start link training immediately after enabling it. This solves
> > +	 * problems for some buggy cards.
> > +	 */
> > +	reg = advk_readl(pcie, PCIE_CORE_LINK_CTRL_STAT_REG);
> > +	reg |= PCIE_CORE_LINK_TRAINING;
> > +	advk_writel(pcie, reg, PCIE_CORE_LINK_CTRL_STAT_REG);
> > +
> > +	ret = advk_pcie_wait_for_link(pcie);
> > +	if (ret)
> > +		return ret;
> > +
> > +	reg = advk_readl(pcie, PCIE_CORE_LINK_CTRL_STAT_REG);
> > +	neg_gen = (reg >> PCIE_CORE_LINK_SPEED_SHIFT) & 0xf;
> > +
> > +	return neg_gen;
> > +}
> > +
> > +static void advk_pcie_train_link(struct advk_pcie *pcie)
> > +{
> > +	struct device *dev = &pcie->pdev->dev;
> > +	int neg_gen = -1, gen;
> > +
> > +	/*
> > +	 * Try link training at link gen specified by device tree property
> > +	 * 'max-link-speed' (defaults to 2, since this controller does not
> > +	 * support higher gen). If this fails, iteratively train at lower gen.
> > +	 */
> > +	for (gen = pcie->link_gen; gen > 0; --gen) {
> > +		neg_gen = advk_pcie_train_at_gen(pcie, gen);
> > +		if (neg_gen > 0)
> > +			break;
> > +	}
> > +
> > +	if (neg_gen < 0)
> > +		goto err;
> > +
> > +	/*
> > +	 * After successful training if negotiated gen is lower than requested,
> > +	 * train again on negotiated gen. This solves some stability issues for
> > +	 * some buggy gen1 cards.
> > +	 */
> > +	if (neg_gen < gen) {
> > +		gen = neg_gen;
> > +		neg_gen = advk_pcie_train_at_gen(pcie, gen);
> > +	}
> > +
> > +	if (neg_gen == gen) {
> > +		dev_info(dev, "link up at gen %i\n", gen);
> > +		return;
> > +	}
> > +
> > +err:
> > +	dev_err(dev, "link never came up\n");
> > +}
> > +
> >  static void advk_pcie_setup_hw(struct advk_pcie *pcie)
> >  {
> >  	u32 reg;
> > @@ -288,12 +364,6 @@ static void advk_pcie_setup_hw(struct advk_pcie *pcie)
> >  		PCIE_CORE_CTRL2_TD_ENABLE;
> >  	advk_writel(pcie, reg, PCIE_CORE_CTRL2_REG);
> >  
> > -	/* Set GEN2 */
> > -	reg = advk_readl(pcie, PCIE_CORE_CTRL0_REG);
> > -	reg &= ~PCIE_GEN_SEL_MSK;
> > -	reg |= SPEED_GEN_2;
> > -	advk_writel(pcie, reg, PCIE_CORE_CTRL0_REG);
> > -
> >  	/* Set lane X1 */
> >  	reg = advk_readl(pcie, PCIE_CORE_CTRL0_REG);
> >  	reg &= ~LANE_CNT_MSK;
> > @@ -341,20 +411,7 @@ static void advk_pcie_setup_hw(struct advk_pcie *pcie)
> >  	 */
> >  	msleep(PCI_PM_D3COLD_WAIT);
> >  
> > -	/* Enable link training */
> > -	reg = advk_readl(pcie, PCIE_CORE_CTRL0_REG);
> > -	reg |= LINK_TRAINING_EN;
> > -	advk_writel(pcie, reg, PCIE_CORE_CTRL0_REG);
> > -
> > -	/*
> > -	 * Start link training immediately after enabling it. This solves
> > -	 * problems for some buggy cards.
> > -	 */
> > -	reg = advk_readl(pcie, PCIE_CORE_LINK_CTRL_STAT_REG);
> > -	reg |= PCIE_CORE_LINK_TRAINING;
> > -	advk_writel(pcie, reg, PCIE_CORE_LINK_CTRL_STAT_REG);
> > -
> > -	advk_pcie_wait_for_link(pcie);
> > +	advk_pcie_train_link(pcie);
> >  
> >  	reg = advk_readl(pcie, PCIE_CORE_CMD_STATUS_REG);
> >  	reg |= PCIE_CORE_CMD_MEM_ACCESS_EN |
> > @@ -988,6 +1045,10 @@ static int advk_pcie_probe(struct platform_device *pdev)
> >  	}
> >  	pcie->root_bus_nr = bus->start;
> >  
> > +	pcie->link_gen = of_pci_get_max_link_speed(dev->of_node);
> > +	if (pcie->link_gen < 1 || pcie->link_gen > 2)
> 
> This is a DT error, isn't it?  Maybe should warn about it and mention
> how you're dealing with it?

Well, negative and zero link_gen is DT error.

But documentation in pci.txt says:

- max-link-speed:
   If present this property specifies PCI gen for link capability.  Host
   drivers could add this as a strategy to avoid unnecessary operation for
   unsupported link speed, for instance, trying to do training for
   unsupported link speed, etc.  Must be '4' for gen4, '3' for gen3, '2'
   for gen2, and '1' for gen1. Any other values are invalid.

I understand this as maximal property which driver could decrease if
needed. I think that '4' and '3' are also valid values as they describe
upper limit and we could map them to 2.

> > +		pcie->link_gen = 2;
> > +
> >  	advk_pcie_setup_hw(pcie);
> >  
> >  	advk_sw_pci_bridge_init(pcie);
> > -- 
> > 2.24.1
> > 

  reply	other threads:[~2020-04-23 18:56 UTC|newest]

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-04-21 11:16 [PATCH v2 0/9] PCI: aardvark: Fix support for Turris MOX and Compex wifi cards Marek Behún
2020-04-21 11:16 ` [PATCH v2 1/9] PCI: aardvark: train link immediately after enabling training Marek Behún
2020-04-21 11:16 ` [PATCH v2 2/9] PCI: aardvark: don't write to read-only register Marek Behún
2020-04-23 17:27   ` Bjorn Helgaas
2020-04-23 17:51     ` Pali Rohár
2020-04-21 11:16 ` [PATCH v2 3/9] PCI: aardvark: improve link training Marek Behún
2020-04-23 18:39   ` Bjorn Helgaas
2020-04-23 18:56     ` Pali Rohár [this message]
2020-04-24 12:49       ` Pali Rohár
2020-04-21 11:16 ` [PATCH v2 4/9] PCI: aardvark: issue PERST via GPIO Marek Behún
2020-04-23 18:41   ` Bjorn Helgaas
2020-04-23 19:02     ` Pali Rohár
2020-04-23 22:17       ` Bjorn Helgaas
2020-04-23 22:23         ` Pali Rohár
2020-04-23 22:40           ` Bjorn Helgaas
2020-04-24  8:13             ` Pali Rohár
2020-04-24  9:25   ` Pali Rohár
2020-04-21 11:16 ` [PATCH v2 5/9] PCI: aardvark: add FIXME comment for PCIE_CORE_CMD_STATUS_REG access Marek Behún
2020-04-23 18:44   ` Bjorn Helgaas
2020-04-23 19:06     ` Pali Rohár
2020-04-21 11:16 ` [PATCH v2 6/9] PCI: aardvark: add PHY support Marek Behún
2020-04-21 11:16 ` [PATCH v2 7/9] dt-bindings: PCI: aardvark: describe new properties Marek Behún
2020-05-11 18:24   ` Rob Herring
2020-04-21 11:17 ` [PATCH v2 8/9] arm64: dts: marvell: armada-37xx: set pcie_reset_pin to gpio function Marek Behún
2020-04-21 11:17 ` [PATCH v2 9/9] arm64: dts: marvell: armada-37xx: move PCIe comphy handle property Marek Behún
2020-04-21 11:42 ` [PATCH v2 0/9] PCI: aardvark: Fix support for Turris MOX and Compex wifi cards Pali Rohár

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200423185627.dm2id6k7da7uvwen@pali \
    --to=pali@kernel.org \
    --cc=amurray@thegoodpenguin.co.uk \
    --cc=andrew@lunn.ch \
    --cc=contact@xogium.me \
    --cc=gregory.clement@bootlin.com \
    --cc=helgaas@kernel.org \
    --cc=jason@lakedaemon.net \
    --cc=linux-pci@vger.kernel.org \
    --cc=lorenzo.pieralisi@arm.com \
    --cc=marek.behun@nic.cz \
    --cc=repk@triplefau.lt \
    --cc=robh@kernel.org \
    --cc=sebastian.hesselbarth@gmail.com \
    --cc=thomas.petazzoni@bootlin.com \
    --cc=tmn505@gmail.com \
    --subject='Re: [PATCH v2 3/9] PCI: aardvark: improve link training' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).