RE: [PATCHv4 26/28] PCI: mobiveil: ls_pcie_g4: add Workaround for A-011451

From: "Z.q. Hou" <zhiqiang.hou@nxp.com>
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
	"linux-arm-kernel@lists.infradead.org" 
	<linux-arm-kernel@lists.infradead.org>,
	"devicetree@vger.kernel.org" <devicetree@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"robh+dt@kernel.org" <robh+dt@kernel.org>,
	"mark.rutland@arm.com" <mark.rutland@arm.com>,
	"l.subrahmanya@mobiveil.co.in" <l.subrahmanya@mobiveil.co.in>,
	"shawnguo@kernel.org" <shawnguo@kernel.org>,
	Leo Li <leoyang.li@nxp.com>,
	"lorenzo.pieralisi@arm.com" <lorenzo.pieralisi@arm.com>,
	"catalin.marinas@arm.com" <catalin.marinas@arm.com>,
	"will.deacon@arm.com" <will.deacon@arm.com>,
	Mingkai Hu <mingkai.hu@nxp.com>,
	"M.h. Lian" <minghuan.lian@nxp.com>,
	Xiaowei Bao <xiaowei.bao@nxp.com>
Subject: RE: [PATCHv4 26/28] PCI: mobiveil: ls_pcie_g4: add Workaround for A-011451
Date: Wed, 13 Mar 2019 14:49:54 +0000	[thread overview]
Message-ID: <AM6PR04MB57819D27C1F4E8A0BFD6E2C5844A0@AM6PR04MB5781.eurprd04.prod.outlook.com> (raw)
In-Reply-To: <20190312133452.GA210027@google.com>

Hi Bjorn,

Thanks a lot for your comments!

> -----Original Message-----
> From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> Sent: 2019年3月12日 21:35
> To: Z.q. Hou <zhiqiang.hou@nxp.com>
> Cc: linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> robh+dt@kernel.org; mark.rutland@arm.com; l.subrahmanya@mobiveil.co.in;
> shawnguo@kernel.org; Leo Li <leoyang.li@nxp.com>;
> lorenzo.pieralisi@arm.com; catalin.marinas@arm.com;
> will.deacon@arm.com; Mingkai Hu <mingkai.hu@nxp.com>; M.h. Lian
> <minghuan.lian@nxp.com>; Xiaowei Bao <xiaowei.bao@nxp.com>
> Subject: Re: [PATCHv4 26/28] PCI: mobiveil: ls_pcie_g4: add Workaround for
> A-011451
> 
> On Tue, Mar 12, 2019 at 09:34:17AM +0000, Z.q. Hou wrote:
> > Hi Bjorn,
> >
> > Thanks a lot for your comments!
> >
> > > -----Original Message-----
> > > From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> > > Sent: 2019年3月12日 1:35
> > > To: Z.q. Hou <zhiqiang.hou@nxp.com>
> > > Cc: linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
> > > devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> > > robh+dt@kernel.org; mark.rutland@arm.com;
> > > robh+l.subrahmanya@mobiveil.co.in;
> > > shawnguo@kernel.org; Leo Li <leoyang.li@nxp.com>;
> > > lorenzo.pieralisi@arm.com; catalin.marinas@arm.com;
> > > will.deacon@arm.com; Mingkai Hu <mingkai.hu@nxp.com>; M.h. Lian
> > > <minghuan.lian@nxp.com>; Xiaowei Bao <xiaowei.bao@nxp.com>
> > > Subject: Re: [PATCHv4 26/28] PCI: mobiveil: ls_pcie_g4: add
> > > Workaround for
> > > A-011451
> > >
> > > On Mon, Mar 11, 2019 at 09:33:32AM +0000, Z.q. Hou wrote:
> > > > From: Hou Zhiqiang <Zhiqiang.Hou@nxp.com>
> > > >
> > > > When LX2 PCIe controller is sending multiple split completions and
> > > > ACK latency expires indicating that ACK should be send at priority.
> > > > But because of large number of split completions and FC update
> > > > DLLP, the controller does not give priority to ACK transmission.
> > > > This results into ACK latency timer timeout error at the link
> > > > partner and the pending TLPs are replayed by the link partner again.
> > > >
> > > > Workaround:
> > > > 1. Reduce the ACK latency timeout value to a very small value.
> > > > 2. Restrict the number of completions from the LX2 PCIe controller
> > > >    to 1, by changing the Max Read Request Size (MRRS) of link partner
> > > >    to the same value as Max Packet size (MPS).
> > > >
> > > > This patch implemented part 1, the part 2 can be set by kernel
> > > > parameter 'pci=pcie_bus_perf'
> > >
> > > If I understand correctly, you're saying that LX2160A Rev1.0 will
> > > only work correctly if you have this patch applied AND you boot with
> > > "pci=pcie_bus_perf".
> >
> > Your understanding is correct.
> >
> > > That might be OK if these rev 1.0 parts are only used in the lab and
> > > are never shipped to customers.  But if these parts are ever shipped
> > > to customers, I don't think it's acceptable for them to have to
> > > figure out that they must boot with "pci=pcie_bus_perf".  Yes, you
> > > can document that in release notes, but it's still a poor user
> > > experience, and users will forget, and they will see mysterious
> hard-to-debug issues.
> 
> Since you didn't respond here, I assume these rev 1.0 parts have been or will
> be shipped to end users?  Please confirm.
> 

Double checked with the marketing and project manager, the rev1.0 will not be shipped to end users, so please ignore this patch and I will stop to send all rev1.0 workaround patches.

> > > Maybe there's a way for you to automatically set that pcie_bus_perf
> mode?
> > > With a dmesg note to indicate that you're overriding any mode the
> > > user may have selected?
> >
> > Actually we don't have a way to set the pcie_bus_perf automatically
> > under Linux, we give this parameter in kernel cmdline.  Do you have
> > any advice about how to set this parameter automatically under Linux?
> 
> The beauty of Linux being open-source is that if you need something that
> doesn't exist, you are empowered to create it.
> 
> We do have one place already (cns3xxx_pcie_hw_init()) that explicitly sets
> pcie_bus_config.  You might be able to do something similar.
>

Thanks for your advice!

> > > > This ERRATA is only for LX2160A Rev1.0, and it will be fixed in
> > > > Rev2.0.
> > > >
> > > > Signed-off-by: Hou Zhiqiang <Zhiqiang.Hou@nxp.com>
> > > > ---
> > > > V4:
> > > >  - no change
> > > >
> > > >  .../pci/controller/mobiveil/pci-layerscape-gen4.c | 15
> +++++++++++++++
> > > >  drivers/pci/controller/mobiveil/pcie-mobiveil.h   |  4 ++++
> > > >  2 files changed, 19 insertions(+)
> > > >
> > > > diff --git a/drivers/pci/controller/mobiveil/pci-layerscape-gen4.c
> > > > b/drivers/pci/controller/mobiveil/pci-layerscape-gen4.c
> > > > index d2c5dbbd5e3c..20ce146788ca 100644
> > > > --- a/drivers/pci/controller/mobiveil/pci-layerscape-gen4.c
> > > > +++ b/drivers/pci/controller/mobiveil/pci-layerscape-gen4.c
> > > > @@ -82,12 +82,27 @@ static bool ls_pcie_g4_is_bridge(struct
> > > > ls_pcie_g4
> > > *pcie)
> > > >  	return header_type == PCI_HEADER_TYPE_BRIDGE;  }
> > > >
> > > > +static void workaround_A011451(struct ls_pcie_g4 *pcie) {
> > > > +	struct mobiveil_pcie *mv_pci = pcie->pci;
> > > > +	u32 val;
> > > > +
> > > > +	/* Set ACK latency timeout */
> > > > +	val = csr_readl(mv_pci, GPEX_ACK_REPLAY_TO);
> > > > +	val &= ~(ACK_LAT_TO_VAL_MASK << ACK_LAT_TO_VAL_SHIFT);
> > > > +	val |= (4 << ACK_LAT_TO_VAL_SHIFT);
> > > > +	csr_writel(mv_pci, val, GPEX_ACK_REPLAY_TO); }
> > > > +
> > > >  static int ls_pcie_g4_host_init(struct mobiveil_pcie *pci)  {
> > > >  	struct ls_pcie_g4 *pcie = to_ls_pcie_g4(pci);
> > > >
> > > >  	pcie->rev = csr_readb(pci, PCI_REVISION_ID);
> > > >
> > > > +	if (pcie->rev == REV_1_0)
> > > > +		workaround_A011451(pcie);
> > > > +
> > > >  	return 0;
> > > >  }
> > > >
> > > > diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > > > b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > > > index ab43de5e4b2b..f0e2e4ae09b5 100644
> > > > --- a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > > > +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > > > @@ -85,6 +85,10 @@
> > > >  #define PAB_AXI_AMAP_PEX_WIN_H(win)
> 	PAB_REG_ADDR(0x0bac,
> > > win)
> > > >  #define PAB_INTP_AXI_PIO_CLASS		0x474
> > > >
> > > > +#define GPEX_ACK_REPLAY_TO		0x438
> > > > +#define  ACK_LAT_TO_VAL_MASK		0x1fff
> > > > +#define  ACK_LAT_TO_VAL_SHIFT		0
> > > > +
> > > >  #define PAB_PEX_AMAP_CTRL(win)		PAB_REG_ADDR(0x4ba0,
> > > win)
> > > >  #define  AMAP_CTRL_EN_SHIFT		0
> > > >  #define  AMAP_CTRL_TYPE_SHIFT		1
> > > > --
> > > > 2.17.1
> > > >
> >

Thanks,
Zhiqiang