[PATCH V3] PCI: tegra: Enable Relaxed Ordering only for Tegra20 & Tegra30

[PATCH V3] PCI: tegra: Enable Relaxed Ordering only for Tegra20 & Tegra30

[Vidya Sagar]

Currently Relaxed Ordering bit in the configuration space is enabled for
all PCIe devices as the quirk uses PCI_ANY_ID for both Vendor-ID and
Device-ID, but, as per the Technical Reference Manual of Tegra20 which is
available at https://developer.nvidia.com/embedded/downloads#?search=tegra%202
in Sec 34.1, it is mentioned that Relexed Ordering bit needs to be enabled in
its root ports to avoid deadlock in hardware. The same is applicable for
Tegra30 as well though it is not explicitly mentioned in Tegra30 TRM document,
but the same must not be extended to root ports of other Tegra SoCs or
other hosts as the same issue doesn't exist there.

Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
---
V3:
* Modified commit message to make it more precise and explicit

V2:
* Modified commit message to include reference to Tegra20 TRM document.

 drivers/pci/controller/pci-tegra.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/controller/pci-tegra.c b/drivers/pci/controller/pci-tegra.c
index 9cc03a2549c0..241760aa15bd 100644
--- a/drivers/pci/controller/pci-tegra.c
+++ b/drivers/pci/controller/pci-tegra.c
@@ -787,12 +787,15 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x0bf1, tegra_pcie_fixup_class);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x0e1c, tegra_pcie_fixup_class);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x0e1d, tegra_pcie_fixup_class);
 
-/* Tegra PCIE requires relaxed ordering */
+/* Tegra20 and Tegra30 PCIE requires relaxed ordering */
 static void tegra_pcie_relax_enable(struct pci_dev *dev)
 {
 	pcie_capability_set_word(dev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_RELAX_EN);
 }
-DECLARE_PCI_FIXUP_FINAL(PCI_ANY_ID, PCI_ANY_ID, tegra_pcie_relax_enable);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x0bf0, tegra_pcie_relax_enable);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x0bf1, tegra_pcie_relax_enable);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x0e1c, tegra_pcie_relax_enable);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x0e1d, tegra_pcie_relax_enable);
 
 static int tegra_pcie_request_resources(struct tegra_pcie *pcie)
 {
-- 
2.17.1

Re: [PATCH V3] PCI: tegra: Enable Relaxed Ordering only for Tegra20 & Tegra30

[Lorenzo Pieralisi]

On Thu, Jul 04, 2019 at 08:34:28PM +0530, Vidya Sagar wrote:
> Currently Relaxed Ordering bit in the configuration space is enabled for
> all PCIe devices as the quirk uses PCI_ANY_ID for both Vendor-ID and
> Device-ID, but, as per the Technical Reference Manual of Tegra20 which is
> available at https://developer.nvidia.com/embedded/downloads#?search=tegra%202
> in Sec 34.1, it is mentioned that Relexed Ordering bit needs to be enabled in
> its root ports to avoid deadlock in hardware. The same is applicable for
> Tegra30 as well though it is not explicitly mentioned in Tegra30 TRM document,
> but the same must not be extended to root ports of other Tegra SoCs or
> other hosts as the same issue doesn't exist there.
> 
> Signed-off-by: Vidya Sagar <vidyas@nvidia.com>

You forgot Thierry's ACK, I added it back but next time pay more
attention please.

You should link the versions through eg git send-email
--in-reply-to=Message-Id so that it is easier to follow.

> ---
> V3:
> * Modified commit message to make it more precise and explicit
> 
> V2:
> * Modified commit message to include reference to Tegra20 TRM document.
> 
>  drivers/pci/controller/pci-tegra.c | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)

I applied it to pci/tegra after rewriting the whole commit log and
adding a Fixes: tag that you or someone at Nvidia will follow up;
I will check.

Please have a look and report back any issues you notice.

Lorenzo

> 
> diff --git a/drivers/pci/controller/pci-tegra.c b/drivers/pci/controller/pci-tegra.c
> index 9cc03a2549c0..241760aa15bd 100644
> --- a/drivers/pci/controller/pci-tegra.c
> +++ b/drivers/pci/controller/pci-tegra.c
> @@ -787,12 +787,15 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x0bf1, tegra_pcie_fixup_class);
>  DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x0e1c, tegra_pcie_fixup_class);
>  DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x0e1d, tegra_pcie_fixup_class);
>  
> -/* Tegra PCIE requires relaxed ordering */
> +/* Tegra20 and Tegra30 PCIE requires relaxed ordering */
>  static void tegra_pcie_relax_enable(struct pci_dev *dev)
>  {
>  	pcie_capability_set_word(dev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_RELAX_EN);
>  }
> -DECLARE_PCI_FIXUP_FINAL(PCI_ANY_ID, PCI_ANY_ID, tegra_pcie_relax_enable);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x0bf0, tegra_pcie_relax_enable);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x0bf1, tegra_pcie_relax_enable);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x0e1c, tegra_pcie_relax_enable);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x0e1d, tegra_pcie_relax_enable);
>  
>  static int tegra_pcie_request_resources(struct tegra_pcie *pcie)
>  {
> -- 
> 2.17.1
> 

Re: [PATCH V3] PCI: tegra: Enable Relaxed Ordering only for Tegra20 & Tegra30

[Jon Hunter]

Hi Lorenzo,

On 04/07/2019 17:09, Lorenzo Pieralisi wrote:
> On Thu, Jul 04, 2019 at 08:34:28PM +0530, Vidya Sagar wrote:
>> Currently Relaxed Ordering bit in the configuration space is enabled for
>> all PCIe devices as the quirk uses PCI_ANY_ID for both Vendor-ID and
>> Device-ID, but, as per the Technical Reference Manual of Tegra20 which is
>> available at https://developer.nvidia.com/embedded/downloads#?search=tegra%202
>> in Sec 34.1, it is mentioned that Relexed Ordering bit needs to be enabled in
>> its root ports to avoid deadlock in hardware. The same is applicable for
>> Tegra30 as well though it is not explicitly mentioned in Tegra30 TRM document,
>> but the same must not be extended to root ports of other Tegra SoCs or
>> other hosts as the same issue doesn't exist there.
>>
>> Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
> 
> You forgot Thierry's ACK, I added it back but next time pay more
> attention please.
> 
> You should link the versions through eg git send-email
> --in-reply-to=Message-Id so that it is easier to follow.
> 
>> ---
>> V3:
>> * Modified commit message to make it more precise and explicit
>>
>> V2:
>> * Modified commit message to include reference to Tegra20 TRM document.
>>
>>  drivers/pci/controller/pci-tegra.c | 7 +++++--
>>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> I applied it to pci/tegra after rewriting the whole commit log and
> adding a Fixes: tag that you or someone at Nvidia will follow up;
> I will check.

I had a chat with Vidya last night to understand the issue, so now I
have a good understanding of the problem this has caused, which is very
unfortunate indeed!

Vidya mentioned that you would like us to get this backported to stable
branches. Please correct me if I am wrong here. We can certainly do
that, but I do have concerns about doing so, for non-Tegra devices
inparticularly, given that this has been around for sometime now. Hence,
I was wondering if we should leave this soak in the mainline for at
least a kernel release cycle before doing so. I really don't want to
break stable for anyone. What are your thoughts on this?

Cheers
Jon

-- 
nvpublic

Re: [PATCH V3] PCI: tegra: Enable Relaxed Ordering only for Tegra20 & Tegra30

[Lorenzo Pieralisi]

[+Greg]

On Fri, Jul 05, 2019 at 09:57:25AM +0100, Jon Hunter wrote:
> Hi Lorenzo,
> 
> On 04/07/2019 17:09, Lorenzo Pieralisi wrote:
> > On Thu, Jul 04, 2019 at 08:34:28PM +0530, Vidya Sagar wrote:
> >> Currently Relaxed Ordering bit in the configuration space is enabled for
> >> all PCIe devices as the quirk uses PCI_ANY_ID for both Vendor-ID and
> >> Device-ID, but, as per the Technical Reference Manual of Tegra20 which is
> >> available at https://developer.nvidia.com/embedded/downloads#?search=tegra%202
> >> in Sec 34.1, it is mentioned that Relexed Ordering bit needs to be enabled in
> >> its root ports to avoid deadlock in hardware. The same is applicable for
> >> Tegra30 as well though it is not explicitly mentioned in Tegra30 TRM document,
> >> but the same must not be extended to root ports of other Tegra SoCs or
> >> other hosts as the same issue doesn't exist there.
> >>
> >> Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
> > 
> > You forgot Thierry's ACK, I added it back but next time pay more
> > attention please.
> > 
> > You should link the versions through eg git send-email
> > --in-reply-to=Message-Id so that it is easier to follow.
> > 
> >> ---
> >> V3:
> >> * Modified commit message to make it more precise and explicit
> >>
> >> V2:
> >> * Modified commit message to include reference to Tegra20 TRM document.
> >>
> >>  drivers/pci/controller/pci-tegra.c | 7 +++++--
> >>  1 file changed, 5 insertions(+), 2 deletions(-)
> > 
> > I applied it to pci/tegra after rewriting the whole commit log and
> > adding a Fixes: tag that you or someone at Nvidia will follow up;
> > I will check.
> 
> I had a chat with Vidya last night to understand the issue, so now I
> have a good understanding of the problem this has caused, which is very
> unfortunate indeed!
> 
> Vidya mentioned that you would like us to get this backported to stable
> branches. Please correct me if I am wrong here. We can certainly do
> that, but I do have concerns about doing so, for non-Tegra devices
> inparticularly, given that this has been around for sometime now. Hence,
> I was wondering if we should leave this soak in the mainline for at
> least a kernel release cycle before doing so. I really don't want to
> break stable for anyone. What are your thoughts on this?

I looped in Greg to pick his brain, since it is unclear how we should
apply the stable kernel rules on this specific patch. Basically, this
technically is not a bug, it is just bad code that forces a feature on
ALL kernels that compile the PCI Tegra Controller driver in the kernel.
I would really really want to have this patch applied to all stable
kernels but first as you said it is better to apply it to mainline and
check it does not cause any issues on any other arch/platform then
we can think about backporting it to stable kernels.

I am not happy to force Relaxed Ordering on any PCIe device on
any platform/arch compiling PCI Tegra controller in, so somehow
we must rectify this situation, this is gross as I said before.

I added a Fixes: tag but I am not sure it is appropriate for the
above to happen and I think it is better for this patch to brew
at least a release cycle in the mainline before sending it back
to stable kernels, if appropriate, how to translate this into
stable kernel rules that's the question I am asking.

Thanks,
Lorenzo

Re: [PATCH V3] PCI: tegra: Enable Relaxed Ordering only for Tegra20 & Tegra30

[Greg Kroah-Hartman]

On Fri, Jul 05, 2019 at 10:38:59AM +0100, Lorenzo Pieralisi wrote:
> [+Greg]
> 
> On Fri, Jul 05, 2019 at 09:57:25AM +0100, Jon Hunter wrote:
> > Hi Lorenzo,
> > 
> > On 04/07/2019 17:09, Lorenzo Pieralisi wrote:
> > > On Thu, Jul 04, 2019 at 08:34:28PM +0530, Vidya Sagar wrote:
> > >> Currently Relaxed Ordering bit in the configuration space is enabled for
> > >> all PCIe devices as the quirk uses PCI_ANY_ID for both Vendor-ID and
> > >> Device-ID, but, as per the Technical Reference Manual of Tegra20 which is
> > >> available at https://developer.nvidia.com/embedded/downloads#?search=tegra%202
> > >> in Sec 34.1, it is mentioned that Relexed Ordering bit needs to be enabled in
> > >> its root ports to avoid deadlock in hardware. The same is applicable for
> > >> Tegra30 as well though it is not explicitly mentioned in Tegra30 TRM document,
> > >> but the same must not be extended to root ports of other Tegra SoCs or
> > >> other hosts as the same issue doesn't exist there.
> > >>
> > >> Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
> > > 
> > > You forgot Thierry's ACK, I added it back but next time pay more
> > > attention please.
> > > 
> > > You should link the versions through eg git send-email
> > > --in-reply-to=Message-Id so that it is easier to follow.
> > > 
> > >> ---
> > >> V3:
> > >> * Modified commit message to make it more precise and explicit
> > >>
> > >> V2:
> > >> * Modified commit message to include reference to Tegra20 TRM document.
> > >>
> > >>  drivers/pci/controller/pci-tegra.c | 7 +++++--
> > >>  1 file changed, 5 insertions(+), 2 deletions(-)
> > > 
> > > I applied it to pci/tegra after rewriting the whole commit log and
> > > adding a Fixes: tag that you or someone at Nvidia will follow up;
> > > I will check.
> > 
> > I had a chat with Vidya last night to understand the issue, so now I
> > have a good understanding of the problem this has caused, which is very
> > unfortunate indeed!
> > 
> > Vidya mentioned that you would like us to get this backported to stable
> > branches. Please correct me if I am wrong here. We can certainly do
> > that, but I do have concerns about doing so, for non-Tegra devices
> > inparticularly, given that this has been around for sometime now. Hence,
> > I was wondering if we should leave this soak in the mainline for at
> > least a kernel release cycle before doing so. I really don't want to
> > break stable for anyone. What are your thoughts on this?
> 
> I looped in Greg to pick his brain, since it is unclear how we should
> apply the stable kernel rules on this specific patch. Basically, this
> technically is not a bug, it is just bad code that forces a feature on
> ALL kernels that compile the PCI Tegra Controller driver in the kernel.
> I would really really want to have this patch applied to all stable
> kernels but first as you said it is better to apply it to mainline and
> check it does not cause any issues on any other arch/platform then
> we can think about backporting it to stable kernels.

You all have read the stable kernel rules, right:
    https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html

Patches have to be upstream first.

After it is merged in Linus's tree, post the patches with the git commit
ids to stable@vger and we will be glad to review them there.

thanks,

greg k-h

Re: [PATCH V3] PCI: tegra: Enable Relaxed Ordering only for Tegra20 & Tegra30

[Lorenzo Pieralisi]

On Fri, Jul 05, 2019 at 12:23:30PM +0200, Greg Kroah-Hartman wrote:
> On Fri, Jul 05, 2019 at 10:38:59AM +0100, Lorenzo Pieralisi wrote:
> > [+Greg]
> > 
> > On Fri, Jul 05, 2019 at 09:57:25AM +0100, Jon Hunter wrote:
> > > Hi Lorenzo,
> > > 
> > > On 04/07/2019 17:09, Lorenzo Pieralisi wrote:
> > > > On Thu, Jul 04, 2019 at 08:34:28PM +0530, Vidya Sagar wrote:
> > > >> Currently Relaxed Ordering bit in the configuration space is enabled for
> > > >> all PCIe devices as the quirk uses PCI_ANY_ID for both Vendor-ID and
> > > >> Device-ID, but, as per the Technical Reference Manual of Tegra20 which is
> > > >> available at https://developer.nvidia.com/embedded/downloads#?search=tegra%202
> > > >> in Sec 34.1, it is mentioned that Relexed Ordering bit needs to be enabled in
> > > >> its root ports to avoid deadlock in hardware. The same is applicable for
> > > >> Tegra30 as well though it is not explicitly mentioned in Tegra30 TRM document,
> > > >> but the same must not be extended to root ports of other Tegra SoCs or
> > > >> other hosts as the same issue doesn't exist there.
> > > >>
> > > >> Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
> > > > 
> > > > You forgot Thierry's ACK, I added it back but next time pay more
> > > > attention please.
> > > > 
> > > > You should link the versions through eg git send-email
> > > > --in-reply-to=Message-Id so that it is easier to follow.
> > > > 
> > > >> ---
> > > >> V3:
> > > >> * Modified commit message to make it more precise and explicit
> > > >>
> > > >> V2:
> > > >> * Modified commit message to include reference to Tegra20 TRM document.
> > > >>
> > > >>  drivers/pci/controller/pci-tegra.c | 7 +++++--
> > > >>  1 file changed, 5 insertions(+), 2 deletions(-)
> > > > 
> > > > I applied it to pci/tegra after rewriting the whole commit log and
> > > > adding a Fixes: tag that you or someone at Nvidia will follow up;
> > > > I will check.
> > > 
> > > I had a chat with Vidya last night to understand the issue, so now I
> > > have a good understanding of the problem this has caused, which is very
> > > unfortunate indeed!
> > > 
> > > Vidya mentioned that you would like us to get this backported to stable
> > > branches. Please correct me if I am wrong here. We can certainly do
> > > that, but I do have concerns about doing so, for non-Tegra devices
> > > inparticularly, given that this has been around for sometime now. Hence,
> > > I was wondering if we should leave this soak in the mainline for at
> > > least a kernel release cycle before doing so. I really don't want to
> > > break stable for anyone. What are your thoughts on this?
> > 
> > I looped in Greg to pick his brain, since it is unclear how we should
> > apply the stable kernel rules on this specific patch. Basically, this
> > technically is not a bug, it is just bad code that forces a feature on
> > ALL kernels that compile the PCI Tegra Controller driver in the kernel.
> > I would really really want to have this patch applied to all stable
> > kernels but first as you said it is better to apply it to mainline and
> > check it does not cause any issues on any other arch/platform then
> > we can think about backporting it to stable kernels.
> 
> You all have read the stable kernel rules, right:
>     https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
> 
> Patches have to be upstream first.
> 
> After it is merged in Linus's tree, post the patches with the git commit
> ids to stable@vger and we will be glad to review them there.

Hi Greg,

thanks, it was just to understand if the "Fixes:" tag would
automatically make it a stable kernel candidate when it hits Linus'
tree; I will drop it from the patch and we will post the patches to
stable@vger when/if we want it considered for stable, we just do not
want it to be automatically picked up, that's it.

Thanks for chiming in,
Lorenzo

> thanks,
> 
> greg k-h

Re: [PATCH V3] PCI: tegra: Enable Relaxed Ordering only for Tegra20 & Tegra30

[Greg Kroah-Hartman]

On Fri, Jul 05, 2019 at 11:35:06AM +0100, Lorenzo Pieralisi wrote:
> On Fri, Jul 05, 2019 at 12:23:30PM +0200, Greg Kroah-Hartman wrote:
> > On Fri, Jul 05, 2019 at 10:38:59AM +0100, Lorenzo Pieralisi wrote:
> > > [+Greg]
> > > 
> > > On Fri, Jul 05, 2019 at 09:57:25AM +0100, Jon Hunter wrote:
> > > > Hi Lorenzo,
> > > > 
> > > > On 04/07/2019 17:09, Lorenzo Pieralisi wrote:
> > > > > On Thu, Jul 04, 2019 at 08:34:28PM +0530, Vidya Sagar wrote:
> > > > >> Currently Relaxed Ordering bit in the configuration space is enabled for
> > > > >> all PCIe devices as the quirk uses PCI_ANY_ID for both Vendor-ID and
> > > > >> Device-ID, but, as per the Technical Reference Manual of Tegra20 which is
> > > > >> available at https://developer.nvidia.com/embedded/downloads#?search=tegra%202
> > > > >> in Sec 34.1, it is mentioned that Relexed Ordering bit needs to be enabled in
> > > > >> its root ports to avoid deadlock in hardware. The same is applicable for
> > > > >> Tegra30 as well though it is not explicitly mentioned in Tegra30 TRM document,
> > > > >> but the same must not be extended to root ports of other Tegra SoCs or
> > > > >> other hosts as the same issue doesn't exist there.
> > > > >>
> > > > >> Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
> > > > > 
> > > > > You forgot Thierry's ACK, I added it back but next time pay more
> > > > > attention please.
> > > > > 
> > > > > You should link the versions through eg git send-email
> > > > > --in-reply-to=Message-Id so that it is easier to follow.
> > > > > 
> > > > >> ---
> > > > >> V3:
> > > > >> * Modified commit message to make it more precise and explicit
> > > > >>
> > > > >> V2:
> > > > >> * Modified commit message to include reference to Tegra20 TRM document.
> > > > >>
> > > > >>  drivers/pci/controller/pci-tegra.c | 7 +++++--
> > > > >>  1 file changed, 5 insertions(+), 2 deletions(-)
> > > > > 
> > > > > I applied it to pci/tegra after rewriting the whole commit log and
> > > > > adding a Fixes: tag that you or someone at Nvidia will follow up;
> > > > > I will check.
> > > > 
> > > > I had a chat with Vidya last night to understand the issue, so now I
> > > > have a good understanding of the problem this has caused, which is very
> > > > unfortunate indeed!
> > > > 
> > > > Vidya mentioned that you would like us to get this backported to stable
> > > > branches. Please correct me if I am wrong here. We can certainly do
> > > > that, but I do have concerns about doing so, for non-Tegra devices
> > > > inparticularly, given that this has been around for sometime now. Hence,
> > > > I was wondering if we should leave this soak in the mainline for at
> > > > least a kernel release cycle before doing so. I really don't want to
> > > > break stable for anyone. What are your thoughts on this?
> > > 
> > > I looped in Greg to pick his brain, since it is unclear how we should
> > > apply the stable kernel rules on this specific patch. Basically, this
> > > technically is not a bug, it is just bad code that forces a feature on
> > > ALL kernels that compile the PCI Tegra Controller driver in the kernel.
> > > I would really really want to have this patch applied to all stable
> > > kernels but first as you said it is better to apply it to mainline and
> > > check it does not cause any issues on any other arch/platform then
> > > we can think about backporting it to stable kernels.
> > 
> > You all have read the stable kernel rules, right:
> >     https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
> > 
> > Patches have to be upstream first.
> > 
> > After it is merged in Linus's tree, post the patches with the git commit
> > ids to stable@vger and we will be glad to review them there.
> 
> Hi Greg,
> 
> thanks, it was just to understand if the "Fixes:" tag would
> automatically make it a stable kernel candidate when it hits Linus'
> tree;

No it will not.  It _might_ get picked up by the "autobot" tool we have,
but it might not as well.  Never rely on that for a patch you know you
want into the stable tree.  Always put a specific cc: stable on it.

> I will drop it from the patch and we will post the patches to
> stable@vger when/if we want it considered for stable, we just do not
> want it to be automatically picked up, that's it.

Good idea.

thanks,

greg k-h

Re: [PATCH V3] PCI: tegra: Enable Relaxed Ordering only for Tegra20 & Tegra30

[Jon Hunter]

On 05/07/2019 10:38, Lorenzo Pieralisi wrote:
> [+Greg]
> 
> On Fri, Jul 05, 2019 at 09:57:25AM +0100, Jon Hunter wrote:
>> Hi Lorenzo,
>>
>> On 04/07/2019 17:09, Lorenzo Pieralisi wrote:
>>> On Thu, Jul 04, 2019 at 08:34:28PM +0530, Vidya Sagar wrote:
>>>> Currently Relaxed Ordering bit in the configuration space is enabled for
>>>> all PCIe devices as the quirk uses PCI_ANY_ID for both Vendor-ID and
>>>> Device-ID, but, as per the Technical Reference Manual of Tegra20 which is
>>>> available at https://developer.nvidia.com/embedded/downloads#?search=tegra%202
>>>> in Sec 34.1, it is mentioned that Relexed Ordering bit needs to be enabled in
>>>> its root ports to avoid deadlock in hardware. The same is applicable for
>>>> Tegra30 as well though it is not explicitly mentioned in Tegra30 TRM document,
>>>> but the same must not be extended to root ports of other Tegra SoCs or
>>>> other hosts as the same issue doesn't exist there.
>>>>
>>>> Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
>>>
>>> You forgot Thierry's ACK, I added it back but next time pay more
>>> attention please.
>>>
>>> You should link the versions through eg git send-email
>>> --in-reply-to=Message-Id so that it is easier to follow.
>>>
>>>> ---
>>>> V3:
>>>> * Modified commit message to make it more precise and explicit
>>>>
>>>> V2:
>>>> * Modified commit message to include reference to Tegra20 TRM document.
>>>>
>>>>  drivers/pci/controller/pci-tegra.c | 7 +++++--
>>>>  1 file changed, 5 insertions(+), 2 deletions(-)
>>>
>>> I applied it to pci/tegra after rewriting the whole commit log and
>>> adding a Fixes: tag that you or someone at Nvidia will follow up;
>>> I will check.
>>
>> I had a chat with Vidya last night to understand the issue, so now I
>> have a good understanding of the problem this has caused, which is very
>> unfortunate indeed!
>>
>> Vidya mentioned that you would like us to get this backported to stable
>> branches. Please correct me if I am wrong here. We can certainly do
>> that, but I do have concerns about doing so, for non-Tegra devices
>> inparticularly, given that this has been around for sometime now. Hence,
>> I was wondering if we should leave this soak in the mainline for at
>> least a kernel release cycle before doing so. I really don't want to
>> break stable for anyone. What are your thoughts on this?
> 
> I looped in Greg to pick his brain, since it is unclear how we should
> apply the stable kernel rules on this specific patch. Basically, this
> technically is not a bug, it is just bad code that forces a feature on
> ALL kernels that compile the PCI Tegra Controller driver in the kernel.
> I would really really want to have this patch applied to all stable
> kernels but first as you said it is better to apply it to mainline and
> check it does not cause any issues on any other arch/platform then
> we can think about backporting it to stable kernels.
> 
> I am not happy to force Relaxed Ordering on any PCIe device on
> any platform/arch compiling PCI Tegra controller in, so somehow
> we must rectify this situation, this is gross as I said before.

Yes understood. Let's plan to sync up on this once v5.3 is out and see
how the land lies. We have an internal issue filed to track this and so
we should not forget!

Cheers
Jon

-- 
nvpublic

Re: [PATCH V3] PCI: tegra: Enable Relaxed Ordering only for Tegra20 & Tegra30

[Jon Hunter]

Hi Lorenzo,

On 09/07/2019 12:02, Jon Hunter wrote:
> On 05/07/2019 10:38, Lorenzo Pieralisi wrote:
>> [+Greg]
>>
>> On Fri, Jul 05, 2019 at 09:57:25AM +0100, Jon Hunter wrote:
>>> Hi Lorenzo,
>>>
>>> On 04/07/2019 17:09, Lorenzo Pieralisi wrote:
>>>> On Thu, Jul 04, 2019 at 08:34:28PM +0530, Vidya Sagar wrote:
>>>>> Currently Relaxed Ordering bit in the configuration space is enabled for
>>>>> all PCIe devices as the quirk uses PCI_ANY_ID for both Vendor-ID and
>>>>> Device-ID, but, as per the Technical Reference Manual of Tegra20 which is
>>>>> available at https://developer.nvidia.com/embedded/downloads#?search=tegra%202
>>>>> in Sec 34.1, it is mentioned that Relexed Ordering bit needs to be enabled in
>>>>> its root ports to avoid deadlock in hardware. The same is applicable for
>>>>> Tegra30 as well though it is not explicitly mentioned in Tegra30 TRM document,
>>>>> but the same must not be extended to root ports of other Tegra SoCs or
>>>>> other hosts as the same issue doesn't exist there.
>>>>>
>>>>> Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
>>>>
>>>> You forgot Thierry's ACK, I added it back but next time pay more
>>>> attention please.
>>>>
>>>> You should link the versions through eg git send-email
>>>> --in-reply-to=Message-Id so that it is easier to follow.
>>>>
>>>>> ---
>>>>> V3:
>>>>> * Modified commit message to make it more precise and explicit
>>>>>
>>>>> V2:
>>>>> * Modified commit message to include reference to Tegra20 TRM document.
>>>>>
>>>>>  drivers/pci/controller/pci-tegra.c | 7 +++++--
>>>>>  1 file changed, 5 insertions(+), 2 deletions(-)
>>>>
>>>> I applied it to pci/tegra after rewriting the whole commit log and
>>>> adding a Fixes: tag that you or someone at Nvidia will follow up;
>>>> I will check.
>>>
>>> I had a chat with Vidya last night to understand the issue, so now I
>>> have a good understanding of the problem this has caused, which is very
>>> unfortunate indeed!
>>>
>>> Vidya mentioned that you would like us to get this backported to stable
>>> branches. Please correct me if I am wrong here. We can certainly do
>>> that, but I do have concerns about doing so, for non-Tegra devices
>>> inparticularly, given that this has been around for sometime now. Hence,
>>> I was wondering if we should leave this soak in the mainline for at
>>> least a kernel release cycle before doing so. I really don't want to
>>> break stable for anyone. What are your thoughts on this?
>>
>> I looped in Greg to pick his brain, since it is unclear how we should
>> apply the stable kernel rules on this specific patch. Basically, this
>> technically is not a bug, it is just bad code that forces a feature on
>> ALL kernels that compile the PCI Tegra Controller driver in the kernel.
>> I would really really want to have this patch applied to all stable
>> kernels but first as you said it is better to apply it to mainline and
>> check it does not cause any issues on any other arch/platform then
>> we can think about backporting it to stable kernels.
>>
>> I am not happy to force Relaxed Ordering on any PCIe device on
>> any platform/arch compiling PCI Tegra controller in, so somehow
>> we must rectify this situation, this is gross as I said before.
> 
> Yes understood. Let's plan to sync up on this once v5.3 is out and see
> how the land lies. We have an internal issue filed to track this and so
> we should not forget!

Please let us know if your preference it still to push this back to
stable. I assume that there has been no fallout from this change.

Cheers
Jon

-- 
nvpublic

Re: [PATCH V3] PCI: tegra: Enable Relaxed Ordering only for Tegra20 & Tegra30

[Lorenzo Pieralisi]

On Tue, Nov 05, 2019 at 10:50:28AM +0000, Jon Hunter wrote:
> Hi Lorenzo,
> 
> On 09/07/2019 12:02, Jon Hunter wrote:
> > On 05/07/2019 10:38, Lorenzo Pieralisi wrote:
> >> [+Greg]
> >>
> >> On Fri, Jul 05, 2019 at 09:57:25AM +0100, Jon Hunter wrote:
> >>> Hi Lorenzo,
> >>>
> >>> On 04/07/2019 17:09, Lorenzo Pieralisi wrote:
> >>>> On Thu, Jul 04, 2019 at 08:34:28PM +0530, Vidya Sagar wrote:
> >>>>> Currently Relaxed Ordering bit in the configuration space is enabled for
> >>>>> all PCIe devices as the quirk uses PCI_ANY_ID for both Vendor-ID and
> >>>>> Device-ID, but, as per the Technical Reference Manual of Tegra20 which is
> >>>>> available at https://developer.nvidia.com/embedded/downloads#?search=tegra%202
> >>>>> in Sec 34.1, it is mentioned that Relexed Ordering bit needs to be enabled in
> >>>>> its root ports to avoid deadlock in hardware. The same is applicable for
> >>>>> Tegra30 as well though it is not explicitly mentioned in Tegra30 TRM document,
> >>>>> but the same must not be extended to root ports of other Tegra SoCs or
> >>>>> other hosts as the same issue doesn't exist there.
> >>>>>
> >>>>> Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
> >>>>
> >>>> You forgot Thierry's ACK, I added it back but next time pay more
> >>>> attention please.
> >>>>
> >>>> You should link the versions through eg git send-email
> >>>> --in-reply-to=Message-Id so that it is easier to follow.
> >>>>
> >>>>> ---
> >>>>> V3:
> >>>>> * Modified commit message to make it more precise and explicit
> >>>>>
> >>>>> V2:
> >>>>> * Modified commit message to include reference to Tegra20 TRM document.
> >>>>>
> >>>>>  drivers/pci/controller/pci-tegra.c | 7 +++++--
> >>>>>  1 file changed, 5 insertions(+), 2 deletions(-)
> >>>>
> >>>> I applied it to pci/tegra after rewriting the whole commit log and
> >>>> adding a Fixes: tag that you or someone at Nvidia will follow up;
> >>>> I will check.
> >>>
> >>> I had a chat with Vidya last night to understand the issue, so now I
> >>> have a good understanding of the problem this has caused, which is very
> >>> unfortunate indeed!
> >>>
> >>> Vidya mentioned that you would like us to get this backported to stable
> >>> branches. Please correct me if I am wrong here. We can certainly do
> >>> that, but I do have concerns about doing so, for non-Tegra devices
> >>> inparticularly, given that this has been around for sometime now. Hence,
> >>> I was wondering if we should leave this soak in the mainline for at
> >>> least a kernel release cycle before doing so. I really don't want to
> >>> break stable for anyone. What are your thoughts on this?
> >>
> >> I looped in Greg to pick his brain, since it is unclear how we should
> >> apply the stable kernel rules on this specific patch. Basically, this
> >> technically is not a bug, it is just bad code that forces a feature on
> >> ALL kernels that compile the PCI Tegra Controller driver in the kernel.
> >> I would really really want to have this patch applied to all stable
> >> kernels but first as you said it is better to apply it to mainline and
> >> check it does not cause any issues on any other arch/platform then
> >> we can think about backporting it to stable kernels.
> >>
> >> I am not happy to force Relaxed Ordering on any PCIe device on
> >> any platform/arch compiling PCI Tegra controller in, so somehow
> >> we must rectify this situation, this is gross as I said before.
> > 
> > Yes understood. Let's plan to sync up on this once v5.3 is out and see
> > how the land lies. We have an internal issue filed to track this and so
> > we should not forget!
> 
> Please let us know if your preference it still to push this back to
> stable. I assume that there has been no fallout from this change.

No fallout reported to the best of my knowledge, yes we should go ahead
and push it to stable kernels, thanks for the reminder.

Cheers,
Lorenzo

Re: [PATCH V3] PCI: tegra: Enable Relaxed Ordering only for Tegra20 & Tegra30

[Bjorn Helgaas]

On Thu, Jul 04, 2019 at 08:34:28PM +0530, Vidya Sagar wrote:
> Currently Relaxed Ordering bit in the configuration space is enabled for
> all PCIe devices as the quirk uses PCI_ANY_ID for both Vendor-ID and
> Device-ID, but, as per the Technical Reference Manual of Tegra20 which is
> available at https://developer.nvidia.com/embedded/downloads#?search=tegra%202
> in Sec 34.1, it is mentioned that Relaxed Ordering bit needs to be enabled in
> its root ports to avoid deadlock in hardware. The same is applicable for
> Tegra30 as well though it is not explicitly mentioned in Tegra30 TRM document,
> but the same must not be extended to root ports of other Tegra SoCs or
> other hosts as the same issue doesn't exist there.

While researching another thread about RO [1], I got concerned about
setting RO for root ports.

Setting RO for *endpoints* makes sense: that allows (but does not
require) the endpoint to issue writes that don't require strong
ordering.

Setting RO for *root ports* seems more problematic.  It allows the
root port to issue PCIe writes that don't require strong ordering.
These would be CPU MMIO writes to devices.  But Linux currently does
not have a way for drivers to indicate that some MMIO writes need to
be ordered while others do not, and I think drivers assume that all
MMIO writes are performed in order. 

We merged this patch as 7be142caabc4 ("PCI: tegra: Enable Relaxed
Ordering only for Tegra20 & Tegra30") [2], so Tegra20 and Tegra30 root
ports are allowed (but again, not required) to set the RO bit for MMIO
writes initiated by a CPU.

Because I think drivers *rely* on MMIO writes being strongly ordered,
this is a potential problem.  I think we should probably consider
explicitly *disabling* RO in all root ports (with exceptions for
quirks like this) in case it's set by any firmware.

So the question is, how do Tegra20 and Tegra30 actually work?  Do they
ever actually set the RO bit for these MMIO writes?  If so, I think
drivers are really at risk, and we probably should log some kind of
warning.

But if Tegra20 and Tegra30 just need "Enable Relaxed Ordering" set as
a bug workaround and they never actually initiate PCIe writes with the
RO bit set, maybe we should add a comment to that effect, but there
should be no actual problem.

[1] https://lore.kernel.org/r/20210830123704.221494-2-verdre@v0yd.nl
[2] https://git.kernel.org/linus/7be142caabc4

> Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
> ---
> V3:
> * Modified commit message to make it more precise and explicit
> 
> V2:
> * Modified commit message to include reference to Tegra20 TRM document.
> 
>  drivers/pci/controller/pci-tegra.c | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/pci/controller/pci-tegra.c b/drivers/pci/controller/pci-tegra.c
> index 9cc03a2549c0..241760aa15bd 100644
> --- a/drivers/pci/controller/pci-tegra.c
> +++ b/drivers/pci/controller/pci-tegra.c
> @@ -787,12 +787,15 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x0bf1, tegra_pcie_fixup_class);
>  DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x0e1c, tegra_pcie_fixup_class);
>  DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x0e1d, tegra_pcie_fixup_class);
>  
> -/* Tegra PCIE requires relaxed ordering */
> +/* Tegra20 and Tegra30 PCIE requires relaxed ordering */
>  static void tegra_pcie_relax_enable(struct pci_dev *dev)
>  {
>  	pcie_capability_set_word(dev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_RELAX_EN);
>  }
> -DECLARE_PCI_FIXUP_FINAL(PCI_ANY_ID, PCI_ANY_ID, tegra_pcie_relax_enable);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x0bf0, tegra_pcie_relax_enable);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x0bf1, tegra_pcie_relax_enable);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x0e1c, tegra_pcie_relax_enable);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x0e1d, tegra_pcie_relax_enable);
>  
>  static int tegra_pcie_request_resources(struct tegra_pcie *pcie)
>  {
> -- 
> 2.17.1
> 

Re: [PATCH V3] PCI: tegra: Enable Relaxed Ordering only for Tegra20 & Tegra30

[Bjorn Helgaas]

[+cc Keith]

On Wed, Sep 01, 2021 at 03:40:47PM -0500, Bjorn Helgaas wrote:
> On Thu, Jul 04, 2019 at 08:34:28PM +0530, Vidya Sagar wrote:
> > Currently Relaxed Ordering bit in the configuration space is enabled for
> > all PCIe devices as the quirk uses PCI_ANY_ID for both Vendor-ID and
> > Device-ID, but, as per the Technical Reference Manual of Tegra20 which is
> > available at https://developer.nvidia.com/embedded/downloads#?search=tegra%202
> > in Sec 34.1, it is mentioned that Relaxed Ordering bit needs to be enabled in
> > its root ports to avoid deadlock in hardware. The same is applicable for
> > Tegra30 as well though it is not explicitly mentioned in Tegra30 TRM document,
> > but the same must not be extended to root ports of other Tegra SoCs or
> > other hosts as the same issue doesn't exist there.
> 
> While researching another thread about RO [1], I got concerned about
> setting RO for root ports.
> 
> Setting RO for *endpoints* makes sense: that allows (but does not
> require) the endpoint to issue writes that don't require strong
> ordering.
> 
> Setting RO for *root ports* seems more problematic.  It allows the
> root port to issue PCIe writes that don't require strong ordering.
> These would be CPU MMIO writes to devices.  But Linux currently does
> not have a way for drivers to indicate that some MMIO writes need to
> be ordered while others do not, and I think drivers assume that all
> MMIO writes are performed in order. 
> 
> We merged this patch as 7be142caabc4 ("PCI: tegra: Enable Relaxed
> Ordering only for Tegra20 & Tegra30") [2], so Tegra20 and Tegra30 root
> ports are allowed (but again, not required) to set the RO bit for MMIO
> writes initiated by a CPU.
> 
> Because I think drivers *rely* on MMIO writes being strongly ordered,
> this is a potential problem.  I think we should probably consider
> explicitly *disabling* RO in all root ports (with exceptions for
> quirks like this) in case it's set by any firmware.
> 
> So the question is, how do Tegra20 and Tegra30 actually work?  Do they
> ever actually set the RO bit for these MMIO writes?  If so, I think
> drivers are really at risk, and we probably should log some kind of
> warning.

Keith reminded me that writel_relaxed() is a way for drivers to
specify that they don't need strong ordering.  So I think the question
really is whether Tegra20 and Tegra30 set the RO bit for normal
writel() writes.  If they set RO for writel_relaxed() writes, that
should be perfectly fine.

> But if Tegra20 and Tegra30 just need "Enable Relaxed Ordering" set as
> a bug workaround and they never actually initiate PCIe writes with the
> RO bit set, maybe we should add a comment to that effect, but there
> should be no actual problem.
> 
> [1] https://lore.kernel.org/r/20210830123704.221494-2-verdre@v0yd.nl
> [2] https://git.kernel.org/linus/7be142caabc4
> 
> > Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
> > ---
> > V3:
> > * Modified commit message to make it more precise and explicit
> > 
> > V2:
> > * Modified commit message to include reference to Tegra20 TRM document.
> > 
> >  drivers/pci/controller/pci-tegra.c | 7 +++++--
> >  1 file changed, 5 insertions(+), 2 deletions(-)
> > 
> > diff --git a/drivers/pci/controller/pci-tegra.c b/drivers/pci/controller/pci-tegra.c
> > index 9cc03a2549c0..241760aa15bd 100644
> > --- a/drivers/pci/controller/pci-tegra.c
> > +++ b/drivers/pci/controller/pci-tegra.c
> > @@ -787,12 +787,15 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x0bf1, tegra_pcie_fixup_class);
> >  DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x0e1c, tegra_pcie_fixup_class);
> >  DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x0e1d, tegra_pcie_fixup_class);
> >  
> > -/* Tegra PCIE requires relaxed ordering */
> > +/* Tegra20 and Tegra30 PCIE requires relaxed ordering */
> >  static void tegra_pcie_relax_enable(struct pci_dev *dev)
> >  {
> >  	pcie_capability_set_word(dev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_RELAX_EN);
> >  }
> > -DECLARE_PCI_FIXUP_FINAL(PCI_ANY_ID, PCI_ANY_ID, tegra_pcie_relax_enable);
> > +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x0bf0, tegra_pcie_relax_enable);
> > +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x0bf1, tegra_pcie_relax_enable);
> > +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x0e1c, tegra_pcie_relax_enable);
> > +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x0e1d, tegra_pcie_relax_enable);
> >  
> >  static int tegra_pcie_request_resources(struct tegra_pcie *pcie)
> >  {
> > -- 
> > 2.17.1
> >