Re: [PATCH V6 02/15] PCI/PME: Export pcie_pme_disable_msi() & pcie_pme_no_msi() APIs

From: Bjorn Helgaas <helgaas@kernel.org>
To: Vidya Sagar <vidyas@nvidia.com>
Cc: Christoph Hellwig <hch@infradead.org>,
	lorenzo.pieralisi@arm.com, robh+dt@kernel.org,
	mark.rutland@arm.com, thierry.reding@gmail.com,
	jonathanh@nvidia.com, kishon@ti.com, catalin.marinas@arm.com,
	will.deacon@arm.com, jingoohan1@gmail.com,
	gustavo.pimentel@synopsys.com, mperttunen@nvidia.com,
	linux-pci@vger.kernel.org, devicetree@vger.kernel.org,
	linux-tegra@vger.kernel.org, linux-kernel@vger.kernel.org,
	linux-arm-kernel@lists.infradead.org, kthota@nvidia.com,
	mmaddireddy@nvidia.com, sagar.tv@gmail.com
Subject: Re: [PATCH V6 02/15] PCI/PME: Export pcie_pme_disable_msi() & pcie_pme_no_msi() APIs
Date: Mon, 20 May 2019 12:57:29 -0500	[thread overview]
Message-ID: <20190520175729.GC49425@google.com> (raw)
In-Reply-To: <bf220eba-f9d7-81f3-6b75-db463c74fbfa@nvidia.com>

On Sat, May 18, 2019 at 07:28:29AM +0530, Vidya Sagar wrote:
> On 5/18/2019 12:25 AM, Bjorn Helgaas wrote:
> > On Fri, May 17, 2019 at 11:23:36PM +0530, Vidya Sagar wrote:
> > > On 5/17/2019 6:54 PM, Bjorn Helgaas wrote:
> > > > Do you have "lspci -vvxxx" output for the root ports handy?
> > > > 
> > > > If there's some clue in the standard config space that would tell us
> > > > that MSI works for some events but not others, we could make the PCI
> > > > core pay attention it.  That would be the best solution because it
> > > > wouldn't require Tegra-specific code.
> > > 
> > > Here is the output of 'lspci vvxxx' for one of Tegra194's root ports.
> > 
> > Thanks!
> > 
> > This port advertises both MSI and MSI-X, and neither one is enabled.
> > This particular port doesn't have a slot, so hotplug isn't applicable
> > to it.
> > 
> > But if I understand correctly, if MSI or MSI-X were enabled and the
> > port had a slot, the port would generate MSI/MSI-X hotplug interrupts.
> > But PME and AER events would still cause INTx interrupts (even with
> > MSI or MSI-X enabled).
> > 
> > Do I have that right?  I just want to make sure that the reason for
> > PME being INTx is a permanent hardware choice and that it's not
> > related to MSI and MSI-X currently being disabled.
>
> Yes. Thats right. Its hardware choice that our hardware engineers made to
> use INTx for PME instead of MSI irrespective of MSI/MSI-X enabled/disabled
> in the root port.

Here are more spec references that seem applicable:

  - PCIe r4.0, sec 7.7.1.2 (Message Control Register for MSI) says:

      MSI Enable – If Set and the MSI-X Enable bit in the MSI-X
      Message Control register (see Section 7.9.2) is Clear, the
      Function is permitted to use MSI to request service and is
      prohibited from using INTx interrupts.

  - PCIe r4.0, sec 7.7.2.2 (Message Control Register for MSI-X) says:

      MSI-X Enable – If Set and the MSI Enable bit in the MSI Message
      Control register (see Section 6.8.1.3) is Clear, the Function is
      permitted to use MSI-X to request service and is prohibited from
      using INTx interrupts (if implemented).

I read that to mean a device is prohibited from using MSI/MSI-X for
some interrupts and INTx for others.  Since Tegra194 cannot use
MSI/MSI-X for PME, it should use INTx for *all* interrupts.  That
makes the MSI/MSI-X Capabilities superfluous, and they should be
omitted.

If we set pdev->no_msi for Tegra194, we'll avoid MSI/MSI-X completely,
so we'll assume *all* interrupts including hotplug will be INTx.  Will
that work?