LinuxPPC-Dev Archive on lore.kernel.org
 help / color / Atom feed
From: Kishon Vijay Abraham I <kishon@ti.com>
To: Gustavo Pimentel <Gustavo.Pimentel@synopsys.com>,
	Andrew Murray <andrew.murray@arm.com>,
	Xiaowei Bao <xiaowei.bao@nxp.com>
Cc: "mark.rutland@arm.com" <mark.rutland@arm.com>,
	Roy Zang <roy.zang@nxp.com>,
	"lorenzo.pieralisi@arm.co" <lorenzo.pieralisi@arm.co>,
	"arnd@arndb.de" <arnd@arndb.de>,
	"devicetree@vger.kernel.org" <devicetree@vger.kernel.org>,
	"gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
	"linuxppc-dev@lists.ozlabs.org" <linuxppc-dev@lists.ozlabs.org>,
	"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	Leo Li <leoyang.li@nxp.com>, "M.h. Lian" <minghuan.lian@nxp.com>,
	"robh+dt@kernel.org" <robh+dt@kernel.org>,
	"linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>,
	"jingoohan1@gmail.com" <jingoohan1@gmail.com>,
	"bhelgaas@google.com" <bhelgaas@google.com>,
	"shawnguo@kernel.org" <shawnguo@kernel.org>,
	Mingkai Hu <mingkai.hu@nxp.com>
Subject: Re: [PATCH v2 07/10] PCI: layerscape: Modify the MSIX to the doorbell way
Date: Wed, 6 Nov 2019 19:09:05 +0530
Message-ID: <e34708a1-1116-89f9-c3f8-7f21b63c9d9c@ti.com> (raw)
In-Reply-To: <DM6PR12MB40107A9B97A8DAF32A4C651EDA790@DM6PR12MB4010.namprd12.prod.outlook.com>

Gustavo,

On 06/11/19 3:10 PM, Gustavo Pimentel wrote:
> On Thu, Aug 29, 2019 at 6:13:18, Kishon Vijay Abraham I <kishon@ti.com> 
> wrote:
> 
> Hi, this email slip away from my attention...
> 
>> Gustavo,
>>
>> On 27/08/19 6:55 PM, Andrew Murray wrote:
>>> On Sat, Aug 24, 2019 at 12:08:40AM +0000, Xiaowei Bao wrote:
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Andrew Murray <andrew.murray@arm.com>
>>>>> Sent: 2019年8月23日 21:58
>>>>> To: Xiaowei Bao <xiaowei.bao@nxp.com>
>>>>> Cc: bhelgaas@google.com; robh+dt@kernel.org; mark.rutland@arm.com;
>>>>> shawnguo@kernel.org; Leo Li <leoyang.li@nxp.com>; kishon@ti.com;
>>>>> lorenzo.pieralisi@arm.co; arnd@arndb.de; gregkh@linuxfoundation.org; M.h.
>>>>> Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
>>>>> Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
>>>>> gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
>>>>> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
>>>>> linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org
>>>>> Subject: Re: [PATCH v2 07/10] PCI: layerscape: Modify the MSIX to the
>>>>> doorbell way
>>>>>
>>>>> On Thu, Aug 22, 2019 at 07:22:39PM +0800, Xiaowei Bao wrote:
>>>>>> The layerscape platform use the doorbell way to trigger MSIX interrupt
>>>>>> in EP mode.
>>>>>>
>>>>>
>>>>> I have no problems with this patch, however...
>>>>>
>>>>> Are you able to add to this message a reason for why you are making this
>>>>> change? Did dw_pcie_ep_raise_msix_irq not work when func_no != 0? Or did
>>>>> it work yet dw_pcie_ep_raise_msix_irq_doorbell is more efficient?
>>>>
>>>> The fact is that, this driver is verified in ls1046a platform of NXP before, and ls1046a don't
>>>> support MSIX feature, so I set the msix_capable of pci_epc_features struct is false,
>>>> but in other platform, e.g. ls1088a, it support the MSIX feature, I verified the MSIX
>>>> feature in ls1088a, it is not OK, so I changed to another way. Thanks.
>>>
>>> Right, so the existing pci-layerscape-ep.c driver never supported MSIX yet it
>>> erroneously had a switch case statement to call dw_pcie_ep_raise_msix_irq which
>>> would never get used.
>>>
>>> Now that we're adding a platform with MSIX support the existing
>>> dw_pcie_ep_raise_msix_irq doesn't work (for this platform) so we are adding a
>>> different method.
>>
>> Gustavo, can you confirm dw_pcie_ep_raise_msix_irq() works for designware as it
>> didn't work for both me and Xiaowei?
> 
> When I implemented the dw_pcie_ep_raise_msix_irq(), the implementation 
> was working quite fine on DesignWare solution. Otherwise, I wouldn't 
> submit it to the kernel.
> From what I have seen and if I recall well, Xiaowei implementation was 
> done having PF's configurated on his solution, which is a configuration 
> that I don't have in my solution, I believe this could be the missing 
> piece that differs between our 2 implementations.

I haven't debugged the issue yet but in my understanding the MSI-X table should
be in the memory (DDR) of EP system. This table will be populated by RC while
configuring MSI-X (with msg address and msg data). The EP will use the
populated msg address and msg data for raising MSI-X interrupt.

From the dw_pcie_ep_raise_msix_irq() (copied below), nowhere the MSI-X table is
being read from the memory of EP system. I've given my comments below.

int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
			     u16 interrupt_num)
{
	.
	.
	reg = PCI_BASE_ADDRESS_0 + (4 * bir);
	bar_addr_upper = 0;
	bar_addr_lower = dw_pcie_readl_dbi(pci, reg);

BAR register will hold the "PCI address" programmed by the host. So
"bar_addr_lower" will have PCI address.

	reg_u64 = (bar_addr_lower & PCI_BASE_ADDRESS_MEM_TYPE_MASK);
	if (reg_u64 == PCI_BASE_ADDRESS_MEM_TYPE_64)
		bar_addr_upper = dw_pcie_readl_dbi(pci, reg + 4);

	tbl_addr = ((u64) bar_addr_upper) << 32 | bar_addr_lower;

The "tbl_addr" now has the PCI address programmed by the host.

	tbl_addr += (tbl_offset + ((interrupt_num - 1) * PCI_MSIX_ENTRY_SIZE));
	tbl_addr &= PCI_BASE_ADDRESS_MEM_MASK;

	msix_tbl = ioremap_nocache(ep->phys_base + tbl_addr,
				   PCI_MSIX_ENTRY_SIZE);

"ep->phys_base" will have EPs outbound memory address and "tbl_addr" will have
PCI address. So msix_tbl points to the EPs outbound memory region.
	if (!msix_tbl)
		return -EINVAL;

	msg_addr_lower = readl(msix_tbl + PCI_MSIX_ENTRY_LOWER_ADDR);
	msg_addr_upper = readl(msix_tbl + PCI_MSIX_ENTRY_UPPER_ADDR);

Here an access to the EP outbound region is made (and the transaction will be
based on ATU configuration).
The message address should ideally be obtained from the MSI-X table present in
the EP system. There need not be any access to the OB region for getting data
from MSI-X table.

	msg_addr = ((u64) msg_addr_upper) << 32 | msg_addr_lower;
	msg_data = readl(msix_tbl + PCI_MSIX_ENTRY_DATA);
	vec_ctrl = readl(msix_tbl + PCI_MSIX_ENTRY_VECTOR_CTRL);

All this should be obtained from the memory of EP.
	.
	.
}

I'm not sure how this worked for you.

Thanks
Kishon

> 
> Since patch submission into the kernel related to msix feature on pcitest 
> tool, I didn't touch or re-tested the msix feature by lack of time (other 
> projects requires my full attention for now). However is on my roadmap to 
> came back to add some other features on DesignWare eDMA driver and I can 
> do at that time some tests to see if the 
> dw_pcie_ep_raise_msix_irq_doorbell() is compatible or not with my 
> solution. If so, I can do some patch to simplify and use the 
> dw_pcie_ep_raise_msix_irq_doorbell() if it still works as expected like 
> on dw_pcie_ep_raise_msix_irq(). Agree?
> 
> Gustavo
> 
>>
>> Thanks
>> Kishon
> 
> 

  parent reply index

Thread overview: 50+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-08-22 11:22 [PATCH v2 01/10] PCI: designware-ep: Add multiple PFs support for DWC Xiaowei Bao
2019-08-22 11:22 ` [PATCH v2 02/10] PCI: designware-ep: Add the doorbell mode of MSI-X in EP mode Xiaowei Bao
2019-08-23 13:35   ` Andrew Murray
2019-08-23 23:51     ` Xiaowei Bao
2019-08-22 11:22 ` [PATCH v2 03/10] PCI: designware-ep: Move the function of getting MSI capability forward Xiaowei Bao
2019-08-23 13:38   ` Andrew Murray
2019-08-24  0:20     ` Xiaowei Bao
2019-08-22 11:22 ` [PATCH v2 04/10] dt-bindings: pci: layerscape-pci: add compatible strings for ls1088a and ls2088a Xiaowei Bao
2019-08-27 22:26   ` Rob Herring
2019-08-29  9:19     ` Xiaowei Bao
2019-08-22 11:22 ` [PATCH v2 05/10] PCI: layerscape: Fix some format issue of the code Xiaowei Bao
2019-08-23 13:45   ` Andrew Murray
2019-08-24  0:00     ` Xiaowei Bao
2019-08-22 11:22 ` [PATCH v2 06/10] PCI: layerscape: Modify the way of getting capability with different PEX Xiaowei Bao
2019-08-22 11:43   ` Kishon Vijay Abraham I
2019-08-23  2:39     ` Xiaowei Bao
2019-08-23  3:39       ` Kishon Vijay Abraham I
2019-08-23  4:13         ` Xiaowei Bao
2019-09-02 13:36           ` Andrew Murray
2019-09-03  2:11             ` Xiaowei Bao
2019-08-22 11:22 ` [PATCH v2 07/10] PCI: layerscape: Modify the MSIX to the doorbell way Xiaowei Bao
2019-08-23 13:58   ` Andrew Murray
2019-08-24  0:08     ` Xiaowei Bao
2019-08-27 13:25       ` Andrew Murray
2019-08-28  2:49         ` Xiaowei Bao
2019-08-29  5:13         ` Kishon Vijay Abraham I
2019-11-05 12:37           ` Lorenzo Pieralisi
2019-11-06  9:33             ` Xiaowei Bao
2019-11-06  9:40           ` Gustavo Pimentel
2019-11-06 10:03             ` Xiaowei Bao
2019-11-06 13:39             ` Kishon Vijay Abraham I [this message]
2019-11-06 15:40               ` Gustavo Pimentel
2019-08-22 11:22 ` [PATCH v2 08/10] PCI: layerscape: Add EP mode support for ls1088a and ls2088a Xiaowei Bao
2019-08-23 14:27   ` Andrew Murray
2019-08-24  0:18     ` Xiaowei Bao
2019-08-24  6:45       ` christophe leroy
2019-08-25  3:07         ` Xiaowei Bao
2019-08-27 14:48           ` Andrew Murray
2019-08-28  3:25             ` Xiaowei Bao
2019-08-26  9:49     ` Xiaowei Bao
2019-08-27 13:34       ` Andrew Murray
2019-08-28  4:29         ` Xiaowei Bao
2019-08-28  9:01           ` Andrew Murray
2019-08-29  2:03             ` Xiaowei Bao
2019-08-22 11:22 ` [PATCH v2 09/10] arm64: dts: layerscape: Add PCIe EP node for ls1088a Xiaowei Bao
2019-08-22 11:22 ` [PATCH v2 10/10] misc: pci_endpoint_test: Add LS1088a in pci_device_id table Xiaowei Bao
2019-08-23 13:25 ` [PATCH v2 01/10] PCI: designware-ep: Add multiple PFs support for DWC Andrew Murray
2019-08-23 23:50   ` Xiaowei Bao
2019-08-27 13:10     ` Andrew Murray
2019-08-28  7:22       ` Xiaowei Bao

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=e34708a1-1116-89f9-c3f8-7f21b63c9d9c@ti.com \
    --to=kishon@ti.com \
    --cc=Gustavo.Pimentel@synopsys.com \
    --cc=andrew.murray@arm.com \
    --cc=arnd@arndb.de \
    --cc=bhelgaas@google.com \
    --cc=devicetree@vger.kernel.org \
    --cc=gregkh@linuxfoundation.org \
    --cc=jingoohan1@gmail.com \
    --cc=leoyang.li@nxp.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=linuxppc-dev@lists.ozlabs.org \
    --cc=lorenzo.pieralisi@arm.co \
    --cc=mark.rutland@arm.com \
    --cc=minghuan.lian@nxp.com \
    --cc=mingkai.hu@nxp.com \
    --cc=robh+dt@kernel.org \
    --cc=roy.zang@nxp.com \
    --cc=shawnguo@kernel.org \
    --cc=xiaowei.bao@nxp.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LinuxPPC-Dev Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linuxppc-dev/0 linuxppc-dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linuxppc-dev linuxppc-dev/ https://lore.kernel.org/linuxppc-dev \
		linuxppc-dev@lists.ozlabs.org linuxppc-dev@ozlabs.org
	public-inbox-index linuxppc-dev

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.ozlabs.lists.linuxppc-dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git