LinuxPPC-Dev Archive on lore.kernel.org
 help / color / Atom feed
From: Gustavo Pimentel <Gustavo.Pimentel@synopsys.com>
To: Kishon Vijay Abraham I <kishon@ti.com>,
	Gustavo Pimentel <Gustavo.Pimentel@synopsys.com>,
	Andrew Murray <andrew.murray@arm.com>,
	Xiaowei Bao <xiaowei.bao@nxp.com>
Cc: "mark.rutland@arm.com" <mark.rutland@arm.com>,
	Roy Zang <roy.zang@nxp.com>,
	"lorenzo.pieralisi@arm.co" <lorenzo.pieralisi@arm.co>,
	"arnd@arndb.de" <arnd@arndb.de>,
	"devicetree@vger.kernel.org" <devicetree@vger.kernel.org>,
	"gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
	"linuxppc-dev@lists.ozlabs.org" <linuxppc-dev@lists.ozlabs.org>,
	"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	Leo Li <leoyang.li@nxp.com>, "M.h. Lian" <minghuan.lian@nxp.com>,
	"robh+dt@kernel.org" <robh+dt@kernel.org>,
	"linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>,
	"jingoohan1@gmail.com" <jingoohan1@gmail.com>,
	"bhelgaas@google.com" <bhelgaas@google.com>,
	"shawnguo@kernel.org" <shawnguo@kernel.org>,
	Mingkai Hu <mingkai.hu@nxp.com>
Subject: RE: [PATCH v2 07/10] PCI: layerscape: Modify the MSIX to the doorbell way
Date: Wed, 6 Nov 2019 15:40:21 +0000
Message-ID: <DM6PR12MB401035203D8962DF4265EAB7DA790@DM6PR12MB4010.namprd12.prod.outlook.com> (raw)
In-Reply-To: <e34708a1-1116-89f9-c3f8-7f21b63c9d9c@ti.com>

On Wed, Nov 6, 2019 at 13:39:5, Kishon Vijay Abraham I <kishon@ti.com> 
wrote:

Hi Kishon,

> Gustavo,
> 
> On 06/11/19 3:10 PM, Gustavo Pimentel wrote:
> > On Thu, Aug 29, 2019 at 6:13:18, Kishon Vijay Abraham I <kishon@ti.com> 
> > wrote:
> > 
> > Hi, this email slip away from my attention...
> > 
> >> Gustavo,
> >>
> >> On 27/08/19 6:55 PM, Andrew Murray wrote:
> >>> On Sat, Aug 24, 2019 at 12:08:40AM +0000, Xiaowei Bao wrote:
> >>>>
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Andrew Murray <andrew.murray@arm.com>
> >>>>> Sent: 2019年8月23日 21:58
> >>>>> To: Xiaowei Bao <xiaowei.bao@nxp.com>
> >>>>> Cc: bhelgaas@google.com; robh+dt@kernel.org; mark.rutland@arm.com;
> >>>>> shawnguo@kernel.org; Leo Li <leoyang.li@nxp.com>; kishon@ti.com;
> >>>>> lorenzo.pieralisi@arm.co; arnd@arndb.de; gregkh@linuxfoundation.org; M.h.
> >>>>> Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> >>>>> Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> >>>>> gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> >>>>> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> >>>>> linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org
> >>>>> Subject: Re: [PATCH v2 07/10] PCI: layerscape: Modify the MSIX to the
> >>>>> doorbell way
> >>>>>
> >>>>> On Thu, Aug 22, 2019 at 07:22:39PM +0800, Xiaowei Bao wrote:
> >>>>>> The layerscape platform use the doorbell way to trigger MSIX interrupt
> >>>>>> in EP mode.
> >>>>>>
> >>>>>
> >>>>> I have no problems with this patch, however...
> >>>>>
> >>>>> Are you able to add to this message a reason for why you are making this
> >>>>> change? Did dw_pcie_ep_raise_msix_irq not work when func_no != 0? Or did
> >>>>> it work yet dw_pcie_ep_raise_msix_irq_doorbell is more efficient?
> >>>>
> >>>> The fact is that, this driver is verified in ls1046a platform of NXP before, and ls1046a don't
> >>>> support MSIX feature, so I set the msix_capable of pci_epc_features struct is false,
> >>>> but in other platform, e.g. ls1088a, it support the MSIX feature, I verified the MSIX
> >>>> feature in ls1088a, it is not OK, so I changed to another way. Thanks.
> >>>
> >>> Right, so the existing pci-layerscape-ep.c driver never supported MSIX yet it
> >>> erroneously had a switch case statement to call dw_pcie_ep_raise_msix_irq which
> >>> would never get used.
> >>>
> >>> Now that we're adding a platform with MSIX support the existing
> >>> dw_pcie_ep_raise_msix_irq doesn't work (for this platform) so we are adding a
> >>> different method.
> >>
> >> Gustavo, can you confirm dw_pcie_ep_raise_msix_irq() works for designware as it
> >> didn't work for both me and Xiaowei?
> > 
> > When I implemented the dw_pcie_ep_raise_msix_irq(), the implementation 
> > was working quite fine on DesignWare solution. Otherwise, I wouldn't 
> > submit it to the kernel.
> > From what I have seen and if I recall well, Xiaowei implementation was 
> > done having PF's configurated on his solution, which is a configuration 
> > that I don't have in my solution, I believe this could be the missing 
> > piece that differs between our 2 implementations.
> 
> I haven't debugged the issue yet but in my understanding the MSI-X table should
> be in the memory (DDR) of EP system. This table will be populated by RC while
> configuring MSI-X (with msg address and msg data). The EP will use the
> populated msg address and msg data for raising MSI-X interrupt.

Right.

> 
> From the dw_pcie_ep_raise_msix_irq() (copied below), nowhere the MSI-X table is
> being read from the memory of EP system. I've given my comments below.
> 
> int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
> 			     u16 interrupt_num)
> {
> 	.
> 	.
> 	reg = PCI_BASE_ADDRESS_0 + (4 * bir);
> 	bar_addr_upper = 0;
> 	bar_addr_lower = dw_pcie_readl_dbi(pci, reg);
> 
> BAR register will hold the "PCI address" programmed by the host. So
> "bar_addr_lower" will have PCI address.
> 
> 	reg_u64 = (bar_addr_lower & PCI_BASE_ADDRESS_MEM_TYPE_MASK);
> 	if (reg_u64 == PCI_BASE_ADDRESS_MEM_TYPE_64)
> 		bar_addr_upper = dw_pcie_readl_dbi(pci, reg + 4);
> 
> 	tbl_addr = ((u64) bar_addr_upper) << 32 | bar_addr_lower;
> 
> The "tbl_addr" now has the PCI address programmed by the host.
> 
> 	tbl_addr += (tbl_offset + ((interrupt_num - 1) * PCI_MSIX_ENTRY_SIZE));
> 	tbl_addr &= PCI_BASE_ADDRESS_MEM_MASK;
> 
> 	msix_tbl = ioremap_nocache(ep->phys_base + tbl_addr,
> 				   PCI_MSIX_ENTRY_SIZE);
> 
> "ep->phys_base" will have EPs outbound memory address and "tbl_addr" will have
> PCI address. So msix_tbl points to the EPs outbound memory region.
> 	if (!msix_tbl)
> 		return -EINVAL;
> 
> 	msg_addr_lower = readl(msix_tbl + PCI_MSIX_ENTRY_LOWER_ADDR);
> 	msg_addr_upper = readl(msix_tbl + PCI_MSIX_ENTRY_UPPER_ADDR);
> 
> Here an access to the EP outbound region is made (and the transaction will be
> based on ATU configuration).
> The message address should ideally be obtained from the MSI-X table present in
> the EP system. There need not be any access to the OB region for getting data
> from MSI-X table.
> 
> 	msg_addr = ((u64) msg_addr_upper) << 32 | msg_addr_lower;
> 	msg_data = readl(msix_tbl + PCI_MSIX_ENTRY_DATA);
> 	vec_ctrl = readl(msix_tbl + PCI_MSIX_ENTRY_VECTOR_CTRL);
> 
> All this should be obtained from the memory of EP.
> 	.
> 	.
> }
> 
> I'm not sure how this worked for you.

Hum, it was a very time ago that I implemented this. I've to debug it to 
give you an accurate answer.
However, it very likely that my DesignWare prototype solution might have 
some peculiarities that could allow this type of access. At the time, 
nobody had the msix feature enabled that was working with pcitest EP in 
their SOCs that could be used to test my implementation against, at least 
that I know of.

As soon as I get back to work on PCI, I will retest this with the other 
function, if it works I'll make a patch to use that function instead, I'm 
assuming that function is working for other SOCs, right? 

Gustavo

> 
> Thanks
> Kishon
> 
> > 
> > Since patch submission into the kernel related to msix feature on pcitest 
> > tool, I didn't touch or re-tested the msix feature by lack of time (other 
> > projects requires my full attention for now). However is on my roadmap to 
> > came back to add some other features on DesignWare eDMA driver and I can 
> > do at that time some tests to see if the 
> > dw_pcie_ep_raise_msix_irq_doorbell() is compatible or not with my 
> > solution. If so, I can do some patch to simplify and use the 
> > dw_pcie_ep_raise_msix_irq_doorbell() if it still works as expected like 
> > on dw_pcie_ep_raise_msix_irq(). Agree?
> > 
> > Gustavo
> > 
> >>
> >> Thanks
> >> Kishon
> > 
> > 



  reply index

Thread overview: 50+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-08-22 11:22 [PATCH v2 01/10] PCI: designware-ep: Add multiple PFs support for DWC Xiaowei Bao
2019-08-22 11:22 ` [PATCH v2 02/10] PCI: designware-ep: Add the doorbell mode of MSI-X in EP mode Xiaowei Bao
2019-08-23 13:35   ` Andrew Murray
2019-08-23 23:51     ` Xiaowei Bao
2019-08-22 11:22 ` [PATCH v2 03/10] PCI: designware-ep: Move the function of getting MSI capability forward Xiaowei Bao
2019-08-23 13:38   ` Andrew Murray
2019-08-24  0:20     ` Xiaowei Bao
2019-08-22 11:22 ` [PATCH v2 04/10] dt-bindings: pci: layerscape-pci: add compatible strings for ls1088a and ls2088a Xiaowei Bao
2019-08-27 22:26   ` Rob Herring
2019-08-29  9:19     ` Xiaowei Bao
2019-08-22 11:22 ` [PATCH v2 05/10] PCI: layerscape: Fix some format issue of the code Xiaowei Bao
2019-08-23 13:45   ` Andrew Murray
2019-08-24  0:00     ` Xiaowei Bao
2019-08-22 11:22 ` [PATCH v2 06/10] PCI: layerscape: Modify the way of getting capability with different PEX Xiaowei Bao
2019-08-22 11:43   ` Kishon Vijay Abraham I
2019-08-23  2:39     ` Xiaowei Bao
2019-08-23  3:39       ` Kishon Vijay Abraham I
2019-08-23  4:13         ` Xiaowei Bao
2019-09-02 13:36           ` Andrew Murray
2019-09-03  2:11             ` Xiaowei Bao
2019-08-22 11:22 ` [PATCH v2 07/10] PCI: layerscape: Modify the MSIX to the doorbell way Xiaowei Bao
2019-08-23 13:58   ` Andrew Murray
2019-08-24  0:08     ` Xiaowei Bao
2019-08-27 13:25       ` Andrew Murray
2019-08-28  2:49         ` Xiaowei Bao
2019-08-29  5:13         ` Kishon Vijay Abraham I
2019-11-05 12:37           ` Lorenzo Pieralisi
2019-11-06  9:33             ` Xiaowei Bao
2019-11-06  9:40           ` Gustavo Pimentel
2019-11-06 10:03             ` Xiaowei Bao
2019-11-06 13:39             ` Kishon Vijay Abraham I
2019-11-06 15:40               ` Gustavo Pimentel [this message]
2019-08-22 11:22 ` [PATCH v2 08/10] PCI: layerscape: Add EP mode support for ls1088a and ls2088a Xiaowei Bao
2019-08-23 14:27   ` Andrew Murray
2019-08-24  0:18     ` Xiaowei Bao
2019-08-24  6:45       ` christophe leroy
2019-08-25  3:07         ` Xiaowei Bao
2019-08-27 14:48           ` Andrew Murray
2019-08-28  3:25             ` Xiaowei Bao
2019-08-26  9:49     ` Xiaowei Bao
2019-08-27 13:34       ` Andrew Murray
2019-08-28  4:29         ` Xiaowei Bao
2019-08-28  9:01           ` Andrew Murray
2019-08-29  2:03             ` Xiaowei Bao
2019-08-22 11:22 ` [PATCH v2 09/10] arm64: dts: layerscape: Add PCIe EP node for ls1088a Xiaowei Bao
2019-08-22 11:22 ` [PATCH v2 10/10] misc: pci_endpoint_test: Add LS1088a in pci_device_id table Xiaowei Bao
2019-08-23 13:25 ` [PATCH v2 01/10] PCI: designware-ep: Add multiple PFs support for DWC Andrew Murray
2019-08-23 23:50   ` Xiaowei Bao
2019-08-27 13:10     ` Andrew Murray
2019-08-28  7:22       ` Xiaowei Bao

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DM6PR12MB401035203D8962DF4265EAB7DA790@DM6PR12MB4010.namprd12.prod.outlook.com \
    --to=gustavo.pimentel@synopsys.com \
    --cc=andrew.murray@arm.com \
    --cc=arnd@arndb.de \
    --cc=bhelgaas@google.com \
    --cc=devicetree@vger.kernel.org \
    --cc=gregkh@linuxfoundation.org \
    --cc=jingoohan1@gmail.com \
    --cc=kishon@ti.com \
    --cc=leoyang.li@nxp.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=linuxppc-dev@lists.ozlabs.org \
    --cc=lorenzo.pieralisi@arm.co \
    --cc=mark.rutland@arm.com \
    --cc=minghuan.lian@nxp.com \
    --cc=mingkai.hu@nxp.com \
    --cc=robh+dt@kernel.org \
    --cc=roy.zang@nxp.com \
    --cc=shawnguo@kernel.org \
    --cc=xiaowei.bao@nxp.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LinuxPPC-Dev Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linuxppc-dev/0 linuxppc-dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linuxppc-dev linuxppc-dev/ https://lore.kernel.org/linuxppc-dev \
		linuxppc-dev@lists.ozlabs.org linuxppc-dev@ozlabs.org
	public-inbox-index linuxppc-dev

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.ozlabs.lists.linuxppc-dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git