linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 00/22] NTB: Alter kernel API to support multi-port devices
@ 2016-11-29 17:15 Serge Semin
  2016-11-29 17:15 ` [PATCH 01/22] NTB: Move link state API being first in sources Serge Semin
                   ` (22 more replies)
  0 siblings, 23 replies; 62+ messages in thread
From: Serge Semin @ 2016-11-29 17:15 UTC (permalink / raw)
  To: jdmason, dave.jiang, Allen.Hubbe, Xiangliang.Yu
  Cc: Sergey.Semin, linux-ntb, linux-kernel, Serge Semin

There are devices, like IDT PCIe-switches, which have more than just two ports.
Particularly one device can have up to eight ports with NTB-function activated.
In order to support such devices, NTB kernel API should be altered since
currently it's optimized to work with two-ports devices only.

Here are the changes I made to conform the design we discussed a few months ago:

1) Port-index-related methods are added to KAPI
 ntb_port_number();
 ntb_peer_port_count();
 ntb_peer_port_number(pdix);
 ntb_peer_port_idx(port);

2) Link state method returns bitfield of link states for each reachable port
 u64 ntb_link_is_up();
 
3) Link enable/disable methods work with local link side of NTB
 ntb_link_enable()/ntb_link_disable();
 
4) NTB memory window related interface does the following things
 ntb_mw_count(pidx); - number of inbound memory windows, which can be allocated
for shared buffer with specified peer device.
 ntb_mw_get_align(pidx, widx); - get alignment and size restrition parameters
to properly allocate inbound memory region.
 ntb_peer_mw_count(); - get number of outbound memory windows.
 ntb_peer_mw_get_addr(widx); - get mapping address of an outbound memory window 

Inbound MW based hardware:
 ntb_mw_set_trans(pidx, widx); - set translation address of allocated inbound
memory window so a peer device could access it.
 ntb_mw_clear_trans(pidx, widx); - clear the translation address of an inbound
memory window.

Outbound MW based hardware:
 ntb_peer_mw_set_trans(pidx, widx); - set translation address of a memory
window retrieved from a peer device
 ntb_peer_mw_clear_trans(pidx, widx); - clear the translation address of an
outbound memory window

5) Scratchpad interface needs to support multi-port devices as well
 ntb_spad_count() - return number of Scratchpad per each port
 ntb_peer_spad_addr(pidx, sidx) - address of Scratchpad register of the
peer device with pidx-index
 ntb_peer_spad_read(pidx, sidx) - read specified Scratchpad register of the
peer with pidx-index
 ntb_peer_spad_write(pidx, sidx) - write data to Scratchpad register of the
peer with pidx-index

6) Introduce new messaging interface of NTB KAPI
 ntb_msg_count(); - get number of message registers
 ntb_msg_inbits(); - get bitfield of inbound message registers status
 ntb_msg_outbits(); - get bitfield of outbound message registers status
 ntb_msg_read_sts(); - read the inbound and outbound message registers status
 ntb_msg_clear_sts(); - clear status bits of message registers
 ntb_msg_set_mask(); - mask interrupts raised by status bits of message
registers.
 ntb_msg_clear_mask(); - clear interrupts mask bits of message registers
 ntb_msg_recv(midx, *pidx); - read message register with specified index,
additionally getting peer port index which data received from
 ntb_msg_send(midx, pidx); - write data to the specified message register
sending it to the passed peer device connected over a pidx port
 ntb_msg_event(); - notify driver context of a new message event
 
7) Topology reduced to be either P2P (port-to-port) or B2B (bridge-to-bridge).
Since there is port number introduced to be part of ntb_dev structure, real
port number can be used to determine Primary and Secondary sides. Intel and AMD
driver are altered to support this novation.

8) Standard test drivers: PingPong, Debugging tool and Raw Perf as well as NTB
Transport drivers don't support multi-port devices at the moment.
Since we haven't got any real multi-port hadrware driver, it's dangerous to
make any serious alterations in the debugging tools. So I have made those
drivers to work the way they wokred before, but using the new NTB API.
The situation will change when I finish porting my current IDT NTB driver to
support new API. Then I'll be able to refactor the tools and test them using
real multi-port hardware.

The changes are made on top of the NTB-fork of the kernel:
https://github.com/jonmason/ntb
of "ntb"-branch with last commit:
9c763584b7c8911106bb77af7e648bef09af9d80 Linux 4.9-rc6

Signed-off-by: Serge Semin <fancer.lancer@gmail.com>

Serge Semin (22):
  NTB: Move link state API being first in sources
  NTB: Add peer indexed ports NTB API
  NTB: Alter NTB API to support both inbound and outbound MW based
    interfaces
  NTB: Add messaging NTB API
  NTB: Alter Scratchpads NTB API to support multi-ports interface
  NTB: Slightly alter link state NTB API
  NTB: Fix a few ntb.h issues
  NTB: Add T-Platforms copyrights to NTB API
  NTB Intel: Move link-related methods being first in the driver
  NTB Intel: Add port-related NTB API callback methods
  NTB Intel: Alter MW interface to fit new NTB API
  NTB Intel: Alter Scratchpads interface to fit new NTB API
  NTB Intel: Add T-Platforms copyrights to Intel NTB driver
  NTB AMD: Move link-related methods being first in the driver
  NTB AMD: Add port-related NTB API callback methods
  NTB AMD: Alter MW interface to fit new NTB API
  NTB AMD: Alter Scratchpads interface to fit new NTB API
  NTB AMD: Add T-Platforms copyrights to AMD NTB driver
  NTB PingPong: Alter driver to work with two-ports NTB API
  NTB Tool: Alter driver to work with two-ports NTB API
  NTB Perf: Alter driver to work with two-ports NTB API
  NTB Transport: Alter driver to work with two-ports NTB API

 drivers/ntb/hw/amd/ntb_hw_amd.c     | 310 +++++++++------
 drivers/ntb/hw/amd/ntb_hw_amd.h     |  12 +
 drivers/ntb/hw/intel/ntb_hw_intel.c | 420 ++++++++++++--------
 drivers/ntb/hw/intel/ntb_hw_intel.h |  12 +
 drivers/ntb/ntb.c                   |  15 +
 drivers/ntb/ntb_transport.c         |  43 +-
 drivers/ntb/test/ntb_perf.c         |  27 +-
 drivers/ntb/test/ntb_pingpong.c     |  11 +-
 drivers/ntb/test/ntb_tool.c         |  87 +++--
 include/linux/ntb.h                 | 753 +++++++++++++++++++++++++++++-------
 10 files changed, 1221 insertions(+), 469 deletions(-)

-- 
2.6.6

^ permalink raw reply	[flat|nested] 62+ messages in thread
* RE: [PATCH v2 1/9] NTB: Make link-state API being declared first
@ 2016-12-13  4:11 Allen Hubbe
  0 siblings, 0 replies; 62+ messages in thread
From: Allen Hubbe @ 2016-12-13  4:11 UTC (permalink / raw)
  To: 'Serge Semin', jdmason, dave.jiang, Xiangliang.Yu
  Cc: Sergey.Semin, linux-ntb, linux-kernel

From: Serge Semin
> Since link operations are usually performed before memory window access
> operations, it's logically better to declared link-related API before any
> other methods. Additionally it's good practice for readability to declare
> NTB device callback methods of hadrware drivers with the same order as it's
> done within ntb.h.

s/hadrware/hardware/

Please limit this change to ntb.h.  Leave it to the hardware driver maintainers to rearrange the methods in each driver if they care to.  Code movement may conflict with other changes in development.

> 
> Signed-off-by: Serge Semin <fancer.lancer@gmail.com>
> 
> ---
>  drivers/ntb/hw/amd/ntb_hw_amd.c     | 188 ++++++++++++++++++------------------
>  drivers/ntb/hw/intel/ntb_hw_intel.c | 168 ++++++++++++++++----------------
>  include/linux/ntb.h                 | 137 +++++++++++++-------------
>  3 files changed, 247 insertions(+), 246 deletions(-)
> 
> diff --git a/drivers/ntb/hw/amd/ntb_hw_amd.c b/drivers/ntb/hw/amd/ntb_hw_amd.c
> index 6ccba0d..6704327 100644
> --- a/drivers/ntb/hw/amd/ntb_hw_amd.c
> +++ b/drivers/ntb/hw/amd/ntb_hw_amd.c
> @@ -71,6 +71,97 @@ MODULE_AUTHOR("AMD Inc.");
>  static const struct file_operations amd_ntb_debugfs_info;
>  static struct dentry *debugfs_dir;
> 
> +static int amd_link_is_up(struct amd_ntb_dev *ndev)
> +{
> +	if (!ndev->peer_sta)
> +		return NTB_LNK_STA_ACTIVE(ndev->cntl_sta);
> +
> +	/* If peer_sta is reset or D0 event, the ISR has
> +	 * started a timer to check link status of hardware.
> +	 * So here just clear status bit. And if peer_sta is
> +	 * D3 or PME_TO, D0/reset event will be happened when
> +	 * system wakeup/poweron, so do nothing here.
> +	 */
> +	if (ndev->peer_sta & AMD_PEER_RESET_EVENT)
> +		ndev->peer_sta &= ~AMD_PEER_RESET_EVENT;
> +	else if (ndev->peer_sta & AMD_PEER_D0_EVENT)
> +		ndev->peer_sta = 0;
> +
> +	return 0;
> +}
> +
> +static int amd_ntb_link_is_up(struct ntb_dev *ntb,
> +			      enum ntb_speed *speed,
> +			      enum ntb_width *width)
> +{
> +	struct amd_ntb_dev *ndev = ntb_ndev(ntb);
> +	int ret = 0;
> +
> +	if (amd_link_is_up(ndev)) {
> +		if (speed)
> +			*speed = NTB_LNK_STA_SPEED(ndev->lnk_sta);
> +		if (width)
> +			*width = NTB_LNK_STA_WIDTH(ndev->lnk_sta);
> +
> +		dev_dbg(ndev_dev(ndev), "link is up.\n");
> +
> +		ret = 1;
> +	} else {
> +		if (speed)
> +			*speed = NTB_SPEED_NONE;
> +		if (width)
> +			*width = NTB_WIDTH_NONE;
> +
> +		dev_dbg(ndev_dev(ndev), "link is down.\n");
> +	}
> +
> +	return ret;
> +}
> +
> +static int amd_ntb_link_enable(struct ntb_dev *ntb,
> +			       enum ntb_speed max_speed,
> +			       enum ntb_width max_width)
> +{
> +	struct amd_ntb_dev *ndev = ntb_ndev(ntb);
> +	void __iomem *mmio = ndev->self_mmio;
> +	u32 ntb_ctl;
> +
> +	/* Enable event interrupt */
> +	ndev->int_mask &= ~AMD_EVENT_INTMASK;
> +	writel(ndev->int_mask, mmio + AMD_INTMASK_OFFSET);
> +
> +	if (ndev->ntb.topo == NTB_TOPO_SEC)
> +		return -EINVAL;
> +	dev_dbg(ndev_dev(ndev), "Enabling Link.\n");
> +
> +	ntb_ctl = readl(mmio + AMD_CNTL_OFFSET);
> +	ntb_ctl |= (PMM_REG_CTL | SMM_REG_CTL);
> +	writel(ntb_ctl, mmio + AMD_CNTL_OFFSET);
> +
> +	return 0;
> +}
> +
> +static int amd_ntb_link_disable(struct ntb_dev *ntb)
> +{
> +	struct amd_ntb_dev *ndev = ntb_ndev(ntb);
> +	void __iomem *mmio = ndev->self_mmio;
> +	u32 ntb_ctl;
> +
> +	/* Disable event interrupt */
> +	ndev->int_mask |= AMD_EVENT_INTMASK;
> +	writel(ndev->int_mask, mmio + AMD_INTMASK_OFFSET);
> +
> +	if (ndev->ntb.topo == NTB_TOPO_SEC)
> +		return -EINVAL;
> +	dev_dbg(ndev_dev(ndev), "Enabling Link.\n");
> +
> +	ntb_ctl = readl(mmio + AMD_CNTL_OFFSET);
> +	ntb_ctl &= ~(PMM_REG_CTL | SMM_REG_CTL);
> +	writel(ntb_ctl, mmio + AMD_CNTL_OFFSET);
> +
> +	return 0;
> +}
> +
>  static int ndev_mw_to_bar(struct amd_ntb_dev *ndev, int idx)
>  {
>  	if (idx < 0 || idx > ndev->mw_count)
> @@ -194,97 +285,6 @@ static int amd_ntb_mw_set_trans(struct ntb_dev *ntb, int idx,
>  	return 0;
>  }
> 
> -static int amd_link_is_up(struct amd_ntb_dev *ndev)
> -{
> -	if (!ndev->peer_sta)
> -		return NTB_LNK_STA_ACTIVE(ndev->cntl_sta);
> -
> -	/* If peer_sta is reset or D0 event, the ISR has
> -	 * started a timer to check link status of hardware.
> -	 * So here just clear status bit. And if peer_sta is
> -	 * D3 or PME_TO, D0/reset event will be happened when
> -	 * system wakeup/poweron, so do nothing here.
> -	 */
> -	if (ndev->peer_sta & AMD_PEER_RESET_EVENT)
> -		ndev->peer_sta &= ~AMD_PEER_RESET_EVENT;
> -	else if (ndev->peer_sta & AMD_PEER_D0_EVENT)
> -		ndev->peer_sta = 0;
> -
> -	return 0;
> -}
> -
> -static int amd_ntb_link_is_up(struct ntb_dev *ntb,
> -			      enum ntb_speed *speed,
> -			      enum ntb_width *width)
> -{
> -	struct amd_ntb_dev *ndev = ntb_ndev(ntb);
> -	int ret = 0;
> -
> -	if (amd_link_is_up(ndev)) {
> -		if (speed)
> -			*speed = NTB_LNK_STA_SPEED(ndev->lnk_sta);
> -		if (width)
> -			*width = NTB_LNK_STA_WIDTH(ndev->lnk_sta);
> -
> -		dev_dbg(ndev_dev(ndev), "link is up.\n");
> -
> -		ret = 1;
> -	} else {
> -		if (speed)
> -			*speed = NTB_SPEED_NONE;
> -		if (width)
> -			*width = NTB_WIDTH_NONE;
> -
> -		dev_dbg(ndev_dev(ndev), "link is down.\n");
> -	}
> -
> -	return ret;
> -}
> -
> -static int amd_ntb_link_enable(struct ntb_dev *ntb,
> -			       enum ntb_speed max_speed,
> -			       enum ntb_width max_width)
> -{
> -	struct amd_ntb_dev *ndev = ntb_ndev(ntb);
> -	void __iomem *mmio = ndev->self_mmio;
> -	u32 ntb_ctl;
> -
> -	/* Enable event interrupt */
> -	ndev->int_mask &= ~AMD_EVENT_INTMASK;
> -	writel(ndev->int_mask, mmio + AMD_INTMASK_OFFSET);
> -
> -	if (ndev->ntb.topo == NTB_TOPO_SEC)
> -		return -EINVAL;
> -	dev_dbg(ndev_dev(ndev), "Enabling Link.\n");
> -
> -	ntb_ctl = readl(mmio + AMD_CNTL_OFFSET);
> -	ntb_ctl |= (PMM_REG_CTL | SMM_REG_CTL);
> -	writel(ntb_ctl, mmio + AMD_CNTL_OFFSET);
> -
> -	return 0;
> -}
> -
> -static int amd_ntb_link_disable(struct ntb_dev *ntb)
> -{
> -	struct amd_ntb_dev *ndev = ntb_ndev(ntb);
> -	void __iomem *mmio = ndev->self_mmio;
> -	u32 ntb_ctl;
> -
> -	/* Disable event interrupt */
> -	ndev->int_mask |= AMD_EVENT_INTMASK;
> -	writel(ndev->int_mask, mmio + AMD_INTMASK_OFFSET);
> -
> -	if (ndev->ntb.topo == NTB_TOPO_SEC)
> -		return -EINVAL;
> -	dev_dbg(ndev_dev(ndev), "Enabling Link.\n");
> -
> -	ntb_ctl = readl(mmio + AMD_CNTL_OFFSET);
> -	ntb_ctl &= ~(PMM_REG_CTL | SMM_REG_CTL);
> -	writel(ntb_ctl, mmio + AMD_CNTL_OFFSET);
> -
> -	return 0;
> -}
> -
>  static u64 amd_ntb_db_valid_mask(struct ntb_dev *ntb)
>  {
>  	return ntb_ndev(ntb)->db_valid_mask;
> @@ -431,12 +431,12 @@ static int amd_ntb_peer_spad_write(struct ntb_dev *ntb,
>  }
> 
>  static const struct ntb_dev_ops amd_ntb_ops = {
> -	.mw_count		= amd_ntb_mw_count,
> -	.mw_get_range		= amd_ntb_mw_get_range,
> -	.mw_set_trans		= amd_ntb_mw_set_trans,
>  	.link_is_up		= amd_ntb_link_is_up,
>  	.link_enable		= amd_ntb_link_enable,
>  	.link_disable		= amd_ntb_link_disable,
> +	.mw_count		= amd_ntb_mw_count,
> +	.mw_get_range		= amd_ntb_mw_get_range,
> +	.mw_set_trans		= amd_ntb_mw_set_trans,
>  	.db_valid_mask		= amd_ntb_db_valid_mask,
>  	.db_vector_count	= amd_ntb_db_vector_count,
>  	.db_vector_mask		= amd_ntb_db_vector_mask,
> diff --git a/drivers/ntb/hw/intel/ntb_hw_intel.c b/drivers/ntb/hw/intel/ntb_hw_intel.c
> index d2ce280..68d9908 100644
> --- a/drivers/ntb/hw/intel/ntb_hw_intel.c
> +++ b/drivers/ntb/hw/intel/ntb_hw_intel.c
> @@ -1035,6 +1035,84 @@ static void ndev_deinit_debugfs(struct intel_ntb_dev *ndev)
>  	debugfs_remove_recursive(ndev->debugfs_dir);
>  }
> 
> +static int intel_ntb_link_is_up(struct ntb_dev *ntb,
> +				enum ntb_speed *speed,
> +				enum ntb_width *width)
> +{
> +	struct intel_ntb_dev *ndev = ntb_ndev(ntb);
> +
> +	if (ndev->reg->link_is_up(ndev)) {
> +		if (speed)
> +			*speed = NTB_LNK_STA_SPEED(ndev->lnk_sta);
> +		if (width)
> +			*width = NTB_LNK_STA_WIDTH(ndev->lnk_sta);
> +		return 1;
> +	} else {
> +		/* TODO MAYBE: is it possible to observe the link speed and
> +		 * width while link is training? */
> +		if (speed)
> +			*speed = NTB_SPEED_NONE;
> +		if (width)
> +			*width = NTB_WIDTH_NONE;
> +		return 0;
> +	}
> +}
> +
> +static int intel_ntb_link_enable(struct ntb_dev *ntb,
> +				 enum ntb_speed max_speed,
> +				 enum ntb_width max_width)
> +{
> +	struct intel_ntb_dev *ndev;
> +	u32 ntb_ctl;
> +
> +	ndev = container_of(ntb, struct intel_ntb_dev, ntb);
> +
> +	if (ndev->ntb.topo == NTB_TOPO_SEC)
> +		return -EINVAL;
> +
> +	dev_dbg(ndev_dev(ndev),
> +		"Enabling link with max_speed %d max_width %d\n",
> +		max_speed, max_width);
> +	if (max_speed != NTB_SPEED_AUTO)
> +		dev_dbg(ndev_dev(ndev), "ignoring max_speed %d\n", max_speed);
> +	if (max_width != NTB_WIDTH_AUTO)
> +		dev_dbg(ndev_dev(ndev), "ignoring max_width %d\n", max_width);
> +
> +	ntb_ctl = ioread32(ndev->self_mmio + ndev->reg->ntb_ctl);
> +	ntb_ctl &= ~(NTB_CTL_DISABLE | NTB_CTL_CFG_LOCK);
> +	ntb_ctl |= NTB_CTL_P2S_BAR2_SNOOP | NTB_CTL_S2P_BAR2_SNOOP;
> +	ntb_ctl |= NTB_CTL_P2S_BAR4_SNOOP | NTB_CTL_S2P_BAR4_SNOOP;
> +	if (ndev->bar4_split)
> +		ntb_ctl |= NTB_CTL_P2S_BAR5_SNOOP | NTB_CTL_S2P_BAR5_SNOOP;
> +	iowrite32(ntb_ctl, ndev->self_mmio + ndev->reg->ntb_ctl);
> +
> +	return 0;
> +}
> +
> +static int intel_ntb_link_disable(struct ntb_dev *ntb)
> +{
> +	struct intel_ntb_dev *ndev;
> +	u32 ntb_cntl;
> +
> +	ndev = container_of(ntb, struct intel_ntb_dev, ntb);
> +
> +	if (ndev->ntb.topo == NTB_TOPO_SEC)
> +		return -EINVAL;
> +
> +	dev_dbg(ndev_dev(ndev), "Disabling link\n");
> +
> +	/* Bring NTB link down */
> +	ntb_cntl = ioread32(ndev->self_mmio + ndev->reg->ntb_ctl);
> +	ntb_cntl &= ~(NTB_CTL_P2S_BAR2_SNOOP | NTB_CTL_S2P_BAR2_SNOOP);
> +	ntb_cntl &= ~(NTB_CTL_P2S_BAR4_SNOOP | NTB_CTL_S2P_BAR4_SNOOP);
> +	if (ndev->bar4_split)
> +		ntb_cntl &= ~(NTB_CTL_P2S_BAR5_SNOOP | NTB_CTL_S2P_BAR5_SNOOP);
> +	ntb_cntl |= NTB_CTL_DISABLE | NTB_CTL_CFG_LOCK;
> +	iowrite32(ntb_cntl, ndev->self_mmio + ndev->reg->ntb_ctl);
> +
> +	return 0;
> +}
> +
>  static int intel_ntb_mw_count(struct ntb_dev *ntb)
>  {
>  	return ntb_ndev(ntb)->mw_count;
> @@ -1171,84 +1249,6 @@ static int intel_ntb_mw_set_trans(struct ntb_dev *ntb, int idx,
>  	return 0;
>  }
> 
> -static int intel_ntb_link_is_up(struct ntb_dev *ntb,
> -				enum ntb_speed *speed,
> -				enum ntb_width *width)
> -{
> -	struct intel_ntb_dev *ndev = ntb_ndev(ntb);
> -
> -	if (ndev->reg->link_is_up(ndev)) {
> -		if (speed)
> -			*speed = NTB_LNK_STA_SPEED(ndev->lnk_sta);
> -		if (width)
> -			*width = NTB_LNK_STA_WIDTH(ndev->lnk_sta);
> -		return 1;
> -	} else {
> -		/* TODO MAYBE: is it possible to observe the link speed and
> -		 * width while link is training? */
> -		if (speed)
> -			*speed = NTB_SPEED_NONE;
> -		if (width)
> -			*width = NTB_WIDTH_NONE;
> -		return 0;
> -	}
> -}
> -
> -static int intel_ntb_link_enable(struct ntb_dev *ntb,
> -				 enum ntb_speed max_speed,
> -				 enum ntb_width max_width)
> -{
> -	struct intel_ntb_dev *ndev;
> -	u32 ntb_ctl;
> -
> -	ndev = container_of(ntb, struct intel_ntb_dev, ntb);
> -
> -	if (ndev->ntb.topo == NTB_TOPO_SEC)
> -		return -EINVAL;
> -
> -	dev_dbg(ndev_dev(ndev),
> -		"Enabling link with max_speed %d max_width %d\n",
> -		max_speed, max_width);
> -	if (max_speed != NTB_SPEED_AUTO)
> -		dev_dbg(ndev_dev(ndev), "ignoring max_speed %d\n", max_speed);
> -	if (max_width != NTB_WIDTH_AUTO)
> -		dev_dbg(ndev_dev(ndev), "ignoring max_width %d\n", max_width);
> -
> -	ntb_ctl = ioread32(ndev->self_mmio + ndev->reg->ntb_ctl);
> -	ntb_ctl &= ~(NTB_CTL_DISABLE | NTB_CTL_CFG_LOCK);
> -	ntb_ctl |= NTB_CTL_P2S_BAR2_SNOOP | NTB_CTL_S2P_BAR2_SNOOP;
> -	ntb_ctl |= NTB_CTL_P2S_BAR4_SNOOP | NTB_CTL_S2P_BAR4_SNOOP;
> -	if (ndev->bar4_split)
> -		ntb_ctl |= NTB_CTL_P2S_BAR5_SNOOP | NTB_CTL_S2P_BAR5_SNOOP;
> -	iowrite32(ntb_ctl, ndev->self_mmio + ndev->reg->ntb_ctl);
> -
> -	return 0;
> -}
> -
> -static int intel_ntb_link_disable(struct ntb_dev *ntb)
> -{
> -	struct intel_ntb_dev *ndev;
> -	u32 ntb_cntl;
> -
> -	ndev = container_of(ntb, struct intel_ntb_dev, ntb);
> -
> -	if (ndev->ntb.topo == NTB_TOPO_SEC)
> -		return -EINVAL;
> -
> -	dev_dbg(ndev_dev(ndev), "Disabling link\n");
> -
> -	/* Bring NTB link down */
> -	ntb_cntl = ioread32(ndev->self_mmio + ndev->reg->ntb_ctl);
> -	ntb_cntl &= ~(NTB_CTL_P2S_BAR2_SNOOP | NTB_CTL_S2P_BAR2_SNOOP);
> -	ntb_cntl &= ~(NTB_CTL_P2S_BAR4_SNOOP | NTB_CTL_S2P_BAR4_SNOOP);
> -	if (ndev->bar4_split)
> -		ntb_cntl &= ~(NTB_CTL_P2S_BAR5_SNOOP | NTB_CTL_S2P_BAR5_SNOOP);
> -	ntb_cntl |= NTB_CTL_DISABLE | NTB_CTL_CFG_LOCK;
> -	iowrite32(ntb_cntl, ndev->self_mmio + ndev->reg->ntb_ctl);
> -
> -	return 0;
> -}
> -
>  static int intel_ntb_db_is_unsafe(struct ntb_dev *ntb)
>  {
>  	return ndev_ignore_unsafe(ntb_ndev(ntb), NTB_UNSAFE_DB);
> @@ -2883,12 +2883,12 @@ static const struct intel_ntb_xlat_reg skx_sec_xlat = {
> 
>  /* operations for primary side of local ntb */
>  static const struct ntb_dev_ops intel_ntb_ops = {
> -	.mw_count		= intel_ntb_mw_count,
> -	.mw_get_range		= intel_ntb_mw_get_range,
> -	.mw_set_trans		= intel_ntb_mw_set_trans,
>  	.link_is_up		= intel_ntb_link_is_up,
>  	.link_enable		= intel_ntb_link_enable,
>  	.link_disable		= intel_ntb_link_disable,
> +	.mw_count		= intel_ntb_mw_count,
> +	.mw_get_range		= intel_ntb_mw_get_range,
> +	.mw_set_trans		= intel_ntb_mw_set_trans,
>  	.db_is_unsafe		= intel_ntb_db_is_unsafe,
>  	.db_valid_mask		= intel_ntb_db_valid_mask,
>  	.db_vector_count	= intel_ntb_db_vector_count,
> @@ -2909,12 +2909,12 @@ static const struct ntb_dev_ops intel_ntb_ops = {
>  };
> 
>  static const struct ntb_dev_ops intel_ntb3_ops = {
> -	.mw_count		= intel_ntb_mw_count,
> -	.mw_get_range		= intel_ntb_mw_get_range,
> -	.mw_set_trans		= intel_ntb3_mw_set_trans,
>  	.link_is_up		= intel_ntb_link_is_up,
>  	.link_enable		= intel_ntb3_link_enable,
>  	.link_disable		= intel_ntb_link_disable,
> +	.mw_count		= intel_ntb_mw_count,
> +	.mw_get_range		= intel_ntb_mw_get_range,
> +	.mw_set_trans		= intel_ntb3_mw_set_trans,
>  	.db_valid_mask		= intel_ntb_db_valid_mask,
>  	.db_vector_count	= intel_ntb_db_vector_count,
>  	.db_vector_mask		= intel_ntb_db_vector_mask,
> diff --git a/include/linux/ntb.h b/include/linux/ntb.h
> index 6f47562..5d1f260 100644
> --- a/include/linux/ntb.h
> +++ b/include/linux/ntb.h
> @@ -179,13 +179,13 @@ static inline int ntb_ctx_ops_is_valid(const struct ntb_ctx_ops
> *ops)
> 
>  /**
>   * struct ntb_ctx_ops - ntb device operations
> + * @link_is_up:		See ntb_link_is_up().
> + * @link_enable:	See ntb_link_enable().
> + * @link_disable:	See ntb_link_disable().
>   * @mw_count:		See ntb_mw_count().
>   * @mw_get_range:	See ntb_mw_get_range().
>   * @mw_set_trans:	See ntb_mw_set_trans().
>   * @mw_clear_trans:	See ntb_mw_clear_trans().
> - * @link_is_up:		See ntb_link_is_up().
> - * @link_enable:	See ntb_link_enable().
> - * @link_disable:	See ntb_link_disable().
>   * @db_is_unsafe:	See ntb_db_is_unsafe().
>   * @db_valid_mask:	See ntb_db_valid_mask().
>   * @db_vector_count:	See ntb_db_vector_count().
> @@ -212,6 +212,12 @@ static inline int ntb_ctx_ops_is_valid(const struct ntb_ctx_ops *ops)
>   * @peer_spad_write:	See ntb_peer_spad_write().
>   */
>  struct ntb_dev_ops {
> +	int (*link_is_up)(struct ntb_dev *ntb,
> +			  enum ntb_speed *speed, enum ntb_width *width);
> +	int (*link_enable)(struct ntb_dev *ntb,
> +			   enum ntb_speed max_speed, enum ntb_width max_width);
> +	int (*link_disable)(struct ntb_dev *ntb);
> +
>  	int (*mw_count)(struct ntb_dev *ntb);
>  	int (*mw_get_range)(struct ntb_dev *ntb, int idx,
>  			    phys_addr_t *base, resource_size_t *size,
> @@ -220,12 +226,6 @@ struct ntb_dev_ops {
>  			    dma_addr_t addr, resource_size_t size);
>  	int (*mw_clear_trans)(struct ntb_dev *ntb, int idx);
> 
> -	int (*link_is_up)(struct ntb_dev *ntb,
> -			  enum ntb_speed *speed, enum ntb_width *width);
> -	int (*link_enable)(struct ntb_dev *ntb,
> -			   enum ntb_speed max_speed, enum ntb_width max_width);
> -	int (*link_disable)(struct ntb_dev *ntb);
> -
>  	int (*db_is_unsafe)(struct ntb_dev *ntb);
>  	u64 (*db_valid_mask)(struct ntb_dev *ntb);
>  	int (*db_vector_count)(struct ntb_dev *ntb);
> @@ -265,13 +265,14 @@ static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops
> *ops)
>  {
>  	/* commented callbacks are not required: */
>  	return
> +		ops->link_is_up				&&
> +		ops->link_enable			&&
> +		ops->link_disable			&&
>  		ops->mw_count				&&
>  		ops->mw_get_range			&&
>  		ops->mw_set_trans			&&
>  		/* ops->mw_clear_trans			&& */
> -		ops->link_is_up				&&
> -		ops->link_enable			&&
> -		ops->link_disable			&&
> +
>  		/* ops->db_is_unsafe			&& */
>  		ops->db_valid_mask			&&
> 
> @@ -441,6 +442,62 @@ void ntb_link_event(struct ntb_dev *ntb);
>  void ntb_db_event(struct ntb_dev *ntb, int vector);
> 
>  /**
> + * ntb_link_is_up() - get the current ntb link state
> + * @ntb:	NTB device context.
> + * @speed:	OUT - The link speed expressed as PCIe generation number.
> + * @width:	OUT - The link width expressed as the number of PCIe lanes.
> + *
> + * Get the current state of the ntb link.  It is recommended to query the link
> + * state once after every link event.  It is safe to query the link state in
> + * the context of the link event callback.
> + *
> + * Return: One if the link is up, zero if the link is down, otherwise a
> + *		negative value indicating the error number.
> + */
> +static inline int ntb_link_is_up(struct ntb_dev *ntb,
> +				 enum ntb_speed *speed, enum ntb_width *width)
> +{
> +	return ntb->ops->link_is_up(ntb, speed, width);
> +}
> +
> +/**
> + * ntb_link_enable() - enable the link on the secondary side of the ntb
> + * @ntb:	NTB device context.
> + * @max_speed:	The maximum link speed expressed as PCIe generation number.
> + * @max_width:	The maximum link width expressed as the number of PCIe lanes.
> + *
> + * Enable the link on the secondary side of the ntb.  This can only be done
> + * from the primary side of the ntb in primary or b2b topology.  The ntb device
> + * should train the link to its maximum speed and width, or the requested speed
> + * and width, whichever is smaller, if supported.
> + *
> + * Return: Zero on success, otherwise an error number.
> + */
> +static inline int ntb_link_enable(struct ntb_dev *ntb,
> +				  enum ntb_speed max_speed,
> +				  enum ntb_width max_width)
> +{
> +	return ntb->ops->link_enable(ntb, max_speed, max_width);
> +}
> +
> +/**
> + * ntb_link_disable() - disable the link on the secondary side of the ntb
> + * @ntb:	NTB device context.
> + *
> + * Disable the link on the secondary side of the ntb.  This can only be
> + * done from the primary side of the ntb in primary or b2b topology.  The ntb
> + * device should disable the link.  Returning from this call must indicate that
> + * a barrier has passed, though with no more writes may pass in either
> + * direction across the link, except if this call returns an error number.
> + *
> + * Return: Zero on success, otherwise an error number.
> + */
> +static inline int ntb_link_disable(struct ntb_dev *ntb)
> +{
> +	return ntb->ops->link_disable(ntb);
> +}
> +
> +/**
>   * ntb_mw_count() - get the number of memory windows
>   * @ntb:	NTB device context.
>   *
> @@ -517,62 +574,6 @@ static inline int ntb_mw_clear_trans(struct ntb_dev *ntb, int idx)
>  }
> 
>  /**
> - * ntb_link_is_up() - get the current ntb link state
> - * @ntb:	NTB device context.
> - * @speed:	OUT - The link speed expressed as PCIe generation number.
> - * @width:	OUT - The link width expressed as the number of PCIe lanes.
> - *
> - * Get the current state of the ntb link.  It is recommended to query the link
> - * state once after every link event.  It is safe to query the link state in
> - * the context of the link event callback.
> - *
> - * Return: One if the link is up, zero if the link is down, otherwise a
> - *		negative value indicating the error number.
> - */
> -static inline int ntb_link_is_up(struct ntb_dev *ntb,
> -				 enum ntb_speed *speed, enum ntb_width *width)
> -{
> -	return ntb->ops->link_is_up(ntb, speed, width);
> -}
> -
> -/**
> - * ntb_link_enable() - enable the link on the secondary side of the ntb
> - * @ntb:	NTB device context.
> - * @max_speed:	The maximum link speed expressed as PCIe generation number.
> - * @max_width:	The maximum link width expressed as the number of PCIe lanes.
> - *
> - * Enable the link on the secondary side of the ntb.  This can only be done
> - * from the primary side of the ntb in primary or b2b topology.  The ntb device
> - * should train the link to its maximum speed and width, or the requested speed
> - * and width, whichever is smaller, if supported.
> - *
> - * Return: Zero on success, otherwise an error number.
> - */
> -static inline int ntb_link_enable(struct ntb_dev *ntb,
> -				  enum ntb_speed max_speed,
> -				  enum ntb_width max_width)
> -{
> -	return ntb->ops->link_enable(ntb, max_speed, max_width);
> -}
> -
> -/**
> - * ntb_link_disable() - disable the link on the secondary side of the ntb
> - * @ntb:	NTB device context.
> - *
> - * Disable the link on the secondary side of the ntb.  This can only be
> - * done from the primary side of the ntb in primary or b2b topology.  The ntb
> - * device should disable the link.  Returning from this call must indicate that
> - * a barrier has passed, though with no more writes may pass in either
> - * direction across the link, except if this call returns an error number.
> - *
> - * Return: Zero on success, otherwise an error number.
> - */
> -static inline int ntb_link_disable(struct ntb_dev *ntb)
> -{
> -	return ntb->ops->link_disable(ntb);
> -}
> -
> -/**
>   * ntb_db_is_unsafe() - check if it is safe to use hardware doorbell
>   * @ntb:	NTB device context.
>   *
> --
> 2.6.6

^ permalink raw reply	[flat|nested] 62+ messages in thread
end of thread, other threads:[~2017-02-01 20:01 UTC | newest]

Thread overview: 62+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-11-29 17:15 [PATCH 00/22] NTB: Alter kernel API to support multi-port devices Serge Semin
2016-11-29 17:15 ` [PATCH 01/22] NTB: Move link state API being first in sources Serge Semin
2016-11-29 17:15 ` [PATCH 02/22] NTB: Add peer indexed ports NTB API Serge Semin
2016-11-30 18:40   ` kbuild test robot
2016-11-30 19:12   ` kbuild test robot
2016-11-30 20:04   ` kbuild test robot
2016-11-29 17:15 ` [PATCH 03/22] NTB: Alter NTB API to support both inbound and outbound MW based interfaces Serge Semin
2016-11-30 18:54   ` kbuild test robot
2016-11-30 19:46   ` kbuild test robot
2016-11-29 17:15 ` [PATCH 04/22] NTB: Add messaging NTB API Serge Semin
2016-11-29 17:16 ` [PATCH 05/22] NTB: Alter Scratchpads NTB API to support multi-ports interface Serge Semin
2016-11-29 17:16 ` [PATCH 06/22] NTB: Slightly alter link state NTB API Serge Semin
2016-11-29 17:16 ` [PATCH 07/22] NTB: Fix a few ntb.h issues Serge Semin
2016-11-29 17:16 ` [PATCH 08/22] NTB: Add T-Platforms copyrights to NTB API Serge Semin
2016-11-29 17:16 ` [PATCH 09/22] NTB Intel: Move link-related methods being first in the driver Serge Semin
2016-11-29 17:16 ` [PATCH 10/22] NTB Intel: Add port-related NTB API callback methods Serge Semin
2016-12-07 22:56   ` Allen Hubbe
2016-11-29 17:16 ` [PATCH 11/22] NTB Intel: Alter MW interface to fit new NTB API Serge Semin
2016-11-29 17:16 ` [PATCH 12/22] NTB Intel: Alter Scratchpads " Serge Semin
2016-11-29 17:16 ` [PATCH 13/22] NTB Intel: Add T-Platforms copyrights to Intel NTB driver Serge Semin
2016-11-29 17:16 ` [PATCH 14/22] NTB AMD: Move link-related methods being first in the driver Serge Semin
2016-11-29 17:16 ` [PATCH 15/22] NTB AMD: Add port-related NTB API callback methods Serge Semin
2016-11-29 17:16 ` [PATCH 16/22] NTB AMD: Alter MW interface to fit new NTB API Serge Semin
2016-11-29 17:16 ` [PATCH 17/22] NTB AMD: Alter Scratchpads " Serge Semin
2016-11-29 17:16 ` [PATCH 18/22] NTB AMD: Add T-Platforms copyrights to AMD NTB driver Serge Semin
2016-11-29 17:16 ` [PATCH 19/22] NTB PingPong: Alter driver to work with two-ports NTB API Serge Semin
2016-11-29 17:16 ` [PATCH 20/22] NTB Tool: " Serge Semin
2016-11-29 17:16 ` [PATCH 21/22] NTB Perf: " Serge Semin
2016-11-29 17:16 ` [PATCH 22/22] NTB Transport: " Serge Semin
2016-12-12 21:08 ` [PATCH v2 0/9] NTB: Alter kernel API to support multi-port devices Serge Semin
2016-12-12 21:08   ` [PATCH v2 1/9] NTB: Make link-state API being declared first Serge Semin
2016-12-12 21:08   ` [PATCH v2 2/9] NTB: Add indexed ports NTB API Serge Semin
2016-12-12 21:08   ` [PATCH v2 3/9] NTB: Alter link-state API to support multi-port devices Serge Semin
2016-12-12 21:08   ` [PATCH v2 4/9] NTB: Alter MW API to support multi-ports devices Serge Semin
2016-12-12 21:08   ` [PATCH v2 5/9] NTB: Alter Scratchpads " Serge Semin
2016-12-12 21:08   ` [PATCH v2 6/9] NTB: Add Messaging NTB API Serge Semin
2016-12-12 21:08   ` [PATCH v2 7/9] NTB: Add new Memory Windows API documentation Serge Semin
2016-12-12 21:08   ` [PATCH v2 8/9] NTB: Add PCIe Gen4 link speed Serge Semin
2016-12-12 21:08   ` [PATCH v2 9/9] NTB: Add ntb.h comments Serge Semin
2016-12-13 23:49   ` [PATCH v3 0/9] NTB: Alter kernel API to support multi-port devices Serge Semin
2016-12-13 23:49     ` [PATCH v3 1/9] NTB: Make link-state API being declared first Serge Semin
2016-12-14  7:07       ` Allen Hubbe
2016-12-13 23:49     ` [PATCH v3 2/9] NTB: Add indexed ports NTB API Serge Semin
2016-12-14  7:07       ` Allen Hubbe
2016-12-13 23:49     ` [PATCH v3 3/9] NTB: Alter link-state API to support multi-port devices Serge Semin
2016-12-13 23:49     ` [PATCH v3 4/9] NTB: Alter MW API to support multi-ports devices Serge Semin
2016-12-14  7:08       ` Allen Hubbe
2017-01-11  0:11       ` Serge Semin
2016-12-13 23:49     ` [PATCH v3 5/9] NTB: Alter Scratchpads " Serge Semin
2016-12-14  7:08       ` Allen Hubbe
2017-01-11  0:13       ` Serge Semin
2017-02-01 20:01         ` Jon Mason
2016-12-13 23:49     ` [PATCH v3 6/9] NTB: Add Messaging NTB API Serge Semin
2016-12-20  9:48       ` Serge Semin
2016-12-13 23:49     ` [PATCH v3 7/9] NTB: Add new Memory Windows API documentation Serge Semin
2016-12-13 23:49     ` [PATCH v3 8/9] NTB: Add PCIe Gen4 link speed Serge Semin
2016-12-13 23:49     ` [PATCH v3 9/9] NTB: Add ntb.h comments Serge Semin
2016-12-14  7:09       ` Allen Hubbe
2016-12-14  7:16       ` Serge Semin
2016-12-14  7:19         ` Allen Hubbe
2016-12-20  9:50         ` Serge Semin
2016-12-13  4:11 [PATCH v2 1/9] NTB: Make link-state API being declared first Allen Hubbe

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).