[PATCH 0/7] Switchtec NTB Crosslink Support

[PATCH 0/7] Switchtec NTB Crosslink Support

[Logan Gunthorpe]

Hi,

This patch series adds support for the Switchtec Crosslink feature.
Crosslink is similar to B2B in that it allows two switches to be
connected back to back.

Two switches can already be connected between two hosts, however with this
setup, there would be no symmetry as one switch would handle NTB and the
other would just need to be a single partition switch. Our customers are
looking for a completely symmetric solution such that a host or switch
can be easily replaced without needing to worry about which half was
replaced.

Crosslink solves this problem by having each of the switches configured
with two partitions. This results in a special host-less partition in
the middle of the connection which can be managed in such a way as
to forward all the traffic and still provide all the same NTB
functionality. In this way, both hosts and switches can be identical.

Please see Patch 6 for a bit more detailed description of how Crosslink
works.

Thanks,

Logan

Kelvin Cao (1):
  ntb_hw_switchtec: Allow using Switchtec NTB in multi-partition setups

Logan Gunthorpe (6):
  ntb_hw_switchtec: Keep track of the number of LUT windows used by the
    driver
  ntb_hw_switchtec: Create helper function to setup reserved LUT MWs
  ntb_hw_switchtec: Make switchtec_ntb_init_req_id_table() more general
  ntb_hw_switchtec: Expand PFF CSR registers
  ntb_hw_switchtec: Add initialization code for crosslink
  ntb_hw_switchtec: Crosslink doorbells and messages

 drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 494 +++++++++++++++++++++++++++------
 include/linux/ntb.h                    |  15 +-
 include/linux/switchtec.h              |  23 +-
 3 files changed, 435 insertions(+), 97 deletions(-)

--
2.11.0

[PATCH 1/7] ntb_hw_switchtec: Allow using Switchtec NTB in multi-partition setups

[Logan Gunthorpe]

From: Kelvin Cao <kelvin.cao@microsemi.com>

Allow using Switchtec NTB in setups that have more than two partitions.
Note: this does not enable having multi-host communication, it only
allows for a single NTB link between two hosts in a network that might
have more than two.

Use following logic to determine the NT peer partition:

1) If there are 2 partitions, and the target vector is set in
   the Switchtec configuration, use the partition specified in target
   vector.
2) If there are 2 partitions and target vector is unset
   use the only other partition as specified in the NT EP map.
3) If there are more than 2 partitions and target vector is set
   use the other partition specified in target vector.
4) If there are more than 2 partitions and target vector is unset,
   this is invalid and report an error.

Signed-off-by: Kelvin Cao <kelvin.cao@microsemi.com>
[logang@deltatee.com: commit message fleshed out]
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
---
 drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 57 ++++++++++++++++++++++++++++------
 include/linux/switchtec.h              |  8 +++++
 2 files changed, 56 insertions(+), 9 deletions(-)

diff --git a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
index 709f37fbe232..088ae220ecb4 100644
--- a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
+++ b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
@@ -777,9 +777,12 @@ static const struct ntb_dev_ops switchtec_ntb_ops = {
 	.peer_spad_addr		= switchtec_ntb_peer_spad_addr,
 };
 
-static void switchtec_ntb_init_sndev(struct switchtec_ntb *sndev)
+static int switchtec_ntb_init_sndev(struct switchtec_ntb *sndev)
 {
+	u64 tpart_vec;
+	int self;
 	u64 part_map;
+	int bit;
 
 	sndev->ntb.pdev = sndev->stdev->pdev;
 	sndev->ntb.topo = NTB_TOPO_SWITCH;
@@ -788,13 +791,47 @@ static void switchtec_ntb_init_sndev(struct switchtec_ntb *sndev)
 	sndev->self_partition = sndev->stdev->partition;
 
 	sndev->mmio_ntb = sndev->stdev->mmio_ntb;
+
+	self = sndev->self_partition;
+	tpart_vec = ioread32(&sndev->mmio_ntb->ntp_info[self].target_part_high);
+	tpart_vec <<= 32;
+	tpart_vec |= ioread32(&sndev->mmio_ntb->ntp_info[self].target_part_low);
+
 	part_map = ioread64(&sndev->mmio_ntb->ep_map);
 	part_map &= ~(1 << sndev->self_partition);
-	sndev->peer_partition = ffs(part_map) - 1;
 
-	dev_dbg(&sndev->stdev->dev, "Partition ID %d of %d (%llx)\n",
-		sndev->self_partition, sndev->stdev->partition_count,
-		part_map);
+	if (!ffs(tpart_vec)) {
+		if (sndev->stdev->partition_count != 2) {
+			dev_err(&sndev->stdev->dev,
+				"ntb target partition not defined\n");
+			return -ENODEV;
+		}
+
+		bit = ffs(part_map);
+		if (!bit) {
+			dev_err(&sndev->stdev->dev,
+				"peer partition is not NT partition\n");
+			return -ENODEV;
+		}
+
+		sndev->peer_partition = bit - 1;
+	} else {
+		if (ffs(tpart_vec) != fls(tpart_vec)) {
+			dev_err(&sndev->stdev->dev,
+				"ntb driver only supports 1 pair of 1-1 ntb mapping\n");
+			return -ENODEV;
+		}
+
+		sndev->peer_partition = ffs(tpart_vec) - 1;
+		if (!(part_map && (1 << sndev->peer_partition))) {
+			dev_err(&sndev->stdev->dev,
+				"ntb target partition is not NT partition\n");
+			return -ENODEV;
+		}
+	}
+
+	dev_dbg(&sndev->stdev->dev, "Partition ID %d of %d\n",
+		sndev->self_partition, sndev->stdev->partition_count);
 
 	sndev->mmio_ctrl = (void * __iomem)sndev->mmio_ntb +
 		SWITCHTEC_NTB_REG_CTRL_OFFSET;
@@ -804,6 +841,8 @@ static void switchtec_ntb_init_sndev(struct switchtec_ntb *sndev)
 	sndev->mmio_self_ctrl = &sndev->mmio_ctrl[sndev->self_partition];
 	sndev->mmio_peer_ctrl = &sndev->mmio_ctrl[sndev->peer_partition];
 	sndev->mmio_self_dbmsg = &sndev->mmio_dbmsg[sndev->self_partition];
+
+	return 0;
 }
 
 static int map_bars(int *map, struct ntb_ctrl_regs __iomem *ctrl)
@@ -1135,15 +1174,15 @@ static int switchtec_ntb_add(struct device *dev,
 	if (stdev->pdev->class != MICROSEMI_NTB_CLASSCODE)
 		return -ENODEV;
 
-	if (stdev->partition_count != 2)
-		dev_warn(dev, "ntb driver only supports 2 partitions\n");
-
 	sndev = kzalloc_node(sizeof(*sndev), GFP_KERNEL, dev_to_node(dev));
 	if (!sndev)
 		return -ENOMEM;
 
 	sndev->stdev = stdev;
-	switchtec_ntb_init_sndev(sndev);
+	rc = switchtec_ntb_init_sndev(sndev);
+	if (rc)
+		goto free_and_exit;
+
 	switchtec_ntb_init_mw(sndev);
 	switchtec_ntb_init_db(sndev);
 	switchtec_ntb_init_msgs(sndev);
diff --git a/include/linux/switchtec.h b/include/linux/switchtec.h
index 09d73d0d1aa8..d4a7c18b42cf 100644
--- a/include/linux/switchtec.h
+++ b/include/linux/switchtec.h
@@ -168,6 +168,14 @@ struct ntb_info_regs {
 	u16 reserved1;
 	u64 ep_map;
 	u16 requester_id;
+	u16 reserved2;
+	u32 reserved3[4];
+	struct nt_partition_info {
+		u32 xlink_enabled;
+		u32 target_part_low;
+		u32 target_part_high;
+		u32 reserved;
+	} ntp_info[48];
 } __packed;
 
 struct part_cfg_regs {
-- 
2.11.0

[PATCH 2/7] ntb_hw_switchtec: Keep track of the number of LUT windows used by the driver

[Logan Gunthorpe]

This is a prep patch in order to support the crosslink feature which will
require the driver to use another reserved LUT window. To simplify this,
we add some code to track the number of reserved LUT windows in use
instead of assuming this is always 1.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
 drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
index 088ae220ecb4..51fec6497164 100644
--- a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
+++ b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
@@ -109,6 +109,7 @@ struct switchtec_ntb {
 
 	int nr_direct_mw;
 	int nr_lut_mw;
+	int nr_rsvd_luts;
 	int direct_mw_to_bar[MAX_DIRECT_MW];
 
 	int peer_nr_direct_mw;
@@ -197,7 +198,7 @@ static int switchtec_ntb_mw_count(struct ntb_dev *ntb, int pidx)
 {
 	struct switchtec_ntb *sndev = ntb_sndev(ntb);
 	int nr_direct_mw = sndev->peer_nr_direct_mw;
-	int nr_lut_mw = sndev->peer_nr_lut_mw - 1;
+	int nr_lut_mw = sndev->peer_nr_lut_mw - sndev->nr_rsvd_luts;
 
 	if (pidx != NTB_DEF_PEER_IDX)
 		return -EINVAL;
@@ -210,12 +211,12 @@ static int switchtec_ntb_mw_count(struct ntb_dev *ntb, int pidx)
 
 static int lut_index(struct switchtec_ntb *sndev, int mw_idx)
 {
-	return mw_idx - sndev->nr_direct_mw + 1;
+	return mw_idx - sndev->nr_direct_mw + sndev->nr_rsvd_luts;
 }
 
 static int peer_lut_index(struct switchtec_ntb *sndev, int mw_idx)
 {
-	return mw_idx - sndev->peer_nr_direct_mw + 1;
+	return mw_idx - sndev->peer_nr_direct_mw + sndev->nr_rsvd_luts;
 }
 
 static int switchtec_ntb_mw_get_align(struct ntb_dev *ntb, int pidx,
@@ -355,8 +356,9 @@ static int switchtec_ntb_mw_set_trans(struct ntb_dev *ntb, int pidx, int widx,
 static int switchtec_ntb_peer_mw_count(struct ntb_dev *ntb)
 {
 	struct switchtec_ntb *sndev = ntb_sndev(ntb);
+	int nr_lut_mw = sndev->nr_lut_mw - sndev->nr_rsvd_luts;
 
-	return sndev->nr_direct_mw + (use_lut_mws ? sndev->nr_lut_mw - 1 : 0);
+	return sndev->nr_direct_mw + (use_lut_mws ? nr_lut_mw : 0);
 }
 
 static int switchtec_ntb_direct_get_addr(struct switchtec_ntb *sndev,
@@ -1008,6 +1010,7 @@ static int switchtec_ntb_init_shared_mw(struct switchtec_ntb *sndev)
 	u32 ctl_val;
 	int rc;
 
+	sndev->nr_rsvd_luts++;
 	sndev->self_shared = dma_zalloc_coherent(&sndev->stdev->pdev->dev,
 						 LUT_SIZE,
 						 &sndev->self_shared_dma,
@@ -1074,6 +1077,7 @@ static void switchtec_ntb_deinit_shared_mw(struct switchtec_ntb *sndev)
 		dma_free_coherent(&sndev->stdev->pdev->dev, LUT_SIZE,
 				  sndev->self_shared,
 				  sndev->self_shared_dma);
+	sndev->nr_rsvd_luts--;
 }
 
 static irqreturn_t switchtec_ntb_doorbell_isr(int irq, void *dev)
-- 
2.11.0

[PATCH 3/7] ntb_hw_switchtec: Create helper function to setup reserved LUT MWs

[Logan Gunthorpe]

This is a prep patch in order to support the crosslink feature which
will require the driver to use another reserved LUT window. To
simplify this we move the code which sets up the reserved LUT window
into a helper function which will be used by the crosslink
initialization.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
 drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 72 ++++++++++++++++++++--------------
 1 file changed, 43 insertions(+), 29 deletions(-)

diff --git a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
index 51fec6497164..b18e938312e1 100644
--- a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
+++ b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
@@ -847,6 +847,46 @@ static int switchtec_ntb_init_sndev(struct switchtec_ntb *sndev)
 	return 0;
 }
 
+static int config_rsvd_lut_win(struct switchtec_ntb *sndev,
+			       struct ntb_ctrl_regs __iomem *ctl,
+			       int lut_idx, int partition,
+			       dma_addr_t addr)
+{
+	int peer_bar = sndev->peer_direct_mw_to_bar[0];
+	u32 ctl_val;
+	int rc;
+
+	rc = switchtec_ntb_part_op(sndev, ctl, NTB_CTRL_PART_OP_LOCK,
+				   NTB_CTRL_PART_STATUS_LOCKED);
+	if (rc)
+		return rc;
+
+	ctl_val = ioread32(&ctl->bar_entry[peer_bar].ctl);
+	ctl_val &= 0xFF;
+	ctl_val |= NTB_CTRL_BAR_LUT_WIN_EN;
+	ctl_val |= ilog2(LUT_SIZE) << 8;
+	ctl_val |= (sndev->nr_lut_mw - 1) << 14;
+	iowrite32(ctl_val, &ctl->bar_entry[peer_bar].ctl);
+
+	iowrite64((NTB_CTRL_LUT_EN | (partition << 1) | addr),
+		  &ctl->lut_entry[lut_idx]);
+
+	rc = switchtec_ntb_part_op(sndev, ctl, NTB_CTRL_PART_OP_CFG,
+				   NTB_CTRL_PART_STATUS_NORMAL);
+	if (rc) {
+		u32 bar_error, lut_error;
+
+		bar_error = ioread32(&ctl->bar_error);
+		lut_error = ioread32(&ctl->lut_error);
+		dev_err(&sndev->stdev->dev,
+			"Error setting up reserved lut window: %08x / %08x\n",
+			bar_error, lut_error);
+		return rc;
+	}
+
+	return 0;
+}
+
 static int map_bars(int *map, struct ntb_ctrl_regs __iomem *ctrl)
 {
 	int i;
@@ -1004,10 +1044,7 @@ static void switchtec_ntb_init_shared(struct switchtec_ntb *sndev)
 
 static int switchtec_ntb_init_shared_mw(struct switchtec_ntb *sndev)
 {
-	struct ntb_ctrl_regs __iomem *ctl = sndev->mmio_peer_ctrl;
 	int self_bar = sndev->direct_mw_to_bar[0];
-	int peer_bar = sndev->peer_direct_mw_to_bar[0];
-	u32 ctl_val;
 	int rc;
 
 	sndev->nr_rsvd_luts++;
@@ -1023,35 +1060,12 @@ static int switchtec_ntb_init_shared_mw(struct switchtec_ntb *sndev)
 
 	switchtec_ntb_init_shared(sndev);
 
-	rc = switchtec_ntb_part_op(sndev, ctl, NTB_CTRL_PART_OP_LOCK,
-				   NTB_CTRL_PART_STATUS_LOCKED);
+	rc = config_rsvd_lut_win(sndev, sndev->mmio_peer_ctrl, 0,
+				 sndev->self_partition,
+				 sndev->self_shared_dma);
 	if (rc)
 		goto unalloc_and_exit;
 
-	ctl_val = ioread32(&ctl->bar_entry[peer_bar].ctl);
-	ctl_val &= 0xFF;
-	ctl_val |= NTB_CTRL_BAR_LUT_WIN_EN;
-	ctl_val |= ilog2(LUT_SIZE) << 8;
-	ctl_val |= (sndev->nr_lut_mw - 1) << 14;
-	iowrite32(ctl_val, &ctl->bar_entry[peer_bar].ctl);
-
-	iowrite64((NTB_CTRL_LUT_EN | (sndev->self_partition << 1) |
-		   sndev->self_shared_dma),
-		  &ctl->lut_entry[0]);
-
-	rc = switchtec_ntb_part_op(sndev, ctl, NTB_CTRL_PART_OP_CFG,
-				   NTB_CTRL_PART_STATUS_NORMAL);
-	if (rc) {
-		u32 bar_error, lut_error;
-
-		bar_error = ioread32(&ctl->bar_error);
-		lut_error = ioread32(&ctl->lut_error);
-		dev_err(&sndev->stdev->dev,
-			"Error setting up shared MW: %08x / %08x\n",
-			bar_error, lut_error);
-		goto unalloc_and_exit;
-	}
-
 	sndev->peer_shared = pci_iomap(sndev->stdev->pdev, self_bar, LUT_SIZE);
 	if (!sndev->peer_shared) {
 		rc = -ENOMEM;
-- 
2.11.0

[PATCH 4/7] ntb_hw_switchtec: Make switchtec_ntb_init_req_id_table() more general

[Logan Gunthorpe]

This is a prep patch in order to support the crosslink feature which
will require the driver to setup the requester ID table in another
partition as well as it's own. To aid this, create a helper function
which sets up the requester IDs from an array.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
 drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 92 +++++++++++++++++++++-------------
 1 file changed, 56 insertions(+), 36 deletions(-)

diff --git a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
index b18e938312e1..4adc32fe035a 100644
--- a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
+++ b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
@@ -887,6 +887,55 @@ static int config_rsvd_lut_win(struct switchtec_ntb *sndev,
 	return 0;
 }
 
+static int config_req_id_table(struct switchtec_ntb *sndev,
+			       struct ntb_ctrl_regs __iomem *mmio_ctrl,
+			       int *req_ids, int count)
+{
+	int i, rc = 0;
+	u32 error;
+	u32 proxy_id;
+
+	if (ioread32(&mmio_ctrl->req_id_table_size) < count) {
+		dev_err(&sndev->stdev->dev,
+			"Not enough requester IDs available.\n");
+		return -EFAULT;
+	}
+
+	rc = switchtec_ntb_part_op(sndev, mmio_ctrl,
+				   NTB_CTRL_PART_OP_LOCK,
+				   NTB_CTRL_PART_STATUS_LOCKED);
+	if (rc)
+		return rc;
+
+	iowrite32(NTB_PART_CTRL_ID_PROT_DIS,
+		  &mmio_ctrl->partition_ctrl);
+
+	for (i = 0; i < count; i++) {
+		iowrite32(req_ids[i] << 16 | NTB_CTRL_REQ_ID_EN,
+			  &mmio_ctrl->req_id_table[i]);
+
+		proxy_id = ioread32(&mmio_ctrl->req_id_table[i]);
+		dev_dbg(&sndev->stdev->dev,
+			"Requester ID %02X:%02X.%X -> BB:%02X.%X\n",
+			req_ids[i] >> 8, (req_ids[i] >> 3) & 0x1F,
+			req_ids[i] & 0x7, (proxy_id >> 4) & 0x1F,
+			(proxy_id >> 1) & 0x7);
+	}
+
+	rc = switchtec_ntb_part_op(sndev, mmio_ctrl,
+				   NTB_CTRL_PART_OP_CFG,
+				   NTB_CTRL_PART_STATUS_NORMAL);
+
+	if (rc == -EIO) {
+		error = ioread32(&mmio_ctrl->req_id_error);
+		dev_err(&sndev->stdev->dev,
+			"Error setting up the requester ID table: %08x\n",
+			error);
+	}
+
+	return 0;
+}
+
 static int map_bars(int *map, struct ntb_ctrl_regs __iomem *ctrl)
 {
 	int i;
@@ -968,52 +1017,23 @@ static void switchtec_ntb_init_msgs(struct switchtec_ntb *sndev)
 			  &sndev->mmio_self_dbmsg->imsg[i]);
 }
 
-static int switchtec_ntb_init_req_id_table(struct switchtec_ntb *sndev)
+static int
+switchtec_ntb_init_req_id_table(struct switchtec_ntb *sndev)
 {
-	int rc = 0;
-	u16 req_id;
-	u32 error;
-
-	req_id = ioread16(&sndev->mmio_ntb->requester_id);
-
-	if (ioread32(&sndev->mmio_self_ctrl->req_id_table_size) < 2) {
-		dev_err(&sndev->stdev->dev,
-			"Not enough requester IDs available\n");
-		return -EFAULT;
-	}
-
-	rc = switchtec_ntb_part_op(sndev, sndev->mmio_self_ctrl,
-				   NTB_CTRL_PART_OP_LOCK,
-				   NTB_CTRL_PART_STATUS_LOCKED);
-	if (rc)
-		return rc;
-
-	iowrite32(NTB_PART_CTRL_ID_PROT_DIS,
-		  &sndev->mmio_self_ctrl->partition_ctrl);
+	int req_ids[2];
 
 	/*
 	 * Root Complex Requester ID (which is 0:00.0)
 	 */
-	iowrite32(0 << 16 | NTB_CTRL_REQ_ID_EN,
-		  &sndev->mmio_self_ctrl->req_id_table[0]);
+	req_ids[0] = 0;
 
 	/*
 	 * Host Bridge Requester ID (as read from the mmap address)
 	 */
-	iowrite32(req_id << 16 | NTB_CTRL_REQ_ID_EN,
-		  &sndev->mmio_self_ctrl->req_id_table[1]);
-
-	rc = switchtec_ntb_part_op(sndev, sndev->mmio_self_ctrl,
-				   NTB_CTRL_PART_OP_CFG,
-				   NTB_CTRL_PART_STATUS_NORMAL);
-	if (rc == -EIO) {
-		error = ioread32(&sndev->mmio_self_ctrl->req_id_error);
-		dev_err(&sndev->stdev->dev,
-			"Error setting up the requester ID table: %08x\n",
-			error);
-	}
+	req_ids[1] = ioread16(&sndev->mmio_ntb->requester_id);
 
-	return rc;
+	return config_req_id_table(sndev, sndev->mmio_self_ctrl, req_ids,
+				   ARRAY_SIZE(req_ids));
 }
 
 static void switchtec_ntb_init_shared(struct switchtec_ntb *sndev)
-- 
2.11.0

[PATCH 5/7] ntb_hw_switchtec: Expand PFF CSR registers

[Logan Gunthorpe]

The PFF CSR registers actual mirrors the PCI configuration space
for all the ports in the switch. Previously, this was not needed by
the driver but will be used by the crosslink code to enumerate the
bus in an host-less centre partition.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
 include/linux/switchtec.h | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/include/linux/switchtec.h b/include/linux/switchtec.h
index d4a7c18b42cf..6d325a7a0c19 100644
--- a/include/linux/switchtec.h
+++ b/include/linux/switchtec.h
@@ -292,7 +292,20 @@ enum {
 struct pff_csr_regs {
 	u16 vendor_id;
 	u16 device_id;
-	u32 pci_cfg_header[15];
+	u16 pcicmd;
+	u16 pcists;
+	u32 pci_class;
+	u32 pci_opts;
+	union {
+		u32 pci_bar[6];
+		u64 pci_bar64[3];
+	};
+	u32 pci_cardbus;
+	u32 pci_subsystem_id;
+	u32 pci_expansion_rom;
+	u32 pci_cap_ptr;
+	u32 reserved1;
+	u32 pci_irq;
 	u32 pci_cap_region[48];
 	u32 pcie_cap_region[448];
 	u32 indirect_gas_window[128];
-- 
2.11.0

[PATCH 6/7] ntb_hw_switchtec: Add initialization code for crosslink

[Logan Gunthorpe]

Crosslink is a feature of the Switchtec switches that is similar to
the B2B mode of other NTB devices. It allows a system to be designed
that is perfectly symmetric with two identical switches that link
two hosts together.

In order for the system to be symmetric, there is an empty host-less
partition between the two switches which the host must enumerate and
assign BAR addresses to. The firmware in the switch manages this
specially so that the BAR addresses on both sides of the empty
partition will be identical despite being in the same partition with
the same address space.

The driver determines whether crosslink is enabled by a flag set in
the NTB partition info registers which are set by the switch's
configuration file.

When crosslink is enabled, a reserved LUT window is setup to point to
the peer's switch's NTB registers and the local MWs are set to forward
to the host-less partition's BARs. (Yes, this hurts my brain too.)
Once this is setup, largely the same NTB infrastructure is used to
communicate between the two hosts.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
 drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 202 ++++++++++++++++++++++++++++++++-
 include/linux/ntb.h                    |  15 ++-
 2 files changed, 206 insertions(+), 11 deletions(-)

diff --git a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
index 4adc32fe035a..17db0f50bb22 100644
--- a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
+++ b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
@@ -95,6 +95,8 @@ struct switchtec_ntb {
 	struct ntb_ctrl_regs __iomem *mmio_peer_ctrl;
 	struct ntb_dbmsg_regs __iomem *mmio_self_dbmsg;
 
+	void __iomem *mmio_xlink_win;
+
 	struct shared_mw *self_shared;
 	struct shared_mw __iomem *peer_shared;
 	dma_addr_t self_shared_dma;
@@ -465,6 +467,13 @@ static void switchtec_ntb_set_link_speed(struct switchtec_ntb *sndev)
 	sndev->link_width = min(self_width, peer_width);
 }
 
+static int crosslink_is_enabled(struct switchtec_ntb *sndev)
+{
+	struct ntb_info_regs __iomem *inf = sndev->mmio_ntb;
+
+	return ioread8(&inf->ntp_info[sndev->peer_partition].xlink_enabled);
+}
+
 enum {
 	LINK_MESSAGE = 0,
 	MSG_LINK_UP = 1,
@@ -849,8 +858,7 @@ static int switchtec_ntb_init_sndev(struct switchtec_ntb *sndev)
 
 static int config_rsvd_lut_win(struct switchtec_ntb *sndev,
 			       struct ntb_ctrl_regs __iomem *ctl,
-			       int lut_idx, int partition,
-			       dma_addr_t addr)
+			       int lut_idx, int partition, u64 addr)
 {
 	int peer_bar = sndev->peer_direct_mw_to_bar[0];
 	u32 ctl_val;
@@ -936,6 +944,182 @@ static int config_req_id_table(struct switchtec_ntb *sndev,
 	return 0;
 }
 
+static int crosslink_setup_mws(struct switchtec_ntb *sndev, int ntb_lut_idx,
+			       u64 *mw_addrs, int mw_count)
+{
+	int rc, i;
+	struct ntb_ctrl_regs __iomem *ctl = sndev->mmio_self_ctrl;
+	u64 addr;
+	size_t size, offset;
+	int bar;
+	int xlate_pos;
+	u32 ctl_val;
+
+	rc = switchtec_ntb_part_op(sndev, ctl, NTB_CTRL_PART_OP_LOCK,
+				   NTB_CTRL_PART_STATUS_LOCKED);
+	if (rc)
+		return rc;
+
+	for (i = 0; i < sndev->nr_lut_mw; i++) {
+		if (i == ntb_lut_idx)
+			continue;
+
+		addr = mw_addrs[0] + LUT_SIZE * i;
+
+		iowrite64((NTB_CTRL_LUT_EN | (sndev->peer_partition << 1) |
+			   addr),
+			  &ctl->lut_entry[i]);
+	}
+
+	sndev->nr_direct_mw = min_t(int, sndev->nr_direct_mw, mw_count);
+
+	for (i = 0; i < sndev->nr_direct_mw; i++) {
+		bar = sndev->direct_mw_to_bar[i];
+		offset = (i == 0) ? LUT_SIZE * sndev->nr_lut_mw : 0;
+		addr = mw_addrs[i] + offset;
+		size = pci_resource_len(sndev->ntb.pdev, bar) - offset;
+		xlate_pos = ilog2(size);
+
+		if (offset && size > offset)
+			size = offset;
+
+		ctl_val = ioread32(&ctl->bar_entry[bar].ctl);
+		ctl_val |= NTB_CTRL_BAR_DIR_WIN_EN;
+
+		iowrite32(ctl_val, &ctl->bar_entry[bar].ctl);
+		iowrite32(xlate_pos | size, &ctl->bar_entry[bar].win_size);
+		iowrite64(sndev->peer_partition | addr,
+			  &ctl->bar_entry[bar].xlate_addr);
+	}
+
+	rc = switchtec_ntb_part_op(sndev, ctl, NTB_CTRL_PART_OP_CFG,
+				   NTB_CTRL_PART_STATUS_NORMAL);
+	if (rc) {
+		u32 bar_error, lut_error;
+
+		bar_error = ioread32(&ctl->bar_error);
+		lut_error = ioread32(&ctl->lut_error);
+		dev_err(&sndev->stdev->dev,
+			"Error setting up cross link windows: %08x / %08x\n",
+			bar_error, lut_error);
+		return rc;
+	}
+
+	return 0;
+}
+
+static int crosslink_setup_req_ids(struct switchtec_ntb *sndev,
+	struct ntb_ctrl_regs __iomem *mmio_ctrl)
+{
+	int req_ids[16];
+	int i;
+	u32 proxy_id;
+
+	for (i = 0; i < ARRAY_SIZE(req_ids); i++) {
+		proxy_id = ioread32(&sndev->mmio_self_ctrl->req_id_table[i]);
+
+		if (!(proxy_id & NTB_CTRL_REQ_ID_EN))
+			break;
+
+		req_ids[i] = ((proxy_id >> 1) & 0xFF);
+	}
+
+	return config_req_id_table(sndev, mmio_ctrl, req_ids, i);
+}
+
+/*
+ * In crosslink configuration there is a virtual partition in the
+ * middle of the two switches. The BARs in this partition have to be
+ * enumerated and assigned addresses.
+ */
+static int crosslink_enum_partition(struct switchtec_ntb *sndev,
+				    u64 *bar_addrs)
+{
+	struct part_cfg_regs __iomem *part_cfg =
+		&sndev->stdev->mmio_part_cfg_all[sndev->peer_partition];
+	u32 pff = ioread32(&part_cfg->vep_pff_inst_id);
+	struct pff_csr_regs __iomem *mmio_pff =
+		&sndev->stdev->mmio_pff_csr[pff];
+	const u64 bar_space = 0x1000000000LL;
+	u64 bar_addr;
+	int bar_cnt = 0;
+	int i;
+
+	iowrite16(0x6, &mmio_pff->pcicmd);
+
+	for (i = 0; i < ARRAY_SIZE(mmio_pff->pci_bar64); i++) {
+		iowrite64(bar_space * i, &mmio_pff->pci_bar64[i]);
+		bar_addr = ioread64(&mmio_pff->pci_bar64[i]);
+		bar_addr &= ~0xf;
+
+		dev_dbg(&sndev->stdev->dev,
+			"Crosslink BAR%d addr: %llx\n",
+			i, bar_addr);
+
+		if (bar_addr != bar_space * i)
+			continue;
+
+		bar_addrs[bar_cnt++] = bar_addr;
+	}
+
+	return bar_cnt;
+}
+
+static int switchtec_ntb_init_crosslink(struct switchtec_ntb *sndev)
+{
+	int rc;
+	int bar = sndev->direct_mw_to_bar[0];
+	const int ntb_lut_idx = 1;
+	u64 bar_addrs[6];
+	u64 addr;
+	int bar_cnt;
+
+	if (!crosslink_is_enabled(sndev))
+		return 0;
+
+	dev_info(&sndev->stdev->dev, "Using crosslink configuration\n");
+	sndev->ntb.topo = NTB_TOPO_CROSSLINK;
+
+	bar_cnt = crosslink_enum_partition(sndev, bar_addrs);
+	if (bar_cnt < sndev->nr_direct_mw + 1) {
+		dev_err(&sndev->stdev->dev,
+			"Error enumerating crosslink partition\n");
+		return -EINVAL;
+	}
+
+	addr = bar_addrs[0];
+	rc = config_rsvd_lut_win(sndev, sndev->mmio_self_ctrl, ntb_lut_idx,
+				 sndev->peer_partition, addr);
+	if (rc)
+		return rc;
+
+	rc = crosslink_setup_mws(sndev, ntb_lut_idx, &bar_addrs[1],
+				 bar_cnt - 1);
+	if (rc)
+		return rc;
+
+	rc = crosslink_setup_req_ids(sndev, sndev->mmio_peer_ctrl);
+	if (rc)
+		return rc;
+
+	sndev->mmio_xlink_win = pci_iomap_range(sndev->stdev->pdev, bar,
+						LUT_SIZE, LUT_SIZE);
+	if (!sndev->mmio_xlink_win) {
+		rc = -ENOMEM;
+		return rc;
+	}
+
+	sndev->nr_rsvd_luts++;
+
+	return 0;
+}
+
+static void switchtec_ntb_deinit_crosslink(struct switchtec_ntb *sndev)
+{
+	if (sndev->mmio_xlink_win)
+		pci_iounmap(sndev->stdev->pdev, sndev->mmio_xlink_win);
+}
+
 static int map_bars(int *map, struct ntb_ctrl_regs __iomem *ctrl)
 {
 	int i;
@@ -1222,17 +1406,22 @@ static int switchtec_ntb_add(struct device *dev,
 		goto free_and_exit;
 
 	switchtec_ntb_init_mw(sndev);
-	switchtec_ntb_init_db(sndev);
-	switchtec_ntb_init_msgs(sndev);
 
 	rc = switchtec_ntb_init_req_id_table(sndev);
 	if (rc)
 		goto free_and_exit;
 
-	rc = switchtec_ntb_init_shared_mw(sndev);
+	rc = switchtec_ntb_init_crosslink(sndev);
 	if (rc)
 		goto free_and_exit;
 
+	switchtec_ntb_init_db(sndev);
+	switchtec_ntb_init_msgs(sndev);
+
+	rc = switchtec_ntb_init_shared_mw(sndev);
+	if (rc)
+		goto deinit_crosslink;
+
 	rc = switchtec_ntb_init_db_msg_irq(sndev);
 	if (rc)
 		goto deinit_shared_and_exit;
@@ -1251,6 +1440,8 @@ static int switchtec_ntb_add(struct device *dev,
 	switchtec_ntb_deinit_db_msg_irq(sndev);
 deinit_shared_and_exit:
 	switchtec_ntb_deinit_shared_mw(sndev);
+deinit_crosslink:
+	switchtec_ntb_deinit_crosslink(sndev);
 free_and_exit:
 	kfree(sndev);
 	dev_err(dev, "failed to register ntb device: %d\n", rc);
@@ -1271,6 +1462,7 @@ void switchtec_ntb_remove(struct device *dev,
 	ntb_unregister_device(&sndev->ntb);
 	switchtec_ntb_deinit_db_msg_irq(sndev);
 	switchtec_ntb_deinit_shared_mw(sndev);
+	switchtec_ntb_deinit_crosslink(sndev);
 	kfree(sndev);
 	dev_info(dev, "ntb device unregistered\n");
 }
diff --git a/include/linux/ntb.h b/include/linux/ntb.h
index c308964777eb..ea3be7275a5e 100644
--- a/include/linux/ntb.h
+++ b/include/linux/ntb.h
@@ -71,6 +71,7 @@ struct pci_dev;
  * @NTB_TOPO_B2B_USD:	On primary side of local ntb upstream of remote ntb.
  * @NTB_TOPO_B2B_DSD:	On primary side of local ntb downstream of remote ntb.
  * @NTB_TOPO_SWITCH:	Connected via a switch which supports ntb.
+ * @NTB_TOPO_CROSSLINK: Connected via two symmetric switchecs
  */
 enum ntb_topo {
 	NTB_TOPO_NONE = -1,
@@ -79,6 +80,7 @@ enum ntb_topo {
 	NTB_TOPO_B2B_USD,
 	NTB_TOPO_B2B_DSD,
 	NTB_TOPO_SWITCH,
+	NTB_TOPO_CROSSLINK,
 };
 
 static inline int ntb_topo_is_b2b(enum ntb_topo topo)
@@ -94,12 +96,13 @@ static inline int ntb_topo_is_b2b(enum ntb_topo topo)
 static inline char *ntb_topo_string(enum ntb_topo topo)
 {
 	switch (topo) {
-	case NTB_TOPO_NONE:	return "NTB_TOPO_NONE";
-	case NTB_TOPO_PRI:	return "NTB_TOPO_PRI";
-	case NTB_TOPO_SEC:	return "NTB_TOPO_SEC";
-	case NTB_TOPO_B2B_USD:	return "NTB_TOPO_B2B_USD";
-	case NTB_TOPO_B2B_DSD:	return "NTB_TOPO_B2B_DSD";
-	case NTB_TOPO_SWITCH:	return "NTB_TOPO_SWITCH";
+	case NTB_TOPO_NONE:		return "NTB_TOPO_NONE";
+	case NTB_TOPO_PRI:		return "NTB_TOPO_PRI";
+	case NTB_TOPO_SEC:		return "NTB_TOPO_SEC";
+	case NTB_TOPO_B2B_USD:		return "NTB_TOPO_B2B_USD";
+	case NTB_TOPO_B2B_DSD:		return "NTB_TOPO_B2B_DSD";
+	case NTB_TOPO_SWITCH:		return "NTB_TOPO_SWITCH";
+	case NTB_TOPO_CROSSLINK:	return "NTB_TOPO_CROSSLINK";
 	}
 	return "NTB_TOPO_INVALID";
 }
-- 
2.11.0

[PATCH 7/7] ntb_hw_switchtec: Crosslink doorbells and messages

[Logan Gunthorpe]

In a crosslink configuration doorbells and messages largely work the
same but the NTB registers must be accessed through the reserved LUT
window. Also, as a bonus, seeing there are now two independent sets of
NTB links, both partitions can actually use all 60 doorbell registers
instead of them having to be split into two for each partition.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
 drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 65 ++++++++++++++++++++++++++++------
 1 file changed, 55 insertions(+), 10 deletions(-)

diff --git a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
index 17db0f50bb22..145b31209f20 100644
--- a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
+++ b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
@@ -94,6 +94,7 @@ struct switchtec_ntb {
 	struct ntb_ctrl_regs __iomem *mmio_self_ctrl;
 	struct ntb_ctrl_regs __iomem *mmio_peer_ctrl;
 	struct ntb_dbmsg_regs __iomem *mmio_self_dbmsg;
+	struct ntb_dbmsg_regs __iomem *mmio_peer_dbmsg;
 
 	void __iomem *mmio_xlink_win;
 
@@ -188,10 +189,10 @@ static int switchtec_ntb_part_op(struct switchtec_ntb *sndev,
 static int switchtec_ntb_send_msg(struct switchtec_ntb *sndev, int idx,
 				  u32 val)
 {
-	if (idx < 0 || idx >= ARRAY_SIZE(sndev->mmio_self_dbmsg->omsg))
+	if (idx < 0 || idx >= ARRAY_SIZE(sndev->mmio_peer_dbmsg->omsg))
 		return -EINVAL;
 
-	iowrite32(val, &sndev->mmio_self_dbmsg->omsg[idx].msg);
+	iowrite32(val, &sndev->mmio_peer_dbmsg->omsg[idx].msg);
 
 	return 0;
 }
@@ -474,6 +475,25 @@ static int crosslink_is_enabled(struct switchtec_ntb *sndev)
 	return ioread8(&inf->ntp_info[sndev->peer_partition].xlink_enabled);
 }
 
+static void crosslink_init_dbmsgs(struct switchtec_ntb *sndev)
+{
+	int i;
+	u32 msg_map = 0;
+
+	if (!crosslink_is_enabled(sndev))
+		return;
+
+	for (i = 0; i < ARRAY_SIZE(sndev->mmio_peer_dbmsg->imsg); i++) {
+		int m = i | sndev->self_partition << 2;
+
+		msg_map |= m << i * 8;
+	}
+
+	iowrite32(msg_map, &sndev->mmio_peer_dbmsg->msg_map);
+	iowrite64(sndev->db_valid_mask << sndev->db_peer_shift,
+		  &sndev->mmio_peer_dbmsg->odb_mask);
+}
+
 enum {
 	LINK_MESSAGE = 0,
 	MSG_LINK_UP = 1,
@@ -504,6 +524,9 @@ static void switchtec_ntb_check_link(struct switchtec_ntb *sndev)
 		ntb_link_event(&sndev->ntb);
 		dev_info(&sndev->stdev->dev, "ntb link %s\n",
 			 link_sta ? "up" : "down");
+
+		if (link_sta)
+			crosslink_init_dbmsgs(sndev);
 	}
 }
 
@@ -649,7 +672,7 @@ static int switchtec_ntb_peer_db_addr(struct ntb_dev *ntb,
 	struct switchtec_ntb *sndev = ntb_sndev(ntb);
 	unsigned long offset;
 
-	offset = (unsigned long)sndev->mmio_self_dbmsg->odb -
+	offset = (unsigned long)sndev->mmio_peer_dbmsg->odb -
 		(unsigned long)sndev->stdev->mmio;
 
 	offset += sndev->db_shift / 8;
@@ -667,7 +690,7 @@ static int switchtec_ntb_peer_db_set(struct ntb_dev *ntb, u64 db_bits)
 	struct switchtec_ntb *sndev = ntb_sndev(ntb);
 
 	iowrite64(db_bits << sndev->db_peer_shift,
-		  &sndev->mmio_self_dbmsg->odb);
+		  &sndev->mmio_peer_dbmsg->odb);
 
 	return 0;
 }
@@ -852,6 +875,7 @@ static int switchtec_ntb_init_sndev(struct switchtec_ntb *sndev)
 	sndev->mmio_self_ctrl = &sndev->mmio_ctrl[sndev->self_partition];
 	sndev->mmio_peer_ctrl = &sndev->mmio_ctrl[sndev->peer_partition];
 	sndev->mmio_self_dbmsg = &sndev->mmio_dbmsg[sndev->self_partition];
+	sndev->mmio_peer_dbmsg = sndev->mmio_self_dbmsg;
 
 	return 0;
 }
@@ -1072,6 +1096,7 @@ static int switchtec_ntb_init_crosslink(struct switchtec_ntb *sndev)
 	const int ntb_lut_idx = 1;
 	u64 bar_addrs[6];
 	u64 addr;
+	int offset;
 	int bar_cnt;
 
 	if (!crosslink_is_enabled(sndev))
@@ -1087,7 +1112,13 @@ static int switchtec_ntb_init_crosslink(struct switchtec_ntb *sndev)
 		return -EINVAL;
 	}
 
-	addr = bar_addrs[0];
+	addr = (bar_addrs[0] + SWITCHTEC_GAS_NTB_OFFSET +
+		SWITCHTEC_NTB_REG_DBMSG_OFFSET +
+		sizeof(struct ntb_dbmsg_regs) * sndev->peer_partition);
+
+	offset = addr & (LUT_SIZE - 1);
+	addr -= offset;
+
 	rc = config_rsvd_lut_win(sndev, sndev->mmio_self_ctrl, ntb_lut_idx,
 				 sndev->peer_partition, addr);
 	if (rc)
@@ -1109,8 +1140,11 @@ static int switchtec_ntb_init_crosslink(struct switchtec_ntb *sndev)
 		return rc;
 	}
 
+	sndev->mmio_peer_dbmsg = sndev->mmio_xlink_win + offset;
 	sndev->nr_rsvd_luts++;
 
+	crosslink_init_dbmsgs(sndev);
+
 	return 0;
 }
 
@@ -1163,24 +1197,35 @@ static void switchtec_ntb_init_mw(struct switchtec_ntb *sndev)
  * shared among all partitions. So we must split them in half
  * (32 for each partition). However, the message interrupts are
  * also shared with the top 4 doorbells so we just limit this to
- * 28 doorbells per partition
+ * 28 doorbells per partition.
+ *
+ * In crosslink mode, each side has it's own dbmsg register so
+ * they can each use all 60 of the available doorbells.
  */
 static void switchtec_ntb_init_db(struct switchtec_ntb *sndev)
 {
-	sndev->db_valid_mask = 0x0FFFFFFF;
+	sndev->db_mask = 0x0FFFFFFFFFFFFFFFULL;
 
-	if (sndev->self_partition < sndev->peer_partition) {
+	if (sndev->mmio_peer_dbmsg != sndev->mmio_self_dbmsg) {
+		sndev->db_shift = 0;
+		sndev->db_peer_shift = 0;
+		sndev->db_valid_mask = sndev->db_mask;
+	} else if (sndev->self_partition < sndev->peer_partition) {
 		sndev->db_shift = 0;
 		sndev->db_peer_shift = 32;
+		sndev->db_valid_mask = 0x0FFFFFFF;
 	} else {
 		sndev->db_shift = 32;
 		sndev->db_peer_shift = 0;
+		sndev->db_valid_mask = 0x0FFFFFFF;
 	}
 
-	sndev->db_mask = 0x0FFFFFFFFFFFFFFFULL;
 	iowrite64(~sndev->db_mask, &sndev->mmio_self_dbmsg->idb_mask);
 	iowrite64(sndev->db_valid_mask << sndev->db_peer_shift,
-		  &sndev->mmio_self_dbmsg->odb_mask);
+		  &sndev->mmio_peer_dbmsg->odb_mask);
+
+	dev_dbg(&sndev->stdev->dev, "dbs: shift %d/%d, mask %016llx\n",
+		sndev->db_shift, sndev->db_peer_shift, sndev->db_valid_mask);
 }
 
 static void switchtec_ntb_init_msgs(struct switchtec_ntb *sndev)
-- 
2.11.0

Re: [PATCH 0/7] Switchtec NTB Crosslink Support

[Logan Gunthorpe]

Also, I forgot to mention, this patch set is based on today's ntb-next.

Logan

On 29/11/17 10:55 AM, Logan Gunthorpe wrote:
> Hi,
> 
> This patch series adds support for the Switchtec Crosslink feature.
> Crosslink is similar to B2B in that it allows two switches to be
> connected back to back.
> 
> Two switches can already be connected between two hosts, however with this
> setup, there would be no symmetry as one switch would handle NTB and the
> other would just need to be a single partition switch. Our customers are
> looking for a completely symmetric solution such that a host or switch
> can be easily replaced without needing to worry about which half was
> replaced.
> 
> Crosslink solves this problem by having each of the switches configured
> with two partitions. This results in a special host-less partition in
> the middle of the connection which can be managed in such a way as
> to forward all the traffic and still provide all the same NTB
> functionality. In this way, both hosts and switches can be identical.
> 
> Please see Patch 6 for a bit more detailed description of how Crosslink
> works.
> 
> Thanks,
> 
> Logan
> 
> Kelvin Cao (1):
>    ntb_hw_switchtec: Allow using Switchtec NTB in multi-partition setups
> 
> Logan Gunthorpe (6):
>    ntb_hw_switchtec: Keep track of the number of LUT windows used by the
>      driver
>    ntb_hw_switchtec: Create helper function to setup reserved LUT MWs
>    ntb_hw_switchtec: Make switchtec_ntb_init_req_id_table() more general
>    ntb_hw_switchtec: Expand PFF CSR registers
>    ntb_hw_switchtec: Add initialization code for crosslink
>    ntb_hw_switchtec: Crosslink doorbells and messages
> 
>   drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 494 +++++++++++++++++++++++++++------
>   include/linux/ntb.h                    |  15 +-
>   include/linux/switchtec.h              |  23 +-
>   3 files changed, 435 insertions(+), 97 deletions(-)
> 
> --
> 2.11.0
> 

Re: [PATCH 5/7] ntb_hw_switchtec: Expand PFF CSR registers

[Jon Mason]

On Wed, Nov 29, 2017 at 12:55 PM, Logan Gunthorpe <logang@deltatee.com> wrote:
> The PFF CSR registers actual mirrors the PCI configuration space
> for all the ports in the switch. Previously, this was not needed by
> the driver but will be used by the crosslink code to enumerate the
> bus in an host-less centre partition.
>
> Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
> ---
>  include/linux/switchtec.h | 15 ++++++++++++++-
>  1 file changed, 14 insertions(+), 1 deletion(-)
>
> diff --git a/include/linux/switchtec.h b/include/linux/switchtec.h
> index d4a7c18b42cf..6d325a7a0c19 100644
> --- a/include/linux/switchtec.h
> +++ b/include/linux/switchtec.h
> @@ -292,7 +292,20 @@ enum {
>  struct pff_csr_regs {
>         u16 vendor_id;
>         u16 device_id;
> -       u32 pci_cfg_header[15];
> +       u16 pcicmd;
> +       u16 pcists;
> +       u32 pci_class;
> +       u32 pci_opts;
> +       union {
> +               u32 pci_bar[6];
> +               u64 pci_bar64[3];
> +       };
> +       u32 pci_cardbus;
> +       u32 pci_subsystem_id;
> +       u32 pci_expansion_rom;
> +       u32 pci_cap_ptr;
> +       u32 reserved1;
> +       u32 pci_irq;

It sucks that we don't already have a struct for PCI config space we
can reuse here.  If you find the time, it would be good to add in the
future to reduce duplicate code here and in the PCI core.  However,
this patch is fine without it.

Thanks,
Jon


>         u32 pci_cap_region[48];
>         u32 pcie_cap_region[448];
>         u32 indirect_gas_window[128];
> --
> 2.11.0
>

Re: [PATCH 0/7] Switchtec NTB Crosslink Support

[Jon Mason]

On Wed, Nov 29, 2017 at 12:58 PM, Logan Gunthorpe <logang@deltatee.com> wrote:
> Also, I forgot to mention, this patch set is based on today's ntb-next.

All of these look sane to me.  Assuming they apply cleanly, adding to ntb-next.

Thanks,
Jon

>
> Logan
>
>
> On 29/11/17 10:55 AM, Logan Gunthorpe wrote:
>>
>> Hi,
>>
>> This patch series adds support for the Switchtec Crosslink feature.
>> Crosslink is similar to B2B in that it allows two switches to be
>> connected back to back.
>>
>> Two switches can already be connected between two hosts, however with this
>> setup, there would be no symmetry as one switch would handle NTB and the
>> other would just need to be a single partition switch. Our customers are
>> looking for a completely symmetric solution such that a host or switch
>> can be easily replaced without needing to worry about which half was
>> replaced.
>>
>> Crosslink solves this problem by having each of the switches configured
>> with two partitions. This results in a special host-less partition in
>> the middle of the connection which can be managed in such a way as
>> to forward all the traffic and still provide all the same NTB
>> functionality. In this way, both hosts and switches can be identical.
>>
>> Please see Patch 6 for a bit more detailed description of how Crosslink
>> works.
>>
>> Thanks,
>>
>> Logan
>>
>> Kelvin Cao (1):
>>    ntb_hw_switchtec: Allow using Switchtec NTB in multi-partition setups
>>
>> Logan Gunthorpe (6):
>>    ntb_hw_switchtec: Keep track of the number of LUT windows used by the
>>      driver
>>    ntb_hw_switchtec: Create helper function to setup reserved LUT MWs
>>    ntb_hw_switchtec: Make switchtec_ntb_init_req_id_table() more general
>>    ntb_hw_switchtec: Expand PFF CSR registers
>>    ntb_hw_switchtec: Add initialization code for crosslink
>>    ntb_hw_switchtec: Crosslink doorbells and messages
>>
>>   drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 494
>> +++++++++++++++++++++++++++------
>>   include/linux/ntb.h                    |  15 +-
>>   include/linux/switchtec.h              |  23 +-
>>   3 files changed, 435 insertions(+), 97 deletions(-)
>>
>> --
>> 2.11.0
>>
>

Re: [PATCH 5/7] ntb_hw_switchtec: Expand PFF CSR registers

[Logan Gunthorpe]

On 05/12/17 12:12 PM, Jon Mason wrote:
> It sucks that we don't already have a struct for PCI config space we
> can reuse here.  If you find the time, it would be good to add in the
> future to reduce duplicate code here and in the PCI core.  However,
> this patch is fine without it.

I agree. And believe me I looked for one. I'm not sure if it makes sense 
to add it to general code at this time when I'm the only user.

Logan

Re: [PATCH 0/7] Switchtec NTB Crosslink Support

[Logan Gunthorpe]

On 05/12/17 12:15 PM, Jon Mason wrote:
> On Wed, Nov 29, 2017 at 12:58 PM, Logan Gunthorpe <logang@deltatee.com> wrote:
>> Also, I forgot to mention, this patch set is based on today's ntb-next.
> 
> All of these look sane to me.  Assuming they apply cleanly, adding to ntb-next.

Great, thanks!

Logan

Re: [PATCH 5/7] ntb_hw_switchtec: Expand PFF CSR registers

[Jon Mason]

On Tue, Dec 5, 2017 at 2:40 PM, Logan Gunthorpe <logang@deltatee.com> wrote:
>
>
> On 05/12/17 12:12 PM, Jon Mason wrote:
>>
>> It sucks that we don't already have a struct for PCI config space we
>> can reuse here.  If you find the time, it would be good to add in the
>> future to reduce duplicate code here and in the PCI core.  However,
>> this patch is fine without it.
>
>
> I agree. And believe me I looked for one. I'm not sure if it makes sense to
> add it to general code at this time when I'm the only user.

No problem.

>
> Logan