[PATCH 0/6] Switchtec NTB Fixes and Improvements

[PATCH 0/6] Switchtec NTB Fixes and Improvements

[Kelvin Cao]

Hi,

Please find a bunch of patches for the Switchtec NTB driver.

Patche 1, 2 and 6 fix three minor bugs. Patch 3 works around a minor
firmware issue. Patch 4 updates the method of getting management VEP
instance ID based on a new firmware change. Patch 5 removes code that
disables ID protection to avoid conflict with static Switchtec config
settings.

This patchset is based on 5.16.0-rc5.

Thanks,
Kelvin

Jeremy Pallotta (2):
  ntb_hw_switchtec: Fix pff ioread to read into mmio_part_cfg_all
  ntb_hw_switchtec: AND with the part_map for a valid tpart_vec

Kelvin Cao (3):
  ntb_hw_switchtec: Update the way of getting VEP instance ID
  ntb_hw_switchtec: Remove code for disabling ID protection
  ntb_hw_switchtec: Fix a minor issue in config_req_id_table()

Wesley Sheng (1):
  ntb_hw_switchtec: Fix bug with more than 32 partitions

 drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 24 +++++++++++-------------
 include/linux/switchtec.h              |  2 --
 2 files changed, 11 insertions(+), 15 deletions(-)

-- 
2.25.1

[PATCH 1/6] ntb_hw_switchtec: Fix pff ioread to read into mmio_part_cfg_all

[Kelvin Cao]

From: Jeremy Pallotta <jmpallotta@gmail.com>

Array mmio_part_cfg_all holds the partition configuration of all
partitions, with partition number as index. Fix this by reading into
mmio_part_cfg_all for pff.

Fixes: 0ee28f26f378 ("NTB: switchtec_ntb: Add link management")
Signed-off-by: Jeremy Pallotta <jmpallotta@gmail.com>
Signed-off-by: Kelvin Cao <kelvin.cao@microchip.com>
---
 drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
index 4c6eb61a6ac6..6603c77c0a84 100644
--- a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
+++ b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
@@ -419,8 +419,8 @@ static void switchtec_ntb_part_link_speed(struct switchtec_ntb *sndev,
 					  enum ntb_width *width)
 {
 	struct switchtec_dev *stdev = sndev->stdev;
-
-	u32 pff = ioread32(&stdev->mmio_part_cfg[partition].vep_pff_inst_id);
+	u32 pff =
+		ioread32(&stdev->mmio_part_cfg_all[partition].vep_pff_inst_id);
 	u32 linksta = ioread32(&stdev->mmio_pff_csr[pff].pci_cap_region[13]);
 
 	if (speed)
-- 
2.25.1

[PATCH 2/6] ntb_hw_switchtec: Fix bug with more than 32 partitions

[Kelvin Cao]

From: Wesley Sheng <wesley.sheng@microchip.com>

Switchtec could support as mush as 48 partitions, but ffs & fls are
for 32 bit argument, in case of partition index larger than 31, the
current code could not parse the peer partition index correctly.
Change to the 64 bit version __ffs64 & fls64 accordingly to fix this
bug.

Fixes: 3df54c870f52 ("ntb_hw_switchtec: Allow using Switchtec NTB in multi-partition setups")
Signed-off-by: Wesley Sheng <wesley.sheng@microchip.com>
Signed-off-by: Kelvin Cao <kelvin.cao@microchip.com>
---
 drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
index 6603c77c0a84..ec9cb6c81eda 100644
--- a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
+++ b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
@@ -840,7 +840,6 @@ static int switchtec_ntb_init_sndev(struct switchtec_ntb *sndev)
 	u64 tpart_vec;
 	int self;
 	u64 part_map;
-	int bit;
 
 	sndev->ntb.pdev = sndev->stdev->pdev;
 	sndev->ntb.topo = NTB_TOPO_SWITCH;
@@ -861,29 +860,28 @@ static int switchtec_ntb_init_sndev(struct switchtec_ntb *sndev)
 	part_map = ioread64(&sndev->mmio_ntb->ep_map);
 	part_map &= ~(1 << sndev->self_partition);
 
-	if (!ffs(tpart_vec)) {
+	if (!tpart_vec) {
 		if (sndev->stdev->partition_count != 2) {
 			dev_err(&sndev->stdev->dev,
 				"ntb target partition not defined\n");
 			return -ENODEV;
 		}
 
-		bit = ffs(part_map);
-		if (!bit) {
+		if (!part_map) {
 			dev_err(&sndev->stdev->dev,
 				"peer partition is not NT partition\n");
 			return -ENODEV;
 		}
 
-		sndev->peer_partition = bit - 1;
+		sndev->peer_partition = __ffs64(part_map);
 	} else {
-		if (ffs(tpart_vec) != fls(tpart_vec)) {
+		if (__ffs64(tpart_vec) != (fls64(tpart_vec) - 1)) {
 			dev_err(&sndev->stdev->dev,
 				"ntb driver only supports 1 pair of 1-1 ntb mapping\n");
 			return -ENODEV;
 		}
 
-		sndev->peer_partition = ffs(tpart_vec) - 1;
+		sndev->peer_partition = __ffs64(tpart_vec);
 		if (!(part_map & (1ULL << sndev->peer_partition))) {
 			dev_err(&sndev->stdev->dev,
 				"ntb target partition is not NT partition\n");
-- 
2.25.1

[PATCH 3/6] ntb_hw_switchtec: AND with the part_map for a valid tpart_vec

[Kelvin Cao]

From: Jeremy Pallotta <jmpallotta@gmail.com>

Some firmware versions return 1 in the target partition vector for
undefined partitions. AND with the part_map to give a valid tpart_vec.

Signed-off-by: Jeremy Pallotta <jmpallotta@gmail.com>
Signed-off-by: Kelvin Cao <kelvin.cao@microchip.com>
---
 drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
index ec9cb6c81eda..25302a384a7d 100644
--- a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
+++ b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
@@ -858,6 +858,7 @@ static int switchtec_ntb_init_sndev(struct switchtec_ntb *sndev)
 	tpart_vec |= ioread32(&sndev->mmio_ntb->ntp_info[self].target_part_low);
 
 	part_map = ioread64(&sndev->mmio_ntb->ep_map);
+	tpart_vec &= part_map;
 	part_map &= ~(1 << sndev->self_partition);
 
 	if (!tpart_vec) {
-- 
2.25.1

[PATCH 4/6] ntb_hw_switchtec: Update the way of getting VEP instance ID

[Kelvin Cao]

Gen4 firmware adds DMA VEP and NVMe VEP support in VEP (virtual EP)
instance ID register in addtion to management EP. Update the way of
getting management VEP instance ID.

Signed-off-by: Kelvin Cao <kelvin.cao@microchip.com>
---
 drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
index 25302a384a7d..03839346233d 100644
--- a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
+++ b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
@@ -419,8 +419,10 @@ static void switchtec_ntb_part_link_speed(struct switchtec_ntb *sndev,
 					  enum ntb_width *width)
 {
 	struct switchtec_dev *stdev = sndev->stdev;
-	u32 pff =
-		ioread32(&stdev->mmio_part_cfg_all[partition].vep_pff_inst_id);
+	struct part_cfg_regs __iomem *part_cfg =
+		&stdev->mmio_part_cfg_all[partition];
+
+	u32 pff = ioread32(&part_cfg->vep_pff_inst_id) & 0xFF;
 	u32 linksta = ioread32(&stdev->mmio_pff_csr[pff].pci_cap_region[13]);
 
 	if (speed)
@@ -1089,7 +1091,7 @@ static int crosslink_enum_partition(struct switchtec_ntb *sndev,
 {
 	struct part_cfg_regs __iomem *part_cfg =
 		&sndev->stdev->mmio_part_cfg_all[sndev->peer_partition];
-	u32 pff = ioread32(&part_cfg->vep_pff_inst_id);
+	u32 pff = ioread32(&part_cfg->vep_pff_inst_id) & 0xFF;
 	struct pff_csr_regs __iomem *mmio_pff =
 		&sndev->stdev->mmio_pff_csr[pff];
 	const u64 bar_space = 0x1000000000LL;
-- 
2.25.1

[PATCH 5/6] ntb_hw_switchtec: Remove code for disabling ID protection

[Kelvin Cao]

ID protection is a firmware setting for NT window access control. With
it enabled, only the posted requests with requester IDs in the requester
ID table will be allowed to access the NT windows. Otherwise all posted
requests are allowed. Normally user will configure it statically via the
Switchtec config file, and it will take effect when the firmware boots
up. The driver can also toggle the ID protection setting dynamically,
which will overwrite the static setting in the Switchtec config file as
a side effect.

Currently, the driver disables the ID protection. However, it's not
necessary to disable the ID protection at the driver level as the driver
has already configured the proper requester IDs in the requester ID
table to allow the corresponding posted requests to hit the NT windows.
Remove the code that disables the ID protection to make the static
setting prevail.

Note: ID protection is not applicable to non-posted requests.

Signed-off-by: Kelvin Cao <kelvin.cao@microchip.com>
---
 drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 3 ---
 include/linux/switchtec.h              | 2 --
 2 files changed, 5 deletions(-)

diff --git a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
index 03839346233d..0e33eef64ec6 100644
--- a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
+++ b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
@@ -967,9 +967,6 @@ static int config_req_id_table(struct switchtec_ntb *sndev,
 	if (rc)
 		return rc;
 
-	iowrite32(NTB_PART_CTRL_ID_PROT_DIS,
-		  &mmio_ctrl->partition_ctrl);
-
 	for (i = 0; i < count; i++) {
 		iowrite32(req_ids[i] << 16 | NTB_CTRL_REQ_ID_EN,
 			  &mmio_ctrl->req_id_table[i]);
diff --git a/include/linux/switchtec.h b/include/linux/switchtec.h
index be24056ac00f..48fabe36509e 100644
--- a/include/linux/switchtec.h
+++ b/include/linux/switchtec.h
@@ -337,8 +337,6 @@ enum {
 	NTB_CTRL_REQ_ID_EN = 1 << 0,
 
 	NTB_CTRL_LUT_EN = 1 << 0,
-
-	NTB_PART_CTRL_ID_PROT_DIS = 1 << 0,
 };
 
 struct ntb_ctrl_regs {
-- 
2.25.1

[PATCH 6/6] ntb_hw_switchtec: Fix a minor issue in config_req_id_table()

[Kelvin Cao]

The req_id_table_size field is 16-bit wide, use ioread16() to read the
value.

Signed-off-by: Kelvin Cao <kelvin.cao@microchip.com>
---
 drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
index 0e33eef64ec6..189faad0d0d5 100644
--- a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
+++ b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
@@ -955,7 +955,7 @@ static int config_req_id_table(struct switchtec_ntb *sndev,
 	u32 error;
 	u32 proxy_id;
 
-	if (ioread32(&mmio_ctrl->req_id_table_size) < count) {
+	if (ioread16(&mmio_ctrl->req_id_table_size) < count) {
 		dev_err(&sndev->stdev->dev,
 			"Not enough requester IDs available.\n");
 		return -EFAULT;
-- 
2.25.1

Re: [PATCH 0/6] Switchtec NTB Fixes and Improvements

[Logan Gunthorpe]

On 2021-12-23 6:23 p.m., Kelvin Cao wrote:
> Hi,
> 
> Please find a bunch of patches for the Switchtec NTB driver.
> 
> Patche 1, 2 and 6 fix three minor bugs. Patch 3 works around a minor
> firmware issue. Patch 4 updates the method of getting management VEP
> instance ID based on a new firmware change. Patch 5 removes code that
> disables ID protection to avoid conflict with static Switchtec config
> settings.
> 
> This patchset is based on 5.16.0-rc5.
> 
> Thanks,
> Kelvin
> 
> Jeremy Pallotta (2):
>   ntb_hw_switchtec: Fix pff ioread to read into mmio_part_cfg_all
>   ntb_hw_switchtec: AND with the part_map for a valid tpart_vec
> 
> Kelvin Cao (3):
>   ntb_hw_switchtec: Update the way of getting VEP instance ID
>   ntb_hw_switchtec: Remove code for disabling ID protection
>   ntb_hw_switchtec: Fix a minor issue in config_req_id_table()
> 
> Wesley Sheng (1):
>   ntb_hw_switchtec: Fix bug with more than 32 partitions

I've reviewed all these patches and they look good to me.

Reviewed-by: Logan Gunthorpe <logang@deltatee.com>

Thanks,

Logan

Re: [PATCH 0/6] Switchtec NTB Fixes and Improvements

[Kelvin.Cao]

On Tue, 2022-01-04 at 09:55 -0700, Logan Gunthorpe wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you
> know the content is safe
> 
> On 2021-12-23 6:23 p.m., Kelvin Cao wrote:
> > Hi,
> > 
> > Please find a bunch of patches for the Switchtec NTB driver.
> > 
> > Patche 1, 2 and 6 fix three minor bugs. Patch 3 works around a
> > minor
> > firmware issue. Patch 4 updates the method of getting management
> > VEP
> > instance ID based on a new firmware change. Patch 5 removes code
> > that
> > disables ID protection to avoid conflict with static Switchtec
> > config
> > settings.
> > 
> > This patchset is based on 5.16.0-rc5.
> > 
> > Thanks,
> > Kelvin
> > 
> > Jeremy Pallotta (2):
> >   ntb_hw_switchtec: Fix pff ioread to read into mmio_part_cfg_all
> >   ntb_hw_switchtec: AND with the part_map for a valid tpart_vec
> > 
> > Kelvin Cao (3):
> >   ntb_hw_switchtec: Update the way of getting VEP instance ID
> >   ntb_hw_switchtec: Remove code for disabling ID protection
> >   ntb_hw_switchtec: Fix a minor issue in config_req_id_table()
> > 
> > Wesley Sheng (1):
> >   ntb_hw_switchtec: Fix bug with more than 32 partitions
> 
> I've reviewed all these patches and they look good to me.
> 
> Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
> 
Thanks Logan!

Kelvin

> Thanks,
> 
> Logan