[PATCH 00/14] bna: Update bna driver version to 3.0.2.1

[PATCH 00/14] bna: Update bna driver version to 3.0.2.1

[Rasesh Mody]

Hello David,

   The following patch set contains changes for adding support for feature like
   Multi TXQ, TX and RX path changes and Ethtool enhancements. This also fixes
   bugs found with new code. It cleans up unused code, formatting changes and
   comments addition/deletion.

   This updates the Brocade BNA driver to v3.0.2.1.

   The driver has been compiled & tested against net-next-2.6(3.0.0-rc7)

Thanks,
Rasesh

Rasesh Mody (14):
  bna: Naming Change and Minor Macro Fix
  bna: PCI Probe Fix
  bna: Interrupt Polling and NAPI Init Changes
  bna: Add Multiple Tx Queue Support
  bna: SKB Check and Drop Macros
  bna: TX Path and RX Path Changes
  bna: Formatting and Code Cleanup
  bna: Initialization and Locking Fix
  bna: Ethtool Enhancements and Fix
  bna: Async Mode Tx Rx Init Fix
  bna: MBOX IRQ Flag Check after Locking
  bna: Queue Depth and SKB Unmap Array Fix
  bna: SKB PCI UNMAP Fix
  bna: Driver Version changed to 3.0.2.1

 drivers/net/ethernet/brocade/bna/bfa_cee.c         |    2 -
 drivers/net/ethernet/brocade/bna/bfa_defs.h        |    8 +-
 .../net/ethernet/brocade/bna/bfa_defs_mfg_comm.h   |    1 -
 drivers/net/ethernet/brocade/bna/bfa_ioc.h         |    6 +-
 drivers/net/ethernet/brocade/bna/bfi.h             |   46 --
 drivers/net/ethernet/brocade/bna/bna.h             |   26 +-
 drivers/net/ethernet/brocade/bna/bna_enet.c        |   29 +-
 drivers/net/ethernet/brocade/bna/bna_hw_defs.h     |    5 +
 drivers/net/ethernet/brocade/bna/bna_types.h       |    2 +-
 drivers/net/ethernet/brocade/bna/bnad.c            |  628 ++++++++++++--------
 drivers/net/ethernet/brocade/bna/bnad.h            |   98 +++-
 drivers/net/ethernet/brocade/bna/bnad_ethtool.c    |   96 +++-
 drivers/net/ethernet/brocade/bna/cna.h             |   11 +-
 13 files changed, 576 insertions(+), 382 deletions(-)

[PATCH 01/14] bna: Naming Change and Minor Macro Fix

[Rasesh Mody]

Naming changes: rename devid, BNAD_MAX_TXS, BNAD_MAX_RXS,
BNAD_MAX_RXPS_PER_RX to device, BNAD_MAX_TX, BNAD_MAX_RX,
BNAD_MAX_RXP_PER_RX respectively and change all the references.

Macro Fix: Add ioc_isr_mod_set check to bfa_nw_ioc_mbox_regisr macro

Signed-off-by: Gurunatha Karaje <gkaraje@brocade.com>
Signed-off-by: Rasesh Mody <rmody@brocade.com>
---
 drivers/net/ethernet/brocade/bna/bfa_defs.h |    8 ++++----
 drivers/net/ethernet/brocade/bna/bfa_ioc.h  |    6 ++++--
 drivers/net/ethernet/brocade/bna/bnad.c     |    6 +++---
 drivers/net/ethernet/brocade/bna/bnad.h     |   20 ++++++++++----------
 4 files changed, 21 insertions(+), 19 deletions(-)

diff --git a/drivers/net/ethernet/brocade/bna/bfa_defs.h b/drivers/net/ethernet/brocade/bna/bfa_defs.h
index 205b92b..a81c0cc 100644
--- a/drivers/net/ethernet/brocade/bna/bfa_defs.h
+++ b/drivers/net/ethernet/brocade/bna/bfa_defs.h
@@ -251,10 +251,10 @@ struct bfa_mfg_block {
  * ---------------------- pci definitions ------------
  */
 
-#define bfa_asic_id_ct(devid)			\
-	((devid) == PCI_DEVICE_ID_BROCADE_CT ||	\
-	(devid) == PCI_DEVICE_ID_BROCADE_CT_FC)
-#define bfa_asic_id_ctc(devid) (bfa_asic_id_ct(devid))
+#define bfa_asic_id_ct(device)			\
+	((device) == PCI_DEVICE_ID_BROCADE_CT ||	\
+	 (device) == PCI_DEVICE_ID_BROCADE_CT_FC)
+#define bfa_asic_id_ctc(device) (bfa_asic_id_ct(device))
 
 enum bfa_mode {
 	BFA_MODE_HBA		= 1,
diff --git a/drivers/net/ethernet/brocade/bna/bfa_ioc.h b/drivers/net/ethernet/brocade/bna/bfa_ioc.h
index f5a3d4e..9116324 100644
--- a/drivers/net/ethernet/brocade/bna/bfa_ioc.h
+++ b/drivers/net/ethernet/brocade/bna/bfa_ioc.h
@@ -274,8 +274,10 @@ void bfa_nw_ioc_mbox_regisr(struct bfa_ioc *ioc, enum bfi_mclass mc,
 	((__ioc)->ioc_hwif->ioc_pll_init((__ioc)->pcidev.pci_bar_kva, \
 			   (__ioc)->asic_mode))
 
-#define	bfa_ioc_isr_mode_set(__ioc, __msix)			\
-			((__ioc)->ioc_hwif->ioc_isr_mode_set(__ioc, __msix))
+#define	bfa_ioc_isr_mode_set(__ioc, __msix) do {			\
+	if ((__ioc)->ioc_hwif->ioc_isr_mode_set)			\
+		((__ioc)->ioc_hwif->ioc_isr_mode_set(__ioc, __msix));	\
+} while (0)
 #define	bfa_ioc_ownership_reset(__ioc)				\
 			((__ioc)->ioc_hwif->ioc_ownership_reset(__ioc))
 
diff --git a/drivers/net/ethernet/brocade/bna/bnad.c b/drivers/net/ethernet/brocade/bna/bnad.c
index 5ad07ea..bc5bfa7 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.c
+++ b/drivers/net/ethernet/brocade/bna/bnad.c
@@ -1001,7 +1001,7 @@ bnad_cb_rx_cleanup(struct bnad *bnad, struct bna_rx *rx)
 
 	mdelay(BNAD_TXRX_SYNC_MDELAY);
 
-	for (i = 0; i < BNAD_MAX_RXPS_PER_RX; i++) {
+	for (i = 0; i < BNAD_MAX_RXP_PER_RX; i++) {
 		rx_ctrl = &rx_info->rx_ctrl[i];
 		ccb = rx_ctrl->ccb;
 		if (!ccb)
@@ -1030,7 +1030,7 @@ bnad_cb_rx_post(struct bnad *bnad, struct bna_rx *rx)
 	int i;
 	int j;
 
-	for (i = 0; i < BNAD_MAX_RXPS_PER_RX; i++) {
+	for (i = 0; i < BNAD_MAX_RXP_PER_RX; i++) {
 		rx_ctrl = &rx_info->rx_ctrl[i];
 		ccb = rx_ctrl->ccb;
 		if (!ccb)
@@ -2227,7 +2227,7 @@ bnad_q_num_init(struct bnad *bnad)
 	int rxps;
 
 	rxps = min((uint)num_online_cpus(),
-			(uint)(BNAD_MAX_RXS * BNAD_MAX_RXPS_PER_RX));
+			(uint)(BNAD_MAX_RX * BNAD_MAX_RXP_PER_RX));
 
 	if (!(bnad->cfg_flags & BNAD_CF_MSIX))
 		rxps = 1;	/* INTx */
diff --git a/drivers/net/ethernet/brocade/bna/bnad.h b/drivers/net/ethernet/brocade/bna/bnad.h
index 5b5451e..3c23139 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.h
+++ b/drivers/net/ethernet/brocade/bna/bnad.h
@@ -38,12 +38,12 @@
 #define BNAD_TXQ_DEPTH		2048
 #define BNAD_RXQ_DEPTH		2048
 
-#define BNAD_MAX_TXS		1
+#define BNAD_MAX_TX		1
 #define BNAD_MAX_TXQ_PER_TX	8	/* 8 priority queues */
 #define BNAD_TXQ_NUM		1
 
-#define BNAD_MAX_RXS		1
-#define BNAD_MAX_RXPS_PER_RX	16
+#define BNAD_MAX_RX		1
+#define BNAD_MAX_RXP_PER_RX	16
 #define BNAD_MAX_RXQ_PER_RXP	2
 
 /*
@@ -190,7 +190,7 @@ struct bnad_tx_info {
 struct bnad_rx_info {
 	struct bna_rx *rx; /* 1:1 between rx_info & rx */
 
-	struct bnad_rx_ctrl rx_ctrl[BNAD_MAX_RXPS_PER_RX];
+	struct bnad_rx_ctrl rx_ctrl[BNAD_MAX_RXP_PER_RX];
 	u32 rx_id;
 } ____cacheline_aligned;
 
@@ -234,8 +234,8 @@ struct bnad {
 	struct net_device	*netdev;
 
 	/* Data path */
-	struct bnad_tx_info tx_info[BNAD_MAX_TXS];
-	struct bnad_rx_info rx_info[BNAD_MAX_RXS];
+	struct bnad_tx_info tx_info[BNAD_MAX_TX];
+	struct bnad_rx_info rx_info[BNAD_MAX_RX];
 
 	unsigned long active_vlans[BITS_TO_LONGS(VLAN_N_VID)];
 	/*
@@ -255,8 +255,8 @@ struct bnad {
 	u8			tx_coalescing_timeo;
 	u8			rx_coalescing_timeo;
 
-	struct bna_rx_config rx_config[BNAD_MAX_RXS];
-	struct bna_tx_config tx_config[BNAD_MAX_TXS];
+	struct bna_rx_config rx_config[BNAD_MAX_RX];
+	struct bna_tx_config tx_config[BNAD_MAX_TX];
 
 	void __iomem		*bar0;	/* BAR0 address */
 
@@ -283,8 +283,8 @@ struct bnad {
 	/* Control path resources, memory & irq */
 	struct bna_res_info res_info[BNA_RES_T_MAX];
 	struct bna_res_info mod_res_info[BNA_MOD_RES_T_MAX];
-	struct bnad_tx_res_info tx_res_info[BNAD_MAX_TXS];
-	struct bnad_rx_res_info rx_res_info[BNAD_MAX_RXS];
+	struct bnad_tx_res_info tx_res_info[BNAD_MAX_TX];
+	struct bnad_rx_res_info rx_res_info[BNAD_MAX_RX];
 
 	struct bnad_completion bnad_completions;
 
-- 
1.7.1

[PATCH 02/14] bna: PCI Probe Fix

[Rasesh Mody]

Change details:
 - Return error as -EIO if bnad_res_alloc fails
 - Release the configuration lock before registering with net_device layer.

Signed-off-by: Gurunatha Karaje <gkaraje@brocade.com>
Signed-off-by: Rasesh Mody <rmody@brocade.com>
---
 drivers/net/ethernet/brocade/bna/bnad.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/drivers/net/ethernet/brocade/bna/bnad.c b/drivers/net/ethernet/brocade/bna/bnad.c
index bc5bfa7..8db7247 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.c
+++ b/drivers/net/ethernet/brocade/bna/bnad.c
@@ -3254,8 +3254,10 @@ bnad_pci_probe(struct pci_dev *pdev,
 	spin_unlock_irqrestore(&bnad->bna_lock, flags);
 
 	err = bnad_res_alloc(bnad, &bnad->mod_res_info[0], BNA_MOD_RES_T_MAX);
-	if (err)
+	if (err) {
+		err = -EIO;
 		goto disable_ioceth;
+	}
 
 	spin_lock_irqsave(&bnad->bna_lock, flags);
 	bna_mod_init(&bnad->bna, &bnad->mod_res_info[0]);
@@ -3267,6 +3269,8 @@ bnad_pci_probe(struct pci_dev *pdev,
 	bnad_set_netdev_perm_addr(bnad);
 	spin_unlock_irqrestore(&bnad->bna_lock, flags);
 
+	mutex_unlock(&bnad->conf_mutex);
+
 	/* Finally, reguister with net_device layer */
 	err = register_netdev(netdev);
 	if (err) {
@@ -3275,6 +3279,8 @@ bnad_pci_probe(struct pci_dev *pdev,
 	}
 	set_bit(BNAD_RF_NETDEV_REGISTERED, &bnad->run_flags);
 
+	return 0;
+
 probe_success:
 	mutex_unlock(&bnad->conf_mutex);
 	return 0;
-- 
1.7.1

[PATCH 03/14] bna: Interrupt Polling and NAPI Init Changes

[Rasesh Mody]

Change details:
 - Remove unnecessary ccb check from bnad_poll_cq
 - Add bnad pointer to rx_ctrl structure, so that bnad can be accessed directly
   from rx_ctrl in the NAPI poll routines, even if ccb is NULL
 - Validate ccb before referencing to it in bnad_msix_rx and bnad_napi_poll_rx
 - Fix the order of NAPI init / uninit in Tx / Rx setup / teardown path:
   a. Kill bnad tx free tasklet ahead of call to bna_tx_destroy()
   b. Call NAPI disable only after call to Rx free_irq(). This makes sure Rx
      interrupt does not schedule a poll when NAPI is already disabled
 - NAPI poll runs before the h/w has completed configuration. This causes a
   crash. Delay enabling NAPI till after bna_rx_enable(). Split NAPI
   initialization into 2 steps, bnad_napi_init() & bnad_napi_enable().

Signed-off-by: Gurunatha Karaje <gkaraje@brocade.com>
Signed-off-by: Rasesh Mody <rmody@brocade.com>
---
 drivers/net/ethernet/brocade/bna/bnad.c |   83 ++++++++++++++++++++-----------
 drivers/net/ethernet/brocade/bna/bnad.h |    1 +
 2 files changed, 54 insertions(+), 30 deletions(-)

diff --git a/drivers/net/ethernet/brocade/bna/bnad.c b/drivers/net/ethernet/brocade/bna/bnad.c
index 8db7247..327b274 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.c
+++ b/drivers/net/ethernet/brocade/bna/bnad.c
@@ -535,16 +535,11 @@ next:
 
 	BNA_QE_INDX_ADD(ccb->producer_index, wis, ccb->q_depth);
 
-	if (likely(ccb)) {
-		if (likely(test_bit(BNAD_RXQ_STARTED, &ccb->rcb[0]->flags)))
-			bna_ib_ack(ccb->i_dbell, packets);
-		bnad_refill_rxq(bnad, ccb->rcb[0]);
-		if (ccb->rcb[1])
-			bnad_refill_rxq(bnad, ccb->rcb[1]);
-	} else {
-		if (likely(test_bit(BNAD_RXQ_STARTED, &ccb->rcb[0]->flags)))
-			bna_ib_ack(ccb->i_dbell, 0);
-	}
+	if (likely(test_bit(BNAD_RXQ_STARTED, &ccb->rcb[0]->flags)))
+		bna_ib_ack(ccb->i_dbell, packets);
+	bnad_refill_rxq(bnad, ccb->rcb[0]);
+	if (ccb->rcb[1])
+		bnad_refill_rxq(bnad, ccb->rcb[1]);
 
 	clear_bit(BNAD_FP_IN_RX_PATH, &rx_ctrl->flags);
 
@@ -590,9 +585,9 @@ static irqreturn_t
 bnad_msix_rx(int irq, void *data)
 {
 	struct bna_ccb *ccb = (struct bna_ccb *)data;
-	struct bnad *bnad = ccb->bnad;
 
-	bnad_netif_rx_schedule_poll(bnad, ccb);
+	if (ccb)
+		bnad_netif_rx_schedule_poll(ccb->bnad, ccb);
 
 	return IRQ_HANDLED;
 }
@@ -1658,18 +1653,14 @@ bnad_napi_poll_rx(struct napi_struct *napi, int budget)
 {
 	struct bnad_rx_ctrl *rx_ctrl =
 		container_of(napi, struct bnad_rx_ctrl, napi);
-	struct bna_ccb *ccb;
-	struct bnad *bnad;
+	struct bnad *bnad = rx_ctrl->bnad;
 	int rcvd = 0;
 
-	ccb = rx_ctrl->ccb;
-
-	bnad = ccb->bnad;
 
 	if (!netif_carrier_ok(bnad->netdev))
 		goto poll_exit;
 
-	rcvd = bnad_poll_cq(bnad, ccb, budget);
+	rcvd = bnad_poll_cq(bnad, rx_ctrl->ccb, budget);
 	if (rcvd == budget)
 		return rcvd;
 
@@ -1678,12 +1669,15 @@ poll_exit:
 
 	BNAD_UPDATE_CTR(bnad, netif_rx_complete);
 
-	bnad_enable_rx_irq(bnad, ccb);
+
+	if (rx_ctrl->ccb)
+		bnad_enable_rx_irq(bnad, rx_ctrl->ccb);
 	return rcvd;
 }
 
+#define BNAD_NAPI_POLL_QUOTA		64
 static void
-bnad_napi_enable(struct bnad *bnad, u32 rx_id)
+bnad_napi_init(struct bnad *bnad, u32 rx_id)
 {
 	struct bnad_rx_ctrl *rx_ctrl;
 	int i;
@@ -1691,9 +1685,20 @@ bnad_napi_enable(struct bnad *bnad, u32 rx_id)
 	/* Initialize & enable NAPI */
 	for (i = 0; i <	bnad->num_rxp_per_rx; i++) {
 		rx_ctrl = &bnad->rx_info[rx_id].rx_ctrl[i];
-
 		netif_napi_add(bnad->netdev, &rx_ctrl->napi,
-			       bnad_napi_poll_rx, 64);
+			       bnad_napi_poll_rx, BNAD_NAPI_POLL_QUOTA);
+	}
+}
+
+static void
+bnad_napi_enable(struct bnad *bnad, u32 rx_id)
+{
+	struct bnad_rx_ctrl *rx_ctrl;
+	int i;
+
+	/* Initialize & enable NAPI */
+	for (i = 0; i <	bnad->num_rxp_per_rx; i++) {
+		rx_ctrl = &bnad->rx_info[rx_id].rx_ctrl[i];
 
 		napi_enable(&rx_ctrl->napi);
 	}
@@ -1732,6 +1737,9 @@ bnad_cleanup_tx(struct bnad *bnad, u32 tx_id)
 		bnad_tx_msix_unregister(bnad, tx_info,
 			bnad->num_txq_per_tx);
 
+	if (0 == tx_id)
+		tasklet_kill(&bnad->tx_free_tasklet);
+
 	spin_lock_irqsave(&bnad->bna_lock, flags);
 	bna_tx_destroy(tx_info->tx);
 	spin_unlock_irqrestore(&bnad->bna_lock, flags);
@@ -1739,9 +1747,6 @@ bnad_cleanup_tx(struct bnad *bnad, u32 tx_id)
 	tx_info->tx = NULL;
 	tx_info->tx_id = 0;
 
-	if (0 == tx_id)
-		tasklet_kill(&bnad->tx_free_tasklet);
-
 	bnad_tx_res_free(bnad, res_info);
 }
 
@@ -1852,6 +1857,16 @@ bnad_init_rx_config(struct bnad *bnad, struct bna_rx_config *rx_config)
 	rx_config->vlan_strip_status = BNA_STATUS_T_ENABLED;
 }
 
+static void
+bnad_rx_ctrl_init(struct bnad *bnad, u32 rx_id)
+{
+	struct bnad_rx_info *rx_info = &bnad->rx_info[rx_id];
+	int i;
+
+	for (i = 0; i < bnad->num_rxp_per_rx; i++)
+		rx_info->rx_ctrl[i].bnad = bnad;
+}
+
 /* Called with mutex_lock(&bnad->conf_mutex) held */
 void
 bnad_cleanup_rx(struct bnad *bnad, u32 rx_id)
@@ -1875,8 +1890,6 @@ bnad_cleanup_rx(struct bnad *bnad, u32 rx_id)
 			del_timer_sync(&bnad->dim_timer);
 	}
 
-	bnad_napi_disable(bnad, rx_id);
-
 	init_completion(&bnad->bnad_completions.rx_comp);
 	spin_lock_irqsave(&bnad->bna_lock, flags);
 	bna_rx_disable(rx_info->rx, BNA_HARD_CLEANUP, bnad_cb_rx_disabled);
@@ -1886,6 +1899,8 @@ bnad_cleanup_rx(struct bnad *bnad, u32 rx_id)
 	if (rx_info->rx_ctrl[0].ccb->intr_type == BNA_INTR_T_MSIX)
 		bnad_rx_msix_unregister(bnad, rx_info, rx_config->num_paths);
 
+	bnad_napi_disable(bnad, rx_id);
+
 	spin_lock_irqsave(&bnad->bna_lock, flags);
 	bna_rx_destroy(rx_info->rx);
 	spin_unlock_irqrestore(&bnad->bna_lock, flags);
@@ -1939,6 +1954,8 @@ bnad_setup_rx(struct bnad *bnad, u32 rx_id)
 	if (err)
 		return err;
 
+	bnad_rx_ctrl_init(bnad, rx_id);
+
 	/* Ask BNA to create one Rx object, supplying required resources */
 	spin_lock_irqsave(&bnad->bna_lock, flags);
 	rx = bna_rx_create(&bnad->bna, bnad, rx_config, &rx_cbfn, res_info,
@@ -1948,6 +1965,12 @@ bnad_setup_rx(struct bnad *bnad, u32 rx_id)
 		goto err_return;
 	rx_info->rx = rx;
 
+	/*
+	 * Init NAPI, so that state is set to NAPI_STATE_SCHED,
+	 * so that IRQ handler cannot schedule NAPI at this point.
+	 */
+	bnad_napi_init(bnad, rx_id);
+
 	/* Register ISR for the Rx object */
 	if (intr_info->intr_type == BNA_INTR_T_MSIX) {
 		err = bnad_rx_msix_register(bnad, rx_info, rx_id,
@@ -1956,9 +1979,6 @@ bnad_setup_rx(struct bnad *bnad, u32 rx_id)
 			goto err_return;
 	}
 
-	/* Enable NAPI */
-	bnad_napi_enable(bnad, rx_id);
-
 	spin_lock_irqsave(&bnad->bna_lock, flags);
 	if (0 == rx_id) {
 		/* Set up Dynamic Interrupt Moderation Vector */
@@ -1975,6 +1995,9 @@ bnad_setup_rx(struct bnad *bnad, u32 rx_id)
 	bna_rx_enable(rx);
 	spin_unlock_irqrestore(&bnad->bna_lock, flags);
 
+	/* Enable scheduling of NAPI */
+	bnad_napi_enable(bnad, rx_id);
+
 	return 0;
 
 err_return:
diff --git a/drivers/net/ethernet/brocade/bna/bnad.h b/drivers/net/ethernet/brocade/bna/bnad.h
index 3c23139..60c2e9d 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.h
+++ b/drivers/net/ethernet/brocade/bna/bnad.h
@@ -53,6 +53,7 @@
  */
 struct bnad_rx_ctrl {
 	struct bna_ccb *ccb;
+	struct bnad *bnad;
 	unsigned long  flags;
 	struct napi_struct	napi;
 };
-- 
1.7.1

[PATCH 04/14] bna: Add Multiple Tx Queue Support

[Rasesh Mody]

Change details:
 - Add macros bna_prio_allow, bna_default_nw_prio, bna_iscsi_prio,
   bna_is_iscsi_over_cee
 - Added support for multipe Tx queues with a separate iSCSI Tx queue based
   on the default value of iSCSI port number. The feature is supported based
   on the underlying hardware and enabled for DCB (CEE) mode only.
 - Allocate multiple TxQ resource in netdev
 - Implement bnad_tx_select_queue() which enables the correct selection of
   TxQ Id (and tcb). This function is called either by the kernel to channel
   packets to the right TxQ
 - bnad_tx_select_queue() returns priority, while only a few packets during
   transition could have wrong priority, all will be associated with a valid
   non-NULL tcb.
 - Implement bnad_iscsi_tcb_get() and BNAD_IS_ISCSI_PKT() for iSCSI packet
   inspection and retrieval of tcb corresponding to the iSCSI priority.
 - Construction of priority indirection table to be used by bnad to direct
   packets into TxQs

Signed-off-by: Gurunatha Karaje <gkaraje@brocade.com>
Signed-off-by: Rasesh Mody <rmody@brocade.com>
---
 drivers/net/ethernet/brocade/bna/bna.h  |    8 ++
 drivers/net/ethernet/brocade/bna/bnad.c |  131 ++++++++++++++++++++++++-------
 drivers/net/ethernet/brocade/bna/bnad.h |   13 +++-
 3 files changed, 121 insertions(+), 31 deletions(-)

diff --git a/drivers/net/ethernet/brocade/bna/bna.h b/drivers/net/ethernet/brocade/bna/bna.h
index 2a587c5..e4f914c 100644
--- a/drivers/net/ethernet/brocade/bna/bna.h
+++ b/drivers/net/ethernet/brocade/bna/bna.h
@@ -351,6 +351,14 @@ do {									\
 	}								\
 } while (0)
 
+#define bna_prio_allowed(_bna, _prio) ((_bna)->tx_mod.prio_map & (1 << _prio))
+
+#define bna_iscsi_prio(_bna) ((_bna)->tx_mod.iscsi_prio)
+
+#define bna_default_prio(_bna) ((_bna)->tx_mod.default_prio)
+
+#define bna_is_iscsi_over_cee(_bna) ((_bna)->tx_mod.iscsi_over_cee)
+
 /**
  *
  *  Inline functions
diff --git a/drivers/net/ethernet/brocade/bna/bnad.c b/drivers/net/ethernet/brocade/bna/bnad.c
index 327b274..895f18b 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.c
+++ b/drivers/net/ethernet/brocade/bna/bnad.c
@@ -246,7 +246,7 @@ bnad_tx_free_tasklet(unsigned long bnad_ptr)
 {
 	struct bnad *bnad = (struct bnad *)bnad_ptr;
 	struct bna_tcb *tcb;
-	u32		acked = 0;
+	u32		acked = 0, txq_id;
 	int			i, j;
 
 	for (i = 0; i < bnad->num_tx; i++) {
@@ -265,14 +265,20 @@ bnad_tx_free_tasklet(unsigned long bnad_ptr)
 				smp_mb__before_clear_bit();
 				clear_bit(BNAD_TXQ_FREE_SENT, &tcb->flags);
 			}
+			/*
+			 * Check again, because this bit can be set from another
+			 * context. This is not lock protected.
+			 */
 			if (unlikely(!test_bit(BNAD_TXQ_TX_STARTED,
 						&tcb->flags)))
 				continue;
-			if (netif_queue_stopped(bnad->netdev)) {
+			txq_id = tcb->id;
+			if (__netif_subqueue_stopped(bnad->netdev, txq_id)) {
 				if (acked && netif_carrier_ok(bnad->netdev) &&
 					BNA_QE_FREE_CNT(tcb, tcb->q_depth) >=
 						BNAD_NETIF_WAKE_THRESHOLD) {
-					netif_wake_queue(bnad->netdev);
+					netif_wake_subqueue(bnad->netdev,
+									txq_id);
 					/* TODO */
 					/* Counters for individual TxQs? */
 					BNAD_UPDATE_CTR(bnad,
@@ -287,19 +293,21 @@ static u32
 bnad_tx(struct bnad *bnad, struct bna_tcb *tcb)
 {
 	struct net_device *netdev = bnad->netdev;
-	u32 sent = 0;
+	u32 sent = 0, txq_id;
 
 	if (test_and_set_bit(BNAD_TXQ_FREE_SENT, &tcb->flags))
 		return 0;
 
 	sent = bnad_free_txbufs(bnad, tcb);
 	if (sent) {
-		if (netif_queue_stopped(netdev) &&
+		txq_id = tcb->id;
+
+		if (__netif_subqueue_stopped(netdev, txq_id) &&
 		    netif_carrier_ok(netdev) &&
 		    BNA_QE_FREE_CNT(tcb, tcb->q_depth) >=
 				    BNAD_NETIF_WAKE_THRESHOLD) {
 			if (test_bit(BNAD_TXQ_TX_STARTED, &tcb->flags)) {
-				netif_wake_queue(netdev);
+				netif_wake_subqueue(netdev, txq_id);
 				BNAD_UPDATE_CTR(bnad, netif_queue_wakeup);
 			}
 		}
@@ -2247,38 +2255,45 @@ bnad_tso_prepare(struct bnad *bnad, struct sk_buff *skb)
 static void
 bnad_q_num_init(struct bnad *bnad)
 {
-	int rxps;
+	int rxps = min((u32)num_online_cpus(),
+			(u32)(BNAD_MAX_RXP_PER_RX));
 
-	rxps = min((uint)num_online_cpus(),
-			(uint)(BNAD_MAX_RX * BNAD_MAX_RXP_PER_RX));
+	BNA_TO_POWER_OF_2(rxps);
 
 	if (!(bnad->cfg_flags & BNAD_CF_MSIX))
 		rxps = 1;	/* INTx */
 
-	bnad->num_rx = 1;
-	bnad->num_tx = 1;
+	bnad->num_rx = BNAD_MAX_RX;
+	bnad->num_tx = BNAD_MAX_TX;
 	bnad->num_rxp_per_rx = rxps;
-	bnad->num_txq_per_tx = BNAD_TXQ_NUM;
+	bnad->num_txq_per_tx = BNAD_MAX_TXQ_PER_TX;
 }
 
 /*
- * Adjusts the Q numbers, given a number of msix vectors
+ * Adjusts the Q numbers, given a number of max possible queues.
  * Give preference to RSS as opposed to Tx priority Queues,
  * in such a case, just use 1 Tx Q
  * Called with bnad->bna_lock held b'cos of cfg_flags access
  */
 static void
-bnad_q_num_adjust(struct bnad *bnad, int msix_vectors, int temp)
+bnad_q_num_adjust(struct bnad *bnad, int max_txq, int max_rxq)
 {
-	bnad->num_txq_per_tx = 1;
-	if ((msix_vectors >= (bnad->num_tx * bnad->num_txq_per_tx)  +
-	     bnad_rxqs_per_cq + BNAD_MAILBOX_MSIX_VECTORS) &&
-	    (bnad->cfg_flags & BNAD_CF_MSIX)) {
-		bnad->num_rxp_per_rx = msix_vectors -
-			(bnad->num_tx * bnad->num_txq_per_tx) -
-			BNAD_MAILBOX_MSIX_VECTORS;
-	} else
-		bnad->num_rxp_per_rx = 1;
+	if (!(bnad->cfg_flags & BNAD_CF_MSIX)) {
+		bnad->num_tx = bnad->num_txq_per_tx = 1;
+		bnad->num_rx = bnad->num_rxp_per_rx = 1;
+		return;
+	}
+
+	if (max_txq < BNAD_NUM_TXQ) {
+		bnad->num_txq_per_tx = 1;
+		bnad->num_tx = 1;
+	}
+
+	bnad->num_rx = 1;
+	bnad->num_rxp_per_rx = min((u32)(min((u32)num_online_cpus(),
+					(u32)(BNAD_MAX_RXP_PER_RX))),
+					(u32)max_rxq);
+	BNA_TO_POWER_OF_2(bnad->num_rxp_per_rx);
 }
 
 /* Enable / disable ioceth */
@@ -2537,15 +2552,15 @@ static netdev_tx_t
 bnad_start_xmit(struct sk_buff *skb, struct net_device *netdev)
 {
 	struct bnad *bnad = netdev_priv(netdev);
-	u32 txq_id = 0;
-	struct bna_tcb *tcb = bnad->tx_info[0].tcb[txq_id];
+	struct bna_tcb *tcb = NULL;
+	u32 txq_id;
+	struct bnad_unmap_q *unmap_q;
 
 	u16		txq_prod, vlan_tag = 0;
 	u32		unmap_prod, wis, wis_used, wi_range;
 	u32		vectors, vect_id, i, acked;
 	int			err;
 
-	struct bnad_unmap_q *unmap_q = tcb->unmap_q;
 	dma_addr_t		dma_addr;
 	struct bna_txq_entry *txqent;
 	u16	flags;
@@ -2556,6 +2571,16 @@ bnad_start_xmit(struct sk_buff *skb, struct net_device *netdev)
 		return NETDEV_TX_OK;
 	}
 
+	txq_id = skb_get_queue_mapping(skb);
+
+	tcb = bnad->tx_info[0].tcb[txq_id];
+
+	if (unlikely(!tcb)) {
+		dev_kfree_skb(skb);
+		return NETDEV_TX_OK;
+	}
+
+	unmap_q = tcb->unmap_q;
 	/*
 	 * Takes care of the Tx that is scheduled between clearing the flag
 	 * and the netif_stop_all_queue() call.
@@ -2583,7 +2608,7 @@ bnad_start_xmit(struct sk_buff *skb, struct net_device *netdev)
 			smp_mb__before_clear_bit();
 			clear_bit(BNAD_TXQ_FREE_SENT, &tcb->flags);
 		} else {
-			netif_stop_queue(netdev);
+			netif_stop_subqueue(netdev, txq_id);
 			BNAD_UPDATE_CTR(bnad, netif_queue_stop);
 		}
 
@@ -2599,7 +2624,7 @@ bnad_start_xmit(struct sk_buff *skb, struct net_device *netdev)
 			BNAD_UPDATE_CTR(bnad, netif_queue_stop);
 			return NETDEV_TX_BUSY;
 		} else {
-			netif_wake_queue(netdev);
+			netif_wake_subqueue(netdev, txq_id);
 			BNAD_UPDATE_CTR(bnad, netif_queue_wakeup);
 		}
 	}
@@ -2624,7 +2649,8 @@ bnad_start_xmit(struct sk_buff *skb, struct net_device *netdev)
 	}
 	if (test_bit(BNAD_RF_CEE_RUNNING, &bnad->run_flags)) {
 		vlan_tag =
-			(tcb->priority & 0x7) << 13 | (vlan_tag & 0x1fff);
+			((tcb->priority & 0x7) << VLAN_PRIO_SHIFT)
+							| (vlan_tag & 0x1fff);
 		flags |= (BNA_TXQ_WI_CF_INS_PRIO | BNA_TXQ_WI_CF_INS_VLAN);
 	}
 
@@ -2771,6 +2797,50 @@ bnad_get_stats64(struct net_device *netdev, struct rtnl_link_stats64 *stats)
 	return stats;
 }
 
+static bool bnad_is_iscsi(struct sk_buff *skb)
+{
+	u16		proto = 0;
+	struct tcphdr		*th;
+
+	if (skb->protocol == htons(ETH_P_IP))
+		proto = ip_hdr(skb)->protocol;
+	else if (skb->protocol == htons(ETH_P_IPV6))
+		/* nexthdr may not be TCP immediately. */
+		proto = ipv6_hdr(skb)->nexthdr;
+	if (proto == IPPROTO_TCP) {
+		th = tcp_hdr(skb);
+		if (BNAD_IS_ISCSI_PKT(th))
+			return true;
+	}
+
+	return false;
+}
+
+static u16
+bnad_tx_select_queue(struct net_device *netdev, struct sk_buff *skb)
+{
+	struct bnad *bnad =  netdev_priv(netdev);
+	struct bna *bna = &bnad->bna;
+	u8	prio = 0;
+
+	if (bnad->num_txq_per_tx < BFI_TX_MAX_PRIO)
+		prio = 0;
+	else if (bna_is_iscsi_over_cee(&bnad->bna) && bnad_is_iscsi(skb))
+		prio = bna_iscsi_prio(bna);
+	else if (vlan_tx_tag_present(skb)) {
+		u8 pkt_vlan_prio = 0;
+		u16 pkt_vlan_tag = 0;
+		pkt_vlan_tag = (u16)vlan_tx_tag_get(skb);
+		pkt_vlan_prio = (pkt_vlan_tag & VLAN_PRIO_MASK)
+					>> VLAN_PRIO_SHIFT;
+		prio = bna_prio_allowed(bna, pkt_vlan_prio) ?
+			pkt_vlan_prio : bna_default_prio(bna);
+	} else
+		prio = bna_default_prio(bna);
+
+	return (u16)prio;
+}
+
 static void
 bnad_set_rx_mode(struct net_device *netdev)
 {
@@ -2977,6 +3047,7 @@ bnad_netpoll(struct net_device *netdev)
 static const struct net_device_ops bnad_netdev_ops = {
 	.ndo_open		= bnad_open,
 	.ndo_stop		= bnad_stop,
+	.ndo_select_queue	= bnad_tx_select_queue,
 	.ndo_start_xmit		= bnad_start_xmit,
 	.ndo_get_stats64		= bnad_get_stats64,
 	.ndo_set_rx_mode	= bnad_set_rx_mode,
@@ -3173,7 +3244,7 @@ bnad_pci_probe(struct pci_dev *pdev,
 	 * Allocates sizeof(struct net_device + struct bnad)
 	 * bnad = netdev->priv
 	 */
-	netdev = alloc_etherdev(sizeof(struct bnad));
+	netdev = alloc_etherdev_mq(sizeof(struct bnad), BNAD_MAX_TXQ);
 	if (!netdev) {
 		dev_err(&pdev->dev, "netdev allocation failed\n");
 		err = -ENOMEM;
diff --git a/drivers/net/ethernet/brocade/bna/bnad.h b/drivers/net/ethernet/brocade/bna/bnad.h
index 60c2e9d..c8664d5 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.h
+++ b/drivers/net/ethernet/brocade/bna/bnad.h
@@ -40,7 +40,6 @@
 
 #define BNAD_MAX_TX		1
 #define BNAD_MAX_TXQ_PER_TX	8	/* 8 priority queues */
-#define BNAD_TXQ_NUM		1
 
 #define BNAD_MAX_RX		1
 #define BNAD_MAX_RXP_PER_RX	16
@@ -98,6 +97,9 @@ struct bnad_rx_ctrl {
 #define BNAD_RXQ_STARTED		1
 
 /* Resource limits */
+#define BNAD_MAX_TXQ			(BNAD_MAX_TX * BNAD_MAX_TXQ_PER_TX)
+#define BNAD_MAX_RXP			(BNAD_MAX_RX * BNAD_MAX_RXP_PER_RX)
+
 #define BNAD_NUM_TXQ			(bnad->num_tx * bnad->num_txq_per_tx)
 #define BNAD_NUM_RXP			(bnad->num_rx * bnad->num_rxp_per_rx)
 
@@ -231,6 +233,15 @@ struct bnad_unmap_q {
 /* Defined as bit positions */
 #define BNAD_FP_IN_RX_PATH	      0
 
+/*
+ * Deep Inspection : Checks if packet is ISCSI based on
+ * standard iSCSI port
+ */
+#define BNAD_TCP_ISCSI_PORT 3260
+#define BNAD_IS_ISCSI_PKT(_tch)				\
+(((_tch)->source == ntohs(BNAD_TCP_ISCSI_PORT)) ||	\
+	((_tch)->dest == ntohs(BNAD_TCP_ISCSI_PORT)))
+
 struct bnad {
 	struct net_device	*netdev;
 
-- 
1.7.1

[PATCH 05/14] bna: SKB Check and Drop Macros

[Rasesh Mody]

Add macros to check and drop skb from transmit path and return.

Signed-off-by: Gurunatha Karaje <gkaraje@brocade.com>
Signed-off-by: Rasesh Mody <rmody@brocade.com>
---
 drivers/net/ethernet/brocade/bna/bnad.c |   68 +++++++++++++++++-------------
 drivers/net/ethernet/brocade/bna/bnad.h |    2 +
 2 files changed, 40 insertions(+), 30 deletions(-)

diff --git a/drivers/net/ethernet/brocade/bna/bnad.c b/drivers/net/ethernet/brocade/bna/bnad.c
index 895f18b..64e2106 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.c
+++ b/drivers/net/ethernet/brocade/bna/bnad.c
@@ -74,6 +74,36 @@ do {								\
 
 #define BNAD_TXRX_SYNC_MDELAY	250	/* 250 msecs */
 
+#define BNAD_DROP_AND_RETURN(_counter)	\
+{ \
+	dev_kfree_skb(skb); \
+	BNAD_UPDATE_CTR(bnad, _counter); \
+	return NETDEV_TX_OK; \
+}
+
+#define BNAD_DROP_AND_RETURN_IF(_condition, _counter)	\
+if (unlikely(_condition)) { \
+	BNAD_DROP_AND_RETURN(_counter); \
+}
+
+#define BNAD_PCI_UNMAP_SKB(_pdev, _array, _index, _depth, _skb, _frag) \
+{ \
+	int j; \
+	(_array)[_index].skb = NULL; \
+	dma_unmap_single(_pdev, dma_unmap_addr(&(_array)[_index], dma_addr), \
+			skb_headlen(_skb), DMA_TO_DEVICE); \
+	dma_unmap_addr_set(&(_array)[_index], dma_addr, 0); \
+	BNA_QE_INDX_ADD(_index, 1, _depth); \
+	for (j = 0; j < (_frag); j++) { \
+		prefetch(&(_array)[(_index) + 1]); \
+		dma_unmap_page(_pdev, dma_unmap_addr(&(_array)[_index], \
+						     dma_addr), \
+			  skb_shinfo(_skb)->frags[j].size, DMA_TO_DEVICE); \
+		dma_unmap_addr_set(&(_array)[_index], dma_addr, 0); \
+		BNA_QE_INDX_ADD(_index, 1, _depth); \
+	} \
+}
+
 /*
  * Reinitialize completions in CQ, once Rx is taken down
  */
@@ -169,7 +199,6 @@ bnad_free_txbufs(struct bnad *bnad,
 	struct bnad_unmap_q *unmap_q = tcb->unmap_q;
 	struct bnad_skb_unmap *unmap_array;
 	struct sk_buff		*skb;
-	int i;
 
 	/*
 	 * Just return if TX is stopped. This check is useful
@@ -195,32 +224,14 @@ bnad_free_txbufs(struct bnad *bnad,
 	while (wis) {
 		skb = unmap_array[unmap_cons].skb;
 
-		unmap_array[unmap_cons].skb = NULL;
-
 		sent_packets++;
 		sent_bytes += skb->len;
 		wis -= BNA_TXQ_WI_NEEDED(1 + skb_shinfo(skb)->nr_frags);
 
-		dma_unmap_single(&bnad->pcidev->dev,
-				 dma_unmap_addr(&unmap_array[unmap_cons],
-						dma_addr), skb_headlen(skb),
-				 DMA_TO_DEVICE);
-		dma_unmap_addr_set(&unmap_array[unmap_cons], dma_addr, 0);
-		BNA_QE_INDX_ADD(unmap_cons, 1, unmap_q->q_depth);
-
-		prefetch(&unmap_array[unmap_cons + 1]);
-		for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
-			prefetch(&unmap_array[unmap_cons + 1]);
+		BNAD_PCI_UNMAP_SKB(&bnad->pcidev->dev, unmap_array, unmap_cons,
+				   unmap_q->q_depth, skb,
+				   skb_shinfo(skb)->nr_frags);
 
-			dma_unmap_page(&bnad->pcidev->dev,
-				       dma_unmap_addr(&unmap_array[unmap_cons],
-						      dma_addr),
-				       skb_shinfo(skb)->frags[i].size,
-				       DMA_TO_DEVICE);
-			dma_unmap_addr_set(&unmap_array[unmap_cons], dma_addr,
-					   0);
-			BNA_QE_INDX_ADD(unmap_cons, 1, unmap_q->q_depth);
-		}
 		dev_kfree_skb_any(skb);
 	}
 
@@ -2585,16 +2596,13 @@ bnad_start_xmit(struct sk_buff *skb, struct net_device *netdev)
 	 * Takes care of the Tx that is scheduled between clearing the flag
 	 * and the netif_stop_all_queue() call.
 	 */
-	if (unlikely(!test_bit(BNAD_TXQ_TX_STARTED, &tcb->flags))) {
-		dev_kfree_skb(skb);
-		return NETDEV_TX_OK;
-	}
+	BNAD_DROP_AND_RETURN_IF(!test_bit(BNAD_TXQ_TX_STARTED, &tcb->flags),
+				tx_skb_stopping);
 
 	vectors = 1 + skb_shinfo(skb)->nr_frags;
-	if (vectors > BFI_TX_MAX_VECTORS_PER_PKT) {
-		dev_kfree_skb(skb);
-		return NETDEV_TX_OK;
-	}
+	BNAD_DROP_AND_RETURN_IF(vectors > BFI_TX_MAX_VECTORS_PER_PKT,
+				tx_skb_max_vectors);
+
 	wis = BNA_TXQ_WI_NEEDED(vectors);	/* 4 vectors per work item */
 	acked = 0;
 	if (unlikely(wis > BNA_QE_FREE_CNT(tcb, tcb->q_depth) ||
diff --git a/drivers/net/ethernet/brocade/bna/bnad.h b/drivers/net/ethernet/brocade/bna/bnad.h
index c8664d5..6074c6d 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.h
+++ b/drivers/net/ethernet/brocade/bna/bnad.h
@@ -150,6 +150,8 @@ struct bnad_drv_stats {
 	u64		udpcsum_offload;
 	u64		csum_help;
 	u64		csum_help_err;
+	u64		tx_skb_stopping;
+	u64		tx_skb_max_vectors;
 
 	u64		hw_stats_updates;
 	u64		netif_rx_schedule;
-- 
1.7.1

[PATCH 06/14] bna: TX Path and RX Path Changes

[Rasesh Mody]

Change details:
 - Disable and enable interrupts from the same polling context to prevent
   reordering in Rx path.
 - Add Rx NAPI debug counters.
 - Make NAPI budget check more generic
 - Add a macro bnad_dim_timer_stop for DIM(Dynamic Interrupt Moderation)
   timer stop
 - Handle reduced MSI-X vectors case in bnad_enable_msix
 - Replace existing checks with macros and add more checks for illegal skbs
   in transmit path. Add more tx_skb counters for dropped skbs.
 - Check for single frame TSO skbs and send them out as non-TSO.
 - Put memory barrier after bna_txq_prod_indx_doorbell()

Signed-off-by: Gurunatha Karaje <gkaraje@brocade.com>
Signed-off-by: Rasesh Mody <rmody@brocade.com>
---
 drivers/net/ethernet/brocade/bna/bnad.c |  207 +++++++++++++++++--------------
 drivers/net/ethernet/brocade/bna/bnad.h |   33 +++++-
 2 files changed, 148 insertions(+), 92 deletions(-)

diff --git a/drivers/net/ethernet/brocade/bna/bnad.c b/drivers/net/ethernet/brocade/bna/bnad.c
index 64e2106..0faa8a1 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.c
+++ b/drivers/net/ethernet/brocade/bna/bnad.c
@@ -532,7 +532,7 @@ bnad_poll_cq(struct bnad *bnad, struct bna_ccb *ccb, int budget)
 		      (flags & BNA_CQ_EF_L4_CKSUM_OK)))
 			skb->ip_summed = CHECKSUM_UNNECESSARY;
 		else
-			skb_checksum_none_assert(skb);
+			skb->ip_summed = CHECKSUM_NONE;
 
 		rcb->rxq->rx_packets++;
 		rcb->rxq->rx_bytes += skb->len;
@@ -555,7 +555,8 @@ next:
 	BNA_QE_INDX_ADD(ccb->producer_index, wis, ccb->q_depth);
 
 	if (likely(test_bit(BNAD_RXQ_STARTED, &ccb->rcb[0]->flags)))
-		bna_ib_ack(ccb->i_dbell, packets);
+		bna_ib_ack_disable_irq(ccb->i_dbell, packets);
+
 	bnad_refill_rxq(bnad, ccb->rcb[0]);
 	if (ccb->rcb[1])
 		bnad_refill_rxq(bnad, ccb->rcb[1]);
@@ -593,10 +594,9 @@ bnad_netif_rx_schedule_poll(struct bnad *bnad, struct bna_ccb *ccb)
 	struct napi_struct *napi = &rx_ctrl->napi;
 
 	if (likely(napi_schedule_prep(napi))) {
-		bnad_disable_rx_irq(bnad, ccb);
 		__napi_schedule(napi);
+		rx_ctrl->rx_schedule++;
 	}
-	BNAD_UPDATE_CTR(bnad, netif_rx_schedule);
 }
 
 /* MSIX Rx Path Handler */
@@ -605,8 +605,10 @@ bnad_msix_rx(int irq, void *data)
 {
 	struct bna_ccb *ccb = (struct bna_ccb *)data;
 
-	if (ccb)
+	if (ccb) {
+		((struct bnad_rx_ctrl *)(ccb->ctrl))->rx_intr_ctr++;
 		bnad_netif_rx_schedule_poll(ccb->bnad, ccb);
+	}
 
 	return IRQ_HANDLED;
 }
@@ -1675,22 +1677,23 @@ bnad_napi_poll_rx(struct napi_struct *napi, int budget)
 	struct bnad *bnad = rx_ctrl->bnad;
 	int rcvd = 0;
 
+	rx_ctrl->rx_poll_ctr++;
 
 	if (!netif_carrier_ok(bnad->netdev))
 		goto poll_exit;
 
 	rcvd = bnad_poll_cq(bnad, rx_ctrl->ccb, budget);
-	if (rcvd == budget)
+	if (rcvd >= budget)
 		return rcvd;
 
 poll_exit:
 	napi_complete((napi));
 
-	BNAD_UPDATE_CTR(bnad, netif_rx_complete);
-
+	rx_ctrl->rx_complete++;
 
 	if (rx_ctrl->ccb)
-		bnad_enable_rx_irq(bnad, rx_ctrl->ccb);
+		bnad_enable_rx_irq_unsafe(rx_ctrl->ccb);
+
 	return rcvd;
 }
 
@@ -1894,20 +1897,14 @@ bnad_cleanup_rx(struct bnad *bnad, u32 rx_id)
 	struct bna_rx_config *rx_config = &bnad->rx_config[rx_id];
 	struct bna_res_info *res_info = &bnad->rx_res_info[rx_id].res_info[0];
 	unsigned long flags;
-	int dim_timer_del = 0;
 
 	if (!rx_info->rx)
 		return;
 
-	if (0 == rx_id) {
-		spin_lock_irqsave(&bnad->bna_lock, flags);
-		dim_timer_del = bnad_dim_timer_running(bnad);
-		if (dim_timer_del)
-			clear_bit(BNAD_RF_DIM_TIMER_RUNNING, &bnad->run_flags);
-		spin_unlock_irqrestore(&bnad->bna_lock, flags);
-		if (dim_timer_del)
-			del_timer_sync(&bnad->dim_timer);
-	}
+	spin_lock_irqsave(&bnad->bna_lock, flags);
+	if (0 == rx_id)
+		bnad_dim_timer_stop(bnad, flags);
+	spin_unlock_irqrestore(&bnad->bna_lock, flags);
 
 	init_completion(&bnad->bnad_completions.rx_comp);
 	spin_lock_irqsave(&bnad->bna_lock, flags);
@@ -2408,12 +2405,11 @@ bnad_enable_msix(struct bnad *bnad)
 
 		spin_lock_irqsave(&bnad->bna_lock, flags);
 		/* ret = #of vectors that we got */
-		bnad_q_num_adjust(bnad, ret, 0);
+		bnad_q_num_adjust(bnad, (ret - BNAD_MAILBOX_MSIX_VECTORS) / 2,
+			(ret - BNAD_MAILBOX_MSIX_VECTORS) / 2);
 		spin_unlock_irqrestore(&bnad->bna_lock, flags);
 
-		bnad->msix_num = (bnad->num_tx * bnad->num_txq_per_tx)
-			+ (bnad->num_rx
-			* bnad->num_rxp_per_rx) +
+		bnad->msix_num = BNAD_NUM_TXQ + BNAD_NUM_RXP +
 			 BNAD_MAILBOX_MSIX_VECTORS;
 
 		if (bnad->msix_num > ret)
@@ -2571,26 +2567,21 @@ bnad_start_xmit(struct sk_buff *skb, struct net_device *netdev)
 	u32		unmap_prod, wis, wis_used, wi_range;
 	u32		vectors, vect_id, i, acked;
 	int			err;
+	unsigned int		len;
 
 	dma_addr_t		dma_addr;
 	struct bna_txq_entry *txqent;
 	u16	flags;
 
-	if (unlikely
-	    (skb->len <= ETH_HLEN || skb->len > BFI_TX_MAX_DATA_PER_PKT)) {
-		dev_kfree_skb(skb);
-		return NETDEV_TX_OK;
-	}
+	BNAD_DROP_AND_RETURN_IF(skb->len <= ETH_HLEN, tx_skb_too_short);
+	BNAD_DROP_AND_RETURN_IF(skb_headlen(skb) > BFI_TX_MAX_DATA_PER_VECTOR,
+				tx_skb_headlen_too_long);
+	BNAD_DROP_AND_RETURN_IF(skb_headlen(skb) == 0, tx_skb_headlen_zero);
 
 	txq_id = skb_get_queue_mapping(skb);
 
 	tcb = bnad->tx_info[0].tcb[txq_id];
 
-	if (unlikely(!tcb)) {
-		dev_kfree_skb(skb);
-		return NETDEV_TX_OK;
-	}
-
 	unmap_q = tcb->unmap_q;
 	/*
 	 * Takes care of the Tx that is scheduled between clearing the flag
@@ -2638,8 +2629,6 @@ bnad_start_xmit(struct sk_buff *skb, struct net_device *netdev)
 	}
 
 	unmap_prod = unmap_q->producer_index;
-	wis_used = 1;
-	vect_id = 0;
 	flags = 0;
 
 	txq_prod = tcb->producer_index;
@@ -2647,9 +2636,6 @@ bnad_start_xmit(struct sk_buff *skb, struct net_device *netdev)
 	BUG_ON(!(wi_range <= tcb->q_depth));
 	txqent->hdr.wi.reserved = 0;
 	txqent->hdr.wi.num_vectors = vectors;
-	txqent->hdr.wi.opcode =
-		htons((skb_is_gso(skb) ? BNA_TXQ_WI_SEND_LSO :
-		       BNA_TXQ_WI_SEND));
 
 	if (vlan_tx_tag_present(skb)) {
 		vlan_tag = (u16) vlan_tx_tag_get(skb);
@@ -2665,62 +2651,74 @@ bnad_start_xmit(struct sk_buff *skb, struct net_device *netdev)
 	txqent->hdr.wi.vlan_tag = htons(vlan_tag);
 
 	if (skb_is_gso(skb)) {
-		err = bnad_tso_prepare(bnad, skb);
-		if (err) {
-			dev_kfree_skb(skb);
-			return NETDEV_TX_OK;
+		BNAD_DROP_AND_RETURN_IF(skb_is_gso(skb) > netdev->mtu,
+					tx_skb_mss_too_long);
+		if (unlikely((skb_is_gso(skb) + skb_transport_offset(skb) +
+			      tcp_hdrlen(skb)) >= skb->len)) {
+			txqent->hdr.wi.opcode =
+				__constant_htons(BNA_TXQ_WI_SEND);
+			txqent->hdr.wi.lso_mss = 0;
+			BNAD_UPDATE_CTR(bnad, tx_skb_tso_too_short);
+		} else {
+			txqent->hdr.wi.opcode =
+				__constant_htons(BNA_TXQ_WI_SEND_LSO);
+			txqent->hdr.wi.lso_mss = htons(skb_is_gso(skb));
 		}
-		txqent->hdr.wi.lso_mss = htons(skb_is_gso(skb));
+
+		err = bnad_tso_prepare(bnad, skb);
+		BNAD_DROP_AND_RETURN_IF(err, tx_skb_tso_prepare);
 		flags |= (BNA_TXQ_WI_CF_IP_CKSUM | BNA_TXQ_WI_CF_TCP_CKSUM);
 		txqent->hdr.wi.l4_hdr_size_n_offset =
 			htons(BNA_TXQ_WI_L4_HDR_N_OFFSET
 			      (tcp_hdrlen(skb) >> 2,
 			       skb_transport_offset(skb)));
-	} else if (skb->ip_summed == CHECKSUM_PARTIAL) {
-		u8 proto = 0;
-
+	} else {
+		txqent->hdr.wi.opcode =	__constant_htons(BNA_TXQ_WI_SEND);
 		txqent->hdr.wi.lso_mss = 0;
 
-		if (skb->protocol == htons(ETH_P_IP))
-			proto = ip_hdr(skb)->protocol;
-		else if (skb->protocol == htons(ETH_P_IPV6)) {
-			/* nexthdr may not be TCP immediately. */
-			proto = ipv6_hdr(skb)->nexthdr;
-		}
-		if (proto == IPPROTO_TCP) {
-			flags |= BNA_TXQ_WI_CF_TCP_CKSUM;
-			txqent->hdr.wi.l4_hdr_size_n_offset =
-				htons(BNA_TXQ_WI_L4_HDR_N_OFFSET
-				      (0, skb_transport_offset(skb)));
-
-			BNAD_UPDATE_CTR(bnad, tcpcsum_offload);
-
-			BUG_ON(!(skb_headlen(skb) >=
-				skb_transport_offset(skb) + tcp_hdrlen(skb)));
+		BNAD_DROP_AND_RETURN_IF(skb->len > (netdev->mtu + ETH_HLEN),
+					tx_skb_non_tso_too_long);
 
-		} else if (proto == IPPROTO_UDP) {
-			flags |= BNA_TXQ_WI_CF_UDP_CKSUM;
-			txqent->hdr.wi.l4_hdr_size_n_offset =
-				htons(BNA_TXQ_WI_L4_HDR_N_OFFSET
-				      (0, skb_transport_offset(skb)));
+		if (skb->ip_summed == CHECKSUM_PARTIAL) {
+			u8 proto = 0;
 
-			BNAD_UPDATE_CTR(bnad, udpcsum_offload);
-
-			BUG_ON(!(skb_headlen(skb) >=
-				   skb_transport_offset(skb) +
-				   sizeof(struct udphdr)));
-		} else {
-			err = skb_checksum_help(skb);
-			BNAD_UPDATE_CTR(bnad, csum_help);
-			if (err) {
-				dev_kfree_skb(skb);
-				BNAD_UPDATE_CTR(bnad, csum_help_err);
-				return NETDEV_TX_OK;
+			if (skb->protocol == __constant_htons(ETH_P_IP))
+				proto = ip_hdr(skb)->protocol;
+			else if (skb->protocol ==
+				 __constant_htons(ETH_P_IPV6)) {
+				/* nexthdr may not be TCP immediately. */
+				proto = ipv6_hdr(skb)->nexthdr;
+			}
+			if (proto == IPPROTO_TCP) {
+				flags |= BNA_TXQ_WI_CF_TCP_CKSUM;
+				txqent->hdr.wi.l4_hdr_size_n_offset =
+					htons(BNA_TXQ_WI_L4_HDR_N_OFFSET
+					      (0, skb_transport_offset(skb)));
+
+				BNAD_UPDATE_CTR(bnad, tcpcsum_offload);
+
+				BNAD_DROP_AND_RETURN_IF(skb_headlen(skb) <
+					    skb_transport_offset(skb) +
+					    tcp_hdrlen(skb), tx_skb_tcp_hdr);
+
+			} else if (proto == IPPROTO_UDP) {
+				flags |= BNA_TXQ_WI_CF_UDP_CKSUM;
+				txqent->hdr.wi.l4_hdr_size_n_offset =
+					htons(BNA_TXQ_WI_L4_HDR_N_OFFSET
+					      (0, skb_transport_offset(skb)));
+
+				BNAD_UPDATE_CTR(bnad, udpcsum_offload);
+				BNAD_DROP_AND_RETURN_IF(skb_headlen(skb) <
+					    skb_transport_offset(skb) +
+					    sizeof(struct udphdr),
+							tx_skb_udp_hdr);
+
+			} else {
+				BNAD_DROP_AND_RETURN(tx_skb_csum_err);
 			}
+		} else {
+			txqent->hdr.wi.l4_hdr_size_n_offset = 0;
 		}
-	} else {
-		txqent->hdr.wi.lso_mss = 0;
-		txqent->hdr.wi.l4_hdr_size_n_offset = 0;
 	}
 
 	txqent->hdr.wi.flags = htons(flags);
@@ -2728,20 +2726,36 @@ bnad_start_xmit(struct sk_buff *skb, struct net_device *netdev)
 	txqent->hdr.wi.frame_length = htonl(skb->len);
 
 	unmap_q->unmap_array[unmap_prod].skb = skb;
-	BUG_ON(!(skb_headlen(skb) <= BFI_TX_MAX_DATA_PER_VECTOR));
-	txqent->vector[vect_id].length = htons(skb_headlen(skb));
+	len = skb_headlen(skb);
+	txqent->vector[0].length = htons(len);
 	dma_addr = dma_map_single(&bnad->pcidev->dev, skb->data,
 				  skb_headlen(skb), DMA_TO_DEVICE);
 	dma_unmap_addr_set(&unmap_q->unmap_array[unmap_prod], dma_addr,
 			   dma_addr);
 
-	BNA_SET_DMA_ADDR(dma_addr, &txqent->vector[vect_id].host_addr);
+	BNA_SET_DMA_ADDR(dma_addr, &txqent->vector[0].host_addr);
 	BNA_QE_INDX_ADD(unmap_prod, 1, unmap_q->q_depth);
 
+	vect_id = 0;
+	wis_used = 1;
+
 	for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
 		struct skb_frag_struct *frag = &skb_shinfo(skb)->frags[i];
 		u16		size = frag->size;
 
+		if (unlikely(size == 0)) {
+			unmap_prod = unmap_q->producer_index;
+			prefetch(&unmap_q->unmap_array[unmap_prod + 1]);
+
+			BNAD_PCI_UNMAP_SKB(&bnad->pcidev->dev,
+					   unmap_q->unmap_array,
+					   unmap_prod, unmap_q->q_depth, skb,
+					   i);
+			BNAD_DROP_AND_RETURN(tx_skb_frag_zero);
+		}
+
+		len += size;
+
 		if (++vect_id == BFI_TX_MAX_VECTORS_PER_WI) {
 			vect_id = 0;
 			if (--wi_range)
@@ -2752,10 +2766,10 @@ bnad_start_xmit(struct sk_buff *skb, struct net_device *netdev)
 				wis_used = 0;
 				BNA_TXQ_QPGE_PTR_GET(txq_prod, tcb->sw_qpt,
 						     txqent, wi_range);
-				BUG_ON(!(wi_range <= tcb->q_depth));
 			}
 			wis_used++;
-			txqent->hdr.wi_ext.opcode = htons(BNA_TXQ_WI_EXTENSION);
+			txqent->hdr.wi_ext.opcode =
+				__constant_htons(BNA_TXQ_WI_EXTENSION);
 		}
 
 		BUG_ON(!(size <= BFI_TX_MAX_DATA_PER_VECTOR));
@@ -2768,6 +2782,16 @@ bnad_start_xmit(struct sk_buff *skb, struct net_device *netdev)
 		BNA_QE_INDX_ADD(unmap_prod, 1, unmap_q->q_depth);
 	}
 
+	if (unlikely(len != skb->len)) {
+		unmap_prod = unmap_q->producer_index;
+		prefetch(&unmap_q->unmap_array[unmap_prod + 1]);
+
+		BNAD_PCI_UNMAP_SKB(&bnad->pcidev->dev, unmap_q->unmap_array,
+				   unmap_prod, unmap_q->q_depth, skb,
+				   skb_shinfo(skb)->nr_frags);
+		BNAD_DROP_AND_RETURN(tx_skb_len_mismatch);
+	}
+
 	unmap_q->producer_index = unmap_prod;
 	BNA_QE_INDX_ADD(txq_prod, wis_used, tcb->q_depth);
 	tcb->producer_index = txq_prod;
@@ -2778,6 +2802,7 @@ bnad_start_xmit(struct sk_buff *skb, struct net_device *netdev)
 		return NETDEV_TX_OK;
 
 	bna_txq_prod_indx_doorbell(tcb);
+	smp_mb();
 
 	if ((u16) (*tcb->hw_consumer_index) != tcb->consumer_index)
 		tasklet_schedule(&bnad->tx_free_tasklet);
@@ -2888,6 +2913,9 @@ bnad_set_rx_mode(struct net_device *netdev)
 		}
 	}
 
+	if (bnad->rx_info[0].rx == NULL)
+		goto unlock;
+
 	bna_rx_mode_set(bnad->rx_info[0].rx, new_mask, valid_mask, NULL);
 
 	if (!netdev_mc_empty(netdev)) {
@@ -3040,12 +3068,9 @@ bnad_netpoll(struct net_device *netdev)
 				continue;
 			for (j = 0; j < bnad->num_rxp_per_rx; j++) {
 				rx_ctrl = &rx_info->rx_ctrl[j];
-				if (rx_ctrl->ccb) {
-					bnad_disable_rx_irq(bnad,
-							    rx_ctrl->ccb);
+				if (rx_ctrl->ccb)
 					bnad_netif_rx_schedule_poll(bnad,
 							    rx_ctrl->ccb);
-				}
 			}
 		}
 	}
diff --git a/drivers/net/ethernet/brocade/bna/bnad.h b/drivers/net/ethernet/brocade/bna/bnad.h
index 6074c6d..f402149 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.h
+++ b/drivers/net/ethernet/brocade/bna/bnad.h
@@ -55,6 +55,11 @@ struct bnad_rx_ctrl {
 	struct bnad *bnad;
 	unsigned long  flags;
 	struct napi_struct	napi;
+	u64		rx_intr_ctr;
+	u64		rx_poll_ctr;
+	u64		rx_schedule;
+	u64		rx_keep_poll;
+	u64		rx_complete;
 };
 
 #define BNAD_RXMODE_PROMISC_DEFAULT	BNA_RXMODE_PROMISC
@@ -150,8 +155,20 @@ struct bnad_drv_stats {
 	u64		udpcsum_offload;
 	u64		csum_help;
 	u64		csum_help_err;
+	u64		tx_skb_too_short;
 	u64		tx_skb_stopping;
 	u64		tx_skb_max_vectors;
+	u64		tx_skb_mss_too_long;
+	u64		tx_skb_tso_too_short;
+	u64		tx_skb_tso_prepare;
+	u64		tx_skb_non_tso_too_long;
+	u64		tx_skb_tcp_hdr;
+	u64		tx_skb_udp_hdr;
+	u64		tx_skb_csum_err;
+	u64		tx_skb_headlen_too_long;
+	u64		tx_skb_headlen_zero;
+	u64		tx_skb_frag_zero;
+	u64		tx_skb_len_mismatch;
 
 	u64		hw_stats_updates;
 	u64		netif_rx_schedule;
@@ -359,7 +376,7 @@ extern void bnad_netdev_hwstats_fill(struct bnad *bnad,
 
 #define bnad_enable_rx_irq_unsafe(_ccb)			\
 {							\
-	if (likely(test_bit(BNAD_RXQ_STARTED, &ccb->rcb[0]->flags))) {\
+	if (likely(test_bit(BNAD_RXQ_STARTED, &(_ccb)->rcb[0]->flags))) {\
 		bna_ib_coalescing_timer_set((_ccb)->i_dbell,	\
 			(_ccb)->rx_coalescing_timeo);		\
 		bna_ib_ack((_ccb)->i_dbell, 0);			\
@@ -369,5 +386,19 @@ extern void bnad_netdev_hwstats_fill(struct bnad *bnad,
 #define bnad_dim_timer_running(_bnad)				\
 	(((_bnad)->cfg_flags & BNAD_CF_DIM_ENABLED) &&		\
 	(test_bit(BNAD_RF_DIM_TIMER_RUNNING, &((_bnad)->run_flags))))
+#define bnad_dim_timer_stop(_bnad, _flags)		\
+do {							\
+	int to_del = 0;					\
+							\
+	if ((_bnad)->cfg_flags & BNAD_CF_DIM_ENABLED &&	\
+	    test_bit(BNAD_RF_DIM_TIMER_RUNNING, &(_bnad)->run_flags)) {\
+		clear_bit(BNAD_RF_DIM_TIMER_RUNNING, &(_bnad)->run_flags);\
+		to_del = 1;				\
+	}						\
+	spin_unlock_irqrestore(&(_bnad)->bna_lock, (_flags));	\
+	if (to_del)					\
+		del_timer_sync(&(_bnad)->dim_timer);	\
+	spin_lock_irqsave(&(_bnad)->bna_lock, (_flags));\
+} while (0)
 
 #endif /* __BNAD_H__ */
-- 
1.7.1

[PATCH 07/14] bna: Formatting and Code Cleanup

[Rasesh Mody]

Change details:
 - Print log messages when running with reduced number of MSI-X vectors
   and when defaulting to INTx mode.
 - Remove BUG_ONs and header file inclusion that are not needed
 - Comments addition/cleanup
 - Unused code cleanup
 - Add New Line to Print msg in bfa_sm_fault
 - Formatting fix

Signed-off-by: Gurunatha Karaje <gkaraje@brocade.com>
Signed-off-by: Rasesh Mody <rmody@brocade.com>
---
 drivers/net/ethernet/brocade/bna/bfa_cee.c         |    2 -
 .../net/ethernet/brocade/bna/bfa_defs_mfg_comm.h   |    1 -
 drivers/net/ethernet/brocade/bna/bfi.h             |   46 --------------------
 drivers/net/ethernet/brocade/bna/bna.h             |   18 +++-----
 drivers/net/ethernet/brocade/bna/bna_types.h       |    1 -
 drivers/net/ethernet/brocade/bna/bnad.c            |   46 ++++++--------------
 drivers/net/ethernet/brocade/bna/bnad.h            |   13 +++---
 drivers/net/ethernet/brocade/bna/cna.h             |   11 ++---
 8 files changed, 31 insertions(+), 107 deletions(-)

diff --git a/drivers/net/ethernet/brocade/bna/bfa_cee.c b/drivers/net/ethernet/brocade/bna/bfa_cee.c
index b45b8eb..8e62718 100644
--- a/drivers/net/ethernet/brocade/bna/bfa_cee.c
+++ b/drivers/net/ethernet/brocade/bna/bfa_cee.c
@@ -16,8 +16,6 @@
  * www.brocade.com
  */
 
-#include "bfa_defs_cna.h"
-#include "cna.h"
 #include "bfa_cee.h"
 #include "bfi_cna.h"
 #include "bfa_ioc.h"
diff --git a/drivers/net/ethernet/brocade/bna/bfa_defs_mfg_comm.h b/drivers/net/ethernet/brocade/bna/bfa_defs_mfg_comm.h
index 7ddd16f..7e5df90 100644
--- a/drivers/net/ethernet/brocade/bna/bfa_defs_mfg_comm.h
+++ b/drivers/net/ethernet/brocade/bna/bfa_defs_mfg_comm.h
@@ -18,7 +18,6 @@
 #ifndef __BFA_DEFS_MFG_COMM_H__
 #define __BFA_DEFS_MFG_COMM_H__
 
-#include "cna.h"
 #include "bfa_defs.h"
 
 /**
diff --git a/drivers/net/ethernet/brocade/bna/bfi.h b/drivers/net/ethernet/brocade/bna/bfi.h
index 19654cc..4e04c14 100644
--- a/drivers/net/ethernet/brocade/bna/bfi.h
+++ b/drivers/net/ethernet/brocade/bna/bfi.h
@@ -73,20 +73,6 @@ struct bfi_mhdr {
  ****************************************************************************
  */
 
-#define BFI_SGE_INLINE	1
-#define BFI_SGE_INLINE_MAX	(BFI_SGE_INLINE + 1)
-
-/**
- * SG Flags
- */
-enum {
-	BFI_SGE_DATA		= 0,	/*!< data address, not last	     */
-	BFI_SGE_DATA_CPL	= 1,	/*!< data addr, last in current page */
-	BFI_SGE_DATA_LAST	= 3,	/*!< data address, last		     */
-	BFI_SGE_LINK		= 2,	/*!< link address		     */
-	BFI_SGE_PGDLEN		= 2,	/*!< cumulative data length for page */
-};
-
 /**
  * DMA addresses
  */
@@ -97,33 +83,6 @@ union bfi_addr_u {
 	} a32;
 };
 
-/**
- * Scatter Gather Element
- */
-struct bfi_sge {
-#ifdef __BIGENDIAN
-	u32	flags:2,
-			rsvd:2,
-			sg_len:28;
-#else
-	u32	sg_len:28,
-			rsvd:2,
-			flags:2;
-#endif
-	union bfi_addr_u sga;
-};
-
-/**
- * Scatter Gather Page
- */
-#define BFI_SGPG_DATA_SGES		7
-#define BFI_SGPG_SGES_MAX		(BFI_SGPG_DATA_SGES + 1)
-#define BFI_SGPG_RSVD_WD_LEN	8
-struct bfi_sgpg {
-	struct bfi_sge sges[BFI_SGPG_SGES_MAX];
-	u32	rsvd[BFI_SGPG_RSVD_WD_LEN];
-};
-
 /*
  * Large Message structure - 128 Bytes size Msgs
  */
@@ -131,11 +90,6 @@ struct bfi_sgpg {
 #define BFI_LMSG_PL_WSZ	\
 			((BFI_LMSG_SZ - sizeof(struct bfi_mhdr)) / 4)
 
-struct bfi_msg {
-	struct bfi_mhdr mhdr;
-	u32	pl[BFI_LMSG_PL_WSZ];
-};
-
 /**
  * Mailbox message structure
  */
diff --git a/drivers/net/ethernet/brocade/bna/bna.h b/drivers/net/ethernet/brocade/bna/bna.h
index e4f914c..c0e5ba8 100644
--- a/drivers/net/ethernet/brocade/bna/bna.h
+++ b/drivers/net/ethernet/brocade/bna/bna.h
@@ -10,12 +10,17 @@
  * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
  * General Public License for more details.
  */
+/*
+ * Copyright (c) 2005-2011 Brocade Communications Systems, Inc.
+ * All rights reserved
+ * www.brocade.com
+ */
 #ifndef __BNA_H__
 #define __BNA_H__
 
-#include "bfa_cs.h"
+#include "bfa_defs.h"
 #include "bfa_ioc.h"
-#include "cna.h"
+#include "bfi_enet.h"
 #include "bna_types.h"
 
 extern const u32 bna_napi_dim_vector[][BNA_BIAS_T_MAX];
@@ -403,12 +408,8 @@ void bna_mod_init(struct bna *bna, struct bna_res_info *res_info);
 void bna_uninit(struct bna *bna);
 int bna_num_txq_set(struct bna *bna, int num_txq);
 int bna_num_rxp_set(struct bna *bna, int num_rxp);
-void bna_stats_get(struct bna *bna);
-void bna_get_perm_mac(struct bna *bna, u8 *mac);
 void bna_hw_stats_get(struct bna *bna);
 
-/* APIs for Rx */
-
 /* APIs for RxF */
 struct bna_mac *bna_ucam_mod_mac_get(struct bna_ucam_mod *ucam_mod);
 void bna_ucam_mod_mac_put(struct bna_ucam_mod *ucam_mod,
@@ -529,11 +530,6 @@ bna_rx_mode_set(struct bna_rx *rx, enum bna_rxmode rxmode,
 void bna_rx_vlan_add(struct bna_rx *rx, int vlan_id);
 void bna_rx_vlan_del(struct bna_rx *rx, int vlan_id);
 void bna_rx_vlanfilter_enable(struct bna_rx *rx);
-void bna_rx_hds_enable(struct bna_rx *rx, struct bna_hds_config *hds_config,
-		       void (*cbfn)(struct bnad *, struct bna_rx *));
-void bna_rx_hds_disable(struct bna_rx *rx,
-			void (*cbfn)(struct bnad *, struct bna_rx *));
-
 /**
  * ENET
  */
diff --git a/drivers/net/ethernet/brocade/bna/bna_types.h b/drivers/net/ethernet/brocade/bna/bna_types.h
index 8a6da0c..59417b1 100644
--- a/drivers/net/ethernet/brocade/bna/bna_types.h
+++ b/drivers/net/ethernet/brocade/bna/bna_types.h
@@ -21,7 +21,6 @@
 #include "cna.h"
 #include "bna_hw_defs.h"
 #include "bfa_cee.h"
-#include "bfi_enet.h"
 #include "bfa_msgq.h"
 
 /**
diff --git a/drivers/net/ethernet/brocade/bna/bnad.c b/drivers/net/ethernet/brocade/bna/bnad.c
index 0faa8a1..869d172 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.c
+++ b/drivers/net/ethernet/brocade/bna/bnad.c
@@ -402,10 +402,9 @@ bnad_alloc_n_post_rxbufs(struct bnad *bnad, struct bna_rcb *rcb)
 	BNA_RXQ_QPGE_PTR_GET(unmap_prod, rcb->sw_qpt, rxent, wi_range);
 
 	while (to_alloc--) {
-		if (!wi_range) {
+		if (!wi_range)
 			BNA_RXQ_QPGE_PTR_GET(unmap_prod, rcb->sw_qpt, rxent,
 					     wi_range);
-		}
 		skb = netdev_alloc_skb_ip_align(bnad->netdev,
 						rcb->rxq->buffer_size);
 		if (unlikely(!skb)) {
@@ -567,27 +566,6 @@ next:
 }
 
 static void
-bnad_disable_rx_irq(struct bnad *bnad, struct bna_ccb *ccb)
-{
-	if (unlikely(!test_bit(BNAD_RXQ_STARTED, &ccb->rcb[0]->flags)))
-		return;
-
-	bna_ib_coalescing_timer_set(ccb->i_dbell, 0);
-	bna_ib_ack(ccb->i_dbell, 0);
-}
-
-static void
-bnad_enable_rx_irq(struct bnad *bnad, struct bna_ccb *ccb)
-{
-	unsigned long flags;
-
-	/* Because of polling context */
-	spin_lock_irqsave(&bnad->bna_lock, flags);
-	bnad_enable_rx_irq_unsafe(ccb);
-	spin_unlock_irqrestore(&bnad->bna_lock, flags);
-}
-
-static void
 bnad_netif_rx_schedule_poll(struct bnad *bnad, struct bna_ccb *ccb)
 {
 	struct bnad_rx_ctrl *rx_ctrl = (struct bnad_rx_ctrl *)(ccb->ctrl);
@@ -1687,7 +1665,7 @@ bnad_napi_poll_rx(struct napi_struct *napi, int budget)
 		return rcvd;
 
 poll_exit:
-	napi_complete((napi));
+	napi_complete(napi);
 
 	rx_ctrl->rx_complete++;
 
@@ -2098,15 +2076,13 @@ bnad_enable_default_bcast(struct bnad *bnad)
 	return 0;
 }
 
-/* Called with bnad_conf_lock() held */
+/* Called with mutex_lock(&bnad->conf_mutex) held */
 static void
 bnad_restore_vlans(struct bnad *bnad, u32 rx_id)
 {
 	u16 vid;
 	unsigned long flags;
 
-	BUG_ON(!(VLAN_N_VID == BFI_ENET_VLAN_ID_MAX));
-
 	for_each_set_bit(vid, bnad->active_vlans, VLAN_N_VID) {
 		spin_lock_irqsave(&bnad->bna_lock, flags);
 		bna_rx_vlan_add(bnad->rx_info[rx_id].rx, vid);
@@ -2215,9 +2191,6 @@ bnad_tso_prepare(struct bnad *bnad, struct sk_buff *skb)
 {
 	int err;
 
-	/* SKB_GSO_TCPV4 and SKB_GSO_TCPV6 is defined since 2.6.18. */
-	BUG_ON(!(skb_shinfo(skb)->gso_type == SKB_GSO_TCPV4 ||
-		   skb_shinfo(skb)->gso_type == SKB_GSO_TCPV6));
 	if (skb_header_cloned(skb)) {
 		err = pskb_expand_head(skb, 0, 0, GFP_ATOMIC);
 		if (err) {
@@ -2244,7 +2217,6 @@ bnad_tso_prepare(struct bnad *bnad, struct sk_buff *skb)
 	} else {
 		struct ipv6hdr *ipv6h = ipv6_hdr(skb);
 
-		BUG_ON(!(skb->protocol == htons(ETH_P_IPV6)));
 		ipv6h->payload_len = 0;
 		tcp_hdr(skb)->check =
 			~csum_ipv6_magic(&ipv6h->saddr, &ipv6h->daddr, 0,
@@ -2402,6 +2374,8 @@ bnad_enable_msix(struct bnad *bnad)
 	ret = pci_enable_msix(bnad->pcidev, bnad->msix_table, bnad->msix_num);
 	if (ret > 0) {
 		/* Not enough MSI-X vectors. */
+		pr_warn("BNA: %d MSI-X vectors allocated < %d requested\n",
+			ret, bnad->msix_num);
 
 		spin_lock_irqsave(&bnad->bna_lock, flags);
 		/* ret = #of vectors that we got */
@@ -2430,6 +2404,7 @@ bnad_enable_msix(struct bnad *bnad)
 	return;
 
 intx_mode:
+	pr_warn("BNA: MSI-X enable failed - operating in INTx mode\n");
 
 	kfree(bnad->msix_table);
 	bnad->msix_table = NULL;
@@ -2585,7 +2560,7 @@ bnad_start_xmit(struct sk_buff *skb, struct net_device *netdev)
 	unmap_q = tcb->unmap_q;
 	/*
 	 * Takes care of the Tx that is scheduled between clearing the flag
-	 * and the netif_stop_all_queue() call.
+	 * and the netif_tx_stop_all_queues() call.
 	 */
 	BNAD_DROP_AND_RETURN_IF(!test_bit(BNAD_TXQ_TX_STARTED, &tcb->flags),
 				tx_skb_stopping);
@@ -2633,7 +2608,6 @@ bnad_start_xmit(struct sk_buff *skb, struct net_device *netdev)
 
 	txq_prod = tcb->producer_index;
 	BNA_TXQ_QPGE_PTR_GET(txq_prod, tcb->sw_qpt, txqent, wi_range);
-	BUG_ON(!(wi_range <= tcb->q_depth));
 	txqent->hdr.wi.reserved = 0;
 	txqent->hdr.wi.num_vectors = vectors;
 
@@ -3062,6 +3036,12 @@ bnad_netpoll(struct net_device *netdev)
 		bnad_isr(bnad->pcidev->irq, netdev);
 		bna_intx_enable(&bnad->bna, curr_mask);
 	} else {
+		/*
+		 * Tx processing may happen in sending context, so no need
+		 * to explicitly process completions here
+		 */
+
+		/* Rx processing */
 		for (i = 0; i < bnad->num_rx; i++) {
 			rx_info = &bnad->rx_info[i];
 			if (!rx_info->rx)
diff --git a/drivers/net/ethernet/brocade/bna/bnad.h b/drivers/net/ethernet/brocade/bna/bnad.h
index f402149..516fbdd 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.h
+++ b/drivers/net/ethernet/brocade/bna/bnad.h
@@ -64,8 +64,6 @@ struct bnad_rx_ctrl {
 
 #define BNAD_RXMODE_PROMISC_DEFAULT	BNA_RXMODE_PROMISC
 
-#define BNAD_GET_TX_ID(_skb)	(0)
-
 /*
  * GLOBAL #defines (CONSTANTS)
  */
@@ -154,7 +152,6 @@ struct bnad_drv_stats {
 	u64		tcpcsum_offload;
 	u64		udpcsum_offload;
 	u64		csum_help;
-	u64		csum_help_err;
 	u64		tx_skb_too_short;
 	u64		tx_skb_stopping;
 	u64		tx_skb_max_vectors;
@@ -171,13 +168,10 @@ struct bnad_drv_stats {
 	u64		tx_skb_len_mismatch;
 
 	u64		hw_stats_updates;
-	u64		netif_rx_schedule;
-	u64		netif_rx_complete;
 	u64		netif_rx_dropped;
 
 	u64		link_toggle;
 	u64		cee_toggle;
-	u64		cee_up;
 
 	u64		rxp_info_alloc_failed;
 	u64		mbox_intr_disabled;
@@ -386,6 +380,13 @@ extern void bnad_netdev_hwstats_fill(struct bnad *bnad,
 #define bnad_dim_timer_running(_bnad)				\
 	(((_bnad)->cfg_flags & BNAD_CF_DIM_ENABLED) &&		\
 	(test_bit(BNAD_RF_DIM_TIMER_RUNNING, &((_bnad)->run_flags))))
+
+/*
+ * Stops the DIM timer
+ * Called with bnad->bna_lock held
+ * Implemented as macro, since we want to use
+ * the correct flags(on stack) while unlocking.
+ */
 #define bnad_dim_timer_stop(_bnad, _flags)		\
 do {							\
 	int to_del = 0;					\
diff --git a/drivers/net/ethernet/brocade/bna/cna.h b/drivers/net/ethernet/brocade/bna/cna.h
index 50fce15..cb48742 100644
--- a/drivers/net/ethernet/brocade/bna/cna.h
+++ b/drivers/net/ethernet/brocade/bna/cna.h
@@ -21,21 +21,18 @@
 
 #include <linux/kernel.h>
 #include <linux/types.h>
+#include <linux/mutex.h>
 #include <linux/pci.h>
 #include <linux/delay.h>
 #include <linux/bitops.h>
 #include <linux/timer.h>
 #include <linux/interrupt.h>
+#include <linux/if_vlan.h>
 #include <linux/if_ether.h>
-#include <asm/page.h>
-#include <asm/io.h>
-#include <asm/string.h>
-
-#include <linux/list.h>
 
 #define bfa_sm_fault(__event)    do {                            \
-	pr_err("SM Assertion failure: %s: %d: event = %d", __FILE__, __LINE__, \
-		__event); \
+	pr_err("SM Assertion failure: %s: %d: event = %d\n",	\
+		 __FILE__, __LINE__, __event);			\
 } while (0)
 
 extern char bfa_version[];
-- 
1.7.1

[PATCH 08/14] bna: Initialization and Locking Fix

[Rasesh Mody]

Change details:
 - Initialize rx_id to 0 for bnad_cleanup_rx
 - Return -ENOMEM in case if bna_rx_create fails
 - Count the Rx buffer allocation failures in bnad_alloc_n_post_rxbufs()
 - Remove unnecessary initialization of using_dac to false in bnad_pci_probe
 - Release lock if error while doing bna_num_txq_set in bnad_pci_probe
 - Release all the locks while doing free_netdev

Signed-off-by: Gurunatha Karaje <gkaraje@brocade.com>
Signed-off-by: Rasesh Mody <rmody@brocade.com>
---
 drivers/net/ethernet/brocade/bna/bna_hw_defs.h |    1 +
 drivers/net/ethernet/brocade/bna/bnad.c        |   15 ++++++++++++---
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/brocade/bna/bna_hw_defs.h b/drivers/net/ethernet/brocade/bna/bna_hw_defs.h
index 07bb792..7ecdca5 100644
--- a/drivers/net/ethernet/brocade/bna/bna_hw_defs.h
+++ b/drivers/net/ethernet/brocade/bna/bna_hw_defs.h
@@ -99,6 +99,7 @@
 	(_bna)->bits.error_status_bits = (__HFN_INT_ERR_MASK);		\
 	(_bna)->bits.error_mask_bits = (__HFN_INT_ERR_MASK);		\
 	(_bna)->bits.halt_status_bits = __HFN_INT_LL_HALT;		\
+	(_bna)->bits.halt_mask_bits = __HFN_INT_LL_HALT;		\
 }
 
 #define ct2_reg_addr_init(_bna, _pcidev)				\
diff --git a/drivers/net/ethernet/brocade/bna/bnad.c b/drivers/net/ethernet/brocade/bna/bnad.c
index 869d172..11c058e 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.c
+++ b/drivers/net/ethernet/brocade/bna/bnad.c
@@ -409,6 +409,7 @@ bnad_alloc_n_post_rxbufs(struct bnad *bnad, struct bna_rcb *rcb)
 						rcb->rxq->buffer_size);
 		if (unlikely(!skb)) {
 			BNAD_UPDATE_CTR(bnad, rxbuf_alloc_failed);
+			rcb->rxq->rxbuf_alloc_failed++;
 			goto finishing;
 		}
 		unmap_array[unmap_prod].skb = skb;
@@ -1900,6 +1901,7 @@ bnad_cleanup_rx(struct bnad *bnad, u32 rx_id)
 	spin_unlock_irqrestore(&bnad->bna_lock, flags);
 
 	rx_info->rx = NULL;
+	rx_info->rx_id = 0;
 
 	bnad_rx_res_free(bnad, res_info);
 }
@@ -1955,8 +1957,10 @@ bnad_setup_rx(struct bnad *bnad, u32 rx_id)
 	rx = bna_rx_create(&bnad->bna, bnad, rx_config, &rx_cbfn, res_info,
 			rx_info);
 	spin_unlock_irqrestore(&bnad->bna_lock, flags);
-	if (!rx)
+	if (!rx) {
+		err = -ENOMEM;
 		goto err_return;
+	}
 	rx_info->rx = rx;
 
 	/*
@@ -3234,7 +3238,7 @@ static int __devinit
 bnad_pci_probe(struct pci_dev *pdev,
 		const struct pci_device_id *pcidev_id)
 {
-	bool	using_dac = false;
+	bool	using_dac;
 	int	err;
 	struct bnad *bnad;
 	struct bna *bna;
@@ -3357,6 +3361,11 @@ bnad_pci_probe(struct pci_dev *pdev,
 			bna_num_rxp_set(bna, BNAD_NUM_RXP + 1))
 			err = -EIO;
 	}
+	spin_unlock_irqrestore(&bnad->bna_lock, flags);
+	if (err)
+		goto disable_ioceth;
+
+	spin_lock_irqsave(&bnad->bna_lock, flags);
 	bna_mod_res_req(&bnad->bna, &bnad->mod_res_info[0]);
 	spin_unlock_irqrestore(&bnad->bna_lock, flags);
 
@@ -3410,9 +3419,9 @@ drv_uninit:
 	bnad_uninit(bnad);
 pci_uninit:
 	bnad_pci_uninit(pdev);
+free_netdev:
 	mutex_unlock(&bnad->conf_mutex);
 	bnad_lock_uninit(bnad);
-free_netdev:
 	free_netdev(netdev);
 	return err;
 }
-- 
1.7.1

[PATCH 09/14] bna: Ethtool Enhancements and Fix

[Rasesh Mody]

Change details:
 - Set coalescing time out for multiple tx objects.
 - Use bnad_dim_timer_stop macro in bnad_set_coalesce.
 - Add tx_skb counters and NAPI debug counters to ethtool stats.
 - Add rlb stats strings to bnad_net_stats_strings{} array. rlb_stats field
   was added to struct bfi_enet_stats {} but the corresponding name structure
   array for ethtool was not initialized with right strings, even though the
   actual name structure array got expanded. This caused a NULL pointer
   violation and a crash when doing ehtool -S <if_name>.
 - While setting the ring parameter restore the rx, vlan configuration and
   set rx mode
 - Indentation fix

Signed-off-by: Gurunatha Karaje <gkaraje@brocade.com>
Signed-off-by: Rasesh Mody <rmody@brocade.com>
---
 drivers/net/ethernet/brocade/bna/bnad.c         |   21 +++---
 drivers/net/ethernet/brocade/bna/bnad.h         |   10 ++-
 drivers/net/ethernet/brocade/bna/bnad_ethtool.c |   88 +++++++++++++++++++----
 3 files changed, 92 insertions(+), 27 deletions(-)

diff --git a/drivers/net/ethernet/brocade/bna/bnad.c b/drivers/net/ethernet/brocade/bna/bnad.c
index 11c058e..76bfa19 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.c
+++ b/drivers/net/ethernet/brocade/bna/bnad.c
@@ -2008,12 +2008,15 @@ void
 bnad_tx_coalescing_timeo_set(struct bnad *bnad)
 {
 	struct bnad_tx_info *tx_info;
+	int i;
 
-	tx_info = &bnad->tx_info[0];
-	if (!tx_info->tx)
-		return;
-
-	bna_tx_coalescing_timeo_set(tx_info->tx, bnad->tx_coalescing_timeo);
+	for (i = 0; i < bnad->num_tx; i++) {
+		tx_info = &bnad->tx_info[i];
+		if (!tx_info->tx)
+			continue;
+		bna_tx_coalescing_timeo_set(tx_info->tx,
+				bnad->tx_coalescing_timeo);
+	}
 }
 
 /* Called with conf_lock & bnad->bna_lock held */
@@ -2035,7 +2038,7 @@ bnad_rx_coalescing_timeo_set(struct bnad *bnad)
 /*
  * Called with bnad->bna_lock held
  */
-static int
+int
 bnad_mac_addr_set_locked(struct bnad *bnad, u8 *mac_addr)
 {
 	int ret;
@@ -2055,7 +2058,7 @@ bnad_mac_addr_set_locked(struct bnad *bnad, u8 *mac_addr)
 }
 
 /* Should be called with conf_lock held */
-static int
+int
 bnad_enable_default_bcast(struct bnad *bnad)
 {
 	struct bnad_rx_info *rx_info = &bnad->rx_info[0];
@@ -2081,7 +2084,7 @@ bnad_enable_default_bcast(struct bnad *bnad)
 }
 
 /* Called with mutex_lock(&bnad->conf_mutex) held */
-static void
+void
 bnad_restore_vlans(struct bnad *bnad, u32 rx_id)
 {
 	u16 vid;
@@ -2852,7 +2855,7 @@ bnad_tx_select_queue(struct net_device *netdev, struct sk_buff *skb)
 	return (u16)prio;
 }
 
-static void
+void
 bnad_set_rx_mode(struct net_device *netdev)
 {
 	struct bnad *bnad = netdev_priv(netdev);
diff --git a/drivers/net/ethernet/brocade/bna/bnad.h b/drivers/net/ethernet/brocade/bna/bnad.h
index 516fbdd..3132c50 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.h
+++ b/drivers/net/ethernet/brocade/bna/bnad.h
@@ -339,6 +339,12 @@ extern u32		bnad_rxqs_per_cq;
  */
 extern u32 *cna_get_firmware_buf(struct pci_dev *pdev);
 /* Netdev entry point prototypes */
+extern void bnad_set_rx_mode(struct net_device *netdev);
+extern struct net_device_stats *bnad_get_netdev_stats(
+				struct net_device *netdev);
+extern int bnad_mac_addr_set_locked(struct bnad *bnad, u8 *mac_addr);
+extern int bnad_enable_default_bcast(struct bnad *bnad);
+extern void bnad_restore_vlans(struct bnad *bnad, u32 rx_id);
 extern void bnad_set_ethtool_ops(struct net_device *netdev);
 
 /* Configuration & setup */
@@ -377,10 +383,6 @@ extern void bnad_netdev_hwstats_fill(struct bnad *bnad,
 	}							\
 }
 
-#define bnad_dim_timer_running(_bnad)				\
-	(((_bnad)->cfg_flags & BNAD_CF_DIM_ENABLED) &&		\
-	(test_bit(BNAD_RF_DIM_TIMER_RUNNING, &((_bnad)->run_flags))))
-
 /*
  * Stops the DIM timer
  * Called with bnad->bna_lock held
diff --git a/drivers/net/ethernet/brocade/bna/bnad_ethtool.c b/drivers/net/ethernet/brocade/bna/bnad_ethtool.c
index 1c19dce..1199f01 100644
--- a/drivers/net/ethernet/brocade/bna/bnad_ethtool.c
+++ b/drivers/net/ethernet/brocade/bna/bnad_ethtool.c
@@ -75,14 +75,25 @@ static char *bnad_net_stats_strings[BNAD_ETHTOOL_STATS_NUM] = {
 	"tcpcsum_offload",
 	"udpcsum_offload",
 	"csum_help",
-	"csum_help_err",
+	"tx_skb_too_short",
+	"tx_skb_stopping",
+	"tx_skb_max_vectors",
+	"tx_skb_mss_too_long",
+	"tx_skb_tso_too_short",
+	"tx_skb_tso_prepare",
+	"tx_skb_non_tso_too_long",
+	"tx_skb_tcp_hdr",
+	"tx_skb_udp_hdr",
+	"tx_skb_csum_err",
+	"tx_skb_headlen_too_long",
+	"tx_skb_headlen_zero",
+	"tx_skb_frag_zero",
+	"tx_skb_len_mismatch",
 	"hw_stats_updates",
-	"netif_rx_schedule",
-	"netif_rx_complete",
 	"netif_rx_dropped",
 
 	"link_toggle",
-	"cee_up",
+	"cee_toggle",
 
 	"rxp_info_alloc_failed",
 	"mbox_intr_disabled",
@@ -201,6 +212,20 @@ static char *bnad_net_stats_strings[BNAD_ETHTOOL_STATS_NUM] = {
 	"rad_rx_bcast_vlan",
 	"rad_rx_drops",
 
+	"rlb_rad_rx_frames",
+	"rlb_rad_rx_octets",
+	"rlb_rad_rx_vlan_frames",
+	"rlb_rad_rx_ucast",
+	"rlb_rad_rx_ucast_octets",
+	"rlb_rad_rx_ucast_vlan",
+	"rlb_rad_rx_mcast",
+	"rlb_rad_rx_mcast_octets",
+	"rlb_rad_rx_mcast_vlan",
+	"rlb_rad_rx_bcast",
+	"rlb_rad_rx_bcast_octets",
+	"rlb_rad_rx_bcast_vlan",
+	"rlb_rad_rx_drops",
+
 	"fc_rx_ucast_octets",
 	"fc_rx_ucast",
 	"fc_rx_ucast_vlan",
@@ -321,7 +346,6 @@ bnad_set_coalesce(struct net_device *netdev, struct ethtool_coalesce *coalesce)
 {
 	struct bnad *bnad = netdev_priv(netdev);
 	unsigned long flags;
-	int dim_timer_del = 0;
 
 	if (coalesce->rx_coalesce_usecs == 0 ||
 	    coalesce->rx_coalesce_usecs >
@@ -348,14 +372,7 @@ bnad_set_coalesce(struct net_device *netdev, struct ethtool_coalesce *coalesce)
 	} else {
 		if (bnad->cfg_flags & BNAD_CF_DIM_ENABLED) {
 			bnad->cfg_flags &= ~BNAD_CF_DIM_ENABLED;
-			dim_timer_del = bnad_dim_timer_running(bnad);
-			if (dim_timer_del) {
-				clear_bit(BNAD_RF_DIM_TIMER_RUNNING,
-							&bnad->run_flags);
-				spin_unlock_irqrestore(&bnad->bna_lock, flags);
-				del_timer_sync(&bnad->dim_timer);
-				spin_lock_irqsave(&bnad->bna_lock, flags);
-			}
+			bnad_dim_timer_stop(bnad, flags);
 			bnad_rx_coalescing_timeo_set(bnad);
 		}
 	}
@@ -407,6 +424,7 @@ bnad_set_ringparam(struct net_device *netdev,
 {
 	int i, current_err, err = 0;
 	struct bnad *bnad = netdev_priv(netdev);
+	unsigned long flags;
 
 	mutex_lock(&bnad->conf_mutex);
 	if (ringparam->rx_pending == bnad->rxq_depth &&
@@ -430,6 +448,11 @@ bnad_set_ringparam(struct net_device *netdev,
 
 	if (ringparam->rx_pending != bnad->rxq_depth) {
 		bnad->rxq_depth = ringparam->rx_pending;
+		if (!netif_running(netdev)) {
+			mutex_unlock(&bnad->conf_mutex);
+			return 0;
+		}
+
 		for (i = 0; i < bnad->num_rx; i++) {
 			if (!bnad->rx_info[i].rx)
 				continue;
@@ -437,10 +460,26 @@ bnad_set_ringparam(struct net_device *netdev,
 			current_err = bnad_setup_rx(bnad, i);
 			if (current_err && !err)
 				err = current_err;
+			if (!err)
+				bnad_restore_vlans(bnad, i);
+		}
+
+		if (!err && bnad->rx_info[0].rx) {
+			/* restore rx configuration */
+			bnad_enable_default_bcast(bnad);
+			spin_lock_irqsave(&bnad->bna_lock, flags);
+			bnad_mac_addr_set_locked(bnad, netdev->dev_addr);
+			spin_unlock_irqrestore(&bnad->bna_lock, flags);
+			bnad_set_rx_mode(netdev);
 		}
 	}
 	if (ringparam->tx_pending != bnad->txq_depth) {
 		bnad->txq_depth = ringparam->tx_pending;
+		if (!netif_running(netdev)) {
+			mutex_unlock(&bnad->conf_mutex);
+			return 0;
+		}
+
 		for (i = 0; i < bnad->num_tx; i++) {
 			if (!bnad->tx_info[i].tx)
 				continue;
@@ -578,6 +617,16 @@ bnad_get_strings(struct net_device *netdev, u32 stringset, u8 * string)
 				sprintf(string, "cq%d_hw_producer_index",
 					q_num);
 				string += ETH_GSTRING_LEN;
+				sprintf(string, "cq%d_intr", q_num);
+				string += ETH_GSTRING_LEN;
+				sprintf(string, "cq%d_poll", q_num);
+				string += ETH_GSTRING_LEN;
+				sprintf(string, "cq%d_schedule", q_num);
+				string += ETH_GSTRING_LEN;
+				sprintf(string, "cq%d_keep_poll", q_num);
+				string += ETH_GSTRING_LEN;
+				sprintf(string, "cq%d_complete", q_num);
+				string += ETH_GSTRING_LEN;
 				q_num++;
 			}
 		}
@@ -660,7 +709,7 @@ static int
 bnad_get_stats_count_locked(struct net_device *netdev)
 {
 	struct bnad *bnad = netdev_priv(netdev);
-	int i, j, count, rxf_active_num = 0, txf_active_num = 0;
+	int i, j, count = 0, rxf_active_num = 0, txf_active_num = 0;
 	u32 bmap;
 
 	bmap = bna_tx_rid_mask(&bnad->bna);
@@ -718,6 +767,17 @@ bnad_per_q_stats_fill(struct bnad *bnad, u64 *buf, int bi)
 				buf[bi++] = 0; /* ccb->consumer_index */
 				buf[bi++] = *(bnad->rx_info[i].rx_ctrl[j].
 						ccb->hw_producer_index);
+
+				buf[bi++] = bnad->rx_info[i].
+						rx_ctrl[j].rx_intr_ctr;
+				buf[bi++] = bnad->rx_info[i].
+						rx_ctrl[j].rx_poll_ctr;
+				buf[bi++] = bnad->rx_info[i].
+						rx_ctrl[j].rx_schedule;
+				buf[bi++] = bnad->rx_info[i].
+						rx_ctrl[j].rx_keep_poll;
+				buf[bi++] = bnad->rx_info[i].
+						rx_ctrl[j].rx_complete;
 			}
 	}
 	for (i = 0; i < bnad->num_rx; i++) {
-- 
1.7.1

[PATCH 10/14] bna: Async Mode Tx Rx Init Fix

[Rasesh Mody]

Change details:
 - Async mode of Tx/Rx queue initialization in BNAD from a task queue context
   runs into non-unique taskq allocation issues. Get rid of Tx/Rx
   initialization from task q context
 - In the attach function, wait for IOC enable, then do Tx/Rx queue
   initialization. Default BNA attributes are used when IOC enable from attach
   fails and values are set to:
   1 TxQ, 1 RxQ, 1 Unicast MAC, 1 RIT entry

Signed-off-by: Gurunatha Karaje <gkaraje@brocade.com>
Signed-off-by: Rasesh Mody <rmody@brocade.com>
---
 drivers/net/ethernet/brocade/bna/bna_enet.c    |   29 ++++++++++++++++++-----
 drivers/net/ethernet/brocade/bna/bna_hw_defs.h |    4 +++
 drivers/net/ethernet/brocade/bna/bna_types.h   |    1 +
 3 files changed, 27 insertions(+), 7 deletions(-)

diff --git a/drivers/net/ethernet/brocade/bna/bna_enet.c b/drivers/net/ethernet/brocade/bna/bna_enet.c
index 68a275d..26f5c5a 100644
--- a/drivers/net/ethernet/brocade/bna/bna_enet.c
+++ b/drivers/net/ethernet/brocade/bna/bna_enet.c
@@ -167,13 +167,14 @@ bna_bfi_attr_get_rsp(struct bna_ioceth *ioceth,
 	 * Store only if not set earlier, since BNAD can override the HW
 	 * attributes
 	 */
-	if (!ioceth->attr.num_txq)
+	if (!ioceth->attr.fw_query_complete) {
 		ioceth->attr.num_txq = ntohl(rsp->max_cfg);
-	if (!ioceth->attr.num_rxp)
 		ioceth->attr.num_rxp = ntohl(rsp->max_cfg);
-	ioceth->attr.num_ucmac = ntohl(rsp->max_ucmac);
-	ioceth->attr.num_mcmac = BFI_ENET_MAX_MCAM;
-	ioceth->attr.max_rit_size = ntohl(rsp->rit_size);
+		ioceth->attr.num_ucmac = ntohl(rsp->max_ucmac);
+		ioceth->attr.num_mcmac = BFI_ENET_MAX_MCAM;
+		ioceth->attr.max_rit_size = ntohl(rsp->rit_size);
+		ioceth->attr.fw_query_complete = true;
+	}
 
 	bfa_fsm_send_event(ioceth, IOCETH_E_ENET_ATTR_RESP);
 }
@@ -1693,6 +1694,16 @@ static struct bfa_ioc_cbfn bna_ioceth_cbfn = {
 	bna_cb_ioceth_reset
 };
 
+static void bna_attr_init(struct bna_ioceth *ioceth)
+{
+	ioceth->attr.num_txq = BFI_ENET_DEF_TXQ;
+	ioceth->attr.num_rxp = BFI_ENET_DEF_RXP;
+	ioceth->attr.num_ucmac = BFI_ENET_DEF_UCAM;
+	ioceth->attr.num_mcmac = BFI_ENET_MAX_MCAM;
+	ioceth->attr.max_rit_size = BFI_ENET_DEF_RITSZ;
+	ioceth->attr.fw_query_complete = false;
+}
+
 static void
 bna_ioceth_init(struct bna_ioceth *ioceth, struct bna *bna,
 		struct bna_res_info *res_info)
@@ -1738,6 +1749,8 @@ bna_ioceth_init(struct bna_ioceth *ioceth, struct bna *bna,
 	ioceth->stop_cbfn = NULL;
 	ioceth->stop_cbarg = NULL;
 
+	bna_attr_init(ioceth);
+
 	bfa_fsm_set_state(ioceth, bna_ioceth_sm_stopped);
 }
 
@@ -2036,7 +2049,8 @@ bna_uninit(struct bna *bna)
 int
 bna_num_txq_set(struct bna *bna, int num_txq)
 {
-	if (num_txq > 0 && (num_txq <= bna->ioceth.attr.num_txq)) {
+	if (bna->ioceth.attr.fw_query_complete &&
+		(num_txq <= bna->ioceth.attr.num_txq)) {
 		bna->ioceth.attr.num_txq = num_txq;
 		return BNA_CB_SUCCESS;
 	}
@@ -2047,7 +2061,8 @@ bna_num_txq_set(struct bna *bna, int num_txq)
 int
 bna_num_rxp_set(struct bna *bna, int num_rxp)
 {
-	if (num_rxp > 0 && (num_rxp <= bna->ioceth.attr.num_rxp)) {
+	if (bna->ioceth.attr.fw_query_complete &&
+		(num_rxp <= bna->ioceth.attr.num_rxp)) {
 		bna->ioceth.attr.num_rxp = num_rxp;
 		return BNA_CB_SUCCESS;
 	}
diff --git a/drivers/net/ethernet/brocade/bna/bna_hw_defs.h b/drivers/net/ethernet/brocade/bna/bna_hw_defs.h
index 7ecdca5..dde8a46 100644
--- a/drivers/net/ethernet/brocade/bna/bna_hw_defs.h
+++ b/drivers/net/ethernet/brocade/bna/bna_hw_defs.h
@@ -30,6 +30,10 @@
  * SW imposed limits
  *
  */
+#define BFI_ENET_DEF_TXQ		1
+#define BFI_ENET_DEF_RXP		1
+#define BFI_ENET_DEF_UCAM		1
+#define BFI_ENET_DEF_RITSZ		1
 
 #define BFI_ENET_MAX_MCAM		256
 
diff --git a/drivers/net/ethernet/brocade/bna/bna_types.h b/drivers/net/ethernet/brocade/bna/bna_types.h
index 59417b1..242d799 100644
--- a/drivers/net/ethernet/brocade/bna/bna_types.h
+++ b/drivers/net/ethernet/brocade/bna/bna_types.h
@@ -323,6 +323,7 @@ struct bna_qpt {
 };
 
 struct bna_attr {
+	bool			fw_query_complete;
 	int			num_txq;
 	int			num_rxp;
 	int			num_ucmac;
-- 
1.7.1

[PATCH 11/14] bna: MBOX IRQ Flag Check after Locking

[Rasesh Mody]

Change details:
 - Check the BNAD_RF_MBOX_IRQ_DISABLED flag after acquiring the bna_lock.

Signed-off-by: Gurunatha Karaje <gkaraje@brocade.com>
Signed-off-by: Rasesh Mody <rmody@brocade.com>
---
 drivers/net/ethernet/brocade/bna/bnad.c |   18 +++++++++++-------
 1 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/drivers/net/ethernet/brocade/bna/bnad.c b/drivers/net/ethernet/brocade/bna/bnad.c
index 76bfa19..5d431de 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.c
+++ b/drivers/net/ethernet/brocade/bna/bnad.c
@@ -602,10 +602,11 @@ bnad_msix_mbox_handler(int irq, void *data)
 	unsigned long flags;
 	struct bnad *bnad = (struct bnad *)data;
 
-	if (unlikely(test_bit(BNAD_RF_MBOX_IRQ_DISABLED, &bnad->run_flags)))
-		return IRQ_HANDLED;
-
 	spin_lock_irqsave(&bnad->bna_lock, flags);
+	if (unlikely(test_bit(BNAD_RF_MBOX_IRQ_DISABLED, &bnad->run_flags))) {
+		spin_unlock_irqrestore(&bnad->bna_lock, flags);
+		return IRQ_HANDLED;
+	}
 
 	bna_intr_status_get(&bnad->bna, intr_status);
 
@@ -628,15 +629,18 @@ bnad_isr(int irq, void *data)
 	struct bnad_rx_ctrl *rx_ctrl;
 	struct bna_tcb *tcb = NULL;
 
-	if (unlikely(test_bit(BNAD_RF_MBOX_IRQ_DISABLED, &bnad->run_flags)))
+	spin_lock_irqsave(&bnad->bna_lock, flags);
+	if (unlikely(test_bit(BNAD_RF_MBOX_IRQ_DISABLED, &bnad->run_flags))) {
+		spin_unlock_irqrestore(&bnad->bna_lock, flags);
 		return IRQ_NONE;
+	}
 
 	bna_intr_status_get(&bnad->bna, intr_status);
 
-	if (unlikely(!intr_status))
+	if (unlikely(!intr_status)) {
+		spin_unlock_irqrestore(&bnad->bna_lock, flags);
 		return IRQ_NONE;
-
-	spin_lock_irqsave(&bnad->bna_lock, flags);
+	}
 
 	if (BNA_IS_MBOX_ERR_INTR(&bnad->bna, intr_status))
 		bna_mbox_handler(&bnad->bna, intr_status);
-- 
1.7.1

[PATCH 12/14] bna: Queue Depth and SKB Unmap Array Fix

[Rasesh Mody]

Change details:
 - Replaced global variable bnad_rxqs_per_cq with define BNAD_RXQS_PER_CQ
 - sk_buff unmap_array grows greater than 65536 (x2) with Tx ring of 65536.
   The index used for accessing it is incorrectly declared as u16. It quickly
   wraps around and accesses null sk_buff ptr. So using u32 to handle
   unmap_array.
 - Reducing TXQ depth and safe(max) acking of Tx events to 32768 (same as Rx)

Signed-off-by: Gurunatha Karaje <gkaraje@brocade.com>
Signed-off-by: Rasesh Mody <rmody@brocade.com>
---
 drivers/net/ethernet/brocade/bna/bnad.c         |    6 ++----
 drivers/net/ethernet/brocade/bna/bnad.h         |    6 +++++-
 drivers/net/ethernet/brocade/bna/bnad_ethtool.c |    8 ++++----
 3 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/drivers/net/ethernet/brocade/bna/bnad.c b/drivers/net/ethernet/brocade/bna/bnad.c
index 5d431de..4e80df3 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.c
+++ b/drivers/net/ethernet/brocade/bna/bnad.c
@@ -46,8 +46,6 @@ MODULE_PARM_DESC(bnad_ioc_auto_recover, "Enable / Disable auto recovery");
 /*
  * Global variables
  */
-u32 bnad_rxqs_per_cq = 2;
-
 static const u8 bnad_bcast_addr[] =  {0xff, 0xff, 0xff, 0xff, 0xff, 0xff};
 
 /*
@@ -194,8 +192,8 @@ static u32
 bnad_free_txbufs(struct bnad *bnad,
 		 struct bna_tcb *tcb)
 {
-	u32		sent_packets = 0, sent_bytes = 0;
-	u16		wis, unmap_cons, updated_hw_cons;
+	u32		unmap_cons, sent_packets = 0, sent_bytes = 0;
+	u16		wis, updated_hw_cons;
 	struct bnad_unmap_q *unmap_q = tcb->unmap_q;
 	struct bnad_skb_unmap *unmap_array;
 	struct sk_buff		*skb;
diff --git a/drivers/net/ethernet/brocade/bna/bnad.h b/drivers/net/ethernet/brocade/bna/bnad.h
index 3132c50..df57e85 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.h
+++ b/drivers/net/ethernet/brocade/bna/bnad.h
@@ -85,6 +85,11 @@ struct bnad_rx_ctrl {
 #define BNAD_MAX_Q_DEPTH		0x10000
 #define BNAD_MIN_Q_DEPTH		0x200
 
+#define BNAD_RXQS_PER_CQ		2
+#define BNAD_MAX_RXQ_DEPTH		(BNAD_MAX_Q_DEPTH / BNAD_RXQS_PER_CQ)
+/* keeping MAX TX and RX Q depth equal */
+#define BNAD_MAX_TXQ_DEPTH		BNAD_MAX_RXQ_DEPTH
+
 #define BNAD_JUMBO_MTU			9000
 
 #define BNAD_NETIF_WAKE_THRESHOLD	8
@@ -332,7 +337,6 @@ struct bnad {
  * EXTERN VARIABLES
  */
 extern struct firmware *bfi_fw;
-extern u32		bnad_rxqs_per_cq;
 
 /*
  * EXTERN PROTOTYPES
diff --git a/drivers/net/ethernet/brocade/bna/bnad_ethtool.c b/drivers/net/ethernet/brocade/bna/bnad_ethtool.c
index 1199f01..e85fb2b 100644
--- a/drivers/net/ethernet/brocade/bna/bnad_ethtool.c
+++ b/drivers/net/ethernet/brocade/bna/bnad_ethtool.c
@@ -407,10 +407,10 @@ bnad_get_ringparam(struct net_device *netdev,
 {
 	struct bnad *bnad = netdev_priv(netdev);
 
-	ringparam->rx_max_pending = BNAD_MAX_Q_DEPTH / bnad_rxqs_per_cq;
+	ringparam->rx_max_pending = BNAD_MAX_RXQ_DEPTH;
 	ringparam->rx_mini_max_pending = 0;
 	ringparam->rx_jumbo_max_pending = 0;
-	ringparam->tx_max_pending = BNAD_MAX_Q_DEPTH;
+	ringparam->tx_max_pending = BNAD_MAX_TXQ_DEPTH;
 
 	ringparam->rx_pending = bnad->rxq_depth;
 	ringparam->rx_mini_max_pending = 0;
@@ -434,13 +434,13 @@ bnad_set_ringparam(struct net_device *netdev,
 	}
 
 	if (ringparam->rx_pending < BNAD_MIN_Q_DEPTH ||
-	    ringparam->rx_pending > BNAD_MAX_Q_DEPTH / bnad_rxqs_per_cq ||
+	    ringparam->rx_pending > BNAD_MAX_RXQ_DEPTH ||
 	    !BNA_POWER_OF_2(ringparam->rx_pending)) {
 		mutex_unlock(&bnad->conf_mutex);
 		return -EINVAL;
 	}
 	if (ringparam->tx_pending < BNAD_MIN_Q_DEPTH ||
-	    ringparam->tx_pending > BNAD_MAX_Q_DEPTH ||
+	    ringparam->tx_pending > BNAD_MAX_TXQ_DEPTH ||
 	    !BNA_POWER_OF_2(ringparam->tx_pending)) {
 		mutex_unlock(&bnad->conf_mutex);
 		return -EINVAL;
-- 
1.7.1

[PATCH 13/14] bna: SKB PCI UNMAP Fix

[Rasesh Mody]

Change details:
 - Found a leak in sk_buff unmapping of PCI dma addresses where boundary
   conditions are not properly handled in freeing all Tx buffers. Freeing
   of all Tx buffers is done considering sk_buffs data and fragments can
   be mapped at the boundary.

Signed-off-by: Gurunatha Karaje <gkaraje@brocade.com>
Signed-off-by: Rasesh Mody <rmody@brocade.com>
---
 drivers/net/ethernet/brocade/bna/bnad.c |   39 ++++++++-----------------------
 1 files changed, 10 insertions(+), 29 deletions(-)

diff --git a/drivers/net/ethernet/brocade/bna/bnad.c b/drivers/net/ethernet/brocade/bna/bnad.c
index 4e80df3..99a4688 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.c
+++ b/drivers/net/ethernet/brocade/bna/bnad.c
@@ -143,39 +143,20 @@ bnad_free_all_txbufs(struct bnad *bnad,
 	struct bnad_unmap_q *unmap_q = tcb->unmap_q;
 	struct bnad_skb_unmap *unmap_array;
 	struct sk_buff		*skb = NULL;
-	int			i;
+	int			q;
 
 	unmap_array = unmap_q->unmap_array;
 
-	unmap_cons = 0;
-	while (unmap_cons < unmap_q->q_depth) {
-		skb = unmap_array[unmap_cons].skb;
-		if (!skb) {
-			unmap_cons++;
+	for (q = 0; q < unmap_q->q_depth; q++) {
+		skb = unmap_array[q].skb;
+		if (!skb)
 			continue;
-		}
-		unmap_array[unmap_cons].skb = NULL;
-
-		dma_unmap_single(&bnad->pcidev->dev,
-				 dma_unmap_addr(&unmap_array[unmap_cons],
-						dma_addr), skb_headlen(skb),
-						DMA_TO_DEVICE);
 
-		dma_unmap_addr_set(&unmap_array[unmap_cons], dma_addr, 0);
-		if (++unmap_cons >= unmap_q->q_depth)
-			break;
+		unmap_cons = q;
+		BNAD_PCI_UNMAP_SKB(&bnad->pcidev->dev, unmap_array, unmap_cons,
+				   unmap_q->q_depth, skb,
+				   skb_shinfo(skb)->nr_frags);
 
-		for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
-			dma_unmap_page(&bnad->pcidev->dev,
-				       dma_unmap_addr(&unmap_array[unmap_cons],
-						      dma_addr),
-				       skb_shinfo(skb)->frags[i].size,
-				       DMA_TO_DEVICE);
-			dma_unmap_addr_set(&unmap_array[unmap_cons], dma_addr,
-					   0);
-			if (++unmap_cons >= unmap_q->q_depth)
-				break;
-		}
 		dev_kfree_skb_any(skb);
 	}
 }
-- 
1.7.1

[PATCH 14/14] bna: Driver Version changed to 3.0.2.1

[Rasesh Mody]

Signed-off-by: Gurunatha Karaje <gkaraje@brocade.com>
Signed-off-by: Rasesh Mody <rmody@brocade.com>
---
 drivers/net/ethernet/brocade/bna/bnad.h |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/net/ethernet/brocade/bna/bnad.h b/drivers/net/ethernet/brocade/bna/bnad.h
index df57e85..6f1e972 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.h
+++ b/drivers/net/ethernet/brocade/bna/bnad.h
@@ -70,7 +70,7 @@ struct bnad_rx_ctrl {
 #define BNAD_NAME			"bna"
 #define BNAD_NAME_LEN			64
 
-#define BNAD_VERSION			"3.0.2.0"
+#define BNAD_VERSION			"3.0.2.1"
 
 #define BNAD_MAILBOX_MSIX_INDEX		0
 #define BNAD_MAILBOX_MSIX_VECTORS	1
-- 
1.7.1

Re: [PATCH 09/14] bna: Ethtool Enhancements and Fix

[Ben Hutchings]

On Tue, 2011-08-16 at 14:19 -0700, Rasesh Mody wrote:
> Change details:
>  - Set coalescing time out for multiple tx objects.
>  - Use bnad_dim_timer_stop macro in bnad_set_coalesce.
>  - Add tx_skb counters and NAPI debug counters to ethtool stats.
>  - Add rlb stats strings to bnad_net_stats_strings{} array. rlb_stats field
>    was added to struct bfi_enet_stats {} but the corresponding name structure
>    array for ethtool was not initialized with right strings, even though the
>    actual name structure array got expanded. This caused a NULL pointer
>    violation and a crash when doing ehtool -S <if_name>.
>  - While setting the ring parameter restore the rx, vlan configuration and
>    set rx mode
>  - Indentation fix
> 
> Signed-off-by: Gurunatha Karaje <gkaraje@brocade.com>
> Signed-off-by: Rasesh Mody <rmody@brocade.com>
> ---
>  drivers/net/ethernet/brocade/bna/bnad.c         |   21 +++---
>  drivers/net/ethernet/brocade/bna/bnad.h         |   10 ++-
>  drivers/net/ethernet/brocade/bna/bnad_ethtool.c |   88 +++++++++++++++++++----
>  3 files changed, 92 insertions(+), 27 deletions(-)
> 
> diff --git a/drivers/net/ethernet/brocade/bna/bnad.c b/drivers/net/ethernet/brocade/bna/bnad.c
> index 11c058e..76bfa19 100644
> --- a/drivers/net/ethernet/brocade/bna/bnad.c
> +++ b/drivers/net/ethernet/brocade/bna/bnad.c
> @@ -2008,12 +2008,15 @@ void
>  bnad_tx_coalescing_timeo_set(struct bnad *bnad)
>  {
>  	struct bnad_tx_info *tx_info;
> +	int i;
>  
> -	tx_info = &bnad->tx_info[0];
> -	if (!tx_info->tx)
> -		return;
> -
> -	bna_tx_coalescing_timeo_set(tx_info->tx, bnad->tx_coalescing_timeo);
> +	for (i = 0; i < bnad->num_tx; i++) {
> +		tx_info = &bnad->tx_info[i];
> +		if (!tx_info->tx)
> +			continue;
> +		bna_tx_coalescing_timeo_set(tx_info->tx,
> +				bnad->tx_coalescing_timeo);
> +	}
>  }
[...]

Doesn't this need to be done at the same time as patch 04/14 "bna: Add
Multiple Tx Queue Support"?

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: [PATCH 06/14] bna: TX Path and RX Path Changes

[Ben Hutchings]

On Tue, 2011-08-16 at 14:19 -0700, Rasesh Mody wrote:
> Change details:
>  - Disable and enable interrupts from the same polling context to prevent
>    reordering in Rx path.
>  - Add Rx NAPI debug counters.
>  - Make NAPI budget check more generic
>  - Add a macro bnad_dim_timer_stop for DIM(Dynamic Interrupt Moderation)
>    timer stop
>  - Handle reduced MSI-X vectors case in bnad_enable_msix
>  - Replace existing checks with macros and add more checks for illegal skbs
>    in transmit path. Add more tx_skb counters for dropped skbs.
>  - Check for single frame TSO skbs and send them out as non-TSO.
>  - Put memory barrier after bna_txq_prod_indx_doorbell()
> 
> Signed-off-by: Gurunatha Karaje <gkaraje@brocade.com>
> Signed-off-by: Rasesh Mody <rmody@brocade.com>
> ---
>  drivers/net/ethernet/brocade/bna/bnad.c |  207 +++++++++++++++++--------------
>  drivers/net/ethernet/brocade/bna/bnad.h |   33 +++++-
>  2 files changed, 148 insertions(+), 92 deletions(-)
> 
> diff --git a/drivers/net/ethernet/brocade/bna/bnad.c b/drivers/net/ethernet/brocade/bna/bnad.c
> index 64e2106..0faa8a1 100644
> --- a/drivers/net/ethernet/brocade/bna/bnad.c
> +++ b/drivers/net/ethernet/brocade/bna/bnad.c
> @@ -532,7 +532,7 @@ bnad_poll_cq(struct bnad *bnad, struct bna_ccb *ccb, int budget)
>  		      (flags & BNA_CQ_EF_L4_CKSUM_OK)))
>  			skb->ip_summed = CHECKSUM_UNNECESSARY;
>  		else
> -			skb_checksum_none_assert(skb);
> +			skb->ip_summed = CHECKSUM_NONE;
>  
>  		rcb->rxq->rx_packets++;
>  		rcb->rxq->rx_bytes += skb->len;
[...]

This is reverting part of:

commit bc8acf2c8c3e43fcc192762a9f964b3e9a17748b
Author: Eric Dumazet <eric.dumazet@gmail.com>
Date:   Thu Sep 2 13:07:41 2010 -0700

    drivers/net: avoid some skb->ip_summed initializations

and I don't see any justification for that.

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: [PATCH 04/14] bna: Add Multiple Tx Queue Support

[Ben Hutchings]

On Tue, 2011-08-16 at 14:19 -0700, Rasesh Mody wrote:
> Change details:
>  - Add macros bna_prio_allow, bna_default_nw_prio, bna_iscsi_prio,
>    bna_is_iscsi_over_cee
>  - Added support for multipe Tx queues with a separate iSCSI Tx queue based
>    on the default value of iSCSI port number. The feature is supported based
>    on the underlying hardware and enabled for DCB (CEE) mode only.
>  - Allocate multiple TxQ resource in netdev
>  - Implement bnad_tx_select_queue() which enables the correct selection of
>    TxQ Id (and tcb). This function is called either by the kernel to channel
>    packets to the right TxQ
>  - bnad_tx_select_queue() returns priority, while only a few packets during
>    transition could have wrong priority, all will be associated with a valid
>    non-NULL tcb.
>  - Implement bnad_iscsi_tcb_get() and BNAD_IS_ISCSI_PKT() for iSCSI packet
>    inspection and retrieval of tcb corresponding to the iSCSI priority.
>  - Construction of priority indirection table to be used by bnad to direct
>    packets into TxQs
[...]

You probably should implement TX priority classes through the
ndo_setup_tc operation, not ndo_select_queue.

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

RE: [PATCH 09/14] bna: Ethtool Enhancements and Fix

[Rasesh Mody]

>From: Ben Hutchings [mailto:bhutchings@solarflare.com]
>Sent: Tuesday, August 16, 2011 2:43 PM
>
>On Tue, 2011-08-16 at 14:19 -0700, Rasesh Mody wrote:
>> Change details:
>>  - Set coalescing time out for multiple tx objects.
>>  - Use bnad_dim_timer_stop macro in bnad_set_coalesce.
>>  - Add tx_skb counters and NAPI debug counters to ethtool stats.
>>  - Add rlb stats strings to bnad_net_stats_strings{} array. rlb_stats
>field
>>    was added to struct bfi_enet_stats {} but the corresponding name
>structure
>>    array for ethtool was not initialized with right strings, even
>though the
>>    actual name structure array got expanded. This caused a NULL
>pointer
>>    violation and a crash when doing ehtool -S <if_name>.
>>  - While setting the ring parameter restore the rx, vlan configuration
>and
>>    set rx mode
>>  - Indentation fix
>>
>> Signed-off-by: Gurunatha Karaje <gkaraje@brocade.com>
>> Signed-off-by: Rasesh Mody <rmody@brocade.com>
>> ---
>>  drivers/net/ethernet/brocade/bna/bnad.c         |   21 +++---
>>  drivers/net/ethernet/brocade/bna/bnad.h         |   10 ++-
>>  drivers/net/ethernet/brocade/bna/bnad_ethtool.c |   88
>+++++++++++++++++++----
>>  3 files changed, 92 insertions(+), 27 deletions(-)
>>
>> diff --git a/drivers/net/ethernet/brocade/bna/bnad.c
>b/drivers/net/ethernet/brocade/bna/bnad.c
>> index 11c058e..76bfa19 100644
>> --- a/drivers/net/ethernet/brocade/bna/bnad.c
>> +++ b/drivers/net/ethernet/brocade/bna/bnad.c
>> @@ -2008,12 +2008,15 @@ void
>>  bnad_tx_coalescing_timeo_set(struct bnad *bnad)
>>  {
>>  	struct bnad_tx_info *tx_info;
>> +	int i;
>>
>> -	tx_info = &bnad->tx_info[0];
>> -	if (!tx_info->tx)
>> -		return;
>> -
>> -	bna_tx_coalescing_timeo_set(tx_info->tx, bnad-
>>tx_coalescing_timeo);
>> +	for (i = 0; i < bnad->num_tx; i++) {
>> +		tx_info = &bnad->tx_info[i];
>> +		if (!tx_info->tx)
>> +			continue;
>> +		bna_tx_coalescing_timeo_set(tx_info->tx,
>> +				bnad->tx_coalescing_timeo);
>> +	}
>>  }
>[...]
>
>Doesn't this need to be done at the same time as patch 04/14 "bna: Add
>Multiple Tx Queue Support"?

Ben,
Thanks for the feedback!

Yes, this is Multiple TXQ related change and will move it to 04/14. Since
ethtool is using it, we thought it would make more sense to do this change
with other ethtool changes.

Rasesh 

RE: [PATCH 06/14] bna: TX Path and RX Path Changes

[Rasesh Mody]

>From: Ben Hutchings [mailto:bhutchings@solarflare.com]
>Sent: Tuesday, August 16, 2011 2:46 PM
>
>On Tue, 2011-08-16 at 14:19 -0700, Rasesh Mody wrote:
>> Change details:
>>  - Disable and enable interrupts from the same polling context to
>prevent
>>    reordering in Rx path.
>>  - Add Rx NAPI debug counters.
>>  - Make NAPI budget check more generic
>>  - Add a macro bnad_dim_timer_stop for DIM(Dynamic Interrupt
>Moderation)
>>    timer stop
>>  - Handle reduced MSI-X vectors case in bnad_enable_msix
>>  - Replace existing checks with macros and add more checks for illegal
>skbs
>>    in transmit path. Add more tx_skb counters for dropped skbs.
>>  - Check for single frame TSO skbs and send them out as non-TSO.
>>  - Put memory barrier after bna_txq_prod_indx_doorbell()
>>
>> Signed-off-by: Gurunatha Karaje <gkaraje@brocade.com>
>> Signed-off-by: Rasesh Mody <rmody@brocade.com>
>> ---
>>  drivers/net/ethernet/brocade/bna/bnad.c |  207 +++++++++++++++++-----
>---------
>>  drivers/net/ethernet/brocade/bna/bnad.h |   33 +++++-
>>  2 files changed, 148 insertions(+), 92 deletions(-)
>>
>> diff --git a/drivers/net/ethernet/brocade/bna/bnad.c
>b/drivers/net/ethernet/brocade/bna/bnad.c
>> index 64e2106..0faa8a1 100644
>> --- a/drivers/net/ethernet/brocade/bna/bnad.c
>> +++ b/drivers/net/ethernet/brocade/bna/bnad.c
>> @@ -532,7 +532,7 @@ bnad_poll_cq(struct bnad *bnad, struct bna_ccb
>*ccb, int budget)
>>  		      (flags & BNA_CQ_EF_L4_CKSUM_OK)))
>>  			skb->ip_summed = CHECKSUM_UNNECESSARY;
>>  		else
>> -			skb_checksum_none_assert(skb);
>> +			skb->ip_summed = CHECKSUM_NONE;
>>
>>  		rcb->rxq->rx_packets++;
>>  		rcb->rxq->rx_bytes += skb->len;
>[...]
>
>This is reverting part of:
>
>commit bc8acf2c8c3e43fcc192762a9f964b3e9a17748b
>Author: Eric Dumazet <eric.dumazet@gmail.com>
>Date:   Thu Sep 2 13:07:41 2010 -0700
>
>    drivers/net: avoid some skb->ip_summed initializations
>
>and I don't see any justification for that.

It is an erroneous change, this change will be dropped when resubmitting
this patch set.
Thanks,
Rasesh

RE: [PATCH 04/14] bna: Add Multiple Tx Queue Support

[Rasesh Mody]

>From: Ben Hutchings [mailto:bhutchings@solarflare.com]
>Sent: Tuesday, August 16, 2011 2:49 PM
>
>On Tue, 2011-08-16 at 14:19 -0700, Rasesh Mody wrote:
>> Change details:
>>  - Add macros bna_prio_allow, bna_default_nw_prio, bna_iscsi_prio,
>>    bna_is_iscsi_over_cee
>>  - Added support for multipe Tx queues with a separate iSCSI Tx queue
>based
>>    on the default value of iSCSI port number. The feature is supported
>based
>>    on the underlying hardware and enabled for DCB (CEE) mode only.
>>  - Allocate multiple TxQ resource in netdev
>>  - Implement bnad_tx_select_queue() which enables the correct
>selection of
>>    TxQ Id (and tcb). This function is called either by the kernel to
>channel
>>    packets to the right TxQ
>>  - bnad_tx_select_queue() returns priority, while only a few packets
>during
>>    transition could have wrong priority, all will be associated with a
>valid
>>    non-NULL tcb.
>>  - Implement bnad_iscsi_tcb_get() and BNAD_IS_ISCSI_PKT() for iSCSI
>packet
>>    inspection and retrieval of tcb corresponding to the iSCSI
>priority.
>>  - Construction of priority indirection table to be used by bnad to
>direct
>>    packets into TxQs
>[...]
>
>You probably should implement TX priority classes through the
>ndo_setup_tc operation, not ndo_select_queue.

The reason we went with ndo_select_queue is due to the need for mapping
iSCSI packets (TCP port 3260) to a priority derived from DCB configuration.
Here the iSCSI packets may not have any tc defined in the packet header. 
Thanks,
Rasesh

RE: [PATCH 04/14] bna: Add Multiple Tx Queue Support

[Ben Hutchings]

On Tue, 2011-08-16 at 16:32 -0700, Rasesh Mody wrote:
> >From: Ben Hutchings [mailto:bhutchings@solarflare.com]
> >Sent: Tuesday, August 16, 2011 2:49 PM
> >
> >On Tue, 2011-08-16 at 14:19 -0700, Rasesh Mody wrote:
> >> Change details:
> >>  - Add macros bna_prio_allow, bna_default_nw_prio, bna_iscsi_prio,
> >>    bna_is_iscsi_over_cee
> >>  - Added support for multipe Tx queues with a separate iSCSI Tx queue
> >based
> >>    on the default value of iSCSI port number. The feature is supported
> >based
> >>    on the underlying hardware and enabled for DCB (CEE) mode only.
> >>  - Allocate multiple TxQ resource in netdev
> >>  - Implement bnad_tx_select_queue() which enables the correct
> >selection of
> >>    TxQ Id (and tcb). This function is called either by the kernel to
> >channel
> >>    packets to the right TxQ
> >>  - bnad_tx_select_queue() returns priority, while only a few packets
> >during
> >>    transition could have wrong priority, all will be associated with a
> >valid
> >>    non-NULL tcb.
> >>  - Implement bnad_iscsi_tcb_get() and BNAD_IS_ISCSI_PKT() for iSCSI
> >packet
> >>    inspection and retrieval of tcb corresponding to the iSCSI
> >priority.
> >>  - Construction of priority indirection table to be used by bnad to
> >direct
> >>    packets into TxQs
> >[...]
> >
> >You probably should implement TX priority classes through the
> >ndo_setup_tc operation, not ndo_select_queue.
> 
> The reason we went with ndo_select_queue is due to the need for mapping
> iSCSI packets (TCP port 3260) to a priority derived from DCB configuration.
> Here the iSCSI packets may not have any tc defined in the packet header. 

There is an skb priority, which is derived from the socket priority
option (SO_PRIORITY).  If you implement ndo_setup_tc then the networking
core will take care of mapping the skb priority onto a different queue
(or range of queues).

I don't know whether the socket priority option is easily configurable
for the existing iSCSI implementations.  But looking at port numbers
really doesn't seem like the right way to do this.

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: [PATCH 04/14] bna: Add Multiple Tx Queue Support

[John Fastabend]

On 8/16/2011 4:43 PM, Ben Hutchings wrote:
> On Tue, 2011-08-16 at 16:32 -0700, Rasesh Mody wrote:
>>> From: Ben Hutchings [mailto:bhutchings@solarflare.com]
>>> Sent: Tuesday, August 16, 2011 2:49 PM
>>>
>>> On Tue, 2011-08-16 at 14:19 -0700, Rasesh Mody wrote:
>>>> Change details:
>>>>  - Add macros bna_prio_allow, bna_default_nw_prio, bna_iscsi_prio,
>>>>    bna_is_iscsi_over_cee
>>>>  - Added support for multipe Tx queues with a separate iSCSI Tx queue
>>> based
>>>>    on the default value of iSCSI port number. The feature is supported
>>> based
>>>>    on the underlying hardware and enabled for DCB (CEE) mode only.
>>>>  - Allocate multiple TxQ resource in netdev
>>>>  - Implement bnad_tx_select_queue() which enables the correct
>>> selection of
>>>>    TxQ Id (and tcb). This function is called either by the kernel to
>>> channel
>>>>    packets to the right TxQ
>>>>  - bnad_tx_select_queue() returns priority, while only a few packets
>>> during
>>>>    transition could have wrong priority, all will be associated with a
>>> valid
>>>>    non-NULL tcb.
>>>>  - Implement bnad_iscsi_tcb_get() and BNAD_IS_ISCSI_PKT() for iSCSI
>>> packet
>>>>    inspection and retrieval of tcb corresponding to the iSCSI
>>> priority.
>>>>  - Construction of priority indirection table to be used by bnad to
>>> direct
>>>>    packets into TxQs
>>> [...]
>>>
>>> You probably should implement TX priority classes through the
>>> ndo_setup_tc operation, not ndo_select_queue.
>>
>> The reason we went with ndo_select_queue is due to the need for mapping
>> iSCSI packets (TCP port 3260) to a priority derived from DCB configuration.
>> Here the iSCSI packets may not have any tc defined in the packet header. 
> 
> There is an skb priority, which is derived from the socket priority
> option (SO_PRIORITY).  If you implement ndo_setup_tc then the networking
> core will take care of mapping the skb priority onto a different queue
> (or range of queues).
> 
> I don't know whether the socket priority option is easily configurable
> for the existing iSCSI implementations.  But looking at port numbers
> really doesn't seem like the right way to do this.
> 
> Ben.
> 

At least open-iscsi supports DCB by listening to the RTNLGRP_DCB for events.
These are generated with dcbnl_cee_notify and dcbnl_ieee_notify. To support
this in your driver you need to call dcb_setapp() or dcb_ieee_setapp() to
add the application data and follow this with the appropriate notify call.
Then assuming you do the necessary tc setup work the stack will handle the
mapping of priority to queues.

John.

RE: [PATCH 04/14] bna: Add Multiple Tx Queue Support

[Rasesh Mody]

>From: John Fastabend [mailto:john.r.fastabend@intel.com]
>Sent: Tuesday, August 16, 2011 5:18 PM
>
>On 8/16/2011 4:43 PM, Ben Hutchings wrote:
>> On Tue, 2011-08-16 at 16:32 -0700, Rasesh Mody wrote:
>>>> From: Ben Hutchings [mailto:bhutchings@solarflare.com]
>>>> Sent: Tuesday, August 16, 2011 2:49 PM
>>>>
>>>> On Tue, 2011-08-16 at 14:19 -0700, Rasesh Mody wrote:
>>>>> Change details:
>>>>>  - Add macros bna_prio_allow, bna_default_nw_prio, bna_iscsi_prio,
>>>>>    bna_is_iscsi_over_cee
>>>>>  - Added support for multipe Tx queues with a separate iSCSI Tx
>queue
>>>> based
>>>>>    on the default value of iSCSI port number. The feature is
>supported
>>>> based
>>>>>    on the underlying hardware and enabled for DCB (CEE) mode only.
>>>>>  - Allocate multiple TxQ resource in netdev
>>>>>  - Implement bnad_tx_select_queue() which enables the correct
>>>> selection of
>>>>>    TxQ Id (and tcb). This function is called either by the kernel
>to
>>>> channel
>>>>>    packets to the right TxQ
>>>>>  - bnad_tx_select_queue() returns priority, while only a few
>packets
>>>> during
>>>>>    transition could have wrong priority, all will be associated
>with a
>>>> valid
>>>>>    non-NULL tcb.
>>>>>  - Implement bnad_iscsi_tcb_get() and BNAD_IS_ISCSI_PKT() for iSCSI
>>>> packet
>>>>>    inspection and retrieval of tcb corresponding to the iSCSI
>>>> priority.
>>>>>  - Construction of priority indirection table to be used by bnad to
>>>> direct
>>>>>    packets into TxQs
>>>> [...]
>>>>
>>>> You probably should implement TX priority classes through the
>>>> ndo_setup_tc operation, not ndo_select_queue.
>>>
>>> The reason we went with ndo_select_queue is due to the need for
>mapping
>>> iSCSI packets (TCP port 3260) to a priority derived from DCB
>configuration.
>>> Here the iSCSI packets may not have any tc defined in the packet
>header.
>>
>> There is an skb priority, which is derived from the socket priority
>> option (SO_PRIORITY).  If you implement ndo_setup_tc then the
>networking
>> core will take care of mapping the skb priority onto a different queue
>> (or range of queues).
>>
>> I don't know whether the socket priority option is easily configurable
>> for the existing iSCSI implementations.  But looking at port numbers
>> really doesn't seem like the right way to do this.
>>
>> Ben.
>>
>
>At least open-iscsi supports DCB by listening to the RTNLGRP_DCB for
>events.
>These are generated with dcbnl_cee_notify and dcbnl_ieee_notify. To
>support
>this in your driver you need to call dcb_setapp() or dcb_ieee_setapp()
>to
>add the application data and follow this with the appropriate notify
>call.
>Then assuming you do the necessary tc setup work the stack will handle
>the
>mapping of priority to queues.

This is how iSCSI over DCB feature is expected to works in BNA driver:-
FW running in the BNA adapter implements the DCB protocol. It learns the
iSCSI priority from the switch through iSCSI TLV exchange. BNA driver
extracts the iSCSI priority from the FW that needs to be used for iSCSI
packets. For every outgoing packet, BNA driver does a TCP header
inspection to classify iSCSI packet and attach right 802.1q priority &
send it on the correct TX queue.

This is expected to work with iSCSI applications that do not configure the
priority with SO_PRIORITY - here the iSCSI priority configuration actually
comes from the switch to the adapter.

The goal of this iSCSI priority is:
a) adapter applies prioritized scheduling for packets in its egress - to
guarantee minimum bandwidth as per ETS
b) packets are tagged with right priority so that switch can also identify
and guarantee BW on its egress.

Hope this explains.

Thanks,
Rasesh

Re: [PATCH 04/14] bna: Add Multiple Tx Queue Support

[John Fastabend]

On 8/16/2011 7:14 PM, Rasesh Mody wrote:
>> From: John Fastabend [mailto:john.r.fastabend@intel.com]
>> Sent: Tuesday, August 16, 2011 5:18 PM
>>
>> On 8/16/2011 4:43 PM, Ben Hutchings wrote:
>>> On Tue, 2011-08-16 at 16:32 -0700, Rasesh Mody wrote:
>>>>> From: Ben Hutchings [mailto:bhutchings@solarflare.com]
>>>>> Sent: Tuesday, August 16, 2011 2:49 PM
>>>>>
>>>>> On Tue, 2011-08-16 at 14:19 -0700, Rasesh Mody wrote:
>>>>>> Change details:
>>>>>>  - Add macros bna_prio_allow, bna_default_nw_prio, bna_iscsi_prio,
>>>>>>    bna_is_iscsi_over_cee
>>>>>>  - Added support for multipe Tx queues with a separate iSCSI Tx
>> queue
>>>>> based
>>>>>>    on the default value of iSCSI port number. The feature is
>> supported
>>>>> based
>>>>>>    on the underlying hardware and enabled for DCB (CEE) mode only.
>>>>>>  - Allocate multiple TxQ resource in netdev
>>>>>>  - Implement bnad_tx_select_queue() which enables the correct
>>>>> selection of
>>>>>>    TxQ Id (and tcb). This function is called either by the kernel
>> to
>>>>> channel
>>>>>>    packets to the right TxQ
>>>>>>  - bnad_tx_select_queue() returns priority, while only a few
>> packets
>>>>> during
>>>>>>    transition could have wrong priority, all will be associated
>> with a
>>>>> valid
>>>>>>    non-NULL tcb.
>>>>>>  - Implement bnad_iscsi_tcb_get() and BNAD_IS_ISCSI_PKT() for iSCSI
>>>>> packet
>>>>>>    inspection and retrieval of tcb corresponding to the iSCSI
>>>>> priority.
>>>>>>  - Construction of priority indirection table to be used by bnad to
>>>>> direct
>>>>>>    packets into TxQs
>>>>> [...]
>>>>>
>>>>> You probably should implement TX priority classes through the
>>>>> ndo_setup_tc operation, not ndo_select_queue.
>>>>
>>>> The reason we went with ndo_select_queue is due to the need for
>> mapping
>>>> iSCSI packets (TCP port 3260) to a priority derived from DCB
>> configuration.
>>>> Here the iSCSI packets may not have any tc defined in the packet
>> header.
>>>
>>> There is an skb priority, which is derived from the socket priority
>>> option (SO_PRIORITY).  If you implement ndo_setup_tc then the
>> networking
>>> core will take care of mapping the skb priority onto a different queue
>>> (or range of queues).
>>>
>>> I don't know whether the socket priority option is easily configurable
>>> for the existing iSCSI implementations.  But looking at port numbers
>>> really doesn't seem like the right way to do this.
>>>
>>> Ben.
>>>
>>
>> At least open-iscsi supports DCB by listening to the RTNLGRP_DCB for
>> events.
>> These are generated with dcbnl_cee_notify and dcbnl_ieee_notify. To
>> support
>> this in your driver you need to call dcb_setapp() or dcb_ieee_setapp()
>> to
>> add the application data and follow this with the appropriate notify
>> call.
>> Then assuming you do the necessary tc setup work the stack will handle
>> the
>> mapping of priority to queues.
> 
> This is how iSCSI over DCB feature is expected to works in BNA driver:-
> FW running in the BNA adapter implements the DCB protocol. It learns the
> iSCSI priority from the switch through iSCSI TLV exchange. BNA driver
> extracts the iSCSI priority from the FW that needs to be used for iSCSI
> packets.

Up to here this is fine. What I was suggesting was to then use the
dcb_setapp() routines to program the iSCSI TLV and generate an event
so any user space applications listening to DCB events could handle
this. It would also be nice if your driver notified user space of
any PG or PFC changes as well. Then management agents (lldpad) could
use the DCB attributes to make policy decisions.

> For every outgoing packet, BNA driver does a TCP header
> inspection to classify iSCSI packet and attach right 802.1q priority &
> send it on the correct TX queue.
> 
> This is expected to work with iSCSI applications that do not configure the
> priority with SO_PRIORITY - here the iSCSI priority configuration actually
> comes from the switch to the adapter.
> 

Although this works I don't think it is optimal for a few reason. Your
L2 driver is inspecting TCP frames which is bad layering IMHO. The
iSCSI port number is hard coded into the driver so it will only work
with the well known port number. Also it adds more driver specific
behavior into select_queue() where I think the trend is to try to
use select_queue() less not more.

To address iSCSI applications that do not configure the priority
we could either work on adding the DCB hooks needed in those
applications. Or look at adding a hook in the qdisc layer to
to do what your select_queue() hook does here. I started prototyping
this awhile ago but this requires running the packet classifiers
and associated actions before picking a queue assuming you want
to use mq or mqprio. I hope to get back to this soon I have some
more details to flush out wrt this and need to run it by some other
folks to make sure its a sane idea.


> The goal of this iSCSI priority is:
> a) adapter applies prioritized scheduling for packets in its egress - to
> guarantee minimum bandwidth as per ETS
> b) packets are tagged with right priority so that switch can also identify
> and guarantee BW on its egress.

Correct. This is the same for other drivers that support DCB and
use tc_setup as Ben suggested. See the bnx2x driver for an example
that also uses a FW based LLDP engine and does this.

> 
> Hope this explains.

I think the one valid item is the unsupported applications but
hopefully that can be addressed. Thanks for the details.

John.

> 
> Thanks,
> Rasesh

RE: [PATCH 04/14] bna: Add Multiple Tx Queue Support

[Rasesh Mody]

>From: John Fastabend [mailto:john.r.fastabend@intel.com]
>Sent: Tuesday, August 16, 2011 10:48 PM
>
>On 8/16/2011 7:14 PM, Rasesh Mody wrote:
>>> From: John Fastabend [mailto:john.r.fastabend@intel.com]
>>> Sent: Tuesday, August 16, 2011 5:18 PM
>>>
>>> On 8/16/2011 4:43 PM, Ben Hutchings wrote:
>>>> On Tue, 2011-08-16 at 16:32 -0700, Rasesh Mody wrote:
>>>>>> From: Ben Hutchings [mailto:bhutchings@solarflare.com]
>>>>>> Sent: Tuesday, August 16, 2011 2:49 PM
>>>>>>
>>>>>> On Tue, 2011-08-16 at 14:19 -0700, Rasesh Mody wrote:
>>>>>>> Change details:
>>>>>>>  - Add macros bna_prio_allow, bna_default_nw_prio,
>bna_iscsi_prio,
>>>>>>>    bna_is_iscsi_over_cee
>>>>>>>  - Added support for multipe Tx queues with a separate iSCSI Tx
>>> queue
>>>>>> based
>>>>>>>    on the default value of iSCSI port number. The feature is
>>> supported
>>>>>> based
>>>>>>>    on the underlying hardware and enabled for DCB (CEE) mode
>only.
>>>>>>>  - Allocate multiple TxQ resource in netdev
>>>>>>>  - Implement bnad_tx_select_queue() which enables the correct
>>>>>> selection of
>>>>>>>    TxQ Id (and tcb). This function is called either by the kernel
>>> to
>>>>>> channel
>>>>>>>    packets to the right TxQ
>>>>>>>  - bnad_tx_select_queue() returns priority, while only a few
>>> packets
>>>>>> during
>>>>>>>    transition could have wrong priority, all will be associated
>>> with a
>>>>>> valid
>>>>>>>    non-NULL tcb.
>>>>>>>  - Implement bnad_iscsi_tcb_get() and BNAD_IS_ISCSI_PKT() for
>iSCSI
>>>>>> packet
>>>>>>>    inspection and retrieval of tcb corresponding to the iSCSI
>>>>>> priority.
>>>>>>>  - Construction of priority indirection table to be used by bnad
>to
>>>>>> direct
>>>>>>>    packets into TxQs
>>>>>> [...]
>>>>>>
>>>>>> You probably should implement TX priority classes through the
>>>>>> ndo_setup_tc operation, not ndo_select_queue.
>>>>>
>>>>> The reason we went with ndo_select_queue is due to the need for
>>> mapping
>>>>> iSCSI packets (TCP port 3260) to a priority derived from DCB
>>> configuration.
>>>>> Here the iSCSI packets may not have any tc defined in the packet
>>> header.
>>>>
>>>> There is an skb priority, which is derived from the socket priority
>>>> option (SO_PRIORITY).  If you implement ndo_setup_tc then the
>>> networking
>>>> core will take care of mapping the skb priority onto a different
>queue
>>>> (or range of queues).
>>>>
>>>> I don't know whether the socket priority option is easily
>configurable
>>>> for the existing iSCSI implementations.  But looking at port numbers
>>>> really doesn't seem like the right way to do this.
>>>>
>>>> Ben.
>>>>
>>>
>>> At least open-iscsi supports DCB by listening to the RTNLGRP_DCB for
>>> events.
>>> These are generated with dcbnl_cee_notify and dcbnl_ieee_notify. To
>>> support
>>> this in your driver you need to call dcb_setapp() or
>dcb_ieee_setapp()
>>> to
>>> add the application data and follow this with the appropriate notify
>>> call.
>>> Then assuming you do the necessary tc setup work the stack will
>handle
>>> the
>>> mapping of priority to queues.
>>
>> This is how iSCSI over DCB feature is expected to works in BNA
>driver:-
>> FW running in the BNA adapter implements the DCB protocol. It learns
>the
>> iSCSI priority from the switch through iSCSI TLV exchange. BNA driver
>> extracts the iSCSI priority from the FW that needs to be used for
>iSCSI
>> packets.
>
>Up to here this is fine. What I was suggesting was to then use the
>dcb_setapp() routines to program the iSCSI TLV and generate an event
>so any user space applications listening to DCB events could handle
>this. It would also be nice if your driver notified user space of
>any PG or PFC changes as well. Then management agents (lldpad) could
>use the DCB attributes to make policy decisions.
>
>> For every outgoing packet, BNA driver does a TCP header
>> inspection to classify iSCSI packet and attach right 802.1q priority &
>> send it on the correct TX queue.
>>
>> This is expected to work with iSCSI applications that do not configure
>the
>> priority with SO_PRIORITY - here the iSCSI priority configuration
>actually
>> comes from the switch to the adapter.
>>
>
>Although this works I don't think it is optimal for a few reason. Your
>L2 driver is inspecting TCP frames which is bad layering IMHO. The
>iSCSI port number is hard coded into the driver so it will only work
>with the well known port number. Also it adds more driver specific
>behavior into select_queue() where I think the trend is to try to
>use select_queue() less not more.
>
>To address iSCSI applications that do not configure the priority
>we could either work on adding the DCB hooks needed in those
>applications. Or look at adding a hook in the qdisc layer to
>to do what your select_queue() hook does here. I started prototyping
>this awhile ago but this requires running the packet classifiers
>and associated actions before picking a queue assuming you want
>to use mq or mqprio. I hope to get back to this soon I have some
>more details to flush out wrt this and need to run it by some other
>folks to make sure its a sane idea.
>
>
>> The goal of this iSCSI priority is:
>> a) adapter applies prioritized scheduling for packets in its egress -
>to
>> guarantee minimum bandwidth as per ETS
>> b) packets are tagged with right priority so that switch can also
>identify
>> and guarantee BW on its egress.
>
>Correct. This is the same for other drivers that support DCB and
>use tc_setup as Ben suggested. See the bnx2x driver for an example
>that also uses a FW based LLDP engine and does this.
>
>>
>> Hope this explains.
>
>I think the one valid item is the unsupported applications but
>hopefully that can be addressed. Thanks for the details.

Can we go with this approach of select_queue() for now till we have DCB hooks added to the iSCSI applications or the qdisc layer? We will address this to use setup_tc() as the framework becomes available.

Thanks,
Rasesh