* [PATCH v2 00/11] enic counter fixes and Tx optimization
@ 2016-05-24 6:32 John Daley
2016-05-24 6:32 ` [PATCH v2 01/11] enic: fix Rx drop counters John Daley
` (11 more replies)
0 siblings, 12 replies; 31+ messages in thread
From: John Daley @ 2016-05-24 6:32 UTC (permalink / raw)
To: dev; +Cc: John Daley
The first 3 patches are related to drop counters. The remaining
patches make up a refactoring, cleanup and optimization of the Tx path.
Changes since v1 are:
- subject line fixups after running check-git-log.sh.
- Errors reported from patchworks fixed. Note: ./scripts/checkpatches.pl was
run locally before upstreaming v1 and the same errors were not caught.
The local host had perl version v5.16.3.
John Daley (11):
enic: fix Rx drop counters
enic: drop bad packets and remove unused Rx error flag
enic: count truncated packets
enic: put Tx and Rx functions into same file
enic: remove some unused functions in Tx path
enic: streamline mbuf handling in Tx path
enic: use Tx completion messages instead of descriptors
enic: refactor Tx mbuf recycling
enic: optimize the Tx function
enic: remove unused files and functions and variables
enic: add an enic assert macro
drivers/net/enic/Makefile | 2 +-
drivers/net/enic/base/enic_vnic_wq.h | 79 ------
drivers/net/enic/base/vnic_cq.h | 44 ----
drivers/net/enic/base/vnic_wq.c | 80 ++----
drivers/net/enic/base/vnic_wq.h | 118 ++-------
drivers/net/enic/enic.h | 47 +++-
drivers/net/enic/enic_ethdev.c | 67 +----
drivers/net/enic/enic_main.c | 156 +++++-------
drivers/net/enic/enic_res.h | 80 +-----
drivers/net/enic/enic_rx.c | 351 -------------------------
drivers/net/enic/enic_rxtx.c | 482 +++++++++++++++++++++++++++++++++++
11 files changed, 634 insertions(+), 872 deletions(-)
delete mode 100644 drivers/net/enic/base/enic_vnic_wq.h
delete mode 100644 drivers/net/enic/enic_rx.c
create mode 100644 drivers/net/enic/enic_rxtx.c
--
2.7.0
^ permalink raw reply [flat|nested] 31+ messages in thread* [PATCH v2 01/11] enic: fix Rx drop counters 2016-05-24 6:32 [PATCH v2 00/11] enic counter fixes and Tx optimization John Daley @ 2016-05-24 6:32 ` John Daley 2016-05-24 6:32 ` [PATCH v2 02/11] enic: drop bad packets and remove unused Rx error flag John Daley ` (10 subsequent siblings) 11 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-05-24 6:32 UTC (permalink / raw) To: dev; +Cc: John Daley rx_no_bufs is a hardware counter of packets dropped on the interface due to no host buffers and should be used to update r_stats->imissed counter instead of rx_nombuf. Include rx_drop in ierrors. rx_drop is incremented if packets arrive when the receive queue is disabled. Add a structure and functions for initializing and clearing software counters. Add count of Rx mbuf allocation failures (rx_nombuf) as the first counter. Fixes: fefed3d1e62c ("enic: new driver") Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/enic.h | 7 +++++++ drivers/net/enic/enic_main.c | 24 +++++++++++++++++++++--- drivers/net/enic/enic_rx.c | 5 +---- 3 files changed, 29 insertions(+), 7 deletions(-) diff --git a/drivers/net/enic/enic.h b/drivers/net/enic/enic.h index 09f3853..584d49b 100644 --- a/drivers/net/enic/enic.h +++ b/drivers/net/enic/enic.h @@ -91,6 +91,10 @@ struct enic_fdir { struct enic_fdir_node *nodes[ENICPMD_FDIR_MAX]; }; +struct enic_soft_stats { + rte_atomic64_t rx_nombuf; +}; + /* Per-instance private data structure */ struct enic { struct enic *next; @@ -133,6 +137,9 @@ struct enic { /* interrupt resource */ struct vnic_intr intr; unsigned int intr_count; + + /* software counters */ + struct enic_soft_stats soft_stats; }; static inline unsigned int enic_cq_rq(__rte_unused struct enic *enic, unsigned int rq) diff --git a/drivers/net/enic/enic_main.c b/drivers/net/enic/enic_main.c index bbbe660..c002ef3 100644 --- a/drivers/net/enic/enic_main.c +++ b/drivers/net/enic/enic_main.c @@ -211,15 +211,30 @@ void enic_send_pkt(struct enic *enic, struct vnic_wq *wq, 0 /*wrid*/); } +static void enic_clear_soft_stats(struct enic *enic) +{ + struct enic_soft_stats *soft_stats = &enic->soft_stats; + rte_atomic64_clear(&soft_stats->rx_nombuf); +} + +static void enic_init_soft_stats(struct enic *enic) +{ + struct enic_soft_stats *soft_stats = &enic->soft_stats; + rte_atomic64_init(&soft_stats->rx_nombuf); + enic_clear_soft_stats(enic); +} + void enic_dev_stats_clear(struct enic *enic) { if (vnic_dev_stats_clear(enic->vdev)) dev_err(enic, "Error in clearing stats\n"); + enic_clear_soft_stats(enic); } void enic_dev_stats_get(struct enic *enic, struct rte_eth_stats *r_stats) { struct vnic_stats *stats; + struct enic_soft_stats *soft_stats; if (vnic_dev_stats_dump(enic->vdev, &stats)) { dev_err(enic, "Error in getting stats\n"); @@ -232,12 +247,13 @@ void enic_dev_stats_get(struct enic *enic, struct rte_eth_stats *r_stats) r_stats->ibytes = stats->rx.rx_bytes_ok; r_stats->obytes = stats->tx.tx_bytes_ok; - r_stats->ierrors = stats->rx.rx_errors; + r_stats->ierrors = stats->rx.rx_errors + stats->rx.rx_drop; r_stats->oerrors = stats->tx.tx_errors; - r_stats->imissed = stats->rx.rx_drop; + r_stats->imissed = stats->rx.rx_no_bufs; - r_stats->rx_nombuf = stats->rx.rx_no_bufs; + soft_stats = &enic->soft_stats; + r_stats->rx_nombuf = rte_atomic64_read(&soft_stats->rx_nombuf); } void enic_del_mac_address(struct enic *enic) @@ -795,6 +811,8 @@ int enic_setup_finish(struct enic *enic) { int ret; + enic_init_soft_stats(enic); + ret = enic_set_rss_nic_cfg(enic); if (ret) { dev_err(enic, "Failed to config nic, aborting.\n"); diff --git a/drivers/net/enic/enic_rx.c b/drivers/net/enic/enic_rx.c index f92f6bc..89c62ce 100644 --- a/drivers/net/enic/enic_rx.c +++ b/drivers/net/enic/enic_rx.c @@ -275,10 +275,7 @@ enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, /* allocate a new mbuf */ nmb = rte_mbuf_raw_alloc(rq->mp); if (nmb == NULL) { - dev_err(enic, "RX mbuf alloc failed port=%u qid=%u", - enic->port_id, (unsigned)rq->index); - rte_eth_devices[enic->port_id]. - data->rx_mbuf_alloc_failed++; + rte_atomic64_inc(&enic->soft_stats.rx_nombuf); break; } -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v2 02/11] enic: drop bad packets and remove unused Rx error flag 2016-05-24 6:32 [PATCH v2 00/11] enic counter fixes and Tx optimization John Daley 2016-05-24 6:32 ` [PATCH v2 01/11] enic: fix Rx drop counters John Daley @ 2016-05-24 6:32 ` John Daley 2016-05-24 6:32 ` [PATCH v2 03/11] enic: count truncated packets John Daley ` (9 subsequent siblings) 11 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-05-24 6:32 UTC (permalink / raw) To: dev; +Cc: John Daley, Olivier Matz Following the discussions from: http://dpdk.org/ml/archives/dev/2015-July/021721.html http://dpdk.org/ml/archives/dev/2016-April/038143.html Remove the unused flag from enic driver. Also, the enic driver is modified to drop bad packets. Signed-off-by: Olivier Matz <olivier.matz@6wind.com> Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/enic_rx.c | 39 +++++++++++++++++---------------------- 1 file changed, 17 insertions(+), 22 deletions(-) diff --git a/drivers/net/enic/enic_rx.c b/drivers/net/enic/enic_rx.c index 89c62ce..c72a80a 100644 --- a/drivers/net/enic/enic_rx.c +++ b/drivers/net/enic/enic_rx.c @@ -134,20 +134,15 @@ enic_cq_rx_desc_n_bytes(struct cq_desc *cqd) } static inline uint8_t -enic_cq_rx_to_pkt_err_flags(struct cq_desc *cqd, uint64_t *pkt_err_flags_out) +enic_cq_rx_check_err(struct cq_desc *cqd) { struct cq_enet_rq_desc *cqrd = (struct cq_enet_rq_desc *)cqd; uint16_t bwflags; - int ret = 0; - uint64_t pkt_err_flags = 0; bwflags = enic_cq_rx_desc_bwflags(cqrd); - if (unlikely(enic_cq_rx_desc_packet_error(bwflags))) { - pkt_err_flags = PKT_RX_MAC_ERR; - ret = 1; - } - *pkt_err_flags_out = pkt_err_flags; - return ret; + if (unlikely(enic_cq_rx_desc_packet_error(bwflags))) + return 1; + return 0; } /* @@ -243,7 +238,7 @@ enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, struct enic *enic = vnic_dev_priv(rq->vdev); unsigned int rx_id; struct rte_mbuf *nmb, *rxmb; - uint16_t nb_rx = 0; + uint16_t nb_rx = 0, nb_err = 0; uint16_t nb_hold; struct vnic_cq *cq; volatile struct cq_desc *cqd_ptr; @@ -259,7 +254,6 @@ enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, volatile struct rq_enet_desc *rqd_ptr; dma_addr_t dma_addr; struct cq_desc cqd; - uint64_t ol_err_flags; uint8_t packet_error; /* Check for pkts available */ @@ -280,7 +274,7 @@ enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, } /* A packet error means descriptor and data are untrusted */ - packet_error = enic_cq_rx_to_pkt_err_flags(&cqd, &ol_err_flags); + packet_error = enic_cq_rx_check_err(&cqd); /* Get the mbuf to return and replace with one just allocated */ rxmb = rq->mbuf_ring[rx_id]; @@ -307,20 +301,21 @@ enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, rqd_ptr->length_type = cpu_to_le16(nmb->buf_len - RTE_PKTMBUF_HEADROOM); + /* Drop incoming bad packet */ + if (unlikely(packet_error)) { + rte_pktmbuf_free(rxmb); + nb_err++; + continue; + } + /* Fill in the rest of the mbuf */ rxmb->data_off = RTE_PKTMBUF_HEADROOM; rxmb->nb_segs = 1; rxmb->next = NULL; rxmb->port = enic->port_id; - if (!packet_error) { - rxmb->pkt_len = enic_cq_rx_desc_n_bytes(&cqd); - rxmb->packet_type = enic_cq_rx_flags_to_pkt_type(&cqd); - enic_cq_rx_to_pkt_flags(&cqd, rxmb); - } else { - rxmb->pkt_len = 0; - rxmb->packet_type = 0; - rxmb->ol_flags = 0; - } + rxmb->pkt_len = enic_cq_rx_desc_n_bytes(&cqd); + rxmb->packet_type = enic_cq_rx_flags_to_pkt_type(&cqd); + enic_cq_rx_to_pkt_flags(&cqd, rxmb); rxmb->data_len = rxmb->pkt_len; /* prefetch mbuf data for caller */ @@ -331,7 +326,7 @@ enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, rx_pkts[nb_rx++] = rxmb; } - nb_hold += nb_rx; + nb_hold += nb_rx + nb_err; cq->to_clean = rx_id; if (nb_hold > rq->rx_free_thresh) { -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v2 03/11] enic: count truncated packets 2016-05-24 6:32 [PATCH v2 00/11] enic counter fixes and Tx optimization John Daley 2016-05-24 6:32 ` [PATCH v2 01/11] enic: fix Rx drop counters John Daley 2016-05-24 6:32 ` [PATCH v2 02/11] enic: drop bad packets and remove unused Rx error flag John Daley @ 2016-05-24 6:32 ` John Daley 2016-05-24 6:32 ` [PATCH v2 04/11] enic: put Tx and Rx functions into same file John Daley ` (8 subsequent siblings) 11 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-05-24 6:32 UTC (permalink / raw) To: dev; +Cc: John Daley, Nelson Escobar Truncated packets occur on enic if an mbuf is not big enough to receive it or there aren't enough mbufs if rx scatter is in use. They show up as error packets but unlike other error packets (like packets bad FCS) there are no nic drop counts incremented for them. Truncated packets are calculated by subtracting hardware errors from software errors. Note: this causes transient inaccuracies in the ipackets count. Also, the length of truncated packets are counted in ibytes even though truncated packets are dropped which can make ibytes be slightly higher than it should be. Signed-off-by: Nelson Escobar <neescoba@cisco.com> Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/enic.h | 1 + drivers/net/enic/enic_main.c | 21 +++++++++++++++++---- 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/drivers/net/enic/enic.h b/drivers/net/enic/enic.h index 584d49b..9b6f349 100644 --- a/drivers/net/enic/enic.h +++ b/drivers/net/enic/enic.h @@ -93,6 +93,7 @@ struct enic_fdir { struct enic_soft_stats { rte_atomic64_t rx_nombuf; + rte_atomic64_t rx_packet_errors; }; /* Per-instance private data structure */ diff --git a/drivers/net/enic/enic_main.c b/drivers/net/enic/enic_main.c index c002ef3..e4ccc7d 100644 --- a/drivers/net/enic/enic_main.c +++ b/drivers/net/enic/enic_main.c @@ -215,12 +215,14 @@ static void enic_clear_soft_stats(struct enic *enic) { struct enic_soft_stats *soft_stats = &enic->soft_stats; rte_atomic64_clear(&soft_stats->rx_nombuf); + rte_atomic64_clear(&soft_stats->rx_packet_errors); } static void enic_init_soft_stats(struct enic *enic) { struct enic_soft_stats *soft_stats = &enic->soft_stats; rte_atomic64_init(&soft_stats->rx_nombuf); + rte_atomic64_init(&soft_stats->rx_packet_errors); enic_clear_soft_stats(enic); } @@ -234,14 +236,26 @@ void enic_dev_stats_clear(struct enic *enic) void enic_dev_stats_get(struct enic *enic, struct rte_eth_stats *r_stats) { struct vnic_stats *stats; - struct enic_soft_stats *soft_stats; + struct enic_soft_stats *soft_stats = &enic->soft_stats; + int64_t rx_truncated; + uint64_t rx_packet_errors; if (vnic_dev_stats_dump(enic->vdev, &stats)) { dev_err(enic, "Error in getting stats\n"); return; } - r_stats->ipackets = stats->rx.rx_frames_ok; + /* The number of truncated packets can only be calculated by + * subtracting a hardware counter from error packets received by + * the driver. Note: this causes transient inaccuracies in the + * ipackets count. Also, the length of truncated packets are + * counted in ibytes even though truncated packets are dropped + * which can make ibytes be slightly higher than it should be. + */ + rx_packet_errors = rte_atomic64_read(&soft_stats->rx_packet_errors); + rx_truncated = rx_packet_errors - stats->rx.rx_errors; + + r_stats->ipackets = stats->rx.rx_frames_ok - rx_truncated; r_stats->opackets = stats->tx.tx_frames_ok; r_stats->ibytes = stats->rx.rx_bytes_ok; @@ -250,9 +264,8 @@ void enic_dev_stats_get(struct enic *enic, struct rte_eth_stats *r_stats) r_stats->ierrors = stats->rx.rx_errors + stats->rx.rx_drop; r_stats->oerrors = stats->tx.tx_errors; - r_stats->imissed = stats->rx.rx_no_bufs; + r_stats->imissed = stats->rx.rx_no_bufs + rx_truncated; - soft_stats = &enic->soft_stats; r_stats->rx_nombuf = rte_atomic64_read(&soft_stats->rx_nombuf); } -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v2 04/11] enic: put Tx and Rx functions into same file 2016-05-24 6:32 [PATCH v2 00/11] enic counter fixes and Tx optimization John Daley ` (2 preceding siblings ...) 2016-05-24 6:32 ` [PATCH v2 03/11] enic: count truncated packets John Daley @ 2016-05-24 6:32 ` John Daley 2016-05-24 6:32 ` [PATCH v2 05/11] enic: remove some unused functions in Tx path John Daley ` (7 subsequent siblings) 11 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-05-24 6:32 UTC (permalink / raw) To: dev; +Cc: John Daley Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/Makefile | 2 +- drivers/net/enic/enic.h | 3 + drivers/net/enic/enic_ethdev.c | 65 ------ drivers/net/enic/enic_main.c | 82 +------ drivers/net/enic/enic_rx.c | 343 ----------------------------- drivers/net/enic/enic_rxtx.c | 481 +++++++++++++++++++++++++++++++++++++++++ 6 files changed, 486 insertions(+), 490 deletions(-) delete mode 100644 drivers/net/enic/enic_rx.c create mode 100644 drivers/net/enic/enic_rxtx.c diff --git a/drivers/net/enic/Makefile b/drivers/net/enic/Makefile index f316274..3926b79 100644 --- a/drivers/net/enic/Makefile +++ b/drivers/net/enic/Makefile @@ -53,7 +53,7 @@ VPATH += $(SRCDIR)/src # SRCS-$(CONFIG_RTE_LIBRTE_ENIC_PMD) += enic_ethdev.c SRCS-$(CONFIG_RTE_LIBRTE_ENIC_PMD) += enic_main.c -SRCS-$(CONFIG_RTE_LIBRTE_ENIC_PMD) += enic_rx.c +SRCS-$(CONFIG_RTE_LIBRTE_ENIC_PMD) += enic_rxtx.c SRCS-$(CONFIG_RTE_LIBRTE_ENIC_PMD) += enic_clsf.c SRCS-$(CONFIG_RTE_LIBRTE_ENIC_PMD) += enic_res.c SRCS-$(CONFIG_RTE_LIBRTE_ENIC_PMD) += base/vnic_cq.c diff --git a/drivers/net/enic/enic.h b/drivers/net/enic/enic.h index 9b6f349..62a8c12 100644 --- a/drivers/net/enic/enic.h +++ b/drivers/net/enic/enic.h @@ -208,4 +208,7 @@ extern void enic_clsf_destroy(struct enic *enic); uint16_t enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); +uint16_t enicpmd_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, + uint16_t nb_pkts); +void enic_free_wq_buf(__rte_unused struct vnic_wq *wq, struct vnic_wq_buf *buf); #endif /* _ENIC_H_ */ diff --git a/drivers/net/enic/enic_ethdev.c b/drivers/net/enic/enic_ethdev.c index 6bea940..fab8124 100644 --- a/drivers/net/enic/enic_ethdev.c +++ b/drivers/net/enic/enic_ethdev.c @@ -519,71 +519,6 @@ static void enicpmd_remove_mac_addr(struct rte_eth_dev *eth_dev, __rte_unused ui enic_del_mac_address(enic); } - -static uint16_t enicpmd_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, - uint16_t nb_pkts) -{ - uint16_t index; - unsigned int frags; - unsigned int pkt_len; - unsigned int seg_len; - unsigned int inc_len; - unsigned int nb_segs; - struct rte_mbuf *tx_pkt, *next_tx_pkt; - struct vnic_wq *wq = (struct vnic_wq *)tx_queue; - struct enic *enic = vnic_dev_priv(wq->vdev); - unsigned short vlan_id; - unsigned short ol_flags; - uint8_t last_seg, eop; - unsigned int host_tx_descs = 0; - - for (index = 0; index < nb_pkts; index++) { - tx_pkt = *tx_pkts++; - inc_len = 0; - nb_segs = tx_pkt->nb_segs; - if (nb_segs > vnic_wq_desc_avail(wq)) { - if (index > 0) - enic_post_wq_index(wq); - - /* wq cleanup and try again */ - if (!enic_cleanup_wq(enic, wq) || - (nb_segs > vnic_wq_desc_avail(wq))) { - return index; - } - } - - pkt_len = tx_pkt->pkt_len; - vlan_id = tx_pkt->vlan_tci; - ol_flags = tx_pkt->ol_flags; - for (frags = 0; inc_len < pkt_len; frags++) { - if (!tx_pkt) - break; - next_tx_pkt = tx_pkt->next; - seg_len = tx_pkt->data_len; - inc_len += seg_len; - - host_tx_descs++; - last_seg = 0; - eop = 0; - if ((pkt_len == inc_len) || !next_tx_pkt) { - eop = 1; - /* post if last packet in batch or > thresh */ - if ((index == (nb_pkts - 1)) || - (host_tx_descs > ENIC_TX_POST_THRESH)) { - last_seg = 1; - host_tx_descs = 0; - } - } - enic_send_pkt(enic, wq, tx_pkt, (unsigned short)seg_len, - !frags, eop, last_seg, ol_flags, vlan_id); - tx_pkt = next_tx_pkt; - } - } - - enic_cleanup_wq(enic, wq); - return index; -} - static const struct eth_dev_ops enicpmd_eth_dev_ops = { .dev_configure = enicpmd_dev_configure, .dev_start = enicpmd_dev_start, diff --git a/drivers/net/enic/enic_main.c b/drivers/net/enic/enic_main.c index e4ccc7d..f41ef86 100644 --- a/drivers/net/enic/enic_main.c +++ b/drivers/net/enic/enic_main.c @@ -58,7 +58,6 @@ #include "vnic_cq.h" #include "vnic_intr.h" #include "vnic_nic.h" -#include "enic_vnic_wq.h" static inline int enic_is_sriov_vf(struct enic *enic) { @@ -104,7 +103,7 @@ void enic_set_hdr_split_size(struct enic *enic, u16 split_hdr_size) vnic_set_hdr_split_size(enic->vdev, split_hdr_size); } -static void enic_free_wq_buf(__rte_unused struct vnic_wq *wq, struct vnic_wq_buf *buf) +void enic_free_wq_buf(__rte_unused struct vnic_wq *wq, struct vnic_wq_buf *buf) { struct rte_mbuf *mbuf = (struct rte_mbuf *)buf->os_buf; @@ -112,26 +111,6 @@ static void enic_free_wq_buf(__rte_unused struct vnic_wq *wq, struct vnic_wq_buf buf->os_buf = NULL; } -static void enic_wq_free_buf(struct vnic_wq *wq, - __rte_unused struct cq_desc *cq_desc, - struct vnic_wq_buf *buf, - __rte_unused void *opaque) -{ - enic_free_wq_buf(wq, buf); -} - -static int enic_wq_service(struct vnic_dev *vdev, struct cq_desc *cq_desc, - __rte_unused u8 type, u16 q_number, u16 completed_index, void *opaque) -{ - struct enic *enic = vnic_dev_priv(vdev); - - vnic_wq_service(&enic->wq[q_number], cq_desc, - completed_index, enic_wq_free_buf, - opaque); - - return 0; -} - static void enic_log_q_error(struct enic *enic) { unsigned int i; @@ -152,65 +131,6 @@ static void enic_log_q_error(struct enic *enic) } } -unsigned int enic_cleanup_wq(struct enic *enic, struct vnic_wq *wq) -{ - unsigned int cq = enic_cq_wq(enic, wq->index); - - /* Return the work done */ - return vnic_cq_service(&enic->cq[cq], - -1 /*wq_work_to_do*/, enic_wq_service, NULL); -} - -void enic_post_wq_index(struct vnic_wq *wq) -{ - enic_vnic_post_wq_index(wq); -} - -void enic_send_pkt(struct enic *enic, struct vnic_wq *wq, - struct rte_mbuf *tx_pkt, unsigned short len, - uint8_t sop, uint8_t eop, uint8_t cq_entry, - uint16_t ol_flags, uint16_t vlan_tag) -{ - struct wq_enet_desc *desc = vnic_wq_next_desc(wq); - uint16_t mss = 0; - uint8_t vlan_tag_insert = 0; - uint64_t bus_addr = (dma_addr_t) - (tx_pkt->buf_physaddr + tx_pkt->data_off); - - if (sop) { - if (ol_flags & PKT_TX_VLAN_PKT) - vlan_tag_insert = 1; - - if (enic->hw_ip_checksum) { - if (ol_flags & PKT_TX_IP_CKSUM) - mss |= ENIC_CALC_IP_CKSUM; - - if (ol_flags & PKT_TX_TCP_UDP_CKSUM) - mss |= ENIC_CALC_TCP_UDP_CKSUM; - } - } - - wq_enet_desc_enc(desc, - bus_addr, - len, - mss, - 0 /* header_length */, - 0 /* offload_mode WQ_ENET_OFFLOAD_MODE_CSUM */, - eop, - cq_entry, - 0 /* fcoe_encap */, - vlan_tag_insert, - vlan_tag, - 0 /* loopback */); - - enic_vnic_post_wq(wq, (void *)tx_pkt, bus_addr, len, - sop, - 1 /*desc_skip_cnt*/, - cq_entry, - 0 /*compressed send*/, - 0 /*wrid*/); -} - static void enic_clear_soft_stats(struct enic *enic) { struct enic_soft_stats *soft_stats = &enic->soft_stats; diff --git a/drivers/net/enic/enic_rx.c b/drivers/net/enic/enic_rx.c deleted file mode 100644 index c72a80a..0000000 --- a/drivers/net/enic/enic_rx.c +++ /dev/null @@ -1,343 +0,0 @@ -/* - * Copyright 2008-2014 Cisco Systems, Inc. All rights reserved. - * Copyright 2007 Nuova Systems, Inc. All rights reserved. - * - * Copyright (c) 2014, Cisco Systems, Inc. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in - * the documentation and/or other materials provided with the - * distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS - * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE - * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, - * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, - * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; - * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER - * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN - * ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * - */ - -#include <rte_mbuf.h> -#include <rte_ethdev.h> -#include <rte_prefetch.h> - -#include "enic_compat.h" -#include "rq_enet_desc.h" -#include "enic.h" - -#define RTE_PMD_USE_PREFETCH - -#ifdef RTE_PMD_USE_PREFETCH -/* - * Prefetch a cache line into all cache levels. - */ -#define rte_enic_prefetch(p) rte_prefetch0(p) -#else -#define rte_enic_prefetch(p) do {} while (0) -#endif - -#ifdef RTE_PMD_PACKET_PREFETCH -#define rte_packet_prefetch(p) rte_prefetch1(p) -#else -#define rte_packet_prefetch(p) do {} while (0) -#endif - -static inline uint16_t -enic_cq_rx_desc_ciflags(struct cq_enet_rq_desc *crd) -{ - return le16_to_cpu(crd->completed_index_flags) & ~CQ_DESC_COMP_NDX_MASK; -} - -static inline uint16_t -enic_cq_rx_desc_bwflags(struct cq_enet_rq_desc *crd) -{ - return(le16_to_cpu(crd->bytes_written_flags) & - ~CQ_ENET_RQ_DESC_BYTES_WRITTEN_MASK); -} - -static inline uint8_t -enic_cq_rx_desc_packet_error(uint16_t bwflags) -{ - return((bwflags & CQ_ENET_RQ_DESC_FLAGS_TRUNCATED) == - CQ_ENET_RQ_DESC_FLAGS_TRUNCATED); -} - -static inline uint8_t -enic_cq_rx_desc_eop(uint16_t ciflags) -{ - return (ciflags & CQ_ENET_RQ_DESC_FLAGS_EOP) - == CQ_ENET_RQ_DESC_FLAGS_EOP; -} - -static inline uint8_t -enic_cq_rx_desc_csum_not_calc(struct cq_enet_rq_desc *cqrd) -{ - return ((le16_to_cpu(cqrd->q_number_rss_type_flags) & - CQ_ENET_RQ_DESC_FLAGS_CSUM_NOT_CALC) == - CQ_ENET_RQ_DESC_FLAGS_CSUM_NOT_CALC); -} - -static inline uint8_t -enic_cq_rx_desc_ipv4_csum_ok(struct cq_enet_rq_desc *cqrd) -{ - return ((cqrd->flags & CQ_ENET_RQ_DESC_FLAGS_IPV4_CSUM_OK) == - CQ_ENET_RQ_DESC_FLAGS_IPV4_CSUM_OK); -} - -static inline uint8_t -enic_cq_rx_desc_tcp_udp_csum_ok(struct cq_enet_rq_desc *cqrd) -{ - return((cqrd->flags & CQ_ENET_RQ_DESC_FLAGS_TCP_UDP_CSUM_OK) == - CQ_ENET_RQ_DESC_FLAGS_TCP_UDP_CSUM_OK); -} - -static inline uint8_t -enic_cq_rx_desc_rss_type(struct cq_enet_rq_desc *cqrd) -{ - return (uint8_t)((le16_to_cpu(cqrd->q_number_rss_type_flags) >> - CQ_DESC_Q_NUM_BITS) & CQ_ENET_RQ_DESC_RSS_TYPE_MASK); -} - -static inline uint32_t -enic_cq_rx_desc_rss_hash(struct cq_enet_rq_desc *cqrd) -{ - return le32_to_cpu(cqrd->rss_hash); -} - -static inline uint16_t -enic_cq_rx_desc_vlan(struct cq_enet_rq_desc *cqrd) -{ - return le16_to_cpu(cqrd->vlan); -} - -static inline uint16_t -enic_cq_rx_desc_n_bytes(struct cq_desc *cqd) -{ - struct cq_enet_rq_desc *cqrd = (struct cq_enet_rq_desc *)cqd; - return le16_to_cpu(cqrd->bytes_written_flags) & - CQ_ENET_RQ_DESC_BYTES_WRITTEN_MASK; -} - -static inline uint8_t -enic_cq_rx_check_err(struct cq_desc *cqd) -{ - struct cq_enet_rq_desc *cqrd = (struct cq_enet_rq_desc *)cqd; - uint16_t bwflags; - - bwflags = enic_cq_rx_desc_bwflags(cqrd); - if (unlikely(enic_cq_rx_desc_packet_error(bwflags))) - return 1; - return 0; -} - -/* - * Lookup table to translate RX CQ flags to mbuf flags. - */ -static inline uint32_t -enic_cq_rx_flags_to_pkt_type(struct cq_desc *cqd) -{ - struct cq_enet_rq_desc *cqrd = (struct cq_enet_rq_desc *)cqd; - uint8_t cqrd_flags = cqrd->flags; - static const uint32_t cq_type_table[128] __rte_cache_aligned = { - [32] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4, - [34] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 - | RTE_PTYPE_L4_UDP, - [36] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 - | RTE_PTYPE_L4_TCP, - [96] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 - | RTE_PTYPE_L4_FRAG, - [16] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6, - [18] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 - | RTE_PTYPE_L4_UDP, - [20] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 - | RTE_PTYPE_L4_TCP, - [80] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 - | RTE_PTYPE_L4_FRAG, - /* All others reserved */ - }; - cqrd_flags &= CQ_ENET_RQ_DESC_FLAGS_IPV4_FRAGMENT - | CQ_ENET_RQ_DESC_FLAGS_IPV4 | CQ_ENET_RQ_DESC_FLAGS_IPV6 - | CQ_ENET_RQ_DESC_FLAGS_TCP | CQ_ENET_RQ_DESC_FLAGS_UDP; - return cq_type_table[cqrd_flags]; -} - -static inline void -enic_cq_rx_to_pkt_flags(struct cq_desc *cqd, struct rte_mbuf *mbuf) -{ - struct cq_enet_rq_desc *cqrd = (struct cq_enet_rq_desc *)cqd; - uint16_t ciflags, bwflags, pkt_flags = 0; - ciflags = enic_cq_rx_desc_ciflags(cqrd); - bwflags = enic_cq_rx_desc_bwflags(cqrd); - - mbuf->ol_flags = 0; - - /* flags are meaningless if !EOP */ - if (unlikely(!enic_cq_rx_desc_eop(ciflags))) - goto mbuf_flags_done; - - /* VLAN stripping */ - if (bwflags & CQ_ENET_RQ_DESC_FLAGS_VLAN_STRIPPED) { - pkt_flags |= PKT_RX_VLAN_PKT; - mbuf->vlan_tci = enic_cq_rx_desc_vlan(cqrd); - } else { - mbuf->vlan_tci = 0; - } - - /* RSS flag */ - if (enic_cq_rx_desc_rss_type(cqrd)) { - pkt_flags |= PKT_RX_RSS_HASH; - mbuf->hash.rss = enic_cq_rx_desc_rss_hash(cqrd); - } - - /* checksum flags */ - if (!enic_cq_rx_desc_csum_not_calc(cqrd) && - (mbuf->packet_type & RTE_PTYPE_L3_IPV4)) { - if (unlikely(!enic_cq_rx_desc_ipv4_csum_ok(cqrd))) - pkt_flags |= PKT_RX_IP_CKSUM_BAD; - if (mbuf->packet_type & (RTE_PTYPE_L4_UDP | RTE_PTYPE_L4_TCP)) { - if (unlikely(!enic_cq_rx_desc_tcp_udp_csum_ok(cqrd))) - pkt_flags |= PKT_RX_L4_CKSUM_BAD; - } - } - - mbuf_flags_done: - mbuf->ol_flags = pkt_flags; -} - -static inline uint32_t -enic_ring_add(uint32_t n_descriptors, uint32_t i0, uint32_t i1) -{ - uint32_t d = i0 + i1; - RTE_ASSERT(i0 < n_descriptors); - RTE_ASSERT(i1 < n_descriptors); - d -= (d >= n_descriptors) ? n_descriptors : 0; - return d; -} - - -uint16_t -enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, - uint16_t nb_pkts) -{ - struct vnic_rq *rq = rx_queue; - struct enic *enic = vnic_dev_priv(rq->vdev); - unsigned int rx_id; - struct rte_mbuf *nmb, *rxmb; - uint16_t nb_rx = 0, nb_err = 0; - uint16_t nb_hold; - struct vnic_cq *cq; - volatile struct cq_desc *cqd_ptr; - uint8_t color; - - cq = &enic->cq[enic_cq_rq(enic, rq->index)]; - rx_id = cq->to_clean; /* index of cqd, rqd, mbuf_table */ - cqd_ptr = (struct cq_desc *)(cq->ring.descs) + rx_id; - - nb_hold = rq->rx_nb_hold; /* mbufs held by software */ - - while (nb_rx < nb_pkts) { - volatile struct rq_enet_desc *rqd_ptr; - dma_addr_t dma_addr; - struct cq_desc cqd; - uint8_t packet_error; - - /* Check for pkts available */ - color = (cqd_ptr->type_color >> CQ_DESC_COLOR_SHIFT) - & CQ_DESC_COLOR_MASK; - if (color == cq->last_color) - break; - - /* Get the cq descriptor and rq pointer */ - cqd = *cqd_ptr; - rqd_ptr = (struct rq_enet_desc *)(rq->ring.descs) + rx_id; - - /* allocate a new mbuf */ - nmb = rte_mbuf_raw_alloc(rq->mp); - if (nmb == NULL) { - rte_atomic64_inc(&enic->soft_stats.rx_nombuf); - break; - } - - /* A packet error means descriptor and data are untrusted */ - packet_error = enic_cq_rx_check_err(&cqd); - - /* Get the mbuf to return and replace with one just allocated */ - rxmb = rq->mbuf_ring[rx_id]; - rq->mbuf_ring[rx_id] = nmb; - - /* Increment cqd, rqd, mbuf_table index */ - rx_id++; - if (unlikely(rx_id == rq->ring.desc_count)) { - rx_id = 0; - cq->last_color = cq->last_color ? 0 : 1; - } - - /* Prefetch next mbuf & desc while processing current one */ - cqd_ptr = (struct cq_desc *)(cq->ring.descs) + rx_id; - rte_enic_prefetch(cqd_ptr); - rte_enic_prefetch(rq->mbuf_ring[rx_id]); - rte_enic_prefetch((struct rq_enet_desc *)(rq->ring.descs) - + rx_id); - - /* Push descriptor for newly allocated mbuf */ - dma_addr = (dma_addr_t)(nmb->buf_physaddr - + RTE_PKTMBUF_HEADROOM); - rqd_ptr->address = rte_cpu_to_le_64(dma_addr); - rqd_ptr->length_type = cpu_to_le16(nmb->buf_len - - RTE_PKTMBUF_HEADROOM); - - /* Drop incoming bad packet */ - if (unlikely(packet_error)) { - rte_pktmbuf_free(rxmb); - nb_err++; - continue; - } - - /* Fill in the rest of the mbuf */ - rxmb->data_off = RTE_PKTMBUF_HEADROOM; - rxmb->nb_segs = 1; - rxmb->next = NULL; - rxmb->port = enic->port_id; - rxmb->pkt_len = enic_cq_rx_desc_n_bytes(&cqd); - rxmb->packet_type = enic_cq_rx_flags_to_pkt_type(&cqd); - enic_cq_rx_to_pkt_flags(&cqd, rxmb); - rxmb->data_len = rxmb->pkt_len; - - /* prefetch mbuf data for caller */ - rte_packet_prefetch(RTE_PTR_ADD(rxmb->buf_addr, - RTE_PKTMBUF_HEADROOM)); - - /* store the mbuf address into the next entry of the array */ - rx_pkts[nb_rx++] = rxmb; - } - - nb_hold += nb_rx + nb_err; - cq->to_clean = rx_id; - - if (nb_hold > rq->rx_free_thresh) { - rq->posted_index = enic_ring_add(rq->ring.desc_count, - rq->posted_index, nb_hold); - nb_hold = 0; - rte_mb(); - iowrite32(rq->posted_index, &rq->ctrl->posted_index); - } - - rq->rx_nb_hold = nb_hold; - - return nb_rx; -} diff --git a/drivers/net/enic/enic_rxtx.c b/drivers/net/enic/enic_rxtx.c new file mode 100644 index 0000000..056a297 --- /dev/null +++ b/drivers/net/enic/enic_rxtx.c @@ -0,0 +1,481 @@ +/* Copyright 2008-2016 Cisco Systems, Inc. All rights reserved. + * Copyright 2007 Nuova Systems, Inc. All rights reserved. + * + * Copyright (c) 2014, Cisco Systems, Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS + * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE + * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, + * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, + * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN + * ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ + +#include <rte_mbuf.h> +#include <rte_ethdev.h> +#include <rte_prefetch.h> + +#include "enic_compat.h" +#include "rq_enet_desc.h" +#include "enic.h" +#include "enic_vnic_wq.h" + +#define RTE_PMD_USE_PREFETCH + +#ifdef RTE_PMD_USE_PREFETCH +/*Prefetch a cache line into all cache levels. */ +#define rte_enic_prefetch(p) rte_prefetch0(p) +#else +#define rte_enic_prefetch(p) do {} while (0) +#endif + +#ifdef RTE_PMD_PACKET_PREFETCH +#define rte_packet_prefetch(p) rte_prefetch1(p) +#else +#define rte_packet_prefetch(p) do {} while (0) +#endif + +static inline uint16_t +enic_cq_rx_desc_ciflags(struct cq_enet_rq_desc *crd) +{ + return le16_to_cpu(crd->completed_index_flags) & ~CQ_DESC_COMP_NDX_MASK; +} + +static inline uint16_t +enic_cq_rx_desc_bwflags(struct cq_enet_rq_desc *crd) +{ + return le16_to_cpu(crd->bytes_written_flags) & + ~CQ_ENET_RQ_DESC_BYTES_WRITTEN_MASK; +} + +static inline uint8_t +enic_cq_rx_desc_packet_error(uint16_t bwflags) +{ + return (bwflags & CQ_ENET_RQ_DESC_FLAGS_TRUNCATED) == + CQ_ENET_RQ_DESC_FLAGS_TRUNCATED; +} + +static inline uint8_t +enic_cq_rx_desc_eop(uint16_t ciflags) +{ + return (ciflags & CQ_ENET_RQ_DESC_FLAGS_EOP) + == CQ_ENET_RQ_DESC_FLAGS_EOP; +} + +static inline uint8_t +enic_cq_rx_desc_csum_not_calc(struct cq_enet_rq_desc *cqrd) +{ + return (le16_to_cpu(cqrd->q_number_rss_type_flags) & + CQ_ENET_RQ_DESC_FLAGS_CSUM_NOT_CALC) == + CQ_ENET_RQ_DESC_FLAGS_CSUM_NOT_CALC; +} + +static inline uint8_t +enic_cq_rx_desc_ipv4_csum_ok(struct cq_enet_rq_desc *cqrd) +{ + return (cqrd->flags & CQ_ENET_RQ_DESC_FLAGS_IPV4_CSUM_OK) == + CQ_ENET_RQ_DESC_FLAGS_IPV4_CSUM_OK; +} + +static inline uint8_t +enic_cq_rx_desc_tcp_udp_csum_ok(struct cq_enet_rq_desc *cqrd) +{ + return (cqrd->flags & CQ_ENET_RQ_DESC_FLAGS_TCP_UDP_CSUM_OK) == + CQ_ENET_RQ_DESC_FLAGS_TCP_UDP_CSUM_OK; +} + +static inline uint8_t +enic_cq_rx_desc_rss_type(struct cq_enet_rq_desc *cqrd) +{ + return (uint8_t)((le16_to_cpu(cqrd->q_number_rss_type_flags) >> + CQ_DESC_Q_NUM_BITS) & CQ_ENET_RQ_DESC_RSS_TYPE_MASK); +} + +static inline uint32_t +enic_cq_rx_desc_rss_hash(struct cq_enet_rq_desc *cqrd) +{ + return le32_to_cpu(cqrd->rss_hash); +} + +static inline uint16_t +enic_cq_rx_desc_vlan(struct cq_enet_rq_desc *cqrd) +{ + return le16_to_cpu(cqrd->vlan); +} + +static inline uint16_t +enic_cq_rx_desc_n_bytes(struct cq_desc *cqd) +{ + struct cq_enet_rq_desc *cqrd = (struct cq_enet_rq_desc *)cqd; + return le16_to_cpu(cqrd->bytes_written_flags) & + CQ_ENET_RQ_DESC_BYTES_WRITTEN_MASK; +} + +static inline uint8_t +enic_cq_rx_check_err(struct cq_desc *cqd) +{ + struct cq_enet_rq_desc *cqrd = (struct cq_enet_rq_desc *)cqd; + uint16_t bwflags; + + bwflags = enic_cq_rx_desc_bwflags(cqrd); + if (unlikely(enic_cq_rx_desc_packet_error(bwflags))) + return 1; + return 0; +} + +/* Lookup table to translate RX CQ flags to mbuf flags. */ +static inline uint32_t +enic_cq_rx_flags_to_pkt_type(struct cq_desc *cqd) +{ + struct cq_enet_rq_desc *cqrd = (struct cq_enet_rq_desc *)cqd; + uint8_t cqrd_flags = cqrd->flags; + static const uint32_t cq_type_table[128] __rte_cache_aligned = { + [32] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4, + [34] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 + | RTE_PTYPE_L4_UDP, + [36] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 + | RTE_PTYPE_L4_TCP, + [96] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 + | RTE_PTYPE_L4_FRAG, + [16] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6, + [18] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 + | RTE_PTYPE_L4_UDP, + [20] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 + | RTE_PTYPE_L4_TCP, + [80] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 + | RTE_PTYPE_L4_FRAG, + /* All others reserved */ + }; + cqrd_flags &= CQ_ENET_RQ_DESC_FLAGS_IPV4_FRAGMENT + | CQ_ENET_RQ_DESC_FLAGS_IPV4 | CQ_ENET_RQ_DESC_FLAGS_IPV6 + | CQ_ENET_RQ_DESC_FLAGS_TCP | CQ_ENET_RQ_DESC_FLAGS_UDP; + return cq_type_table[cqrd_flags]; +} + +static inline void +enic_cq_rx_to_pkt_flags(struct cq_desc *cqd, struct rte_mbuf *mbuf) +{ + struct cq_enet_rq_desc *cqrd = (struct cq_enet_rq_desc *)cqd; + uint16_t ciflags, bwflags, pkt_flags = 0; + ciflags = enic_cq_rx_desc_ciflags(cqrd); + bwflags = enic_cq_rx_desc_bwflags(cqrd); + + mbuf->ol_flags = 0; + + /* flags are meaningless if !EOP */ + if (unlikely(!enic_cq_rx_desc_eop(ciflags))) + goto mbuf_flags_done; + + /* VLAN stripping */ + if (bwflags & CQ_ENET_RQ_DESC_FLAGS_VLAN_STRIPPED) { + pkt_flags |= PKT_RX_VLAN_PKT; + mbuf->vlan_tci = enic_cq_rx_desc_vlan(cqrd); + } else { + mbuf->vlan_tci = 0; + } + + /* RSS flag */ + if (enic_cq_rx_desc_rss_type(cqrd)) { + pkt_flags |= PKT_RX_RSS_HASH; + mbuf->hash.rss = enic_cq_rx_desc_rss_hash(cqrd); + } + + /* checksum flags */ + if (!enic_cq_rx_desc_csum_not_calc(cqrd) && + (mbuf->packet_type & RTE_PTYPE_L3_IPV4)) { + if (unlikely(!enic_cq_rx_desc_ipv4_csum_ok(cqrd))) + pkt_flags |= PKT_RX_IP_CKSUM_BAD; + if (mbuf->packet_type & (RTE_PTYPE_L4_UDP | RTE_PTYPE_L4_TCP)) { + if (unlikely(!enic_cq_rx_desc_tcp_udp_csum_ok(cqrd))) + pkt_flags |= PKT_RX_L4_CKSUM_BAD; + } + } + + mbuf_flags_done: + mbuf->ol_flags = pkt_flags; +} + +static inline uint32_t +enic_ring_add(uint32_t n_descriptors, uint32_t i0, uint32_t i1) +{ + uint32_t d = i0 + i1; + RTE_ASSERT(i0 < n_descriptors); + RTE_ASSERT(i1 < n_descriptors); + d -= (d >= n_descriptors) ? n_descriptors : 0; + return d; +} + + +uint16_t +enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, + uint16_t nb_pkts) +{ + struct vnic_rq *rq = rx_queue; + struct enic *enic = vnic_dev_priv(rq->vdev); + unsigned int rx_id; + struct rte_mbuf *nmb, *rxmb; + uint16_t nb_rx = 0, nb_err = 0; + uint16_t nb_hold; + struct vnic_cq *cq; + volatile struct cq_desc *cqd_ptr; + uint8_t color; + + cq = &enic->cq[enic_cq_rq(enic, rq->index)]; + rx_id = cq->to_clean; /* index of cqd, rqd, mbuf_table */ + cqd_ptr = (struct cq_desc *)(cq->ring.descs) + rx_id; + + nb_hold = rq->rx_nb_hold; /* mbufs held by software */ + + while (nb_rx < nb_pkts) { + volatile struct rq_enet_desc *rqd_ptr; + dma_addr_t dma_addr; + struct cq_desc cqd; + uint8_t packet_error; + + /* Check for pkts available */ + color = (cqd_ptr->type_color >> CQ_DESC_COLOR_SHIFT) + & CQ_DESC_COLOR_MASK; + if (color == cq->last_color) + break; + + /* Get the cq descriptor and rq pointer */ + cqd = *cqd_ptr; + rqd_ptr = (struct rq_enet_desc *)(rq->ring.descs) + rx_id; + + /* allocate a new mbuf */ + nmb = rte_mbuf_raw_alloc(rq->mp); + if (nmb == NULL) { + rte_atomic64_inc(&enic->soft_stats.rx_nombuf); + break; + } + + /* A packet error means descriptor and data are untrusted */ + packet_error = enic_cq_rx_check_err(&cqd); + + /* Get the mbuf to return and replace with one just allocated */ + rxmb = rq->mbuf_ring[rx_id]; + rq->mbuf_ring[rx_id] = nmb; + + /* Increment cqd, rqd, mbuf_table index */ + rx_id++; + if (unlikely(rx_id == rq->ring.desc_count)) { + rx_id = 0; + cq->last_color = cq->last_color ? 0 : 1; + } + + /* Prefetch next mbuf & desc while processing current one */ + cqd_ptr = (struct cq_desc *)(cq->ring.descs) + rx_id; + rte_enic_prefetch(cqd_ptr); + rte_enic_prefetch(rq->mbuf_ring[rx_id]); + rte_enic_prefetch((struct rq_enet_desc *)(rq->ring.descs) + + rx_id); + + /* Push descriptor for newly allocated mbuf */ + dma_addr = (dma_addr_t)(nmb->buf_physaddr + + RTE_PKTMBUF_HEADROOM); + rqd_ptr->address = rte_cpu_to_le_64(dma_addr); + rqd_ptr->length_type = cpu_to_le16(nmb->buf_len + - RTE_PKTMBUF_HEADROOM); + + /* Drop incoming bad packet */ + if (unlikely(packet_error)) { + rte_pktmbuf_free(rxmb); + nb_err++; + continue; + } + + /* Fill in the rest of the mbuf */ + rxmb->data_off = RTE_PKTMBUF_HEADROOM; + rxmb->nb_segs = 1; + rxmb->next = NULL; + rxmb->port = enic->port_id; + rxmb->pkt_len = enic_cq_rx_desc_n_bytes(&cqd); + rxmb->packet_type = enic_cq_rx_flags_to_pkt_type(&cqd); + enic_cq_rx_to_pkt_flags(&cqd, rxmb); + rxmb->data_len = rxmb->pkt_len; + + /* prefetch mbuf data for caller */ + rte_packet_prefetch(RTE_PTR_ADD(rxmb->buf_addr, + RTE_PKTMBUF_HEADROOM)); + + /* store the mbuf address into the next entry of the array */ + rx_pkts[nb_rx++] = rxmb; + } + + nb_hold += nb_rx + nb_err; + cq->to_clean = rx_id; + + if (nb_hold > rq->rx_free_thresh) { + rq->posted_index = enic_ring_add(rq->ring.desc_count, + rq->posted_index, nb_hold); + nb_hold = 0; + rte_mb(); + iowrite32(rq->posted_index, &rq->ctrl->posted_index); + } + + rq->rx_nb_hold = nb_hold; + + return nb_rx; +} + +static void enic_wq_free_buf(struct vnic_wq *wq, + __rte_unused struct cq_desc *cq_desc, + struct vnic_wq_buf *buf, + __rte_unused void *opaque) +{ + enic_free_wq_buf(wq, buf); +} + +static int enic_wq_service(struct vnic_dev *vdev, struct cq_desc *cq_desc, + __rte_unused u8 type, u16 q_number, u16 completed_index, void *opaque) +{ + struct enic *enic = vnic_dev_priv(vdev); + + vnic_wq_service(&enic->wq[q_number], cq_desc, + completed_index, enic_wq_free_buf, + opaque); + + return 0; +} + +unsigned int enic_cleanup_wq(struct enic *enic, struct vnic_wq *wq) +{ + unsigned int cq = enic_cq_wq(enic, wq->index); + + /* Return the work done */ + return vnic_cq_service(&enic->cq[cq], + -1 /*wq_work_to_do*/, enic_wq_service, NULL); +} + +void enic_post_wq_index(struct vnic_wq *wq) +{ + enic_vnic_post_wq_index(wq); +} + +void enic_send_pkt(struct enic *enic, struct vnic_wq *wq, + struct rte_mbuf *tx_pkt, unsigned short len, + uint8_t sop, uint8_t eop, uint8_t cq_entry, + uint16_t ol_flags, uint16_t vlan_tag) +{ + struct wq_enet_desc *desc = vnic_wq_next_desc(wq); + uint16_t mss = 0; + uint8_t vlan_tag_insert = 0; + uint64_t bus_addr = (dma_addr_t) + (tx_pkt->buf_physaddr + tx_pkt->data_off); + + if (sop) { + if (ol_flags & PKT_TX_VLAN_PKT) + vlan_tag_insert = 1; + + if (enic->hw_ip_checksum) { + if (ol_flags & PKT_TX_IP_CKSUM) + mss |= ENIC_CALC_IP_CKSUM; + + if (ol_flags & PKT_TX_TCP_UDP_CKSUM) + mss |= ENIC_CALC_TCP_UDP_CKSUM; + } + } + + wq_enet_desc_enc(desc, + bus_addr, + len, + mss, + 0 /* header_length */, + 0 /* offload_mode WQ_ENET_OFFLOAD_MODE_CSUM */, + eop, + cq_entry, + 0 /* fcoe_encap */, + vlan_tag_insert, + vlan_tag, + 0 /* loopback */); + + enic_vnic_post_wq(wq, (void *)tx_pkt, bus_addr, len, + sop, + 1 /*desc_skip_cnt*/, + cq_entry, + 0 /*compressed send*/, + 0 /*wrid*/); +} + +uint16_t enicpmd_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, + uint16_t nb_pkts) +{ + uint16_t index; + unsigned int frags; + unsigned int pkt_len; + unsigned int seg_len; + unsigned int inc_len; + unsigned int nb_segs; + struct rte_mbuf *tx_pkt, *next_tx_pkt; + struct vnic_wq *wq = (struct vnic_wq *)tx_queue; + struct enic *enic = vnic_dev_priv(wq->vdev); + unsigned short vlan_id; + unsigned short ol_flags; + uint8_t last_seg, eop; + unsigned int host_tx_descs = 0; + + for (index = 0; index < nb_pkts; index++) { + tx_pkt = *tx_pkts++; + inc_len = 0; + nb_segs = tx_pkt->nb_segs; + if (nb_segs > vnic_wq_desc_avail(wq)) { + if (index > 0) + enic_post_wq_index(wq); + + /* wq cleanup and try again */ + if (!enic_cleanup_wq(enic, wq) || + (nb_segs > vnic_wq_desc_avail(wq))) { + return index; + } + } + + pkt_len = tx_pkt->pkt_len; + vlan_id = tx_pkt->vlan_tci; + ol_flags = tx_pkt->ol_flags; + for (frags = 0; inc_len < pkt_len; frags++) { + if (!tx_pkt) + break; + next_tx_pkt = tx_pkt->next; + seg_len = tx_pkt->data_len; + inc_len += seg_len; + + host_tx_descs++; + last_seg = 0; + eop = 0; + if ((pkt_len == inc_len) || !next_tx_pkt) { + eop = 1; + /* post if last packet in batch or > thresh */ + if ((index == (nb_pkts - 1)) || + (host_tx_descs > ENIC_TX_POST_THRESH)) { + last_seg = 1; + host_tx_descs = 0; + } + } + enic_send_pkt(enic, wq, tx_pkt, (unsigned short)seg_len, + !frags, eop, last_seg, ol_flags, vlan_id); + tx_pkt = next_tx_pkt; + } + } + + enic_cleanup_wq(enic, wq); + return index; +} -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v2 05/11] enic: remove some unused functions in Tx path 2016-05-24 6:32 [PATCH v2 00/11] enic counter fixes and Tx optimization John Daley ` (3 preceding siblings ...) 2016-05-24 6:32 ` [PATCH v2 04/11] enic: put Tx and Rx functions into same file John Daley @ 2016-05-24 6:32 ` John Daley 2016-05-24 6:32 ` [PATCH v2 06/11] enic: streamline mbuf handling " John Daley ` (6 subsequent siblings) 11 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-05-24 6:32 UTC (permalink / raw) To: dev; +Cc: John Daley Functions existed which were never called. Removed them. Also rename the 'pmd' from the name of the Tx function to improve clarity. Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/base/vnic_wq.h | 45 ------------------------ drivers/net/enic/enic.h | 5 ++- drivers/net/enic/enic_ethdev.c | 2 +- drivers/net/enic/enic_res.h | 78 ----------------------------------------- drivers/net/enic/enic_rxtx.c | 2 +- 5 files changed, 4 insertions(+), 128 deletions(-) diff --git a/drivers/net/enic/base/vnic_wq.h b/drivers/net/enic/base/vnic_wq.h index c23de62..d8660e4 100644 --- a/drivers/net/enic/base/vnic_wq.h +++ b/drivers/net/enic/base/vnic_wq.h @@ -191,51 +191,6 @@ static inline u64 vnic_cached_posted_index(dma_addr_t addr, unsigned int len, PI_PREFETCH_ADDR_MASK) << PI_PREFETCH_ADDR_OFF); } -static inline void vnic_wq_post(struct vnic_wq *wq, - void *os_buf, dma_addr_t dma_addr, - unsigned int len, int sop, int eop, - uint8_t desc_skip_cnt, uint8_t cq_entry, - uint8_t compressed_send, uint64_t wrid) -{ - struct vnic_wq_buf *buf = wq->to_use; - - buf->sop = sop; - buf->cq_entry = cq_entry; - buf->compressed_send = compressed_send; - buf->desc_skip_cnt = desc_skip_cnt; - buf->os_buf = os_buf; - buf->dma_addr = dma_addr; - buf->len = len; - buf->wr_id = wrid; - - buf = buf->next; - if (eop) { -#ifdef DO_PREFETCH - uint64_t wr = vnic_cached_posted_index(dma_addr, len, - buf->index); -#endif - /* Adding write memory barrier prevents compiler and/or CPU - * reordering, thus avoiding descriptor posting before - * descriptor is initialized. Otherwise, hardware can read - * stale descriptor fields. - */ - wmb(); -#ifdef DO_PREFETCH - /* Intel chipsets seem to limit the rate of PIOs that we can - * push on the bus. Thus, it is very important to do a single - * 64 bit write here. With two 32-bit writes, my maximum - * pkt/sec rate was cut almost in half. -AJF - */ - iowrite64((uint64_t)wr, &wq->ctrl->posted_index); -#else - iowrite32(buf->index, &wq->ctrl->posted_index); -#endif - } - wq->to_use = buf; - - wq->ring.desc_avail -= desc_skip_cnt; -} - static inline void vnic_wq_service(struct vnic_wq *wq, struct cq_desc *cq_desc, u16 completed_index, void (*buf_service)(struct vnic_wq *wq, diff --git a/drivers/net/enic/enic.h b/drivers/net/enic/enic.h index 62a8c12..5b58a65 100644 --- a/drivers/net/enic/enic.h +++ b/drivers/net/enic/enic.h @@ -207,8 +207,7 @@ extern int enic_clsf_init(struct enic *enic); extern void enic_clsf_destroy(struct enic *enic); uint16_t enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); - -uint16_t enicpmd_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, - uint16_t nb_pkts); +uint16_t enic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, + uint16_t nb_pkts); void enic_free_wq_buf(__rte_unused struct vnic_wq *wq, struct vnic_wq_buf *buf); #endif /* _ENIC_H_ */ diff --git a/drivers/net/enic/enic_ethdev.c b/drivers/net/enic/enic_ethdev.c index fab8124..697ff82 100644 --- a/drivers/net/enic/enic_ethdev.c +++ b/drivers/net/enic/enic_ethdev.c @@ -577,7 +577,7 @@ static int eth_enicpmd_dev_init(struct rte_eth_dev *eth_dev) enic->rte_dev = eth_dev; eth_dev->dev_ops = &enicpmd_eth_dev_ops; eth_dev->rx_pkt_burst = &enic_recv_pkts; - eth_dev->tx_pkt_burst = &enicpmd_xmit_pkts; + eth_dev->tx_pkt_burst = &enic_xmit_pkts; pdev = eth_dev->pci_dev; rte_eth_copy_pci_info(eth_dev, pdev); diff --git a/drivers/net/enic/enic_res.h b/drivers/net/enic/enic_res.h index 00fa71d..955db71 100644 --- a/drivers/net/enic/enic_res.h +++ b/drivers/net/enic/enic_res.h @@ -57,84 +57,6 @@ #define ENIC_SETTING(enic, f) ((enic->config.flags & VENETF_##f) ? 1 : 0) -static inline void enic_queue_wq_desc_ex(struct vnic_wq *wq, - void *os_buf, dma_addr_t dma_addr, unsigned int len, - unsigned int mss_or_csum_offset, unsigned int hdr_len, - int vlan_tag_insert, unsigned int vlan_tag, - int offload_mode, int cq_entry, int sop, int eop, int loopback) -{ - struct wq_enet_desc *desc = vnic_wq_next_desc(wq); - u8 desc_skip_cnt = 1; - u8 compressed_send = 0; - u64 wrid = 0; - - wq_enet_desc_enc(desc, - (u64)dma_addr | VNIC_PADDR_TARGET, - (u16)len, - (u16)mss_or_csum_offset, - (u16)hdr_len, (u8)offload_mode, - (u8)eop, (u8)cq_entry, - 0, /* fcoe_encap */ - (u8)vlan_tag_insert, - (u16)vlan_tag, - (u8)loopback); - - vnic_wq_post(wq, os_buf, dma_addr, len, sop, eop, desc_skip_cnt, - (u8)cq_entry, compressed_send, wrid); -} - -static inline void enic_queue_wq_desc_cont(struct vnic_wq *wq, - void *os_buf, dma_addr_t dma_addr, unsigned int len, - int eop, int loopback) -{ - enic_queue_wq_desc_ex(wq, os_buf, dma_addr, len, - 0, 0, 0, 0, 0, - eop, 0 /* !SOP */, eop, loopback); -} - -static inline void enic_queue_wq_desc(struct vnic_wq *wq, void *os_buf, - dma_addr_t dma_addr, unsigned int len, int vlan_tag_insert, - unsigned int vlan_tag, int eop, int loopback) -{ - enic_queue_wq_desc_ex(wq, os_buf, dma_addr, len, - 0, 0, vlan_tag_insert, vlan_tag, - WQ_ENET_OFFLOAD_MODE_CSUM, - eop, 1 /* SOP */, eop, loopback); -} - -static inline void enic_queue_wq_desc_csum(struct vnic_wq *wq, - void *os_buf, dma_addr_t dma_addr, unsigned int len, - int ip_csum, int tcpudp_csum, int vlan_tag_insert, - unsigned int vlan_tag, int eop, int loopback) -{ - enic_queue_wq_desc_ex(wq, os_buf, dma_addr, len, - (ip_csum ? 1 : 0) + (tcpudp_csum ? 2 : 0), - 0, vlan_tag_insert, vlan_tag, - WQ_ENET_OFFLOAD_MODE_CSUM, - eop, 1 /* SOP */, eop, loopback); -} - -static inline void enic_queue_wq_desc_csum_l4(struct vnic_wq *wq, - void *os_buf, dma_addr_t dma_addr, unsigned int len, - unsigned int csum_offset, unsigned int hdr_len, - int vlan_tag_insert, unsigned int vlan_tag, int eop, int loopback) -{ - enic_queue_wq_desc_ex(wq, os_buf, dma_addr, len, - csum_offset, hdr_len, vlan_tag_insert, vlan_tag, - WQ_ENET_OFFLOAD_MODE_CSUM_L4, - eop, 1 /* SOP */, eop, loopback); -} - -static inline void enic_queue_wq_desc_tso(struct vnic_wq *wq, - void *os_buf, dma_addr_t dma_addr, unsigned int len, - unsigned int mss, unsigned int hdr_len, int vlan_tag_insert, - unsigned int vlan_tag, int eop, int loopback) -{ - enic_queue_wq_desc_ex(wq, os_buf, dma_addr, len, - mss, hdr_len, vlan_tag_insert, vlan_tag, - WQ_ENET_OFFLOAD_MODE_TSO, - eop, 1 /* SOP */, eop, loopback); -} struct enic; diff --git a/drivers/net/enic/enic_rxtx.c b/drivers/net/enic/enic_rxtx.c index 056a297..530e8a3 100644 --- a/drivers/net/enic/enic_rxtx.c +++ b/drivers/net/enic/enic_rxtx.c @@ -416,7 +416,7 @@ void enic_send_pkt(struct enic *enic, struct vnic_wq *wq, 0 /*wrid*/); } -uint16_t enicpmd_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, +uint16_t enic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) { uint16_t index; -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v2 06/11] enic: streamline mbuf handling in Tx path 2016-05-24 6:32 [PATCH v2 00/11] enic counter fixes and Tx optimization John Daley ` (4 preceding siblings ...) 2016-05-24 6:32 ` [PATCH v2 05/11] enic: remove some unused functions in Tx path John Daley @ 2016-05-24 6:32 ` John Daley 2016-05-24 6:32 ` [PATCH v2 07/11] enic: use Tx completion messages instead of descriptors John Daley ` (5 subsequent siblings) 11 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-05-24 6:32 UTC (permalink / raw) To: dev; +Cc: John Daley The list of mbufs held by the driver on Tx was allocated in chunks (a hold-over from the enic kernel mode driver). The structure used next pointers across chunks which led to cache misses. Allocate the array used to hold mbufs in flight on Tx with rte_zmalloc_socket(). Remove unnecessary fields from the structure and use head and tail pointers instead of next pointers. Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/base/enic_vnic_wq.h | 26 +++---------- drivers/net/enic/base/vnic_wq.c | 75 +++++++++--------------------------- drivers/net/enic/base/vnic_wq.h | 56 ++++++++++----------------- drivers/net/enic/enic.h | 24 ++++++++++++ drivers/net/enic/enic_main.c | 4 +- drivers/net/enic/enic_rxtx.c | 23 +++-------- 6 files changed, 75 insertions(+), 133 deletions(-) diff --git a/drivers/net/enic/base/enic_vnic_wq.h b/drivers/net/enic/base/enic_vnic_wq.h index b019109..55c5664 100644 --- a/drivers/net/enic/base/enic_vnic_wq.h +++ b/drivers/net/enic/base/enic_vnic_wq.h @@ -40,37 +40,21 @@ static inline void enic_vnic_post_wq_index(struct vnic_wq *wq) { - struct vnic_wq_buf *buf = wq->to_use; - /* Adding write memory barrier prevents compiler and/or CPU * reordering, thus avoiding descriptor posting before * descriptor is initialized. Otherwise, hardware can read * stale descriptor fields. */ wmb(); - iowrite32(buf->index, &wq->ctrl->posted_index); + iowrite32(wq->head_idx, &wq->ctrl->posted_index); } static inline void enic_vnic_post_wq(struct vnic_wq *wq, - void *os_buf, dma_addr_t dma_addr, - unsigned int len, int sop, - uint8_t desc_skip_cnt, uint8_t cq_entry, - uint8_t compressed_send, uint64_t wrid) + void *os_buf, uint8_t cq_entry) { - struct vnic_wq_buf *buf = wq->to_use; - - buf->sop = sop; - buf->cq_entry = cq_entry; - buf->compressed_send = compressed_send; - buf->desc_skip_cnt = desc_skip_cnt; - buf->os_buf = os_buf; - buf->dma_addr = dma_addr; - buf->len = len; - buf->wr_id = wrid; - - buf = buf->next; - wq->ring.desc_avail -= desc_skip_cnt; - wq->to_use = buf; + struct vnic_wq_buf *buf = &wq->bufs[wq->head_idx]; + buf->mb = os_buf; + wq->head_idx = enic_ring_incr(wq->ring.desc_count, wq->head_idx); if (cq_entry) enic_vnic_post_wq_index(wq); diff --git a/drivers/net/enic/base/vnic_wq.c b/drivers/net/enic/base/vnic_wq.c index a3ef417..ab81c7e 100644 --- a/drivers/net/enic/base/vnic_wq.c +++ b/drivers/net/enic/base/vnic_wq.c @@ -59,71 +59,30 @@ int vnic_wq_alloc_ring(struct vnic_dev *vdev, struct vnic_wq *wq, static int vnic_wq_alloc_bufs(struct vnic_wq *wq) { - struct vnic_wq_buf *buf; - unsigned int i, j, count = wq->ring.desc_count; - unsigned int blks = VNIC_WQ_BUF_BLKS_NEEDED(count); - - for (i = 0; i < blks; i++) { - wq->bufs[i] = kzalloc(VNIC_WQ_BUF_BLK_SZ(count), GFP_ATOMIC); - if (!wq->bufs[i]) - return -ENOMEM; - } - - for (i = 0; i < blks; i++) { - buf = wq->bufs[i]; - for (j = 0; j < VNIC_WQ_BUF_BLK_ENTRIES(count); j++) { - buf->index = i * VNIC_WQ_BUF_BLK_ENTRIES(count) + j; - buf->desc = (u8 *)wq->ring.descs + - wq->ring.desc_size * buf->index; - if (buf->index + 1 == count) { - buf->next = wq->bufs[0]; - break; - } else if (j + 1 == VNIC_WQ_BUF_BLK_ENTRIES(count)) { - buf->next = wq->bufs[i + 1]; - } else { - buf->next = buf + 1; - buf++; - } - } - } - - wq->to_use = wq->to_clean = wq->bufs[0]; - + unsigned int count = wq->ring.desc_count; + /* Allocate the mbuf ring */ + wq->bufs = (struct vnic_wq_buf *)rte_zmalloc_socket("wq->bufs", + sizeof(struct vnic_wq_buf) * count, + RTE_CACHE_LINE_SIZE, wq->socket_id); + wq->head_idx = 0; + wq->tail_idx = 0; + if (wq->bufs == NULL) + return -ENOMEM; return 0; } void vnic_wq_free(struct vnic_wq *wq) { struct vnic_dev *vdev; - unsigned int i; vdev = wq->vdev; vnic_dev_free_desc_ring(vdev, &wq->ring); - for (i = 0; i < VNIC_WQ_BUF_BLKS_MAX; i++) { - if (wq->bufs[i]) { - kfree(wq->bufs[i]); - wq->bufs[i] = NULL; - } - } - + rte_free(wq->bufs); wq->ctrl = NULL; } -int vnic_wq_mem_size(struct vnic_wq *wq, unsigned int desc_count, - unsigned int desc_size) -{ - int mem_size = 0; - - mem_size += vnic_dev_desc_ring_size(&wq->ring, desc_count, desc_size); - - mem_size += VNIC_WQ_BUF_BLKS_NEEDED(wq->ring.desc_count) * - VNIC_WQ_BUF_BLK_SZ(wq->ring.desc_count); - - return mem_size; -} - int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index, unsigned int desc_count, unsigned int desc_size) @@ -172,9 +131,8 @@ void vnic_wq_init_start(struct vnic_wq *wq, unsigned int cq_index, iowrite32(error_interrupt_offset, &wq->ctrl->error_interrupt_offset); iowrite32(0, &wq->ctrl->error_status); - wq->to_use = wq->to_clean = - &wq->bufs[fetch_index / VNIC_WQ_BUF_BLK_ENTRIES(count)] - [fetch_index % VNIC_WQ_BUF_BLK_ENTRIES(count)]; + wq->head_idx = fetch_index; + wq->tail_idx = wq->head_idx; } void vnic_wq_init(struct vnic_wq *wq, unsigned int cq_index, @@ -223,18 +181,21 @@ void vnic_wq_clean(struct vnic_wq *wq, void (*buf_clean)(struct vnic_wq *wq, struct vnic_wq_buf *buf)) { struct vnic_wq_buf *buf; + unsigned int to_clean = wq->tail_idx; - buf = wq->to_clean; + buf = &wq->bufs[to_clean]; while (vnic_wq_desc_used(wq) > 0) { (*buf_clean)(wq, buf); + to_clean = buf_idx_incr(wq->ring.desc_count, to_clean); - buf = wq->to_clean = buf->next; + buf = &wq->bufs[to_clean]; wq->ring.desc_avail++; } - wq->to_use = wq->to_clean = wq->bufs[0]; + wq->head_idx = 0; + wq->tail_idx = 0; iowrite32(0, &wq->ctrl->fetch_index); iowrite32(0, &wq->ctrl->posted_index); diff --git a/drivers/net/enic/base/vnic_wq.h b/drivers/net/enic/base/vnic_wq.h index d8660e4..a6759f5 100644 --- a/drivers/net/enic/base/vnic_wq.h +++ b/drivers/net/enic/base/vnic_wq.h @@ -64,40 +64,19 @@ struct vnic_wq_ctrl { u32 pad9; }; +/* 16 bytes */ struct vnic_wq_buf { - struct vnic_wq_buf *next; - dma_addr_t dma_addr; - void *os_buf; - unsigned int len; - unsigned int index; - int sop; - void *desc; - uint64_t wr_id; /* Cookie */ - uint8_t cq_entry; /* Gets completion event from hw */ - uint8_t desc_skip_cnt; /* Num descs to occupy */ - uint8_t compressed_send; /* Both hdr and payload in one desc */ + void *mb; }; -/* Break the vnic_wq_buf allocations into blocks of 32/64 entries */ -#define VNIC_WQ_BUF_MIN_BLK_ENTRIES 32 -#define VNIC_WQ_BUF_DFLT_BLK_ENTRIES 64 -#define VNIC_WQ_BUF_BLK_ENTRIES(entries) \ - ((unsigned int)((entries < VNIC_WQ_BUF_DFLT_BLK_ENTRIES) ? \ - VNIC_WQ_BUF_MIN_BLK_ENTRIES : VNIC_WQ_BUF_DFLT_BLK_ENTRIES)) -#define VNIC_WQ_BUF_BLK_SZ(entries) \ - (VNIC_WQ_BUF_BLK_ENTRIES(entries) * sizeof(struct vnic_wq_buf)) -#define VNIC_WQ_BUF_BLKS_NEEDED(entries) \ - DIV_ROUND_UP(entries, VNIC_WQ_BUF_BLK_ENTRIES(entries)) -#define VNIC_WQ_BUF_BLKS_MAX VNIC_WQ_BUF_BLKS_NEEDED(4096) - struct vnic_wq { unsigned int index; struct vnic_dev *vdev; struct vnic_wq_ctrl __iomem *ctrl; /* memory-mapped */ struct vnic_dev_ring ring; - struct vnic_wq_buf *bufs[VNIC_WQ_BUF_BLKS_MAX]; - struct vnic_wq_buf *to_use; - struct vnic_wq_buf *to_clean; + struct vnic_wq_buf *bufs; + unsigned int head_idx; + unsigned int tail_idx; unsigned int pkts_outstanding; unsigned int socket_id; }; @@ -114,11 +93,6 @@ static inline unsigned int vnic_wq_desc_used(struct vnic_wq *wq) return wq->ring.desc_count - wq->ring.desc_avail - 1; } -static inline void *vnic_wq_next_desc(struct vnic_wq *wq) -{ - return wq->to_use->desc; -} - #define PI_LOG2_CACHE_LINE_SIZE 5 #define PI_INDEX_BITS 12 #define PI_INDEX_MASK ((1U << PI_INDEX_BITS) - 1) @@ -191,6 +165,15 @@ static inline u64 vnic_cached_posted_index(dma_addr_t addr, unsigned int len, PI_PREFETCH_ADDR_MASK) << PI_PREFETCH_ADDR_OFF); } +static inline uint32_t +buf_idx_incr(uint32_t n_descriptors, uint32_t idx) +{ + idx++; + if (unlikely(idx == n_descriptors)) + idx = 0; + return idx; +} + static inline void vnic_wq_service(struct vnic_wq *wq, struct cq_desc *cq_desc, u16 completed_index, void (*buf_service)(struct vnic_wq *wq, @@ -198,21 +181,24 @@ static inline void vnic_wq_service(struct vnic_wq *wq, void *opaque) { struct vnic_wq_buf *buf; + unsigned int to_clean = wq->tail_idx; - buf = wq->to_clean; + buf = &wq->bufs[to_clean]; while (1) { (*buf_service)(wq, cq_desc, buf, opaque); wq->ring.desc_avail++; - wq->to_clean = buf->next; - if (buf->index == completed_index) + to_clean = buf_idx_incr(wq->ring.desc_count, to_clean); + + if (to_clean == completed_index) break; - buf = wq->to_clean; + buf = &wq->bufs[to_clean]; } + wq->tail_idx = to_clean; } void vnic_wq_free(struct vnic_wq *wq); diff --git a/drivers/net/enic/enic.h b/drivers/net/enic/enic.h index 5b58a65..fe186de 100644 --- a/drivers/net/enic/enic.h +++ b/drivers/net/enic/enic.h @@ -163,6 +163,30 @@ static inline struct enic *pmd_priv(struct rte_eth_dev *eth_dev) return (struct enic *)eth_dev->data->dev_private; } +static inline uint32_t +enic_ring_add(uint32_t n_descriptors, uint32_t i0, uint32_t i1) +{ + uint32_t d = i0 + i1; + d -= (d >= n_descriptors) ? n_descriptors : 0; + return d; +} + +static inline uint32_t +enic_ring_sub(uint32_t n_descriptors, uint32_t i0, uint32_t i1) +{ + int32_t d = i1 - i0; + return (uint32_t)((d < 0) ? ((int32_t)n_descriptors + d) : d); +} + +static inline uint32_t +enic_ring_incr(uint32_t n_descriptors, uint32_t idx) +{ + idx++; + if (unlikely(idx == n_descriptors)) + idx = 0; + return idx; +} + extern void enic_fdir_stats_get(struct enic *enic, struct rte_eth_fdir_stats *stats); extern int enic_fdir_add_fltr(struct enic *enic, diff --git a/drivers/net/enic/enic_main.c b/drivers/net/enic/enic_main.c index f41ef86..5bf5fcf 100644 --- a/drivers/net/enic/enic_main.c +++ b/drivers/net/enic/enic_main.c @@ -105,10 +105,10 @@ void enic_set_hdr_split_size(struct enic *enic, u16 split_hdr_size) void enic_free_wq_buf(__rte_unused struct vnic_wq *wq, struct vnic_wq_buf *buf) { - struct rte_mbuf *mbuf = (struct rte_mbuf *)buf->os_buf; + struct rte_mbuf *mbuf = (struct rte_mbuf *)buf->mb; rte_mempool_put(mbuf->pool, mbuf); - buf->os_buf = NULL; + buf->mb = NULL; } static void enic_log_q_error(struct enic *enic) diff --git a/drivers/net/enic/enic_rxtx.c b/drivers/net/enic/enic_rxtx.c index 530e8a3..ea31dfa 100644 --- a/drivers/net/enic/enic_rxtx.c +++ b/drivers/net/enic/enic_rxtx.c @@ -214,17 +214,6 @@ enic_cq_rx_to_pkt_flags(struct cq_desc *cqd, struct rte_mbuf *mbuf) mbuf->ol_flags = pkt_flags; } -static inline uint32_t -enic_ring_add(uint32_t n_descriptors, uint32_t i0, uint32_t i1) -{ - uint32_t d = i0 + i1; - RTE_ASSERT(i0 < n_descriptors); - RTE_ASSERT(i1 < n_descriptors); - d -= (d >= n_descriptors) ? n_descriptors : 0; - return d; -} - - uint16_t enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) @@ -376,12 +365,15 @@ void enic_send_pkt(struct enic *enic, struct vnic_wq *wq, uint8_t sop, uint8_t eop, uint8_t cq_entry, uint16_t ol_flags, uint16_t vlan_tag) { - struct wq_enet_desc *desc = vnic_wq_next_desc(wq); + struct wq_enet_desc *desc, *descs; uint16_t mss = 0; uint8_t vlan_tag_insert = 0; uint64_t bus_addr = (dma_addr_t) (tx_pkt->buf_physaddr + tx_pkt->data_off); + descs = (struct wq_enet_desc *)wq->ring.descs; + desc = descs + wq->head_idx; + if (sop) { if (ol_flags & PKT_TX_VLAN_PKT) vlan_tag_insert = 1; @@ -408,12 +400,7 @@ void enic_send_pkt(struct enic *enic, struct vnic_wq *wq, vlan_tag, 0 /* loopback */); - enic_vnic_post_wq(wq, (void *)tx_pkt, bus_addr, len, - sop, - 1 /*desc_skip_cnt*/, - cq_entry, - 0 /*compressed send*/, - 0 /*wrid*/); + enic_vnic_post_wq(wq, (void *)tx_pkt, cq_entry); } uint16_t enic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v2 07/11] enic: use Tx completion messages instead of descriptors 2016-05-24 6:32 [PATCH v2 00/11] enic counter fixes and Tx optimization John Daley ` (5 preceding siblings ...) 2016-05-24 6:32 ` [PATCH v2 06/11] enic: streamline mbuf handling " John Daley @ 2016-05-24 6:32 ` John Daley 2016-05-24 6:32 ` [PATCH v2 08/11] enic: refactor Tx mbuf recycling John Daley ` (4 subsequent siblings) 11 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-05-24 6:32 UTC (permalink / raw) To: dev; +Cc: John Daley The NIC can either DMA a separate completion message for each completed send or periodically just DMA an index of the last completed send. Switch to the second method which improves cache locality and performance. Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/base/vnic_wq.c | 1 + drivers/net/enic/base/vnic_wq.h | 3 +++ drivers/net/enic/enic_main.c | 43 ++++++++++++++++++++++++++++++++--------- drivers/net/enic/enic_rxtx.c | 11 +++++++---- 4 files changed, 45 insertions(+), 13 deletions(-) diff --git a/drivers/net/enic/base/vnic_wq.c b/drivers/net/enic/base/vnic_wq.c index ab81c7e..cfef1af 100644 --- a/drivers/net/enic/base/vnic_wq.c +++ b/drivers/net/enic/base/vnic_wq.c @@ -142,6 +142,7 @@ void vnic_wq_init(struct vnic_wq *wq, unsigned int cq_index, vnic_wq_init_start(wq, cq_index, 0, 0, error_interrupt_enable, error_interrupt_offset); + wq->last_completed_index = 0; } void vnic_wq_error_out(struct vnic_wq *wq, unsigned int error) diff --git a/drivers/net/enic/base/vnic_wq.h b/drivers/net/enic/base/vnic_wq.h index a6759f5..fe46bb4 100644 --- a/drivers/net/enic/base/vnic_wq.h +++ b/drivers/net/enic/base/vnic_wq.h @@ -38,6 +38,7 @@ #include "vnic_dev.h" #include "vnic_cq.h" +#include <rte_memzone.h> /* Work queue control */ struct vnic_wq_ctrl { @@ -79,6 +80,8 @@ struct vnic_wq { unsigned int tail_idx; unsigned int pkts_outstanding; unsigned int socket_id; + const struct rte_memzone *cqmsg_rz; + uint16_t last_completed_index; }; static inline unsigned int vnic_wq_desc_avail(struct vnic_wq *wq) diff --git a/drivers/net/enic/enic_main.c b/drivers/net/enic/enic_main.c index 5bf5fcf..eaa206e 100644 --- a/drivers/net/enic/enic_main.c +++ b/drivers/net/enic/enic_main.c @@ -97,7 +97,6 @@ enic_rxmbuf_queue_release(struct enic *enic, struct vnic_rq *rq) } } - void enic_set_hdr_split_size(struct enic *enic, u16 split_hdr_size) { vnic_set_hdr_split_size(enic->vdev, split_hdr_size); @@ -235,12 +234,26 @@ void enic_init_vnic_resources(struct enic *enic) unsigned int error_interrupt_enable = 1; unsigned int error_interrupt_offset = 0; unsigned int index = 0; + unsigned int cq_idx; for (index = 0; index < enic->rq_count; index++) { vnic_rq_init(&enic->rq[index], enic_cq_rq(enic, index), error_interrupt_enable, error_interrupt_offset); + + cq_idx = enic_cq_rq(enic, index); + vnic_cq_init(&enic->cq[cq_idx], + 0 /* flow_control_enable */, + 1 /* color_enable */, + 0 /* cq_head */, + 0 /* cq_tail */, + 1 /* cq_tail_color */, + 0 /* interrupt_enable */, + 1 /* cq_entry_enable */, + 0 /* cq_message_enable */, + 0 /* interrupt offset */, + 0 /* cq_message_addr */); } for (index = 0; index < enic->wq_count; index++) { @@ -248,22 +261,19 @@ void enic_init_vnic_resources(struct enic *enic) enic_cq_wq(enic, index), error_interrupt_enable, error_interrupt_offset); - } - vnic_dev_stats_clear(enic->vdev); - - for (index = 0; index < enic->cq_count; index++) { - vnic_cq_init(&enic->cq[index], + cq_idx = enic_cq_wq(enic, index); + vnic_cq_init(&enic->cq[cq_idx], 0 /* flow_control_enable */, 1 /* color_enable */, 0 /* cq_head */, 0 /* cq_tail */, 1 /* cq_tail_color */, 0 /* interrupt_enable */, - 1 /* cq_entry_enable */, - 0 /* cq_message_enable */, + 0 /* cq_entry_enable */, + 1 /* cq_message_enable */, 0 /* interrupt offset */, - 0 /* cq_message_addr */); + (u64)enic->wq[index].cqmsg_rz->phys_addr); } vnic_intr_init(&enic->intr, @@ -507,6 +517,7 @@ void enic_free_wq(void *txq) struct vnic_wq *wq = (struct vnic_wq *)txq; struct enic *enic = vnic_dev_priv(wq->vdev); + rte_memzone_free(wq->cqmsg_rz); vnic_wq_free(wq); vnic_cq_free(&enic->cq[enic->rq_count + wq->index]); } @@ -517,6 +528,8 @@ int enic_alloc_wq(struct enic *enic, uint16_t queue_idx, int err; struct vnic_wq *wq = &enic->wq[queue_idx]; unsigned int cq_index = enic_cq_wq(enic, queue_idx); + char name[NAME_MAX]; + static int instance; wq->socket_id = socket_id; if (nb_desc) { @@ -552,6 +565,18 @@ int enic_alloc_wq(struct enic *enic, uint16_t queue_idx, dev_err(enic, "error in allocation of cq for wq\n"); } + /* setup up CQ message */ + snprintf((char *)name, sizeof(name), + "vnic_cqmsg-%s-%d-%d", enic->bdf_name, queue_idx, + instance++); + + wq->cqmsg_rz = rte_memzone_reserve_aligned((const char *)name, + sizeof(uint32_t), + SOCKET_ID_ANY, 0, + ENIC_ALIGN); + if (!wq->cqmsg_rz) + return -ENOMEM; + return err; } diff --git a/drivers/net/enic/enic_rxtx.c b/drivers/net/enic/enic_rxtx.c index ea31dfa..2a54333 100644 --- a/drivers/net/enic/enic_rxtx.c +++ b/drivers/net/enic/enic_rxtx.c @@ -348,11 +348,14 @@ static int enic_wq_service(struct vnic_dev *vdev, struct cq_desc *cq_desc, unsigned int enic_cleanup_wq(struct enic *enic, struct vnic_wq *wq) { - unsigned int cq = enic_cq_wq(enic, wq->index); + u16 completed_index = *((uint32_t *)wq->cqmsg_rz->addr) & 0xffff; - /* Return the work done */ - return vnic_cq_service(&enic->cq[cq], - -1 /*wq_work_to_do*/, enic_wq_service, NULL); + if (wq->last_completed_index != completed_index) { + enic_wq_service(enic->vdev, NULL, 0, wq->index, + completed_index, NULL); + wq->last_completed_index = completed_index; + } + return 0; } void enic_post_wq_index(struct vnic_wq *wq) -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v2 08/11] enic: refactor Tx mbuf recycling 2016-05-24 6:32 [PATCH v2 00/11] enic counter fixes and Tx optimization John Daley ` (6 preceding siblings ...) 2016-05-24 6:32 ` [PATCH v2 07/11] enic: use Tx completion messages instead of descriptors John Daley @ 2016-05-24 6:32 ` John Daley 2016-05-24 6:32 ` [PATCH v2 09/11] enic: optimize the Tx function John Daley ` (3 subsequent siblings) 11 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-05-24 6:32 UTC (permalink / raw) To: dev; +Cc: John Daley Mbufs were returned to the pool one at a time. Use rte_mempool_put_bulk instead. There were muiltiple function calls for each buffer returned. Refactor this code into just 2 functions. Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/base/vnic_wq.h | 27 --------------------- drivers/net/enic/enic_rxtx.c | 54 ++++++++++++++++++++++++++--------------- 2 files changed, 35 insertions(+), 46 deletions(-) diff --git a/drivers/net/enic/base/vnic_wq.h b/drivers/net/enic/base/vnic_wq.h index fe46bb4..689b81c 100644 --- a/drivers/net/enic/base/vnic_wq.h +++ b/drivers/net/enic/base/vnic_wq.h @@ -177,33 +177,6 @@ buf_idx_incr(uint32_t n_descriptors, uint32_t idx) return idx; } -static inline void vnic_wq_service(struct vnic_wq *wq, - struct cq_desc *cq_desc, u16 completed_index, - void (*buf_service)(struct vnic_wq *wq, - struct cq_desc *cq_desc, struct vnic_wq_buf *buf, void *opaque), - void *opaque) -{ - struct vnic_wq_buf *buf; - unsigned int to_clean = wq->tail_idx; - - buf = &wq->bufs[to_clean]; - while (1) { - - (*buf_service)(wq, cq_desc, buf, opaque); - - wq->ring.desc_avail++; - - - to_clean = buf_idx_incr(wq->ring.desc_count, to_clean); - - if (to_clean == completed_index) - break; - - buf = &wq->bufs[to_clean]; - } - wq->tail_idx = to_clean; -} - void vnic_wq_free(struct vnic_wq *wq); int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index, unsigned int desc_count, unsigned int desc_size); diff --git a/drivers/net/enic/enic_rxtx.c b/drivers/net/enic/enic_rxtx.c index 2a54333..ec8d90a 100644 --- a/drivers/net/enic/enic_rxtx.c +++ b/drivers/net/enic/enic_rxtx.c @@ -326,33 +326,49 @@ enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, return nb_rx; } -static void enic_wq_free_buf(struct vnic_wq *wq, - __rte_unused struct cq_desc *cq_desc, - struct vnic_wq_buf *buf, - __rte_unused void *opaque) +static inline void enic_free_wq_bufs(struct vnic_wq *wq, u16 completed_index) { - enic_free_wq_buf(wq, buf); -} - -static int enic_wq_service(struct vnic_dev *vdev, struct cq_desc *cq_desc, - __rte_unused u8 type, u16 q_number, u16 completed_index, void *opaque) -{ - struct enic *enic = vnic_dev_priv(vdev); + struct vnic_wq_buf *buf; + struct rte_mbuf *m, *free[ENIC_MAX_WQ_DESCS]; + unsigned int nb_to_free, nb_free = 0, i; + struct rte_mempool *pool; + unsigned int tail_idx; + unsigned int desc_count = wq->ring.desc_count; + + nb_to_free = enic_ring_sub(desc_count, wq->tail_idx, completed_index) + + 1; + tail_idx = wq->tail_idx; + buf = &wq->bufs[tail_idx]; + pool = ((struct rte_mbuf *)buf->mb)->pool; + for (i = 0; i < nb_to_free; i++) { + buf = &wq->bufs[tail_idx]; + m = (struct rte_mbuf *)(buf->mb); + if (likely(m->pool == pool)) { + free[nb_free++] = m; + } else { + rte_mempool_put_bulk(pool, (void *)free, nb_free); + free[0] = m; + nb_free = 1; + pool = m->pool; + } + tail_idx = enic_ring_incr(desc_count, tail_idx); + buf->mb = NULL; + } - vnic_wq_service(&enic->wq[q_number], cq_desc, - completed_index, enic_wq_free_buf, - opaque); + rte_mempool_put_bulk(pool, (void **)free, nb_free); - return 0; + wq->tail_idx = tail_idx; + wq->ring.desc_avail += nb_to_free; } -unsigned int enic_cleanup_wq(struct enic *enic, struct vnic_wq *wq) +unsigned int enic_cleanup_wq(__rte_unused struct enic *enic, struct vnic_wq *wq) { - u16 completed_index = *((uint32_t *)wq->cqmsg_rz->addr) & 0xffff; + u16 completed_index; + + completed_index = *((uint32_t *)wq->cqmsg_rz->addr) & 0xffff; if (wq->last_completed_index != completed_index) { - enic_wq_service(enic->vdev, NULL, 0, wq->index, - completed_index, NULL); + enic_free_wq_bufs(wq, completed_index); wq->last_completed_index = completed_index; } return 0; -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v2 09/11] enic: optimize the Tx function 2016-05-24 6:32 [PATCH v2 00/11] enic counter fixes and Tx optimization John Daley ` (7 preceding siblings ...) 2016-05-24 6:32 ` [PATCH v2 08/11] enic: refactor Tx mbuf recycling John Daley @ 2016-05-24 6:32 ` John Daley 2016-05-30 10:05 ` Azarewicz, PiotrX T 2016-05-24 6:32 ` [PATCH v2 10/11] enic: remove unused files and functions and variables John Daley ` (2 subsequent siblings) 11 siblings, 1 reply; 31+ messages in thread From: John Daley @ 2016-05-24 6:32 UTC (permalink / raw) To: dev; +Cc: John Daley, Nelson Escobar Reduce host CPU overhead of Tx packet processing: * Use local variables inside per packet loop instead of fields in structs. * Factor book keeping and conditionals out of the per packet loop where possible. * Post buffers to the nic at a maximum of every 64 packets Signed-off-by: Nelson Escobar <neescoba@cisco.com> Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/base/vnic_wq.h | 1 + drivers/net/enic/enic_res.h | 2 +- drivers/net/enic/enic_rxtx.c | 167 +++++++++++++++++++--------------------- 3 files changed, 83 insertions(+), 87 deletions(-) diff --git a/drivers/net/enic/base/vnic_wq.h b/drivers/net/enic/base/vnic_wq.h index 689b81c..7a66813 100644 --- a/drivers/net/enic/base/vnic_wq.h +++ b/drivers/net/enic/base/vnic_wq.h @@ -67,6 +67,7 @@ struct vnic_wq_ctrl { /* 16 bytes */ struct vnic_wq_buf { + struct rte_mempool *pool; void *mb; }; diff --git a/drivers/net/enic/enic_res.h b/drivers/net/enic/enic_res.h index 955db71..3c8e303 100644 --- a/drivers/net/enic/enic_res.h +++ b/drivers/net/enic/enic_res.h @@ -53,7 +53,7 @@ #define ENIC_NON_TSO_MAX_DESC 16 #define ENIC_DEFAULT_RX_FREE_THRESH 32 -#define ENIC_TX_POST_THRESH (ENIC_MIN_WQ_DESCS / 2) +#define ENIC_TX_XMIT_MAX 64 #define ENIC_SETTING(enic, f) ((enic->config.flags & VENETF_##f) ? 1 : 0) diff --git a/drivers/net/enic/enic_rxtx.c b/drivers/net/enic/enic_rxtx.c index ec8d90a..ba15670 100644 --- a/drivers/net/enic/enic_rxtx.c +++ b/drivers/net/enic/enic_rxtx.c @@ -374,114 +374,109 @@ unsigned int enic_cleanup_wq(__rte_unused struct enic *enic, struct vnic_wq *wq) return 0; } -void enic_post_wq_index(struct vnic_wq *wq) -{ - enic_vnic_post_wq_index(wq); -} - -void enic_send_pkt(struct enic *enic, struct vnic_wq *wq, - struct rte_mbuf *tx_pkt, unsigned short len, - uint8_t sop, uint8_t eop, uint8_t cq_entry, - uint16_t ol_flags, uint16_t vlan_tag) -{ - struct wq_enet_desc *desc, *descs; - uint16_t mss = 0; - uint8_t vlan_tag_insert = 0; - uint64_t bus_addr = (dma_addr_t) - (tx_pkt->buf_physaddr + tx_pkt->data_off); - - descs = (struct wq_enet_desc *)wq->ring.descs; - desc = descs + wq->head_idx; - - if (sop) { - if (ol_flags & PKT_TX_VLAN_PKT) - vlan_tag_insert = 1; - - if (enic->hw_ip_checksum) { - if (ol_flags & PKT_TX_IP_CKSUM) - mss |= ENIC_CALC_IP_CKSUM; - - if (ol_flags & PKT_TX_TCP_UDP_CKSUM) - mss |= ENIC_CALC_TCP_UDP_CKSUM; - } - } - - wq_enet_desc_enc(desc, - bus_addr, - len, - mss, - 0 /* header_length */, - 0 /* offload_mode WQ_ENET_OFFLOAD_MODE_CSUM */, - eop, - cq_entry, - 0 /* fcoe_encap */, - vlan_tag_insert, - vlan_tag, - 0 /* loopback */); - - enic_vnic_post_wq(wq, (void *)tx_pkt, cq_entry); -} - uint16_t enic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) { uint16_t index; - unsigned int frags; - unsigned int pkt_len; - unsigned int seg_len; - unsigned int inc_len; + unsigned int pkt_len, data_len; unsigned int nb_segs; - struct rte_mbuf *tx_pkt, *next_tx_pkt; + struct rte_mbuf *tx_pkt; struct vnic_wq *wq = (struct vnic_wq *)tx_queue; struct enic *enic = vnic_dev_priv(wq->vdev); unsigned short vlan_id; unsigned short ol_flags; - uint8_t last_seg, eop; - unsigned int host_tx_descs = 0; + unsigned int wq_desc_avail; + int head_idx; + struct vnic_wq_buf *buf; + unsigned int hw_ip_cksum_enabled; + unsigned int desc_count; + struct wq_enet_desc *descs, *desc_p, desc_tmp; + uint16_t mss; + uint8_t vlan_tag_insert; + uint8_t eop; + uint64_t bus_addr; + enic_cleanup_wq(enic, wq); + wq_desc_avail = vnic_wq_desc_avail(wq); + head_idx = wq->head_idx; + desc_count = wq->ring.desc_count; + + nb_pkts = RTE_MIN(nb_pkts, ENIC_TX_XMIT_MAX); + + hw_ip_cksum_enabled = enic->hw_ip_checksum; for (index = 0; index < nb_pkts; index++) { tx_pkt = *tx_pkts++; - inc_len = 0; nb_segs = tx_pkt->nb_segs; - if (nb_segs > vnic_wq_desc_avail(wq)) { + if (nb_segs > wq_desc_avail) { if (index > 0) - enic_post_wq_index(wq); - - /* wq cleanup and try again */ - if (!enic_cleanup_wq(enic, wq) || - (nb_segs > vnic_wq_desc_avail(wq))) { - return index; - } + goto post; + goto done; } pkt_len = tx_pkt->pkt_len; + data_len = tx_pkt->data_len; vlan_id = tx_pkt->vlan_tci; ol_flags = tx_pkt->ol_flags; - for (frags = 0; inc_len < pkt_len; frags++) { - if (!tx_pkt) - break; - next_tx_pkt = tx_pkt->next; - seg_len = tx_pkt->data_len; - inc_len += seg_len; - - host_tx_descs++; - last_seg = 0; - eop = 0; - if ((pkt_len == inc_len) || !next_tx_pkt) { - eop = 1; - /* post if last packet in batch or > thresh */ - if ((index == (nb_pkts - 1)) || - (host_tx_descs > ENIC_TX_POST_THRESH)) { - last_seg = 1; - host_tx_descs = 0; - } + + mss = 0; + vlan_tag_insert = 0; + bus_addr = (dma_addr_t) + (tx_pkt->buf_physaddr + tx_pkt->data_off); + + descs = (struct wq_enet_desc *)wq->ring.descs; + desc_p = descs + head_idx; + + eop = (data_len == pkt_len); + + if (ol_flags & PKT_TX_VLAN_PKT) + vlan_tag_insert = 1; + + if (hw_ip_cksum_enabled && (ol_flags & PKT_TX_IP_CKSUM)) + mss |= ENIC_CALC_IP_CKSUM; + + if (hw_ip_cksum_enabled && (ol_flags & PKT_TX_TCP_UDP_CKSUM)) + mss |= ENIC_CALC_TCP_UDP_CKSUM; + + wq_enet_desc_enc(&desc_tmp, bus_addr, data_len, mss, 0, 0, eop, + eop, 0, vlan_tag_insert, vlan_id, 0); + + *desc_p = desc_tmp; + buf = &wq->bufs[head_idx]; + buf->mb = (void *)tx_pkt; + head_idx = enic_ring_incr(desc_count, head_idx); + wq_desc_avail--; + + if (!eop) { + for (tx_pkt = tx_pkt->next; tx_pkt; tx_pkt = + tx_pkt->next) { + data_len = tx_pkt->data_len; + + if (tx_pkt->next == NULL) + eop = 1; + desc_p = descs + head_idx; + bus_addr = (dma_addr_t)(tx_pkt->buf_physaddr + + tx_pkt->data_off); + wq_enet_desc_enc((struct wq_enet_desc *) + &desc_tmp, bus_addr, data_len, + mss, 0, 0, eop, eop, 0, + vlan_tag_insert, vlan_id, 0); + + *desc_p = desc_tmp; + buf = &wq->bufs[head_idx]; + buf->mb = (void *)tx_pkt; + head_idx = enic_ring_incr(desc_count, head_idx); + wq_desc_avail--; } - enic_send_pkt(enic, wq, tx_pkt, (unsigned short)seg_len, - !frags, eop, last_seg, ol_flags, vlan_id); - tx_pkt = next_tx_pkt; } } + post: + rte_wmb(); + iowrite32(head_idx, &wq->ctrl->posted_index); + done: + wq->ring.desc_avail = wq_desc_avail; + wq->head_idx = head_idx; - enic_cleanup_wq(enic, wq); return index; } + + -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* Re: [PATCH v2 09/11] enic: optimize the Tx function 2016-05-24 6:32 ` [PATCH v2 09/11] enic: optimize the Tx function John Daley @ 2016-05-30 10:05 ` Azarewicz, PiotrX T 0 siblings, 0 replies; 31+ messages in thread From: Azarewicz, PiotrX T @ 2016-05-30 10:05 UTC (permalink / raw) To: John Daley, Nelson Escobar; +Cc: dev Hi, > uint16_t enic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, > uint16_t nb_pkts) > { > uint16_t index; > - unsigned int frags; > - unsigned int pkt_len; > - unsigned int seg_len; > - unsigned int inc_len; > + unsigned int pkt_len, data_len; > unsigned int nb_segs; > - struct rte_mbuf *tx_pkt, *next_tx_pkt; > + struct rte_mbuf *tx_pkt; > struct vnic_wq *wq = (struct vnic_wq *)tx_queue; > struct enic *enic = vnic_dev_priv(wq->vdev); > unsigned short vlan_id; > unsigned short ol_flags; Above ol_flags should be uint64_t. > - uint8_t last_seg, eop; > - unsigned int host_tx_descs = 0; > + unsigned int wq_desc_avail; > + int head_idx; > + struct vnic_wq_buf *buf; > + unsigned int hw_ip_cksum_enabled; > + unsigned int desc_count; > + struct wq_enet_desc *descs, *desc_p, desc_tmp; > + uint16_t mss; > + uint8_t vlan_tag_insert; > + uint8_t eop; > + uint64_t bus_addr; > > + enic_cleanup_wq(enic, wq); > + wq_desc_avail = vnic_wq_desc_avail(wq); > + head_idx = wq->head_idx; > + desc_count = wq->ring.desc_count; > + > + nb_pkts = RTE_MIN(nb_pkts, ENIC_TX_XMIT_MAX); > + > + hw_ip_cksum_enabled = enic->hw_ip_checksum; > for (index = 0; index < nb_pkts; index++) { > tx_pkt = *tx_pkts++; > - inc_len = 0; > nb_segs = tx_pkt->nb_segs; > - if (nb_segs > vnic_wq_desc_avail(wq)) { > + if (nb_segs > wq_desc_avail) { > if (index > 0) > - enic_post_wq_index(wq); > - > - /* wq cleanup and try again */ > - if (!enic_cleanup_wq(enic, wq) || > - (nb_segs > vnic_wq_desc_avail(wq))) { > - return index; > - } > + goto post; > + goto done; > } > > pkt_len = tx_pkt->pkt_len; > + data_len = tx_pkt->data_len; > vlan_id = tx_pkt->vlan_tci; > ol_flags = tx_pkt->ol_flags; Cause you may miss a lot of flags in here. Piotr > - for (frags = 0; inc_len < pkt_len; frags++) { > - if (!tx_pkt) > - break; > - next_tx_pkt = tx_pkt->next; > - seg_len = tx_pkt->data_len; > - inc_len += seg_len; > - > - host_tx_descs++; > - last_seg = 0; > - eop = 0; > - if ((pkt_len == inc_len) || !next_tx_pkt) { > - eop = 1; > - /* post if last packet in batch or > thresh */ > - if ((index == (nb_pkts - 1)) || > - (host_tx_descs > ENIC_TX_POST_THRESH)) > { > - last_seg = 1; > - host_tx_descs = 0; > - } > + > + mss = 0; > + vlan_tag_insert = 0; > + bus_addr = (dma_addr_t) > + (tx_pkt->buf_physaddr + tx_pkt->data_off); > + > + descs = (struct wq_enet_desc *)wq->ring.descs; > + desc_p = descs + head_idx; > + > + eop = (data_len == pkt_len); > + > + if (ol_flags & PKT_TX_VLAN_PKT) > + vlan_tag_insert = 1; > + > + if (hw_ip_cksum_enabled && (ol_flags & > PKT_TX_IP_CKSUM)) > + mss |= ENIC_CALC_IP_CKSUM; > + > + if (hw_ip_cksum_enabled && (ol_flags & > PKT_TX_TCP_UDP_CKSUM)) > + mss |= ENIC_CALC_TCP_UDP_CKSUM; > + > + wq_enet_desc_enc(&desc_tmp, bus_addr, data_len, mss, 0, > 0, eop, > + eop, 0, vlan_tag_insert, vlan_id, 0); > + > + *desc_p = desc_tmp; > + buf = &wq->bufs[head_idx]; > + buf->mb = (void *)tx_pkt; > + head_idx = enic_ring_incr(desc_count, head_idx); > + wq_desc_avail--; > + > + if (!eop) { > + for (tx_pkt = tx_pkt->next; tx_pkt; tx_pkt = > + tx_pkt->next) { > + data_len = tx_pkt->data_len; > + > + if (tx_pkt->next == NULL) > + eop = 1; > + desc_p = descs + head_idx; > + bus_addr = (dma_addr_t)(tx_pkt- > >buf_physaddr > + + tx_pkt->data_off); > + wq_enet_desc_enc((struct wq_enet_desc *) > + &desc_tmp, bus_addr, > data_len, > + mss, 0, 0, eop, eop, 0, > + vlan_tag_insert, vlan_id, 0); > + > + *desc_p = desc_tmp; > + buf = &wq->bufs[head_idx]; > + buf->mb = (void *)tx_pkt; > + head_idx = enic_ring_incr(desc_count, > head_idx); > + wq_desc_avail--; > } > - enic_send_pkt(enic, wq, tx_pkt, (unsigned > short)seg_len, > - !frags, eop, last_seg, ol_flags, vlan_id); > - tx_pkt = next_tx_pkt; > } > } > + post: > + rte_wmb(); > + iowrite32(head_idx, &wq->ctrl->posted_index); > + done: > + wq->ring.desc_avail = wq_desc_avail; > + wq->head_idx = head_idx; > > - enic_cleanup_wq(enic, wq); > return index; > } > + > + > -- > 2.7.0 ^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v2 10/11] enic: remove unused files and functions and variables 2016-05-24 6:32 [PATCH v2 00/11] enic counter fixes and Tx optimization John Daley ` (8 preceding siblings ...) 2016-05-24 6:32 ` [PATCH v2 09/11] enic: optimize the Tx function John Daley @ 2016-05-24 6:32 ` John Daley 2016-05-24 6:32 ` [PATCH v2 11/11] enic: add an enic assert macro John Daley 2016-06-03 0:22 ` [PATCH v3 00/13] enic counter fixes and Tx optimization John Daley 11 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-05-24 6:32 UTC (permalink / raw) To: dev; +Cc: John Daley Remove some files, functions and variables left unused after Tx performance improvements. Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/base/enic_vnic_wq.h | 63 ------------------------------------ drivers/net/enic/base/vnic_cq.h | 44 ------------------------- drivers/net/enic/base/vnic_wq.c | 4 +-- drivers/net/enic/base/vnic_wq.h | 6 +--- drivers/net/enic/enic.h | 1 - drivers/net/enic/enic_main.c | 2 +- drivers/net/enic/enic_rxtx.c | 1 - 7 files changed, 4 insertions(+), 117 deletions(-) delete mode 100644 drivers/net/enic/base/enic_vnic_wq.h diff --git a/drivers/net/enic/base/enic_vnic_wq.h b/drivers/net/enic/base/enic_vnic_wq.h deleted file mode 100644 index 55c5664..0000000 --- a/drivers/net/enic/base/enic_vnic_wq.h +++ /dev/null @@ -1,63 +0,0 @@ -/* - * Copyright 2008-2015 Cisco Systems, Inc. All rights reserved. - * Copyright 2007 Nuova Systems, Inc. All rights reserved. - * - * Copyright (c) 2015, Cisco Systems, Inc. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in - * the documentation and/or other materials provided with the - * distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS - * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE - * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, - * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, - * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; - * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER - * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN - * ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * - */ - -#ifndef _ENIC_VNIC_WQ_H_ -#define _ENIC_VNIC_WQ_H_ - -#include "vnic_dev.h" -#include "vnic_cq.h" - -static inline void enic_vnic_post_wq_index(struct vnic_wq *wq) -{ - /* Adding write memory barrier prevents compiler and/or CPU - * reordering, thus avoiding descriptor posting before - * descriptor is initialized. Otherwise, hardware can read - * stale descriptor fields. - */ - wmb(); - iowrite32(wq->head_idx, &wq->ctrl->posted_index); -} - -static inline void enic_vnic_post_wq(struct vnic_wq *wq, - void *os_buf, uint8_t cq_entry) -{ - struct vnic_wq_buf *buf = &wq->bufs[wq->head_idx]; - buf->mb = os_buf; - wq->head_idx = enic_ring_incr(wq->ring.desc_count, wq->head_idx); - - if (cq_entry) - enic_vnic_post_wq_index(wq); -} - -#endif /* _ENIC_VNIC_WQ_H_ */ diff --git a/drivers/net/enic/base/vnic_cq.h b/drivers/net/enic/base/vnic_cq.h index 922391b..13ab87c 100644 --- a/drivers/net/enic/base/vnic_cq.h +++ b/drivers/net/enic/base/vnic_cq.h @@ -90,50 +90,6 @@ struct vnic_cq { #endif }; -static inline unsigned int vnic_cq_service(struct vnic_cq *cq, - unsigned int work_to_do, - int (*q_service)(struct vnic_dev *vdev, struct cq_desc *cq_desc, - u8 type, u16 q_number, u16 completed_index, void *opaque), - void *opaque) -{ - struct cq_desc *cq_desc; - unsigned int work_done = 0; - u16 q_number, completed_index; - u8 type, color; - struct rte_mbuf **rx_pkts = opaque; - unsigned int ret; - - cq_desc = (struct cq_desc *)((u8 *)cq->ring.descs + - cq->ring.desc_size * cq->to_clean); - cq_desc_dec(cq_desc, &type, &color, - &q_number, &completed_index); - - while (color != cq->last_color) { - if (opaque) - opaque = (void *)&(rx_pkts[work_done]); - - ret = (*q_service)(cq->vdev, cq_desc, type, - q_number, completed_index, opaque); - cq->to_clean++; - if (cq->to_clean == cq->ring.desc_count) { - cq->to_clean = 0; - cq->last_color = cq->last_color ? 0 : 1; - } - - cq_desc = (struct cq_desc *)((u8 *)cq->ring.descs + - cq->ring.desc_size * cq->to_clean); - cq_desc_dec(cq_desc, &type, &color, - &q_number, &completed_index); - - if (ret) - work_done++; - if (work_done >= work_to_do) - break; - } - - return work_done; -} - void vnic_cq_free(struct vnic_cq *cq); int vnic_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index, unsigned int socket_id, diff --git a/drivers/net/enic/base/vnic_wq.c b/drivers/net/enic/base/vnic_wq.c index cfef1af..9b9ff4d 100644 --- a/drivers/net/enic/base/vnic_wq.c +++ b/drivers/net/enic/base/vnic_wq.c @@ -179,7 +179,7 @@ int vnic_wq_disable(struct vnic_wq *wq) } void vnic_wq_clean(struct vnic_wq *wq, - void (*buf_clean)(struct vnic_wq *wq, struct vnic_wq_buf *buf)) + void (*buf_clean)(struct vnic_wq_buf *buf)) { struct vnic_wq_buf *buf; unsigned int to_clean = wq->tail_idx; @@ -188,7 +188,7 @@ void vnic_wq_clean(struct vnic_wq *wq, while (vnic_wq_desc_used(wq) > 0) { - (*buf_clean)(wq, buf); + (*buf_clean)(buf); to_clean = buf_idx_incr(wq->ring.desc_count, to_clean); buf = &wq->bufs[to_clean]; diff --git a/drivers/net/enic/base/vnic_wq.h b/drivers/net/enic/base/vnic_wq.h index 7a66813..38a217f 100644 --- a/drivers/net/enic/base/vnic_wq.h +++ b/drivers/net/enic/base/vnic_wq.h @@ -79,7 +79,6 @@ struct vnic_wq { struct vnic_wq_buf *bufs; unsigned int head_idx; unsigned int tail_idx; - unsigned int pkts_outstanding; unsigned int socket_id; const struct rte_memzone *cqmsg_rz; uint16_t last_completed_index; @@ -193,8 +192,5 @@ unsigned int vnic_wq_error_status(struct vnic_wq *wq); void vnic_wq_enable(struct vnic_wq *wq); int vnic_wq_disable(struct vnic_wq *wq); void vnic_wq_clean(struct vnic_wq *wq, - void (*buf_clean)(struct vnic_wq *wq, struct vnic_wq_buf *buf)); -int vnic_wq_mem_size(struct vnic_wq *wq, unsigned int desc_count, - unsigned int desc_size); - + void (*buf_clean)(struct vnic_wq_buf *buf)); #endif /* _VNIC_WQ_H_ */ diff --git a/drivers/net/enic/enic.h b/drivers/net/enic/enic.h index fe186de..4eb28ee 100644 --- a/drivers/net/enic/enic.h +++ b/drivers/net/enic/enic.h @@ -233,5 +233,4 @@ uint16_t enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); uint16_t enic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts); -void enic_free_wq_buf(__rte_unused struct vnic_wq *wq, struct vnic_wq_buf *buf); #endif /* _ENIC_H_ */ diff --git a/drivers/net/enic/enic_main.c b/drivers/net/enic/enic_main.c index eaa206e..996f999 100644 --- a/drivers/net/enic/enic_main.c +++ b/drivers/net/enic/enic_main.c @@ -102,7 +102,7 @@ void enic_set_hdr_split_size(struct enic *enic, u16 split_hdr_size) vnic_set_hdr_split_size(enic->vdev, split_hdr_size); } -void enic_free_wq_buf(__rte_unused struct vnic_wq *wq, struct vnic_wq_buf *buf) +static void enic_free_wq_buf(struct vnic_wq_buf *buf) { struct rte_mbuf *mbuf = (struct rte_mbuf *)buf->mb; diff --git a/drivers/net/enic/enic_rxtx.c b/drivers/net/enic/enic_rxtx.c index ba15670..a04ebd0 100644 --- a/drivers/net/enic/enic_rxtx.c +++ b/drivers/net/enic/enic_rxtx.c @@ -37,7 +37,6 @@ #include "enic_compat.h" #include "rq_enet_desc.h" #include "enic.h" -#include "enic_vnic_wq.h" #define RTE_PMD_USE_PREFETCH -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v2 11/11] enic: add an enic assert macro 2016-05-24 6:32 [PATCH v2 00/11] enic counter fixes and Tx optimization John Daley ` (9 preceding siblings ...) 2016-05-24 6:32 ` [PATCH v2 10/11] enic: remove unused files and functions and variables John Daley @ 2016-05-24 6:32 ` John Daley 2016-06-03 0:22 ` [PATCH v3 00/13] enic counter fixes and Tx optimization John Daley 11 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-05-24 6:32 UTC (permalink / raw) To: dev; +Cc: John Daley Add an ASSERT macro for the enic driver which is enabled when the log level is >= RTE_LOG_DEBUG. Assert that number of mbufs to return to the pool in the Tx function is never greater than the max allowed. Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/enic.h | 12 ++++++++++++ drivers/net/enic/enic_rxtx.c | 1 + 2 files changed, 13 insertions(+) diff --git a/drivers/net/enic/enic.h b/drivers/net/enic/enic.h index 4eb28ee..4b76e6d 100644 --- a/drivers/net/enic/enic.h +++ b/drivers/net/enic/enic.h @@ -187,6 +187,18 @@ enic_ring_incr(uint32_t n_descriptors, uint32_t idx) return idx; } +#if RTE_LOG_LEVEL >= RTE_LOG_DEBUG +#define ENIC_ASSERT(cond) \ + do { \ + if (unlikely(!(cond))) { \ + rte_panic("line %d\tassert \"" #cond "\"" \ + "failed\n", __LINE__); \ + } \ + } while (0) +#else +#define ENIC_ASSERT(cond) do {} while (0) +#endif + extern void enic_fdir_stats_get(struct enic *enic, struct rte_eth_fdir_stats *stats); extern int enic_fdir_add_fltr(struct enic *enic, diff --git a/drivers/net/enic/enic_rxtx.c b/drivers/net/enic/enic_rxtx.c index a04ebd0..7527bce 100644 --- a/drivers/net/enic/enic_rxtx.c +++ b/drivers/net/enic/enic_rxtx.c @@ -343,6 +343,7 @@ static inline void enic_free_wq_bufs(struct vnic_wq *wq, u16 completed_index) buf = &wq->bufs[tail_idx]; m = (struct rte_mbuf *)(buf->mb); if (likely(m->pool == pool)) { + ENIC_ASSERT(nb_free < ENIC_MAX_WQ_DESCS); free[nb_free++] = m; } else { rte_mempool_put_bulk(pool, (void *)free, nb_free); -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v3 00/13] enic counter fixes and Tx optimization 2016-05-24 6:32 [PATCH v2 00/11] enic counter fixes and Tx optimization John Daley ` (10 preceding siblings ...) 2016-05-24 6:32 ` [PATCH v2 11/11] enic: add an enic assert macro John Daley @ 2016-06-03 0:22 ` John Daley 2016-06-03 0:22 ` [PATCH v3 01/13] enic: fix Rx drop counters John Daley ` (13 more replies) 11 siblings, 14 replies; 31+ messages in thread From: John Daley @ 2016-06-03 0:22 UTC (permalink / raw) To: dev; +Cc: bruce.richarsdon, John Daley The first 3 patches are related to drop counters. The remaining patches make up refactoring, cleanup bug fixes and optimization of the Tx path. Changes since v2 are: - Piotr Azarewicz's ol_flags patch http://www.dpdk.org/dev/patchwork/patch/12642 - fix Tx IP and UDP/TCP checksum offload John Daley (13): enic: fix Rx drop counters enic: drop bad packets and remove unused Rx error flag enic: count truncated packets enic: put Tx and Rx functions into same file enic: remove some unused functions in Tx path enic: streamline mbuf handling in Tx path enic: use Tx completion messages instead of descriptors enic: refactor Tx mbuf recycling enic: optimize the Tx function enic: remove unused files and functions and variables enic: add an enic assert macro enic: expand local Tx mbuf flags variable to 64-bits enic: fix Tx IP and UDP/TCP checksum offload drivers/net/enic/Makefile | 2 +- drivers/net/enic/base/enic_vnic_wq.h | 79 ------ drivers/net/enic/base/vnic_cq.h | 44 ---- drivers/net/enic/base/vnic_wq.c | 80 ++---- drivers/net/enic/base/vnic_wq.h | 118 ++------- drivers/net/enic/enic.h | 48 +++- drivers/net/enic/enic_ethdev.c | 67 +---- drivers/net/enic/enic_main.c | 156 +++++------ drivers/net/enic/enic_res.h | 80 +----- drivers/net/enic/enic_rx.c | 351 ------------------------- drivers/net/enic/enic_rxtx.c | 490 +++++++++++++++++++++++++++++++++++ 11 files changed, 642 insertions(+), 873 deletions(-) delete mode 100644 drivers/net/enic/base/enic_vnic_wq.h delete mode 100644 drivers/net/enic/enic_rx.c create mode 100644 drivers/net/enic/enic_rxtx.c -- 2.7.0 ^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v3 01/13] enic: fix Rx drop counters 2016-06-03 0:22 ` [PATCH v3 00/13] enic counter fixes and Tx optimization John Daley @ 2016-06-03 0:22 ` John Daley 2016-06-03 0:22 ` [PATCH v3 02/13] enic: drop bad packets and remove unused Rx error flag John Daley ` (12 subsequent siblings) 13 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-06-03 0:22 UTC (permalink / raw) To: dev; +Cc: bruce.richarsdon, John Daley rx_no_bufs is a hardware counter of packets dropped on the interface due to no host buffers and should be used to update r_stats->imissed counter instead of rx_nombuf. Include rx_drop in ierrors. rx_drop is incremented if packets arrive when the receive queue is disabled. Add a structure and functions for initializing and clearing software counters. Add count of Rx mbuf allocation failures (rx_nombuf) as the first counter. Fixes: fefed3d1e62c ("enic: new driver") Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/enic.h | 7 +++++++ drivers/net/enic/enic_main.c | 24 +++++++++++++++++++++--- drivers/net/enic/enic_rx.c | 5 +---- 3 files changed, 29 insertions(+), 7 deletions(-) diff --git a/drivers/net/enic/enic.h b/drivers/net/enic/enic.h index 09f3853..584d49b 100644 --- a/drivers/net/enic/enic.h +++ b/drivers/net/enic/enic.h @@ -91,6 +91,10 @@ struct enic_fdir { struct enic_fdir_node *nodes[ENICPMD_FDIR_MAX]; }; +struct enic_soft_stats { + rte_atomic64_t rx_nombuf; +}; + /* Per-instance private data structure */ struct enic { struct enic *next; @@ -133,6 +137,9 @@ struct enic { /* interrupt resource */ struct vnic_intr intr; unsigned int intr_count; + + /* software counters */ + struct enic_soft_stats soft_stats; }; static inline unsigned int enic_cq_rq(__rte_unused struct enic *enic, unsigned int rq) diff --git a/drivers/net/enic/enic_main.c b/drivers/net/enic/enic_main.c index bbbe660..c002ef3 100644 --- a/drivers/net/enic/enic_main.c +++ b/drivers/net/enic/enic_main.c @@ -211,15 +211,30 @@ void enic_send_pkt(struct enic *enic, struct vnic_wq *wq, 0 /*wrid*/); } +static void enic_clear_soft_stats(struct enic *enic) +{ + struct enic_soft_stats *soft_stats = &enic->soft_stats; + rte_atomic64_clear(&soft_stats->rx_nombuf); +} + +static void enic_init_soft_stats(struct enic *enic) +{ + struct enic_soft_stats *soft_stats = &enic->soft_stats; + rte_atomic64_init(&soft_stats->rx_nombuf); + enic_clear_soft_stats(enic); +} + void enic_dev_stats_clear(struct enic *enic) { if (vnic_dev_stats_clear(enic->vdev)) dev_err(enic, "Error in clearing stats\n"); + enic_clear_soft_stats(enic); } void enic_dev_stats_get(struct enic *enic, struct rte_eth_stats *r_stats) { struct vnic_stats *stats; + struct enic_soft_stats *soft_stats; if (vnic_dev_stats_dump(enic->vdev, &stats)) { dev_err(enic, "Error in getting stats\n"); @@ -232,12 +247,13 @@ void enic_dev_stats_get(struct enic *enic, struct rte_eth_stats *r_stats) r_stats->ibytes = stats->rx.rx_bytes_ok; r_stats->obytes = stats->tx.tx_bytes_ok; - r_stats->ierrors = stats->rx.rx_errors; + r_stats->ierrors = stats->rx.rx_errors + stats->rx.rx_drop; r_stats->oerrors = stats->tx.tx_errors; - r_stats->imissed = stats->rx.rx_drop; + r_stats->imissed = stats->rx.rx_no_bufs; - r_stats->rx_nombuf = stats->rx.rx_no_bufs; + soft_stats = &enic->soft_stats; + r_stats->rx_nombuf = rte_atomic64_read(&soft_stats->rx_nombuf); } void enic_del_mac_address(struct enic *enic) @@ -795,6 +811,8 @@ int enic_setup_finish(struct enic *enic) { int ret; + enic_init_soft_stats(enic); + ret = enic_set_rss_nic_cfg(enic); if (ret) { dev_err(enic, "Failed to config nic, aborting.\n"); diff --git a/drivers/net/enic/enic_rx.c b/drivers/net/enic/enic_rx.c index f92f6bc..89c62ce 100644 --- a/drivers/net/enic/enic_rx.c +++ b/drivers/net/enic/enic_rx.c @@ -275,10 +275,7 @@ enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, /* allocate a new mbuf */ nmb = rte_mbuf_raw_alloc(rq->mp); if (nmb == NULL) { - dev_err(enic, "RX mbuf alloc failed port=%u qid=%u", - enic->port_id, (unsigned)rq->index); - rte_eth_devices[enic->port_id]. - data->rx_mbuf_alloc_failed++; + rte_atomic64_inc(&enic->soft_stats.rx_nombuf); break; } -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v3 02/13] enic: drop bad packets and remove unused Rx error flag 2016-06-03 0:22 ` [PATCH v3 00/13] enic counter fixes and Tx optimization John Daley 2016-06-03 0:22 ` [PATCH v3 01/13] enic: fix Rx drop counters John Daley @ 2016-06-03 0:22 ` John Daley 2016-06-03 0:22 ` [PATCH v3 03/13] enic: count truncated packets John Daley ` (11 subsequent siblings) 13 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-06-03 0:22 UTC (permalink / raw) To: dev; +Cc: bruce.richarsdon, John Daley, Olivier Matz Following the discussions from: http://dpdk.org/ml/archives/dev/2015-July/021721.html http://dpdk.org/ml/archives/dev/2016-April/038143.html Remove the unused flag from enic driver. Also, the enic driver is modified to drop bad packets. Signed-off-by: Olivier Matz <olivier.matz@6wind.com> Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/enic_rx.c | 39 +++++++++++++++++---------------------- 1 file changed, 17 insertions(+), 22 deletions(-) diff --git a/drivers/net/enic/enic_rx.c b/drivers/net/enic/enic_rx.c index 89c62ce..c72a80a 100644 --- a/drivers/net/enic/enic_rx.c +++ b/drivers/net/enic/enic_rx.c @@ -134,20 +134,15 @@ enic_cq_rx_desc_n_bytes(struct cq_desc *cqd) } static inline uint8_t -enic_cq_rx_to_pkt_err_flags(struct cq_desc *cqd, uint64_t *pkt_err_flags_out) +enic_cq_rx_check_err(struct cq_desc *cqd) { struct cq_enet_rq_desc *cqrd = (struct cq_enet_rq_desc *)cqd; uint16_t bwflags; - int ret = 0; - uint64_t pkt_err_flags = 0; bwflags = enic_cq_rx_desc_bwflags(cqrd); - if (unlikely(enic_cq_rx_desc_packet_error(bwflags))) { - pkt_err_flags = PKT_RX_MAC_ERR; - ret = 1; - } - *pkt_err_flags_out = pkt_err_flags; - return ret; + if (unlikely(enic_cq_rx_desc_packet_error(bwflags))) + return 1; + return 0; } /* @@ -243,7 +238,7 @@ enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, struct enic *enic = vnic_dev_priv(rq->vdev); unsigned int rx_id; struct rte_mbuf *nmb, *rxmb; - uint16_t nb_rx = 0; + uint16_t nb_rx = 0, nb_err = 0; uint16_t nb_hold; struct vnic_cq *cq; volatile struct cq_desc *cqd_ptr; @@ -259,7 +254,6 @@ enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, volatile struct rq_enet_desc *rqd_ptr; dma_addr_t dma_addr; struct cq_desc cqd; - uint64_t ol_err_flags; uint8_t packet_error; /* Check for pkts available */ @@ -280,7 +274,7 @@ enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, } /* A packet error means descriptor and data are untrusted */ - packet_error = enic_cq_rx_to_pkt_err_flags(&cqd, &ol_err_flags); + packet_error = enic_cq_rx_check_err(&cqd); /* Get the mbuf to return and replace with one just allocated */ rxmb = rq->mbuf_ring[rx_id]; @@ -307,20 +301,21 @@ enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, rqd_ptr->length_type = cpu_to_le16(nmb->buf_len - RTE_PKTMBUF_HEADROOM); + /* Drop incoming bad packet */ + if (unlikely(packet_error)) { + rte_pktmbuf_free(rxmb); + nb_err++; + continue; + } + /* Fill in the rest of the mbuf */ rxmb->data_off = RTE_PKTMBUF_HEADROOM; rxmb->nb_segs = 1; rxmb->next = NULL; rxmb->port = enic->port_id; - if (!packet_error) { - rxmb->pkt_len = enic_cq_rx_desc_n_bytes(&cqd); - rxmb->packet_type = enic_cq_rx_flags_to_pkt_type(&cqd); - enic_cq_rx_to_pkt_flags(&cqd, rxmb); - } else { - rxmb->pkt_len = 0; - rxmb->packet_type = 0; - rxmb->ol_flags = 0; - } + rxmb->pkt_len = enic_cq_rx_desc_n_bytes(&cqd); + rxmb->packet_type = enic_cq_rx_flags_to_pkt_type(&cqd); + enic_cq_rx_to_pkt_flags(&cqd, rxmb); rxmb->data_len = rxmb->pkt_len; /* prefetch mbuf data for caller */ @@ -331,7 +326,7 @@ enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, rx_pkts[nb_rx++] = rxmb; } - nb_hold += nb_rx; + nb_hold += nb_rx + nb_err; cq->to_clean = rx_id; if (nb_hold > rq->rx_free_thresh) { -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v3 03/13] enic: count truncated packets 2016-06-03 0:22 ` [PATCH v3 00/13] enic counter fixes and Tx optimization John Daley 2016-06-03 0:22 ` [PATCH v3 01/13] enic: fix Rx drop counters John Daley 2016-06-03 0:22 ` [PATCH v3 02/13] enic: drop bad packets and remove unused Rx error flag John Daley @ 2016-06-03 0:22 ` John Daley 2016-06-03 0:22 ` [PATCH v3 04/13] enic: put Tx and Rx functions into same file John Daley ` (10 subsequent siblings) 13 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-06-03 0:22 UTC (permalink / raw) To: dev; +Cc: bruce.richarsdon, John Daley, Nelson Escobar Truncated packets occur on enic if an mbuf is not big enough to receive it or there aren't enough mbufs if rx scatter is in use. They show up as error packets but unlike other error packets (like packets bad FCS) there are no nic drop counts incremented for them. Truncated packets are calculated by subtracting hardware errors from software errors. Note: this causes transient inaccuracies in the ipackets count. Also, the length of truncated packets are counted in ibytes even though truncated packets are dropped which can make ibytes be slightly higher than it should be. Signed-off-by: Nelson Escobar <neescoba@cisco.com> Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/enic.h | 1 + drivers/net/enic/enic_main.c | 21 +++++++++++++++++---- 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/drivers/net/enic/enic.h b/drivers/net/enic/enic.h index 584d49b..9b6f349 100644 --- a/drivers/net/enic/enic.h +++ b/drivers/net/enic/enic.h @@ -93,6 +93,7 @@ struct enic_fdir { struct enic_soft_stats { rte_atomic64_t rx_nombuf; + rte_atomic64_t rx_packet_errors; }; /* Per-instance private data structure */ diff --git a/drivers/net/enic/enic_main.c b/drivers/net/enic/enic_main.c index c002ef3..e4ccc7d 100644 --- a/drivers/net/enic/enic_main.c +++ b/drivers/net/enic/enic_main.c @@ -215,12 +215,14 @@ static void enic_clear_soft_stats(struct enic *enic) { struct enic_soft_stats *soft_stats = &enic->soft_stats; rte_atomic64_clear(&soft_stats->rx_nombuf); + rte_atomic64_clear(&soft_stats->rx_packet_errors); } static void enic_init_soft_stats(struct enic *enic) { struct enic_soft_stats *soft_stats = &enic->soft_stats; rte_atomic64_init(&soft_stats->rx_nombuf); + rte_atomic64_init(&soft_stats->rx_packet_errors); enic_clear_soft_stats(enic); } @@ -234,14 +236,26 @@ void enic_dev_stats_clear(struct enic *enic) void enic_dev_stats_get(struct enic *enic, struct rte_eth_stats *r_stats) { struct vnic_stats *stats; - struct enic_soft_stats *soft_stats; + struct enic_soft_stats *soft_stats = &enic->soft_stats; + int64_t rx_truncated; + uint64_t rx_packet_errors; if (vnic_dev_stats_dump(enic->vdev, &stats)) { dev_err(enic, "Error in getting stats\n"); return; } - r_stats->ipackets = stats->rx.rx_frames_ok; + /* The number of truncated packets can only be calculated by + * subtracting a hardware counter from error packets received by + * the driver. Note: this causes transient inaccuracies in the + * ipackets count. Also, the length of truncated packets are + * counted in ibytes even though truncated packets are dropped + * which can make ibytes be slightly higher than it should be. + */ + rx_packet_errors = rte_atomic64_read(&soft_stats->rx_packet_errors); + rx_truncated = rx_packet_errors - stats->rx.rx_errors; + + r_stats->ipackets = stats->rx.rx_frames_ok - rx_truncated; r_stats->opackets = stats->tx.tx_frames_ok; r_stats->ibytes = stats->rx.rx_bytes_ok; @@ -250,9 +264,8 @@ void enic_dev_stats_get(struct enic *enic, struct rte_eth_stats *r_stats) r_stats->ierrors = stats->rx.rx_errors + stats->rx.rx_drop; r_stats->oerrors = stats->tx.tx_errors; - r_stats->imissed = stats->rx.rx_no_bufs; + r_stats->imissed = stats->rx.rx_no_bufs + rx_truncated; - soft_stats = &enic->soft_stats; r_stats->rx_nombuf = rte_atomic64_read(&soft_stats->rx_nombuf); } -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v3 04/13] enic: put Tx and Rx functions into same file 2016-06-03 0:22 ` [PATCH v3 00/13] enic counter fixes and Tx optimization John Daley ` (2 preceding siblings ...) 2016-06-03 0:22 ` [PATCH v3 03/13] enic: count truncated packets John Daley @ 2016-06-03 0:22 ` John Daley 2016-06-03 0:22 ` [PATCH v3 05/13] enic: remove some unused functions in Tx path John Daley ` (9 subsequent siblings) 13 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-06-03 0:22 UTC (permalink / raw) To: dev; +Cc: bruce.richarsdon, John Daley Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/Makefile | 2 +- drivers/net/enic/enic.h | 3 + drivers/net/enic/enic_ethdev.c | 65 ------ drivers/net/enic/enic_main.c | 82 +------ drivers/net/enic/enic_rx.c | 343 ----------------------------- drivers/net/enic/enic_rxtx.c | 481 +++++++++++++++++++++++++++++++++++++++++ 6 files changed, 486 insertions(+), 490 deletions(-) delete mode 100644 drivers/net/enic/enic_rx.c create mode 100644 drivers/net/enic/enic_rxtx.c diff --git a/drivers/net/enic/Makefile b/drivers/net/enic/Makefile index f316274..3926b79 100644 --- a/drivers/net/enic/Makefile +++ b/drivers/net/enic/Makefile @@ -53,7 +53,7 @@ VPATH += $(SRCDIR)/src # SRCS-$(CONFIG_RTE_LIBRTE_ENIC_PMD) += enic_ethdev.c SRCS-$(CONFIG_RTE_LIBRTE_ENIC_PMD) += enic_main.c -SRCS-$(CONFIG_RTE_LIBRTE_ENIC_PMD) += enic_rx.c +SRCS-$(CONFIG_RTE_LIBRTE_ENIC_PMD) += enic_rxtx.c SRCS-$(CONFIG_RTE_LIBRTE_ENIC_PMD) += enic_clsf.c SRCS-$(CONFIG_RTE_LIBRTE_ENIC_PMD) += enic_res.c SRCS-$(CONFIG_RTE_LIBRTE_ENIC_PMD) += base/vnic_cq.c diff --git a/drivers/net/enic/enic.h b/drivers/net/enic/enic.h index 9b6f349..62a8c12 100644 --- a/drivers/net/enic/enic.h +++ b/drivers/net/enic/enic.h @@ -208,4 +208,7 @@ extern void enic_clsf_destroy(struct enic *enic); uint16_t enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); +uint16_t enicpmd_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, + uint16_t nb_pkts); +void enic_free_wq_buf(__rte_unused struct vnic_wq *wq, struct vnic_wq_buf *buf); #endif /* _ENIC_H_ */ diff --git a/drivers/net/enic/enic_ethdev.c b/drivers/net/enic/enic_ethdev.c index 6bea940..fab8124 100644 --- a/drivers/net/enic/enic_ethdev.c +++ b/drivers/net/enic/enic_ethdev.c @@ -519,71 +519,6 @@ static void enicpmd_remove_mac_addr(struct rte_eth_dev *eth_dev, __rte_unused ui enic_del_mac_address(enic); } - -static uint16_t enicpmd_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, - uint16_t nb_pkts) -{ - uint16_t index; - unsigned int frags; - unsigned int pkt_len; - unsigned int seg_len; - unsigned int inc_len; - unsigned int nb_segs; - struct rte_mbuf *tx_pkt, *next_tx_pkt; - struct vnic_wq *wq = (struct vnic_wq *)tx_queue; - struct enic *enic = vnic_dev_priv(wq->vdev); - unsigned short vlan_id; - unsigned short ol_flags; - uint8_t last_seg, eop; - unsigned int host_tx_descs = 0; - - for (index = 0; index < nb_pkts; index++) { - tx_pkt = *tx_pkts++; - inc_len = 0; - nb_segs = tx_pkt->nb_segs; - if (nb_segs > vnic_wq_desc_avail(wq)) { - if (index > 0) - enic_post_wq_index(wq); - - /* wq cleanup and try again */ - if (!enic_cleanup_wq(enic, wq) || - (nb_segs > vnic_wq_desc_avail(wq))) { - return index; - } - } - - pkt_len = tx_pkt->pkt_len; - vlan_id = tx_pkt->vlan_tci; - ol_flags = tx_pkt->ol_flags; - for (frags = 0; inc_len < pkt_len; frags++) { - if (!tx_pkt) - break; - next_tx_pkt = tx_pkt->next; - seg_len = tx_pkt->data_len; - inc_len += seg_len; - - host_tx_descs++; - last_seg = 0; - eop = 0; - if ((pkt_len == inc_len) || !next_tx_pkt) { - eop = 1; - /* post if last packet in batch or > thresh */ - if ((index == (nb_pkts - 1)) || - (host_tx_descs > ENIC_TX_POST_THRESH)) { - last_seg = 1; - host_tx_descs = 0; - } - } - enic_send_pkt(enic, wq, tx_pkt, (unsigned short)seg_len, - !frags, eop, last_seg, ol_flags, vlan_id); - tx_pkt = next_tx_pkt; - } - } - - enic_cleanup_wq(enic, wq); - return index; -} - static const struct eth_dev_ops enicpmd_eth_dev_ops = { .dev_configure = enicpmd_dev_configure, .dev_start = enicpmd_dev_start, diff --git a/drivers/net/enic/enic_main.c b/drivers/net/enic/enic_main.c index e4ccc7d..f41ef86 100644 --- a/drivers/net/enic/enic_main.c +++ b/drivers/net/enic/enic_main.c @@ -58,7 +58,6 @@ #include "vnic_cq.h" #include "vnic_intr.h" #include "vnic_nic.h" -#include "enic_vnic_wq.h" static inline int enic_is_sriov_vf(struct enic *enic) { @@ -104,7 +103,7 @@ void enic_set_hdr_split_size(struct enic *enic, u16 split_hdr_size) vnic_set_hdr_split_size(enic->vdev, split_hdr_size); } -static void enic_free_wq_buf(__rte_unused struct vnic_wq *wq, struct vnic_wq_buf *buf) +void enic_free_wq_buf(__rte_unused struct vnic_wq *wq, struct vnic_wq_buf *buf) { struct rte_mbuf *mbuf = (struct rte_mbuf *)buf->os_buf; @@ -112,26 +111,6 @@ static void enic_free_wq_buf(__rte_unused struct vnic_wq *wq, struct vnic_wq_buf buf->os_buf = NULL; } -static void enic_wq_free_buf(struct vnic_wq *wq, - __rte_unused struct cq_desc *cq_desc, - struct vnic_wq_buf *buf, - __rte_unused void *opaque) -{ - enic_free_wq_buf(wq, buf); -} - -static int enic_wq_service(struct vnic_dev *vdev, struct cq_desc *cq_desc, - __rte_unused u8 type, u16 q_number, u16 completed_index, void *opaque) -{ - struct enic *enic = vnic_dev_priv(vdev); - - vnic_wq_service(&enic->wq[q_number], cq_desc, - completed_index, enic_wq_free_buf, - opaque); - - return 0; -} - static void enic_log_q_error(struct enic *enic) { unsigned int i; @@ -152,65 +131,6 @@ static void enic_log_q_error(struct enic *enic) } } -unsigned int enic_cleanup_wq(struct enic *enic, struct vnic_wq *wq) -{ - unsigned int cq = enic_cq_wq(enic, wq->index); - - /* Return the work done */ - return vnic_cq_service(&enic->cq[cq], - -1 /*wq_work_to_do*/, enic_wq_service, NULL); -} - -void enic_post_wq_index(struct vnic_wq *wq) -{ - enic_vnic_post_wq_index(wq); -} - -void enic_send_pkt(struct enic *enic, struct vnic_wq *wq, - struct rte_mbuf *tx_pkt, unsigned short len, - uint8_t sop, uint8_t eop, uint8_t cq_entry, - uint16_t ol_flags, uint16_t vlan_tag) -{ - struct wq_enet_desc *desc = vnic_wq_next_desc(wq); - uint16_t mss = 0; - uint8_t vlan_tag_insert = 0; - uint64_t bus_addr = (dma_addr_t) - (tx_pkt->buf_physaddr + tx_pkt->data_off); - - if (sop) { - if (ol_flags & PKT_TX_VLAN_PKT) - vlan_tag_insert = 1; - - if (enic->hw_ip_checksum) { - if (ol_flags & PKT_TX_IP_CKSUM) - mss |= ENIC_CALC_IP_CKSUM; - - if (ol_flags & PKT_TX_TCP_UDP_CKSUM) - mss |= ENIC_CALC_TCP_UDP_CKSUM; - } - } - - wq_enet_desc_enc(desc, - bus_addr, - len, - mss, - 0 /* header_length */, - 0 /* offload_mode WQ_ENET_OFFLOAD_MODE_CSUM */, - eop, - cq_entry, - 0 /* fcoe_encap */, - vlan_tag_insert, - vlan_tag, - 0 /* loopback */); - - enic_vnic_post_wq(wq, (void *)tx_pkt, bus_addr, len, - sop, - 1 /*desc_skip_cnt*/, - cq_entry, - 0 /*compressed send*/, - 0 /*wrid*/); -} - static void enic_clear_soft_stats(struct enic *enic) { struct enic_soft_stats *soft_stats = &enic->soft_stats; diff --git a/drivers/net/enic/enic_rx.c b/drivers/net/enic/enic_rx.c deleted file mode 100644 index c72a80a..0000000 --- a/drivers/net/enic/enic_rx.c +++ /dev/null @@ -1,343 +0,0 @@ -/* - * Copyright 2008-2014 Cisco Systems, Inc. All rights reserved. - * Copyright 2007 Nuova Systems, Inc. All rights reserved. - * - * Copyright (c) 2014, Cisco Systems, Inc. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in - * the documentation and/or other materials provided with the - * distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS - * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE - * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, - * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, - * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; - * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER - * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN - * ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * - */ - -#include <rte_mbuf.h> -#include <rte_ethdev.h> -#include <rte_prefetch.h> - -#include "enic_compat.h" -#include "rq_enet_desc.h" -#include "enic.h" - -#define RTE_PMD_USE_PREFETCH - -#ifdef RTE_PMD_USE_PREFETCH -/* - * Prefetch a cache line into all cache levels. - */ -#define rte_enic_prefetch(p) rte_prefetch0(p) -#else -#define rte_enic_prefetch(p) do {} while (0) -#endif - -#ifdef RTE_PMD_PACKET_PREFETCH -#define rte_packet_prefetch(p) rte_prefetch1(p) -#else -#define rte_packet_prefetch(p) do {} while (0) -#endif - -static inline uint16_t -enic_cq_rx_desc_ciflags(struct cq_enet_rq_desc *crd) -{ - return le16_to_cpu(crd->completed_index_flags) & ~CQ_DESC_COMP_NDX_MASK; -} - -static inline uint16_t -enic_cq_rx_desc_bwflags(struct cq_enet_rq_desc *crd) -{ - return(le16_to_cpu(crd->bytes_written_flags) & - ~CQ_ENET_RQ_DESC_BYTES_WRITTEN_MASK); -} - -static inline uint8_t -enic_cq_rx_desc_packet_error(uint16_t bwflags) -{ - return((bwflags & CQ_ENET_RQ_DESC_FLAGS_TRUNCATED) == - CQ_ENET_RQ_DESC_FLAGS_TRUNCATED); -} - -static inline uint8_t -enic_cq_rx_desc_eop(uint16_t ciflags) -{ - return (ciflags & CQ_ENET_RQ_DESC_FLAGS_EOP) - == CQ_ENET_RQ_DESC_FLAGS_EOP; -} - -static inline uint8_t -enic_cq_rx_desc_csum_not_calc(struct cq_enet_rq_desc *cqrd) -{ - return ((le16_to_cpu(cqrd->q_number_rss_type_flags) & - CQ_ENET_RQ_DESC_FLAGS_CSUM_NOT_CALC) == - CQ_ENET_RQ_DESC_FLAGS_CSUM_NOT_CALC); -} - -static inline uint8_t -enic_cq_rx_desc_ipv4_csum_ok(struct cq_enet_rq_desc *cqrd) -{ - return ((cqrd->flags & CQ_ENET_RQ_DESC_FLAGS_IPV4_CSUM_OK) == - CQ_ENET_RQ_DESC_FLAGS_IPV4_CSUM_OK); -} - -static inline uint8_t -enic_cq_rx_desc_tcp_udp_csum_ok(struct cq_enet_rq_desc *cqrd) -{ - return((cqrd->flags & CQ_ENET_RQ_DESC_FLAGS_TCP_UDP_CSUM_OK) == - CQ_ENET_RQ_DESC_FLAGS_TCP_UDP_CSUM_OK); -} - -static inline uint8_t -enic_cq_rx_desc_rss_type(struct cq_enet_rq_desc *cqrd) -{ - return (uint8_t)((le16_to_cpu(cqrd->q_number_rss_type_flags) >> - CQ_DESC_Q_NUM_BITS) & CQ_ENET_RQ_DESC_RSS_TYPE_MASK); -} - -static inline uint32_t -enic_cq_rx_desc_rss_hash(struct cq_enet_rq_desc *cqrd) -{ - return le32_to_cpu(cqrd->rss_hash); -} - -static inline uint16_t -enic_cq_rx_desc_vlan(struct cq_enet_rq_desc *cqrd) -{ - return le16_to_cpu(cqrd->vlan); -} - -static inline uint16_t -enic_cq_rx_desc_n_bytes(struct cq_desc *cqd) -{ - struct cq_enet_rq_desc *cqrd = (struct cq_enet_rq_desc *)cqd; - return le16_to_cpu(cqrd->bytes_written_flags) & - CQ_ENET_RQ_DESC_BYTES_WRITTEN_MASK; -} - -static inline uint8_t -enic_cq_rx_check_err(struct cq_desc *cqd) -{ - struct cq_enet_rq_desc *cqrd = (struct cq_enet_rq_desc *)cqd; - uint16_t bwflags; - - bwflags = enic_cq_rx_desc_bwflags(cqrd); - if (unlikely(enic_cq_rx_desc_packet_error(bwflags))) - return 1; - return 0; -} - -/* - * Lookup table to translate RX CQ flags to mbuf flags. - */ -static inline uint32_t -enic_cq_rx_flags_to_pkt_type(struct cq_desc *cqd) -{ - struct cq_enet_rq_desc *cqrd = (struct cq_enet_rq_desc *)cqd; - uint8_t cqrd_flags = cqrd->flags; - static const uint32_t cq_type_table[128] __rte_cache_aligned = { - [32] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4, - [34] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 - | RTE_PTYPE_L4_UDP, - [36] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 - | RTE_PTYPE_L4_TCP, - [96] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 - | RTE_PTYPE_L4_FRAG, - [16] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6, - [18] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 - | RTE_PTYPE_L4_UDP, - [20] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 - | RTE_PTYPE_L4_TCP, - [80] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 - | RTE_PTYPE_L4_FRAG, - /* All others reserved */ - }; - cqrd_flags &= CQ_ENET_RQ_DESC_FLAGS_IPV4_FRAGMENT - | CQ_ENET_RQ_DESC_FLAGS_IPV4 | CQ_ENET_RQ_DESC_FLAGS_IPV6 - | CQ_ENET_RQ_DESC_FLAGS_TCP | CQ_ENET_RQ_DESC_FLAGS_UDP; - return cq_type_table[cqrd_flags]; -} - -static inline void -enic_cq_rx_to_pkt_flags(struct cq_desc *cqd, struct rte_mbuf *mbuf) -{ - struct cq_enet_rq_desc *cqrd = (struct cq_enet_rq_desc *)cqd; - uint16_t ciflags, bwflags, pkt_flags = 0; - ciflags = enic_cq_rx_desc_ciflags(cqrd); - bwflags = enic_cq_rx_desc_bwflags(cqrd); - - mbuf->ol_flags = 0; - - /* flags are meaningless if !EOP */ - if (unlikely(!enic_cq_rx_desc_eop(ciflags))) - goto mbuf_flags_done; - - /* VLAN stripping */ - if (bwflags & CQ_ENET_RQ_DESC_FLAGS_VLAN_STRIPPED) { - pkt_flags |= PKT_RX_VLAN_PKT; - mbuf->vlan_tci = enic_cq_rx_desc_vlan(cqrd); - } else { - mbuf->vlan_tci = 0; - } - - /* RSS flag */ - if (enic_cq_rx_desc_rss_type(cqrd)) { - pkt_flags |= PKT_RX_RSS_HASH; - mbuf->hash.rss = enic_cq_rx_desc_rss_hash(cqrd); - } - - /* checksum flags */ - if (!enic_cq_rx_desc_csum_not_calc(cqrd) && - (mbuf->packet_type & RTE_PTYPE_L3_IPV4)) { - if (unlikely(!enic_cq_rx_desc_ipv4_csum_ok(cqrd))) - pkt_flags |= PKT_RX_IP_CKSUM_BAD; - if (mbuf->packet_type & (RTE_PTYPE_L4_UDP | RTE_PTYPE_L4_TCP)) { - if (unlikely(!enic_cq_rx_desc_tcp_udp_csum_ok(cqrd))) - pkt_flags |= PKT_RX_L4_CKSUM_BAD; - } - } - - mbuf_flags_done: - mbuf->ol_flags = pkt_flags; -} - -static inline uint32_t -enic_ring_add(uint32_t n_descriptors, uint32_t i0, uint32_t i1) -{ - uint32_t d = i0 + i1; - RTE_ASSERT(i0 < n_descriptors); - RTE_ASSERT(i1 < n_descriptors); - d -= (d >= n_descriptors) ? n_descriptors : 0; - return d; -} - - -uint16_t -enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, - uint16_t nb_pkts) -{ - struct vnic_rq *rq = rx_queue; - struct enic *enic = vnic_dev_priv(rq->vdev); - unsigned int rx_id; - struct rte_mbuf *nmb, *rxmb; - uint16_t nb_rx = 0, nb_err = 0; - uint16_t nb_hold; - struct vnic_cq *cq; - volatile struct cq_desc *cqd_ptr; - uint8_t color; - - cq = &enic->cq[enic_cq_rq(enic, rq->index)]; - rx_id = cq->to_clean; /* index of cqd, rqd, mbuf_table */ - cqd_ptr = (struct cq_desc *)(cq->ring.descs) + rx_id; - - nb_hold = rq->rx_nb_hold; /* mbufs held by software */ - - while (nb_rx < nb_pkts) { - volatile struct rq_enet_desc *rqd_ptr; - dma_addr_t dma_addr; - struct cq_desc cqd; - uint8_t packet_error; - - /* Check for pkts available */ - color = (cqd_ptr->type_color >> CQ_DESC_COLOR_SHIFT) - & CQ_DESC_COLOR_MASK; - if (color == cq->last_color) - break; - - /* Get the cq descriptor and rq pointer */ - cqd = *cqd_ptr; - rqd_ptr = (struct rq_enet_desc *)(rq->ring.descs) + rx_id; - - /* allocate a new mbuf */ - nmb = rte_mbuf_raw_alloc(rq->mp); - if (nmb == NULL) { - rte_atomic64_inc(&enic->soft_stats.rx_nombuf); - break; - } - - /* A packet error means descriptor and data are untrusted */ - packet_error = enic_cq_rx_check_err(&cqd); - - /* Get the mbuf to return and replace with one just allocated */ - rxmb = rq->mbuf_ring[rx_id]; - rq->mbuf_ring[rx_id] = nmb; - - /* Increment cqd, rqd, mbuf_table index */ - rx_id++; - if (unlikely(rx_id == rq->ring.desc_count)) { - rx_id = 0; - cq->last_color = cq->last_color ? 0 : 1; - } - - /* Prefetch next mbuf & desc while processing current one */ - cqd_ptr = (struct cq_desc *)(cq->ring.descs) + rx_id; - rte_enic_prefetch(cqd_ptr); - rte_enic_prefetch(rq->mbuf_ring[rx_id]); - rte_enic_prefetch((struct rq_enet_desc *)(rq->ring.descs) - + rx_id); - - /* Push descriptor for newly allocated mbuf */ - dma_addr = (dma_addr_t)(nmb->buf_physaddr - + RTE_PKTMBUF_HEADROOM); - rqd_ptr->address = rte_cpu_to_le_64(dma_addr); - rqd_ptr->length_type = cpu_to_le16(nmb->buf_len - - RTE_PKTMBUF_HEADROOM); - - /* Drop incoming bad packet */ - if (unlikely(packet_error)) { - rte_pktmbuf_free(rxmb); - nb_err++; - continue; - } - - /* Fill in the rest of the mbuf */ - rxmb->data_off = RTE_PKTMBUF_HEADROOM; - rxmb->nb_segs = 1; - rxmb->next = NULL; - rxmb->port = enic->port_id; - rxmb->pkt_len = enic_cq_rx_desc_n_bytes(&cqd); - rxmb->packet_type = enic_cq_rx_flags_to_pkt_type(&cqd); - enic_cq_rx_to_pkt_flags(&cqd, rxmb); - rxmb->data_len = rxmb->pkt_len; - - /* prefetch mbuf data for caller */ - rte_packet_prefetch(RTE_PTR_ADD(rxmb->buf_addr, - RTE_PKTMBUF_HEADROOM)); - - /* store the mbuf address into the next entry of the array */ - rx_pkts[nb_rx++] = rxmb; - } - - nb_hold += nb_rx + nb_err; - cq->to_clean = rx_id; - - if (nb_hold > rq->rx_free_thresh) { - rq->posted_index = enic_ring_add(rq->ring.desc_count, - rq->posted_index, nb_hold); - nb_hold = 0; - rte_mb(); - iowrite32(rq->posted_index, &rq->ctrl->posted_index); - } - - rq->rx_nb_hold = nb_hold; - - return nb_rx; -} diff --git a/drivers/net/enic/enic_rxtx.c b/drivers/net/enic/enic_rxtx.c new file mode 100644 index 0000000..056a297 --- /dev/null +++ b/drivers/net/enic/enic_rxtx.c @@ -0,0 +1,481 @@ +/* Copyright 2008-2016 Cisco Systems, Inc. All rights reserved. + * Copyright 2007 Nuova Systems, Inc. All rights reserved. + * + * Copyright (c) 2014, Cisco Systems, Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS + * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE + * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, + * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, + * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN + * ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ + +#include <rte_mbuf.h> +#include <rte_ethdev.h> +#include <rte_prefetch.h> + +#include "enic_compat.h" +#include "rq_enet_desc.h" +#include "enic.h" +#include "enic_vnic_wq.h" + +#define RTE_PMD_USE_PREFETCH + +#ifdef RTE_PMD_USE_PREFETCH +/*Prefetch a cache line into all cache levels. */ +#define rte_enic_prefetch(p) rte_prefetch0(p) +#else +#define rte_enic_prefetch(p) do {} while (0) +#endif + +#ifdef RTE_PMD_PACKET_PREFETCH +#define rte_packet_prefetch(p) rte_prefetch1(p) +#else +#define rte_packet_prefetch(p) do {} while (0) +#endif + +static inline uint16_t +enic_cq_rx_desc_ciflags(struct cq_enet_rq_desc *crd) +{ + return le16_to_cpu(crd->completed_index_flags) & ~CQ_DESC_COMP_NDX_MASK; +} + +static inline uint16_t +enic_cq_rx_desc_bwflags(struct cq_enet_rq_desc *crd) +{ + return le16_to_cpu(crd->bytes_written_flags) & + ~CQ_ENET_RQ_DESC_BYTES_WRITTEN_MASK; +} + +static inline uint8_t +enic_cq_rx_desc_packet_error(uint16_t bwflags) +{ + return (bwflags & CQ_ENET_RQ_DESC_FLAGS_TRUNCATED) == + CQ_ENET_RQ_DESC_FLAGS_TRUNCATED; +} + +static inline uint8_t +enic_cq_rx_desc_eop(uint16_t ciflags) +{ + return (ciflags & CQ_ENET_RQ_DESC_FLAGS_EOP) + == CQ_ENET_RQ_DESC_FLAGS_EOP; +} + +static inline uint8_t +enic_cq_rx_desc_csum_not_calc(struct cq_enet_rq_desc *cqrd) +{ + return (le16_to_cpu(cqrd->q_number_rss_type_flags) & + CQ_ENET_RQ_DESC_FLAGS_CSUM_NOT_CALC) == + CQ_ENET_RQ_DESC_FLAGS_CSUM_NOT_CALC; +} + +static inline uint8_t +enic_cq_rx_desc_ipv4_csum_ok(struct cq_enet_rq_desc *cqrd) +{ + return (cqrd->flags & CQ_ENET_RQ_DESC_FLAGS_IPV4_CSUM_OK) == + CQ_ENET_RQ_DESC_FLAGS_IPV4_CSUM_OK; +} + +static inline uint8_t +enic_cq_rx_desc_tcp_udp_csum_ok(struct cq_enet_rq_desc *cqrd) +{ + return (cqrd->flags & CQ_ENET_RQ_DESC_FLAGS_TCP_UDP_CSUM_OK) == + CQ_ENET_RQ_DESC_FLAGS_TCP_UDP_CSUM_OK; +} + +static inline uint8_t +enic_cq_rx_desc_rss_type(struct cq_enet_rq_desc *cqrd) +{ + return (uint8_t)((le16_to_cpu(cqrd->q_number_rss_type_flags) >> + CQ_DESC_Q_NUM_BITS) & CQ_ENET_RQ_DESC_RSS_TYPE_MASK); +} + +static inline uint32_t +enic_cq_rx_desc_rss_hash(struct cq_enet_rq_desc *cqrd) +{ + return le32_to_cpu(cqrd->rss_hash); +} + +static inline uint16_t +enic_cq_rx_desc_vlan(struct cq_enet_rq_desc *cqrd) +{ + return le16_to_cpu(cqrd->vlan); +} + +static inline uint16_t +enic_cq_rx_desc_n_bytes(struct cq_desc *cqd) +{ + struct cq_enet_rq_desc *cqrd = (struct cq_enet_rq_desc *)cqd; + return le16_to_cpu(cqrd->bytes_written_flags) & + CQ_ENET_RQ_DESC_BYTES_WRITTEN_MASK; +} + +static inline uint8_t +enic_cq_rx_check_err(struct cq_desc *cqd) +{ + struct cq_enet_rq_desc *cqrd = (struct cq_enet_rq_desc *)cqd; + uint16_t bwflags; + + bwflags = enic_cq_rx_desc_bwflags(cqrd); + if (unlikely(enic_cq_rx_desc_packet_error(bwflags))) + return 1; + return 0; +} + +/* Lookup table to translate RX CQ flags to mbuf flags. */ +static inline uint32_t +enic_cq_rx_flags_to_pkt_type(struct cq_desc *cqd) +{ + struct cq_enet_rq_desc *cqrd = (struct cq_enet_rq_desc *)cqd; + uint8_t cqrd_flags = cqrd->flags; + static const uint32_t cq_type_table[128] __rte_cache_aligned = { + [32] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4, + [34] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 + | RTE_PTYPE_L4_UDP, + [36] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 + | RTE_PTYPE_L4_TCP, + [96] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 + | RTE_PTYPE_L4_FRAG, + [16] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6, + [18] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 + | RTE_PTYPE_L4_UDP, + [20] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 + | RTE_PTYPE_L4_TCP, + [80] = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 + | RTE_PTYPE_L4_FRAG, + /* All others reserved */ + }; + cqrd_flags &= CQ_ENET_RQ_DESC_FLAGS_IPV4_FRAGMENT + | CQ_ENET_RQ_DESC_FLAGS_IPV4 | CQ_ENET_RQ_DESC_FLAGS_IPV6 + | CQ_ENET_RQ_DESC_FLAGS_TCP | CQ_ENET_RQ_DESC_FLAGS_UDP; + return cq_type_table[cqrd_flags]; +} + +static inline void +enic_cq_rx_to_pkt_flags(struct cq_desc *cqd, struct rte_mbuf *mbuf) +{ + struct cq_enet_rq_desc *cqrd = (struct cq_enet_rq_desc *)cqd; + uint16_t ciflags, bwflags, pkt_flags = 0; + ciflags = enic_cq_rx_desc_ciflags(cqrd); + bwflags = enic_cq_rx_desc_bwflags(cqrd); + + mbuf->ol_flags = 0; + + /* flags are meaningless if !EOP */ + if (unlikely(!enic_cq_rx_desc_eop(ciflags))) + goto mbuf_flags_done; + + /* VLAN stripping */ + if (bwflags & CQ_ENET_RQ_DESC_FLAGS_VLAN_STRIPPED) { + pkt_flags |= PKT_RX_VLAN_PKT; + mbuf->vlan_tci = enic_cq_rx_desc_vlan(cqrd); + } else { + mbuf->vlan_tci = 0; + } + + /* RSS flag */ + if (enic_cq_rx_desc_rss_type(cqrd)) { + pkt_flags |= PKT_RX_RSS_HASH; + mbuf->hash.rss = enic_cq_rx_desc_rss_hash(cqrd); + } + + /* checksum flags */ + if (!enic_cq_rx_desc_csum_not_calc(cqrd) && + (mbuf->packet_type & RTE_PTYPE_L3_IPV4)) { + if (unlikely(!enic_cq_rx_desc_ipv4_csum_ok(cqrd))) + pkt_flags |= PKT_RX_IP_CKSUM_BAD; + if (mbuf->packet_type & (RTE_PTYPE_L4_UDP | RTE_PTYPE_L4_TCP)) { + if (unlikely(!enic_cq_rx_desc_tcp_udp_csum_ok(cqrd))) + pkt_flags |= PKT_RX_L4_CKSUM_BAD; + } + } + + mbuf_flags_done: + mbuf->ol_flags = pkt_flags; +} + +static inline uint32_t +enic_ring_add(uint32_t n_descriptors, uint32_t i0, uint32_t i1) +{ + uint32_t d = i0 + i1; + RTE_ASSERT(i0 < n_descriptors); + RTE_ASSERT(i1 < n_descriptors); + d -= (d >= n_descriptors) ? n_descriptors : 0; + return d; +} + + +uint16_t +enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, + uint16_t nb_pkts) +{ + struct vnic_rq *rq = rx_queue; + struct enic *enic = vnic_dev_priv(rq->vdev); + unsigned int rx_id; + struct rte_mbuf *nmb, *rxmb; + uint16_t nb_rx = 0, nb_err = 0; + uint16_t nb_hold; + struct vnic_cq *cq; + volatile struct cq_desc *cqd_ptr; + uint8_t color; + + cq = &enic->cq[enic_cq_rq(enic, rq->index)]; + rx_id = cq->to_clean; /* index of cqd, rqd, mbuf_table */ + cqd_ptr = (struct cq_desc *)(cq->ring.descs) + rx_id; + + nb_hold = rq->rx_nb_hold; /* mbufs held by software */ + + while (nb_rx < nb_pkts) { + volatile struct rq_enet_desc *rqd_ptr; + dma_addr_t dma_addr; + struct cq_desc cqd; + uint8_t packet_error; + + /* Check for pkts available */ + color = (cqd_ptr->type_color >> CQ_DESC_COLOR_SHIFT) + & CQ_DESC_COLOR_MASK; + if (color == cq->last_color) + break; + + /* Get the cq descriptor and rq pointer */ + cqd = *cqd_ptr; + rqd_ptr = (struct rq_enet_desc *)(rq->ring.descs) + rx_id; + + /* allocate a new mbuf */ + nmb = rte_mbuf_raw_alloc(rq->mp); + if (nmb == NULL) { + rte_atomic64_inc(&enic->soft_stats.rx_nombuf); + break; + } + + /* A packet error means descriptor and data are untrusted */ + packet_error = enic_cq_rx_check_err(&cqd); + + /* Get the mbuf to return and replace with one just allocated */ + rxmb = rq->mbuf_ring[rx_id]; + rq->mbuf_ring[rx_id] = nmb; + + /* Increment cqd, rqd, mbuf_table index */ + rx_id++; + if (unlikely(rx_id == rq->ring.desc_count)) { + rx_id = 0; + cq->last_color = cq->last_color ? 0 : 1; + } + + /* Prefetch next mbuf & desc while processing current one */ + cqd_ptr = (struct cq_desc *)(cq->ring.descs) + rx_id; + rte_enic_prefetch(cqd_ptr); + rte_enic_prefetch(rq->mbuf_ring[rx_id]); + rte_enic_prefetch((struct rq_enet_desc *)(rq->ring.descs) + + rx_id); + + /* Push descriptor for newly allocated mbuf */ + dma_addr = (dma_addr_t)(nmb->buf_physaddr + + RTE_PKTMBUF_HEADROOM); + rqd_ptr->address = rte_cpu_to_le_64(dma_addr); + rqd_ptr->length_type = cpu_to_le16(nmb->buf_len + - RTE_PKTMBUF_HEADROOM); + + /* Drop incoming bad packet */ + if (unlikely(packet_error)) { + rte_pktmbuf_free(rxmb); + nb_err++; + continue; + } + + /* Fill in the rest of the mbuf */ + rxmb->data_off = RTE_PKTMBUF_HEADROOM; + rxmb->nb_segs = 1; + rxmb->next = NULL; + rxmb->port = enic->port_id; + rxmb->pkt_len = enic_cq_rx_desc_n_bytes(&cqd); + rxmb->packet_type = enic_cq_rx_flags_to_pkt_type(&cqd); + enic_cq_rx_to_pkt_flags(&cqd, rxmb); + rxmb->data_len = rxmb->pkt_len; + + /* prefetch mbuf data for caller */ + rte_packet_prefetch(RTE_PTR_ADD(rxmb->buf_addr, + RTE_PKTMBUF_HEADROOM)); + + /* store the mbuf address into the next entry of the array */ + rx_pkts[nb_rx++] = rxmb; + } + + nb_hold += nb_rx + nb_err; + cq->to_clean = rx_id; + + if (nb_hold > rq->rx_free_thresh) { + rq->posted_index = enic_ring_add(rq->ring.desc_count, + rq->posted_index, nb_hold); + nb_hold = 0; + rte_mb(); + iowrite32(rq->posted_index, &rq->ctrl->posted_index); + } + + rq->rx_nb_hold = nb_hold; + + return nb_rx; +} + +static void enic_wq_free_buf(struct vnic_wq *wq, + __rte_unused struct cq_desc *cq_desc, + struct vnic_wq_buf *buf, + __rte_unused void *opaque) +{ + enic_free_wq_buf(wq, buf); +} + +static int enic_wq_service(struct vnic_dev *vdev, struct cq_desc *cq_desc, + __rte_unused u8 type, u16 q_number, u16 completed_index, void *opaque) +{ + struct enic *enic = vnic_dev_priv(vdev); + + vnic_wq_service(&enic->wq[q_number], cq_desc, + completed_index, enic_wq_free_buf, + opaque); + + return 0; +} + +unsigned int enic_cleanup_wq(struct enic *enic, struct vnic_wq *wq) +{ + unsigned int cq = enic_cq_wq(enic, wq->index); + + /* Return the work done */ + return vnic_cq_service(&enic->cq[cq], + -1 /*wq_work_to_do*/, enic_wq_service, NULL); +} + +void enic_post_wq_index(struct vnic_wq *wq) +{ + enic_vnic_post_wq_index(wq); +} + +void enic_send_pkt(struct enic *enic, struct vnic_wq *wq, + struct rte_mbuf *tx_pkt, unsigned short len, + uint8_t sop, uint8_t eop, uint8_t cq_entry, + uint16_t ol_flags, uint16_t vlan_tag) +{ + struct wq_enet_desc *desc = vnic_wq_next_desc(wq); + uint16_t mss = 0; + uint8_t vlan_tag_insert = 0; + uint64_t bus_addr = (dma_addr_t) + (tx_pkt->buf_physaddr + tx_pkt->data_off); + + if (sop) { + if (ol_flags & PKT_TX_VLAN_PKT) + vlan_tag_insert = 1; + + if (enic->hw_ip_checksum) { + if (ol_flags & PKT_TX_IP_CKSUM) + mss |= ENIC_CALC_IP_CKSUM; + + if (ol_flags & PKT_TX_TCP_UDP_CKSUM) + mss |= ENIC_CALC_TCP_UDP_CKSUM; + } + } + + wq_enet_desc_enc(desc, + bus_addr, + len, + mss, + 0 /* header_length */, + 0 /* offload_mode WQ_ENET_OFFLOAD_MODE_CSUM */, + eop, + cq_entry, + 0 /* fcoe_encap */, + vlan_tag_insert, + vlan_tag, + 0 /* loopback */); + + enic_vnic_post_wq(wq, (void *)tx_pkt, bus_addr, len, + sop, + 1 /*desc_skip_cnt*/, + cq_entry, + 0 /*compressed send*/, + 0 /*wrid*/); +} + +uint16_t enicpmd_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, + uint16_t nb_pkts) +{ + uint16_t index; + unsigned int frags; + unsigned int pkt_len; + unsigned int seg_len; + unsigned int inc_len; + unsigned int nb_segs; + struct rte_mbuf *tx_pkt, *next_tx_pkt; + struct vnic_wq *wq = (struct vnic_wq *)tx_queue; + struct enic *enic = vnic_dev_priv(wq->vdev); + unsigned short vlan_id; + unsigned short ol_flags; + uint8_t last_seg, eop; + unsigned int host_tx_descs = 0; + + for (index = 0; index < nb_pkts; index++) { + tx_pkt = *tx_pkts++; + inc_len = 0; + nb_segs = tx_pkt->nb_segs; + if (nb_segs > vnic_wq_desc_avail(wq)) { + if (index > 0) + enic_post_wq_index(wq); + + /* wq cleanup and try again */ + if (!enic_cleanup_wq(enic, wq) || + (nb_segs > vnic_wq_desc_avail(wq))) { + return index; + } + } + + pkt_len = tx_pkt->pkt_len; + vlan_id = tx_pkt->vlan_tci; + ol_flags = tx_pkt->ol_flags; + for (frags = 0; inc_len < pkt_len; frags++) { + if (!tx_pkt) + break; + next_tx_pkt = tx_pkt->next; + seg_len = tx_pkt->data_len; + inc_len += seg_len; + + host_tx_descs++; + last_seg = 0; + eop = 0; + if ((pkt_len == inc_len) || !next_tx_pkt) { + eop = 1; + /* post if last packet in batch or > thresh */ + if ((index == (nb_pkts - 1)) || + (host_tx_descs > ENIC_TX_POST_THRESH)) { + last_seg = 1; + host_tx_descs = 0; + } + } + enic_send_pkt(enic, wq, tx_pkt, (unsigned short)seg_len, + !frags, eop, last_seg, ol_flags, vlan_id); + tx_pkt = next_tx_pkt; + } + } + + enic_cleanup_wq(enic, wq); + return index; +} -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v3 05/13] enic: remove some unused functions in Tx path 2016-06-03 0:22 ` [PATCH v3 00/13] enic counter fixes and Tx optimization John Daley ` (3 preceding siblings ...) 2016-06-03 0:22 ` [PATCH v3 04/13] enic: put Tx and Rx functions into same file John Daley @ 2016-06-03 0:22 ` John Daley 2016-06-03 0:22 ` [PATCH v3 06/13] enic: streamline mbuf handling " John Daley ` (8 subsequent siblings) 13 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-06-03 0:22 UTC (permalink / raw) To: dev; +Cc: bruce.richarsdon, John Daley Functions existed which were never called. Removed them. Also rename the 'pmd' from the name of the Tx function to improve clarity. Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/base/vnic_wq.h | 45 ------------------------ drivers/net/enic/enic.h | 5 ++- drivers/net/enic/enic_ethdev.c | 2 +- drivers/net/enic/enic_res.h | 78 ----------------------------------------- drivers/net/enic/enic_rxtx.c | 2 +- 5 files changed, 4 insertions(+), 128 deletions(-) diff --git a/drivers/net/enic/base/vnic_wq.h b/drivers/net/enic/base/vnic_wq.h index c23de62..d8660e4 100644 --- a/drivers/net/enic/base/vnic_wq.h +++ b/drivers/net/enic/base/vnic_wq.h @@ -191,51 +191,6 @@ static inline u64 vnic_cached_posted_index(dma_addr_t addr, unsigned int len, PI_PREFETCH_ADDR_MASK) << PI_PREFETCH_ADDR_OFF); } -static inline void vnic_wq_post(struct vnic_wq *wq, - void *os_buf, dma_addr_t dma_addr, - unsigned int len, int sop, int eop, - uint8_t desc_skip_cnt, uint8_t cq_entry, - uint8_t compressed_send, uint64_t wrid) -{ - struct vnic_wq_buf *buf = wq->to_use; - - buf->sop = sop; - buf->cq_entry = cq_entry; - buf->compressed_send = compressed_send; - buf->desc_skip_cnt = desc_skip_cnt; - buf->os_buf = os_buf; - buf->dma_addr = dma_addr; - buf->len = len; - buf->wr_id = wrid; - - buf = buf->next; - if (eop) { -#ifdef DO_PREFETCH - uint64_t wr = vnic_cached_posted_index(dma_addr, len, - buf->index); -#endif - /* Adding write memory barrier prevents compiler and/or CPU - * reordering, thus avoiding descriptor posting before - * descriptor is initialized. Otherwise, hardware can read - * stale descriptor fields. - */ - wmb(); -#ifdef DO_PREFETCH - /* Intel chipsets seem to limit the rate of PIOs that we can - * push on the bus. Thus, it is very important to do a single - * 64 bit write here. With two 32-bit writes, my maximum - * pkt/sec rate was cut almost in half. -AJF - */ - iowrite64((uint64_t)wr, &wq->ctrl->posted_index); -#else - iowrite32(buf->index, &wq->ctrl->posted_index); -#endif - } - wq->to_use = buf; - - wq->ring.desc_avail -= desc_skip_cnt; -} - static inline void vnic_wq_service(struct vnic_wq *wq, struct cq_desc *cq_desc, u16 completed_index, void (*buf_service)(struct vnic_wq *wq, diff --git a/drivers/net/enic/enic.h b/drivers/net/enic/enic.h index 62a8c12..5b58a65 100644 --- a/drivers/net/enic/enic.h +++ b/drivers/net/enic/enic.h @@ -207,8 +207,7 @@ extern int enic_clsf_init(struct enic *enic); extern void enic_clsf_destroy(struct enic *enic); uint16_t enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); - -uint16_t enicpmd_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, - uint16_t nb_pkts); +uint16_t enic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, + uint16_t nb_pkts); void enic_free_wq_buf(__rte_unused struct vnic_wq *wq, struct vnic_wq_buf *buf); #endif /* _ENIC_H_ */ diff --git a/drivers/net/enic/enic_ethdev.c b/drivers/net/enic/enic_ethdev.c index fab8124..697ff82 100644 --- a/drivers/net/enic/enic_ethdev.c +++ b/drivers/net/enic/enic_ethdev.c @@ -577,7 +577,7 @@ static int eth_enicpmd_dev_init(struct rte_eth_dev *eth_dev) enic->rte_dev = eth_dev; eth_dev->dev_ops = &enicpmd_eth_dev_ops; eth_dev->rx_pkt_burst = &enic_recv_pkts; - eth_dev->tx_pkt_burst = &enicpmd_xmit_pkts; + eth_dev->tx_pkt_burst = &enic_xmit_pkts; pdev = eth_dev->pci_dev; rte_eth_copy_pci_info(eth_dev, pdev); diff --git a/drivers/net/enic/enic_res.h b/drivers/net/enic/enic_res.h index 00fa71d..955db71 100644 --- a/drivers/net/enic/enic_res.h +++ b/drivers/net/enic/enic_res.h @@ -57,84 +57,6 @@ #define ENIC_SETTING(enic, f) ((enic->config.flags & VENETF_##f) ? 1 : 0) -static inline void enic_queue_wq_desc_ex(struct vnic_wq *wq, - void *os_buf, dma_addr_t dma_addr, unsigned int len, - unsigned int mss_or_csum_offset, unsigned int hdr_len, - int vlan_tag_insert, unsigned int vlan_tag, - int offload_mode, int cq_entry, int sop, int eop, int loopback) -{ - struct wq_enet_desc *desc = vnic_wq_next_desc(wq); - u8 desc_skip_cnt = 1; - u8 compressed_send = 0; - u64 wrid = 0; - - wq_enet_desc_enc(desc, - (u64)dma_addr | VNIC_PADDR_TARGET, - (u16)len, - (u16)mss_or_csum_offset, - (u16)hdr_len, (u8)offload_mode, - (u8)eop, (u8)cq_entry, - 0, /* fcoe_encap */ - (u8)vlan_tag_insert, - (u16)vlan_tag, - (u8)loopback); - - vnic_wq_post(wq, os_buf, dma_addr, len, sop, eop, desc_skip_cnt, - (u8)cq_entry, compressed_send, wrid); -} - -static inline void enic_queue_wq_desc_cont(struct vnic_wq *wq, - void *os_buf, dma_addr_t dma_addr, unsigned int len, - int eop, int loopback) -{ - enic_queue_wq_desc_ex(wq, os_buf, dma_addr, len, - 0, 0, 0, 0, 0, - eop, 0 /* !SOP */, eop, loopback); -} - -static inline void enic_queue_wq_desc(struct vnic_wq *wq, void *os_buf, - dma_addr_t dma_addr, unsigned int len, int vlan_tag_insert, - unsigned int vlan_tag, int eop, int loopback) -{ - enic_queue_wq_desc_ex(wq, os_buf, dma_addr, len, - 0, 0, vlan_tag_insert, vlan_tag, - WQ_ENET_OFFLOAD_MODE_CSUM, - eop, 1 /* SOP */, eop, loopback); -} - -static inline void enic_queue_wq_desc_csum(struct vnic_wq *wq, - void *os_buf, dma_addr_t dma_addr, unsigned int len, - int ip_csum, int tcpudp_csum, int vlan_tag_insert, - unsigned int vlan_tag, int eop, int loopback) -{ - enic_queue_wq_desc_ex(wq, os_buf, dma_addr, len, - (ip_csum ? 1 : 0) + (tcpudp_csum ? 2 : 0), - 0, vlan_tag_insert, vlan_tag, - WQ_ENET_OFFLOAD_MODE_CSUM, - eop, 1 /* SOP */, eop, loopback); -} - -static inline void enic_queue_wq_desc_csum_l4(struct vnic_wq *wq, - void *os_buf, dma_addr_t dma_addr, unsigned int len, - unsigned int csum_offset, unsigned int hdr_len, - int vlan_tag_insert, unsigned int vlan_tag, int eop, int loopback) -{ - enic_queue_wq_desc_ex(wq, os_buf, dma_addr, len, - csum_offset, hdr_len, vlan_tag_insert, vlan_tag, - WQ_ENET_OFFLOAD_MODE_CSUM_L4, - eop, 1 /* SOP */, eop, loopback); -} - -static inline void enic_queue_wq_desc_tso(struct vnic_wq *wq, - void *os_buf, dma_addr_t dma_addr, unsigned int len, - unsigned int mss, unsigned int hdr_len, int vlan_tag_insert, - unsigned int vlan_tag, int eop, int loopback) -{ - enic_queue_wq_desc_ex(wq, os_buf, dma_addr, len, - mss, hdr_len, vlan_tag_insert, vlan_tag, - WQ_ENET_OFFLOAD_MODE_TSO, - eop, 1 /* SOP */, eop, loopback); -} struct enic; diff --git a/drivers/net/enic/enic_rxtx.c b/drivers/net/enic/enic_rxtx.c index 056a297..530e8a3 100644 --- a/drivers/net/enic/enic_rxtx.c +++ b/drivers/net/enic/enic_rxtx.c @@ -416,7 +416,7 @@ void enic_send_pkt(struct enic *enic, struct vnic_wq *wq, 0 /*wrid*/); } -uint16_t enicpmd_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, +uint16_t enic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) { uint16_t index; -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v3 06/13] enic: streamline mbuf handling in Tx path 2016-06-03 0:22 ` [PATCH v3 00/13] enic counter fixes and Tx optimization John Daley ` (4 preceding siblings ...) 2016-06-03 0:22 ` [PATCH v3 05/13] enic: remove some unused functions in Tx path John Daley @ 2016-06-03 0:22 ` John Daley 2016-06-03 0:22 ` [PATCH v3 07/13] enic: use Tx completion messages instead of descriptors John Daley ` (7 subsequent siblings) 13 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-06-03 0:22 UTC (permalink / raw) To: dev; +Cc: bruce.richarsdon, John Daley The list of mbufs held by the driver on Tx was allocated in chunks (a hold-over from the enic kernel mode driver). The structure used next pointers across chunks which led to cache misses. Allocate the array used to hold mbufs in flight on Tx with rte_zmalloc_socket(). Remove unnecessary fields from the structure and use head and tail pointers instead of next pointers. Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/base/enic_vnic_wq.h | 26 +++---------- drivers/net/enic/base/vnic_wq.c | 75 +++++++++--------------------------- drivers/net/enic/base/vnic_wq.h | 56 ++++++++++----------------- drivers/net/enic/enic.h | 24 ++++++++++++ drivers/net/enic/enic_main.c | 4 +- drivers/net/enic/enic_rxtx.c | 23 +++-------- 6 files changed, 75 insertions(+), 133 deletions(-) diff --git a/drivers/net/enic/base/enic_vnic_wq.h b/drivers/net/enic/base/enic_vnic_wq.h index b019109..55c5664 100644 --- a/drivers/net/enic/base/enic_vnic_wq.h +++ b/drivers/net/enic/base/enic_vnic_wq.h @@ -40,37 +40,21 @@ static inline void enic_vnic_post_wq_index(struct vnic_wq *wq) { - struct vnic_wq_buf *buf = wq->to_use; - /* Adding write memory barrier prevents compiler and/or CPU * reordering, thus avoiding descriptor posting before * descriptor is initialized. Otherwise, hardware can read * stale descriptor fields. */ wmb(); - iowrite32(buf->index, &wq->ctrl->posted_index); + iowrite32(wq->head_idx, &wq->ctrl->posted_index); } static inline void enic_vnic_post_wq(struct vnic_wq *wq, - void *os_buf, dma_addr_t dma_addr, - unsigned int len, int sop, - uint8_t desc_skip_cnt, uint8_t cq_entry, - uint8_t compressed_send, uint64_t wrid) + void *os_buf, uint8_t cq_entry) { - struct vnic_wq_buf *buf = wq->to_use; - - buf->sop = sop; - buf->cq_entry = cq_entry; - buf->compressed_send = compressed_send; - buf->desc_skip_cnt = desc_skip_cnt; - buf->os_buf = os_buf; - buf->dma_addr = dma_addr; - buf->len = len; - buf->wr_id = wrid; - - buf = buf->next; - wq->ring.desc_avail -= desc_skip_cnt; - wq->to_use = buf; + struct vnic_wq_buf *buf = &wq->bufs[wq->head_idx]; + buf->mb = os_buf; + wq->head_idx = enic_ring_incr(wq->ring.desc_count, wq->head_idx); if (cq_entry) enic_vnic_post_wq_index(wq); diff --git a/drivers/net/enic/base/vnic_wq.c b/drivers/net/enic/base/vnic_wq.c index a3ef417..ab81c7e 100644 --- a/drivers/net/enic/base/vnic_wq.c +++ b/drivers/net/enic/base/vnic_wq.c @@ -59,71 +59,30 @@ int vnic_wq_alloc_ring(struct vnic_dev *vdev, struct vnic_wq *wq, static int vnic_wq_alloc_bufs(struct vnic_wq *wq) { - struct vnic_wq_buf *buf; - unsigned int i, j, count = wq->ring.desc_count; - unsigned int blks = VNIC_WQ_BUF_BLKS_NEEDED(count); - - for (i = 0; i < blks; i++) { - wq->bufs[i] = kzalloc(VNIC_WQ_BUF_BLK_SZ(count), GFP_ATOMIC); - if (!wq->bufs[i]) - return -ENOMEM; - } - - for (i = 0; i < blks; i++) { - buf = wq->bufs[i]; - for (j = 0; j < VNIC_WQ_BUF_BLK_ENTRIES(count); j++) { - buf->index = i * VNIC_WQ_BUF_BLK_ENTRIES(count) + j; - buf->desc = (u8 *)wq->ring.descs + - wq->ring.desc_size * buf->index; - if (buf->index + 1 == count) { - buf->next = wq->bufs[0]; - break; - } else if (j + 1 == VNIC_WQ_BUF_BLK_ENTRIES(count)) { - buf->next = wq->bufs[i + 1]; - } else { - buf->next = buf + 1; - buf++; - } - } - } - - wq->to_use = wq->to_clean = wq->bufs[0]; - + unsigned int count = wq->ring.desc_count; + /* Allocate the mbuf ring */ + wq->bufs = (struct vnic_wq_buf *)rte_zmalloc_socket("wq->bufs", + sizeof(struct vnic_wq_buf) * count, + RTE_CACHE_LINE_SIZE, wq->socket_id); + wq->head_idx = 0; + wq->tail_idx = 0; + if (wq->bufs == NULL) + return -ENOMEM; return 0; } void vnic_wq_free(struct vnic_wq *wq) { struct vnic_dev *vdev; - unsigned int i; vdev = wq->vdev; vnic_dev_free_desc_ring(vdev, &wq->ring); - for (i = 0; i < VNIC_WQ_BUF_BLKS_MAX; i++) { - if (wq->bufs[i]) { - kfree(wq->bufs[i]); - wq->bufs[i] = NULL; - } - } - + rte_free(wq->bufs); wq->ctrl = NULL; } -int vnic_wq_mem_size(struct vnic_wq *wq, unsigned int desc_count, - unsigned int desc_size) -{ - int mem_size = 0; - - mem_size += vnic_dev_desc_ring_size(&wq->ring, desc_count, desc_size); - - mem_size += VNIC_WQ_BUF_BLKS_NEEDED(wq->ring.desc_count) * - VNIC_WQ_BUF_BLK_SZ(wq->ring.desc_count); - - return mem_size; -} - int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index, unsigned int desc_count, unsigned int desc_size) @@ -172,9 +131,8 @@ void vnic_wq_init_start(struct vnic_wq *wq, unsigned int cq_index, iowrite32(error_interrupt_offset, &wq->ctrl->error_interrupt_offset); iowrite32(0, &wq->ctrl->error_status); - wq->to_use = wq->to_clean = - &wq->bufs[fetch_index / VNIC_WQ_BUF_BLK_ENTRIES(count)] - [fetch_index % VNIC_WQ_BUF_BLK_ENTRIES(count)]; + wq->head_idx = fetch_index; + wq->tail_idx = wq->head_idx; } void vnic_wq_init(struct vnic_wq *wq, unsigned int cq_index, @@ -223,18 +181,21 @@ void vnic_wq_clean(struct vnic_wq *wq, void (*buf_clean)(struct vnic_wq *wq, struct vnic_wq_buf *buf)) { struct vnic_wq_buf *buf; + unsigned int to_clean = wq->tail_idx; - buf = wq->to_clean; + buf = &wq->bufs[to_clean]; while (vnic_wq_desc_used(wq) > 0) { (*buf_clean)(wq, buf); + to_clean = buf_idx_incr(wq->ring.desc_count, to_clean); - buf = wq->to_clean = buf->next; + buf = &wq->bufs[to_clean]; wq->ring.desc_avail++; } - wq->to_use = wq->to_clean = wq->bufs[0]; + wq->head_idx = 0; + wq->tail_idx = 0; iowrite32(0, &wq->ctrl->fetch_index); iowrite32(0, &wq->ctrl->posted_index); diff --git a/drivers/net/enic/base/vnic_wq.h b/drivers/net/enic/base/vnic_wq.h index d8660e4..a6759f5 100644 --- a/drivers/net/enic/base/vnic_wq.h +++ b/drivers/net/enic/base/vnic_wq.h @@ -64,40 +64,19 @@ struct vnic_wq_ctrl { u32 pad9; }; +/* 16 bytes */ struct vnic_wq_buf { - struct vnic_wq_buf *next; - dma_addr_t dma_addr; - void *os_buf; - unsigned int len; - unsigned int index; - int sop; - void *desc; - uint64_t wr_id; /* Cookie */ - uint8_t cq_entry; /* Gets completion event from hw */ - uint8_t desc_skip_cnt; /* Num descs to occupy */ - uint8_t compressed_send; /* Both hdr and payload in one desc */ + void *mb; }; -/* Break the vnic_wq_buf allocations into blocks of 32/64 entries */ -#define VNIC_WQ_BUF_MIN_BLK_ENTRIES 32 -#define VNIC_WQ_BUF_DFLT_BLK_ENTRIES 64 -#define VNIC_WQ_BUF_BLK_ENTRIES(entries) \ - ((unsigned int)((entries < VNIC_WQ_BUF_DFLT_BLK_ENTRIES) ? \ - VNIC_WQ_BUF_MIN_BLK_ENTRIES : VNIC_WQ_BUF_DFLT_BLK_ENTRIES)) -#define VNIC_WQ_BUF_BLK_SZ(entries) \ - (VNIC_WQ_BUF_BLK_ENTRIES(entries) * sizeof(struct vnic_wq_buf)) -#define VNIC_WQ_BUF_BLKS_NEEDED(entries) \ - DIV_ROUND_UP(entries, VNIC_WQ_BUF_BLK_ENTRIES(entries)) -#define VNIC_WQ_BUF_BLKS_MAX VNIC_WQ_BUF_BLKS_NEEDED(4096) - struct vnic_wq { unsigned int index; struct vnic_dev *vdev; struct vnic_wq_ctrl __iomem *ctrl; /* memory-mapped */ struct vnic_dev_ring ring; - struct vnic_wq_buf *bufs[VNIC_WQ_BUF_BLKS_MAX]; - struct vnic_wq_buf *to_use; - struct vnic_wq_buf *to_clean; + struct vnic_wq_buf *bufs; + unsigned int head_idx; + unsigned int tail_idx; unsigned int pkts_outstanding; unsigned int socket_id; }; @@ -114,11 +93,6 @@ static inline unsigned int vnic_wq_desc_used(struct vnic_wq *wq) return wq->ring.desc_count - wq->ring.desc_avail - 1; } -static inline void *vnic_wq_next_desc(struct vnic_wq *wq) -{ - return wq->to_use->desc; -} - #define PI_LOG2_CACHE_LINE_SIZE 5 #define PI_INDEX_BITS 12 #define PI_INDEX_MASK ((1U << PI_INDEX_BITS) - 1) @@ -191,6 +165,15 @@ static inline u64 vnic_cached_posted_index(dma_addr_t addr, unsigned int len, PI_PREFETCH_ADDR_MASK) << PI_PREFETCH_ADDR_OFF); } +static inline uint32_t +buf_idx_incr(uint32_t n_descriptors, uint32_t idx) +{ + idx++; + if (unlikely(idx == n_descriptors)) + idx = 0; + return idx; +} + static inline void vnic_wq_service(struct vnic_wq *wq, struct cq_desc *cq_desc, u16 completed_index, void (*buf_service)(struct vnic_wq *wq, @@ -198,21 +181,24 @@ static inline void vnic_wq_service(struct vnic_wq *wq, void *opaque) { struct vnic_wq_buf *buf; + unsigned int to_clean = wq->tail_idx; - buf = wq->to_clean; + buf = &wq->bufs[to_clean]; while (1) { (*buf_service)(wq, cq_desc, buf, opaque); wq->ring.desc_avail++; - wq->to_clean = buf->next; - if (buf->index == completed_index) + to_clean = buf_idx_incr(wq->ring.desc_count, to_clean); + + if (to_clean == completed_index) break; - buf = wq->to_clean; + buf = &wq->bufs[to_clean]; } + wq->tail_idx = to_clean; } void vnic_wq_free(struct vnic_wq *wq); diff --git a/drivers/net/enic/enic.h b/drivers/net/enic/enic.h index 5b58a65..fe186de 100644 --- a/drivers/net/enic/enic.h +++ b/drivers/net/enic/enic.h @@ -163,6 +163,30 @@ static inline struct enic *pmd_priv(struct rte_eth_dev *eth_dev) return (struct enic *)eth_dev->data->dev_private; } +static inline uint32_t +enic_ring_add(uint32_t n_descriptors, uint32_t i0, uint32_t i1) +{ + uint32_t d = i0 + i1; + d -= (d >= n_descriptors) ? n_descriptors : 0; + return d; +} + +static inline uint32_t +enic_ring_sub(uint32_t n_descriptors, uint32_t i0, uint32_t i1) +{ + int32_t d = i1 - i0; + return (uint32_t)((d < 0) ? ((int32_t)n_descriptors + d) : d); +} + +static inline uint32_t +enic_ring_incr(uint32_t n_descriptors, uint32_t idx) +{ + idx++; + if (unlikely(idx == n_descriptors)) + idx = 0; + return idx; +} + extern void enic_fdir_stats_get(struct enic *enic, struct rte_eth_fdir_stats *stats); extern int enic_fdir_add_fltr(struct enic *enic, diff --git a/drivers/net/enic/enic_main.c b/drivers/net/enic/enic_main.c index f41ef86..5bf5fcf 100644 --- a/drivers/net/enic/enic_main.c +++ b/drivers/net/enic/enic_main.c @@ -105,10 +105,10 @@ void enic_set_hdr_split_size(struct enic *enic, u16 split_hdr_size) void enic_free_wq_buf(__rte_unused struct vnic_wq *wq, struct vnic_wq_buf *buf) { - struct rte_mbuf *mbuf = (struct rte_mbuf *)buf->os_buf; + struct rte_mbuf *mbuf = (struct rte_mbuf *)buf->mb; rte_mempool_put(mbuf->pool, mbuf); - buf->os_buf = NULL; + buf->mb = NULL; } static void enic_log_q_error(struct enic *enic) diff --git a/drivers/net/enic/enic_rxtx.c b/drivers/net/enic/enic_rxtx.c index 530e8a3..ea31dfa 100644 --- a/drivers/net/enic/enic_rxtx.c +++ b/drivers/net/enic/enic_rxtx.c @@ -214,17 +214,6 @@ enic_cq_rx_to_pkt_flags(struct cq_desc *cqd, struct rte_mbuf *mbuf) mbuf->ol_flags = pkt_flags; } -static inline uint32_t -enic_ring_add(uint32_t n_descriptors, uint32_t i0, uint32_t i1) -{ - uint32_t d = i0 + i1; - RTE_ASSERT(i0 < n_descriptors); - RTE_ASSERT(i1 < n_descriptors); - d -= (d >= n_descriptors) ? n_descriptors : 0; - return d; -} - - uint16_t enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) @@ -376,12 +365,15 @@ void enic_send_pkt(struct enic *enic, struct vnic_wq *wq, uint8_t sop, uint8_t eop, uint8_t cq_entry, uint16_t ol_flags, uint16_t vlan_tag) { - struct wq_enet_desc *desc = vnic_wq_next_desc(wq); + struct wq_enet_desc *desc, *descs; uint16_t mss = 0; uint8_t vlan_tag_insert = 0; uint64_t bus_addr = (dma_addr_t) (tx_pkt->buf_physaddr + tx_pkt->data_off); + descs = (struct wq_enet_desc *)wq->ring.descs; + desc = descs + wq->head_idx; + if (sop) { if (ol_flags & PKT_TX_VLAN_PKT) vlan_tag_insert = 1; @@ -408,12 +400,7 @@ void enic_send_pkt(struct enic *enic, struct vnic_wq *wq, vlan_tag, 0 /* loopback */); - enic_vnic_post_wq(wq, (void *)tx_pkt, bus_addr, len, - sop, - 1 /*desc_skip_cnt*/, - cq_entry, - 0 /*compressed send*/, - 0 /*wrid*/); + enic_vnic_post_wq(wq, (void *)tx_pkt, cq_entry); } uint16_t enic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v3 07/13] enic: use Tx completion messages instead of descriptors 2016-06-03 0:22 ` [PATCH v3 00/13] enic counter fixes and Tx optimization John Daley ` (5 preceding siblings ...) 2016-06-03 0:22 ` [PATCH v3 06/13] enic: streamline mbuf handling " John Daley @ 2016-06-03 0:22 ` John Daley 2016-06-10 21:18 ` Bruce Richardson 2016-06-03 0:22 ` [PATCH v3 08/13] enic: refactor Tx mbuf recycling John Daley ` (6 subsequent siblings) 13 siblings, 1 reply; 31+ messages in thread From: John Daley @ 2016-06-03 0:22 UTC (permalink / raw) To: dev; +Cc: bruce.richarsdon, John Daley The NIC can either DMA a separate completion message for each completed send or periodically just DMA an index of the last completed send. Switch to the second method which improves cache locality and performance. Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/base/vnic_wq.c | 1 + drivers/net/enic/base/vnic_wq.h | 3 +++ drivers/net/enic/enic_main.c | 43 ++++++++++++++++++++++++++++++++--------- drivers/net/enic/enic_rxtx.c | 11 +++++++---- 4 files changed, 45 insertions(+), 13 deletions(-) diff --git a/drivers/net/enic/base/vnic_wq.c b/drivers/net/enic/base/vnic_wq.c index ab81c7e..cfef1af 100644 --- a/drivers/net/enic/base/vnic_wq.c +++ b/drivers/net/enic/base/vnic_wq.c @@ -142,6 +142,7 @@ void vnic_wq_init(struct vnic_wq *wq, unsigned int cq_index, vnic_wq_init_start(wq, cq_index, 0, 0, error_interrupt_enable, error_interrupt_offset); + wq->last_completed_index = 0; } void vnic_wq_error_out(struct vnic_wq *wq, unsigned int error) diff --git a/drivers/net/enic/base/vnic_wq.h b/drivers/net/enic/base/vnic_wq.h index a6759f5..fe46bb4 100644 --- a/drivers/net/enic/base/vnic_wq.h +++ b/drivers/net/enic/base/vnic_wq.h @@ -38,6 +38,7 @@ #include "vnic_dev.h" #include "vnic_cq.h" +#include <rte_memzone.h> /* Work queue control */ struct vnic_wq_ctrl { @@ -79,6 +80,8 @@ struct vnic_wq { unsigned int tail_idx; unsigned int pkts_outstanding; unsigned int socket_id; + const struct rte_memzone *cqmsg_rz; + uint16_t last_completed_index; }; static inline unsigned int vnic_wq_desc_avail(struct vnic_wq *wq) diff --git a/drivers/net/enic/enic_main.c b/drivers/net/enic/enic_main.c index 5bf5fcf..eaa206e 100644 --- a/drivers/net/enic/enic_main.c +++ b/drivers/net/enic/enic_main.c @@ -97,7 +97,6 @@ enic_rxmbuf_queue_release(struct enic *enic, struct vnic_rq *rq) } } - void enic_set_hdr_split_size(struct enic *enic, u16 split_hdr_size) { vnic_set_hdr_split_size(enic->vdev, split_hdr_size); @@ -235,12 +234,26 @@ void enic_init_vnic_resources(struct enic *enic) unsigned int error_interrupt_enable = 1; unsigned int error_interrupt_offset = 0; unsigned int index = 0; + unsigned int cq_idx; for (index = 0; index < enic->rq_count; index++) { vnic_rq_init(&enic->rq[index], enic_cq_rq(enic, index), error_interrupt_enable, error_interrupt_offset); + + cq_idx = enic_cq_rq(enic, index); + vnic_cq_init(&enic->cq[cq_idx], + 0 /* flow_control_enable */, + 1 /* color_enable */, + 0 /* cq_head */, + 0 /* cq_tail */, + 1 /* cq_tail_color */, + 0 /* interrupt_enable */, + 1 /* cq_entry_enable */, + 0 /* cq_message_enable */, + 0 /* interrupt offset */, + 0 /* cq_message_addr */); } for (index = 0; index < enic->wq_count; index++) { @@ -248,22 +261,19 @@ void enic_init_vnic_resources(struct enic *enic) enic_cq_wq(enic, index), error_interrupt_enable, error_interrupt_offset); - } - vnic_dev_stats_clear(enic->vdev); - - for (index = 0; index < enic->cq_count; index++) { - vnic_cq_init(&enic->cq[index], + cq_idx = enic_cq_wq(enic, index); + vnic_cq_init(&enic->cq[cq_idx], 0 /* flow_control_enable */, 1 /* color_enable */, 0 /* cq_head */, 0 /* cq_tail */, 1 /* cq_tail_color */, 0 /* interrupt_enable */, - 1 /* cq_entry_enable */, - 0 /* cq_message_enable */, + 0 /* cq_entry_enable */, + 1 /* cq_message_enable */, 0 /* interrupt offset */, - 0 /* cq_message_addr */); + (u64)enic->wq[index].cqmsg_rz->phys_addr); } vnic_intr_init(&enic->intr, @@ -507,6 +517,7 @@ void enic_free_wq(void *txq) struct vnic_wq *wq = (struct vnic_wq *)txq; struct enic *enic = vnic_dev_priv(wq->vdev); + rte_memzone_free(wq->cqmsg_rz); vnic_wq_free(wq); vnic_cq_free(&enic->cq[enic->rq_count + wq->index]); } @@ -517,6 +528,8 @@ int enic_alloc_wq(struct enic *enic, uint16_t queue_idx, int err; struct vnic_wq *wq = &enic->wq[queue_idx]; unsigned int cq_index = enic_cq_wq(enic, queue_idx); + char name[NAME_MAX]; + static int instance; wq->socket_id = socket_id; if (nb_desc) { @@ -552,6 +565,18 @@ int enic_alloc_wq(struct enic *enic, uint16_t queue_idx, dev_err(enic, "error in allocation of cq for wq\n"); } + /* setup up CQ message */ + snprintf((char *)name, sizeof(name), + "vnic_cqmsg-%s-%d-%d", enic->bdf_name, queue_idx, + instance++); + + wq->cqmsg_rz = rte_memzone_reserve_aligned((const char *)name, + sizeof(uint32_t), + SOCKET_ID_ANY, 0, + ENIC_ALIGN); + if (!wq->cqmsg_rz) + return -ENOMEM; + return err; } diff --git a/drivers/net/enic/enic_rxtx.c b/drivers/net/enic/enic_rxtx.c index ea31dfa..2a54333 100644 --- a/drivers/net/enic/enic_rxtx.c +++ b/drivers/net/enic/enic_rxtx.c @@ -348,11 +348,14 @@ static int enic_wq_service(struct vnic_dev *vdev, struct cq_desc *cq_desc, unsigned int enic_cleanup_wq(struct enic *enic, struct vnic_wq *wq) { - unsigned int cq = enic_cq_wq(enic, wq->index); + u16 completed_index = *((uint32_t *)wq->cqmsg_rz->addr) & 0xffff; - /* Return the work done */ - return vnic_cq_service(&enic->cq[cq], - -1 /*wq_work_to_do*/, enic_wq_service, NULL); + if (wq->last_completed_index != completed_index) { + enic_wq_service(enic->vdev, NULL, 0, wq->index, + completed_index, NULL); + wq->last_completed_index = completed_index; + } + return 0; } void enic_post_wq_index(struct vnic_wq *wq) -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* Re: [PATCH v3 07/13] enic: use Tx completion messages instead of descriptors 2016-06-03 0:22 ` [PATCH v3 07/13] enic: use Tx completion messages instead of descriptors John Daley @ 2016-06-10 21:18 ` Bruce Richardson 2016-06-10 22:28 ` John Daley (johndale) 0 siblings, 1 reply; 31+ messages in thread From: Bruce Richardson @ 2016-06-10 21:18 UTC (permalink / raw) To: John Daley; +Cc: dev, bruce.richarsdon On Thu, Jun 02, 2016 at 05:22:51PM -0700, John Daley wrote: > The NIC can either DMA a separate completion message for each > completed send or periodically just DMA an index of the last > completed send. Switch to the second method which improves > cache locality and performance. > > Signed-off-by: John Daley <johndale@cisco.com> Can you perhaps send me an updated wording for this commit message as the title and commit message conflict. The title says to use completion messages not descriptors, while the body talks about moving away from a completion message way of working. Is the former method a descriptor writeback method, while the latter a head pointer writeback? If so, I think the title could be: "enic: use Tx head pointer not descriptor writeback" or something similar. Again, if you send on the updated commit text, I'll just update it on apply. I'd ideally like to get this patchset pushed to next-net first thing Monday. /Bruce ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v3 07/13] enic: use Tx completion messages instead of descriptors 2016-06-10 21:18 ` Bruce Richardson @ 2016-06-10 22:28 ` John Daley (johndale) 0 siblings, 0 replies; 31+ messages in thread From: John Daley (johndale) @ 2016-06-10 22:28 UTC (permalink / raw) To: Bruce Richardson; +Cc: dev, bruce.richarsdon > -----Original Message----- > From: Bruce Richardson [mailto:bruce.richardson@intel.com] > Sent: Friday, June 10, 2016 2:18 PM > To: John Daley (johndale) <johndale@cisco.com> > Cc: dev@dpdk.org; bruce.richarsdon@intel.com > Subject: Re: [dpdk-dev] [PATCH v3 07/13] enic: use Tx completion messages > instead of descriptors > > On Thu, Jun 02, 2016 at 05:22:51PM -0700, John Daley wrote: > > The NIC can either DMA a separate completion message for each > > completed send or periodically just DMA an index of the last completed > > send. Switch to the second method which improves cache locality and > > performance. > > > > Signed-off-by: John Daley <johndale@cisco.com> > > Can you perhaps send me an updated wording for this commit message as > the title and commit message conflict. The title says to use completion > messages not descriptors, while the body talks about moving away from a > completion message way of working. > Is the former method a descriptor writeback method, while the latter a head > pointer writeback? If so, I think the title could be: > > "enic: use Tx head pointer not descriptor writeback" > > or something similar. > > Again, if you send on the updated commit text, I'll just update it on apply. I'd > ideally like to get this patchset pushed to next-net first thing Monday. Ok, I agree that it is confusing. We moved from having the hardware send a completion descriptor for every packet to having it send the index of the last completed packet every once in a while. We can use the word 'index' and 'message' to describe the 2 methods and drop the word 'descriptor'. Here is a suggestion: enic: use Tx completion index instead of completion messages The NIC can either DMA a separate completion message for each completed send or periodically just DMA an index of the last completed send. Switch to the latter method which improves cache locality and performance. Thank you, John > > /Bruce ^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v3 08/13] enic: refactor Tx mbuf recycling 2016-06-03 0:22 ` [PATCH v3 00/13] enic counter fixes and Tx optimization John Daley ` (6 preceding siblings ...) 2016-06-03 0:22 ` [PATCH v3 07/13] enic: use Tx completion messages instead of descriptors John Daley @ 2016-06-03 0:22 ` John Daley 2016-06-03 0:22 ` [PATCH v3 09/13] enic: optimize the Tx function John Daley ` (5 subsequent siblings) 13 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-06-03 0:22 UTC (permalink / raw) To: dev; +Cc: bruce.richarsdon, John Daley Mbufs were returned to the pool one at a time. Use rte_mempool_put_bulk instead. There were muiltiple function calls for each buffer returned. Refactor this code into just 2 functions. Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/base/vnic_wq.h | 27 --------------------- drivers/net/enic/enic_rxtx.c | 54 ++++++++++++++++++++++++++--------------- 2 files changed, 35 insertions(+), 46 deletions(-) diff --git a/drivers/net/enic/base/vnic_wq.h b/drivers/net/enic/base/vnic_wq.h index fe46bb4..689b81c 100644 --- a/drivers/net/enic/base/vnic_wq.h +++ b/drivers/net/enic/base/vnic_wq.h @@ -177,33 +177,6 @@ buf_idx_incr(uint32_t n_descriptors, uint32_t idx) return idx; } -static inline void vnic_wq_service(struct vnic_wq *wq, - struct cq_desc *cq_desc, u16 completed_index, - void (*buf_service)(struct vnic_wq *wq, - struct cq_desc *cq_desc, struct vnic_wq_buf *buf, void *opaque), - void *opaque) -{ - struct vnic_wq_buf *buf; - unsigned int to_clean = wq->tail_idx; - - buf = &wq->bufs[to_clean]; - while (1) { - - (*buf_service)(wq, cq_desc, buf, opaque); - - wq->ring.desc_avail++; - - - to_clean = buf_idx_incr(wq->ring.desc_count, to_clean); - - if (to_clean == completed_index) - break; - - buf = &wq->bufs[to_clean]; - } - wq->tail_idx = to_clean; -} - void vnic_wq_free(struct vnic_wq *wq); int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index, unsigned int desc_count, unsigned int desc_size); diff --git a/drivers/net/enic/enic_rxtx.c b/drivers/net/enic/enic_rxtx.c index 2a54333..ec8d90a 100644 --- a/drivers/net/enic/enic_rxtx.c +++ b/drivers/net/enic/enic_rxtx.c @@ -326,33 +326,49 @@ enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, return nb_rx; } -static void enic_wq_free_buf(struct vnic_wq *wq, - __rte_unused struct cq_desc *cq_desc, - struct vnic_wq_buf *buf, - __rte_unused void *opaque) +static inline void enic_free_wq_bufs(struct vnic_wq *wq, u16 completed_index) { - enic_free_wq_buf(wq, buf); -} - -static int enic_wq_service(struct vnic_dev *vdev, struct cq_desc *cq_desc, - __rte_unused u8 type, u16 q_number, u16 completed_index, void *opaque) -{ - struct enic *enic = vnic_dev_priv(vdev); + struct vnic_wq_buf *buf; + struct rte_mbuf *m, *free[ENIC_MAX_WQ_DESCS]; + unsigned int nb_to_free, nb_free = 0, i; + struct rte_mempool *pool; + unsigned int tail_idx; + unsigned int desc_count = wq->ring.desc_count; + + nb_to_free = enic_ring_sub(desc_count, wq->tail_idx, completed_index) + + 1; + tail_idx = wq->tail_idx; + buf = &wq->bufs[tail_idx]; + pool = ((struct rte_mbuf *)buf->mb)->pool; + for (i = 0; i < nb_to_free; i++) { + buf = &wq->bufs[tail_idx]; + m = (struct rte_mbuf *)(buf->mb); + if (likely(m->pool == pool)) { + free[nb_free++] = m; + } else { + rte_mempool_put_bulk(pool, (void *)free, nb_free); + free[0] = m; + nb_free = 1; + pool = m->pool; + } + tail_idx = enic_ring_incr(desc_count, tail_idx); + buf->mb = NULL; + } - vnic_wq_service(&enic->wq[q_number], cq_desc, - completed_index, enic_wq_free_buf, - opaque); + rte_mempool_put_bulk(pool, (void **)free, nb_free); - return 0; + wq->tail_idx = tail_idx; + wq->ring.desc_avail += nb_to_free; } -unsigned int enic_cleanup_wq(struct enic *enic, struct vnic_wq *wq) +unsigned int enic_cleanup_wq(__rte_unused struct enic *enic, struct vnic_wq *wq) { - u16 completed_index = *((uint32_t *)wq->cqmsg_rz->addr) & 0xffff; + u16 completed_index; + + completed_index = *((uint32_t *)wq->cqmsg_rz->addr) & 0xffff; if (wq->last_completed_index != completed_index) { - enic_wq_service(enic->vdev, NULL, 0, wq->index, - completed_index, NULL); + enic_free_wq_bufs(wq, completed_index); wq->last_completed_index = completed_index; } return 0; -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v3 09/13] enic: optimize the Tx function 2016-06-03 0:22 ` [PATCH v3 00/13] enic counter fixes and Tx optimization John Daley ` (7 preceding siblings ...) 2016-06-03 0:22 ` [PATCH v3 08/13] enic: refactor Tx mbuf recycling John Daley @ 2016-06-03 0:22 ` John Daley 2016-06-03 0:22 ` [PATCH v3 10/13] enic: remove unused files and functions and variables John Daley ` (4 subsequent siblings) 13 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-06-03 0:22 UTC (permalink / raw) To: dev; +Cc: bruce.richarsdon, John Daley, Nelson Escobar Reduce host CPU overhead of Tx packet processing: * Use local variables inside per packet loop instead of fields in structs. * Factor book keeping and conditionals out of the per packet loop where possible. * Post buffers to the nic at a maximum of every 64 packets Signed-off-by: Nelson Escobar <neescoba@cisco.com> Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/base/vnic_wq.h | 1 + drivers/net/enic/enic_res.h | 2 +- drivers/net/enic/enic_rxtx.c | 167 +++++++++++++++++++--------------------- 3 files changed, 83 insertions(+), 87 deletions(-) diff --git a/drivers/net/enic/base/vnic_wq.h b/drivers/net/enic/base/vnic_wq.h index 689b81c..7a66813 100644 --- a/drivers/net/enic/base/vnic_wq.h +++ b/drivers/net/enic/base/vnic_wq.h @@ -67,6 +67,7 @@ struct vnic_wq_ctrl { /* 16 bytes */ struct vnic_wq_buf { + struct rte_mempool *pool; void *mb; }; diff --git a/drivers/net/enic/enic_res.h b/drivers/net/enic/enic_res.h index 955db71..3c8e303 100644 --- a/drivers/net/enic/enic_res.h +++ b/drivers/net/enic/enic_res.h @@ -53,7 +53,7 @@ #define ENIC_NON_TSO_MAX_DESC 16 #define ENIC_DEFAULT_RX_FREE_THRESH 32 -#define ENIC_TX_POST_THRESH (ENIC_MIN_WQ_DESCS / 2) +#define ENIC_TX_XMIT_MAX 64 #define ENIC_SETTING(enic, f) ((enic->config.flags & VENETF_##f) ? 1 : 0) diff --git a/drivers/net/enic/enic_rxtx.c b/drivers/net/enic/enic_rxtx.c index ec8d90a..ba15670 100644 --- a/drivers/net/enic/enic_rxtx.c +++ b/drivers/net/enic/enic_rxtx.c @@ -374,114 +374,109 @@ unsigned int enic_cleanup_wq(__rte_unused struct enic *enic, struct vnic_wq *wq) return 0; } -void enic_post_wq_index(struct vnic_wq *wq) -{ - enic_vnic_post_wq_index(wq); -} - -void enic_send_pkt(struct enic *enic, struct vnic_wq *wq, - struct rte_mbuf *tx_pkt, unsigned short len, - uint8_t sop, uint8_t eop, uint8_t cq_entry, - uint16_t ol_flags, uint16_t vlan_tag) -{ - struct wq_enet_desc *desc, *descs; - uint16_t mss = 0; - uint8_t vlan_tag_insert = 0; - uint64_t bus_addr = (dma_addr_t) - (tx_pkt->buf_physaddr + tx_pkt->data_off); - - descs = (struct wq_enet_desc *)wq->ring.descs; - desc = descs + wq->head_idx; - - if (sop) { - if (ol_flags & PKT_TX_VLAN_PKT) - vlan_tag_insert = 1; - - if (enic->hw_ip_checksum) { - if (ol_flags & PKT_TX_IP_CKSUM) - mss |= ENIC_CALC_IP_CKSUM; - - if (ol_flags & PKT_TX_TCP_UDP_CKSUM) - mss |= ENIC_CALC_TCP_UDP_CKSUM; - } - } - - wq_enet_desc_enc(desc, - bus_addr, - len, - mss, - 0 /* header_length */, - 0 /* offload_mode WQ_ENET_OFFLOAD_MODE_CSUM */, - eop, - cq_entry, - 0 /* fcoe_encap */, - vlan_tag_insert, - vlan_tag, - 0 /* loopback */); - - enic_vnic_post_wq(wq, (void *)tx_pkt, cq_entry); -} - uint16_t enic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) { uint16_t index; - unsigned int frags; - unsigned int pkt_len; - unsigned int seg_len; - unsigned int inc_len; + unsigned int pkt_len, data_len; unsigned int nb_segs; - struct rte_mbuf *tx_pkt, *next_tx_pkt; + struct rte_mbuf *tx_pkt; struct vnic_wq *wq = (struct vnic_wq *)tx_queue; struct enic *enic = vnic_dev_priv(wq->vdev); unsigned short vlan_id; unsigned short ol_flags; - uint8_t last_seg, eop; - unsigned int host_tx_descs = 0; + unsigned int wq_desc_avail; + int head_idx; + struct vnic_wq_buf *buf; + unsigned int hw_ip_cksum_enabled; + unsigned int desc_count; + struct wq_enet_desc *descs, *desc_p, desc_tmp; + uint16_t mss; + uint8_t vlan_tag_insert; + uint8_t eop; + uint64_t bus_addr; + enic_cleanup_wq(enic, wq); + wq_desc_avail = vnic_wq_desc_avail(wq); + head_idx = wq->head_idx; + desc_count = wq->ring.desc_count; + + nb_pkts = RTE_MIN(nb_pkts, ENIC_TX_XMIT_MAX); + + hw_ip_cksum_enabled = enic->hw_ip_checksum; for (index = 0; index < nb_pkts; index++) { tx_pkt = *tx_pkts++; - inc_len = 0; nb_segs = tx_pkt->nb_segs; - if (nb_segs > vnic_wq_desc_avail(wq)) { + if (nb_segs > wq_desc_avail) { if (index > 0) - enic_post_wq_index(wq); - - /* wq cleanup and try again */ - if (!enic_cleanup_wq(enic, wq) || - (nb_segs > vnic_wq_desc_avail(wq))) { - return index; - } + goto post; + goto done; } pkt_len = tx_pkt->pkt_len; + data_len = tx_pkt->data_len; vlan_id = tx_pkt->vlan_tci; ol_flags = tx_pkt->ol_flags; - for (frags = 0; inc_len < pkt_len; frags++) { - if (!tx_pkt) - break; - next_tx_pkt = tx_pkt->next; - seg_len = tx_pkt->data_len; - inc_len += seg_len; - - host_tx_descs++; - last_seg = 0; - eop = 0; - if ((pkt_len == inc_len) || !next_tx_pkt) { - eop = 1; - /* post if last packet in batch or > thresh */ - if ((index == (nb_pkts - 1)) || - (host_tx_descs > ENIC_TX_POST_THRESH)) { - last_seg = 1; - host_tx_descs = 0; - } + + mss = 0; + vlan_tag_insert = 0; + bus_addr = (dma_addr_t) + (tx_pkt->buf_physaddr + tx_pkt->data_off); + + descs = (struct wq_enet_desc *)wq->ring.descs; + desc_p = descs + head_idx; + + eop = (data_len == pkt_len); + + if (ol_flags & PKT_TX_VLAN_PKT) + vlan_tag_insert = 1; + + if (hw_ip_cksum_enabled && (ol_flags & PKT_TX_IP_CKSUM)) + mss |= ENIC_CALC_IP_CKSUM; + + if (hw_ip_cksum_enabled && (ol_flags & PKT_TX_TCP_UDP_CKSUM)) + mss |= ENIC_CALC_TCP_UDP_CKSUM; + + wq_enet_desc_enc(&desc_tmp, bus_addr, data_len, mss, 0, 0, eop, + eop, 0, vlan_tag_insert, vlan_id, 0); + + *desc_p = desc_tmp; + buf = &wq->bufs[head_idx]; + buf->mb = (void *)tx_pkt; + head_idx = enic_ring_incr(desc_count, head_idx); + wq_desc_avail--; + + if (!eop) { + for (tx_pkt = tx_pkt->next; tx_pkt; tx_pkt = + tx_pkt->next) { + data_len = tx_pkt->data_len; + + if (tx_pkt->next == NULL) + eop = 1; + desc_p = descs + head_idx; + bus_addr = (dma_addr_t)(tx_pkt->buf_physaddr + + tx_pkt->data_off); + wq_enet_desc_enc((struct wq_enet_desc *) + &desc_tmp, bus_addr, data_len, + mss, 0, 0, eop, eop, 0, + vlan_tag_insert, vlan_id, 0); + + *desc_p = desc_tmp; + buf = &wq->bufs[head_idx]; + buf->mb = (void *)tx_pkt; + head_idx = enic_ring_incr(desc_count, head_idx); + wq_desc_avail--; } - enic_send_pkt(enic, wq, tx_pkt, (unsigned short)seg_len, - !frags, eop, last_seg, ol_flags, vlan_id); - tx_pkt = next_tx_pkt; } } + post: + rte_wmb(); + iowrite32(head_idx, &wq->ctrl->posted_index); + done: + wq->ring.desc_avail = wq_desc_avail; + wq->head_idx = head_idx; - enic_cleanup_wq(enic, wq); return index; } + + -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v3 10/13] enic: remove unused files and functions and variables 2016-06-03 0:22 ` [PATCH v3 00/13] enic counter fixes and Tx optimization John Daley ` (8 preceding siblings ...) 2016-06-03 0:22 ` [PATCH v3 09/13] enic: optimize the Tx function John Daley @ 2016-06-03 0:22 ` John Daley 2016-06-03 0:22 ` [PATCH v3 11/13] enic: add an enic assert macro John Daley ` (3 subsequent siblings) 13 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-06-03 0:22 UTC (permalink / raw) To: dev; +Cc: bruce.richarsdon, John Daley Remove some files, functions and variables left unused after Tx performance improvements. Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/base/enic_vnic_wq.h | 63 ------------------------------------ drivers/net/enic/base/vnic_cq.h | 44 ------------------------- drivers/net/enic/base/vnic_wq.c | 4 +-- drivers/net/enic/base/vnic_wq.h | 6 +--- drivers/net/enic/enic.h | 1 - drivers/net/enic/enic_main.c | 2 +- drivers/net/enic/enic_rxtx.c | 1 - 7 files changed, 4 insertions(+), 117 deletions(-) delete mode 100644 drivers/net/enic/base/enic_vnic_wq.h diff --git a/drivers/net/enic/base/enic_vnic_wq.h b/drivers/net/enic/base/enic_vnic_wq.h deleted file mode 100644 index 55c5664..0000000 --- a/drivers/net/enic/base/enic_vnic_wq.h +++ /dev/null @@ -1,63 +0,0 @@ -/* - * Copyright 2008-2015 Cisco Systems, Inc. All rights reserved. - * Copyright 2007 Nuova Systems, Inc. All rights reserved. - * - * Copyright (c) 2015, Cisco Systems, Inc. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in - * the documentation and/or other materials provided with the - * distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS - * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE - * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, - * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, - * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; - * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER - * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN - * ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * - */ - -#ifndef _ENIC_VNIC_WQ_H_ -#define _ENIC_VNIC_WQ_H_ - -#include "vnic_dev.h" -#include "vnic_cq.h" - -static inline void enic_vnic_post_wq_index(struct vnic_wq *wq) -{ - /* Adding write memory barrier prevents compiler and/or CPU - * reordering, thus avoiding descriptor posting before - * descriptor is initialized. Otherwise, hardware can read - * stale descriptor fields. - */ - wmb(); - iowrite32(wq->head_idx, &wq->ctrl->posted_index); -} - -static inline void enic_vnic_post_wq(struct vnic_wq *wq, - void *os_buf, uint8_t cq_entry) -{ - struct vnic_wq_buf *buf = &wq->bufs[wq->head_idx]; - buf->mb = os_buf; - wq->head_idx = enic_ring_incr(wq->ring.desc_count, wq->head_idx); - - if (cq_entry) - enic_vnic_post_wq_index(wq); -} - -#endif /* _ENIC_VNIC_WQ_H_ */ diff --git a/drivers/net/enic/base/vnic_cq.h b/drivers/net/enic/base/vnic_cq.h index 922391b..13ab87c 100644 --- a/drivers/net/enic/base/vnic_cq.h +++ b/drivers/net/enic/base/vnic_cq.h @@ -90,50 +90,6 @@ struct vnic_cq { #endif }; -static inline unsigned int vnic_cq_service(struct vnic_cq *cq, - unsigned int work_to_do, - int (*q_service)(struct vnic_dev *vdev, struct cq_desc *cq_desc, - u8 type, u16 q_number, u16 completed_index, void *opaque), - void *opaque) -{ - struct cq_desc *cq_desc; - unsigned int work_done = 0; - u16 q_number, completed_index; - u8 type, color; - struct rte_mbuf **rx_pkts = opaque; - unsigned int ret; - - cq_desc = (struct cq_desc *)((u8 *)cq->ring.descs + - cq->ring.desc_size * cq->to_clean); - cq_desc_dec(cq_desc, &type, &color, - &q_number, &completed_index); - - while (color != cq->last_color) { - if (opaque) - opaque = (void *)&(rx_pkts[work_done]); - - ret = (*q_service)(cq->vdev, cq_desc, type, - q_number, completed_index, opaque); - cq->to_clean++; - if (cq->to_clean == cq->ring.desc_count) { - cq->to_clean = 0; - cq->last_color = cq->last_color ? 0 : 1; - } - - cq_desc = (struct cq_desc *)((u8 *)cq->ring.descs + - cq->ring.desc_size * cq->to_clean); - cq_desc_dec(cq_desc, &type, &color, - &q_number, &completed_index); - - if (ret) - work_done++; - if (work_done >= work_to_do) - break; - } - - return work_done; -} - void vnic_cq_free(struct vnic_cq *cq); int vnic_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index, unsigned int socket_id, diff --git a/drivers/net/enic/base/vnic_wq.c b/drivers/net/enic/base/vnic_wq.c index cfef1af..9b9ff4d 100644 --- a/drivers/net/enic/base/vnic_wq.c +++ b/drivers/net/enic/base/vnic_wq.c @@ -179,7 +179,7 @@ int vnic_wq_disable(struct vnic_wq *wq) } void vnic_wq_clean(struct vnic_wq *wq, - void (*buf_clean)(struct vnic_wq *wq, struct vnic_wq_buf *buf)) + void (*buf_clean)(struct vnic_wq_buf *buf)) { struct vnic_wq_buf *buf; unsigned int to_clean = wq->tail_idx; @@ -188,7 +188,7 @@ void vnic_wq_clean(struct vnic_wq *wq, while (vnic_wq_desc_used(wq) > 0) { - (*buf_clean)(wq, buf); + (*buf_clean)(buf); to_clean = buf_idx_incr(wq->ring.desc_count, to_clean); buf = &wq->bufs[to_clean]; diff --git a/drivers/net/enic/base/vnic_wq.h b/drivers/net/enic/base/vnic_wq.h index 7a66813..38a217f 100644 --- a/drivers/net/enic/base/vnic_wq.h +++ b/drivers/net/enic/base/vnic_wq.h @@ -79,7 +79,6 @@ struct vnic_wq { struct vnic_wq_buf *bufs; unsigned int head_idx; unsigned int tail_idx; - unsigned int pkts_outstanding; unsigned int socket_id; const struct rte_memzone *cqmsg_rz; uint16_t last_completed_index; @@ -193,8 +192,5 @@ unsigned int vnic_wq_error_status(struct vnic_wq *wq); void vnic_wq_enable(struct vnic_wq *wq); int vnic_wq_disable(struct vnic_wq *wq); void vnic_wq_clean(struct vnic_wq *wq, - void (*buf_clean)(struct vnic_wq *wq, struct vnic_wq_buf *buf)); -int vnic_wq_mem_size(struct vnic_wq *wq, unsigned int desc_count, - unsigned int desc_size); - + void (*buf_clean)(struct vnic_wq_buf *buf)); #endif /* _VNIC_WQ_H_ */ diff --git a/drivers/net/enic/enic.h b/drivers/net/enic/enic.h index fe186de..4eb28ee 100644 --- a/drivers/net/enic/enic.h +++ b/drivers/net/enic/enic.h @@ -233,5 +233,4 @@ uint16_t enic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); uint16_t enic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts); -void enic_free_wq_buf(__rte_unused struct vnic_wq *wq, struct vnic_wq_buf *buf); #endif /* _ENIC_H_ */ diff --git a/drivers/net/enic/enic_main.c b/drivers/net/enic/enic_main.c index eaa206e..996f999 100644 --- a/drivers/net/enic/enic_main.c +++ b/drivers/net/enic/enic_main.c @@ -102,7 +102,7 @@ void enic_set_hdr_split_size(struct enic *enic, u16 split_hdr_size) vnic_set_hdr_split_size(enic->vdev, split_hdr_size); } -void enic_free_wq_buf(__rte_unused struct vnic_wq *wq, struct vnic_wq_buf *buf) +static void enic_free_wq_buf(struct vnic_wq_buf *buf) { struct rte_mbuf *mbuf = (struct rte_mbuf *)buf->mb; diff --git a/drivers/net/enic/enic_rxtx.c b/drivers/net/enic/enic_rxtx.c index ba15670..a04ebd0 100644 --- a/drivers/net/enic/enic_rxtx.c +++ b/drivers/net/enic/enic_rxtx.c @@ -37,7 +37,6 @@ #include "enic_compat.h" #include "rq_enet_desc.h" #include "enic.h" -#include "enic_vnic_wq.h" #define RTE_PMD_USE_PREFETCH -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v3 11/13] enic: add an enic assert macro 2016-06-03 0:22 ` [PATCH v3 00/13] enic counter fixes and Tx optimization John Daley ` (9 preceding siblings ...) 2016-06-03 0:22 ` [PATCH v3 10/13] enic: remove unused files and functions and variables John Daley @ 2016-06-03 0:22 ` John Daley 2016-06-03 0:22 ` [PATCH v3 12/13] enic: expand local Tx mbuf flags variable to 64-bits John Daley ` (2 subsequent siblings) 13 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-06-03 0:22 UTC (permalink / raw) To: dev; +Cc: bruce.richarsdon, John Daley Add an ASSERT macro for the enic driver which is enabled when the log level is >= RTE_LOG_DEBUG. Assert that number of mbufs to return to the pool in the Tx function is never greater than the max allowed. Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/enic.h | 12 ++++++++++++ drivers/net/enic/enic_rxtx.c | 1 + 2 files changed, 13 insertions(+) diff --git a/drivers/net/enic/enic.h b/drivers/net/enic/enic.h index 4eb28ee..4b76e6d 100644 --- a/drivers/net/enic/enic.h +++ b/drivers/net/enic/enic.h @@ -187,6 +187,18 @@ enic_ring_incr(uint32_t n_descriptors, uint32_t idx) return idx; } +#if RTE_LOG_LEVEL >= RTE_LOG_DEBUG +#define ENIC_ASSERT(cond) \ + do { \ + if (unlikely(!(cond))) { \ + rte_panic("line %d\tassert \"" #cond "\"" \ + "failed\n", __LINE__); \ + } \ + } while (0) +#else +#define ENIC_ASSERT(cond) do {} while (0) +#endif + extern void enic_fdir_stats_get(struct enic *enic, struct rte_eth_fdir_stats *stats); extern int enic_fdir_add_fltr(struct enic *enic, diff --git a/drivers/net/enic/enic_rxtx.c b/drivers/net/enic/enic_rxtx.c index a04ebd0..7527bce 100644 --- a/drivers/net/enic/enic_rxtx.c +++ b/drivers/net/enic/enic_rxtx.c @@ -343,6 +343,7 @@ static inline void enic_free_wq_bufs(struct vnic_wq *wq, u16 completed_index) buf = &wq->bufs[tail_idx]; m = (struct rte_mbuf *)(buf->mb); if (likely(m->pool == pool)) { + ENIC_ASSERT(nb_free < ENIC_MAX_WQ_DESCS); free[nb_free++] = m; } else { rte_mempool_put_bulk(pool, (void *)free, nb_free); -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v3 12/13] enic: expand local Tx mbuf flags variable to 64-bits 2016-06-03 0:22 ` [PATCH v3 00/13] enic counter fixes and Tx optimization John Daley ` (10 preceding siblings ...) 2016-06-03 0:22 ` [PATCH v3 11/13] enic: add an enic assert macro John Daley @ 2016-06-03 0:22 ` John Daley 2016-06-03 8:05 ` Azarewicz, PiotrX T 2016-06-03 0:22 ` [PATCH v3 13/13] enic: fix Tx IP and UDP/TCP checksum offload John Daley 2016-06-10 22:38 ` [PATCH v3 00/13] enic counter fixes and Tx optimization Bruce Richardson 13 siblings, 1 reply; 31+ messages in thread From: John Daley @ 2016-06-03 0:22 UTC (permalink / raw) To: dev; +Cc: bruce.richarsdon, John Daley The offload flags variable (ol_flags) in rte_mbuf structure is 64-bits, so local copy of it must be 64-bits too. Moreover bit comparison between 16-bits variable and 64-bits value make no sense. This breaks Tx vlan IP and L4 offloads. CID 13218 : Operands don't affect result (CONSTANT_EXPRESSION_RESULT) result_independent_of_operands: ol_flags & (18014398509481984ULL /* 1ULL << 54 */) is always 0 regardless of the values of its operands. This occurs as the logical operand of if. Coverity issue: 13218 Fixes: fefed3d1e62c ("enic: new driver") Suggested-by: Piotr Azarewicz <piotrx.t.azarewicz@intel.com> Signed-off-by: John Daley <johndale@cisco.com> --- This is essentially patch http://www.dpdk.org/dev/patchwork/patch/12642 applied after the enic_send_packet function was melded into the main transmit funciton. drivers/net/enic/enic_rxtx.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/net/enic/enic_rxtx.c b/drivers/net/enic/enic_rxtx.c index 7527bce..350d04b 100644 --- a/drivers/net/enic/enic_rxtx.c +++ b/drivers/net/enic/enic_rxtx.c @@ -384,7 +384,7 @@ uint16_t enic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, struct vnic_wq *wq = (struct vnic_wq *)tx_queue; struct enic *enic = vnic_dev_priv(wq->vdev); unsigned short vlan_id; - unsigned short ol_flags; + uint64_t ol_flags; unsigned int wq_desc_avail; int head_idx; struct vnic_wq_buf *buf; -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* Re: [PATCH v3 12/13] enic: expand local Tx mbuf flags variable to 64-bits 2016-06-03 0:22 ` [PATCH v3 12/13] enic: expand local Tx mbuf flags variable to 64-bits John Daley @ 2016-06-03 8:05 ` Azarewicz, PiotrX T 0 siblings, 0 replies; 31+ messages in thread From: Azarewicz, PiotrX T @ 2016-06-03 8:05 UTC (permalink / raw) To: John Daley, dev; +Cc: bruce.richarsdon > Coverity issue: 13218 > Fixes: fefed3d1e62c ("enic: new driver") > > Suggested-by: Piotr Azarewicz <piotrx.t.azarewicz@intel.com> > Signed-off-by: John Daley <johndale@cisco.com> > --- > This is essentially patch http://www.dpdk.org/dev/patchwork/patch/12642 > applied after the enic_send_packet function was melded into the main > transmit funciton. > Acked-by: Piotr Azarewicz <piotrx.t.azarewicz@intel.com> ^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v3 13/13] enic: fix Tx IP and UDP/TCP checksum offload 2016-06-03 0:22 ` [PATCH v3 00/13] enic counter fixes and Tx optimization John Daley ` (11 preceding siblings ...) 2016-06-03 0:22 ` [PATCH v3 12/13] enic: expand local Tx mbuf flags variable to 64-bits John Daley @ 2016-06-03 0:22 ` John Daley 2016-06-10 22:38 ` [PATCH v3 00/13] enic counter fixes and Tx optimization Bruce Richardson 13 siblings, 0 replies; 31+ messages in thread From: John Daley @ 2016-06-03 0:22 UTC (permalink / raw) To: dev; +Cc: bruce.richarsdon, John Daley Private/confilicting ol_flags where used to enable UDP/TCP Tx offloads. Use the common flags in PKT_TX_L4_MASK to support them. Also, do some minor code rearranging for slightly better performane. Fixes: fefed3d1e62c ("enic: new driver") Signed-off-by: John Daley <johndale@cisco.com> --- drivers/net/enic/enic.h | 1 - drivers/net/enic/enic_rxtx.c | 28 ++++++++++++++++++---------- 2 files changed, 18 insertions(+), 11 deletions(-) diff --git a/drivers/net/enic/enic.h b/drivers/net/enic/enic.h index 4b76e6d..1e6914e 100644 --- a/drivers/net/enic/enic.h +++ b/drivers/net/enic/enic.h @@ -62,7 +62,6 @@ #define ENICPMD_SETTING(enic, f) ((enic->config.flags & VENETF_##f) ? 1 : 0) #define ENICPMD_BDF_LENGTH 13 /* 0000:00:00.0'\0' */ -#define PKT_TX_TCP_UDP_CKSUM 0x6000 #define ENIC_CALC_IP_CKSUM 1 #define ENIC_CALC_TCP_UDP_CKSUM 2 #define ENIC_MAX_MTU 9000 diff --git a/drivers/net/enic/enic_rxtx.c b/drivers/net/enic/enic_rxtx.c index 350d04b..8fe334f 100644 --- a/drivers/net/enic/enic_rxtx.c +++ b/drivers/net/enic/enic_rxtx.c @@ -385,10 +385,10 @@ uint16_t enic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, struct enic *enic = vnic_dev_priv(wq->vdev); unsigned short vlan_id; uint64_t ol_flags; + uint64_t ol_flags_mask; unsigned int wq_desc_avail; int head_idx; struct vnic_wq_buf *buf; - unsigned int hw_ip_cksum_enabled; unsigned int desc_count; struct wq_enet_desc *descs, *desc_p, desc_tmp; uint16_t mss; @@ -400,10 +400,10 @@ uint16_t enic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, wq_desc_avail = vnic_wq_desc_avail(wq); head_idx = wq->head_idx; desc_count = wq->ring.desc_count; + ol_flags_mask = PKT_TX_VLAN_PKT | PKT_TX_IP_CKSUM | PKT_TX_L4_MASK; nb_pkts = RTE_MIN(nb_pkts, ENIC_TX_XMIT_MAX); - hw_ip_cksum_enabled = enic->hw_ip_checksum; for (index = 0; index < nb_pkts; index++) { tx_pkt = *tx_pkts++; nb_segs = tx_pkt->nb_segs; @@ -415,10 +415,9 @@ uint16_t enic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, pkt_len = tx_pkt->pkt_len; data_len = tx_pkt->data_len; - vlan_id = tx_pkt->vlan_tci; ol_flags = tx_pkt->ol_flags; - mss = 0; + vlan_id = 0; vlan_tag_insert = 0; bus_addr = (dma_addr_t) (tx_pkt->buf_physaddr + tx_pkt->data_off); @@ -428,14 +427,23 @@ uint16_t enic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, eop = (data_len == pkt_len); - if (ol_flags & PKT_TX_VLAN_PKT) - vlan_tag_insert = 1; + if (ol_flags & ol_flags_mask) { + if (ol_flags & PKT_TX_VLAN_PKT) { + vlan_tag_insert = 1; + vlan_id = tx_pkt->vlan_tci; + } - if (hw_ip_cksum_enabled && (ol_flags & PKT_TX_IP_CKSUM)) - mss |= ENIC_CALC_IP_CKSUM; + if (ol_flags & PKT_TX_IP_CKSUM) + mss |= ENIC_CALC_IP_CKSUM; - if (hw_ip_cksum_enabled && (ol_flags & PKT_TX_TCP_UDP_CKSUM)) - mss |= ENIC_CALC_TCP_UDP_CKSUM; + /* Nic uses just 1 bit for UDP and TCP */ + switch (ol_flags & PKT_TX_L4_MASK) { + case PKT_TX_TCP_CKSUM: + case PKT_TX_UDP_CKSUM: + mss |= ENIC_CALC_TCP_UDP_CKSUM; + break; + } + } wq_enet_desc_enc(&desc_tmp, bus_addr, data_len, mss, 0, 0, eop, eop, 0, vlan_tag_insert, vlan_id, 0); -- 2.7.0 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* Re: [PATCH v3 00/13] enic counter fixes and Tx optimization 2016-06-03 0:22 ` [PATCH v3 00/13] enic counter fixes and Tx optimization John Daley ` (12 preceding siblings ...) 2016-06-03 0:22 ` [PATCH v3 13/13] enic: fix Tx IP and UDP/TCP checksum offload John Daley @ 2016-06-10 22:38 ` Bruce Richardson 13 siblings, 0 replies; 31+ messages in thread From: Bruce Richardson @ 2016-06-10 22:38 UTC (permalink / raw) To: John Daley; +Cc: dev, bruce.richarsdon On Thu, Jun 02, 2016 at 05:22:44PM -0700, John Daley wrote: > The first 3 patches are related to drop counters. The remaining > patches make up refactoring, cleanup bug fixes and optimization of > the Tx path. > > Changes since v2 are: > - Piotr Azarewicz's ol_flags patch > http://www.dpdk.org/dev/patchwork/patch/12642 > - fix Tx IP and UDP/TCP checksum offload > > John Daley (13): > enic: fix Rx drop counters > enic: drop bad packets and remove unused Rx error flag > enic: count truncated packets > enic: put Tx and Rx functions into same file > enic: remove some unused functions in Tx path > enic: streamline mbuf handling in Tx path > enic: use Tx completion messages instead of descriptors > enic: refactor Tx mbuf recycling > enic: optimize the Tx function > enic: remove unused files and functions and variables > enic: add an enic assert macro > enic: expand local Tx mbuf flags variable to 64-bits > enic: fix Tx IP and UDP/TCP checksum offload > Patchset applied to dpdk-next-net/rel_16_07 Thanks, /Bruce ^ permalink raw reply [flat|nested] 31+ messages in thread
end of thread, other threads:[~2016-06-10 22:38 UTC | newest] Thread overview: 31+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-05-24 6:32 [PATCH v2 00/11] enic counter fixes and Tx optimization John Daley 2016-05-24 6:32 ` [PATCH v2 01/11] enic: fix Rx drop counters John Daley 2016-05-24 6:32 ` [PATCH v2 02/11] enic: drop bad packets and remove unused Rx error flag John Daley 2016-05-24 6:32 ` [PATCH v2 03/11] enic: count truncated packets John Daley 2016-05-24 6:32 ` [PATCH v2 04/11] enic: put Tx and Rx functions into same file John Daley 2016-05-24 6:32 ` [PATCH v2 05/11] enic: remove some unused functions in Tx path John Daley 2016-05-24 6:32 ` [PATCH v2 06/11] enic: streamline mbuf handling " John Daley 2016-05-24 6:32 ` [PATCH v2 07/11] enic: use Tx completion messages instead of descriptors John Daley 2016-05-24 6:32 ` [PATCH v2 08/11] enic: refactor Tx mbuf recycling John Daley 2016-05-24 6:32 ` [PATCH v2 09/11] enic: optimize the Tx function John Daley 2016-05-30 10:05 ` Azarewicz, PiotrX T 2016-05-24 6:32 ` [PATCH v2 10/11] enic: remove unused files and functions and variables John Daley 2016-05-24 6:32 ` [PATCH v2 11/11] enic: add an enic assert macro John Daley 2016-06-03 0:22 ` [PATCH v3 00/13] enic counter fixes and Tx optimization John Daley 2016-06-03 0:22 ` [PATCH v3 01/13] enic: fix Rx drop counters John Daley 2016-06-03 0:22 ` [PATCH v3 02/13] enic: drop bad packets and remove unused Rx error flag John Daley 2016-06-03 0:22 ` [PATCH v3 03/13] enic: count truncated packets John Daley 2016-06-03 0:22 ` [PATCH v3 04/13] enic: put Tx and Rx functions into same file John Daley 2016-06-03 0:22 ` [PATCH v3 05/13] enic: remove some unused functions in Tx path John Daley 2016-06-03 0:22 ` [PATCH v3 06/13] enic: streamline mbuf handling " John Daley 2016-06-03 0:22 ` [PATCH v3 07/13] enic: use Tx completion messages instead of descriptors John Daley 2016-06-10 21:18 ` Bruce Richardson 2016-06-10 22:28 ` John Daley (johndale) 2016-06-03 0:22 ` [PATCH v3 08/13] enic: refactor Tx mbuf recycling John Daley 2016-06-03 0:22 ` [PATCH v3 09/13] enic: optimize the Tx function John Daley 2016-06-03 0:22 ` [PATCH v3 10/13] enic: remove unused files and functions and variables John Daley 2016-06-03 0:22 ` [PATCH v3 11/13] enic: add an enic assert macro John Daley 2016-06-03 0:22 ` [PATCH v3 12/13] enic: expand local Tx mbuf flags variable to 64-bits John Daley 2016-06-03 8:05 ` Azarewicz, PiotrX T 2016-06-03 0:22 ` [PATCH v3 13/13] enic: fix Tx IP and UDP/TCP checksum offload John Daley 2016-06-10 22:38 ` [PATCH v3 00/13] enic counter fixes and Tx optimization Bruce Richardson
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.