* [PATCH 0/7] add XDP support to mvneta driver
@ 2019-10-05 20:44 Lorenzo Bianconi
2019-10-05 20:44 ` [PATCH 1/7] net: mvneta: introduce mvneta_update_stats routine Lorenzo Bianconi
` (7 more replies)
0 siblings, 8 replies; 12+ messages in thread
From: Lorenzo Bianconi @ 2019-10-05 20:44 UTC (permalink / raw)
To: netdev
Cc: davem, thomas.petazzoni, ilias.apalodimas, brouer,
lorenzo.bianconi, matteo.croce
Add XDP support to mvneta driver for devices that rely on software
buffer management. Supported verdicts are:
- XDP_DROP
- XDP_PASS
- XDP_REDIRECT
- XDP_TX
Moreover set ndo_xdp_xmit net_device_ops function pointer in order to support
redirecting from other device (e.g. virtio-net).
Convert mvneta driver to page_pool API.
This series is based on previous work done by Jesper and Ilias.
Changes since RFC:
- implement XDP_TX
- make tx pending buffer list agnostic
- code refactoring
- check if device is running in mvneta_xdp_setup
Lorenzo Bianconi (7):
net: mvneta: introduce mvneta_update_stats routine
net: mvneta: introduce page pool API for sw buffer manager
net: mvneta: rely on build_skb in mvneta_rx_swbm poll routine
net: mvneta: add basic XDP support
net: mvneta: move header prefetch in mvneta_swbm_rx_frame
net: mvneta: make tx buffer array agnostic
net: mvneta: add XDP_TX support
drivers/net/ethernet/marvell/Kconfig | 1 +
drivers/net/ethernet/marvell/mvneta.c | 627 +++++++++++++++++++-------
2 files changed, 468 insertions(+), 160 deletions(-)
--
2.21.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 1/7] net: mvneta: introduce mvneta_update_stats routine
2019-10-05 20:44 [PATCH 0/7] add XDP support to mvneta driver Lorenzo Bianconi
@ 2019-10-05 20:44 ` Lorenzo Bianconi
2019-10-05 20:44 ` [PATCH 2/7] net: mvneta: introduce page pool API for sw buffer manager Lorenzo Bianconi
` (6 subsequent siblings)
7 siblings, 0 replies; 12+ messages in thread
From: Lorenzo Bianconi @ 2019-10-05 20:44 UTC (permalink / raw)
To: netdev
Cc: davem, thomas.petazzoni, ilias.apalodimas, brouer,
lorenzo.bianconi, matteo.croce
Introduce mvneta_update_stats routine to collect {rx/tx} statistics
(packets and bytes). This is a preliminary patch to add XDP support to
mvneta driver
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
drivers/net/ethernet/marvell/mvneta.c | 41 +++++++++++++--------------
1 file changed, 20 insertions(+), 21 deletions(-)
diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
index e49820675c8c..128b9fded959 100644
--- a/drivers/net/ethernet/marvell/mvneta.c
+++ b/drivers/net/ethernet/marvell/mvneta.c
@@ -1900,6 +1900,23 @@ static void mvneta_rxq_drop_pkts(struct mvneta_port *pp,
}
}
+static void
+mvneta_update_stats(struct mvneta_port *pp, u32 pkts,
+ u32 len, bool tx)
+{
+ struct mvneta_pcpu_stats *stats = this_cpu_ptr(pp->stats);
+
+ u64_stats_update_begin(&stats->syncp);
+ if (tx) {
+ stats->tx_packets += pkts;
+ stats->tx_bytes += len;
+ } else {
+ stats->rx_packets += pkts;
+ stats->rx_bytes += len;
+ }
+ u64_stats_update_end(&stats->syncp);
+}
+
static inline
int mvneta_rx_refill_queue(struct mvneta_port *pp, struct mvneta_rx_queue *rxq)
{
@@ -2075,14 +2092,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi,
rxq->left_size = 0;
}
- if (rcvd_pkts) {
- struct mvneta_pcpu_stats *stats = this_cpu_ptr(pp->stats);
-
- u64_stats_update_begin(&stats->syncp);
- stats->rx_packets += rcvd_pkts;
- stats->rx_bytes += rcvd_bytes;
- u64_stats_update_end(&stats->syncp);
- }
+ mvneta_update_stats(pp, rcvd_pkts, rcvd_bytes, false);
/* return some buffers to hardware queue, one at a time is too slow */
refill = mvneta_rx_refill_queue(pp, rxq);
@@ -2206,14 +2216,7 @@ static int mvneta_rx_hwbm(struct napi_struct *napi,
napi_gro_receive(napi, skb);
}
- if (rcvd_pkts) {
- struct mvneta_pcpu_stats *stats = this_cpu_ptr(pp->stats);
-
- u64_stats_update_begin(&stats->syncp);
- stats->rx_packets += rcvd_pkts;
- stats->rx_bytes += rcvd_bytes;
- u64_stats_update_end(&stats->syncp);
- }
+ mvneta_update_stats(pp, rcvd_pkts, rcvd_bytes, false);
/* Update rxq management counters */
mvneta_rxq_desc_num_update(pp, rxq, rx_done, rx_done);
@@ -2459,7 +2462,6 @@ static netdev_tx_t mvneta_tx(struct sk_buff *skb, struct net_device *dev)
out:
if (frags > 0) {
- struct mvneta_pcpu_stats *stats = this_cpu_ptr(pp->stats);
struct netdev_queue *nq = netdev_get_tx_queue(dev, txq_id);
netdev_tx_sent_queue(nq, len);
@@ -2474,10 +2476,7 @@ static netdev_tx_t mvneta_tx(struct sk_buff *skb, struct net_device *dev)
else
txq->pending += frags;
- u64_stats_update_begin(&stats->syncp);
- stats->tx_packets++;
- stats->tx_bytes += len;
- u64_stats_update_end(&stats->syncp);
+ mvneta_update_stats(pp, 1, len, true);
} else {
dev->stats.tx_dropped++;
dev_kfree_skb_any(skb);
--
2.21.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 2/7] net: mvneta: introduce page pool API for sw buffer manager
2019-10-05 20:44 [PATCH 0/7] add XDP support to mvneta driver Lorenzo Bianconi
2019-10-05 20:44 ` [PATCH 1/7] net: mvneta: introduce mvneta_update_stats routine Lorenzo Bianconi
@ 2019-10-05 20:44 ` Lorenzo Bianconi
2019-10-05 21:34 ` Ilias Apalodimas
2019-10-05 20:44 ` [PATCH 3/7] net: mvneta: rely on build_skb in mvneta_rx_swbm poll routine Lorenzo Bianconi
` (5 subsequent siblings)
7 siblings, 1 reply; 12+ messages in thread
From: Lorenzo Bianconi @ 2019-10-05 20:44 UTC (permalink / raw)
To: netdev
Cc: davem, thomas.petazzoni, ilias.apalodimas, brouer,
lorenzo.bianconi, matteo.croce
Use the page_pool api for allocations and DMA handling instead of
__dev_alloc_page()/dma_map_page() and free_page()/dma_unmap_page().
Pages are unmapped using page_pool_release_page before packets
go into the network stack.
The page_pool API offers buffer recycling capabilities for XDP but
allocates one page per packet, unless the driver splits and manages
the allocated page.
This is a preliminary patch to add XDP support to mvneta driver
Tested-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Signed-off-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
drivers/net/ethernet/marvell/Kconfig | 1 +
drivers/net/ethernet/marvell/mvneta.c | 76 ++++++++++++++++++++-------
2 files changed, 58 insertions(+), 19 deletions(-)
diff --git a/drivers/net/ethernet/marvell/Kconfig b/drivers/net/ethernet/marvell/Kconfig
index fb942167ee54..3d5caea096fb 100644
--- a/drivers/net/ethernet/marvell/Kconfig
+++ b/drivers/net/ethernet/marvell/Kconfig
@@ -61,6 +61,7 @@ config MVNETA
depends on ARCH_MVEBU || COMPILE_TEST
select MVMDIO
select PHYLINK
+ select PAGE_POOL
---help---
This driver supports the network interface units in the
Marvell ARMADA XP, ARMADA 370, ARMADA 38x and
diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
index 128b9fded959..8beae0e1eda7 100644
--- a/drivers/net/ethernet/marvell/mvneta.c
+++ b/drivers/net/ethernet/marvell/mvneta.c
@@ -37,6 +37,7 @@
#include <net/ip.h>
#include <net/ipv6.h>
#include <net/tso.h>
+#include <net/page_pool.h>
/* Registers */
#define MVNETA_RXQ_CONFIG_REG(q) (0x1400 + ((q) << 2))
@@ -603,6 +604,10 @@ struct mvneta_rx_queue {
u32 pkts_coal;
u32 time_coal;
+ /* page_pool */
+ struct page_pool *page_pool;
+ struct xdp_rxq_info xdp_rxq;
+
/* Virtual address of the RX buffer */
void **buf_virt_addr;
@@ -1815,19 +1820,12 @@ static int mvneta_rx_refill(struct mvneta_port *pp,
dma_addr_t phys_addr;
struct page *page;
- page = __dev_alloc_page(gfp_mask);
+ page = page_pool_alloc_pages(rxq->page_pool,
+ gfp_mask | __GFP_NOWARN);
if (!page)
return -ENOMEM;
- /* map page for use */
- phys_addr = dma_map_page(pp->dev->dev.parent, page, 0, PAGE_SIZE,
- DMA_FROM_DEVICE);
- if (unlikely(dma_mapping_error(pp->dev->dev.parent, phys_addr))) {
- __free_page(page);
- return -ENOMEM;
- }
-
- phys_addr += pp->rx_offset_correction;
+ phys_addr = page_pool_get_dma_addr(page) + pp->rx_offset_correction;
mvneta_rx_desc_fill(rx_desc, phys_addr, page, rxq);
return 0;
}
@@ -1894,10 +1892,11 @@ static void mvneta_rxq_drop_pkts(struct mvneta_port *pp,
if (!data || !(rx_desc->buf_phys_addr))
continue;
- dma_unmap_page(pp->dev->dev.parent, rx_desc->buf_phys_addr,
- PAGE_SIZE, DMA_FROM_DEVICE);
- __free_page(data);
+ page_pool_put_page(rxq->page_pool, data, false);
}
+ if (xdp_rxq_info_is_reg(&rxq->xdp_rxq))
+ xdp_rxq_info_unreg(&rxq->xdp_rxq);
+ page_pool_destroy(rxq->page_pool);
}
static void
@@ -2029,8 +2028,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi,
skb_add_rx_frag(rxq->skb, frag_num, page,
frag_offset, frag_size,
PAGE_SIZE);
- dma_unmap_page(dev->dev.parent, phys_addr,
- PAGE_SIZE, DMA_FROM_DEVICE);
+ page_pool_release_page(rxq->page_pool, page);
rxq->left_size -= frag_size;
}
} else {
@@ -2060,9 +2058,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi,
frag_offset, frag_size,
PAGE_SIZE);
- dma_unmap_page(dev->dev.parent, phys_addr,
- PAGE_SIZE, DMA_FROM_DEVICE);
-
+ page_pool_release_page(rxq->page_pool, page);
rxq->left_size -= frag_size;
}
} /* Middle or Last descriptor */
@@ -2829,11 +2825,53 @@ static int mvneta_poll(struct napi_struct *napi, int budget)
return rx_done;
}
+static int mvneta_create_page_pool(struct mvneta_port *pp,
+ struct mvneta_rx_queue *rxq, int size)
+{
+ struct page_pool_params pp_params = {
+ .order = 0,
+ .flags = PP_FLAG_DMA_MAP,
+ .pool_size = size,
+ .nid = cpu_to_node(0),
+ .dev = pp->dev->dev.parent,
+ .dma_dir = DMA_FROM_DEVICE,
+ };
+ int err;
+
+ rxq->page_pool = page_pool_create(&pp_params);
+ if (IS_ERR(rxq->page_pool)) {
+ err = PTR_ERR(rxq->page_pool);
+ rxq->page_pool = NULL;
+ return err;
+ }
+
+ err = xdp_rxq_info_reg(&rxq->xdp_rxq, pp->dev, 0);
+ if (err < 0)
+ goto err_free_pp;
+
+ err = xdp_rxq_info_reg_mem_model(&rxq->xdp_rxq, MEM_TYPE_PAGE_POOL,
+ rxq->page_pool);
+ if (err)
+ goto err_unregister_rxq;
+
+ return 0;
+
+err_unregister_rxq:
+ xdp_rxq_info_unreg(&rxq->xdp_rxq);
+err_free_pp:
+ page_pool_destroy(rxq->page_pool);
+ return err;
+}
+
/* Handle rxq fill: allocates rxq skbs; called when initializing a port */
static int mvneta_rxq_fill(struct mvneta_port *pp, struct mvneta_rx_queue *rxq,
int num)
{
- int i;
+ int i, err;
+
+ err = mvneta_create_page_pool(pp, rxq, num);
+ if (err < 0)
+ return err;
for (i = 0; i < num; i++) {
memset(rxq->descs + i, 0, sizeof(struct mvneta_rx_desc));
--
2.21.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 3/7] net: mvneta: rely on build_skb in mvneta_rx_swbm poll routine
2019-10-05 20:44 [PATCH 0/7] add XDP support to mvneta driver Lorenzo Bianconi
2019-10-05 20:44 ` [PATCH 1/7] net: mvneta: introduce mvneta_update_stats routine Lorenzo Bianconi
2019-10-05 20:44 ` [PATCH 2/7] net: mvneta: introduce page pool API for sw buffer manager Lorenzo Bianconi
@ 2019-10-05 20:44 ` Lorenzo Bianconi
2019-10-07 11:05 ` Ilias Apalodimas
2019-10-05 20:44 ` [PATCH 4/7] net: mvneta: add basic XDP support Lorenzo Bianconi
` (4 subsequent siblings)
7 siblings, 1 reply; 12+ messages in thread
From: Lorenzo Bianconi @ 2019-10-05 20:44 UTC (permalink / raw)
To: netdev
Cc: davem, thomas.petazzoni, ilias.apalodimas, brouer,
lorenzo.bianconi, matteo.croce
Refactor mvneta_rx_swbm code introducing mvneta_swbm_rx_frame and
mvneta_swbm_add_rx_fragment routines. Rely on build_skb in order to
allocate skb since the previous patch introduced buffer recycling using
the page_pool API
Tested-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Signed-off-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
drivers/net/ethernet/marvell/mvneta.c | 198 ++++++++++++++------------
1 file changed, 104 insertions(+), 94 deletions(-)
diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
index 8beae0e1eda7..d775fcae9353 100644
--- a/drivers/net/ethernet/marvell/mvneta.c
+++ b/drivers/net/ethernet/marvell/mvneta.c
@@ -323,6 +323,11 @@
ETH_HLEN + ETH_FCS_LEN, \
cache_line_size())
+#define MVNETA_SKB_PAD (SKB_DATA_ALIGN(sizeof(struct skb_shared_info) + \
+ NET_SKB_PAD))
+#define MVNETA_SKB_SIZE(len) (SKB_DATA_ALIGN(len) + MVNETA_SKB_PAD)
+#define MVNETA_MAX_RX_BUF_SIZE (PAGE_SIZE - MVNETA_SKB_PAD)
+
#define IS_TSO_HEADER(txq, addr) \
((addr >= txq->tso_hdrs_phys) && \
(addr < txq->tso_hdrs_phys + txq->size * TSO_HEADER_SIZE))
@@ -646,7 +651,6 @@ static int txq_number = 8;
static int rxq_def;
static int rx_copybreak __read_mostly = 256;
-static int rx_header_size __read_mostly = 128;
/* HW BM need that each port be identify by a unique ID */
static int global_port_id;
@@ -1941,30 +1945,102 @@ int mvneta_rx_refill_queue(struct mvneta_port *pp, struct mvneta_rx_queue *rxq)
return i;
}
+static int
+mvneta_swbm_rx_frame(struct mvneta_port *pp,
+ struct mvneta_rx_desc *rx_desc,
+ struct mvneta_rx_queue *rxq,
+ struct page *page)
+{
+ unsigned char *data = page_address(page);
+ int data_len = -MVNETA_MH_SIZE, len;
+ struct net_device *dev = pp->dev;
+ enum dma_data_direction dma_dir;
+
+ if (MVNETA_SKB_SIZE(rx_desc->data_size) > PAGE_SIZE) {
+ len = MVNETA_MAX_RX_BUF_SIZE;
+ data_len += len;
+ } else {
+ len = rx_desc->data_size;
+ data_len += len - ETH_FCS_LEN;
+ }
+
+ dma_dir = page_pool_get_dma_dir(rxq->page_pool);
+ dma_sync_single_range_for_cpu(dev->dev.parent,
+ rx_desc->buf_phys_addr, 0,
+ len, dma_dir);
+
+ rxq->skb = build_skb(data, PAGE_SIZE);
+ if (unlikely(!rxq->skb)) {
+ netdev_err(dev,
+ "Can't allocate skb on queue %d\n",
+ rxq->id);
+ dev->stats.rx_dropped++;
+ rxq->skb_alloc_err++;
+ return -ENOMEM;
+ }
+ page_pool_release_page(rxq->page_pool, page);
+
+ skb_reserve(rxq->skb, MVNETA_MH_SIZE + NET_SKB_PAD);
+ skb_put(rxq->skb, data_len);
+ mvneta_rx_csum(pp, rx_desc->status, rxq->skb);
+
+ rxq->left_size = rx_desc->data_size - len;
+ rx_desc->buf_phys_addr = 0;
+
+ return 0;
+}
+
+static void
+mvneta_swbm_add_rx_fragment(struct mvneta_port *pp,
+ struct mvneta_rx_desc *rx_desc,
+ struct mvneta_rx_queue *rxq,
+ struct page *page)
+{
+ struct net_device *dev = pp->dev;
+ enum dma_data_direction dma_dir;
+ int data_len, len;
+
+ if (rxq->left_size > MVNETA_MAX_RX_BUF_SIZE) {
+ len = MVNETA_MAX_RX_BUF_SIZE;
+ data_len = len;
+ } else {
+ len = rxq->left_size;
+ data_len = len - ETH_FCS_LEN;
+ }
+ dma_dir = page_pool_get_dma_dir(rxq->page_pool);
+ dma_sync_single_range_for_cpu(dev->dev.parent,
+ rx_desc->buf_phys_addr, 0,
+ len, dma_dir);
+ if (data_len > 0) {
+ /* refill descriptor with new buffer later */
+ skb_add_rx_frag(rxq->skb,
+ skb_shinfo(rxq->skb)->nr_frags,
+ page, NET_SKB_PAD, data_len,
+ PAGE_SIZE);
+
+ page_pool_release_page(rxq->page_pool, page);
+ rx_desc->buf_phys_addr = 0;
+ }
+ rxq->left_size -= len;
+}
+
/* Main rx processing when using software buffer management */
static int mvneta_rx_swbm(struct napi_struct *napi,
struct mvneta_port *pp, int budget,
struct mvneta_rx_queue *rxq)
{
- struct net_device *dev = pp->dev;
- int rx_todo, rx_proc;
- int refill = 0;
- u32 rcvd_pkts = 0;
- u32 rcvd_bytes = 0;
+ int rcvd_pkts = 0, rcvd_bytes = 0;
+ int rx_pending, refill, done = 0;
/* Get number of received packets */
- rx_todo = mvneta_rxq_busy_desc_num_get(pp, rxq);
- rx_proc = 0;
+ rx_pending = mvneta_rxq_busy_desc_num_get(pp, rxq);
/* Fairness NAPI loop */
- while ((rcvd_pkts < budget) && (rx_proc < rx_todo)) {
+ while (done < budget && done < rx_pending) {
struct mvneta_rx_desc *rx_desc = mvneta_rxq_next_desc_get(rxq);
unsigned char *data;
struct page *page;
- dma_addr_t phys_addr;
- u32 rx_status, index;
- int rx_bytes, skb_size, copy_size;
- int frag_num, frag_size, frag_offset;
+ int index;
index = rx_desc - rxq->descs;
page = (struct page *)rxq->buf_virt_addr[index];
@@ -1972,98 +2048,33 @@ static int mvneta_rx_swbm(struct napi_struct *napi,
/* Prefetch header */
prefetch(data);
- phys_addr = rx_desc->buf_phys_addr;
- rx_status = rx_desc->status;
- rx_proc++;
rxq->refill_num++;
+ done++;
+
+ if (rx_desc->status & MVNETA_RXD_FIRST_DESC) {
+ int err;
- if (rx_status & MVNETA_RXD_FIRST_DESC) {
/* Check errors only for FIRST descriptor */
- if (rx_status & MVNETA_RXD_ERR_SUMMARY) {
+ if (rx_desc->status & MVNETA_RXD_ERR_SUMMARY) {
mvneta_rx_error(pp, rx_desc);
- dev->stats.rx_errors++;
+ pp->dev->stats.rx_errors++;
/* leave the descriptor untouched */
continue;
}
- rx_bytes = rx_desc->data_size -
- (ETH_FCS_LEN + MVNETA_MH_SIZE);
- /* Allocate small skb for each new packet */
- skb_size = max(rx_copybreak, rx_header_size);
- rxq->skb = netdev_alloc_skb_ip_align(dev, skb_size);
- if (unlikely(!rxq->skb)) {
- netdev_err(dev,
- "Can't allocate skb on queue %d\n",
- rxq->id);
- dev->stats.rx_dropped++;
- rxq->skb_alloc_err++;
+ err = mvneta_swbm_rx_frame(pp, rx_desc, rxq, page);
+ if (err)
continue;
- }
- copy_size = min(skb_size, rx_bytes);
-
- /* Copy data from buffer to SKB, skip Marvell header */
- memcpy(rxq->skb->data, data + MVNETA_MH_SIZE,
- copy_size);
- skb_put(rxq->skb, copy_size);
- rxq->left_size = rx_bytes - copy_size;
-
- mvneta_rx_csum(pp, rx_status, rxq->skb);
- if (rxq->left_size == 0) {
- int size = copy_size + MVNETA_MH_SIZE;
-
- dma_sync_single_range_for_cpu(dev->dev.parent,
- phys_addr, 0,
- size,
- DMA_FROM_DEVICE);
-
- /* leave the descriptor and buffer untouched */
- } else {
- /* refill descriptor with new buffer later */
- rx_desc->buf_phys_addr = 0;
-
- frag_num = 0;
- frag_offset = copy_size + MVNETA_MH_SIZE;
- frag_size = min(rxq->left_size,
- (int)(PAGE_SIZE - frag_offset));
- skb_add_rx_frag(rxq->skb, frag_num, page,
- frag_offset, frag_size,
- PAGE_SIZE);
- page_pool_release_page(rxq->page_pool, page);
- rxq->left_size -= frag_size;
- }
} else {
- /* Middle or Last descriptor */
if (unlikely(!rxq->skb)) {
pr_debug("no skb for rx_status 0x%x\n",
- rx_status);
+ rx_desc->status);
continue;
}
- if (!rxq->left_size) {
- /* last descriptor has only FCS */
- /* and can be discarded */
- dma_sync_single_range_for_cpu(dev->dev.parent,
- phys_addr, 0,
- ETH_FCS_LEN,
- DMA_FROM_DEVICE);
- /* leave the descriptor and buffer untouched */
- } else {
- /* refill descriptor with new buffer later */
- rx_desc->buf_phys_addr = 0;
-
- frag_num = skb_shinfo(rxq->skb)->nr_frags;
- frag_offset = 0;
- frag_size = min(rxq->left_size,
- (int)(PAGE_SIZE - frag_offset));
- skb_add_rx_frag(rxq->skb, frag_num, page,
- frag_offset, frag_size,
- PAGE_SIZE);
-
- page_pool_release_page(rxq->page_pool, page);
- rxq->left_size -= frag_size;
- }
+ mvneta_swbm_add_rx_fragment(pp, rx_desc, rxq, page);
} /* Middle or Last descriptor */
- if (!(rx_status & MVNETA_RXD_LAST_DESC))
+ if (!(rx_desc->status & MVNETA_RXD_LAST_DESC))
/* no last descriptor this time */
continue;
@@ -2079,13 +2090,12 @@ static int mvneta_rx_swbm(struct napi_struct *napi,
rcvd_bytes += rxq->skb->len;
/* Linux processing */
- rxq->skb->protocol = eth_type_trans(rxq->skb, dev);
+ rxq->skb->protocol = eth_type_trans(rxq->skb, pp->dev);
napi_gro_receive(napi, rxq->skb);
/* clean uncomplete skb pointer in queue */
rxq->skb = NULL;
- rxq->left_size = 0;
}
mvneta_update_stats(pp, rcvd_pkts, rcvd_bytes, false);
@@ -2094,7 +2104,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi,
refill = mvneta_rx_refill_queue(pp, rxq);
/* Update rxq management counters */
- mvneta_rxq_desc_num_update(pp, rxq, rx_proc, refill);
+ mvneta_rxq_desc_num_update(pp, rxq, done, refill);
return rcvd_pkts;
}
@@ -2945,7 +2955,7 @@ static void mvneta_rxq_hw_init(struct mvneta_port *pp,
/* Set Offset */
mvneta_rxq_offset_set(pp, rxq, 0);
mvneta_rxq_buf_size_set(pp, rxq, PAGE_SIZE < SZ_64K ?
- PAGE_SIZE :
+ MVNETA_MAX_RX_BUF_SIZE :
MVNETA_RX_BUF_SIZE(pp->pkt_size));
mvneta_rxq_bm_disable(pp, rxq);
mvneta_rxq_fill(pp, rxq, rxq->size);
@@ -4655,7 +4665,7 @@ static int mvneta_probe(struct platform_device *pdev)
SET_NETDEV_DEV(dev, &pdev->dev);
pp->id = global_port_id++;
- pp->rx_offset_correction = 0; /* not relevant for SW BM */
+ pp->rx_offset_correction = NET_SKB_PAD;
/* Obtain access to BM resources if enabled and already initialized */
bm_node = of_parse_phandle(dn, "buffer-manager", 0);
--
2.21.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 4/7] net: mvneta: add basic XDP support
2019-10-05 20:44 [PATCH 0/7] add XDP support to mvneta driver Lorenzo Bianconi
` (2 preceding siblings ...)
2019-10-05 20:44 ` [PATCH 3/7] net: mvneta: rely on build_skb in mvneta_rx_swbm poll routine Lorenzo Bianconi
@ 2019-10-05 20:44 ` Lorenzo Bianconi
2019-10-05 20:44 ` [PATCH 5/7] net: mvneta: move header prefetch in mvneta_swbm_rx_frame Lorenzo Bianconi
` (3 subsequent siblings)
7 siblings, 0 replies; 12+ messages in thread
From: Lorenzo Bianconi @ 2019-10-05 20:44 UTC (permalink / raw)
To: netdev
Cc: davem, thomas.petazzoni, ilias.apalodimas, brouer,
lorenzo.bianconi, matteo.croce
Add basic XDP support to mvneta driver for devices that rely on software
buffer management. Currently supported verdicts are:
- XDP_DROP
- XDP_PASS
- XDP_REDIRECT
- XDP_ABORTED
Tested-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
drivers/net/ethernet/marvell/mvneta.c | 146 ++++++++++++++++++++++++--
1 file changed, 137 insertions(+), 9 deletions(-)
diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
index d775fcae9353..7cbcf1b54da6 100644
--- a/drivers/net/ethernet/marvell/mvneta.c
+++ b/drivers/net/ethernet/marvell/mvneta.c
@@ -38,6 +38,7 @@
#include <net/ipv6.h>
#include <net/tso.h>
#include <net/page_pool.h>
+#include <linux/bpf_trace.h>
/* Registers */
#define MVNETA_RXQ_CONFIG_REG(q) (0x1400 + ((q) << 2))
@@ -323,8 +324,10 @@
ETH_HLEN + ETH_FCS_LEN, \
cache_line_size())
+#define MVNETA_SKB_HEADROOM (max(XDP_PACKET_HEADROOM, NET_SKB_PAD) + \
+ NET_IP_ALIGN)
#define MVNETA_SKB_PAD (SKB_DATA_ALIGN(sizeof(struct skb_shared_info) + \
- NET_SKB_PAD))
+ MVNETA_SKB_HEADROOM))
#define MVNETA_SKB_SIZE(len) (SKB_DATA_ALIGN(len) + MVNETA_SKB_PAD)
#define MVNETA_MAX_RX_BUF_SIZE (PAGE_SIZE - MVNETA_SKB_PAD)
@@ -352,6 +355,11 @@ struct mvneta_statistic {
#define T_REG_64 64
#define T_SW 1
+#define MVNETA_XDP_PASS BIT(0)
+#define MVNETA_XDP_CONSUMED BIT(1)
+#define MVNETA_XDP_TX BIT(2)
+#define MVNETA_XDP_REDIR BIT(3)
+
static const struct mvneta_statistic mvneta_statistics[] = {
{ 0x3000, T_REG_64, "good_octets_received", },
{ 0x3010, T_REG_32, "good_frames_received", },
@@ -431,6 +439,8 @@ struct mvneta_port {
u32 cause_rx_tx;
struct napi_struct napi;
+ struct bpf_prog *xdp_prog;
+
/* Core clock */
struct clk *clk;
/* AXI clock */
@@ -1945,16 +1955,60 @@ int mvneta_rx_refill_queue(struct mvneta_port *pp, struct mvneta_rx_queue *rxq)
return i;
}
+static int
+mvneta_run_xdp(struct mvneta_port *pp, struct bpf_prog *prog,
+ struct xdp_buff *xdp)
+{
+ u32 ret, act = bpf_prog_run_xdp(prog, xdp);
+
+ switch (act) {
+ case XDP_PASS:
+ ret = MVNETA_XDP_PASS;
+ break;
+ case XDP_REDIRECT: {
+ int err;
+
+ err = xdp_do_redirect(pp->dev, xdp, prog);
+ if (err) {
+ ret = MVNETA_XDP_CONSUMED;
+ xdp_return_buff(xdp);
+ } else {
+ ret = MVNETA_XDP_REDIR;
+ }
+ break;
+ }
+ default:
+ bpf_warn_invalid_xdp_action(act);
+ /* fall through */
+ case XDP_ABORTED:
+ trace_xdp_exception(pp->dev, prog, act);
+ /* fall through */
+ case XDP_DROP:
+ ret = MVNETA_XDP_CONSUMED;
+ xdp_return_buff(xdp);
+ break;
+ }
+
+ return ret;
+}
+
static int
mvneta_swbm_rx_frame(struct mvneta_port *pp,
struct mvneta_rx_desc *rx_desc,
struct mvneta_rx_queue *rxq,
- struct page *page)
+ struct bpf_prog *xdp_prog,
+ struct page *page, u32 *xdp_ret)
{
unsigned char *data = page_address(page);
int data_len = -MVNETA_MH_SIZE, len;
struct net_device *dev = pp->dev;
enum dma_data_direction dma_dir;
+ struct xdp_buff xdp = {
+ .data_hard_start = data,
+ .data = data + MVNETA_SKB_HEADROOM + MVNETA_MH_SIZE,
+ .rxq = &rxq->xdp_rxq,
+ };
+ xdp_set_data_meta_invalid(&xdp);
if (MVNETA_SKB_SIZE(rx_desc->data_size) > PAGE_SIZE) {
len = MVNETA_MAX_RX_BUF_SIZE;
@@ -1963,13 +2017,29 @@ mvneta_swbm_rx_frame(struct mvneta_port *pp,
len = rx_desc->data_size;
data_len += len - ETH_FCS_LEN;
}
+ xdp.data_end = xdp.data + data_len;
dma_dir = page_pool_get_dma_dir(rxq->page_pool);
dma_sync_single_range_for_cpu(dev->dev.parent,
rx_desc->buf_phys_addr, 0,
len, dma_dir);
- rxq->skb = build_skb(data, PAGE_SIZE);
+ if (xdp_prog) {
+ u32 ret;
+
+ ret = mvneta_run_xdp(pp, xdp_prog, &xdp);
+ if (ret == MVNETA_XDP_REDIR ||
+ ret == MVNETA_XDP_TX)
+ mvneta_update_stats(pp, 1, xdp.data_end - xdp.data,
+ false);
+ if (ret != MVNETA_XDP_PASS) {
+ rx_desc->buf_phys_addr = 0;
+ *xdp_ret |= ret;
+ return ret;
+ }
+ }
+
+ rxq->skb = build_skb(xdp.data_hard_start, PAGE_SIZE);
if (unlikely(!rxq->skb)) {
netdev_err(dev,
"Can't allocate skb on queue %d\n",
@@ -1980,8 +2050,9 @@ mvneta_swbm_rx_frame(struct mvneta_port *pp,
}
page_pool_release_page(rxq->page_pool, page);
- skb_reserve(rxq->skb, MVNETA_MH_SIZE + NET_SKB_PAD);
- skb_put(rxq->skb, data_len);
+ skb_reserve(rxq->skb,
+ xdp.data - xdp.data_hard_start);
+ skb_put(rxq->skb, xdp.data_end - xdp.data);
mvneta_rx_csum(pp, rx_desc->status, rxq->skb);
rxq->left_size = rx_desc->data_size - len;
@@ -2015,7 +2086,7 @@ mvneta_swbm_add_rx_fragment(struct mvneta_port *pp,
/* refill descriptor with new buffer later */
skb_add_rx_frag(rxq->skb,
skb_shinfo(rxq->skb)->nr_frags,
- page, NET_SKB_PAD, data_len,
+ page, MVNETA_SKB_HEADROOM, data_len,
PAGE_SIZE);
page_pool_release_page(rxq->page_pool, page);
@@ -2031,10 +2102,15 @@ static int mvneta_rx_swbm(struct napi_struct *napi,
{
int rcvd_pkts = 0, rcvd_bytes = 0;
int rx_pending, refill, done = 0;
+ struct bpf_prog *xdp_prog;
+ u32 xdp_ret = 0;
/* Get number of received packets */
rx_pending = mvneta_rxq_busy_desc_num_get(pp, rxq);
+ rcu_read_lock();
+ xdp_prog = READ_ONCE(pp->xdp_prog);
+
/* Fairness NAPI loop */
while (done < budget && done < rx_pending) {
struct mvneta_rx_desc *rx_desc = mvneta_rxq_next_desc_get(rxq);
@@ -2062,7 +2138,8 @@ static int mvneta_rx_swbm(struct napi_struct *napi,
continue;
}
- err = mvneta_swbm_rx_frame(pp, rx_desc, rxq, page);
+ err = mvneta_swbm_rx_frame(pp, rx_desc, rxq,
+ xdp_prog, page, &xdp_ret);
if (err)
continue;
} else {
@@ -2097,6 +2174,10 @@ static int mvneta_rx_swbm(struct napi_struct *napi,
/* clean uncomplete skb pointer in queue */
rxq->skb = NULL;
}
+ rcu_read_unlock();
+
+ if (xdp_ret & MVNETA_XDP_REDIR)
+ xdp_do_flush_map();
mvneta_update_stats(pp, rcvd_pkts, rcvd_bytes, false);
@@ -2838,13 +2919,14 @@ static int mvneta_poll(struct napi_struct *napi, int budget)
static int mvneta_create_page_pool(struct mvneta_port *pp,
struct mvneta_rx_queue *rxq, int size)
{
+ struct bpf_prog *xdp_prog = READ_ONCE(pp->xdp_prog);
struct page_pool_params pp_params = {
.order = 0,
.flags = PP_FLAG_DMA_MAP,
.pool_size = size,
.nid = cpu_to_node(0),
.dev = pp->dev->dev.parent,
- .dma_dir = DMA_FROM_DEVICE,
+ .dma_dir = xdp_prog ? DMA_BIDIRECTIONAL : DMA_FROM_DEVICE,
};
int err;
@@ -3310,6 +3392,11 @@ static int mvneta_change_mtu(struct net_device *dev, int mtu)
mtu = ALIGN(MVNETA_RX_PKT_SIZE(mtu), 8);
}
+ if (pp->xdp_prog && mtu > MVNETA_MAX_RX_BUF_SIZE) {
+ netdev_info(dev, "Illegal MTU value %d for XDP mode\n", mtu);
+ return -EINVAL;
+ }
+
dev->mtu = mtu;
if (!netif_running(dev)) {
@@ -3979,6 +4066,46 @@ static int mvneta_ioctl(struct net_device *dev, struct ifreq *ifr, int cmd)
return phylink_mii_ioctl(pp->phylink, ifr, cmd);
}
+static int mvneta_xdp_setup(struct net_device *dev, struct bpf_prog *prog,
+ struct netlink_ext_ack *extack)
+{
+ struct mvneta_port *pp = netdev_priv(dev);
+ struct bpf_prog *old_prog;
+
+ if (prog && dev->mtu > MVNETA_MAX_RX_BUF_SIZE) {
+ NL_SET_ERR_MSG_MOD(extack, "Jumbo frames not supported on XDP");
+ return -EOPNOTSUPP;
+ }
+
+ if (netif_running(dev))
+ mvneta_stop(dev);
+
+ old_prog = xchg(&pp->xdp_prog, prog);
+ if (old_prog)
+ bpf_prog_put(old_prog);
+
+ if (netif_running(dev))
+ mvneta_open(dev);
+
+ return 0;
+}
+
+static int mvneta_xdp(struct net_device *dev, struct netdev_bpf *xdp)
+{
+ struct mvneta_port *pp = netdev_priv(dev);
+
+ switch (xdp->command) {
+ case XDP_SETUP_PROG:
+ return mvneta_xdp_setup(dev, xdp->prog, xdp->extack);
+ case XDP_QUERY_PROG:
+ xdp->prog_id = pp->xdp_prog ? pp->xdp_prog->aux->id : 0;
+ return 0;
+ default:
+ NL_SET_ERR_MSG_MOD(xdp->extack, "unknown XDP command");
+ return -EINVAL;
+ }
+}
+
/* Ethtool methods */
/* Set link ksettings (phy address, speed) for ethtools */
@@ -4375,6 +4502,7 @@ static const struct net_device_ops mvneta_netdev_ops = {
.ndo_fix_features = mvneta_fix_features,
.ndo_get_stats64 = mvneta_get_stats64,
.ndo_do_ioctl = mvneta_ioctl,
+ .ndo_bpf = mvneta_xdp,
};
static const struct ethtool_ops mvneta_eth_tool_ops = {
@@ -4665,7 +4793,7 @@ static int mvneta_probe(struct platform_device *pdev)
SET_NETDEV_DEV(dev, &pdev->dev);
pp->id = global_port_id++;
- pp->rx_offset_correction = NET_SKB_PAD;
+ pp->rx_offset_correction = MVNETA_SKB_HEADROOM;
/* Obtain access to BM resources if enabled and already initialized */
bm_node = of_parse_phandle(dn, "buffer-manager", 0);
--
2.21.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 5/7] net: mvneta: move header prefetch in mvneta_swbm_rx_frame
2019-10-05 20:44 [PATCH 0/7] add XDP support to mvneta driver Lorenzo Bianconi
` (3 preceding siblings ...)
2019-10-05 20:44 ` [PATCH 4/7] net: mvneta: add basic XDP support Lorenzo Bianconi
@ 2019-10-05 20:44 ` Lorenzo Bianconi
2019-10-05 20:44 ` [PATCH 6/7] net: mvneta: make tx buffer array agnostic Lorenzo Bianconi
` (2 subsequent siblings)
7 siblings, 0 replies; 12+ messages in thread
From: Lorenzo Bianconi @ 2019-10-05 20:44 UTC (permalink / raw)
To: netdev
Cc: davem, thomas.petazzoni, ilias.apalodimas, brouer,
lorenzo.bianconi, matteo.croce
Move data buffer prefetch in mvneta_swbm_rx_frame after
dma_sync_single_range_for_cpu
Tested-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Signed-off-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
drivers/net/ethernet/marvell/mvneta.c | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
index 7cbcf1b54da6..45dcae52f34b 100644
--- a/drivers/net/ethernet/marvell/mvneta.c
+++ b/drivers/net/ethernet/marvell/mvneta.c
@@ -2004,11 +2004,8 @@ mvneta_swbm_rx_frame(struct mvneta_port *pp,
struct net_device *dev = pp->dev;
enum dma_data_direction dma_dir;
struct xdp_buff xdp = {
- .data_hard_start = data,
- .data = data + MVNETA_SKB_HEADROOM + MVNETA_MH_SIZE,
.rxq = &rxq->xdp_rxq,
};
- xdp_set_data_meta_invalid(&xdp);
if (MVNETA_SKB_SIZE(rx_desc->data_size) > PAGE_SIZE) {
len = MVNETA_MAX_RX_BUF_SIZE;
@@ -2017,13 +2014,20 @@ mvneta_swbm_rx_frame(struct mvneta_port *pp,
len = rx_desc->data_size;
data_len += len - ETH_FCS_LEN;
}
- xdp.data_end = xdp.data + data_len;
dma_dir = page_pool_get_dma_dir(rxq->page_pool);
dma_sync_single_range_for_cpu(dev->dev.parent,
rx_desc->buf_phys_addr, 0,
len, dma_dir);
+ /* Prefetch header */
+ prefetch(data);
+
+ xdp.data_hard_start = data;
+ xdp.data = data + MVNETA_SKB_HEADROOM + MVNETA_MH_SIZE;
+ xdp.data_end = xdp.data + data_len;
+ xdp_set_data_meta_invalid(&xdp);
+
if (xdp_prog) {
u32 ret;
@@ -2114,15 +2118,11 @@ static int mvneta_rx_swbm(struct napi_struct *napi,
/* Fairness NAPI loop */
while (done < budget && done < rx_pending) {
struct mvneta_rx_desc *rx_desc = mvneta_rxq_next_desc_get(rxq);
- unsigned char *data;
struct page *page;
int index;
index = rx_desc - rxq->descs;
page = (struct page *)rxq->buf_virt_addr[index];
- data = page_address(page);
- /* Prefetch header */
- prefetch(data);
rxq->refill_num++;
done++;
--
2.21.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 6/7] net: mvneta: make tx buffer array agnostic
2019-10-05 20:44 [PATCH 0/7] add XDP support to mvneta driver Lorenzo Bianconi
` (4 preceding siblings ...)
2019-10-05 20:44 ` [PATCH 5/7] net: mvneta: move header prefetch in mvneta_swbm_rx_frame Lorenzo Bianconi
@ 2019-10-05 20:44 ` Lorenzo Bianconi
2019-10-05 20:44 ` [PATCH 7/7] net: mvneta: add XDP_TX support Lorenzo Bianconi
2019-10-05 21:02 ` [PATCH 0/7] add XDP support to mvneta driver Lorenzo Bianconi
7 siblings, 0 replies; 12+ messages in thread
From: Lorenzo Bianconi @ 2019-10-05 20:44 UTC (permalink / raw)
To: netdev
Cc: davem, thomas.petazzoni, ilias.apalodimas, brouer,
lorenzo.bianconi, matteo.croce
Allow tx buffer array to contain both skb and xdp buffers in order to
enable xdp frame recycling adding XDP_TX verdict support
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
drivers/net/ethernet/marvell/mvneta.c | 66 +++++++++++++++++----------
1 file changed, 43 insertions(+), 23 deletions(-)
diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
index 45dcae52f34b..28b35e55aa35 100644
--- a/drivers/net/ethernet/marvell/mvneta.c
+++ b/drivers/net/ethernet/marvell/mvneta.c
@@ -561,6 +561,20 @@ struct mvneta_rx_desc {
};
#endif
+enum mvneta_tx_buf_type {
+ MVNETA_TYPE_SKB,
+ MVNETA_TYPE_XDP_TX,
+ MVNETA_TYPE_XDP_NDO,
+};
+
+struct mvneta_tx_buf {
+ enum mvneta_tx_buf_type type;
+ union {
+ struct xdp_frame *xdpf;
+ struct sk_buff *skb;
+ };
+};
+
struct mvneta_tx_queue {
/* Number of this TX queue, in the range 0-7 */
u8 id;
@@ -576,8 +590,8 @@ struct mvneta_tx_queue {
int tx_stop_threshold;
int tx_wake_threshold;
- /* Array of transmitted skb */
- struct sk_buff **tx_skb;
+ /* Array of transmitted buffers */
+ struct mvneta_tx_buf *buf;
/* Index of last TX DMA descriptor that was inserted */
int txq_put_index;
@@ -1780,14 +1794,9 @@ static void mvneta_txq_bufs_free(struct mvneta_port *pp,
int i;
for (i = 0; i < num; i++) {
+ struct mvneta_tx_buf *buf = &txq->buf[txq->txq_get_index];
struct mvneta_tx_desc *tx_desc = txq->descs +
txq->txq_get_index;
- struct sk_buff *skb = txq->tx_skb[txq->txq_get_index];
-
- if (skb) {
- bytes_compl += skb->len;
- pkts_compl++;
- }
mvneta_txq_inc_get(txq);
@@ -1795,9 +1804,12 @@ static void mvneta_txq_bufs_free(struct mvneta_port *pp,
dma_unmap_single(pp->dev->dev.parent,
tx_desc->buf_phys_addr,
tx_desc->data_size, DMA_TO_DEVICE);
- if (!skb)
+ if (!buf->skb)
continue;
- dev_kfree_skb_any(skb);
+
+ bytes_compl += buf->skb->len;
+ pkts_compl++;
+ dev_kfree_skb_any(buf->skb);
}
netdev_tx_completed_queue(nq, pkts_compl, bytes_compl);
@@ -2315,16 +2327,19 @@ static inline void
mvneta_tso_put_hdr(struct sk_buff *skb,
struct mvneta_port *pp, struct mvneta_tx_queue *txq)
{
- struct mvneta_tx_desc *tx_desc;
int hdr_len = skb_transport_offset(skb) + tcp_hdrlen(skb);
+ struct mvneta_tx_buf *buf = &txq->buf[txq->txq_put_index];
+ struct mvneta_tx_desc *tx_desc;
- txq->tx_skb[txq->txq_put_index] = NULL;
tx_desc = mvneta_txq_next_desc_get(txq);
tx_desc->data_size = hdr_len;
tx_desc->command = mvneta_skb_tx_csum(pp, skb);
tx_desc->command |= MVNETA_TXD_F_DESC;
tx_desc->buf_phys_addr = txq->tso_hdrs_phys +
txq->txq_put_index * TSO_HEADER_SIZE;
+ buf->type = MVNETA_TYPE_SKB;
+ buf->skb = NULL;
+
mvneta_txq_inc_put(txq);
}
@@ -2333,6 +2348,7 @@ mvneta_tso_put_data(struct net_device *dev, struct mvneta_tx_queue *txq,
struct sk_buff *skb, char *data, int size,
bool last_tcp, bool is_last)
{
+ struct mvneta_tx_buf *buf = &txq->buf[txq->txq_put_index];
struct mvneta_tx_desc *tx_desc;
tx_desc = mvneta_txq_next_desc_get(txq);
@@ -2346,7 +2362,8 @@ mvneta_tso_put_data(struct net_device *dev, struct mvneta_tx_queue *txq,
}
tx_desc->command = 0;
- txq->tx_skb[txq->txq_put_index] = NULL;
+ buf->type = MVNETA_TYPE_SKB;
+ buf->skb = NULL;
if (last_tcp) {
/* last descriptor in the TCP packet */
@@ -2354,7 +2371,7 @@ mvneta_tso_put_data(struct net_device *dev, struct mvneta_tx_queue *txq,
/* last descriptor in SKB */
if (is_last)
- txq->tx_skb[txq->txq_put_index] = skb;
+ buf->skb = skb;
}
mvneta_txq_inc_put(txq);
return 0;
@@ -2439,6 +2456,7 @@ static int mvneta_tx_frag_process(struct mvneta_port *pp, struct sk_buff *skb,
int i, nr_frags = skb_shinfo(skb)->nr_frags;
for (i = 0; i < nr_frags; i++) {
+ struct mvneta_tx_buf *buf = &txq->buf[txq->txq_put_index];
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
void *addr = skb_frag_address(frag);
@@ -2458,12 +2476,13 @@ static int mvneta_tx_frag_process(struct mvneta_port *pp, struct sk_buff *skb,
if (i == nr_frags - 1) {
/* Last descriptor */
tx_desc->command = MVNETA_TXD_L_DESC | MVNETA_TXD_Z_PAD;
- txq->tx_skb[txq->txq_put_index] = skb;
+ buf->skb = skb;
} else {
/* Descriptor in the middle: Not First, Not Last */
tx_desc->command = 0;
- txq->tx_skb[txq->txq_put_index] = NULL;
+ buf->skb = NULL;
}
+ buf->type = MVNETA_TYPE_SKB;
mvneta_txq_inc_put(txq);
}
@@ -2491,6 +2510,7 @@ static netdev_tx_t mvneta_tx(struct sk_buff *skb, struct net_device *dev)
struct mvneta_port *pp = netdev_priv(dev);
u16 txq_id = skb_get_queue_mapping(skb);
struct mvneta_tx_queue *txq = &pp->txqs[txq_id];
+ struct mvneta_tx_buf *buf = &txq->buf[txq->txq_put_index];
struct mvneta_tx_desc *tx_desc;
int len = skb->len;
int frags = 0;
@@ -2523,16 +2543,17 @@ static netdev_tx_t mvneta_tx(struct sk_buff *skb, struct net_device *dev)
goto out;
}
+ buf->type = MVNETA_TYPE_SKB;
if (frags == 1) {
/* First and Last descriptor */
tx_cmd |= MVNETA_TXD_FLZ_DESC;
tx_desc->command = tx_cmd;
- txq->tx_skb[txq->txq_put_index] = skb;
+ buf->skb = skb;
mvneta_txq_inc_put(txq);
} else {
/* First but not Last */
tx_cmd |= MVNETA_TXD_F_DESC;
- txq->tx_skb[txq->txq_put_index] = NULL;
+ buf->skb = NULL;
mvneta_txq_inc_put(txq);
tx_desc->command = tx_cmd;
/* Continue with other skb fragments */
@@ -3118,9 +3139,8 @@ static int mvneta_txq_sw_init(struct mvneta_port *pp,
txq->last_desc = txq->size - 1;
- txq->tx_skb = kmalloc_array(txq->size, sizeof(*txq->tx_skb),
- GFP_KERNEL);
- if (!txq->tx_skb) {
+ txq->buf = kmalloc_array(txq->size, sizeof(*txq->buf), GFP_KERNEL);
+ if (!txq->buf) {
dma_free_coherent(pp->dev->dev.parent,
txq->size * MVNETA_DESC_ALIGNED_SIZE,
txq->descs, txq->descs_phys);
@@ -3132,7 +3152,7 @@ static int mvneta_txq_sw_init(struct mvneta_port *pp,
txq->size * TSO_HEADER_SIZE,
&txq->tso_hdrs_phys, GFP_KERNEL);
if (!txq->tso_hdrs) {
- kfree(txq->tx_skb);
+ kfree(txq->buf);
dma_free_coherent(pp->dev->dev.parent,
txq->size * MVNETA_DESC_ALIGNED_SIZE,
txq->descs, txq->descs_phys);
@@ -3185,7 +3205,7 @@ static void mvneta_txq_sw_deinit(struct mvneta_port *pp,
{
struct netdev_queue *nq = netdev_get_tx_queue(pp->dev, txq->id);
- kfree(txq->tx_skb);
+ kfree(txq->buf);
if (txq->tso_hdrs)
dma_free_coherent(pp->dev->dev.parent,
--
2.21.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 7/7] net: mvneta: add XDP_TX support
2019-10-05 20:44 [PATCH 0/7] add XDP support to mvneta driver Lorenzo Bianconi
` (5 preceding siblings ...)
2019-10-05 20:44 ` [PATCH 6/7] net: mvneta: make tx buffer array agnostic Lorenzo Bianconi
@ 2019-10-05 20:44 ` Lorenzo Bianconi
2019-10-05 21:02 ` [PATCH 0/7] add XDP support to mvneta driver Lorenzo Bianconi
7 siblings, 0 replies; 12+ messages in thread
From: Lorenzo Bianconi @ 2019-10-05 20:44 UTC (permalink / raw)
To: netdev
Cc: davem, thomas.petazzoni, ilias.apalodimas, brouer,
lorenzo.bianconi, matteo.croce
Implement XDP_TX verdict and ndo_xdp_xmit net_device_ops function
pointer
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
drivers/net/ethernet/marvell/mvneta.c | 126 ++++++++++++++++++++++++--
1 file changed, 119 insertions(+), 7 deletions(-)
diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
index 28b35e55aa35..9ec97086f991 100644
--- a/drivers/net/ethernet/marvell/mvneta.c
+++ b/drivers/net/ethernet/marvell/mvneta.c
@@ -1800,16 +1800,19 @@ static void mvneta_txq_bufs_free(struct mvneta_port *pp,
mvneta_txq_inc_get(txq);
- if (!IS_TSO_HEADER(txq, tx_desc->buf_phys_addr))
+ if (!IS_TSO_HEADER(txq, tx_desc->buf_phys_addr) &&
+ buf->type != MVNETA_TYPE_XDP_TX)
dma_unmap_single(pp->dev->dev.parent,
tx_desc->buf_phys_addr,
tx_desc->data_size, DMA_TO_DEVICE);
- if (!buf->skb)
- continue;
-
- bytes_compl += buf->skb->len;
- pkts_compl++;
- dev_kfree_skb_any(buf->skb);
+ if (buf->type == MVNETA_TYPE_SKB && buf->skb) {
+ bytes_compl += buf->skb->len;
+ pkts_compl++;
+ dev_kfree_skb_any(buf->skb);
+ } else if (buf->type == MVNETA_TYPE_XDP_TX ||
+ buf->type == MVNETA_TYPE_XDP_NDO) {
+ xdp_return_frame(buf->xdpf);
+ }
}
netdev_tx_completed_queue(nq, pkts_compl, bytes_compl);
@@ -1967,6 +1970,109 @@ int mvneta_rx_refill_queue(struct mvneta_port *pp, struct mvneta_rx_queue *rxq)
return i;
}
+static int
+mvneta_xdp_submit_frame(struct mvneta_port *pp, struct mvneta_tx_queue *txq,
+ struct xdp_frame *xdpf, bool dma_map)
+{
+ struct mvneta_tx_desc *tx_desc;
+ struct mvneta_tx_buf *buf;
+ dma_addr_t dma_addr;
+
+ if (txq->count >= txq->tx_stop_threshold)
+ return MVNETA_XDP_CONSUMED;
+
+ tx_desc = mvneta_txq_next_desc_get(txq);
+
+ buf = &txq->buf[txq->txq_put_index];
+ if (dma_map) {
+ /* ndo_xdp_xmit */
+ dma_addr = dma_map_single(pp->dev->dev.parent, xdpf->data,
+ xdpf->len, DMA_TO_DEVICE);
+ if (dma_mapping_error(pp->dev->dev.parent, dma_addr)) {
+ mvneta_txq_desc_put(txq);
+ return MVNETA_XDP_CONSUMED;
+ }
+ buf->type = MVNETA_TYPE_XDP_NDO;
+ } else {
+ struct page *page = virt_to_page(xdpf->data);
+
+ dma_addr = page_pool_get_dma_addr(page) +
+ pp->rx_offset_correction + MVNETA_MH_SIZE;
+ dma_sync_single_for_device(pp->dev->dev.parent, dma_addr,
+ xdpf->len, DMA_BIDIRECTIONAL);
+ buf->type = MVNETA_TYPE_XDP_TX;
+ }
+ buf->xdpf = xdpf;
+
+ tx_desc->command = MVNETA_TXD_FLZ_DESC;
+ tx_desc->buf_phys_addr = dma_addr;
+ tx_desc->data_size = xdpf->len;
+
+ mvneta_update_stats(pp, 1, xdpf->len, true);
+ mvneta_txq_inc_put(txq);
+ txq->pending++;
+ txq->count++;
+
+ return MVNETA_XDP_TX;
+}
+
+static int
+mvneta_xdp_xmit_back(struct mvneta_port *pp, struct xdp_buff *xdp)
+{
+ struct xdp_frame *xdpf = convert_to_xdp_frame(xdp);
+ int cpu = smp_processor_id();
+ struct mvneta_tx_queue *txq;
+ struct netdev_queue *nq;
+ u32 ret;
+
+ if (unlikely(!xdpf))
+ return MVNETA_XDP_CONSUMED;
+
+ txq = &pp->txqs[cpu % txq_number];
+ nq = netdev_get_tx_queue(pp->dev, txq->id);
+
+ __netif_tx_lock(nq, cpu);
+ ret = mvneta_xdp_submit_frame(pp, txq, xdpf, false);
+ if (ret == MVNETA_XDP_TX)
+ mvneta_txq_pend_desc_add(pp, txq, 0);
+ __netif_tx_unlock(nq);
+
+ return ret;
+}
+
+static int
+mvneta_xdp_xmit(struct net_device *dev, int num_frame,
+ struct xdp_frame **frames, u32 flags)
+{
+ struct mvneta_port *pp = netdev_priv(dev);
+ int cpu = smp_processor_id();
+ struct mvneta_tx_queue *txq;
+ struct netdev_queue *nq;
+ int i, drops = 0;
+ u32 ret;
+
+ if (unlikely(flags & ~XDP_XMIT_FLAGS_MASK))
+ return -EINVAL;
+
+ txq = &pp->txqs[cpu % txq_number];
+ nq = netdev_get_tx_queue(pp->dev, txq->id);
+
+ __netif_tx_lock(nq, cpu);
+ for (i = 0; i < num_frame; i++) {
+ ret = mvneta_xdp_submit_frame(pp, txq, frames[i], true);
+ if (ret != MVNETA_XDP_TX) {
+ xdp_return_frame_rx_napi(frames[i]);
+ drops++;
+ }
+ }
+
+ if (unlikely(flags & XDP_XMIT_FLUSH))
+ mvneta_txq_pend_desc_add(pp, txq, 0);
+ __netif_tx_unlock(nq);
+
+ return num_frame - drops;
+}
+
static int
mvneta_run_xdp(struct mvneta_port *pp, struct bpf_prog *prog,
struct xdp_buff *xdp)
@@ -1989,6 +2095,11 @@ mvneta_run_xdp(struct mvneta_port *pp, struct bpf_prog *prog,
}
break;
}
+ case XDP_TX:
+ ret = mvneta_xdp_xmit_back(pp, xdp);
+ if (ret != MVNETA_XDP_TX)
+ xdp_return_buff(xdp);
+ break;
default:
bpf_warn_invalid_xdp_action(act);
/* fall through */
@@ -4523,6 +4634,7 @@ static const struct net_device_ops mvneta_netdev_ops = {
.ndo_get_stats64 = mvneta_get_stats64,
.ndo_do_ioctl = mvneta_ioctl,
.ndo_bpf = mvneta_xdp,
+ .ndo_xdp_xmit = mvneta_xdp_xmit,
};
static const struct ethtool_ops mvneta_eth_tool_ops = {
--
2.21.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH 0/7] add XDP support to mvneta driver
2019-10-05 20:44 [PATCH 0/7] add XDP support to mvneta driver Lorenzo Bianconi
` (6 preceding siblings ...)
2019-10-05 20:44 ` [PATCH 7/7] net: mvneta: add XDP_TX support Lorenzo Bianconi
@ 2019-10-05 21:02 ` Lorenzo Bianconi
7 siblings, 0 replies; 12+ messages in thread
From: Lorenzo Bianconi @ 2019-10-05 21:02 UTC (permalink / raw)
To: Lorenzo Bianconi
Cc: Network Development, David S. Miller, thomas.petazzoni,
Ilias Apalodimas, Jesper Dangaard Brouer, matteo.croce
>
> Add XDP support to mvneta driver for devices that rely on software
> buffer management. Supported verdicts are:
> - XDP_DROP
> - XDP_PASS
> - XDP_REDIRECT
> - XDP_TX
> Moreover set ndo_xdp_xmit net_device_ops function pointer in order to support
> redirecting from other device (e.g. virtio-net).
> Convert mvneta driver to page_pool API.
> This series is based on previous work done by Jesper and Ilias.
>
> Changes since RFC:
> - implement XDP_TX
> - make tx pending buffer list agnostic
> - code refactoring
> - check if device is running in mvneta_xdp_setup
>
This series is clearly intended for net-next :)
Regards,
Lorenzo
> Lorenzo Bianconi (7):
> net: mvneta: introduce mvneta_update_stats routine
> net: mvneta: introduce page pool API for sw buffer manager
> net: mvneta: rely on build_skb in mvneta_rx_swbm poll routine
> net: mvneta: add basic XDP support
> net: mvneta: move header prefetch in mvneta_swbm_rx_frame
> net: mvneta: make tx buffer array agnostic
> net: mvneta: add XDP_TX support
>
> drivers/net/ethernet/marvell/Kconfig | 1 +
> drivers/net/ethernet/marvell/mvneta.c | 627 +++++++++++++++++++-------
> 2 files changed, 468 insertions(+), 160 deletions(-)
>
> --
> 2.21.0
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 2/7] net: mvneta: introduce page pool API for sw buffer manager
2019-10-05 20:44 ` [PATCH 2/7] net: mvneta: introduce page pool API for sw buffer manager Lorenzo Bianconi
@ 2019-10-05 21:34 ` Ilias Apalodimas
2019-10-07 12:27 ` Lorenzo Bianconi
0 siblings, 1 reply; 12+ messages in thread
From: Ilias Apalodimas @ 2019-10-05 21:34 UTC (permalink / raw)
To: Lorenzo Bianconi
Cc: netdev, davem, thomas.petazzoni, brouer, lorenzo.bianconi, matteo.croce
Hi Lorenzo,
On Sat, Oct 05, 2019 at 10:44:35PM +0200, Lorenzo Bianconi wrote:
> Use the page_pool api for allocations and DMA handling instead of
> __dev_alloc_page()/dma_map_page() and free_page()/dma_unmap_page().
> Pages are unmapped using page_pool_release_page before packets
> go into the network stack.
>
> The page_pool API offers buffer recycling capabilities for XDP but
> allocates one page per packet, unless the driver splits and manages
> the allocated page.
> This is a preliminary patch to add XDP support to mvneta driver
>
> Tested-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
> Signed-off-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
> Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
> ---
> drivers/net/ethernet/marvell/Kconfig | 1 +
> drivers/net/ethernet/marvell/mvneta.c | 76 ++++++++++++++++++++-------
> 2 files changed, 58 insertions(+), 19 deletions(-)
>
> diff --git a/drivers/net/ethernet/marvell/Kconfig b/drivers/net/ethernet/marvell/Kconfig
> index fb942167ee54..3d5caea096fb 100644
> --- a/drivers/net/ethernet/marvell/Kconfig
> +++ b/drivers/net/ethernet/marvell/Kconfig
> @@ -61,6 +61,7 @@ config MVNETA
> depends on ARCH_MVEBU || COMPILE_TEST
> select MVMDIO
> select PHYLINK
> + select PAGE_POOL
> ---help---
> This driver supports the network interface units in the
> Marvell ARMADA XP, ARMADA 370, ARMADA 38x and
> diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
> index 128b9fded959..8beae0e1eda7 100644
> --- a/drivers/net/ethernet/marvell/mvneta.c
> +++ b/drivers/net/ethernet/marvell/mvneta.c
> @@ -37,6 +37,7 @@
> #include <net/ip.h>
> #include <net/ipv6.h>
> #include <net/tso.h>
> +#include <net/page_pool.h>
>
> /* Registers */
> #define MVNETA_RXQ_CONFIG_REG(q) (0x1400 + ((q) << 2))
> @@ -603,6 +604,10 @@ struct mvneta_rx_queue {
> u32 pkts_coal;
> u32 time_coal;
>
> + /* page_pool */
> + struct page_pool *page_pool;
> + struct xdp_rxq_info xdp_rxq;
> +
> /* Virtual address of the RX buffer */
> void **buf_virt_addr;
>
> @@ -1815,19 +1820,12 @@ static int mvneta_rx_refill(struct mvneta_port *pp,
> dma_addr_t phys_addr;
> struct page *page;
>
> - page = __dev_alloc_page(gfp_mask);
> + page = page_pool_alloc_pages(rxq->page_pool,
> + gfp_mask | __GFP_NOWARN);
> if (!page)
> return -ENOMEM;
Is the driver syncing the buffer somewhere else? (for_device)
If not you'll have to do this here.
On a non-cache coherent machine (and i think this one is) you may get dirty
cache lines handed to the device. Those dirty cache lines might get written back
*after* the device has DMA'ed it's data. You need to flush those first to avoid
any data corruption
>
> - /* map page for use */
> - phys_addr = dma_map_page(pp->dev->dev.parent, page, 0, PAGE_SIZE,
> - DMA_FROM_DEVICE);
> - if (unlikely(dma_mapping_error(pp->dev->dev.parent, phys_addr))) {
> - __free_page(page);
> - return -ENOMEM;
> - }
> -
> - phys_addr += pp->rx_offset_correction;
> + phys_addr = page_pool_get_dma_addr(page) + pp->rx_offset_correction;
> mvneta_rx_desc_fill(rx_desc, phys_addr, page, rxq);
> return 0;
> }
> @@ -1894,10 +1892,11 @@ static void mvneta_rxq_drop_pkts(struct mvneta_port *pp,
> if (!data || !(rx_desc->buf_phys_addr))
> continue;
>
> - dma_unmap_page(pp->dev->dev.parent, rx_desc->buf_phys_addr,
> - PAGE_SIZE, DMA_FROM_DEVICE);
> - __free_page(data);
> + page_pool_put_page(rxq->page_pool, data, false);
> }
> + if (xdp_rxq_info_is_reg(&rxq->xdp_rxq))
> + xdp_rxq_info_unreg(&rxq->xdp_rxq);
> + page_pool_destroy(rxq->page_pool);
> }
>
> static void
> @@ -2029,8 +2028,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi,
> skb_add_rx_frag(rxq->skb, frag_num, page,
> frag_offset, frag_size,
> PAGE_SIZE);
> - dma_unmap_page(dev->dev.parent, phys_addr,
> - PAGE_SIZE, DMA_FROM_DEVICE);
> + page_pool_release_page(rxq->page_pool, page);
> rxq->left_size -= frag_size;
> }
> } else {
> @@ -2060,9 +2058,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi,
> frag_offset, frag_size,
> PAGE_SIZE);
>
> - dma_unmap_page(dev->dev.parent, phys_addr,
> - PAGE_SIZE, DMA_FROM_DEVICE);
> -
> + page_pool_release_page(rxq->page_pool, page);
> rxq->left_size -= frag_size;
> }
> } /* Middle or Last descriptor */
> @@ -2829,11 +2825,53 @@ static int mvneta_poll(struct napi_struct *napi, int budget)
> return rx_done;
> }
>
> +static int mvneta_create_page_pool(struct mvneta_port *pp,
> + struct mvneta_rx_queue *rxq, int size)
> +{
> + struct page_pool_params pp_params = {
> + .order = 0,
> + .flags = PP_FLAG_DMA_MAP,
> + .pool_size = size,
> + .nid = cpu_to_node(0),
> + .dev = pp->dev->dev.parent,
> + .dma_dir = DMA_FROM_DEVICE,
> + };
> + int err;
> +
> + rxq->page_pool = page_pool_create(&pp_params);
> + if (IS_ERR(rxq->page_pool)) {
> + err = PTR_ERR(rxq->page_pool);
> + rxq->page_pool = NULL;
> + return err;
> + }
> +
> + err = xdp_rxq_info_reg(&rxq->xdp_rxq, pp->dev, 0);
> + if (err < 0)
> + goto err_free_pp;
> +
> + err = xdp_rxq_info_reg_mem_model(&rxq->xdp_rxq, MEM_TYPE_PAGE_POOL,
> + rxq->page_pool);
> + if (err)
> + goto err_unregister_rxq;
> +
> + return 0;
> +
> +err_unregister_rxq:
> + xdp_rxq_info_unreg(&rxq->xdp_rxq);
> +err_free_pp:
> + page_pool_destroy(rxq->page_pool);
> + return err;
> +}
> +
> /* Handle rxq fill: allocates rxq skbs; called when initializing a port */
> static int mvneta_rxq_fill(struct mvneta_port *pp, struct mvneta_rx_queue *rxq,
> int num)
> {
> - int i;
> + int i, err;
> +
> + err = mvneta_create_page_pool(pp, rxq, num);
> + if (err < 0)
> + return err;
>
> for (i = 0; i < num; i++) {
> memset(rxq->descs + i, 0, sizeof(struct mvneta_rx_desc));
> --
> 2.21.0
>
Thanks
/Ilias
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 3/7] net: mvneta: rely on build_skb in mvneta_rx_swbm poll routine
2019-10-05 20:44 ` [PATCH 3/7] net: mvneta: rely on build_skb in mvneta_rx_swbm poll routine Lorenzo Bianconi
@ 2019-10-07 11:05 ` Ilias Apalodimas
0 siblings, 0 replies; 12+ messages in thread
From: Ilias Apalodimas @ 2019-10-07 11:05 UTC (permalink / raw)
To: Lorenzo Bianconi
Cc: netdev, davem, thomas.petazzoni, brouer, lorenzo.bianconi, matteo.croce
Hi Lorenzo,
On Sat, Oct 05, 2019 at 10:44:36PM +0200, Lorenzo Bianconi wrote:
> Refactor mvneta_rx_swbm code introducing mvneta_swbm_rx_frame and
> mvneta_swbm_add_rx_fragment routines. Rely on build_skb in order to
> allocate skb since the previous patch introduced buffer recycling using
> the page_pool API
>
> Tested-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
> Signed-off-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
> Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
> ---
> drivers/net/ethernet/marvell/mvneta.c | 198 ++++++++++++++------------
> 1 file changed, 104 insertions(+), 94 deletions(-)
>
> diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
> index 8beae0e1eda7..d775fcae9353 100644
> --- a/drivers/net/ethernet/marvell/mvneta.c
> +++ b/drivers/net/ethernet/marvell/mvneta.c
> @@ -323,6 +323,11 @@
> ETH_HLEN + ETH_FCS_LEN, \
> cache_line_size())
>
> +#define MVNETA_SKB_PAD (SKB_DATA_ALIGN(sizeof(struct skb_shared_info) + \
> + NET_SKB_PAD))
> +#define MVNETA_SKB_SIZE(len) (SKB_DATA_ALIGN(len) + MVNETA_SKB_PAD)
> +#define MVNETA_MAX_RX_BUF_SIZE (PAGE_SIZE - MVNETA_SKB_PAD)
> +
> #define IS_TSO_HEADER(txq, addr) \
> ((addr >= txq->tso_hdrs_phys) && \
> (addr < txq->tso_hdrs_phys + txq->size * TSO_HEADER_SIZE))
> @@ -646,7 +651,6 @@ static int txq_number = 8;
> static int rxq_def;
>
> static int rx_copybreak __read_mostly = 256;
> -static int rx_header_size __read_mostly = 128;
>
> /* HW BM need that each port be identify by a unique ID */
> static int global_port_id;
> @@ -1941,30 +1945,102 @@ int mvneta_rx_refill_queue(struct mvneta_port *pp, struct mvneta_rx_queue *rxq)
> return i;
> }
>
> +static int
> +mvneta_swbm_rx_frame(struct mvneta_port *pp,
> + struct mvneta_rx_desc *rx_desc,
> + struct mvneta_rx_queue *rxq,
> + struct page *page)
> +{
> + unsigned char *data = page_address(page);
> + int data_len = -MVNETA_MH_SIZE, len;
> + struct net_device *dev = pp->dev;
> + enum dma_data_direction dma_dir;
> +
> + if (MVNETA_SKB_SIZE(rx_desc->data_size) > PAGE_SIZE) {
> + len = MVNETA_MAX_RX_BUF_SIZE;
> + data_len += len;
> + } else {
> + len = rx_desc->data_size;
> + data_len += len - ETH_FCS_LEN;
> + }
> +
> + dma_dir = page_pool_get_dma_dir(rxq->page_pool);
> + dma_sync_single_range_for_cpu(dev->dev.parent,
> + rx_desc->buf_phys_addr, 0,
> + len, dma_dir);
> +
> + rxq->skb = build_skb(data, PAGE_SIZE);
> + if (unlikely(!rxq->skb)) {
> + netdev_err(dev,
> + "Can't allocate skb on queue %d\n",
> + rxq->id);
> + dev->stats.rx_dropped++;
> + rxq->skb_alloc_err++;
> + return -ENOMEM;
> + }
> + page_pool_release_page(rxq->page_pool, page);
> +
> + skb_reserve(rxq->skb, MVNETA_MH_SIZE + NET_SKB_PAD);
> + skb_put(rxq->skb, data_len);
> + mvneta_rx_csum(pp, rx_desc->status, rxq->skb);
> +
> + rxq->left_size = rx_desc->data_size - len;
> + rx_desc->buf_phys_addr = 0;
> +
> + return 0;
> +}
> +
> +static void
> +mvneta_swbm_add_rx_fragment(struct mvneta_port *pp,
> + struct mvneta_rx_desc *rx_desc,
> + struct mvneta_rx_queue *rxq,
> + struct page *page)
> +{
> + struct net_device *dev = pp->dev;
> + enum dma_data_direction dma_dir;
> + int data_len, len;
> +
> + if (rxq->left_size > MVNETA_MAX_RX_BUF_SIZE) {
> + len = MVNETA_MAX_RX_BUF_SIZE;
> + data_len = len;
> + } else {
> + len = rxq->left_size;
> + data_len = len - ETH_FCS_LEN;
> + }
> + dma_dir = page_pool_get_dma_dir(rxq->page_pool);
> + dma_sync_single_range_for_cpu(dev->dev.parent,
> + rx_desc->buf_phys_addr, 0,
> + len, dma_dir);
> + if (data_len > 0) {
> + /* refill descriptor with new buffer later */
> + skb_add_rx_frag(rxq->skb,
> + skb_shinfo(rxq->skb)->nr_frags,
> + page, NET_SKB_PAD, data_len,
> + PAGE_SIZE);
> +
> + page_pool_release_page(rxq->page_pool, page);
> + rx_desc->buf_phys_addr = 0;
> + }
> + rxq->left_size -= len;
> +}
> +
> /* Main rx processing when using software buffer management */
> static int mvneta_rx_swbm(struct napi_struct *napi,
> struct mvneta_port *pp, int budget,
> struct mvneta_rx_queue *rxq)
> {
> - struct net_device *dev = pp->dev;
> - int rx_todo, rx_proc;
> - int refill = 0;
> - u32 rcvd_pkts = 0;
> - u32 rcvd_bytes = 0;
> + int rcvd_pkts = 0, rcvd_bytes = 0;
> + int rx_pending, refill, done = 0;
>
> /* Get number of received packets */
> - rx_todo = mvneta_rxq_busy_desc_num_get(pp, rxq);
> - rx_proc = 0;
> + rx_pending = mvneta_rxq_busy_desc_num_get(pp, rxq);
>
> /* Fairness NAPI loop */
> - while ((rcvd_pkts < budget) && (rx_proc < rx_todo)) {
> + while (done < budget && done < rx_pending) {
> struct mvneta_rx_desc *rx_desc = mvneta_rxq_next_desc_get(rxq);
> unsigned char *data;
> struct page *page;
> - dma_addr_t phys_addr;
> - u32 rx_status, index;
> - int rx_bytes, skb_size, copy_size;
> - int frag_num, frag_size, frag_offset;
> + int index;
>
> index = rx_desc - rxq->descs;
> page = (struct page *)rxq->buf_virt_addr[index];
> @@ -1972,98 +2048,33 @@ static int mvneta_rx_swbm(struct napi_struct *napi,
> /* Prefetch header */
> prefetch(data);
>
> - phys_addr = rx_desc->buf_phys_addr;
> - rx_status = rx_desc->status;
> - rx_proc++;
> rxq->refill_num++;
> + done++;
> +
> + if (rx_desc->status & MVNETA_RXD_FIRST_DESC) {
> + int err;
>
> - if (rx_status & MVNETA_RXD_FIRST_DESC) {
> /* Check errors only for FIRST descriptor */
> - if (rx_status & MVNETA_RXD_ERR_SUMMARY) {
> + if (rx_desc->status & MVNETA_RXD_ERR_SUMMARY) {
> mvneta_rx_error(pp, rx_desc);
> - dev->stats.rx_errors++;
> + pp->dev->stats.rx_errors++;
> /* leave the descriptor untouched */
> continue;
> }
> - rx_bytes = rx_desc->data_size -
> - (ETH_FCS_LEN + MVNETA_MH_SIZE);
>
> - /* Allocate small skb for each new packet */
> - skb_size = max(rx_copybreak, rx_header_size);
> - rxq->skb = netdev_alloc_skb_ip_align(dev, skb_size);
> - if (unlikely(!rxq->skb)) {
> - netdev_err(dev,
> - "Can't allocate skb on queue %d\n",
> - rxq->id);
> - dev->stats.rx_dropped++;
> - rxq->skb_alloc_err++;
> + err = mvneta_swbm_rx_frame(pp, rx_desc, rxq, page);
> + if (err)
> continue;
The error is practically a build_skb failure probably due to low memory.
I think it's better to break the loop and let the system reclaim some of it's
free memory instead of continuing to process (and allocate more skb's)
> - }
> - copy_size = min(skb_size, rx_bytes);
> -
> - /* Copy data from buffer to SKB, skip Marvell header */
> - memcpy(rxq->skb->data, data + MVNETA_MH_SIZE,
> - copy_size);
> - skb_put(rxq->skb, copy_size);
> - rxq->left_size = rx_bytes - copy_size;
> -
> - mvneta_rx_csum(pp, rx_status, rxq->skb);
> - if (rxq->left_size == 0) {
> - int size = copy_size + MVNETA_MH_SIZE;
> -
> - dma_sync_single_range_for_cpu(dev->dev.parent,
> - phys_addr, 0,
> - size,
> - DMA_FROM_DEVICE);
> -
> - /* leave the descriptor and buffer untouched */
> - } else {
> - /* refill descriptor with new buffer later */
> - rx_desc->buf_phys_addr = 0;
> -
> - frag_num = 0;
> - frag_offset = copy_size + MVNETA_MH_SIZE;
> - frag_size = min(rxq->left_size,
> - (int)(PAGE_SIZE - frag_offset));
> - skb_add_rx_frag(rxq->skb, frag_num, page,
> - frag_offset, frag_size,
> - PAGE_SIZE);
> - page_pool_release_page(rxq->page_pool, page);
> - rxq->left_size -= frag_size;
> - }
> } else {
> - /* Middle or Last descriptor */
> if (unlikely(!rxq->skb)) {
> pr_debug("no skb for rx_status 0x%x\n",
> - rx_status);
> + rx_desc->status);
> continue;
> }
> - if (!rxq->left_size) {
> - /* last descriptor has only FCS */
> - /* and can be discarded */
> - dma_sync_single_range_for_cpu(dev->dev.parent,
> - phys_addr, 0,
> - ETH_FCS_LEN,
> - DMA_FROM_DEVICE);
> - /* leave the descriptor and buffer untouched */
> - } else {
> - /* refill descriptor with new buffer later */
> - rx_desc->buf_phys_addr = 0;
> -
> - frag_num = skb_shinfo(rxq->skb)->nr_frags;
> - frag_offset = 0;
> - frag_size = min(rxq->left_size,
> - (int)(PAGE_SIZE - frag_offset));
> - skb_add_rx_frag(rxq->skb, frag_num, page,
> - frag_offset, frag_size,
> - PAGE_SIZE);
> -
> - page_pool_release_page(rxq->page_pool, page);
> - rxq->left_size -= frag_size;
> - }
> + mvneta_swbm_add_rx_fragment(pp, rx_desc, rxq, page);
> } /* Middle or Last descriptor */
>
> - if (!(rx_status & MVNETA_RXD_LAST_DESC))
> + if (!(rx_desc->status & MVNETA_RXD_LAST_DESC))
> /* no last descriptor this time */
> continue;
>
> @@ -2079,13 +2090,12 @@ static int mvneta_rx_swbm(struct napi_struct *napi,
> rcvd_bytes += rxq->skb->len;
>
> /* Linux processing */
> - rxq->skb->protocol = eth_type_trans(rxq->skb, dev);
> + rxq->skb->protocol = eth_type_trans(rxq->skb, pp->dev);
>
> napi_gro_receive(napi, rxq->skb);
>
> /* clean uncomplete skb pointer in queue */
> rxq->skb = NULL;
> - rxq->left_size = 0;
> }
>
> mvneta_update_stats(pp, rcvd_pkts, rcvd_bytes, false);
> @@ -2094,7 +2104,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi,
> refill = mvneta_rx_refill_queue(pp, rxq);
>
> /* Update rxq management counters */
> - mvneta_rxq_desc_num_update(pp, rxq, rx_proc, refill);
> + mvneta_rxq_desc_num_update(pp, rxq, done, refill);
>
> return rcvd_pkts;
> }
> @@ -2945,7 +2955,7 @@ static void mvneta_rxq_hw_init(struct mvneta_port *pp,
> /* Set Offset */
> mvneta_rxq_offset_set(pp, rxq, 0);
> mvneta_rxq_buf_size_set(pp, rxq, PAGE_SIZE < SZ_64K ?
> - PAGE_SIZE :
> + MVNETA_MAX_RX_BUF_SIZE :
> MVNETA_RX_BUF_SIZE(pp->pkt_size));
> mvneta_rxq_bm_disable(pp, rxq);
> mvneta_rxq_fill(pp, rxq, rxq->size);
> @@ -4655,7 +4665,7 @@ static int mvneta_probe(struct platform_device *pdev)
> SET_NETDEV_DEV(dev, &pdev->dev);
>
> pp->id = global_port_id++;
> - pp->rx_offset_correction = 0; /* not relevant for SW BM */
> + pp->rx_offset_correction = NET_SKB_PAD;
>
> /* Obtain access to BM resources if enabled and already initialized */
> bm_node = of_parse_phandle(dn, "buffer-manager", 0);
> --
> 2.21.0
>
Regards
/Ilias
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 2/7] net: mvneta: introduce page pool API for sw buffer manager
2019-10-05 21:34 ` Ilias Apalodimas
@ 2019-10-07 12:27 ` Lorenzo Bianconi
0 siblings, 0 replies; 12+ messages in thread
From: Lorenzo Bianconi @ 2019-10-07 12:27 UTC (permalink / raw)
To: Ilias Apalodimas
Cc: netdev, davem, thomas.petazzoni, brouer, lorenzo.bianconi, matteo.croce
[-- Attachment #1: Type: text/plain, Size: 6643 bytes --]
> Hi Lorenzo,
>
Hi Ilias,
> On Sat, Oct 05, 2019 at 10:44:35PM +0200, Lorenzo Bianconi wrote:
> > Use the page_pool api for allocations and DMA handling instead of
> > __dev_alloc_page()/dma_map_page() and free_page()/dma_unmap_page().
> > Pages are unmapped using page_pool_release_page before packets
> > go into the network stack.
> >
> > The page_pool API offers buffer recycling capabilities for XDP but
> > allocates one page per packet, unless the driver splits and manages
> > the allocated page.
> > This is a preliminary patch to add XDP support to mvneta driver
> >
> > Tested-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
> > Signed-off-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
> > Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
> > Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
> > ---
> > drivers/net/ethernet/marvell/Kconfig | 1 +
> > drivers/net/ethernet/marvell/mvneta.c | 76 ++++++++++++++++++++-------
> > 2 files changed, 58 insertions(+), 19 deletions(-)
> >
> > diff --git a/drivers/net/ethernet/marvell/Kconfig b/drivers/net/ethernet/marvell/Kconfig
> > index fb942167ee54..3d5caea096fb 100644
> > --- a/drivers/net/ethernet/marvell/Kconfig
> > +++ b/drivers/net/ethernet/marvell/Kconfig
> > @@ -61,6 +61,7 @@ config MVNETA
> > depends on ARCH_MVEBU || COMPILE_TEST
> > select MVMDIO
> > select PHYLINK
> > + select PAGE_POOL
> > ---help---
> > This driver supports the network interface units in the
> > Marvell ARMADA XP, ARMADA 370, ARMADA 38x and
> > diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
> > index 128b9fded959..8beae0e1eda7 100644
> > --- a/drivers/net/ethernet/marvell/mvneta.c
> > +++ b/drivers/net/ethernet/marvell/mvneta.c
> > @@ -37,6 +37,7 @@
> > #include <net/ip.h>
> > #include <net/ipv6.h>
> > #include <net/tso.h>
> > +#include <net/page_pool.h>
> >
> > /* Registers */
> > #define MVNETA_RXQ_CONFIG_REG(q) (0x1400 + ((q) << 2))
> > @@ -603,6 +604,10 @@ struct mvneta_rx_queue {
> > u32 pkts_coal;
> > u32 time_coal;
> >
> > + /* page_pool */
> > + struct page_pool *page_pool;
> > + struct xdp_rxq_info xdp_rxq;
> > +
> > /* Virtual address of the RX buffer */
> > void **buf_virt_addr;
> >
> > @@ -1815,19 +1820,12 @@ static int mvneta_rx_refill(struct mvneta_port *pp,
> > dma_addr_t phys_addr;
> > struct page *page;
> >
> > - page = __dev_alloc_page(gfp_mask);
> > + page = page_pool_alloc_pages(rxq->page_pool,
> > + gfp_mask | __GFP_NOWARN);
> > if (!page)
> > return -ENOMEM;
>
> Is the driver syncing the buffer somewhere else? (for_device)
> If not you'll have to do this here.
ack, I will add a dma_sync_to_device() in v2, thx for the hint :)
>
> On a non-cache coherent machine (and i think this one is) you may get dirty
> cache lines handed to the device. Those dirty cache lines might get written back
> *after* the device has DMA'ed it's data. You need to flush those first to avoid
> any data corruption
Right.
Regards,
Lorenzo
>
> >
> > - /* map page for use */
> > - phys_addr = dma_map_page(pp->dev->dev.parent, page, 0, PAGE_SIZE,
> > - DMA_FROM_DEVICE);
> > - if (unlikely(dma_mapping_error(pp->dev->dev.parent, phys_addr))) {
> > - __free_page(page);
> > - return -ENOMEM;
> > - }
> > -
> > - phys_addr += pp->rx_offset_correction;
> > + phys_addr = page_pool_get_dma_addr(page) + pp->rx_offset_correction;
> > mvneta_rx_desc_fill(rx_desc, phys_addr, page, rxq);
> > return 0;
> > }
> > @@ -1894,10 +1892,11 @@ static void mvneta_rxq_drop_pkts(struct mvneta_port *pp,
> > if (!data || !(rx_desc->buf_phys_addr))
> > continue;
> >
> > - dma_unmap_page(pp->dev->dev.parent, rx_desc->buf_phys_addr,
> > - PAGE_SIZE, DMA_FROM_DEVICE);
> > - __free_page(data);
> > + page_pool_put_page(rxq->page_pool, data, false);
> > }
> > + if (xdp_rxq_info_is_reg(&rxq->xdp_rxq))
> > + xdp_rxq_info_unreg(&rxq->xdp_rxq);
> > + page_pool_destroy(rxq->page_pool);
> > }
> >
> > static void
> > @@ -2029,8 +2028,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi,
> > skb_add_rx_frag(rxq->skb, frag_num, page,
> > frag_offset, frag_size,
> > PAGE_SIZE);
> > - dma_unmap_page(dev->dev.parent, phys_addr,
> > - PAGE_SIZE, DMA_FROM_DEVICE);
> > + page_pool_release_page(rxq->page_pool, page);
> > rxq->left_size -= frag_size;
> > }
> > } else {
> > @@ -2060,9 +2058,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi,
> > frag_offset, frag_size,
> > PAGE_SIZE);
> >
> > - dma_unmap_page(dev->dev.parent, phys_addr,
> > - PAGE_SIZE, DMA_FROM_DEVICE);
> > -
> > + page_pool_release_page(rxq->page_pool, page);
> > rxq->left_size -= frag_size;
> > }
> > } /* Middle or Last descriptor */
> > @@ -2829,11 +2825,53 @@ static int mvneta_poll(struct napi_struct *napi, int budget)
> > return rx_done;
> > }
> >
> > +static int mvneta_create_page_pool(struct mvneta_port *pp,
> > + struct mvneta_rx_queue *rxq, int size)
> > +{
> > + struct page_pool_params pp_params = {
> > + .order = 0,
> > + .flags = PP_FLAG_DMA_MAP,
> > + .pool_size = size,
> > + .nid = cpu_to_node(0),
> > + .dev = pp->dev->dev.parent,
> > + .dma_dir = DMA_FROM_DEVICE,
> > + };
> > + int err;
> > +
> > + rxq->page_pool = page_pool_create(&pp_params);
> > + if (IS_ERR(rxq->page_pool)) {
> > + err = PTR_ERR(rxq->page_pool);
> > + rxq->page_pool = NULL;
> > + return err;
> > + }
> > +
> > + err = xdp_rxq_info_reg(&rxq->xdp_rxq, pp->dev, 0);
> > + if (err < 0)
> > + goto err_free_pp;
> > +
> > + err = xdp_rxq_info_reg_mem_model(&rxq->xdp_rxq, MEM_TYPE_PAGE_POOL,
> > + rxq->page_pool);
> > + if (err)
> > + goto err_unregister_rxq;
> > +
> > + return 0;
> > +
> > +err_unregister_rxq:
> > + xdp_rxq_info_unreg(&rxq->xdp_rxq);
> > +err_free_pp:
> > + page_pool_destroy(rxq->page_pool);
> > + return err;
> > +}
> > +
> > /* Handle rxq fill: allocates rxq skbs; called when initializing a port */
> > static int mvneta_rxq_fill(struct mvneta_port *pp, struct mvneta_rx_queue *rxq,
> > int num)
> > {
> > - int i;
> > + int i, err;
> > +
> > + err = mvneta_create_page_pool(pp, rxq, num);
> > + if (err < 0)
> > + return err;
> >
> > for (i = 0; i < num; i++) {
> > memset(rxq->descs + i, 0, sizeof(struct mvneta_rx_desc));
> > --
> > 2.21.0
> >
>
>
> Thanks
> /Ilias
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2019-10-07 12:28 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-05 20:44 [PATCH 0/7] add XDP support to mvneta driver Lorenzo Bianconi
2019-10-05 20:44 ` [PATCH 1/7] net: mvneta: introduce mvneta_update_stats routine Lorenzo Bianconi
2019-10-05 20:44 ` [PATCH 2/7] net: mvneta: introduce page pool API for sw buffer manager Lorenzo Bianconi
2019-10-05 21:34 ` Ilias Apalodimas
2019-10-07 12:27 ` Lorenzo Bianconi
2019-10-05 20:44 ` [PATCH 3/7] net: mvneta: rely on build_skb in mvneta_rx_swbm poll routine Lorenzo Bianconi
2019-10-07 11:05 ` Ilias Apalodimas
2019-10-05 20:44 ` [PATCH 4/7] net: mvneta: add basic XDP support Lorenzo Bianconi
2019-10-05 20:44 ` [PATCH 5/7] net: mvneta: move header prefetch in mvneta_swbm_rx_frame Lorenzo Bianconi
2019-10-05 20:44 ` [PATCH 6/7] net: mvneta: make tx buffer array agnostic Lorenzo Bianconi
2019-10-05 20:44 ` [PATCH 7/7] net: mvneta: add XDP_TX support Lorenzo Bianconi
2019-10-05 21:02 ` [PATCH 0/7] add XDP support to mvneta driver Lorenzo Bianconi
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).