* [PATCH net-next 0/6] stmmac: Add XDP support
@ 2021-03-29 13:40 ` Ong Boon Leong
0 siblings, 0 replies; 15+ messages in thread
From: Ong Boon Leong @ 2021-03-29 13:40 UTC (permalink / raw)
To: Giuseppe Cavallaro, Alexandre Torgue, Jose Abreu,
David S . Miller, Jakub Kicinski, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend
Cc: Maxime Coquelin, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, netdev, linux-stm32, linux-arm-kernel,
linux-kernel, bpf, Ong Boon Leong
Hi,
This is the patch series to adds XDP support to stmmac driver.
Summary of the changes as follow:-
1/6: Add IRQ affinity hint to make RXQ and TXQ to be by default serviced
by the same CPU. This is required for Intel mGbE controller which
has independent per DMA channel RX & and TX IRQ resources.
2/6: To add capability to enable/disable Split Header (SPH) dynamically.
Disable SPH for XDP and revert back to the SPH capability of the IP
when XDP exits.
3/6: To rearrange TX tail pointer update into common function for reuse
across driver.
4/6: Add basic XDP framework into stmmac driver. This patch only support
XDP_DROP only.
5/6: Add support for XDP_TX.
6/6: Add support for XDP_REDIRECT and ndo_xdp_xmit() implementation.
To send burst traffics to DUT, we use samples/pktgen script called
pktgen_sample03_burst_single_flow.sh. Following shows the test steps
and results that are obtained using Intel mGbE controller under
preempt-rt environment. In such environment, I also have sent a patch
that fixes an issue found in xdp_return_frame() earlier:
https://patchwork.kernel.org/project/netdevbpf/list/?series=457139
########################################################################
root@intel-corei7-64:~# ./xdp1 eth0
libbpf: elf: skipping unrecognized data section(16) .eh_frame
libbpf: elf: skipping relo section(17) .rel.eh_frame for section(16) .eh_frame
proto 17: 421844 pkt/s
proto 17: 684396 pkt/s
proto 17: 684332 pkt/s
proto 17: 684074 pkt/s
proto 17: 683837 pkt/s
proto 17: 684357 pkt/s
proto 17: 684371 pkt/s
proto 17: 684419 pkt/s
proto 17: 683996 pkt/s
root@intel-corei7-64:~# ./xdp1 -S eth0
libbpf: elf: skipping unrecognized data section(16) .eh_frame
libbpf: elf: skipping relo section(17) .rel.eh_frame for section(16) .eh_frame
proto 0: 1 pkt/s
proto 17: 1 pkt/s
proto 17: 1 pkt/s
proto 17: 65564 pkt/s
proto 17: 117450 pkt/s
proto 17: 117600 pkt/s
proto 17: 118108 pkt/s
proto 17: 118032 pkt/s
proto 17: 118092 pkt/s
root@intel-corei7-64:~# ./xdp2 eth0
libbpf: elf: skipping unrecognized data section(16) .eh_frame
libbpf: elf: skipping relo section(17) .rel.eh_frame for section(16) .eh_frame
proto 17: 0 pkt/s
proto 17: 49338 pkt/s
proto 17: 714255 pkt/s
proto 17: 714495 pkt/s
proto 17: 710120 pkt/s
proto 17: 712892 pkt/s
proto 17: 712585 pkt/s
proto 17: 713883 pkt/s
root@intel-corei7-64:~# ./xdp2 -S eth0
libbpf: elf: skipping unrecognized data section(16) .eh_frame
libbpf: elf: skipping relo section(17) .rel.eh_frame for section(16) .eh_frame
proto 0: 0 pkt/s
proto 17: 89483 pkt/s
proto 17: 89540 pkt/s
proto 17: 89358 pkt/s
proto 17: 89797 pkt/s
proto 0: 1 pkt/s
proto 17: 89672 pkt/s
root@intel-corei7-64:~# ./xdp_redirect eth0 eth1
input: 7 output: 8
libbpf: elf: skipping unrecognized data section(20) .eh_frame
libbpf: elf: skipping relo section(21) .rel.eh_frame for section(20) .eh_frame
ifindex 8: 143805 pkt/s
ifindex 8: 676900 pkt/s
ifindex 8: 676801 pkt/s
ifindex 8: 677657 pkt/s
ifindex 8: 677916 pkt/s
root@intel-corei7-64:~# ./xdp_redirect -S eth0 eth1
input: 7 output: 8
libbpf: elf: skipping unrecognized data section(20) .eh_frame
libbpf: elf: skipping relo section(21) .rel.eh_frame for section(20) .eh_frame
ifindex 8: 2 pkt/s
ifindex 8: 0 pkt/s
ifindex 8: 33505 pkt/s
ifindex 8: 73536 pkt/s
ifindex 8: 52512 pkt/s
ifindex 8: 97600 pkt/s
ifindex 8: 96928 pkt/s
ifindex 8: 96480 pkt/s
ifindex 8: 96760 pkt/s
ifindex 8: 96949 pkt/s
ifindex 8: 96591 pkt/s
root@intel-corei7-64:~# ./xdp_redirect_cpu --dev eth0 --cpu 0
libbpf: elf: skipping unrecognized data section(39) .eh_frame
libbpf: elf: skipping relo section(40) .rel.eh_frame for section(39) .eh_frame
libbpf: elf: skipping unrecognized data section(20) .eh_frame
libbpf: elf: skipping relo section(21) .rel.eh_frame for section(20) .eh_frame
Add-new CPU:0 as idx:0 qsize:192 prog_fd: 33 (cpus_count:1)
Running XDP/eBPF prog_name:xdp_cpu_map5_lb_hash_ip_pairs
XDP-cpumap CPU:to pps drop-pps extra-info
XDP-RX 0 667,952 0 0
XDP-RX total 667,952 0
cpumap-enqueue 0:0 667,952 550,150 7.82 bulk-average
cpumap-enqueue sum:0 667,952 550,150 7.82 bulk-average
cpumap_kthread 0 117,821 0 0
cpumap_kthread total 117,821 0 0
redirect_err total 0 0
xdp_exception total 0 0
2nd remote XDP/eBPF prog_name: xdp_redirect_dummy
XDP-cpumap CPU:to xdp-pass xdp-drop xdp-redir
xdp-in-kthread 0 117,822 0 0
xdp-in-kthread total 117,822 0 0
Running XDP/eBPF prog_name:xdp_cpu_map5_lb_hash_ip_pairs
XDP-cpumap CPU:to pps drop-pps extra-info
XDP-RX 0 667,871 0 0
XDP-RX total 667,871 0
cpumap-enqueue 0:0 667,877 550,820 7.82 bulk-average
cpumap-enqueue sum:0 667,877 550,820 7.82 bulk-average
cpumap_kthread 0 117,041 0 0
cpumap_kthread total 117,041 0 0
redirect_err total 0 0
xdp_exception total 0 0
2nd remote XDP/eBPF prog_name: xdp_redirect_dummy
XDP-cpumap CPU:to xdp-pass xdp-drop xdp-redir
xdp-in-kthread 0 117,041 0 0
xdp-in-kthread total 117,041 0 0
Running XDP/eBPF prog_name:xdp_cpu_map5_lb_hash_ip_pairs
XDP-cpumap CPU:to pps drop-pps extra-info
XDP-RX 0 667,856 0 0
XDP-RX total 667,856 0
cpumap-enqueue 0:0 667,849 549,672 7.82 bulk-average
cpumap-enqueue sum:0 667,849 549,672 7.82 bulk-average
cpumap_kthread 0 118,177 0 0
cpumap_kthread total 118,177 0 0
redirect_err total 0 0
xdp_exception total 0 0
2nd remote XDP/eBPF prog_name: xdp_redirect_dummy
XDP-cpumap CPU:to xdp-pass xdp-drop xdp-redir
xdp-in-kthread 0 118,177 0 0
xdp-in-kthread total 118,177 0 0
########################################################################
As stmmac driver is using page_pool mechanism, I waited > 1min after
each of above apps is terminated to make sure there is no stalled
pool warning prints on the terminal, and I found none which looks good
on Intel mGbE platform.
It will be great if community help tes out these patch series on your
platform and provide me feedback.
Thank you very much,
Boon Leong
Ong Boon Leong (6):
stmmac: intel: set IRQ affinity hint for multi MSI vectors
net: stmmac: make SPH enable/disable to be configurable
net: stmmac: arrange Tx tail pointer update to
stmmac_flush_tx_descriptors
net: stmmac: Add initial XDP support
net: stmmac: Add support for XDP_TX action
net: stmmac: Add support for XDP_REDIRECT action
drivers/net/ethernet/stmicro/stmmac/Makefile | 1 +
.../net/ethernet/stmicro/stmmac/dwmac-intel.c | 7 +
drivers/net/ethernet/stmicro/stmmac/stmmac.h | 35 +-
.../net/ethernet/stmicro/stmmac/stmmac_main.c | 502 +++++++++++++++---
.../net/ethernet/stmicro/stmmac/stmmac_xdp.c | 40 ++
.../net/ethernet/stmicro/stmmac/stmmac_xdp.h | 12 +
6 files changed, 523 insertions(+), 74 deletions(-)
create mode 100644 drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.c
create mode 100644 drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.h
--
2.25.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH net-next 0/6] stmmac: Add XDP support
@ 2021-03-29 13:40 ` Ong Boon Leong
0 siblings, 0 replies; 15+ messages in thread
From: Ong Boon Leong @ 2021-03-29 13:40 UTC (permalink / raw)
To: Giuseppe Cavallaro, Alexandre Torgue, Jose Abreu,
David S . Miller, Jakub Kicinski, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend
Cc: Maxime Coquelin, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, netdev, linux-stm32, linux-arm-kernel,
linux-kernel, bpf, Ong Boon Leong
Hi,
This is the patch series to adds XDP support to stmmac driver.
Summary of the changes as follow:-
1/6: Add IRQ affinity hint to make RXQ and TXQ to be by default serviced
by the same CPU. This is required for Intel mGbE controller which
has independent per DMA channel RX & and TX IRQ resources.
2/6: To add capability to enable/disable Split Header (SPH) dynamically.
Disable SPH for XDP and revert back to the SPH capability of the IP
when XDP exits.
3/6: To rearrange TX tail pointer update into common function for reuse
across driver.
4/6: Add basic XDP framework into stmmac driver. This patch only support
XDP_DROP only.
5/6: Add support for XDP_TX.
6/6: Add support for XDP_REDIRECT and ndo_xdp_xmit() implementation.
To send burst traffics to DUT, we use samples/pktgen script called
pktgen_sample03_burst_single_flow.sh. Following shows the test steps
and results that are obtained using Intel mGbE controller under
preempt-rt environment. In such environment, I also have sent a patch
that fixes an issue found in xdp_return_frame() earlier:
https://patchwork.kernel.org/project/netdevbpf/list/?series=457139
########################################################################
root@intel-corei7-64:~# ./xdp1 eth0
libbpf: elf: skipping unrecognized data section(16) .eh_frame
libbpf: elf: skipping relo section(17) .rel.eh_frame for section(16) .eh_frame
proto 17: 421844 pkt/s
proto 17: 684396 pkt/s
proto 17: 684332 pkt/s
proto 17: 684074 pkt/s
proto 17: 683837 pkt/s
proto 17: 684357 pkt/s
proto 17: 684371 pkt/s
proto 17: 684419 pkt/s
proto 17: 683996 pkt/s
root@intel-corei7-64:~# ./xdp1 -S eth0
libbpf: elf: skipping unrecognized data section(16) .eh_frame
libbpf: elf: skipping relo section(17) .rel.eh_frame for section(16) .eh_frame
proto 0: 1 pkt/s
proto 17: 1 pkt/s
proto 17: 1 pkt/s
proto 17: 65564 pkt/s
proto 17: 117450 pkt/s
proto 17: 117600 pkt/s
proto 17: 118108 pkt/s
proto 17: 118032 pkt/s
proto 17: 118092 pkt/s
root@intel-corei7-64:~# ./xdp2 eth0
libbpf: elf: skipping unrecognized data section(16) .eh_frame
libbpf: elf: skipping relo section(17) .rel.eh_frame for section(16) .eh_frame
proto 17: 0 pkt/s
proto 17: 49338 pkt/s
proto 17: 714255 pkt/s
proto 17: 714495 pkt/s
proto 17: 710120 pkt/s
proto 17: 712892 pkt/s
proto 17: 712585 pkt/s
proto 17: 713883 pkt/s
root@intel-corei7-64:~# ./xdp2 -S eth0
libbpf: elf: skipping unrecognized data section(16) .eh_frame
libbpf: elf: skipping relo section(17) .rel.eh_frame for section(16) .eh_frame
proto 0: 0 pkt/s
proto 17: 89483 pkt/s
proto 17: 89540 pkt/s
proto 17: 89358 pkt/s
proto 17: 89797 pkt/s
proto 0: 1 pkt/s
proto 17: 89672 pkt/s
root@intel-corei7-64:~# ./xdp_redirect eth0 eth1
input: 7 output: 8
libbpf: elf: skipping unrecognized data section(20) .eh_frame
libbpf: elf: skipping relo section(21) .rel.eh_frame for section(20) .eh_frame
ifindex 8: 143805 pkt/s
ifindex 8: 676900 pkt/s
ifindex 8: 676801 pkt/s
ifindex 8: 677657 pkt/s
ifindex 8: 677916 pkt/s
root@intel-corei7-64:~# ./xdp_redirect -S eth0 eth1
input: 7 output: 8
libbpf: elf: skipping unrecognized data section(20) .eh_frame
libbpf: elf: skipping relo section(21) .rel.eh_frame for section(20) .eh_frame
ifindex 8: 2 pkt/s
ifindex 8: 0 pkt/s
ifindex 8: 33505 pkt/s
ifindex 8: 73536 pkt/s
ifindex 8: 52512 pkt/s
ifindex 8: 97600 pkt/s
ifindex 8: 96928 pkt/s
ifindex 8: 96480 pkt/s
ifindex 8: 96760 pkt/s
ifindex 8: 96949 pkt/s
ifindex 8: 96591 pkt/s
root@intel-corei7-64:~# ./xdp_redirect_cpu --dev eth0 --cpu 0
libbpf: elf: skipping unrecognized data section(39) .eh_frame
libbpf: elf: skipping relo section(40) .rel.eh_frame for section(39) .eh_frame
libbpf: elf: skipping unrecognized data section(20) .eh_frame
libbpf: elf: skipping relo section(21) .rel.eh_frame for section(20) .eh_frame
Add-new CPU:0 as idx:0 qsize:192 prog_fd: 33 (cpus_count:1)
Running XDP/eBPF prog_name:xdp_cpu_map5_lb_hash_ip_pairs
XDP-cpumap CPU:to pps drop-pps extra-info
XDP-RX 0 667,952 0 0
XDP-RX total 667,952 0
cpumap-enqueue 0:0 667,952 550,150 7.82 bulk-average
cpumap-enqueue sum:0 667,952 550,150 7.82 bulk-average
cpumap_kthread 0 117,821 0 0
cpumap_kthread total 117,821 0 0
redirect_err total 0 0
xdp_exception total 0 0
2nd remote XDP/eBPF prog_name: xdp_redirect_dummy
XDP-cpumap CPU:to xdp-pass xdp-drop xdp-redir
xdp-in-kthread 0 117,822 0 0
xdp-in-kthread total 117,822 0 0
Running XDP/eBPF prog_name:xdp_cpu_map5_lb_hash_ip_pairs
XDP-cpumap CPU:to pps drop-pps extra-info
XDP-RX 0 667,871 0 0
XDP-RX total 667,871 0
cpumap-enqueue 0:0 667,877 550,820 7.82 bulk-average
cpumap-enqueue sum:0 667,877 550,820 7.82 bulk-average
cpumap_kthread 0 117,041 0 0
cpumap_kthread total 117,041 0 0
redirect_err total 0 0
xdp_exception total 0 0
2nd remote XDP/eBPF prog_name: xdp_redirect_dummy
XDP-cpumap CPU:to xdp-pass xdp-drop xdp-redir
xdp-in-kthread 0 117,041 0 0
xdp-in-kthread total 117,041 0 0
Running XDP/eBPF prog_name:xdp_cpu_map5_lb_hash_ip_pairs
XDP-cpumap CPU:to pps drop-pps extra-info
XDP-RX 0 667,856 0 0
XDP-RX total 667,856 0
cpumap-enqueue 0:0 667,849 549,672 7.82 bulk-average
cpumap-enqueue sum:0 667,849 549,672 7.82 bulk-average
cpumap_kthread 0 118,177 0 0
cpumap_kthread total 118,177 0 0
redirect_err total 0 0
xdp_exception total 0 0
2nd remote XDP/eBPF prog_name: xdp_redirect_dummy
XDP-cpumap CPU:to xdp-pass xdp-drop xdp-redir
xdp-in-kthread 0 118,177 0 0
xdp-in-kthread total 118,177 0 0
########################################################################
As stmmac driver is using page_pool mechanism, I waited > 1min after
each of above apps is terminated to make sure there is no stalled
pool warning prints on the terminal, and I found none which looks good
on Intel mGbE platform.
It will be great if community help tes out these patch series on your
platform and provide me feedback.
Thank you very much,
Boon Leong
Ong Boon Leong (6):
stmmac: intel: set IRQ affinity hint for multi MSI vectors
net: stmmac: make SPH enable/disable to be configurable
net: stmmac: arrange Tx tail pointer update to
stmmac_flush_tx_descriptors
net: stmmac: Add initial XDP support
net: stmmac: Add support for XDP_TX action
net: stmmac: Add support for XDP_REDIRECT action
drivers/net/ethernet/stmicro/stmmac/Makefile | 1 +
.../net/ethernet/stmicro/stmmac/dwmac-intel.c | 7 +
drivers/net/ethernet/stmicro/stmmac/stmmac.h | 35 +-
.../net/ethernet/stmicro/stmmac/stmmac_main.c | 502 +++++++++++++++---
.../net/ethernet/stmicro/stmmac/stmmac_xdp.c | 40 ++
.../net/ethernet/stmicro/stmmac/stmmac_xdp.h | 12 +
6 files changed, 523 insertions(+), 74 deletions(-)
create mode 100644 drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.c
create mode 100644 drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.h
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH net-next 1/6] stmmac: intel: set IRQ affinity hint for multi MSI vectors
2021-03-29 13:40 ` Ong Boon Leong
@ 2021-03-29 13:40 ` Ong Boon Leong
-1 siblings, 0 replies; 15+ messages in thread
From: Ong Boon Leong @ 2021-03-29 13:40 UTC (permalink / raw)
To: Giuseppe Cavallaro, Alexandre Torgue, Jose Abreu,
David S . Miller, Jakub Kicinski, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend
Cc: Maxime Coquelin, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, netdev, linux-stm32, linux-arm-kernel,
linux-kernel, bpf, Ong Boon Leong
Intel mGBE has independent hardware IRQ resources for TX and RX DMA
operation. In preparation to support XDP TX, we add IRQ affinity hint
to group both RX and TX queue of the same queue ID to the same CPU.
Signed-off-by: Ong Boon Leong <boon.leong.ong@intel.com>
---
drivers/net/ethernet/stmicro/stmmac/dwmac-intel.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.c b/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.c
index 08b4852eed4c..53a24932a192 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.c
@@ -810,6 +810,7 @@ static int stmmac_config_multi_msi(struct pci_dev *pdev,
struct plat_stmmacenet_data *plat,
struct stmmac_resources *res)
{
+ cpumask_t cpu_mask;
int ret;
int i;
@@ -832,12 +833,18 @@ static int stmmac_config_multi_msi(struct pci_dev *pdev,
for (i = 0; i < plat->rx_queues_to_use; i++) {
res->rx_irq[i] = pci_irq_vector(pdev,
plat->msi_rx_base_vec + i * 2);
+ cpumask_clear(&cpu_mask);
+ cpumask_set_cpu(i % num_online_cpus(), &cpu_mask);
+ irq_set_affinity_hint(res->rx_irq[i], &cpu_mask);
}
/* For TX MSI */
for (i = 0; i < plat->tx_queues_to_use; i++) {
res->tx_irq[i] = pci_irq_vector(pdev,
plat->msi_tx_base_vec + i * 2);
+ cpumask_clear(&cpu_mask);
+ cpumask_set_cpu(i % num_online_cpus(), &cpu_mask);
+ irq_set_affinity_hint(res->tx_irq[i], &cpu_mask);
}
if (plat->msi_mac_vec < STMMAC_MSI_VEC_MAX)
--
2.25.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH net-next 1/6] stmmac: intel: set IRQ affinity hint for multi MSI vectors
@ 2021-03-29 13:40 ` Ong Boon Leong
0 siblings, 0 replies; 15+ messages in thread
From: Ong Boon Leong @ 2021-03-29 13:40 UTC (permalink / raw)
To: Giuseppe Cavallaro, Alexandre Torgue, Jose Abreu,
David S . Miller, Jakub Kicinski, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend
Cc: Maxime Coquelin, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, netdev, linux-stm32, linux-arm-kernel,
linux-kernel, bpf, Ong Boon Leong
Intel mGBE has independent hardware IRQ resources for TX and RX DMA
operation. In preparation to support XDP TX, we add IRQ affinity hint
to group both RX and TX queue of the same queue ID to the same CPU.
Signed-off-by: Ong Boon Leong <boon.leong.ong@intel.com>
---
drivers/net/ethernet/stmicro/stmmac/dwmac-intel.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.c b/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.c
index 08b4852eed4c..53a24932a192 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.c
@@ -810,6 +810,7 @@ static int stmmac_config_multi_msi(struct pci_dev *pdev,
struct plat_stmmacenet_data *plat,
struct stmmac_resources *res)
{
+ cpumask_t cpu_mask;
int ret;
int i;
@@ -832,12 +833,18 @@ static int stmmac_config_multi_msi(struct pci_dev *pdev,
for (i = 0; i < plat->rx_queues_to_use; i++) {
res->rx_irq[i] = pci_irq_vector(pdev,
plat->msi_rx_base_vec + i * 2);
+ cpumask_clear(&cpu_mask);
+ cpumask_set_cpu(i % num_online_cpus(), &cpu_mask);
+ irq_set_affinity_hint(res->rx_irq[i], &cpu_mask);
}
/* For TX MSI */
for (i = 0; i < plat->tx_queues_to_use; i++) {
res->tx_irq[i] = pci_irq_vector(pdev,
plat->msi_tx_base_vec + i * 2);
+ cpumask_clear(&cpu_mask);
+ cpumask_set_cpu(i % num_online_cpus(), &cpu_mask);
+ irq_set_affinity_hint(res->tx_irq[i], &cpu_mask);
}
if (plat->msi_mac_vec < STMMAC_MSI_VEC_MAX)
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH net-next 2/6] net: stmmac: make SPH enable/disable to be configurable
2021-03-29 13:40 ` Ong Boon Leong
@ 2021-03-29 13:40 ` Ong Boon Leong
-1 siblings, 0 replies; 15+ messages in thread
From: Ong Boon Leong @ 2021-03-29 13:40 UTC (permalink / raw)
To: Giuseppe Cavallaro, Alexandre Torgue, Jose Abreu,
David S . Miller, Jakub Kicinski, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend
Cc: Maxime Coquelin, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, netdev, linux-stm32, linux-arm-kernel,
linux-kernel, bpf, Ong Boon Leong
SPH functionality splits header and payload according to split mode and
offsef fields (SPLM and SPLOFST). It is beneficials for Linux network
stack RX processing however it adds a lot of complexity in XDP
processing.
So, this patch makes the split-header (SPH) capability of the controller
is stored in "priv->sph_cap" and the enabling/disabling of SPH is decided
by "priv->sph".
This is to prepare initial XDP enabling for stmmac to disable the use of
SPH whenever XDP is enabled.
Signed-off-by: Ong Boon Leong <boon.leong.ong@intel.com>
---
drivers/net/ethernet/stmicro/stmmac/stmmac.h | 1 +
drivers/net/ethernet/stmicro/stmmac/stmmac_main.c | 12 +++++++-----
2 files changed, 8 insertions(+), 5 deletions(-)
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac.h b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
index 9966f6f10905..e293423f98c3 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac.h
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
@@ -160,6 +160,7 @@ struct stmmac_priv {
bool tx_path_in_lpi_mode;
bool tso;
int sph;
+ int sph_cap;
u32 sarc_type;
unsigned int dma_buf_sz;
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
index d34388b1ffcc..ace3c3835a9f 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
@@ -2858,6 +2858,7 @@ static int stmmac_hw_setup(struct net_device *dev, bool init_ptp)
struct stmmac_priv *priv = netdev_priv(dev);
u32 rx_cnt = priv->plat->rx_queues_to_use;
u32 tx_cnt = priv->plat->tx_queues_to_use;
+ bool sph_en;
u32 chan;
int ret;
@@ -2952,10 +2953,10 @@ static int stmmac_hw_setup(struct net_device *dev, bool init_ptp)
}
/* Enable Split Header */
- if (priv->sph && priv->hw->rx_csum) {
- for (chan = 0; chan < rx_cnt; chan++)
- stmmac_enable_sph(priv, priv->ioaddr, 1, chan);
- }
+ sph_en = (priv->hw->rx_csum > 0) && priv->sph;
+ for (chan = 0; chan < rx_cnt; chan++)
+ stmmac_enable_sph(priv, priv->ioaddr, sph_en, chan);
+
/* VLAN Tag Insertion */
if (priv->dma_cap.vlins)
@@ -5697,7 +5698,8 @@ int stmmac_dvr_probe(struct device *device,
if (priv->dma_cap.sphen) {
ndev->hw_features |= NETIF_F_GRO;
- priv->sph = true;
+ priv->sph_cap = true;
+ priv->sph = priv->sph_cap;
dev_info(priv->device, "SPH feature enabled\n");
}
--
2.25.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH net-next 2/6] net: stmmac: make SPH enable/disable to be configurable
@ 2021-03-29 13:40 ` Ong Boon Leong
0 siblings, 0 replies; 15+ messages in thread
From: Ong Boon Leong @ 2021-03-29 13:40 UTC (permalink / raw)
To: Giuseppe Cavallaro, Alexandre Torgue, Jose Abreu,
David S . Miller, Jakub Kicinski, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend
Cc: Maxime Coquelin, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, netdev, linux-stm32, linux-arm-kernel,
linux-kernel, bpf, Ong Boon Leong
SPH functionality splits header and payload according to split mode and
offsef fields (SPLM and SPLOFST). It is beneficials for Linux network
stack RX processing however it adds a lot of complexity in XDP
processing.
So, this patch makes the split-header (SPH) capability of the controller
is stored in "priv->sph_cap" and the enabling/disabling of SPH is decided
by "priv->sph".
This is to prepare initial XDP enabling for stmmac to disable the use of
SPH whenever XDP is enabled.
Signed-off-by: Ong Boon Leong <boon.leong.ong@intel.com>
---
drivers/net/ethernet/stmicro/stmmac/stmmac.h | 1 +
drivers/net/ethernet/stmicro/stmmac/stmmac_main.c | 12 +++++++-----
2 files changed, 8 insertions(+), 5 deletions(-)
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac.h b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
index 9966f6f10905..e293423f98c3 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac.h
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
@@ -160,6 +160,7 @@ struct stmmac_priv {
bool tx_path_in_lpi_mode;
bool tso;
int sph;
+ int sph_cap;
u32 sarc_type;
unsigned int dma_buf_sz;
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
index d34388b1ffcc..ace3c3835a9f 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
@@ -2858,6 +2858,7 @@ static int stmmac_hw_setup(struct net_device *dev, bool init_ptp)
struct stmmac_priv *priv = netdev_priv(dev);
u32 rx_cnt = priv->plat->rx_queues_to_use;
u32 tx_cnt = priv->plat->tx_queues_to_use;
+ bool sph_en;
u32 chan;
int ret;
@@ -2952,10 +2953,10 @@ static int stmmac_hw_setup(struct net_device *dev, bool init_ptp)
}
/* Enable Split Header */
- if (priv->sph && priv->hw->rx_csum) {
- for (chan = 0; chan < rx_cnt; chan++)
- stmmac_enable_sph(priv, priv->ioaddr, 1, chan);
- }
+ sph_en = (priv->hw->rx_csum > 0) && priv->sph;
+ for (chan = 0; chan < rx_cnt; chan++)
+ stmmac_enable_sph(priv, priv->ioaddr, sph_en, chan);
+
/* VLAN Tag Insertion */
if (priv->dma_cap.vlins)
@@ -5697,7 +5698,8 @@ int stmmac_dvr_probe(struct device *device,
if (priv->dma_cap.sphen) {
ndev->hw_features |= NETIF_F_GRO;
- priv->sph = true;
+ priv->sph_cap = true;
+ priv->sph = priv->sph_cap;
dev_info(priv->device, "SPH feature enabled\n");
}
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH net-next 3/6] net: stmmac: arrange Tx tail pointer update to stmmac_flush_tx_descriptors
2021-03-29 13:40 ` Ong Boon Leong
@ 2021-03-29 13:40 ` Ong Boon Leong
-1 siblings, 0 replies; 15+ messages in thread
From: Ong Boon Leong @ 2021-03-29 13:40 UTC (permalink / raw)
To: Giuseppe Cavallaro, Alexandre Torgue, Jose Abreu,
David S . Miller, Jakub Kicinski, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend
Cc: Maxime Coquelin, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, netdev, linux-stm32, linux-arm-kernel,
linux-kernel, bpf, Ong Boon Leong
This patch organizes TX tail pointer update into a new function called
stmmac_flush_tx_descriptors() so that we can reuse it in stmmac_xmit(),
stmmac_tso_xmit() and up-coming XDP implementation.
Signed-off-by: Ong Boon Leong <boon.leong.ong@intel.com>
---
.../net/ethernet/stmicro/stmmac/stmmac_main.c | 52 +++++++++----------
1 file changed, 24 insertions(+), 28 deletions(-)
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
index ace3c3835a9f..18578239b438 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
@@ -3507,6 +3507,28 @@ static void stmmac_tso_allocator(struct stmmac_priv *priv, dma_addr_t des,
}
}
+static void stmmac_flush_tx_descriptors(struct stmmac_priv *priv, int queue)
+{
+ struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
+ int desc_size;
+
+ if (likely(priv->extend_desc))
+ desc_size = sizeof(struct dma_extended_desc);
+ else if (tx_q->tbs & STMMAC_TBS_AVAIL)
+ desc_size = sizeof(struct dma_edesc);
+ else
+ desc_size = sizeof(struct dma_desc);
+
+ /* The own bit must be the latest setting done when prepare the
+ * descriptor and then barrier is needed to make sure that
+ * all is coherent before granting the DMA engine.
+ */
+ wmb();
+
+ tx_q->tx_tail_addr = tx_q->dma_tx_phy + (tx_q->cur_tx * desc_size);
+ stmmac_set_tx_tail_ptr(priv, priv->ioaddr, tx_q->tx_tail_addr, queue);
+}
+
/**
* stmmac_tso_xmit - Tx entry point of the driver for oversized frames (TSO)
* @skb : the socket buffer
@@ -3739,12 +3761,6 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
stmmac_set_tx_owner(priv, mss_desc);
}
- /* The own bit must be the latest setting done when prepare the
- * descriptor and then barrier is needed to make sure that
- * all is coherent before granting the DMA engine.
- */
- wmb();
-
if (netif_msg_pktdata(priv)) {
pr_info("%s: curr=%d dirty=%d f=%d, e=%d, f_p=%p, nfrags %d\n",
__func__, tx_q->cur_tx, tx_q->dirty_tx, first_entry,
@@ -3755,13 +3771,7 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
netdev_tx_sent_queue(netdev_get_tx_queue(dev, queue), skb->len);
- if (tx_q->tbs & STMMAC_TBS_AVAIL)
- desc_size = sizeof(struct dma_edesc);
- else
- desc_size = sizeof(struct dma_desc);
-
- tx_q->tx_tail_addr = tx_q->dma_tx_phy + (tx_q->cur_tx * desc_size);
- stmmac_set_tx_tail_ptr(priv, priv->ioaddr, tx_q->tx_tail_addr, queue);
+ stmmac_flush_tx_descriptors(priv, queue);
stmmac_tx_timer_arm(priv, queue);
return NETDEV_TX_OK;
@@ -3996,25 +4006,11 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
stmmac_set_tx_owner(priv, first);
- /* The own bit must be the latest setting done when prepare the
- * descriptor and then barrier is needed to make sure that
- * all is coherent before granting the DMA engine.
- */
- wmb();
-
netdev_tx_sent_queue(netdev_get_tx_queue(dev, queue), skb->len);
stmmac_enable_dma_transmission(priv, priv->ioaddr);
- if (likely(priv->extend_desc))
- desc_size = sizeof(struct dma_extended_desc);
- else if (tx_q->tbs & STMMAC_TBS_AVAIL)
- desc_size = sizeof(struct dma_edesc);
- else
- desc_size = sizeof(struct dma_desc);
-
- tx_q->tx_tail_addr = tx_q->dma_tx_phy + (tx_q->cur_tx * desc_size);
- stmmac_set_tx_tail_ptr(priv, priv->ioaddr, tx_q->tx_tail_addr, queue);
+ stmmac_flush_tx_descriptors(priv, queue);
stmmac_tx_timer_arm(priv, queue);
return NETDEV_TX_OK;
--
2.25.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH net-next 3/6] net: stmmac: arrange Tx tail pointer update to stmmac_flush_tx_descriptors
@ 2021-03-29 13:40 ` Ong Boon Leong
0 siblings, 0 replies; 15+ messages in thread
From: Ong Boon Leong @ 2021-03-29 13:40 UTC (permalink / raw)
To: Giuseppe Cavallaro, Alexandre Torgue, Jose Abreu,
David S . Miller, Jakub Kicinski, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend
Cc: Maxime Coquelin, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, netdev, linux-stm32, linux-arm-kernel,
linux-kernel, bpf, Ong Boon Leong
This patch organizes TX tail pointer update into a new function called
stmmac_flush_tx_descriptors() so that we can reuse it in stmmac_xmit(),
stmmac_tso_xmit() and up-coming XDP implementation.
Signed-off-by: Ong Boon Leong <boon.leong.ong@intel.com>
---
.../net/ethernet/stmicro/stmmac/stmmac_main.c | 52 +++++++++----------
1 file changed, 24 insertions(+), 28 deletions(-)
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
index ace3c3835a9f..18578239b438 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
@@ -3507,6 +3507,28 @@ static void stmmac_tso_allocator(struct stmmac_priv *priv, dma_addr_t des,
}
}
+static void stmmac_flush_tx_descriptors(struct stmmac_priv *priv, int queue)
+{
+ struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
+ int desc_size;
+
+ if (likely(priv->extend_desc))
+ desc_size = sizeof(struct dma_extended_desc);
+ else if (tx_q->tbs & STMMAC_TBS_AVAIL)
+ desc_size = sizeof(struct dma_edesc);
+ else
+ desc_size = sizeof(struct dma_desc);
+
+ /* The own bit must be the latest setting done when prepare the
+ * descriptor and then barrier is needed to make sure that
+ * all is coherent before granting the DMA engine.
+ */
+ wmb();
+
+ tx_q->tx_tail_addr = tx_q->dma_tx_phy + (tx_q->cur_tx * desc_size);
+ stmmac_set_tx_tail_ptr(priv, priv->ioaddr, tx_q->tx_tail_addr, queue);
+}
+
/**
* stmmac_tso_xmit - Tx entry point of the driver for oversized frames (TSO)
* @skb : the socket buffer
@@ -3739,12 +3761,6 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
stmmac_set_tx_owner(priv, mss_desc);
}
- /* The own bit must be the latest setting done when prepare the
- * descriptor and then barrier is needed to make sure that
- * all is coherent before granting the DMA engine.
- */
- wmb();
-
if (netif_msg_pktdata(priv)) {
pr_info("%s: curr=%d dirty=%d f=%d, e=%d, f_p=%p, nfrags %d\n",
__func__, tx_q->cur_tx, tx_q->dirty_tx, first_entry,
@@ -3755,13 +3771,7 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
netdev_tx_sent_queue(netdev_get_tx_queue(dev, queue), skb->len);
- if (tx_q->tbs & STMMAC_TBS_AVAIL)
- desc_size = sizeof(struct dma_edesc);
- else
- desc_size = sizeof(struct dma_desc);
-
- tx_q->tx_tail_addr = tx_q->dma_tx_phy + (tx_q->cur_tx * desc_size);
- stmmac_set_tx_tail_ptr(priv, priv->ioaddr, tx_q->tx_tail_addr, queue);
+ stmmac_flush_tx_descriptors(priv, queue);
stmmac_tx_timer_arm(priv, queue);
return NETDEV_TX_OK;
@@ -3996,25 +4006,11 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
stmmac_set_tx_owner(priv, first);
- /* The own bit must be the latest setting done when prepare the
- * descriptor and then barrier is needed to make sure that
- * all is coherent before granting the DMA engine.
- */
- wmb();
-
netdev_tx_sent_queue(netdev_get_tx_queue(dev, queue), skb->len);
stmmac_enable_dma_transmission(priv, priv->ioaddr);
- if (likely(priv->extend_desc))
- desc_size = sizeof(struct dma_extended_desc);
- else if (tx_q->tbs & STMMAC_TBS_AVAIL)
- desc_size = sizeof(struct dma_edesc);
- else
- desc_size = sizeof(struct dma_desc);
-
- tx_q->tx_tail_addr = tx_q->dma_tx_phy + (tx_q->cur_tx * desc_size);
- stmmac_set_tx_tail_ptr(priv, priv->ioaddr, tx_q->tx_tail_addr, queue);
+ stmmac_flush_tx_descriptors(priv, queue);
stmmac_tx_timer_arm(priv, queue);
return NETDEV_TX_OK;
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH net-next 4/6] net: stmmac: Add initial XDP support
2021-03-29 13:40 ` Ong Boon Leong
@ 2021-03-29 13:40 ` Ong Boon Leong
-1 siblings, 0 replies; 15+ messages in thread
From: Ong Boon Leong @ 2021-03-29 13:40 UTC (permalink / raw)
To: Giuseppe Cavallaro, Alexandre Torgue, Jose Abreu,
David S . Miller, Jakub Kicinski, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend
Cc: Maxime Coquelin, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, netdev, linux-stm32, linux-arm-kernel,
linux-kernel, bpf, Ong Boon Leong
This patch adds the initial XDP support to stmmac driver. It supports
XDP_PASS, XDP_DROP and XDP_ABORTED actions. Upcoming patches will add
support for XDP_TX and XDP_REDIRECT.
To support XDP headroom, this patch adds page_offset into RX buffer and
change the dma_sync_single_for_device|cpu(). The DMA address used for
RX operation are changed to take into page_offset too. As page_pool
can handle dma_sync_single_for_device() on behalf of driver with
PP_FLAG_DMA_SYNC_DEV flag, we skip doing that in stmmac driver.
Current stmmac driver supports split header support (SPH) in RX but
the flexibility of splitting header and payload at different position
makes it very complex to be supported for XDP processing. In addition,
jumbo frame is not supported in XDP to keep the initial codes simple.
This patch has been tested with the sample app "xdp1" located in
samples/bpf directory for both SKB and Native (XDP) mode. The burst
traffic generated using pktgen_sample03_burst_single_flow.sh in
samples/pktgen directory.
Signed-off-by: Ong Boon Leong <boon.leong.ong@intel.com>
---
drivers/net/ethernet/stmicro/stmmac/Makefile | 1 +
drivers/net/ethernet/stmicro/stmmac/stmmac.h | 21 ++-
.../net/ethernet/stmicro/stmmac/stmmac_main.c | 146 +++++++++++++++---
.../net/ethernet/stmicro/stmmac/stmmac_xdp.c | 40 +++++
.../net/ethernet/stmicro/stmmac/stmmac_xdp.h | 12 ++
5 files changed, 195 insertions(+), 25 deletions(-)
create mode 100644 drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.c
create mode 100644 drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.h
diff --git a/drivers/net/ethernet/stmicro/stmmac/Makefile b/drivers/net/ethernet/stmicro/stmmac/Makefile
index 366740ab9c5a..f2e478b884b0 100644
--- a/drivers/net/ethernet/stmicro/stmmac/Makefile
+++ b/drivers/net/ethernet/stmicro/stmmac/Makefile
@@ -6,6 +6,7 @@ stmmac-objs:= stmmac_main.o stmmac_ethtool.o stmmac_mdio.o ring_mode.o \
mmc_core.o stmmac_hwtstamp.o stmmac_ptp.o dwmac4_descs.o \
dwmac4_dma.o dwmac4_lib.o dwmac4_core.o dwmac5.o hwif.o \
stmmac_tc.o dwxgmac2_core.o dwxgmac2_dma.o dwxgmac2_descs.o \
+ stmmac_xdp.o \
$(stmmac-y)
stmmac-$(CONFIG_STMMAC_SELFTESTS) += stmmac_selftests.o
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac.h b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
index e293423f98c3..e72224c8fbac 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac.h
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
@@ -68,8 +68,9 @@ struct stmmac_tx_queue {
struct stmmac_rx_buffer {
struct page *page;
- struct page *sec_page;
dma_addr_t addr;
+ __u32 page_offset;
+ struct page *sec_page;
dma_addr_t sec_addr;
};
@@ -269,6 +270,9 @@ struct stmmac_priv {
/* Receive Side Scaling */
struct stmmac_rss rss;
+
+ /* XDP BPF Program */
+ struct bpf_prog *xdp_prog;
};
enum stmmac_state {
@@ -285,6 +289,8 @@ void stmmac_set_ethtool_ops(struct net_device *netdev);
void stmmac_ptp_register(struct stmmac_priv *priv);
void stmmac_ptp_unregister(struct stmmac_priv *priv);
+int stmmac_open(struct net_device *dev);
+int stmmac_release(struct net_device *dev);
int stmmac_resume(struct device *dev);
int stmmac_suspend(struct device *dev);
int stmmac_dvr_remove(struct device *dev);
@@ -298,6 +304,19 @@ int stmmac_reinit_ringparam(struct net_device *dev, u32 rx_size, u32 tx_size);
int stmmac_bus_clks_config(struct stmmac_priv *priv, bool enabled);
void stmmac_fpe_handshake(struct stmmac_priv *priv, bool enable);
+static inline bool stmmac_xdp_is_enabled(struct stmmac_priv *priv)
+{
+ return !!priv->xdp_prog;
+}
+
+static inline unsigned int stmmac_rx_offset(struct stmmac_priv *priv)
+{
+ if (stmmac_xdp_is_enabled(priv))
+ return XDP_PACKET_HEADROOM;
+
+ return 0;
+}
+
#if IS_ENABLED(CONFIG_STMMAC_SELFTESTS)
void stmmac_selftest_run(struct net_device *dev,
struct ethtool_test *etest, u64 *buf);
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
index 18578239b438..fd29c36860c9 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
@@ -38,9 +38,11 @@
#include <linux/net_tstamp.h>
#include <linux/phylink.h>
#include <linux/udp.h>
+#include <linux/bpf_trace.h>
#include <net/pkt_cls.h>
#include "stmmac_ptp.h"
#include "stmmac.h"
+#include "stmmac_xdp.h"
#include <linux/reset.h>
#include <linux/of_mdio.h>
#include "dwmac1000.h"
@@ -67,6 +69,9 @@ MODULE_PARM_DESC(phyaddr, "Physical device address");
#define STMMAC_TX_THRESH(x) ((x)->dma_tx_size / 4)
#define STMMAC_RX_THRESH(x) ((x)->dma_rx_size / 4)
+#define STMMAC_XDP_PASS 0
+#define STMMAC_XDP_CONSUMED BIT(0)
+
static int flow_ctrl = FLOW_AUTO;
module_param(flow_ctrl, int, 0644);
MODULE_PARM_DESC(flow_ctrl, "Flow control ability [on/off]");
@@ -1384,6 +1389,7 @@ static int stmmac_init_rx_buffers(struct stmmac_priv *priv, struct dma_desc *p,
buf->page = page_pool_dev_alloc_pages(rx_q->page_pool);
if (!buf->page)
return -ENOMEM;
+ buf->page_offset = stmmac_rx_offset(priv);
if (priv->sph) {
buf->sec_page = page_pool_dev_alloc_pages(rx_q->page_pool);
@@ -1397,7 +1403,8 @@ static int stmmac_init_rx_buffers(struct stmmac_priv *priv, struct dma_desc *p,
stmmac_set_desc_sec_addr(priv, p, buf->sec_addr, false);
}
- buf->addr = page_pool_get_dma_addr(buf->page);
+ buf->addr = page_pool_get_dma_addr(buf->page) + buf->page_offset;
+
stmmac_set_desc_addr(priv, p, buf->addr);
if (priv->dma_buf_sz == BUF_SIZE_16KiB)
stmmac_init_desc3(priv, p);
@@ -1503,7 +1510,8 @@ static void stmmac_reinit_rx_buffers(struct stmmac_priv *priv)
if (!buf->page)
goto err_reinit_rx_buffers;
- buf->addr = page_pool_get_dma_addr(buf->page);
+ buf->addr = page_pool_get_dma_addr(buf->page) +
+ buf->page_offset;
}
if (priv->sph && !buf->sec_page) {
@@ -1821,6 +1829,7 @@ static void free_dma_tx_desc_resources(struct stmmac_priv *priv)
*/
static int alloc_dma_rx_desc_resources(struct stmmac_priv *priv)
{
+ bool xdp_prog = stmmac_xdp_is_enabled(priv);
u32 rx_count = priv->plat->rx_queues_to_use;
int ret = -ENOMEM;
u32 queue;
@@ -1834,13 +1843,15 @@ static int alloc_dma_rx_desc_resources(struct stmmac_priv *priv)
rx_q->queue_index = queue;
rx_q->priv_data = priv;
- pp_params.flags = PP_FLAG_DMA_MAP;
+ pp_params.flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV;
pp_params.pool_size = priv->dma_rx_size;
num_pages = DIV_ROUND_UP(priv->dma_buf_sz, PAGE_SIZE);
pp_params.order = ilog2(num_pages);
pp_params.nid = dev_to_node(priv->device);
pp_params.dev = priv->device;
- pp_params.dma_dir = DMA_FROM_DEVICE;
+ pp_params.dma_dir = xdp_prog ? DMA_BIDIRECTIONAL : DMA_FROM_DEVICE;
+ pp_params.offset = stmmac_rx_offset(priv);
+ pp_params.max_len = STMMAC_MAX_RX_BUF_SIZE(num_pages);
rx_q->page_pool = page_pool_create(&pp_params);
if (IS_ERR(rx_q->page_pool)) {
@@ -3257,7 +3268,7 @@ static int stmmac_request_irq(struct net_device *dev)
* 0 on success and an appropriate (-)ve integer as defined in errno.h
* file on failure.
*/
-static int stmmac_open(struct net_device *dev)
+int stmmac_open(struct net_device *dev)
{
struct stmmac_priv *priv = netdev_priv(dev);
int bfsize = 0;
@@ -3380,7 +3391,7 @@ static void stmmac_fpe_stop_wq(struct stmmac_priv *priv)
* Description:
* This is the stop entry point of the driver.
*/
-static int stmmac_release(struct net_device *dev)
+int stmmac_release(struct net_device *dev)
{
struct stmmac_priv *priv = netdev_priv(dev);
u32 chan;
@@ -3560,10 +3571,10 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
{
struct dma_desc *desc, *first, *mss_desc = NULL;
struct stmmac_priv *priv = netdev_priv(dev);
- int desc_size, tmp_pay_len = 0, first_tx;
int nfrags = skb_shinfo(skb)->nr_frags;
u32 queue = skb_get_queue_mapping(skb);
unsigned int first_entry, tx_packets;
+ int tmp_pay_len = 0, first_tx;
struct stmmac_tx_queue *tx_q;
bool has_vlan, set_ic;
u8 proto_hdr_len, hdr;
@@ -3801,10 +3812,10 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
int nfrags = skb_shinfo(skb)->nr_frags;
int gso = skb_shinfo(skb)->gso_type;
struct dma_edesc *tbs_desc = NULL;
- int entry, desc_size, first_tx;
struct dma_desc *desc, *first;
struct stmmac_tx_queue *tx_q;
bool has_vlan, set_ic;
+ int entry, first_tx;
dma_addr_t des;
tx_q = &priv->tx_queue[queue];
@@ -4080,18 +4091,9 @@ static inline void stmmac_rx_refill(struct stmmac_priv *priv, u32 queue)
break;
buf->sec_addr = page_pool_get_dma_addr(buf->sec_page);
-
- dma_sync_single_for_device(priv->device, buf->sec_addr,
- len, DMA_FROM_DEVICE);
}
- buf->addr = page_pool_get_dma_addr(buf->page);
-
- /* Sync whole allocation to device. This will invalidate old
- * data.
- */
- dma_sync_single_for_device(priv->device, buf->addr, len,
- DMA_FROM_DEVICE);
+ buf->addr = page_pool_get_dma_addr(buf->page) + buf->page_offset;
stmmac_set_desc_addr(priv, p, buf->addr);
if (priv->sph)
@@ -4170,6 +4172,42 @@ static unsigned int stmmac_rx_buf2_len(struct stmmac_priv *priv,
return plen - len;
}
+static struct sk_buff *stmmac_xdp_run_prog(struct stmmac_priv *priv,
+ struct xdp_buff *xdp)
+{
+ struct bpf_prog *prog;
+ int res;
+ u32 act;
+
+ rcu_read_lock();
+
+ prog = READ_ONCE(priv->xdp_prog);
+ if (!prog) {
+ res = STMMAC_XDP_PASS;
+ goto unlock;
+ }
+
+ act = bpf_prog_run_xdp(prog, xdp);
+ switch (act) {
+ case XDP_PASS:
+ res = STMMAC_XDP_PASS;
+ break;
+ default:
+ bpf_warn_invalid_xdp_action(act);
+ fallthrough;
+ case XDP_ABORTED:
+ trace_xdp_exception(priv->dev, prog, act);
+ fallthrough;
+ case XDP_DROP:
+ res = STMMAC_XDP_CONSUMED;
+ break;
+ }
+
+unlock:
+ rcu_read_unlock();
+ return ERR_PTR(-res);
+}
+
/**
* stmmac_rx - manage the receive process
* @priv: driver private structure
@@ -4185,8 +4223,14 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
unsigned int count = 0, error = 0, len = 0;
int status = 0, coe = priv->hw->rx_csum;
unsigned int next_entry = rx_q->cur_rx;
+ enum dma_data_direction dma_dir;
unsigned int desc_size;
struct sk_buff *skb = NULL;
+ struct xdp_buff xdp;
+ int buf_sz;
+
+ dma_dir = page_pool_get_dma_dir(rx_q->page_pool);
+ buf_sz = DIV_ROUND_UP(priv->dma_buf_sz, PAGE_SIZE) * PAGE_SIZE;
if (netif_msg_rx_status(priv)) {
void *rx_head;
@@ -4303,6 +4347,42 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
len -= ETH_FCS_LEN;
}
+ if (!skb) {
+ dma_sync_single_for_cpu(priv->device, buf->addr,
+ buf1_len, dma_dir);
+
+ xdp.data = page_address(buf->page) + buf->page_offset;
+ xdp.data_end = xdp.data + len;
+ xdp.data_hard_start = page_address(buf->page);
+ xdp_set_data_meta_invalid(&xdp);
+ xdp.frame_sz = buf_sz;
+
+ skb = stmmac_xdp_run_prog(priv, &xdp);
+
+ /* For Not XDP_PASS verdict */
+ if (IS_ERR(skb)) {
+ unsigned int xdp_res = -PTR_ERR(skb);
+
+ if (xdp_res & STMMAC_XDP_CONSUMED) {
+ page_pool_recycle_direct(rx_q->page_pool,
+ buf->page);
+ buf->page = NULL;
+ priv->dev->stats.rx_dropped++;
+
+ /* Clear skb as it was set as
+ * status by XDP program.
+ */
+ skb = NULL;
+
+ if (unlikely((status & rx_not_ls)))
+ goto read_again;
+
+ count++;
+ continue;
+ }
+ }
+ }
+
if (!skb) {
skb = napi_alloc_skb(&ch->rx_napi, buf1_len);
if (!skb) {
@@ -4311,9 +4391,8 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
goto drain_data;
}
- dma_sync_single_for_cpu(priv->device, buf->addr,
- buf1_len, DMA_FROM_DEVICE);
- skb_copy_to_linear_data(skb, page_address(buf->page),
+ skb_copy_to_linear_data(skb, page_address(buf->page) +
+ buf->page_offset,
buf1_len);
skb_put(skb, buf1_len);
@@ -4322,9 +4401,9 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
buf->page = NULL;
} else if (buf1_len) {
dma_sync_single_for_cpu(priv->device, buf->addr,
- buf1_len, DMA_FROM_DEVICE);
+ buf1_len, dma_dir);
skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags,
- buf->page, 0, buf1_len,
+ buf->page, buf->page_offset, buf1_len,
priv->dma_buf_sz);
/* Data payload appended into SKB */
@@ -4334,7 +4413,7 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
if (buf2_len) {
dma_sync_single_for_cpu(priv->device, buf->sec_addr,
- buf2_len, DMA_FROM_DEVICE);
+ buf2_len, dma_dir);
skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags,
buf->sec_page, 0, buf2_len,
priv->dma_buf_sz);
@@ -4492,6 +4571,11 @@ static int stmmac_change_mtu(struct net_device *dev, int new_mtu)
return -EBUSY;
}
+ if (stmmac_xdp_is_enabled(priv) && new_mtu > ETH_DATA_LEN) {
+ netdev_dbg(priv->dev, "Jumbo frames not supported for XDP\n");
+ return -EINVAL;
+ }
+
new_mtu = STMMAC_ALIGN(new_mtu);
/* If condition true, FIFO is too small or MTU too large */
@@ -4553,6 +4637,7 @@ static int stmmac_set_features(struct net_device *netdev,
stmmac_rx_ipc(priv, priv->hw);
sph_en = (priv->hw->rx_csum > 0) && priv->sph;
+
for (chan = 0; chan < priv->plat->rx_queues_to_use; chan++)
stmmac_enable_sph(priv, priv->ioaddr, sph_en, chan);
@@ -5288,6 +5373,18 @@ static int stmmac_vlan_rx_kill_vid(struct net_device *ndev, __be16 proto, u16 vi
return ret;
}
+static int stmmac_bpf(struct net_device *dev, struct netdev_bpf *bpf)
+{
+ struct stmmac_priv *priv = netdev_priv(dev);
+
+ switch (bpf->command) {
+ case XDP_SETUP_PROG:
+ return stmmac_xdp_set_prog(priv, bpf->prog, bpf->extack);
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+
static const struct net_device_ops stmmac_netdev_ops = {
.ndo_open = stmmac_open,
.ndo_start_xmit = stmmac_xmit,
@@ -5306,6 +5403,7 @@ static const struct net_device_ops stmmac_netdev_ops = {
.ndo_set_mac_address = stmmac_set_mac_address,
.ndo_vlan_rx_add_vid = stmmac_vlan_rx_add_vid,
.ndo_vlan_rx_kill_vid = stmmac_vlan_rx_kill_vid,
+ .ndo_bpf = stmmac_bpf,
};
static void stmmac_reset_subtask(struct stmmac_priv *priv)
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.c
new file mode 100644
index 000000000000..bf38d231860b
--- /dev/null
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.c
@@ -0,0 +1,40 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2021, Intel Corporation. */
+
+#include "stmmac.h"
+#include "stmmac_xdp.h"
+
+int stmmac_xdp_set_prog(struct stmmac_priv *priv, struct bpf_prog *prog,
+ struct netlink_ext_ack *extack)
+{
+ struct net_device *dev = priv->dev;
+ struct bpf_prog *old_prog;
+ bool need_update;
+ bool if_running;
+
+ if_running = netif_running(dev);
+
+ if (prog && dev->mtu > ETH_DATA_LEN) {
+ /* For now, the driver doesn't support XDP functionality with
+ * jumbo frames so we return error.
+ */
+ NL_SET_ERR_MSG_MOD(extack, "Jumbo frames not supported");
+ return -EOPNOTSUPP;
+ }
+
+ need_update = !!priv->xdp_prog != !!prog;
+ if (if_running && need_update)
+ stmmac_release(dev);
+
+ old_prog = xchg(&priv->xdp_prog, prog);
+ if (old_prog)
+ bpf_prog_put(old_prog);
+
+ /* Disable RX SPH for XDP operation */
+ priv->sph = priv->sph_cap && !stmmac_xdp_is_enabled(priv);
+
+ if (if_running && need_update)
+ stmmac_open(dev);
+
+ return 0;
+}
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.h b/drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.h
new file mode 100644
index 000000000000..93948569d92a
--- /dev/null
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2021, Intel Corporation. */
+
+#ifndef _STMMAC_XDP_H_
+#define _STMMAC_XDP_H_
+
+#define STMMAC_MAX_RX_BUF_SIZE(num) (((num) * PAGE_SIZE) - XDP_PACKET_HEADROOM)
+
+int stmmac_xdp_set_prog(struct stmmac_priv *priv, struct bpf_prog *prog,
+ struct netlink_ext_ack *extack);
+
+#endif /* _STMMAC_XDP_H_ */
--
2.25.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH net-next 4/6] net: stmmac: Add initial XDP support
@ 2021-03-29 13:40 ` Ong Boon Leong
0 siblings, 0 replies; 15+ messages in thread
From: Ong Boon Leong @ 2021-03-29 13:40 UTC (permalink / raw)
To: Giuseppe Cavallaro, Alexandre Torgue, Jose Abreu,
David S . Miller, Jakub Kicinski, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend
Cc: Maxime Coquelin, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, netdev, linux-stm32, linux-arm-kernel,
linux-kernel, bpf, Ong Boon Leong
This patch adds the initial XDP support to stmmac driver. It supports
XDP_PASS, XDP_DROP and XDP_ABORTED actions. Upcoming patches will add
support for XDP_TX and XDP_REDIRECT.
To support XDP headroom, this patch adds page_offset into RX buffer and
change the dma_sync_single_for_device|cpu(). The DMA address used for
RX operation are changed to take into page_offset too. As page_pool
can handle dma_sync_single_for_device() on behalf of driver with
PP_FLAG_DMA_SYNC_DEV flag, we skip doing that in stmmac driver.
Current stmmac driver supports split header support (SPH) in RX but
the flexibility of splitting header and payload at different position
makes it very complex to be supported for XDP processing. In addition,
jumbo frame is not supported in XDP to keep the initial codes simple.
This patch has been tested with the sample app "xdp1" located in
samples/bpf directory for both SKB and Native (XDP) mode. The burst
traffic generated using pktgen_sample03_burst_single_flow.sh in
samples/pktgen directory.
Signed-off-by: Ong Boon Leong <boon.leong.ong@intel.com>
---
drivers/net/ethernet/stmicro/stmmac/Makefile | 1 +
drivers/net/ethernet/stmicro/stmmac/stmmac.h | 21 ++-
.../net/ethernet/stmicro/stmmac/stmmac_main.c | 146 +++++++++++++++---
.../net/ethernet/stmicro/stmmac/stmmac_xdp.c | 40 +++++
.../net/ethernet/stmicro/stmmac/stmmac_xdp.h | 12 ++
5 files changed, 195 insertions(+), 25 deletions(-)
create mode 100644 drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.c
create mode 100644 drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.h
diff --git a/drivers/net/ethernet/stmicro/stmmac/Makefile b/drivers/net/ethernet/stmicro/stmmac/Makefile
index 366740ab9c5a..f2e478b884b0 100644
--- a/drivers/net/ethernet/stmicro/stmmac/Makefile
+++ b/drivers/net/ethernet/stmicro/stmmac/Makefile
@@ -6,6 +6,7 @@ stmmac-objs:= stmmac_main.o stmmac_ethtool.o stmmac_mdio.o ring_mode.o \
mmc_core.o stmmac_hwtstamp.o stmmac_ptp.o dwmac4_descs.o \
dwmac4_dma.o dwmac4_lib.o dwmac4_core.o dwmac5.o hwif.o \
stmmac_tc.o dwxgmac2_core.o dwxgmac2_dma.o dwxgmac2_descs.o \
+ stmmac_xdp.o \
$(stmmac-y)
stmmac-$(CONFIG_STMMAC_SELFTESTS) += stmmac_selftests.o
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac.h b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
index e293423f98c3..e72224c8fbac 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac.h
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
@@ -68,8 +68,9 @@ struct stmmac_tx_queue {
struct stmmac_rx_buffer {
struct page *page;
- struct page *sec_page;
dma_addr_t addr;
+ __u32 page_offset;
+ struct page *sec_page;
dma_addr_t sec_addr;
};
@@ -269,6 +270,9 @@ struct stmmac_priv {
/* Receive Side Scaling */
struct stmmac_rss rss;
+
+ /* XDP BPF Program */
+ struct bpf_prog *xdp_prog;
};
enum stmmac_state {
@@ -285,6 +289,8 @@ void stmmac_set_ethtool_ops(struct net_device *netdev);
void stmmac_ptp_register(struct stmmac_priv *priv);
void stmmac_ptp_unregister(struct stmmac_priv *priv);
+int stmmac_open(struct net_device *dev);
+int stmmac_release(struct net_device *dev);
int stmmac_resume(struct device *dev);
int stmmac_suspend(struct device *dev);
int stmmac_dvr_remove(struct device *dev);
@@ -298,6 +304,19 @@ int stmmac_reinit_ringparam(struct net_device *dev, u32 rx_size, u32 tx_size);
int stmmac_bus_clks_config(struct stmmac_priv *priv, bool enabled);
void stmmac_fpe_handshake(struct stmmac_priv *priv, bool enable);
+static inline bool stmmac_xdp_is_enabled(struct stmmac_priv *priv)
+{
+ return !!priv->xdp_prog;
+}
+
+static inline unsigned int stmmac_rx_offset(struct stmmac_priv *priv)
+{
+ if (stmmac_xdp_is_enabled(priv))
+ return XDP_PACKET_HEADROOM;
+
+ return 0;
+}
+
#if IS_ENABLED(CONFIG_STMMAC_SELFTESTS)
void stmmac_selftest_run(struct net_device *dev,
struct ethtool_test *etest, u64 *buf);
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
index 18578239b438..fd29c36860c9 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
@@ -38,9 +38,11 @@
#include <linux/net_tstamp.h>
#include <linux/phylink.h>
#include <linux/udp.h>
+#include <linux/bpf_trace.h>
#include <net/pkt_cls.h>
#include "stmmac_ptp.h"
#include "stmmac.h"
+#include "stmmac_xdp.h"
#include <linux/reset.h>
#include <linux/of_mdio.h>
#include "dwmac1000.h"
@@ -67,6 +69,9 @@ MODULE_PARM_DESC(phyaddr, "Physical device address");
#define STMMAC_TX_THRESH(x) ((x)->dma_tx_size / 4)
#define STMMAC_RX_THRESH(x) ((x)->dma_rx_size / 4)
+#define STMMAC_XDP_PASS 0
+#define STMMAC_XDP_CONSUMED BIT(0)
+
static int flow_ctrl = FLOW_AUTO;
module_param(flow_ctrl, int, 0644);
MODULE_PARM_DESC(flow_ctrl, "Flow control ability [on/off]");
@@ -1384,6 +1389,7 @@ static int stmmac_init_rx_buffers(struct stmmac_priv *priv, struct dma_desc *p,
buf->page = page_pool_dev_alloc_pages(rx_q->page_pool);
if (!buf->page)
return -ENOMEM;
+ buf->page_offset = stmmac_rx_offset(priv);
if (priv->sph) {
buf->sec_page = page_pool_dev_alloc_pages(rx_q->page_pool);
@@ -1397,7 +1403,8 @@ static int stmmac_init_rx_buffers(struct stmmac_priv *priv, struct dma_desc *p,
stmmac_set_desc_sec_addr(priv, p, buf->sec_addr, false);
}
- buf->addr = page_pool_get_dma_addr(buf->page);
+ buf->addr = page_pool_get_dma_addr(buf->page) + buf->page_offset;
+
stmmac_set_desc_addr(priv, p, buf->addr);
if (priv->dma_buf_sz == BUF_SIZE_16KiB)
stmmac_init_desc3(priv, p);
@@ -1503,7 +1510,8 @@ static void stmmac_reinit_rx_buffers(struct stmmac_priv *priv)
if (!buf->page)
goto err_reinit_rx_buffers;
- buf->addr = page_pool_get_dma_addr(buf->page);
+ buf->addr = page_pool_get_dma_addr(buf->page) +
+ buf->page_offset;
}
if (priv->sph && !buf->sec_page) {
@@ -1821,6 +1829,7 @@ static void free_dma_tx_desc_resources(struct stmmac_priv *priv)
*/
static int alloc_dma_rx_desc_resources(struct stmmac_priv *priv)
{
+ bool xdp_prog = stmmac_xdp_is_enabled(priv);
u32 rx_count = priv->plat->rx_queues_to_use;
int ret = -ENOMEM;
u32 queue;
@@ -1834,13 +1843,15 @@ static int alloc_dma_rx_desc_resources(struct stmmac_priv *priv)
rx_q->queue_index = queue;
rx_q->priv_data = priv;
- pp_params.flags = PP_FLAG_DMA_MAP;
+ pp_params.flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV;
pp_params.pool_size = priv->dma_rx_size;
num_pages = DIV_ROUND_UP(priv->dma_buf_sz, PAGE_SIZE);
pp_params.order = ilog2(num_pages);
pp_params.nid = dev_to_node(priv->device);
pp_params.dev = priv->device;
- pp_params.dma_dir = DMA_FROM_DEVICE;
+ pp_params.dma_dir = xdp_prog ? DMA_BIDIRECTIONAL : DMA_FROM_DEVICE;
+ pp_params.offset = stmmac_rx_offset(priv);
+ pp_params.max_len = STMMAC_MAX_RX_BUF_SIZE(num_pages);
rx_q->page_pool = page_pool_create(&pp_params);
if (IS_ERR(rx_q->page_pool)) {
@@ -3257,7 +3268,7 @@ static int stmmac_request_irq(struct net_device *dev)
* 0 on success and an appropriate (-)ve integer as defined in errno.h
* file on failure.
*/
-static int stmmac_open(struct net_device *dev)
+int stmmac_open(struct net_device *dev)
{
struct stmmac_priv *priv = netdev_priv(dev);
int bfsize = 0;
@@ -3380,7 +3391,7 @@ static void stmmac_fpe_stop_wq(struct stmmac_priv *priv)
* Description:
* This is the stop entry point of the driver.
*/
-static int stmmac_release(struct net_device *dev)
+int stmmac_release(struct net_device *dev)
{
struct stmmac_priv *priv = netdev_priv(dev);
u32 chan;
@@ -3560,10 +3571,10 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
{
struct dma_desc *desc, *first, *mss_desc = NULL;
struct stmmac_priv *priv = netdev_priv(dev);
- int desc_size, tmp_pay_len = 0, first_tx;
int nfrags = skb_shinfo(skb)->nr_frags;
u32 queue = skb_get_queue_mapping(skb);
unsigned int first_entry, tx_packets;
+ int tmp_pay_len = 0, first_tx;
struct stmmac_tx_queue *tx_q;
bool has_vlan, set_ic;
u8 proto_hdr_len, hdr;
@@ -3801,10 +3812,10 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
int nfrags = skb_shinfo(skb)->nr_frags;
int gso = skb_shinfo(skb)->gso_type;
struct dma_edesc *tbs_desc = NULL;
- int entry, desc_size, first_tx;
struct dma_desc *desc, *first;
struct stmmac_tx_queue *tx_q;
bool has_vlan, set_ic;
+ int entry, first_tx;
dma_addr_t des;
tx_q = &priv->tx_queue[queue];
@@ -4080,18 +4091,9 @@ static inline void stmmac_rx_refill(struct stmmac_priv *priv, u32 queue)
break;
buf->sec_addr = page_pool_get_dma_addr(buf->sec_page);
-
- dma_sync_single_for_device(priv->device, buf->sec_addr,
- len, DMA_FROM_DEVICE);
}
- buf->addr = page_pool_get_dma_addr(buf->page);
-
- /* Sync whole allocation to device. This will invalidate old
- * data.
- */
- dma_sync_single_for_device(priv->device, buf->addr, len,
- DMA_FROM_DEVICE);
+ buf->addr = page_pool_get_dma_addr(buf->page) + buf->page_offset;
stmmac_set_desc_addr(priv, p, buf->addr);
if (priv->sph)
@@ -4170,6 +4172,42 @@ static unsigned int stmmac_rx_buf2_len(struct stmmac_priv *priv,
return plen - len;
}
+static struct sk_buff *stmmac_xdp_run_prog(struct stmmac_priv *priv,
+ struct xdp_buff *xdp)
+{
+ struct bpf_prog *prog;
+ int res;
+ u32 act;
+
+ rcu_read_lock();
+
+ prog = READ_ONCE(priv->xdp_prog);
+ if (!prog) {
+ res = STMMAC_XDP_PASS;
+ goto unlock;
+ }
+
+ act = bpf_prog_run_xdp(prog, xdp);
+ switch (act) {
+ case XDP_PASS:
+ res = STMMAC_XDP_PASS;
+ break;
+ default:
+ bpf_warn_invalid_xdp_action(act);
+ fallthrough;
+ case XDP_ABORTED:
+ trace_xdp_exception(priv->dev, prog, act);
+ fallthrough;
+ case XDP_DROP:
+ res = STMMAC_XDP_CONSUMED;
+ break;
+ }
+
+unlock:
+ rcu_read_unlock();
+ return ERR_PTR(-res);
+}
+
/**
* stmmac_rx - manage the receive process
* @priv: driver private structure
@@ -4185,8 +4223,14 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
unsigned int count = 0, error = 0, len = 0;
int status = 0, coe = priv->hw->rx_csum;
unsigned int next_entry = rx_q->cur_rx;
+ enum dma_data_direction dma_dir;
unsigned int desc_size;
struct sk_buff *skb = NULL;
+ struct xdp_buff xdp;
+ int buf_sz;
+
+ dma_dir = page_pool_get_dma_dir(rx_q->page_pool);
+ buf_sz = DIV_ROUND_UP(priv->dma_buf_sz, PAGE_SIZE) * PAGE_SIZE;
if (netif_msg_rx_status(priv)) {
void *rx_head;
@@ -4303,6 +4347,42 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
len -= ETH_FCS_LEN;
}
+ if (!skb) {
+ dma_sync_single_for_cpu(priv->device, buf->addr,
+ buf1_len, dma_dir);
+
+ xdp.data = page_address(buf->page) + buf->page_offset;
+ xdp.data_end = xdp.data + len;
+ xdp.data_hard_start = page_address(buf->page);
+ xdp_set_data_meta_invalid(&xdp);
+ xdp.frame_sz = buf_sz;
+
+ skb = stmmac_xdp_run_prog(priv, &xdp);
+
+ /* For Not XDP_PASS verdict */
+ if (IS_ERR(skb)) {
+ unsigned int xdp_res = -PTR_ERR(skb);
+
+ if (xdp_res & STMMAC_XDP_CONSUMED) {
+ page_pool_recycle_direct(rx_q->page_pool,
+ buf->page);
+ buf->page = NULL;
+ priv->dev->stats.rx_dropped++;
+
+ /* Clear skb as it was set as
+ * status by XDP program.
+ */
+ skb = NULL;
+
+ if (unlikely((status & rx_not_ls)))
+ goto read_again;
+
+ count++;
+ continue;
+ }
+ }
+ }
+
if (!skb) {
skb = napi_alloc_skb(&ch->rx_napi, buf1_len);
if (!skb) {
@@ -4311,9 +4391,8 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
goto drain_data;
}
- dma_sync_single_for_cpu(priv->device, buf->addr,
- buf1_len, DMA_FROM_DEVICE);
- skb_copy_to_linear_data(skb, page_address(buf->page),
+ skb_copy_to_linear_data(skb, page_address(buf->page) +
+ buf->page_offset,
buf1_len);
skb_put(skb, buf1_len);
@@ -4322,9 +4401,9 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
buf->page = NULL;
} else if (buf1_len) {
dma_sync_single_for_cpu(priv->device, buf->addr,
- buf1_len, DMA_FROM_DEVICE);
+ buf1_len, dma_dir);
skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags,
- buf->page, 0, buf1_len,
+ buf->page, buf->page_offset, buf1_len,
priv->dma_buf_sz);
/* Data payload appended into SKB */
@@ -4334,7 +4413,7 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
if (buf2_len) {
dma_sync_single_for_cpu(priv->device, buf->sec_addr,
- buf2_len, DMA_FROM_DEVICE);
+ buf2_len, dma_dir);
skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags,
buf->sec_page, 0, buf2_len,
priv->dma_buf_sz);
@@ -4492,6 +4571,11 @@ static int stmmac_change_mtu(struct net_device *dev, int new_mtu)
return -EBUSY;
}
+ if (stmmac_xdp_is_enabled(priv) && new_mtu > ETH_DATA_LEN) {
+ netdev_dbg(priv->dev, "Jumbo frames not supported for XDP\n");
+ return -EINVAL;
+ }
+
new_mtu = STMMAC_ALIGN(new_mtu);
/* If condition true, FIFO is too small or MTU too large */
@@ -4553,6 +4637,7 @@ static int stmmac_set_features(struct net_device *netdev,
stmmac_rx_ipc(priv, priv->hw);
sph_en = (priv->hw->rx_csum > 0) && priv->sph;
+
for (chan = 0; chan < priv->plat->rx_queues_to_use; chan++)
stmmac_enable_sph(priv, priv->ioaddr, sph_en, chan);
@@ -5288,6 +5373,18 @@ static int stmmac_vlan_rx_kill_vid(struct net_device *ndev, __be16 proto, u16 vi
return ret;
}
+static int stmmac_bpf(struct net_device *dev, struct netdev_bpf *bpf)
+{
+ struct stmmac_priv *priv = netdev_priv(dev);
+
+ switch (bpf->command) {
+ case XDP_SETUP_PROG:
+ return stmmac_xdp_set_prog(priv, bpf->prog, bpf->extack);
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+
static const struct net_device_ops stmmac_netdev_ops = {
.ndo_open = stmmac_open,
.ndo_start_xmit = stmmac_xmit,
@@ -5306,6 +5403,7 @@ static const struct net_device_ops stmmac_netdev_ops = {
.ndo_set_mac_address = stmmac_set_mac_address,
.ndo_vlan_rx_add_vid = stmmac_vlan_rx_add_vid,
.ndo_vlan_rx_kill_vid = stmmac_vlan_rx_kill_vid,
+ .ndo_bpf = stmmac_bpf,
};
static void stmmac_reset_subtask(struct stmmac_priv *priv)
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.c
new file mode 100644
index 000000000000..bf38d231860b
--- /dev/null
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.c
@@ -0,0 +1,40 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2021, Intel Corporation. */
+
+#include "stmmac.h"
+#include "stmmac_xdp.h"
+
+int stmmac_xdp_set_prog(struct stmmac_priv *priv, struct bpf_prog *prog,
+ struct netlink_ext_ack *extack)
+{
+ struct net_device *dev = priv->dev;
+ struct bpf_prog *old_prog;
+ bool need_update;
+ bool if_running;
+
+ if_running = netif_running(dev);
+
+ if (prog && dev->mtu > ETH_DATA_LEN) {
+ /* For now, the driver doesn't support XDP functionality with
+ * jumbo frames so we return error.
+ */
+ NL_SET_ERR_MSG_MOD(extack, "Jumbo frames not supported");
+ return -EOPNOTSUPP;
+ }
+
+ need_update = !!priv->xdp_prog != !!prog;
+ if (if_running && need_update)
+ stmmac_release(dev);
+
+ old_prog = xchg(&priv->xdp_prog, prog);
+ if (old_prog)
+ bpf_prog_put(old_prog);
+
+ /* Disable RX SPH for XDP operation */
+ priv->sph = priv->sph_cap && !stmmac_xdp_is_enabled(priv);
+
+ if (if_running && need_update)
+ stmmac_open(dev);
+
+ return 0;
+}
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.h b/drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.h
new file mode 100644
index 000000000000..93948569d92a
--- /dev/null
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2021, Intel Corporation. */
+
+#ifndef _STMMAC_XDP_H_
+#define _STMMAC_XDP_H_
+
+#define STMMAC_MAX_RX_BUF_SIZE(num) (((num) * PAGE_SIZE) - XDP_PACKET_HEADROOM)
+
+int stmmac_xdp_set_prog(struct stmmac_priv *priv, struct bpf_prog *prog,
+ struct netlink_ext_ack *extack);
+
+#endif /* _STMMAC_XDP_H_ */
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH net-next 5/6] net: stmmac: Add support for XDP_TX action
2021-03-29 13:40 ` Ong Boon Leong
@ 2021-03-29 13:40 ` Ong Boon Leong
-1 siblings, 0 replies; 15+ messages in thread
From: Ong Boon Leong @ 2021-03-29 13:40 UTC (permalink / raw)
To: Giuseppe Cavallaro, Alexandre Torgue, Jose Abreu,
David S . Miller, Jakub Kicinski, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend
Cc: Maxime Coquelin, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, netdev, linux-stm32, linux-arm-kernel,
linux-kernel, bpf, Ong Boon Leong
This patch adds support for XDP_TX action which enables XDP program to
transmit back received frames.
This patch has been tested with the "xdp2" app located in samples/bpf
dir. The DUT receives burst traffic packet generated using pktgen script
'pktgen_sample03_burst_single_flow.sh'.
Signed-off-by: Ong Boon Leong <boon.leong.ong@intel.com>
---
drivers/net/ethernet/stmicro/stmmac/stmmac.h | 12 +-
.../net/ethernet/stmicro/stmmac/stmmac_main.c | 220 ++++++++++++++++--
2 files changed, 214 insertions(+), 18 deletions(-)
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac.h b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
index e72224c8fbac..a93e22a6be59 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac.h
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
@@ -36,12 +36,18 @@ struct stmmac_resources {
int tx_irq[MTL_MAX_TX_QUEUES];
};
+enum stmmac_txbuf_type {
+ STMMAC_TXBUF_T_SKB,
+ STMMAC_TXBUF_T_XDP_TX,
+};
+
struct stmmac_tx_info {
dma_addr_t buf;
bool map_as_page;
unsigned len;
bool last_segment;
bool is_jumbo;
+ enum stmmac_txbuf_type buf_type;
};
#define STMMAC_TBS_AVAIL BIT(0)
@@ -57,7 +63,10 @@ struct stmmac_tx_queue {
struct dma_extended_desc *dma_etx ____cacheline_aligned_in_smp;
struct dma_edesc *dma_entx;
struct dma_desc *dma_tx;
- struct sk_buff **tx_skbuff;
+ union {
+ struct sk_buff **tx_skbuff;
+ struct xdp_frame **xdpf;
+ };
struct stmmac_tx_info *tx_skbuff_dma;
unsigned int cur_tx;
unsigned int dirty_tx;
@@ -77,6 +86,7 @@ struct stmmac_rx_buffer {
struct stmmac_rx_queue {
u32 rx_count_frames;
u32 queue_index;
+ struct xdp_rxq_info xdp_rxq;
struct page_pool *page_pool;
struct stmmac_rx_buffer *buf_pool;
struct stmmac_priv *priv_data;
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
index fd29c36860c9..b92355561609 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
@@ -71,6 +71,7 @@ MODULE_PARM_DESC(phyaddr, "Physical device address");
#define STMMAC_XDP_PASS 0
#define STMMAC_XDP_CONSUMED BIT(0)
+#define STMMAC_XDP_TX BIT(1)
static int flow_ctrl = FLOW_AUTO;
module_param(flow_ctrl, int, 0644);
@@ -1442,7 +1443,8 @@ static void stmmac_free_tx_buffer(struct stmmac_priv *priv, u32 queue, int i)
{
struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
- if (tx_q->tx_skbuff_dma[i].buf) {
+ if (tx_q->tx_skbuff_dma[i].buf &&
+ tx_q->tx_skbuff_dma[i].buf_type != STMMAC_TXBUF_T_XDP_TX) {
if (tx_q->tx_skbuff_dma[i].map_as_page)
dma_unmap_page(priv->device,
tx_q->tx_skbuff_dma[i].buf,
@@ -1455,12 +1457,20 @@ static void stmmac_free_tx_buffer(struct stmmac_priv *priv, u32 queue, int i)
DMA_TO_DEVICE);
}
- if (tx_q->tx_skbuff[i]) {
+ if (tx_q->xdpf[i] &&
+ tx_q->tx_skbuff_dma[i].buf_type == STMMAC_TXBUF_T_XDP_TX) {
+ xdp_return_frame(tx_q->xdpf[i]);
+ tx_q->xdpf[i] = NULL;
+ }
+
+ if (tx_q->tx_skbuff[i] &&
+ tx_q->tx_skbuff_dma[i].buf_type == STMMAC_TXBUF_T_SKB) {
dev_kfree_skb_any(tx_q->tx_skbuff[i]);
tx_q->tx_skbuff[i] = NULL;
- tx_q->tx_skbuff_dma[i].buf = 0;
- tx_q->tx_skbuff_dma[i].map_as_page = false;
}
+
+ tx_q->tx_skbuff_dma[i].buf = 0;
+ tx_q->tx_skbuff_dma[i].map_as_page = false;
}
/**
@@ -1568,6 +1578,7 @@ static int init_dma_rx_desc_rings(struct net_device *dev, gfp_t flags)
for (queue = 0; queue < rx_count; queue++) {
struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
+ int ret;
netif_dbg(priv, probe, priv->dev,
"(%s) dma_rx_phy=0x%08x\n", __func__,
@@ -1575,6 +1586,14 @@ static int init_dma_rx_desc_rings(struct net_device *dev, gfp_t flags)
stmmac_clear_rx_descriptors(priv, queue);
+ WARN_ON(xdp_rxq_info_reg_mem_model(&rx_q->xdp_rxq,
+ MEM_TYPE_PAGE_POOL,
+ rx_q->page_pool));
+
+ netdev_info(priv->dev,
+ "Register MEM_TYPE_PAGE_POOL RxQ-%d\n",
+ rx_q->queue_index);
+
for (i = 0; i < priv->dma_rx_size; i++) {
struct dma_desc *p;
@@ -1775,6 +1794,9 @@ static void free_dma_rx_desc_resources(struct stmmac_priv *priv)
sizeof(struct dma_extended_desc),
rx_q->dma_erx, rx_q->dma_rx_phy);
+ if (xdp_rxq_info_is_reg(&rx_q->xdp_rxq))
+ xdp_rxq_info_unreg(&rx_q->xdp_rxq);
+
kfree(rx_q->buf_pool);
if (rx_q->page_pool)
page_pool_destroy(rx_q->page_pool);
@@ -1837,8 +1859,10 @@ static int alloc_dma_rx_desc_resources(struct stmmac_priv *priv)
/* RX queues buffers and DMA */
for (queue = 0; queue < rx_count; queue++) {
struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
+ struct stmmac_channel *ch = &priv->channel[queue];
struct page_pool_params pp_params = { 0 };
unsigned int num_pages;
+ int ret;
rx_q->queue_index = queue;
rx_q->priv_data = priv;
@@ -1884,6 +1908,14 @@ static int alloc_dma_rx_desc_resources(struct stmmac_priv *priv)
if (!rx_q->dma_rx)
goto err_dma;
}
+
+ ret = xdp_rxq_info_reg(&rx_q->xdp_rxq, priv->dev,
+ rx_q->queue_index,
+ ch->rx_napi.napi_id);
+ if (ret) {
+ netdev_err(priv->dev, "Failed to register xdp rxq info\n");
+ goto err_dma;
+ }
}
return 0;
@@ -1985,11 +2017,13 @@ static int alloc_dma_desc_resources(struct stmmac_priv *priv)
*/
static void free_dma_desc_resources(struct stmmac_priv *priv)
{
- /* Release the DMA RX socket buffers */
- free_dma_rx_desc_resources(priv);
-
/* Release the DMA TX socket buffers */
free_dma_tx_desc_resources(priv);
+
+ /* Release the DMA RX socket buffers later
+ * to ensure all pending XDP_TX buffers are returned.
+ */
+ free_dma_rx_desc_resources(priv);
}
/**
@@ -2181,10 +2215,22 @@ static int stmmac_tx_clean(struct stmmac_priv *priv, int budget, u32 queue)
entry = tx_q->dirty_tx;
while ((entry != tx_q->cur_tx) && (count < budget)) {
- struct sk_buff *skb = tx_q->tx_skbuff[entry];
+ struct xdp_frame *xdpf;
+ struct sk_buff *skb;
struct dma_desc *p;
int status;
+ if (tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_XDP_TX) {
+ xdpf = tx_q->xdpf[entry];
+ skb = NULL;
+ } else if (tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_SKB) {
+ xdpf = NULL;
+ skb = tx_q->tx_skbuff[entry];
+ } else {
+ xdpf = NULL;
+ skb = NULL;
+ }
+
if (priv->extend_desc)
p = (struct dma_desc *)(tx_q->dma_etx + entry);
else if (tx_q->tbs & STMMAC_TBS_AVAIL)
@@ -2214,10 +2260,12 @@ static int stmmac_tx_clean(struct stmmac_priv *priv, int budget, u32 queue)
priv->dev->stats.tx_packets++;
priv->xstats.tx_pkt_n++;
}
- stmmac_get_tx_hwtstamp(priv, p, skb);
+ if (skb)
+ stmmac_get_tx_hwtstamp(priv, p, skb);
}
- if (likely(tx_q->tx_skbuff_dma[entry].buf)) {
+ if (likely(tx_q->tx_skbuff_dma[entry].buf &&
+ tx_q->tx_skbuff_dma[entry].buf_type != STMMAC_TXBUF_T_XDP_TX)) {
if (tx_q->tx_skbuff_dma[entry].map_as_page)
dma_unmap_page(priv->device,
tx_q->tx_skbuff_dma[entry].buf,
@@ -2238,11 +2286,19 @@ static int stmmac_tx_clean(struct stmmac_priv *priv, int budget, u32 queue)
tx_q->tx_skbuff_dma[entry].last_segment = false;
tx_q->tx_skbuff_dma[entry].is_jumbo = false;
- if (likely(skb != NULL)) {
- pkts_compl++;
- bytes_compl += skb->len;
- dev_consume_skb_any(skb);
- tx_q->tx_skbuff[entry] = NULL;
+ if (xdpf &&
+ tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_XDP_TX) {
+ xdp_return_frame_rx_napi(xdpf);
+ tx_q->xdpf[entry] = NULL;
+ }
+
+ if (tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_SKB) {
+ if (likely(skb)) {
+ pkts_compl++;
+ bytes_compl += skb->len;
+ dev_consume_skb_any(skb);
+ tx_q->tx_skbuff[entry] = NULL;
+ }
}
stmmac_release_tx_desc(priv, p, priv->mode);
@@ -3656,6 +3712,8 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
tx_q->tx_skbuff_dma[first_entry].buf = des;
tx_q->tx_skbuff_dma[first_entry].len = skb_headlen(skb);
+ tx_q->tx_skbuff_dma[first_entry].map_as_page = false;
+ tx_q->tx_skbuff_dma[first_entry].buf_type = STMMAC_TXBUF_T_SKB;
if (priv->dma_cap.addr64 <= 32) {
first->des0 = cpu_to_le32(des);
@@ -3691,12 +3749,14 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
tx_q->tx_skbuff_dma[tx_q->cur_tx].buf = des;
tx_q->tx_skbuff_dma[tx_q->cur_tx].len = skb_frag_size(frag);
tx_q->tx_skbuff_dma[tx_q->cur_tx].map_as_page = true;
+ tx_q->tx_skbuff_dma[tx_q->cur_tx].buf_type = STMMAC_TXBUF_T_SKB;
}
tx_q->tx_skbuff_dma[tx_q->cur_tx].last_segment = true;
/* Only the last descriptor gets to point to the skb. */
tx_q->tx_skbuff[tx_q->cur_tx] = skb;
+ tx_q->tx_skbuff_dma[tx_q->cur_tx].buf_type = STMMAC_TXBUF_T_SKB;
/* Manage tx mitigation */
tx_packets = (tx_q->cur_tx + 1) - first_tx;
@@ -3903,6 +3963,7 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
tx_q->tx_skbuff_dma[entry].map_as_page = true;
tx_q->tx_skbuff_dma[entry].len = len;
tx_q->tx_skbuff_dma[entry].last_segment = last_segment;
+ tx_q->tx_skbuff_dma[entry].buf_type = STMMAC_TXBUF_T_SKB;
/* Prepare the descriptor and set the own bit too */
stmmac_prepare_tx_desc(priv, desc, 0, len, csum_insertion,
@@ -3911,6 +3972,7 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
/* Only the last descriptor gets to point to the skb. */
tx_q->tx_skbuff[entry] = skb;
+ tx_q->tx_skbuff_dma[entry].buf_type = STMMAC_TXBUF_T_SKB;
/* According to the coalesce parameter the IC bit for the latest
* segment is reset and the timer re-started to clean the tx status.
@@ -3989,6 +4051,8 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
goto dma_map_err;
tx_q->tx_skbuff_dma[first_entry].buf = des;
+ tx_q->tx_skbuff_dma[first_entry].buf_type = STMMAC_TXBUF_T_SKB;
+ tx_q->tx_skbuff_dma[first_entry].map_as_page = false;
stmmac_set_desc_addr(priv, first, des);
@@ -4172,6 +4236,108 @@ static unsigned int stmmac_rx_buf2_len(struct stmmac_priv *priv,
return plen - len;
}
+static int stmmac_xdp_xmit_xdpf(struct stmmac_priv *priv, int queue,
+ struct xdp_frame *xdpf)
+{
+ struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
+ struct page *page = virt_to_page(xdpf->data);
+ unsigned int entry = tx_q->cur_tx;
+ struct dma_desc *tx_desc;
+ dma_addr_t dma_addr;
+ bool set_ic;
+
+ if (stmmac_tx_avail(priv, queue) < STMMAC_TX_THRESH(priv))
+ return STMMAC_XDP_CONSUMED;
+
+ if (likely(priv->extend_desc))
+ tx_desc = (struct dma_desc *)(tx_q->dma_etx + entry);
+ else if (tx_q->tbs & STMMAC_TBS_AVAIL)
+ tx_desc = &tx_q->dma_entx[entry].basic;
+ else
+ tx_desc = tx_q->dma_tx + entry;
+
+ dma_addr = page_pool_get_dma_addr(page) + sizeof(*xdpf) +
+ xdpf->headroom;
+ dma_sync_single_for_device(priv->device, dma_addr,
+ xdpf->len, DMA_BIDIRECTIONAL);
+
+ tx_q->tx_skbuff_dma[entry].buf_type = STMMAC_TXBUF_T_XDP_TX;
+
+ tx_q->tx_skbuff_dma[entry].buf = dma_addr;
+ tx_q->tx_skbuff_dma[entry].map_as_page = false;
+ tx_q->tx_skbuff_dma[entry].len = xdpf->len;
+ tx_q->tx_skbuff_dma[entry].last_segment = true;
+ tx_q->tx_skbuff_dma[entry].is_jumbo = false;
+
+ tx_q->xdpf[entry] = xdpf;
+
+ stmmac_set_desc_addr(priv, tx_desc, dma_addr);
+
+ stmmac_prepare_tx_desc(priv, tx_desc, 1, xdpf->len,
+ true, priv->mode, true, true,
+ xdpf->len);
+
+ tx_q->tx_count_frames++;
+
+ if (tx_q->tx_count_frames % priv->tx_coal_frames[queue] == 0)
+ set_ic = true;
+ else
+ set_ic = false;
+
+ if (set_ic) {
+ tx_q->tx_count_frames = 0;
+ stmmac_set_tx_ic(priv, tx_desc);
+ priv->xstats.tx_set_ic_bit++;
+ }
+
+ stmmac_enable_dma_transmission(priv, priv->ioaddr);
+
+ entry = STMMAC_GET_ENTRY(entry, priv->dma_tx_size);
+ tx_q->cur_tx = entry;
+
+ return STMMAC_XDP_TX;
+}
+
+static int stmmac_xdp_get_tx_queue(struct stmmac_priv *priv,
+ int cpu)
+{
+ int index = cpu;
+
+ if (unlikely(index < 0))
+ index = 0;
+
+ while (index >= priv->plat->tx_queues_to_use)
+ index -= priv->plat->tx_queues_to_use;
+
+ return index;
+}
+
+static int stmmac_xdp_xmit_back(struct stmmac_priv *priv,
+ struct xdp_buff *xdp)
+{
+ struct xdp_frame *xdpf = xdp_convert_buff_to_frame(xdp);
+ int cpu = smp_processor_id();
+ struct netdev_queue *nq;
+ int queue;
+ int res;
+
+ if (unlikely(!xdpf))
+ return -EFAULT;
+
+ queue = stmmac_xdp_get_tx_queue(priv, cpu);
+ nq = netdev_get_tx_queue(priv->dev, queue);
+
+ __netif_tx_lock(nq, cpu);
+ res = stmmac_xdp_xmit_xdpf(priv, queue, xdpf);
+ if (res == STMMAC_XDP_TX) {
+ stmmac_flush_tx_descriptors(priv, queue);
+ stmmac_tx_timer_arm(priv, queue);
+ }
+ __netif_tx_unlock(nq);
+
+ return res;
+}
+
static struct sk_buff *stmmac_xdp_run_prog(struct stmmac_priv *priv,
struct xdp_buff *xdp)
{
@@ -4192,6 +4358,9 @@ static struct sk_buff *stmmac_xdp_run_prog(struct stmmac_priv *priv,
case XDP_PASS:
res = STMMAC_XDP_PASS;
break;
+ case XDP_TX:
+ res = stmmac_xdp_xmit_back(priv, xdp);
+ break;
default:
bpf_warn_invalid_xdp_action(act);
fallthrough;
@@ -4348,6 +4517,8 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
}
if (!skb) {
+ unsigned int pre_len, sync_len;
+
dma_sync_single_for_cpu(priv->device, buf->addr,
buf1_len, dma_dir);
@@ -4356,16 +4527,26 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
xdp.data_hard_start = page_address(buf->page);
xdp_set_data_meta_invalid(&xdp);
xdp.frame_sz = buf_sz;
+ xdp.rxq = &rx_q->xdp_rxq;
+ pre_len = xdp.data_end - xdp.data_hard_start -
+ buf->page_offset;
skb = stmmac_xdp_run_prog(priv, &xdp);
+ /* Due xdp_adjust_tail: DMA sync for_device
+ * cover max len CPU touch
+ */
+ sync_len = xdp.data_end - xdp.data_hard_start -
+ buf->page_offset;
+ sync_len = max(sync_len, pre_len);
/* For Not XDP_PASS verdict */
if (IS_ERR(skb)) {
unsigned int xdp_res = -PTR_ERR(skb);
if (xdp_res & STMMAC_XDP_CONSUMED) {
- page_pool_recycle_direct(rx_q->page_pool,
- buf->page);
+ page_pool_put_page(rx_q->page_pool,
+ virt_to_head_page(xdp.data),
+ sync_len, true);
buf->page = NULL;
priv->dev->stats.rx_dropped++;
@@ -4379,6 +4560,11 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
count++;
continue;
+ } else if (xdp_res & STMMAC_XDP_TX) {
+ buf->page = NULL;
+ skb = NULL;
+ count++;
+ continue;
}
}
}
--
2.25.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH net-next 5/6] net: stmmac: Add support for XDP_TX action
@ 2021-03-29 13:40 ` Ong Boon Leong
0 siblings, 0 replies; 15+ messages in thread
From: Ong Boon Leong @ 2021-03-29 13:40 UTC (permalink / raw)
To: Giuseppe Cavallaro, Alexandre Torgue, Jose Abreu,
David S . Miller, Jakub Kicinski, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend
Cc: Maxime Coquelin, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, netdev, linux-stm32, linux-arm-kernel,
linux-kernel, bpf, Ong Boon Leong
This patch adds support for XDP_TX action which enables XDP program to
transmit back received frames.
This patch has been tested with the "xdp2" app located in samples/bpf
dir. The DUT receives burst traffic packet generated using pktgen script
'pktgen_sample03_burst_single_flow.sh'.
Signed-off-by: Ong Boon Leong <boon.leong.ong@intel.com>
---
drivers/net/ethernet/stmicro/stmmac/stmmac.h | 12 +-
.../net/ethernet/stmicro/stmmac/stmmac_main.c | 220 ++++++++++++++++--
2 files changed, 214 insertions(+), 18 deletions(-)
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac.h b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
index e72224c8fbac..a93e22a6be59 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac.h
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
@@ -36,12 +36,18 @@ struct stmmac_resources {
int tx_irq[MTL_MAX_TX_QUEUES];
};
+enum stmmac_txbuf_type {
+ STMMAC_TXBUF_T_SKB,
+ STMMAC_TXBUF_T_XDP_TX,
+};
+
struct stmmac_tx_info {
dma_addr_t buf;
bool map_as_page;
unsigned len;
bool last_segment;
bool is_jumbo;
+ enum stmmac_txbuf_type buf_type;
};
#define STMMAC_TBS_AVAIL BIT(0)
@@ -57,7 +63,10 @@ struct stmmac_tx_queue {
struct dma_extended_desc *dma_etx ____cacheline_aligned_in_smp;
struct dma_edesc *dma_entx;
struct dma_desc *dma_tx;
- struct sk_buff **tx_skbuff;
+ union {
+ struct sk_buff **tx_skbuff;
+ struct xdp_frame **xdpf;
+ };
struct stmmac_tx_info *tx_skbuff_dma;
unsigned int cur_tx;
unsigned int dirty_tx;
@@ -77,6 +86,7 @@ struct stmmac_rx_buffer {
struct stmmac_rx_queue {
u32 rx_count_frames;
u32 queue_index;
+ struct xdp_rxq_info xdp_rxq;
struct page_pool *page_pool;
struct stmmac_rx_buffer *buf_pool;
struct stmmac_priv *priv_data;
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
index fd29c36860c9..b92355561609 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
@@ -71,6 +71,7 @@ MODULE_PARM_DESC(phyaddr, "Physical device address");
#define STMMAC_XDP_PASS 0
#define STMMAC_XDP_CONSUMED BIT(0)
+#define STMMAC_XDP_TX BIT(1)
static int flow_ctrl = FLOW_AUTO;
module_param(flow_ctrl, int, 0644);
@@ -1442,7 +1443,8 @@ static void stmmac_free_tx_buffer(struct stmmac_priv *priv, u32 queue, int i)
{
struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
- if (tx_q->tx_skbuff_dma[i].buf) {
+ if (tx_q->tx_skbuff_dma[i].buf &&
+ tx_q->tx_skbuff_dma[i].buf_type != STMMAC_TXBUF_T_XDP_TX) {
if (tx_q->tx_skbuff_dma[i].map_as_page)
dma_unmap_page(priv->device,
tx_q->tx_skbuff_dma[i].buf,
@@ -1455,12 +1457,20 @@ static void stmmac_free_tx_buffer(struct stmmac_priv *priv, u32 queue, int i)
DMA_TO_DEVICE);
}
- if (tx_q->tx_skbuff[i]) {
+ if (tx_q->xdpf[i] &&
+ tx_q->tx_skbuff_dma[i].buf_type == STMMAC_TXBUF_T_XDP_TX) {
+ xdp_return_frame(tx_q->xdpf[i]);
+ tx_q->xdpf[i] = NULL;
+ }
+
+ if (tx_q->tx_skbuff[i] &&
+ tx_q->tx_skbuff_dma[i].buf_type == STMMAC_TXBUF_T_SKB) {
dev_kfree_skb_any(tx_q->tx_skbuff[i]);
tx_q->tx_skbuff[i] = NULL;
- tx_q->tx_skbuff_dma[i].buf = 0;
- tx_q->tx_skbuff_dma[i].map_as_page = false;
}
+
+ tx_q->tx_skbuff_dma[i].buf = 0;
+ tx_q->tx_skbuff_dma[i].map_as_page = false;
}
/**
@@ -1568,6 +1578,7 @@ static int init_dma_rx_desc_rings(struct net_device *dev, gfp_t flags)
for (queue = 0; queue < rx_count; queue++) {
struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
+ int ret;
netif_dbg(priv, probe, priv->dev,
"(%s) dma_rx_phy=0x%08x\n", __func__,
@@ -1575,6 +1586,14 @@ static int init_dma_rx_desc_rings(struct net_device *dev, gfp_t flags)
stmmac_clear_rx_descriptors(priv, queue);
+ WARN_ON(xdp_rxq_info_reg_mem_model(&rx_q->xdp_rxq,
+ MEM_TYPE_PAGE_POOL,
+ rx_q->page_pool));
+
+ netdev_info(priv->dev,
+ "Register MEM_TYPE_PAGE_POOL RxQ-%d\n",
+ rx_q->queue_index);
+
for (i = 0; i < priv->dma_rx_size; i++) {
struct dma_desc *p;
@@ -1775,6 +1794,9 @@ static void free_dma_rx_desc_resources(struct stmmac_priv *priv)
sizeof(struct dma_extended_desc),
rx_q->dma_erx, rx_q->dma_rx_phy);
+ if (xdp_rxq_info_is_reg(&rx_q->xdp_rxq))
+ xdp_rxq_info_unreg(&rx_q->xdp_rxq);
+
kfree(rx_q->buf_pool);
if (rx_q->page_pool)
page_pool_destroy(rx_q->page_pool);
@@ -1837,8 +1859,10 @@ static int alloc_dma_rx_desc_resources(struct stmmac_priv *priv)
/* RX queues buffers and DMA */
for (queue = 0; queue < rx_count; queue++) {
struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
+ struct stmmac_channel *ch = &priv->channel[queue];
struct page_pool_params pp_params = { 0 };
unsigned int num_pages;
+ int ret;
rx_q->queue_index = queue;
rx_q->priv_data = priv;
@@ -1884,6 +1908,14 @@ static int alloc_dma_rx_desc_resources(struct stmmac_priv *priv)
if (!rx_q->dma_rx)
goto err_dma;
}
+
+ ret = xdp_rxq_info_reg(&rx_q->xdp_rxq, priv->dev,
+ rx_q->queue_index,
+ ch->rx_napi.napi_id);
+ if (ret) {
+ netdev_err(priv->dev, "Failed to register xdp rxq info\n");
+ goto err_dma;
+ }
}
return 0;
@@ -1985,11 +2017,13 @@ static int alloc_dma_desc_resources(struct stmmac_priv *priv)
*/
static void free_dma_desc_resources(struct stmmac_priv *priv)
{
- /* Release the DMA RX socket buffers */
- free_dma_rx_desc_resources(priv);
-
/* Release the DMA TX socket buffers */
free_dma_tx_desc_resources(priv);
+
+ /* Release the DMA RX socket buffers later
+ * to ensure all pending XDP_TX buffers are returned.
+ */
+ free_dma_rx_desc_resources(priv);
}
/**
@@ -2181,10 +2215,22 @@ static int stmmac_tx_clean(struct stmmac_priv *priv, int budget, u32 queue)
entry = tx_q->dirty_tx;
while ((entry != tx_q->cur_tx) && (count < budget)) {
- struct sk_buff *skb = tx_q->tx_skbuff[entry];
+ struct xdp_frame *xdpf;
+ struct sk_buff *skb;
struct dma_desc *p;
int status;
+ if (tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_XDP_TX) {
+ xdpf = tx_q->xdpf[entry];
+ skb = NULL;
+ } else if (tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_SKB) {
+ xdpf = NULL;
+ skb = tx_q->tx_skbuff[entry];
+ } else {
+ xdpf = NULL;
+ skb = NULL;
+ }
+
if (priv->extend_desc)
p = (struct dma_desc *)(tx_q->dma_etx + entry);
else if (tx_q->tbs & STMMAC_TBS_AVAIL)
@@ -2214,10 +2260,12 @@ static int stmmac_tx_clean(struct stmmac_priv *priv, int budget, u32 queue)
priv->dev->stats.tx_packets++;
priv->xstats.tx_pkt_n++;
}
- stmmac_get_tx_hwtstamp(priv, p, skb);
+ if (skb)
+ stmmac_get_tx_hwtstamp(priv, p, skb);
}
- if (likely(tx_q->tx_skbuff_dma[entry].buf)) {
+ if (likely(tx_q->tx_skbuff_dma[entry].buf &&
+ tx_q->tx_skbuff_dma[entry].buf_type != STMMAC_TXBUF_T_XDP_TX)) {
if (tx_q->tx_skbuff_dma[entry].map_as_page)
dma_unmap_page(priv->device,
tx_q->tx_skbuff_dma[entry].buf,
@@ -2238,11 +2286,19 @@ static int stmmac_tx_clean(struct stmmac_priv *priv, int budget, u32 queue)
tx_q->tx_skbuff_dma[entry].last_segment = false;
tx_q->tx_skbuff_dma[entry].is_jumbo = false;
- if (likely(skb != NULL)) {
- pkts_compl++;
- bytes_compl += skb->len;
- dev_consume_skb_any(skb);
- tx_q->tx_skbuff[entry] = NULL;
+ if (xdpf &&
+ tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_XDP_TX) {
+ xdp_return_frame_rx_napi(xdpf);
+ tx_q->xdpf[entry] = NULL;
+ }
+
+ if (tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_SKB) {
+ if (likely(skb)) {
+ pkts_compl++;
+ bytes_compl += skb->len;
+ dev_consume_skb_any(skb);
+ tx_q->tx_skbuff[entry] = NULL;
+ }
}
stmmac_release_tx_desc(priv, p, priv->mode);
@@ -3656,6 +3712,8 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
tx_q->tx_skbuff_dma[first_entry].buf = des;
tx_q->tx_skbuff_dma[first_entry].len = skb_headlen(skb);
+ tx_q->tx_skbuff_dma[first_entry].map_as_page = false;
+ tx_q->tx_skbuff_dma[first_entry].buf_type = STMMAC_TXBUF_T_SKB;
if (priv->dma_cap.addr64 <= 32) {
first->des0 = cpu_to_le32(des);
@@ -3691,12 +3749,14 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
tx_q->tx_skbuff_dma[tx_q->cur_tx].buf = des;
tx_q->tx_skbuff_dma[tx_q->cur_tx].len = skb_frag_size(frag);
tx_q->tx_skbuff_dma[tx_q->cur_tx].map_as_page = true;
+ tx_q->tx_skbuff_dma[tx_q->cur_tx].buf_type = STMMAC_TXBUF_T_SKB;
}
tx_q->tx_skbuff_dma[tx_q->cur_tx].last_segment = true;
/* Only the last descriptor gets to point to the skb. */
tx_q->tx_skbuff[tx_q->cur_tx] = skb;
+ tx_q->tx_skbuff_dma[tx_q->cur_tx].buf_type = STMMAC_TXBUF_T_SKB;
/* Manage tx mitigation */
tx_packets = (tx_q->cur_tx + 1) - first_tx;
@@ -3903,6 +3963,7 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
tx_q->tx_skbuff_dma[entry].map_as_page = true;
tx_q->tx_skbuff_dma[entry].len = len;
tx_q->tx_skbuff_dma[entry].last_segment = last_segment;
+ tx_q->tx_skbuff_dma[entry].buf_type = STMMAC_TXBUF_T_SKB;
/* Prepare the descriptor and set the own bit too */
stmmac_prepare_tx_desc(priv, desc, 0, len, csum_insertion,
@@ -3911,6 +3972,7 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
/* Only the last descriptor gets to point to the skb. */
tx_q->tx_skbuff[entry] = skb;
+ tx_q->tx_skbuff_dma[entry].buf_type = STMMAC_TXBUF_T_SKB;
/* According to the coalesce parameter the IC bit for the latest
* segment is reset and the timer re-started to clean the tx status.
@@ -3989,6 +4051,8 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
goto dma_map_err;
tx_q->tx_skbuff_dma[first_entry].buf = des;
+ tx_q->tx_skbuff_dma[first_entry].buf_type = STMMAC_TXBUF_T_SKB;
+ tx_q->tx_skbuff_dma[first_entry].map_as_page = false;
stmmac_set_desc_addr(priv, first, des);
@@ -4172,6 +4236,108 @@ static unsigned int stmmac_rx_buf2_len(struct stmmac_priv *priv,
return plen - len;
}
+static int stmmac_xdp_xmit_xdpf(struct stmmac_priv *priv, int queue,
+ struct xdp_frame *xdpf)
+{
+ struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
+ struct page *page = virt_to_page(xdpf->data);
+ unsigned int entry = tx_q->cur_tx;
+ struct dma_desc *tx_desc;
+ dma_addr_t dma_addr;
+ bool set_ic;
+
+ if (stmmac_tx_avail(priv, queue) < STMMAC_TX_THRESH(priv))
+ return STMMAC_XDP_CONSUMED;
+
+ if (likely(priv->extend_desc))
+ tx_desc = (struct dma_desc *)(tx_q->dma_etx + entry);
+ else if (tx_q->tbs & STMMAC_TBS_AVAIL)
+ tx_desc = &tx_q->dma_entx[entry].basic;
+ else
+ tx_desc = tx_q->dma_tx + entry;
+
+ dma_addr = page_pool_get_dma_addr(page) + sizeof(*xdpf) +
+ xdpf->headroom;
+ dma_sync_single_for_device(priv->device, dma_addr,
+ xdpf->len, DMA_BIDIRECTIONAL);
+
+ tx_q->tx_skbuff_dma[entry].buf_type = STMMAC_TXBUF_T_XDP_TX;
+
+ tx_q->tx_skbuff_dma[entry].buf = dma_addr;
+ tx_q->tx_skbuff_dma[entry].map_as_page = false;
+ tx_q->tx_skbuff_dma[entry].len = xdpf->len;
+ tx_q->tx_skbuff_dma[entry].last_segment = true;
+ tx_q->tx_skbuff_dma[entry].is_jumbo = false;
+
+ tx_q->xdpf[entry] = xdpf;
+
+ stmmac_set_desc_addr(priv, tx_desc, dma_addr);
+
+ stmmac_prepare_tx_desc(priv, tx_desc, 1, xdpf->len,
+ true, priv->mode, true, true,
+ xdpf->len);
+
+ tx_q->tx_count_frames++;
+
+ if (tx_q->tx_count_frames % priv->tx_coal_frames[queue] == 0)
+ set_ic = true;
+ else
+ set_ic = false;
+
+ if (set_ic) {
+ tx_q->tx_count_frames = 0;
+ stmmac_set_tx_ic(priv, tx_desc);
+ priv->xstats.tx_set_ic_bit++;
+ }
+
+ stmmac_enable_dma_transmission(priv, priv->ioaddr);
+
+ entry = STMMAC_GET_ENTRY(entry, priv->dma_tx_size);
+ tx_q->cur_tx = entry;
+
+ return STMMAC_XDP_TX;
+}
+
+static int stmmac_xdp_get_tx_queue(struct stmmac_priv *priv,
+ int cpu)
+{
+ int index = cpu;
+
+ if (unlikely(index < 0))
+ index = 0;
+
+ while (index >= priv->plat->tx_queues_to_use)
+ index -= priv->plat->tx_queues_to_use;
+
+ return index;
+}
+
+static int stmmac_xdp_xmit_back(struct stmmac_priv *priv,
+ struct xdp_buff *xdp)
+{
+ struct xdp_frame *xdpf = xdp_convert_buff_to_frame(xdp);
+ int cpu = smp_processor_id();
+ struct netdev_queue *nq;
+ int queue;
+ int res;
+
+ if (unlikely(!xdpf))
+ return -EFAULT;
+
+ queue = stmmac_xdp_get_tx_queue(priv, cpu);
+ nq = netdev_get_tx_queue(priv->dev, queue);
+
+ __netif_tx_lock(nq, cpu);
+ res = stmmac_xdp_xmit_xdpf(priv, queue, xdpf);
+ if (res == STMMAC_XDP_TX) {
+ stmmac_flush_tx_descriptors(priv, queue);
+ stmmac_tx_timer_arm(priv, queue);
+ }
+ __netif_tx_unlock(nq);
+
+ return res;
+}
+
static struct sk_buff *stmmac_xdp_run_prog(struct stmmac_priv *priv,
struct xdp_buff *xdp)
{
@@ -4192,6 +4358,9 @@ static struct sk_buff *stmmac_xdp_run_prog(struct stmmac_priv *priv,
case XDP_PASS:
res = STMMAC_XDP_PASS;
break;
+ case XDP_TX:
+ res = stmmac_xdp_xmit_back(priv, xdp);
+ break;
default:
bpf_warn_invalid_xdp_action(act);
fallthrough;
@@ -4348,6 +4517,8 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
}
if (!skb) {
+ unsigned int pre_len, sync_len;
+
dma_sync_single_for_cpu(priv->device, buf->addr,
buf1_len, dma_dir);
@@ -4356,16 +4527,26 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
xdp.data_hard_start = page_address(buf->page);
xdp_set_data_meta_invalid(&xdp);
xdp.frame_sz = buf_sz;
+ xdp.rxq = &rx_q->xdp_rxq;
+ pre_len = xdp.data_end - xdp.data_hard_start -
+ buf->page_offset;
skb = stmmac_xdp_run_prog(priv, &xdp);
+ /* Due xdp_adjust_tail: DMA sync for_device
+ * cover max len CPU touch
+ */
+ sync_len = xdp.data_end - xdp.data_hard_start -
+ buf->page_offset;
+ sync_len = max(sync_len, pre_len);
/* For Not XDP_PASS verdict */
if (IS_ERR(skb)) {
unsigned int xdp_res = -PTR_ERR(skb);
if (xdp_res & STMMAC_XDP_CONSUMED) {
- page_pool_recycle_direct(rx_q->page_pool,
- buf->page);
+ page_pool_put_page(rx_q->page_pool,
+ virt_to_head_page(xdp.data),
+ sync_len, true);
buf->page = NULL;
priv->dev->stats.rx_dropped++;
@@ -4379,6 +4560,11 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
count++;
continue;
+ } else if (xdp_res & STMMAC_XDP_TX) {
+ buf->page = NULL;
+ skb = NULL;
+ count++;
+ continue;
}
}
}
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH net-next 6/6] net: stmmac: Add support for XDP_REDIRECT action
2021-03-29 13:40 ` Ong Boon Leong
@ 2021-03-29 13:40 ` Ong Boon Leong
-1 siblings, 0 replies; 15+ messages in thread
From: Ong Boon Leong @ 2021-03-29 13:40 UTC (permalink / raw)
To: Giuseppe Cavallaro, Alexandre Torgue, Jose Abreu,
David S . Miller, Jakub Kicinski, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend
Cc: Maxime Coquelin, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, netdev, linux-stm32, linux-arm-kernel,
linux-kernel, bpf, Ong Boon Leong
This patch adds the support of XDP_REDIRECT to another remote cpu for
further action. It also implements ndo_xdp_xmit ops, enabling the driver
to transmit packets forwarded to it by XDP program running on another
interface.
This patch has been tested using "xdp_redirect_cpu" for XDP_REDIRECT
+ drop testing. It also been tested with "xdp_redirect" sample app
which can be used to exercise ndo_xdp_xmit ops. The burst traffics are
generated using pktgen_sample03_burst_single_flow.sh in samples/pktgen
directory.
Signed-off-by: Ong Boon Leong <boon.leong.ong@intel.com>
---
drivers/net/ethernet/stmicro/stmmac/stmmac.h | 1 +
.../net/ethernet/stmicro/stmmac/stmmac_main.c | 96 +++++++++++++++++--
2 files changed, 87 insertions(+), 10 deletions(-)
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac.h b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
index a93e22a6be59..c49debb62b05 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac.h
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
@@ -39,6 +39,7 @@ struct stmmac_resources {
enum stmmac_txbuf_type {
STMMAC_TXBUF_T_SKB,
STMMAC_TXBUF_T_XDP_TX,
+ STMMAC_TXBUF_T_XDP_NDO,
};
struct stmmac_tx_info {
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
index b92355561609..3af783b7a0a9 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
@@ -72,6 +72,7 @@ MODULE_PARM_DESC(phyaddr, "Physical device address");
#define STMMAC_XDP_PASS 0
#define STMMAC_XDP_CONSUMED BIT(0)
#define STMMAC_XDP_TX BIT(1)
+#define STMMAC_XDP_REDIRECT BIT(2)
static int flow_ctrl = FLOW_AUTO;
module_param(flow_ctrl, int, 0644);
@@ -1458,7 +1459,8 @@ static void stmmac_free_tx_buffer(struct stmmac_priv *priv, u32 queue, int i)
}
if (tx_q->xdpf[i] &&
- tx_q->tx_skbuff_dma[i].buf_type == STMMAC_TXBUF_T_XDP_TX) {
+ (tx_q->tx_skbuff_dma[i].buf_type == STMMAC_TXBUF_T_XDP_TX ||
+ tx_q->tx_skbuff_dma[i].buf_type == STMMAC_TXBUF_T_XDP_NDO)) {
xdp_return_frame(tx_q->xdpf[i]);
tx_q->xdpf[i] = NULL;
}
@@ -2220,7 +2222,8 @@ static int stmmac_tx_clean(struct stmmac_priv *priv, int budget, u32 queue)
struct dma_desc *p;
int status;
- if (tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_XDP_TX) {
+ if (tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_XDP_TX ||
+ tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_XDP_NDO) {
xdpf = tx_q->xdpf[entry];
skb = NULL;
} else if (tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_SKB) {
@@ -2292,6 +2295,12 @@ static int stmmac_tx_clean(struct stmmac_priv *priv, int budget, u32 queue)
tx_q->xdpf[entry] = NULL;
}
+ if (xdpf &&
+ tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_XDP_NDO) {
+ xdp_return_frame(xdpf);
+ tx_q->xdpf[entry] = NULL;
+ }
+
if (tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_SKB) {
if (likely(skb)) {
pkts_compl++;
@@ -4237,10 +4246,9 @@ static unsigned int stmmac_rx_buf2_len(struct stmmac_priv *priv,
}
static int stmmac_xdp_xmit_xdpf(struct stmmac_priv *priv, int queue,
- struct xdp_frame *xdpf)
+ struct xdp_frame *xdpf, bool dma_map)
{
struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
- struct page *page = virt_to_page(xdpf->data);
unsigned int entry = tx_q->cur_tx;
struct dma_desc *tx_desc;
dma_addr_t dma_addr;
@@ -4256,12 +4264,23 @@ static int stmmac_xdp_xmit_xdpf(struct stmmac_priv *priv, int queue,
else
tx_desc = tx_q->dma_tx + entry;
- dma_addr = page_pool_get_dma_addr(page) + sizeof(*xdpf) +
- xdpf->headroom;
- dma_sync_single_for_device(priv->device, dma_addr,
- xdpf->len, DMA_BIDIRECTIONAL);
+ if (dma_map) {
+ dma_addr = dma_map_single(priv->device, xdpf->data,
+ xdpf->len, DMA_TO_DEVICE);
+ if (dma_mapping_error(priv->device, dma_addr))
+ return STMMAC_XDP_CONSUMED;
+
+ tx_q->tx_skbuff_dma[entry].buf_type = STMMAC_TXBUF_T_XDP_NDO;
+ } else {
+ struct page *page = virt_to_page(xdpf->data);
- tx_q->tx_skbuff_dma[entry].buf_type = STMMAC_TXBUF_T_XDP_TX;
+ dma_addr = page_pool_get_dma_addr(page) + sizeof(*xdpf) +
+ xdpf->headroom;
+ dma_sync_single_for_device(priv->device, dma_addr,
+ xdpf->len, DMA_BIDIRECTIONAL);
+
+ tx_q->tx_skbuff_dma[entry].buf_type = STMMAC_TXBUF_T_XDP_TX;
+ }
tx_q->tx_skbuff_dma[entry].buf = dma_addr;
tx_q->tx_skbuff_dma[entry].map_as_page = false;
@@ -4328,7 +4347,7 @@ static int stmmac_xdp_xmit_back(struct stmmac_priv *priv,
nq = netdev_get_tx_queue(priv->dev, queue);
__netif_tx_lock(nq, cpu);
- res = stmmac_xdp_xmit_xdpf(priv, queue, xdpf);
+ res = stmmac_xdp_xmit_xdpf(priv, queue, xdpf, false);
if (res == STMMAC_XDP_TX) {
stmmac_flush_tx_descriptors(priv, queue);
stmmac_tx_timer_arm(priv, queue);
@@ -4361,6 +4380,12 @@ static struct sk_buff *stmmac_xdp_run_prog(struct stmmac_priv *priv,
case XDP_TX:
res = stmmac_xdp_xmit_back(priv, xdp);
break;
+ case XDP_REDIRECT:
+ if (xdp_do_redirect(priv->dev, xdp, prog) < 0)
+ res = STMMAC_XDP_CONSUMED;
+ else
+ res = STMMAC_XDP_REDIRECT;
+ break;
default:
bpf_warn_invalid_xdp_action(act);
fallthrough;
@@ -4396,6 +4421,7 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
unsigned int desc_size;
struct sk_buff *skb = NULL;
struct xdp_buff xdp;
+ int xdp_status = 0;
int buf_sz;
dma_dir = page_pool_get_dma_dir(rx_q->page_pool);
@@ -4565,6 +4591,12 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
skb = NULL;
count++;
continue;
+ } else if (xdp_res & STMMAC_XDP_REDIRECT) {
+ xdp_status |= xdp_res;
+ buf->page = NULL;
+ skb = NULL;
+ count++;
+ continue;
}
}
}
@@ -4645,6 +4677,9 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
rx_q->state.len = len;
}
+ if (xdp_status & STMMAC_XDP_REDIRECT)
+ xdp_do_flush();
+
stmmac_rx_refill(priv, queue);
priv->xstats.rx_pkt_n += count;
@@ -5571,6 +5606,46 @@ static int stmmac_bpf(struct net_device *dev, struct netdev_bpf *bpf)
}
}
+static int stmmac_xdp_xmit(struct net_device *dev, int num_frames,
+ struct xdp_frame **frames, u32 flags)
+{
+ struct stmmac_priv *priv = netdev_priv(dev);
+ int cpu = smp_processor_id();
+ struct netdev_queue *nq;
+ int i, nxmit = 0;
+ int queue;
+
+ if (unlikely(test_bit(STMMAC_DOWN, &priv->state)))
+ return -ENETDOWN;
+
+ if (unlikely(flags & ~XDP_XMIT_FLAGS_MASK))
+ return -EINVAL;
+
+ queue = stmmac_xdp_get_tx_queue(priv, cpu);
+ nq = netdev_get_tx_queue(priv->dev, queue);
+
+ __netif_tx_lock(nq, cpu);
+
+ for (i = 0; i < num_frames; i++) {
+ int res;
+
+ res = stmmac_xdp_xmit_xdpf(priv, queue, frames[i], true);
+ if (res == STMMAC_XDP_CONSUMED)
+ break;
+
+ nxmit++;
+ }
+
+ if (flags & XDP_XMIT_FLUSH) {
+ stmmac_flush_tx_descriptors(priv, queue);
+ stmmac_tx_timer_arm(priv, queue);
+ }
+
+ __netif_tx_unlock(nq);
+
+ return nxmit;
+}
+
static const struct net_device_ops stmmac_netdev_ops = {
.ndo_open = stmmac_open,
.ndo_start_xmit = stmmac_xmit,
@@ -5590,6 +5665,7 @@ static const struct net_device_ops stmmac_netdev_ops = {
.ndo_vlan_rx_add_vid = stmmac_vlan_rx_add_vid,
.ndo_vlan_rx_kill_vid = stmmac_vlan_rx_kill_vid,
.ndo_bpf = stmmac_bpf,
+ .ndo_xdp_xmit = stmmac_xdp_xmit,
};
static void stmmac_reset_subtask(struct stmmac_priv *priv)
--
2.25.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH net-next 6/6] net: stmmac: Add support for XDP_REDIRECT action
@ 2021-03-29 13:40 ` Ong Boon Leong
0 siblings, 0 replies; 15+ messages in thread
From: Ong Boon Leong @ 2021-03-29 13:40 UTC (permalink / raw)
To: Giuseppe Cavallaro, Alexandre Torgue, Jose Abreu,
David S . Miller, Jakub Kicinski, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend
Cc: Maxime Coquelin, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, netdev, linux-stm32, linux-arm-kernel,
linux-kernel, bpf, Ong Boon Leong
This patch adds the support of XDP_REDIRECT to another remote cpu for
further action. It also implements ndo_xdp_xmit ops, enabling the driver
to transmit packets forwarded to it by XDP program running on another
interface.
This patch has been tested using "xdp_redirect_cpu" for XDP_REDIRECT
+ drop testing. It also been tested with "xdp_redirect" sample app
which can be used to exercise ndo_xdp_xmit ops. The burst traffics are
generated using pktgen_sample03_burst_single_flow.sh in samples/pktgen
directory.
Signed-off-by: Ong Boon Leong <boon.leong.ong@intel.com>
---
drivers/net/ethernet/stmicro/stmmac/stmmac.h | 1 +
.../net/ethernet/stmicro/stmmac/stmmac_main.c | 96 +++++++++++++++++--
2 files changed, 87 insertions(+), 10 deletions(-)
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac.h b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
index a93e22a6be59..c49debb62b05 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac.h
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
@@ -39,6 +39,7 @@ struct stmmac_resources {
enum stmmac_txbuf_type {
STMMAC_TXBUF_T_SKB,
STMMAC_TXBUF_T_XDP_TX,
+ STMMAC_TXBUF_T_XDP_NDO,
};
struct stmmac_tx_info {
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
index b92355561609..3af783b7a0a9 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
@@ -72,6 +72,7 @@ MODULE_PARM_DESC(phyaddr, "Physical device address");
#define STMMAC_XDP_PASS 0
#define STMMAC_XDP_CONSUMED BIT(0)
#define STMMAC_XDP_TX BIT(1)
+#define STMMAC_XDP_REDIRECT BIT(2)
static int flow_ctrl = FLOW_AUTO;
module_param(flow_ctrl, int, 0644);
@@ -1458,7 +1459,8 @@ static void stmmac_free_tx_buffer(struct stmmac_priv *priv, u32 queue, int i)
}
if (tx_q->xdpf[i] &&
- tx_q->tx_skbuff_dma[i].buf_type == STMMAC_TXBUF_T_XDP_TX) {
+ (tx_q->tx_skbuff_dma[i].buf_type == STMMAC_TXBUF_T_XDP_TX ||
+ tx_q->tx_skbuff_dma[i].buf_type == STMMAC_TXBUF_T_XDP_NDO)) {
xdp_return_frame(tx_q->xdpf[i]);
tx_q->xdpf[i] = NULL;
}
@@ -2220,7 +2222,8 @@ static int stmmac_tx_clean(struct stmmac_priv *priv, int budget, u32 queue)
struct dma_desc *p;
int status;
- if (tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_XDP_TX) {
+ if (tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_XDP_TX ||
+ tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_XDP_NDO) {
xdpf = tx_q->xdpf[entry];
skb = NULL;
} else if (tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_SKB) {
@@ -2292,6 +2295,12 @@ static int stmmac_tx_clean(struct stmmac_priv *priv, int budget, u32 queue)
tx_q->xdpf[entry] = NULL;
}
+ if (xdpf &&
+ tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_XDP_NDO) {
+ xdp_return_frame(xdpf);
+ tx_q->xdpf[entry] = NULL;
+ }
+
if (tx_q->tx_skbuff_dma[entry].buf_type == STMMAC_TXBUF_T_SKB) {
if (likely(skb)) {
pkts_compl++;
@@ -4237,10 +4246,9 @@ static unsigned int stmmac_rx_buf2_len(struct stmmac_priv *priv,
}
static int stmmac_xdp_xmit_xdpf(struct stmmac_priv *priv, int queue,
- struct xdp_frame *xdpf)
+ struct xdp_frame *xdpf, bool dma_map)
{
struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
- struct page *page = virt_to_page(xdpf->data);
unsigned int entry = tx_q->cur_tx;
struct dma_desc *tx_desc;
dma_addr_t dma_addr;
@@ -4256,12 +4264,23 @@ static int stmmac_xdp_xmit_xdpf(struct stmmac_priv *priv, int queue,
else
tx_desc = tx_q->dma_tx + entry;
- dma_addr = page_pool_get_dma_addr(page) + sizeof(*xdpf) +
- xdpf->headroom;
- dma_sync_single_for_device(priv->device, dma_addr,
- xdpf->len, DMA_BIDIRECTIONAL);
+ if (dma_map) {
+ dma_addr = dma_map_single(priv->device, xdpf->data,
+ xdpf->len, DMA_TO_DEVICE);
+ if (dma_mapping_error(priv->device, dma_addr))
+ return STMMAC_XDP_CONSUMED;
+
+ tx_q->tx_skbuff_dma[entry].buf_type = STMMAC_TXBUF_T_XDP_NDO;
+ } else {
+ struct page *page = virt_to_page(xdpf->data);
- tx_q->tx_skbuff_dma[entry].buf_type = STMMAC_TXBUF_T_XDP_TX;
+ dma_addr = page_pool_get_dma_addr(page) + sizeof(*xdpf) +
+ xdpf->headroom;
+ dma_sync_single_for_device(priv->device, dma_addr,
+ xdpf->len, DMA_BIDIRECTIONAL);
+
+ tx_q->tx_skbuff_dma[entry].buf_type = STMMAC_TXBUF_T_XDP_TX;
+ }
tx_q->tx_skbuff_dma[entry].buf = dma_addr;
tx_q->tx_skbuff_dma[entry].map_as_page = false;
@@ -4328,7 +4347,7 @@ static int stmmac_xdp_xmit_back(struct stmmac_priv *priv,
nq = netdev_get_tx_queue(priv->dev, queue);
__netif_tx_lock(nq, cpu);
- res = stmmac_xdp_xmit_xdpf(priv, queue, xdpf);
+ res = stmmac_xdp_xmit_xdpf(priv, queue, xdpf, false);
if (res == STMMAC_XDP_TX) {
stmmac_flush_tx_descriptors(priv, queue);
stmmac_tx_timer_arm(priv, queue);
@@ -4361,6 +4380,12 @@ static struct sk_buff *stmmac_xdp_run_prog(struct stmmac_priv *priv,
case XDP_TX:
res = stmmac_xdp_xmit_back(priv, xdp);
break;
+ case XDP_REDIRECT:
+ if (xdp_do_redirect(priv->dev, xdp, prog) < 0)
+ res = STMMAC_XDP_CONSUMED;
+ else
+ res = STMMAC_XDP_REDIRECT;
+ break;
default:
bpf_warn_invalid_xdp_action(act);
fallthrough;
@@ -4396,6 +4421,7 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
unsigned int desc_size;
struct sk_buff *skb = NULL;
struct xdp_buff xdp;
+ int xdp_status = 0;
int buf_sz;
dma_dir = page_pool_get_dma_dir(rx_q->page_pool);
@@ -4565,6 +4591,12 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
skb = NULL;
count++;
continue;
+ } else if (xdp_res & STMMAC_XDP_REDIRECT) {
+ xdp_status |= xdp_res;
+ buf->page = NULL;
+ skb = NULL;
+ count++;
+ continue;
}
}
}
@@ -4645,6 +4677,9 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
rx_q->state.len = len;
}
+ if (xdp_status & STMMAC_XDP_REDIRECT)
+ xdp_do_flush();
+
stmmac_rx_refill(priv, queue);
priv->xstats.rx_pkt_n += count;
@@ -5571,6 +5606,46 @@ static int stmmac_bpf(struct net_device *dev, struct netdev_bpf *bpf)
}
}
+static int stmmac_xdp_xmit(struct net_device *dev, int num_frames,
+ struct xdp_frame **frames, u32 flags)
+{
+ struct stmmac_priv *priv = netdev_priv(dev);
+ int cpu = smp_processor_id();
+ struct netdev_queue *nq;
+ int i, nxmit = 0;
+ int queue;
+
+ if (unlikely(test_bit(STMMAC_DOWN, &priv->state)))
+ return -ENETDOWN;
+
+ if (unlikely(flags & ~XDP_XMIT_FLAGS_MASK))
+ return -EINVAL;
+
+ queue = stmmac_xdp_get_tx_queue(priv, cpu);
+ nq = netdev_get_tx_queue(priv->dev, queue);
+
+ __netif_tx_lock(nq, cpu);
+
+ for (i = 0; i < num_frames; i++) {
+ int res;
+
+ res = stmmac_xdp_xmit_xdpf(priv, queue, frames[i], true);
+ if (res == STMMAC_XDP_CONSUMED)
+ break;
+
+ nxmit++;
+ }
+
+ if (flags & XDP_XMIT_FLUSH) {
+ stmmac_flush_tx_descriptors(priv, queue);
+ stmmac_tx_timer_arm(priv, queue);
+ }
+
+ __netif_tx_unlock(nq);
+
+ return nxmit;
+}
+
static const struct net_device_ops stmmac_netdev_ops = {
.ndo_open = stmmac_open,
.ndo_start_xmit = stmmac_xmit,
@@ -5590,6 +5665,7 @@ static const struct net_device_ops stmmac_netdev_ops = {
.ndo_vlan_rx_add_vid = stmmac_vlan_rx_add_vid,
.ndo_vlan_rx_kill_vid = stmmac_vlan_rx_kill_vid,
.ndo_bpf = stmmac_bpf,
+ .ndo_xdp_xmit = stmmac_xdp_xmit,
};
static void stmmac_reset_subtask(struct stmmac_priv *priv)
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH net-next 4/6] net: stmmac: Add initial XDP support
2021-03-29 13:40 ` Ong Boon Leong
(?)
@ 2021-03-29 15:49 ` kernel test robot
-1 siblings, 0 replies; 15+ messages in thread
From: kernel test robot @ 2021-03-29 15:49 UTC (permalink / raw)
To: kbuild-all
[-- Attachment #1: Type: text/plain, Size: 10885 bytes --]
Hi Ong,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on net-next/master]
url: https://github.com/0day-ci/linux/commits/Ong-Boon-Leong/stmmac-Add-XDP-support/20210329-213915
base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git 9d0365448b5b954bba1b551ade5b273d629446bb
config: arc-allyesconfig (attached as .config)
compiler: arceb-elf-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/2eb2e788a389bf190a004f6416ea2bfd57783f4a
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Ong-Boon-Leong/stmmac-Add-XDP-support/20210329-213915
git checkout 2eb2e788a389bf190a004f6416ea2bfd57783f4a
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=arc
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All warnings (new ones prefixed by >>):
drivers/net/ethernet/stmicro/stmmac/stmmac_main.c: In function 'stmmac_rx_refill':
>> drivers/net/ethernet/stmicro/stmmac/stmmac_main.c:4067:6: warning: variable 'len' set but not used [-Wunused-but-set-variable]
4067 | int len, dirty = stmmac_rx_dirty(priv, queue);
| ^~~
vim +/len +4067 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
b93819854d6e79 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Vince Bridgers 2014-01-14 4056
32ceabcad3c8ab drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Giuseppe CAVALLARO 2013-04-08 4057 /**
732fdf0e5253e9 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Giuseppe CAVALLARO 2014-11-18 4058 * stmmac_rx_refill - refill used skb preallocated buffers
32ceabcad3c8ab drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Giuseppe CAVALLARO 2013-04-08 4059 * @priv: driver private structure
54139cf3bb33fa drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Joao Pinto 2017-04-06 4060 * @queue: RX queue index
32ceabcad3c8ab drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Giuseppe CAVALLARO 2013-04-08 4061 * Description : this is to reallocate the skb for the reception process
32ceabcad3c8ab drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Giuseppe CAVALLARO 2013-04-08 4062 * that is based on zero-copy.
32ceabcad3c8ab drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Giuseppe CAVALLARO 2013-04-08 4063 */
54139cf3bb33fa drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Joao Pinto 2017-04-06 4064 static inline void stmmac_rx_refill(struct stmmac_priv *priv, u32 queue)
47dd7a540b8a0c drivers/net/stmmac/stmmac_main.c Giuseppe Cavallaro 2009-10-14 4065 {
54139cf3bb33fa drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Joao Pinto 2017-04-06 4066 struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
3caa61c2087534 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-07-30 @4067 int len, dirty = stmmac_rx_dirty(priv, queue);
54139cf3bb33fa drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Joao Pinto 2017-04-06 4068 unsigned int entry = rx_q->dirty_rx;
54139cf3bb33fa drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Joao Pinto 2017-04-06 4069
3caa61c2087534 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-07-30 4070 len = DIV_ROUND_UP(priv->dma_buf_sz, PAGE_SIZE) * PAGE_SIZE;
3caa61c2087534 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-07-30 4071
e3ad57c96715df drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Giuseppe Cavallaro 2016-02-29 4072 while (dirty-- > 0) {
2af6106ae94965 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-07-09 4073 struct stmmac_rx_buffer *buf = &rx_q->buf_pool[entry];
c24602ef866493 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Giuseppe CAVALLARO 2013-03-26 4074 struct dma_desc *p;
d429b66e2ebcec drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-07-09 4075 bool use_rx_wd;
c24602ef866493 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Giuseppe CAVALLARO 2013-03-26 4076
c24602ef866493 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Giuseppe CAVALLARO 2013-03-26 4077 if (priv->extend_desc)
54139cf3bb33fa drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Joao Pinto 2017-04-06 4078 p = (struct dma_desc *)(rx_q->dma_erx + entry);
c24602ef866493 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Giuseppe CAVALLARO 2013-03-26 4079 else
54139cf3bb33fa drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Joao Pinto 2017-04-06 4080 p = rx_q->dma_rx + entry;
c24602ef866493 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Giuseppe CAVALLARO 2013-03-26 4081
2af6106ae94965 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-07-09 4082 if (!buf->page) {
2af6106ae94965 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-07-09 4083 buf->page = page_pool_dev_alloc_pages(rx_q->page_pool);
2af6106ae94965 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-07-09 4084 if (!buf->page)
362b37be01edc7 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Giuseppe CAVALLARO 2014-08-27 4085 break;
362b37be01edc7 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Giuseppe CAVALLARO 2014-08-27 4086 }
286a837217204b drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Giuseppe CAVALLARO 2011-10-18 4087
67afd6d1cfdf0d drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-08-17 4088 if (priv->sph && !buf->sec_page) {
67afd6d1cfdf0d drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-08-17 4089 buf->sec_page = page_pool_dev_alloc_pages(rx_q->page_pool);
67afd6d1cfdf0d drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-08-17 4090 if (!buf->sec_page)
67afd6d1cfdf0d drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-08-17 4091 break;
67afd6d1cfdf0d drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-08-17 4092
67afd6d1cfdf0d drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-08-17 4093 buf->sec_addr = page_pool_get_dma_addr(buf->sec_page);
67afd6d1cfdf0d drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-08-17 4094 }
67afd6d1cfdf0d drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-08-17 4095
2eb2e788a389bf drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Ong Boon Leong 2021-03-29 4096 buf->addr = page_pool_get_dma_addr(buf->page) + buf->page_offset;
3caa61c2087534 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-07-30 4097
2af6106ae94965 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-07-09 4098 stmmac_set_desc_addr(priv, p, buf->addr);
396e13e11577b6 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Joakim Zhang 2021-02-25 4099 if (priv->sph)
396e13e11577b6 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Joakim Zhang 2021-02-25 4100 stmmac_set_desc_sec_addr(priv, p, buf->sec_addr, true);
396e13e11577b6 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Joakim Zhang 2021-02-25 4101 else
396e13e11577b6 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Joakim Zhang 2021-02-25 4102 stmmac_set_desc_sec_addr(priv, p, buf->sec_addr, false);
2c520b1c9cfa7d drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2018-04-16 4103 stmmac_refill_desc3(priv, rx_q, p);
286a837217204b drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Giuseppe CAVALLARO 2011-10-18 4104
d429b66e2ebcec drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-07-09 4105 rx_q->rx_count_frames++;
db2f2842e6f560 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Ong Boon Leong 2021-03-17 4106 rx_q->rx_count_frames += priv->rx_coal_frames[queue];
db2f2842e6f560 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Ong Boon Leong 2021-03-17 4107 if (rx_q->rx_count_frames > priv->rx_coal_frames[queue])
6fa9d691b91ac6 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-09-10 4108 rx_q->rx_count_frames = 0;
09146abebc788a drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-11-14 4109
db2f2842e6f560 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Ong Boon Leong 2021-03-17 4110 use_rx_wd = !priv->rx_coal_frames[queue];
09146abebc788a drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-11-14 4111 use_rx_wd |= rx_q->rx_count_frames > 0;
09146abebc788a drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-11-14 4112 if (!priv->use_riwt)
09146abebc788a drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-11-14 4113 use_rx_wd = false;
d429b66e2ebcec drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-07-09 4114
ad688cdbb07683 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Pavel Machek 2016-12-18 4115 dma_wmb();
2af6106ae94965 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-07-09 4116 stmmac_set_rx_owner(priv, p, use_rx_wd);
e3ad57c96715df drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Giuseppe Cavallaro 2016-02-29 4117
aa042f60e4961d drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Song, Yoong Siang 2020-09-16 4118 entry = STMMAC_GET_ENTRY(entry, priv->dma_rx_size);
47dd7a540b8a0c drivers/net/stmmac/stmmac_main.c Giuseppe Cavallaro 2009-10-14 4119 }
54139cf3bb33fa drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Joao Pinto 2017-04-06 4120 rx_q->dirty_rx = entry;
858a31ffc3d97a drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-06-28 4121 rx_q->rx_tail_addr = rx_q->dma_rx_phy +
858a31ffc3d97a drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Jose Abreu 2019-06-28 4122 (rx_q->dirty_rx * sizeof(struct dma_desc));
4523a561152670 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c Biao Huang 2019-05-24 4123 stmmac_set_rx_tail_ptr(priv, priv->ioaddr, rx_q->rx_tail_addr, queue);
47dd7a540b8a0c drivers/net/stmmac/stmmac_main.c Giuseppe Cavallaro 2009-10-14 4124 }
47dd7a540b8a0c drivers/net/stmmac/stmmac_main.c Giuseppe Cavallaro 2009-10-14 4125
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 67556 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2021-03-29 22:36 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-29 13:40 [PATCH net-next 0/6] stmmac: Add XDP support Ong Boon Leong
2021-03-29 13:40 ` Ong Boon Leong
2021-03-29 13:40 ` [PATCH net-next 1/6] stmmac: intel: set IRQ affinity hint for multi MSI vectors Ong Boon Leong
2021-03-29 13:40 ` Ong Boon Leong
2021-03-29 13:40 ` [PATCH net-next 2/6] net: stmmac: make SPH enable/disable to be configurable Ong Boon Leong
2021-03-29 13:40 ` Ong Boon Leong
2021-03-29 13:40 ` [PATCH net-next 3/6] net: stmmac: arrange Tx tail pointer update to stmmac_flush_tx_descriptors Ong Boon Leong
2021-03-29 13:40 ` Ong Boon Leong
2021-03-29 13:40 ` [PATCH net-next 4/6] net: stmmac: Add initial XDP support Ong Boon Leong
2021-03-29 13:40 ` Ong Boon Leong
2021-03-29 15:49 ` kernel test robot
2021-03-29 13:40 ` [PATCH net-next 5/6] net: stmmac: Add support for XDP_TX action Ong Boon Leong
2021-03-29 13:40 ` Ong Boon Leong
2021-03-29 13:40 ` [PATCH net-next 6/6] net: stmmac: Add support for XDP_REDIRECT action Ong Boon Leong
2021-03-29 13:40 ` Ong Boon Leong
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.