* [net-next PATCH 00/12] Implement XDP bpf_redirect
@ 2017-07-17 16:26 John Fastabend
2017-07-17 16:26 ` [net-next PATCH 01/12] ixgbe: NULL xdp_tx rings on resource cleanup John Fastabend
` (12 more replies)
0 siblings, 13 replies; 19+ messages in thread
From: John Fastabend @ 2017-07-17 16:26 UTC (permalink / raw)
To: davem; +Cc: daniel, ast, netdev, john.fastabend, brouer, andy
This series adds two new XDP helper routines bpf_redirect() and
bpf_redirect_map(). The first variant bpf_redirect() is meant
to be used the same way it is currently being used by the cls_bpf
classifier. An xdp packet will be redirected immediately when this
is called.
The other variant bpf_redirect_map(map, key, flags) uses a new
map type called devmap. A devmap uses integers as keys and
net_devices as values. The user provies key/ifindex pairs to
update the map with new net_devices. This provides two benefits
over the normal variant 'bpf_redirect()'. First the datapath
bpf program is abstracted away from using hard-coded ifindex
values. Allowing a single bpf program to be run any many different
environments. Second, and perhaps more important, the map enables
batching packet transmits. The map plus small driver changes
allows for batching all send requests across a NAPI poll loop.
This allows driver writers to optimize the driver xmit path
and only call expensive operations once for a batch of xdp_buffs.
The devmap was designed to support possible future work for
multicast and broadcast as follow-up patches.
To see, in more detail, how to leverage the new helpers and
map from the userspace side please review these two patches,
xdp: sample program for new bpf_redirect helper
xdp: bpf redirect with map sample program
Performance numbers provided by Jesper are the following, tested
using the ixgbe driver with CPU E5-1650 v4 @ 3.60GHz:
13,939,674 pkt/s = XDP_DROP without touching memory
14,290,650 pkt/s = xdp1: XDP_DROP with reading packet data
13,221,812 pkt/s = xdp2: XDP_TX with swap mac (writes into pkt)
7,596,576 pkt/s = xdp_redirect: XDP_REDIRECT with swap mac (like XDP_TX)
13,058,435 pkt/s = xdp_redirect_map:XDP_REDIRECT with swap mac + devmap
A big thanks to everyone who helped with this series. Jesper
provided fixes, debugging, code review, performance benchmarks!
Daniel provided lots of useful feedback and code review. And last
but not least Andy provided useful feedback related to supporting
additional drivers, generic xdp implementation, testing, etc. Any
other feedback is welcome but I believe at this point these are
ready to be merged!
Whats left... get the rest of the drivers developers to implement
this in all the drivers.
---
John Fastabend (12):
ixgbe: NULL xdp_tx rings on resource cleanup
net: xdp: support xdp generic on virtual devices
xdp: add bpf_redirect helper function
xdp: sample program for new bpf_redirect helper
net: implement XDP_REDIRECT for xdp generic
ixgbe: add initial support for xdp redirect
xdp: add trace event for xdp redirect
bpf: add devmap, a map for storing net device references
bpf: add bpf_redirect_map helper routine
xdp: Add batching support to redirect map
net: add notifier hooks for devmap bpf map
xdp: bpf redirect with map sample program
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 8
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 63 ++++
include/linux/bpf.h | 5
include/linux/bpf_types.h | 3
include/linux/filter.h | 14 +
include/linux/netdevice.h | 11 +
include/trace/events/xdp.h | 31 ++
include/uapi/linux/bpf.h | 10 +
kernel/bpf/Makefile | 3
kernel/bpf/devmap.c | 431 +++++++++++++++++++++++++
kernel/bpf/verifier.c | 12 +
net/core/dev.c | 226 ++++++++-----
net/core/filter.c | 170 ++++++++++
samples/bpf/Makefile | 8
samples/bpf/xdp_redirect_kern.c | 81 +++++
samples/bpf/xdp_redirect_map_kern.c | 83 +++++
samples/bpf/xdp_redirect_map_user.c | 105 ++++++
samples/bpf/xdp_redirect_user.c | 102 ++++++
tools/testing/selftests/bpf/bpf_helpers.h | 2
tools/testing/selftests/bpf/test_maps.c | 15 +
20 files changed, 1282 insertions(+), 101 deletions(-)
create mode 100644 kernel/bpf/devmap.c
create mode 100644 samples/bpf/xdp_redirect_kern.c
create mode 100644 samples/bpf/xdp_redirect_map_kern.c
create mode 100644 samples/bpf/xdp_redirect_map_user.c
create mode 100644 samples/bpf/xdp_redirect_user.c
^ permalink raw reply [flat|nested] 19+ messages in thread
* [net-next PATCH 01/12] ixgbe: NULL xdp_tx rings on resource cleanup
2017-07-17 16:26 [net-next PATCH 00/12] Implement XDP bpf_redirect John Fastabend
@ 2017-07-17 16:26 ` John Fastabend
2017-07-17 16:26 ` [net-next PATCH 02/12] net: xdp: support xdp generic on virtual devices John Fastabend
` (11 subsequent siblings)
12 siblings, 0 replies; 19+ messages in thread
From: John Fastabend @ 2017-07-17 16:26 UTC (permalink / raw)
To: davem; +Cc: daniel, ast, netdev, john.fastabend, brouer, andy
tx_rings and rx_rings are cleaned up on close paths in ixgbe driver
however, xdp_rings are not. Set the xdp_rings to NULL here so that
we can use the pointer to indicate if the XDP rings are initialized.
Signed-off-by: John Fastabend <john.fastabend@gmail.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c
index b45fdc9..f1bfae0 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c
@@ -1018,8 +1018,12 @@ static void ixgbe_free_q_vector(struct ixgbe_adapter *adapter, int v_idx)
struct ixgbe_q_vector *q_vector = adapter->q_vector[v_idx];
struct ixgbe_ring *ring;
- ixgbe_for_each_ring(ring, q_vector->tx)
- adapter->tx_ring[ring->queue_index] = NULL;
+ ixgbe_for_each_ring(ring, q_vector->tx) {
+ if (ring_is_xdp(ring))
+ adapter->xdp_ring[ring->queue_index] = NULL;
+ else
+ adapter->tx_ring[ring->queue_index] = NULL;
+ }
ixgbe_for_each_ring(ring, q_vector->rx)
adapter->rx_ring[ring->queue_index] = NULL;
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [net-next PATCH 02/12] net: xdp: support xdp generic on virtual devices
2017-07-17 16:26 [net-next PATCH 00/12] Implement XDP bpf_redirect John Fastabend
2017-07-17 16:26 ` [net-next PATCH 01/12] ixgbe: NULL xdp_tx rings on resource cleanup John Fastabend
@ 2017-07-17 16:26 ` John Fastabend
2017-07-17 16:27 ` [net-next PATCH 03/12] xdp: add bpf_redirect helper function John Fastabend
` (10 subsequent siblings)
12 siblings, 0 replies; 19+ messages in thread
From: John Fastabend @ 2017-07-17 16:26 UTC (permalink / raw)
To: davem; +Cc: daniel, ast, netdev, john.fastabend, brouer, andy
XDP generic allows users to test XDP programs and/or run them with
degraded performance on devices that do not yet support XDP. For
testing I typically test eBPF programs using a set of veth devices.
This allows testing topologies that would otherwise be difficult to
setup especially in the early stages of development.
This patch adds a xdp generic hook to the netif_rx_internal()
function which is called from dev_forward_skb(). With this addition
attaching XDP programs to veth devices works as expected! Also I
noticed multiple drivers using netif_rx(). These devices will also
benefit and generic XDP will work for them as well.
Signed-off-by: John Fastabend <john.fastabend@gmail.com>
Tested-by: Andy Gospodarek <andy@greyhouse.net>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
net/core/dev.c | 208 ++++++++++++++++++++++++++++++--------------------------
1 file changed, 113 insertions(+), 95 deletions(-)
diff --git a/net/core/dev.c b/net/core/dev.c
index 0244051..a1ed7b4 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -3865,6 +3865,107 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
return NET_RX_DROP;
}
+static u32 netif_receive_generic_xdp(struct sk_buff *skb,
+ struct bpf_prog *xdp_prog)
+{
+ struct xdp_buff xdp;
+ u32 act = XDP_DROP;
+ void *orig_data;
+ int hlen, off;
+ u32 mac_len;
+
+ /* Reinjected packets coming from act_mirred or similar should
+ * not get XDP generic processing.
+ */
+ if (skb_cloned(skb))
+ return XDP_PASS;
+
+ if (skb_linearize(skb))
+ goto do_drop;
+
+ /* The XDP program wants to see the packet starting at the MAC
+ * header.
+ */
+ mac_len = skb->data - skb_mac_header(skb);
+ hlen = skb_headlen(skb) + mac_len;
+ xdp.data = skb->data - mac_len;
+ xdp.data_end = xdp.data + hlen;
+ xdp.data_hard_start = skb->data - skb_headroom(skb);
+ orig_data = xdp.data;
+
+ act = bpf_prog_run_xdp(xdp_prog, &xdp);
+
+ off = xdp.data - orig_data;
+ if (off > 0)
+ __skb_pull(skb, off);
+ else if (off < 0)
+ __skb_push(skb, -off);
+
+ switch (act) {
+ case XDP_TX:
+ __skb_push(skb, mac_len);
+ /* fall through */
+ case XDP_PASS:
+ break;
+
+ default:
+ bpf_warn_invalid_xdp_action(act);
+ /* fall through */
+ case XDP_ABORTED:
+ trace_xdp_exception(skb->dev, xdp_prog, act);
+ /* fall through */
+ case XDP_DROP:
+ do_drop:
+ kfree_skb(skb);
+ break;
+ }
+
+ return act;
+}
+
+/* When doing generic XDP we have to bypass the qdisc layer and the
+ * network taps in order to match in-driver-XDP behavior.
+ */
+static void generic_xdp_tx(struct sk_buff *skb, struct bpf_prog *xdp_prog)
+{
+ struct net_device *dev = skb->dev;
+ struct netdev_queue *txq;
+ bool free_skb = true;
+ int cpu, rc;
+
+ txq = netdev_pick_tx(dev, skb, NULL);
+ cpu = smp_processor_id();
+ HARD_TX_LOCK(dev, txq, cpu);
+ if (!netif_xmit_stopped(txq)) {
+ rc = netdev_start_xmit(skb, dev, txq, 0);
+ if (dev_xmit_complete(rc))
+ free_skb = false;
+ }
+ HARD_TX_UNLOCK(dev, txq);
+ if (free_skb) {
+ trace_xdp_exception(dev, xdp_prog, XDP_TX);
+ kfree_skb(skb);
+ }
+}
+
+static struct static_key generic_xdp_needed __read_mostly;
+
+static int do_xdp_generic(struct sk_buff *skb)
+{
+ struct bpf_prog *xdp_prog = rcu_dereference(skb->dev->xdp_prog);
+
+ if (xdp_prog) {
+ u32 act = netif_receive_generic_xdp(skb, xdp_prog);
+
+ if (act != XDP_PASS) {
+ if (act == XDP_TX)
+ generic_xdp_tx(skb, xdp_prog);
+ return XDP_DROP;
+ }
+ }
+ return XDP_PASS;
+}
+
static int netif_rx_internal(struct sk_buff *skb)
{
int ret;
@@ -3872,6 +3973,14 @@ static int netif_rx_internal(struct sk_buff *skb)
net_timestamp_check(netdev_tstamp_prequeue, skb);
trace_netif_rx(skb);
+
+ if (static_key_false(&generic_xdp_needed)) {
+ int ret = do_xdp_generic(skb);
+
+ if (ret != XDP_PASS)
+ return NET_RX_DROP;
+ }
+
#ifdef CONFIG_RPS
if (static_key_false(&rps_needed)) {
struct rps_dev_flow voidflow, *rflow = &voidflow;
@@ -4338,8 +4447,6 @@ static int __netif_receive_skb(struct sk_buff *skb)
return ret;
}
-static struct static_key generic_xdp_needed __read_mostly;
-
static int generic_xdp_install(struct net_device *dev, struct netdev_xdp *xdp)
{
struct bpf_prog *old = rtnl_dereference(dev->xdp_prog);
@@ -4373,89 +4480,6 @@ static int generic_xdp_install(struct net_device *dev, struct netdev_xdp *xdp)
return ret;
}
-static u32 netif_receive_generic_xdp(struct sk_buff *skb,
- struct bpf_prog *xdp_prog)
-{
- struct xdp_buff xdp;
- u32 act = XDP_DROP;
- void *orig_data;
- int hlen, off;
- u32 mac_len;
-
- /* Reinjected packets coming from act_mirred or similar should
- * not get XDP generic processing.
- */
- if (skb_cloned(skb))
- return XDP_PASS;
-
- if (skb_linearize(skb))
- goto do_drop;
-
- /* The XDP program wants to see the packet starting at the MAC
- * header.
- */
- mac_len = skb->data - skb_mac_header(skb);
- hlen = skb_headlen(skb) + mac_len;
- xdp.data = skb->data - mac_len;
- xdp.data_end = xdp.data + hlen;
- xdp.data_hard_start = skb->data - skb_headroom(skb);
- orig_data = xdp.data;
-
- act = bpf_prog_run_xdp(xdp_prog, &xdp);
-
- off = xdp.data - orig_data;
- if (off > 0)
- __skb_pull(skb, off);
- else if (off < 0)
- __skb_push(skb, -off);
-
- switch (act) {
- case XDP_TX:
- __skb_push(skb, mac_len);
- /* fall through */
- case XDP_PASS:
- break;
-
- default:
- bpf_warn_invalid_xdp_action(act);
- /* fall through */
- case XDP_ABORTED:
- trace_xdp_exception(skb->dev, xdp_prog, act);
- /* fall through */
- case XDP_DROP:
- do_drop:
- kfree_skb(skb);
- break;
- }
-
- return act;
-}
-
-/* When doing generic XDP we have to bypass the qdisc layer and the
- * network taps in order to match in-driver-XDP behavior.
- */
-static void generic_xdp_tx(struct sk_buff *skb, struct bpf_prog *xdp_prog)
-{
- struct net_device *dev = skb->dev;
- struct netdev_queue *txq;
- bool free_skb = true;
- int cpu, rc;
-
- txq = netdev_pick_tx(dev, skb, NULL);
- cpu = smp_processor_id();
- HARD_TX_LOCK(dev, txq, cpu);
- if (!netif_xmit_stopped(txq)) {
- rc = netdev_start_xmit(skb, dev, txq, 0);
- if (dev_xmit_complete(rc))
- free_skb = false;
- }
- HARD_TX_UNLOCK(dev, txq);
- if (free_skb) {
- trace_xdp_exception(dev, xdp_prog, XDP_TX);
- kfree_skb(skb);
- }
-}
-
static int netif_receive_skb_internal(struct sk_buff *skb)
{
int ret;
@@ -4468,17 +4492,11 @@ static int netif_receive_skb_internal(struct sk_buff *skb)
rcu_read_lock();
if (static_key_false(&generic_xdp_needed)) {
- struct bpf_prog *xdp_prog = rcu_dereference(skb->dev->xdp_prog);
-
- if (xdp_prog) {
- u32 act = netif_receive_generic_xdp(skb, xdp_prog);
+ int ret = do_xdp_generic(skb);
- if (act != XDP_PASS) {
- rcu_read_unlock();
- if (act == XDP_TX)
- generic_xdp_tx(skb, xdp_prog);
- return NET_RX_DROP;
- }
+ if (ret != XDP_PASS) {
+ rcu_read_unlock();
+ return NET_RX_DROP;
}
}
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [net-next PATCH 03/12] xdp: add bpf_redirect helper function
2017-07-17 16:26 [net-next PATCH 00/12] Implement XDP bpf_redirect John Fastabend
2017-07-17 16:26 ` [net-next PATCH 01/12] ixgbe: NULL xdp_tx rings on resource cleanup John Fastabend
2017-07-17 16:26 ` [net-next PATCH 02/12] net: xdp: support xdp generic on virtual devices John Fastabend
@ 2017-07-17 16:27 ` John Fastabend
2017-07-17 16:27 ` [net-next PATCH 04/12] xdp: sample program for new bpf_redirect helper John Fastabend
` (9 subsequent siblings)
12 siblings, 0 replies; 19+ messages in thread
From: John Fastabend @ 2017-07-17 16:27 UTC (permalink / raw)
To: davem; +Cc: daniel, ast, netdev, john.fastabend, brouer, andy
This adds support for a bpf_redirect helper function to the XDP
infrastructure. For now this only supports redirecting to the egress
path of a port.
In order to support drivers handling a xdp_buff natively this patches
uses a new ndo operation ndo_xdp_xmit() that takes pushes a xdp_buff
to the specified device.
If the program specifies either (a) an unknown device or (b) a device
that does not support the operation a BPF warning is thrown and the
XDP_ABORTED error code is returned.
Signed-off-by: John Fastabend <john.fastabend@gmail.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
include/linux/filter.h | 4 +++
include/linux/netdevice.h | 6 +++++
include/uapi/linux/bpf.h | 1 +
net/core/filter.c | 52 +++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 63 insertions(+)
diff --git a/include/linux/filter.h b/include/linux/filter.h
index bfef1e5..64cae7a 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -711,7 +711,11 @@ int sk_get_filter(struct sock *sk, struct sock_filter __user *filter,
struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off,
const struct bpf_insn *patch, u32 len);
+
+int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp);
+
void bpf_warn_invalid_xdp_action(u32 act);
+void bpf_warn_invalid_xdp_redirect(u32 ifindex);
#ifdef CONFIG_BPF_JIT
extern int bpf_jit_enable;
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 779b235..77f5376 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -66,6 +66,7 @@
/* UDP Tunnel offloads */
struct udp_tunnel_info;
struct bpf_prog;
+struct xdp_buff;
void netdev_set_default_ethtool_ops(struct net_device *dev,
const struct ethtool_ops *ops);
@@ -1138,6 +1139,9 @@ struct xfrmdev_ops {
* int (*ndo_xdp)(struct net_device *dev, struct netdev_xdp *xdp);
* This function is used to set or query state related to XDP on the
* netdevice. See definition of enum xdp_netdev_command for details.
+ * int (*ndo_xdp_xmit)(struct net_device *dev, struct xdp_buff *xdp);
+ * This function is used to submit a XDP packet for transmit on a
+ * netdevice.
*
*/
struct net_device_ops {
@@ -1323,6 +1327,8 @@ struct net_device_ops {
int needed_headroom);
int (*ndo_xdp)(struct net_device *dev,
struct netdev_xdp *xdp);
+ int (*ndo_xdp_xmit)(struct net_device *dev,
+ struct xdp_buff *xdp);
};
/**
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index e99e3e6..4dbb7a3 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -717,6 +717,7 @@ enum xdp_action {
XDP_DROP,
XDP_PASS,
XDP_TX,
+ XDP_REDIRECT,
};
/* user accessible metadata for XDP packet hook
diff --git a/net/core/filter.c b/net/core/filter.c
index c7f7370..d606a66 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -2412,6 +2412,51 @@ static int bpf_skb_trim_rcsum(struct sk_buff *skb, unsigned int new_len)
.arg2_type = ARG_ANYTHING,
};
+static int __bpf_tx_xdp(struct net_device *dev, struct xdp_buff *xdp)
+{
+ if (dev->netdev_ops->ndo_xdp_xmit) {
+ dev->netdev_ops->ndo_xdp_xmit(dev, xdp);
+ return 0;
+ }
+ bpf_warn_invalid_xdp_redirect(dev->ifindex);
+ return -EOPNOTSUPP;
+}
+
+int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp)
+{
+ struct redirect_info *ri = this_cpu_ptr(&redirect_info);
+
+ dev = dev_get_by_index_rcu(dev_net(dev), ri->ifindex);
+ ri->ifindex = 0;
+ if (unlikely(!dev)) {
+ bpf_warn_invalid_xdp_redirect(ri->ifindex);
+ return -EINVAL;
+ }
+
+ return __bpf_tx_xdp(dev, xdp);
+}
+EXPORT_SYMBOL_GPL(xdp_do_redirect);
+
+BPF_CALL_2(bpf_xdp_redirect, u32, ifindex, u64, flags)
+{
+ struct redirect_info *ri = this_cpu_ptr(&redirect_info);
+
+ if (unlikely(flags))
+ return XDP_ABORTED;
+
+ ri->ifindex = ifindex;
+ ri->flags = flags;
+ return XDP_REDIRECT;
+}
+
+static const struct bpf_func_proto bpf_xdp_redirect_proto = {
+ .func = bpf_xdp_redirect,
+ .gpl_only = false,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_ANYTHING,
+ .arg2_type = ARG_ANYTHING,
+};
+
bool bpf_helper_changes_pkt_data(void *func)
{
if (func == bpf_skb_vlan_push ||
@@ -3011,6 +3056,8 @@ static unsigned long bpf_xdp_copy(void *dst_buff, const void *src_buff,
return &bpf_get_smp_processor_id_proto;
case BPF_FUNC_xdp_adjust_head:
return &bpf_xdp_adjust_head_proto;
+ case BPF_FUNC_redirect:
+ return &bpf_xdp_redirect_proto;
default:
return bpf_base_func_proto(func_id);
}
@@ -3310,6 +3357,11 @@ void bpf_warn_invalid_xdp_action(u32 act)
}
EXPORT_SYMBOL_GPL(bpf_warn_invalid_xdp_action);
+void bpf_warn_invalid_xdp_redirect(u32 ifindex)
+{
+ WARN_ONCE(1, "Illegal XDP redirect to unsupported device ifindex(%i)\n", ifindex);
+}
+
static bool __is_valid_sock_ops_access(int off, int size)
{
if (off < 0 || off >= sizeof(struct bpf_sock_ops))
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [net-next PATCH 04/12] xdp: sample program for new bpf_redirect helper
2017-07-17 16:26 [net-next PATCH 00/12] Implement XDP bpf_redirect John Fastabend
` (2 preceding siblings ...)
2017-07-17 16:27 ` [net-next PATCH 03/12] xdp: add bpf_redirect helper function John Fastabend
@ 2017-07-17 16:27 ` John Fastabend
2017-07-17 16:27 ` [net-next PATCH 05/12] net: implement XDP_REDIRECT for xdp generic John Fastabend
` (8 subsequent siblings)
12 siblings, 0 replies; 19+ messages in thread
From: John Fastabend @ 2017-07-17 16:27 UTC (permalink / raw)
To: davem; +Cc: daniel, ast, netdev, john.fastabend, brouer, andy
This implements a sample program for testing bpf_redirect. It reports
the number of packets redirected per second and as input takes the
ifindex of the device to run the xdp program on and the ifindex of the
interface to redirect packets to.
Signed-off-by: John Fastabend <john.fastabend@gmail.com>
Tested-by: Andy Gospodarek <andy@greyhouse.net>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
samples/bpf/Makefile | 4 ++
samples/bpf/xdp_redirect_kern.c | 81 +++++++++++++++++++++++++++++++
samples/bpf/xdp_redirect_user.c | 102 +++++++++++++++++++++++++++++++++++++++
3 files changed, 187 insertions(+)
create mode 100644 samples/bpf/xdp_redirect_kern.c
create mode 100644 samples/bpf/xdp_redirect_user.c
diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile
index 87246be..97734ce 100644
--- a/samples/bpf/Makefile
+++ b/samples/bpf/Makefile
@@ -37,6 +37,7 @@ hostprogs-y += xdp_tx_iptunnel
hostprogs-y += test_map_in_map
hostprogs-y += per_socket_stats_example
hostprogs-y += load_sock_ops
+hostprogs-y += xdp_redirect
# Libbpf dependencies
LIBBPF := ../../tools/lib/bpf/bpf.o
@@ -78,6 +79,7 @@ lwt_len_hist-objs := bpf_load.o $(LIBBPF) lwt_len_hist_user.o
xdp_tx_iptunnel-objs := bpf_load.o $(LIBBPF) xdp_tx_iptunnel_user.o
test_map_in_map-objs := bpf_load.o $(LIBBPF) test_map_in_map_user.o
per_socket_stats_example-objs := $(LIBBPF) cookie_uid_helper_example.o
+xdp_redirect-objs := bpf_load.o $(LIBBPF) xdp_redirect_user.o
# Tell kbuild to always build the programs
always := $(hostprogs-y)
@@ -119,6 +121,7 @@ always += tcp_bufs_kern.o
always += tcp_cong_kern.o
always += tcp_iw_kern.o
always += tcp_clamp_kern.o
+always += xdp_redirect_kern.o
HOSTCFLAGS += -I$(objtree)/usr/include
HOSTCFLAGS += -I$(srctree)/tools/lib/
@@ -155,6 +158,7 @@ HOSTLOADLIBES_tc_l2_redirect += -l elf
HOSTLOADLIBES_lwt_len_hist += -l elf
HOSTLOADLIBES_xdp_tx_iptunnel += -lelf
HOSTLOADLIBES_test_map_in_map += -lelf
+HOSTLOADLIBES_xdp_redirect += -lelf
# Allows pointing LLC/CLANG to a LLVM backend with bpf support, redefine on cmdline:
# make samples/bpf/ LLC=~/git/llvm/build/bin/llc CLANG=~/git/llvm/build/bin/clang
diff --git a/samples/bpf/xdp_redirect_kern.c b/samples/bpf/xdp_redirect_kern.c
new file mode 100644
index 0000000..a34ad45
--- /dev/null
+++ b/samples/bpf/xdp_redirect_kern.c
@@ -0,0 +1,81 @@
+/* Copyright (c) 2016 John Fastabend <john.r.fastabend@intel.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+#define KBUILD_MODNAME "foo"
+#include <uapi/linux/bpf.h>
+#include <linux/in.h>
+#include <linux/if_ether.h>
+#include <linux/if_packet.h>
+#include <linux/if_vlan.h>
+#include <linux/ip.h>
+#include <linux/ipv6.h>
+#include "bpf_helpers.h"
+
+struct bpf_map_def SEC("maps") tx_port = {
+ .type = BPF_MAP_TYPE_ARRAY,
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .max_entries = 1,
+};
+
+struct bpf_map_def SEC("maps") rxcnt = {
+ .type = BPF_MAP_TYPE_PERCPU_ARRAY,
+ .key_size = sizeof(u32),
+ .value_size = sizeof(long),
+ .max_entries = 1,
+};
+
+
+static void swap_src_dst_mac(void *data)
+{
+ unsigned short *p = data;
+ unsigned short dst[3];
+
+ dst[0] = p[0];
+ dst[1] = p[1];
+ dst[2] = p[2];
+ p[0] = p[3];
+ p[1] = p[4];
+ p[2] = p[5];
+ p[3] = dst[0];
+ p[4] = dst[1];
+ p[5] = dst[2];
+}
+
+SEC("xdp_redirect")
+int xdp_redirect_prog(struct xdp_md *ctx)
+{
+ void *data_end = (void *)(long)ctx->data_end;
+ void *data = (void *)(long)ctx->data;
+ struct ethhdr *eth = data;
+ int rc = XDP_DROP;
+ int *ifindex, port = 0;
+ long *value;
+ u32 key = 0;
+ u64 nh_off;
+
+ nh_off = sizeof(*eth);
+ if (data + nh_off > data_end)
+ return rc;
+
+ ifindex = bpf_map_lookup_elem(&tx_port, &port);
+ if (!ifindex)
+ return rc;
+
+ value = bpf_map_lookup_elem(&rxcnt, &key);
+ if (value)
+ *value += 1;
+
+ swap_src_dst_mac(data);
+ return bpf_redirect(*ifindex, 0);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/samples/bpf/xdp_redirect_user.c b/samples/bpf/xdp_redirect_user.c
new file mode 100644
index 0000000..761a91d
--- /dev/null
+++ b/samples/bpf/xdp_redirect_user.c
@@ -0,0 +1,102 @@
+/* Copyright (c) 2016 John Fastabend <john.r.fastabend@intel.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+#include <linux/bpf.h>
+#include <assert.h>
+#include <errno.h>
+#include <signal.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#include "bpf_load.h"
+#include "bpf_util.h"
+#include "libbpf.h"
+
+static int ifindex_in;
+static int ifindex_out;
+
+static void int_exit(int sig)
+{
+ set_link_xdp_fd(ifindex_in, -1, 0);
+ exit(0);
+}
+
+/* simple per-protocol drop counter
+ */
+static void poll_stats(int interval, int ifindex)
+{
+ unsigned int nr_cpus = bpf_num_possible_cpus();
+ __u64 values[nr_cpus], prev[nr_cpus];
+
+ memset(prev, 0, sizeof(prev));
+
+ while (1) {
+ __u64 sum = 0;
+ __u32 key = 0;
+ int i;
+
+ sleep(interval);
+ assert(bpf_map_lookup_elem(map_fd[1], &key, values) == 0);
+ for (i = 0; i < nr_cpus; i++)
+ sum += (values[i] - prev[i]);
+ if (sum)
+ printf("ifindex %i: %10llu pkt/s\n",
+ ifindex, sum / interval);
+ memcpy(prev, values, sizeof(values));
+ }
+}
+
+int main(int ac, char **argv)
+{
+ char filename[256];
+ int ret, key = 0;
+
+ snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
+
+ if (ac != 3) {
+ printf("usage: %s IFINDEX_IN IFINDEX_OUT\n", argv[0]);
+ return 1;
+ }
+
+ ifindex_in = strtoul(argv[1], NULL, 0);
+ ifindex_out = strtoul(argv[2], NULL, 0);
+
+ if (load_bpf_file(filename)) {
+ printf("%s", bpf_log_buf);
+ return 1;
+ }
+
+ if (!prog_fd[0]) {
+ printf("load_bpf_file: %s\n", strerror(errno));
+ return 1;
+ }
+
+ signal(SIGINT, int_exit);
+
+ if (set_link_xdp_fd(ifindex_in, prog_fd[0], 0) < 0) {
+ printf("link set xdp fd failed\n");
+ return 1;
+ }
+
+ /* bpf redirect port */
+ ret = bpf_map_update_elem(map_fd[0], &key, &ifindex_out, 0);
+ if (ret) {
+ perror("bpf_update_elem");
+ goto out;
+ }
+
+ poll_stats(2, ifindex_out);
+
+out:
+ return 0;
+}
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [net-next PATCH 05/12] net: implement XDP_REDIRECT for xdp generic
2017-07-17 16:26 [net-next PATCH 00/12] Implement XDP bpf_redirect John Fastabend
` (3 preceding siblings ...)
2017-07-17 16:27 ` [net-next PATCH 04/12] xdp: sample program for new bpf_redirect helper John Fastabend
@ 2017-07-17 16:27 ` John Fastabend
2017-07-17 16:28 ` [net-next PATCH 06/12] ixgbe: add initial support for xdp redirect John Fastabend
` (7 subsequent siblings)
12 siblings, 0 replies; 19+ messages in thread
From: John Fastabend @ 2017-07-17 16:27 UTC (permalink / raw)
To: davem; +Cc: daniel, ast, netdev, john.fastabend, brouer, andy
Add support for redirect to xdp generic creating a fall back for
devices that do not yet have support and allowing test infrastructure
using veth pairs to be built.
Signed-off-by: John Fastabend <john.fastabend@gmail.com>
Tested-by: Andy Gospodarek <andy@greyhouse.net>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
include/linux/filter.h | 1 +
net/core/dev.c | 22 ++++++++++++++++++++--
net/core/filter.c | 26 ++++++++++++++++++++++++++
3 files changed, 47 insertions(+), 2 deletions(-)
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 64cae7a..10df7da 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -712,6 +712,7 @@ int sk_get_filter(struct sock *sk, struct sock_filter __user *filter,
struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off,
const struct bpf_insn *patch, u32 len);
+int xdp_do_generic_redirect(struct net_device *dev, struct sk_buff *skb);
int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp);
void bpf_warn_invalid_xdp_action(u32 act);
diff --git a/net/core/dev.c b/net/core/dev.c
index a1ed7b4..9f3f408 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -3902,6 +3902,7 @@ static u32 netif_receive_generic_xdp(struct sk_buff *skb,
__skb_push(skb, -off);
switch (act) {
+ case XDP_REDIRECT:
case XDP_TX:
__skb_push(skb, mac_len);
/* fall through */
@@ -3956,14 +3957,27 @@ static int do_xdp_generic(struct sk_buff *skb)
if (xdp_prog) {
u32 act = netif_receive_generic_xdp(skb, xdp_prog);
+ int err;
if (act != XDP_PASS) {
- if (act == XDP_TX)
+ switch (act) {
+ case XDP_REDIRECT:
+ err = xdp_do_generic_redirect(skb->dev, skb);
+ if (err)
+ goto out_redir;
+ /* fallthru to submit skb */
+ case XDP_TX:
generic_xdp_tx(skb, xdp_prog);
+ break;
+ }
return XDP_DROP;
}
}
return XDP_PASS;
+out_redir:
+ trace_xdp_exception(skb->dev, xdp_prog, XDP_REDIRECT);
+ kfree_skb(skb);
+ return XDP_DROP;
}
static int netif_rx_internal(struct sk_buff *skb)
@@ -3977,8 +3991,12 @@ static int netif_rx_internal(struct sk_buff *skb)
if (static_key_false(&generic_xdp_needed)) {
int ret = do_xdp_generic(skb);
+ /* Consider XDP consuming the packet a success from
+ * the netdev point of view we do not want to count
+ * this as an error.
+ */
if (ret != XDP_PASS)
- return NET_RX_DROP;
+ return NET_RX_SUCCESS;
}
#ifdef CONFIG_RPS
diff --git a/net/core/filter.c b/net/core/filter.c
index d606a66..eeb7134 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -2437,6 +2437,32 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp)
}
EXPORT_SYMBOL_GPL(xdp_do_redirect);
+int xdp_do_generic_redirect(struct net_device *dev, struct sk_buff *skb)
+{
+ struct redirect_info *ri = this_cpu_ptr(&redirect_info);
+ unsigned int len;
+
+ dev = dev_get_by_index_rcu(dev_net(dev), ri->ifindex);
+ ri->ifindex = 0;
+ if (unlikely(!dev)) {
+ bpf_warn_invalid_xdp_redirect(ri->ifindex);
+ goto err;
+ }
+
+ if (unlikely(!(dev->flags & IFF_UP)))
+ goto err;
+
+ len = dev->mtu + dev->hard_header_len + VLAN_HLEN;
+ if (skb->len > len)
+ goto err;
+
+ skb->dev = dev;
+ return 0;
+err:
+ return -EINVAL;
+}
+EXPORT_SYMBOL_GPL(xdp_do_generic_redirect);
+
BPF_CALL_2(bpf_xdp_redirect, u32, ifindex, u64, flags)
{
struct redirect_info *ri = this_cpu_ptr(&redirect_info);
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [net-next PATCH 06/12] ixgbe: add initial support for xdp redirect
2017-07-17 16:26 [net-next PATCH 00/12] Implement XDP bpf_redirect John Fastabend
` (4 preceding siblings ...)
2017-07-17 16:27 ` [net-next PATCH 05/12] net: implement XDP_REDIRECT for xdp generic John Fastabend
@ 2017-07-17 16:28 ` John Fastabend
2017-07-17 16:28 ` [net-next PATCH 07/12] xdp: add trace event " John Fastabend
` (6 subsequent siblings)
12 siblings, 0 replies; 19+ messages in thread
From: John Fastabend @ 2017-07-17 16:28 UTC (permalink / raw)
To: davem; +Cc: daniel, ast, netdev, john.fastabend, brouer, andy
There are optimizations we can add after the basic feature is
enabled. But, for now keep the patch simple.
Signed-off-by: John Fastabend <john.fastabend@gmail.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 41 ++++++++++++++++++++++++-
1 file changed, 40 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
index f1dbdf2..3db0473 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
@@ -2214,7 +2214,7 @@ static struct sk_buff *ixgbe_run_xdp(struct ixgbe_adapter *adapter,
struct ixgbe_ring *rx_ring,
struct xdp_buff *xdp)
{
- int result = IXGBE_XDP_PASS;
+ int err, result = IXGBE_XDP_PASS;
struct bpf_prog *xdp_prog;
u32 act;
@@ -2231,6 +2231,13 @@ static struct sk_buff *ixgbe_run_xdp(struct ixgbe_adapter *adapter,
case XDP_TX:
result = ixgbe_xmit_xdp_ring(adapter, xdp);
break;
+ case XDP_REDIRECT:
+ err = xdp_do_redirect(adapter->netdev, xdp);
+ if (!err)
+ result = IXGBE_XDP_TX;
+ else
+ result = IXGBE_XDP_CONSUMED;
+ break;
default:
bpf_warn_invalid_xdp_action(act);
/* fallthrough */
@@ -9823,6 +9830,37 @@ static int ixgbe_xdp(struct net_device *dev, struct netdev_xdp *xdp)
}
}
+static int ixgbe_xdp_xmit(struct net_device *dev, struct xdp_buff *xdp)
+{
+ struct ixgbe_adapter *adapter = netdev_priv(dev);
+ struct ixgbe_ring *ring;
+ int err;
+
+ if (unlikely(test_bit(__IXGBE_DOWN, &adapter->state)))
+ return -EINVAL;
+
+ /* During program transitions its possible adapter->xdp_prog is assigned
+ * but ring has not been configured yet. In this case simply abort xmit.
+ */
+ ring = adapter->xdp_prog ? adapter->xdp_ring[smp_processor_id()] : NULL;
+ if (unlikely(!ring))
+ return -EINVAL;
+
+ err = ixgbe_xmit_xdp_ring(adapter, xdp);
+ if (err != IXGBE_XDP_TX)
+ return -ENOMEM;
+
+ /* Force memory writes to complete before letting h/w know there
+ * are new descriptors to fetch.
+ */
+ wmb();
+
+ ring = adapter->xdp_ring[smp_processor_id()];
+ writel(ring->next_to_use, ring->tail);
+
+ return 0;
+}
+
static const struct net_device_ops ixgbe_netdev_ops = {
.ndo_open = ixgbe_open,
.ndo_stop = ixgbe_close,
@@ -9869,6 +9907,7 @@ static int ixgbe_xdp(struct net_device *dev, struct netdev_xdp *xdp)
.ndo_udp_tunnel_del = ixgbe_del_udp_tunnel_port,
.ndo_features_check = ixgbe_features_check,
.ndo_xdp = ixgbe_xdp,
+ .ndo_xdp_xmit = ixgbe_xdp_xmit,
};
/**
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [net-next PATCH 07/12] xdp: add trace event for xdp redirect
2017-07-17 16:26 [net-next PATCH 00/12] Implement XDP bpf_redirect John Fastabend
` (5 preceding siblings ...)
2017-07-17 16:28 ` [net-next PATCH 06/12] ixgbe: add initial support for xdp redirect John Fastabend
@ 2017-07-17 16:28 ` John Fastabend
2017-07-17 16:28 ` [net-next PATCH 08/12] bpf: add devmap, a map for storing net device references John Fastabend
` (5 subsequent siblings)
12 siblings, 0 replies; 19+ messages in thread
From: John Fastabend @ 2017-07-17 16:28 UTC (permalink / raw)
To: davem; +Cc: daniel, ast, netdev, john.fastabend, brouer, andy
This adds a trace event for xdp redirect which may help when debugging
XDP programs that use redirect bpf commands.
Signed-off-by: John Fastabend <john.fastabend@gmail.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 2 +-
include/linux/filter.h | 4 ++-
include/trace/events/xdp.h | 31 ++++++++++++++++++++++++-
net/core/filter.c | 13 +++++++---
4 files changed, 43 insertions(+), 7 deletions(-)
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
index 3db0473..38f7ff9 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
@@ -2232,7 +2232,7 @@ static struct sk_buff *ixgbe_run_xdp(struct ixgbe_adapter *adapter,
result = ixgbe_xmit_xdp_ring(adapter, xdp);
break;
case XDP_REDIRECT:
- err = xdp_do_redirect(adapter->netdev, xdp);
+ err = xdp_do_redirect(adapter->netdev, xdp, xdp_prog);
if (!err)
result = IXGBE_XDP_TX;
else
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 10df7da..ce8211f 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -713,7 +713,9 @@ struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off,
const struct bpf_insn *patch, u32 len);
int xdp_do_generic_redirect(struct net_device *dev, struct sk_buff *skb);
-int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp);
+int xdp_do_redirect(struct net_device *dev,
+ struct xdp_buff *xdp,
+ struct bpf_prog *prog);
void bpf_warn_invalid_xdp_action(u32 act);
void bpf_warn_invalid_xdp_redirect(u32 ifindex);
diff --git a/include/trace/events/xdp.h b/include/trace/events/xdp.h
index 1b61357..7b1eb7b 100644
--- a/include/trace/events/xdp.h
+++ b/include/trace/events/xdp.h
@@ -12,7 +12,8 @@
FN(ABORTED) \
FN(DROP) \
FN(PASS) \
- FN(TX)
+ FN(TX) \
+ FN(REDIRECT)
#define __XDP_ACT_TP_FN(x) \
TRACE_DEFINE_ENUM(XDP_##x);
@@ -48,6 +49,34 @@
__print_symbolic(__entry->act, __XDP_ACT_SYM_TAB))
);
+TRACE_EVENT(xdp_redirect,
+
+ TP_PROTO(const struct net_device *from,
+ const struct net_device *to,
+ const struct bpf_prog *xdp, u32 act),
+
+ TP_ARGS(from, to, xdp, act),
+
+ TP_STRUCT__entry(
+ __string(name_from, from->name)
+ __string(name_to, to->name)
+ __array(u8, prog_tag, 8)
+ __field(u32, act)
+ ),
+
+ TP_fast_assign(
+ BUILD_BUG_ON(sizeof(__entry->prog_tag) != sizeof(xdp->tag));
+ memcpy(__entry->prog_tag, xdp->tag, sizeof(xdp->tag));
+ __assign_str(name_from, from->name);
+ __assign_str(name_to, to->name);
+ __entry->act = act;
+ ),
+
+ TP_printk("prog=%s from=%s to=%s action=%s",
+ __print_hex_str(__entry->prog_tag, 8),
+ __get_str(name_from), __get_str(name_to),
+ __print_symbolic(__entry->act, __XDP_ACT_SYM_TAB))
+);
#endif /* _TRACE_XDP_H */
#include <trace/define_trace.h>
diff --git a/net/core/filter.c b/net/core/filter.c
index eeb7134..e30d38b 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -55,6 +55,7 @@
#include <net/sock_reuseport.h>
#include <net/busy_poll.h>
#include <net/tcp.h>
+#include <linux/bpf_trace.h>
/**
* sk_filter_trim_cap - run a packet through a socket filter
@@ -2422,18 +2423,22 @@ static int __bpf_tx_xdp(struct net_device *dev, struct xdp_buff *xdp)
return -EOPNOTSUPP;
}
-int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp)
+int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp,
+ struct bpf_prog *xdp_prog)
{
struct redirect_info *ri = this_cpu_ptr(&redirect_info);
+ struct net_device *fwd;
- dev = dev_get_by_index_rcu(dev_net(dev), ri->ifindex);
+ fwd = dev_get_by_index_rcu(dev_net(dev), ri->ifindex);
ri->ifindex = 0;
- if (unlikely(!dev)) {
+ if (unlikely(!fwd)) {
bpf_warn_invalid_xdp_redirect(ri->ifindex);
return -EINVAL;
}
- return __bpf_tx_xdp(dev, xdp);
+ trace_xdp_redirect(dev, fwd, xdp_prog, XDP_REDIRECT);
+
+ return __bpf_tx_xdp(fwd, xdp);
}
EXPORT_SYMBOL_GPL(xdp_do_redirect);
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [net-next PATCH 08/12] bpf: add devmap, a map for storing net device references
2017-07-17 16:26 [net-next PATCH 00/12] Implement XDP bpf_redirect John Fastabend
` (6 preceding siblings ...)
2017-07-17 16:28 ` [net-next PATCH 07/12] xdp: add trace event " John Fastabend
@ 2017-07-17 16:28 ` John Fastabend
2017-07-17 16:29 ` [net-next PATCH 09/12] bpf: add bpf_redirect_map helper routine John Fastabend
` (4 subsequent siblings)
12 siblings, 0 replies; 19+ messages in thread
From: John Fastabend @ 2017-07-17 16:28 UTC (permalink / raw)
To: davem; +Cc: daniel, ast, netdev, john.fastabend, brouer, andy
Device map (devmap) is a BPF map, primarily useful for networking
applications, that uses a key to lookup a reference to a netdevice.
The map provides a clean way for BPF programs to build virtual port
to physical port maps. Additionally, it provides a scoping function
for the redirect action itself allowing multiple optimizations. Future
patches will leverage the map to provide batching at the XDP layer.
Another optimization/feature, that is not yet implemented, would be
to support multiple netdevices per key to support efficient multicast
and broadcast support.
Signed-off-by: John Fastabend <john.fastabend@gmail.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
include/linux/bpf_types.h | 3
include/uapi/linux/bpf.h | 1
kernel/bpf/Makefile | 3
kernel/bpf/devmap.c | 264 +++++++++++++++++++++++++++++++
kernel/bpf/verifier.c | 8 +
tools/testing/selftests/bpf/test_maps.c | 15 ++
6 files changed, 294 insertions(+)
create mode 100644 kernel/bpf/devmap.c
diff --git a/include/linux/bpf_types.h b/include/linux/bpf_types.h
index 3d137c3..b1e1035 100644
--- a/include/linux/bpf_types.h
+++ b/include/linux/bpf_types.h
@@ -35,3 +35,6 @@
#endif
BPF_MAP_TYPE(BPF_MAP_TYPE_ARRAY_OF_MAPS, array_of_maps_map_ops)
BPF_MAP_TYPE(BPF_MAP_TYPE_HASH_OF_MAPS, htab_of_maps_map_ops)
+#ifdef CONFIG_NET
+BPF_MAP_TYPE(BPF_MAP_TYPE_DEVMAP, dev_map_ops)
+#endif
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 4dbb7a3..ecbb0e7 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -104,6 +104,7 @@ enum bpf_map_type {
BPF_MAP_TYPE_LPM_TRIE,
BPF_MAP_TYPE_ARRAY_OF_MAPS,
BPF_MAP_TYPE_HASH_OF_MAPS,
+ BPF_MAP_TYPE_DEVMAP,
};
enum bpf_prog_type {
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index e1e5e65..48e9270 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -2,6 +2,9 @@ obj-y := core.o
obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o
obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o
+ifeq ($(CONFIG_NET),y)
+obj-$(CONFIG_BPF_SYSCALL) += devmap.o
+endif
ifeq ($(CONFIG_PERF_EVENTS),y)
obj-$(CONFIG_BPF_SYSCALL) += stackmap.o
endif
diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c
new file mode 100644
index 0000000..1a87835
--- /dev/null
+++ b/kernel/bpf/devmap.c
@@ -0,0 +1,264 @@
+/* Copyright (c) 2017 Covalent IO, Inc. http://covalent.io
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+
+/* Devmaps primary use is as a backend map for XDP BPF helper call
+ * bpf_redirect_map(). Because XDP is mostly concerned with performance we
+ * spent some effort to ensure the datapath with redirect maps does not use
+ * any locking. This is a quick note on the details.
+ *
+ * We have three possible paths to get into the devmap control plane bpf
+ * syscalls, bpf programs, and driver side xmit/flush operations. A bpf syscall
+ * will invoke an update, delete, or lookup operation. To ensure updates and
+ * deletes appear atomic from the datapath side xchg() is used to modify the
+ * netdev_map array. Then because the datapath does a lookup into the netdev_map
+ * array (read-only) from an RCU critical section we use call_rcu() to wait for
+ * an rcu grace period before free'ing the old data structures. This ensures the
+ * datapath always has a valid copy. However, the datapath does a "flush"
+ * operation that pushes any pending packets in the driver outside the RCU
+ * critical section. Each bpf_dtab_netdev tracks these pending operations using
+ * an atomic per-cpu bitmap. The bpf_dtab_netdev object will not be destroyed
+ * until all bits are cleared indicating outstanding flush operations have
+ * completed.
+ *
+ * BPF syscalls may race with BPF program calls on any of the update, delete
+ * or lookup operations. As noted above the xchg() operation also keep the
+ * netdev_map consistent in this case. From the devmap side BPF programs
+ * calling into these operations are the same as multiple user space threads
+ * making system calls.
+ */
+#include <linux/bpf.h>
+#include <linux/jhash.h>
+#include <linux/filter.h>
+#include <linux/rculist_nulls.h>
+#include "percpu_freelist.h"
+#include "bpf_lru_list.h"
+#include "map_in_map.h"
+
+struct bpf_dtab_netdev {
+ struct net_device *dev;
+ int key;
+ struct rcu_head rcu;
+ struct bpf_dtab *dtab;
+};
+
+struct bpf_dtab {
+ struct bpf_map map;
+ struct bpf_dtab_netdev **netdev_map;
+};
+
+static struct bpf_map *dev_map_alloc(union bpf_attr *attr)
+{
+ struct bpf_dtab *dtab;
+ u64 cost;
+ int err;
+
+ /* check sanity of attributes */
+ if (attr->max_entries == 0 || attr->key_size != 4 ||
+ attr->value_size != 4 || attr->map_flags)
+ return ERR_PTR(-EINVAL);
+
+ /* if value_size is bigger, the user space won't be able to
+ * access the elements.
+ */
+ if (attr->value_size > KMALLOC_MAX_SIZE)
+ return ERR_PTR(-E2BIG);
+
+ dtab = kzalloc(sizeof(*dtab), GFP_USER);
+ if (!dtab)
+ return ERR_PTR(-ENOMEM);
+
+ /* mandatory map attributes */
+ dtab->map.map_type = attr->map_type;
+ dtab->map.key_size = attr->key_size;
+ dtab->map.value_size = attr->value_size;
+ dtab->map.max_entries = attr->max_entries;
+ dtab->map.map_flags = attr->map_flags;
+
+ err = -ENOMEM;
+
+ /* make sure page count doesn't overflow */
+ cost = (u64) dtab->map.max_entries * sizeof(struct bpf_dtab_netdev *);
+ if (cost >= U32_MAX - PAGE_SIZE)
+ goto free_dtab;
+
+ dtab->map.pages = round_up(cost, PAGE_SIZE) >> PAGE_SHIFT;
+
+ /* if map size is larger than memlock limit, reject it early */
+ err = bpf_map_precharge_memlock(dtab->map.pages);
+ if (err)
+ goto free_dtab;
+
+ dtab->netdev_map = bpf_map_area_alloc(dtab->map.max_entries *
+ sizeof(struct bpf_dtab_netdev *));
+ if (!dtab->netdev_map)
+ goto free_dtab;
+
+ return &dtab->map;
+
+free_dtab:
+ kfree(dtab);
+ return ERR_PTR(err);
+}
+
+static void dev_map_free(struct bpf_map *map)
+{
+ struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);
+ int i;
+
+ /* At this point bpf_prog->aux->refcnt == 0 and this map->refcnt == 0,
+ * so the programs (can be more than one that used this map) were
+ * disconnected from events. Wait for outstanding critical sections in
+ * these programs to complete. The rcu critical section only guarantees
+ * no further reads against netdev_map. It does __not__ ensure pending
+ * flush operations (if any) are complete.
+ */
+ synchronize_rcu();
+
+ for (i = 0; i < dtab->map.max_entries; i++) {
+ struct bpf_dtab_netdev *dev;
+
+ dev = dtab->netdev_map[i];
+ if (!dev)
+ continue;
+
+ dev_put(dev->dev);
+ kfree(dev);
+ }
+
+ /* At this point bpf program is detached and all pending operations
+ * _must_ be complete
+ */
+ bpf_map_area_free(dtab->netdev_map);
+ kfree(dtab);
+}
+
+static int dev_map_get_next_key(struct bpf_map *map, void *key, void *next_key)
+{
+ struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);
+ u32 index = key ? *(u32 *)key : U32_MAX;
+ u32 *next = (u32 *)next_key;
+
+ if (index >= dtab->map.max_entries) {
+ *next = 0;
+ return 0;
+ }
+
+ if (index == dtab->map.max_entries - 1)
+ return -ENOENT;
+
+ *next = index + 1;
+ return 0;
+}
+
+/* rcu_read_lock (from syscall and BPF contexts) ensures that if a delete and/or
+ * update happens in parallel here a dev_put wont happen until after reading the
+ * ifindex.
+ */
+static void *dev_map_lookup_elem(struct bpf_map *map, void *key)
+{
+ struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);
+ struct bpf_dtab_netdev *dev;
+ u32 i = *(u32 *)key;
+
+ if (i >= map->max_entries)
+ return NULL;
+
+ dev = READ_ONCE(dtab->netdev_map[i]);
+ return dev ? &dev->dev->ifindex : NULL;
+}
+
+static void __dev_map_entry_free(struct rcu_head *rcu)
+{
+ struct bpf_dtab_netdev *old_dev;
+
+ old_dev = container_of(rcu, struct bpf_dtab_netdev, rcu);
+ dev_put(old_dev->dev);
+ kfree(old_dev);
+}
+
+static int dev_map_delete_elem(struct bpf_map *map, void *key)
+{
+ struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);
+ struct bpf_dtab_netdev *old_dev;
+ int k = *(u32 *)key;
+
+ if (k >= map->max_entries)
+ return -EINVAL;
+
+ /* Use synchronize_rcu() here to ensure any rcu critical sections
+ * have completed, but this does not guarantee a flush has happened
+ * yet. Because driver side rcu_read_lock/unlock only protects the
+ * running XDP program. However, for pending flush operations the
+ * dev and ctx are stored in another per cpu map. And additionally,
+ * the driver tear down ensures all soft irqs are complete before
+ * removing the net device in the case of dev_put equals zero.
+ */
+ old_dev = xchg(&dtab->netdev_map[k], NULL);
+ if (old_dev)
+ call_rcu(&old_dev->rcu, __dev_map_entry_free);
+ return 0;
+}
+
+static int dev_map_update_elem(struct bpf_map *map, void *key, void *value,
+ u64 map_flags)
+{
+ struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);
+ struct net *net = current->nsproxy->net_ns;
+ struct bpf_dtab_netdev *dev, *old_dev;
+ u32 i = *(u32 *)key;
+ u32 ifindex = *(u32 *)value;
+
+ if (unlikely(map_flags > BPF_EXIST))
+ return -EINVAL;
+
+ if (unlikely(i >= dtab->map.max_entries))
+ return -E2BIG;
+
+ if (unlikely(map_flags == BPF_NOEXIST))
+ return -EEXIST;
+
+ if (!ifindex) {
+ dev = NULL;
+ } else {
+ dev = kmalloc(sizeof(*dev), GFP_ATOMIC | __GFP_NOWARN);
+ if (!dev)
+ return -ENOMEM;
+
+ dev->dev = dev_get_by_index(net, ifindex);
+ if (!dev->dev) {
+ kfree(dev);
+ return -EINVAL;
+ }
+
+ dev->key = i;
+ dev->dtab = dtab;
+ }
+
+ /* Use call_rcu() here to ensure rcu critical sections have completed
+ * Remembering the driver side flush operation will happen before the
+ * net device is removed.
+ */
+ old_dev = xchg(&dtab->netdev_map[i], dev);
+ if (old_dev)
+ call_rcu(&old_dev->rcu, __dev_map_entry_free);
+
+ return 0;
+}
+
+const struct bpf_map_ops dev_map_ops = {
+ .map_alloc = dev_map_alloc,
+ .map_free = dev_map_free,
+ .map_get_next_key = dev_map_get_next_key,
+ .map_lookup_elem = dev_map_lookup_elem,
+ .map_update_elem = dev_map_update_elem,
+ .map_delete_elem = dev_map_delete_elem,
+};
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 6a86723..4016774 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -1276,6 +1276,14 @@ static int check_map_func_compatibility(struct bpf_map *map, int func_id)
func_id != BPF_FUNC_current_task_under_cgroup)
goto error;
break;
+ /* devmap returns a pointer to a live net_device ifindex that we cannot
+ * allow to be modified from bpf side. So do not allow lookup elements
+ * for now.
+ */
+ case BPF_MAP_TYPE_DEVMAP:
+ if (func_id == BPF_FUNC_map_lookup_elem)
+ goto error;
+ break;
case BPF_MAP_TYPE_ARRAY_OF_MAPS:
case BPF_MAP_TYPE_HASH_OF_MAPS:
if (func_id != BPF_FUNC_map_lookup_elem)
diff --git a/tools/testing/selftests/bpf/test_maps.c b/tools/testing/selftests/bpf/test_maps.c
index 79601c8..36d6ac3 100644
--- a/tools/testing/selftests/bpf/test_maps.c
+++ b/tools/testing/selftests/bpf/test_maps.c
@@ -438,6 +438,21 @@ static void test_arraymap_percpu_many_keys(void)
close(fd);
}
+static void test_devmap(int task, void *data)
+{
+ int next_key, fd;
+ __u32 key, value;
+
+ fd = bpf_create_map(BPF_MAP_TYPE_DEVMAP, sizeof(key), sizeof(value),
+ 2, 0);
+ if (fd < 0) {
+ printf("Failed to create arraymap '%s'!\n", strerror(errno));
+ exit(1);
+ }
+
+ close(fd);
+}
+
#define MAP_SIZE (32 * 1024)
static void test_map_large(void)
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [net-next PATCH 09/12] bpf: add bpf_redirect_map helper routine
2017-07-17 16:26 [net-next PATCH 00/12] Implement XDP bpf_redirect John Fastabend
` (7 preceding siblings ...)
2017-07-17 16:28 ` [net-next PATCH 08/12] bpf: add devmap, a map for storing net device references John Fastabend
@ 2017-07-17 16:29 ` John Fastabend
2017-07-17 17:00 ` Alexei Starovoitov
2017-07-17 16:29 ` [net-next PATCH 10/12] xdp: Add batching support to redirect map John Fastabend
` (3 subsequent siblings)
12 siblings, 1 reply; 19+ messages in thread
From: John Fastabend @ 2017-07-17 16:29 UTC (permalink / raw)
To: davem; +Cc: daniel, ast, netdev, john.fastabend, brouer, andy
BPF programs can use the devmap with a bpf_redirect_map() helper
routine to forward packets to netdevice in map.
Signed-off-by: John Fastabend <john.fastabend@gmail.com>
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
---
include/linux/bpf.h | 3 +++
include/uapi/linux/bpf.h | 8 ++++++-
kernel/bpf/devmap.c | 12 +++++++++++
kernel/bpf/verifier.c | 4 ++++
net/core/filter.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++
5 files changed, 78 insertions(+), 1 deletion(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index b69e7a5..d0d3281 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -379,4 +379,7 @@ static inline void __bpf_prog_uncharge(struct user_struct *user, u32 pages)
void bpf_user_rnd_init_once(void);
u64 bpf_user_rnd_u32(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
+/* Map specifics */
+struct net_device *__dev_map_lookup_elem(struct bpf_map *map, u32 key);
+
#endif /* _LINUX_BPF_H */
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index ecbb0e7..1106a8c 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -348,6 +348,11 @@ enum bpf_attach_type {
* @flags: bit 0 - if set, redirect to ingress instead of egress
* other bits - reserved
* Return: TC_ACT_REDIRECT
+ * int bpf_redirect_map(key, map, flags)
+ * redirect to endpoint in map
+ * @key: index in map to lookup
+ * @map: fd of map to do lookup in
+ * @flags: --
*
* u32 bpf_get_route_realm(skb)
* retrieve a dst's tclassid
@@ -592,7 +597,8 @@ enum bpf_attach_type {
FN(get_socket_uid), \
FN(set_hash), \
FN(setsockopt), \
- FN(skb_adjust_room),
+ FN(skb_adjust_room), \
+ FN(redirect_map),
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
* function eBPF program intends to call
diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c
index 1a87835..36dc13de 100644
--- a/kernel/bpf/devmap.c
+++ b/kernel/bpf/devmap.c
@@ -159,6 +159,18 @@ static int dev_map_get_next_key(struct bpf_map *map, void *key, void *next_key)
return 0;
}
+struct net_device *__dev_map_lookup_elem(struct bpf_map *map, u32 key)
+{
+ struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);
+ struct bpf_dtab_netdev *dev;
+
+ if (key >= map->max_entries)
+ return NULL;
+
+ dev = READ_ONCE(dtab->netdev_map[key]);
+ return dev ? dev->dev : NULL;
+}
+
/* rcu_read_lock (from syscall and BPF contexts) ensures that if a delete and/or
* update happens in parallel here a dev_put wont happen until after reading the
* ifindex.
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 4016774..df05d65 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -1312,6 +1312,10 @@ static int check_map_func_compatibility(struct bpf_map *map, int func_id)
if (map->map_type != BPF_MAP_TYPE_CGROUP_ARRAY)
goto error;
break;
+ case BPF_FUNC_redirect_map:
+ if (map->map_type != BPF_MAP_TYPE_DEVMAP)
+ goto error;
+ break;
default:
break;
}
diff --git a/net/core/filter.c b/net/core/filter.c
index e30d38b..e93a558 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -1779,6 +1779,7 @@ static int __bpf_redirect(struct sk_buff *skb, struct net_device *dev,
struct redirect_info {
u32 ifindex;
u32 flags;
+ struct bpf_map *map;
};
static DEFINE_PER_CPU(struct redirect_info, redirect_info);
@@ -1792,6 +1793,7 @@ struct redirect_info {
ri->ifindex = ifindex;
ri->flags = flags;
+ ri->map = NULL;
return TC_ACT_REDIRECT;
}
@@ -1819,6 +1821,29 @@ int skb_do_redirect(struct sk_buff *skb)
.arg2_type = ARG_ANYTHING,
};
+BPF_CALL_3(bpf_redirect_map, struct bpf_map *, map, u32, ifindex, u64, flags)
+{
+ struct redirect_info *ri = this_cpu_ptr(&redirect_info);
+
+ if (unlikely(flags))
+ return XDP_ABORTED;
+
+ ri->ifindex = ifindex;
+ ri->flags = flags;
+ ri->map = map;
+
+ return XDP_REDIRECT;
+}
+
+static const struct bpf_func_proto bpf_redirect_map_proto = {
+ .func = bpf_redirect_map,
+ .gpl_only = false,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_CONST_MAP_PTR,
+ .arg2_type = ARG_ANYTHING,
+ .arg3_type = ARG_ANYTHING,
+};
+
BPF_CALL_1(bpf_get_cgroup_classid, const struct sk_buff *, skb)
{
return task_get_classid(skb);
@@ -2423,14 +2448,39 @@ static int __bpf_tx_xdp(struct net_device *dev, struct xdp_buff *xdp)
return -EOPNOTSUPP;
}
+int xdp_do_redirect_map(struct net_device *dev, struct xdp_buff *xdp,
+ struct bpf_prog *xdp_prog)
+{
+ struct redirect_info *ri = this_cpu_ptr(&redirect_info);
+ struct bpf_map *map = ri->map;
+ struct net_device *fwd;
+ int err = -EINVAL;
+
+ ri->ifindex = 0;
+ ri->map = NULL;
+
+ fwd = __dev_map_lookup_elem(map, ri->ifindex);
+ if (!fwd)
+ goto out;
+
+ trace_xdp_redirect(dev, fwd, xdp_prog, XDP_REDIRECT);
+ err = __bpf_tx_xdp(fwd, xdp);
+out:
+ return err;
+}
+
int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp,
struct bpf_prog *xdp_prog)
{
struct redirect_info *ri = this_cpu_ptr(&redirect_info);
struct net_device *fwd;
+ if (ri->map)
+ return xdp_do_redirect_map(dev, xdp, xdp_prog);
+
fwd = dev_get_by_index_rcu(dev_net(dev), ri->ifindex);
ri->ifindex = 0;
+ ri->map = NULL;
if (unlikely(!fwd)) {
bpf_warn_invalid_xdp_redirect(ri->ifindex);
return -EINVAL;
@@ -3089,6 +3139,8 @@ static unsigned long bpf_xdp_copy(void *dst_buff, const void *src_buff,
return &bpf_xdp_adjust_head_proto;
case BPF_FUNC_redirect:
return &bpf_xdp_redirect_proto;
+ case BPF_FUNC_redirect_map:
+ return &bpf_redirect_map_proto;
default:
return bpf_base_func_proto(func_id);
}
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [net-next PATCH 10/12] xdp: Add batching support to redirect map
2017-07-17 16:26 [net-next PATCH 00/12] Implement XDP bpf_redirect John Fastabend
` (8 preceding siblings ...)
2017-07-17 16:29 ` [net-next PATCH 09/12] bpf: add bpf_redirect_map helper routine John Fastabend
@ 2017-07-17 16:29 ` John Fastabend
2017-07-17 16:30 ` [net-next PATCH 11/12] net: add notifier hooks for devmap bpf map John Fastabend
` (2 subsequent siblings)
12 siblings, 0 replies; 19+ messages in thread
From: John Fastabend @ 2017-07-17 16:29 UTC (permalink / raw)
To: davem; +Cc: daniel, ast, netdev, john.fastabend, brouer, andy
For performance reasons we want to avoid updating the tail pointer in
the driver tx ring as much as possible. To accomplish this we add
batching support to the redirect path in XDP.
This adds another ndo op "xdp_flush" that is used to inform the driver
that it should bump the tail pointer on the TX ring.
Signed-off-by: John Fastabend <john.fastabend@gmail.com>
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
---
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 28 +++++++-
include/linux/bpf.h | 2 +
include/linux/filter.h | 7 ++
include/linux/netdevice.h | 5 +
kernel/bpf/devmap.c | 84 +++++++++++++++++++++++++
net/core/filter.c | 55 +++++++++++++---
6 files changed, 166 insertions(+), 15 deletions(-)
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
index 38f7ff9..0f867dc 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
@@ -2415,6 +2415,8 @@ static int ixgbe_clean_rx_irq(struct ixgbe_q_vector *q_vector,
*/
wmb();
writel(ring->next_to_use, ring->tail);
+
+ xdp_do_flush_map();
}
u64_stats_update_begin(&rx_ring->syncp);
@@ -5817,6 +5819,9 @@ void ixgbe_down(struct ixgbe_adapter *adapter)
usleep_range(10000, 20000);
+ /* synchronize_sched() needed for pending XDP buffers to drain */
+ if (adapter->xdp_ring[0])
+ synchronize_sched();
netif_tx_stop_all_queues(netdev);
/* call carrier off first to avoid false dev_watchdog timeouts */
@@ -9850,15 +9855,31 @@ static int ixgbe_xdp_xmit(struct net_device *dev, struct xdp_buff *xdp)
if (err != IXGBE_XDP_TX)
return -ENOMEM;
+ return 0;
+}
+
+static void ixgbe_xdp_flush(struct net_device *dev)
+{
+ struct ixgbe_adapter *adapter = netdev_priv(dev);
+ struct ixgbe_ring *ring;
+
+ /* Its possible the device went down between xdp xmit and flush so
+ * we need to ensure device is still up.
+ */
+ if (unlikely(test_bit(__IXGBE_DOWN, &adapter->state)))
+ return;
+
+ ring = adapter->xdp_prog ? adapter->xdp_ring[smp_processor_id()] : NULL;
+ if (unlikely(!ring))
+ return;
+
/* Force memory writes to complete before letting h/w know there
* are new descriptors to fetch.
*/
wmb();
-
- ring = adapter->xdp_ring[smp_processor_id()];
writel(ring->next_to_use, ring->tail);
- return 0;
+ return;
}
static const struct net_device_ops ixgbe_netdev_ops = {
@@ -9908,6 +9929,7 @@ static int ixgbe_xdp_xmit(struct net_device *dev, struct xdp_buff *xdp)
.ndo_features_check = ixgbe_features_check,
.ndo_xdp = ixgbe_xdp,
.ndo_xdp_xmit = ixgbe_xdp_xmit,
+ .ndo_xdp_flush = ixgbe_xdp_flush,
};
/**
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index d0d3281..6850a76 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -381,5 +381,7 @@ static inline void __bpf_prog_uncharge(struct user_struct *user, u32 pages)
/* Map specifics */
struct net_device *__dev_map_lookup_elem(struct bpf_map *map, u32 key);
+void __dev_map_insert_ctx(struct bpf_map *map, u32 index);
+void __dev_map_flush(struct bpf_map *map);
#endif /* _LINUX_BPF_H */
diff --git a/include/linux/filter.h b/include/linux/filter.h
index ce8211f..3323ee9 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -712,10 +712,17 @@ int sk_get_filter(struct sock *sk, struct sock_filter __user *filter,
struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off,
const struct bpf_insn *patch, u32 len);
+/* The pair of xdp_do_redirect and xdp_do_flush_map MUST be called in the
+ * same cpu context. Further for best results no more than a single map
+ * for the do_redirect/do_flush pair should be used. This limitation is
+ * because we only track one map and force a flush when the map changes.
+ * This does not appear to be a real limiation for existing software.
+ */
int xdp_do_generic_redirect(struct net_device *dev, struct sk_buff *skb);
int xdp_do_redirect(struct net_device *dev,
struct xdp_buff *xdp,
struct bpf_prog *prog);
+void xdp_do_flush_map(void);
void bpf_warn_invalid_xdp_action(u32 act);
void bpf_warn_invalid_xdp_redirect(u32 ifindex);
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 77f5376..03b1049 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -1142,7 +1142,9 @@ struct xfrmdev_ops {
* int (*ndo_xdp_xmit)(struct net_device *dev, struct xdp_buff *xdp);
* This function is used to submit a XDP packet for transmit on a
* netdevice.
- *
+ * void (*ndo_xdp_flush)(struct net_device *dev);
+ * This function is used to inform the driver to flush a paticular
+ * xpd tx queue. Must be called on same CPU as xdp_xmit.
*/
struct net_device_ops {
int (*ndo_init)(struct net_device *dev);
@@ -1329,6 +1331,7 @@ struct net_device_ops {
struct netdev_xdp *xdp);
int (*ndo_xdp_xmit)(struct net_device *dev,
struct xdp_buff *xdp);
+ void (*ndo_xdp_flush)(struct net_device *dev);
};
/**
diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c
index 36dc13de..b2ef04a 100644
--- a/kernel/bpf/devmap.c
+++ b/kernel/bpf/devmap.c
@@ -53,6 +53,7 @@ struct bpf_dtab_netdev {
struct bpf_dtab {
struct bpf_map map;
struct bpf_dtab_netdev **netdev_map;
+ unsigned long int __percpu *flush_needed;
};
static struct bpf_map *dev_map_alloc(union bpf_attr *attr)
@@ -87,6 +88,7 @@ static struct bpf_map *dev_map_alloc(union bpf_attr *attr)
/* make sure page count doesn't overflow */
cost = (u64) dtab->map.max_entries * sizeof(struct bpf_dtab_netdev *);
+ cost += BITS_TO_LONGS(attr->max_entries) * sizeof(unsigned long);
if (cost >= U32_MAX - PAGE_SIZE)
goto free_dtab;
@@ -97,6 +99,14 @@ static struct bpf_map *dev_map_alloc(union bpf_attr *attr)
if (err)
goto free_dtab;
+ /* A per cpu bitfield with a bit per possible net device */
+ dtab->flush_needed = __alloc_percpu(
+ BITS_TO_LONGS(attr->max_entries) *
+ sizeof(unsigned long),
+ __alignof__(unsigned long));
+ if (!dtab->flush_needed)
+ goto free_dtab;
+
dtab->netdev_map = bpf_map_area_alloc(dtab->map.max_entries *
sizeof(struct bpf_dtab_netdev *));
if (!dtab->netdev_map)
@@ -105,6 +115,7 @@ static struct bpf_map *dev_map_alloc(union bpf_attr *attr)
return &dtab->map;
free_dtab:
+ free_percpu(dtab->flush_needed);
kfree(dtab);
return ERR_PTR(err);
}
@@ -112,7 +123,7 @@ static struct bpf_map *dev_map_alloc(union bpf_attr *attr)
static void dev_map_free(struct bpf_map *map)
{
struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);
- int i;
+ int i, cpu;
/* At this point bpf_prog->aux->refcnt == 0 and this map->refcnt == 0,
* so the programs (can be more than one that used this map) were
@@ -123,6 +134,18 @@ static void dev_map_free(struct bpf_map *map)
*/
synchronize_rcu();
+ /* To ensure all pending flush operations have completed wait for flush
+ * bitmap to indicate all flush_needed bits to be zero on _all_ cpus.
+ * Because the above synchronize_rcu() ensures the map is disconnected
+ * from the program we can assume no new bits will be set.
+ */
+ for_each_online_cpu(cpu) {
+ unsigned long *bitmap = per_cpu_ptr(dtab->flush_needed, cpu);
+
+ while (!bitmap_empty(bitmap, dtab->map.max_entries))
+ cpu_relax();
+ }
+
for (i = 0; i < dtab->map.max_entries; i++) {
struct bpf_dtab_netdev *dev;
@@ -137,6 +160,7 @@ static void dev_map_free(struct bpf_map *map)
/* At this point bpf program is detached and all pending operations
* _must_ be complete
*/
+ free_percpu(dtab->flush_needed);
bpf_map_area_free(dtab->netdev_map);
kfree(dtab);
}
@@ -159,6 +183,14 @@ static int dev_map_get_next_key(struct bpf_map *map, void *key, void *next_key)
return 0;
}
+void __dev_map_insert_ctx(struct bpf_map *map, u32 key)
+{
+ struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);
+ unsigned long *bitmap = this_cpu_ptr(dtab->flush_needed);
+
+ __set_bit(key, bitmap);
+}
+
struct net_device *__dev_map_lookup_elem(struct bpf_map *map, u32 key)
{
struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);
@@ -171,6 +203,39 @@ struct net_device *__dev_map_lookup_elem(struct bpf_map *map, u32 key)
return dev ? dev->dev : NULL;
}
+/* __dev_map_flush is called from xdp_do_flush_map() which _must_ be signaled
+ * from the driver before returning from its napi->poll() routine. The poll()
+ * routine is called either from busy_poll context or net_rx_action signaled
+ * from NET_RX_SOFTIRQ. Either way the poll routine must complete before the
+ * net device can be torn down. On devmap tear down we ensure the ctx bitmap
+ * is zeroed before completing to ensure all flush operations have completed.
+ */
+void __dev_map_flush(struct bpf_map *map)
+{
+ struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);
+ unsigned long *bitmap = this_cpu_ptr(dtab->flush_needed);
+ u32 bit;
+
+ for_each_set_bit(bit, bitmap, map->max_entries) {
+ struct bpf_dtab_netdev *dev = READ_ONCE(dtab->netdev_map[bit]);
+ struct net_device *netdev;
+
+ /* This is possible if the dev entry is removed by user space
+ * between xdp redirect and flush op.
+ */
+ if (unlikely(!dev))
+ continue;
+
+ netdev = dev->dev;
+
+ __clear_bit(bit, bitmap);
+ if (unlikely(!netdev || !netdev->netdev_ops->ndo_xdp_flush))
+ continue;
+
+ netdev->netdev_ops->ndo_xdp_flush(netdev);
+ }
+}
+
/* rcu_read_lock (from syscall and BPF contexts) ensures that if a delete and/or
* update happens in parallel here a dev_put wont happen until after reading the
* ifindex.
@@ -188,11 +253,28 @@ static void *dev_map_lookup_elem(struct bpf_map *map, void *key)
return dev ? &dev->dev->ifindex : NULL;
}
+static void dev_map_flush_old(struct bpf_dtab_netdev *old_dev)
+{
+ if (old_dev->dev->netdev_ops->ndo_xdp_flush) {
+ struct net_device *fl = old_dev->dev;
+ unsigned long *bitmap;
+ int cpu;
+
+ for_each_online_cpu(cpu) {
+ bitmap = per_cpu_ptr(old_dev->dtab->flush_needed, cpu);
+ __clear_bit(old_dev->key, bitmap);
+
+ fl->netdev_ops->ndo_xdp_flush(old_dev->dev);
+ }
+ }
+}
+
static void __dev_map_entry_free(struct rcu_head *rcu)
{
struct bpf_dtab_netdev *old_dev;
old_dev = container_of(rcu, struct bpf_dtab_netdev, rcu);
+ dev_map_flush_old(old_dev);
dev_put(old_dev->dev);
kfree(old_dev);
}
diff --git a/net/core/filter.c b/net/core/filter.c
index e93a558..e23aa6f 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -1780,6 +1780,7 @@ struct redirect_info {
u32 ifindex;
u32 flags;
struct bpf_map *map;
+ struct bpf_map *map_to_flush;
};
static DEFINE_PER_CPU(struct redirect_info, redirect_info);
@@ -2438,34 +2439,68 @@ static int bpf_skb_trim_rcsum(struct sk_buff *skb, unsigned int new_len)
.arg2_type = ARG_ANYTHING,
};
-static int __bpf_tx_xdp(struct net_device *dev, struct xdp_buff *xdp)
+static int __bpf_tx_xdp(struct net_device *dev,
+ struct bpf_map *map,
+ struct xdp_buff *xdp,
+ u32 index)
{
- if (dev->netdev_ops->ndo_xdp_xmit) {
- dev->netdev_ops->ndo_xdp_xmit(dev, xdp);
- return 0;
+ int err;
+
+ if (!dev->netdev_ops->ndo_xdp_xmit) {
+ bpf_warn_invalid_xdp_redirect(dev->ifindex);
+ return -EOPNOTSUPP;
}
- bpf_warn_invalid_xdp_redirect(dev->ifindex);
- return -EOPNOTSUPP;
+
+ err = dev->netdev_ops->ndo_xdp_xmit(dev, xdp);
+ if (err)
+ return err;
+
+ if (map)
+ __dev_map_insert_ctx(map, index);
+ else
+ dev->netdev_ops->ndo_xdp_flush(dev);
+
+ return err;
}
+void xdp_do_flush_map(void)
+{
+ struct redirect_info *ri = this_cpu_ptr(&redirect_info);
+ struct bpf_map *map = ri->map_to_flush;
+
+ ri->map = NULL;
+ ri->map_to_flush = NULL;
+
+ if (map)
+ __dev_map_flush(map);
+}
+EXPORT_SYMBOL_GPL(xdp_do_flush_map);
+
int xdp_do_redirect_map(struct net_device *dev, struct xdp_buff *xdp,
struct bpf_prog *xdp_prog)
{
struct redirect_info *ri = this_cpu_ptr(&redirect_info);
struct bpf_map *map = ri->map;
+ u32 index = ri->ifindex;
struct net_device *fwd;
int err = -EINVAL;
ri->ifindex = 0;
ri->map = NULL;
- fwd = __dev_map_lookup_elem(map, ri->ifindex);
+ fwd = __dev_map_lookup_elem(map, index);
if (!fwd)
goto out;
- trace_xdp_redirect(dev, fwd, xdp_prog, XDP_REDIRECT);
- err = __bpf_tx_xdp(fwd, xdp);
+ if (ri->map_to_flush && (ri->map_to_flush != map))
+ xdp_do_flush_map();
+
+ err = __bpf_tx_xdp(fwd, map, xdp, index);
+ if (likely(!err))
+ ri->map_to_flush = map;
+
out:
+ trace_xdp_redirect(dev, fwd, xdp_prog, XDP_REDIRECT);
return err;
}
@@ -2488,7 +2523,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp,
trace_xdp_redirect(dev, fwd, xdp_prog, XDP_REDIRECT);
- return __bpf_tx_xdp(fwd, xdp);
+ return __bpf_tx_xdp(fwd, NULL, xdp, 0);
}
EXPORT_SYMBOL_GPL(xdp_do_redirect);
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [net-next PATCH 11/12] net: add notifier hooks for devmap bpf map
2017-07-17 16:26 [net-next PATCH 00/12] Implement XDP bpf_redirect John Fastabend
` (9 preceding siblings ...)
2017-07-17 16:29 ` [net-next PATCH 10/12] xdp: Add batching support to redirect map John Fastabend
@ 2017-07-17 16:30 ` John Fastabend
2017-07-30 13:28 ` Levin, Alexander (Sasha Levin)
2017-07-17 16:30 ` [net-next PATCH 12/12] xdp: bpf redirect with map sample program John Fastabend
2017-07-17 16:48 ` [net-next PATCH 00/12] Implement XDP bpf_redirect David Miller
12 siblings, 1 reply; 19+ messages in thread
From: John Fastabend @ 2017-07-17 16:30 UTC (permalink / raw)
To: davem; +Cc: daniel, ast, netdev, john.fastabend, brouer, andy
The BPF map devmap holds a refcnt on the net_device structure when
it is in the map. We need to do this to ensure on driver unload we
don't lose a dev reference.
However, its not very convenient to have to manually unload the map
when destroying a net device so add notifier handlers to do the cleanup
automatically. But this creates a race between update/destroy BPF
syscall and programs and the unregister netdev hook.
Unfortunately, the best I could come up with is either to live with
requiring manual removal of net devices from the map before removing
the net device OR to add a mutex in devmap to ensure the map is not
modified while we are removing a device. The fallout also requires
that BPF programs no longer update/delete the map from the BPF program
side because the mutex may sleep and this can not be done from inside
an rcu critical section. This is not a real problem though because I
have not come up with any use cases where this is actually useful in
practice. If/when we come up with a compelling user for this we may
need to revisit this.
Signed-off-by: John Fastabend <john.fastabend@gmail.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
include/linux/filter.h | 2 +
kernel/bpf/devmap.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++
kernel/bpf/verifier.c | 2 +
3 files changed, 75 insertions(+), 2 deletions(-)
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 3323ee9..d19ed3c 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -716,7 +716,7 @@ struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off,
* same cpu context. Further for best results no more than a single map
* for the do_redirect/do_flush pair should be used. This limitation is
* because we only track one map and force a flush when the map changes.
- * This does not appear to be a real limiation for existing software.
+ * This does not appear to be a real limitation for existing software.
*/
int xdp_do_generic_redirect(struct net_device *dev, struct sk_buff *skb);
int xdp_do_redirect(struct net_device *dev,
diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c
index b2ef04a..899364d 100644
--- a/kernel/bpf/devmap.c
+++ b/kernel/bpf/devmap.c
@@ -34,6 +34,17 @@
* netdev_map consistent in this case. From the devmap side BPF programs
* calling into these operations are the same as multiple user space threads
* making system calls.
+ *
+ * Finally, any of the above may race with a netdev_unregister notifier. The
+ * unregister notifier must search for net devices in the map structure that
+ * contain a reference to the net device and remove them. This is a two step
+ * process (a) dereference the bpf_dtab_netdev object in netdev_map and (b)
+ * check to see if the ifindex is the same as the net_device being removed.
+ * Unfortunately, the xchg() operations do not protect against this. To avoid
+ * potentially removing incorrect objects the dev_map_list_mutex protects
+ * conflicting netdev unregister and BPF syscall operations. Updates and
+ * deletes from a BPF program (done in rcu critical section) are blocked
+ * because of this mutex.
*/
#include <linux/bpf.h>
#include <linux/jhash.h>
@@ -54,8 +65,12 @@ struct bpf_dtab {
struct bpf_map map;
struct bpf_dtab_netdev **netdev_map;
unsigned long int __percpu *flush_needed;
+ struct list_head list;
};
+static DEFINE_MUTEX(dev_map_list_mutex);
+static LIST_HEAD(dev_map_list);
+
static struct bpf_map *dev_map_alloc(union bpf_attr *attr)
{
struct bpf_dtab *dtab;
@@ -112,6 +127,9 @@ static struct bpf_map *dev_map_alloc(union bpf_attr *attr)
if (!dtab->netdev_map)
goto free_dtab;
+ mutex_lock(&dev_map_list_mutex);
+ list_add_tail(&dtab->list, &dev_map_list);
+ mutex_unlock(&dev_map_list_mutex);
return &dtab->map;
free_dtab:
@@ -146,6 +164,11 @@ static void dev_map_free(struct bpf_map *map)
cpu_relax();
}
+ /* Although we should no longer have datapath or bpf syscall operations
+ * at this point we we can still race with netdev notifier, hence the
+ * lock.
+ */
+ mutex_lock(&dev_map_list_mutex);
for (i = 0; i < dtab->map.max_entries; i++) {
struct bpf_dtab_netdev *dev;
@@ -160,6 +183,8 @@ static void dev_map_free(struct bpf_map *map)
/* At this point bpf program is detached and all pending operations
* _must_ be complete
*/
+ list_del(&dtab->list);
+ mutex_unlock(&dev_map_list_mutex);
free_percpu(dtab->flush_needed);
bpf_map_area_free(dtab->netdev_map);
kfree(dtab);
@@ -296,9 +321,11 @@ static int dev_map_delete_elem(struct bpf_map *map, void *key)
* the driver tear down ensures all soft irqs are complete before
* removing the net device in the case of dev_put equals zero.
*/
+ mutex_lock(&dev_map_list_mutex);
old_dev = xchg(&dtab->netdev_map[k], NULL);
if (old_dev)
call_rcu(&old_dev->rcu, __dev_map_entry_free);
+ mutex_unlock(&dev_map_list_mutex);
return 0;
}
@@ -341,9 +368,11 @@ static int dev_map_update_elem(struct bpf_map *map, void *key, void *value,
* Remembering the driver side flush operation will happen before the
* net device is removed.
*/
+ mutex_lock(&dev_map_list_mutex);
old_dev = xchg(&dtab->netdev_map[i], dev);
if (old_dev)
call_rcu(&old_dev->rcu, __dev_map_entry_free);
+ mutex_unlock(&dev_map_list_mutex);
return 0;
}
@@ -356,3 +385,47 @@ static int dev_map_update_elem(struct bpf_map *map, void *key, void *value,
.map_update_elem = dev_map_update_elem,
.map_delete_elem = dev_map_delete_elem,
};
+
+static int dev_map_notification(struct notifier_block *notifier,
+ ulong event, void *ptr)
+{
+ struct net_device *netdev = netdev_notifier_info_to_dev(ptr);
+ struct bpf_dtab *dtab;
+ int i;
+
+ switch (event) {
+ case NETDEV_UNREGISTER:
+ mutex_lock(&dev_map_list_mutex);
+ list_for_each_entry(dtab, &dev_map_list, list) {
+ for (i = 0; i < dtab->map.max_entries; i++) {
+ struct bpf_dtab_netdev *dev;
+
+ dev = dtab->netdev_map[i];
+ if (!dev ||
+ dev->dev->ifindex != netdev->ifindex)
+ continue;
+ dev = xchg(&dtab->netdev_map[i], NULL);
+ if (dev)
+ call_rcu(&dev->rcu,
+ __dev_map_entry_free);
+ }
+ }
+ mutex_unlock(&dev_map_list_mutex);
+ break;
+ default:
+ break;
+ }
+ return NOTIFY_OK;
+}
+
+static struct notifier_block dev_map_notifier = {
+ .notifier_call = dev_map_notification,
+};
+
+static int __init dev_map_init(void)
+{
+ register_netdevice_notifier(&dev_map_notifier);
+ return 0;
+}
+
+subsys_initcall(dev_map_init);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index df05d65..ebe9b38 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -1281,7 +1281,7 @@ static int check_map_func_compatibility(struct bpf_map *map, int func_id)
* for now.
*/
case BPF_MAP_TYPE_DEVMAP:
- if (func_id == BPF_FUNC_map_lookup_elem)
+ if (func_id != BPF_FUNC_redirect_map)
goto error;
break;
case BPF_MAP_TYPE_ARRAY_OF_MAPS:
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [net-next PATCH 12/12] xdp: bpf redirect with map sample program
2017-07-17 16:26 [net-next PATCH 00/12] Implement XDP bpf_redirect John Fastabend
` (10 preceding siblings ...)
2017-07-17 16:30 ` [net-next PATCH 11/12] net: add notifier hooks for devmap bpf map John Fastabend
@ 2017-07-17 16:30 ` John Fastabend
2017-07-17 16:48 ` [net-next PATCH 00/12] Implement XDP bpf_redirect David Miller
12 siblings, 0 replies; 19+ messages in thread
From: John Fastabend @ 2017-07-17 16:30 UTC (permalink / raw)
To: davem; +Cc: daniel, ast, netdev, john.fastabend, brouer, andy
Signed-off-by: John Fastabend <john.fastabend@gmail.com>
Tested-by: Andy Gospodarek <andy@greyhouse.net>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
samples/bpf/Makefile | 4 +
samples/bpf/xdp_redirect_map_kern.c | 83 +++++++++++++++++++++++
samples/bpf/xdp_redirect_map_user.c | 105 +++++++++++++++++++++++++++++
tools/testing/selftests/bpf/bpf_helpers.h | 2 +
4 files changed, 194 insertions(+)
create mode 100644 samples/bpf/xdp_redirect_map_kern.c
create mode 100644 samples/bpf/xdp_redirect_map_user.c
diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile
index 97734ce..770d46c 100644
--- a/samples/bpf/Makefile
+++ b/samples/bpf/Makefile
@@ -38,6 +38,7 @@ hostprogs-y += test_map_in_map
hostprogs-y += per_socket_stats_example
hostprogs-y += load_sock_ops
hostprogs-y += xdp_redirect
+hostprogs-y += xdp_redirect_map
# Libbpf dependencies
LIBBPF := ../../tools/lib/bpf/bpf.o
@@ -80,6 +81,7 @@ xdp_tx_iptunnel-objs := bpf_load.o $(LIBBPF) xdp_tx_iptunnel_user.o
test_map_in_map-objs := bpf_load.o $(LIBBPF) test_map_in_map_user.o
per_socket_stats_example-objs := $(LIBBPF) cookie_uid_helper_example.o
xdp_redirect-objs := bpf_load.o $(LIBBPF) xdp_redirect_user.o
+xdp_redirect_map-objs := bpf_load.o $(LIBBPF) xdp_redirect_map_user.o
# Tell kbuild to always build the programs
always := $(hostprogs-y)
@@ -122,6 +124,7 @@ always += tcp_cong_kern.o
always += tcp_iw_kern.o
always += tcp_clamp_kern.o
always += xdp_redirect_kern.o
+always += xdp_redirect_map_kern.o
HOSTCFLAGS += -I$(objtree)/usr/include
HOSTCFLAGS += -I$(srctree)/tools/lib/
@@ -159,6 +162,7 @@ HOSTLOADLIBES_lwt_len_hist += -l elf
HOSTLOADLIBES_xdp_tx_iptunnel += -lelf
HOSTLOADLIBES_test_map_in_map += -lelf
HOSTLOADLIBES_xdp_redirect += -lelf
+HOSTLOADLIBES_xdp_redirect_map += -lelf
# Allows pointing LLC/CLANG to a LLVM backend with bpf support, redefine on cmdline:
# make samples/bpf/ LLC=~/git/llvm/build/bin/llc CLANG=~/git/llvm/build/bin/clang
diff --git a/samples/bpf/xdp_redirect_map_kern.c b/samples/bpf/xdp_redirect_map_kern.c
new file mode 100644
index 0000000..2faf196
--- /dev/null
+++ b/samples/bpf/xdp_redirect_map_kern.c
@@ -0,0 +1,83 @@
+/* Copyright (c) 2017 Covalent IO, Inc. http://covalent.io
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+#define KBUILD_MODNAME "foo"
+#include <uapi/linux/bpf.h>
+#include <linux/in.h>
+#include <linux/if_ether.h>
+#include <linux/if_packet.h>
+#include <linux/if_vlan.h>
+#include <linux/ip.h>
+#include <linux/ipv6.h>
+#include "bpf_helpers.h"
+
+struct bpf_map_def SEC("maps") tx_port = {
+ .type = BPF_MAP_TYPE_DEVMAP,
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .max_entries = 100,
+};
+
+struct bpf_map_def SEC("maps") rxcnt = {
+ .type = BPF_MAP_TYPE_PERCPU_ARRAY,
+ .key_size = sizeof(u32),
+ .value_size = sizeof(long),
+ .max_entries = 1,
+};
+
+
+static void swap_src_dst_mac(void *data)
+{
+ unsigned short *p = data;
+ unsigned short dst[3];
+
+ dst[0] = p[0];
+ dst[1] = p[1];
+ dst[2] = p[2];
+ p[0] = p[3];
+ p[1] = p[4];
+ p[2] = p[5];
+ p[3] = dst[0];
+ p[4] = dst[1];
+ p[5] = dst[2];
+}
+
+SEC("xdp_redirect_map")
+int xdp_redirect_map_prog(struct xdp_md *ctx)
+{
+ void *data_end = (void *)(long)ctx->data_end;
+ void *data = (void *)(long)ctx->data;
+ struct ethhdr *eth = data;
+ int rc = XDP_DROP;
+ int vport, port = 0, m = 0;
+ long *value;
+ u32 key = 0;
+ u64 nh_off;
+
+ nh_off = sizeof(*eth);
+ if (data + nh_off > data_end)
+ return rc;
+
+ /* constant virtual port */
+ vport = 0;
+
+ /* count packet in global counter */
+ value = bpf_map_lookup_elem(&rxcnt, &key);
+ if (value)
+ *value += 1;
+
+ swap_src_dst_mac(data);
+
+ /* send packet out physical port */
+ return bpf_redirect_map(&tx_port, vport, 0);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/samples/bpf/xdp_redirect_map_user.c b/samples/bpf/xdp_redirect_map_user.c
new file mode 100644
index 0000000..0b8009a
--- /dev/null
+++ b/samples/bpf/xdp_redirect_map_user.c
@@ -0,0 +1,105 @@
+/* Copyright (c) 2017 Covalent IO, Inc. http://covalent.io
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+#include <linux/bpf.h>
+#include <assert.h>
+#include <errno.h>
+#include <signal.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#include "bpf_load.h"
+#include "bpf_util.h"
+#include "libbpf.h"
+
+static int ifindex_in;
+static int ifindex_out;
+
+static void int_exit(int sig)
+{
+ set_link_xdp_fd(ifindex_in, -1, 0);
+ exit(0);
+}
+
+/* simple per-protocol drop counter
+ */
+static void poll_stats(int interval, int ifindex)
+{
+ unsigned int nr_cpus = bpf_num_possible_cpus();
+ __u64 values[nr_cpus], prev[nr_cpus];
+
+ memset(prev, 0, sizeof(prev));
+
+ while (1) {
+ __u64 sum = 0;
+ __u32 key = 0;
+ int i;
+
+ sleep(interval);
+ assert(bpf_map_lookup_elem(map_fd[1], &key, values) == 0);
+ for (i = 0; i < nr_cpus; i++)
+ sum += (values[i] - prev[i]);
+ if (sum)
+ printf("ifindex %i: %10llu pkt/s\n",
+ ifindex, sum / interval);
+ memcpy(prev, values, sizeof(values));
+ }
+}
+
+int main(int ac, char **argv)
+{
+ char filename[256];
+ int ret, key = 0;
+
+ snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
+
+ if (ac != 3) {
+ printf("usage: %s IFINDEX_IN IFINDEX_OUT\n", argv[0]);
+ return 1;
+ }
+
+ ifindex_in = strtoul(argv[1], NULL, 0);
+ ifindex_out = strtoul(argv[2], NULL, 0);
+
+ if (load_bpf_file(filename)) {
+ printf("%s", bpf_log_buf);
+ return 1;
+ }
+
+ if (!prog_fd[0]) {
+ printf("load_bpf_file: %s\n", strerror(errno));
+ return 1;
+ }
+
+ signal(SIGINT, int_exit);
+
+ if (set_link_xdp_fd(ifindex_in, prog_fd[0], 0) < 0) {
+ printf("link set xdp fd failed\n");
+ return 1;
+ }
+
+ printf("map[0] (vports) = %i, map[1] (map) = %i, map[2] (count) = %i\n",
+ map_fd[0], map_fd[1], map_fd[2]);
+
+ /* populate virtual to physical port map */
+ ret = bpf_map_update_elem(map_fd[0], &key, &ifindex_out, 0);
+ if (ret) {
+ perror("bpf_update_elem");
+ goto out;
+ }
+
+ poll_stats(2, ifindex_out);
+
+out:
+ return 0;
+}
diff --git a/tools/testing/selftests/bpf/bpf_helpers.h b/tools/testing/selftests/bpf/bpf_helpers.h
index d50ac34..acbd605 100644
--- a/tools/testing/selftests/bpf/bpf_helpers.h
+++ b/tools/testing/selftests/bpf/bpf_helpers.h
@@ -38,6 +38,8 @@ static int (*bpf_clone_redirect)(void *ctx, int ifindex, int flags) =
(void *) BPF_FUNC_clone_redirect;
static int (*bpf_redirect)(int ifindex, int flags) =
(void *) BPF_FUNC_redirect;
+static int (*bpf_redirect_map)(void *map, int key, int flags) =
+ (void *) BPF_FUNC_redirect_map;
static int (*bpf_perf_event_output)(void *ctx, void *map,
unsigned long long flags, void *data,
int size) =
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [net-next PATCH 00/12] Implement XDP bpf_redirect
2017-07-17 16:26 [net-next PATCH 00/12] Implement XDP bpf_redirect John Fastabend
` (11 preceding siblings ...)
2017-07-17 16:30 ` [net-next PATCH 12/12] xdp: bpf redirect with map sample program John Fastabend
@ 2017-07-17 16:48 ` David Miller
12 siblings, 0 replies; 19+ messages in thread
From: David Miller @ 2017-07-17 16:48 UTC (permalink / raw)
To: john.fastabend; +Cc: daniel, ast, netdev, brouer, andy
From: John Fastabend <john.fastabend@gmail.com>
Date: Mon, 17 Jul 2017 09:26:02 -0700
> This series adds two new XDP helper routines bpf_redirect() and
> bpf_redirect_map(). The first variant bpf_redirect() is meant
> to be used the same way it is currently being used by the cls_bpf
> classifier. An xdp packet will be redirected immediately when this
> is called.
Awesome, series applied, thanks!
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [net-next PATCH 09/12] bpf: add bpf_redirect_map helper routine
2017-07-17 16:29 ` [net-next PATCH 09/12] bpf: add bpf_redirect_map helper routine John Fastabend
@ 2017-07-17 17:00 ` Alexei Starovoitov
2017-07-17 17:16 ` John Fastabend
0 siblings, 1 reply; 19+ messages in thread
From: Alexei Starovoitov @ 2017-07-17 17:00 UTC (permalink / raw)
To: John Fastabend, davem; +Cc: daniel, netdev, brouer, andy
On 7/17/17 9:29 AM, John Fastabend wrote:
> + * int bpf_redirect_map(key, map, flags)
> + * redirect to endpoint in map
> + * @key: index in map to lookup
> + * @map: fd of map to do lookup in
> + * @flags: --
could you please adjust the comment describing return value
and also fix the comment next to bpf_redirect()
since that function got reused between tc and xdp.
Great stuff!
looking forward to support in other drivers.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [net-next PATCH 09/12] bpf: add bpf_redirect_map helper routine
2017-07-17 17:00 ` Alexei Starovoitov
@ 2017-07-17 17:16 ` John Fastabend
0 siblings, 0 replies; 19+ messages in thread
From: John Fastabend @ 2017-07-17 17:16 UTC (permalink / raw)
To: Alexei Starovoitov, davem; +Cc: daniel, netdev, brouer, andy
On 07/17/2017 10:00 AM, Alexei Starovoitov wrote:
> On 7/17/17 9:29 AM, John Fastabend wrote:
>> + * int bpf_redirect_map(key, map, flags)
>> + * redirect to endpoint in map
>> + * @key: index in map to lookup
>> + * @map: fd of map to do lookup in
>> + * @flags: --
>
> could you please adjust the comment describing return value
> and also fix the comment next to bpf_redirect()
> since that function got reused between tc and xdp.
Looks like Dave pulled it in so I'll roll a patch on top of that
with the comment description fix. Thanks.
>
> Great stuff!
> looking forward to support in other drivers.
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [net-next PATCH 11/12] net: add notifier hooks for devmap bpf map
2017-07-17 16:30 ` [net-next PATCH 11/12] net: add notifier hooks for devmap bpf map John Fastabend
@ 2017-07-30 13:28 ` Levin, Alexander (Sasha Levin)
2017-07-31 8:55 ` Daniel Borkmann
0 siblings, 1 reply; 19+ messages in thread
From: Levin, Alexander (Sasha Levin) @ 2017-07-30 13:28 UTC (permalink / raw)
To: John Fastabend; +Cc: davem, daniel, ast, netdev, brouer, andy
On Mon, Jul 17, 2017 at 09:30:02AM -0700, John Fastabend wrote:
>@@ -341,9 +368,11 @@ static int dev_map_update_elem(struct bpf_map *map, void *key, void *value,
> * Remembering the driver side flush operation will happen before the
> * net device is removed.
> */
>+ mutex_lock(&dev_map_list_mutex);
> old_dev = xchg(&dtab->netdev_map[i], dev);
> if (old_dev)
> call_rcu(&old_dev->rcu, __dev_map_entry_free);
>+ mutex_unlock(&dev_map_list_mutex);
>
> return 0;
> }
This function gets called under rcu critical section, where we can't grab mutexes:
BUG: sleeping function called from invalid context at kernel/locking/mutex.c:747
in_atomic(): 1, irqs_disabled(): 0, pid: 16315, name: syz-executor1
1 lock held by syz-executor1/16315:
#0: (rcu_read_lock){......}, at: [<ffffffff8c363bc2>] map_delete_elem kernel/bpf/syscall.c:577 [inline]
#0: (rcu_read_lock){......}, at: [<ffffffff8c363bc2>] SYSC_bpf kernel/bpf/syscall.c:1427 [inline]
#0: (rcu_read_lock){......}, at: [<ffffffff8c363bc2>] SyS_bpf+0x1d32/0x4ba0 kernel/bpf/syscall.c:1388
Preemption disabled at:
[<ffffffff8c363bd1>] map_delete_elem kernel/bpf/syscall.c:582 [inline]
[<ffffffff8c363bd1>] SYSC_bpf kernel/bpf/syscall.c:1427 [inline]
[<ffffffff8c363bd1>] SyS_bpf+0x1d41/0x4ba0 kernel/bpf/syscall.c:1388
CPU: 2 PID: 16315 Comm: syz-executor1 Not tainted 4.13.0-rc2-next-20170727 #235
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.1-1ubuntu1 04/01/2014
Call Trace:
__dump_stack lib/dump_stack.c:16 [inline]
dump_stack+0x11d/0x1e5 lib/dump_stack.c:52
___might_sleep+0x3cc/0x520 kernel/sched/core.c:6001
__might_sleep+0x95/0x190 kernel/sched/core.c:5954
__mutex_lock_common kernel/locking/mutex.c:747 [inline]
__mutex_lock+0x146/0x19b0 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
dev_map_delete_elem+0x82/0x110 kernel/bpf/devmap.c:325
map_delete_elem kernel/bpf/syscall.c:585 [inline]
SYSC_bpf kernel/bpf/syscall.c:1427 [inline]
SyS_bpf+0x1deb/0x4ba0 kernel/bpf/syscall.c:1388
do_syscall_64+0x26a/0x800 arch/x86/entry/common.c:287
entry_SYSCALL64_slow_path+0x25/0x25
RIP: 0033:0x452309
RSP: 002b:00007f8d83d66c08 EFLAGS: 00000216 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 0000000000718000 RCX: 0000000000452309
RDX: 0000000000000010 RSI: 0000000020007000 RDI: 0000000000000003
RBP: 0000000000000270 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000216 R12: 00000000004b85e4
R13: 00000000ffffffff R14: 0000000000000003 R15: 0000000020007000
--
Thanks,
Sasha
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [net-next PATCH 11/12] net: add notifier hooks for devmap bpf map
2017-07-30 13:28 ` Levin, Alexander (Sasha Levin)
@ 2017-07-31 8:55 ` Daniel Borkmann
2017-07-31 14:47 ` John Fastabend
0 siblings, 1 reply; 19+ messages in thread
From: Daniel Borkmann @ 2017-07-31 8:55 UTC (permalink / raw)
To: Levin, Alexander (Sasha Levin), John Fastabend
Cc: davem, ast, netdev, brouer, andy
On 07/30/2017 03:28 PM, Levin, Alexander (Sasha Levin) wrote:
> On Mon, Jul 17, 2017 at 09:30:02AM -0700, John Fastabend wrote:
>> @@ -341,9 +368,11 @@ static int dev_map_update_elem(struct bpf_map *map, void *key, void *value,
>> * Remembering the driver side flush operation will happen before the
>> * net device is removed.
>> */
>> + mutex_lock(&dev_map_list_mutex);
>> old_dev = xchg(&dtab->netdev_map[i], dev);
>> if (old_dev)
>> call_rcu(&old_dev->rcu, __dev_map_entry_free);
>> + mutex_unlock(&dev_map_list_mutex);
>>
>> return 0;
>> }
>
> This function gets called under rcu critical section, where we can't grab mutexes:
Agree, same goes for the delete callback that mutex is not allowed
in this context. If I recall, this was for the devmap netdev notifier
in order to check whether we need to purge dev entries from the map,
so that the device can be unregistered gracefully. Given that devmap
ops like update/delete are only allowed from user space, we could
look into whether this map type actually needs to hold RCU at all
here, or other option is to try and get rid of the mutex altogether.
John, could you take a look for a fix?
Thanks a lot,
Daniel
> BUG: sleeping function called from invalid context at kernel/locking/mutex.c:747
> in_atomic(): 1, irqs_disabled(): 0, pid: 16315, name: syz-executor1
> 1 lock held by syz-executor1/16315:
> #0: (rcu_read_lock){......}, at: [<ffffffff8c363bc2>] map_delete_elem kernel/bpf/syscall.c:577 [inline]
> #0: (rcu_read_lock){......}, at: [<ffffffff8c363bc2>] SYSC_bpf kernel/bpf/syscall.c:1427 [inline]
> #0: (rcu_read_lock){......}, at: [<ffffffff8c363bc2>] SyS_bpf+0x1d32/0x4ba0 kernel/bpf/syscall.c:1388
> Preemption disabled at:
> [<ffffffff8c363bd1>] map_delete_elem kernel/bpf/syscall.c:582 [inline]
> [<ffffffff8c363bd1>] SYSC_bpf kernel/bpf/syscall.c:1427 [inline]
> [<ffffffff8c363bd1>] SyS_bpf+0x1d41/0x4ba0 kernel/bpf/syscall.c:1388
> CPU: 2 PID: 16315 Comm: syz-executor1 Not tainted 4.13.0-rc2-next-20170727 #235
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.1-1ubuntu1 04/01/2014
> Call Trace:
> __dump_stack lib/dump_stack.c:16 [inline]
> dump_stack+0x11d/0x1e5 lib/dump_stack.c:52
> ___might_sleep+0x3cc/0x520 kernel/sched/core.c:6001
> __might_sleep+0x95/0x190 kernel/sched/core.c:5954
> __mutex_lock_common kernel/locking/mutex.c:747 [inline]
> __mutex_lock+0x146/0x19b0 kernel/locking/mutex.c:893
> mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
> dev_map_delete_elem+0x82/0x110 kernel/bpf/devmap.c:325
> map_delete_elem kernel/bpf/syscall.c:585 [inline]
> SYSC_bpf kernel/bpf/syscall.c:1427 [inline]
> SyS_bpf+0x1deb/0x4ba0 kernel/bpf/syscall.c:1388
> do_syscall_64+0x26a/0x800 arch/x86/entry/common.c:287
> entry_SYSCALL64_slow_path+0x25/0x25
> RIP: 0033:0x452309
> RSP: 002b:00007f8d83d66c08 EFLAGS: 00000216 ORIG_RAX: 0000000000000141
> RAX: ffffffffffffffda RBX: 0000000000718000 RCX: 0000000000452309
> RDX: 0000000000000010 RSI: 0000000020007000 RDI: 0000000000000003
> RBP: 0000000000000270 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000216 R12: 00000000004b85e4
> R13: 00000000ffffffff R14: 0000000000000003 R15: 0000000020007000
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [net-next PATCH 11/12] net: add notifier hooks for devmap bpf map
2017-07-31 8:55 ` Daniel Borkmann
@ 2017-07-31 14:47 ` John Fastabend
0 siblings, 0 replies; 19+ messages in thread
From: John Fastabend @ 2017-07-31 14:47 UTC (permalink / raw)
To: Daniel Borkmann, Levin, Alexander (Sasha Levin)
Cc: davem, ast, netdev, brouer, andy
On 07/31/2017 01:55 AM, Daniel Borkmann wrote:
> On 07/30/2017 03:28 PM, Levin, Alexander (Sasha Levin) wrote:
>> On Mon, Jul 17, 2017 at 09:30:02AM -0700, John Fastabend wrote:
>>> @@ -341,9 +368,11 @@ static int dev_map_update_elem(struct bpf_map *map, void *key, void *value,
>>> * Remembering the driver side flush operation will happen before the
>>> * net device is removed.
>>> */
>>> + mutex_lock(&dev_map_list_mutex);
>>> old_dev = xchg(&dtab->netdev_map[i], dev);
>>> if (old_dev)
>>> call_rcu(&old_dev->rcu, __dev_map_entry_free);
>>> + mutex_unlock(&dev_map_list_mutex);
>>>
>>> return 0;
>>> }
>>
>> This function gets called under rcu critical section, where we can't grab mutexes:
>
> Agree, same goes for the delete callback that mutex is not allowed
> in this context. If I recall, this was for the devmap netdev notifier
> in order to check whether we need to purge dev entries from the map,
> so that the device can be unregistered gracefully. Given that devmap
> ops like update/delete are only allowed from user space, we could
> look into whether this map type actually needs to hold RCU at all
> here, or other option is to try and get rid of the mutex altogether.
> John, could you take a look for a fix?
>
> Thanks a lot,
> Daniel
>
I'll work up a fix today/tomorrow. Thanks.
^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2017-07-31 14:47 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-07-17 16:26 [net-next PATCH 00/12] Implement XDP bpf_redirect John Fastabend
2017-07-17 16:26 ` [net-next PATCH 01/12] ixgbe: NULL xdp_tx rings on resource cleanup John Fastabend
2017-07-17 16:26 ` [net-next PATCH 02/12] net: xdp: support xdp generic on virtual devices John Fastabend
2017-07-17 16:27 ` [net-next PATCH 03/12] xdp: add bpf_redirect helper function John Fastabend
2017-07-17 16:27 ` [net-next PATCH 04/12] xdp: sample program for new bpf_redirect helper John Fastabend
2017-07-17 16:27 ` [net-next PATCH 05/12] net: implement XDP_REDIRECT for xdp generic John Fastabend
2017-07-17 16:28 ` [net-next PATCH 06/12] ixgbe: add initial support for xdp redirect John Fastabend
2017-07-17 16:28 ` [net-next PATCH 07/12] xdp: add trace event " John Fastabend
2017-07-17 16:28 ` [net-next PATCH 08/12] bpf: add devmap, a map for storing net device references John Fastabend
2017-07-17 16:29 ` [net-next PATCH 09/12] bpf: add bpf_redirect_map helper routine John Fastabend
2017-07-17 17:00 ` Alexei Starovoitov
2017-07-17 17:16 ` John Fastabend
2017-07-17 16:29 ` [net-next PATCH 10/12] xdp: Add batching support to redirect map John Fastabend
2017-07-17 16:30 ` [net-next PATCH 11/12] net: add notifier hooks for devmap bpf map John Fastabend
2017-07-30 13:28 ` Levin, Alexander (Sasha Levin)
2017-07-31 8:55 ` Daniel Borkmann
2017-07-31 14:47 ` John Fastabend
2017-07-17 16:30 ` [net-next PATCH 12/12] xdp: bpf redirect with map sample program John Fastabend
2017-07-17 16:48 ` [net-next PATCH 00/12] Implement XDP bpf_redirect David Miller
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.