* [RFC PATCH bpf-next 0/2] xdp: add dev map multicast support
@ 2020-04-15 8:54 Hangbin Liu
2020-04-15 8:54 ` [RFC PATCH bpf-next 1/2] " Hangbin Liu
` (4 more replies)
0 siblings, 5 replies; 219+ messages in thread
From: Hangbin Liu @ 2020-04-15 8:54 UTC (permalink / raw)
To: bpf
Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc,
Jesper Dangaard Brouer, Eelco Chaudron, Alexei Starovoitov,
Daniel Borkmann, Hangbin Liu
Hi all,
This is a prototype for xdp multicast support, which has been discussed
before[0]. The goal is to be able to implement an OVS-like data plane in
XDP, i.e., a software switch that can forward XDP frames to multiple
ports.
To achieve this, an application needs to specify a group of interfaces
to forward a packet to. It is also common to want to exclude one or more
physical interfaces from the forwarding operation - e.g., to forward a
packet to all interfaces in the multicast group except the interface it
arrived on. While this could be done simply by adding more groups, this
quickly leads to a combinatorial explosion in the number of groups an
application has to maintain.
To avoid the combinatorial explosion, we propose to include the ability
to specify an "exclude group" as part of the forwarding operation. This
needs to be a group (instead of just a single port index), because a
physical interface can be part of a logical grouping, such as a bond
device.
Thus, the logical forwarding operation becomes a "set difference"
operation, i.e. "forward to all ports in group A that are not also in
group B". This series implements such an operation using device maps to
represent the groups. This means that the XDP program specifies two
device maps, one containing the list of netdevs to redirect to, and the
other containing the exclude list.
To be able to reuse the existing bpf_redirect_map() helper, we use a
containing map-in-map type to store the forwarding and exclude groups.
When a map-in-map type is passed to the redirect helper, it will
interpret the index as encoding the forwarding group in the upper 16
bits and the exclude group in the lower 16 bits. The enqueue logic will
unpack the two halves of the index and perform separate lookups in the
containing map. E.g., an index of 0x00010001 will look for the
forwarding group at map index 0x10000 and the exclude group at map index
0x1; the application is expected to populate the map accordingly.
For this RFC series we are primarily looking for feedback on the concept
and API: the example in patch 2 is functional, but not a lot of effort
has been made on performance optimisation.
Last but not least, thanks a lot to Jiri, Eelco, Toke and Jesper for
suggestions and help on implementation.
[0] https://xdp-project.net/#Handling-multicast
Hangbin Liu (2):
xdp: add dev map multicast support
sample/bpf: add xdp_redirect_map_multicast test
include/linux/bpf.h | 29 ++
include/net/xdp.h | 1 +
kernel/bpf/arraymap.c | 2 +-
kernel/bpf/devmap.c | 118 +++++++
kernel/bpf/hashtab.c | 2 +-
kernel/bpf/verifier.c | 15 +-
net/core/filter.c | 69 +++-
net/core/xdp.c | 26 ++
samples/bpf/Makefile | 3 +
samples/bpf/xdp_redirect_map_multicast.sh | 142 ++++++++
samples/bpf/xdp_redirect_map_multicast_kern.c | 147 +++++++++
samples/bpf/xdp_redirect_map_multicast_user.c | 306 ++++++++++++++++++
12 files changed, 854 insertions(+), 6 deletions(-)
create mode 100755 samples/bpf/xdp_redirect_map_multicast.sh
create mode 100644 samples/bpf/xdp_redirect_map_multicast_kern.c
create mode 100644 samples/bpf/xdp_redirect_map_multicast_user.c
--
2.19.2
^ permalink raw reply [flat|nested] 219+ messages in thread* [RFC PATCH bpf-next 1/2] xdp: add dev map multicast support 2020-04-15 8:54 [RFC PATCH bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu @ 2020-04-15 8:54 ` Hangbin Liu 2020-04-20 9:52 ` Hangbin Liu 2020-04-15 8:54 ` [RFC PATCH bpf-next 2/2] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu ` (3 subsequent siblings) 4 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-04-15 8:54 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, Alexei Starovoitov, Daniel Borkmann, Hangbin Liu This is a prototype for xdp multicast support. In this implemention we use map-in-map to store the multicast groups, because we may have both include and exclude groups on one interface. The include and exclude groups are seperated by a 32 bits map key. the high 16 bits keys are used for include groups and low 16 bits keys are for exclude groups. The general data path is kept in net/core/filter.c. The native data path is in kernel/bpf/devmap.c so we can use direct calls to get better performace. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- include/linux/bpf.h | 29 +++++++++++ include/net/xdp.h | 1 + kernel/bpf/arraymap.c | 2 +- kernel/bpf/devmap.c | 118 ++++++++++++++++++++++++++++++++++++++++++ kernel/bpf/hashtab.c | 2 +- kernel/bpf/verifier.c | 15 +++++- net/core/filter.c | 69 +++++++++++++++++++++++- net/core/xdp.c | 26 ++++++++++ 8 files changed, 256 insertions(+), 6 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index fd2b2322412d..72797667bca8 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1156,11 +1156,17 @@ struct sk_buff; struct bpf_dtab_netdev *__dev_map_lookup_elem(struct bpf_map *map, u32 key); struct bpf_dtab_netdev *__dev_map_hash_lookup_elem(struct bpf_map *map, u32 key); +void *array_of_map_lookup_elem(struct bpf_map *map, void *key); +void *htab_of_map_lookup_elem(struct bpf_map *map, void *key); void __dev_flush(void); int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx); int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, struct net_device *dev_rx); +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map); +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, u32 index); + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); @@ -1276,6 +1282,16 @@ static inline struct net_device *__dev_map_hash_lookup_elem(struct bpf_map *map return NULL; } +static void *array_of_map_lookup_elem(struct bpf_map *map, void *key) +{ + +} + +static void *htab_of_map_lookup_elem(struct bpf_map *map, void *key) +{ + +} + static inline void __dev_flush(void) { } @@ -1297,6 +1313,19 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return 0; } +static inline +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map) +{ + return true; +} + +static inline +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, u32 index) +{ + return 0; +} + struct sk_buff; static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, diff --git a/include/net/xdp.h b/include/net/xdp.h index 40c6d3398458..a214dce8579c 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -92,6 +92,7 @@ static inline void xdp_scrub_frame(struct xdp_frame *frame) } struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); /* Convert xdp_buff to xdp_frame */ static inline diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 95d77770353c..26ac66a05015 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -1031,7 +1031,7 @@ static void array_of_map_free(struct bpf_map *map) fd_array_map_free(map); } -static void *array_of_map_lookup_elem(struct bpf_map *map, void *key) +void *array_of_map_lookup_elem(struct bpf_map *map, void *key) { struct bpf_map **inner_map = array_map_lookup_elem(map, key); diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 58bdca5d978a..3a60cb209ae1 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -85,6 +85,9 @@ static DEFINE_PER_CPU(struct list_head, dev_flush_list); static DEFINE_SPINLOCK(dev_map_lock); static LIST_HEAD(dev_map_list); +static void *dev_map_lookup_elem(struct bpf_map *map, void *key); +static void *dev_map_hash_lookup_elem(struct bpf_map *map, void *key); + static struct hlist_head *dev_map_create_hash(unsigned int entries) { int i; @@ -456,6 +459,121 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return __xdp_enqueue(dev, xdp, dev_rx); } +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + return dev_map_get_next_key(map, key, next_key); + case BPF_MAP_TYPE_DEVMAP_HASH: + return dev_map_hash_get_next_key(map, key, next_key); + default: + break; + } + + return -ENOENT; +} + +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map) +{ + struct bpf_dtab_netdev *in_obj = NULL; + u32 key, next_key; + int err; + + devmap_get_next_key(map, NULL, &key); + + for (;;) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + in_obj = __dev_map_lookup_elem(map, key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + in_obj = __dev_map_hash_lookup_elem(map, key); + break; + default: + break; + } + + if (in_obj && in_obj->dev->ifindex == obj->dev->ifindex) + return true; + + err = devmap_get_next_key(map, &key, &next_key); + + if (err) + break; + + key = next_key; + } + + return false; +} + +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, u32 index) +{ + struct bpf_dtab_netdev *obj = NULL; + struct bpf_map *in_map, *ex_map; + struct xdp_frame *xdpf, *nxdpf; + struct net_device *dev; + u32 in_index, ex_index; + u32 key, next_key; + int err; + + in_index = index >> 16; + in_index = in_index << 16; + ex_index = in_index ^ index; + + in_map = map->ops->map_lookup_elem(map, &in_index); + /* ex_map could be NULL */ + ex_map = map->ops->map_lookup_elem(map, &ex_index); + + devmap_get_next_key(in_map, NULL, &key); + + xdpf = convert_to_xdp_frame(xdp); + if (unlikely(!xdpf)) + return -EOVERFLOW; + + for (;;) { + switch (in_map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + obj = __dev_map_lookup_elem(in_map, key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + obj = __dev_map_hash_lookup_elem(in_map, key); + break; + default: + break; + } + if (!obj) + goto find_next; + + if (ex_map && !dev_in_exclude_map(obj, ex_map)) { + dev = obj->dev; + + if (!dev->netdev_ops->ndo_xdp_xmit) + return -EOPNOTSUPP; + + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + if (unlikely(err)) + return err; + + nxdpf = xdpf_clone(xdpf); + if (unlikely(!nxdpf)) + return -ENOMEM; + + bq_enqueue(dev, nxdpf, dev_rx); + } +find_next: + err = devmap_get_next_key(in_map, &key, &next_key); + if (err) + break; + key = next_key; + } + + return 0; +} + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog) { diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index d541c8486c95..4e0a2eebd38d 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -1853,7 +1853,7 @@ static struct bpf_map *htab_of_map_alloc(union bpf_attr *attr) return map; } -static void *htab_of_map_lookup_elem(struct bpf_map *map, void *key) +void *htab_of_map_lookup_elem(struct bpf_map *map, void *key) { struct bpf_map **inner_map = htab_map_lookup_elem(map, key); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 04c6630cc18f..84d23418823a 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -3898,7 +3898,9 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, break; case BPF_MAP_TYPE_ARRAY_OF_MAPS: case BPF_MAP_TYPE_HASH_OF_MAPS: - if (func_id != BPF_FUNC_map_lookup_elem) + /* Used by multicast redirect */ + if (func_id != BPF_FUNC_redirect_map && + func_id != BPF_FUNC_map_lookup_elem) goto error; break; case BPF_MAP_TYPE_SOCKMAP: @@ -3968,8 +3970,17 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, if (map->map_type != BPF_MAP_TYPE_DEVMAP && map->map_type != BPF_MAP_TYPE_DEVMAP_HASH && map->map_type != BPF_MAP_TYPE_CPUMAP && - map->map_type != BPF_MAP_TYPE_XSKMAP) + map->map_type != BPF_MAP_TYPE_XSKMAP && + map->map_type != BPF_MAP_TYPE_ARRAY_OF_MAPS && + map->map_type != BPF_MAP_TYPE_HASH_OF_MAPS) goto error; + if (map->map_type == BPF_MAP_TYPE_ARRAY_OF_MAPS || + map->map_type == BPF_MAP_TYPE_HASH_OF_MAPS) { + /* FIXME: Maybe we should also strict the key size here ?? */ + if (map->inner_map_meta->map_type != BPF_MAP_TYPE_DEVMAP && + map->inner_map_meta->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + goto error; + } break; case BPF_FUNC_sk_redirect_map: case BPF_FUNC_msg_redirect_map: diff --git a/net/core/filter.c b/net/core/filter.c index 7628b947dbc3..7d2076f5b0a4 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3473,12 +3473,17 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { }; static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) + struct bpf_map *map, struct xdp_buff *xdp, + u32 index) { switch (map->map_type) { case BPF_MAP_TYPE_DEVMAP: + /* fall through */ case BPF_MAP_TYPE_DEVMAP_HASH: return dev_map_enqueue(fwd, xdp, dev_rx); + case BPF_MAP_TYPE_HASH_OF_MAPS: + case BPF_MAP_TYPE_ARRAY_OF_MAPS: + return dev_map_enqueue_multi(xdp, dev_rx, map, index); case BPF_MAP_TYPE_CPUMAP: return cpu_map_enqueue(fwd, xdp, dev_rx); case BPF_MAP_TYPE_XSKMAP: @@ -3508,6 +3513,10 @@ static inline void *__xdp_map_lookup_elem(struct bpf_map *map, u32 index) return __cpu_map_lookup_elem(map, index); case BPF_MAP_TYPE_XSKMAP: return __xsk_map_lookup_elem(map, index); + case BPF_MAP_TYPE_ARRAY_OF_MAPS: + return array_of_map_lookup_elem(map, (index >> 16) << 16); + case BPF_MAP_TYPE_HASH_OF_MAPS: + return htab_of_map_lookup_elem(map, (index >> 16) << 16); default: return NULL; } @@ -3552,7 +3561,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = dev_xdp_enqueue(fwd, xdp, dev); } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, index); } if (unlikely(err)) @@ -3566,6 +3575,55 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, } EXPORT_SYMBOL_GPL(xdp_do_redirect); +static int dev_map_redirect_multi(struct sk_buff *skb, struct bpf_prog *xdp_prog, + struct bpf_map *map, u32 index) + +{ + struct bpf_map *in_map, *ex_map; + struct bpf_dtab_netdev *dst; + u32 in_index, ex_index; + struct sk_buff *nskb; + u32 key, next_key; + int err; + void *fwd; + + in_index = index >> 16; + in_index = in_index << 16; + ex_index = in_index ^ index; + + in_map = map->ops->map_lookup_elem(map, &in_index); + /* ex_map could be NULL */ + ex_map = map->ops->map_lookup_elem(map, &ex_index); + + in_map->ops->map_get_next_key(in_map, NULL, &key); + + for (;;) { + fwd = __xdp_map_lookup_elem(in_map, key); + if (fwd) { + dst = (struct bpf_dtab_netdev *)fwd; + if (ex_map && dev_in_exclude_map(dst, ex_map)) + goto find_next; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -EOVERFLOW; + + err = dev_map_generic_redirect(dst, nskb, xdp_prog); + if (unlikely(err)) + return err; + } + +find_next: + err = in_map->ops->map_get_next_key(in_map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + return 0; +} + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, @@ -3588,6 +3646,13 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; + } else if (map->map_type == BPF_MAP_TYPE_ARRAY_OF_MAPS || + map->map_type == BPF_MAP_TYPE_HASH_OF_MAPS) { + /* Do multicast redirecting */ + err = dev_map_redirect_multi(skb, xdp_prog, map, index); + if (unlikely(err)) + goto err; + consume_skb(skb); } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { struct xdp_sock *xs = fwd; diff --git a/net/core/xdp.c b/net/core/xdp.c index 4c7ea85486af..70dfb4910f84 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -496,3 +496,29 @@ struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp) return xdpf; } EXPORT_SYMBOL_GPL(xdp_convert_zc_to_xdp_frame); + +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) +{ + unsigned int headroom, totalsize; + struct xdp_frame *nxdpf; + struct page *page; + void *addr; + + headroom = xdpf->headroom + sizeof(*xdpf); + totalsize = headroom + xdpf->len; + + if (unlikely(totalsize > PAGE_SIZE)) + return NULL; + page = dev_alloc_page(); + if (!page) + return NULL; + addr = page_to_virt(page); + + memcpy(addr, xdpf, totalsize); + + nxdpf = addr; + nxdpf->data = addr + headroom; + + return nxdpf; +} +EXPORT_SYMBOL_GPL(xdpf_clone); -- 2.19.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [RFC PATCH bpf-next 1/2] xdp: add dev map multicast support 2020-04-15 8:54 ` [RFC PATCH bpf-next 1/2] " Hangbin Liu @ 2020-04-20 9:52 ` Hangbin Liu 0 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-04-20 9:52 UTC (permalink / raw) To: Daniel Borkmann Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, Alexei Starovoitov, bpf Hi Daniel, Would you please help review the RFC and give some comments? Especially for the ifindex parameter of bpf_redirect_map() which contains both include and exclude map id. Should we keep the current designing, or find a way to make it flexible, or even add a new syscall to accept two index parameters? Thanks Hangbin On Wed, Apr 15, 2020 at 04:54:36PM +0800, Hangbin Liu wrote: > This is a prototype for xdp multicast support. In this implemention we > use map-in-map to store the multicast groups, because we may have both > include and exclude groups on one interface. > > The include and exclude groups are seperated by a 32 bits map key. > the high 16 bits keys are used for include groups and low 16 bits > keys are for exclude groups. > > The general data path is kept in net/core/filter.c. The native data > path is in kernel/bpf/devmap.c so we can use direct calls to > get better performace. > > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > --- > include/linux/bpf.h | 29 +++++++++++ > include/net/xdp.h | 1 + > kernel/bpf/arraymap.c | 2 +- > kernel/bpf/devmap.c | 118 ++++++++++++++++++++++++++++++++++++++++++ > kernel/bpf/hashtab.c | 2 +- > kernel/bpf/verifier.c | 15 +++++- > net/core/filter.c | 69 +++++++++++++++++++++++- > net/core/xdp.c | 26 ++++++++++ > 8 files changed, 256 insertions(+), 6 deletions(-) > > diff --git a/include/linux/bpf.h b/include/linux/bpf.h > index fd2b2322412d..72797667bca8 100644 > --- a/include/linux/bpf.h > +++ b/include/linux/bpf.h > @@ -1156,11 +1156,17 @@ struct sk_buff; > > struct bpf_dtab_netdev *__dev_map_lookup_elem(struct bpf_map *map, u32 key); > struct bpf_dtab_netdev *__dev_map_hash_lookup_elem(struct bpf_map *map, u32 key); > +void *array_of_map_lookup_elem(struct bpf_map *map, void *key); > +void *htab_of_map_lookup_elem(struct bpf_map *map, void *key); > void __dev_flush(void); > int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, > struct net_device *dev_rx); > int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > struct net_device *dev_rx); > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map); > +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, > + struct bpf_map *map, u32 index); > + > int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, > struct bpf_prog *xdp_prog); > > @@ -1276,6 +1282,16 @@ static inline struct net_device *__dev_map_hash_lookup_elem(struct bpf_map *map > return NULL; > } > > +static void *array_of_map_lookup_elem(struct bpf_map *map, void *key) > +{ > + > +} > + > +static void *htab_of_map_lookup_elem(struct bpf_map *map, void *key) > +{ > + > +} > + > static inline void __dev_flush(void) > { > } > @@ -1297,6 +1313,19 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > return 0; > } > > +static inline > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map) > +{ > + return true; > +} > + > +static inline > +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, > + struct bpf_map *map, u32 index) > +{ > + return 0; > +} > + > struct sk_buff; > > static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, > diff --git a/include/net/xdp.h b/include/net/xdp.h > index 40c6d3398458..a214dce8579c 100644 > --- a/include/net/xdp.h > +++ b/include/net/xdp.h > @@ -92,6 +92,7 @@ static inline void xdp_scrub_frame(struct xdp_frame *frame) > } > > struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); > +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); > > /* Convert xdp_buff to xdp_frame */ > static inline > diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c > index 95d77770353c..26ac66a05015 100644 > --- a/kernel/bpf/arraymap.c > +++ b/kernel/bpf/arraymap.c > @@ -1031,7 +1031,7 @@ static void array_of_map_free(struct bpf_map *map) > fd_array_map_free(map); > } > > -static void *array_of_map_lookup_elem(struct bpf_map *map, void *key) > +void *array_of_map_lookup_elem(struct bpf_map *map, void *key) > { > struct bpf_map **inner_map = array_map_lookup_elem(map, key); > > diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c > index 58bdca5d978a..3a60cb209ae1 100644 > --- a/kernel/bpf/devmap.c > +++ b/kernel/bpf/devmap.c > @@ -85,6 +85,9 @@ static DEFINE_PER_CPU(struct list_head, dev_flush_list); > static DEFINE_SPINLOCK(dev_map_lock); > static LIST_HEAD(dev_map_list); > > +static void *dev_map_lookup_elem(struct bpf_map *map, void *key); > +static void *dev_map_hash_lookup_elem(struct bpf_map *map, void *key); > + > static struct hlist_head *dev_map_create_hash(unsigned int entries) > { > int i; > @@ -456,6 +459,121 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > return __xdp_enqueue(dev, xdp, dev_rx); > } > > +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ > +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) > +{ > + > + switch (map->map_type) { > + case BPF_MAP_TYPE_DEVMAP: > + return dev_map_get_next_key(map, key, next_key); > + case BPF_MAP_TYPE_DEVMAP_HASH: > + return dev_map_hash_get_next_key(map, key, next_key); > + default: > + break; > + } > + > + return -ENOENT; > +} > + > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map) > +{ > + struct bpf_dtab_netdev *in_obj = NULL; > + u32 key, next_key; > + int err; > + > + devmap_get_next_key(map, NULL, &key); > + > + for (;;) { > + switch (map->map_type) { > + case BPF_MAP_TYPE_DEVMAP: > + in_obj = __dev_map_lookup_elem(map, key); > + break; > + case BPF_MAP_TYPE_DEVMAP_HASH: > + in_obj = __dev_map_hash_lookup_elem(map, key); > + break; > + default: > + break; > + } > + > + if (in_obj && in_obj->dev->ifindex == obj->dev->ifindex) > + return true; > + > + err = devmap_get_next_key(map, &key, &next_key); > + > + if (err) > + break; > + > + key = next_key; > + } > + > + return false; > +} > + > +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, > + struct bpf_map *map, u32 index) > +{ > + struct bpf_dtab_netdev *obj = NULL; > + struct bpf_map *in_map, *ex_map; > + struct xdp_frame *xdpf, *nxdpf; > + struct net_device *dev; > + u32 in_index, ex_index; > + u32 key, next_key; > + int err; > + > + in_index = index >> 16; > + in_index = in_index << 16; > + ex_index = in_index ^ index; > + > + in_map = map->ops->map_lookup_elem(map, &in_index); > + /* ex_map could be NULL */ > + ex_map = map->ops->map_lookup_elem(map, &ex_index); > + > + devmap_get_next_key(in_map, NULL, &key); > + > + xdpf = convert_to_xdp_frame(xdp); > + if (unlikely(!xdpf)) > + return -EOVERFLOW; > + > + for (;;) { > + switch (in_map->map_type) { > + case BPF_MAP_TYPE_DEVMAP: > + obj = __dev_map_lookup_elem(in_map, key); > + break; > + case BPF_MAP_TYPE_DEVMAP_HASH: > + obj = __dev_map_hash_lookup_elem(in_map, key); > + break; > + default: > + break; > + } > + if (!obj) > + goto find_next; > + > + if (ex_map && !dev_in_exclude_map(obj, ex_map)) { > + dev = obj->dev; > + > + if (!dev->netdev_ops->ndo_xdp_xmit) > + return -EOPNOTSUPP; > + > + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); > + if (unlikely(err)) > + return err; > + > + nxdpf = xdpf_clone(xdpf); > + if (unlikely(!nxdpf)) > + return -ENOMEM; > + > + bq_enqueue(dev, nxdpf, dev_rx); > + } > +find_next: > + err = devmap_get_next_key(in_map, &key, &next_key); > + if (err) > + break; > + key = next_key; > + } > + > + return 0; > +} > + > int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, > struct bpf_prog *xdp_prog) > { > diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c > index d541c8486c95..4e0a2eebd38d 100644 > --- a/kernel/bpf/hashtab.c > +++ b/kernel/bpf/hashtab.c > @@ -1853,7 +1853,7 @@ static struct bpf_map *htab_of_map_alloc(union bpf_attr *attr) > return map; > } > > -static void *htab_of_map_lookup_elem(struct bpf_map *map, void *key) > +void *htab_of_map_lookup_elem(struct bpf_map *map, void *key) > { > struct bpf_map **inner_map = htab_map_lookup_elem(map, key); > > diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c > index 04c6630cc18f..84d23418823a 100644 > --- a/kernel/bpf/verifier.c > +++ b/kernel/bpf/verifier.c > @@ -3898,7 +3898,9 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, > break; > case BPF_MAP_TYPE_ARRAY_OF_MAPS: > case BPF_MAP_TYPE_HASH_OF_MAPS: > - if (func_id != BPF_FUNC_map_lookup_elem) > + /* Used by multicast redirect */ > + if (func_id != BPF_FUNC_redirect_map && > + func_id != BPF_FUNC_map_lookup_elem) > goto error; > break; > case BPF_MAP_TYPE_SOCKMAP: > @@ -3968,8 +3970,17 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, > if (map->map_type != BPF_MAP_TYPE_DEVMAP && > map->map_type != BPF_MAP_TYPE_DEVMAP_HASH && > map->map_type != BPF_MAP_TYPE_CPUMAP && > - map->map_type != BPF_MAP_TYPE_XSKMAP) > + map->map_type != BPF_MAP_TYPE_XSKMAP && > + map->map_type != BPF_MAP_TYPE_ARRAY_OF_MAPS && > + map->map_type != BPF_MAP_TYPE_HASH_OF_MAPS) > goto error; > + if (map->map_type == BPF_MAP_TYPE_ARRAY_OF_MAPS || > + map->map_type == BPF_MAP_TYPE_HASH_OF_MAPS) { > + /* FIXME: Maybe we should also strict the key size here ?? */ > + if (map->inner_map_meta->map_type != BPF_MAP_TYPE_DEVMAP && > + map->inner_map_meta->map_type != BPF_MAP_TYPE_DEVMAP_HASH) > + goto error; > + } > break; > case BPF_FUNC_sk_redirect_map: > case BPF_FUNC_msg_redirect_map: > diff --git a/net/core/filter.c b/net/core/filter.c > index 7628b947dbc3..7d2076f5b0a4 100644 > --- a/net/core/filter.c > +++ b/net/core/filter.c > @@ -3473,12 +3473,17 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { > }; > > static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, > - struct bpf_map *map, struct xdp_buff *xdp) > + struct bpf_map *map, struct xdp_buff *xdp, > + u32 index) > { > switch (map->map_type) { > case BPF_MAP_TYPE_DEVMAP: > + /* fall through */ > case BPF_MAP_TYPE_DEVMAP_HASH: > return dev_map_enqueue(fwd, xdp, dev_rx); > + case BPF_MAP_TYPE_HASH_OF_MAPS: > + case BPF_MAP_TYPE_ARRAY_OF_MAPS: > + return dev_map_enqueue_multi(xdp, dev_rx, map, index); > case BPF_MAP_TYPE_CPUMAP: > return cpu_map_enqueue(fwd, xdp, dev_rx); > case BPF_MAP_TYPE_XSKMAP: > @@ -3508,6 +3513,10 @@ static inline void *__xdp_map_lookup_elem(struct bpf_map *map, u32 index) > return __cpu_map_lookup_elem(map, index); > case BPF_MAP_TYPE_XSKMAP: > return __xsk_map_lookup_elem(map, index); > + case BPF_MAP_TYPE_ARRAY_OF_MAPS: > + return array_of_map_lookup_elem(map, (index >> 16) << 16); > + case BPF_MAP_TYPE_HASH_OF_MAPS: > + return htab_of_map_lookup_elem(map, (index >> 16) << 16); > default: > return NULL; > } > @@ -3552,7 +3561,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, > > err = dev_xdp_enqueue(fwd, xdp, dev); > } else { > - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); > + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, index); > } > > if (unlikely(err)) > @@ -3566,6 +3575,55 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, > } > EXPORT_SYMBOL_GPL(xdp_do_redirect); > > +static int dev_map_redirect_multi(struct sk_buff *skb, struct bpf_prog *xdp_prog, > + struct bpf_map *map, u32 index) > + > +{ > + struct bpf_map *in_map, *ex_map; > + struct bpf_dtab_netdev *dst; > + u32 in_index, ex_index; > + struct sk_buff *nskb; > + u32 key, next_key; > + int err; > + void *fwd; > + > + in_index = index >> 16; > + in_index = in_index << 16; > + ex_index = in_index ^ index; > + > + in_map = map->ops->map_lookup_elem(map, &in_index); > + /* ex_map could be NULL */ > + ex_map = map->ops->map_lookup_elem(map, &ex_index); > + > + in_map->ops->map_get_next_key(in_map, NULL, &key); > + > + for (;;) { > + fwd = __xdp_map_lookup_elem(in_map, key); > + if (fwd) { > + dst = (struct bpf_dtab_netdev *)fwd; > + if (ex_map && dev_in_exclude_map(dst, ex_map)) > + goto find_next; > + > + nskb = skb_clone(skb, GFP_ATOMIC); > + if (!nskb) > + return -EOVERFLOW; > + > + err = dev_map_generic_redirect(dst, nskb, xdp_prog); > + if (unlikely(err)) > + return err; > + } > + > +find_next: > + err = in_map->ops->map_get_next_key(in_map, &key, &next_key); > + if (err) > + break; > + > + key = next_key; > + } > + > + return 0; > +} > + > static int xdp_do_generic_redirect_map(struct net_device *dev, > struct sk_buff *skb, > struct xdp_buff *xdp, > @@ -3588,6 +3646,13 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, > err = dev_map_generic_redirect(dst, skb, xdp_prog); > if (unlikely(err)) > goto err; > + } else if (map->map_type == BPF_MAP_TYPE_ARRAY_OF_MAPS || > + map->map_type == BPF_MAP_TYPE_HASH_OF_MAPS) { > + /* Do multicast redirecting */ > + err = dev_map_redirect_multi(skb, xdp_prog, map, index); > + if (unlikely(err)) > + goto err; > + consume_skb(skb); > } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { > struct xdp_sock *xs = fwd; > > diff --git a/net/core/xdp.c b/net/core/xdp.c > index 4c7ea85486af..70dfb4910f84 100644 > --- a/net/core/xdp.c > +++ b/net/core/xdp.c > @@ -496,3 +496,29 @@ struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp) > return xdpf; > } > EXPORT_SYMBOL_GPL(xdp_convert_zc_to_xdp_frame); > + > +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) > +{ > + unsigned int headroom, totalsize; > + struct xdp_frame *nxdpf; > + struct page *page; > + void *addr; > + > + headroom = xdpf->headroom + sizeof(*xdpf); > + totalsize = headroom + xdpf->len; > + > + if (unlikely(totalsize > PAGE_SIZE)) > + return NULL; > + page = dev_alloc_page(); > + if (!page) > + return NULL; > + addr = page_to_virt(page); > + > + memcpy(addr, xdpf, totalsize); > + > + nxdpf = addr; > + nxdpf->data = addr + headroom; > + > + return nxdpf; > +} > +EXPORT_SYMBOL_GPL(xdpf_clone); > -- > 2.19.2 > ^ permalink raw reply [flat|nested] 219+ messages in thread
* [RFC PATCH bpf-next 2/2] sample/bpf: add xdp_redirect_map_multicast test 2020-04-15 8:54 [RFC PATCH bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu 2020-04-15 8:54 ` [RFC PATCH bpf-next 1/2] " Hangbin Liu @ 2020-04-15 8:54 ` Hangbin Liu 2020-04-24 8:56 ` [RFC PATCHv2 bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu ` (2 subsequent siblings) 4 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-04-15 8:54 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, Alexei Starovoitov, Daniel Borkmann, Hangbin Liu This test is used for testing xdp multicast. It defined 3 groups for different usage. Each interface in init net has different exclude interfaces. In the test it tests both generic/native mode and 3 different map-in-map types. For more testing details, please see the test description in xdp_redirect_map_multicast.sh. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multicast.sh | 142 ++++++++ samples/bpf/xdp_redirect_map_multicast_kern.c | 147 +++++++++ samples/bpf/xdp_redirect_map_multicast_user.c | 306 ++++++++++++++++++ 4 files changed, 598 insertions(+) create mode 100755 samples/bpf/xdp_redirect_map_multicast.sh create mode 100644 samples/bpf/xdp_redirect_map_multicast_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multicast_user.c diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 424f6fe7ce38..55555b0267cf 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map tprogs-y += per_socket_stats_example tprogs-y += xdp_redirect tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_map_multicast tprogs-y += xdp_redirect_cpu tprogs-y += xdp_monitor tprogs-y += xdp_rxq_info @@ -97,6 +98,7 @@ test_map_in_map-objs := bpf_load.o test_map_in_map_user.o per_socket_stats_example-objs := cookie_uid_helper_example.o xdp_redirect-objs := xdp_redirect_user.o xdp_redirect_map-objs := xdp_redirect_map_user.o +xdp_redirect_map_multicast-objs := bpf_load.o xdp_redirect_map_multicast_user.o xdp_redirect_cpu-objs := bpf_load.o xdp_redirect_cpu_user.o xdp_monitor-objs := bpf_load.o xdp_monitor_user.o xdp_rxq_info-objs := xdp_rxq_info_user.o @@ -156,6 +158,7 @@ always-y += tcp_tos_reflect_kern.o always-y += tcp_dumpstats_kern.o always-y += xdp_redirect_kern.o always-y += xdp_redirect_map_kern.o +always-y += xdp_redirect_map_multicast_kern.o always-y += xdp_redirect_cpu_kern.o always-y += xdp_monitor_kern.o always-y += xdp_rxq_info_kern.o diff --git a/samples/bpf/xdp_redirect_map_multicast.sh b/samples/bpf/xdp_redirect_map_multicast.sh new file mode 100755 index 000000000000..01f825e33060 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multicast.sh @@ -0,0 +1,142 @@ +#!/bin/bash +# Test topology: +# - - - - - - - - - - - - - - - - - - - - - - - - - +# | veth1 veth2 veth3 veth4 | init net +# - -| - - - - - - | - - - - - - | - - - - - - | - - +# --------- --------- --------- --------- +# | veth0 | | veth0 | | veth0 | | veth0 | +# --------- --------- --------- --------- +# ns1 ns2 ns3 ns4 +# +# Include Groups: +# Group 1 has interfaces: veth1, veth2, veth3, veth4 (All traffic except IPv4, IPv6) +# Group 2 has interfaces: veth1, veth3 (For IPv4 traffic only) +# Group 3 has interfaces: veth2, veth4 (For IPv6 traffic only) +# Exclude Groups: +# veth1: exclude veth1 +# veth2: exclude veth2 +# veth3: exclude veth3, veth4 +# veth4: exclude veth3, veth4 +# +# Testing: +# XDP modes: generic, native +# map types: array of array, hash of array, hash of hash +# Include: +# IPv4: +# ns1 -> ns2 (fail), ns1 -> ns3 (pass) +# IPv6 +# ns4 -> ns1 (fail), ns4 -> ns2 (pass) +# Exclude: +# arp ns1 -> ns2: ns2, ns3, ns4 should receive the arp request +# arp ns4 -> ns1: ns1, ns2 should receive the arp request, ns3 should not +# + + +# netns numbers +NUM=4 +IFACES="" +DRV_MODE="generic native" +MAP_TYPE="aa ha hh" + +test_pass() +{ + echo "Pass: $@" +} + +test_fail() +{ + echo "fail: $@" +} + +clean_up() +{ + for i in $(seq $NUM); do + ip netns del ns$i + done +} + +setup_ns() +{ + local mode=$1 + + for i in $(seq $NUM); do + ip netns add ns$i + ip link add veth0 type veth peer name veth$i + ip link set veth0 netns ns$i + ip netns exec ns$i ip link set veth0 up + ip link set veth$i up + + ip netns exec ns$i ip addr add 192.0.2.$i/24 dev veth0 + ip netns exec ns$i ip addr add 2001:db8::$i/24 dev veth0 + # Use xdp_redirect_map_kern.o because the dummy section in + # xdp_redirect_map_multicast_kern.o does not support iproute2 loading + ip netns exec ns$i ip link set veth0 xdp$mode obj xdp_redirect_map_kern.o sec xdp_redirect_dummy &> /dev/null || \ + { test_fail "Unable to load dummy xdp" && exit 1; } + IFACES="$IFACES veth$i" + done +} + +do_tests() +{ + local drv_mode=$1 + local map_type=$2 + local drv_p + + [ ${drv_mode} == "drv" ] && drv_p="-N" || drv_p="-S" + + ./xdp_redirect_map_multicast $drv_p -M $map_type $IFACES &> xdp_${drv_mode}_${map_type}.log & + xdp_pid=$! + sleep 10 + + # arp test + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns1-2_${drv_mode}_${map_type}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns1-3_${drv_mode}_${map_type}.log & + ip netns exec ns4 tcpdump -i veth0 -nn -l -e &> arp_ns1-4_${drv_mode}_${map_type}.log & + ip netns exec ns1 ping 192.0.2.100 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.100 tell 192.0.2.1" arp_ns1-2_${drv_mode}_${map_type}.log && \ + test_pass "$drv_mode $map_type arp ns1-2" || test_fail "$drv_mode $map_type arp ns1-2" + grep -q "Request who-has 192.0.2.100 tell 192.0.2.1" arp_ns1-3_${drv_mode}_${map_type}.log && \ + test_pass "$drv_mode $map_type arp ns1-3" || test_fail "$drv_mode $map_type arp ns1-3" + grep -q "Request who-has 192.0.2.100 tell 192.0.2.1" arp_ns1-4_${drv_mode}_${map_type}.log && \ + test_pass "$drv_mode $map_type arp ns1-4" || test_fail "$drv_mode $map_type arp ns1-4" + + ip netns exec ns1 tcpdump -i veth0 -nn -l -e &> arp_ns4-1_${drv_mode}_${map_type}.log & + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns4-2_${drv_mode}_${map_type}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns4-3_${drv_mode}_${map_type}.log & + ip netns exec ns4 ping 192.0.2.100 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.100 tell 192.0.2.4" arp_ns4-1_${drv_mode}_${map_type}.log && \ + test_pass "$drv_mode $map_type arp ns4-1" || test_fail "$drv_mode $map_type arp ns4-1" + grep -q "Request who-has 192.0.2.100 tell 192.0.2.4" arp_ns4-2_${drv_mode}_${map_type}.log && \ + test_pass "$drv_mode $map_type arp ns4-2" || test_fail "$drv_mode $map_type arp ns4-2" + grep -q "Request who-has 192.0.2.100 tell 192.0.2.4" arp_ns4-3_${drv_mode}_${map_type}.log && \ + test_fail "$drv_mode $map_type arp ns4-3" || test_pass "$drv_mode $map_type arp ns4-3" + + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_fail "$drv_mode $map_type ping ns1-2" || test_pass "$drv_mode $map_type ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_pass "$drv_mode $map_type ping ns1-3" || test_fail "$drv_mode $map_type ping ns1-3" + + # ping6 test + ip netns exec ns4 ping6 2001:db8::1 -c 4 &> /dev/null && \ + test_fail "$drv_mode $map_type ping6 ns4-1" || test_pass "$drv_mode $map_type ping6 ns4-1" + ip netns exec ns4 ping6 2001:db8::2 -c 4 &> /dev/null && \ + test_pass "$drv_mode $map_type ping6 ns4-2" || test_fail "$drv_mode $map_type ping6 ns4-2" + + kill $xdp_pid +} + +for mode in ${DRV_MODE}; do + sleep 2 + setup_ns $mode + for type in ${MAP_TYPE}; do + do_tests $mode $type + done + sleep 20 + clean_up +done diff --git a/samples/bpf/xdp_redirect_map_multicast_kern.c b/samples/bpf/xdp_redirect_map_multicast_kern.c new file mode 100644 index 000000000000..f2ac36eed0e9 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multicast_kern.c @@ -0,0 +1,147 @@ +/* This program is free software; you can redistribute it and/or + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <uapi/linux/bpf.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ip.h> +#include <linux/ipv6.h> +#include <bpf/bpf_helpers.h> + +#define MAX_NR_PORTS 65536 + +/* In the sample we will store all ports to group 1. + * And add two multicast groups: + * group 2 for even number interfaces, group 3 for odd number interfaces + */ + +/* This is an array map template(NOT used) for multicast group storage + * The format could be lined by index:ifindex, like + * [0, 0], [1, 1], [2, 0], [3, 3], [4,4], [5, 0], [6, 0] ... + * which would be easier to modify and update. + * + * This map also could be used as multicast exclude array map. + * */ +struct bpf_map_def SEC("maps") group_a = { + .type = BPF_MAP_TYPE_DEVMAP, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = MAX_NR_PORTS, +}; + +/* This is a hash map template(NOT used) for multicast group storage + * The format could be none-lined index:ifindex, like + * [1, 1], [3, 3], [4, 4]... + * which would save more spaces for storage + * + * This map also could be used as multicast exclude hash map. + * */ +struct bpf_map_def SEC("maps") group_h = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = MAX_NR_PORTS, +}; + +/* This is an array map-in-map, the inner maps will store all the + * include array maps and exclude array maps + * + * The max_entries is MAX_NR_PORTS * 32 as I only use 3 groups. + * */ +struct bpf_map_def SEC("maps") a_of_group_a = { + .type = BPF_MAP_TYPE_ARRAY_OF_MAPS, + .key_size = sizeof(u32), + .value_size = sizeof(u32), + .max_entries = MAX_NR_PORTS * 32, +}; + +/* This is a hash map-in-map, the inner maps will store all the + * include array maps and exclude array maps + * */ +struct bpf_map_def SEC("maps") h_of_group_a = { + .type = BPF_MAP_TYPE_HASH_OF_MAPS, + .key_size = sizeof(u32), + .value_size = sizeof(u32), + .max_entries = MAX_NR_PORTS * 32, +}; + +/* This is a hash map-in-map, the inner maps will store all the + * include hash maps and exclude hash maps + * */ +struct bpf_map_def SEC("maps") h_of_group_h = { + .type = BPF_MAP_TYPE_HASH_OF_MAPS, + .key_size = sizeof(u32), + .value_size = sizeof(u32), + .max_entries = MAX_NR_PORTS * 32, +}; + +/* Note: This map is not used yet, we get the gourp id based on IP version at + * present. + * + * This map is used to store all the include groups fds based on ip/mac dest. + */ +struct bpf_map_def SEC("maps") mcast_route_map = { + .type = BPF_MAP_TYPE_HASH, + .key_size = sizeof(__u32), + .value_size = sizeof(__u16), + .max_entries = MAX_NR_PORTS, +}; + +/* TODO: This is used for broadcast redirecting/forwarding, + * how to do the redirecting/forwarding one on one based on neigh tables?? */ +SEC("xdp_redirect_map") +int xdp_redirect_map_prog(struct xdp_md *ctx) +{ + u32 key, mcast_group_id, exclude_group_id, redirect_key; + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + struct ethhdr *eth = data; + int *inmap_id; + u16 h_proto; + u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + h_proto = eth->h_proto; + + if (h_proto == htons(ETH_P_IPV6)) + mcast_group_id = 3; + else if (h_proto == htons(ETH_P_IP)) + mcast_group_id = 2; + else + mcast_group_id = 1; + + exclude_group_id = ctx->ingress_ifindex; + redirect_key = (mcast_group_id << 16) | exclude_group_id; + + key = 1 << 16; + if ((inmap_id = bpf_map_lookup_elem(&a_of_group_a, &key)) && inmap_id) + return bpf_redirect_map(&a_of_group_a, redirect_key, 0); + else if ((inmap_id = bpf_map_lookup_elem(&h_of_group_a, &key)) && inmap_id) + return bpf_redirect_map(&h_of_group_a, redirect_key, 0); + else if ((inmap_id = bpf_map_lookup_elem(&h_of_group_h, &key)) && inmap_id) + return bpf_redirect_map(&h_of_group_h, redirect_key, 0); + + return XDP_PASS; +} + +/* FIXME: This prog could not be load by iproute2 as the map-in-map need + * set inner map fd first. + */ +SEC("xdp_redirect_dummy") +int xdp_redirect_dummy_prog(struct xdp_md *ctx) +{ + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/samples/bpf/xdp_redirect_map_multicast_user.c b/samples/bpf/xdp_redirect_map_multicast_user.c new file mode 100644 index 000000000000..a451c90a05b6 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multicast_user.c @@ -0,0 +1,306 @@ +/* SPDX-License-Identifier: GPL-2.0-only + */ +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <stdbool.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> +#include <sys/resource.h> + +#include "bpf_load.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 +#define MAX_NR_PORTS 65536 + +static int ifaces[MAX_IFACE_NUM] = {}; +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static int init_map_in_map(struct bpf_object *obj, char *outmap_name, bool inmap_hash) +{ + struct bpf_map *outmap; + int inmap_fd, ret; + + inmap_fd = bpf_create_map(inmap_hash ? BPF_MAP_TYPE_DEVMAP_HASH : BPF_MAP_TYPE_DEVMAP, sizeof(__u32), sizeof(int), MAX_NR_PORTS, 0); + if (inmap_fd < 0) { + printf("Failed to create inner map '%s'!\n", strerror(errno)); + return 1; + } + outmap = bpf_object__find_map_by_name(obj, outmap_name); + if (!outmap) { + printf("Failed to load map %s from test prog\n", outmap_name); + return 1; + } + ret = bpf_map__set_inner_map_fd(outmap, inmap_fd); + if (ret) { + printf("Failed to set inner_map_fd for map %s\n", outmap_name); + return 1; + } + + return 0; +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n" + " -M map-in-map mode, could be aa, ha, hh(default)\n", + prog); +} + +int main(int argc, char **argv) +{ + struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY}; + struct bpf_object_open_attr obj_open_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + struct bpf_program *prog; + struct bpf_object *obj; + struct bpf_map *outmap; + char *outmap_name; + char ifname[IF_NAMESIZE]; + int pro_fd, inmap_fd, outmap_fd; + int i, j, ret, opt, ifindex; + __u32 inmap_id, key; + char filename[256]; + bool inmap_hash; + char *mode = NULL; + + while ((opt = getopt(argc, argv, "SNFM:")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + case 'M': + mode = optarg; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) + xdp_flags |= XDP_FLAGS_DRV_MODE; + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + /* array of array */ + if (strncmp(mode, "aa", 2) == 0) { + outmap_name = "a_of_group_a"; + inmap_hash = false; + /* hash of array */ + } else if (strncmp(mode, "ha", 2) == 0) { + outmap_name = "h_of_group_a"; + inmap_hash = false; + /* hash of hash */ + } else { + outmap_name = "h_of_group_h"; + inmap_hash = true; + } + + if (setrlimit(RLIMIT_MEMLOCK, &r)) { + perror("setrlimit(RLIMIT_MEMLOCK)"); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i ++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + /* open bpf obj, set inner fd for out map-in-map and load bpf obj */ + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + obj_open_attr.file = filename; + obj = bpf_object__open_xattr(&obj_open_attr); + + /* We need set inner map fd for all outmaps. */ + if(init_map_in_map(obj, "a_of_group_a", false)) { + printf("Failed to init out map a_of_group_a\n"); + goto err_out; + } + if(init_map_in_map(obj, "h_of_group_a", false)) { + printf("Failed to init out map h_of_group_a\n"); + goto err_out; + } + if(init_map_in_map(obj, "h_of_group_h", true)) { + printf("Failed to init out map h_of_group_h\n"); + goto err_out; + } + + bpf_object__load(obj); + + prog = bpf_program__next(NULL, obj); + pro_fd = bpf_program__fd(prog); + + outmap = bpf_object__find_map_by_name(obj, outmap_name); + if (!outmap) { + printf("Failed to load map %s from test prog\n", outmap_name); + goto err_out; + } + outmap_fd = bpf_map__fd(outmap); + if (outmap_fd < 0) { + printf("Failed to get fd from map %s\n", outmap_name); + goto err_out; + } + /* Init 3 multicast groups first. + * group 1: this is used for all ports group + * group 2: this is used for even number interfaces + * group 3: this is used for odd number interfaces + * You can store the group number in mcast_route_map for furture + * IP/MAC -> Multicast Group lookup. + */ + for (i = 1; i <= 3; i++) { + /* Split the include/exclude groups by 16 bit + * FIXME: is there a flexible way? how to let + * kernel side know this? + */ + key = i << 16; + inmap_fd = bpf_create_map(inmap_hash ? BPF_MAP_TYPE_DEVMAP_HASH : BPF_MAP_TYPE_DEVMAP, sizeof(__u32), sizeof(int), MAX_NR_PORTS, 0); + if (inmap_fd < 0) { + printf("Failed to create inner map '%s'!\n", strerror(errno)); + goto err_out; + } + ret = bpf_map_update_elem(outmap_fd, &key, &inmap_fd, 0); + if (ret) { + printf("Failed to update map %s\n", outmap_name); + goto err_out; + } + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Set values for each map + * 1. set all ports to group 1 + * 2. set ports to group 2 or 3 based on interface index + * 3. set exclude group for each interface + */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* bind pro_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, pro_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + /* Add the interface to group 1 */ + key = 1 << 16; + ret = bpf_map_lookup_elem(outmap_fd, &key, &inmap_id); + if (ret) { + printf("Failed to lookup inmap by key %u from map %s\n", key, outmap_name); + goto err_out; + } + inmap_fd = bpf_map_get_fd_by_id(inmap_id); + ret = bpf_map_update_elem(inmap_fd, &ifindex, &ifindex, 0); + if (ret) { + printf("Failed to update key %d, value %d for inmap id %u\n", ifindex, ifindex, inmap_id); + goto err_out; + } + + /* Add the even number ifaces to group 2 and odd ifaces to group 3 */ + if (i % 2 == 0) + key = 2 << 16; + else + key = 3 << 16; + ret = bpf_map_lookup_elem(outmap_fd, &key, &inmap_id); + if (ret) { + printf("Failed to lookup inmap by key %u from map %s\n", key, outmap_name); + goto err_out; + } + inmap_fd = bpf_map_get_fd_by_id(inmap_id); + ret = bpf_map_update_elem(inmap_fd, &ifindex, &ifindex, 0); + if (ret) { + printf("Failed to update key %d, value %d for inmap id %u\n", ifindex, ifindex, inmap_id); + goto err_out; + } + + /* Set the exclude map for the interfaces */ + key = ifindex; + inmap_fd = bpf_create_map(inmap_hash ? BPF_MAP_TYPE_DEVMAP_HASH : BPF_MAP_TYPE_DEVMAP, sizeof(__u32), sizeof(int), MAX_NR_PORTS, 0); + if (inmap_fd < 0) { + printf("Failed to create inner map '%s'!\n", strerror(errno)); + goto err_out; + } + ret = bpf_map_update_elem(inmap_fd, &ifindex, &ifindex, 0); + if (ret) { + printf("Failed to update key %d, value %d for exclude inmap\n", ifindex, ifindex); + goto err_out; + } + + /* let test exclude map by excluding all the interfaces except + * the first one. The first two interfaces are not affect. e.g. + * iface_1 = [1], iface_2 = [2], iface_3 = [3, 4,..], + * iface_4 = [3, 4,...] + */ + for (j = 2; ifaces[j] > 0; j++) { + if (i <= 1 || i == j) + continue; + ifindex = ifaces[j]; + ret = bpf_map_update_elem(inmap_fd, &ifindex, &ifindex, 0); + if (ret) { + printf("Failed to update key %d, value %d for exclude inmap\n", ifindex, ifindex); + goto err_out; + } + + } + ret = bpf_map_update_elem(outmap_fd, &key, &inmap_fd, 0); + if (ret) { + printf("Failed to update map %s\n", outmap_name); + goto err_out; + } + } + + sleep(600); + return 0; + +err_out: + return 1; +} -- 2.19.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [RFC PATCHv2 bpf-next 0/2] xdp: add dev map multicast support 2020-04-15 8:54 [RFC PATCH bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu 2020-04-15 8:54 ` [RFC PATCH bpf-next 1/2] " Hangbin Liu 2020-04-15 8:54 ` [RFC PATCH bpf-next 2/2] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu @ 2020-04-24 8:56 ` Hangbin Liu 2020-04-24 8:56 ` [RFC PATCHv2 bpf-next 1/2] xdp: add a new helper for " Hangbin Liu 2020-04-24 8:56 ` [RFC PATCHv2 bpf-next 2/2] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2020-05-23 6:05 ` [PATCHv3 bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu 2020-05-26 14:05 ` [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu 4 siblings, 2 replies; 219+ messages in thread From: Hangbin Liu @ 2020-04-24 8:56 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu Hi all, This is a prototype for xdp multicast support, which has been discussed before[0]. The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because a physical interface can be part of a logical grouping, such as a bond device. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. For this RFC series we are primarily looking for feedback on the concept and API: the example in patch 2 is functional, but not a lot of effort has been made on performance optimisation. Last but not least, thanks a lot to Jiri, Eelco, Toke and Jesper for suggestions and help on implementation. [0] https://xdp-project.net/#Handling-multicast v2: Discussed with Jiri, Toke, Jesper, Eelco, we think the v1 is doing a trick and may make user confused. So let's just add a new helper to make the implemention more clear. Hangbin Liu (2): xdp: add a new helper for dev map multicast support sample/bpf: add xdp_redirect_map_multicast test include/linux/bpf.h | 20 +++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 23 ++- kernel/bpf/devmap.c | 114 +++++++++++++++ kernel/bpf/verifier.c | 6 + net/core/filter.c | 98 ++++++++++++- net/core/xdp.c | 26 ++++ samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi.sh | 124 ++++++++++++++++ samples/bpf/xdp_redirect_map_multi_kern.c | 100 +++++++++++++ samples/bpf/xdp_redirect_map_multi_user.c | 170 ++++++++++++++++++++++ tools/include/uapi/linux/bpf.h | 23 ++- 13 files changed, 702 insertions(+), 7 deletions(-) create mode 100755 samples/bpf/xdp_redirect_map_multi.sh create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c -- 2.19.2 ^ permalink raw reply [flat|nested] 219+ messages in thread
* [RFC PATCHv2 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-04-24 8:56 ` [RFC PATCHv2 bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu @ 2020-04-24 8:56 ` Hangbin Liu 2020-04-24 14:19 ` Lorenzo Bianconi 2020-04-24 14:34 ` Toke Høiland-Jørgensen 2020-04-24 8:56 ` [RFC PATCHv2 bpf-next 2/2] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 1 sibling, 2 replies; 219+ messages in thread From: Hangbin Liu @ 2020-04-24 8:56 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu This is a prototype for xdp multicast support. In this implemention we add a new helper to accept two maps, forward map and exclude map. We will redirect the packet to all the interfaces in *forward map*, but exclude the interfaces that in *exclude map*. To achive this I add a new ex_map for struct bpf_redirect_info. in the helper I set tgt_value to NULL to make a difference with bpf_xdp_redirect_map() We also add a flag *BPF_F_EXCLUDE_INGRESS* incase you don't want to create a exclude map for each interface and just want to exclude the ingress interface. The general data path is kept in net/core/filter.c. The native data path is in kernel/bpf/devmap.c so we can use direct calls to get better performace. v2: add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- include/linux/bpf.h | 20 ++++++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 23 ++++++- kernel/bpf/devmap.c | 114 +++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++ net/core/filter.c | 98 ++++++++++++++++++++++++++-- net/core/xdp.c | 26 ++++++++ tools/include/uapi/linux/bpf.h | 23 ++++++- 9 files changed, 305 insertions(+), 7 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index fd2b2322412d..3fd2903def3f 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1161,6 +1161,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx); int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, struct net_device *dev_rx); +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex); +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + bool exclude_ingress); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); @@ -1297,6 +1302,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return 0; } +static inline +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + return false; +} + +static inline +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + bool exclude_ingress) +{ + return 0; +} + struct sk_buff; static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, diff --git a/include/linux/filter.h b/include/linux/filter.h index 9b5aa5c483cc..5b4e1ccd2d37 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -614,6 +614,7 @@ struct bpf_redirect_info { u32 tgt_index; void *tgt_value; struct bpf_map *map; + struct bpf_map *ex_map; u32 kern_flags; }; diff --git a/include/net/xdp.h b/include/net/xdp.h index 40c6d3398458..a214dce8579c 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -92,6 +92,7 @@ static inline void xdp_scrub_frame(struct xdp_frame *frame) } struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); /* Convert xdp_buff to xdp_frame */ static inline diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 2e29a671d67e..1dbe42290223 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3025,6 +3025,21 @@ union bpf_attr { * * **-EOPNOTSUPP** Unsupported operation, for example a * call from outside of TC ingress. * * **-ESOCKTNOSUPPORT** Socket type not supported (reuseport). + * + * int bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * Redirect the packet to all the interfaces in *map*, and + * exclude the interfaces that in *ex_map*. The *ex_map* could + * be NULL. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which could exlcude redirect to the ingress device. + * + * See also bpf_redirect_map(), which supports redirecting + * packet to a specific ifindex in the map. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. + * */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3151,7 +3166,8 @@ union bpf_attr { FN(xdp_output), \ FN(get_netns_cookie), \ FN(get_current_ancestor_cgroup_id), \ - FN(sk_assign), + FN(sk_assign), \ + FN(redirect_map_multi), /* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call @@ -3280,6 +3296,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 58bdca5d978a..34b171f7826c 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -456,6 +456,120 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return __xdp_enqueue(dev, xdp, dev_rx); } +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + return dev_map_get_next_key(map, key, next_key); + case BPF_MAP_TYPE_DEVMAP_HASH: + return dev_map_hash_get_next_key(map, key, next_key); + default: + break; + } + + return -ENOENT; +} + +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + struct bpf_dtab_netdev *in_obj = NULL; + u32 key, next_key; + int err; + + if (!map) + return false; + + if (obj->dev->ifindex == exclude_ifindex) + return true; + + devmap_get_next_key(map, NULL, &key); + + for (;;) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + in_obj = __dev_map_lookup_elem(map, key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + in_obj = __dev_map_hash_lookup_elem(map, key); + break; + default: + break; + } + + if (in_obj && in_obj->dev->ifindex == obj->dev->ifindex) + return true; + + err = devmap_get_next_key(map, &key, &next_key); + + if (err) + break; + + key = next_key; + } + + return false; +} + +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + bool exclude_ingress) +{ + struct bpf_dtab_netdev *obj = NULL; + struct xdp_frame *xdpf, *nxdpf; + struct net_device *dev; + u32 key, next_key; + int err; + + devmap_get_next_key(map, NULL, &key); + + xdpf = convert_to_xdp_frame(xdp); + if (unlikely(!xdpf)) + return -EOVERFLOW; + + for (;;) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + obj = __dev_map_lookup_elem(map, key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + obj = __dev_map_hash_lookup_elem(map, key); + break; + default: + break; + } + + if (!obj || dev_in_exclude_map(obj, ex_map, + exclude_ingress ? dev_rx->ifindex : 0)) + goto find_next; + + dev = obj->dev; + + if (!dev->netdev_ops->ndo_xdp_xmit) + return -EOPNOTSUPP; + + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + if (unlikely(err)) + return err; + + nxdpf = xdpf_clone(xdpf); + if (unlikely(!nxdpf)) + return -ENOMEM; + + bq_enqueue(dev, nxdpf, dev_rx); + +find_next: + err = devmap_get_next_key(map, &key, &next_key); + if (err) + break; + key = next_key; + } + + return 0; +} + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 38cfcf701eeb..f77213a0e354 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -3880,6 +3880,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: if (func_id != BPF_FUNC_redirect_map && + func_id != BPF_FUNC_redirect_map_multi && func_id != BPF_FUNC_map_lookup_elem) goto error; break; @@ -3970,6 +3971,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, map->map_type != BPF_MAP_TYPE_XSKMAP) goto error; break; + case BPF_FUNC_redirect_map_multi: + if (map->map_type != BPF_MAP_TYPE_DEVMAP && + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + goto error; + break; case BPF_FUNC_sk_redirect_map: case BPF_FUNC_msg_redirect_map: case BPF_FUNC_sock_map_update: diff --git a/net/core/filter.c b/net/core/filter.c index 7d6ceaa54d21..94d1530e5ac6 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3473,12 +3473,17 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { }; static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) + struct bpf_map *map, struct xdp_buff *xdp, + struct bpf_map *ex_map, bool exclude_ingress) { switch (map->map_type) { case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: - return dev_map_enqueue(fwd, xdp, dev_rx); + if (fwd) + return dev_map_enqueue(fwd, xdp, dev_rx); + else + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, + exclude_ingress); case BPF_MAP_TYPE_CPUMAP: return cpu_map_enqueue(fwd, xdp, dev_rx); case BPF_MAP_TYPE_XSKMAP: @@ -3534,6 +3539,8 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, struct bpf_prog *xdp_prog) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + bool exclude_ingress = !!(ri->flags & BPF_F_EXCLUDE_INGRESS); + struct bpf_map *ex_map = READ_ONCE(ri->ex_map); struct bpf_map *map = READ_ONCE(ri->map); u32 index = ri->tgt_index; void *fwd = ri->tgt_value; @@ -3552,7 +3559,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = dev_xdp_enqueue(fwd, xdp, dev); } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, exclude_ingress); } if (unlikely(err)) @@ -3566,6 +3573,49 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, } EXPORT_SYMBOL_GPL(xdp_do_redirect); +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, + struct bpf_prog *xdp_prog, + struct bpf_map *map, struct bpf_map *ex_map, + bool exclude_ingress) + +{ + struct bpf_dtab_netdev *dst; + struct sk_buff *nskb; + u32 key, next_key; + int err; + void *fwd; + + /* Get first key from forward map */ + map->ops->map_get_next_key(map, NULL, &key); + + for (;;) { + fwd = __xdp_map_lookup_elem(map, key); + if (fwd) { + dst = (struct bpf_dtab_netdev *)fwd; + if (dev_in_exclude_map(dst, ex_map, + exclude_ingress ? dev->ifindex : 0)) + goto find_next; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -EOVERFLOW; + + err = dev_map_generic_redirect(dst, nskb, xdp_prog); + if (unlikely(err)) + return err; + } + +find_next: + err = map->ops->map_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + return 0; +} + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, @@ -3573,6 +3623,8 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, struct bpf_map *map) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + bool exclude_ingress = !!(ri->flags & BPF_F_EXCLUDE_INGRESS); + struct bpf_map *ex_map = READ_ONCE(ri->ex_map); u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err = 0; @@ -3583,9 +3635,16 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, if (map->map_type == BPF_MAP_TYPE_DEVMAP || map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { - struct bpf_dtab_netdev *dst = fwd; + if (fwd) { + struct bpf_dtab_netdev *dst = fwd; + + err = dev_map_generic_redirect(dst, skb, xdp_prog); + } else { + /* Deal with multicast maps */ + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, + ex_map, exclude_ingress); + } - err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { @@ -3699,6 +3758,33 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { .arg3_type = ARG_ANYTHING, }; +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, + struct bpf_map *, ex_map, u64, flags) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + + if (unlikely(!map || flags > BPF_F_EXCLUDE_INGRESS)) + return XDP_ABORTED; + + ri->tgt_index = 0; + ri->tgt_value = NULL; + ri->flags = flags; + + WRITE_ONCE(ri->map, map); + WRITE_ONCE(ri->ex_map, ex_map); + + return XDP_REDIRECT; +} + +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { + .func = bpf_xdp_redirect_map_multi, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_CONST_MAP_PTR, + .arg1_type = ARG_CONST_MAP_PTR, + .arg3_type = ARG_ANYTHING, +}; + static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, unsigned long off, unsigned long len) { @@ -6304,6 +6390,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_redirect_proto; case BPF_FUNC_redirect_map: return &bpf_xdp_redirect_map_proto; + case BPF_FUNC_redirect_map_multi: + return &bpf_xdp_redirect_map_multi_proto; case BPF_FUNC_xdp_adjust_tail: return &bpf_xdp_adjust_tail_proto; case BPF_FUNC_fib_lookup: diff --git a/net/core/xdp.c b/net/core/xdp.c index 4c7ea85486af..70dfb4910f84 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -496,3 +496,29 @@ struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp) return xdpf; } EXPORT_SYMBOL_GPL(xdp_convert_zc_to_xdp_frame); + +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) +{ + unsigned int headroom, totalsize; + struct xdp_frame *nxdpf; + struct page *page; + void *addr; + + headroom = xdpf->headroom + sizeof(*xdpf); + totalsize = headroom + xdpf->len; + + if (unlikely(totalsize > PAGE_SIZE)) + return NULL; + page = dev_alloc_page(); + if (!page) + return NULL; + addr = page_to_virt(page); + + memcpy(addr, xdpf, totalsize); + + nxdpf = addr; + nxdpf->data = addr + headroom; + + return nxdpf; +} +EXPORT_SYMBOL_GPL(xdpf_clone); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 2e29a671d67e..1dbe42290223 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3025,6 +3025,21 @@ union bpf_attr { * * **-EOPNOTSUPP** Unsupported operation, for example a * call from outside of TC ingress. * * **-ESOCKTNOSUPPORT** Socket type not supported (reuseport). + * + * int bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * Redirect the packet to all the interfaces in *map*, and + * exclude the interfaces that in *ex_map*. The *ex_map* could + * be NULL. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which could exlcude redirect to the ingress device. + * + * See also bpf_redirect_map(), which supports redirecting + * packet to a specific ifindex in the map. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. + * */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3151,7 +3166,8 @@ union bpf_attr { FN(xdp_output), \ FN(get_netns_cookie), \ FN(get_current_ancestor_cgroup_id), \ - FN(sk_assign), + FN(sk_assign), \ + FN(redirect_map_multi), /* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call @@ -3280,6 +3296,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ -- 2.19.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [RFC PATCHv2 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-04-24 8:56 ` [RFC PATCHv2 bpf-next 1/2] xdp: add a new helper for " Hangbin Liu @ 2020-04-24 14:19 ` Lorenzo Bianconi 2020-04-28 11:09 ` Eelco Chaudron 2020-05-06 9:35 ` Hangbin Liu 2020-04-24 14:34 ` Toke Høiland-Jørgensen 1 sibling, 2 replies; 219+ messages in thread From: Lorenzo Bianconi @ 2020-04-24 14:19 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann [-- Attachment #1: Type: text/plain, Size: 13985 bytes --] > This is a prototype for xdp multicast support. In this implemention we > add a new helper to accept two maps, forward map and exclude map. > We will redirect the packet to all the interfaces in *forward map*, but > exclude the interfaces that in *exclude map*. > > To achive this I add a new ex_map for struct bpf_redirect_info. > in the helper I set tgt_value to NULL to make a difference with > bpf_xdp_redirect_map() > > We also add a flag *BPF_F_EXCLUDE_INGRESS* incase you don't want to > create a exclude map for each interface and just want to exclude the > ingress interface. > > The general data path is kept in net/core/filter.c. The native data > path is in kernel/bpf/devmap.c so we can use direct calls to > get better performace. > > v2: add new syscall bpf_xdp_redirect_map_multi() which could accept > include/exclude maps directly. > > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > --- > include/linux/bpf.h | 20 ++++++ > include/linux/filter.h | 1 + > include/net/xdp.h | 1 + > include/uapi/linux/bpf.h | 23 ++++++- > kernel/bpf/devmap.c | 114 +++++++++++++++++++++++++++++++++ > kernel/bpf/verifier.c | 6 ++ > net/core/filter.c | 98 ++++++++++++++++++++++++++-- > net/core/xdp.c | 26 ++++++++ > tools/include/uapi/linux/bpf.h | 23 ++++++- > 9 files changed, 305 insertions(+), 7 deletions(-) > [...] > +{ > + > + switch (map->map_type) { > + case BPF_MAP_TYPE_DEVMAP: > + return dev_map_get_next_key(map, key, next_key); > + case BPF_MAP_TYPE_DEVMAP_HASH: > + return dev_map_hash_get_next_key(map, key, next_key); > + default: > + break; > + } > + > + return -ENOENT; > +} > + > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, > + int exclude_ifindex) > +{ > + struct bpf_dtab_netdev *in_obj = NULL; > + u32 key, next_key; > + int err; > + > + if (!map) > + return false; doing so it seems mandatory to define an exclude_map even if we want just to do not forward the packet to the "ingress" interface. Moreover I was thinking that we can assume to never forward to in the incoming interface. Doing so the code would be simpler I guess. Is there a use case for it? (forward even to the ingress interface) > + > + if (obj->dev->ifindex == exclude_ifindex) > + return true; > + > + devmap_get_next_key(map, NULL, &key); > + > + for (;;) { > + switch (map->map_type) { > + case BPF_MAP_TYPE_DEVMAP: > + in_obj = __dev_map_lookup_elem(map, key); > + break; > + case BPF_MAP_TYPE_DEVMAP_HASH: > + in_obj = __dev_map_hash_lookup_elem(map, key); > + break; > + default: > + break; > + } > + > + if (in_obj && in_obj->dev->ifindex == obj->dev->ifindex) > + return true; > + > + err = devmap_get_next_key(map, &key, &next_key); > + > + if (err) > + break; > + > + key = next_key; > + } > + > + return false; > +} > + > +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, > + struct bpf_map *map, struct bpf_map *ex_map, > + bool exclude_ingress) > +{ > + struct bpf_dtab_netdev *obj = NULL; > + struct xdp_frame *xdpf, *nxdpf; > + struct net_device *dev; > + u32 key, next_key; > + int err; > + > + devmap_get_next_key(map, NULL, &key); > + > + xdpf = convert_to_xdp_frame(xdp); > + if (unlikely(!xdpf)) > + return -EOVERFLOW; > + > + for (;;) { > + switch (map->map_type) { > + case BPF_MAP_TYPE_DEVMAP: > + obj = __dev_map_lookup_elem(map, key); > + break; > + case BPF_MAP_TYPE_DEVMAP_HASH: > + obj = __dev_map_hash_lookup_elem(map, key); > + break; > + default: > + break; > + } > + > + if (!obj || dev_in_exclude_map(obj, ex_map, > + exclude_ingress ? dev_rx->ifindex : 0)) > + goto find_next; > + > + dev = obj->dev; > + > + if (!dev->netdev_ops->ndo_xdp_xmit) > + return -EOPNOTSUPP; > + > + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); > + if (unlikely(err)) > + return err; > + > + nxdpf = xdpf_clone(xdpf); > + if (unlikely(!nxdpf)) > + return -ENOMEM; > + > + bq_enqueue(dev, nxdpf, dev_rx); > + > +find_next: > + err = devmap_get_next_key(map, &key, &next_key); > + if (err) > + break; > + key = next_key; > + } Do we need to free 'incoming' xdp buffer here? I think most of the drivers assume the packet is owned by the stack if xdp_do_redirect returns 0 > + > + return 0; > +} > + > int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, > struct bpf_prog *xdp_prog) > { > diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c > index 38cfcf701eeb..f77213a0e354 100644 > --- a/kernel/bpf/verifier.c > +++ b/kernel/bpf/verifier.c > @@ -3880,6 +3880,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, > case BPF_MAP_TYPE_DEVMAP: > case BPF_MAP_TYPE_DEVMAP_HASH: > if (func_id != BPF_FUNC_redirect_map && > + func_id != BPF_FUNC_redirect_map_multi && > func_id != BPF_FUNC_map_lookup_elem) > goto error; > break; > @@ -3970,6 +3971,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, > map->map_type != BPF_MAP_TYPE_XSKMAP) > goto error; > break; > + case BPF_FUNC_redirect_map_multi: > + if (map->map_type != BPF_MAP_TYPE_DEVMAP && > + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) > + goto error; > + break; > case BPF_FUNC_sk_redirect_map: > case BPF_FUNC_msg_redirect_map: > case BPF_FUNC_sock_map_update: > diff --git a/net/core/filter.c b/net/core/filter.c > index 7d6ceaa54d21..94d1530e5ac6 100644 > --- a/net/core/filter.c > +++ b/net/core/filter.c > @@ -3473,12 +3473,17 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { > }; > > static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, > - struct bpf_map *map, struct xdp_buff *xdp) > + struct bpf_map *map, struct xdp_buff *xdp, > + struct bpf_map *ex_map, bool exclude_ingress) > { > switch (map->map_type) { > case BPF_MAP_TYPE_DEVMAP: > case BPF_MAP_TYPE_DEVMAP_HASH: > - return dev_map_enqueue(fwd, xdp, dev_rx); > + if (fwd) > + return dev_map_enqueue(fwd, xdp, dev_rx); > + else > + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, > + exclude_ingress); I guess it would be better to do not make it the default case. Maybe you can add a bit in flags to mark it for "multicast" > case BPF_MAP_TYPE_CPUMAP: > return cpu_map_enqueue(fwd, xdp, dev_rx); > case BPF_MAP_TYPE_XSKMAP: > @@ -3534,6 +3539,8 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, > struct bpf_prog *xdp_prog) > { > struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); > + bool exclude_ingress = !!(ri->flags & BPF_F_EXCLUDE_INGRESS); > + struct bpf_map *ex_map = READ_ONCE(ri->ex_map); > struct bpf_map *map = READ_ONCE(ri->map); > u32 index = ri->tgt_index; > void *fwd = ri->tgt_value; > @@ -3552,7 +3559,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, > > err = dev_xdp_enqueue(fwd, xdp, dev); > } else { > - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); > + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, exclude_ingress); > } > > if (unlikely(err)) > @@ -3566,6 +3573,49 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, > } > EXPORT_SYMBOL_GPL(xdp_do_redirect); > > +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, > + struct bpf_prog *xdp_prog, > + struct bpf_map *map, struct bpf_map *ex_map, > + bool exclude_ingress) > + > +{ > + struct bpf_dtab_netdev *dst; > + struct sk_buff *nskb; > + u32 key, next_key; > + int err; > + void *fwd; > + > + /* Get first key from forward map */ > + map->ops->map_get_next_key(map, NULL, &key); > + > + for (;;) { > + fwd = __xdp_map_lookup_elem(map, key); > + if (fwd) { > + dst = (struct bpf_dtab_netdev *)fwd; > + if (dev_in_exclude_map(dst, ex_map, > + exclude_ingress ? dev->ifindex : 0)) > + goto find_next; > + > + nskb = skb_clone(skb, GFP_ATOMIC); > + if (!nskb) > + return -EOVERFLOW; > + > + err = dev_map_generic_redirect(dst, nskb, xdp_prog); > + if (unlikely(err)) > + return err; > + } > + > +find_next: > + err = map->ops->map_get_next_key(map, &key, &next_key); > + if (err) > + break; > + > + key = next_key; > + } > + > + return 0; > +} > + > static int xdp_do_generic_redirect_map(struct net_device *dev, > struct sk_buff *skb, > struct xdp_buff *xdp, > @@ -3573,6 +3623,8 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, > struct bpf_map *map) > { > struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); > + bool exclude_ingress = !!(ri->flags & BPF_F_EXCLUDE_INGRESS); > + struct bpf_map *ex_map = READ_ONCE(ri->ex_map); > u32 index = ri->tgt_index; > void *fwd = ri->tgt_value; > int err = 0; > @@ -3583,9 +3635,16 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, > > if (map->map_type == BPF_MAP_TYPE_DEVMAP || > map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { > - struct bpf_dtab_netdev *dst = fwd; > + if (fwd) { > + struct bpf_dtab_netdev *dst = fwd; > + > + err = dev_map_generic_redirect(dst, skb, xdp_prog); > + } else { > + /* Deal with multicast maps */ > + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, > + ex_map, exclude_ingress); > + } > > - err = dev_map_generic_redirect(dst, skb, xdp_prog); > if (unlikely(err)) > goto err; > } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { > @@ -3699,6 +3758,33 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { > .arg3_type = ARG_ANYTHING, > }; > > +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, > + struct bpf_map *, ex_map, u64, flags) > +{ > + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); > + > + if (unlikely(!map || flags > BPF_F_EXCLUDE_INGRESS)) > + return XDP_ABORTED; > + > + ri->tgt_index = 0; > + ri->tgt_value = NULL; > + ri->flags = flags; > + > + WRITE_ONCE(ri->map, map); > + WRITE_ONCE(ri->ex_map, ex_map); > + > + return XDP_REDIRECT; > +} > + > +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { > + .func = bpf_xdp_redirect_map_multi, > + .gpl_only = false, > + .ret_type = RET_INTEGER, > + .arg1_type = ARG_CONST_MAP_PTR, > + .arg1_type = ARG_CONST_MAP_PTR, > + .arg3_type = ARG_ANYTHING, > +}; > + > static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, > unsigned long off, unsigned long len) > { > @@ -6304,6 +6390,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) > return &bpf_xdp_redirect_proto; > case BPF_FUNC_redirect_map: > return &bpf_xdp_redirect_map_proto; > + case BPF_FUNC_redirect_map_multi: > + return &bpf_xdp_redirect_map_multi_proto; > case BPF_FUNC_xdp_adjust_tail: > return &bpf_xdp_adjust_tail_proto; > case BPF_FUNC_fib_lookup: > diff --git a/net/core/xdp.c b/net/core/xdp.c > index 4c7ea85486af..70dfb4910f84 100644 > --- a/net/core/xdp.c > +++ b/net/core/xdp.c > @@ -496,3 +496,29 @@ struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp) > return xdpf; > } > EXPORT_SYMBOL_GPL(xdp_convert_zc_to_xdp_frame); > + > +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) > +{ > + unsigned int headroom, totalsize; > + struct xdp_frame *nxdpf; > + struct page *page; > + void *addr; > + > + headroom = xdpf->headroom + sizeof(*xdpf); > + totalsize = headroom + xdpf->len; > + > + if (unlikely(totalsize > PAGE_SIZE)) > + return NULL; > + page = dev_alloc_page(); > + if (!page) > + return NULL; > + addr = page_to_virt(page); > + > + memcpy(addr, xdpf, totalsize); > + > + nxdpf = addr; > + nxdpf->data = addr + headroom; > + > + return nxdpf; > +} > +EXPORT_SYMBOL_GPL(xdpf_clone); > diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h > index 2e29a671d67e..1dbe42290223 100644 > --- a/tools/include/uapi/linux/bpf.h > +++ b/tools/include/uapi/linux/bpf.h > @@ -3025,6 +3025,21 @@ union bpf_attr { > * * **-EOPNOTSUPP** Unsupported operation, for example a > * call from outside of TC ingress. > * * **-ESOCKTNOSUPPORT** Socket type not supported (reuseport). > + * > + * int bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) > + * Description > + * Redirect the packet to all the interfaces in *map*, and > + * exclude the interfaces that in *ex_map*. The *ex_map* could > + * be NULL. > + * > + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, > + * which could exlcude redirect to the ingress device. > + * > + * See also bpf_redirect_map(), which supports redirecting > + * packet to a specific ifindex in the map. > + * Return > + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. > + * > */ > #define __BPF_FUNC_MAPPER(FN) \ > FN(unspec), \ > @@ -3151,7 +3166,8 @@ union bpf_attr { > FN(xdp_output), \ > FN(get_netns_cookie), \ > FN(get_current_ancestor_cgroup_id), \ > - FN(sk_assign), > + FN(sk_assign), \ > + FN(redirect_map_multi), > > /* integer value in 'imm' field of BPF_CALL instruction selects which helper > * function eBPF program intends to call > @@ -3280,6 +3296,11 @@ enum bpf_lwt_encap_mode { > BPF_LWT_ENCAP_IP, > }; > > +/* BPF_FUNC_redirect_map_multi flags. */ > +enum { > + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), > +}; > + > #define __bpf_md_ptr(type, name) \ > union { \ > type name; \ > -- > 2.19.2 > [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 228 bytes --] ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [RFC PATCHv2 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-04-24 14:19 ` Lorenzo Bianconi @ 2020-04-28 11:09 ` Eelco Chaudron 2020-05-06 9:35 ` Hangbin Liu 1 sibling, 0 replies; 219+ messages in thread From: Eelco Chaudron @ 2020-04-28 11:09 UTC (permalink / raw) To: Lorenzo Bianconi Cc: Hangbin Liu, bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, ast, Daniel Borkmann On 24 Apr 2020, at 16:19, Lorenzo Bianconi wrote: [...] >> +{ >> + >> + switch (map->map_type) { >> + case BPF_MAP_TYPE_DEVMAP: >> + return dev_map_get_next_key(map, key, next_key); >> + case BPF_MAP_TYPE_DEVMAP_HASH: >> + return dev_map_hash_get_next_key(map, key, next_key); >> + default: >> + break; >> + } >> + >> + return -ENOENT; >> +} >> + >> +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map >> *map, >> + int exclude_ifindex) >> +{ >> + struct bpf_dtab_netdev *in_obj = NULL; >> + u32 key, next_key; >> + int err; >> + >> + if (!map) >> + return false; > > doing so it seems mandatory to define an exclude_map even if we want > just to do > not forward the packet to the "ingress" interface. > Moreover I was thinking that we can assume to never forward to in the > incoming > interface. Doing so the code would be simpler I guess. Is there a use > case for > it? (forward even to the ingress interface) > This part I can answer, it’s called VEPA, I think it’s part of IEEE 802.1Qbg. ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [RFC PATCHv2 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-04-24 14:19 ` Lorenzo Bianconi 2020-04-28 11:09 ` Eelco Chaudron @ 2020-05-06 9:35 ` Hangbin Liu 1 sibling, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-05-06 9:35 UTC (permalink / raw) To: Lorenzo Bianconi Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann Hi Lorenzo, Thanks for the comments, please see replies below. On Fri, Apr 24, 2020 at 04:19:08PM +0200, Lorenzo Bianconi wrote: > > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, > > + int exclude_ifindex) > > +{ > > + struct bpf_dtab_netdev *in_obj = NULL; > > + u32 key, next_key; > > + int err; > > + > > + if (!map) > > + return false; > > doing so it seems mandatory to define an exclude_map even if we want just to do > not forward the packet to the "ingress" interface. > Moreover I was thinking that we can assume to never forward to in the incoming > interface. Doing so the code would be simpler I guess. Is there a use case for > it? (forward even to the ingress interface) Eelco has help answered one use case: VEPA. Another reason I added this flag is that the other syscalls like bpf_redirect() or bpf_redirect_map() are also able to forward to ingress interface. So we need to behave the same by default. > > > +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, > > + struct bpf_map *map, struct bpf_map *ex_map, > > + bool exclude_ingress) > > +{ [...] > > + } > > Do we need to free 'incoming' xdp buffer here? I think most of the drivers assume > the packet is owned by the stack if xdp_do_redirect returns 0 Yes, we need. I will fix it. > > diff --git a/net/core/filter.c b/net/core/filter.c > > index 7d6ceaa54d21..94d1530e5ac6 100644 > > --- a/net/core/filter.c > > +++ b/net/core/filter.c > > @@ -3473,12 +3473,17 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { > > }; > > > > static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, > > - struct bpf_map *map, struct xdp_buff *xdp) > > + struct bpf_map *map, struct xdp_buff *xdp, > > + struct bpf_map *ex_map, bool exclude_ingress) > > { > > switch (map->map_type) { > > case BPF_MAP_TYPE_DEVMAP: > > case BPF_MAP_TYPE_DEVMAP_HASH: > > - return dev_map_enqueue(fwd, xdp, dev_rx); > > + if (fwd) > > + return dev_map_enqueue(fwd, xdp, dev_rx); > > + else > > + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, > > + exclude_ingress); > > I guess it would be better to do not make it the default case. Maybe you can > add a bit in flags to mark it for "multicast" But how do we distinguish the flag bit with other syscalls? e.g. If we define 0x02 as the "do_multicast" flag. What if other syscalls also used this flag. Currently __bpf_tx_xdp_map() is only called by xdp_do_redirect(). If there is a map and no fwd, it must be multicast forward. So we are still safe now. Maybe we need an update in future. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [RFC PATCHv2 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-04-24 8:56 ` [RFC PATCHv2 bpf-next 1/2] xdp: add a new helper for " Hangbin Liu 2020-04-24 14:19 ` Lorenzo Bianconi @ 2020-04-24 14:34 ` Toke Høiland-Jørgensen 2020-05-06 9:14 ` Hangbin Liu 2020-05-18 8:45 ` Hangbin Liu 1 sibling, 2 replies; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2020-04-24 14:34 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu Hangbin Liu <liuhangbin@gmail.com> writes: > This is a prototype for xdp multicast support. In this implemention we > add a new helper to accept two maps, forward map and exclude map. > We will redirect the packet to all the interfaces in *forward map*, but > exclude the interfaces that in *exclude map*. Yeah, the new helper is much cleaner! > To achive this I add a new ex_map for struct bpf_redirect_info. > in the helper I set tgt_value to NULL to make a difference with > bpf_xdp_redirect_map() > > We also add a flag *BPF_F_EXCLUDE_INGRESS* incase you don't want to > create a exclude map for each interface and just want to exclude the > ingress interface. > > The general data path is kept in net/core/filter.c. The native data > path is in kernel/bpf/devmap.c so we can use direct calls to > get better performace. Got any performance numbers? :) > v2: add new syscall bpf_xdp_redirect_map_multi() which could accept > include/exclude maps directly. > > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > --- > include/linux/bpf.h | 20 ++++++ > include/linux/filter.h | 1 + > include/net/xdp.h | 1 + > include/uapi/linux/bpf.h | 23 ++++++- > kernel/bpf/devmap.c | 114 +++++++++++++++++++++++++++++++++ > kernel/bpf/verifier.c | 6 ++ > net/core/filter.c | 98 ++++++++++++++++++++++++++-- > net/core/xdp.c | 26 ++++++++ > tools/include/uapi/linux/bpf.h | 23 ++++++- > 9 files changed, 305 insertions(+), 7 deletions(-) > > diff --git a/include/linux/bpf.h b/include/linux/bpf.h > index fd2b2322412d..3fd2903def3f 100644 > --- a/include/linux/bpf.h > +++ b/include/linux/bpf.h > @@ -1161,6 +1161,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, > struct net_device *dev_rx); > int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > struct net_device *dev_rx); > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, > + int exclude_ifindex); > +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, > + struct bpf_map *map, struct bpf_map *ex_map, > + bool exclude_ingress); > int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, > struct bpf_prog *xdp_prog); > > @@ -1297,6 +1302,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > return 0; > } > > +static inline > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, > + int exclude_ifindex) > +{ > + return false; > +} > + > +static inline > +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, > + struct bpf_map *map, struct bpf_map *ex_map, > + bool exclude_ingress) > +{ > + return 0; > +} > + > struct sk_buff; > > static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, > diff --git a/include/linux/filter.h b/include/linux/filter.h > index 9b5aa5c483cc..5b4e1ccd2d37 100644 > --- a/include/linux/filter.h > +++ b/include/linux/filter.h > @@ -614,6 +614,7 @@ struct bpf_redirect_info { > u32 tgt_index; > void *tgt_value; > struct bpf_map *map; > + struct bpf_map *ex_map; > u32 kern_flags; > }; > > diff --git a/include/net/xdp.h b/include/net/xdp.h > index 40c6d3398458..a214dce8579c 100644 > --- a/include/net/xdp.h > +++ b/include/net/xdp.h > @@ -92,6 +92,7 @@ static inline void xdp_scrub_frame(struct xdp_frame *frame) > } > > struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); > +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); > > /* Convert xdp_buff to xdp_frame */ > static inline > diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h > index 2e29a671d67e..1dbe42290223 100644 > --- a/include/uapi/linux/bpf.h > +++ b/include/uapi/linux/bpf.h > @@ -3025,6 +3025,21 @@ union bpf_attr { > * * **-EOPNOTSUPP** Unsupported operation, for example a > * call from outside of TC ingress. > * * **-ESOCKTNOSUPPORT** Socket type not supported (reuseport). > + * > + * int bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) > + * Description > + * Redirect the packet to all the interfaces in *map*, and > + * exclude the interfaces that in *ex_map*. The *ex_map* could > + * be NULL. > + * > + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, > + * which could exlcude redirect to the ingress device. I'd suggest rewording this to: * Redirect the packet to ALL the interfaces in *map*, but * exclude the interfaces in *ex_map* (which may be NULL). * * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, * which additionally excludes the current ingress device. > + * See also bpf_redirect_map(), which supports redirecting > + * packet to a specific ifindex in the map. > + * Return > + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. > + * > */ > #define __BPF_FUNC_MAPPER(FN) \ > FN(unspec), \ > @@ -3151,7 +3166,8 @@ union bpf_attr { > FN(xdp_output), \ > FN(get_netns_cookie), \ > FN(get_current_ancestor_cgroup_id), \ > - FN(sk_assign), > + FN(sk_assign), \ > + FN(redirect_map_multi), > > /* integer value in 'imm' field of BPF_CALL instruction selects which helper > * function eBPF program intends to call > @@ -3280,6 +3296,11 @@ enum bpf_lwt_encap_mode { > BPF_LWT_ENCAP_IP, > }; > > +/* BPF_FUNC_redirect_map_multi flags. */ > +enum { > + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), > +}; > + > #define __bpf_md_ptr(type, name) \ > union { \ > type name; \ > diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c > index 58bdca5d978a..34b171f7826c 100644 > --- a/kernel/bpf/devmap.c > +++ b/kernel/bpf/devmap.c > @@ -456,6 +456,120 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > return __xdp_enqueue(dev, xdp, dev_rx); > } > > +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ > +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) > +{ > + > + switch (map->map_type) { > + case BPF_MAP_TYPE_DEVMAP: > + return dev_map_get_next_key(map, key, next_key); > + case BPF_MAP_TYPE_DEVMAP_HASH: > + return dev_map_hash_get_next_key(map, key, next_key); > + default: > + break; > + } > + > + return -ENOENT; > +} > + > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, > + int exclude_ifindex) > +{ > + struct bpf_dtab_netdev *in_obj = NULL; > + u32 key, next_key; > + int err; > + > + if (!map) > + return false; > + > + if (obj->dev->ifindex == exclude_ifindex) > + return true; We probably want the EXCLUDE_INGRESS flag to work even if ex_map is NULL, right? In that case you want to switch the order of the two checks above. > + devmap_get_next_key(map, NULL, &key); > + > + for (;;) { I wonder if we should require DEVMAP_HASH maps to be indexed by ifindex to avoid the loop? > + switch (map->map_type) { > + case BPF_MAP_TYPE_DEVMAP: > + in_obj = __dev_map_lookup_elem(map, key); > + break; > + case BPF_MAP_TYPE_DEVMAP_HASH: > + in_obj = __dev_map_hash_lookup_elem(map, key); > + break; > + default: > + break; > + } > + > + if (in_obj && in_obj->dev->ifindex == obj->dev->ifindex) > + return true; > + > + err = devmap_get_next_key(map, &key, &next_key); > + > + if (err) > + break; > + > + key = next_key; > + } > + > + return false; > +} > + > +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, > + struct bpf_map *map, struct bpf_map *ex_map, > + bool exclude_ingress) > +{ > + struct bpf_dtab_netdev *obj = NULL; > + struct xdp_frame *xdpf, *nxdpf; > + struct net_device *dev; > + u32 key, next_key; > + int err; > + > + devmap_get_next_key(map, NULL, &key); > + > + xdpf = convert_to_xdp_frame(xdp); > + if (unlikely(!xdpf)) > + return -EOVERFLOW; You do a clone for each map entry below, so I think you end up leaking this initial xdpf? Also, you'll end up with one clone more than necessary - redirecting to two interfaces should only require 1 clone, you're doing 2. > + for (;;) { > + switch (map->map_type) { > + case BPF_MAP_TYPE_DEVMAP: > + obj = __dev_map_lookup_elem(map, key); > + break; > + case BPF_MAP_TYPE_DEVMAP_HASH: > + obj = __dev_map_hash_lookup_elem(map, key); > + break; > + default: > + break; > + } > + > + if (!obj || dev_in_exclude_map(obj, ex_map, > + exclude_ingress ? dev_rx->ifindex : 0)) > + goto find_next; > + > + dev = obj->dev; > + > + if (!dev->netdev_ops->ndo_xdp_xmit) > + return -EOPNOTSUPP; > + > + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); > + if (unlikely(err)) > + return err; These abort the whole operation midway through the loop if any error occurs. That is probably not what we want? I think the right thing to do is just continue the loop and only return an error if *all* of the forwarding attempts failed. Maybe we need a tracepoint to catch individual errors? > + nxdpf = xdpf_clone(xdpf); > + if (unlikely(!nxdpf)) > + return -ENOMEM; As this is a memory error it's likely fatal on the nest loop iteration as well, so probably OK to abort everything here. > + bq_enqueue(dev, nxdpf, dev_rx); > + > +find_next: > + err = devmap_get_next_key(map, &key, &next_key); > + if (err) > + break; > + key = next_key; > + } > + > + return 0; > +} > + > int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, > struct bpf_prog *xdp_prog) > { > diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c > index 38cfcf701eeb..f77213a0e354 100644 > --- a/kernel/bpf/verifier.c > +++ b/kernel/bpf/verifier.c > @@ -3880,6 +3880,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, > case BPF_MAP_TYPE_DEVMAP: > case BPF_MAP_TYPE_DEVMAP_HASH: > if (func_id != BPF_FUNC_redirect_map && > + func_id != BPF_FUNC_redirect_map_multi && > func_id != BPF_FUNC_map_lookup_elem) > goto error; > break; > @@ -3970,6 +3971,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, > map->map_type != BPF_MAP_TYPE_XSKMAP) > goto error; > break; > + case BPF_FUNC_redirect_map_multi: > + if (map->map_type != BPF_MAP_TYPE_DEVMAP && > + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) > + goto error; > + break; > case BPF_FUNC_sk_redirect_map: > case BPF_FUNC_msg_redirect_map: > case BPF_FUNC_sock_map_update: > diff --git a/net/core/filter.c b/net/core/filter.c > index 7d6ceaa54d21..94d1530e5ac6 100644 > --- a/net/core/filter.c > +++ b/net/core/filter.c > @@ -3473,12 +3473,17 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { > }; > > static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, > - struct bpf_map *map, struct xdp_buff *xdp) > + struct bpf_map *map, struct xdp_buff *xdp, > + struct bpf_map *ex_map, bool exclude_ingress) > { > switch (map->map_type) { > case BPF_MAP_TYPE_DEVMAP: > case BPF_MAP_TYPE_DEVMAP_HASH: > - return dev_map_enqueue(fwd, xdp, dev_rx); > + if (fwd) > + return dev_map_enqueue(fwd, xdp, dev_rx); > + else > + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, > + exclude_ingress); > case BPF_MAP_TYPE_CPUMAP: > return cpu_map_enqueue(fwd, xdp, dev_rx); > case BPF_MAP_TYPE_XSKMAP: > @@ -3534,6 +3539,8 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, > struct bpf_prog *xdp_prog) > { > struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); > + bool exclude_ingress = !!(ri->flags & BPF_F_EXCLUDE_INGRESS); > + struct bpf_map *ex_map = READ_ONCE(ri->ex_map); I don't think you need the READ_ONCE here since there's already one below? > struct bpf_map *map = READ_ONCE(ri->map); > u32 index = ri->tgt_index; > void *fwd = ri->tgt_value; > @@ -3552,7 +3559,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, > > err = dev_xdp_enqueue(fwd, xdp, dev); > } else { > - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); > + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, exclude_ingress); > } > > if (unlikely(err)) > @@ -3566,6 +3573,49 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, > } > EXPORT_SYMBOL_GPL(xdp_do_redirect); > > +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, > + struct bpf_prog *xdp_prog, > + struct bpf_map *map, struct bpf_map *ex_map, > + bool exclude_ingress) > + > +{ > + struct bpf_dtab_netdev *dst; > + struct sk_buff *nskb; > + u32 key, next_key; > + int err; > + void *fwd; > + > + /* Get first key from forward map */ > + map->ops->map_get_next_key(map, NULL, &key); > + > + for (;;) { > + fwd = __xdp_map_lookup_elem(map, key); > + if (fwd) { > + dst = (struct bpf_dtab_netdev *)fwd; > + if (dev_in_exclude_map(dst, ex_map, > + exclude_ingress ? dev->ifindex : 0)) > + goto find_next; > + > + nskb = skb_clone(skb, GFP_ATOMIC); > + if (!nskb) > + return -EOVERFLOW; > + > + err = dev_map_generic_redirect(dst, nskb, xdp_prog); > + if (unlikely(err)) > + return err; > + } > + > +find_next: > + err = map->ops->map_get_next_key(map, &key, &next_key); > + if (err) > + break; > + > + key = next_key; > + } > + > + return 0; > +} This duplication bugs me; maybe we should try to consolidate the generic and native XDP code paths? > static int xdp_do_generic_redirect_map(struct net_device *dev, > struct sk_buff *skb, > struct xdp_buff *xdp, > @@ -3573,6 +3623,8 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, > struct bpf_map *map) > { > struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); > + bool exclude_ingress = !!(ri->flags & BPF_F_EXCLUDE_INGRESS); > + struct bpf_map *ex_map = READ_ONCE(ri->ex_map); > u32 index = ri->tgt_index; > void *fwd = ri->tgt_value; > int err = 0; > @@ -3583,9 +3635,16 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, > > if (map->map_type == BPF_MAP_TYPE_DEVMAP || > map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { > - struct bpf_dtab_netdev *dst = fwd; > + if (fwd) { > + struct bpf_dtab_netdev *dst = fwd; > + > + err = dev_map_generic_redirect(dst, skb, xdp_prog); > + } else { > + /* Deal with multicast maps */ > + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, > + ex_map, exclude_ingress); > + } > > - err = dev_map_generic_redirect(dst, skb, xdp_prog); > if (unlikely(err)) > goto err; > } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { > @@ -3699,6 +3758,33 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { > .arg3_type = ARG_ANYTHING, > }; > > +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, > + struct bpf_map *, ex_map, u64, flags) > +{ > + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); > + > + if (unlikely(!map || flags > BPF_F_EXCLUDE_INGRESS)) > + return XDP_ABORTED; > + > + ri->tgt_index = 0; > + ri->tgt_value = NULL; > + ri->flags = flags; > + > + WRITE_ONCE(ri->map, map); > + WRITE_ONCE(ri->ex_map, ex_map); > + > + return XDP_REDIRECT; > +} > + > +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { > + .func = bpf_xdp_redirect_map_multi, > + .gpl_only = false, > + .ret_type = RET_INTEGER, > + .arg1_type = ARG_CONST_MAP_PTR, > + .arg1_type = ARG_CONST_MAP_PTR, > + .arg3_type = ARG_ANYTHING, > +}; > + > static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, > unsigned long off, unsigned long len) > { > @@ -6304,6 +6390,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) > return &bpf_xdp_redirect_proto; > case BPF_FUNC_redirect_map: > return &bpf_xdp_redirect_map_proto; > + case BPF_FUNC_redirect_map_multi: > + return &bpf_xdp_redirect_map_multi_proto; > case BPF_FUNC_xdp_adjust_tail: > return &bpf_xdp_adjust_tail_proto; > case BPF_FUNC_fib_lookup: > diff --git a/net/core/xdp.c b/net/core/xdp.c > index 4c7ea85486af..70dfb4910f84 100644 > --- a/net/core/xdp.c > +++ b/net/core/xdp.c > @@ -496,3 +496,29 @@ struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp) > return xdpf; > } > EXPORT_SYMBOL_GPL(xdp_convert_zc_to_xdp_frame); > + > +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) > +{ > + unsigned int headroom, totalsize; > + struct xdp_frame *nxdpf; > + struct page *page; > + void *addr; > + > + headroom = xdpf->headroom + sizeof(*xdpf); > + totalsize = headroom + xdpf->len; > + > + if (unlikely(totalsize > PAGE_SIZE)) > + return NULL; > + page = dev_alloc_page(); > + if (!page) > + return NULL; > + addr = page_to_virt(page); > + > + memcpy(addr, xdpf, totalsize); > + > + nxdpf = addr; > + nxdpf->data = addr + headroom; > + > + return nxdpf; > +} > +EXPORT_SYMBOL_GPL(xdpf_clone); > diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h > index 2e29a671d67e..1dbe42290223 100644 > --- a/tools/include/uapi/linux/bpf.h > +++ b/tools/include/uapi/linux/bpf.h Updates to tools/include should generally go into a separate patch. > @@ -3025,6 +3025,21 @@ union bpf_attr { > * * **-EOPNOTSUPP** Unsupported operation, for example a > * call from outside of TC ingress. > * * **-ESOCKTNOSUPPORT** Socket type not supported (reuseport). > + * > + * int bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) > + * Description > + * Redirect the packet to all the interfaces in *map*, and > + * exclude the interfaces that in *ex_map*. The *ex_map* could > + * be NULL. > + * > + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, > + * which could exlcude redirect to the ingress device. > + * > + * See also bpf_redirect_map(), which supports redirecting > + * packet to a specific ifindex in the map. > + * Return > + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. > + * > */ > #define __BPF_FUNC_MAPPER(FN) \ > FN(unspec), \ > @@ -3151,7 +3166,8 @@ union bpf_attr { > FN(xdp_output), \ > FN(get_netns_cookie), \ > FN(get_current_ancestor_cgroup_id), \ > - FN(sk_assign), > + FN(sk_assign), \ > + FN(redirect_map_multi), > > /* integer value in 'imm' field of BPF_CALL instruction selects which helper > * function eBPF program intends to call > @@ -3280,6 +3296,11 @@ enum bpf_lwt_encap_mode { > BPF_LWT_ENCAP_IP, > }; > > +/* BPF_FUNC_redirect_map_multi flags. */ > +enum { > + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), > +}; > + > #define __bpf_md_ptr(type, name) \ > union { \ > type name; \ > -- > 2.19.2 ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [RFC PATCHv2 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-04-24 14:34 ` Toke Høiland-Jørgensen @ 2020-05-06 9:14 ` Hangbin Liu 2020-05-06 10:00 ` Toke Høiland-Jørgensen 2020-05-18 8:45 ` Hangbin Liu 1 sibling, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-05-06 9:14 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi Hi Toke, Thanks for your review, please see replies below. On Fri, Apr 24, 2020 at 04:34:49PM +0200, Toke Høiland-Jørgensen wrote: > > > > The general data path is kept in net/core/filter.c. The native data > > path is in kernel/bpf/devmap.c so we can use direct calls to > > get better performace. > > Got any performance numbers? :) No, I haven't test the performance. Do you have any suggestions about how to test it? I'd like to try forwarding pkts to 10+ ports. But I don't know how to test the throughput. I don't think netperf or iperf supports this. > > > + * int bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) > > + * Description > > + * Redirect the packet to all the interfaces in *map*, and > > + * exclude the interfaces that in *ex_map*. The *ex_map* could > > + * be NULL. > > + * > > + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, > > + * which could exlcude redirect to the ingress device. > > I'd suggest rewording this to: > > * Redirect the packet to ALL the interfaces in *map*, but > * exclude the interfaces in *ex_map* (which may be NULL). > * > * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, > * which additionally excludes the current ingress device. Thanks, I will update it > > + > > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, > > + int exclude_ifindex) > > +{ > > + struct bpf_dtab_netdev *in_obj = NULL; > > + u32 key, next_key; > > + int err; > > + > > + if (!map) > > + return false; > > + > > + if (obj->dev->ifindex == exclude_ifindex) > > + return true; > > We probably want the EXCLUDE_INGRESS flag to work even if ex_map is > NULL, right? In that case you want to switch the order of the two checks > above. Yes, will fix it. > > > + devmap_get_next_key(map, NULL, &key); > > + > > + for (;;) { > > I wonder if we should require DEVMAP_HASH maps to be indexed by ifindex > to avoid the loop? I guess it's not easy to force user to index the map by ifindex. > > + xdpf = convert_to_xdp_frame(xdp); > > + if (unlikely(!xdpf)) > > + return -EOVERFLOW; > > You do a clone for each map entry below, so I think you end up leaking > this initial xdpf? Also, you'll end up with one clone more than > necessary - redirecting to two interfaces should only require 1 clone, > you're doing 2. We don't know which is the latest one. So we need to keep the initial for clone. Is it enough to call xdp_release_frame() after the for loop? > > > + for (;;) { > > + switch (map->map_type) { > > + case BPF_MAP_TYPE_DEVMAP: > > + obj = __dev_map_lookup_elem(map, key); > > + break; > > + case BPF_MAP_TYPE_DEVMAP_HASH: > > + obj = __dev_map_hash_lookup_elem(map, key); > > + break; > > + default: > > + break; > > + } > > + > > + if (!obj || dev_in_exclude_map(obj, ex_map, > > + exclude_ingress ? dev_rx->ifindex : 0)) > > + goto find_next; > > + > > + dev = obj->dev; > > + > > + if (!dev->netdev_ops->ndo_xdp_xmit) > > + return -EOPNOTSUPP; > > + > > + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); > > + if (unlikely(err)) > > + return err; > > These abort the whole operation midway through the loop if any error > occurs. That is probably not what we want? I think the right thing to do > is just continue the loop and only return an error if *all* of the > forwarding attempts failed. Maybe we need a tracepoint to catch > individual errors? Makes sense. I will see if we can add a tracepoint here. > > > > +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, > > + struct bpf_prog *xdp_prog, > > + struct bpf_map *map, struct bpf_map *ex_map, > > + bool exclude_ingress) > > + > > +{ > > + struct bpf_dtab_netdev *dst; > > + struct sk_buff *nskb; > > + u32 key, next_key; > > + int err; > > + void *fwd; > > + > > + /* Get first key from forward map */ > > + map->ops->map_get_next_key(map, NULL, &key); > > + > > + for (;;) { > > + fwd = __xdp_map_lookup_elem(map, key); > > + if (fwd) { > > + dst = (struct bpf_dtab_netdev *)fwd; > > + if (dev_in_exclude_map(dst, ex_map, > > + exclude_ingress ? dev->ifindex : 0)) > > + goto find_next; > > + > > + nskb = skb_clone(skb, GFP_ATOMIC); > > + if (!nskb) > > + return -EOVERFLOW; > > + > > + err = dev_map_generic_redirect(dst, nskb, xdp_prog); > > + if (unlikely(err)) > > + return err; > > + } > > + > > +find_next: > > + err = map->ops->map_get_next_key(map, &key, &next_key); > > + if (err) > > + break; > > + > > + key = next_key; > > + } > > + > > + return 0; > > +} > > This duplication bugs me; maybe we should try to consolidate the generic > and native XDP code paths? Yes, I have tried to combine these two functions together. But one is generic code path and another is XDP code patch. One use skb_clone and another use xdpf_clone(). There are also some extra checks for XDP code. So maybe we'd better just keep it as it is. > > diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h > > index 2e29a671d67e..1dbe42290223 100644 > > --- a/tools/include/uapi/linux/bpf.h > > +++ b/tools/include/uapi/linux/bpf.h > > Updates to tools/include should generally go into a separate patch. Will fix it, thanks. Best Regards Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [RFC PATCHv2 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-05-06 9:14 ` Hangbin Liu @ 2020-05-06 10:00 ` Toke Høiland-Jørgensen 2020-05-08 8:53 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2020-05-06 10:00 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi Hangbin Liu <liuhangbin@gmail.com> writes: > Hi Toke, > > Thanks for your review, please see replies below. > > On Fri, Apr 24, 2020 at 04:34:49PM +0200, Toke Høiland-Jørgensen wrote: >> > >> > The general data path is kept in net/core/filter.c. The native data >> > path is in kernel/bpf/devmap.c so we can use direct calls to >> > get better performace. >> >> Got any performance numbers? :) > > No, I haven't test the performance. Do you have any suggestions about how > to test it? I'd like to try forwarding pkts to 10+ ports. But I don't know > how to test the throughput. I don't think netperf or iperf supports > this. What I usually do when benchmarking XDP_REDIRECT is to just use pktgen (samples/pktgen in the kernel source tree) on another machine, specifically, like this: ./pktgen_sample03_burst_single_flow.sh -i enp1s0f1 -d 10.70.2.2 -m ec:0d:9a:db:11:35 -t 4 -s 64 (adjust iface, IP and MAC address to your system, of course). That'll flood the target machine with small UDP packets. On that machine, I then run the 'xdp_redirect_map' program from samples/bpf. The bpf program used by that sample will update an internal counter for every packet, and the userspace prints it out, which gives you the performance (in PPS). So just modifying that sample to using your new multicast helper (and comparing it to regular REDIRECT to a single device) would be a first approximation of a performance test. [...] >> > + devmap_get_next_key(map, NULL, &key); >> > + >> > + for (;;) { >> >> I wonder if we should require DEVMAP_HASH maps to be indexed by ifindex >> to avoid the loop? > > I guess it's not easy to force user to index the map by ifindex. Well, the way to 'force the user' is just to assume that this is the case, and if the map is filled in wrong, things just won't work ;) >> > + xdpf = convert_to_xdp_frame(xdp); >> > + if (unlikely(!xdpf)) >> > + return -EOVERFLOW; >> >> You do a clone for each map entry below, so I think you end up leaking >> this initial xdpf? Also, you'll end up with one clone more than >> necessary - redirecting to two interfaces should only require 1 clone, >> you're doing 2. > > We don't know which is the latest one. So we need to keep the initial > for clone. Is it enough to call xdp_release_frame() after the for > loop? You could do something like: bool first = true; for (;;) { [...] if (!first) { nxdpf = xdpf_clone(xdpf); if (unlikely(!nxdpf)) return -ENOMEM; bq_enqueue(dev, nxdpf, dev_rx); } else { bq_enqueue(dev, xdpf, dev_rx); first = false; } } /* didn't find anywhere to forward to, free buf */ if (first) xdp_return_frame_rx_napi(xdpf); [...] >> This duplication bugs me; maybe we should try to consolidate the generic >> and native XDP code paths? > > Yes, I have tried to combine these two functions together. But one is generic > code path and another is XDP code patch. One use skb_clone and another > use xdpf_clone(). There are also some extra checks for XDP code. So maybe > we'd better just keep it as it is. Yeah, guess it may not be as simple as I'd like it to be ;) Let's keep it this way for now at least; we can always consolidate in a separate patch series. -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [RFC PATCHv2 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-05-06 10:00 ` Toke Høiland-Jørgensen @ 2020-05-08 8:53 ` Hangbin Liu 2020-05-08 14:58 ` Toke Høiland-Jørgensen 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-05-08 8:53 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On Wed, May 06, 2020 at 12:00:08PM +0200, Toke Høiland-Jørgensen wrote: > > No, I haven't test the performance. Do you have any suggestions about how > > to test it? I'd like to try forwarding pkts to 10+ ports. But I don't know > > how to test the throughput. I don't think netperf or iperf supports > > this. > > What I usually do when benchmarking XDP_REDIRECT is to just use pktgen > (samples/pktgen in the kernel source tree) on another machine, > specifically, like this: > > ./pktgen_sample03_burst_single_flow.sh -i enp1s0f1 -d 10.70.2.2 -m ec:0d:9a:db:11:35 -t 4 -s 64 > > (adjust iface, IP and MAC address to your system, of course). That'll > flood the target machine with small UDP packets. On that machine, I then > run the 'xdp_redirect_map' program from samples/bpf. The bpf program > used by that sample will update an internal counter for every packet, > and the userspace prints it out, which gives you the performance (in > PPS). So just modifying that sample to using your new multicast helper > (and comparing it to regular REDIRECT to a single device) would be a > first approximation of a performance test. Thanks for this method. I will update the sample and do some more tests. > > You could do something like: > > bool first = true; > for (;;) { > > [...] > > if (!first) { > nxdpf = xdpf_clone(xdpf); > if (unlikely(!nxdpf)) > return -ENOMEM; > bq_enqueue(dev, nxdpf, dev_rx); > } else { > bq_enqueue(dev, xdpf, dev_rx); > first = false; > } > } > > /* didn't find anywhere to forward to, free buf */ > if (first) > xdp_return_frame_rx_napi(xdpf); I think the first xdpf will be consumed by the driver and the later xdpf_clone() will failed, won't it? How about just do a xdp_return_frame_rx_napi(xdpf) after all nxdpf enqueue? > > @@ -3534,6 +3539,8 @@ int xdp_do_redirect(struct net_device *dev, struct > > xdp_buff *xdp, > > struct bpf_prog *xdp_prog) > > { > > struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); > > + bool exclude_ingress = !!(ri->flags & BPF_F_EXCLUDE_INGRESS); > > + struct bpf_map *ex_map = READ_ONCE(ri->ex_map); > > I don't think you need the READ_ONCE here since there's already one > below? BTW, I forgot to ask, why we don't need the READ_ONCE for ex_map? I though the map and ex_map are two different pointers. > > struct bpf_map *map = READ_ONCE(ri->map); Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [RFC PATCHv2 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-05-08 8:53 ` Hangbin Liu @ 2020-05-08 14:58 ` Toke Høiland-Jørgensen 0 siblings, 0 replies; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2020-05-08 14:58 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi Hangbin Liu <liuhangbin@gmail.com> writes: > On Wed, May 06, 2020 at 12:00:08PM +0200, Toke Høiland-Jørgensen wrote: >> > No, I haven't test the performance. Do you have any suggestions about how >> > to test it? I'd like to try forwarding pkts to 10+ ports. But I don't know >> > how to test the throughput. I don't think netperf or iperf supports >> > this. >> >> What I usually do when benchmarking XDP_REDIRECT is to just use pktgen >> (samples/pktgen in the kernel source tree) on another machine, >> specifically, like this: >> >> ./pktgen_sample03_burst_single_flow.sh -i enp1s0f1 -d 10.70.2.2 -m ec:0d:9a:db:11:35 -t 4 -s 64 >> >> (adjust iface, IP and MAC address to your system, of course). That'll >> flood the target machine with small UDP packets. On that machine, I then >> run the 'xdp_redirect_map' program from samples/bpf. The bpf program >> used by that sample will update an internal counter for every packet, >> and the userspace prints it out, which gives you the performance (in >> PPS). So just modifying that sample to using your new multicast helper >> (and comparing it to regular REDIRECT to a single device) would be a >> first approximation of a performance test. > > Thanks for this method. I will update the sample and do some more tests. Great! >> You could do something like: >> >> bool first = true; >> for (;;) { >> >> [...] >> >> if (!first) { >> nxdpf = xdpf_clone(xdpf); >> if (unlikely(!nxdpf)) >> return -ENOMEM; >> bq_enqueue(dev, nxdpf, dev_rx); >> } else { >> bq_enqueue(dev, xdpf, dev_rx); >> first = false; >> } >> } >> >> /* didn't find anywhere to forward to, free buf */ >> if (first) >> xdp_return_frame_rx_napi(xdpf); > > I think the first xdpf will be consumed by the driver and the later > xdpf_clone() will failed, won't it? No, bq_enqueue just sticks the frame on a list, it's not consumed until after the NAPI cycle ends (and the driver calls xdp_do_flush()). > How about just do a xdp_return_frame_rx_napi(xdpf) after all nxdpf enqueue? Yeah, that would be the semantically obvious thing to do, but it is wasteful in that you end up performing one more clone than you strictly have to :) >> > @@ -3534,6 +3539,8 @@ int xdp_do_redirect(struct net_device *dev, struct >> > xdp_buff *xdp, >> > struct bpf_prog *xdp_prog) >> > { >> > struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); >> > + bool exclude_ingress = !!(ri->flags & BPF_F_EXCLUDE_INGRESS); >> > + struct bpf_map *ex_map = READ_ONCE(ri->ex_map); >> >> I don't think you need the READ_ONCE here since there's already one >> below? > > BTW, I forgot to ask, why we don't need the READ_ONCE for ex_map? > I though the map and ex_map are two different pointers. It isn't, but not for the reason I thought, so I can understand why my comment might have been somewhat confusing (I have been confused by this myself until just now...). The READ_ONCE() is not needed because the ex_map field is only ever read from or written to by the CPU owning the per-cpu pointer. Whereas the 'map' field is manipulated by remote CPUs in bpf_clear_redirect_map(). So you need neither READ_ONCE() nor WRITE_ONCE() on ex_map, just like there are none on tgt_index and tgt_value. -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [RFC PATCHv2 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-04-24 14:34 ` Toke Høiland-Jørgensen 2020-05-06 9:14 ` Hangbin Liu @ 2020-05-18 8:45 ` Hangbin Liu 2020-05-19 10:15 ` Jesper Dangaard Brouer 1 sibling, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-05-18 8:45 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi Hi Toke, On Fri, Apr 24, 2020 at 04:34:49PM +0200, Toke Høiland-Jørgensen wrote: > > Yeah, the new helper is much cleaner! > > > To achive this I add a new ex_map for struct bpf_redirect_info. > > in the helper I set tgt_value to NULL to make a difference with > > bpf_xdp_redirect_map() > > > > We also add a flag *BPF_F_EXCLUDE_INGRESS* incase you don't want to > > create a exclude map for each interface and just want to exclude the > > ingress interface. > > > > The general data path is kept in net/core/filter.c. The native data > > path is in kernel/bpf/devmap.c so we can use direct calls to > > get better performace. > > Got any performance numbers? :) Recently I tried with pktgen to get the performance number. It works with native mode, although the number looks not high. I tested it on VM with 1 cpu core. By forwarding to 7 ports, With pktgen config like: echo "count 10000000" > /proc/net/pktgen/veth0 echo "clone_skb 0" > /proc/net/pktgen/veth0 echo "pkt_size 64" > /proc/net/pktgen/veth0 echo "dst 224.1.1.10" > /proc/net/pktgen/veth0 I got forwarding number like: Forwarding 159958 pkt/s Forwarding 160213 pkt/s Forwarding 160448 pkt/s But when testing generic mode, I got system crashed directly. The code path is: do_xdp_generic() - netif_receive_generic_xdp() - pskb_expand_head() <- skb_is_nonlinear(skb) - BUG_ON(skb_shared(skb)) So I want to ask do you have the same issue with pktgen? Any workaround? > > diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h > > index 2e29a671d67e..1dbe42290223 100644 > > --- a/tools/include/uapi/linux/bpf.h > > +++ b/tools/include/uapi/linux/bpf.h > > Updates to tools/include should generally go into a separate patch. Is this a must to? It looks strange to separate the same implementation into two patches. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [RFC PATCHv2 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-05-18 8:45 ` Hangbin Liu @ 2020-05-19 10:15 ` Jesper Dangaard Brouer 2020-05-20 1:24 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: Jesper Dangaard Brouer @ 2020-05-19 10:15 UTC (permalink / raw) To: Hangbin Liu Cc: Toke Høiland-Jørgensen, bpf, netdev, Jiri Benc, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, brouer On Mon, 18 May 2020 16:45:27 +0800 Hangbin Liu <liuhangbin@gmail.com> wrote: > Hi Toke, > > On Fri, Apr 24, 2020 at 04:34:49PM +0200, Toke Høiland-Jørgensen wrote: > > > > Yeah, the new helper is much cleaner! > > > > > To achive this I add a new ex_map for struct bpf_redirect_info. > > > in the helper I set tgt_value to NULL to make a difference with > > > bpf_xdp_redirect_map() > > > > > > We also add a flag *BPF_F_EXCLUDE_INGRESS* incase you don't want to > > > create a exclude map for each interface and just want to exclude the > > > ingress interface. > > > > > > The general data path is kept in net/core/filter.c. The native data > > > path is in kernel/bpf/devmap.c so we can use direct calls to > > > get better performace. > > > > Got any performance numbers? :) > > Recently I tried with pktgen to get the performance number. It works > with native mode, although the number looks not high. > > I tested it on VM with 1 cpu core. Performance testing on a VM doesn't really make much sense. > By forwarding to 7 ports, With pktgen > config like: > echo "count 10000000" > /proc/net/pktgen/veth0 > echo "clone_skb 0" > /proc/net/pktgen/veth0 > echo "pkt_size 64" > /proc/net/pktgen/veth0 > echo "dst 224.1.1.10" > /proc/net/pktgen/veth0 > > I got forwarding number like: > Forwarding 159958 pkt/s > Forwarding 160213 pkt/s > Forwarding 160448 pkt/s > > But when testing generic mode, I got system crashed directly. The code > path is: > do_xdp_generic() > - netif_receive_generic_xdp() > - pskb_expand_head() <- skb_is_nonlinear(skb) > - BUG_ON(skb_shared(skb)) > > So I want to ask do you have the same issue with pktgen? Any workaround? Pktgen is not meant to be used on virtual devices. -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [RFC PATCHv2 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-05-19 10:15 ` Jesper Dangaard Brouer @ 2020-05-20 1:24 ` Hangbin Liu 0 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-05-20 1:24 UTC (permalink / raw) To: Jesper Dangaard Brouer Cc: Toke Høiland-Jørgensen, bpf, netdev, Jiri Benc, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On Tue, May 19, 2020 at 12:15:12PM +0200, Jesper Dangaard Brouer wrote: > Performance testing on a VM doesn't really make much sense. > > Pktgen is not meant to be used on virtual devices. Thanks, I will try on a physical machine. Cheers Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* [RFC PATCHv2 bpf-next 2/2] sample/bpf: add xdp_redirect_map_multicast test 2020-04-24 8:56 ` [RFC PATCHv2 bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu 2020-04-24 8:56 ` [RFC PATCHv2 bpf-next 1/2] xdp: add a new helper for " Hangbin Liu @ 2020-04-24 8:56 ` Hangbin Liu 2020-04-24 14:21 ` Lorenzo Bianconi 1 sibling, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-04-24 8:56 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu This is a sample for xdp multicast. In the sample we have 3 forward groups and 1 exclude group. It will redirect each interface's packets to all the interfaces in the forward group, and exclude the interface in exclude map. For more testing details, please see the test description in xdp_redirect_map_multi.sh. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi.sh | 124 ++++++++++++++++ samples/bpf/xdp_redirect_map_multi_kern.c | 100 +++++++++++++ samples/bpf/xdp_redirect_map_multi_user.c | 170 ++++++++++++++++++++++ 4 files changed, 397 insertions(+) create mode 100755 samples/bpf/xdp_redirect_map_multi.sh create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 424f6fe7ce38..eb7306efe85e 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map tprogs-y += per_socket_stats_example tprogs-y += xdp_redirect tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_map_multi tprogs-y += xdp_redirect_cpu tprogs-y += xdp_monitor tprogs-y += xdp_rxq_info @@ -97,6 +98,7 @@ test_map_in_map-objs := bpf_load.o test_map_in_map_user.o per_socket_stats_example-objs := cookie_uid_helper_example.o xdp_redirect-objs := xdp_redirect_user.o xdp_redirect_map-objs := xdp_redirect_map_user.o +xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o xdp_redirect_cpu-objs := bpf_load.o xdp_redirect_cpu_user.o xdp_monitor-objs := bpf_load.o xdp_monitor_user.o xdp_rxq_info-objs := xdp_rxq_info_user.o @@ -156,6 +158,7 @@ always-y += tcp_tos_reflect_kern.o always-y += tcp_dumpstats_kern.o always-y += xdp_redirect_kern.o always-y += xdp_redirect_map_kern.o +always-y += xdp_redirect_map_multi_kern.o always-y += xdp_redirect_cpu_kern.o always-y += xdp_monitor_kern.o always-y += xdp_rxq_info_kern.o diff --git a/samples/bpf/xdp_redirect_map_multi.sh b/samples/bpf/xdp_redirect_map_multi.sh new file mode 100755 index 000000000000..1999f261a1e8 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi.sh @@ -0,0 +1,124 @@ +#!/bin/bash +# Test topology: +# - - - - - - - - - - - - - - - - - - - - - - - - - +# | veth1 veth2 veth3 veth4 | init net +# - -| - - - - - - | - - - - - - | - - - - - - | - - +# --------- --------- --------- --------- +# | veth0 | | veth0 | | veth0 | | veth0 | +# --------- --------- --------- --------- +# ns1 ns2 ns3 ns4 +# +# Forward multicast groups: +# Forward group all has interfaces: veth1, veth2, veth3, veth4 (All traffic except IPv4, IPv6) +# Forward group v4 has interfaces: veth1, veth3, veth4 (For IPv4 traffic only) +# Forward group v6 has interfaces: veth2, veth3, veth4 (For IPv6 traffic only) +# Exclude Groups: +# Exclude group: veth3 (assume ns3 is in black list) +# +# Test modules: +# XDP modes: generic, native +# map types: group v4 use DEVMAP, others use DEVMAP_HASH +# +# Test cases: +# ARP(we didn't exclude ns3 in kern.c for ARP): +# ns1 -> gw: ns2, ns3, ns4 should receive the arp request +# IPv4: +# ns1 -> ns2 (fail), ns1 -> ns3 (fail), ns1 -> ns4 (pass) +# IPv6 +# ns2 -> ns1 (fail), ns2 -> ns3 (fail), ns2 -> ns4 (pass) +# + + +# netns numbers +NUM=4 +IFACES="" +DRV_MODE="drv generic" + +test_pass() +{ + echo "Pass: $@" +} + +test_fail() +{ + echo "fail: $@" +} + +clean_up() +{ + for i in $(seq $NUM); do + ip netns del ns$i + done +} + +setup_ns() +{ + local mode=$1 + + for i in $(seq $NUM); do + ip netns add ns$i + ip link add veth0 type veth peer name veth$i + ip link set veth0 netns ns$i + ip netns exec ns$i ip link set veth0 up + ip link set veth$i up + + ip netns exec ns$i ip addr add 192.0.2.$i/24 dev veth0 + ip netns exec ns$i ip addr add 2001:db8::$i/24 dev veth0 + ip netns exec ns$i ip link set veth0 xdp$mode obj \ + xdp_redirect_map_multi_kern.o sec xdp_redirect_dummy &> /dev/null || \ + { test_fail "Unable to load dummy xdp" && exit 1; } + IFACES="$IFACES veth$i" + done +} + +do_tests() +{ + local drv_mode=$1 + local drv_p + + [ ${drv_mode} == "drv" ] && drv_p="-N" || drv_p="-S" + + ./xdp_redirect_map_multi $drv_p $IFACES &> xdp_${drv_mode}.log & + xdp_pid=$! + sleep 10 + + # arp test + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns1-2_${drv_mode}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns1-3_${drv_mode}.log & + ip netns exec ns4 tcpdump -i veth0 -nn -l -e &> arp_ns1-4_${drv_mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-2_${drv_mode}.log && \ + test_pass "$drv_mode arp ns1-2" || test_fail "$drv_mode arp ns1-2" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-3_${drv_mode}.log && \ + test_pass "$drv_mode arp ns1-3" || test_fail "$drv_mode arp ns1-3" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-4_${drv_mode}.log && \ + test_pass "$drv_mode arp ns1-4" || test_fail "$drv_mode arp ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_fail "$drv_mode ping ns1-2" || test_pass "$drv_mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_fail "$drv_mode ping ns1-3" || test_pass "$drv_mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$drv_mode ping ns1-4" || test_fail "$drv_mode ping ns1-4" + + # ping6 test + ip netns exec ns2 ping6 2001:db8::1 -c 4 &> /dev/null && \ + test_fail "$drv_mode ping6 ns2-1" || test_pass "$drv_mode ping6 ns2-1" + ip netns exec ns2 ping6 2001:db8::3 -c 4 &> /dev/null && \ + test_fail "$drv_mode ping6 ns2-3" || test_pass "$drv_mode ping6 ns2-3" + ip netns exec ns2 ping6 2001:db8::4 -c 4 &> /dev/null && \ + test_pass "$drv_mode ping6 ns2-4" || test_fail "$drv_mode ping6 ns2-4" + + kill $xdp_pid +} + +for mode in ${DRV_MODE}; do + sleep 2 + setup_ns $mode + do_tests $mode + sleep 20 + clean_up +done diff --git a/samples/bpf/xdp_redirect_map_multi_kern.c b/samples/bpf/xdp_redirect_map_multi_kern.c new file mode 100644 index 000000000000..c98985683ba2 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_kern.c @@ -0,0 +1,100 @@ +/* + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <uapi/linux/bpf.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ip.h> +#include <linux/ipv6.h> +#include <bpf/bpf_helpers.h> + +/* In this sample we will use 3 forward maps and 1 exclude map to + * show how to use the helper bpf_redirect_map_multi(). + * + * In real world, there may have multi forward maps and exclude map. You can + * use map-in-map type to store the forward and exlude maps. e.g. + * forward_map_in_map[group_a_index] = forward_group_a_map + * forward_map_in_map[group_b_index] = forward_group_b_map + * exclude_map_in_map[iface_1_index] = iface_1_exclude_map + * exclude_map_in_map[iface_2_index] = iface_2_exclude_map + * Then store the forward group indexes based on IP/MAC policy in another + * hash map, e.g.: + * mcast_route_map[hash(subnet_a)] = group_a_index + * mcast_route_map[hash(subnet_b)] = group_b_index + * + * You can init the maps in user.c, and find the forward group index from + * mcast_route_map bye key hash(subnet) in kern.c, Then you could find + * the forward group by the group index. You can also get the exclude map + * simply by iface index in exclude_map_in_map. + */ +struct bpf_map_def SEC("maps") forward_map_v4 = { + .type = BPF_MAP_TYPE_DEVMAP, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") forward_map_v6 = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") forward_map_all = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") exclude_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + u32 key, mcast_group_id, exclude_group_id; + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + struct ethhdr *eth = data; + int *inmap_id; + u16 h_proto; + u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + h_proto = eth->h_proto; + + if (h_proto == htons(ETH_P_IP)) + return bpf_redirect_map_multi(&forward_map_v4, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else if (h_proto == htons(ETH_P_IPV6)) + return bpf_redirect_map_multi(&forward_map_v6, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else + return bpf_redirect_map_multi(&forward_map_all, NULL, + BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_redirect_dummy") +int xdp_redirect_dummy_prog(struct xdp_md *ctx) +{ + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/samples/bpf/xdp_redirect_map_multi_user.c b/samples/bpf/xdp_redirect_map_multi_user.c new file mode 100644 index 000000000000..2fcd15322201 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_user.c @@ -0,0 +1,170 @@ +/* SPDX-License-Identifier: GPL-2.0-only + */ +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> + +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static int ifaces[MAX_IFACE_NUM] = {}; +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ... \n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n", + prog); +} + +int main(int argc, char **argv) +{ + int prog_fd, group_all, group_v4, group_v6, exclude; + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + int i, ret, opt, ifindex; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNF")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) + xdp_flags |= XDP_FLAGS_DRV_MODE; + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i ++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + return 1; + + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_all"); + group_v4 = bpf_object__find_map_fd_by_name(obj, "forward_map_v4"); + group_v6 = bpf_object__find_map_fd_by_name(obj, "forward_map_v6"); + exclude = bpf_object__find_map_fd_by_name(obj, "exclude_map"); + + if (group_all < 0 || group_v4 < 0 || group_v6 < 0 || exclude < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + return 1; + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* Add all the interfaces to group all */ + ret = bpf_map_update_elem(group_all, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* For testing: remove the 2nd interfaces from group v4 */ + if (i != 1) { + ret = bpf_map_update_elem(group_v4, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: remove the 1st interfaces from group v6 */ + if (i != 0) { + ret = bpf_map_update_elem(group_v6, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: add the 3rd interfaces to exclude map */ + if (i == 2) { + ret = bpf_map_update_elem(exclude, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + } + + sleep(600); + return 0; + +err_out: + return 1; +} -- 2.19.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [RFC PATCHv2 bpf-next 2/2] sample/bpf: add xdp_redirect_map_multicast test 2020-04-24 8:56 ` [RFC PATCHv2 bpf-next 2/2] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu @ 2020-04-24 14:21 ` Lorenzo Bianconi 0 siblings, 0 replies; 219+ messages in thread From: Lorenzo Bianconi @ 2020-04-24 14:21 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann [-- Attachment #1: Type: text/plain, Size: 6979 bytes --] On Apr 24, Hangbin Liu wrote: > This is a sample for xdp multicast. In the sample we have 3 forward > groups and 1 exclude group. It will redirect each interface's > packets to all the interfaces in the forward group, and exclude > the interface in exclude map. > > For more testing details, please see the test description in > xdp_redirect_map_multi.sh. > > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > --- > samples/bpf/Makefile | 3 + > samples/bpf/xdp_redirect_map_multi.sh | 124 ++++++++++++++++ > samples/bpf/xdp_redirect_map_multi_kern.c | 100 +++++++++++++ > samples/bpf/xdp_redirect_map_multi_user.c | 170 ++++++++++++++++++++++ > 4 files changed, 397 insertions(+) > create mode 100755 samples/bpf/xdp_redirect_map_multi.sh > create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c > create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c > > diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile > index 424f6fe7ce38..eb7306efe85e 100644 [...] > + > +SEC("xdp_redirect_map_multi") > +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) > +{ > + u32 key, mcast_group_id, exclude_group_id; > + void *data_end = (void *)(long)ctx->data_end; > + void *data = (void *)(long)ctx->data; > + struct ethhdr *eth = data; > + int *inmap_id; > + u16 h_proto; > + u64 nh_off; > + > + nh_off = sizeof(*eth); > + if (data + nh_off > data_end) > + return XDP_DROP; > + > + h_proto = eth->h_proto; > + > + if (h_proto == htons(ETH_P_IP)) > + return bpf_redirect_map_multi(&forward_map_v4, &exclude_map, > + BPF_F_EXCLUDE_INGRESS); Do we need the 'BPF_F_EXCLUDE_INGRESS' here? > + else if (h_proto == htons(ETH_P_IPV6)) > + return bpf_redirect_map_multi(&forward_map_v6, &exclude_map, > + BPF_F_EXCLUDE_INGRESS); ditto > + else > + return bpf_redirect_map_multi(&forward_map_all, NULL, > + BPF_F_EXCLUDE_INGRESS); > +} > + > +SEC("xdp_redirect_dummy") > +int xdp_redirect_dummy_prog(struct xdp_md *ctx) > +{ > + return XDP_PASS; > +} > + > +char _license[] SEC("license") = "GPL"; > diff --git a/samples/bpf/xdp_redirect_map_multi_user.c b/samples/bpf/xdp_redirect_map_multi_user.c > new file mode 100644 > index 000000000000..2fcd15322201 > --- /dev/null > +++ b/samples/bpf/xdp_redirect_map_multi_user.c > @@ -0,0 +1,170 @@ > +/* SPDX-License-Identifier: GPL-2.0-only > + */ > +#include <linux/bpf.h> > +#include <linux/if_link.h> > +#include <errno.h> > +#include <signal.h> > +#include <stdio.h> > +#include <stdlib.h> > +#include <string.h> > +#include <net/if.h> > +#include <unistd.h> > +#include <libgen.h> > + > +#include <bpf/bpf.h> > +#include <bpf/libbpf.h> > + > +#define MAX_IFACE_NUM 32 > + > +static int ifaces[MAX_IFACE_NUM] = {}; > +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; > + > +static void int_exit(int sig) > +{ > + __u32 prog_id = 0; > + int i; > + > + for (i = 0; ifaces[i] > 0; i++) { > + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { > + printf("bpf_get_link_xdp_id failed\n"); > + exit(1); > + } > + if (prog_id) > + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); > + } > + > + exit(0); > +} > + > +static void usage(const char *prog) > +{ > + fprintf(stderr, > + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ... \n" > + "OPTS:\n" > + " -S use skb-mode\n" > + " -N enforce native mode\n" > + " -F force loading prog\n", > + prog); > +} > + > +int main(int argc, char **argv) > +{ > + int prog_fd, group_all, group_v4, group_v6, exclude; > + struct bpf_prog_load_attr prog_load_attr = { > + .prog_type = BPF_PROG_TYPE_XDP, > + }; > + int i, ret, opt, ifindex; > + char ifname[IF_NAMESIZE]; > + struct bpf_object *obj; > + char filename[256]; > + > + while ((opt = getopt(argc, argv, "SNF")) != -1) { > + switch (opt) { > + case 'S': > + xdp_flags |= XDP_FLAGS_SKB_MODE; > + break; > + case 'N': > + /* default, set below */ > + break; > + case 'F': > + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; > + break; > + default: > + usage(basename(argv[0])); > + return 1; > + } > + } > + > + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) > + xdp_flags |= XDP_FLAGS_DRV_MODE; > + > + if (optind == argc) { > + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); > + return 1; > + } > + > + printf("Get interfaces"); > + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i ++) { > + ifaces[i] = if_nametoindex(argv[optind + i]); > + if (!ifaces[i]) > + ifaces[i] = strtoul(argv[optind + i], NULL, 0); > + if (!if_indextoname(ifaces[i], ifname)) { > + perror("Invalid interface name or i"); > + return 1; > + } > + printf(" %d", ifaces[i]); > + } > + printf("\n"); > + > + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); > + prog_load_attr.file = filename; > + > + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) > + return 1; > + > + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_all"); > + group_v4 = bpf_object__find_map_fd_by_name(obj, "forward_map_v4"); > + group_v6 = bpf_object__find_map_fd_by_name(obj, "forward_map_v6"); > + exclude = bpf_object__find_map_fd_by_name(obj, "exclude_map"); > + > + if (group_all < 0 || group_v4 < 0 || group_v6 < 0 || exclude < 0) { > + printf("bpf_object__find_map_fd_by_name failed\n"); > + return 1; > + } > + > + signal(SIGINT, int_exit); > + signal(SIGTERM, int_exit); > + > + /* Init forward multicast groups and exclude group */ > + for (i = 0; ifaces[i] > 0; i++) { > + ifindex = ifaces[i]; > + > + /* Add all the interfaces to group all */ > + ret = bpf_map_update_elem(group_all, &ifindex, &ifindex, 0); > + if (ret) { > + perror("bpf_map_update_elem"); > + goto err_out; > + } > + > + /* For testing: remove the 2nd interfaces from group v4 */ > + if (i != 1) { > + ret = bpf_map_update_elem(group_v4, &ifindex, &ifindex, 0); > + if (ret) { > + perror("bpf_map_update_elem"); > + goto err_out; > + } > + } > + > + /* For testing: remove the 1st interfaces from group v6 */ > + if (i != 0) { > + ret = bpf_map_update_elem(group_v6, &ifindex, &ifindex, 0); > + if (ret) { > + perror("bpf_map_update_elem"); > + goto err_out; > + } > + } > + > + /* For testing: add the 3rd interfaces to exclude map */ > + if (i == 2) { > + ret = bpf_map_update_elem(exclude, &ifindex, &ifindex, 0); > + if (ret) { > + perror("bpf_map_update_elem"); > + goto err_out; > + } > + } > + > + /* bind prog_fd to each interface */ > + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); > + if (ret) { > + printf("Set xdp fd failed on %d\n", ifindex); > + goto err_out; > + } > + > + } > + > + sleep(600); > + return 0; > + > +err_out: > + return 1; > +} > -- > 2.19.2 > [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 228 bytes --] ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv3 bpf-next 0/2] xdp: add dev map multicast support 2020-04-15 8:54 [RFC PATCH bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu ` (2 preceding siblings ...) 2020-04-24 8:56 ` [RFC PATCHv2 bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu @ 2020-05-23 6:05 ` Hangbin Liu 2020-05-23 6:05 ` [PATCHv3 bpf-next 1/2] xdp: add a new helper for " Hangbin Liu 2020-05-23 6:05 ` [PATCHv3 bpf-next 2/2] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2020-05-26 14:05 ` [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu 4 siblings, 2 replies; 219+ messages in thread From: Hangbin Liu @ 2020-05-23 6:05 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu Hi all, This patchset is for xdp multicast support, which has been discussed before[0]. The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because a physical interface can be part of a logical grouping, such as a bond device. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. The example in patch 2 is functional, but not a lot of effort has been made on performance optimisation. I did a simple test(pkt size 64) with pktgen. Here is the test result with BPF_MAP_TYPE_DEVMAP_HASH arrays: bpf_redirect_map() with 1 ingress, 1 egress: generic path: ~1600k pps native path: ~980k pps bpf_redirect_map_multi() with 1 ingress, 3 egress: generic path: ~600k pps native path: ~480k pps bpf_redirect_map_multi() with 1 ingress, 9 egress: generic path: ~125k pps native path: ~100k pps The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop the arrays and do clone skb/xdpf. The native path is slower than generic path as we send skbs by pktgen. So the result looks reasonable. We need also note that the performace number will get slower if we use large BPF_MAP_TYPE_DEVMAP arrays. Last but not least, thanks a lot to Jiri, Eelco, Toke and Jesper for suggestions and help on implementation. [0] https://xdp-project.net/#Handling-multicast v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. f) Add rxcnt map to show the packet transmit speed in sample test. g) Add performace test number. I didn't split the tools/include to a separate patch because I think they are all the same change, and I saw some others also do like this. But I can re-post the patch and split it if you insist. v2: Discussed with Jiri, Toke, Jesper, Eelco, we think the v1 is doing a trick and may make user confused. So let's just add a new helper to make the implementation more clear. Hangbin Liu (2): xdp: add a new helper for dev map multicast support sample/bpf: add xdp_redirect_map_multicast test include/linux/bpf.h | 20 +++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 22 ++- kernel/bpf/devmap.c | 124 ++++++++++++++ kernel/bpf/verifier.c | 6 + net/core/filter.c | 101 ++++++++++- net/core/xdp.c | 26 +++ samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi.sh | 133 +++++++++++++++ samples/bpf/xdp_redirect_map_multi_kern.c | 112 ++++++++++++ samples/bpf/xdp_redirect_map_multi_user.c | 198 ++++++++++++++++++++++ tools/include/uapi/linux/bpf.h | 22 ++- 13 files changed, 762 insertions(+), 7 deletions(-) create mode 100755 samples/bpf/xdp_redirect_map_multi.sh create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c -- 2.25.4 ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv3 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-05-23 6:05 ` [PATCHv3 bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu @ 2020-05-23 6:05 ` Hangbin Liu 2020-05-26 7:34 ` kbuild test robot 2020-05-23 6:05 ` [PATCHv3 bpf-next 2/2] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 1 sibling, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-05-23 6:05 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu This patch is for xdp multicast support. In this implementation we add a new helper to accept two maps: forward map and exclude map. We will redirect the packet to all the interfaces in *forward map*, but exclude the interfaces that in *exclude map*. To achive this I add a new ex_map for struct bpf_redirect_info. in the helper I set tgt_value to NULL to make a difference with bpf_xdp_redirect_map() We also add a flag *BPF_F_EXCLUDE_INGRESS* incase you don't want to create a exclude map for each interface and just want to exclude the ingress interface. The general data path is kept in net/core/filter.c. The native data path is in kernel/bpf/devmap.c so we can use direct calls to get better performace. v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- include/linux/bpf.h | 20 ++++++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 22 +++++- kernel/bpf/devmap.c | 124 +++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++ net/core/filter.c | 101 +++++++++++++++++++++++++-- net/core/xdp.c | 26 +++++++ tools/include/uapi/linux/bpf.h | 22 +++++- 9 files changed, 316 insertions(+), 7 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index efe8836b5c48..d1c169bec6b5 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1240,6 +1240,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx); int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, struct net_device *dev_rx); +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex); +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + bool exclude_ingress); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); @@ -1377,6 +1382,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return 0; } +static inline +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + return false; +} + +static inline +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + bool exclude_ingress) +{ + return 0; +} + struct sk_buff; static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, diff --git a/include/linux/filter.h b/include/linux/filter.h index 73d06a39e2d6..5d9c6ac6ade3 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -612,6 +612,7 @@ struct bpf_redirect_info { u32 tgt_index; void *tgt_value; struct bpf_map *map; + struct bpf_map *ex_map; u32 kern_flags; }; diff --git a/include/net/xdp.h b/include/net/xdp.h index 90f11760bd12..967684aa096a 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -105,6 +105,7 @@ void xdp_warn(const char *msg, const char *func, const int line); #define XDP_WARN(msg) xdp_warn(msg, __func__, __LINE__) struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); /* Convert xdp_buff to xdp_frame */ static inline diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 97e1fd19ff58..000b0cf961ea 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3157,6 +3157,20 @@ union bpf_attr { * **bpf_sk_cgroup_id**\ (). * Return * The id is returned or 0 in case the id could not be retrieved. + * + * int bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * Redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map* (which may be NULL). + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map(), which supports redirecting + * packet to a specific ifindex in the map. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. + * */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3288,7 +3302,8 @@ union bpf_attr { FN(seq_printf), \ FN(seq_write), \ FN(sk_cgroup_id), \ - FN(sk_ancestor_cgroup_id), + FN(sk_ancestor_cgroup_id), \ + FN(redirect_map_multi), /* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call @@ -3417,6 +3432,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index a51d9fb7a359..ecc5c44a5bab 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -455,6 +455,130 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return __xdp_enqueue(dev, xdp, dev_rx); } +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + return dev_map_get_next_key(map, key, next_key); + case BPF_MAP_TYPE_DEVMAP_HASH: + return dev_map_hash_get_next_key(map, key, next_key); + default: + break; + } + + return -ENOENT; +} + +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + struct bpf_dtab_netdev *in_obj = NULL; + u32 key, next_key; + int err; + + if (obj->dev->ifindex == exclude_ifindex) + return true; + + if (!map) + return false; + + devmap_get_next_key(map, NULL, &key); + + for (;;) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + in_obj = __dev_map_lookup_elem(map, key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + in_obj = __dev_map_hash_lookup_elem(map, key); + break; + default: + break; + } + + if (in_obj && in_obj->dev->ifindex == obj->dev->ifindex) + return true; + + err = devmap_get_next_key(map, &key, &next_key); + + if (err) + break; + + key = next_key; + } + + return false; +} + +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + bool exclude_ingress) +{ + struct bpf_dtab_netdev *obj = NULL; + struct xdp_frame *xdpf, *nxdpf; + struct net_device *dev; + bool first = true; + u32 key, next_key; + int err; + + devmap_get_next_key(map, NULL, &key); + + xdpf = convert_to_xdp_frame(xdp); + if (unlikely(!xdpf)) + return -EOVERFLOW; + + for (;;) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + obj = __dev_map_lookup_elem(map, key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + obj = __dev_map_hash_lookup_elem(map, key); + break; + default: + break; + } + + if (!obj || dev_in_exclude_map(obj, ex_map, + exclude_ingress ? dev_rx->ifindex : 0)) + goto find_next; + + dev = obj->dev; + + if (!dev->netdev_ops->ndo_xdp_xmit) + goto find_next; + + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + if (unlikely(err)) + goto find_next; + + if (!first) { + nxdpf = xdpf_clone(xdpf); + if (unlikely(!nxdpf)) + return -ENOMEM; + + bq_enqueue(dev, nxdpf, dev_rx); + } else { + bq_enqueue(dev, xdpf, dev_rx); + first = false; + } + +find_next: + err = devmap_get_next_key(map, &key, &next_key); + if (err) + break; + key = next_key; + } + + /* didn't find anywhere to forward to, free buf */ + if (first) + xdp_return_frame_rx_napi(xdpf); + + return 0; +} + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d2e27dba4ac6..a5857953248d 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -3946,6 +3946,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: if (func_id != BPF_FUNC_redirect_map && + func_id != BPF_FUNC_redirect_map_multi && func_id != BPF_FUNC_map_lookup_elem) goto error; break; @@ -4038,6 +4039,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, map->map_type != BPF_MAP_TYPE_XSKMAP) goto error; break; + case BPF_FUNC_redirect_map_multi: + if (map->map_type != BPF_MAP_TYPE_DEVMAP && + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + goto error; + break; case BPF_FUNC_sk_redirect_map: case BPF_FUNC_msg_redirect_map: case BPF_FUNC_sock_map_update: diff --git a/net/core/filter.c b/net/core/filter.c index bd2853d23b50..f07eb1408f70 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3473,12 +3473,17 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { }; static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) + struct bpf_map *map, struct xdp_buff *xdp, + struct bpf_map *ex_map, bool exclude_ingress) { switch (map->map_type) { case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: - return dev_map_enqueue(fwd, xdp, dev_rx); + if (fwd) + return dev_map_enqueue(fwd, xdp, dev_rx); + else + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, + exclude_ingress); case BPF_MAP_TYPE_CPUMAP: return cpu_map_enqueue(fwd, xdp, dev_rx); case BPF_MAP_TYPE_XSKMAP: @@ -3534,6 +3539,8 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, struct bpf_prog *xdp_prog) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + bool exclude_ingress = !!(ri->flags & BPF_F_EXCLUDE_INGRESS); + struct bpf_map *ex_map = ri->ex_map; struct bpf_map *map = READ_ONCE(ri->map); u32 index = ri->tgt_index; void *fwd = ri->tgt_value; @@ -3541,6 +3548,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (unlikely(!map)) { @@ -3552,7 +3560,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = dev_xdp_enqueue(fwd, xdp, dev); } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, exclude_ingress); } if (unlikely(err)) @@ -3566,6 +3574,50 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, } EXPORT_SYMBOL_GPL(xdp_do_redirect); +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, + struct bpf_prog *xdp_prog, + struct bpf_map *map, struct bpf_map *ex_map, + bool exclude_ingress) + +{ + struct bpf_dtab_netdev *dst; + struct sk_buff *nskb; + u32 key, next_key; + int err; + void *fwd; + + /* Get first key from forward map */ + map->ops->map_get_next_key(map, NULL, &key); + + for (;;) { + fwd = __xdp_map_lookup_elem(map, key); + if (fwd) { + dst = (struct bpf_dtab_netdev *)fwd; + if (dev_in_exclude_map(dst, ex_map, + exclude_ingress ? dev->ifindex : 0)) + goto find_next; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -ENOMEM; + + err = dev_map_generic_redirect(dst, nskb, xdp_prog); + if (unlikely(err)) + return err; + } + +find_next: + err = map->ops->map_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + consume_skb(skb); + return 0; +} + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, @@ -3573,19 +3625,29 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, struct bpf_map *map) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + bool exclude_ingress = !!(ri->flags & BPF_F_EXCLUDE_INGRESS); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err = 0; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (map->map_type == BPF_MAP_TYPE_DEVMAP || map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { - struct bpf_dtab_netdev *dst = fwd; + if (fwd) { + struct bpf_dtab_netdev *dst = fwd; + + err = dev_map_generic_redirect(dst, skb, xdp_prog); + } else { + /* Deal with multicast maps */ + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, + ex_map, exclude_ingress); + } - err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { @@ -3699,6 +3761,33 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { .arg3_type = ARG_ANYTHING, }; +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, + struct bpf_map *, ex_map, u64, flags) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + + if (unlikely(!map || flags > BPF_F_EXCLUDE_INGRESS)) + return XDP_ABORTED; + + ri->tgt_index = 0; + ri->tgt_value = NULL; + ri->flags = flags; + ri->ex_map = ex_map; + + WRITE_ONCE(ri->map, map); + + return XDP_REDIRECT; +} + +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { + .func = bpf_xdp_redirect_map_multi, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_CONST_MAP_PTR, + .arg1_type = ARG_CONST_MAP_PTR, + .arg3_type = ARG_ANYTHING, +}; + static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, unsigned long off, unsigned long len) { @@ -6363,6 +6452,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_redirect_proto; case BPF_FUNC_redirect_map: return &bpf_xdp_redirect_map_proto; + case BPF_FUNC_redirect_map_multi: + return &bpf_xdp_redirect_map_multi_proto; case BPF_FUNC_xdp_adjust_tail: return &bpf_xdp_adjust_tail_proto; case BPF_FUNC_fib_lookup: diff --git a/net/core/xdp.c b/net/core/xdp.c index 90f44f382115..acdc63833b1f 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -475,3 +475,29 @@ void xdp_warn(const char *msg, const char *func, const int line) WARN(1, "XDP_WARN: %s(line:%d): %s\n", func, line, msg); }; EXPORT_SYMBOL_GPL(xdp_warn); + +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) +{ + unsigned int headroom, totalsize; + struct xdp_frame *nxdpf; + struct page *page; + void *addr; + + headroom = xdpf->headroom + sizeof(*xdpf); + totalsize = headroom + xdpf->len; + + if (unlikely(totalsize > PAGE_SIZE)) + return NULL; + page = dev_alloc_page(); + if (!page) + return NULL; + addr = page_to_virt(page); + + memcpy(addr, xdpf, totalsize); + + nxdpf = addr; + nxdpf->data = addr + headroom; + + return nxdpf; +} +EXPORT_SYMBOL_GPL(xdpf_clone); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 97e1fd19ff58..000b0cf961ea 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3157,6 +3157,20 @@ union bpf_attr { * **bpf_sk_cgroup_id**\ (). * Return * The id is returned or 0 in case the id could not be retrieved. + * + * int bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * Redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map* (which may be NULL). + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map(), which supports redirecting + * packet to a specific ifindex in the map. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. + * */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3288,7 +3302,8 @@ union bpf_attr { FN(seq_printf), \ FN(seq_write), \ FN(sk_cgroup_id), \ - FN(sk_ancestor_cgroup_id), + FN(sk_ancestor_cgroup_id), \ + FN(redirect_map_multi), /* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call @@ -3417,6 +3432,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv3 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-05-23 6:05 ` [PATCHv3 bpf-next 1/2] xdp: add a new helper for " Hangbin Liu @ 2020-05-26 7:34 ` kbuild test robot 0 siblings, 0 replies; 219+ messages in thread From: kbuild test robot @ 2020-05-26 7:34 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: kbuild-all, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu [-- Attachment #1: Type: text/plain, Size: 5721 bytes --] Hi Hangbin, Thank you for the patch! Perhaps something to improve: [auto build test WARNING on bpf-next/master] [also build test WARNING on net-next/master next-20200525] [cannot apply to bpf/master net/master linus/master v5.7-rc7] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982] url: https://github.com/0day-ci/linux/commits/Hangbin-Liu/xdp-add-dev-map-multicast-support/20200523-141019 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master config: c6x-randconfig-r003-20200526 (attached as .config) compiler: c6x-elf-gcc (GCC) 9.3.0 reproduce (this is a W=1 build): wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # save the attached .config to linux build tree COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=c6x If you fix the issue, kindly add following tag as appropriate Reported-by: kbuild test robot <lkp@intel.com> All warnings (new ones prefixed by >>, old ones prefixed by <<): In file included from include/asm-generic/atomic.h:12, from ./arch/c6x/include/generated/asm/atomic.h:1, from include/linux/atomic.h:7, from include/asm-generic/bitops/lock.h:5, from arch/c6x/include/asm/bitops.h:87, from include/linux/bitops.h:29, from include/linux/kernel.h:12, from include/linux/list.h:9, from include/linux/module.h:12, from net/core/filter.c:20: net/core/filter.c: In function 'bpf_clear_redirect_map': arch/c6x/include/asm/cmpxchg.h:55:3: warning: value computed is not used [-Wunused-value] 55 | ((__typeof__(*(ptr)))__cmpxchg_local_generic((ptr), | ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 56 | (unsigned long)(o), | ~~~~~~~~~~~~~~~~~~~~~ 57 | (unsigned long)(n), | ~~~~~~~~~~~~~~~~~~~~~ 58 | sizeof(*(ptr)))) | ~~~~~~~~~~~~~~~~ include/asm-generic/cmpxchg.h:106:28: note: in expansion of macro 'cmpxchg_local' 106 | #define cmpxchg(ptr, o, n) cmpxchg_local((ptr), (o), (n)) | ^~~~~~~~~~~~~ net/core/filter.c:3534:4: note: in expansion of macro 'cmpxchg' 3534 | cmpxchg(&ri->map, map, NULL); | ^~~~~~~ net/core/filter.c: At top level: >> net/core/filter.c:3787:20: warning: initialized field overwritten [-Woverride-init] 3787 | .arg1_type = ARG_CONST_MAP_PTR, | ^~~~~~~~~~~~~~~~~ net/core/filter.c:3787:20: note: (near initialization for 'bpf_xdp_redirect_map_multi_proto.<anonymous>.<anonymous>.arg1_type') /tmp/cc2n7hPR.s: Assembler messages: /tmp/cc2n7hPR.s:69347: Warning: ignoring changed section type for .far /tmp/cc2n7hPR.s:69347: Warning: ignoring changed section attributes for .far /tmp/cc2n7hPR.s:69454: Warning: ignoring changed section type for .far /tmp/cc2n7hPR.s:69454: Warning: ignoring changed section attributes for .far /tmp/cc2n7hPR.s:69503: Warning: ignoring changed section type for .far /tmp/cc2n7hPR.s:69503: Warning: ignoring changed section attributes for .far -- In file included from include/asm-generic/atomic.h:12, from ./arch/c6x/include/generated/asm/atomic.h:1, from include/linux/atomic.h:7, from include/asm-generic/bitops/lock.h:5, from arch/c6x/include/asm/bitops.h:87, from include/linux/bitops.h:29, from include/linux/kernel.h:12, from include/linux/list.h:9, from include/linux/module.h:12, from net/core/filter.c:20: net/core/filter.c: In function 'bpf_clear_redirect_map': arch/c6x/include/asm/cmpxchg.h:55:3: warning: value computed is not used [-Wunused-value] 55 | ((__typeof__(*(ptr)))__cmpxchg_local_generic((ptr), | ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 56 | (unsigned long)(o), | ~~~~~~~~~~~~~~~~~~~~~ 57 | (unsigned long)(n), | ~~~~~~~~~~~~~~~~~~~~~ 58 | sizeof(*(ptr)))) | ~~~~~~~~~~~~~~~~ include/asm-generic/cmpxchg.h:106:28: note: in expansion of macro 'cmpxchg_local' 106 | #define cmpxchg(ptr, o, n) cmpxchg_local((ptr), (o), (n)) | ^~~~~~~~~~~~~ net/core/filter.c:3534:4: note: in expansion of macro 'cmpxchg' 3534 | cmpxchg(&ri->map, map, NULL); | ^~~~~~~ net/core/filter.c: At top level: >> net/core/filter.c:3787:20: warning: initialized field overwritten [-Woverride-init] 3787 | .arg1_type = ARG_CONST_MAP_PTR, | ^~~~~~~~~~~~~~~~~ net/core/filter.c:3787:20: note: (near initialization for 'bpf_xdp_redirect_map_multi_proto.<anonymous>.<anonymous>.arg1_type') /tmp/ccHtg48M.s: Assembler messages: /tmp/ccHtg48M.s:69347: Warning: ignoring changed section type for .far /tmp/ccHtg48M.s:69347: Warning: ignoring changed section attributes for .far /tmp/ccHtg48M.s:69454: Warning: ignoring changed section type for .far /tmp/ccHtg48M.s:69454: Warning: ignoring changed section attributes for .far /tmp/ccHtg48M.s:69503: Warning: ignoring changed section type for .far /tmp/ccHtg48M.s:69503: Warning: ignoring changed section attributes for .far vim +3787 net/core/filter.c 3781 3782 static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { 3783 .func = bpf_xdp_redirect_map_multi, 3784 .gpl_only = false, 3785 .ret_type = RET_INTEGER, 3786 .arg1_type = ARG_CONST_MAP_PTR, > 3787 .arg1_type = ARG_CONST_MAP_PTR, 3788 .arg3_type = ARG_ANYTHING, 3789 }; 3790 --- 0-DAY CI Kernel Test Service, Intel Corporation https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org [-- Attachment #2: .config.gz --] [-- Type: application/gzip, Size: 23268 bytes --] ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv3 bpf-next 2/2] sample/bpf: add xdp_redirect_map_multicast test 2020-05-23 6:05 ` [PATCHv3 bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu 2020-05-23 6:05 ` [PATCHv3 bpf-next 1/2] xdp: add a new helper for " Hangbin Liu @ 2020-05-23 6:05 ` Hangbin Liu 1 sibling, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-05-23 6:05 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu This is a sample for xdp multicast. In the sample we have 3 forward groups and 1 exclude group. It will redirect each interface's packets to all the interfaces in the forward group, and exclude the interface in exclude map. For more testing details, please see the test description in xdp_redirect_map_multi.sh. v3: add rxcnt map to show the packet transmit speed. v2: no update. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi.sh | 135 +++++++++++++++ samples/bpf/xdp_redirect_map_multi_kern.c | 113 +++++++++++++ samples/bpf/xdp_redirect_map_multi_user.c | 197 ++++++++++++++++++++++ 4 files changed, 448 insertions(+) create mode 100755 samples/bpf/xdp_redirect_map_multi.sh create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 8403e4762306..000709bb89c3 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map tprogs-y += per_socket_stats_example tprogs-y += xdp_redirect tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_map_multi tprogs-y += xdp_redirect_cpu tprogs-y += xdp_monitor tprogs-y += xdp_rxq_info @@ -97,6 +98,7 @@ test_map_in_map-objs := bpf_load.o test_map_in_map_user.o per_socket_stats_example-objs := cookie_uid_helper_example.o xdp_redirect-objs := xdp_redirect_user.o xdp_redirect_map-objs := xdp_redirect_map_user.o +xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o xdp_redirect_cpu-objs := bpf_load.o xdp_redirect_cpu_user.o xdp_monitor-objs := bpf_load.o xdp_monitor_user.o xdp_rxq_info-objs := xdp_rxq_info_user.o @@ -156,6 +158,7 @@ always-y += tcp_tos_reflect_kern.o always-y += tcp_dumpstats_kern.o always-y += xdp_redirect_kern.o always-y += xdp_redirect_map_kern.o +always-y += xdp_redirect_map_multi_kern.o always-y += xdp_redirect_cpu_kern.o always-y += xdp_monitor_kern.o always-y += xdp_rxq_info_kern.o diff --git a/samples/bpf/xdp_redirect_map_multi.sh b/samples/bpf/xdp_redirect_map_multi.sh new file mode 100755 index 000000000000..bbf10ca06720 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi.sh @@ -0,0 +1,135 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Test topology: +# - - - - - - - - - - - - - - - - - - - - - - - - - +# | veth1 veth2 veth3 veth4 | ... init net +# - -| - - - - - - | - - - - - - | - - - - - - | - - +# --------- --------- --------- --------- +# | veth0 | | veth0 | | veth0 | | veth0 | ... +# --------- --------- --------- --------- +# ns1 ns2 ns3 ns4 +# +# Forward multicast groups: +# Forward group all has interfaces: veth1, veth2, veth3, veth4, ... (All traffic except IPv4, IPv6) +# Forward group v4 has interfaces: veth1, veth3, veth4, ... (For IPv4 traffic only) +# Forward group v6 has interfaces: veth2, veth3, veth4, ... (For IPv6 traffic only) +# Exclude Groups: +# Exclude group: veth3 (assume ns3 is in black list) +# +# Test modules: +# XDP modes: generic, native +# map types: group v4 use DEVMAP, others use DEVMAP_HASH +# +# Test cases: +# ARP(we didn't exclude ns3 in kern.c for ARP): +# ns1 -> gw: ns2, ns3, ns4 should receive the arp request +# IPv4: +# ns1 -> ns2 (fail), ns1 -> ns3 (fail), ns1 -> ns4 (pass) +# IPv6 +# ns2 -> ns1 (fail), ns2 -> ns3 (fail), ns2 -> ns4 (pass) +# + + +# netns numbers +NUM=10 +IFACES="" +DRV_MODE="generic drv" + +test_pass() +{ + echo "Pass: $@" +} + +test_fail() +{ + echo "fail: $@" +} + +clean_up() +{ + for i in $(seq $NUM); do + ip netns del ns$i + done +} + +setup_ns() +{ + local mode=$1 + + for i in $(seq $NUM); do + ip netns add ns$i + ip link add veth0 type veth peer name veth$i + ip link set veth0 netns ns$i + ip -n ns$i link set veth0 up + ip link set veth$i up + + ip -n ns$i addr add 192.0.2.$i/24 dev veth0 + ip -n ns$i addr add 2001:db8::$i/24 dev veth0 + ip -n ns$i link set veth0 xdp$mode obj \ + xdp_redirect_map_multi_kern.o sec xdp_redirect_dummy &> /dev/null || \ + { test_fail "Unable to load dummy xdp" && exit 1; } + IFACES="$IFACES veth$i" + done +} + +do_ping_tests() +{ + local drv_mode=$1 + + # arp test + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns1-2_${drv_mode}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns1-3_${drv_mode}.log & + ip netns exec ns4 tcpdump -i veth0 -nn -l -e &> arp_ns1-4_${drv_mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-2_${drv_mode}.log && \ + test_pass "$drv_mode arp ns1-2" || test_fail "$drv_mode arp ns1-2" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-3_${drv_mode}.log && \ + test_pass "$drv_mode arp ns1-3" || test_fail "$drv_mode arp ns1-3" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-4_${drv_mode}.log && \ + test_pass "$drv_mode arp ns1-4" || test_fail "$drv_mode arp ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_fail "$drv_mode ping ns1-2" || test_pass "$drv_mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_fail "$drv_mode ping ns1-3" || test_pass "$drv_mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$drv_mode ping ns1-4" || test_fail "$drv_mode ping ns1-4" + + # ping6 test + ip netns exec ns2 ping6 2001:db8::1 -c 4 &> /dev/null && \ + test_fail "$drv_mode ping6 ns2-1" || test_pass "$drv_mode ping6 ns2-1" + ip netns exec ns2 ping6 2001:db8::3 -c 4 &> /dev/null && \ + test_fail "$drv_mode ping6 ns2-3" || test_pass "$drv_mode ping6 ns2-3" + ip netns exec ns2 ping6 2001:db8::4 -c 4 &> /dev/null && \ + test_pass "$drv_mode ping6 ns2-4" || test_fail "$drv_mode ping6 ns2-4" +} + +do_tests() +{ + local drv_mode=$1 + local drv_p + + [ ${drv_mode} == "drv" ] && drv_p="-N" || drv_p="-S" + + # run `ulimit -l unlimited` if you got errors like + # libbpf: Error in bpf_object__probe_global_data():Operation not permitted(1). + ./xdp_redirect_map_multi $drv_p $IFACES &> xdp_${drv_mode}.log & + xdp_pid=$! + sleep 10 + + do_ping_tests $drv_mode + + kill $xdp_pid +} + +for mode in ${DRV_MODE}; do + setup_ns $mode + do_tests $mode + sleep 10 + clean_up + sleep 5 +done diff --git a/samples/bpf/xdp_redirect_map_multi_kern.c b/samples/bpf/xdp_redirect_map_multi_kern.c new file mode 100644 index 000000000000..81f71461a252 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_kern.c @@ -0,0 +1,113 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <uapi/linux/bpf.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ip.h> +#include <linux/ipv6.h> +#include <bpf/bpf_helpers.h> + +/* In this sample we will use 3 forward maps and 1 exclude map to + * show how to use the helper bpf_redirect_map_multi(). + * + * In real world, there may have multi forward maps and exclude map. You can + * use map-in-map type to store the forward and exlude maps. e.g. + * forward_map_in_map[group_a_index] = forward_group_a_map + * forward_map_in_map[group_b_index] = forward_group_b_map + * exclude_map_in_map[iface_1_index] = iface_1_exclude_map + * exclude_map_in_map[iface_2_index] = iface_2_exclude_map + * Then store the forward group indexes based on IP/MAC policy in another + * hash map, e.g.: + * mcast_route_map[hash(subnet_a)] = group_a_index + * mcast_route_map[hash(subnet_b)] = group_b_index + * + * You can init the maps in user.c, and find the forward group index from + * mcast_route_map bye key hash(subnet) in kern.c, Then you could find + * the forward group by the group index. You can also get the exclude map + * simply by iface index in exclude_map_in_map. + */ +struct bpf_map_def SEC("maps") forward_map_v4 = { + .type = BPF_MAP_TYPE_DEVMAP, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 4096, +}; + +struct bpf_map_def SEC("maps") forward_map_v6 = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") forward_map_all = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") exclude_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") rxcnt = { + .type = BPF_MAP_TYPE_PERCPU_ARRAY, + .key_size = sizeof(u32), + .value_size = sizeof(long), + .max_entries = 1, +}; + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + struct ethhdr *eth = data; + long *value; + u16 h_proto; + u32 key = 0; + u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + h_proto = eth->h_proto; + + /* count packet in global counter */ + value = bpf_map_lookup_elem(&rxcnt, &key); + if (value) + *value += 1; + + if (h_proto == htons(ETH_P_IP)) + return bpf_redirect_map_multi(&forward_map_v4, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else if (h_proto == htons(ETH_P_IPV6)) + return bpf_redirect_map_multi(&forward_map_v6, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else + return bpf_redirect_map_multi(&forward_map_all, NULL, + BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_redirect_dummy") +int xdp_redirect_dummy_prog(struct xdp_md *ctx) +{ + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/samples/bpf/xdp_redirect_map_multi_user.c b/samples/bpf/xdp_redirect_map_multi_user.c new file mode 100644 index 000000000000..7339ce4c7f9c --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_user.c @@ -0,0 +1,197 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static int ifaces[MAX_IFACE_NUM] = {}; +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int rxcnt; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void poll_stats(int interval) +{ + unsigned int nr_cpus = bpf_num_possible_cpus(); + __u64 values[nr_cpus], prev[nr_cpus]; + + memset(prev, 0, sizeof(prev)); + + while (1) { + __u64 sum = 0; + __u32 key = 0; + int i; + + sleep(interval); + assert(bpf_map_lookup_elem(rxcnt, &key, values) == 0); + for (i = 0; i < nr_cpus; i++) + sum += (values[i] - prev[i]); + if (sum) + printf("Forwarding %10llu pkt/s\n", sum / interval); + memcpy(prev, values, sizeof(values)); + } +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n", + prog); +} + +int main(int argc, char **argv) +{ + int prog_fd, group_all, group_v4, group_v6, exclude; + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + int i, ret, opt, ifindex; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNF")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) + xdp_flags |= XDP_FLAGS_DRV_MODE; + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + return 1; + + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_all"); + group_v4 = bpf_object__find_map_fd_by_name(obj, "forward_map_v4"); + group_v6 = bpf_object__find_map_fd_by_name(obj, "forward_map_v6"); + exclude = bpf_object__find_map_fd_by_name(obj, "exclude_map"); + rxcnt = bpf_object__find_map_fd_by_name(obj, "rxcnt"); + + if (group_all < 0 || group_v4 < 0 || group_v6 < 0 || exclude < 0 || + rxcnt < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + return 1; + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* Add all the interfaces to group all */ + ret = bpf_map_update_elem(group_all, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* For testing: remove the 2nd interfaces from group v4 */ + if (i != 1) { + ret = bpf_map_update_elem(group_v4, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: remove the 1st interfaces from group v6 */ + if (i != 0) { + ret = bpf_map_update_elem(group_v6, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: add the 3rd interfaces to exclude map */ + if (i == 2) { + ret = bpf_map_update_elem(exclude, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + } + + poll_stats(2); + + return 0; + +err_out: + return 1; +} -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-04-15 8:54 [RFC PATCH bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu ` (3 preceding siblings ...) 2020-05-23 6:05 ` [PATCHv3 bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu @ 2020-05-26 14:05 ` Hangbin Liu 2020-05-26 14:05 ` [PATCHv4 bpf-next 1/2] xdp: add a new helper for " Hangbin Liu ` (3 more replies) 4 siblings, 4 replies; 219+ messages in thread From: Hangbin Liu @ 2020-05-26 14:05 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu Hi all, This patchset is for xdp multicast support, which has been discussed before[0]. The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because a physical interface can be part of a logical grouping, such as a bond device. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. The example in patch 2 is functional, but not a lot of effort has been made on performance optimisation. I did a simple test(pkt size 64) with pktgen. Here is the test result with BPF_MAP_TYPE_DEVMAP_HASH arrays: bpf_redirect_map() with 1 ingress, 1 egress: generic path: ~1600k pps native path: ~980k pps bpf_redirect_map_multi() with 1 ingress, 3 egress: generic path: ~600k pps native path: ~480k pps bpf_redirect_map_multi() with 1 ingress, 9 egress: generic path: ~125k pps native path: ~100k pps The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop the arrays and do clone skb/xdpf. The native path is slower than generic path as we send skbs by pktgen. So the result looks reasonable. We need also note that the performace number will get slower if we use large BPF_MAP_TYPE_DEVMAP arrays. Last but not least, thanks a lot to Jiri, Eelco, Toke and Jesper for suggestions and help on implementation. [0] https://xdp-project.net/#Handling-multicast v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. f) Add rxcnt map to show the packet transmit speed in sample test. g) Add performace test number. I didn't split the tools/include to a separate patch because I think they are all the same change, and I saw some others also do like this. But I can re-post the patch and split it if you insist. v2: Discussed with Jiri, Toke, Jesper, Eelco, we think the v1 is doing a trick and may make user confused. So let's just add a new helper to make the implementation more clear. Hangbin Liu (2): xdp: add a new helper for dev map multicast support sample/bpf: add xdp_redirect_map_multicast test include/linux/bpf.h | 20 +++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 22 ++- kernel/bpf/devmap.c | 124 ++++++++++++++ kernel/bpf/verifier.c | 6 + net/core/filter.c | 101 ++++++++++- net/core/xdp.c | 26 +++ samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi.sh | 133 +++++++++++++++ samples/bpf/xdp_redirect_map_multi_kern.c | 112 ++++++++++++ samples/bpf/xdp_redirect_map_multi_user.c | 198 ++++++++++++++++++++++ tools/include/uapi/linux/bpf.h | 22 ++- 13 files changed, 762 insertions(+), 7 deletions(-) create mode 100755 samples/bpf/xdp_redirect_map_multi.sh create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c -- 2.25.4 ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv4 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-05-26 14:05 ` [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu @ 2020-05-26 14:05 ` Hangbin Liu 2020-05-27 10:29 ` Toke Høiland-Jørgensen ` (2 more replies) 2020-05-26 14:05 ` [PATCHv4 bpf-next 2/2] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu ` (2 subsequent siblings) 3 siblings, 3 replies; 219+ messages in thread From: Hangbin Liu @ 2020-05-26 14:05 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu This patch is for xdp multicast support. In this implementation we add a new helper to accept two maps: forward map and exclude map. We will redirect the packet to all the interfaces in *forward map*, but exclude the interfaces that in *exclude map*. To achive this I add a new ex_map for struct bpf_redirect_info. in the helper I set tgt_value to NULL to make a difference with bpf_xdp_redirect_map() We also add a flag *BPF_F_EXCLUDE_INGRESS* incase you don't want to create a exclude map for each interface and just want to exclude the ingress interface. The general data path is kept in net/core/filter.c. The native data path is in kernel/bpf/devmap.c so we can use direct calls to get better performace. v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- include/linux/bpf.h | 20 ++++++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 22 +++++- kernel/bpf/devmap.c | 124 +++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++ net/core/filter.c | 101 +++++++++++++++++++++++++-- net/core/xdp.c | 26 +++++++ tools/include/uapi/linux/bpf.h | 22 +++++- 9 files changed, 316 insertions(+), 7 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index efe8836b5c48..d1c169bec6b5 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1240,6 +1240,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx); int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, struct net_device *dev_rx); +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex); +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + bool exclude_ingress); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); @@ -1377,6 +1382,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return 0; } +static inline +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + return false; +} + +static inline +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + bool exclude_ingress) +{ + return 0; +} + struct sk_buff; static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, diff --git a/include/linux/filter.h b/include/linux/filter.h index 73d06a39e2d6..5d9c6ac6ade3 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -612,6 +612,7 @@ struct bpf_redirect_info { u32 tgt_index; void *tgt_value; struct bpf_map *map; + struct bpf_map *ex_map; u32 kern_flags; }; diff --git a/include/net/xdp.h b/include/net/xdp.h index 90f11760bd12..967684aa096a 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -105,6 +105,7 @@ void xdp_warn(const char *msg, const char *func, const int line); #define XDP_WARN(msg) xdp_warn(msg, __func__, __LINE__) struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); /* Convert xdp_buff to xdp_frame */ static inline diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 97e1fd19ff58..000b0cf961ea 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3157,6 +3157,20 @@ union bpf_attr { * **bpf_sk_cgroup_id**\ (). * Return * The id is returned or 0 in case the id could not be retrieved. + * + * int bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * Redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map* (which may be NULL). + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map(), which supports redirecting + * packet to a specific ifindex in the map. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. + * */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3288,7 +3302,8 @@ union bpf_attr { FN(seq_printf), \ FN(seq_write), \ FN(sk_cgroup_id), \ - FN(sk_ancestor_cgroup_id), + FN(sk_ancestor_cgroup_id), \ + FN(redirect_map_multi), /* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call @@ -3417,6 +3432,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index a51d9fb7a359..ecc5c44a5bab 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -455,6 +455,130 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return __xdp_enqueue(dev, xdp, dev_rx); } +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + return dev_map_get_next_key(map, key, next_key); + case BPF_MAP_TYPE_DEVMAP_HASH: + return dev_map_hash_get_next_key(map, key, next_key); + default: + break; + } + + return -ENOENT; +} + +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + struct bpf_dtab_netdev *in_obj = NULL; + u32 key, next_key; + int err; + + if (obj->dev->ifindex == exclude_ifindex) + return true; + + if (!map) + return false; + + devmap_get_next_key(map, NULL, &key); + + for (;;) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + in_obj = __dev_map_lookup_elem(map, key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + in_obj = __dev_map_hash_lookup_elem(map, key); + break; + default: + break; + } + + if (in_obj && in_obj->dev->ifindex == obj->dev->ifindex) + return true; + + err = devmap_get_next_key(map, &key, &next_key); + + if (err) + break; + + key = next_key; + } + + return false; +} + +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + bool exclude_ingress) +{ + struct bpf_dtab_netdev *obj = NULL; + struct xdp_frame *xdpf, *nxdpf; + struct net_device *dev; + bool first = true; + u32 key, next_key; + int err; + + devmap_get_next_key(map, NULL, &key); + + xdpf = convert_to_xdp_frame(xdp); + if (unlikely(!xdpf)) + return -EOVERFLOW; + + for (;;) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + obj = __dev_map_lookup_elem(map, key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + obj = __dev_map_hash_lookup_elem(map, key); + break; + default: + break; + } + + if (!obj || dev_in_exclude_map(obj, ex_map, + exclude_ingress ? dev_rx->ifindex : 0)) + goto find_next; + + dev = obj->dev; + + if (!dev->netdev_ops->ndo_xdp_xmit) + goto find_next; + + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + if (unlikely(err)) + goto find_next; + + if (!first) { + nxdpf = xdpf_clone(xdpf); + if (unlikely(!nxdpf)) + return -ENOMEM; + + bq_enqueue(dev, nxdpf, dev_rx); + } else { + bq_enqueue(dev, xdpf, dev_rx); + first = false; + } + +find_next: + err = devmap_get_next_key(map, &key, &next_key); + if (err) + break; + key = next_key; + } + + /* didn't find anywhere to forward to, free buf */ + if (first) + xdp_return_frame_rx_napi(xdpf); + + return 0; +} + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d2e27dba4ac6..a5857953248d 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -3946,6 +3946,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: if (func_id != BPF_FUNC_redirect_map && + func_id != BPF_FUNC_redirect_map_multi && func_id != BPF_FUNC_map_lookup_elem) goto error; break; @@ -4038,6 +4039,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, map->map_type != BPF_MAP_TYPE_XSKMAP) goto error; break; + case BPF_FUNC_redirect_map_multi: + if (map->map_type != BPF_MAP_TYPE_DEVMAP && + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + goto error; + break; case BPF_FUNC_sk_redirect_map: case BPF_FUNC_msg_redirect_map: case BPF_FUNC_sock_map_update: diff --git a/net/core/filter.c b/net/core/filter.c index bd2853d23b50..f07eb1408f70 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3473,12 +3473,17 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { }; static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) + struct bpf_map *map, struct xdp_buff *xdp, + struct bpf_map *ex_map, bool exclude_ingress) { switch (map->map_type) { case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: - return dev_map_enqueue(fwd, xdp, dev_rx); + if (fwd) + return dev_map_enqueue(fwd, xdp, dev_rx); + else + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, + exclude_ingress); case BPF_MAP_TYPE_CPUMAP: return cpu_map_enqueue(fwd, xdp, dev_rx); case BPF_MAP_TYPE_XSKMAP: @@ -3534,6 +3539,8 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, struct bpf_prog *xdp_prog) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + bool exclude_ingress = !!(ri->flags & BPF_F_EXCLUDE_INGRESS); + struct bpf_map *ex_map = ri->ex_map; struct bpf_map *map = READ_ONCE(ri->map); u32 index = ri->tgt_index; void *fwd = ri->tgt_value; @@ -3541,6 +3548,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (unlikely(!map)) { @@ -3552,7 +3560,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = dev_xdp_enqueue(fwd, xdp, dev); } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, exclude_ingress); } if (unlikely(err)) @@ -3566,6 +3574,50 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, } EXPORT_SYMBOL_GPL(xdp_do_redirect); +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, + struct bpf_prog *xdp_prog, + struct bpf_map *map, struct bpf_map *ex_map, + bool exclude_ingress) + +{ + struct bpf_dtab_netdev *dst; + struct sk_buff *nskb; + u32 key, next_key; + int err; + void *fwd; + + /* Get first key from forward map */ + map->ops->map_get_next_key(map, NULL, &key); + + for (;;) { + fwd = __xdp_map_lookup_elem(map, key); + if (fwd) { + dst = (struct bpf_dtab_netdev *)fwd; + if (dev_in_exclude_map(dst, ex_map, + exclude_ingress ? dev->ifindex : 0)) + goto find_next; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -ENOMEM; + + err = dev_map_generic_redirect(dst, nskb, xdp_prog); + if (unlikely(err)) + return err; + } + +find_next: + err = map->ops->map_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + consume_skb(skb); + return 0; +} + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, @@ -3573,19 +3625,29 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, struct bpf_map *map) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + bool exclude_ingress = !!(ri->flags & BPF_F_EXCLUDE_INGRESS); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err = 0; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (map->map_type == BPF_MAP_TYPE_DEVMAP || map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { - struct bpf_dtab_netdev *dst = fwd; + if (fwd) { + struct bpf_dtab_netdev *dst = fwd; + + err = dev_map_generic_redirect(dst, skb, xdp_prog); + } else { + /* Deal with multicast maps */ + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, + ex_map, exclude_ingress); + } - err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { @@ -3699,6 +3761,33 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { .arg3_type = ARG_ANYTHING, }; +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, + struct bpf_map *, ex_map, u64, flags) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + + if (unlikely(!map || flags > BPF_F_EXCLUDE_INGRESS)) + return XDP_ABORTED; + + ri->tgt_index = 0; + ri->tgt_value = NULL; + ri->flags = flags; + ri->ex_map = ex_map; + + WRITE_ONCE(ri->map, map); + + return XDP_REDIRECT; +} + +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { + .func = bpf_xdp_redirect_map_multi, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_CONST_MAP_PTR, + .arg2_type = ARG_CONST_MAP_PTR, + .arg3_type = ARG_ANYTHING, +}; + static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, unsigned long off, unsigned long len) { @@ -6363,6 +6452,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_redirect_proto; case BPF_FUNC_redirect_map: return &bpf_xdp_redirect_map_proto; + case BPF_FUNC_redirect_map_multi: + return &bpf_xdp_redirect_map_multi_proto; case BPF_FUNC_xdp_adjust_tail: return &bpf_xdp_adjust_tail_proto; case BPF_FUNC_fib_lookup: diff --git a/net/core/xdp.c b/net/core/xdp.c index 90f44f382115..acdc63833b1f 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -475,3 +475,29 @@ void xdp_warn(const char *msg, const char *func, const int line) WARN(1, "XDP_WARN: %s(line:%d): %s\n", func, line, msg); }; EXPORT_SYMBOL_GPL(xdp_warn); + +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) +{ + unsigned int headroom, totalsize; + struct xdp_frame *nxdpf; + struct page *page; + void *addr; + + headroom = xdpf->headroom + sizeof(*xdpf); + totalsize = headroom + xdpf->len; + + if (unlikely(totalsize > PAGE_SIZE)) + return NULL; + page = dev_alloc_page(); + if (!page) + return NULL; + addr = page_to_virt(page); + + memcpy(addr, xdpf, totalsize); + + nxdpf = addr; + nxdpf->data = addr + headroom; + + return nxdpf; +} +EXPORT_SYMBOL_GPL(xdpf_clone); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 97e1fd19ff58..000b0cf961ea 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3157,6 +3157,20 @@ union bpf_attr { * **bpf_sk_cgroup_id**\ (). * Return * The id is returned or 0 in case the id could not be retrieved. + * + * int bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * Redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map* (which may be NULL). + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map(), which supports redirecting + * packet to a specific ifindex in the map. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. + * */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3288,7 +3302,8 @@ union bpf_attr { FN(seq_printf), \ FN(seq_write), \ FN(sk_cgroup_id), \ - FN(sk_ancestor_cgroup_id), + FN(sk_ancestor_cgroup_id), \ + FN(redirect_map_multi), /* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call @@ -3417,6 +3432,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-05-26 14:05 ` [PATCHv4 bpf-next 1/2] xdp: add a new helper for " Hangbin Liu @ 2020-05-27 10:29 ` Toke Høiland-Jørgensen 2020-06-10 10:18 ` Jesper Dangaard Brouer 2020-06-10 10:21 ` Jesper Dangaard Brouer 2 siblings, 0 replies; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2020-05-27 10:29 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu Hangbin Liu <liuhangbin@gmail.com> writes: > This patch is for xdp multicast support. In this implementation we > add a new helper to accept two maps: forward map and exclude map. > We will redirect the packet to all the interfaces in *forward map*, but > exclude the interfaces that in *exclude map*. > > To achive this I add a new ex_map for struct bpf_redirect_info. > in the helper I set tgt_value to NULL to make a difference with > bpf_xdp_redirect_map() > > We also add a flag *BPF_F_EXCLUDE_INGRESS* incase you don't want to > create a exclude map for each interface and just want to exclude the > ingress interface. > > The general data path is kept in net/core/filter.c. The native data > path is in kernel/bpf/devmap.c so we can use direct calls to > get better performace. > > v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo > > v3: Based on Toke's suggestion, do the following update > a) Update bpf_redirect_map_multi() description in bpf.h. > b) Fix exclude_ifindex checking order in dev_in_exclude_map(). > c) Fix one more xdpf clone in dev_map_enqueue_multi(). > d) Go find next one in dev_map_enqueue_multi() if the interface is not > able to forward instead of abort the whole loop. > e) Remove READ_ONCE/WRITE_ONCE for ex_map. > > v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept > include/exclude maps directly. > > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > --- > include/linux/bpf.h | 20 ++++++ > include/linux/filter.h | 1 + > include/net/xdp.h | 1 + > include/uapi/linux/bpf.h | 22 +++++- > kernel/bpf/devmap.c | 124 +++++++++++++++++++++++++++++++++ > kernel/bpf/verifier.c | 6 ++ > net/core/filter.c | 101 +++++++++++++++++++++++++-- > net/core/xdp.c | 26 +++++++ > tools/include/uapi/linux/bpf.h | 22 +++++- > 9 files changed, 316 insertions(+), 7 deletions(-) > > diff --git a/include/linux/bpf.h b/include/linux/bpf.h > index efe8836b5c48..d1c169bec6b5 100644 > --- a/include/linux/bpf.h > +++ b/include/linux/bpf.h > @@ -1240,6 +1240,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, > struct net_device *dev_rx); > int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > struct net_device *dev_rx); > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, > + int exclude_ifindex); > +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, > + struct bpf_map *map, struct bpf_map *ex_map, > + bool exclude_ingress); > int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, > struct bpf_prog *xdp_prog); > > @@ -1377,6 +1382,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > return 0; > } > > +static inline > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, > + int exclude_ifindex) > +{ > + return false; > +} > + > +static inline > +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, > + struct bpf_map *map, struct bpf_map *ex_map, > + bool exclude_ingress) > +{ > + return 0; > +} > + > struct sk_buff; > > static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, > diff --git a/include/linux/filter.h b/include/linux/filter.h > index 73d06a39e2d6..5d9c6ac6ade3 100644 > --- a/include/linux/filter.h > +++ b/include/linux/filter.h > @@ -612,6 +612,7 @@ struct bpf_redirect_info { > u32 tgt_index; > void *tgt_value; > struct bpf_map *map; > + struct bpf_map *ex_map; > u32 kern_flags; > }; > > diff --git a/include/net/xdp.h b/include/net/xdp.h > index 90f11760bd12..967684aa096a 100644 > --- a/include/net/xdp.h > +++ b/include/net/xdp.h > @@ -105,6 +105,7 @@ void xdp_warn(const char *msg, const char *func, const int line); > #define XDP_WARN(msg) xdp_warn(msg, __func__, __LINE__) > > struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); > +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); > > /* Convert xdp_buff to xdp_frame */ > static inline > diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h > index 97e1fd19ff58..000b0cf961ea 100644 > --- a/include/uapi/linux/bpf.h > +++ b/include/uapi/linux/bpf.h > @@ -3157,6 +3157,20 @@ union bpf_attr { > * **bpf_sk_cgroup_id**\ (). > * Return > * The id is returned or 0 in case the id could not be retrieved. > + * > + * int bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) > + * Description > + * Redirect the packet to ALL the interfaces in *map*, but > + * exclude the interfaces in *ex_map* (which may be NULL). > + * > + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, > + * which additionally excludes the current ingress device. > + * > + * See also bpf_redirect_map(), which supports redirecting > + * packet to a specific ifindex in the map. > + * Return > + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. > + * > */ > #define __BPF_FUNC_MAPPER(FN) \ > FN(unspec), \ > @@ -3288,7 +3302,8 @@ union bpf_attr { > FN(seq_printf), \ > FN(seq_write), \ > FN(sk_cgroup_id), \ > - FN(sk_ancestor_cgroup_id), > + FN(sk_ancestor_cgroup_id), \ > + FN(redirect_map_multi), > > /* integer value in 'imm' field of BPF_CALL instruction selects which helper > * function eBPF program intends to call > @@ -3417,6 +3432,11 @@ enum bpf_lwt_encap_mode { > BPF_LWT_ENCAP_IP, > }; > > +/* BPF_FUNC_redirect_map_multi flags. */ > +enum { > + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), > +}; > + > #define __bpf_md_ptr(type, name) \ > union { \ > type name; \ > diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c > index a51d9fb7a359..ecc5c44a5bab 100644 > --- a/kernel/bpf/devmap.c > +++ b/kernel/bpf/devmap.c > @@ -455,6 +455,130 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > return __xdp_enqueue(dev, xdp, dev_rx); > } > > +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ > +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) > +{ > + > + switch (map->map_type) { > + case BPF_MAP_TYPE_DEVMAP: > + return dev_map_get_next_key(map, key, next_key); > + case BPF_MAP_TYPE_DEVMAP_HASH: > + return dev_map_hash_get_next_key(map, key, next_key); > + default: > + break; > + } > + > + return -ENOENT; > +} > + > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, > + int exclude_ifindex) > +{ > + struct bpf_dtab_netdev *in_obj = NULL; > + u32 key, next_key; > + int err; > + > + if (obj->dev->ifindex == exclude_ifindex) > + return true; > + > + if (!map) > + return false; > + > + devmap_get_next_key(map, NULL, &key); You also need to check if this fails; the map could be empty... This goes for all the places you loop through maps below, but not going to repeat the comment :) > + for (;;) { > + switch (map->map_type) { > + case BPF_MAP_TYPE_DEVMAP: > + in_obj = __dev_map_lookup_elem(map, key); > + break; > + case BPF_MAP_TYPE_DEVMAP_HASH: > + in_obj = __dev_map_hash_lookup_elem(map, key); > + break; > + default: > + break; > + } > + > + if (in_obj && in_obj->dev->ifindex == obj->dev->ifindex) > + return true; > + > + err = devmap_get_next_key(map, &key, &next_key); > + > + if (err) > + break; > + > + key = next_key; > + } > + > + return false; > +} > + > +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, > + struct bpf_map *map, struct bpf_map *ex_map, > + bool exclude_ingress) > +{ > + struct bpf_dtab_netdev *obj = NULL; > + struct xdp_frame *xdpf, *nxdpf; > + struct net_device *dev; > + bool first = true; > + u32 key, next_key; > + int err; > + > + devmap_get_next_key(map, NULL, &key); > + > + xdpf = convert_to_xdp_frame(xdp); > + if (unlikely(!xdpf)) > + return -EOVERFLOW; > + > + for (;;) { > + switch (map->map_type) { > + case BPF_MAP_TYPE_DEVMAP: > + obj = __dev_map_lookup_elem(map, key); > + break; > + case BPF_MAP_TYPE_DEVMAP_HASH: > + obj = __dev_map_hash_lookup_elem(map, key); > + break; > + default: > + break; > + } > + > + if (!obj || dev_in_exclude_map(obj, ex_map, > + exclude_ingress ? dev_rx->ifindex : 0)) > + goto find_next; > + > + dev = obj->dev; > + > + if (!dev->netdev_ops->ndo_xdp_xmit) > + goto find_next; > + > + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); > + if (unlikely(err)) > + goto find_next; > + > + if (!first) { > + nxdpf = xdpf_clone(xdpf); > + if (unlikely(!nxdpf)) > + return -ENOMEM; > + > + bq_enqueue(dev, nxdpf, dev_rx); > + } else { > + bq_enqueue(dev, xdpf, dev_rx); > + first = false; > + } > + > +find_next: > + err = devmap_get_next_key(map, &key, &next_key); > + if (err) > + break; > + key = next_key; > + } > + > + /* didn't find anywhere to forward to, free buf */ > + if (first) > + xdp_return_frame_rx_napi(xdpf); > + > + return 0; > +} > + > int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, > struct bpf_prog *xdp_prog) > { > diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c > index d2e27dba4ac6..a5857953248d 100644 > --- a/kernel/bpf/verifier.c > +++ b/kernel/bpf/verifier.c > @@ -3946,6 +3946,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, > case BPF_MAP_TYPE_DEVMAP: > case BPF_MAP_TYPE_DEVMAP_HASH: > if (func_id != BPF_FUNC_redirect_map && > + func_id != BPF_FUNC_redirect_map_multi && > func_id != BPF_FUNC_map_lookup_elem) > goto error; > break; > @@ -4038,6 +4039,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, > map->map_type != BPF_MAP_TYPE_XSKMAP) > goto error; > break; > + case BPF_FUNC_redirect_map_multi: > + if (map->map_type != BPF_MAP_TYPE_DEVMAP && > + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) > + goto error; > + break; > case BPF_FUNC_sk_redirect_map: > case BPF_FUNC_msg_redirect_map: > case BPF_FUNC_sock_map_update: > diff --git a/net/core/filter.c b/net/core/filter.c > index bd2853d23b50..f07eb1408f70 100644 > --- a/net/core/filter.c > +++ b/net/core/filter.c > @@ -3473,12 +3473,17 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { > }; > > static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, > - struct bpf_map *map, struct xdp_buff *xdp) > + struct bpf_map *map, struct xdp_buff *xdp, > + struct bpf_map *ex_map, bool exclude_ingress) Maybe just pass through the flags argument here? > { > switch (map->map_type) { > case BPF_MAP_TYPE_DEVMAP: > case BPF_MAP_TYPE_DEVMAP_HASH: > - return dev_map_enqueue(fwd, xdp, dev_rx); Using a NULL target_value to distinguish between multicast and unicast forwarding is clever, but bordering on 'too clever' :) - took me a little while to figure out this was what you were doing, at least. So please add a comment explaining this, here and in the helper. > + if (fwd) > + return dev_map_enqueue(fwd, xdp, dev_rx); > + else > + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, > + exclude_ingress); > case BPF_MAP_TYPE_CPUMAP: > return cpu_map_enqueue(fwd, xdp, dev_rx); > case BPF_MAP_TYPE_XSKMAP: > @@ -3534,6 +3539,8 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, > struct bpf_prog *xdp_prog) > { > struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); > + bool exclude_ingress = !!(ri->flags & BPF_F_EXCLUDE_INGRESS); > + struct bpf_map *ex_map = ri->ex_map; > struct bpf_map *map = READ_ONCE(ri->map); > u32 index = ri->tgt_index; > void *fwd = ri->tgt_value; > @@ -3541,6 +3548,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, > > ri->tgt_index = 0; > ri->tgt_value = NULL; > + ri->ex_map = NULL; > WRITE_ONCE(ri->map, NULL); > > if (unlikely(!map)) { > @@ -3552,7 +3560,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, > > err = dev_xdp_enqueue(fwd, xdp, dev); > } else { > - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); > + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, exclude_ingress); > } > > if (unlikely(err)) > @@ -3566,6 +3574,50 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, > } > EXPORT_SYMBOL_GPL(xdp_do_redirect); > > +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, > + struct bpf_prog *xdp_prog, > + struct bpf_map *map, struct bpf_map *ex_map, > + bool exclude_ingress) > + > +{ > + struct bpf_dtab_netdev *dst; > + struct sk_buff *nskb; > + u32 key, next_key; > + int err; > + void *fwd; > + > + /* Get first key from forward map */ > + map->ops->map_get_next_key(map, NULL, &key); > + > + for (;;) { > + fwd = __xdp_map_lookup_elem(map, key); > + if (fwd) { > + dst = (struct bpf_dtab_netdev *)fwd; > + if (dev_in_exclude_map(dst, ex_map, > + exclude_ingress ? dev->ifindex : 0)) > + goto find_next; > + > + nskb = skb_clone(skb, GFP_ATOMIC); > + if (!nskb) > + return -ENOMEM; > + > + err = dev_map_generic_redirect(dst, nskb, xdp_prog); > + if (unlikely(err)) > + return err; > + } > + > +find_next: > + err = map->ops->map_get_next_key(map, &key, &next_key); > + if (err) > + break; > + > + key = next_key; > + } > + > + consume_skb(skb); > + return 0; > +} > + > static int xdp_do_generic_redirect_map(struct net_device *dev, > struct sk_buff *skb, > struct xdp_buff *xdp, > @@ -3573,19 +3625,29 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, > struct bpf_map *map) > { > struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); > + bool exclude_ingress = !!(ri->flags & BPF_F_EXCLUDE_INGRESS); > + struct bpf_map *ex_map = ri->ex_map; > u32 index = ri->tgt_index; > void *fwd = ri->tgt_value; > int err = 0; > > ri->tgt_index = 0; > ri->tgt_value = NULL; > + ri->ex_map = NULL; > WRITE_ONCE(ri->map, NULL); > > if (map->map_type == BPF_MAP_TYPE_DEVMAP || > map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { > - struct bpf_dtab_netdev *dst = fwd; Same as above - please add a comment explaining this test... > + if (fwd) { > + struct bpf_dtab_netdev *dst = fwd; > + > + err = dev_map_generic_redirect(dst, skb, xdp_prog); > + } else { > + /* Deal with multicast maps */ > + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, > + ex_map, exclude_ingress); > + } > > - err = dev_map_generic_redirect(dst, skb, xdp_prog); > if (unlikely(err)) > goto err; > } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { > @@ -3699,6 +3761,33 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { > .arg3_type = ARG_ANYTHING, > }; > > +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, > + struct bpf_map *, ex_map, u64, flags) > +{ > + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); > + > + if (unlikely(!map || flags > BPF_F_EXCLUDE_INGRESS)) > + return XDP_ABORTED; > + > + ri->tgt_index = 0; > + ri->tgt_value = NULL; > + ri->flags = flags; > + ri->ex_map = ex_map; > + > + WRITE_ONCE(ri->map, map); > + > + return XDP_REDIRECT; > +} > + > +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { > + .func = bpf_xdp_redirect_map_multi, > + .gpl_only = false, > + .ret_type = RET_INTEGER, > + .arg1_type = ARG_CONST_MAP_PTR, > + .arg2_type = ARG_CONST_MAP_PTR, > + .arg3_type = ARG_ANYTHING, > +}; > + > static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, > unsigned long off, unsigned long len) > { > @@ -6363,6 +6452,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) > return &bpf_xdp_redirect_proto; > case BPF_FUNC_redirect_map: > return &bpf_xdp_redirect_map_proto; > + case BPF_FUNC_redirect_map_multi: > + return &bpf_xdp_redirect_map_multi_proto; > case BPF_FUNC_xdp_adjust_tail: > return &bpf_xdp_adjust_tail_proto; > case BPF_FUNC_fib_lookup: > diff --git a/net/core/xdp.c b/net/core/xdp.c > index 90f44f382115..acdc63833b1f 100644 > --- a/net/core/xdp.c > +++ b/net/core/xdp.c > @@ -475,3 +475,29 @@ void xdp_warn(const char *msg, const char *func, const int line) > WARN(1, "XDP_WARN: %s(line:%d): %s\n", func, line, msg); > }; > EXPORT_SYMBOL_GPL(xdp_warn); > + > +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) > +{ > + unsigned int headroom, totalsize; > + struct xdp_frame *nxdpf; > + struct page *page; > + void *addr; > + > + headroom = xdpf->headroom + sizeof(*xdpf); > + totalsize = headroom + xdpf->len; > + > + if (unlikely(totalsize > PAGE_SIZE)) > + return NULL; > + page = dev_alloc_page(); > + if (!page) > + return NULL; > + addr = page_to_virt(page); > + > + memcpy(addr, xdpf, totalsize); > + > + nxdpf = addr; > + nxdpf->data = addr + headroom; > + > + return nxdpf; > +} > +EXPORT_SYMBOL_GPL(xdpf_clone); > diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h > index 97e1fd19ff58..000b0cf961ea 100644 > --- a/tools/include/uapi/linux/bpf.h > +++ b/tools/include/uapi/linux/bpf.h > @@ -3157,6 +3157,20 @@ union bpf_attr { > * **bpf_sk_cgroup_id**\ (). > * Return > * The id is returned or 0 in case the id could not be retrieved. > + * > + * int bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) > + * Description > + * Redirect the packet to ALL the interfaces in *map*, but > + * exclude the interfaces in *ex_map* (which may be NULL). > + * > + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, > + * which additionally excludes the current ingress device. > + * > + * See also bpf_redirect_map(), which supports redirecting > + * packet to a specific ifindex in the map. > + * Return > + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. > + * > */ > #define __BPF_FUNC_MAPPER(FN) \ > FN(unspec), \ > @@ -3288,7 +3302,8 @@ union bpf_attr { > FN(seq_printf), \ > FN(seq_write), \ > FN(sk_cgroup_id), \ > - FN(sk_ancestor_cgroup_id), > + FN(sk_ancestor_cgroup_id), \ > + FN(redirect_map_multi), > > /* integer value in 'imm' field of BPF_CALL instruction selects which helper > * function eBPF program intends to call > @@ -3417,6 +3432,11 @@ enum bpf_lwt_encap_mode { > BPF_LWT_ENCAP_IP, > }; > > +/* BPF_FUNC_redirect_map_multi flags. */ > +enum { > + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), > +}; > + > #define __bpf_md_ptr(type, name) \ > union { \ > type name; \ > -- > 2.25.4 ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-05-26 14:05 ` [PATCHv4 bpf-next 1/2] xdp: add a new helper for " Hangbin Liu 2020-05-27 10:29 ` Toke Høiland-Jørgensen @ 2020-06-10 10:18 ` Jesper Dangaard Brouer 2020-06-12 8:54 ` Hangbin Liu 2020-06-10 10:21 ` Jesper Dangaard Brouer 2 siblings, 1 reply; 219+ messages in thread From: Jesper Dangaard Brouer @ 2020-06-10 10:18 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, brouer On Tue, 26 May 2020 22:05:38 +0800 Hangbin Liu <liuhangbin@gmail.com> wrote: > diff --git a/net/core/xdp.c b/net/core/xdp.c > index 90f44f382115..acdc63833b1f 100644 > --- a/net/core/xdp.c > +++ b/net/core/xdp.c > @@ -475,3 +475,29 @@ void xdp_warn(const char *msg, const char *func, const int line) > WARN(1, "XDP_WARN: %s(line:%d): %s\n", func, line, msg); > }; > EXPORT_SYMBOL_GPL(xdp_warn); > + > +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) > +{ > + unsigned int headroom, totalsize; > + struct xdp_frame *nxdpf; > + struct page *page; > + void *addr; > + > + headroom = xdpf->headroom + sizeof(*xdpf); > + totalsize = headroom + xdpf->len; > + > + if (unlikely(totalsize > PAGE_SIZE)) > + return NULL; > + page = dev_alloc_page(); > + if (!page) > + return NULL; > + addr = page_to_virt(page); > + > + memcpy(addr, xdpf, totalsize); I don't think this will work. You are assuming that the memory model (xdp_mem_info) is the same. You happened to use i40, that have MEM_TYPE_PAGE_SHARED, and you should have changed this to MEM_TYPE_PAGE_ORDER0, but it doesn't crash as they are compatible. If you were using mlx5, I suspect that this would result in memory leaking. You also need to update xdpf->frame_sz, as you also cannot assume it is the same. > + > + nxdpf = addr; > + nxdpf->data = addr + headroom; > + > + return nxdpf; > +} > +EXPORT_SYMBOL_GPL(xdpf_clone); -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer struct xdp_frame { void *data; u16 len; u16 headroom; u32 metasize:8; u32 frame_sz:24; /* Lifetime of xdp_rxq_info is limited to NAPI/enqueue time, * while mem info is valid on remote CPU. */ struct xdp_mem_info mem; struct net_device *dev_rx; /* used by cpumap */ }; ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-06-10 10:18 ` Jesper Dangaard Brouer @ 2020-06-12 8:54 ` Hangbin Liu 2020-06-16 8:55 ` Jesper Dangaard Brouer 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-06-12 8:54 UTC (permalink / raw) To: Jesper Dangaard Brouer Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On Wed, Jun 10, 2020 at 12:18:59PM +0200, Jesper Dangaard Brouer wrote: > On Tue, 26 May 2020 22:05:38 +0800 > Hangbin Liu <liuhangbin@gmail.com> wrote: > > > diff --git a/net/core/xdp.c b/net/core/xdp.c > > index 90f44f382115..acdc63833b1f 100644 > > --- a/net/core/xdp.c > > +++ b/net/core/xdp.c > > @@ -475,3 +475,29 @@ void xdp_warn(const char *msg, const char *func, const int line) > > WARN(1, "XDP_WARN: %s(line:%d): %s\n", func, line, msg); > > }; > > EXPORT_SYMBOL_GPL(xdp_warn); > > + > > +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) > > +{ > > + unsigned int headroom, totalsize; > > + struct xdp_frame *nxdpf; > > + struct page *page; > > + void *addr; > > + > > + headroom = xdpf->headroom + sizeof(*xdpf); > > + totalsize = headroom + xdpf->len; > > + > > + if (unlikely(totalsize > PAGE_SIZE)) > > + return NULL; > > + page = dev_alloc_page(); > > + if (!page) > > + return NULL; > > + addr = page_to_virt(page); > > + > > + memcpy(addr, xdpf, totalsize); > > I don't think this will work. You are assuming that the memory model > (xdp_mem_info) is the same. > > You happened to use i40, that have MEM_TYPE_PAGE_SHARED, and you should > have changed this to MEM_TYPE_PAGE_ORDER0, but it doesn't crash as they > are compatible. If you were using mlx5, I suspect that this would > result in memory leaking. Is there anything else I should do except add the following line? nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0; > > You also need to update xdpf->frame_sz, as you also cannot assume it is > the same. Won't the memcpy() copy xdpf->frame_sz to nxdpf? And I didn't see xdpf->frame_sz is set in xdp_convert_zc_to_xdp_frame(), do we need a fix? Thanks Hangbin > > > + > > + nxdpf = addr; > > + nxdpf->data = addr + headroom; > > + > > + return nxdpf; > > +} > > +EXPORT_SYMBOL_GPL(xdpf_clone); > > > -- > Best regards, > Jesper Dangaard Brouer > MSc.CS, Principal Kernel Engineer at Red Hat > LinkedIn: http://www.linkedin.com/in/brouer > > > struct xdp_frame { > void *data; > u16 len; > u16 headroom; > u32 metasize:8; > u32 frame_sz:24; > /* Lifetime of xdp_rxq_info is limited to NAPI/enqueue time, > * while mem info is valid on remote CPU. > */ > struct xdp_mem_info mem; > struct net_device *dev_rx; /* used by cpumap */ > }; > ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-06-12 8:54 ` Hangbin Liu @ 2020-06-16 8:55 ` Jesper Dangaard Brouer 2020-06-16 10:11 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: Jesper Dangaard Brouer @ 2020-06-16 8:55 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, brouer On Fri, 12 Jun 2020 16:54:08 +0800 Hangbin Liu <liuhangbin@gmail.com> wrote: > On Wed, Jun 10, 2020 at 12:18:59PM +0200, Jesper Dangaard Brouer wrote: > > On Tue, 26 May 2020 22:05:38 +0800 > > Hangbin Liu <liuhangbin@gmail.com> wrote: > > > > > diff --git a/net/core/xdp.c b/net/core/xdp.c > > > index 90f44f382115..acdc63833b1f 100644 > > > --- a/net/core/xdp.c > > > +++ b/net/core/xdp.c > > > @@ -475,3 +475,29 @@ void xdp_warn(const char *msg, const char *func, const int line) > > > WARN(1, "XDP_WARN: %s(line:%d): %s\n", func, line, msg); > > > }; > > > EXPORT_SYMBOL_GPL(xdp_warn); > > > + > > > +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) > > > +{ > > > + unsigned int headroom, totalsize; > > > + struct xdp_frame *nxdpf; > > > + struct page *page; > > > + void *addr; > > > + > > > + headroom = xdpf->headroom + sizeof(*xdpf); > > > + totalsize = headroom + xdpf->len; > > > + > > > + if (unlikely(totalsize > PAGE_SIZE)) > > > + return NULL; > > > + page = dev_alloc_page(); > > > + if (!page) > > > + return NULL; > > > + addr = page_to_virt(page); > > > + > > > + memcpy(addr, xdpf, totalsize); > > > > I don't think this will work. You are assuming that the memory model > > (xdp_mem_info) is the same. > > > > You happened to use i40, that have MEM_TYPE_PAGE_SHARED, and you should > > have changed this to MEM_TYPE_PAGE_ORDER0, but it doesn't crash as they > > are compatible. If you were using mlx5, I suspect that this would > > result in memory leaking. > > Is there anything else I should do except add the following line? > nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0; You do realize that you also have copied over the mem.id, right? And as I wrote below you also need to update frame_sz. > > > > You also need to update xdpf->frame_sz, as you also cannot assume it is > > the same. > > Won't the memcpy() copy xdpf->frame_sz to nxdpf? You obviously cannot use the frame_sz from the existing frame, as you just allocated a new page for the new xdp_frame, that have another size (here PAGE_SIZE). > And I didn't see xdpf->frame_sz is set in xdp_convert_zc_to_xdp_frame(), > do we need a fix? Good catch, that sounds like a bug, that should be fixed. Will you send a fix? > > > + > > > + nxdpf = addr; > > > + nxdpf->data = addr + headroom; > > > + > > > + return nxdpf; > > > +} > > > +EXPORT_SYMBOL_GPL(xdpf_clone); > > > > > > struct xdp_frame { > > void *data; > > u16 len; > > u16 headroom; > > u32 metasize:8; > > u32 frame_sz:24; > > /* Lifetime of xdp_rxq_info is limited to NAPI/enqueue time, > > * while mem info is valid on remote CPU. > > */ > > struct xdp_mem_info mem; > > struct net_device *dev_rx; /* used by cpumap */ > > }; > > > struct xdp_mem_info { u32 type; /* 0 4 */ u32 id; /* 4 4 */ /* size: 8, cachelines: 1, members: 2 */ /* last cacheline: 8 bytes */ }; -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-06-16 8:55 ` Jesper Dangaard Brouer @ 2020-06-16 10:11 ` Hangbin Liu 2020-06-16 14:38 ` Jesper Dangaard Brouer 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-06-16 10:11 UTC (permalink / raw) To: Jesper Dangaard Brouer Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi HI Jesper, On Tue, Jun 16, 2020 at 10:55:06AM +0200, Jesper Dangaard Brouer wrote: > > Is there anything else I should do except add the following line? > > nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0; > > You do realize that you also have copied over the mem.id, right? Thanks for the reminding. To confirm, set mem.id to 0 is enough, right? > > And as I wrote below you also need to update frame_sz. > > > > > > > You also need to update xdpf->frame_sz, as you also cannot assume it is > > > the same. > > > > Won't the memcpy() copy xdpf->frame_sz to nxdpf? > > You obviously cannot use the frame_sz from the existing frame, as you > just allocated a new page for the new xdp_frame, that have another size > (here PAGE_SIZE). Thanks, I didn't understand the frame_sz correctly before. > > > > And I didn't see xdpf->frame_sz is set in xdp_convert_zc_to_xdp_frame(), > > do we need a fix? > > Good catch, that sounds like a bug, that should be fixed. > Will you send a fix? OK, I will. > > > > > > + > > > > + nxdpf = addr; > > > > + nxdpf->data = addr + headroom; > > > > + > > > > + return nxdpf; > > > > +} > > > > +EXPORT_SYMBOL_GPL(xdpf_clone); > > > > > > > > > struct xdp_frame { > > > void *data; > > > u16 len; > > > u16 headroom; > > > u32 metasize:8; > > > u32 frame_sz:24; > > > /* Lifetime of xdp_rxq_info is limited to NAPI/enqueue time, > > > * while mem info is valid on remote CPU. > > > */ > > > struct xdp_mem_info mem; > > > struct net_device *dev_rx; /* used by cpumap */ > > > }; > > > > > > > struct xdp_mem_info { > u32 type; /* 0 4 */ > u32 id; /* 4 4 */ > > /* size: 8, cachelines: 1, members: 2 */ > /* last cacheline: 8 bytes */ > }; > Is this a struct reference or you want to remind me something else? Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-06-16 10:11 ` Hangbin Liu @ 2020-06-16 14:38 ` Jesper Dangaard Brouer 0 siblings, 0 replies; 219+ messages in thread From: Jesper Dangaard Brouer @ 2020-06-16 14:38 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, brouer On Tue, 16 Jun 2020 18:11:33 +0800 Hangbin Liu <liuhangbin@gmail.com> wrote: > HI Jesper, > > On Tue, Jun 16, 2020 at 10:55:06AM +0200, Jesper Dangaard Brouer wrote: > > > Is there anything else I should do except add the following line? > > > nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0; > > > > You do realize that you also have copied over the mem.id, right? > > Thanks for the reminding. To confirm, set mem.id to 0 is enough, right? Yes. > > And as I wrote below you also need to update frame_sz. > > > > > > > > > > You also need to update xdpf->frame_sz, as you also cannot assume it is > > > > the same. > > > > > > Won't the memcpy() copy xdpf->frame_sz to nxdpf? > > > > You obviously cannot use the frame_sz from the existing frame, as you > > just allocated a new page for the new xdp_frame, that have another size > > (here PAGE_SIZE). > > Thanks, I didn't understand the frame_sz correctly before. > > > > > > > And I didn't see xdpf->frame_sz is set in xdp_convert_zc_to_xdp_frame(), > > > do we need a fix? > > > > Good catch, that sounds like a bug, that should be fixed. > > Will you send a fix? > > OK, I will. Thanks. > > > > > > > + > > > > > + nxdpf = addr; > > > > > + nxdpf->data = addr + headroom; > > > > > + > > > > > + return nxdpf; > > > > > +} > > > > > +EXPORT_SYMBOL_GPL(xdpf_clone); > > > > > > > > > > > > struct xdp_frame { > > > > void *data; > > > > u16 len; > > > > u16 headroom; > > > > u32 metasize:8; > > > > u32 frame_sz:24; > > > > /* Lifetime of xdp_rxq_info is limited to NAPI/enqueue time, > > > > * while mem info is valid on remote CPU. > > > > */ > > > > struct xdp_mem_info mem; > > > > struct net_device *dev_rx; /* used by cpumap */ > > > > }; > > > > > > > > > > > struct xdp_mem_info { > > u32 type; /* 0 4 */ > > u32 id; /* 4 4 */ > > > > /* size: 8, cachelines: 1, members: 2 */ > > /* last cacheline: 8 bytes */ > > }; > > > > Is this a struct reference or you want to remind me something else? This is just a struct reference to help the readers of this email. I had to lookup the struct to review this code, so I included it to save time for other reviewers. -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-05-26 14:05 ` [PATCHv4 bpf-next 1/2] xdp: add a new helper for " Hangbin Liu 2020-05-27 10:29 ` Toke Høiland-Jørgensen 2020-06-10 10:18 ` Jesper Dangaard Brouer @ 2020-06-10 10:21 ` Jesper Dangaard Brouer 2020-06-10 10:29 ` Toke Høiland-Jørgensen 2 siblings, 1 reply; 219+ messages in thread From: Jesper Dangaard Brouer @ 2020-06-10 10:21 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, brouer On Tue, 26 May 2020 22:05:38 +0800 Hangbin Liu <liuhangbin@gmail.com> wrote: > diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c > index a51d9fb7a359..ecc5c44a5bab 100644 > --- a/kernel/bpf/devmap.c > +++ b/kernel/bpf/devmap.c [...] > +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, > + struct bpf_map *map, struct bpf_map *ex_map, > + bool exclude_ingress) > +{ > + struct bpf_dtab_netdev *obj = NULL; > + struct xdp_frame *xdpf, *nxdpf; > + struct net_device *dev; > + bool first = true; > + u32 key, next_key; > + int err; > + > + devmap_get_next_key(map, NULL, &key); > + > + xdpf = convert_to_xdp_frame(xdp); > + if (unlikely(!xdpf)) > + return -EOVERFLOW; > + > + for (;;) { > + switch (map->map_type) { > + case BPF_MAP_TYPE_DEVMAP: > + obj = __dev_map_lookup_elem(map, key); > + break; > + case BPF_MAP_TYPE_DEVMAP_HASH: > + obj = __dev_map_hash_lookup_elem(map, key); > + break; > + default: > + break; > + } > + > + if (!obj || dev_in_exclude_map(obj, ex_map, > + exclude_ingress ? dev_rx->ifindex : 0)) > + goto find_next; > + > + dev = obj->dev; > + > + if (!dev->netdev_ops->ndo_xdp_xmit) > + goto find_next; > + > + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); > + if (unlikely(err)) > + goto find_next; > + > + if (!first) { > + nxdpf = xdpf_clone(xdpf); > + if (unlikely(!nxdpf)) > + return -ENOMEM; > + > + bq_enqueue(dev, nxdpf, dev_rx); > + } else { > + bq_enqueue(dev, xdpf, dev_rx); This looks racy. You enqueue the original frame, and then later xdpf_clone it. The original frame might have been freed at that point. > + first = false; > + } > + > +find_next: > + err = devmap_get_next_key(map, &key, &next_key); > + if (err) > + break; > + key = next_key; > + } > + > + /* didn't find anywhere to forward to, free buf */ > + if (first) > + xdp_return_frame_rx_napi(xdpf); > + > + return 0; > +} > + -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-06-10 10:21 ` Jesper Dangaard Brouer @ 2020-06-10 10:29 ` Toke Høiland-Jørgensen 2020-06-16 9:04 ` Jesper Dangaard Brouer 0 siblings, 1 reply; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2020-06-10 10:29 UTC (permalink / raw) To: Jesper Dangaard Brouer, Hangbin Liu Cc: bpf, netdev, Jiri Benc, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, brouer Jesper Dangaard Brouer <brouer@redhat.com> writes: > On Tue, 26 May 2020 22:05:38 +0800 > Hangbin Liu <liuhangbin@gmail.com> wrote: > >> diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c >> index a51d9fb7a359..ecc5c44a5bab 100644 >> --- a/kernel/bpf/devmap.c >> +++ b/kernel/bpf/devmap.c > [...] > >> +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, >> + struct bpf_map *map, struct bpf_map *ex_map, >> + bool exclude_ingress) >> +{ >> + struct bpf_dtab_netdev *obj = NULL; >> + struct xdp_frame *xdpf, *nxdpf; >> + struct net_device *dev; >> + bool first = true; >> + u32 key, next_key; >> + int err; >> + >> + devmap_get_next_key(map, NULL, &key); >> + >> + xdpf = convert_to_xdp_frame(xdp); >> + if (unlikely(!xdpf)) >> + return -EOVERFLOW; >> + >> + for (;;) { >> + switch (map->map_type) { >> + case BPF_MAP_TYPE_DEVMAP: >> + obj = __dev_map_lookup_elem(map, key); >> + break; >> + case BPF_MAP_TYPE_DEVMAP_HASH: >> + obj = __dev_map_hash_lookup_elem(map, key); >> + break; >> + default: >> + break; >> + } >> + >> + if (!obj || dev_in_exclude_map(obj, ex_map, >> + exclude_ingress ? dev_rx->ifindex : 0)) >> + goto find_next; >> + >> + dev = obj->dev; >> + >> + if (!dev->netdev_ops->ndo_xdp_xmit) >> + goto find_next; >> + >> + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); >> + if (unlikely(err)) >> + goto find_next; >> + >> + if (!first) { >> + nxdpf = xdpf_clone(xdpf); >> + if (unlikely(!nxdpf)) >> + return -ENOMEM; >> + >> + bq_enqueue(dev, nxdpf, dev_rx); >> + } else { >> + bq_enqueue(dev, xdpf, dev_rx); > > This looks racy. You enqueue the original frame, and then later > xdpf_clone it. The original frame might have been freed at that > point. This was actually my suggestion; on the assumption that bq_enqueue() just puts the frame on a list that won't be flushed until we exit the NAPI loop. But I guess now that you mention it that bq_enqueue() may flush the queue, so you're right that this won't work. Sorry about that, Hangbin :/ Jesper, the reason I suggested this was to avoid an "extra" copy (i.e., if we have two destinations, ideally we should only clone once instead of twice). Got any clever ideas for a safe way to achieve this? :) -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 1/2] xdp: add a new helper for dev map multicast support 2020-06-10 10:29 ` Toke Høiland-Jørgensen @ 2020-06-16 9:04 ` Jesper Dangaard Brouer 0 siblings, 0 replies; 219+ messages in thread From: Jesper Dangaard Brouer @ 2020-06-16 9:04 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: Hangbin Liu, bpf, netdev, Jiri Benc, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, brouer On Wed, 10 Jun 2020 12:29:35 +0200 Toke Høiland-Jørgensen <toke@redhat.com> wrote: > Jesper Dangaard Brouer <brouer@redhat.com> writes: > > > On Tue, 26 May 2020 22:05:38 +0800 > > Hangbin Liu <liuhangbin@gmail.com> wrote: > > > >> diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c > >> index a51d9fb7a359..ecc5c44a5bab 100644 > >> --- a/kernel/bpf/devmap.c > >> +++ b/kernel/bpf/devmap.c > > [...] > > > >> +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, > >> + struct bpf_map *map, struct bpf_map *ex_map, > >> + bool exclude_ingress) > >> +{ [...] > >> + if (!first) { > >> + nxdpf = xdpf_clone(xdpf); > >> + if (unlikely(!nxdpf)) > >> + return -ENOMEM; > >> + > >> + bq_enqueue(dev, nxdpf, dev_rx); > >> + } else { > >> + bq_enqueue(dev, xdpf, dev_rx); > > > > This looks racy. You enqueue the original frame, and then later > > xdpf_clone it. The original frame might have been freed at that > > point. > > This was actually my suggestion; on the assumption that bq_enqueue() > just puts the frame on a list that won't be flushed until we exit the > NAPI loop. > > But I guess now that you mention it that bq_enqueue() may flush the > queue, so you're right that this won't work. Sorry about that, Hangbin :/ > > Jesper, the reason I suggested this was to avoid an "extra" copy (i.e., > if we have two destinations, ideally we should only clone once instead > of twice). Got any clever ideas for a safe way to achieve this? :) Maybe you/we could avoid the clone on the last destination? -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv4 bpf-next 2/2] sample/bpf: add xdp_redirect_map_multicast test 2020-05-26 14:05 ` [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu 2020-05-26 14:05 ` [PATCHv4 bpf-next 1/2] xdp: add a new helper for " Hangbin Liu @ 2020-05-26 14:05 ` Hangbin Liu 2020-05-27 10:21 ` [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support Toke Høiland-Jørgensen 2020-07-01 4:19 ` [PATCHv5 bpf-next 0/3] xdp: add a new helper for " Hangbin Liu 3 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-05-26 14:05 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu This is a sample for xdp multicast. In the sample we have 3 forward groups and 1 exclude group. It will redirect each interface's packets to all the interfaces in the forward group, and exclude the interface in exclude map. For more testing details, please see the test description in xdp_redirect_map_multi.sh. v4: no update. v3: add rxcnt map to show the packet transmit speed. v2: no update. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi.sh | 135 +++++++++++++++ samples/bpf/xdp_redirect_map_multi_kern.c | 113 +++++++++++++ samples/bpf/xdp_redirect_map_multi_user.c | 197 ++++++++++++++++++++++ 4 files changed, 448 insertions(+) create mode 100755 samples/bpf/xdp_redirect_map_multi.sh create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 8403e4762306..000709bb89c3 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map tprogs-y += per_socket_stats_example tprogs-y += xdp_redirect tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_map_multi tprogs-y += xdp_redirect_cpu tprogs-y += xdp_monitor tprogs-y += xdp_rxq_info @@ -97,6 +98,7 @@ test_map_in_map-objs := bpf_load.o test_map_in_map_user.o per_socket_stats_example-objs := cookie_uid_helper_example.o xdp_redirect-objs := xdp_redirect_user.o xdp_redirect_map-objs := xdp_redirect_map_user.o +xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o xdp_redirect_cpu-objs := bpf_load.o xdp_redirect_cpu_user.o xdp_monitor-objs := bpf_load.o xdp_monitor_user.o xdp_rxq_info-objs := xdp_rxq_info_user.o @@ -156,6 +158,7 @@ always-y += tcp_tos_reflect_kern.o always-y += tcp_dumpstats_kern.o always-y += xdp_redirect_kern.o always-y += xdp_redirect_map_kern.o +always-y += xdp_redirect_map_multi_kern.o always-y += xdp_redirect_cpu_kern.o always-y += xdp_monitor_kern.o always-y += xdp_rxq_info_kern.o diff --git a/samples/bpf/xdp_redirect_map_multi.sh b/samples/bpf/xdp_redirect_map_multi.sh new file mode 100755 index 000000000000..bbf10ca06720 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi.sh @@ -0,0 +1,135 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Test topology: +# - - - - - - - - - - - - - - - - - - - - - - - - - +# | veth1 veth2 veth3 veth4 | ... init net +# - -| - - - - - - | - - - - - - | - - - - - - | - - +# --------- --------- --------- --------- +# | veth0 | | veth0 | | veth0 | | veth0 | ... +# --------- --------- --------- --------- +# ns1 ns2 ns3 ns4 +# +# Forward multicast groups: +# Forward group all has interfaces: veth1, veth2, veth3, veth4, ... (All traffic except IPv4, IPv6) +# Forward group v4 has interfaces: veth1, veth3, veth4, ... (For IPv4 traffic only) +# Forward group v6 has interfaces: veth2, veth3, veth4, ... (For IPv6 traffic only) +# Exclude Groups: +# Exclude group: veth3 (assume ns3 is in black list) +# +# Test modules: +# XDP modes: generic, native +# map types: group v4 use DEVMAP, others use DEVMAP_HASH +# +# Test cases: +# ARP(we didn't exclude ns3 in kern.c for ARP): +# ns1 -> gw: ns2, ns3, ns4 should receive the arp request +# IPv4: +# ns1 -> ns2 (fail), ns1 -> ns3 (fail), ns1 -> ns4 (pass) +# IPv6 +# ns2 -> ns1 (fail), ns2 -> ns3 (fail), ns2 -> ns4 (pass) +# + + +# netns numbers +NUM=10 +IFACES="" +DRV_MODE="generic drv" + +test_pass() +{ + echo "Pass: $@" +} + +test_fail() +{ + echo "fail: $@" +} + +clean_up() +{ + for i in $(seq $NUM); do + ip netns del ns$i + done +} + +setup_ns() +{ + local mode=$1 + + for i in $(seq $NUM); do + ip netns add ns$i + ip link add veth0 type veth peer name veth$i + ip link set veth0 netns ns$i + ip -n ns$i link set veth0 up + ip link set veth$i up + + ip -n ns$i addr add 192.0.2.$i/24 dev veth0 + ip -n ns$i addr add 2001:db8::$i/24 dev veth0 + ip -n ns$i link set veth0 xdp$mode obj \ + xdp_redirect_map_multi_kern.o sec xdp_redirect_dummy &> /dev/null || \ + { test_fail "Unable to load dummy xdp" && exit 1; } + IFACES="$IFACES veth$i" + done +} + +do_ping_tests() +{ + local drv_mode=$1 + + # arp test + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns1-2_${drv_mode}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns1-3_${drv_mode}.log & + ip netns exec ns4 tcpdump -i veth0 -nn -l -e &> arp_ns1-4_${drv_mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-2_${drv_mode}.log && \ + test_pass "$drv_mode arp ns1-2" || test_fail "$drv_mode arp ns1-2" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-3_${drv_mode}.log && \ + test_pass "$drv_mode arp ns1-3" || test_fail "$drv_mode arp ns1-3" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-4_${drv_mode}.log && \ + test_pass "$drv_mode arp ns1-4" || test_fail "$drv_mode arp ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_fail "$drv_mode ping ns1-2" || test_pass "$drv_mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_fail "$drv_mode ping ns1-3" || test_pass "$drv_mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$drv_mode ping ns1-4" || test_fail "$drv_mode ping ns1-4" + + # ping6 test + ip netns exec ns2 ping6 2001:db8::1 -c 4 &> /dev/null && \ + test_fail "$drv_mode ping6 ns2-1" || test_pass "$drv_mode ping6 ns2-1" + ip netns exec ns2 ping6 2001:db8::3 -c 4 &> /dev/null && \ + test_fail "$drv_mode ping6 ns2-3" || test_pass "$drv_mode ping6 ns2-3" + ip netns exec ns2 ping6 2001:db8::4 -c 4 &> /dev/null && \ + test_pass "$drv_mode ping6 ns2-4" || test_fail "$drv_mode ping6 ns2-4" +} + +do_tests() +{ + local drv_mode=$1 + local drv_p + + [ ${drv_mode} == "drv" ] && drv_p="-N" || drv_p="-S" + + # run `ulimit -l unlimited` if you got errors like + # libbpf: Error in bpf_object__probe_global_data():Operation not permitted(1). + ./xdp_redirect_map_multi $drv_p $IFACES &> xdp_${drv_mode}.log & + xdp_pid=$! + sleep 10 + + do_ping_tests $drv_mode + + kill $xdp_pid +} + +for mode in ${DRV_MODE}; do + setup_ns $mode + do_tests $mode + sleep 10 + clean_up + sleep 5 +done diff --git a/samples/bpf/xdp_redirect_map_multi_kern.c b/samples/bpf/xdp_redirect_map_multi_kern.c new file mode 100644 index 000000000000..81f71461a252 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_kern.c @@ -0,0 +1,113 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <uapi/linux/bpf.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ip.h> +#include <linux/ipv6.h> +#include <bpf/bpf_helpers.h> + +/* In this sample we will use 3 forward maps and 1 exclude map to + * show how to use the helper bpf_redirect_map_multi(). + * + * In real world, there may have multi forward maps and exclude map. You can + * use map-in-map type to store the forward and exlude maps. e.g. + * forward_map_in_map[group_a_index] = forward_group_a_map + * forward_map_in_map[group_b_index] = forward_group_b_map + * exclude_map_in_map[iface_1_index] = iface_1_exclude_map + * exclude_map_in_map[iface_2_index] = iface_2_exclude_map + * Then store the forward group indexes based on IP/MAC policy in another + * hash map, e.g.: + * mcast_route_map[hash(subnet_a)] = group_a_index + * mcast_route_map[hash(subnet_b)] = group_b_index + * + * You can init the maps in user.c, and find the forward group index from + * mcast_route_map bye key hash(subnet) in kern.c, Then you could find + * the forward group by the group index. You can also get the exclude map + * simply by iface index in exclude_map_in_map. + */ +struct bpf_map_def SEC("maps") forward_map_v4 = { + .type = BPF_MAP_TYPE_DEVMAP, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 4096, +}; + +struct bpf_map_def SEC("maps") forward_map_v6 = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") forward_map_all = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") exclude_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") rxcnt = { + .type = BPF_MAP_TYPE_PERCPU_ARRAY, + .key_size = sizeof(u32), + .value_size = sizeof(long), + .max_entries = 1, +}; + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + struct ethhdr *eth = data; + long *value; + u16 h_proto; + u32 key = 0; + u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + h_proto = eth->h_proto; + + /* count packet in global counter */ + value = bpf_map_lookup_elem(&rxcnt, &key); + if (value) + *value += 1; + + if (h_proto == htons(ETH_P_IP)) + return bpf_redirect_map_multi(&forward_map_v4, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else if (h_proto == htons(ETH_P_IPV6)) + return bpf_redirect_map_multi(&forward_map_v6, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else + return bpf_redirect_map_multi(&forward_map_all, NULL, + BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_redirect_dummy") +int xdp_redirect_dummy_prog(struct xdp_md *ctx) +{ + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/samples/bpf/xdp_redirect_map_multi_user.c b/samples/bpf/xdp_redirect_map_multi_user.c new file mode 100644 index 000000000000..7339ce4c7f9c --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_user.c @@ -0,0 +1,197 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static int ifaces[MAX_IFACE_NUM] = {}; +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int rxcnt; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void poll_stats(int interval) +{ + unsigned int nr_cpus = bpf_num_possible_cpus(); + __u64 values[nr_cpus], prev[nr_cpus]; + + memset(prev, 0, sizeof(prev)); + + while (1) { + __u64 sum = 0; + __u32 key = 0; + int i; + + sleep(interval); + assert(bpf_map_lookup_elem(rxcnt, &key, values) == 0); + for (i = 0; i < nr_cpus; i++) + sum += (values[i] - prev[i]); + if (sum) + printf("Forwarding %10llu pkt/s\n", sum / interval); + memcpy(prev, values, sizeof(values)); + } +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n", + prog); +} + +int main(int argc, char **argv) +{ + int prog_fd, group_all, group_v4, group_v6, exclude; + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + int i, ret, opt, ifindex; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNF")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) + xdp_flags |= XDP_FLAGS_DRV_MODE; + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + return 1; + + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_all"); + group_v4 = bpf_object__find_map_fd_by_name(obj, "forward_map_v4"); + group_v6 = bpf_object__find_map_fd_by_name(obj, "forward_map_v6"); + exclude = bpf_object__find_map_fd_by_name(obj, "exclude_map"); + rxcnt = bpf_object__find_map_fd_by_name(obj, "rxcnt"); + + if (group_all < 0 || group_v4 < 0 || group_v6 < 0 || exclude < 0 || + rxcnt < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + return 1; + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* Add all the interfaces to group all */ + ret = bpf_map_update_elem(group_all, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* For testing: remove the 2nd interfaces from group v4 */ + if (i != 1) { + ret = bpf_map_update_elem(group_v4, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: remove the 1st interfaces from group v6 */ + if (i != 0) { + ret = bpf_map_update_elem(group_v6, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: add the 3rd interfaces to exclude map */ + if (i == 2) { + ret = bpf_map_update_elem(exclude, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + } + + poll_stats(2); + + return 0; + +err_out: + return 1; +} -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-05-26 14:05 ` [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu 2020-05-26 14:05 ` [PATCHv4 bpf-next 1/2] xdp: add a new helper for " Hangbin Liu 2020-05-26 14:05 ` [PATCHv4 bpf-next 2/2] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu @ 2020-05-27 10:21 ` Toke Høiland-Jørgensen 2020-05-27 10:32 ` Eelco Chaudron ` (2 more replies) 2020-07-01 4:19 ` [PATCHv5 bpf-next 0/3] xdp: add a new helper for " Hangbin Liu 3 siblings, 3 replies; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2020-05-27 10:21 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu Hangbin Liu <liuhangbin@gmail.com> writes: > Hi all, > > This patchset is for xdp multicast support, which has been discussed > before[0]. The goal is to be able to implement an OVS-like data plane in > XDP, i.e., a software switch that can forward XDP frames to multiple > ports. > > To achieve this, an application needs to specify a group of interfaces > to forward a packet to. It is also common to want to exclude one or more > physical interfaces from the forwarding operation - e.g., to forward a > packet to all interfaces in the multicast group except the interface it > arrived on. While this could be done simply by adding more groups, this > quickly leads to a combinatorial explosion in the number of groups an > application has to maintain. > > To avoid the combinatorial explosion, we propose to include the ability > to specify an "exclude group" as part of the forwarding operation. This > needs to be a group (instead of just a single port index), because a > physical interface can be part of a logical grouping, such as a bond > device. > > Thus, the logical forwarding operation becomes a "set difference" > operation, i.e. "forward to all ports in group A that are not also in > group B". This series implements such an operation using device maps to > represent the groups. This means that the XDP program specifies two > device maps, one containing the list of netdevs to redirect to, and the > other containing the exclude list. > > To achieve this, I re-implement a new helper bpf_redirect_map_multi() > to accept two maps, the forwarding map and exclude map. If user > don't want to use exclude map and just want simply stop redirecting back > to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. > > The example in patch 2 is functional, but not a lot of effort > has been made on performance optimisation. I did a simple test(pkt size 64) > with pktgen. Here is the test result with BPF_MAP_TYPE_DEVMAP_HASH > arrays: > > bpf_redirect_map() with 1 ingress, 1 egress: > generic path: ~1600k pps > native path: ~980k pps > > bpf_redirect_map_multi() with 1 ingress, 3 egress: > generic path: ~600k pps > native path: ~480k pps > > bpf_redirect_map_multi() with 1 ingress, 9 egress: > generic path: ~125k pps > native path: ~100k pps > > The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop > the arrays and do clone skb/xdpf. The native path is slower than generic > path as we send skbs by pktgen. So the result looks reasonable. How are you running these tests? Still on virtual devices? We really need results from a physical setup in native mode to assess the impact on the native-XDP fast path. The numbers above don't tell much in this regard. I'd also like to see a before/after patch for straight bpf_redirect_map(), since you're messing with the fast path, and we want to make sure it's not causing a performance regression for regular redirect. Finally, since the overhead seems to be quite substantial: A comparison with a regular network stack bridge might make sense? After all we also want to make sure it's a performance win over that :) -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-05-27 10:21 ` [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support Toke Høiland-Jørgensen @ 2020-05-27 10:32 ` Eelco Chaudron 2020-05-27 12:38 ` Hangbin Liu 2020-06-03 2:40 ` Hangbin Liu 2 siblings, 0 replies; 219+ messages in thread From: Eelco Chaudron @ 2020-05-27 10:32 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: Hangbin Liu, bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, ast, Daniel Borkmann, Lorenzo Bianconi On 27 May 2020, at 12:21, Toke Høiland-Jørgensen wrote: > Hangbin Liu <liuhangbin@gmail.com> writes: > >> Hi all, >> >> This patchset is for xdp multicast support, which has been discussed >> before[0]. The goal is to be able to implement an OVS-like data plane >> in >> XDP, i.e., a software switch that can forward XDP frames to multiple >> ports. >> >> To achieve this, an application needs to specify a group of >> interfaces >> to forward a packet to. It is also common to want to exclude one or >> more >> physical interfaces from the forwarding operation - e.g., to forward >> a >> packet to all interfaces in the multicast group except the interface >> it >> arrived on. While this could be done simply by adding more groups, >> this >> quickly leads to a combinatorial explosion in the number of groups an >> application has to maintain. >> >> To avoid the combinatorial explosion, we propose to include the >> ability >> to specify an "exclude group" as part of the forwarding operation. >> This >> needs to be a group (instead of just a single port index), because a >> physical interface can be part of a logical grouping, such as a bond >> device. >> >> Thus, the logical forwarding operation becomes a "set difference" >> operation, i.e. "forward to all ports in group A that are not also in >> group B". This series implements such an operation using device maps >> to >> represent the groups. This means that the XDP program specifies two >> device maps, one containing the list of netdevs to redirect to, and >> the >> other containing the exclude list. >> >> To achieve this, I re-implement a new helper bpf_redirect_map_multi() >> to accept two maps, the forwarding map and exclude map. If user >> don't want to use exclude map and just want simply stop redirecting >> back >> to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. >> >> The example in patch 2 is functional, but not a lot of effort >> has been made on performance optimisation. I did a simple test(pkt >> size 64) >> with pktgen. Here is the test result with BPF_MAP_TYPE_DEVMAP_HASH >> arrays: >> >> bpf_redirect_map() with 1 ingress, 1 egress: >> generic path: ~1600k pps >> native path: ~980k pps >> >> bpf_redirect_map_multi() with 1 ingress, 3 egress: >> generic path: ~600k pps >> native path: ~480k pps >> >> bpf_redirect_map_multi() with 1 ingress, 9 egress: >> generic path: ~125k pps >> native path: ~100k pps >> >> The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we >> loop >> the arrays and do clone skb/xdpf. The native path is slower than >> generic >> path as we send skbs by pktgen. So the result looks reasonable. > > How are you running these tests? Still on virtual devices? We really > need results from a physical setup in native mode to assess the impact > on the native-XDP fast path. The numbers above don't tell much in this > regard. I'd also like to see a before/after patch for straight > bpf_redirect_map(), since you're messing with the fast path, and we > want > to make sure it's not causing a performance regression for regular > redirect. > > Finally, since the overhead seems to be quite substantial: A > comparison > with a regular network stack bridge might make sense? After all we > also > want to make sure it's a performance win over that :) What about adding a test with only one egress port? So it compares better to bpf_redirect_map(), i.e. “bpf_redirect_map_multi() with 1 ingress, 1 egress”. ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-05-27 10:21 ` [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support Toke Høiland-Jørgensen 2020-05-27 10:32 ` Eelco Chaudron @ 2020-05-27 12:38 ` Hangbin Liu 2020-05-27 15:04 ` Toke Høiland-Jørgensen 2020-06-03 2:40 ` Hangbin Liu 2 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-05-27 12:38 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On Wed, May 27, 2020 at 12:21:54PM +0200, Toke Høiland-Jørgensen wrote: > > The example in patch 2 is functional, but not a lot of effort > > has been made on performance optimisation. I did a simple test(pkt size 64) > > with pktgen. Here is the test result with BPF_MAP_TYPE_DEVMAP_HASH > > arrays: > > > > bpf_redirect_map() with 1 ingress, 1 egress: > > generic path: ~1600k pps > > native path: ~980k pps > > > > bpf_redirect_map_multi() with 1 ingress, 3 egress: > > generic path: ~600k pps > > native path: ~480k pps > > > > bpf_redirect_map_multi() with 1 ingress, 9 egress: > > generic path: ~125k pps > > native path: ~100k pps > > > > The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop > > the arrays and do clone skb/xdpf. The native path is slower than generic > > path as we send skbs by pktgen. So the result looks reasonable. > > How are you running these tests? Still on virtual devices? We really I run it with the test topology in patch 2/2. The test is run on physical machines, but I use veth interface. Do you mean use a physical NIC driver for testing? BTW, when using pktgen, I got an panic because the skb don't have enough header room. The code path looks like do_xdp_generic() - netif_receive_generic_xdp() - skb_headroom(skb) < XDP_PACKET_HEADROOM - pskb_expand_head() - BUG_ON(skb_shared(skb)) So I added a draft patch for pktgen, not sure if it has any influence. index 08e2811b5274..fee17310c178 100644 --- a/net/core/pktgen.c +++ b/net/core/pktgen.c @@ -170,6 +170,7 @@ #include <linux/uaccess.h> #include <asm/dma.h> #include <asm/div64.h> /* do_div */ +#include <linux/bpf.h> #define VERSION "2.75" #define IP_NAME_SZ 32 @@ -2692,7 +2693,7 @@ static void pktgen_finalize_skb(struct pktgen_dev *pkt_dev, struct sk_buff *skb, static struct sk_buff *pktgen_alloc_skb(struct net_device *dev, struct pktgen_dev *pkt_dev) { - unsigned int extralen = LL_RESERVED_SPACE(dev); + unsigned int extralen = LL_RESERVED_SPACE(dev) + XDP_PACKET_HEADROOM; struct sk_buff *skb = NULL; unsigned int size; > need results from a physical setup in native mode to assess the impact > on the native-XDP fast path. The numbers above don't tell much in this > regard. I'd also like to see a before/after patch for straight > bpf_redirect_map(), since you're messing with the fast path, and we want > to make sure it's not causing a performance regression for regular > redirect. OK, I will write a test with 1 ingress + 1 egress for bpf_redirect_map_multi. Just as Eelco said. > > Finally, since the overhead seems to be quite substantial: A comparison > with a regular network stack bridge might make sense? After all we also > want to make sure it's a performance win over that :) OK, Will do it. Thanks Hangbin ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-05-27 12:38 ` Hangbin Liu @ 2020-05-27 15:04 ` Toke Høiland-Jørgensen 2020-06-16 9:09 ` Jesper Dangaard Brouer 0 siblings, 1 reply; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2020-05-27 15:04 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi Hangbin Liu <liuhangbin@gmail.com> writes: > On Wed, May 27, 2020 at 12:21:54PM +0200, Toke Høiland-Jørgensen wrote: >> > The example in patch 2 is functional, but not a lot of effort >> > has been made on performance optimisation. I did a simple test(pkt size 64) >> > with pktgen. Here is the test result with BPF_MAP_TYPE_DEVMAP_HASH >> > arrays: >> > >> > bpf_redirect_map() with 1 ingress, 1 egress: >> > generic path: ~1600k pps >> > native path: ~980k pps >> > >> > bpf_redirect_map_multi() with 1 ingress, 3 egress: >> > generic path: ~600k pps >> > native path: ~480k pps >> > >> > bpf_redirect_map_multi() with 1 ingress, 9 egress: >> > generic path: ~125k pps >> > native path: ~100k pps >> > >> > The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop >> > the arrays and do clone skb/xdpf. The native path is slower than generic >> > path as we send skbs by pktgen. So the result looks reasonable. >> >> How are you running these tests? Still on virtual devices? We really > > I run it with the test topology in patch 2/2. The test is run on physical > machines, but I use veth interface. Do you mean use a physical NIC driver > for testing? Yes, sorry, when I said 'physical machine' I should have also 'physical NIC'. We really need to know how the performance of this is on the XDP fast path, i.e., when there are no skbs involved at all. > BTW, when using pktgen, I got an panic because the skb don't have enough > header room. The code path looks like > > do_xdp_generic() > - netif_receive_generic_xdp() > - skb_headroom(skb) < XDP_PACKET_HEADROOM > - pskb_expand_head() > - BUG_ON(skb_shared(skb)) > > So I added a draft patch for pktgen, not sure if it has any influence. Hmm, as Jesper said pktgen was really not intended to be used this way, so I guess that's why. I guess I'll let him comment on whether he thinks it's worth fixing; or you could send this as a proper patch and see if anyone complains about it ;) -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-05-27 15:04 ` Toke Høiland-Jørgensen @ 2020-06-16 9:09 ` Jesper Dangaard Brouer 2020-06-16 9:47 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: Jesper Dangaard Brouer @ 2020-06-16 9:09 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: Hangbin Liu, bpf, netdev, Jiri Benc, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, brouer On Wed, 27 May 2020 17:04:50 +0200 Toke Høiland-Jørgensen <toke@redhat.com> wrote: > Hangbin Liu <liuhangbin@gmail.com> writes: > > > On Wed, May 27, 2020 at 12:21:54PM +0200, Toke Høiland-Jørgensen wrote: > >> > The example in patch 2 is functional, but not a lot of effort > >> > has been made on performance optimisation. I did a simple test(pkt size 64) > >> > with pktgen. Here is the test result with BPF_MAP_TYPE_DEVMAP_HASH > >> > arrays: > >> > > >> > bpf_redirect_map() with 1 ingress, 1 egress: > >> > generic path: ~1600k pps > >> > native path: ~980k pps > >> > > >> > bpf_redirect_map_multi() with 1 ingress, 3 egress: > >> > generic path: ~600k pps > >> > native path: ~480k pps > >> > > >> > bpf_redirect_map_multi() with 1 ingress, 9 egress: > >> > generic path: ~125k pps > >> > native path: ~100k pps > >> > > >> > The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop > >> > the arrays and do clone skb/xdpf. The native path is slower than generic > >> > path as we send skbs by pktgen. So the result looks reasonable. > >> > >> How are you running these tests? Still on virtual devices? We really > > > > I run it with the test topology in patch 2/2. The test is run on physical > > machines, but I use veth interface. Do you mean use a physical NIC driver > > for testing? > > Yes, sorry, when I said 'physical machine' I should have also 'physical > NIC'. We really need to know how the performance of this is on the XDP > fast path, i.e., when there are no skbs involved at all. > > > BTW, when using pktgen, I got an panic because the skb don't have enough > > header room. The code path looks like > > > > do_xdp_generic() > > - netif_receive_generic_xdp() > > - skb_headroom(skb) < XDP_PACKET_HEADROOM > > - pskb_expand_head() > > - BUG_ON(skb_shared(skb)) > > > > So I added a draft patch for pktgen, not sure if it has any influence. > > Hmm, as Jesper said pktgen was really not intended to be used this way, > so I guess that's why. I guess I'll let him comment on whether he thinks > it's worth fixing; or you could send this as a proper patch and see if > anyone complains about it ;) Don't use pktgen in this way with veth. If anything pktgen should detect that you use pktgen in virtual interfaces and reject/disallow that you do this. Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-06-16 9:09 ` Jesper Dangaard Brouer @ 2020-06-16 9:47 ` Hangbin Liu 0 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-06-16 9:47 UTC (permalink / raw) To: Jesper Dangaard Brouer Cc: Toke Høiland-Jørgensen, bpf, netdev, Jiri Benc, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On Tue, Jun 16, 2020 at 11:09:22AM +0200, Jesper Dangaard Brouer wrote: > > > BTW, when using pktgen, I got an panic because the skb don't have enough > > > header room. The code path looks like > > > > > > do_xdp_generic() > > > - netif_receive_generic_xdp() > > > - skb_headroom(skb) < XDP_PACKET_HEADROOM > > > - pskb_expand_head() > > > - BUG_ON(skb_shared(skb)) > > > > > > So I added a draft patch for pktgen, not sure if it has any influence. > > > > Hmm, as Jesper said pktgen was really not intended to be used this way, > > so I guess that's why. I guess I'll let him comment on whether he thinks > > it's worth fixing; or you could send this as a proper patch and see if > > anyone complains about it ;) > > Don't use pktgen in this way with veth. If anything pktgen should > detect that you use pktgen in virtual interfaces and reject/disallow > that you do this. OK, got it. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-05-27 10:21 ` [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support Toke Høiland-Jørgensen 2020-05-27 10:32 ` Eelco Chaudron 2020-05-27 12:38 ` Hangbin Liu @ 2020-06-03 2:40 ` Hangbin Liu 2020-06-03 11:05 ` Toke Høiland-Jørgensen 2 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-06-03 2:40 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On Wed, May 27, 2020 at 12:21:54PM +0200, Toke Høiland-Jørgensen wrote: > > The example in patch 2 is functional, but not a lot of effort > > has been made on performance optimisation. I did a simple test(pkt size 64) > > with pktgen. Here is the test result with BPF_MAP_TYPE_DEVMAP_HASH > > arrays: > > > > bpf_redirect_map() with 1 ingress, 1 egress: > > generic path: ~1600k pps > > native path: ~980k pps > > > > bpf_redirect_map_multi() with 1 ingress, 3 egress: > > generic path: ~600k pps > > native path: ~480k pps > > > > bpf_redirect_map_multi() with 1 ingress, 9 egress: > > generic path: ~125k pps > > native path: ~100k pps > > > > The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop > > the arrays and do clone skb/xdpf. The native path is slower than generic > > path as we send skbs by pktgen. So the result looks reasonable. > > How are you running these tests? Still on virtual devices? We really > need results from a physical setup in native mode to assess the impact > on the native-XDP fast path. The numbers above don't tell much in this > regard. I'd also like to see a before/after patch for straight > bpf_redirect_map(), since you're messing with the fast path, and we want > to make sure it's not causing a performance regression for regular > redirect. > > Finally, since the overhead seems to be quite substantial: A comparison > with a regular network stack bridge might make sense? After all we also > want to make sure it's a performance win over that :) Hi Toke, Here is the result I tested with 2 i40e 10G ports on physical machine. The pktgen pkt_size is 64. Bridge forwarding(I use sample/bpf/xdp1 to count the PPS, so there are two modes data): generic mode: 1.32M PPS driver mode: 1.66M PPS xdp_redirect_map: generic mode: 1.88M PPS driver mode: 2.74M PPS xdp_redirect_map_multi: generic mode: 1.38M PPS driver mode: 2.73M PPS So what do you think about the data. If you are OK, I will update my patch and re-post it. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-06-03 2:40 ` Hangbin Liu @ 2020-06-03 11:05 ` Toke Høiland-Jørgensen 2020-06-04 4:09 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2020-06-03 11:05 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi Hangbin Liu <liuhangbin@gmail.com> writes: > On Wed, May 27, 2020 at 12:21:54PM +0200, Toke Høiland-Jørgensen wrote: >> > The example in patch 2 is functional, but not a lot of effort >> > has been made on performance optimisation. I did a simple test(pkt size 64) >> > with pktgen. Here is the test result with BPF_MAP_TYPE_DEVMAP_HASH >> > arrays: >> > >> > bpf_redirect_map() with 1 ingress, 1 egress: >> > generic path: ~1600k pps >> > native path: ~980k pps >> > >> > bpf_redirect_map_multi() with 1 ingress, 3 egress: >> > generic path: ~600k pps >> > native path: ~480k pps >> > >> > bpf_redirect_map_multi() with 1 ingress, 9 egress: >> > generic path: ~125k pps >> > native path: ~100k pps >> > >> > The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop >> > the arrays and do clone skb/xdpf. The native path is slower than generic >> > path as we send skbs by pktgen. So the result looks reasonable. >> >> How are you running these tests? Still on virtual devices? We really >> need results from a physical setup in native mode to assess the impact >> on the native-XDP fast path. The numbers above don't tell much in this >> regard. I'd also like to see a before/after patch for straight >> bpf_redirect_map(), since you're messing with the fast path, and we want >> to make sure it's not causing a performance regression for regular >> redirect. >> >> Finally, since the overhead seems to be quite substantial: A comparison >> with a regular network stack bridge might make sense? After all we also >> want to make sure it's a performance win over that :) > > Hi Toke, > > Here is the result I tested with 2 i40e 10G ports on physical machine. > The pktgen pkt_size is 64. These numbers seem a bit low (I'm getting ~8.5MPPS on my test machine for a simple redirect). Some of that may just be performance of the machine, I guess (what are you running this on?), but please check that you are not limited by pktgen itself - i.e., that pktgen is generating traffic at a higher rate than what XDP is processing. > Bridge forwarding(I use sample/bpf/xdp1 to count the PPS, so there are two modes data): > generic mode: 1.32M PPS > driver mode: 1.66M PPS I'm not sure I understand this - what are you measuring here exactly? > xdp_redirect_map: > generic mode: 1.88M PPS > driver mode: 2.74M PPS Please add numbers without your patch applied as well, for comparison. > xdp_redirect_map_multi: > generic mode: 1.38M PPS > driver mode: 2.73M PPS I assume this is with a single interface only, right? Could you please add a test with a second interface (so the packet is cloned) as well? You can just use a veth as the second target device. -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-06-03 11:05 ` Toke Høiland-Jørgensen @ 2020-06-04 4:09 ` Hangbin Liu 2020-06-04 9:44 ` Toke Høiland-Jørgensen 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-06-04 4:09 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On Wed, Jun 03, 2020 at 01:05:28PM +0200, Toke Høiland-Jørgensen wrote: > > Hi Toke, > > > > Here is the result I tested with 2 i40e 10G ports on physical machine. > > The pktgen pkt_size is 64. > > These numbers seem a bit low (I'm getting ~8.5MPPS on my test machine > for a simple redirect). Some of that may just be performance of the > machine, I guess (what are you running this on?), but please check that > you are not limited by pktgen itself - i.e., that pktgen is generating > traffic at a higher rate than what XDP is processing. Here is the test topology, which looks like Host A | Host B | Host C eth0 + eth0 - eth1 + eth0 I did pktgen sending on Host A, forwarding on Host B. Host B is a Dell PowerEdge R730 (128G memory, Intel(R) Xeon(R) CPU E5-2690 v3) eth0, eth1 is an onboard i40e 10G driver Test 1: add eth0, eth1 to br0 and test bridge forwarding Test 2: Test xdp_redirect_map(), eth0 is ingress, eth1 is egress Test 3: Test xdp_redirect_map_multi(), eth0 is ingress, eth1 is egress > > > Bridge forwarding(I use sample/bpf/xdp1 to count the PPS, so there are two modes data): > > generic mode: 1.32M PPS > > driver mode: 1.66M PPS > > I'm not sure I understand this - what are you measuring here exactly? > Finally, since the overhead seems to be quite substantial: A comparison > with a regular network stack bridge might make sense? After all we also > want to make sure it's a performance win over that :) I though you want me also test with bridge forwarding. Am I missing something? > > > xdp_redirect_map: > > generic mode: 1.88M PPS > > driver mode: 2.74M PPS > > Please add numbers without your patch applied as well, for comparison. OK, I will. > > > xdp_redirect_map_multi: > > generic mode: 1.38M PPS > > driver mode: 2.73M PPS > > I assume this is with a single interface only, right? Could you please > add a test with a second interface (so the packet is cloned) as well? > You can just use a veth as the second target device. OK, so the topology on Host B should be like eth0 + eth1 + veth0, eth0 as ingress, eth1 and veth0 as egress, right? Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-06-04 4:09 ` Hangbin Liu @ 2020-06-04 9:44 ` Toke Høiland-Jørgensen 2020-06-04 12:12 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2020-06-04 9:44 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi Hangbin Liu <liuhangbin@gmail.com> writes: > On Wed, Jun 03, 2020 at 01:05:28PM +0200, Toke Høiland-Jørgensen wrote: >> > Hi Toke, >> > >> > Here is the result I tested with 2 i40e 10G ports on physical machine. >> > The pktgen pkt_size is 64. >> >> These numbers seem a bit low (I'm getting ~8.5MPPS on my test machine >> for a simple redirect). Some of that may just be performance of the >> machine, I guess (what are you running this on?), but please check that >> you are not limited by pktgen itself - i.e., that pktgen is generating >> traffic at a higher rate than what XDP is processing. > > Here is the test topology, which looks like > > Host A | Host B | Host C > eth0 + eth0 - eth1 + eth0 > > I did pktgen sending on Host A, forwarding on Host B. > Host B is a Dell PowerEdge R730 (128G memory, Intel(R) Xeon(R) CPU E5-2690 v3) > eth0, eth1 is an onboard i40e 10G driver > > Test 1: add eth0, eth1 to br0 and test bridge forwarding > Test 2: Test xdp_redirect_map(), eth0 is ingress, eth1 is egress > Test 3: Test xdp_redirect_map_multi(), eth0 is ingress, eth1 is egress Right, that all seems reasonable, but that machine is comparable to my test machine, so you should be getting way more than 2.75 MPPS on a regular redirect test. Are you bottlenecked on pktgen or something? Could you please try running Jesper's ethtool stats poller: https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl on eth0 on Host B, and see what PPS values you get on the different counters? >> > Bridge forwarding(I use sample/bpf/xdp1 to count the PPS, so there are two modes data): >> > generic mode: 1.32M PPS >> > driver mode: 1.66M PPS >> >> I'm not sure I understand this - what are you measuring here exactly? > >> Finally, since the overhead seems to be quite substantial: A comparison >> with a regular network stack bridge might make sense? After all we also >> want to make sure it's a performance win over that :) > > I though you want me also test with bridge forwarding. Am I missing something? Yes, but what does this mean: > (I use sample/bpf/xdp1 to count the PPS, so there are two modes data): or rather, why are there two numbers? :) >> > xdp_redirect_map: >> > generic mode: 1.88M PPS >> > driver mode: 2.74M PPS >> >> Please add numbers without your patch applied as well, for comparison. > > OK, I will. >> >> > xdp_redirect_map_multi: >> > generic mode: 1.38M PPS >> > driver mode: 2.73M PPS >> >> I assume this is with a single interface only, right? Could you please >> add a test with a second interface (so the packet is cloned) as well? >> You can just use a veth as the second target device. > > OK, so the topology on Host B should be like > > eth0 + eth1 + veth0, eth0 as ingress, eth1 and veth0 as egress, right? Yup, exactly! -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-06-04 9:44 ` Toke Høiland-Jørgensen @ 2020-06-04 12:12 ` Hangbin Liu 2020-06-04 12:37 ` Toke Høiland-Jørgensen 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-06-04 12:12 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On Thu, Jun 04, 2020 at 11:44:24AM +0200, Toke Høiland-Jørgensen wrote: > Hangbin Liu <liuhangbin@gmail.com> writes: > > Here is the test topology, which looks like > > > > Host A | Host B | Host C > > eth0 + eth0 - eth1 + eth0 > > > > I did pktgen sending on Host A, forwarding on Host B. > > Host B is a Dell PowerEdge R730 (128G memory, Intel(R) Xeon(R) CPU E5-2690 v3) > > eth0, eth1 is an onboard i40e 10G driver > > > > Test 1: add eth0, eth1 to br0 and test bridge forwarding > > Test 2: Test xdp_redirect_map(), eth0 is ingress, eth1 is egress > > Test 3: Test xdp_redirect_map_multi(), eth0 is ingress, eth1 is egress > > Right, that all seems reasonable, but that machine is comparable to > my test machine, so you should be getting way more than 2.75 MPPS on a > regular redirect test. Are you bottlenecked on pktgen or something? Yes, I found the pktgen is bottleneck. I only use 1 thread. By using the cmd you gave to me ./pktgen_sample03_burst_single_flow.sh -i eno1 -d 192.168.200.1 -m f8:bc:12:14:11:20 -t 4 -s 64 Now I could get higher speed. > > Could you please try running Jesper's ethtool stats poller: > https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl Nice tool. > > I though you want me also test with bridge forwarding. Am I missing something? > > Yes, but what does this mean: > > (I use sample/bpf/xdp1 to count the PPS, so there are two modes data): > > or rather, why are there two numbers? :) Just as it said, to test bridge forwarding speed. I use the xdp tool sample/bpf/xdp1 to count the PPS. But there are two modes when attach xdp to eth0, general and driver mode. So there are 2 number.. Now I use the ethtool_stats.pl to count forwarding speed and here is the result: With kernel 5.7(ingress i40e, egress i40e) XDP: bridge: 1.8M PPS xdp_redirect_map: generic mode: 1.9M PPS driver mode: 10.4M PPS Kernel 5.7 + my patch(ingress i40e, egress i40e) bridge: 1.8M xdp_redirect_map: generic mode: 1.86M PPS driver mode: 10.17M PPS xdp_redirect_map_multi: generic mode: 1.53M PPS driver mode: 7.22M PPS Kernel 5.7 + my patch(ingress i40e, egress veth) xdp_redirect_map: generic mode: 1.38M PPS driver mode: 4.15M PPS xdp_redirect_map_multi: generic mode: 1.13M PPS driver mode: 3.55M PPS Kernel 5.7 + my patch(ingress i40e, egress i40e + veth) xdp_redirect_map_multi: generic mode: 1.13M PPS driver mode: 3.47M PPS I added a group that with i40e ingress and veth egress, which shows a significant drop on the speed. It looks like veth driver is a bottleneck, but I don't have more i40e NICs on the test bed... Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-06-04 12:12 ` Hangbin Liu @ 2020-06-04 12:37 ` Toke Høiland-Jørgensen 2020-06-04 14:41 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2020-06-04 12:37 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi Hangbin Liu <liuhangbin@gmail.com> writes: > On Thu, Jun 04, 2020 at 11:44:24AM +0200, Toke Høiland-Jørgensen wrote: >> Hangbin Liu <liuhangbin@gmail.com> writes: >> > Here is the test topology, which looks like >> > >> > Host A | Host B | Host C >> > eth0 + eth0 - eth1 + eth0 >> > >> > I did pktgen sending on Host A, forwarding on Host B. >> > Host B is a Dell PowerEdge R730 (128G memory, Intel(R) Xeon(R) CPU E5-2690 v3) >> > eth0, eth1 is an onboard i40e 10G driver >> > >> > Test 1: add eth0, eth1 to br0 and test bridge forwarding >> > Test 2: Test xdp_redirect_map(), eth0 is ingress, eth1 is egress >> > Test 3: Test xdp_redirect_map_multi(), eth0 is ingress, eth1 is egress >> >> Right, that all seems reasonable, but that machine is comparable to >> my test machine, so you should be getting way more than 2.75 MPPS on a >> regular redirect test. Are you bottlenecked on pktgen or something? > > Yes, I found the pktgen is bottleneck. I only use 1 thread. > By using the cmd you gave to me > ./pktgen_sample03_burst_single_flow.sh -i eno1 -d 192.168.200.1 -m f8:bc:12:14:11:20 -t 4 -s 64 > > Now I could get higher speed. > >> >> Could you please try running Jesper's ethtool stats poller: >> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl > > Nice tool. > >> > I though you want me also test with bridge forwarding. Am I missing something? >> >> Yes, but what does this mean: >> > (I use sample/bpf/xdp1 to count the PPS, so there are two modes data): >> >> or rather, why are there two numbers? :) > > Just as it said, to test bridge forwarding speed. I use the xdp tool > sample/bpf/xdp1 to count the PPS. But there are two modes when attach xdp > to eth0, general and driver mode. So there are 2 number.. > > Now I use the ethtool_stats.pl to count forwarding speed and here is the result: > > With kernel 5.7(ingress i40e, egress i40e) > XDP: > bridge: 1.8M PPS > xdp_redirect_map: > generic mode: 1.9M PPS > driver mode: 10.4M PPS Ah, now we're getting somewhere! :) > Kernel 5.7 + my patch(ingress i40e, egress i40e) > bridge: 1.8M > xdp_redirect_map: > generic mode: 1.86M PPS > driver mode: 10.17M PPS Right, so this corresponds to a ~2ns overhead (10**9/10400000 - 10**9/10170000). This is not too far from being in the noise, I suppose; is the difference consistent? > xdp_redirect_map_multi: > generic mode: 1.53M PPS > driver mode: 7.22M PPS > > Kernel 5.7 + my patch(ingress i40e, egress veth) > xdp_redirect_map: > generic mode: 1.38M PPS > driver mode: 4.15M PPS > xdp_redirect_map_multi: > generic mode: 1.13M PPS > driver mode: 3.55M PPS > > Kernel 5.7 + my patch(ingress i40e, egress i40e + veth) > xdp_redirect_map_multi: > generic mode: 1.13M PPS > driver mode: 3.47M PPS > > I added a group that with i40e ingress and veth egress, which shows > a significant drop on the speed. It looks like veth driver is a bottleneck, > but I don't have more i40e NICs on the test bed... I suspect this may be because veth ends up creating an SKB for each packet after receiving the frame on the peer device (even though it's immediately dropped). Could you please try adding an XDP program that drops the packets on the veth peer of your target, and see if that helps? -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-06-04 12:37 ` Toke Høiland-Jørgensen @ 2020-06-04 14:41 ` Hangbin Liu 2020-06-04 16:02 ` Toke Høiland-Jørgensen 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-06-04 14:41 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On Thu, Jun 04, 2020 at 02:37:23PM +0200, Toke Høiland-Jørgensen wrote: > > Now I use the ethtool_stats.pl to count forwarding speed and here is the result: > > > > With kernel 5.7(ingress i40e, egress i40e) > > XDP: > > bridge: 1.8M PPS > > xdp_redirect_map: > > generic mode: 1.9M PPS > > driver mode: 10.4M PPS > > Ah, now we're getting somewhere! :) > > > Kernel 5.7 + my patch(ingress i40e, egress i40e) > > bridge: 1.8M > > xdp_redirect_map: > > generic mode: 1.86M PPS > > driver mode: 10.17M PPS > > Right, so this corresponds to a ~2ns overhead (10**9/10400000 - > 10**9/10170000). This is not too far from being in the noise, I suppose; > is the difference consistent? Sorry, I didn't get, what different consistent do you mean? > > > xdp_redirect_map_multi: > > generic mode: 1.53M PPS > > driver mode: 7.22M PPS > > > > Kernel 5.7 + my patch(ingress i40e, egress veth) > > xdp_redirect_map: > > generic mode: 1.38M PPS > > driver mode: 4.15M PPS > > xdp_redirect_map_multi: > > generic mode: 1.13M PPS > > driver mode: 3.55M PPS With XDP_DROP in veth perr, the number looks much better xdp_redirect_map: generic mode: 1.64M PPS driver mode: 13.3M PPS xdp_redirect_map_multi: generic mode: 1.29M PPS driver mode: 8.5M PPS > > > > Kernel 5.7 + my patch(ingress i40e, egress i40e + veth) > > xdp_redirect_map_multi: > > generic mode: 1.13M PPS > > driver mode: 3.47M PPS But I don't know why this one get even a little slower.. xdp_redirect_map_multi: generic mode: 0.96M PPS driver mode: 3.14M PPS Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-06-04 14:41 ` Hangbin Liu @ 2020-06-04 16:02 ` Toke Høiland-Jørgensen 2020-06-05 6:26 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2020-06-04 16:02 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi Hangbin Liu <liuhangbin@gmail.com> writes: > On Thu, Jun 04, 2020 at 02:37:23PM +0200, Toke Høiland-Jørgensen wrote: >> > Now I use the ethtool_stats.pl to count forwarding speed and here is the result: >> > >> > With kernel 5.7(ingress i40e, egress i40e) >> > XDP: >> > bridge: 1.8M PPS >> > xdp_redirect_map: >> > generic mode: 1.9M PPS >> > driver mode: 10.4M PPS >> >> Ah, now we're getting somewhere! :) >> >> > Kernel 5.7 + my patch(ingress i40e, egress i40e) >> > bridge: 1.8M >> > xdp_redirect_map: >> > generic mode: 1.86M PPS >> > driver mode: 10.17M PPS >> >> Right, so this corresponds to a ~2ns overhead (10**9/10400000 - >> 10**9/10170000). This is not too far from being in the noise, I suppose; >> is the difference consistent? > > Sorry, I didn't get, what different consistent do you mean? I meant, how much do the numbers vary between each test run? >> > xdp_redirect_map_multi: >> > generic mode: 1.53M PPS >> > driver mode: 7.22M PPS >> > >> > Kernel 5.7 + my patch(ingress i40e, egress veth) >> > xdp_redirect_map: >> > generic mode: 1.38M PPS >> > driver mode: 4.15M PPS >> > xdp_redirect_map_multi: >> > generic mode: 1.13M PPS >> > driver mode: 3.55M PPS > > With XDP_DROP in veth perr, the number looks much better > > xdp_redirect_map: > generic mode: 1.64M PPS > driver mode: 13.3M PPS > xdp_redirect_map_multi: > generic mode: 1.29M PPS > driver mode: 8.5M PPS Is this for a single interface in both cases? Look a bit odd that you get such a big difference all of a sudden; is the redirect failing in one of those cases (should be a hint in the ethtool stats, I think, otherwise check xdp_monitor)? >> > Kernel 5.7 + my patch(ingress i40e, egress i40e + veth) >> > xdp_redirect_map_multi: >> > generic mode: 1.13M PPS >> > driver mode: 3.47M PPS > > But I don't know why this one get even a little slower.. > > xdp_redirect_map_multi: > generic mode: 0.96M PPS > driver mode: 3.14M PPS Yeah, this does seem a bit odd. Don't have any good ideas off the top of my head, but maybe worth double-checking where the time is spent. You can use 'perf' for this, but you need to make sure it's recording the CPU that is processing packets... -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-06-04 16:02 ` Toke Høiland-Jørgensen @ 2020-06-05 6:26 ` Hangbin Liu 2020-06-08 15:32 ` Toke Høiland-Jørgensen 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-06-05 6:26 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On Thu, Jun 04, 2020 at 06:02:54PM +0200, Toke Høiland-Jørgensen wrote: > Hangbin Liu <liuhangbin@gmail.com> writes: > > > On Thu, Jun 04, 2020 at 02:37:23PM +0200, Toke Høiland-Jørgensen wrote: > >> > Now I use the ethtool_stats.pl to count forwarding speed and here is the result: > >> > > >> > With kernel 5.7(ingress i40e, egress i40e) > >> > XDP: > >> > bridge: 1.8M PPS > >> > xdp_redirect_map: > >> > generic mode: 1.9M PPS > >> > driver mode: 10.4M PPS > >> > >> Ah, now we're getting somewhere! :) > >> > >> > Kernel 5.7 + my patch(ingress i40e, egress i40e) > >> > bridge: 1.8M > >> > xdp_redirect_map: > >> > generic mode: 1.86M PPS > >> > driver mode: 10.17M PPS > >> > >> Right, so this corresponds to a ~2ns overhead (10**9/10400000 - > >> 10**9/10170000). This is not too far from being in the noise, I suppose; > >> is the difference consistent? > > > > Sorry, I didn't get, what different consistent do you mean? > > I meant, how much do the numbers vary between each test run? Oh, when run it at the same period, the number is stable, the range is about ~0.05M PPS. But after a long time or reboot, the speed may changed a little. Here is the new test result after I reboot the system: Kernel 5.7 + my patch(ingress i40e, egress i40e) xdp_redirect_map: generic mode: 1.9M PPS driver mode: 10.2M PPS xdp_redirect_map_multi: generic mode: 1.58M PPS driver mode: 7.16M PPS Kernel 5.7 + my patch(ingress i40e, egress i40e + veth(No XDP on peer)) xdp_redirect_map: generic mode: 2.2M PPS driver mode: 14.2M PPS xdp_redirect_map_multi: generic mode: 1.6M PPS driver mode: 9.9M PPS Kernel 5.7 + my patch(ingress i40e, egress i40e + veth(with XDP_DROP on peer)) xdp_redirect_map: generic mode: 1.6M PPS driver mode: 13.6M PPS xdp_redirect_map_multi: generic mode: 1.3M PPS driver mode: 8.7M PPS Kernel 5.7 + my patch(ingress i40e, egress i40e + veth(No XDP on peer)) xdp_redirect_map_multi: generic mode: 1.15M PPS driver mode: 3.48M PPS Kernel 5.7 + my patch(ingress i40e, egress i40e + veth(with XDP_DROP on peer)) xdp_redirect_map_multi: generic mode: 0.98M PPS driver mode: 3.15M PPS This time the number looks more reasonable. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-06-05 6:26 ` Hangbin Liu @ 2020-06-08 15:32 ` Toke Høiland-Jørgensen 2020-06-09 3:03 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2020-06-08 15:32 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi Hangbin Liu <liuhangbin@gmail.com> writes: > On Thu, Jun 04, 2020 at 06:02:54PM +0200, Toke Høiland-Jørgensen wrote: >> Hangbin Liu <liuhangbin@gmail.com> writes: >> >> > On Thu, Jun 04, 2020 at 02:37:23PM +0200, Toke Høiland-Jørgensen wrote: >> >> > Now I use the ethtool_stats.pl to count forwarding speed and here is the result: >> >> > >> >> > With kernel 5.7(ingress i40e, egress i40e) >> >> > XDP: >> >> > bridge: 1.8M PPS >> >> > xdp_redirect_map: >> >> > generic mode: 1.9M PPS >> >> > driver mode: 10.4M PPS >> >> >> >> Ah, now we're getting somewhere! :) >> >> >> >> > Kernel 5.7 + my patch(ingress i40e, egress i40e) >> >> > bridge: 1.8M >> >> > xdp_redirect_map: >> >> > generic mode: 1.86M PPS >> >> > driver mode: 10.17M PPS >> >> >> >> Right, so this corresponds to a ~2ns overhead (10**9/10400000 - >> >> 10**9/10170000). This is not too far from being in the noise, I suppose; >> >> is the difference consistent? >> > >> > Sorry, I didn't get, what different consistent do you mean? >> >> I meant, how much do the numbers vary between each test run? > > Oh, when run it at the same period, the number is stable, the range is about > ~0.05M PPS. But after a long time or reboot, the speed may changed a little. > Here is the new test result after I reboot the system: > > Kernel 5.7 + my patch(ingress i40e, egress i40e) > xdp_redirect_map: > generic mode: 1.9M PPS > driver mode: 10.2M PPS > > xdp_redirect_map_multi: > generic mode: 1.58M PPS > driver mode: 7.16M PPS > > Kernel 5.7 + my patch(ingress i40e, egress i40e + veth(No XDP on peer)) > xdp_redirect_map: > generic mode: 2.2M PPS > driver mode: 14.2M PPS This looks wrong - why is performance increasing when adding another target? How are you even adding another target to regular xdp_redirect_map? -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-06-08 15:32 ` Toke Høiland-Jørgensen @ 2020-06-09 3:03 ` Hangbin Liu 2020-06-09 20:31 ` Toke Høiland-Jørgensen 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-06-09 3:03 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On Mon, Jun 08, 2020 at 05:32:54PM +0200, Toke Høiland-Jørgensen wrote: > Hangbin Liu <liuhangbin@gmail.com> writes: > > > On Thu, Jun 04, 2020 at 06:02:54PM +0200, Toke Høiland-Jørgensen wrote: > >> Hangbin Liu <liuhangbin@gmail.com> writes: > >> > >> > On Thu, Jun 04, 2020 at 02:37:23PM +0200, Toke Høiland-Jørgensen wrote: > >> >> > Now I use the ethtool_stats.pl to count forwarding speed and here is the result: > >> >> > > >> >> > With kernel 5.7(ingress i40e, egress i40e) > >> >> > XDP: > >> >> > bridge: 1.8M PPS > >> >> > xdp_redirect_map: > >> >> > generic mode: 1.9M PPS > >> >> > driver mode: 10.4M PPS > >> >> > >> >> Ah, now we're getting somewhere! :) > >> >> > >> >> > Kernel 5.7 + my patch(ingress i40e, egress i40e) > >> >> > bridge: 1.8M > >> >> > xdp_redirect_map: > >> >> > generic mode: 1.86M PPS > >> >> > driver mode: 10.17M PPS > >> >> > >> >> Right, so this corresponds to a ~2ns overhead (10**9/10400000 - > >> >> 10**9/10170000). This is not too far from being in the noise, I suppose; > >> >> is the difference consistent? > >> > > >> > Sorry, I didn't get, what different consistent do you mean? > >> > >> I meant, how much do the numbers vary between each test run? > > > > Oh, when run it at the same period, the number is stable, the range is about > > ~0.05M PPS. But after a long time or reboot, the speed may changed a little. > > Here is the new test result after I reboot the system: > > > > Kernel 5.7 + my patch(ingress i40e, egress i40e) > > xdp_redirect_map: > > generic mode: 1.9M PPS > > driver mode: 10.2M PPS > > > > xdp_redirect_map_multi: > > generic mode: 1.58M PPS > > driver mode: 7.16M PPS > > > > Kernel 5.7 + my patch(ingress i40e, egress i40e + veth(No XDP on peer)) > > xdp_redirect_map: > > generic mode: 2.2M PPS > > driver mode: 14.2M PPS > > This looks wrong - why is performance increasing when adding another > target? How are you even adding another target to regular > xdp_redirect_map? > Oh, sorry for the typo, the numbers make me crazy, it should be only ingress i40e, egress veth. Here is the right description: Kernel 5.7 + my patch(ingress i40e, egress i40e) xdp_redirect_map: generic mode: 1.9M PPS driver mode: 10.2M PPS xdp_redirect_map_multi: generic mode: 1.58M PPS driver mode: 7.16M PPS Kernel 5.7 + my patch(ingress i40e, egress veth(No XDP on peer)) xdp_redirect_map: generic mode: 2.2M PPS driver mode: 14.2M PPS xdp_redirect_map_multi: generic mode: 1.6M PPS driver mode: 9.9M PPS Kernel 5.7 + my patch(ingress i40e, egress veth(with XDP_DROP on peer)) xdp_redirect_map: generic mode: 1.6M PPS driver mode: 13.6M PPS xdp_redirect_map_multi: generic mode: 1.3M PPS driver mode: 8.7M PPS Kernel 5.7 + my patch(ingress i40e, egress i40e + veth(No XDP on peer)) xdp_redirect_map_multi: generic mode: 1.15M PPS driver mode: 3.48M PPS Kernel 5.7 + my patch(ingress i40e, egress i40e + veth(with XDP_DROP on peer)) xdp_redirect_map_multi: generic mode: 0.98M PPS driver mode: 3.15M PPS The performance number for xdp_redirect_map_multi is not very well. But I think we can optimize after the implementation. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-06-09 3:03 ` Hangbin Liu @ 2020-06-09 20:31 ` Toke Høiland-Jørgensen 2020-06-10 2:35 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2020-06-09 20:31 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi Hangbin Liu <liuhangbin@gmail.com> writes: > On Mon, Jun 08, 2020 at 05:32:54PM +0200, Toke Høiland-Jørgensen wrote: >> Hangbin Liu <liuhangbin@gmail.com> writes: >> >> > On Thu, Jun 04, 2020 at 06:02:54PM +0200, Toke Høiland-Jørgensen wrote: >> >> Hangbin Liu <liuhangbin@gmail.com> writes: >> >> >> >> > On Thu, Jun 04, 2020 at 02:37:23PM +0200, Toke Høiland-Jørgensen wrote: >> >> >> > Now I use the ethtool_stats.pl to count forwarding speed and here is the result: >> >> >> > >> >> >> > With kernel 5.7(ingress i40e, egress i40e) >> >> >> > XDP: >> >> >> > bridge: 1.8M PPS >> >> >> > xdp_redirect_map: >> >> >> > generic mode: 1.9M PPS >> >> >> > driver mode: 10.4M PPS >> >> >> >> >> >> Ah, now we're getting somewhere! :) >> >> >> >> >> >> > Kernel 5.7 + my patch(ingress i40e, egress i40e) >> >> >> > bridge: 1.8M >> >> >> > xdp_redirect_map: >> >> >> > generic mode: 1.86M PPS >> >> >> > driver mode: 10.17M PPS >> >> >> >> >> >> Right, so this corresponds to a ~2ns overhead (10**9/10400000 - >> >> >> 10**9/10170000). This is not too far from being in the noise, I suppose; >> >> >> is the difference consistent? >> >> > >> >> > Sorry, I didn't get, what different consistent do you mean? >> >> >> >> I meant, how much do the numbers vary between each test run? >> > >> > Oh, when run it at the same period, the number is stable, the range is about >> > ~0.05M PPS. But after a long time or reboot, the speed may changed a little. >> > Here is the new test result after I reboot the system: >> > >> > Kernel 5.7 + my patch(ingress i40e, egress i40e) >> > xdp_redirect_map: >> > generic mode: 1.9M PPS >> > driver mode: 10.2M PPS >> > >> > xdp_redirect_map_multi: >> > generic mode: 1.58M PPS >> > driver mode: 7.16M PPS >> > >> > Kernel 5.7 + my patch(ingress i40e, egress i40e + veth(No XDP on peer)) >> > xdp_redirect_map: >> > generic mode: 2.2M PPS >> > driver mode: 14.2M PPS >> >> This looks wrong - why is performance increasing when adding another >> target? How are you even adding another target to regular >> xdp_redirect_map? >> > Oh, sorry for the typo, the numbers make me crazy, it should be only > ingress i40e, egress veth. Here is the right description: > > Kernel 5.7 + my patch(ingress i40e, egress i40e) > xdp_redirect_map: > generic mode: 1.9M PPS > driver mode: 10.2M PPS > > xdp_redirect_map_multi: > generic mode: 1.58M PPS > driver mode: 7.16M PPS > > Kernel 5.7 + my patch(ingress i40e, egress veth(No XDP on peer)) > xdp_redirect_map: > generic mode: 2.2M PPS > driver mode: 14.2M PPS A few messages up-thread you were getting 4.15M PPS in this case - what changed? It's inconsistencies like these that make me suspicious of the whole set of results :/ Are you getting these numbers from ethtool_stats.pl or from the XDP program? What counter are you looking at, exactly? -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-06-09 20:31 ` Toke Høiland-Jørgensen @ 2020-06-10 2:35 ` Hangbin Liu 2020-06-10 10:03 ` Jesper Dangaard Brouer 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-06-10 2:35 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On Tue, Jun 09, 2020 at 10:31:19PM +0200, Toke Høiland-Jørgensen wrote: > > Oh, sorry for the typo, the numbers make me crazy, it should be only > > ingress i40e, egress veth. Here is the right description: > > > > Kernel 5.7 + my patch(ingress i40e, egress i40e) > > xdp_redirect_map: > > generic mode: 1.9M PPS > > driver mode: 10.2M PPS > > > > xdp_redirect_map_multi: > > generic mode: 1.58M PPS > > driver mode: 7.16M PPS > > > > Kernel 5.7 + my patch(ingress i40e, egress veth(No XDP on peer)) > > xdp_redirect_map: > > generic mode: 2.2M PPS > > driver mode: 14.2M PPS > > A few messages up-thread you were getting 4.15M PPS in this case - what > changed? It's inconsistencies like these that make me suspicious of the > whole set of results :/ I got the number after a reboot, not sure what happened. And I also feel surprised... But the result shows the number, so I have to put it here. > > Are you getting these numbers from ethtool_stats.pl or from the XDP > program? What counter are you looking at, exactly? For bridge testing I use ethtool_stats.pl. For later xdp_redirect_map and xdp_redirect_map_multi testing, I checked that ethtool_stats.pl and XDP program shows the same number. When run ethtool_stats.pl the number will go a little bit slower. So at the end I use the xdp program's number. I'm going to re-setup the test environment and share it with you. Hope we could get a final number that we all accept. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support 2020-06-10 2:35 ` Hangbin Liu @ 2020-06-10 10:03 ` Jesper Dangaard Brouer 0 siblings, 0 replies; 219+ messages in thread From: Jesper Dangaard Brouer @ 2020-06-10 10:03 UTC (permalink / raw) To: Hangbin Liu Cc: Toke Høiland-Jørgensen, bpf, netdev, Jiri Benc, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, brouer On Wed, 10 Jun 2020 10:35:08 +0800 Hangbin Liu <liuhangbin@gmail.com> wrote: > On Tue, Jun 09, 2020 at 10:31:19PM +0200, Toke Høiland-Jørgensen wrote: > > > Oh, sorry for the typo, the numbers make me crazy, it should be only > > > ingress i40e, egress veth. Here is the right description: > > > > > > Kernel 5.7 + my patch(ingress i40e, egress i40e) > > > xdp_redirect_map: > > > generic mode: 1.9M PPS > > > driver mode: 10.2M PPS > > > > > > xdp_redirect_map_multi: > > > generic mode: 1.58M PPS > > > driver mode: 7.16M PPS > > > > > > Kernel 5.7 + my patch(ingress i40e, egress veth(No XDP on peer)) > > > xdp_redirect_map: > > > generic mode: 2.2M PPS > > > driver mode: 14.2M PPS > > > > A few messages up-thread you were getting 4.15M PPS in this case - what > > changed? It's inconsistencies like these that make me suspicious of the > > whole set of results :/ > > I got the number after a reboot, not sure what happened. > And I also feel surprised... But the result shows the number, so I have > to put it here. > > > > > Are you getting these numbers from ethtool_stats.pl or from the XDP > > program? What counter are you looking at, exactly? > > For bridge testing I use ethtool_stats.pl. For later xdp_redirect_map > and xdp_redirect_map_multi testing, I checked that ethtool_stats.pl and > XDP program shows the same number. When run ethtool_stats.pl the number > will go a little bit slower. So at the end I use the xdp program's number. You cannot trust the xdp program's number, because it just counts all RX-packets, and don't take into account if the packets are getting dropped. We really want to verify (e.g. with ethtool_stats.pl) that the packets were successfully transmitted. -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv5 bpf-next 0/3] xdp: add a new helper for dev map multicast support 2020-05-26 14:05 ` [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu ` (2 preceding siblings ...) 2020-05-27 10:21 ` [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support Toke Høiland-Jørgensen @ 2020-07-01 4:19 ` Hangbin Liu 2020-07-01 4:19 ` [PATCHv5 bpf-next 1/3] " Hangbin Liu ` (3 more replies) 3 siblings, 4 replies; 219+ messages in thread From: Hangbin Liu @ 2020-07-01 4:19 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because a physical interface can be part of a logical grouping, such as a bond device. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. The 2nd and 3rd patches are for usage sample and testing purpose, so there is no effort has been made on performance optimisation. I did same tests with pktgen(pkt size 64) to compire with xdp_redirect_map(). Here is the test result(the veth peer has a dummy xdp program with XDP_DROP directly): Version | Test | Native | Generic 5.8 rc1 | xdp_redirect_map i40e->i40e | 10.0M | 1.9M 5.8 rc1 | xdp_redirect_map i40e->veth | 12.7M | 1.6M 5.8 rc1 + patch | xdp_redirect_map i40e->i40e | 10.0M | 1.9M 5.8 rc1 + patch | xdp_redirect_map i40e->veth | 12.3M | 1.6M 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e | 7.2M | 1.5M 5.8 rc1 + patch | xdp_redirect_map_multi i40e->veth | 8.5M | 1.3M 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e+veth | 3.0M | 0.98M The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop the arrays and do clone skb/xdpf. The native path is slower than generic path as we send skbs by pktgen. So the result looks reasonable. Last but not least, thanks a lot to Jiri, Eelco, Toke and Jesper for suggestions and help on implementation. [0] https://xdp-project.net/#Handling-multicast v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). f) Split the tests from sample and add a bpf kernel selftest patch. v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Hangbin Liu (3): xdp: add a new helper for dev map multicast support sample/bpf: add xdp_redirect_map_multicast test selftests/bpf: add xdp_redirect_multi test include/linux/bpf.h | 20 ++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 25 ++- kernel/bpf/devmap.c | 154 ++++++++++++++++ kernel/bpf/verifier.c | 6 + net/core/filter.c | 109 ++++++++++- net/core/xdp.c | 29 +++ samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 57 ++++++ samples/bpf/xdp_redirect_map_multi_user.c | 166 +++++++++++++++++ tools/include/uapi/linux/bpf.h | 25 ++- tools/testing/selftests/bpf/Makefile | 4 +- .../bpf/progs/xdp_redirect_multi_kern.c | 90 +++++++++ .../selftests/bpf/test_xdp_redirect_multi.sh | 164 +++++++++++++++++ .../selftests/bpf/xdp_redirect_multi.c | 173 ++++++++++++++++++ 16 files changed, 1019 insertions(+), 8 deletions(-) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c -- 2.25.4 ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv5 bpf-next 1/3] xdp: add a new helper for dev map multicast support 2020-07-01 4:19 ` [PATCHv5 bpf-next 0/3] xdp: add a new helper for " Hangbin Liu @ 2020-07-01 4:19 ` Hangbin Liu 2020-07-01 5:09 ` Andrii Nakryiko 2020-07-01 18:33 ` kernel test robot 2020-07-01 4:19 ` [PATCHv5 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu ` (2 subsequent siblings) 3 siblings, 2 replies; 219+ messages in thread From: Hangbin Liu @ 2020-07-01 4:19 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu This patch is for xdp multicast support. In this implementation we add a new helper to accept two maps: forward map and exclude map. We will redirect the packet to all the interfaces in *forward map*, but exclude the interfaces that in *exclude map*. To achive this I add a new ex_map for struct bpf_redirect_info. in the helper I set tgt_value to NULL to make a difference with bpf_xdp_redirect_map() We also add a flag *BPF_F_EXCLUDE_INGRESS* incase you don't want to create a exclude map for each interface and just want to exclude the ingress interface. The general data path is kept in net/core/filter.c. The native data path is in kernel/bpf/devmap.c so we can use direct calls to get better performace. v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- include/linux/bpf.h | 20 +++++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 25 +++++- kernel/bpf/devmap.c | 154 +++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++ net/core/filter.c | 109 +++++++++++++++++++++-- net/core/xdp.c | 29 +++++++ tools/include/uapi/linux/bpf.h | 25 +++++- 9 files changed, 363 insertions(+), 7 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 3d2ade703a35..c77bc70dba87 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1264,6 +1264,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx); int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, struct net_device *dev_rx); +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex); +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); bool dev_map_can_have_prog(struct bpf_map *map); @@ -1406,6 +1411,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return 0; } +static inline +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + return false; +} + +static inline +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + return 0; +} + struct sk_buff; static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, diff --git a/include/linux/filter.h b/include/linux/filter.h index 259377723603..cf5b5b1d9ae5 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -612,6 +612,7 @@ struct bpf_redirect_info { u32 tgt_index; void *tgt_value; struct bpf_map *map; + struct bpf_map *ex_map; u32 kern_flags; }; diff --git a/include/net/xdp.h b/include/net/xdp.h index 609f819ed08b..deb6c104e698 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -110,6 +110,7 @@ void xdp_warn(const char *msg, const char *func, const int line); #define XDP_WARN(msg) xdp_warn(msg, __func__, __LINE__) struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); static inline void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 0cb8ec948816..d7de6c0b32e4 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3285,6 +3285,23 @@ union bpf_attr { * Dynamically cast a *sk* pointer to a *udp6_sock* pointer. * Return * *sk* if casting is valid, or NULL otherwise. + * + * int bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3427,7 +3444,8 @@ union bpf_attr { FN(skc_to_tcp_sock), \ FN(skc_to_tcp_timewait_sock), \ FN(skc_to_tcp_request_sock), \ - FN(skc_to_udp6_sock), + FN(skc_to_udp6_sock), \ + FN(redirect_map_multi), /* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call @@ -3588,6 +3606,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 58acc46861ef..8a45fc9e2ccb 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -510,6 +510,160 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return __xdp_enqueue(dev, xdp, dev_rx); } +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + return dev_map_get_next_key(map, key, next_key); + case BPF_MAP_TYPE_DEVMAP_HASH: + return dev_map_hash_get_next_key(map, key, next_key); + default: + break; + } + + return -ENOENT; +} + +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + struct bpf_dtab_netdev *ex_obj = NULL; + u32 key, next_key; + int err; + + if (obj->dev->ifindex == exclude_ifindex) + return true; + + if (!map) + return false; + + err = devmap_get_next_key(map, NULL, &key); + if (err) + return false; + + for (;;) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + ex_obj = __dev_map_lookup_elem(map, key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + ex_obj = __dev_map_hash_lookup_elem(map, key); + break; + default: + break; + } + + if (ex_obj && ex_obj->dev->ifindex == obj->dev->ifindex) + return true; + + err = devmap_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + return false; +} + +struct bpf_dtab_netdev *devmap_get_next_obj(struct xdp_buff *xdp, struct bpf_map *map, + struct bpf_map *ex_map, u32 *key, + u32 *next_key, int ex_ifindex) +{ + struct bpf_dtab_netdev *obj; + struct net_device *dev; + u32 *tmp_key = key; + int err; + + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + return NULL; + + for (;;) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + obj = __dev_map_lookup_elem(map, *next_key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + obj = __dev_map_hash_lookup_elem(map, *next_key); + break; + default: + break; + } + + if (!obj || dev_in_exclude_map(obj, ex_map, ex_ifindex)) + goto find_next; + + dev = obj->dev; + + if (!dev->netdev_ops->ndo_xdp_xmit) + goto find_next; + + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + if (unlikely(err)) + goto find_next; + + return obj; + +find_next: + tmp_key = next_key; + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + break; + } + + return NULL; +} + +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + struct bpf_dtab_netdev *obj = NULL, *next_obj = NULL; + struct xdp_frame *xdpf, *nxdpf; + bool last_one = false; + int ex_ifindex; + u32 key, next_key; + + ex_ifindex = flags & BPF_F_EXCLUDE_INGRESS ? dev_rx->ifindex : 0; + + /* Find first available obj */ + obj = devmap_get_next_obj(xdp, map, ex_map, NULL, &key, ex_ifindex); + if (!obj) + return 0; + + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) + return -EOVERFLOW; + + for (;;) { + /* Check if we still have one more available obj */ + next_obj = devmap_get_next_obj(xdp, map, ex_map, &key, + &next_key, ex_ifindex); + if (!next_obj) + last_one = true; + + if (last_one) { + bq_enqueue(obj->dev, xdpf, dev_rx); + return 0; + } + + nxdpf = xdpf_clone(xdpf); + if (unlikely(!nxdpf)) { + xdp_return_frame_rx_napi(xdpf); + return -ENOMEM; + } + + bq_enqueue(obj->dev, nxdpf, dev_rx); + + /* Deal with next obj */ + obj = next_obj; + key = next_key; + } +} + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 7de98906ddf4..8302b68ef953 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4110,6 +4110,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: if (func_id != BPF_FUNC_redirect_map && + func_id != BPF_FUNC_redirect_map_multi && func_id != BPF_FUNC_map_lookup_elem) goto error; break; @@ -4202,6 +4203,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, map->map_type != BPF_MAP_TYPE_XSKMAP) goto error; break; + case BPF_FUNC_redirect_map_multi: + if (map->map_type != BPF_MAP_TYPE_DEVMAP && + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + goto error; + break; case BPF_FUNC_sk_redirect_map: case BPF_FUNC_msg_redirect_map: case BPF_FUNC_sock_map_update: diff --git a/net/core/filter.c b/net/core/filter.c index c796e141ea8e..37df1ea747ae 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3515,12 +3515,19 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { }; static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) + struct bpf_map *map, struct xdp_buff *xdp, + struct bpf_map *ex_map, u32 flags) { switch (map->map_type) { case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: - return dev_map_enqueue(fwd, xdp, dev_rx); + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) + return dev_map_enqueue(fwd, xdp, dev_rx); + else + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, flags); case BPF_MAP_TYPE_CPUMAP: return cpu_map_enqueue(fwd, xdp, dev_rx); case BPF_MAP_TYPE_XSKMAP: @@ -3577,12 +3584,14 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); struct bpf_map *map = READ_ONCE(ri->map); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (unlikely(!map)) { @@ -3594,7 +3603,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = dev_xdp_enqueue(fwd, xdp, dev); } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, ri->flags); } if (unlikely(err)) @@ -3608,6 +3617,55 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, } EXPORT_SYMBOL_GPL(xdp_do_redirect); +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, + struct bpf_prog *xdp_prog, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) + +{ + struct bpf_dtab_netdev *dst; + struct sk_buff *nskb; + bool exclude_ingress; + u32 key, next_key; + void *fwd; + int err; + + /* Get first key from forward map */ + err = map->ops->map_get_next_key(map, NULL, &key); + if (err) + return err; + + exclude_ingress = !!(flags & BPF_F_EXCLUDE_INGRESS); + + for (;;) { + fwd = __xdp_map_lookup_elem(map, key); + if (fwd) { + dst = (struct bpf_dtab_netdev *)fwd; + if (dev_in_exclude_map(dst, ex_map, + exclude_ingress ? dev->ifindex : 0)) + goto find_next; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -ENOMEM; + + /* Try forword next one no mater the current forward + * succeed or not */ + dev_map_generic_redirect(dst, nskb, xdp_prog); + } + +find_next: + err = map->ops->map_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + consume_skb(skb); + return 0; +} + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, @@ -3615,19 +3673,30 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, struct bpf_map *map) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err = 0; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (map->map_type == BPF_MAP_TYPE_DEVMAP || map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { - struct bpf_dtab_netdev *dst = fwd; + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) { + struct bpf_dtab_netdev *dst = fwd; + + err = dev_map_generic_redirect(dst, skb, xdp_prog); + } else { + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, + ex_map, ri->flags); + } - err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { @@ -3741,6 +3810,34 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { .arg3_type = ARG_ANYTHING, }; +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, + struct bpf_map *, ex_map, u64, flags) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + + if (unlikely(!map || flags > BPF_F_EXCLUDE_INGRESS)) + return XDP_ABORTED; + + ri->tgt_index = 0; + /* Set the tgt_value to NULL to distinguish with bpf_xdp_redirect_map */ + ri->tgt_value = NULL; + ri->flags = flags; + ri->ex_map = ex_map; + + WRITE_ONCE(ri->map, map); + + return XDP_REDIRECT; +} + +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { + .func = bpf_xdp_redirect_map_multi, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_CONST_MAP_PTR, + .arg2_type = ARG_CONST_MAP_PTR, + .arg3_type = ARG_ANYTHING, +}; + static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, unsigned long off, unsigned long len) { @@ -6464,6 +6561,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_redirect_proto; case BPF_FUNC_redirect_map: return &bpf_xdp_redirect_map_proto; + case BPF_FUNC_redirect_map_multi: + return &bpf_xdp_redirect_map_multi_proto; case BPF_FUNC_xdp_adjust_tail: return &bpf_xdp_adjust_tail_proto; case BPF_FUNC_fib_lookup: diff --git a/net/core/xdp.c b/net/core/xdp.c index 90f44f382115..7e291f1015d7 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -475,3 +475,32 @@ void xdp_warn(const char *msg, const char *func, const int line) WARN(1, "XDP_WARN: %s(line:%d): %s\n", func, line, msg); }; EXPORT_SYMBOL_GPL(xdp_warn); + +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) +{ + unsigned int headroom, totalsize; + struct xdp_frame *nxdpf; + struct page *page; + void *addr; + + headroom = xdpf->headroom + sizeof(*xdpf); + totalsize = headroom + xdpf->len; + + if (unlikely(totalsize > PAGE_SIZE)) + return NULL; + page = dev_alloc_page(); + if (!page) + return NULL; + addr = page_to_virt(page); + + memcpy(addr, xdpf, totalsize); + + nxdpf = addr; + nxdpf->data = addr + headroom; + nxdpf->frame_sz = PAGE_SIZE; + nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0; + nxdpf->mem.id = 0; + + return nxdpf; +} +EXPORT_SYMBOL_GPL(xdpf_clone); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 0cb8ec948816..d7de6c0b32e4 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3285,6 +3285,23 @@ union bpf_attr { * Dynamically cast a *sk* pointer to a *udp6_sock* pointer. * Return * *sk* if casting is valid, or NULL otherwise. + * + * int bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3427,7 +3444,8 @@ union bpf_attr { FN(skc_to_tcp_sock), \ FN(skc_to_tcp_timewait_sock), \ FN(skc_to_tcp_request_sock), \ - FN(skc_to_udp6_sock), + FN(skc_to_udp6_sock), \ + FN(redirect_map_multi), /* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call @@ -3588,6 +3606,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv5 bpf-next 1/3] xdp: add a new helper for dev map multicast support 2020-07-01 4:19 ` [PATCHv5 bpf-next 1/3] " Hangbin Liu @ 2020-07-01 5:09 ` Andrii Nakryiko 2020-07-01 6:51 ` Hangbin Liu 2020-07-01 18:33 ` kernel test robot 1 sibling, 1 reply; 219+ messages in thread From: Andrii Nakryiko @ 2020-07-01 5:09 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, Networking, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, Alexei Starovoitov, Daniel Borkmann, Lorenzo Bianconi On Tue, Jun 30, 2020 at 9:21 PM Hangbin Liu <liuhangbin@gmail.com> wrote: > > This patch is for xdp multicast support. In this implementation we > add a new helper to accept two maps: forward map and exclude map. > We will redirect the packet to all the interfaces in *forward map*, but > exclude the interfaces that in *exclude map*. > > To achive this I add a new ex_map for struct bpf_redirect_info. > in the helper I set tgt_value to NULL to make a difference with > bpf_xdp_redirect_map() > > We also add a flag *BPF_F_EXCLUDE_INGRESS* incase you don't want to > create a exclude map for each interface and just want to exclude the > ingress interface. > > The general data path is kept in net/core/filter.c. The native data > path is in kernel/bpf/devmap.c so we can use direct calls to > get better performace. > > v5: > a) Check devmap_get_next_key() return value. > b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. > c) In function dev_map_enqueue_multi(), consume xdpf for the last > obj instead of the first on. > d) Update helper description and code comments to explain that we > use NULL target value to distinguish multicast and unicast > forwarding. > e) Update memory model, memory id and frame_sz in xdpf_clone(). > > v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo > > v3: Based on Toke's suggestion, do the following update > a) Update bpf_redirect_map_multi() description in bpf.h. > b) Fix exclude_ifindex checking order in dev_in_exclude_map(). > c) Fix one more xdpf clone in dev_map_enqueue_multi(). > d) Go find next one in dev_map_enqueue_multi() if the interface is not > able to forward instead of abort the whole loop. > e) Remove READ_ONCE/WRITE_ONCE for ex_map. > > v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept > include/exclude maps directly. > > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > --- > include/linux/bpf.h | 20 +++++ > include/linux/filter.h | 1 + > include/net/xdp.h | 1 + > include/uapi/linux/bpf.h | 25 +++++- > kernel/bpf/devmap.c | 154 +++++++++++++++++++++++++++++++++ > kernel/bpf/verifier.c | 6 ++ > net/core/filter.c | 109 +++++++++++++++++++++-- > net/core/xdp.c | 29 +++++++ > tools/include/uapi/linux/bpf.h | 25 +++++- > 9 files changed, 363 insertions(+), 7 deletions(-) > [...] > diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h > index 0cb8ec948816..d7de6c0b32e4 100644 > --- a/tools/include/uapi/linux/bpf.h > +++ b/tools/include/uapi/linux/bpf.h > @@ -3285,6 +3285,23 @@ union bpf_attr { > * Dynamically cast a *sk* pointer to a *udp6_sock* pointer. > * Return > * *sk* if casting is valid, or NULL otherwise. > + * > + * int bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) We've recently converted all return types for helpers from int to long, please update accordingly. Thanks. [...] ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv5 bpf-next 1/3] xdp: add a new helper for dev map multicast support 2020-07-01 5:09 ` Andrii Nakryiko @ 2020-07-01 6:51 ` Hangbin Liu 0 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-07-01 6:51 UTC (permalink / raw) To: Andrii Nakryiko Cc: bpf, Networking, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, Alexei Starovoitov, Daniel Borkmann, Lorenzo Bianconi On Tue, Jun 30, 2020 at 10:09:39PM -0700, Andrii Nakryiko wrote: > > diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h > > index 0cb8ec948816..d7de6c0b32e4 100644 > > --- a/tools/include/uapi/linux/bpf.h > > +++ b/tools/include/uapi/linux/bpf.h > > @@ -3285,6 +3285,23 @@ union bpf_attr { > > * Dynamically cast a *sk* pointer to a *udp6_sock* pointer. > > * Return > > * *sk* if casting is valid, or NULL otherwise. > > + * > > + * int bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) > > We've recently converted all return types for helpers from int to > long, please update accordingly. Thanks. > Thanks, I will fix it. - Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv5 bpf-next 1/3] xdp: add a new helper for dev map multicast support 2020-07-01 4:19 ` [PATCHv5 bpf-next 1/3] " Hangbin Liu 2020-07-01 5:09 ` Andrii Nakryiko @ 2020-07-01 18:33 ` kernel test robot 1 sibling, 0 replies; 219+ messages in thread From: kernel test robot @ 2020-07-01 18:33 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: kbuild-all, clang-built-linux, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu [-- Attachment #1: Type: text/plain, Size: 3059 bytes --] Hi Hangbin, Thank you for the patch! Perhaps something to improve: [auto build test WARNING on bpf-next/master] url: https://github.com/0day-ci/linux/commits/Hangbin-Liu/xdp-add-a-new-helper-for-dev-map-multicast-support/20200701-122334 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master config: arm-randconfig-r013-20200701 (attached as .config) compiler: clang version 11.0.0 (https://github.com/llvm/llvm-project c8f1d442d0858f66fd4128fde6f67eb5202fa2b1) reproduce (this is a W=1 build): wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # install arm cross compiling tool for clang build # apt-get install binutils-arm-linux-gnueabi # save the attached .config to linux build tree COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=arm If you fix the issue, kindly add following tag as appropriate Reported-by: kernel test robot <lkp@intel.com> All warnings (new ones prefixed by >>): >> kernel/bpf/devmap.c:571:25: warning: no previous prototype for function 'devmap_get_next_obj' [-Wmissing-prototypes] struct bpf_dtab_netdev *devmap_get_next_obj(struct xdp_buff *xdp, struct bpf_map *map, ^ kernel/bpf/devmap.c:571:1: note: declare 'static' if the function is not intended to be used outside of this translation unit struct bpf_dtab_netdev *devmap_get_next_obj(struct xdp_buff *xdp, struct bpf_map *map, ^ static 1 warning generated. vim +/devmap_get_next_obj +571 kernel/bpf/devmap.c 570 > 571 struct bpf_dtab_netdev *devmap_get_next_obj(struct xdp_buff *xdp, struct bpf_map *map, 572 struct bpf_map *ex_map, u32 *key, 573 u32 *next_key, int ex_ifindex) 574 { 575 struct bpf_dtab_netdev *obj; 576 struct net_device *dev; 577 u32 *tmp_key = key; 578 int err; 579 580 err = devmap_get_next_key(map, tmp_key, next_key); 581 if (err) 582 return NULL; 583 584 for (;;) { 585 switch (map->map_type) { 586 case BPF_MAP_TYPE_DEVMAP: 587 obj = __dev_map_lookup_elem(map, *next_key); 588 break; 589 case BPF_MAP_TYPE_DEVMAP_HASH: 590 obj = __dev_map_hash_lookup_elem(map, *next_key); 591 break; 592 default: 593 break; 594 } 595 596 if (!obj || dev_in_exclude_map(obj, ex_map, ex_ifindex)) 597 goto find_next; 598 599 dev = obj->dev; 600 601 if (!dev->netdev_ops->ndo_xdp_xmit) 602 goto find_next; 603 604 err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); 605 if (unlikely(err)) 606 goto find_next; 607 608 return obj; 609 610 find_next: 611 tmp_key = next_key; 612 err = devmap_get_next_key(map, tmp_key, next_key); 613 if (err) 614 break; 615 } 616 617 return NULL; 618 } 619 --- 0-DAY CI Kernel Test Service, Intel Corporation https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org [-- Attachment #2: .config.gz --] [-- Type: application/gzip, Size: 28561 bytes --] ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv5 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test 2020-07-01 4:19 ` [PATCHv5 bpf-next 0/3] xdp: add a new helper for " Hangbin Liu 2020-07-01 4:19 ` [PATCHv5 bpf-next 1/3] " Hangbin Liu @ 2020-07-01 4:19 ` Hangbin Liu 2020-07-01 4:19 ` [PATCHv5 bpf-next 3/3] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2020-07-09 1:30 ` [PATCHv6 bpf-next 0/3] xdp: add a new helper for dev map multicast support Hangbin Liu 3 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-07-01 4:19 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu This is a sample for xdp multicast. In the sample we could forward all packets between given interfaces. v5: add a null_map as we have strict the arg2 to ARG_CONST_MAP_PTR. Move the testing part to bpf selftest in next patch. v4: no update. v3: add rxcnt map to show the packet transmit speed. v2: no update. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 57 ++++++++ samples/bpf/xdp_redirect_map_multi_user.c | 166 ++++++++++++++++++++++ 3 files changed, 226 insertions(+) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 8403e4762306..000709bb89c3 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map tprogs-y += per_socket_stats_example tprogs-y += xdp_redirect tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_map_multi tprogs-y += xdp_redirect_cpu tprogs-y += xdp_monitor tprogs-y += xdp_rxq_info @@ -97,6 +98,7 @@ test_map_in_map-objs := bpf_load.o test_map_in_map_user.o per_socket_stats_example-objs := cookie_uid_helper_example.o xdp_redirect-objs := xdp_redirect_user.o xdp_redirect_map-objs := xdp_redirect_map_user.o +xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o xdp_redirect_cpu-objs := bpf_load.o xdp_redirect_cpu_user.o xdp_monitor-objs := bpf_load.o xdp_monitor_user.o xdp_rxq_info-objs := xdp_rxq_info_user.o @@ -156,6 +158,7 @@ always-y += tcp_tos_reflect_kern.o always-y += tcp_dumpstats_kern.o always-y += xdp_redirect_kern.o always-y += xdp_redirect_map_kern.o +always-y += xdp_redirect_map_multi_kern.o always-y += xdp_redirect_cpu_kern.o always-y += xdp_monitor_kern.o always-y += xdp_rxq_info_kern.o diff --git a/samples/bpf/xdp_redirect_map_multi_kern.c b/samples/bpf/xdp_redirect_map_multi_kern.c new file mode 100644 index 000000000000..cc7ebaedf55a --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_kern.c @@ -0,0 +1,57 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <uapi/linux/bpf.h> +#include <bpf/bpf_helpers.h> + +struct bpf_map_def SEC("maps") forward_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 256, +}; + +struct bpf_map_def SEC("maps") null_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 1, +}; + +struct bpf_map_def SEC("maps") rxcnt = { + .type = BPF_MAP_TYPE_PERCPU_ARRAY, + .key_size = sizeof(u32), + .value_size = sizeof(long), + .max_entries = 1, +}; + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + long *value; + u32 key = 0; + + /* count packet in global counter */ + value = bpf_map_lookup_elem(&rxcnt, &key); + if (value) + *value += 1; + + return bpf_redirect_map_multi(&forward_map, &null_map, + BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_dummy") +int xdp_pass(struct xdp_md *ctx) +{ + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/samples/bpf/xdp_redirect_map_multi_user.c b/samples/bpf/xdp_redirect_map_multi_user.c new file mode 100644 index 000000000000..49f44c91b672 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_user.c @@ -0,0 +1,166 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; +static int rxcnt; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void poll_stats(int interval) +{ + unsigned int nr_cpus = bpf_num_possible_cpus(); + __u64 values[nr_cpus], prev[nr_cpus]; + + memset(prev, 0, sizeof(prev)); + + while (1) { + __u64 sum = 0; + __u32 key = 0; + int i; + + sleep(interval); + assert(bpf_map_lookup_elem(rxcnt, &key, values) == 0); + for (i = 0; i < nr_cpus; i++) + sum += (values[i] - prev[i]); + if (sum) + printf("Forwarding %10llu pkt/s\n", sum / interval); + memcpy(prev, values, sizeof(values)); + } +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n", + prog); +} + +int main(int argc, char **argv) +{ + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + int prog_fd, forward_map; + int i, ret, opt, ifindex; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNF")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) + xdp_flags |= XDP_FLAGS_DRV_MODE; + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + return 1; + + forward_map = bpf_object__find_map_fd_by_name(obj, "forward_map"); + rxcnt = bpf_object__find_map_fd_by_name(obj, "rxcnt"); + + if (forward_map < 0 || rxcnt < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + return 1; + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* Add all the interfaces to group all */ + ret = bpf_map_update_elem(forward_map, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + } + + poll_stats(2); + + return 0; + +err_out: + return 1; +} -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv5 bpf-next 3/3] selftests/bpf: add xdp_redirect_multi test 2020-07-01 4:19 ` [PATCHv5 bpf-next 0/3] xdp: add a new helper for " Hangbin Liu 2020-07-01 4:19 ` [PATCHv5 bpf-next 1/3] " Hangbin Liu 2020-07-01 4:19 ` [PATCHv5 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu @ 2020-07-01 4:19 ` Hangbin Liu 2020-07-09 1:30 ` [PATCHv6 bpf-next 0/3] xdp: add a new helper for dev map multicast support Hangbin Liu 3 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-07-01 4:19 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu Add a bpf selftest for new helper xdp_redirect_map_multi(). In this test we have 3 forward groups groups and 1 exclude group. The test will redirect each interface's packets to all the interfaces in the forward group, and exclude the interface in exclude map. We will also test both DEVMAP and DEVMAP_HASH with xdp generic and drv. For more test details, you can find it in the test script. Here is the test result. ]# ./test_xdp_redirect_multi.sh Pass: xdpgeneric arp ns1-2 Pass: xdpgeneric arp ns1-3 Pass: xdpgeneric arp ns1-4 Pass: xdpgeneric ping ns1-2 Pass: xdpgeneric ping ns1-3 Pass: xdpgeneric ping ns1-4 Pass: xdpgeneric ping6 ns2-1 Pass: xdpgeneric ping6 ns2-3 Pass: xdpgeneric ping6 ns2-4 Pass: xdpdrv arp ns1-2 Pass: xdpdrv arp ns1-3 Pass: xdpdrv arp ns1-4 Pass: xdpdrv ping ns1-2 Pass: xdpdrv ping ns1-3 Pass: xdpdrv ping ns1-4 Pass: xdpdrv ping6 ns2-1 Pass: xdpdrv ping6 ns2-3 Pass: xdpdrv ping6 ns2-4 Summary: PASS 18, FAIL 0 Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- tools/testing/selftests/bpf/Makefile | 4 +- .../bpf/progs/xdp_redirect_multi_kern.c | 90 +++++++++ .../selftests/bpf/test_xdp_redirect_multi.sh | 164 +++++++++++++++++ .../selftests/bpf/xdp_redirect_multi.c | 173 ++++++++++++++++++ 4 files changed, 430 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 1f9c696b3edf..66b857210814 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -51,6 +51,7 @@ TEST_FILES = test_lwt_ip_encap.o \ # Order correspond to 'make run_tests' order TEST_PROGS := test_kmod.sh \ test_xdp_redirect.sh \ + test_xdp_redirect_multi.sh \ test_xdp_meta.sh \ test_xdp_veth.sh \ test_offload.py \ @@ -79,7 +80,8 @@ TEST_PROGS_EXTENDED := with_addr.sh \ # Compile but not part of 'make run_tests' TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ - test_lirc_mode2_user xdping test_cpp runqslower bench + test_lirc_mode2_user xdping test_cpp runqslower bench \ + xdp_redirect_multi TEST_CUSTOM_PROGS = urandom_read diff --git a/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c new file mode 100644 index 000000000000..70b8476b9df3 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c @@ -0,0 +1,90 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <string.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ip.h> +#include <linux/ipv6.h> + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_endian.h> + +struct bpf_map_def SEC("maps") forward_map_v4 = { + .type = BPF_MAP_TYPE_DEVMAP, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 4096, +}; + +struct bpf_map_def SEC("maps") forward_map_v6 = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") forward_map_all = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") exclude_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") null_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 1, +}; + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + struct ethhdr *eth = data; + __u16 h_proto; + __u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + h_proto = eth->h_proto; + + if (h_proto == bpf_htons(ETH_P_IP)) + return bpf_redirect_map_multi(&forward_map_v4, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else if (h_proto == bpf_htons(ETH_P_IPV6)) + return bpf_redirect_map_multi(&forward_map_v6, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else + return bpf_redirect_map_multi(&forward_map_all, &null_map, + BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_dummy") +int xdp_pass(struct xdp_md *ctx) +{ + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh new file mode 100755 index 000000000000..f4f8f751854e --- /dev/null +++ b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh @@ -0,0 +1,164 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Test topology: +# - - - - - - - - - - - - - - - - - - - - - - - - - +# | veth1 veth2 veth3 veth4 | ... init net +# - -| - - - - - - | - - - - - - | - - - - - - | - - +# --------- --------- --------- --------- +# | veth0 | | veth0 | | veth0 | | veth0 | ... +# --------- --------- --------- --------- +# ns1 ns2 ns3 ns4 +# +# Forward multicast groups: +# Forward group all has interfaces: veth1, veth2, veth3, veth4, ... (All traffic except IPv4, IPv6) +# Forward group v4 has interfaces: veth1, veth3, veth4, ... (For IPv4 traffic only) +# Forward group v6 has interfaces: veth2, veth3, veth4, ... (For IPv6 traffic only) +# Exclude Groups: +# Exclude group: veth3 (assume ns3 is in black list) +# +# Test modules: +# XDP modes: generic, native +# map types: group v4 use DEVMAP, others use DEVMAP_HASH +# +# Test cases: +# ARP(we didn't block ARP for ns3): +# ns1 -> gw: ns2, ns3, ns4 should receive the arp request +# IPv4: +# ns1 -> ns2 (fail), ns1 -> ns3 (fail), ns1 -> ns4 (pass) +# IPv6 +# ns2 -> ns1 (fail), ns2 -> ns3 (fail), ns2 -> ns4 (pass) +# + + +# netns numbers +NUM=4 +IFACES="" +DRV_MODE="xdpgeneric xdpdrv" +PASS=0 +FAIL=0 + +test_pass() +{ + echo "Pass: $@" + PASS=$((PASS + 1)) +} + +test_fail() +{ + echo "fail: $@" + FAIL=$((FAIL + 1)) +} + +clean_up() +{ + for i in $(seq $NUM); do + ip link del veth$i 2> /dev/null + ip netns del ns$i 2> /dev/null + done + rm -f xdp_redirect_*.log arp_ns*.log +} + +# Kselftest framework requirement - SKIP code is 4. +check_env() +{ + ip link set dev lo xdpgeneric off &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without the ip xdpgeneric support" + exit 4 + fi + + which tcpdump &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without tcpdump" + exit 4 + fi +} + +setup_ns() +{ + local mode=$1 + IFACES="" + + for i in $(seq $NUM); do + ip netns add ns$i + ip link add veth$i type veth peer name veth0 netns ns$i + ip link set veth$i up + ip -n ns$i link set veth0 up + + ip -n ns$i addr add 192.0.2.$i/24 dev veth0 + ip -n ns$i addr add 2001:db8::$i/64 dev veth0 + ip -n ns$i link set veth0 $mode obj \ + xdp_redirect_multi_kern.o sec xdp_dummy &> /dev/null || \ + { test_fail "Unable to load dummy xdp" && exit 1; } + IFACES="$IFACES veth$i" + done +} + +do_ping_tests() +{ + local mode=$1 + + # arp test + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -i veth0 -nn -l -e &> arp_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-2_${mode}.log && \ + test_pass "$mode arp ns1-2" || test_fail "$mode arp ns1-2" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-3_${mode}.log && \ + test_pass "$mode arp ns1-3" || test_fail "$mode arp ns1-3" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-4_${mode}.log && \ + test_pass "$mode arp ns1-4" || test_fail "$mode arp ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-2" || test_pass "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-3" || test_pass "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" + + # ping6 test + ip netns exec ns2 ping6 2001:db8::1 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-1" || test_pass "$mode ping6 ns2-1" + ip netns exec ns2 ping6 2001:db8::3 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-3" || test_pass "$mode ping6 ns2-3" + ip netns exec ns2 ping6 2001:db8::4 -c 4 &> /dev/null && \ + test_pass "$mode ping6 ns2-4" || test_fail "$mode ping6 ns2-4" +} + +do_tests() +{ + local mode=$1 + local drv_p + + [ ${mode} == "xdpdrv" ] && drv_p="-N" || drv_p="-S" + + # run `ulimit -l unlimited` if you got errors like + # libbpf: Error in bpf_object__probe_global_data():Operation not permitted(1). + ./xdp_redirect_multi $drv_p $IFACES &> xdp_redirect_${mode}.log & + xdp_pid=$! + sleep 10 + + do_ping_tests $mode + + kill $xdp_pid +} + +trap clean_up 0 2 3 6 9 + +check_env + +for mode in ${DRV_MODE}; do + setup_ns $mode + do_tests $mode + sleep 10 + clean_up + sleep 5 +done + +echo "Summary: PASS $PASS, FAIL $FAIL" +[ $FAIL -eq 0 ] && exit 0 || exit 1 diff --git a/tools/testing/selftests/bpf/xdp_redirect_multi.c b/tools/testing/selftests/bpf/xdp_redirect_multi.c new file mode 100644 index 000000000000..5626005cb679 --- /dev/null +++ b/tools/testing/selftests/bpf/xdp_redirect_multi.c @@ -0,0 +1,173 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n", + prog); +} + +int main(int argc, char **argv) +{ + int prog_fd, group_all, group_v4, group_v6, exclude; + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + int i, ret, opt, ifindex; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNF")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) + xdp_flags |= XDP_FLAGS_DRV_MODE; + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + return 1; + + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_all"); + group_v4 = bpf_object__find_map_fd_by_name(obj, "forward_map_v4"); + group_v6 = bpf_object__find_map_fd_by_name(obj, "forward_map_v6"); + exclude = bpf_object__find_map_fd_by_name(obj, "exclude_map"); + + if (group_all < 0 || group_v4 < 0 || group_v6 < 0 || exclude < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + return 1; + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* Add all the interfaces to group all */ + ret = bpf_map_update_elem(group_all, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* For testing: remove the 1st interfaces from group v6 */ + if (i != 0) { + ret = bpf_map_update_elem(group_v6, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: remove the 2nd interfaces from group v4 */ + if (i != 1) { + ret = bpf_map_update_elem(group_v4, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: add the 3rd interfaces to exclude map */ + if (i == 2) { + ret = bpf_map_update_elem(exclude, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + } + + /* sleep some time for testing */ + sleep(999); + + return 0; + +err_out: + return 1; +} -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv6 bpf-next 0/3] xdp: add a new helper for dev map multicast support 2020-07-01 4:19 ` [PATCHv5 bpf-next 0/3] xdp: add a new helper for " Hangbin Liu ` (2 preceding siblings ...) 2020-07-01 4:19 ` [PATCHv5 bpf-next 3/3] selftests/bpf: add xdp_redirect_multi test Hangbin Liu @ 2020-07-09 1:30 ` Hangbin Liu 2020-07-09 1:30 ` [PATCHv6 bpf-next 1/3] " Hangbin Liu ` (4 more replies) 3 siblings, 5 replies; 219+ messages in thread From: Hangbin Liu @ 2020-07-09 1:30 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because a physical interface can be part of a logical grouping, such as a bond device. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. The 2nd and 3rd patches are for usage sample and testing purpose, so there is no effort has been made on performance optimisation. I did same tests with pktgen(pkt size 64) to compire with xdp_redirect_map(). Here is the test result(the veth peer has a dummy xdp program with XDP_DROP directly): Version | Test | Native | Generic 5.8 rc1 | xdp_redirect_map i40e->i40e | 10.0M | 1.9M 5.8 rc1 | xdp_redirect_map i40e->veth | 12.7M | 1.6M 5.8 rc1 + patch | xdp_redirect_map i40e->i40e | 10.0M | 1.9M 5.8 rc1 + patch | xdp_redirect_map i40e->veth | 12.3M | 1.6M 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e | 7.2M | 1.5M 5.8 rc1 + patch | xdp_redirect_map_multi i40e->veth | 8.5M | 1.3M 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e+veth | 3.0M | 0.98M The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop the arrays and do clone skb/xdpf. The native path is slower than generic path as we send skbs by pktgen. So the result looks reasonable. Last but not least, thanks a lot to Jiri, Eelco, Toke and Jesper for suggestions and help on implementation. [0] https://xdp-project.net/#Handling-multicast v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). f) Split the tests from sample and add a bpf kernel selftest patch. v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Hangbin Liu (3): xdp: add a new helper for dev map multicast support sample/bpf: add xdp_redirect_map_multicast test selftests/bpf: add xdp_redirect_multi test include/linux/bpf.h | 20 ++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 22 +++ kernel/bpf/devmap.c | 154 ++++++++++++++++ kernel/bpf/verifier.c | 6 + net/core/filter.c | 109 ++++++++++- net/core/xdp.c | 29 +++ samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 57 ++++++ samples/bpf/xdp_redirect_map_multi_user.c | 166 +++++++++++++++++ tools/include/uapi/linux/bpf.h | 22 +++ tools/testing/selftests/bpf/Makefile | 4 +- .../bpf/progs/xdp_redirect_multi_kern.c | 90 +++++++++ .../selftests/bpf/test_xdp_redirect_multi.sh | 164 +++++++++++++++++ .../selftests/bpf/xdp_redirect_multi.c | 173 ++++++++++++++++++ 16 files changed, 1015 insertions(+), 6 deletions(-) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c -- 2.25.4 ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv6 bpf-next 1/3] xdp: add a new helper for dev map multicast support 2020-07-09 1:30 ` [PATCHv6 bpf-next 0/3] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2020-07-09 1:30 ` Hangbin Liu 2020-07-09 16:33 ` David Ahern 2020-07-09 1:30 ` [PATCHv6 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu ` (3 subsequent siblings) 4 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-07-09 1:30 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu This patch is for xdp multicast support. In this implementation we add a new helper to accept two maps: forward map and exclude map. We will redirect the packet to all the interfaces in *forward map*, but exclude the interfaces that in *exclude map*. To achive this I add a new ex_map for struct bpf_redirect_info. in the helper I set tgt_value to NULL to make a difference with bpf_xdp_redirect_map() We also add a flag *BPF_F_EXCLUDE_INGRESS* incase you don't want to create a exclude map for each interface and just want to exclude the ingress interface. The general data path is kept in net/core/filter.c. The native data path is in kernel/bpf/devmap.c so we can use direct calls to get better performace. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- include/linux/bpf.h | 20 +++++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 22 +++++ kernel/bpf/devmap.c | 154 +++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++ net/core/filter.c | 109 +++++++++++++++++++++-- net/core/xdp.c | 29 +++++++ tools/include/uapi/linux/bpf.h | 22 +++++ 9 files changed, 359 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 0cd7f6884c5c..b48d587b8b3b 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1264,6 +1264,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx); int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, struct net_device *dev_rx); +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex); +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); bool dev_map_can_have_prog(struct bpf_map *map); @@ -1406,6 +1411,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return 0; } +static inline +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + return false; +} + +static inline +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + return 0; +} + struct sk_buff; static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, diff --git a/include/linux/filter.h b/include/linux/filter.h index 259377723603..cf5b5b1d9ae5 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -612,6 +612,7 @@ struct bpf_redirect_info { u32 tgt_index; void *tgt_value; struct bpf_map *map; + struct bpf_map *ex_map; u32 kern_flags; }; diff --git a/include/net/xdp.h b/include/net/xdp.h index 609f819ed08b..deb6c104e698 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -110,6 +110,7 @@ void xdp_warn(const char *msg, const char *func, const int line); #define XDP_WARN(msg) xdp_warn(msg, __func__, __LINE__) struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); static inline void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 548a749aebb3..a14e41309e73 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3319,6 +3319,22 @@ union bpf_attr { * A non-negative value equal to or less than *size* on success, * or a negative error in case of failure. * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3463,6 +3479,7 @@ union bpf_attr { FN(skc_to_tcp_request_sock), \ FN(skc_to_udp6_sock), \ FN(get_task_stack), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -3624,6 +3641,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 10abb06065bb..617a51391971 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -512,6 +512,160 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return __xdp_enqueue(dev, xdp, dev_rx); } +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + return dev_map_get_next_key(map, key, next_key); + case BPF_MAP_TYPE_DEVMAP_HASH: + return dev_map_hash_get_next_key(map, key, next_key); + default: + break; + } + + return -ENOENT; +} + +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + struct bpf_dtab_netdev *ex_obj = NULL; + u32 key, next_key; + int err; + + if (obj->dev->ifindex == exclude_ifindex) + return true; + + if (!map) + return false; + + err = devmap_get_next_key(map, NULL, &key); + if (err) + return false; + + for (;;) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + ex_obj = __dev_map_lookup_elem(map, key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + ex_obj = __dev_map_hash_lookup_elem(map, key); + break; + default: + break; + } + + if (ex_obj && ex_obj->dev->ifindex == obj->dev->ifindex) + return true; + + err = devmap_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + return false; +} + +static struct bpf_dtab_netdev *devmap_get_next_obj(struct xdp_buff *xdp, struct bpf_map *map, + struct bpf_map *ex_map, u32 *key, + u32 *next_key, int ex_ifindex) +{ + struct bpf_dtab_netdev *obj; + struct net_device *dev; + u32 *tmp_key = key; + int err; + + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + return NULL; + + for (;;) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + obj = __dev_map_lookup_elem(map, *next_key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + obj = __dev_map_hash_lookup_elem(map, *next_key); + break; + default: + break; + } + + if (!obj || dev_in_exclude_map(obj, ex_map, ex_ifindex)) + goto find_next; + + dev = obj->dev; + + if (!dev->netdev_ops->ndo_xdp_xmit) + goto find_next; + + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + if (unlikely(err)) + goto find_next; + + return obj; + +find_next: + tmp_key = next_key; + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + break; + } + + return NULL; +} + +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + struct bpf_dtab_netdev *obj = NULL, *next_obj = NULL; + struct xdp_frame *xdpf, *nxdpf; + bool last_one = false; + int ex_ifindex; + u32 key, next_key; + + ex_ifindex = flags & BPF_F_EXCLUDE_INGRESS ? dev_rx->ifindex : 0; + + /* Find first available obj */ + obj = devmap_get_next_obj(xdp, map, ex_map, NULL, &key, ex_ifindex); + if (!obj) + return 0; + + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) + return -EOVERFLOW; + + for (;;) { + /* Check if we still have one more available obj */ + next_obj = devmap_get_next_obj(xdp, map, ex_map, &key, + &next_key, ex_ifindex); + if (!next_obj) + last_one = true; + + if (last_one) { + bq_enqueue(obj->dev, xdpf, dev_rx); + return 0; + } + + nxdpf = xdpf_clone(xdpf); + if (unlikely(!nxdpf)) { + xdp_return_frame_rx_napi(xdpf); + return -ENOMEM; + } + + bq_enqueue(obj->dev, nxdpf, dev_rx); + + /* Deal with next obj */ + obj = next_obj; + key = next_key; + } +} + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index b608185e1ffd..ceaf28ec111a 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4110,6 +4110,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: if (func_id != BPF_FUNC_redirect_map && + func_id != BPF_FUNC_redirect_map_multi && func_id != BPF_FUNC_map_lookup_elem) goto error; break; @@ -4202,6 +4203,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, map->map_type != BPF_MAP_TYPE_XSKMAP) goto error; break; + case BPF_FUNC_redirect_map_multi: + if (map->map_type != BPF_MAP_TYPE_DEVMAP && + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + goto error; + break; case BPF_FUNC_sk_redirect_map: case BPF_FUNC_msg_redirect_map: case BPF_FUNC_sock_map_update: diff --git a/net/core/filter.c b/net/core/filter.c index ddcc0d6209e1..673d12a051ef 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3515,12 +3515,19 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { }; static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) + struct bpf_map *map, struct xdp_buff *xdp, + struct bpf_map *ex_map, u32 flags) { switch (map->map_type) { case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: - return dev_map_enqueue(fwd, xdp, dev_rx); + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) + return dev_map_enqueue(fwd, xdp, dev_rx); + else + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, flags); case BPF_MAP_TYPE_CPUMAP: return cpu_map_enqueue(fwd, xdp, dev_rx); case BPF_MAP_TYPE_XSKMAP: @@ -3577,12 +3584,14 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); struct bpf_map *map = READ_ONCE(ri->map); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (unlikely(!map)) { @@ -3594,7 +3603,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = dev_xdp_enqueue(fwd, xdp, dev); } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, ri->flags); } if (unlikely(err)) @@ -3608,6 +3617,55 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, } EXPORT_SYMBOL_GPL(xdp_do_redirect); +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, + struct bpf_prog *xdp_prog, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) + +{ + struct bpf_dtab_netdev *dst; + struct sk_buff *nskb; + bool exclude_ingress; + u32 key, next_key; + void *fwd; + int err; + + /* Get first key from forward map */ + err = map->ops->map_get_next_key(map, NULL, &key); + if (err) + return err; + + exclude_ingress = !!(flags & BPF_F_EXCLUDE_INGRESS); + + for (;;) { + fwd = __xdp_map_lookup_elem(map, key); + if (fwd) { + dst = (struct bpf_dtab_netdev *)fwd; + if (dev_in_exclude_map(dst, ex_map, + exclude_ingress ? dev->ifindex : 0)) + goto find_next; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -ENOMEM; + + /* Try forword next one no mater the current forward + * succeed or not */ + dev_map_generic_redirect(dst, nskb, xdp_prog); + } + +find_next: + err = map->ops->map_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + consume_skb(skb); + return 0; +} + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, @@ -3615,19 +3673,30 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, struct bpf_map *map) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err = 0; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (map->map_type == BPF_MAP_TYPE_DEVMAP || map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { - struct bpf_dtab_netdev *dst = fwd; + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) { + struct bpf_dtab_netdev *dst = fwd; + + err = dev_map_generic_redirect(dst, skb, xdp_prog); + } else { + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, + ex_map, ri->flags); + } - err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { @@ -3741,6 +3810,34 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { .arg3_type = ARG_ANYTHING, }; +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, + struct bpf_map *, ex_map, u64, flags) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + + if (unlikely(!map || flags > BPF_F_EXCLUDE_INGRESS)) + return XDP_ABORTED; + + ri->tgt_index = 0; + /* Set the tgt_value to NULL to distinguish with bpf_xdp_redirect_map */ + ri->tgt_value = NULL; + ri->flags = flags; + ri->ex_map = ex_map; + + WRITE_ONCE(ri->map, map); + + return XDP_REDIRECT; +} + +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { + .func = bpf_xdp_redirect_map_multi, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_CONST_MAP_PTR, + .arg2_type = ARG_CONST_MAP_PTR, + .arg3_type = ARG_ANYTHING, +}; + static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, unsigned long off, unsigned long len) { @@ -6464,6 +6561,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_redirect_proto; case BPF_FUNC_redirect_map: return &bpf_xdp_redirect_map_proto; + case BPF_FUNC_redirect_map_multi: + return &bpf_xdp_redirect_map_multi_proto; case BPF_FUNC_xdp_adjust_tail: return &bpf_xdp_adjust_tail_proto; case BPF_FUNC_fib_lookup: diff --git a/net/core/xdp.c b/net/core/xdp.c index 3c45f99e26d5..9b43d0a208a7 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -476,3 +476,32 @@ void xdp_warn(const char *msg, const char *func, const int line) WARN(1, "XDP_WARN: %s(line:%d): %s\n", func, line, msg); }; EXPORT_SYMBOL_GPL(xdp_warn); + +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) +{ + unsigned int headroom, totalsize; + struct xdp_frame *nxdpf; + struct page *page; + void *addr; + + headroom = xdpf->headroom + sizeof(*xdpf); + totalsize = headroom + xdpf->len; + + if (unlikely(totalsize > PAGE_SIZE)) + return NULL; + page = dev_alloc_page(); + if (!page) + return NULL; + addr = page_to_virt(page); + + memcpy(addr, xdpf, totalsize); + + nxdpf = addr; + nxdpf->data = addr + headroom; + nxdpf->frame_sz = PAGE_SIZE; + nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0; + nxdpf->mem.id = 0; + + return nxdpf; +} +EXPORT_SYMBOL_GPL(xdpf_clone); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 548a749aebb3..a14e41309e73 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3319,6 +3319,22 @@ union bpf_attr { * A non-negative value equal to or less than *size* on success, * or a negative error in case of failure. * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3463,6 +3479,7 @@ union bpf_attr { FN(skc_to_tcp_request_sock), \ FN(skc_to_udp6_sock), \ FN(get_task_stack), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -3624,6 +3641,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv6 bpf-next 1/3] xdp: add a new helper for dev map multicast support 2020-07-09 1:30 ` [PATCHv6 bpf-next 1/3] " Hangbin Liu @ 2020-07-09 16:33 ` David Ahern 2020-07-10 6:55 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: David Ahern @ 2020-07-09 16:33 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On 7/8/20 7:30 PM, Hangbin Liu wrote: > This patch is for xdp multicast support. In this implementation we > add a new helper to accept two maps: forward map and exclude map. > We will redirect the packet to all the interfaces in *forward map*, but > exclude the interfaces that in *exclude map*. > good feature. I bet we could use this to create a simpler xdp dumper - redirect to an xdpmon device which converts to an skb and passes to any attached sockets. > diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c > index 10abb06065bb..617a51391971 100644 > --- a/kernel/bpf/devmap.c > +++ b/kernel/bpf/devmap.c > @@ -512,6 +512,160 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > return __xdp_enqueue(dev, xdp, dev_rx); > } > > +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ > +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) > +{ > + > + switch (map->map_type) { > + case BPF_MAP_TYPE_DEVMAP: > + return dev_map_get_next_key(map, key, next_key); > + case BPF_MAP_TYPE_DEVMAP_HASH: > + return dev_map_hash_get_next_key(map, key, next_key); > + default: > + break; > + } > + > + return -ENOENT; > +} > + > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, > + int exclude_ifindex) > +{ > + struct bpf_dtab_netdev *ex_obj = NULL; > + u32 key, next_key; > + int err; > + > + if (obj->dev->ifindex == exclude_ifindex) > + return true; > + > + if (!map) > + return false; > + > + err = devmap_get_next_key(map, NULL, &key); > + if (err) > + return false; > + > + for (;;) { > + switch (map->map_type) { > + case BPF_MAP_TYPE_DEVMAP: > + ex_obj = __dev_map_lookup_elem(map, key); > + break; > + case BPF_MAP_TYPE_DEVMAP_HASH: > + ex_obj = __dev_map_hash_lookup_elem(map, key); > + break; > + default: > + break; > + } > + > + if (ex_obj && ex_obj->dev->ifindex == obj->dev->ifindex) I'm probably missing something fundamental, but why do you need to walk the keys? Why not just do a lookup on the device index? > + return true; > + > + err = devmap_get_next_key(map, &key, &next_key); > + if (err) > + break; > + > + key = next_key; > + } > + > + return false; > +} > + > @@ -3741,6 +3810,34 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { > .arg3_type = ARG_ANYTHING, > }; > > +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, > + struct bpf_map *, ex_map, u64, flags) > +{ > + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); > + > + if (unlikely(!map || flags > BPF_F_EXCLUDE_INGRESS)) If flags is a bitfield, the check should be: flags & ~BPF_F_EXCLUDE_INGRESS ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv6 bpf-next 1/3] xdp: add a new helper for dev map multicast support 2020-07-09 16:33 ` David Ahern @ 2020-07-10 6:55 ` Hangbin Liu 2020-07-10 13:46 ` David Ahern 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-07-10 6:55 UTC (permalink / raw) To: David Ahern Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi Hi David, On Thu, Jul 09, 2020 at 10:33:38AM -0600, David Ahern wrote: > > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, > > + int exclude_ifindex) > > +{ > > + struct bpf_dtab_netdev *ex_obj = NULL; > > + u32 key, next_key; > > + int err; > > + > > + if (obj->dev->ifindex == exclude_ifindex) > > + return true; > > + > > + if (!map) > > + return false; > > + > > + err = devmap_get_next_key(map, NULL, &key); > > + if (err) > > + return false; > > + > > + for (;;) { > > + switch (map->map_type) { > > + case BPF_MAP_TYPE_DEVMAP: > > + ex_obj = __dev_map_lookup_elem(map, key); > > + break; > > + case BPF_MAP_TYPE_DEVMAP_HASH: > > + ex_obj = __dev_map_hash_lookup_elem(map, key); > > + break; > > + default: > > + break; > > + } > > + > > + if (ex_obj && ex_obj->dev->ifindex == obj->dev->ifindex) > > I'm probably missing something fundamental, but why do you need to walk > the keys? Why not just do a lookup on the device index? This functions is to check if the device index is in exclude map. The device indexes are stored as values in the map. The user could store the values by any key number. There is no way to lookup the device index directly unless loop the map and check each values we stored. Is there a map feature which could get an exact value directly? > > +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, > > + struct bpf_map *, ex_map, u64, flags) > > +{ > > + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); > > + > > + if (unlikely(!map || flags > BPF_F_EXCLUDE_INGRESS)) > > If flags is a bitfield, the check should be: > flags & ~BPF_F_EXCLUDE_INGRESS Thanks for the tips, I will fix it. Cheers Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv6 bpf-next 1/3] xdp: add a new helper for dev map multicast support 2020-07-10 6:55 ` Hangbin Liu @ 2020-07-10 13:46 ` David Ahern 2020-07-11 0:26 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: David Ahern @ 2020-07-10 13:46 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On 7/10/20 12:55 AM, Hangbin Liu wrote: > Hi David, > On Thu, Jul 09, 2020 at 10:33:38AM -0600, David Ahern wrote: >>> +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, >>> + int exclude_ifindex) >>> +{ >>> + struct bpf_dtab_netdev *ex_obj = NULL; >>> + u32 key, next_key; >>> + int err; >>> + >>> + if (obj->dev->ifindex == exclude_ifindex) >>> + return true; >>> + >>> + if (!map) >>> + return false; >>> + >>> + err = devmap_get_next_key(map, NULL, &key); >>> + if (err) >>> + return false; >>> + >>> + for (;;) { >>> + switch (map->map_type) { >>> + case BPF_MAP_TYPE_DEVMAP: >>> + ex_obj = __dev_map_lookup_elem(map, key); >>> + break; >>> + case BPF_MAP_TYPE_DEVMAP_HASH: >>> + ex_obj = __dev_map_hash_lookup_elem(map, key); >>> + break; >>> + default: >>> + break; >>> + } >>> + >>> + if (ex_obj && ex_obj->dev->ifindex == obj->dev->ifindex) >> >> I'm probably missing something fundamental, but why do you need to walk >> the keys? Why not just do a lookup on the device index? > > This functions is to check if the device index is in exclude map. > > The device indexes are stored as values in the map. The user could store > the values by any key number. There is no way to lookup the device index > directly unless loop the map and check each values we stored. Right. The point of DEVMAP_HASH is to allow map management where key == device index (vs DEVMAP which for any non-trivial use case is going to require key != device index). You could require the exclude map to be DEVMAP_HASH and the key to be the index allowing you to do a direct lookup. Having to roam the entire map looking for a match does not scale and is going to have poor performance with increasing number of entries. XDP is targeted at performance with expert level of control, so constraints like this have to be part of the deal. ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv6 bpf-next 1/3] xdp: add a new helper for dev map multicast support 2020-07-10 13:46 ` David Ahern @ 2020-07-11 0:26 ` Hangbin Liu 2020-07-11 16:09 ` David Ahern 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-07-11 0:26 UTC (permalink / raw) To: David Ahern Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On Fri, Jul 10, 2020 at 07:46:18AM -0600, David Ahern wrote: > >> > >> I'm probably missing something fundamental, but why do you need to walk > >> the keys? Why not just do a lookup on the device index? > > > > This functions is to check if the device index is in exclude map. > > > > The device indexes are stored as values in the map. The user could store > > the values by any key number. There is no way to lookup the device index > > directly unless loop the map and check each values we stored. > > Right. > > The point of DEVMAP_HASH is to allow map management where key == device > index (vs DEVMAP which for any non-trivial use case is going to require > key != device index). You could require the exclude map to be > DEVMAP_HASH and the key to be the index allowing you to do a direct > lookup. Having to roam the entire map looking for a match does not scale > and is going to have poor performance with increasing number of entries. > XDP is targeted at performance with expert level of control, so > constraints like this have to be part of the deal. Yes, if we have this constraints the performance should have some improvement. Do you think we should do it right now or in later performance update patch. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv6 bpf-next 1/3] xdp: add a new helper for dev map multicast support 2020-07-11 0:26 ` Hangbin Liu @ 2020-07-11 16:09 ` David Ahern 0 siblings, 0 replies; 219+ messages in thread From: David Ahern @ 2020-07-11 16:09 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On 7/10/20 6:26 PM, Hangbin Liu wrote: >> >> The point of DEVMAP_HASH is to allow map management where key == device >> index (vs DEVMAP which for any non-trivial use case is going to require >> key != device index). You could require the exclude map to be >> DEVMAP_HASH and the key to be the index allowing you to do a direct >> lookup. Having to roam the entire map looking for a match does not scale >> and is going to have poor performance with increasing number of entries. >> XDP is targeted at performance with expert level of control, so >> constraints like this have to be part of the deal. > > Yes, if we have this constraints the performance should have some improvement. > > Do you think we should do it right now or in later performance update patch. > It needs to be in the same release as the initial patches. Easiest to include with the initial set, but a followup is fine if it happens before this dev cycle is over. ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv6 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test 2020-07-09 1:30 ` [PATCHv6 bpf-next 0/3] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-07-09 1:30 ` [PATCHv6 bpf-next 1/3] " Hangbin Liu @ 2020-07-09 1:30 ` Hangbin Liu 2020-07-09 22:40 ` Daniel Borkmann 2020-07-09 1:30 ` [PATCHv6 bpf-next 3/3] selftests/bpf: add xdp_redirect_multi test Hangbin Liu ` (2 subsequent siblings) 4 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-07-09 1:30 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu This is a sample for xdp multicast. In the sample we could forward all packets between given interfaces. v5: add a null_map as we have strict the arg2 to ARG_CONST_MAP_PTR. Move the testing part to bpf selftest in next patch. v4: no update. v3: add rxcnt map to show the packet transmit speed. v2: no update. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 57 ++++++++ samples/bpf/xdp_redirect_map_multi_user.c | 166 ++++++++++++++++++++++ 3 files changed, 226 insertions(+) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index f87ee02073ba..fddca6cb76b8 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map tprogs-y += per_socket_stats_example tprogs-y += xdp_redirect tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_map_multi tprogs-y += xdp_redirect_cpu tprogs-y += xdp_monitor tprogs-y += xdp_rxq_info @@ -97,6 +98,7 @@ test_map_in_map-objs := test_map_in_map_user.o per_socket_stats_example-objs := cookie_uid_helper_example.o xdp_redirect-objs := xdp_redirect_user.o xdp_redirect_map-objs := xdp_redirect_map_user.o +xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o xdp_redirect_cpu-objs := bpf_load.o xdp_redirect_cpu_user.o xdp_monitor-objs := bpf_load.o xdp_monitor_user.o xdp_rxq_info-objs := xdp_rxq_info_user.o @@ -156,6 +158,7 @@ always-y += tcp_tos_reflect_kern.o always-y += tcp_dumpstats_kern.o always-y += xdp_redirect_kern.o always-y += xdp_redirect_map_kern.o +always-y += xdp_redirect_map_multi_kern.o always-y += xdp_redirect_cpu_kern.o always-y += xdp_monitor_kern.o always-y += xdp_rxq_info_kern.o diff --git a/samples/bpf/xdp_redirect_map_multi_kern.c b/samples/bpf/xdp_redirect_map_multi_kern.c new file mode 100644 index 000000000000..cc7ebaedf55a --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_kern.c @@ -0,0 +1,57 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <uapi/linux/bpf.h> +#include <bpf/bpf_helpers.h> + +struct bpf_map_def SEC("maps") forward_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 256, +}; + +struct bpf_map_def SEC("maps") null_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 1, +}; + +struct bpf_map_def SEC("maps") rxcnt = { + .type = BPF_MAP_TYPE_PERCPU_ARRAY, + .key_size = sizeof(u32), + .value_size = sizeof(long), + .max_entries = 1, +}; + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + long *value; + u32 key = 0; + + /* count packet in global counter */ + value = bpf_map_lookup_elem(&rxcnt, &key); + if (value) + *value += 1; + + return bpf_redirect_map_multi(&forward_map, &null_map, + BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_dummy") +int xdp_pass(struct xdp_md *ctx) +{ + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/samples/bpf/xdp_redirect_map_multi_user.c b/samples/bpf/xdp_redirect_map_multi_user.c new file mode 100644 index 000000000000..49f44c91b672 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_user.c @@ -0,0 +1,166 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; +static int rxcnt; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void poll_stats(int interval) +{ + unsigned int nr_cpus = bpf_num_possible_cpus(); + __u64 values[nr_cpus], prev[nr_cpus]; + + memset(prev, 0, sizeof(prev)); + + while (1) { + __u64 sum = 0; + __u32 key = 0; + int i; + + sleep(interval); + assert(bpf_map_lookup_elem(rxcnt, &key, values) == 0); + for (i = 0; i < nr_cpus; i++) + sum += (values[i] - prev[i]); + if (sum) + printf("Forwarding %10llu pkt/s\n", sum / interval); + memcpy(prev, values, sizeof(values)); + } +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n", + prog); +} + +int main(int argc, char **argv) +{ + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + int prog_fd, forward_map; + int i, ret, opt, ifindex; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNF")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) + xdp_flags |= XDP_FLAGS_DRV_MODE; + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + return 1; + + forward_map = bpf_object__find_map_fd_by_name(obj, "forward_map"); + rxcnt = bpf_object__find_map_fd_by_name(obj, "rxcnt"); + + if (forward_map < 0 || rxcnt < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + return 1; + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* Add all the interfaces to group all */ + ret = bpf_map_update_elem(forward_map, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + } + + poll_stats(2); + + return 0; + +err_out: + return 1; +} -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv6 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test 2020-07-09 1:30 ` [PATCHv6 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu @ 2020-07-09 22:40 ` Daniel Borkmann 2020-07-10 6:41 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: Daniel Borkmann @ 2020-07-09 22:40 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Lorenzo Bianconi On 7/9/20 3:30 AM, Hangbin Liu wrote: > This is a sample for xdp multicast. In the sample we could forward all > packets between given interfaces. > > v5: add a null_map as we have strict the arg2 to ARG_CONST_MAP_PTR. > Move the testing part to bpf selftest in next patch. > v4: no update. > v3: add rxcnt map to show the packet transmit speed. > v2: no update. > > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > --- > samples/bpf/Makefile | 3 + > samples/bpf/xdp_redirect_map_multi_kern.c | 57 ++++++++ > samples/bpf/xdp_redirect_map_multi_user.c | 166 ++++++++++++++++++++++ > 3 files changed, 226 insertions(+) > create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c > create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c > > diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile > index f87ee02073ba..fddca6cb76b8 100644 > --- a/samples/bpf/Makefile > +++ b/samples/bpf/Makefile > @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map > tprogs-y += per_socket_stats_example > tprogs-y += xdp_redirect > tprogs-y += xdp_redirect_map > +tprogs-y += xdp_redirect_map_multi > tprogs-y += xdp_redirect_cpu > tprogs-y += xdp_monitor > tprogs-y += xdp_rxq_info > @@ -97,6 +98,7 @@ test_map_in_map-objs := test_map_in_map_user.o > per_socket_stats_example-objs := cookie_uid_helper_example.o > xdp_redirect-objs := xdp_redirect_user.o > xdp_redirect_map-objs := xdp_redirect_map_user.o > +xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o > xdp_redirect_cpu-objs := bpf_load.o xdp_redirect_cpu_user.o > xdp_monitor-objs := bpf_load.o xdp_monitor_user.o > xdp_rxq_info-objs := xdp_rxq_info_user.o > @@ -156,6 +158,7 @@ always-y += tcp_tos_reflect_kern.o > always-y += tcp_dumpstats_kern.o > always-y += xdp_redirect_kern.o > always-y += xdp_redirect_map_kern.o > +always-y += xdp_redirect_map_multi_kern.o > always-y += xdp_redirect_cpu_kern.o > always-y += xdp_monitor_kern.o > always-y += xdp_rxq_info_kern.o > diff --git a/samples/bpf/xdp_redirect_map_multi_kern.c b/samples/bpf/xdp_redirect_map_multi_kern.c > new file mode 100644 > index 000000000000..cc7ebaedf55a > --- /dev/null > +++ b/samples/bpf/xdp_redirect_map_multi_kern.c > @@ -0,0 +1,57 @@ > +/* SPDX-License-Identifier: GPL-2.0 > + * > + * modify it under the terms of version 2 of the GNU General Public > + * License as published by the Free Software Foundation. > + * > + * This program is distributed in the hope that it will be useful, but > + * WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + * General Public License for more details. > + */ > +#define KBUILD_MODNAME "foo" > +#include <uapi/linux/bpf.h> > +#include <bpf/bpf_helpers.h> > + > +struct bpf_map_def SEC("maps") forward_map = { > + .type = BPF_MAP_TYPE_DEVMAP_HASH, > + .key_size = sizeof(u32), > + .value_size = sizeof(int), > + .max_entries = 256, > +}; > + > +struct bpf_map_def SEC("maps") null_map = { > + .type = BPF_MAP_TYPE_DEVMAP_HASH, > + .key_size = sizeof(u32), > + .value_size = sizeof(int), > + .max_entries = 1, > +}; > + > +struct bpf_map_def SEC("maps") rxcnt = { > + .type = BPF_MAP_TYPE_PERCPU_ARRAY, > + .key_size = sizeof(u32), > + .value_size = sizeof(long), > + .max_entries = 1, > +}; > + > +SEC("xdp_redirect_map_multi") > +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) > +{ > + long *value; > + u32 key = 0; > + > + /* count packet in global counter */ > + value = bpf_map_lookup_elem(&rxcnt, &key); > + if (value) > + *value += 1; > + > + return bpf_redirect_map_multi(&forward_map, &null_map, > + BPF_F_EXCLUDE_INGRESS); Why not extending to allow use-case like ... return bpf_redirect_map_multi(&fwd_map, NULL, BPF_F_EXCLUDE_INGRESS); ... instead of requiring a dummy/'null' map? Thanks, Daniel ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv6 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test 2020-07-09 22:40 ` Daniel Borkmann @ 2020-07-10 6:41 ` Hangbin Liu 2020-07-10 14:32 ` Daniel Borkmann 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-07-10 6:41 UTC (permalink / raw) To: Daniel Borkmann Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Lorenzo Bianconi On Fri, Jul 10, 2020 at 12:40:11AM +0200, Daniel Borkmann wrote: > > +SEC("xdp_redirect_map_multi") > > +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) > > +{ > > + long *value; > > + u32 key = 0; > > + > > + /* count packet in global counter */ > > + value = bpf_map_lookup_elem(&rxcnt, &key); > > + if (value) > > + *value += 1; > > + > > + return bpf_redirect_map_multi(&forward_map, &null_map, > > + BPF_F_EXCLUDE_INGRESS); > > Why not extending to allow use-case like ... > > return bpf_redirect_map_multi(&fwd_map, NULL, BPF_F_EXCLUDE_INGRESS); > > ... instead of requiring a dummy/'null' map? > I planed to let user set NULL, but the arg2_type is ARG_CONST_MAP_PTR, which not allow NULL pointer. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv6 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test 2020-07-10 6:41 ` Hangbin Liu @ 2020-07-10 14:32 ` Daniel Borkmann 2020-07-11 0:21 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: Daniel Borkmann @ 2020-07-10 14:32 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Lorenzo Bianconi On 7/10/20 8:41 AM, Hangbin Liu wrote: > On Fri, Jul 10, 2020 at 12:40:11AM +0200, Daniel Borkmann wrote: >>> +SEC("xdp_redirect_map_multi") >>> +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) >>> +{ >>> + long *value; >>> + u32 key = 0; >>> + >>> + /* count packet in global counter */ >>> + value = bpf_map_lookup_elem(&rxcnt, &key); >>> + if (value) >>> + *value += 1; >>> + >>> + return bpf_redirect_map_multi(&forward_map, &null_map, >>> + BPF_F_EXCLUDE_INGRESS); >> >> Why not extending to allow use-case like ... >> >> return bpf_redirect_map_multi(&fwd_map, NULL, BPF_F_EXCLUDE_INGRESS); >> >> ... instead of requiring a dummy/'null' map? > > I planed to let user set NULL, but the arg2_type is ARG_CONST_MAP_PTR, which > not allow NULL pointer. Right, but then why not adding a new type ARG_CONST_MAP_PTR_OR_NULL ? ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv6 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test 2020-07-10 14:32 ` Daniel Borkmann @ 2020-07-11 0:21 ` Hangbin Liu 0 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-07-11 0:21 UTC (permalink / raw) To: Daniel Borkmann Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Lorenzo Bianconi On Fri, Jul 10, 2020 at 04:32:35PM +0200, Daniel Borkmann wrote: > On 7/10/20 8:41 AM, Hangbin Liu wrote: > > On Fri, Jul 10, 2020 at 12:40:11AM +0200, Daniel Borkmann wrote: > > > > +SEC("xdp_redirect_map_multi") > > > > +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) > > > > +{ > > > > + long *value; > > > > + u32 key = 0; > > > > + > > > > + /* count packet in global counter */ > > > > + value = bpf_map_lookup_elem(&rxcnt, &key); > > > > + if (value) > > > > + *value += 1; > > > > + > > > > + return bpf_redirect_map_multi(&forward_map, &null_map, > > > > + BPF_F_EXCLUDE_INGRESS); > > > > > > Why not extending to allow use-case like ... > > > > > > return bpf_redirect_map_multi(&fwd_map, NULL, BPF_F_EXCLUDE_INGRESS); > > > > > > ... instead of requiring a dummy/'null' map? > > > > I planed to let user set NULL, but the arg2_type is ARG_CONST_MAP_PTR, which > > not allow NULL pointer. > > Right, but then why not adding a new type ARG_CONST_MAP_PTR_OR_NULL ? Yes, that's what I plan for next step. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv6 bpf-next 3/3] selftests/bpf: add xdp_redirect_multi test 2020-07-09 1:30 ` [PATCHv6 bpf-next 0/3] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-07-09 1:30 ` [PATCHv6 bpf-next 1/3] " Hangbin Liu 2020-07-09 1:30 ` [PATCHv6 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu @ 2020-07-09 1:30 ` Hangbin Liu 2020-07-09 22:37 ` [PATCHv6 bpf-next 0/3] xdp: add a new helper for dev map multicast support Daniel Borkmann 2020-07-14 6:32 ` [PATCHv7 " Hangbin Liu 4 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-07-09 1:30 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu Add a bpf selftest for new helper xdp_redirect_map_multi(). In this test we have 3 forward groups groups and 1 exclude group. The test will redirect each interface's packets to all the interfaces in the forward group, and exclude the interface in exclude map. We will also test both DEVMAP and DEVMAP_HASH with xdp generic and drv. For more test details, you can find it in the test script. Here is the test result. ]# ./test_xdp_redirect_multi.sh Pass: xdpgeneric arp ns1-2 Pass: xdpgeneric arp ns1-3 Pass: xdpgeneric arp ns1-4 Pass: xdpgeneric ping ns1-2 Pass: xdpgeneric ping ns1-3 Pass: xdpgeneric ping ns1-4 Pass: xdpgeneric ping6 ns2-1 Pass: xdpgeneric ping6 ns2-3 Pass: xdpgeneric ping6 ns2-4 Pass: xdpdrv arp ns1-2 Pass: xdpdrv arp ns1-3 Pass: xdpdrv arp ns1-4 Pass: xdpdrv ping ns1-2 Pass: xdpdrv ping ns1-3 Pass: xdpdrv ping ns1-4 Pass: xdpdrv ping6 ns2-1 Pass: xdpdrv ping6 ns2-3 Pass: xdpdrv ping6 ns2-4 Summary: PASS 18, FAIL 0 Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- tools/testing/selftests/bpf/Makefile | 4 +- .../bpf/progs/xdp_redirect_multi_kern.c | 90 +++++++++ .../selftests/bpf/test_xdp_redirect_multi.sh | 164 +++++++++++++++++ .../selftests/bpf/xdp_redirect_multi.c | 173 ++++++++++++++++++ 4 files changed, 430 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 1f9c696b3edf..66b857210814 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -51,6 +51,7 @@ TEST_FILES = test_lwt_ip_encap.o \ # Order correspond to 'make run_tests' order TEST_PROGS := test_kmod.sh \ test_xdp_redirect.sh \ + test_xdp_redirect_multi.sh \ test_xdp_meta.sh \ test_xdp_veth.sh \ test_offload.py \ @@ -79,7 +80,8 @@ TEST_PROGS_EXTENDED := with_addr.sh \ # Compile but not part of 'make run_tests' TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ - test_lirc_mode2_user xdping test_cpp runqslower bench + test_lirc_mode2_user xdping test_cpp runqslower bench \ + xdp_redirect_multi TEST_CUSTOM_PROGS = urandom_read diff --git a/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c new file mode 100644 index 000000000000..70b8476b9df3 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c @@ -0,0 +1,90 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <string.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ip.h> +#include <linux/ipv6.h> + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_endian.h> + +struct bpf_map_def SEC("maps") forward_map_v4 = { + .type = BPF_MAP_TYPE_DEVMAP, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 4096, +}; + +struct bpf_map_def SEC("maps") forward_map_v6 = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") forward_map_all = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") exclude_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") null_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 1, +}; + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + struct ethhdr *eth = data; + __u16 h_proto; + __u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + h_proto = eth->h_proto; + + if (h_proto == bpf_htons(ETH_P_IP)) + return bpf_redirect_map_multi(&forward_map_v4, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else if (h_proto == bpf_htons(ETH_P_IPV6)) + return bpf_redirect_map_multi(&forward_map_v6, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else + return bpf_redirect_map_multi(&forward_map_all, &null_map, + BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_dummy") +int xdp_pass(struct xdp_md *ctx) +{ + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh new file mode 100755 index 000000000000..f4f8f751854e --- /dev/null +++ b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh @@ -0,0 +1,164 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Test topology: +# - - - - - - - - - - - - - - - - - - - - - - - - - +# | veth1 veth2 veth3 veth4 | ... init net +# - -| - - - - - - | - - - - - - | - - - - - - | - - +# --------- --------- --------- --------- +# | veth0 | | veth0 | | veth0 | | veth0 | ... +# --------- --------- --------- --------- +# ns1 ns2 ns3 ns4 +# +# Forward multicast groups: +# Forward group all has interfaces: veth1, veth2, veth3, veth4, ... (All traffic except IPv4, IPv6) +# Forward group v4 has interfaces: veth1, veth3, veth4, ... (For IPv4 traffic only) +# Forward group v6 has interfaces: veth2, veth3, veth4, ... (For IPv6 traffic only) +# Exclude Groups: +# Exclude group: veth3 (assume ns3 is in black list) +# +# Test modules: +# XDP modes: generic, native +# map types: group v4 use DEVMAP, others use DEVMAP_HASH +# +# Test cases: +# ARP(we didn't block ARP for ns3): +# ns1 -> gw: ns2, ns3, ns4 should receive the arp request +# IPv4: +# ns1 -> ns2 (fail), ns1 -> ns3 (fail), ns1 -> ns4 (pass) +# IPv6 +# ns2 -> ns1 (fail), ns2 -> ns3 (fail), ns2 -> ns4 (pass) +# + + +# netns numbers +NUM=4 +IFACES="" +DRV_MODE="xdpgeneric xdpdrv" +PASS=0 +FAIL=0 + +test_pass() +{ + echo "Pass: $@" + PASS=$((PASS + 1)) +} + +test_fail() +{ + echo "fail: $@" + FAIL=$((FAIL + 1)) +} + +clean_up() +{ + for i in $(seq $NUM); do + ip link del veth$i 2> /dev/null + ip netns del ns$i 2> /dev/null + done + rm -f xdp_redirect_*.log arp_ns*.log +} + +# Kselftest framework requirement - SKIP code is 4. +check_env() +{ + ip link set dev lo xdpgeneric off &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without the ip xdpgeneric support" + exit 4 + fi + + which tcpdump &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without tcpdump" + exit 4 + fi +} + +setup_ns() +{ + local mode=$1 + IFACES="" + + for i in $(seq $NUM); do + ip netns add ns$i + ip link add veth$i type veth peer name veth0 netns ns$i + ip link set veth$i up + ip -n ns$i link set veth0 up + + ip -n ns$i addr add 192.0.2.$i/24 dev veth0 + ip -n ns$i addr add 2001:db8::$i/64 dev veth0 + ip -n ns$i link set veth0 $mode obj \ + xdp_redirect_multi_kern.o sec xdp_dummy &> /dev/null || \ + { test_fail "Unable to load dummy xdp" && exit 1; } + IFACES="$IFACES veth$i" + done +} + +do_ping_tests() +{ + local mode=$1 + + # arp test + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -i veth0 -nn -l -e &> arp_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-2_${mode}.log && \ + test_pass "$mode arp ns1-2" || test_fail "$mode arp ns1-2" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-3_${mode}.log && \ + test_pass "$mode arp ns1-3" || test_fail "$mode arp ns1-3" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-4_${mode}.log && \ + test_pass "$mode arp ns1-4" || test_fail "$mode arp ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-2" || test_pass "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-3" || test_pass "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" + + # ping6 test + ip netns exec ns2 ping6 2001:db8::1 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-1" || test_pass "$mode ping6 ns2-1" + ip netns exec ns2 ping6 2001:db8::3 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-3" || test_pass "$mode ping6 ns2-3" + ip netns exec ns2 ping6 2001:db8::4 -c 4 &> /dev/null && \ + test_pass "$mode ping6 ns2-4" || test_fail "$mode ping6 ns2-4" +} + +do_tests() +{ + local mode=$1 + local drv_p + + [ ${mode} == "xdpdrv" ] && drv_p="-N" || drv_p="-S" + + # run `ulimit -l unlimited` if you got errors like + # libbpf: Error in bpf_object__probe_global_data():Operation not permitted(1). + ./xdp_redirect_multi $drv_p $IFACES &> xdp_redirect_${mode}.log & + xdp_pid=$! + sleep 10 + + do_ping_tests $mode + + kill $xdp_pid +} + +trap clean_up 0 2 3 6 9 + +check_env + +for mode in ${DRV_MODE}; do + setup_ns $mode + do_tests $mode + sleep 10 + clean_up + sleep 5 +done + +echo "Summary: PASS $PASS, FAIL $FAIL" +[ $FAIL -eq 0 ] && exit 0 || exit 1 diff --git a/tools/testing/selftests/bpf/xdp_redirect_multi.c b/tools/testing/selftests/bpf/xdp_redirect_multi.c new file mode 100644 index 000000000000..5626005cb679 --- /dev/null +++ b/tools/testing/selftests/bpf/xdp_redirect_multi.c @@ -0,0 +1,173 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n", + prog); +} + +int main(int argc, char **argv) +{ + int prog_fd, group_all, group_v4, group_v6, exclude; + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + int i, ret, opt, ifindex; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNF")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) + xdp_flags |= XDP_FLAGS_DRV_MODE; + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + return 1; + + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_all"); + group_v4 = bpf_object__find_map_fd_by_name(obj, "forward_map_v4"); + group_v6 = bpf_object__find_map_fd_by_name(obj, "forward_map_v6"); + exclude = bpf_object__find_map_fd_by_name(obj, "exclude_map"); + + if (group_all < 0 || group_v4 < 0 || group_v6 < 0 || exclude < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + return 1; + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* Add all the interfaces to group all */ + ret = bpf_map_update_elem(group_all, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* For testing: remove the 1st interfaces from group v6 */ + if (i != 0) { + ret = bpf_map_update_elem(group_v6, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: remove the 2nd interfaces from group v4 */ + if (i != 1) { + ret = bpf_map_update_elem(group_v4, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: add the 3rd interfaces to exclude map */ + if (i == 2) { + ret = bpf_map_update_elem(exclude, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + } + + /* sleep some time for testing */ + sleep(999); + + return 0; + +err_out: + return 1; +} -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv6 bpf-next 0/3] xdp: add a new helper for dev map multicast support 2020-07-09 1:30 ` [PATCHv6 bpf-next 0/3] xdp: add a new helper for dev map multicast support Hangbin Liu ` (2 preceding siblings ...) 2020-07-09 1:30 ` [PATCHv6 bpf-next 3/3] selftests/bpf: add xdp_redirect_multi test Hangbin Liu @ 2020-07-09 22:37 ` Daniel Borkmann 2020-07-10 7:36 ` Hangbin Liu 2020-07-14 6:32 ` [PATCHv7 " Hangbin Liu 4 siblings, 1 reply; 219+ messages in thread From: Daniel Borkmann @ 2020-07-09 22:37 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Lorenzo Bianconi On 7/9/20 3:30 AM, Hangbin Liu wrote: > This patch is for xdp multicast support. which has been discussed before[0], > The goal is to be able to implement an OVS-like data plane in XDP, i.e., > a software switch that can forward XDP frames to multiple ports. > > To achieve this, an application needs to specify a group of interfaces > to forward a packet to. It is also common to want to exclude one or more > physical interfaces from the forwarding operation - e.g., to forward a > packet to all interfaces in the multicast group except the interface it > arrived on. While this could be done simply by adding more groups, this > quickly leads to a combinatorial explosion in the number of groups an > application has to maintain. > > To avoid the combinatorial explosion, we propose to include the ability > to specify an "exclude group" as part of the forwarding operation. This > needs to be a group (instead of just a single port index), because a > physical interface can be part of a logical grouping, such as a bond > device. > > Thus, the logical forwarding operation becomes a "set difference" > operation, i.e. "forward to all ports in group A that are not also in > group B". This series implements such an operation using device maps to > represent the groups. This means that the XDP program specifies two > device maps, one containing the list of netdevs to redirect to, and the > other containing the exclude list. Could you move this description as part of patch 1/3 instead of cover letter? Mostly given this helps understanding the rationale wrt exclusion map which is otherwise lacking from just looking at the patch itself. Assuming you have a bond, how does this look in practice for your mentioned ovs-like data plane in XDP? The map for 'group A' is shared among all XDP progs and the map for 'group B' is managed per prog? The BPF_F_EXCLUDE_INGRESS is clear, but how would this look wrt forwarding from a phys dev /to/ the bond iface w/ XDP? Also, what about tc BPF helper support for the case where not every device might have native XDP (but they could still share the maps)? > To achieve this, I re-implement a new helper bpf_redirect_map_multi() > to accept two maps, the forwarding map and exclude map. If user > don't want to use exclude map and just want simply stop redirecting back > to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. > > The 2nd and 3rd patches are for usage sample and testing purpose, so there > is no effort has been made on performance optimisation. I did same tests > with pktgen(pkt size 64) to compire with xdp_redirect_map(). Here is the > test result(the veth peer has a dummy xdp program with XDP_DROP directly): > > Version | Test | Native | Generic > 5.8 rc1 | xdp_redirect_map i40e->i40e | 10.0M | 1.9M > 5.8 rc1 | xdp_redirect_map i40e->veth | 12.7M | 1.6M > 5.8 rc1 + patch | xdp_redirect_map i40e->i40e | 10.0M | 1.9M > 5.8 rc1 + patch | xdp_redirect_map i40e->veth | 12.3M | 1.6M > 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e | 7.2M | 1.5M > 5.8 rc1 + patch | xdp_redirect_map_multi i40e->veth | 8.5M | 1.3M > 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e+veth | 3.0M | 0.98M > > The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop > the arrays and do clone skb/xdpf. The native path is slower than generic > path as we send skbs by pktgen. So the result looks reasonable. > > Last but not least, thanks a lot to Jiri, Eelco, Toke and Jesper for > suggestions and help on implementation. > > [0] https://xdp-project.net/#Handling-multicast > > v6: converted helper return types from int to long > > v5: > a) Check devmap_get_next_key() return value. > b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. > c) In function dev_map_enqueue_multi(), consume xdpf for the last > obj instead of the first on. > d) Update helper description and code comments to explain that we > use NULL target value to distinguish multicast and unicast > forwarding. > e) Update memory model, memory id and frame_sz in xdpf_clone(). > f) Split the tests from sample and add a bpf kernel selftest patch. > > v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo > > v3: Based on Toke's suggestion, do the following update > a) Update bpf_redirect_map_multi() description in bpf.h. > b) Fix exclude_ifindex checking order in dev_in_exclude_map(). > c) Fix one more xdpf clone in dev_map_enqueue_multi(). > d) Go find next one in dev_map_enqueue_multi() if the interface is not > able to forward instead of abort the whole loop. > e) Remove READ_ONCE/WRITE_ONCE for ex_map. > > v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept > include/exclude maps directly. > > Hangbin Liu (3): > xdp: add a new helper for dev map multicast support > sample/bpf: add xdp_redirect_map_multicast test > selftests/bpf: add xdp_redirect_multi test > > include/linux/bpf.h | 20 ++ > include/linux/filter.h | 1 + > include/net/xdp.h | 1 + > include/uapi/linux/bpf.h | 22 +++ > kernel/bpf/devmap.c | 154 ++++++++++++++++ > kernel/bpf/verifier.c | 6 + > net/core/filter.c | 109 ++++++++++- > net/core/xdp.c | 29 +++ > samples/bpf/Makefile | 3 + > samples/bpf/xdp_redirect_map_multi_kern.c | 57 ++++++ > samples/bpf/xdp_redirect_map_multi_user.c | 166 +++++++++++++++++ > tools/include/uapi/linux/bpf.h | 22 +++ > tools/testing/selftests/bpf/Makefile | 4 +- > .../bpf/progs/xdp_redirect_multi_kern.c | 90 +++++++++ > .../selftests/bpf/test_xdp_redirect_multi.sh | 164 +++++++++++++++++ > .../selftests/bpf/xdp_redirect_multi.c | 173 ++++++++++++++++++ > 16 files changed, 1015 insertions(+), 6 deletions(-) > create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c > create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c > create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c > create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh > create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c > ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv6 bpf-next 0/3] xdp: add a new helper for dev map multicast support 2020-07-09 22:37 ` [PATCHv6 bpf-next 0/3] xdp: add a new helper for dev map multicast support Daniel Borkmann @ 2020-07-10 7:36 ` Hangbin Liu 2020-07-10 15:02 ` Daniel Borkmann 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-07-10 7:36 UTC (permalink / raw) To: Daniel Borkmann Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Lorenzo Bianconi On Fri, Jul 10, 2020 at 12:37:59AM +0200, Daniel Borkmann wrote: > On 7/9/20 3:30 AM, Hangbin Liu wrote: > > This patch is for xdp multicast support. which has been discussed before[0], > > The goal is to be able to implement an OVS-like data plane in XDP, i.e., > > a software switch that can forward XDP frames to multiple ports. > > > > To achieve this, an application needs to specify a group of interfaces > > to forward a packet to. It is also common to want to exclude one or more > > physical interfaces from the forwarding operation - e.g., to forward a > > packet to all interfaces in the multicast group except the interface it > > arrived on. While this could be done simply by adding more groups, this > > quickly leads to a combinatorial explosion in the number of groups an > > application has to maintain. > > > > To avoid the combinatorial explosion, we propose to include the ability > > to specify an "exclude group" as part of the forwarding operation. This > > needs to be a group (instead of just a single port index), because a > > physical interface can be part of a logical grouping, such as a bond > > device. > > > > Thus, the logical forwarding operation becomes a "set difference" > > operation, i.e. "forward to all ports in group A that are not also in > > group B". This series implements such an operation using device maps to > > represent the groups. This means that the XDP program specifies two > > device maps, one containing the list of netdevs to redirect to, and the > > other containing the exclude list. > > Could you move this description as part of patch 1/3 instead of cover > letter? Mostly given this helps understanding the rationale wrt exclusion > map which is otherwise lacking from just looking at the patch itself. OK, I will > > Assuming you have a bond, how does this look in practice for your mentioned > ovs-like data plane in XDP? The map for 'group A' is shared among all XDP > progs and the map for 'group B' is managed per prog? The BPF_F_EXCLUDE_INGRESS Yes, kind of. Since we have two maps as parameter. The 'group A map'(include map) will be shared between the interfaces in same group/vlan. The 'group B map' (exclude map) is interface specific. Each interface will hold it's own exclude map. As most time each interface only exclude itself, a null map + BPF_F_EXCLUDE_INGRESS should be enough. For bond situation. e.g. A active-backup bond0 with eth1 + eth2 as slaves. If eth1 is active interface, we can add eth2 to the exclude map. > is clear, but how would this look wrt forwarding from a phys dev /to/ the > bond iface w/ XDP? As bond interface doesn't support native XDP, This forwarding only works for physical slave interfaces. For generic xdp, maybe we can forward to bond interface directly, but I haven't tried. > > Also, what about tc BPF helper support for the case where not every device > might have native XDP (but they could still share the maps)? I haven't tried tc BPF. This helper works for both generic and native xdp forwarding. I think it should also works if we load the prog with native xdp mode in one interface and generic xdp mode in another interface, couldn't we? Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv6 bpf-next 0/3] xdp: add a new helper for dev map multicast support 2020-07-10 7:36 ` Hangbin Liu @ 2020-07-10 15:02 ` Daniel Borkmann 2020-07-10 16:52 ` David Ahern 0 siblings, 1 reply; 219+ messages in thread From: Daniel Borkmann @ 2020-07-10 15:02 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Lorenzo Bianconi On 7/10/20 9:36 AM, Hangbin Liu wrote: > On Fri, Jul 10, 2020 at 12:37:59AM +0200, Daniel Borkmann wrote: >> On 7/9/20 3:30 AM, Hangbin Liu wrote: >>> This patch is for xdp multicast support. which has been discussed before[0], >>> The goal is to be able to implement an OVS-like data plane in XDP, i.e., >>> a software switch that can forward XDP frames to multiple ports. >>> >>> To achieve this, an application needs to specify a group of interfaces >>> to forward a packet to. It is also common to want to exclude one or more >>> physical interfaces from the forwarding operation - e.g., to forward a >>> packet to all interfaces in the multicast group except the interface it >>> arrived on. While this could be done simply by adding more groups, this >>> quickly leads to a combinatorial explosion in the number of groups an >>> application has to maintain. >>> >>> To avoid the combinatorial explosion, we propose to include the ability >>> to specify an "exclude group" as part of the forwarding operation. This >>> needs to be a group (instead of just a single port index), because a >>> physical interface can be part of a logical grouping, such as a bond >>> device. >>> >>> Thus, the logical forwarding operation becomes a "set difference" >>> operation, i.e. "forward to all ports in group A that are not also in >>> group B". This series implements such an operation using device maps to >>> represent the groups. This means that the XDP program specifies two >>> device maps, one containing the list of netdevs to redirect to, and the >>> other containing the exclude list. >> >> Could you move this description as part of patch 1/3 instead of cover >> letter? Mostly given this helps understanding the rationale wrt exclusion >> map which is otherwise lacking from just looking at the patch itself. > > OK, I will > >> Assuming you have a bond, how does this look in practice for your mentioned >> ovs-like data plane in XDP? The map for 'group A' is shared among all XDP >> progs and the map for 'group B' is managed per prog? The BPF_F_EXCLUDE_INGRESS > > Yes, kind of. Since we have two maps as parameter. The 'group A map'(include map) > will be shared between the interfaces in same group/vlan. The 'group B map' > (exclude map) is interface specific. Each interface will hold it's own exclude map. > > As most time each interface only exclude itself, a null map + BPF_F_EXCLUDE_INGRESS > should be enough. > > For bond situation. e.g. A active-backup bond0 with eth1 + eth2 as slaves. > If eth1 is active interface, we can add eth2 to the exclude map. Right, but what about the other direction where one device forwards to a bond, presumably eth1 + eth2 are in the include map and shared also between other ifaces? Given the logic for the bond mode is on bond0, so one layer higher, how do you determine which of eth1 + eth2 to send to in the BPF prog? Daemon listening for link events via arp or mii monitor and then update include map? Ideally would be nice to have some sort of a bond0 pass-through for the XDP buffer so it ends up eventually at one of the two through the native logic, e.g. what do you do when it's configured in xor mode or when slave dev is selected via hash or some other user logic (e.g. via team driver); how would this be modeled via inclusion map? I guess the issue can be regarded independently to this set, but given you mention explicitly bond here as a use case for the exclusion map, I was wondering how you solve the inclusion one for bond devices for your data plane? >> is clear, but how would this look wrt forwarding from a phys dev /to/ the >> bond iface w/ XDP? > > As bond interface doesn't support native XDP, This forwarding only works for > physical slave interfaces. > > For generic xdp, maybe we can forward to bond interface directly, but I > haven't tried. > >> Also, what about tc BPF helper support for the case where not every device >> might have native XDP (but they could still share the maps)? > > I haven't tried tc BPF. This helper works for both generic and native xdp > forwarding. I think it should also works if we load the prog with native > xdp mode in one interface and generic xdp mode in another interface, couldn't > we? Yes, that would work though generic XDP comes with its own set of issues, but presumably this sort of traffic could be considered slow-path anyway. Thanks, Daniel ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv6 bpf-next 0/3] xdp: add a new helper for dev map multicast support 2020-07-10 15:02 ` Daniel Borkmann @ 2020-07-10 16:52 ` David Ahern 0 siblings, 0 replies; 219+ messages in thread From: David Ahern @ 2020-07-10 16:52 UTC (permalink / raw) To: Daniel Borkmann, Hangbin Liu Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Lorenzo Bianconi On 7/10/20 9:02 AM, Daniel Borkmann wrote: > Right, but what about the other direction where one device forwards to a > bond, > presumably eth1 + eth2 are in the include map and shared also between other > ifaces? Given the logic for the bond mode is on bond0, so one layer > higher, how > do you determine which of eth1 + eth2 to send to in the BPF prog? Daemon > listening > for link events via arp or mii monitor and then update include map? > Ideally would > be nice to have some sort of a bond0 pass-through for the XDP buffer so > it ends > up eventually at one of the two through the native logic, e.g. what do > you do when > it's configured in xor mode or when slave dev is selected via hash or > some other > user logic (e.g. via team driver); how would this be modeled via > inclusion map? I > guess the issue can be regarded independently to this set, but given you > mention > explicitly bond here as a use case for the exclusion map, I was > wondering how you > solve the inclusion one for bond devices for your data plane? bond driver does not support xdp_xmit, and I do not believe there is a good ROI for adapting it to handle xdp buffers. For round robin and active-backup modes it is straightforward to adapt the new ndo_get_xmit_slave to work with ebpf. That is not the case for any of them that use a hash on the skb. e.g., for L3+L4 hashing I found it easier to replicate the algorithm in bpf than trying to adapt the bond code to work with XDP buffers. I put that in the category of 'XDP is advanced networking that requires unraveling the generic for a specific deployment.' In short, for bonds and Tx the bpf program needs to pick the slave device. ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv7 bpf-next 0/3] xdp: add a new helper for dev map multicast support 2020-07-09 1:30 ` [PATCHv6 bpf-next 0/3] xdp: add a new helper for dev map multicast support Hangbin Liu ` (3 preceding siblings ...) 2020-07-09 22:37 ` [PATCHv6 bpf-next 0/3] xdp: add a new helper for dev map multicast support Daniel Borkmann @ 2020-07-14 6:32 ` Hangbin Liu 2020-07-14 6:32 ` [PATCHv7 bpf-next 1/3] " Hangbin Liu ` (4 more replies) 4 siblings, 5 replies; 219+ messages in thread From: Hangbin Liu @ 2020-07-14 6:32 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because there may have multi interfaces you want to exclude. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. The 2nd and 3rd patches are for usage sample and testing purpose, so there is no effort has been made on performance optimisation. I did same tests with pktgen(pkt size 64) to compire with xdp_redirect_map(). Here is the test result(the veth peer has a dummy xdp program with XDP_DROP directly): Version | Test | Native | Generic 5.8 rc1 | xdp_redirect_map i40e->i40e | 10.0M | 1.9M 5.8 rc1 | xdp_redirect_map i40e->veth | 12.7M | 1.6M 5.8 rc1 + patch | xdp_redirect_map i40e->i40e | 10.0M | 1.9M 5.8 rc1 + patch | xdp_redirect_map i40e->veth | 12.3M | 1.6M 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e | 7.2M | 1.5M 5.8 rc1 + patch | xdp_redirect_map_multi i40e->veth | 8.5M | 1.3M 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e+veth | 3.0M | 0.98M The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop the arrays and do clone skb/xdpf. The native path is slower than generic path as we send skbs by pktgen. So the result looks reasonable. Last but not least, thanks a lot to Jiri, Eelco, Toke and Jesper for suggestions and help on implementation. [0] https://xdp-project.net/#Handling-multicast v7: Fix helper flag check Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). f) Split the tests from sample and add a bpf kernel selftest patch. v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Hangbin Liu (3): xdp: add a new helper for dev map multicast support sample/bpf: add xdp_redirect_map_multicast test selftests/bpf: add xdp_redirect_multi test include/linux/bpf.h | 20 ++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 26 +++ kernel/bpf/devmap.c | 140 ++++++++++++++ kernel/bpf/verifier.c | 6 + net/core/filter.c | 111 ++++++++++- net/core/xdp.c | 29 +++ samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 57 ++++++ samples/bpf/xdp_redirect_map_multi_user.c | 166 +++++++++++++++++ tools/include/uapi/linux/bpf.h | 26 +++ tools/testing/selftests/bpf/Makefile | 4 +- .../bpf/progs/xdp_redirect_multi_kern.c | 90 +++++++++ .../selftests/bpf/test_xdp_redirect_multi.sh | 164 +++++++++++++++++ .../selftests/bpf/xdp_redirect_multi.c | 173 ++++++++++++++++++ 16 files changed, 1011 insertions(+), 6 deletions(-) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c -- 2.25.4 ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv7 bpf-next 1/3] xdp: add a new helper for dev map multicast support 2020-07-14 6:32 ` [PATCHv7 " Hangbin Liu @ 2020-07-14 6:32 ` Hangbin Liu 2020-07-14 21:52 ` Toke Høiland-Jørgensen 2020-07-14 6:32 ` [PATCHv7 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu ` (3 subsequent siblings) 4 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-07-14 6:32 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because there may have multi interfaces you want to exclude. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. The forwarding map could be DEVMAP or DEVMAP_HASH, but the exclude map *must* be DEVMAP_HASH to get better performace. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. As both bpf_xdp_redirect_map() and this new helpers are using struct bpf_redirect_info, I add a new ex_map and set tgt_value to NULL in the new helper to make a difference with bpf_xdp_redirect_map(). Also I keep the the general data path in net/core/filter.c, the native data path in kernel/bpf/devmap.c so we can use direct calls to get better performace. [0] https://xdp-project.net/#Handling-multicast Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v7: Fix helper flag check Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. --- include/linux/bpf.h | 20 +++++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 26 ++++++ kernel/bpf/devmap.c | 140 +++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++ net/core/filter.c | 111 ++++++++++++++++++++++++-- net/core/xdp.c | 29 +++++++ tools/include/uapi/linux/bpf.h | 26 ++++++ 9 files changed, 355 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 0cd7f6884c5c..b48d587b8b3b 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1264,6 +1264,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx); int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, struct net_device *dev_rx); +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex); +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); bool dev_map_can_have_prog(struct bpf_map *map); @@ -1406,6 +1411,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return 0; } +static inline +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + return false; +} + +static inline +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + return 0; +} + struct sk_buff; static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, diff --git a/include/linux/filter.h b/include/linux/filter.h index 259377723603..cf5b5b1d9ae5 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -612,6 +612,7 @@ struct bpf_redirect_info { u32 tgt_index; void *tgt_value; struct bpf_map *map; + struct bpf_map *ex_map; u32 kern_flags; }; diff --git a/include/net/xdp.h b/include/net/xdp.h index 609f819ed08b..deb6c104e698 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -110,6 +110,7 @@ void xdp_warn(const char *msg, const char *func, const int line); #define XDP_WARN(msg) xdp_warn(msg, __func__, __LINE__) struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); static inline void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 548a749aebb3..ce0fb7c8bd5e 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3319,6 +3319,26 @@ union bpf_attr { * A non-negative value equal to or less than *size* on success, * or a negative error in case of failure. * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The frowarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3463,6 +3483,7 @@ union bpf_attr { FN(skc_to_tcp_request_sock), \ FN(skc_to_udp6_sock), \ FN(get_task_stack), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -3624,6 +3645,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 10abb06065bb..bef81f869728 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -512,6 +512,146 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return __xdp_enqueue(dev, xdp, dev_rx); } +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + return dev_map_get_next_key(map, key, next_key); + case BPF_MAP_TYPE_DEVMAP_HASH: + return dev_map_hash_get_next_key(map, key, next_key); + default: + break; + } + + return -ENOENT; +} + +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map); + struct bpf_dtab_netdev *dev; + struct hlist_head *head; + int i = 0; + + if (obj->dev->ifindex == exclude_ifindex) + return true; + + if (!map || map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + return false; + + for (; i < dtab->n_buckets; i++) { + head = dev_map_index_hash(dtab, i); + + dev = hlist_entry_safe(rcu_dereference_raw(hlist_first_rcu(head)), + struct bpf_dtab_netdev, + index_hlist); + + if (dev && dev->idx == exclude_ifindex) + return true; + } + + return false; +} + +static struct bpf_dtab_netdev *devmap_get_next_obj(struct xdp_buff *xdp, struct bpf_map *map, + struct bpf_map *ex_map, u32 *key, + u32 *next_key, int ex_ifindex) +{ + struct bpf_dtab_netdev *obj; + struct net_device *dev; + u32 *tmp_key = key; + int err; + + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + return NULL; + + for (;;) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + obj = __dev_map_lookup_elem(map, *next_key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + obj = __dev_map_hash_lookup_elem(map, *next_key); + break; + default: + break; + } + + if (!obj || dev_in_exclude_map(obj, ex_map, ex_ifindex)) + goto find_next; + + dev = obj->dev; + + if (!dev->netdev_ops->ndo_xdp_xmit) + goto find_next; + + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + if (unlikely(err)) + goto find_next; + + return obj; + +find_next: + tmp_key = next_key; + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + break; + } + + return NULL; +} + +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + struct bpf_dtab_netdev *obj = NULL, *next_obj = NULL; + struct xdp_frame *xdpf, *nxdpf; + bool last_one = false; + int ex_ifindex; + u32 key, next_key; + + ex_ifindex = flags & BPF_F_EXCLUDE_INGRESS ? dev_rx->ifindex : 0; + + /* Find first available obj */ + obj = devmap_get_next_obj(xdp, map, ex_map, NULL, &key, ex_ifindex); + if (!obj) + return 0; + + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) + return -EOVERFLOW; + + for (;;) { + /* Check if we still have one more available obj */ + next_obj = devmap_get_next_obj(xdp, map, ex_map, &key, + &next_key, ex_ifindex); + if (!next_obj) + last_one = true; + + if (last_one) { + bq_enqueue(obj->dev, xdpf, dev_rx); + return 0; + } + + nxdpf = xdpf_clone(xdpf); + if (unlikely(!nxdpf)) { + xdp_return_frame_rx_napi(xdpf); + return -ENOMEM; + } + + bq_enqueue(obj->dev, nxdpf, dev_rx); + + /* Deal with next obj */ + obj = next_obj; + key = next_key; + } +} + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index b608185e1ffd..ceaf28ec111a 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4110,6 +4110,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: if (func_id != BPF_FUNC_redirect_map && + func_id != BPF_FUNC_redirect_map_multi && func_id != BPF_FUNC_map_lookup_elem) goto error; break; @@ -4202,6 +4203,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, map->map_type != BPF_MAP_TYPE_XSKMAP) goto error; break; + case BPF_FUNC_redirect_map_multi: + if (map->map_type != BPF_MAP_TYPE_DEVMAP && + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + goto error; + break; case BPF_FUNC_sk_redirect_map: case BPF_FUNC_msg_redirect_map: case BPF_FUNC_sock_map_update: diff --git a/net/core/filter.c b/net/core/filter.c index 4e572441e64a..ad34edd7eac9 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3516,12 +3516,19 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { }; static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) + struct bpf_map *map, struct xdp_buff *xdp, + struct bpf_map *ex_map, u32 flags) { switch (map->map_type) { case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: - return dev_map_enqueue(fwd, xdp, dev_rx); + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) + return dev_map_enqueue(fwd, xdp, dev_rx); + else + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, flags); case BPF_MAP_TYPE_CPUMAP: return cpu_map_enqueue(fwd, xdp, dev_rx); case BPF_MAP_TYPE_XSKMAP: @@ -3578,12 +3585,14 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); struct bpf_map *map = READ_ONCE(ri->map); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (unlikely(!map)) { @@ -3595,7 +3604,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = dev_xdp_enqueue(fwd, xdp, dev); } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, ri->flags); } if (unlikely(err)) @@ -3609,6 +3618,55 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, } EXPORT_SYMBOL_GPL(xdp_do_redirect); +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, + struct bpf_prog *xdp_prog, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) + +{ + struct bpf_dtab_netdev *dst; + struct sk_buff *nskb; + bool exclude_ingress; + u32 key, next_key; + void *fwd; + int err; + + /* Get first key from forward map */ + err = map->ops->map_get_next_key(map, NULL, &key); + if (err) + return err; + + exclude_ingress = !!(flags & BPF_F_EXCLUDE_INGRESS); + + for (;;) { + fwd = __xdp_map_lookup_elem(map, key); + if (fwd) { + dst = (struct bpf_dtab_netdev *)fwd; + if (dev_in_exclude_map(dst, ex_map, + exclude_ingress ? dev->ifindex : 0)) + goto find_next; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -ENOMEM; + + /* Try forword next one no mater the current forward + * succeed or not */ + dev_map_generic_redirect(dst, nskb, xdp_prog); + } + +find_next: + err = map->ops->map_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + consume_skb(skb); + return 0; +} + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, @@ -3616,19 +3674,30 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, struct bpf_map *map) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err = 0; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (map->map_type == BPF_MAP_TYPE_DEVMAP || map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { - struct bpf_dtab_netdev *dst = fwd; + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) { + struct bpf_dtab_netdev *dst = fwd; + + err = dev_map_generic_redirect(dst, skb, xdp_prog); + } else { + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, + ex_map, ri->flags); + } - err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { @@ -3742,6 +3811,36 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { .arg3_type = ARG_ANYTHING, }; +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, + struct bpf_map *, ex_map, u64, flags) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + + /* Limit ex_map type to DEVMAP_HASH to get better performance */ + if (unlikely(!map || ex_map->map_type != BPF_MAP_TYPE_DEVMAP_HASH || + flags & ~BPF_F_EXCLUDE_INGRESS)) + return XDP_ABORTED; + + ri->tgt_index = 0; + /* Set the tgt_value to NULL to distinguish with bpf_xdp_redirect_map */ + ri->tgt_value = NULL; + ri->flags = flags; + ri->ex_map = ex_map; + + WRITE_ONCE(ri->map, map); + + return XDP_REDIRECT; +} + +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { + .func = bpf_xdp_redirect_map_multi, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_CONST_MAP_PTR, + .arg2_type = ARG_CONST_MAP_PTR, + .arg3_type = ARG_ANYTHING, +}; + static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, unsigned long off, unsigned long len) { @@ -6469,6 +6568,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_redirect_proto; case BPF_FUNC_redirect_map: return &bpf_xdp_redirect_map_proto; + case BPF_FUNC_redirect_map_multi: + return &bpf_xdp_redirect_map_multi_proto; case BPF_FUNC_xdp_adjust_tail: return &bpf_xdp_adjust_tail_proto; case BPF_FUNC_fib_lookup: diff --git a/net/core/xdp.c b/net/core/xdp.c index 3c45f99e26d5..9b43d0a208a7 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -476,3 +476,32 @@ void xdp_warn(const char *msg, const char *func, const int line) WARN(1, "XDP_WARN: %s(line:%d): %s\n", func, line, msg); }; EXPORT_SYMBOL_GPL(xdp_warn); + +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) +{ + unsigned int headroom, totalsize; + struct xdp_frame *nxdpf; + struct page *page; + void *addr; + + headroom = xdpf->headroom + sizeof(*xdpf); + totalsize = headroom + xdpf->len; + + if (unlikely(totalsize > PAGE_SIZE)) + return NULL; + page = dev_alloc_page(); + if (!page) + return NULL; + addr = page_to_virt(page); + + memcpy(addr, xdpf, totalsize); + + nxdpf = addr; + nxdpf->data = addr + headroom; + nxdpf->frame_sz = PAGE_SIZE; + nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0; + nxdpf->mem.id = 0; + + return nxdpf; +} +EXPORT_SYMBOL_GPL(xdpf_clone); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 548a749aebb3..ce0fb7c8bd5e 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3319,6 +3319,26 @@ union bpf_attr { * A non-negative value equal to or less than *size* on success, * or a negative error in case of failure. * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The frowarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3463,6 +3483,7 @@ union bpf_attr { FN(skc_to_tcp_request_sock), \ FN(skc_to_udp6_sock), \ FN(get_task_stack), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -3624,6 +3645,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv7 bpf-next 1/3] xdp: add a new helper for dev map multicast support 2020-07-14 6:32 ` [PATCHv7 bpf-next 1/3] " Hangbin Liu @ 2020-07-14 21:52 ` Toke Høiland-Jørgensen 2020-07-15 12:25 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2020-07-14 21:52 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu Hangbin Liu <liuhangbin@gmail.com> writes: > This patch is for xdp multicast support. which has been discussed > before[0], The goal is to be able to implement an OVS-like data plane in XDP, > i.e., a software switch that can forward XDP frames to multiple ports. > > To achieve this, an application needs to specify a group of interfaces > to forward a packet to. It is also common to want to exclude one or more > physical interfaces from the forwarding operation - e.g., to forward a > packet to all interfaces in the multicast group except the interface it > arrived on. While this could be done simply by adding more groups, this > quickly leads to a combinatorial explosion in the number of groups an > application has to maintain. > > To avoid the combinatorial explosion, we propose to include the ability > to specify an "exclude group" as part of the forwarding operation. This > needs to be a group (instead of just a single port index), because there > may have multi interfaces you want to exclude. > > Thus, the logical forwarding operation becomes a "set difference" > operation, i.e. "forward to all ports in group A that are not also in > group B". This series implements such an operation using device maps to > represent the groups. This means that the XDP program specifies two > device maps, one containing the list of netdevs to redirect to, and the > other containing the exclude list. > > To achieve this, I re-implement a new helper bpf_redirect_map_multi() > to accept two maps, the forwarding map and exclude map. The forwarding > map could be DEVMAP or DEVMAP_HASH, but the exclude map *must* be > DEVMAP_HASH to get better performace. If user don't want to use exclude > map and just want simply stop redirecting back to ingress device, they > can use flag BPF_F_EXCLUDE_INGRESS. > > As both bpf_xdp_redirect_map() and this new helpers are using struct > bpf_redirect_info, I add a new ex_map and set tgt_value to NULL in the > new helper to make a difference with bpf_xdp_redirect_map(). > > Also I keep the the general data path in net/core/filter.c, the native data > path in kernel/bpf/devmap.c so we can use direct calls to get better > performace. > > [0] https://xdp-project.net/#Handling-multicast > > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > > --- > v7: Fix helper flag check > Limit the *ex_map* to use DEVMAP_HASH only and update function > dev_in_exclude_map() to get better performance. > > v6: converted helper return types from int to long > > v5: > a) Check devmap_get_next_key() return value. > b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. > c) In function dev_map_enqueue_multi(), consume xdpf for the last > obj instead of the first on. > d) Update helper description and code comments to explain that we > use NULL target value to distinguish multicast and unicast > forwarding. > e) Update memory model, memory id and frame_sz in xdpf_clone(). > > v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo > > v3: Based on Toke's suggestion, do the following update > a) Update bpf_redirect_map_multi() description in bpf.h. > b) Fix exclude_ifindex checking order in dev_in_exclude_map(). > c) Fix one more xdpf clone in dev_map_enqueue_multi(). > d) Go find next one in dev_map_enqueue_multi() if the interface is not > able to forward instead of abort the whole loop. > e) Remove READ_ONCE/WRITE_ONCE for ex_map. > > v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept > include/exclude maps directly. > > --- > include/linux/bpf.h | 20 +++++ > include/linux/filter.h | 1 + > include/net/xdp.h | 1 + > include/uapi/linux/bpf.h | 26 ++++++ > kernel/bpf/devmap.c | 140 +++++++++++++++++++++++++++++++++ > kernel/bpf/verifier.c | 6 ++ > net/core/filter.c | 111 ++++++++++++++++++++++++-- > net/core/xdp.c | 29 +++++++ > tools/include/uapi/linux/bpf.h | 26 ++++++ > 9 files changed, 355 insertions(+), 5 deletions(-) > > diff --git a/include/linux/bpf.h b/include/linux/bpf.h > index 0cd7f6884c5c..b48d587b8b3b 100644 > --- a/include/linux/bpf.h > +++ b/include/linux/bpf.h > @@ -1264,6 +1264,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, > struct net_device *dev_rx); > int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > struct net_device *dev_rx); > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, > + int exclude_ifindex); > +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, > + struct bpf_map *map, struct bpf_map *ex_map, > + u32 flags); > int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, > struct bpf_prog *xdp_prog); > bool dev_map_can_have_prog(struct bpf_map *map); > @@ -1406,6 +1411,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > return 0; > } > > +static inline > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, > + int exclude_ifindex) > +{ > + return false; > +} > + > +static inline > +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, > + struct bpf_map *map, struct bpf_map *ex_map, > + u32 flags) > +{ > + return 0; > +} > + > struct sk_buff; > > static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, > diff --git a/include/linux/filter.h b/include/linux/filter.h > index 259377723603..cf5b5b1d9ae5 100644 > --- a/include/linux/filter.h > +++ b/include/linux/filter.h > @@ -612,6 +612,7 @@ struct bpf_redirect_info { > u32 tgt_index; > void *tgt_value; > struct bpf_map *map; > + struct bpf_map *ex_map; > u32 kern_flags; > }; > > diff --git a/include/net/xdp.h b/include/net/xdp.h > index 609f819ed08b..deb6c104e698 100644 > --- a/include/net/xdp.h > +++ b/include/net/xdp.h > @@ -110,6 +110,7 @@ void xdp_warn(const char *msg, const char *func, const int line); > #define XDP_WARN(msg) xdp_warn(msg, __func__, __LINE__) > > struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); > +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); > > static inline > void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp) > diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h > index 548a749aebb3..ce0fb7c8bd5e 100644 > --- a/include/uapi/linux/bpf.h > +++ b/include/uapi/linux/bpf.h > @@ -3319,6 +3319,26 @@ union bpf_attr { > * A non-negative value equal to or less than *size* on success, > * or a negative error in case of failure. > * > + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) > + * Description > + * This is a multicast implementation for XDP redirect. It will > + * redirect the packet to ALL the interfaces in *map*, but > + * exclude the interfaces in *ex_map*. > + * > + * The frowarding *map* could be either BPF_MAP_TYPE_DEVMAP or > + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be > + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. > + * > + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, > + * which additionally excludes the current ingress device. > + * > + * See also bpf_redirect_map() as a unicast implementation, > + * which supports redirecting packet to a specific ifindex > + * in the map. As both helpers use struct bpf_redirect_info > + * to store the redirect info, we will use a a NULL tgt_value > + * to distinguish multicast and unicast redirecting. > + * Return > + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. > */ > #define __BPF_FUNC_MAPPER(FN) \ > FN(unspec), \ > @@ -3463,6 +3483,7 @@ union bpf_attr { > FN(skc_to_tcp_request_sock), \ > FN(skc_to_udp6_sock), \ > FN(get_task_stack), \ > + FN(redirect_map_multi), \ > /* */ > > /* integer value in 'imm' field of BPF_CALL instruction selects which helper > @@ -3624,6 +3645,11 @@ enum bpf_lwt_encap_mode { > BPF_LWT_ENCAP_IP, > }; > > +/* BPF_FUNC_redirect_map_multi flags. */ > +enum { > + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), > +}; > + > #define __bpf_md_ptr(type, name) \ > union { \ > type name; \ > diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c > index 10abb06065bb..bef81f869728 100644 > --- a/kernel/bpf/devmap.c > +++ b/kernel/bpf/devmap.c > @@ -512,6 +512,146 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > return __xdp_enqueue(dev, xdp, dev_rx); > } > > +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ > +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) > +{ > + > + switch (map->map_type) { > + case BPF_MAP_TYPE_DEVMAP: > + return dev_map_get_next_key(map, key, next_key); > + case BPF_MAP_TYPE_DEVMAP_HASH: > + return dev_map_hash_get_next_key(map, key, next_key); > + default: > + break; > + } > + > + return -ENOENT; > +} > + > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, > + int exclude_ifindex) > +{ > + struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map); > + struct bpf_dtab_netdev *dev; > + struct hlist_head *head; > + int i = 0; > + > + if (obj->dev->ifindex == exclude_ifindex) > + return true; > + > + if (!map || map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) > + return false; The map type should probably be checked earlier and the whole operation aborted if it is wrong... > + > + for (; i < dtab->n_buckets; i++) { > + head = dev_map_index_hash(dtab, i); > + > + dev = hlist_entry_safe(rcu_dereference_raw(hlist_first_rcu(head)), > + struct bpf_dtab_netdev, > + index_hlist); > + > + if (dev && dev->idx == exclude_ifindex) > + return true; > + } This looks broken; why are you iterating through the buckets? Shouldn't this just be something like: return __dev_map_hash_lookup_elem(map, obj->dev->ifindex) != NULL; -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv7 bpf-next 1/3] xdp: add a new helper for dev map multicast support 2020-07-14 21:52 ` Toke Høiland-Jørgensen @ 2020-07-15 12:25 ` Hangbin Liu 0 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-07-15 12:25 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On Tue, Jul 14, 2020 at 11:52:14PM +0200, Toke Høiland-Jørgensen wrote: > > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, > > + int exclude_ifindex) > > +{ > > + struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map); > > + struct bpf_dtab_netdev *dev; > > + struct hlist_head *head; > > + int i = 0; > > + > > + if (obj->dev->ifindex == exclude_ifindex) > > + return true; > > + > > + if (!map || map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) > > + return false; > > The map type should probably be checked earlier and the whole operation > aborted if it is wrong... Yes, I have already checked it in the helper, there should no need to double check. I will remove this check. > > > + > > + for (; i < dtab->n_buckets; i++) { > > + head = dev_map_index_hash(dtab, i); > > + > > + dev = hlist_entry_safe(rcu_dereference_raw(hlist_first_rcu(head)), > > + struct bpf_dtab_netdev, > > + index_hlist); > > + > > + if (dev && dev->idx == exclude_ifindex) > > + return true; > > + } > > This looks broken; why are you iterating through the buckets? Shouldn't > this just be something like: > > return __dev_map_hash_lookup_elem(map, obj->dev->ifindex) != NULL; Ah, yes, I forgot this. I will update the code. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv7 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test 2020-07-14 6:32 ` [PATCHv7 " Hangbin Liu 2020-07-14 6:32 ` [PATCHv7 bpf-next 1/3] " Hangbin Liu @ 2020-07-14 6:32 ` Hangbin Liu 2020-07-14 6:32 ` [PATCHv7 bpf-next 3/3] selftests/bpf: add xdp_redirect_multi test Hangbin Liu ` (2 subsequent siblings) 4 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-07-14 6:32 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu This is a sample for xdp multicast. In the sample we could forward all packets between given interfaces. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v5: add a null_map as we have strict the arg2 to ARG_CONST_MAP_PTR. Move the testing part to bpf selftest in next patch. v4: no update. v3: add rxcnt map to show the packet transmit speed. v2: no update. --- samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 57 ++++++++ samples/bpf/xdp_redirect_map_multi_user.c | 166 ++++++++++++++++++++++ 3 files changed, 226 insertions(+) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index f87ee02073ba..fddca6cb76b8 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map tprogs-y += per_socket_stats_example tprogs-y += xdp_redirect tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_map_multi tprogs-y += xdp_redirect_cpu tprogs-y += xdp_monitor tprogs-y += xdp_rxq_info @@ -97,6 +98,7 @@ test_map_in_map-objs := test_map_in_map_user.o per_socket_stats_example-objs := cookie_uid_helper_example.o xdp_redirect-objs := xdp_redirect_user.o xdp_redirect_map-objs := xdp_redirect_map_user.o +xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o xdp_redirect_cpu-objs := bpf_load.o xdp_redirect_cpu_user.o xdp_monitor-objs := bpf_load.o xdp_monitor_user.o xdp_rxq_info-objs := xdp_rxq_info_user.o @@ -156,6 +158,7 @@ always-y += tcp_tos_reflect_kern.o always-y += tcp_dumpstats_kern.o always-y += xdp_redirect_kern.o always-y += xdp_redirect_map_kern.o +always-y += xdp_redirect_map_multi_kern.o always-y += xdp_redirect_cpu_kern.o always-y += xdp_monitor_kern.o always-y += xdp_rxq_info_kern.o diff --git a/samples/bpf/xdp_redirect_map_multi_kern.c b/samples/bpf/xdp_redirect_map_multi_kern.c new file mode 100644 index 000000000000..cc7ebaedf55a --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_kern.c @@ -0,0 +1,57 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <uapi/linux/bpf.h> +#include <bpf/bpf_helpers.h> + +struct bpf_map_def SEC("maps") forward_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 256, +}; + +struct bpf_map_def SEC("maps") null_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 1, +}; + +struct bpf_map_def SEC("maps") rxcnt = { + .type = BPF_MAP_TYPE_PERCPU_ARRAY, + .key_size = sizeof(u32), + .value_size = sizeof(long), + .max_entries = 1, +}; + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + long *value; + u32 key = 0; + + /* count packet in global counter */ + value = bpf_map_lookup_elem(&rxcnt, &key); + if (value) + *value += 1; + + return bpf_redirect_map_multi(&forward_map, &null_map, + BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_dummy") +int xdp_pass(struct xdp_md *ctx) +{ + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/samples/bpf/xdp_redirect_map_multi_user.c b/samples/bpf/xdp_redirect_map_multi_user.c new file mode 100644 index 000000000000..49f44c91b672 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_user.c @@ -0,0 +1,166 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; +static int rxcnt; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void poll_stats(int interval) +{ + unsigned int nr_cpus = bpf_num_possible_cpus(); + __u64 values[nr_cpus], prev[nr_cpus]; + + memset(prev, 0, sizeof(prev)); + + while (1) { + __u64 sum = 0; + __u32 key = 0; + int i; + + sleep(interval); + assert(bpf_map_lookup_elem(rxcnt, &key, values) == 0); + for (i = 0; i < nr_cpus; i++) + sum += (values[i] - prev[i]); + if (sum) + printf("Forwarding %10llu pkt/s\n", sum / interval); + memcpy(prev, values, sizeof(values)); + } +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n", + prog); +} + +int main(int argc, char **argv) +{ + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + int prog_fd, forward_map; + int i, ret, opt, ifindex; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNF")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) + xdp_flags |= XDP_FLAGS_DRV_MODE; + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + return 1; + + forward_map = bpf_object__find_map_fd_by_name(obj, "forward_map"); + rxcnt = bpf_object__find_map_fd_by_name(obj, "rxcnt"); + + if (forward_map < 0 || rxcnt < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + return 1; + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* Add all the interfaces to group all */ + ret = bpf_map_update_elem(forward_map, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + } + + poll_stats(2); + + return 0; + +err_out: + return 1; +} -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv7 bpf-next 3/3] selftests/bpf: add xdp_redirect_multi test 2020-07-14 6:32 ` [PATCHv7 " Hangbin Liu 2020-07-14 6:32 ` [PATCHv7 bpf-next 1/3] " Hangbin Liu 2020-07-14 6:32 ` [PATCHv7 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu @ 2020-07-14 6:32 ` Hangbin Liu 2020-07-14 12:29 ` [PATCHv7 bpf-next 0/3] xdp: add a new helper for dev map multicast support Toke Høiland-Jørgensen 2020-07-15 13:08 ` [PATCHv8 " Hangbin Liu 4 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-07-14 6:32 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu Add a bpf selftest for new helper xdp_redirect_map_multi(). In this test we have 3 forward groups groups and 1 exclude group. The test will redirect each interface's packets to all the interfaces in the forward group, and exclude the interface in exclude map. We will also test both DEVMAP and DEVMAP_HASH with xdp generic and drv. For more test details, you can find it in the test script. Here is the test result. ]# ./test_xdp_redirect_multi.sh Pass: xdpgeneric arp ns1-2 Pass: xdpgeneric arp ns1-3 Pass: xdpgeneric arp ns1-4 Pass: xdpgeneric ping ns1-2 Pass: xdpgeneric ping ns1-3 Pass: xdpgeneric ping ns1-4 Pass: xdpgeneric ping6 ns2-1 Pass: xdpgeneric ping6 ns2-3 Pass: xdpgeneric ping6 ns2-4 Pass: xdpdrv arp ns1-2 Pass: xdpdrv arp ns1-3 Pass: xdpdrv arp ns1-4 Pass: xdpdrv ping ns1-2 Pass: xdpdrv ping ns1-3 Pass: xdpdrv ping ns1-4 Pass: xdpdrv ping6 ns2-1 Pass: xdpdrv ping6 ns2-3 Pass: xdpdrv ping6 ns2-4 Summary: PASS 18, FAIL 0 Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- tools/testing/selftests/bpf/Makefile | 4 +- .../bpf/progs/xdp_redirect_multi_kern.c | 90 +++++++++ .../selftests/bpf/test_xdp_redirect_multi.sh | 164 +++++++++++++++++ .../selftests/bpf/xdp_redirect_multi.c | 173 ++++++++++++++++++ 4 files changed, 430 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index e7a8cf83ba48..c2434a3d2d15 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -51,6 +51,7 @@ TEST_FILES = test_lwt_ip_encap.o \ # Order correspond to 'make run_tests' order TEST_PROGS := test_kmod.sh \ test_xdp_redirect.sh \ + test_xdp_redirect_multi.sh \ test_xdp_meta.sh \ test_xdp_veth.sh \ test_offload.py \ @@ -79,7 +80,8 @@ TEST_PROGS_EXTENDED := with_addr.sh \ # Compile but not part of 'make run_tests' TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ - test_lirc_mode2_user xdping test_cpp runqslower bench + test_lirc_mode2_user xdping test_cpp runqslower bench \ + xdp_redirect_multi TEST_CUSTOM_PROGS = urandom_read diff --git a/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c new file mode 100644 index 000000000000..70b8476b9df3 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c @@ -0,0 +1,90 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <string.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ip.h> +#include <linux/ipv6.h> + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_endian.h> + +struct bpf_map_def SEC("maps") forward_map_v4 = { + .type = BPF_MAP_TYPE_DEVMAP, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 4096, +}; + +struct bpf_map_def SEC("maps") forward_map_v6 = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") forward_map_all = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") exclude_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") null_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 1, +}; + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + struct ethhdr *eth = data; + __u16 h_proto; + __u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + h_proto = eth->h_proto; + + if (h_proto == bpf_htons(ETH_P_IP)) + return bpf_redirect_map_multi(&forward_map_v4, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else if (h_proto == bpf_htons(ETH_P_IPV6)) + return bpf_redirect_map_multi(&forward_map_v6, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else + return bpf_redirect_map_multi(&forward_map_all, &null_map, + BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_dummy") +int xdp_pass(struct xdp_md *ctx) +{ + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh new file mode 100755 index 000000000000..f4f8f751854e --- /dev/null +++ b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh @@ -0,0 +1,164 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Test topology: +# - - - - - - - - - - - - - - - - - - - - - - - - - +# | veth1 veth2 veth3 veth4 | ... init net +# - -| - - - - - - | - - - - - - | - - - - - - | - - +# --------- --------- --------- --------- +# | veth0 | | veth0 | | veth0 | | veth0 | ... +# --------- --------- --------- --------- +# ns1 ns2 ns3 ns4 +# +# Forward multicast groups: +# Forward group all has interfaces: veth1, veth2, veth3, veth4, ... (All traffic except IPv4, IPv6) +# Forward group v4 has interfaces: veth1, veth3, veth4, ... (For IPv4 traffic only) +# Forward group v6 has interfaces: veth2, veth3, veth4, ... (For IPv6 traffic only) +# Exclude Groups: +# Exclude group: veth3 (assume ns3 is in black list) +# +# Test modules: +# XDP modes: generic, native +# map types: group v4 use DEVMAP, others use DEVMAP_HASH +# +# Test cases: +# ARP(we didn't block ARP for ns3): +# ns1 -> gw: ns2, ns3, ns4 should receive the arp request +# IPv4: +# ns1 -> ns2 (fail), ns1 -> ns3 (fail), ns1 -> ns4 (pass) +# IPv6 +# ns2 -> ns1 (fail), ns2 -> ns3 (fail), ns2 -> ns4 (pass) +# + + +# netns numbers +NUM=4 +IFACES="" +DRV_MODE="xdpgeneric xdpdrv" +PASS=0 +FAIL=0 + +test_pass() +{ + echo "Pass: $@" + PASS=$((PASS + 1)) +} + +test_fail() +{ + echo "fail: $@" + FAIL=$((FAIL + 1)) +} + +clean_up() +{ + for i in $(seq $NUM); do + ip link del veth$i 2> /dev/null + ip netns del ns$i 2> /dev/null + done + rm -f xdp_redirect_*.log arp_ns*.log +} + +# Kselftest framework requirement - SKIP code is 4. +check_env() +{ + ip link set dev lo xdpgeneric off &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without the ip xdpgeneric support" + exit 4 + fi + + which tcpdump &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without tcpdump" + exit 4 + fi +} + +setup_ns() +{ + local mode=$1 + IFACES="" + + for i in $(seq $NUM); do + ip netns add ns$i + ip link add veth$i type veth peer name veth0 netns ns$i + ip link set veth$i up + ip -n ns$i link set veth0 up + + ip -n ns$i addr add 192.0.2.$i/24 dev veth0 + ip -n ns$i addr add 2001:db8::$i/64 dev veth0 + ip -n ns$i link set veth0 $mode obj \ + xdp_redirect_multi_kern.o sec xdp_dummy &> /dev/null || \ + { test_fail "Unable to load dummy xdp" && exit 1; } + IFACES="$IFACES veth$i" + done +} + +do_ping_tests() +{ + local mode=$1 + + # arp test + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -i veth0 -nn -l -e &> arp_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-2_${mode}.log && \ + test_pass "$mode arp ns1-2" || test_fail "$mode arp ns1-2" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-3_${mode}.log && \ + test_pass "$mode arp ns1-3" || test_fail "$mode arp ns1-3" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-4_${mode}.log && \ + test_pass "$mode arp ns1-4" || test_fail "$mode arp ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-2" || test_pass "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-3" || test_pass "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" + + # ping6 test + ip netns exec ns2 ping6 2001:db8::1 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-1" || test_pass "$mode ping6 ns2-1" + ip netns exec ns2 ping6 2001:db8::3 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-3" || test_pass "$mode ping6 ns2-3" + ip netns exec ns2 ping6 2001:db8::4 -c 4 &> /dev/null && \ + test_pass "$mode ping6 ns2-4" || test_fail "$mode ping6 ns2-4" +} + +do_tests() +{ + local mode=$1 + local drv_p + + [ ${mode} == "xdpdrv" ] && drv_p="-N" || drv_p="-S" + + # run `ulimit -l unlimited` if you got errors like + # libbpf: Error in bpf_object__probe_global_data():Operation not permitted(1). + ./xdp_redirect_multi $drv_p $IFACES &> xdp_redirect_${mode}.log & + xdp_pid=$! + sleep 10 + + do_ping_tests $mode + + kill $xdp_pid +} + +trap clean_up 0 2 3 6 9 + +check_env + +for mode in ${DRV_MODE}; do + setup_ns $mode + do_tests $mode + sleep 10 + clean_up + sleep 5 +done + +echo "Summary: PASS $PASS, FAIL $FAIL" +[ $FAIL -eq 0 ] && exit 0 || exit 1 diff --git a/tools/testing/selftests/bpf/xdp_redirect_multi.c b/tools/testing/selftests/bpf/xdp_redirect_multi.c new file mode 100644 index 000000000000..5626005cb679 --- /dev/null +++ b/tools/testing/selftests/bpf/xdp_redirect_multi.c @@ -0,0 +1,173 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n", + prog); +} + +int main(int argc, char **argv) +{ + int prog_fd, group_all, group_v4, group_v6, exclude; + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + int i, ret, opt, ifindex; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNF")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) + xdp_flags |= XDP_FLAGS_DRV_MODE; + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + return 1; + + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_all"); + group_v4 = bpf_object__find_map_fd_by_name(obj, "forward_map_v4"); + group_v6 = bpf_object__find_map_fd_by_name(obj, "forward_map_v6"); + exclude = bpf_object__find_map_fd_by_name(obj, "exclude_map"); + + if (group_all < 0 || group_v4 < 0 || group_v6 < 0 || exclude < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + return 1; + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* Add all the interfaces to group all */ + ret = bpf_map_update_elem(group_all, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* For testing: remove the 1st interfaces from group v6 */ + if (i != 0) { + ret = bpf_map_update_elem(group_v6, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: remove the 2nd interfaces from group v4 */ + if (i != 1) { + ret = bpf_map_update_elem(group_v4, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: add the 3rd interfaces to exclude map */ + if (i == 2) { + ret = bpf_map_update_elem(exclude, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + } + + /* sleep some time for testing */ + sleep(999); + + return 0; + +err_out: + return 1; +} -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv7 bpf-next 0/3] xdp: add a new helper for dev map multicast support 2020-07-14 6:32 ` [PATCHv7 " Hangbin Liu ` (2 preceding siblings ...) 2020-07-14 6:32 ` [PATCHv7 bpf-next 3/3] selftests/bpf: add xdp_redirect_multi test Hangbin Liu @ 2020-07-14 12:29 ` Toke Høiland-Jørgensen 2020-07-14 17:12 ` David Ahern 2020-07-15 13:08 ` [PATCHv8 " Hangbin Liu 4 siblings, 1 reply; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2020-07-14 12:29 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu Hangbin Liu <liuhangbin@gmail.com> writes: > This patch is for xdp multicast support. which has been discussed before[0], > The goal is to be able to implement an OVS-like data plane in XDP, i.e., > a software switch that can forward XDP frames to multiple ports. > > To achieve this, an application needs to specify a group of interfaces > to forward a packet to. It is also common to want to exclude one or more > physical interfaces from the forwarding operation - e.g., to forward a > packet to all interfaces in the multicast group except the interface it > arrived on. While this could be done simply by adding more groups, this > quickly leads to a combinatorial explosion in the number of groups an > application has to maintain. > > To avoid the combinatorial explosion, we propose to include the ability > to specify an "exclude group" as part of the forwarding operation. This > needs to be a group (instead of just a single port index), because there > may have multi interfaces you want to exclude. > > Thus, the logical forwarding operation becomes a "set difference" > operation, i.e. "forward to all ports in group A that are not also in > group B". This series implements such an operation using device maps to > represent the groups. This means that the XDP program specifies two > device maps, one containing the list of netdevs to redirect to, and the > other containing the exclude list. > > To achieve this, I re-implement a new helper bpf_redirect_map_multi() > to accept two maps, the forwarding map and exclude map. If user > don't want to use exclude map and just want simply stop redirecting back > to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. > > The 2nd and 3rd patches are for usage sample and testing purpose, so there > is no effort has been made on performance optimisation. I did same tests > with pktgen(pkt size 64) to compire with xdp_redirect_map(). Here is the > test result(the veth peer has a dummy xdp program with XDP_DROP directly): > > Version | Test | Native | Generic > 5.8 rc1 | xdp_redirect_map i40e->i40e | 10.0M | 1.9M > 5.8 rc1 | xdp_redirect_map i40e->veth | 12.7M | 1.6M > 5.8 rc1 + patch | xdp_redirect_map i40e->i40e | 10.0M | 1.9M > 5.8 rc1 + patch | xdp_redirect_map i40e->veth | 12.3M | 1.6M > 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e | 7.2M | 1.5M > 5.8 rc1 + patch | xdp_redirect_map_multi i40e->veth | 8.5M | 1.3M > 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e+veth | 3.0M | 0.98M > > The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop > the arrays and do clone skb/xdpf. The native path is slower than generic > path as we send skbs by pktgen. So the result looks reasonable. > > Last but not least, thanks a lot to Jiri, Eelco, Toke and Jesper for > suggestions and help on implementation. > > [0] https://xdp-project.net/#Handling-multicast > > v7: Fix helper flag check > Limit the *ex_map* to use DEVMAP_HASH only and update function > dev_in_exclude_map() to get better performance. Did it help? The performance numbers in the table above are the same as in v6... -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv7 bpf-next 0/3] xdp: add a new helper for dev map multicast support 2020-07-14 12:29 ` [PATCHv7 bpf-next 0/3] xdp: add a new helper for dev map multicast support Toke Høiland-Jørgensen @ 2020-07-14 17:12 ` David Ahern 2020-07-14 21:53 ` Toke Høiland-Jørgensen 2020-07-15 3:45 ` Hangbin Liu 0 siblings, 2 replies; 219+ messages in thread From: David Ahern @ 2020-07-14 17:12 UTC (permalink / raw) To: Toke Høiland-Jørgensen, Hangbin Liu, bpf Cc: netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On 7/14/20 6:29 AM, Toke Høiland-Jørgensen wrote: > Hangbin Liu <liuhangbin@gmail.com> writes: > >> This patch is for xdp multicast support. which has been discussed before[0], >> The goal is to be able to implement an OVS-like data plane in XDP, i.e., >> a software switch that can forward XDP frames to multiple ports. >> >> To achieve this, an application needs to specify a group of interfaces >> to forward a packet to. It is also common to want to exclude one or more >> physical interfaces from the forwarding operation - e.g., to forward a >> packet to all interfaces in the multicast group except the interface it >> arrived on. While this could be done simply by adding more groups, this >> quickly leads to a combinatorial explosion in the number of groups an >> application has to maintain. >> >> To avoid the combinatorial explosion, we propose to include the ability >> to specify an "exclude group" as part of the forwarding operation. This >> needs to be a group (instead of just a single port index), because there >> may have multi interfaces you want to exclude. >> >> Thus, the logical forwarding operation becomes a "set difference" >> operation, i.e. "forward to all ports in group A that are not also in >> group B". This series implements such an operation using device maps to >> represent the groups. This means that the XDP program specifies two >> device maps, one containing the list of netdevs to redirect to, and the >> other containing the exclude list. >> >> To achieve this, I re-implement a new helper bpf_redirect_map_multi() >> to accept two maps, the forwarding map and exclude map. If user >> don't want to use exclude map and just want simply stop redirecting back >> to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. >> >> The 2nd and 3rd patches are for usage sample and testing purpose, so there >> is no effort has been made on performance optimisation. I did same tests >> with pktgen(pkt size 64) to compire with xdp_redirect_map(). Here is the >> test result(the veth peer has a dummy xdp program with XDP_DROP directly): >> >> Version | Test | Native | Generic >> 5.8 rc1 | xdp_redirect_map i40e->i40e | 10.0M | 1.9M >> 5.8 rc1 | xdp_redirect_map i40e->veth | 12.7M | 1.6M >> 5.8 rc1 + patch | xdp_redirect_map i40e->i40e | 10.0M | 1.9M >> 5.8 rc1 + patch | xdp_redirect_map i40e->veth | 12.3M | 1.6M >> 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e | 7.2M | 1.5M >> 5.8 rc1 + patch | xdp_redirect_map_multi i40e->veth | 8.5M | 1.3M >> 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e+veth | 3.0M | 0.98M >> >> The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop >> the arrays and do clone skb/xdpf. The native path is slower than generic >> path as we send skbs by pktgen. So the result looks reasonable. >> >> Last but not least, thanks a lot to Jiri, Eelco, Toke and Jesper for >> suggestions and help on implementation. >> >> [0] https://xdp-project.net/#Handling-multicast >> >> v7: Fix helper flag check >> Limit the *ex_map* to use DEVMAP_HASH only and update function >> dev_in_exclude_map() to get better performance. > > Did it help? The performance numbers in the table above are the same as > in v6... > If there is only 1 entry in the exclude map, then the numbers should be about the same. ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv7 bpf-next 0/3] xdp: add a new helper for dev map multicast support 2020-07-14 17:12 ` David Ahern @ 2020-07-14 21:53 ` Toke Høiland-Jørgensen 2020-07-15 12:31 ` Hangbin Liu 2020-07-15 3:45 ` Hangbin Liu 1 sibling, 1 reply; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2020-07-14 21:53 UTC (permalink / raw) To: David Ahern, Hangbin Liu, bpf Cc: netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi >David Ahern <dsahern@gmail.com> writes: > On 7/14/20 6:29 AM, Toke Høiland-Jørgensen wrote: >> Hangbin Liu <liuhangbin@gmail.com> writes: >> >>> This patch is for xdp multicast support. which has been discussed before[0], >>> The goal is to be able to implement an OVS-like data plane in XDP, i.e., >>> a software switch that can forward XDP frames to multiple ports. >>> >>> To achieve this, an application needs to specify a group of interfaces >>> to forward a packet to. It is also common to want to exclude one or more >>> physical interfaces from the forwarding operation - e.g., to forward a >>> packet to all interfaces in the multicast group except the interface it >>> arrived on. While this could be done simply by adding more groups, this >>> quickly leads to a combinatorial explosion in the number of groups an >>> application has to maintain. >>> >>> To avoid the combinatorial explosion, we propose to include the ability >>> to specify an "exclude group" as part of the forwarding operation. This >>> needs to be a group (instead of just a single port index), because there >>> may have multi interfaces you want to exclude. >>> >>> Thus, the logical forwarding operation becomes a "set difference" >>> operation, i.e. "forward to all ports in group A that are not also in >>> group B". This series implements such an operation using device maps to >>> represent the groups. This means that the XDP program specifies two >>> device maps, one containing the list of netdevs to redirect to, and the >>> other containing the exclude list. >>> >>> To achieve this, I re-implement a new helper bpf_redirect_map_multi() >>> to accept two maps, the forwarding map and exclude map. If user >>> don't want to use exclude map and just want simply stop redirecting back >>> to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. >>> >>> The 2nd and 3rd patches are for usage sample and testing purpose, so there >>> is no effort has been made on performance optimisation. I did same tests >>> with pktgen(pkt size 64) to compire with xdp_redirect_map(). Here is the >>> test result(the veth peer has a dummy xdp program with XDP_DROP directly): >>> >>> Version | Test | Native | Generic >>> 5.8 rc1 | xdp_redirect_map i40e->i40e | 10.0M | 1.9M >>> 5.8 rc1 | xdp_redirect_map i40e->veth | 12.7M | 1.6M >>> 5.8 rc1 + patch | xdp_redirect_map i40e->i40e | 10.0M | 1.9M >>> 5.8 rc1 + patch | xdp_redirect_map i40e->veth | 12.3M | 1.6M >>> 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e | 7.2M | 1.5M >>> 5.8 rc1 + patch | xdp_redirect_map_multi i40e->veth | 8.5M | 1.3M >>> 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e+veth | 3.0M | 0.98M >>> >>> The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop >>> the arrays and do clone skb/xdpf. The native path is slower than generic >>> path as we send skbs by pktgen. So the result looks reasonable. >>> >>> Last but not least, thanks a lot to Jiri, Eelco, Toke and Jesper for >>> suggestions and help on implementation. >>> >>> [0] https://xdp-project.net/#Handling-multicast >>> >>> v7: Fix helper flag check >>> Limit the *ex_map* to use DEVMAP_HASH only and update function >>> dev_in_exclude_map() to get better performance. >> >> Did it help? The performance numbers in the table above are the same as >> in v6... >> > > If there is only 1 entry in the exclude map, then the numbers should be > about the same. I would still expect the lack of the calls to devmap_get_next_key() to at least provide a small speedup, no? That the numbers are completely unchanged looks a bit suspicious... -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv7 bpf-next 0/3] xdp: add a new helper for dev map multicast support 2020-07-14 21:53 ` Toke Høiland-Jørgensen @ 2020-07-15 12:31 ` Hangbin Liu 0 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-07-15 12:31 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: David Ahern, bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On Tue, Jul 14, 2020 at 11:53:20PM +0200, Toke Høiland-Jørgensen wrote: > >David Ahern <dsahern@gmail.com> writes: > >>> Version | Test | Native | Generic > >>> 5.8 rc1 | xdp_redirect_map i40e->i40e | 10.0M | 1.9M > >>> 5.8 rc1 | xdp_redirect_map i40e->veth | 12.7M | 1.6M > >>> 5.8 rc1 + patch | xdp_redirect_map i40e->i40e | 10.0M | 1.9M > >>> 5.8 rc1 + patch | xdp_redirect_map i40e->veth | 12.3M | 1.6M > >>> 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e | 7.2M | 1.5M > >>> 5.8 rc1 + patch | xdp_redirect_map_multi i40e->veth | 8.5M | 1.3M > >>> 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e+veth | 3.0M | 0.98M > >>> > >>> The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop > >>> the arrays and do clone skb/xdpf. The native path is slower than generic > >>> path as we send skbs by pktgen. So the result looks reasonable. > >>> > >>> Last but not least, thanks a lot to Jiri, Eelco, Toke and Jesper for > >>> suggestions and help on implementation. > >>> > >>> [0] https://xdp-project.net/#Handling-multicast > >>> > >>> v7: Fix helper flag check > >>> Limit the *ex_map* to use DEVMAP_HASH only and update function > >>> dev_in_exclude_map() to get better performance. > >> > >> Did it help? The performance numbers in the table above are the same as > >> in v6... > >> > > > > If there is only 1 entry in the exclude map, then the numbers should be > > about the same. > > I would still expect the lack of the calls to devmap_get_next_key() to > at least provide a small speedup, no? That the numbers are completely > unchanged looks a bit suspicious... As I replied to David, I didn't re-run the test as I thought there should no much difference as the exclude map on has 1 entry. There should be a small speedup compared with previous patch. But as the test system re-installed and rebooted, there will be some jitter to the test result. It would be a little hard to observe the improvement. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv7 bpf-next 0/3] xdp: add a new helper for dev map multicast support 2020-07-14 17:12 ` David Ahern 2020-07-14 21:53 ` Toke Høiland-Jørgensen @ 2020-07-15 3:45 ` Hangbin Liu 1 sibling, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-07-15 3:45 UTC (permalink / raw) To: David Ahern Cc: Toke Høiland-Jørgensen, bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi On Tue, Jul 14, 2020 at 11:12:59AM -0600, David Ahern wrote: > >> with pktgen(pkt size 64) to compire with xdp_redirect_map(). Here is the > >> test result(the veth peer has a dummy xdp program with XDP_DROP directly): > >> > >> Version | Test | Native | Generic > >> 5.8 rc1 | xdp_redirect_map i40e->i40e | 10.0M | 1.9M > >> 5.8 rc1 | xdp_redirect_map i40e->veth | 12.7M | 1.6M > >> 5.8 rc1 + patch | xdp_redirect_map i40e->i40e | 10.0M | 1.9M > >> 5.8 rc1 + patch | xdp_redirect_map i40e->veth | 12.3M | 1.6M > >> 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e | 7.2M | 1.5M > >> 5.8 rc1 + patch | xdp_redirect_map_multi i40e->veth | 8.5M | 1.3M > >> 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e+veth | 3.0M | 0.98M > >> > >> The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop > >> the arrays and do clone skb/xdpf. The native path is slower than generic > >> path as we send skbs by pktgen. So the result looks reasonable. > >> > >> Last but not least, thanks a lot to Jiri, Eelco, Toke and Jesper for > >> suggestions and help on implementation. > >> > >> [0] https://xdp-project.net/#Handling-multicast > >> > >> v7: Fix helper flag check > >> Limit the *ex_map* to use DEVMAP_HASH only and update function > >> dev_in_exclude_map() to get better performance. > > > > Did it help? The performance numbers in the table above are the same as > > in v6... > > > > If there is only 1 entry in the exclude map, then the numbers should be > about the same. Yes, I didn't re-run the test. Because when do the testing, I use null exclude map + flag BPF_F_EXCLUDE_INGRESS. So the perf number should have no difference with last patch. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv8 bpf-next 0/3] xdp: add a new helper for dev map multicast support 2020-07-14 6:32 ` [PATCHv7 " Hangbin Liu ` (3 preceding siblings ...) 2020-07-14 12:29 ` [PATCHv7 bpf-next 0/3] xdp: add a new helper for dev map multicast support Toke Høiland-Jørgensen @ 2020-07-15 13:08 ` Hangbin Liu 2020-07-15 13:08 ` [PATCHv8 bpf-next 1/3] " Hangbin Liu ` (3 more replies) 4 siblings, 4 replies; 219+ messages in thread From: Hangbin Liu @ 2020-07-15 13:08 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because there may have multi interfaces you want to exclude. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. The 2nd and 3rd patches are for usage sample and testing purpose, so there is no effort has been made on performance optimisation. I did same tests with pktgen(pkt size 64) to compire with xdp_redirect_map(). Here is the test result(the veth peer has a dummy xdp program with XDP_DROP directly): Version | Test | Native | Generic 5.8 rc1 | xdp_redirect_map i40e->i40e | 10.0M | 1.9M 5.8 rc1 | xdp_redirect_map i40e->veth | 12.7M | 1.6M 5.8 rc1 + patch | xdp_redirect_map i40e->i40e | 10.0M | 1.9M 5.8 rc1 + patch | xdp_redirect_map i40e->veth | 12.3M | 1.6M 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e | 7.2M | 1.5M 5.8 rc1 + patch | xdp_redirect_map_multi i40e->veth | 8.5M | 1.3M 5.8 rc1 + patch | xdp_redirect_map_multi i40e->i40e+veth | 3.0M | 0.98M The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop the arrays and do clone skb/xdpf. The native path is slower than generic path as we send skbs by pktgen. So the result looks reasonable. Last but not least, thanks a lot to Jiri, Eelco, Toke and Jesper for suggestions and help on implementation. [0] https://xdp-project.net/#Handling-multicast v8: a) Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). f) Split the tests from sample and add a bpf kernel selftest patch. v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Hangbin Liu (3): xdp: add a new helper for dev map multicast support sample/bpf: add xdp_redirect_map_multicast test selftests/bpf: add xdp_redirect_multi test include/linux/bpf.h | 20 ++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 26 +++ kernel/bpf/devmap.c | 124 +++++++++++++ kernel/bpf/verifier.c | 6 + net/core/filter.c | 111 ++++++++++- net/core/xdp.c | 29 +++ samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 57 ++++++ samples/bpf/xdp_redirect_map_multi_user.c | 166 +++++++++++++++++ tools/include/uapi/linux/bpf.h | 26 +++ tools/testing/selftests/bpf/Makefile | 4 +- .../bpf/progs/xdp_redirect_multi_kern.c | 90 +++++++++ .../selftests/bpf/test_xdp_redirect_multi.sh | 164 +++++++++++++++++ .../selftests/bpf/xdp_redirect_multi.c | 173 ++++++++++++++++++ 16 files changed, 995 insertions(+), 6 deletions(-) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c -- 2.25.4 ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv8 bpf-next 1/3] xdp: add a new helper for dev map multicast support 2020-07-15 13:08 ` [PATCHv8 " Hangbin Liu @ 2020-07-15 13:08 ` Hangbin Liu 2020-07-15 13:08 ` [PATCHv8 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu ` (2 subsequent siblings) 3 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-07-15 13:08 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because a physical interface can be part of a logical grouping, such as a bond device. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. The forwarding map could be DEVMAP or DEVMAP_HASH, but the exclude map *must* be DEVMAP_HASH to get better performace. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. As both bpf_xdp_redirect_map() and this new helpers are using struct bpf_redirect_info, I add a new ex_map and set tgt_value to NULL in the new helper to make a difference with bpf_xdp_redirect_map(). Also I keep the the general data path in net/core/filter.c, the native data path in kernel/bpf/devmap.c so we can use direct calls to get better performace. [0] https://xdp-project.net/#Handling-multicast Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v8: a) Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. --- include/linux/bpf.h | 20 ++++++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 26 +++++++ kernel/bpf/devmap.c | 124 +++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++ net/core/filter.c | 111 +++++++++++++++++++++++++++-- net/core/xdp.c | 29 ++++++++ tools/include/uapi/linux/bpf.h | 26 +++++++ 9 files changed, 339 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index c67c88ad35f8..ed1dc0a084f5 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1264,6 +1264,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx); int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, struct net_device *dev_rx); +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex); +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); bool dev_map_can_have_prog(struct bpf_map *map); @@ -1406,6 +1411,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return 0; } +static inline +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + return false; +} + +static inline +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + return 0; +} + struct sk_buff; static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, diff --git a/include/linux/filter.h b/include/linux/filter.h index 0b0144752d78..cf7debed9bc0 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -612,6 +612,7 @@ struct bpf_redirect_info { u32 tgt_index; void *tgt_value; struct bpf_map *map; + struct bpf_map *ex_map; u32 kern_flags; }; diff --git a/include/net/xdp.h b/include/net/xdp.h index 609f819ed08b..deb6c104e698 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -110,6 +110,7 @@ void xdp_warn(const char *msg, const char *func, const int line); #define XDP_WARN(msg) xdp_warn(msg, __func__, __LINE__) struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); static inline void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 5e386389913a..54d5b42d282c 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3320,6 +3320,26 @@ union bpf_attr { * A non-negative value equal to or less than *size* on success, * or a negative error in case of failure. * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The frowarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3464,6 +3484,7 @@ union bpf_attr { FN(skc_to_tcp_request_sock), \ FN(skc_to_udp6_sock), \ FN(get_task_stack), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -3625,6 +3646,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 10abb06065bb..1e6695f38f78 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -512,6 +512,130 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return __xdp_enqueue(dev, xdp, dev_rx); } +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + return dev_map_get_next_key(map, key, next_key); + case BPF_MAP_TYPE_DEVMAP_HASH: + return dev_map_hash_get_next_key(map, key, next_key); + default: + break; + } + + return -ENOENT; +} + +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + if (obj->dev->ifindex == exclude_ifindex) + return true; + + if (!map) + return false; + + return __dev_map_hash_lookup_elem(map, obj->dev->ifindex) != NULL; +} + +static struct bpf_dtab_netdev *devmap_get_next_obj(struct xdp_buff *xdp, struct bpf_map *map, + struct bpf_map *ex_map, u32 *key, + u32 *next_key, int ex_ifindex) +{ + struct bpf_dtab_netdev *obj; + struct net_device *dev; + u32 *tmp_key = key; + int err; + + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + return NULL; + + for (;;) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + obj = __dev_map_lookup_elem(map, *next_key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + obj = __dev_map_hash_lookup_elem(map, *next_key); + break; + default: + break; + } + + if (!obj || dev_in_exclude_map(obj, ex_map, ex_ifindex)) + goto find_next; + + dev = obj->dev; + + if (!dev->netdev_ops->ndo_xdp_xmit) + goto find_next; + + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + if (unlikely(err)) + goto find_next; + + return obj; + +find_next: + tmp_key = next_key; + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + break; + } + + return NULL; +} + +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + struct bpf_dtab_netdev *obj = NULL, *next_obj = NULL; + struct xdp_frame *xdpf, *nxdpf; + bool last_one = false; + int ex_ifindex; + u32 key, next_key; + + ex_ifindex = flags & BPF_F_EXCLUDE_INGRESS ? dev_rx->ifindex : 0; + + /* Find first available obj */ + obj = devmap_get_next_obj(xdp, map, ex_map, NULL, &key, ex_ifindex); + if (!obj) + return 0; + + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) + return -EOVERFLOW; + + for (;;) { + /* Check if we still have one more available obj */ + next_obj = devmap_get_next_obj(xdp, map, ex_map, &key, + &next_key, ex_ifindex); + if (!next_obj) + last_one = true; + + if (last_one) { + bq_enqueue(obj->dev, xdpf, dev_rx); + return 0; + } + + nxdpf = xdpf_clone(xdpf); + if (unlikely(!nxdpf)) { + xdp_return_frame_rx_napi(xdpf); + return -ENOMEM; + } + + bq_enqueue(obj->dev, nxdpf, dev_rx); + + /* Deal with next obj */ + obj = next_obj; + key = next_key; + } +} + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 3c1efc9d08fd..487f973e1d64 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4109,6 +4109,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: if (func_id != BPF_FUNC_redirect_map && + func_id != BPF_FUNC_redirect_map_multi && func_id != BPF_FUNC_map_lookup_elem) goto error; break; @@ -4201,6 +4202,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, map->map_type != BPF_MAP_TYPE_XSKMAP) goto error; break; + case BPF_FUNC_redirect_map_multi: + if (map->map_type != BPF_MAP_TYPE_DEVMAP && + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + goto error; + break; case BPF_FUNC_sk_redirect_map: case BPF_FUNC_msg_redirect_map: case BPF_FUNC_sock_map_update: diff --git a/net/core/filter.c b/net/core/filter.c index bdd2382e655d..9fc5ec554647 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3516,12 +3516,19 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { }; static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) + struct bpf_map *map, struct xdp_buff *xdp, + struct bpf_map *ex_map, u32 flags) { switch (map->map_type) { case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: - return dev_map_enqueue(fwd, xdp, dev_rx); + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) + return dev_map_enqueue(fwd, xdp, dev_rx); + else + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, flags); case BPF_MAP_TYPE_CPUMAP: return cpu_map_enqueue(fwd, xdp, dev_rx); case BPF_MAP_TYPE_XSKMAP: @@ -3578,12 +3585,14 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); struct bpf_map *map = READ_ONCE(ri->map); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (unlikely(!map)) { @@ -3595,7 +3604,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = dev_xdp_enqueue(fwd, xdp, dev); } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, ri->flags); } if (unlikely(err)) @@ -3609,6 +3618,55 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, } EXPORT_SYMBOL_GPL(xdp_do_redirect); +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, + struct bpf_prog *xdp_prog, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) + +{ + struct bpf_dtab_netdev *dst; + struct sk_buff *nskb; + bool exclude_ingress; + u32 key, next_key; + void *fwd; + int err; + + /* Get first key from forward map */ + err = map->ops->map_get_next_key(map, NULL, &key); + if (err) + return err; + + exclude_ingress = !!(flags & BPF_F_EXCLUDE_INGRESS); + + for (;;) { + fwd = __xdp_map_lookup_elem(map, key); + if (fwd) { + dst = (struct bpf_dtab_netdev *)fwd; + if (dev_in_exclude_map(dst, ex_map, + exclude_ingress ? dev->ifindex : 0)) + goto find_next; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -ENOMEM; + + /* Try forword next one no mater the current forward + * succeed or not */ + dev_map_generic_redirect(dst, nskb, xdp_prog); + } + +find_next: + err = map->ops->map_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + consume_skb(skb); + return 0; +} + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, @@ -3616,19 +3674,30 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, struct bpf_map *map) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err = 0; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (map->map_type == BPF_MAP_TYPE_DEVMAP || map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { - struct bpf_dtab_netdev *dst = fwd; + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) { + struct bpf_dtab_netdev *dst = fwd; + + err = dev_map_generic_redirect(dst, skb, xdp_prog); + } else { + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, + ex_map, ri->flags); + } - err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { @@ -3742,6 +3811,36 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { .arg3_type = ARG_ANYTHING, }; +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, + struct bpf_map *, ex_map, u64, flags) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + + /* Limit ex_map type to DEVMAP_HASH to get better performance */ + if (unlikely(!map || ex_map->map_type != BPF_MAP_TYPE_DEVMAP_HASH || + flags & ~BPF_F_EXCLUDE_INGRESS)) + return XDP_ABORTED; + + ri->tgt_index = 0; + /* Set the tgt_value to NULL to distinguish with bpf_xdp_redirect_map */ + ri->tgt_value = NULL; + ri->flags = flags; + ri->ex_map = ex_map; + + WRITE_ONCE(ri->map, map); + + return XDP_REDIRECT; +} + +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { + .func = bpf_xdp_redirect_map_multi, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_CONST_MAP_PTR, + .arg2_type = ARG_CONST_MAP_PTR, + .arg3_type = ARG_ANYTHING, +}; + static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, unsigned long off, unsigned long len) { @@ -6473,6 +6572,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_redirect_proto; case BPF_FUNC_redirect_map: return &bpf_xdp_redirect_map_proto; + case BPF_FUNC_redirect_map_multi: + return &bpf_xdp_redirect_map_multi_proto; case BPF_FUNC_xdp_adjust_tail: return &bpf_xdp_adjust_tail_proto; case BPF_FUNC_fib_lookup: diff --git a/net/core/xdp.c b/net/core/xdp.c index 3c45f99e26d5..9b43d0a208a7 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -476,3 +476,32 @@ void xdp_warn(const char *msg, const char *func, const int line) WARN(1, "XDP_WARN: %s(line:%d): %s\n", func, line, msg); }; EXPORT_SYMBOL_GPL(xdp_warn); + +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) +{ + unsigned int headroom, totalsize; + struct xdp_frame *nxdpf; + struct page *page; + void *addr; + + headroom = xdpf->headroom + sizeof(*xdpf); + totalsize = headroom + xdpf->len; + + if (unlikely(totalsize > PAGE_SIZE)) + return NULL; + page = dev_alloc_page(); + if (!page) + return NULL; + addr = page_to_virt(page); + + memcpy(addr, xdpf, totalsize); + + nxdpf = addr; + nxdpf->data = addr + headroom; + nxdpf->frame_sz = PAGE_SIZE; + nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0; + nxdpf->mem.id = 0; + + return nxdpf; +} +EXPORT_SYMBOL_GPL(xdpf_clone); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 5e386389913a..54d5b42d282c 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3320,6 +3320,26 @@ union bpf_attr { * A non-negative value equal to or less than *size* on success, * or a negative error in case of failure. * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The frowarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3464,6 +3484,7 @@ union bpf_attr { FN(skc_to_tcp_request_sock), \ FN(skc_to_udp6_sock), \ FN(get_task_stack), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -3625,6 +3646,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv8 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test 2020-07-15 13:08 ` [PATCHv8 " Hangbin Liu 2020-07-15 13:08 ` [PATCHv8 bpf-next 1/3] " Hangbin Liu @ 2020-07-15 13:08 ` Hangbin Liu 2020-07-15 13:08 ` [PATCHv8 bpf-next 3/3] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2020-08-26 13:19 ` [PATCHv9 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu 3 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-07-15 13:08 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu This is a sample for xdp multicast. In the sample we could forward all packets between given interfaces. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v6-v8: no update v5: add a null_map as we have strict the arg2 to ARG_CONST_MAP_PTR. Move the testing part to bpf selftest in next patch. v4: no update. v3: add rxcnt map to show the packet transmit speed. v2: no update. --- samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 57 ++++++++ samples/bpf/xdp_redirect_map_multi_user.c | 166 ++++++++++++++++++++++ 3 files changed, 226 insertions(+) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index f87ee02073ba..fddca6cb76b8 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map tprogs-y += per_socket_stats_example tprogs-y += xdp_redirect tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_map_multi tprogs-y += xdp_redirect_cpu tprogs-y += xdp_monitor tprogs-y += xdp_rxq_info @@ -97,6 +98,7 @@ test_map_in_map-objs := test_map_in_map_user.o per_socket_stats_example-objs := cookie_uid_helper_example.o xdp_redirect-objs := xdp_redirect_user.o xdp_redirect_map-objs := xdp_redirect_map_user.o +xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o xdp_redirect_cpu-objs := bpf_load.o xdp_redirect_cpu_user.o xdp_monitor-objs := bpf_load.o xdp_monitor_user.o xdp_rxq_info-objs := xdp_rxq_info_user.o @@ -156,6 +158,7 @@ always-y += tcp_tos_reflect_kern.o always-y += tcp_dumpstats_kern.o always-y += xdp_redirect_kern.o always-y += xdp_redirect_map_kern.o +always-y += xdp_redirect_map_multi_kern.o always-y += xdp_redirect_cpu_kern.o always-y += xdp_monitor_kern.o always-y += xdp_rxq_info_kern.o diff --git a/samples/bpf/xdp_redirect_map_multi_kern.c b/samples/bpf/xdp_redirect_map_multi_kern.c new file mode 100644 index 000000000000..cc7ebaedf55a --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_kern.c @@ -0,0 +1,57 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <uapi/linux/bpf.h> +#include <bpf/bpf_helpers.h> + +struct bpf_map_def SEC("maps") forward_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 256, +}; + +struct bpf_map_def SEC("maps") null_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(u32), + .value_size = sizeof(int), + .max_entries = 1, +}; + +struct bpf_map_def SEC("maps") rxcnt = { + .type = BPF_MAP_TYPE_PERCPU_ARRAY, + .key_size = sizeof(u32), + .value_size = sizeof(long), + .max_entries = 1, +}; + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + long *value; + u32 key = 0; + + /* count packet in global counter */ + value = bpf_map_lookup_elem(&rxcnt, &key); + if (value) + *value += 1; + + return bpf_redirect_map_multi(&forward_map, &null_map, + BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_dummy") +int xdp_pass(struct xdp_md *ctx) +{ + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/samples/bpf/xdp_redirect_map_multi_user.c b/samples/bpf/xdp_redirect_map_multi_user.c new file mode 100644 index 000000000000..49f44c91b672 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_user.c @@ -0,0 +1,166 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; +static int rxcnt; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void poll_stats(int interval) +{ + unsigned int nr_cpus = bpf_num_possible_cpus(); + __u64 values[nr_cpus], prev[nr_cpus]; + + memset(prev, 0, sizeof(prev)); + + while (1) { + __u64 sum = 0; + __u32 key = 0; + int i; + + sleep(interval); + assert(bpf_map_lookup_elem(rxcnt, &key, values) == 0); + for (i = 0; i < nr_cpus; i++) + sum += (values[i] - prev[i]); + if (sum) + printf("Forwarding %10llu pkt/s\n", sum / interval); + memcpy(prev, values, sizeof(values)); + } +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n", + prog); +} + +int main(int argc, char **argv) +{ + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + int prog_fd, forward_map; + int i, ret, opt, ifindex; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNF")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) + xdp_flags |= XDP_FLAGS_DRV_MODE; + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + return 1; + + forward_map = bpf_object__find_map_fd_by_name(obj, "forward_map"); + rxcnt = bpf_object__find_map_fd_by_name(obj, "rxcnt"); + + if (forward_map < 0 || rxcnt < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + return 1; + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* Add all the interfaces to group all */ + ret = bpf_map_update_elem(forward_map, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + } + + poll_stats(2); + + return 0; + +err_out: + return 1; +} -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv8 bpf-next 3/3] selftests/bpf: add xdp_redirect_multi test 2020-07-15 13:08 ` [PATCHv8 " Hangbin Liu 2020-07-15 13:08 ` [PATCHv8 bpf-next 1/3] " Hangbin Liu 2020-07-15 13:08 ` [PATCHv8 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu @ 2020-07-15 13:08 ` Hangbin Liu 2020-08-26 13:19 ` [PATCHv9 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu 3 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-07-15 13:08 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Hangbin Liu Add a bpf selftest for new helper xdp_redirect_map_multi(). In this test we have 3 forward groups groups and 1 exclude group. The test will redirect each interface's packets to all the interfaces in the forward group, and exclude the interface in exclude map. We will also test both DEVMAP and DEVMAP_HASH with xdp generic and drv. For more test details, you can find it in the test script. Here is the test result. ]# ./test_xdp_redirect_multi.sh Pass: xdpgeneric arp ns1-2 Pass: xdpgeneric arp ns1-3 Pass: xdpgeneric arp ns1-4 Pass: xdpgeneric ping ns1-2 Pass: xdpgeneric ping ns1-3 Pass: xdpgeneric ping ns1-4 Pass: xdpgeneric ping6 ns2-1 Pass: xdpgeneric ping6 ns2-3 Pass: xdpgeneric ping6 ns2-4 Pass: xdpdrv arp ns1-2 Pass: xdpdrv arp ns1-3 Pass: xdpdrv arp ns1-4 Pass: xdpdrv ping ns1-2 Pass: xdpdrv ping ns1-3 Pass: xdpdrv ping ns1-4 Pass: xdpdrv ping6 ns2-1 Pass: xdpdrv ping6 ns2-3 Pass: xdpdrv ping6 ns2-4 Summary: PASS 18, FAIL 0 Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- tools/testing/selftests/bpf/Makefile | 4 +- .../bpf/progs/xdp_redirect_multi_kern.c | 90 +++++++++ .../selftests/bpf/test_xdp_redirect_multi.sh | 164 +++++++++++++++++ .../selftests/bpf/xdp_redirect_multi.c | 173 ++++++++++++++++++ 4 files changed, 430 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index e7a8cf83ba48..c2434a3d2d15 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -51,6 +51,7 @@ TEST_FILES = test_lwt_ip_encap.o \ # Order correspond to 'make run_tests' order TEST_PROGS := test_kmod.sh \ test_xdp_redirect.sh \ + test_xdp_redirect_multi.sh \ test_xdp_meta.sh \ test_xdp_veth.sh \ test_offload.py \ @@ -79,7 +80,8 @@ TEST_PROGS_EXTENDED := with_addr.sh \ # Compile but not part of 'make run_tests' TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ - test_lirc_mode2_user xdping test_cpp runqslower bench + test_lirc_mode2_user xdping test_cpp runqslower bench \ + xdp_redirect_multi TEST_CUSTOM_PROGS = urandom_read diff --git a/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c new file mode 100644 index 000000000000..70b8476b9df3 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c @@ -0,0 +1,90 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <string.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ip.h> +#include <linux/ipv6.h> + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_endian.h> + +struct bpf_map_def SEC("maps") forward_map_v4 = { + .type = BPF_MAP_TYPE_DEVMAP, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 4096, +}; + +struct bpf_map_def SEC("maps") forward_map_v6 = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") forward_map_all = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") exclude_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 128, +}; + +struct bpf_map_def SEC("maps") null_map = { + .type = BPF_MAP_TYPE_DEVMAP_HASH, + .key_size = sizeof(__u32), + .value_size = sizeof(int), + .max_entries = 1, +}; + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + struct ethhdr *eth = data; + __u16 h_proto; + __u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + h_proto = eth->h_proto; + + if (h_proto == bpf_htons(ETH_P_IP)) + return bpf_redirect_map_multi(&forward_map_v4, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else if (h_proto == bpf_htons(ETH_P_IPV6)) + return bpf_redirect_map_multi(&forward_map_v6, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else + return bpf_redirect_map_multi(&forward_map_all, &null_map, + BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_dummy") +int xdp_pass(struct xdp_md *ctx) +{ + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh new file mode 100755 index 000000000000..f4f8f751854e --- /dev/null +++ b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh @@ -0,0 +1,164 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Test topology: +# - - - - - - - - - - - - - - - - - - - - - - - - - +# | veth1 veth2 veth3 veth4 | ... init net +# - -| - - - - - - | - - - - - - | - - - - - - | - - +# --------- --------- --------- --------- +# | veth0 | | veth0 | | veth0 | | veth0 | ... +# --------- --------- --------- --------- +# ns1 ns2 ns3 ns4 +# +# Forward multicast groups: +# Forward group all has interfaces: veth1, veth2, veth3, veth4, ... (All traffic except IPv4, IPv6) +# Forward group v4 has interfaces: veth1, veth3, veth4, ... (For IPv4 traffic only) +# Forward group v6 has interfaces: veth2, veth3, veth4, ... (For IPv6 traffic only) +# Exclude Groups: +# Exclude group: veth3 (assume ns3 is in black list) +# +# Test modules: +# XDP modes: generic, native +# map types: group v4 use DEVMAP, others use DEVMAP_HASH +# +# Test cases: +# ARP(we didn't block ARP for ns3): +# ns1 -> gw: ns2, ns3, ns4 should receive the arp request +# IPv4: +# ns1 -> ns2 (fail), ns1 -> ns3 (fail), ns1 -> ns4 (pass) +# IPv6 +# ns2 -> ns1 (fail), ns2 -> ns3 (fail), ns2 -> ns4 (pass) +# + + +# netns numbers +NUM=4 +IFACES="" +DRV_MODE="xdpgeneric xdpdrv" +PASS=0 +FAIL=0 + +test_pass() +{ + echo "Pass: $@" + PASS=$((PASS + 1)) +} + +test_fail() +{ + echo "fail: $@" + FAIL=$((FAIL + 1)) +} + +clean_up() +{ + for i in $(seq $NUM); do + ip link del veth$i 2> /dev/null + ip netns del ns$i 2> /dev/null + done + rm -f xdp_redirect_*.log arp_ns*.log +} + +# Kselftest framework requirement - SKIP code is 4. +check_env() +{ + ip link set dev lo xdpgeneric off &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without the ip xdpgeneric support" + exit 4 + fi + + which tcpdump &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without tcpdump" + exit 4 + fi +} + +setup_ns() +{ + local mode=$1 + IFACES="" + + for i in $(seq $NUM); do + ip netns add ns$i + ip link add veth$i type veth peer name veth0 netns ns$i + ip link set veth$i up + ip -n ns$i link set veth0 up + + ip -n ns$i addr add 192.0.2.$i/24 dev veth0 + ip -n ns$i addr add 2001:db8::$i/64 dev veth0 + ip -n ns$i link set veth0 $mode obj \ + xdp_redirect_multi_kern.o sec xdp_dummy &> /dev/null || \ + { test_fail "Unable to load dummy xdp" && exit 1; } + IFACES="$IFACES veth$i" + done +} + +do_ping_tests() +{ + local mode=$1 + + # arp test + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -i veth0 -nn -l -e &> arp_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-2_${mode}.log && \ + test_pass "$mode arp ns1-2" || test_fail "$mode arp ns1-2" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-3_${mode}.log && \ + test_pass "$mode arp ns1-3" || test_fail "$mode arp ns1-3" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-4_${mode}.log && \ + test_pass "$mode arp ns1-4" || test_fail "$mode arp ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-2" || test_pass "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-3" || test_pass "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" + + # ping6 test + ip netns exec ns2 ping6 2001:db8::1 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-1" || test_pass "$mode ping6 ns2-1" + ip netns exec ns2 ping6 2001:db8::3 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-3" || test_pass "$mode ping6 ns2-3" + ip netns exec ns2 ping6 2001:db8::4 -c 4 &> /dev/null && \ + test_pass "$mode ping6 ns2-4" || test_fail "$mode ping6 ns2-4" +} + +do_tests() +{ + local mode=$1 + local drv_p + + [ ${mode} == "xdpdrv" ] && drv_p="-N" || drv_p="-S" + + # run `ulimit -l unlimited` if you got errors like + # libbpf: Error in bpf_object__probe_global_data():Operation not permitted(1). + ./xdp_redirect_multi $drv_p $IFACES &> xdp_redirect_${mode}.log & + xdp_pid=$! + sleep 10 + + do_ping_tests $mode + + kill $xdp_pid +} + +trap clean_up 0 2 3 6 9 + +check_env + +for mode in ${DRV_MODE}; do + setup_ns $mode + do_tests $mode + sleep 10 + clean_up + sleep 5 +done + +echo "Summary: PASS $PASS, FAIL $FAIL" +[ $FAIL -eq 0 ] && exit 0 || exit 1 diff --git a/tools/testing/selftests/bpf/xdp_redirect_multi.c b/tools/testing/selftests/bpf/xdp_redirect_multi.c new file mode 100644 index 000000000000..5626005cb679 --- /dev/null +++ b/tools/testing/selftests/bpf/xdp_redirect_multi.c @@ -0,0 +1,173 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n", + prog); +} + +int main(int argc, char **argv) +{ + int prog_fd, group_all, group_v4, group_v6, exclude; + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + int i, ret, opt, ifindex; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNF")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) + xdp_flags |= XDP_FLAGS_DRV_MODE; + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + return 1; + + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_all"); + group_v4 = bpf_object__find_map_fd_by_name(obj, "forward_map_v4"); + group_v6 = bpf_object__find_map_fd_by_name(obj, "forward_map_v6"); + exclude = bpf_object__find_map_fd_by_name(obj, "exclude_map"); + + if (group_all < 0 || group_v4 < 0 || group_v6 < 0 || exclude < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + return 1; + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* Add all the interfaces to group all */ + ret = bpf_map_update_elem(group_all, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* For testing: remove the 1st interfaces from group v6 */ + if (i != 0) { + ret = bpf_map_update_elem(group_v6, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: remove the 2nd interfaces from group v4 */ + if (i != 1) { + ret = bpf_map_update_elem(group_v4, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: add the 3rd interfaces to exclude map */ + if (i == 2) { + ret = bpf_map_update_elem(exclude, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + } + + /* sleep some time for testing */ + sleep(999); + + return 0; + +err_out: + return 1; +} -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv9 bpf-next 0/5] xdp: add a new helper for dev map multicast support 2020-07-15 13:08 ` [PATCHv8 " Hangbin Liu ` (2 preceding siblings ...) 2020-07-15 13:08 ` [PATCHv8 bpf-next 3/3] selftests/bpf: add xdp_redirect_multi test Hangbin Liu @ 2020-08-26 13:19 ` Hangbin Liu 2020-08-26 13:19 ` [PATCHv9 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu ` (5 more replies) 3 siblings, 6 replies; 219+ messages in thread From: Hangbin Liu @ 2020-08-26 13:19 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko B, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because there may have multi interfaces you want to exclude. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. The 1st patch add a new bpf arg to allow NULL map pointer. The 2nd patch add the new bpf_redirect_map_multi() helper. The 3rd and 4th patches are for usage sample and testing purpose, there is no effort has been made on performance optimisation. The 5th patch added some verifier test for new bpf arg ARG_CONST_MAP_PTR_OR_NULL I did same tests with pktgen(pkt size 64) to compire with xdp_redirect_map(). Here is the test result(the veth peer has a dummy xdp program with XDP_DROP directly): Version | Test | Native | Generic 5.9 rc1 | xdp_redirect_map i40e->i40e | 10.4M | 1.9M 5.9 rc1 | xdp_redirect_map i40e->veth | 14.2M | 2.2M 5.9 rc1 + patch | xdp_redirect_map i40e->i40e | 10.3M | 1.9M 5.9 rc1 + patch | xdp_redirect_map i40e->veth | 14.2M | 2.2M 5.9 rc1 + patch | xdp_redirect_map_multi i40e->i40e | 8.0M | 1.5M 5.9 rc1 + patch | xdp_redirect_map_multi i40e->veth | 11.2M | 1.6M 5.9 rc1 + patch | xdp_redirect_map_multi i40e->i40e+veth | 3.5M | 1.1M The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop the map and do clone skb/xdpf. The generic path is slower than native path as we send skbs by pktgen. So the result looks reasonable. There is some performance improvement for veth port compared with 5.8 rc1. Last but not least, thanks a lot to Toke, Jesper, Jiri and Eelco for suggestions and help on implementation. [0] https://xdp-project.net/#Handling-multicast v9: Merge the new bpf argument type ARG_CONST_MAP_PTR_OR_NULL to this patchset v8: a) Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). f) Split the tests from sample and add a bpf kernel selftest patch. v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Hangbin Liu (5): bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL xdp: add a new helper for dev map multicast support sample/bpf: add xdp_redirect_map_multicast test selftests/bpf: add xdp_redirect_multi test selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL include/linux/bpf.h | 22 +++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 27 +++ kernel/bpf/devmap.c | 124 +++++++++++++ kernel/bpf/verifier.c | 29 ++- net/core/filter.c | 112 +++++++++++- net/core/xdp.c | 29 +++ samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 43 +++++ samples/bpf/xdp_redirect_map_multi_user.c | 166 +++++++++++++++++ tools/include/uapi/linux/bpf.h | 27 +++ tools/testing/selftests/bpf/Makefile | 4 +- .../bpf/progs/xdp_redirect_multi_kern.c | 77 ++++++++ tools/testing/selftests/bpf/test_verifier.c | 22 ++- .../selftests/bpf/test_xdp_redirect_multi.sh | 164 +++++++++++++++++ .../testing/selftests/bpf/verifier/map_ptr.c | 70 +++++++ .../selftests/bpf/xdp_redirect_multi.c | 173 ++++++++++++++++++ 18 files changed, 1080 insertions(+), 14 deletions(-) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c -- 2.25.4 ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv9 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL 2020-08-26 13:19 ` [PATCHv9 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2020-08-26 13:19 ` Hangbin Liu 2020-08-28 21:56 ` Daniel Borkmann 2020-08-26 13:19 ` [PATCHv9 bpf-next 2/5] xdp: add a new helper for dev map multicast support Hangbin Liu ` (4 subsequent siblings) 5 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-08-26 13:19 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko B, Hangbin Liu Add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL which could be used when we want to allow NULL pointer for map parameter. The bpf helper need to take care and check if the map is NULL when use this type. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v9: merge the patch from [1] in to this series. v1-v8: no this patch [1] https://lore.kernel.org/bpf/20200715070001.2048207-1-liuhangbin@gmail.com/ --- include/linux/bpf.h | 2 ++ kernel/bpf/verifier.c | 23 ++++++++++++++++------- 2 files changed, 18 insertions(+), 7 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index a6131d95e31e..cb40a1281ea2 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -276,6 +276,7 @@ enum bpf_arg_type { ARG_PTR_TO_ALLOC_MEM, /* pointer to dynamically allocated memory */ ARG_PTR_TO_ALLOC_MEM_OR_NULL, /* pointer to dynamically allocated memory or NULL */ ARG_CONST_ALLOC_SIZE_OR_ZERO, /* number of allocated bytes requested */ + ARG_CONST_MAP_PTR_OR_NULL, /* const argument used as pointer to bpf_map or NULL */ }; /* type of values returned from helper functions */ @@ -369,6 +370,7 @@ enum bpf_reg_type { PTR_TO_RDONLY_BUF_OR_NULL, /* reg points to a readonly buffer or NULL */ PTR_TO_RDWR_BUF, /* reg points to a read/write buffer */ PTR_TO_RDWR_BUF_OR_NULL, /* reg points to a read/write buffer or NULL */ + CONST_PTR_TO_MAP_OR_NULL, /* reg points to struct bpf_map or NULL */ }; /* The information passed from prog-specific *_is_valid_access diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 7e5908b83ec7..53a84335a8fd 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -411,7 +411,8 @@ static bool reg_type_may_be_null(enum bpf_reg_type type) type == PTR_TO_BTF_ID_OR_NULL || type == PTR_TO_MEM_OR_NULL || type == PTR_TO_RDONLY_BUF_OR_NULL || - type == PTR_TO_RDWR_BUF_OR_NULL; + type == PTR_TO_RDWR_BUF_OR_NULL || + type == CONST_PTR_TO_MAP_OR_NULL; } static bool reg_may_point_to_spin_lock(const struct bpf_reg_state *reg) @@ -427,7 +428,8 @@ static bool reg_type_may_be_refcounted_or_null(enum bpf_reg_type type) type == PTR_TO_TCP_SOCK || type == PTR_TO_TCP_SOCK_OR_NULL || type == PTR_TO_MEM || - type == PTR_TO_MEM_OR_NULL; + type == PTR_TO_MEM_OR_NULL || + type == CONST_PTR_TO_MAP_OR_NULL; } static bool arg_type_may_be_refcounted(enum bpf_arg_type type) @@ -509,6 +511,7 @@ static const char * const reg_type_str[] = { [PTR_TO_RDONLY_BUF_OR_NULL] = "rdonly_buf_or_null", [PTR_TO_RDWR_BUF] = "rdwr_buf", [PTR_TO_RDWR_BUF_OR_NULL] = "rdwr_buf_or_null", + [CONST_PTR_TO_MAP_OR_NULL] = "map_ptr_or_null", }; static char slot_type_char[] = { @@ -3957,9 +3960,13 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, expected_type = SCALAR_VALUE; if (type != expected_type) goto err_type; - } else if (arg_type == ARG_CONST_MAP_PTR) { + } else if (arg_type == ARG_CONST_MAP_PTR || + arg_type == ARG_CONST_MAP_PTR_OR_NULL) { expected_type = CONST_PTR_TO_MAP; - if (type != expected_type) + if (register_is_null(reg) && + arg_type == ARG_CONST_MAP_PTR_OR_NULL) + /* final test in check_stack_boundary() */; + else if (type != expected_type) goto err_type; } else if (arg_type == ARG_PTR_TO_CTX || arg_type == ARG_PTR_TO_CTX_OR_NULL) { @@ -4076,9 +4083,9 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, return -EFAULT; } - if (arg_type == ARG_CONST_MAP_PTR) { - /* bpf_map_xxx(map_ptr) call: remember that map_ptr */ - meta->map_ptr = reg->map_ptr; + if (arg_type == ARG_CONST_MAP_PTR || + arg_type == ARG_CONST_MAP_PTR_OR_NULL) { + meta->map_ptr = register_is_null(reg) ? NULL : reg->map_ptr; } else if (arg_type == ARG_PTR_TO_MAP_KEY) { /* bpf_map_xxx(..., map_ptr, ..., key) call: * check that [key, key + map->key_size) are within @@ -6977,6 +6984,8 @@ static void mark_ptr_or_null_reg(struct bpf_func_state *state, reg->type = PTR_TO_RDONLY_BUF; } else if (reg->type == PTR_TO_RDWR_BUF_OR_NULL) { reg->type = PTR_TO_RDWR_BUF; + } else if (reg->type == CONST_PTR_TO_MAP_OR_NULL) { + reg->type = CONST_PTR_TO_MAP; } if (is_null) { /* We don't need id and ref_obj_id from this point -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv9 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL 2020-08-26 13:19 ` [PATCHv9 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2020-08-28 21:56 ` Daniel Borkmann 2020-09-03 9:15 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: Daniel Borkmann @ 2020-08-28 21:56 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Lorenzo Bianconi, David Ahern, Andrii Nakryiko B On 8/26/20 3:19 PM, Hangbin Liu wrote: > Add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL which could be > used when we want to allow NULL pointer for map parameter. The bpf helper > need to take care and check if the map is NULL when use this type. > > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > --- > > v9: merge the patch from [1] in to this series. > v1-v8: no this patch > > [1] https://lore.kernel.org/bpf/20200715070001.2048207-1-liuhangbin@gmail.com/ > --- > include/linux/bpf.h | 2 ++ > kernel/bpf/verifier.c | 23 ++++++++++++++++------- > 2 files changed, 18 insertions(+), 7 deletions(-) > > diff --git a/include/linux/bpf.h b/include/linux/bpf.h > index a6131d95e31e..cb40a1281ea2 100644 > --- a/include/linux/bpf.h > +++ b/include/linux/bpf.h > @@ -276,6 +276,7 @@ enum bpf_arg_type { > ARG_PTR_TO_ALLOC_MEM, /* pointer to dynamically allocated memory */ > ARG_PTR_TO_ALLOC_MEM_OR_NULL, /* pointer to dynamically allocated memory or NULL */ > ARG_CONST_ALLOC_SIZE_OR_ZERO, /* number of allocated bytes requested */ > + ARG_CONST_MAP_PTR_OR_NULL, /* const argument used as pointer to bpf_map or NULL */ > }; > > /* type of values returned from helper functions */ > @@ -369,6 +370,7 @@ enum bpf_reg_type { > PTR_TO_RDONLY_BUF_OR_NULL, /* reg points to a readonly buffer or NULL */ > PTR_TO_RDWR_BUF, /* reg points to a read/write buffer */ > PTR_TO_RDWR_BUF_OR_NULL, /* reg points to a read/write buffer or NULL */ > + CONST_PTR_TO_MAP_OR_NULL, /* reg points to struct bpf_map or NULL */ Why is this needed & where do you assign it? Also, if we were to use CONST_PTR_TO_MAP_OR_NULL then it's missing few things like rejection of arithmetic in adjust_ptr_min_max_vals(), handling in pruning logic etc. Either way, given no helper currently returns CONST_PTR_TO_MAP_OR_NULL, the ARG_CONST_MAP_PTR_OR_NULL one should be sufficient, so I'd suggest to remove the CONST_PTR_TO_MAP_OR_NULL bits. > }; > > /* The information passed from prog-specific *_is_valid_access > diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c > index 7e5908b83ec7..53a84335a8fd 100644 > --- a/kernel/bpf/verifier.c > +++ b/kernel/bpf/verifier.c > @@ -411,7 +411,8 @@ static bool reg_type_may_be_null(enum bpf_reg_type type) > type == PTR_TO_BTF_ID_OR_NULL || > type == PTR_TO_MEM_OR_NULL || > type == PTR_TO_RDONLY_BUF_OR_NULL || > - type == PTR_TO_RDWR_BUF_OR_NULL; > + type == PTR_TO_RDWR_BUF_OR_NULL || > + type == CONST_PTR_TO_MAP_OR_NULL; > } > > static bool reg_may_point_to_spin_lock(const struct bpf_reg_state *reg) > @@ -427,7 +428,8 @@ static bool reg_type_may_be_refcounted_or_null(enum bpf_reg_type type) > type == PTR_TO_TCP_SOCK || > type == PTR_TO_TCP_SOCK_OR_NULL || > type == PTR_TO_MEM || > - type == PTR_TO_MEM_OR_NULL; > + type == PTR_TO_MEM_OR_NULL || > + type == CONST_PTR_TO_MAP_OR_NULL; > } > > static bool arg_type_may_be_refcounted(enum bpf_arg_type type) > @@ -509,6 +511,7 @@ static const char * const reg_type_str[] = { > [PTR_TO_RDONLY_BUF_OR_NULL] = "rdonly_buf_or_null", > [PTR_TO_RDWR_BUF] = "rdwr_buf", > [PTR_TO_RDWR_BUF_OR_NULL] = "rdwr_buf_or_null", > + [CONST_PTR_TO_MAP_OR_NULL] = "map_ptr_or_null", > }; > > static char slot_type_char[] = { > @@ -3957,9 +3960,13 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, > expected_type = SCALAR_VALUE; > if (type != expected_type) > goto err_type; > - } else if (arg_type == ARG_CONST_MAP_PTR) { > + } else if (arg_type == ARG_CONST_MAP_PTR || > + arg_type == ARG_CONST_MAP_PTR_OR_NULL) { > expected_type = CONST_PTR_TO_MAP; > - if (type != expected_type) > + if (register_is_null(reg) && > + arg_type == ARG_CONST_MAP_PTR_OR_NULL) > + /* final test in check_stack_boundary() */; Where is that test in the code? Copy-paste leftover comment? > + else if (type != expected_type) > goto err_type; > } else if (arg_type == ARG_PTR_TO_CTX || > arg_type == ARG_PTR_TO_CTX_OR_NULL) { > @@ -4076,9 +4083,9 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, > return -EFAULT; > } > > - if (arg_type == ARG_CONST_MAP_PTR) { > - /* bpf_map_xxx(map_ptr) call: remember that map_ptr */ > - meta->map_ptr = reg->map_ptr; > + if (arg_type == ARG_CONST_MAP_PTR || > + arg_type == ARG_CONST_MAP_PTR_OR_NULL) { > + meta->map_ptr = register_is_null(reg) ? NULL : reg->map_ptr; > } else if (arg_type == ARG_PTR_TO_MAP_KEY) { > /* bpf_map_xxx(..., map_ptr, ..., key) call: > * check that [key, key + map->key_size) are within > @@ -6977,6 +6984,8 @@ static void mark_ptr_or_null_reg(struct bpf_func_state *state, > reg->type = PTR_TO_RDONLY_BUF; > } else if (reg->type == PTR_TO_RDWR_BUF_OR_NULL) { > reg->type = PTR_TO_RDWR_BUF; > + } else if (reg->type == CONST_PTR_TO_MAP_OR_NULL) { > + reg->type = CONST_PTR_TO_MAP; > } > if (is_null) { > /* We don't need id and ref_obj_id from this point > ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv9 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL 2020-08-28 21:56 ` Daniel Borkmann @ 2020-09-03 9:15 ` Hangbin Liu 0 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-09-03 9:15 UTC (permalink / raw) To: Daniel Borkmann Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Lorenzo Bianconi, David Ahern, Andrii Nakryiko B Hi Daniel, Sorry for the late reply. I was in PTO last few days. On Fri, Aug 28, 2020 at 11:56:37PM +0200, Daniel Borkmann wrote: > On 8/26/20 3:19 PM, Hangbin Liu wrote: > > Add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL which could be > > used when we want to allow NULL pointer for map parameter. The bpf helper > > need to take care and check if the map is NULL when use this type. > > > > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > > --- > > > > v9: merge the patch from [1] in to this series. > > v1-v8: no this patch > > > > [1] https://lore.kernel.org/bpf/20200715070001.2048207-1-liuhangbin@gmail.com/ > > --- > > include/linux/bpf.h | 2 ++ > > kernel/bpf/verifier.c | 23 ++++++++++++++++------- > > 2 files changed, 18 insertions(+), 7 deletions(-) > > > > diff --git a/include/linux/bpf.h b/include/linux/bpf.h > > index a6131d95e31e..cb40a1281ea2 100644 > > --- a/include/linux/bpf.h > > +++ b/include/linux/bpf.h > > @@ -276,6 +276,7 @@ enum bpf_arg_type { > > ARG_PTR_TO_ALLOC_MEM, /* pointer to dynamically allocated memory */ > > ARG_PTR_TO_ALLOC_MEM_OR_NULL, /* pointer to dynamically allocated memory or NULL */ > > ARG_CONST_ALLOC_SIZE_OR_ZERO, /* number of allocated bytes requested */ > > + ARG_CONST_MAP_PTR_OR_NULL, /* const argument used as pointer to bpf_map or NULL */ > > }; > > /* type of values returned from helper functions */ > > @@ -369,6 +370,7 @@ enum bpf_reg_type { > > PTR_TO_RDONLY_BUF_OR_NULL, /* reg points to a readonly buffer or NULL */ > > PTR_TO_RDWR_BUF, /* reg points to a read/write buffer */ > > PTR_TO_RDWR_BUF_OR_NULL, /* reg points to a read/write buffer or NULL */ > > + CONST_PTR_TO_MAP_OR_NULL, /* reg points to struct bpf_map or NULL */ > > Why is this needed & where do you assign it? Also, if we were to use CONST_PTR_TO_MAP_OR_NULL > then it's missing few things like rejection of arithmetic in adjust_ptr_min_max_vals(), handling > in pruning logic etc. > > Either way, given no helper currently returns CONST_PTR_TO_MAP_OR_NULL, the ARG_CONST_MAP_PTR_OR_NULL > one should be sufficient, so I'd suggest to remove the CONST_PTR_TO_MAP_OR_NULL bits. Sorry, I misunderstand the bpf_reg_type when added it. Thanks for the comment. I will remove it. > > - } else if (arg_type == ARG_CONST_MAP_PTR) { > > + } else if (arg_type == ARG_CONST_MAP_PTR || > > + arg_type == ARG_CONST_MAP_PTR_OR_NULL) { > > expected_type = CONST_PTR_TO_MAP; > > - if (type != expected_type) > > + if (register_is_null(reg) && > > + arg_type == ARG_CONST_MAP_PTR_OR_NULL) > > + /* final test in check_stack_boundary() */; > > Where is that test in the code? Copy-paste leftover comment? Yeah... I will remove it. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv9 bpf-next 2/5] xdp: add a new helper for dev map multicast support 2020-08-26 13:19 ` [PATCHv9 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-08-26 13:19 ` [PATCHv9 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2020-08-26 13:19 ` Hangbin Liu 2020-08-28 22:05 ` Daniel Borkmann 2020-08-26 13:20 ` [PATCHv9 bpf-next 3/5] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu ` (3 subsequent siblings) 5 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-08-26 13:19 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko B, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because a physical interface can be part of a logical grouping, such as a bond device. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. The forwarding map could be DEVMAP or DEVMAP_HASH, but the exclude map *must* be DEVMAP_HASH to get better performace. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. As both bpf_xdp_redirect_map() and this new helpers are using struct bpf_redirect_info, I add a new ex_map and set tgt_value to NULL in the new helper to make a difference with bpf_xdp_redirect_map(). Also I keep the the general data path in net/core/filter.c, the native data path in kernel/bpf/devmap.c so we can use direct calls to get better performace. [0] https://xdp-project.net/#Handling-multicast Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v9: a) Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: a) Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. --- include/linux/bpf.h | 20 ++++++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 27 +++++++ kernel/bpf/devmap.c | 124 +++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++ net/core/filter.c | 112 +++++++++++++++++++++++++++-- net/core/xdp.c | 29 ++++++++ tools/include/uapi/linux/bpf.h | 27 +++++++ 9 files changed, 342 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index cb40a1281ea2..3aed07836df0 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1324,6 +1324,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx); int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, struct net_device *dev_rx); +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex); +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); bool dev_map_can_have_prog(struct bpf_map *map); @@ -1499,6 +1504,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return 0; } +static inline +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + return false; +} + +static inline +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + return 0; +} + struct sk_buff; static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, diff --git a/include/linux/filter.h b/include/linux/filter.h index 995625950cc1..583dbd4c8dce 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -612,6 +612,7 @@ struct bpf_redirect_info { u32 tgt_index; void *tgt_value; struct bpf_map *map; + struct bpf_map *ex_map; u32 kern_flags; }; diff --git a/include/net/xdp.h b/include/net/xdp.h index 3814fb631d52..8453d477bb22 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -132,6 +132,7 @@ void xdp_warn(const char *msg, const char *func, const int line); #define XDP_WARN(msg) xdp_warn(msg, __func__, __LINE__) struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); static inline void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 0388bc0200b0..bc31ea0b715b 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3561,6 +3561,27 @@ union bpf_attr { * On success, the strictly positive length of the string, * including the trailing NUL character. On error, a negative * value. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The frowarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3711,6 +3732,7 @@ union bpf_attr { FN(inode_storage_get), \ FN(inode_storage_delete), \ FN(d_path), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -3882,6 +3904,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 10abb06065bb..1e6695f38f78 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -512,6 +512,130 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return __xdp_enqueue(dev, xdp, dev_rx); } +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + return dev_map_get_next_key(map, key, next_key); + case BPF_MAP_TYPE_DEVMAP_HASH: + return dev_map_hash_get_next_key(map, key, next_key); + default: + break; + } + + return -ENOENT; +} + +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + if (obj->dev->ifindex == exclude_ifindex) + return true; + + if (!map) + return false; + + return __dev_map_hash_lookup_elem(map, obj->dev->ifindex) != NULL; +} + +static struct bpf_dtab_netdev *devmap_get_next_obj(struct xdp_buff *xdp, struct bpf_map *map, + struct bpf_map *ex_map, u32 *key, + u32 *next_key, int ex_ifindex) +{ + struct bpf_dtab_netdev *obj; + struct net_device *dev; + u32 *tmp_key = key; + int err; + + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + return NULL; + + for (;;) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + obj = __dev_map_lookup_elem(map, *next_key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + obj = __dev_map_hash_lookup_elem(map, *next_key); + break; + default: + break; + } + + if (!obj || dev_in_exclude_map(obj, ex_map, ex_ifindex)) + goto find_next; + + dev = obj->dev; + + if (!dev->netdev_ops->ndo_xdp_xmit) + goto find_next; + + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + if (unlikely(err)) + goto find_next; + + return obj; + +find_next: + tmp_key = next_key; + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + break; + } + + return NULL; +} + +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + struct bpf_dtab_netdev *obj = NULL, *next_obj = NULL; + struct xdp_frame *xdpf, *nxdpf; + bool last_one = false; + int ex_ifindex; + u32 key, next_key; + + ex_ifindex = flags & BPF_F_EXCLUDE_INGRESS ? dev_rx->ifindex : 0; + + /* Find first available obj */ + obj = devmap_get_next_obj(xdp, map, ex_map, NULL, &key, ex_ifindex); + if (!obj) + return 0; + + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) + return -EOVERFLOW; + + for (;;) { + /* Check if we still have one more available obj */ + next_obj = devmap_get_next_obj(xdp, map, ex_map, &key, + &next_key, ex_ifindex); + if (!next_obj) + last_one = true; + + if (last_one) { + bq_enqueue(obj->dev, xdpf, dev_rx); + return 0; + } + + nxdpf = xdpf_clone(xdpf); + if (unlikely(!nxdpf)) { + xdp_return_frame_rx_napi(xdpf); + return -ENOMEM; + } + + bq_enqueue(obj->dev, nxdpf, dev_rx); + + /* Deal with next obj */ + obj = next_obj; + key = next_key; + } +} + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 53a84335a8fd..39685a7aa157 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4267,6 +4267,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: if (func_id != BPF_FUNC_redirect_map && + func_id != BPF_FUNC_redirect_map_multi && func_id != BPF_FUNC_map_lookup_elem) goto error; break; @@ -4366,6 +4367,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, map->map_type != BPF_MAP_TYPE_XSKMAP) goto error; break; + case BPF_FUNC_redirect_map_multi: + if (map->map_type != BPF_MAP_TYPE_DEVMAP && + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + goto error; + break; case BPF_FUNC_sk_redirect_map: case BPF_FUNC_msg_redirect_map: case BPF_FUNC_sock_map_update: diff --git a/net/core/filter.c b/net/core/filter.c index 47eef9a0be6a..aa10ae72809b 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3539,12 +3539,19 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { }; static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) + struct bpf_map *map, struct xdp_buff *xdp, + struct bpf_map *ex_map, u32 flags) { switch (map->map_type) { case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: - return dev_map_enqueue(fwd, xdp, dev_rx); + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) + return dev_map_enqueue(fwd, xdp, dev_rx); + else + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, flags); case BPF_MAP_TYPE_CPUMAP: return cpu_map_enqueue(fwd, xdp, dev_rx); case BPF_MAP_TYPE_XSKMAP: @@ -3601,12 +3608,14 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); struct bpf_map *map = READ_ONCE(ri->map); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (unlikely(!map)) { @@ -3618,7 +3627,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = dev_xdp_enqueue(fwd, xdp, dev); } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, ri->flags); } if (unlikely(err)) @@ -3632,6 +3641,55 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, } EXPORT_SYMBOL_GPL(xdp_do_redirect); +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, + struct bpf_prog *xdp_prog, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) + +{ + struct bpf_dtab_netdev *dst; + struct sk_buff *nskb; + bool exclude_ingress; + u32 key, next_key; + void *fwd; + int err; + + /* Get first key from forward map */ + err = map->ops->map_get_next_key(map, NULL, &key); + if (err) + return err; + + exclude_ingress = !!(flags & BPF_F_EXCLUDE_INGRESS); + + for (;;) { + fwd = __xdp_map_lookup_elem(map, key); + if (fwd) { + dst = (struct bpf_dtab_netdev *)fwd; + if (dev_in_exclude_map(dst, ex_map, + exclude_ingress ? dev->ifindex : 0)) + goto find_next; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -ENOMEM; + + /* Try forword next one no mater the current forward + * succeed or not */ + dev_map_generic_redirect(dst, nskb, xdp_prog); + } + +find_next: + err = map->ops->map_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + consume_skb(skb); + return 0; +} + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, @@ -3639,19 +3697,30 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, struct bpf_map *map) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err = 0; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (map->map_type == BPF_MAP_TYPE_DEVMAP || map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { - struct bpf_dtab_netdev *dst = fwd; + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) { + struct bpf_dtab_netdev *dst = fwd; + + err = dev_map_generic_redirect(dst, skb, xdp_prog); + } else { + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, + ex_map, ri->flags); + } - err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { @@ -3765,6 +3834,37 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { .arg3_type = ARG_ANYTHING, }; +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, + struct bpf_map *, ex_map, u64, flags) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + + /* Limit ex_map type to DEVMAP_HASH to get better performance */ + if (unlikely(!map || + (ex_map && ex_map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) || + flags & ~BPF_F_EXCLUDE_INGRESS)) + return XDP_ABORTED; + + ri->tgt_index = 0; + /* Set the tgt_value to NULL to distinguish with bpf_xdp_redirect_map */ + ri->tgt_value = NULL; + ri->flags = flags; + ri->ex_map = ex_map; + + WRITE_ONCE(ri->map, map); + + return XDP_REDIRECT; +} + +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { + .func = bpf_xdp_redirect_map_multi, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_CONST_MAP_PTR, + .arg2_type = ARG_CONST_MAP_PTR_OR_NULL, + .arg3_type = ARG_ANYTHING, +}; + static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, unsigned long off, unsigned long len) { @@ -6833,6 +6933,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_redirect_proto; case BPF_FUNC_redirect_map: return &bpf_xdp_redirect_map_proto; + case BPF_FUNC_redirect_map_multi: + return &bpf_xdp_redirect_map_multi_proto; case BPF_FUNC_xdp_adjust_tail: return &bpf_xdp_adjust_tail_proto; case BPF_FUNC_fib_lookup: diff --git a/net/core/xdp.c b/net/core/xdp.c index 48aba933a5a8..9fd3e89768c4 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -467,3 +467,32 @@ void xdp_warn(const char *msg, const char *func, const int line) WARN(1, "XDP_WARN: %s(line:%d): %s\n", func, line, msg); }; EXPORT_SYMBOL_GPL(xdp_warn); + +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) +{ + unsigned int headroom, totalsize; + struct xdp_frame *nxdpf; + struct page *page; + void *addr; + + headroom = xdpf->headroom + sizeof(*xdpf); + totalsize = headroom + xdpf->len; + + if (unlikely(totalsize > PAGE_SIZE)) + return NULL; + page = dev_alloc_page(); + if (!page) + return NULL; + addr = page_to_virt(page); + + memcpy(addr, xdpf, totalsize); + + nxdpf = addr; + nxdpf->data = addr + headroom; + nxdpf->frame_sz = PAGE_SIZE; + nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0; + nxdpf->mem.id = 0; + + return nxdpf; +} +EXPORT_SYMBOL_GPL(xdpf_clone); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 0388bc0200b0..bc31ea0b715b 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3561,6 +3561,27 @@ union bpf_attr { * On success, the strictly positive length of the string, * including the trailing NUL character. On error, a negative * value. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The frowarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3711,6 +3732,7 @@ union bpf_attr { FN(inode_storage_get), \ FN(inode_storage_delete), \ FN(d_path), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -3882,6 +3904,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv9 bpf-next 2/5] xdp: add a new helper for dev map multicast support 2020-08-26 13:19 ` [PATCHv9 bpf-next 2/5] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2020-08-28 22:05 ` Daniel Borkmann 0 siblings, 0 replies; 219+ messages in thread From: Daniel Borkmann @ 2020-08-28 22:05 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Lorenzo Bianconi, David Ahern, Andrii Nakryiko B On 8/26/20 3:19 PM, Hangbin Liu wrote: [...] > +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, > + struct bpf_map *, ex_map, u64, flags) > +{ > + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); > + > + /* Limit ex_map type to DEVMAP_HASH to get better performance */ > + if (unlikely(!map || Why is test on !map needed when arg1_type is ARG_CONST_MAP_PTR? Verifier must guarantee that it's a valid map ptr .. are you saying this is not the case for this helper? > + (ex_map && ex_map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) || > + flags & ~BPF_F_EXCLUDE_INGRESS)) > + return XDP_ABORTED; > + > + ri->tgt_index = 0; > + /* Set the tgt_value to NULL to distinguish with bpf_xdp_redirect_map */ > + ri->tgt_value = NULL; > + ri->flags = flags; > + ri->ex_map = ex_map; > + > + WRITE_ONCE(ri->map, map); > + > + return XDP_REDIRECT; > +} > + > +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { > + .func = bpf_xdp_redirect_map_multi, > + .gpl_only = false, > + .ret_type = RET_INTEGER, > + .arg1_type = ARG_CONST_MAP_PTR, > + .arg2_type = ARG_CONST_MAP_PTR_OR_NULL, > + .arg3_type = ARG_ANYTHING, > +}; > + > static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, > unsigned long off, unsigned long len) > { > @@ -6833,6 +6933,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) > return &bpf_xdp_redirect_proto; > case BPF_FUNC_redirect_map: > return &bpf_xdp_redirect_map_proto; > + case BPF_FUNC_redirect_map_multi: > + return &bpf_xdp_redirect_map_multi_proto; > case BPF_FUNC_xdp_adjust_tail: > return &bpf_xdp_adjust_tail_proto; > case BPF_FUNC_fib_lookup: ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv9 bpf-next 3/5] sample/bpf: add xdp_redirect_map_multicast test 2020-08-26 13:19 ` [PATCHv9 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-08-26 13:19 ` [PATCHv9 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2020-08-26 13:19 ` [PATCHv9 bpf-next 2/5] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2020-08-26 13:20 ` Hangbin Liu 2020-08-26 13:20 ` [PATCHv9 bpf-next 4/5] selftests/bpf: add xdp_redirect_multi test Hangbin Liu ` (2 subsequent siblings) 5 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-08-26 13:20 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko B, Hangbin Liu This is a sample for xdp multicast. In the sample we could forward all packets between given interfaces. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v9: use NULL directly for arg2 and redefine the maps with btf format v6-v8: no update v5: add a null_map as we have strict the arg2 to ARG_CONST_MAP_PTR. Move the testing part to bpf selftest in next patch. v4: no update. v3: add rxcnt map to show the packet transmit speed. v2: no update. --- samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 43 ++++++ samples/bpf/xdp_redirect_map_multi_user.c | 166 ++++++++++++++++++++++ 3 files changed, 212 insertions(+) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index a6d3646b3818..177cab805768 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map tprogs-y += per_socket_stats_example tprogs-y += xdp_redirect tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_map_multi tprogs-y += xdp_redirect_cpu tprogs-y += xdp_monitor tprogs-y += xdp_rxq_info @@ -97,6 +98,7 @@ test_map_in_map-objs := test_map_in_map_user.o per_socket_stats_example-objs := cookie_uid_helper_example.o xdp_redirect-objs := xdp_redirect_user.o xdp_redirect_map-objs := xdp_redirect_map_user.o +xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o xdp_redirect_cpu-objs := bpf_load.o xdp_redirect_cpu_user.o xdp_monitor-objs := bpf_load.o xdp_monitor_user.o xdp_rxq_info-objs := xdp_rxq_info_user.o @@ -156,6 +158,7 @@ always-y += tcp_tos_reflect_kern.o always-y += tcp_dumpstats_kern.o always-y += xdp_redirect_kern.o always-y += xdp_redirect_map_kern.o +always-y += xdp_redirect_map_multi_kern.o always-y += xdp_redirect_cpu_kern.o always-y += xdp_monitor_kern.o always-y += xdp_rxq_info_kern.o diff --git a/samples/bpf/xdp_redirect_map_multi_kern.c b/samples/bpf/xdp_redirect_map_multi_kern.c new file mode 100644 index 000000000000..db58d56cef89 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_kern.c @@ -0,0 +1,43 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <uapi/linux/bpf.h> +#include <bpf/bpf_helpers.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 256); +} forward_map SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __type(key, u32); + __type(value, long); + __uint(max_entries, 1); +} rxcnt SEC(".maps"); + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + long *value; + u32 key = 0; + + /* count packet in global counter */ + value = bpf_map_lookup_elem(&rxcnt, &key); + if (value) + *value += 1; + + return bpf_redirect_map_multi(&forward_map, NULL, BPF_F_EXCLUDE_INGRESS); +} + +char _license[] SEC("license") = "GPL"; diff --git a/samples/bpf/xdp_redirect_map_multi_user.c b/samples/bpf/xdp_redirect_map_multi_user.c new file mode 100644 index 000000000000..49f44c91b672 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_user.c @@ -0,0 +1,166 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; +static int rxcnt; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void poll_stats(int interval) +{ + unsigned int nr_cpus = bpf_num_possible_cpus(); + __u64 values[nr_cpus], prev[nr_cpus]; + + memset(prev, 0, sizeof(prev)); + + while (1) { + __u64 sum = 0; + __u32 key = 0; + int i; + + sleep(interval); + assert(bpf_map_lookup_elem(rxcnt, &key, values) == 0); + for (i = 0; i < nr_cpus; i++) + sum += (values[i] - prev[i]); + if (sum) + printf("Forwarding %10llu pkt/s\n", sum / interval); + memcpy(prev, values, sizeof(values)); + } +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n", + prog); +} + +int main(int argc, char **argv) +{ + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + int prog_fd, forward_map; + int i, ret, opt, ifindex; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNF")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) + xdp_flags |= XDP_FLAGS_DRV_MODE; + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + return 1; + + forward_map = bpf_object__find_map_fd_by_name(obj, "forward_map"); + rxcnt = bpf_object__find_map_fd_by_name(obj, "rxcnt"); + + if (forward_map < 0 || rxcnt < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + return 1; + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* Add all the interfaces to group all */ + ret = bpf_map_update_elem(forward_map, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + } + + poll_stats(2); + + return 0; + +err_out: + return 1; +} -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv9 bpf-next 4/5] selftests/bpf: add xdp_redirect_multi test 2020-08-26 13:19 ` [PATCHv9 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu ` (2 preceding siblings ...) 2020-08-26 13:20 ` [PATCHv9 bpf-next 3/5] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu @ 2020-08-26 13:20 ` Hangbin Liu 2020-08-26 13:20 ` [PATCHv9 bpf-next 5/5] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2020-09-03 10:26 ` [PATCHv10 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu 5 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-08-26 13:20 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko B, Hangbin Liu Add a bpf selftest for new helper xdp_redirect_map_multi(). In this test we have 3 forward groups and 1 exclude group. The test will redirect each interface's packets to all the interfaces in the forward group, and exclude the interface in exclude map. We will also test both DEVMAP and DEVMAP_HASH with xdp generic and drv. For more test details, you can find it in the test script. Here is the test result. ]# ./test_xdp_redirect_multi.sh Pass: xdpgeneric arp ns1-2 Pass: xdpgeneric arp ns1-3 Pass: xdpgeneric arp ns1-4 Pass: xdpgeneric ping ns1-2 Pass: xdpgeneric ping ns1-3 Pass: xdpgeneric ping ns1-4 Pass: xdpgeneric ping6 ns2-1 Pass: xdpgeneric ping6 ns2-3 Pass: xdpgeneric ping6 ns2-4 Pass: xdpdrv arp ns1-2 Pass: xdpdrv arp ns1-3 Pass: xdpdrv arp ns1-4 Pass: xdpdrv ping ns1-2 Pass: xdpdrv ping ns1-3 Pass: xdpdrv ping ns1-4 Pass: xdpdrv ping6 ns2-1 Pass: xdpdrv ping6 ns2-3 Pass: xdpdrv ping6 ns2-4 Summary: PASS 18, FAIL 0 Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v9: use NULL directly for arg2 and redefine the maps with btf format --- tools/testing/selftests/bpf/Makefile | 4 +- .../bpf/progs/xdp_redirect_multi_kern.c | 77 ++++++++ .../selftests/bpf/test_xdp_redirect_multi.sh | 164 +++++++++++++++++ .../selftests/bpf/xdp_redirect_multi.c | 173 ++++++++++++++++++ 4 files changed, 417 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 09657d0afb5c..ae44bd708d0e 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -51,6 +51,7 @@ TEST_FILES = test_lwt_ip_encap.o \ # Order correspond to 'make run_tests' order TEST_PROGS := test_kmod.sh \ test_xdp_redirect.sh \ + test_xdp_redirect_multi.sh \ test_xdp_meta.sh \ test_xdp_veth.sh \ test_offload.py \ @@ -79,7 +80,8 @@ TEST_PROGS_EXTENDED := with_addr.sh \ # Compile but not part of 'make run_tests' TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ - test_lirc_mode2_user xdping test_cpp runqslower bench + test_lirc_mode2_user xdping test_cpp runqslower bench \ + xdp_redirect_multi TEST_CUSTOM_PROGS = urandom_read diff --git a/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c new file mode 100644 index 000000000000..30c703efcb28 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c @@ -0,0 +1,77 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <string.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ip.h> +#include <linux/ipv6.h> + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_endian.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v4 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v6 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_all SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} exclude_map SEC(".maps"); + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + struct ethhdr *eth = data; + __u16 h_proto; + __u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + h_proto = eth->h_proto; + + if (h_proto == bpf_htons(ETH_P_IP)) + return bpf_redirect_map_multi(&forward_map_v4, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else if (h_proto == bpf_htons(ETH_P_IPV6)) + return bpf_redirect_map_multi(&forward_map_v6, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else + return bpf_redirect_map_multi(&forward_map_all, NULL, + BPF_F_EXCLUDE_INGRESS); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh new file mode 100755 index 000000000000..c1395d8634c5 --- /dev/null +++ b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh @@ -0,0 +1,164 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Test topology: +# - - - - - - - - - - - - - - - - - - - - - - - - - +# | veth1 veth2 veth3 veth4 | ... init net +# - -| - - - - - - | - - - - - - | - - - - - - | - - +# --------- --------- --------- --------- +# | veth0 | | veth0 | | veth0 | | veth0 | ... +# --------- --------- --------- --------- +# ns1 ns2 ns3 ns4 +# +# Forward multicast groups: +# Forward group all has interfaces: veth1, veth2, veth3, veth4, ... (All traffic except IPv4, IPv6) +# Forward group v4 has interfaces: veth1, veth3, veth4, ... (For IPv4 traffic only) +# Forward group v6 has interfaces: veth2, veth3, veth4, ... (For IPv6 traffic only) +# Exclude Groups: +# Exclude group: veth3 (assume ns3 is in black list) +# +# Test modules: +# XDP modes: generic, native +# map types: group v4 use DEVMAP, others use DEVMAP_HASH +# +# Test cases: +# ARP(we didn't block ARP for ns3): +# ns1 -> gw: ns2, ns3, ns4 should receive the arp request +# IPv4: +# ns1 -> ns2 (fail), ns1 -> ns3 (fail), ns1 -> ns4 (pass) +# IPv6 +# ns2 -> ns1 (fail), ns2 -> ns3 (fail), ns2 -> ns4 (pass) +# + + +# netns numbers +NUM=4 +IFACES="" +DRV_MODE="xdpgeneric xdpdrv" +PASS=0 +FAIL=0 + +test_pass() +{ + echo "Pass: $@" + PASS=$((PASS + 1)) +} + +test_fail() +{ + echo "fail: $@" + FAIL=$((FAIL + 1)) +} + +clean_up() +{ + for i in $(seq $NUM); do + ip link del veth$i 2> /dev/null + ip netns del ns$i 2> /dev/null + done + rm -f xdp_redirect_*.log arp_ns*.log +} + +# Kselftest framework requirement - SKIP code is 4. +check_env() +{ + ip link set dev lo xdpgeneric off &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without the ip xdpgeneric support" + exit 4 + fi + + which tcpdump &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without tcpdump" + exit 4 + fi +} + +setup_ns() +{ + local mode=$1 + IFACES="" + + for i in $(seq $NUM); do + ip netns add ns$i + ip link add veth$i type veth peer name veth0 netns ns$i + ip link set veth$i up + ip -n ns$i link set veth0 up + + ip -n ns$i addr add 192.0.2.$i/24 dev veth0 + ip -n ns$i addr add 2001:db8::$i/64 dev veth0 + ip -n ns$i link set veth0 $mode obj \ + xdp_dummy.o sec xdp_dummy &> /dev/null || \ + { test_fail "Unable to load dummy xdp" && exit 1; } + IFACES="$IFACES veth$i" + done +} + +do_ping_tests() +{ + local mode=$1 + + # arp test + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -i veth0 -nn -l -e &> arp_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-2_${mode}.log && \ + test_pass "$mode arp ns1-2" || test_fail "$mode arp ns1-2" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-3_${mode}.log && \ + test_pass "$mode arp ns1-3" || test_fail "$mode arp ns1-3" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-4_${mode}.log && \ + test_pass "$mode arp ns1-4" || test_fail "$mode arp ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-2" || test_pass "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-3" || test_pass "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" + + # ping6 test + ip netns exec ns2 ping6 2001:db8::1 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-1" || test_pass "$mode ping6 ns2-1" + ip netns exec ns2 ping6 2001:db8::3 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-3" || test_pass "$mode ping6 ns2-3" + ip netns exec ns2 ping6 2001:db8::4 -c 4 &> /dev/null && \ + test_pass "$mode ping6 ns2-4" || test_fail "$mode ping6 ns2-4" +} + +do_tests() +{ + local mode=$1 + local drv_p + + [ ${mode} == "xdpdrv" ] && drv_p="-N" || drv_p="-S" + + # run `ulimit -l unlimited` if you got errors like + # libbpf: Error in bpf_object__probe_global_data():Operation not permitted(1). + ./xdp_redirect_multi $drv_p $IFACES &> xdp_redirect_${mode}.log & + xdp_pid=$! + sleep 10 + + do_ping_tests $mode + + kill $xdp_pid +} + +trap clean_up 0 2 3 6 9 + +check_env + +for mode in ${DRV_MODE}; do + setup_ns $mode + do_tests $mode + sleep 10 + clean_up + sleep 5 +done + +echo "Summary: PASS $PASS, FAIL $FAIL" +[ $FAIL -eq 0 ] && exit 0 || exit 1 diff --git a/tools/testing/selftests/bpf/xdp_redirect_multi.c b/tools/testing/selftests/bpf/xdp_redirect_multi.c new file mode 100644 index 000000000000..5626005cb679 --- /dev/null +++ b/tools/testing/selftests/bpf/xdp_redirect_multi.c @@ -0,0 +1,173 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n", + prog); +} + +int main(int argc, char **argv) +{ + int prog_fd, group_all, group_v4, group_v6, exclude; + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + int i, ret, opt, ifindex; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNF")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) + xdp_flags |= XDP_FLAGS_DRV_MODE; + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + return 1; + + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_all"); + group_v4 = bpf_object__find_map_fd_by_name(obj, "forward_map_v4"); + group_v6 = bpf_object__find_map_fd_by_name(obj, "forward_map_v6"); + exclude = bpf_object__find_map_fd_by_name(obj, "exclude_map"); + + if (group_all < 0 || group_v4 < 0 || group_v6 < 0 || exclude < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + return 1; + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* Add all the interfaces to group all */ + ret = bpf_map_update_elem(group_all, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* For testing: remove the 1st interfaces from group v6 */ + if (i != 0) { + ret = bpf_map_update_elem(group_v6, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: remove the 2nd interfaces from group v4 */ + if (i != 1) { + ret = bpf_map_update_elem(group_v4, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: add the 3rd interfaces to exclude map */ + if (i == 2) { + ret = bpf_map_update_elem(exclude, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + } + + /* sleep some time for testing */ + sleep(999); + + return 0; + +err_out: + return 1; +} -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv9 bpf-next 5/5] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL 2020-08-26 13:19 ` [PATCHv9 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu ` (3 preceding siblings ...) 2020-08-26 13:20 ` [PATCHv9 bpf-next 4/5] selftests/bpf: add xdp_redirect_multi test Hangbin Liu @ 2020-08-26 13:20 ` Hangbin Liu 2020-09-03 10:26 ` [PATCHv10 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu 5 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-08-26 13:20 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko B, Hangbin Liu Use helper bpf_redirect_map() and bpf_redirect_map_multi() to test bpf arg ARG_CONST_MAP_PTR and ARG_CONST_MAP_PTR_OR_NULL. Make sure the map arg could be verified correctly when it is NULL or valid map pointer. Add devmap and devmap_hash in struct bpf_test due to bpf_redirect_{map, map_multi} limit. Test result: ]# ./test_verifier 702 705 #702/p ARG_CONST_MAP_PTR: null pointer OK #703/p ARG_CONST_MAP_PTR: valid map pointer OK #704/p ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map OK #705/p ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map OK Summary: 4 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v1-v9: no update, new added --- tools/testing/selftests/bpf/test_verifier.c | 22 +++++- .../testing/selftests/bpf/verifier/map_ptr.c | 70 +++++++++++++++++++ 2 files changed, 91 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c index 9be395d9dc64..f89a13e60692 100644 --- a/tools/testing/selftests/bpf/test_verifier.c +++ b/tools/testing/selftests/bpf/test_verifier.c @@ -50,7 +50,7 @@ #define MAX_INSNS BPF_MAXINSNS #define MAX_TEST_INSNS 1000000 #define MAX_FIXUPS 8 -#define MAX_NR_MAPS 20 +#define MAX_NR_MAPS 22 #define MAX_TEST_RUNS 8 #define POINTER_VALUE 0xcafe4all #define TEST_DATA_LEN 64 @@ -87,6 +87,8 @@ struct bpf_test { int fixup_sk_storage_map[MAX_FIXUPS]; int fixup_map_event_output[MAX_FIXUPS]; int fixup_map_reuseport_array[MAX_FIXUPS]; + int fixup_map_devmap[MAX_FIXUPS]; + int fixup_map_devmap_hash[MAX_FIXUPS]; const char *errstr; const char *errstr_unpriv; uint32_t insn_processed; @@ -640,6 +642,8 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, int *fixup_sk_storage_map = test->fixup_sk_storage_map; int *fixup_map_event_output = test->fixup_map_event_output; int *fixup_map_reuseport_array = test->fixup_map_reuseport_array; + int *fixup_map_devmap = test->fixup_map_devmap; + int *fixup_map_devmap_hash = test->fixup_map_devmap_hash; if (test->fill_helper) { test->fill_insns = calloc(MAX_TEST_INSNS, sizeof(struct bpf_insn)); @@ -817,6 +821,22 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, fixup_map_reuseport_array++; } while (*fixup_map_reuseport_array); } + if (*fixup_map_devmap) { + map_fds[20] = __create_map(BPF_MAP_TYPE_DEVMAP, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap].imm = map_fds[20]; + fixup_map_devmap++; + } while (*fixup_map_devmap); + } + if (*fixup_map_devmap_hash) { + map_fds[21] = __create_map(BPF_MAP_TYPE_DEVMAP_HASH, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap_hash].imm = map_fds[21]; + fixup_map_devmap_hash++; + } while (*fixup_map_devmap_hash); + } } struct libcap { diff --git a/tools/testing/selftests/bpf/verifier/map_ptr.c b/tools/testing/selftests/bpf/verifier/map_ptr.c index b52209db8250..51df7d8784dc 100644 --- a/tools/testing/selftests/bpf/verifier/map_ptr.c +++ b/tools/testing/selftests/bpf/verifier/map_ptr.c @@ -60,3 +60,73 @@ .result = ACCEPT, .retval = 1, }, +{ + "ARG_CONST_MAP_PTR: null pointer", + .insns = { + /* bpf_redirect_map arg1 (map) */ + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .result = REJECT, + .prog_type = BPF_PROG_TYPE_XDP, + .errstr = "R1 type=inv expected=map_ptr", +}, +{ + "ARG_CONST_MAP_PTR: valid map pointer", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg1 (map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_LD_MAP_FD(BPF_REG_2, 1), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .fixup_map_devmap_hash = { 3 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv10 bpf-next 0/5] xdp: add a new helper for dev map multicast support 2020-08-26 13:19 ` [PATCHv9 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu ` (4 preceding siblings ...) 2020-08-26 13:20 ` [PATCHv9 bpf-next 5/5] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2020-09-03 10:26 ` Hangbin Liu 2020-09-03 10:26 ` [PATCHv10 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu ` (5 more replies) 5 siblings, 6 replies; 219+ messages in thread From: Hangbin Liu @ 2020-09-03 10:26 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because there may have multi interfaces you want to exclude. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. The 1st patch add a new bpf arg to allow NULL map pointer. The 2nd patch add the new bpf_redirect_map_multi() helper. The 3rd and 4th patches are for usage sample and testing purpose, there is no effort has been made on performance optimisation. The 5th patch added some verifier test for new bpf arg ARG_CONST_MAP_PTR_OR_NULL I did same tests with pktgen(pkt size 64) to compire with xdp_redirect_map(). Here is the test result(the veth peer has a dummy xdp program with XDP_DROP directly): Version | Test | Native | Generic 5.9 rc1 | xdp_redirect_map i40e->i40e | 10.4M | 1.9M 5.9 rc1 | xdp_redirect_map i40e->veth | 14.2M | 2.2M 5.9 rc1 + patch | xdp_redirect_map i40e->i40e | 10.3M | 1.9M 5.9 rc1 + patch | xdp_redirect_map i40e->veth | 14.2M | 2.2M 5.9 rc1 + patch | xdp_redirect_map_multi i40e->i40e | 8.0M | 1.5M 5.9 rc1 + patch | xdp_redirect_map_multi i40e->veth | 11.2M | 1.6M 5.9 rc1 + patch | xdp_redirect_map_multi i40e->i40e+veth | 3.5M | 1.1M The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop the map and do clone skb/xdpf. The generic path is slower than native path as we send skbs by pktgen. So the result looks reasonable. There is some performance improvement for veth port compared with 5.8 rc1. Last but not least, thanks a lot to Toke, Jesper, Jiri and Eelco for suggestions and help on implementation. [0] https://xdp-project.net/#Handling-multicast v10: Rebase the code to latest bpf-next. Update helper bpf_xdp_redirect_map_multi() - No need to check map pointer as we will do the check in verifier. v9: Update helper bpf_xdp_redirect_map_multi() - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: a) Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). f) Split the tests from sample and add a bpf kernel selftest patch. v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Hangbin Liu (5): bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL xdp: add a new helper for dev map multicast support sample/bpf: add xdp_redirect_map_multicast test selftests/bpf: add xdp_redirect_multi test selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL include/linux/bpf.h | 21 +++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 27 +++ kernel/bpf/devmap.c | 124 +++++++++++++ kernel/bpf/verifier.c | 20 +- net/core/filter.c | 111 ++++++++++- net/core/xdp.c | 29 +++ samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 43 +++++ samples/bpf/xdp_redirect_map_multi_user.c | 166 +++++++++++++++++ tools/include/uapi/linux/bpf.h | 27 +++ tools/testing/selftests/bpf/Makefile | 4 +- .../bpf/progs/xdp_redirect_multi_kern.c | 77 ++++++++ tools/testing/selftests/bpf/test_verifier.c | 22 ++- .../selftests/bpf/test_xdp_redirect_multi.sh | 164 +++++++++++++++++ .../testing/selftests/bpf/verifier/map_ptr.c | 70 +++++++ .../selftests/bpf/xdp_redirect_multi.c | 173 ++++++++++++++++++ 18 files changed, 1071 insertions(+), 12 deletions(-) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c -- 2.25.4 ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv10 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL 2020-09-03 10:26 ` [PATCHv10 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2020-09-03 10:26 ` Hangbin Liu 2020-09-04 14:26 ` Daniel Borkmann 2020-09-03 10:26 ` [PATCHv10 bpf-next 2/5] xdp: add a new helper for dev map multicast support Hangbin Liu ` (4 subsequent siblings) 5 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-09-03 10:26 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Hangbin Liu Add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL which could be used when we want to allow NULL pointer for map parameter. The bpf helper need to take care and check if the map is NULL when use this type. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v10: remove useless CONST_PTR_TO_MAP_OR_NULL and Copy-paste comment. v9: merge the patch from [1] in to this series. v1-v8: no this patch [1] https://lore.kernel.org/bpf/20200715070001.2048207-1-liuhangbin@gmail.com/ --- include/linux/bpf.h | 1 + kernel/bpf/verifier.c | 14 +++++++++----- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index c6d9f2c444f4..884392297874 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -292,6 +292,7 @@ enum bpf_arg_type { ARG_PTR_TO_ALLOC_MEM, /* pointer to dynamically allocated memory */ ARG_PTR_TO_ALLOC_MEM_OR_NULL, /* pointer to dynamically allocated memory or NULL */ ARG_CONST_ALLOC_SIZE_OR_ZERO, /* number of allocated bytes requested */ + ARG_CONST_MAP_PTR_OR_NULL, /* const argument used as pointer to bpf_map or NULL */ }; /* type of values returned from helper functions */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index b4e9c56b8b32..95444022f74c 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -3966,9 +3966,13 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, expected_type = SCALAR_VALUE; if (type != expected_type) goto err_type; - } else if (arg_type == ARG_CONST_MAP_PTR) { + } else if (arg_type == ARG_CONST_MAP_PTR || + arg_type == ARG_CONST_MAP_PTR_OR_NULL) { expected_type = CONST_PTR_TO_MAP; - if (type != expected_type) + if (register_is_null(reg) && + arg_type == ARG_CONST_MAP_PTR_OR_NULL) + /* fall through to next check */; + else if (type != expected_type) goto err_type; } else if (arg_type == ARG_PTR_TO_CTX || arg_type == ARG_PTR_TO_CTX_OR_NULL) { @@ -4085,9 +4089,9 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, return -EFAULT; } - if (arg_type == ARG_CONST_MAP_PTR) { - /* bpf_map_xxx(map_ptr) call: remember that map_ptr */ - meta->map_ptr = reg->map_ptr; + if (arg_type == ARG_CONST_MAP_PTR || + arg_type == ARG_CONST_MAP_PTR_OR_NULL) { + meta->map_ptr = register_is_null(reg) ? NULL : reg->map_ptr; } else if (arg_type == ARG_PTR_TO_MAP_KEY) { /* bpf_map_xxx(..., map_ptr, ..., key) call: * check that [key, key + map->key_size) are within -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv10 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL 2020-09-03 10:26 ` [PATCHv10 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2020-09-04 14:26 ` Daniel Borkmann 0 siblings, 0 replies; 219+ messages in thread From: Daniel Borkmann @ 2020-09-04 14:26 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Lorenzo Bianconi, David Ahern, Andrii Nakryiko On 9/3/20 12:26 PM, Hangbin Liu wrote: > Add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL which could be > used when we want to allow NULL pointer for map parameter. The bpf helper > need to take care and check if the map is NULL when use this type. > > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > --- > > v10: remove useless CONST_PTR_TO_MAP_OR_NULL and Copy-paste comment. > v9: merge the patch from [1] in to this series. > v1-v8: no this patch > > [1] https://lore.kernel.org/bpf/20200715070001.2048207-1-liuhangbin@gmail.com/ > > --- > include/linux/bpf.h | 1 + > kernel/bpf/verifier.c | 14 +++++++++----- > 2 files changed, 10 insertions(+), 5 deletions(-) > > diff --git a/include/linux/bpf.h b/include/linux/bpf.h > index c6d9f2c444f4..884392297874 100644 > --- a/include/linux/bpf.h > +++ b/include/linux/bpf.h > @@ -292,6 +292,7 @@ enum bpf_arg_type { > ARG_PTR_TO_ALLOC_MEM, /* pointer to dynamically allocated memory */ > ARG_PTR_TO_ALLOC_MEM_OR_NULL, /* pointer to dynamically allocated memory or NULL */ > ARG_CONST_ALLOC_SIZE_OR_ZERO, /* number of allocated bytes requested */ > + ARG_CONST_MAP_PTR_OR_NULL, /* const argument used as pointer to bpf_map or NULL */ > }; > > /* type of values returned from helper functions */ > diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c > index b4e9c56b8b32..95444022f74c 100644 > --- a/kernel/bpf/verifier.c > +++ b/kernel/bpf/verifier.c > @@ -3966,9 +3966,13 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, > expected_type = SCALAR_VALUE; > if (type != expected_type) > goto err_type; > - } else if (arg_type == ARG_CONST_MAP_PTR) { > + } else if (arg_type == ARG_CONST_MAP_PTR || > + arg_type == ARG_CONST_MAP_PTR_OR_NULL) { > expected_type = CONST_PTR_TO_MAP; > - if (type != expected_type) > + if (register_is_null(reg) && > + arg_type == ARG_CONST_MAP_PTR_OR_NULL) > + /* fall through to next check */; > + else if (type != expected_type) > goto err_type; > } else if (arg_type == ARG_PTR_TO_CTX || > arg_type == ARG_PTR_TO_CTX_OR_NULL) { > @@ -4085,9 +4089,9 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, > return -EFAULT; > } > > - if (arg_type == ARG_CONST_MAP_PTR) { > - /* bpf_map_xxx(map_ptr) call: remember that map_ptr */ > - meta->map_ptr = reg->map_ptr; > + if (arg_type == ARG_CONST_MAP_PTR || > + arg_type == ARG_CONST_MAP_PTR_OR_NULL) { > + meta->map_ptr = register_is_null(reg) ? NULL : reg->map_ptr; Lgtm, one thing to note is that meta->map_ptr is just a single var right now, so if there are two maps as args like in your helper case, this would confuse verifier if we were to do more elaborate verification on the types. Not the case here, but to keep in mind if there are ever other helpers with two map args. I think we would need some bpf_func_proto sanity check on this at some point to prevent such situations. > } else if (arg_type == ARG_PTR_TO_MAP_KEY) { > /* bpf_map_xxx(..., map_ptr, ..., key) call: > * check that [key, key + map->key_size) are within > ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv10 bpf-next 2/5] xdp: add a new helper for dev map multicast support 2020-09-03 10:26 ` [PATCHv10 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-09-03 10:26 ` [PATCHv10 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2020-09-03 10:26 ` Hangbin Liu 2020-09-04 14:22 ` Daniel Borkmann 2020-09-03 10:26 ` [PATCHv10 bpf-next 3/5] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu ` (3 subsequent siblings) 5 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-09-03 10:26 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because a physical interface can be part of a logical grouping, such as a bond device. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. The forwarding map could be DEVMAP or DEVMAP_HASH, but the exclude map *must* be DEVMAP_HASH to get better performace. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. As both bpf_xdp_redirect_map() and this new helpers are using struct bpf_redirect_info, I add a new ex_map and set tgt_value to NULL in the new helper to make a difference with bpf_xdp_redirect_map(). Also I keep the the general data path in net/core/filter.c, the native data path in kernel/bpf/devmap.c so we can use direct calls to get better performace. [0] https://xdp-project.net/#Handling-multicast Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v10: Update helper bpf_xdp_redirect_map_multi() - No need to check map pointer as we will do the check in verifier. v9: Update helper bpf_xdp_redirect_map_multi() - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. --- include/linux/bpf.h | 20 ++++++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 27 +++++++ kernel/bpf/devmap.c | 124 +++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++ net/core/filter.c | 111 +++++++++++++++++++++++++++-- net/core/xdp.c | 29 ++++++++ tools/include/uapi/linux/bpf.h | 27 +++++++ 9 files changed, 341 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 884392297874..01c8d82ff2e4 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1342,6 +1342,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx); int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, struct net_device *dev_rx); +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex); +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); bool dev_map_can_have_prog(struct bpf_map *map); @@ -1517,6 +1522,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return 0; } +static inline +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + return false; +} + +static inline +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + return 0; +} + struct sk_buff; static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, diff --git a/include/linux/filter.h b/include/linux/filter.h index 995625950cc1..583dbd4c8dce 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -612,6 +612,7 @@ struct bpf_redirect_info { u32 tgt_index; void *tgt_value; struct bpf_map *map; + struct bpf_map *ex_map; u32 kern_flags; }; diff --git a/include/net/xdp.h b/include/net/xdp.h index 3814fb631d52..8453d477bb22 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -132,6 +132,7 @@ void xdp_warn(const char *msg, const char *func, const int line); #define XDP_WARN(msg) xdp_warn(msg, __func__, __LINE__) struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); static inline void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 8dda13880957..e897c4a04061 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3576,6 +3576,27 @@ union bpf_attr { * the data in *dst*. This is a wrapper of copy_from_user(). * Return * 0 on success, or a negative error in case of failure. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The frowarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3727,6 +3748,7 @@ union bpf_attr { FN(inode_storage_delete), \ FN(d_path), \ FN(copy_from_user), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -3898,6 +3920,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 2b5ca93c17de..04950e96282c 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -511,6 +511,130 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return __xdp_enqueue(dev, xdp, dev_rx); } +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + return dev_map_get_next_key(map, key, next_key); + case BPF_MAP_TYPE_DEVMAP_HASH: + return dev_map_hash_get_next_key(map, key, next_key); + default: + break; + } + + return -ENOENT; +} + +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + if (obj->dev->ifindex == exclude_ifindex) + return true; + + if (!map) + return false; + + return __dev_map_hash_lookup_elem(map, obj->dev->ifindex) != NULL; +} + +static struct bpf_dtab_netdev *devmap_get_next_obj(struct xdp_buff *xdp, struct bpf_map *map, + struct bpf_map *ex_map, u32 *key, + u32 *next_key, int ex_ifindex) +{ + struct bpf_dtab_netdev *obj; + struct net_device *dev; + u32 *tmp_key = key; + int err; + + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + return NULL; + + for (;;) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + obj = __dev_map_lookup_elem(map, *next_key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + obj = __dev_map_hash_lookup_elem(map, *next_key); + break; + default: + break; + } + + if (!obj || dev_in_exclude_map(obj, ex_map, ex_ifindex)) + goto find_next; + + dev = obj->dev; + + if (!dev->netdev_ops->ndo_xdp_xmit) + goto find_next; + + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + if (unlikely(err)) + goto find_next; + + return obj; + +find_next: + tmp_key = next_key; + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + break; + } + + return NULL; +} + +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + struct bpf_dtab_netdev *obj = NULL, *next_obj = NULL; + struct xdp_frame *xdpf, *nxdpf; + bool last_one = false; + int ex_ifindex; + u32 key, next_key; + + ex_ifindex = flags & BPF_F_EXCLUDE_INGRESS ? dev_rx->ifindex : 0; + + /* Find first available obj */ + obj = devmap_get_next_obj(xdp, map, ex_map, NULL, &key, ex_ifindex); + if (!obj) + return 0; + + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) + return -EOVERFLOW; + + for (;;) { + /* Check if we still have one more available obj */ + next_obj = devmap_get_next_obj(xdp, map, ex_map, &key, + &next_key, ex_ifindex); + if (!next_obj) + last_one = true; + + if (last_one) { + bq_enqueue(obj->dev, xdpf, dev_rx); + return 0; + } + + nxdpf = xdpf_clone(xdpf); + if (unlikely(!nxdpf)) { + xdp_return_frame_rx_napi(xdpf); + return -ENOMEM; + } + + bq_enqueue(obj->dev, nxdpf, dev_rx); + + /* Deal with next obj */ + obj = next_obj; + key = next_key; + } +} + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 95444022f74c..d79068df2b10 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4273,6 +4273,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: if (func_id != BPF_FUNC_redirect_map && + func_id != BPF_FUNC_redirect_map_multi && func_id != BPF_FUNC_map_lookup_elem) goto error; break; @@ -4372,6 +4373,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, map->map_type != BPF_MAP_TYPE_XSKMAP) goto error; break; + case BPF_FUNC_redirect_map_multi: + if (map->map_type != BPF_MAP_TYPE_DEVMAP && + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + goto error; + break; case BPF_FUNC_sk_redirect_map: case BPF_FUNC_msg_redirect_map: case BPF_FUNC_sock_map_update: diff --git a/net/core/filter.c b/net/core/filter.c index 47eef9a0be6a..8e596d5e308e 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3539,12 +3539,19 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { }; static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) + struct bpf_map *map, struct xdp_buff *xdp, + struct bpf_map *ex_map, u32 flags) { switch (map->map_type) { case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: - return dev_map_enqueue(fwd, xdp, dev_rx); + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) + return dev_map_enqueue(fwd, xdp, dev_rx); + else + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, flags); case BPF_MAP_TYPE_CPUMAP: return cpu_map_enqueue(fwd, xdp, dev_rx); case BPF_MAP_TYPE_XSKMAP: @@ -3601,12 +3608,14 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); struct bpf_map *map = READ_ONCE(ri->map); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (unlikely(!map)) { @@ -3618,7 +3627,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = dev_xdp_enqueue(fwd, xdp, dev); } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, ri->flags); } if (unlikely(err)) @@ -3632,6 +3641,55 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, } EXPORT_SYMBOL_GPL(xdp_do_redirect); +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, + struct bpf_prog *xdp_prog, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) + +{ + struct bpf_dtab_netdev *dst; + struct sk_buff *nskb; + bool exclude_ingress; + u32 key, next_key; + void *fwd; + int err; + + /* Get first key from forward map */ + err = map->ops->map_get_next_key(map, NULL, &key); + if (err) + return err; + + exclude_ingress = !!(flags & BPF_F_EXCLUDE_INGRESS); + + for (;;) { + fwd = __xdp_map_lookup_elem(map, key); + if (fwd) { + dst = (struct bpf_dtab_netdev *)fwd; + if (dev_in_exclude_map(dst, ex_map, + exclude_ingress ? dev->ifindex : 0)) + goto find_next; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -ENOMEM; + + /* Try forword next one no mater the current forward + * succeed or not */ + dev_map_generic_redirect(dst, nskb, xdp_prog); + } + +find_next: + err = map->ops->map_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + consume_skb(skb); + return 0; +} + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, @@ -3639,19 +3697,30 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, struct bpf_map *map) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err = 0; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (map->map_type == BPF_MAP_TYPE_DEVMAP || map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { - struct bpf_dtab_netdev *dst = fwd; + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) { + struct bpf_dtab_netdev *dst = fwd; + + err = dev_map_generic_redirect(dst, skb, xdp_prog); + } else { + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, + ex_map, ri->flags); + } - err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { @@ -3765,6 +3834,36 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { .arg3_type = ARG_ANYTHING, }; +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, + struct bpf_map *, ex_map, u64, flags) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + + /* Limit ex_map type to DEVMAP_HASH to get better performance */ + if (unlikely((ex_map && ex_map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) || + flags & ~BPF_F_EXCLUDE_INGRESS)) + return XDP_ABORTED; + + ri->tgt_index = 0; + /* Set the tgt_value to NULL to distinguish with bpf_xdp_redirect_map */ + ri->tgt_value = NULL; + ri->flags = flags; + ri->ex_map = ex_map; + + WRITE_ONCE(ri->map, map); + + return XDP_REDIRECT; +} + +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { + .func = bpf_xdp_redirect_map_multi, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_CONST_MAP_PTR, + .arg2_type = ARG_CONST_MAP_PTR_OR_NULL, + .arg3_type = ARG_ANYTHING, +}; + static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, unsigned long off, unsigned long len) { @@ -6833,6 +6932,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_redirect_proto; case BPF_FUNC_redirect_map: return &bpf_xdp_redirect_map_proto; + case BPF_FUNC_redirect_map_multi: + return &bpf_xdp_redirect_map_multi_proto; case BPF_FUNC_xdp_adjust_tail: return &bpf_xdp_adjust_tail_proto; case BPF_FUNC_fib_lookup: diff --git a/net/core/xdp.c b/net/core/xdp.c index 48aba933a5a8..9fd3e89768c4 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -467,3 +467,32 @@ void xdp_warn(const char *msg, const char *func, const int line) WARN(1, "XDP_WARN: %s(line:%d): %s\n", func, line, msg); }; EXPORT_SYMBOL_GPL(xdp_warn); + +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) +{ + unsigned int headroom, totalsize; + struct xdp_frame *nxdpf; + struct page *page; + void *addr; + + headroom = xdpf->headroom + sizeof(*xdpf); + totalsize = headroom + xdpf->len; + + if (unlikely(totalsize > PAGE_SIZE)) + return NULL; + page = dev_alloc_page(); + if (!page) + return NULL; + addr = page_to_virt(page); + + memcpy(addr, xdpf, totalsize); + + nxdpf = addr; + nxdpf->data = addr + headroom; + nxdpf->frame_sz = PAGE_SIZE; + nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0; + nxdpf->mem.id = 0; + + return nxdpf; +} +EXPORT_SYMBOL_GPL(xdpf_clone); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 8dda13880957..e897c4a04061 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3576,6 +3576,27 @@ union bpf_attr { * the data in *dst*. This is a wrapper of copy_from_user(). * Return * 0 on success, or a negative error in case of failure. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The frowarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3727,6 +3748,7 @@ union bpf_attr { FN(inode_storage_delete), \ FN(d_path), \ FN(copy_from_user), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -3898,6 +3920,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv10 bpf-next 2/5] xdp: add a new helper for dev map multicast support 2020-09-03 10:26 ` [PATCHv10 bpf-next 2/5] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2020-09-04 14:22 ` Daniel Borkmann 0 siblings, 0 replies; 219+ messages in thread From: Daniel Borkmann @ 2020-09-04 14:22 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Lorenzo Bianconi, David Ahern, Andrii Nakryiko On 9/3/20 12:26 PM, Hangbin Liu wrote: [...] > diff --git a/include/net/xdp.h b/include/net/xdp.h > index 3814fb631d52..8453d477bb22 100644 > --- a/include/net/xdp.h > +++ b/include/net/xdp.h > @@ -132,6 +132,7 @@ void xdp_warn(const char *msg, const char *func, const int line); > #define XDP_WARN(msg) xdp_warn(msg, __func__, __LINE__) > > struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); > +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); > > static inline > void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp) > diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h > index 8dda13880957..e897c4a04061 100644 > --- a/include/uapi/linux/bpf.h > +++ b/include/uapi/linux/bpf.h > @@ -3576,6 +3576,27 @@ union bpf_attr { > * the data in *dst*. This is a wrapper of copy_from_user(). > * Return > * 0 on success, or a negative error in case of failure. > + * > + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) > + * Description > + * This is a multicast implementation for XDP redirect. It will > + * redirect the packet to ALL the interfaces in *map*, but > + * exclude the interfaces in *ex_map*. > + * > + * The frowarding *map* could be either BPF_MAP_TYPE_DEVMAP or nit: typo > + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be > + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. > + * > + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, > + * which additionally excludes the current ingress device. > + * > + * See also bpf_redirect_map() as a unicast implementation, > + * which supports redirecting packet to a specific ifindex > + * in the map. As both helpers use struct bpf_redirect_info > + * to store the redirect info, we will use a a NULL tgt_value > + * to distinguish multicast and unicast redirecting. > + * Return > + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. > */ > #define __BPF_FUNC_MAPPER(FN) \ > FN(unspec), \ > @@ -3727,6 +3748,7 @@ union bpf_attr { > FN(inode_storage_delete), \ > FN(d_path), \ > FN(copy_from_user), \ > + FN(redirect_map_multi), \ > /* */ > > /* integer value in 'imm' field of BPF_CALL instruction selects which helper > @@ -3898,6 +3920,11 @@ enum bpf_lwt_encap_mode { > BPF_LWT_ENCAP_IP, > }; > > +/* BPF_FUNC_redirect_map_multi flags. */ > +enum { > + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), > +}; > + > #define __bpf_md_ptr(type, name) \ > union { \ > type name; \ > diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c > index 2b5ca93c17de..04950e96282c 100644 > --- a/kernel/bpf/devmap.c > +++ b/kernel/bpf/devmap.c > @@ -511,6 +511,130 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > return __xdp_enqueue(dev, xdp, dev_rx); > } > > +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ > +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) > +{ > + > + switch (map->map_type) { > + case BPF_MAP_TYPE_DEVMAP: > + return dev_map_get_next_key(map, key, next_key); > + case BPF_MAP_TYPE_DEVMAP_HASH: > + return dev_map_hash_get_next_key(map, key, next_key); > + default: > + break; > + } > + > + return -ENOENT; > +} > + > +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, > + int exclude_ifindex) > +{ > + if (obj->dev->ifindex == exclude_ifindex) > + return true; > + > + if (!map) > + return false; > + > + return __dev_map_hash_lookup_elem(map, obj->dev->ifindex) != NULL; > +} > + > +static struct bpf_dtab_netdev *devmap_get_next_obj(struct xdp_buff *xdp, struct bpf_map *map, > + struct bpf_map *ex_map, u32 *key, > + u32 *next_key, int ex_ifindex) > +{ > + struct bpf_dtab_netdev *obj; > + struct net_device *dev; > + u32 *tmp_key = key; > + int err; > + > + err = devmap_get_next_key(map, tmp_key, next_key); > + if (err) > + return NULL; > + > + for (;;) { > + switch (map->map_type) { > + case BPF_MAP_TYPE_DEVMAP: > + obj = __dev_map_lookup_elem(map, *next_key); > + break; > + case BPF_MAP_TYPE_DEVMAP_HASH: > + obj = __dev_map_hash_lookup_elem(map, *next_key); > + break; > + default: > + break; > + } > + > + if (!obj || dev_in_exclude_map(obj, ex_map, ex_ifindex)) > + goto find_next; > + > + dev = obj->dev; > + > + if (!dev->netdev_ops->ndo_xdp_xmit) > + goto find_next; > + > + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); > + if (unlikely(err)) > + goto find_next; > + > + return obj; > + > +find_next: > + tmp_key = next_key; > + err = devmap_get_next_key(map, tmp_key, next_key); For all the devmap_get_next_key() or map->ops->map_get_next_key() (in dev_map_redirect_multi()) in the case of dev map hash, we could restart the hashtab traversal in case the key has been updated/removed in the mean time, so we'd end up potentially looping due to traversal restarts from first elem. Instead of for (;;) there should be an upper limit, so we don't perform this forever, afaics. > + if (err) > + break; > + } > + > + return NULL; > +} > + ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv10 bpf-next 3/5] sample/bpf: add xdp_redirect_map_multicast test 2020-09-03 10:26 ` [PATCHv10 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-09-03 10:26 ` [PATCHv10 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2020-09-03 10:26 ` [PATCHv10 bpf-next 2/5] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2020-09-03 10:26 ` Hangbin Liu 2020-09-03 10:27 ` [PATCHv10 bpf-next 4/5] selftests/bpf: add xdp_redirect_multi test Hangbin Liu ` (2 subsequent siblings) 5 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-09-03 10:26 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Hangbin Liu This is a sample for xdp multicast. In the sample we could forward all packets between given interfaces. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v10: no update v9: use NULL directly for arg2 and redefine the maps with btf format v5: add a null_map as we have strict the arg2 to ARG_CONST_MAP_PTR. Move the testing part to bpf selftest in next patch. v4: no update. v3: add rxcnt map to show the packet transmit speed. v2: no update. --- samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 43 ++++++ samples/bpf/xdp_redirect_map_multi_user.c | 166 ++++++++++++++++++++++ 3 files changed, 212 insertions(+) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 4f1ed0e3cf9f..cad63d4ea164 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map tprogs-y += per_socket_stats_example tprogs-y += xdp_redirect tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_map_multi tprogs-y += xdp_redirect_cpu tprogs-y += xdp_monitor tprogs-y += xdp_rxq_info @@ -98,6 +99,7 @@ test_map_in_map-objs := test_map_in_map_user.o per_socket_stats_example-objs := cookie_uid_helper_example.o xdp_redirect-objs := xdp_redirect_user.o xdp_redirect_map-objs := xdp_redirect_map_user.o +xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o xdp_redirect_cpu-objs := bpf_load.o xdp_redirect_cpu_user.o xdp_monitor-objs := bpf_load.o xdp_monitor_user.o xdp_rxq_info-objs := xdp_rxq_info_user.o @@ -158,6 +160,7 @@ always-y += tcp_tos_reflect_kern.o always-y += tcp_dumpstats_kern.o always-y += xdp_redirect_kern.o always-y += xdp_redirect_map_kern.o +always-y += xdp_redirect_map_multi_kern.o always-y += xdp_redirect_cpu_kern.o always-y += xdp_monitor_kern.o always-y += xdp_rxq_info_kern.o diff --git a/samples/bpf/xdp_redirect_map_multi_kern.c b/samples/bpf/xdp_redirect_map_multi_kern.c new file mode 100644 index 000000000000..db58d56cef89 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_kern.c @@ -0,0 +1,43 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <uapi/linux/bpf.h> +#include <bpf/bpf_helpers.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 256); +} forward_map SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __type(key, u32); + __type(value, long); + __uint(max_entries, 1); +} rxcnt SEC(".maps"); + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + long *value; + u32 key = 0; + + /* count packet in global counter */ + value = bpf_map_lookup_elem(&rxcnt, &key); + if (value) + *value += 1; + + return bpf_redirect_map_multi(&forward_map, NULL, BPF_F_EXCLUDE_INGRESS); +} + +char _license[] SEC("license") = "GPL"; diff --git a/samples/bpf/xdp_redirect_map_multi_user.c b/samples/bpf/xdp_redirect_map_multi_user.c new file mode 100644 index 000000000000..49f44c91b672 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_user.c @@ -0,0 +1,166 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; +static int rxcnt; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void poll_stats(int interval) +{ + unsigned int nr_cpus = bpf_num_possible_cpus(); + __u64 values[nr_cpus], prev[nr_cpus]; + + memset(prev, 0, sizeof(prev)); + + while (1) { + __u64 sum = 0; + __u32 key = 0; + int i; + + sleep(interval); + assert(bpf_map_lookup_elem(rxcnt, &key, values) == 0); + for (i = 0; i < nr_cpus; i++) + sum += (values[i] - prev[i]); + if (sum) + printf("Forwarding %10llu pkt/s\n", sum / interval); + memcpy(prev, values, sizeof(values)); + } +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n", + prog); +} + +int main(int argc, char **argv) +{ + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + int prog_fd, forward_map; + int i, ret, opt, ifindex; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNF")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) + xdp_flags |= XDP_FLAGS_DRV_MODE; + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + return 1; + + forward_map = bpf_object__find_map_fd_by_name(obj, "forward_map"); + rxcnt = bpf_object__find_map_fd_by_name(obj, "rxcnt"); + + if (forward_map < 0 || rxcnt < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + return 1; + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* Add all the interfaces to group all */ + ret = bpf_map_update_elem(forward_map, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + } + + poll_stats(2); + + return 0; + +err_out: + return 1; +} -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv10 bpf-next 4/5] selftests/bpf: add xdp_redirect_multi test 2020-09-03 10:26 ` [PATCHv10 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu ` (2 preceding siblings ...) 2020-09-03 10:26 ` [PATCHv10 bpf-next 3/5] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu @ 2020-09-03 10:27 ` Hangbin Liu 2020-09-03 10:27 ` [PATCHv10 bpf-next 5/5] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2020-09-07 8:27 ` [PATCHv11 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu 5 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-09-03 10:27 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Hangbin Liu Add a bpf selftest for new helper xdp_redirect_map_multi(). In this test we have 3 forward groups and 1 exclude group. The test will redirect each interface's packets to all the interfaces in the forward group, and exclude the interface in exclude map. We will also test both DEVMAP and DEVMAP_HASH with xdp generic and drv. For more test details, you can find it in the test script. Here is the test result. ]# ./test_xdp_redirect_multi.sh Pass: xdpgeneric arp ns1-2 Pass: xdpgeneric arp ns1-3 Pass: xdpgeneric arp ns1-4 Pass: xdpgeneric ping ns1-2 Pass: xdpgeneric ping ns1-3 Pass: xdpgeneric ping ns1-4 Pass: xdpgeneric ping6 ns2-1 Pass: xdpgeneric ping6 ns2-3 Pass: xdpgeneric ping6 ns2-4 Pass: xdpdrv arp ns1-2 Pass: xdpdrv arp ns1-3 Pass: xdpdrv arp ns1-4 Pass: xdpdrv ping ns1-2 Pass: xdpdrv ping ns1-3 Pass: xdpdrv ping ns1-4 Pass: xdpdrv ping6 ns2-1 Pass: xdpdrv ping6 ns2-3 Pass: xdpdrv ping6 ns2-4 Summary: PASS 18, FAIL 0 Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v10: no update v9: use NULL directly for arg2 and redefine the maps with btf format --- tools/testing/selftests/bpf/Makefile | 4 +- .../bpf/progs/xdp_redirect_multi_kern.c | 77 ++++++++ .../selftests/bpf/test_xdp_redirect_multi.sh | 164 +++++++++++++++++ .../selftests/bpf/xdp_redirect_multi.c | 173 ++++++++++++++++++ 4 files changed, 417 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 65d3d9aaeb31..61f063618e48 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -51,6 +51,7 @@ TEST_FILES = test_lwt_ip_encap.o \ # Order correspond to 'make run_tests' order TEST_PROGS := test_kmod.sh \ test_xdp_redirect.sh \ + test_xdp_redirect_multi.sh \ test_xdp_meta.sh \ test_xdp_veth.sh \ test_offload.py \ @@ -79,7 +80,8 @@ TEST_PROGS_EXTENDED := with_addr.sh \ # Compile but not part of 'make run_tests' TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ - test_lirc_mode2_user xdping test_cpp runqslower bench + test_lirc_mode2_user xdping test_cpp runqslower bench \ + xdp_redirect_multi TEST_CUSTOM_PROGS = urandom_read diff --git a/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c new file mode 100644 index 000000000000..30c703efcb28 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c @@ -0,0 +1,77 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <string.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ip.h> +#include <linux/ipv6.h> + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_endian.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v4 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v6 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_all SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} exclude_map SEC(".maps"); + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + struct ethhdr *eth = data; + __u16 h_proto; + __u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + h_proto = eth->h_proto; + + if (h_proto == bpf_htons(ETH_P_IP)) + return bpf_redirect_map_multi(&forward_map_v4, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else if (h_proto == bpf_htons(ETH_P_IPV6)) + return bpf_redirect_map_multi(&forward_map_v6, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else + return bpf_redirect_map_multi(&forward_map_all, NULL, + BPF_F_EXCLUDE_INGRESS); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh new file mode 100755 index 000000000000..c1395d8634c5 --- /dev/null +++ b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh @@ -0,0 +1,164 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Test topology: +# - - - - - - - - - - - - - - - - - - - - - - - - - +# | veth1 veth2 veth3 veth4 | ... init net +# - -| - - - - - - | - - - - - - | - - - - - - | - - +# --------- --------- --------- --------- +# | veth0 | | veth0 | | veth0 | | veth0 | ... +# --------- --------- --------- --------- +# ns1 ns2 ns3 ns4 +# +# Forward multicast groups: +# Forward group all has interfaces: veth1, veth2, veth3, veth4, ... (All traffic except IPv4, IPv6) +# Forward group v4 has interfaces: veth1, veth3, veth4, ... (For IPv4 traffic only) +# Forward group v6 has interfaces: veth2, veth3, veth4, ... (For IPv6 traffic only) +# Exclude Groups: +# Exclude group: veth3 (assume ns3 is in black list) +# +# Test modules: +# XDP modes: generic, native +# map types: group v4 use DEVMAP, others use DEVMAP_HASH +# +# Test cases: +# ARP(we didn't block ARP for ns3): +# ns1 -> gw: ns2, ns3, ns4 should receive the arp request +# IPv4: +# ns1 -> ns2 (fail), ns1 -> ns3 (fail), ns1 -> ns4 (pass) +# IPv6 +# ns2 -> ns1 (fail), ns2 -> ns3 (fail), ns2 -> ns4 (pass) +# + + +# netns numbers +NUM=4 +IFACES="" +DRV_MODE="xdpgeneric xdpdrv" +PASS=0 +FAIL=0 + +test_pass() +{ + echo "Pass: $@" + PASS=$((PASS + 1)) +} + +test_fail() +{ + echo "fail: $@" + FAIL=$((FAIL + 1)) +} + +clean_up() +{ + for i in $(seq $NUM); do + ip link del veth$i 2> /dev/null + ip netns del ns$i 2> /dev/null + done + rm -f xdp_redirect_*.log arp_ns*.log +} + +# Kselftest framework requirement - SKIP code is 4. +check_env() +{ + ip link set dev lo xdpgeneric off &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without the ip xdpgeneric support" + exit 4 + fi + + which tcpdump &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without tcpdump" + exit 4 + fi +} + +setup_ns() +{ + local mode=$1 + IFACES="" + + for i in $(seq $NUM); do + ip netns add ns$i + ip link add veth$i type veth peer name veth0 netns ns$i + ip link set veth$i up + ip -n ns$i link set veth0 up + + ip -n ns$i addr add 192.0.2.$i/24 dev veth0 + ip -n ns$i addr add 2001:db8::$i/64 dev veth0 + ip -n ns$i link set veth0 $mode obj \ + xdp_dummy.o sec xdp_dummy &> /dev/null || \ + { test_fail "Unable to load dummy xdp" && exit 1; } + IFACES="$IFACES veth$i" + done +} + +do_ping_tests() +{ + local mode=$1 + + # arp test + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -i veth0 -nn -l -e &> arp_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-2_${mode}.log && \ + test_pass "$mode arp ns1-2" || test_fail "$mode arp ns1-2" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-3_${mode}.log && \ + test_pass "$mode arp ns1-3" || test_fail "$mode arp ns1-3" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-4_${mode}.log && \ + test_pass "$mode arp ns1-4" || test_fail "$mode arp ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-2" || test_pass "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-3" || test_pass "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" + + # ping6 test + ip netns exec ns2 ping6 2001:db8::1 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-1" || test_pass "$mode ping6 ns2-1" + ip netns exec ns2 ping6 2001:db8::3 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-3" || test_pass "$mode ping6 ns2-3" + ip netns exec ns2 ping6 2001:db8::4 -c 4 &> /dev/null && \ + test_pass "$mode ping6 ns2-4" || test_fail "$mode ping6 ns2-4" +} + +do_tests() +{ + local mode=$1 + local drv_p + + [ ${mode} == "xdpdrv" ] && drv_p="-N" || drv_p="-S" + + # run `ulimit -l unlimited` if you got errors like + # libbpf: Error in bpf_object__probe_global_data():Operation not permitted(1). + ./xdp_redirect_multi $drv_p $IFACES &> xdp_redirect_${mode}.log & + xdp_pid=$! + sleep 10 + + do_ping_tests $mode + + kill $xdp_pid +} + +trap clean_up 0 2 3 6 9 + +check_env + +for mode in ${DRV_MODE}; do + setup_ns $mode + do_tests $mode + sleep 10 + clean_up + sleep 5 +done + +echo "Summary: PASS $PASS, FAIL $FAIL" +[ $FAIL -eq 0 ] && exit 0 || exit 1 diff --git a/tools/testing/selftests/bpf/xdp_redirect_multi.c b/tools/testing/selftests/bpf/xdp_redirect_multi.c new file mode 100644 index 000000000000..5626005cb679 --- /dev/null +++ b/tools/testing/selftests/bpf/xdp_redirect_multi.c @@ -0,0 +1,173 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n", + prog); +} + +int main(int argc, char **argv) +{ + int prog_fd, group_all, group_v4, group_v6, exclude; + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + int i, ret, opt, ifindex; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNF")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) + xdp_flags |= XDP_FLAGS_DRV_MODE; + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + return 1; + + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_all"); + group_v4 = bpf_object__find_map_fd_by_name(obj, "forward_map_v4"); + group_v6 = bpf_object__find_map_fd_by_name(obj, "forward_map_v6"); + exclude = bpf_object__find_map_fd_by_name(obj, "exclude_map"); + + if (group_all < 0 || group_v4 < 0 || group_v6 < 0 || exclude < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + return 1; + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* Add all the interfaces to group all */ + ret = bpf_map_update_elem(group_all, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* For testing: remove the 1st interfaces from group v6 */ + if (i != 0) { + ret = bpf_map_update_elem(group_v6, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: remove the 2nd interfaces from group v4 */ + if (i != 1) { + ret = bpf_map_update_elem(group_v4, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: add the 3rd interfaces to exclude map */ + if (i == 2) { + ret = bpf_map_update_elem(exclude, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + } + + /* sleep some time for testing */ + sleep(999); + + return 0; + +err_out: + return 1; +} -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv10 bpf-next 5/5] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL 2020-09-03 10:26 ` [PATCHv10 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu ` (3 preceding siblings ...) 2020-09-03 10:27 ` [PATCHv10 bpf-next 4/5] selftests/bpf: add xdp_redirect_multi test Hangbin Liu @ 2020-09-03 10:27 ` Hangbin Liu 2020-09-07 8:27 ` [PATCHv11 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu 5 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-09-03 10:27 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Hangbin Liu Use helper bpf_redirect_map() and bpf_redirect_map_multi() to test bpf arg ARG_CONST_MAP_PTR and ARG_CONST_MAP_PTR_OR_NULL. Make sure the map arg could be verified correctly when it is NULL or valid map pointer. Add devmap and devmap_hash in struct bpf_test due to bpf_redirect_{map, map_multi} limit. Test result: ]# ./test_verifier 702 705 #702/p ARG_CONST_MAP_PTR: null pointer OK #703/p ARG_CONST_MAP_PTR: valid map pointer OK #704/p ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map OK #705/p ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map OK Summary: 4 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- tools/testing/selftests/bpf/test_verifier.c | 22 +++++- .../testing/selftests/bpf/verifier/map_ptr.c | 70 +++++++++++++++++++ 2 files changed, 91 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c index 9be395d9dc64..f89a13e60692 100644 --- a/tools/testing/selftests/bpf/test_verifier.c +++ b/tools/testing/selftests/bpf/test_verifier.c @@ -50,7 +50,7 @@ #define MAX_INSNS BPF_MAXINSNS #define MAX_TEST_INSNS 1000000 #define MAX_FIXUPS 8 -#define MAX_NR_MAPS 20 +#define MAX_NR_MAPS 22 #define MAX_TEST_RUNS 8 #define POINTER_VALUE 0xcafe4all #define TEST_DATA_LEN 64 @@ -87,6 +87,8 @@ struct bpf_test { int fixup_sk_storage_map[MAX_FIXUPS]; int fixup_map_event_output[MAX_FIXUPS]; int fixup_map_reuseport_array[MAX_FIXUPS]; + int fixup_map_devmap[MAX_FIXUPS]; + int fixup_map_devmap_hash[MAX_FIXUPS]; const char *errstr; const char *errstr_unpriv; uint32_t insn_processed; @@ -640,6 +642,8 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, int *fixup_sk_storage_map = test->fixup_sk_storage_map; int *fixup_map_event_output = test->fixup_map_event_output; int *fixup_map_reuseport_array = test->fixup_map_reuseport_array; + int *fixup_map_devmap = test->fixup_map_devmap; + int *fixup_map_devmap_hash = test->fixup_map_devmap_hash; if (test->fill_helper) { test->fill_insns = calloc(MAX_TEST_INSNS, sizeof(struct bpf_insn)); @@ -817,6 +821,22 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, fixup_map_reuseport_array++; } while (*fixup_map_reuseport_array); } + if (*fixup_map_devmap) { + map_fds[20] = __create_map(BPF_MAP_TYPE_DEVMAP, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap].imm = map_fds[20]; + fixup_map_devmap++; + } while (*fixup_map_devmap); + } + if (*fixup_map_devmap_hash) { + map_fds[21] = __create_map(BPF_MAP_TYPE_DEVMAP_HASH, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap_hash].imm = map_fds[21]; + fixup_map_devmap_hash++; + } while (*fixup_map_devmap_hash); + } } struct libcap { diff --git a/tools/testing/selftests/bpf/verifier/map_ptr.c b/tools/testing/selftests/bpf/verifier/map_ptr.c index b52209db8250..51df7d8784dc 100644 --- a/tools/testing/selftests/bpf/verifier/map_ptr.c +++ b/tools/testing/selftests/bpf/verifier/map_ptr.c @@ -60,3 +60,73 @@ .result = ACCEPT, .retval = 1, }, +{ + "ARG_CONST_MAP_PTR: null pointer", + .insns = { + /* bpf_redirect_map arg1 (map) */ + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .result = REJECT, + .prog_type = BPF_PROG_TYPE_XDP, + .errstr = "R1 type=inv expected=map_ptr", +}, +{ + "ARG_CONST_MAP_PTR: valid map pointer", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg1 (map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_LD_MAP_FD(BPF_REG_2, 1), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .fixup_map_devmap_hash = { 3 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv11 bpf-next 0/5] xdp: add a new helper for dev map multicast support 2020-09-03 10:26 ` [PATCHv10 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu ` (4 preceding siblings ...) 2020-09-03 10:27 ` [PATCHv10 bpf-next 5/5] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2020-09-07 8:27 ` Hangbin Liu 2020-09-07 8:27 ` [PATCHv11 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu ` (5 more replies) 5 siblings, 6 replies; 219+ messages in thread From: Hangbin Liu @ 2020-09-07 8:27 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because there may have multi interfaces you want to exclude. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. The 1st patch add a new bpf arg to allow NULL map pointer. The 2nd patch add the new bpf_redirect_map_multi() helper. The 3rd and 4th patches are for usage sample and testing purpose, there is no effort has been made on performance optimisation. The 5th patch added some verifier test for new bpf arg ARG_CONST_MAP_PTR_OR_NULL I did same tests with pktgen(pkt size 64) to compire with xdp_redirect_map(). Here is the test result(the veth peer has a dummy xdp program with XDP_DROP directly): Version | Test | Native | Generic 5.9 rc1 | xdp_redirect_map i40e->i40e | 10.4M | 1.9M 5.9 rc1 | xdp_redirect_map i40e->veth | 14.2M | 2.2M 5.9 rc1 + patch | xdp_redirect_map i40e->i40e | 10.3M | 1.9M 5.9 rc1 + patch | xdp_redirect_map i40e->veth | 14.2M | 2.2M 5.9 rc1 + patch | xdp_redirect_map_multi i40e->i40e | 8.0M | 1.5M 5.9 rc1 + patch | xdp_redirect_map_multi i40e->veth | 11.2M | 1.6M 5.9 rc1 + patch | xdp_redirect_map_multi i40e->i40e+veth | 3.5M | 1.1M The bpf_redirect_map_multi() is slower than bpf_redirect_map() as we loop the map and do clone skb/xdpf. The generic path is slower than native path as we send skbs by pktgen. So the result looks reasonable. There is some performance improvement for veth port compared with 5.8 rc1. Last but not least, thanks a lot to Toke, Jesper, Jiri and Eelco for suggestions and help on implementation. [0] https://xdp-project.net/#Handling-multicast v11: Fix bpf_redirect_map_multi() helper description typo. Add loop limit for devmap_get_next_obj() and dev_map_redirect_multi(). v10: Rebase the code to latest bpf-next. Update helper bpf_xdp_redirect_map_multi() - No need to check map pointer as we will do the check in verifier. v9: Update helper bpf_xdp_redirect_map_multi() - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: a) Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). f) Split the tests from sample and add a bpf kernel selftest patch. v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Hangbin Liu (5): bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL xdp: add a new helper for dev map multicast support sample/bpf: add xdp_redirect_map_multicast test selftests/bpf: add xdp_redirect_multi test selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL include/linux/bpf.h | 21 +++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 27 +++ kernel/bpf/devmap.c | 132 +++++++++++++ kernel/bpf/verifier.c | 20 +- net/core/filter.c | 118 +++++++++++- net/core/xdp.c | 29 +++ samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 43 +++++ samples/bpf/xdp_redirect_map_multi_user.c | 166 +++++++++++++++++ tools/include/uapi/linux/bpf.h | 27 +++ tools/testing/selftests/bpf/Makefile | 4 +- .../bpf/progs/xdp_redirect_multi_kern.c | 77 ++++++++ tools/testing/selftests/bpf/test_verifier.c | 22 ++- .../selftests/bpf/test_xdp_redirect_multi.sh | 164 +++++++++++++++++ .../testing/selftests/bpf/verifier/map_ptr.c | 70 +++++++ .../selftests/bpf/xdp_redirect_multi.c | 173 ++++++++++++++++++ 18 files changed, 1086 insertions(+), 12 deletions(-) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c -- 2.25.4 ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv11 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL 2020-09-07 8:27 ` [PATCHv11 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2020-09-07 8:27 ` Hangbin Liu 2020-09-07 8:27 ` [PATCHv11 bpf-next 2/5] xdp: add a new helper for dev map multicast support Hangbin Liu ` (4 subsequent siblings) 5 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-09-07 8:27 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Hangbin Liu Add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL which could be used when we want to allow NULL pointer for map parameter. The bpf helper need to take care and check if the map is NULL when use this type. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v11: no update v10: remove useless CONST_PTR_TO_MAP_OR_NULL and Copy-paste comment. v9: merge the patch from [1] in to this series. v1-v8: no this patch [1] https://lore.kernel.org/bpf/20200715070001.2048207-1-liuhangbin@gmail.com/ --- include/linux/bpf.h | 1 + kernel/bpf/verifier.c | 14 +++++++++----- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index c6d9f2c444f4..884392297874 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -292,6 +292,7 @@ enum bpf_arg_type { ARG_PTR_TO_ALLOC_MEM, /* pointer to dynamically allocated memory */ ARG_PTR_TO_ALLOC_MEM_OR_NULL, /* pointer to dynamically allocated memory or NULL */ ARG_CONST_ALLOC_SIZE_OR_ZERO, /* number of allocated bytes requested */ + ARG_CONST_MAP_PTR_OR_NULL, /* const argument used as pointer to bpf_map or NULL */ }; /* type of values returned from helper functions */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index b4e9c56b8b32..95444022f74c 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -3966,9 +3966,13 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, expected_type = SCALAR_VALUE; if (type != expected_type) goto err_type; - } else if (arg_type == ARG_CONST_MAP_PTR) { + } else if (arg_type == ARG_CONST_MAP_PTR || + arg_type == ARG_CONST_MAP_PTR_OR_NULL) { expected_type = CONST_PTR_TO_MAP; - if (type != expected_type) + if (register_is_null(reg) && + arg_type == ARG_CONST_MAP_PTR_OR_NULL) + /* fall through to next check */; + else if (type != expected_type) goto err_type; } else if (arg_type == ARG_PTR_TO_CTX || arg_type == ARG_PTR_TO_CTX_OR_NULL) { @@ -4085,9 +4089,9 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, return -EFAULT; } - if (arg_type == ARG_CONST_MAP_PTR) { - /* bpf_map_xxx(map_ptr) call: remember that map_ptr */ - meta->map_ptr = reg->map_ptr; + if (arg_type == ARG_CONST_MAP_PTR || + arg_type == ARG_CONST_MAP_PTR_OR_NULL) { + meta->map_ptr = register_is_null(reg) ? NULL : reg->map_ptr; } else if (arg_type == ARG_PTR_TO_MAP_KEY) { /* bpf_map_xxx(..., map_ptr, ..., key) call: * check that [key, key + map->key_size) are within -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv11 bpf-next 2/5] xdp: add a new helper for dev map multicast support 2020-09-07 8:27 ` [PATCHv11 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-09-07 8:27 ` [PATCHv11 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2020-09-07 8:27 ` Hangbin Liu 2020-09-09 21:52 ` Alexei Starovoitov 2020-09-07 8:27 ` [PATCHv11 bpf-next 3/5] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu ` (3 subsequent siblings) 5 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-09-07 8:27 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because a physical interface can be part of a logical grouping, such as a bond device. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. The forwarding map could be DEVMAP or DEVMAP_HASH, but the exclude map *must* be DEVMAP_HASH to get better performace. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. As both bpf_xdp_redirect_map() and this new helpers are using struct bpf_redirect_info, I add a new ex_map and set tgt_value to NULL in the new helper to make a difference with bpf_xdp_redirect_map(). Also I keep the the general data path in net/core/filter.c, the native data path in kernel/bpf/devmap.c so we can use direct calls to get better performace. [0] https://xdp-project.net/#Handling-multicast Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v11: Fix bpf_redirect_map_multi() helper description typo. Add loop limit for devmap_get_next_obj() and dev_map_redirect_multi(). v10: Update helper bpf_xdp_redirect_map_multi() - No need to check map pointer as we will do the check in verifier. v9: Update helper bpf_xdp_redirect_map_multi() - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. --- include/linux/bpf.h | 20 +++++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 27 +++++++ kernel/bpf/devmap.c | 132 +++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++ net/core/filter.c | 118 +++++++++++++++++++++++++++-- net/core/xdp.c | 29 ++++++++ tools/include/uapi/linux/bpf.h | 27 +++++++ 9 files changed, 356 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 884392297874..01c8d82ff2e4 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1342,6 +1342,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx); int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, struct net_device *dev_rx); +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex); +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); bool dev_map_can_have_prog(struct bpf_map *map); @@ -1517,6 +1522,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return 0; } +static inline +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + return false; +} + +static inline +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + return 0; +} + struct sk_buff; static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, diff --git a/include/linux/filter.h b/include/linux/filter.h index 995625950cc1..583dbd4c8dce 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -612,6 +612,7 @@ struct bpf_redirect_info { u32 tgt_index; void *tgt_value; struct bpf_map *map; + struct bpf_map *ex_map; u32 kern_flags; }; diff --git a/include/net/xdp.h b/include/net/xdp.h index 3814fb631d52..8453d477bb22 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -132,6 +132,7 @@ void xdp_warn(const char *msg, const char *func, const int line); #define XDP_WARN(msg) xdp_warn(msg, __func__, __LINE__) struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); static inline void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 8dda13880957..60785cf1989c 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3576,6 +3576,27 @@ union bpf_attr { * the data in *dst*. This is a wrapper of copy_from_user(). * Return * 0 on success, or a negative error in case of failure. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3727,6 +3748,7 @@ union bpf_attr { FN(inode_storage_delete), \ FN(d_path), \ FN(copy_from_user), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -3898,6 +3920,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 2b5ca93c17de..f9a4b663c713 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -511,6 +511,138 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return __xdp_enqueue(dev, xdp, dev_rx); } +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + return dev_map_get_next_key(map, key, next_key); + case BPF_MAP_TYPE_DEVMAP_HASH: + return dev_map_hash_get_next_key(map, key, next_key); + default: + break; + } + + return -ENOENT; +} + +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + if (obj->dev->ifindex == exclude_ifindex) + return true; + + if (!map) + return false; + + return __dev_map_hash_lookup_elem(map, obj->dev->ifindex) != NULL; +} + +static struct bpf_dtab_netdev *devmap_get_next_obj(struct xdp_buff *xdp, struct bpf_map *map, + struct bpf_map *ex_map, u32 *key, + u32 *next_key, int ex_ifindex) +{ + struct bpf_dtab_netdev *obj; + struct net_device *dev; + u32 *tmp_key = key; + u32 index; + int err; + + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + return NULL; + + /* When using dev map hash, we could restart the hashtab traversal + * in case the key has been updated/removed in the mean time. + * So we may end up potentially looping due to traversal restarts + * from first elem. + * + * Let's use map's max_entries to limit the loop number. + */ + for (index = 0; index < map->max_entries; index++) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + obj = __dev_map_lookup_elem(map, *next_key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + obj = __dev_map_hash_lookup_elem(map, *next_key); + break; + default: + break; + } + + if (!obj || dev_in_exclude_map(obj, ex_map, ex_ifindex)) + goto find_next; + + dev = obj->dev; + + if (!dev->netdev_ops->ndo_xdp_xmit) + goto find_next; + + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + if (unlikely(err)) + goto find_next; + + return obj; + +find_next: + tmp_key = next_key; + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + break; + } + + return NULL; +} + +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + struct bpf_dtab_netdev *obj = NULL, *next_obj = NULL; + struct xdp_frame *xdpf, *nxdpf; + bool last_one = false; + int ex_ifindex; + u32 key, next_key; + + ex_ifindex = flags & BPF_F_EXCLUDE_INGRESS ? dev_rx->ifindex : 0; + + /* Find first available obj */ + obj = devmap_get_next_obj(xdp, map, ex_map, NULL, &key, ex_ifindex); + if (!obj) + return 0; + + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) + return -EOVERFLOW; + + for (;;) { + /* Check if we still have one more available obj */ + next_obj = devmap_get_next_obj(xdp, map, ex_map, &key, + &next_key, ex_ifindex); + if (!next_obj) + last_one = true; + + if (last_one) { + bq_enqueue(obj->dev, xdpf, dev_rx); + return 0; + } + + nxdpf = xdpf_clone(xdpf); + if (unlikely(!nxdpf)) { + xdp_return_frame_rx_napi(xdpf); + return -ENOMEM; + } + + bq_enqueue(obj->dev, nxdpf, dev_rx); + + /* Deal with next obj */ + obj = next_obj; + key = next_key; + } +} + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 95444022f74c..d79068df2b10 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4273,6 +4273,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: if (func_id != BPF_FUNC_redirect_map && + func_id != BPF_FUNC_redirect_map_multi && func_id != BPF_FUNC_map_lookup_elem) goto error; break; @@ -4372,6 +4373,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, map->map_type != BPF_MAP_TYPE_XSKMAP) goto error; break; + case BPF_FUNC_redirect_map_multi: + if (map->map_type != BPF_MAP_TYPE_DEVMAP && + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + goto error; + break; case BPF_FUNC_sk_redirect_map: case BPF_FUNC_msg_redirect_map: case BPF_FUNC_sock_map_update: diff --git a/net/core/filter.c b/net/core/filter.c index 47eef9a0be6a..a2999ea8178b 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3539,12 +3539,19 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { }; static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) + struct bpf_map *map, struct xdp_buff *xdp, + struct bpf_map *ex_map, u32 flags) { switch (map->map_type) { case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: - return dev_map_enqueue(fwd, xdp, dev_rx); + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) + return dev_map_enqueue(fwd, xdp, dev_rx); + else + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, flags); case BPF_MAP_TYPE_CPUMAP: return cpu_map_enqueue(fwd, xdp, dev_rx); case BPF_MAP_TYPE_XSKMAP: @@ -3601,12 +3608,14 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); struct bpf_map *map = READ_ONCE(ri->map); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (unlikely(!map)) { @@ -3618,7 +3627,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = dev_xdp_enqueue(fwd, xdp, dev); } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, ri->flags); } if (unlikely(err)) @@ -3632,6 +3641,62 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, } EXPORT_SYMBOL_GPL(xdp_do_redirect); +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, + struct bpf_prog *xdp_prog, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) + +{ + struct bpf_dtab_netdev *dst; + struct sk_buff *nskb; + bool exclude_ingress; + u32 key, next_key, index; + void *fwd; + int err; + + /* Get first key from forward map */ + err = map->ops->map_get_next_key(map, NULL, &key); + if (err) + return err; + + exclude_ingress = !!(flags & BPF_F_EXCLUDE_INGRESS); + + /* When using dev map hash, we could restart the hashtab traversal + * in case the key has been updated/removed in the mean time. + * So we may end up potentially looping due to traversal restarts + * from first elem. + * + * Let's use map's max_entries to limit the loop number. + */ + for (index = 0; index < map->max_entries; index++) { + fwd = __xdp_map_lookup_elem(map, key); + if (fwd) { + dst = (struct bpf_dtab_netdev *)fwd; + if (dev_in_exclude_map(dst, ex_map, + exclude_ingress ? dev->ifindex : 0)) + goto find_next; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -ENOMEM; + + /* Try forword next one no mater the current forward + * succeed or not */ + dev_map_generic_redirect(dst, nskb, xdp_prog); + } + +find_next: + err = map->ops->map_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + consume_skb(skb); + return 0; +} + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, @@ -3639,19 +3704,30 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, struct bpf_map *map) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err = 0; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (map->map_type == BPF_MAP_TYPE_DEVMAP || map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { - struct bpf_dtab_netdev *dst = fwd; + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) { + struct bpf_dtab_netdev *dst = fwd; + + err = dev_map_generic_redirect(dst, skb, xdp_prog); + } else { + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, + ex_map, ri->flags); + } - err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { @@ -3765,6 +3841,36 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { .arg3_type = ARG_ANYTHING, }; +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, + struct bpf_map *, ex_map, u64, flags) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + + /* Limit ex_map type to DEVMAP_HASH to get better performance */ + if (unlikely((ex_map && ex_map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) || + flags & ~BPF_F_EXCLUDE_INGRESS)) + return XDP_ABORTED; + + ri->tgt_index = 0; + /* Set the tgt_value to NULL to distinguish with bpf_xdp_redirect_map */ + ri->tgt_value = NULL; + ri->flags = flags; + ri->ex_map = ex_map; + + WRITE_ONCE(ri->map, map); + + return XDP_REDIRECT; +} + +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { + .func = bpf_xdp_redirect_map_multi, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_CONST_MAP_PTR, + .arg2_type = ARG_CONST_MAP_PTR_OR_NULL, + .arg3_type = ARG_ANYTHING, +}; + static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, unsigned long off, unsigned long len) { @@ -6833,6 +6939,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_redirect_proto; case BPF_FUNC_redirect_map: return &bpf_xdp_redirect_map_proto; + case BPF_FUNC_redirect_map_multi: + return &bpf_xdp_redirect_map_multi_proto; case BPF_FUNC_xdp_adjust_tail: return &bpf_xdp_adjust_tail_proto; case BPF_FUNC_fib_lookup: diff --git a/net/core/xdp.c b/net/core/xdp.c index 48aba933a5a8..9fd3e89768c4 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -467,3 +467,32 @@ void xdp_warn(const char *msg, const char *func, const int line) WARN(1, "XDP_WARN: %s(line:%d): %s\n", func, line, msg); }; EXPORT_SYMBOL_GPL(xdp_warn); + +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) +{ + unsigned int headroom, totalsize; + struct xdp_frame *nxdpf; + struct page *page; + void *addr; + + headroom = xdpf->headroom + sizeof(*xdpf); + totalsize = headroom + xdpf->len; + + if (unlikely(totalsize > PAGE_SIZE)) + return NULL; + page = dev_alloc_page(); + if (!page) + return NULL; + addr = page_to_virt(page); + + memcpy(addr, xdpf, totalsize); + + nxdpf = addr; + nxdpf->data = addr + headroom; + nxdpf->frame_sz = PAGE_SIZE; + nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0; + nxdpf->mem.id = 0; + + return nxdpf; +} +EXPORT_SYMBOL_GPL(xdpf_clone); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 8dda13880957..60785cf1989c 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3576,6 +3576,27 @@ union bpf_attr { * the data in *dst*. This is a wrapper of copy_from_user(). * Return * 0 on success, or a negative error in case of failure. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3727,6 +3748,7 @@ union bpf_attr { FN(inode_storage_delete), \ FN(d_path), \ FN(copy_from_user), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -3898,6 +3920,11 @@ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_IP, }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv11 bpf-next 2/5] xdp: add a new helper for dev map multicast support 2020-09-07 8:27 ` [PATCHv11 bpf-next 2/5] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2020-09-09 21:52 ` Alexei Starovoitov 2020-09-10 2:35 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: Alexei Starovoitov @ 2020-09-09 21:52 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko On Mon, Sep 07, 2020 at 04:27:21PM +0800, Hangbin Liu wrote: > This patch is for xdp multicast support. which has been discussed > before[0], The goal is to be able to implement an OVS-like data plane in > XDP, i.e., a software switch that can forward XDP frames to multiple ports. > > To achieve this, an application needs to specify a group of interfaces > to forward a packet to. It is also common to want to exclude one or more > physical interfaces from the forwarding operation - e.g., to forward a > packet to all interfaces in the multicast group except the interface it > arrived on. While this could be done simply by adding more groups, this > quickly leads to a combinatorial explosion in the number of groups an > application has to maintain. > > To avoid the combinatorial explosion, we propose to include the ability > to specify an "exclude group" as part of the forwarding operation. This > needs to be a group (instead of just a single port index), because a > physical interface can be part of a logical grouping, such as a bond > device. > > Thus, the logical forwarding operation becomes a "set difference" > operation, i.e. "forward to all ports in group A that are not also in > group B". This series implements such an operation using device maps to > represent the groups. This means that the XDP program specifies two > device maps, one containing the list of netdevs to redirect to, and the > other containing the exclude list. "set difference" and BPF_F_EXCLUDE_INGRESS makes sense to me as high level api, but I don't see how program or helper is going to modify the packet before multicasting it. Even to implement a basic switch the program would need to modify destination mac addresses before xmiting it on the device. In case of XDP_TX the bpf program is doing it manually. With this api the program is out of the loop. It can prepare a packet for one target netdev, but sending the same packet as-is to other netdevs isn't going to to work correctly. Veth-s and tap-s don't care about mac and the stack will silently accept packets even with wrong mac. The same thing may happen with physical netdevs. The driver won't care that dst mac is wrong. It will xmit it out, but the other side of the wire will likely drop that packet unless it's promisc. Properly implemented bridge shouldn't be doing it, but I really don't see how this api can work in practice to implement real bridge. What am I missing? ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv11 bpf-next 2/5] xdp: add a new helper for dev map multicast support 2020-09-09 21:52 ` Alexei Starovoitov @ 2020-09-10 2:35 ` Hangbin Liu 2020-09-10 3:30 ` David Ahern 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-09-10 2:35 UTC (permalink / raw) To: Alexei Starovoitov Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko Hi Alexei, On Wed, Sep 09, 2020 at 02:52:06PM -0700, Alexei Starovoitov wrote: > On Mon, Sep 07, 2020 at 04:27:21PM +0800, Hangbin Liu wrote: > > This patch is for xdp multicast support. which has been discussed > > before[0], The goal is to be able to implement an OVS-like data plane in > > XDP, i.e., a software switch that can forward XDP frames to multiple ports. > > > > To achieve this, an application needs to specify a group of interfaces > > to forward a packet to. It is also common to want to exclude one or more > > physical interfaces from the forwarding operation - e.g., to forward a > > packet to all interfaces in the multicast group except the interface it > > arrived on. While this could be done simply by adding more groups, this > > quickly leads to a combinatorial explosion in the number of groups an > > application has to maintain. > > > > To avoid the combinatorial explosion, we propose to include the ability > > to specify an "exclude group" as part of the forwarding operation. This > > needs to be a group (instead of just a single port index), because a > > physical interface can be part of a logical grouping, such as a bond > > device. > > > > Thus, the logical forwarding operation becomes a "set difference" > > operation, i.e. "forward to all ports in group A that are not also in > > group B". This series implements such an operation using device maps to > > represent the groups. This means that the XDP program specifies two > > device maps, one containing the list of netdevs to redirect to, and the > > other containing the exclude list. > > "set difference" and BPF_F_EXCLUDE_INGRESS makes sense to me as high level api, > but I don't see how program or helper is going to modify the packet > before multicasting it. > Even to implement a basic switch the program would need to modify destination > mac addresses before xmiting it on the device. > In case of XDP_TX the bpf program is doing it manually. > With this api the program is out of the loop. > It can prepare a packet for one target netdev, but sending the same > packet as-is to other netdevs isn't going to to work correctly. Yes, we can't modify the packets on ingress as there are multi egress ports and each one may has different requirements. So this helper will only forward the packets to other group(looks like a multicast group) devices. I think the packets modification (edit dst mac, add vlan tag, etc) should be done on egress, which rely on David's XDP egress support. > Veth-s and tap-s don't care about mac and the stack will silently accept > packets even with wrong mac. > The same thing may happen with physical netdevs. The driver won't care > that dst mac is wrong. It will xmit it out, but the other side of the wire > will likely drop that packet unless it's promisc. > Properly implemented bridge shouldn't be doing it, but > I really don't see how this api can work in practice to implement real bridge. > What am I missing? Not sure if I missed something. Does current linux bridge do dst mac modification? I thought it only forward packets(although it has fdb instead of flush the packet to all ports) On patch 4/5 there is an example about forwarding packets. It still need to get remote's mac address by arp/nd. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv11 bpf-next 2/5] xdp: add a new helper for dev map multicast support 2020-09-10 2:35 ` Hangbin Liu @ 2020-09-10 3:30 ` David Ahern 2020-09-10 5:35 ` Alexei Starovoitov 0 siblings, 1 reply; 219+ messages in thread From: David Ahern @ 2020-09-10 3:30 UTC (permalink / raw) To: Hangbin Liu, Alexei Starovoitov Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Andrii Nakryiko On 9/9/20 8:35 PM, Hangbin Liu wrote: > Hi Alexei, > > On Wed, Sep 09, 2020 at 02:52:06PM -0700, Alexei Starovoitov wrote: >> On Mon, Sep 07, 2020 at 04:27:21PM +0800, Hangbin Liu wrote: >>> This patch is for xdp multicast support. which has been discussed >>> before[0], The goal is to be able to implement an OVS-like data plane in >>> XDP, i.e., a software switch that can forward XDP frames to multiple ports. >>> >>> To achieve this, an application needs to specify a group of interfaces >>> to forward a packet to. It is also common to want to exclude one or more >>> physical interfaces from the forwarding operation - e.g., to forward a >>> packet to all interfaces in the multicast group except the interface it >>> arrived on. While this could be done simply by adding more groups, this >>> quickly leads to a combinatorial explosion in the number of groups an >>> application has to maintain. >>> >>> To avoid the combinatorial explosion, we propose to include the ability >>> to specify an "exclude group" as part of the forwarding operation. This >>> needs to be a group (instead of just a single port index), because a >>> physical interface can be part of a logical grouping, such as a bond >>> device. >>> >>> Thus, the logical forwarding operation becomes a "set difference" >>> operation, i.e. "forward to all ports in group A that are not also in >>> group B". This series implements such an operation using device maps to >>> represent the groups. This means that the XDP program specifies two >>> device maps, one containing the list of netdevs to redirect to, and the >>> other containing the exclude list. >> >> "set difference" and BPF_F_EXCLUDE_INGRESS makes sense to me as high level api, >> but I don't see how program or helper is going to modify the packet >> before multicasting it. >> Even to implement a basic switch the program would need to modify destination >> mac addresses before xmiting it on the device. >> In case of XDP_TX the bpf program is doing it manually. >> With this api the program is out of the loop. >> It can prepare a packet for one target netdev, but sending the same >> packet as-is to other netdevs isn't going to to work correctly. > > Yes, we can't modify the packets on ingress as there are multi egress ports > and each one may has different requirements. So this helper will only forward > the packets to other group(looks like a multicast group) devices. > > I think the packets modification (edit dst mac, add vlan tag, etc) should be > done on egress, which rely on David's XDP egress support. agreed. The DEVMAP used for redirect can have programs attached that update the packet headers - assuming you want to update them. This is tagged as "multicast" support but it really is redirecting a packet to multiple devices. One use case I see that evolves from this set is the ability to both forward packets (e.g., host ingress to VM) and grab a copy tcpdump style by redirecting packets to a virtual device (similar to a patch set for dropwatch). ie., no need for an perf-events style copy to push to userspace. ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv11 bpf-next 2/5] xdp: add a new helper for dev map multicast support 2020-09-10 3:30 ` David Ahern @ 2020-09-10 5:35 ` Alexei Starovoitov 2020-09-10 9:44 ` Toke Høiland-Jørgensen 0 siblings, 1 reply; 219+ messages in thread From: Alexei Starovoitov @ 2020-09-10 5:35 UTC (permalink / raw) To: David Ahern Cc: Hangbin Liu, bpf, Network Development, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, Alexei Starovoitov, Daniel Borkmann, Lorenzo Bianconi, Andrii Nakryiko On Wed, Sep 9, 2020 at 8:30 PM David Ahern <dsahern@gmail.com> wrote: > > > > I think the packets modification (edit dst mac, add vlan tag, etc) should be > > done on egress, which rely on David's XDP egress support. > > agreed. The DEVMAP used for redirect can have programs attached that > update the packet headers - assuming you want to update them. Then you folks have to submit them as one set. As-is the programmer cannot achieve correct behavior. ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv11 bpf-next 2/5] xdp: add a new helper for dev map multicast support 2020-09-10 5:35 ` Alexei Starovoitov @ 2020-09-10 9:44 ` Toke Høiland-Jørgensen 2020-09-10 15:39 ` Alexei Starovoitov 2020-09-10 17:50 ` Jesper Dangaard Brouer 0 siblings, 2 replies; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2020-09-10 9:44 UTC (permalink / raw) To: Alexei Starovoitov, David Ahern Cc: Hangbin Liu, bpf, Network Development, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, Alexei Starovoitov, Daniel Borkmann, Lorenzo Bianconi, Andrii Nakryiko Alexei Starovoitov <alexei.starovoitov@gmail.com> writes: > On Wed, Sep 9, 2020 at 8:30 PM David Ahern <dsahern@gmail.com> wrote: >> > >> > I think the packets modification (edit dst mac, add vlan tag, etc) should be >> > done on egress, which rely on David's XDP egress support. >> >> agreed. The DEVMAP used for redirect can have programs attached that >> update the packet headers - assuming you want to update them. > > Then you folks have to submit them as one set. > As-is the programmer cannot achieve correct behavior. The ability to attach a program to devmaps is already there. See: fbee97feed9b ("bpf: Add support to attach bpf program to a devmap entry") But now that you mention it, it does appear that this series is skipping the hook that will actually run such a program. Didn't realise that was in the caller of bq_enqueue() and not inside bq_enqueue() itself... Hangbin, you'll need to add the hook for dev_map_run_prog() before bq_enqueue(); see the existing dev_map_enqueue() function. -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv11 bpf-next 2/5] xdp: add a new helper for dev map multicast support 2020-09-10 9:44 ` Toke Høiland-Jørgensen @ 2020-09-10 15:39 ` Alexei Starovoitov 2020-09-10 17:50 ` Jesper Dangaard Brouer 1 sibling, 0 replies; 219+ messages in thread From: Alexei Starovoitov @ 2020-09-10 15:39 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: David Ahern, Hangbin Liu, bpf, Network Development, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, Alexei Starovoitov, Daniel Borkmann, Lorenzo Bianconi, Andrii Nakryiko On Thu, Sep 10, 2020 at 2:44 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote: > > Alexei Starovoitov <alexei.starovoitov@gmail.com> writes: > > > On Wed, Sep 9, 2020 at 8:30 PM David Ahern <dsahern@gmail.com> wrote: > >> > > >> > I think the packets modification (edit dst mac, add vlan tag, etc) should be > >> > done on egress, which rely on David's XDP egress support. > >> > >> agreed. The DEVMAP used for redirect can have programs attached that > >> update the packet headers - assuming you want to update them. > > > > Then you folks have to submit them as one set. > > As-is the programmer cannot achieve correct behavior. > > The ability to attach a program to devmaps is already there. See: > > fbee97feed9b ("bpf: Add support to attach bpf program to a devmap entry") ahh. you meant that one. > But now that you mention it, it does appear that this series is skipping > the hook that will actually run such a program. Didn't realise that was > in the caller of bq_enqueue() and not inside bq_enqueue() itself... > > Hangbin, you'll need to add the hook for dev_map_run_prog() before > bq_enqueue(); see the existing dev_map_enqueue() function. If that's the expected usage it should have been described in the commit log and thoroughly exercised in the tests. ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv11 bpf-next 2/5] xdp: add a new helper for dev map multicast support 2020-09-10 9:44 ` Toke Høiland-Jørgensen 2020-09-10 15:39 ` Alexei Starovoitov @ 2020-09-10 17:50 ` Jesper Dangaard Brouer 2020-09-10 18:35 ` David Ahern 1 sibling, 1 reply; 219+ messages in thread From: Jesper Dangaard Brouer @ 2020-09-10 17:50 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: Alexei Starovoitov, David Ahern, Hangbin Liu, bpf, Network Development, Jiri Benc, Eelco Chaudron, Alexei Starovoitov, Daniel Borkmann, Lorenzo Bianconi, Andrii Nakryiko, brouer On Thu, 10 Sep 2020 11:44:50 +0200 Toke Høiland-Jørgensen <toke@redhat.com> wrote: > Alexei Starovoitov <alexei.starovoitov@gmail.com> writes: > > > On Wed, Sep 9, 2020 at 8:30 PM David Ahern <dsahern@gmail.com> wrote: > >> > > >> > I think the packets modification (edit dst mac, add vlan tag, etc) should be > >> > done on egress, which rely on David's XDP egress support. > >> > >> agreed. The DEVMAP used for redirect can have programs attached that > >> update the packet headers - assuming you want to update them. > > > > Then you folks have to submit them as one set. > > As-is the programmer cannot achieve correct behavior. > > The ability to attach a program to devmaps is already there. See: > > fbee97feed9b ("bpf: Add support to attach bpf program to a devmap entry") > > But now that you mention it, it does appear that this series is skipping > the hook that will actually run such a program. Didn't realise that was > in the caller of bq_enqueue() and not inside bq_enqueue() itself... In the first revisions of Ahern's patchset (before fully integrated in devmap), this was the case, but it changed in some of the last revisions. (This also lost the sort-n-bulk effect in the process, that optimize I-cache). In these earlier revisions it operated on xdp_frame's. It would have been a lot easier for Hangbin's patch if the devmap-prog operated on these xdp_frame's. Maybe we should change the devmap-prog approach, and run this on the xdp_frame's (in bq_xmit_all() to be precise) . Hangbin's patchset clearly shows that we need this "layer" between running the xdp_prog and the devmap-prog. -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv11 bpf-next 2/5] xdp: add a new helper for dev map multicast support 2020-09-10 17:50 ` Jesper Dangaard Brouer @ 2020-09-10 18:35 ` David Ahern 2020-09-11 7:58 ` Jesper Dangaard Brouer 0 siblings, 1 reply; 219+ messages in thread From: David Ahern @ 2020-09-10 18:35 UTC (permalink / raw) To: Jesper Dangaard Brouer, Toke Høiland-Jørgensen Cc: Alexei Starovoitov, Hangbin Liu, bpf, Network Development, Jiri Benc, Eelco Chaudron, Alexei Starovoitov, Daniel Borkmann, Lorenzo Bianconi, Andrii Nakryiko On 9/10/20 11:50 AM, Jesper Dangaard Brouer wrote: > Maybe we should change the devmap-prog approach, and run this on the > xdp_frame's (in bq_xmit_all() to be precise) . Hangbin's patchset > clearly shows that we need this "layer" between running the xdp_prog and > the devmap-prog. I would prefer to leave it in dev_map_enqueue. The main premise at the moment is that the program attached to the DEVMAP entry is an ACL specific to that dev. If the program is going to drop the packet, then no sense queueing it. I also expect a follow on feature will be useful to allow the DEVMAP program to do another REDIRECT (e.g., potentially after modifying). It is not handled at the moment as it needs thought - e.g., limiting the number of iterative redirects. If such a feature does happen, then no sense queueing it to the current device. ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv11 bpf-next 2/5] xdp: add a new helper for dev map multicast support 2020-09-10 18:35 ` David Ahern @ 2020-09-11 7:58 ` Jesper Dangaard Brouer 2020-09-15 16:12 ` David Ahern 0 siblings, 1 reply; 219+ messages in thread From: Jesper Dangaard Brouer @ 2020-09-11 7:58 UTC (permalink / raw) To: David Ahern Cc: Toke Høiland-Jørgensen, Alexei Starovoitov, Hangbin Liu, bpf, Network Development, Jiri Benc, Eelco Chaudron, Alexei Starovoitov, Daniel Borkmann, Lorenzo Bianconi, Andrii Nakryiko, brouer On Thu, 10 Sep 2020 12:35:33 -0600 David Ahern <dsahern@gmail.com> wrote: > On 9/10/20 11:50 AM, Jesper Dangaard Brouer wrote: > > Maybe we should change the devmap-prog approach, and run this on the > > xdp_frame's (in bq_xmit_all() to be precise) . Hangbin's patchset > > clearly shows that we need this "layer" between running the xdp_prog and > > the devmap-prog. > > I would prefer to leave it in dev_map_enqueue. > > The main premise at the moment is that the program attached to the > DEVMAP entry is an ACL specific to that dev. If the program is going to > drop the packet, then no sense queueing it. > > I also expect a follow on feature will be useful to allow the DEVMAP > program to do another REDIRECT (e.g., potentially after modifying). It > is not handled at the moment as it needs thought - e.g., limiting the > number of iterative redirects. If such a feature does happen, then no > sense queueing it to the current device. It makes a lot of sense to do queuing before redirecting again. The (hidden) bulking we do at XDP redirect is the primary reason for the performance boost. We all remember performance difference between non-map version of redirect (which Toke fixed via always having the bulking available in net_device->xdp_bulkq). In a simple micro-benchmark I bet it will look better running the devmap-prog right after the xdp_prog (which is what we have today). But I claim this is the wrong approach, as soon as (1) traffic is more intermixed, and (2) devmap-prog gets bigger and becomes more specific to the egress-device (e.g. BPF update constants per egress-device). When this happens performance suffers, as I-cache and data-access to each egress-device gets pushed out of cache. (Hint VPP/fd.io approach) Queuing xdp_frames up for your devmap-prog makes sense, as these share common properties. With intermix traffic the first xdp_prog will sort packets into egress-devices, and then the devmap-prog can operate on these. The best illustration[1] of this sorting I saw in a Netflix blogpost[2] about FreeBSD, section "RSS Assisted LRO" (not directly related, but illustration was good). [1] https://miro.medium.com/max/700/1%2alTGL1_D6hTMEMa7EDV8yZA.png [2] https://netflixtechblog.com/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99 -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv11 bpf-next 2/5] xdp: add a new helper for dev map multicast support 2020-09-11 7:58 ` Jesper Dangaard Brouer @ 2020-09-15 16:12 ` David Ahern 0 siblings, 0 replies; 219+ messages in thread From: David Ahern @ 2020-09-15 16:12 UTC (permalink / raw) To: Jesper Dangaard Brouer Cc: Toke Høiland-Jørgensen, Alexei Starovoitov, Hangbin Liu, bpf, Network Development, Jiri Benc, Eelco Chaudron, Alexei Starovoitov, Daniel Borkmann, Lorenzo Bianconi, Andrii Nakryiko On 9/11/20 1:58 AM, Jesper Dangaard Brouer wrote: > On Thu, 10 Sep 2020 12:35:33 -0600 > David Ahern <dsahern@gmail.com> wrote: > >> On 9/10/20 11:50 AM, Jesper Dangaard Brouer wrote: >>> Maybe we should change the devmap-prog approach, and run this on the >>> xdp_frame's (in bq_xmit_all() to be precise) . Hangbin's patchset >>> clearly shows that we need this "layer" between running the xdp_prog and >>> the devmap-prog. >> >> I would prefer to leave it in dev_map_enqueue. >> >> The main premise at the moment is that the program attached to the >> DEVMAP entry is an ACL specific to that dev. If the program is going to >> drop the packet, then no sense queueing it. >> >> I also expect a follow on feature will be useful to allow the DEVMAP >> program to do another REDIRECT (e.g., potentially after modifying). It >> is not handled at the moment as it needs thought - e.g., limiting the >> number of iterative redirects. If such a feature does happen, then no >> sense queueing it to the current device. > > It makes a lot of sense to do queuing before redirecting again. The > (hidden) bulking we do at XDP redirect is the primary reason for the > performance boost. We all remember performance difference between > non-map version of redirect (which Toke fixed via always having the > bulking available in net_device->xdp_bulkq). > > In a simple micro-benchmark I bet it will look better running the > devmap-prog right after the xdp_prog (which is what we have today). But > I claim this is the wrong approach, as soon as (1) traffic is more > intermixed, and (2) devmap-prog gets bigger and becomes more specific > to the egress-device (e.g. BPF update constants per egress-device). > When this happens performance suffers, as I-cache and data-access to > each egress-device gets pushed out of cache. (Hint VPP/fd.io approach) > > Queuing xdp_frames up for your devmap-prog makes sense, as these share > common properties. With intermix traffic the first xdp_prog will sort > packets into egress-devices, and then the devmap-prog can operate on > these. The best illustration[1] of this sorting I saw in a Netflix > blogpost[2] about FreeBSD, section "RSS Assisted LRO" (not directly > related, but illustration was good). > > > [1] https://miro.medium.com/max/700/1%2alTGL1_D6hTMEMa7EDV8yZA.png > [2] https://netflixtechblog.com/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99 > I understand the theory and testing will need to bear that out. There is a bit of distance (code wise) between where the program is run now and where you want to put it - the conversion from xdp_buff to xdp_frame, the enqueue, and what it means to do a redirect to another device in bq_xmit_all. More importantly though for a redirect is the current xdp_ok_fwd_dev check in __xdp_enqueue which for a redirect could be doing the wrong checks for the wrong device. ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv11 bpf-next 3/5] sample/bpf: add xdp_redirect_map_multicast test 2020-09-07 8:27 ` [PATCHv11 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-09-07 8:27 ` [PATCHv11 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2020-09-07 8:27 ` [PATCHv11 bpf-next 2/5] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2020-09-07 8:27 ` Hangbin Liu 2020-09-07 8:27 ` [PATCHv11 bpf-next 4/5] selftests/bpf: add xdp_redirect_multi test Hangbin Liu ` (2 subsequent siblings) 5 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-09-07 8:27 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Hangbin Liu This is a sample for xdp multicast. In the sample we could forward all packets between given interfaces. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v10-v11: no update v9: use NULL directly for arg2 and redefine the maps with btf format v5: add a null_map as we have strict the arg2 to ARG_CONST_MAP_PTR. Move the testing part to bpf selftest in next patch. v4: no update. v3: add rxcnt map to show the packet transmit speed. v2: no update. --- samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 43 ++++++ samples/bpf/xdp_redirect_map_multi_user.c | 166 ++++++++++++++++++++++ 3 files changed, 212 insertions(+) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 4f1ed0e3cf9f..cad63d4ea164 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map tprogs-y += per_socket_stats_example tprogs-y += xdp_redirect tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_map_multi tprogs-y += xdp_redirect_cpu tprogs-y += xdp_monitor tprogs-y += xdp_rxq_info @@ -98,6 +99,7 @@ test_map_in_map-objs := test_map_in_map_user.o per_socket_stats_example-objs := cookie_uid_helper_example.o xdp_redirect-objs := xdp_redirect_user.o xdp_redirect_map-objs := xdp_redirect_map_user.o +xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o xdp_redirect_cpu-objs := bpf_load.o xdp_redirect_cpu_user.o xdp_monitor-objs := bpf_load.o xdp_monitor_user.o xdp_rxq_info-objs := xdp_rxq_info_user.o @@ -158,6 +160,7 @@ always-y += tcp_tos_reflect_kern.o always-y += tcp_dumpstats_kern.o always-y += xdp_redirect_kern.o always-y += xdp_redirect_map_kern.o +always-y += xdp_redirect_map_multi_kern.o always-y += xdp_redirect_cpu_kern.o always-y += xdp_monitor_kern.o always-y += xdp_rxq_info_kern.o diff --git a/samples/bpf/xdp_redirect_map_multi_kern.c b/samples/bpf/xdp_redirect_map_multi_kern.c new file mode 100644 index 000000000000..db58d56cef89 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_kern.c @@ -0,0 +1,43 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <uapi/linux/bpf.h> +#include <bpf/bpf_helpers.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 256); +} forward_map SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __type(key, u32); + __type(value, long); + __uint(max_entries, 1); +} rxcnt SEC(".maps"); + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + long *value; + u32 key = 0; + + /* count packet in global counter */ + value = bpf_map_lookup_elem(&rxcnt, &key); + if (value) + *value += 1; + + return bpf_redirect_map_multi(&forward_map, NULL, BPF_F_EXCLUDE_INGRESS); +} + +char _license[] SEC("license") = "GPL"; diff --git a/samples/bpf/xdp_redirect_map_multi_user.c b/samples/bpf/xdp_redirect_map_multi_user.c new file mode 100644 index 000000000000..49f44c91b672 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_user.c @@ -0,0 +1,166 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; +static int rxcnt; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void poll_stats(int interval) +{ + unsigned int nr_cpus = bpf_num_possible_cpus(); + __u64 values[nr_cpus], prev[nr_cpus]; + + memset(prev, 0, sizeof(prev)); + + while (1) { + __u64 sum = 0; + __u32 key = 0; + int i; + + sleep(interval); + assert(bpf_map_lookup_elem(rxcnt, &key, values) == 0); + for (i = 0; i < nr_cpus; i++) + sum += (values[i] - prev[i]); + if (sum) + printf("Forwarding %10llu pkt/s\n", sum / interval); + memcpy(prev, values, sizeof(values)); + } +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n", + prog); +} + +int main(int argc, char **argv) +{ + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + int prog_fd, forward_map; + int i, ret, opt, ifindex; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNF")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) + xdp_flags |= XDP_FLAGS_DRV_MODE; + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + return 1; + + forward_map = bpf_object__find_map_fd_by_name(obj, "forward_map"); + rxcnt = bpf_object__find_map_fd_by_name(obj, "rxcnt"); + + if (forward_map < 0 || rxcnt < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + return 1; + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* Add all the interfaces to group all */ + ret = bpf_map_update_elem(forward_map, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + } + + poll_stats(2); + + return 0; + +err_out: + return 1; +} -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv11 bpf-next 4/5] selftests/bpf: add xdp_redirect_multi test 2020-09-07 8:27 ` [PATCHv11 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu ` (2 preceding siblings ...) 2020-09-07 8:27 ` [PATCHv11 bpf-next 3/5] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu @ 2020-09-07 8:27 ` Hangbin Liu 2020-09-07 8:27 ` [PATCHv11 bpf-next 5/5] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2020-12-16 14:30 ` [PATCHv12 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 5 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-09-07 8:27 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Hangbin Liu Add a bpf selftest for new helper xdp_redirect_map_multi(). In this test we have 3 forward groups and 1 exclude group. The test will redirect each interface's packets to all the interfaces in the forward group, and exclude the interface in exclude map. We will also test both DEVMAP and DEVMAP_HASH with xdp generic and drv. For more test details, you can find it in the test script. Here is the test result. ]# ./test_xdp_redirect_multi.sh Pass: xdpgeneric arp ns1-2 Pass: xdpgeneric arp ns1-3 Pass: xdpgeneric arp ns1-4 Pass: xdpgeneric ping ns1-2 Pass: xdpgeneric ping ns1-3 Pass: xdpgeneric ping ns1-4 Pass: xdpgeneric ping6 ns2-1 Pass: xdpgeneric ping6 ns2-3 Pass: xdpgeneric ping6 ns2-4 Pass: xdpdrv arp ns1-2 Pass: xdpdrv arp ns1-3 Pass: xdpdrv arp ns1-4 Pass: xdpdrv ping ns1-2 Pass: xdpdrv ping ns1-3 Pass: xdpdrv ping ns1-4 Pass: xdpdrv ping6 ns2-1 Pass: xdpdrv ping6 ns2-3 Pass: xdpdrv ping6 ns2-4 Summary: PASS 18, FAIL 0 Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v10-v11: no update v9: use NULL directly for arg2 and redefine the maps with btf format v2-v8: no update --- tools/testing/selftests/bpf/Makefile | 4 +- .../bpf/progs/xdp_redirect_multi_kern.c | 77 ++++++++ .../selftests/bpf/test_xdp_redirect_multi.sh | 164 +++++++++++++++++ .../selftests/bpf/xdp_redirect_multi.c | 173 ++++++++++++++++++ 4 files changed, 417 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 65d3d9aaeb31..61f063618e48 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -51,6 +51,7 @@ TEST_FILES = test_lwt_ip_encap.o \ # Order correspond to 'make run_tests' order TEST_PROGS := test_kmod.sh \ test_xdp_redirect.sh \ + test_xdp_redirect_multi.sh \ test_xdp_meta.sh \ test_xdp_veth.sh \ test_offload.py \ @@ -79,7 +80,8 @@ TEST_PROGS_EXTENDED := with_addr.sh \ # Compile but not part of 'make run_tests' TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ - test_lirc_mode2_user xdping test_cpp runqslower bench + test_lirc_mode2_user xdping test_cpp runqslower bench \ + xdp_redirect_multi TEST_CUSTOM_PROGS = urandom_read diff --git a/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c new file mode 100644 index 000000000000..30c703efcb28 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c @@ -0,0 +1,77 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <string.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ip.h> +#include <linux/ipv6.h> + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_endian.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v4 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v6 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_all SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} exclude_map SEC(".maps"); + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + struct ethhdr *eth = data; + __u16 h_proto; + __u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + h_proto = eth->h_proto; + + if (h_proto == bpf_htons(ETH_P_IP)) + return bpf_redirect_map_multi(&forward_map_v4, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else if (h_proto == bpf_htons(ETH_P_IPV6)) + return bpf_redirect_map_multi(&forward_map_v6, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else + return bpf_redirect_map_multi(&forward_map_all, NULL, + BPF_F_EXCLUDE_INGRESS); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh new file mode 100755 index 000000000000..c1395d8634c5 --- /dev/null +++ b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh @@ -0,0 +1,164 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Test topology: +# - - - - - - - - - - - - - - - - - - - - - - - - - +# | veth1 veth2 veth3 veth4 | ... init net +# - -| - - - - - - | - - - - - - | - - - - - - | - - +# --------- --------- --------- --------- +# | veth0 | | veth0 | | veth0 | | veth0 | ... +# --------- --------- --------- --------- +# ns1 ns2 ns3 ns4 +# +# Forward multicast groups: +# Forward group all has interfaces: veth1, veth2, veth3, veth4, ... (All traffic except IPv4, IPv6) +# Forward group v4 has interfaces: veth1, veth3, veth4, ... (For IPv4 traffic only) +# Forward group v6 has interfaces: veth2, veth3, veth4, ... (For IPv6 traffic only) +# Exclude Groups: +# Exclude group: veth3 (assume ns3 is in black list) +# +# Test modules: +# XDP modes: generic, native +# map types: group v4 use DEVMAP, others use DEVMAP_HASH +# +# Test cases: +# ARP(we didn't block ARP for ns3): +# ns1 -> gw: ns2, ns3, ns4 should receive the arp request +# IPv4: +# ns1 -> ns2 (fail), ns1 -> ns3 (fail), ns1 -> ns4 (pass) +# IPv6 +# ns2 -> ns1 (fail), ns2 -> ns3 (fail), ns2 -> ns4 (pass) +# + + +# netns numbers +NUM=4 +IFACES="" +DRV_MODE="xdpgeneric xdpdrv" +PASS=0 +FAIL=0 + +test_pass() +{ + echo "Pass: $@" + PASS=$((PASS + 1)) +} + +test_fail() +{ + echo "fail: $@" + FAIL=$((FAIL + 1)) +} + +clean_up() +{ + for i in $(seq $NUM); do + ip link del veth$i 2> /dev/null + ip netns del ns$i 2> /dev/null + done + rm -f xdp_redirect_*.log arp_ns*.log +} + +# Kselftest framework requirement - SKIP code is 4. +check_env() +{ + ip link set dev lo xdpgeneric off &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without the ip xdpgeneric support" + exit 4 + fi + + which tcpdump &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without tcpdump" + exit 4 + fi +} + +setup_ns() +{ + local mode=$1 + IFACES="" + + for i in $(seq $NUM); do + ip netns add ns$i + ip link add veth$i type veth peer name veth0 netns ns$i + ip link set veth$i up + ip -n ns$i link set veth0 up + + ip -n ns$i addr add 192.0.2.$i/24 dev veth0 + ip -n ns$i addr add 2001:db8::$i/64 dev veth0 + ip -n ns$i link set veth0 $mode obj \ + xdp_dummy.o sec xdp_dummy &> /dev/null || \ + { test_fail "Unable to load dummy xdp" && exit 1; } + IFACES="$IFACES veth$i" + done +} + +do_ping_tests() +{ + local mode=$1 + + # arp test + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -i veth0 -nn -l -e &> arp_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-2_${mode}.log && \ + test_pass "$mode arp ns1-2" || test_fail "$mode arp ns1-2" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-3_${mode}.log && \ + test_pass "$mode arp ns1-3" || test_fail "$mode arp ns1-3" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-4_${mode}.log && \ + test_pass "$mode arp ns1-4" || test_fail "$mode arp ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-2" || test_pass "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-3" || test_pass "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" + + # ping6 test + ip netns exec ns2 ping6 2001:db8::1 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-1" || test_pass "$mode ping6 ns2-1" + ip netns exec ns2 ping6 2001:db8::3 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-3" || test_pass "$mode ping6 ns2-3" + ip netns exec ns2 ping6 2001:db8::4 -c 4 &> /dev/null && \ + test_pass "$mode ping6 ns2-4" || test_fail "$mode ping6 ns2-4" +} + +do_tests() +{ + local mode=$1 + local drv_p + + [ ${mode} == "xdpdrv" ] && drv_p="-N" || drv_p="-S" + + # run `ulimit -l unlimited` if you got errors like + # libbpf: Error in bpf_object__probe_global_data():Operation not permitted(1). + ./xdp_redirect_multi $drv_p $IFACES &> xdp_redirect_${mode}.log & + xdp_pid=$! + sleep 10 + + do_ping_tests $mode + + kill $xdp_pid +} + +trap clean_up 0 2 3 6 9 + +check_env + +for mode in ${DRV_MODE}; do + setup_ns $mode + do_tests $mode + sleep 10 + clean_up + sleep 5 +done + +echo "Summary: PASS $PASS, FAIL $FAIL" +[ $FAIL -eq 0 ] && exit 0 || exit 1 diff --git a/tools/testing/selftests/bpf/xdp_redirect_multi.c b/tools/testing/selftests/bpf/xdp_redirect_multi.c new file mode 100644 index 000000000000..5626005cb679 --- /dev/null +++ b/tools/testing/selftests/bpf/xdp_redirect_multi.c @@ -0,0 +1,173 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n", + prog); +} + +int main(int argc, char **argv) +{ + int prog_fd, group_all, group_v4, group_v6, exclude; + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + int i, ret, opt, ifindex; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNF")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) + xdp_flags |= XDP_FLAGS_DRV_MODE; + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + return 1; + + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_all"); + group_v4 = bpf_object__find_map_fd_by_name(obj, "forward_map_v4"); + group_v6 = bpf_object__find_map_fd_by_name(obj, "forward_map_v6"); + exclude = bpf_object__find_map_fd_by_name(obj, "exclude_map"); + + if (group_all < 0 || group_v4 < 0 || group_v6 < 0 || exclude < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + return 1; + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* Add all the interfaces to group all */ + ret = bpf_map_update_elem(group_all, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* For testing: remove the 1st interfaces from group v6 */ + if (i != 0) { + ret = bpf_map_update_elem(group_v6, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: remove the 2nd interfaces from group v4 */ + if (i != 1) { + ret = bpf_map_update_elem(group_v4, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: add the 3rd interfaces to exclude map */ + if (i == 2) { + ret = bpf_map_update_elem(exclude, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + } + + /* sleep some time for testing */ + sleep(999); + + return 0; + +err_out: + return 1; +} -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv11 bpf-next 5/5] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL 2020-09-07 8:27 ` [PATCHv11 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu ` (3 preceding siblings ...) 2020-09-07 8:27 ` [PATCHv11 bpf-next 4/5] selftests/bpf: add xdp_redirect_multi test Hangbin Liu @ 2020-09-07 8:27 ` Hangbin Liu 2020-12-16 14:30 ` [PATCHv12 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 5 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-09-07 8:27 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Hangbin Liu Use helper bpf_redirect_map() and bpf_redirect_map_multi() to test bpf arg ARG_CONST_MAP_PTR and ARG_CONST_MAP_PTR_OR_NULL. Make sure the map arg could be verified correctly when it is NULL or valid map pointer. Add devmap and devmap_hash in struct bpf_test due to bpf_redirect_{map, map_multi} limit. Test result: ]# ./test_verifier 702 705 #702/p ARG_CONST_MAP_PTR: null pointer OK #703/p ARG_CONST_MAP_PTR: valid map pointer OK #704/p ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map OK #705/p ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map OK Summary: 4 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v2-v11: no update --- tools/testing/selftests/bpf/test_verifier.c | 22 +++++- .../testing/selftests/bpf/verifier/map_ptr.c | 70 +++++++++++++++++++ 2 files changed, 91 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c index 9be395d9dc64..f89a13e60692 100644 --- a/tools/testing/selftests/bpf/test_verifier.c +++ b/tools/testing/selftests/bpf/test_verifier.c @@ -50,7 +50,7 @@ #define MAX_INSNS BPF_MAXINSNS #define MAX_TEST_INSNS 1000000 #define MAX_FIXUPS 8 -#define MAX_NR_MAPS 20 +#define MAX_NR_MAPS 22 #define MAX_TEST_RUNS 8 #define POINTER_VALUE 0xcafe4all #define TEST_DATA_LEN 64 @@ -87,6 +87,8 @@ struct bpf_test { int fixup_sk_storage_map[MAX_FIXUPS]; int fixup_map_event_output[MAX_FIXUPS]; int fixup_map_reuseport_array[MAX_FIXUPS]; + int fixup_map_devmap[MAX_FIXUPS]; + int fixup_map_devmap_hash[MAX_FIXUPS]; const char *errstr; const char *errstr_unpriv; uint32_t insn_processed; @@ -640,6 +642,8 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, int *fixup_sk_storage_map = test->fixup_sk_storage_map; int *fixup_map_event_output = test->fixup_map_event_output; int *fixup_map_reuseport_array = test->fixup_map_reuseport_array; + int *fixup_map_devmap = test->fixup_map_devmap; + int *fixup_map_devmap_hash = test->fixup_map_devmap_hash; if (test->fill_helper) { test->fill_insns = calloc(MAX_TEST_INSNS, sizeof(struct bpf_insn)); @@ -817,6 +821,22 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, fixup_map_reuseport_array++; } while (*fixup_map_reuseport_array); } + if (*fixup_map_devmap) { + map_fds[20] = __create_map(BPF_MAP_TYPE_DEVMAP, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap].imm = map_fds[20]; + fixup_map_devmap++; + } while (*fixup_map_devmap); + } + if (*fixup_map_devmap_hash) { + map_fds[21] = __create_map(BPF_MAP_TYPE_DEVMAP_HASH, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap_hash].imm = map_fds[21]; + fixup_map_devmap_hash++; + } while (*fixup_map_devmap_hash); + } } struct libcap { diff --git a/tools/testing/selftests/bpf/verifier/map_ptr.c b/tools/testing/selftests/bpf/verifier/map_ptr.c index b52209db8250..51df7d8784dc 100644 --- a/tools/testing/selftests/bpf/verifier/map_ptr.c +++ b/tools/testing/selftests/bpf/verifier/map_ptr.c @@ -60,3 +60,73 @@ .result = ACCEPT, .retval = 1, }, +{ + "ARG_CONST_MAP_PTR: null pointer", + .insns = { + /* bpf_redirect_map arg1 (map) */ + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .result = REJECT, + .prog_type = BPF_PROG_TYPE_XDP, + .errstr = "R1 type=inv expected=map_ptr", +}, +{ + "ARG_CONST_MAP_PTR: valid map pointer", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg1 (map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_LD_MAP_FD(BPF_REG_2, 1), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .fixup_map_devmap_hash = { 3 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, -- 2.25.4 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv12 bpf-next 0/6] xdp: add a new helper for dev map multicast support 2020-09-07 8:27 ` [PATCHv11 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu ` (4 preceding siblings ...) 2020-09-07 8:27 ` [PATCHv11 bpf-next 5/5] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2020-12-16 14:30 ` Hangbin Liu 2020-12-16 14:30 ` [PATCHv12 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu ` (6 more replies) 5 siblings, 7 replies; 219+ messages in thread From: Hangbin Liu @ 2020-12-16 14:30 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because there may have multi interfaces you want to exclude. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. The 1st patch is Jesper's run devmap xdp_prog later in bulking step. The 2st patch add a new bpf arg to allow NULL map pointer. The 3rd patch add the new bpf_redirect_map_multi() helper. The 4-6 patches are for usage sample and testing purpose. I did same perf tests with the following topo: --------------------- --------------------- | Host A (i40e 10G) | ---------- | eno1(i40e 10G) | --------------------- | | | Host B | --------------------- | | | Host C (i40e 10G) | ---------- | eno2(i40e 10G) | --------------------- vlan2 | -------- | | veth1 -- | veth0| | | -------- | --------------------| On Host A: # pktgen/pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -s 64 On Host B(Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, 128G Memory): Use xdp_redirect_map and xdp_redirect_map_multi in samples/bpf for testing. The veth0 in netns load dummy drop program. The forward_map max_entries in xdp_redirect_map_multi is modify to 4. Here is the perf result with 5.10 rc6: The are about +/- 0.1M deviation for native testing Version | Test | Generic | Native | Native + 2nd 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M 5.10 rc6 + patch1 | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch1 | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M 5.10 rc6 + patch1-6 | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch1-6 | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->i40e | 1.7M | 7.8M | 6.4M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->veth | 1.4M | 9.3M | 7.5M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->i40e+veth | 1.0M | 3.2M | 2.7M Last but not least, thanks a lot to Toke, Jesper, Jiri and Eelco for suggestions and help on implementation. [0] https://xdp-project.net/#Handling-multicast v12: Add Jesper's xdp_prog patch, rebase my works on this and latest bpf-next Add 2nd xdp_prog test on the sample and selftests. v11: Fix bpf_redirect_map_multi() helper description typo. Add loop limit for devmap_get_next_obj() and dev_map_redirect_multi(). v10: Rebase the code to latest bpf-next. Update helper bpf_xdp_redirect_map_multi() - No need to check map pointer as we will do the check in verifier. v9: Update helper bpf_xdp_redirect_map_multi() - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: a) Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). f) Split the tests from sample and add a bpf kernel selftest patch. v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Hangbin Liu (5): bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL xdp: add a new helper for dev map multicast support sample/bpf: add xdp_redirect_map_multicast test selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL selftests/bpf: add xdp_redirect_multi test Jesper Dangaard Brouer (1): bpf: run devmap xdp_prog on flush instead of bulk enqueue include/linux/bpf.h | 21 ++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 27 ++ kernel/bpf/devmap.c | 236 +++++++++++--- kernel/bpf/verifier.c | 16 +- net/core/filter.c | 118 ++++++- net/core/xdp.c | 29 ++ samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 96 ++++++ samples/bpf/xdp_redirect_map_multi_user.c | 301 ++++++++++++++++++ tools/include/uapi/linux/bpf.h | 27 ++ tools/testing/selftests/bpf/Makefile | 3 +- .../bpf/progs/xdp_redirect_multi_kern.c | 120 +++++++ tools/testing/selftests/bpf/test_verifier.c | 22 +- .../selftests/bpf/test_xdp_redirect_multi.sh | 208 ++++++++++++ .../testing/selftests/bpf/verifier/map_ptr.c | 70 ++++ .../selftests/bpf/xdp_redirect_multi.c | 258 +++++++++++++++ 18 files changed, 1509 insertions(+), 48 deletions(-) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c -- 2.26.2 ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv12 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2020-12-16 14:30 ` [PATCHv12 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2020-12-16 14:30 ` Hangbin Liu 2020-12-17 16:07 ` David Ahern 2020-12-16 14:30 ` [PATCHv12 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu ` (5 subsequent siblings) 6 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2020-12-16 14:30 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu From: Jesper Dangaard Brouer <brouer@redhat.com> This changes the devmap XDP program support to run the program when the bulk queue is flushed instead of before the frame is enqueued. This has a couple of benefits: - It "sorts" the packets by destination devmap entry, and then runs the same BPF program on all the packets in sequence. This ensures that we keep the XDP program and destination device properties hot in I-cache. - It makes the multicast implementation simpler because it can just enqueue packets using bq_enqueue() without having to deal with the devmap program at all. The drawback is that if the devmap program drops the packet, the enqueue step is redundant. However, arguably this is mostly visible in a micro-benchmark, and with more mixed traffic the I-cache benefit should win out. The performance impact of just this patch is as follows: Using xdp_redirect_map(with a 2nd xdp_prog patch[1]) in sample/bpf and send pkts via pktgen cmd: ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 There are about +/- 0.1M deviation for native testing, the performance improved for the base-case, but some drop back with xdp devmap prog attached. Version | Test | Generic | Native | Native + 2nd xdp_prog 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M 5.10 rc6 + patch | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M [1] https://patchwork.ozlabs.org/project/netdev/patch/20201208120159.2278277-1-liuhangbin@gmail.com/ Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com> Signed-off-by: Hangbin Liu <haliu@redhat.com> --- kernel/bpf/devmap.c | 116 ++++++++++++++++++++++++++++---------------- 1 file changed, 73 insertions(+), 43 deletions(-) diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index f6e9c68afdd4..2a83232cf63a 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -57,6 +57,7 @@ struct xdp_dev_bulk_queue { struct list_head flush_node; struct net_device *dev; struct net_device *dev_rx; + struct bpf_prog *xdp_prog; unsigned int count; }; @@ -327,40 +328,92 @@ bool dev_map_can_have_prog(struct bpf_map *map) return false; } +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, + struct xdp_frame **frames, int n, + struct net_device *dev) +{ + struct xdp_txq_info txq = { .dev = dev }; + struct xdp_buff xdp; + int i, nframes = 0; + + for (i = 0; i < n; i++) { + struct xdp_frame *xdpf = frames[i]; + u32 act; + int err; + + xdp_convert_frame_to_buff(xdpf, &xdp); + xdp.txq = &txq; + + act = bpf_prog_run_xdp(xdp_prog, &xdp); + switch (act) { + case XDP_PASS: + err = xdp_update_frame_from_buff(&xdp, xdpf); + if (unlikely(err < 0)) + xdp_return_frame_rx_napi(xdpf); + else + frames[nframes++] = xdpf; + break; + default: + bpf_warn_invalid_xdp_action(act); + fallthrough; + case XDP_ABORTED: + trace_xdp_exception(dev, xdp_prog, act); + fallthrough; + case XDP_DROP: + xdp_return_frame_rx_napi(xdpf); + break; + } + } + return n - nframes; /* dropped frames count */ +} + static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) { struct net_device *dev = bq->dev; int sent = 0, drops = 0, err = 0; + unsigned int cnt = bq->count; + unsigned int xdp_drop; int i; - if (unlikely(!bq->count)) + if (unlikely(!cnt)) return; - for (i = 0; i < bq->count; i++) { + for (i = 0; i < cnt; i++) { struct xdp_frame *xdpf = bq->q[i]; prefetch(xdpf); } - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); + if (unlikely(bq->xdp_prog)) { + xdp_drop = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); + cnt -= xdp_drop; + if (!cnt) { + sent = 0; + drops = xdp_drop; + goto out; + } + } + + sent = dev->netdev_ops->ndo_xdp_xmit(dev, cnt, bq->q, flags); if (sent < 0) { err = sent; sent = 0; goto error; } - drops = bq->count - sent; + drops = (cnt - sent) + xdp_drop; out: bq->count = 0; trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); bq->dev_rx = NULL; + bq->xdp_prog = NULL; __list_del_clearprev(&bq->flush_node); return; error: /* If ndo_xdp_xmit fails with an errno, no frames have been * xmit'ed and it's our responsibility to them free all. */ - for (i = 0; i < bq->count; i++) { + for (i = 0; i < cnt; i++) { struct xdp_frame *xdpf = bq->q[i]; xdp_return_frame_rx_napi(xdpf); @@ -408,7 +461,8 @@ struct bpf_dtab_netdev *__dev_map_lookup_elem(struct bpf_map *map, u32 key) * Thus, safe percpu variable access. */ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, - struct net_device *dev_rx) + struct net_device *dev_rx, + struct bpf_dtab_netdev *dst) { struct list_head *flush_list = this_cpu_ptr(&dev_flush_list); struct xdp_dev_bulk_queue *bq = this_cpu_ptr(dev->xdp_bulkq); @@ -423,6 +477,14 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, if (!bq->dev_rx) bq->dev_rx = dev_rx; + /* Store (potential) xdp_prog that run before egress to dev as + * part of bulk_queue. This will be same xdp_prog for all + * xdp_frame's in bulk_queue, because this per-CPU store must + * be flushed from net_device drivers NAPI func end. + */ + if (dst && dst->xdp_prog && !bq->xdp_prog) + bq->xdp_prog = dst->xdp_prog; + bq->q[bq->count++] = xdpf; if (!bq->flush_node.prev) @@ -430,7 +492,8 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, } static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, - struct net_device *dev_rx) + struct net_device *dev_rx, + struct bpf_dtab_netdev *dst) { struct xdp_frame *xdpf; int err; @@ -446,42 +509,14 @@ static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, if (unlikely(!xdpf)) return -EOVERFLOW; - bq_enqueue(dev, xdpf, dev_rx); + bq_enqueue(dev, xdpf, dev_rx, dst); return 0; } -static struct xdp_buff *dev_map_run_prog(struct net_device *dev, - struct xdp_buff *xdp, - struct bpf_prog *xdp_prog) -{ - struct xdp_txq_info txq = { .dev = dev }; - u32 act; - - xdp_set_data_meta_invalid(xdp); - xdp->txq = &txq; - - act = bpf_prog_run_xdp(xdp_prog, xdp); - switch (act) { - case XDP_PASS: - return xdp; - case XDP_DROP: - break; - default: - bpf_warn_invalid_xdp_action(act); - fallthrough; - case XDP_ABORTED: - trace_xdp_exception(dev, xdp_prog, act); - break; - } - - xdp_return_buff(xdp); - return NULL; -} - int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx) { - return __xdp_enqueue(dev, xdp, dev_rx); + return __xdp_enqueue(dev, xdp, dev_rx, NULL); } int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, @@ -489,12 +524,7 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, { struct net_device *dev = dst->dev; - if (dst->xdp_prog) { - xdp = dev_map_run_prog(dev, xdp, dst->xdp_prog); - if (!xdp) - return 0; - } - return __xdp_enqueue(dev, xdp, dev_rx); + return __xdp_enqueue(dev, xdp, dev_rx, dst); } int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv12 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2020-12-16 14:30 ` [PATCHv12 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu @ 2020-12-17 16:07 ` David Ahern 2020-12-18 9:43 ` Hangbin Liu 2020-12-21 8:38 ` Hangbin Liu 0 siblings, 2 replies; 219+ messages in thread From: David Ahern @ 2020-12-17 16:07 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu On 12/16/20 7:30 AM, Hangbin Liu wrote: > @@ -327,40 +328,92 @@ bool dev_map_can_have_prog(struct bpf_map *map) > return false; > } > > +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, > + struct xdp_frame **frames, int n, > + struct net_device *dev) > +{ > + struct xdp_txq_info txq = { .dev = dev }; > + struct xdp_buff xdp; > + int i, nframes = 0; > + > + for (i = 0; i < n; i++) { > + struct xdp_frame *xdpf = frames[i]; > + u32 act; > + int err; > + > + xdp_convert_frame_to_buff(xdpf, &xdp); > + xdp.txq = &txq; > + > + act = bpf_prog_run_xdp(xdp_prog, &xdp); > + switch (act) { > + case XDP_PASS: > + err = xdp_update_frame_from_buff(&xdp, xdpf); > + if (unlikely(err < 0)) > + xdp_return_frame_rx_napi(xdpf); > + else > + frames[nframes++] = xdpf; > + break; > + default: > + bpf_warn_invalid_xdp_action(act); > + fallthrough; > + case XDP_ABORTED: > + trace_xdp_exception(dev, xdp_prog, act); > + fallthrough; > + case XDP_DROP: > + xdp_return_frame_rx_napi(xdpf); > + break; > + } > + } > + return n - nframes; /* dropped frames count */ just return nframes here, since ... > +} > + > static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) > { > struct net_device *dev = bq->dev; > int sent = 0, drops = 0, err = 0; > + unsigned int cnt = bq->count; > + unsigned int xdp_drop; > int i; > > - if (unlikely(!bq->count)) > + if (unlikely(!cnt)) > return; > > - for (i = 0; i < bq->count; i++) { > + for (i = 0; i < cnt; i++) { > struct xdp_frame *xdpf = bq->q[i]; > > prefetch(xdpf); > } > > - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); > + if (unlikely(bq->xdp_prog)) { > + xdp_drop = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); > + cnt -= xdp_drop; ... that is apparently what you really want. > + if (!cnt) { > + sent = 0; > + drops = xdp_drop; > + goto out; > + } > + } > + > + sent = dev->netdev_ops->ndo_xdp_xmit(dev, cnt, bq->q, flags); > if (sent < 0) { > err = sent; > sent = 0; > goto error; > } > - drops = bq->count - sent; > + drops = (cnt - sent) + xdp_drop; > out: > bq->count = 0; > > trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); > bq->dev_rx = NULL; > + bq->xdp_prog = NULL; > __list_del_clearprev(&bq->flush_node); > return; > error: > /* If ndo_xdp_xmit fails with an errno, no frames have been > * xmit'ed and it's our responsibility to them free all. > */ > - for (i = 0; i < bq->count; i++) { > + for (i = 0; i < cnt; i++) { > struct xdp_frame *xdpf = bq->q[i]; > > xdp_return_frame_rx_napi(xdpf); > @@ -408,7 +461,8 @@ struct bpf_dtab_netdev *__dev_map_lookup_elem(struct bpf_map *map, u32 key) > * Thus, safe percpu variable access. > */ > static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, > - struct net_device *dev_rx) > + struct net_device *dev_rx, > + struct bpf_dtab_netdev *dst) > { > struct list_head *flush_list = this_cpu_ptr(&dev_flush_list); > struct xdp_dev_bulk_queue *bq = this_cpu_ptr(dev->xdp_bulkq); > @@ -423,6 +477,14 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, > if (!bq->dev_rx) > bq->dev_rx = dev_rx; > > + /* Store (potential) xdp_prog that run before egress to dev as > + * part of bulk_queue. This will be same xdp_prog for all > + * xdp_frame's in bulk_queue, because this per-CPU store must > + * be flushed from net_device drivers NAPI func end. > + */ > + if (dst && dst->xdp_prog && !bq->xdp_prog) > + bq->xdp_prog = dst->xdp_prog; if you pass in xdp_prog through __xdp_enqueue you can reduce that to just: if (!bq->xdp_prog) bq->xdp_prog = xdp_prog; > bq->q[bq->count++] = xdpf; > > if (!bq->flush_node.prev) > @@ -430,7 +492,8 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, > } > > static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, > - struct net_device *dev_rx) > + struct net_device *dev_rx, > + struct bpf_dtab_netdev *dst) > { > struct xdp_frame *xdpf; > int err; > @@ -446,42 +509,14 @@ static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, > if (unlikely(!xdpf)) > return -EOVERFLOW; > > - bq_enqueue(dev, xdpf, dev_rx); > + bq_enqueue(dev, xdpf, dev_rx, dst); > return 0; > } > > -static struct xdp_buff *dev_map_run_prog(struct net_device *dev, > - struct xdp_buff *xdp, > - struct bpf_prog *xdp_prog) > -{ > - struct xdp_txq_info txq = { .dev = dev }; > - u32 act; > - > - xdp_set_data_meta_invalid(xdp); > - xdp->txq = &txq; > - > - act = bpf_prog_run_xdp(xdp_prog, xdp); > - switch (act) { > - case XDP_PASS: > - return xdp; > - case XDP_DROP: > - break; > - default: > - bpf_warn_invalid_xdp_action(act); > - fallthrough; > - case XDP_ABORTED: > - trace_xdp_exception(dev, xdp_prog, act); > - break; > - } > - > - xdp_return_buff(xdp); > - return NULL; > -} > - > int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, > struct net_device *dev_rx) > { > - return __xdp_enqueue(dev, xdp, dev_rx); > + return __xdp_enqueue(dev, xdp, dev_rx, NULL); > } > > int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > @@ -489,12 +524,7 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > { > struct net_device *dev = dst->dev; > > - if (dst->xdp_prog) { > - xdp = dev_map_run_prog(dev, xdp, dst->xdp_prog); > - if (!xdp) > - return 0; > - } > - return __xdp_enqueue(dev, xdp, dev_rx); > + return __xdp_enqueue(dev, xdp, dev_rx, dst); > } > > int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, > ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv12 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2020-12-17 16:07 ` David Ahern @ 2020-12-18 9:43 ` Hangbin Liu 2020-12-21 8:38 ` Hangbin Liu 1 sibling, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-12-18 9:43 UTC (permalink / raw) To: David Ahern Cc: Hangbin Liu, bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Andrii Nakryiko, Alexei Starovoitov Hi David, Thanks for the comment. On Thu, Dec 17, 2020 at 09:07:03AM -0700, David Ahern wrote: > > + return n - nframes; /* dropped frames count */ > > just return nframes here, since ... > > > + xdp_drop = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); > > + cnt -= xdp_drop; > > ... that is apparently what you really want. I will fix this > > + if (dst && dst->xdp_prog && !bq->xdp_prog) > > + bq->xdp_prog = dst->xdp_prog; > > > if you pass in xdp_prog through __xdp_enqueue you can reduce that to just: > > if (!bq->xdp_prog) > bq->xdp_prog = xdp_prog; And this in the next PATCH version. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv12 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2020-12-17 16:07 ` David Ahern 2020-12-18 9:43 ` Hangbin Liu @ 2020-12-21 8:38 ` Hangbin Liu 1 sibling, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-12-21 8:38 UTC (permalink / raw) To: David Ahern Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu Hi David, I just aware that, On Thu, Dec 17, 2020 at 09:07:03AM -0700, David Ahern wrote: > > +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, > > + struct xdp_frame **frames, int n, > > + struct net_device *dev) > > +{ > > + struct xdp_txq_info txq = { .dev = dev }; > > + struct xdp_buff xdp; > > + int i, nframes = 0; > > + > > + for (i = 0; i < n; i++) { > > + struct xdp_frame *xdpf = frames[i]; > > + u32 act; > > + int err; > > + > > + xdp_convert_frame_to_buff(xdpf, &xdp); > > + xdp.txq = &txq; > > + > > + act = bpf_prog_run_xdp(xdp_prog, &xdp); > > + switch (act) { > > + case XDP_PASS: > > + err = xdp_update_frame_from_buff(&xdp, xdpf); > > + if (unlikely(err < 0)) > > + xdp_return_frame_rx_napi(xdpf); > > + else > > + frames[nframes++] = xdpf; > > + break; > > + default: > > + bpf_warn_invalid_xdp_action(act); > > + fallthrough; > > + case XDP_ABORTED: > > + trace_xdp_exception(dev, xdp_prog, act); > > + fallthrough; > > + case XDP_DROP: > > + xdp_return_frame_rx_napi(xdpf); > > + break; > > + } > > + } > > + return n - nframes; /* dropped frames count */ > > just return nframes here, since ... If we return nframes here, > > > +} > > + > > static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) > > { > > struct net_device *dev = bq->dev; > > int sent = 0, drops = 0, err = 0; > > + unsigned int cnt = bq->count; > > + unsigned int xdp_drop; > > int i; > > > > - if (unlikely(!bq->count)) > > + if (unlikely(!cnt)) > > return; > > > > - for (i = 0; i < bq->count; i++) { > > + for (i = 0; i < cnt; i++) { > > struct xdp_frame *xdpf = bq->q[i]; > > > > prefetch(xdpf); > > } > > > > - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); > > + if (unlikely(bq->xdp_prog)) { > > + xdp_drop = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); > > + cnt -= xdp_drop; > > ... that is apparently what you really want. then this will be cnt = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); xdp_drop = bq->count - cnt; So there is no much difference whether we return passed frames or dropped frames. > > > + if (!cnt) { > > + sent = 0; > > + drops = xdp_drop; > > + goto out; > > + } > > + } Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv12 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL 2020-12-16 14:30 ` [PATCHv12 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-12-16 14:30 ` [PATCHv12 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu @ 2020-12-16 14:30 ` Hangbin Liu 2020-12-16 14:30 ` [PATCHv12 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (4 subsequent siblings) 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-12-16 14:30 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu Add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL which could be used when we want to allow NULL pointer for map parameter. The bpf helper need to take care and check if the map is NULL when use this type. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v11-v12: rebase the patch to latest bpf-next v10: remove useless CONST_PTR_TO_MAP_OR_NULL and Copy-paste comment. v9: merge the patch from [1] in to this series. v1-v8: no this patch [1] https://lore.kernel.org/bpf/20200715070001.2048207-1-liuhangbin@gmail.com/ --- include/linux/bpf.h | 1 + kernel/bpf/verifier.c | 10 ++++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 07cb5d15e743..7850c87456fc 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -296,6 +296,7 @@ enum bpf_arg_type { ARG_CONST_ALLOC_SIZE_OR_ZERO, /* number of allocated bytes requested */ ARG_PTR_TO_BTF_ID_SOCK_COMMON, /* pointer to in-kernel sock_common or bpf-mirrored bpf_sock */ ARG_PTR_TO_PERCPU_BTF_ID, /* pointer to in-kernel percpu type */ + ARG_CONST_MAP_PTR_OR_NULL, /* const argument used as pointer to bpf_map or NULL */ __BPF_ARG_TYPE_MAX, }; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 17270b8404f1..9f6633c9ea12 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -445,7 +445,8 @@ static bool arg_type_may_be_null(enum bpf_arg_type type) type == ARG_PTR_TO_MEM_OR_NULL || type == ARG_PTR_TO_CTX_OR_NULL || type == ARG_PTR_TO_SOCKET_OR_NULL || - type == ARG_PTR_TO_ALLOC_MEM_OR_NULL; + type == ARG_PTR_TO_ALLOC_MEM_OR_NULL || + type == ARG_CONST_MAP_PTR_OR_NULL; } /* Determine whether the function releases some resources allocated by another @@ -4065,6 +4066,7 @@ static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = { [ARG_CONST_SIZE_OR_ZERO] = &scalar_types, [ARG_CONST_ALLOC_SIZE_OR_ZERO] = &scalar_types, [ARG_CONST_MAP_PTR] = &const_map_ptr_types, + [ARG_CONST_MAP_PTR_OR_NULL] = &const_map_ptr_types, [ARG_PTR_TO_CTX] = &context_types, [ARG_PTR_TO_CTX_OR_NULL] = &context_types, [ARG_PTR_TO_SOCK_COMMON] = &sock_types, @@ -4210,9 +4212,9 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, meta->ref_obj_id = reg->ref_obj_id; } - if (arg_type == ARG_CONST_MAP_PTR) { - /* bpf_map_xxx(map_ptr) call: remember that map_ptr */ - meta->map_ptr = reg->map_ptr; + if (arg_type == ARG_CONST_MAP_PTR || + arg_type == ARG_CONST_MAP_PTR_OR_NULL) { + meta->map_ptr = register_is_null(reg) ? NULL : reg->map_ptr; } else if (arg_type == ARG_PTR_TO_MAP_KEY) { /* bpf_map_xxx(..., map_ptr, ..., key) call: * check that [key, key + map->key_size) are within -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv12 bpf-next 3/6] xdp: add a new helper for dev map multicast support 2020-12-16 14:30 ` [PATCHv12 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-12-16 14:30 ` [PATCHv12 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu 2020-12-16 14:30 ` [PATCHv12 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2020-12-16 14:30 ` Hangbin Liu 2020-12-16 14:30 ` [PATCHv12 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu ` (3 subsequent siblings) 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-12-16 14:30 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because a physical interface can be part of a logical grouping, such as a bond device. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. The forwarding map could be DEVMAP or DEVMAP_HASH, but the exclude map *must* be DEVMAP_HASH to get better performace. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. As both bpf_xdp_redirect_map() and this new helpers are using struct bpf_redirect_info, I add a new ex_map and set tgt_value to NULL in the new helper to make a difference with bpf_xdp_redirect_map(). Also I keep the general data path in net/core/filter.c, the native data path in kernel/bpf/devmap.c so we can use direct calls to get better performace. [0] https://xdp-project.net/#Handling-multicast Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v12: rebase the code based on Jespoer's devmap xdp_prog patch v11: Fix bpf_redirect_map_multi() helper description typo. Add loop limit for devmap_get_next_obj() and dev_map_redirect_multi(). v10: Update helper bpf_xdp_redirect_map_multi() - No need to check map pointer as we will do the check in verifier. v9: Update helper bpf_xdp_redirect_map_multi() - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. --- include/linux/bpf.h | 20 +++++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 27 +++++++ kernel/bpf/devmap.c | 132 +++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++ net/core/filter.c | 118 +++++++++++++++++++++++++++-- net/core/xdp.c | 29 ++++++++ tools/include/uapi/linux/bpf.h | 27 +++++++ 9 files changed, 356 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 7850c87456fc..2b4eb7b7f2e2 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1419,6 +1419,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx); int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, struct net_device *dev_rx); +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex); +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); bool dev_map_can_have_prog(struct bpf_map *map); @@ -1587,6 +1592,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return 0; } +static inline +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + return false; +} + +static inline +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + return 0; +} + struct sk_buff; static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, diff --git a/include/linux/filter.h b/include/linux/filter.h index 29c27656165b..55095e4759b4 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -620,6 +620,7 @@ struct bpf_redirect_info { u32 tgt_index; void *tgt_value; struct bpf_map *map; + struct bpf_map *ex_map; u32 kern_flags; struct bpf_nh_params nh; }; diff --git a/include/net/xdp.h b/include/net/xdp.h index 600acb307db6..2d90f641d9ac 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -145,6 +145,7 @@ void xdp_warn(const char *msg, const char *func, const int line); #define XDP_WARN(msg) xdp_warn(msg, __func__, __LINE__) struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); static inline void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 77d7c1bb2923..b22e79220df5 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3830,6 +3830,27 @@ union bpf_attr { * Return * A pointer to a struct socket on success or NULL if the file is * not a socket. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3995,6 +4016,7 @@ union bpf_attr { FN(ktime_get_coarse_ns), \ FN(ima_inode_hash), \ FN(sock_from_file), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -4171,6 +4193,11 @@ enum { BPF_F_BPRM_SECUREEXEC = (1ULL << 0), }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 2a83232cf63a..c4ecfa6b2873 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -527,6 +527,138 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return __xdp_enqueue(dev, xdp, dev_rx, dst); } +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + return dev_map_get_next_key(map, key, next_key); + case BPF_MAP_TYPE_DEVMAP_HASH: + return dev_map_hash_get_next_key(map, key, next_key); + default: + break; + } + + return -ENOENT; +} + +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + if (obj->dev->ifindex == exclude_ifindex) + return true; + + if (!map) + return false; + + return __dev_map_hash_lookup_elem(map, obj->dev->ifindex) != NULL; +} + +static struct bpf_dtab_netdev *devmap_get_next_obj(struct xdp_buff *xdp, struct bpf_map *map, + struct bpf_map *ex_map, u32 *key, + u32 *next_key, int ex_ifindex) +{ + struct bpf_dtab_netdev *obj; + struct net_device *dev; + u32 *tmp_key = key; + u32 index; + int err; + + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + return NULL; + + /* When using dev map hash, we could restart the hashtab traversal + * in case the key has been updated/removed in the mean time. + * So we may end up potentially looping due to traversal restarts + * from first elem. + * + * Let's use map's max_entries to limit the loop number. + */ + for (index = 0; index < map->max_entries; index++) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + obj = __dev_map_lookup_elem(map, *next_key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + obj = __dev_map_hash_lookup_elem(map, *next_key); + break; + default: + break; + } + + if (!obj || dev_in_exclude_map(obj, ex_map, ex_ifindex)) + goto find_next; + + dev = obj->dev; + + if (!dev->netdev_ops->ndo_xdp_xmit) + goto find_next; + + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + if (unlikely(err)) + goto find_next; + + return obj; + +find_next: + tmp_key = next_key; + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + break; + } + + return NULL; +} + +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + struct bpf_dtab_netdev *obj = NULL, *next_obj = NULL; + struct xdp_frame *xdpf, *nxdpf; + bool last_one = false; + int ex_ifindex; + u32 key, next_key; + + ex_ifindex = flags & BPF_F_EXCLUDE_INGRESS ? dev_rx->ifindex : 0; + + /* Find first available obj */ + obj = devmap_get_next_obj(xdp, map, ex_map, NULL, &key, ex_ifindex); + if (!obj) + return 0; + + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) + return -EOVERFLOW; + + for (;;) { + /* Check if we still have one more available obj */ + next_obj = devmap_get_next_obj(xdp, map, ex_map, &key, + &next_key, ex_ifindex); + if (!next_obj) + last_one = true; + + if (last_one) { + bq_enqueue(obj->dev, xdpf, dev_rx, obj); + return 0; + } + + nxdpf = xdpf_clone(xdpf); + if (unlikely(!nxdpf)) { + xdp_return_frame_rx_napi(xdpf); + return -ENOMEM; + } + + bq_enqueue(obj->dev, nxdpf, dev_rx, obj); + + /* Deal with next obj */ + obj = next_obj; + key = next_key; + } +} + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9f6633c9ea12..fd297b929edd 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4420,6 +4420,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: if (func_id != BPF_FUNC_redirect_map && + func_id != BPF_FUNC_redirect_map_multi && func_id != BPF_FUNC_map_lookup_elem) goto error; break; @@ -4524,6 +4525,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, map->map_type != BPF_MAP_TYPE_XSKMAP) goto error; break; + case BPF_FUNC_redirect_map_multi: + if (map->map_type != BPF_MAP_TYPE_DEVMAP && + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + goto error; + break; case BPF_FUNC_sk_redirect_map: case BPF_FUNC_msg_redirect_map: case BPF_FUNC_sock_map_update: diff --git a/net/core/filter.c b/net/core/filter.c index 255aeee72402..a5d97cb1054b 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3924,12 +3924,19 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { }; static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) + struct bpf_map *map, struct xdp_buff *xdp, + struct bpf_map *ex_map, u32 flags) { switch (map->map_type) { case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: - return dev_map_enqueue(fwd, xdp, dev_rx); + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) + return dev_map_enqueue(fwd, xdp, dev_rx); + else + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, flags); case BPF_MAP_TYPE_CPUMAP: return cpu_map_enqueue(fwd, xdp, dev_rx); case BPF_MAP_TYPE_XSKMAP: @@ -3986,12 +3993,14 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); struct bpf_map *map = READ_ONCE(ri->map); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (unlikely(!map)) { @@ -4003,7 +4012,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = dev_xdp_enqueue(fwd, xdp, dev); } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, ri->flags); } if (unlikely(err)) @@ -4017,6 +4026,62 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, } EXPORT_SYMBOL_GPL(xdp_do_redirect); +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, + struct bpf_prog *xdp_prog, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) + +{ + struct bpf_dtab_netdev *dst; + struct sk_buff *nskb; + bool exclude_ingress; + u32 key, next_key, index; + void *fwd; + int err; + + /* Get first key from forward map */ + err = map->ops->map_get_next_key(map, NULL, &key); + if (err) + return err; + + exclude_ingress = !!(flags & BPF_F_EXCLUDE_INGRESS); + + /* When using dev map hash, we could restart the hashtab traversal + * in case the key has been updated/removed in the mean time. + * So we may end up potentially looping due to traversal restarts + * from first elem. + * + * Let's use map's max_entries to limit the loop number. + */ + for (index = 0; index < map->max_entries; index++) { + fwd = __xdp_map_lookup_elem(map, key); + if (fwd) { + dst = (struct bpf_dtab_netdev *)fwd; + if (dev_in_exclude_map(dst, ex_map, + exclude_ingress ? dev->ifindex : 0)) + goto find_next; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -ENOMEM; + + /* Try forword next one no mater the current forward + * succeed or not */ + dev_map_generic_redirect(dst, nskb, xdp_prog); + } + +find_next: + err = map->ops->map_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + consume_skb(skb); + return 0; +} + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, @@ -4024,19 +4089,30 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, struct bpf_map *map) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err = 0; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (map->map_type == BPF_MAP_TYPE_DEVMAP || map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { - struct bpf_dtab_netdev *dst = fwd; + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) { + struct bpf_dtab_netdev *dst = fwd; + + err = dev_map_generic_redirect(dst, skb, xdp_prog); + } else { + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, + ex_map, ri->flags); + } - err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { @@ -4150,6 +4226,36 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { .arg3_type = ARG_ANYTHING, }; +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, + struct bpf_map *, ex_map, u64, flags) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + + /* Limit ex_map type to DEVMAP_HASH to get better performance */ + if (unlikely((ex_map && ex_map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) || + flags & ~BPF_F_EXCLUDE_INGRESS)) + return XDP_ABORTED; + + ri->tgt_index = 0; + /* Set the tgt_value to NULL to distinguish with bpf_xdp_redirect_map */ + ri->tgt_value = NULL; + ri->flags = flags; + ri->ex_map = ex_map; + + WRITE_ONCE(ri->map, map); + + return XDP_REDIRECT; +} + +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { + .func = bpf_xdp_redirect_map_multi, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_CONST_MAP_PTR, + .arg2_type = ARG_CONST_MAP_PTR_OR_NULL, + .arg3_type = ARG_ANYTHING, +}; + static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, unsigned long off, unsigned long len) { @@ -7227,6 +7333,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_redirect_proto; case BPF_FUNC_redirect_map: return &bpf_xdp_redirect_map_proto; + case BPF_FUNC_redirect_map_multi: + return &bpf_xdp_redirect_map_multi_proto; case BPF_FUNC_xdp_adjust_tail: return &bpf_xdp_adjust_tail_proto; case BPF_FUNC_fib_lookup: diff --git a/net/core/xdp.c b/net/core/xdp.c index 3a8c9ab4ecbe..6d86af029dc5 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -513,3 +513,32 @@ void xdp_warn(const char *msg, const char *func, const int line) WARN(1, "XDP_WARN: %s(line:%d): %s\n", func, line, msg); }; EXPORT_SYMBOL_GPL(xdp_warn); + +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) +{ + unsigned int headroom, totalsize; + struct xdp_frame *nxdpf; + struct page *page; + void *addr; + + headroom = xdpf->headroom + sizeof(*xdpf); + totalsize = headroom + xdpf->len; + + if (unlikely(totalsize > PAGE_SIZE)) + return NULL; + page = dev_alloc_page(); + if (!page) + return NULL; + addr = page_to_virt(page); + + memcpy(addr, xdpf, totalsize); + + nxdpf = addr; + nxdpf->data = addr + headroom; + nxdpf->frame_sz = PAGE_SIZE; + nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0; + nxdpf->mem.id = 0; + + return nxdpf; +} +EXPORT_SYMBOL_GPL(xdpf_clone); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 77d7c1bb2923..b22e79220df5 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3830,6 +3830,27 @@ union bpf_attr { * Return * A pointer to a struct socket on success or NULL if the file is * not a socket. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3995,6 +4016,7 @@ union bpf_attr { FN(ktime_get_coarse_ns), \ FN(ima_inode_hash), \ FN(sock_from_file), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -4171,6 +4193,11 @@ enum { BPF_F_BPRM_SECUREEXEC = (1ULL << 0), }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv12 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test 2020-12-16 14:30 ` [PATCHv12 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (2 preceding siblings ...) 2020-12-16 14:30 ` [PATCHv12 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2020-12-16 14:30 ` Hangbin Liu 2020-12-16 14:30 ` [PATCHv12 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu ` (2 subsequent siblings) 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-12-16 14:30 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu This is a sample for xdp multicast. In the sample we could forward all packets between given interfaces. There is also an option -X that could enable 2nd xdp_prog on egress interface. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v12: add devmap xdp_prog on egress support v10-v11: no update v9: use NULL directly for arg2 and redefine the maps with btf format v5: add a null_map as we have strict the arg2 to ARG_CONST_MAP_PTR. Move the testing part to bpf selftest in next patch. v4: no update. v3: add rxcnt map to show the packet transmit speed. v2: no update. --- samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 96 +++++++ samples/bpf/xdp_redirect_map_multi_user.c | 301 ++++++++++++++++++++++ 3 files changed, 400 insertions(+) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 26fc96ca619e..200029fcf53c 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map tprogs-y += per_socket_stats_example tprogs-y += xdp_redirect tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_map_multi tprogs-y += xdp_redirect_cpu tprogs-y += xdp_monitor tprogs-y += xdp_rxq_info @@ -99,6 +100,7 @@ test_map_in_map-objs := test_map_in_map_user.o per_socket_stats_example-objs := cookie_uid_helper_example.o xdp_redirect-objs := xdp_redirect_user.o xdp_redirect_map-objs := xdp_redirect_map_user.o +xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o xdp_redirect_cpu-objs := xdp_redirect_cpu_user.o xdp_monitor-objs := xdp_monitor_user.o xdp_rxq_info-objs := xdp_rxq_info_user.o @@ -160,6 +162,7 @@ always-y += tcp_tos_reflect_kern.o always-y += tcp_dumpstats_kern.o always-y += xdp_redirect_kern.o always-y += xdp_redirect_map_kern.o +always-y += xdp_redirect_map_multi_kern.o always-y += xdp_redirect_cpu_kern.o always-y += xdp_monitor_kern.o always-y += xdp_rxq_info_kern.o diff --git a/samples/bpf/xdp_redirect_map_multi_kern.c b/samples/bpf/xdp_redirect_map_multi_kern.c new file mode 100644 index 000000000000..0c63aace9bd2 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_kern.c @@ -0,0 +1,96 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <uapi/linux/bpf.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/ip.h> +#include <linux/ipv6.h> +#include <bpf/bpf_helpers.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 32); +} forward_map_general SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(struct bpf_devmap_val)); + __uint(max_entries, 32); +} forward_map_native SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __type(key, u32); + __type(value, long); + __uint(max_entries, 1); +} rxcnt SEC(".maps"); + +/* map to stroe egress interfaces mac addresses, set the + * max_entries to 1 and extend it in user sapce prog. + */ +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, u32); + __type(value, __be64); + __uint(max_entries, 1); +} mac_map SEC(".maps"); + +static int xdp_redirect_map(struct xdp_md *ctx, void *forward_map) +{ + long *value; + u32 key = 0; + + /* count packet in global counter */ + value = bpf_map_lookup_elem(&rxcnt, &key); + if (value) + *value += 1; + + return bpf_redirect_map_multi(forward_map, NULL, BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_redirect_general") +int xdp_redirect_map_general(struct xdp_md *ctx) +{ + return xdp_redirect_map(ctx, &forward_map_general); +} + +SEC("xdp_redirect_native") +int xdp_redirect_map_native(struct xdp_md *ctx) +{ + return xdp_redirect_map(ctx, &forward_map_native); +} + +SEC("xdp_devmap/map_prog") +int xdp_devmap_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + u32 key = ctx->egress_ifindex; + struct ethhdr *eth = data; + __be64 *mac; + u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + mac = bpf_map_lookup_elem(&mac_map, &key); + if (mac) + __builtin_memcpy(eth->h_source, mac, ETH_ALEN); + + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/samples/bpf/xdp_redirect_map_multi_user.c b/samples/bpf/xdp_redirect_map_multi_user.c new file mode 100644 index 000000000000..67ffc294567c --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_user.c @@ -0,0 +1,301 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> +#include <sys/resource.h> +#include <sys/ioctl.h> +#include <sys/types.h> +#include <sys/socket.h> +#include <netinet/in.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; +static int rxcnt_map_fd; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void poll_stats(int interval) +{ + unsigned int nr_cpus = bpf_num_possible_cpus(); + __u64 values[nr_cpus], prev[nr_cpus]; + + memset(prev, 0, sizeof(prev)); + + while (1) { + __u64 sum = 0; + __u32 key = 0; + int i; + + sleep(interval); + assert(bpf_map_lookup_elem(rxcnt_map_fd, &key, values) == 0); + for (i = 0; i < nr_cpus; i++) + sum += (values[i] - prev[i]); + if (sum) + printf("Forwarding %10llu pkt/s\n", sum / interval); + memcpy(prev, values, sizeof(values)); + } +} + +static int get_mac_addr(unsigned int ifindex, void *mac_addr) +{ + char ifname[IF_NAMESIZE]; + struct ifreq ifr; + int fd, ret = -1; + + fd = socket(AF_INET, SOCK_DGRAM, 0); + if (fd < 0) + return ret; + + if (!if_indextoname(ifindex, ifname)) + goto err_out; + + strcpy(ifr.ifr_name, ifname); + + if (ioctl(fd, SIOCGIFHWADDR, &ifr) != 0) + goto err_out; + + memcpy(mac_addr, ifr.ifr_hwaddr.sa_data, 6 * sizeof(char)); + ret = 0; + +err_out: + close(fd); + return ret; +} + +static int update_mac_map(struct bpf_object *obj) +{ + int i, ret = -1, mac_map_fd; + unsigned char mac_addr[6]; + unsigned int ifindex; + + mac_map_fd = bpf_object__find_map_fd_by_name(obj, "mac_map"); + if (mac_map_fd < 0) { + printf("find mac map fd failed\n"); + return ret; + } + + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + ret = get_mac_addr(ifindex, mac_addr); + if (ret < 0) { + printf("get interface %d mac failed\n", ifindex); + return ret; + } + + ret = bpf_map_update_elem(mac_map_fd, &ifindex, mac_addr, 0); + if (ret) { + perror("bpf_update_elem mac_map_fd"); + return ret; + } + } + + return 0; +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n" + " -X load xdp program on egress\n", + prog); +} + +int main(int argc, char **argv) +{ + int i, ret, opt, forward_map_fd, max_ifindex = 0; + struct bpf_program *ingress_prog, *egress_prog; + int ingress_prog_fd, egress_prog_fd = -1; + struct bpf_devmap_val devmap_val; + bool attach_egress_prog = false; + char ifname[IF_NAMESIZE]; + struct bpf_map *mac_map; + struct bpf_object *obj; + unsigned int ifindex; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNFX")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + case 'X': + attach_egress_prog = true; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) { + xdp_flags |= XDP_FLAGS_DRV_MODE; + } else if (attach_egress_prog) { + printf("Load xdp program on egress with SKB mode not supported yet\n"); + return 1; + } + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + + /* Find the largest index number */ + if (ifaces[i] > max_ifindex) + max_ifindex = ifaces[i]; + + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + + obj = bpf_object__open(filename); + if (libbpf_get_error(obj)) { + printf("ERROR: opening BPF object file failed\n"); + obj = NULL; + goto err_out; + } + + /* Reset the map size to max ifindex + 1 */ + if (attach_egress_prog) { + mac_map = bpf_object__find_map_by_name(obj, "mac_map"); + ret = bpf_map__resize(mac_map, max_ifindex + 1); + if (ret < 0) { + printf("ERROR: reset mac map size failed\n"); + goto err_out; + } + } + + /* load BPF program */ + if (bpf_object__load(obj)) { + printf("ERROR: loading BPF object file failed\n"); + goto err_out; + } + + if (xdp_flags & XDP_FLAGS_SKB_MODE) { + ingress_prog = bpf_object__find_program_by_title(obj, "xdp_redirect_general"); + forward_map_fd = bpf_object__find_map_fd_by_name(obj, "forward_map_general"); + } else { + ingress_prog = bpf_object__find_program_by_title(obj, "xdp_redirect_native"); + forward_map_fd = bpf_object__find_map_fd_by_name(obj, "forward_map_native"); + } + if (!ingress_prog || forward_map_fd < 0) { + printf("finding ingress_prog/forward_map in obj file failed\n"); + goto err_out; + } + + ingress_prog_fd = bpf_program__fd(ingress_prog); + if (ingress_prog_fd < 0) { + printf("find ingress_prog fd failed\n"); + goto err_out; + } + + rxcnt_map_fd = bpf_object__find_map_fd_by_name(obj, "rxcnt"); + if (rxcnt_map_fd < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + goto err_out; + } + + if (attach_egress_prog) { + /* Update mac_map with all egress interfaces' mac addr */ + if (update_mac_map(obj) < 0) { + printf("Error: update mac map failed"); + goto err_out; + } + + /* Find egress prog fd */ + egress_prog = bpf_object__find_program_by_title(obj, "xdp_devmap/map_prog"); + if (!egress_prog) { + printf("finding egress_prog in obj file failed\n"); + goto err_out; + } + egress_prog_fd = bpf_program__fd(egress_prog); + if (egress_prog_fd < 0) { + printf("find egress_prog fd failed\n"); + goto err_out; + } + } + + /* Remove attached program when program is interrupted or killed */ + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, ingress_prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + /* Add all the interfaces to forward group and attach + * egress devmap programe if exist */ + devmap_val.ifindex = ifindex; + devmap_val.bpf_prog.fd = egress_prog_fd; + ret = bpf_map_update_elem(forward_map_fd, &ifindex, &devmap_val, 0); + if (ret) { + perror("bpf_map_update_elem forward_map"); + goto err_out; + } + } + + poll_stats(2); + + return 0; + +err_out: + return 1; +} -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv12 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL 2020-12-16 14:30 ` [PATCHv12 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (3 preceding siblings ...) 2020-12-16 14:30 ` [PATCHv12 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu @ 2020-12-16 14:30 ` Hangbin Liu 2020-12-16 14:30 ` [PATCHv12 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2020-12-21 12:34 ` [PATCHv13 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-12-16 14:30 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu Use helper bpf_redirect_map() and bpf_redirect_map_multi() to test bpf arg ARG_CONST_MAP_PTR and ARG_CONST_MAP_PTR_OR_NULL. Make sure the map arg could be verified correctly when it is NULL or valid map pointer. Add devmap and devmap_hash in struct bpf_test due to bpf_redirect_{map, map_multi} limit. Test result: ]# ./test_verifier 713 716 #713/p ARG_CONST_MAP_PTR: null pointer OK #714/p ARG_CONST_MAP_PTR: valid map pointer OK #715/p ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map OK #716/p ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map OK Summary: 4 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- tools/testing/selftests/bpf/test_verifier.c | 22 +++++- .../testing/selftests/bpf/verifier/map_ptr.c | 70 +++++++++++++++++++ 2 files changed, 91 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c index 777a81404fdb..17eb3958ce6d 100644 --- a/tools/testing/selftests/bpf/test_verifier.c +++ b/tools/testing/selftests/bpf/test_verifier.c @@ -50,7 +50,7 @@ #define MAX_INSNS BPF_MAXINSNS #define MAX_TEST_INSNS 1000000 #define MAX_FIXUPS 8 -#define MAX_NR_MAPS 20 +#define MAX_NR_MAPS 22 #define MAX_TEST_RUNS 8 #define POINTER_VALUE 0xcafe4all #define TEST_DATA_LEN 64 @@ -87,6 +87,8 @@ struct bpf_test { int fixup_sk_storage_map[MAX_FIXUPS]; int fixup_map_event_output[MAX_FIXUPS]; int fixup_map_reuseport_array[MAX_FIXUPS]; + int fixup_map_devmap[MAX_FIXUPS]; + int fixup_map_devmap_hash[MAX_FIXUPS]; const char *errstr; const char *errstr_unpriv; uint32_t insn_processed; @@ -640,6 +642,8 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, int *fixup_sk_storage_map = test->fixup_sk_storage_map; int *fixup_map_event_output = test->fixup_map_event_output; int *fixup_map_reuseport_array = test->fixup_map_reuseport_array; + int *fixup_map_devmap = test->fixup_map_devmap; + int *fixup_map_devmap_hash = test->fixup_map_devmap_hash; if (test->fill_helper) { test->fill_insns = calloc(MAX_TEST_INSNS, sizeof(struct bpf_insn)); @@ -817,6 +821,22 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, fixup_map_reuseport_array++; } while (*fixup_map_reuseport_array); } + if (*fixup_map_devmap) { + map_fds[20] = __create_map(BPF_MAP_TYPE_DEVMAP, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap].imm = map_fds[20]; + fixup_map_devmap++; + } while (*fixup_map_devmap); + } + if (*fixup_map_devmap_hash) { + map_fds[21] = __create_map(BPF_MAP_TYPE_DEVMAP_HASH, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap_hash].imm = map_fds[21]; + fixup_map_devmap_hash++; + } while (*fixup_map_devmap_hash); + } } struct libcap { diff --git a/tools/testing/selftests/bpf/verifier/map_ptr.c b/tools/testing/selftests/bpf/verifier/map_ptr.c index b117bdd3806d..1a532198c9c1 100644 --- a/tools/testing/selftests/bpf/verifier/map_ptr.c +++ b/tools/testing/selftests/bpf/verifier/map_ptr.c @@ -93,3 +93,73 @@ .fixup_map_hash_16b = { 4 }, .result = ACCEPT, }, +{ + "ARG_CONST_MAP_PTR: null pointer", + .insns = { + /* bpf_redirect_map arg1 (map) */ + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .result = REJECT, + .prog_type = BPF_PROG_TYPE_XDP, + .errstr = "R1 type=inv expected=map_ptr", +}, +{ + "ARG_CONST_MAP_PTR: valid map pointer", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg1 (map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_LD_MAP_FD(BPF_REG_2, 1), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .fixup_map_devmap_hash = { 3 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv12 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test 2020-12-16 14:30 ` [PATCHv12 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (4 preceding siblings ...) 2020-12-16 14:30 ` [PATCHv12 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2020-12-16 14:30 ` Hangbin Liu 2020-12-21 12:34 ` [PATCHv13 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-12-16 14:30 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu Add a bpf selftest for new helper xdp_redirect_map_multi(). In this test we have 3 forward groups and 1 exclude group. The test will redirect each interface's packets to all the interfaces in the forward group, and exclude the interface in exclude map. We will also test both DEVMAP and DEVMAP_HASH with xdp generic and drv. For more test details, you can find it in the test script. Here is the test result. ]# ./test_xdp_redirect_multi.sh Pass: xdpgeneric arp ns1-2 Pass: xdpgeneric arp ns1-3 Pass: xdpgeneric arp ns1-4 Pass: xdpgeneric ping ns1-2 Pass: xdpgeneric ping ns1-3 Pass: xdpgeneric ping ns1-4 Pass: xdpgeneric ping6 ns2-1 Pass: xdpgeneric ping6 ns2-3 Pass: xdpgeneric ping6 ns2-4 Pass: xdpdrv arp ns1-2 Pass: xdpdrv arp ns1-3 Pass: xdpdrv arp ns1-4 Pass: xdpdrv ping ns1-2 Pass: xdpdrv ping ns1-3 Pass: xdpdrv ping ns1-4 Pass: xdpdrv ping6 ns2-1 Pass: xdpdrv ping6 ns2-3 Pass: xdpdrv ping6 ns2-4 Pass: xdpegress mac ns1-2 Pass: xdpegress mac ns1-3 Pass: xdpegress mac ns1-4 Pass: xdpegress ping ns1-2 Pass: xdpegress ping ns1-3 Pass: xdpegress ping ns1-4 Summary: PASS 24, FAIL 0 Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v12: add devmap prog test on egress v9: use NULL directly for arg2 and redefine the maps with btf format --- tools/testing/selftests/bpf/Makefile | 3 +- .../bpf/progs/xdp_redirect_multi_kern.c | 120 ++++++++ .../selftests/bpf/test_xdp_redirect_multi.sh | 208 ++++++++++++++ .../selftests/bpf/xdp_redirect_multi.c | 258 ++++++++++++++++++ 4 files changed, 588 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 8c33e999319a..20b9481c679f 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -51,6 +51,7 @@ TEST_FILES = test_lwt_ip_encap.o \ # Order correspond to 'make run_tests' order TEST_PROGS := test_kmod.sh \ test_xdp_redirect.sh \ + test_xdp_redirect_multi.sh \ test_xdp_meta.sh \ test_xdp_veth.sh \ test_offload.py \ @@ -80,7 +81,7 @@ TEST_PROGS_EXTENDED := with_addr.sh \ TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ test_lirc_mode2_user xdping test_cpp runqslower bench bpf_testmod.ko \ - xdpxceiver + xdpxceiver xdp_redirect_multi TEST_CUSTOM_PROGS = urandom_read diff --git a/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c new file mode 100644 index 000000000000..a9785b28175a --- /dev/null +++ b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c @@ -0,0 +1,120 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <string.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ip.h> +#include <linux/ipv6.h> + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_endian.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v4 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v6 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_all SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(struct bpf_devmap_val)); + __uint(max_entries, 128); +} forward_map_egress SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} exclude_map SEC(".maps"); + +/* map to stroe egress interfaces mac addresses */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, __u32); + __type(value, __be64); + __uint(max_entries, 128); +} mac_map SEC(".maps"); + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + struct ethhdr *eth = data; + __u16 h_proto; + __u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + h_proto = eth->h_proto; + + if (h_proto == bpf_htons(ETH_P_IP)) + return bpf_redirect_map_multi(&forward_map_v4, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else if (h_proto == bpf_htons(ETH_P_IPV6)) + return bpf_redirect_map_multi(&forward_map_v6, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else + return bpf_redirect_map_multi(&forward_map_all, NULL, + BPF_F_EXCLUDE_INGRESS); +} + +/* The following 2 progs are for 2nd devmap prog testing */ +SEC("xdp_redirect_map_ingress") +int xdp_redirect_map_all_prog(struct xdp_md *ctx) +{ + return bpf_redirect_map_multi(&forward_map_egress, NULL, BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_devmap/map_prog") +int xdp_devmap_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + __u32 key = ctx->egress_ifindex; + struct ethhdr *eth = data; + __u64 nh_off; + __be64 *mac; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + mac = bpf_map_lookup_elem(&mac_map, &key); + if (mac) + __builtin_memcpy(eth->h_source, mac, ETH_ALEN); + + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh new file mode 100755 index 000000000000..6503751fdca5 --- /dev/null +++ b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh @@ -0,0 +1,208 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Test topology: +# - - - - - - - - - - - - - - - - - - - - - - - - - +# | veth1 veth2 veth3 veth4 | ... init net +# - -| - - - - - - | - - - - - - | - - - - - - | - - +# --------- --------- --------- --------- +# | veth0 | | veth0 | | veth0 | | veth0 | ... +# --------- --------- --------- --------- +# ns1 ns2 ns3 ns4 +# +# Forward maps: +# Forward map_all has interfaces: veth1, veth2, veth3, veth4, ... (All traffic except IPv4, IPv6) +# Forward map_v4 has interfaces: veth1, veth3, veth4, ... (For IPv4 traffic only) +# Forward map_v6 has interfaces: veth2, veth3, veth4, ... (For IPv6 traffic only) +# Forward map_egress has all interfaces and redirect all pkts +# Exclude Groups: +# Exclude map: veth3 (assume ns3 is in black list) +# Map type: +# map_v4 use DEVMAP, others use DEVMAP_HASH +# +# Test modules: +# XDP modes: generic, native, native + egress_prog +# +# Test cases: +# ARP(we didn't block ARP for ns3): +# ns1 -> gw: ns2, ns3, ns4 should receive the arp request +# IPv4: +# ns1 -> ns2 (block), ns1 -> ns3 (block), ns1 -> ns4 (pass) +# IPv6 +# ns2 -> ns1 (block), ns2 -> ns3 (block), ns2 -> ns4 (pass) +# egress_prog: +# all ping test should pass, the src mac should be egress interface's mac +# + + +# netns numbers +NUM=4 +IFACES="" +DRV_MODE="xdpgeneric xdpdrv xdpegress" +PASS=0 +FAIL=0 + +test_pass() +{ + echo "Pass: $@" + PASS=$((PASS + 1)) +} + +test_fail() +{ + echo "fail: $@" + FAIL=$((FAIL + 1)) +} + +clean_up() +{ + for i in $(seq $NUM); do + ip link del veth$i 2> /dev/null + ip netns del ns$i 2> /dev/null + done +} + +# Kselftest framework requirement - SKIP code is 4. +check_env() +{ + ip link set dev lo xdpgeneric off &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without the ip xdpgeneric support" + exit 4 + fi + + which tcpdump &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without tcpdump" + exit 4 + fi +} + +setup_ns() +{ + local mode=$1 + IFACES="" + + if [ "$mode" = "xdpegress" ]; then + mode="xdpdrv" + fi + + for i in $(seq $NUM); do + ip netns add ns$i + ip link add veth$i type veth peer name veth0 netns ns$i + ip link set veth$i up + ip -n ns$i link set veth0 up + + ip -n ns$i addr add 192.0.2.$i/24 dev veth0 + ip -n ns$i addr add 2001:db8::$i/64 dev veth0 + ip -n ns$i link set veth0 $mode obj \ + xdp_dummy.o sec xdp_dummy &> /dev/null || \ + { test_fail "Unable to load dummy xdp" && exit 1; } + IFACES="$IFACES veth$i" + veth_mac[$i]=$(ip link show veth$i | awk '/link\/ether/ {print $2}') + done +} + +do_egress_tests() +{ + local mode=$1 + + # mac test + ip netns exec ns2 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + + # mac check + grep -q "${veth_mac[2]} > ff:ff:ff:ff:ff:ff" mac_ns1-2_${mode}.log && \ + test_pass "$mode mac ns1-2" || test_fail "$mode mac ns1-2" + grep -q "${veth_mac[3]} > ff:ff:ff:ff:ff:ff" mac_ns1-3_${mode}.log && \ + test_pass "$mode mac ns1-3" || test_fail "$mode mac ns1-3" + grep -q "${veth_mac[4]} > ff:ff:ff:ff:ff:ff" mac_ns1-4_${mode}.log && \ + test_pass "$mode mac ns1-4" || test_fail "$mode mac ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-2" || test_fail "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-3" || test_fail "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" +} + +do_ping_tests() +{ + local mode=$1 + + # arp test + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -i veth0 -nn -l -e &> arp_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-2_${mode}.log && \ + test_pass "$mode arp ns1-2" || test_fail "$mode arp ns1-2" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-3_${mode}.log && \ + test_pass "$mode arp ns1-3" || test_fail "$mode arp ns1-3" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-4_${mode}.log && \ + test_pass "$mode arp ns1-4" || test_fail "$mode arp ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-2" || test_pass "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-3" || test_pass "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" + + # ping6 test + ip netns exec ns2 ping6 2001:db8::1 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-1" || test_pass "$mode ping6 ns2-1" + ip netns exec ns2 ping6 2001:db8::3 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-3" || test_pass "$mode ping6 ns2-3" + ip netns exec ns2 ping6 2001:db8::4 -c 4 &> /dev/null && \ + test_pass "$mode ping6 ns2-4" || test_fail "$mode ping6 ns2-4" +} + +do_tests() +{ + local mode=$1 + local drv_p + + case ${mode} in + xdpdrv) drv_p="-N";; + xdpegress) drv_p="-X";; + xdpgeneric) drv_p="-S";; + esac + + ./xdp_redirect_multi $drv_p $IFACES &> xdp_redirect_${mode}.log & + xdp_pid=$! + sleep 10 + + if [ "$mode" = "xdpegress" ]; then + do_egress_tests $mode + else + do_ping_tests $mode + fi + + kill $xdp_pid +} + +trap clean_up 0 2 3 6 9 + +check_env +rm -f xdp_redirect_*.log arp_ns*.log mac_ns*.log + +for mode in ${DRV_MODE}; do + setup_ns $mode + do_tests $mode + sleep 10 + clean_up + sleep 5 +done + +echo "Summary: PASS $PASS, FAIL $FAIL" +[ $FAIL -eq 0 ] && exit 0 || exit 1 diff --git a/tools/testing/selftests/bpf/xdp_redirect_multi.c b/tools/testing/selftests/bpf/xdp_redirect_multi.c new file mode 100644 index 000000000000..0a39dfe3246c --- /dev/null +++ b/tools/testing/selftests/bpf/xdp_redirect_multi.c @@ -0,0 +1,258 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> +#include <sys/resource.h> +#include <sys/ioctl.h> +#include <sys/types.h> +#include <sys/socket.h> +#include <netinet/in.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static int get_mac_addr(unsigned int ifindex, void *mac_addr) +{ + char ifname[IF_NAMESIZE]; + struct ifreq ifr; + int fd, ret = -1; + + fd = socket(AF_INET, SOCK_DGRAM, 0); + if (fd < 0) + return ret; + + if (!if_indextoname(ifindex, ifname)) + goto err_out; + + strcpy(ifr.ifr_name, ifname); + + if (ioctl(fd, SIOCGIFHWADDR, &ifr) != 0) + goto err_out; + + memcpy(mac_addr, ifr.ifr_hwaddr.sa_data, 6 * sizeof(char)); + ret = 0; + +err_out: + close(fd); + return ret; +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n" + " -X load xdp program on egress\n", + prog); +} + +int main(int argc, char **argv) +{ + int prog_fd, group_all, group_v4, group_v6, exclude, mac_map; + struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY}; + struct bpf_program *ingress_prog, *egress_prog; + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_UNSPEC, + }; + int i, ret, opt, egress_prog_fd = -1; + struct bpf_devmap_val devmap_val; + bool attach_egress_prog = false; + unsigned char mac_addr[6]; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + unsigned int ifindex; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNFX")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + case 'X': + attach_egress_prog = true; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) { + xdp_flags |= XDP_FLAGS_DRV_MODE; + } else if (attach_egress_prog) { + printf("Load xdp program on egress with SKB mode not supported yet\n"); + goto err_out; + } + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + goto err_out; + } + + if (setrlimit(RLIMIT_MEMLOCK, &r)) { + perror("setrlimit(RLIMIT_MEMLOCK)"); + goto err_out; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + goto err_out; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + goto err_out; + + if (attach_egress_prog) + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_egress"); + else + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_all"); + group_v4 = bpf_object__find_map_fd_by_name(obj, "forward_map_v4"); + group_v6 = bpf_object__find_map_fd_by_name(obj, "forward_map_v6"); + exclude = bpf_object__find_map_fd_by_name(obj, "exclude_map"); + mac_map = bpf_object__find_map_fd_by_name(obj, "mac_map"); + + if (group_all < 0 || group_v4 < 0 || group_v6 < 0 || exclude < 0 || + mac_map < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + goto err_out; + } + + if (attach_egress_prog) { + /* Find ingress/egress prog for 2nd xdp prog */ + ingress_prog = bpf_object__find_program_by_title(obj, "xdp_redirect_map_ingress"); + egress_prog = bpf_object__find_program_by_title(obj, "xdp_devmap/map_prog"); + if (!ingress_prog || !egress_prog) { + printf("finding ingress/egress_prog in obj file failed\n"); + goto err_out; + } + prog_fd = bpf_program__fd(ingress_prog); + egress_prog_fd = bpf_program__fd(egress_prog); + if (prog_fd < 0 || egress_prog_fd < 0) { + printf("find egress_prog fd failed\n"); + goto err_out; + } + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + if (attach_egress_prog) { + ret = get_mac_addr(ifindex, mac_addr); + if (ret < 0) { + printf("get interface %d mac failed\n", ifindex); + goto err_out; + } + ret = bpf_map_update_elem(mac_map, &ifindex, mac_addr, 0); + if (ret) { + perror("bpf_update_elem mac_map failed\n"); + goto err_out; + } + } + + /* Add all the interfaces to group all */ + devmap_val.ifindex = ifindex; + devmap_val.bpf_prog.fd = egress_prog_fd; + ret = bpf_map_update_elem(group_all, &ifindex, &devmap_val, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* For testing: remove the 1st interfaces from group v6 */ + if (i != 0) { + ret = bpf_map_update_elem(group_v6, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: remove the 2nd interfaces from group v4 */ + if (i != 1) { + ret = bpf_map_update_elem(group_v4, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: add the 3rd interfaces to exclude map */ + if (i == 2) { + ret = bpf_map_update_elem(exclude, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + } + + /* sleep some time for testing */ + sleep(999); + + return 0; + +err_out: + return 1; +} -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv13 bpf-next 0/6] xdp: add a new helper for dev map multicast support 2020-12-16 14:30 ` [PATCHv12 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (5 preceding siblings ...) 2020-12-16 14:30 ` [PATCHv12 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu @ 2020-12-21 12:34 ` Hangbin Liu 2020-12-21 12:35 ` [PATCHv13 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu ` (6 more replies) 6 siblings, 7 replies; 219+ messages in thread From: Hangbin Liu @ 2020-12-21 12:34 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because there may have multi interfaces you want to exclude. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. The 1st patch is Jesper's run devmap xdp_prog later in bulking step. The 2st patch add a new bpf arg to allow NULL map pointer. The 3rd patch add the new bpf_redirect_map_multi() helper. The 4-6 patches are for usage sample and testing purpose. I did same perf tests with the following topo: --------------------- --------------------- | Host A (i40e 10G) | ---------- | eno1(i40e 10G) | --------------------- | | | Host B | --------------------- | | | Host C (i40e 10G) | ---------- | eno2(i40e 10G) | --------------------- vlan2 | -------- | | veth1 -- | veth0| | | -------- | --------------------| On Host A: # pktgen/pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -s 64 On Host B(Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, 128G Memory): Use xdp_redirect_map and xdp_redirect_map_multi in samples/bpf for testing. The veth0 in netns load dummy drop program. The forward_map max_entries in xdp_redirect_map_multi is modify to 4. Here is the perf result with 5.10 rc6: The are about +/- 0.1M deviation for native testing Version | Test | Generic | Native | Native + 2nd 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M 5.10 rc6 + patch1 | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch1 | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M 5.10 rc6 + patch1-6 | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch1-6 | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->i40e | 1.7M | 7.8M | 6.4M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->veth | 1.4M | 9.3M | 7.5M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->i40e+veth | 1.0M | 3.2M | 2.7M Last but not least, thanks a lot to Toke, Jesper, Jiri and Eelco for suggestions and help on implementation. [0] https://xdp-project.net/#Handling-multicast v13: Pass in xdp_prog through __xdp_enqueue() for patch 01. Update related code in patch 03. v12: Add Jesper's xdp_prog patch, rebase my works on this and latest bpf-next Add 2nd xdp_prog test on the sample and selftests. v11: Fix bpf_redirect_map_multi() helper description typo. Add loop limit for devmap_get_next_obj() and dev_map_redirect_multi(). v10: Rebase the code to latest bpf-next. Update helper bpf_xdp_redirect_map_multi() - No need to check map pointer as we will do the check in verifier. v9: Update helper bpf_xdp_redirect_map_multi() - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: a) Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). f) Split the tests from sample and add a bpf kernel selftest patch. v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Hangbin Liu (5): bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL xdp: add a new helper for dev map multicast support sample/bpf: add xdp_redirect_map_multicast test selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL selftests/bpf: add xdp_redirect_multi test Jesper Dangaard Brouer (1): bpf: run devmap xdp_prog on flush instead of bulk enqueue include/linux/bpf.h | 21 ++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 27 ++ kernel/bpf/devmap.c | 235 +++++++++++--- kernel/bpf/verifier.c | 16 +- net/core/filter.c | 118 ++++++- net/core/xdp.c | 29 ++ samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 96 ++++++ samples/bpf/xdp_redirect_map_multi_user.c | 301 ++++++++++++++++++ tools/include/uapi/linux/bpf.h | 27 ++ tools/testing/selftests/bpf/Makefile | 3 +- .../bpf/progs/xdp_redirect_multi_kern.c | 120 +++++++ tools/testing/selftests/bpf/test_verifier.c | 22 +- .../selftests/bpf/test_xdp_redirect_multi.sh | 208 ++++++++++++ .../testing/selftests/bpf/verifier/map_ptr.c | 70 ++++ .../selftests/bpf/xdp_redirect_multi.c | 252 +++++++++++++++ 18 files changed, 1502 insertions(+), 48 deletions(-) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c -- 2.26.2 ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv13 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2020-12-21 12:34 ` [PATCHv13 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2020-12-21 12:35 ` Hangbin Liu 2020-12-21 12:35 ` [PATCHv13 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu ` (5 subsequent siblings) 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-12-21 12:35 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu From: Jesper Dangaard Brouer <brouer@redhat.com> This changes the devmap XDP program support to run the program when the bulk queue is flushed instead of before the frame is enqueued. This has a couple of benefits: - It "sorts" the packets by destination devmap entry, and then runs the same BPF program on all the packets in sequence. This ensures that we keep the XDP program and destination device properties hot in I-cache. - It makes the multicast implementation simpler because it can just enqueue packets using bq_enqueue() without having to deal with the devmap program at all. The drawback is that if the devmap program drops the packet, the enqueue step is redundant. However, arguably this is mostly visible in a micro-benchmark, and with more mixed traffic the I-cache benefit should win out. The performance impact of just this patch is as follows: Using xdp_redirect_map(with a 2nd xdp_prog patch[1]) in sample/bpf and send pkts via pktgen cmd: ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 There are about +/- 0.1M deviation for native testing, the performance improved for the base-case, but some drop back with xdp devmap prog attached. Version | Test | Generic | Native | Native + 2nd xdp_prog 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M 5.10 rc6 + patch | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M [1] https://patchwork.ozlabs.org/project/netdev/patch/20201208120159.2278277-1-liuhangbin@gmail.com/ Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v13: pass in xdp_prog through __xdp_enqueue() v2-v12: no this patch --- kernel/bpf/devmap.c | 115 +++++++++++++++++++++++++++----------------- 1 file changed, 72 insertions(+), 43 deletions(-) diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index f6e9c68afdd4..84fe15950e44 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -57,6 +57,7 @@ struct xdp_dev_bulk_queue { struct list_head flush_node; struct net_device *dev; struct net_device *dev_rx; + struct bpf_prog *xdp_prog; unsigned int count; }; @@ -327,40 +328,92 @@ bool dev_map_can_have_prog(struct bpf_map *map) return false; } +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, + struct xdp_frame **frames, int n, + struct net_device *dev) +{ + struct xdp_txq_info txq = { .dev = dev }; + struct xdp_buff xdp; + int i, nframes = 0; + + for (i = 0; i < n; i++) { + struct xdp_frame *xdpf = frames[i]; + u32 act; + int err; + + xdp_convert_frame_to_buff(xdpf, &xdp); + xdp.txq = &txq; + + act = bpf_prog_run_xdp(xdp_prog, &xdp); + switch (act) { + case XDP_PASS: + err = xdp_update_frame_from_buff(&xdp, xdpf); + if (unlikely(err < 0)) + xdp_return_frame_rx_napi(xdpf); + else + frames[nframes++] = xdpf; + break; + default: + bpf_warn_invalid_xdp_action(act); + fallthrough; + case XDP_ABORTED: + trace_xdp_exception(dev, xdp_prog, act); + fallthrough; + case XDP_DROP: + xdp_return_frame_rx_napi(xdpf); + break; + } + } + return n - nframes; /* dropped frames count */ +} + static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) { struct net_device *dev = bq->dev; int sent = 0, drops = 0, err = 0; + unsigned int cnt = bq->count; + unsigned int xdp_drop; int i; - if (unlikely(!bq->count)) + if (unlikely(!cnt)) return; - for (i = 0; i < bq->count; i++) { + for (i = 0; i < cnt; i++) { struct xdp_frame *xdpf = bq->q[i]; prefetch(xdpf); } - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); + if (unlikely(bq->xdp_prog)) { + xdp_drop = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); + cnt -= xdp_drop; + if (!cnt) { + sent = 0; + drops = xdp_drop; + goto out; + } + } + + sent = dev->netdev_ops->ndo_xdp_xmit(dev, cnt, bq->q, flags); if (sent < 0) { err = sent; sent = 0; goto error; } - drops = bq->count - sent; + drops = (cnt - sent) + xdp_drop; out: bq->count = 0; trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); bq->dev_rx = NULL; + bq->xdp_prog = NULL; __list_del_clearprev(&bq->flush_node); return; error: /* If ndo_xdp_xmit fails with an errno, no frames have been * xmit'ed and it's our responsibility to them free all. */ - for (i = 0; i < bq->count; i++) { + for (i = 0; i < cnt; i++) { struct xdp_frame *xdpf = bq->q[i]; xdp_return_frame_rx_napi(xdpf); @@ -408,7 +461,7 @@ struct bpf_dtab_netdev *__dev_map_lookup_elem(struct bpf_map *map, u32 key) * Thus, safe percpu variable access. */ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, - struct net_device *dev_rx) + struct net_device *dev_rx, struct bpf_prog *xdp_prog) { struct list_head *flush_list = this_cpu_ptr(&dev_flush_list); struct xdp_dev_bulk_queue *bq = this_cpu_ptr(dev->xdp_bulkq); @@ -423,6 +476,14 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, if (!bq->dev_rx) bq->dev_rx = dev_rx; + /* Store (potential) xdp_prog that run before egress to dev as + * part of bulk_queue. This will be same xdp_prog for all + * xdp_frame's in bulk_queue, because this per-CPU store must + * be flushed from net_device drivers NAPI func end. + */ + if (!bq->xdp_prog) + bq->xdp_prog = xdp_prog; + bq->q[bq->count++] = xdpf; if (!bq->flush_node.prev) @@ -430,7 +491,8 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, } static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, - struct net_device *dev_rx) + struct net_device *dev_rx, + struct bpf_prog *xdp_prog) { struct xdp_frame *xdpf; int err; @@ -446,42 +508,14 @@ static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, if (unlikely(!xdpf)) return -EOVERFLOW; - bq_enqueue(dev, xdpf, dev_rx); + bq_enqueue(dev, xdpf, dev_rx, xdp_prog); return 0; } -static struct xdp_buff *dev_map_run_prog(struct net_device *dev, - struct xdp_buff *xdp, - struct bpf_prog *xdp_prog) -{ - struct xdp_txq_info txq = { .dev = dev }; - u32 act; - - xdp_set_data_meta_invalid(xdp); - xdp->txq = &txq; - - act = bpf_prog_run_xdp(xdp_prog, xdp); - switch (act) { - case XDP_PASS: - return xdp; - case XDP_DROP: - break; - default: - bpf_warn_invalid_xdp_action(act); - fallthrough; - case XDP_ABORTED: - trace_xdp_exception(dev, xdp_prog, act); - break; - } - - xdp_return_buff(xdp); - return NULL; -} - int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx) { - return __xdp_enqueue(dev, xdp, dev_rx); + return __xdp_enqueue(dev, xdp, dev_rx, NULL); } int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, @@ -489,12 +523,7 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, { struct net_device *dev = dst->dev; - if (dst->xdp_prog) { - xdp = dev_map_run_prog(dev, xdp, dst->xdp_prog); - if (!xdp) - return 0; - } - return __xdp_enqueue(dev, xdp, dev_rx); + return __xdp_enqueue(dev, xdp, dev_rx, dst->xdp_prog); } int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv13 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL 2020-12-21 12:34 ` [PATCHv13 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-12-21 12:35 ` [PATCHv13 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu @ 2020-12-21 12:35 ` Hangbin Liu 2020-12-21 12:35 ` [PATCHv13 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (4 subsequent siblings) 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-12-21 12:35 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu Add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL which could be used when we want to allow NULL pointer for map parameter. The bpf helper need to take care and check if the map is NULL when use this type. [1] https://lore.kernel.org/bpf/20200715070001.2048207-1-liuhangbin@gmail.com/ Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v11-v12: rebase the patch to latest bpf-next v10: remove useless CONST_PTR_TO_MAP_OR_NULL and Copy-paste comment. v9: merge the patch from [1] in to this series. v1-v8: no this patch --- include/linux/bpf.h | 1 + kernel/bpf/verifier.c | 10 ++++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 07cb5d15e743..7850c87456fc 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -296,6 +296,7 @@ enum bpf_arg_type { ARG_CONST_ALLOC_SIZE_OR_ZERO, /* number of allocated bytes requested */ ARG_PTR_TO_BTF_ID_SOCK_COMMON, /* pointer to in-kernel sock_common or bpf-mirrored bpf_sock */ ARG_PTR_TO_PERCPU_BTF_ID, /* pointer to in-kernel percpu type */ + ARG_CONST_MAP_PTR_OR_NULL, /* const argument used as pointer to bpf_map or NULL */ __BPF_ARG_TYPE_MAX, }; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 17270b8404f1..9f6633c9ea12 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -445,7 +445,8 @@ static bool arg_type_may_be_null(enum bpf_arg_type type) type == ARG_PTR_TO_MEM_OR_NULL || type == ARG_PTR_TO_CTX_OR_NULL || type == ARG_PTR_TO_SOCKET_OR_NULL || - type == ARG_PTR_TO_ALLOC_MEM_OR_NULL; + type == ARG_PTR_TO_ALLOC_MEM_OR_NULL || + type == ARG_CONST_MAP_PTR_OR_NULL; } /* Determine whether the function releases some resources allocated by another @@ -4065,6 +4066,7 @@ static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = { [ARG_CONST_SIZE_OR_ZERO] = &scalar_types, [ARG_CONST_ALLOC_SIZE_OR_ZERO] = &scalar_types, [ARG_CONST_MAP_PTR] = &const_map_ptr_types, + [ARG_CONST_MAP_PTR_OR_NULL] = &const_map_ptr_types, [ARG_PTR_TO_CTX] = &context_types, [ARG_PTR_TO_CTX_OR_NULL] = &context_types, [ARG_PTR_TO_SOCK_COMMON] = &sock_types, @@ -4210,9 +4212,9 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, meta->ref_obj_id = reg->ref_obj_id; } - if (arg_type == ARG_CONST_MAP_PTR) { - /* bpf_map_xxx(map_ptr) call: remember that map_ptr */ - meta->map_ptr = reg->map_ptr; + if (arg_type == ARG_CONST_MAP_PTR || + arg_type == ARG_CONST_MAP_PTR_OR_NULL) { + meta->map_ptr = register_is_null(reg) ? NULL : reg->map_ptr; } else if (arg_type == ARG_PTR_TO_MAP_KEY) { /* bpf_map_xxx(..., map_ptr, ..., key) call: * check that [key, key + map->key_size) are within -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv13 bpf-next 3/6] xdp: add a new helper for dev map multicast support 2020-12-21 12:34 ` [PATCHv13 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-12-21 12:35 ` [PATCHv13 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu 2020-12-21 12:35 ` [PATCHv13 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2020-12-21 12:35 ` Hangbin Liu 2020-12-21 12:35 ` [PATCHv13 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu ` (3 subsequent siblings) 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-12-21 12:35 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because a physical interface can be part of a logical grouping, such as a bond device. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. The forwarding map could be DEVMAP or DEVMAP_HASH, but the exclude map *must* be DEVMAP_HASH to get better performace. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. As both bpf_xdp_redirect_map() and this new helpers are using struct bpf_redirect_info, I add a new ex_map and set tgt_value to NULL in the new helper to make a difference with bpf_xdp_redirect_map(). Also I keep the general data path in net/core/filter.c, the native data path in kernel/bpf/devmap.c so we can use direct calls to get better performace. [0] https://xdp-project.net/#Handling-multicast Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v13: pass xdp_prog through bq_enqueue v12: rebase the code based on Jespoer's devmap xdp_prog patch v11: Fix bpf_redirect_map_multi() helper description typo. Add loop limit for devmap_get_next_obj() and dev_map_redirect_multi(). v10: Update helper bpf_xdp_redirect_map_multi() - No need to check map pointer as we will do the check in verifier. v9: Update helper bpf_xdp_redirect_map_multi() - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. --- include/linux/bpf.h | 20 +++++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 27 +++++++ kernel/bpf/devmap.c | 132 +++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++ net/core/filter.c | 118 +++++++++++++++++++++++++++-- net/core/xdp.c | 29 ++++++++ tools/include/uapi/linux/bpf.h | 27 +++++++ 9 files changed, 356 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 7850c87456fc..2b4eb7b7f2e2 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1419,6 +1419,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx); int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, struct net_device *dev_rx); +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex); +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); bool dev_map_can_have_prog(struct bpf_map *map); @@ -1587,6 +1592,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return 0; } +static inline +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + return false; +} + +static inline +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + return 0; +} + struct sk_buff; static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, diff --git a/include/linux/filter.h b/include/linux/filter.h index 29c27656165b..55095e4759b4 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -620,6 +620,7 @@ struct bpf_redirect_info { u32 tgt_index; void *tgt_value; struct bpf_map *map; + struct bpf_map *ex_map; u32 kern_flags; struct bpf_nh_params nh; }; diff --git a/include/net/xdp.h b/include/net/xdp.h index 600acb307db6..2d90f641d9ac 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -145,6 +145,7 @@ void xdp_warn(const char *msg, const char *func, const int line); #define XDP_WARN(msg) xdp_warn(msg, __func__, __LINE__) struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); static inline void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 77d7c1bb2923..b22e79220df5 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3830,6 +3830,27 @@ union bpf_attr { * Return * A pointer to a struct socket on success or NULL if the file is * not a socket. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3995,6 +4016,7 @@ union bpf_attr { FN(ktime_get_coarse_ns), \ FN(ima_inode_hash), \ FN(sock_from_file), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -4171,6 +4193,11 @@ enum { BPF_F_BPRM_SECUREEXEC = (1ULL << 0), }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 84fe15950e44..098cff728f7f 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -526,6 +526,138 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return __xdp_enqueue(dev, xdp, dev_rx, dst->xdp_prog); } +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + return dev_map_get_next_key(map, key, next_key); + case BPF_MAP_TYPE_DEVMAP_HASH: + return dev_map_hash_get_next_key(map, key, next_key); + default: + break; + } + + return -ENOENT; +} + +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + if (obj->dev->ifindex == exclude_ifindex) + return true; + + if (!map) + return false; + + return __dev_map_hash_lookup_elem(map, obj->dev->ifindex) != NULL; +} + +static struct bpf_dtab_netdev *devmap_get_next_obj(struct xdp_buff *xdp, struct bpf_map *map, + struct bpf_map *ex_map, u32 *key, + u32 *next_key, int ex_ifindex) +{ + struct bpf_dtab_netdev *obj; + struct net_device *dev; + u32 *tmp_key = key; + u32 index; + int err; + + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + return NULL; + + /* When using dev map hash, we could restart the hashtab traversal + * in case the key has been updated/removed in the mean time. + * So we may end up potentially looping due to traversal restarts + * from first elem. + * + * Let's use map's max_entries to limit the loop number. + */ + for (index = 0; index < map->max_entries; index++) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + obj = __dev_map_lookup_elem(map, *next_key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + obj = __dev_map_hash_lookup_elem(map, *next_key); + break; + default: + break; + } + + if (!obj || dev_in_exclude_map(obj, ex_map, ex_ifindex)) + goto find_next; + + dev = obj->dev; + + if (!dev->netdev_ops->ndo_xdp_xmit) + goto find_next; + + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + if (unlikely(err)) + goto find_next; + + return obj; + +find_next: + tmp_key = next_key; + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + break; + } + + return NULL; +} + +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + struct bpf_dtab_netdev *obj = NULL, *next_obj = NULL; + struct xdp_frame *xdpf, *nxdpf; + bool last_one = false; + int ex_ifindex; + u32 key, next_key; + + ex_ifindex = flags & BPF_F_EXCLUDE_INGRESS ? dev_rx->ifindex : 0; + + /* Find first available obj */ + obj = devmap_get_next_obj(xdp, map, ex_map, NULL, &key, ex_ifindex); + if (!obj) + return 0; + + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) + return -EOVERFLOW; + + for (;;) { + /* Check if we still have one more available obj */ + next_obj = devmap_get_next_obj(xdp, map, ex_map, &key, + &next_key, ex_ifindex); + if (!next_obj) + last_one = true; + + if (last_one) { + bq_enqueue(obj->dev, xdpf, dev_rx, obj->xdp_prog); + return 0; + } + + nxdpf = xdpf_clone(xdpf); + if (unlikely(!nxdpf)) { + xdp_return_frame_rx_napi(xdpf); + return -ENOMEM; + } + + bq_enqueue(obj->dev, nxdpf, dev_rx, obj->xdp_prog); + + /* Deal with next obj */ + obj = next_obj; + key = next_key; + } +} + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9f6633c9ea12..fd297b929edd 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4420,6 +4420,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: if (func_id != BPF_FUNC_redirect_map && + func_id != BPF_FUNC_redirect_map_multi && func_id != BPF_FUNC_map_lookup_elem) goto error; break; @@ -4524,6 +4525,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, map->map_type != BPF_MAP_TYPE_XSKMAP) goto error; break; + case BPF_FUNC_redirect_map_multi: + if (map->map_type != BPF_MAP_TYPE_DEVMAP && + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + goto error; + break; case BPF_FUNC_sk_redirect_map: case BPF_FUNC_msg_redirect_map: case BPF_FUNC_sock_map_update: diff --git a/net/core/filter.c b/net/core/filter.c index 255aeee72402..a5d97cb1054b 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3924,12 +3924,19 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { }; static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) + struct bpf_map *map, struct xdp_buff *xdp, + struct bpf_map *ex_map, u32 flags) { switch (map->map_type) { case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: - return dev_map_enqueue(fwd, xdp, dev_rx); + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) + return dev_map_enqueue(fwd, xdp, dev_rx); + else + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, flags); case BPF_MAP_TYPE_CPUMAP: return cpu_map_enqueue(fwd, xdp, dev_rx); case BPF_MAP_TYPE_XSKMAP: @@ -3986,12 +3993,14 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); struct bpf_map *map = READ_ONCE(ri->map); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (unlikely(!map)) { @@ -4003,7 +4012,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = dev_xdp_enqueue(fwd, xdp, dev); } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, ri->flags); } if (unlikely(err)) @@ -4017,6 +4026,62 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, } EXPORT_SYMBOL_GPL(xdp_do_redirect); +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, + struct bpf_prog *xdp_prog, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) + +{ + struct bpf_dtab_netdev *dst; + struct sk_buff *nskb; + bool exclude_ingress; + u32 key, next_key, index; + void *fwd; + int err; + + /* Get first key from forward map */ + err = map->ops->map_get_next_key(map, NULL, &key); + if (err) + return err; + + exclude_ingress = !!(flags & BPF_F_EXCLUDE_INGRESS); + + /* When using dev map hash, we could restart the hashtab traversal + * in case the key has been updated/removed in the mean time. + * So we may end up potentially looping due to traversal restarts + * from first elem. + * + * Let's use map's max_entries to limit the loop number. + */ + for (index = 0; index < map->max_entries; index++) { + fwd = __xdp_map_lookup_elem(map, key); + if (fwd) { + dst = (struct bpf_dtab_netdev *)fwd; + if (dev_in_exclude_map(dst, ex_map, + exclude_ingress ? dev->ifindex : 0)) + goto find_next; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -ENOMEM; + + /* Try forword next one no mater the current forward + * succeed or not */ + dev_map_generic_redirect(dst, nskb, xdp_prog); + } + +find_next: + err = map->ops->map_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + consume_skb(skb); + return 0; +} + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, @@ -4024,19 +4089,30 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, struct bpf_map *map) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err = 0; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (map->map_type == BPF_MAP_TYPE_DEVMAP || map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { - struct bpf_dtab_netdev *dst = fwd; + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) { + struct bpf_dtab_netdev *dst = fwd; + + err = dev_map_generic_redirect(dst, skb, xdp_prog); + } else { + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, + ex_map, ri->flags); + } - err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { @@ -4150,6 +4226,36 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { .arg3_type = ARG_ANYTHING, }; +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, + struct bpf_map *, ex_map, u64, flags) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + + /* Limit ex_map type to DEVMAP_HASH to get better performance */ + if (unlikely((ex_map && ex_map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) || + flags & ~BPF_F_EXCLUDE_INGRESS)) + return XDP_ABORTED; + + ri->tgt_index = 0; + /* Set the tgt_value to NULL to distinguish with bpf_xdp_redirect_map */ + ri->tgt_value = NULL; + ri->flags = flags; + ri->ex_map = ex_map; + + WRITE_ONCE(ri->map, map); + + return XDP_REDIRECT; +} + +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { + .func = bpf_xdp_redirect_map_multi, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_CONST_MAP_PTR, + .arg2_type = ARG_CONST_MAP_PTR_OR_NULL, + .arg3_type = ARG_ANYTHING, +}; + static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, unsigned long off, unsigned long len) { @@ -7227,6 +7333,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_redirect_proto; case BPF_FUNC_redirect_map: return &bpf_xdp_redirect_map_proto; + case BPF_FUNC_redirect_map_multi: + return &bpf_xdp_redirect_map_multi_proto; case BPF_FUNC_xdp_adjust_tail: return &bpf_xdp_adjust_tail_proto; case BPF_FUNC_fib_lookup: diff --git a/net/core/xdp.c b/net/core/xdp.c index 3a8c9ab4ecbe..6d86af029dc5 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -513,3 +513,32 @@ void xdp_warn(const char *msg, const char *func, const int line) WARN(1, "XDP_WARN: %s(line:%d): %s\n", func, line, msg); }; EXPORT_SYMBOL_GPL(xdp_warn); + +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) +{ + unsigned int headroom, totalsize; + struct xdp_frame *nxdpf; + struct page *page; + void *addr; + + headroom = xdpf->headroom + sizeof(*xdpf); + totalsize = headroom + xdpf->len; + + if (unlikely(totalsize > PAGE_SIZE)) + return NULL; + page = dev_alloc_page(); + if (!page) + return NULL; + addr = page_to_virt(page); + + memcpy(addr, xdpf, totalsize); + + nxdpf = addr; + nxdpf->data = addr + headroom; + nxdpf->frame_sz = PAGE_SIZE; + nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0; + nxdpf->mem.id = 0; + + return nxdpf; +} +EXPORT_SYMBOL_GPL(xdpf_clone); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 77d7c1bb2923..b22e79220df5 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3830,6 +3830,27 @@ union bpf_attr { * Return * A pointer to a struct socket on success or NULL if the file is * not a socket. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3995,6 +4016,7 @@ union bpf_attr { FN(ktime_get_coarse_ns), \ FN(ima_inode_hash), \ FN(sock_from_file), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -4171,6 +4193,11 @@ enum { BPF_F_BPRM_SECUREEXEC = (1ULL << 0), }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv13 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test 2020-12-21 12:34 ` [PATCHv13 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (2 preceding siblings ...) 2020-12-21 12:35 ` [PATCHv13 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2020-12-21 12:35 ` Hangbin Liu 2020-12-21 12:35 ` [PATCHv13 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu ` (2 subsequent siblings) 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-12-21 12:35 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu This is a sample for xdp multicast. In the sample we could forward all packets between given interfaces. There is also an option -X that could enable 2nd xdp_prog on egress interface. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v12: add devmap xdp_prog on egress support v10-v11: no update v9: use NULL directly for arg2 and redefine the maps with btf format v5: add a null_map as we have strict the arg2 to ARG_CONST_MAP_PTR. Move the testing part to bpf selftest in next patch. v4: no update. v3: add rxcnt map to show the packet transmit speed. v2: no update. --- samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 96 +++++++ samples/bpf/xdp_redirect_map_multi_user.c | 301 ++++++++++++++++++++++ 3 files changed, 400 insertions(+) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 26fc96ca619e..200029fcf53c 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map tprogs-y += per_socket_stats_example tprogs-y += xdp_redirect tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_map_multi tprogs-y += xdp_redirect_cpu tprogs-y += xdp_monitor tprogs-y += xdp_rxq_info @@ -99,6 +100,7 @@ test_map_in_map-objs := test_map_in_map_user.o per_socket_stats_example-objs := cookie_uid_helper_example.o xdp_redirect-objs := xdp_redirect_user.o xdp_redirect_map-objs := xdp_redirect_map_user.o +xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o xdp_redirect_cpu-objs := xdp_redirect_cpu_user.o xdp_monitor-objs := xdp_monitor_user.o xdp_rxq_info-objs := xdp_rxq_info_user.o @@ -160,6 +162,7 @@ always-y += tcp_tos_reflect_kern.o always-y += tcp_dumpstats_kern.o always-y += xdp_redirect_kern.o always-y += xdp_redirect_map_kern.o +always-y += xdp_redirect_map_multi_kern.o always-y += xdp_redirect_cpu_kern.o always-y += xdp_monitor_kern.o always-y += xdp_rxq_info_kern.o diff --git a/samples/bpf/xdp_redirect_map_multi_kern.c b/samples/bpf/xdp_redirect_map_multi_kern.c new file mode 100644 index 000000000000..0c63aace9bd2 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_kern.c @@ -0,0 +1,96 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <uapi/linux/bpf.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/ip.h> +#include <linux/ipv6.h> +#include <bpf/bpf_helpers.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 32); +} forward_map_general SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(struct bpf_devmap_val)); + __uint(max_entries, 32); +} forward_map_native SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __type(key, u32); + __type(value, long); + __uint(max_entries, 1); +} rxcnt SEC(".maps"); + +/* map to stroe egress interfaces mac addresses, set the + * max_entries to 1 and extend it in user sapce prog. + */ +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, u32); + __type(value, __be64); + __uint(max_entries, 1); +} mac_map SEC(".maps"); + +static int xdp_redirect_map(struct xdp_md *ctx, void *forward_map) +{ + long *value; + u32 key = 0; + + /* count packet in global counter */ + value = bpf_map_lookup_elem(&rxcnt, &key); + if (value) + *value += 1; + + return bpf_redirect_map_multi(forward_map, NULL, BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_redirect_general") +int xdp_redirect_map_general(struct xdp_md *ctx) +{ + return xdp_redirect_map(ctx, &forward_map_general); +} + +SEC("xdp_redirect_native") +int xdp_redirect_map_native(struct xdp_md *ctx) +{ + return xdp_redirect_map(ctx, &forward_map_native); +} + +SEC("xdp_devmap/map_prog") +int xdp_devmap_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + u32 key = ctx->egress_ifindex; + struct ethhdr *eth = data; + __be64 *mac; + u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + mac = bpf_map_lookup_elem(&mac_map, &key); + if (mac) + __builtin_memcpy(eth->h_source, mac, ETH_ALEN); + + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/samples/bpf/xdp_redirect_map_multi_user.c b/samples/bpf/xdp_redirect_map_multi_user.c new file mode 100644 index 000000000000..67ffc294567c --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_user.c @@ -0,0 +1,301 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> +#include <sys/resource.h> +#include <sys/ioctl.h> +#include <sys/types.h> +#include <sys/socket.h> +#include <netinet/in.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; +static int rxcnt_map_fd; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void poll_stats(int interval) +{ + unsigned int nr_cpus = bpf_num_possible_cpus(); + __u64 values[nr_cpus], prev[nr_cpus]; + + memset(prev, 0, sizeof(prev)); + + while (1) { + __u64 sum = 0; + __u32 key = 0; + int i; + + sleep(interval); + assert(bpf_map_lookup_elem(rxcnt_map_fd, &key, values) == 0); + for (i = 0; i < nr_cpus; i++) + sum += (values[i] - prev[i]); + if (sum) + printf("Forwarding %10llu pkt/s\n", sum / interval); + memcpy(prev, values, sizeof(values)); + } +} + +static int get_mac_addr(unsigned int ifindex, void *mac_addr) +{ + char ifname[IF_NAMESIZE]; + struct ifreq ifr; + int fd, ret = -1; + + fd = socket(AF_INET, SOCK_DGRAM, 0); + if (fd < 0) + return ret; + + if (!if_indextoname(ifindex, ifname)) + goto err_out; + + strcpy(ifr.ifr_name, ifname); + + if (ioctl(fd, SIOCGIFHWADDR, &ifr) != 0) + goto err_out; + + memcpy(mac_addr, ifr.ifr_hwaddr.sa_data, 6 * sizeof(char)); + ret = 0; + +err_out: + close(fd); + return ret; +} + +static int update_mac_map(struct bpf_object *obj) +{ + int i, ret = -1, mac_map_fd; + unsigned char mac_addr[6]; + unsigned int ifindex; + + mac_map_fd = bpf_object__find_map_fd_by_name(obj, "mac_map"); + if (mac_map_fd < 0) { + printf("find mac map fd failed\n"); + return ret; + } + + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + ret = get_mac_addr(ifindex, mac_addr); + if (ret < 0) { + printf("get interface %d mac failed\n", ifindex); + return ret; + } + + ret = bpf_map_update_elem(mac_map_fd, &ifindex, mac_addr, 0); + if (ret) { + perror("bpf_update_elem mac_map_fd"); + return ret; + } + } + + return 0; +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n" + " -X load xdp program on egress\n", + prog); +} + +int main(int argc, char **argv) +{ + int i, ret, opt, forward_map_fd, max_ifindex = 0; + struct bpf_program *ingress_prog, *egress_prog; + int ingress_prog_fd, egress_prog_fd = -1; + struct bpf_devmap_val devmap_val; + bool attach_egress_prog = false; + char ifname[IF_NAMESIZE]; + struct bpf_map *mac_map; + struct bpf_object *obj; + unsigned int ifindex; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNFX")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + case 'X': + attach_egress_prog = true; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) { + xdp_flags |= XDP_FLAGS_DRV_MODE; + } else if (attach_egress_prog) { + printf("Load xdp program on egress with SKB mode not supported yet\n"); + return 1; + } + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + + /* Find the largest index number */ + if (ifaces[i] > max_ifindex) + max_ifindex = ifaces[i]; + + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + + obj = bpf_object__open(filename); + if (libbpf_get_error(obj)) { + printf("ERROR: opening BPF object file failed\n"); + obj = NULL; + goto err_out; + } + + /* Reset the map size to max ifindex + 1 */ + if (attach_egress_prog) { + mac_map = bpf_object__find_map_by_name(obj, "mac_map"); + ret = bpf_map__resize(mac_map, max_ifindex + 1); + if (ret < 0) { + printf("ERROR: reset mac map size failed\n"); + goto err_out; + } + } + + /* load BPF program */ + if (bpf_object__load(obj)) { + printf("ERROR: loading BPF object file failed\n"); + goto err_out; + } + + if (xdp_flags & XDP_FLAGS_SKB_MODE) { + ingress_prog = bpf_object__find_program_by_title(obj, "xdp_redirect_general"); + forward_map_fd = bpf_object__find_map_fd_by_name(obj, "forward_map_general"); + } else { + ingress_prog = bpf_object__find_program_by_title(obj, "xdp_redirect_native"); + forward_map_fd = bpf_object__find_map_fd_by_name(obj, "forward_map_native"); + } + if (!ingress_prog || forward_map_fd < 0) { + printf("finding ingress_prog/forward_map in obj file failed\n"); + goto err_out; + } + + ingress_prog_fd = bpf_program__fd(ingress_prog); + if (ingress_prog_fd < 0) { + printf("find ingress_prog fd failed\n"); + goto err_out; + } + + rxcnt_map_fd = bpf_object__find_map_fd_by_name(obj, "rxcnt"); + if (rxcnt_map_fd < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + goto err_out; + } + + if (attach_egress_prog) { + /* Update mac_map with all egress interfaces' mac addr */ + if (update_mac_map(obj) < 0) { + printf("Error: update mac map failed"); + goto err_out; + } + + /* Find egress prog fd */ + egress_prog = bpf_object__find_program_by_title(obj, "xdp_devmap/map_prog"); + if (!egress_prog) { + printf("finding egress_prog in obj file failed\n"); + goto err_out; + } + egress_prog_fd = bpf_program__fd(egress_prog); + if (egress_prog_fd < 0) { + printf("find egress_prog fd failed\n"); + goto err_out; + } + } + + /* Remove attached program when program is interrupted or killed */ + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, ingress_prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + /* Add all the interfaces to forward group and attach + * egress devmap programe if exist */ + devmap_val.ifindex = ifindex; + devmap_val.bpf_prog.fd = egress_prog_fd; + ret = bpf_map_update_elem(forward_map_fd, &ifindex, &devmap_val, 0); + if (ret) { + perror("bpf_map_update_elem forward_map"); + goto err_out; + } + } + + poll_stats(2); + + return 0; + +err_out: + return 1; +} -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv13 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL 2020-12-21 12:34 ` [PATCHv13 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (3 preceding siblings ...) 2020-12-21 12:35 ` [PATCHv13 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu @ 2020-12-21 12:35 ` Hangbin Liu 2020-12-21 12:35 ` [PATCHv13 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2021-01-14 14:23 ` [PATCHv14 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-12-21 12:35 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu Use helper bpf_redirect_map() and bpf_redirect_map_multi() to test bpf arg ARG_CONST_MAP_PTR and ARG_CONST_MAP_PTR_OR_NULL. Make sure the map arg could be verified correctly when it is NULL or valid map pointer. Add devmap and devmap_hash in struct bpf_test due to bpf_redirect_{map, map_multi} limit. Test result: ]# ./test_verifier 713 716 #713/p ARG_CONST_MAP_PTR: null pointer OK #714/p ARG_CONST_MAP_PTR: valid map pointer OK #715/p ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map OK #716/p ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map OK Summary: 4 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- tools/testing/selftests/bpf/test_verifier.c | 22 +++++- .../testing/selftests/bpf/verifier/map_ptr.c | 70 +++++++++++++++++++ 2 files changed, 91 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c index 777a81404fdb..17eb3958ce6d 100644 --- a/tools/testing/selftests/bpf/test_verifier.c +++ b/tools/testing/selftests/bpf/test_verifier.c @@ -50,7 +50,7 @@ #define MAX_INSNS BPF_MAXINSNS #define MAX_TEST_INSNS 1000000 #define MAX_FIXUPS 8 -#define MAX_NR_MAPS 20 +#define MAX_NR_MAPS 22 #define MAX_TEST_RUNS 8 #define POINTER_VALUE 0xcafe4all #define TEST_DATA_LEN 64 @@ -87,6 +87,8 @@ struct bpf_test { int fixup_sk_storage_map[MAX_FIXUPS]; int fixup_map_event_output[MAX_FIXUPS]; int fixup_map_reuseport_array[MAX_FIXUPS]; + int fixup_map_devmap[MAX_FIXUPS]; + int fixup_map_devmap_hash[MAX_FIXUPS]; const char *errstr; const char *errstr_unpriv; uint32_t insn_processed; @@ -640,6 +642,8 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, int *fixup_sk_storage_map = test->fixup_sk_storage_map; int *fixup_map_event_output = test->fixup_map_event_output; int *fixup_map_reuseport_array = test->fixup_map_reuseport_array; + int *fixup_map_devmap = test->fixup_map_devmap; + int *fixup_map_devmap_hash = test->fixup_map_devmap_hash; if (test->fill_helper) { test->fill_insns = calloc(MAX_TEST_INSNS, sizeof(struct bpf_insn)); @@ -817,6 +821,22 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, fixup_map_reuseport_array++; } while (*fixup_map_reuseport_array); } + if (*fixup_map_devmap) { + map_fds[20] = __create_map(BPF_MAP_TYPE_DEVMAP, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap].imm = map_fds[20]; + fixup_map_devmap++; + } while (*fixup_map_devmap); + } + if (*fixup_map_devmap_hash) { + map_fds[21] = __create_map(BPF_MAP_TYPE_DEVMAP_HASH, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap_hash].imm = map_fds[21]; + fixup_map_devmap_hash++; + } while (*fixup_map_devmap_hash); + } } struct libcap { diff --git a/tools/testing/selftests/bpf/verifier/map_ptr.c b/tools/testing/selftests/bpf/verifier/map_ptr.c index b117bdd3806d..1a532198c9c1 100644 --- a/tools/testing/selftests/bpf/verifier/map_ptr.c +++ b/tools/testing/selftests/bpf/verifier/map_ptr.c @@ -93,3 +93,73 @@ .fixup_map_hash_16b = { 4 }, .result = ACCEPT, }, +{ + "ARG_CONST_MAP_PTR: null pointer", + .insns = { + /* bpf_redirect_map arg1 (map) */ + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .result = REJECT, + .prog_type = BPF_PROG_TYPE_XDP, + .errstr = "R1 type=inv expected=map_ptr", +}, +{ + "ARG_CONST_MAP_PTR: valid map pointer", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg1 (map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_LD_MAP_FD(BPF_REG_2, 1), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .fixup_map_devmap_hash = { 3 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv13 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test 2020-12-21 12:34 ` [PATCHv13 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (4 preceding siblings ...) 2020-12-21 12:35 ` [PATCHv13 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2020-12-21 12:35 ` Hangbin Liu 2021-01-14 14:23 ` [PATCHv14 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2020-12-21 12:35 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu Add a bpf selftest for new helper xdp_redirect_map_multi(). In this test we have 3 forward groups and 1 exclude group. The test will redirect each interface's packets to all the interfaces in the forward group, and exclude the interface in exclude map. We will also test both DEVMAP and DEVMAP_HASH with xdp generic and drv. For more test details, you can find it in the test script. Here is the test result. ]# ./test_xdp_redirect_multi.sh Pass: xdpgeneric arp ns1-2 Pass: xdpgeneric arp ns1-3 Pass: xdpgeneric arp ns1-4 Pass: xdpgeneric ping ns1-2 Pass: xdpgeneric ping ns1-3 Pass: xdpgeneric ping ns1-4 Pass: xdpgeneric ping6 ns2-1 Pass: xdpgeneric ping6 ns2-3 Pass: xdpgeneric ping6 ns2-4 Pass: xdpdrv arp ns1-2 Pass: xdpdrv arp ns1-3 Pass: xdpdrv arp ns1-4 Pass: xdpdrv ping ns1-2 Pass: xdpdrv ping ns1-3 Pass: xdpdrv ping ns1-4 Pass: xdpdrv ping6 ns2-1 Pass: xdpdrv ping6 ns2-3 Pass: xdpdrv ping6 ns2-4 Pass: xdpegress mac ns1-2 Pass: xdpegress mac ns1-3 Pass: xdpegress mac ns1-4 Pass: xdpegress ping ns1-2 Pass: xdpegress ping ns1-3 Pass: xdpegress ping ns1-4 Summary: PASS 24, FAIL 0 Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v13: remove setrlimit v12: add devmap prog test on egress v9: use NULL directly for arg2 and redefine the maps with btf format --- tools/testing/selftests/bpf/Makefile | 3 +- .../bpf/progs/xdp_redirect_multi_kern.c | 120 +++++++++ .../selftests/bpf/test_xdp_redirect_multi.sh | 208 +++++++++++++++ .../selftests/bpf/xdp_redirect_multi.c | 252 ++++++++++++++++++ 4 files changed, 582 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 8c33e999319a..20b9481c679f 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -51,6 +51,7 @@ TEST_FILES = test_lwt_ip_encap.o \ # Order correspond to 'make run_tests' order TEST_PROGS := test_kmod.sh \ test_xdp_redirect.sh \ + test_xdp_redirect_multi.sh \ test_xdp_meta.sh \ test_xdp_veth.sh \ test_offload.py \ @@ -80,7 +81,7 @@ TEST_PROGS_EXTENDED := with_addr.sh \ TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ test_lirc_mode2_user xdping test_cpp runqslower bench bpf_testmod.ko \ - xdpxceiver + xdpxceiver xdp_redirect_multi TEST_CUSTOM_PROGS = urandom_read diff --git a/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c new file mode 100644 index 000000000000..a9785b28175a --- /dev/null +++ b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c @@ -0,0 +1,120 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <string.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ip.h> +#include <linux/ipv6.h> + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_endian.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v4 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v6 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_all SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(struct bpf_devmap_val)); + __uint(max_entries, 128); +} forward_map_egress SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} exclude_map SEC(".maps"); + +/* map to stroe egress interfaces mac addresses */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, __u32); + __type(value, __be64); + __uint(max_entries, 128); +} mac_map SEC(".maps"); + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + struct ethhdr *eth = data; + __u16 h_proto; + __u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + h_proto = eth->h_proto; + + if (h_proto == bpf_htons(ETH_P_IP)) + return bpf_redirect_map_multi(&forward_map_v4, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else if (h_proto == bpf_htons(ETH_P_IPV6)) + return bpf_redirect_map_multi(&forward_map_v6, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else + return bpf_redirect_map_multi(&forward_map_all, NULL, + BPF_F_EXCLUDE_INGRESS); +} + +/* The following 2 progs are for 2nd devmap prog testing */ +SEC("xdp_redirect_map_ingress") +int xdp_redirect_map_all_prog(struct xdp_md *ctx) +{ + return bpf_redirect_map_multi(&forward_map_egress, NULL, BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_devmap/map_prog") +int xdp_devmap_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + __u32 key = ctx->egress_ifindex; + struct ethhdr *eth = data; + __u64 nh_off; + __be64 *mac; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + mac = bpf_map_lookup_elem(&mac_map, &key); + if (mac) + __builtin_memcpy(eth->h_source, mac, ETH_ALEN); + + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh new file mode 100755 index 000000000000..6503751fdca5 --- /dev/null +++ b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh @@ -0,0 +1,208 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Test topology: +# - - - - - - - - - - - - - - - - - - - - - - - - - +# | veth1 veth2 veth3 veth4 | ... init net +# - -| - - - - - - | - - - - - - | - - - - - - | - - +# --------- --------- --------- --------- +# | veth0 | | veth0 | | veth0 | | veth0 | ... +# --------- --------- --------- --------- +# ns1 ns2 ns3 ns4 +# +# Forward maps: +# Forward map_all has interfaces: veth1, veth2, veth3, veth4, ... (All traffic except IPv4, IPv6) +# Forward map_v4 has interfaces: veth1, veth3, veth4, ... (For IPv4 traffic only) +# Forward map_v6 has interfaces: veth2, veth3, veth4, ... (For IPv6 traffic only) +# Forward map_egress has all interfaces and redirect all pkts +# Exclude Groups: +# Exclude map: veth3 (assume ns3 is in black list) +# Map type: +# map_v4 use DEVMAP, others use DEVMAP_HASH +# +# Test modules: +# XDP modes: generic, native, native + egress_prog +# +# Test cases: +# ARP(we didn't block ARP for ns3): +# ns1 -> gw: ns2, ns3, ns4 should receive the arp request +# IPv4: +# ns1 -> ns2 (block), ns1 -> ns3 (block), ns1 -> ns4 (pass) +# IPv6 +# ns2 -> ns1 (block), ns2 -> ns3 (block), ns2 -> ns4 (pass) +# egress_prog: +# all ping test should pass, the src mac should be egress interface's mac +# + + +# netns numbers +NUM=4 +IFACES="" +DRV_MODE="xdpgeneric xdpdrv xdpegress" +PASS=0 +FAIL=0 + +test_pass() +{ + echo "Pass: $@" + PASS=$((PASS + 1)) +} + +test_fail() +{ + echo "fail: $@" + FAIL=$((FAIL + 1)) +} + +clean_up() +{ + for i in $(seq $NUM); do + ip link del veth$i 2> /dev/null + ip netns del ns$i 2> /dev/null + done +} + +# Kselftest framework requirement - SKIP code is 4. +check_env() +{ + ip link set dev lo xdpgeneric off &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without the ip xdpgeneric support" + exit 4 + fi + + which tcpdump &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without tcpdump" + exit 4 + fi +} + +setup_ns() +{ + local mode=$1 + IFACES="" + + if [ "$mode" = "xdpegress" ]; then + mode="xdpdrv" + fi + + for i in $(seq $NUM); do + ip netns add ns$i + ip link add veth$i type veth peer name veth0 netns ns$i + ip link set veth$i up + ip -n ns$i link set veth0 up + + ip -n ns$i addr add 192.0.2.$i/24 dev veth0 + ip -n ns$i addr add 2001:db8::$i/64 dev veth0 + ip -n ns$i link set veth0 $mode obj \ + xdp_dummy.o sec xdp_dummy &> /dev/null || \ + { test_fail "Unable to load dummy xdp" && exit 1; } + IFACES="$IFACES veth$i" + veth_mac[$i]=$(ip link show veth$i | awk '/link\/ether/ {print $2}') + done +} + +do_egress_tests() +{ + local mode=$1 + + # mac test + ip netns exec ns2 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + + # mac check + grep -q "${veth_mac[2]} > ff:ff:ff:ff:ff:ff" mac_ns1-2_${mode}.log && \ + test_pass "$mode mac ns1-2" || test_fail "$mode mac ns1-2" + grep -q "${veth_mac[3]} > ff:ff:ff:ff:ff:ff" mac_ns1-3_${mode}.log && \ + test_pass "$mode mac ns1-3" || test_fail "$mode mac ns1-3" + grep -q "${veth_mac[4]} > ff:ff:ff:ff:ff:ff" mac_ns1-4_${mode}.log && \ + test_pass "$mode mac ns1-4" || test_fail "$mode mac ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-2" || test_fail "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-3" || test_fail "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" +} + +do_ping_tests() +{ + local mode=$1 + + # arp test + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -i veth0 -nn -l -e &> arp_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-2_${mode}.log && \ + test_pass "$mode arp ns1-2" || test_fail "$mode arp ns1-2" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-3_${mode}.log && \ + test_pass "$mode arp ns1-3" || test_fail "$mode arp ns1-3" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-4_${mode}.log && \ + test_pass "$mode arp ns1-4" || test_fail "$mode arp ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-2" || test_pass "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-3" || test_pass "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" + + # ping6 test + ip netns exec ns2 ping6 2001:db8::1 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-1" || test_pass "$mode ping6 ns2-1" + ip netns exec ns2 ping6 2001:db8::3 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-3" || test_pass "$mode ping6 ns2-3" + ip netns exec ns2 ping6 2001:db8::4 -c 4 &> /dev/null && \ + test_pass "$mode ping6 ns2-4" || test_fail "$mode ping6 ns2-4" +} + +do_tests() +{ + local mode=$1 + local drv_p + + case ${mode} in + xdpdrv) drv_p="-N";; + xdpegress) drv_p="-X";; + xdpgeneric) drv_p="-S";; + esac + + ./xdp_redirect_multi $drv_p $IFACES &> xdp_redirect_${mode}.log & + xdp_pid=$! + sleep 10 + + if [ "$mode" = "xdpegress" ]; then + do_egress_tests $mode + else + do_ping_tests $mode + fi + + kill $xdp_pid +} + +trap clean_up 0 2 3 6 9 + +check_env +rm -f xdp_redirect_*.log arp_ns*.log mac_ns*.log + +for mode in ${DRV_MODE}; do + setup_ns $mode + do_tests $mode + sleep 10 + clean_up + sleep 5 +done + +echo "Summary: PASS $PASS, FAIL $FAIL" +[ $FAIL -eq 0 ] && exit 0 || exit 1 diff --git a/tools/testing/selftests/bpf/xdp_redirect_multi.c b/tools/testing/selftests/bpf/xdp_redirect_multi.c new file mode 100644 index 000000000000..906db3d76071 --- /dev/null +++ b/tools/testing/selftests/bpf/xdp_redirect_multi.c @@ -0,0 +1,252 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> +#include <sys/resource.h> +#include <sys/ioctl.h> +#include <sys/types.h> +#include <sys/socket.h> +#include <netinet/in.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static int get_mac_addr(unsigned int ifindex, void *mac_addr) +{ + char ifname[IF_NAMESIZE]; + struct ifreq ifr; + int fd, ret = -1; + + fd = socket(AF_INET, SOCK_DGRAM, 0); + if (fd < 0) + return ret; + + if (!if_indextoname(ifindex, ifname)) + goto err_out; + + strcpy(ifr.ifr_name, ifname); + + if (ioctl(fd, SIOCGIFHWADDR, &ifr) != 0) + goto err_out; + + memcpy(mac_addr, ifr.ifr_hwaddr.sa_data, 6 * sizeof(char)); + ret = 0; + +err_out: + close(fd); + return ret; +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n" + " -X load xdp program on egress\n", + prog); +} + +int main(int argc, char **argv) +{ + int prog_fd, group_all, group_v4, group_v6, exclude, mac_map; + struct bpf_program *ingress_prog, *egress_prog; + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_UNSPEC, + }; + int i, ret, opt, egress_prog_fd = -1; + struct bpf_devmap_val devmap_val; + bool attach_egress_prog = false; + unsigned char mac_addr[6]; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + unsigned int ifindex; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNFX")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + case 'X': + attach_egress_prog = true; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) { + xdp_flags |= XDP_FLAGS_DRV_MODE; + } else if (attach_egress_prog) { + printf("Load xdp program on egress with SKB mode not supported yet\n"); + goto err_out; + } + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + goto err_out; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + goto err_out; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + goto err_out; + + if (attach_egress_prog) + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_egress"); + else + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_all"); + group_v4 = bpf_object__find_map_fd_by_name(obj, "forward_map_v4"); + group_v6 = bpf_object__find_map_fd_by_name(obj, "forward_map_v6"); + exclude = bpf_object__find_map_fd_by_name(obj, "exclude_map"); + mac_map = bpf_object__find_map_fd_by_name(obj, "mac_map"); + + if (group_all < 0 || group_v4 < 0 || group_v6 < 0 || exclude < 0 || + mac_map < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + goto err_out; + } + + if (attach_egress_prog) { + /* Find ingress/egress prog for 2nd xdp prog */ + ingress_prog = bpf_object__find_program_by_title(obj, "xdp_redirect_map_ingress"); + egress_prog = bpf_object__find_program_by_title(obj, "xdp_devmap/map_prog"); + if (!ingress_prog || !egress_prog) { + printf("finding ingress/egress_prog in obj file failed\n"); + goto err_out; + } + prog_fd = bpf_program__fd(ingress_prog); + egress_prog_fd = bpf_program__fd(egress_prog); + if (prog_fd < 0 || egress_prog_fd < 0) { + printf("find egress_prog fd failed\n"); + goto err_out; + } + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + if (attach_egress_prog) { + ret = get_mac_addr(ifindex, mac_addr); + if (ret < 0) { + printf("get interface %d mac failed\n", ifindex); + goto err_out; + } + ret = bpf_map_update_elem(mac_map, &ifindex, mac_addr, 0); + if (ret) { + perror("bpf_update_elem mac_map failed\n"); + goto err_out; + } + } + + /* Add all the interfaces to group all */ + devmap_val.ifindex = ifindex; + devmap_val.bpf_prog.fd = egress_prog_fd; + ret = bpf_map_update_elem(group_all, &ifindex, &devmap_val, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* For testing: remove the 1st interfaces from group v6 */ + if (i != 0) { + ret = bpf_map_update_elem(group_v6, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: remove the 2nd interfaces from group v4 */ + if (i != 1) { + ret = bpf_map_update_elem(group_v4, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: add the 3rd interfaces to exclude map */ + if (i == 2) { + ret = bpf_map_update_elem(exclude, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + } + + /* sleep some time for testing */ + sleep(999); + + return 0; + +err_out: + return 1; +} -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv14 bpf-next 0/6] xdp: add a new helper for dev map multicast support 2020-12-21 12:34 ` [PATCHv13 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (5 preceding siblings ...) 2020-12-21 12:35 ` [PATCHv13 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu @ 2021-01-14 14:23 ` Hangbin Liu 2021-01-14 14:23 ` [PATCHv14 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu ` (6 more replies) 6 siblings, 7 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-14 14:23 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because there may have multi interfaces you want to exclude. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. The 1st patch is Jesper's run devmap xdp_prog later in bulking step. The 2st patch add a new bpf arg to allow NULL map pointer. The 3rd patch add the new bpf_redirect_map_multi() helper. The 4-6 patches are for usage sample and testing purpose. I did same perf tests with the following topo: --------------------- --------------------- | Host A (i40e 10G) | ---------- | eno1(i40e 10G) | --------------------- | | | Host B | --------------------- | | | Host C (i40e 10G) | ---------- | eno2(i40e 10G) | --------------------- vlan2 | -------- | | veth1 -- | veth0| | | -------- | --------------------| On Host A: # pktgen/pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -s 64 On Host B(Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, 128G Memory): Use xdp_redirect_map and xdp_redirect_map_multi in samples/bpf for testing. The veth0 in netns load dummy drop program. The forward_map max_entries in xdp_redirect_map_multi is modify to 4. Here is the perf result with 5.10 rc6: The are about +/- 0.1M deviation for native testing Version | Test | Generic | Native | Native + 2nd 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M 5.10 rc6 + patch1 | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch1 | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M 5.10 rc6 + patch1-6 | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch1-6 | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->i40e | 1.7M | 7.8M | 6.4M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->veth | 1.4M | 9.3M | 7.5M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->i40e+veth | 1.0M | 3.2M | 2.7M Last but not least, thanks a lot to Toke, Jesper, Jiri and Eelco for suggestions and help on implementation. [0] https://xdp-project.net/#Handling-multicast v14: No code update, just rebase the code on latest bpf-next v13: Pass in xdp_prog through __xdp_enqueue() for patch 01. Update related code in patch 03. v12: Add Jesper's xdp_prog patch, rebase my works on this and latest bpf-next Add 2nd xdp_prog test on the sample and selftests. v11: Fix bpf_redirect_map_multi() helper description typo. Add loop limit for devmap_get_next_obj() and dev_map_redirect_multi(). v10: Rebase the code to latest bpf-next. Update helper bpf_xdp_redirect_map_multi() - No need to check map pointer as we will do the check in verifier. v9: Update helper bpf_xdp_redirect_map_multi() - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: a) Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). f) Split the tests from sample and add a bpf kernel selftest patch. v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Hangbin Liu (5): bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL xdp: add a new helper for dev map multicast support sample/bpf: add xdp_redirect_map_multicast test selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL selftests/bpf: add xdp_redirect_multi test Jesper Dangaard Brouer (1): bpf: run devmap xdp_prog on flush instead of bulk enqueue include/linux/bpf.h | 21 ++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 27 ++ kernel/bpf/devmap.c | 235 +++++++++++--- kernel/bpf/verifier.c | 16 +- net/core/filter.c | 118 ++++++- net/core/xdp.c | 29 ++ samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 96 ++++++ samples/bpf/xdp_redirect_map_multi_user.c | 301 ++++++++++++++++++ tools/include/uapi/linux/bpf.h | 27 ++ tools/testing/selftests/bpf/Makefile | 3 +- .../bpf/progs/xdp_redirect_multi_kern.c | 120 +++++++ tools/testing/selftests/bpf/test_verifier.c | 22 +- .../selftests/bpf/test_xdp_redirect_multi.sh | 208 ++++++++++++ .../testing/selftests/bpf/verifier/map_ptr.c | 70 ++++ .../selftests/bpf/xdp_redirect_multi.c | 252 +++++++++++++++ 18 files changed, 1502 insertions(+), 48 deletions(-) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c -- 2.26.2 ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv14 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-14 14:23 ` [PATCHv14 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2021-01-14 14:23 ` Hangbin Liu 2021-01-17 22:57 ` John Fastabend 2021-01-14 14:23 ` [PATCHv14 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu ` (5 subsequent siblings) 6 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2021-01-14 14:23 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu From: Jesper Dangaard Brouer <brouer@redhat.com> This changes the devmap XDP program support to run the program when the bulk queue is flushed instead of before the frame is enqueued. This has a couple of benefits: - It "sorts" the packets by destination devmap entry, and then runs the same BPF program on all the packets in sequence. This ensures that we keep the XDP program and destination device properties hot in I-cache. - It makes the multicast implementation simpler because it can just enqueue packets using bq_enqueue() without having to deal with the devmap program at all. The drawback is that if the devmap program drops the packet, the enqueue step is redundant. However, arguably this is mostly visible in a micro-benchmark, and with more mixed traffic the I-cache benefit should win out. The performance impact of just this patch is as follows: Using xdp_redirect_map(with a 2nd xdp_prog patch[1]) in sample/bpf and send pkts via pktgen cmd: ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 There are about +/- 0.1M deviation for native testing, the performance improved for the base-case, but some drop back with xdp devmap prog attached. Version | Test | Generic | Native | Native + 2nd xdp_prog 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M 5.10 rc6 + patch | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M [1] https://patchwork.ozlabs.org/project/netdev/patch/20201208120159.2278277-1-liuhangbin@gmail.com/ Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> -- v14: no update, only rebase the code v13: pass in xdp_prog through __xdp_enqueue() v2-v12: no this patch --- kernel/bpf/devmap.c | 115 +++++++++++++++++++++++++++----------------- 1 file changed, 72 insertions(+), 43 deletions(-) diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index f6e9c68afdd4..84fe15950e44 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -57,6 +57,7 @@ struct xdp_dev_bulk_queue { struct list_head flush_node; struct net_device *dev; struct net_device *dev_rx; + struct bpf_prog *xdp_prog; unsigned int count; }; @@ -327,40 +328,92 @@ bool dev_map_can_have_prog(struct bpf_map *map) return false; } +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, + struct xdp_frame **frames, int n, + struct net_device *dev) +{ + struct xdp_txq_info txq = { .dev = dev }; + struct xdp_buff xdp; + int i, nframes = 0; + + for (i = 0; i < n; i++) { + struct xdp_frame *xdpf = frames[i]; + u32 act; + int err; + + xdp_convert_frame_to_buff(xdpf, &xdp); + xdp.txq = &txq; + + act = bpf_prog_run_xdp(xdp_prog, &xdp); + switch (act) { + case XDP_PASS: + err = xdp_update_frame_from_buff(&xdp, xdpf); + if (unlikely(err < 0)) + xdp_return_frame_rx_napi(xdpf); + else + frames[nframes++] = xdpf; + break; + default: + bpf_warn_invalid_xdp_action(act); + fallthrough; + case XDP_ABORTED: + trace_xdp_exception(dev, xdp_prog, act); + fallthrough; + case XDP_DROP: + xdp_return_frame_rx_napi(xdpf); + break; + } + } + return n - nframes; /* dropped frames count */ +} + static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) { struct net_device *dev = bq->dev; int sent = 0, drops = 0, err = 0; + unsigned int cnt = bq->count; + unsigned int xdp_drop; int i; - if (unlikely(!bq->count)) + if (unlikely(!cnt)) return; - for (i = 0; i < bq->count; i++) { + for (i = 0; i < cnt; i++) { struct xdp_frame *xdpf = bq->q[i]; prefetch(xdpf); } - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); + if (unlikely(bq->xdp_prog)) { + xdp_drop = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); + cnt -= xdp_drop; + if (!cnt) { + sent = 0; + drops = xdp_drop; + goto out; + } + } + + sent = dev->netdev_ops->ndo_xdp_xmit(dev, cnt, bq->q, flags); if (sent < 0) { err = sent; sent = 0; goto error; } - drops = bq->count - sent; + drops = (cnt - sent) + xdp_drop; out: bq->count = 0; trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); bq->dev_rx = NULL; + bq->xdp_prog = NULL; __list_del_clearprev(&bq->flush_node); return; error: /* If ndo_xdp_xmit fails with an errno, no frames have been * xmit'ed and it's our responsibility to them free all. */ - for (i = 0; i < bq->count; i++) { + for (i = 0; i < cnt; i++) { struct xdp_frame *xdpf = bq->q[i]; xdp_return_frame_rx_napi(xdpf); @@ -408,7 +461,7 @@ struct bpf_dtab_netdev *__dev_map_lookup_elem(struct bpf_map *map, u32 key) * Thus, safe percpu variable access. */ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, - struct net_device *dev_rx) + struct net_device *dev_rx, struct bpf_prog *xdp_prog) { struct list_head *flush_list = this_cpu_ptr(&dev_flush_list); struct xdp_dev_bulk_queue *bq = this_cpu_ptr(dev->xdp_bulkq); @@ -423,6 +476,14 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, if (!bq->dev_rx) bq->dev_rx = dev_rx; + /* Store (potential) xdp_prog that run before egress to dev as + * part of bulk_queue. This will be same xdp_prog for all + * xdp_frame's in bulk_queue, because this per-CPU store must + * be flushed from net_device drivers NAPI func end. + */ + if (!bq->xdp_prog) + bq->xdp_prog = xdp_prog; + bq->q[bq->count++] = xdpf; if (!bq->flush_node.prev) @@ -430,7 +491,8 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, } static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, - struct net_device *dev_rx) + struct net_device *dev_rx, + struct bpf_prog *xdp_prog) { struct xdp_frame *xdpf; int err; @@ -446,42 +508,14 @@ static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, if (unlikely(!xdpf)) return -EOVERFLOW; - bq_enqueue(dev, xdpf, dev_rx); + bq_enqueue(dev, xdpf, dev_rx, xdp_prog); return 0; } -static struct xdp_buff *dev_map_run_prog(struct net_device *dev, - struct xdp_buff *xdp, - struct bpf_prog *xdp_prog) -{ - struct xdp_txq_info txq = { .dev = dev }; - u32 act; - - xdp_set_data_meta_invalid(xdp); - xdp->txq = &txq; - - act = bpf_prog_run_xdp(xdp_prog, xdp); - switch (act) { - case XDP_PASS: - return xdp; - case XDP_DROP: - break; - default: - bpf_warn_invalid_xdp_action(act); - fallthrough; - case XDP_ABORTED: - trace_xdp_exception(dev, xdp_prog, act); - break; - } - - xdp_return_buff(xdp); - return NULL; -} - int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx) { - return __xdp_enqueue(dev, xdp, dev_rx); + return __xdp_enqueue(dev, xdp, dev_rx, NULL); } int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, @@ -489,12 +523,7 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, { struct net_device *dev = dst->dev; - if (dst->xdp_prog) { - xdp = dev_map_run_prog(dev, xdp, dst->xdp_prog); - if (!xdp) - return 0; - } - return __xdp_enqueue(dev, xdp, dev_rx); + return __xdp_enqueue(dev, xdp, dev_rx, dst->xdp_prog); } int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* RE: [PATCHv14 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-14 14:23 ` [PATCHv14 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu @ 2021-01-17 22:57 ` John Fastabend 2021-01-18 10:07 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: John Fastabend @ 2021-01-17 22:57 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu Hangbin Liu wrote: > From: Jesper Dangaard Brouer <brouer@redhat.com> > > This changes the devmap XDP program support to run the program when the > bulk queue is flushed instead of before the frame is enqueued. This has > a couple of benefits: > > - It "sorts" the packets by destination devmap entry, and then runs the > same BPF program on all the packets in sequence. This ensures that we > keep the XDP program and destination device properties hot in I-cache. > > - It makes the multicast implementation simpler because it can just > enqueue packets using bq_enqueue() without having to deal with the > devmap program at all. > > The drawback is that if the devmap program drops the packet, the enqueue > step is redundant. However, arguably this is mostly visible in a > micro-benchmark, and with more mixed traffic the I-cache benefit should > win out. The performance impact of just this patch is as follows: > > Using xdp_redirect_map(with a 2nd xdp_prog patch[1]) in sample/bpf and send > pkts via pktgen cmd: > ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 > > There are about +/- 0.1M deviation for native testing, the performance > improved for the base-case, but some drop back with xdp devmap prog attached. > > Version | Test | Generic | Native | Native + 2nd xdp_prog > 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M > 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M > 5.10 rc6 + patch | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M > 5.10 rc6 + patch | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M > > [1] https://patchwork.ozlabs.org/project/netdev/patch/20201208120159.2278277-1-liuhangbin@gmail.com/ > > Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com> > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > > -- > v14: no update, only rebase the code > v13: pass in xdp_prog through __xdp_enqueue() > v2-v12: no this patch > --- > kernel/bpf/devmap.c | 115 +++++++++++++++++++++++++++----------------- > 1 file changed, 72 insertions(+), 43 deletions(-) > > diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c > index f6e9c68afdd4..84fe15950e44 100644 > --- a/kernel/bpf/devmap.c > +++ b/kernel/bpf/devmap.c > @@ -57,6 +57,7 @@ struct xdp_dev_bulk_queue { > struct list_head flush_node; > struct net_device *dev; > struct net_device *dev_rx; > + struct bpf_prog *xdp_prog; > unsigned int count; > }; > > @@ -327,40 +328,92 @@ bool dev_map_can_have_prog(struct bpf_map *map) > return false; > } > > +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, > + struct xdp_frame **frames, int n, > + struct net_device *dev) > +{ > + struct xdp_txq_info txq = { .dev = dev }; > + struct xdp_buff xdp; > + int i, nframes = 0; > + > + for (i = 0; i < n; i++) { > + struct xdp_frame *xdpf = frames[i]; > + u32 act; > + int err; > + > + xdp_convert_frame_to_buff(xdpf, &xdp); Hi, slightly higher level question about the desgin. How come we have to bounce the xdp_frame back and forth between an xdp_buff<->xdp-frame? Seems a bit wasteful. > + xdp.txq = &txq; > + > + act = bpf_prog_run_xdp(xdp_prog, &xdp); > + switch (act) { > + case XDP_PASS: > + err = xdp_update_frame_from_buff(&xdp, xdpf); xdp_update_frame_from_buff will then convert it back from the xdp_buff? struct xdp_buff { void *data; void *data_end; void *data_meta; void *data_hard_start; struct xdp_rxq_info *rxq; struct xdp_txq_info *txq; u32 frame_sz; /* frame size to deduce data_hard_end/reserved tailroom*/ }; struct xdp_frame { void *data; u16 len; u16 headroom; u32 metasize:8; u32 frame_sz:24; /* Lifetime of xdp_rxq_info is limited to NAPI/enqueue time, * while mem info is valid on remote CPU. */ struct xdp_mem_info mem; struct net_device *dev_rx; /* used by cpumap */ }; It looks like we could embed xdp_buff in xdp_frame and then keep the metadata at the end. Because you are working performance here wdyt? <- @Jesper as well. > + if (unlikely(err < 0)) > + xdp_return_frame_rx_napi(xdpf); > + else > + frames[nframes++] = xdpf; > + break; > + default: > + bpf_warn_invalid_xdp_action(act); > + fallthrough; > + case XDP_ABORTED: > + trace_xdp_exception(dev, xdp_prog, act); > + fallthrough; > + case XDP_DROP: > + xdp_return_frame_rx_napi(xdpf); > + break; > + } > + } > + return n - nframes; /* dropped frames count */ > +} > + > static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) > { > struct net_device *dev = bq->dev; > int sent = 0, drops = 0, err = 0; > + unsigned int cnt = bq->count; > + unsigned int xdp_drop; > int i; > > - if (unlikely(!bq->count)) > + if (unlikely(!cnt)) > return; > > - for (i = 0; i < bq->count; i++) { > + for (i = 0; i < cnt; i++) { > struct xdp_frame *xdpf = bq->q[i]; > > prefetch(xdpf); > } > > - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); > + if (unlikely(bq->xdp_prog)) { Whats the rational for making above unlikely()? Seems for users its not unlikely. Can you measure a performance increase/decrease here? I think its probably fine to just let compiler/prefetcher do its thing here. Or I'm not reading this right, but seems users of bq->xdp_prog would disagree on unlikely case? Either way a comment might be nice to give us some insight in 6 months why we decided this is unlikely. > + xdp_drop = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); > + cnt -= xdp_drop; > + if (!cnt) { if dev_map_bpf_prog_run() returned sent packets this would read better imo. sent = dev_map_bpf_prog_run(...) if (!sent) goto out; > + sent = 0; > + drops = xdp_drop; > + goto out; > + } > + } > + > + sent = dev->netdev_ops->ndo_xdp_xmit(dev, cnt, bq->q, flags); And, sent = dev->netdev_ops->ndo_xdp_xmit(dev, sent, bq->q, flags); > if (sent < 0) { > err = sent; > sent = 0; > goto error; > } > - drops = bq->count - sent; > + drops = (cnt - sent) + xdp_drop; With about 'sent' logic then drops will still be just, drops = bq->count - sent and move the calculation below the out label and I think you clean up above as well. Did I miss something... > out: > bq->count = 0; > > trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); > bq->dev_rx = NULL; > + bq->xdp_prog = NULL; > __list_del_clearprev(&bq->flush_node); > return; > error: > /* If ndo_xdp_xmit fails with an errno, no frames have been > * xmit'ed and it's our responsibility to them free all. > */ > - for (i = 0; i < bq->count; i++) { > + for (i = 0; i < cnt; i++) { > struct xdp_frame *xdpf = bq->q[i]; Patch looks overall good to me, but cleaning up the logic a bit seems like a plus. Thanks, John ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv14 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-17 22:57 ` John Fastabend @ 2021-01-18 10:07 ` Hangbin Liu 2021-01-21 14:33 ` Jesper Dangaard Brouer 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2021-01-18 10:07 UTC (permalink / raw) To: John Fastabend Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov On Sun, Jan 17, 2021 at 02:57:02PM -0800, John Fastabend wrote: [...] > It looks like we could embed xdp_buff in xdp_frame and then keep the metadata > at the end. > > Because you are working performance here wdyt? <- @Jesper as well. Leave this question to Jesper. > > > > - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); > > + if (unlikely(bq->xdp_prog)) { > > Whats the rational for making above unlikely()? Seems for users its not > unlikely. Can you measure a performance increase/decrease here? I think > its probably fine to just let compiler/prefetcher do its thing here. Or > I'm not reading this right, but seems users of bq->xdp_prog would disagree > on unlikely case? > > Either way a comment might be nice to give us some insight in 6 months > why we decided this is unlikely. I agree that there is no need to use unlikely() here. > > > + xdp_drop = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); > > + cnt -= xdp_drop; > > + if (!cnt) { > > > if dev_map_bpf_prog_run() returned sent packets this would read better > imo. > > sent = dev_map_bpf_prog_run(...) > if (!sent) > goto out; > > > + sent = 0; > > + drops = xdp_drop; > > + goto out; > > + } > > + } > > + > > + sent = dev->netdev_ops->ndo_xdp_xmit(dev, cnt, bq->q, flags); > > And, sent = dev->netdev_ops->ndo_xdp_xmit(dev, sent, bq->q, flags); > > > if (sent < 0) { > > err = sent; > > sent = 0; > > goto error; > > } > > - drops = bq->count - sent; > > + drops = (cnt - sent) + xdp_drop; > > With about 'sent' logic then drops will still be just, drops = bq->count - sent > and move the calculation below the out label and I think you clean up above If we use the 'sent' logic, we should also backup the drop value before xmit as the erro label also need it. > as well. Did I miss something... > > > out: > > bq->count = 0; > > > > trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); > > bq->dev_rx = NULL; > > + bq->xdp_prog = NULL; > > __list_del_clearprev(&bq->flush_node); > > return; > > error: > > /* If ndo_xdp_xmit fails with an errno, no frames have been > > * xmit'ed and it's our responsibility to them free all. > > */ > > - for (i = 0; i < bq->count; i++) { > > + for (i = 0; i < cnt; i++) { > > struct xdp_frame *xdpf = bq->q[i]; here it will be "for (i = 0; i < cnt - drops; i++)" to free none xmit'ed frames. To make the logic more clear, here is the full code: [...] if (bq->xdp_prog) { sent = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); if (!sent) goto out; } /* Backup drops value before xmit as we may need it in error label */ drops = cnt - sent; sent = dev->netdev_ops->ndo_xdp_xmit(dev, sent, bq->q, flags); if (sent < 0) { err = sent; sent = 0; goto error; } out: drops = cnt - sent; bq->count = 0; trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); bq->dev_rx = NULL; bq->xdp_prog = NULL; __list_del_clearprev(&bq->flush_node); return; error: /* If ndo_xdp_xmit fails with an errno, no frames have been * xmit'ed and it's our responsibility to them free all. */ for (i = 0; i < cnt - drops; i++) { struct xdp_frame *xdpf = bq->q[i]; xdp_return_frame_rx_napi(xdpf); } goto out; } Thanks hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv14 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-18 10:07 ` Hangbin Liu @ 2021-01-21 14:33 ` Jesper Dangaard Brouer 0 siblings, 0 replies; 219+ messages in thread From: Jesper Dangaard Brouer @ 2021-01-21 14:33 UTC (permalink / raw) To: Hangbin Liu Cc: John Fastabend, bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, brouer On Mon, 18 Jan 2021 18:07:17 +0800 Hangbin Liu <liuhangbin@gmail.com> wrote: > On Sun, Jan 17, 2021 at 02:57:02PM -0800, John Fastabend wrote: > [...] > > It looks like we could embed xdp_buff in xdp_frame and then keep the metadata > > at the end. > > > > Because you are working performance here wdyt? <- @Jesper as well. > > Leave this question to Jesper. The struct xdp_buff is larger than struct xdp_frame. The size of xdp_frame matters. It is a reserved areas in top of the frame. An XDP BPF-program cannot access this area (and limit headroom grow). This is why this code works, as afterwards xdp_frame is still valid. Looking at the code xdp_update_frame_from_buff() we do seem to update more fields than actually needed. > > > > > > - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); > > > + if (unlikely(bq->xdp_prog)) { > > > > Whats the rational for making above unlikely()? Seems for users its not > > unlikely. Can you measure a performance increase/decrease here? I think > > its probably fine to just let compiler/prefetcher do its thing here. Or > > I'm not reading this right, but seems users of bq->xdp_prog would disagree > > on unlikely case? > > > > Either way a comment might be nice to give us some insight in 6 months > > why we decided this is unlikely. > > I agree that there is no need to use unlikely() here. I added the unlikely() to preserve the baseline performance when not having the 2nd prog loaded. But I'm fine with removing that. -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv14 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL 2021-01-14 14:23 ` [PATCHv14 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2021-01-14 14:23 ` [PATCHv14 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu @ 2021-01-14 14:23 ` Hangbin Liu 2021-01-14 14:23 ` [PATCHv14 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (4 subsequent siblings) 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-14 14:23 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu Add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL which could be used when we want to allow NULL pointer for map parameter. The bpf helper need to take care and check if the map is NULL when use this type. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v11-v14: rebase the patch to latest bpf-next v10: remove useless CONST_PTR_TO_MAP_OR_NULL and Copy-paste comment. v9: merge the patch from [1] in to this series. v1-v8: no this patch [1] https://lore.kernel.org/bpf/20200715070001.2048207-1-liuhangbin@gmail.com/ --- include/linux/bpf.h | 1 + kernel/bpf/verifier.c | 10 ++++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 1aac2af12fed..b75207a2484c 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -296,6 +296,7 @@ enum bpf_arg_type { ARG_CONST_ALLOC_SIZE_OR_ZERO, /* number of allocated bytes requested */ ARG_PTR_TO_BTF_ID_SOCK_COMMON, /* pointer to in-kernel sock_common or bpf-mirrored bpf_sock */ ARG_PTR_TO_PERCPU_BTF_ID, /* pointer to in-kernel percpu type */ + ARG_CONST_MAP_PTR_OR_NULL, /* const argument used as pointer to bpf_map or NULL */ __BPF_ARG_TYPE_MAX, }; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index ae2aee48cf82..3e4b5d9fce78 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -445,7 +445,8 @@ static bool arg_type_may_be_null(enum bpf_arg_type type) type == ARG_PTR_TO_MEM_OR_NULL || type == ARG_PTR_TO_CTX_OR_NULL || type == ARG_PTR_TO_SOCKET_OR_NULL || - type == ARG_PTR_TO_ALLOC_MEM_OR_NULL; + type == ARG_PTR_TO_ALLOC_MEM_OR_NULL || + type == ARG_CONST_MAP_PTR_OR_NULL; } /* Determine whether the function releases some resources allocated by another @@ -4065,6 +4066,7 @@ static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = { [ARG_CONST_SIZE_OR_ZERO] = &scalar_types, [ARG_CONST_ALLOC_SIZE_OR_ZERO] = &scalar_types, [ARG_CONST_MAP_PTR] = &const_map_ptr_types, + [ARG_CONST_MAP_PTR_OR_NULL] = &const_map_ptr_types, [ARG_PTR_TO_CTX] = &context_types, [ARG_PTR_TO_CTX_OR_NULL] = &context_types, [ARG_PTR_TO_SOCK_COMMON] = &sock_types, @@ -4210,9 +4212,9 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, meta->ref_obj_id = reg->ref_obj_id; } - if (arg_type == ARG_CONST_MAP_PTR) { - /* bpf_map_xxx(map_ptr) call: remember that map_ptr */ - meta->map_ptr = reg->map_ptr; + if (arg_type == ARG_CONST_MAP_PTR || + arg_type == ARG_CONST_MAP_PTR_OR_NULL) { + meta->map_ptr = register_is_null(reg) ? NULL : reg->map_ptr; } else if (arg_type == ARG_PTR_TO_MAP_KEY) { /* bpf_map_xxx(..., map_ptr, ..., key) call: * check that [key, key + map->key_size) are within -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv14 bpf-next 3/6] xdp: add a new helper for dev map multicast support 2021-01-14 14:23 ` [PATCHv14 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2021-01-14 14:23 ` [PATCHv14 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu 2021-01-14 14:23 ` [PATCHv14 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2021-01-14 14:23 ` Hangbin Liu 2021-01-18 0:10 ` John Fastabend 2021-01-14 14:23 ` [PATCHv14 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu ` (3 subsequent siblings) 6 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2021-01-14 14:23 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because a physical interface can be part of a logical grouping, such as a bond device. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. The forwarding map could be DEVMAP or DEVMAP_HASH, but the exclude map *must* be DEVMAP_HASH to get better performace. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. As both bpf_xdp_redirect_map() and this new helpers are using struct bpf_redirect_info, I add a new ex_map and set tgt_value to NULL in the new helper to make a difference with bpf_xdp_redirect_map(). Also I keep the general data path in net/core/filter.c, the native data path in kernel/bpf/devmap.c so we can use direct calls to get better performace. [0] https://xdp-project.net/#Handling-multicast Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v14: no update, only rebase the code v13: pass xdp_prog through bq_enqueue v12: rebase the code based on Jespoer's devmap xdp_prog patch v11: Fix bpf_redirect_map_multi() helper description typo. Add loop limit for devmap_get_next_obj() and dev_map_redirect_multi(). v10: Update helper bpf_xdp_redirect_map_multi() - No need to check map pointer as we will do the check in verifier. v9: Update helper bpf_xdp_redirect_map_multi() - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. --- include/linux/bpf.h | 20 +++++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 27 +++++++ kernel/bpf/devmap.c | 132 +++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++ net/core/filter.c | 118 +++++++++++++++++++++++++++-- net/core/xdp.c | 29 ++++++++ tools/include/uapi/linux/bpf.h | 27 +++++++ 9 files changed, 356 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index b75207a2484c..4b68e975e990 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1427,6 +1427,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx); int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, struct net_device *dev_rx); +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex); +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); bool dev_map_can_have_prog(struct bpf_map *map); @@ -1595,6 +1600,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return 0; } +static inline +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + return false; +} + +static inline +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + return 0; +} + struct sk_buff; static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, diff --git a/include/linux/filter.h b/include/linux/filter.h index 5edf2b660881..9ad109b3ba3e 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -620,6 +620,7 @@ struct bpf_redirect_info { u32 tgt_index; void *tgt_value; struct bpf_map *map; + struct bpf_map *ex_map; u32 kern_flags; struct bpf_nh_params nh; }; diff --git a/include/net/xdp.h b/include/net/xdp.h index 0cf3976ce77c..0e6468cd0ab9 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -164,6 +164,7 @@ void xdp_warn(const char *msg, const char *func, const int line); #define XDP_WARN(msg) xdp_warn(msg, __func__, __LINE__) struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); static inline void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index a1ad32456f89..ecf5d117b96a 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3830,6 +3830,27 @@ union bpf_attr { * Return * A pointer to a struct socket on success or NULL if the file is * not a socket. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3995,6 +4016,7 @@ union bpf_attr { FN(ktime_get_coarse_ns), \ FN(ima_inode_hash), \ FN(sock_from_file), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -4171,6 +4193,11 @@ enum { BPF_F_BPRM_SECUREEXEC = (1ULL << 0), }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 84fe15950e44..098cff728f7f 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -526,6 +526,138 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return __xdp_enqueue(dev, xdp, dev_rx, dst->xdp_prog); } +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + return dev_map_get_next_key(map, key, next_key); + case BPF_MAP_TYPE_DEVMAP_HASH: + return dev_map_hash_get_next_key(map, key, next_key); + default: + break; + } + + return -ENOENT; +} + +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + if (obj->dev->ifindex == exclude_ifindex) + return true; + + if (!map) + return false; + + return __dev_map_hash_lookup_elem(map, obj->dev->ifindex) != NULL; +} + +static struct bpf_dtab_netdev *devmap_get_next_obj(struct xdp_buff *xdp, struct bpf_map *map, + struct bpf_map *ex_map, u32 *key, + u32 *next_key, int ex_ifindex) +{ + struct bpf_dtab_netdev *obj; + struct net_device *dev; + u32 *tmp_key = key; + u32 index; + int err; + + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + return NULL; + + /* When using dev map hash, we could restart the hashtab traversal + * in case the key has been updated/removed in the mean time. + * So we may end up potentially looping due to traversal restarts + * from first elem. + * + * Let's use map's max_entries to limit the loop number. + */ + for (index = 0; index < map->max_entries; index++) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + obj = __dev_map_lookup_elem(map, *next_key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + obj = __dev_map_hash_lookup_elem(map, *next_key); + break; + default: + break; + } + + if (!obj || dev_in_exclude_map(obj, ex_map, ex_ifindex)) + goto find_next; + + dev = obj->dev; + + if (!dev->netdev_ops->ndo_xdp_xmit) + goto find_next; + + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + if (unlikely(err)) + goto find_next; + + return obj; + +find_next: + tmp_key = next_key; + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + break; + } + + return NULL; +} + +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + struct bpf_dtab_netdev *obj = NULL, *next_obj = NULL; + struct xdp_frame *xdpf, *nxdpf; + bool last_one = false; + int ex_ifindex; + u32 key, next_key; + + ex_ifindex = flags & BPF_F_EXCLUDE_INGRESS ? dev_rx->ifindex : 0; + + /* Find first available obj */ + obj = devmap_get_next_obj(xdp, map, ex_map, NULL, &key, ex_ifindex); + if (!obj) + return 0; + + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) + return -EOVERFLOW; + + for (;;) { + /* Check if we still have one more available obj */ + next_obj = devmap_get_next_obj(xdp, map, ex_map, &key, + &next_key, ex_ifindex); + if (!next_obj) + last_one = true; + + if (last_one) { + bq_enqueue(obj->dev, xdpf, dev_rx, obj->xdp_prog); + return 0; + } + + nxdpf = xdpf_clone(xdpf); + if (unlikely(!nxdpf)) { + xdp_return_frame_rx_napi(xdpf); + return -ENOMEM; + } + + bq_enqueue(obj->dev, nxdpf, dev_rx, obj->xdp_prog); + + /* Deal with next obj */ + obj = next_obj; + key = next_key; + } +} + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 3e4b5d9fce78..2139398057cf 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4420,6 +4420,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: if (func_id != BPF_FUNC_redirect_map && + func_id != BPF_FUNC_redirect_map_multi && func_id != BPF_FUNC_map_lookup_elem) goto error; break; @@ -4524,6 +4525,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, map->map_type != BPF_MAP_TYPE_XSKMAP) goto error; break; + case BPF_FUNC_redirect_map_multi: + if (map->map_type != BPF_MAP_TYPE_DEVMAP && + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + goto error; + break; case BPF_FUNC_sk_redirect_map: case BPF_FUNC_msg_redirect_map: case BPF_FUNC_sock_map_update: diff --git a/net/core/filter.c b/net/core/filter.c index 9ab94e90d660..123efaf4ab88 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3924,12 +3924,19 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { }; static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) + struct bpf_map *map, struct xdp_buff *xdp, + struct bpf_map *ex_map, u32 flags) { switch (map->map_type) { case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: - return dev_map_enqueue(fwd, xdp, dev_rx); + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) + return dev_map_enqueue(fwd, xdp, dev_rx); + else + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, flags); case BPF_MAP_TYPE_CPUMAP: return cpu_map_enqueue(fwd, xdp, dev_rx); case BPF_MAP_TYPE_XSKMAP: @@ -3986,12 +3993,14 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); struct bpf_map *map = READ_ONCE(ri->map); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (unlikely(!map)) { @@ -4003,7 +4012,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = dev_xdp_enqueue(fwd, xdp, dev); } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, ri->flags); } if (unlikely(err)) @@ -4017,6 +4026,62 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, } EXPORT_SYMBOL_GPL(xdp_do_redirect); +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, + struct bpf_prog *xdp_prog, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) + +{ + struct bpf_dtab_netdev *dst; + struct sk_buff *nskb; + bool exclude_ingress; + u32 key, next_key, index; + void *fwd; + int err; + + /* Get first key from forward map */ + err = map->ops->map_get_next_key(map, NULL, &key); + if (err) + return err; + + exclude_ingress = !!(flags & BPF_F_EXCLUDE_INGRESS); + + /* When using dev map hash, we could restart the hashtab traversal + * in case the key has been updated/removed in the mean time. + * So we may end up potentially looping due to traversal restarts + * from first elem. + * + * Let's use map's max_entries to limit the loop number. + */ + for (index = 0; index < map->max_entries; index++) { + fwd = __xdp_map_lookup_elem(map, key); + if (fwd) { + dst = (struct bpf_dtab_netdev *)fwd; + if (dev_in_exclude_map(dst, ex_map, + exclude_ingress ? dev->ifindex : 0)) + goto find_next; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -ENOMEM; + + /* Try forword next one no mater the current forward + * succeed or not */ + dev_map_generic_redirect(dst, nskb, xdp_prog); + } + +find_next: + err = map->ops->map_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + consume_skb(skb); + return 0; +} + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, @@ -4024,19 +4089,30 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, struct bpf_map *map) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err = 0; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (map->map_type == BPF_MAP_TYPE_DEVMAP || map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { - struct bpf_dtab_netdev *dst = fwd; + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) { + struct bpf_dtab_netdev *dst = fwd; + + err = dev_map_generic_redirect(dst, skb, xdp_prog); + } else { + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, + ex_map, ri->flags); + } - err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { @@ -4150,6 +4226,36 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { .arg3_type = ARG_ANYTHING, }; +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, + struct bpf_map *, ex_map, u64, flags) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + + /* Limit ex_map type to DEVMAP_HASH to get better performance */ + if (unlikely((ex_map && ex_map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) || + flags & ~BPF_F_EXCLUDE_INGRESS)) + return XDP_ABORTED; + + ri->tgt_index = 0; + /* Set the tgt_value to NULL to distinguish with bpf_xdp_redirect_map */ + ri->tgt_value = NULL; + ri->flags = flags; + ri->ex_map = ex_map; + + WRITE_ONCE(ri->map, map); + + return XDP_REDIRECT; +} + +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { + .func = bpf_xdp_redirect_map_multi, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_CONST_MAP_PTR, + .arg2_type = ARG_CONST_MAP_PTR_OR_NULL, + .arg3_type = ARG_ANYTHING, +}; + static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, unsigned long off, unsigned long len) { @@ -7248,6 +7354,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_redirect_proto; case BPF_FUNC_redirect_map: return &bpf_xdp_redirect_map_proto; + case BPF_FUNC_redirect_map_multi: + return &bpf_xdp_redirect_map_multi_proto; case BPF_FUNC_xdp_adjust_tail: return &bpf_xdp_adjust_tail_proto; case BPF_FUNC_fib_lookup: diff --git a/net/core/xdp.c b/net/core/xdp.c index 3a8c9ab4ecbe..6d86af029dc5 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -513,3 +513,32 @@ void xdp_warn(const char *msg, const char *func, const int line) WARN(1, "XDP_WARN: %s(line:%d): %s\n", func, line, msg); }; EXPORT_SYMBOL_GPL(xdp_warn); + +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) +{ + unsigned int headroom, totalsize; + struct xdp_frame *nxdpf; + struct page *page; + void *addr; + + headroom = xdpf->headroom + sizeof(*xdpf); + totalsize = headroom + xdpf->len; + + if (unlikely(totalsize > PAGE_SIZE)) + return NULL; + page = dev_alloc_page(); + if (!page) + return NULL; + addr = page_to_virt(page); + + memcpy(addr, xdpf, totalsize); + + nxdpf = addr; + nxdpf->data = addr + headroom; + nxdpf->frame_sz = PAGE_SIZE; + nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0; + nxdpf->mem.id = 0; + + return nxdpf; +} +EXPORT_SYMBOL_GPL(xdpf_clone); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index a1ad32456f89..ecf5d117b96a 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3830,6 +3830,27 @@ union bpf_attr { * Return * A pointer to a struct socket on success or NULL if the file is * not a socket. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3995,6 +4016,7 @@ union bpf_attr { FN(ktime_get_coarse_ns), \ FN(ima_inode_hash), \ FN(sock_from_file), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -4171,6 +4193,11 @@ enum { BPF_F_BPRM_SECUREEXEC = (1ULL << 0), }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* RE: [PATCHv14 bpf-next 3/6] xdp: add a new helper for dev map multicast support 2021-01-14 14:23 ` [PATCHv14 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2021-01-18 0:10 ` John Fastabend 2021-01-18 8:44 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: John Fastabend @ 2021-01-18 0:10 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu Hangbin Liu wrote: > This patch is for xdp multicast support. which has been discussed > before[0], The goal is to be able to implement an OVS-like data plane in > XDP, i.e., a software switch that can forward XDP frames to multiple ports. > > To achieve this, an application needs to specify a group of interfaces > to forward a packet to. It is also common to want to exclude one or more > physical interfaces from the forwarding operation - e.g., to forward a > packet to all interfaces in the multicast group except the interface it > arrived on. While this could be done simply by adding more groups, this > quickly leads to a combinatorial explosion in the number of groups an > application has to maintain. > > To avoid the combinatorial explosion, we propose to include the ability > to specify an "exclude group" as part of the forwarding operation. This > needs to be a group (instead of just a single port index), because a > physical interface can be part of a logical grouping, such as a bond > device. > > Thus, the logical forwarding operation becomes a "set difference" > operation, i.e. "forward to all ports in group A that are not also in > group B". This series implements such an operation using device maps to > represent the groups. This means that the XDP program specifies two > device maps, one containing the list of netdevs to redirect to, and the > other containing the exclude list. > > To achieve this, I re-implement a new helper bpf_redirect_map_multi() > to accept two maps, the forwarding map and exclude map. The forwarding > map could be DEVMAP or DEVMAP_HASH, but the exclude map *must* be > DEVMAP_HASH to get better performace. If user don't want to use exclude > map and just want simply stop redirecting back to ingress device, they > can use flag BPF_F_EXCLUDE_INGRESS. > > As both bpf_xdp_redirect_map() and this new helpers are using struct > bpf_redirect_info, I add a new ex_map and set tgt_value to NULL in the > new helper to make a difference with bpf_xdp_redirect_map(). > > Also I keep the general data path in net/core/filter.c, the native data > path in kernel/bpf/devmap.c so we can use direct calls to get better > performace. [...] > diff --git a/include/net/xdp.h b/include/net/xdp.h > index 0cf3976ce77c..0e6468cd0ab9 100644 > --- a/include/net/xdp.h > +++ b/include/net/xdp.h > @@ -164,6 +164,7 @@ void xdp_warn(const char *msg, const char *func, const int line); > #define XDP_WARN(msg) xdp_warn(msg, __func__, __LINE__) > > struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); > +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); > > static inline > void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp) > diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h > index a1ad32456f89..ecf5d117b96a 100644 > --- a/include/uapi/linux/bpf.h > +++ b/include/uapi/linux/bpf.h > @@ -3830,6 +3830,27 @@ union bpf_attr { > * Return > * A pointer to a struct socket on success or NULL if the file is > * not a socket. > + * > + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) > + * Description > + * This is a multicast implementation for XDP redirect. It will > + * redirect the packet to ALL the interfaces in *map*, but > + * exclude the interfaces in *ex_map*. > + * > + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or > + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be > + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. Would be good to add a note ex_map _must_ be keyed by ifindex for the helper to work. Its the obvious way to key a hashmap, but not required iirc. > + * > + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, > + * which additionally excludes the current ingress device. > + * > + * See also bpf_redirect_map() as a unicast implementation, > + * which supports redirecting packet to a specific ifindex > + * in the map. As both helpers use struct bpf_redirect_info > + * to store the redirect info, we will use a a NULL tgt_value > + * to distinguish multicast and unicast redirecting. > + * Return > + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. > */ [...] > + > +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, > + struct bpf_map *map, struct bpf_map *ex_map, > + u32 flags) > +{ > + struct bpf_dtab_netdev *obj = NULL, *next_obj = NULL; > + struct xdp_frame *xdpf, *nxdpf; > + bool last_one = false; > + int ex_ifindex; > + u32 key, next_key; > + > + ex_ifindex = flags & BPF_F_EXCLUDE_INGRESS ? dev_rx->ifindex : 0; > + > + /* Find first available obj */ > + obj = devmap_get_next_obj(xdp, map, ex_map, NULL, &key, ex_ifindex); > + if (!obj) > + return 0; > + > + xdpf = xdp_convert_buff_to_frame(xdp); > + if (unlikely(!xdpf)) > + return -EOVERFLOW; > + > + for (;;) { > + /* Check if we still have one more available obj */ > + next_obj = devmap_get_next_obj(xdp, map, ex_map, &key, > + &next_key, ex_ifindex); > + if (!next_obj) > + last_one = true; > + > + if (last_one) { > + bq_enqueue(obj->dev, xdpf, dev_rx, obj->xdp_prog); > + return 0; > + } Just collapse above to if (!next_obj) { bq_enqueue() return } 'last_one' is a bit pointless here. > + > + nxdpf = xdpf_clone(xdpf); > + if (unlikely(!nxdpf)) { > + xdp_return_frame_rx_napi(xdpf); > + return -ENOMEM; > + } > + > + bq_enqueue(obj->dev, nxdpf, dev_rx, obj->xdp_prog); > + > + /* Deal with next obj */ > + obj = next_obj; > + key = next_key; > + } > +} > + > int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, > struct bpf_prog *xdp_prog) > { > diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c > index 3e4b5d9fce78..2139398057cf 100644 > --- a/kernel/bpf/verifier.c > +++ b/kernel/bpf/verifier.c > @@ -4420,6 +4420,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, > case BPF_MAP_TYPE_DEVMAP: > case BPF_MAP_TYPE_DEVMAP_HASH: > if (func_id != BPF_FUNC_redirect_map && > + func_id != BPF_FUNC_redirect_map_multi && > func_id != BPF_FUNC_map_lookup_elem) > goto error; > break; > @@ -4524,6 +4525,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, > map->map_type != BPF_MAP_TYPE_XSKMAP) > goto error; > break; > + case BPF_FUNC_redirect_map_multi: > + if (map->map_type != BPF_MAP_TYPE_DEVMAP && > + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) > + goto error; > + break; > case BPF_FUNC_sk_redirect_map: > case BPF_FUNC_msg_redirect_map: > case BPF_FUNC_sock_map_update: > diff --git a/net/core/filter.c b/net/core/filter.c > index 9ab94e90d660..123efaf4ab88 100644 > --- a/net/core/filter.c > +++ b/net/core/filter.c > @@ -3924,12 +3924,19 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { > }; > > static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, > - struct bpf_map *map, struct xdp_buff *xdp) > + struct bpf_map *map, struct xdp_buff *xdp, > + struct bpf_map *ex_map, u32 flags) > { > switch (map->map_type) { > case BPF_MAP_TYPE_DEVMAP: > case BPF_MAP_TYPE_DEVMAP_HASH: > - return dev_map_enqueue(fwd, xdp, dev_rx); > + /* We use a NULL fwd value to distinguish multicast > + * and unicast forwarding > + */ > + if (fwd) > + return dev_map_enqueue(fwd, xdp, dev_rx); > + else > + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, flags); > case BPF_MAP_TYPE_CPUMAP: > return cpu_map_enqueue(fwd, xdp, dev_rx); > case BPF_MAP_TYPE_XSKMAP: > @@ -3986,12 +3993,14 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, > { > struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); > struct bpf_map *map = READ_ONCE(ri->map); > + struct bpf_map *ex_map = ri->ex_map; READ_ONCE(ri->ex_map)? > u32 index = ri->tgt_index; > void *fwd = ri->tgt_value; > int err; > > ri->tgt_index = 0; > ri->tgt_value = NULL; > + ri->ex_map = NULL; WRITE_ONCE(ri->ex_map)? > WRITE_ONCE(ri->map, NULL); So we needed write_once, read_once pairs for ri->map do we also need them in the ex_map case? > > if (unlikely(!map)) { > @@ -4003,7 +4012,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, > > err = dev_xdp_enqueue(fwd, xdp, dev); > } else { > - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); > + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, ri->flags); > } > > if (unlikely(err)) > @@ -4017,6 +4026,62 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, > } > EXPORT_SYMBOL_GPL(xdp_do_redirect); [...] > +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, > + struct bpf_map *, ex_map, u64, flags) > +{ > + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); > + > + /* Limit ex_map type to DEVMAP_HASH to get better performance */ > + if (unlikely((ex_map && ex_map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) || > + flags & ~BPF_F_EXCLUDE_INGRESS)) > + return XDP_ABORTED; > + > + ri->tgt_index = 0; > + /* Set the tgt_value to NULL to distinguish with bpf_xdp_redirect_map */ > + ri->tgt_value = NULL; > + ri->flags = flags; > + ri->ex_map = ex_map; WRITE_ONCE? > + > + WRITE_ONCE(ri->map, map); > + > + return XDP_REDIRECT; > +} > + > +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { > + .func = bpf_xdp_redirect_map_multi, > + .gpl_only = false, > + .ret_type = RET_INTEGER, > + .arg1_type = ARG_CONST_MAP_PTR, > + .arg2_type = ARG_CONST_MAP_PTR_OR_NULL, > + .arg3_type = ARG_ANYTHING, > +}; > + Thanks, John ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv14 bpf-next 3/6] xdp: add a new helper for dev map multicast support 2021-01-18 0:10 ` John Fastabend @ 2021-01-18 8:44 ` Hangbin Liu 2021-01-18 10:47 ` Toke Høiland-Jørgensen 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2021-01-18 8:44 UTC (permalink / raw) To: John Fastabend Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov Hi John, Thanks for the reviewing. On Sun, Jan 17, 2021 at 04:10:40PM -0800, John Fastabend wrote: > > + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or > > + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be > > + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. > > Would be good to add a note ex_map _must_ be keyed by ifindex for the > helper to work. Its the obvious way to key a hashmap, but not required > iirc. OK, I will. > > + if (!next_obj) > > + last_one = true; > > + > > + if (last_one) { > > + bq_enqueue(obj->dev, xdpf, dev_rx, obj->xdp_prog); > > + return 0; > > + } > > Just collapse above to > > if (!next_obj) { > bq_enqueue() > return > } > > 'last_one' is a bit pointless here. Yes, thanks. > > @@ -3986,12 +3993,14 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, > > { > > struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); > > struct bpf_map *map = READ_ONCE(ri->map); > > + struct bpf_map *ex_map = ri->ex_map; > > READ_ONCE(ri->ex_map)? > > > u32 index = ri->tgt_index; > > void *fwd = ri->tgt_value; > > int err; > > > > ri->tgt_index = 0; > > ri->tgt_value = NULL; > > + ri->ex_map = NULL; > > WRITE_ONCE(ri->ex_map)? > > > WRITE_ONCE(ri->map, NULL); > > So we needed write_once, read_once pairs for ri->map do we also need them in > the ex_map case? Toke said this is no need for this read/write_once as there is already one. https://lore.kernel.org/bpf/87r1wd2bqu.fsf@toke.dk/ Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv14 bpf-next 3/6] xdp: add a new helper for dev map multicast support 2021-01-18 8:44 ` Hangbin Liu @ 2021-01-18 10:47 ` Toke Høiland-Jørgensen 2021-01-18 15:14 ` John Fastabend 0 siblings, 1 reply; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2021-01-18 10:47 UTC (permalink / raw) To: Hangbin Liu, John Fastabend Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov Hangbin Liu <liuhangbin@gmail.com> writes: > Hi John, > > Thanks for the reviewing. > > On Sun, Jan 17, 2021 at 04:10:40PM -0800, John Fastabend wrote: >> > + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or >> > + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be >> > + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. >> >> Would be good to add a note ex_map _must_ be keyed by ifindex for the >> helper to work. Its the obvious way to key a hashmap, but not required >> iirc. > > OK, I will. >> > + if (!next_obj) >> > + last_one = true; >> > + >> > + if (last_one) { >> > + bq_enqueue(obj->dev, xdpf, dev_rx, obj->xdp_prog); >> > + return 0; >> > + } >> >> Just collapse above to >> >> if (!next_obj) { >> bq_enqueue() >> return >> } >> >> 'last_one' is a bit pointless here. > > Yes, thanks. > >> > @@ -3986,12 +3993,14 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, >> > { >> > struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); >> > struct bpf_map *map = READ_ONCE(ri->map); >> > + struct bpf_map *ex_map = ri->ex_map; >> >> READ_ONCE(ri->ex_map)? >> >> > u32 index = ri->tgt_index; >> > void *fwd = ri->tgt_value; >> > int err; >> > >> > ri->tgt_index = 0; >> > ri->tgt_value = NULL; >> > + ri->ex_map = NULL; >> >> WRITE_ONCE(ri->ex_map)? >> >> > WRITE_ONCE(ri->map, NULL); >> >> So we needed write_once, read_once pairs for ri->map do we also need them in >> the ex_map case? > > Toke said this is no need for this read/write_once as there is already one. > > https://lore.kernel.org/bpf/87r1wd2bqu.fsf@toke.dk/ And then I corrected that after I figured out the real reason :) https://lore.kernel.org/bpf/878si2h3sb.fsf@toke.dk/ - Quote: > The READ_ONCE() is not needed because the ex_map field is only ever read > from or written to by the CPU owning the per-cpu pointer. Whereas the > 'map' field is manipulated by remote CPUs in bpf_clear_redirect_map(). > So you need neither READ_ONCE() nor WRITE_ONCE() on ex_map, just like > there are none on tgt_index and tgt_value. -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv14 bpf-next 3/6] xdp: add a new helper for dev map multicast support 2021-01-18 10:47 ` Toke Høiland-Jørgensen @ 2021-01-18 15:14 ` John Fastabend 0 siblings, 0 replies; 219+ messages in thread From: John Fastabend @ 2021-01-18 15:14 UTC (permalink / raw) To: Toke Høiland-Jørgensen, Hangbin Liu, John Fastabend Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov Toke Høiland-Jørgensen wrote: > Hangbin Liu <liuhangbin@gmail.com> writes: > > > Hi John, > > > > Thanks for the reviewing. > > > > On Sun, Jan 17, 2021 at 04:10:40PM -0800, John Fastabend wrote: > >> > + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or > >> > + * BPF_MAP_TYPE_DEVMAP_HASH. But the *ex_map* must be > >> > + * BPF_MAP_TYPE_DEVMAP_HASH to get better performance. > >> > >> Would be good to add a note ex_map _must_ be keyed by ifindex for the > >> helper to work. Its the obvious way to key a hashmap, but not required > >> iirc. > > > > OK, I will. [...] > >> WRITE_ONCE(ri->ex_map)? > >> > >> > WRITE_ONCE(ri->map, NULL); > >> > >> So we needed write_once, read_once pairs for ri->map do we also need them in > >> the ex_map case? > > > > Toke said this is no need for this read/write_once as there is already one. > > > > https://lore.kernel.org/bpf/87r1wd2bqu.fsf@toke.dk/ > > And then I corrected that after I figured out the real reason :) > > https://lore.kernel.org/bpf/878si2h3sb.fsf@toke.dk/ - Quote: > > > The READ_ONCE() is not needed because the ex_map field is only ever read > > from or written to by the CPU owning the per-cpu pointer. Whereas the > > 'map' field is manipulated by remote CPUs in bpf_clear_redirect_map(). > > So you need neither READ_ONCE() nor WRITE_ONCE() on ex_map, just like > > there are none on tgt_index and tgt_value. > > -Toke > Hi Hangbin, please add a comment above that code block to remind us why the READ_ONCE/WRITE_ONCE is not needed or add it in the commit message so we don't lose it. It seems we've hashed it over already, but I forgot after the holidays/break so presumably I'll forget next time I read this code as well and commit-msg or comment will help. Thanks, John ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv14 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test 2021-01-14 14:23 ` [PATCHv14 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (2 preceding siblings ...) 2021-01-14 14:23 ` [PATCHv14 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2021-01-14 14:23 ` Hangbin Liu 2021-01-14 14:23 ` [PATCHv14 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu ` (2 subsequent siblings) 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-14 14:23 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu This is a sample for xdp multicast. In the sample we could forward all packets between given interfaces. There is also an option -X that could enable 2nd xdp_prog on egress interface. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v13-v14: no update, only rebase the code v12: add devmap xdp_prog on egress support v10-v11: no update v9: use NULL directly for arg2 and redefine the maps with btf format v5: add a null_map as we have strict the arg2 to ARG_CONST_MAP_PTR. Move the testing part to bpf selftest in next patch. v4: no update. v3: add rxcnt map to show the packet transmit speed. v2: no update. --- samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 96 +++++++ samples/bpf/xdp_redirect_map_multi_user.c | 301 ++++++++++++++++++++++ 3 files changed, 400 insertions(+) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 26fc96ca619e..200029fcf53c 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map tprogs-y += per_socket_stats_example tprogs-y += xdp_redirect tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_map_multi tprogs-y += xdp_redirect_cpu tprogs-y += xdp_monitor tprogs-y += xdp_rxq_info @@ -99,6 +100,7 @@ test_map_in_map-objs := test_map_in_map_user.o per_socket_stats_example-objs := cookie_uid_helper_example.o xdp_redirect-objs := xdp_redirect_user.o xdp_redirect_map-objs := xdp_redirect_map_user.o +xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o xdp_redirect_cpu-objs := xdp_redirect_cpu_user.o xdp_monitor-objs := xdp_monitor_user.o xdp_rxq_info-objs := xdp_rxq_info_user.o @@ -160,6 +162,7 @@ always-y += tcp_tos_reflect_kern.o always-y += tcp_dumpstats_kern.o always-y += xdp_redirect_kern.o always-y += xdp_redirect_map_kern.o +always-y += xdp_redirect_map_multi_kern.o always-y += xdp_redirect_cpu_kern.o always-y += xdp_monitor_kern.o always-y += xdp_rxq_info_kern.o diff --git a/samples/bpf/xdp_redirect_map_multi_kern.c b/samples/bpf/xdp_redirect_map_multi_kern.c new file mode 100644 index 000000000000..0c63aace9bd2 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_kern.c @@ -0,0 +1,96 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <uapi/linux/bpf.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/ip.h> +#include <linux/ipv6.h> +#include <bpf/bpf_helpers.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 32); +} forward_map_general SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(struct bpf_devmap_val)); + __uint(max_entries, 32); +} forward_map_native SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __type(key, u32); + __type(value, long); + __uint(max_entries, 1); +} rxcnt SEC(".maps"); + +/* map to stroe egress interfaces mac addresses, set the + * max_entries to 1 and extend it in user sapce prog. + */ +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, u32); + __type(value, __be64); + __uint(max_entries, 1); +} mac_map SEC(".maps"); + +static int xdp_redirect_map(struct xdp_md *ctx, void *forward_map) +{ + long *value; + u32 key = 0; + + /* count packet in global counter */ + value = bpf_map_lookup_elem(&rxcnt, &key); + if (value) + *value += 1; + + return bpf_redirect_map_multi(forward_map, NULL, BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_redirect_general") +int xdp_redirect_map_general(struct xdp_md *ctx) +{ + return xdp_redirect_map(ctx, &forward_map_general); +} + +SEC("xdp_redirect_native") +int xdp_redirect_map_native(struct xdp_md *ctx) +{ + return xdp_redirect_map(ctx, &forward_map_native); +} + +SEC("xdp_devmap/map_prog") +int xdp_devmap_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + u32 key = ctx->egress_ifindex; + struct ethhdr *eth = data; + __be64 *mac; + u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + mac = bpf_map_lookup_elem(&mac_map, &key); + if (mac) + __builtin_memcpy(eth->h_source, mac, ETH_ALEN); + + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/samples/bpf/xdp_redirect_map_multi_user.c b/samples/bpf/xdp_redirect_map_multi_user.c new file mode 100644 index 000000000000..67ffc294567c --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_user.c @@ -0,0 +1,301 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> +#include <sys/resource.h> +#include <sys/ioctl.h> +#include <sys/types.h> +#include <sys/socket.h> +#include <netinet/in.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; +static int rxcnt_map_fd; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void poll_stats(int interval) +{ + unsigned int nr_cpus = bpf_num_possible_cpus(); + __u64 values[nr_cpus], prev[nr_cpus]; + + memset(prev, 0, sizeof(prev)); + + while (1) { + __u64 sum = 0; + __u32 key = 0; + int i; + + sleep(interval); + assert(bpf_map_lookup_elem(rxcnt_map_fd, &key, values) == 0); + for (i = 0; i < nr_cpus; i++) + sum += (values[i] - prev[i]); + if (sum) + printf("Forwarding %10llu pkt/s\n", sum / interval); + memcpy(prev, values, sizeof(values)); + } +} + +static int get_mac_addr(unsigned int ifindex, void *mac_addr) +{ + char ifname[IF_NAMESIZE]; + struct ifreq ifr; + int fd, ret = -1; + + fd = socket(AF_INET, SOCK_DGRAM, 0); + if (fd < 0) + return ret; + + if (!if_indextoname(ifindex, ifname)) + goto err_out; + + strcpy(ifr.ifr_name, ifname); + + if (ioctl(fd, SIOCGIFHWADDR, &ifr) != 0) + goto err_out; + + memcpy(mac_addr, ifr.ifr_hwaddr.sa_data, 6 * sizeof(char)); + ret = 0; + +err_out: + close(fd); + return ret; +} + +static int update_mac_map(struct bpf_object *obj) +{ + int i, ret = -1, mac_map_fd; + unsigned char mac_addr[6]; + unsigned int ifindex; + + mac_map_fd = bpf_object__find_map_fd_by_name(obj, "mac_map"); + if (mac_map_fd < 0) { + printf("find mac map fd failed\n"); + return ret; + } + + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + ret = get_mac_addr(ifindex, mac_addr); + if (ret < 0) { + printf("get interface %d mac failed\n", ifindex); + return ret; + } + + ret = bpf_map_update_elem(mac_map_fd, &ifindex, mac_addr, 0); + if (ret) { + perror("bpf_update_elem mac_map_fd"); + return ret; + } + } + + return 0; +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n" + " -X load xdp program on egress\n", + prog); +} + +int main(int argc, char **argv) +{ + int i, ret, opt, forward_map_fd, max_ifindex = 0; + struct bpf_program *ingress_prog, *egress_prog; + int ingress_prog_fd, egress_prog_fd = -1; + struct bpf_devmap_val devmap_val; + bool attach_egress_prog = false; + char ifname[IF_NAMESIZE]; + struct bpf_map *mac_map; + struct bpf_object *obj; + unsigned int ifindex; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNFX")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + case 'X': + attach_egress_prog = true; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) { + xdp_flags |= XDP_FLAGS_DRV_MODE; + } else if (attach_egress_prog) { + printf("Load xdp program on egress with SKB mode not supported yet\n"); + return 1; + } + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + + /* Find the largest index number */ + if (ifaces[i] > max_ifindex) + max_ifindex = ifaces[i]; + + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + + obj = bpf_object__open(filename); + if (libbpf_get_error(obj)) { + printf("ERROR: opening BPF object file failed\n"); + obj = NULL; + goto err_out; + } + + /* Reset the map size to max ifindex + 1 */ + if (attach_egress_prog) { + mac_map = bpf_object__find_map_by_name(obj, "mac_map"); + ret = bpf_map__resize(mac_map, max_ifindex + 1); + if (ret < 0) { + printf("ERROR: reset mac map size failed\n"); + goto err_out; + } + } + + /* load BPF program */ + if (bpf_object__load(obj)) { + printf("ERROR: loading BPF object file failed\n"); + goto err_out; + } + + if (xdp_flags & XDP_FLAGS_SKB_MODE) { + ingress_prog = bpf_object__find_program_by_title(obj, "xdp_redirect_general"); + forward_map_fd = bpf_object__find_map_fd_by_name(obj, "forward_map_general"); + } else { + ingress_prog = bpf_object__find_program_by_title(obj, "xdp_redirect_native"); + forward_map_fd = bpf_object__find_map_fd_by_name(obj, "forward_map_native"); + } + if (!ingress_prog || forward_map_fd < 0) { + printf("finding ingress_prog/forward_map in obj file failed\n"); + goto err_out; + } + + ingress_prog_fd = bpf_program__fd(ingress_prog); + if (ingress_prog_fd < 0) { + printf("find ingress_prog fd failed\n"); + goto err_out; + } + + rxcnt_map_fd = bpf_object__find_map_fd_by_name(obj, "rxcnt"); + if (rxcnt_map_fd < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + goto err_out; + } + + if (attach_egress_prog) { + /* Update mac_map with all egress interfaces' mac addr */ + if (update_mac_map(obj) < 0) { + printf("Error: update mac map failed"); + goto err_out; + } + + /* Find egress prog fd */ + egress_prog = bpf_object__find_program_by_title(obj, "xdp_devmap/map_prog"); + if (!egress_prog) { + printf("finding egress_prog in obj file failed\n"); + goto err_out; + } + egress_prog_fd = bpf_program__fd(egress_prog); + if (egress_prog_fd < 0) { + printf("find egress_prog fd failed\n"); + goto err_out; + } + } + + /* Remove attached program when program is interrupted or killed */ + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, ingress_prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + /* Add all the interfaces to forward group and attach + * egress devmap programe if exist */ + devmap_val.ifindex = ifindex; + devmap_val.bpf_prog.fd = egress_prog_fd; + ret = bpf_map_update_elem(forward_map_fd, &ifindex, &devmap_val, 0); + if (ret) { + perror("bpf_map_update_elem forward_map"); + goto err_out; + } + } + + poll_stats(2); + + return 0; + +err_out: + return 1; +} -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv14 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL 2021-01-14 14:23 ` [PATCHv14 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (3 preceding siblings ...) 2021-01-14 14:23 ` [PATCHv14 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu @ 2021-01-14 14:23 ` Hangbin Liu 2021-01-14 14:23 ` [PATCHv14 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2021-01-20 2:25 ` [PATCHv15 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-14 14:23 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu Use helper bpf_redirect_map() and bpf_redirect_map_multi() to test bpf arg ARG_CONST_MAP_PTR and ARG_CONST_MAP_PTR_OR_NULL. Make sure the map arg could be verified correctly when it is NULL or valid map pointer. Add devmap and devmap_hash in struct bpf_test due to bpf_redirect_{map, map_multi} limit. Test result: ]# ./test_verifier 713 716 #713/p ARG_CONST_MAP_PTR: null pointer OK #714/p ARG_CONST_MAP_PTR: valid map pointer OK #715/p ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map OK #716/p ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map OK Summary: 4 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- tools/testing/selftests/bpf/test_verifier.c | 22 +++++- .../testing/selftests/bpf/verifier/map_ptr.c | 70 +++++++++++++++++++ 2 files changed, 91 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c index 777a81404fdb..17eb3958ce6d 100644 --- a/tools/testing/selftests/bpf/test_verifier.c +++ b/tools/testing/selftests/bpf/test_verifier.c @@ -50,7 +50,7 @@ #define MAX_INSNS BPF_MAXINSNS #define MAX_TEST_INSNS 1000000 #define MAX_FIXUPS 8 -#define MAX_NR_MAPS 20 +#define MAX_NR_MAPS 22 #define MAX_TEST_RUNS 8 #define POINTER_VALUE 0xcafe4all #define TEST_DATA_LEN 64 @@ -87,6 +87,8 @@ struct bpf_test { int fixup_sk_storage_map[MAX_FIXUPS]; int fixup_map_event_output[MAX_FIXUPS]; int fixup_map_reuseport_array[MAX_FIXUPS]; + int fixup_map_devmap[MAX_FIXUPS]; + int fixup_map_devmap_hash[MAX_FIXUPS]; const char *errstr; const char *errstr_unpriv; uint32_t insn_processed; @@ -640,6 +642,8 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, int *fixup_sk_storage_map = test->fixup_sk_storage_map; int *fixup_map_event_output = test->fixup_map_event_output; int *fixup_map_reuseport_array = test->fixup_map_reuseport_array; + int *fixup_map_devmap = test->fixup_map_devmap; + int *fixup_map_devmap_hash = test->fixup_map_devmap_hash; if (test->fill_helper) { test->fill_insns = calloc(MAX_TEST_INSNS, sizeof(struct bpf_insn)); @@ -817,6 +821,22 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, fixup_map_reuseport_array++; } while (*fixup_map_reuseport_array); } + if (*fixup_map_devmap) { + map_fds[20] = __create_map(BPF_MAP_TYPE_DEVMAP, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap].imm = map_fds[20]; + fixup_map_devmap++; + } while (*fixup_map_devmap); + } + if (*fixup_map_devmap_hash) { + map_fds[21] = __create_map(BPF_MAP_TYPE_DEVMAP_HASH, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap_hash].imm = map_fds[21]; + fixup_map_devmap_hash++; + } while (*fixup_map_devmap_hash); + } } struct libcap { diff --git a/tools/testing/selftests/bpf/verifier/map_ptr.c b/tools/testing/selftests/bpf/verifier/map_ptr.c index b117bdd3806d..1a532198c9c1 100644 --- a/tools/testing/selftests/bpf/verifier/map_ptr.c +++ b/tools/testing/selftests/bpf/verifier/map_ptr.c @@ -93,3 +93,73 @@ .fixup_map_hash_16b = { 4 }, .result = ACCEPT, }, +{ + "ARG_CONST_MAP_PTR: null pointer", + .insns = { + /* bpf_redirect_map arg1 (map) */ + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .result = REJECT, + .prog_type = BPF_PROG_TYPE_XDP, + .errstr = "R1 type=inv expected=map_ptr", +}, +{ + "ARG_CONST_MAP_PTR: valid map pointer", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg1 (map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_LD_MAP_FD(BPF_REG_2, 1), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .fixup_map_devmap_hash = { 3 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv14 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test 2021-01-14 14:23 ` [PATCHv14 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (4 preceding siblings ...) 2021-01-14 14:23 ` [PATCHv14 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2021-01-14 14:23 ` Hangbin Liu 2021-01-20 2:25 ` [PATCHv15 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-14 14:23 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Hangbin Liu Add a bpf selftest for new helper xdp_redirect_map_multi(). In this test we have 3 forward groups and 1 exclude group. The test will redirect each interface's packets to all the interfaces in the forward group, and exclude the interface in exclude map. We will also test both DEVMAP and DEVMAP_HASH with xdp generic and drv. For more test details, you can find it in the test script. Here is the test result. ]# ./test_xdp_redirect_multi.sh Pass: xdpgeneric arp ns1-2 Pass: xdpgeneric arp ns1-3 Pass: xdpgeneric arp ns1-4 Pass: xdpgeneric ping ns1-2 Pass: xdpgeneric ping ns1-3 Pass: xdpgeneric ping ns1-4 Pass: xdpgeneric ping6 ns2-1 Pass: xdpgeneric ping6 ns2-3 Pass: xdpgeneric ping6 ns2-4 Pass: xdpdrv arp ns1-2 Pass: xdpdrv arp ns1-3 Pass: xdpdrv arp ns1-4 Pass: xdpdrv ping ns1-2 Pass: xdpdrv ping ns1-3 Pass: xdpdrv ping ns1-4 Pass: xdpdrv ping6 ns2-1 Pass: xdpdrv ping6 ns2-3 Pass: xdpdrv ping6 ns2-4 Pass: xdpegress mac ns1-2 Pass: xdpegress mac ns1-3 Pass: xdpegress mac ns1-4 Pass: xdpegress ping ns1-2 Pass: xdpegress ping ns1-3 Pass: xdpegress ping ns1-4 Summary: PASS 24, FAIL 0 Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v14: no update, only rebase the code v13: remove setrlimit v12: add devmap prog test on egress v9: use NULL directly for arg2 and redefine the maps with btf format --- tools/testing/selftests/bpf/Makefile | 3 +- .../bpf/progs/xdp_redirect_multi_kern.c | 120 +++++++++ .../selftests/bpf/test_xdp_redirect_multi.sh | 208 +++++++++++++++ .../selftests/bpf/xdp_redirect_multi.c | 252 ++++++++++++++++++ 4 files changed, 582 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 7f8667ad113e..208dc56e3672 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -52,6 +52,7 @@ TEST_FILES = xsk_prereqs.sh \ # Order correspond to 'make run_tests' order TEST_PROGS := test_kmod.sh \ test_xdp_redirect.sh \ + test_xdp_redirect_multi.sh \ test_xdp_meta.sh \ test_xdp_veth.sh \ test_offload.py \ @@ -81,7 +82,7 @@ TEST_PROGS_EXTENDED := with_addr.sh \ TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ test_lirc_mode2_user xdping test_cpp runqslower bench bpf_testmod.ko \ - xdpxceiver + xdpxceiver xdp_redirect_multi TEST_CUSTOM_PROGS = $(OUTPUT)/urandom_read diff --git a/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c new file mode 100644 index 000000000000..a9785b28175a --- /dev/null +++ b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c @@ -0,0 +1,120 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define KBUILD_MODNAME "foo" +#include <string.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ip.h> +#include <linux/ipv6.h> + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_endian.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v4 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v6 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_all SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(struct bpf_devmap_val)); + __uint(max_entries, 128); +} forward_map_egress SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} exclude_map SEC(".maps"); + +/* map to stroe egress interfaces mac addresses */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, __u32); + __type(value, __be64); + __uint(max_entries, 128); +} mac_map SEC(".maps"); + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + struct ethhdr *eth = data; + __u16 h_proto; + __u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + h_proto = eth->h_proto; + + if (h_proto == bpf_htons(ETH_P_IP)) + return bpf_redirect_map_multi(&forward_map_v4, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else if (h_proto == bpf_htons(ETH_P_IPV6)) + return bpf_redirect_map_multi(&forward_map_v6, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else + return bpf_redirect_map_multi(&forward_map_all, NULL, + BPF_F_EXCLUDE_INGRESS); +} + +/* The following 2 progs are for 2nd devmap prog testing */ +SEC("xdp_redirect_map_ingress") +int xdp_redirect_map_all_prog(struct xdp_md *ctx) +{ + return bpf_redirect_map_multi(&forward_map_egress, NULL, BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_devmap/map_prog") +int xdp_devmap_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + __u32 key = ctx->egress_ifindex; + struct ethhdr *eth = data; + __u64 nh_off; + __be64 *mac; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + mac = bpf_map_lookup_elem(&mac_map, &key); + if (mac) + __builtin_memcpy(eth->h_source, mac, ETH_ALEN); + + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh new file mode 100755 index 000000000000..6503751fdca5 --- /dev/null +++ b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh @@ -0,0 +1,208 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Test topology: +# - - - - - - - - - - - - - - - - - - - - - - - - - +# | veth1 veth2 veth3 veth4 | ... init net +# - -| - - - - - - | - - - - - - | - - - - - - | - - +# --------- --------- --------- --------- +# | veth0 | | veth0 | | veth0 | | veth0 | ... +# --------- --------- --------- --------- +# ns1 ns2 ns3 ns4 +# +# Forward maps: +# Forward map_all has interfaces: veth1, veth2, veth3, veth4, ... (All traffic except IPv4, IPv6) +# Forward map_v4 has interfaces: veth1, veth3, veth4, ... (For IPv4 traffic only) +# Forward map_v6 has interfaces: veth2, veth3, veth4, ... (For IPv6 traffic only) +# Forward map_egress has all interfaces and redirect all pkts +# Exclude Groups: +# Exclude map: veth3 (assume ns3 is in black list) +# Map type: +# map_v4 use DEVMAP, others use DEVMAP_HASH +# +# Test modules: +# XDP modes: generic, native, native + egress_prog +# +# Test cases: +# ARP(we didn't block ARP for ns3): +# ns1 -> gw: ns2, ns3, ns4 should receive the arp request +# IPv4: +# ns1 -> ns2 (block), ns1 -> ns3 (block), ns1 -> ns4 (pass) +# IPv6 +# ns2 -> ns1 (block), ns2 -> ns3 (block), ns2 -> ns4 (pass) +# egress_prog: +# all ping test should pass, the src mac should be egress interface's mac +# + + +# netns numbers +NUM=4 +IFACES="" +DRV_MODE="xdpgeneric xdpdrv xdpegress" +PASS=0 +FAIL=0 + +test_pass() +{ + echo "Pass: $@" + PASS=$((PASS + 1)) +} + +test_fail() +{ + echo "fail: $@" + FAIL=$((FAIL + 1)) +} + +clean_up() +{ + for i in $(seq $NUM); do + ip link del veth$i 2> /dev/null + ip netns del ns$i 2> /dev/null + done +} + +# Kselftest framework requirement - SKIP code is 4. +check_env() +{ + ip link set dev lo xdpgeneric off &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without the ip xdpgeneric support" + exit 4 + fi + + which tcpdump &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without tcpdump" + exit 4 + fi +} + +setup_ns() +{ + local mode=$1 + IFACES="" + + if [ "$mode" = "xdpegress" ]; then + mode="xdpdrv" + fi + + for i in $(seq $NUM); do + ip netns add ns$i + ip link add veth$i type veth peer name veth0 netns ns$i + ip link set veth$i up + ip -n ns$i link set veth0 up + + ip -n ns$i addr add 192.0.2.$i/24 dev veth0 + ip -n ns$i addr add 2001:db8::$i/64 dev veth0 + ip -n ns$i link set veth0 $mode obj \ + xdp_dummy.o sec xdp_dummy &> /dev/null || \ + { test_fail "Unable to load dummy xdp" && exit 1; } + IFACES="$IFACES veth$i" + veth_mac[$i]=$(ip link show veth$i | awk '/link\/ether/ {print $2}') + done +} + +do_egress_tests() +{ + local mode=$1 + + # mac test + ip netns exec ns2 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + + # mac check + grep -q "${veth_mac[2]} > ff:ff:ff:ff:ff:ff" mac_ns1-2_${mode}.log && \ + test_pass "$mode mac ns1-2" || test_fail "$mode mac ns1-2" + grep -q "${veth_mac[3]} > ff:ff:ff:ff:ff:ff" mac_ns1-3_${mode}.log && \ + test_pass "$mode mac ns1-3" || test_fail "$mode mac ns1-3" + grep -q "${veth_mac[4]} > ff:ff:ff:ff:ff:ff" mac_ns1-4_${mode}.log && \ + test_pass "$mode mac ns1-4" || test_fail "$mode mac ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-2" || test_fail "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-3" || test_fail "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" +} + +do_ping_tests() +{ + local mode=$1 + + # arp test + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -i veth0 -nn -l -e &> arp_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-2_${mode}.log && \ + test_pass "$mode arp ns1-2" || test_fail "$mode arp ns1-2" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-3_${mode}.log && \ + test_pass "$mode arp ns1-3" || test_fail "$mode arp ns1-3" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-4_${mode}.log && \ + test_pass "$mode arp ns1-4" || test_fail "$mode arp ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-2" || test_pass "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-3" || test_pass "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" + + # ping6 test + ip netns exec ns2 ping6 2001:db8::1 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-1" || test_pass "$mode ping6 ns2-1" + ip netns exec ns2 ping6 2001:db8::3 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-3" || test_pass "$mode ping6 ns2-3" + ip netns exec ns2 ping6 2001:db8::4 -c 4 &> /dev/null && \ + test_pass "$mode ping6 ns2-4" || test_fail "$mode ping6 ns2-4" +} + +do_tests() +{ + local mode=$1 + local drv_p + + case ${mode} in + xdpdrv) drv_p="-N";; + xdpegress) drv_p="-X";; + xdpgeneric) drv_p="-S";; + esac + + ./xdp_redirect_multi $drv_p $IFACES &> xdp_redirect_${mode}.log & + xdp_pid=$! + sleep 10 + + if [ "$mode" = "xdpegress" ]; then + do_egress_tests $mode + else + do_ping_tests $mode + fi + + kill $xdp_pid +} + +trap clean_up 0 2 3 6 9 + +check_env +rm -f xdp_redirect_*.log arp_ns*.log mac_ns*.log + +for mode in ${DRV_MODE}; do + setup_ns $mode + do_tests $mode + sleep 10 + clean_up + sleep 5 +done + +echo "Summary: PASS $PASS, FAIL $FAIL" +[ $FAIL -eq 0 ] && exit 0 || exit 1 diff --git a/tools/testing/selftests/bpf/xdp_redirect_multi.c b/tools/testing/selftests/bpf/xdp_redirect_multi.c new file mode 100644 index 000000000000..906db3d76071 --- /dev/null +++ b/tools/testing/selftests/bpf/xdp_redirect_multi.c @@ -0,0 +1,252 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> +#include <sys/resource.h> +#include <sys/ioctl.h> +#include <sys/types.h> +#include <sys/socket.h> +#include <netinet/in.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static int get_mac_addr(unsigned int ifindex, void *mac_addr) +{ + char ifname[IF_NAMESIZE]; + struct ifreq ifr; + int fd, ret = -1; + + fd = socket(AF_INET, SOCK_DGRAM, 0); + if (fd < 0) + return ret; + + if (!if_indextoname(ifindex, ifname)) + goto err_out; + + strcpy(ifr.ifr_name, ifname); + + if (ioctl(fd, SIOCGIFHWADDR, &ifr) != 0) + goto err_out; + + memcpy(mac_addr, ifr.ifr_hwaddr.sa_data, 6 * sizeof(char)); + ret = 0; + +err_out: + close(fd); + return ret; +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n" + " -X load xdp program on egress\n", + prog); +} + +int main(int argc, char **argv) +{ + int prog_fd, group_all, group_v4, group_v6, exclude, mac_map; + struct bpf_program *ingress_prog, *egress_prog; + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_UNSPEC, + }; + int i, ret, opt, egress_prog_fd = -1; + struct bpf_devmap_val devmap_val; + bool attach_egress_prog = false; + unsigned char mac_addr[6]; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + unsigned int ifindex; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNFX")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + case 'X': + attach_egress_prog = true; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) { + xdp_flags |= XDP_FLAGS_DRV_MODE; + } else if (attach_egress_prog) { + printf("Load xdp program on egress with SKB mode not supported yet\n"); + goto err_out; + } + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + goto err_out; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + goto err_out; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + goto err_out; + + if (attach_egress_prog) + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_egress"); + else + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_all"); + group_v4 = bpf_object__find_map_fd_by_name(obj, "forward_map_v4"); + group_v6 = bpf_object__find_map_fd_by_name(obj, "forward_map_v6"); + exclude = bpf_object__find_map_fd_by_name(obj, "exclude_map"); + mac_map = bpf_object__find_map_fd_by_name(obj, "mac_map"); + + if (group_all < 0 || group_v4 < 0 || group_v6 < 0 || exclude < 0 || + mac_map < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + goto err_out; + } + + if (attach_egress_prog) { + /* Find ingress/egress prog for 2nd xdp prog */ + ingress_prog = bpf_object__find_program_by_title(obj, "xdp_redirect_map_ingress"); + egress_prog = bpf_object__find_program_by_title(obj, "xdp_devmap/map_prog"); + if (!ingress_prog || !egress_prog) { + printf("finding ingress/egress_prog in obj file failed\n"); + goto err_out; + } + prog_fd = bpf_program__fd(ingress_prog); + egress_prog_fd = bpf_program__fd(egress_prog); + if (prog_fd < 0 || egress_prog_fd < 0) { + printf("find egress_prog fd failed\n"); + goto err_out; + } + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + if (attach_egress_prog) { + ret = get_mac_addr(ifindex, mac_addr); + if (ret < 0) { + printf("get interface %d mac failed\n", ifindex); + goto err_out; + } + ret = bpf_map_update_elem(mac_map, &ifindex, mac_addr, 0); + if (ret) { + perror("bpf_update_elem mac_map failed\n"); + goto err_out; + } + } + + /* Add all the interfaces to group all */ + devmap_val.ifindex = ifindex; + devmap_val.bpf_prog.fd = egress_prog_fd; + ret = bpf_map_update_elem(group_all, &ifindex, &devmap_val, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* For testing: remove the 1st interfaces from group v6 */ + if (i != 0) { + ret = bpf_map_update_elem(group_v6, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: remove the 2nd interfaces from group v4 */ + if (i != 1) { + ret = bpf_map_update_elem(group_v4, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: add the 3rd interfaces to exclude map */ + if (i == 2) { + ret = bpf_map_update_elem(exclude, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + } + + /* sleep some time for testing */ + sleep(999); + + return 0; + +err_out: + return 1; +} -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv15 bpf-next 0/6] xdp: add a new helper for dev map multicast support 2021-01-14 14:23 ` [PATCHv14 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (5 preceding siblings ...) 2021-01-14 14:23 ` [PATCHv14 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu @ 2021-01-20 2:25 ` Hangbin Liu 2021-01-20 2:25 ` [PATCHv15 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu ` (6 more replies) 6 siblings, 7 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-20 2:25 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because there may have multi interfaces you want to exclude. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. The 1st patch is Jesper's run devmap xdp_prog later in bulking step. The 2st patch add a new bpf arg to allow NULL map pointer. The 3rd patch add the new bpf_redirect_map_multi() helper. The 4-6 patches are for usage sample and testing purpose. I did same perf tests with the following topo: --------------------- --------------------- | Host A (i40e 10G) | ---------- | eno1(i40e 10G) | --------------------- | | | Host B | --------------------- | | | Host C (i40e 10G) | ---------- | eno2(i40e 10G) | --------------------- vlan2 | -------- | | veth1 -- | veth0| | | -------- | --------------------| On Host A: # pktgen/pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -s 64 On Host B(Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, 128G Memory): Use xdp_redirect_map and xdp_redirect_map_multi in samples/bpf for testing. The veth0 in netns load dummy drop program. The forward_map max_entries in xdp_redirect_map_multi is modify to 4. Here is the perf result with 5.10 rc6: The are about +/- 0.1M deviation for native testing Version | Test | Generic | Native | Native + 2nd 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M 5.10 rc6 + patch1 | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch1 | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M 5.10 rc6 + patch1-6 | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch1-6 | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->i40e | 1.7M | 7.8M | 6.4M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->veth | 1.4M | 9.3M | 7.5M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->i40e+veth | 1.0M | 3.2M | 2.7M Last but not least, thanks a lot to Toke, Jesper, Jiri and Eelco for suggestions and help on implementation. [0] https://xdp-project.net/#Handling-multicast v15: Update bq_xmit_all() logic for patch 01. Add some comments and remove useless variable for patch 03. Use bpf_object__find_program_by_title() for patch 04 and 06. v14: No code update, just rebase the code on latest bpf-next v13: Pass in xdp_prog through __xdp_enqueue() for patch 01. Update related code in patch 03. v12: Add Jesper's xdp_prog patch, rebase my works on this and latest bpf-next Add 2nd xdp_prog test on the sample and selftests. v11: Fix bpf_redirect_map_multi() helper description typo. Add loop limit for devmap_get_next_obj() and dev_map_redirect_multi(). v10: Rebase the code to latest bpf-next. Update helper bpf_xdp_redirect_map_multi() - No need to check map pointer as we will do the check in verifier. v9: Update helper bpf_xdp_redirect_map_multi() - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: a) Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). f) Split the tests from sample and add a bpf kernel selftest patch. v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Hangbin Liu (5): bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL xdp: add a new helper for dev map multicast support sample/bpf: add xdp_redirect_map_multicast test selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL selftests/bpf: add xdp_redirect_multi test Jesper Dangaard Brouer (1): bpf: run devmap xdp_prog on flush instead of bulk enqueue include/linux/bpf.h | 21 ++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 28 ++ kernel/bpf/devmap.c | 232 +++++++++++--- kernel/bpf/verifier.c | 16 +- net/core/filter.c | 124 ++++++- net/core/xdp.c | 29 ++ samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 87 +++++ samples/bpf/xdp_redirect_map_multi_user.c | 302 ++++++++++++++++++ tools/include/uapi/linux/bpf.h | 28 ++ tools/testing/selftests/bpf/Makefile | 3 +- .../bpf/progs/xdp_redirect_multi_kern.c | 111 +++++++ tools/testing/selftests/bpf/test_verifier.c | 22 +- .../selftests/bpf/test_xdp_redirect_multi.sh | 208 ++++++++++++ .../testing/selftests/bpf/verifier/map_ptr.c | 70 ++++ .../selftests/bpf/xdp_redirect_multi.c | 252 +++++++++++++++ 18 files changed, 1488 insertions(+), 50 deletions(-) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c -- 2.26.2 ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv15 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-20 2:25 ` [PATCHv15 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2021-01-20 2:25 ` Hangbin Liu 2021-01-20 22:42 ` Maciej Fijalkowski 2021-01-20 2:25 ` [PATCHv15 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu ` (5 subsequent siblings) 6 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2021-01-20 2:25 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Hangbin Liu From: Jesper Dangaard Brouer <brouer@redhat.com> This changes the devmap XDP program support to run the program when the bulk queue is flushed instead of before the frame is enqueued. This has a couple of benefits: - It "sorts" the packets by destination devmap entry, and then runs the same BPF program on all the packets in sequence. This ensures that we keep the XDP program and destination device properties hot in I-cache. - It makes the multicast implementation simpler because it can just enqueue packets using bq_enqueue() without having to deal with the devmap program at all. The drawback is that if the devmap program drops the packet, the enqueue step is redundant. However, arguably this is mostly visible in a micro-benchmark, and with more mixed traffic the I-cache benefit should win out. The performance impact of just this patch is as follows: Using xdp_redirect_map(with a 2nd xdp_prog patch[1]) in sample/bpf and send pkts via pktgen cmd: ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 There are about +/- 0.1M deviation for native testing, the performance improved for the base-case, but some drop back with xdp devmap prog attached. Version | Test | Generic | Native | Native + 2nd xdp_prog 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M 5.10 rc6 + patch | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M [1] https://patchwork.ozlabs.org/project/netdev/patch/20201208120159.2278277-1-liuhangbin@gmail.com/ Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v15: a) do not use unlikely when checking bq->xdp_prog b) return sent frames for dev_map_bpf_prog_run() v14: no update, only rebase the code v13: pass in xdp_prog through __xdp_enqueue() v2-v12: no this patch --- kernel/bpf/devmap.c | 116 +++++++++++++++++++++++++++----------------- 1 file changed, 71 insertions(+), 45 deletions(-) diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index f6e9c68afdd4..13ed68c24aad 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -57,6 +57,7 @@ struct xdp_dev_bulk_queue { struct list_head flush_node; struct net_device *dev; struct net_device *dev_rx; + struct bpf_prog *xdp_prog; unsigned int count; }; @@ -327,44 +328,93 @@ bool dev_map_can_have_prog(struct bpf_map *map) return false; } +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, + struct xdp_frame **frames, int n, + struct net_device *dev) +{ + struct xdp_txq_info txq = { .dev = dev }; + struct xdp_buff xdp; + int i, nframes = 0; + + for (i = 0; i < n; i++) { + struct xdp_frame *xdpf = frames[i]; + u32 act; + int err; + + xdp_convert_frame_to_buff(xdpf, &xdp); + xdp.txq = &txq; + + act = bpf_prog_run_xdp(xdp_prog, &xdp); + switch (act) { + case XDP_PASS: + err = xdp_update_frame_from_buff(&xdp, xdpf); + if (unlikely(err < 0)) + xdp_return_frame_rx_napi(xdpf); + else + frames[nframes++] = xdpf; + break; + default: + bpf_warn_invalid_xdp_action(act); + fallthrough; + case XDP_ABORTED: + trace_xdp_exception(dev, xdp_prog, act); + fallthrough; + case XDP_DROP: + xdp_return_frame_rx_napi(xdpf); + break; + } + } + return nframes; /* sent frames count */ +} + static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) { struct net_device *dev = bq->dev; int sent = 0, drops = 0, err = 0; + unsigned int cnt = bq->count; int i; - if (unlikely(!bq->count)) + if (unlikely(!cnt)) return; - for (i = 0; i < bq->count; i++) { + for (i = 0; i < cnt; i++) { struct xdp_frame *xdpf = bq->q[i]; prefetch(xdpf); } - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); + /* Init sent to cnt in case there is no xdp_prog */ + sent = cnt; + if (bq->xdp_prog) { + sent = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); + if (!sent) + goto out; + } + + /* Backup drops value before xmit as we may need it in error label */ + drops = cnt - sent; + sent = dev->netdev_ops->ndo_xdp_xmit(dev, sent, bq->q, flags); if (sent < 0) { err = sent; sent = 0; goto error; } - drops = bq->count - sent; out: + drops = cnt - sent; bq->count = 0; trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); bq->dev_rx = NULL; + bq->xdp_prog = NULL; __list_del_clearprev(&bq->flush_node); return; error: /* If ndo_xdp_xmit fails with an errno, no frames have been * xmit'ed and it's our responsibility to them free all. */ - for (i = 0; i < bq->count; i++) { + for (i = 0; i < cnt - drops; i++) { struct xdp_frame *xdpf = bq->q[i]; - xdp_return_frame_rx_napi(xdpf); - drops++; } goto out; } @@ -408,7 +458,7 @@ struct bpf_dtab_netdev *__dev_map_lookup_elem(struct bpf_map *map, u32 key) * Thus, safe percpu variable access. */ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, - struct net_device *dev_rx) + struct net_device *dev_rx, struct bpf_prog *xdp_prog) { struct list_head *flush_list = this_cpu_ptr(&dev_flush_list); struct xdp_dev_bulk_queue *bq = this_cpu_ptr(dev->xdp_bulkq); @@ -423,6 +473,14 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, if (!bq->dev_rx) bq->dev_rx = dev_rx; + /* Store (potential) xdp_prog that run before egress to dev as + * part of bulk_queue. This will be same xdp_prog for all + * xdp_frame's in bulk_queue, because this per-CPU store must + * be flushed from net_device drivers NAPI func end. + */ + if (!bq->xdp_prog) + bq->xdp_prog = xdp_prog; + bq->q[bq->count++] = xdpf; if (!bq->flush_node.prev) @@ -430,7 +488,8 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, } static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, - struct net_device *dev_rx) + struct net_device *dev_rx, + struct bpf_prog *xdp_prog) { struct xdp_frame *xdpf; int err; @@ -446,42 +505,14 @@ static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, if (unlikely(!xdpf)) return -EOVERFLOW; - bq_enqueue(dev, xdpf, dev_rx); + bq_enqueue(dev, xdpf, dev_rx, xdp_prog); return 0; } -static struct xdp_buff *dev_map_run_prog(struct net_device *dev, - struct xdp_buff *xdp, - struct bpf_prog *xdp_prog) -{ - struct xdp_txq_info txq = { .dev = dev }; - u32 act; - - xdp_set_data_meta_invalid(xdp); - xdp->txq = &txq; - - act = bpf_prog_run_xdp(xdp_prog, xdp); - switch (act) { - case XDP_PASS: - return xdp; - case XDP_DROP: - break; - default: - bpf_warn_invalid_xdp_action(act); - fallthrough; - case XDP_ABORTED: - trace_xdp_exception(dev, xdp_prog, act); - break; - } - - xdp_return_buff(xdp); - return NULL; -} - int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx) { - return __xdp_enqueue(dev, xdp, dev_rx); + return __xdp_enqueue(dev, xdp, dev_rx, NULL); } int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, @@ -489,12 +520,7 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, { struct net_device *dev = dst->dev; - if (dst->xdp_prog) { - xdp = dev_map_run_prog(dev, xdp, dst->xdp_prog); - if (!xdp) - return 0; - } - return __xdp_enqueue(dev, xdp, dev_rx); + return __xdp_enqueue(dev, xdp, dev_rx, dst->xdp_prog); } int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv15 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-20 2:25 ` [PATCHv15 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu @ 2021-01-20 22:42 ` Maciej Fijalkowski 2021-01-21 3:54 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: Maciej Fijalkowski @ 2021-01-20 22:42 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend On Wed, Jan 20, 2021 at 10:25:09AM +0800, Hangbin Liu wrote: > From: Jesper Dangaard Brouer <brouer@redhat.com> > > This changes the devmap XDP program support to run the program when the > bulk queue is flushed instead of before the frame is enqueued. This has > a couple of benefits: > > - It "sorts" the packets by destination devmap entry, and then runs the > same BPF program on all the packets in sequence. This ensures that we > keep the XDP program and destination device properties hot in I-cache. > > - It makes the multicast implementation simpler because it can just > enqueue packets using bq_enqueue() without having to deal with the > devmap program at all. > > The drawback is that if the devmap program drops the packet, the enqueue > step is redundant. However, arguably this is mostly visible in a > micro-benchmark, and with more mixed traffic the I-cache benefit should > win out. The performance impact of just this patch is as follows: > > Using xdp_redirect_map(with a 2nd xdp_prog patch[1]) in sample/bpf and send > pkts via pktgen cmd: > ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 > > There are about +/- 0.1M deviation for native testing, the performance > improved for the base-case, but some drop back with xdp devmap prog attached. > > Version | Test | Generic | Native | Native + 2nd xdp_prog > 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M > 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M > 5.10 rc6 + patch | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M > 5.10 rc6 + patch | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M > > [1] https://patchwork.ozlabs.org/project/netdev/patch/20201208120159.2278277-1-liuhangbin@gmail.com/ nit: probably would be good to update the link to patch, I see it's v8 already whereas link refers to v4. > > Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com> > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > > --- > v15: > a) do not use unlikely when checking bq->xdp_prog > b) return sent frames for dev_map_bpf_prog_run() > > v14: no update, only rebase the code > v13: pass in xdp_prog through __xdp_enqueue() > v2-v12: no this patch > --- > kernel/bpf/devmap.c | 116 +++++++++++++++++++++++++++----------------- > 1 file changed, 71 insertions(+), 45 deletions(-) > > diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c > index f6e9c68afdd4..13ed68c24aad 100644 > --- a/kernel/bpf/devmap.c > +++ b/kernel/bpf/devmap.c > @@ -57,6 +57,7 @@ struct xdp_dev_bulk_queue { > struct list_head flush_node; > struct net_device *dev; > struct net_device *dev_rx; > + struct bpf_prog *xdp_prog; > unsigned int count; > }; > > @@ -327,44 +328,93 @@ bool dev_map_can_have_prog(struct bpf_map *map) > return false; > } > > +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, > + struct xdp_frame **frames, int n, > + struct net_device *dev) > +{ > + struct xdp_txq_info txq = { .dev = dev }; > + struct xdp_buff xdp; > + int i, nframes = 0; > + > + for (i = 0; i < n; i++) { > + struct xdp_frame *xdpf = frames[i]; > + u32 act; > + int err; > + > + xdp_convert_frame_to_buff(xdpf, &xdp); > + xdp.txq = &txq; > + > + act = bpf_prog_run_xdp(xdp_prog, &xdp); > + switch (act) { > + case XDP_PASS: > + err = xdp_update_frame_from_buff(&xdp, xdpf); Bump on John's question. > + if (unlikely(err < 0)) > + xdp_return_frame_rx_napi(xdpf); > + else > + frames[nframes++] = xdpf; > + break; > + default: > + bpf_warn_invalid_xdp_action(act); > + fallthrough; > + case XDP_ABORTED: > + trace_xdp_exception(dev, xdp_prog, act); > + fallthrough; > + case XDP_DROP: > + xdp_return_frame_rx_napi(xdpf); > + break; > + } > + } > + return nframes; /* sent frames count */ > +} > + > static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) > { > struct net_device *dev = bq->dev; > int sent = 0, drops = 0, err = 0; if sent is inited below, the assignment above is useless. Couldn't we do sent = cnt at the declaration over here? > + unsigned int cnt = bq->count; > int i; > > - if (unlikely(!bq->count)) > + if (unlikely(!cnt)) > return; > > - for (i = 0; i < bq->count; i++) { > + for (i = 0; i < cnt; i++) { > struct xdp_frame *xdpf = bq->q[i]; > > prefetch(xdpf); > } > > - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); > + /* Init sent to cnt in case there is no xdp_prog */ > + sent = cnt; > + if (bq->xdp_prog) { > + sent = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); > + if (!sent) > + goto out; Sorry, but 'sent' is a bit confusing to me, actual sending happens below via ndo_xdp_xmit, right? This hook will not actually send frames. Can we do a subtle change to have it in separate variable 'to_send' ? > + } > + > + /* Backup drops value before xmit as we may need it in error label */ > + drops = cnt - sent; > + sent = dev->netdev_ops->ndo_xdp_xmit(dev, sent, bq->q, flags); > if (sent < 0) { > err = sent; > sent = 0; > goto error; > } > - drops = bq->count - sent; > out: > + drops = cnt - sent; > bq->count = 0; > > trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); > bq->dev_rx = NULL; > + bq->xdp_prog = NULL; > __list_del_clearprev(&bq->flush_node); > return; > error: > /* If ndo_xdp_xmit fails with an errno, no frames have been > * xmit'ed and it's our responsibility to them free all. > */ > - for (i = 0; i < bq->count; i++) { > + for (i = 0; i < cnt - drops; i++) { > struct xdp_frame *xdpf = bq->q[i]; > - > xdp_return_frame_rx_napi(xdpf); > - drops++; > } > goto out; Although I'm a huge goto advocate, I feel like this particular usage could be simplified. Not sure why we had that in first place. I gave a shot at rewriting/refactoring whole bq_xmit_all and I feel like it's more readable. I introduced 'to_send' variable and got rid of 'error' label. Thoughts? I might have missed something, though. static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) { struct net_device *dev = bq->dev; unsigned int cnt = bq->count; int drops = 0, err = 0; int to_send = 0; int sent = cnt; int i; if (unlikely(!cnt)) return; for (i = 0; i < cnt; i++) { struct xdp_frame *xdpf = bq->q[i]; prefetch(xdpf); } if (bq->xdp_prog) { to_send = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); if (!to_send) { sent = 0; goto out; } } drops = cnt - to_send; sent = dev->netdev_ops->ndo_xdp_xmit(dev, to_send, bq->q, flags); if (sent < 0) { err = sent; sent = 0; /* If ndo_xdp_xmit fails with an errno, no frames have been * xmit'ed and it's our responsibility to them free all. */ for (i = 0; i < cnt - drops; i++) { struct xdp_frame *xdpf = bq->q[i]; xdp_return_frame_rx_napi(xdpf); } } out: drops = cnt - sent; bq->count = 0; trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); bq->dev_rx = NULL; bq->xdp_prog = NULL; __list_del_clearprev(&bq->flush_node); return; } > } > @@ -408,7 +458,7 @@ struct bpf_dtab_netdev *__dev_map_lookup_elem(struct bpf_map *map, u32 key) > * Thus, safe percpu variable access. > */ > static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, > - struct net_device *dev_rx) > + struct net_device *dev_rx, struct bpf_prog *xdp_prog) > { > struct list_head *flush_list = this_cpu_ptr(&dev_flush_list); > struct xdp_dev_bulk_queue *bq = this_cpu_ptr(dev->xdp_bulkq); > @@ -423,6 +473,14 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, > if (!bq->dev_rx) > bq->dev_rx = dev_rx; > > + /* Store (potential) xdp_prog that run before egress to dev as > + * part of bulk_queue. This will be same xdp_prog for all > + * xdp_frame's in bulk_queue, because this per-CPU store must > + * be flushed from net_device drivers NAPI func end. > + */ > + if (!bq->xdp_prog) > + bq->xdp_prog = xdp_prog; > + > bq->q[bq->count++] = xdpf; > > if (!bq->flush_node.prev) > @@ -430,7 +488,8 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, > } > > static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, > - struct net_device *dev_rx) > + struct net_device *dev_rx, > + struct bpf_prog *xdp_prog) > { > struct xdp_frame *xdpf; > int err; > @@ -446,42 +505,14 @@ static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, > if (unlikely(!xdpf)) > return -EOVERFLOW; > > - bq_enqueue(dev, xdpf, dev_rx); > + bq_enqueue(dev, xdpf, dev_rx, xdp_prog); > return 0; > } > > -static struct xdp_buff *dev_map_run_prog(struct net_device *dev, > - struct xdp_buff *xdp, > - struct bpf_prog *xdp_prog) > -{ > - struct xdp_txq_info txq = { .dev = dev }; > - u32 act; > - > - xdp_set_data_meta_invalid(xdp); > - xdp->txq = &txq; > - > - act = bpf_prog_run_xdp(xdp_prog, xdp); > - switch (act) { > - case XDP_PASS: > - return xdp; > - case XDP_DROP: > - break; > - default: > - bpf_warn_invalid_xdp_action(act); > - fallthrough; > - case XDP_ABORTED: > - trace_xdp_exception(dev, xdp_prog, act); > - break; > - } > - > - xdp_return_buff(xdp); > - return NULL; > -} > - > int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, > struct net_device *dev_rx) > { > - return __xdp_enqueue(dev, xdp, dev_rx); > + return __xdp_enqueue(dev, xdp, dev_rx, NULL); > } > > int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > @@ -489,12 +520,7 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > { > struct net_device *dev = dst->dev; > > - if (dst->xdp_prog) { > - xdp = dev_map_run_prog(dev, xdp, dst->xdp_prog); > - if (!xdp) > - return 0; > - } > - return __xdp_enqueue(dev, xdp, dev_rx); > + return __xdp_enqueue(dev, xdp, dev_rx, dst->xdp_prog); > } > > int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, > -- > 2.26.2 > ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv15 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-20 22:42 ` Maciej Fijalkowski @ 2021-01-21 3:54 ` Hangbin Liu 2021-01-21 13:35 ` Maciej Fijalkowski 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2021-01-21 3:54 UTC (permalink / raw) To: Maciej Fijalkowski Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend Hi Maciej, On Wed, Jan 20, 2021 at 11:42:38PM +0100, Maciej Fijalkowski wrote: > > +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, > > + struct xdp_frame **frames, int n, > > + struct net_device *dev) > > +{ > > + struct xdp_txq_info txq = { .dev = dev }; > > + struct xdp_buff xdp; > > + int i, nframes = 0; > > + > > + for (i = 0; i < n; i++) { > > + struct xdp_frame *xdpf = frames[i]; > > + u32 act; > > + int err; > > + > > + xdp_convert_frame_to_buff(xdpf, &xdp); > > + xdp.txq = &txq; > > + > > + act = bpf_prog_run_xdp(xdp_prog, &xdp); > > + switch (act) { > > + case XDP_PASS: > > + err = xdp_update_frame_from_buff(&xdp, xdpf); > > Bump on John's question. Hi Jesper, would you please help answer John's question? > > > > - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); > > + /* Init sent to cnt in case there is no xdp_prog */ > > + sent = cnt; > > + if (bq->xdp_prog) { > > + sent = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); > > + if (!sent) > > + goto out; > > Sorry, but 'sent' is a bit confusing to me, actual sending happens below > via ndo_xdp_xmit, right? This hook will not actually send frames. > Can we do a subtle change to have it in separate variable 'to_send' ? Makes sense to me. > > Although I'm a huge goto advocate, I feel like this particular usage could > be simplified. Not sure why we had that in first place. > > I gave a shot at rewriting/refactoring whole bq_xmit_all and I feel like > it's more readable. I introduced 'to_send' variable and got rid of 'error' > label. > > Thoughts? > > I might have missed something, though. > > static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) > { > struct net_device *dev = bq->dev; > unsigned int cnt = bq->count; > int drops = 0, err = 0; > int to_send = 0; The to_send also need to init to cnt. > int sent = cnt; > int i; > > if (unlikely(!cnt)) > return; > > for (i = 0; i < cnt; i++) { > struct xdp_frame *xdpf = bq->q[i]; > > prefetch(xdpf); > } > > if (bq->xdp_prog) { > to_send = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); > if (!to_send) { > sent = 0; > goto out; > } > } > > drops = cnt - to_send; This line could move in to the xdp_prog brackets to save time when no xdp_prog. if (bq->xdp_prog) { to_send = ... if (!to_send) { ... } drops = cnt - to_send; } > sent = dev->netdev_ops->ndo_xdp_xmit(dev, to_send, bq->q, flags); If we don't have xdp_prog, the to_send should be cnt. > if (sent < 0) { > err = sent; > sent = 0; > > /* If ndo_xdp_xmit fails with an errno, no frames have been > * xmit'ed and it's our responsibility to them free all. > */ > for (i = 0; i < cnt - drops; i++) { > struct xdp_frame *xdpf = bq->q[i]; > > xdp_return_frame_rx_napi(xdpf); > } > } > out: > drops = cnt - sent; > bq->count = 0; > > trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); > bq->dev_rx = NULL; > bq->xdp_prog = NULL; > __list_del_clearprev(&bq->flush_node); > > return; > } Thanks for your code, looks much clear now. Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv15 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-21 3:54 ` Hangbin Liu @ 2021-01-21 13:35 ` Maciej Fijalkowski 0 siblings, 0 replies; 219+ messages in thread From: Maciej Fijalkowski @ 2021-01-21 13:35 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend On Thu, Jan 21, 2021 at 11:54:24AM +0800, Hangbin Liu wrote: > Hi Maciej, > On Wed, Jan 20, 2021 at 11:42:38PM +0100, Maciej Fijalkowski wrote: > > > +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, > > > + struct xdp_frame **frames, int n, > > > + struct net_device *dev) > > > +{ > > > + struct xdp_txq_info txq = { .dev = dev }; > > > + struct xdp_buff xdp; > > > + int i, nframes = 0; > > > + > > > + for (i = 0; i < n; i++) { > > > + struct xdp_frame *xdpf = frames[i]; > > > + u32 act; > > > + int err; > > > + > > > + xdp_convert_frame_to_buff(xdpf, &xdp); > > > + xdp.txq = &txq; > > > + > > > + act = bpf_prog_run_xdp(xdp_prog, &xdp); > > > + switch (act) { > > > + case XDP_PASS: > > > + err = xdp_update_frame_from_buff(&xdp, xdpf); > > > > Bump on John's question. > > Hi Jesper, would you please help answer John's question? > > > > > > - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); > > > + /* Init sent to cnt in case there is no xdp_prog */ > > > + sent = cnt; > > > + if (bq->xdp_prog) { > > > + sent = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); > > > + if (!sent) > > > + goto out; > > > > Sorry, but 'sent' is a bit confusing to me, actual sending happens below > > via ndo_xdp_xmit, right? This hook will not actually send frames. > > Can we do a subtle change to have it in separate variable 'to_send' ? > > Makes sense to me. > > > > Although I'm a huge goto advocate, I feel like this particular usage could > > be simplified. Not sure why we had that in first place. > > > > I gave a shot at rewriting/refactoring whole bq_xmit_all and I feel like > > it's more readable. I introduced 'to_send' variable and got rid of 'error' > > label. > > > > Thoughts? > > > > I might have missed something, though. > > > > static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) > > { > > struct net_device *dev = bq->dev; > > unsigned int cnt = bq->count; > > int drops = 0, err = 0; > > int to_send = 0; > > The to_send also need to init to cnt. So I missed something indeed :P you're correct > > > int sent = cnt; > > int i; > > > > if (unlikely(!cnt)) > > return; > > > > for (i = 0; i < cnt; i++) { > > struct xdp_frame *xdpf = bq->q[i]; > > > > prefetch(xdpf); > > } > > > > if (bq->xdp_prog) { > > to_send = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); > > if (!to_send) { > > sent = 0; > > goto out; > > } > > } > > > > drops = cnt - to_send; > > This line could move in to the xdp_prog brackets to save time when no xdp_prog. Hmm, looks like we can do it. For scenario where there was no bq->xdp_prog and failure of ndo_xdp_xmit, we didn't alter the count of frames to be sent, so we would basically free all of the frames (as drops is 0, cnt = bq->count). After that we recalculate drops and correct value will be reported in tracepoint. (needed to explain it to myself) > > if (bq->xdp_prog) { > to_send = ... > if (!to_send) { > ... > } > drops = cnt - to_send; > } > > > sent = dev->netdev_ops->ndo_xdp_xmit(dev, to_send, bq->q, flags); > > If we don't have xdp_prog, the to_send should be cnt. Yes, we should init to_send to cnt as you're suggesting above. > > > if (sent < 0) { > > err = sent; > > sent = 0; > > > > /* If ndo_xdp_xmit fails with an errno, no frames have been > > * xmit'ed and it's our responsibility to them free all. > > */ > > for (i = 0; i < cnt - drops; i++) { > > struct xdp_frame *xdpf = bq->q[i]; > > > > xdp_return_frame_rx_napi(xdpf); > > } > > } > > out: > > drops = cnt - sent; > > bq->count = 0; > > > > trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); > > bq->dev_rx = NULL; > > bq->xdp_prog = NULL; > > __list_del_clearprev(&bq->flush_node); > > > > return; > > } > > Thanks for your code, looks much clear now. Good to hear! I agree on your points as well. > > Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv15 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL 2021-01-20 2:25 ` [PATCHv15 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2021-01-20 2:25 ` [PATCHv15 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu @ 2021-01-20 2:25 ` Hangbin Liu 2021-01-20 2:25 ` [PATCHv15 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (4 subsequent siblings) 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-20 2:25 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Hangbin Liu Add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL which could be used when we want to allow NULL pointer for map parameter. The bpf helper need to take care and check if the map is NULL when use this type. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v13-v15: no update v11-v12: rebase the patch to latest bpf-next v10: remove useless CONST_PTR_TO_MAP_OR_NULL and Copy-paste comment. v9: merge the patch from [1] in to this series. v1-v8: no this patch [1] https://lore.kernel.org/bpf/20200715070001.2048207-1-liuhangbin@gmail.com/ --- include/linux/bpf.h | 1 + kernel/bpf/verifier.c | 10 ++++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 1aac2af12fed..b75207a2484c 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -296,6 +296,7 @@ enum bpf_arg_type { ARG_CONST_ALLOC_SIZE_OR_ZERO, /* number of allocated bytes requested */ ARG_PTR_TO_BTF_ID_SOCK_COMMON, /* pointer to in-kernel sock_common or bpf-mirrored bpf_sock */ ARG_PTR_TO_PERCPU_BTF_ID, /* pointer to in-kernel percpu type */ + ARG_CONST_MAP_PTR_OR_NULL, /* const argument used as pointer to bpf_map or NULL */ __BPF_ARG_TYPE_MAX, }; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 0f82d5d46e2c..89c60494dd69 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -445,7 +445,8 @@ static bool arg_type_may_be_null(enum bpf_arg_type type) type == ARG_PTR_TO_MEM_OR_NULL || type == ARG_PTR_TO_CTX_OR_NULL || type == ARG_PTR_TO_SOCKET_OR_NULL || - type == ARG_PTR_TO_ALLOC_MEM_OR_NULL; + type == ARG_PTR_TO_ALLOC_MEM_OR_NULL || + type == ARG_CONST_MAP_PTR_OR_NULL; } /* Determine whether the function releases some resources allocated by another @@ -4106,6 +4107,7 @@ static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = { [ARG_CONST_SIZE_OR_ZERO] = &scalar_types, [ARG_CONST_ALLOC_SIZE_OR_ZERO] = &scalar_types, [ARG_CONST_MAP_PTR] = &const_map_ptr_types, + [ARG_CONST_MAP_PTR_OR_NULL] = &const_map_ptr_types, [ARG_PTR_TO_CTX] = &context_types, [ARG_PTR_TO_CTX_OR_NULL] = &context_types, [ARG_PTR_TO_SOCK_COMMON] = &sock_types, @@ -4251,9 +4253,9 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, meta->ref_obj_id = reg->ref_obj_id; } - if (arg_type == ARG_CONST_MAP_PTR) { - /* bpf_map_xxx(map_ptr) call: remember that map_ptr */ - meta->map_ptr = reg->map_ptr; + if (arg_type == ARG_CONST_MAP_PTR || + arg_type == ARG_CONST_MAP_PTR_OR_NULL) { + meta->map_ptr = register_is_null(reg) ? NULL : reg->map_ptr; } else if (arg_type == ARG_PTR_TO_MAP_KEY) { /* bpf_map_xxx(..., map_ptr, ..., key) call: * check that [key, key + map->key_size) are within -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv15 bpf-next 3/6] xdp: add a new helper for dev map multicast support 2021-01-20 2:25 ` [PATCHv15 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2021-01-20 2:25 ` [PATCHv15 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu 2021-01-20 2:25 ` [PATCHv15 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2021-01-20 2:25 ` Hangbin Liu 2021-01-20 2:25 ` [PATCHv15 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu ` (3 subsequent siblings) 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-20 2:25 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because a physical interface can be part of a logical grouping, such as a bond device. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. The forwarding map could be DEVMAP or DEVMAP_HASH, but the exclude map *must* be DEVMAP_HASH to get better performace. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. As both bpf_xdp_redirect_map() and this new helpers are using struct bpf_redirect_info, I add a new ex_map and set tgt_value to NULL in the new helper to make a difference with bpf_xdp_redirect_map(). Also I keep the general data path in net/core/filter.c, the native data path in kernel/bpf/devmap.c so we can use direct calls to get better performace. [0] https://xdp-project.net/#Handling-multicast Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v15: a) Update bpf_redirect_map_multi() helper description that ex_map must be keyed by ifindex. b) remove variable last_one in dev_map_enqueue_multi() as it's pointless. c) add a comment about why we don't use READ/WRITE_ONCE() for ex_map. v14: no update, only rebase the code v13: pass xdp_prog through bq_enqueue v12: rebase the code based on Jespoer's devmap xdp_prog patch v11: Fix bpf_redirect_map_multi() helper description typo. Add loop limit for devmap_get_next_obj() and dev_map_redirect_multi(). v10: Update helper bpf_xdp_redirect_map_multi() - No need to check map pointer as we will do the check in verifier. v9: Update helper bpf_xdp_redirect_map_multi() - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. --- include/linux/bpf.h | 20 ++++++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 28 ++++++++ kernel/bpf/devmap.c | 128 +++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++ net/core/filter.c | 124 ++++++++++++++++++++++++++++++-- net/core/xdp.c | 29 ++++++++ tools/include/uapi/linux/bpf.h | 28 ++++++++ 9 files changed, 360 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index b75207a2484c..4b68e975e990 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1427,6 +1427,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx); int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, struct net_device *dev_rx); +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex); +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); bool dev_map_can_have_prog(struct bpf_map *map); @@ -1595,6 +1600,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return 0; } +static inline +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + return false; +} + +static inline +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + return 0; +} + struct sk_buff; static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, diff --git a/include/linux/filter.h b/include/linux/filter.h index 7fdce5407214..dc9d79651c24 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -637,6 +637,7 @@ struct bpf_redirect_info { u32 tgt_index; void *tgt_value; struct bpf_map *map; + struct bpf_map *ex_map; u32 kern_flags; struct bpf_nh_params nh; }; diff --git a/include/net/xdp.h b/include/net/xdp.h index c4bfdc9a8b79..fcc0f6f8bd91 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -169,6 +169,7 @@ struct sk_buff *__xdp_build_skb_from_frame(struct xdp_frame *xdpf, struct net_device *dev); struct sk_buff *xdp_build_skb_from_frame(struct xdp_frame *xdpf, struct net_device *dev); +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); static inline void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index c001766adcbc..ef943c024322 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3836,6 +3836,28 @@ union bpf_attr { * Return * A pointer to a struct socket on success or NULL if the file is * not a socket. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. To get better performance, the + * *ex_map* is limited to BPF_MAP_TYPE_DEVMAP_HASH and must be + * keyed by ifindex for the helper to work. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -4001,6 +4023,7 @@ union bpf_attr { FN(ktime_get_coarse_ns), \ FN(ima_inode_hash), \ FN(sock_from_file), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -4177,6 +4200,11 @@ enum { BPF_F_BPRM_SECUREEXEC = (1ULL << 0), }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 13ed68c24aad..6c9bc3c359e0 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -523,6 +523,134 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return __xdp_enqueue(dev, xdp, dev_rx, dst->xdp_prog); } +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + return dev_map_get_next_key(map, key, next_key); + case BPF_MAP_TYPE_DEVMAP_HASH: + return dev_map_hash_get_next_key(map, key, next_key); + default: + break; + } + + return -ENOENT; +} + +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + if (obj->dev->ifindex == exclude_ifindex) + return true; + + if (!map) + return false; + + return __dev_map_hash_lookup_elem(map, obj->dev->ifindex) != NULL; +} + +static struct bpf_dtab_netdev *devmap_get_next_obj(struct xdp_buff *xdp, struct bpf_map *map, + struct bpf_map *ex_map, u32 *key, + u32 *next_key, int ex_ifindex) +{ + struct bpf_dtab_netdev *obj; + struct net_device *dev; + u32 *tmp_key = key; + u32 index; + int err; + + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + return NULL; + + /* When using dev map hash, we could restart the hashtab traversal + * in case the key has been updated/removed in the mean time. + * So we may end up potentially looping due to traversal restarts + * from first elem. + * + * Let's use map's max_entries to limit the loop number. + */ + for (index = 0; index < map->max_entries; index++) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + obj = __dev_map_lookup_elem(map, *next_key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + obj = __dev_map_hash_lookup_elem(map, *next_key); + break; + default: + break; + } + + if (!obj || dev_in_exclude_map(obj, ex_map, ex_ifindex)) + goto find_next; + + dev = obj->dev; + + if (!dev->netdev_ops->ndo_xdp_xmit) + goto find_next; + + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + if (unlikely(err)) + goto find_next; + + return obj; + +find_next: + tmp_key = next_key; + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + break; + } + + return NULL; +} + +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + struct bpf_dtab_netdev *obj = NULL, *next_obj = NULL; + struct xdp_frame *xdpf, *nxdpf; + int ex_ifindex; + u32 key, next_key; + + ex_ifindex = flags & BPF_F_EXCLUDE_INGRESS ? dev_rx->ifindex : 0; + + /* Find first available obj */ + obj = devmap_get_next_obj(xdp, map, ex_map, NULL, &key, ex_ifindex); + if (!obj) + return 0; + + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) + return -EOVERFLOW; + + for (;;) { + /* Check if we still have one more available obj */ + next_obj = devmap_get_next_obj(xdp, map, ex_map, &key, + &next_key, ex_ifindex); + if (!next_obj) { + bq_enqueue(obj->dev, xdpf, dev_rx, obj->xdp_prog); + return 0; + } + + nxdpf = xdpf_clone(xdpf); + if (unlikely(!nxdpf)) { + xdp_return_frame_rx_napi(xdpf); + return -ENOMEM; + } + + bq_enqueue(obj->dev, nxdpf, dev_rx, obj->xdp_prog); + + /* Deal with next obj */ + obj = next_obj; + key = next_key; + } +} + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 89c60494dd69..d645ec1f6d88 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4461,6 +4461,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: if (func_id != BPF_FUNC_redirect_map && + func_id != BPF_FUNC_redirect_map_multi && func_id != BPF_FUNC_map_lookup_elem) goto error; break; @@ -4565,6 +4566,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, map->map_type != BPF_MAP_TYPE_XSKMAP) goto error; break; + case BPF_FUNC_redirect_map_multi: + if (map->map_type != BPF_MAP_TYPE_DEVMAP && + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + goto error; + break; case BPF_FUNC_sk_redirect_map: case BPF_FUNC_msg_redirect_map: case BPF_FUNC_sock_map_update: diff --git a/net/core/filter.c b/net/core/filter.c index 9ab94e90d660..03b3088ce8db 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3924,12 +3924,19 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { }; static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) + struct bpf_map *map, struct xdp_buff *xdp, + struct bpf_map *ex_map, u32 flags) { switch (map->map_type) { case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: - return dev_map_enqueue(fwd, xdp, dev_rx); + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) + return dev_map_enqueue(fwd, xdp, dev_rx); + else + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, flags); case BPF_MAP_TYPE_CPUMAP: return cpu_map_enqueue(fwd, xdp, dev_rx); case BPF_MAP_TYPE_XSKMAP: @@ -3986,12 +3993,19 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); struct bpf_map *map = READ_ONCE(ri->map); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err; + /* The READ/WRITE_ONCE() is not needed for ex_map because the field + * is only read from or written to by the CPU owning the per-cpu + * pointer. Whereas the 'map' field is manipulated by remote CPUs + * in bpf_clear_redirect_map(). + */ ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (unlikely(!map)) { @@ -4003,7 +4017,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = dev_xdp_enqueue(fwd, xdp, dev); } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, ri->flags); } if (unlikely(err)) @@ -4017,6 +4031,63 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, } EXPORT_SYMBOL_GPL(xdp_do_redirect); +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, + struct bpf_prog *xdp_prog, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) + +{ + struct bpf_dtab_netdev *dst; + struct sk_buff *nskb; + bool exclude_ingress; + u32 key, next_key, index; + void *fwd; + int err; + + /* Get first key from forward map */ + err = map->ops->map_get_next_key(map, NULL, &key); + if (err) + return err; + + exclude_ingress = !!(flags & BPF_F_EXCLUDE_INGRESS); + + /* When using dev map hash, we could restart the hashtab traversal + * in case the key has been updated/removed in the mean time. + * So we may end up potentially looping due to traversal restarts + * from first elem. + * + * Let's use map's max_entries to limit the loop number. + */ + for (index = 0; index < map->max_entries; index++) { + fwd = __xdp_map_lookup_elem(map, key); + if (fwd) { + dst = (struct bpf_dtab_netdev *)fwd; + if (dev_in_exclude_map(dst, ex_map, + exclude_ingress ? dev->ifindex : 0)) + goto find_next; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -ENOMEM; + + /* Try forword next one no mater the current forward + * succeed or not. + */ + dev_map_generic_redirect(dst, nskb, xdp_prog); + } + +find_next: + err = map->ops->map_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + consume_skb(skb); + return 0; +} + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, @@ -4024,19 +4095,30 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, struct bpf_map *map) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err = 0; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (map->map_type == BPF_MAP_TYPE_DEVMAP || map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { - struct bpf_dtab_netdev *dst = fwd; + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) { + struct bpf_dtab_netdev *dst = fwd; + + err = dev_map_generic_redirect(dst, skb, xdp_prog); + } else { + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, + ex_map, ri->flags); + } - err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { @@ -4150,6 +4232,36 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { .arg3_type = ARG_ANYTHING, }; +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, + struct bpf_map *, ex_map, u64, flags) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + + /* Limit ex_map type to DEVMAP_HASH to get better performance */ + if (unlikely((ex_map && ex_map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) || + flags & ~BPF_F_EXCLUDE_INGRESS)) + return XDP_ABORTED; + + ri->tgt_index = 0; + /* Set the tgt_value to NULL to distinguish with bpf_xdp_redirect_map */ + ri->tgt_value = NULL; + ri->flags = flags; + ri->ex_map = ex_map; + + WRITE_ONCE(ri->map, map); + + return XDP_REDIRECT; +} + +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { + .func = bpf_xdp_redirect_map_multi, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_CONST_MAP_PTR, + .arg2_type = ARG_CONST_MAP_PTR_OR_NULL, + .arg3_type = ARG_ANYTHING, +}; + static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, unsigned long off, unsigned long len) { @@ -7248,6 +7360,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_redirect_proto; case BPF_FUNC_redirect_map: return &bpf_xdp_redirect_map_proto; + case BPF_FUNC_redirect_map_multi: + return &bpf_xdp_redirect_map_multi_proto; case BPF_FUNC_xdp_adjust_tail: return &bpf_xdp_adjust_tail_proto; case BPF_FUNC_fib_lookup: diff --git a/net/core/xdp.c b/net/core/xdp.c index 0d2630a35c3e..b44d05d24a27 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -572,3 +572,32 @@ struct sk_buff *xdp_build_skb_from_frame(struct xdp_frame *xdpf, return __xdp_build_skb_from_frame(xdpf, skb, dev); } EXPORT_SYMBOL_GPL(xdp_build_skb_from_frame); + +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) +{ + unsigned int headroom, totalsize; + struct xdp_frame *nxdpf; + struct page *page; + void *addr; + + headroom = xdpf->headroom + sizeof(*xdpf); + totalsize = headroom + xdpf->len; + + if (unlikely(totalsize > PAGE_SIZE)) + return NULL; + page = dev_alloc_page(); + if (!page) + return NULL; + addr = page_to_virt(page); + + memcpy(addr, xdpf, totalsize); + + nxdpf = addr; + nxdpf->data = addr + headroom; + nxdpf->frame_sz = PAGE_SIZE; + nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0; + nxdpf->mem.id = 0; + + return nxdpf; +} +EXPORT_SYMBOL_GPL(xdpf_clone); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index c001766adcbc..ef943c024322 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3836,6 +3836,28 @@ union bpf_attr { * Return * A pointer to a struct socket on success or NULL if the file is * not a socket. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. To get better performance, the + * *ex_map* is limited to BPF_MAP_TYPE_DEVMAP_HASH and must be + * keyed by ifindex for the helper to work. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -4001,6 +4023,7 @@ union bpf_attr { FN(ktime_get_coarse_ns), \ FN(ima_inode_hash), \ FN(sock_from_file), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -4177,6 +4200,11 @@ enum { BPF_F_BPRM_SECUREEXEC = (1ULL << 0), }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv15 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test 2021-01-20 2:25 ` [PATCHv15 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (2 preceding siblings ...) 2021-01-20 2:25 ` [PATCHv15 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2021-01-20 2:25 ` Hangbin Liu 2021-01-20 2:25 ` [PATCHv15 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu ` (2 subsequent siblings) 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-20 2:25 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Hangbin Liu This is a sample for xdp multicast. In the sample we could forward all packets between given interfaces. There is also an option -X that could enable 2nd xdp_prog on egress interface. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v15: use bpf_object__find_program_by_name() instead of bpf_object__find_program_by_title() v13-v14: no update, only rebase the code v12: add devmap xdp_prog on egress support v10-v11: no update v9: use NULL directly for arg2 and redefine the maps with btf format v5: add a null_map as we have strict the arg2 to ARG_CONST_MAP_PTR. Move the testing part to bpf selftest in next patch. v4: no update. v3: add rxcnt map to show the packet transmit speed. v2: no update. --- samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 87 +++++++ samples/bpf/xdp_redirect_map_multi_user.c | 302 ++++++++++++++++++++++ 3 files changed, 392 insertions(+) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 26fc96ca619e..200029fcf53c 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map tprogs-y += per_socket_stats_example tprogs-y += xdp_redirect tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_map_multi tprogs-y += xdp_redirect_cpu tprogs-y += xdp_monitor tprogs-y += xdp_rxq_info @@ -99,6 +100,7 @@ test_map_in_map-objs := test_map_in_map_user.o per_socket_stats_example-objs := cookie_uid_helper_example.o xdp_redirect-objs := xdp_redirect_user.o xdp_redirect_map-objs := xdp_redirect_map_user.o +xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o xdp_redirect_cpu-objs := xdp_redirect_cpu_user.o xdp_monitor-objs := xdp_monitor_user.o xdp_rxq_info-objs := xdp_rxq_info_user.o @@ -160,6 +162,7 @@ always-y += tcp_tos_reflect_kern.o always-y += tcp_dumpstats_kern.o always-y += xdp_redirect_kern.o always-y += xdp_redirect_map_kern.o +always-y += xdp_redirect_map_multi_kern.o always-y += xdp_redirect_cpu_kern.o always-y += xdp_monitor_kern.o always-y += xdp_rxq_info_kern.o diff --git a/samples/bpf/xdp_redirect_map_multi_kern.c b/samples/bpf/xdp_redirect_map_multi_kern.c new file mode 100644 index 000000000000..e422340d1251 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_kern.c @@ -0,0 +1,87 @@ +// SPDX-License-Identifier: GPL-2.0 +#define KBUILD_MODNAME "foo" +#include <uapi/linux/bpf.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/ip.h> +#include <linux/ipv6.h> +#include <bpf/bpf_helpers.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 32); +} forward_map_general SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(struct bpf_devmap_val)); + __uint(max_entries, 32); +} forward_map_native SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __type(key, u32); + __type(value, long); + __uint(max_entries, 1); +} rxcnt SEC(".maps"); + +/* map to store egress interfaces mac addresses, set the + * max_entries to 1 and extend it in user sapce prog. + */ +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, u32); + __type(value, __be64); + __uint(max_entries, 1); +} mac_map SEC(".maps"); + +static int xdp_redirect_map(struct xdp_md *ctx, void *forward_map) +{ + long *value; + u32 key = 0; + + /* count packet in global counter */ + value = bpf_map_lookup_elem(&rxcnt, &key); + if (value) + *value += 1; + + return bpf_redirect_map_multi(forward_map, NULL, BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_redirect_general") +int xdp_redirect_map_general(struct xdp_md *ctx) +{ + return xdp_redirect_map(ctx, &forward_map_general); +} + +SEC("xdp_redirect_native") +int xdp_redirect_map_native(struct xdp_md *ctx) +{ + return xdp_redirect_map(ctx, &forward_map_native); +} + +SEC("xdp_devmap/map_prog") +int xdp_devmap_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + u32 key = ctx->egress_ifindex; + struct ethhdr *eth = data; + __be64 *mac; + u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + mac = bpf_map_lookup_elem(&mac_map, &key); + if (mac) + __builtin_memcpy(eth->h_source, mac, ETH_ALEN); + + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/samples/bpf/xdp_redirect_map_multi_user.c b/samples/bpf/xdp_redirect_map_multi_user.c new file mode 100644 index 000000000000..84cdbbed20b7 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_user.c @@ -0,0 +1,302 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> +#include <sys/resource.h> +#include <sys/ioctl.h> +#include <sys/types.h> +#include <sys/socket.h> +#include <netinet/in.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; +static int rxcnt_map_fd; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void poll_stats(int interval) +{ + unsigned int nr_cpus = bpf_num_possible_cpus(); + __u64 values[nr_cpus], prev[nr_cpus]; + + memset(prev, 0, sizeof(prev)); + + while (1) { + __u64 sum = 0; + __u32 key = 0; + int i; + + sleep(interval); + assert(bpf_map_lookup_elem(rxcnt_map_fd, &key, values) == 0); + for (i = 0; i < nr_cpus; i++) + sum += (values[i] - prev[i]); + if (sum) + printf("Forwarding %10llu pkt/s\n", sum / interval); + memcpy(prev, values, sizeof(values)); + } +} + +static int get_mac_addr(unsigned int ifindex, void *mac_addr) +{ + char ifname[IF_NAMESIZE]; + struct ifreq ifr; + int fd, ret = -1; + + fd = socket(AF_INET, SOCK_DGRAM, 0); + if (fd < 0) + return ret; + + if (!if_indextoname(ifindex, ifname)) + goto err_out; + + strcpy(ifr.ifr_name, ifname); + + if (ioctl(fd, SIOCGIFHWADDR, &ifr) != 0) + goto err_out; + + memcpy(mac_addr, ifr.ifr_hwaddr.sa_data, 6 * sizeof(char)); + ret = 0; + +err_out: + close(fd); + return ret; +} + +static int update_mac_map(struct bpf_object *obj) +{ + int i, ret = -1, mac_map_fd; + unsigned char mac_addr[6]; + unsigned int ifindex; + + mac_map_fd = bpf_object__find_map_fd_by_name(obj, "mac_map"); + if (mac_map_fd < 0) { + printf("find mac map fd failed\n"); + return ret; + } + + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + ret = get_mac_addr(ifindex, mac_addr); + if (ret < 0) { + printf("get interface %d mac failed\n", ifindex); + return ret; + } + + ret = bpf_map_update_elem(mac_map_fd, &ifindex, mac_addr, 0); + if (ret) { + perror("bpf_update_elem mac_map_fd"); + return ret; + } + } + + return 0; +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n" + " -X load xdp program on egress\n", + prog); +} + +int main(int argc, char **argv) +{ + int i, ret, opt, forward_map_fd, max_ifindex = 0; + struct bpf_program *ingress_prog, *egress_prog; + int ingress_prog_fd, egress_prog_fd = 0; + struct bpf_devmap_val devmap_val; + bool attach_egress_prog = false; + char ifname[IF_NAMESIZE]; + struct bpf_map *mac_map; + struct bpf_object *obj; + unsigned int ifindex; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNFX")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + case 'X': + attach_egress_prog = true; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) { + xdp_flags |= XDP_FLAGS_DRV_MODE; + } else if (attach_egress_prog) { + printf("Load xdp program on egress with SKB mode not supported yet\n"); + return 1; + } + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + + /* Find the largest index number */ + if (ifaces[i] > max_ifindex) + max_ifindex = ifaces[i]; + + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + + obj = bpf_object__open(filename); + if (libbpf_get_error(obj)) { + printf("ERROR: opening BPF object file failed\n"); + obj = NULL; + goto err_out; + } + + /* Reset the map size to max ifindex + 1 */ + if (attach_egress_prog) { + mac_map = bpf_object__find_map_by_name(obj, "mac_map"); + ret = bpf_map__resize(mac_map, max_ifindex + 1); + if (ret < 0) { + printf("ERROR: reset mac map size failed\n"); + goto err_out; + } + } + + /* load BPF program */ + if (bpf_object__load(obj)) { + printf("ERROR: loading BPF object file failed\n"); + goto err_out; + } + + if (xdp_flags & XDP_FLAGS_SKB_MODE) { + ingress_prog = bpf_object__find_program_by_name(obj, "xdp_redirect_map_general"); + forward_map_fd = bpf_object__find_map_fd_by_name(obj, "forward_map_general"); + } else { + ingress_prog = bpf_object__find_program_by_name(obj, "xdp_redirect_map_native"); + forward_map_fd = bpf_object__find_map_fd_by_name(obj, "forward_map_native"); + } + if (!ingress_prog || forward_map_fd < 0) { + printf("finding ingress_prog/forward_map in obj file failed\n"); + goto err_out; + } + + ingress_prog_fd = bpf_program__fd(ingress_prog); + if (ingress_prog_fd < 0) { + printf("find ingress_prog fd failed\n"); + goto err_out; + } + + rxcnt_map_fd = bpf_object__find_map_fd_by_name(obj, "rxcnt"); + if (rxcnt_map_fd < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + goto err_out; + } + + if (attach_egress_prog) { + /* Update mac_map with all egress interfaces' mac addr */ + if (update_mac_map(obj) < 0) { + printf("Error: update mac map failed"); + goto err_out; + } + + /* Find egress prog fd */ + egress_prog = bpf_object__find_program_by_name(obj, "xdp_devmap_prog"); + if (!egress_prog) { + printf("finding egress_prog in obj file failed\n"); + goto err_out; + } + egress_prog_fd = bpf_program__fd(egress_prog); + if (egress_prog_fd < 0) { + printf("find egress_prog fd failed\n"); + goto err_out; + } + } + + /* Remove attached program when program is interrupted or killed */ + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, ingress_prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + /* Add all the interfaces to forward group and attach + * egress devmap programe if exist + */ + devmap_val.ifindex = ifindex; + devmap_val.bpf_prog.fd = egress_prog_fd; + ret = bpf_map_update_elem(forward_map_fd, &ifindex, &devmap_val, 0); + if (ret) { + perror("bpf_map_update_elem forward_map"); + goto err_out; + } + } + + poll_stats(2); + + return 0; + +err_out: + return 1; +} -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv15 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL 2021-01-20 2:25 ` [PATCHv15 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (3 preceding siblings ...) 2021-01-20 2:25 ` [PATCHv15 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu @ 2021-01-20 2:25 ` Hangbin Liu 2021-01-20 2:25 ` [PATCHv15 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2021-01-22 7:46 ` [PATCHv16 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-20 2:25 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Hangbin Liu Use helper bpf_redirect_map() and bpf_redirect_map_multi() to test bpf arg ARG_CONST_MAP_PTR and ARG_CONST_MAP_PTR_OR_NULL. Make sure the map arg could be verified correctly when it is NULL or valid map pointer. Add devmap and devmap_hash in struct bpf_test due to bpf_redirect_{map, map_multi} limit. Test result: ]# ./test_verifier 713 716 #713/p ARG_CONST_MAP_PTR: null pointer OK #714/p ARG_CONST_MAP_PTR: valid map pointer OK #715/p ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map OK #716/p ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map OK Summary: 4 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- tools/testing/selftests/bpf/test_verifier.c | 22 +++++- .../testing/selftests/bpf/verifier/map_ptr.c | 70 +++++++++++++++++++ 2 files changed, 91 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c index 777a81404fdb..17eb3958ce6d 100644 --- a/tools/testing/selftests/bpf/test_verifier.c +++ b/tools/testing/selftests/bpf/test_verifier.c @@ -50,7 +50,7 @@ #define MAX_INSNS BPF_MAXINSNS #define MAX_TEST_INSNS 1000000 #define MAX_FIXUPS 8 -#define MAX_NR_MAPS 20 +#define MAX_NR_MAPS 22 #define MAX_TEST_RUNS 8 #define POINTER_VALUE 0xcafe4all #define TEST_DATA_LEN 64 @@ -87,6 +87,8 @@ struct bpf_test { int fixup_sk_storage_map[MAX_FIXUPS]; int fixup_map_event_output[MAX_FIXUPS]; int fixup_map_reuseport_array[MAX_FIXUPS]; + int fixup_map_devmap[MAX_FIXUPS]; + int fixup_map_devmap_hash[MAX_FIXUPS]; const char *errstr; const char *errstr_unpriv; uint32_t insn_processed; @@ -640,6 +642,8 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, int *fixup_sk_storage_map = test->fixup_sk_storage_map; int *fixup_map_event_output = test->fixup_map_event_output; int *fixup_map_reuseport_array = test->fixup_map_reuseport_array; + int *fixup_map_devmap = test->fixup_map_devmap; + int *fixup_map_devmap_hash = test->fixup_map_devmap_hash; if (test->fill_helper) { test->fill_insns = calloc(MAX_TEST_INSNS, sizeof(struct bpf_insn)); @@ -817,6 +821,22 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, fixup_map_reuseport_array++; } while (*fixup_map_reuseport_array); } + if (*fixup_map_devmap) { + map_fds[20] = __create_map(BPF_MAP_TYPE_DEVMAP, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap].imm = map_fds[20]; + fixup_map_devmap++; + } while (*fixup_map_devmap); + } + if (*fixup_map_devmap_hash) { + map_fds[21] = __create_map(BPF_MAP_TYPE_DEVMAP_HASH, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap_hash].imm = map_fds[21]; + fixup_map_devmap_hash++; + } while (*fixup_map_devmap_hash); + } } struct libcap { diff --git a/tools/testing/selftests/bpf/verifier/map_ptr.c b/tools/testing/selftests/bpf/verifier/map_ptr.c index b117bdd3806d..1a532198c9c1 100644 --- a/tools/testing/selftests/bpf/verifier/map_ptr.c +++ b/tools/testing/selftests/bpf/verifier/map_ptr.c @@ -93,3 +93,73 @@ .fixup_map_hash_16b = { 4 }, .result = ACCEPT, }, +{ + "ARG_CONST_MAP_PTR: null pointer", + .insns = { + /* bpf_redirect_map arg1 (map) */ + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .result = REJECT, + .prog_type = BPF_PROG_TYPE_XDP, + .errstr = "R1 type=inv expected=map_ptr", +}, +{ + "ARG_CONST_MAP_PTR: valid map pointer", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg1 (map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_LD_MAP_FD(BPF_REG_2, 1), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .fixup_map_devmap_hash = { 3 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv15 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test 2021-01-20 2:25 ` [PATCHv15 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (4 preceding siblings ...) 2021-01-20 2:25 ` [PATCHv15 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2021-01-20 2:25 ` Hangbin Liu 2021-01-22 7:46 ` [PATCHv16 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 6 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-20 2:25 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Hangbin Liu Add a bpf selftest for new helper xdp_redirect_map_multi(). In this test we have 3 forward groups and 1 exclude group. The test will redirect each interface's packets to all the interfaces in the forward group, and exclude the interface in exclude map. We will also test both DEVMAP and DEVMAP_HASH with xdp generic and drv. For more test details, you can find it in the test script. Here is the test result. ]# ./test_xdp_redirect_multi.sh Pass: xdpgeneric arp ns1-2 Pass: xdpgeneric arp ns1-3 Pass: xdpgeneric arp ns1-4 Pass: xdpgeneric ping ns1-2 Pass: xdpgeneric ping ns1-3 Pass: xdpgeneric ping ns1-4 Pass: xdpgeneric ping6 ns2-1 Pass: xdpgeneric ping6 ns2-3 Pass: xdpgeneric ping6 ns2-4 Pass: xdpdrv arp ns1-2 Pass: xdpdrv arp ns1-3 Pass: xdpdrv arp ns1-4 Pass: xdpdrv ping ns1-2 Pass: xdpdrv ping ns1-3 Pass: xdpdrv ping ns1-4 Pass: xdpdrv ping6 ns2-1 Pass: xdpdrv ping6 ns2-3 Pass: xdpdrv ping6 ns2-4 Pass: xdpegress mac ns1-2 Pass: xdpegress mac ns1-3 Pass: xdpegress mac ns1-4 Pass: xdpegress ping ns1-2 Pass: xdpegress ping ns1-3 Pass: xdpegress ping ns1-4 Summary: PASS 24, FAIL 0 Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v15: use bpf_object__find_program_by_name instead of bpf_object__find_program_by_title v14: no update, only rebase the code v13: remove setrlimit v12: add devmap prog test on egress v9: use NULL directly for arg2 and redefine the maps with btf format --- tools/testing/selftests/bpf/Makefile | 3 +- .../bpf/progs/xdp_redirect_multi_kern.c | 111 ++++++++ .../selftests/bpf/test_xdp_redirect_multi.sh | 208 +++++++++++++++ .../selftests/bpf/xdp_redirect_multi.c | 252 ++++++++++++++++++ 4 files changed, 573 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 63d6288e419c..621dceddb249 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -51,6 +51,7 @@ TEST_FILES = xsk_prereqs.sh \ # Order correspond to 'make run_tests' order TEST_PROGS := test_kmod.sh \ test_xdp_redirect.sh \ + test_xdp_redirect_multi.sh \ test_xdp_meta.sh \ test_xdp_veth.sh \ test_offload.py \ @@ -80,7 +81,7 @@ TEST_PROGS_EXTENDED := with_addr.sh \ TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ test_lirc_mode2_user xdping test_cpp runqslower bench bpf_testmod.ko \ - xdpxceiver + xdpxceiver xdp_redirect_multi TEST_CUSTOM_PROGS = $(OUTPUT)/urandom_read diff --git a/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c new file mode 100644 index 000000000000..dce4df40d9de --- /dev/null +++ b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c @@ -0,0 +1,111 @@ +// SPDX-License-Identifier: GPL-2.0 +#define KBUILD_MODNAME "foo" +#include <string.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ip.h> +#include <linux/ipv6.h> + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_endian.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v4 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v6 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_all SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(struct bpf_devmap_val)); + __uint(max_entries, 128); +} forward_map_egress SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} exclude_map SEC(".maps"); + +/* map to store egress interfaces mac addresses */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, __u32); + __type(value, __be64); + __uint(max_entries, 128); +} mac_map SEC(".maps"); + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + struct ethhdr *eth = data; + __u16 h_proto; + __u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + h_proto = eth->h_proto; + + if (h_proto == bpf_htons(ETH_P_IP)) + return bpf_redirect_map_multi(&forward_map_v4, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else if (h_proto == bpf_htons(ETH_P_IPV6)) + return bpf_redirect_map_multi(&forward_map_v6, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else + return bpf_redirect_map_multi(&forward_map_all, NULL, + BPF_F_EXCLUDE_INGRESS); +} + +/* The following 2 progs are for 2nd devmap prog testing */ +SEC("xdp_redirect_map_ingress") +int xdp_redirect_map_all_prog(struct xdp_md *ctx) +{ + return bpf_redirect_map_multi(&forward_map_egress, NULL, BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_devmap/map_prog") +int xdp_devmap_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + __u32 key = ctx->egress_ifindex; + struct ethhdr *eth = data; + __u64 nh_off; + __be64 *mac; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + mac = bpf_map_lookup_elem(&mac_map, &key); + if (mac) + __builtin_memcpy(eth->h_source, mac, ETH_ALEN); + + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh new file mode 100755 index 000000000000..6503751fdca5 --- /dev/null +++ b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh @@ -0,0 +1,208 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Test topology: +# - - - - - - - - - - - - - - - - - - - - - - - - - +# | veth1 veth2 veth3 veth4 | ... init net +# - -| - - - - - - | - - - - - - | - - - - - - | - - +# --------- --------- --------- --------- +# | veth0 | | veth0 | | veth0 | | veth0 | ... +# --------- --------- --------- --------- +# ns1 ns2 ns3 ns4 +# +# Forward maps: +# Forward map_all has interfaces: veth1, veth2, veth3, veth4, ... (All traffic except IPv4, IPv6) +# Forward map_v4 has interfaces: veth1, veth3, veth4, ... (For IPv4 traffic only) +# Forward map_v6 has interfaces: veth2, veth3, veth4, ... (For IPv6 traffic only) +# Forward map_egress has all interfaces and redirect all pkts +# Exclude Groups: +# Exclude map: veth3 (assume ns3 is in black list) +# Map type: +# map_v4 use DEVMAP, others use DEVMAP_HASH +# +# Test modules: +# XDP modes: generic, native, native + egress_prog +# +# Test cases: +# ARP(we didn't block ARP for ns3): +# ns1 -> gw: ns2, ns3, ns4 should receive the arp request +# IPv4: +# ns1 -> ns2 (block), ns1 -> ns3 (block), ns1 -> ns4 (pass) +# IPv6 +# ns2 -> ns1 (block), ns2 -> ns3 (block), ns2 -> ns4 (pass) +# egress_prog: +# all ping test should pass, the src mac should be egress interface's mac +# + + +# netns numbers +NUM=4 +IFACES="" +DRV_MODE="xdpgeneric xdpdrv xdpegress" +PASS=0 +FAIL=0 + +test_pass() +{ + echo "Pass: $@" + PASS=$((PASS + 1)) +} + +test_fail() +{ + echo "fail: $@" + FAIL=$((FAIL + 1)) +} + +clean_up() +{ + for i in $(seq $NUM); do + ip link del veth$i 2> /dev/null + ip netns del ns$i 2> /dev/null + done +} + +# Kselftest framework requirement - SKIP code is 4. +check_env() +{ + ip link set dev lo xdpgeneric off &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without the ip xdpgeneric support" + exit 4 + fi + + which tcpdump &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without tcpdump" + exit 4 + fi +} + +setup_ns() +{ + local mode=$1 + IFACES="" + + if [ "$mode" = "xdpegress" ]; then + mode="xdpdrv" + fi + + for i in $(seq $NUM); do + ip netns add ns$i + ip link add veth$i type veth peer name veth0 netns ns$i + ip link set veth$i up + ip -n ns$i link set veth0 up + + ip -n ns$i addr add 192.0.2.$i/24 dev veth0 + ip -n ns$i addr add 2001:db8::$i/64 dev veth0 + ip -n ns$i link set veth0 $mode obj \ + xdp_dummy.o sec xdp_dummy &> /dev/null || \ + { test_fail "Unable to load dummy xdp" && exit 1; } + IFACES="$IFACES veth$i" + veth_mac[$i]=$(ip link show veth$i | awk '/link\/ether/ {print $2}') + done +} + +do_egress_tests() +{ + local mode=$1 + + # mac test + ip netns exec ns2 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + + # mac check + grep -q "${veth_mac[2]} > ff:ff:ff:ff:ff:ff" mac_ns1-2_${mode}.log && \ + test_pass "$mode mac ns1-2" || test_fail "$mode mac ns1-2" + grep -q "${veth_mac[3]} > ff:ff:ff:ff:ff:ff" mac_ns1-3_${mode}.log && \ + test_pass "$mode mac ns1-3" || test_fail "$mode mac ns1-3" + grep -q "${veth_mac[4]} > ff:ff:ff:ff:ff:ff" mac_ns1-4_${mode}.log && \ + test_pass "$mode mac ns1-4" || test_fail "$mode mac ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-2" || test_fail "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-3" || test_fail "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" +} + +do_ping_tests() +{ + local mode=$1 + + # arp test + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -i veth0 -nn -l -e &> arp_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-2_${mode}.log && \ + test_pass "$mode arp ns1-2" || test_fail "$mode arp ns1-2" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-3_${mode}.log && \ + test_pass "$mode arp ns1-3" || test_fail "$mode arp ns1-3" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-4_${mode}.log && \ + test_pass "$mode arp ns1-4" || test_fail "$mode arp ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-2" || test_pass "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-3" || test_pass "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" + + # ping6 test + ip netns exec ns2 ping6 2001:db8::1 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-1" || test_pass "$mode ping6 ns2-1" + ip netns exec ns2 ping6 2001:db8::3 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-3" || test_pass "$mode ping6 ns2-3" + ip netns exec ns2 ping6 2001:db8::4 -c 4 &> /dev/null && \ + test_pass "$mode ping6 ns2-4" || test_fail "$mode ping6 ns2-4" +} + +do_tests() +{ + local mode=$1 + local drv_p + + case ${mode} in + xdpdrv) drv_p="-N";; + xdpegress) drv_p="-X";; + xdpgeneric) drv_p="-S";; + esac + + ./xdp_redirect_multi $drv_p $IFACES &> xdp_redirect_${mode}.log & + xdp_pid=$! + sleep 10 + + if [ "$mode" = "xdpegress" ]; then + do_egress_tests $mode + else + do_ping_tests $mode + fi + + kill $xdp_pid +} + +trap clean_up 0 2 3 6 9 + +check_env +rm -f xdp_redirect_*.log arp_ns*.log mac_ns*.log + +for mode in ${DRV_MODE}; do + setup_ns $mode + do_tests $mode + sleep 10 + clean_up + sleep 5 +done + +echo "Summary: PASS $PASS, FAIL $FAIL" +[ $FAIL -eq 0 ] && exit 0 || exit 1 diff --git a/tools/testing/selftests/bpf/xdp_redirect_multi.c b/tools/testing/selftests/bpf/xdp_redirect_multi.c new file mode 100644 index 000000000000..b43cd3c9eefd --- /dev/null +++ b/tools/testing/selftests/bpf/xdp_redirect_multi.c @@ -0,0 +1,252 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> +#include <sys/resource.h> +#include <sys/ioctl.h> +#include <sys/types.h> +#include <sys/socket.h> +#include <netinet/in.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static int get_mac_addr(unsigned int ifindex, void *mac_addr) +{ + char ifname[IF_NAMESIZE]; + struct ifreq ifr; + int fd, ret = -1; + + fd = socket(AF_INET, SOCK_DGRAM, 0); + if (fd < 0) + return ret; + + if (!if_indextoname(ifindex, ifname)) + goto err_out; + + strcpy(ifr.ifr_name, ifname); + + if (ioctl(fd, SIOCGIFHWADDR, &ifr) != 0) + goto err_out; + + memcpy(mac_addr, ifr.ifr_hwaddr.sa_data, 6 * sizeof(char)); + ret = 0; + +err_out: + close(fd); + return ret; +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n" + " -X load xdp program on egress\n", + prog); +} + +int main(int argc, char **argv) +{ + int prog_fd, group_all, group_v4, group_v6, exclude, mac_map; + struct bpf_program *ingress_prog, *egress_prog; + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_UNSPEC, + }; + int i, ret, opt, egress_prog_fd = 0; + struct bpf_devmap_val devmap_val; + bool attach_egress_prog = false; + unsigned char mac_addr[6]; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + unsigned int ifindex; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNFX")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + case 'X': + attach_egress_prog = true; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) { + xdp_flags |= XDP_FLAGS_DRV_MODE; + } else if (attach_egress_prog) { + printf("Load xdp program on egress with SKB mode not supported yet\n"); + goto err_out; + } + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + goto err_out; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + goto err_out; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + goto err_out; + + if (attach_egress_prog) + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_egress"); + else + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_all"); + group_v4 = bpf_object__find_map_fd_by_name(obj, "forward_map_v4"); + group_v6 = bpf_object__find_map_fd_by_name(obj, "forward_map_v6"); + exclude = bpf_object__find_map_fd_by_name(obj, "exclude_map"); + mac_map = bpf_object__find_map_fd_by_name(obj, "mac_map"); + + if (group_all < 0 || group_v4 < 0 || group_v6 < 0 || exclude < 0 || + mac_map < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + goto err_out; + } + + if (attach_egress_prog) { + /* Find ingress/egress prog for 2nd xdp prog */ + ingress_prog = bpf_object__find_program_by_name(obj, "xdp_redirect_map_all_prog"); + egress_prog = bpf_object__find_program_by_name(obj, "xdp_devmap_prog"); + if (!ingress_prog || !egress_prog) { + printf("finding ingress/egress_prog in obj file failed\n"); + goto err_out; + } + prog_fd = bpf_program__fd(ingress_prog); + egress_prog_fd = bpf_program__fd(egress_prog); + if (prog_fd < 0 || egress_prog_fd < 0) { + printf("find egress_prog fd failed\n"); + goto err_out; + } + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + if (attach_egress_prog) { + ret = get_mac_addr(ifindex, mac_addr); + if (ret < 0) { + printf("get interface %d mac failed\n", ifindex); + goto err_out; + } + ret = bpf_map_update_elem(mac_map, &ifindex, mac_addr, 0); + if (ret) { + perror("bpf_update_elem mac_map failed\n"); + goto err_out; + } + } + + /* Add all the interfaces to group all */ + devmap_val.ifindex = ifindex; + devmap_val.bpf_prog.fd = egress_prog_fd; + ret = bpf_map_update_elem(group_all, &ifindex, &devmap_val, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* For testing: remove the 1st interfaces from group v6 */ + if (i != 0) { + ret = bpf_map_update_elem(group_v6, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: remove the 2nd interfaces from group v4 */ + if (i != 1) { + ret = bpf_map_update_elem(group_v4, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: add the 3rd interfaces to exclude map */ + if (i == 2) { + ret = bpf_map_update_elem(exclude, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + } + + /* sleep some time for testing */ + sleep(999); + + return 0; + +err_out: + return 1; +} -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv16 bpf-next 0/6] xdp: add a new helper for dev map multicast support 2021-01-20 2:25 ` [PATCHv15 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (5 preceding siblings ...) 2021-01-20 2:25 ` [PATCHv15 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu @ 2021-01-22 7:46 ` Hangbin Liu 2021-01-22 7:46 ` [PATCHv16 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu ` (7 more replies) 6 siblings, 8 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-22 7:46 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because there may have multi interfaces you want to exclude. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. The 1st patch is Jesper's run devmap xdp_prog later in bulking step. The 2st patch add a new bpf arg to allow NULL map pointer. The 3rd patch add the new bpf_redirect_map_multi() helper. The 4-6 patches are for usage sample and testing purpose. I did same perf tests with the following topo: --------------------- --------------------- | Host A (i40e 10G) | ---------- | eno1(i40e 10G) | --------------------- | | | Host B | --------------------- | | | Host C (i40e 10G) | ---------- | eno2(i40e 10G) | --------------------- vlan2 | -------- | | veth1 -- | veth0| | | -------- | --------------------| On Host A: # pktgen/pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -s 64 On Host B(Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, 128G Memory): Use xdp_redirect_map and xdp_redirect_map_multi in samples/bpf for testing. The veth0 in netns load dummy drop program. The forward_map max_entries in xdp_redirect_map_multi is modify to 4. Here is the perf result with 5.10 rc6: The are about +/- 0.1M deviation for native testing Version | Test | Generic | Native | Native + 2nd 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M 5.10 rc6 + patch1 | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch1 | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M 5.10 rc6 + patch1-6 | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch1-6 | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->i40e | 1.7M | 7.8M | 6.4M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->veth | 1.4M | 9.3M | 7.5M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->i40e+veth | 1.0M | 3.2M | 2.7M Last but not least, thanks a lot to Toke, Jesper, Jiri and Eelco for suggestions and help on implementation. [0] https://xdp-project.net/#Handling-multicast v16: refactor bq_xmit_all logic and remove error label for patch 01 v15: Update bq_xmit_all() logic for patch 01. Add some comments and remove useless variable for patch 03. Use bpf_object__find_program_by_title() for patch 04 and 06. v14: No code update, just rebase the code on latest bpf-next v13: Pass in xdp_prog through __xdp_enqueue() for patch 01. Update related code in patch 03. v12: Add Jesper's xdp_prog patch, rebase my works on this and latest bpf-next Add 2nd xdp_prog test on the sample and selftests. v11: Fix bpf_redirect_map_multi() helper description typo. Add loop limit for devmap_get_next_obj() and dev_map_redirect_multi(). v10: Rebase the code to latest bpf-next. Update helper bpf_xdp_redirect_map_multi() - No need to check map pointer as we will do the check in verifier. v9: Update helper bpf_xdp_redirect_map_multi() - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: a) Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). f) Split the tests from sample and add a bpf kernel selftest patch. v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Hangbin Liu (5): bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL xdp: add a new helper for dev map multicast support sample/bpf: add xdp_redirect_map_multicast test selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL selftests/bpf: add xdp_redirect_multi test Jesper Dangaard Brouer (1): bpf: run devmap xdp_prog on flush instead of bulk enqueue include/linux/bpf.h | 21 ++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 28 ++ kernel/bpf/devmap.c | 252 ++++++++++++--- kernel/bpf/verifier.c | 16 +- net/core/filter.c | 124 ++++++- net/core/xdp.c | 29 ++ samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 87 +++++ samples/bpf/xdp_redirect_map_multi_user.c | 302 ++++++++++++++++++ tools/include/uapi/linux/bpf.h | 28 ++ tools/testing/selftests/bpf/Makefile | 3 +- .../bpf/progs/xdp_redirect_multi_kern.c | 111 +++++++ tools/testing/selftests/bpf/test_verifier.c | 21 +- .../selftests/bpf/test_xdp_redirect_multi.sh | 208 ++++++++++++ .../testing/selftests/bpf/verifier/map_ptr.c | 70 ++++ .../selftests/bpf/xdp_redirect_multi.c | 252 +++++++++++++++ 18 files changed, 1497 insertions(+), 60 deletions(-) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c -- 2.26.2 ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv16 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-22 7:46 ` [PATCHv16 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2021-01-22 7:46 ` Hangbin Liu 2021-01-22 10:50 ` Maciej Fijalkowski 2021-01-22 7:46 ` [PATCHv16 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu ` (6 subsequent siblings) 7 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2021-01-22 7:46 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu From: Jesper Dangaard Brouer <brouer@redhat.com> This changes the devmap XDP program support to run the program when the bulk queue is flushed instead of before the frame is enqueued. This has a couple of benefits: - It "sorts" the packets by destination devmap entry, and then runs the same BPF program on all the packets in sequence. This ensures that we keep the XDP program and destination device properties hot in I-cache. - It makes the multicast implementation simpler because it can just enqueue packets using bq_enqueue() without having to deal with the devmap program at all. The drawback is that if the devmap program drops the packet, the enqueue step is redundant. However, arguably this is mostly visible in a micro-benchmark, and with more mixed traffic the I-cache benefit should win out. The performance impact of just this patch is as follows: Using xdp_redirect_map(with a 2nd xdp_prog patch[1]) in sample/bpf and send pkts via pktgen cmd: ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 There are about +/- 0.1M deviation for native testing, the performance improved for the base-case, but some drop back with xdp devmap prog attached. Version | Test | Generic | Native | Native + 2nd xdp_prog 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M 5.10 rc6 + patch | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M [1] https://lore.kernel.org/bpf/20210122025007.2968381-1-liuhangbin@gmail.com Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v16: a) refactor bq_xmit_all logic and remove error label v15: a) do not use unlikely when checking bq->xdp_prog b) return sent frames for dev_map_bpf_prog_run() v14: no update, only rebase the code v13: pass in xdp_prog through __xdp_enqueue() v2-v12: no this patch --- kernel/bpf/devmap.c | 136 ++++++++++++++++++++++++++------------------ 1 file changed, 81 insertions(+), 55 deletions(-) diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index f6e9c68afdd4..c24fcffbbfad 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -57,6 +57,7 @@ struct xdp_dev_bulk_queue { struct list_head flush_node; struct net_device *dev; struct net_device *dev_rx; + struct bpf_prog *xdp_prog; unsigned int count; }; @@ -327,46 +328,95 @@ bool dev_map_can_have_prog(struct bpf_map *map) return false; } +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, + struct xdp_frame **frames, int n, + struct net_device *dev) +{ + struct xdp_txq_info txq = { .dev = dev }; + struct xdp_buff xdp; + int i, nframes = 0; + + for (i = 0; i < n; i++) { + struct xdp_frame *xdpf = frames[i]; + u32 act; + int err; + + xdp_convert_frame_to_buff(xdpf, &xdp); + xdp.txq = &txq; + + act = bpf_prog_run_xdp(xdp_prog, &xdp); + switch (act) { + case XDP_PASS: + err = xdp_update_frame_from_buff(&xdp, xdpf); + if (unlikely(err < 0)) + xdp_return_frame_rx_napi(xdpf); + else + frames[nframes++] = xdpf; + break; + default: + bpf_warn_invalid_xdp_action(act); + fallthrough; + case XDP_ABORTED: + trace_xdp_exception(dev, xdp_prog, act); + fallthrough; + case XDP_DROP: + xdp_return_frame_rx_napi(xdpf); + break; + } + } + return nframes; /* sent frames count */ +} + static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) { struct net_device *dev = bq->dev; - int sent = 0, drops = 0, err = 0; + unsigned int cnt = bq->count; + int drops = 0, err = 0; + int to_sent = cnt; + int sent = cnt; int i; - if (unlikely(!bq->count)) + if (unlikely(!cnt)) return; - for (i = 0; i < bq->count; i++) { + for (i = 0; i < cnt; i++) { struct xdp_frame *xdpf = bq->q[i]; prefetch(xdpf); } - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); + if (bq->xdp_prog) { + to_sent = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); + if (!to_sent) { + sent = 0; + goto out; + } + drops = cnt - to_sent; + } + + sent = dev->netdev_ops->ndo_xdp_xmit(dev, to_sent, bq->q, flags); if (sent < 0) { err = sent; sent = 0; - goto error; + + /* If ndo_xdp_xmit fails with an errno, no frames have been + * xmit'ed and it's our responsibility to them free all. + */ + for (i = 0; i < cnt - drops; i++) { + struct xdp_frame *xdpf = bq->q[i]; + + xdp_return_frame_rx_napi(xdpf); + } } - drops = bq->count - sent; out: + drops = cnt - sent; bq->count = 0; trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); bq->dev_rx = NULL; + bq->xdp_prog = NULL; __list_del_clearprev(&bq->flush_node); return; -error: - /* If ndo_xdp_xmit fails with an errno, no frames have been - * xmit'ed and it's our responsibility to them free all. - */ - for (i = 0; i < bq->count; i++) { - struct xdp_frame *xdpf = bq->q[i]; - - xdp_return_frame_rx_napi(xdpf); - drops++; - } - goto out; } /* __dev_flush is called from xdp_do_flush() which _must_ be signaled @@ -408,7 +458,7 @@ struct bpf_dtab_netdev *__dev_map_lookup_elem(struct bpf_map *map, u32 key) * Thus, safe percpu variable access. */ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, - struct net_device *dev_rx) + struct net_device *dev_rx, struct bpf_prog *xdp_prog) { struct list_head *flush_list = this_cpu_ptr(&dev_flush_list); struct xdp_dev_bulk_queue *bq = this_cpu_ptr(dev->xdp_bulkq); @@ -423,6 +473,14 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, if (!bq->dev_rx) bq->dev_rx = dev_rx; + /* Store (potential) xdp_prog that run before egress to dev as + * part of bulk_queue. This will be same xdp_prog for all + * xdp_frame's in bulk_queue, because this per-CPU store must + * be flushed from net_device drivers NAPI func end. + */ + if (!bq->xdp_prog) + bq->xdp_prog = xdp_prog; + bq->q[bq->count++] = xdpf; if (!bq->flush_node.prev) @@ -430,7 +488,8 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, } static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, - struct net_device *dev_rx) + struct net_device *dev_rx, + struct bpf_prog *xdp_prog) { struct xdp_frame *xdpf; int err; @@ -446,42 +505,14 @@ static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, if (unlikely(!xdpf)) return -EOVERFLOW; - bq_enqueue(dev, xdpf, dev_rx); + bq_enqueue(dev, xdpf, dev_rx, xdp_prog); return 0; } -static struct xdp_buff *dev_map_run_prog(struct net_device *dev, - struct xdp_buff *xdp, - struct bpf_prog *xdp_prog) -{ - struct xdp_txq_info txq = { .dev = dev }; - u32 act; - - xdp_set_data_meta_invalid(xdp); - xdp->txq = &txq; - - act = bpf_prog_run_xdp(xdp_prog, xdp); - switch (act) { - case XDP_PASS: - return xdp; - case XDP_DROP: - break; - default: - bpf_warn_invalid_xdp_action(act); - fallthrough; - case XDP_ABORTED: - trace_xdp_exception(dev, xdp_prog, act); - break; - } - - xdp_return_buff(xdp); - return NULL; -} - int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx) { - return __xdp_enqueue(dev, xdp, dev_rx); + return __xdp_enqueue(dev, xdp, dev_rx, NULL); } int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, @@ -489,12 +520,7 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, { struct net_device *dev = dst->dev; - if (dst->xdp_prog) { - xdp = dev_map_run_prog(dev, xdp, dst->xdp_prog); - if (!xdp) - return 0; - } - return __xdp_enqueue(dev, xdp, dev_rx); + return __xdp_enqueue(dev, xdp, dev_rx, dst->xdp_prog); } int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv16 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-22 7:46 ` [PATCHv16 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu @ 2021-01-22 10:50 ` Maciej Fijalkowski 2021-01-22 13:38 ` Toke Høiland-Jørgensen 0 siblings, 1 reply; 219+ messages in thread From: Maciej Fijalkowski @ 2021-01-22 10:50 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend On Fri, Jan 22, 2021 at 03:46:47PM +0800, Hangbin Liu wrote: > From: Jesper Dangaard Brouer <brouer@redhat.com> > > This changes the devmap XDP program support to run the program when the > bulk queue is flushed instead of before the frame is enqueued. This has > a couple of benefits: > > - It "sorts" the packets by destination devmap entry, and then runs the > same BPF program on all the packets in sequence. This ensures that we > keep the XDP program and destination device properties hot in I-cache. > > - It makes the multicast implementation simpler because it can just > enqueue packets using bq_enqueue() without having to deal with the > devmap program at all. > > The drawback is that if the devmap program drops the packet, the enqueue > step is redundant. However, arguably this is mostly visible in a > micro-benchmark, and with more mixed traffic the I-cache benefit should > win out. The performance impact of just this patch is as follows: > > Using xdp_redirect_map(with a 2nd xdp_prog patch[1]) in sample/bpf and send > pkts via pktgen cmd: > ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 > > There are about +/- 0.1M deviation for native testing, the performance > improved for the base-case, but some drop back with xdp devmap prog attached. > > Version | Test | Generic | Native | Native + 2nd xdp_prog > 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M > 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M > 5.10 rc6 + patch | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M > 5.10 rc6 + patch | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M > > [1] https://lore.kernel.org/bpf/20210122025007.2968381-1-liuhangbin@gmail.com > > Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com> > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > > --- > v16: > a) refactor bq_xmit_all logic and remove error label > > v15: > a) do not use unlikely when checking bq->xdp_prog > b) return sent frames for dev_map_bpf_prog_run() > > v14: no update, only rebase the code > v13: pass in xdp_prog through __xdp_enqueue() > v2-v12: no this patch > --- > kernel/bpf/devmap.c | 136 ++++++++++++++++++++++++++------------------ > 1 file changed, 81 insertions(+), 55 deletions(-) > > diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c > index f6e9c68afdd4..c24fcffbbfad 100644 > --- a/kernel/bpf/devmap.c > +++ b/kernel/bpf/devmap.c > @@ -57,6 +57,7 @@ struct xdp_dev_bulk_queue { > struct list_head flush_node; > struct net_device *dev; > struct net_device *dev_rx; > + struct bpf_prog *xdp_prog; > unsigned int count; > }; > > @@ -327,46 +328,95 @@ bool dev_map_can_have_prog(struct bpf_map *map) > return false; > } > > +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, > + struct xdp_frame **frames, int n, > + struct net_device *dev) > +{ > + struct xdp_txq_info txq = { .dev = dev }; > + struct xdp_buff xdp; > + int i, nframes = 0; > + > + for (i = 0; i < n; i++) { > + struct xdp_frame *xdpf = frames[i]; > + u32 act; > + int err; > + > + xdp_convert_frame_to_buff(xdpf, &xdp); > + xdp.txq = &txq; > + > + act = bpf_prog_run_xdp(xdp_prog, &xdp); > + switch (act) { > + case XDP_PASS: > + err = xdp_update_frame_from_buff(&xdp, xdpf); > + if (unlikely(err < 0)) > + xdp_return_frame_rx_napi(xdpf); > + else > + frames[nframes++] = xdpf; > + break; > + default: > + bpf_warn_invalid_xdp_action(act); > + fallthrough; > + case XDP_ABORTED: > + trace_xdp_exception(dev, xdp_prog, act); > + fallthrough; > + case XDP_DROP: > + xdp_return_frame_rx_napi(xdpf); > + break; > + } > + } > + return nframes; /* sent frames count */ > +} > + > static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) > { > struct net_device *dev = bq->dev; > - int sent = 0, drops = 0, err = 0; > + unsigned int cnt = bq->count; > + int drops = 0, err = 0; > + int to_sent = cnt; Hmm if I would be super picky then I'd like to have this variable as "to_send", as we spoke. Could you change that? With that, you can add my: Reviewed-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com> to next revision. > + int sent = cnt; > int i; > > - if (unlikely(!bq->count)) > + if (unlikely(!cnt)) > return; > > - for (i = 0; i < bq->count; i++) { > + for (i = 0; i < cnt; i++) { > struct xdp_frame *xdpf = bq->q[i]; > > prefetch(xdpf); > } > > - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); > + if (bq->xdp_prog) { > + to_sent = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); > + if (!to_sent) { > + sent = 0; > + goto out; > + } > + drops = cnt - to_sent; > + } > + > + sent = dev->netdev_ops->ndo_xdp_xmit(dev, to_sent, bq->q, flags); > if (sent < 0) { > err = sent; > sent = 0; > - goto error; > + > + /* If ndo_xdp_xmit fails with an errno, no frames have been > + * xmit'ed and it's our responsibility to them free all. > + */ > + for (i = 0; i < cnt - drops; i++) { > + struct xdp_frame *xdpf = bq->q[i]; > + > + xdp_return_frame_rx_napi(xdpf); > + } > } > - drops = bq->count - sent; > out: > + drops = cnt - sent; > bq->count = 0; > > trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); > bq->dev_rx = NULL; > + bq->xdp_prog = NULL; One more question, do you really have to do that per each bq_xmit_all call? Couldn't you clear it in __dev_flush ? Or IOW - what's the rationale behind storing xdp_prog in xdp_dev_bulk_queue. Why can't you propagate the dst->xdp_prog and rely on that without that local pointer? You probably have an answer for that, so maybe include it in commit message. BTW same question for clearing dev_rx. To me this will be the same for all bq_xmit_all() calls that will happen within same napi. > __list_del_clearprev(&bq->flush_node); > return; > -error: > - /* If ndo_xdp_xmit fails with an errno, no frames have been > - * xmit'ed and it's our responsibility to them free all. > - */ > - for (i = 0; i < bq->count; i++) { > - struct xdp_frame *xdpf = bq->q[i]; > - > - xdp_return_frame_rx_napi(xdpf); > - drops++; > - } > - goto out; > } > > /* __dev_flush is called from xdp_do_flush() which _must_ be signaled > @@ -408,7 +458,7 @@ struct bpf_dtab_netdev *__dev_map_lookup_elem(struct bpf_map *map, u32 key) > * Thus, safe percpu variable access. > */ > static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, > - struct net_device *dev_rx) > + struct net_device *dev_rx, struct bpf_prog *xdp_prog) > { > struct list_head *flush_list = this_cpu_ptr(&dev_flush_list); > struct xdp_dev_bulk_queue *bq = this_cpu_ptr(dev->xdp_bulkq); > @@ -423,6 +473,14 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, > if (!bq->dev_rx) > bq->dev_rx = dev_rx; > > + /* Store (potential) xdp_prog that run before egress to dev as > + * part of bulk_queue. This will be same xdp_prog for all > + * xdp_frame's in bulk_queue, because this per-CPU store must > + * be flushed from net_device drivers NAPI func end. > + */ > + if (!bq->xdp_prog) > + bq->xdp_prog = xdp_prog; > + > bq->q[bq->count++] = xdpf; > > if (!bq->flush_node.prev) > @@ -430,7 +488,8 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, > } > > static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, > - struct net_device *dev_rx) > + struct net_device *dev_rx, > + struct bpf_prog *xdp_prog) > { > struct xdp_frame *xdpf; > int err; > @@ -446,42 +505,14 @@ static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, > if (unlikely(!xdpf)) > return -EOVERFLOW; > > - bq_enqueue(dev, xdpf, dev_rx); > + bq_enqueue(dev, xdpf, dev_rx, xdp_prog); > return 0; > } > > -static struct xdp_buff *dev_map_run_prog(struct net_device *dev, > - struct xdp_buff *xdp, > - struct bpf_prog *xdp_prog) > -{ > - struct xdp_txq_info txq = { .dev = dev }; > - u32 act; > - > - xdp_set_data_meta_invalid(xdp); > - xdp->txq = &txq; > - > - act = bpf_prog_run_xdp(xdp_prog, xdp); > - switch (act) { > - case XDP_PASS: > - return xdp; > - case XDP_DROP: > - break; > - default: > - bpf_warn_invalid_xdp_action(act); > - fallthrough; > - case XDP_ABORTED: > - trace_xdp_exception(dev, xdp_prog, act); > - break; > - } > - > - xdp_return_buff(xdp); > - return NULL; > -} > - > int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, > struct net_device *dev_rx) > { > - return __xdp_enqueue(dev, xdp, dev_rx); > + return __xdp_enqueue(dev, xdp, dev_rx, NULL); > } > > int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > @@ -489,12 +520,7 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > { > struct net_device *dev = dst->dev; > > - if (dst->xdp_prog) { > - xdp = dev_map_run_prog(dev, xdp, dst->xdp_prog); > - if (!xdp) > - return 0; > - } > - return __xdp_enqueue(dev, xdp, dev_rx); > + return __xdp_enqueue(dev, xdp, dev_rx, dst->xdp_prog); > } > > int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, > -- > 2.26.2 > ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv16 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-22 10:50 ` Maciej Fijalkowski @ 2021-01-22 13:38 ` Toke Høiland-Jørgensen 2021-01-25 3:30 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2021-01-22 13:38 UTC (permalink / raw) To: Maciej Fijalkowski, Hangbin Liu Cc: bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend Maciej Fijalkowski <maciej.fijalkowski@intel.com> writes: > On Fri, Jan 22, 2021 at 03:46:47PM +0800, Hangbin Liu wrote: >> From: Jesper Dangaard Brouer <brouer@redhat.com> >> >> This changes the devmap XDP program support to run the program when the >> bulk queue is flushed instead of before the frame is enqueued. This has >> a couple of benefits: >> >> - It "sorts" the packets by destination devmap entry, and then runs the >> same BPF program on all the packets in sequence. This ensures that we >> keep the XDP program and destination device properties hot in I-cache. >> >> - It makes the multicast implementation simpler because it can just >> enqueue packets using bq_enqueue() without having to deal with the >> devmap program at all. >> >> The drawback is that if the devmap program drops the packet, the enqueue >> step is redundant. However, arguably this is mostly visible in a >> micro-benchmark, and with more mixed traffic the I-cache benefit should >> win out. The performance impact of just this patch is as follows: >> >> Using xdp_redirect_map(with a 2nd xdp_prog patch[1]) in sample/bpf and send >> pkts via pktgen cmd: >> ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 >> >> There are about +/- 0.1M deviation for native testing, the performance >> improved for the base-case, but some drop back with xdp devmap prog attached. >> >> Version | Test | Generic | Native | Native + 2nd xdp_prog >> 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M >> 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M >> 5.10 rc6 + patch | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M >> 5.10 rc6 + patch | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M >> >> [1] https://lore.kernel.org/bpf/20210122025007.2968381-1-liuhangbin@gmail.com >> >> Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com> >> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> >> >> --- >> v16: >> a) refactor bq_xmit_all logic and remove error label >> >> v15: >> a) do not use unlikely when checking bq->xdp_prog >> b) return sent frames for dev_map_bpf_prog_run() >> >> v14: no update, only rebase the code >> v13: pass in xdp_prog through __xdp_enqueue() >> v2-v12: no this patch >> --- >> kernel/bpf/devmap.c | 136 ++++++++++++++++++++++++++------------------ >> 1 file changed, 81 insertions(+), 55 deletions(-) >> >> diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c >> index f6e9c68afdd4..c24fcffbbfad 100644 >> --- a/kernel/bpf/devmap.c >> +++ b/kernel/bpf/devmap.c >> @@ -57,6 +57,7 @@ struct xdp_dev_bulk_queue { >> struct list_head flush_node; >> struct net_device *dev; >> struct net_device *dev_rx; >> + struct bpf_prog *xdp_prog; >> unsigned int count; >> }; >> >> @@ -327,46 +328,95 @@ bool dev_map_can_have_prog(struct bpf_map *map) >> return false; >> } >> >> +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, >> + struct xdp_frame **frames, int n, >> + struct net_device *dev) >> +{ >> + struct xdp_txq_info txq = { .dev = dev }; >> + struct xdp_buff xdp; >> + int i, nframes = 0; >> + >> + for (i = 0; i < n; i++) { >> + struct xdp_frame *xdpf = frames[i]; >> + u32 act; >> + int err; >> + >> + xdp_convert_frame_to_buff(xdpf, &xdp); >> + xdp.txq = &txq; >> + >> + act = bpf_prog_run_xdp(xdp_prog, &xdp); >> + switch (act) { >> + case XDP_PASS: >> + err = xdp_update_frame_from_buff(&xdp, xdpf); >> + if (unlikely(err < 0)) >> + xdp_return_frame_rx_napi(xdpf); >> + else >> + frames[nframes++] = xdpf; >> + break; >> + default: >> + bpf_warn_invalid_xdp_action(act); >> + fallthrough; >> + case XDP_ABORTED: >> + trace_xdp_exception(dev, xdp_prog, act); >> + fallthrough; >> + case XDP_DROP: >> + xdp_return_frame_rx_napi(xdpf); >> + break; >> + } >> + } >> + return nframes; /* sent frames count */ >> +} >> + >> static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) >> { >> struct net_device *dev = bq->dev; >> - int sent = 0, drops = 0, err = 0; >> + unsigned int cnt = bq->count; >> + int drops = 0, err = 0; >> + int to_sent = cnt; > > Hmm if I would be super picky then I'd like to have this variable as > "to_send", as we spoke. Could you change that? > > With that, you can add my: > > Reviewed-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com> > > to next revision. > >> + int sent = cnt; >> int i; >> >> - if (unlikely(!bq->count)) >> + if (unlikely(!cnt)) >> return; >> >> - for (i = 0; i < bq->count; i++) { >> + for (i = 0; i < cnt; i++) { >> struct xdp_frame *xdpf = bq->q[i]; >> >> prefetch(xdpf); >> } >> >> - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); >> + if (bq->xdp_prog) { >> + to_sent = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); >> + if (!to_sent) { >> + sent = 0; >> + goto out; >> + } >> + drops = cnt - to_sent; >> + } >> + >> + sent = dev->netdev_ops->ndo_xdp_xmit(dev, to_sent, bq->q, flags); >> if (sent < 0) { >> err = sent; >> sent = 0; >> - goto error; >> + >> + /* If ndo_xdp_xmit fails with an errno, no frames have been >> + * xmit'ed and it's our responsibility to them free all. >> + */ >> + for (i = 0; i < cnt - drops; i++) { >> + struct xdp_frame *xdpf = bq->q[i]; >> + >> + xdp_return_frame_rx_napi(xdpf); >> + } >> } >> - drops = bq->count - sent; >> out: >> + drops = cnt - sent; >> bq->count = 0; >> >> trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); >> bq->dev_rx = NULL; >> + bq->xdp_prog = NULL; > > One more question, do you really have to do that per each bq_xmit_all > call? Couldn't you clear it in __dev_flush ? > > Or IOW - what's the rationale behind storing xdp_prog in > xdp_dev_bulk_queue. Why can't you propagate the dst->xdp_prog and rely on > that without that local pointer? > > You probably have an answer for that, so maybe include it in commit > message. > > BTW same question for clearing dev_rx. To me this will be the same for all > bq_xmit_all() calls that will happen within same napi. I think you're right: When bq_xmit_all() is called from bq_enqueue(), another packet will always be enqueued immediately after, so clearing out all of those things in bq_xmit_all() is redundant. This also includes the list_del on bq->flush_node, BTW. And while we're getting into e micro-optimisations: In bq_enqueue() we have two checks: if (!bq->dev_rx) bq->dev_rx = dev_rx; bq->q[bq->count++] = xdpf; if (!bq->flush_node.prev) list_add(&bq->flush_node, flush_list); those two if() checks can be collapsed into one, since the list and the dev_rx field are only ever modified together. This will also be the case for bq->xdp_prog, so putting all three under the same check in bq_enqueue() and only clearing them in __dev_flush() would be a win, I suppose - nice catch! :) -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv16 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-22 13:38 ` Toke Høiland-Jørgensen @ 2021-01-25 3:30 ` Hangbin Liu 2021-01-25 11:21 ` Toke Høiland-Jørgensen 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2021-01-25 3:30 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: Maciej Fijalkowski, bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend On Fri, Jan 22, 2021 at 02:38:40PM +0100, Toke Høiland-Jørgensen wrote: > >> out: > >> + drops = cnt - sent; > >> bq->count = 0; > >> > >> trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); > >> bq->dev_rx = NULL; > >> + bq->xdp_prog = NULL; > > > > One more question, do you really have to do that per each bq_xmit_all > > call? Couldn't you clear it in __dev_flush ? > > > > Or IOW - what's the rationale behind storing xdp_prog in > > xdp_dev_bulk_queue. Why can't you propagate the dst->xdp_prog and rely on > > that without that local pointer? > > > > You probably have an answer for that, so maybe include it in commit > > message. > > > > BTW same question for clearing dev_rx. To me this will be the same for all > > bq_xmit_all() calls that will happen within same napi. > > I think you're right: When bq_xmit_all() is called from bq_enqueue(), > another packet will always be enqueued immediately after, so clearing > out all of those things in bq_xmit_all() is redundant. This also > includes the list_del on bq->flush_node, BTW. > > And while we're getting into e micro-optimisations: In bq_enqueue() we > have two checks: > > if (!bq->dev_rx) > bq->dev_rx = dev_rx; > > bq->q[bq->count++] = xdpf; > > if (!bq->flush_node.prev) > list_add(&bq->flush_node, flush_list); > > > those two if() checks can be collapsed into one, since the list and the > dev_rx field are only ever modified together. This will also be the case > for bq->xdp_prog, so putting all three under the same check in > bq_enqueue() and only clearing them in __dev_flush() would be a win, I > suppose - nice catch! :) Thanks for the advice, so how about modify it like: diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index bc38f7193149..217e09533097 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -413,9 +413,6 @@ static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) bq->count = 0; trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); - bq->dev_rx = NULL; - bq->xdp_prog = NULL; - __list_del_clearprev(&bq->flush_node); return; } @@ -434,8 +431,12 @@ void __dev_flush(void) struct list_head *flush_list = this_cpu_ptr(&dev_flush_list); struct xdp_dev_bulk_queue *bq, *tmp; - list_for_each_entry_safe(bq, tmp, flush_list, flush_node) + list_for_each_entry_safe(bq, tmp, flush_list, flush_node) { bq_xmit_all(bq, XDP_XMIT_FLUSH); + bq->dev_rx = NULL; + bq->xdp_prog = NULL; + __list_del_clearprev(&bq->flush_node); + } } /* rcu_read_lock (from syscall and BPF contexts) ensures that if a delete and/or @@ -469,22 +470,17 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, /* Ingress dev_rx will be the same for all xdp_frame's in * bulk_queue, because bq stored per-CPU and must be flushed * from net_device drivers NAPI func end. + * + * Do the same with xdp_prog and flush_list since these fields + * are modified together. */ - if (!bq->dev_rx) + if (!bq->dev_rx) { bq->dev_rx = dev_rx; - - /* Store (potential) xdp_prog that run before egress to dev as - * part of bulk_queue. This will be same xdp_prog for all - * xdp_frame's in bulk_queue, because this per-CPU store must - * be flushed from net_device drivers NAPI func end. - */ - if (!bq->xdp_prog) bq->xdp_prog = xdp_prog; + list_add(&bq->flush_node, flush_list); + } bq->q[bq->count++] = xdpf; - - if (!bq->flush_node.prev) - list_add(&bq->flush_node, flush_list); } static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv16 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-25 3:30 ` Hangbin Liu @ 2021-01-25 11:21 ` Toke Høiland-Jørgensen 2021-01-25 12:29 ` Maciej Fijalkowski 0 siblings, 1 reply; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2021-01-25 11:21 UTC (permalink / raw) To: Hangbin Liu Cc: Maciej Fijalkowski, bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend Hangbin Liu <liuhangbin@gmail.com> writes: > On Fri, Jan 22, 2021 at 02:38:40PM +0100, Toke Høiland-Jørgensen wrote: >> >> out: >> >> + drops = cnt - sent; >> >> bq->count = 0; >> >> >> >> trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); >> >> bq->dev_rx = NULL; >> >> + bq->xdp_prog = NULL; >> > >> > One more question, do you really have to do that per each bq_xmit_all >> > call? Couldn't you clear it in __dev_flush ? >> > >> > Or IOW - what's the rationale behind storing xdp_prog in >> > xdp_dev_bulk_queue. Why can't you propagate the dst->xdp_prog and rely on >> > that without that local pointer? >> > >> > You probably have an answer for that, so maybe include it in commit >> > message. >> > >> > BTW same question for clearing dev_rx. To me this will be the same for all >> > bq_xmit_all() calls that will happen within same napi. >> >> I think you're right: When bq_xmit_all() is called from bq_enqueue(), >> another packet will always be enqueued immediately after, so clearing >> out all of those things in bq_xmit_all() is redundant. This also >> includes the list_del on bq->flush_node, BTW. >> >> And while we're getting into e micro-optimisations: In bq_enqueue() we >> have two checks: >> >> if (!bq->dev_rx) >> bq->dev_rx = dev_rx; >> >> bq->q[bq->count++] = xdpf; >> >> if (!bq->flush_node.prev) >> list_add(&bq->flush_node, flush_list); >> >> >> those two if() checks can be collapsed into one, since the list and the >> dev_rx field are only ever modified together. This will also be the case >> for bq->xdp_prog, so putting all three under the same check in >> bq_enqueue() and only clearing them in __dev_flush() would be a win, I >> suppose - nice catch! :) > > Thanks for the advice, so how about modify it like: Yup, exactly! :) -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv16 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-25 11:21 ` Toke Høiland-Jørgensen @ 2021-01-25 12:29 ` Maciej Fijalkowski 0 siblings, 0 replies; 219+ messages in thread From: Maciej Fijalkowski @ 2021-01-25 12:29 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: Hangbin Liu, bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend On Mon, Jan 25, 2021 at 12:21:26PM +0100, Toke Høiland-Jørgensen wrote: > Hangbin Liu <liuhangbin@gmail.com> writes: > > > On Fri, Jan 22, 2021 at 02:38:40PM +0100, Toke Høiland-Jørgensen wrote: > >> >> out: > >> >> + drops = cnt - sent; > >> >> bq->count = 0; > >> >> > >> >> trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); > >> >> bq->dev_rx = NULL; > >> >> + bq->xdp_prog = NULL; > >> > > >> > One more question, do you really have to do that per each bq_xmit_all > >> > call? Couldn't you clear it in __dev_flush ? > >> > > >> > Or IOW - what's the rationale behind storing xdp_prog in > >> > xdp_dev_bulk_queue. Why can't you propagate the dst->xdp_prog and rely on > >> > that without that local pointer? > >> > > >> > You probably have an answer for that, so maybe include it in commit > >> > message. > >> > > >> > BTW same question for clearing dev_rx. To me this will be the same for all > >> > bq_xmit_all() calls that will happen within same napi. > >> > >> I think you're right: When bq_xmit_all() is called from bq_enqueue(), > >> another packet will always be enqueued immediately after, so clearing > >> out all of those things in bq_xmit_all() is redundant. This also > >> includes the list_del on bq->flush_node, BTW. > >> > >> And while we're getting into e micro-optimisations: In bq_enqueue() we > >> have two checks: > >> > >> if (!bq->dev_rx) > >> bq->dev_rx = dev_rx; > >> > >> bq->q[bq->count++] = xdpf; > >> > >> if (!bq->flush_node.prev) > >> list_add(&bq->flush_node, flush_list); > >> > >> > >> those two if() checks can be collapsed into one, since the list and the > >> dev_rx field are only ever modified together. This will also be the case > >> for bq->xdp_prog, so putting all three under the same check in > >> bq_enqueue() and only clearing them in __dev_flush() would be a win, I > >> suppose - nice catch! :) Huh, nice further optimization! :) Of course I agree on that. > > > > Thanks for the advice, so how about modify it like: > > Yup, exactly! :) > > -Toke > ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv16 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL 2021-01-22 7:46 ` [PATCHv16 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2021-01-22 7:46 ` [PATCHv16 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu @ 2021-01-22 7:46 ` Hangbin Liu 2021-01-22 7:46 ` [PATCHv16 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (5 subsequent siblings) 7 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-22 7:46 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu Add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL which could be used when we want to allow NULL pointer for map parameter. The bpf helper need to take care and check if the map is NULL when use this type. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v13-v15: no update v11-v12: rebase the patch to latest bpf-next v10: remove useless CONST_PTR_TO_MAP_OR_NULL and Copy-paste comment. v9: merge the patch from [1] in to this series. v1-v8: no this patch [1] https://lore.kernel.org/bpf/20200715070001.2048207-1-liuhangbin@gmail.com/ --- include/linux/bpf.h | 1 + kernel/bpf/verifier.c | 10 ++++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 1aac2af12fed..b75207a2484c 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -296,6 +296,7 @@ enum bpf_arg_type { ARG_CONST_ALLOC_SIZE_OR_ZERO, /* number of allocated bytes requested */ ARG_PTR_TO_BTF_ID_SOCK_COMMON, /* pointer to in-kernel sock_common or bpf-mirrored bpf_sock */ ARG_PTR_TO_PERCPU_BTF_ID, /* pointer to in-kernel percpu type */ + ARG_CONST_MAP_PTR_OR_NULL, /* const argument used as pointer to bpf_map or NULL */ __BPF_ARG_TYPE_MAX, }; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 785d25392ead..62920bea2454 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -445,7 +445,8 @@ static bool arg_type_may_be_null(enum bpf_arg_type type) type == ARG_PTR_TO_MEM_OR_NULL || type == ARG_PTR_TO_CTX_OR_NULL || type == ARG_PTR_TO_SOCKET_OR_NULL || - type == ARG_PTR_TO_ALLOC_MEM_OR_NULL; + type == ARG_PTR_TO_ALLOC_MEM_OR_NULL || + type == ARG_CONST_MAP_PTR_OR_NULL; } /* Determine whether the function releases some resources allocated by another @@ -4108,6 +4109,7 @@ static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = { [ARG_CONST_SIZE_OR_ZERO] = &scalar_types, [ARG_CONST_ALLOC_SIZE_OR_ZERO] = &scalar_types, [ARG_CONST_MAP_PTR] = &const_map_ptr_types, + [ARG_CONST_MAP_PTR_OR_NULL] = &const_map_ptr_types, [ARG_PTR_TO_CTX] = &context_types, [ARG_PTR_TO_CTX_OR_NULL] = &context_types, [ARG_PTR_TO_SOCK_COMMON] = &sock_types, @@ -4253,9 +4255,9 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, meta->ref_obj_id = reg->ref_obj_id; } - if (arg_type == ARG_CONST_MAP_PTR) { - /* bpf_map_xxx(map_ptr) call: remember that map_ptr */ - meta->map_ptr = reg->map_ptr; + if (arg_type == ARG_CONST_MAP_PTR || + arg_type == ARG_CONST_MAP_PTR_OR_NULL) { + meta->map_ptr = register_is_null(reg) ? NULL : reg->map_ptr; } else if (arg_type == ARG_PTR_TO_MAP_KEY) { /* bpf_map_xxx(..., map_ptr, ..., key) call: * check that [key, key + map->key_size) are within -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv16 bpf-next 3/6] xdp: add a new helper for dev map multicast support 2021-01-22 7:46 ` [PATCHv16 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2021-01-22 7:46 ` [PATCHv16 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu 2021-01-22 7:46 ` [PATCHv16 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2021-01-22 7:46 ` Hangbin Liu 2021-01-25 12:27 ` Maciej Fijalkowski 2021-01-27 22:13 ` John Fastabend 2021-01-22 7:46 ` [PATCHv16 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu ` (4 subsequent siblings) 7 siblings, 2 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-22 7:46 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because a physical interface can be part of a logical grouping, such as a bond device. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. The forwarding map could be DEVMAP or DEVMAP_HASH, but the exclude map *must* be DEVMAP_HASH to get better performace. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. As both bpf_xdp_redirect_map() and this new helpers are using struct bpf_redirect_info, I add a new ex_map and set tgt_value to NULL in the new helper to make a difference with bpf_xdp_redirect_map(). Also I keep the general data path in net/core/filter.c, the native data path in kernel/bpf/devmap.c so we can use direct calls to get better performace. [0] https://xdp-project.net/#Handling-multicast Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v15: a) Update bpf_redirect_map_multi() helper description that ex_map must be keyed by ifindex. b) remove variable last_one in dev_map_enqueue_multi() as it's pointless. c) add a comment about why we don't use READ/WRITE_ONCE() for ex_map. v14: no update, only rebase the code v13: pass xdp_prog through bq_enqueue v12: rebase the code based on Jespoer's devmap xdp_prog patch v11: Fix bpf_redirect_map_multi() helper description typo. Add loop limit for devmap_get_next_obj() and dev_map_redirect_multi(). v10: Update helper bpf_xdp_redirect_map_multi() - No need to check map pointer as we will do the check in verifier. v9: Update helper bpf_xdp_redirect_map_multi() - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. --- include/linux/bpf.h | 20 ++++++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 28 ++++++++ kernel/bpf/devmap.c | 128 +++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++ net/core/filter.c | 124 ++++++++++++++++++++++++++++++-- net/core/xdp.c | 29 ++++++++ tools/include/uapi/linux/bpf.h | 28 ++++++++ 9 files changed, 360 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index b75207a2484c..4b68e975e990 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1427,6 +1427,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx); int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, struct net_device *dev_rx); +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex); +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); bool dev_map_can_have_prog(struct bpf_map *map); @@ -1595,6 +1600,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return 0; } +static inline +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + return false; +} + +static inline +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + return 0; +} + struct sk_buff; static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, diff --git a/include/linux/filter.h b/include/linux/filter.h index 5b3137d7b690..aa9f87ded63c 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -637,6 +637,7 @@ struct bpf_redirect_info { u32 tgt_index; void *tgt_value; struct bpf_map *map; + struct bpf_map *ex_map; u32 kern_flags; struct bpf_nh_params nh; }; diff --git a/include/net/xdp.h b/include/net/xdp.h index c4bfdc9a8b79..fcc0f6f8bd91 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -169,6 +169,7 @@ struct sk_buff *__xdp_build_skb_from_frame(struct xdp_frame *xdpf, struct net_device *dev); struct sk_buff *xdp_build_skb_from_frame(struct xdp_frame *xdpf, struct net_device *dev); +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); static inline void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index c001766adcbc..ef943c024322 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3836,6 +3836,28 @@ union bpf_attr { * Return * A pointer to a struct socket on success or NULL if the file is * not a socket. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. To get better performance, the + * *ex_map* is limited to BPF_MAP_TYPE_DEVMAP_HASH and must be + * keyed by ifindex for the helper to work. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -4001,6 +4023,7 @@ union bpf_attr { FN(ktime_get_coarse_ns), \ FN(ima_inode_hash), \ FN(sock_from_file), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -4177,6 +4200,11 @@ enum { BPF_F_BPRM_SECUREEXEC = (1ULL << 0), }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index c24fcffbbfad..212c91b8f063 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -523,6 +523,134 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return __xdp_enqueue(dev, xdp, dev_rx, dst->xdp_prog); } +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + return dev_map_get_next_key(map, key, next_key); + case BPF_MAP_TYPE_DEVMAP_HASH: + return dev_map_hash_get_next_key(map, key, next_key); + default: + break; + } + + return -ENOENT; +} + +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + if (obj->dev->ifindex == exclude_ifindex) + return true; + + if (!map) + return false; + + return __dev_map_hash_lookup_elem(map, obj->dev->ifindex) != NULL; +} + +static struct bpf_dtab_netdev *devmap_get_next_obj(struct xdp_buff *xdp, struct bpf_map *map, + struct bpf_map *ex_map, u32 *key, + u32 *next_key, int ex_ifindex) +{ + struct bpf_dtab_netdev *obj; + struct net_device *dev; + u32 *tmp_key = key; + u32 index; + int err; + + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + return NULL; + + /* When using dev map hash, we could restart the hashtab traversal + * in case the key has been updated/removed in the mean time. + * So we may end up potentially looping due to traversal restarts + * from first elem. + * + * Let's use map's max_entries to limit the loop number. + */ + for (index = 0; index < map->max_entries; index++) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + obj = __dev_map_lookup_elem(map, *next_key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + obj = __dev_map_hash_lookup_elem(map, *next_key); + break; + default: + break; + } + + if (!obj || dev_in_exclude_map(obj, ex_map, ex_ifindex)) + goto find_next; + + dev = obj->dev; + + if (!dev->netdev_ops->ndo_xdp_xmit) + goto find_next; + + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + if (unlikely(err)) + goto find_next; + + return obj; + +find_next: + tmp_key = next_key; + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + break; + } + + return NULL; +} + +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + struct bpf_dtab_netdev *obj = NULL, *next_obj = NULL; + struct xdp_frame *xdpf, *nxdpf; + int ex_ifindex; + u32 key, next_key; + + ex_ifindex = flags & BPF_F_EXCLUDE_INGRESS ? dev_rx->ifindex : 0; + + /* Find first available obj */ + obj = devmap_get_next_obj(xdp, map, ex_map, NULL, &key, ex_ifindex); + if (!obj) + return 0; + + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) + return -EOVERFLOW; + + for (;;) { + /* Check if we still have one more available obj */ + next_obj = devmap_get_next_obj(xdp, map, ex_map, &key, + &next_key, ex_ifindex); + if (!next_obj) { + bq_enqueue(obj->dev, xdpf, dev_rx, obj->xdp_prog); + return 0; + } + + nxdpf = xdpf_clone(xdpf); + if (unlikely(!nxdpf)) { + xdp_return_frame_rx_napi(xdpf); + return -ENOMEM; + } + + bq_enqueue(obj->dev, nxdpf, dev_rx, obj->xdp_prog); + + /* Deal with next obj */ + obj = next_obj; + key = next_key; + } +} + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 62920bea2454..342674767292 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4463,6 +4463,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: if (func_id != BPF_FUNC_redirect_map && + func_id != BPF_FUNC_redirect_map_multi && func_id != BPF_FUNC_map_lookup_elem) goto error; break; @@ -4567,6 +4568,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, map->map_type != BPF_MAP_TYPE_XSKMAP) goto error; break; + case BPF_FUNC_redirect_map_multi: + if (map->map_type != BPF_MAP_TYPE_DEVMAP && + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + goto error; + break; case BPF_FUNC_sk_redirect_map: case BPF_FUNC_msg_redirect_map: case BPF_FUNC_sock_map_update: diff --git a/net/core/filter.c b/net/core/filter.c index 9ab94e90d660..03b3088ce8db 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3924,12 +3924,19 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { }; static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) + struct bpf_map *map, struct xdp_buff *xdp, + struct bpf_map *ex_map, u32 flags) { switch (map->map_type) { case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: - return dev_map_enqueue(fwd, xdp, dev_rx); + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) + return dev_map_enqueue(fwd, xdp, dev_rx); + else + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, flags); case BPF_MAP_TYPE_CPUMAP: return cpu_map_enqueue(fwd, xdp, dev_rx); case BPF_MAP_TYPE_XSKMAP: @@ -3986,12 +3993,19 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); struct bpf_map *map = READ_ONCE(ri->map); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err; + /* The READ/WRITE_ONCE() is not needed for ex_map because the field + * is only read from or written to by the CPU owning the per-cpu + * pointer. Whereas the 'map' field is manipulated by remote CPUs + * in bpf_clear_redirect_map(). + */ ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (unlikely(!map)) { @@ -4003,7 +4017,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = dev_xdp_enqueue(fwd, xdp, dev); } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, ri->flags); } if (unlikely(err)) @@ -4017,6 +4031,63 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, } EXPORT_SYMBOL_GPL(xdp_do_redirect); +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, + struct bpf_prog *xdp_prog, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) + +{ + struct bpf_dtab_netdev *dst; + struct sk_buff *nskb; + bool exclude_ingress; + u32 key, next_key, index; + void *fwd; + int err; + + /* Get first key from forward map */ + err = map->ops->map_get_next_key(map, NULL, &key); + if (err) + return err; + + exclude_ingress = !!(flags & BPF_F_EXCLUDE_INGRESS); + + /* When using dev map hash, we could restart the hashtab traversal + * in case the key has been updated/removed in the mean time. + * So we may end up potentially looping due to traversal restarts + * from first elem. + * + * Let's use map's max_entries to limit the loop number. + */ + for (index = 0; index < map->max_entries; index++) { + fwd = __xdp_map_lookup_elem(map, key); + if (fwd) { + dst = (struct bpf_dtab_netdev *)fwd; + if (dev_in_exclude_map(dst, ex_map, + exclude_ingress ? dev->ifindex : 0)) + goto find_next; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -ENOMEM; + + /* Try forword next one no mater the current forward + * succeed or not. + */ + dev_map_generic_redirect(dst, nskb, xdp_prog); + } + +find_next: + err = map->ops->map_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + consume_skb(skb); + return 0; +} + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, @@ -4024,19 +4095,30 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, struct bpf_map *map) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err = 0; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (map->map_type == BPF_MAP_TYPE_DEVMAP || map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { - struct bpf_dtab_netdev *dst = fwd; + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) { + struct bpf_dtab_netdev *dst = fwd; + + err = dev_map_generic_redirect(dst, skb, xdp_prog); + } else { + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, + ex_map, ri->flags); + } - err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { @@ -4150,6 +4232,36 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { .arg3_type = ARG_ANYTHING, }; +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, + struct bpf_map *, ex_map, u64, flags) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + + /* Limit ex_map type to DEVMAP_HASH to get better performance */ + if (unlikely((ex_map && ex_map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) || + flags & ~BPF_F_EXCLUDE_INGRESS)) + return XDP_ABORTED; + + ri->tgt_index = 0; + /* Set the tgt_value to NULL to distinguish with bpf_xdp_redirect_map */ + ri->tgt_value = NULL; + ri->flags = flags; + ri->ex_map = ex_map; + + WRITE_ONCE(ri->map, map); + + return XDP_REDIRECT; +} + +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { + .func = bpf_xdp_redirect_map_multi, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_CONST_MAP_PTR, + .arg2_type = ARG_CONST_MAP_PTR_OR_NULL, + .arg3_type = ARG_ANYTHING, +}; + static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, unsigned long off, unsigned long len) { @@ -7248,6 +7360,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_redirect_proto; case BPF_FUNC_redirect_map: return &bpf_xdp_redirect_map_proto; + case BPF_FUNC_redirect_map_multi: + return &bpf_xdp_redirect_map_multi_proto; case BPF_FUNC_xdp_adjust_tail: return &bpf_xdp_adjust_tail_proto; case BPF_FUNC_fib_lookup: diff --git a/net/core/xdp.c b/net/core/xdp.c index 0d2630a35c3e..b44d05d24a27 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -572,3 +572,32 @@ struct sk_buff *xdp_build_skb_from_frame(struct xdp_frame *xdpf, return __xdp_build_skb_from_frame(xdpf, skb, dev); } EXPORT_SYMBOL_GPL(xdp_build_skb_from_frame); + +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) +{ + unsigned int headroom, totalsize; + struct xdp_frame *nxdpf; + struct page *page; + void *addr; + + headroom = xdpf->headroom + sizeof(*xdpf); + totalsize = headroom + xdpf->len; + + if (unlikely(totalsize > PAGE_SIZE)) + return NULL; + page = dev_alloc_page(); + if (!page) + return NULL; + addr = page_to_virt(page); + + memcpy(addr, xdpf, totalsize); + + nxdpf = addr; + nxdpf->data = addr + headroom; + nxdpf->frame_sz = PAGE_SIZE; + nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0; + nxdpf->mem.id = 0; + + return nxdpf; +} +EXPORT_SYMBOL_GPL(xdpf_clone); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index c001766adcbc..ef943c024322 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3836,6 +3836,28 @@ union bpf_attr { * Return * A pointer to a struct socket on success or NULL if the file is * not a socket. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. To get better performance, the + * *ex_map* is limited to BPF_MAP_TYPE_DEVMAP_HASH and must be + * keyed by ifindex for the helper to work. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -4001,6 +4023,7 @@ union bpf_attr { FN(ktime_get_coarse_ns), \ FN(ima_inode_hash), \ FN(sock_from_file), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -4177,6 +4200,11 @@ enum { BPF_F_BPRM_SECUREEXEC = (1ULL << 0), }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv16 bpf-next 3/6] xdp: add a new helper for dev map multicast support 2021-01-22 7:46 ` [PATCHv16 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2021-01-25 12:27 ` Maciej Fijalkowski 2021-01-26 1:17 ` Hangbin Liu 2021-01-27 22:13 ` John Fastabend 1 sibling, 1 reply; 219+ messages in thread From: Maciej Fijalkowski @ 2021-01-25 12:27 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend On Fri, Jan 22, 2021 at 03:46:49PM +0800, Hangbin Liu wrote: > This patch is for xdp multicast support. which has been discussed > before[0], The goal is to be able to implement an OVS-like data plane in > XDP, i.e., a software switch that can forward XDP frames to multiple ports. > > To achieve this, an application needs to specify a group of interfaces > to forward a packet to. It is also common to want to exclude one or more > physical interfaces from the forwarding operation - e.g., to forward a > packet to all interfaces in the multicast group except the interface it > arrived on. While this could be done simply by adding more groups, this > quickly leads to a combinatorial explosion in the number of groups an > application has to maintain. > > To avoid the combinatorial explosion, we propose to include the ability > to specify an "exclude group" as part of the forwarding operation. This > needs to be a group (instead of just a single port index), because a > physical interface can be part of a logical grouping, such as a bond > device. > > Thus, the logical forwarding operation becomes a "set difference" > operation, i.e. "forward to all ports in group A that are not also in > group B". This series implements such an operation using device maps to > represent the groups. This means that the XDP program specifies two > device maps, one containing the list of netdevs to redirect to, and the > other containing the exclude list. > > To achieve this, I re-implement a new helper bpf_redirect_map_multi() > to accept two maps, the forwarding map and exclude map. The forwarding > map could be DEVMAP or DEVMAP_HASH, but the exclude map *must* be > DEVMAP_HASH to get better performace. If user don't want to use exclude > map and just want simply stop redirecting back to ingress device, they > can use flag BPF_F_EXCLUDE_INGRESS. Hangbin, before you submit next revision, could you try to apply imperative mood to your commit messages? From Documentation/process/submitting-patches.rst: <quote> Describe your changes in imperative mood, e.g. "make xyzzy do frotz" instead of "[This patch] makes xyzzy do frotz" or "[I] changed xyzzy to do frotz", as if you are giving orders to the codebase to change its behaviour. </quote> That's the thing I'm trying to remind people internally and it feels like we keep on forgetting about that. Thanks! > > As both bpf_xdp_redirect_map() and this new helpers are using struct > bpf_redirect_info, I add a new ex_map and set tgt_value to NULL in the > new helper to make a difference with bpf_xdp_redirect_map(). > > Also I keep the general data path in net/core/filter.c, the native data > path in kernel/bpf/devmap.c so we can use direct calls to get better > performace. > > [0] https://xdp-project.net/#Handling-multicast > > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv16 bpf-next 3/6] xdp: add a new helper for dev map multicast support 2021-01-25 12:27 ` Maciej Fijalkowski @ 2021-01-26 1:17 ` Hangbin Liu 0 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-26 1:17 UTC (permalink / raw) To: Maciej Fijalkowski Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend On Mon, Jan 25, 2021 at 01:27:24PM +0100, Maciej Fijalkowski wrote: > Hangbin, > > before you submit next revision, could you try to apply imperative mood to > your commit messages? > > From Documentation/process/submitting-patches.rst: > > <quote> > Describe your changes in imperative mood, e.g. "make xyzzy do frotz" > instead of "[This patch] makes xyzzy do frotz" or "[I] changed xyzzy > to do frotz", as if you are giving orders to the codebase to change > its behaviour. > </quote> > > That's the thing I'm trying to remind people internally and it feels like > we keep on forgetting about that. Hi Maciej, Thanks for the reminder. I just see your reply after post the new version. As I'm not a native speaker, I always not very sure what kind of mood/words I should use in the commit message. So I just try to as polite as possible (although I may pick some rude works that I didn't realize) and not order people or something else based on our culture/background. I guess that's also the reason some people forget to use "imperative mood". I will keep this in mind for future works. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* RE: [PATCHv16 bpf-next 3/6] xdp: add a new helper for dev map multicast support 2021-01-22 7:46 ` [PATCHv16 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2021-01-25 12:27 ` Maciej Fijalkowski @ 2021-01-27 22:13 ` John Fastabend 1 sibling, 0 replies; 219+ messages in thread From: John Fastabend @ 2021-01-27 22:13 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu Hangbin Liu wrote: > This patch is for xdp multicast support. which has been discussed > before[0], The goal is to be able to implement an OVS-like data plane in > XDP, i.e., a software switch that can forward XDP frames to multiple ports. > > To achieve this, an application needs to specify a group of interfaces > to forward a packet to. It is also common to want to exclude one or more > physical interfaces from the forwarding operation - e.g., to forward a > packet to all interfaces in the multicast group except the interface it > arrived on. While this could be done simply by adding more groups, this > quickly leads to a combinatorial explosion in the number of groups an > application has to maintain. > > To avoid the combinatorial explosion, we propose to include the ability > to specify an "exclude group" as part of the forwarding operation. This > needs to be a group (instead of just a single port index), because a > physical interface can be part of a logical grouping, such as a bond > device. > > Thus, the logical forwarding operation becomes a "set difference" > operation, i.e. "forward to all ports in group A that are not also in > group B". This series implements such an operation using device maps to > represent the groups. This means that the XDP program specifies two > device maps, one containing the list of netdevs to redirect to, and the > other containing the exclude list. > > To achieve this, I re-implement a new helper bpf_redirect_map_multi() > to accept two maps, the forwarding map and exclude map. The forwarding > map could be DEVMAP or DEVMAP_HASH, but the exclude map *must* be > DEVMAP_HASH to get better performace. If user don't want to use exclude > map and just want simply stop redirecting back to ingress device, they > can use flag BPF_F_EXCLUDE_INGRESS. > > As both bpf_xdp_redirect_map() and this new helpers are using struct > bpf_redirect_info, I add a new ex_map and set tgt_value to NULL in the > new helper to make a difference with bpf_xdp_redirect_map(). > > Also I keep the general data path in net/core/filter.c, the native data > path in kernel/bpf/devmap.c so we can use direct calls to get better > performace. > > [0] https://xdp-project.net/#Handling-multicast > > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > Thanks for the updates. Acked-by: John Fastabend <john.fastabend@gmail.com> ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv16 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test 2021-01-22 7:46 ` [PATCHv16 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (2 preceding siblings ...) 2021-01-22 7:46 ` [PATCHv16 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2021-01-22 7:46 ` Hangbin Liu 2021-01-22 7:46 ` [PATCHv16 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu ` (3 subsequent siblings) 7 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-22 7:46 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu This is a sample for xdp multicast. In the sample we could forward all packets between given interfaces. There is also an option -X that could enable 2nd xdp_prog on egress interface. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v15: use bpf_object__find_program_by_name() instead of bpf_object__find_program_by_title() v13-v14: no update, only rebase the code v12: add devmap xdp_prog on egress support v10-v11: no update v9: use NULL directly for arg2 and redefine the maps with btf format v5: add a null_map as we have strict the arg2 to ARG_CONST_MAP_PTR. Move the testing part to bpf selftest in next patch. v4: no update. v3: add rxcnt map to show the packet transmit speed. v2: no update. --- samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 87 +++++++ samples/bpf/xdp_redirect_map_multi_user.c | 302 ++++++++++++++++++++++ 3 files changed, 392 insertions(+) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 26fc96ca619e..200029fcf53c 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map tprogs-y += per_socket_stats_example tprogs-y += xdp_redirect tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_map_multi tprogs-y += xdp_redirect_cpu tprogs-y += xdp_monitor tprogs-y += xdp_rxq_info @@ -99,6 +100,7 @@ test_map_in_map-objs := test_map_in_map_user.o per_socket_stats_example-objs := cookie_uid_helper_example.o xdp_redirect-objs := xdp_redirect_user.o xdp_redirect_map-objs := xdp_redirect_map_user.o +xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o xdp_redirect_cpu-objs := xdp_redirect_cpu_user.o xdp_monitor-objs := xdp_monitor_user.o xdp_rxq_info-objs := xdp_rxq_info_user.o @@ -160,6 +162,7 @@ always-y += tcp_tos_reflect_kern.o always-y += tcp_dumpstats_kern.o always-y += xdp_redirect_kern.o always-y += xdp_redirect_map_kern.o +always-y += xdp_redirect_map_multi_kern.o always-y += xdp_redirect_cpu_kern.o always-y += xdp_monitor_kern.o always-y += xdp_rxq_info_kern.o diff --git a/samples/bpf/xdp_redirect_map_multi_kern.c b/samples/bpf/xdp_redirect_map_multi_kern.c new file mode 100644 index 000000000000..e422340d1251 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_kern.c @@ -0,0 +1,87 @@ +// SPDX-License-Identifier: GPL-2.0 +#define KBUILD_MODNAME "foo" +#include <uapi/linux/bpf.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/ip.h> +#include <linux/ipv6.h> +#include <bpf/bpf_helpers.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 32); +} forward_map_general SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(struct bpf_devmap_val)); + __uint(max_entries, 32); +} forward_map_native SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __type(key, u32); + __type(value, long); + __uint(max_entries, 1); +} rxcnt SEC(".maps"); + +/* map to store egress interfaces mac addresses, set the + * max_entries to 1 and extend it in user sapce prog. + */ +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, u32); + __type(value, __be64); + __uint(max_entries, 1); +} mac_map SEC(".maps"); + +static int xdp_redirect_map(struct xdp_md *ctx, void *forward_map) +{ + long *value; + u32 key = 0; + + /* count packet in global counter */ + value = bpf_map_lookup_elem(&rxcnt, &key); + if (value) + *value += 1; + + return bpf_redirect_map_multi(forward_map, NULL, BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_redirect_general") +int xdp_redirect_map_general(struct xdp_md *ctx) +{ + return xdp_redirect_map(ctx, &forward_map_general); +} + +SEC("xdp_redirect_native") +int xdp_redirect_map_native(struct xdp_md *ctx) +{ + return xdp_redirect_map(ctx, &forward_map_native); +} + +SEC("xdp_devmap/map_prog") +int xdp_devmap_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + u32 key = ctx->egress_ifindex; + struct ethhdr *eth = data; + __be64 *mac; + u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + mac = bpf_map_lookup_elem(&mac_map, &key); + if (mac) + __builtin_memcpy(eth->h_source, mac, ETH_ALEN); + + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/samples/bpf/xdp_redirect_map_multi_user.c b/samples/bpf/xdp_redirect_map_multi_user.c new file mode 100644 index 000000000000..84cdbbed20b7 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_user.c @@ -0,0 +1,302 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> +#include <sys/resource.h> +#include <sys/ioctl.h> +#include <sys/types.h> +#include <sys/socket.h> +#include <netinet/in.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; +static int rxcnt_map_fd; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void poll_stats(int interval) +{ + unsigned int nr_cpus = bpf_num_possible_cpus(); + __u64 values[nr_cpus], prev[nr_cpus]; + + memset(prev, 0, sizeof(prev)); + + while (1) { + __u64 sum = 0; + __u32 key = 0; + int i; + + sleep(interval); + assert(bpf_map_lookup_elem(rxcnt_map_fd, &key, values) == 0); + for (i = 0; i < nr_cpus; i++) + sum += (values[i] - prev[i]); + if (sum) + printf("Forwarding %10llu pkt/s\n", sum / interval); + memcpy(prev, values, sizeof(values)); + } +} + +static int get_mac_addr(unsigned int ifindex, void *mac_addr) +{ + char ifname[IF_NAMESIZE]; + struct ifreq ifr; + int fd, ret = -1; + + fd = socket(AF_INET, SOCK_DGRAM, 0); + if (fd < 0) + return ret; + + if (!if_indextoname(ifindex, ifname)) + goto err_out; + + strcpy(ifr.ifr_name, ifname); + + if (ioctl(fd, SIOCGIFHWADDR, &ifr) != 0) + goto err_out; + + memcpy(mac_addr, ifr.ifr_hwaddr.sa_data, 6 * sizeof(char)); + ret = 0; + +err_out: + close(fd); + return ret; +} + +static int update_mac_map(struct bpf_object *obj) +{ + int i, ret = -1, mac_map_fd; + unsigned char mac_addr[6]; + unsigned int ifindex; + + mac_map_fd = bpf_object__find_map_fd_by_name(obj, "mac_map"); + if (mac_map_fd < 0) { + printf("find mac map fd failed\n"); + return ret; + } + + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + ret = get_mac_addr(ifindex, mac_addr); + if (ret < 0) { + printf("get interface %d mac failed\n", ifindex); + return ret; + } + + ret = bpf_map_update_elem(mac_map_fd, &ifindex, mac_addr, 0); + if (ret) { + perror("bpf_update_elem mac_map_fd"); + return ret; + } + } + + return 0; +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n" + " -X load xdp program on egress\n", + prog); +} + +int main(int argc, char **argv) +{ + int i, ret, opt, forward_map_fd, max_ifindex = 0; + struct bpf_program *ingress_prog, *egress_prog; + int ingress_prog_fd, egress_prog_fd = 0; + struct bpf_devmap_val devmap_val; + bool attach_egress_prog = false; + char ifname[IF_NAMESIZE]; + struct bpf_map *mac_map; + struct bpf_object *obj; + unsigned int ifindex; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNFX")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + case 'X': + attach_egress_prog = true; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) { + xdp_flags |= XDP_FLAGS_DRV_MODE; + } else if (attach_egress_prog) { + printf("Load xdp program on egress with SKB mode not supported yet\n"); + return 1; + } + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + + /* Find the largest index number */ + if (ifaces[i] > max_ifindex) + max_ifindex = ifaces[i]; + + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + + obj = bpf_object__open(filename); + if (libbpf_get_error(obj)) { + printf("ERROR: opening BPF object file failed\n"); + obj = NULL; + goto err_out; + } + + /* Reset the map size to max ifindex + 1 */ + if (attach_egress_prog) { + mac_map = bpf_object__find_map_by_name(obj, "mac_map"); + ret = bpf_map__resize(mac_map, max_ifindex + 1); + if (ret < 0) { + printf("ERROR: reset mac map size failed\n"); + goto err_out; + } + } + + /* load BPF program */ + if (bpf_object__load(obj)) { + printf("ERROR: loading BPF object file failed\n"); + goto err_out; + } + + if (xdp_flags & XDP_FLAGS_SKB_MODE) { + ingress_prog = bpf_object__find_program_by_name(obj, "xdp_redirect_map_general"); + forward_map_fd = bpf_object__find_map_fd_by_name(obj, "forward_map_general"); + } else { + ingress_prog = bpf_object__find_program_by_name(obj, "xdp_redirect_map_native"); + forward_map_fd = bpf_object__find_map_fd_by_name(obj, "forward_map_native"); + } + if (!ingress_prog || forward_map_fd < 0) { + printf("finding ingress_prog/forward_map in obj file failed\n"); + goto err_out; + } + + ingress_prog_fd = bpf_program__fd(ingress_prog); + if (ingress_prog_fd < 0) { + printf("find ingress_prog fd failed\n"); + goto err_out; + } + + rxcnt_map_fd = bpf_object__find_map_fd_by_name(obj, "rxcnt"); + if (rxcnt_map_fd < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + goto err_out; + } + + if (attach_egress_prog) { + /* Update mac_map with all egress interfaces' mac addr */ + if (update_mac_map(obj) < 0) { + printf("Error: update mac map failed"); + goto err_out; + } + + /* Find egress prog fd */ + egress_prog = bpf_object__find_program_by_name(obj, "xdp_devmap_prog"); + if (!egress_prog) { + printf("finding egress_prog in obj file failed\n"); + goto err_out; + } + egress_prog_fd = bpf_program__fd(egress_prog); + if (egress_prog_fd < 0) { + printf("find egress_prog fd failed\n"); + goto err_out; + } + } + + /* Remove attached program when program is interrupted or killed */ + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, ingress_prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + /* Add all the interfaces to forward group and attach + * egress devmap programe if exist + */ + devmap_val.ifindex = ifindex; + devmap_val.bpf_prog.fd = egress_prog_fd; + ret = bpf_map_update_elem(forward_map_fd, &ifindex, &devmap_val, 0); + if (ret) { + perror("bpf_map_update_elem forward_map"); + goto err_out; + } + } + + poll_stats(2); + + return 0; + +err_out: + return 1; +} -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv16 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL 2021-01-22 7:46 ` [PATCHv16 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (3 preceding siblings ...) 2021-01-22 7:46 ` [PATCHv16 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu @ 2021-01-22 7:46 ` Hangbin Liu 2021-01-22 7:46 ` [PATCHv16 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu ` (2 subsequent siblings) 7 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-22 7:46 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu Use helper bpf_redirect_map() and bpf_redirect_map_multi() to test bpf arg ARG_CONST_MAP_PTR and ARG_CONST_MAP_PTR_OR_NULL. Make sure the map arg could be verified correctly when it is NULL or valid map pointer. Add devmap and devmap_hash in struct bpf_test due to bpf_redirect_{map, map_multi} limit. Test result: ]# ./test_verifier 713 716 #713/p ARG_CONST_MAP_PTR: null pointer OK #714/p ARG_CONST_MAP_PTR: valid map pointer OK #715/p ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map OK #716/p ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map OK Summary: 4 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- tools/testing/selftests/bpf/test_verifier.c | 21 +++++- .../testing/selftests/bpf/verifier/map_ptr.c | 70 +++++++++++++++++++ 2 files changed, 90 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c index 59bfa6201d1d..15f83ca8288a 100644 --- a/tools/testing/selftests/bpf/test_verifier.c +++ b/tools/testing/selftests/bpf/test_verifier.c @@ -50,7 +50,7 @@ #define MAX_INSNS BPF_MAXINSNS #define MAX_TEST_INSNS 1000000 #define MAX_FIXUPS 8 -#define MAX_NR_MAPS 21 +#define MAX_NR_MAPS 23 #define MAX_TEST_RUNS 8 #define POINTER_VALUE 0xcafe4all #define TEST_DATA_LEN 64 @@ -88,6 +88,8 @@ struct bpf_test { int fixup_map_event_output[MAX_FIXUPS]; int fixup_map_reuseport_array[MAX_FIXUPS]; int fixup_map_ringbuf[MAX_FIXUPS]; + int fixup_map_devmap[MAX_FIXUPS]; + int fixup_map_devmap_hash[MAX_FIXUPS]; const char *errstr; const char *errstr_unpriv; uint32_t insn_processed; @@ -714,6 +716,8 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, int *fixup_map_event_output = test->fixup_map_event_output; int *fixup_map_reuseport_array = test->fixup_map_reuseport_array; int *fixup_map_ringbuf = test->fixup_map_ringbuf; + int *fixup_map_devmap = test->fixup_map_devmap; + int *fixup_map_devmap_hash = test->fixup_map_devmap_hash; if (test->fill_helper) { test->fill_insns = calloc(MAX_TEST_INSNS, sizeof(struct bpf_insn)); @@ -898,6 +902,21 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, prog[*fixup_map_ringbuf].imm = map_fds[20]; fixup_map_ringbuf++; } while (*fixup_map_ringbuf); + if (*fixup_map_devmap) { + map_fds[20] = __create_map(BPF_MAP_TYPE_DEVMAP, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap].imm = map_fds[20]; + fixup_map_devmap++; + } while (*fixup_map_devmap); + } + if (*fixup_map_devmap_hash) { + map_fds[21] = __create_map(BPF_MAP_TYPE_DEVMAP_HASH, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap_hash].imm = map_fds[21]; + fixup_map_devmap_hash++; + } while (*fixup_map_devmap_hash); } } diff --git a/tools/testing/selftests/bpf/verifier/map_ptr.c b/tools/testing/selftests/bpf/verifier/map_ptr.c index b117bdd3806d..1a532198c9c1 100644 --- a/tools/testing/selftests/bpf/verifier/map_ptr.c +++ b/tools/testing/selftests/bpf/verifier/map_ptr.c @@ -93,3 +93,73 @@ .fixup_map_hash_16b = { 4 }, .result = ACCEPT, }, +{ + "ARG_CONST_MAP_PTR: null pointer", + .insns = { + /* bpf_redirect_map arg1 (map) */ + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .result = REJECT, + .prog_type = BPF_PROG_TYPE_XDP, + .errstr = "R1 type=inv expected=map_ptr", +}, +{ + "ARG_CONST_MAP_PTR: valid map pointer", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg1 (map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_LD_MAP_FD(BPF_REG_2, 1), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .fixup_map_devmap_hash = { 3 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv16 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test 2021-01-22 7:46 ` [PATCHv16 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (4 preceding siblings ...) 2021-01-22 7:46 ` [PATCHv16 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2021-01-22 7:46 ` Hangbin Liu 2021-01-22 13:43 ` [PATCHv16 bpf-next 0/6] xdp: add a new helper for dev map multicast support Toke Høiland-Jørgensen 2021-01-25 12:45 ` [PATCHv17 " Hangbin Liu 7 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-22 7:46 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu Add a bpf selftest for new helper xdp_redirect_map_multi(). In this test we have 3 forward groups and 1 exclude group. The test will redirect each interface's packets to all the interfaces in the forward group, and exclude the interface in exclude map. We will also test both DEVMAP and DEVMAP_HASH with xdp generic and drv. For more test details, you can find it in the test script. Here is the test result. ]# ./test_xdp_redirect_multi.sh Pass: xdpgeneric arp ns1-2 Pass: xdpgeneric arp ns1-3 Pass: xdpgeneric arp ns1-4 Pass: xdpgeneric ping ns1-2 Pass: xdpgeneric ping ns1-3 Pass: xdpgeneric ping ns1-4 Pass: xdpgeneric ping6 ns2-1 Pass: xdpgeneric ping6 ns2-3 Pass: xdpgeneric ping6 ns2-4 Pass: xdpdrv arp ns1-2 Pass: xdpdrv arp ns1-3 Pass: xdpdrv arp ns1-4 Pass: xdpdrv ping ns1-2 Pass: xdpdrv ping ns1-3 Pass: xdpdrv ping ns1-4 Pass: xdpdrv ping6 ns2-1 Pass: xdpdrv ping6 ns2-3 Pass: xdpdrv ping6 ns2-4 Pass: xdpegress mac ns1-2 Pass: xdpegress mac ns1-3 Pass: xdpegress mac ns1-4 Pass: xdpegress ping ns1-2 Pass: xdpegress ping ns1-3 Pass: xdpegress ping ns1-4 Summary: PASS 24, FAIL 0 Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v15: use bpf_object__find_program_by_name instead of bpf_object__find_program_by_title v14: no update, only rebase the code v13: remove setrlimit v12: add devmap prog test on egress v9: use NULL directly for arg2 and redefine the maps with btf format --- tools/testing/selftests/bpf/Makefile | 3 +- .../bpf/progs/xdp_redirect_multi_kern.c | 111 ++++++++ .../selftests/bpf/test_xdp_redirect_multi.sh | 208 +++++++++++++++ .../selftests/bpf/xdp_redirect_multi.c | 252 ++++++++++++++++++ 4 files changed, 573 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 63d6288e419c..621dceddb249 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -51,6 +51,7 @@ TEST_FILES = xsk_prereqs.sh \ # Order correspond to 'make run_tests' order TEST_PROGS := test_kmod.sh \ test_xdp_redirect.sh \ + test_xdp_redirect_multi.sh \ test_xdp_meta.sh \ test_xdp_veth.sh \ test_offload.py \ @@ -80,7 +81,7 @@ TEST_PROGS_EXTENDED := with_addr.sh \ TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ test_lirc_mode2_user xdping test_cpp runqslower bench bpf_testmod.ko \ - xdpxceiver + xdpxceiver xdp_redirect_multi TEST_CUSTOM_PROGS = $(OUTPUT)/urandom_read diff --git a/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c new file mode 100644 index 000000000000..dce4df40d9de --- /dev/null +++ b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c @@ -0,0 +1,111 @@ +// SPDX-License-Identifier: GPL-2.0 +#define KBUILD_MODNAME "foo" +#include <string.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ip.h> +#include <linux/ipv6.h> + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_endian.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v4 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v6 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_all SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(struct bpf_devmap_val)); + __uint(max_entries, 128); +} forward_map_egress SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} exclude_map SEC(".maps"); + +/* map to store egress interfaces mac addresses */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, __u32); + __type(value, __be64); + __uint(max_entries, 128); +} mac_map SEC(".maps"); + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + struct ethhdr *eth = data; + __u16 h_proto; + __u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + h_proto = eth->h_proto; + + if (h_proto == bpf_htons(ETH_P_IP)) + return bpf_redirect_map_multi(&forward_map_v4, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else if (h_proto == bpf_htons(ETH_P_IPV6)) + return bpf_redirect_map_multi(&forward_map_v6, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else + return bpf_redirect_map_multi(&forward_map_all, NULL, + BPF_F_EXCLUDE_INGRESS); +} + +/* The following 2 progs are for 2nd devmap prog testing */ +SEC("xdp_redirect_map_ingress") +int xdp_redirect_map_all_prog(struct xdp_md *ctx) +{ + return bpf_redirect_map_multi(&forward_map_egress, NULL, BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_devmap/map_prog") +int xdp_devmap_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + __u32 key = ctx->egress_ifindex; + struct ethhdr *eth = data; + __u64 nh_off; + __be64 *mac; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + mac = bpf_map_lookup_elem(&mac_map, &key); + if (mac) + __builtin_memcpy(eth->h_source, mac, ETH_ALEN); + + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh new file mode 100755 index 000000000000..6503751fdca5 --- /dev/null +++ b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh @@ -0,0 +1,208 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Test topology: +# - - - - - - - - - - - - - - - - - - - - - - - - - +# | veth1 veth2 veth3 veth4 | ... init net +# - -| - - - - - - | - - - - - - | - - - - - - | - - +# --------- --------- --------- --------- +# | veth0 | | veth0 | | veth0 | | veth0 | ... +# --------- --------- --------- --------- +# ns1 ns2 ns3 ns4 +# +# Forward maps: +# Forward map_all has interfaces: veth1, veth2, veth3, veth4, ... (All traffic except IPv4, IPv6) +# Forward map_v4 has interfaces: veth1, veth3, veth4, ... (For IPv4 traffic only) +# Forward map_v6 has interfaces: veth2, veth3, veth4, ... (For IPv6 traffic only) +# Forward map_egress has all interfaces and redirect all pkts +# Exclude Groups: +# Exclude map: veth3 (assume ns3 is in black list) +# Map type: +# map_v4 use DEVMAP, others use DEVMAP_HASH +# +# Test modules: +# XDP modes: generic, native, native + egress_prog +# +# Test cases: +# ARP(we didn't block ARP for ns3): +# ns1 -> gw: ns2, ns3, ns4 should receive the arp request +# IPv4: +# ns1 -> ns2 (block), ns1 -> ns3 (block), ns1 -> ns4 (pass) +# IPv6 +# ns2 -> ns1 (block), ns2 -> ns3 (block), ns2 -> ns4 (pass) +# egress_prog: +# all ping test should pass, the src mac should be egress interface's mac +# + + +# netns numbers +NUM=4 +IFACES="" +DRV_MODE="xdpgeneric xdpdrv xdpegress" +PASS=0 +FAIL=0 + +test_pass() +{ + echo "Pass: $@" + PASS=$((PASS + 1)) +} + +test_fail() +{ + echo "fail: $@" + FAIL=$((FAIL + 1)) +} + +clean_up() +{ + for i in $(seq $NUM); do + ip link del veth$i 2> /dev/null + ip netns del ns$i 2> /dev/null + done +} + +# Kselftest framework requirement - SKIP code is 4. +check_env() +{ + ip link set dev lo xdpgeneric off &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without the ip xdpgeneric support" + exit 4 + fi + + which tcpdump &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without tcpdump" + exit 4 + fi +} + +setup_ns() +{ + local mode=$1 + IFACES="" + + if [ "$mode" = "xdpegress" ]; then + mode="xdpdrv" + fi + + for i in $(seq $NUM); do + ip netns add ns$i + ip link add veth$i type veth peer name veth0 netns ns$i + ip link set veth$i up + ip -n ns$i link set veth0 up + + ip -n ns$i addr add 192.0.2.$i/24 dev veth0 + ip -n ns$i addr add 2001:db8::$i/64 dev veth0 + ip -n ns$i link set veth0 $mode obj \ + xdp_dummy.o sec xdp_dummy &> /dev/null || \ + { test_fail "Unable to load dummy xdp" && exit 1; } + IFACES="$IFACES veth$i" + veth_mac[$i]=$(ip link show veth$i | awk '/link\/ether/ {print $2}') + done +} + +do_egress_tests() +{ + local mode=$1 + + # mac test + ip netns exec ns2 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + + # mac check + grep -q "${veth_mac[2]} > ff:ff:ff:ff:ff:ff" mac_ns1-2_${mode}.log && \ + test_pass "$mode mac ns1-2" || test_fail "$mode mac ns1-2" + grep -q "${veth_mac[3]} > ff:ff:ff:ff:ff:ff" mac_ns1-3_${mode}.log && \ + test_pass "$mode mac ns1-3" || test_fail "$mode mac ns1-3" + grep -q "${veth_mac[4]} > ff:ff:ff:ff:ff:ff" mac_ns1-4_${mode}.log && \ + test_pass "$mode mac ns1-4" || test_fail "$mode mac ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-2" || test_fail "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-3" || test_fail "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" +} + +do_ping_tests() +{ + local mode=$1 + + # arp test + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -i veth0 -nn -l -e &> arp_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-2_${mode}.log && \ + test_pass "$mode arp ns1-2" || test_fail "$mode arp ns1-2" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-3_${mode}.log && \ + test_pass "$mode arp ns1-3" || test_fail "$mode arp ns1-3" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-4_${mode}.log && \ + test_pass "$mode arp ns1-4" || test_fail "$mode arp ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-2" || test_pass "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-3" || test_pass "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" + + # ping6 test + ip netns exec ns2 ping6 2001:db8::1 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-1" || test_pass "$mode ping6 ns2-1" + ip netns exec ns2 ping6 2001:db8::3 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-3" || test_pass "$mode ping6 ns2-3" + ip netns exec ns2 ping6 2001:db8::4 -c 4 &> /dev/null && \ + test_pass "$mode ping6 ns2-4" || test_fail "$mode ping6 ns2-4" +} + +do_tests() +{ + local mode=$1 + local drv_p + + case ${mode} in + xdpdrv) drv_p="-N";; + xdpegress) drv_p="-X";; + xdpgeneric) drv_p="-S";; + esac + + ./xdp_redirect_multi $drv_p $IFACES &> xdp_redirect_${mode}.log & + xdp_pid=$! + sleep 10 + + if [ "$mode" = "xdpegress" ]; then + do_egress_tests $mode + else + do_ping_tests $mode + fi + + kill $xdp_pid +} + +trap clean_up 0 2 3 6 9 + +check_env +rm -f xdp_redirect_*.log arp_ns*.log mac_ns*.log + +for mode in ${DRV_MODE}; do + setup_ns $mode + do_tests $mode + sleep 10 + clean_up + sleep 5 +done + +echo "Summary: PASS $PASS, FAIL $FAIL" +[ $FAIL -eq 0 ] && exit 0 || exit 1 diff --git a/tools/testing/selftests/bpf/xdp_redirect_multi.c b/tools/testing/selftests/bpf/xdp_redirect_multi.c new file mode 100644 index 000000000000..b43cd3c9eefd --- /dev/null +++ b/tools/testing/selftests/bpf/xdp_redirect_multi.c @@ -0,0 +1,252 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> +#include <sys/resource.h> +#include <sys/ioctl.h> +#include <sys/types.h> +#include <sys/socket.h> +#include <netinet/in.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static int get_mac_addr(unsigned int ifindex, void *mac_addr) +{ + char ifname[IF_NAMESIZE]; + struct ifreq ifr; + int fd, ret = -1; + + fd = socket(AF_INET, SOCK_DGRAM, 0); + if (fd < 0) + return ret; + + if (!if_indextoname(ifindex, ifname)) + goto err_out; + + strcpy(ifr.ifr_name, ifname); + + if (ioctl(fd, SIOCGIFHWADDR, &ifr) != 0) + goto err_out; + + memcpy(mac_addr, ifr.ifr_hwaddr.sa_data, 6 * sizeof(char)); + ret = 0; + +err_out: + close(fd); + return ret; +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n" + " -X load xdp program on egress\n", + prog); +} + +int main(int argc, char **argv) +{ + int prog_fd, group_all, group_v4, group_v6, exclude, mac_map; + struct bpf_program *ingress_prog, *egress_prog; + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_UNSPEC, + }; + int i, ret, opt, egress_prog_fd = 0; + struct bpf_devmap_val devmap_val; + bool attach_egress_prog = false; + unsigned char mac_addr[6]; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + unsigned int ifindex; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNFX")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + case 'X': + attach_egress_prog = true; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) { + xdp_flags |= XDP_FLAGS_DRV_MODE; + } else if (attach_egress_prog) { + printf("Load xdp program on egress with SKB mode not supported yet\n"); + goto err_out; + } + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + goto err_out; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + goto err_out; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + goto err_out; + + if (attach_egress_prog) + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_egress"); + else + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_all"); + group_v4 = bpf_object__find_map_fd_by_name(obj, "forward_map_v4"); + group_v6 = bpf_object__find_map_fd_by_name(obj, "forward_map_v6"); + exclude = bpf_object__find_map_fd_by_name(obj, "exclude_map"); + mac_map = bpf_object__find_map_fd_by_name(obj, "mac_map"); + + if (group_all < 0 || group_v4 < 0 || group_v6 < 0 || exclude < 0 || + mac_map < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + goto err_out; + } + + if (attach_egress_prog) { + /* Find ingress/egress prog for 2nd xdp prog */ + ingress_prog = bpf_object__find_program_by_name(obj, "xdp_redirect_map_all_prog"); + egress_prog = bpf_object__find_program_by_name(obj, "xdp_devmap_prog"); + if (!ingress_prog || !egress_prog) { + printf("finding ingress/egress_prog in obj file failed\n"); + goto err_out; + } + prog_fd = bpf_program__fd(ingress_prog); + egress_prog_fd = bpf_program__fd(egress_prog); + if (prog_fd < 0 || egress_prog_fd < 0) { + printf("find egress_prog fd failed\n"); + goto err_out; + } + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + if (attach_egress_prog) { + ret = get_mac_addr(ifindex, mac_addr); + if (ret < 0) { + printf("get interface %d mac failed\n", ifindex); + goto err_out; + } + ret = bpf_map_update_elem(mac_map, &ifindex, mac_addr, 0); + if (ret) { + perror("bpf_update_elem mac_map failed\n"); + goto err_out; + } + } + + /* Add all the interfaces to group all */ + devmap_val.ifindex = ifindex; + devmap_val.bpf_prog.fd = egress_prog_fd; + ret = bpf_map_update_elem(group_all, &ifindex, &devmap_val, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* For testing: remove the 1st interfaces from group v6 */ + if (i != 0) { + ret = bpf_map_update_elem(group_v6, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: remove the 2nd interfaces from group v4 */ + if (i != 1) { + ret = bpf_map_update_elem(group_v4, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: add the 3rd interfaces to exclude map */ + if (i == 2) { + ret = bpf_map_update_elem(exclude, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + } + + /* sleep some time for testing */ + sleep(999); + + return 0; + +err_out: + return 1; +} -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* Re: [PATCHv16 bpf-next 0/6] xdp: add a new helper for dev map multicast support 2021-01-22 7:46 ` [PATCHv16 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (5 preceding siblings ...) 2021-01-22 7:46 ` [PATCHv16 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu @ 2021-01-22 13:43 ` Toke Høiland-Jørgensen 2021-01-25 12:45 ` [PATCHv17 " Hangbin Liu 7 siblings, 0 replies; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2021-01-22 13:43 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu Hangbin Liu <liuhangbin@gmail.com> writes: > This patch is for xdp multicast support. which has been discussed before[0], > The goal is to be able to implement an OVS-like data plane in XDP, i.e., > a software switch that can forward XDP frames to multiple ports. > > To achieve this, an application needs to specify a group of interfaces > to forward a packet to. It is also common to want to exclude one or more > physical interfaces from the forwarding operation - e.g., to forward a > packet to all interfaces in the multicast group except the interface it > arrived on. While this could be done simply by adding more groups, this > quickly leads to a combinatorial explosion in the number of groups an > application has to maintain. > > To avoid the combinatorial explosion, we propose to include the ability > to specify an "exclude group" as part of the forwarding operation. This > needs to be a group (instead of just a single port index), because there > may have multi interfaces you want to exclude. > > Thus, the logical forwarding operation becomes a "set difference" > operation, i.e. "forward to all ports in group A that are not also in > group B". This series implements such an operation using device maps to > represent the groups. This means that the XDP program specifies two > device maps, one containing the list of netdevs to redirect to, and the > other containing the exclude list. > > To achieve this, I re-implement a new helper bpf_redirect_map_multi() > to accept two maps, the forwarding map and exclude map. If user > don't want to use exclude map and just want simply stop redirecting back > to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. > > The 1st patch is Jesper's run devmap xdp_prog later in bulking step. > The 2st patch add a new bpf arg to allow NULL map pointer. > The 3rd patch add the new bpf_redirect_map_multi() helper. > The 4-6 patches are for usage sample and testing purpose. > > I did same perf tests with the following topo: > > --------------------- --------------------- > | Host A (i40e 10G) | ---------- | eno1(i40e 10G) | > --------------------- | | > | Host B | > --------------------- | | > | Host C (i40e 10G) | ---------- | eno2(i40e 10G) | > --------------------- vlan2 | -------- | > | veth1 -- | veth0| | > | -------- | > --------------------| > On Host A: > # pktgen/pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -s 64 > > On Host B(Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, 128G Memory): > Use xdp_redirect_map and xdp_redirect_map_multi in samples/bpf for testing. > The veth0 in netns load dummy drop program. The forward_map max_entries in > xdp_redirect_map_multi is modify to 4. > > Here is the perf result with 5.10 rc6: > > The are about +/- 0.1M deviation for native testing > Version | Test | Generic | Native | Native + 2nd > 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M > 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M > 5.10 rc6 + patch1 | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M > 5.10 rc6 + patch1 | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M > 5.10 rc6 + patch1-6 | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M > 5.10 rc6 + patch1-6 | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M > 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->i40e | 1.7M | 7.8M | 6.4M > 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->veth | 1.4M | 9.3M | 7.5M > 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->i40e+veth | 1.0M | 3.2M | 2.7M > > Last but not least, thanks a lot to Toke, Jesper, Jiri and Eelco for > suggestions and help on implementation. Nice work, and thank you for sticking with this! With the last couple of fixes discussed for patch 1, when you resubmit please add my: Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> for the series! -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv17 bpf-next 0/6] xdp: add a new helper for dev map multicast support 2021-01-22 7:46 ` [PATCHv16 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (6 preceding siblings ...) 2021-01-22 13:43 ` [PATCHv16 bpf-next 0/6] xdp: add a new helper for dev map multicast support Toke Høiland-Jørgensen @ 2021-01-25 12:45 ` Hangbin Liu 2021-01-25 12:45 ` [PATCHv17 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu ` (7 more replies) 7 siblings, 8 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-25 12:45 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because there may have multi interfaces you want to exclude. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. The 1st patch is Jesper's run devmap xdp_prog later in bulking step. The 2st patch add a new bpf arg to allow NULL map pointer. The 3rd patch add the new bpf_redirect_map_multi() helper. The 4-6 patches are for usage sample and testing purpose. I did same perf tests with the following topo: --------------------- --------------------- | Host A (i40e 10G) | ---------- | eno1(i40e 10G) | --------------------- | | | Host B | --------------------- | | | Host C (i40e 10G) | ---------- | eno2(i40e 10G) | --------------------- vlan2 | -------- | | veth1 -- | veth0| | | -------- | --------------------| On Host A: # pktgen/pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -s 64 On Host B(Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, 128G Memory): Use xdp_redirect_map and xdp_redirect_map_multi in samples/bpf for testing. The veth0 in netns load dummy drop program. The forward_map max_entries in xdp_redirect_map_multi is modify to 4. Here is the perf result with 5.10 rc6: The are about +/- 0.1M deviation for native testing Version | Test | Generic | Native | Native + 2nd 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M 5.10 rc6 + patch1 | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch1 | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M 5.10 rc6 + patch1-6 | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch1-6 | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->i40e | 1.7M | 7.8M | 6.4M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->veth | 1.4M | 9.3M | 7.5M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->i40e+veth | 1.0M | 3.2M | 2.7M Last but not least, thanks a lot to Toke, Jesper, Jiri and Eelco for suggestions and help on implementation. [0] https://xdp-project.net/#Handling-multicast v17: For patch 01: a) rename to_sent to to_send. b) clear bq dev_rx, xdp_prog and flush_node in __dev_flush(). v16: refactor bq_xmit_all logic and remove error label for patch 01 v15: Update bq_xmit_all() logic for patch 01. Add some comments and remove useless variable for patch 03. Use bpf_object__find_program_by_title() for patch 04 and 06. v14: No code update, just rebase the code on latest bpf-next v13: Pass in xdp_prog through __xdp_enqueue() for patch 01. Update related code in patch 03. v12: Add Jesper's xdp_prog patch, rebase my works on this and latest bpf-next Add 2nd xdp_prog test on the sample and selftests. v11: Fix bpf_redirect_map_multi() helper description typo. Add loop limit for devmap_get_next_obj() and dev_map_redirect_multi(). v10: Rebase the code to latest bpf-next. Update helper bpf_xdp_redirect_map_multi() - No need to check map pointer as we will do the check in verifier. v9: Update helper bpf_xdp_redirect_map_multi() - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: a) Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). f) Split the tests from sample and add a bpf kernel selftest patch. v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Hangbin Liu (5): bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL xdp: add a new helper for dev map multicast support sample/bpf: add xdp_redirect_map_multicast test selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL selftests/bpf: add xdp_redirect_multi test Jesper Dangaard Brouer (1): bpf: run devmap xdp_prog on flush instead of bulk enqueue include/linux/bpf.h | 21 ++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 28 ++ kernel/bpf/devmap.c | 262 +++++++++++---- kernel/bpf/verifier.c | 16 +- net/core/filter.c | 124 ++++++- net/core/xdp.c | 29 ++ samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 87 +++++ samples/bpf/xdp_redirect_map_multi_user.c | 302 ++++++++++++++++++ tools/include/uapi/linux/bpf.h | 28 ++ tools/testing/selftests/bpf/Makefile | 3 +- .../bpf/progs/xdp_redirect_multi_kern.c | 111 +++++++ tools/testing/selftests/bpf/test_verifier.c | 22 +- .../selftests/bpf/test_xdp_redirect_multi.sh | 208 ++++++++++++ .../testing/selftests/bpf/verifier/map_ptr.c | 70 ++++ .../selftests/bpf/xdp_redirect_multi.c | 252 +++++++++++++++ 18 files changed, 1501 insertions(+), 67 deletions(-) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c -- 2.26.2 ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv17 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-25 12:45 ` [PATCHv17 " Hangbin Liu @ 2021-01-25 12:45 ` Hangbin Liu 2021-01-27 7:37 ` John Fastabend 2021-01-27 15:58 ` Jesper Dangaard Brouer 2021-01-25 12:45 ` [PATCHv17 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu ` (6 subsequent siblings) 7 siblings, 2 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-25 12:45 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu From: Jesper Dangaard Brouer <brouer@redhat.com> This changes the devmap XDP program support to run the program when the bulk queue is flushed instead of before the frame is enqueued. This has a couple of benefits: - It "sorts" the packets by destination devmap entry, and then runs the same BPF program on all the packets in sequence. This ensures that we keep the XDP program and destination device properties hot in I-cache. - It makes the multicast implementation simpler because it can just enqueue packets using bq_enqueue() without having to deal with the devmap program at all. The drawback is that if the devmap program drops the packet, the enqueue step is redundant. However, arguably this is mostly visible in a micro-benchmark, and with more mixed traffic the I-cache benefit should win out. The performance impact of just this patch is as follows: The bq_xmit_all's logic is also refactored and error label is removed. When bq_xmit_all() is called from bq_enqueue(), another packet will always be enqueued immediately after, so clearing dev_rx, xdp_prog and flush_node in bq_xmit_all() is redundant. Let's move the clear to __dev_flush(), and only check them once in bq_enqueue() since they are all modified together. By using xdp_redirect_map in sample/bpf and send pkts via pktgen cmd: ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 There are about +/- 0.1M deviation for native testing, the performance improved for the base-case, but some drop back with xdp devmap prog attached. Version | Test | Generic | Native | Native + 2nd xdp_prog 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M 5.10 rc6 + patch | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M [1] https://lore.kernel.org/bpf/20210122025007.2968381-1-liuhangbin@gmail.com Reviewed-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com> Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v17: a) rename to_sent to to_send. b) clear bq dev_rx, xdp_prog and flush_node in __dev_flush(). v16: a) refactor bq_xmit_all logic and remove error label v15: a) do not use unlikely when checking bq->xdp_prog b) return sent frames for dev_map_bpf_prog_run() v14: no update, only rebase the code v13: pass in xdp_prog through __xdp_enqueue() v2-v12: no this patch --- kernel/bpf/devmap.c | 146 +++++++++++++++++++++++++------------------- 1 file changed, 84 insertions(+), 62 deletions(-) diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index f6e9c68afdd4..bf8b6b5c9cab 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -57,6 +57,7 @@ struct xdp_dev_bulk_queue { struct list_head flush_node; struct net_device *dev; struct net_device *dev_rx; + struct bpf_prog *xdp_prog; unsigned int count; }; @@ -327,46 +328,92 @@ bool dev_map_can_have_prog(struct bpf_map *map) return false; } +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, + struct xdp_frame **frames, int n, + struct net_device *dev) +{ + struct xdp_txq_info txq = { .dev = dev }; + struct xdp_buff xdp; + int i, nframes = 0; + + for (i = 0; i < n; i++) { + struct xdp_frame *xdpf = frames[i]; + u32 act; + int err; + + xdp_convert_frame_to_buff(xdpf, &xdp); + xdp.txq = &txq; + + act = bpf_prog_run_xdp(xdp_prog, &xdp); + switch (act) { + case XDP_PASS: + err = xdp_update_frame_from_buff(&xdp, xdpf); + if (unlikely(err < 0)) + xdp_return_frame_rx_napi(xdpf); + else + frames[nframes++] = xdpf; + break; + default: + bpf_warn_invalid_xdp_action(act); + fallthrough; + case XDP_ABORTED: + trace_xdp_exception(dev, xdp_prog, act); + fallthrough; + case XDP_DROP: + xdp_return_frame_rx_napi(xdpf); + break; + } + } + return nframes; /* sent frames count */ +} + static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) { struct net_device *dev = bq->dev; - int sent = 0, drops = 0, err = 0; + unsigned int cnt = bq->count; + int drops = 0, err = 0; + int to_send = cnt; + int sent = cnt; int i; - if (unlikely(!bq->count)) + if (unlikely(!cnt)) return; - for (i = 0; i < bq->count; i++) { + for (i = 0; i < cnt; i++) { struct xdp_frame *xdpf = bq->q[i]; prefetch(xdpf); } - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); + if (bq->xdp_prog) { + to_send = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); + if (!to_send) { + sent = 0; + goto out; + } + drops = cnt - to_send; + } + + sent = dev->netdev_ops->ndo_xdp_xmit(dev, to_send, bq->q, flags); if (sent < 0) { err = sent; sent = 0; - goto error; + + /* If ndo_xdp_xmit fails with an errno, no frames have been + * xmit'ed and it's our responsibility to them free all. + */ + for (i = 0; i < cnt - drops; i++) { + struct xdp_frame *xdpf = bq->q[i]; + + xdp_return_frame_rx_napi(xdpf); + } } - drops = bq->count - sent; out: + drops = cnt - sent; bq->count = 0; trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); - bq->dev_rx = NULL; - __list_del_clearprev(&bq->flush_node); return; -error: - /* If ndo_xdp_xmit fails with an errno, no frames have been - * xmit'ed and it's our responsibility to them free all. - */ - for (i = 0; i < bq->count; i++) { - struct xdp_frame *xdpf = bq->q[i]; - - xdp_return_frame_rx_napi(xdpf); - drops++; - } - goto out; } /* __dev_flush is called from xdp_do_flush() which _must_ be signaled @@ -384,8 +431,12 @@ void __dev_flush(void) struct list_head *flush_list = this_cpu_ptr(&dev_flush_list); struct xdp_dev_bulk_queue *bq, *tmp; - list_for_each_entry_safe(bq, tmp, flush_list, flush_node) + list_for_each_entry_safe(bq, tmp, flush_list, flush_node) { bq_xmit_all(bq, XDP_XMIT_FLUSH); + bq->dev_rx = NULL; + bq->xdp_prog = NULL; + __list_del_clearprev(&bq->flush_node); + } } /* rcu_read_lock (from syscall and BPF contexts) ensures that if a delete and/or @@ -408,7 +459,7 @@ struct bpf_dtab_netdev *__dev_map_lookup_elem(struct bpf_map *map, u32 key) * Thus, safe percpu variable access. */ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, - struct net_device *dev_rx) + struct net_device *dev_rx, struct bpf_prog *xdp_prog) { struct list_head *flush_list = this_cpu_ptr(&dev_flush_list); struct xdp_dev_bulk_queue *bq = this_cpu_ptr(dev->xdp_bulkq); @@ -419,18 +470,22 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, /* Ingress dev_rx will be the same for all xdp_frame's in * bulk_queue, because bq stored per-CPU and must be flushed * from net_device drivers NAPI func end. + * + * Do the same with xdp_prog and flush_list since these fields + * are only ever modified together. */ - if (!bq->dev_rx) + if (!bq->dev_rx) { bq->dev_rx = dev_rx; + bq->xdp_prog = xdp_prog; + list_add(&bq->flush_node, flush_list); + } bq->q[bq->count++] = xdpf; - - if (!bq->flush_node.prev) - list_add(&bq->flush_node, flush_list); } static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, - struct net_device *dev_rx) + struct net_device *dev_rx, + struct bpf_prog *xdp_prog) { struct xdp_frame *xdpf; int err; @@ -446,42 +501,14 @@ static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, if (unlikely(!xdpf)) return -EOVERFLOW; - bq_enqueue(dev, xdpf, dev_rx); + bq_enqueue(dev, xdpf, dev_rx, xdp_prog); return 0; } -static struct xdp_buff *dev_map_run_prog(struct net_device *dev, - struct xdp_buff *xdp, - struct bpf_prog *xdp_prog) -{ - struct xdp_txq_info txq = { .dev = dev }; - u32 act; - - xdp_set_data_meta_invalid(xdp); - xdp->txq = &txq; - - act = bpf_prog_run_xdp(xdp_prog, xdp); - switch (act) { - case XDP_PASS: - return xdp; - case XDP_DROP: - break; - default: - bpf_warn_invalid_xdp_action(act); - fallthrough; - case XDP_ABORTED: - trace_xdp_exception(dev, xdp_prog, act); - break; - } - - xdp_return_buff(xdp); - return NULL; -} - int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx) { - return __xdp_enqueue(dev, xdp, dev_rx); + return __xdp_enqueue(dev, xdp, dev_rx, NULL); } int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, @@ -489,12 +516,7 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, { struct net_device *dev = dst->dev; - if (dst->xdp_prog) { - xdp = dev_map_run_prog(dev, xdp, dst->xdp_prog); - if (!xdp) - return 0; - } - return __xdp_enqueue(dev, xdp, dev_rx); + return __xdp_enqueue(dev, xdp, dev_rx, dst->xdp_prog); } int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* RE: [PATCHv17 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-25 12:45 ` [PATCHv17 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu @ 2021-01-27 7:37 ` John Fastabend 2021-01-27 9:41 ` Toke Høiland-Jørgensen 2021-01-27 15:58 ` Jesper Dangaard Brouer 1 sibling, 1 reply; 219+ messages in thread From: John Fastabend @ 2021-01-27 7:37 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu Hangbin Liu wrote: > From: Jesper Dangaard Brouer <brouer@redhat.com> > > This changes the devmap XDP program support to run the program when the > bulk queue is flushed instead of before the frame is enqueued. This has > a couple of benefits: > > - It "sorts" the packets by destination devmap entry, and then runs the > same BPF program on all the packets in sequence. This ensures that we > keep the XDP program and destination device properties hot in I-cache. > > - It makes the multicast implementation simpler because it can just > enqueue packets using bq_enqueue() without having to deal with the > devmap program at all. > > The drawback is that if the devmap program drops the packet, the enqueue > step is redundant. However, arguably this is mostly visible in a > micro-benchmark, and with more mixed traffic the I-cache benefit should > win out. The performance impact of just this patch is as follows: > > The bq_xmit_all's logic is also refactored and error label is removed. > When bq_xmit_all() is called from bq_enqueue(), another packet will > always be enqueued immediately after, so clearing dev_rx, xdp_prog and > flush_node in bq_xmit_all() is redundant. Let's move the clear to > __dev_flush(), and only check them once in bq_enqueue() since they are > all modified together. > > By using xdp_redirect_map in sample/bpf and send pkts via pktgen cmd: > ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 > > There are about +/- 0.1M deviation for native testing, the performance > improved for the base-case, but some drop back with xdp devmap prog attached. > > Version | Test | Generic | Native | Native + 2nd xdp_prog > 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M > 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M > 5.10 rc6 + patch | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M > 5.10 rc6 + patch | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M > [...] > +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, > + struct xdp_frame **frames, int n, > + struct net_device *dev) > +{ > + struct xdp_txq_info txq = { .dev = dev }; > + struct xdp_buff xdp; > + int i, nframes = 0; > + > + for (i = 0; i < n; i++) { > + struct xdp_frame *xdpf = frames[i]; > + u32 act; > + int err; > + > + xdp_convert_frame_to_buff(xdpf, &xdp); > + xdp.txq = &txq; > + > + act = bpf_prog_run_xdp(xdp_prog, &xdp); > + switch (act) { > + case XDP_PASS: > + err = xdp_update_frame_from_buff(&xdp, xdpf); > + if (unlikely(err < 0)) > + xdp_return_frame_rx_napi(xdpf); > + else > + frames[nframes++] = xdpf; > + break; > + default: > + bpf_warn_invalid_xdp_action(act); > + fallthrough; > + case XDP_ABORTED: > + trace_xdp_exception(dev, xdp_prog, act); > + fallthrough; > + case XDP_DROP: > + xdp_return_frame_rx_napi(xdpf); > + break; > + } > + } > + return nframes; /* sent frames count */ > +} > + > static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) > { > struct net_device *dev = bq->dev; > - int sent = 0, drops = 0, err = 0; > + unsigned int cnt = bq->count; > + int drops = 0, err = 0; > + int to_send = cnt; > + int sent = cnt; > int i; > > - if (unlikely(!bq->count)) > + if (unlikely(!cnt)) > return; > > - for (i = 0; i < bq->count; i++) { > + for (i = 0; i < cnt; i++) { > struct xdp_frame *xdpf = bq->q[i]; > > prefetch(xdpf); > } > > - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); > + if (bq->xdp_prog) { > + to_send = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); > + if (!to_send) { > + sent = 0; > + goto out; > + } > + drops = cnt - to_send; > + } I might be missing something about how *bq works here. What happens when dev_map_bpf_prog_run returns to_send < cnt? So I read this as it will send [0, to_send] and [to_send, cnt] will be dropped? How do we know the bpf prog would have dropped the set, [to_send+1, cnt]? > + > + sent = dev->netdev_ops->ndo_xdp_xmit(dev, to_send, bq->q, flags); > if (sent < 0) { > err = sent; > sent = 0; > - goto error; > + > + /* If ndo_xdp_xmit fails with an errno, no frames have been > + * xmit'ed and it's our responsibility to them free all. > + */ > + for (i = 0; i < cnt - drops; i++) { > + struct xdp_frame *xdpf = bq->q[i]; > + > + xdp_return_frame_rx_napi(xdpf); > + } > } > - drops = bq->count - sent; > out: > + drops = cnt - sent; > bq->count = 0; > > trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); I gather the remaining [to_send+1, cnt] packets are in fact dropped because we set bq->count=0 and trace_xdp_devmap_xmit seems to think they are dropped. Also update the comment in trace_xdp_devmap_xmit, /* Catch API error of drv ndo_xdp_xmit sent more than count */ so that it reads to account for devmap progs as well? > - bq->dev_rx = NULL; > - __list_del_clearprev(&bq->flush_node); > return; > -error: > - /* If ndo_xdp_xmit fails with an errno, no frames have been > - * xmit'ed and it's our responsibility to them free all. > - */ > - for (i = 0; i < bq->count; i++) { > - struct xdp_frame *xdpf = bq->q[i]; > - > - xdp_return_frame_rx_napi(xdpf); > - drops++; > - } > - goto out; > } [...] > -static struct xdp_buff *dev_map_run_prog(struct net_device *dev, > - struct xdp_buff *xdp, > - struct bpf_prog *xdp_prog) > -{ > - struct xdp_txq_info txq = { .dev = dev }; > - u32 act; > - > - xdp_set_data_meta_invalid(xdp); > - xdp->txq = &txq; > - > - act = bpf_prog_run_xdp(xdp_prog, xdp); > - switch (act) { > - case XDP_PASS: > - return xdp; > - case XDP_DROP: > - break; > - default: > - bpf_warn_invalid_xdp_action(act); > - fallthrough; > - case XDP_ABORTED: > - trace_xdp_exception(dev, xdp_prog, act); > - break; > - } > - > - xdp_return_buff(xdp); > - return NULL; > -} > - > int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, > struct net_device *dev_rx) > { > - return __xdp_enqueue(dev, xdp, dev_rx); > + return __xdp_enqueue(dev, xdp, dev_rx, NULL); > } > > int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > @@ -489,12 +516,7 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > { > struct net_device *dev = dst->dev; > > - if (dst->xdp_prog) { > - xdp = dev_map_run_prog(dev, xdp, dst->xdp_prog); > - if (!xdp) > - return 0; So here it looks like dev_map_run_prog will not drop extra packets, but will see every single packet. Are we changing the semantics subtle here? This looks like a problem to me. We should not drop packets in the new case unless bpf program tells us to. > - } > - return __xdp_enqueue(dev, xdp, dev_rx); > + return __xdp_enqueue(dev, xdp, dev_rx, dst->xdp_prog); > } Did I miss something? If not maybe a comment in the commit message would help or in the code itself. Thanks, John ^ permalink raw reply [flat|nested] 219+ messages in thread
* RE: [PATCHv17 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-27 7:37 ` John Fastabend @ 2021-01-27 9:41 ` Toke Høiland-Jørgensen 2021-01-27 12:20 ` Maciej Fijalkowski 0 siblings, 1 reply; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2021-01-27 9:41 UTC (permalink / raw) To: John Fastabend, Hangbin Liu, bpf Cc: netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu John Fastabend <john.fastabend@gmail.com> writes: > Hangbin Liu wrote: >> From: Jesper Dangaard Brouer <brouer@redhat.com> >> >> This changes the devmap XDP program support to run the program when the >> bulk queue is flushed instead of before the frame is enqueued. This has >> a couple of benefits: >> >> - It "sorts" the packets by destination devmap entry, and then runs the >> same BPF program on all the packets in sequence. This ensures that we >> keep the XDP program and destination device properties hot in I-cache. >> >> - It makes the multicast implementation simpler because it can just >> enqueue packets using bq_enqueue() without having to deal with the >> devmap program at all. >> >> The drawback is that if the devmap program drops the packet, the enqueue >> step is redundant. However, arguably this is mostly visible in a >> micro-benchmark, and with more mixed traffic the I-cache benefit should >> win out. The performance impact of just this patch is as follows: >> >> The bq_xmit_all's logic is also refactored and error label is removed. >> When bq_xmit_all() is called from bq_enqueue(), another packet will >> always be enqueued immediately after, so clearing dev_rx, xdp_prog and >> flush_node in bq_xmit_all() is redundant. Let's move the clear to >> __dev_flush(), and only check them once in bq_enqueue() since they are >> all modified together. >> >> By using xdp_redirect_map in sample/bpf and send pkts via pktgen cmd: >> ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 >> >> There are about +/- 0.1M deviation for native testing, the performance >> improved for the base-case, but some drop back with xdp devmap prog attached. >> >> Version | Test | Generic | Native | Native + 2nd xdp_prog >> 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M >> 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M >> 5.10 rc6 + patch | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M >> 5.10 rc6 + patch | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M >> > > [...] > >> +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, >> + struct xdp_frame **frames, int n, >> + struct net_device *dev) >> +{ >> + struct xdp_txq_info txq = { .dev = dev }; >> + struct xdp_buff xdp; >> + int i, nframes = 0; >> + >> + for (i = 0; i < n; i++) { >> + struct xdp_frame *xdpf = frames[i]; >> + u32 act; >> + int err; >> + >> + xdp_convert_frame_to_buff(xdpf, &xdp); >> + xdp.txq = &txq; >> + >> + act = bpf_prog_run_xdp(xdp_prog, &xdp); >> + switch (act) { >> + case XDP_PASS: >> + err = xdp_update_frame_from_buff(&xdp, xdpf); >> + if (unlikely(err < 0)) >> + xdp_return_frame_rx_napi(xdpf); >> + else >> + frames[nframes++] = xdpf; >> + break; >> + default: >> + bpf_warn_invalid_xdp_action(act); >> + fallthrough; >> + case XDP_ABORTED: >> + trace_xdp_exception(dev, xdp_prog, act); >> + fallthrough; >> + case XDP_DROP: >> + xdp_return_frame_rx_napi(xdpf); >> + break; >> + } >> + } >> + return nframes; /* sent frames count */ >> +} >> + >> static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) >> { >> struct net_device *dev = bq->dev; >> - int sent = 0, drops = 0, err = 0; >> + unsigned int cnt = bq->count; >> + int drops = 0, err = 0; >> + int to_send = cnt; >> + int sent = cnt; >> int i; >> >> - if (unlikely(!bq->count)) >> + if (unlikely(!cnt)) >> return; >> >> - for (i = 0; i < bq->count; i++) { >> + for (i = 0; i < cnt; i++) { >> struct xdp_frame *xdpf = bq->q[i]; >> >> prefetch(xdpf); >> } >> >> - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); >> + if (bq->xdp_prog) { >> + to_send = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); >> + if (!to_send) { >> + sent = 0; >> + goto out; >> + } >> + drops = cnt - to_send; >> + } > > I might be missing something about how *bq works here. What happens when > dev_map_bpf_prog_run returns to_send < cnt? > > So I read this as it will send [0, to_send] and [to_send, cnt] will be > dropped? How do we know the bpf prog would have dropped the set, > [to_send+1, cnt]? Because dev_map_bpf_prog_run() compacts the array: + case XDP_PASS: + err = xdp_update_frame_from_buff(&xdp, xdpf); + if (unlikely(err < 0)) + xdp_return_frame_rx_napi(xdpf); + else + frames[nframes++] = xdpf; + break; [...] >> int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, >> @@ -489,12 +516,7 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, >> { >> struct net_device *dev = dst->dev; >> >> - if (dst->xdp_prog) { >> - xdp = dev_map_run_prog(dev, xdp, dst->xdp_prog); >> - if (!xdp) >> - return 0; > > So here it looks like dev_map_run_prog will not drop extra > packets, but will see every single packet. > > Are we changing the semantics subtle here? This looks like > a problem to me. We should not drop packets in the new case > unless bpf program tells us to. It's not a change in semantics (see above), but I'll grant you that it's subtle :) -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv17 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-27 9:41 ` Toke Høiland-Jørgensen @ 2021-01-27 12:20 ` Maciej Fijalkowski 2021-01-27 15:00 ` Jesper Dangaard Brouer 0 siblings, 1 reply; 219+ messages in thread From: Maciej Fijalkowski @ 2021-01-27 12:20 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: John Fastabend, Hangbin Liu, bpf, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov On Wed, Jan 27, 2021 at 10:41:44AM +0100, Toke Høiland-Jørgensen wrote: > John Fastabend <john.fastabend@gmail.com> writes: > > > Hangbin Liu wrote: > >> From: Jesper Dangaard Brouer <brouer@redhat.com> > >> > >> This changes the devmap XDP program support to run the program when the > >> bulk queue is flushed instead of before the frame is enqueued. This has > >> a couple of benefits: > >> > >> - It "sorts" the packets by destination devmap entry, and then runs the > >> same BPF program on all the packets in sequence. This ensures that we > >> keep the XDP program and destination device properties hot in I-cache. > >> > >> - It makes the multicast implementation simpler because it can just > >> enqueue packets using bq_enqueue() without having to deal with the > >> devmap program at all. > >> > >> The drawback is that if the devmap program drops the packet, the enqueue > >> step is redundant. However, arguably this is mostly visible in a > >> micro-benchmark, and with more mixed traffic the I-cache benefit should > >> win out. The performance impact of just this patch is as follows: > >> > >> The bq_xmit_all's logic is also refactored and error label is removed. > >> When bq_xmit_all() is called from bq_enqueue(), another packet will > >> always be enqueued immediately after, so clearing dev_rx, xdp_prog and > >> flush_node in bq_xmit_all() is redundant. Let's move the clear to > >> __dev_flush(), and only check them once in bq_enqueue() since they are > >> all modified together. > >> > >> By using xdp_redirect_map in sample/bpf and send pkts via pktgen cmd: > >> ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 > >> > >> There are about +/- 0.1M deviation for native testing, the performance > >> improved for the base-case, but some drop back with xdp devmap prog attached. > >> > >> Version | Test | Generic | Native | Native + 2nd xdp_prog > >> 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M > >> 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M > >> 5.10 rc6 + patch | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M > >> 5.10 rc6 + patch | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M > >> > > > > [...] > > > >> +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, > >> + struct xdp_frame **frames, int n, > >> + struct net_device *dev) > >> +{ > >> + struct xdp_txq_info txq = { .dev = dev }; > >> + struct xdp_buff xdp; > >> + int i, nframes = 0; > >> + > >> + for (i = 0; i < n; i++) { > >> + struct xdp_frame *xdpf = frames[i]; > >> + u32 act; > >> + int err; > >> + > >> + xdp_convert_frame_to_buff(xdpf, &xdp); > >> + xdp.txq = &txq; > >> + > >> + act = bpf_prog_run_xdp(xdp_prog, &xdp); > >> + switch (act) { > >> + case XDP_PASS: > >> + err = xdp_update_frame_from_buff(&xdp, xdpf); > >> + if (unlikely(err < 0)) > >> + xdp_return_frame_rx_napi(xdpf); > >> + else > >> + frames[nframes++] = xdpf; > >> + break; > >> + default: > >> + bpf_warn_invalid_xdp_action(act); > >> + fallthrough; > >> + case XDP_ABORTED: > >> + trace_xdp_exception(dev, xdp_prog, act); > >> + fallthrough; > >> + case XDP_DROP: > >> + xdp_return_frame_rx_napi(xdpf); > >> + break; > >> + } > >> + } > >> + return nframes; /* sent frames count */ > >> +} > >> + > >> static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) > >> { > >> struct net_device *dev = bq->dev; > >> - int sent = 0, drops = 0, err = 0; > >> + unsigned int cnt = bq->count; > >> + int drops = 0, err = 0; > >> + int to_send = cnt; > >> + int sent = cnt; > >> int i; > >> > >> - if (unlikely(!bq->count)) > >> + if (unlikely(!cnt)) > >> return; > >> > >> - for (i = 0; i < bq->count; i++) { > >> + for (i = 0; i < cnt; i++) { > >> struct xdp_frame *xdpf = bq->q[i]; > >> > >> prefetch(xdpf); > >> } > >> > >> - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); > >> + if (bq->xdp_prog) { > >> + to_send = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); > >> + if (!to_send) { > >> + sent = 0; > >> + goto out; > >> + } > >> + drops = cnt - to_send; > >> + } > > > > I might be missing something about how *bq works here. What happens when > > dev_map_bpf_prog_run returns to_send < cnt? > > > > So I read this as it will send [0, to_send] and [to_send, cnt] will be > > dropped? How do we know the bpf prog would have dropped the set, > > [to_send+1, cnt]? You know that via recalculation of 'drops' value after you returned from dev_map_bpf_prog_run() which later on is provided onto trace_xdp_devmap_xmit. > > Because dev_map_bpf_prog_run() compacts the array: > > + case XDP_PASS: > + err = xdp_update_frame_from_buff(&xdp, xdpf); > + if (unlikely(err < 0)) > + xdp_return_frame_rx_napi(xdpf); > + else > + frames[nframes++] = xdpf; > + break; To expand this a little, 'frames' array is reused and 'nframes' above is the value that is returned and we store it onto 'to_send' variable. > > [...] > > >> int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > >> @@ -489,12 +516,7 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > >> { > >> struct net_device *dev = dst->dev; > >> > >> - if (dst->xdp_prog) { > >> - xdp = dev_map_run_prog(dev, xdp, dst->xdp_prog); > >> - if (!xdp) > >> - return 0; > > > > So here it looks like dev_map_run_prog will not drop extra > > packets, but will see every single packet. > > > > Are we changing the semantics subtle here? This looks like > > a problem to me. We should not drop packets in the new case > > unless bpf program tells us to. > > It's not a change in semantics (see above), but I'll grant you that it's > subtle :) dev map xdp prog still sees all of the frames. Maybe you were referring to a fact that for XDP_PASS action you might fail with xdp->xdpf conversion? I'm wondering if we could actually do a further optimization and avoid xdpf/xdp juggling. What if xdp_dev_bulk_queue would be storing the xdp_buffs instead of xdp_frames ? Then you hit bq_xmit_all and if prog is present it doesn't have to do that dance like we have right now. After that you walk through xdp_buff array and do the conversion so that xdp_frame array will be passed do ndo_xdp_xmit. I had a bad sleep so maybe I'm talking nonsense over here, will take another look in the evening though :) > > -Toke > ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv17 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-27 12:20 ` Maciej Fijalkowski @ 2021-01-27 15:00 ` Jesper Dangaard Brouer 2021-01-27 15:52 ` John Fastabend 0 siblings, 1 reply; 219+ messages in thread From: Jesper Dangaard Brouer @ 2021-01-27 15:00 UTC (permalink / raw) To: Maciej Fijalkowski Cc: Toke Høiland-Jørgensen, John Fastabend, Hangbin Liu, bpf, netdev, Jiri Benc, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, brouer On Wed, 27 Jan 2021 13:20:50 +0100 Maciej Fijalkowski <maciej.fijalkowski@intel.com> wrote: > On Wed, Jan 27, 2021 at 10:41:44AM +0100, Toke Høiland-Jørgensen wrote: > > John Fastabend <john.fastabend@gmail.com> writes: > > > > > Hangbin Liu wrote: > > >> From: Jesper Dangaard Brouer <brouer@redhat.com> > > >> > > >> This changes the devmap XDP program support to run the program when the > > >> bulk queue is flushed instead of before the frame is enqueued. This has > > >> a couple of benefits: > > >> > > >> - It "sorts" the packets by destination devmap entry, and then runs the > > >> same BPF program on all the packets in sequence. This ensures that we > > >> keep the XDP program and destination device properties hot in I-cache. > > >> > > >> - It makes the multicast implementation simpler because it can just > > >> enqueue packets using bq_enqueue() without having to deal with the > > >> devmap program at all. > > >> > > >> The drawback is that if the devmap program drops the packet, the enqueue > > >> step is redundant. However, arguably this is mostly visible in a > > >> micro-benchmark, and with more mixed traffic the I-cache benefit should > > >> win out. The performance impact of just this patch is as follows: > > >> > > >> The bq_xmit_all's logic is also refactored and error label is removed. > > >> When bq_xmit_all() is called from bq_enqueue(), another packet will > > >> always be enqueued immediately after, so clearing dev_rx, xdp_prog and > > >> flush_node in bq_xmit_all() is redundant. Let's move the clear to > > >> __dev_flush(), and only check them once in bq_enqueue() since they are > > >> all modified together. > > >> > > >> By using xdp_redirect_map in sample/bpf and send pkts via pktgen cmd: > > >> ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 > > >> > > >> There are about +/- 0.1M deviation for native testing, the performance > > >> improved for the base-case, but some drop back with xdp devmap prog attached. > > >> > > >> Version | Test | Generic | Native | Native + 2nd xdp_prog > > >> 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M > > >> 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M > > >> 5.10 rc6 + patch | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M > > >> 5.10 rc6 + patch | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M > > >> > > > > > > [...] > > > > > >> +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, > > >> + struct xdp_frame **frames, int n, > > >> + struct net_device *dev) > > >> +{ > > >> + struct xdp_txq_info txq = { .dev = dev }; > > >> + struct xdp_buff xdp; > > >> + int i, nframes = 0; > > >> + > > >> + for (i = 0; i < n; i++) { > > >> + struct xdp_frame *xdpf = frames[i]; > > >> + u32 act; > > >> + int err; > > >> + > > >> + xdp_convert_frame_to_buff(xdpf, &xdp); > > >> + xdp.txq = &txq; > > >> + > > >> + act = bpf_prog_run_xdp(xdp_prog, &xdp); > > >> + switch (act) { > > >> + case XDP_PASS: > > >> + err = xdp_update_frame_from_buff(&xdp, xdpf); > > >> + if (unlikely(err < 0)) > > >> + xdp_return_frame_rx_napi(xdpf); > > >> + else > > >> + frames[nframes++] = xdpf; > > >> + break; > > >> + default: > > >> + bpf_warn_invalid_xdp_action(act); > > >> + fallthrough; > > >> + case XDP_ABORTED: > > >> + trace_xdp_exception(dev, xdp_prog, act); > > >> + fallthrough; > > >> + case XDP_DROP: > > >> + xdp_return_frame_rx_napi(xdpf); > > >> + break; > > >> + } > > >> + } > > >> + return nframes; /* sent frames count */ > > >> +} > > >> + > > >> static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) > > >> { > > >> struct net_device *dev = bq->dev; > > >> - int sent = 0, drops = 0, err = 0; > > >> + unsigned int cnt = bq->count; > > >> + int drops = 0, err = 0; > > >> + int to_send = cnt; > > >> + int sent = cnt; > > >> int i; > > >> > > >> - if (unlikely(!bq->count)) > > >> + if (unlikely(!cnt)) > > >> return; > > >> > > >> - for (i = 0; i < bq->count; i++) { > > >> + for (i = 0; i < cnt; i++) { > > >> struct xdp_frame *xdpf = bq->q[i]; > > >> > > >> prefetch(xdpf); > > >> } > > >> > > >> - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); > > >> + if (bq->xdp_prog) { > > >> + to_send = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); > > >> + if (!to_send) { > > >> + sent = 0; > > >> + goto out; > > >> + } > > >> + drops = cnt - to_send; > > >> + } > > > > > > I might be missing something about how *bq works here. What happens when > > > dev_map_bpf_prog_run returns to_send < cnt? > > > > > > So I read this as it will send [0, to_send] and [to_send, cnt] will be > > > dropped? How do we know the bpf prog would have dropped the set, > > > [to_send+1, cnt]? > > You know that via recalculation of 'drops' value after you returned from > dev_map_bpf_prog_run() which later on is provided onto trace_xdp_devmap_xmit. > > > > > Because dev_map_bpf_prog_run() compacts the array: > > > > + case XDP_PASS: > > + err = xdp_update_frame_from_buff(&xdp, xdpf); > > + if (unlikely(err < 0)) > > + xdp_return_frame_rx_napi(xdpf); > > + else > > + frames[nframes++] = xdpf; > > + break; > > To expand this a little, 'frames' array is reused and 'nframes' above is > the value that is returned and we store it onto 'to_send' variable. > > > > > [...] > > > > >> int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > > >> @@ -489,12 +516,7 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, > > >> { > > >> struct net_device *dev = dst->dev; > > >> > > >> - if (dst->xdp_prog) { > > >> - xdp = dev_map_run_prog(dev, xdp, dst->xdp_prog); > > >> - if (!xdp) > > >> - return 0; > > > > > > So here it looks like dev_map_run_prog will not drop extra > > > packets, but will see every single packet. > > > > > > Are we changing the semantics subtle here? This looks like > > > a problem to me. We should not drop packets in the new case > > > unless bpf program tells us to. > > > > It's not a change in semantics (see above), but I'll grant you that it's > > subtle :) > > dev map xdp prog still sees all of the frames. > > Maybe you were referring to a fact that for XDP_PASS action you might fail > with xdp->xdpf conversion? > > I'm wondering if we could actually do a further optimization and avoid > xdpf/xdp juggling. > > What if xdp_dev_bulk_queue would be storing the xdp_buffs instead of > xdp_frames ? Not possible. Remember that struct xdp_buff is "allocated" on the call stack. Thus, you cannot store a pointer to the xdp_buffs in xdp_dev_bulk_queue. The xdp_frame also avoids allocation, via using memory placed in top of data-frame. Thus, you can store a pointer to the xdp_frame, as it is actually backed by real memory. See[1] slide-11 ("Fundamental structs") > Then you hit bq_xmit_all and if prog is present it doesn't have to do that > dance like we have right now. After that you walk through xdp_buff array > and do the conversion so that xdp_frame array will be passed do > ndo_xdp_xmit. If you want to performance optimize this, I suggest that we detect if we need to call xdp_update_frame_from_buff(&xdp, xdpf) after the 2nd XDP-prog ran. In many case the BPF-prog don't move head/tail/metadata, so that call becomes unnecessary. > I had a bad sleep so maybe I'm talking nonsense over here, will take > another look in the evening though :) :) [1] https://people.netfilter.org/hawk/presentations/KernelRecipes2019/xdp-netstack-concert.pdf -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv17 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-27 15:00 ` Jesper Dangaard Brouer @ 2021-01-27 15:52 ` John Fastabend 2021-01-27 16:05 ` Toke Høiland-Jørgensen 0 siblings, 1 reply; 219+ messages in thread From: John Fastabend @ 2021-01-27 15:52 UTC (permalink / raw) To: Jesper Dangaard Brouer, Maciej Fijalkowski Cc: Toke Høiland-Jørgensen, John Fastabend, Hangbin Liu, bpf, netdev, Jiri Benc, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, brouer Jesper Dangaard Brouer wrote: > On Wed, 27 Jan 2021 13:20:50 +0100 > Maciej Fijalkowski <maciej.fijalkowski@intel.com> wrote: > > > On Wed, Jan 27, 2021 at 10:41:44AM +0100, Toke Høiland-Jørgensen wrote: > > > John Fastabend <john.fastabend@gmail.com> writes: > > > > > > > Hangbin Liu wrote: > > > >> From: Jesper Dangaard Brouer <brouer@redhat.com> > > > >> > > > >> This changes the devmap XDP program support to run the program when the > > > >> bulk queue is flushed instead of before the frame is enqueued. This has > > > >> a couple of benefits: > > > >> > > > >> - It "sorts" the packets by destination devmap entry, and then runs the > > > >> same BPF program on all the packets in sequence. This ensures that we > > > >> keep the XDP program and destination device properties hot in I-cache. > > > >> > > > >> - It makes the multicast implementation simpler because it can just > > > >> enqueue packets using bq_enqueue() without having to deal with the > > > >> devmap program at all. > > > >> > > > >> The drawback is that if the devmap program drops the packet, the enqueue > > > >> step is redundant. However, arguably this is mostly visible in a > > > >> micro-benchmark, and with more mixed traffic the I-cache benefit should > > > >> win out. The performance impact of just this patch is as follows: > > > >> > > > >> The bq_xmit_all's logic is also refactored and error label is removed. > > > >> When bq_xmit_all() is called from bq_enqueue(), another packet will > > > >> always be enqueued immediately after, so clearing dev_rx, xdp_prog and > > > >> flush_node in bq_xmit_all() is redundant. Let's move the clear to > > > >> __dev_flush(), and only check them once in bq_enqueue() since they are > > > >> all modified together. > > > >> > > > >> By using xdp_redirect_map in sample/bpf and send pkts via pktgen cmd: > > > >> ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 > > > >> > > > >> There are about +/- 0.1M deviation for native testing, the performance > > > >> improved for the base-case, but some drop back with xdp devmap prog attached. > > > >> > > > >> Version | Test | Generic | Native | Native + 2nd xdp_prog > > > >> 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M > > > >> 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M > > > >> 5.10 rc6 + patch | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M > > > >> 5.10 rc6 + patch | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M > > > >> > > > > > > > > [...] Acked-by: John Fastabend <john.fastabend@gmail.com> > > > >> static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) > > > >> { > > > >> struct net_device *dev = bq->dev; > > > >> - int sent = 0, drops = 0, err = 0; > > > >> + unsigned int cnt = bq->count; > > > >> + int drops = 0, err = 0; > > > >> + int to_send = cnt; > > > >> + int sent = cnt; > > > >> int i; > > > >> > > > >> - if (unlikely(!bq->count)) > > > >> + if (unlikely(!cnt)) > > > >> return; > > > >> > > > >> - for (i = 0; i < bq->count; i++) { > > > >> + for (i = 0; i < cnt; i++) { > > > >> struct xdp_frame *xdpf = bq->q[i]; > > > >> > > > >> prefetch(xdpf); > > > >> } > > > >> > > > >> - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); > > > >> + if (bq->xdp_prog) { > > > >> + to_send = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); > > > >> + if (!to_send) { > > > >> + sent = 0; > > > >> + goto out; > > > >> + } > > > >> + drops = cnt - to_send; > > > >> + } > > > > > > > > I might be missing something about how *bq works here. What happens when > > > > dev_map_bpf_prog_run returns to_send < cnt? > > > > > > > > So I read this as it will send [0, to_send] and [to_send, cnt] will be > > > > dropped? How do we know the bpf prog would have dropped the set, > > > > [to_send+1, cnt]? > > > > You know that via recalculation of 'drops' value after you returned from > > dev_map_bpf_prog_run() which later on is provided onto trace_xdp_devmap_xmit. > > > > > > > > Because dev_map_bpf_prog_run() compacts the array: > > > > > > + case XDP_PASS: > > > + err = xdp_update_frame_from_buff(&xdp, xdpf); > > > + if (unlikely(err < 0)) > > > + xdp_return_frame_rx_napi(xdpf); > > > + else > > > + frames[nframes++] = xdpf; > > > + break; > > > > To expand this a little, 'frames' array is reused and 'nframes' above is > > the value that is returned and we store it onto 'to_send' variable. > > In the morning with coffee looks good to me. Thanks Toke, Jesper. ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv17 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-27 15:52 ` John Fastabend @ 2021-01-27 16:05 ` Toke Høiland-Jørgensen 0 siblings, 0 replies; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2021-01-27 16:05 UTC (permalink / raw) To: John Fastabend, Jesper Dangaard Brouer, Maciej Fijalkowski Cc: John Fastabend, Hangbin Liu, bpf, netdev, Jiri Benc, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, brouer John Fastabend <john.fastabend@gmail.com> writes: > Jesper Dangaard Brouer wrote: >> On Wed, 27 Jan 2021 13:20:50 +0100 >> Maciej Fijalkowski <maciej.fijalkowski@intel.com> wrote: >> >> > On Wed, Jan 27, 2021 at 10:41:44AM +0100, Toke Høiland-Jørgensen wrote: >> > > John Fastabend <john.fastabend@gmail.com> writes: >> > > >> > > > Hangbin Liu wrote: >> > > >> From: Jesper Dangaard Brouer <brouer@redhat.com> >> > > >> >> > > >> This changes the devmap XDP program support to run the program when the >> > > >> bulk queue is flushed instead of before the frame is enqueued. This has >> > > >> a couple of benefits: >> > > >> >> > > >> - It "sorts" the packets by destination devmap entry, and then runs the >> > > >> same BPF program on all the packets in sequence. This ensures that we >> > > >> keep the XDP program and destination device properties hot in I-cache. >> > > >> >> > > >> - It makes the multicast implementation simpler because it can just >> > > >> enqueue packets using bq_enqueue() without having to deal with the >> > > >> devmap program at all. >> > > >> >> > > >> The drawback is that if the devmap program drops the packet, the enqueue >> > > >> step is redundant. However, arguably this is mostly visible in a >> > > >> micro-benchmark, and with more mixed traffic the I-cache benefit should >> > > >> win out. The performance impact of just this patch is as follows: >> > > >> >> > > >> The bq_xmit_all's logic is also refactored and error label is removed. >> > > >> When bq_xmit_all() is called from bq_enqueue(), another packet will >> > > >> always be enqueued immediately after, so clearing dev_rx, xdp_prog and >> > > >> flush_node in bq_xmit_all() is redundant. Let's move the clear to >> > > >> __dev_flush(), and only check them once in bq_enqueue() since they are >> > > >> all modified together. >> > > >> >> > > >> By using xdp_redirect_map in sample/bpf and send pkts via pktgen cmd: >> > > >> ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 >> > > >> >> > > >> There are about +/- 0.1M deviation for native testing, the performance >> > > >> improved for the base-case, but some drop back with xdp devmap prog attached. >> > > >> >> > > >> Version | Test | Generic | Native | Native + 2nd xdp_prog >> > > >> 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M >> > > >> 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M >> > > >> 5.10 rc6 + patch | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M >> > > >> 5.10 rc6 + patch | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M >> > > >> >> > > > >> > > > [...] > > Acked-by: John Fastabend <john.fastabend@gmail.com> > >> > > >> static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) >> > > >> { >> > > >> struct net_device *dev = bq->dev; >> > > >> - int sent = 0, drops = 0, err = 0; >> > > >> + unsigned int cnt = bq->count; >> > > >> + int drops = 0, err = 0; >> > > >> + int to_send = cnt; >> > > >> + int sent = cnt; >> > > >> int i; >> > > >> >> > > >> - if (unlikely(!bq->count)) >> > > >> + if (unlikely(!cnt)) >> > > >> return; >> > > >> >> > > >> - for (i = 0; i < bq->count; i++) { >> > > >> + for (i = 0; i < cnt; i++) { >> > > >> struct xdp_frame *xdpf = bq->q[i]; >> > > >> >> > > >> prefetch(xdpf); >> > > >> } >> > > >> >> > > >> - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); >> > > >> + if (bq->xdp_prog) { >> > > >> + to_send = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); >> > > >> + if (!to_send) { >> > > >> + sent = 0; >> > > >> + goto out; >> > > >> + } >> > > >> + drops = cnt - to_send; >> > > >> + } >> > > > >> > > > I might be missing something about how *bq works here. What happens when >> > > > dev_map_bpf_prog_run returns to_send < cnt? >> > > > >> > > > So I read this as it will send [0, to_send] and [to_send, cnt] will be >> > > > dropped? How do we know the bpf prog would have dropped the set, >> > > > [to_send+1, cnt]? >> > >> > You know that via recalculation of 'drops' value after you returned from >> > dev_map_bpf_prog_run() which later on is provided onto trace_xdp_devmap_xmit. >> > >> > > >> > > Because dev_map_bpf_prog_run() compacts the array: >> > > >> > > + case XDP_PASS: >> > > + err = xdp_update_frame_from_buff(&xdp, xdpf); >> > > + if (unlikely(err < 0)) >> > > + xdp_return_frame_rx_napi(xdpf); >> > > + else >> > > + frames[nframes++] = xdpf; >> > > + break; >> > >> > To expand this a little, 'frames' array is reused and 'nframes' above is >> > the value that is returned and we store it onto 'to_send' variable. >> > > > In the morning with coffee looks good to me. Thanks Toke, Jesper. Haha, yeah, coffee does tend to help, doesn't it? You're welcome :) -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv17 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-01-25 12:45 ` [PATCHv17 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu 2021-01-27 7:37 ` John Fastabend @ 2021-01-27 15:58 ` Jesper Dangaard Brouer 1 sibling, 0 replies; 219+ messages in thread From: Jesper Dangaard Brouer @ 2021-01-27 15:58 UTC (permalink / raw) To: Hangbin Liu Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, brouer On Mon, 25 Jan 2021 20:45:11 +0800 Hangbin Liu <liuhangbin@gmail.com> wrote: > By using xdp_redirect_map in sample/bpf and send pkts via pktgen cmd: > ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 > > There are about +/- 0.1M deviation for native testing, the performance > improved for the base-case, but some drop back with xdp devmap prog attached. > > Version | Test | Generic | Native | Native + 2nd xdp_prog > 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M > 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M > 5.10 rc6 + patch | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M > 5.10 rc6 + patch | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M I want to highlight the improvement 9.1M -> 9.5M. This is the native (40e->i40e) case where the isn't any "2nd xdp_prog". This means that when we introduced the "2nd xdp_prog", we lost a little performance without noticing (death-by-a-1000-paper-cuts), for the baseline case where this feature is not used/activated. This patch regains that performance for our baseline. That in itself make this patch worth it. -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv17 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL 2021-01-25 12:45 ` [PATCHv17 " Hangbin Liu 2021-01-25 12:45 ` [PATCHv17 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu @ 2021-01-25 12:45 ` Hangbin Liu 2021-01-27 21:50 ` John Fastabend 2021-01-25 12:45 ` [PATCHv17 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (5 subsequent siblings) 7 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2021-01-25 12:45 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu Add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL which could be used when we want to allow NULL pointer for map parameter. The bpf helper need to take care and check if the map is NULL when use this type. Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v13-v17: no update v11-v12: rebase the patch to latest bpf-next v10: remove useless CONST_PTR_TO_MAP_OR_NULL and Copy-paste comment. v9: merge the patch from [1] in to this series. v1-v8: no this patch [1] https://lore.kernel.org/bpf/20200715070001.2048207-1-liuhangbin@gmail.com/ --- include/linux/bpf.h | 1 + kernel/bpf/verifier.c | 10 ++++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 1aac2af12fed..b75207a2484c 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -296,6 +296,7 @@ enum bpf_arg_type { ARG_CONST_ALLOC_SIZE_OR_ZERO, /* number of allocated bytes requested */ ARG_PTR_TO_BTF_ID_SOCK_COMMON, /* pointer to in-kernel sock_common or bpf-mirrored bpf_sock */ ARG_PTR_TO_PERCPU_BTF_ID, /* pointer to in-kernel percpu type */ + ARG_CONST_MAP_PTR_OR_NULL, /* const argument used as pointer to bpf_map or NULL */ __BPF_ARG_TYPE_MAX, }; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d0eae51b31e4..4d147f39bfe7 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -445,7 +445,8 @@ static bool arg_type_may_be_null(enum bpf_arg_type type) type == ARG_PTR_TO_MEM_OR_NULL || type == ARG_PTR_TO_CTX_OR_NULL || type == ARG_PTR_TO_SOCKET_OR_NULL || - type == ARG_PTR_TO_ALLOC_MEM_OR_NULL; + type == ARG_PTR_TO_ALLOC_MEM_OR_NULL || + type == ARG_CONST_MAP_PTR_OR_NULL; } /* Determine whether the function releases some resources allocated by another @@ -4108,6 +4109,7 @@ static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = { [ARG_CONST_SIZE_OR_ZERO] = &scalar_types, [ARG_CONST_ALLOC_SIZE_OR_ZERO] = &scalar_types, [ARG_CONST_MAP_PTR] = &const_map_ptr_types, + [ARG_CONST_MAP_PTR_OR_NULL] = &const_map_ptr_types, [ARG_PTR_TO_CTX] = &context_types, [ARG_PTR_TO_CTX_OR_NULL] = &context_types, [ARG_PTR_TO_SOCK_COMMON] = &sock_types, @@ -4253,9 +4255,9 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, meta->ref_obj_id = reg->ref_obj_id; } - if (arg_type == ARG_CONST_MAP_PTR) { - /* bpf_map_xxx(map_ptr) call: remember that map_ptr */ - meta->map_ptr = reg->map_ptr; + if (arg_type == ARG_CONST_MAP_PTR || + arg_type == ARG_CONST_MAP_PTR_OR_NULL) { + meta->map_ptr = register_is_null(reg) ? NULL : reg->map_ptr; } else if (arg_type == ARG_PTR_TO_MAP_KEY) { /* bpf_map_xxx(..., map_ptr, ..., key) call: * check that [key, key + map->key_size) are within -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* RE: [PATCHv17 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL 2021-01-25 12:45 ` [PATCHv17 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2021-01-27 21:50 ` John Fastabend 0 siblings, 0 replies; 219+ messages in thread From: John Fastabend @ 2021-01-27 21:50 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu Hangbin Liu wrote: > Add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL which could be > used when we want to allow NULL pointer for map parameter. The bpf helper > need to take care and check if the map is NULL when use this type. > > Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > > --- > v13-v17: no update > v11-v12: rebase the patch to latest bpf-next > v10: remove useless CONST_PTR_TO_MAP_OR_NULL and Copy-paste comment. > v9: merge the patch from [1] in to this series. > v1-v8: no this patch > > [1] https://lore.kernel.org/bpf/20200715070001.2048207-1-liuhangbin@gmail.com/ > --- LGTM Acked-by: John Fastabend <john.fastabend@gmail.com> ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv17 bpf-next 3/6] xdp: add a new helper for dev map multicast support 2021-01-25 12:45 ` [PATCHv17 " Hangbin Liu 2021-01-25 12:45 ` [PATCHv17 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu 2021-01-25 12:45 ` [PATCHv17 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2021-01-25 12:45 ` Hangbin Liu 2021-01-25 12:45 ` [PATCHv17 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu ` (4 subsequent siblings) 7 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-25 12:45 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because a physical interface can be part of a logical grouping, such as a bond device. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, I re-implement a new helper bpf_redirect_map_multi() to accept two maps, the forwarding map and exclude map. The forwarding map could be DEVMAP or DEVMAP_HASH, but the exclude map *must* be DEVMAP_HASH to get better performace. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. As both bpf_xdp_redirect_map() and this new helpers are using struct bpf_redirect_info, I add a new ex_map and set tgt_value to NULL in the new helper to make a difference with bpf_xdp_redirect_map(). Also I keep the general data path in net/core/filter.c, the native data path in kernel/bpf/devmap.c so we can use direct calls to get better performace. [0] https://xdp-project.net/#Handling-multicast Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v16-v17: no update v15: a) Update bpf_redirect_map_multi() helper description that ex_map must be keyed by ifindex. b) remove variable last_one in dev_map_enqueue_multi() as it's pointless. c) add a comment about why we don't use READ/WRITE_ONCE() for ex_map. v14: no update, only rebase the code v13: pass xdp_prog through bq_enqueue v12: rebase the code based on Jespoer's devmap xdp_prog patch v11: Fix bpf_redirect_map_multi() helper description typo. Add loop limit for devmap_get_next_obj() and dev_map_redirect_multi(). v10: Update helper bpf_xdp_redirect_map_multi() - No need to check map pointer as we will do the check in verifier. v9: Update helper bpf_xdp_redirect_map_multi() - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. --- include/linux/bpf.h | 20 ++++++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 28 ++++++++ kernel/bpf/devmap.c | 128 +++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++ net/core/filter.c | 124 ++++++++++++++++++++++++++++++-- net/core/xdp.c | 29 ++++++++ tools/include/uapi/linux/bpf.h | 28 ++++++++ 9 files changed, 360 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index b75207a2484c..4b68e975e990 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1427,6 +1427,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx); int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, struct net_device *dev_rx); +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex); +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); bool dev_map_can_have_prog(struct bpf_map *map); @@ -1595,6 +1600,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return 0; } +static inline +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + return false; +} + +static inline +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + return 0; +} + struct sk_buff; static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, diff --git a/include/linux/filter.h b/include/linux/filter.h index 5b3137d7b690..aa9f87ded63c 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -637,6 +637,7 @@ struct bpf_redirect_info { u32 tgt_index; void *tgt_value; struct bpf_map *map; + struct bpf_map *ex_map; u32 kern_flags; struct bpf_nh_params nh; }; diff --git a/include/net/xdp.h b/include/net/xdp.h index c4bfdc9a8b79..fcc0f6f8bd91 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -169,6 +169,7 @@ struct sk_buff *__xdp_build_skb_from_frame(struct xdp_frame *xdpf, struct net_device *dev); struct sk_buff *xdp_build_skb_from_frame(struct xdp_frame *xdpf, struct net_device *dev); +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); static inline void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index c001766adcbc..ef943c024322 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3836,6 +3836,28 @@ union bpf_attr { * Return * A pointer to a struct socket on success or NULL if the file is * not a socket. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. To get better performance, the + * *ex_map* is limited to BPF_MAP_TYPE_DEVMAP_HASH and must be + * keyed by ifindex for the helper to work. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -4001,6 +4023,7 @@ union bpf_attr { FN(ktime_get_coarse_ns), \ FN(ima_inode_hash), \ FN(sock_from_file), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -4177,6 +4200,11 @@ enum { BPF_F_BPRM_SECUREEXEC = (1ULL << 0), }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index bf8b6b5c9cab..217e09533097 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -519,6 +519,134 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return __xdp_enqueue(dev, xdp, dev_rx, dst->xdp_prog); } +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + return dev_map_get_next_key(map, key, next_key); + case BPF_MAP_TYPE_DEVMAP_HASH: + return dev_map_hash_get_next_key(map, key, next_key); + default: + break; + } + + return -ENOENT; +} + +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + if (obj->dev->ifindex == exclude_ifindex) + return true; + + if (!map) + return false; + + return __dev_map_hash_lookup_elem(map, obj->dev->ifindex) != NULL; +} + +static struct bpf_dtab_netdev *devmap_get_next_obj(struct xdp_buff *xdp, struct bpf_map *map, + struct bpf_map *ex_map, u32 *key, + u32 *next_key, int ex_ifindex) +{ + struct bpf_dtab_netdev *obj; + struct net_device *dev; + u32 *tmp_key = key; + u32 index; + int err; + + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + return NULL; + + /* When using dev map hash, we could restart the hashtab traversal + * in case the key has been updated/removed in the mean time. + * So we may end up potentially looping due to traversal restarts + * from first elem. + * + * Let's use map's max_entries to limit the loop number. + */ + for (index = 0; index < map->max_entries; index++) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + obj = __dev_map_lookup_elem(map, *next_key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + obj = __dev_map_hash_lookup_elem(map, *next_key); + break; + default: + break; + } + + if (!obj || dev_in_exclude_map(obj, ex_map, ex_ifindex)) + goto find_next; + + dev = obj->dev; + + if (!dev->netdev_ops->ndo_xdp_xmit) + goto find_next; + + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + if (unlikely(err)) + goto find_next; + + return obj; + +find_next: + tmp_key = next_key; + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + break; + } + + return NULL; +} + +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + struct bpf_dtab_netdev *obj = NULL, *next_obj = NULL; + struct xdp_frame *xdpf, *nxdpf; + int ex_ifindex; + u32 key, next_key; + + ex_ifindex = flags & BPF_F_EXCLUDE_INGRESS ? dev_rx->ifindex : 0; + + /* Find first available obj */ + obj = devmap_get_next_obj(xdp, map, ex_map, NULL, &key, ex_ifindex); + if (!obj) + return 0; + + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) + return -EOVERFLOW; + + for (;;) { + /* Check if we still have one more available obj */ + next_obj = devmap_get_next_obj(xdp, map, ex_map, &key, + &next_key, ex_ifindex); + if (!next_obj) { + bq_enqueue(obj->dev, xdpf, dev_rx, obj->xdp_prog); + return 0; + } + + nxdpf = xdpf_clone(xdpf); + if (unlikely(!nxdpf)) { + xdp_return_frame_rx_napi(xdpf); + return -ENOMEM; + } + + bq_enqueue(obj->dev, nxdpf, dev_rx, obj->xdp_prog); + + /* Deal with next obj */ + obj = next_obj; + key = next_key; + } +} + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 4d147f39bfe7..9e1c21bcb8f4 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4463,6 +4463,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: if (func_id != BPF_FUNC_redirect_map && + func_id != BPF_FUNC_redirect_map_multi && func_id != BPF_FUNC_map_lookup_elem) goto error; break; @@ -4567,6 +4568,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, map->map_type != BPF_MAP_TYPE_XSKMAP) goto error; break; + case BPF_FUNC_redirect_map_multi: + if (map->map_type != BPF_MAP_TYPE_DEVMAP && + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + goto error; + break; case BPF_FUNC_sk_redirect_map: case BPF_FUNC_msg_redirect_map: case BPF_FUNC_sock_map_update: diff --git a/net/core/filter.c b/net/core/filter.c index 9ab94e90d660..03b3088ce8db 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3924,12 +3924,19 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { }; static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) + struct bpf_map *map, struct xdp_buff *xdp, + struct bpf_map *ex_map, u32 flags) { switch (map->map_type) { case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: - return dev_map_enqueue(fwd, xdp, dev_rx); + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) + return dev_map_enqueue(fwd, xdp, dev_rx); + else + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, flags); case BPF_MAP_TYPE_CPUMAP: return cpu_map_enqueue(fwd, xdp, dev_rx); case BPF_MAP_TYPE_XSKMAP: @@ -3986,12 +3993,19 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); struct bpf_map *map = READ_ONCE(ri->map); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err; + /* The READ/WRITE_ONCE() is not needed for ex_map because the field + * is only read from or written to by the CPU owning the per-cpu + * pointer. Whereas the 'map' field is manipulated by remote CPUs + * in bpf_clear_redirect_map(). + */ ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (unlikely(!map)) { @@ -4003,7 +4017,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = dev_xdp_enqueue(fwd, xdp, dev); } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, ri->flags); } if (unlikely(err)) @@ -4017,6 +4031,63 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, } EXPORT_SYMBOL_GPL(xdp_do_redirect); +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, + struct bpf_prog *xdp_prog, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) + +{ + struct bpf_dtab_netdev *dst; + struct sk_buff *nskb; + bool exclude_ingress; + u32 key, next_key, index; + void *fwd; + int err; + + /* Get first key from forward map */ + err = map->ops->map_get_next_key(map, NULL, &key); + if (err) + return err; + + exclude_ingress = !!(flags & BPF_F_EXCLUDE_INGRESS); + + /* When using dev map hash, we could restart the hashtab traversal + * in case the key has been updated/removed in the mean time. + * So we may end up potentially looping due to traversal restarts + * from first elem. + * + * Let's use map's max_entries to limit the loop number. + */ + for (index = 0; index < map->max_entries; index++) { + fwd = __xdp_map_lookup_elem(map, key); + if (fwd) { + dst = (struct bpf_dtab_netdev *)fwd; + if (dev_in_exclude_map(dst, ex_map, + exclude_ingress ? dev->ifindex : 0)) + goto find_next; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -ENOMEM; + + /* Try forword next one no mater the current forward + * succeed or not. + */ + dev_map_generic_redirect(dst, nskb, xdp_prog); + } + +find_next: + err = map->ops->map_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + consume_skb(skb); + return 0; +} + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, @@ -4024,19 +4095,30 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, struct bpf_map *map) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err = 0; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (map->map_type == BPF_MAP_TYPE_DEVMAP || map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { - struct bpf_dtab_netdev *dst = fwd; + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) { + struct bpf_dtab_netdev *dst = fwd; + + err = dev_map_generic_redirect(dst, skb, xdp_prog); + } else { + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, + ex_map, ri->flags); + } - err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { @@ -4150,6 +4232,36 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { .arg3_type = ARG_ANYTHING, }; +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, + struct bpf_map *, ex_map, u64, flags) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + + /* Limit ex_map type to DEVMAP_HASH to get better performance */ + if (unlikely((ex_map && ex_map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) || + flags & ~BPF_F_EXCLUDE_INGRESS)) + return XDP_ABORTED; + + ri->tgt_index = 0; + /* Set the tgt_value to NULL to distinguish with bpf_xdp_redirect_map */ + ri->tgt_value = NULL; + ri->flags = flags; + ri->ex_map = ex_map; + + WRITE_ONCE(ri->map, map); + + return XDP_REDIRECT; +} + +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { + .func = bpf_xdp_redirect_map_multi, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_CONST_MAP_PTR, + .arg2_type = ARG_CONST_MAP_PTR_OR_NULL, + .arg3_type = ARG_ANYTHING, +}; + static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, unsigned long off, unsigned long len) { @@ -7248,6 +7360,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_redirect_proto; case BPF_FUNC_redirect_map: return &bpf_xdp_redirect_map_proto; + case BPF_FUNC_redirect_map_multi: + return &bpf_xdp_redirect_map_multi_proto; case BPF_FUNC_xdp_adjust_tail: return &bpf_xdp_adjust_tail_proto; case BPF_FUNC_fib_lookup: diff --git a/net/core/xdp.c b/net/core/xdp.c index 0d2630a35c3e..b44d05d24a27 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -572,3 +572,32 @@ struct sk_buff *xdp_build_skb_from_frame(struct xdp_frame *xdpf, return __xdp_build_skb_from_frame(xdpf, skb, dev); } EXPORT_SYMBOL_GPL(xdp_build_skb_from_frame); + +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) +{ + unsigned int headroom, totalsize; + struct xdp_frame *nxdpf; + struct page *page; + void *addr; + + headroom = xdpf->headroom + sizeof(*xdpf); + totalsize = headroom + xdpf->len; + + if (unlikely(totalsize > PAGE_SIZE)) + return NULL; + page = dev_alloc_page(); + if (!page) + return NULL; + addr = page_to_virt(page); + + memcpy(addr, xdpf, totalsize); + + nxdpf = addr; + nxdpf->data = addr + headroom; + nxdpf->frame_sz = PAGE_SIZE; + nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0; + nxdpf->mem.id = 0; + + return nxdpf; +} +EXPORT_SYMBOL_GPL(xdpf_clone); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index c001766adcbc..ef943c024322 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3836,6 +3836,28 @@ union bpf_attr { * Return * A pointer to a struct socket on success or NULL if the file is * not a socket. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. To get better performance, the + * *ex_map* is limited to BPF_MAP_TYPE_DEVMAP_HASH and must be + * keyed by ifindex for the helper to work. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -4001,6 +4023,7 @@ union bpf_attr { FN(ktime_get_coarse_ns), \ FN(ima_inode_hash), \ FN(sock_from_file), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -4177,6 +4200,11 @@ enum { BPF_F_BPRM_SECUREEXEC = (1ULL << 0), }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv17 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test 2021-01-25 12:45 ` [PATCHv17 " Hangbin Liu ` (2 preceding siblings ...) 2021-01-25 12:45 ` [PATCHv17 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2021-01-25 12:45 ` Hangbin Liu 2021-01-25 12:45 ` [PATCHv17 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu ` (3 subsequent siblings) 7 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-25 12:45 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu This is a sample for xdp multicast. In the sample we could forward all packets between given interfaces. There is also an option -X that could enable 2nd xdp_prog on egress interface. Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v16-v17: no update v15: use bpf_object__find_program_by_name() instead of bpf_object__find_program_by_title() v13-v14: no update, only rebase the code v12: add devmap xdp_prog on egress support v10-v11: no update v9: use NULL directly for arg2 and redefine the maps with btf format v5: add a null_map as we have strict the arg2 to ARG_CONST_MAP_PTR. Move the testing part to bpf selftest in next patch. v4: no update. v3: add rxcnt map to show the packet transmit speed. v2: no update. --- samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 87 +++++++ samples/bpf/xdp_redirect_map_multi_user.c | 302 ++++++++++++++++++++++ 3 files changed, 392 insertions(+) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index d06144613ca2..539af70b5a98 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map tprogs-y += per_socket_stats_example tprogs-y += xdp_redirect tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_map_multi tprogs-y += xdp_redirect_cpu tprogs-y += xdp_monitor tprogs-y += xdp_rxq_info @@ -99,6 +100,7 @@ test_map_in_map-objs := test_map_in_map_user.o per_socket_stats_example-objs := cookie_uid_helper_example.o xdp_redirect-objs := xdp_redirect_user.o xdp_redirect_map-objs := xdp_redirect_map_user.o +xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o xdp_redirect_cpu-objs := xdp_redirect_cpu_user.o xdp_monitor-objs := xdp_monitor_user.o xdp_rxq_info-objs := xdp_rxq_info_user.o @@ -160,6 +162,7 @@ always-y += tcp_tos_reflect_kern.o always-y += tcp_dumpstats_kern.o always-y += xdp_redirect_kern.o always-y += xdp_redirect_map_kern.o +always-y += xdp_redirect_map_multi_kern.o always-y += xdp_redirect_cpu_kern.o always-y += xdp_monitor_kern.o always-y += xdp_rxq_info_kern.o diff --git a/samples/bpf/xdp_redirect_map_multi_kern.c b/samples/bpf/xdp_redirect_map_multi_kern.c new file mode 100644 index 000000000000..e422340d1251 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_kern.c @@ -0,0 +1,87 @@ +// SPDX-License-Identifier: GPL-2.0 +#define KBUILD_MODNAME "foo" +#include <uapi/linux/bpf.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/ip.h> +#include <linux/ipv6.h> +#include <bpf/bpf_helpers.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 32); +} forward_map_general SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(struct bpf_devmap_val)); + __uint(max_entries, 32); +} forward_map_native SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __type(key, u32); + __type(value, long); + __uint(max_entries, 1); +} rxcnt SEC(".maps"); + +/* map to store egress interfaces mac addresses, set the + * max_entries to 1 and extend it in user sapce prog. + */ +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, u32); + __type(value, __be64); + __uint(max_entries, 1); +} mac_map SEC(".maps"); + +static int xdp_redirect_map(struct xdp_md *ctx, void *forward_map) +{ + long *value; + u32 key = 0; + + /* count packet in global counter */ + value = bpf_map_lookup_elem(&rxcnt, &key); + if (value) + *value += 1; + + return bpf_redirect_map_multi(forward_map, NULL, BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_redirect_general") +int xdp_redirect_map_general(struct xdp_md *ctx) +{ + return xdp_redirect_map(ctx, &forward_map_general); +} + +SEC("xdp_redirect_native") +int xdp_redirect_map_native(struct xdp_md *ctx) +{ + return xdp_redirect_map(ctx, &forward_map_native); +} + +SEC("xdp_devmap/map_prog") +int xdp_devmap_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + u32 key = ctx->egress_ifindex; + struct ethhdr *eth = data; + __be64 *mac; + u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + mac = bpf_map_lookup_elem(&mac_map, &key); + if (mac) + __builtin_memcpy(eth->h_source, mac, ETH_ALEN); + + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/samples/bpf/xdp_redirect_map_multi_user.c b/samples/bpf/xdp_redirect_map_multi_user.c new file mode 100644 index 000000000000..84cdbbed20b7 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_user.c @@ -0,0 +1,302 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> +#include <sys/resource.h> +#include <sys/ioctl.h> +#include <sys/types.h> +#include <sys/socket.h> +#include <netinet/in.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; +static int rxcnt_map_fd; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void poll_stats(int interval) +{ + unsigned int nr_cpus = bpf_num_possible_cpus(); + __u64 values[nr_cpus], prev[nr_cpus]; + + memset(prev, 0, sizeof(prev)); + + while (1) { + __u64 sum = 0; + __u32 key = 0; + int i; + + sleep(interval); + assert(bpf_map_lookup_elem(rxcnt_map_fd, &key, values) == 0); + for (i = 0; i < nr_cpus; i++) + sum += (values[i] - prev[i]); + if (sum) + printf("Forwarding %10llu pkt/s\n", sum / interval); + memcpy(prev, values, sizeof(values)); + } +} + +static int get_mac_addr(unsigned int ifindex, void *mac_addr) +{ + char ifname[IF_NAMESIZE]; + struct ifreq ifr; + int fd, ret = -1; + + fd = socket(AF_INET, SOCK_DGRAM, 0); + if (fd < 0) + return ret; + + if (!if_indextoname(ifindex, ifname)) + goto err_out; + + strcpy(ifr.ifr_name, ifname); + + if (ioctl(fd, SIOCGIFHWADDR, &ifr) != 0) + goto err_out; + + memcpy(mac_addr, ifr.ifr_hwaddr.sa_data, 6 * sizeof(char)); + ret = 0; + +err_out: + close(fd); + return ret; +} + +static int update_mac_map(struct bpf_object *obj) +{ + int i, ret = -1, mac_map_fd; + unsigned char mac_addr[6]; + unsigned int ifindex; + + mac_map_fd = bpf_object__find_map_fd_by_name(obj, "mac_map"); + if (mac_map_fd < 0) { + printf("find mac map fd failed\n"); + return ret; + } + + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + ret = get_mac_addr(ifindex, mac_addr); + if (ret < 0) { + printf("get interface %d mac failed\n", ifindex); + return ret; + } + + ret = bpf_map_update_elem(mac_map_fd, &ifindex, mac_addr, 0); + if (ret) { + perror("bpf_update_elem mac_map_fd"); + return ret; + } + } + + return 0; +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n" + " -X load xdp program on egress\n", + prog); +} + +int main(int argc, char **argv) +{ + int i, ret, opt, forward_map_fd, max_ifindex = 0; + struct bpf_program *ingress_prog, *egress_prog; + int ingress_prog_fd, egress_prog_fd = 0; + struct bpf_devmap_val devmap_val; + bool attach_egress_prog = false; + char ifname[IF_NAMESIZE]; + struct bpf_map *mac_map; + struct bpf_object *obj; + unsigned int ifindex; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNFX")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + case 'X': + attach_egress_prog = true; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) { + xdp_flags |= XDP_FLAGS_DRV_MODE; + } else if (attach_egress_prog) { + printf("Load xdp program on egress with SKB mode not supported yet\n"); + return 1; + } + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + + /* Find the largest index number */ + if (ifaces[i] > max_ifindex) + max_ifindex = ifaces[i]; + + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + + obj = bpf_object__open(filename); + if (libbpf_get_error(obj)) { + printf("ERROR: opening BPF object file failed\n"); + obj = NULL; + goto err_out; + } + + /* Reset the map size to max ifindex + 1 */ + if (attach_egress_prog) { + mac_map = bpf_object__find_map_by_name(obj, "mac_map"); + ret = bpf_map__resize(mac_map, max_ifindex + 1); + if (ret < 0) { + printf("ERROR: reset mac map size failed\n"); + goto err_out; + } + } + + /* load BPF program */ + if (bpf_object__load(obj)) { + printf("ERROR: loading BPF object file failed\n"); + goto err_out; + } + + if (xdp_flags & XDP_FLAGS_SKB_MODE) { + ingress_prog = bpf_object__find_program_by_name(obj, "xdp_redirect_map_general"); + forward_map_fd = bpf_object__find_map_fd_by_name(obj, "forward_map_general"); + } else { + ingress_prog = bpf_object__find_program_by_name(obj, "xdp_redirect_map_native"); + forward_map_fd = bpf_object__find_map_fd_by_name(obj, "forward_map_native"); + } + if (!ingress_prog || forward_map_fd < 0) { + printf("finding ingress_prog/forward_map in obj file failed\n"); + goto err_out; + } + + ingress_prog_fd = bpf_program__fd(ingress_prog); + if (ingress_prog_fd < 0) { + printf("find ingress_prog fd failed\n"); + goto err_out; + } + + rxcnt_map_fd = bpf_object__find_map_fd_by_name(obj, "rxcnt"); + if (rxcnt_map_fd < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + goto err_out; + } + + if (attach_egress_prog) { + /* Update mac_map with all egress interfaces' mac addr */ + if (update_mac_map(obj) < 0) { + printf("Error: update mac map failed"); + goto err_out; + } + + /* Find egress prog fd */ + egress_prog = bpf_object__find_program_by_name(obj, "xdp_devmap_prog"); + if (!egress_prog) { + printf("finding egress_prog in obj file failed\n"); + goto err_out; + } + egress_prog_fd = bpf_program__fd(egress_prog); + if (egress_prog_fd < 0) { + printf("find egress_prog fd failed\n"); + goto err_out; + } + } + + /* Remove attached program when program is interrupted or killed */ + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, ingress_prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + /* Add all the interfaces to forward group and attach + * egress devmap programe if exist + */ + devmap_val.ifindex = ifindex; + devmap_val.bpf_prog.fd = egress_prog_fd; + ret = bpf_map_update_elem(forward_map_fd, &ifindex, &devmap_val, 0); + if (ret) { + perror("bpf_map_update_elem forward_map"); + goto err_out; + } + } + + poll_stats(2); + + return 0; + +err_out: + return 1; +} -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv17 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL 2021-01-25 12:45 ` [PATCHv17 " Hangbin Liu ` (3 preceding siblings ...) 2021-01-25 12:45 ` [PATCHv17 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu @ 2021-01-25 12:45 ` Hangbin Liu 2021-01-27 22:24 ` John Fastabend 2021-01-25 12:45 ` [PATCHv17 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu ` (2 subsequent siblings) 7 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2021-01-25 12:45 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu Use helper bpf_redirect_map() and bpf_redirect_map_multi() to test bpf arg ARG_CONST_MAP_PTR and ARG_CONST_MAP_PTR_OR_NULL. Make sure the map arg could be verified correctly when it is NULL or valid map pointer. Add devmap and devmap_hash in struct bpf_test due to bpf_redirect_{map, map_multi} limit. Test result: ]# ./test_verifier 713 716 #713/p ARG_CONST_MAP_PTR: null pointer OK #714/p ARG_CONST_MAP_PTR: valid map pointer OK #715/p ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map OK #716/p ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map OK Summary: 4 PASSED, 0 SKIPPED, 0 FAILED Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- tools/testing/selftests/bpf/test_verifier.c | 22 +++++- .../testing/selftests/bpf/verifier/map_ptr.c | 70 +++++++++++++++++++ 2 files changed, 91 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c index 59bfa6201d1d..c0e10a6f1911 100644 --- a/tools/testing/selftests/bpf/test_verifier.c +++ b/tools/testing/selftests/bpf/test_verifier.c @@ -50,7 +50,7 @@ #define MAX_INSNS BPF_MAXINSNS #define MAX_TEST_INSNS 1000000 #define MAX_FIXUPS 8 -#define MAX_NR_MAPS 21 +#define MAX_NR_MAPS 23 #define MAX_TEST_RUNS 8 #define POINTER_VALUE 0xcafe4all #define TEST_DATA_LEN 64 @@ -88,6 +88,8 @@ struct bpf_test { int fixup_map_event_output[MAX_FIXUPS]; int fixup_map_reuseport_array[MAX_FIXUPS]; int fixup_map_ringbuf[MAX_FIXUPS]; + int fixup_map_devmap[MAX_FIXUPS]; + int fixup_map_devmap_hash[MAX_FIXUPS]; const char *errstr; const char *errstr_unpriv; uint32_t insn_processed; @@ -714,6 +716,8 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, int *fixup_map_event_output = test->fixup_map_event_output; int *fixup_map_reuseport_array = test->fixup_map_reuseport_array; int *fixup_map_ringbuf = test->fixup_map_ringbuf; + int *fixup_map_devmap = test->fixup_map_devmap; + int *fixup_map_devmap_hash = test->fixup_map_devmap_hash; if (test->fill_helper) { test->fill_insns = calloc(MAX_TEST_INSNS, sizeof(struct bpf_insn)); @@ -899,6 +903,22 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, fixup_map_ringbuf++; } while (*fixup_map_ringbuf); } + if (*fixup_map_devmap) { + map_fds[20] = __create_map(BPF_MAP_TYPE_DEVMAP, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap].imm = map_fds[20]; + fixup_map_devmap++; + } while (*fixup_map_devmap); + } + if (*fixup_map_devmap_hash) { + map_fds[21] = __create_map(BPF_MAP_TYPE_DEVMAP_HASH, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap_hash].imm = map_fds[21]; + fixup_map_devmap_hash++; + } while (*fixup_map_devmap_hash); + } } struct libcap { diff --git a/tools/testing/selftests/bpf/verifier/map_ptr.c b/tools/testing/selftests/bpf/verifier/map_ptr.c index b117bdd3806d..1a532198c9c1 100644 --- a/tools/testing/selftests/bpf/verifier/map_ptr.c +++ b/tools/testing/selftests/bpf/verifier/map_ptr.c @@ -93,3 +93,73 @@ .fixup_map_hash_16b = { 4 }, .result = ACCEPT, }, +{ + "ARG_CONST_MAP_PTR: null pointer", + .insns = { + /* bpf_redirect_map arg1 (map) */ + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .result = REJECT, + .prog_type = BPF_PROG_TYPE_XDP, + .errstr = "R1 type=inv expected=map_ptr", +}, +{ + "ARG_CONST_MAP_PTR: valid map pointer", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg1 (map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_LD_MAP_FD(BPF_REG_2, 1), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .fixup_map_devmap_hash = { 3 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* RE: [PATCHv17 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL 2021-01-25 12:45 ` [PATCHv17 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2021-01-27 22:24 ` John Fastabend 2021-01-28 3:53 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: John Fastabend @ 2021-01-27 22:24 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu Hangbin Liu wrote: > Use helper bpf_redirect_map() and bpf_redirect_map_multi() to test bpf > arg ARG_CONST_MAP_PTR and ARG_CONST_MAP_PTR_OR_NULL. Make sure the > map arg could be verified correctly when it is NULL or valid map > pointer. > > Add devmap and devmap_hash in struct bpf_test due to bpf_redirect_{map, > map_multi} limit. > > Test result: > ]# ./test_verifier 713 716 > #713/p ARG_CONST_MAP_PTR: null pointer OK > #714/p ARG_CONST_MAP_PTR: valid map pointer OK > #715/p ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map OK > #716/p ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map OK > Summary: 4 PASSED, 0 SKIPPED, 0 FAILED > > Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > --- > tools/testing/selftests/bpf/test_verifier.c | 22 +++++- > .../testing/selftests/bpf/verifier/map_ptr.c | 70 +++++++++++++++++++ > 2 files changed, 91 insertions(+), 1 deletion(-) > [...] > +{ > + "ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map", > + .insns = { > + BPF_MOV64_IMM(BPF_REG_1, 0), > + /* bpf_redirect_map_multi arg1 (in_map) */ > + BPF_LD_MAP_FD(BPF_REG_1, 0), > + /* bpf_redirect_map_multi arg2 (ex_map) */ > + BPF_MOV64_IMM(BPF_REG_2, 0), > + /* bpf_redirect_map_multi arg3 (flags) */ > + BPF_MOV64_IMM(BPF_REG_3, 0), > + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), > + BPF_EXIT_INSN(), > + }, > + .fixup_map_devmap = { 1 }, > + .result = ACCEPT, > + .prog_type = BPF_PROG_TYPE_XDP, > + .retval = 4, Do we need one more case where this is map_or_null? In above ex_map will be scalar tnum_const=0 and be exactly a null. This will push verifier here, meta->map_ptr = register_is_null(reg) ? NULL : reg->map_ptr; In the below case it is known to be not null. Is it also interesting to have a case where register_is_null(reg) check fails and reg->map_ptr is set, but may be null. > +}, > +{ > + "ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map", > + .insns = { > + BPF_MOV64_IMM(BPF_REG_1, 0), > + /* bpf_redirect_map_multi arg1 (in_map) */ > + BPF_LD_MAP_FD(BPF_REG_1, 0), > + /* bpf_redirect_map_multi arg2 (ex_map) */ > + BPF_LD_MAP_FD(BPF_REG_2, 1), > + /* bpf_redirect_map_multi arg3 (flags) */ > + BPF_MOV64_IMM(BPF_REG_3, 0), > + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), > + BPF_EXIT_INSN(), > + }, > + .fixup_map_devmap = { 1 }, > + .fixup_map_devmap_hash = { 3 }, > + .result = ACCEPT, > + .prog_type = BPF_PROG_TYPE_XDP, > + .retval = 4, > +}, > -- > 2.26.2 > ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv17 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL 2021-01-27 22:24 ` John Fastabend @ 2021-01-28 3:53 ` Hangbin Liu 2021-01-28 23:14 ` John Fastabend 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2021-01-28 3:53 UTC (permalink / raw) To: John Fastabend Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Maciej Fijalkowski On Wed, Jan 27, 2021 at 02:24:47PM -0800, John Fastabend wrote: > [...] > > > +{ > > + "ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map", > > + .insns = { > > + BPF_MOV64_IMM(BPF_REG_1, 0), > > + /* bpf_redirect_map_multi arg1 (in_map) */ > > + BPF_LD_MAP_FD(BPF_REG_1, 0), > > + /* bpf_redirect_map_multi arg2 (ex_map) */ > > + BPF_MOV64_IMM(BPF_REG_2, 0), > > + /* bpf_redirect_map_multi arg3 (flags) */ > > + BPF_MOV64_IMM(BPF_REG_3, 0), > > + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), > > + BPF_EXIT_INSN(), > > + }, > > + .fixup_map_devmap = { 1 }, > > + .result = ACCEPT, > > + .prog_type = BPF_PROG_TYPE_XDP, > > + .retval = 4, > > Do we need one more case where this is map_or_null? In above > ex_map will be scalar tnum_const=0 and be exactly a null. This > will push verifier here, > > meta->map_ptr = register_is_null(reg) ? NULL : reg->map_ptr; > > In the below case it is known to be not null. > > Is it also interesting to have a case where register_is_null(reg) > check fails and reg->map_ptr is set, but may be null. Hi John, I'm not familiar with the test_verifier syntax. Doesn't BPF_LD_MAP_FD(BPF_REG_1, 0) just assign the register with map NULL? Thanks hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv17 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL 2021-01-28 3:53 ` Hangbin Liu @ 2021-01-28 23:14 ` John Fastabend 0 siblings, 0 replies; 219+ messages in thread From: John Fastabend @ 2021-01-28 23:14 UTC (permalink / raw) To: Hangbin Liu, John Fastabend Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Maciej Fijalkowski Hangbin Liu wrote: > On Wed, Jan 27, 2021 at 02:24:47PM -0800, John Fastabend wrote: > > [...] > > > > > +{ > > > + "ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map", > > > + .insns = { > > > + BPF_MOV64_IMM(BPF_REG_1, 0), > > > + /* bpf_redirect_map_multi arg1 (in_map) */ > > > + BPF_LD_MAP_FD(BPF_REG_1, 0), > > > + /* bpf_redirect_map_multi arg2 (ex_map) */ > > > + BPF_MOV64_IMM(BPF_REG_2, 0), > > > + /* bpf_redirect_map_multi arg3 (flags) */ > > > + BPF_MOV64_IMM(BPF_REG_3, 0), > > > + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), > > > + BPF_EXIT_INSN(), > > > + }, > > > + .fixup_map_devmap = { 1 }, > > > + .result = ACCEPT, > > > + .prog_type = BPF_PROG_TYPE_XDP, > > > + .retval = 4, > > > > Do we need one more case where this is map_or_null? In above > > ex_map will be scalar tnum_const=0 and be exactly a null. This > > will push verifier here, > > > > meta->map_ptr = register_is_null(reg) ? NULL : reg->map_ptr; > > > > In the below case it is known to be not null. > > > > Is it also interesting to have a case where register_is_null(reg) > > check fails and reg->map_ptr is set, but may be null. > > Hi John, > > I'm not familiar with the test_verifier syntax. Doesn't > BPF_LD_MAP_FD(BPF_REG_1, 0) just assign the register with map NULL? On second thought because we are only running the verifier here and not actually calling the helper I guess both paths are in fact covered here. Acked-by: John Fastabend <john.fastabend@gmail.com> ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv17 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test 2021-01-25 12:45 ` [PATCHv17 " Hangbin Liu ` (4 preceding siblings ...) 2021-01-25 12:45 ` [PATCHv17 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2021-01-25 12:45 ` Hangbin Liu 2021-01-28 23:37 ` John Fastabend 2021-02-04 0:14 ` [PATCHv17 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2021-02-04 14:03 ` [PATCHv18 " Hangbin Liu 7 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2021-01-25 12:45 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu Add a bpf selftest for new helper xdp_redirect_map_multi(). In this test we have 3 forward groups and 1 exclude group. The test will redirect each interface's packets to all the interfaces in the forward group, and exclude the interface in exclude map. We will also test both DEVMAP and DEVMAP_HASH with xdp generic and drv. For more test details, you can find it in the test script. Here is the test result. ]# ./test_xdp_redirect_multi.sh Pass: xdpgeneric arp ns1-2 Pass: xdpgeneric arp ns1-3 Pass: xdpgeneric arp ns1-4 Pass: xdpgeneric ping ns1-2 Pass: xdpgeneric ping ns1-3 Pass: xdpgeneric ping ns1-4 Pass: xdpgeneric ping6 ns2-1 Pass: xdpgeneric ping6 ns2-3 Pass: xdpgeneric ping6 ns2-4 Pass: xdpdrv arp ns1-2 Pass: xdpdrv arp ns1-3 Pass: xdpdrv arp ns1-4 Pass: xdpdrv ping ns1-2 Pass: xdpdrv ping ns1-3 Pass: xdpdrv ping ns1-4 Pass: xdpdrv ping6 ns2-1 Pass: xdpdrv ping6 ns2-3 Pass: xdpdrv ping6 ns2-4 Pass: xdpegress mac ns1-2 Pass: xdpegress mac ns1-3 Pass: xdpegress mac ns1-4 Pass: xdpegress ping ns1-2 Pass: xdpegress ping ns1-3 Pass: xdpegress ping ns1-4 Summary: PASS 24, FAIL 0 Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v16-v17: no update v15: use bpf_object__find_program_by_name instead of bpf_object__find_program_by_title v14: no update, only rebase the code v13: remove setrlimit v12: add devmap prog test on egress v9: use NULL directly for arg2 and redefine the maps with btf format --- tools/testing/selftests/bpf/Makefile | 3 +- .../bpf/progs/xdp_redirect_multi_kern.c | 111 ++++++++ .../selftests/bpf/test_xdp_redirect_multi.sh | 208 +++++++++++++++ .../selftests/bpf/xdp_redirect_multi.c | 252 ++++++++++++++++++ 4 files changed, 573 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 63d6288e419c..621dceddb249 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -51,6 +51,7 @@ TEST_FILES = xsk_prereqs.sh \ # Order correspond to 'make run_tests' order TEST_PROGS := test_kmod.sh \ test_xdp_redirect.sh \ + test_xdp_redirect_multi.sh \ test_xdp_meta.sh \ test_xdp_veth.sh \ test_offload.py \ @@ -80,7 +81,7 @@ TEST_PROGS_EXTENDED := with_addr.sh \ TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ test_lirc_mode2_user xdping test_cpp runqslower bench bpf_testmod.ko \ - xdpxceiver + xdpxceiver xdp_redirect_multi TEST_CUSTOM_PROGS = $(OUTPUT)/urandom_read diff --git a/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c new file mode 100644 index 000000000000..dce4df40d9de --- /dev/null +++ b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c @@ -0,0 +1,111 @@ +// SPDX-License-Identifier: GPL-2.0 +#define KBUILD_MODNAME "foo" +#include <string.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ip.h> +#include <linux/ipv6.h> + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_endian.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v4 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v6 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_all SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(struct bpf_devmap_val)); + __uint(max_entries, 128); +} forward_map_egress SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} exclude_map SEC(".maps"); + +/* map to store egress interfaces mac addresses */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, __u32); + __type(value, __be64); + __uint(max_entries, 128); +} mac_map SEC(".maps"); + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + struct ethhdr *eth = data; + __u16 h_proto; + __u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + h_proto = eth->h_proto; + + if (h_proto == bpf_htons(ETH_P_IP)) + return bpf_redirect_map_multi(&forward_map_v4, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else if (h_proto == bpf_htons(ETH_P_IPV6)) + return bpf_redirect_map_multi(&forward_map_v6, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else + return bpf_redirect_map_multi(&forward_map_all, NULL, + BPF_F_EXCLUDE_INGRESS); +} + +/* The following 2 progs are for 2nd devmap prog testing */ +SEC("xdp_redirect_map_ingress") +int xdp_redirect_map_all_prog(struct xdp_md *ctx) +{ + return bpf_redirect_map_multi(&forward_map_egress, NULL, BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_devmap/map_prog") +int xdp_devmap_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + __u32 key = ctx->egress_ifindex; + struct ethhdr *eth = data; + __u64 nh_off; + __be64 *mac; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + mac = bpf_map_lookup_elem(&mac_map, &key); + if (mac) + __builtin_memcpy(eth->h_source, mac, ETH_ALEN); + + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh new file mode 100755 index 000000000000..6503751fdca5 --- /dev/null +++ b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh @@ -0,0 +1,208 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Test topology: +# - - - - - - - - - - - - - - - - - - - - - - - - - +# | veth1 veth2 veth3 veth4 | ... init net +# - -| - - - - - - | - - - - - - | - - - - - - | - - +# --------- --------- --------- --------- +# | veth0 | | veth0 | | veth0 | | veth0 | ... +# --------- --------- --------- --------- +# ns1 ns2 ns3 ns4 +# +# Forward maps: +# Forward map_all has interfaces: veth1, veth2, veth3, veth4, ... (All traffic except IPv4, IPv6) +# Forward map_v4 has interfaces: veth1, veth3, veth4, ... (For IPv4 traffic only) +# Forward map_v6 has interfaces: veth2, veth3, veth4, ... (For IPv6 traffic only) +# Forward map_egress has all interfaces and redirect all pkts +# Exclude Groups: +# Exclude map: veth3 (assume ns3 is in black list) +# Map type: +# map_v4 use DEVMAP, others use DEVMAP_HASH +# +# Test modules: +# XDP modes: generic, native, native + egress_prog +# +# Test cases: +# ARP(we didn't block ARP for ns3): +# ns1 -> gw: ns2, ns3, ns4 should receive the arp request +# IPv4: +# ns1 -> ns2 (block), ns1 -> ns3 (block), ns1 -> ns4 (pass) +# IPv6 +# ns2 -> ns1 (block), ns2 -> ns3 (block), ns2 -> ns4 (pass) +# egress_prog: +# all ping test should pass, the src mac should be egress interface's mac +# + + +# netns numbers +NUM=4 +IFACES="" +DRV_MODE="xdpgeneric xdpdrv xdpegress" +PASS=0 +FAIL=0 + +test_pass() +{ + echo "Pass: $@" + PASS=$((PASS + 1)) +} + +test_fail() +{ + echo "fail: $@" + FAIL=$((FAIL + 1)) +} + +clean_up() +{ + for i in $(seq $NUM); do + ip link del veth$i 2> /dev/null + ip netns del ns$i 2> /dev/null + done +} + +# Kselftest framework requirement - SKIP code is 4. +check_env() +{ + ip link set dev lo xdpgeneric off &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without the ip xdpgeneric support" + exit 4 + fi + + which tcpdump &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without tcpdump" + exit 4 + fi +} + +setup_ns() +{ + local mode=$1 + IFACES="" + + if [ "$mode" = "xdpegress" ]; then + mode="xdpdrv" + fi + + for i in $(seq $NUM); do + ip netns add ns$i + ip link add veth$i type veth peer name veth0 netns ns$i + ip link set veth$i up + ip -n ns$i link set veth0 up + + ip -n ns$i addr add 192.0.2.$i/24 dev veth0 + ip -n ns$i addr add 2001:db8::$i/64 dev veth0 + ip -n ns$i link set veth0 $mode obj \ + xdp_dummy.o sec xdp_dummy &> /dev/null || \ + { test_fail "Unable to load dummy xdp" && exit 1; } + IFACES="$IFACES veth$i" + veth_mac[$i]=$(ip link show veth$i | awk '/link\/ether/ {print $2}') + done +} + +do_egress_tests() +{ + local mode=$1 + + # mac test + ip netns exec ns2 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + + # mac check + grep -q "${veth_mac[2]} > ff:ff:ff:ff:ff:ff" mac_ns1-2_${mode}.log && \ + test_pass "$mode mac ns1-2" || test_fail "$mode mac ns1-2" + grep -q "${veth_mac[3]} > ff:ff:ff:ff:ff:ff" mac_ns1-3_${mode}.log && \ + test_pass "$mode mac ns1-3" || test_fail "$mode mac ns1-3" + grep -q "${veth_mac[4]} > ff:ff:ff:ff:ff:ff" mac_ns1-4_${mode}.log && \ + test_pass "$mode mac ns1-4" || test_fail "$mode mac ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-2" || test_fail "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-3" || test_fail "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" +} + +do_ping_tests() +{ + local mode=$1 + + # arp test + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -i veth0 -nn -l -e &> arp_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-2_${mode}.log && \ + test_pass "$mode arp ns1-2" || test_fail "$mode arp ns1-2" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-3_${mode}.log && \ + test_pass "$mode arp ns1-3" || test_fail "$mode arp ns1-3" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-4_${mode}.log && \ + test_pass "$mode arp ns1-4" || test_fail "$mode arp ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-2" || test_pass "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-3" || test_pass "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" + + # ping6 test + ip netns exec ns2 ping6 2001:db8::1 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-1" || test_pass "$mode ping6 ns2-1" + ip netns exec ns2 ping6 2001:db8::3 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-3" || test_pass "$mode ping6 ns2-3" + ip netns exec ns2 ping6 2001:db8::4 -c 4 &> /dev/null && \ + test_pass "$mode ping6 ns2-4" || test_fail "$mode ping6 ns2-4" +} + +do_tests() +{ + local mode=$1 + local drv_p + + case ${mode} in + xdpdrv) drv_p="-N";; + xdpegress) drv_p="-X";; + xdpgeneric) drv_p="-S";; + esac + + ./xdp_redirect_multi $drv_p $IFACES &> xdp_redirect_${mode}.log & + xdp_pid=$! + sleep 10 + + if [ "$mode" = "xdpegress" ]; then + do_egress_tests $mode + else + do_ping_tests $mode + fi + + kill $xdp_pid +} + +trap clean_up 0 2 3 6 9 + +check_env +rm -f xdp_redirect_*.log arp_ns*.log mac_ns*.log + +for mode in ${DRV_MODE}; do + setup_ns $mode + do_tests $mode + sleep 10 + clean_up + sleep 5 +done + +echo "Summary: PASS $PASS, FAIL $FAIL" +[ $FAIL -eq 0 ] && exit 0 || exit 1 diff --git a/tools/testing/selftests/bpf/xdp_redirect_multi.c b/tools/testing/selftests/bpf/xdp_redirect_multi.c new file mode 100644 index 000000000000..b43cd3c9eefd --- /dev/null +++ b/tools/testing/selftests/bpf/xdp_redirect_multi.c @@ -0,0 +1,252 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> +#include <sys/resource.h> +#include <sys/ioctl.h> +#include <sys/types.h> +#include <sys/socket.h> +#include <netinet/in.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static int get_mac_addr(unsigned int ifindex, void *mac_addr) +{ + char ifname[IF_NAMESIZE]; + struct ifreq ifr; + int fd, ret = -1; + + fd = socket(AF_INET, SOCK_DGRAM, 0); + if (fd < 0) + return ret; + + if (!if_indextoname(ifindex, ifname)) + goto err_out; + + strcpy(ifr.ifr_name, ifname); + + if (ioctl(fd, SIOCGIFHWADDR, &ifr) != 0) + goto err_out; + + memcpy(mac_addr, ifr.ifr_hwaddr.sa_data, 6 * sizeof(char)); + ret = 0; + +err_out: + close(fd); + return ret; +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n" + " -X load xdp program on egress\n", + prog); +} + +int main(int argc, char **argv) +{ + int prog_fd, group_all, group_v4, group_v6, exclude, mac_map; + struct bpf_program *ingress_prog, *egress_prog; + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_UNSPEC, + }; + int i, ret, opt, egress_prog_fd = 0; + struct bpf_devmap_val devmap_val; + bool attach_egress_prog = false; + unsigned char mac_addr[6]; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + unsigned int ifindex; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNFX")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + case 'X': + attach_egress_prog = true; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) { + xdp_flags |= XDP_FLAGS_DRV_MODE; + } else if (attach_egress_prog) { + printf("Load xdp program on egress with SKB mode not supported yet\n"); + goto err_out; + } + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + goto err_out; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + goto err_out; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + goto err_out; + + if (attach_egress_prog) + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_egress"); + else + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_all"); + group_v4 = bpf_object__find_map_fd_by_name(obj, "forward_map_v4"); + group_v6 = bpf_object__find_map_fd_by_name(obj, "forward_map_v6"); + exclude = bpf_object__find_map_fd_by_name(obj, "exclude_map"); + mac_map = bpf_object__find_map_fd_by_name(obj, "mac_map"); + + if (group_all < 0 || group_v4 < 0 || group_v6 < 0 || exclude < 0 || + mac_map < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + goto err_out; + } + + if (attach_egress_prog) { + /* Find ingress/egress prog for 2nd xdp prog */ + ingress_prog = bpf_object__find_program_by_name(obj, "xdp_redirect_map_all_prog"); + egress_prog = bpf_object__find_program_by_name(obj, "xdp_devmap_prog"); + if (!ingress_prog || !egress_prog) { + printf("finding ingress/egress_prog in obj file failed\n"); + goto err_out; + } + prog_fd = bpf_program__fd(ingress_prog); + egress_prog_fd = bpf_program__fd(egress_prog); + if (prog_fd < 0 || egress_prog_fd < 0) { + printf("find egress_prog fd failed\n"); + goto err_out; + } + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + if (attach_egress_prog) { + ret = get_mac_addr(ifindex, mac_addr); + if (ret < 0) { + printf("get interface %d mac failed\n", ifindex); + goto err_out; + } + ret = bpf_map_update_elem(mac_map, &ifindex, mac_addr, 0); + if (ret) { + perror("bpf_update_elem mac_map failed\n"); + goto err_out; + } + } + + /* Add all the interfaces to group all */ + devmap_val.ifindex = ifindex; + devmap_val.bpf_prog.fd = egress_prog_fd; + ret = bpf_map_update_elem(group_all, &ifindex, &devmap_val, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* For testing: remove the 1st interfaces from group v6 */ + if (i != 0) { + ret = bpf_map_update_elem(group_v6, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: remove the 2nd interfaces from group v4 */ + if (i != 1) { + ret = bpf_map_update_elem(group_v4, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: add the 3rd interfaces to exclude map */ + if (i == 2) { + ret = bpf_map_update_elem(exclude, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + } + + /* sleep some time for testing */ + sleep(999); + + return 0; + +err_out: + return 1; +} -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* RE: [PATCHv17 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test 2021-01-25 12:45 ` [PATCHv17 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu @ 2021-01-28 23:37 ` John Fastabend 2021-01-29 1:46 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: John Fastabend @ 2021-01-28 23:37 UTC (permalink / raw) To: Hangbin Liu, bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu Hangbin Liu wrote: > Add a bpf selftest for new helper xdp_redirect_map_multi(). In this > test we have 3 forward groups and 1 exclude group. The test will > redirect each interface's packets to all the interfaces in the forward > group, and exclude the interface in exclude map. We will also test both > DEVMAP and DEVMAP_HASH with xdp generic and drv. > > For more test details, you can find it in the test script. Here is > the test result. > ]# ./test_xdp_redirect_multi.sh > Pass: xdpgeneric arp ns1-2 > Pass: xdpgeneric arp ns1-3 > Pass: xdpgeneric arp ns1-4 > Pass: xdpgeneric ping ns1-2 > Pass: xdpgeneric ping ns1-3 > Pass: xdpgeneric ping ns1-4 > Pass: xdpgeneric ping6 ns2-1 > Pass: xdpgeneric ping6 ns2-3 > Pass: xdpgeneric ping6 ns2-4 > Pass: xdpdrv arp ns1-2 > Pass: xdpdrv arp ns1-3 > Pass: xdpdrv arp ns1-4 > Pass: xdpdrv ping ns1-2 > Pass: xdpdrv ping ns1-3 > Pass: xdpdrv ping ns1-4 > Pass: xdpdrv ping6 ns2-1 > Pass: xdpdrv ping6 ns2-3 > Pass: xdpdrv ping6 ns2-4 > Pass: xdpegress mac ns1-2 > Pass: xdpegress mac ns1-3 > Pass: xdpegress mac ns1-4 > Pass: xdpegress ping ns1-2 > Pass: xdpegress ping ns1-3 > Pass: xdpegress ping ns1-4 > Summary: PASS 24, FAIL 0 > > Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> > > --- > v16-v17: no update > v15: use bpf_object__find_program_by_name instead of > bpf_object__find_program_by_title > v14: no update, only rebase the code > v13: remove setrlimit > v12: add devmap prog test on egress > v9: use NULL directly for arg2 and redefine the maps with btf format > --- [...] > +SEC("xdp_devmap/map_prog") > +int xdp_devmap_prog(struct xdp_md *ctx) > +{ > + void *data_end = (void *)(long)ctx->data_end; > + void *data = (void *)(long)ctx->data; > + __u32 key = ctx->egress_ifindex; > + struct ethhdr *eth = data; > + __u64 nh_off; > + __be64 *mac; > + > + nh_off = sizeof(*eth); > + if (data + nh_off > data_end) > + return XDP_DROP; > + > + mac = bpf_map_lookup_elem(&mac_map, &key); > + if (mac) > + __builtin_memcpy(eth->h_source, mac, ETH_ALEN); > + > + return XDP_PASS; > +} Might be nice to also have a test for XDP_DROP. I guess the above 'data + nh_off > data' case should not happen. Otherwise, its not the most elegant, but testing XDP at the moment doesn't fit into the normal test framework very well either. Acked-by: John Fastabend <john.fastabend@gmail.com> ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv17 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test 2021-01-28 23:37 ` John Fastabend @ 2021-01-29 1:46 ` Hangbin Liu 0 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-01-29 1:46 UTC (permalink / raw) To: John Fastabend Cc: bpf, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, Maciej Fijalkowski Hi John, On Thu, Jan 28, 2021 at 03:37:09PM -0800, John Fastabend wrote: > Otherwise, its not the most elegant, but testing XDP at the moment > doesn't fit into the normal test framework very well either. Thanks a lot for your help in reviewing the patches. I will add updating XDP test in my todo list. Thanks hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv17 bpf-next 0/6] xdp: add a new helper for dev map multicast support 2021-01-25 12:45 ` [PATCHv17 " Hangbin Liu ` (5 preceding siblings ...) 2021-01-25 12:45 ` [PATCHv17 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu @ 2021-02-04 0:14 ` Hangbin Liu 2021-02-04 2:53 ` John Fastabend 2021-02-04 14:03 ` [PATCHv18 " Hangbin Liu 7 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2021-02-04 0:14 UTC (permalink / raw) To: Daniel Borkmann, Alexei Starovoitov Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, bpf, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, John Fastabend, Maciej Fijalkowski Hi Daniel, Alexei, It has been one week after Maciej, Toke, John's review/ack. What should I do to make a progress for this patch set? Thanks Hangbin On Mon, Jan 25, 2021 at 08:45:10PM +0800, Hangbin Liu wrote: > This patch is for xdp multicast support. which has been discussed before[0], > The goal is to be able to implement an OVS-like data plane in XDP, i.e., > a software switch that can forward XDP frames to multiple ports. > > To achieve this, an application needs to specify a group of interfaces > to forward a packet to. It is also common to want to exclude one or more > physical interfaces from the forwarding operation - e.g., to forward a > packet to all interfaces in the multicast group except the interface it > arrived on. While this could be done simply by adding more groups, this > quickly leads to a combinatorial explosion in the number of groups an > application has to maintain. > > To avoid the combinatorial explosion, we propose to include the ability > to specify an "exclude group" as part of the forwarding operation. This > needs to be a group (instead of just a single port index), because there > may have multi interfaces you want to exclude. > > Thus, the logical forwarding operation becomes a "set difference" > operation, i.e. "forward to all ports in group A that are not also in > group B". This series implements such an operation using device maps to > represent the groups. This means that the XDP program specifies two > device maps, one containing the list of netdevs to redirect to, and the > other containing the exclude list. > > To achieve this, I re-implement a new helper bpf_redirect_map_multi() > to accept two maps, the forwarding map and exclude map. If user > don't want to use exclude map and just want simply stop redirecting back > to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. > > The 1st patch is Jesper's run devmap xdp_prog later in bulking step. > The 2st patch add a new bpf arg to allow NULL map pointer. > The 3rd patch add the new bpf_redirect_map_multi() helper. > The 4-6 patches are for usage sample and testing purpose. > > I did same perf tests with the following topo: > > --------------------- --------------------- > | Host A (i40e 10G) | ---------- | eno1(i40e 10G) | > --------------------- | | > | Host B | > --------------------- | | > | Host C (i40e 10G) | ---------- | eno2(i40e 10G) | > --------------------- vlan2 | -------- | > | veth1 -- | veth0| | > | -------- | > --------------------| > On Host A: > # pktgen/pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -s 64 > > On Host B(Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, 128G Memory): > Use xdp_redirect_map and xdp_redirect_map_multi in samples/bpf for testing. > The veth0 in netns load dummy drop program. The forward_map max_entries in > xdp_redirect_map_multi is modify to 4. > > Here is the perf result with 5.10 rc6: > > The are about +/- 0.1M deviation for native testing > Version | Test | Generic | Native | Native + 2nd > 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M > 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M > 5.10 rc6 + patch1 | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M > 5.10 rc6 + patch1 | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M > 5.10 rc6 + patch1-6 | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M > 5.10 rc6 + patch1-6 | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M > 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->i40e | 1.7M | 7.8M | 6.4M > 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->veth | 1.4M | 9.3M | 7.5M > 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->i40e+veth | 1.0M | 3.2M | 2.7M > > Last but not least, thanks a lot to Toke, Jesper, Jiri and Eelco for > suggestions and help on implementation. > > [0] https://xdp-project.net/#Handling-multicast > > v17: > For patch 01: > a) rename to_sent to to_send. > b) clear bq dev_rx, xdp_prog and flush_node in __dev_flush(). > > v16: > refactor bq_xmit_all logic and remove error label for patch 01 > > v15: > Update bq_xmit_all() logic for patch 01. > Add some comments and remove useless variable for patch 03. > Use bpf_object__find_program_by_title() for patch 04 and 06. > > v14: > No code update, just rebase the code on latest bpf-next > > v13: > Pass in xdp_prog through __xdp_enqueue() for patch 01. Update related > code in patch 03. > > v12: > Add Jesper's xdp_prog patch, rebase my works on this and latest bpf-next > Add 2nd xdp_prog test on the sample and selftests. > > v11: > Fix bpf_redirect_map_multi() helper description typo. > Add loop limit for devmap_get_next_obj() and dev_map_redirect_multi(). > > v10: > Rebase the code to latest bpf-next. > Update helper bpf_xdp_redirect_map_multi() > - No need to check map pointer as we will do the check in verifier. > > v9: > Update helper bpf_xdp_redirect_map_multi() > - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 > > v8: > a) Update function dev_in_exclude_map(): > - remove duplicate ex_map map_type check in > - lookup the element in dev map by obj dev index directly instead > of looping all the map > > v7: > a) Fix helper flag check > b) Limit the *ex_map* to use DEVMAP_HASH only and update function > dev_in_exclude_map() to get better performance. > > v6: converted helper return types from int to long > > v5: > a) Check devmap_get_next_key() return value. > b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. > c) In function dev_map_enqueue_multi(), consume xdpf for the last > obj instead of the first on. > d) Update helper description and code comments to explain that we > use NULL target value to distinguish multicast and unicast > forwarding. > e) Update memory model, memory id and frame_sz in xdpf_clone(). > f) Split the tests from sample and add a bpf kernel selftest patch. > > v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo > > v3: Based on Toke's suggestion, do the following update > a) Update bpf_redirect_map_multi() description in bpf.h. > b) Fix exclude_ifindex checking order in dev_in_exclude_map(). > c) Fix one more xdpf clone in dev_map_enqueue_multi(). > d) Go find next one in dev_map_enqueue_multi() if the interface is not > able to forward instead of abort the whole loop. > e) Remove READ_ONCE/WRITE_ONCE for ex_map. > > v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept > include/exclude maps directly. > > Hangbin Liu (5): > bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL > xdp: add a new helper for dev map multicast support > sample/bpf: add xdp_redirect_map_multicast test > selftests/bpf: Add verifier tests for bpf arg > ARG_CONST_MAP_PTR_OR_NULL > selftests/bpf: add xdp_redirect_multi test > > Jesper Dangaard Brouer (1): > bpf: run devmap xdp_prog on flush instead of bulk enqueue > > include/linux/bpf.h | 21 ++ > include/linux/filter.h | 1 + > include/net/xdp.h | 1 + > include/uapi/linux/bpf.h | 28 ++ > kernel/bpf/devmap.c | 262 +++++++++++---- > kernel/bpf/verifier.c | 16 +- > net/core/filter.c | 124 ++++++- > net/core/xdp.c | 29 ++ > samples/bpf/Makefile | 3 + > samples/bpf/xdp_redirect_map_multi_kern.c | 87 +++++ > samples/bpf/xdp_redirect_map_multi_user.c | 302 ++++++++++++++++++ > tools/include/uapi/linux/bpf.h | 28 ++ > tools/testing/selftests/bpf/Makefile | 3 +- > .../bpf/progs/xdp_redirect_multi_kern.c | 111 +++++++ > tools/testing/selftests/bpf/test_verifier.c | 22 +- > .../selftests/bpf/test_xdp_redirect_multi.sh | 208 ++++++++++++ > .../testing/selftests/bpf/verifier/map_ptr.c | 70 ++++ > .../selftests/bpf/xdp_redirect_multi.c | 252 +++++++++++++++ > 18 files changed, 1501 insertions(+), 67 deletions(-) > create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c > create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c > create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c > create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh > create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c > > -- > 2.26.2 > ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv17 bpf-next 0/6] xdp: add a new helper for dev map multicast support 2021-02-04 0:14 ` [PATCHv17 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2021-02-04 2:53 ` John Fastabend 2021-02-04 3:12 ` Hangbin Liu 0 siblings, 1 reply; 219+ messages in thread From: John Fastabend @ 2021-02-04 2:53 UTC (permalink / raw) To: Hangbin Liu, Daniel Borkmann, Alexei Starovoitov Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, bpf, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, John Fastabend, Maciej Fijalkowski Hangbin Liu wrote: > Hi Daniel, Alexei, > > It has been one week after Maciej, Toke, John's review/ack. What should > I do to make a progress for this patch set? > Patchwork is usually the first place to check: https://patchwork.kernel.org/project/netdevbpf/list/?series=421095&state=* Looks like it was marked changed requested. After this its unlikely anyone will follow up on it, rightly so given the assumption another revision is coming. In this case my guess is it was moved into changes requested because I asked for a change, but then after some discussion you convinced me the change was not in fact needed. Alexei, Daniel can probably tell you if its easier to just send a v18 or pull in the v17 assuming any final reviews don't kick anything else up. Thanks John ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv17 bpf-next 0/6] xdp: add a new helper for dev map multicast support 2021-02-04 2:53 ` John Fastabend @ 2021-02-04 3:12 ` Hangbin Liu 2021-02-04 11:00 ` Toke Høiland-Jørgensen 0 siblings, 1 reply; 219+ messages in thread From: Hangbin Liu @ 2021-02-04 3:12 UTC (permalink / raw) To: John Fastabend Cc: Daniel Borkmann, Alexei Starovoitov, netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, bpf, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Maciej Fijalkowski On Wed, Feb 03, 2021 at 06:53:20PM -0800, John Fastabend wrote: > Hangbin Liu wrote: > > Hi Daniel, Alexei, > > > > It has been one week after Maciej, Toke, John's review/ack. What should > > I do to make a progress for this patch set? > > > > Patchwork is usually the first place to check: Thanks John for the link. > > https://patchwork.kernel.org/project/netdevbpf/list/?series=421095&state=* Before I sent the email I only checked link https://patchwork.kernel.org/project/netdevbpf/list/ but can't find my patch. How do you get the series number? > > Looks like it was marked changed requested. After this its unlikely > anyone will follow up on it, rightly so given the assumption another > revision is coming. > > In this case my guess is it was moved into changes requested because > I asked for a change, but then after some discussion you convinced me > the change was not in fact needed. > > Alexei, Daniel can probably tell you if its easier to just send a v18 > or pull in the v17 assuming any final reviews don't kick anything > else up. OK, I will wait for Alexei, Daniel and see if I need to do a rebase. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv17 bpf-next 0/6] xdp: add a new helper for dev map multicast support 2021-02-04 3:12 ` Hangbin Liu @ 2021-02-04 11:00 ` Toke Høiland-Jørgensen 2021-02-04 12:09 ` Maciej Fijalkowski 2021-02-04 17:03 ` Jakub Kicinski 0 siblings, 2 replies; 219+ messages in thread From: Toke Høiland-Jørgensen @ 2021-02-04 11:00 UTC (permalink / raw) To: Hangbin Liu, John Fastabend Cc: Daniel Borkmann, Alexei Starovoitov, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, bpf, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Maciej Fijalkowski Hangbin Liu <liuhangbin@gmail.com> writes: > On Wed, Feb 03, 2021 at 06:53:20PM -0800, John Fastabend wrote: >> Hangbin Liu wrote: >> > Hi Daniel, Alexei, >> > >> > It has been one week after Maciej, Toke, John's review/ack. What should >> > I do to make a progress for this patch set? >> > >> >> Patchwork is usually the first place to check: > > Thanks John for the link. >> >> https://patchwork.kernel.org/project/netdevbpf/list/?series=421095&state=* > > Before I sent the email I only checked link > https://patchwork.kernel.org/project/netdevbpf/list/ but can't find my patch. > > How do you get the series number? If you click the "show patches with" link at the top you can twiddle the filtering; state = any + your own name as submitter usually finds things, I've found. >> Looks like it was marked changed requested. After this its unlikely >> anyone will follow up on it, rightly so given the assumption another >> revision is coming. >> >> In this case my guess is it was moved into changes requested because >> I asked for a change, but then after some discussion you convinced me >> the change was not in fact needed. >> >> Alexei, Daniel can probably tell you if its easier to just send a v18 >> or pull in the v17 assuming any final reviews don't kick anything >> else up. > > OK, I will wait for Alexei, Daniel and see if I need to do a rebase. I think I would just resubmit with a rebase + a note in the changelog that we concluded no further change was needed :) -Toke ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv17 bpf-next 0/6] xdp: add a new helper for dev map multicast support 2021-02-04 11:00 ` Toke Høiland-Jørgensen @ 2021-02-04 12:09 ` Maciej Fijalkowski 2021-02-04 13:33 ` Hangbin Liu 2021-02-04 17:03 ` Jakub Kicinski 1 sibling, 1 reply; 219+ messages in thread From: Maciej Fijalkowski @ 2021-02-04 12:09 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: Hangbin Liu, John Fastabend, Daniel Borkmann, Alexei Starovoitov, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, bpf, Lorenzo Bianconi, David Ahern, Andrii Nakryiko On Thu, Feb 04, 2021 at 12:00:29PM +0100, Toke Høiland-Jørgensen wrote: > Hangbin Liu <liuhangbin@gmail.com> writes: > > > On Wed, Feb 03, 2021 at 06:53:20PM -0800, John Fastabend wrote: > >> Hangbin Liu wrote: > >> > Hi Daniel, Alexei, > >> > > >> > It has been one week after Maciej, Toke, John's review/ack. What should > >> > I do to make a progress for this patch set? > >> > > >> > >> Patchwork is usually the first place to check: > > > > Thanks John for the link. > >> > >> https://patchwork.kernel.org/project/netdevbpf/list/?series=421095&state=* > > > > Before I sent the email I only checked link > > https://patchwork.kernel.org/project/netdevbpf/list/ but can't find my patch. > > > > How do you get the series number? > > If you click the "show patches with" link at the top you can twiddle the > filtering; state = any + your own name as submitter usually finds > things, I've found. > > >> Looks like it was marked changed requested. After this its unlikely > >> anyone will follow up on it, rightly so given the assumption another > >> revision is coming. > >> > >> In this case my guess is it was moved into changes requested because > >> I asked for a change, but then after some discussion you convinced me > >> the change was not in fact needed. > >> > >> Alexei, Daniel can probably tell you if its easier to just send a v18 > >> or pull in the v17 assuming any final reviews don't kick anything > >> else up. > > > > OK, I will wait for Alexei, Daniel and see if I need to do a rebase. > > I think I would just resubmit with a rebase + a note in the changelog > that we concluded no further change was needed :) I only asked for imperative mood in commit messages, but not sure if anyone cares ;) > > -Toke > ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv17 bpf-next 0/6] xdp: add a new helper for dev map multicast support 2021-02-04 12:09 ` Maciej Fijalkowski @ 2021-02-04 13:33 ` Hangbin Liu 0 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-02-04 13:33 UTC (permalink / raw) To: Maciej Fijalkowski Cc: Toke Høiland-Jørgensen, John Fastabend, Daniel Borkmann, Alexei Starovoitov, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, bpf, Lorenzo Bianconi, David Ahern, Andrii Nakryiko On Thu, Feb 04, 2021 at 01:09:22PM +0100, Maciej Fijalkowski wrote: > > I think I would just resubmit with a rebase + a note in the changelog > > that we concluded no further change was needed :) > > I only asked for imperative mood in commit messages, but not sure if > anyone cares ;) I will try, but could not guarantee I can fix all the sentences. Thanks hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv17 bpf-next 0/6] xdp: add a new helper for dev map multicast support 2021-02-04 11:00 ` Toke Høiland-Jørgensen 2021-02-04 12:09 ` Maciej Fijalkowski @ 2021-02-04 17:03 ` Jakub Kicinski 2021-02-05 3:07 ` Hangbin Liu 1 sibling, 1 reply; 219+ messages in thread From: Jakub Kicinski @ 2021-02-04 17:03 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: Hangbin Liu, John Fastabend, Daniel Borkmann, Alexei Starovoitov, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, bpf, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Maciej Fijalkowski On Thu, 04 Feb 2021 12:00:29 +0100 Toke Høiland-Jørgensen wrote: > >> Patchwork is usually the first place to check: > > > > Thanks John for the link. > >> > >> https://patchwork.kernel.org/project/netdevbpf/list/?series=421095&state=* > > > > Before I sent the email I only checked link > > https://patchwork.kernel.org/project/netdevbpf/list/ but can't find my patch. > > > > How do you get the series number? > > If you click the "show patches with" link at the top you can twiddle the > filtering; state = any + your own name as submitter usually finds > things, I've found. New patchwork can actually find messages by Message-ID header. Just slap message ID of one of the patches at the end of: https://patchwork.kernel.org/project/netdevbpf/patch/ And there is a link to entire series there. Since I'm speaking, Hangbin I'd discourage posting new version as a reply to previous posting. It brings out this massive 100+ message thread and breaks natural ordering of patches to review. ^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [PATCHv17 bpf-next 0/6] xdp: add a new helper for dev map multicast support 2021-02-04 17:03 ` Jakub Kicinski @ 2021-02-05 3:07 ` Hangbin Liu 0 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-02-05 3:07 UTC (permalink / raw) To: Jakub Kicinski Cc: Toke Høiland-Jørgensen, John Fastabend, Daniel Borkmann, Alexei Starovoitov, netdev, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, bpf, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Maciej Fijalkowski Hi John, On Thu, Feb 04, 2021 at 09:03:23AM -0800, Jakub Kicinski wrote: > New patchwork can actually find messages by Message-ID header. > > Just slap message ID of one of the patches at the end of: > > https://patchwork.kernel.org/project/netdevbpf/patch/ > > And there is a link to entire series there. Thanks for the tips. > > Since I'm speaking, Hangbin I'd discourage posting new version > as a reply to previous posting. It brings out this massive 100+ > message thread and breaks natural ordering of patches to review. Thanks for the reminder. I will not reply to previous version and will only use a link in future. Thanks Hangbin ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv18 bpf-next 0/6] xdp: add a new helper for dev map multicast support 2021-01-25 12:45 ` [PATCHv17 " Hangbin Liu ` (6 preceding siblings ...) 2021-02-04 0:14 ` [PATCHv17 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2021-02-04 14:03 ` Hangbin Liu 2021-02-04 14:03 ` [PATCHv18 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu ` (5 more replies) 7 siblings, 6 replies; 219+ messages in thread From: Hangbin Liu @ 2021-02-04 14:03 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu This patch is for xdp multicast support. which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because there may have multi interfaces you want to exclude. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, a new helper bpf_redirect_map_multi() is implemented to accept two maps, the forwarding map and exclude map. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. The 1st patch is Jesper's run devmap xdp_prog later in bulking step. The 2st patch add a new bpf arg to allow NULL map pointer. The 3rd patch add the new bpf_redirect_map_multi() helper. The 4-6 patches are for usage sample and testing purpose. I did same perf tests with the following topo: --------------------- --------------------- | Host A (i40e 10G) | ---------- | eno1(i40e 10G) | --------------------- | | | Host B | --------------------- | | | Host C (i40e 10G) | ---------- | eno2(i40e 10G) | --------------------- vlan2 | -------- | | veth1 -- | veth0| | | -------- | --------------------| On Host A: # pktgen/pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -s 64 On Host B(Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, 128G Memory): Use xdp_redirect_map and xdp_redirect_map_multi in samples/bpf for testing. The veth0 in netns load dummy drop program. The forward_map max_entries in xdp_redirect_map_multi is modify to 4. Here is the perf result with 5.10 rc6: The are about +/- 0.1M deviation for native testing Version | Test | Generic | Native | Native + 2nd 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M 5.10 rc6 + patch1 | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch1 | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M 5.10 rc6 + patch1-6 | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch1-6 | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->i40e | 1.7M | 7.8M | 6.4M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->veth | 1.4M | 9.3M | 7.5M 5.10 rc6 + patch1-6 | xdp_redirect_map_multi i40e->i40e+veth | 1.0M | 3.2M | 2.7M Last but not least, thanks a lot to Toke, Jesper, Jiri and Eelco for suggestions and help on implementation. [0] https://xdp-project.net/#Handling-multicast v18: no update, just rebase the code to latest bpf-next v17: For patch 01: a) rename to_sent to to_send. b) clear bq dev_rx, xdp_prog and flush_node in __dev_flush(). v16: refactor bq_xmit_all logic and remove error label for patch 01 v15: Update bq_xmit_all() logic for patch 01. Add some comments and remove useless variable for patch 03. Use bpf_object__find_program_by_title() for patch 04 and 06. v14: No code update, just rebase the code on latest bpf-next v13: Pass in xdp_prog through __xdp_enqueue() for patch 01. Update related code in patch 03. v12: Add Jesper's xdp_prog patch, rebase my works on this and latest bpf-next Add 2nd xdp_prog test on the sample and selftests. v11: Fix bpf_redirect_map_multi() helper description typo. Add loop limit for devmap_get_next_obj() and dev_map_redirect_multi(). v10: Rebase the code to latest bpf-next. Update helper bpf_xdp_redirect_map_multi() - No need to check map pointer as we will do the check in verifier. v9: Update helper bpf_xdp_redirect_map_multi() - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: a) Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). f) Split the tests from sample and add a bpf kernel selftest patch. v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. Hangbin Liu (5): bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL xdp: add a new helper for dev map multicast support sample/bpf: add xdp_redirect_map_multicast test selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL selftests/bpf: add xdp_redirect_multi test Jesper Dangaard Brouer (1): bpf: run devmap xdp_prog on flush instead of bulk enqueue include/linux/bpf.h | 21 ++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 28 ++ kernel/bpf/devmap.c | 262 +++++++++++---- kernel/bpf/verifier.c | 16 +- net/core/filter.c | 124 ++++++- net/core/xdp.c | 29 ++ samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 87 +++++ samples/bpf/xdp_redirect_map_multi_user.c | 302 ++++++++++++++++++ tools/include/uapi/linux/bpf.h | 28 ++ tools/testing/selftests/bpf/Makefile | 3 +- .../bpf/progs/xdp_redirect_multi_kern.c | 111 +++++++ tools/testing/selftests/bpf/test_verifier.c | 22 +- .../selftests/bpf/test_xdp_redirect_multi.sh | 208 ++++++++++++ .../testing/selftests/bpf/verifier/map_ptr.c | 70 ++++ .../selftests/bpf/xdp_redirect_multi.c | 252 +++++++++++++++ 18 files changed, 1501 insertions(+), 67 deletions(-) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c -- 2.26.2 ^ permalink raw reply [flat|nested] 219+ messages in thread
* [PATCHv18 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue 2021-02-04 14:03 ` [PATCHv18 " Hangbin Liu @ 2021-02-04 14:03 ` Hangbin Liu 2021-02-04 14:03 ` [PATCHv18 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu ` (4 subsequent siblings) 5 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-02-04 14:03 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu From: Jesper Dangaard Brouer <brouer@redhat.com> This changes the devmap XDP program support to run the program when the bulk queue is flushed instead of before the frame is enqueued. This has a couple of benefits: - It "sorts" the packets by destination devmap entry, and then runs the same BPF program on all the packets in sequence. This ensures that we keep the XDP program and destination device properties hot in I-cache. - It makes the multicast implementation simpler because it can just enqueue packets using bq_enqueue() without having to deal with the devmap program at all. The drawback is that if the devmap program drops the packet, the enqueue step is redundant. However, arguably this is mostly visible in a micro-benchmark, and with more mixed traffic the I-cache benefit should win out. The performance impact of just this patch is as follows: The bq_xmit_all's logic is also refactored and error label is removed. When bq_xmit_all() is called from bq_enqueue(), another packet will always be enqueued immediately after, so clearing dev_rx, xdp_prog and flush_node in bq_xmit_all() is redundant. Let's move the clear to __dev_flush(), and only check them once in bq_enqueue() since they are all modified together. By using xdp_redirect_map in sample/bpf and send pkts via pktgen cmd: ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 There are about +/- 0.1M deviation for native testing, the performance improved for the base-case, but some drop back with xdp devmap prog attached. Version | Test | Generic | Native | Native + 2nd xdp_prog 5.10 rc6 | xdp_redirect_map i40e->i40e | 2.0M | 9.1M | 8.0M 5.10 rc6 | xdp_redirect_map i40e->veth | 1.7M | 11.0M | 9.7M 5.10 rc6 + patch | xdp_redirect_map i40e->i40e | 2.0M | 9.5M | 7.5M 5.10 rc6 + patch | xdp_redirect_map i40e->veth | 1.7M | 11.6M | 9.1M Reviewed-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com> Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> Acked-by: John Fastabend <john.fastabend@gmail.com> Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> v18: no update, only rebase the code to latest bpf-next v17: a) rename to_sent to to_send. b) clear bq dev_rx, xdp_prog and flush_node in __dev_flush(). v16: a) refactor bq_xmit_all logic and remove error label v15: a) do not use unlikely when checking bq->xdp_prog b) return sent frames for dev_map_bpf_prog_run() v14: no update, only rebase the code v13: pass in xdp_prog through __xdp_enqueue() v2-v12: no this patch --- kernel/bpf/devmap.c | 146 +++++++++++++++++++++++++------------------- 1 file changed, 84 insertions(+), 62 deletions(-) diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index f6e9c68afdd4..bf8b6b5c9cab 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -57,6 +57,7 @@ struct xdp_dev_bulk_queue { struct list_head flush_node; struct net_device *dev; struct net_device *dev_rx; + struct bpf_prog *xdp_prog; unsigned int count; }; @@ -327,46 +328,92 @@ bool dev_map_can_have_prog(struct bpf_map *map) return false; } +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, + struct xdp_frame **frames, int n, + struct net_device *dev) +{ + struct xdp_txq_info txq = { .dev = dev }; + struct xdp_buff xdp; + int i, nframes = 0; + + for (i = 0; i < n; i++) { + struct xdp_frame *xdpf = frames[i]; + u32 act; + int err; + + xdp_convert_frame_to_buff(xdpf, &xdp); + xdp.txq = &txq; + + act = bpf_prog_run_xdp(xdp_prog, &xdp); + switch (act) { + case XDP_PASS: + err = xdp_update_frame_from_buff(&xdp, xdpf); + if (unlikely(err < 0)) + xdp_return_frame_rx_napi(xdpf); + else + frames[nframes++] = xdpf; + break; + default: + bpf_warn_invalid_xdp_action(act); + fallthrough; + case XDP_ABORTED: + trace_xdp_exception(dev, xdp_prog, act); + fallthrough; + case XDP_DROP: + xdp_return_frame_rx_napi(xdpf); + break; + } + } + return nframes; /* sent frames count */ +} + static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags) { struct net_device *dev = bq->dev; - int sent = 0, drops = 0, err = 0; + unsigned int cnt = bq->count; + int drops = 0, err = 0; + int to_send = cnt; + int sent = cnt; int i; - if (unlikely(!bq->count)) + if (unlikely(!cnt)) return; - for (i = 0; i < bq->count; i++) { + for (i = 0; i < cnt; i++) { struct xdp_frame *xdpf = bq->q[i]; prefetch(xdpf); } - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags); + if (bq->xdp_prog) { + to_send = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev); + if (!to_send) { + sent = 0; + goto out; + } + drops = cnt - to_send; + } + + sent = dev->netdev_ops->ndo_xdp_xmit(dev, to_send, bq->q, flags); if (sent < 0) { err = sent; sent = 0; - goto error; + + /* If ndo_xdp_xmit fails with an errno, no frames have been + * xmit'ed and it's our responsibility to them free all. + */ + for (i = 0; i < cnt - drops; i++) { + struct xdp_frame *xdpf = bq->q[i]; + + xdp_return_frame_rx_napi(xdpf); + } } - drops = bq->count - sent; out: + drops = cnt - sent; bq->count = 0; trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err); - bq->dev_rx = NULL; - __list_del_clearprev(&bq->flush_node); return; -error: - /* If ndo_xdp_xmit fails with an errno, no frames have been - * xmit'ed and it's our responsibility to them free all. - */ - for (i = 0; i < bq->count; i++) { - struct xdp_frame *xdpf = bq->q[i]; - - xdp_return_frame_rx_napi(xdpf); - drops++; - } - goto out; } /* __dev_flush is called from xdp_do_flush() which _must_ be signaled @@ -384,8 +431,12 @@ void __dev_flush(void) struct list_head *flush_list = this_cpu_ptr(&dev_flush_list); struct xdp_dev_bulk_queue *bq, *tmp; - list_for_each_entry_safe(bq, tmp, flush_list, flush_node) + list_for_each_entry_safe(bq, tmp, flush_list, flush_node) { bq_xmit_all(bq, XDP_XMIT_FLUSH); + bq->dev_rx = NULL; + bq->xdp_prog = NULL; + __list_del_clearprev(&bq->flush_node); + } } /* rcu_read_lock (from syscall and BPF contexts) ensures that if a delete and/or @@ -408,7 +459,7 @@ struct bpf_dtab_netdev *__dev_map_lookup_elem(struct bpf_map *map, u32 key) * Thus, safe percpu variable access. */ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, - struct net_device *dev_rx) + struct net_device *dev_rx, struct bpf_prog *xdp_prog) { struct list_head *flush_list = this_cpu_ptr(&dev_flush_list); struct xdp_dev_bulk_queue *bq = this_cpu_ptr(dev->xdp_bulkq); @@ -419,18 +470,22 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, /* Ingress dev_rx will be the same for all xdp_frame's in * bulk_queue, because bq stored per-CPU and must be flushed * from net_device drivers NAPI func end. + * + * Do the same with xdp_prog and flush_list since these fields + * are only ever modified together. */ - if (!bq->dev_rx) + if (!bq->dev_rx) { bq->dev_rx = dev_rx; + bq->xdp_prog = xdp_prog; + list_add(&bq->flush_node, flush_list); + } bq->q[bq->count++] = xdpf; - - if (!bq->flush_node.prev) - list_add(&bq->flush_node, flush_list); } static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, - struct net_device *dev_rx) + struct net_device *dev_rx, + struct bpf_prog *xdp_prog) { struct xdp_frame *xdpf; int err; @@ -446,42 +501,14 @@ static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, if (unlikely(!xdpf)) return -EOVERFLOW; - bq_enqueue(dev, xdpf, dev_rx); + bq_enqueue(dev, xdpf, dev_rx, xdp_prog); return 0; } -static struct xdp_buff *dev_map_run_prog(struct net_device *dev, - struct xdp_buff *xdp, - struct bpf_prog *xdp_prog) -{ - struct xdp_txq_info txq = { .dev = dev }; - u32 act; - - xdp_set_data_meta_invalid(xdp); - xdp->txq = &txq; - - act = bpf_prog_run_xdp(xdp_prog, xdp); - switch (act) { - case XDP_PASS: - return xdp; - case XDP_DROP: - break; - default: - bpf_warn_invalid_xdp_action(act); - fallthrough; - case XDP_ABORTED: - trace_xdp_exception(dev, xdp_prog, act); - break; - } - - xdp_return_buff(xdp); - return NULL; -} - int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx) { - return __xdp_enqueue(dev, xdp, dev_rx); + return __xdp_enqueue(dev, xdp, dev_rx, NULL); } int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, @@ -489,12 +516,7 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, { struct net_device *dev = dst->dev; - if (dst->xdp_prog) { - xdp = dev_map_run_prog(dev, xdp, dst->xdp_prog); - if (!xdp) - return 0; - } - return __xdp_enqueue(dev, xdp, dev_rx); + return __xdp_enqueue(dev, xdp, dev_rx, dst->xdp_prog); } int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv18 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL 2021-02-04 14:03 ` [PATCHv18 " Hangbin Liu 2021-02-04 14:03 ` [PATCHv18 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu @ 2021-02-04 14:03 ` Hangbin Liu 2021-02-04 14:03 ` [PATCHv18 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu ` (3 subsequent siblings) 5 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-02-04 14:03 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu Add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL which could be used when we want to allow NULL pointer for map parameter. The bpf helper need to take care and check if the map is NULL when use this type. Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> Acked-by: John Fastabend <john.fastabend@gmail.com> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> v13-v18: no update v11-v12: rebase the patch to latest bpf-next v10: remove useless CONST_PTR_TO_MAP_OR_NULL and Copy-paste comment. v9: merge the patch from [1] in to this series. v1-v8: no this patch [1] https://lore.kernel.org/bpf/20200715070001.2048207-1-liuhangbin@gmail.com/ --- include/linux/bpf.h | 1 + kernel/bpf/verifier.c | 10 ++++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 321966fc35db..b0777c8c03fd 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -296,6 +296,7 @@ enum bpf_arg_type { ARG_CONST_ALLOC_SIZE_OR_ZERO, /* number of allocated bytes requested */ ARG_PTR_TO_BTF_ID_SOCK_COMMON, /* pointer to in-kernel sock_common or bpf-mirrored bpf_sock */ ARG_PTR_TO_PERCPU_BTF_ID, /* pointer to in-kernel percpu type */ + ARG_CONST_MAP_PTR_OR_NULL, /* const argument used as pointer to bpf_map or NULL */ __BPF_ARG_TYPE_MAX, }; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 5e09632efddb..50a17f80358b 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -445,7 +445,8 @@ static bool arg_type_may_be_null(enum bpf_arg_type type) type == ARG_PTR_TO_MEM_OR_NULL || type == ARG_PTR_TO_CTX_OR_NULL || type == ARG_PTR_TO_SOCKET_OR_NULL || - type == ARG_PTR_TO_ALLOC_MEM_OR_NULL; + type == ARG_PTR_TO_ALLOC_MEM_OR_NULL || + type == ARG_CONST_MAP_PTR_OR_NULL; } /* Determine whether the function releases some resources allocated by another @@ -4112,6 +4113,7 @@ static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = { [ARG_CONST_SIZE_OR_ZERO] = &scalar_types, [ARG_CONST_ALLOC_SIZE_OR_ZERO] = &scalar_types, [ARG_CONST_MAP_PTR] = &const_map_ptr_types, + [ARG_CONST_MAP_PTR_OR_NULL] = &const_map_ptr_types, [ARG_PTR_TO_CTX] = &context_types, [ARG_PTR_TO_CTX_OR_NULL] = &context_types, [ARG_PTR_TO_SOCK_COMMON] = &sock_types, @@ -4257,9 +4259,9 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, meta->ref_obj_id = reg->ref_obj_id; } - if (arg_type == ARG_CONST_MAP_PTR) { - /* bpf_map_xxx(map_ptr) call: remember that map_ptr */ - meta->map_ptr = reg->map_ptr; + if (arg_type == ARG_CONST_MAP_PTR || + arg_type == ARG_CONST_MAP_PTR_OR_NULL) { + meta->map_ptr = register_is_null(reg) ? NULL : reg->map_ptr; } else if (arg_type == ARG_PTR_TO_MAP_KEY) { /* bpf_map_xxx(..., map_ptr, ..., key) call: * check that [key, key + map->key_size) are within -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv18 bpf-next 3/6] xdp: add a new helper for dev map multicast support 2021-02-04 14:03 ` [PATCHv18 " Hangbin Liu 2021-02-04 14:03 ` [PATCHv18 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu 2021-02-04 14:03 ` [PATCHv18 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2021-02-04 14:03 ` Hangbin Liu 2021-02-04 14:03 ` [PATCHv18 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu ` (2 subsequent siblings) 5 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-02-04 14:03 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu This patch is for xdp multicast support, which has been discussed before[0], The goal is to be able to implement an OVS-like data plane in XDP, i.e., a software switch that can forward XDP frames to multiple ports. To achieve this, an application needs to specify a group of interfaces to forward a packet to. It is also common to want to exclude one or more physical interfaces from the forwarding operation - e.g., to forward a packet to all interfaces in the multicast group except the interface it arrived on. While this could be done simply by adding more groups, this quickly leads to a combinatorial explosion in the number of groups an application has to maintain. To avoid the combinatorial explosion, we propose to include the ability to specify an "exclude group" as part of the forwarding operation. This needs to be a group (instead of just a single port index), because a physical interface can be part of a logical grouping, such as a bond device. Thus, the logical forwarding operation becomes a "set difference" operation, i.e. "forward to all ports in group A that are not also in group B". This series implements such an operation using device maps to represent the groups. This means that the XDP program specifies two device maps, one containing the list of netdevs to redirect to, and the other containing the exclude list. To achieve this, a new helper bpf_redirect_map_multi() is implemented to accept two maps, the forwarding map and exclude map. The forwarding map could be DEVMAP or DEVMAP_HASH, but the exclude map *must* be DEVMAP_HASH to get better performace. If user don't want to use exclude map and just want simply stop redirecting back to ingress device, they can use flag BPF_F_EXCLUDE_INGRESS. As both bpf_xdp_redirect_map() and this new helpers are using struct bpf_redirect_info, a new field ex_map is added and tgt_value is set to NULL in the new helper to make a difference with bpf_xdp_redirect_map(). At last, keep the general data path in net/core/filter.c, the native data path in kernel/bpf/devmap.c so we can use direct calls to get better performace. [0] https://xdp-project.net/#Handling-multicast Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> v16-v18: no update v15: a) Update bpf_redirect_map_multi() helper description that ex_map must be keyed by ifindex. b) remove variable last_one in dev_map_enqueue_multi() as it's pointless. c) add a comment about why we don't use READ/WRITE_ONCE() for ex_map. v14: no update, only rebase the code v13: pass xdp_prog through bq_enqueue v12: rebase the code based on Jespoer's devmap xdp_prog patch v11: Fix bpf_redirect_map_multi() helper description typo. Add loop limit for devmap_get_next_obj() and dev_map_redirect_multi(). v10: Update helper bpf_xdp_redirect_map_multi() - No need to check map pointer as we will do the check in verifier. v9: Update helper bpf_xdp_redirect_map_multi() - Use ARG_CONST_MAP_PTR_OR_NULL for helper arg2 v8: Update function dev_in_exclude_map(): - remove duplicate ex_map map_type check in - lookup the element in dev map by obj dev index directly instead of looping all the map v7: a) Fix helper flag check b) Limit the *ex_map* to use DEVMAP_HASH only and update function dev_in_exclude_map() to get better performance. v6: converted helper return types from int to long v5: a) Check devmap_get_next_key() return value. b) Pass through flags to __bpf_tx_xdp_map() instead of bool value. c) In function dev_map_enqueue_multi(), consume xdpf for the last obj instead of the first on. d) Update helper description and code comments to explain that we use NULL target value to distinguish multicast and unicast forwarding. e) Update memory model, memory id and frame_sz in xdpf_clone(). v4: Fix bpf_xdp_redirect_map_multi_proto arg2_type typo v3: Based on Toke's suggestion, do the following update a) Update bpf_redirect_map_multi() description in bpf.h. b) Fix exclude_ifindex checking order in dev_in_exclude_map(). c) Fix one more xdpf clone in dev_map_enqueue_multi(). d) Go find next one in dev_map_enqueue_multi() if the interface is not able to forward instead of abort the whole loop. e) Remove READ_ONCE/WRITE_ONCE for ex_map. v2: Add new syscall bpf_xdp_redirect_map_multi() which could accept include/exclude maps directly. --- include/linux/bpf.h | 20 ++++++ include/linux/filter.h | 1 + include/net/xdp.h | 1 + include/uapi/linux/bpf.h | 28 ++++++++ kernel/bpf/devmap.c | 128 +++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++ net/core/filter.c | 124 ++++++++++++++++++++++++++++++-- net/core/xdp.c | 29 ++++++++ tools/include/uapi/linux/bpf.h | 28 ++++++++ 9 files changed, 360 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index b0777c8c03fd..68c136bfde8f 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1441,6 +1441,11 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx); int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, struct net_device *dev_rx); +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex); +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); bool dev_map_can_have_prog(struct bpf_map *map); @@ -1609,6 +1614,21 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return 0; } +static inline +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + return false; +} + +static inline +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + return 0; +} + struct sk_buff; static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, diff --git a/include/linux/filter.h b/include/linux/filter.h index 5b3137d7b690..aa9f87ded63c 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -637,6 +637,7 @@ struct bpf_redirect_info { u32 tgt_index; void *tgt_value; struct bpf_map *map; + struct bpf_map *ex_map; u32 kern_flags; struct bpf_nh_params nh; }; diff --git a/include/net/xdp.h b/include/net/xdp.h index a5bc214a49d9..5533f0ab2afc 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -170,6 +170,7 @@ struct sk_buff *__xdp_build_skb_from_frame(struct xdp_frame *xdpf, struct sk_buff *xdp_build_skb_from_frame(struct xdp_frame *xdpf, struct net_device *dev); int xdp_alloc_skb_bulk(void **skbs, int n_skb, gfp_t gfp); +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); static inline void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index c001766adcbc..ef943c024322 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3836,6 +3836,28 @@ union bpf_attr { * Return * A pointer to a struct socket on success or NULL if the file is * not a socket. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. To get better performance, the + * *ex_map* is limited to BPF_MAP_TYPE_DEVMAP_HASH and must be + * keyed by ifindex for the helper to work. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -4001,6 +4023,7 @@ union bpf_attr { FN(ktime_get_coarse_ns), \ FN(ima_inode_hash), \ FN(sock_from_file), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -4177,6 +4200,11 @@ enum { BPF_F_BPRM_SECUREEXEC = (1ULL << 0), }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index bf8b6b5c9cab..217e09533097 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -519,6 +519,134 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, return __xdp_enqueue(dev, xdp, dev_rx, dst->xdp_prog); } +/* Use direct call in fast path instead of map->ops->map_get_next_key() */ +static int devmap_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + return dev_map_get_next_key(map, key, next_key); + case BPF_MAP_TYPE_DEVMAP_HASH: + return dev_map_hash_get_next_key(map, key, next_key); + default: + break; + } + + return -ENOENT; +} + +bool dev_in_exclude_map(struct bpf_dtab_netdev *obj, struct bpf_map *map, + int exclude_ifindex) +{ + if (obj->dev->ifindex == exclude_ifindex) + return true; + + if (!map) + return false; + + return __dev_map_hash_lookup_elem(map, obj->dev->ifindex) != NULL; +} + +static struct bpf_dtab_netdev *devmap_get_next_obj(struct xdp_buff *xdp, struct bpf_map *map, + struct bpf_map *ex_map, u32 *key, + u32 *next_key, int ex_ifindex) +{ + struct bpf_dtab_netdev *obj; + struct net_device *dev; + u32 *tmp_key = key; + u32 index; + int err; + + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + return NULL; + + /* When using dev map hash, we could restart the hashtab traversal + * in case the key has been updated/removed in the mean time. + * So we may end up potentially looping due to traversal restarts + * from first elem. + * + * Let's use map's max_entries to limit the loop number. + */ + for (index = 0; index < map->max_entries; index++) { + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + obj = __dev_map_lookup_elem(map, *next_key); + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + obj = __dev_map_hash_lookup_elem(map, *next_key); + break; + default: + break; + } + + if (!obj || dev_in_exclude_map(obj, ex_map, ex_ifindex)) + goto find_next; + + dev = obj->dev; + + if (!dev->netdev_ops->ndo_xdp_xmit) + goto find_next; + + err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + if (unlikely(err)) + goto find_next; + + return obj; + +find_next: + tmp_key = next_key; + err = devmap_get_next_key(map, tmp_key, next_key); + if (err) + break; + } + + return NULL; +} + +int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) +{ + struct bpf_dtab_netdev *obj = NULL, *next_obj = NULL; + struct xdp_frame *xdpf, *nxdpf; + int ex_ifindex; + u32 key, next_key; + + ex_ifindex = flags & BPF_F_EXCLUDE_INGRESS ? dev_rx->ifindex : 0; + + /* Find first available obj */ + obj = devmap_get_next_obj(xdp, map, ex_map, NULL, &key, ex_ifindex); + if (!obj) + return 0; + + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) + return -EOVERFLOW; + + for (;;) { + /* Check if we still have one more available obj */ + next_obj = devmap_get_next_obj(xdp, map, ex_map, &key, + &next_key, ex_ifindex); + if (!next_obj) { + bq_enqueue(obj->dev, xdpf, dev_rx, obj->xdp_prog); + return 0; + } + + nxdpf = xdpf_clone(xdpf); + if (unlikely(!nxdpf)) { + xdp_return_frame_rx_napi(xdpf); + return -ENOMEM; + } + + bq_enqueue(obj->dev, nxdpf, dev_rx, obj->xdp_prog); + + /* Deal with next obj */ + obj = next_obj; + key = next_key; + } +} + int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 50a17f80358b..99644d52ae0e 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4467,6 +4467,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: if (func_id != BPF_FUNC_redirect_map && + func_id != BPF_FUNC_redirect_map_multi && func_id != BPF_FUNC_map_lookup_elem) goto error; break; @@ -4571,6 +4572,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, map->map_type != BPF_MAP_TYPE_XSKMAP) goto error; break; + case BPF_FUNC_redirect_map_multi: + if (map->map_type != BPF_MAP_TYPE_DEVMAP && + map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) + goto error; + break; case BPF_FUNC_sk_redirect_map: case BPF_FUNC_msg_redirect_map: case BPF_FUNC_sock_map_update: diff --git a/net/core/filter.c b/net/core/filter.c index e15d4741719a..a5617d850237 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3924,12 +3924,19 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { }; static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) + struct bpf_map *map, struct xdp_buff *xdp, + struct bpf_map *ex_map, u32 flags) { switch (map->map_type) { case BPF_MAP_TYPE_DEVMAP: case BPF_MAP_TYPE_DEVMAP_HASH: - return dev_map_enqueue(fwd, xdp, dev_rx); + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) + return dev_map_enqueue(fwd, xdp, dev_rx); + else + return dev_map_enqueue_multi(xdp, dev_rx, map, ex_map, flags); case BPF_MAP_TYPE_CPUMAP: return cpu_map_enqueue(fwd, xdp, dev_rx); case BPF_MAP_TYPE_XSKMAP: @@ -3986,12 +3993,19 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); struct bpf_map *map = READ_ONCE(ri->map); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err; + /* The READ/WRITE_ONCE() is not needed for ex_map because the field + * is only read from or written to by the CPU owning the per-cpu + * pointer. Whereas the 'map' field is manipulated by remote CPUs + * in bpf_clear_redirect_map(). + */ ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (unlikely(!map)) { @@ -4003,7 +4017,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = dev_xdp_enqueue(fwd, xdp, dev); } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + err = __bpf_tx_xdp_map(dev, fwd, map, xdp, ex_map, ri->flags); } if (unlikely(err)) @@ -4017,6 +4031,63 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, } EXPORT_SYMBOL_GPL(xdp_do_redirect); +static int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, + struct bpf_prog *xdp_prog, + struct bpf_map *map, struct bpf_map *ex_map, + u32 flags) + +{ + struct bpf_dtab_netdev *dst; + struct sk_buff *nskb; + bool exclude_ingress; + u32 key, next_key, index; + void *fwd; + int err; + + /* Get first key from forward map */ + err = map->ops->map_get_next_key(map, NULL, &key); + if (err) + return err; + + exclude_ingress = !!(flags & BPF_F_EXCLUDE_INGRESS); + + /* When using dev map hash, we could restart the hashtab traversal + * in case the key has been updated/removed in the mean time. + * So we may end up potentially looping due to traversal restarts + * from first elem. + * + * Let's use map's max_entries to limit the loop number. + */ + for (index = 0; index < map->max_entries; index++) { + fwd = __xdp_map_lookup_elem(map, key); + if (fwd) { + dst = (struct bpf_dtab_netdev *)fwd; + if (dev_in_exclude_map(dst, ex_map, + exclude_ingress ? dev->ifindex : 0)) + goto find_next; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -ENOMEM; + + /* Try forword next one no mater the current forward + * succeed or not. + */ + dev_map_generic_redirect(dst, nskb, xdp_prog); + } + +find_next: + err = map->ops->map_get_next_key(map, &key, &next_key); + if (err) + break; + + key = next_key; + } + + consume_skb(skb); + return 0; +} + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, @@ -4024,19 +4095,30 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, struct bpf_map *map) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + struct bpf_map *ex_map = ri->ex_map; u32 index = ri->tgt_index; void *fwd = ri->tgt_value; int err = 0; ri->tgt_index = 0; ri->tgt_value = NULL; + ri->ex_map = NULL; WRITE_ONCE(ri->map, NULL); if (map->map_type == BPF_MAP_TYPE_DEVMAP || map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { - struct bpf_dtab_netdev *dst = fwd; + /* We use a NULL fwd value to distinguish multicast + * and unicast forwarding + */ + if (fwd) { + struct bpf_dtab_netdev *dst = fwd; + + err = dev_map_generic_redirect(dst, skb, xdp_prog); + } else { + err = dev_map_redirect_multi(dev, skb, xdp_prog, map, + ex_map, ri->flags); + } - err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { @@ -4150,6 +4232,36 @@ static const struct bpf_func_proto bpf_xdp_redirect_map_proto = { .arg3_type = ARG_ANYTHING, }; +BPF_CALL_3(bpf_xdp_redirect_map_multi, struct bpf_map *, map, + struct bpf_map *, ex_map, u64, flags) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + + /* Limit ex_map type to DEVMAP_HASH to get better performance */ + if (unlikely((ex_map && ex_map->map_type != BPF_MAP_TYPE_DEVMAP_HASH) || + flags & ~BPF_F_EXCLUDE_INGRESS)) + return XDP_ABORTED; + + ri->tgt_index = 0; + /* Set the tgt_value to NULL to distinguish with bpf_xdp_redirect_map */ + ri->tgt_value = NULL; + ri->flags = flags; + ri->ex_map = ex_map; + + WRITE_ONCE(ri->map, map); + + return XDP_REDIRECT; +} + +static const struct bpf_func_proto bpf_xdp_redirect_map_multi_proto = { + .func = bpf_xdp_redirect_map_multi, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_CONST_MAP_PTR, + .arg2_type = ARG_CONST_MAP_PTR_OR_NULL, + .arg3_type = ARG_ANYTHING, +}; + static unsigned long bpf_skb_copy(void *dst_buff, const void *skb, unsigned long off, unsigned long len) { @@ -7264,6 +7376,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_redirect_proto; case BPF_FUNC_redirect_map: return &bpf_xdp_redirect_map_proto; + case BPF_FUNC_redirect_map_multi: + return &bpf_xdp_redirect_map_multi_proto; case BPF_FUNC_xdp_adjust_tail: return &bpf_xdp_adjust_tail_proto; case BPF_FUNC_fib_lookup: diff --git a/net/core/xdp.c b/net/core/xdp.c index 05354976c1fc..aba84d04642b 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -583,3 +583,32 @@ struct sk_buff *xdp_build_skb_from_frame(struct xdp_frame *xdpf, return __xdp_build_skb_from_frame(xdpf, skb, dev); } EXPORT_SYMBOL_GPL(xdp_build_skb_from_frame); + +struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) +{ + unsigned int headroom, totalsize; + struct xdp_frame *nxdpf; + struct page *page; + void *addr; + + headroom = xdpf->headroom + sizeof(*xdpf); + totalsize = headroom + xdpf->len; + + if (unlikely(totalsize > PAGE_SIZE)) + return NULL; + page = dev_alloc_page(); + if (!page) + return NULL; + addr = page_to_virt(page); + + memcpy(addr, xdpf, totalsize); + + nxdpf = addr; + nxdpf->data = addr + headroom; + nxdpf->frame_sz = PAGE_SIZE; + nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0; + nxdpf->mem.id = 0; + + return nxdpf; +} +EXPORT_SYMBOL_GPL(xdpf_clone); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index c001766adcbc..ef943c024322 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3836,6 +3836,28 @@ union bpf_attr { * Return * A pointer to a struct socket on success or NULL if the file is * not a socket. + * + * long bpf_redirect_map_multi(struct bpf_map *map, struct bpf_map *ex_map, u64 flags) + * Description + * This is a multicast implementation for XDP redirect. It will + * redirect the packet to ALL the interfaces in *map*, but + * exclude the interfaces in *ex_map*. + * + * The forwarding *map* could be either BPF_MAP_TYPE_DEVMAP or + * BPF_MAP_TYPE_DEVMAP_HASH. To get better performance, the + * *ex_map* is limited to BPF_MAP_TYPE_DEVMAP_HASH and must be + * keyed by ifindex for the helper to work. + * + * Currently the *flags* only supports *BPF_F_EXCLUDE_INGRESS*, + * which additionally excludes the current ingress device. + * + * See also bpf_redirect_map() as a unicast implementation, + * which supports redirecting packet to a specific ifindex + * in the map. As both helpers use struct bpf_redirect_info + * to store the redirect info, we will use a a NULL tgt_value + * to distinguish multicast and unicast redirecting. + * Return + * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -4001,6 +4023,7 @@ union bpf_attr { FN(ktime_get_coarse_ns), \ FN(ima_inode_hash), \ FN(sock_from_file), \ + FN(redirect_map_multi), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -4177,6 +4200,11 @@ enum { BPF_F_BPRM_SECUREEXEC = (1ULL << 0), }; +/* BPF_FUNC_redirect_map_multi flags. */ +enum { + BPF_F_EXCLUDE_INGRESS = (1ULL << 0), +}; + #define __bpf_md_ptr(type, name) \ union { \ type name; \ -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv18 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test 2021-02-04 14:03 ` [PATCHv18 " Hangbin Liu ` (2 preceding siblings ...) 2021-02-04 14:03 ` [PATCHv18 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu @ 2021-02-04 14:03 ` Hangbin Liu 2021-02-04 14:03 ` [PATCHv18 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2021-02-04 14:03 ` [PATCHv18 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 5 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-02-04 14:03 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu This is a sample for xdp multicast. In the sample we could forward all packets between given interfaces. There is also an option -X that could enable 2nd xdp_prog on egress interface. Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> v16-v18: no update v15: use bpf_object__find_program_by_name() instead of bpf_object__find_program_by_title() v13-v14: no update, only rebase the code v12: add devmap xdp_prog on egress support v10-v11: no update v9: use NULL directly for arg2 and redefine the maps with btf format v5: add a null_map as we have strict the arg2 to ARG_CONST_MAP_PTR. Move the testing part to bpf selftest in next patch. v4: no update. v3: add rxcnt map to show the packet transmit speed. v2: no update. --- samples/bpf/Makefile | 3 + samples/bpf/xdp_redirect_map_multi_kern.c | 87 +++++++ samples/bpf/xdp_redirect_map_multi_user.c | 302 ++++++++++++++++++++++ 3 files changed, 392 insertions(+) create mode 100644 samples/bpf/xdp_redirect_map_multi_kern.c create mode 100644 samples/bpf/xdp_redirect_map_multi_user.c diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 45ceca4e2c70..520434ea966f 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -41,6 +41,7 @@ tprogs-y += test_map_in_map tprogs-y += per_socket_stats_example tprogs-y += xdp_redirect tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_map_multi tprogs-y += xdp_redirect_cpu tprogs-y += xdp_monitor tprogs-y += xdp_rxq_info @@ -99,6 +100,7 @@ test_map_in_map-objs := test_map_in_map_user.o per_socket_stats_example-objs := cookie_uid_helper_example.o xdp_redirect-objs := xdp_redirect_user.o xdp_redirect_map-objs := xdp_redirect_map_user.o +xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o xdp_redirect_cpu-objs := xdp_redirect_cpu_user.o xdp_monitor-objs := xdp_monitor_user.o xdp_rxq_info-objs := xdp_rxq_info_user.o @@ -160,6 +162,7 @@ always-y += tcp_tos_reflect_kern.o always-y += tcp_dumpstats_kern.o always-y += xdp_redirect_kern.o always-y += xdp_redirect_map_kern.o +always-y += xdp_redirect_map_multi_kern.o always-y += xdp_redirect_cpu_kern.o always-y += xdp_monitor_kern.o always-y += xdp_rxq_info_kern.o diff --git a/samples/bpf/xdp_redirect_map_multi_kern.c b/samples/bpf/xdp_redirect_map_multi_kern.c new file mode 100644 index 000000000000..e422340d1251 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_kern.c @@ -0,0 +1,87 @@ +// SPDX-License-Identifier: GPL-2.0 +#define KBUILD_MODNAME "foo" +#include <uapi/linux/bpf.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/ip.h> +#include <linux/ipv6.h> +#include <bpf/bpf_helpers.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 32); +} forward_map_general SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(struct bpf_devmap_val)); + __uint(max_entries, 32); +} forward_map_native SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __type(key, u32); + __type(value, long); + __uint(max_entries, 1); +} rxcnt SEC(".maps"); + +/* map to store egress interfaces mac addresses, set the + * max_entries to 1 and extend it in user sapce prog. + */ +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, u32); + __type(value, __be64); + __uint(max_entries, 1); +} mac_map SEC(".maps"); + +static int xdp_redirect_map(struct xdp_md *ctx, void *forward_map) +{ + long *value; + u32 key = 0; + + /* count packet in global counter */ + value = bpf_map_lookup_elem(&rxcnt, &key); + if (value) + *value += 1; + + return bpf_redirect_map_multi(forward_map, NULL, BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_redirect_general") +int xdp_redirect_map_general(struct xdp_md *ctx) +{ + return xdp_redirect_map(ctx, &forward_map_general); +} + +SEC("xdp_redirect_native") +int xdp_redirect_map_native(struct xdp_md *ctx) +{ + return xdp_redirect_map(ctx, &forward_map_native); +} + +SEC("xdp_devmap/map_prog") +int xdp_devmap_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + u32 key = ctx->egress_ifindex; + struct ethhdr *eth = data; + __be64 *mac; + u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + mac = bpf_map_lookup_elem(&mac_map, &key); + if (mac) + __builtin_memcpy(eth->h_source, mac, ETH_ALEN); + + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/samples/bpf/xdp_redirect_map_multi_user.c b/samples/bpf/xdp_redirect_map_multi_user.c new file mode 100644 index 000000000000..84cdbbed20b7 --- /dev/null +++ b/samples/bpf/xdp_redirect_map_multi_user.c @@ -0,0 +1,302 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> +#include <sys/resource.h> +#include <sys/ioctl.h> +#include <sys/types.h> +#include <sys/socket.h> +#include <netinet/in.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; +static int rxcnt_map_fd; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static void poll_stats(int interval) +{ + unsigned int nr_cpus = bpf_num_possible_cpus(); + __u64 values[nr_cpus], prev[nr_cpus]; + + memset(prev, 0, sizeof(prev)); + + while (1) { + __u64 sum = 0; + __u32 key = 0; + int i; + + sleep(interval); + assert(bpf_map_lookup_elem(rxcnt_map_fd, &key, values) == 0); + for (i = 0; i < nr_cpus; i++) + sum += (values[i] - prev[i]); + if (sum) + printf("Forwarding %10llu pkt/s\n", sum / interval); + memcpy(prev, values, sizeof(values)); + } +} + +static int get_mac_addr(unsigned int ifindex, void *mac_addr) +{ + char ifname[IF_NAMESIZE]; + struct ifreq ifr; + int fd, ret = -1; + + fd = socket(AF_INET, SOCK_DGRAM, 0); + if (fd < 0) + return ret; + + if (!if_indextoname(ifindex, ifname)) + goto err_out; + + strcpy(ifr.ifr_name, ifname); + + if (ioctl(fd, SIOCGIFHWADDR, &ifr) != 0) + goto err_out; + + memcpy(mac_addr, ifr.ifr_hwaddr.sa_data, 6 * sizeof(char)); + ret = 0; + +err_out: + close(fd); + return ret; +} + +static int update_mac_map(struct bpf_object *obj) +{ + int i, ret = -1, mac_map_fd; + unsigned char mac_addr[6]; + unsigned int ifindex; + + mac_map_fd = bpf_object__find_map_fd_by_name(obj, "mac_map"); + if (mac_map_fd < 0) { + printf("find mac map fd failed\n"); + return ret; + } + + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + ret = get_mac_addr(ifindex, mac_addr); + if (ret < 0) { + printf("get interface %d mac failed\n", ifindex); + return ret; + } + + ret = bpf_map_update_elem(mac_map_fd, &ifindex, mac_addr, 0); + if (ret) { + perror("bpf_update_elem mac_map_fd"); + return ret; + } + } + + return 0; +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n" + " -X load xdp program on egress\n", + prog); +} + +int main(int argc, char **argv) +{ + int i, ret, opt, forward_map_fd, max_ifindex = 0; + struct bpf_program *ingress_prog, *egress_prog; + int ingress_prog_fd, egress_prog_fd = 0; + struct bpf_devmap_val devmap_val; + bool attach_egress_prog = false; + char ifname[IF_NAMESIZE]; + struct bpf_map *mac_map; + struct bpf_object *obj; + unsigned int ifindex; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNFX")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + case 'X': + attach_egress_prog = true; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) { + xdp_flags |= XDP_FLAGS_DRV_MODE; + } else if (attach_egress_prog) { + printf("Load xdp program on egress with SKB mode not supported yet\n"); + return 1; + } + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + return 1; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + return 1; + } + + /* Find the largest index number */ + if (ifaces[i] > max_ifindex) + max_ifindex = ifaces[i]; + + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + + obj = bpf_object__open(filename); + if (libbpf_get_error(obj)) { + printf("ERROR: opening BPF object file failed\n"); + obj = NULL; + goto err_out; + } + + /* Reset the map size to max ifindex + 1 */ + if (attach_egress_prog) { + mac_map = bpf_object__find_map_by_name(obj, "mac_map"); + ret = bpf_map__resize(mac_map, max_ifindex + 1); + if (ret < 0) { + printf("ERROR: reset mac map size failed\n"); + goto err_out; + } + } + + /* load BPF program */ + if (bpf_object__load(obj)) { + printf("ERROR: loading BPF object file failed\n"); + goto err_out; + } + + if (xdp_flags & XDP_FLAGS_SKB_MODE) { + ingress_prog = bpf_object__find_program_by_name(obj, "xdp_redirect_map_general"); + forward_map_fd = bpf_object__find_map_fd_by_name(obj, "forward_map_general"); + } else { + ingress_prog = bpf_object__find_program_by_name(obj, "xdp_redirect_map_native"); + forward_map_fd = bpf_object__find_map_fd_by_name(obj, "forward_map_native"); + } + if (!ingress_prog || forward_map_fd < 0) { + printf("finding ingress_prog/forward_map in obj file failed\n"); + goto err_out; + } + + ingress_prog_fd = bpf_program__fd(ingress_prog); + if (ingress_prog_fd < 0) { + printf("find ingress_prog fd failed\n"); + goto err_out; + } + + rxcnt_map_fd = bpf_object__find_map_fd_by_name(obj, "rxcnt"); + if (rxcnt_map_fd < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + goto err_out; + } + + if (attach_egress_prog) { + /* Update mac_map with all egress interfaces' mac addr */ + if (update_mac_map(obj) < 0) { + printf("Error: update mac map failed"); + goto err_out; + } + + /* Find egress prog fd */ + egress_prog = bpf_object__find_program_by_name(obj, "xdp_devmap_prog"); + if (!egress_prog) { + printf("finding egress_prog in obj file failed\n"); + goto err_out; + } + egress_prog_fd = bpf_program__fd(egress_prog); + if (egress_prog_fd < 0) { + printf("find egress_prog fd failed\n"); + goto err_out; + } + } + + /* Remove attached program when program is interrupted or killed */ + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, ingress_prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + + /* Add all the interfaces to forward group and attach + * egress devmap programe if exist + */ + devmap_val.ifindex = ifindex; + devmap_val.bpf_prog.fd = egress_prog_fd; + ret = bpf_map_update_elem(forward_map_fd, &ifindex, &devmap_val, 0); + if (ret) { + perror("bpf_map_update_elem forward_map"); + goto err_out; + } + } + + poll_stats(2); + + return 0; + +err_out: + return 1; +} -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv18 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL 2021-02-04 14:03 ` [PATCHv18 " Hangbin Liu ` (3 preceding siblings ...) 2021-02-04 14:03 ` [PATCHv18 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu @ 2021-02-04 14:03 ` Hangbin Liu 2021-02-04 14:03 ` [PATCHv18 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 5 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-02-04 14:03 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu Use helper bpf_redirect_map() and bpf_redirect_map_multi() to test bpf arg ARG_CONST_MAP_PTR and ARG_CONST_MAP_PTR_OR_NULL. Make sure the map arg could be verified correctly when it is NULL or valid map pointer. Add devmap and devmap_hash in struct bpf_test due to bpf_redirect_{map, map_multi} limit. Test result: ]# ./test_verifier 713 716 #713/p ARG_CONST_MAP_PTR: null pointer OK #714/p ARG_CONST_MAP_PTR: valid map pointer OK #715/p ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map OK #716/p ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map OK Summary: 4 PASSED, 0 SKIPPED, 0 FAILED Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> Acked-by: John Fastabend <john.fastabend@gmail.com> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- tools/testing/selftests/bpf/test_verifier.c | 22 +++++- .../testing/selftests/bpf/verifier/map_ptr.c | 70 +++++++++++++++++++ 2 files changed, 91 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c index 58b5a349d3ba..8b474177dab0 100644 --- a/tools/testing/selftests/bpf/test_verifier.c +++ b/tools/testing/selftests/bpf/test_verifier.c @@ -50,7 +50,7 @@ #define MAX_INSNS BPF_MAXINSNS #define MAX_TEST_INSNS 1000000 #define MAX_FIXUPS 8 -#define MAX_NR_MAPS 21 +#define MAX_NR_MAPS 23 #define MAX_TEST_RUNS 8 #define POINTER_VALUE 0xcafe4all #define TEST_DATA_LEN 64 @@ -88,6 +88,8 @@ struct bpf_test { int fixup_map_event_output[MAX_FIXUPS]; int fixup_map_reuseport_array[MAX_FIXUPS]; int fixup_map_ringbuf[MAX_FIXUPS]; + int fixup_map_devmap[MAX_FIXUPS]; + int fixup_map_devmap_hash[MAX_FIXUPS]; /* Expected verifier log output for result REJECT or VERBOSE_ACCEPT. * Can be a tab-separated sequence of expected strings. An empty string * means no log verification. @@ -718,6 +720,8 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, int *fixup_map_event_output = test->fixup_map_event_output; int *fixup_map_reuseport_array = test->fixup_map_reuseport_array; int *fixup_map_ringbuf = test->fixup_map_ringbuf; + int *fixup_map_devmap = test->fixup_map_devmap; + int *fixup_map_devmap_hash = test->fixup_map_devmap_hash; if (test->fill_helper) { test->fill_insns = calloc(MAX_TEST_INSNS, sizeof(struct bpf_insn)); @@ -903,6 +907,22 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, fixup_map_ringbuf++; } while (*fixup_map_ringbuf); } + if (*fixup_map_devmap) { + map_fds[20] = __create_map(BPF_MAP_TYPE_DEVMAP, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap].imm = map_fds[20]; + fixup_map_devmap++; + } while (*fixup_map_devmap); + } + if (*fixup_map_devmap_hash) { + map_fds[21] = __create_map(BPF_MAP_TYPE_DEVMAP_HASH, + sizeof(u32), sizeof(u32), 1, 0); + do { + prog[*fixup_map_devmap_hash].imm = map_fds[21]; + fixup_map_devmap_hash++; + } while (*fixup_map_devmap_hash); + } } struct libcap { diff --git a/tools/testing/selftests/bpf/verifier/map_ptr.c b/tools/testing/selftests/bpf/verifier/map_ptr.c index b117bdd3806d..1a532198c9c1 100644 --- a/tools/testing/selftests/bpf/verifier/map_ptr.c +++ b/tools/testing/selftests/bpf/verifier/map_ptr.c @@ -93,3 +93,73 @@ .fixup_map_hash_16b = { 4 }, .result = ACCEPT, }, +{ + "ARG_CONST_MAP_PTR: null pointer", + .insns = { + /* bpf_redirect_map arg1 (map) */ + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .result = REJECT, + .prog_type = BPF_PROG_TYPE_XDP, + .errstr = "R1 type=inv expected=map_ptr", +}, +{ + "ARG_CONST_MAP_PTR: valid map pointer", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map arg1 (map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map arg2 (ifindex) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: null pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, +{ + "ARG_CONST_MAP_PTR_OR_NULL: valid map pointer for ex_map", + .insns = { + BPF_MOV64_IMM(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg1 (in_map) */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* bpf_redirect_map_multi arg2 (ex_map) */ + BPF_LD_MAP_FD(BPF_REG_2, 1), + /* bpf_redirect_map_multi arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_redirect_map_multi), + BPF_EXIT_INSN(), + }, + .fixup_map_devmap = { 1 }, + .fixup_map_devmap_hash = { 3 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + .retval = 4, +}, -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
* [PATCHv18 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test 2021-02-04 14:03 ` [PATCHv18 " Hangbin Liu ` (4 preceding siblings ...) 2021-02-04 14:03 ` [PATCHv18 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu @ 2021-02-04 14:03 ` Hangbin Liu 5 siblings, 0 replies; 219+ messages in thread From: Hangbin Liu @ 2021-02-04 14:03 UTC (permalink / raw) To: bpf Cc: netdev, Toke Høiland-Jørgensen, Jiri Benc, Jesper Dangaard Brouer, Eelco Chaudron, ast, Daniel Borkmann, Lorenzo Bianconi, David Ahern, Andrii Nakryiko, Alexei Starovoitov, John Fastabend, Maciej Fijalkowski, Hangbin Liu Add a bpf selftest for new helper xdp_redirect_map_multi(). In this test there are 3 forward groups and 1 exclude group. The test will redirect each interface's packets to all the interfaces in the forward group, and exclude the interface in exclude map. Two maps (DEVMAP, DEVMAP_HASH) and two xdp modes (generic, drive) will be tested. XDP egress program will also be tested by setting pkt src MAC to egress interface's MAC address. For more test details, you can find it in the test script. Here is the test result. ]# ./test_xdp_redirect_multi.sh Pass: xdpgeneric arp ns1-2 Pass: xdpgeneric arp ns1-3 Pass: xdpgeneric arp ns1-4 Pass: xdpgeneric ping ns1-2 Pass: xdpgeneric ping ns1-3 Pass: xdpgeneric ping ns1-4 Pass: xdpgeneric ping6 ns2-1 Pass: xdpgeneric ping6 ns2-3 Pass: xdpgeneric ping6 ns2-4 Pass: xdpdrv arp ns1-2 Pass: xdpdrv arp ns1-3 Pass: xdpdrv arp ns1-4 Pass: xdpdrv ping ns1-2 Pass: xdpdrv ping ns1-3 Pass: xdpdrv ping ns1-4 Pass: xdpdrv ping6 ns2-1 Pass: xdpdrv ping6 ns2-3 Pass: xdpdrv ping6 ns2-4 Pass: xdpegress mac ns1-2 Pass: xdpegress mac ns1-3 Pass: xdpegress mac ns1-4 Pass: xdpegress ping ns1-2 Pass: xdpegress ping ns1-3 Pass: xdpegress ping ns1-4 Summary: PASS 24, FAIL 0 Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> Acked-by: John Fastabend <john.fastabend@gmail.com> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> v16-v18: no update v15: use bpf_object__find_program_by_name instead of bpf_object__find_program_by_title v14: no update, only rebase the code v13: remove setrlimit v12: add devmap prog test on egress v9: use NULL directly for arg2 and redefine the maps with btf format --- tools/testing/selftests/bpf/Makefile | 3 +- .../bpf/progs/xdp_redirect_multi_kern.c | 111 ++++++++ .../selftests/bpf/test_xdp_redirect_multi.sh | 208 +++++++++++++++ .../selftests/bpf/xdp_redirect_multi.c | 252 ++++++++++++++++++ 4 files changed, 573 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c create mode 100755 tools/testing/selftests/bpf/test_xdp_redirect_multi.sh create mode 100644 tools/testing/selftests/bpf/xdp_redirect_multi.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index f0674d406f40..7e730498bf70 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -49,6 +49,7 @@ TEST_FILES = xsk_prereqs.sh \ # Order correspond to 'make run_tests' order TEST_PROGS := test_kmod.sh \ test_xdp_redirect.sh \ + test_xdp_redirect_multi.sh \ test_xdp_meta.sh \ test_xdp_veth.sh \ test_offload.py \ @@ -78,7 +79,7 @@ TEST_PROGS_EXTENDED := with_addr.sh \ TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ test_lirc_mode2_user xdping test_cpp runqslower bench bpf_testmod.ko \ - xdpxceiver + xdpxceiver xdp_redirect_multi TEST_CUSTOM_PROGS = $(OUTPUT)/urandom_read diff --git a/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c new file mode 100644 index 000000000000..dce4df40d9de --- /dev/null +++ b/tools/testing/selftests/bpf/progs/xdp_redirect_multi_kern.c @@ -0,0 +1,111 @@ +// SPDX-License-Identifier: GPL-2.0 +#define KBUILD_MODNAME "foo" +#include <string.h> +#include <linux/in.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ip.h> +#include <linux/ipv6.h> + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_endian.h> + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v4 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_v6 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} forward_map_all SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(struct bpf_devmap_val)); + __uint(max_entries, 128); +} forward_map_egress SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_DEVMAP_HASH); + __uint(key_size, sizeof(int)); + __uint(value_size, sizeof(int)); + __uint(max_entries, 128); +} exclude_map SEC(".maps"); + +/* map to store egress interfaces mac addresses */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, __u32); + __type(value, __be64); + __uint(max_entries, 128); +} mac_map SEC(".maps"); + +SEC("xdp_redirect_map_multi") +int xdp_redirect_map_multi_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + struct ethhdr *eth = data; + __u16 h_proto; + __u64 nh_off; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + h_proto = eth->h_proto; + + if (h_proto == bpf_htons(ETH_P_IP)) + return bpf_redirect_map_multi(&forward_map_v4, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else if (h_proto == bpf_htons(ETH_P_IPV6)) + return bpf_redirect_map_multi(&forward_map_v6, &exclude_map, + BPF_F_EXCLUDE_INGRESS); + else + return bpf_redirect_map_multi(&forward_map_all, NULL, + BPF_F_EXCLUDE_INGRESS); +} + +/* The following 2 progs are for 2nd devmap prog testing */ +SEC("xdp_redirect_map_ingress") +int xdp_redirect_map_all_prog(struct xdp_md *ctx) +{ + return bpf_redirect_map_multi(&forward_map_egress, NULL, BPF_F_EXCLUDE_INGRESS); +} + +SEC("xdp_devmap/map_prog") +int xdp_devmap_prog(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + __u32 key = ctx->egress_ifindex; + struct ethhdr *eth = data; + __u64 nh_off; + __be64 *mac; + + nh_off = sizeof(*eth); + if (data + nh_off > data_end) + return XDP_DROP; + + mac = bpf_map_lookup_elem(&mac_map, &key); + if (mac) + __builtin_memcpy(eth->h_source, mac, ETH_ALEN); + + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh new file mode 100755 index 000000000000..6503751fdca5 --- /dev/null +++ b/tools/testing/selftests/bpf/test_xdp_redirect_multi.sh @@ -0,0 +1,208 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Test topology: +# - - - - - - - - - - - - - - - - - - - - - - - - - +# | veth1 veth2 veth3 veth4 | ... init net +# - -| - - - - - - | - - - - - - | - - - - - - | - - +# --------- --------- --------- --------- +# | veth0 | | veth0 | | veth0 | | veth0 | ... +# --------- --------- --------- --------- +# ns1 ns2 ns3 ns4 +# +# Forward maps: +# Forward map_all has interfaces: veth1, veth2, veth3, veth4, ... (All traffic except IPv4, IPv6) +# Forward map_v4 has interfaces: veth1, veth3, veth4, ... (For IPv4 traffic only) +# Forward map_v6 has interfaces: veth2, veth3, veth4, ... (For IPv6 traffic only) +# Forward map_egress has all interfaces and redirect all pkts +# Exclude Groups: +# Exclude map: veth3 (assume ns3 is in black list) +# Map type: +# map_v4 use DEVMAP, others use DEVMAP_HASH +# +# Test modules: +# XDP modes: generic, native, native + egress_prog +# +# Test cases: +# ARP(we didn't block ARP for ns3): +# ns1 -> gw: ns2, ns3, ns4 should receive the arp request +# IPv4: +# ns1 -> ns2 (block), ns1 -> ns3 (block), ns1 -> ns4 (pass) +# IPv6 +# ns2 -> ns1 (block), ns2 -> ns3 (block), ns2 -> ns4 (pass) +# egress_prog: +# all ping test should pass, the src mac should be egress interface's mac +# + + +# netns numbers +NUM=4 +IFACES="" +DRV_MODE="xdpgeneric xdpdrv xdpegress" +PASS=0 +FAIL=0 + +test_pass() +{ + echo "Pass: $@" + PASS=$((PASS + 1)) +} + +test_fail() +{ + echo "fail: $@" + FAIL=$((FAIL + 1)) +} + +clean_up() +{ + for i in $(seq $NUM); do + ip link del veth$i 2> /dev/null + ip netns del ns$i 2> /dev/null + done +} + +# Kselftest framework requirement - SKIP code is 4. +check_env() +{ + ip link set dev lo xdpgeneric off &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without the ip xdpgeneric support" + exit 4 + fi + + which tcpdump &>/dev/null + if [ $? -ne 0 ];then + echo "selftests: [SKIP] Could not run test without tcpdump" + exit 4 + fi +} + +setup_ns() +{ + local mode=$1 + IFACES="" + + if [ "$mode" = "xdpegress" ]; then + mode="xdpdrv" + fi + + for i in $(seq $NUM); do + ip netns add ns$i + ip link add veth$i type veth peer name veth0 netns ns$i + ip link set veth$i up + ip -n ns$i link set veth0 up + + ip -n ns$i addr add 192.0.2.$i/24 dev veth0 + ip -n ns$i addr add 2001:db8::$i/64 dev veth0 + ip -n ns$i link set veth0 $mode obj \ + xdp_dummy.o sec xdp_dummy &> /dev/null || \ + { test_fail "Unable to load dummy xdp" && exit 1; } + IFACES="$IFACES veth$i" + veth_mac[$i]=$(ip link show veth$i | awk '/link\/ether/ {print $2}') + done +} + +do_egress_tests() +{ + local mode=$1 + + # mac test + ip netns exec ns2 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + + # mac check + grep -q "${veth_mac[2]} > ff:ff:ff:ff:ff:ff" mac_ns1-2_${mode}.log && \ + test_pass "$mode mac ns1-2" || test_fail "$mode mac ns1-2" + grep -q "${veth_mac[3]} > ff:ff:ff:ff:ff:ff" mac_ns1-3_${mode}.log && \ + test_pass "$mode mac ns1-3" || test_fail "$mode mac ns1-3" + grep -q "${veth_mac[4]} > ff:ff:ff:ff:ff:ff" mac_ns1-4_${mode}.log && \ + test_pass "$mode mac ns1-4" || test_fail "$mode mac ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-2" || test_fail "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-3" || test_fail "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" +} + +do_ping_tests() +{ + local mode=$1 + + # arp test + ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> arp_ns1-2_${mode}.log & + ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> arp_ns1-3_${mode}.log & + ip netns exec ns4 tcpdump -i veth0 -nn -l -e &> arp_ns1-4_${mode}.log & + ip netns exec ns1 ping 192.0.2.254 -c 4 &> /dev/null + sleep 2 + pkill -9 tcpdump + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-2_${mode}.log && \ + test_pass "$mode arp ns1-2" || test_fail "$mode arp ns1-2" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-3_${mode}.log && \ + test_pass "$mode arp ns1-3" || test_fail "$mode arp ns1-3" + grep -q "Request who-has 192.0.2.254 tell 192.0.2.1" arp_ns1-4_${mode}.log && \ + test_pass "$mode arp ns1-4" || test_fail "$mode arp ns1-4" + + # ping test + ip netns exec ns1 ping 192.0.2.2 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-2" || test_pass "$mode ping ns1-2" + ip netns exec ns1 ping 192.0.2.3 -c 4 &> /dev/null && \ + test_fail "$mode ping ns1-3" || test_pass "$mode ping ns1-3" + ip netns exec ns1 ping 192.0.2.4 -c 4 &> /dev/null && \ + test_pass "$mode ping ns1-4" || test_fail "$mode ping ns1-4" + + # ping6 test + ip netns exec ns2 ping6 2001:db8::1 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-1" || test_pass "$mode ping6 ns2-1" + ip netns exec ns2 ping6 2001:db8::3 -c 4 &> /dev/null && \ + test_fail "$mode ping6 ns2-3" || test_pass "$mode ping6 ns2-3" + ip netns exec ns2 ping6 2001:db8::4 -c 4 &> /dev/null && \ + test_pass "$mode ping6 ns2-4" || test_fail "$mode ping6 ns2-4" +} + +do_tests() +{ + local mode=$1 + local drv_p + + case ${mode} in + xdpdrv) drv_p="-N";; + xdpegress) drv_p="-X";; + xdpgeneric) drv_p="-S";; + esac + + ./xdp_redirect_multi $drv_p $IFACES &> xdp_redirect_${mode}.log & + xdp_pid=$! + sleep 10 + + if [ "$mode" = "xdpegress" ]; then + do_egress_tests $mode + else + do_ping_tests $mode + fi + + kill $xdp_pid +} + +trap clean_up 0 2 3 6 9 + +check_env +rm -f xdp_redirect_*.log arp_ns*.log mac_ns*.log + +for mode in ${DRV_MODE}; do + setup_ns $mode + do_tests $mode + sleep 10 + clean_up + sleep 5 +done + +echo "Summary: PASS $PASS, FAIL $FAIL" +[ $FAIL -eq 0 ] && exit 0 || exit 1 diff --git a/tools/testing/selftests/bpf/xdp_redirect_multi.c b/tools/testing/selftests/bpf/xdp_redirect_multi.c new file mode 100644 index 000000000000..b43cd3c9eefd --- /dev/null +++ b/tools/testing/selftests/bpf/xdp_redirect_multi.c @@ -0,0 +1,252 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/if_link.h> +#include <assert.h> +#include <errno.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <net/if.h> +#include <unistd.h> +#include <libgen.h> +#include <sys/resource.h> +#include <sys/ioctl.h> +#include <sys/types.h> +#include <sys/socket.h> +#include <netinet/in.h> + +#include "bpf_util.h" +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#define MAX_IFACE_NUM 32 + +static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST; +static int ifaces[MAX_IFACE_NUM] = {}; + +static void int_exit(int sig) +{ + __u32 prog_id = 0; + int i; + + for (i = 0; ifaces[i] > 0; i++) { + if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) { + printf("bpf_get_link_xdp_id failed\n"); + exit(1); + } + if (prog_id) + bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags); + } + + exit(0); +} + +static int get_mac_addr(unsigned int ifindex, void *mac_addr) +{ + char ifname[IF_NAMESIZE]; + struct ifreq ifr; + int fd, ret = -1; + + fd = socket(AF_INET, SOCK_DGRAM, 0); + if (fd < 0) + return ret; + + if (!if_indextoname(ifindex, ifname)) + goto err_out; + + strcpy(ifr.ifr_name, ifname); + + if (ioctl(fd, SIOCGIFHWADDR, &ifr) != 0) + goto err_out; + + memcpy(mac_addr, ifr.ifr_hwaddr.sa_data, 6 * sizeof(char)); + ret = 0; + +err_out: + close(fd); + return ret; +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n" + "OPTS:\n" + " -S use skb-mode\n" + " -N enforce native mode\n" + " -F force loading prog\n" + " -X load xdp program on egress\n", + prog); +} + +int main(int argc, char **argv) +{ + int prog_fd, group_all, group_v4, group_v6, exclude, mac_map; + struct bpf_program *ingress_prog, *egress_prog; + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_UNSPEC, + }; + int i, ret, opt, egress_prog_fd = 0; + struct bpf_devmap_val devmap_val; + bool attach_egress_prog = false; + unsigned char mac_addr[6]; + char ifname[IF_NAMESIZE]; + struct bpf_object *obj; + unsigned int ifindex; + char filename[256]; + + while ((opt = getopt(argc, argv, "SNFX")) != -1) { + switch (opt) { + case 'S': + xdp_flags |= XDP_FLAGS_SKB_MODE; + break; + case 'N': + /* default, set below */ + break; + case 'F': + xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST; + break; + case 'X': + attach_egress_prog = true; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) { + xdp_flags |= XDP_FLAGS_DRV_MODE; + } else if (attach_egress_prog) { + printf("Load xdp program on egress with SKB mode not supported yet\n"); + goto err_out; + } + + if (optind == argc) { + printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]); + goto err_out; + } + + printf("Get interfaces"); + for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) { + ifaces[i] = if_nametoindex(argv[optind + i]); + if (!ifaces[i]) + ifaces[i] = strtoul(argv[optind + i], NULL, 0); + if (!if_indextoname(ifaces[i], ifname)) { + perror("Invalid interface name or i"); + goto err_out; + } + printf(" %d", ifaces[i]); + } + printf("\n"); + + snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]); + prog_load_attr.file = filename; + + if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd)) + goto err_out; + + if (attach_egress_prog) + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_egress"); + else + group_all = bpf_object__find_map_fd_by_name(obj, "forward_map_all"); + group_v4 = bpf_object__find_map_fd_by_name(obj, "forward_map_v4"); + group_v6 = bpf_object__find_map_fd_by_name(obj, "forward_map_v6"); + exclude = bpf_object__find_map_fd_by_name(obj, "exclude_map"); + mac_map = bpf_object__find_map_fd_by_name(obj, "mac_map"); + + if (group_all < 0 || group_v4 < 0 || group_v6 < 0 || exclude < 0 || + mac_map < 0) { + printf("bpf_object__find_map_fd_by_name failed\n"); + goto err_out; + } + + if (attach_egress_prog) { + /* Find ingress/egress prog for 2nd xdp prog */ + ingress_prog = bpf_object__find_program_by_name(obj, "xdp_redirect_map_all_prog"); + egress_prog = bpf_object__find_program_by_name(obj, "xdp_devmap_prog"); + if (!ingress_prog || !egress_prog) { + printf("finding ingress/egress_prog in obj file failed\n"); + goto err_out; + } + prog_fd = bpf_program__fd(ingress_prog); + egress_prog_fd = bpf_program__fd(egress_prog); + if (prog_fd < 0 || egress_prog_fd < 0) { + printf("find egress_prog fd failed\n"); + goto err_out; + } + } + + signal(SIGINT, int_exit); + signal(SIGTERM, int_exit); + + /* Init forward multicast groups and exclude group */ + for (i = 0; ifaces[i] > 0; i++) { + ifindex = ifaces[i]; + + if (attach_egress_prog) { + ret = get_mac_addr(ifindex, mac_addr); + if (ret < 0) { + printf("get interface %d mac failed\n", ifindex); + goto err_out; + } + ret = bpf_map_update_elem(mac_map, &ifindex, mac_addr, 0); + if (ret) { + perror("bpf_update_elem mac_map failed\n"); + goto err_out; + } + } + + /* Add all the interfaces to group all */ + devmap_val.ifindex = ifindex; + devmap_val.bpf_prog.fd = egress_prog_fd; + ret = bpf_map_update_elem(group_all, &ifindex, &devmap_val, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + + /* For testing: remove the 1st interfaces from group v6 */ + if (i != 0) { + ret = bpf_map_update_elem(group_v6, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: remove the 2nd interfaces from group v4 */ + if (i != 1) { + ret = bpf_map_update_elem(group_v4, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* For testing: add the 3rd interfaces to exclude map */ + if (i == 2) { + ret = bpf_map_update_elem(exclude, &ifindex, &ifindex, 0); + if (ret) { + perror("bpf_map_update_elem"); + goto err_out; + } + } + + /* bind prog_fd to each interface */ + ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags); + if (ret) { + printf("Set xdp fd failed on %d\n", ifindex); + goto err_out; + } + } + + /* sleep some time for testing */ + sleep(999); + + return 0; + +err_out: + return 1; +} -- 2.26.2 ^ permalink raw reply related [flat|nested] 219+ messages in thread
end of thread, other threads:[~2021-02-05 3:09 UTC | newest] Thread overview: 219+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-04-15 8:54 [RFC PATCH bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu 2020-04-15 8:54 ` [RFC PATCH bpf-next 1/2] " Hangbin Liu 2020-04-20 9:52 ` Hangbin Liu 2020-04-15 8:54 ` [RFC PATCH bpf-next 2/2] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2020-04-24 8:56 ` [RFC PATCHv2 bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu 2020-04-24 8:56 ` [RFC PATCHv2 bpf-next 1/2] xdp: add a new helper for " Hangbin Liu 2020-04-24 14:19 ` Lorenzo Bianconi 2020-04-28 11:09 ` Eelco Chaudron 2020-05-06 9:35 ` Hangbin Liu 2020-04-24 14:34 ` Toke Høiland-Jørgensen 2020-05-06 9:14 ` Hangbin Liu 2020-05-06 10:00 ` Toke Høiland-Jørgensen 2020-05-08 8:53 ` Hangbin Liu 2020-05-08 14:58 ` Toke Høiland-Jørgensen 2020-05-18 8:45 ` Hangbin Liu 2020-05-19 10:15 ` Jesper Dangaard Brouer 2020-05-20 1:24 ` Hangbin Liu 2020-04-24 8:56 ` [RFC PATCHv2 bpf-next 2/2] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2020-04-24 14:21 ` Lorenzo Bianconi 2020-05-23 6:05 ` [PATCHv3 bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu 2020-05-23 6:05 ` [PATCHv3 bpf-next 1/2] xdp: add a new helper for " Hangbin Liu 2020-05-26 7:34 ` kbuild test robot 2020-05-23 6:05 ` [PATCHv3 bpf-next 2/2] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2020-05-26 14:05 ` [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support Hangbin Liu 2020-05-26 14:05 ` [PATCHv4 bpf-next 1/2] xdp: add a new helper for " Hangbin Liu 2020-05-27 10:29 ` Toke Høiland-Jørgensen 2020-06-10 10:18 ` Jesper Dangaard Brouer 2020-06-12 8:54 ` Hangbin Liu 2020-06-16 8:55 ` Jesper Dangaard Brouer 2020-06-16 10:11 ` Hangbin Liu 2020-06-16 14:38 ` Jesper Dangaard Brouer 2020-06-10 10:21 ` Jesper Dangaard Brouer 2020-06-10 10:29 ` Toke Høiland-Jørgensen 2020-06-16 9:04 ` Jesper Dangaard Brouer 2020-05-26 14:05 ` [PATCHv4 bpf-next 2/2] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2020-05-27 10:21 ` [PATCHv4 bpf-next 0/2] xdp: add dev map multicast support Toke Høiland-Jørgensen 2020-05-27 10:32 ` Eelco Chaudron 2020-05-27 12:38 ` Hangbin Liu 2020-05-27 15:04 ` Toke Høiland-Jørgensen 2020-06-16 9:09 ` Jesper Dangaard Brouer 2020-06-16 9:47 ` Hangbin Liu 2020-06-03 2:40 ` Hangbin Liu 2020-06-03 11:05 ` Toke Høiland-Jørgensen 2020-06-04 4:09 ` Hangbin Liu 2020-06-04 9:44 ` Toke Høiland-Jørgensen 2020-06-04 12:12 ` Hangbin Liu 2020-06-04 12:37 ` Toke Høiland-Jørgensen 2020-06-04 14:41 ` Hangbin Liu 2020-06-04 16:02 ` Toke Høiland-Jørgensen 2020-06-05 6:26 ` Hangbin Liu 2020-06-08 15:32 ` Toke Høiland-Jørgensen 2020-06-09 3:03 ` Hangbin Liu 2020-06-09 20:31 ` Toke Høiland-Jørgensen 2020-06-10 2:35 ` Hangbin Liu 2020-06-10 10:03 ` Jesper Dangaard Brouer 2020-07-01 4:19 ` [PATCHv5 bpf-next 0/3] xdp: add a new helper for " Hangbin Liu 2020-07-01 4:19 ` [PATCHv5 bpf-next 1/3] " Hangbin Liu 2020-07-01 5:09 ` Andrii Nakryiko 2020-07-01 6:51 ` Hangbin Liu 2020-07-01 18:33 ` kernel test robot 2020-07-01 4:19 ` [PATCHv5 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2020-07-01 4:19 ` [PATCHv5 bpf-next 3/3] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2020-07-09 1:30 ` [PATCHv6 bpf-next 0/3] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-07-09 1:30 ` [PATCHv6 bpf-next 1/3] " Hangbin Liu 2020-07-09 16:33 ` David Ahern 2020-07-10 6:55 ` Hangbin Liu 2020-07-10 13:46 ` David Ahern 2020-07-11 0:26 ` Hangbin Liu 2020-07-11 16:09 ` David Ahern 2020-07-09 1:30 ` [PATCHv6 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2020-07-09 22:40 ` Daniel Borkmann 2020-07-10 6:41 ` Hangbin Liu 2020-07-10 14:32 ` Daniel Borkmann 2020-07-11 0:21 ` Hangbin Liu 2020-07-09 1:30 ` [PATCHv6 bpf-next 3/3] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2020-07-09 22:37 ` [PATCHv6 bpf-next 0/3] xdp: add a new helper for dev map multicast support Daniel Borkmann 2020-07-10 7:36 ` Hangbin Liu 2020-07-10 15:02 ` Daniel Borkmann 2020-07-10 16:52 ` David Ahern 2020-07-14 6:32 ` [PATCHv7 " Hangbin Liu 2020-07-14 6:32 ` [PATCHv7 bpf-next 1/3] " Hangbin Liu 2020-07-14 21:52 ` Toke Høiland-Jørgensen 2020-07-15 12:25 ` Hangbin Liu 2020-07-14 6:32 ` [PATCHv7 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2020-07-14 6:32 ` [PATCHv7 bpf-next 3/3] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2020-07-14 12:29 ` [PATCHv7 bpf-next 0/3] xdp: add a new helper for dev map multicast support Toke Høiland-Jørgensen 2020-07-14 17:12 ` David Ahern 2020-07-14 21:53 ` Toke Høiland-Jørgensen 2020-07-15 12:31 ` Hangbin Liu 2020-07-15 3:45 ` Hangbin Liu 2020-07-15 13:08 ` [PATCHv8 " Hangbin Liu 2020-07-15 13:08 ` [PATCHv8 bpf-next 1/3] " Hangbin Liu 2020-07-15 13:08 ` [PATCHv8 bpf-next 2/3] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2020-07-15 13:08 ` [PATCHv8 bpf-next 3/3] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2020-08-26 13:19 ` [PATCHv9 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-08-26 13:19 ` [PATCHv9 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2020-08-28 21:56 ` Daniel Borkmann 2020-09-03 9:15 ` Hangbin Liu 2020-08-26 13:19 ` [PATCHv9 bpf-next 2/5] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-08-28 22:05 ` Daniel Borkmann 2020-08-26 13:20 ` [PATCHv9 bpf-next 3/5] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2020-08-26 13:20 ` [PATCHv9 bpf-next 4/5] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2020-08-26 13:20 ` [PATCHv9 bpf-next 5/5] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2020-09-03 10:26 ` [PATCHv10 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-09-03 10:26 ` [PATCHv10 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2020-09-04 14:26 ` Daniel Borkmann 2020-09-03 10:26 ` [PATCHv10 bpf-next 2/5] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-09-04 14:22 ` Daniel Borkmann 2020-09-03 10:26 ` [PATCHv10 bpf-next 3/5] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2020-09-03 10:27 ` [PATCHv10 bpf-next 4/5] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2020-09-03 10:27 ` [PATCHv10 bpf-next 5/5] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2020-09-07 8:27 ` [PATCHv11 bpf-next 0/5] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-09-07 8:27 ` [PATCHv11 bpf-next 1/5] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2020-09-07 8:27 ` [PATCHv11 bpf-next 2/5] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-09-09 21:52 ` Alexei Starovoitov 2020-09-10 2:35 ` Hangbin Liu 2020-09-10 3:30 ` David Ahern 2020-09-10 5:35 ` Alexei Starovoitov 2020-09-10 9:44 ` Toke Høiland-Jørgensen 2020-09-10 15:39 ` Alexei Starovoitov 2020-09-10 17:50 ` Jesper Dangaard Brouer 2020-09-10 18:35 ` David Ahern 2020-09-11 7:58 ` Jesper Dangaard Brouer 2020-09-15 16:12 ` David Ahern 2020-09-07 8:27 ` [PATCHv11 bpf-next 3/5] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2020-09-07 8:27 ` [PATCHv11 bpf-next 4/5] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2020-09-07 8:27 ` [PATCHv11 bpf-next 5/5] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2020-12-16 14:30 ` [PATCHv12 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-12-16 14:30 ` [PATCHv12 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu 2020-12-17 16:07 ` David Ahern 2020-12-18 9:43 ` Hangbin Liu 2020-12-21 8:38 ` Hangbin Liu 2020-12-16 14:30 ` [PATCHv12 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2020-12-16 14:30 ` [PATCHv12 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-12-16 14:30 ` [PATCHv12 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2020-12-16 14:30 ` [PATCHv12 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2020-12-16 14:30 ` [PATCHv12 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2020-12-21 12:34 ` [PATCHv13 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-12-21 12:35 ` [PATCHv13 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu 2020-12-21 12:35 ` [PATCHv13 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2020-12-21 12:35 ` [PATCHv13 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2020-12-21 12:35 ` [PATCHv13 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2020-12-21 12:35 ` [PATCHv13 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2020-12-21 12:35 ` [PATCHv13 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2021-01-14 14:23 ` [PATCHv14 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2021-01-14 14:23 ` [PATCHv14 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu 2021-01-17 22:57 ` John Fastabend 2021-01-18 10:07 ` Hangbin Liu 2021-01-21 14:33 ` Jesper Dangaard Brouer 2021-01-14 14:23 ` [PATCHv14 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2021-01-14 14:23 ` [PATCHv14 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2021-01-18 0:10 ` John Fastabend 2021-01-18 8:44 ` Hangbin Liu 2021-01-18 10:47 ` Toke Høiland-Jørgensen 2021-01-18 15:14 ` John Fastabend 2021-01-14 14:23 ` [PATCHv14 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2021-01-14 14:23 ` [PATCHv14 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2021-01-14 14:23 ` [PATCHv14 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2021-01-20 2:25 ` [PATCHv15 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2021-01-20 2:25 ` [PATCHv15 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu 2021-01-20 22:42 ` Maciej Fijalkowski 2021-01-21 3:54 ` Hangbin Liu 2021-01-21 13:35 ` Maciej Fijalkowski 2021-01-20 2:25 ` [PATCHv15 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2021-01-20 2:25 ` [PATCHv15 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2021-01-20 2:25 ` [PATCHv15 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2021-01-20 2:25 ` [PATCHv15 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2021-01-20 2:25 ` [PATCHv15 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2021-01-22 7:46 ` [PATCHv16 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2021-01-22 7:46 ` [PATCHv16 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu 2021-01-22 10:50 ` Maciej Fijalkowski 2021-01-22 13:38 ` Toke Høiland-Jørgensen 2021-01-25 3:30 ` Hangbin Liu 2021-01-25 11:21 ` Toke Høiland-Jørgensen 2021-01-25 12:29 ` Maciej Fijalkowski 2021-01-22 7:46 ` [PATCHv16 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2021-01-22 7:46 ` [PATCHv16 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2021-01-25 12:27 ` Maciej Fijalkowski 2021-01-26 1:17 ` Hangbin Liu 2021-01-27 22:13 ` John Fastabend 2021-01-22 7:46 ` [PATCHv16 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2021-01-22 7:46 ` [PATCHv16 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2021-01-22 7:46 ` [PATCHv16 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2021-01-22 13:43 ` [PATCHv16 bpf-next 0/6] xdp: add a new helper for dev map multicast support Toke Høiland-Jørgensen 2021-01-25 12:45 ` [PATCHv17 " Hangbin Liu 2021-01-25 12:45 ` [PATCHv17 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu 2021-01-27 7:37 ` John Fastabend 2021-01-27 9:41 ` Toke Høiland-Jørgensen 2021-01-27 12:20 ` Maciej Fijalkowski 2021-01-27 15:00 ` Jesper Dangaard Brouer 2021-01-27 15:52 ` John Fastabend 2021-01-27 16:05 ` Toke Høiland-Jørgensen 2021-01-27 15:58 ` Jesper Dangaard Brouer 2021-01-25 12:45 ` [PATCHv17 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2021-01-27 21:50 ` John Fastabend 2021-01-25 12:45 ` [PATCHv17 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2021-01-25 12:45 ` [PATCHv17 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2021-01-25 12:45 ` [PATCHv17 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2021-01-27 22:24 ` John Fastabend 2021-01-28 3:53 ` Hangbin Liu 2021-01-28 23:14 ` John Fastabend 2021-01-25 12:45 ` [PATCHv17 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu 2021-01-28 23:37 ` John Fastabend 2021-01-29 1:46 ` Hangbin Liu 2021-02-04 0:14 ` [PATCHv17 bpf-next 0/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2021-02-04 2:53 ` John Fastabend 2021-02-04 3:12 ` Hangbin Liu 2021-02-04 11:00 ` Toke Høiland-Jørgensen 2021-02-04 12:09 ` Maciej Fijalkowski 2021-02-04 13:33 ` Hangbin Liu 2021-02-04 17:03 ` Jakub Kicinski 2021-02-05 3:07 ` Hangbin Liu 2021-02-04 14:03 ` [PATCHv18 " Hangbin Liu 2021-02-04 14:03 ` [PATCHv18 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue Hangbin Liu 2021-02-04 14:03 ` [PATCHv18 bpf-next 2/6] bpf: add a new bpf argument type ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2021-02-04 14:03 ` [PATCHv18 bpf-next 3/6] xdp: add a new helper for dev map multicast support Hangbin Liu 2021-02-04 14:03 ` [PATCHv18 bpf-next 4/6] sample/bpf: add xdp_redirect_map_multicast test Hangbin Liu 2021-02-04 14:03 ` [PATCHv18 bpf-next 5/6] selftests/bpf: Add verifier tests for bpf arg ARG_CONST_MAP_PTR_OR_NULL Hangbin Liu 2021-02-04 14:03 ` [PATCHv18 bpf-next 6/6] selftests/bpf: add xdp_redirect_multi test Hangbin Liu
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).
