netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH net-next 0/6] skbuff: micro-optimize flow dissection
@ 2021-03-12 19:46 Alexander Lobakin
  2021-03-12 19:46 ` [PATCH net-next 1/6] flow_dissector: constify bpf_flow_dissector's data pointers Alexander Lobakin
                   ` (5 more replies)
  0 siblings, 6 replies; 7+ messages in thread
From: Alexander Lobakin @ 2021-03-12 19:46 UTC (permalink / raw)
  To: David S. Miller, Jakub Kicinski
  Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
	Martin KaFai Lau, Song Liu, Yonghong Song, John Fastabend,
	KP Singh, Jonathan Lemon, Alexander Lobakin, Eric Dumazet,
	Willem de Bruijn, Kevin Hao, Pablo Neira Ayuso, Jakub Sitnicki,
	Marco Elver, Dexuan Cui, Vladimir Oltean, Ariel Levkovich,
	Wang Qing, Davide Caratti, Guillaume Nault, Eran Ben Elisha,
	Mauro Carvalho Chehab, Kirill Tkhai, Bartosz Golaszewski, netdev,
	linux-kernel, bpf

This little number makes all of the flow dissection functions take
raw input data pointer as const (1-5) and shuffles the branches in
__skb_header_pointer() according to their hit probability.

The result is +20 Mbps per flow/core with one Flow Dissector pass
per packet. This affects RPS (with software hashing), drivers that
use eth_get_headlen() on their Rx path and so on.

Alexander Lobakin (6):
  flow_dissector: constify bpf_flow_dissector's data pointers
  skbuff: make __skb_header_pointer()'s data argument const
  flow_dissector: constify raw input @data argument
  linux/etherdevice.h: misc trailing whitespace cleanup
  ethernet: constify eth_get_headlen()'s @data argument
  skbuff: micro-optimize {,__}skb_header_pointer()

 include/linux/etherdevice.h  |  4 ++--
 include/linux/skbuff.h       | 26 +++++++++++------------
 include/net/flow_dissector.h |  6 +++---
 net/core/flow_dissector.c    | 41 +++++++++++++++++++-----------------
 net/ethernet/eth.c           |  2 +-
 5 files changed, 40 insertions(+), 39 deletions(-)

--
2.30.2



^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH net-next 1/6] flow_dissector: constify bpf_flow_dissector's data pointers
  2021-03-12 19:46 [PATCH net-next 0/6] skbuff: micro-optimize flow dissection Alexander Lobakin
@ 2021-03-12 19:46 ` Alexander Lobakin
  2021-03-12 19:46 ` [PATCH net-next 2/6] skbuff: make __skb_header_pointer()'s data argument const Alexander Lobakin
                   ` (4 subsequent siblings)
  5 siblings, 0 replies; 7+ messages in thread
From: Alexander Lobakin @ 2021-03-12 19:46 UTC (permalink / raw)
  To: David S. Miller, Jakub Kicinski
  Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
	Martin KaFai Lau, Song Liu, Yonghong Song, John Fastabend,
	KP Singh, Jonathan Lemon, Alexander Lobakin, Eric Dumazet,
	Willem de Bruijn, Kevin Hao, Pablo Neira Ayuso, Jakub Sitnicki,
	Marco Elver, Dexuan Cui, Vladimir Oltean, Ariel Levkovich,
	Wang Qing, Davide Caratti, Guillaume Nault, Eran Ben Elisha,
	Mauro Carvalho Chehab, Kirill Tkhai, Bartosz Golaszewski, netdev,
	linux-kernel, bpf

BPF Flow dissection programs are read-only and don't touch input
buffers.
Mark @data and @data_end in struct bpf_flow_dissector as const in
preparation for global input constifying.

Signed-off-by: Alexander Lobakin <alobakin@pm.me>
---
 include/net/flow_dissector.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h
index cc10b10dc3a1..bf00e71816ed 100644
--- a/include/net/flow_dissector.h
+++ b/include/net/flow_dissector.h
@@ -368,8 +368,8 @@ static inline void *skb_flow_dissector_target(struct flow_dissector *flow_dissec
 struct bpf_flow_dissector {
 	struct bpf_flow_keys	*flow_keys;
 	const struct sk_buff	*skb;
-	void			*data;
-	void			*data_end;
+	const void		*data;
+	const void		*data_end;
 };

 static inline void
--
2.30.2



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH net-next 2/6] skbuff: make __skb_header_pointer()'s data argument const
  2021-03-12 19:46 [PATCH net-next 0/6] skbuff: micro-optimize flow dissection Alexander Lobakin
  2021-03-12 19:46 ` [PATCH net-next 1/6] flow_dissector: constify bpf_flow_dissector's data pointers Alexander Lobakin
@ 2021-03-12 19:46 ` Alexander Lobakin
  2021-03-12 19:46 ` [PATCH net-next 3/6] flow_dissector: constify raw input @data argument Alexander Lobakin
                   ` (3 subsequent siblings)
  5 siblings, 0 replies; 7+ messages in thread
From: Alexander Lobakin @ 2021-03-12 19:46 UTC (permalink / raw)
  To: David S. Miller, Jakub Kicinski
  Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
	Martin KaFai Lau, Song Liu, Yonghong Song, John Fastabend,
	KP Singh, Jonathan Lemon, Alexander Lobakin, Eric Dumazet,
	Willem de Bruijn, Kevin Hao, Pablo Neira Ayuso, Jakub Sitnicki,
	Marco Elver, Dexuan Cui, Vladimir Oltean, Ariel Levkovich,
	Wang Qing, Davide Caratti, Guillaume Nault, Eran Ben Elisha,
	Mauro Carvalho Chehab, Kirill Tkhai, Bartosz Golaszewski, netdev,
	linux-kernel, bpf

The function never modifies the input buffer, so @data argument
can be marked as const.
This implies one harmless cast-away.

Signed-off-by: Alexander Lobakin <alobakin@pm.me>
---
 include/linux/skbuff.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 0503c917d773..d93ab74063e5 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -3678,11 +3678,11 @@ __wsum skb_checksum(const struct sk_buff *skb, int offset, int len,
 		    __wsum csum);

 static inline void * __must_check
-__skb_header_pointer(const struct sk_buff *skb, int offset,
-		     int len, void *data, int hlen, void *buffer)
+__skb_header_pointer(const struct sk_buff *skb, int offset, int len,
+		     const void *data, int hlen, void *buffer)
 {
 	if (hlen - offset >= len)
-		return data + offset;
+		return (void *)data + offset;

 	if (!skb ||
 	    skb_copy_bits(skb, offset, buffer, len) < 0)
--
2.30.2



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH net-next 3/6] flow_dissector: constify raw input @data argument
  2021-03-12 19:46 [PATCH net-next 0/6] skbuff: micro-optimize flow dissection Alexander Lobakin
  2021-03-12 19:46 ` [PATCH net-next 1/6] flow_dissector: constify bpf_flow_dissector's data pointers Alexander Lobakin
  2021-03-12 19:46 ` [PATCH net-next 2/6] skbuff: make __skb_header_pointer()'s data argument const Alexander Lobakin
@ 2021-03-12 19:46 ` Alexander Lobakin
  2021-03-12 19:46 ` [PATCH net-next 4/6] linux/etherdevice.h: misc trailing whitespace cleanup Alexander Lobakin
                   ` (2 subsequent siblings)
  5 siblings, 0 replies; 7+ messages in thread
From: Alexander Lobakin @ 2021-03-12 19:46 UTC (permalink / raw)
  To: David S. Miller, Jakub Kicinski
  Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
	Martin KaFai Lau, Song Liu, Yonghong Song, John Fastabend,
	KP Singh, Jonathan Lemon, Alexander Lobakin, Eric Dumazet,
	Willem de Bruijn, Kevin Hao, Pablo Neira Ayuso, Jakub Sitnicki,
	Marco Elver, Dexuan Cui, Vladimir Oltean, Ariel Levkovich,
	Wang Qing, Davide Caratti, Guillaume Nault, Eran Ben Elisha,
	Mauro Carvalho Chehab, Kirill Tkhai, Bartosz Golaszewski, netdev,
	linux-kernel, bpf

Flow Dissector code never modifies the input buffer, neither skb nor
raw data.
Make @data argument const for all of the Flow dissector's functions.

Signed-off-by: Alexander Lobakin <alobakin@pm.me>
---
 include/linux/skbuff.h       | 15 ++++++-------
 include/net/flow_dissector.h |  2 +-
 net/core/flow_dissector.c    | 41 +++++++++++++++++++-----------------
 3 files changed, 30 insertions(+), 28 deletions(-)

diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index d93ab74063e5..7873f24c0ae5 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -1292,10 +1292,10 @@ __skb_set_sw_hash(struct sk_buff *skb, __u32 hash, bool is_l4)
 void __skb_get_hash(struct sk_buff *skb);
 u32 __skb_get_hash_symmetric(const struct sk_buff *skb);
 u32 skb_get_poff(const struct sk_buff *skb);
-u32 __skb_get_poff(const struct sk_buff *skb, void *data,
+u32 __skb_get_poff(const struct sk_buff *skb, const void *data,
 		   const struct flow_keys_basic *keys, int hlen);
 __be32 __skb_flow_get_ports(const struct sk_buff *skb, int thoff, u8 ip_proto,
-			    void *data, int hlen_proto);
+			    const void *data, int hlen_proto);

 static inline __be32 skb_flow_get_ports(const struct sk_buff *skb,
 					int thoff, u8 ip_proto)
@@ -1314,9 +1314,8 @@ bool bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx,
 bool __skb_flow_dissect(const struct net *net,
 			const struct sk_buff *skb,
 			struct flow_dissector *flow_dissector,
-			void *target_container,
-			void *data, __be16 proto, int nhoff, int hlen,
-			unsigned int flags);
+			void *target_container, const void *data,
+			__be16 proto, int nhoff, int hlen, unsigned int flags);

 static inline bool skb_flow_dissect(const struct sk_buff *skb,
 				    struct flow_dissector *flow_dissector,
@@ -1338,9 +1337,9 @@ static inline bool skb_flow_dissect_flow_keys(const struct sk_buff *skb,
 static inline bool
 skb_flow_dissect_flow_keys_basic(const struct net *net,
 				 const struct sk_buff *skb,
-				 struct flow_keys_basic *flow, void *data,
-				 __be16 proto, int nhoff, int hlen,
-				 unsigned int flags)
+				 struct flow_keys_basic *flow,
+				 const void *data, __be16 proto,
+				 int nhoff, int hlen, unsigned int flags)
 {
 	memset(flow, 0, sizeof(*flow));
 	return __skb_flow_dissect(net, skb, &flow_keys_basic_dissector, flow,
diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h
index bf00e71816ed..ffd386ea0dbb 100644
--- a/include/net/flow_dissector.h
+++ b/include/net/flow_dissector.h
@@ -350,7 +350,7 @@ static inline bool flow_keys_have_l4(const struct flow_keys *keys)
 u32 flow_hash_from_keys(struct flow_keys *keys);
 void skb_flow_get_icmp_tci(const struct sk_buff *skb,
 			   struct flow_dissector_key_icmp *key_icmp,
-			   void *data, int thoff, int hlen);
+			   const void *data, int thoff, int hlen);

 static inline bool dissector_uses_key(const struct flow_dissector *flow_dissector,
 				      enum flow_dissector_key_id key_id)
diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c
index 2ef2224b3bff..2ed380d096ce 100644
--- a/net/core/flow_dissector.c
+++ b/net/core/flow_dissector.c
@@ -114,7 +114,7 @@ int flow_dissector_bpf_prog_attach_check(struct net *net,
  * is the protocol port offset returned from proto_ports_offset
  */
 __be32 __skb_flow_get_ports(const struct sk_buff *skb, int thoff, u8 ip_proto,
-			    void *data, int hlen)
+			    const void *data, int hlen)
 {
 	int poff = proto_ports_offset(ip_proto);

@@ -161,7 +161,7 @@ static bool icmp_has_id(u8 type)
  */
 void skb_flow_get_icmp_tci(const struct sk_buff *skb,
 			   struct flow_dissector_key_icmp *key_icmp,
-			   void *data, int thoff, int hlen)
+			   const void *data, int thoff, int hlen)
 {
 	struct icmphdr *ih, _ih;

@@ -187,8 +187,8 @@ EXPORT_SYMBOL(skb_flow_get_icmp_tci);
  */
 static void __skb_flow_dissect_icmp(const struct sk_buff *skb,
 				    struct flow_dissector *flow_dissector,
-				    void *target_container,
-				    void *data, int thoff, int hlen)
+				    void *target_container, const void *data,
+				    int thoff, int hlen)
 {
 	struct flow_dissector_key_icmp *key_icmp;

@@ -409,8 +409,8 @@ EXPORT_SYMBOL(skb_flow_dissect_hash);
 static enum flow_dissect_ret
 __skb_flow_dissect_mpls(const struct sk_buff *skb,
 			struct flow_dissector *flow_dissector,
-			void *target_container, void *data, int nhoff, int hlen,
-			int lse_index, bool *entropy_label)
+			void *target_container, const void *data, int nhoff,
+			int hlen, int lse_index, bool *entropy_label)
 {
 	struct mpls_label *hdr, _hdr;
 	u32 entry, label, bos;
@@ -467,7 +467,8 @@ __skb_flow_dissect_mpls(const struct sk_buff *skb,
 static enum flow_dissect_ret
 __skb_flow_dissect_arp(const struct sk_buff *skb,
 		       struct flow_dissector *flow_dissector,
-		       void *target_container, void *data, int nhoff, int hlen)
+		       void *target_container, const void *data,
+		       int nhoff, int hlen)
 {
 	struct flow_dissector_key_arp *key_arp;
 	struct {
@@ -523,7 +524,7 @@ static enum flow_dissect_ret
 __skb_flow_dissect_gre(const struct sk_buff *skb,
 		       struct flow_dissector_key_control *key_control,
 		       struct flow_dissector *flow_dissector,
-		       void *target_container, void *data,
+		       void *target_container, const void *data,
 		       __be16 *p_proto, int *p_nhoff, int *p_hlen,
 		       unsigned int flags)
 {
@@ -663,8 +664,8 @@ __skb_flow_dissect_gre(const struct sk_buff *skb,
 static enum flow_dissect_ret
 __skb_flow_dissect_batadv(const struct sk_buff *skb,
 			  struct flow_dissector_key_control *key_control,
-			  void *data, __be16 *p_proto, int *p_nhoff, int hlen,
-			  unsigned int flags)
+			  const void *data, __be16 *p_proto, int *p_nhoff,
+			  int hlen, unsigned int flags)
 {
 	struct {
 		struct batadv_unicast_packet batadv_unicast;
@@ -695,7 +696,8 @@ __skb_flow_dissect_batadv(const struct sk_buff *skb,
 static void
 __skb_flow_dissect_tcp(const struct sk_buff *skb,
 		       struct flow_dissector *flow_dissector,
-		       void *target_container, void *data, int thoff, int hlen)
+		       void *target_container, const void *data,
+		       int thoff, int hlen)
 {
 	struct flow_dissector_key_tcp *key_tcp;
 	struct tcphdr *th, _th;
@@ -719,8 +721,8 @@ __skb_flow_dissect_tcp(const struct sk_buff *skb,
 static void
 __skb_flow_dissect_ports(const struct sk_buff *skb,
 			 struct flow_dissector *flow_dissector,
-			 void *target_container, void *data, int nhoff,
-			 u8 ip_proto, int hlen)
+			 void *target_container, const void *data,
+			 int nhoff, u8 ip_proto, int hlen)
 {
 	enum flow_dissector_key_id dissector_ports = FLOW_DISSECTOR_KEY_MAX;
 	struct flow_dissector_key_ports *key_ports;
@@ -744,7 +746,8 @@ __skb_flow_dissect_ports(const struct sk_buff *skb,
 static void
 __skb_flow_dissect_ipv4(const struct sk_buff *skb,
 			struct flow_dissector *flow_dissector,
-			void *target_container, void *data, const struct iphdr *iph)
+			void *target_container, const void *data,
+			const struct iphdr *iph)
 {
 	struct flow_dissector_key_ip *key_ip;

@@ -761,7 +764,8 @@ __skb_flow_dissect_ipv4(const struct sk_buff *skb,
 static void
 __skb_flow_dissect_ipv6(const struct sk_buff *skb,
 			struct flow_dissector *flow_dissector,
-			void *target_container, void *data, const struct ipv6hdr *iph)
+			void *target_container, const void *data,
+			const struct ipv6hdr *iph)
 {
 	struct flow_dissector_key_ip *key_ip;

@@ -908,9 +912,8 @@ bool bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx,
 bool __skb_flow_dissect(const struct net *net,
 			const struct sk_buff *skb,
 			struct flow_dissector *flow_dissector,
-			void *target_container,
-			void *data, __be16 proto, int nhoff, int hlen,
-			unsigned int flags)
+			void *target_container, const void *data,
+			__be16 proto, int nhoff, int hlen, unsigned int flags)
 {
 	struct flow_dissector_key_control *key_control;
 	struct flow_dissector_key_basic *key_basic;
@@ -1642,7 +1645,7 @@ __u32 skb_get_hash_perturb(const struct sk_buff *skb,
 }
 EXPORT_SYMBOL(skb_get_hash_perturb);

-u32 __skb_get_poff(const struct sk_buff *skb, void *data,
+u32 __skb_get_poff(const struct sk_buff *skb, const void *data,
 		   const struct flow_keys_basic *keys, int hlen)
 {
 	u32 poff = keys->control.thoff;
--
2.30.2



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH net-next 4/6] linux/etherdevice.h: misc trailing whitespace cleanup
  2021-03-12 19:46 [PATCH net-next 0/6] skbuff: micro-optimize flow dissection Alexander Lobakin
                   ` (2 preceding siblings ...)
  2021-03-12 19:46 ` [PATCH net-next 3/6] flow_dissector: constify raw input @data argument Alexander Lobakin
@ 2021-03-12 19:46 ` Alexander Lobakin
  2021-03-12 19:46 ` [PATCH net-next 5/6] ethernet: constify eth_get_headlen()'s @data argument Alexander Lobakin
  2021-03-12 19:47 ` [PATCH net-next 6/6] skbuff: micro-optimize {,__}skb_header_pointer() Alexander Lobakin
  5 siblings, 0 replies; 7+ messages in thread
From: Alexander Lobakin @ 2021-03-12 19:46 UTC (permalink / raw)
  To: David S. Miller, Jakub Kicinski
  Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
	Martin KaFai Lau, Song Liu, Yonghong Song, John Fastabend,
	KP Singh, Jonathan Lemon, Alexander Lobakin, Eric Dumazet,
	Willem de Bruijn, Kevin Hao, Pablo Neira Ayuso, Jakub Sitnicki,
	Marco Elver, Dexuan Cui, Vladimir Oltean, Ariel Levkovich,
	Wang Qing, Davide Caratti, Guillaume Nault, Eran Ben Elisha,
	Mauro Carvalho Chehab, Kirill Tkhai, Bartosz Golaszewski, netdev,
	linux-kernel, bpf

Caught by the text editor. Fix it separately from the actual changes.

Signed-off-by: Alexander Lobakin <alobakin@pm.me>
---
 include/linux/etherdevice.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/etherdevice.h b/include/linux/etherdevice.h
index 2e5debc0373c..bcb2f81baafb 100644
--- a/include/linux/etherdevice.h
+++ b/include/linux/etherdevice.h
@@ -11,7 +11,7 @@
  * Authors:	Ross Biro
  *		Fred N. van Kempen, <waltje@uWalt.NL.Mugnet.ORG>
  *
- *		Relocated to include/linux where it belongs by Alan Cox
+ *		Relocated to include/linux where it belongs by Alan Cox
  *							<gw4pts@gw4pts.ampr.org>
  */
 #ifndef _LINUX_ETHERDEVICE_H
--
2.30.2



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH net-next 5/6] ethernet: constify eth_get_headlen()'s @data argument
  2021-03-12 19:46 [PATCH net-next 0/6] skbuff: micro-optimize flow dissection Alexander Lobakin
                   ` (3 preceding siblings ...)
  2021-03-12 19:46 ` [PATCH net-next 4/6] linux/etherdevice.h: misc trailing whitespace cleanup Alexander Lobakin
@ 2021-03-12 19:46 ` Alexander Lobakin
  2021-03-12 19:47 ` [PATCH net-next 6/6] skbuff: micro-optimize {,__}skb_header_pointer() Alexander Lobakin
  5 siblings, 0 replies; 7+ messages in thread
From: Alexander Lobakin @ 2021-03-12 19:46 UTC (permalink / raw)
  To: David S. Miller, Jakub Kicinski
  Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
	Martin KaFai Lau, Song Liu, Yonghong Song, John Fastabend,
	KP Singh, Jonathan Lemon, Alexander Lobakin, Eric Dumazet,
	Willem de Bruijn, Kevin Hao, Pablo Neira Ayuso, Jakub Sitnicki,
	Marco Elver, Dexuan Cui, Vladimir Oltean, Ariel Levkovich,
	Wang Qing, Davide Caratti, Guillaume Nault, Eran Ben Elisha,
	Mauro Carvalho Chehab, Kirill Tkhai, Bartosz Golaszewski, netdev,
	linux-kernel, bpf

It's used only for flow dissection, which now takes constant data
pointers.

Signed-off-by: Alexander Lobakin <alobakin@pm.me>
---
 include/linux/etherdevice.h | 2 +-
 net/ethernet/eth.c          | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/linux/etherdevice.h b/include/linux/etherdevice.h
index bcb2f81baafb..330345b1be54 100644
--- a/include/linux/etherdevice.h
+++ b/include/linux/etherdevice.h
@@ -29,7 +29,7 @@ struct device;
 int eth_platform_get_mac_address(struct device *dev, u8 *mac_addr);
 unsigned char *arch_get_platform_mac_address(void);
 int nvmem_get_mac_address(struct device *dev, void *addrbuf);
-u32 eth_get_headlen(const struct net_device *dev, void *data, unsigned int len);
+u32 eth_get_headlen(const struct net_device *dev, const void *data, u32 len);
 __be16 eth_type_trans(struct sk_buff *skb, struct net_device *dev);
 extern const struct header_ops eth_header_ops;

diff --git a/net/ethernet/eth.c b/net/ethernet/eth.c
index 4106373180c6..e01cf766d2c5 100644
--- a/net/ethernet/eth.c
+++ b/net/ethernet/eth.c
@@ -122,7 +122,7 @@ EXPORT_SYMBOL(eth_header);
  * Make a best effort attempt to pull the length for all of the headers for
  * a given frame in a linear buffer.
  */
-u32 eth_get_headlen(const struct net_device *dev, void *data, unsigned int len)
+u32 eth_get_headlen(const struct net_device *dev, const void *data, u32 len)
 {
 	const unsigned int flags = FLOW_DISSECTOR_F_PARSE_1ST_FRAG;
 	const struct ethhdr *eth = (const struct ethhdr *)data;
--
2.30.2



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH net-next 6/6] skbuff: micro-optimize {,__}skb_header_pointer()
  2021-03-12 19:46 [PATCH net-next 0/6] skbuff: micro-optimize flow dissection Alexander Lobakin
                   ` (4 preceding siblings ...)
  2021-03-12 19:46 ` [PATCH net-next 5/6] ethernet: constify eth_get_headlen()'s @data argument Alexander Lobakin
@ 2021-03-12 19:47 ` Alexander Lobakin
  5 siblings, 0 replies; 7+ messages in thread
From: Alexander Lobakin @ 2021-03-12 19:47 UTC (permalink / raw)
  To: David S. Miller, Jakub Kicinski
  Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
	Martin KaFai Lau, Song Liu, Yonghong Song, John Fastabend,
	KP Singh, Jonathan Lemon, Alexander Lobakin, Eric Dumazet,
	Willem de Bruijn, Kevin Hao, Pablo Neira Ayuso, Jakub Sitnicki,
	Marco Elver, Dexuan Cui, Vladimir Oltean, Ariel Levkovich,
	Wang Qing, Davide Caratti, Guillaume Nault, Eran Ben Elisha,
	Mauro Carvalho Chehab, Kirill Tkhai, Bartosz Golaszewski, netdev,
	linux-kernel, bpf

{,__}skb_header_pointer() helpers exist mainly for preventing
accesses-beyond-end of the linear data.
In the vast majorify of cases, they bail out on the first condition.
All code going after is mostly a fallback.
Mark the most common branch as 'likely' one to move it in-line.
Also, skb_copy_bits() can return negative values only when the input
arguments are invalid, e.g. offset is greater than skb->len. It can
be safely marked as 'unlikely' branch, assuming that hotpath code
provides sane input to not fail here.

These two bump the throughput with a single Flow Dissector pass on
every packet (e.g. with RPS or driver that uses eth_get_headlen())
on 20 Mbps per flow/core.

Signed-off-by: Alexander Lobakin <alobakin@pm.me>
---
 include/linux/skbuff.h | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 7873f24c0ae5..71f4d609819e 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -3680,11 +3680,10 @@ static inline void * __must_check
 __skb_header_pointer(const struct sk_buff *skb, int offset, int len,
 		     const void *data, int hlen, void *buffer)
 {
-	if (hlen - offset >= len)
+	if (likely(hlen - offset >= len))
 		return (void *)data + offset;

-	if (!skb ||
-	    skb_copy_bits(skb, offset, buffer, len) < 0)
+	if (!skb || unlikely(skb_copy_bits(skb, offset, buffer, len) < 0))
 		return NULL;

 	return buffer;
--
2.30.2



^ permalink raw reply related	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2021-03-12 20:01 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-12 19:46 [PATCH net-next 0/6] skbuff: micro-optimize flow dissection Alexander Lobakin
2021-03-12 19:46 ` [PATCH net-next 1/6] flow_dissector: constify bpf_flow_dissector's data pointers Alexander Lobakin
2021-03-12 19:46 ` [PATCH net-next 2/6] skbuff: make __skb_header_pointer()'s data argument const Alexander Lobakin
2021-03-12 19:46 ` [PATCH net-next 3/6] flow_dissector: constify raw input @data argument Alexander Lobakin
2021-03-12 19:46 ` [PATCH net-next 4/6] linux/etherdevice.h: misc trailing whitespace cleanup Alexander Lobakin
2021-03-12 19:46 ` [PATCH net-next 5/6] ethernet: constify eth_get_headlen()'s @data argument Alexander Lobakin
2021-03-12 19:47 ` [PATCH net-next 6/6] skbuff: micro-optimize {,__}skb_header_pointer() Alexander Lobakin

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).