* [PATCH net-next 0/2] tcp: remove two indirect calls from xmit path
@ 2020-06-19 19:12 Eric Dumazet
2020-06-19 19:12 ` [PATCH net-next 1/2] tcp: remove indirect calls for icsk->icsk_af_ops->queue_xmit Eric Dumazet
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: Eric Dumazet @ 2020-06-19 19:12 UTC (permalink / raw)
To: David S . Miller; +Cc: netdev, Eric Dumazet, Eric Dumazet, Jakub Kicinski
__tcp_transmit_skb() does two indirect calls per packet, lets get rid
of them.
Eric Dumazet (2):
tcp: remove indirect calls for icsk->icsk_af_ops->queue_xmit
tcp: remove indirect calls for icsk->icsk_af_ops->send_check
include/net/ip.h | 6 +-----
include/net/ip6_checksum.h | 9 ---------
include/net/tcp.h | 4 ++++
net/ipv4/ip_output.c | 6 ++++++
net/ipv4/tcp_output.c | 12 ++++++++++--
net/ipv6/tcp_ipv6.c | 7 +++++++
6 files changed, 28 insertions(+), 16 deletions(-)
--
2.27.0.111.gc72c7da667-goog
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH net-next 1/2] tcp: remove indirect calls for icsk->icsk_af_ops->queue_xmit
2020-06-19 19:12 [PATCH net-next 0/2] tcp: remove two indirect calls from xmit path Eric Dumazet
@ 2020-06-19 19:12 ` Eric Dumazet
2020-06-19 19:12 ` [PATCH net-next 2/2] tcp: remove indirect calls for icsk->icsk_af_ops->send_check Eric Dumazet
2020-06-21 0:48 ` [PATCH net-next 0/2] tcp: remove two indirect calls from xmit path David Miller
2 siblings, 0 replies; 4+ messages in thread
From: Eric Dumazet @ 2020-06-19 19:12 UTC (permalink / raw)
To: David S . Miller; +Cc: netdev, Eric Dumazet, Eric Dumazet, Jakub Kicinski
Mitigate RETPOLINE costs in __tcp_transmit_skb()
by using INDIRECT_CALL_INET() wrapper.
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
include/net/ip.h | 6 +-----
include/net/tcp.h | 1 +
net/ipv4/ip_output.c | 6 ++++++
net/ipv4/tcp_output.c | 7 ++++++-
4 files changed, 14 insertions(+), 6 deletions(-)
diff --git a/include/net/ip.h b/include/net/ip.h
index 04ebe7bf54c6a8b21cf7894691b36e564ef1aef1..862c9545833a95e25fdcddafcfc84cd07421835d 100644
--- a/include/net/ip.h
+++ b/include/net/ip.h
@@ -231,11 +231,7 @@ struct sk_buff *ip_make_skb(struct sock *sk, struct flowi4 *fl4,
struct ipcm_cookie *ipc, struct rtable **rtp,
struct inet_cork *cork, unsigned int flags);
-static inline int ip_queue_xmit(struct sock *sk, struct sk_buff *skb,
- struct flowi *fl)
-{
- return __ip_queue_xmit(sk, skb, fl, inet_sk(sk)->tos);
-}
+int ip_queue_xmit(struct sock *sk, struct sk_buff *skb, struct flowi *fl);
static inline struct sk_buff *ip_finish_skb(struct sock *sk, struct flowi4 *fl4)
{
diff --git a/include/net/tcp.h b/include/net/tcp.h
index 4de9485f73d97d0b44f1e423bcbc8a3598749adc..e5d7e0b099245cf245a5f1c994d164a9fff66124 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -25,6 +25,7 @@
#include <linux/skbuff.h>
#include <linux/kref.h>
#include <linux/ktime.h>
+#include <linux/indirect_call_wrapper.h>
#include <net/inet_connection_sock.h>
#include <net/inet_timewait_sock.h>
diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
index 090d3097ee15baa87695c530278761fb26534ad5..d946356187eddfb0f81a2f632ee863fafad5f89c 100644
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -539,6 +539,12 @@ int __ip_queue_xmit(struct sock *sk, struct sk_buff *skb, struct flowi *fl,
}
EXPORT_SYMBOL(__ip_queue_xmit);
+int ip_queue_xmit(struct sock *sk, struct sk_buff *skb, struct flowi *fl)
+{
+ return __ip_queue_xmit(sk, skb, fl, inet_sk(sk)->tos);
+}
+EXPORT_SYMBOL(ip_queue_xmit);
+
static void ip_copy_metadata(struct sk_buff *to, struct sk_buff *from)
{
to->pkt_type = from->pkt_type;
diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index a50e1990a845a258d4cc6a2a989d09068ea3a973..be1bd37185d82e5827dfc5105ae74cd815ba1877 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -1064,6 +1064,9 @@ static void tcp_update_skb_after_send(struct sock *sk, struct sk_buff *skb,
list_move_tail(&skb->tcp_tsorted_anchor, &tp->tsorted_sent_queue);
}
+INDIRECT_CALLABLE_DECLARE(int ip_queue_xmit(struct sock *sk, struct sk_buff *skb, struct flowi *fl));
+INDIRECT_CALLABLE_DECLARE(int inet6_csk_xmit(struct sock *sk, struct sk_buff *skb, struct flowi *fl));
+
/* This routine actually transmits TCP packets queued in by
* tcp_do_sendmsg(). This is used by both the initial
* transmission and possible later retransmissions.
@@ -1235,7 +1238,9 @@ static int __tcp_transmit_skb(struct sock *sk, struct sk_buff *skb,
tcp_add_tx_delay(skb, tp);
- err = icsk->icsk_af_ops->queue_xmit(sk, skb, &inet->cork.fl);
+ err = INDIRECT_CALL_INET(icsk->icsk_af_ops->queue_xmit,
+ inet6_csk_xmit, ip_queue_xmit,
+ sk, skb, &inet->cork.fl);
if (unlikely(err > 0)) {
tcp_enter_cwr(sk);
--
2.27.0.111.gc72c7da667-goog
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH net-next 2/2] tcp: remove indirect calls for icsk->icsk_af_ops->send_check
2020-06-19 19:12 [PATCH net-next 0/2] tcp: remove two indirect calls from xmit path Eric Dumazet
2020-06-19 19:12 ` [PATCH net-next 1/2] tcp: remove indirect calls for icsk->icsk_af_ops->queue_xmit Eric Dumazet
@ 2020-06-19 19:12 ` Eric Dumazet
2020-06-21 0:48 ` [PATCH net-next 0/2] tcp: remove two indirect calls from xmit path David Miller
2 siblings, 0 replies; 4+ messages in thread
From: Eric Dumazet @ 2020-06-19 19:12 UTC (permalink / raw)
To: David S . Miller; +Cc: netdev, Eric Dumazet, Eric Dumazet, Jakub Kicinski
Mitigate RETPOLINE costs in __tcp_transmit_skb()
by using INDIRECT_CALL_INET() wrapper.
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
include/net/ip6_checksum.h | 9 ---------
include/net/tcp.h | 3 +++
net/ipv4/tcp_output.c | 5 ++++-
net/ipv6/tcp_ipv6.c | 7 +++++++
4 files changed, 14 insertions(+), 10 deletions(-)
diff --git a/include/net/ip6_checksum.h b/include/net/ip6_checksum.h
index 27ec612cd4a4598dc01e4984e61bc42c6ee5e77b..b3f4eaa88672a2e64ec3fbb3e77a60fe383e59d9 100644
--- a/include/net/ip6_checksum.h
+++ b/include/net/ip6_checksum.h
@@ -85,15 +85,6 @@ static inline void tcp_v6_gso_csum_prep(struct sk_buff *skb)
th->check = ~tcp_v6_check(0, &ipv6h->saddr, &ipv6h->daddr, 0);
}
-#if IS_ENABLED(CONFIG_IPV6)
-static inline void tcp_v6_send_check(struct sock *sk, struct sk_buff *skb)
-{
- struct ipv6_pinfo *np = inet6_sk(sk);
-
- __tcp_v6_send_check(skb, &np->saddr, &sk->sk_v6_daddr);
-}
-#endif
-
static inline __sum16 udp_v6_check(int len,
const struct in6_addr *saddr,
const struct in6_addr *daddr,
diff --git a/include/net/tcp.h b/include/net/tcp.h
index e5d7e0b099245cf245a5f1c994d164a9fff66124..cd9cc348dbf9c62efa30d909f23a0ed1b39e4492 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -932,6 +932,9 @@ static inline int tcp_v6_sdif(const struct sk_buff *skb)
#endif
return 0;
}
+
+INDIRECT_CALLABLE_DECLARE(void tcp_v6_send_check(struct sock *sk, struct sk_buff *skb));
+
#endif
static inline bool inet_exact_dif_match(struct net *net, struct sk_buff *skb)
diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index be1bd37185d82e5827dfc5105ae74cd815ba1877..04b70fe31fa2cdad739ca49447cf4dd7e9d90cce 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -1066,6 +1066,7 @@ static void tcp_update_skb_after_send(struct sock *sk, struct sk_buff *skb,
INDIRECT_CALLABLE_DECLARE(int ip_queue_xmit(struct sock *sk, struct sk_buff *skb, struct flowi *fl));
INDIRECT_CALLABLE_DECLARE(int inet6_csk_xmit(struct sock *sk, struct sk_buff *skb, struct flowi *fl));
+INDIRECT_CALLABLE_DECLARE(void tcp_v4_send_check(struct sock *sk, struct sk_buff *skb));
/* This routine actually transmits TCP packets queued in by
* tcp_do_sendmsg(). This is used by both the initial
@@ -1210,7 +1211,9 @@ static int __tcp_transmit_skb(struct sock *sk, struct sk_buff *skb,
}
#endif
- icsk->icsk_af_ops->send_check(sk, skb);
+ INDIRECT_CALL_INET(icsk->icsk_af_ops->send_check,
+ tcp_v6_send_check, tcp_v4_send_check,
+ sk, skb);
if (likely(tcb->tcp_flags & TCPHDR_ACK))
tcp_event_ack_sent(sk, tcp_skb_pcount(skb), rcv_nxt);
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index f67d45ff00b468f8c50053f6d2e430a1f5247db5..4502db706f75349b49672a859492a036a8722bb6 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -1811,6 +1811,13 @@ static struct timewait_sock_ops tcp6_timewait_sock_ops = {
.twsk_destructor = tcp_twsk_destructor,
};
+INDIRECT_CALLABLE_SCOPE void tcp_v6_send_check(struct sock *sk, struct sk_buff *skb)
+{
+ struct ipv6_pinfo *np = inet6_sk(sk);
+
+ __tcp_v6_send_check(skb, &np->saddr, &sk->sk_v6_daddr);
+}
+
const struct inet_connection_sock_af_ops ipv6_specific = {
.queue_xmit = inet6_csk_xmit,
.send_check = tcp_v6_send_check,
--
2.27.0.111.gc72c7da667-goog
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH net-next 0/2] tcp: remove two indirect calls from xmit path
2020-06-19 19:12 [PATCH net-next 0/2] tcp: remove two indirect calls from xmit path Eric Dumazet
2020-06-19 19:12 ` [PATCH net-next 1/2] tcp: remove indirect calls for icsk->icsk_af_ops->queue_xmit Eric Dumazet
2020-06-19 19:12 ` [PATCH net-next 2/2] tcp: remove indirect calls for icsk->icsk_af_ops->send_check Eric Dumazet
@ 2020-06-21 0:48 ` David Miller
2 siblings, 0 replies; 4+ messages in thread
From: David Miller @ 2020-06-21 0:48 UTC (permalink / raw)
To: edumazet; +Cc: netdev, eric.dumazet, kuba
From: Eric Dumazet <edumazet@google.com>
Date: Fri, 19 Jun 2020 12:12:33 -0700
> __tcp_transmit_skb() does two indirect calls per packet, lets get rid
> of them.
Series applied, thanks.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-06-21 0:48 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-19 19:12 [PATCH net-next 0/2] tcp: remove two indirect calls from xmit path Eric Dumazet
2020-06-19 19:12 ` [PATCH net-next 1/2] tcp: remove indirect calls for icsk->icsk_af_ops->queue_xmit Eric Dumazet
2020-06-19 19:12 ` [PATCH net-next 2/2] tcp: remove indirect calls for icsk->icsk_af_ops->send_check Eric Dumazet
2020-06-21 0:48 ` [PATCH net-next 0/2] tcp: remove two indirect calls from xmit path David Miller
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).