* [PATCH RFC v2 net-next 1/5] net: Introduce Qdisc backpressure infrastructure
2022-08-22 9:10 ` [PATCH RFC v2 net-next 0/5] " Peilin Ye
@ 2022-08-22 9:11 ` Peilin Ye
2022-08-22 9:12 ` [PATCH RFC v2 net-next 2/5] net/udp: Implement Qdisc backpressure algorithm Peilin Ye
` (5 subsequent siblings)
6 siblings, 0 replies; 29+ messages in thread
From: Peilin Ye @ 2022-08-22 9:11 UTC (permalink / raw)
To: David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jonathan Corbet, Hideaki YOSHIFUJI, David Ahern,
Jamal Hadi Salim, Cong Wang, Jiri Pirko
Cc: Peilin Ye, netdev, linux-doc, linux-kernel, Cong Wang,
Stephen Hemminger, Dave Taht, Peilin Ye
From: Peilin Ye <peilin.ye@bytedance.com>
Currently sockets (especially UDP ones) can drop a lot of traffic at TC
egress when rate limited by shaper Qdiscs like HTB. Improve this by
introducing a Qdisc backpressure infrastructure:
a. A new 'sock struct' field, @sk_overlimits, which keeps track of the
number of bytes in socket send buffer that are currently
unavailable due to TC egress congestion. The size of an overlimit
socket's "effective" send buffer is represented by @sk_sndbuf minus
@sk_overlimits, with a lower limit of SOCK_MIN_SNDBUF:
max(@sk_sndbuf - @sk_overlimits, SOCK_MIN_SNDBUF)
b. A new (*backpressure) 'struct proto' callback, which is the
protocol's private algorithm for Qdisc backpressure.
Working together:
1. When a shaper Qdisc (TBF, HTB, CBQ, etc.) drops a packet that
belongs to a local socket, it calls qdisc_backpressure().
2. qdisc_backpressure() eventually invokes the socket protocol's
(*backpressure) callback, which should increase @sk_overlimits.
3. The transport layer then sees a smaller "effective" send buffer and
will send slower.
4. It is the per-protocol (*backpressure) implementation's
responsibility to decrease @sk_overlimits when TC egress becomes
idle again, potentially by using a timer.
Suggested-by: Cong Wang <cong.wang@bytedance.com>
Signed-off-by: Peilin Ye <peilin.ye@bytedance.com>
---
include/net/sch_generic.h | 11 +++++++++++
include/net/sock.h | 21 +++++++++++++++++++++
net/core/sock.c | 1 +
3 files changed, 33 insertions(+)
diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h
index ec693fe7c553..afdf4bf64936 100644
--- a/include/net/sch_generic.h
+++ b/include/net/sch_generic.h
@@ -19,6 +19,7 @@
#include <net/gen_stats.h>
#include <net/rtnetlink.h>
#include <net/flow_offload.h>
+#include <net/sock.h>
struct Qdisc_ops;
struct qdisc_walker;
@@ -1188,6 +1189,16 @@ static inline int qdisc_drop_all(struct sk_buff *skb, struct Qdisc *sch,
return NET_XMIT_DROP;
}
+static inline void qdisc_backpressure(struct sk_buff *skb)
+{
+ struct sock *sk = skb->sk;
+
+ if (!sk || !sk_fullsock(sk))
+ return;
+
+ sk_backpressure(sk);
+}
+
/* Length to Time (L2T) lookup in a qdisc_rate_table, to determine how
long it will take to send a packet given its size.
*/
diff --git a/include/net/sock.h b/include/net/sock.h
index 05a1bbdf5805..ef10ca66cf26 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -277,6 +277,7 @@ struct sk_filter;
* @sk_pacing_status: Pacing status (requested, handled by sch_fq)
* @sk_max_pacing_rate: Maximum pacing rate (%SO_MAX_PACING_RATE)
* @sk_sndbuf: size of send buffer in bytes
+ * @sk_overlimits: size of temporarily unavailable send buffer in bytes
* @__sk_flags_offset: empty field used to determine location of bitfield
* @sk_padding: unused element for alignment
* @sk_no_check_tx: %SO_NO_CHECK setting, set checksum in TX packets
@@ -439,6 +440,7 @@ struct sock {
struct dst_entry __rcu *sk_dst_cache;
atomic_t sk_omem_alloc;
int sk_sndbuf;
+ int sk_overlimits;
/* ===== cache line for TX ===== */
int sk_wmem_queued;
@@ -1264,6 +1266,7 @@ struct proto {
bool (*stream_memory_free)(const struct sock *sk, int wake);
bool (*sock_is_readable)(struct sock *sk);
+ void (*backpressure)(struct sock *sk);
/* Memory pressure */
void (*enter_memory_pressure)(struct sock *sk);
void (*leave_memory_pressure)(struct sock *sk);
@@ -2499,6 +2502,24 @@ static inline void sk_stream_moderate_sndbuf(struct sock *sk)
WRITE_ONCE(sk->sk_sndbuf, max_t(u32, val, SOCK_MIN_SNDBUF));
}
+static inline int sk_sndbuf_avail(struct sock *sk)
+{
+ int overlimits, sndbuf = READ_ONCE(sk->sk_sndbuf);
+
+ if (!sk->sk_prot->backpressure)
+ return sndbuf;
+
+ overlimits = READ_ONCE(sk->sk_overlimits);
+
+ return max_t(int, sndbuf - overlimits, SOCK_MIN_SNDBUF);
+}
+
+static inline void sk_backpressure(struct sock *sk)
+{
+ if (sk->sk_prot->backpressure)
+ sk->sk_prot->backpressure(sk);
+}
+
/**
* sk_page_frag - return an appropriate page_frag
* @sk: socket
diff --git a/net/core/sock.c b/net/core/sock.c
index 4cb957d934a2..167d471b176f 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -2194,6 +2194,7 @@ struct sock *sk_clone_lock(const struct sock *sk, const gfp_t priority)
/* sk_wmem_alloc set to one (see sk_free() and sock_wfree()) */
refcount_set(&newsk->sk_wmem_alloc, 1);
+ newsk->sk_overlimits = 0;
atomic_set(&newsk->sk_omem_alloc, 0);
sk_init_common(newsk);
--
2.20.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH RFC v2 net-next 2/5] net/udp: Implement Qdisc backpressure algorithm
2022-08-22 9:10 ` [PATCH RFC v2 net-next 0/5] " Peilin Ye
2022-08-22 9:11 ` [PATCH RFC v2 net-next 1/5] net: Introduce " Peilin Ye
@ 2022-08-22 9:12 ` Peilin Ye
2022-08-31 10:42 ` Hillf Danton
2022-08-22 9:12 ` [PATCH RFC v2 net-next 3/5] net/sched: sch_tbf: Use Qdisc backpressure infrastructure Peilin Ye
` (4 subsequent siblings)
6 siblings, 1 reply; 29+ messages in thread
From: Peilin Ye @ 2022-08-22 9:12 UTC (permalink / raw)
To: David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jonathan Corbet, Hideaki YOSHIFUJI, David Ahern,
Jamal Hadi Salim, Cong Wang, Jiri Pirko
Cc: Peilin Ye, netdev, linux-doc, linux-kernel, Cong Wang,
Stephen Hemminger, Dave Taht, Peilin Ye
From: Peilin Ye <peilin.ye@bytedance.com>
Support Qdisc backpressure for UDP (IPv4 and IPv6) sockets by
implementing the (*backpressure) callback:
1. When a shaper Qdisc drops a packet due to TC egress congestion,
halve the effective send buffer [1], then (re)scedule the
backpressure timer.
[1] sndbuf - overlimits_new == 1/2 * (sndbuf - overlimits_old)
2. When the timer expires, double the effective send buffer [2]. If
the socket is still overlimit, reschedule the timer itself.
[2] sndbuf - overlimits_new == 2 * (sndbuf - overlimits_old)
In sock_wait_for_wmem() and sock_alloc_send_pskb(), check the size of
effective send buffer instead, so that overlimit sockets send slower.
See sk_sndbuf_avail().
The timer interval is specified by a new per-net sysctl,
sysctl_udp_backpressure_interval. Default is 100 milliseconds, meaning
that an overlimit UDP socket will try to double its effective send
buffer every 100 milliseconds. Use 0 to disable Qdisc backpressure for
UDP sockets.
Generally, longer interval means lower packet drop rate, but also makes
overlimit sockets slower to recover when TC egress becomes idle (or the
shaper Qdisc gets removed, etc.)
Test results with TBF + SFQ Qdiscs, 500 Mbits/sec rate limit with 16
iperf UDP '-b 1G' clients:
Interval Throughput Drop Rate CPU Usage [3]
0 (disabled) 480.0 Mb/s 96.50% 68.38%
10 ms 486.4 Mb/s 9.28% 1.30%
100 ms 486.4 Mb/s 1.10% 1.11%
1000 ms 486.4 Mb/s 0.13% 0.81%
[3] perf-top, __pv_queued_spin_lock_slowpath()
Signed-off-by: Peilin Ye <peilin.ye@bytedance.com>
---
Documentation/networking/ip-sysctl.rst | 11 ++++
include/linux/udp.h | 3 ++
include/net/netns/ipv4.h | 1 +
include/net/udp.h | 1 +
net/core/sock.c | 4 +-
net/ipv4/sysctl_net_ipv4.c | 7 +++
net/ipv4/udp.c | 69 +++++++++++++++++++++++++-
net/ipv6/udp.c | 2 +-
8 files changed, 94 insertions(+), 4 deletions(-)
diff --git a/Documentation/networking/ip-sysctl.rst b/Documentation/networking/ip-sysctl.rst
index 56cd4ea059b2..a0d8e9518fda 100644
--- a/Documentation/networking/ip-sysctl.rst
+++ b/Documentation/networking/ip-sysctl.rst
@@ -1070,6 +1070,17 @@ udp_rmem_min - INTEGER
udp_wmem_min - INTEGER
UDP does not have tx memory accounting and this tunable has no effect.
+udp_backpressure_interval - INTEGER
+ The time interval (in milliseconds) in which an overlimit UDP socket
+ tries to increase its effective send buffer size, used by Qdisc
+ backpressure. A longer interval typically results in a lower packet
+ drop rate, but also makes it slower for overlimit UDP sockets to
+ recover from backpressure when TC egress becomes idle.
+
+ 0 to disable Qdisc backpressure for UDP sockets.
+
+ Default: 100
+
RAW variables
=============
diff --git a/include/linux/udp.h b/include/linux/udp.h
index 254a2654400f..dd017994738b 100644
--- a/include/linux/udp.h
+++ b/include/linux/udp.h
@@ -86,6 +86,9 @@ struct udp_sock {
/* This field is dirtied by udp_recvmsg() */
int forward_deficit;
+
+ /* Qdisc backpressure timer */
+ struct timer_list backpressure_timer;
};
#define UDP_MAX_SEGMENTS (1 << 6UL)
diff --git a/include/net/netns/ipv4.h b/include/net/netns/ipv4.h
index c7320ef356d9..01f72ddf23e0 100644
--- a/include/net/netns/ipv4.h
+++ b/include/net/netns/ipv4.h
@@ -182,6 +182,7 @@ struct netns_ipv4 {
int sysctl_udp_wmem_min;
int sysctl_udp_rmem_min;
+ int sysctl_udp_backpressure_interval;
u8 sysctl_fib_notify_on_flag_change;
diff --git a/include/net/udp.h b/include/net/udp.h
index 5ee88ddf79c3..82018e58659b 100644
--- a/include/net/udp.h
+++ b/include/net/udp.h
@@ -279,6 +279,7 @@ int udp_init_sock(struct sock *sk);
int udp_pre_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len);
int __udp_disconnect(struct sock *sk, int flags);
int udp_disconnect(struct sock *sk, int flags);
+void udp_backpressure(struct sock *sk);
__poll_t udp_poll(struct file *file, struct socket *sock, poll_table *wait);
struct sk_buff *skb_udp_tunnel_segment(struct sk_buff *skb,
netdev_features_t features,
diff --git a/net/core/sock.c b/net/core/sock.c
index 167d471b176f..cb6ba66f80c8 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -2614,7 +2614,7 @@ static long sock_wait_for_wmem(struct sock *sk, long timeo)
break;
set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
- if (refcount_read(&sk->sk_wmem_alloc) < READ_ONCE(sk->sk_sndbuf))
+ if (refcount_read(&sk->sk_wmem_alloc) < sk_sndbuf_avail(sk))
break;
if (sk->sk_shutdown & SEND_SHUTDOWN)
break;
@@ -2649,7 +2649,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
if (sk->sk_shutdown & SEND_SHUTDOWN)
goto failure;
- if (sk_wmem_alloc_get(sk) < READ_ONCE(sk->sk_sndbuf))
+ if (sk_wmem_alloc_get(sk) < sk_sndbuf_avail(sk))
break;
sk_set_bit(SOCKWQ_ASYNC_NOSPACE, sk);
diff --git a/net/ipv4/sysctl_net_ipv4.c b/net/ipv4/sysctl_net_ipv4.c
index 5490c285668b..1e509a417b92 100644
--- a/net/ipv4/sysctl_net_ipv4.c
+++ b/net/ipv4/sysctl_net_ipv4.c
@@ -1337,6 +1337,13 @@ static struct ctl_table ipv4_net_table[] = {
.proc_handler = proc_dointvec_minmax,
.extra1 = SYSCTL_ONE
},
+ {
+ .procname = "udp_backpressure_interval",
+ .data = &init_net.ipv4.sysctl_udp_backpressure_interval,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec_ms_jiffies,
+ },
{
.procname = "fib_notify_on_flag_change",
.data = &init_net.ipv4.sysctl_fib_notify_on_flag_change,
diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
index 34eda973bbf1..ff58f638c834 100644
--- a/net/ipv4/udp.c
+++ b/net/ipv4/udp.c
@@ -110,6 +110,7 @@
#include <trace/events/skb.h>
#include <net/busy_poll.h>
#include "udp_impl.h"
+#include <net/sock.h>
#include <net/sock_reuseport.h>
#include <net/addrconf.h>
#include <net/udp_tunnel.h>
@@ -1614,10 +1615,73 @@ void udp_destruct_sock(struct sock *sk)
}
EXPORT_SYMBOL_GPL(udp_destruct_sock);
+static inline int udp_backpressure_interval_get(struct sock *sk)
+{
+ return READ_ONCE(sock_net(sk)->ipv4.sysctl_udp_backpressure_interval);
+}
+
+static inline void udp_reset_backpressure_timer(struct sock *sk,
+ unsigned long expires)
+{
+ sk_reset_timer(sk, &udp_sk(sk)->backpressure_timer, expires);
+}
+
+static void udp_backpressure_timer(struct timer_list *t)
+{
+ struct udp_sock *up = from_timer(up, t, backpressure_timer);
+ int interval, sndbuf, overlimits;
+ struct sock *sk = &up->inet.sk;
+
+ interval = udp_backpressure_interval_get(sk);
+ if (!interval) {
+ /* Qdisc backpressure has been turned off */
+ WRITE_ONCE(sk->sk_overlimits, 0);
+ goto out;
+ }
+
+ sndbuf = READ_ONCE(sk->sk_sndbuf);
+ overlimits = READ_ONCE(sk->sk_overlimits);
+
+ /* sndbuf - overlimits_new == 2 * (sndbuf - overlimits_old) */
+ overlimits = min_t(int, overlimits, sndbuf - SOCK_MIN_SNDBUF);
+ overlimits = max_t(int, (2 * overlimits) - sndbuf, 0);
+ WRITE_ONCE(sk->sk_overlimits, overlimits);
+
+ if (overlimits > 0)
+ udp_reset_backpressure_timer(sk, jiffies + interval);
+
+out:
+ sock_put(sk);
+}
+
+void udp_backpressure(struct sock *sk)
+{
+ int interval, sndbuf, overlimits;
+
+ interval = udp_backpressure_interval_get(sk);
+ if (!interval) /* Qdisc backpressure is off */
+ return;
+
+ sndbuf = READ_ONCE(sk->sk_sndbuf);
+ overlimits = READ_ONCE(sk->sk_overlimits);
+
+ /* sndbuf - overlimits_new == 1/2 * (sndbuf - overlimits_old) */
+ overlimits = min_t(int, overlimits, sndbuf - SOCK_MIN_SNDBUF);
+ overlimits += (sndbuf - overlimits) >> 1;
+ WRITE_ONCE(sk->sk_overlimits, overlimits);
+
+ if (overlimits > 0)
+ udp_reset_backpressure_timer(sk, jiffies + interval);
+}
+EXPORT_SYMBOL_GPL(udp_backpressure);
+
int udp_init_sock(struct sock *sk)
{
- skb_queue_head_init(&udp_sk(sk)->reader_queue);
+ struct udp_sock *up = udp_sk(sk);
+
+ skb_queue_head_init(&up->reader_queue);
sk->sk_destruct = udp_destruct_sock;
+ timer_setup(&up->backpressure_timer, udp_backpressure_timer, 0);
return 0;
}
EXPORT_SYMBOL_GPL(udp_init_sock);
@@ -2653,6 +2717,7 @@ void udp_destroy_sock(struct sock *sk)
/* protects from races with udp_abort() */
sock_set_flag(sk, SOCK_DEAD);
udp_flush_pending_frames(sk);
+ sk_stop_timer(sk, &up->backpressure_timer);
unlock_sock_fast(sk, slow);
if (static_branch_unlikely(&udp_encap_needed_key)) {
if (up->encap_type) {
@@ -2946,6 +3011,7 @@ struct proto udp_prot = {
#ifdef CONFIG_BPF_SYSCALL
.psock_update_sk_prot = udp_bpf_update_proto,
#endif
+ .backpressure = udp_backpressure,
.memory_allocated = &udp_memory_allocated,
.per_cpu_fw_alloc = &udp_memory_per_cpu_fw_alloc,
@@ -3268,6 +3334,7 @@ static int __net_init udp_sysctl_init(struct net *net)
{
net->ipv4.sysctl_udp_rmem_min = PAGE_SIZE;
net->ipv4.sysctl_udp_wmem_min = PAGE_SIZE;
+ net->ipv4.sysctl_udp_backpressure_interval = msecs_to_jiffies(100);
#ifdef CONFIG_NET_L3_MASTER_DEV
net->ipv4.sysctl_udp_l3mdev_accept = 0;
diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c
index 16c176e7c69a..106032af6756 100644
--- a/net/ipv6/udp.c
+++ b/net/ipv6/udp.c
@@ -1735,7 +1735,7 @@ struct proto udpv6_prot = {
#ifdef CONFIG_BPF_SYSCALL
.psock_update_sk_prot = udp_bpf_update_proto,
#endif
-
+ .backpressure = udp_backpressure,
.memory_allocated = &udp_memory_allocated,
.per_cpu_fw_alloc = &udp_memory_per_cpu_fw_alloc,
--
2.20.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* Re: [PATCH RFC v2 net-next 2/5] net/udp: Implement Qdisc backpressure algorithm
2022-08-22 9:12 ` [PATCH RFC v2 net-next 2/5] net/udp: Implement Qdisc backpressure algorithm Peilin Ye
@ 2022-08-31 10:42 ` Hillf Danton
0 siblings, 0 replies; 29+ messages in thread
From: Hillf Danton @ 2022-08-31 10:42 UTC (permalink / raw)
To: Peilin Ye; +Cc: Eric Dumazet, Cong Wang, linux-mm, linux-kernel
On Mon, 22 Aug 2022 02:12:20 -0700 Peilin Ye <peilin.ye@bytedance.com> wrote
>
> Support Qdisc backpressure for UDP (IPv4 and IPv6) sockets by
> implementing the (*backpressure) callback:
>
> 1. When a shaper Qdisc drops a packet due to TC egress congestion,
> halve the effective send buffer [1], then (re)scedule the
> backpressure timer.
>
> [1] sndbuf - overlimits_new == 1/2 * (sndbuf - overlimits_old)
>
> 2. When the timer expires, double the effective send buffer [2]. If
> the socket is still overlimit, reschedule the timer itself.
>
> [2] sndbuf - overlimits_new == 2 * (sndbuf - overlimits_old)
>
> In sock_wait_for_wmem() and sock_alloc_send_pskb(), check the size of
> effective send buffer instead, so that overlimit sockets send slower.
> See sk_sndbuf_avail().
Make sense to me.
>
> The timer interval is specified by a new per-net sysctl,
> sysctl_udp_backpressure_interval. Default is 100 milliseconds, meaning
> that an overlimit UDP socket will try to double its effective send
> buffer every 100 milliseconds. Use 0 to disable Qdisc backpressure for
> UDP sockets.
>
> Generally, longer interval means lower packet drop rate, but also makes
> overlimit sockets slower to recover when TC egress becomes idle (or the
> shaper Qdisc gets removed, etc.)
>
> Test results with TBF + SFQ Qdiscs, 500 Mbits/sec rate limit with 16
> iperf UDP '-b 1G' clients:
>
> Interval Throughput Drop Rate CPU Usage [3]
> 0 (disabled) 480.0 Mb/s 96.50% 68.38%
> 10 ms 486.4 Mb/s 9.28% 1.30%
> 100 ms 486.4 Mb/s 1.10% 1.11%
> 1000 ms 486.4 Mb/s 0.13% 0.81%
>
> [3] perf-top, __pv_queued_spin_lock_slowpath()
>
> Signed-off-by: Peilin Ye <peilin.ye@bytedance.com>
> ---
> Documentation/networking/ip-sysctl.rst | 11 ++++
> include/linux/udp.h | 3 ++
> include/net/netns/ipv4.h | 1 +
> include/net/udp.h | 1 +
> net/core/sock.c | 4 +-
> net/ipv4/sysctl_net_ipv4.c | 7 +++
> net/ipv4/udp.c | 69 +++++++++++++++++++++++++-
> net/ipv6/udp.c | 2 +-
> 8 files changed, 94 insertions(+), 4 deletions(-)
>
> diff --git a/Documentation/networking/ip-sysctl.rst b/Documentation/networking/ip-sysctl.rst
> index 56cd4ea059b2..a0d8e9518fda 100644
> --- a/Documentation/networking/ip-sysctl.rst
> +++ b/Documentation/networking/ip-sysctl.rst
> @@ -1070,6 +1070,17 @@ udp_rmem_min - INTEGER
> udp_wmem_min - INTEGER
> UDP does not have tx memory accounting and this tunable has no effect.
>
> +udp_backpressure_interval - INTEGER
> + The time interval (in milliseconds) in which an overlimit UDP socket
> + tries to increase its effective send buffer size, used by Qdisc
> + backpressure. A longer interval typically results in a lower packet
> + drop rate, but also makes it slower for overlimit UDP sockets to
> + recover from backpressure when TC egress becomes idle.
> +
> + 0 to disable Qdisc backpressure for UDP sockets.
> +
> + Default: 100
> +
> RAW variables
> =============
>
> diff --git a/include/linux/udp.h b/include/linux/udp.h
> index 254a2654400f..dd017994738b 100644
> --- a/include/linux/udp.h
> +++ b/include/linux/udp.h
> @@ -86,6 +86,9 @@ struct udp_sock {
>
> /* This field is dirtied by udp_recvmsg() */
> int forward_deficit;
> +
> + /* Qdisc backpressure timer */
> + struct timer_list backpressure_timer;
> };
My $0.02 is s/backpressure_timer/backpressure_ts/
>
> #define UDP_MAX_SEGMENTS (1 << 6UL)
> diff --git a/include/net/netns/ipv4.h b/include/net/netns/ipv4.h
> index c7320ef356d9..01f72ddf23e0 100644
> --- a/include/net/netns/ipv4.h
> +++ b/include/net/netns/ipv4.h
> @@ -182,6 +182,7 @@ struct netns_ipv4 {
>
> int sysctl_udp_wmem_min;
> int sysctl_udp_rmem_min;
> + int sysctl_udp_backpressure_interval;
>
> u8 sysctl_fib_notify_on_flag_change;
>
> diff --git a/include/net/udp.h b/include/net/udp.h
> index 5ee88ddf79c3..82018e58659b 100644
> --- a/include/net/udp.h
> +++ b/include/net/udp.h
> @@ -279,6 +279,7 @@ int udp_init_sock(struct sock *sk);
> int udp_pre_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len);
> int __udp_disconnect(struct sock *sk, int flags);
> int udp_disconnect(struct sock *sk, int flags);
> +void udp_backpressure(struct sock *sk);
> __poll_t udp_poll(struct file *file, struct socket *sock, poll_table *wait);
> struct sk_buff *skb_udp_tunnel_segment(struct sk_buff *skb,
> netdev_features_t features,
> diff --git a/net/core/sock.c b/net/core/sock.c
> index 167d471b176f..cb6ba66f80c8 100644
> --- a/net/core/sock.c
> +++ b/net/core/sock.c
> @@ -2614,7 +2614,7 @@ static long sock_wait_for_wmem(struct sock *sk, long timeo)
> break;
> set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
> prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
> - if (refcount_read(&sk->sk_wmem_alloc) < READ_ONCE(sk->sk_sndbuf))
> + if (refcount_read(&sk->sk_wmem_alloc) < sk_sndbuf_avail(sk))
> break;
> if (sk->sk_shutdown & SEND_SHUTDOWN)
> break;
> @@ -2649,7 +2649,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
> if (sk->sk_shutdown & SEND_SHUTDOWN)
> goto failure;
>
> - if (sk_wmem_alloc_get(sk) < READ_ONCE(sk->sk_sndbuf))
> + if (sk_wmem_alloc_get(sk) < sk_sndbuf_avail(sk))
> break;
>
> sk_set_bit(SOCKWQ_ASYNC_NOSPACE, sk);
> diff --git a/net/ipv4/sysctl_net_ipv4.c b/net/ipv4/sysctl_net_ipv4.c
> index 5490c285668b..1e509a417b92 100644
> --- a/net/ipv4/sysctl_net_ipv4.c
> +++ b/net/ipv4/sysctl_net_ipv4.c
> @@ -1337,6 +1337,13 @@ static struct ctl_table ipv4_net_table[] = {
> .proc_handler = proc_dointvec_minmax,
> .extra1 = SYSCTL_ONE
> },
> + {
> + .procname = "udp_backpressure_interval",
> + .data = &init_net.ipv4.sysctl_udp_backpressure_interval,
> + .maxlen = sizeof(int),
> + .mode = 0644,
> + .proc_handler = proc_dointvec_ms_jiffies,
> + },
> {
> .procname = "fib_notify_on_flag_change",
> .data = &init_net.ipv4.sysctl_fib_notify_on_flag_change,
> diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
> index 34eda973bbf1..ff58f638c834 100644
> --- a/net/ipv4/udp.c
> +++ b/net/ipv4/udp.c
> @@ -110,6 +110,7 @@
> #include <trace/events/skb.h>
> #include <net/busy_poll.h>
> #include "udp_impl.h"
> +#include <net/sock.h>
> #include <net/sock_reuseport.h>
> #include <net/addrconf.h>
> #include <net/udp_tunnel.h>
> @@ -1614,10 +1615,73 @@ void udp_destruct_sock(struct sock *sk)
> }
> EXPORT_SYMBOL_GPL(udp_destruct_sock);
>
> +static inline int udp_backpressure_interval_get(struct sock *sk)
> +{
> + return READ_ONCE(sock_net(sk)->ipv4.sysctl_udp_backpressure_interval);
> +}
> +
> +static inline void udp_reset_backpressure_timer(struct sock *sk,
> + unsigned long expires)
> +{
> + sk_reset_timer(sk, &udp_sk(sk)->backpressure_timer, expires);
> +}
> +
> +static void udp_backpressure_timer(struct timer_list *t)
> +{
> + struct udp_sock *up = from_timer(up, t, backpressure_timer);
> + int interval, sndbuf, overlimits;
> + struct sock *sk = &up->inet.sk;
> +
> + interval = udp_backpressure_interval_get(sk);
> + if (!interval) {
> + /* Qdisc backpressure has been turned off */
> + WRITE_ONCE(sk->sk_overlimits, 0);
> + goto out;
> + }
> +
> + sndbuf = READ_ONCE(sk->sk_sndbuf);
> + overlimits = READ_ONCE(sk->sk_overlimits);
> +
> + /* sndbuf - overlimits_new == 2 * (sndbuf - overlimits_old) */
> + overlimits = min_t(int, overlimits, sndbuf - SOCK_MIN_SNDBUF);
> + overlimits = max_t(int, (2 * overlimits) - sndbuf, 0);
> + WRITE_ONCE(sk->sk_overlimits, overlimits);
> +
> + if (overlimits > 0)
> + udp_reset_backpressure_timer(sk, jiffies + interval);
> +
> +out:
> + sock_put(sk);
> +}
> +
> +void udp_backpressure(struct sock *sk)
> +{
> + int interval, sndbuf, overlimits;
> +
> + interval = udp_backpressure_interval_get(sk);
> + if (!interval) /* Qdisc backpressure is off */
> + return;
> +
> + sndbuf = READ_ONCE(sk->sk_sndbuf);
> + overlimits = READ_ONCE(sk->sk_overlimits);
> +
> + /* sndbuf - overlimits_new == 1/2 * (sndbuf - overlimits_old) */
> + overlimits = min_t(int, overlimits, sndbuf - SOCK_MIN_SNDBUF);
> + overlimits += (sndbuf - overlimits) >> 1;
> + WRITE_ONCE(sk->sk_overlimits, overlimits);
> +
> + if (overlimits > 0)
> + udp_reset_backpressure_timer(sk, jiffies + interval);
> +}
> +EXPORT_SYMBOL_GPL(udp_backpressure);
> +
2, add sndbuf callback in addition to backpressure.
static int udp_sndbuf(struct sock *sk)
{
int interval = udp_backpressure_interval_get(sk);
unsigned long ts = udp_sk(sk)->backpressure_ts;
int sndbuf = READ_ONCE(sk->sk_sndbuf);
unsigned long now = jiffies;
if (!interval || !time_before(now, interval + ts))
return sndbuf;
/* sndbuf = (sndbuf * (now - ts)) / interval; */
if (now - ts > (interval / 2))
sndbuf /= 2;
else
sndbuf /= 4;
return max_t(int, sndbuf, SOCK_MIN_SNDBUF);
}
1, sk_sndbuf_avail() should be in sock.h as it is.
static inline int sk_sndbuf_avail(struct sock *sk)
{
if (sk->sk_prot->backpressure)
return sk->sk_prot->sndbuf(sk);
else
return READ_ONCE(sk->sk_sndbuf);
}
0, add backpressure callback.
static void udp_backpressure(struct sock *sk)
{
if (udp_backpressure_interval_get(sk))
udp_sk(sk)->backpressure_ts = jiffies;
}
Only for thoughts now.
Hillf
> int udp_init_sock(struct sock *sk)
> {
> - skb_queue_head_init(&udp_sk(sk)->reader_queue);
> + struct udp_sock *up = udp_sk(sk);
> +
> + skb_queue_head_init(&up->reader_queue);
> sk->sk_destruct = udp_destruct_sock;
> + timer_setup(&up->backpressure_timer, udp_backpressure_timer, 0);
> return 0;
> }
> EXPORT_SYMBOL_GPL(udp_init_sock);
> @@ -2653,6 +2717,7 @@ void udp_destroy_sock(struct sock *sk)
> /* protects from races with udp_abort() */
> sock_set_flag(sk, SOCK_DEAD);
> udp_flush_pending_frames(sk);
> + sk_stop_timer(sk, &up->backpressure_timer);
> unlock_sock_fast(sk, slow);
> if (static_branch_unlikely(&udp_encap_needed_key)) {
> if (up->encap_type) {
> @@ -2946,6 +3011,7 @@ struct proto udp_prot = {
> #ifdef CONFIG_BPF_SYSCALL
> .psock_update_sk_prot = udp_bpf_update_proto,
> #endif
> + .backpressure = udp_backpressure,
> .memory_allocated = &udp_memory_allocated,
> .per_cpu_fw_alloc = &udp_memory_per_cpu_fw_alloc,
>
> @@ -3268,6 +3334,7 @@ static int __net_init udp_sysctl_init(struct net *net)
> {
> net->ipv4.sysctl_udp_rmem_min = PAGE_SIZE;
> net->ipv4.sysctl_udp_wmem_min = PAGE_SIZE;
> + net->ipv4.sysctl_udp_backpressure_interval = msecs_to_jiffies(100);
>
> #ifdef CONFIG_NET_L3_MASTER_DEV
> net->ipv4.sysctl_udp_l3mdev_accept = 0;
> diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c
> index 16c176e7c69a..106032af6756 100644
> --- a/net/ipv6/udp.c
> +++ b/net/ipv6/udp.c
> @@ -1735,7 +1735,7 @@ struct proto udpv6_prot = {
> #ifdef CONFIG_BPF_SYSCALL
> .psock_update_sk_prot = udp_bpf_update_proto,
> #endif
> -
> + .backpressure = udp_backpressure,
> .memory_allocated = &udp_memory_allocated,
> .per_cpu_fw_alloc = &udp_memory_per_cpu_fw_alloc,
>
> --
> 2.20.1
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH RFC v2 net-next 3/5] net/sched: sch_tbf: Use Qdisc backpressure infrastructure
2022-08-22 9:10 ` [PATCH RFC v2 net-next 0/5] " Peilin Ye
2022-08-22 9:11 ` [PATCH RFC v2 net-next 1/5] net: Introduce " Peilin Ye
2022-08-22 9:12 ` [PATCH RFC v2 net-next 2/5] net/udp: Implement Qdisc backpressure algorithm Peilin Ye
@ 2022-08-22 9:12 ` Peilin Ye
2022-08-22 9:12 ` [PATCH RFC v2 net-next 4/5] net/sched: sch_htb: " Peilin Ye
` (3 subsequent siblings)
6 siblings, 0 replies; 29+ messages in thread
From: Peilin Ye @ 2022-08-22 9:12 UTC (permalink / raw)
To: David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jonathan Corbet, Hideaki YOSHIFUJI, David Ahern,
Jamal Hadi Salim, Cong Wang, Jiri Pirko
Cc: Peilin Ye, netdev, linux-doc, linux-kernel, Cong Wang,
Stephen Hemminger, Dave Taht, Peilin Ye
From: Peilin Ye <peilin.ye@bytedance.com>
Recently we introduced a Qdisc backpressure infrastructure (currently
supports UDP sockets). Use it in TBF Qdisc.
Tested with 500 Mbits/sec rate limit and SFQ inner Qdisc using 16 iperf UDP
1 Gbit/sec clients. Before:
[ 3] 0.0-15.0 sec 53.6 MBytes 30.0 Mbits/sec 0.208 ms 1190234/1228450 (97%)
[ 3] 0.0-15.0 sec 54.7 MBytes 30.6 Mbits/sec 0.085 ms 955591/994593 (96%)
[ 3] 0.0-15.0 sec 55.4 MBytes 31.0 Mbits/sec 0.170 ms 966364/1005868 (96%)
[ 3] 0.0-15.0 sec 55.0 MBytes 30.8 Mbits/sec 0.167 ms 925083/964333 (96%)
<...> ^^^^^^^^^^^^^^^^^^^
Total throughput is 480.2 Mbits/sec and average drop rate is 96.5%.
Now enable Qdisc backpressure for UDP sockets, with
udp_backpressure_interval default to 100 milliseconds:
[ 3] 0.0-15.0 sec 54.4 MBytes 30.4 Mbits/sec 0.097 ms 450/39246 (1.1%)
[ 3] 0.0-15.0 sec 54.4 MBytes 30.4 Mbits/sec 0.331 ms 435/39232 (1.1%)
[ 3] 0.0-15.0 sec 54.4 MBytes 30.4 Mbits/sec 0.040 ms 435/39212 (1.1%)
[ 3] 0.0-15.0 sec 54.4 MBytes 30.4 Mbits/sec 0.031 ms 426/39208 (1.1%)
<...> ^^^^^^^^^^^^^^^^
Total throughput is 486.4 Mbits/sec (1.29% higher) and average drop rate
is 1.1% (98.86% lower).
However, enabling Qdisc backpressure affects fairness between flow if we
use TBF Qdisc with default bfifo inner Qdisc:
[ 3] 0.0-15.0 sec 46.1 MBytes 25.8 Mbits/sec 1.102 ms 142/33048 (0.43%)
[ 3] 0.0-15.0 sec 72.8 MBytes 40.7 Mbits/sec 0.476 ms 145/52081 (0.28%)
[ 3] 0.0-15.0 sec 53.2 MBytes 29.7 Mbits/sec 1.047 ms 141/38086 (0.37%)
[ 3] 0.0-15.0 sec 45.5 MBytes 25.4 Mbits/sec 1.600 ms 141/32573 (0.43%)
<...> ^^^^^^^^^^^^^^^^^
In the test, per-flow throughput ranged from 16.4 to 68.7 Mbits/sec.
However, total throughput was still 486.4 Mbits/sec (0.87% higher than
before), and average drop rate was 0.41% (99.58% lower than before).
Signed-off-by: Peilin Ye <peilin.ye@bytedance.com>
---
net/sched/sch_tbf.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/net/sched/sch_tbf.c b/net/sched/sch_tbf.c
index 72102277449e..cf9cc7dbf078 100644
--- a/net/sched/sch_tbf.c
+++ b/net/sched/sch_tbf.c
@@ -222,6 +222,7 @@ static int tbf_segment(struct sk_buff *skb, struct Qdisc *sch,
len += segs->len;
ret = qdisc_enqueue(segs, q->qdisc, to_free);
if (ret != NET_XMIT_SUCCESS) {
+ qdisc_backpressure(skb);
if (net_xmit_drop_count(ret))
qdisc_qstats_drop(sch);
} else {
@@ -250,6 +251,7 @@ static int tbf_enqueue(struct sk_buff *skb, struct Qdisc *sch,
}
ret = qdisc_enqueue(skb, q->qdisc, to_free);
if (ret != NET_XMIT_SUCCESS) {
+ qdisc_backpressure(skb);
if (net_xmit_drop_count(ret))
qdisc_qstats_drop(sch);
return ret;
--
2.20.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH RFC v2 net-next 4/5] net/sched: sch_htb: Use Qdisc backpressure infrastructure
2022-08-22 9:10 ` [PATCH RFC v2 net-next 0/5] " Peilin Ye
` (2 preceding siblings ...)
2022-08-22 9:12 ` [PATCH RFC v2 net-next 3/5] net/sched: sch_tbf: Use Qdisc backpressure infrastructure Peilin Ye
@ 2022-08-22 9:12 ` Peilin Ye
2022-08-22 9:12 ` [PATCH RFC v2 net-next 5/5] net/sched: sch_cbq: " Peilin Ye
` (2 subsequent siblings)
6 siblings, 0 replies; 29+ messages in thread
From: Peilin Ye @ 2022-08-22 9:12 UTC (permalink / raw)
To: David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jonathan Corbet, Hideaki YOSHIFUJI, David Ahern,
Jamal Hadi Salim, Cong Wang, Jiri Pirko
Cc: Peilin Ye, netdev, linux-doc, linux-kernel, Cong Wang,
Stephen Hemminger, Dave Taht, Peilin Ye
From: Peilin Ye <peilin.ye@bytedance.com>
Recently we introduced a Qdisc backpressure infrastructure (currently
supports UDP sockets). Use it in HTB Qdisc.
Tested with 500 Mbits/sec rate limit using 16 iperf UDP 1 Gbit/sec
clients. Before:
[ 3] 0.0-15.0 sec 54.2 MBytes 30.4 Mbits/sec 0.875 ms 1245750/1284444 (97%)
[ 3] 0.0-15.0 sec 54.2 MBytes 30.3 Mbits/sec 1.288 ms 1238753/1277402 (97%)
[ 3] 0.0-15.0 sec 54.8 MBytes 30.6 Mbits/sec 1.761 ms 1261762/1300817 (97%)
[ 3] 0.0-15.0 sec 53.9 MBytes 30.1 Mbits/sec 1.635 ms 1241690/1280133 (97%)
<...> ^^^^^^^^^^^^^^^^^^^^^
Total throughput is 482.0 Mbits/sec and average drop rate is 97.0%.
Now enable Qdisc backpressure for UDP sockets, with
udp_backpressure_interval default to 100 milliseconds:
[ 3] 0.0-15.0 sec 53.0 MBytes 29.6 Mbits/sec 1.621 ms 54/37856 (0.14%)
[ 3] 0.0-15.0 sec 55.9 MBytes 31.3 Mbits/sec 1.368 ms 6/39895 (0.015%)
[ 3] 0.0-15.0 sec 52.3 MBytes 29.2 Mbits/sec 1.560 ms 56/37340 (0.15%)
[ 3] 0.0-15.0 sec 52.7 MBytes 29.5 Mbits/sec 1.495 ms 57/37677 (0.15%)
<...> ^^^^^^^^^^^^^^^^
Total throughput is 485.9 Mbits/sec (0.81% higher) and average drop rate
is 0.1% (99.9% lower).
Fairness between flows is slightly affected, with per-flow average
throughput ranging from 29.2 to 31.8 Mbits/sec (compared with 29.7 to
30.6 Mbits/sec).
Signed-off-by: Peilin Ye <peilin.ye@bytedance.com>
---
net/sched/sch_htb.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c
index 23a9d6242429..e337b3d0dab3 100644
--- a/net/sched/sch_htb.c
+++ b/net/sched/sch_htb.c
@@ -623,6 +623,7 @@ static int htb_enqueue(struct sk_buff *skb, struct Qdisc *sch,
__qdisc_enqueue_tail(skb, &q->direct_queue);
q->direct_pkts++;
} else {
+ qdisc_backpressure(skb);
return qdisc_drop(skb, sch, to_free);
}
#ifdef CONFIG_NET_CLS_ACT
@@ -634,6 +635,7 @@ static int htb_enqueue(struct sk_buff *skb, struct Qdisc *sch,
#endif
} else if ((ret = qdisc_enqueue(skb, cl->leaf.q,
to_free)) != NET_XMIT_SUCCESS) {
+ qdisc_backpressure(skb);
if (net_xmit_drop_count(ret)) {
qdisc_qstats_drop(sch);
cl->drops++;
--
2.20.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH RFC v2 net-next 5/5] net/sched: sch_cbq: Use Qdisc backpressure infrastructure
2022-08-22 9:10 ` [PATCH RFC v2 net-next 0/5] " Peilin Ye
` (3 preceding siblings ...)
2022-08-22 9:12 ` [PATCH RFC v2 net-next 4/5] net/sched: sch_htb: " Peilin Ye
@ 2022-08-22 9:12 ` Peilin Ye
2022-08-22 16:17 ` [PATCH RFC v2 net-next 0/5] net: " Jakub Kicinski
2022-08-22 16:22 ` Eric Dumazet
6 siblings, 0 replies; 29+ messages in thread
From: Peilin Ye @ 2022-08-22 9:12 UTC (permalink / raw)
To: David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jonathan Corbet, Hideaki YOSHIFUJI, David Ahern,
Jamal Hadi Salim, Cong Wang, Jiri Pirko
Cc: Peilin Ye, netdev, linux-doc, linux-kernel, Cong Wang,
Stephen Hemminger, Dave Taht, Peilin Ye
From: Peilin Ye <peilin.ye@bytedance.com>
Recently we introduced a Qdisc backpressure infrastructure (currently
supports UDP sockets). Use it in CBQ Qdisc.
Tested with 500 Mbits/sec rate limit using 16 iperf UDP 1 Gbit/sec
clients. Before:
[ 3] 0.0-15.0 sec 55.8 MBytes 31.2 Mbits/sec 1.185 ms 1073326/1113110 (96%)
[ 3] 0.0-15.0 sec 55.9 MBytes 31.3 Mbits/sec 1.001 ms 1080330/1120201 (96%)
[ 3] 0.0-15.0 sec 55.6 MBytes 31.1 Mbits/sec 1.750 ms 1078292/1117980 (96%)
[ 3] 0.0-15.0 sec 55.3 MBytes 30.9 Mbits/sec 0.895 ms 1089200/1128640 (97%)
<...> ^^^^^^^^^^^^^^^^^^^^^
Total throughput is 493.7 Mbits/sec and average drop rate is 96.13%.
Now enable Qdisc backpressure for UDP sockets, with
udp_backpressure_interval default to 100 milliseconds:
[ 3] 0.0-15.0 sec 54.2 MBytes 30.3 Mbits/sec 2.302 ms 54/38692 (0.14%)
[ 3] 0.0-15.0 sec 54.1 MBytes 30.2 Mbits/sec 2.227 ms 54/38671 (0.14%)
[ 3] 0.0-15.0 sec 53.5 MBytes 29.9 Mbits/sec 2.043 ms 57/38203 (0.15%)
[ 3] 0.0-15.0 sec 58.1 MBytes 32.5 Mbits/sec 1.843 ms 1/41480 (0.0024%)
<...> ^^^^^^^^^^^^^^^^^
Total throughput is 497.1 Mbits/sec (0.69% higher), average drop rate is
0.08% (99.9% lower).
Fairness between flows is slightly affected, with per-flow average
throughput ranging from 29.9 to 32.6 Mbits/sec (compared with 30.3 to
31.3 Mbits/sec).
Signed-off-by: Peilin Ye <peilin.ye@bytedance.com>
---
net/sched/sch_cbq.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/net/sched/sch_cbq.c b/net/sched/sch_cbq.c
index 91a0dc463c48..42e44f570988 100644
--- a/net/sched/sch_cbq.c
+++ b/net/sched/sch_cbq.c
@@ -381,6 +381,7 @@ cbq_enqueue(struct sk_buff *skb, struct Qdisc *sch,
return ret;
}
+ qdisc_backpressure(skb);
if (net_xmit_drop_count(ret)) {
qdisc_qstats_drop(sch);
cbq_mark_toplevel(q, cl);
--
2.20.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* Re: [PATCH RFC v2 net-next 0/5] net: Qdisc backpressure infrastructure
2022-08-22 9:10 ` [PATCH RFC v2 net-next 0/5] " Peilin Ye
` (4 preceding siblings ...)
2022-08-22 9:12 ` [PATCH RFC v2 net-next 5/5] net/sched: sch_cbq: " Peilin Ye
@ 2022-08-22 16:17 ` Jakub Kicinski
2022-08-29 16:53 ` Cong Wang
2022-08-22 16:22 ` Eric Dumazet
6 siblings, 1 reply; 29+ messages in thread
From: Jakub Kicinski @ 2022-08-22 16:17 UTC (permalink / raw)
To: Peilin Ye
Cc: David S. Miller, Eric Dumazet, Paolo Abeni, Jonathan Corbet,
Hideaki YOSHIFUJI, David Ahern, Jamal Hadi Salim, Cong Wang,
Jiri Pirko, Peilin Ye, netdev, linux-doc, linux-kernel,
Cong Wang, Stephen Hemminger, Dave Taht
On Mon, 22 Aug 2022 02:10:17 -0700 Peilin Ye wrote:
> Currently sockets (especially UDP ones) can drop a lot of packets at TC
> egress when rate limited by shaper Qdiscs like HTB. This patchset series
> tries to solve this by introducing a Qdisc backpressure mechanism.
>
> RFC v1 [1] used a throttle & unthrottle approach, which introduced several
> issues, including a thundering herd problem and a socket reference count
> issue [2]. This RFC v2 uses a different approach to avoid those issues:
>
> 1. When a shaper Qdisc drops a packet that belongs to a local socket due
> to TC egress congestion, we make part of the socket's sndbuf
> temporarily unavailable, so it sends slower.
>
> 2. Later, when TC egress becomes idle again, we gradually recover the
> socket's sndbuf back to normal. Patch 2 implements this step using a
> timer for UDP sockets.
>
> The thundering herd problem is avoided, since we no longer wake up all
> throttled sockets at the same time in qdisc_watchdog(). The socket
> reference count issue is also avoided, since we no longer maintain socket
> list on Qdisc.
>
> Performance is better than RFC v1. There is one concern about fairness
> between flows for TBF Qdisc, which could be solved by using a SFQ inner
> Qdisc.
>
> Please see the individual patches for details and numbers. Any comments,
> suggestions would be much appreciated. Thanks!
>
> [1] https://lore.kernel.org/netdev/cover.1651800598.git.peilin.ye@bytedance.com/
> [2] https://lore.kernel.org/netdev/20220506133111.1d4bebf3@hermes.local/
Similarly to Eric's comments on v1 I'm not seeing the clear motivation
here. Modern high speed UDP users will have a CC in user space, back
off and set transmission time on the packets. Could you describe your
_actual_ use case / application in more detail?
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH RFC v2 net-next 0/5] net: Qdisc backpressure infrastructure
2022-08-22 16:17 ` [PATCH RFC v2 net-next 0/5] net: " Jakub Kicinski
@ 2022-08-29 16:53 ` Cong Wang
2022-08-30 0:21 ` Jakub Kicinski
0 siblings, 1 reply; 29+ messages in thread
From: Cong Wang @ 2022-08-29 16:53 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Peilin Ye, David S. Miller, Eric Dumazet, Paolo Abeni,
Jonathan Corbet, Hideaki YOSHIFUJI, David Ahern,
Jamal Hadi Salim, Jiri Pirko, Peilin Ye, netdev, linux-doc,
linux-kernel, Cong Wang, Stephen Hemminger, Dave Taht
On Mon, Aug 22, 2022 at 09:17:37AM -0700, Jakub Kicinski wrote:
> On Mon, 22 Aug 2022 02:10:17 -0700 Peilin Ye wrote:
> > Currently sockets (especially UDP ones) can drop a lot of packets at TC
> > egress when rate limited by shaper Qdiscs like HTB. This patchset series
> > tries to solve this by introducing a Qdisc backpressure mechanism.
> >
> > RFC v1 [1] used a throttle & unthrottle approach, which introduced several
> > issues, including a thundering herd problem and a socket reference count
> > issue [2]. This RFC v2 uses a different approach to avoid those issues:
> >
> > 1. When a shaper Qdisc drops a packet that belongs to a local socket due
> > to TC egress congestion, we make part of the socket's sndbuf
> > temporarily unavailable, so it sends slower.
> >
> > 2. Later, when TC egress becomes idle again, we gradually recover the
> > socket's sndbuf back to normal. Patch 2 implements this step using a
> > timer for UDP sockets.
> >
> > The thundering herd problem is avoided, since we no longer wake up all
> > throttled sockets at the same time in qdisc_watchdog(). The socket
> > reference count issue is also avoided, since we no longer maintain socket
> > list on Qdisc.
> >
> > Performance is better than RFC v1. There is one concern about fairness
> > between flows for TBF Qdisc, which could be solved by using a SFQ inner
> > Qdisc.
> >
> > Please see the individual patches for details and numbers. Any comments,
> > suggestions would be much appreciated. Thanks!
> >
> > [1] https://lore.kernel.org/netdev/cover.1651800598.git.peilin.ye@bytedance.com/
> > [2] https://lore.kernel.org/netdev/20220506133111.1d4bebf3@hermes.local/
>
> Similarly to Eric's comments on v1 I'm not seeing the clear motivation
> here. Modern high speed UDP users will have a CC in user space, back
> off and set transmission time on the packets. Could you describe your
> _actual_ use case / application in more detail?
Not everyone implements QUIC or CC, it is really hard to implement CC
from scratch. This backpressure mechnism is much simpler than CC (TCP or
QUIC), as clearly it does not deal with any remote congestions.
And, although this patchset only implements UDP backpressure, it can be
applied to any other protocol easily, it is protocol-independent.
Thanks.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH RFC v2 net-next 0/5] net: Qdisc backpressure infrastructure
2022-08-29 16:53 ` Cong Wang
@ 2022-08-30 0:21 ` Jakub Kicinski
2022-09-19 17:00 ` Cong Wang
0 siblings, 1 reply; 29+ messages in thread
From: Jakub Kicinski @ 2022-08-30 0:21 UTC (permalink / raw)
To: Cong Wang
Cc: Peilin Ye, David S. Miller, Eric Dumazet, Paolo Abeni,
Jonathan Corbet, Hideaki YOSHIFUJI, David Ahern,
Jamal Hadi Salim, Jiri Pirko, Peilin Ye, netdev, linux-doc,
linux-kernel, Cong Wang, Stephen Hemminger, Dave Taht
On Mon, 29 Aug 2022 09:53:17 -0700 Cong Wang wrote:
> > Similarly to Eric's comments on v1 I'm not seeing the clear motivation
> > here. Modern high speed UDP users will have a CC in user space, back
> > off and set transmission time on the packets. Could you describe your
> > _actual_ use case / application in more detail?
>
> Not everyone implements QUIC or CC, it is really hard to implement CC
> from scratch. This backpressure mechnism is much simpler than CC (TCP or
> QUIC), as clearly it does not deal with any remote congestions.
>
> And, although this patchset only implements UDP backpressure, it can be
> applied to any other protocol easily, it is protocol-independent.
No disagreement on any of your points. But I don't feel like
you answered my question about the details of the use case.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH RFC v2 net-next 0/5] net: Qdisc backpressure infrastructure
2022-08-30 0:21 ` Jakub Kicinski
@ 2022-09-19 17:00 ` Cong Wang
0 siblings, 0 replies; 29+ messages in thread
From: Cong Wang @ 2022-09-19 17:00 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Peilin Ye, David S. Miller, Eric Dumazet, Paolo Abeni,
Jonathan Corbet, Hideaki YOSHIFUJI, David Ahern,
Jamal Hadi Salim, Jiri Pirko, Peilin Ye, netdev, linux-doc,
linux-kernel, Cong Wang, Stephen Hemminger, Dave Taht
On Mon, Aug 29, 2022 at 05:21:11PM -0700, Jakub Kicinski wrote:
> On Mon, 29 Aug 2022 09:53:17 -0700 Cong Wang wrote:
> > > Similarly to Eric's comments on v1 I'm not seeing the clear motivation
> > > here. Modern high speed UDP users will have a CC in user space, back
> > > off and set transmission time on the packets. Could you describe your
> > > _actual_ use case / application in more detail?
> >
> > Not everyone implements QUIC or CC, it is really hard to implement CC
> > from scratch. This backpressure mechnism is much simpler than CC (TCP or
> > QUIC), as clearly it does not deal with any remote congestions.
> >
> > And, although this patchset only implements UDP backpressure, it can be
> > applied to any other protocol easily, it is protocol-independent.
>
> No disagreement on any of your points. But I don't feel like
> you answered my question about the details of the use case.
Do you need a use case for UDP w/o QUIC? Seriously??? There must be
tons of it...
Take a look at UDP tunnels, for instance, wireguard which is our use
case. ByteDance has wireguard-based VPN solution for bussiness. (I hate
to brand ourselves, but you are asking for it...)
Please do research on your side, as a netdev maintainer, you are
supposed to know this much better than me.
Thanks.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH RFC v2 net-next 0/5] net: Qdisc backpressure infrastructure
2022-08-22 9:10 ` [PATCH RFC v2 net-next 0/5] " Peilin Ye
` (5 preceding siblings ...)
2022-08-22 16:17 ` [PATCH RFC v2 net-next 0/5] net: " Jakub Kicinski
@ 2022-08-22 16:22 ` Eric Dumazet
2022-08-29 16:47 ` Cong Wang
2022-08-30 2:28 ` Yafang Shao
6 siblings, 2 replies; 29+ messages in thread
From: Eric Dumazet @ 2022-08-22 16:22 UTC (permalink / raw)
To: Peilin Ye
Cc: David S. Miller, Jakub Kicinski, Paolo Abeni, Jonathan Corbet,
Hideaki YOSHIFUJI, David Ahern, Jamal Hadi Salim, Cong Wang,
Jiri Pirko, Peilin Ye, netdev, open list:DOCUMENTATION, LKML,
Cong Wang, Stephen Hemminger, Dave Taht
On Mon, Aug 22, 2022 at 2:10 AM Peilin Ye <yepeilin.cs@gmail.com> wrote:
>
> From: Peilin Ye <peilin.ye@bytedance.com>
>
> Hi all,
>
> Currently sockets (especially UDP ones) can drop a lot of packets at TC
> egress when rate limited by shaper Qdiscs like HTB. This patchset series
> tries to solve this by introducing a Qdisc backpressure mechanism.
>
> RFC v1 [1] used a throttle & unthrottle approach, which introduced several
> issues, including a thundering herd problem and a socket reference count
> issue [2]. This RFC v2 uses a different approach to avoid those issues:
>
> 1. When a shaper Qdisc drops a packet that belongs to a local socket due
> to TC egress congestion, we make part of the socket's sndbuf
> temporarily unavailable, so it sends slower.
>
> 2. Later, when TC egress becomes idle again, we gradually recover the
> socket's sndbuf back to normal. Patch 2 implements this step using a
> timer for UDP sockets.
>
> The thundering herd problem is avoided, since we no longer wake up all
> throttled sockets at the same time in qdisc_watchdog(). The socket
> reference count issue is also avoided, since we no longer maintain socket
> list on Qdisc.
>
> Performance is better than RFC v1. There is one concern about fairness
> between flows for TBF Qdisc, which could be solved by using a SFQ inner
> Qdisc.
>
> Please see the individual patches for details and numbers. Any comments,
> suggestions would be much appreciated. Thanks!
>
> [1] https://lore.kernel.org/netdev/cover.1651800598.git.peilin.ye@bytedance.com/
> [2] https://lore.kernel.org/netdev/20220506133111.1d4bebf3@hermes.local/
>
> Peilin Ye (5):
> net: Introduce Qdisc backpressure infrastructure
> net/udp: Implement Qdisc backpressure algorithm
> net/sched: sch_tbf: Use Qdisc backpressure infrastructure
> net/sched: sch_htb: Use Qdisc backpressure infrastructure
> net/sched: sch_cbq: Use Qdisc backpressure infrastructure
>
I think the whole idea is wrong.
Packet schedulers can be remote (offloaded, or on another box)
The idea of going back to socket level from a packet scheduler should
really be a last resort.
Issue of having UDP sockets being able to flood a network is tough, I
am not sure the core networking stack
should pretend it can solve the issue.
Note that FQ based packet schedulers can also help already.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH RFC v2 net-next 0/5] net: Qdisc backpressure infrastructure
2022-08-22 16:22 ` Eric Dumazet
@ 2022-08-29 16:47 ` Cong Wang
2022-08-29 16:53 ` Eric Dumazet
2022-08-30 2:28 ` Yafang Shao
1 sibling, 1 reply; 29+ messages in thread
From: Cong Wang @ 2022-08-29 16:47 UTC (permalink / raw)
To: Eric Dumazet
Cc: Peilin Ye, David S. Miller, Jakub Kicinski, Paolo Abeni,
Jonathan Corbet, Hideaki YOSHIFUJI, David Ahern,
Jamal Hadi Salim, Jiri Pirko, Peilin Ye, netdev,
open list:DOCUMENTATION, LKML, Cong Wang, Stephen Hemminger,
Dave Taht
On Mon, Aug 22, 2022 at 09:22:39AM -0700, Eric Dumazet wrote:
> On Mon, Aug 22, 2022 at 2:10 AM Peilin Ye <yepeilin.cs@gmail.com> wrote:
> >
> > From: Peilin Ye <peilin.ye@bytedance.com>
> >
> > Hi all,
> >
> > Currently sockets (especially UDP ones) can drop a lot of packets at TC
> > egress when rate limited by shaper Qdiscs like HTB. This patchset series
> > tries to solve this by introducing a Qdisc backpressure mechanism.
> >
> > RFC v1 [1] used a throttle & unthrottle approach, which introduced several
> > issues, including a thundering herd problem and a socket reference count
> > issue [2]. This RFC v2 uses a different approach to avoid those issues:
> >
> > 1. When a shaper Qdisc drops a packet that belongs to a local socket due
> > to TC egress congestion, we make part of the socket's sndbuf
> > temporarily unavailable, so it sends slower.
> >
> > 2. Later, when TC egress becomes idle again, we gradually recover the
> > socket's sndbuf back to normal. Patch 2 implements this step using a
> > timer for UDP sockets.
> >
> > The thundering herd problem is avoided, since we no longer wake up all
> > throttled sockets at the same time in qdisc_watchdog(). The socket
> > reference count issue is also avoided, since we no longer maintain socket
> > list on Qdisc.
> >
> > Performance is better than RFC v1. There is one concern about fairness
> > between flows for TBF Qdisc, which could be solved by using a SFQ inner
> > Qdisc.
> >
> > Please see the individual patches for details and numbers. Any comments,
> > suggestions would be much appreciated. Thanks!
> >
> > [1] https://lore.kernel.org/netdev/cover.1651800598.git.peilin.ye@bytedance.com/
> > [2] https://lore.kernel.org/netdev/20220506133111.1d4bebf3@hermes.local/
> >
> > Peilin Ye (5):
> > net: Introduce Qdisc backpressure infrastructure
> > net/udp: Implement Qdisc backpressure algorithm
> > net/sched: sch_tbf: Use Qdisc backpressure infrastructure
> > net/sched: sch_htb: Use Qdisc backpressure infrastructure
> > net/sched: sch_cbq: Use Qdisc backpressure infrastructure
> >
>
> I think the whole idea is wrong.
>
Be more specific?
> Packet schedulers can be remote (offloaded, or on another box)
This is not the case we are dealing with (yet).
>
> The idea of going back to socket level from a packet scheduler should
> really be a last resort.
I think it should be the first resort, as we should backpressure to the
source, rather than anything in the middle.
>
> Issue of having UDP sockets being able to flood a network is tough, I
> am not sure the core networking stack
> should pretend it can solve the issue.
It seems you misunderstand it here, we are not dealing with UDP on the
network, just on an end host. The backpressure we are dealing with is
from Qdisc to socket on _TX side_ and on one single host.
>
> Note that FQ based packet schedulers can also help already.
It only helps TCP pacing.
Thanks.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH RFC v2 net-next 0/5] net: Qdisc backpressure infrastructure
2022-08-29 16:47 ` Cong Wang
@ 2022-08-29 16:53 ` Eric Dumazet
2022-09-19 17:06 ` Cong Wang
0 siblings, 1 reply; 29+ messages in thread
From: Eric Dumazet @ 2022-08-29 16:53 UTC (permalink / raw)
To: Cong Wang
Cc: Peilin Ye, David S. Miller, Jakub Kicinski, Paolo Abeni,
Jonathan Corbet, Hideaki YOSHIFUJI, David Ahern,
Jamal Hadi Salim, Jiri Pirko, Peilin Ye, netdev,
open list:DOCUMENTATION, LKML, Cong Wang, Stephen Hemminger,
Dave Taht
On Mon, Aug 29, 2022 at 9:47 AM Cong Wang <xiyou.wangcong@gmail.com> wrote:
>
> On Mon, Aug 22, 2022 at 09:22:39AM -0700, Eric Dumazet wrote:
> > On Mon, Aug 22, 2022 at 2:10 AM Peilin Ye <yepeilin.cs@gmail.com> wrote:
> > >
> > > From: Peilin Ye <peilin.ye@bytedance.com>
> > >
> > > Hi all,
> > >
> > > Currently sockets (especially UDP ones) can drop a lot of packets at TC
> > > egress when rate limited by shaper Qdiscs like HTB. This patchset series
> > > tries to solve this by introducing a Qdisc backpressure mechanism.
> > >
> > > RFC v1 [1] used a throttle & unthrottle approach, which introduced several
> > > issues, including a thundering herd problem and a socket reference count
> > > issue [2]. This RFC v2 uses a different approach to avoid those issues:
> > >
> > > 1. When a shaper Qdisc drops a packet that belongs to a local socket due
> > > to TC egress congestion, we make part of the socket's sndbuf
> > > temporarily unavailable, so it sends slower.
> > >
> > > 2. Later, when TC egress becomes idle again, we gradually recover the
> > > socket's sndbuf back to normal. Patch 2 implements this step using a
> > > timer for UDP sockets.
> > >
> > > The thundering herd problem is avoided, since we no longer wake up all
> > > throttled sockets at the same time in qdisc_watchdog(). The socket
> > > reference count issue is also avoided, since we no longer maintain socket
> > > list on Qdisc.
> > >
> > > Performance is better than RFC v1. There is one concern about fairness
> > > between flows for TBF Qdisc, which could be solved by using a SFQ inner
> > > Qdisc.
> > >
> > > Please see the individual patches for details and numbers. Any comments,
> > > suggestions would be much appreciated. Thanks!
> > >
> > > [1] https://lore.kernel.org/netdev/cover.1651800598.git.peilin.ye@bytedance.com/
> > > [2] https://lore.kernel.org/netdev/20220506133111.1d4bebf3@hermes.local/
> > >
> > > Peilin Ye (5):
> > > net: Introduce Qdisc backpressure infrastructure
> > > net/udp: Implement Qdisc backpressure algorithm
> > > net/sched: sch_tbf: Use Qdisc backpressure infrastructure
> > > net/sched: sch_htb: Use Qdisc backpressure infrastructure
> > > net/sched: sch_cbq: Use Qdisc backpressure infrastructure
> > >
> >
> > I think the whole idea is wrong.
> >
>
> Be more specific?
>
> > Packet schedulers can be remote (offloaded, or on another box)
>
> This is not the case we are dealing with (yet).
>
> >
> > The idea of going back to socket level from a packet scheduler should
> > really be a last resort.
>
> I think it should be the first resort, as we should backpressure to the
> source, rather than anything in the middle.
>
> >
> > Issue of having UDP sockets being able to flood a network is tough, I
> > am not sure the core networking stack
> > should pretend it can solve the issue.
>
> It seems you misunderstand it here, we are not dealing with UDP on the
> network, just on an end host. The backpressure we are dealing with is
> from Qdisc to socket on _TX side_ and on one single host.
>
> >
> > Note that FQ based packet schedulers can also help already.
>
> It only helps TCP pacing.
FQ : Fair Queue.
It definitely helps without the pacing part...
>
> Thanks.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH RFC v2 net-next 0/5] net: Qdisc backpressure infrastructure
2022-08-29 16:53 ` Eric Dumazet
@ 2022-09-19 17:06 ` Cong Wang
0 siblings, 0 replies; 29+ messages in thread
From: Cong Wang @ 2022-09-19 17:06 UTC (permalink / raw)
To: Eric Dumazet
Cc: Peilin Ye, David S. Miller, Jakub Kicinski, Paolo Abeni,
Jonathan Corbet, Hideaki YOSHIFUJI, David Ahern,
Jamal Hadi Salim, Jiri Pirko, Peilin Ye, netdev,
open list:DOCUMENTATION, LKML, Cong Wang, Stephen Hemminger,
Dave Taht
On Mon, Aug 29, 2022 at 09:53:43AM -0700, Eric Dumazet wrote:
> On Mon, Aug 29, 2022 at 9:47 AM Cong Wang <xiyou.wangcong@gmail.com> wrote:
> >
> > On Mon, Aug 22, 2022 at 09:22:39AM -0700, Eric Dumazet wrote:
> > > On Mon, Aug 22, 2022 at 2:10 AM Peilin Ye <yepeilin.cs@gmail.com> wrote:
> > > >
> > > > From: Peilin Ye <peilin.ye@bytedance.com>
> > > >
> > > > Hi all,
> > > >
> > > > Currently sockets (especially UDP ones) can drop a lot of packets at TC
> > > > egress when rate limited by shaper Qdiscs like HTB. This patchset series
> > > > tries to solve this by introducing a Qdisc backpressure mechanism.
> > > >
> > > > RFC v1 [1] used a throttle & unthrottle approach, which introduced several
> > > > issues, including a thundering herd problem and a socket reference count
> > > > issue [2]. This RFC v2 uses a different approach to avoid those issues:
> > > >
> > > > 1. When a shaper Qdisc drops a packet that belongs to a local socket due
> > > > to TC egress congestion, we make part of the socket's sndbuf
> > > > temporarily unavailable, so it sends slower.
> > > >
> > > > 2. Later, when TC egress becomes idle again, we gradually recover the
> > > > socket's sndbuf back to normal. Patch 2 implements this step using a
> > > > timer for UDP sockets.
> > > >
> > > > The thundering herd problem is avoided, since we no longer wake up all
> > > > throttled sockets at the same time in qdisc_watchdog(). The socket
> > > > reference count issue is also avoided, since we no longer maintain socket
> > > > list on Qdisc.
> > > >
> > > > Performance is better than RFC v1. There is one concern about fairness
> > > > between flows for TBF Qdisc, which could be solved by using a SFQ inner
> > > > Qdisc.
> > > >
> > > > Please see the individual patches for details and numbers. Any comments,
> > > > suggestions would be much appreciated. Thanks!
> > > >
> > > > [1] https://lore.kernel.org/netdev/cover.1651800598.git.peilin.ye@bytedance.com/
> > > > [2] https://lore.kernel.org/netdev/20220506133111.1d4bebf3@hermes.local/
> > > >
> > > > Peilin Ye (5):
> > > > net: Introduce Qdisc backpressure infrastructure
> > > > net/udp: Implement Qdisc backpressure algorithm
> > > > net/sched: sch_tbf: Use Qdisc backpressure infrastructure
> > > > net/sched: sch_htb: Use Qdisc backpressure infrastructure
> > > > net/sched: sch_cbq: Use Qdisc backpressure infrastructure
> > > >
> > >
> > > I think the whole idea is wrong.
> > >
> >
> > Be more specific?
> >
> > > Packet schedulers can be remote (offloaded, or on another box)
> >
> > This is not the case we are dealing with (yet).
> >
> > >
> > > The idea of going back to socket level from a packet scheduler should
> > > really be a last resort.
> >
> > I think it should be the first resort, as we should backpressure to the
> > source, rather than anything in the middle.
> >
> > >
> > > Issue of having UDP sockets being able to flood a network is tough, I
> > > am not sure the core networking stack
> > > should pretend it can solve the issue.
> >
> > It seems you misunderstand it here, we are not dealing with UDP on the
> > network, just on an end host. The backpressure we are dealing with is
> > from Qdisc to socket on _TX side_ and on one single host.
> >
> > >
> > > Note that FQ based packet schedulers can also help already.
> >
> > It only helps TCP pacing.
>
> FQ : Fair Queue.
>
> It definitely helps without the pacing part...
True. but the fair queuing part has nothing related to this patchset...
Only the pacing part is related to this topic, and it is merely about
TCP.
Thanks.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH RFC v2 net-next 0/5] net: Qdisc backpressure infrastructure
2022-08-22 16:22 ` Eric Dumazet
2022-08-29 16:47 ` Cong Wang
@ 2022-08-30 2:28 ` Yafang Shao
2022-09-19 17:04 ` Cong Wang
1 sibling, 1 reply; 29+ messages in thread
From: Yafang Shao @ 2022-08-30 2:28 UTC (permalink / raw)
To: Eric Dumazet
Cc: Peilin Ye, David S. Miller, Jakub Kicinski, Paolo Abeni,
Jonathan Corbet, Hideaki YOSHIFUJI, David Ahern,
Jamal Hadi Salim, Cong Wang, Jiri Pirko, Peilin Ye, netdev,
open list:DOCUMENTATION, LKML, Cong Wang, Stephen Hemminger,
Dave Taht
On Tue, Aug 23, 2022 at 1:02 AM Eric Dumazet <edumazet@google.com> wrote:
>
> On Mon, Aug 22, 2022 at 2:10 AM Peilin Ye <yepeilin.cs@gmail.com> wrote:
> >
> > From: Peilin Ye <peilin.ye@bytedance.com>
> >
> > Hi all,
> >
> > Currently sockets (especially UDP ones) can drop a lot of packets at TC
> > egress when rate limited by shaper Qdiscs like HTB. This patchset series
> > tries to solve this by introducing a Qdisc backpressure mechanism.
> >
> > RFC v1 [1] used a throttle & unthrottle approach, which introduced several
> > issues, including a thundering herd problem and a socket reference count
> > issue [2]. This RFC v2 uses a different approach to avoid those issues:
> >
> > 1. When a shaper Qdisc drops a packet that belongs to a local socket due
> > to TC egress congestion, we make part of the socket's sndbuf
> > temporarily unavailable, so it sends slower.
> >
> > 2. Later, when TC egress becomes idle again, we gradually recover the
> > socket's sndbuf back to normal. Patch 2 implements this step using a
> > timer for UDP sockets.
> >
> > The thundering herd problem is avoided, since we no longer wake up all
> > throttled sockets at the same time in qdisc_watchdog(). The socket
> > reference count issue is also avoided, since we no longer maintain socket
> > list on Qdisc.
> >
> > Performance is better than RFC v1. There is one concern about fairness
> > between flows for TBF Qdisc, which could be solved by using a SFQ inner
> > Qdisc.
> >
> > Please see the individual patches for details and numbers. Any comments,
> > suggestions would be much appreciated. Thanks!
> >
> > [1] https://lore.kernel.org/netdev/cover.1651800598.git.peilin.ye@bytedance.com/
> > [2] https://lore.kernel.org/netdev/20220506133111.1d4bebf3@hermes.local/
> >
> > Peilin Ye (5):
> > net: Introduce Qdisc backpressure infrastructure
> > net/udp: Implement Qdisc backpressure algorithm
> > net/sched: sch_tbf: Use Qdisc backpressure infrastructure
> > net/sched: sch_htb: Use Qdisc backpressure infrastructure
> > net/sched: sch_cbq: Use Qdisc backpressure infrastructure
> >
>
> I think the whole idea is wrong.
>
> Packet schedulers can be remote (offloaded, or on another box)
>
> The idea of going back to socket level from a packet scheduler should
> really be a last resort.
>
> Issue of having UDP sockets being able to flood a network is tough, I
> am not sure the core networking stack
> should pretend it can solve the issue.
>
> Note that FQ based packet schedulers can also help already.
We encounter a similar issue when using (fq + edt-bpf) to limit UDP
packet, because of the qdisc buffer limit.
If the qdisc buffer limit is too small, the UDP packet will be dropped
in the qdisc layer. But the sender doesn't know that the packets has
been dropped, so it will continue to send packets, and thus more and
more packets will be dropped there. IOW, the qdisc will be a
bottleneck before the bandwidth limit is reached.
We workaround this issue by enlarging the buffer limit and flow_limit
(the proper values can be calculated from net.ipv4.udp_mem and
net.core.wmem_default).
But obviously this is not a perfect solution, because
net.ipv4.udp_mem or net.core.wmem_default may be changed dynamically.
We also think about a solution to build a connection between udp
memory and qdisc limit, but not sure if it is a good idea neither.
--
Regards
Yafang
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH RFC v2 net-next 0/5] net: Qdisc backpressure infrastructure
2022-08-30 2:28 ` Yafang Shao
@ 2022-09-19 17:04 ` Cong Wang
0 siblings, 0 replies; 29+ messages in thread
From: Cong Wang @ 2022-09-19 17:04 UTC (permalink / raw)
To: Yafang Shao
Cc: Eric Dumazet, Peilin Ye, David S. Miller, Jakub Kicinski,
Paolo Abeni, Jonathan Corbet, Hideaki YOSHIFUJI, David Ahern,
Jamal Hadi Salim, Jiri Pirko, Peilin Ye, netdev,
open list:DOCUMENTATION, LKML, Cong Wang, Stephen Hemminger,
Dave Taht
On Tue, Aug 30, 2022 at 10:28:01AM +0800, Yafang Shao wrote:
> On Tue, Aug 23, 2022 at 1:02 AM Eric Dumazet <edumazet@google.com> wrote:
> >
> > On Mon, Aug 22, 2022 at 2:10 AM Peilin Ye <yepeilin.cs@gmail.com> wrote:
> > >
> > > From: Peilin Ye <peilin.ye@bytedance.com>
> > >
> > > Hi all,
> > >
> > > Currently sockets (especially UDP ones) can drop a lot of packets at TC
> > > egress when rate limited by shaper Qdiscs like HTB. This patchset series
> > > tries to solve this by introducing a Qdisc backpressure mechanism.
> > >
> > > RFC v1 [1] used a throttle & unthrottle approach, which introduced several
> > > issues, including a thundering herd problem and a socket reference count
> > > issue [2]. This RFC v2 uses a different approach to avoid those issues:
> > >
> > > 1. When a shaper Qdisc drops a packet that belongs to a local socket due
> > > to TC egress congestion, we make part of the socket's sndbuf
> > > temporarily unavailable, so it sends slower.
> > >
> > > 2. Later, when TC egress becomes idle again, we gradually recover the
> > > socket's sndbuf back to normal. Patch 2 implements this step using a
> > > timer for UDP sockets.
> > >
> > > The thundering herd problem is avoided, since we no longer wake up all
> > > throttled sockets at the same time in qdisc_watchdog(). The socket
> > > reference count issue is also avoided, since we no longer maintain socket
> > > list on Qdisc.
> > >
> > > Performance is better than RFC v1. There is one concern about fairness
> > > between flows for TBF Qdisc, which could be solved by using a SFQ inner
> > > Qdisc.
> > >
> > > Please see the individual patches for details and numbers. Any comments,
> > > suggestions would be much appreciated. Thanks!
> > >
> > > [1] https://lore.kernel.org/netdev/cover.1651800598.git.peilin.ye@bytedance.com/
> > > [2] https://lore.kernel.org/netdev/20220506133111.1d4bebf3@hermes.local/
> > >
> > > Peilin Ye (5):
> > > net: Introduce Qdisc backpressure infrastructure
> > > net/udp: Implement Qdisc backpressure algorithm
> > > net/sched: sch_tbf: Use Qdisc backpressure infrastructure
> > > net/sched: sch_htb: Use Qdisc backpressure infrastructure
> > > net/sched: sch_cbq: Use Qdisc backpressure infrastructure
> > >
> >
> > I think the whole idea is wrong.
> >
> > Packet schedulers can be remote (offloaded, or on another box)
> >
> > The idea of going back to socket level from a packet scheduler should
> > really be a last resort.
> >
> > Issue of having UDP sockets being able to flood a network is tough, I
> > am not sure the core networking stack
> > should pretend it can solve the issue.
> >
> > Note that FQ based packet schedulers can also help already.
>
> We encounter a similar issue when using (fq + edt-bpf) to limit UDP
> packet, because of the qdisc buffer limit.
> If the qdisc buffer limit is too small, the UDP packet will be dropped
> in the qdisc layer. But the sender doesn't know that the packets has
> been dropped, so it will continue to send packets, and thus more and
> more packets will be dropped there. IOW, the qdisc will be a
> bottleneck before the bandwidth limit is reached.
> We workaround this issue by enlarging the buffer limit and flow_limit
> (the proper values can be calculated from net.ipv4.udp_mem and
> net.core.wmem_default).
> But obviously this is not a perfect solution, because
> net.ipv4.udp_mem or net.core.wmem_default may be changed dynamically.
> We also think about a solution to build a connection between udp
> memory and qdisc limit, but not sure if it is a good idea neither.
This is literally what this patchset does. Although this patchset does
not touch any TCP (as TCP has TSQ), I think this is a better approach
than TSQ, because TSQ has no idea about Qdisc limit.
Thanks.
^ permalink raw reply [flat|nested] 29+ messages in thread