[PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections

All of lore.kernel.org
 help / color / mirror / Atom feed

* [PATCH net-next] tcp: optimise  receiver buffer autotuning initialisation for high latency connections
@ 2020-12-04 18:06 Hazem Mohamed Abuelfotoh
  2020-12-04 18:19 ` Mohamed Abuelfotoh, Hazem
                   ` (2 more replies)
  0 siblings, 3 replies; 22+ messages in thread
From: Hazem Mohamed Abuelfotoh @ 2020-12-04 18:06 UTC (permalink / raw)
  To: netdev
  Cc: stable, edumazet, ycheng, ncardwell, weiwan, astroh, benh,
	Hazem Mohamed Abuelfotoh

    Previously receiver buffer auto-tuning starts after receiving
    one advertised window amount of data.After the initial
    receiver buffer was raised by
    commit a337531b942b ("tcp: up initial rmem to 128KB
    and SYN rwin to around 64KB"),the receiver buffer may
    take too long for TCP autotuning to start raising
    the receiver buffer size.
    commit 041a14d26715 ("tcp: start receiver buffer autotuning sooner")
    tried to decrease the threshold at which TCP auto-tuning starts
    but it's doesn't work well in some environments
    where the receiver has large MTU (9001) configured
    specially within environments where RTT is high.
    To address this issue this patch is relying on RCV_MSS
    so auto-tuning can start early regardless
    the receiver configured MTU.

    Fixes: a337531b942b ("tcp: up initial rmem to 128KB and SYN rwin to around 64KB")
    Fixes: 041a14d26715 ("tcp: start receiver buffer autotuning sooner")

Signed-off-by: Hazem Mohamed Abuelfotoh <abuehaze@amazon.com>
---
 net/ipv4/tcp_input.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 389d1b340248..f0ffac9e937b 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -504,13 +504,14 @@ static void tcp_grow_window(struct sock *sk, const struct sk_buff *skb)
 static void tcp_init_buffer_space(struct sock *sk)
 {
 	int tcp_app_win = sock_net(sk)->ipv4.sysctl_tcp_app_win;
+	struct inet_connection_sock *icsk = inet_csk(sk);
 	struct tcp_sock *tp = tcp_sk(sk);
 	int maxwin;
 
 	if (!(sk->sk_userlocks & SOCK_SNDBUF_LOCK))
 		tcp_sndbuf_expand(sk);
 
-	tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND * tp->advmss);
+	tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND * icsk->icsk_ack.rcv_mss);
 	tcp_mstamp_refresh(tp);
 	tp->rcvq_space.time = tp->tcp_mstamp;
 	tp->rcvq_space.seq = tp->copied_seq;
-- 
2.16.6




Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284

Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705




^ permalink raw reply related	[flat|nested] 22+ messages in thread

* Re: [PATCH net-next] tcp: optimise  receiver buffer autotuning initialisation for high latency connections
  2020-12-04 18:06 [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections Hazem Mohamed Abuelfotoh
@ 2020-12-04 18:19 ` Mohamed Abuelfotoh, Hazem
  2020-12-04 18:41   ` Eric Dumazet
  2020-12-04 19:10 ` Eric Dumazet
  2020-12-04 21:28 ` Neal Cardwell
  2 siblings, 1 reply; 22+ messages in thread
From: Mohamed Abuelfotoh, Hazem @ 2020-12-04 18:19 UTC (permalink / raw)
  To: netdev
  Cc: stable, edumazet, ycheng, ncardwell, weiwan, Strohman, Andy,
	Herrenschmidt, Benjamin

[-- Attachment #1: Type: text/plain, Size: 13343 bytes --]

Hey Team,

I am sending you this e-mail as a follow-up to provide more context about the patch that I proposed in my previous e-mail.

1-We have received a customer complain[1] about degraded download speed   from google endpoints after they upgraded their Ubuntu kernel from 4.14 to 5.4.These customers were getting around 80MB/s on kernel 4.14 which became 3MB/s after the upgrade to kernel 5.4.
2-We tried to reproduce the issue locally between EC2 instances within the same region but we couldn’t however we were able to reproduce it when fetching data from google endpoint.
3-The issue could only be reproduced in Regions where we have high RTT(around 12msec  or more ) with Google endpoints.
4-We have found some workarounds that can be applied on the receiver side which has proven to be effective and I am listing them below:
            A) Decrease TCP socket default rmem from 131072 to 87380
            B) Decrease MTU from 9001 to 1500.
            C) Change sysctl_tcp_adv_win_scale from default 1 to 0 or 2
            D)We have also found that disabling net.ipv4.tcp_moderate_rcvbuf on kernel 4.14 is giving exactly the same bad performance speed.
5-We have done some kernel bisect to understand when this behaviour has been introduced and found that   commit a337531b942b ("tcp: up initial rmem to 128KB and SYN rwin to around 64KB")[2] which has been merged to mainline kernel 4.19.86 is the culprit behind this download performance degradation, The commit  mainly did two main changes:
A)Raising the initial TCP receive buffer size and receive window.
B)Changing the way in which TCP Dynamic Right Sizing (DRS) is been kicked off.

6)There was a regression that has been introduced because of the above patch causing the receive window scaling  to take long time after raising the initial receiver buffer & receive window  and there was additional fix for that  in commit 041a14d26715 ("tcp: start receiver buffer autotuning sooner")[3].

7)Commit 041a14d26715 ("tcp: start receiver buffer autotuning sooner") was trying to decrease the initial rcvq_space.space which  is used in TCP's internal auto-tuning to grow socket buffers based on how much data the kernel estimates the sender can send and It should  change over the life of any connection based on the amount of data that the sender is sending. This patch is relying on advmss (which is the MSS configured on the receiver) to identify the initial receive space, although this works very well with receivers with small MTUs like 1500 it’s doesn’t help if the receiver is configured to use Jumbo frames (9001 MTU) which is the default MTU on AWS EC2 instances and this is why we think this hasn’t been reported before beside the high RTT >=12msec required to see the issue as well.

8)After further debugging and testing we have found that the issue can only be reproduced under any of the  below conditions:
A)Sender (MTU 1500) using bbr/bbrv2 as congestion control algorithm ——> Receiver (MTU 9001) with default ipv4.sysctl_tcp_rmem[1] = 131072   running kernel 4.19.86 or later with RTT >=12msec.——>consistently reproducible
B)Sender (MTU 1500) using cubic as congestion control algorithm with fq as disc ——> Receiver (MTU 9001) with default ipv4.sysctl_tcp_rmem[1] = 131072 running kernel 4.19.86 or later with RTT >=30msec.——>consistently reproducible.
C)Sender (MTU 1500) using cubic as congestion control algorithm with pfifo_fast as qdisc ——> Receiver (MTU 9001) with default ipv4.sysctl_tcp_rmem[1] = 131072   running kernel 4.19.86 or later with RTT >=30msec.——>intermittently  reproducible
D)Sender needs a MTU of 1500. If the sender is using MTU of 9001  with no MSS clamping , then we  couldn’t  reproduce the issue.
E)AWS EC2 instances are using 9001 as MTU by default hence they are likely more impacted by this.

9)With some kernel hacking & packet capture analysis we found that the main issue is that under the above mentioned conditions the receive window never scales up as it looks like the tcp receiver autotuning never kicks off, I have attached to this e-mail  screenshots showing Window scaling with and without the proposed patch.
We also found that all workarounds either decreasing initial rcvq_space (this includes decreasing receiver advertised MSS from 9001 to 1500 or  default receive buffer size from 131072 to 87380) or increasing the maximum advertised receive window (before TCP autotuning start scaling) and this includes changing net.ipv4.tcp_adv_win_scale from 1 to 0 or 2.

10)It looks like when the issue happen we have a  kind of deadlock here so advertised receive window has to exceed rcvq_space for the tcp auto tuning to kickoff at the same time with the initial default  configuration the receive window is not going to exceed rcvq_space because it can only get half of the initial receive socket buffer size.

11)The current code which is based on patch has main drawback  which should be handled:
A)It relies on receiver configured MTU to define the initial receive space(threshold where tcp autotuning starts), as mentioned above this works well with 1500 MTU because with that it will make sure that initial receive space is lower than receive window so tcp autotuning will work just fine while it won’t work with Jumbo frames in use on the receiver because at this case the receiver won’t start tcp autotuning especially with high RTT and we will be hitting the regression that commit  041a14d26715 ("tcp: start receiver buffer autotuning sooner") was trying to handle.
12)I am proposing  the below patch which is relying on RCV_MSS (our guess about MSS used by the peer which is equal to TCP_MSS_DEFAULT 536 bytes by default) this should work regardless the receiver configured MSS. I am also sharing  my iperf test results with and without the patch and also verified  that the connection won’t get stuck in the middle in case of packet loss or latency spike which I emulated using tc netem on the sender side.

Test Results using the same sender & receiver:

-Without our proposed patch

#iperf3 -c xx.xx.xx.xx -t15 -i1 -R
Connecting to host xx.xx.xx.xx, port 5201
Reverse mode, remote host xx.xx.xx.xx is sending
[  4] local 172.31.37.167 port 52838 connected to xx.xx.xx.xx port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   269 KBytes  2.20 Mbits/sec
[  4]   1.00-2.00   sec   332 KBytes  2.72 Mbits/sec
[  4]   2.00-3.00   sec   334 KBytes  2.73 Mbits/sec
[  4]   3.00-4.00   sec   335 KBytes  2.75 Mbits/sec
[  4]   4.00-5.00   sec   332 KBytes  2.72 Mbits/sec
[  4]   5.00-6.00   sec   283 KBytes  2.32 Mbits/sec
[  4]   6.00-7.00   sec   332 KBytes  2.72 Mbits/sec
[  4]   7.00-8.00   sec   335 KBytes  2.75 Mbits/sec
[  4]   8.00-9.00   sec   335 KBytes  2.75 Mbits/sec
[  4]   9.00-10.00  sec   334 KBytes  2.73 Mbits/sec
[  4]  10.00-11.00  sec   332 KBytes  2.72 Mbits/sec
[  4]  11.00-12.00  sec   332 KBytes  2.72 Mbits/sec
[  4]  12.00-13.00  sec   338 KBytes  2.77 Mbits/sec
[  4]  13.00-14.00  sec   334 KBytes  2.73 Mbits/sec
[  4]  14.00-15.00  sec   332 KBytes  2.72 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-15.00  sec  6.07 MBytes  3.39 Mbits/sec    0             sender
[  4]   0.00-15.00  sec  4.90 MBytes  2.74 Mbits/sec                  receiver

iperf Done.

Test downloading from google endpoint:

# wget https://storage.googleapis.com/kubernetes-release/release/v1.18.9/bin/linux/amd64/kubelet
--2020-12-04 16:53:00--  https://storage.googleapis.com/kubernetes-release/release/v1.18.9/bin/linux/amd64/kubelet
Resolving storage.googleapis.com (storage.googleapis.com)... 172.217.1.48, 172.217.8.176, 172.217.4.48, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.1.48|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 113320760 (108M) [application/octet-stream]
Saving to: ‘kubelet.45’

100%[===================================================================================================================================================>] 113,320,760 3.04MB/s   in 36s

2020-12-04 16:53:36 (3.02 MB/s) - ‘kubelet’ saved [113320760/113320760]

########################################################################################################################

-With the proposed  patch:

#iperf3 -c xx.xx.xx.xx -t15 -i1 -R
Connecting to host xx.xx.xx.xx, port 5201
Reverse mode, remote host xx.xx.xx.xx is sending
[  4] local 172.31.37.167 port 44514 connected to xx.xx.xx.xx port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   911 KBytes  7.46 Mbits/sec
[  4]   1.00-2.00   sec  8.95 MBytes  75.1 Mbits/sec
[  4]   2.00-3.00   sec  9.57 MBytes  80.3 Mbits/sec
[  4]   3.00-4.00   sec  9.56 MBytes  80.2 Mbits/sec
[  4]   4.00-5.00   sec  9.58 MBytes  80.3 Mbits/sec
[  4]   5.00-6.00   sec  9.58 MBytes  80.4 Mbits/sec
[  4]   6.00-7.00   sec  9.59 MBytes  80.4 Mbits/sec
[  4]   7.00-8.00   sec  9.59 MBytes  80.5 Mbits/sec
[  4]   8.00-9.00   sec  9.58 MBytes  80.4 Mbits/sec
[  4]   9.00-10.00  sec  9.58 MBytes  80.4 Mbits/sec
[  4]  10.00-11.00  sec  9.59 MBytes  80.4 Mbits/sec
[  4]  11.00-12.00  sec  9.59 MBytes  80.5 Mbits/sec
[  4]  12.00-13.00  sec  8.05 MBytes  67.5 Mbits/sec
[  4]  13.00-14.00  sec  9.57 MBytes  80.3 Mbits/sec
[  4]  14.00-15.00  sec  9.57 MBytes  80.3 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-15.00  sec   136 MBytes  76.3 Mbits/sec    0             sender
[  4]   0.00-15.00  sec   134 MBytes  75.2 Mbits/sec                  receiver

iperf Done.

Test downloading from google endpoint:

# wget https://storage.googleapis.com/kubernetes-release/release/v1.18.9/bin/linux/amd64/kubelet
--2020-12-04 16:54:34--  https://storage.googleapis.com/kubernetes-release/release/v1.18.9/bin/linux/amd64/kubelet
Resolving storage.googleapis.com (storage.googleapis.com)... 172.217.0.16, 216.58.192.144, 172.217.6.16, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.0.16|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 113320760 (108M) [application/octet-stream]
Saving to: ‘kubelet’

100%[===================================================================================================================================================>] 113,320,760 80.0MB/s   in 1.4s

2020-12-04 16:54:36 (80.0 MB/s) - ‘kubelet.1’ saved [113320760/113320760]

Links:

[1] https://github.com/kubernetes/kops/issues/10206
[2] https://lore.kernel.org/patchwork/patch/1157936/
[3] https://lore.kernel.org/patchwork/patch/1157883/

Thank you.

Hazem

On 04/12/2020, 18:08, "Hazem Mohamed Abuelfotoh" <abuehaze@amazon.com> wrote:

        Previously receiver buffer auto-tuning starts after receiving
        one advertised window amount of data.After the initial
        receiver buffer was raised by
        commit a337531b942b ("tcp: up initial rmem to 128KB
        and SYN rwin to around 64KB"),the receiver buffer may
        take too long for TCP autotuning to start raising
        the receiver buffer size.
        commit 041a14d26715 ("tcp: start receiver buffer autotuning sooner")
        tried to decrease the threshold at which TCP auto-tuning starts
        but it's doesn't work well in some environments
        where the receiver has large MTU (9001) configured
        specially within environments where RTT is high.
        To address this issue this patch is relying on RCV_MSS
        so auto-tuning can start early regardless
        the receiver configured MTU.

        Fixes: a337531b942b ("tcp: up initial rmem to 128KB and SYN rwin to around 64KB")
        Fixes: 041a14d26715 ("tcp: start receiver buffer autotuning sooner")

    Signed-off-by: Hazem Mohamed Abuelfotoh <abuehaze@amazon.com>
    ---
     net/ipv4/tcp_input.c | 3 ++-
     1 file changed, 2 insertions(+), 1 deletion(-)

    diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
    index 389d1b340248..f0ffac9e937b 100644
    --- a/net/ipv4/tcp_input.c
    +++ b/net/ipv4/tcp_input.c
    @@ -504,13 +504,14 @@ static void tcp_grow_window(struct sock *sk, const struct sk_buff *skb)
     static void tcp_init_buffer_space(struct sock *sk)
     {
     	int tcp_app_win = sock_net(sk)->ipv4.sysctl_tcp_app_win;
    +	struct inet_connection_sock *icsk = inet_csk(sk);
     	struct tcp_sock *tp = tcp_sk(sk);
     	int maxwin;

     	if (!(sk->sk_userlocks & SOCK_SNDBUF_LOCK))
     		tcp_sndbuf_expand(sk);

    -	tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND * tp->advmss);
    +	tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND * icsk->icsk_ack.rcv_mss);
     	tcp_mstamp_refresh(tp);
     	tp->rcvq_space.time = tp->tcp_mstamp;
     	tp->rcvq_space.seq = tp->copied_seq;
    -- 
    2.16.6

Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284

Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705

[-- Attachment #2: sender-bbr-bad-unpatched.png --]
[-- Type: image/png, Size: 125913 bytes --]

[-- Attachment #3: sender-bbr-good-patched.png --]
[-- Type: image/png, Size: 119698 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections
  2020-12-04 18:19 ` Mohamed Abuelfotoh, Hazem
@ 2020-12-04 18:41   ` Eric Dumazet
       [not found]     ` <3F02FF08-EDA6-4DFD-8D93-479A5B05E25A@amazon.com>
  0 siblings, 1 reply; 22+ messages in thread
From: Eric Dumazet @ 2020-12-04 18:41 UTC (permalink / raw)
  To: Mohamed Abuelfotoh, Hazem
  Cc: netdev, stable, ycheng, ncardwell, weiwan, Strohman, Andy,
	Herrenschmidt, Benjamin

On Fri, Dec 4, 2020 at 7:19 PM Mohamed Abuelfotoh, Hazem
<abuehaze@amazon.com> wrote:
>
> Hey Team,
>
> I am sending you this e-mail as a follow-up to provide more context about the patch that I proposed in my previous e-mail.
>
>
> 1-We have received a customer complain[1] about degraded download speed   from google endpoints after they upgraded their Ubuntu kernel from 4.14 to 5.4.These customers were getting around 80MB/s on kernel 4.14 which became 3MB/s after the upgrade to kernel 5.4.
> 2-We tried to reproduce the issue locally between EC2 instances within the same region but we couldn’t however we were able to reproduce it when fetching data from google endpoint.
> 3-The issue could only be reproduced in Regions where we have high RTT(around 12msec  or more ) with Google endpoints.
> 4-We have found some workarounds that can be applied on the receiver side which has proven to be effective and I am listing them below:
>             A) Decrease TCP socket default rmem from 131072 to 87380
>             B) Decrease MTU from 9001 to 1500.
>             C) Change sysctl_tcp_adv_win_scale from default 1 to 0 or 2
>             D)We have also found that disabling net.ipv4.tcp_moderate_rcvbuf on kernel 4.14 is giving exactly the same bad performance speed.
> 5-We have done some kernel bisect to understand when this behaviour has been introduced and found that   commit a337531b942b ("tcp: up initial rmem to 128KB and SYN rwin to around 64KB")[2] which has been merged to mainline kernel 4.19.86 is the culprit behind this download performance degradation, The commit  mainly did two main changes:
> A)Raising the initial TCP receive buffer size and receive window.
> B)Changing the way in which TCP Dynamic Right Sizing (DRS) is been kicked off.
>
> 6)There was a regression that has been introduced because of the above patch causing the receive window scaling  to take long time after raising the initial receiver buffer & receive window  and there was additional fix for that  in commit 041a14d26715 ("tcp: start receiver buffer autotuning sooner")[3].
>
> 7)Commit 041a14d26715 ("tcp: start receiver buffer autotuning sooner") was trying to decrease the initial rcvq_space.space which  is used in TCP's internal auto-tuning to grow socket buffers based on how much data the kernel estimates the sender can send and It should  change over the life of any connection based on the amount of data that the sender is sending. This patch is relying on advmss (which is the MSS configured on the receiver) to identify the initial receive space, although this works very well with receivers with small MTUs like 1500 it’s doesn’t help if the receiver is configured to use Jumbo frames (9001 MTU) which is the default MTU on AWS EC2 instances and this is why we think this hasn’t been reported before beside the high RTT >=12msec required to see the issue as well.
>
> 8)After further debugging and testing we have found that the issue can only be reproduced under any of the  below conditions:
> A)Sender (MTU 1500) using bbr/bbrv2 as congestion control algorithm ——> Receiver (MTU 9001) with default ipv4.sysctl_tcp_rmem[1] = 131072   running kernel 4.19.86 or later with RTT >=12msec.——>consistently reproducible
> B)Sender (MTU 1500) using cubic as congestion control algorithm with fq as disc ——> Receiver (MTU 9001) with default ipv4.sysctl_tcp_rmem[1] = 131072 running kernel 4.19.86 or later with RTT >=30msec.——>consistently reproducible.
> C)Sender (MTU 1500) using cubic as congestion control algorithm with pfifo_fast as qdisc ——> Receiver (MTU 9001) with default ipv4.sysctl_tcp_rmem[1] = 131072   running kernel 4.19.86 or later with RTT >=30msec.——>intermittently  reproducible
> D)Sender needs a MTU of 1500. If the sender is using MTU of 9001  with no MSS clamping , then we  couldn’t  reproduce the issue.
> E)AWS EC2 instances are using 9001 as MTU by default hence they are likely more impacted by this.
>
>
> 9)With some kernel hacking & packet capture analysis we found that the main issue is that under the above mentioned conditions the receive window never scales up as it looks like the tcp receiver autotuning never kicks off, I have attached to this e-mail  screenshots showing Window scaling with and without the proposed patch.
> We also found that all workarounds either decreasing initial rcvq_space (this includes decreasing receiver advertised MSS from 9001 to 1500 or  default receive buffer size from 131072 to 87380) or increasing the maximum advertised receive window (before TCP autotuning start scaling) and this includes changing net.ipv4.tcp_adv_win_scale from 1 to 0 or 2.
>
> 10)It looks like when the issue happen we have a  kind of deadlock here so advertised receive window has to exceed rcvq_space for the tcp auto tuning to kickoff at the same time with the initial default  configuration the receive window is not going to exceed rcvq_space because it can only get half of the initial receive socket buffer size.
>
> 11)The current code which is based on patch has main drawback  which should be handled:
> A)It relies on receiver configured MTU to define the initial receive space(threshold where tcp autotuning starts), as mentioned above this works well with 1500 MTU because with that it will make sure that initial receive space is lower than receive window so tcp autotuning will work just fine while it won’t work with Jumbo frames in use on the receiver because at this case the receiver won’t start tcp autotuning especially with high RTT and we will be hitting the regression that commit  041a14d26715 ("tcp: start receiver buffer autotuning sooner") was trying to handle.
> 12)I am proposing  the below patch which is relying on RCV_MSS (our guess about MSS used by the peer which is equal to TCP_MSS_DEFAULT 536 bytes by default) this should work regardless the receiver configured MSS. I am also sharing  my iperf test results with and without the patch and also verified  that the connection won’t get stuck in the middle in case of packet loss or latency spike which I emulated using tc netem on the sender side.
>
>
> Test Results using the same sender & receiver:
>
> -Without our proposed patch
>
> #iperf3 -c xx.xx.xx.xx -t15 -i1 -R
> Connecting to host xx.xx.xx.xx, port 5201
> Reverse mode, remote host xx.xx.xx.xx is sending
> [  4] local 172.31.37.167 port 52838 connected to xx.xx.xx.xx port 5201
> [ ID] Interval           Transfer     Bandwidth
> [  4]   0.00-1.00   sec   269 KBytes  2.20 Mbits/sec
> [  4]   1.00-2.00   sec   332 KBytes  2.72 Mbits/sec
> [  4]   2.00-3.00   sec   334 KBytes  2.73 Mbits/sec
> [  4]   3.00-4.00   sec   335 KBytes  2.75 Mbits/sec
> [  4]   4.00-5.00   sec   332 KBytes  2.72 Mbits/sec
> [  4]   5.00-6.00   sec   283 KBytes  2.32 Mbits/sec
> [  4]   6.00-7.00   sec   332 KBytes  2.72 Mbits/sec
> [  4]   7.00-8.00   sec   335 KBytes  2.75 Mbits/sec
> [  4]   8.00-9.00   sec   335 KBytes  2.75 Mbits/sec
> [  4]   9.00-10.00  sec   334 KBytes  2.73 Mbits/sec
> [  4]  10.00-11.00  sec   332 KBytes  2.72 Mbits/sec
> [  4]  11.00-12.00  sec   332 KBytes  2.72 Mbits/sec
> [  4]  12.00-13.00  sec   338 KBytes  2.77 Mbits/sec
> [  4]  13.00-14.00  sec   334 KBytes  2.73 Mbits/sec
> [  4]  14.00-15.00  sec   332 KBytes  2.72 Mbits/sec
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bandwidth       Retr
> [  4]   0.00-15.00  sec  6.07 MBytes  3.39 Mbits/sec    0             sender
> [  4]   0.00-15.00  sec  4.90 MBytes  2.74 Mbits/sec                  receiver
>
> iperf Done.
>
>
> Test downloading from google endpoint:
>
> # wget https://storage.googleapis.com/kubernetes-release/release/v1.18.9/bin/linux/amd64/kubelet
> --2020-12-04 16:53:00--  https://storage.googleapis.com/kubernetes-release/release/v1.18.9/bin/linux/amd64/kubelet
> Resolving storage.googleapis.com (storage.googleapis.com)... 172.217.1.48, 172.217.8.176, 172.217.4.48, ...
> Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.1.48|:443... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 113320760 (108M) [application/octet-stream]
> Saving to: ‘kubelet.45’
>
> 100%[===================================================================================================================================================>] 113,320,760 3.04MB/s   in 36s
>
> 2020-12-04 16:53:36 (3.02 MB/s) - ‘kubelet’ saved [113320760/113320760]
>
>
> ########################################################################################################################
>
> -With the proposed  patch:
>
> #iperf3 -c xx.xx.xx.xx -t15 -i1 -R
> Connecting to host xx.xx.xx.xx, port 5201
> Reverse mode, remote host xx.xx.xx.xx is sending
> [  4] local 172.31.37.167 port 44514 connected to xx.xx.xx.xx port 5201
> [ ID] Interval           Transfer     Bandwidth
> [  4]   0.00-1.00   sec   911 KBytes  7.46 Mbits/sec
> [  4]   1.00-2.00   sec  8.95 MBytes  75.1 Mbits/sec
> [  4]   2.00-3.00   sec  9.57 MBytes  80.3 Mbits/sec
> [  4]   3.00-4.00   sec  9.56 MBytes  80.2 Mbits/sec
> [  4]   4.00-5.00   sec  9.58 MBytes  80.3 Mbits/sec
> [  4]   5.00-6.00   sec  9.58 MBytes  80.4 Mbits/sec
> [  4]   6.00-7.00   sec  9.59 MBytes  80.4 Mbits/sec
> [  4]   7.00-8.00   sec  9.59 MBytes  80.5 Mbits/sec
> [  4]   8.00-9.00   sec  9.58 MBytes  80.4 Mbits/sec
> [  4]   9.00-10.00  sec  9.58 MBytes  80.4 Mbits/sec
> [  4]  10.00-11.00  sec  9.59 MBytes  80.4 Mbits/sec
> [  4]  11.00-12.00  sec  9.59 MBytes  80.5 Mbits/sec
> [  4]  12.00-13.00  sec  8.05 MBytes  67.5 Mbits/sec
> [  4]  13.00-14.00  sec  9.57 MBytes  80.3 Mbits/sec
> [  4]  14.00-15.00  sec  9.57 MBytes  80.3 Mbits/sec
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bandwidth       Retr
> [  4]   0.00-15.00  sec   136 MBytes  76.3 Mbits/sec    0             sender
> [  4]   0.00-15.00  sec   134 MBytes  75.2 Mbits/sec                  receiver
>
> iperf Done.
>
> Test downloading from google endpoint:
>
>
> # wget https://storage.googleapis.com/kubernetes-release/release/v1.18.9/bin/linux/amd64/kubelet
> --2020-12-04 16:54:34--  https://storage.googleapis.com/kubernetes-release/release/v1.18.9/bin/linux/amd64/kubelet
> Resolving storage.googleapis.com (storage.googleapis.com)... 172.217.0.16, 216.58.192.144, 172.217.6.16, ...
> Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.0.16|:443... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 113320760 (108M) [application/octet-stream]
> Saving to: ‘kubelet’
>
> 100%[===================================================================================================================================================>] 113,320,760 80.0MB/s   in 1.4s
>
> 2020-12-04 16:54:36 (80.0 MB/s) - ‘kubelet.1’ saved [113320760/113320760]
>
> Links:
>
> [1] https://github.com/kubernetes/kops/issues/10206
> [2] https://lore.kernel.org/patchwork/patch/1157936/
> [3] https://lore.kernel.org/patchwork/patch/1157883/
>

Unfortunately few things are missing in this report.

What is the RTT between hosts in your test ?

What driver is used at the receiving side ?

Usually, this kind of problem comes when s(kb->len / skb->truesize) is
pathologically small.
This could be caused by a driver lacking scatter gather support at RX
(a 1500 bytes incoming packet would use 12KB of memory or so, because
driver MTU was set to 9000)

Also worth noting that if you set MTU to 9000 (instead of standard
1500), you probably need to tweak a few sysctls.

autotuning is tricky, changing initial values can be good in some
cases, bad in others.

It would be nice if you send "ss -temoi"  output taken at receiver
while transfer is in progress.

>
>
> Thank you.
>
> Hazem
>
> On 04/12/2020, 18:08, "Hazem Mohamed Abuelfotoh" <abuehaze@amazon.com> wrote:
>
>         Previously receiver buffer auto-tuning starts after receiving
>         one advertised window amount of data.After the initial
>         receiver buffer was raised by
>         commit a337531b942b ("tcp: up initial rmem to 128KB
>         and SYN rwin to around 64KB"),the receiver buffer may
>         take too long for TCP autotuning to start raising
>         the receiver buffer size.
>         commit 041a14d26715 ("tcp: start receiver buffer autotuning sooner")
>         tried to decrease the threshold at which TCP auto-tuning starts
>         but it's doesn't work well in some environments
>         where the receiver has large MTU (9001) configured
>         specially within environments where RTT is high.
>         To address this issue this patch is relying on RCV_MSS
>         so auto-tuning can start early regardless
>         the receiver configured MTU.
>
>         Fixes: a337531b942b ("tcp: up initial rmem to 128KB and SYN rwin to around 64KB")
>         Fixes: 041a14d26715 ("tcp: start receiver buffer autotuning sooner")
>
>     Signed-off-by: Hazem Mohamed Abuelfotoh <abuehaze@amazon.com>
>     ---
>      net/ipv4/tcp_input.c | 3 ++-
>      1 file changed, 2 insertions(+), 1 deletion(-)
>
>     diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
>     index 389d1b340248..f0ffac9e937b 100644
>     --- a/net/ipv4/tcp_input.c
>     +++ b/net/ipv4/tcp_input.c
>     @@ -504,13 +504,14 @@ static void tcp_grow_window(struct sock *sk, const struct sk_buff *skb)
>      static void tcp_init_buffer_space(struct sock *sk)
>      {
>         int tcp_app_win = sock_net(sk)->ipv4.sysctl_tcp_app_win;
>     +   struct inet_connection_sock *icsk = inet_csk(sk);
>         struct tcp_sock *tp = tcp_sk(sk);
>         int maxwin;
>
>         if (!(sk->sk_userlocks & SOCK_SNDBUF_LOCK))
>                 tcp_sndbuf_expand(sk);
>
>     -   tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND * tp->advmss);
>     +   tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND * icsk->icsk_ack.rcv_mss);
>         tcp_mstamp_refresh(tp);
>         tp->rcvq_space.time = tp->tcp_mstamp;
>         tp->rcvq_space.seq = tp->copied_seq;
>     --
>     2.16.6
>
>
>
>
>
> Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284
>
> Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705
>
>

^ permalink raw reply	[flat|nested] 22+ messages in thread

[parent not found: <3F02FF08-EDA6-4DFD-8D93-479A5B05E25A@amazon.com>]

* Re: [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections
       [not found]     ` <3F02FF08-EDA6-4DFD-8D93-479A5B05E25A@amazon.com>
@ 2020-12-07 15:25       ` Eric Dumazet
  2020-12-07 16:09         ` Mohamed Abuelfotoh, Hazem
  0 siblings, 1 reply; 22+ messages in thread
From: Eric Dumazet @ 2020-12-07 15:25 UTC (permalink / raw)
  To: Mohamed Abuelfotoh, Hazem
  Cc: netdev, stable, ycheng, ncardwell, weiwan, Strohman, Andy,
	Herrenschmidt, Benjamin

On Sat, Dec 5, 2020 at 1:03 PM Mohamed Abuelfotoh, Hazem
<abuehaze@amazon.com> wrote:
>
> Unfortunately few things are missing in this report.
>
>     What is the RTT between hosts in your test ?
>      >>>>>RTT in my test is 162 msec, but I am able to reproduce it with lower RTTs for example I could see the issue downloading from google   endpoint with RTT of 16.7 msec, as mentioned in my previous e-mail the issue is reproducible whenever RTT exceeded 12msec given that    the sender is using bbr.
>
>         RTT between hosts where I run the iperf test.
>         # ping 54.199.163.187
>         PING 54.199.163.187 (54.199.163.187) 56(84) bytes of data.
>         64 bytes from 54.199.163.187: icmp_seq=1 ttl=33 time=162 ms
>         64 bytes from 54.199.163.187: icmp_seq=2 ttl=33 time=162 ms
>         64 bytes from 54.199.163.187: icmp_seq=3 ttl=33 time=162 ms
>         64 bytes from 54.199.163.187: icmp_seq=4 ttl=33 time=162 ms
>
>         RTT between my EC2 instances and google endpoint.
>         # ping 172.217.4.240
>         PING 172.217.4.240 (172.217.4.240) 56(84) bytes of data.
>         64 bytes from 172.217.4.240: icmp_seq=1 ttl=101 time=16.7 ms
>         64 bytes from 172.217.4.240: icmp_seq=2 ttl=101 time=16.7 ms
>         64 bytes from 172.217.4.240: icmp_seq=3 ttl=101 time=16.7 ms
>         64 bytes from 172.217.4.240: icmp_seq=4 ttl=101 time=16.7 ms
>
>     What driver is used at the receiving side ?
>       >>>>>>I am using ENA driver version version: 2.2.10g on the receiver with scatter gathering enabled.
>
>         # ethtool -k eth0 | grep scatter-gather
>         scatter-gather: on
>                 tx-scatter-gather: on
>                 tx-scatter-gather-fraglist: off [fixed]

This ethtool output refers to TX scatter gather, which is not relevant
for this bug.

I see ENA driver might use 16 KB per incoming packet (if ENA_PAGE_SIZE is 16 KB)

Since I can not reproduce this problem with another NIC on x86, I
really wonder if this is not an issue with ENA driver on PowerPC
perhaps ?

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections
  2020-12-07 15:25       ` Eric Dumazet
@ 2020-12-07 16:09         ` Mohamed Abuelfotoh, Hazem
  2020-12-07 16:22           ` Eric Dumazet
  0 siblings, 1 reply; 22+ messages in thread
From: Mohamed Abuelfotoh, Hazem @ 2020-12-07 16:09 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: netdev, stable, ycheng, ncardwell, weiwan, Strohman, Andy,
	Herrenschmidt, Benjamin

    >Since I can not reproduce this problem with another NIC on x86, I
    >really wonder if this is not an issue with ENA driver on PowerPC
    >perhaps ?


I am able to reproduce it on x86 based EC2 instances using ENA  or  Xen netfront or Intel ixgbevf driver on the receiver so it's not specific to ENA, we were able to easily reproduce it between 2 VMs running in virtual box on the same physical host considering the environment requirements I mentioned in my first e-mail.

What's the RTT between the sender & receiver in your reproduction? Are you using bbr on the sender side?

Thank you.

Hazem

On 07/12/2020, 15:26, "Eric Dumazet" <edumazet@google.com> wrote:

    CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



    On Sat, Dec 5, 2020 at 1:03 PM Mohamed Abuelfotoh, Hazem
    <abuehaze@amazon.com> wrote:
    >
    > Unfortunately few things are missing in this report.
    >
    >     What is the RTT between hosts in your test ?
    >      >>>>>RTT in my test is 162 msec, but I am able to reproduce it with lower RTTs for example I could see the issue downloading from google   endpoint with RTT of 16.7 msec, as mentioned in my previous e-mail the issue is reproducible whenever RTT exceeded 12msec given that    the sender is using bbr.
    >
    >         RTT between hosts where I run the iperf test.
    >         # ping 54.199.163.187
    >         PING 54.199.163.187 (54.199.163.187) 56(84) bytes of data.
    >         64 bytes from 54.199.163.187: icmp_seq=1 ttl=33 time=162 ms
    >         64 bytes from 54.199.163.187: icmp_seq=2 ttl=33 time=162 ms
    >         64 bytes from 54.199.163.187: icmp_seq=3 ttl=33 time=162 ms
    >         64 bytes from 54.199.163.187: icmp_seq=4 ttl=33 time=162 ms
    >
    >         RTT between my EC2 instances and google endpoint.
    >         # ping 172.217.4.240
    >         PING 172.217.4.240 (172.217.4.240) 56(84) bytes of data.
    >         64 bytes from 172.217.4.240: icmp_seq=1 ttl=101 time=16.7 ms
    >         64 bytes from 172.217.4.240: icmp_seq=2 ttl=101 time=16.7 ms
    >         64 bytes from 172.217.4.240: icmp_seq=3 ttl=101 time=16.7 ms
    >         64 bytes from 172.217.4.240: icmp_seq=4 ttl=101 time=16.7 ms
    >
    >     What driver is used at the receiving side ?
    >       >>>>>>I am using ENA driver version version: 2.2.10g on the receiver with scatter gathering enabled.
    >
    >         # ethtool -k eth0 | grep scatter-gather
    >         scatter-gather: on
    >                 tx-scatter-gather: on
    >                 tx-scatter-gather-fraglist: off [fixed]

    This ethtool output refers to TX scatter gather, which is not relevant
    for this bug.

    I see ENA driver might use 16 KB per incoming packet (if ENA_PAGE_SIZE is 16 KB)

    Since I can not reproduce this problem with another NIC on x86, I
    really wonder if this is not an issue with ENA driver on PowerPC
    perhaps ?




Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284

Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705



^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections
  2020-12-07 16:09         ` Mohamed Abuelfotoh, Hazem
@ 2020-12-07 16:22           ` Eric Dumazet
  2020-12-07 16:33             ` Neal Cardwell
  2020-12-07 16:34             ` Mohamed Abuelfotoh, Hazem
  0 siblings, 2 replies; 22+ messages in thread
From: Eric Dumazet @ 2020-12-07 16:22 UTC (permalink / raw)
  To: Mohamed Abuelfotoh, Hazem
  Cc: netdev, stable, ycheng, ncardwell, weiwan, Strohman, Andy,
	Herrenschmidt, Benjamin

On Mon, Dec 7, 2020 at 5:09 PM Mohamed Abuelfotoh, Hazem
<abuehaze@amazon.com> wrote:
>
>     >Since I can not reproduce this problem with another NIC on x86, I
>     >really wonder if this is not an issue with ENA driver on PowerPC
>     >perhaps ?
>
>
> I am able to reproduce it on x86 based EC2 instances using ENA  or  Xen netfront or Intel ixgbevf driver on the receiver so it's not specific to ENA, we were able to easily reproduce it between 2 VMs running in virtual box on the same physical host considering the environment requirements I mentioned in my first e-mail.
>
> What's the RTT between the sender & receiver in your reproduction? Are you using bbr on the sender side?


100ms RTT

Which exact version of linux kernel are you using ?



>
> Thank you.
>
> Hazem
>
> On 07/12/2020, 15:26, "Eric Dumazet" <edumazet@google.com> wrote:
>
>     CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
>     On Sat, Dec 5, 2020 at 1:03 PM Mohamed Abuelfotoh, Hazem
>     <abuehaze@amazon.com> wrote:
>     >
>     > Unfortunately few things are missing in this report.
>     >
>     >     What is the RTT between hosts in your test ?
>     >      >>>>>RTT in my test is 162 msec, but I am able to reproduce it with lower RTTs for example I could see the issue downloading from google   endpoint with RTT of 16.7 msec, as mentioned in my previous e-mail the issue is reproducible whenever RTT exceeded 12msec given that    the sender is using bbr.
>     >
>     >         RTT between hosts where I run the iperf test.
>     >         # ping 54.199.163.187
>     >         PING 54.199.163.187 (54.199.163.187) 56(84) bytes of data.
>     >         64 bytes from 54.199.163.187: icmp_seq=1 ttl=33 time=162 ms
>     >         64 bytes from 54.199.163.187: icmp_seq=2 ttl=33 time=162 ms
>     >         64 bytes from 54.199.163.187: icmp_seq=3 ttl=33 time=162 ms
>     >         64 bytes from 54.199.163.187: icmp_seq=4 ttl=33 time=162 ms
>     >
>     >         RTT between my EC2 instances and google endpoint.
>     >         # ping 172.217.4.240
>     >         PING 172.217.4.240 (172.217.4.240) 56(84) bytes of data.
>     >         64 bytes from 172.217.4.240: icmp_seq=1 ttl=101 time=16.7 ms
>     >         64 bytes from 172.217.4.240: icmp_seq=2 ttl=101 time=16.7 ms
>     >         64 bytes from 172.217.4.240: icmp_seq=3 ttl=101 time=16.7 ms
>     >         64 bytes from 172.217.4.240: icmp_seq=4 ttl=101 time=16.7 ms
>     >
>     >     What driver is used at the receiving side ?
>     >       >>>>>>I am using ENA driver version version: 2.2.10g on the receiver with scatter gathering enabled.
>     >
>     >         # ethtool -k eth0 | grep scatter-gather
>     >         scatter-gather: on
>     >                 tx-scatter-gather: on
>     >                 tx-scatter-gather-fraglist: off [fixed]
>
>     This ethtool output refers to TX scatter gather, which is not relevant
>     for this bug.
>
>     I see ENA driver might use 16 KB per incoming packet (if ENA_PAGE_SIZE is 16 KB)
>
>     Since I can not reproduce this problem with another NIC on x86, I
>     really wonder if this is not an issue with ENA driver on PowerPC
>     perhaps ?
>
>
>
>
> Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284
>
> Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705
>
>

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections
  2020-12-07 16:22           ` Eric Dumazet
@ 2020-12-07 16:33             ` Neal Cardwell
  2020-12-07 17:08               ` Eric Dumazet
  2020-12-07 17:16               ` Mohamed Abuelfotoh, Hazem
  2020-12-07 16:34             ` Mohamed Abuelfotoh, Hazem
  1 sibling, 2 replies; 22+ messages in thread
From: Neal Cardwell @ 2020-12-07 16:33 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: Mohamed Abuelfotoh, Hazem, netdev, stable, ycheng, weiwan,
	Strohman, Andy, Herrenschmidt, Benjamin

On Mon, Dec 7, 2020 at 11:23 AM Eric Dumazet <edumazet@google.com> wrote:
>
> On Mon, Dec 7, 2020 at 5:09 PM Mohamed Abuelfotoh, Hazem
> <abuehaze@amazon.com> wrote:
> >
> >     >Since I can not reproduce this problem with another NIC on x86, I
> >     >really wonder if this is not an issue with ENA driver on PowerPC
> >     >perhaps ?
> >
> >
> > I am able to reproduce it on x86 based EC2 instances using ENA  or  Xen netfront or Intel ixgbevf driver on the receiver so it's not specific to ENA, we were able to easily reproduce it between 2 VMs running in virtual box on the same physical host considering the environment requirements I mentioned in my first e-mail.
> >
> > What's the RTT between the sender & receiver in your reproduction? Are you using bbr on the sender side?
>
>
> 100ms RTT
>
> Which exact version of linux kernel are you using ?

Thanks for testing this, Eric. Would you be able to share the MTU
config commands you used, and the tcpdump traces you get? I'm
surprised that receive buffer autotuning would work for advmss of
around 6500 or higher.

thanks,
neal

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections
  2020-12-07 16:33             ` Neal Cardwell
@ 2020-12-07 17:08               ` Eric Dumazet
  2020-12-07 20:09                 ` Mohamed Abuelfotoh, Hazem
  2020-12-07 17:16               ` Mohamed Abuelfotoh, Hazem
  1 sibling, 1 reply; 22+ messages in thread
From: Eric Dumazet @ 2020-12-07 17:08 UTC (permalink / raw)
  To: Neal Cardwell
  Cc: Mohamed Abuelfotoh, Hazem, netdev, stable, ycheng, weiwan,
	Strohman, Andy, Herrenschmidt, Benjamin

On Mon, Dec 7, 2020 at 5:34 PM Neal Cardwell <ncardwell@google.com> wrote:
>
> On Mon, Dec 7, 2020 at 11:23 AM Eric Dumazet <edumazet@google.com> wrote:
> >
> > On Mon, Dec 7, 2020 at 5:09 PM Mohamed Abuelfotoh, Hazem
> > <abuehaze@amazon.com> wrote:
> > >
> > >     >Since I can not reproduce this problem with another NIC on x86, I
> > >     >really wonder if this is not an issue with ENA driver on PowerPC
> > >     >perhaps ?
> > >
> > >
> > > I am able to reproduce it on x86 based EC2 instances using ENA  or  Xen netfront or Intel ixgbevf driver on the receiver so it's not specific to ENA, we were able to easily reproduce it between 2 VMs running in virtual box on the same physical host considering the environment requirements I mentioned in my first e-mail.
> > >
> > > What's the RTT between the sender & receiver in your reproduction? Are you using bbr on the sender side?
> >
> >
> > 100ms RTT
> >
> > Which exact version of linux kernel are you using ?
>
> Thanks for testing this, Eric. Would you be able to share the MTU
> config commands you used, and the tcpdump traces you get? I'm
> surprised that receive buffer autotuning would work for advmss of
> around 6500 or higher.

autotuning might be delayed by one RTT, this does not match numbers
given by Mohamed (flows stuck in low speed)

autotuning is an heuristic, and because it has one RTT latency, it is
crucial to get proper initial rcvmem values.

People using MTU=9000 should know they have to tune tcp_rmem[1]
accordingly, especially when using drivers consuming one page per
incoming MSS.


(mlx4 driver only uses ome 2048 bytes fragment for a 1500 MTU packet.
even with MTU set to 9000)

I want to state again that using 536 bytes as a magic value makes no
sense to me.


For the record, Google has increased tcp_rmem[1] when switching to a bigger MTU.

The reason is simple : If we intend to receive 10 MSS, we should allow
for 90000 bytes of payload, or tcp_rmem[1] set to 180,000
Because of autotuning latency, doubling the value is advised : 360000

Another problem with kicking autotuning too fast is that it might
allow bigger sk->sk_rcvbuf values even for small flows, opening more
surface to malicious attacks.

I _think_ that if we want to allow admins to set high MTU without
having to tune tcp_rmem[], we need something different than current
proposal.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections
  2020-12-07 17:08               ` Eric Dumazet
@ 2020-12-07 20:09                 ` Mohamed Abuelfotoh, Hazem
  2020-12-07 23:22                   ` Eric Dumazet
  0 siblings, 1 reply; 22+ messages in thread
From: Mohamed Abuelfotoh, Hazem @ 2020-12-07 20:09 UTC (permalink / raw)
  To: Eric Dumazet, Neal Cardwell
  Cc: netdev, stable, ycheng, weiwan, Strohman, Andy, Herrenschmidt, Benjamin

    >I want to state again that using 536 bytes as a magic value makes no
    sense to me.

 >autotuning might be delayed by one RTT, this does not match numbers
 >given by Mohamed (flows stuck in low speed)

  >autotuning is an heuristic, and because it has one RTT latency, it is
   >crucial to get proper initial rcvmem values.

   >People using MTU=9000 should know they have to tune tcp_rmem[1]
   >accordingly, especially when using drivers consuming one page per
   >+incoming MSS.



The magic number would be 10*rcv_mss=5360 not 536 and in my opinion it's a big amount of data to be sent in security attack so if we are talking about DDos attack triggering Autotuning at 5360 bytes I'd say he will also be able to trigger it sending 64KB but I totally agree that it would be easier with lower rcvq_space.space, it's always a tradeoff between security and performance.

Other options would be to either consider the configured MTU in the rcv_wnd calculation or probably check the MTU before calculating the initial rcvspace. We have to make sure that initial receive space is lower than initial receive window so Autotuning would work regardless the configured MTU on the receiver and only people using Jumbo frames will be paying the price if we agreed that it's expected for Jumbo frame users to have machines with more memory,  I'd say something as below should work:

void tcp_init_buffer_space(struct sock *sk)
{
	int tcp_app_win = sock_net(sk)->ipv4.sysctl_tcp_app_win;
	struct inet_connection_sock *icsk = inet_csk(sk);
	struct tcp_sock *tp = tcp_sk(sk);
	int maxwin;

	if (!(sk->sk_userlocks & SOCK_SNDBUF_LOCK))
		tcp_sndbuf_expand(sk);
	if(tp->advmss < 6000)
		tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND * tp->advmss);
	else
		tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND * icsk->icsk_ack.rcv_mss);
	tcp_mstamp_refresh(tp);
	tp->rcvq_space.time = tp->tcp_mstamp;
	tp->rcvq_space.seq = tp->copied_seq;



I don't think that we should rely on Admins manually tuning this tcp_rmem[1] with Jumbo frame in use also Linux users shouldn't expect performance degradation after kernel upgrade. although [1] is the only public reporting of this issue, I am pretty sure we will see more users reporting this with Linux Main distributions moving to kernel 5.4 as stable version. In Summary we should come up with something either the proposed patch or something else to avoid admins doing the manual job.


Links

[1] https://github.com/kubernetes/kops/issues/10206

On 07/12/2020, 17:08, "Eric Dumazet" <edumazet@google.com> wrote:

    CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



    On Mon, Dec 7, 2020 at 5:34 PM Neal Cardwell <ncardwell@google.com> wrote:
    >
    > On Mon, Dec 7, 2020 at 11:23 AM Eric Dumazet <edumazet@google.com> wrote:
    > >
    > > On Mon, Dec 7, 2020 at 5:09 PM Mohamed Abuelfotoh, Hazem
    > > <abuehaze@amazon.com> wrote:
    > > >
    > > >     >Since I can not reproduce this problem with another NIC on x86, I
    > > >     >really wonder if this is not an issue with ENA driver on PowerPC
    > > >     >perhaps ?
    > > >
    > > >
    > > > I am able to reproduce it on x86 based EC2 instances using ENA  or  Xen netfront or Intel ixgbevf driver on the receiver so it's not specific to ENA, we were able to easily reproduce it between 2 VMs running in virtual box on the same physical host considering the environment requirements I mentioned in my first e-mail.
    > > >
    > > > What's the RTT between the sender & receiver in your reproduction? Are you using bbr on the sender side?
    > >
    > >
    > > 100ms RTT
    > >
    > > Which exact version of linux kernel are you using ?
    >
    > Thanks for testing this, Eric. Would you be able to share the MTU
    > config commands you used, and the tcpdump traces you get? I'm
    > surprised that receive buffer autotuning would work for advmss of
    > around 6500 or higher.

    autotuning might be delayed by one RTT, this does not match numbers
    given by Mohamed (flows stuck in low speed)

    autotuning is an heuristic, and because it has one RTT latency, it is
    crucial to get proper initial rcvmem values.

    People using MTU=9000 should know they have to tune tcp_rmem[1]
    accordingly, especially when using drivers consuming one page per
    incoming MSS.


    (mlx4 driver only uses ome 2048 bytes fragment for a 1500 MTU packet.
    even with MTU set to 9000)

    I want to state again that using 536 bytes as a magic value makes no
    sense to me.


    For the record, Google has increased tcp_rmem[1] when switching to a bigger MTU.

    The reason is simple : If we intend to receive 10 MSS, we should allow
    for 90000 bytes of payload, or tcp_rmem[1] set to 180,000
    Because of autotuning latency, doubling the value is advised : 360000

    Another problem with kicking autotuning too fast is that it might
    allow bigger sk->sk_rcvbuf values even for small flows, opening more
    surface to malicious attacks.

    I _think_ that if we want to allow admins to set high MTU without
    having to tune tcp_rmem[], we need something different than current
    proposal.




Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284

Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705



^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections
  2020-12-07 20:09                 ` Mohamed Abuelfotoh, Hazem
@ 2020-12-07 23:22                   ` Eric Dumazet
  0 siblings, 0 replies; 22+ messages in thread
From: Eric Dumazet @ 2020-12-07 23:22 UTC (permalink / raw)
  To: Mohamed Abuelfotoh, Hazem
  Cc: Neal Cardwell, netdev, stable, ycheng, weiwan, Strohman, Andy,
	Herrenschmidt, Benjamin

On Mon, Dec 7, 2020 at 9:09 PM Mohamed Abuelfotoh, Hazem
<abuehaze@amazon.com> wrote:
>
>     >I want to state again that using 536 bytes as a magic value makes no
>     sense to me.
>
>  >autotuning might be delayed by one RTT, this does not match numbers
>  >given by Mohamed (flows stuck in low speed)
>
>   >autotuning is an heuristic, and because it has one RTT latency, it is
>    >crucial to get proper initial rcvmem values.
>
>    >People using MTU=9000 should know they have to tune tcp_rmem[1]
>    >accordingly, especially when using drivers consuming one page per
>    >+incoming MSS.
>
>
>
> The magic number would be 10*rcv_mss=5360 not 536 and in my opinion it's a big amount of data to be sent in security attack so if we are talking about DDos attack triggering Autotuning at 5360 bytes I'd say he will also be able to trigger it sending 64KB but I totally agree that it would be easier with lower rcvq_space.space, it's always a tradeoff between security and performance.



>
> Other options would be to either consider the configured MTU in the rcv_wnd calculation or probably check the MTU before calculating the initial rcvspace. We have to make sure that initial receive space is lower than initial receive window so Autotuning would work regardless the configured MTU on the receiver and only people using Jumbo frames will be paying the price if we agreed that it's expected for Jumbo frame users to have machines with more memory,  I'd say something as below should work:
>
> void tcp_init_buffer_space(struct sock *sk)
> {
>         int tcp_app_win = sock_net(sk)->ipv4.sysctl_tcp_app_win;
>         struct inet_connection_sock *icsk = inet_csk(sk);
>         struct tcp_sock *tp = tcp_sk(sk);
>         int maxwin;
>
>         if (!(sk->sk_userlocks & SOCK_SNDBUF_LOCK))
>                 tcp_sndbuf_expand(sk);
>         if(tp->advmss < 6000)
>                 tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND * tp->advmss);

This is just another hack, based on 'magic' numbers.

>         else
>                 tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND * icsk->icsk_ack.rcv_mss);
>         tcp_mstamp_refresh(tp);
>         tp->rcvq_space.time = tp->tcp_mstamp;
>         tp->rcvq_space.seq = tp->copied_seq;
>
>
>
> I don't think that we should rely on Admins manually tuning this tcp_rmem[1] with Jumbo frame in use also Linux users shouldn't expect performance degradation after kernel upgrade. although [1] is the only public reporting of this issue, I am pretty sure we will see more users reporting this with Linux Main distributions moving to kernel 5.4 as stable version. In Summary we should come up with something either the proposed patch or something else to avoid admins doing the manual job.
>



Default MTU is 1500, not 9000.

I hinted in my very first reply to you that MTU  9000 is not easy and
needs tuning. We could argue and try to make this less of a pain in
future kernel (net-next)

<quote>Also worth noting that if you set MTU to 9000 (instead of
standard 1500), you probably need to tweak a few sysctls.
</quote>

I think I have asked you multiple times to test appropriate
tcp_rmem[1] settings...

I gave the reason why tcp_rmem[1] set to 131072 is not good for MTU
9000, I will prefer a solution that involves no kernel patch, no
backports, just a matter of educating sysadmins, for increased TCP
performance,
especially when really using 9000 MTU...

Your patch would change the behavior of TCP stack for standard
MTU=1500 flows which are yet the majority. This is very risky.

Anyway. _if_ we really wanted to change the kernel, ( keeping stupid
tcp_rmem[1] value ) :

In the tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND *
tp->advmss);  formula, really the bug is in the tp->rcv_wnd term, not
the second one.

This is buggy, because tcp_init_buffer_space() ends up with
tp->window_clamp smaller than tp->rcv_wnd, so tcp_grow_window() is not
able to change tp->rcv_ssthresh

The only mechanism allowing to change tp->window_clamp later would be
DRS, so we better use the proper limit when initializing
tp->rcvq_space.space

This issue disappears if tcp_rmem[1] is slightly above 131072, because
then the following is not needed.

diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 9e8a6c1aa0190cc248b3b99b073a4c6e45884cf5..81b5d9375860ae583e08045fb25b089c456c60ab
100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -534,6 +534,7 @@ static void tcp_init_buffer_space(struct sock *sk)

        tp->rcv_ssthresh = min(tp->rcv_ssthresh, tp->window_clamp);
        tp->snd_cwnd_stamp = tcp_jiffies32;
+       tp->rcvq_space.space = min(tp->rcv_ssthresh, tp->rcvq_space.space);
 }

 /* 4. Recalculate window clamp after socket hit its memory bounds. */

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections
  2020-12-07 16:33             ` Neal Cardwell
  2020-12-07 17:08               ` Eric Dumazet
@ 2020-12-07 17:16               ` Mohamed Abuelfotoh, Hazem
  2020-12-07 17:27                 ` Eric Dumazet
  1 sibling, 1 reply; 22+ messages in thread
From: Mohamed Abuelfotoh, Hazem @ 2020-12-07 17:16 UTC (permalink / raw)
  To: Neal Cardwell, Eric Dumazet
  Cc: netdev, stable, ycheng, weiwan, Strohman, Andy, Herrenschmidt, Benjamin

[-- Attachment #1: Type: text/plain, Size: 3060 bytes --]

    >Thanks for testing this, Eric. Would you be able to share the MTU
    >config commands you used, and the tcpdump traces you get? I'm
    >surprised that receive buffer autotuning would work for advmss of
    >around 6500 or higher.

Packet capture before applying the proposed patch

https://tcpautotuningpcaps.s3.eu-west-1.amazonaws.com/sender-bbr-bad-unpatched.pcap?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJNMP5ZZ3I4FAQGAQ%2F20201207%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20201207T170123Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=a599a0e0e6632a957e5619007ba5ce4f63c8e8535ea24470b7093fef440a8300

Packet capture after applying the proposed patch

https://tcpautotuningpcaps.s3.eu-west-1.amazonaws.com/sender-bbr-good-patched.pcap?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJNMP5ZZ3I4FAQGAQ%2F20201207%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20201207T165831Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=f18ec7246107590e8ac35c24322af699e4c2a73d174067c51cf6b0a06bbbca77

kernel version & MTU and configuration  from my receiver & sender is attached to this e-mail, please be aware that EC2 is doing MSS clamping so you need to configure MTU as 1500 on the sender side if you don’t have any MSS clamping between sender & receiver.

Thank you.

Hazem


On 07/12/2020, 16:34, "Neal Cardwell" <ncardwell@google.com> wrote:

    CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



    On Mon, Dec 7, 2020 at 11:23 AM Eric Dumazet <edumazet@google.com> wrote:
    >
    > On Mon, Dec 7, 2020 at 5:09 PM Mohamed Abuelfotoh, Hazem
    > <abuehaze@amazon.com> wrote:
    > >
    > >     >Since I can not reproduce this problem with another NIC on x86, I
    > >     >really wonder if this is not an issue with ENA driver on PowerPC
    > >     >perhaps ?
    > >
    > >
    > > I am able to reproduce it on x86 based EC2 instances using ENA  or  Xen netfront or Intel ixgbevf driver on the receiver so it's not specific to ENA, we were able to easily reproduce it between 2 VMs running in virtual box on the same physical host considering the environment requirements I mentioned in my first e-mail.
    > >
    > > What's the RTT between the sender & receiver in your reproduction? Are you using bbr on the sender side?
    >
    >
    > 100ms RTT
    >
    > Which exact version of linux kernel are you using ?

    Thanks for testing this, Eric. Would you be able to share the MTU
    config commands you used, and the tcpdump traces you get? I'm
    surprised that receive buffer autotuning would work for advmss of
    around 6500 or higher.

    thanks,
    neal




Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284

Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705



[-- Attachment #2: receiverconf --]
[-- Type: application/octet-stream, Size: 73007 bytes --]

5.4.74-36.135.amzn2.x86_64
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 172.31.37.167  netmask 255.255.240.0  broadcast 172.31.47.255
        inet6 fe80::8f5:b8ff:fefa:827e  prefixlen 64  scopeid 0x20<link>
        ether 0a:f5:b8:fa:82:7e  txqueuelen 1000  (Ethernet)
        RX packets 322997  bytes 293467446 (279.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 213839  bytes 23587696 (22.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

abi.vsyscall32 = 1
crypto.fips_enabled = 0
debug.exception-trace = 1
debug.kprobes-optimization = 1
dev.hpet.max-user-freq = 64
dev.raid.speed_limit_max = 200000
dev.raid.speed_limit_min = 1000
dev.tty.ldisc_autoload = 1
fs.aio-max-nr = 65536
fs.aio-nr = 0
fs.binfmt_misc.status = enabled
fs.dentry-state = 57902	42848	45	0	13000	0
fs.dir-notify-enable = 1
fs.epoll.max_user_watches = 19795722
fs.file-max = 9663372
fs.file-nr = 1984	0	9663372
fs.inode-nr = 45204	360
fs.inode-state = 45203	360	0	0	0	0	0
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 8192
fs.lease-break-time = 45
fs.leases-enable = 1
fs.mount-max = 100000
fs.mqueue.msg_default = 10
fs.mqueue.msg_max = 10
fs.mqueue.msgsize_default = 8192
fs.mqueue.msgsize_max = 8192
fs.mqueue.queues_max = 256
fs.nr_open = 1048576
fs.overflowgid = 65534
fs.overflowuid = 65534
fs.pipe-max-size = 1048576
fs.pipe-user-pages-hard = 0
fs.pipe-user-pages-soft = 16384
fs.protected_fifos = 0
fs.protected_hardlinks = 1
fs.protected_regular = 0
fs.protected_symlinks = 1
fs.quota.allocated_dquots = 0
fs.quota.cache_hits = 0
fs.quota.drops = 0
fs.quota.free_dquots = 0
fs.quota.lookups = 0
fs.quota.reads = 0
fs.quota.syncs = 0
fs.quota.writes = 0
fs.suid_dumpable = 0
fs.xfs.error_level = 3
fs.xfs.filestream_centisecs = 3000
fs.xfs.inherit_noatime = 1
fs.xfs.inherit_nodefrag = 1
fs.xfs.inherit_nodump = 1
fs.xfs.inherit_nosymlinks = 0
fs.xfs.inherit_sync = 1
fs.xfs.irix_sgid_inherit = 0
fs.xfs.irix_symlink_mode = 0
fs.xfs.panic_mask = 0
fs.xfs.rotorstep = 1
fs.xfs.speculative_cow_prealloc_lifetime = 1800
fs.xfs.speculative_prealloc_lifetime = 300
fs.xfs.stats_clear = 0
fs.xfs.xfssyncd_centisecs = 3000
kernel.acct = 4	2	30
kernel.acpi_video_flags = 0
kernel.auto_msgmni = 0
kernel.bootloader_type = 114
kernel.bootloader_version = 2
kernel.bpf_stats_enabled = 0
kernel.cad_pid = 1
kernel.cap_last_cap = 37
kernel.core_pattern = core
kernel.core_pipe_limit = 0
kernel.core_uses_pid = 1
kernel.ctrl-alt-del = 0
kernel.dmesg_restrict = 0
kernel.domainname = (none)
kernel.firmware_config.force_sysfs_fallback = 0
kernel.firmware_config.ignore_sysfs_fallback = 0
kernel.ftrace_dump_on_oops = 0
kernel.ftrace_enabled = 1
kernel.hardlockup_all_cpu_backtrace = 0
kernel.hardlockup_panic = 0
kernel.hostname = ip-172-31-37-167.us-east-2.compute.internal
kernel.hotplug = /sbin/hotplug
kernel.hung_task_check_count = 4194304
kernel.hung_task_check_interval_secs = 0
kernel.hung_task_panic = 0
kernel.hung_task_timeout_secs = 120
kernel.hung_task_warnings = 10
kernel.io_delay_type = 0
kernel.kexec_load_disabled = 0
kernel.keys.gc_delay = 300
kernel.keys.maxbytes = 20000
kernel.keys.maxkeys = 200
kernel.keys.persistent_keyring_expiry = 259200
kernel.keys.root_maxbytes = 25000000
kernel.keys.root_maxkeys = 1000000
kernel.kptr_restrict = 0
kernel.latencytop = 0
kernel.max_lock_depth = 1024
kernel.modprobe = /sbin/modprobe
kernel.modules_disabled = 0
kernel.msg_next_id = -1
kernel.msgmax = 8192
kernel.msgmnb = 16384
kernel.msgmni = 32000
kernel.ngroups_max = 65536
kernel.nmi_watchdog = 0
kernel.ns_last_pid = 28406
kernel.numa_balancing = 0
kernel.numa_balancing_scan_delay_ms = 1000
kernel.numa_balancing_scan_period_max_ms = 60000
kernel.numa_balancing_scan_period_min_ms = 1000
kernel.numa_balancing_scan_size_mb = 256
kernel.osrelease = 5.4.74-36.135.amzn2.x86_64
kernel.ostype = Linux
kernel.overflowgid = 65534
kernel.overflowuid = 65534
kernel.panic = 30
kernel.panic_on_io_nmi = 0
kernel.panic_on_oops = 0
kernel.panic_on_rcu_stall = 0
kernel.panic_on_unrecovered_nmi = 0
kernel.panic_on_warn = 0
kernel.panic_print = 0
kernel.perf_cpu_time_max_percent = 25
kernel.perf_event_max_contexts_per_stack = 8
kernel.perf_event_max_sample_rate = 100000
kernel.perf_event_max_stack = 127
kernel.perf_event_mlock_kb = 516
kernel.perf_event_paranoid = 2
kernel.pid_max = 49152
kernel.poweroff_cmd = /sbin/poweroff
kernel.print-fatal-signals = 0
kernel.printk = 8	4	1	7
kernel.printk_delay = 0
kernel.printk_devkmsg = ratelimit
kernel.printk_ratelimit = 5
kernel.printk_ratelimit_burst = 10
kernel.pty.max = 4096
kernel.pty.nr = 5
kernel.pty.reserve = 1024
kernel.random.boot_id = 0d65fac5-e5fd-4154-93aa-2518eb2db2b4
kernel.random.entropy_avail = 3158
kernel.random.poolsize = 4096
kernel.random.read_wakeup_threshold = 64
kernel.random.urandom_min_reseed_secs = 60
kernel.random.uuid = ae1c1815-2be2-4404-a6ab-36cd6b3165b9
kernel.random.write_wakeup_threshold = 3072
kernel.randomize_va_space = 2
kernel.real-root-dev = 0
kernel.sched_autogroup_enabled = 0
kernel.sched_cfs_bandwidth_slice_us = 5000
kernel.sched_child_runs_first = 0
kernel.sched_domain.cpu0.domain0.busy_factor = 32
kernel.sched_domain.cpu0.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu0.domain0.flags = 4783
kernel.sched_domain.cpu0.domain0.imbalance_pct = 110
kernel.sched_domain.cpu0.domain0.max_interval = 4
kernel.sched_domain.cpu0.domain0.max_newidle_lb_cost = 552
kernel.sched_domain.cpu0.domain0.min_interval = 2
kernel.sched_domain.cpu0.domain0.name = SMT
kernel.sched_domain.cpu0.domain1.busy_factor = 32
kernel.sched_domain.cpu0.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu0.domain1.flags = 4655
kernel.sched_domain.cpu0.domain1.imbalance_pct = 117
kernel.sched_domain.cpu0.domain1.max_interval = 96
kernel.sched_domain.cpu0.domain1.max_newidle_lb_cost = 4293
kernel.sched_domain.cpu0.domain1.min_interval = 48
kernel.sched_domain.cpu0.domain1.name = MC
kernel.sched_domain.cpu1.domain0.busy_factor = 32
kernel.sched_domain.cpu1.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu1.domain0.flags = 4783
kernel.sched_domain.cpu1.domain0.imbalance_pct = 110
kernel.sched_domain.cpu1.domain0.max_interval = 4
kernel.sched_domain.cpu1.domain0.max_newidle_lb_cost = 79
kernel.sched_domain.cpu1.domain0.min_interval = 2
kernel.sched_domain.cpu1.domain0.name = SMT
kernel.sched_domain.cpu1.domain1.busy_factor = 32
kernel.sched_domain.cpu1.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu1.domain1.flags = 4655
kernel.sched_domain.cpu1.domain1.imbalance_pct = 117
kernel.sched_domain.cpu1.domain1.max_interval = 96
kernel.sched_domain.cpu1.domain1.max_newidle_lb_cost = 1061
kernel.sched_domain.cpu1.domain1.min_interval = 48
kernel.sched_domain.cpu1.domain1.name = MC
kernel.sched_domain.cpu10.domain0.busy_factor = 32
kernel.sched_domain.cpu10.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu10.domain0.flags = 4783
kernel.sched_domain.cpu10.domain0.imbalance_pct = 110
kernel.sched_domain.cpu10.domain0.max_interval = 4
kernel.sched_domain.cpu10.domain0.max_newidle_lb_cost = 608
kernel.sched_domain.cpu10.domain0.min_interval = 2
kernel.sched_domain.cpu10.domain0.name = SMT
kernel.sched_domain.cpu10.domain1.busy_factor = 32
kernel.sched_domain.cpu10.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu10.domain1.flags = 4655
kernel.sched_domain.cpu10.domain1.imbalance_pct = 117
kernel.sched_domain.cpu10.domain1.max_interval = 96
kernel.sched_domain.cpu10.domain1.max_newidle_lb_cost = 2687
kernel.sched_domain.cpu10.domain1.min_interval = 48
kernel.sched_domain.cpu10.domain1.name = MC
kernel.sched_domain.cpu11.domain0.busy_factor = 32
kernel.sched_domain.cpu11.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu11.domain0.flags = 4783
kernel.sched_domain.cpu11.domain0.imbalance_pct = 110
kernel.sched_domain.cpu11.domain0.max_interval = 4
kernel.sched_domain.cpu11.domain0.max_newidle_lb_cost = 2445
kernel.sched_domain.cpu11.domain0.min_interval = 2
kernel.sched_domain.cpu11.domain0.name = SMT
kernel.sched_domain.cpu11.domain1.busy_factor = 32
kernel.sched_domain.cpu11.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu11.domain1.flags = 4655
kernel.sched_domain.cpu11.domain1.imbalance_pct = 117
kernel.sched_domain.cpu11.domain1.max_interval = 96
kernel.sched_domain.cpu11.domain1.max_newidle_lb_cost = 8333
kernel.sched_domain.cpu11.domain1.min_interval = 48
kernel.sched_domain.cpu11.domain1.name = MC
kernel.sched_domain.cpu12.domain0.busy_factor = 32
kernel.sched_domain.cpu12.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu12.domain0.flags = 4783
kernel.sched_domain.cpu12.domain0.imbalance_pct = 110
kernel.sched_domain.cpu12.domain0.max_interval = 4
kernel.sched_domain.cpu12.domain0.max_newidle_lb_cost = 877
kernel.sched_domain.cpu12.domain0.min_interval = 2
kernel.sched_domain.cpu12.domain0.name = SMT
kernel.sched_domain.cpu12.domain1.busy_factor = 32
kernel.sched_domain.cpu12.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu12.domain1.flags = 4655
kernel.sched_domain.cpu12.domain1.imbalance_pct = 117
kernel.sched_domain.cpu12.domain1.max_interval = 96
kernel.sched_domain.cpu12.domain1.max_newidle_lb_cost = 11776
kernel.sched_domain.cpu12.domain1.min_interval = 48
kernel.sched_domain.cpu12.domain1.name = MC
kernel.sched_domain.cpu13.domain0.busy_factor = 32
kernel.sched_domain.cpu13.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu13.domain0.flags = 4783
kernel.sched_domain.cpu13.domain0.imbalance_pct = 110
kernel.sched_domain.cpu13.domain0.max_interval = 4
kernel.sched_domain.cpu13.domain0.max_newidle_lb_cost = 0
kernel.sched_domain.cpu13.domain0.min_interval = 2
kernel.sched_domain.cpu13.domain0.name = SMT
kernel.sched_domain.cpu13.domain1.busy_factor = 32
kernel.sched_domain.cpu13.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu13.domain1.flags = 4655
kernel.sched_domain.cpu13.domain1.imbalance_pct = 117
kernel.sched_domain.cpu13.domain1.max_interval = 96
kernel.sched_domain.cpu13.domain1.max_newidle_lb_cost = 0
kernel.sched_domain.cpu13.domain1.min_interval = 48
kernel.sched_domain.cpu13.domain1.name = MC
kernel.sched_domain.cpu14.domain0.busy_factor = 32
kernel.sched_domain.cpu14.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu14.domain0.flags = 4783
kernel.sched_domain.cpu14.domain0.imbalance_pct = 110
kernel.sched_domain.cpu14.domain0.max_interval = 4
kernel.sched_domain.cpu14.domain0.max_newidle_lb_cost = 105
kernel.sched_domain.cpu14.domain0.min_interval = 2
kernel.sched_domain.cpu14.domain0.name = SMT
kernel.sched_domain.cpu14.domain1.busy_factor = 32
kernel.sched_domain.cpu14.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu14.domain1.flags = 4655
kernel.sched_domain.cpu14.domain1.imbalance_pct = 117
kernel.sched_domain.cpu14.domain1.max_interval = 96
kernel.sched_domain.cpu14.domain1.max_newidle_lb_cost = 1284
kernel.sched_domain.cpu14.domain1.min_interval = 48
kernel.sched_domain.cpu14.domain1.name = MC
kernel.sched_domain.cpu15.domain0.busy_factor = 32
kernel.sched_domain.cpu15.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu15.domain0.flags = 4783
kernel.sched_domain.cpu15.domain0.imbalance_pct = 110
kernel.sched_domain.cpu15.domain0.max_interval = 4
kernel.sched_domain.cpu15.domain0.max_newidle_lb_cost = 965
kernel.sched_domain.cpu15.domain0.min_interval = 2
kernel.sched_domain.cpu15.domain0.name = SMT
kernel.sched_domain.cpu15.domain1.busy_factor = 32
kernel.sched_domain.cpu15.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu15.domain1.flags = 4655
kernel.sched_domain.cpu15.domain1.imbalance_pct = 117
kernel.sched_domain.cpu15.domain1.max_interval = 96
kernel.sched_domain.cpu15.domain1.max_newidle_lb_cost = 15395
kernel.sched_domain.cpu15.domain1.min_interval = 48
kernel.sched_domain.cpu15.domain1.name = MC
kernel.sched_domain.cpu16.domain0.busy_factor = 32
kernel.sched_domain.cpu16.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu16.domain0.flags = 4783
kernel.sched_domain.cpu16.domain0.imbalance_pct = 110
kernel.sched_domain.cpu16.domain0.max_interval = 4
kernel.sched_domain.cpu16.domain0.max_newidle_lb_cost = 367
kernel.sched_domain.cpu16.domain0.min_interval = 2
kernel.sched_domain.cpu16.domain0.name = SMT
kernel.sched_domain.cpu16.domain1.busy_factor = 32
kernel.sched_domain.cpu16.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu16.domain1.flags = 4655
kernel.sched_domain.cpu16.domain1.imbalance_pct = 117
kernel.sched_domain.cpu16.domain1.max_interval = 96
kernel.sched_domain.cpu16.domain1.max_newidle_lb_cost = 1605
kernel.sched_domain.cpu16.domain1.min_interval = 48
kernel.sched_domain.cpu16.domain1.name = MC
kernel.sched_domain.cpu17.domain0.busy_factor = 32
kernel.sched_domain.cpu17.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu17.domain0.flags = 4783
kernel.sched_domain.cpu17.domain0.imbalance_pct = 110
kernel.sched_domain.cpu17.domain0.max_interval = 4
kernel.sched_domain.cpu17.domain0.max_newidle_lb_cost = 33
kernel.sched_domain.cpu17.domain0.min_interval = 2
kernel.sched_domain.cpu17.domain0.name = SMT
kernel.sched_domain.cpu17.domain1.busy_factor = 32
kernel.sched_domain.cpu17.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu17.domain1.flags = 4655
kernel.sched_domain.cpu17.domain1.imbalance_pct = 117
kernel.sched_domain.cpu17.domain1.max_interval = 96
kernel.sched_domain.cpu17.domain1.max_newidle_lb_cost = 506
kernel.sched_domain.cpu17.domain1.min_interval = 48
kernel.sched_domain.cpu17.domain1.name = MC
kernel.sched_domain.cpu18.domain0.busy_factor = 32
kernel.sched_domain.cpu18.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu18.domain0.flags = 4783
kernel.sched_domain.cpu18.domain0.imbalance_pct = 110
kernel.sched_domain.cpu18.domain0.max_interval = 4
kernel.sched_domain.cpu18.domain0.max_newidle_lb_cost = 0
kernel.sched_domain.cpu18.domain0.min_interval = 2
kernel.sched_domain.cpu18.domain0.name = SMT
kernel.sched_domain.cpu18.domain1.busy_factor = 32
kernel.sched_domain.cpu18.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu18.domain1.flags = 4655
kernel.sched_domain.cpu18.domain1.imbalance_pct = 117
kernel.sched_domain.cpu18.domain1.max_interval = 96
kernel.sched_domain.cpu18.domain1.max_newidle_lb_cost = 582
kernel.sched_domain.cpu18.domain1.min_interval = 48
kernel.sched_domain.cpu18.domain1.name = MC
kernel.sched_domain.cpu19.domain0.busy_factor = 32
kernel.sched_domain.cpu19.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu19.domain0.flags = 4783
kernel.sched_domain.cpu19.domain0.imbalance_pct = 110
kernel.sched_domain.cpu19.domain0.max_interval = 4
kernel.sched_domain.cpu19.domain0.max_newidle_lb_cost = 1605
kernel.sched_domain.cpu19.domain0.min_interval = 2
kernel.sched_domain.cpu19.domain0.name = SMT
kernel.sched_domain.cpu19.domain1.busy_factor = 32
kernel.sched_domain.cpu19.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu19.domain1.flags = 4655
kernel.sched_domain.cpu19.domain1.imbalance_pct = 117
kernel.sched_domain.cpu19.domain1.max_interval = 96
kernel.sched_domain.cpu19.domain1.max_newidle_lb_cost = 1385
kernel.sched_domain.cpu19.domain1.min_interval = 48
kernel.sched_domain.cpu19.domain1.name = MC
kernel.sched_domain.cpu2.domain0.busy_factor = 32
kernel.sched_domain.cpu2.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu2.domain0.flags = 4783
kernel.sched_domain.cpu2.domain0.imbalance_pct = 110
kernel.sched_domain.cpu2.domain0.max_interval = 4
kernel.sched_domain.cpu2.domain0.max_newidle_lb_cost = 269
kernel.sched_domain.cpu2.domain0.min_interval = 2
kernel.sched_domain.cpu2.domain0.name = SMT
kernel.sched_domain.cpu2.domain1.busy_factor = 32
kernel.sched_domain.cpu2.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu2.domain1.flags = 4655
kernel.sched_domain.cpu2.domain1.imbalance_pct = 117
kernel.sched_domain.cpu2.domain1.max_interval = 96
kernel.sched_domain.cpu2.domain1.max_newidle_lb_cost = 3490
kernel.sched_domain.cpu2.domain1.min_interval = 48
kernel.sched_domain.cpu2.domain1.name = MC
kernel.sched_domain.cpu20.domain0.busy_factor = 32
kernel.sched_domain.cpu20.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu20.domain0.flags = 4783
kernel.sched_domain.cpu20.domain0.imbalance_pct = 110
kernel.sched_domain.cpu20.domain0.max_interval = 4
kernel.sched_domain.cpu20.domain0.max_newidle_lb_cost = 1665
kernel.sched_domain.cpu20.domain0.min_interval = 2
kernel.sched_domain.cpu20.domain0.name = SMT
kernel.sched_domain.cpu20.domain1.busy_factor = 32
kernel.sched_domain.cpu20.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu20.domain1.flags = 4655
kernel.sched_domain.cpu20.domain1.imbalance_pct = 117
kernel.sched_domain.cpu20.domain1.max_interval = 96
kernel.sched_domain.cpu20.domain1.max_newidle_lb_cost = 6083
kernel.sched_domain.cpu20.domain1.min_interval = 48
kernel.sched_domain.cpu20.domain1.name = MC
kernel.sched_domain.cpu21.domain0.busy_factor = 32
kernel.sched_domain.cpu21.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu21.domain0.flags = 4783
kernel.sched_domain.cpu21.domain0.imbalance_pct = 110
kernel.sched_domain.cpu21.domain0.max_interval = 4
kernel.sched_domain.cpu21.domain0.max_newidle_lb_cost = 1676
kernel.sched_domain.cpu21.domain0.min_interval = 2
kernel.sched_domain.cpu21.domain0.name = SMT
kernel.sched_domain.cpu21.domain1.busy_factor = 32
kernel.sched_domain.cpu21.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu21.domain1.flags = 4655
kernel.sched_domain.cpu21.domain1.imbalance_pct = 117
kernel.sched_domain.cpu21.domain1.max_interval = 96
kernel.sched_domain.cpu21.domain1.max_newidle_lb_cost = 22147
kernel.sched_domain.cpu21.domain1.min_interval = 48
kernel.sched_domain.cpu21.domain1.name = MC
kernel.sched_domain.cpu22.domain0.busy_factor = 32
kernel.sched_domain.cpu22.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu22.domain0.flags = 4783
kernel.sched_domain.cpu22.domain0.imbalance_pct = 110
kernel.sched_domain.cpu22.domain0.max_interval = 4
kernel.sched_domain.cpu22.domain0.max_newidle_lb_cost = 1275
kernel.sched_domain.cpu22.domain0.min_interval = 2
kernel.sched_domain.cpu22.domain0.name = SMT
kernel.sched_domain.cpu22.domain1.busy_factor = 32
kernel.sched_domain.cpu22.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu22.domain1.flags = 4655
kernel.sched_domain.cpu22.domain1.imbalance_pct = 117
kernel.sched_domain.cpu22.domain1.max_interval = 96
kernel.sched_domain.cpu22.domain1.max_newidle_lb_cost = 12732
kernel.sched_domain.cpu22.domain1.min_interval = 48
kernel.sched_domain.cpu22.domain1.name = MC
kernel.sched_domain.cpu23.domain0.busy_factor = 32
kernel.sched_domain.cpu23.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu23.domain0.flags = 4783
kernel.sched_domain.cpu23.domain0.imbalance_pct = 110
kernel.sched_domain.cpu23.domain0.max_interval = 4
kernel.sched_domain.cpu23.domain0.max_newidle_lb_cost = 708
kernel.sched_domain.cpu23.domain0.min_interval = 2
kernel.sched_domain.cpu23.domain0.name = SMT
kernel.sched_domain.cpu23.domain1.busy_factor = 32
kernel.sched_domain.cpu23.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu23.domain1.flags = 4655
kernel.sched_domain.cpu23.domain1.imbalance_pct = 117
kernel.sched_domain.cpu23.domain1.max_interval = 96
kernel.sched_domain.cpu23.domain1.max_newidle_lb_cost = 4719
kernel.sched_domain.cpu23.domain1.min_interval = 48
kernel.sched_domain.cpu23.domain1.name = MC
kernel.sched_domain.cpu24.domain0.busy_factor = 32
kernel.sched_domain.cpu24.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu24.domain0.flags = 4783
kernel.sched_domain.cpu24.domain0.imbalance_pct = 110
kernel.sched_domain.cpu24.domain0.max_interval = 4
kernel.sched_domain.cpu24.domain0.max_newidle_lb_cost = 79
kernel.sched_domain.cpu24.domain0.min_interval = 2
kernel.sched_domain.cpu24.domain0.name = SMT
kernel.sched_domain.cpu24.domain1.busy_factor = 32
kernel.sched_domain.cpu24.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu24.domain1.flags = 4655
kernel.sched_domain.cpu24.domain1.imbalance_pct = 117
kernel.sched_domain.cpu24.domain1.max_interval = 96
kernel.sched_domain.cpu24.domain1.max_newidle_lb_cost = 1064
kernel.sched_domain.cpu24.domain1.min_interval = 48
kernel.sched_domain.cpu24.domain1.name = MC
kernel.sched_domain.cpu25.domain0.busy_factor = 32
kernel.sched_domain.cpu25.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu25.domain0.flags = 4783
kernel.sched_domain.cpu25.domain0.imbalance_pct = 110
kernel.sched_domain.cpu25.domain0.max_interval = 4
kernel.sched_domain.cpu25.domain0.max_newidle_lb_cost = 0
kernel.sched_domain.cpu25.domain0.min_interval = 2
kernel.sched_domain.cpu25.domain0.name = SMT
kernel.sched_domain.cpu25.domain1.busy_factor = 32
kernel.sched_domain.cpu25.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu25.domain1.flags = 4655
kernel.sched_domain.cpu25.domain1.imbalance_pct = 117
kernel.sched_domain.cpu25.domain1.max_interval = 96
kernel.sched_domain.cpu25.domain1.max_newidle_lb_cost = 0
kernel.sched_domain.cpu25.domain1.min_interval = 48
kernel.sched_domain.cpu25.domain1.name = MC
kernel.sched_domain.cpu26.domain0.busy_factor = 32
kernel.sched_domain.cpu26.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu26.domain0.flags = 4783
kernel.sched_domain.cpu26.domain0.imbalance_pct = 110
kernel.sched_domain.cpu26.domain0.max_interval = 4
kernel.sched_domain.cpu26.domain0.max_newidle_lb_cost = 1794
kernel.sched_domain.cpu26.domain0.min_interval = 2
kernel.sched_domain.cpu26.domain0.name = SMT
kernel.sched_domain.cpu26.domain1.busy_factor = 32
kernel.sched_domain.cpu26.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu26.domain1.flags = 4655
kernel.sched_domain.cpu26.domain1.imbalance_pct = 117
kernel.sched_domain.cpu26.domain1.max_interval = 96
kernel.sched_domain.cpu26.domain1.max_newidle_lb_cost = 7461
kernel.sched_domain.cpu26.domain1.min_interval = 48
kernel.sched_domain.cpu26.domain1.name = MC
kernel.sched_domain.cpu27.domain0.busy_factor = 32
kernel.sched_domain.cpu27.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu27.domain0.flags = 4783
kernel.sched_domain.cpu27.domain0.imbalance_pct = 110
kernel.sched_domain.cpu27.domain0.max_interval = 4
kernel.sched_domain.cpu27.domain0.max_newidle_lb_cost = 271
kernel.sched_domain.cpu27.domain0.min_interval = 2
kernel.sched_domain.cpu27.domain0.name = SMT
kernel.sched_domain.cpu27.domain1.busy_factor = 32
kernel.sched_domain.cpu27.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu27.domain1.flags = 4655
kernel.sched_domain.cpu27.domain1.imbalance_pct = 117
kernel.sched_domain.cpu27.domain1.max_interval = 96
kernel.sched_domain.cpu27.domain1.max_newidle_lb_cost = 2378
kernel.sched_domain.cpu27.domain1.min_interval = 48
kernel.sched_domain.cpu27.domain1.name = MC
kernel.sched_domain.cpu28.domain0.busy_factor = 32
kernel.sched_domain.cpu28.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu28.domain0.flags = 4783
kernel.sched_domain.cpu28.domain0.imbalance_pct = 110
kernel.sched_domain.cpu28.domain0.max_interval = 4
kernel.sched_domain.cpu28.domain0.max_newidle_lb_cost = 1204
kernel.sched_domain.cpu28.domain0.min_interval = 2
kernel.sched_domain.cpu28.domain0.name = SMT
kernel.sched_domain.cpu28.domain1.busy_factor = 32
kernel.sched_domain.cpu28.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu28.domain1.flags = 4655
kernel.sched_domain.cpu28.domain1.imbalance_pct = 117
kernel.sched_domain.cpu28.domain1.max_interval = 96
kernel.sched_domain.cpu28.domain1.max_newidle_lb_cost = 7083
kernel.sched_domain.cpu28.domain1.min_interval = 48
kernel.sched_domain.cpu28.domain1.name = MC
kernel.sched_domain.cpu29.domain0.busy_factor = 32
kernel.sched_domain.cpu29.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu29.domain0.flags = 4783
kernel.sched_domain.cpu29.domain0.imbalance_pct = 110
kernel.sched_domain.cpu29.domain0.max_interval = 4
kernel.sched_domain.cpu29.domain0.max_newidle_lb_cost = 1301
kernel.sched_domain.cpu29.domain0.min_interval = 2
kernel.sched_domain.cpu29.domain0.name = SMT
kernel.sched_domain.cpu29.domain1.busy_factor = 32
kernel.sched_domain.cpu29.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu29.domain1.flags = 4655
kernel.sched_domain.cpu29.domain1.imbalance_pct = 117
kernel.sched_domain.cpu29.domain1.max_interval = 96
kernel.sched_domain.cpu29.domain1.max_newidle_lb_cost = 8313
kernel.sched_domain.cpu29.domain1.min_interval = 48
kernel.sched_domain.cpu29.domain1.name = MC
kernel.sched_domain.cpu3.domain0.busy_factor = 32
kernel.sched_domain.cpu3.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu3.domain0.flags = 4783
kernel.sched_domain.cpu3.domain0.imbalance_pct = 110
kernel.sched_domain.cpu3.domain0.max_interval = 4
kernel.sched_domain.cpu3.domain0.max_newidle_lb_cost = 47
kernel.sched_domain.cpu3.domain0.min_interval = 2
kernel.sched_domain.cpu3.domain0.name = SMT
kernel.sched_domain.cpu3.domain1.busy_factor = 32
kernel.sched_domain.cpu3.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu3.domain1.flags = 4655
kernel.sched_domain.cpu3.domain1.imbalance_pct = 117
kernel.sched_domain.cpu3.domain1.max_interval = 96
kernel.sched_domain.cpu3.domain1.max_newidle_lb_cost = 367
kernel.sched_domain.cpu3.domain1.min_interval = 48
kernel.sched_domain.cpu3.domain1.name = MC
kernel.sched_domain.cpu30.domain0.busy_factor = 32
kernel.sched_domain.cpu30.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu30.domain0.flags = 4783
kernel.sched_domain.cpu30.domain0.imbalance_pct = 110
kernel.sched_domain.cpu30.domain0.max_interval = 4
kernel.sched_domain.cpu30.domain0.max_newidle_lb_cost = 600
kernel.sched_domain.cpu30.domain0.min_interval = 2
kernel.sched_domain.cpu30.domain0.name = SMT
kernel.sched_domain.cpu30.domain1.busy_factor = 32
kernel.sched_domain.cpu30.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu30.domain1.flags = 4655
kernel.sched_domain.cpu30.domain1.imbalance_pct = 117
kernel.sched_domain.cpu30.domain1.max_interval = 96
kernel.sched_domain.cpu30.domain1.max_newidle_lb_cost = 2303
kernel.sched_domain.cpu30.domain1.min_interval = 48
kernel.sched_domain.cpu30.domain1.name = MC
kernel.sched_domain.cpu31.domain0.busy_factor = 32
kernel.sched_domain.cpu31.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu31.domain0.flags = 4783
kernel.sched_domain.cpu31.domain0.imbalance_pct = 110
kernel.sched_domain.cpu31.domain0.max_interval = 4
kernel.sched_domain.cpu31.domain0.max_newidle_lb_cost = 791
kernel.sched_domain.cpu31.domain0.min_interval = 2
kernel.sched_domain.cpu31.domain0.name = SMT
kernel.sched_domain.cpu31.domain1.busy_factor = 32
kernel.sched_domain.cpu31.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu31.domain1.flags = 4655
kernel.sched_domain.cpu31.domain1.imbalance_pct = 117
kernel.sched_domain.cpu31.domain1.max_interval = 96
kernel.sched_domain.cpu31.domain1.max_newidle_lb_cost = 5335
kernel.sched_domain.cpu31.domain1.min_interval = 48
kernel.sched_domain.cpu31.domain1.name = MC
kernel.sched_domain.cpu32.domain0.busy_factor = 32
kernel.sched_domain.cpu32.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu32.domain0.flags = 4783
kernel.sched_domain.cpu32.domain0.imbalance_pct = 110
kernel.sched_domain.cpu32.domain0.max_interval = 4
kernel.sched_domain.cpu32.domain0.max_newidle_lb_cost = 1926
kernel.sched_domain.cpu32.domain0.min_interval = 2
kernel.sched_domain.cpu32.domain0.name = SMT
kernel.sched_domain.cpu32.domain1.busy_factor = 32
kernel.sched_domain.cpu32.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu32.domain1.flags = 4655
kernel.sched_domain.cpu32.domain1.imbalance_pct = 117
kernel.sched_domain.cpu32.domain1.max_interval = 96
kernel.sched_domain.cpu32.domain1.max_newidle_lb_cost = 11608
kernel.sched_domain.cpu32.domain1.min_interval = 48
kernel.sched_domain.cpu32.domain1.name = MC
kernel.sched_domain.cpu33.domain0.busy_factor = 32
kernel.sched_domain.cpu33.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu33.domain0.flags = 4783
kernel.sched_domain.cpu33.domain0.imbalance_pct = 110
kernel.sched_domain.cpu33.domain0.max_interval = 4
kernel.sched_domain.cpu33.domain0.max_newidle_lb_cost = 1505
kernel.sched_domain.cpu33.domain0.min_interval = 2
kernel.sched_domain.cpu33.domain0.name = SMT
kernel.sched_domain.cpu33.domain1.busy_factor = 32
kernel.sched_domain.cpu33.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu33.domain1.flags = 4655
kernel.sched_domain.cpu33.domain1.imbalance_pct = 117
kernel.sched_domain.cpu33.domain1.max_interval = 96
kernel.sched_domain.cpu33.domain1.max_newidle_lb_cost = 15512
kernel.sched_domain.cpu33.domain1.min_interval = 48
kernel.sched_domain.cpu33.domain1.name = MC
kernel.sched_domain.cpu34.domain0.busy_factor = 32
kernel.sched_domain.cpu34.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu34.domain0.flags = 4783
kernel.sched_domain.cpu34.domain0.imbalance_pct = 110
kernel.sched_domain.cpu34.domain0.max_interval = 4
kernel.sched_domain.cpu34.domain0.max_newidle_lb_cost = 1150
kernel.sched_domain.cpu34.domain0.min_interval = 2
kernel.sched_domain.cpu34.domain0.name = SMT
kernel.sched_domain.cpu34.domain1.busy_factor = 32
kernel.sched_domain.cpu34.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu34.domain1.flags = 4655
kernel.sched_domain.cpu34.domain1.imbalance_pct = 117
kernel.sched_domain.cpu34.domain1.max_interval = 96
kernel.sched_domain.cpu34.domain1.max_newidle_lb_cost = 7719
kernel.sched_domain.cpu34.domain1.min_interval = 48
kernel.sched_domain.cpu34.domain1.name = MC
kernel.sched_domain.cpu35.domain0.busy_factor = 32
kernel.sched_domain.cpu35.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu35.domain0.flags = 4783
kernel.sched_domain.cpu35.domain0.imbalance_pct = 110
kernel.sched_domain.cpu35.domain0.max_interval = 4
kernel.sched_domain.cpu35.domain0.max_newidle_lb_cost = 1930
kernel.sched_domain.cpu35.domain0.min_interval = 2
kernel.sched_domain.cpu35.domain0.name = SMT
kernel.sched_domain.cpu35.domain1.busy_factor = 32
kernel.sched_domain.cpu35.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu35.domain1.flags = 4655
kernel.sched_domain.cpu35.domain1.imbalance_pct = 117
kernel.sched_domain.cpu35.domain1.max_interval = 96
kernel.sched_domain.cpu35.domain1.max_newidle_lb_cost = 6693
kernel.sched_domain.cpu35.domain1.min_interval = 48
kernel.sched_domain.cpu35.domain1.name = MC
kernel.sched_domain.cpu36.domain0.busy_factor = 32
kernel.sched_domain.cpu36.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu36.domain0.flags = 4783
kernel.sched_domain.cpu36.domain0.imbalance_pct = 110
kernel.sched_domain.cpu36.domain0.max_interval = 4
kernel.sched_domain.cpu36.domain0.max_newidle_lb_cost = 545
kernel.sched_domain.cpu36.domain0.min_interval = 2
kernel.sched_domain.cpu36.domain0.name = SMT
kernel.sched_domain.cpu36.domain1.busy_factor = 32
kernel.sched_domain.cpu36.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu36.domain1.flags = 4655
kernel.sched_domain.cpu36.domain1.imbalance_pct = 117
kernel.sched_domain.cpu36.domain1.max_interval = 96
kernel.sched_domain.cpu36.domain1.max_newidle_lb_cost = 4101
kernel.sched_domain.cpu36.domain1.min_interval = 48
kernel.sched_domain.cpu36.domain1.name = MC
kernel.sched_domain.cpu37.domain0.busy_factor = 32
kernel.sched_domain.cpu37.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu37.domain0.flags = 4783
kernel.sched_domain.cpu37.domain0.imbalance_pct = 110
kernel.sched_domain.cpu37.domain0.max_interval = 4
kernel.sched_domain.cpu37.domain0.max_newidle_lb_cost = 1054
kernel.sched_domain.cpu37.domain0.min_interval = 2
kernel.sched_domain.cpu37.domain0.name = SMT
kernel.sched_domain.cpu37.domain1.busy_factor = 32
kernel.sched_domain.cpu37.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu37.domain1.flags = 4655
kernel.sched_domain.cpu37.domain1.imbalance_pct = 117
kernel.sched_domain.cpu37.domain1.max_interval = 96
kernel.sched_domain.cpu37.domain1.max_newidle_lb_cost = 5982
kernel.sched_domain.cpu37.domain1.min_interval = 48
kernel.sched_domain.cpu37.domain1.name = MC
kernel.sched_domain.cpu38.domain0.busy_factor = 32
kernel.sched_domain.cpu38.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu38.domain0.flags = 4783
kernel.sched_domain.cpu38.domain0.imbalance_pct = 110
kernel.sched_domain.cpu38.domain0.max_interval = 4
kernel.sched_domain.cpu38.domain0.max_newidle_lb_cost = 229
kernel.sched_domain.cpu38.domain0.min_interval = 2
kernel.sched_domain.cpu38.domain0.name = SMT
kernel.sched_domain.cpu38.domain1.busy_factor = 32
kernel.sched_domain.cpu38.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu38.domain1.flags = 4655
kernel.sched_domain.cpu38.domain1.imbalance_pct = 117
kernel.sched_domain.cpu38.domain1.max_interval = 96
kernel.sched_domain.cpu38.domain1.max_newidle_lb_cost = 2758
kernel.sched_domain.cpu38.domain1.min_interval = 48
kernel.sched_domain.cpu38.domain1.name = MC
kernel.sched_domain.cpu39.domain0.busy_factor = 32
kernel.sched_domain.cpu39.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu39.domain0.flags = 4783
kernel.sched_domain.cpu39.domain0.imbalance_pct = 110
kernel.sched_domain.cpu39.domain0.max_interval = 4
kernel.sched_domain.cpu39.domain0.max_newidle_lb_cost = 1698
kernel.sched_domain.cpu39.domain0.min_interval = 2
kernel.sched_domain.cpu39.domain0.name = SMT
kernel.sched_domain.cpu39.domain1.busy_factor = 32
kernel.sched_domain.cpu39.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu39.domain1.flags = 4655
kernel.sched_domain.cpu39.domain1.imbalance_pct = 117
kernel.sched_domain.cpu39.domain1.max_interval = 96
kernel.sched_domain.cpu39.domain1.max_newidle_lb_cost = 9714
kernel.sched_domain.cpu39.domain1.min_interval = 48
kernel.sched_domain.cpu39.domain1.name = MC
kernel.sched_domain.cpu4.domain0.busy_factor = 32
kernel.sched_domain.cpu4.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu4.domain0.flags = 4783
kernel.sched_domain.cpu4.domain0.imbalance_pct = 110
kernel.sched_domain.cpu4.domain0.max_interval = 4
kernel.sched_domain.cpu4.domain0.max_newidle_lb_cost = 143
kernel.sched_domain.cpu4.domain0.min_interval = 2
kernel.sched_domain.cpu4.domain0.name = SMT
kernel.sched_domain.cpu4.domain1.busy_factor = 32
kernel.sched_domain.cpu4.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu4.domain1.flags = 4655
kernel.sched_domain.cpu4.domain1.imbalance_pct = 117
kernel.sched_domain.cpu4.domain1.max_interval = 96
kernel.sched_domain.cpu4.domain1.max_newidle_lb_cost = 1224
kernel.sched_domain.cpu4.domain1.min_interval = 48
kernel.sched_domain.cpu4.domain1.name = MC
kernel.sched_domain.cpu40.domain0.busy_factor = 32
kernel.sched_domain.cpu40.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu40.domain0.flags = 4783
kernel.sched_domain.cpu40.domain0.imbalance_pct = 110
kernel.sched_domain.cpu40.domain0.max_interval = 4
kernel.sched_domain.cpu40.domain0.max_newidle_lb_cost = 1559
kernel.sched_domain.cpu40.domain0.min_interval = 2
kernel.sched_domain.cpu40.domain0.name = SMT
kernel.sched_domain.cpu40.domain1.busy_factor = 32
kernel.sched_domain.cpu40.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu40.domain1.flags = 4655
kernel.sched_domain.cpu40.domain1.imbalance_pct = 117
kernel.sched_domain.cpu40.domain1.max_interval = 96
kernel.sched_domain.cpu40.domain1.max_newidle_lb_cost = 5159
kernel.sched_domain.cpu40.domain1.min_interval = 48
kernel.sched_domain.cpu40.domain1.name = MC
kernel.sched_domain.cpu41.domain0.busy_factor = 32
kernel.sched_domain.cpu41.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu41.domain0.flags = 4783
kernel.sched_domain.cpu41.domain0.imbalance_pct = 110
kernel.sched_domain.cpu41.domain0.max_interval = 4
kernel.sched_domain.cpu41.domain0.max_newidle_lb_cost = 721
kernel.sched_domain.cpu41.domain0.min_interval = 2
kernel.sched_domain.cpu41.domain0.name = SMT
kernel.sched_domain.cpu41.domain1.busy_factor = 32
kernel.sched_domain.cpu41.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu41.domain1.flags = 4655
kernel.sched_domain.cpu41.domain1.imbalance_pct = 117
kernel.sched_domain.cpu41.domain1.max_interval = 96
kernel.sched_domain.cpu41.domain1.max_newidle_lb_cost = 5866
kernel.sched_domain.cpu41.domain1.min_interval = 48
kernel.sched_domain.cpu41.domain1.name = MC
kernel.sched_domain.cpu42.domain0.busy_factor = 32
kernel.sched_domain.cpu42.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu42.domain0.flags = 4783
kernel.sched_domain.cpu42.domain0.imbalance_pct = 110
kernel.sched_domain.cpu42.domain0.max_interval = 4
kernel.sched_domain.cpu42.domain0.max_newidle_lb_cost = 150
kernel.sched_domain.cpu42.domain0.min_interval = 2
kernel.sched_domain.cpu42.domain0.name = SMT
kernel.sched_domain.cpu42.domain1.busy_factor = 32
kernel.sched_domain.cpu42.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu42.domain1.flags = 4655
kernel.sched_domain.cpu42.domain1.imbalance_pct = 117
kernel.sched_domain.cpu42.domain1.max_interval = 96
kernel.sched_domain.cpu42.domain1.max_newidle_lb_cost = 672
kernel.sched_domain.cpu42.domain1.min_interval = 48
kernel.sched_domain.cpu42.domain1.name = MC
kernel.sched_domain.cpu43.domain0.busy_factor = 32
kernel.sched_domain.cpu43.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu43.domain0.flags = 4783
kernel.sched_domain.cpu43.domain0.imbalance_pct = 110
kernel.sched_domain.cpu43.domain0.max_interval = 4
kernel.sched_domain.cpu43.domain0.max_newidle_lb_cost = 1015
kernel.sched_domain.cpu43.domain0.min_interval = 2
kernel.sched_domain.cpu43.domain0.name = SMT
kernel.sched_domain.cpu43.domain1.busy_factor = 32
kernel.sched_domain.cpu43.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu43.domain1.flags = 4655
kernel.sched_domain.cpu43.domain1.imbalance_pct = 117
kernel.sched_domain.cpu43.domain1.max_interval = 96
kernel.sched_domain.cpu43.domain1.max_newidle_lb_cost = 10008
kernel.sched_domain.cpu43.domain1.min_interval = 48
kernel.sched_domain.cpu43.domain1.name = MC
kernel.sched_domain.cpu44.domain0.busy_factor = 32
kernel.sched_domain.cpu44.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu44.domain0.flags = 4783
kernel.sched_domain.cpu44.domain0.imbalance_pct = 110
kernel.sched_domain.cpu44.domain0.max_interval = 4
kernel.sched_domain.cpu44.domain0.max_newidle_lb_cost = 0
kernel.sched_domain.cpu44.domain0.min_interval = 2
kernel.sched_domain.cpu44.domain0.name = SMT
kernel.sched_domain.cpu44.domain1.busy_factor = 32
kernel.sched_domain.cpu44.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu44.domain1.flags = 4655
kernel.sched_domain.cpu44.domain1.imbalance_pct = 117
kernel.sched_domain.cpu44.domain1.max_interval = 96
kernel.sched_domain.cpu44.domain1.max_newidle_lb_cost = 440
kernel.sched_domain.cpu44.domain1.min_interval = 48
kernel.sched_domain.cpu44.domain1.name = MC
kernel.sched_domain.cpu45.domain0.busy_factor = 32
kernel.sched_domain.cpu45.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu45.domain0.flags = 4783
kernel.sched_domain.cpu45.domain0.imbalance_pct = 110
kernel.sched_domain.cpu45.domain0.max_interval = 4
kernel.sched_domain.cpu45.domain0.max_newidle_lb_cost = 31
kernel.sched_domain.cpu45.domain0.min_interval = 2
kernel.sched_domain.cpu45.domain0.name = SMT
kernel.sched_domain.cpu45.domain1.busy_factor = 32
kernel.sched_domain.cpu45.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu45.domain1.flags = 4655
kernel.sched_domain.cpu45.domain1.imbalance_pct = 117
kernel.sched_domain.cpu45.domain1.max_interval = 96
kernel.sched_domain.cpu45.domain1.max_newidle_lb_cost = 495
kernel.sched_domain.cpu45.domain1.min_interval = 48
kernel.sched_domain.cpu45.domain1.name = MC
kernel.sched_domain.cpu46.domain0.busy_factor = 32
kernel.sched_domain.cpu46.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu46.domain0.flags = 4783
kernel.sched_domain.cpu46.domain0.imbalance_pct = 110
kernel.sched_domain.cpu46.domain0.max_interval = 4
kernel.sched_domain.cpu46.domain0.max_newidle_lb_cost = 402
kernel.sched_domain.cpu46.domain0.min_interval = 2
kernel.sched_domain.cpu46.domain0.name = SMT
kernel.sched_domain.cpu46.domain1.busy_factor = 32
kernel.sched_domain.cpu46.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu46.domain1.flags = 4655
kernel.sched_domain.cpu46.domain1.imbalance_pct = 117
kernel.sched_domain.cpu46.domain1.max_interval = 96
kernel.sched_domain.cpu46.domain1.max_newidle_lb_cost = 1802
kernel.sched_domain.cpu46.domain1.min_interval = 48
kernel.sched_domain.cpu46.domain1.name = MC
kernel.sched_domain.cpu47.domain0.busy_factor = 32
kernel.sched_domain.cpu47.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu47.domain0.flags = 4783
kernel.sched_domain.cpu47.domain0.imbalance_pct = 110
kernel.sched_domain.cpu47.domain0.max_interval = 4
kernel.sched_domain.cpu47.domain0.max_newidle_lb_cost = 842
kernel.sched_domain.cpu47.domain0.min_interval = 2
kernel.sched_domain.cpu47.domain0.name = SMT
kernel.sched_domain.cpu47.domain1.busy_factor = 32
kernel.sched_domain.cpu47.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu47.domain1.flags = 4655
kernel.sched_domain.cpu47.domain1.imbalance_pct = 117
kernel.sched_domain.cpu47.domain1.max_interval = 96
kernel.sched_domain.cpu47.domain1.max_newidle_lb_cost = 4531
kernel.sched_domain.cpu47.domain1.min_interval = 48
kernel.sched_domain.cpu47.domain1.name = MC
kernel.sched_domain.cpu5.domain0.busy_factor = 32
kernel.sched_domain.cpu5.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu5.domain0.flags = 4783
kernel.sched_domain.cpu5.domain0.imbalance_pct = 110
kernel.sched_domain.cpu5.domain0.max_interval = 4
kernel.sched_domain.cpu5.domain0.max_newidle_lb_cost = 1780
kernel.sched_domain.cpu5.domain0.min_interval = 2
kernel.sched_domain.cpu5.domain0.name = SMT
kernel.sched_domain.cpu5.domain1.busy_factor = 32
kernel.sched_domain.cpu5.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu5.domain1.flags = 4655
kernel.sched_domain.cpu5.domain1.imbalance_pct = 117
kernel.sched_domain.cpu5.domain1.max_interval = 96
kernel.sched_domain.cpu5.domain1.max_newidle_lb_cost = 7787
kernel.sched_domain.cpu5.domain1.min_interval = 48
kernel.sched_domain.cpu5.domain1.name = MC
kernel.sched_domain.cpu6.domain0.busy_factor = 32
kernel.sched_domain.cpu6.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu6.domain0.flags = 4783
kernel.sched_domain.cpu6.domain0.imbalance_pct = 110
kernel.sched_domain.cpu6.domain0.max_interval = 4
kernel.sched_domain.cpu6.domain0.max_newidle_lb_cost = 524
kernel.sched_domain.cpu6.domain0.min_interval = 2
kernel.sched_domain.cpu6.domain0.name = SMT
kernel.sched_domain.cpu6.domain1.busy_factor = 32
kernel.sched_domain.cpu6.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu6.domain1.flags = 4655
kernel.sched_domain.cpu6.domain1.imbalance_pct = 117
kernel.sched_domain.cpu6.domain1.max_interval = 96
kernel.sched_domain.cpu6.domain1.max_newidle_lb_cost = 2517
kernel.sched_domain.cpu6.domain1.min_interval = 48
kernel.sched_domain.cpu6.domain1.name = MC
kernel.sched_domain.cpu7.domain0.busy_factor = 32
kernel.sched_domain.cpu7.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu7.domain0.flags = 4783
kernel.sched_domain.cpu7.domain0.imbalance_pct = 110
kernel.sched_domain.cpu7.domain0.max_interval = 4
kernel.sched_domain.cpu7.domain0.max_newidle_lb_cost = 593
kernel.sched_domain.cpu7.domain0.min_interval = 2
kernel.sched_domain.cpu7.domain0.name = SMT
kernel.sched_domain.cpu7.domain1.busy_factor = 32
kernel.sched_domain.cpu7.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu7.domain1.flags = 4655
kernel.sched_domain.cpu7.domain1.imbalance_pct = 117
kernel.sched_domain.cpu7.domain1.max_interval = 96
kernel.sched_domain.cpu7.domain1.max_newidle_lb_cost = 8795
kernel.sched_domain.cpu7.domain1.min_interval = 48
kernel.sched_domain.cpu7.domain1.name = MC
kernel.sched_domain.cpu8.domain0.busy_factor = 32
kernel.sched_domain.cpu8.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu8.domain0.flags = 4783
kernel.sched_domain.cpu8.domain0.imbalance_pct = 110
kernel.sched_domain.cpu8.domain0.max_interval = 4
kernel.sched_domain.cpu8.domain0.max_newidle_lb_cost = 1600
kernel.sched_domain.cpu8.domain0.min_interval = 2
kernel.sched_domain.cpu8.domain0.name = SMT
kernel.sched_domain.cpu8.domain1.busy_factor = 32
kernel.sched_domain.cpu8.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu8.domain1.flags = 4655
kernel.sched_domain.cpu8.domain1.imbalance_pct = 117
kernel.sched_domain.cpu8.domain1.max_interval = 96
kernel.sched_domain.cpu8.domain1.max_newidle_lb_cost = 6126
kernel.sched_domain.cpu8.domain1.min_interval = 48
kernel.sched_domain.cpu8.domain1.name = MC
kernel.sched_domain.cpu9.domain0.busy_factor = 32
kernel.sched_domain.cpu9.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu9.domain0.flags = 4783
kernel.sched_domain.cpu9.domain0.imbalance_pct = 110
kernel.sched_domain.cpu9.domain0.max_interval = 4
kernel.sched_domain.cpu9.domain0.max_newidle_lb_cost = 84
kernel.sched_domain.cpu9.domain0.min_interval = 2
kernel.sched_domain.cpu9.domain0.name = SMT
kernel.sched_domain.cpu9.domain1.busy_factor = 32
kernel.sched_domain.cpu9.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu9.domain1.flags = 4655
kernel.sched_domain.cpu9.domain1.imbalance_pct = 117
kernel.sched_domain.cpu9.domain1.max_interval = 96
kernel.sched_domain.cpu9.domain1.max_newidle_lb_cost = 914
kernel.sched_domain.cpu9.domain1.min_interval = 48
kernel.sched_domain.cpu9.domain1.name = MC
kernel.sched_latency_ns = 24000000
kernel.sched_migration_cost_ns = 500000
kernel.sched_min_granularity_ns = 3000000
kernel.sched_nr_migrate = 32
kernel.sched_rr_timeslice_ms = 100
kernel.sched_rt_period_us = 1000000
kernel.sched_rt_runtime_us = 950000
kernel.sched_schedstats = 0
kernel.sched_tunable_scaling = 1
kernel.sched_wakeup_granularity_ns = 4000000
kernel.seccomp.actions_avail = kill_process kill_thread trap errno user_notif trace log allow
kernel.seccomp.actions_logged = kill_process kill_thread trap errno user_notif trace log
kernel.sem = 32000	1024000000	500	32000
kernel.sem_next_id = -1
kernel.shm_next_id = -1
kernel.shm_rmid_forced = 0
kernel.shmall = 18446744073692774399
kernel.shmmax = 18446744073692774399
kernel.shmmni = 4096
kernel.soft_watchdog = 1
kernel.softlockup_all_cpu_backtrace = 0
kernel.softlockup_panic = 0
kernel.stack_tracer_enabled = 0
kernel.sysctl_writes_strict = 1
kernel.sysrq = 16
kernel.tainted = 0
kernel.threads-max = 755146
kernel.timer_migration = 1
kernel.traceoff_on_warning = 0
kernel.tracepoint_printk = 0
kernel.unknown_nmi_panic = 0
kernel.unprivileged_bpf_disabled = 0
kernel.usermodehelper.bset = 4294967295	63
kernel.usermodehelper.inheritable = 4294967295	63
kernel.version = #1 SMP Wed Nov 4 17:56:35 UTC 2020
kernel.watchdog = 1
kernel.watchdog_cpumask = 0-47
kernel.watchdog_thresh = 10
net.core.bpf_jit_enable = 1
net.core.bpf_jit_harden = 0
net.core.bpf_jit_kallsyms = 0
net.core.bpf_jit_limit = 398458880
net.core.busy_poll = 0
net.core.busy_read = 0
net.core.default_qdisc = pfifo_fast
net.core.dev_weight = 64
net.core.dev_weight_rx_bias = 1
net.core.dev_weight_tx_bias = 1
net.core.devconf_inherit_init_net = 0
net.core.fb_tunnels_only_for_init_net = 0
net.core.flow_limit_cpu_bitmap = 0000,00000000
net.core.flow_limit_table_len = 4096
net.core.gro_normal_batch = 8
net.core.high_order_alloc_disable = 0
net.core.max_skb_frags = 17
net.core.message_burst = 10
net.core.message_cost = 5
net.core.netdev_budget = 300
net.core.netdev_budget_usecs = 8000
net.core.netdev_max_backlog = 1000
net.core.netdev_rss_key = 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
net.core.netdev_tstamp_prequeue = 1
net.core.optmem_max = 20480
net.core.rmem_default = 212992
net.core.rmem_max = 212992
net.core.rps_sock_flow_entries = 0
net.core.somaxconn = 4096
net.core.tstamp_allow_data = 1
net.core.warnings = 0
net.core.wmem_default = 212992
net.core.wmem_max = 212992
net.core.xfrm_acq_expires = 30
net.core.xfrm_aevent_etime = 10
net.core.xfrm_aevent_rseqth = 2
net.core.xfrm_larval_drop = 1
net.ipv4.cipso_cache_bucket_size = 10
net.ipv4.cipso_cache_enable = 1
net.ipv4.cipso_rbm_optfmt = 0
net.ipv4.cipso_rbm_strictvalid = 1
net.ipv4.conf.all.accept_local = 0
net.ipv4.conf.all.accept_redirects = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_notify = 0
net.ipv4.conf.all.bc_forwarding = 0
net.ipv4.conf.all.bootp_relay = 0
net.ipv4.conf.all.disable_policy = 0
net.ipv4.conf.all.disable_xfrm = 0
net.ipv4.conf.all.drop_gratuitous_arp = 0
net.ipv4.conf.all.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.all.force_igmp_version = 0
net.ipv4.conf.all.forwarding = 0
net.ipv4.conf.all.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.all.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.all.ignore_routes_with_linkdown = 0
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.all.medium_id = 0
net.ipv4.conf.all.promote_secondaries = 1
net.ipv4.conf.all.proxy_arp = 0
net.ipv4.conf.all.proxy_arp_pvlan = 0
net.ipv4.conf.all.route_localnet = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.secure_redirects = 1
net.ipv4.conf.all.send_redirects = 1
net.ipv4.conf.all.shared_media = 1
net.ipv4.conf.all.src_valid_mark = 0
net.ipv4.conf.all.tag = 0
net.ipv4.conf.default.accept_local = 0
net.ipv4.conf.default.accept_redirects = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_notify = 0
net.ipv4.conf.default.bc_forwarding = 0
net.ipv4.conf.default.bootp_relay = 0
net.ipv4.conf.default.disable_policy = 0
net.ipv4.conf.default.disable_xfrm = 0
net.ipv4.conf.default.drop_gratuitous_arp = 0
net.ipv4.conf.default.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.default.force_igmp_version = 0
net.ipv4.conf.default.forwarding = 0
net.ipv4.conf.default.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.default.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.default.ignore_routes_with_linkdown = 0
net.ipv4.conf.default.log_martians = 0
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.default.medium_id = 0
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.default.proxy_arp_pvlan = 0
net.ipv4.conf.default.route_localnet = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.secure_redirects = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.default.shared_media = 1
net.ipv4.conf.default.src_valid_mark = 0
net.ipv4.conf.default.tag = 0
net.ipv4.conf.eth0.accept_local = 0
net.ipv4.conf.eth0.accept_redirects = 1
net.ipv4.conf.eth0.accept_source_route = 0
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_notify = 0
net.ipv4.conf.eth0.bc_forwarding = 0
net.ipv4.conf.eth0.bootp_relay = 0
net.ipv4.conf.eth0.disable_policy = 0
net.ipv4.conf.eth0.disable_xfrm = 0
net.ipv4.conf.eth0.drop_gratuitous_arp = 0
net.ipv4.conf.eth0.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.eth0.force_igmp_version = 0
net.ipv4.conf.eth0.forwarding = 0
net.ipv4.conf.eth0.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.eth0.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.eth0.ignore_routes_with_linkdown = 0
net.ipv4.conf.eth0.log_martians = 0
net.ipv4.conf.eth0.mc_forwarding = 0
net.ipv4.conf.eth0.medium_id = 0
net.ipv4.conf.eth0.promote_secondaries = 1
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.eth0.proxy_arp_pvlan = 0
net.ipv4.conf.eth0.route_localnet = 0
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.eth0.secure_redirects = 1
net.ipv4.conf.eth0.send_redirects = 1
net.ipv4.conf.eth0.shared_media = 1
net.ipv4.conf.eth0.src_valid_mark = 0
net.ipv4.conf.eth0.tag = 0
net.ipv4.conf.lo.accept_local = 0
net.ipv4.conf.lo.accept_redirects = 1
net.ipv4.conf.lo.accept_source_route = 1
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_notify = 0
net.ipv4.conf.lo.bc_forwarding = 0
net.ipv4.conf.lo.bootp_relay = 0
net.ipv4.conf.lo.disable_policy = 1
net.ipv4.conf.lo.disable_xfrm = 1
net.ipv4.conf.lo.drop_gratuitous_arp = 0
net.ipv4.conf.lo.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.lo.force_igmp_version = 0
net.ipv4.conf.lo.forwarding = 0
net.ipv4.conf.lo.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.lo.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.lo.ignore_routes_with_linkdown = 0
net.ipv4.conf.lo.log_martians = 0
net.ipv4.conf.lo.mc_forwarding = 0
net.ipv4.conf.lo.medium_id = 0
net.ipv4.conf.lo.promote_secondaries = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.lo.proxy_arp_pvlan = 0
net.ipv4.conf.lo.route_localnet = 0
net.ipv4.conf.lo.rp_filter = 0
net.ipv4.conf.lo.secure_redirects = 1
net.ipv4.conf.lo.send_redirects = 1
net.ipv4.conf.lo.shared_media = 1
net.ipv4.conf.lo.src_valid_mark = 0
net.ipv4.conf.lo.tag = 0
net.ipv4.fib_multipath_hash_policy = 0
net.ipv4.fib_multipath_use_neigh = 0
net.ipv4.fib_sync_mem = 524288
net.ipv4.fwmark_reflect = 0
net.ipv4.icmp_echo_ignore_all = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_errors_use_inbound_ifaddr = 0
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.icmp_msgs_burst = 50
net.ipv4.icmp_msgs_per_sec = 1000
net.ipv4.icmp_ratelimit = 1000
net.ipv4.icmp_ratemask = 6168
net.ipv4.igmp_link_local_mcast_reports = 1
net.ipv4.igmp_max_memberships = 20
net.ipv4.igmp_max_msf = 10
net.ipv4.igmp_qrv = 2
net.ipv4.inet_peer_maxttl = 600
net.ipv4.inet_peer_minttl = 120
net.ipv4.inet_peer_threshold = 65664
net.ipv4.ip_default_ttl = 255
net.ipv4.ip_dynaddr = 0
net.ipv4.ip_early_demux = 1
net.ipv4.ip_forward = 0
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0
net.ipv4.ip_local_port_range = 32768	60999
net.ipv4.ip_local_reserved_ports = 
net.ipv4.ip_no_pmtu_disc = 0
net.ipv4.ip_nonlocal_bind = 0
net.ipv4.ip_unprivileged_port_start = 1024
net.ipv4.ipfrag_high_thresh = 4194304
net.ipv4.ipfrag_low_thresh = 3145728
net.ipv4.ipfrag_max_dist = 64
net.ipv4.ipfrag_secret_interval = 0
net.ipv4.ipfrag_time = 30
net.ipv4.neigh.default.anycast_delay = 100
net.ipv4.neigh.default.app_solicit = 0
net.ipv4.neigh.default.base_reachable_time_ms = 30000
net.ipv4.neigh.default.delay_first_probe_time = 5
net.ipv4.neigh.default.gc_interval = 30
net.ipv4.neigh.default.gc_stale_time = 60
net.ipv4.neigh.default.gc_thresh1 = 0
net.ipv4.neigh.default.gc_thresh2 = 15360
net.ipv4.neigh.default.gc_thresh3 = 16384
net.ipv4.neigh.default.locktime = 100
net.ipv4.neigh.default.mcast_resolicit = 0
net.ipv4.neigh.default.mcast_solicit = 3
net.ipv4.neigh.default.proxy_delay = 80
net.ipv4.neigh.default.proxy_qlen = 64
net.ipv4.neigh.default.retrans_time_ms = 1000
net.ipv4.neigh.default.ucast_solicit = 3
net.ipv4.neigh.default.unres_qlen = 101
net.ipv4.neigh.default.unres_qlen_bytes = 212992
net.ipv4.neigh.eth0.anycast_delay = 100
net.ipv4.neigh.eth0.app_solicit = 0
net.ipv4.neigh.eth0.base_reachable_time_ms = 30000
net.ipv4.neigh.eth0.delay_first_probe_time = 5
net.ipv4.neigh.eth0.gc_stale_time = 60
net.ipv4.neigh.eth0.locktime = 100
net.ipv4.neigh.eth0.mcast_resolicit = 0
net.ipv4.neigh.eth0.mcast_solicit = 3
net.ipv4.neigh.eth0.proxy_delay = 80
net.ipv4.neigh.eth0.proxy_qlen = 64
net.ipv4.neigh.eth0.retrans_time_ms = 1000
net.ipv4.neigh.eth0.ucast_solicit = 3
net.ipv4.neigh.eth0.unres_qlen = 101
net.ipv4.neigh.eth0.unres_qlen_bytes = 212992
net.ipv4.neigh.lo.anycast_delay = 100
net.ipv4.neigh.lo.app_solicit = 0
net.ipv4.neigh.lo.base_reachable_time_ms = 30000
net.ipv4.neigh.lo.delay_first_probe_time = 5
net.ipv4.neigh.lo.gc_stale_time = 60
net.ipv4.neigh.lo.locktime = 100
net.ipv4.neigh.lo.mcast_resolicit = 0
net.ipv4.neigh.lo.mcast_solicit = 3
net.ipv4.neigh.lo.proxy_delay = 80
net.ipv4.neigh.lo.proxy_qlen = 64
net.ipv4.neigh.lo.retrans_time_ms = 1000
net.ipv4.neigh.lo.ucast_solicit = 3
net.ipv4.neigh.lo.unres_qlen = 101
net.ipv4.neigh.lo.unres_qlen_bytes = 212992
net.ipv4.ping_group_range = 1	0
net.ipv4.raw_l3mdev_accept = 1
net.ipv4.route.error_burst = 1250
net.ipv4.route.error_cost = 250
net.ipv4.route.gc_elasticity = 8
net.ipv4.route.gc_interval = 60
net.ipv4.route.gc_min_interval = 0
net.ipv4.route.gc_min_interval_ms = 500
net.ipv4.route.gc_thresh = -1
net.ipv4.route.gc_timeout = 300
net.ipv4.route.max_size = 2147483647
net.ipv4.route.min_adv_mss = 256
net.ipv4.route.min_pmtu = 552
net.ipv4.route.mtu_expires = 600
net.ipv4.route.redirect_load = 5
net.ipv4.route.redirect_number = 9
net.ipv4.route.redirect_silence = 5120
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_allowed_congestion_control = reno cubic
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_autocorking = 1
net.ipv4.tcp_available_congestion_control = reno cubic
net.ipv4.tcp_available_ulp = 
net.ipv4.tcp_base_mss = 1024
net.ipv4.tcp_challenge_ack_limit = 1000
net.ipv4.tcp_comp_sack_delay_ns = 1000000
net.ipv4.tcp_comp_sack_nr = 44
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_early_demux = 1
net.ipv4.tcp_early_retrans = 3
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_ecn_fallback = 1
net.ipv4.tcp_fack = 0
net.ipv4.tcp_fastopen = 1
net.ipv4.tcp_fastopen_blackhole_timeout_sec = 3600
net.ipv4.tcp_fastopen_key = 00000000-00000000-00000000-00000000
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_frto = 2
net.ipv4.tcp_fwmark_accept = 0
net.ipv4.tcp_invalid_ratelimit = 500
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_l3mdev_accept = 0
net.ipv4.tcp_limit_output_bytes = 1048576
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_reordering = 300
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_max_tw_buckets = 262144
net.ipv4.tcp_mem = 1129662	1506217	2259324
net.ipv4.tcp_min_rtt_wlen = 300
net.ipv4.tcp_min_snd_mss = 48
net.ipv4.tcp_min_tso_segs = 2
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_mtu_probe_floor = 48
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_notsent_lowat = 4294967295
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_pacing_ca_ratio = 120
net.ipv4.tcp_pacing_ss_ratio = 200
net.ipv4.tcp_probe_interval = 600
net.ipv4.tcp_probe_threshold = 8
net.ipv4.tcp_recovery = 1
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_rmem = 4096	131072	6291456
net.ipv4.tcp_rx_skb_cache = 0
net.ipv4.tcp_sack = 1
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_syn_retries = 6
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_thin_linear_timeouts = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_tw_reuse = 2
net.ipv4.tcp_tx_skb_cache = 0
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_wmem = 4096	20480	4194304
net.ipv4.tcp_workaround_signed_windows = 0
net.ipv4.udp_early_demux = 1
net.ipv4.udp_l3mdev_accept = 0
net.ipv4.udp_mem = 2259324	3012435	4518648
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
net.ipv4.xfrm4_gc_thresh = 32768
net.ipv6.anycast_src_echo_reply = 0
net.ipv6.auto_flowlabels = 1
net.ipv6.bindv6only = 0
net.ipv6.calipso_cache_bucket_size = 10
net.ipv6.calipso_cache_enable = 1
net.ipv6.conf.all.accept_dad = 0
net.ipv6.conf.all.accept_ra = 1
net.ipv6.conf.all.accept_ra_defrtr = 1
net.ipv6.conf.all.accept_ra_from_local = 0
net.ipv6.conf.all.accept_ra_min_hop_limit = 1
net.ipv6.conf.all.accept_ra_mtu = 1
net.ipv6.conf.all.accept_ra_pinfo = 1
net.ipv6.conf.all.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.all.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.all.accept_ra_rtr_pref = 1
net.ipv6.conf.all.accept_redirects = 1
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.all.addr_gen_mode = 0
net.ipv6.conf.all.autoconf = 1
net.ipv6.conf.all.dad_transmits = 1
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.all.disable_policy = 0
net.ipv6.conf.all.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.all.drop_unsolicited_na = 0
net.ipv6.conf.all.enhanced_dad = 1
net.ipv6.conf.all.force_mld_version = 0
net.ipv6.conf.all.force_tllao = 0
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.all.hop_limit = 64
net.ipv6.conf.all.ignore_routes_with_linkdown = 0
net.ipv6.conf.all.keep_addr_on_down = 0
net.ipv6.conf.all.max_addresses = 16
net.ipv6.conf.all.max_desync_factor = 600
net.ipv6.conf.all.mc_forwarding = 0
net.ipv6.conf.all.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.all.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.all.mtu = 1280
net.ipv6.conf.all.ndisc_notify = 0
net.ipv6.conf.all.ndisc_tclass = 0
net.ipv6.conf.all.optimistic_dad = 0
net.ipv6.conf.all.proxy_ndp = 0
net.ipv6.conf.all.regen_max_retry = 3
net.ipv6.conf.all.router_probe_interval = 60
net.ipv6.conf.all.router_solicitation_delay = 1
net.ipv6.conf.all.router_solicitation_interval = 4
net.ipv6.conf.all.router_solicitation_max_interval = 3600
net.ipv6.conf.all.router_solicitations = -1
net.ipv6.conf.all.seg6_enabled = 0
net.ipv6.conf.all.seg6_require_hmac = 0
net.ipv6.conf.all.suppress_frag_ndisc = 1
net.ipv6.conf.all.temp_prefered_lft = 86400
net.ipv6.conf.all.temp_valid_lft = 604800
net.ipv6.conf.all.use_oif_addrs_only = 0
net.ipv6.conf.all.use_optimistic = 0
net.ipv6.conf.all.use_tempaddr = 0
net.ipv6.conf.default.accept_dad = 1
net.ipv6.conf.default.accept_ra = 1
net.ipv6.conf.default.accept_ra_defrtr = 1
net.ipv6.conf.default.accept_ra_from_local = 0
net.ipv6.conf.default.accept_ra_min_hop_limit = 1
net.ipv6.conf.default.accept_ra_mtu = 1
net.ipv6.conf.default.accept_ra_pinfo = 1
net.ipv6.conf.default.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.default.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.default.accept_ra_rtr_pref = 1
net.ipv6.conf.default.accept_redirects = 1
net.ipv6.conf.default.accept_source_route = 0
net.ipv6.conf.default.addr_gen_mode = 0
net.ipv6.conf.default.autoconf = 1
net.ipv6.conf.default.dad_transmits = 1
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.default.disable_policy = 0
net.ipv6.conf.default.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.default.drop_unsolicited_na = 0
net.ipv6.conf.default.enhanced_dad = 1
net.ipv6.conf.default.force_mld_version = 0
net.ipv6.conf.default.force_tllao = 0
net.ipv6.conf.default.forwarding = 0
net.ipv6.conf.default.hop_limit = 64
net.ipv6.conf.default.ignore_routes_with_linkdown = 0
net.ipv6.conf.default.keep_addr_on_down = 0
net.ipv6.conf.default.max_addresses = 16
net.ipv6.conf.default.max_desync_factor = 600
net.ipv6.conf.default.mc_forwarding = 0
net.ipv6.conf.default.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.default.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.default.mtu = 1280
net.ipv6.conf.default.ndisc_notify = 0
net.ipv6.conf.default.ndisc_tclass = 0
net.ipv6.conf.default.optimistic_dad = 0
net.ipv6.conf.default.proxy_ndp = 0
net.ipv6.conf.default.regen_max_retry = 3
net.ipv6.conf.default.router_probe_interval = 60
net.ipv6.conf.default.router_solicitation_delay = 1
net.ipv6.conf.default.router_solicitation_interval = 4
net.ipv6.conf.default.router_solicitation_max_interval = 3600
net.ipv6.conf.default.router_solicitations = -1
net.ipv6.conf.default.seg6_enabled = 0
net.ipv6.conf.default.seg6_require_hmac = 0
net.ipv6.conf.default.suppress_frag_ndisc = 1
net.ipv6.conf.default.temp_prefered_lft = 86400
net.ipv6.conf.default.temp_valid_lft = 604800
net.ipv6.conf.default.use_oif_addrs_only = 0
net.ipv6.conf.default.use_optimistic = 0
net.ipv6.conf.default.use_tempaddr = 0
net.ipv6.conf.eth0.accept_dad = 1
net.ipv6.conf.eth0.accept_ra = 1
net.ipv6.conf.eth0.accept_ra_defrtr = 1
net.ipv6.conf.eth0.accept_ra_from_local = 0
net.ipv6.conf.eth0.accept_ra_min_hop_limit = 1
net.ipv6.conf.eth0.accept_ra_mtu = 1
net.ipv6.conf.eth0.accept_ra_pinfo = 1
net.ipv6.conf.eth0.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.eth0.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.eth0.accept_ra_rtr_pref = 1
net.ipv6.conf.eth0.accept_redirects = 1
net.ipv6.conf.eth0.accept_source_route = 0
net.ipv6.conf.eth0.addr_gen_mode = 0
net.ipv6.conf.eth0.autoconf = 1
net.ipv6.conf.eth0.dad_transmits = 1
net.ipv6.conf.eth0.disable_ipv6 = 0
net.ipv6.conf.eth0.disable_policy = 0
net.ipv6.conf.eth0.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.eth0.drop_unsolicited_na = 0
net.ipv6.conf.eth0.enhanced_dad = 1
net.ipv6.conf.eth0.force_mld_version = 0
net.ipv6.conf.eth0.force_tllao = 0
net.ipv6.conf.eth0.forwarding = 0
net.ipv6.conf.eth0.hop_limit = 64
net.ipv6.conf.eth0.ignore_routes_with_linkdown = 0
net.ipv6.conf.eth0.keep_addr_on_down = 0
net.ipv6.conf.eth0.max_addresses = 16
net.ipv6.conf.eth0.max_desync_factor = 600
net.ipv6.conf.eth0.mc_forwarding = 0
net.ipv6.conf.eth0.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.eth0.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.eth0.mtu = 9001
net.ipv6.conf.eth0.ndisc_notify = 0
net.ipv6.conf.eth0.ndisc_tclass = 0
net.ipv6.conf.eth0.optimistic_dad = 0
net.ipv6.conf.eth0.proxy_ndp = 0
net.ipv6.conf.eth0.regen_max_retry = 3
net.ipv6.conf.eth0.router_probe_interval = 60
net.ipv6.conf.eth0.router_solicitation_delay = 1
net.ipv6.conf.eth0.router_solicitation_interval = 4
net.ipv6.conf.eth0.router_solicitation_max_interval = 3600
net.ipv6.conf.eth0.router_solicitations = -1
net.ipv6.conf.eth0.seg6_enabled = 0
net.ipv6.conf.eth0.seg6_require_hmac = 0
net.ipv6.conf.eth0.suppress_frag_ndisc = 1
net.ipv6.conf.eth0.temp_prefered_lft = 86400
net.ipv6.conf.eth0.temp_valid_lft = 604800
net.ipv6.conf.eth0.use_oif_addrs_only = 0
net.ipv6.conf.eth0.use_optimistic = 0
net.ipv6.conf.eth0.use_tempaddr = 0
net.ipv6.conf.lo.accept_dad = -1
net.ipv6.conf.lo.accept_ra = 1
net.ipv6.conf.lo.accept_ra_defrtr = 1
net.ipv6.conf.lo.accept_ra_from_local = 0
net.ipv6.conf.lo.accept_ra_min_hop_limit = 1
net.ipv6.conf.lo.accept_ra_mtu = 1
net.ipv6.conf.lo.accept_ra_pinfo = 1
net.ipv6.conf.lo.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.lo.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.lo.accept_ra_rtr_pref = 1
net.ipv6.conf.lo.accept_redirects = 1
net.ipv6.conf.lo.accept_source_route = 0
net.ipv6.conf.lo.addr_gen_mode = 0
net.ipv6.conf.lo.autoconf = 1
net.ipv6.conf.lo.dad_transmits = 1
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.lo.disable_policy = 0
net.ipv6.conf.lo.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.lo.drop_unsolicited_na = 0
net.ipv6.conf.lo.enhanced_dad = 1
net.ipv6.conf.lo.force_mld_version = 0
net.ipv6.conf.lo.force_tllao = 0
net.ipv6.conf.lo.forwarding = 0
net.ipv6.conf.lo.hop_limit = 64
net.ipv6.conf.lo.ignore_routes_with_linkdown = 0
net.ipv6.conf.lo.keep_addr_on_down = 0
net.ipv6.conf.lo.max_addresses = 16
net.ipv6.conf.lo.max_desync_factor = 600
net.ipv6.conf.lo.mc_forwarding = 0
net.ipv6.conf.lo.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.lo.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.lo.mtu = 65536
net.ipv6.conf.lo.ndisc_notify = 0
net.ipv6.conf.lo.ndisc_tclass = 0
net.ipv6.conf.lo.optimistic_dad = 0
net.ipv6.conf.lo.proxy_ndp = 0
net.ipv6.conf.lo.regen_max_retry = 3
net.ipv6.conf.lo.router_probe_interval = 60
net.ipv6.conf.lo.router_solicitation_delay = 1
net.ipv6.conf.lo.router_solicitation_interval = 4
net.ipv6.conf.lo.router_solicitation_max_interval = 3600
net.ipv6.conf.lo.router_solicitations = -1
net.ipv6.conf.lo.seg6_enabled = 0
net.ipv6.conf.lo.seg6_require_hmac = 0
net.ipv6.conf.lo.suppress_frag_ndisc = 1
net.ipv6.conf.lo.temp_prefered_lft = 86400
net.ipv6.conf.lo.temp_valid_lft = 604800
net.ipv6.conf.lo.use_oif_addrs_only = 0
net.ipv6.conf.lo.use_optimistic = 0
net.ipv6.conf.lo.use_tempaddr = -1
net.ipv6.fib_multipath_hash_policy = 0
net.ipv6.flowlabel_consistency = 1
net.ipv6.flowlabel_reflect = 0
net.ipv6.flowlabel_state_ranges = 0
net.ipv6.fwmark_reflect = 0
net.ipv6.icmp.echo_ignore_all = 0
net.ipv6.icmp.echo_ignore_anycast = 0
net.ipv6.icmp.echo_ignore_multicast = 0
net.ipv6.icmp.ratelimit = 1000
net.ipv6.icmp.ratemask = 0-1,3-127
net.ipv6.idgen_delay = 1
net.ipv6.idgen_retries = 3
net.ipv6.ip6frag_high_thresh = 4194304
net.ipv6.ip6frag_low_thresh = 3145728
net.ipv6.ip6frag_secret_interval = 0
net.ipv6.ip6frag_time = 60
net.ipv6.ip_nonlocal_bind = 0
net.ipv6.max_dst_opts_length = 2147483647
net.ipv6.max_dst_opts_number = 8
net.ipv6.max_hbh_length = 2147483647
net.ipv6.max_hbh_opts_number = 8
net.ipv6.mld_max_msf = 64
net.ipv6.mld_qrv = 2
net.ipv6.neigh.default.anycast_delay = 100
net.ipv6.neigh.default.app_solicit = 0
net.ipv6.neigh.default.base_reachable_time_ms = 30000
net.ipv6.neigh.default.delay_first_probe_time = 5
net.ipv6.neigh.default.gc_interval = 30
net.ipv6.neigh.default.gc_stale_time = 60
net.ipv6.neigh.default.gc_thresh1 = 0
net.ipv6.neigh.default.gc_thresh2 = 15360
net.ipv6.neigh.default.gc_thresh3 = 16384
net.ipv6.neigh.default.locktime = 0
net.ipv6.neigh.default.mcast_resolicit = 0
net.ipv6.neigh.default.mcast_solicit = 3
net.ipv6.neigh.default.proxy_delay = 80
net.ipv6.neigh.default.proxy_qlen = 64
net.ipv6.neigh.default.retrans_time_ms = 1000
net.ipv6.neigh.default.ucast_solicit = 3
net.ipv6.neigh.default.unres_qlen = 101
net.ipv6.neigh.default.unres_qlen_bytes = 212992
net.ipv6.neigh.eth0.anycast_delay = 100
net.ipv6.neigh.eth0.app_solicit = 0
net.ipv6.neigh.eth0.base_reachable_time_ms = 30000
net.ipv6.neigh.eth0.delay_first_probe_time = 5
net.ipv6.neigh.eth0.gc_stale_time = 60
net.ipv6.neigh.eth0.locktime = 0
net.ipv6.neigh.eth0.mcast_resolicit = 0
net.ipv6.neigh.eth0.mcast_solicit = 3
net.ipv6.neigh.eth0.proxy_delay = 80
net.ipv6.neigh.eth0.proxy_qlen = 64
net.ipv6.neigh.eth0.retrans_time_ms = 1000
net.ipv6.neigh.eth0.ucast_solicit = 3
net.ipv6.neigh.eth0.unres_qlen = 101
net.ipv6.neigh.eth0.unres_qlen_bytes = 212992
net.ipv6.neigh.lo.anycast_delay = 100
net.ipv6.neigh.lo.app_solicit = 0
net.ipv6.neigh.lo.base_reachable_time_ms = 30000
net.ipv6.neigh.lo.delay_first_probe_time = 5
net.ipv6.neigh.lo.gc_stale_time = 60
net.ipv6.neigh.lo.locktime = 0
net.ipv6.neigh.lo.mcast_resolicit = 0
net.ipv6.neigh.lo.mcast_solicit = 3
net.ipv6.neigh.lo.proxy_delay = 80
net.ipv6.neigh.lo.proxy_qlen = 64
net.ipv6.neigh.lo.retrans_time_ms = 1000
net.ipv6.neigh.lo.ucast_solicit = 3
net.ipv6.neigh.lo.unres_qlen = 101
net.ipv6.neigh.lo.unres_qlen_bytes = 212992
net.ipv6.route.gc_elasticity = 9
net.ipv6.route.gc_interval = 30
net.ipv6.route.gc_min_interval = 0
net.ipv6.route.gc_min_interval_ms = 500
net.ipv6.route.gc_thresh = 1024
net.ipv6.route.gc_timeout = 60
net.ipv6.route.max_size = 4096
net.ipv6.route.min_adv_mss = 1220
net.ipv6.route.mtu_expires = 600
net.ipv6.route.skip_notify_on_dev_down = 0
net.ipv6.seg6_flowlabel = 0
net.ipv6.xfrm6_gc_thresh = 32768
net.netfilter.nf_log.0 = NONE
net.netfilter.nf_log.1 = NONE
net.netfilter.nf_log.10 = NONE
net.netfilter.nf_log.11 = NONE
net.netfilter.nf_log.12 = NONE
net.netfilter.nf_log.2 = NONE
net.netfilter.nf_log.3 = NONE
net.netfilter.nf_log.4 = NONE
net.netfilter.nf_log.5 = NONE
net.netfilter.nf_log.6 = NONE
net.netfilter.nf_log.7 = NONE
net.netfilter.nf_log.8 = NONE
net.netfilter.nf_log.9 = NONE
net.netfilter.nf_log_all_netns = 0
net.unix.max_dgram_qlen = 512
sunrpc.max_resvport = 1023
sunrpc.min_resvport = 665
sunrpc.nfs_debug = 0x0000
sunrpc.nfsd_debug = 0x0000
sunrpc.nlm_debug = 0x0000
sunrpc.rpc_debug = 0x0000
sunrpc.tcp_fin_timeout = 15
sunrpc.tcp_max_slot_table_entries = 65536
sunrpc.tcp_slot_table_entries = 2
sunrpc.transports = tcp 1048576
sunrpc.transports = udp 32768
sunrpc.udp_slot_table_entries = 16
user.max_cgroup_namespaces = 377573
user.max_inotify_instances = 128
user.max_inotify_watches = 8192
user.max_ipc_namespaces = 377573
user.max_mnt_namespaces = 377573
user.max_net_namespaces = 377573
user.max_pid_namespaces = 377573
user.max_user_namespaces = 377573
user.max_uts_namespaces = 377573
vm.admin_reserve_kbytes = 8192
vm.block_dump = 0
vm.compact_unevictable_allowed = 1
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
vm.extfrag_threshold = 500
vm.hugetlb_shm_group = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 256	256	32	0
vm.max_map_count = 65530
vm.memory_failure_early_kill = 0
vm.memory_failure_recovery = 1
vm.min_free_kbytes = 67584
vm.min_slab_ratio = 5
vm.min_unmapped_ratio = 1
vm.mmap_min_addr = 4096
vm.mmap_rnd_bits = 28
vm.mmap_rnd_compat_bits = 8
vm.nr_hugepages = 0
vm.nr_hugepages_mempolicy = 0
vm.nr_overcommit_hugepages = 0
vm.numa_stat = 1
vm.numa_zonelist_order = Node
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_kbytes = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.stat_interval = 1
vm.swappiness = 60
vm.unprivileged_userfaultfd = 1
vm.user_reserve_kbytes = 131072
vm.vfs_cache_pressure = 100
vm.watermark_boost_factor = 15000
vm.watermark_scale_factor = 10
vm.zone_reclaim_mode = 0

[-- Attachment #3: senderconf --]
[-- Type: application/octet-stream, Size: 73639 bytes --]

ens5      Link encap:Ethernet  HWaddr 06:a7:ff:c5:b7:36  
          inet addr:172.31.24.182  Bcast:172.31.31.255  Mask:255.255.240.0
          inet6 addr: fe80::4a7:ffff:fec5:b736/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
          RX packets:3224349 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6165225 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2845092012 (2.8 GB)  TX bytes:8458720408 (8.4 GB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:200 errors:0 dropped:0 overruns:0 frame:0
          TX packets:200 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:15040 (15.0 KB)  TX bytes:15040 (15.0 KB)

abi.vsyscall32 = 1
debug.exception-trace = 1
debug.kprobes-optimization = 1
dev.cdrom.autoclose = 1
dev.cdrom.autoeject = 0
dev.cdrom.check_media = 0
dev.cdrom.debug = 0
dev.cdrom.info = CD-ROM information, Id: cdrom.c 3.20 2003/12/17
dev.cdrom.info = 
dev.cdrom.info = drive name:	
dev.cdrom.info = drive speed:	
dev.cdrom.info = drive # of slots:
dev.cdrom.info = Can close tray:	
dev.cdrom.info = Can open tray:	
dev.cdrom.info = Can lock tray:	
dev.cdrom.info = Can change speed:
dev.cdrom.info = Can select disk:
dev.cdrom.info = Can read multisession:
dev.cdrom.info = Can read MCN:	
dev.cdrom.info = Reports media changed:
dev.cdrom.info = Can play audio:	
dev.cdrom.info = Can write CD-R:	
dev.cdrom.info = Can write CD-RW:
dev.cdrom.info = Can read DVD:	
dev.cdrom.info = Can write DVD-R:
dev.cdrom.info = Can write DVD-RAM:
dev.cdrom.info = Can read MRW:	
dev.cdrom.info = Can write MRW:	
dev.cdrom.info = Can write RAM:	
dev.cdrom.info = 
dev.cdrom.info = 
dev.cdrom.lock = 1
dev.hpet.max-user-freq = 64
dev.mac_hid.mouse_button2_keycode = 97
dev.mac_hid.mouse_button3_keycode = 100
dev.mac_hid.mouse_button_emulation = 0
dev.parport.default.spintime = 500
dev.parport.default.timeslice = 200
dev.raid.speed_limit_max = 200000
dev.raid.speed_limit_min = 1000
dev.scsi.logging_level = 0
dev.tty.ldisc_autoload = 1
fs.aio-max-nr = 65536
fs.aio-nr = 0
fs.binfmt_misc.status = enabled
fs.dentry-state = 130931	109108	45	0	29329	0
fs.dir-notify-enable = 1
fs.epoll.max_user_watches = 19795148
fs.file-max = 9665226
fs.file-nr = 1824	0	9665226
fs.inode-nr = 101719	437
fs.inode-state = 101719	437	0	0	0	0	0
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 8192
fs.lease-break-time = 45
fs.leases-enable = 1
fs.mount-max = 100000
fs.mqueue.msg_default = 10
fs.mqueue.msg_max = 10
fs.mqueue.msgsize_default = 8192
fs.mqueue.msgsize_max = 8192
fs.mqueue.queues_max = 256
fs.nr_open = 1048576
fs.overflowgid = 65534
fs.overflowuid = 65534
fs.pipe-max-size = 1048576
fs.pipe-user-pages-hard = 0
fs.pipe-user-pages-soft = 16384
fs.protected_fifos = 0
fs.protected_hardlinks = 1
fs.protected_regular = 0
fs.protected_symlinks = 1
fs.quota.allocated_dquots = 0
fs.quota.cache_hits = 0
fs.quota.drops = 0
fs.quota.free_dquots = 0
fs.quota.lookups = 0
fs.quota.reads = 0
fs.quota.syncs = 4
fs.quota.writes = 0
fs.suid_dumpable = 2
fs.verity.require_signatures = 0
fs.xfs.error_level = 3
fs.xfs.filestream_centisecs = 3000
fs.xfs.inherit_noatime = 1
fs.xfs.inherit_nodefrag = 1
fs.xfs.inherit_nodump = 1
fs.xfs.inherit_nosymlinks = 0
fs.xfs.inherit_sync = 1
fs.xfs.irix_sgid_inherit = 0
fs.xfs.irix_symlink_mode = 0
fs.xfs.panic_mask = 0
fs.xfs.rotorstep = 1
fs.xfs.speculative_cow_prealloc_lifetime = 1800
fs.xfs.speculative_prealloc_lifetime = 300
fs.xfs.stats_clear = 0
fs.xfs.xfssyncd_centisecs = 3000
kernel.acct = 4	2	30
kernel.acpi_video_flags = 0
kernel.auto_msgmni = 0
kernel.bootloader_type = 114
kernel.bootloader_version = 2
kernel.bpf_stats_enabled = 0
kernel.cad_pid = 1
kernel.cap_last_cap = 37
kernel.core_pattern = |/usr/share/apport/apport %p %s %c %d %P %E
kernel.core_pipe_limit = 0
kernel.core_uses_pid = 0
kernel.ctrl-alt-del = 0
kernel.dmesg_restrict = 0
kernel.domainname = (none)
kernel.firmware_config.force_sysfs_fallback = 0
kernel.firmware_config.ignore_sysfs_fallback = 0
kernel.ftrace_dump_on_oops = 0
kernel.ftrace_enabled = 1
kernel.hardlockup_all_cpu_backtrace = 0
kernel.hardlockup_panic = 0
kernel.hostname = ip-172-31-24-182
kernel.hotplug = 
kernel.hung_task_check_count = 4194304
kernel.hung_task_check_interval_secs = 0
kernel.hung_task_panic = 0
kernel.hung_task_timeout_secs = 120
kernel.hung_task_warnings = 10
kernel.io_delay_type = 1
kernel.kexec_load_disabled = 0
kernel.keys.gc_delay = 300
kernel.keys.maxbytes = 20000
kernel.keys.maxkeys = 200
kernel.keys.persistent_keyring_expiry = 259200
kernel.keys.root_maxbytes = 25000000
kernel.keys.root_maxkeys = 1000000
kernel.kptr_restrict = 1
kernel.max_lock_depth = 1024
kernel.modprobe = /sbin/modprobe
kernel.modules_disabled = 0
kernel.msg_next_id = -1
kernel.msgmax = 8192
kernel.msgmnb = 16384
kernel.msgmni = 32000
kernel.ngroups_max = 65536
kernel.nmi_watchdog = 0
kernel.ns_last_pid = 33416
kernel.numa_balancing = 0
kernel.numa_balancing_scan_delay_ms = 1000
kernel.numa_balancing_scan_period_max_ms = 60000
kernel.numa_balancing_scan_period_min_ms = 1000
kernel.numa_balancing_scan_size_mb = 256
kernel.osrelease = 5.4.0-rc6
kernel.ostype = Linux
kernel.overflowgid = 65534
kernel.overflowuid = 65534
kernel.panic = 0
kernel.panic_on_io_nmi = 0
kernel.panic_on_oops = 0
kernel.panic_on_rcu_stall = 0
kernel.panic_on_unrecovered_nmi = 0
kernel.panic_on_warn = 0
kernel.panic_print = 0
kernel.perf_cpu_time_max_percent = 25
kernel.perf_event_max_contexts_per_stack = 8
kernel.perf_event_max_sample_rate = 100000
kernel.perf_event_max_stack = 127
kernel.perf_event_mlock_kb = 516
kernel.perf_event_paranoid = 2
kernel.pid_max = 49152
kernel.poweroff_cmd = /sbin/poweroff
kernel.print-fatal-signals = 0
kernel.printk = 4	4	1	7
kernel.printk_delay = 0
kernel.printk_devkmsg = ratelimit
kernel.printk_ratelimit = 5
kernel.printk_ratelimit_burst = 10
kernel.pty.max = 4096
kernel.pty.nr = 8
kernel.pty.reserve = 1024
kernel.random.boot_id = 30408b35-2574-413d-8841-fe755aba6248
kernel.random.entropy_avail = 1178
kernel.random.poolsize = 4096
kernel.random.read_wakeup_threshold = 64
kernel.random.urandom_min_reseed_secs = 60
kernel.random.uuid = a7b78e29-9112-4465-86a0-fdf9abb8ebd1
kernel.random.write_wakeup_threshold = 896
kernel.randomize_va_space = 2
kernel.real-root-dev = 0
kernel.sched_autogroup_enabled = 1
kernel.sched_cfs_bandwidth_slice_us = 5000
kernel.sched_child_runs_first = 0
kernel.sched_domain.cpu0.domain0.busy_factor = 32
kernel.sched_domain.cpu0.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu0.domain0.flags = 4783
kernel.sched_domain.cpu0.domain0.imbalance_pct = 110
kernel.sched_domain.cpu0.domain0.max_interval = 4
kernel.sched_domain.cpu0.domain0.max_newidle_lb_cost = 891
kernel.sched_domain.cpu0.domain0.min_interval = 2
kernel.sched_domain.cpu0.domain0.name = SMT
kernel.sched_domain.cpu0.domain1.busy_factor = 32
kernel.sched_domain.cpu0.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu0.domain1.flags = 4655
kernel.sched_domain.cpu0.domain1.imbalance_pct = 117
kernel.sched_domain.cpu0.domain1.max_interval = 96
kernel.sched_domain.cpu0.domain1.max_newidle_lb_cost = 5252
kernel.sched_domain.cpu0.domain1.min_interval = 48
kernel.sched_domain.cpu0.domain1.name = MC
kernel.sched_domain.cpu1.domain0.busy_factor = 32
kernel.sched_domain.cpu1.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu1.domain0.flags = 4783
kernel.sched_domain.cpu1.domain0.imbalance_pct = 110
kernel.sched_domain.cpu1.domain0.max_interval = 4
kernel.sched_domain.cpu1.domain0.max_newidle_lb_cost = 291
kernel.sched_domain.cpu1.domain0.min_interval = 2
kernel.sched_domain.cpu1.domain0.name = SMT
kernel.sched_domain.cpu1.domain1.busy_factor = 32
kernel.sched_domain.cpu1.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu1.domain1.flags = 4655
kernel.sched_domain.cpu1.domain1.imbalance_pct = 117
kernel.sched_domain.cpu1.domain1.max_interval = 96
kernel.sched_domain.cpu1.domain1.max_newidle_lb_cost = 3068
kernel.sched_domain.cpu1.domain1.min_interval = 48
kernel.sched_domain.cpu1.domain1.name = MC
kernel.sched_domain.cpu10.domain0.busy_factor = 32
kernel.sched_domain.cpu10.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu10.domain0.flags = 4783
kernel.sched_domain.cpu10.domain0.imbalance_pct = 110
kernel.sched_domain.cpu10.domain0.max_interval = 4
kernel.sched_domain.cpu10.domain0.max_newidle_lb_cost = 0
kernel.sched_domain.cpu10.domain0.min_interval = 2
kernel.sched_domain.cpu10.domain0.name = SMT
kernel.sched_domain.cpu10.domain1.busy_factor = 32
kernel.sched_domain.cpu10.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu10.domain1.flags = 4655
kernel.sched_domain.cpu10.domain1.imbalance_pct = 117
kernel.sched_domain.cpu10.domain1.max_interval = 96
kernel.sched_domain.cpu10.domain1.max_newidle_lb_cost = 0
kernel.sched_domain.cpu10.domain1.min_interval = 48
kernel.sched_domain.cpu10.domain1.name = MC
kernel.sched_domain.cpu11.domain0.busy_factor = 32
kernel.sched_domain.cpu11.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu11.domain0.flags = 4783
kernel.sched_domain.cpu11.domain0.imbalance_pct = 110
kernel.sched_domain.cpu11.domain0.max_interval = 4
kernel.sched_domain.cpu11.domain0.max_newidle_lb_cost = 72
kernel.sched_domain.cpu11.domain0.min_interval = 2
kernel.sched_domain.cpu11.domain0.name = SMT
kernel.sched_domain.cpu11.domain1.busy_factor = 32
kernel.sched_domain.cpu11.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu11.domain1.flags = 4655
kernel.sched_domain.cpu11.domain1.imbalance_pct = 117
kernel.sched_domain.cpu11.domain1.max_interval = 96
kernel.sched_domain.cpu11.domain1.max_newidle_lb_cost = 838
kernel.sched_domain.cpu11.domain1.min_interval = 48
kernel.sched_domain.cpu11.domain1.name = MC
kernel.sched_domain.cpu12.domain0.busy_factor = 32
kernel.sched_domain.cpu12.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu12.domain0.flags = 4783
kernel.sched_domain.cpu12.domain0.imbalance_pct = 110
kernel.sched_domain.cpu12.domain0.max_interval = 4
kernel.sched_domain.cpu12.domain0.max_newidle_lb_cost = 512
kernel.sched_domain.cpu12.domain0.min_interval = 2
kernel.sched_domain.cpu12.domain0.name = SMT
kernel.sched_domain.cpu12.domain1.busy_factor = 32
kernel.sched_domain.cpu12.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu12.domain1.flags = 4655
kernel.sched_domain.cpu12.domain1.imbalance_pct = 117
kernel.sched_domain.cpu12.domain1.max_interval = 96
kernel.sched_domain.cpu12.domain1.max_newidle_lb_cost = 5713
kernel.sched_domain.cpu12.domain1.min_interval = 48
kernel.sched_domain.cpu12.domain1.name = MC
kernel.sched_domain.cpu13.domain0.busy_factor = 32
kernel.sched_domain.cpu13.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu13.domain0.flags = 4783
kernel.sched_domain.cpu13.domain0.imbalance_pct = 110
kernel.sched_domain.cpu13.domain0.max_interval = 4
kernel.sched_domain.cpu13.domain0.max_newidle_lb_cost = 658
kernel.sched_domain.cpu13.domain0.min_interval = 2
kernel.sched_domain.cpu13.domain0.name = SMT
kernel.sched_domain.cpu13.domain1.busy_factor = 32
kernel.sched_domain.cpu13.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu13.domain1.flags = 4655
kernel.sched_domain.cpu13.domain1.imbalance_pct = 117
kernel.sched_domain.cpu13.domain1.max_interval = 96
kernel.sched_domain.cpu13.domain1.max_newidle_lb_cost = 2805
kernel.sched_domain.cpu13.domain1.min_interval = 48
kernel.sched_domain.cpu13.domain1.name = MC
kernel.sched_domain.cpu14.domain0.busy_factor = 32
kernel.sched_domain.cpu14.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu14.domain0.flags = 4783
kernel.sched_domain.cpu14.domain0.imbalance_pct = 110
kernel.sched_domain.cpu14.domain0.max_interval = 4
kernel.sched_domain.cpu14.domain0.max_newidle_lb_cost = 0
kernel.sched_domain.cpu14.domain0.min_interval = 2
kernel.sched_domain.cpu14.domain0.name = SMT
kernel.sched_domain.cpu14.domain1.busy_factor = 32
kernel.sched_domain.cpu14.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu14.domain1.flags = 4655
kernel.sched_domain.cpu14.domain1.imbalance_pct = 117
kernel.sched_domain.cpu14.domain1.max_interval = 96
kernel.sched_domain.cpu14.domain1.max_newidle_lb_cost = 57
kernel.sched_domain.cpu14.domain1.min_interval = 48
kernel.sched_domain.cpu14.domain1.name = MC
kernel.sched_domain.cpu15.domain0.busy_factor = 32
kernel.sched_domain.cpu15.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu15.domain0.flags = 4783
kernel.sched_domain.cpu15.domain0.imbalance_pct = 110
kernel.sched_domain.cpu15.domain0.max_interval = 4
kernel.sched_domain.cpu15.domain0.max_newidle_lb_cost = 143
kernel.sched_domain.cpu15.domain0.min_interval = 2
kernel.sched_domain.cpu15.domain0.name = SMT
kernel.sched_domain.cpu15.domain1.busy_factor = 32
kernel.sched_domain.cpu15.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu15.domain1.flags = 4655
kernel.sched_domain.cpu15.domain1.imbalance_pct = 117
kernel.sched_domain.cpu15.domain1.max_interval = 96
kernel.sched_domain.cpu15.domain1.max_newidle_lb_cost = 900
kernel.sched_domain.cpu15.domain1.min_interval = 48
kernel.sched_domain.cpu15.domain1.name = MC
kernel.sched_domain.cpu16.domain0.busy_factor = 32
kernel.sched_domain.cpu16.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu16.domain0.flags = 4783
kernel.sched_domain.cpu16.domain0.imbalance_pct = 110
kernel.sched_domain.cpu16.domain0.max_interval = 4
kernel.sched_domain.cpu16.domain0.max_newidle_lb_cost = 392
kernel.sched_domain.cpu16.domain0.min_interval = 2
kernel.sched_domain.cpu16.domain0.name = SMT
kernel.sched_domain.cpu16.domain1.busy_factor = 32
kernel.sched_domain.cpu16.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu16.domain1.flags = 4655
kernel.sched_domain.cpu16.domain1.imbalance_pct = 117
kernel.sched_domain.cpu16.domain1.max_interval = 96
kernel.sched_domain.cpu16.domain1.max_newidle_lb_cost = 2618
kernel.sched_domain.cpu16.domain1.min_interval = 48
kernel.sched_domain.cpu16.domain1.name = MC
kernel.sched_domain.cpu17.domain0.busy_factor = 32
kernel.sched_domain.cpu17.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu17.domain0.flags = 4783
kernel.sched_domain.cpu17.domain0.imbalance_pct = 110
kernel.sched_domain.cpu17.domain0.max_interval = 4
kernel.sched_domain.cpu17.domain0.max_newidle_lb_cost = 246
kernel.sched_domain.cpu17.domain0.min_interval = 2
kernel.sched_domain.cpu17.domain0.name = SMT
kernel.sched_domain.cpu17.domain1.busy_factor = 32
kernel.sched_domain.cpu17.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu17.domain1.flags = 4655
kernel.sched_domain.cpu17.domain1.imbalance_pct = 117
kernel.sched_domain.cpu17.domain1.max_interval = 96
kernel.sched_domain.cpu17.domain1.max_newidle_lb_cost = 13723
kernel.sched_domain.cpu17.domain1.min_interval = 48
kernel.sched_domain.cpu17.domain1.name = MC
kernel.sched_domain.cpu18.domain0.busy_factor = 32
kernel.sched_domain.cpu18.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu18.domain0.flags = 4783
kernel.sched_domain.cpu18.domain0.imbalance_pct = 110
kernel.sched_domain.cpu18.domain0.max_interval = 4
kernel.sched_domain.cpu18.domain0.max_newidle_lb_cost = 564
kernel.sched_domain.cpu18.domain0.min_interval = 2
kernel.sched_domain.cpu18.domain0.name = SMT
kernel.sched_domain.cpu18.domain1.busy_factor = 32
kernel.sched_domain.cpu18.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu18.domain1.flags = 4655
kernel.sched_domain.cpu18.domain1.imbalance_pct = 117
kernel.sched_domain.cpu18.domain1.max_interval = 96
kernel.sched_domain.cpu18.domain1.max_newidle_lb_cost = 4134
kernel.sched_domain.cpu18.domain1.min_interval = 48
kernel.sched_domain.cpu18.domain1.name = MC
kernel.sched_domain.cpu19.domain0.busy_factor = 32
kernel.sched_domain.cpu19.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu19.domain0.flags = 4783
kernel.sched_domain.cpu19.domain0.imbalance_pct = 110
kernel.sched_domain.cpu19.domain0.max_interval = 4
kernel.sched_domain.cpu19.domain0.max_newidle_lb_cost = 82
kernel.sched_domain.cpu19.domain0.min_interval = 2
kernel.sched_domain.cpu19.domain0.name = SMT
kernel.sched_domain.cpu19.domain1.busy_factor = 32
kernel.sched_domain.cpu19.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu19.domain1.flags = 4655
kernel.sched_domain.cpu19.domain1.imbalance_pct = 117
kernel.sched_domain.cpu19.domain1.max_interval = 96
kernel.sched_domain.cpu19.domain1.max_newidle_lb_cost = 2186
kernel.sched_domain.cpu19.domain1.min_interval = 48
kernel.sched_domain.cpu19.domain1.name = MC
kernel.sched_domain.cpu2.domain0.busy_factor = 32
kernel.sched_domain.cpu2.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu2.domain0.flags = 4783
kernel.sched_domain.cpu2.domain0.imbalance_pct = 110
kernel.sched_domain.cpu2.domain0.max_interval = 4
kernel.sched_domain.cpu2.domain0.max_newidle_lb_cost = 926
kernel.sched_domain.cpu2.domain0.min_interval = 2
kernel.sched_domain.cpu2.domain0.name = SMT
kernel.sched_domain.cpu2.domain1.busy_factor = 32
kernel.sched_domain.cpu2.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu2.domain1.flags = 4655
kernel.sched_domain.cpu2.domain1.imbalance_pct = 117
kernel.sched_domain.cpu2.domain1.max_interval = 96
kernel.sched_domain.cpu2.domain1.max_newidle_lb_cost = 8898
kernel.sched_domain.cpu2.domain1.min_interval = 48
kernel.sched_domain.cpu2.domain1.name = MC
kernel.sched_domain.cpu20.domain0.busy_factor = 32
kernel.sched_domain.cpu20.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu20.domain0.flags = 4783
kernel.sched_domain.cpu20.domain0.imbalance_pct = 110
kernel.sched_domain.cpu20.domain0.max_interval = 4
kernel.sched_domain.cpu20.domain0.max_newidle_lb_cost = 259
kernel.sched_domain.cpu20.domain0.min_interval = 2
kernel.sched_domain.cpu20.domain0.name = SMT
kernel.sched_domain.cpu20.domain1.busy_factor = 32
kernel.sched_domain.cpu20.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu20.domain1.flags = 4655
kernel.sched_domain.cpu20.domain1.imbalance_pct = 117
kernel.sched_domain.cpu20.domain1.max_interval = 96
kernel.sched_domain.cpu20.domain1.max_newidle_lb_cost = 3818
kernel.sched_domain.cpu20.domain1.min_interval = 48
kernel.sched_domain.cpu20.domain1.name = MC
kernel.sched_domain.cpu21.domain0.busy_factor = 32
kernel.sched_domain.cpu21.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu21.domain0.flags = 4783
kernel.sched_domain.cpu21.domain0.imbalance_pct = 110
kernel.sched_domain.cpu21.domain0.max_interval = 4
kernel.sched_domain.cpu21.domain0.max_newidle_lb_cost = 346
kernel.sched_domain.cpu21.domain0.min_interval = 2
kernel.sched_domain.cpu21.domain0.name = SMT
kernel.sched_domain.cpu21.domain1.busy_factor = 32
kernel.sched_domain.cpu21.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu21.domain1.flags = 4655
kernel.sched_domain.cpu21.domain1.imbalance_pct = 117
kernel.sched_domain.cpu21.domain1.max_interval = 96
kernel.sched_domain.cpu21.domain1.max_newidle_lb_cost = 3853
kernel.sched_domain.cpu21.domain1.min_interval = 48
kernel.sched_domain.cpu21.domain1.name = MC
kernel.sched_domain.cpu22.domain0.busy_factor = 32
kernel.sched_domain.cpu22.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu22.domain0.flags = 4783
kernel.sched_domain.cpu22.domain0.imbalance_pct = 110
kernel.sched_domain.cpu22.domain0.max_interval = 4
kernel.sched_domain.cpu22.domain0.max_newidle_lb_cost = 317
kernel.sched_domain.cpu22.domain0.min_interval = 2
kernel.sched_domain.cpu22.domain0.name = SMT
kernel.sched_domain.cpu22.domain1.busy_factor = 32
kernel.sched_domain.cpu22.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu22.domain1.flags = 4655
kernel.sched_domain.cpu22.domain1.imbalance_pct = 117
kernel.sched_domain.cpu22.domain1.max_interval = 96
kernel.sched_domain.cpu22.domain1.max_newidle_lb_cost = 1984
kernel.sched_domain.cpu22.domain1.min_interval = 48
kernel.sched_domain.cpu22.domain1.name = MC
kernel.sched_domain.cpu23.domain0.busy_factor = 32
kernel.sched_domain.cpu23.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu23.domain0.flags = 4783
kernel.sched_domain.cpu23.domain0.imbalance_pct = 110
kernel.sched_domain.cpu23.domain0.max_interval = 4
kernel.sched_domain.cpu23.domain0.max_newidle_lb_cost = 641
kernel.sched_domain.cpu23.domain0.min_interval = 2
kernel.sched_domain.cpu23.domain0.name = SMT
kernel.sched_domain.cpu23.domain1.busy_factor = 32
kernel.sched_domain.cpu23.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu23.domain1.flags = 4655
kernel.sched_domain.cpu23.domain1.imbalance_pct = 117
kernel.sched_domain.cpu23.domain1.max_interval = 96
kernel.sched_domain.cpu23.domain1.max_newidle_lb_cost = 13867
kernel.sched_domain.cpu23.domain1.min_interval = 48
kernel.sched_domain.cpu23.domain1.name = MC
kernel.sched_domain.cpu24.domain0.busy_factor = 32
kernel.sched_domain.cpu24.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu24.domain0.flags = 4783
kernel.sched_domain.cpu24.domain0.imbalance_pct = 110
kernel.sched_domain.cpu24.domain0.max_interval = 4
kernel.sched_domain.cpu24.domain0.max_newidle_lb_cost = 901
kernel.sched_domain.cpu24.domain0.min_interval = 2
kernel.sched_domain.cpu24.domain0.name = SMT
kernel.sched_domain.cpu24.domain1.busy_factor = 32
kernel.sched_domain.cpu24.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu24.domain1.flags = 4655
kernel.sched_domain.cpu24.domain1.imbalance_pct = 117
kernel.sched_domain.cpu24.domain1.max_interval = 96
kernel.sched_domain.cpu24.domain1.max_newidle_lb_cost = 5323
kernel.sched_domain.cpu24.domain1.min_interval = 48
kernel.sched_domain.cpu24.domain1.name = MC
kernel.sched_domain.cpu25.domain0.busy_factor = 32
kernel.sched_domain.cpu25.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu25.domain0.flags = 4783
kernel.sched_domain.cpu25.domain0.imbalance_pct = 110
kernel.sched_domain.cpu25.domain0.max_interval = 4
kernel.sched_domain.cpu25.domain0.max_newidle_lb_cost = 370
kernel.sched_domain.cpu25.domain0.min_interval = 2
kernel.sched_domain.cpu25.domain0.name = SMT
kernel.sched_domain.cpu25.domain1.busy_factor = 32
kernel.sched_domain.cpu25.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu25.domain1.flags = 4655
kernel.sched_domain.cpu25.domain1.imbalance_pct = 117
kernel.sched_domain.cpu25.domain1.max_interval = 96
kernel.sched_domain.cpu25.domain1.max_newidle_lb_cost = 3380
kernel.sched_domain.cpu25.domain1.min_interval = 48
kernel.sched_domain.cpu25.domain1.name = MC
kernel.sched_domain.cpu26.domain0.busy_factor = 32
kernel.sched_domain.cpu26.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu26.domain0.flags = 4783
kernel.sched_domain.cpu26.domain0.imbalance_pct = 110
kernel.sched_domain.cpu26.domain0.max_interval = 4
kernel.sched_domain.cpu26.domain0.max_newidle_lb_cost = 67
kernel.sched_domain.cpu26.domain0.min_interval = 2
kernel.sched_domain.cpu26.domain0.name = SMT
kernel.sched_domain.cpu26.domain1.busy_factor = 32
kernel.sched_domain.cpu26.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu26.domain1.flags = 4655
kernel.sched_domain.cpu26.domain1.imbalance_pct = 117
kernel.sched_domain.cpu26.domain1.max_interval = 96
kernel.sched_domain.cpu26.domain1.max_newidle_lb_cost = 840
kernel.sched_domain.cpu26.domain1.min_interval = 48
kernel.sched_domain.cpu26.domain1.name = MC
kernel.sched_domain.cpu27.domain0.busy_factor = 32
kernel.sched_domain.cpu27.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu27.domain0.flags = 4783
kernel.sched_domain.cpu27.domain0.imbalance_pct = 110
kernel.sched_domain.cpu27.domain0.max_interval = 4
kernel.sched_domain.cpu27.domain0.max_newidle_lb_cost = 1146
kernel.sched_domain.cpu27.domain0.min_interval = 2
kernel.sched_domain.cpu27.domain0.name = SMT
kernel.sched_domain.cpu27.domain1.busy_factor = 32
kernel.sched_domain.cpu27.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu27.domain1.flags = 4655
kernel.sched_domain.cpu27.domain1.imbalance_pct = 117
kernel.sched_domain.cpu27.domain1.max_interval = 96
kernel.sched_domain.cpu27.domain1.max_newidle_lb_cost = 4451
kernel.sched_domain.cpu27.domain1.min_interval = 48
kernel.sched_domain.cpu27.domain1.name = MC
kernel.sched_domain.cpu28.domain0.busy_factor = 32
kernel.sched_domain.cpu28.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu28.domain0.flags = 4783
kernel.sched_domain.cpu28.domain0.imbalance_pct = 110
kernel.sched_domain.cpu28.domain0.max_interval = 4
kernel.sched_domain.cpu28.domain0.max_newidle_lb_cost = 483
kernel.sched_domain.cpu28.domain0.min_interval = 2
kernel.sched_domain.cpu28.domain0.name = SMT
kernel.sched_domain.cpu28.domain1.busy_factor = 32
kernel.sched_domain.cpu28.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu28.domain1.flags = 4655
kernel.sched_domain.cpu28.domain1.imbalance_pct = 117
kernel.sched_domain.cpu28.domain1.max_interval = 96
kernel.sched_domain.cpu28.domain1.max_newidle_lb_cost = 3296
kernel.sched_domain.cpu28.domain1.min_interval = 48
kernel.sched_domain.cpu28.domain1.name = MC
kernel.sched_domain.cpu29.domain0.busy_factor = 32
kernel.sched_domain.cpu29.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu29.domain0.flags = 4783
kernel.sched_domain.cpu29.domain0.imbalance_pct = 110
kernel.sched_domain.cpu29.domain0.max_interval = 4
kernel.sched_domain.cpu29.domain0.max_newidle_lb_cost = 163
kernel.sched_domain.cpu29.domain0.min_interval = 2
kernel.sched_domain.cpu29.domain0.name = SMT
kernel.sched_domain.cpu29.domain1.busy_factor = 32
kernel.sched_domain.cpu29.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu29.domain1.flags = 4655
kernel.sched_domain.cpu29.domain1.imbalance_pct = 117
kernel.sched_domain.cpu29.domain1.max_interval = 96
kernel.sched_domain.cpu29.domain1.max_newidle_lb_cost = 1191
kernel.sched_domain.cpu29.domain1.min_interval = 48
kernel.sched_domain.cpu29.domain1.name = MC
kernel.sched_domain.cpu3.domain0.busy_factor = 32
kernel.sched_domain.cpu3.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu3.domain0.flags = 4783
kernel.sched_domain.cpu3.domain0.imbalance_pct = 110
kernel.sched_domain.cpu3.domain0.max_interval = 4
kernel.sched_domain.cpu3.domain0.max_newidle_lb_cost = 528
kernel.sched_domain.cpu3.domain0.min_interval = 2
kernel.sched_domain.cpu3.domain0.name = SMT
kernel.sched_domain.cpu3.domain1.busy_factor = 32
kernel.sched_domain.cpu3.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu3.domain1.flags = 4655
kernel.sched_domain.cpu3.domain1.imbalance_pct = 117
kernel.sched_domain.cpu3.domain1.max_interval = 96
kernel.sched_domain.cpu3.domain1.max_newidle_lb_cost = 4269
kernel.sched_domain.cpu3.domain1.min_interval = 48
kernel.sched_domain.cpu3.domain1.name = MC
kernel.sched_domain.cpu30.domain0.busy_factor = 32
kernel.sched_domain.cpu30.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu30.domain0.flags = 4783
kernel.sched_domain.cpu30.domain0.imbalance_pct = 110
kernel.sched_domain.cpu30.domain0.max_interval = 4
kernel.sched_domain.cpu30.domain0.max_newidle_lb_cost = 0
kernel.sched_domain.cpu30.domain0.min_interval = 2
kernel.sched_domain.cpu30.domain0.name = SMT
kernel.sched_domain.cpu30.domain1.busy_factor = 32
kernel.sched_domain.cpu30.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu30.domain1.flags = 4655
kernel.sched_domain.cpu30.domain1.imbalance_pct = 117
kernel.sched_domain.cpu30.domain1.max_interval = 96
kernel.sched_domain.cpu30.domain1.max_newidle_lb_cost = 0
kernel.sched_domain.cpu30.domain1.min_interval = 48
kernel.sched_domain.cpu30.domain1.name = MC
kernel.sched_domain.cpu31.domain0.busy_factor = 32
kernel.sched_domain.cpu31.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu31.domain0.flags = 4783
kernel.sched_domain.cpu31.domain0.imbalance_pct = 110
kernel.sched_domain.cpu31.domain0.max_interval = 4
kernel.sched_domain.cpu31.domain0.max_newidle_lb_cost = 629
kernel.sched_domain.cpu31.domain0.min_interval = 2
kernel.sched_domain.cpu31.domain0.name = SMT
kernel.sched_domain.cpu31.domain1.busy_factor = 32
kernel.sched_domain.cpu31.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu31.domain1.flags = 4655
kernel.sched_domain.cpu31.domain1.imbalance_pct = 117
kernel.sched_domain.cpu31.domain1.max_interval = 96
kernel.sched_domain.cpu31.domain1.max_newidle_lb_cost = 3757
kernel.sched_domain.cpu31.domain1.min_interval = 48
kernel.sched_domain.cpu31.domain1.name = MC
kernel.sched_domain.cpu32.domain0.busy_factor = 32
kernel.sched_domain.cpu32.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu32.domain0.flags = 4783
kernel.sched_domain.cpu32.domain0.imbalance_pct = 110
kernel.sched_domain.cpu32.domain0.max_interval = 4
kernel.sched_domain.cpu32.domain0.max_newidle_lb_cost = 617
kernel.sched_domain.cpu32.domain0.min_interval = 2
kernel.sched_domain.cpu32.domain0.name = SMT
kernel.sched_domain.cpu32.domain1.busy_factor = 32
kernel.sched_domain.cpu32.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu32.domain1.flags = 4655
kernel.sched_domain.cpu32.domain1.imbalance_pct = 117
kernel.sched_domain.cpu32.domain1.max_interval = 96
kernel.sched_domain.cpu32.domain1.max_newidle_lb_cost = 3582
kernel.sched_domain.cpu32.domain1.min_interval = 48
kernel.sched_domain.cpu32.domain1.name = MC
kernel.sched_domain.cpu33.domain0.busy_factor = 32
kernel.sched_domain.cpu33.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu33.domain0.flags = 4783
kernel.sched_domain.cpu33.domain0.imbalance_pct = 110
kernel.sched_domain.cpu33.domain0.max_interval = 4
kernel.sched_domain.cpu33.domain0.max_newidle_lb_cost = 338
kernel.sched_domain.cpu33.domain0.min_interval = 2
kernel.sched_domain.cpu33.domain0.name = SMT
kernel.sched_domain.cpu33.domain1.busy_factor = 32
kernel.sched_domain.cpu33.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu33.domain1.flags = 4655
kernel.sched_domain.cpu33.domain1.imbalance_pct = 117
kernel.sched_domain.cpu33.domain1.max_interval = 96
kernel.sched_domain.cpu33.domain1.max_newidle_lb_cost = 4608
kernel.sched_domain.cpu33.domain1.min_interval = 48
kernel.sched_domain.cpu33.domain1.name = MC
kernel.sched_domain.cpu34.domain0.busy_factor = 32
kernel.sched_domain.cpu34.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu34.domain0.flags = 4783
kernel.sched_domain.cpu34.domain0.imbalance_pct = 110
kernel.sched_domain.cpu34.domain0.max_interval = 4
kernel.sched_domain.cpu34.domain0.max_newidle_lb_cost = 662
kernel.sched_domain.cpu34.domain0.min_interval = 2
kernel.sched_domain.cpu34.domain0.name = SMT
kernel.sched_domain.cpu34.domain1.busy_factor = 32
kernel.sched_domain.cpu34.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu34.domain1.flags = 4655
kernel.sched_domain.cpu34.domain1.imbalance_pct = 117
kernel.sched_domain.cpu34.domain1.max_interval = 96
kernel.sched_domain.cpu34.domain1.max_newidle_lb_cost = 5312
kernel.sched_domain.cpu34.domain1.min_interval = 48
kernel.sched_domain.cpu34.domain1.name = MC
kernel.sched_domain.cpu35.domain0.busy_factor = 32
kernel.sched_domain.cpu35.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu35.domain0.flags = 4783
kernel.sched_domain.cpu35.domain0.imbalance_pct = 110
kernel.sched_domain.cpu35.domain0.max_interval = 4
kernel.sched_domain.cpu35.domain0.max_newidle_lb_cost = 769
kernel.sched_domain.cpu35.domain0.min_interval = 2
kernel.sched_domain.cpu35.domain0.name = SMT
kernel.sched_domain.cpu35.domain1.busy_factor = 32
kernel.sched_domain.cpu35.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu35.domain1.flags = 4655
kernel.sched_domain.cpu35.domain1.imbalance_pct = 117
kernel.sched_domain.cpu35.domain1.max_interval = 96
kernel.sched_domain.cpu35.domain1.max_newidle_lb_cost = 9419
kernel.sched_domain.cpu35.domain1.min_interval = 48
kernel.sched_domain.cpu35.domain1.name = MC
kernel.sched_domain.cpu36.domain0.busy_factor = 32
kernel.sched_domain.cpu36.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu36.domain0.flags = 4783
kernel.sched_domain.cpu36.domain0.imbalance_pct = 110
kernel.sched_domain.cpu36.domain0.max_interval = 4
kernel.sched_domain.cpu36.domain0.max_newidle_lb_cost = 813
kernel.sched_domain.cpu36.domain0.min_interval = 2
kernel.sched_domain.cpu36.domain0.name = SMT
kernel.sched_domain.cpu36.domain1.busy_factor = 32
kernel.sched_domain.cpu36.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu36.domain1.flags = 4655
kernel.sched_domain.cpu36.domain1.imbalance_pct = 117
kernel.sched_domain.cpu36.domain1.max_interval = 96
kernel.sched_domain.cpu36.domain1.max_newidle_lb_cost = 3822
kernel.sched_domain.cpu36.domain1.min_interval = 48
kernel.sched_domain.cpu36.domain1.name = MC
kernel.sched_domain.cpu37.domain0.busy_factor = 32
kernel.sched_domain.cpu37.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu37.domain0.flags = 4783
kernel.sched_domain.cpu37.domain0.imbalance_pct = 110
kernel.sched_domain.cpu37.domain0.max_interval = 4
kernel.sched_domain.cpu37.domain0.max_newidle_lb_cost = 106
kernel.sched_domain.cpu37.domain0.min_interval = 2
kernel.sched_domain.cpu37.domain0.name = SMT
kernel.sched_domain.cpu37.domain1.busy_factor = 32
kernel.sched_domain.cpu37.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu37.domain1.flags = 4655
kernel.sched_domain.cpu37.domain1.imbalance_pct = 117
kernel.sched_domain.cpu37.domain1.max_interval = 96
kernel.sched_domain.cpu37.domain1.max_newidle_lb_cost = 783
kernel.sched_domain.cpu37.domain1.min_interval = 48
kernel.sched_domain.cpu37.domain1.name = MC
kernel.sched_domain.cpu38.domain0.busy_factor = 32
kernel.sched_domain.cpu38.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu38.domain0.flags = 4783
kernel.sched_domain.cpu38.domain0.imbalance_pct = 110
kernel.sched_domain.cpu38.domain0.max_interval = 4
kernel.sched_domain.cpu38.domain0.max_newidle_lb_cost = 0
kernel.sched_domain.cpu38.domain0.min_interval = 2
kernel.sched_domain.cpu38.domain0.name = SMT
kernel.sched_domain.cpu38.domain1.busy_factor = 32
kernel.sched_domain.cpu38.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu38.domain1.flags = 4655
kernel.sched_domain.cpu38.domain1.imbalance_pct = 117
kernel.sched_domain.cpu38.domain1.max_interval = 96
kernel.sched_domain.cpu38.domain1.max_newidle_lb_cost = 0
kernel.sched_domain.cpu38.domain1.min_interval = 48
kernel.sched_domain.cpu38.domain1.name = MC
kernel.sched_domain.cpu39.domain0.busy_factor = 32
kernel.sched_domain.cpu39.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu39.domain0.flags = 4783
kernel.sched_domain.cpu39.domain0.imbalance_pct = 110
kernel.sched_domain.cpu39.domain0.max_interval = 4
kernel.sched_domain.cpu39.domain0.max_newidle_lb_cost = 165
kernel.sched_domain.cpu39.domain0.min_interval = 2
kernel.sched_domain.cpu39.domain0.name = SMT
kernel.sched_domain.cpu39.domain1.busy_factor = 32
kernel.sched_domain.cpu39.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu39.domain1.flags = 4655
kernel.sched_domain.cpu39.domain1.imbalance_pct = 117
kernel.sched_domain.cpu39.domain1.max_interval = 96
kernel.sched_domain.cpu39.domain1.max_newidle_lb_cost = 935
kernel.sched_domain.cpu39.domain1.min_interval = 48
kernel.sched_domain.cpu39.domain1.name = MC
kernel.sched_domain.cpu4.domain0.busy_factor = 32
kernel.sched_domain.cpu4.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu4.domain0.flags = 4783
kernel.sched_domain.cpu4.domain0.imbalance_pct = 110
kernel.sched_domain.cpu4.domain0.max_interval = 4
kernel.sched_domain.cpu4.domain0.max_newidle_lb_cost = 655
kernel.sched_domain.cpu4.domain0.min_interval = 2
kernel.sched_domain.cpu4.domain0.name = SMT
kernel.sched_domain.cpu4.domain1.busy_factor = 32
kernel.sched_domain.cpu4.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu4.domain1.flags = 4655
kernel.sched_domain.cpu4.domain1.imbalance_pct = 117
kernel.sched_domain.cpu4.domain1.max_interval = 96
kernel.sched_domain.cpu4.domain1.max_newidle_lb_cost = 13492
kernel.sched_domain.cpu4.domain1.min_interval = 48
kernel.sched_domain.cpu4.domain1.name = MC
kernel.sched_domain.cpu40.domain0.busy_factor = 32
kernel.sched_domain.cpu40.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu40.domain0.flags = 4783
kernel.sched_domain.cpu40.domain0.imbalance_pct = 110
kernel.sched_domain.cpu40.domain0.max_interval = 4
kernel.sched_domain.cpu40.domain0.max_newidle_lb_cost = 94
kernel.sched_domain.cpu40.domain0.min_interval = 2
kernel.sched_domain.cpu40.domain0.name = SMT
kernel.sched_domain.cpu40.domain1.busy_factor = 32
kernel.sched_domain.cpu40.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu40.domain1.flags = 4655
kernel.sched_domain.cpu40.domain1.imbalance_pct = 117
kernel.sched_domain.cpu40.domain1.max_interval = 96
kernel.sched_domain.cpu40.domain1.max_newidle_lb_cost = 630
kernel.sched_domain.cpu40.domain1.min_interval = 48
kernel.sched_domain.cpu40.domain1.name = MC
kernel.sched_domain.cpu41.domain0.busy_factor = 32
kernel.sched_domain.cpu41.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu41.domain0.flags = 4783
kernel.sched_domain.cpu41.domain0.imbalance_pct = 110
kernel.sched_domain.cpu41.domain0.max_interval = 4
kernel.sched_domain.cpu41.domain0.max_newidle_lb_cost = 0
kernel.sched_domain.cpu41.domain0.min_interval = 2
kernel.sched_domain.cpu41.domain0.name = SMT
kernel.sched_domain.cpu41.domain1.busy_factor = 32
kernel.sched_domain.cpu41.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu41.domain1.flags = 4655
kernel.sched_domain.cpu41.domain1.imbalance_pct = 117
kernel.sched_domain.cpu41.domain1.max_interval = 96
kernel.sched_domain.cpu41.domain1.max_newidle_lb_cost = 0
kernel.sched_domain.cpu41.domain1.min_interval = 48
kernel.sched_domain.cpu41.domain1.name = MC
kernel.sched_domain.cpu42.domain0.busy_factor = 32
kernel.sched_domain.cpu42.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu42.domain0.flags = 4783
kernel.sched_domain.cpu42.domain0.imbalance_pct = 110
kernel.sched_domain.cpu42.domain0.max_interval = 4
kernel.sched_domain.cpu42.domain0.max_newidle_lb_cost = 532
kernel.sched_domain.cpu42.domain0.min_interval = 2
kernel.sched_domain.cpu42.domain0.name = SMT
kernel.sched_domain.cpu42.domain1.busy_factor = 32
kernel.sched_domain.cpu42.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu42.domain1.flags = 4655
kernel.sched_domain.cpu42.domain1.imbalance_pct = 117
kernel.sched_domain.cpu42.domain1.max_interval = 96
kernel.sched_domain.cpu42.domain1.max_newidle_lb_cost = 3944
kernel.sched_domain.cpu42.domain1.min_interval = 48
kernel.sched_domain.cpu42.domain1.name = MC
kernel.sched_domain.cpu43.domain0.busy_factor = 32
kernel.sched_domain.cpu43.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu43.domain0.flags = 4783
kernel.sched_domain.cpu43.domain0.imbalance_pct = 110
kernel.sched_domain.cpu43.domain0.max_interval = 4
kernel.sched_domain.cpu43.domain0.max_newidle_lb_cost = 696
kernel.sched_domain.cpu43.domain0.min_interval = 2
kernel.sched_domain.cpu43.domain0.name = SMT
kernel.sched_domain.cpu43.domain1.busy_factor = 32
kernel.sched_domain.cpu43.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu43.domain1.flags = 4655
kernel.sched_domain.cpu43.domain1.imbalance_pct = 117
kernel.sched_domain.cpu43.domain1.max_interval = 96
kernel.sched_domain.cpu43.domain1.max_newidle_lb_cost = 5533
kernel.sched_domain.cpu43.domain1.min_interval = 48
kernel.sched_domain.cpu43.domain1.name = MC
kernel.sched_domain.cpu44.domain0.busy_factor = 32
kernel.sched_domain.cpu44.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu44.domain0.flags = 4783
kernel.sched_domain.cpu44.domain0.imbalance_pct = 110
kernel.sched_domain.cpu44.domain0.max_interval = 4
kernel.sched_domain.cpu44.domain0.max_newidle_lb_cost = 281
kernel.sched_domain.cpu44.domain0.min_interval = 2
kernel.sched_domain.cpu44.domain0.name = SMT
kernel.sched_domain.cpu44.domain1.busy_factor = 32
kernel.sched_domain.cpu44.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu44.domain1.flags = 4655
kernel.sched_domain.cpu44.domain1.imbalance_pct = 117
kernel.sched_domain.cpu44.domain1.max_interval = 96
kernel.sched_domain.cpu44.domain1.max_newidle_lb_cost = 3730
kernel.sched_domain.cpu44.domain1.min_interval = 48
kernel.sched_domain.cpu44.domain1.name = MC
kernel.sched_domain.cpu45.domain0.busy_factor = 32
kernel.sched_domain.cpu45.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu45.domain0.flags = 4783
kernel.sched_domain.cpu45.domain0.imbalance_pct = 110
kernel.sched_domain.cpu45.domain0.max_interval = 4
kernel.sched_domain.cpu45.domain0.max_newidle_lb_cost = 214
kernel.sched_domain.cpu45.domain0.min_interval = 2
kernel.sched_domain.cpu45.domain0.name = SMT
kernel.sched_domain.cpu45.domain1.busy_factor = 32
kernel.sched_domain.cpu45.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu45.domain1.flags = 4655
kernel.sched_domain.cpu45.domain1.imbalance_pct = 117
kernel.sched_domain.cpu45.domain1.max_interval = 96
kernel.sched_domain.cpu45.domain1.max_newidle_lb_cost = 2498
kernel.sched_domain.cpu45.domain1.min_interval = 48
kernel.sched_domain.cpu45.domain1.name = MC
kernel.sched_domain.cpu46.domain0.busy_factor = 32
kernel.sched_domain.cpu46.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu46.domain0.flags = 4783
kernel.sched_domain.cpu46.domain0.imbalance_pct = 110
kernel.sched_domain.cpu46.domain0.max_interval = 4
kernel.sched_domain.cpu46.domain0.max_newidle_lb_cost = 0
kernel.sched_domain.cpu46.domain0.min_interval = 2
kernel.sched_domain.cpu46.domain0.name = SMT
kernel.sched_domain.cpu46.domain1.busy_factor = 32
kernel.sched_domain.cpu46.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu46.domain1.flags = 4655
kernel.sched_domain.cpu46.domain1.imbalance_pct = 117
kernel.sched_domain.cpu46.domain1.max_interval = 96
kernel.sched_domain.cpu46.domain1.max_newidle_lb_cost = 0
kernel.sched_domain.cpu46.domain1.min_interval = 48
kernel.sched_domain.cpu46.domain1.name = MC
kernel.sched_domain.cpu47.domain0.busy_factor = 32
kernel.sched_domain.cpu47.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu47.domain0.flags = 4783
kernel.sched_domain.cpu47.domain0.imbalance_pct = 110
kernel.sched_domain.cpu47.domain0.max_interval = 4
kernel.sched_domain.cpu47.domain0.max_newidle_lb_cost = 148
kernel.sched_domain.cpu47.domain0.min_interval = 2
kernel.sched_domain.cpu47.domain0.name = SMT
kernel.sched_domain.cpu47.domain1.busy_factor = 32
kernel.sched_domain.cpu47.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu47.domain1.flags = 4655
kernel.sched_domain.cpu47.domain1.imbalance_pct = 117
kernel.sched_domain.cpu47.domain1.max_interval = 96
kernel.sched_domain.cpu47.domain1.max_newidle_lb_cost = 1138
kernel.sched_domain.cpu47.domain1.min_interval = 48
kernel.sched_domain.cpu47.domain1.name = MC
kernel.sched_domain.cpu5.domain0.busy_factor = 32
kernel.sched_domain.cpu5.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu5.domain0.flags = 4783
kernel.sched_domain.cpu5.domain0.imbalance_pct = 110
kernel.sched_domain.cpu5.domain0.max_interval = 4
kernel.sched_domain.cpu5.domain0.max_newidle_lb_cost = 292
kernel.sched_domain.cpu5.domain0.min_interval = 2
kernel.sched_domain.cpu5.domain0.name = SMT
kernel.sched_domain.cpu5.domain1.busy_factor = 32
kernel.sched_domain.cpu5.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu5.domain1.flags = 4655
kernel.sched_domain.cpu5.domain1.imbalance_pct = 117
kernel.sched_domain.cpu5.domain1.max_interval = 96
kernel.sched_domain.cpu5.domain1.max_newidle_lb_cost = 2502
kernel.sched_domain.cpu5.domain1.min_interval = 48
kernel.sched_domain.cpu5.domain1.name = MC
kernel.sched_domain.cpu6.domain0.busy_factor = 32
kernel.sched_domain.cpu6.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu6.domain0.flags = 4783
kernel.sched_domain.cpu6.domain0.imbalance_pct = 110
kernel.sched_domain.cpu6.domain0.max_interval = 4
kernel.sched_domain.cpu6.domain0.max_newidle_lb_cost = 698
kernel.sched_domain.cpu6.domain0.min_interval = 2
kernel.sched_domain.cpu6.domain0.name = SMT
kernel.sched_domain.cpu6.domain1.busy_factor = 32
kernel.sched_domain.cpu6.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu6.domain1.flags = 4655
kernel.sched_domain.cpu6.domain1.imbalance_pct = 117
kernel.sched_domain.cpu6.domain1.max_interval = 96
kernel.sched_domain.cpu6.domain1.max_newidle_lb_cost = 14389
kernel.sched_domain.cpu6.domain1.min_interval = 48
kernel.sched_domain.cpu6.domain1.name = MC
kernel.sched_domain.cpu7.domain0.busy_factor = 32
kernel.sched_domain.cpu7.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu7.domain0.flags = 4783
kernel.sched_domain.cpu7.domain0.imbalance_pct = 110
kernel.sched_domain.cpu7.domain0.max_interval = 4
kernel.sched_domain.cpu7.domain0.max_newidle_lb_cost = 517
kernel.sched_domain.cpu7.domain0.min_interval = 2
kernel.sched_domain.cpu7.domain0.name = SMT
kernel.sched_domain.cpu7.domain1.busy_factor = 32
kernel.sched_domain.cpu7.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu7.domain1.flags = 4655
kernel.sched_domain.cpu7.domain1.imbalance_pct = 117
kernel.sched_domain.cpu7.domain1.max_interval = 96
kernel.sched_domain.cpu7.domain1.max_newidle_lb_cost = 3315
kernel.sched_domain.cpu7.domain1.min_interval = 48
kernel.sched_domain.cpu7.domain1.name = MC
kernel.sched_domain.cpu8.domain0.busy_factor = 32
kernel.sched_domain.cpu8.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu8.domain0.flags = 4783
kernel.sched_domain.cpu8.domain0.imbalance_pct = 110
kernel.sched_domain.cpu8.domain0.max_interval = 4
kernel.sched_domain.cpu8.domain0.max_newidle_lb_cost = 1873
kernel.sched_domain.cpu8.domain0.min_interval = 2
kernel.sched_domain.cpu8.domain0.name = SMT
kernel.sched_domain.cpu8.domain1.busy_factor = 32
kernel.sched_domain.cpu8.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu8.domain1.flags = 4655
kernel.sched_domain.cpu8.domain1.imbalance_pct = 117
kernel.sched_domain.cpu8.domain1.max_interval = 96
kernel.sched_domain.cpu8.domain1.max_newidle_lb_cost = 12517
kernel.sched_domain.cpu8.domain1.min_interval = 48
kernel.sched_domain.cpu8.domain1.name = MC
kernel.sched_domain.cpu9.domain0.busy_factor = 32
kernel.sched_domain.cpu9.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu9.domain0.flags = 4783
kernel.sched_domain.cpu9.domain0.imbalance_pct = 110
kernel.sched_domain.cpu9.domain0.max_interval = 4
kernel.sched_domain.cpu9.domain0.max_newidle_lb_cost = 44
kernel.sched_domain.cpu9.domain0.min_interval = 2
kernel.sched_domain.cpu9.domain0.name = SMT
kernel.sched_domain.cpu9.domain1.busy_factor = 32
kernel.sched_domain.cpu9.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu9.domain1.flags = 4655
kernel.sched_domain.cpu9.domain1.imbalance_pct = 117
kernel.sched_domain.cpu9.domain1.max_interval = 96
kernel.sched_domain.cpu9.domain1.max_newidle_lb_cost = 931
kernel.sched_domain.cpu9.domain1.min_interval = 48
kernel.sched_domain.cpu9.domain1.name = MC
kernel.sched_latency_ns = 24000000
kernel.sched_migration_cost_ns = 500000
kernel.sched_min_granularity_ns = 3000000
kernel.sched_nr_migrate = 32
kernel.sched_rr_timeslice_ms = 100
kernel.sched_rt_period_us = 1000000
kernel.sched_rt_runtime_us = 950000
kernel.sched_schedstats = 0
kernel.sched_tunable_scaling = 1
kernel.sched_util_clamp_max = 1024
kernel.sched_util_clamp_min = 1024
kernel.sched_wakeup_granularity_ns = 4000000
kernel.seccomp.actions_avail = kill_process kill_thread trap errno user_notif trace log allow
kernel.seccomp.actions_logged = kill_process kill_thread trap errno user_notif trace log
kernel.sem = 32000	1024000000	500	32000
kernel.sem_next_id = -1
kernel.sg-big-buff = 32768
kernel.shm_next_id = -1
kernel.shm_rmid_forced = 0
kernel.shmall = 18446744073692774399
kernel.shmmax = 18446744073692774399
kernel.shmmni = 4096
kernel.soft_watchdog = 1
kernel.softlockup_all_cpu_backtrace = 0
kernel.softlockup_panic = 0
kernel.stack_tracer_enabled = 0
kernel.sysctl_writes_strict = 1
kernel.sysrq = 176
kernel.tainted = 0
kernel.threads-max = 755124
kernel.timer_migration = 1
kernel.traceoff_on_warning = 0
kernel.tracepoint_printk = 0
kernel.unknown_nmi_panic = 0
kernel.unprivileged_bpf_disabled = 0
kernel.unprivileged_userns_apparmor_policy = 1
kernel.usermodehelper.bset = 4294967295	63
kernel.usermodehelper.inheritable = 4294967295	63
kernel.version = #2 SMP Thu Nov 21 16:52:26 HKT 2019
kernel.watchdog = 1
kernel.watchdog_cpumask = 0-47
kernel.watchdog_thresh = 10
kernel.yama.ptrace_scope = 1
net.core.bpf_jit_enable = 1
net.core.bpf_jit_harden = 0
net.core.bpf_jit_kallsyms = 0
net.core.bpf_jit_limit = 264241152
net.core.busy_poll = 0
net.core.busy_read = 0
net.core.default_qdisc = fq
net.core.dev_weight = 64
net.core.dev_weight_rx_bias = 1
net.core.dev_weight_tx_bias = 1
net.core.devconf_inherit_init_net = 0
net.core.fb_tunnels_only_for_init_net = 0
net.core.flow_limit_cpu_bitmap = 0000,00000000
net.core.flow_limit_table_len = 4096
net.core.gro_normal_batch = 8
net.core.high_order_alloc_disable = 0
net.core.max_skb_frags = 17
net.core.message_burst = 10
net.core.message_cost = 5
net.core.netdev_budget = 300
net.core.netdev_budget_usecs = 2000
net.core.netdev_max_backlog = 1000
net.core.netdev_rss_key = 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
net.core.netdev_tstamp_prequeue = 1
net.core.optmem_max = 20480
net.core.rmem_default = 212992
net.core.rmem_max = 212992
net.core.rps_sock_flow_entries = 0
net.core.somaxconn = 4096
net.core.tstamp_allow_data = 1
net.core.warnings = 0
net.core.wmem_default = 212992
net.core.wmem_max = 212992
net.core.xfrm_acq_expires = 30
net.core.xfrm_aevent_etime = 10
net.core.xfrm_aevent_rseqth = 2
net.core.xfrm_larval_drop = 1
net.ipv4.cipso_cache_bucket_size = 10
net.ipv4.cipso_cache_enable = 1
net.ipv4.cipso_rbm_optfmt = 0
net.ipv4.cipso_rbm_strictvalid = 1
net.ipv4.conf.all.accept_local = 0
net.ipv4.conf.all.accept_redirects = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_notify = 0
net.ipv4.conf.all.bc_forwarding = 0
net.ipv4.conf.all.bootp_relay = 0
net.ipv4.conf.all.disable_policy = 0
net.ipv4.conf.all.disable_xfrm = 0
net.ipv4.conf.all.drop_gratuitous_arp = 0
net.ipv4.conf.all.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.all.force_igmp_version = 0
net.ipv4.conf.all.forwarding = 0
net.ipv4.conf.all.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.all.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.all.ignore_routes_with_linkdown = 0
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.all.medium_id = 0
net.ipv4.conf.all.promote_secondaries = 0
net.ipv4.conf.all.proxy_arp = 0
net.ipv4.conf.all.proxy_arp_pvlan = 0
net.ipv4.conf.all.route_localnet = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.secure_redirects = 1
net.ipv4.conf.all.send_redirects = 1
net.ipv4.conf.all.shared_media = 1
net.ipv4.conf.all.src_valid_mark = 0
net.ipv4.conf.all.tag = 0
net.ipv4.conf.default.accept_local = 0
net.ipv4.conf.default.accept_redirects = 1
net.ipv4.conf.default.accept_source_route = 1
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_notify = 0
net.ipv4.conf.default.bc_forwarding = 0
net.ipv4.conf.default.bootp_relay = 0
net.ipv4.conf.default.disable_policy = 0
net.ipv4.conf.default.disable_xfrm = 0
net.ipv4.conf.default.drop_gratuitous_arp = 0
net.ipv4.conf.default.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.default.force_igmp_version = 0
net.ipv4.conf.default.forwarding = 0
net.ipv4.conf.default.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.default.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.default.ignore_routes_with_linkdown = 0
net.ipv4.conf.default.log_martians = 0
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.default.medium_id = 0
net.ipv4.conf.default.promote_secondaries = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.default.proxy_arp_pvlan = 0
net.ipv4.conf.default.route_localnet = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.secure_redirects = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.default.shared_media = 1
net.ipv4.conf.default.src_valid_mark = 0
net.ipv4.conf.default.tag = 0
net.ipv4.conf.ens5.accept_local = 0
net.ipv4.conf.ens5.accept_redirects = 1
net.ipv4.conf.ens5.accept_source_route = 1
net.ipv4.conf.ens5.arp_accept = 0
net.ipv4.conf.ens5.arp_announce = 0
net.ipv4.conf.ens5.arp_filter = 0
net.ipv4.conf.ens5.arp_ignore = 0
net.ipv4.conf.ens5.arp_notify = 0
net.ipv4.conf.ens5.bc_forwarding = 0
net.ipv4.conf.ens5.bootp_relay = 0
net.ipv4.conf.ens5.disable_policy = 0
net.ipv4.conf.ens5.disable_xfrm = 0
net.ipv4.conf.ens5.drop_gratuitous_arp = 0
net.ipv4.conf.ens5.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.ens5.force_igmp_version = 0
net.ipv4.conf.ens5.forwarding = 0
net.ipv4.conf.ens5.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.ens5.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.ens5.ignore_routes_with_linkdown = 0
net.ipv4.conf.ens5.log_martians = 0
net.ipv4.conf.ens5.mc_forwarding = 0
net.ipv4.conf.ens5.medium_id = 0
net.ipv4.conf.ens5.promote_secondaries = 0
net.ipv4.conf.ens5.proxy_arp = 0
net.ipv4.conf.ens5.proxy_arp_pvlan = 0
net.ipv4.conf.ens5.route_localnet = 0
net.ipv4.conf.ens5.rp_filter = 1
net.ipv4.conf.ens5.secure_redirects = 1
net.ipv4.conf.ens5.send_redirects = 1
net.ipv4.conf.ens5.shared_media = 1
net.ipv4.conf.ens5.src_valid_mark = 0
net.ipv4.conf.ens5.tag = 0
net.ipv4.conf.lo.accept_local = 0
net.ipv4.conf.lo.accept_redirects = 1
net.ipv4.conf.lo.accept_source_route = 1
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_notify = 0
net.ipv4.conf.lo.bc_forwarding = 0
net.ipv4.conf.lo.bootp_relay = 0
net.ipv4.conf.lo.disable_policy = 1
net.ipv4.conf.lo.disable_xfrm = 1
net.ipv4.conf.lo.drop_gratuitous_arp = 0
net.ipv4.conf.lo.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.lo.force_igmp_version = 0
net.ipv4.conf.lo.forwarding = 0
net.ipv4.conf.lo.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.lo.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.lo.ignore_routes_with_linkdown = 0
net.ipv4.conf.lo.log_martians = 0
net.ipv4.conf.lo.mc_forwarding = 0
net.ipv4.conf.lo.medium_id = 0
net.ipv4.conf.lo.promote_secondaries = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.lo.proxy_arp_pvlan = 0
net.ipv4.conf.lo.route_localnet = 0
net.ipv4.conf.lo.rp_filter = 0
net.ipv4.conf.lo.secure_redirects = 1
net.ipv4.conf.lo.send_redirects = 1
net.ipv4.conf.lo.shared_media = 1
net.ipv4.conf.lo.src_valid_mark = 0
net.ipv4.conf.lo.tag = 0
net.ipv4.fib_multipath_hash_policy = 0
net.ipv4.fib_multipath_use_neigh = 0
net.ipv4.fib_sync_mem = 524288
net.ipv4.fwmark_reflect = 0
net.ipv4.icmp_echo_ignore_all = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_errors_use_inbound_ifaddr = 0
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.icmp_msgs_burst = 50
net.ipv4.icmp_msgs_per_sec = 1000
net.ipv4.icmp_ratelimit = 1000
net.ipv4.icmp_ratemask = 6168
net.ipv4.igmp_link_local_mcast_reports = 1
net.ipv4.igmp_max_memberships = 20
net.ipv4.igmp_max_msf = 10
net.ipv4.igmp_qrv = 2
net.ipv4.inet_peer_maxttl = 600
net.ipv4.inet_peer_minttl = 120
net.ipv4.inet_peer_threshold = 65664
net.ipv4.ip_default_ttl = 64
net.ipv4.ip_dynaddr = 0
net.ipv4.ip_early_demux = 1
net.ipv4.ip_forward = 0
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0
net.ipv4.ip_local_port_range = 32768	60999
net.ipv4.ip_local_reserved_ports = 
net.ipv4.ip_no_pmtu_disc = 0
net.ipv4.ip_nonlocal_bind = 0
net.ipv4.ip_unprivileged_port_start = 1024
net.ipv4.ipfrag_high_thresh = 4194304
net.ipv4.ipfrag_low_thresh = 3145728
net.ipv4.ipfrag_max_dist = 64
net.ipv4.ipfrag_secret_interval = 0
net.ipv4.ipfrag_time = 30
net.ipv4.neigh.default.anycast_delay = 100
net.ipv4.neigh.default.app_solicit = 0
net.ipv4.neigh.default.base_reachable_time_ms = 30000
net.ipv4.neigh.default.delay_first_probe_time = 5
net.ipv4.neigh.default.gc_interval = 30
net.ipv4.neigh.default.gc_stale_time = 60
net.ipv4.neigh.default.gc_thresh1 = 128
net.ipv4.neigh.default.gc_thresh2 = 512
net.ipv4.neigh.default.gc_thresh3 = 1024
net.ipv4.neigh.default.locktime = 100
net.ipv4.neigh.default.mcast_resolicit = 0
net.ipv4.neigh.default.mcast_solicit = 3
net.ipv4.neigh.default.proxy_delay = 80
net.ipv4.neigh.default.proxy_qlen = 64
net.ipv4.neigh.default.retrans_time_ms = 1000
net.ipv4.neigh.default.ucast_solicit = 3
net.ipv4.neigh.default.unres_qlen = 101
net.ipv4.neigh.default.unres_qlen_bytes = 212992
net.ipv4.neigh.ens5.anycast_delay = 100
net.ipv4.neigh.ens5.app_solicit = 0
net.ipv4.neigh.ens5.base_reachable_time_ms = 30000
net.ipv4.neigh.ens5.delay_first_probe_time = 5
net.ipv4.neigh.ens5.gc_stale_time = 60
net.ipv4.neigh.ens5.locktime = 100
net.ipv4.neigh.ens5.mcast_resolicit = 0
net.ipv4.neigh.ens5.mcast_solicit = 3
net.ipv4.neigh.ens5.proxy_delay = 80
net.ipv4.neigh.ens5.proxy_qlen = 64
net.ipv4.neigh.ens5.retrans_time_ms = 1000
net.ipv4.neigh.ens5.ucast_solicit = 3
net.ipv4.neigh.ens5.unres_qlen = 101
net.ipv4.neigh.ens5.unres_qlen_bytes = 212992
net.ipv4.neigh.lo.anycast_delay = 100
net.ipv4.neigh.lo.app_solicit = 0
net.ipv4.neigh.lo.base_reachable_time_ms = 30000
net.ipv4.neigh.lo.delay_first_probe_time = 5
net.ipv4.neigh.lo.gc_stale_time = 60
net.ipv4.neigh.lo.locktime = 100
net.ipv4.neigh.lo.mcast_resolicit = 0
net.ipv4.neigh.lo.mcast_solicit = 3
net.ipv4.neigh.lo.proxy_delay = 80
net.ipv4.neigh.lo.proxy_qlen = 64
net.ipv4.neigh.lo.retrans_time_ms = 1000
net.ipv4.neigh.lo.ucast_solicit = 3
net.ipv4.neigh.lo.unres_qlen = 101
net.ipv4.neigh.lo.unres_qlen_bytes = 212992
net.ipv4.ping_group_range = 1	0
net.ipv4.raw_l3mdev_accept = 1
net.ipv4.route.error_burst = 1250
net.ipv4.route.error_cost = 250
net.ipv4.route.gc_elasticity = 8
net.ipv4.route.gc_interval = 60
net.ipv4.route.gc_min_interval = 0
net.ipv4.route.gc_min_interval_ms = 500
net.ipv4.route.gc_thresh = -1
net.ipv4.route.gc_timeout = 300
net.ipv4.route.max_size = 2147483647
net.ipv4.route.min_adv_mss = 256
net.ipv4.route.min_pmtu = 552
net.ipv4.route.mtu_expires = 600
net.ipv4.route.redirect_load = 5
net.ipv4.route.redirect_number = 9
net.ipv4.route.redirect_silence = 5120
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_allowed_congestion_control = reno cubic bbr2
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_autocorking = 1
net.ipv4.tcp_available_congestion_control = reno cubic bbr2
net.ipv4.tcp_available_ulp = 
net.ipv4.tcp_base_mss = 1024
net.ipv4.tcp_challenge_ack_limit = 1000
net.ipv4.tcp_comp_sack_delay_ns = 1000000
net.ipv4.tcp_comp_sack_nr = 44
net.ipv4.tcp_congestion_control = bbr2
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_early_demux = 1
net.ipv4.tcp_early_retrans = 3
net.ipv4.tcp_ecn = 1
net.ipv4.tcp_ecn_fallback = 1
net.ipv4.tcp_fack = 0
net.ipv4.tcp_fastopen = 1
net.ipv4.tcp_fastopen_blackhole_timeout_sec = 3600
net.ipv4.tcp_fastopen_key = 00000000-00000000-00000000-00000000
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_frto = 2
net.ipv4.tcp_fwmark_accept = 0
net.ipv4.tcp_invalid_ratelimit = 500
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_l3mdev_accept = 0
net.ipv4.tcp_limit_output_bytes = 1048576
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_reordering = 300
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_max_tw_buckets = 262144
net.ipv4.tcp_mem = 1129629	1506173	2259258
net.ipv4.tcp_min_rtt_wlen = 300
net.ipv4.tcp_min_snd_mss = 48
net.ipv4.tcp_min_tso_segs = 2
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_mtu_probe_floor = 48
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_notsent_lowat = 4294967295
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_pacing_ca_ratio = 120
net.ipv4.tcp_pacing_ss_ratio = 200
net.ipv4.tcp_probe_interval = 600
net.ipv4.tcp_probe_threshold = 8
net.ipv4.tcp_recovery = 1
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_rmem = 4096	131072	6291456
net.ipv4.tcp_rx_skb_cache = 0
net.ipv4.tcp_sack = 1
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_syn_retries = 6
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_thin_linear_timeouts = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_tw_reuse = 2
net.ipv4.tcp_tx_skb_cache = 0
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_wmem = 4096	16384	4194304
net.ipv4.tcp_workaround_signed_windows = 0
net.ipv4.udp_early_demux = 1
net.ipv4.udp_l3mdev_accept = 0
net.ipv4.udp_mem = 2259258	3012347	4518516
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
net.ipv4.xfrm4_gc_thresh = 32768
net.ipv6.anycast_src_echo_reply = 0
net.ipv6.auto_flowlabels = 1
net.ipv6.bindv6only = 0
net.ipv6.calipso_cache_bucket_size = 10
net.ipv6.calipso_cache_enable = 1
net.ipv6.conf.all.accept_dad = 0
net.ipv6.conf.all.accept_ra = 1
net.ipv6.conf.all.accept_ra_defrtr = 1
net.ipv6.conf.all.accept_ra_from_local = 0
net.ipv6.conf.all.accept_ra_min_hop_limit = 1
net.ipv6.conf.all.accept_ra_mtu = 1
net.ipv6.conf.all.accept_ra_pinfo = 1
net.ipv6.conf.all.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.all.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.all.accept_ra_rtr_pref = 1
net.ipv6.conf.all.accept_redirects = 1
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.all.addr_gen_mode = 0
net.ipv6.conf.all.autoconf = 1
net.ipv6.conf.all.dad_transmits = 1
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.all.disable_policy = 0
net.ipv6.conf.all.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.all.drop_unsolicited_na = 0
net.ipv6.conf.all.enhanced_dad = 1
net.ipv6.conf.all.force_mld_version = 0
net.ipv6.conf.all.force_tllao = 0
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.all.hop_limit = 64
net.ipv6.conf.all.ignore_routes_with_linkdown = 0
net.ipv6.conf.all.keep_addr_on_down = 0
net.ipv6.conf.all.max_addresses = 16
net.ipv6.conf.all.max_desync_factor = 600
net.ipv6.conf.all.mc_forwarding = 0
net.ipv6.conf.all.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.all.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.all.mtu = 1280
net.ipv6.conf.all.ndisc_notify = 0
net.ipv6.conf.all.ndisc_tclass = 0
net.ipv6.conf.all.proxy_ndp = 0
net.ipv6.conf.all.regen_max_retry = 3
net.ipv6.conf.all.router_probe_interval = 60
net.ipv6.conf.all.router_solicitation_delay = 1
net.ipv6.conf.all.router_solicitation_interval = 4
net.ipv6.conf.all.router_solicitation_max_interval = 3600
net.ipv6.conf.all.router_solicitations = -1
net.ipv6.conf.all.seg6_enabled = 0
net.ipv6.conf.all.seg6_require_hmac = 0
net.ipv6.conf.all.suppress_frag_ndisc = 1
net.ipv6.conf.all.temp_prefered_lft = 86400
net.ipv6.conf.all.temp_valid_lft = 604800
net.ipv6.conf.all.use_oif_addrs_only = 0
net.ipv6.conf.all.use_tempaddr = 0
net.ipv6.conf.default.accept_dad = 1
net.ipv6.conf.default.accept_ra = 1
net.ipv6.conf.default.accept_ra_defrtr = 1
net.ipv6.conf.default.accept_ra_from_local = 0
net.ipv6.conf.default.accept_ra_min_hop_limit = 1
net.ipv6.conf.default.accept_ra_mtu = 1
net.ipv6.conf.default.accept_ra_pinfo = 1
net.ipv6.conf.default.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.default.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.default.accept_ra_rtr_pref = 1
net.ipv6.conf.default.accept_redirects = 1
net.ipv6.conf.default.accept_source_route = 0
net.ipv6.conf.default.addr_gen_mode = 0
net.ipv6.conf.default.autoconf = 1
net.ipv6.conf.default.dad_transmits = 1
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.default.disable_policy = 0
net.ipv6.conf.default.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.default.drop_unsolicited_na = 0
net.ipv6.conf.default.enhanced_dad = 1
net.ipv6.conf.default.force_mld_version = 0
net.ipv6.conf.default.force_tllao = 0
net.ipv6.conf.default.forwarding = 0
net.ipv6.conf.default.hop_limit = 64
net.ipv6.conf.default.ignore_routes_with_linkdown = 0
net.ipv6.conf.default.keep_addr_on_down = 0
net.ipv6.conf.default.max_addresses = 16
net.ipv6.conf.default.max_desync_factor = 600
net.ipv6.conf.default.mc_forwarding = 0
net.ipv6.conf.default.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.default.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.default.mtu = 1280
net.ipv6.conf.default.ndisc_notify = 0
net.ipv6.conf.default.ndisc_tclass = 0
net.ipv6.conf.default.proxy_ndp = 0
net.ipv6.conf.default.regen_max_retry = 3
net.ipv6.conf.default.router_probe_interval = 60
net.ipv6.conf.default.router_solicitation_delay = 1
net.ipv6.conf.default.router_solicitation_interval = 4
net.ipv6.conf.default.router_solicitation_max_interval = 3600
net.ipv6.conf.default.router_solicitations = -1
net.ipv6.conf.default.seg6_enabled = 0
net.ipv6.conf.default.seg6_require_hmac = 0
net.ipv6.conf.default.suppress_frag_ndisc = 1
net.ipv6.conf.default.temp_prefered_lft = 86400
net.ipv6.conf.default.temp_valid_lft = 604800
net.ipv6.conf.default.use_oif_addrs_only = 0
net.ipv6.conf.default.use_tempaddr = 0
net.ipv6.conf.ens5.accept_dad = 1
net.ipv6.conf.ens5.accept_ra = 1
net.ipv6.conf.ens5.accept_ra_defrtr = 1
net.ipv6.conf.ens5.accept_ra_from_local = 0
net.ipv6.conf.ens5.accept_ra_min_hop_limit = 1
net.ipv6.conf.ens5.accept_ra_mtu = 1
net.ipv6.conf.ens5.accept_ra_pinfo = 1
net.ipv6.conf.ens5.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.ens5.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.ens5.accept_ra_rtr_pref = 1
net.ipv6.conf.ens5.accept_redirects = 1
net.ipv6.conf.ens5.accept_source_route = 0
net.ipv6.conf.ens5.addr_gen_mode = 0
net.ipv6.conf.ens5.autoconf = 1
net.ipv6.conf.ens5.dad_transmits = 1
net.ipv6.conf.ens5.disable_ipv6 = 0
net.ipv6.conf.ens5.disable_policy = 0
net.ipv6.conf.ens5.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.ens5.drop_unsolicited_na = 0
net.ipv6.conf.ens5.enhanced_dad = 1
net.ipv6.conf.ens5.force_mld_version = 0
net.ipv6.conf.ens5.force_tllao = 0
net.ipv6.conf.ens5.forwarding = 0
net.ipv6.conf.ens5.hop_limit = 64
net.ipv6.conf.ens5.ignore_routes_with_linkdown = 0
net.ipv6.conf.ens5.keep_addr_on_down = 0
net.ipv6.conf.ens5.max_addresses = 16
net.ipv6.conf.ens5.max_desync_factor = 600
net.ipv6.conf.ens5.mc_forwarding = 0
net.ipv6.conf.ens5.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.ens5.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.ens5.mtu = 9001
net.ipv6.conf.ens5.ndisc_notify = 0
net.ipv6.conf.ens5.ndisc_tclass = 0
net.ipv6.conf.ens5.proxy_ndp = 0
net.ipv6.conf.ens5.regen_max_retry = 3
net.ipv6.conf.ens5.router_probe_interval = 60
net.ipv6.conf.ens5.router_solicitation_delay = 1
net.ipv6.conf.ens5.router_solicitation_interval = 4
net.ipv6.conf.ens5.router_solicitation_max_interval = 3600
net.ipv6.conf.ens5.router_solicitations = -1
net.ipv6.conf.ens5.seg6_enabled = 0
net.ipv6.conf.ens5.seg6_require_hmac = 0
net.ipv6.conf.ens5.suppress_frag_ndisc = 1
net.ipv6.conf.ens5.temp_prefered_lft = 86400
net.ipv6.conf.ens5.temp_valid_lft = 604800
net.ipv6.conf.ens5.use_oif_addrs_only = 0
net.ipv6.conf.ens5.use_tempaddr = 0
net.ipv6.conf.lo.accept_dad = -1
net.ipv6.conf.lo.accept_ra = 1
net.ipv6.conf.lo.accept_ra_defrtr = 1
net.ipv6.conf.lo.accept_ra_from_local = 0
net.ipv6.conf.lo.accept_ra_min_hop_limit = 1
net.ipv6.conf.lo.accept_ra_mtu = 1
net.ipv6.conf.lo.accept_ra_pinfo = 1
net.ipv6.conf.lo.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.lo.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.lo.accept_ra_rtr_pref = 1
net.ipv6.conf.lo.accept_redirects = 1
net.ipv6.conf.lo.accept_source_route = 0
net.ipv6.conf.lo.addr_gen_mode = 0
net.ipv6.conf.lo.autoconf = 1
net.ipv6.conf.lo.dad_transmits = 1
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.lo.disable_policy = 0
net.ipv6.conf.lo.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.lo.drop_unsolicited_na = 0
net.ipv6.conf.lo.enhanced_dad = 1
net.ipv6.conf.lo.force_mld_version = 0
net.ipv6.conf.lo.force_tllao = 0
net.ipv6.conf.lo.forwarding = 0
net.ipv6.conf.lo.hop_limit = 64
net.ipv6.conf.lo.ignore_routes_with_linkdown = 0
net.ipv6.conf.lo.keep_addr_on_down = 0
net.ipv6.conf.lo.max_addresses = 16
net.ipv6.conf.lo.max_desync_factor = 600
net.ipv6.conf.lo.mc_forwarding = 0
net.ipv6.conf.lo.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.lo.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.lo.mtu = 65536
net.ipv6.conf.lo.ndisc_notify = 0
net.ipv6.conf.lo.ndisc_tclass = 0
net.ipv6.conf.lo.proxy_ndp = 0
net.ipv6.conf.lo.regen_max_retry = 3
net.ipv6.conf.lo.router_probe_interval = 60
net.ipv6.conf.lo.router_solicitation_delay = 1
net.ipv6.conf.lo.router_solicitation_interval = 4
net.ipv6.conf.lo.router_solicitation_max_interval = 3600
net.ipv6.conf.lo.router_solicitations = -1
net.ipv6.conf.lo.seg6_enabled = 0
net.ipv6.conf.lo.seg6_require_hmac = 0
net.ipv6.conf.lo.suppress_frag_ndisc = 1
net.ipv6.conf.lo.temp_prefered_lft = 86400
net.ipv6.conf.lo.temp_valid_lft = 604800
net.ipv6.conf.lo.use_oif_addrs_only = 0
net.ipv6.conf.lo.use_tempaddr = -1
net.ipv6.fib_multipath_hash_policy = 0
net.ipv6.flowlabel_consistency = 1
net.ipv6.flowlabel_reflect = 0
net.ipv6.flowlabel_state_ranges = 0
net.ipv6.fwmark_reflect = 0
net.ipv6.icmp.echo_ignore_all = 0
net.ipv6.icmp.echo_ignore_anycast = 0
net.ipv6.icmp.echo_ignore_multicast = 0
net.ipv6.icmp.ratelimit = 1000
net.ipv6.icmp.ratemask = 0-1,3-127
net.ipv6.idgen_delay = 1
net.ipv6.idgen_retries = 3
net.ipv6.ip6frag_high_thresh = 4194304
net.ipv6.ip6frag_low_thresh = 3145728
net.ipv6.ip6frag_secret_interval = 0
net.ipv6.ip6frag_time = 60
net.ipv6.ip_nonlocal_bind = 0
net.ipv6.max_dst_opts_length = 2147483647
net.ipv6.max_dst_opts_number = 8
net.ipv6.max_hbh_length = 2147483647
net.ipv6.max_hbh_opts_number = 8
net.ipv6.mld_max_msf = 64
net.ipv6.mld_qrv = 2
net.ipv6.neigh.default.anycast_delay = 100
net.ipv6.neigh.default.app_solicit = 0
net.ipv6.neigh.default.base_reachable_time_ms = 30000
net.ipv6.neigh.default.delay_first_probe_time = 5
net.ipv6.neigh.default.gc_interval = 30
net.ipv6.neigh.default.gc_stale_time = 60
net.ipv6.neigh.default.gc_thresh1 = 128
net.ipv6.neigh.default.gc_thresh2 = 512
net.ipv6.neigh.default.gc_thresh3 = 1024
net.ipv6.neigh.default.locktime = 0
net.ipv6.neigh.default.mcast_resolicit = 0
net.ipv6.neigh.default.mcast_solicit = 3
net.ipv6.neigh.default.proxy_delay = 80
net.ipv6.neigh.default.proxy_qlen = 64
net.ipv6.neigh.default.retrans_time_ms = 1000
net.ipv6.neigh.default.ucast_solicit = 3
net.ipv6.neigh.default.unres_qlen = 101
net.ipv6.neigh.default.unres_qlen_bytes = 212992
net.ipv6.neigh.ens5.anycast_delay = 100
net.ipv6.neigh.ens5.app_solicit = 0
net.ipv6.neigh.ens5.base_reachable_time_ms = 30000
net.ipv6.neigh.ens5.delay_first_probe_time = 5
net.ipv6.neigh.ens5.gc_stale_time = 60
net.ipv6.neigh.ens5.locktime = 0
net.ipv6.neigh.ens5.mcast_resolicit = 0
net.ipv6.neigh.ens5.mcast_solicit = 3
net.ipv6.neigh.ens5.proxy_delay = 80
net.ipv6.neigh.ens5.proxy_qlen = 64
net.ipv6.neigh.ens5.retrans_time_ms = 1000
net.ipv6.neigh.ens5.ucast_solicit = 3
net.ipv6.neigh.ens5.unres_qlen = 101
net.ipv6.neigh.ens5.unres_qlen_bytes = 212992
net.ipv6.neigh.lo.anycast_delay = 100
net.ipv6.neigh.lo.app_solicit = 0
net.ipv6.neigh.lo.base_reachable_time_ms = 30000
net.ipv6.neigh.lo.delay_first_probe_time = 5
net.ipv6.neigh.lo.gc_stale_time = 60
net.ipv6.neigh.lo.locktime = 0
net.ipv6.neigh.lo.mcast_resolicit = 0
net.ipv6.neigh.lo.mcast_solicit = 3
net.ipv6.neigh.lo.proxy_delay = 80
net.ipv6.neigh.lo.proxy_qlen = 64
net.ipv6.neigh.lo.retrans_time_ms = 1000
net.ipv6.neigh.lo.ucast_solicit = 3
net.ipv6.neigh.lo.unres_qlen = 101
net.ipv6.neigh.lo.unres_qlen_bytes = 212992
net.ipv6.route.gc_elasticity = 9
net.ipv6.route.gc_interval = 30
net.ipv6.route.gc_min_interval = 0
net.ipv6.route.gc_min_interval_ms = 500
net.ipv6.route.gc_thresh = 1024
net.ipv6.route.gc_timeout = 60
net.ipv6.route.max_size = 4096
net.ipv6.route.min_adv_mss = 1220
net.ipv6.route.mtu_expires = 600
net.ipv6.route.skip_notify_on_dev_down = 0
net.ipv6.seg6_flowlabel = 0
net.ipv6.xfrm6_gc_thresh = 32768
net.iw_cm.default_backlog = 256
net.netfilter.nf_log.0 = NONE
net.netfilter.nf_log.1 = NONE
net.netfilter.nf_log.10 = NONE
net.netfilter.nf_log.11 = NONE
net.netfilter.nf_log.12 = NONE
net.netfilter.nf_log.2 = NONE
net.netfilter.nf_log.3 = NONE
net.netfilter.nf_log.4 = NONE
net.netfilter.nf_log.5 = NONE
net.netfilter.nf_log.6 = NONE
net.netfilter.nf_log.7 = NONE
net.netfilter.nf_log.8 = NONE
net.netfilter.nf_log.9 = NONE
net.netfilter.nf_log_all_netns = 0
net.unix.max_dgram_qlen = 512
user.max_cgroup_namespaces = 377562
user.max_inotify_instances = 1024
user.max_inotify_watches = 8192
user.max_ipc_namespaces = 377562
user.max_mnt_namespaces = 377562
user.max_net_namespaces = 377562
user.max_pid_namespaces = 377562
user.max_user_namespaces = 377562
user.max_uts_namespaces = 377562
vm.admin_reserve_kbytes = 8192
vm.block_dump = 0
vm.compact_unevictable_allowed = 1
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
vm.drop_caches = 0
vm.extfrag_threshold = 500
vm.hugetlb_shm_group = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 256	256	32	0	0
vm.max_map_count = 65530
vm.memory_failure_early_kill = 0
vm.memory_failure_recovery = 1
vm.min_free_kbytes = 67584
vm.min_slab_ratio = 5
vm.min_unmapped_ratio = 1
vm.mmap_min_addr = 65536
vm.mmap_rnd_bits = 28
vm.mmap_rnd_compat_bits = 8
vm.nr_hugepages = 0
vm.nr_hugepages_mempolicy = 0
vm.nr_overcommit_hugepages = 0
vm.numa_stat = 1
vm.numa_zonelist_order = Node
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_kbytes = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.stat_interval = 1
vm.swappiness = 60
vm.unprivileged_userfaultfd = 1
vm.user_reserve_kbytes = 131072
vm.vfs_cache_pressure = 100
vm.watermark_boost_factor = 15000
vm.watermark_scale_factor = 10
vm.zone_reclaim_mode = 0

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections
  2020-12-07 17:16               ` Mohamed Abuelfotoh, Hazem
@ 2020-12-07 17:27                 ` Eric Dumazet
  2020-12-08 16:28                   ` Mohamed Abuelfotoh, Hazem
  0 siblings, 1 reply; 22+ messages in thread
From: Eric Dumazet @ 2020-12-07 17:27 UTC (permalink / raw)
  To: Mohamed Abuelfotoh, Hazem
  Cc: Neal Cardwell, netdev, stable, ycheng, weiwan, Strohman, Andy,
	Herrenschmidt, Benjamin

On Mon, Dec 7, 2020 at 6:17 PM Mohamed Abuelfotoh, Hazem
<abuehaze@amazon.com> wrote:
>
>     >Thanks for testing this, Eric. Would you be able to share the MTU
>     >config commands you used, and the tcpdump traces you get? I'm
>     >surprised that receive buffer autotuning would work for advmss of
>     >around 6500 or higher.
>
> Packet capture before applying the proposed patch
>
> https://tcpautotuningpcaps.s3.eu-west-1.amazonaws.com/sender-bbr-bad-unpatched.pcap?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJNMP5ZZ3I4FAQGAQ%2F20201207%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20201207T170123Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=a599a0e0e6632a957e5619007ba5ce4f63c8e8535ea24470b7093fef440a8300
>
> Packet capture after applying the proposed patch
>
> https://tcpautotuningpcaps.s3.eu-west-1.amazonaws.com/sender-bbr-good-patched.pcap?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJNMP5ZZ3I4FAQGAQ%2F20201207%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20201207T165831Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=f18ec7246107590e8ac35c24322af699e4c2a73d174067c51cf6b0a06bbbca77
>
> kernel version & MTU and configuration  from my receiver & sender is attached to this e-mail, please be aware that EC2 is doing MSS clamping so you need to configure MTU as 1500 on the sender side if you don’t have any MSS clamping between sender & receiver.
>
> Thank you.
>
> Hazem

Please try again, with a fixed tcp_rmem[1] on receiver, taking into
account bigger memory requirement for MTU 9000

Rationale : TCP should be ready to receive 10 full frames before
autotuning takes place (these 10 MSS are typically in a single GRO
packet)

At 9000 MTU, one frame typically consumes 12KB (or 16KB on some arches/drivers)

TCP uses a 50% factor rule, accounting 18000 bytes of kernel memory per MSS.

->

echo "4096 180000 15728640" >/proc/sys/net/ipv4/tcp_rmem



>
>
> On 07/12/2020, 16:34, "Neal Cardwell" <ncardwell@google.com> wrote:
>
>     CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
>     On Mon, Dec 7, 2020 at 11:23 AM Eric Dumazet <edumazet@google.com> wrote:
>     >
>     > On Mon, Dec 7, 2020 at 5:09 PM Mohamed Abuelfotoh, Hazem
>     > <abuehaze@amazon.com> wrote:
>     > >
>     > >     >Since I can not reproduce this problem with another NIC on x86, I
>     > >     >really wonder if this is not an issue with ENA driver on PowerPC
>     > >     >perhaps ?
>     > >
>     > >
>     > > I am able to reproduce it on x86 based EC2 instances using ENA  or  Xen netfront or Intel ixgbevf driver on the receiver so it's not specific to ENA, we were able to easily reproduce it between 2 VMs running in virtual box on the same physical host considering the environment requirements I mentioned in my first e-mail.
>     > >
>     > > What's the RTT between the sender & receiver in your reproduction? Are you using bbr on the sender side?
>     >
>     >
>     > 100ms RTT
>     >
>     > Which exact version of linux kernel are you using ?
>
>     Thanks for testing this, Eric. Would you be able to share the MTU
>     config commands you used, and the tcpdump traces you get? I'm
>     surprised that receive buffer autotuning would work for advmss of
>     around 6500 or higher.
>
>     thanks,
>     neal
>
>
>
>
> Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284
>
> Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705
>
>

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections
  2020-12-07 17:27                 ` Eric Dumazet
@ 2020-12-08 16:28                   ` Mohamed Abuelfotoh, Hazem
  2020-12-08 16:30                     ` Mohamed Abuelfotoh, Hazem
  2020-12-08 16:46                     ` Eric Dumazet
  0 siblings, 2 replies; 22+ messages in thread
From: Mohamed Abuelfotoh, Hazem @ 2020-12-08 16:28 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: Neal Cardwell, netdev, stable, ycheng, weiwan, Strohman, Andy,
	Herrenschmidt, Benjamin

    >Please try again, with a fixed tcp_rmem[1] on receiver, taking into
    >account bigger memory requirement for MTU 9000

    >Rationale : TCP should be ready to receive 10 full frames before
    >autotuning takes place (these 10 MSS are typically in a single GRO
   > packet)

    >At 9000 MTU, one frame typically consumes 12KB (or 16KB on some arches/drivers)

   >TCP uses a 50% factor rule, accounting 18000 bytes of kernel memory per MSS.

    ->

    >echo "4096 180000 15728640" >/proc/sys/net/ipv4/tcp_rmem



>diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
>index 9e8a6c1aa0190cc248b3b99b073a4c6e45884cf5..81b5d9375860ae583e08045fb25b089c456c60ab
>100644
>--- a/net/ipv4/tcp_input.c
>+++ b/net/ipv4/tcp_input.c
>@@ -534,6 +534,7 @@ static void tcp_init_buffer_space(struct sock *sk)
>
>        tp->rcv_ssthresh = min(tp->rcv_ssthresh, tp->window_clamp);
>       tp->snd_cwnd_stamp = tcp_jiffies32;
>+       tp->rcvq_space.space = min(tp->rcv_ssthresh, tp->rcvq_space.space);
>}

Yes this worked and it looks like echo "4096 140000 15728640" >/proc/sys/net/ipv4/tcp_rmem is actually enough to trigger TCP autotuning, if the current default tcp_rmem[1] doesn't work well with 9000 MTU I am curious to know  if there is specific reason behind having 131072 specifically   as  tcp_rmem[1]?I think the number itself has to be divisible by page size (4K) and 16KB given what you said that each Jumbo frame packet may consume up to 16KB.

if the patch I proposed would be risky for users who have MTU of 1500 because of its higher memory footprint in my opinion we should  get the patch you proposed merged instead of asking the Admins doing the manual work.

Thank you.

Hazem

On 07/12/2020, 17:28, "Eric Dumazet" <edumazet@google.com> wrote:

    CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



    On Mon, Dec 7, 2020 at 6:17 PM Mohamed Abuelfotoh, Hazem
    <abuehaze@amazon.com> wrote:
    >
    >     >Thanks for testing this, Eric. Would you be able to share the MTU
    >     >config commands you used, and the tcpdump traces you get? I'm
    >     >surprised that receive buffer autotuning would work for advmss of
    >     >around 6500 or higher.
    >
    > Packet capture before applying the proposed patch
    >
    > https://tcpautotuningpcaps.s3.eu-west-1.amazonaws.com/sender-bbr-bad-unpatched.pcap?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJNMP5ZZ3I4FAQGAQ%2F20201207%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20201207T170123Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=a599a0e0e6632a957e5619007ba5ce4f63c8e8535ea24470b7093fef440a8300
    >
    > Packet capture after applying the proposed patch
    >
    > https://tcpautotuningpcaps.s3.eu-west-1.amazonaws.com/sender-bbr-good-patched.pcap?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJNMP5ZZ3I4FAQGAQ%2F20201207%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20201207T165831Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=f18ec7246107590e8ac35c24322af699e4c2a73d174067c51cf6b0a06bbbca77
    >
    > kernel version & MTU and configuration  from my receiver & sender is attached to this e-mail, please be aware that EC2 is doing MSS clamping so you need to configure MTU as 1500 on the sender side if you don’t have any MSS clamping between sender & receiver.
    >
    > Thank you.
    >
    > Hazem

    Please try again, with a fixed tcp_rmem[1] on receiver, taking into
    account bigger memory requirement for MTU 9000

    Rationale : TCP should be ready to receive 10 full frames before
    autotuning takes place (these 10 MSS are typically in a single GRO
    packet)

    At 9000 MTU, one frame typically consumes 12KB (or 16KB on some arches/drivers)

    TCP uses a 50% factor rule, accounting 18000 bytes of kernel memory per MSS.

    ->

    echo "4096 180000 15728640" >/proc/sys/net/ipv4/tcp_rmem



    >
    >
    > On 07/12/2020, 16:34, "Neal Cardwell" <ncardwell@google.com> wrote:
    >
    >     CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
    >
    >
    >
    >     On Mon, Dec 7, 2020 at 11:23 AM Eric Dumazet <edumazet@google.com> wrote:
    >     >
    >     > On Mon, Dec 7, 2020 at 5:09 PM Mohamed Abuelfotoh, Hazem
    >     > <abuehaze@amazon.com> wrote:
    >     > >
    >     > >     >Since I can not reproduce this problem with another NIC on x86, I
    >     > >     >really wonder if this is not an issue with ENA driver on PowerPC
    >     > >     >perhaps ?
    >     > >
    >     > >
    >     > > I am able to reproduce it on x86 based EC2 instances using ENA  or  Xen netfront or Intel ixgbevf driver on the receiver so it's not specific to ENA, we were able to easily reproduce it between 2 VMs running in virtual box on the same physical host considering the environment requirements I mentioned in my first e-mail.
    >     > >
    >     > > What's the RTT between the sender & receiver in your reproduction? Are you using bbr on the sender side?
    >     >
    >     >
    >     > 100ms RTT
    >     >
    >     > Which exact version of linux kernel are you using ?
    >
    >     Thanks for testing this, Eric. Would you be able to share the MTU
    >     config commands you used, and the tcpdump traces you get? I'm
    >     surprised that receive buffer autotuning would work for advmss of
    >     around 6500 or higher.
    >
    >     thanks,
    >     neal
    >
    >
    >
    >
    > Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284
    >
    > Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705
    >
    >




Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284

Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705



^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections
  2020-12-08 16:28                   ` Mohamed Abuelfotoh, Hazem
@ 2020-12-08 16:30                     ` Mohamed Abuelfotoh, Hazem
  2020-12-08 16:46                     ` Eric Dumazet
  1 sibling, 0 replies; 22+ messages in thread
From: Mohamed Abuelfotoh, Hazem @ 2020-12-08 16:30 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: Neal Cardwell, netdev, stable, ycheng, weiwan, Strohman, Andy,
	Herrenschmidt, Benjamin

Feel free to ignore this message  as I sent it before seeing  your newly submitted patch (

Thank you.

Hazem



On 08/12/2020, 16:28, "Mohamed Abuelfotoh, Hazem" <abuehaze@amazon.com> wrote:

        >Please try again, with a fixed tcp_rmem[1] on receiver, taking into
        >account bigger memory requirement for MTU 9000

        >Rationale : TCP should be ready to receive 10 full frames before
        >autotuning takes place (these 10 MSS are typically in a single GRO
       > packet)

        >At 9000 MTU, one frame typically consumes 12KB (or 16KB on some arches/drivers)

       >TCP uses a 50% factor rule, accounting 18000 bytes of kernel memory per MSS.

        ->

        >echo "4096 180000 15728640" >/proc/sys/net/ipv4/tcp_rmem



    >diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
    >index 9e8a6c1aa0190cc248b3b99b073a4c6e45884cf5..81b5d9375860ae583e08045fb25b089c456c60ab
    >100644
    >--- a/net/ipv4/tcp_input.c
    >+++ b/net/ipv4/tcp_input.c
    >@@ -534,6 +534,7 @@ static void tcp_init_buffer_space(struct sock *sk)
    >
    >        tp->rcv_ssthresh = min(tp->rcv_ssthresh, tp->window_clamp);
    >       tp->snd_cwnd_stamp = tcp_jiffies32;
    >+       tp->rcvq_space.space = min(tp->rcv_ssthresh, tp->rcvq_space.space);
    >}

    Yes this worked and it looks like echo "4096 140000 15728640" >/proc/sys/net/ipv4/tcp_rmem is actually enough to trigger TCP autotuning, if the current default tcp_rmem[1] doesn't work well with 9000 MTU I am curious to know  if there is specific reason behind having 131072 specifically   as  tcp_rmem[1]?I think the number itself has to be divisible by page size (4K) and 16KB given what you said that each Jumbo frame packet may consume up to 16KB.

    if the patch I proposed would be risky for users who have MTU of 1500 because of its higher memory footprint in my opinion we should  get the patch you proposed merged instead of asking the Admins doing the manual work.

    Thank you.

    Hazem

    On 07/12/2020, 17:28, "Eric Dumazet" <edumazet@google.com> wrote:

        CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



        On Mon, Dec 7, 2020 at 6:17 PM Mohamed Abuelfotoh, Hazem
        <abuehaze@amazon.com> wrote:
        >
        >     >Thanks for testing this, Eric. Would you be able to share the MTU
        >     >config commands you used, and the tcpdump traces you get? I'm
        >     >surprised that receive buffer autotuning would work for advmss of
        >     >around 6500 or higher.
        >
        > Packet capture before applying the proposed patch
        >
        > https://tcpautotuningpcaps.s3.eu-west-1.amazonaws.com/sender-bbr-bad-unpatched.pcap?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJNMP5ZZ3I4FAQGAQ%2F20201207%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20201207T170123Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=a599a0e0e6632a957e5619007ba5ce4f63c8e8535ea24470b7093fef440a8300
        >
        > Packet capture after applying the proposed patch
        >
        > https://tcpautotuningpcaps.s3.eu-west-1.amazonaws.com/sender-bbr-good-patched.pcap?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJNMP5ZZ3I4FAQGAQ%2F20201207%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20201207T165831Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=f18ec7246107590e8ac35c24322af699e4c2a73d174067c51cf6b0a06bbbca77
        >
        > kernel version & MTU and configuration  from my receiver & sender is attached to this e-mail, please be aware that EC2 is doing MSS clamping so you need to configure MTU as 1500 on the sender side if you don’t have any MSS clamping between sender & receiver.
        >
        > Thank you.
        >
        > Hazem

        Please try again, with a fixed tcp_rmem[1] on receiver, taking into
        account bigger memory requirement for MTU 9000

        Rationale : TCP should be ready to receive 10 full frames before
        autotuning takes place (these 10 MSS are typically in a single GRO
        packet)

        At 9000 MTU, one frame typically consumes 12KB (or 16KB on some arches/drivers)

        TCP uses a 50% factor rule, accounting 18000 bytes of kernel memory per MSS.

        ->

        echo "4096 180000 15728640" >/proc/sys/net/ipv4/tcp_rmem



        >
        >
        > On 07/12/2020, 16:34, "Neal Cardwell" <ncardwell@google.com> wrote:
        >
        >     CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
        >
        >
        >
        >     On Mon, Dec 7, 2020 at 11:23 AM Eric Dumazet <edumazet@google.com> wrote:
        >     >
        >     > On Mon, Dec 7, 2020 at 5:09 PM Mohamed Abuelfotoh, Hazem
        >     > <abuehaze@amazon.com> wrote:
        >     > >
        >     > >     >Since I can not reproduce this problem with another NIC on x86, I
        >     > >     >really wonder if this is not an issue with ENA driver on PowerPC
        >     > >     >perhaps ?
        >     > >
        >     > >
        >     > > I am able to reproduce it on x86 based EC2 instances using ENA  or  Xen netfront or Intel ixgbevf driver on the receiver so it's not specific to ENA, we were able to easily reproduce it between 2 VMs running in virtual box on the same physical host considering the environment requirements I mentioned in my first e-mail.
        >     > >
        >     > > What's the RTT between the sender & receiver in your reproduction? Are you using bbr on the sender side?
        >     >
        >     >
        >     > 100ms RTT
        >     >
        >     > Which exact version of linux kernel are you using ?
        >
        >     Thanks for testing this, Eric. Would you be able to share the MTU
        >     config commands you used, and the tcpdump traces you get? I'm
        >     surprised that receive buffer autotuning would work for advmss of
        >     around 6500 or higher.
        >
        >     thanks,
        >     neal
        >
        >
        >
        >
        > Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284
        >
        > Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705
        >
        >





Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284

Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705



^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections
  2020-12-08 16:28                   ` Mohamed Abuelfotoh, Hazem
  2020-12-08 16:30                     ` Mohamed Abuelfotoh, Hazem
@ 2020-12-08 16:46                     ` Eric Dumazet
  1 sibling, 0 replies; 22+ messages in thread
From: Eric Dumazet @ 2020-12-08 16:46 UTC (permalink / raw)
  To: Mohamed Abuelfotoh, Hazem
  Cc: Neal Cardwell, netdev, stable, ycheng, weiwan, Strohman, Andy,
	Herrenschmidt, Benjamin

On Tue, Dec 8, 2020 at 5:28 PM Mohamed Abuelfotoh, Hazem
<abuehaze@amazon.com> wrote:
>
>     >Please try again, with a fixed tcp_rmem[1] on receiver, taking into
>     >account bigger memory requirement for MTU 9000
>
>     >Rationale : TCP should be ready to receive 10 full frames before
>     >autotuning takes place (these 10 MSS are typically in a single GRO
>    > packet)
>
>     >At 9000 MTU, one frame typically consumes 12KB (or 16KB on some arches/drivers)
>
>    >TCP uses a 50% factor rule, accounting 18000 bytes of kernel memory per MSS.
>
>     ->
>
>     >echo "4096 180000 15728640" >/proc/sys/net/ipv4/tcp_rmem
>
>
>
> >diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
> >index 9e8a6c1aa0190cc248b3b99b073a4c6e45884cf5..81b5d9375860ae583e08045fb25b089c456c60ab
> >100644
> >--- a/net/ipv4/tcp_input.c
> >+++ b/net/ipv4/tcp_input.c
> >@@ -534,6 +534,7 @@ static void tcp_init_buffer_space(struct sock *sk)
> >
> >        tp->rcv_ssthresh = min(tp->rcv_ssthresh, tp->window_clamp);
> >       tp->snd_cwnd_stamp = tcp_jiffies32;
> >+       tp->rcvq_space.space = min(tp->rcv_ssthresh, tp->rcvq_space.space);
> >}
>
> Yes this worked and it looks like echo "4096 140000 15728640" >/proc/sys/net/ipv4/tcp_rmem is actually enough to trigger TCP autotuning, if the current default tcp_rmem[1] doesn't work well with 9000 MTU I am curious to know  if there is specific reason behind having 131072 specifically   as  tcp_rmem[1]?I think the number itself has to be divisible by page size (4K) and 16KB given what you said that each Jumbo frame packet may consume up to 16KB.


I think the idea behind the value of 131072 was that because TCP RWIN
was set to 65535, we had to reserve twice this amount of memory ->
131072 bytes.

Assuming DRS works well, the exact value should matter only for
unresponsive applications (slow to read/drain the receive queue),
since DRS is delayed for them.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections
  2020-12-07 16:22           ` Eric Dumazet
  2020-12-07 16:33             ` Neal Cardwell
@ 2020-12-07 16:34             ` Mohamed Abuelfotoh, Hazem
  2020-12-07 17:46               ` Greg KH
  1 sibling, 1 reply; 22+ messages in thread
From: Mohamed Abuelfotoh, Hazem @ 2020-12-07 16:34 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: netdev, stable, ycheng, ncardwell, weiwan, Strohman, Andy,
	Herrenschmidt, Benjamin

100ms RTT

>Which exact version of linux kernel are you using ?
On the receiver side I could see the issue with any mainline kernel version >=4.19.86 which is the first kernel version that has patches [1] & [2] included.
On the sender I am using kernel 5.4.0-rc6.

Links:

[1] https://lore.kernel.org/patchwork/patch/1157936/
[2] https://lore.kernel.org/patchwork/patch/1157883/

Thank you.

Hazem



On 07/12/2020, 16:24, "Eric Dumazet" <edumazet@google.com> wrote:

    CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



    On Mon, Dec 7, 2020 at 5:09 PM Mohamed Abuelfotoh, Hazem
    <abuehaze@amazon.com> wrote:
    >
    >     >Since I can not reproduce this problem with another NIC on x86, I
    >     >really wonder if this is not an issue with ENA driver on PowerPC
    >     >perhaps ?
    >
    >
    > I am able to reproduce it on x86 based EC2 instances using ENA  or  Xen netfront or Intel ixgbevf driver on the receiver so it's not specific to ENA, we were able to easily reproduce it between 2 VMs running in virtual box on the same physical host considering the environment requirements I mentioned in my first e-mail.
    >
    > What's the RTT between the sender & receiver in your reproduction? Are you using bbr on the sender side?


    100ms RTT

    Which exact version of linux kernel are you using ?



    >
    > Thank you.
    >
    > Hazem
    >
    > On 07/12/2020, 15:26, "Eric Dumazet" <edumazet@google.com> wrote:
    >
    >     CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
    >
    >
    >
    >     On Sat, Dec 5, 2020 at 1:03 PM Mohamed Abuelfotoh, Hazem
    >     <abuehaze@amazon.com> wrote:
    >     >
    >     > Unfortunately few things are missing in this report.
    >     >
    >     >     What is the RTT between hosts in your test ?
    >     >      >>>>>RTT in my test is 162 msec, but I am able to reproduce it with lower RTTs for example I could see the issue downloading from google   endpoint with RTT of 16.7 msec, as mentioned in my previous e-mail the issue is reproducible whenever RTT exceeded 12msec given that    the sender is using bbr.
    >     >
    >     >         RTT between hosts where I run the iperf test.
    >     >         # ping 54.199.163.187
    >     >         PING 54.199.163.187 (54.199.163.187) 56(84) bytes of data.
    >     >         64 bytes from 54.199.163.187: icmp_seq=1 ttl=33 time=162 ms
    >     >         64 bytes from 54.199.163.187: icmp_seq=2 ttl=33 time=162 ms
    >     >         64 bytes from 54.199.163.187: icmp_seq=3 ttl=33 time=162 ms
    >     >         64 bytes from 54.199.163.187: icmp_seq=4 ttl=33 time=162 ms
    >     >
    >     >         RTT between my EC2 instances and google endpoint.
    >     >         # ping 172.217.4.240
    >     >         PING 172.217.4.240 (172.217.4.240) 56(84) bytes of data.
    >     >         64 bytes from 172.217.4.240: icmp_seq=1 ttl=101 time=16.7 ms
    >     >         64 bytes from 172.217.4.240: icmp_seq=2 ttl=101 time=16.7 ms
    >     >         64 bytes from 172.217.4.240: icmp_seq=3 ttl=101 time=16.7 ms
    >     >         64 bytes from 172.217.4.240: icmp_seq=4 ttl=101 time=16.7 ms
    >     >
    >     >     What driver is used at the receiving side ?
    >     >       >>>>>>I am using ENA driver version version: 2.2.10g on the receiver with scatter gathering enabled.
    >     >
    >     >         # ethtool -k eth0 | grep scatter-gather
    >     >         scatter-gather: on
    >     >                 tx-scatter-gather: on
    >     >                 tx-scatter-gather-fraglist: off [fixed]
    >
    >     This ethtool output refers to TX scatter gather, which is not relevant
    >     for this bug.
    >
    >     I see ENA driver might use 16 KB per incoming packet (if ENA_PAGE_SIZE is 16 KB)
    >
    >     Since I can not reproduce this problem with another NIC on x86, I
    >     really wonder if this is not an issue with ENA driver on PowerPC
    >     perhaps ?
    >
    >
    >
    >
    > Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284
    >
    > Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705
    >
    >




Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284

Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705



^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections
  2020-12-07 16:34             ` Mohamed Abuelfotoh, Hazem
@ 2020-12-07 17:46               ` Greg KH
  2020-12-07 17:54                 ` Mohamed Abuelfotoh, Hazem
  0 siblings, 1 reply; 22+ messages in thread
From: Greg KH @ 2020-12-07 17:46 UTC (permalink / raw)
  To: Mohamed Abuelfotoh, Hazem
  Cc: Eric Dumazet, netdev, stable, ycheng, ncardwell, weiwan,
	Strohman, Andy, Herrenschmidt, Benjamin

On Mon, Dec 07, 2020 at 04:34:57PM +0000, Mohamed Abuelfotoh, Hazem wrote:
> 100ms RTT
> 
> >Which exact version of linux kernel are you using ?
> On the receiver side I could see the issue with any mainline kernel
> version >=4.19.86 which is the first kernel version that has patches
> [1] & [2] included.  On the sender I am using kernel 5.4.0-rc6.

5.4.0-rc6 is a very old and odd kernel to be doing anything with.  Are
you sure you don't mean "5.10-rc6" here?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections
  2020-12-07 17:46               ` Greg KH
@ 2020-12-07 17:54                 ` Mohamed Abuelfotoh, Hazem
  0 siblings, 0 replies; 22+ messages in thread
From: Mohamed Abuelfotoh, Hazem @ 2020-12-07 17:54 UTC (permalink / raw)
  To: Greg KH
  Cc: Eric Dumazet, netdev, stable, ycheng, ncardwell, weiwan,
	Strohman, Andy, Herrenschmidt, Benjamin

>5.4.0-rc6 is a very old and odd kernel to be doing anything with.  Are
>you sure you don't mean "5.10-rc6" here?

I was able to reproduce it on the latest mainline kernel as well  so anything newer than 4.19.85 is just broken.

Thank you.

Hazem

On 07/12/2020, 17:45, "Greg KH" <gregkh@linuxfoundation.org> wrote:

    CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



    On Mon, Dec 07, 2020 at 04:34:57PM +0000, Mohamed Abuelfotoh, Hazem wrote:
    > 100ms RTT
    >
    > >Which exact version of linux kernel are you using ?
    > On the receiver side I could see the issue with any mainline kernel
    > version >=4.19.86 which is the first kernel version that has patches
    > [1] & [2] included.  On the sender I am using kernel 5.4.0-rc6.

    5.4.0-rc6 is a very old and odd kernel to be doing anything with.  Are
    you sure you don't mean "5.10-rc6" here?

    thanks,

    greg k-h




Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284

Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705



^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections
  2020-12-04 18:06 [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections Hazem Mohamed Abuelfotoh
  2020-12-04 18:19 ` Mohamed Abuelfotoh, Hazem
@ 2020-12-04 19:10 ` Eric Dumazet
  2020-12-04 21:28 ` Neal Cardwell
  2 siblings, 0 replies; 22+ messages in thread
From: Eric Dumazet @ 2020-12-04 19:10 UTC (permalink / raw)
  To: Hazem Mohamed Abuelfotoh
  Cc: netdev, stable, Yuchung Cheng, Neal Cardwell, Wei Wang, Strohman,
	Andy, Benjamin Herrenschmidt

On Fri, Dec 4, 2020 at 7:08 PM Hazem Mohamed Abuelfotoh
<abuehaze@amazon.com> wrote:
>
>     Previously receiver buffer auto-tuning starts after receiving
>     one advertised window amount of data.After the initial
>     receiver buffer was raised by
>     commit a337531b942b ("tcp: up initial rmem to 128KB
>     and SYN rwin to around 64KB"),the receiver buffer may
>     take too long for TCP autotuning to start raising
>     the receiver buffer size.
>     commit 041a14d26715 ("tcp: start receiver buffer autotuning sooner")
>     tried to decrease the threshold at which TCP auto-tuning starts
>     but it's doesn't work well in some environments
>     where the receiver has large MTU (9001) configured
>     specially within environments where RTT is high.
>     To address this issue this patch is relying on RCV_MSS
>     so auto-tuning can start early regardless
>     the receiver configured MTU.
>
>     Fixes: a337531b942b ("tcp: up initial rmem to 128KB and SYN rwin to around 64KB")
>     Fixes: 041a14d26715 ("tcp: start receiver buffer autotuning sooner")
>
> Signed-off-by: Hazem Mohamed Abuelfotoh <abuehaze@amazon.com>
> ---
>  net/ipv4/tcp_input.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
> index 389d1b340248..f0ffac9e937b 100644
> --- a/net/ipv4/tcp_input.c
> +++ b/net/ipv4/tcp_input.c
> @@ -504,13 +504,14 @@ static void tcp_grow_window(struct sock *sk, const struct sk_buff *skb)
>  static void tcp_init_buffer_space(struct sock *sk)
>  {
>         int tcp_app_win = sock_net(sk)->ipv4.sysctl_tcp_app_win;
> +       struct inet_connection_sock *icsk = inet_csk(sk);
>         struct tcp_sock *tp = tcp_sk(sk);
>         int maxwin;
>
>         if (!(sk->sk_userlocks & SOCK_SNDBUF_LOCK))
>                 tcp_sndbuf_expand(sk);
>
> -       tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND * tp->advmss);
> +       tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND * icsk->icsk_ack.rcv_mss);

So are you claiming icsk->icsk_ack.rcv_mss is related to MTU 9000 ?

RCV_MSS is not known until we receive actual packets... The initial
value is somthing like 536 if I am not mistaken.

I think your patch does not match the changelog.

>         tcp_mstamp_refresh(tp);
>         tp->rcvq_space.time = tp->tcp_mstamp;
>         tp->rcvq_space.seq = tp->copied_seq;
> --
> 2.16.6
>
>
>
>
> Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284
>
> Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705
>
>
>

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections
  2020-12-04 18:06 [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections Hazem Mohamed Abuelfotoh
  2020-12-04 18:19 ` Mohamed Abuelfotoh, Hazem
  2020-12-04 19:10 ` Eric Dumazet
@ 2020-12-04 21:28 ` Neal Cardwell
  2020-12-07 11:46   ` [PATCH net] tcp: fix receive buffer autotuning to trigger for any valid advertised MSS Hazem Mohamed Abuelfotoh
  2 siblings, 1 reply; 22+ messages in thread
From: Neal Cardwell @ 2020-12-04 21:28 UTC (permalink / raw)
  To: Hazem Mohamed Abuelfotoh
  Cc: Netdev, stable, Eric Dumazet, Yuchung Cheng, Wei Wang, astroh, benh

On Fri, Dec 4, 2020 at 1:08 PM Hazem Mohamed Abuelfotoh
<abuehaze@amazon.com> wrote:
>
>     Previously receiver buffer auto-tuning starts after receiving
>     one advertised window amount of data.After the initial
>     receiver buffer was raised by
>     commit a337531b942b ("tcp: up initial rmem to 128KB
>     and SYN rwin to around 64KB"),the receiver buffer may
>     take too long for TCP autotuning to start raising
>     the receiver buffer size.
>     commit 041a14d26715 ("tcp: start receiver buffer autotuning sooner")
>     tried to decrease the threshold at which TCP auto-tuning starts
>     but it's doesn't work well in some environments
>     where the receiver has large MTU (9001) configured
>     specially within environments where RTT is high.
>     To address this issue this patch is relying on RCV_MSS
>     so auto-tuning can start early regardless
>     the receiver configured MTU.
>
>     Fixes: a337531b942b ("tcp: up initial rmem to 128KB and SYN rwin to around 64KB")
>     Fixes: 041a14d26715 ("tcp: start receiver buffer autotuning sooner")
>
> Signed-off-by: Hazem Mohamed Abuelfotoh <abuehaze@amazon.com>
> ---
>  net/ipv4/tcp_input.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
> index 389d1b340248..f0ffac9e937b 100644
> --- a/net/ipv4/tcp_input.c
> +++ b/net/ipv4/tcp_input.c
> @@ -504,13 +504,14 @@ static void tcp_grow_window(struct sock *sk, const struct sk_buff *skb)
>  static void tcp_init_buffer_space(struct sock *sk)
>  {
>         int tcp_app_win = sock_net(sk)->ipv4.sysctl_tcp_app_win;
> +       struct inet_connection_sock *icsk = inet_csk(sk);
>         struct tcp_sock *tp = tcp_sk(sk);
>         int maxwin;
>
>         if (!(sk->sk_userlocks & SOCK_SNDBUF_LOCK))
>                 tcp_sndbuf_expand(sk);
>
> -       tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND * tp->advmss);
> +       tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND * icsk->icsk_ack.rcv_mss);

Thanks for the detailed report and the proposed fix.

AFAICT the core of the bug is related to this part of your follow-up email:

> 10)It looks like when the issue happen we have a  kind of deadlock here
> so advertised receive window has to exceed rcvq_space for the tcp auto
> tuning to kickoff at the same time with the initial default  configuration the
> receive window is not going to exceed rcvq_space

The existing code is:

> -       tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND * tp->advmss);

(1) With a typical case where both sides have an advmss based on an
MTU of 1500 bytes, the governing limit here is around 10*1460 or so,
or around 15Kbytes. With a tp->rcvq_space.space of that size, it is
common for the data sender to send that amount of data in one round
trip, which will trigger the receive buffer autotuning code in
tcp_rcv_space_adjust(), so autotuning works well.

(2) With a case where the sender has a 1500B NIC and the receiver has
an advmss based on an MTU of 9KB, then the  expression becomes
governed by the receive window of 64KBytes:

  tp->rcvq_space.space
    ~=  min(rcv_wnd, 10*9KBytes)
    ~= min(64KByte,90KByte) = 65536 bytes

But because tp->rcvq_space.space is set to the rcv_wnd, and
the sender is not allowed to send more than the receive
window, this check in tcp_rcv_space_adjust() will always fail,
and the receiver will take the new_measure path:

        /* Number of bytes copied to user in last RTT */
        copied = tp->copied_seq - tp->rcvq_space.seq;
        if (copied <= tp->rcvq_space.space)
                goto new_measure;

This seems to be why receive buffer autotuning is never triggered.

Furthermore, if we try to fix it with:

-        if (copied <= tp->rcvq_space.space)
+        if (copied < tp->rcvq_space.space)

The buggy behavior will still remain, because 65536 is not a multiple
of the MSS, so the number of bytes copied to the user in the last RTT
will never, in practice, exactly match the tp->rcvq_space.space.

AFAICT the proposed patch fixes this bug by setting the
tp->rcvq_space.space to 10*536 = 5360. This is a number of bytes that
*can* actually be delivered in a round trip in a mixed 1500B/9KB
scenario, so this allows receive window auto-tuning to actually be
triggered.

It seems like a reasonable approach to fix this issue, but I agree
with Eric that it would be good to improve the commit message a bit.

Also, since this is a bug fix, it seems it should be directed to the
"net" branch rather than the "net-next" branch.

Also, FWIW, I think the "for high latency connections" can be dropped
from the commit summary/first-line, since the bug can be hit at any
RTT, and is simply easier to notice at high RTTs. You might consider
something like:

  tcp: fix receive buffer autotuning to trigger for any valid advertised MSS

best,
neal

^ permalink raw reply	[flat|nested] 22+ messages in thread

* [PATCH net] tcp: fix receive buffer autotuning to trigger for any valid advertised MSS
  2020-12-04 21:28 ` Neal Cardwell
@ 2020-12-07 11:46   ` Hazem Mohamed Abuelfotoh
  2020-12-07 18:53     ` Jakub Kicinski
  0 siblings, 1 reply; 22+ messages in thread
From: Hazem Mohamed Abuelfotoh @ 2020-12-07 11:46 UTC (permalink / raw)
  To: netdev
  Cc: stable, edumazet, ycheng, ncardwell, weiwan, astroh, benh,
	Hazem Mohamed Abuelfotoh

    Previously receiver buffer auto-tuning starts after receiving
    one advertised window amount of data.After the initial
    receiver buffer was raised by
    commit a337531b942b ("tcp: up initial rmem to 128KB
    and SYN rwin to around 64KB"),the receiver buffer may
    take too long for TCP autotuning to start raising
    the receiver buffer size.
    commit 041a14d26715 ("tcp: start receiver buffer autotuning sooner")
    tried to decrease the threshold at which TCP auto-tuning starts
    but it's doesn't work well in some environments
    where the receiver has large MTU (9001) especially with high RTT
    connections as in these environments rcvq_space.space will be the same
    as rcv_wnd so TCP autotuning will never start because
    sender can't send more than rcv_wnd size in one round trip.
    To address this issue this patch is decreasing the initial
    rcvq_space.space so TCP autotuning kicks in whenever the sender is
    able to send more than 5360 bytes in one round trip regardless the
    receiver's configured MTU.

    Fixes: a337531b942b ("tcp: up initial rmem to 128KB and SYN rwin to around 64KB")
    Fixes: 041a14d26715 ("tcp: start receiver buffer autotuning sooner")

Signed-off-by: Hazem Mohamed Abuelfotoh <abuehaze@amazon.com>
---
 net/ipv4/tcp_input.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 389d1b340248..f0ffac9e937b 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -504,13 +504,14 @@ static void tcp_grow_window(struct sock *sk, const struct sk_buff *skb)
 static void tcp_init_buffer_space(struct sock *sk)
 {
 	int tcp_app_win = sock_net(sk)->ipv4.sysctl_tcp_app_win;
+	struct inet_connection_sock *icsk = inet_csk(sk);
 	struct tcp_sock *tp = tcp_sk(sk);
 	int maxwin;
 
 	if (!(sk->sk_userlocks & SOCK_SNDBUF_LOCK))
 		tcp_sndbuf_expand(sk);
 
-	tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND * tp->advmss);
+	tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND * icsk->icsk_ack.rcv_mss);
 	tcp_mstamp_refresh(tp);
 	tp->rcvq_space.time = tp->tcp_mstamp;
 	tp->rcvq_space.seq = tp->copied_seq;
-- 
2.16.6




Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284

Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705




^ permalink raw reply related	[flat|nested] 22+ messages in thread

* Re: [PATCH net] tcp: fix receive buffer autotuning to trigger for any valid advertised MSS
  2020-12-07 11:46   ` [PATCH net] tcp: fix receive buffer autotuning to trigger for any valid advertised MSS Hazem Mohamed Abuelfotoh
@ 2020-12-07 18:53     ` Jakub Kicinski
  0 siblings, 0 replies; 22+ messages in thread
From: Jakub Kicinski @ 2020-12-07 18:53 UTC (permalink / raw)
  To: Hazem Mohamed Abuelfotoh
  Cc: netdev, stable, edumazet, ycheng, ncardwell, weiwan, astroh, benh

On Mon, 7 Dec 2020 11:46:25 +0000 Hazem Mohamed Abuelfotoh wrote:
>     Previously receiver buffer auto-tuning starts after receiving
>     one advertised window amount of data.After the initial
>     receiver buffer was raised by
>     commit a337531b942b ("tcp: up initial rmem to 128KB
>     and SYN rwin to around 64KB"),the receiver buffer may
>     take too long for TCP autotuning to start raising
>     the receiver buffer size.
>     commit 041a14d26715 ("tcp: start receiver buffer autotuning sooner")
>     tried to decrease the threshold at which TCP auto-tuning starts
>     but it's doesn't work well in some environments
>     where the receiver has large MTU (9001) especially with high RTT
>     connections as in these environments rcvq_space.space will be the same
>     as rcv_wnd so TCP autotuning will never start because
>     sender can't send more than rcv_wnd size in one round trip.
>     To address this issue this patch is decreasing the initial
>     rcvq_space.space so TCP autotuning kicks in whenever the sender is
>     able to send more than 5360 bytes in one round trip regardless the
>     receiver's configured MTU.
> 
>     Fixes: a337531b942b ("tcp: up initial rmem to 128KB and SYN rwin to around 64KB")
>     Fixes: 041a14d26715 ("tcp: start receiver buffer autotuning sooner")
> 
> Signed-off-by: Hazem Mohamed Abuelfotoh <abuehaze@amazon.com>

If the discussion concludes in favor of this patch please un-indent
this commit message, remove the empty line after the fixes tag, and 
repost.

^ permalink raw reply	[flat|nested] 22+ messages in thread

end of thread, other threads:[~2020-12-08 16:47 UTC | newest]

Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-04 18:06 [PATCH net-next] tcp: optimise receiver buffer autotuning initialisation for high latency connections Hazem Mohamed Abuelfotoh
2020-12-04 18:19 ` Mohamed Abuelfotoh, Hazem
2020-12-04 18:41   ` Eric Dumazet
     [not found]     ` <3F02FF08-EDA6-4DFD-8D93-479A5B05E25A@amazon.com>
2020-12-07 15:25       ` Eric Dumazet
2020-12-07 16:09         ` Mohamed Abuelfotoh, Hazem
2020-12-07 16:22           ` Eric Dumazet
2020-12-07 16:33             ` Neal Cardwell
2020-12-07 17:08               ` Eric Dumazet
2020-12-07 20:09                 ` Mohamed Abuelfotoh, Hazem
2020-12-07 23:22                   ` Eric Dumazet
2020-12-07 17:16               ` Mohamed Abuelfotoh, Hazem
2020-12-07 17:27                 ` Eric Dumazet
2020-12-08 16:28                   ` Mohamed Abuelfotoh, Hazem
2020-12-08 16:30                     ` Mohamed Abuelfotoh, Hazem
2020-12-08 16:46                     ` Eric Dumazet
2020-12-07 16:34             ` Mohamed Abuelfotoh, Hazem
2020-12-07 17:46               ` Greg KH
2020-12-07 17:54                 ` Mohamed Abuelfotoh, Hazem
2020-12-04 19:10 ` Eric Dumazet
2020-12-04 21:28 ` Neal Cardwell
2020-12-07 11:46   ` [PATCH net] tcp: fix receive buffer autotuning to trigger for any valid advertised MSS Hazem Mohamed Abuelfotoh
2020-12-07 18:53     ` Jakub Kicinski

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.