Fwd: 3.10.0 network trace

Fwd: 3.10.0 network trace

[Michael Sterrett]

Upgraded to 3.10.0 and started getting these appearing in the log:

[21893.282786] ------------[ cut here ]------------
[21893.282802] WARNING: at net/ipv4/tcp_input.c:2776
tcp_fastretrans_alert+0xbac/0xbf0()
[21893.282805] Modules linked in:
[21893.282813] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.0 #1
[21893.282816] Hardware name: ECS IC780M-A/IC780M-A, BIOS 080015  04/19/2010
[21893.282819]  ffffffff815186b2 0000000000000000 ffffffff810317ec
ffff88000a5ef500
[21893.282825]  0000000000000000 0000000000000007 0000000000004120
0000000000000001
[21893.282830]  ffffffff814647cc 0000000700000009 0000000000000000
2c81743c00000000
[21893.282835] Call Trace:
[21893.282838]  <IRQ>  [<ffffffff815186b2>] ? dump_stack+0xd/0x17
[21893.282854]  [<ffffffff810317ec>] ? warn_slowpath_common+0x6c/0xa0
[21893.282860]  [<ffffffff814647cc>] ? tcp_fastretrans_alert+0xbac/0xbf0
[21893.282865]  [<ffffffff81465257>] ? tcp_ack+0x997/0xd60
[21893.282872]  [<ffffffff81465aa1>] ? tcp_rcv_established+0xf1/0x640
[21893.282878]  [<ffffffff8146eb9e>] ? tcp_v4_do_rcv+0x1fe/0x4b0
[21893.282885]  [<ffffffff81434d9e>] ? nf_iterate+0x8e/0xc0
[21893.282890]  [<ffffffff81003cfc>] ? do_IRQ+0x5c/0xd0
[21893.282896]  [<ffffffff814711d7>] ? tcp_v4_rcv+0x757/0x780
[21893.282901]  [<ffffffff81434e47>] ? nf_hook_slow+0x77/0x150
[21893.282907]  [<ffffffff8144dfb0>] ? ip_rcv_finish+0x2a0/0x2a0
[21893.282914]  [<ffffffff8144e057>] ? ip_local_deliver_finish+0xa7/0x1e0
[21893.282921]  [<ffffffff8140c7cf>] ? __netif_receive_skb_core+0x49f/0x690
[21893.282926]  [<ffffffff8140cbb4>] ? netif_receive_skb+0x24/0x80
[21893.282931]  [<ffffffff8140e368>] ? napi_gro_receive+0xc8/0x100
[21893.282937]  [<ffffffff812ffecc>] ? rtl8169_poll+0x2cc/0x600
[21893.282942]  [<ffffffff8140ce29>] ? net_rx_action+0x79/0x130
[21893.282948]  [<ffffffff810382e8>] ? __do_softirq+0xc8/0x1a0
[21893.282953]  [<ffffffff81038516>] ? irq_exit+0x96/0xc0
[21893.282957]  [<ffffffff81003cfc>] ? do_IRQ+0x5c/0xd0
[21893.282963]  [<ffffffff8151caea>] ? common_interrupt+0x6a/0x6a
[21893.282965]  <EOI>  [<ffffffff8100a0a0>] ?
arch_remove_reservations+0x130/0x130
[21893.282978]  [<ffffffff8100a0a2>] ? default_idle+0x2/0x10
[21893.282983]  [<ffffffff810627f9>] ? cpu_startup_entry+0x79/0x160
[21893.282989]  [<ffffffff81732d54>] ? start_kernel+0x325/0x330
[21893.282993]  [<ffffffff81732848>] ? repair_env_string+0x58/0x58
[21893.282997] ---[ end trace d2be5719b9c30b65 ]---

If this is is already a known issue please let me know.  Otherwise,
I'm happy to provide any additional information that might be helpful
in tracking it down.

Re: 3.10.0 network trace

[Yuchung Cheng]

On Fri, Jul 12, 2013 at 1:44 PM, Michael Sterrett <michael@sterretts.net> wrote:
> Upgraded to 3.10.0 and started getting these appearing in the log:
>
> [21893.282786] ------------[ cut here ]------------
> [21893.282802] WARNING: at net/ipv4/tcp_input.c:2776
> tcp_fastretrans_alert+0xbac/0xbf0()
> [21893.282805] Modules linked in:
> [21893.282813] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.0 #1
> [21893.282816] Hardware name: ECS IC780M-A/IC780M-A, BIOS 080015  04/19/2010
> [21893.282819]  ffffffff815186b2 0000000000000000 ffffffff810317ec
> ffff88000a5ef500
> [21893.282825]  0000000000000000 0000000000000007 0000000000004120
> 0000000000000001
> [21893.282830]  ffffffff814647cc 0000000700000009 0000000000000000
> 2c81743c00000000
> [21893.282835] Call Trace:
> [21893.282838]  <IRQ>  [<ffffffff815186b2>] ? dump_stack+0xd/0x17
> [21893.282854]  [<ffffffff810317ec>] ? warn_slowpath_common+0x6c/0xa0
> [21893.282860]  [<ffffffff814647cc>] ? tcp_fastretrans_alert+0xbac/0xbf0
> [21893.282865]  [<ffffffff81465257>] ? tcp_ack+0x997/0xd60
> [21893.282872]  [<ffffffff81465aa1>] ? tcp_rcv_established+0xf1/0x640
> [21893.282878]  [<ffffffff8146eb9e>] ? tcp_v4_do_rcv+0x1fe/0x4b0
> [21893.282885]  [<ffffffff81434d9e>] ? nf_iterate+0x8e/0xc0
> [21893.282890]  [<ffffffff81003cfc>] ? do_IRQ+0x5c/0xd0
> [21893.282896]  [<ffffffff814711d7>] ? tcp_v4_rcv+0x757/0x780
> [21893.282901]  [<ffffffff81434e47>] ? nf_hook_slow+0x77/0x150
> [21893.282907]  [<ffffffff8144dfb0>] ? ip_rcv_finish+0x2a0/0x2a0
> [21893.282914]  [<ffffffff8144e057>] ? ip_local_deliver_finish+0xa7/0x1e0
> [21893.282921]  [<ffffffff8140c7cf>] ? __netif_receive_skb_core+0x49f/0x690
> [21893.282926]  [<ffffffff8140cbb4>] ? netif_receive_skb+0x24/0x80
> [21893.282931]  [<ffffffff8140e368>] ? napi_gro_receive+0xc8/0x100
> [21893.282937]  [<ffffffff812ffecc>] ? rtl8169_poll+0x2cc/0x600
> [21893.282942]  [<ffffffff8140ce29>] ? net_rx_action+0x79/0x130
> [21893.282948]  [<ffffffff810382e8>] ? __do_softirq+0xc8/0x1a0
> [21893.282953]  [<ffffffff81038516>] ? irq_exit+0x96/0xc0
> [21893.282957]  [<ffffffff81003cfc>] ? do_IRQ+0x5c/0xd0
> [21893.282963]  [<ffffffff8151caea>] ? common_interrupt+0x6a/0x6a
> [21893.282965]  <EOI>  [<ffffffff8100a0a0>] ?
> arch_remove_reservations+0x130/0x130
> [21893.282978]  [<ffffffff8100a0a2>] ? default_idle+0x2/0x10
> [21893.282983]  [<ffffffff810627f9>] ? cpu_startup_entry+0x79/0x160
> [21893.282989]  [<ffffffff81732d54>] ? start_kernel+0x325/0x330
> [21893.282993]  [<ffffffff81732848>] ? repair_env_string+0x58/0x58
> [21893.282997] ---[ end trace d2be5719b9c30b65 ]---
>
> If this is is already a known issue please let me know.  Otherwise,
> I'm happy to provide any additional information that might be helpful
> in tracking it down.
can you send the output of 'sysctl -a'.
one related fix is commit 93b174ad71b08e504c2cf6e8a58ecce778b77a40



> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3.10.0 network trace

[Michael Sterrett]

[-- Attachment #1: Type: text/plain, Size: 57 bytes --]

Thanks for the reply.  The sysctl -a output is attached.

[-- Attachment #2: sysctl-a --]
[-- Type: application/octet-stream, Size: 28172 bytes --]

abi.vsyscall32 = 1
debug.exception-trace = 1
dev.cdrom.autoclose = 1
dev.cdrom.autoeject = 0
dev.cdrom.check_media = 0
dev.cdrom.debug = 0
dev.cdrom.info = CD-ROM information, Id: cdrom.c 3.20 2003/12/17
dev.cdrom.info = 
dev.cdrom.info = drive name:	
dev.cdrom.info = drive speed:	
dev.cdrom.info = drive # of slots:
dev.cdrom.info = Can close tray:	
dev.cdrom.info = Can open tray:	
dev.cdrom.info = Can lock tray:	
dev.cdrom.info = Can change speed:
dev.cdrom.info = Can select disk:
dev.cdrom.info = Can read multisession:
dev.cdrom.info = Can read MCN:	
dev.cdrom.info = Reports media changed:
dev.cdrom.info = Can play audio:	
dev.cdrom.info = Can write CD-R:	
dev.cdrom.info = Can write CD-RW:
dev.cdrom.info = Can read DVD:	
dev.cdrom.info = Can write DVD-R:
dev.cdrom.info = Can write DVD-RAM:
dev.cdrom.info = Can read MRW:	
dev.cdrom.info = Can write MRW:	
dev.cdrom.info = Can write RAM:	
dev.cdrom.info = 
dev.cdrom.info = 
dev.cdrom.lock = 1
dev.hpet.max-user-freq = 64
dev.raid.speed_limit_max = 200000
dev.raid.speed_limit_min = 1000
dev.scsi.logging_level = 0
fs.aio-max-nr = 65536
fs.aio-nr = 0
fs.dentry-state = 6950	4020	45	0	0	0
fs.dir-notify-enable = 1
fs.epoll.max_user_watches = 209080
fs.file-max = 102048
fs.file-nr = 1568	0	102048
fs.inode-nr = 10000	3641
fs.inode-state = 10000	3641	0	0	0	0	0
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 8192
fs.lease-break-time = 45
fs.leases-enable = 1
fs.mqueue.msg_default = 10
fs.mqueue.msg_max = 10
fs.mqueue.msgsize_default = 8192
fs.mqueue.msgsize_max = 8192
fs.mqueue.queues_max = 256
fs.nfs.nfs_congestion_kb = 32320
fs.nfs.nfs_mountpoint_timeout = 500
fs.nfs.nlm_grace_period = 0
fs.nfs.nlm_tcpport = 0
fs.nfs.nlm_timeout = 10
fs.nfs.nlm_udpport = 0
fs.nfs.nsm_local_state = 81
fs.nfs.nsm_use_hostnames = 0
fs.nr_open = 1048576
fs.overflowgid = 65534
fs.overflowuid = 65534
fs.pipe-max-size = 1048576
fs.protected_hardlinks = 0
fs.protected_symlinks = 0
fs.suid_dumpable = 0
kernel.acct = 4	2	30
kernel.auto_msgmni = 1
kernel.blk_iopoll = 1
kernel.bootloader_type = 113
kernel.bootloader_version = 1
kernel.cad_pid = 1
kernel.cap_last_cap = 36
kernel.compat-log = 1
kernel.core_pattern = core
kernel.core_pipe_limit = 0
kernel.core_uses_pid = 0
kernel.ctrl-alt-del = 0
kernel.dmesg_restrict = 0
kernel.domainname = yp.sterrettnet
kernel.hostname = eyes
kernel.hotplug = 
kernel.io_delay_type = 0
kernel.kptr_restrict = 0
kernel.kstack_depth_to_print = 12
kernel.max_lock_depth = 1024
kernel.modprobe = /sbin/modprobe
kernel.modules_disabled = 0
kernel.msgmax = 8192
kernel.msgmnb = 16384
kernel.msgmni = 1993
kernel.ngroups_max = 65536
kernel.osrelease = 3.10.0
kernel.ostype = Linux
kernel.overflowgid = 65534
kernel.overflowuid = 65534
kernel.panic = 0
kernel.panic_on_io_nmi = 0
kernel.panic_on_oops = 0
kernel.panic_on_unrecovered_nmi = 0
kernel.perf_event_max_sample_rate = 100000
kernel.perf_event_mlock_kb = 516
kernel.perf_event_paranoid = 1
kernel.pid_max = 32768
kernel.poweroff_cmd = /sbin/poweroff
kernel.print-fatal-signals = 0
kernel.printk = 1	4	1	7
kernel.printk_delay = 0
kernel.printk_ratelimit = 5
kernel.printk_ratelimit_burst = 10
kernel.pty.max = 4096
kernel.pty.nr = 1
kernel.pty.reserve = 1024
kernel.random.boot_id = 9cc64c2b-66ba-49a3-a0bf-e3080f97efdb
kernel.random.entropy_avail = 3968
kernel.random.poolsize = 4096
kernel.random.read_wakeup_threshold = 64
kernel.random.uuid = 993b1af4-4653-4dc4-b300-228834e09423
kernel.random.write_wakeup_threshold = 128
kernel.randomize_va_space = 2
kernel.real-root-dev = 0
kernel.sched_child_runs_first = 0
kernel.sched_rr_timeslice_ms = 100
kernel.sched_rt_period_us = 1000000
kernel.sched_rt_runtime_us = 950000
kernel.sem = 250	32000	32	128
kernel.shm_rmid_forced = 0
kernel.shmall = 2097152
kernel.shmmax = 33554432
kernel.shmmni = 4096
kernel.tainted = 512
kernel.threads-max = 15951
kernel.unknown_nmi_panic = 0
kernel.usermodehelper.bset = 4294967295	4294967295
kernel.usermodehelper.inheritable = 4294967295	4294967295
kernel.version = #1 SMP Wed Jul 10 00:47:26 EDT 2013
net.core.dev_weight = 64
net.core.message_burst = 10
net.core.message_cost = 5
net.core.netdev_budget = 300
net.core.netdev_max_backlog = 1000
net.core.netdev_tstamp_prequeue = 1
net.core.optmem_max = 20480
net.core.rmem_default = 212992
net.core.rmem_max = 212992
net.core.rps_sock_flow_entries = 0
net.core.somaxconn = 128
net.core.warnings = 1
net.core.wmem_default = 212992
net.core.wmem_max = 212992
net.core.xfrm_acq_expires = 30
net.core.xfrm_aevent_etime = 10
net.core.xfrm_aevent_rseqth = 2
net.core.xfrm_larval_drop = 1
net.ipv4.conf.all.accept_local = 0
net.ipv4.conf.all.accept_redirects = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_notify = 0
net.ipv4.conf.all.bootp_relay = 0
net.ipv4.conf.all.disable_policy = 0
net.ipv4.conf.all.disable_xfrm = 0
net.ipv4.conf.all.force_igmp_version = 0
net.ipv4.conf.all.forwarding = 0
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.all.medium_id = 0
net.ipv4.conf.all.promote_secondaries = 0
net.ipv4.conf.all.proxy_arp = 0
net.ipv4.conf.all.proxy_arp_pvlan = 0
net.ipv4.conf.all.route_localnet = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.secure_redirects = 1
net.ipv4.conf.all.send_redirects = 1
net.ipv4.conf.all.shared_media = 1
net.ipv4.conf.all.src_valid_mark = 0
net.ipv4.conf.all.tag = 0
net.ipv4.conf.default.accept_local = 0
net.ipv4.conf.default.accept_redirects = 1
net.ipv4.conf.default.accept_source_route = 1
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_notify = 0
net.ipv4.conf.default.bootp_relay = 0
net.ipv4.conf.default.disable_policy = 0
net.ipv4.conf.default.disable_xfrm = 0
net.ipv4.conf.default.force_igmp_version = 0
net.ipv4.conf.default.forwarding = 0
net.ipv4.conf.default.log_martians = 0
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.default.medium_id = 0
net.ipv4.conf.default.promote_secondaries = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.default.proxy_arp_pvlan = 0
net.ipv4.conf.default.route_localnet = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.secure_redirects = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.default.shared_media = 1
net.ipv4.conf.default.src_valid_mark = 0
net.ipv4.conf.default.tag = 0
net.ipv4.conf.dummy0.accept_local = 0
net.ipv4.conf.dummy0.accept_redirects = 1
net.ipv4.conf.dummy0.accept_source_route = 1
net.ipv4.conf.dummy0.arp_accept = 0
net.ipv4.conf.dummy0.arp_announce = 0
net.ipv4.conf.dummy0.arp_filter = 0
net.ipv4.conf.dummy0.arp_ignore = 0
net.ipv4.conf.dummy0.arp_notify = 0
net.ipv4.conf.dummy0.bootp_relay = 0
net.ipv4.conf.dummy0.disable_policy = 0
net.ipv4.conf.dummy0.disable_xfrm = 0
net.ipv4.conf.dummy0.force_igmp_version = 0
net.ipv4.conf.dummy0.forwarding = 0
net.ipv4.conf.dummy0.log_martians = 0
net.ipv4.conf.dummy0.mc_forwarding = 0
net.ipv4.conf.dummy0.medium_id = 0
net.ipv4.conf.dummy0.promote_secondaries = 0
net.ipv4.conf.dummy0.proxy_arp = 0
net.ipv4.conf.dummy0.proxy_arp_pvlan = 0
net.ipv4.conf.dummy0.route_localnet = 0
net.ipv4.conf.dummy0.rp_filter = 1
net.ipv4.conf.dummy0.secure_redirects = 1
net.ipv4.conf.dummy0.send_redirects = 1
net.ipv4.conf.dummy0.shared_media = 1
net.ipv4.conf.dummy0.src_valid_mark = 0
net.ipv4.conf.dummy0.tag = 0
net.ipv4.conf.eth0.accept_local = 0
net.ipv4.conf.eth0.accept_redirects = 1
net.ipv4.conf.eth0.accept_source_route = 1
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_notify = 0
net.ipv4.conf.eth0.bootp_relay = 0
net.ipv4.conf.eth0.disable_policy = 0
net.ipv4.conf.eth0.disable_xfrm = 0
net.ipv4.conf.eth0.force_igmp_version = 0
net.ipv4.conf.eth0.forwarding = 0
net.ipv4.conf.eth0.log_martians = 0
net.ipv4.conf.eth0.mc_forwarding = 0
net.ipv4.conf.eth0.medium_id = 0
net.ipv4.conf.eth0.promote_secondaries = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.eth0.proxy_arp_pvlan = 0
net.ipv4.conf.eth0.route_localnet = 0
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.eth0.secure_redirects = 1
net.ipv4.conf.eth0.send_redirects = 1
net.ipv4.conf.eth0.shared_media = 1
net.ipv4.conf.eth0.src_valid_mark = 0
net.ipv4.conf.eth0.tag = 0
net.ipv4.conf.lo.accept_local = 0
net.ipv4.conf.lo.accept_redirects = 1
net.ipv4.conf.lo.accept_source_route = 1
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_notify = 0
net.ipv4.conf.lo.bootp_relay = 0
net.ipv4.conf.lo.disable_policy = 1
net.ipv4.conf.lo.disable_xfrm = 1
net.ipv4.conf.lo.force_igmp_version = 0
net.ipv4.conf.lo.forwarding = 0
net.ipv4.conf.lo.log_martians = 0
net.ipv4.conf.lo.mc_forwarding = 0
net.ipv4.conf.lo.medium_id = 0
net.ipv4.conf.lo.promote_secondaries = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.lo.proxy_arp_pvlan = 0
net.ipv4.conf.lo.route_localnet = 0
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.lo.secure_redirects = 1
net.ipv4.conf.lo.send_redirects = 1
net.ipv4.conf.lo.shared_media = 1
net.ipv4.conf.lo.src_valid_mark = 0
net.ipv4.conf.lo.tag = 0
net.ipv4.conf.sit0.accept_local = 0
net.ipv4.conf.sit0.accept_redirects = 1
net.ipv4.conf.sit0.accept_source_route = 1
net.ipv4.conf.sit0.arp_accept = 0
net.ipv4.conf.sit0.arp_announce = 0
net.ipv4.conf.sit0.arp_filter = 0
net.ipv4.conf.sit0.arp_ignore = 0
net.ipv4.conf.sit0.arp_notify = 0
net.ipv4.conf.sit0.bootp_relay = 0
net.ipv4.conf.sit0.disable_policy = 0
net.ipv4.conf.sit0.disable_xfrm = 0
net.ipv4.conf.sit0.force_igmp_version = 0
net.ipv4.conf.sit0.forwarding = 0
net.ipv4.conf.sit0.log_martians = 0
net.ipv4.conf.sit0.mc_forwarding = 0
net.ipv4.conf.sit0.medium_id = 0
net.ipv4.conf.sit0.promote_secondaries = 0
net.ipv4.conf.sit0.proxy_arp = 0
net.ipv4.conf.sit0.proxy_arp_pvlan = 0
net.ipv4.conf.sit0.route_localnet = 0
net.ipv4.conf.sit0.rp_filter = 1
net.ipv4.conf.sit0.secure_redirects = 1
net.ipv4.conf.sit0.send_redirects = 1
net.ipv4.conf.sit0.shared_media = 1
net.ipv4.conf.sit0.src_valid_mark = 0
net.ipv4.conf.sit0.tag = 0
net.ipv4.icmp_echo_ignore_all = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_errors_use_inbound_ifaddr = 0
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.icmp_ratelimit = 1000
net.ipv4.icmp_ratemask = 6168
net.ipv4.igmp_max_memberships = 20
net.ipv4.igmp_max_msf = 10
net.ipv4.inet_peer_maxttl = 600
net.ipv4.inet_peer_minttl = 120
net.ipv4.inet_peer_threshold = 65664
net.ipv4.ip_default_ttl = 64
net.ipv4.ip_dynaddr = 0
net.ipv4.ip_early_demux = 1
net.ipv4.ip_forward = 0
net.ipv4.ip_local_port_range = 32768	61000
net.ipv4.ip_local_reserved_ports = 
net.ipv4.ip_no_pmtu_disc = 0
net.ipv4.ip_nonlocal_bind = 0
net.ipv4.ipfrag_high_thresh = 4194304
net.ipv4.ipfrag_low_thresh = 3145728
net.ipv4.ipfrag_max_dist = 64
net.ipv4.ipfrag_secret_interval = 600
net.ipv4.ipfrag_time = 30
net.ipv4.neigh.default.anycast_delay = 100
net.ipv4.neigh.default.app_solicit = 0
net.ipv4.neigh.default.base_reachable_time_ms = 30000
net.ipv4.neigh.default.delay_first_probe_time = 5
net.ipv4.neigh.default.gc_interval = 30
net.ipv4.neigh.default.gc_stale_time = 60
net.ipv4.neigh.default.gc_thresh1 = 128
net.ipv4.neigh.default.gc_thresh2 = 512
net.ipv4.neigh.default.gc_thresh3 = 1024
net.ipv4.neigh.default.locktime = 100
net.ipv4.neigh.default.mcast_solicit = 3
net.ipv4.neigh.default.proxy_delay = 80
net.ipv4.neigh.default.proxy_qlen = 64
net.ipv4.neigh.default.retrans_time_ms = 1000
net.ipv4.neigh.default.ucast_solicit = 3
net.ipv4.neigh.default.unres_qlen = 31
net.ipv4.neigh.default.unres_qlen_bytes = 65536
net.ipv4.neigh.dummy0.anycast_delay = 100
net.ipv4.neigh.dummy0.app_solicit = 0
net.ipv4.neigh.dummy0.base_reachable_time_ms = 30000
net.ipv4.neigh.dummy0.delay_first_probe_time = 5
net.ipv4.neigh.dummy0.gc_stale_time = 60
net.ipv4.neigh.dummy0.locktime = 100
net.ipv4.neigh.dummy0.mcast_solicit = 3
net.ipv4.neigh.dummy0.proxy_delay = 80
net.ipv4.neigh.dummy0.proxy_qlen = 64
net.ipv4.neigh.dummy0.retrans_time_ms = 1000
net.ipv4.neigh.dummy0.ucast_solicit = 3
net.ipv4.neigh.dummy0.unres_qlen = 31
net.ipv4.neigh.dummy0.unres_qlen_bytes = 65536
net.ipv4.neigh.eth0.anycast_delay = 100
net.ipv4.neigh.eth0.app_solicit = 0
net.ipv4.neigh.eth0.base_reachable_time_ms = 30000
net.ipv4.neigh.eth0.delay_first_probe_time = 5
net.ipv4.neigh.eth0.gc_stale_time = 60
net.ipv4.neigh.eth0.locktime = 100
net.ipv4.neigh.eth0.mcast_solicit = 3
net.ipv4.neigh.eth0.proxy_delay = 80
net.ipv4.neigh.eth0.proxy_qlen = 64
net.ipv4.neigh.eth0.retrans_time_ms = 1000
net.ipv4.neigh.eth0.ucast_solicit = 3
net.ipv4.neigh.eth0.unres_qlen = 31
net.ipv4.neigh.eth0.unres_qlen_bytes = 65536
net.ipv4.neigh.lo.anycast_delay = 100
net.ipv4.neigh.lo.app_solicit = 0
net.ipv4.neigh.lo.base_reachable_time_ms = 30000
net.ipv4.neigh.lo.delay_first_probe_time = 5
net.ipv4.neigh.lo.gc_stale_time = 60
net.ipv4.neigh.lo.locktime = 100
net.ipv4.neigh.lo.mcast_solicit = 3
net.ipv4.neigh.lo.proxy_delay = 80
net.ipv4.neigh.lo.proxy_qlen = 64
net.ipv4.neigh.lo.retrans_time_ms = 1000
net.ipv4.neigh.lo.ucast_solicit = 3
net.ipv4.neigh.lo.unres_qlen = 31
net.ipv4.neigh.lo.unres_qlen_bytes = 65536
net.ipv4.neigh.sit0.anycast_delay = 100
net.ipv4.neigh.sit0.app_solicit = 0
net.ipv4.neigh.sit0.base_reachable_time_ms = 30000
net.ipv4.neigh.sit0.delay_first_probe_time = 5
net.ipv4.neigh.sit0.gc_stale_time = 60
net.ipv4.neigh.sit0.locktime = 100
net.ipv4.neigh.sit0.mcast_solicit = 3
net.ipv4.neigh.sit0.proxy_delay = 80
net.ipv4.neigh.sit0.proxy_qlen = 64
net.ipv4.neigh.sit0.retrans_time_ms = 1000
net.ipv4.neigh.sit0.ucast_solicit = 3
net.ipv4.neigh.sit0.unres_qlen = 31
net.ipv4.neigh.sit0.unres_qlen_bytes = 65536
net.ipv4.netfilter.ip_conntrack_buckets = 8192
net.ipv4.netfilter.ip_conntrack_checksum = 1
net.ipv4.netfilter.ip_conntrack_count = 165
net.ipv4.netfilter.ip_conntrack_generic_timeout = 600
net.ipv4.netfilter.ip_conntrack_icmp_timeout = 30
net.ipv4.netfilter.ip_conntrack_log_invalid = 0
net.ipv4.netfilter.ip_conntrack_max = 31900
net.ipv4.netfilter.ip_conntrack_tcp_be_liberal = 0
net.ipv4.netfilter.ip_conntrack_tcp_loose = 1
net.ipv4.netfilter.ip_conntrack_tcp_max_retrans = 3
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close = 10
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 432000
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_max_retrans = 300
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent2 = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 120
net.ipv4.netfilter.ip_conntrack_udp_timeout = 30
net.ipv4.netfilter.ip_conntrack_udp_timeout_stream = 180
net.ipv4.ping_group_range = 1	0
net.ipv4.route.error_burst = 5000
net.ipv4.route.error_cost = 1000
net.ipv4.route.gc_elasticity = 8
net.ipv4.route.gc_interval = 60
net.ipv4.route.gc_min_interval = 0
net.ipv4.route.gc_min_interval_ms = 500
net.ipv4.route.gc_thresh = -1
net.ipv4.route.gc_timeout = 300
net.ipv4.route.max_size = 2147483647
net.ipv4.route.min_adv_mss = 256
net.ipv4.route.min_pmtu = 552
net.ipv4.route.mtu_expires = 600
net.ipv4.route.redirect_load = 20
net.ipv4.route.redirect_number = 9
net.ipv4.route.redirect_silence = 20480
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_allowed_congestion_control = cubic reno
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_available_congestion_control = cubic reno
net.ipv4.tcp_base_mss = 512
net.ipv4.tcp_challenge_ack_limit = 100
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_early_retrans = 3
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_fack = 1
net.ipv4.tcp_fastopen = 0
net.ipv4.tcp_fastopen_key = c1a9388e-f9e19b5a-27b15630-7b5cd491
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_frto = 2
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_limit_output_bytes = 131072
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_max_orphans = 4096
net.ipv4.tcp_max_ssthresh = 0
net.ipv4.tcp_max_syn_backlog = 128
net.ipv4.tcp_max_tw_buckets = 4096
net.ipv4.tcp_mem = 23784	31713	47568
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_rmem = 4096	87380	6291456
net.ipv4.tcp_sack = 1
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_syn_retries = 6
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_thin_dupack = 0
net.ipv4.tcp_thin_linear_timeouts = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_wmem = 4096	16384	4194304
net.ipv4.tcp_workaround_signed_windows = 0
net.ipv4.udp_mem = 23925	31902	47850
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
net.ipv4.xfrm4_gc_thresh = 1024
net.ipv6.bindv6only = 0
net.ipv6.conf.all.accept_dad = 1
net.ipv6.conf.all.accept_ra = 1
net.ipv6.conf.all.accept_ra_defrtr = 1
net.ipv6.conf.all.accept_ra_pinfo = 1
net.ipv6.conf.all.accept_redirects = 1
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.all.autoconf = 1
net.ipv6.conf.all.dad_transmits = 1
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.all.force_mld_version = 0
net.ipv6.conf.all.force_tllao = 0
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.all.hop_limit = 64
net.ipv6.conf.all.max_addresses = 16
net.ipv6.conf.all.mtu = 1280
net.ipv6.conf.all.ndisc_notify = 0
net.ipv6.conf.all.proxy_ndp = 0
net.ipv6.conf.all.router_solicitation_delay = 1
net.ipv6.conf.all.router_solicitation_interval = 4
net.ipv6.conf.all.router_solicitations = 3
net.ipv6.conf.default.accept_dad = 1
net.ipv6.conf.default.accept_ra = 1
net.ipv6.conf.default.accept_ra_defrtr = 1
net.ipv6.conf.default.accept_ra_pinfo = 1
net.ipv6.conf.default.accept_redirects = 1
net.ipv6.conf.default.accept_source_route = 0
net.ipv6.conf.default.autoconf = 1
net.ipv6.conf.default.dad_transmits = 1
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.default.force_mld_version = 0
net.ipv6.conf.default.force_tllao = 0
net.ipv6.conf.default.forwarding = 0
net.ipv6.conf.default.hop_limit = 64
net.ipv6.conf.default.max_addresses = 16
net.ipv6.conf.default.mtu = 1280
net.ipv6.conf.default.ndisc_notify = 0
net.ipv6.conf.default.proxy_ndp = 0
net.ipv6.conf.default.router_solicitation_delay = 1
net.ipv6.conf.default.router_solicitation_interval = 4
net.ipv6.conf.default.router_solicitations = 3
net.ipv6.conf.dummy0.accept_dad = -1
net.ipv6.conf.dummy0.accept_ra = 1
net.ipv6.conf.dummy0.accept_ra_defrtr = 1
net.ipv6.conf.dummy0.accept_ra_pinfo = 1
net.ipv6.conf.dummy0.accept_redirects = 1
net.ipv6.conf.dummy0.accept_source_route = 0
net.ipv6.conf.dummy0.autoconf = 1
net.ipv6.conf.dummy0.dad_transmits = 1
net.ipv6.conf.dummy0.disable_ipv6 = 0
net.ipv6.conf.dummy0.force_mld_version = 0
net.ipv6.conf.dummy0.force_tllao = 0
net.ipv6.conf.dummy0.forwarding = 0
net.ipv6.conf.dummy0.hop_limit = 64
net.ipv6.conf.dummy0.max_addresses = 16
net.ipv6.conf.dummy0.mtu = 1500
net.ipv6.conf.dummy0.ndisc_notify = 0
net.ipv6.conf.dummy0.proxy_ndp = 0
net.ipv6.conf.dummy0.router_solicitation_delay = 1
net.ipv6.conf.dummy0.router_solicitation_interval = 4
net.ipv6.conf.dummy0.router_solicitations = 3
net.ipv6.conf.eth0.accept_dad = 1
net.ipv6.conf.eth0.accept_ra = 1
net.ipv6.conf.eth0.accept_ra_defrtr = 1
net.ipv6.conf.eth0.accept_ra_pinfo = 1
net.ipv6.conf.eth0.accept_redirects = 1
net.ipv6.conf.eth0.accept_source_route = 0
net.ipv6.conf.eth0.autoconf = 1
net.ipv6.conf.eth0.dad_transmits = 1
net.ipv6.conf.eth0.disable_ipv6 = 0
net.ipv6.conf.eth0.force_mld_version = 0
net.ipv6.conf.eth0.force_tllao = 0
net.ipv6.conf.eth0.forwarding = 0
net.ipv6.conf.eth0.hop_limit = 64
net.ipv6.conf.eth0.max_addresses = 16
net.ipv6.conf.eth0.mtu = 1500
net.ipv6.conf.eth0.ndisc_notify = 0
net.ipv6.conf.eth0.proxy_ndp = 0
net.ipv6.conf.eth0.router_solicitation_delay = 1
net.ipv6.conf.eth0.router_solicitation_interval = 4
net.ipv6.conf.eth0.router_solicitations = 3
net.ipv6.conf.lo.accept_dad = -1
net.ipv6.conf.lo.accept_ra = 1
net.ipv6.conf.lo.accept_ra_defrtr = 1
net.ipv6.conf.lo.accept_ra_pinfo = 1
net.ipv6.conf.lo.accept_redirects = 1
net.ipv6.conf.lo.accept_source_route = 0
net.ipv6.conf.lo.autoconf = 1
net.ipv6.conf.lo.dad_transmits = 1
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.lo.force_mld_version = 0
net.ipv6.conf.lo.force_tllao = 0
net.ipv6.conf.lo.forwarding = 0
net.ipv6.conf.lo.hop_limit = 64
net.ipv6.conf.lo.max_addresses = 16
net.ipv6.conf.lo.mtu = 65536
net.ipv6.conf.lo.ndisc_notify = 0
net.ipv6.conf.lo.proxy_ndp = 0
net.ipv6.conf.lo.router_solicitation_delay = 1
net.ipv6.conf.lo.router_solicitation_interval = 4
net.ipv6.conf.lo.router_solicitations = 3
net.ipv6.conf.sit0.accept_dad = -1
net.ipv6.conf.sit0.accept_ra = 1
net.ipv6.conf.sit0.accept_ra_defrtr = 1
net.ipv6.conf.sit0.accept_ra_pinfo = 1
net.ipv6.conf.sit0.accept_redirects = 1
net.ipv6.conf.sit0.accept_source_route = 0
net.ipv6.conf.sit0.autoconf = 1
net.ipv6.conf.sit0.dad_transmits = 1
net.ipv6.conf.sit0.disable_ipv6 = 0
net.ipv6.conf.sit0.force_mld_version = 0
net.ipv6.conf.sit0.force_tllao = 0
net.ipv6.conf.sit0.forwarding = 0
net.ipv6.conf.sit0.hop_limit = 64
net.ipv6.conf.sit0.max_addresses = 16
net.ipv6.conf.sit0.mtu = 1480
net.ipv6.conf.sit0.ndisc_notify = 0
net.ipv6.conf.sit0.proxy_ndp = 0
net.ipv6.conf.sit0.router_solicitation_delay = 1
net.ipv6.conf.sit0.router_solicitation_interval = 4
net.ipv6.conf.sit0.router_solicitations = 3
net.ipv6.icmp.ratelimit = 1000
net.ipv6.ip6frag_high_thresh = 4194304
net.ipv6.ip6frag_low_thresh = 3145728
net.ipv6.ip6frag_secret_interval = 600
net.ipv6.ip6frag_time = 60
net.ipv6.mld_max_msf = 64
net.ipv6.neigh.default.anycast_delay = 100
net.ipv6.neigh.default.app_solicit = 0
net.ipv6.neigh.default.base_reachable_time_ms = 30000
net.ipv6.neigh.default.delay_first_probe_time = 5
net.ipv6.neigh.default.gc_interval = 30
net.ipv6.neigh.default.gc_stale_time = 60
net.ipv6.neigh.default.gc_thresh1 = 128
net.ipv6.neigh.default.gc_thresh2 = 512
net.ipv6.neigh.default.gc_thresh3 = 1024
net.ipv6.neigh.default.locktime = 0
net.ipv6.neigh.default.mcast_solicit = 3
net.ipv6.neigh.default.proxy_delay = 80
net.ipv6.neigh.default.proxy_qlen = 64
net.ipv6.neigh.default.retrans_time_ms = 1000
net.ipv6.neigh.default.ucast_solicit = 3
net.ipv6.neigh.default.unres_qlen = 31
net.ipv6.neigh.default.unres_qlen_bytes = 65536
net.ipv6.neigh.dummy0.anycast_delay = 100
net.ipv6.neigh.dummy0.app_solicit = 0
net.ipv6.neigh.dummy0.base_reachable_time_ms = 30000
net.ipv6.neigh.dummy0.delay_first_probe_time = 5
net.ipv6.neigh.dummy0.gc_stale_time = 60
net.ipv6.neigh.dummy0.locktime = 0
net.ipv6.neigh.dummy0.mcast_solicit = 3
net.ipv6.neigh.dummy0.proxy_delay = 80
net.ipv6.neigh.dummy0.proxy_qlen = 64
net.ipv6.neigh.dummy0.retrans_time_ms = 1000
net.ipv6.neigh.dummy0.ucast_solicit = 3
net.ipv6.neigh.dummy0.unres_qlen = 31
net.ipv6.neigh.dummy0.unres_qlen_bytes = 65536
net.ipv6.neigh.eth0.anycast_delay = 100
net.ipv6.neigh.eth0.app_solicit = 0
net.ipv6.neigh.eth0.base_reachable_time_ms = 30000
net.ipv6.neigh.eth0.delay_first_probe_time = 5
net.ipv6.neigh.eth0.gc_stale_time = 60
net.ipv6.neigh.eth0.locktime = 0
net.ipv6.neigh.eth0.mcast_solicit = 3
net.ipv6.neigh.eth0.proxy_delay = 80
net.ipv6.neigh.eth0.proxy_qlen = 64
net.ipv6.neigh.eth0.retrans_time_ms = 1000
net.ipv6.neigh.eth0.ucast_solicit = 3
net.ipv6.neigh.eth0.unres_qlen = 31
net.ipv6.neigh.eth0.unres_qlen_bytes = 65536
net.ipv6.neigh.lo.anycast_delay = 100
net.ipv6.neigh.lo.app_solicit = 0
net.ipv6.neigh.lo.base_reachable_time_ms = 30000
net.ipv6.neigh.lo.delay_first_probe_time = 5
net.ipv6.neigh.lo.gc_stale_time = 60
net.ipv6.neigh.lo.locktime = 0
net.ipv6.neigh.lo.mcast_solicit = 3
net.ipv6.neigh.lo.proxy_delay = 80
net.ipv6.neigh.lo.proxy_qlen = 64
net.ipv6.neigh.lo.retrans_time_ms = 1000
net.ipv6.neigh.lo.ucast_solicit = 3
net.ipv6.neigh.lo.unres_qlen = 31
net.ipv6.neigh.lo.unres_qlen_bytes = 65536
net.ipv6.neigh.sit0.anycast_delay = 100
net.ipv6.neigh.sit0.app_solicit = 0
net.ipv6.neigh.sit0.base_reachable_time_ms = 30000
net.ipv6.neigh.sit0.delay_first_probe_time = 5
net.ipv6.neigh.sit0.gc_stale_time = 60
net.ipv6.neigh.sit0.locktime = 0
net.ipv6.neigh.sit0.mcast_solicit = 3
net.ipv6.neigh.sit0.proxy_delay = 80
net.ipv6.neigh.sit0.proxy_qlen = 64
net.ipv6.neigh.sit0.retrans_time_ms = 1000
net.ipv6.neigh.sit0.ucast_solicit = 3
net.ipv6.neigh.sit0.unres_qlen = 31
net.ipv6.neigh.sit0.unres_qlen_bytes = 65536
net.ipv6.route.gc_elasticity = 9
net.ipv6.route.gc_interval = 30
net.ipv6.route.gc_min_interval = 0
net.ipv6.route.gc_min_interval_ms = 500
net.ipv6.route.gc_thresh = 1024
net.ipv6.route.gc_timeout = 60
net.ipv6.route.max_size = 4096
net.ipv6.route.min_adv_mss = 1220
net.ipv6.route.mtu_expires = 600
net.ipv6.xfrm6_gc_thresh = 1024
net.netfilter.nf_conntrack_acct = 0
net.netfilter.nf_conntrack_buckets = 8192
net.netfilter.nf_conntrack_checksum = 1
net.netfilter.nf_conntrack_count = 165
net.netfilter.nf_conntrack_expect_max = 124
net.netfilter.nf_conntrack_frag6_high_thresh = 4194304
net.netfilter.nf_conntrack_frag6_low_thresh = 3145728
net.netfilter.nf_conntrack_frag6_timeout = 60
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_helper = 1
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_icmpv6_timeout = 30
net.netfilter.nf_conntrack_log_invalid = 0
net.netfilter.nf_conntrack_max = 31900
net.netfilter.nf_conntrack_tcp_be_liberal = 0
net.netfilter.nf_conntrack_tcp_loose = 1
net.netfilter.nf_conntrack_tcp_max_retrans = 3
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 180
net.netfilter.nf_log.0 = NONE
net.netfilter.nf_log.1 = NONE
net.netfilter.nf_log.10 = NONE
net.netfilter.nf_log.11 = NONE
net.netfilter.nf_log.12 = NONE
net.netfilter.nf_log.2 = ipt_ULOG
net.netfilter.nf_log.3 = NONE
net.netfilter.nf_log.4 = NONE
net.netfilter.nf_log.5 = NONE
net.netfilter.nf_log.6 = NONE
net.netfilter.nf_log.7 = NONE
net.netfilter.nf_log.8 = NONE
net.netfilter.nf_log.9 = NONE
net.nf_conntrack_max = 31900
net.unix.max_dgram_qlen = 10
vm.admin_reserve_kbytes = 8192
vm.block_dump = 0
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.drop_caches = 0
vm.hugepages_treat_as_movable = 0
vm.hugetlb_shm_group = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 256	256	32
vm.max_map_count = 65530
vm.memory_failure_early_kill = 0
vm.memory_failure_recovery = 1
vm.min_free_kbytes = 5120
vm.mmap_min_addr = 4096
vm.nr_hugepages = 0
vm.nr_overcommit_hugepages = 0
vm.nr_pdflush_threads = 0
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.scan_unevictable_pages = 0
vm.stat_interval = 1
vm.swappiness = 60
vm.user_reserve_kbytes = 31797
vm.vfs_cache_pressure = 100

Re: 3.10.0 network trace

[Yuchung Cheng]

On Fri, Jul 12, 2013 at 2:33 PM, Michael Sterrett <michael@sterretts.net> wrote:
> Thanks for the reply.  The sysctl -a output is attached.

Hi Michael: my guess is some of the recent changes like TLP or F-RTO
rewrite has bugs. It'd be best if you can provide tcp traces that
triggers this. If not and this WARN appears frequently enough, please
try i) sysctl tcp_early_retrans=2 ii) sysctl tcp_frto=0 and see if
that WARN goes away. Thanks.

Re: 3.10.0 network trace

[Michael Sterrett]

After changing those two sysctl settings the WARN hasn't reappeared.
I'm willing to mess around with this box though if you want to try to
narrow down the root cause with different settings and/or patches.

On Fri, Jul 12, 2013 at 5:39 PM, Yuchung Cheng <ycheng@google.com> wrote:
> On Fri, Jul 12, 2013 at 2:33 PM, Michael Sterrett <michael@sterretts.net> wrote:
>> Thanks for the reply.  The sysctl -a output is attached.
>
> Hi Michael: my guess is some of the recent changes like TLP or F-RTO
> rewrite has bugs. It'd be best if you can provide tcp traces that
> triggers this. If not and this WARN appears frequently enough, please
> try i) sysctl tcp_early_retrans=2 ii) sysctl tcp_frto=0 and see if
> that WARN goes away. Thanks.

Re: 3.10.0 network trace

[Yuchung Cheng]

On Sat, Jul 13, 2013 at 2:10 PM, Michael Sterrett <michael@sterretts.net> wrote:
> After changing those two sysctl settings the WARN hasn't reappeared.
> I'm willing to mess around with this box though if you want to try to
> narrow down the root cause with different settings and/or patches.
Could you try to disable F-RTO but enable loss probe by
sysctl net.ipv4.tcp_frto=0 net.ipv4.tcp_early_retrans=3
to help us identify if this is loss probe bug or f-rto. Thanks.

>
> On Fri, Jul 12, 2013 at 5:39 PM, Yuchung Cheng <ycheng@google.com> wrote:
>> On Fri, Jul 12, 2013 at 2:33 PM, Michael Sterrett <michael@sterretts.net> wrote:
>>> Thanks for the reply.  The sysctl -a output is attached.
>>
>> Hi Michael: my guess is some of the recent changes like TLP or F-RTO
>> rewrite has bugs. It'd be best if you can provide tcp traces that
>> triggers this. If not and this WARN appears frequently enough, please
>> try i) sysctl tcp_early_retrans=2 ii) sysctl tcp_frto=0 and see if
>> that WARN goes away. Thanks.

Re: 3.10.0 network trace

[Michael Sterrett]

Since upgraded to 3.10.1 which exhibits the same issue (probably not
surprising).

sysctl net.ipv4.tcp_frto=0 net.ipv4.tcp_early_retrans=3 seems to take
care of it.

Re: 3.10.0 network trace

[Josh Boyer]

On Sun, Jul 21, 2013 at 9:36 PM, Michael Sterrett <michael@sterretts.net> wrote:
> Since upgraded to 3.10.1 which exhibits the same issue (probably not
> surprising).
>
> sysctl net.ipv4.tcp_frto=0 net.ipv4.tcp_early_retrans=3 seems to take
> care of it.

We're still getting reports of this with 3.10.10 in Fedora [1].  At
least one of the reporters has said those sysctl settings didn't help.

Is there anything else that can be gathered to help track this down?

josh

[1] https://bugzilla.redhat.com/show_bug.cgi?id=989251

Re: 3.10.0 network trace

[Yuchung Cheng]

[-- Attachment #1: Type: text/plain, Size: 1080 bytes --]

On Wed, Sep 11, 2013 at 5:49 AM, Josh Boyer <jwboyer@fedoraproject.org> wrote:
> On Sun, Jul 21, 2013 at 9:36 PM, Michael Sterrett <michael@sterretts.net> wrote:
>> Since upgraded to 3.10.1 which exhibits the same issue (probably not
>> surprising).
>>
>> sysctl net.ipv4.tcp_frto=0 net.ipv4.tcp_early_retrans=3 seems to take
>> care of it.
>
> We're still getting reports of this with 3.10.10 in Fedora [1].  At
> least one of the reporters has said those sysctl settings didn't help.
>
> Is there anything else that can be gathered to help track this down?
since frto=0 didn't help, I suspect it's either a bug introduced in
the tail loss probe patch 9b717a8d (tcp: TLP loss detection) or other
changes related loss recovery.

could you try disable TLP by
sysctl net.ipv4.tcp_frto=0 net.ipv4.tcp_early_retrans=1

if it still does not work. disable any form of early retransmit by
sysctl net.ipv4.tcp_frto=0 net.ipv4.tcp_early_retrans=0

if you can test a custom build kernel, please try this debugging patch.

>
> josh
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=989251

[-- Attachment #2: 0001-tcp-debug-fastretrans-warning.patch --]
[-- Type: application/octet-stream, Size: 1525 bytes --]

From 62561c91d4655539781113e0fa04ee681b22a2a4 Mon Sep 17 00:00:00 2001
From: Yuchung Cheng <ycheng@google.com>
Date: Wed, 31 Jul 2013 09:22:35 -0700
Subject: [PATCH] tcp: debug fastretrans warning

printk a ton of states to debug fastretrans warning

Signed-off-by: Yuchung Cheng <ycheng@google.com>
---
 net/ipv4/tcp_input.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 9c62257..c18dab1 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -2773,7 +2773,20 @@ static void tcp_fastretrans_alert(struct sock *sk, int pkts_acked,
 	/* D. Check state exit conditions. State can be terminated
 	 *    when high_seq is ACKed. */
 	if (icsk->icsk_ca_state == TCP_CA_Open) {
-		WARN_ON(tp->retrans_out != 0);
+		if (WARN_ON(tp->retrans_out != 0)) {
+			printk(KERN_DEBUG "%pI4:%u F0x%x S%u s%d IF%u+%u-%u-%u"
+			       "f%u ur%u rr%u rt%u um%u hs%u nxt%u\n",
+			       &inet_sk(sk)->inet_daddr,
+			       ntohs(inet_sk(sk)->inet_dport),
+			       flag, sk->sk_state, tp->rx_opt.sack_ok,
+			       tp->packets_out, tp->retrans_out,
+			       tp->sacked_out, tp->lost_out,
+			       tp->frto, tp->undo_retrans,
+			       tp->reordering, icsk->icsk_retransmits,
+			       tp->undo_marker ? tp->undo_marker-tp->snd_una:0,
+			       tp->high_seq - tp->snd_una,
+			       tp->snd_nxt - tp->snd_una);
+		}
 		tp->retrans_stamp = 0;
 	} else if (!before(tp->snd_una, tp->high_seq)) {
 		switch (icsk->icsk_ca_state) {
-- 
1.8.3

Re: 3.10.0 network trace

[Josh Boyer]

On Wed, Sep 11, 2013 at 12:32 PM, Yuchung Cheng <ycheng@google.com> wrote:
> On Wed, Sep 11, 2013 at 5:49 AM, Josh Boyer <jwboyer@fedoraproject.org> wrote:
>> On Sun, Jul 21, 2013 at 9:36 PM, Michael Sterrett <michael@sterretts.net> wrote:
>>> Since upgraded to 3.10.1 which exhibits the same issue (probably not
>>> surprising).
>>>
>>> sysctl net.ipv4.tcp_frto=0 net.ipv4.tcp_early_retrans=3 seems to take
>>> care of it.
>>
>> We're still getting reports of this with 3.10.10 in Fedora [1].  At
>> least one of the reporters has said those sysctl settings didn't help.
>>
>> Is there anything else that can be gathered to help track this down?
> since frto=0 didn't help, I suspect it's either a bug introduced in
> the tail loss probe patch 9b717a8d (tcp: TLP loss detection) or other
> changes related loss recovery.
>
> could you try disable TLP by
> sysctl net.ipv4.tcp_frto=0 net.ipv4.tcp_early_retrans=1
>
> if it still does not work. disable any form of early retransmit by
> sysctl net.ipv4.tcp_frto=0 net.ipv4.tcp_early_retrans=0

Thanks.  I've asked the people in the bug to try those settings.

> if you can test a custom build kernel, please try this debugging patch.

I'll get the patch included in a test build for the tomorrow as well.

josh