* Possible bug in TCP retry logic/Kernel crash @ 2019-11-10 5:59 Avinash Patil 2019-11-10 23:47 ` Eric Dumazet 0 siblings, 1 reply; 8+ messages in thread From: Avinash Patil @ 2019-11-10 5:59 UTC (permalink / raw) To: netdev Hi everyone, Kernel: Linux 4.19.35 kernel built from linux-stable I am seeing this issue on our platform and suspect this is TCP issue: [ 3148.796319] Oops [ 3148.799789] Path: /usr/bin/qtn_dut [ 3148.803306] CPU: 0 PID: 1341 Comm: qtn_dut Tainted: P O 4.19.35 #4 [ 3148.810876] [ 3148.810876] [ECR ]: 0x00220100 => Invalid Read @ 0x00000008 by insn @ 0x8b1bc7e8 [ 3148.820064] [EFA ]: 0x00000008 [ 3148.820064] [BLINK ]: tcp_try_coalesce+0x3c/0xf0 [ 3148.820064] [ERET ]: skb_try_coalesce+0x94/0x3a0 [ 3148.832704] [STAT32]: 0x00000206 : K E2 E1 [ 3148.837677] BTA: 0x8b309ca3 SP: 0x8c92db44 FP: 0x00000000 [ 3148.843338] LPS: 0x8b304b94 LPE: 0x8b304b9c LPC: 0x00000000 [ 3148.849023] r00: 0x8c8743c0 r01: 0x8c92a0e0 r02: 0x8c92dbaa [ 3148.849023] r03: 0x00000000 r04: 0x40000214 r05: 0x8b221ab8 [ 3148.849023] r06: 0x8bab8f3d r07: 0x00000000 r08: 0x00000000 [ 3148.849023] r09: 0x00000000 r10: 0x1f4f9e47 r11: 0x00000000 [ 3148.849023] r12: 0x00000000 r13: 0x8afcecfc r14: 0x000d2bb8 [ 3148.849023] r15: 0x5682fbc0 r16: 0xffffffff r17: 0x00000000 [ 3148.849023] r18: 0x00000001 r19: 0x5682faa4 r20: 0x5682fa84 [ 3148.849023] r21: 0x5682fa64 r22: 0x5682fb30 r23: 0x00000020 [ 3148.849023] r24: 0x000d2bb8 r25: 0x5682fbc0 [ 3148.849023] [ 3148.849023] [ 3148.901689] [ 3148.901689] Stack Trace: [ 3148.905781] Firmware build version: AAA [ 3148.905781] Firmware configuration: BBB [ 3148.905781] Hardware ID : CCC [ 3148.920879] skb_try_coalesce+0x94/0x3a0 [ 3148.925026] tcp_try_coalesce+0x3c/0xf0 [ 3148.929079] tcp_queue_rcv+0x44/0x164 [ 3148.932953] tcp_data_queue+0x32a/0x75c [ 3148.936946] tcp_rcv_established+0x37e/0x7d4 [ 3148.941438] tcp_v4_do_rcv+0xda/0x120 [ 3148.945320] tcp_v4_rcv+0x8f2/0xa04 [ 3148.949034] ip_local_deliver+0x72/0x208 [ 3148.953179] process_backlog+0xbe/0x1b0 [ 3148.957169] net_rx_action+0xfe/0x27c [ 3148.961057] __do_softirq+0xf0/0x228 [ 3148.964863] __local_bh_enable_ip+0xae/0xb4 [ 3148.969277] ip_finish_output2.constprop.6+0x116/0x368 [ 3148.974641] __tcp_transmit_skb+0x56e/0xb3c [ 3148.979039] tcp_write_xmit+0x34a/0x126c [ 3148.983174] __tcp_push_pending_frames+0x28/0x94 [ 3148.987992] tcp_sendmsg_locked+0xa7a/0xc14 [ 3148.992386] tcp_sendmsg+0x1e/0x34 [ 3148.995935] __sys_sendto+0xc8/0xf4 [ 3148.999642] EV_Trap+0x11c/0x120 [ 3149.003057] Conditions under which this happens: There are 2 processes running on platform which communicate with TCP sockets- P1 and P2. 1. P1 has 2 TCP sockets- one TCP client to communicate with P2 while another TCP server to listen to client running on another machine. 2. P1 has issued command to P2 and P2 is preparing response. 3. While P2 is preparing response, P1 receives zero sized packet from remote server and closes its server socket treating this as error. Note: client socket is open/active. P2 prepares its response but its buffered 4. P1 respawns server socket and issues another command to P2 and waits for response. 5. P2 now sends 2 sets of data- one for old session and one response for current command. I see kernel panic with backtrace as above. There is another symptom of this issue : # [ 194.416963] Alignment trap: fault in fix-up 0000a260 at [<00000001>] [ 194.423419] [ 194.423419] Misaligned Access [ 194.427950] Path: (null) [ 194.430517] CPU: 0 PID: 0 Comm: swapper Tainted: P O 4.19.35 #3 [ 194.437816] [ 194.437816] [ECR ]: 0x00230400 => Misaligned r/w from 0x00000001 [ 194.445597] [EFA ]: 0x00000001 [ 194.445597] [BLINK ]: tcp_ack+0x5e6/0x1598 [ 194.445597] [ERET ]: tcp_ack+0x606/0x1598 [ 194.457087] [STAT32]: 0x0000020e : K A1 E2 E1 [ 194.462137] BTA: 0x8b3bf3b7 SP: 0x8b4c9c04 FP: 0x00000000 [ 194.467777] LPS: 0x8b3ba37c LPE: 0x8b3ba384 LPC: 0x00000000 [ 194.473454] r00: 0x8ce503c0 r01: 0x8f3b34e4 r02: 0x00000001 [ 194.473454] r03: 0xa0076e71 r04: 0x00000000 r05: 0x06e0ed35 [ 194.473454] r06: 0x8f3b3800 r07: 0x00000000 r08: 0x0b9362f8 [ 194.473454] r09: 0x00000000 r10: 0x00032e0c r11: 0x00000000 [ 194.473454] r12: 0xefec0000 r13: 0x8f3b3400 r14: 0x8f3b3c00 [ 194.473454] r15: 0x8ce503c0 r16: 0x00000001 r17: 0x8f3b3800 [ 194.473454] r18: 0x00000000 r19: 0x00000000 r20: 0x66c2443f [ 194.473454] r21: 0x8b4c9c80 r22: 0x00000004 r23: 0x00000001 [ 194.473454] r24: 0x00000000 r25: 0x8b4cb2e0 [ 194.473454] [ 194.473454] [ 194.526128] [ 194.526128] Stack Trace: [ 194.530209] [ 194.530209] Firmware build version: pyang_sh-swbuild04_main2ac-cl101263 [ 194.530216] [ 194.530216] Firmware configuration: pearl_10gax_config [ 194.538389] [ 194.538389] Hardware ID : 65535 [ 194.550632] tcp_ack+0x606/0x1598 [ 194.554160] tcp_rcv_established+0x458/0x7d4 [ 194.558646] tcp_v4_do_rcv+0xda/0x120 [ 194.562521] tcp_v4_rcv+0x8f2/0xa04 [ 194.566162] ip_local_deliver+0x72/0x208 [ 194.570287] netif_receive_skb+0x62/0x104 [ 194.574510] br_handle_frame_finish.constprop.2+0x1a6/0x270 [ 194.580299] br_handle_frame+0x170/0x2a0 [ 194.584427] __netif_receive_skb_core+0x156/0x650 [ 194.589346] netif_receive_skb+0x50/0x104 [ 194.593582] wowlan_magic_packet_check+0xc68/0x16b8 [switch_tqe] [ 194.599825] net_rx_action+0xfe/0x27c [ 194.603695] __do_softirq+0xf0/0x228 [ 194.607477] __handle_domain_irq+0x5c/0x98 [ 194.611732] handle_interrupt_level1+0xcc/0xd8 Do you happen to know if this is already reported/fixed? I can run more experiments/gather more debug data/stats if required. Thanks in advance. -Avinash ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Possible bug in TCP retry logic/Kernel crash 2019-11-10 5:59 Possible bug in TCP retry logic/Kernel crash Avinash Patil @ 2019-11-10 23:47 ` Eric Dumazet 2019-11-11 2:20 ` Avinash Patil 2019-11-16 1:52 ` Avinash Patil 0 siblings, 2 replies; 8+ messages in thread From: Eric Dumazet @ 2019-11-10 23:47 UTC (permalink / raw) To: Avinash Patil, netdev On 11/9/19 9:59 PM, Avinash Patil wrote: > Hi everyone, > > Kernel: Linux 4.19.35 kernel built from linux-stable > This is quite an old version. Please upgrade to the latest one. $ git log --oneline v4.19.35..v4.19.82 -- net/ipv4/tcp*c 3fdcf6a88ded2bb5c3c0f0aabaff253dd3564013 tcp: better handle TCP_USER_TIMEOUT in SYN_SENT state 67fe3b94a833779caf4504ececa7097fba9b2627 tcp: fix tcp_ecn_withdraw_cwr() to clear TCP_ECN_QUEUE_CWR 5977bc19ce7f1ed25bf20d09d8e93e56873a9abb tcp: remove empty skb from write queue in error cases 6f3126379879bb2b9148174f0a4b6b65e04dede9 tcp: inherit timestamp on mtu probe 1b200acde418f4d6d87279d3f6f976ebf188f272 tcp: Reset bytes_acked and bytes_received when disconnecting c60f57dfe995172c2f01e59266e3ffa3419c6cd9 tcp: fix tcp_set_congestion_control() use from bpf hook 6323c238bb4374d1477348cfbd5854f2bebe9a21 tcp: be more careful in tcp_fragment() dad3a9314ac95dedc007bc7dacacb396ea10e376 tcp: refine memory limit test in tcp_fragment() 59222807fcc99951dc769cd50e132e319d73d699 tcp: enforce tcp_min_snd_mss in tcp_mtu_probing() 7f9f8a37e563c67b24ccd57da1d541a95538e8d9 tcp: add tcp_min_snd_mss sysctl ec83921899a571ad70d582934ee9e3e07f478848 tcp: tcp_fragment() should apply sane memory limits c09be31461ed140976c60a87364415454a2c3d42 tcp: limit payload size of sacked skbs 6728c6174a47b8a04ceec89aca9e1195dee7ff6b tcp: tcp_grow_window() needs to respect tcp_space() ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Possible bug in TCP retry logic/Kernel crash 2019-11-10 23:47 ` Eric Dumazet @ 2019-11-11 2:20 ` Avinash Patil 2019-11-16 1:52 ` Avinash Patil 1 sibling, 0 replies; 8+ messages in thread From: Avinash Patil @ 2019-11-11 2:20 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev Thanks for the response, Eric. I have tested 4.19.80 as well and I see same issue there. I can run more tests(for additional data/logs) if needed- let me know. Best, Avinash On Sun, Nov 10, 2019 at 3:48 PM Eric Dumazet <eric.dumazet@gmail.com> wrote: > > > > On 11/9/19 9:59 PM, Avinash Patil wrote: > > Hi everyone, > > > > Kernel: Linux 4.19.35 kernel built from linux-stable > > > > This is quite an old version. > > Please upgrade to the latest one. > > $ git log --oneline v4.19.35..v4.19.82 -- net/ipv4/tcp*c > 3fdcf6a88ded2bb5c3c0f0aabaff253dd3564013 tcp: better handle TCP_USER_TIMEOUT in SYN_SENT state > 67fe3b94a833779caf4504ececa7097fba9b2627 tcp: fix tcp_ecn_withdraw_cwr() to clear TCP_ECN_QUEUE_CWR > 5977bc19ce7f1ed25bf20d09d8e93e56873a9abb tcp: remove empty skb from write queue in error cases > 6f3126379879bb2b9148174f0a4b6b65e04dede9 tcp: inherit timestamp on mtu probe > 1b200acde418f4d6d87279d3f6f976ebf188f272 tcp: Reset bytes_acked and bytes_received when disconnecting > c60f57dfe995172c2f01e59266e3ffa3419c6cd9 tcp: fix tcp_set_congestion_control() use from bpf hook > 6323c238bb4374d1477348cfbd5854f2bebe9a21 tcp: be more careful in tcp_fragment() > dad3a9314ac95dedc007bc7dacacb396ea10e376 tcp: refine memory limit test in tcp_fragment() > 59222807fcc99951dc769cd50e132e319d73d699 tcp: enforce tcp_min_snd_mss in tcp_mtu_probing() > 7f9f8a37e563c67b24ccd57da1d541a95538e8d9 tcp: add tcp_min_snd_mss sysctl > ec83921899a571ad70d582934ee9e3e07f478848 tcp: tcp_fragment() should apply sane memory limits > c09be31461ed140976c60a87364415454a2c3d42 tcp: limit payload size of sacked skbs > 6728c6174a47b8a04ceec89aca9e1195dee7ff6b tcp: tcp_grow_window() needs to respect tcp_space() > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Possible bug in TCP retry logic/Kernel crash 2019-11-10 23:47 ` Eric Dumazet 2019-11-11 2:20 ` Avinash Patil @ 2019-11-16 1:52 ` Avinash Patil 2019-11-16 2:01 ` Eric Dumazet 1 sibling, 1 reply; 8+ messages in thread From: Avinash Patil @ 2019-11-16 1:52 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev Hi Eric, I integrated all patches till 4.19.83 as per your suggestion. Unfortunately, I still see crash. [ 991.291968] [ECR ]: 0x00230400 => Misaligned r/w from 0x00000001 [ 991.299766] [EFA ]: 0x00000001 [ 991.299766] [BLINK ]: tcp_clean_rtx_queue+0x2a4/0xc88 [ 991.299766] [ERET ]: __list_del_entry_valid+0x12/0x1a0 [ 991.313350] [STAT32]: 0x00000206 : K E2 E1 [ 991.318323] BTA: 0x8b188843 SP: 0x8f04fcf4 FP: 0x00000000 [ 991.323977] LPS: 0x8b2e2942 LPE: 0x8b2e2946 LPC: 0x00000000 [ 991.329660] r00: 0x8cbf2224 r01: 0x8cbf21c0 r02: 0x6e6c53fe [ 991.329660] r03: 0x00000001 r04: 0x00000000 r05: 0x39ef6f1d [ 991.329660] r06: 0x8f3e3180 r07: 0x00000000 r08: 0x3b14d54c [ 991.329660] r09: 0x00000000 r10: 0x0000c0ef r11: 0x00000000 [ 991.329660] r12: 0x04c80000 r13: 0x8cbf21c0 r14: 0x00000000 [ 991.329660] r15: 0x00000000 r16: 0x00000001 r17: 0x00000000 [ 991.329660] r18: 0x8f3e3580 r19: 0x8f3e2d80 r20: 0x00000004 [ 991.329660] r21: 0x8f04fd7c r22: 0x00000001 r23: 0x3b14d54c [ 991.329660] r24: 0x00000000 r25: 0x8f0315c0 [ 991.329660] [ 991.329660] [ 991.382333] [ 991.382333] Stack Trace: [ 991.386374] Firmware build version: avinashp6_bbic5_a-cl103643 [ 991.386374] Firmware configuration: pearl_10gax_config [ 991.386374] Hardware ID : 65535 [ 991.401461] __list_del_entry_valid+0x12/0x1a0 [ 991.406125] tcp_clean_rtx_queue+0x2a4/0xc88 [ 991.410618] tcp_ack+0x484/0x914 [ 991.414073] tcp_rcv_established+0x538/0x724 [ 991.418564] tcp_v4_do_rcv+0xda/0x120 [ 991.422456] tcp_v4_rcv+0x954/0xa7c [ 991.426105] ip_local_deliver+0x72/0x208 [ 991.430257] process_backlog+0xbe/0x1b0 [ 991.434319] net_rx_action+0x106/0x294 [ 991.438286] __do_softirq+0xf0/0x218 [ 991.442014] run_ksoftirqd+0x2a/0x3c [ 991.445812] smpboot_thread_fn+0xb4/0x10c [ 991.450054] kthread+0xd8/0xdc [ 991.453338] ret_from_fork+0x18/0x1c [ 991.457067] Thank you! -Avinash On Sun, Nov 10, 2019 at 3:48 PM Eric Dumazet <eric.dumazet@gmail.com> wrote: > > > > On 11/9/19 9:59 PM, Avinash Patil wrote: > > Hi everyone, > > > > Kernel: Linux 4.19.35 kernel built from linux-stable > > > > This is quite an old version. > > Please upgrade to the latest one. > > $ git log --oneline v4.19.35..v4.19.82 -- net/ipv4/tcp*c > 3fdcf6a88ded2bb5c3c0f0aabaff253dd3564013 tcp: better handle TCP_USER_TIMEOUT in SYN_SENT state > 67fe3b94a833779caf4504ececa7097fba9b2627 tcp: fix tcp_ecn_withdraw_cwr() to clear TCP_ECN_QUEUE_CWR > 5977bc19ce7f1ed25bf20d09d8e93e56873a9abb tcp: remove empty skb from write queue in error cases > 6f3126379879bb2b9148174f0a4b6b65e04dede9 tcp: inherit timestamp on mtu probe > 1b200acde418f4d6d87279d3f6f976ebf188f272 tcp: Reset bytes_acked and bytes_received when disconnecting > c60f57dfe995172c2f01e59266e3ffa3419c6cd9 tcp: fix tcp_set_congestion_control() use from bpf hook > 6323c238bb4374d1477348cfbd5854f2bebe9a21 tcp: be more careful in tcp_fragment() > dad3a9314ac95dedc007bc7dacacb396ea10e376 tcp: refine memory limit test in tcp_fragment() > 59222807fcc99951dc769cd50e132e319d73d699 tcp: enforce tcp_min_snd_mss in tcp_mtu_probing() > 7f9f8a37e563c67b24ccd57da1d541a95538e8d9 tcp: add tcp_min_snd_mss sysctl > ec83921899a571ad70d582934ee9e3e07f478848 tcp: tcp_fragment() should apply sane memory limits > c09be31461ed140976c60a87364415454a2c3d42 tcp: limit payload size of sacked skbs > 6728c6174a47b8a04ceec89aca9e1195dee7ff6b tcp: tcp_grow_window() needs to respect tcp_space() > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Possible bug in TCP retry logic/Kernel crash 2019-11-16 1:52 ` Avinash Patil @ 2019-11-16 2:01 ` Eric Dumazet 2019-11-16 2:07 ` Avinash Patil 0 siblings, 1 reply; 8+ messages in thread From: Eric Dumazet @ 2019-11-16 2:01 UTC (permalink / raw) To: Avinash Patil; +Cc: netdev On 11/15/19 5:52 PM, Avinash Patil wrote: > Hi Eric, > > I integrated all patches till 4.19.83 as per your suggestion. > Unfortunately, I still see crash. What do you mean by integrating patches ? Are you using an official/pristine tree ? > > [ 991.291968] [ECR ]: 0x00230400 => Misaligned r/w from 0x00000001 > [ 991.299766] [EFA ]: 0x00000001 > [ 991.299766] [BLINK ]: tcp_clean_rtx_queue+0x2a4/0xc88 > [ 991.299766] [ERET ]: __list_del_entry_valid+0x12/0x1a0 > [ 991.313350] [STAT32]: 0x00000206 : K E2 E1 > [ 991.318323] BTA: 0x8b188843 SP: 0x8f04fcf4 FP: 0x00000000 > [ 991.323977] LPS: 0x8b2e2942 LPE: 0x8b2e2946 LPC: 0x00000000 > [ 991.329660] r00: 0x8cbf2224 r01: 0x8cbf21c0 r02: 0x6e6c53fe > [ 991.329660] r03: 0x00000001 r04: 0x00000000 r05: 0x39ef6f1d > [ 991.329660] r06: 0x8f3e3180 r07: 0x00000000 r08: 0x3b14d54c > [ 991.329660] r09: 0x00000000 r10: 0x0000c0ef r11: 0x00000000 > [ 991.329660] r12: 0x04c80000 r13: 0x8cbf21c0 r14: 0x00000000 > [ 991.329660] r15: 0x00000000 r16: 0x00000001 r17: 0x00000000 > [ 991.329660] r18: 0x8f3e3580 r19: 0x8f3e2d80 r20: 0x00000004 > [ 991.329660] r21: 0x8f04fd7c r22: 0x00000001 r23: 0x3b14d54c > [ 991.329660] r24: 0x00000000 r25: 0x8f0315c0 > [ 991.329660] > [ 991.329660] > [ 991.382333] > [ 991.382333] Stack Trace: > [ 991.386374] Firmware build version: avinashp6_bbic5_a-cl103643 > [ 991.386374] Firmware configuration: pearl_10gax_config > [ 991.386374] Hardware ID : 65535 > [ 991.401461] __list_del_entry_valid+0x12/0x1a0 > [ 991.406125] tcp_clean_rtx_queue+0x2a4/0xc88 > [ 991.410618] tcp_ack+0x484/0x914 > [ 991.414073] tcp_rcv_established+0x538/0x724 > [ 991.418564] tcp_v4_do_rcv+0xda/0x120 > [ 991.422456] tcp_v4_rcv+0x954/0xa7c > [ 991.426105] ip_local_deliver+0x72/0x208 > [ 991.430257] process_backlog+0xbe/0x1b0 > [ 991.434319] net_rx_action+0x106/0x294 > [ 991.438286] __do_softirq+0xf0/0x218 > [ 991.442014] run_ksoftirqd+0x2a/0x3c > [ 991.445812] smpboot_thread_fn+0xb4/0x10c > [ 991.450054] kthread+0xd8/0xdc > [ 991.453338] ret_from_fork+0x18/0x1c > [ 991.457067] > > Thank you! > > -Avinash > > On Sun, Nov 10, 2019 at 3:48 PM Eric Dumazet <eric.dumazet@gmail.com> wrote: >> >> >> >> On 11/9/19 9:59 PM, Avinash Patil wrote: >>> Hi everyone, >>> >>> Kernel: Linux 4.19.35 kernel built from linux-stable >>> >> >> This is quite an old version. >> >> Please upgrade to the latest one. >> >> $ git log --oneline v4.19.35..v4.19.82 -- net/ipv4/tcp*c >> 3fdcf6a88ded2bb5c3c0f0aabaff253dd3564013 tcp: better handle TCP_USER_TIMEOUT in SYN_SENT state >> 67fe3b94a833779caf4504ececa7097fba9b2627 tcp: fix tcp_ecn_withdraw_cwr() to clear TCP_ECN_QUEUE_CWR >> 5977bc19ce7f1ed25bf20d09d8e93e56873a9abb tcp: remove empty skb from write queue in error cases >> 6f3126379879bb2b9148174f0a4b6b65e04dede9 tcp: inherit timestamp on mtu probe >> 1b200acde418f4d6d87279d3f6f976ebf188f272 tcp: Reset bytes_acked and bytes_received when disconnecting >> c60f57dfe995172c2f01e59266e3ffa3419c6cd9 tcp: fix tcp_set_congestion_control() use from bpf hook >> 6323c238bb4374d1477348cfbd5854f2bebe9a21 tcp: be more careful in tcp_fragment() >> dad3a9314ac95dedc007bc7dacacb396ea10e376 tcp: refine memory limit test in tcp_fragment() >> 59222807fcc99951dc769cd50e132e319d73d699 tcp: enforce tcp_min_snd_mss in tcp_mtu_probing() >> 7f9f8a37e563c67b24ccd57da1d541a95538e8d9 tcp: add tcp_min_snd_mss sysctl >> ec83921899a571ad70d582934ee9e3e07f478848 tcp: tcp_fragment() should apply sane memory limits >> c09be31461ed140976c60a87364415454a2c3d42 tcp: limit payload size of sacked skbs >> 6728c6174a47b8a04ceec89aca9e1195dee7ff6b tcp: tcp_grow_window() needs to respect tcp_space() >> ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Possible bug in TCP retry logic/Kernel crash 2019-11-16 2:01 ` Eric Dumazet @ 2019-11-16 2:07 ` Avinash Patil 2019-11-16 2:29 ` Eric Dumazet 0 siblings, 1 reply; 8+ messages in thread From: Avinash Patil @ 2019-11-16 2:07 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev No; I am not using an official tree. There are certain changes to arch/arc directory which are specific to Synposys ARC platform. However, there are no changes to TCP module and this is why I suspect crash is generic TCP issue. Thanks, Avinash On Fri, Nov 15, 2019 at 6:01 PM Eric Dumazet <eric.dumazet@gmail.com> wrote: > > > > On 11/15/19 5:52 PM, Avinash Patil wrote: > > Hi Eric, > > > > I integrated all patches till 4.19.83 as per your suggestion. > > Unfortunately, I still see crash. > > What do you mean by integrating patches ? > > Are you using an official/pristine tree ? > > > > > [ 991.291968] [ECR ]: 0x00230400 => Misaligned r/w from 0x00000001 > > [ 991.299766] [EFA ]: 0x00000001 > > [ 991.299766] [BLINK ]: tcp_clean_rtx_queue+0x2a4/0xc88 > > [ 991.299766] [ERET ]: __list_del_entry_valid+0x12/0x1a0 > > [ 991.313350] [STAT32]: 0x00000206 : K E2 E1 > > [ 991.318323] BTA: 0x8b188843 SP: 0x8f04fcf4 FP: 0x00000000 > > [ 991.323977] LPS: 0x8b2e2942 LPE: 0x8b2e2946 LPC: 0x00000000 > > [ 991.329660] r00: 0x8cbf2224 r01: 0x8cbf21c0 r02: 0x6e6c53fe > > [ 991.329660] r03: 0x00000001 r04: 0x00000000 r05: 0x39ef6f1d > > [ 991.329660] r06: 0x8f3e3180 r07: 0x00000000 r08: 0x3b14d54c > > [ 991.329660] r09: 0x00000000 r10: 0x0000c0ef r11: 0x00000000 > > [ 991.329660] r12: 0x04c80000 r13: 0x8cbf21c0 r14: 0x00000000 > > [ 991.329660] r15: 0x00000000 r16: 0x00000001 r17: 0x00000000 > > [ 991.329660] r18: 0x8f3e3580 r19: 0x8f3e2d80 r20: 0x00000004 > > [ 991.329660] r21: 0x8f04fd7c r22: 0x00000001 r23: 0x3b14d54c > > [ 991.329660] r24: 0x00000000 r25: 0x8f0315c0 > > [ 991.329660] > > [ 991.329660] > > [ 991.382333] > > [ 991.382333] Stack Trace: > > [ 991.386374] Firmware build version: avinashp6_bbic5_a-cl103643 > > [ 991.386374] Firmware configuration: pearl_10gax_config > > [ 991.386374] Hardware ID : 65535 > > [ 991.401461] __list_del_entry_valid+0x12/0x1a0 > > [ 991.406125] tcp_clean_rtx_queue+0x2a4/0xc88 > > [ 991.410618] tcp_ack+0x484/0x914 > > [ 991.414073] tcp_rcv_established+0x538/0x724 > > [ 991.418564] tcp_v4_do_rcv+0xda/0x120 > > [ 991.422456] tcp_v4_rcv+0x954/0xa7c > > [ 991.426105] ip_local_deliver+0x72/0x208 > > [ 991.430257] process_backlog+0xbe/0x1b0 > > [ 991.434319] net_rx_action+0x106/0x294 > > [ 991.438286] __do_softirq+0xf0/0x218 > > [ 991.442014] run_ksoftirqd+0x2a/0x3c > > [ 991.445812] smpboot_thread_fn+0xb4/0x10c > > [ 991.450054] kthread+0xd8/0xdc > > [ 991.453338] ret_from_fork+0x18/0x1c > > [ 991.457067] > > > > Thank you! > > > > -Avinash > > > > On Sun, Nov 10, 2019 at 3:48 PM Eric Dumazet <eric.dumazet@gmail.com> wrote: > >> > >> > >> > >> On 11/9/19 9:59 PM, Avinash Patil wrote: > >>> Hi everyone, > >>> > >>> Kernel: Linux 4.19.35 kernel built from linux-stable > >>> > >> > >> This is quite an old version. > >> > >> Please upgrade to the latest one. > >> > >> $ git log --oneline v4.19.35..v4.19.82 -- net/ipv4/tcp*c > >> 3fdcf6a88ded2bb5c3c0f0aabaff253dd3564013 tcp: better handle TCP_USER_TIMEOUT in SYN_SENT state > >> 67fe3b94a833779caf4504ececa7097fba9b2627 tcp: fix tcp_ecn_withdraw_cwr() to clear TCP_ECN_QUEUE_CWR > >> 5977bc19ce7f1ed25bf20d09d8e93e56873a9abb tcp: remove empty skb from write queue in error cases > >> 6f3126379879bb2b9148174f0a4b6b65e04dede9 tcp: inherit timestamp on mtu probe > >> 1b200acde418f4d6d87279d3f6f976ebf188f272 tcp: Reset bytes_acked and bytes_received when disconnecting > >> c60f57dfe995172c2f01e59266e3ffa3419c6cd9 tcp: fix tcp_set_congestion_control() use from bpf hook > >> 6323c238bb4374d1477348cfbd5854f2bebe9a21 tcp: be more careful in tcp_fragment() > >> dad3a9314ac95dedc007bc7dacacb396ea10e376 tcp: refine memory limit test in tcp_fragment() > >> 59222807fcc99951dc769cd50e132e319d73d699 tcp: enforce tcp_min_snd_mss in tcp_mtu_probing() > >> 7f9f8a37e563c67b24ccd57da1d541a95538e8d9 tcp: add tcp_min_snd_mss sysctl > >> ec83921899a571ad70d582934ee9e3e07f478848 tcp: tcp_fragment() should apply sane memory limits > >> c09be31461ed140976c60a87364415454a2c3d42 tcp: limit payload size of sacked skbs > >> 6728c6174a47b8a04ceec89aca9e1195dee7ff6b tcp: tcp_grow_window() needs to respect tcp_space() > >> ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Possible bug in TCP retry logic/Kernel crash 2019-11-16 2:07 ` Avinash Patil @ 2019-11-16 2:29 ` Eric Dumazet 2019-11-19 22:39 ` Avinash Patil 0 siblings, 1 reply; 8+ messages in thread From: Eric Dumazet @ 2019-11-16 2:29 UTC (permalink / raw) To: Avinash Patil, Eric Dumazet; +Cc: netdev On 11/15/19 6:07 PM, Avinash Patil wrote: > No; I am not using an official tree. There are certain changes to > arch/arc directory which are specific to Synposys ARC platform. > However, there are no changes to TCP module and this is why I suspect > crash is generic TCP issue. Please provide a git tree then, and exact line numbers of the stack trace. Really I can not spend days working on non pristibe kernels, you have to help a bit. Thanks. > > Thanks, > Avinash > > > On Fri, Nov 15, 2019 at 6:01 PM Eric Dumazet <eric.dumazet@gmail.com> wrote: >> >> >> >> On 11/15/19 5:52 PM, Avinash Patil wrote: >>> Hi Eric, >>> >>> I integrated all patches till 4.19.83 as per your suggestion. >>> Unfortunately, I still see crash. >> >> What do you mean by integrating patches ? >> >> Are you using an official/pristine tree ? >> >>> >>> [ 991.291968] [ECR ]: 0x00230400 => Misaligned r/w from 0x00000001 >>> [ 991.299766] [EFA ]: 0x00000001 >>> [ 991.299766] [BLINK ]: tcp_clean_rtx_queue+0x2a4/0xc88 >>> [ 991.299766] [ERET ]: __list_del_entry_valid+0x12/0x1a0 >>> [ 991.313350] [STAT32]: 0x00000206 : K E2 E1 >>> [ 991.318323] BTA: 0x8b188843 SP: 0x8f04fcf4 FP: 0x00000000 >>> [ 991.323977] LPS: 0x8b2e2942 LPE: 0x8b2e2946 LPC: 0x00000000 >>> [ 991.329660] r00: 0x8cbf2224 r01: 0x8cbf21c0 r02: 0x6e6c53fe >>> [ 991.329660] r03: 0x00000001 r04: 0x00000000 r05: 0x39ef6f1d >>> [ 991.329660] r06: 0x8f3e3180 r07: 0x00000000 r08: 0x3b14d54c >>> [ 991.329660] r09: 0x00000000 r10: 0x0000c0ef r11: 0x00000000 >>> [ 991.329660] r12: 0x04c80000 r13: 0x8cbf21c0 r14: 0x00000000 >>> [ 991.329660] r15: 0x00000000 r16: 0x00000001 r17: 0x00000000 >>> [ 991.329660] r18: 0x8f3e3580 r19: 0x8f3e2d80 r20: 0x00000004 >>> [ 991.329660] r21: 0x8f04fd7c r22: 0x00000001 r23: 0x3b14d54c >>> [ 991.329660] r24: 0x00000000 r25: 0x8f0315c0 >>> [ 991.329660] >>> [ 991.329660] >>> [ 991.382333] >>> [ 991.382333] Stack Trace: >>> [ 991.386374] Firmware build version: avinashp6_bbic5_a-cl103643 >>> [ 991.386374] Firmware configuration: pearl_10gax_config >>> [ 991.386374] Hardware ID : 65535 >>> [ 991.401461] __list_del_entry_valid+0x12/0x1a0 >>> [ 991.406125] tcp_clean_rtx_queue+0x2a4/0xc88 >>> [ 991.410618] tcp_ack+0x484/0x914 >>> [ 991.414073] tcp_rcv_established+0x538/0x724 >>> [ 991.418564] tcp_v4_do_rcv+0xda/0x120 >>> [ 991.422456] tcp_v4_rcv+0x954/0xa7c >>> [ 991.426105] ip_local_deliver+0x72/0x208 >>> [ 991.430257] process_backlog+0xbe/0x1b0 >>> [ 991.434319] net_rx_action+0x106/0x294 >>> [ 991.438286] __do_softirq+0xf0/0x218 >>> [ 991.442014] run_ksoftirqd+0x2a/0x3c >>> [ 991.445812] smpboot_thread_fn+0xb4/0x10c >>> [ 991.450054] kthread+0xd8/0xdc >>> [ 991.453338] ret_from_fork+0x18/0x1c >>> [ 991.457067] >>> >>> Thank you! >>> >>> -Avinash >>> >>> On Sun, Nov 10, 2019 at 3:48 PM Eric Dumazet <eric.dumazet@gmail.com> wrote: >>>> >>>> >>>> >>>> On 11/9/19 9:59 PM, Avinash Patil wrote: >>>>> Hi everyone, >>>>> >>>>> Kernel: Linux 4.19.35 kernel built from linux-stable >>>>> >>>> >>>> This is quite an old version. >>>> >>>> Please upgrade to the latest one. >>>> >>>> $ git log --oneline v4.19.35..v4.19.82 -- net/ipv4/tcp*c >>>> 3fdcf6a88ded2bb5c3c0f0aabaff253dd3564013 tcp: better handle TCP_USER_TIMEOUT in SYN_SENT state >>>> 67fe3b94a833779caf4504ececa7097fba9b2627 tcp: fix tcp_ecn_withdraw_cwr() to clear TCP_ECN_QUEUE_CWR >>>> 5977bc19ce7f1ed25bf20d09d8e93e56873a9abb tcp: remove empty skb from write queue in error cases >>>> 6f3126379879bb2b9148174f0a4b6b65e04dede9 tcp: inherit timestamp on mtu probe >>>> 1b200acde418f4d6d87279d3f6f976ebf188f272 tcp: Reset bytes_acked and bytes_received when disconnecting >>>> c60f57dfe995172c2f01e59266e3ffa3419c6cd9 tcp: fix tcp_set_congestion_control() use from bpf hook >>>> 6323c238bb4374d1477348cfbd5854f2bebe9a21 tcp: be more careful in tcp_fragment() >>>> dad3a9314ac95dedc007bc7dacacb396ea10e376 tcp: refine memory limit test in tcp_fragment() >>>> 59222807fcc99951dc769cd50e132e319d73d699 tcp: enforce tcp_min_snd_mss in tcp_mtu_probing() >>>> 7f9f8a37e563c67b24ccd57da1d541a95538e8d9 tcp: add tcp_min_snd_mss sysctl >>>> ec83921899a571ad70d582934ee9e3e07f478848 tcp: tcp_fragment() should apply sane memory limits >>>> c09be31461ed140976c60a87364415454a2c3d42 tcp: limit payload size of sacked skbs >>>> 6728c6174a47b8a04ceec89aca9e1195dee7ff6b tcp: tcp_grow_window() needs to respect tcp_space() >>>> ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Possible bug in TCP retry logic/Kernel crash 2019-11-16 2:29 ` Eric Dumazet @ 2019-11-19 22:39 ` Avinash Patil 0 siblings, 0 replies; 8+ messages in thread From: Avinash Patil @ 2019-11-19 22:39 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev Hi Eric, Thanks for your help so far. I am in no way suggesting you to debug this issue. I have already said in first email itself that I am debugging this myself and looking for pointers from experts for conditions under which this could happen. BTW, I am also checking about the possibility of sharing my changes to kernel sources. Here is another crash I observed today. [ 139.786529] =============== [ 139.789445] tcp_rcv_established: sk=8f3eb400, skb=8f3c0c00, len=47 [ 139.795833] tcp_rcv_established: step5 skb=8f3c0c00 [ 139.800814] tcp_queue_rcv: sk=8f3eb400, skb=8f3c0c00, src-port=39620, len=15 [ 139.808199] =============== [ 139.811033] tcp_rcv_established: sk=8f3eb400, skb=8f3c0900, len=32 [ 139.817409] tcp_rcv_established: slow_path skb=8f3c0900 [ 139.822774] tcp_rcv_established: step5 skb=8f3c0900 [ 139.827753] tcp_queue_rcv: sk=8f3eb400, skb=8f3c0900, src-port=39620, len=0 [ 139.834965] tcp_drop: sk=8f3eb400, skb=8f3c0900 [ 140.846604] TCP: tcp_recvmsg: found a SYN, please report ! [ 140.852198] ------------[ cut here ]------------ [ 140.856948] WARNING: CPU: 0 PID: 1353 at net/ipv4/tcp.c:2039 tcp_recvmsg+0x648/0x9f4 [ 140.864793] TCP recvmsg seq # bug 2: copied F5058AF9, seq 8C96A6D0, rcvnxt F5058C85, fl 0 [ 140.873073] Modules linked in: wlan_scan_ap(O) wlan_acl(O) wlan_gcmp_256(O) wlan_gcmp(O) wlan_ccmp_256(O) wlan_wep(O) wlan_tkip(O) wlan_ccmp(O) wlan_xauth(O) qdrv(O) auc_fw(PO) wlan(O) radar(PO) i2cbu) [ 140.904278] CPU: 0 PID: 1353 Comm: qtn_ca Tainted: P O 4.19.35 #135 [ 140.911948] [ 140.911948] Stack Trace: [ 140.916040] Firmware build version: avinashp6_bbic5_a-cl103746 [ 140.916040] Firmware configuration: pearl_10gax_config [ 140.916040] Hardware ID : 65535 [ 140.931150] arc_unwind_core+0xc8/0xe8 [ 140.935127] __warn+0x9c/0xbc [ 140.938316] warn_slowpath_fmt+0x32/0x3c [ 140.942385] tcp_recvmsg+0x648/0x9f4 [ 140.946185] inet_recvmsg+0x2a/0x3c [ 140.949906] __sys_recvfrom+0xba/0x100 [ 140.953903] EV_Trap+0x11c/0x120 [ 140.957263] ---[ end trace 0e4059a49f3521b9 ]--- [ 140.961983] [ 140.961983] Oops [ 140.965397] Path: /usr/bin/qtn_ca [ 140.968815] CPU: 0 PID: 1353 Comm: qtn_ca Tainted: P W O 4.19.35 #135 [ 140.976473] [ 140.976473] [ECR ]: 0x00220100 => Invalid Read @ 0x00000020 by insn @ 0x8b39d8a0 [ 140.985651] [EFA ]: 0x00000020 [ 140.985651] [BLINK ]: tcp_recvmsg+0x648/0x9f4 [ 140.985651] [ERET ]: tcp_recvmsg+0x654/0x9f4 [ 140.997675] [STAT32]: 0x00000406 : K E2 E1 [ 141.002648] BTA: 0x8b188843 SP: 0x8f32fe28 FP: 0x00000000 [ 141.008305] LPS: 0x8b2e2942 LPE: 0x8b2e2946 LPC: 0x00000000 [ 141.013990] r00: 0x00000009 r01: 0x00000000 r02: 0x8b5c003c [ 141.013990] r03: 0xf5058af9 r04: 0x00001001 r05: 0x00000000 [ 141.013990] r06: 0x00000136 r07: 0x00000000 r08: 0x8f32fc90 [ 141.013990] r09: 0x00000011 r10: 0x00000000 r11: 0x00000000 [ 141.013990] r12: 0x0000002d r13: 0x00000000 r14: 0x00000000 [ 141.013990] r15: 0x00000000 r16: 0x00000000 r17: 0x00000000 [ 141.013990] r18: 0x00000000 r19: 0x00000000 r20: 0x00000000 [ 141.013990] r21: 0x00000000 r22: 0x00000000 r23: 0x00000000 [ 141.013990] r24: 0x00000000 r25: 0x00000000 [ 141.013990] [ 141.013990] [ 141.066650] [ 141.066650] Stack Trace: [ 141.070740] Firmware build version: avinashp6_bbic5_a-cl103746 [ 141.070740] Firmware configuration: pearl_10gax_config [ 141.070740] Hardware ID : 65535 [ 141.085836] tcp_recvmsg+0x654/0x9f4 [ 141.089634] inet_recvmsg+0x2a/0x3c [ 141.093350] __sys_recvfrom+0xba/0x100 [ 141.097314] EV_Trap+0x11c/0x120 [ 141.100773] [ 141.100773] Stack not used for current process: 6200 Thanks, Avinash On Fri, Nov 15, 2019 at 6:29 PM Eric Dumazet <eric.dumazet@gmail.com> wrote: > > > > On 11/15/19 6:07 PM, Avinash Patil wrote: > > No; I am not using an official tree. There are certain changes to > > arch/arc directory which are specific to Synposys ARC platform. > > However, there are no changes to TCP module and this is why I suspect > > crash is generic TCP issue. > > > Please provide a git tree then, and exact line numbers of the stack trace. > > Really I can not spend days working on non pristibe kernels, you have to help a bit. > Thanks. > > > > > Thanks, > > Avinash > > > > > > On Fri, Nov 15, 2019 at 6:01 PM Eric Dumazet <eric.dumazet@gmail.com> wrote: > >> > >> > >> > >> On 11/15/19 5:52 PM, Avinash Patil wrote: > >>> Hi Eric, > >>> > >>> I integrated all patches till 4.19.83 as per your suggestion. > >>> Unfortunately, I still see crash. > >> > >> What do you mean by integrating patches ? > >> > >> Are you using an official/pristine tree ? > >> > >>> > >>> [ 991.291968] [ECR ]: 0x00230400 => Misaligned r/w from 0x00000001 > >>> [ 991.299766] [EFA ]: 0x00000001 > >>> [ 991.299766] [BLINK ]: tcp_clean_rtx_queue+0x2a4/0xc88 > >>> [ 991.299766] [ERET ]: __list_del_entry_valid+0x12/0x1a0 > >>> [ 991.313350] [STAT32]: 0x00000206 : K E2 E1 > >>> [ 991.318323] BTA: 0x8b188843 SP: 0x8f04fcf4 FP: 0x00000000 > >>> [ 991.323977] LPS: 0x8b2e2942 LPE: 0x8b2e2946 LPC: 0x00000000 > >>> [ 991.329660] r00: 0x8cbf2224 r01: 0x8cbf21c0 r02: 0x6e6c53fe > >>> [ 991.329660] r03: 0x00000001 r04: 0x00000000 r05: 0x39ef6f1d > >>> [ 991.329660] r06: 0x8f3e3180 r07: 0x00000000 r08: 0x3b14d54c > >>> [ 991.329660] r09: 0x00000000 r10: 0x0000c0ef r11: 0x00000000 > >>> [ 991.329660] r12: 0x04c80000 r13: 0x8cbf21c0 r14: 0x00000000 > >>> [ 991.329660] r15: 0x00000000 r16: 0x00000001 r17: 0x00000000 > >>> [ 991.329660] r18: 0x8f3e3580 r19: 0x8f3e2d80 r20: 0x00000004 > >>> [ 991.329660] r21: 0x8f04fd7c r22: 0x00000001 r23: 0x3b14d54c > >>> [ 991.329660] r24: 0x00000000 r25: 0x8f0315c0 > >>> [ 991.329660] > >>> [ 991.329660] > >>> [ 991.382333] > >>> [ 991.382333] Stack Trace: > >>> [ 991.386374] Firmware build version: avinashp6_bbic5_a-cl103643 > >>> [ 991.386374] Firmware configuration: pearl_10gax_config > >>> [ 991.386374] Hardware ID : 65535 > >>> [ 991.401461] __list_del_entry_valid+0x12/0x1a0 > >>> [ 991.406125] tcp_clean_rtx_queue+0x2a4/0xc88 > >>> [ 991.410618] tcp_ack+0x484/0x914 > >>> [ 991.414073] tcp_rcv_established+0x538/0x724 > >>> [ 991.418564] tcp_v4_do_rcv+0xda/0x120 > >>> [ 991.422456] tcp_v4_rcv+0x954/0xa7c > >>> [ 991.426105] ip_local_deliver+0x72/0x208 > >>> [ 991.430257] process_backlog+0xbe/0x1b0 > >>> [ 991.434319] net_rx_action+0x106/0x294 > >>> [ 991.438286] __do_softirq+0xf0/0x218 > >>> [ 991.442014] run_ksoftirqd+0x2a/0x3c > >>> [ 991.445812] smpboot_thread_fn+0xb4/0x10c > >>> [ 991.450054] kthread+0xd8/0xdc > >>> [ 991.453338] ret_from_fork+0x18/0x1c > >>> [ 991.457067] > >>> > >>> Thank you! > >>> > >>> -Avinash > >>> > >>> On Sun, Nov 10, 2019 at 3:48 PM Eric Dumazet <eric.dumazet@gmail.com> wrote: > >>>> > >>>> > >>>> > >>>> On 11/9/19 9:59 PM, Avinash Patil wrote: > >>>>> Hi everyone, > >>>>> > >>>>> Kernel: Linux 4.19.35 kernel built from linux-stable > >>>>> > >>>> > >>>> This is quite an old version. > >>>> > >>>> Please upgrade to the latest one. > >>>> > >>>> $ git log --oneline v4.19.35..v4.19.82 -- net/ipv4/tcp*c > >>>> 3fdcf6a88ded2bb5c3c0f0aabaff253dd3564013 tcp: better handle TCP_USER_TIMEOUT in SYN_SENT state > >>>> 67fe3b94a833779caf4504ececa7097fba9b2627 tcp: fix tcp_ecn_withdraw_cwr() to clear TCP_ECN_QUEUE_CWR > >>>> 5977bc19ce7f1ed25bf20d09d8e93e56873a9abb tcp: remove empty skb from write queue in error cases > >>>> 6f3126379879bb2b9148174f0a4b6b65e04dede9 tcp: inherit timestamp on mtu probe > >>>> 1b200acde418f4d6d87279d3f6f976ebf188f272 tcp: Reset bytes_acked and bytes_received when disconnecting > >>>> c60f57dfe995172c2f01e59266e3ffa3419c6cd9 tcp: fix tcp_set_congestion_control() use from bpf hook > >>>> 6323c238bb4374d1477348cfbd5854f2bebe9a21 tcp: be more careful in tcp_fragment() > >>>> dad3a9314ac95dedc007bc7dacacb396ea10e376 tcp: refine memory limit test in tcp_fragment() > >>>> 59222807fcc99951dc769cd50e132e319d73d699 tcp: enforce tcp_min_snd_mss in tcp_mtu_probing() > >>>> 7f9f8a37e563c67b24ccd57da1d541a95538e8d9 tcp: add tcp_min_snd_mss sysctl > >>>> ec83921899a571ad70d582934ee9e3e07f478848 tcp: tcp_fragment() should apply sane memory limits > >>>> c09be31461ed140976c60a87364415454a2c3d42 tcp: limit payload size of sacked skbs > >>>> 6728c6174a47b8a04ceec89aca9e1195dee7ff6b tcp: tcp_grow_window() needs to respect tcp_space() > >>>> ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2019-11-19 22:39 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-11-10 5:59 Possible bug in TCP retry logic/Kernel crash Avinash Patil 2019-11-10 23:47 ` Eric Dumazet 2019-11-11 2:20 ` Avinash Patil 2019-11-16 1:52 ` Avinash Patil 2019-11-16 2:01 ` Eric Dumazet 2019-11-16 2:07 ` Avinash Patil 2019-11-16 2:29 ` Eric Dumazet 2019-11-19 22:39 ` Avinash Patil
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).