* tcp hang when socket fills up ?
@ 2018-04-06 9:07 Dominique Martinet
2018-04-13 9:42 ` Dominique Martinet
` (2 more replies)
0 siblings, 3 replies; 26+ messages in thread
From: Dominique Martinet @ 2018-04-06 9:07 UTC (permalink / raw)
To: netdev
(current kernel: vanilla 4.14.29)
I've been running into troubles recently where a sockets "fills up" (as
in, select() will no longer return it in its outfd / attempting to write
to it after setting it to NONBLOCK will return -EWOULDBLOCK) and it
doesn't seem to ever "unfill" until the tcp connexion timeout.
The previous time I pushed it down to the application for not trying to
read the socket either as I assume the buffers could be shared and
just waiting won't take data out, but this time I'm a bit more
skeptical as socat waits for the fd in both read and write...
Let me take a minute to describe my setup, I don't think that how the
socket was created matters but it might be interesting:
- I have two computers behind NATs, no port forwarding on either side
- One (call it C for client) runs ssh with a proxycommand ncat/socat to
control the source port, e.g.
$ ssh -o ProxyCommand="socat stdio tcp:<server public ip>:<port1>,sourceport=<port2>" server
- The server runs another socat to connect to that and forwards to ssh
locally, e.g.
$ socat tcp:<client public ip>:<port2>,sourceport=<port1> tcp:127.0.0.1:22
(yes, both are connect() calls, and that just works™ thanks to initial
syn replay and conntrack on routers)
When things stall, the first socat is in a select with both fd in
reading, so it's waiting data.
The second socat has data coming from ssh and is in a select with the
client-facing socket in both read and write - that select never returns
so the socket is not available for reading or writing and does not free
up until the connection eventually times out a few minutes later.
At this point, I only see tcp replays in tcpdump/wireshark. I've
compared dumps from both sides and there are no lost packets, only
reordering - there always is a batch of acks that were sent regularily
coming in shortly before the hang. Here's the trace on the server:
16:49:26.735042 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 70476:71850, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
16:49:26.735046 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 71850:73224, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
16:49:26.735334 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 41622, win 918, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
16:49:26.736005 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 42996, win 940, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
16:49:26.736402 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 73224:74598, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
16:49:26.736408 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 74598:75972, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
16:49:26.738561 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 44370, win 963, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
16:49:26.739539 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 45744, win 986, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
16:49:26.739882 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 47118, win 1008, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
16:49:26.740255 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 48492, win 1031, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
16:49:26.746756 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 49866, win 1053, options [nop,nop,TS val 1617129493 ecr 1313937627], length 0
16:49:26.747923 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 51240, win 1076, options [nop,nop,TS val 1617129494 ecr 1313937627], length 0
16:49:26.749083 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 52614, win 1099, options [nop,nop,TS val 1617129495 ecr 1313937629], length 0
16:49:26.750171 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 53988, win 1121, options [nop,nop,TS val 1617129496 ecr 1313937629], length 0
16:49:26.750808 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 55362, win 1144, options [nop,nop,TS val 1617129497 ecr 1313937629], length 0
16:49:26.754648 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 56736, win 1167, options [nop,nop,TS val 1617129500 ecr 1313937629], length 0
16:49:26.755985 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 58110, win 1189, options [nop,nop,TS val 1617129501 ecr 1313937630], length 0
16:49:26.758513 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 59484, win 1212, options [nop,nop,TS val 1617129502 ecr 1313937630], length 0
16:49:26.759096 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 60858, win 1234, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
16:49:26.759421 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 62232, win 1257, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
16:49:26.759755 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 63606, win 1280, options [nop,nop,TS val 1617129504 ecr 1313937636], length 0
16:49:26.760653 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 64980, win 1302, options [nop,nop,TS val 1617129505 ecr 1313937636], length 0
16:49:26.761453 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 66354, win 1325, options [nop,nop,TS val 1617129506 ecr 1313937638], length 0
16:49:26.762199 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 0
16:49:26.763547 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 36
16:49:26.763553 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 70476, win 1393, options [nop,nop,TS val 1617129508 ecr 1313937639], length 0
16:49:26.764298 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 73224, win 1438, options [nop,nop,TS val 1617129509 ecr 1313937641], length 0
16:49:26.764676 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 75972, win 1444, options [nop,nop,TS val 1617129510 ecr 1313937643], length 0
16:49:26.807754 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 75972:77346, ack 4190, win 307, options [nop,nop,TS val 1313937714 ecr 1617129473], length 1374
16:49:26.876467 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129620 ecr 1313937714], length 0
16:49:27.048760 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937955 ecr 1617129473], length 1374
16:49:27.051791 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617129762 ecr 1313937714], length 36
16:49:27.076444 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129822 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:27.371182 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130018 ecr 1313937714], length 36
16:49:27.519862 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313938426 ecr 1617129473], length 1374
16:49:27.547662 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617130293 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:27.883372 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130530 ecr 1313937714], length 36
16:49:28.511861 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313939418 ecr 1617129473], length 1374
16:49:28.538891 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617131285 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:28.907197 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617131554 ecr 1313937714], length 36
16:49:30.431864 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313941338 ecr 1617129473], length 1374
16:49:30.459127 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617133204 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:30.955388 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617133602 ecr 1313937714], length 36
16:49:34.207879 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313945114 ecr 1617129473], length 1374
16:49:34.235726 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617136981 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:35.256285 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617137954 ecr 1313937714], length 36
16:49:42.143864 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313953050 ecr 1617129473], length 1374
16:49:42.171531 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617144917 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:43.448262 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617146146 ecr 1313937714], length 36
(I still have the pcap files if someone wants to see them, but I'd
rather not give my work IP publicly so will send it privately on
request)
At this point, only the same 3 packets keep being replayed over and
over... According to 'ss' the Send-Q isn't empty on either side, the
client has some ~1k to send but the server has much more (87k)
After increasing the window size through net.*wmem sysctl it got stuck
with over 1MB in Send-Q, which makes sense because the socket is full...
So, what I don't get is, why are these acks continuously replayed? Given
the timing it looks like the server doesn't take the client acks into
account, despite having received that precise 33378 ack earlier and I
believe it should accept a higher ack value anyway ?
The ultimate question being, how can I go about debugging that?
I'm working on getting perf probe/crash to work on the server so I can
look at the kernel side now, I can reproduce this semi-easily so I'm
sure I'll get down to it eventually, but if anyone has an idea I'm all
ears.
Thanks!
--
Dominique Martinet | Asmadeus
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-06 9:07 tcp hang when socket fills up ? Dominique Martinet
@ 2018-04-13 9:42 ` Dominique Martinet
2018-04-13 15:01 ` Eric Dumazet
2018-04-13 16:32 ` Michal Kubecek
2 siblings, 0 replies; 26+ messages in thread
From: Dominique Martinet @ 2018-04-13 9:42 UTC (permalink / raw)
To: netdev
Note this is mostly me talking to myself, but in case anyone else hits
the same issues I might as well post my meagre progress.
Dominique Martinet wrote on Fri, Apr 06, 2018:
> (current kernel: vanilla 4.14.29)
reproduced in a fedora VM on that host with a 4.16.1-300.fc28.x86_64
kernel, since this one has DEBUG_INFO=y and I was lazy (but haven't seen
any patch about that kind of stuff recently so probably still valid)
Other main difference is the qdisc, VM is using fq_codel, host is
directly on wireless so mq with 4 pfifo_fast queues - it is harder to
reproduce on the VM (even on another VM with the same kernel) so I'd
put the difference down to the qdisc, but I was able to reproduce with
both ultimately.
(update: it actually was still fairly easy to reproduce until it got
later (coworkers left?), going from ~5-15s to reproduce to multiple
minutes, so this likely depends on net quality a lot. I couldn't
reproduce by fiddling with netem on a local network though...)
> [please find previous email for setup/tcpdump output]
So I have a crash dump with a socket / inet_sock that are blocked, since
this is a VM I even get gdb in bonus..
With the crash dump, I can confirm that the socket is not available for
writing (sk->sk_sndbuf - sk->sk_wmem_queued < sk->sk_wmem_queued >> 1),
but that doesn't help much if I can't tell why we're not taking acks in
With gdb, I set a breakpoint to tcp_ack (net/ipv4/tcp_input.c) as I
think that's the function that should handle my ack, and that gets the
replay.
First, abusing next, the flow seems to be something like
(I folded if/else not taken)
static int tcp_ack(struct sock *sk, const struct sk_buff *skb, int flag)
{
struct inet_connection_sock *icsk = inet_csk(sk);
struct tcp_sock *tp = tcp_sk(sk);
struct tcp_sacktag_state sack_state;
struct rate_sample rs = { .prior_delivered = 0 };
u32 prior_snd_una = tp->snd_una;
bool is_sack_reneg = tp->is_sack_reneg;
u32 ack_seq = TCP_SKB_CB(skb)->seq;
u32 ack = TCP_SKB_CB(skb)->ack_seq;
bool is_dupack = false;
int prior_packets = tp->packets_out;
u32 delivered = tp->delivered;
u32 lost = tp->lost;
int rexmit = REXMIT_NONE; /* Flag to (re)transmit to recover
losses */
u32 prior_fack;
sack_state.first_sackt = 0;
sack_state.rate = &rs;
/* We very likely will need to access rtx queue. */
prefetch(sk->tcp_rtx_queue.rb_node);
/* If the ack is older than previous acks
* then we can probably ignore it.
*/
if (before(ack, prior_snd_una)) {
}
/* If the ack includes data we haven't sent yet, discard
* this segment (RFC793 Section 3.9).
*/
if (after(ack, tp->snd_nxt))
if (after(ack, prior_snd_una)) {
}
prior_fack = tcp_is_sack(tp) ? tcp_highest_sack_seq(tp) :
tp->snd_una;
rs.prior_in_flight = tcp_packets_in_flight(tp);
/* ts_recent update must be made after we are sure that the
packet
* is in window.
*/
if (flag & FLAG_UPDATE_TS_RECENT)
if (!(flag & FLAG_SLOWPATH) && after(ack, prior_snd_una)) {
} else {
u32 ack_ev_flags = CA_ACK_SLOWPATH;
if (ack_seq != TCP_SKB_CB(skb)->end_seq)
else
NET_INC_STATS(sock_net(sk),
LINUX_MIB_TCPPUREACKS);
flag |= tcp_ack_update_window(sk, skb, ack, ack_seq);
if (TCP_SKB_CB(skb)->sacked)
if (tcp_ecn_rcv_ecn_echo(tp, tcp_hdr(skb))) {
}
if (flag & FLAG_WIN_UPDATE)
tcp_in_ack_event(sk, ack_ev_flags);
}
/* We passed data and got it acked, remove any soft error
* log. Something worked...
*/
sk->sk_err_soft = 0;
icsk->icsk_probes_out = 0;
tp->rcv_tstamp = tcp_jiffies32;
if (!prior_packets)
goto no_queue;
no_queue:
/* If data was DSACKed, see if we can undo a cwnd reduction. */
if (flag & FLAG_DSACKING_ACK)
tcp_fastretrans_alert(sk, prior_snd_una, is_dupack,
&flag,
&rexmit);
/* If this ack opens up a zero window, clear backoff. It was
* being used to time the probes, and is probably far higher
than
* it needs to be for normal retransmission.
*/
tcp_ack_probe(sk);
if (tp->tlp_high_seq)
tcp_process_tlp_ack(sk, ack, flag);
return 1;
And here is 'info local' towards the end of the function:
icsk = 0xffff88001740b800
tp = 0xffff88001740b800
sack_state = {reord = 84, first_sackt = 0, last_sackt = 0, rate = 0xffff88003fc83ae0, flag = 0, mss_now = 0}
rs = {prior_mstamp = 0, prior_delivered = 0, delivered = 0, interval_us = 0, rtt_us = 0, losses = 0, acked_sacked = 0, prior_in_flight = 0, is_app_limited = false, is_retrans = false, is_ack_delayed = false}
prior_snd_una = 2896339291
is_sack_reneg = false
ack_seq = 2651638615
ack = 2896339291
is_dupack = false
prior_packets = 0
delivered = 172
lost = 0
rexmit = 0
prior_fack = 2896339291
__vpp_verify = <optimized out>
pao_ID__ = <optimized out>
pao_tmp__ = <optimized out>
pao_ID__ = <optimized out>
pao_tmp__ = <optimized out>
pao_ID__ = <optimized out>
pao_tmp__ = <optimized out>
So as I was seeing, we have ack = prior_fack = prior_snd_una (which is > ack_seq)
We're in the !prior_packets goto to no_queue though so we don't pass
tcp_clean_rtx_queue - so that explains why the replays don't help. I'm
ok with that. But all these traces actually don't tell me why the first
time we saw that packed didn't take the ack and decreate sk_wmem_queued
properly.
This isn't easy enough to reproduce that I feel confident grabbing the
first call to that function that fails though, so I guess I'll actually
have to read the code I'm debugging instead of trying to just plow
through it...
Hints still welcome, if anyone has an idea.
--
Dominique Martinet | Asmadeus
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-06 9:07 tcp hang when socket fills up ? Dominique Martinet
2018-04-13 9:42 ` Dominique Martinet
@ 2018-04-13 15:01 ` Eric Dumazet
2018-04-13 16:32 ` Michal Kubecek
2 siblings, 0 replies; 26+ messages in thread
From: Eric Dumazet @ 2018-04-13 15:01 UTC (permalink / raw)
To: Dominique Martinet, netdev
On 04/06/2018 02:07 AM, Dominique Martinet wrote:
> (current kernel: vanilla 4.14.29)
>
> I've been running into troubles recently where a sockets "fills up" (as
> in, select() will no longer return it in its outfd / attempting to write
> to it after setting it to NONBLOCK will return -EWOULDBLOCK) and it
> doesn't seem to ever "unfill" until the tcp connexion timeout.
>
> The previous time I pushed it down to the application for not trying to
> read the socket either as I assume the buffers could be shared and
> just waiting won't take data out, but this time I'm a bit more
> skeptical as socat waits for the fd in both read and write...
>
> Let me take a minute to describe my setup, I don't think that how the
> socket was created matters but it might be interesting:
> - I have two computers behind NATs, no port forwarding on either side
> - One (call it C for client) runs ssh with a proxycommand ncat/socat to
> control the source port, e.g.
> $ ssh -o ProxyCommand="socat stdio tcp:<server public ip>:<port1>,sourceport=<port2>" server
> - The server runs another socat to connect to that and forwards to ssh
> locally, e.g.
> $ socat tcp:<client public ip>:<port2>,sourceport=<port1> tcp:127.0.0.1:22
>
> (yes, both are connect() calls, and that just works™ thanks to initial
> syn replay and conntrack on routers)
>
> When things stall, the first socat is in a select with both fd in
> reading, so it's waiting data.
> The second socat has data coming from ssh and is in a select with the
> client-facing socket in both read and write - that select never returns
> so the socket is not available for reading or writing and does not free
> up until the connection eventually times out a few minutes later.
>
> At this point, I only see tcp replays in tcpdump/wireshark. I've
> compared dumps from both sides and there are no lost packets, only
> reordering - there always is a batch of acks that were sent regularily
> coming in shortly before the hang. Here's the trace on the server:
>
> 16:49:26.735042 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 70476:71850, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
> 16:49:26.735046 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 71850:73224, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
> 16:49:26.735334 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 41622, win 918, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
> 16:49:26.736005 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 42996, win 940, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
> 16:49:26.736402 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 73224:74598, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
> 16:49:26.736408 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 74598:75972, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
> 16:49:26.738561 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 44370, win 963, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
> 16:49:26.739539 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 45744, win 986, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
> 16:49:26.739882 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 47118, win 1008, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
> 16:49:26.740255 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 48492, win 1031, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
> 16:49:26.746756 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 49866, win 1053, options [nop,nop,TS val 1617129493 ecr 1313937627], length 0
> 16:49:26.747923 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 51240, win 1076, options [nop,nop,TS val 1617129494 ecr 1313937627], length 0
> 16:49:26.749083 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 52614, win 1099, options [nop,nop,TS val 1617129495 ecr 1313937629], length 0
> 16:49:26.750171 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 53988, win 1121, options [nop,nop,TS val 1617129496 ecr 1313937629], length 0
> 16:49:26.750808 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 55362, win 1144, options [nop,nop,TS val 1617129497 ecr 1313937629], length 0
> 16:49:26.754648 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 56736, win 1167, options [nop,nop,TS val 1617129500 ecr 1313937629], length 0
> 16:49:26.755985 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 58110, win 1189, options [nop,nop,TS val 1617129501 ecr 1313937630], length 0
> 16:49:26.758513 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 59484, win 1212, options [nop,nop,TS val 1617129502 ecr 1313937630], length 0
> 16:49:26.759096 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 60858, win 1234, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
> 16:49:26.759421 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 62232, win 1257, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
> 16:49:26.759755 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 63606, win 1280, options [nop,nop,TS val 1617129504 ecr 1313937636], length 0
> 16:49:26.760653 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 64980, win 1302, options [nop,nop,TS val 1617129505 ecr 1313937636], length 0
> 16:49:26.761453 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 66354, win 1325, options [nop,nop,TS val 1617129506 ecr 1313937638], length 0
> 16:49:26.762199 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 0
> 16:49:26.763547 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 36
> 16:49:26.763553 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 70476, win 1393, options [nop,nop,TS val 1617129508 ecr 1313937639], length 0
> 16:49:26.764298 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 73224, win 1438, options [nop,nop,TS val 1617129509 ecr 1313937641], length 0
> 16:49:26.764676 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 75972, win 1444, options [nop,nop,TS val 1617129510 ecr 1313937643], length 0
> 16:49:26.807754 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 75972:77346, ack 4190, win 307, options [nop,nop,TS val 1313937714 ecr 1617129473], length 1374
> 16:49:26.876467 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129620 ecr 1313937714], length 0
There is no way a regular TCP stack (including linux) could send the following frame.
> 16:49:27.048760 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937955 ecr 1617129473], length 1374
So something is mangling the packet, maybe NAT or something.
> 16:49:27.051791 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617129762 ecr 1313937714], length 36
> 16:49:27.076444 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129822 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
> 16:49:27.371182 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130018 ecr 1313937714], length 36
> 16:49:27.519862 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313938426 ecr 1617129473], length 1374
> 16:49:27.547662 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617130293 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
> 16:49:27.883372 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130530 ecr 1313937714], length 36
> 16:49:28.511861 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313939418 ecr 1617129473], length 1374
> 16:49:28.538891 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617131285 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
> 16:49:28.907197 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617131554 ecr 1313937714], length 36
> 16:49:30.431864 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313941338 ecr 1617129473], length 1374
> 16:49:30.459127 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617133204 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
> 16:49:30.955388 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617133602 ecr 1313937714], length 36
> 16:49:34.207879 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313945114 ecr 1617129473], length 1374
> 16:49:34.235726 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617136981 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
> 16:49:35.256285 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617137954 ecr 1313937714], length 36
> 16:49:42.143864 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313953050 ecr 1617129473], length 1374
> 16:49:42.171531 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617144917 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
> 16:49:43.448262 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617146146 ecr 1313937714], length 36
>
> (I still have the pcap files if someone wants to see them, but I'd
> rather not give my work IP publicly so will send it privately on
> request)
>
>
> At this point, only the same 3 packets keep being replayed over and
> over... According to 'ss' the Send-Q isn't empty on either side, the
> client has some ~1k to send but the server has much more (87k)
> After increasing the window size through net.*wmem sysctl it got stuck
> with over 1MB in Send-Q, which makes sense because the socket is full...
>
>
> So, what I don't get is, why are these acks continuously replayed? Given
> the timing it looks like the server doesn't take the client acks into
> account, despite having received that precise 33378 ack earlier and I
> believe it should accept a higher ack value anyway ?
>
> The ultimate question being, how can I go about debugging that?
> I'm working on getting perf probe/crash to work on the server so I can
> look at the kernel side now, I can reproduce this semi-easily so I'm
> sure I'll get down to it eventually, but if anyone has an idea I'm all
> ears.
>
>
> Thanks!
>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-06 9:07 tcp hang when socket fills up ? Dominique Martinet
2018-04-13 9:42 ` Dominique Martinet
2018-04-13 15:01 ` Eric Dumazet
@ 2018-04-13 16:32 ` Michal Kubecek
2018-04-14 1:09 ` Dominique Martinet
2 siblings, 1 reply; 26+ messages in thread
From: Michal Kubecek @ 2018-04-13 16:32 UTC (permalink / raw)
To: netdev; +Cc: Dominique Martinet
On Fri, Apr 06, 2018 at 11:07:20AM +0200, Dominique Martinet wrote:
> 16:49:26.735042 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 70476:71850, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
> 16:49:26.735046 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 71850:73224, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
> 16:49:26.735334 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 41622, win 918, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
> 16:49:26.736005 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 42996, win 940, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
> 16:49:26.736402 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 73224:74598, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
> 16:49:26.736408 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 74598:75972, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
> 16:49:26.738561 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 44370, win 963, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
> 16:49:26.739539 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 45744, win 986, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
> 16:49:26.739882 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 47118, win 1008, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
> 16:49:26.740255 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 48492, win 1031, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
> 16:49:26.746756 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 49866, win 1053, options [nop,nop,TS val 1617129493 ecr 1313937627], length 0
> 16:49:26.747923 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 51240, win 1076, options [nop,nop,TS val 1617129494 ecr 1313937627], length 0
> 16:49:26.749083 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 52614, win 1099, options [nop,nop,TS val 1617129495 ecr 1313937629], length 0
> 16:49:26.750171 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 53988, win 1121, options [nop,nop,TS val 1617129496 ecr 1313937629], length 0
> 16:49:26.750808 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 55362, win 1144, options [nop,nop,TS val 1617129497 ecr 1313937629], length 0
> 16:49:26.754648 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 56736, win 1167, options [nop,nop,TS val 1617129500 ecr 1313937629], length 0
> 16:49:26.755985 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 58110, win 1189, options [nop,nop,TS val 1617129501 ecr 1313937630], length 0
> 16:49:26.758513 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 59484, win 1212, options [nop,nop,TS val 1617129502 ecr 1313937630], length 0
> 16:49:26.759096 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 60858, win 1234, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
> 16:49:26.759421 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 62232, win 1257, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
> 16:49:26.759755 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 63606, win 1280, options [nop,nop,TS val 1617129504 ecr 1313937636], length 0
> 16:49:26.760653 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 64980, win 1302, options [nop,nop,TS val 1617129505 ecr 1313937636], length 0
> 16:49:26.761453 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 66354, win 1325, options [nop,nop,TS val 1617129506 ecr 1313937638], length 0
> 16:49:26.762199 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 0
> 16:49:26.763547 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 36
> 16:49:26.763553 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 70476, win 1393, options [nop,nop,TS val 1617129508 ecr 1313937639], length 0
> 16:49:26.764298 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 73224, win 1438, options [nop,nop,TS val 1617129509 ecr 1313937641], length 0
> 16:49:26.764676 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 75972, win 1444, options [nop,nop,TS val 1617129510 ecr 1313937643], length 0
> 16:49:26.807754 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 75972:77346, ack 4190, win 307, options [nop,nop,TS val 1313937714 ecr 1617129473], length 1374
> 16:49:26.876467 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129620 ecr 1313937714], length 0
> 16:49:27.048760 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937955 ecr 1617129473], length 1374
> 16:49:27.051791 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617129762 ecr 1313937714], length 36
> 16:49:27.076444 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129822 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
> 16:49:27.371182 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130018 ecr 1313937714], length 36
> 16:49:27.519862 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313938426 ecr 1617129473], length 1374
> 16:49:27.547662 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617130293 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
> 16:49:27.883372 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130530 ecr 1313937714], length 36
> 16:49:28.511861 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313939418 ecr 1617129473], length 1374
> 16:49:28.538891 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617131285 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
> 16:49:28.907197 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617131554 ecr 1313937714], length 36
> 16:49:30.431864 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313941338 ecr 1617129473], length 1374
> 16:49:30.459127 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617133204 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
> 16:49:30.955388 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617133602 ecr 1313937714], length 36
> 16:49:34.207879 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313945114 ecr 1617129473], length 1374
> 16:49:34.235726 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617136981 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
> 16:49:35.256285 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617137954 ecr 1313937714], length 36
> 16:49:42.143864 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313953050 ecr 1617129473], length 1374
> 16:49:42.171531 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617144917 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
> 16:49:43.448262 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617146146 ecr 1313937714], length 36
The way I read this, server doesn't see anything sent by client since
some point shortly before the dump shown here starts (about 5ms). It
keeps sending data until 16:49:26.807754 (seq 77346) and then keeps
resending first (from its point of view) unacknowledged segment
(32004:33378) in exponentially growing intervals and ignores replies
from the client. Client apparently receives these retransmits and
replies with dupack (with D-SACK for 32004:33378) and retransmits of its
own first unacknowledged segment (4190:4226).
As we can see the client packets in the dump (which was taken on
server), it would mean they are dropped after the point where packet
socket would pass them to libpcap. That might be e.g. netfilter
(conntrack?) or the IP/TCP code detecting them to be invalid for some
reason (which is not obvious to me from the dump above).
There are two strange points:
1. While client apparently responds to all server retransmits, it does
so with TSecr=1313937714 (matching server packet from 16:49:26.807754)
rather than TSval of the packets it dupacks (1313937955 through
1313953050). This doesn't seem to follow the rules of RFC 7323
Section 4.3.
2. Window size values in acks from client grow with each acked packet by
22-23 (which might be ~1400 with scaling factor of 64). I would rather
expect advertised receive window to go down by 1374 with each received
segment and to grow by bigger steps with each read()/recv() call from
application.
We might get more insight if we saw the same connection on both sides.
>From what was presented here, my guess is that
(1) received packets are dropped somewhere on server side (after they
are cloned for the packet socket)
(2) there is something wrong either on client side or between the two
hosts (there is at least a NAT, IIUC)
Michal Kubecek
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-13 16:32 ` Michal Kubecek
@ 2018-04-14 1:09 ` Dominique Martinet
2018-04-14 1:39 ` Eric Dumazet
0 siblings, 1 reply; 26+ messages in thread
From: Dominique Martinet @ 2018-04-14 1:09 UTC (permalink / raw)
To: Michal Kubecek, Eric Dumazet; +Cc: netdev
Thank you for the replies,
Eric Dumazet wrote on Fri, Apr 13, 2018:
> There is no way a regular TCP stack (including linux) could send the following frame.
>
> > 16:49:27.048760 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937955 ecr 1617129473], length 1374
>
> So something is mangling the packet, maybe NAT or something.
The pcap was produced on the server which emitted the frame, so it
should be exactly as the server intended it to be without any mangling
involved?
If you could point at what strikes you as odd I can check if the same
happened on other hang traces I might still have. Is it just that it
replays a very old seq?
(if it's odd wrt client packets, the same trace as captured on client is
at the end of this mail)
Michal Kubecek wrote on Fri, Apr 13, 2018:
> The way I read this, server doesn't see anything sent by client since
> some point shortly before the dump shown here starts (about 5ms). It
> keeps sending data until 16:49:26.807754 (seq 77346) and then keeps
> resending first (from its point of view) unacknowledged segment
> (32004:33378) in exponentially growing intervals and ignores replies
> from the client. Client apparently receives these retransmits and
> replies with dupack (with D-SACK for 32004:33378) and retransmits of its
> own first unacknowledged segment (4190:4226).
I have the same understanding.
> As we can see the client packets in the dump (which was taken on
> server), it would mean they are dropped after the point where packet
> socket would pass them to libpcap. That might be e.g. netfilter
> (conntrack?) or the IP/TCP code detecting them to be invalid for some
> reason (which is not obvious to me from the dump above).
In my second mail, I got as far as `tcp_ack`, where the replays didn't
get considered for ack because it was seen as having the same ack number
as the lastest seen. I'm not sure why it didn't get considered the first
time, I need to add more instrumentation.
> There are two strange points:
>
> 1. While client apparently responds to all server retransmits, it does
> so with TSecr=1313937714 (matching server packet from 16:49:26.807754)
> rather than TSval of the packets it dupacks (1313937955 through
> 1313953050). This doesn't seem to follow the rules of RFC 7323
> Section 4.3.
I have a pcap of the same sequence on the client as well, and it does
get the packets with higher val= unchanged, so that might be something
to look at.
I'll post a similarily formated tcpdump output at the end of this mail,
the timestamps and verious numbers in the traces match with my original
mail.
> 2. Window size values in acks from client grow with each acked packet by
> 22-23 (which might be ~1400 with scaling factor of 64). I would rather
> expect advertised receive window to go down by 1374 with each received
> segment and to grow by bigger steps with each read()/recv() call from
> application.
I have a very poor understanding of this part of the protocol/code, but
I think the window grows when a socket starts receiving a bulk of data.
Given this hangs very fast, there were only individual small packets up
till this trace where data comes in quickly, and I think this might just
be window scaling?
I might very well be wrong though..
> We might get more insight if we saw the same connection on both sides.
> From what was presented here, my guess is that
>
> (1) received packets are dropped somewhere on server side (after they
> are cloned for the packet socket)
I'd agree for received replays, but I don't get why the first time that
ack was received didn't work.
> (2) there is something wrong either on client side or between the two
> hosts (there is at least a NAT, IIUC)
My minimal reproducer involves the internet, so there is no telling what
is in the middle, but there is at least two NAT, and the connection is
established through two connect() calls with carefully crafted
source/destination ports if that makes a difference.
I've done a bit more fiddling with netem/tbf and a single nat but I
still can't reproduce it on a local network; I need to spend more time
trying as this is very frustrating because I don't control any of the
two NATs in the real case.
I don't think the NATs are culprits, though - looking at the traces I
don't see any reordering within a direction, missing packets or fiddling
with values so this seems fairly straightforward.
Here's the client-side traces. pcap files still available on request, if
you prefer to look at it in wireshark or another tool.
16:49:26.726793 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 40248:41622, ack 4190, win 307, options [nop,nop,TS val 1313937609 ecr 1617129440], length 1374
16:49:26.726798 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 41622, win 918, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
16:49:26.726802 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 41622:42996, ack 4190, win 307, options [nop,nop,TS val 1313937609 ecr 1617129440], length 1374
16:49:26.726807 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 42996, win 940, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
16:49:26.730611 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 42996:44370, ack 4190, win 307, options [nop,nop,TS val 1313937616 ecr 1617129456], length 1374
16:49:26.730619 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 44370, win 963, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
16:49:26.730835 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 44370:45744, ack 4190, win 307, options [nop,nop,TS val 1313937616 ecr 1617129456], length 1374
16:49:26.730847 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 45744, win 986, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
16:49:26.732371 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 45744:47118, ack 4190, win 307, options [nop,nop,TS val 1313937617 ecr 1617129456], length 1374
16:49:26.732383 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 47118, win 1008, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
16:49:26.732393 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 47118:48492, ack 4190, win 307, options [nop,nop,TS val 1313937617 ecr 1617129456], length 1374
16:49:26.732400 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 48492, win 1031, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
16:49:26.740896 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 48492:49866, ack 4190, win 307, options [nop,nop,TS val 1313937627 ecr 1617129464], length 1374
16:49:26.740910 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 49866, win 1053, options [nop,nop,TS val 1617129493 ecr 1313937627], length 0
16:49:26.742220 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 49866:51240, ack 4190, win 307, options [nop,nop,TS val 1313937627 ecr 1617129464], length 1374
16:49:26.742232 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 51240, win 1076, options [nop,nop,TS val 1617129494 ecr 1313937627], length 0
16:49:26.743463 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 51240:52614, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.743476 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 52614, win 1099, options [nop,nop,TS val 1617129495 ecr 1313937629], length 0
16:49:26.744478 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 52614:53988, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.744490 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 53988, win 1121, options [nop,nop,TS val 1617129496 ecr 1313937629], length 0
16:49:26.744975 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 53988:55362, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.744982 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 55362, win 1144, options [nop,nop,TS val 1617129497 ecr 1313937629], length 0
16:49:26.748722 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 55362:56736, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.748733 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 56736, win 1167, options [nop,nop,TS val 1617129500 ecr 1313937629], length 0
16:49:26.749615 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 56736:58110, ack 4190, win 307, options [nop,nop,TS val 1313937630 ecr 1617129467], length 1374
16:49:26.749627 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 58110, win 1189, options [nop,nop,TS val 1617129501 ecr 1313937630], length 0
16:49:26.750248 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 58110:59484, ack 4190, win 307, options [nop,nop,TS val 1313937630 ecr 1617129467], length 1374
16:49:26.750260 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 59484, win 1212, options [nop,nop,TS val 1617129502 ecr 1313937630], length 0
16:49:26.750968 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 59484:60858, ack 4190, win 307, options [nop,nop,TS val 1313937635 ecr 1617129473], length 1374
16:49:26.750980 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 60858, win 1234, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
16:49:26.751486 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 60858:62232, ack 4190, win 307, options [nop,nop,TS val 1313937635 ecr 1617129473], length 1374
16:49:26.751498 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 62232, win 1257, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
16:49:26.752095 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 62232:63606, ack 4190, win 307, options [nop,nop,TS val 1313937636 ecr 1617129473], length 1374
16:49:26.752107 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 63606, win 1280, options [nop,nop,TS val 1617129504 ecr 1313937636], length 0
16:49:26.752920 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 63606:64980, ack 4190, win 307, options [nop,nop,TS val 1313937636 ecr 1617129473], length 1374
16:49:26.752932 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 64980, win 1302, options [nop,nop,TS val 1617129505 ecr 1313937636], length 0
16:49:26.754573 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 64980:66354, ack 4190, win 307, options [nop,nop,TS val 1313937638 ecr 1617129473], length 1374
16:49:26.754585 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 66354, win 1325, options [nop,nop,TS val 1617129506 ecr 1313937638], length 0
16:49:26.755260 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 66354:67728, ack 4190, win 307, options [nop,nop,TS val 1313937638 ecr 1617129473], length 1374
16:49:26.755271 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 0
16:49:26.755493 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 36
16:49:26.756076 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 67728:69102, ack 4190, win 307, options [nop,nop,TS val 1313937639 ecr 1617129473], length 1374
16:49:26.756391 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 69102:70476, ack 4190, win 307, options [nop,nop,TS val 1313937639 ecr 1617129473], length 1374
16:49:26.756402 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 70476, win 1393, options [nop,nop,TS val 1617129508 ecr 1313937639], length 0
16:49:26.757020 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 70476:71850, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
16:49:26.757349 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 71850:73224, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
16:49:26.757360 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 73224, win 1438, options [nop,nop,TS val 1617129509 ecr 1313937641], length 0
16:49:26.757878 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 73224:74598, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
16:49:26.758457 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 74598:75972, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
16:49:26.758465 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 75972, win 1444, options [nop,nop,TS val 1617129510 ecr 1313937643], length 0
16:49:26.828268 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 75972:77346, ack 4190, win 307, options [nop,nop,TS val 1313937714 ecr 1617129473], length 1374
16:49:26.868867 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129620 ecr 1313937714], length 0
16:49:27.010733 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617129762 ecr 1313937714], length 36
16:49:27.070035 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937955 ecr 1617129473], length 1374
16:49:27.070062 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129822 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:27.266769 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130018 ecr 1313937714], length 36
16:49:27.540998 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313938426 ecr 1617129473], length 1374
16:49:27.541022 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617130293 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:27.778687 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130530 ecr 1313937714], length 36
16:49:28.532876 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313939418 ecr 1617129473], length 1374
16:49:28.532907 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617131285 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:28.802879 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617131554 ecr 1313937714], length 36
16:49:30.452830 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313941338 ecr 1617129473], length 1374
16:49:30.452861 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617133204 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:30.850734 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617133602 ecr 1313937714], length 36
16:49:34.229420 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313945114 ecr 1617129473], length 1374
16:49:34.229447 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617136981 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:35.202854 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617137954 ecr 1313937714], length 36
16:49:42.165044 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313953050 ecr 1617129473], length 1374
16:49:42.165073 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617144917 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:43.394773 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617146146 ecr 1313937714], length 36
Once again, thank you both.
--
Dominique Martinet | Asmadeus
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-14 1:09 ` Dominique Martinet
@ 2018-04-14 1:39 ` Eric Dumazet
2018-04-14 1:55 ` Dominique Martinet
0 siblings, 1 reply; 26+ messages in thread
From: Eric Dumazet @ 2018-04-14 1:39 UTC (permalink / raw)
To: Dominique Martinet, Michal Kubecek, Eric Dumazet; +Cc: netdev
On 04/13/2018 06:09 PM, Dominique Martinet wrote:
> Thank you for the replies,
>
> Eric Dumazet wrote on Fri, Apr 13, 2018:
>> There is no way a regular TCP stack (including linux) could send the following frame.
>>
>>> 16:49:27.048760 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937955 ecr 1617129473], length 1374
>>
>> So something is mangling the packet, maybe NAT or something.
>
>
> The pcap was produced on the server which emitted the frame, so it
> should be exactly as the server intended it to be without any mangling
> involved?
> If you could point at what strikes you as odd I can check if the same
> happened on other hang traces I might still have. Is it just that it
> replays a very old seq?
> (if it's odd wrt client packets, the same trace as captured on client is
> at the end of this mail)
>
Ah sorry, your trace was truncated, we need more packets _before_ the excerpt.
That might be caused by some TS val/ecr breakage :
Many acks were received by the server tcpdump,
but none of them was accepted by TCP stack, for some reason.
Try to disable TCP timestamps, it will give some hint if bug does not reproduce.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-14 1:39 ` Eric Dumazet
@ 2018-04-14 1:55 ` Dominique Martinet
2018-04-16 1:47 ` Dominique Martinet
2018-04-16 20:43 ` Marcelo Ricardo Leitner
0 siblings, 2 replies; 26+ messages in thread
From: Dominique Martinet @ 2018-04-14 1:55 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Michal Kubecek, netdev
Eric Dumazet wrote on Fri, Apr 13, 2018:
> Ah sorry, your trace was truncated, we need more packets _before_ the excerpt.
Ah, sorry as well. I tried to go far back enough to include the replayed
packets, but I see it didn't include the original ack for that packet.
I'm resending both full traces from connect to fin this time.
> That might be caused by some TS val/ecr breakage :
>
> Many acks were received by the server tcpdump,
> but none of them was accepted by TCP stack, for some reason.
>
> Try to disable TCP timestamps, it will give some hint if bug does not reproduce.
I unfortunately cannot reproduce on a local network, but will give this
a try on Monday and report. (sysctl net.ipv4.tcp_timestamps=0, correct?)
Here's the traces again, server-capture first:
16:49:22.374452 IP <server local ip>.13317 > <client public ip>.31872: Flags [S], seq 2026966826, win 29200, options [mss 1460,sackOK,TS val 1313933281 ecr 0,nop,wscale 7], length 0
16:49:22.700553 IP <client public ip>.31872 > <server local ip>.13317: Flags [S], seq 882075258, win 29200, options [mss 1386,sackOK,TS val 1617125446 ecr 0,nop,wscale 7], length 0
16:49:22.700577 IP <server local ip>.13317 > <client public ip>.31872: Flags [S.], seq 2026966826, ack 882075259, win 29200, options [mss 1460,sackOK,TS val 1313933607 ecr 1617125446,nop,wscale 7], length 0
16:49:22.727131 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 1, win 229, options [nop,nop,TS val 1617125472 ecr 1313933607], length 0
16:49:22.729193 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 1:22, ack 1, win 229, options [nop,nop,TS val 1617125472 ecr 1313933607], length 21
16:49:22.729230 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], ack 22, win 229, options [nop,nop,TS val 1313933635 ecr 1617125472], length 0
16:49:22.732726 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 1:22, ack 22, win 229, options [nop,nop,TS val 1313933639 ecr 1617125472], length 21
16:49:22.759809 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 22, win 229, options [nop,nop,TS val 1617125503 ecr 1313933639], length 0
16:49:22.759835 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 22:734, ack 22, win 229, options [nop,nop,TS val 1313933666 ecr 1617125503], length 712
16:49:22.761643 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 22:1270, ack 22, win 229, options [nop,nop,TS val 1617125504 ecr 1313933639], length 1248
16:49:22.801860 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], ack 1270, win 248, options [nop,nop,TS val 1313933708 ecr 1617125504], length 0
16:49:22.826902 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 734, win 240, options [nop,nop,TS val 1617125572 ecr 1313933666], length 0
16:49:22.827576 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 1270:1318, ack 734, win 240, options [nop,nop,TS val 1617125573 ecr 1313933708], length 48
16:49:22.827600 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], ack 1318, win 248, options [nop,nop,TS val 1313933734 ecr 1617125573], length 0
16:49:22.833028 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 734:1122, ack 1318, win 248, options [nop,nop,TS val 1313933739 ecr 1617125573], length 388
16:49:22.858455 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 1122, win 251, options [nop,nop,TS val 1617125604 ecr 1313933739], length 0
16:49:22.865866 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 1318:1334, ack 1122, win 251, options [nop,nop,TS val 1617125612 ecr 1313933739], length 16
16:49:22.906865 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], ack 1334, win 248, options [nop,nop,TS val 1313933813 ecr 1617125612], length 0
16:49:22.944474 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 1334:1386, ack 1122, win 251, options [nop,nop,TS val 1617125678 ecr 1313933813], length 52
16:49:22.944497 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], ack 1386, win 248, options [nop,nop,TS val 1313933851 ecr 1617125678], length 0
16:49:22.944747 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 1122:1174, ack 1386, win 248, options [nop,nop,TS val 1313933851 ecr 1617125678], length 52
16:49:22.971083 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 1386:1454, ack 1174, win 251, options [nop,nop,TS val 1617125716 ecr 1313933851], length 68
16:49:22.971607 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 1174:1258, ack 1454, win 248, options [nop,nop,TS val 1313933878 ecr 1617125716], length 84
16:49:22.998201 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 1454:2082, ack 1258, win 251, options [nop,nop,TS val 1617125742 ecr 1313933878], length 628
16:49:22.998987 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 1258:1838, ack 2082, win 268, options [nop,nop,TS val 1313933905 ecr 1617125742], length 580
16:49:23.065102 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 1838, win 262, options [nop,nop,TS val 1617125810 ecr 1313933905], length 0
16:49:24.811297 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 2082:3254, ack 1838, win 262, options [nop,nop,TS val 1617127527 ecr 1313933905], length 1172
16:49:24.812562 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 1838:1874, ack 3254, win 287, options [nop,nop,TS val 1313935719 ecr 1617127527], length 36
16:49:24.838210 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 1874, win 262, options [nop,nop,TS val 1617127583 ecr 1313935719], length 0
16:49:24.838236 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 1874:2534, ack 3254, win 287, options [nop,nop,TS val 1313935744 ecr 1617127583], length 660
16:49:24.839175 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 3254:3374, ack 1874, win 262, options [nop,nop,TS val 1617127584 ecr 1313935719], length 120
16:49:24.879860 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], ack 3374, win 287, options [nop,nop,TS val 1313935786 ecr 1617127584], length 0
16:49:24.906889 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 2534, win 273, options [nop,nop,TS val 1617127652 ecr 1313935744], length 0
16:49:24.906916 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 2534:2586, ack 3374, win 287, options [nop,nop,TS val 1313935813 ecr 1617127652], length 52
16:49:24.932283 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 2586, win 273, options [nop,nop,TS val 1617127678 ecr 1313935813], length 0
16:49:24.933286 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 3374:3902, ack 2586, win 273, options [nop,nop,TS val 1617127678 ecr 1313935813], length 528
16:49:24.933314 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], ack 3902, win 307, options [nop,nop,TS val 1313935839 ecr 1617127678], length 0
16:49:24.934315 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 2586:2694, ack 3902, win 307, options [nop,nop,TS val 1313935840 ecr 1617127678], length 108
16:49:24.999902 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 2694, win 273, options [nop,nop,TS val 1617127745 ecr 1313935840], length 0
16:49:24.999927 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 2694:2862, ack 3902, win 307, options [nop,nop,TS val 1313935906 ecr 1617127745], length 168
16:49:25.025121 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 2862, win 284, options [nop,nop,TS val 1617127771 ecr 1313935906], length 0
16:49:25.732946 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 3902:3938, ack 2862, win 284, options [nop,nop,TS val 1617128391 ecr 1313935906], length 36
16:49:25.733531 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 2862:2898, ack 3938, win 307, options [nop,nop,TS val 1313936640 ecr 1617128391], length 36
16:49:25.763138 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 3938:3974, ack 2898, win 284, options [nop,nop,TS val 1617128507 ecr 1313936640], length 36
16:49:25.763721 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 2898:2934, ack 3974, win 307, options [nop,nop,TS val 1313936670 ecr 1617128507], length 36
16:49:25.832807 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 3974:4010, ack 2934, win 284, options [nop,nop,TS val 1617128576 ecr 1313936670], length 36
16:49:25.833392 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 2934:2970, ack 4010, win 307, options [nop,nop,TS val 1313936740 ecr 1617128576], length 36
16:49:25.873065 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4010:4046, ack 2970, win 284, options [nop,nop,TS val 1617128617 ecr 1313936740], length 36
16:49:25.873608 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 2970:3006, ack 4046, win 307, options [nop,nop,TS val 1313936780 ecr 1617128617], length 36
16:49:25.942195 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 3006, win 284, options [nop,nop,TS val 1617128687 ecr 1313936780], length 0
16:49:26.244880 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4046:4082, ack 3006, win 284, options [nop,nop,TS val 1617128954 ecr 1313936780], length 36
16:49:26.245455 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 3006:3042, ack 4082, win 307, options [nop,nop,TS val 1313937152 ecr 1617128954], length 36
16:49:26.270697 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 3042, win 284, options [nop,nop,TS val 1617129016 ecr 1313937152], length 0
16:49:26.289989 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4082:4118, ack 3042, win 284, options [nop,nop,TS val 1617129035 ecr 1313937152], length 36
16:49:26.290569 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 3042:3078, ack 4118, win 307, options [nop,nop,TS val 1313937197 ecr 1617129035], length 36
16:49:26.357222 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 3078, win 284, options [nop,nop,TS val 1617129102 ecr 1313937197], length 0
16:49:26.409189 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4118:4154, ack 3078, win 284, options [nop,nop,TS val 1617129154 ecr 1313937197], length 36
16:49:26.409773 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 3078:3114, ack 4154, win 307, options [nop,nop,TS val 1313937316 ecr 1617129154], length 36
16:49:26.435224 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 3114, win 284, options [nop,nop,TS val 1617129181 ecr 1313937316], length 0
16:49:26.654517 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4154:4190, ack 3114, win 284, options [nop,nop,TS val 1617129290 ecr 1313937316], length 36
16:49:26.654979 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 3114:3150, ack 4190, win 307, options [nop,nop,TS val 1313937561 ecr 1617129290], length 36
16:49:26.664510 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 3150:4524, ack 4190, win 307, options [nop,nop,TS val 1313937571 ecr 1617129290], length 1374
16:49:26.664538 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 4524:5898, ack 4190, win 307, options [nop,nop,TS val 1313937571 ecr 1617129290], length 1374
16:49:26.664645 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 5898:7272, ack 4190, win 307, options [nop,nop,TS val 1313937571 ecr 1617129290], length 1374
16:49:26.664648 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 7272:8646, ack 4190, win 307, options [nop,nop,TS val 1313937571 ecr 1617129290], length 1374
16:49:26.665977 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 8646:10020, ack 4190, win 307, options [nop,nop,TS val 1313937572 ecr 1617129290], length 1374
16:49:26.666414 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 10020:11394, ack 4190, win 307, options [nop,nop,TS val 1313937573 ecr 1617129290], length 1374
16:49:26.666424 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 11394:12768, ack 4190, win 307, options [nop,nop,TS val 1313937573 ecr 1617129290], length 1374
16:49:26.666427 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 12768:14142, ack 4190, win 307, options [nop,nop,TS val 1313937573 ecr 1617129290], length 1374
16:49:26.667536 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 14142:15516, ack 4190, win 307, options [nop,nop,TS val 1313937574 ecr 1617129290], length 1374
16:49:26.684298 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 3150, win 284, options [nop,nop,TS val 1617129430 ecr 1313937561], length 0
16:49:26.684314 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 15516:16890, ack 4190, win 307, options [nop,nop,TS val 1313937590 ecr 1617129430], length 1374
16:49:26.684320 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 16890:18264, ack 4190, win 307, options [nop,nop,TS val 1313937590 ecr 1617129430], length 1374
16:49:26.691620 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 4524, win 307, options [nop,nop,TS val 1617129437 ecr 1313937571], length 0
16:49:26.691635 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 18264:19638, ack 4190, win 307, options [nop,nop,TS val 1313937598 ecr 1617129437], length 1374
16:49:26.691642 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 19638:21012, ack 4190, win 307, options [nop,nop,TS val 1313937598 ecr 1617129437], length 1374
16:49:26.692236 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 5898, win 329, options [nop,nop,TS val 1617129437 ecr 1313937571], length 0
16:49:26.692250 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 21012:22386, ack 4190, win 307, options [nop,nop,TS val 1313937598 ecr 1617129437], length 1374
16:49:26.692254 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 22386:23760, ack 4190, win 307, options [nop,nop,TS val 1313937598 ecr 1617129437], length 1374
16:49:26.692551 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 7272, win 352, options [nop,nop,TS val 1617129437 ecr 1313937571], length 0
16:49:26.692954 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 23760:25134, ack 4190, win 307, options [nop,nop,TS val 1313937599 ecr 1617129437], length 1374
16:49:26.692958 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 25134:26508, ack 4190, win 307, options [nop,nop,TS val 1313937599 ecr 1617129437], length 1374
16:49:26.693801 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 8646, win 375, options [nop,nop,TS val 1617129438 ecr 1313937571], length 0
16:49:26.694488 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 10020, win 397, options [nop,nop,TS val 1617129439 ecr 1313937572], length 0
16:49:26.694920 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 26508:27882, ack 4190, win 307, options [nop,nop,TS val 1313937601 ecr 1617129439], length 1374
16:49:26.694925 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 27882:29256, ack 4190, win 307, options [nop,nop,TS val 1313937601 ecr 1617129439], length 1374
16:49:26.695593 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 29256:30630, ack 4190, win 307, options [nop,nop,TS val 1313937602 ecr 1617129439], length 1374
16:49:26.695597 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 30630:32004, ack 4190, win 307, options [nop,nop,TS val 1313937602 ecr 1617129439], length 1374
16:49:26.696500 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 11394, win 420, options [nop,nop,TS val 1617129440 ecr 1313937573], length 0
16:49:26.697258 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 12768, win 443, options [nop,nop,TS val 1617129440 ecr 1313937573], length 0
16:49:26.697576 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 14142, win 465, options [nop,nop,TS val 1617129440 ecr 1313937573], length 0
16:49:26.698057 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 15516, win 488, options [nop,nop,TS val 1617129440 ecr 1313937574], length 0
16:49:26.700531 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937607 ecr 1617129440], length 1374
16:49:26.700536 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 33378:34752, ack 4190, win 307, options [nop,nop,TS val 1313937607 ecr 1617129440], length 1374
16:49:26.701255 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 34752:36126, ack 4190, win 307, options [nop,nop,TS val 1313937607 ecr 1617129440], length 1374
16:49:26.701260 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 36126:37500, ack 4190, win 307, options [nop,nop,TS val 1313937607 ecr 1617129440], length 1374
16:49:26.702122 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 37500:38874, ack 4190, win 307, options [nop,nop,TS val 1313937608 ecr 1617129440], length 1374
16:49:26.702128 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 38874:40248, ack 4190, win 307, options [nop,nop,TS val 1313937608 ecr 1617129440], length 1374
16:49:26.703142 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 40248:41622, ack 4190, win 307, options [nop,nop,TS val 1313937609 ecr 1617129440], length 1374
16:49:26.703148 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 41622:42996, ack 4190, win 307, options [nop,nop,TS val 1313937609 ecr 1617129440], length 1374
16:49:26.710201 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 16890, win 510, options [nop,nop,TS val 1617129456 ecr 1313937590], length 0
16:49:26.710214 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 42996:44370, ack 4190, win 307, options [nop,nop,TS val 1313937616 ecr 1617129456], length 1374
16:49:26.710218 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 44370:45744, ack 4190, win 307, options [nop,nop,TS val 1313937616 ecr 1617129456], length 1374
16:49:26.710568 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 18264, win 533, options [nop,nop,TS val 1617129456 ecr 1313937590], length 0
16:49:26.710577 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 45744:47118, ack 4190, win 307, options [nop,nop,TS val 1313937617 ecr 1617129456], length 1374
16:49:26.710580 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 47118:48492, ack 4190, win 307, options [nop,nop,TS val 1313937617 ecr 1617129456], length 1374
16:49:26.720991 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 19638, win 556, options [nop,nop,TS val 1617129464 ecr 1313937598], length 0
16:49:26.721006 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 48492:49866, ack 4190, win 307, options [nop,nop,TS val 1313937627 ecr 1617129464], length 1374
16:49:26.721011 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 49866:51240, ack 4190, win 307, options [nop,nop,TS val 1313937627 ecr 1617129464], length 1374
16:49:26.722681 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 21012, win 578, options [nop,nop,TS val 1617129466 ecr 1313937598], length 0
16:49:26.722692 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 51240:52614, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.722696 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 52614:53988, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.723268 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 22386, win 601, options [nop,nop,TS val 1617129466 ecr 1313937598], length 0
16:49:26.723280 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 53988:55362, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.723284 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 55362:56736, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.723597 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 23760, win 624, options [nop,nop,TS val 1617129467 ecr 1313937598], length 0
16:49:26.724171 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 56736:58110, ack 4190, win 307, options [nop,nop,TS val 1313937630 ecr 1617129467], length 1374
16:49:26.724175 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 58110:59484, ack 4190, win 307, options [nop,nop,TS val 1313937630 ecr 1617129467], length 1374
16:49:26.724457 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 25134, win 646, options [nop,nop,TS val 1617129467 ecr 1313937599], length 0
16:49:26.726081 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 26508, win 669, options [nop,nop,TS val 1617129471 ecr 1313937599], length 0
16:49:26.726391 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 27882, win 691, options [nop,nop,TS val 1617129471 ecr 1313937601], length 0
16:49:26.726962 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 29256, win 714, options [nop,nop,TS val 1617129472 ecr 1313937601], length 0
16:49:26.727614 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 30630, win 737, options [nop,nop,TS val 1617129473 ecr 1313937602], length 0
16:49:26.728084 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 32004, win 759, options [nop,nop,TS val 1617129473 ecr 1313937602], length 0
16:49:26.728507 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 59484:60858, ack 4190, win 307, options [nop,nop,TS val 1313937635 ecr 1617129473], length 1374
16:49:26.728511 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 60858:62232, ack 4190, win 307, options [nop,nop,TS val 1313937635 ecr 1617129473], length 1374
16:49:26.729531 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 33378, win 782, options [nop,nop,TS val 1617129475 ecr 1313937607], length 0
16:49:26.730002 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 34752, win 805, options [nop,nop,TS val 1617129475 ecr 1313937607], length 0
16:49:26.730340 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 62232:63606, ack 4190, win 307, options [nop,nop,TS val 1313937636 ecr 1617129473], length 1374
16:49:26.730344 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 63606:64980, ack 4190, win 307, options [nop,nop,TS val 1313937636 ecr 1617129473], length 1374
16:49:26.731398 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 64980:66354, ack 4190, win 307, options [nop,nop,TS val 1313937638 ecr 1617129473], length 1374
16:49:26.731402 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 66354:67728, ack 4190, win 307, options [nop,nop,TS val 1313937638 ecr 1617129473], length 1374
16:49:26.731634 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 36126, win 827, options [nop,nop,TS val 1617129476 ecr 1313937607], length 0
16:49:26.732955 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 67728:69102, ack 4190, win 307, options [nop,nop,TS val 1313937639 ecr 1617129473], length 1374
16:49:26.732963 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 69102:70476, ack 4190, win 307, options [nop,nop,TS val 1313937639 ecr 1617129473], length 1374
16:49:26.733956 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 37500, win 850, options [nop,nop,TS val 1617129476 ecr 1313937607], length 0
16:49:26.734242 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 38874, win 872, options [nop,nop,TS val 1617129477 ecr 1313937608], length 0
16:49:26.734653 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 40248, win 895, options [nop,nop,TS val 1617129478 ecr 1313937608], length 0
16:49:26.735042 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 70476:71850, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
16:49:26.735046 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 71850:73224, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
16:49:26.735334 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 41622, win 918, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
16:49:26.736005 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 42996, win 940, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
16:49:26.736402 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 73224:74598, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
16:49:26.736408 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 74598:75972, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
16:49:26.738561 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 44370, win 963, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
16:49:26.739539 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 45744, win 986, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
16:49:26.739882 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 47118, win 1008, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
16:49:26.740255 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 48492, win 1031, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
16:49:26.746756 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 49866, win 1053, options [nop,nop,TS val 1617129493 ecr 1313937627], length 0
16:49:26.747923 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 51240, win 1076, options [nop,nop,TS val 1617129494 ecr 1313937627], length 0
16:49:26.749083 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 52614, win 1099, options [nop,nop,TS val 1617129495 ecr 1313937629], length 0
16:49:26.750171 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 53988, win 1121, options [nop,nop,TS val 1617129496 ecr 1313937629], length 0
16:49:26.750808 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 55362, win 1144, options [nop,nop,TS val 1617129497 ecr 1313937629], length 0
16:49:26.754648 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 56736, win 1167, options [nop,nop,TS val 1617129500 ecr 1313937629], length 0
16:49:26.755985 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 58110, win 1189, options [nop,nop,TS val 1617129501 ecr 1313937630], length 0
16:49:26.758513 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 59484, win 1212, options [nop,nop,TS val 1617129502 ecr 1313937630], length 0
16:49:26.759096 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 60858, win 1234, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
16:49:26.759421 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 62232, win 1257, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
16:49:26.759755 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 63606, win 1280, options [nop,nop,TS val 1617129504 ecr 1313937636], length 0
16:49:26.760653 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 64980, win 1302, options [nop,nop,TS val 1617129505 ecr 1313937636], length 0
16:49:26.761453 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 66354, win 1325, options [nop,nop,TS val 1617129506 ecr 1313937638], length 0
16:49:26.762199 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 0
16:49:26.763547 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 36
16:49:26.763553 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 70476, win 1393, options [nop,nop,TS val 1617129508 ecr 1313937639], length 0
16:49:26.764298 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 73224, win 1438, options [nop,nop,TS val 1617129509 ecr 1313937641], length 0
16:49:26.764676 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 75972, win 1444, options [nop,nop,TS val 1617129510 ecr 1313937643], length 0
16:49:26.807754 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 75972:77346, ack 4190, win 307, options [nop,nop,TS val 1313937714 ecr 1617129473], length 1374
16:49:26.876467 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129620 ecr 1313937714], length 0
16:49:27.048760 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937955 ecr 1617129473], length 1374
16:49:27.051791 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617129762 ecr 1313937714], length 36
16:49:27.076444 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129822 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:27.371182 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130018 ecr 1313937714], length 36
16:49:27.519862 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313938426 ecr 1617129473], length 1374
16:49:27.547662 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617130293 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:27.883372 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130530 ecr 1313937714], length 36
16:49:28.511861 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313939418 ecr 1617129473], length 1374
16:49:28.538891 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617131285 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:28.907197 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617131554 ecr 1313937714], length 36
16:49:30.431864 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313941338 ecr 1617129473], length 1374
16:49:30.459127 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617133204 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:30.955388 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617133602 ecr 1313937714], length 36
16:49:34.207879 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313945114 ecr 1617129473], length 1374
16:49:34.235726 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617136981 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:35.256285 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617137954 ecr 1313937714], length 36
16:49:42.143864 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313953050 ecr 1617129473], length 1374
16:49:42.171531 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617144917 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:43.448262 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617146146 ecr 1313937714], length 36
16:49:57.503875 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313968410 ecr 1617129473], length 1374
16:49:57.533167 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617160279 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:59.832712 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617162530 ecr 1313937714], length 36
16:50:27.711882 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313998617 ecr 1617129473], length 1374
16:50:27.745163 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617190488 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:50:34.649363 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617197346 ecr 1313937714], length 36
16:51:29.151870 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1314060057 ecr 1617129473], length 1374
16:51:29.186204 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617251927 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:51:40.186590 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617262882 ecr 1313937714], length 36
16:53:32.031867 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1314182936 ecr 1617129473], length 1374
16:53:32.062310 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617374807 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:53:43.016872 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617385762 ecr 1313937714], length 36
16:54:30.276051 IP <client public ip>.31872 > <server local ip>.13317: Flags [FP.], seq 4226:5390, ack 77346, win 1444, options [nop,nop,TS val 1617432931 ecr 1313937714], length 1164
16:55:34.911864 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1314305815 ecr 1617129473], length 1374
16:55:34.941093 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617497688 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
and client side
16:49:22.693949 IP <client local ip>.31872 > <server public ip>.13317: Flags [S], seq 882075258, win 29200, options [mss 1460,sackOK,TS val 1617125446 ecr 0,nop,wscale 7], length 0
16:49:22.720391 IP <server public ip>.13317 > <client local ip>.31872: Flags [S.], seq 2026966826, ack 882075259, win 29200, options [mss 1386,sackOK,TS val 1313933607 ecr 1617125446,nop,wscale 7], length 0
16:49:22.720437 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 1, win 229, options [nop,nop,TS val 1617125472 ecr 1313933607], length 0
16:49:22.720502 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 1:22, ack 1, win 229, options [nop,nop,TS val 1617125472 ecr 1313933607], length 21
16:49:22.748570 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], ack 22, win 229, options [nop,nop,TS val 1313933635 ecr 1617125472], length 0
16:49:22.751807 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 1:22, ack 22, win 229, options [nop,nop,TS val 1313933639 ecr 1617125472], length 21
16:49:22.751823 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 22, win 229, options [nop,nop,TS val 1617125503 ecr 1313933639], length 0
16:49:22.752235 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 22:1270, ack 22, win 229, options [nop,nop,TS val 1617125504 ecr 1313933639], length 1248
16:49:22.779858 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 22:734, ack 22, win 229, options [nop,nop,TS val 1313933666 ecr 1617125503], length 712
16:49:22.820776 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 734, win 240, options [nop,nop,TS val 1617125572 ecr 1313933666], length 0
16:49:22.820910 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], ack 1270, win 248, options [nop,nop,TS val 1313933708 ecr 1617125504], length 0
16:49:22.820923 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 1270:1318, ack 734, win 240, options [nop,nop,TS val 1617125573 ecr 1313933708], length 48
16:49:22.846614 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], ack 1318, win 248, options [nop,nop,TS val 1313933734 ecr 1617125573], length 0
16:49:22.852230 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 734:1122, ack 1318, win 248, options [nop,nop,TS val 1313933739 ecr 1617125573], length 388
16:49:22.852249 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 1122, win 251, options [nop,nop,TS val 1617125604 ecr 1313933739], length 0
16:49:22.859964 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 1318:1334, ack 1122, win 251, options [nop,nop,TS val 1617125612 ecr 1313933739], length 16
16:49:22.926230 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], ack 1334, win 248, options [nop,nop,TS val 1313933813 ecr 1617125612], length 0
16:49:22.926256 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 1334:1386, ack 1122, win 251, options [nop,nop,TS val 1617125678 ecr 1313933813], length 52
16:49:22.964007 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], ack 1386, win 248, options [nop,nop,TS val 1313933851 ecr 1617125678], length 0
16:49:22.964240 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 1122:1174, ack 1386, win 248, options [nop,nop,TS val 1313933851 ecr 1617125678], length 52
16:49:22.964367 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 1386:1454, ack 1174, win 251, options [nop,nop,TS val 1617125716 ecr 1313933851], length 68
16:49:22.990580 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 1174:1258, ack 1454, win 248, options [nop,nop,TS val 1313933878 ecr 1617125716], length 84
16:49:22.990705 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 1454:2082, ack 1258, win 251, options [nop,nop,TS val 1617125742 ecr 1313933878], length 628
16:49:23.018404 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 1258:1838, ack 2082, win 268, options [nop,nop,TS val 1313933905 ecr 1617125742], length 580
16:49:23.058783 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 1838, win 262, options [nop,nop,TS val 1617125810 ecr 1313933905], length 0
16:49:24.775119 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 2082:3254, ack 1838, win 262, options [nop,nop,TS val 1617127527 ecr 1313933905], length 1172
16:49:24.831833 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 1838:1874, ack 3254, win 287, options [nop,nop,TS val 1313935719 ecr 1617127527], length 36
16:49:24.831862 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 1874, win 262, options [nop,nop,TS val 1617127583 ecr 1313935719], length 0
16:49:24.832031 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 3254:3374, ack 1874, win 262, options [nop,nop,TS val 1617127584 ecr 1313935719], length 120
16:49:24.859918 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 1874:2534, ack 3254, win 287, options [nop,nop,TS val 1313935744 ecr 1617127583], length 660
16:49:24.898914 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], ack 3374, win 287, options [nop,nop,TS val 1313935786 ecr 1617127584], length 0
16:49:24.900776 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 2534, win 273, options [nop,nop,TS val 1617127652 ecr 1313935744], length 0
16:49:24.926271 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 2534:2586, ack 3374, win 287, options [nop,nop,TS val 1313935813 ecr 1617127652], length 52
16:49:24.926284 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 2586, win 273, options [nop,nop,TS val 1617127678 ecr 1313935813], length 0
16:49:24.926452 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 3374:3902, ack 2586, win 273, options [nop,nop,TS val 1617127678 ecr 1313935813], length 528
16:49:24.952442 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], ack 3902, win 307, options [nop,nop,TS val 1313935839 ecr 1617127678], length 0
16:49:24.953230 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 2586:2694, ack 3902, win 307, options [nop,nop,TS val 1313935840 ecr 1617127678], length 108
16:49:24.993782 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 2694, win 273, options [nop,nop,TS val 1617127745 ecr 1313935840], length 0
16:49:25.019012 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 2694:2862, ack 3902, win 307, options [nop,nop,TS val 1313935906 ecr 1617127745], length 168
16:49:25.019033 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 2862, win 284, options [nop,nop,TS val 1617127771 ecr 1313935906], length 0
16:49:25.639817 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 3902:3938, ack 2862, win 284, options [nop,nop,TS val 1617128391 ecr 1313935906], length 36
16:49:25.754886 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 2862:2898, ack 3938, win 307, options [nop,nop,TS val 1313936640 ecr 1617128391], length 36
16:49:25.754915 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 3938:3974, ack 2898, win 284, options [nop,nop,TS val 1617128507 ecr 1313936640], length 36
16:49:25.782855 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 2898:2934, ack 3974, win 307, options [nop,nop,TS val 1313936670 ecr 1617128507], length 36
16:49:25.824544 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 3974:4010, ack 2934, win 284, options [nop,nop,TS val 1617128576 ecr 1313936670], length 36
16:49:25.852650 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 2934:2970, ack 4010, win 307, options [nop,nop,TS val 1313936740 ecr 1617128576], length 36
16:49:25.865736 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4010:4046, ack 2970, win 284, options [nop,nop,TS val 1617128617 ecr 1313936740], length 36
16:49:25.893292 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 2970:3006, ack 4046, win 307, options [nop,nop,TS val 1313936780 ecr 1617128617], length 36
16:49:25.935768 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 3006, win 284, options [nop,nop,TS val 1617128687 ecr 1313936780], length 0
16:49:26.202625 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4046:4082, ack 3006, win 284, options [nop,nop,TS val 1617128954 ecr 1313936780], length 36
16:49:26.264448 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 3006:3042, ack 4082, win 307, options [nop,nop,TS val 1313937152 ecr 1617128954], length 36
16:49:26.264480 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 3042, win 284, options [nop,nop,TS val 1617129016 ecr 1313937152], length 0
16:49:26.282882 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4082:4118, ack 3042, win 284, options [nop,nop,TS val 1617129035 ecr 1313937152], length 36
16:49:26.309893 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 3042:3078, ack 4118, win 307, options [nop,nop,TS val 1313937197 ecr 1617129035], length 36
16:49:26.350729 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 3078, win 284, options [nop,nop,TS val 1617129102 ecr 1313937197], length 0
16:49:26.402501 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4118:4154, ack 3078, win 284, options [nop,nop,TS val 1617129154 ecr 1313937197], length 36
16:49:26.428995 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 3078:3114, ack 4154, win 307, options [nop,nop,TS val 1313937316 ecr 1617129154], length 36
16:49:26.429016 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 3114, win 284, options [nop,nop,TS val 1617129181 ecr 1313937316], length 0
16:49:26.538794 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4154:4190, ack 3114, win 284, options [nop,nop,TS val 1617129290 ecr 1313937316], length 36
16:49:26.678342 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 3114:3150, ack 4190, win 307, options [nop,nop,TS val 1313937561 ecr 1617129290], length 36
16:49:26.678376 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 3150, win 284, options [nop,nop,TS val 1617129430 ecr 1313937561], length 0
16:49:26.685030 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 3150:4524, ack 4190, win 307, options [nop,nop,TS val 1313937571 ecr 1617129290], length 1374
16:49:26.685061 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 4524, win 307, options [nop,nop,TS val 1617129437 ecr 1313937571], length 0
16:49:26.685337 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 4524:5898, ack 4190, win 307, options [nop,nop,TS val 1313937571 ecr 1617129290], length 1374
16:49:26.685356 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 5898, win 329, options [nop,nop,TS val 1617129437 ecr 1313937571], length 0
16:49:26.685709 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 5898:7272, ack 4190, win 307, options [nop,nop,TS val 1313937571 ecr 1617129290], length 1374
16:49:26.685736 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 7272, win 352, options [nop,nop,TS val 1617129437 ecr 1313937571], length 0
16:49:26.686323 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 7272:8646, ack 4190, win 307, options [nop,nop,TS val 1313937571 ecr 1617129290], length 1374
16:49:26.686345 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 8646, win 375, options [nop,nop,TS val 1617129438 ecr 1313937571], length 0
16:49:26.687216 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 8646:10020, ack 4190, win 307, options [nop,nop,TS val 1313937572 ecr 1617129290], length 1374
16:49:26.687225 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 10020, win 397, options [nop,nop,TS val 1617129439 ecr 1313937572], length 0
16:49:26.688495 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 10020:11394, ack 4190, win 307, options [nop,nop,TS val 1313937573 ecr 1617129290], length 1374
16:49:26.688507 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 11394, win 420, options [nop,nop,TS val 1617129440 ecr 1313937573], length 0
16:49:26.688517 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 11394:12768, ack 4190, win 307, options [nop,nop,TS val 1313937573 ecr 1617129290], length 1374
16:49:26.688525 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 12768, win 443, options [nop,nop,TS val 1617129440 ecr 1313937573], length 0
16:49:26.688530 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 12768:14142, ack 4190, win 307, options [nop,nop,TS val 1313937573 ecr 1617129290], length 1374
16:49:26.688535 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 14142, win 465, options [nop,nop,TS val 1617129440 ecr 1313937573], length 0
16:49:26.688834 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 14142:15516, ack 4190, win 307, options [nop,nop,TS val 1313937574 ecr 1617129290], length 1374
16:49:26.688846 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 15516, win 488, options [nop,nop,TS val 1617129440 ecr 1313937574], length 0
16:49:26.704046 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 15516:16890, ack 4190, win 307, options [nop,nop,TS val 1313937590 ecr 1617129430], length 1374
16:49:26.704071 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 16890, win 510, options [nop,nop,TS val 1617129456 ecr 1313937590], length 0
16:49:26.704631 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 16890:18264, ack 4190, win 307, options [nop,nop,TS val 1313937590 ecr 1617129430], length 1374
16:49:26.704644 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 18264, win 533, options [nop,nop,TS val 1617129456 ecr 1313937590], length 0
16:49:26.712707 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 18264:19638, ack 4190, win 307, options [nop,nop,TS val 1313937598 ecr 1617129437], length 1374
16:49:26.712738 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 19638, win 556, options [nop,nop,TS val 1617129464 ecr 1313937598], length 0
16:49:26.714041 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 19638:21012, ack 4190, win 307, options [nop,nop,TS val 1313937598 ecr 1617129437], length 1374
16:49:26.714067 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 21012, win 578, options [nop,nop,TS val 1617129466 ecr 1313937598], length 0
16:49:26.714639 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 21012:22386, ack 4190, win 307, options [nop,nop,TS val 1313937598 ecr 1617129437], length 1374
16:49:26.714660 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 22386, win 601, options [nop,nop,TS val 1617129466 ecr 1313937598], length 0
16:49:26.714997 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 22386:23760, ack 4190, win 307, options [nop,nop,TS val 1313937598 ecr 1617129437], length 1374
16:49:26.715020 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 23760, win 624, options [nop,nop,TS val 1617129467 ecr 1313937598], length 0
16:49:26.715438 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 23760:25134, ack 4190, win 307, options [nop,nop,TS val 1313937599 ecr 1617129437], length 1374
16:49:26.715462 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 25134, win 646, options [nop,nop,TS val 1617129467 ecr 1313937599], length 0
16:49:26.718975 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 25134:26508, ack 4190, win 307, options [nop,nop,TS val 1313937599 ecr 1617129437], length 1374
16:49:26.718992 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 26508, win 669, options [nop,nop,TS val 1617129471 ecr 1313937599], length 0
16:49:26.719636 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 26508:27882, ack 4190, win 307, options [nop,nop,TS val 1313937601 ecr 1617129439], length 1374
16:49:26.719648 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 27882, win 691, options [nop,nop,TS val 1617129471 ecr 1313937601], length 0
16:49:26.720637 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 27882:29256, ack 4190, win 307, options [nop,nop,TS val 1313937601 ecr 1617129439], length 1374
16:49:26.720649 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 29256, win 714, options [nop,nop,TS val 1617129472 ecr 1313937601], length 0
16:49:26.720960 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 29256:30630, ack 4190, win 307, options [nop,nop,TS val 1313937602 ecr 1617129439], length 1374
16:49:26.720972 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 30630, win 737, options [nop,nop,TS val 1617129473 ecr 1313937602], length 0
16:49:26.721446 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 30630:32004, ack 4190, win 307, options [nop,nop,TS val 1313937602 ecr 1617129439], length 1374
16:49:26.721459 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 32004, win 759, options [nop,nop,TS val 1617129473 ecr 1313937602], length 0
16:49:26.723143 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937607 ecr 1617129440], length 1374
16:49:26.723155 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 33378, win 782, options [nop,nop,TS val 1617129475 ecr 1313937607], length 0
16:49:26.723687 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 33378:34752, ack 4190, win 307, options [nop,nop,TS val 1313937607 ecr 1617129440], length 1374
16:49:26.723699 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 34752, win 805, options [nop,nop,TS val 1617129475 ecr 1313937607], length 0
16:49:26.724294 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 34752:36126, ack 4190, win 307, options [nop,nop,TS val 1313937607 ecr 1617129440], length 1374
16:49:26.724306 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 36126, win 827, options [nop,nop,TS val 1617129476 ecr 1313937607], length 0
16:49:26.724835 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 36126:37500, ack 4190, win 307, options [nop,nop,TS val 1313937607 ecr 1617129440], length 1374
16:49:26.724847 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 37500, win 850, options [nop,nop,TS val 1617129476 ecr 1313937607], length 0
16:49:26.724859 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 37500:38874, ack 4190, win 307, options [nop,nop,TS val 1313937608 ecr 1617129440], length 1374
16:49:26.724867 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 38874, win 872, options [nop,nop,TS val 1617129477 ecr 1313937608], length 0
16:49:26.726775 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 38874:40248, ack 4190, win 307, options [nop,nop,TS val 1313937608 ecr 1617129440], length 1374
16:49:26.726785 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 40248, win 895, options [nop,nop,TS val 1617129478 ecr 1313937608], length 0
16:49:26.726793 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 40248:41622, ack 4190, win 307, options [nop,nop,TS val 1313937609 ecr 1617129440], length 1374
16:49:26.726798 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 41622, win 918, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
16:49:26.726802 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 41622:42996, ack 4190, win 307, options [nop,nop,TS val 1313937609 ecr 1617129440], length 1374
16:49:26.726807 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 42996, win 940, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
16:49:26.730611 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 42996:44370, ack 4190, win 307, options [nop,nop,TS val 1313937616 ecr 1617129456], length 1374
16:49:26.730619 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 44370, win 963, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
16:49:26.730835 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 44370:45744, ack 4190, win 307, options [nop,nop,TS val 1313937616 ecr 1617129456], length 1374
16:49:26.730847 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 45744, win 986, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
16:49:26.732371 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 45744:47118, ack 4190, win 307, options [nop,nop,TS val 1313937617 ecr 1617129456], length 1374
16:49:26.732383 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 47118, win 1008, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
16:49:26.732393 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 47118:48492, ack 4190, win 307, options [nop,nop,TS val 1313937617 ecr 1617129456], length 1374
16:49:26.732400 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 48492, win 1031, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
16:49:26.740896 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 48492:49866, ack 4190, win 307, options [nop,nop,TS val 1313937627 ecr 1617129464], length 1374
16:49:26.740910 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 49866, win 1053, options [nop,nop,TS val 1617129493 ecr 1313937627], length 0
16:49:26.742220 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 49866:51240, ack 4190, win 307, options [nop,nop,TS val 1313937627 ecr 1617129464], length 1374
16:49:26.742232 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 51240, win 1076, options [nop,nop,TS val 1617129494 ecr 1313937627], length 0
16:49:26.743463 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 51240:52614, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.743476 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 52614, win 1099, options [nop,nop,TS val 1617129495 ecr 1313937629], length 0
16:49:26.744478 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 52614:53988, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.744490 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 53988, win 1121, options [nop,nop,TS val 1617129496 ecr 1313937629], length 0
16:49:26.744975 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 53988:55362, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.744982 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 55362, win 1144, options [nop,nop,TS val 1617129497 ecr 1313937629], length 0
16:49:26.748722 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 55362:56736, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.748733 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 56736, win 1167, options [nop,nop,TS val 1617129500 ecr 1313937629], length 0
16:49:26.749615 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 56736:58110, ack 4190, win 307, options [nop,nop,TS val 1313937630 ecr 1617129467], length 1374
16:49:26.749627 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 58110, win 1189, options [nop,nop,TS val 1617129501 ecr 1313937630], length 0
16:49:26.750248 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 58110:59484, ack 4190, win 307, options [nop,nop,TS val 1313937630 ecr 1617129467], length 1374
16:49:26.750260 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 59484, win 1212, options [nop,nop,TS val 1617129502 ecr 1313937630], length 0
16:49:26.750968 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 59484:60858, ack 4190, win 307, options [nop,nop,TS val 1313937635 ecr 1617129473], length 1374
16:49:26.750980 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 60858, win 1234, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
16:49:26.751486 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 60858:62232, ack 4190, win 307, options [nop,nop,TS val 1313937635 ecr 1617129473], length 1374
16:49:26.751498 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 62232, win 1257, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
16:49:26.752095 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 62232:63606, ack 4190, win 307, options [nop,nop,TS val 1313937636 ecr 1617129473], length 1374
16:49:26.752107 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 63606, win 1280, options [nop,nop,TS val 1617129504 ecr 1313937636], length 0
16:49:26.752920 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 63606:64980, ack 4190, win 307, options [nop,nop,TS val 1313937636 ecr 1617129473], length 1374
16:49:26.752932 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 64980, win 1302, options [nop,nop,TS val 1617129505 ecr 1313937636], length 0
16:49:26.754573 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 64980:66354, ack 4190, win 307, options [nop,nop,TS val 1313937638 ecr 1617129473], length 1374
16:49:26.754585 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 66354, win 1325, options [nop,nop,TS val 1617129506 ecr 1313937638], length 0
16:49:26.755260 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 66354:67728, ack 4190, win 307, options [nop,nop,TS val 1313937638 ecr 1617129473], length 1374
16:49:26.755271 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 0
16:49:26.755493 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 36
16:49:26.756076 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 67728:69102, ack 4190, win 307, options [nop,nop,TS val 1313937639 ecr 1617129473], length 1374
16:49:26.756391 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 69102:70476, ack 4190, win 307, options [nop,nop,TS val 1313937639 ecr 1617129473], length 1374
16:49:26.756402 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 70476, win 1393, options [nop,nop,TS val 1617129508 ecr 1313937639], length 0
16:49:26.757020 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 70476:71850, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
16:49:26.757349 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 71850:73224, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
16:49:26.757360 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 73224, win 1438, options [nop,nop,TS val 1617129509 ecr 1313937641], length 0
16:49:26.757878 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 73224:74598, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
16:49:26.758457 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 74598:75972, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
16:49:26.758465 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 75972, win 1444, options [nop,nop,TS val 1617129510 ecr 1313937643], length 0
16:49:26.828268 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 75972:77346, ack 4190, win 307, options [nop,nop,TS val 1313937714 ecr 1617129473], length 1374
16:49:26.868867 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129620 ecr 1313937714], length 0
16:49:27.010733 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617129762 ecr 1313937714], length 36
16:49:27.070035 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937955 ecr 1617129473], length 1374
16:49:27.070062 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129822 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:27.266769 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130018 ecr 1313937714], length 36
16:49:27.540998 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313938426 ecr 1617129473], length 1374
16:49:27.541022 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617130293 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:27.778687 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130530 ecr 1313937714], length 36
16:49:28.532876 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313939418 ecr 1617129473], length 1374
16:49:28.532907 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617131285 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:28.802879 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617131554 ecr 1313937714], length 36
16:49:30.452830 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313941338 ecr 1617129473], length 1374
16:49:30.452861 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617133204 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:30.850734 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617133602 ecr 1313937714], length 36
16:49:34.229420 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313945114 ecr 1617129473], length 1374
16:49:34.229447 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617136981 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:35.202854 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617137954 ecr 1313937714], length 36
16:49:42.165044 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313953050 ecr 1617129473], length 1374
16:49:42.165073 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617144917 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:43.394773 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617146146 ecr 1313937714], length 36
16:49:57.527155 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313968410 ecr 1617129473], length 1374
16:49:57.527184 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617160279 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:59.778790 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617162530 ecr 1313937714], length 36
16:50:27.736331 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313998617 ecr 1617129473], length 1374
16:50:27.736360 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617190488 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:50:34.594866 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617197346 ecr 1313937714], length 36
16:51:29.175879 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1314060057 ecr 1617129473], length 1374
16:51:29.175917 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617251927 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:51:40.130722 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617262882 ecr 1313937714], length 36
16:53:32.055922 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1314182936 ecr 1617129473], length 1374
16:53:32.055963 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617374807 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:53:43.010868 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617385762 ecr 1313937714], length 36
16:54:30.179770 IP <client local ip>.31872 > <server public ip>.13317: Flags [FP.], seq 4226:5390, ack 77346, win 1444, options [nop,nop,TS val 1617432931 ecr 1313937714], length 1164
16:55:34.936179 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1314305815 ecr 1617129473], length 1374
16:55:34.936205 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617497688 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
Thanks,
--
Dominique Martinet | Asmadeus
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-14 1:55 ` Dominique Martinet
@ 2018-04-16 1:47 ` Dominique Martinet
2018-04-16 2:26 ` Eric Dumazet
2018-04-16 20:43 ` Marcelo Ricardo Leitner
1 sibling, 1 reply; 26+ messages in thread
From: Dominique Martinet @ 2018-04-16 1:47 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Michal Kubecek, netdev
Eric Dumazet wrote on Fri, Apr 13, 2018:
> That might be caused by some TS val/ecr breakage :
>
> Many acks were received by the server tcpdump,
> but none of them was accepted by TCP stack, for some reason.
>
> Try to disable TCP timestamps, it will give some hint if bug does not reproduce.
This was spot on, after disabling tcp timestamps I cannot reproduce the
hang anymore.
I've had another look at the original sequence (as seen by the server)
and I don't see much wrong; tell me what I missed:
- the replayed packet has seq 32004:33378, so the first ignored ack
would be the one with ack 33378, is that right? (meaning the server did
accept the one for 32004 and none after that)
Assuming it is, excerpt from around then (first emission of that packet then
client replies):
16:49:26.700531 IP <server>.13317 > <client>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937607 ecr 1617129440], length 1374
...
16:49:26.728084 IP <client>.31872 > <server>.13317: Flags [.], ack 32004, win 759, options [nop,nop,TS val 1617129473 ecr 1313937602], length 0
...
16:49:26.729531 IP <client>.31872 > <server>.13317: Flags [.], ack 33378, win 782, options [nop,nop,TS val 1617129475 ecr 1313937607], length 0
16:49:26.730002 IP <client>.31872 > <server>.13317: Flags [.], ack 34752, win 805, options [nop,nop,TS val 1617129475 ecr 1313937607], length 0
...
16:49:26.731634 IP <client>.31872 > <server>.13317: Flags [.], ack 36126, win 827, options [nop,nop,TS val 1617129476 ecr 1313937607], length 0
- the ecr value matches the val of the packet it acks
- the val is >= that of previous packet (won't be considered
reorder/should pass paws check?)
- even if the packets are processed in parallel and some kind of race
occurs, a "bigger" ack should ack all the previous packets, right?
- Just to make sure, I checked /proc/net/netstat for PAWSEstab but that
is 0:
TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps ArpFilter TW TWRecycled TWKilled PAWSActive PAWSEstab DelayedACKs DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops TCPHPHits TCPPureAcks TCPHPAcks TCPRenoRecovery TCPSackRecovery TCPSACKReneging TCPFACKReorder TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo TCPDSACKUndo TCPLossUndo TCPLostRetransmit TCPRenoFailures TCPSackFailures TCPLossFailures TCPFastRetrans TCPSlowStartRetrans TCPTimeouts TCPLossProbes TCPLossProbeRecovery TCPRenoRecoveryFail TCPSackRecoveryFail TCPRcvCollapsed TCPDSACKOldSent TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv TCPAbortOnData TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout TCPAbortOnLinger T
CPAbortFailed TCPMemoryPressures TCPMemoryPressuresChrono TCPSACKDiscard TCPDSACKIgnoredOld TCPDSACKIgnoredNoUndo TCPSpuriousRTOs TCPMD5NotFound TCPMD5Unexpected TCPMD5Failure TCPSackShifted TCPSackMerged TCPSackShiftFallback TCPBacklogDrop PFMemallocDrop TCPMinTTLDrop TCPDeferAcceptDrop IPReversePathFilter TCPTimeWaitOverflow TCPReqQFullDoCookies TCPReqQFullDrop TCPRetransFail TCPRcvCoalesce TCPOFOQueue TCPOFODrop TCPOFOMerge TCPChallengeACK TCPSYNChallenge TCPFastOpenActive TCPFastOpenActiveFail TCPFastOpenPassive TCPFastOpenPassiveFail TCPFastOpenListenOverflow TCPFastOpenCookieReqd TCPFastOpenBlackhole TCPSpuriousRtxHostQueues BusyPollRxPackets TCPAutoCorking TCPFromZeroWindowAdv TCPToZeroWindowAdv TCPWantZeroWindowAdv TCPSynRetrans TCPOrigDataSent TCPHystartTrainDetect TCPHystartTra
inCwnd TCPHystartDelayDetect TCPHystartDelayCwnd TCPACKSkippedSynRecv TCPACKSkippedPAWS TCPACKSkippedSeq TCPACKSkippedFinWait2 TCPACKSkippedTimeWait TCPACKSkippedChallenge TCPWinProbe TCPKeepAlive TCPMTUPFail TCPMTUPSuccess
TcpExt: 0 0 0 0 58 0 0 26 0 0 50 0 0 0 0 75402 17 201 0 0 6876848 59804 2258387 0 33 0 0 3 0 0 0 0 1 102 15 0 0 0 1306 60 386 292 10 0 0 108750 201 1 228 1 8 4 0 63 0 3 0 0 0 0 107 1 0 0 0 2834 1962 622 0 0 0 0 0 0 0 0 0 1065022 54160 0 1 3 3 0 0 0 0 0 0 0 475 0 9578 6 8 71 257 5116325 0 0 0 0 0 0 6 0 0 0 61 85 0 0
IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts InBcastPkts OutBcastPkts InOctets OutOctets InMcastOctets OutMcastOctets InBcastOctets OutBcastOctets InCsumErrors InNoECTPkts InECT1Pkts InECT0Pkts InCEPkts
IpExt: 0 0 0 0 206 0 16405602866 8921427728 0 0 57928 0 0 16121410 0 5388 0
Also, here are the per-socket stats I could find (ss -i after having
reproduced hang):
reno wscale:7,7 rto:7456 backoff:5 rtt:32.924/1.41 ato:40 mss:1374
pmtu:1500 rcvmss:1248 advmss:1448 cwnd:1 ssthresh:16
bytes_acked:32004 bytes_received:4189 segs_out:85 segs_in:54
data_segs_out:78 data_segs_in:18 send 333.9Kbps lastsnd:3912
lastrcv:11464 lastack:11387 pacing_rate 21.4Mbps delivery_rate
3.5Mbps busy:12188ms unacked:33 retrans:1/5 lost:33 rcv_rtt:37
rcv_space:29200 rcv_ssthresh:39184 notsent:28796 minrtt:24.986
Here are the same stats with tcp timestamp disabled (after running my
reproducer, e.g. outputing a big chunk of text quickly):
reno wscale:7,7 rto:228 rtt:27.267/1.423 ato:40 mss:1386 pmtu:1500
rcvmss:1248 advmss:1460 cwnd:10 ssthresh:12 bytes_acked:17311070
bytes_received:40445 segs_out:13331 segs_in:7523
data_segs_out:13279 data_segs_in:947 send 4.1Mbps lastsnd:5
lastrcv:5 lastack:5 pacing_rate 4.9Mbps delivery_rate 431.4Kbps
app_limited busy:36064ms unacked:1 retrans:0/6 rcv_rtt:9112.95
rcv_space:29233 rcv_ssthresh:39184 minrtt:25.566
So I guess lost:33 matching unacked:33 might be another hint?
I'll need a bit more time reading the code to understand what this all
implies ; feel free to beat me to it.
Thanks,
--
Dominique Martinet | Asmadeus
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-16 1:47 ` Dominique Martinet
@ 2018-04-16 2:26 ` Eric Dumazet
2018-04-16 3:55 ` Dominique Martinet
0 siblings, 1 reply; 26+ messages in thread
From: Eric Dumazet @ 2018-04-16 2:26 UTC (permalink / raw)
To: Dominique Martinet, Eric Dumazet; +Cc: Michal Kubecek, netdev
On 04/15/2018 06:47 PM, Dominique Martinet wrote:
> Also, here are the per-socket stats I could find (ss -i after having
> reproduced hang):
> reno wscale:7,7 rto:7456 backoff:5 rtt:32.924/1.41 ato:40 mss:1374
> pmtu:1500 rcvmss:1248 advmss:1448 cwnd:1 ssthresh:16
> bytes_acked:32004 bytes_received:4189 segs_out:85 segs_in:54
> data_segs_out:78 data_segs_in:18 send 333.9Kbps lastsnd:3912
> lastrcv:11464 lastack:11387 pacing_rate 21.4Mbps delivery_rate
> 3.5Mbps busy:12188ms unacked:33 retrans:1/5 lost:33 rcv_rtt:37
> rcv_space:29200 rcv_ssthresh:39184 notsent:28796 minrtt:24.986
>
ss -temoi might give us more info
Really it looks like at some point, all incoming packets are shown by tcpdump but do not reach the TCP socket anymore.
(segs_in: might be steady, look at the d0 counter shown by ss -temoi (dX : drop counters, sk->sk_drops)
Are you sure you do not have some iptables/netfilter stuff ?
While running your experiment, try on the server.
perf record -a -g -e skb:kfree_skb sleep 30
perf report
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-16 2:26 ` Eric Dumazet
@ 2018-04-16 3:55 ` Dominique Martinet
2018-04-16 4:03 ` Dominique Martinet
2018-04-16 11:01 ` Florian Westphal
0 siblings, 2 replies; 26+ messages in thread
From: Dominique Martinet @ 2018-04-16 3:55 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Michal Kubecek, netdev
Eric Dumazet wrote on Sun, Apr 15, 2018:
> Are you sure you do not have some iptables/netfilter stuff ?
I have a basic firewall setup with default rules e.g. starts with
-m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
in the INPUT chain...
That said, I just dropped it on the server to check and that seems to
workaround the issue?!
When logging everything dropped it appears to decide that the connection
is no longer established at some point, but only if there is
tcp_timestamp, just, err, how?
And certainly enough, if I restore the firewall while a connection is up
that just hangs; conntrack doesn't consider it connected anymore at some
point (but it worked for a while!)
Here's the kind of logs I get from iptables:
IN=wlp1s0 OUT= MAC=00:c2:c6:b4:7e:c7:a4:12:42:b5:5d:fc:08:00 SRC=client DST=server LEN=52 TOS=0x00 PREC=0x00 TTL=52 ID=17038 DF PROTO=TCP SPT=41558 DPT=15609 WINDOW=1212 RES=0x00 ACK URGP=0
> ss -temoi might give us more info
hang
ESTAB 0 81406 server:15609 client:41558 users:(("socat",pid=17818,fd=5)) timer:(on,48sec,11) uid:1000 ino:137253 sk:6a <->
skmem:(r0,rb369280,t0,tb147456,f2050,w104446,o0,bl0,d1) ts sack
reno wscale:7,7 rto:15168 backoff:6 rtt:36.829/6.492 ato:40
mss:1374 pmtu:1500 rcvmss:1248 advmss:1448 cwnd:1 ssthresh:16
bytes_acked:32004 bytes_received:4189 segs_out:84 segs_in:55
data_segs_out:77 data_segs_in:18 send 298.5Kbps lastsnd:12483
lastrcv:27801 lastack:27726 pacing_rate 19.1Mbps delivery_rate
4.1Mbps busy:28492ms unacked:31 retrans:1/6 lost:31 rcv_rtt:29
rcv_space:29200 rcv_ssthresh:39184 notsent:38812 minrtt:25.152
working (tcp_timestamp=0)
ESTAB 0 36 server:15080 client:32979 users:(("socat",pid=17047,fd=5)) timer:(on,226ms,0) uid:1000 ino:90917 sk:23 <->
skmem:(r0,rb369280,t0,tb1170432,f1792,w2304,o0,bl0,d3) sack reno
wscale:7,7 rto:230 rtt:29.413/5.345 ato:64 mss:1386 pmtu:1500
rcvmss:1248 advmss:1460 cwnd:4 ssthresh:3 bytes_acked:17391762
bytes_received:62397 segs_out:13964 segs_in:8642
data_segs_out:13895 data_segs_in:1494 send 1.5Mbps lastsnd:4
lastrcv:5 lastack:5 pacing_rate 1.8Mbps delivery_rate 1.2Mbps
busy:56718ms unacked:1 retrans:0/11 rcv_rtt:9112.95 rcv_space:29233
rcv_ssthresh:41680 minrtt:25.95
working (no iptables)
ESTAB 0 0 server:61460 client:20468 users:(("socat",pid=17880,fd=5)) uid:1000 ino:129982 sk:6f <->
skmem:(r0,rb369280,t0,tb1852416,f0,w0,o0,bl0,d1) ts sack reno
wscale:7,7 rto:244 rtt:43.752/7.726 ato:40 mss:1374 pmtu:1500
rcvmss:1248 advmss:1448 cwnd:10 bytes_acked:2617302
bytes_received:5441 segs_out:1929 segs_in:976 data_segs_out:1919
data_segs_in:41 send 2.5Mbps lastsnd:2734 lastrcv:2734 lastack:2705
pacing_rate 5.0Mbps delivery_rate 12.7Mbps busy:1884ms rcv_rtt:30
rcv_space:29200 rcv_ssthresh:39184 minrtt:26.156
> Really it looks like at some point, all incoming packets are shown by
> tcpdump but do not reach the TCP socket anymore.
>
> (segs_in: might be steady, look at the d0 counter shown by ss -temoi
> (dX : drop counters, sk->sk_drops)
segs_in does not increase with replays; the d1 seems stable.
> While running your experiment, try on the server.
>
> perf record -a -g -e skb:kfree_skb sleep 30
> perf report
While I understand what that should do, I am not sure why I do not get
any graph so that doesn't help tell what called kfree_skb and thus what
decided to drop the packet (although we no longer really need that
now..)
perf script just shows kfree_skb e.g.
swapper 0 [001] 237244.869321: skb:kfree_skb: skbaddr=0xffff8800360fda00 protocol=2048 location=0xffffffff817a1a77
9458e3 kfree_skb (/usr/lib/debug/lib/modules/4.16.0-300.fc28.x86_64/vmlinux)
---
So I guess that ultimately the problem is why conntrack suddenly decides
that an established connection suddenly isn't anymore, despite being
listed as established by ss..
I'm discovering `conntrack(8)`, but what strikes me as interesting is
that even that points at the connection being established (looking at a
new connection after iptables started dropping packets)
# conntrack -L | grep 21308
tcp 6 267 ESTABLISHED src=server dst=client sport=21308 dport=37552 src=client dst=server sport=37552 dport=21308 [ASSURED] mark=0 use=1
compared to another that isn't dropped (the old connection without
tcp_timestamp)
tcp 6 299 ESTABLISHED src=server dst=client sport=15080 dport=32979 src=client dst=server sport=32979 dport=15080 [ASSURED] mark=0 use=1
The expect/dying/unconfirmed tables all are empty.
. . . Oh, there is something interesting there, the connection doesn't
come up with -G?
working:
conntrack -G --protonum tcp --src server --dst client --sport 15080 --dport 32979
tcp 6 299 ESTABLISHED src=server dst=client sport=15080 dport=32979 src=client dst=server sport=32979 dport=15080 [ASSURED] mark=0 use=3
conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown.
hang:
# conntrack -G --protonum tcp --src server --dst client --sport 21308 --dport 37552
conntrack v1.4.4 (conntrack-tools): 0 flow entries have been shown.
So something happened that makes it show up in -L (table dump) but not
when querying...?
And only when there is enough traffic: I have previously kept such a
connection without workaround for hours just fine as long as I made sure
not to display more than a screen at a time.
Thanks again,
--
Dominique Martinet | Asmadeus
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-16 3:55 ` Dominique Martinet
@ 2018-04-16 4:03 ` Dominique Martinet
2018-04-16 11:01 ` Florian Westphal
1 sibling, 0 replies; 26+ messages in thread
From: Dominique Martinet @ 2018-04-16 4:03 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Michal Kubecek, netdev
Dominique Martinet wrote on Mon, Apr 16, 2018:
> . . . Oh, there is something interesting there, the connection doesn't
> come up with -G?
Hm, sorry, I take this last part back. I cannot reproduce -G not working
reliably.
I'll dig around the conntrack table a bit more.
--
Dominique Martinet | Asmadeus
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-16 3:55 ` Dominique Martinet
2018-04-16 4:03 ` Dominique Martinet
@ 2018-04-16 11:01 ` Florian Westphal
2018-04-17 3:52 ` Dominique Martinet
1 sibling, 1 reply; 26+ messages in thread
From: Florian Westphal @ 2018-04-16 11:01 UTC (permalink / raw)
To: Dominique Martinet; +Cc: Eric Dumazet, Michal Kubecek, netdev
Dominique Martinet <asmadeus@codewreck.org> wrote:
> Eric Dumazet wrote on Sun, Apr 15, 2018:
> > Are you sure you do not have some iptables/netfilter stuff ?
>
> I have a basic firewall setup with default rules e.g. starts with
> -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
> in the INPUT chain...
> That said, I just dropped it on the server to check and that seems to
> workaround the issue?!
> When logging everything dropped it appears to decide that the connection
> is no longer established at some point, but only if there is
> tcp_timestamp, just, err, how?
>
> And certainly enough, if I restore the firewall while a connection is up
> that just hangs; conntrack doesn't consider it connected anymore at some
> point (but it worked for a while!)
>
> Here's the kind of logs I get from iptables:
> IN=wlp1s0 OUT= MAC=00:c2:c6:b4:7e:c7:a4:12:42:b5:5d:fc:08:00 SRC=client DST=server LEN=52 TOS=0x00 PREC=0x00 TTL=52 ID=17038 DF PROTO=TCP SPT=41558 DPT=15609 WINDOW=1212 RES=0x00 ACK URGP=0
You could do
echo 6 > /proc/sys/net/netfilter/nf_conntrack_log_invalid
to have conntrack log when/why it thinks packet is invalid.
You can also set
echo 1 > /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal
which stops conntrack from marking packets with out-of-window
acks as invalid.
(Earlier email implies this is related to timestamps, but unfortunately
to best of my knowledge conntrack doesn't look at tcp timestamps).
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-14 1:55 ` Dominique Martinet
2018-04-16 1:47 ` Dominique Martinet
@ 2018-04-16 20:43 ` Marcelo Ricardo Leitner
1 sibling, 0 replies; 26+ messages in thread
From: Marcelo Ricardo Leitner @ 2018-04-16 20:43 UTC (permalink / raw)
To: Dominique Martinet; +Cc: Eric Dumazet, Michal Kubecek, netdev, Florian Westphal
On Sat, Apr 14, 2018 at 03:55:15AM +0200, Dominique Martinet wrote:
> 16:49:26.724457 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 25134, win 646, options [nop,nop,TS val 1617129467 ecr 1313937599], length 0
> 16:49:26.726081 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 26508, win 669, options [nop,nop,TS val 1617129471 ecr 1313937599], length 0
> 16:49:26.726391 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 27882, win 691, options [nop,nop,TS val 1617129471 ecr 1313937601], length 0
> 16:49:26.726962 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 29256, win 714, options [nop,nop,TS val 1617129472 ecr 1313937601], length 0
> 16:49:26.727614 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 30630, win 737, options [nop,nop,TS val 1617129473 ecr 1313937602], length 0
> 16:49:26.728084 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 32004, win 759, options [nop,nop,TS val 1617129473 ecr 1313937602], length 0
[A] (packet above)
> 16:49:26.728507 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 59484:60858, ack 4190, win 307, options [nop,nop,TS val 1313937635 ecr 1617129473], length 1374
> 16:49:26.728511 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 60858:62232, ack 4190, win 307, options [nop,nop,TS val 1313937635 ecr 1617129473], length 1374
> 16:49:26.729531 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 33378, win 782, options [nop,nop,TS val 1617129475 ecr 1313937607], length 0
> 16:49:26.730002 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 34752, win 805, options [nop,nop,TS val 1617129475 ecr 1313937607], length 0
> 16:49:26.730340 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 62232:63606, ack 4190, win 307, options [nop,nop,TS val 1313937636 ecr 1617129473], length 1374
> 16:49:26.730344 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 63606:64980, ack 4190, win 307, options [nop,nop,TS val 1313937636 ecr 1617129473], length 1374
> 16:49:26.731398 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 64980:66354, ack 4190, win 307, options [nop,nop,TS val 1313937638 ecr 1617129473], length 1374
> 16:49:26.731402 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 66354:67728, ack 4190, win 307, options [nop,nop,TS val 1313937638 ecr 1617129473], length 1374
> 16:49:26.731634 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 36126, win 827, options [nop,nop,TS val 1617129476 ecr 1313937607], length 0
> 16:49:26.732955 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 67728:69102, ack 4190, win 307, options [nop,nop,TS val 1313937639 ecr 1617129473], length 1374
> 16:49:26.732963 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 69102:70476, ack 4190, win 307, options [nop,nop,TS val 1313937639 ecr 1617129473], length 1374
> 16:49:26.733956 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 37500, win 850, options [nop,nop,TS val 1617129476 ecr 1313937607], length 0
> 16:49:26.734242 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 38874, win 872, options [nop,nop,TS val 1617129477 ecr 1313937608], length 0
> 16:49:26.734653 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 40248, win 895, options [nop,nop,TS val 1617129478 ecr 1313937608], length 0
> 16:49:26.735042 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 70476:71850, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
> 16:49:26.735046 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 71850:73224, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
> 16:49:26.735334 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 41622, win 918, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
> 16:49:26.736005 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 42996, win 940, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
> 16:49:26.736402 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 73224:74598, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
> 16:49:26.736408 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 74598:75972, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
> 16:49:26.738561 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 44370, win 963, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
> 16:49:26.739539 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 45744, win 986, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
> 16:49:26.739882 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 47118, win 1008, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
> 16:49:26.740255 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 48492, win 1031, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
> 16:49:26.746756 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 49866, win 1053, options [nop,nop,TS val 1617129493 ecr 1313937627], length 0
> 16:49:26.747923 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 51240, win 1076, options [nop,nop,TS val 1617129494 ecr 1313937627], length 0
> 16:49:26.749083 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 52614, win 1099, options [nop,nop,TS val 1617129495 ecr 1313937629], length 0
> 16:49:26.750171 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 53988, win 1121, options [nop,nop,TS val 1617129496 ecr 1313937629], length 0
> 16:49:26.750808 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 55362, win 1144, options [nop,nop,TS val 1617129497 ecr 1313937629], length 0
> 16:49:26.754648 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 56736, win 1167, options [nop,nop,TS val 1617129500 ecr 1313937629], length 0
> 16:49:26.755985 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 58110, win 1189, options [nop,nop,TS val 1617129501 ecr 1313937630], length 0
> 16:49:26.758513 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 59484, win 1212, options [nop,nop,TS val 1617129502 ecr 1313937630], length 0
> 16:49:26.759096 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 60858, win 1234, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
> 16:49:26.759421 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 62232, win 1257, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
> 16:49:26.759755 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 63606, win 1280, options [nop,nop,TS val 1617129504 ecr 1313937636], length 0
> 16:49:26.760653 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 64980, win 1302, options [nop,nop,TS val 1617129505 ecr 1313937636], length 0
> 16:49:26.761453 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 66354, win 1325, options [nop,nop,TS val 1617129506 ecr 1313937638], length 0
> 16:49:26.762199 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 0
> 16:49:26.763547 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 36
> 16:49:26.763553 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 70476, win 1393, options [nop,nop,TS val 1617129508 ecr 1313937639], length 0
> 16:49:26.764298 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 73224, win 1438, options [nop,nop,TS val 1617129509 ecr 1313937641], length 0
> 16:49:26.764676 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 75972, win 1444, options [nop,nop,TS val 1617129510 ecr 1313937643], length 0
> 16:49:26.807754 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 75972:77346, ack 4190, win 307, options [nop,nop,TS val 1313937714 ecr 1617129473], length 1374
> 16:49:26.876467 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129620 ecr 1313937714], length 0
> 16:49:27.048760 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937955 ecr 1617129473], length 1374
Probably bogus retrans for conntrack, but OUTPUT is not filtered.
> 16:49:27.051791 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617129762 ecr 1313937714], length 36
> 16:49:27.076444 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129822 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
I guess the issue lies with the dup-sack here.
AFAIR Disabling timestamps also disables SACK, which would avoid the
issue as conntrack will not see these as they won't happen anymore.
Then it gets locked into a loopy behavior:
> 16:49:27.371182 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130018 ecr 1313937714], length 36
> 16:49:27.519862 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313938426 ecr 1617129473], length 1374
> 16:49:27.547662 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617130293 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
> 16:49:27.883372 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130530 ecr 1313937714], length 36
Cycle 1
> 16:49:28.511861 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313939418 ecr 1617129473], length 1374
> 16:49:28.538891 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617131285 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
> 16:49:28.907197 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617131554 ecr 1313937714], length 36
Cycle 2
> 16:49:30.431864 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313941338 ecr 1617129473], length 1374
> 16:49:30.459127 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617133204 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
> 16:49:30.955388 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617133602 ecr 1313937714], length 36
Cycle 3...
Based on server RTO.
But I can't tell why packet [A] and subsequent packets from client
never made to the server's socket.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-16 11:01 ` Florian Westphal
@ 2018-04-17 3:52 ` Dominique Martinet
2018-04-17 5:28 ` Eric Dumazet
0 siblings, 1 reply; 26+ messages in thread
From: Dominique Martinet @ 2018-04-17 3:52 UTC (permalink / raw)
To: Florian Westphal
Cc: Eric Dumazet, Marcelo Ricardo Leitner, Michal Kubecek, netdev
[-- Attachment #1: Type: text/plain, Size: 8673 bytes --]
Thank you for the hints. Out of order reply.
Florian Westphal wrote on Mon, Apr 16, 2018:
> echo 1 > /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal
>
> which stops conntrack from marking packets with out-of-window
> acks as invalid.
That worked, the connection stays established with this set.
> echo 6 > /proc/sys/net/netfilter/nf_conntrack_log_invalid
Took me a while to figure how to log that properly (why do distros not
ship ulogd?!) but that is helpful.
I have attached the full logs to this mail as text attachments in case I
missed something in my excerpts
Specifically, there are many outbound invalid packets, but as Marcelo
pointed out there is no conntrack netfilter for outbound packets...
The first invalid incoming packet is that one (json output seemed to be
the most convenient as it has tcp seq/ackseq and the oob.prefix message,
feel free to ask for some other format)
{"timestamp": "2018-04-17T10:29:14.956485", "dvc": "Netfilter",
"raw.pktlen": 52, "raw.pktcount": 1, "oob.prefix": "nf_ct_tcp: ACK is
over the upper bound (ACKed data not seen yet) ", "oob.time.sec":
1523928554, "oob.time.usec": 956485, "oob.mark": 0, "oob.hook": 0,
"oob.family": 2, "oob.protocol": 2048, "raw.label": 0, "ip.protocol": 6,
"ip.tos": 0, "ip.ttl": 52, "ip.totlen": 52, "ip.ihl": 5, "ip.csum":
50909, "ip.id": 30328, "ip.fragoff": 16384, "src_port": 30280,
"dest_port": 29543, "tcp.seq": 4048786673, "tcp.ackseq": 3024392506,
"tcp.window": 781, "tcp.offset": 0, "tcp.reserved": 0, "tcp.urg": 0,
"tcp.ack": 1, "tcp.psh": 0, "tcp.rst": 0, "tcp.syn": 0, "tcp.fin": 0,
"tcp.res1": 0, "tcp.res2": 0, "tcp.csum": 17752, "oob.in": "",
"oob.out": "", "src_ip": "client", "dest_ip": "server"}
which is the first ack that isn't seen alright (sent packet, blocked
ack, replays):
10:29:14.926115 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681965809 ecr 1634528339], length 1374
...
10:29:14.956485 IP client.30280 > server.29543: Flags [.], ack 3024392506, win 781, options [nop,nop,TS val 1634528369 ecr 681965809], length 0
...
10:29:15.255489 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681966138 ecr 1634528369], length 1374
10:29:15.281581 IP client.30280 > server.29543: Flags [.], ack 3024436474, win 1444, options [nop,nop,TS val 1634528697 ecr 681965904,nop,nop,sack 1 {3024391132:3024392506}], length 0
10:29:15.555466 IP client.30280 > server.29543: Flags [P.], seq 4048786673:4048786709, ack 3024436474, win 1444, options [nop,nop,TS val 1634528869 ecr 681965904], length 36
10:29:15.719510 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681966602 ecr 1634528369], length 1374
So now I guess the question is why netfilter think this is over the
upper bound.
..And surely enough the answer is within the invalid outbound packets
that aren't considered in the conntrack state machine, the first of whih
is this one:
{"timestamp": "2018-04-17T10:29:14.926143", "dvc": "Netfilter",
"raw.pktlen": 2800, "raw.pktcount": 1, "oob.prefix": "nf_ct_tcp: SEQ is
over the upper bound (over the window of the receiver)", "oob.time.sec":
1523928554, "oob.time.usec": 926143, "oob.mark": 0, "oob.hook": 0,
"oob.family": 2, "oob.protocol": 2048, "oob.uid": 1000, "oob.gid": 100,
"raw.label": 0, "ip.protocol": 6, "ip.tos": 8, "ip.ttl": 64,
"ip.totlen": 2800, "ip.ihl": 5, "ip.csum": 47600, "ip.id": 27809,
"ip.fragoff": 16384, "src_port": 29543, "dest_port": 30280, "tcp.seq":
3024391132, "tcp.ackseq": 4048786673, "tcp.window": 307, "tcp.offset":
0, "tcp.reserved": 0, "tcp.urg": 0, "tcp.ack": 1, "tcp.psh": 0,
"tcp.rst": 0, "tcp.syn": 0, "tcp.fin": 0, "tcp.res1": 0, "tcp.res2": 0,
"tcp.csum": 5201, "oob.in": "", "oob.out": "", "src_ip": "server",
"dest_ip": "client"}
which sequence matches the start seq of the first packet we don't ack,
so that's why conntrack doesn't recongnize that as a valid ack.
Here's some context before that packet:
10:29:14.921305 IP server.29543 > client.30280: Flags [.], seq 3024378314:3024379688, ack 4048786673, win 307, options [nop,nop,TS val 681965804 ecr 1634528336], length 1374
10:29:14.921311 IP server.29543 > client.30280: Flags [.], seq 3024379688:3024381062, ack 4048786673, win 307, options [nop,nop,TS val 681965804 ecr 1634528336], length 1374
10:29:14.922166 IP client.30280 > server.29543: Flags [.], ack 3024365948, win 329, options [nop,nop,TS val 1634528336 ecr 681965778], length 0
10:29:14.922178 IP server.29543 > client.30280: Flags [.], seq 3024381062:3024382436, ack 4048786673, win 307, options [nop,nop,TS val 681965805 ecr 1634528336], length 1374
10:29:14.922182 IP server.29543 > client.30280: Flags [.], seq 3024382436:3024383810, ack 4048786673, win 307, options [nop,nop,TS val 681965805 ecr 1634528336], length 1374
10:29:14.923848 IP client.30280 > server.29543: Flags [.], ack 3024367322, win 352, options [nop,nop,TS val 1634528338 ecr 681965778], length 0
10:29:14.923860 IP server.29543 > client.30280: Flags [.], seq 3024383810:3024385184, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
10:29:14.923863 IP server.29543 > client.30280: Flags [.], seq 3024385184:3024386558, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
10:29:14.924175 IP client.30280 > server.29543: Flags [.], ack 3024368696, win 375, options [nop,nop,TS val 1634528338 ecr 681965778], length 0
10:29:14.924187 IP server.29543 > client.30280: Flags [.], seq 3024386558:3024387932, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
10:29:14.924190 IP server.29543 > client.30280: Flags [.], seq 3024387932:3024389306, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
10:29:14.925410 IP client.30280 > server.29543: Flags [.], ack 3024370070, win 397, options [nop,nop,TS val 1634528339 ecr 681965779], length 0
10:29:14.925422 IP server.29543 > client.30280: Flags [.], seq 3024389306:3024390680, ack 4048786673, win 307, options [nop,nop,TS val 681965808 ecr 1634528339], length 1374
10:29:14.925425 IP server.29543 > client.30280: Flags [P.], seq 3024390680:3024391132, ack 4048786673, win 307, options [nop,nop,TS val 681965808 ecr 1634528339], length 452
10:29:14.926030 IP client.30280 > server.29543: Flags [.], ack 3024371444, win 420, options [nop,nop,TS val 1634528339 ecr 681965779], length 0
10:29:14.926115 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681965809 ecr 1634528339], length 1374
But the way I see it, (3024392506-3024371444) = 21064 is smaller than
420*64 = 26880.
Even looking at the previous (ack 3024370070, win 397) we get 22436 and
25408 which still work, same with ack 3024368696, win 375 (23810 <
24000)
So that means conntrack is looking at an ack at least as old as ack
3024367322, win 352?...
Is there any way to confirm that? In the code it looks like we match
! before(seq, sender->td_maxend + 1)
I turned pr_debug on in tcp_in_window() for another try and it's a bit
mangled because the information on multiple lines and the function is
called in parallel but it looks like I do have some seq > maxend +1
Although it's weird, the maxend was set WAY earlier apparently?
Apr 17 11:13:14 res=1 sender end=1913287798 maxend=1913316998 maxwin=29312 receiver end=505004284 maxend=505033596 maxwin=29200
then window decreased drastically e.g. previous ack just before refusal:
Apr 17 11:13:53 seq=1913292311 ack=505007789+(0) sack=505007789+(0) win=284 end=1913292311
Apr 17 11:13:53 sender end=1913292311 maxend=1913331607 maxwin=284 scale=0 receiver end=505020155 maxend=505033596 maxwin=39296 scale=7
first res=0 (refused) packet:
Apr 17 11:13:53 odin kernel: seq=505042139 ack=1913292311+(0) ack=1913292311+(0) win=307 end=505044887
Apr 17 11:13:53 res=0 sender end=505033895 maxend=505033596 maxwin=39296 receiver end=1913292311 maxend=1913331607 maxwin=674
> (Earlier email implies this is related to timestamps, but unfortunately
> to best of my knowledge conntrack doesn't look at tcp timestamps).
tcp_timestamp introduces other changes in behaviour so it might still be
relateed.
Recap of attachments:
- new tcpdump output of a failed attempt
- matching invalid packet dumped in json format
- logs of tcp_in_window function unfortunately of a different attempt
Happy to provide new/different logs if it would be useful. Once again
thanks for the extra eyes and useful commands.
--
Dominique Martinet | Asmadeus
[-- Attachment #2: tcpdump --]
[-- Type: text/plain, Size: 27334 bytes --]
10:29:13.429532 IP server.29543 > client.30280: Flags [S], seq 3024360198, win 29200, options [mss 1460,sackOK,TS val 681964312 ecr 0,nop,wscale 7], length 0
10:29:13.712771 IP client.30280 > server.29543: Flags [S], seq 4048782683, win 29200, options [mss 1386,sackOK,TS val 1634527123 ecr 0,nop,wscale 7], length 0
10:29:13.712791 IP server.29543 > client.30280: Flags [S.], seq 3024360198, ack 4048782684, win 29200, options [mss 1460,sackOK,TS val 681964596 ecr 1634527123,nop,wscale 7], length 0
10:29:13.738441 IP client.30280 > server.29543: Flags [.], ack 3024360199, win 229, options [nop,nop,TS val 1634527153 ecr 681964596], length 0
10:29:13.740587 IP client.30280 > server.29543: Flags [P.], seq 4048782684:4048782705, ack 3024360199, win 229, options [nop,nop,TS val 1634527154 ecr 681964596], length 21
10:29:13.740610 IP server.29543 > client.30280: Flags [.], ack 4048782705, win 229, options [nop,nop,TS val 681964623 ecr 1634527154], length 0
10:29:13.744143 IP server.29543 > client.30280: Flags [P.], seq 3024360199:3024360220, ack 4048782705, win 229, options [nop,nop,TS val 681964627 ecr 1634527154], length 21
10:29:13.769430 IP client.30280 > server.29543: Flags [.], ack 3024360220, win 229, options [nop,nop,TS val 1634527185 ecr 681964627], length 0
10:29:13.769458 IP server.29543 > client.30280: Flags [P.], seq 3024360220:3024360932, ack 4048782705, win 229, options [nop,nop,TS val 681964652 ecr 1634527185], length 712
10:29:13.772401 IP client.30280 > server.29543: Flags [P.], seq 4048782705:4048783953, ack 3024360220, win 229, options [nop,nop,TS val 1634527187 ecr 681964627], length 1248
10:29:13.813512 IP server.29543 > client.30280: Flags [.], ack 4048783953, win 248, options [nop,nop,TS val 681964696 ecr 1634527187], length 0
10:29:13.835970 IP client.30280 > server.29543: Flags [.], ack 3024360932, win 240, options [nop,nop,TS val 1634527251 ecr 681964652], length 0
10:29:13.841601 IP client.30280 > server.29543: Flags [P.], seq 4048783953:4048784001, ack 3024360932, win 240, options [nop,nop,TS val 1634527255 ecr 681964696], length 48
10:29:13.841626 IP server.29543 > client.30280: Flags [.], ack 4048784001, win 248, options [nop,nop,TS val 681964724 ecr 1634527255], length 0
10:29:13.847085 IP server.29543 > client.30280: Flags [P.], seq 3024360932:3024361320, ack 4048784001, win 248, options [nop,nop,TS val 681964730 ecr 1634527255], length 388
10:29:13.872177 IP client.30280 > server.29543: Flags [.], ack 3024361320, win 251, options [nop,nop,TS val 1634527287 ecr 681964730], length 0
10:29:13.906359 IP client.30280 > server.29543: Flags [P.], seq 4048784001:4048784017, ack 3024361320, win 251, options [nop,nop,TS val 1634527321 ecr 681964730], length 16
10:29:13.946509 IP server.29543 > client.30280: Flags [.], ack 4048784017, win 248, options [nop,nop,TS val 681964829 ecr 1634527321], length 0
10:29:13.979888 IP client.30280 > server.29543: Flags [P.], seq 4048784017:4048784069, ack 3024361320, win 251, options [nop,nop,TS val 1634527387 ecr 681964829], length 52
10:29:13.979896 IP server.29543 > client.30280: Flags [.], ack 4048784069, win 248, options [nop,nop,TS val 681964863 ecr 1634527387], length 0
10:29:13.980189 IP server.29543 > client.30280: Flags [P.], seq 3024361320:3024361372, ack 4048784069, win 248, options [nop,nop,TS val 681964863 ecr 1634527387], length 52
10:29:14.007950 IP client.30280 > server.29543: Flags [P.], seq 4048784069:4048784137, ack 3024361372, win 251, options [nop,nop,TS val 1634527422 ecr 681964863], length 68
10:29:14.008365 IP server.29543 > client.30280: Flags [P.], seq 3024361372:3024361456, ack 4048784137, win 248, options [nop,nop,TS val 681964891 ecr 1634527422], length 84
10:29:14.035877 IP client.30280 > server.29543: Flags [P.], seq 4048784137:4048784765, ack 3024361456, win 251, options [nop,nop,TS val 1634527449 ecr 681964891], length 628
10:29:14.036743 IP server.29543 > client.30280: Flags [P.], seq 3024361456:3024362036, ack 4048784765, win 268, options [nop,nop,TS val 681964920 ecr 1634527449], length 580
10:29:14.075014 IP client.30280 > server.29543: Flags [P.], seq 4048784765:4048785937, ack 3024362036, win 262, options [nop,nop,TS val 1634527489 ecr 681964920], length 1172
10:29:14.076317 IP server.29543 > client.30280: Flags [P.], seq 3024362036:3024362072, ack 4048785937, win 287, options [nop,nop,TS val 681964959 ecr 1634527489], length 36
10:29:14.102624 IP client.30280 > server.29543: Flags [P.], seq 4048785937:4048786057, ack 3024362072, win 262, options [nop,nop,TS val 1634527517 ecr 681964959], length 120
10:29:14.102857 IP server.29543 > client.30280: Flags [P.], seq 3024362072:3024362732, ack 4048786057, win 287, options [nop,nop,TS val 681964986 ecr 1634527517], length 660
10:29:14.171874 IP client.30280 > server.29543: Flags [.], ack 3024362732, win 273, options [nop,nop,TS val 1634527584 ecr 681964986], length 0
10:29:14.171890 IP server.29543 > client.30280: Flags [P.], seq 3024362732:3024362784, ack 4048786057, win 287, options [nop,nop,TS val 681965055 ecr 1634527584], length 52
10:29:14.199156 IP client.30280 > server.29543: Flags [.], ack 3024362784, win 273, options [nop,nop,TS val 1634527614 ecr 681965055], length 0
10:29:14.200359 IP client.30280 > server.29543: Flags [P.], seq 4048786057:4048786585, ack 3024362784, win 273, options [nop,nop,TS val 1634527615 ecr 681965055], length 528
10:29:14.201431 IP server.29543 > client.30280: Flags [P.], seq 3024362784:3024362892, ack 4048786585, win 307, options [nop,nop,TS val 681965084 ecr 1634527615], length 108
10:29:14.227696 IP client.30280 > server.29543: Flags [P.], seq 4048786585:4048786637, ack 3024362892, win 273, options [nop,nop,TS val 1634527643 ecr 681965084], length 52
10:29:14.227737 IP server.29543 > client.30280: Flags [P.], seq 3024362892:3024362992, ack 4048786637, win 307, options [nop,nop,TS val 681965111 ecr 1634527643], length 100
10:29:14.296663 IP client.30280 > server.29543: Flags [.], ack 3024362992, win 273, options [nop,nop,TS val 1634527709 ecr 681965111], length 0
10:29:14.296678 IP server.29543 > client.30280: Flags [P.], seq 3024362992:3024363164, ack 4048786637, win 307, options [nop,nop,TS val 681965179 ecr 1634527709], length 172
10:29:14.322190 IP client.30280 > server.29543: Flags [.], ack 3024363164, win 284, options [nop,nop,TS val 1634527738 ecr 681965179], length 0
10:29:14.891087 IP client.30280 > server.29543: Flags [P.], seq 4048786637:4048786673, ack 3024363164, win 284, options [nop,nop,TS val 1634528306 ecr 681965179], length 36
10:29:14.891605 IP server.29543 > client.30280: Flags [P.], seq 3024363164:3024363200, ack 4048786673, win 307, options [nop,nop,TS val 681965774 ecr 1634528306], length 36
10:29:14.895150 IP server.29543 > client.30280: Flags [.], seq 3024363200:3024364574, ack 4048786673, win 307, options [nop,nop,TS val 681965778 ecr 1634528306], length 1374
10:29:14.895201 IP server.29543 > client.30280: Flags [.], seq 3024364574:3024365948, ack 4048786673, win 307, options [nop,nop,TS val 681965778 ecr 1634528306], length 1374
10:29:14.895329 IP server.29543 > client.30280: Flags [.], seq 3024365948:3024367322, ack 4048786673, win 307, options [nop,nop,TS val 681965778 ecr 1634528306], length 1374
10:29:14.895334 IP server.29543 > client.30280: Flags [.], seq 3024367322:3024368696, ack 4048786673, win 307, options [nop,nop,TS val 681965778 ecr 1634528306], length 1374
10:29:14.896045 IP server.29543 > client.30280: Flags [.], seq 3024368696:3024370070, ack 4048786673, win 307, options [nop,nop,TS val 681965779 ecr 1634528306], length 1374
10:29:14.896056 IP server.29543 > client.30280: Flags [.], seq 3024370070:3024371444, ack 4048786673, win 307, options [nop,nop,TS val 681965779 ecr 1634528306], length 1374
10:29:14.896058 IP server.29543 > client.30280: Flags [.], seq 3024371444:3024372818, ack 4048786673, win 307, options [nop,nop,TS val 681965779 ecr 1634528306], length 1374
10:29:14.896380 IP server.29543 > client.30280: Flags [.], seq 3024372818:3024374192, ack 4048786673, win 307, options [nop,nop,TS val 681965779 ecr 1634528306], length 1374
10:29:14.896384 IP server.29543 > client.30280: Flags [.], seq 3024374192:3024375566, ack 4048786673, win 307, options [nop,nop,TS val 681965779 ecr 1634528306], length 1374
10:29:14.917086 IP client.30280 > server.29543: Flags [.], ack 3024363200, win 284, options [nop,nop,TS val 1634528332 ecr 681965774], length 0
10:29:14.917104 IP server.29543 > client.30280: Flags [.], seq 3024375566:3024376940, ack 4048786673, win 307, options [nop,nop,TS val 681965800 ecr 1634528332], length 1374
10:29:14.917110 IP server.29543 > client.30280: Flags [.], seq 3024376940:3024378314, ack 4048786673, win 307, options [nop,nop,TS val 681965800 ecr 1634528332], length 1374
10:29:14.921290 IP client.30280 > server.29543: Flags [.], ack 3024364574, win 307, options [nop,nop,TS val 1634528336 ecr 681965778], length 0
10:29:14.921305 IP server.29543 > client.30280: Flags [.], seq 3024378314:3024379688, ack 4048786673, win 307, options [nop,nop,TS val 681965804 ecr 1634528336], length 1374
10:29:14.921311 IP server.29543 > client.30280: Flags [.], seq 3024379688:3024381062, ack 4048786673, win 307, options [nop,nop,TS val 681965804 ecr 1634528336], length 1374
10:29:14.922166 IP client.30280 > server.29543: Flags [.], ack 3024365948, win 329, options [nop,nop,TS val 1634528336 ecr 681965778], length 0
10:29:14.922178 IP server.29543 > client.30280: Flags [.], seq 3024381062:3024382436, ack 4048786673, win 307, options [nop,nop,TS val 681965805 ecr 1634528336], length 1374
10:29:14.922182 IP server.29543 > client.30280: Flags [.], seq 3024382436:3024383810, ack 4048786673, win 307, options [nop,nop,TS val 681965805 ecr 1634528336], length 1374
10:29:14.923848 IP client.30280 > server.29543: Flags [.], ack 3024367322, win 352, options [nop,nop,TS val 1634528338 ecr 681965778], length 0
10:29:14.923860 IP server.29543 > client.30280: Flags [.], seq 3024383810:3024385184, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
10:29:14.923863 IP server.29543 > client.30280: Flags [.], seq 3024385184:3024386558, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
10:29:14.924175 IP client.30280 > server.29543: Flags [.], ack 3024368696, win 375, options [nop,nop,TS val 1634528338 ecr 681965778], length 0
10:29:14.924187 IP server.29543 > client.30280: Flags [.], seq 3024386558:3024387932, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
10:29:14.924190 IP server.29543 > client.30280: Flags [.], seq 3024387932:3024389306, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
10:29:14.925410 IP client.30280 > server.29543: Flags [.], ack 3024370070, win 397, options [nop,nop,TS val 1634528339 ecr 681965779], length 0
10:29:14.925422 IP server.29543 > client.30280: Flags [.], seq 3024389306:3024390680, ack 4048786673, win 307, options [nop,nop,TS val 681965808 ecr 1634528339], length 1374
10:29:14.925425 IP server.29543 > client.30280: Flags [P.], seq 3024390680:3024391132, ack 4048786673, win 307, options [nop,nop,TS val 681965808 ecr 1634528339], length 452
10:29:14.926030 IP client.30280 > server.29543: Flags [.], ack 3024371444, win 420, options [nop,nop,TS val 1634528339 ecr 681965779], length 0
10:29:14.926115 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681965809 ecr 1634528339], length 1374
10:29:14.926122 IP server.29543 > client.30280: Flags [.], seq 3024392506:3024393880, ack 4048786673, win 307, options [nop,nop,TS val 681965809 ecr 1634528339], length 1374
10:29:14.926353 IP client.30280 > server.29543: Flags [.], ack 3024372818, win 443, options [nop,nop,TS val 1634528339 ecr 681965779], length 0
10:29:14.926924 IP client.30280 > server.29543: Flags [.], ack 3024374192, win 465, options [nop,nop,TS val 1634528339 ecr 681965779], length 0
10:29:14.927677 IP client.30280 > server.29543: Flags [.], ack 3024375566, win 488, options [nop,nop,TS val 1634528340 ecr 681965779], length 0
10:29:14.928109 IP server.29543 > client.30280: Flags [.], seq 3024393880:3024395254, ack 4048786673, win 307, options [nop,nop,TS val 681965811 ecr 1634528340], length 1374
10:29:14.928113 IP server.29543 > client.30280: Flags [.], seq 3024395254:3024396628, ack 4048786673, win 307, options [nop,nop,TS val 681965811 ecr 1634528340], length 1374
10:29:14.928701 IP server.29543 > client.30280: Flags [.], seq 3024396628:3024398002, ack 4048786673, win 307, options [nop,nop,TS val 681965811 ecr 1634528340], length 1374
10:29:14.928705 IP server.29543 > client.30280: Flags [.], seq 3024398002:3024399376, ack 4048786673, win 307, options [nop,nop,TS val 681965811 ecr 1634528340], length 1374
10:29:14.929817 IP server.29543 > client.30280: Flags [.], seq 3024399376:3024400750, ack 4048786673, win 307, options [nop,nop,TS val 681965813 ecr 1634528340], length 1374
10:29:14.929821 IP server.29543 > client.30280: Flags [.], seq 3024400750:3024402124, ack 4048786673, win 307, options [nop,nop,TS val 681965813 ecr 1634528340], length 1374
10:29:14.942186 IP client.30280 > server.29543: Flags [.], ack 3024376940, win 510, options [nop,nop,TS val 1634528358 ecr 681965800], length 0
10:29:14.942203 IP server.29543 > client.30280: Flags [.], seq 3024402124:3024403498, ack 4048786673, win 307, options [nop,nop,TS val 681965825 ecr 1634528358], length 1374
10:29:14.942208 IP server.29543 > client.30280: Flags [.], seq 3024403498:3024404872, ack 4048786673, win 307, options [nop,nop,TS val 681965825 ecr 1634528358], length 1374
10:29:14.942782 IP client.30280 > server.29543: Flags [.], ack 3024378314, win 533, options [nop,nop,TS val 1634528358 ecr 681965800], length 0
10:29:14.942797 IP server.29543 > client.30280: Flags [.], seq 3024404872:3024406246, ack 4048786673, win 307, options [nop,nop,TS val 681965826 ecr 1634528358], length 1374
10:29:14.942800 IP server.29543 > client.30280: Flags [.], seq 3024406246:3024407620, ack 4048786673, win 307, options [nop,nop,TS val 681965826 ecr 1634528358], length 1374
10:29:14.946310 IP client.30280 > server.29543: Flags [.], ack 3024379688, win 556, options [nop,nop,TS val 1634528362 ecr 681965804], length 0
10:29:14.946326 IP server.29543 > client.30280: Flags [.], seq 3024407620:3024408994, ack 4048786673, win 307, options [nop,nop,TS val 681965829 ecr 1634528362], length 1374
10:29:14.946332 IP server.29543 > client.30280: Flags [.], seq 3024408994:3024410368, ack 4048786673, win 307, options [nop,nop,TS val 681965829 ecr 1634528362], length 1374
10:29:14.946922 IP client.30280 > server.29543: Flags [.], ack 3024381062, win 578, options [nop,nop,TS val 1634528363 ecr 681965804], length 0
10:29:14.946937 IP server.29543 > client.30280: Flags [.], seq 3024410368:3024411742, ack 4048786673, win 307, options [nop,nop,TS val 681965830 ecr 1634528363], length 1374
10:29:14.946940 IP server.29543 > client.30280: Flags [.], seq 3024411742:3024413116, ack 4048786673, win 307, options [nop,nop,TS val 681965830 ecr 1634528363], length 1374
10:29:14.947854 IP client.30280 > server.29543: Flags [.], ack 3024382436, win 601, options [nop,nop,TS val 1634528363 ecr 681965805], length 0
10:29:14.947870 IP server.29543 > client.30280: Flags [.], seq 3024413116:3024414490, ack 4048786673, win 307, options [nop,nop,TS val 681965831 ecr 1634528363], length 1374
10:29:14.947873 IP server.29543 > client.30280: Flags [.], seq 3024414490:3024415864, ack 4048786673, win 307, options [nop,nop,TS val 681965831 ecr 1634528363], length 1374
10:29:14.948437 IP client.30280 > server.29543: Flags [.], ack 3024383810, win 624, options [nop,nop,TS val 1634528364 ecr 681965805], length 0
10:29:14.949374 IP server.29543 > client.30280: Flags [.], seq 3024415864:3024417238, ack 4048786673, win 307, options [nop,nop,TS val 681965832 ecr 1634528364], length 1374
10:29:14.949378 IP server.29543 > client.30280: Flags [.], seq 3024417238:3024418612, ack 4048786673, win 307, options [nop,nop,TS val 681965832 ecr 1634528364], length 1374
10:29:14.950039 IP client.30280 > server.29543: Flags [.], ack 3024385184, win 646, options [nop,nop,TS val 1634528365 ecr 681965807], length 0
10:29:14.951071 IP server.29543 > client.30280: Flags [.], seq 3024418612:3024419986, ack 4048786673, win 307, options [nop,nop,TS val 681965834 ecr 1634528365], length 1374
10:29:14.951075 IP server.29543 > client.30280: Flags [.], seq 3024419986:3024421360, ack 4048786673, win 307, options [nop,nop,TS val 681965834 ecr 1634528365], length 1374
10:29:14.951334 IP client.30280 > server.29543: Flags [.], ack 3024386558, win 669, options [nop,nop,TS val 1634528366 ecr 681965807], length 0
10:29:14.951824 IP client.30280 > server.29543: Flags [.], ack 3024387932, win 691, options [nop,nop,TS val 1634528366 ecr 681965807], length 0
10:29:14.952159 IP client.30280 > server.29543: Flags [.], ack 3024389306, win 714, options [nop,nop,TS val 1634528367 ecr 681965807], length 0
10:29:14.953216 IP server.29543 > client.30280: Flags [.], seq 3024421360:3024422734, ack 4048786673, win 307, options [nop,nop,TS val 681965836 ecr 1634528367], length 1374
10:29:14.953220 IP server.29543 > client.30280: Flags [.], seq 3024422734:3024424108, ack 4048786673, win 307, options [nop,nop,TS val 681965836 ecr 1634528367], length 1374
10:29:14.953473 IP client.30280 > server.29543: Flags [.], ack 3024390680, win 737, options [nop,nop,TS val 1634528368 ecr 681965808], length 0
10:29:14.954411 IP server.29543 > client.30280: Flags [.], seq 3024424108:3024425482, ack 4048786673, win 307, options [nop,nop,TS val 681965837 ecr 1634528368], length 1374
10:29:14.954415 IP server.29543 > client.30280: Flags [.], seq 3024425482:3024426856, ack 4048786673, win 307, options [nop,nop,TS val 681965837 ecr 1634528368], length 1374
10:29:14.955336 IP client.30280 > server.29543: Flags [.], ack 3024391132, win 758, options [nop,nop,TS val 1634528369 ecr 681965808], length 0
10:29:14.955646 IP server.29543 > client.30280: Flags [.], seq 3024426856:3024428230, ack 4048786673, win 307, options [nop,nop,TS val 681965838 ecr 1634528369], length 1374
10:29:14.955650 IP server.29543 > client.30280: Flags [.], seq 3024428230:3024429604, ack 4048786673, win 307, options [nop,nop,TS val 681965838 ecr 1634528369], length 1374
10:29:14.956485 IP client.30280 > server.29543: Flags [.], ack 3024392506, win 781, options [nop,nop,TS val 1634528369 ecr 681965809], length 0
10:29:14.956836 IP server.29543 > client.30280: Flags [.], seq 3024429604:3024430978, ack 4048786673, win 307, options [nop,nop,TS val 681965840 ecr 1634528369], length 1374
10:29:14.956841 IP server.29543 > client.30280: Flags [.], seq 3024430978:3024432352, ack 4048786673, win 307, options [nop,nop,TS val 681965840 ecr 1634528369], length 1374
10:29:14.957123 IP client.30280 > server.29543: Flags [.], ack 3024393880, win 803, options [nop,nop,TS val 1634528370 ecr 681965809], length 0
10:29:14.958055 IP client.30280 > server.29543: Flags [.], ack 3024395254, win 826, options [nop,nop,TS val 1634528371 ecr 681965811], length 0
10:29:14.958541 IP client.30280 > server.29543: Flags [.], ack 3024396628, win 849, options [nop,nop,TS val 1634528371 ecr 681965811], length 0
10:29:14.958931 IP client.30280 > server.29543: Flags [.], ack 3024398002, win 871, options [nop,nop,TS val 1634528372 ecr 681965811], length 0
10:29:14.959265 IP client.30280 > server.29543: Flags [.], ack 3024399376, win 894, options [nop,nop,TS val 1634528373 ecr 681965811], length 0
10:29:14.960799 IP client.30280 > server.29543: Flags [.], ack 3024400750, win 917, options [nop,nop,TS val 1634528375 ecr 681965813], length 0
10:29:14.961117 IP client.30280 > server.29543: Flags [.], ack 3024402124, win 939, options [nop,nop,TS val 1634528376 ecr 681965813], length 0
10:29:14.961443 IP server.29543 > client.30280: Flags [.], seq 3024432352:3024433726, ack 4048786673, win 307, options [nop,nop,TS val 681965844 ecr 1634528369], length 1374
10:29:14.961449 IP server.29543 > client.30280: Flags [.], seq 3024433726:3024435100, ack 4048786673, win 307, options [nop,nop,TS val 681965844 ecr 1634528369], length 1374
10:29:14.969116 IP client.30280 > server.29543: Flags [.], ack 3024403498, win 962, options [nop,nop,TS val 1634528384 ecr 681965825], length 0
10:29:14.969924 IP client.30280 > server.29543: Flags [.], ack 3024404872, win 984, options [nop,nop,TS val 1634528384 ecr 681965825], length 0
10:29:14.970246 IP client.30280 > server.29543: Flags [.], ack 3024406246, win 1007, options [nop,nop,TS val 1634528384 ecr 681965826], length 0
10:29:14.970652 IP client.30280 > server.29543: Flags [.], ack 3024407620, win 1030, options [nop,nop,TS val 1634528385 ecr 681965826], length 0
10:29:14.971939 IP client.30280 > server.29543: Flags [.], ack 3024408994, win 1052, options [nop,nop,TS val 1634528388 ecr 681965829], length 0
10:29:14.973628 IP client.30280 > server.29543: Flags [.], ack 3024410368, win 1075, options [nop,nop,TS val 1634528389 ecr 681965829], length 0
10:29:14.974381 IP client.30280 > server.29543: Flags [.], ack 3024411742, win 1098, options [nop,nop,TS val 1634528390 ecr 681965830], length 0
10:29:14.975803 IP client.30280 > server.29543: Flags [.], ack 3024413116, win 1120, options [nop,nop,TS val 1634528391 ecr 681965830], length 0
10:29:14.976415 IP client.30280 > server.29543: Flags [.], ack 3024414490, win 1143, options [nop,nop,TS val 1634528392 ecr 681965831], length 0
10:29:14.977895 IP client.30280 > server.29543: Flags [.], ack 3024415864, win 1165, options [nop,nop,TS val 1634528393 ecr 681965831], length 0
10:29:14.978221 IP client.30280 > server.29543: Flags [.], ack 3024417238, win 1188, options [nop,nop,TS val 1634528393 ecr 681965832], length 0
10:29:14.978864 IP client.30280 > server.29543: Flags [.], ack 3024418612, win 1211, options [nop,nop,TS val 1634528394 ecr 681965832], length 0
10:29:14.979653 IP client.30280 > server.29543: Flags [.], ack 3024419986, win 1233, options [nop,nop,TS val 1634528395 ecr 681965834], length 0
10:29:14.979976 IP client.30280 > server.29543: Flags [.], ack 3024421360, win 1256, options [nop,nop,TS val 1634528396 ecr 681965834], length 0
10:29:14.980864 IP client.30280 > server.29543: Flags [.], ack 3024422734, win 1279, options [nop,nop,TS val 1634528396 ecr 681965836], length 0
10:29:14.981489 IP client.30280 > server.29543: Flags [.], ack 3024424108, win 1301, options [nop,nop,TS val 1634528397 ecr 681965836], length 0
10:29:14.981877 IP client.30280 > server.29543: Flags [.], ack 3024425482, win 1324, options [nop,nop,TS val 1634528397 ecr 681965837], length 0
10:29:14.985230 IP client.30280 > server.29543: Flags [.], ack 3024426856, win 1346, options [nop,nop,TS val 1634528401 ecr 681965837], length 0
10:29:14.986222 IP client.30280 > server.29543: Flags [P.], seq 4048786673:4048786709, ack 3024426856, win 1346, options [nop,nop,TS val 1634528401 ecr 681965837], length 36
10:29:14.987058 IP client.30280 > server.29543: Flags [.], ack 3024429604, win 1392, options [nop,nop,TS val 1634528402 ecr 681965838], length 0
10:29:14.988349 IP client.30280 > server.29543: Flags [.], ack 3024432352, win 1437, options [nop,nop,TS val 1634528403 ecr 681965840], length 0
10:29:14.989103 IP client.30280 > server.29543: Flags [.], ack 3024435100, win 1444, options [nop,nop,TS val 1634528404 ecr 681965844], length 0
10:29:15.021406 IP server.29543 > client.30280: Flags [.], seq 3024435100:3024436474, ack 4048786673, win 307, options [nop,nop,TS val 681965904 ecr 1634528369], length 1374
10:29:15.087837 IP client.30280 > server.29543: Flags [.], ack 3024436474, win 1444, options [nop,nop,TS val 1634528503 ecr 681965904], length 0
10:29:15.242785 IP client.30280 > server.29543: Flags [P.], seq 4048786673:4048786709, ack 3024436474, win 1444, options [nop,nop,TS val 1634528637 ecr 681965904], length 36
10:29:15.255489 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681966138 ecr 1634528369], length 1374
10:29:15.281581 IP client.30280 > server.29543: Flags [.], ack 3024436474, win 1444, options [nop,nop,TS val 1634528697 ecr 681965904,nop,nop,sack 1 {3024391132:3024392506}], length 0
10:29:15.555466 IP client.30280 > server.29543: Flags [P.], seq 4048786673:4048786709, ack 3024436474, win 1444, options [nop,nop,TS val 1634528869 ecr 681965904], length 36
10:29:15.719510 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681966602 ecr 1634528369], length 1374
10:29:15.746624 IP client.30280 > server.29543: Flags [.], ack 3024436474, win 1444, options [nop,nop,TS val 1634529162 ecr 681965904,nop,nop,sack 1 {3024391132:3024392506}], length 0
10:29:15.918363 IP client.30280 > server.29543: Flags [P.], seq 4048786673:4048786709, ack 3024436474, win 1444, options [nop,nop,TS val 1634529333 ecr 681965904], length 36
10:29:16.695512 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681967578 ecr 1634528369], length 1374
10:29:16.737481 IP client.30280 > server.29543: Flags [.], ack 3024436474, win 1444, options [nop,nop,TS val 1634530143 ecr 681965904,nop,nop,sack 1 {3024391132:3024392506}], length 0
10:29:16.886594 IP client.30280 > server.29543: Flags [P.], seq 4048786673:4048786709, ack 3024436474, win 1444, options [nop,nop,TS val 1634530301 ecr 681965904], length 36
10:29:18.551512 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681969434 ecr 1634528369], length 1374
10:29:18.578152 IP client.30280 > server.29543: Flags [.], ack 3024436474, win 1444, options [nop,nop,TS val 1634531993 ecr 681965904,nop,nop,sack 1 {3024391132:3024392506}], length 0
10:29:18.742471 IP client.30280 > server.29543: Flags [P.], seq 4048786673:4048786709, ack 3024436474, win 1444, options [nop,nop,TS val 1634532157 ecr 681965904], length 36
10:29:22.263513 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681973146 ecr 1634528369], length 1374
10:29:22.298045 IP client.30280 > server.29543: Flags [.], ack 3024436474, win 1444, options [nop,nop,TS val 1634535706 ecr 681965904,nop,nop,sack 1 {3024391132:3024392506}], length 0
10:29:22.456055 IP client.30280 > server.29543: Flags [P.], seq 4048786673:4048786709, ack 3024436474, win 1444, options [nop,nop,TS val 1634535869 ecr 681965904], length 36
10:29:22.698033 IP client.30280 > server.29543: Flags [FP.], seq 4048786709:4048786849, ack 3024436474, win 1444, options [nop,nop,TS val 1634536113 ecr 681965904], length 140
10:29:29.943515 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681980826 ecr 1634528369], length 1374
10:29:29.976496 IP client.30280 > server.29543: Flags [.], ack 3024436474, win 1444, options [nop,nop,TS val 1634543386 ecr 681965904,nop,nop,sack 1 {3024391132:3024392506}], length 0
10:29:30.326520 IP client.30280 > server.29543: Flags [FP.], seq 4048786673:4048786849, ack 3024436474, win 1444, options [nop,nop,TS val 1634543741 ecr 681965904], length 176
[-- Attachment #3: ulogd.json --]
[-- Type: application/json, Size: 48712 bytes --]
[-- Attachment #4: dmesg.xz --]
[-- Type: application/octet-stream, Size: 500772 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-17 3:52 ` Dominique Martinet
@ 2018-04-17 5:28 ` Eric Dumazet
2018-04-17 6:52 ` Michal Kubecek
2018-04-17 9:20 ` Michal Kubecek
0 siblings, 2 replies; 26+ messages in thread
From: Eric Dumazet @ 2018-04-17 5:28 UTC (permalink / raw)
To: Florian Westphal, Eric Dumazet, Marcelo Ricardo Leitner,
Michal Kubecek, netdev
On 04/16/2018 08:52 PM, Dominique Martinet wrote:
> Thank you for the hints. Out of order reply.
>
> Florian Westphal wrote on Mon, Apr 16, 2018:
>> echo 1 > /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal
>>
>> which stops conntrack from marking packets with out-of-window
>> acks as invalid.
>
> That worked, the connection stays established with this set.
>
>
>> echo 6 > /proc/sys/net/netfilter/nf_conntrack_log_invalid
>
> Took me a while to figure how to log that properly (why do distros not
> ship ulogd?!) but that is helpful.
>
> I have attached the full logs to this mail as text attachments in case I
> missed something in my excerpts
>
>
> Specifically, there are many outbound invalid packets, but as Marcelo
> pointed out there is no conntrack netfilter for outbound packets...
> The first invalid incoming packet is that one (json output seemed to be
> the most convenient as it has tcp seq/ackseq and the oob.prefix message,
> feel free to ask for some other format)
>
> {"timestamp": "2018-04-17T10:29:14.956485", "dvc": "Netfilter",
> "raw.pktlen": 52, "raw.pktcount": 1, "oob.prefix": "nf_ct_tcp: ACK is
> over the upper bound (ACKed data not seen yet) ", "oob.time.sec":
> 1523928554, "oob.time.usec": 956485, "oob.mark": 0, "oob.hook": 0,
> "oob.family": 2, "oob.protocol": 2048, "raw.label": 0, "ip.protocol": 6,
> "ip.tos": 0, "ip.ttl": 52, "ip.totlen": 52, "ip.ihl": 5, "ip.csum":
> 50909, "ip.id": 30328, "ip.fragoff": 16384, "src_port": 30280,
> "dest_port": 29543, "tcp.seq": 4048786673, "tcp.ackseq": 3024392506,
> "tcp.window": 781, "tcp.offset": 0, "tcp.reserved": 0, "tcp.urg": 0,
> "tcp.ack": 1, "tcp.psh": 0, "tcp.rst": 0, "tcp.syn": 0, "tcp.fin": 0,
> "tcp.res1": 0, "tcp.res2": 0, "tcp.csum": 17752, "oob.in": "",
> "oob.out": "", "src_ip": "client", "dest_ip": "server"}
>
> which is the first ack that isn't seen alright (sent packet, blocked
> ack, replays):
> 10:29:14.926115 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681965809 ecr 1634528339], length 1374
> ...
> 10:29:14.956485 IP client.30280 > server.29543: Flags [.], ack 3024392506, win 781, options [nop,nop,TS val 1634528369 ecr 681965809], length 0
> ...
> 10:29:15.255489 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681966138 ecr 1634528369], length 1374
> 10:29:15.281581 IP client.30280 > server.29543: Flags [.], ack 3024436474, win 1444, options [nop,nop,TS val 1634528697 ecr 681965904,nop,nop,sack 1 {3024391132:3024392506}], length 0
> 10:29:15.555466 IP client.30280 > server.29543: Flags [P.], seq 4048786673:4048786709, ack 3024436474, win 1444, options [nop,nop,TS val 1634528869 ecr 681965904], length 36
> 10:29:15.719510 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681966602 ecr 1634528369], length 1374
>
> So now I guess the question is why netfilter think this is over the
> upper bound.
> ..And surely enough the answer is within the invalid outbound packets
> that aren't considered in the conntrack state machine, the first of whih
> is this one:
> {"timestamp": "2018-04-17T10:29:14.926143", "dvc": "Netfilter",
> "raw.pktlen": 2800, "raw.pktcount": 1, "oob.prefix": "nf_ct_tcp: SEQ is
> over the upper bound (over the window of the receiver)", "oob.time.sec":
> 1523928554, "oob.time.usec": 926143, "oob.mark": 0, "oob.hook": 0,
> "oob.family": 2, "oob.protocol": 2048, "oob.uid": 1000, "oob.gid": 100,
> "raw.label": 0, "ip.protocol": 6, "ip.tos": 8, "ip.ttl": 64,
> "ip.totlen": 2800, "ip.ihl": 5, "ip.csum": 47600, "ip.id": 27809,
> "ip.fragoff": 16384, "src_port": 29543, "dest_port": 30280, "tcp.seq":
> 3024391132, "tcp.ackseq": 4048786673, "tcp.window": 307, "tcp.offset":
> 0, "tcp.reserved": 0, "tcp.urg": 0, "tcp.ack": 1, "tcp.psh": 0,
> "tcp.rst": 0, "tcp.syn": 0, "tcp.fin": 0, "tcp.res1": 0, "tcp.res2": 0,
> "tcp.csum": 5201, "oob.in": "", "oob.out": "", "src_ip": "server",
> "dest_ip": "client"}
>
> which sequence matches the start seq of the first packet we don't ack,
> so that's why conntrack doesn't recongnize that as a valid ack.
>
>
> Here's some context before that packet:
> 10:29:14.921305 IP server.29543 > client.30280: Flags [.], seq 3024378314:3024379688, ack 4048786673, win 307, options [nop,nop,TS val 681965804 ecr 1634528336], length 1374
> 10:29:14.921311 IP server.29543 > client.30280: Flags [.], seq 3024379688:3024381062, ack 4048786673, win 307, options [nop,nop,TS val 681965804 ecr 1634528336], length 1374
> 10:29:14.922166 IP client.30280 > server.29543: Flags [.], ack 3024365948, win 329, options [nop,nop,TS val 1634528336 ecr 681965778], length 0
> 10:29:14.922178 IP server.29543 > client.30280: Flags [.], seq 3024381062:3024382436, ack 4048786673, win 307, options [nop,nop,TS val 681965805 ecr 1634528336], length 1374
> 10:29:14.922182 IP server.29543 > client.30280: Flags [.], seq 3024382436:3024383810, ack 4048786673, win 307, options [nop,nop,TS val 681965805 ecr 1634528336], length 1374
> 10:29:14.923848 IP client.30280 > server.29543: Flags [.], ack 3024367322, win 352, options [nop,nop,TS val 1634528338 ecr 681965778], length 0
> 10:29:14.923860 IP server.29543 > client.30280: Flags [.], seq 3024383810:3024385184, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
> 10:29:14.923863 IP server.29543 > client.30280: Flags [.], seq 3024385184:3024386558, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
> 10:29:14.924175 IP client.30280 > server.29543: Flags [.], ack 3024368696, win 375, options [nop,nop,TS val 1634528338 ecr 681965778], length 0
> 10:29:14.924187 IP server.29543 > client.30280: Flags [.], seq 3024386558:3024387932, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
> 10:29:14.924190 IP server.29543 > client.30280: Flags [.], seq 3024387932:3024389306, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
> 10:29:14.925410 IP client.30280 > server.29543: Flags [.], ack 3024370070, win 397, options [nop,nop,TS val 1634528339 ecr 681965779], length 0
> 10:29:14.925422 IP server.29543 > client.30280: Flags [.], seq 3024389306:3024390680, ack 4048786673, win 307, options [nop,nop,TS val 681965808 ecr 1634528339], length 1374
> 10:29:14.925425 IP server.29543 > client.30280: Flags [P.], seq 3024390680:3024391132, ack 4048786673, win 307, options [nop,nop,TS val 681965808 ecr 1634528339], length 452
> 10:29:14.926030 IP client.30280 > server.29543: Flags [.], ack 3024371444, win 420, options [nop,nop,TS val 1634528339 ecr 681965779], length 0
> 10:29:14.926115 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681965809 ecr 1634528339], length 1374
>
> But the way I see it, (3024392506-3024371444) = 21064 is smaller than
> 420*64 = 26880.
> Even looking at the previous (ack 3024370070, win 397) we get 22436 and
> 25408 which still work, same with ack 3024368696, win 375 (23810 <
> 24000)
> So that means conntrack is looking at an ack at least as old as ack
> 3024367322, win 352?...
> Is there any way to confirm that? In the code it looks like we match
> ! before(seq, sender->td_maxend + 1)
>
> I turned pr_debug on in tcp_in_window() for another try and it's a bit
> mangled because the information on multiple lines and the function is
> called in parallel but it looks like I do have some seq > maxend +1
>
> Although it's weird, the maxend was set WAY earlier apparently?
> Apr 17 11:13:14 res=1 sender end=1913287798 maxend=1913316998 maxwin=29312 receiver end=505004284 maxend=505033596 maxwin=29200
> then window decreased drastically e.g. previous ack just before refusal:
> Apr 17 11:13:53 seq=1913292311 ack=505007789+(0) sack=505007789+(0) win=284 end=1913292311
> Apr 17 11:13:53 sender end=1913292311 maxend=1913331607 maxwin=284 scale=0 receiver end=505020155 maxend=505033596 maxwin=39296 scale=7
scale=0 is suspect.
Really if conntrack does not see SYN SYNACK packets, it should not make any window check,
since windows can be scaled up to 14 :/
>
> first res=0 (refused) packet:
> Apr 17 11:13:53 odin kernel: seq=505042139 ack=1913292311+(0) ack=1913292311+(0) win=307 end=505044887
> Apr 17 11:13:53 res=0 sender end=505033895 maxend=505033596 maxwin=39296 receiver end=1913292311 maxend=1913331607 maxwin=674
>
>
>> (Earlier email implies this is related to timestamps, but unfortunately
>> to best of my knowledge conntrack doesn't look at tcp timestamps).
>
> tcp_timestamp introduces other changes in behaviour so it might still be
> relateed.
>
>
>
> Recap of attachments:
> - new tcpdump output of a failed attempt
> - matching invalid packet dumped in json format
> - logs of tcp_in_window function unfortunately of a different attempt
>
> Happy to provide new/different logs if it would be useful. Once again
> thanks for the extra eyes and useful commands.
>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-17 5:28 ` Eric Dumazet
@ 2018-04-17 6:52 ` Michal Kubecek
2018-04-17 9:20 ` Michal Kubecek
1 sibling, 0 replies; 26+ messages in thread
From: Michal Kubecek @ 2018-04-17 6:52 UTC (permalink / raw)
To: netdev; +Cc: Florian Westphal, Marcelo Ricardo Leitner, Eric Dumazet
On Mon, Apr 16, 2018 at 10:28:11PM -0700, Eric Dumazet wrote:
> > I turned pr_debug on in tcp_in_window() for another try and it's a bit
> > mangled because the information on multiple lines and the function is
> > called in parallel but it looks like I do have some seq > maxend +1
> >
> > Although it's weird, the maxend was set WAY earlier apparently?
> > Apr 17 11:13:14 res=1 sender end=1913287798 maxend=1913316998 maxwin=29312 receiver end=505004284 maxend=505033596 maxwin=29200
> > then window decreased drastically e.g. previous ack just before refusal:
> > Apr 17 11:13:53 seq=1913292311 ack=505007789+(0) sack=505007789+(0) win=284 end=1913292311
> > Apr 17 11:13:53 sender end=1913292311 maxend=1913331607 maxwin=284 scale=0 receiver end=505020155 maxend=505033596 maxwin=39296 scale=7
>
> scale=0 is suspect.
>
> Really if conntrack does not see SYN SYNACK packets, it should not
> make any window check, since windows can be scaled up to 14 :/
Or maybe set the scaling to
- TCP_MAX_WSCALE (14) by default
- 0 when SYN or SYNACK without window scale option is seen
- value of window scale option when SYN or SYNACK with it is seen
Michal Kubecek
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-17 5:28 ` Eric Dumazet
2018-04-17 6:52 ` Michal Kubecek
@ 2018-04-17 9:20 ` Michal Kubecek
2018-04-17 12:34 ` Dominique Martinet
1 sibling, 1 reply; 26+ messages in thread
From: Michal Kubecek @ 2018-04-17 9:20 UTC (permalink / raw)
To: netdev; +Cc: Florian Westphal, Marcelo Ricardo Leitner, Eric Dumazet
On Mon, Apr 16, 2018 at 10:28:11PM -0700, Eric Dumazet wrote:
> > I turned pr_debug on in tcp_in_window() for another try and it's a bit
> > mangled because the information on multiple lines and the function is
> > called in parallel but it looks like I do have some seq > maxend +1
> >
> > Although it's weird, the maxend was set WAY earlier apparently?
> > Apr 17 11:13:14 res=1 sender end=1913287798 maxend=1913316998 maxwin=29312 receiver end=505004284 maxend=505033596 maxwin=29200
> > then window decreased drastically e.g. previous ack just before refusal:
> > Apr 17 11:13:53 seq=1913292311 ack=505007789+(0) sack=505007789+(0) win=284 end=1913292311
> > Apr 17 11:13:53 sender end=1913292311 maxend=1913331607 maxwin=284 scale=0 receiver end=505020155 maxend=505033596 maxwin=39296 scale=7
>
> scale=0 is suspect.
>
> Really if conntrack does not see SYN SYNACK packets, it should not
> make any window check, since windows can be scaled up to 14 :/
Hm... it doesn't seem to be the case here:
14.364038 tcp_in_window: START
14.364065 tcp_in_window:
14.364090 seq=505004283 ack=0+(0) sack=0+(0) win=29200 end=505004284
14.364129 tcp_in_window: sender end=505004284 maxend=505004284 maxwin=29200 scale=7 receiver end=0 maxend=0 maxwin=0 scale=0
14.364158 tcp_in_window:
14.364185 seq=505004283 ack=0+(0) sack=0+(0) win=29200 end=505004284
14.364210 tcp_in_window: sender end=505004284 maxend=505004284 maxwin=29200 scale=7 receiver end=0 maxend=0 maxwin=0 scale=0
14.364237 tcp_in_window: I=1 II=1 III=1 IV=1
14.364262 tcp_in_window: res=1 sender end=505004284 maxend=505004284 maxwin=29200 receiver end=0 maxend=29200 maxwin=0
looks like SYN packet
14.661682 tcp_in_window: START
14.661706 tcp_in_window:
14.661731 seq=1913287797 ack=0+(0) sack=0+(0) win=29200 end=1913287798
14.661828 tcp_in_window: sender end=0 maxend=29200 maxwin=0 scale=0 receiver end=505004284 maxend=505004284 maxwin=29200 scale=7
14.661867 tcp_in_window: START
14.661893 tcp_in_window:
14.661917 seq=1025597635 ack=1542862349+(0) sack=1542862349+(0) win=2414 end=1025597635
14.661942 tcp_in_window: START
14.661966 tcp_in_window:
14.661993 tcp_in_window: sender end=1025597635 maxend=1025635103 maxwin=354378 scale=7 receiver end=1542862349 maxend=1543168175 maxwin=37504 scale=7
14.662020 seq=505004283 ack=1913287798+(0) sack=1913287798+(0) win=29200 end=505004284
14.662045 tcp_in_window:
14.662072 seq=1025597635 ack=1542862349+(0) sack=1542862349+(0) win=2414 end=1025597635
14.662097 tcp_in_window: sender end=505004284 maxend=505004284 maxwin=29200 scale=7 receiver end=1913287798 maxend=1913287798 maxwin=29200 scale=7
14.662125 tcp_in_window:
14.662150 tcp_in_window: sender end=1025597635 maxend=1025635103 maxwin=354378 scale=7 receiver end=1542862349 maxend=1543168175 maxwin=37504 scale=7
14.662175 seq=505004283 ack=1913287798+(0) sack=1913287798+(0) win=29200 end=505004284
14.662202 tcp_in_window: sender end=505004284 maxend=505004284 maxwin=29200 scale=7 receiver end=1913287798 maxend=1913287798 maxwin=29200 scale=7
14.662226 tcp_in_window: I=1 II=1 III=1 IV=1
14.662251 tcp_in_window: I=1 II=1 III=1 IV=1
14.662277 tcp_in_window: res=1 sender end=505004284 maxend=505004284 maxwin=29200 receiver end=1913287798 maxend=1913316998 maxwin=29200
14.662302 tcp_in_window: res=1 sender end=1025597635 maxend=1025635103 maxwin=354378 receiver end=1542862349 maxend=1543171341 maxwin=37504
SYNACK response and (dataless) ACK in the original direction, mixed with
an unrelated packet.
14.687411 tcp_in_window: START
14.687522 tcp_in_window:
14.687570 seq=1913287798 ack=505004284+(0) sack=505004284+(0) win=229 end=1913287798
14.687619 tcp_in_window: sender end=1913287798 maxend=1913316998 maxwin=29200 scale=7 receiver end=505004284 maxend=505004284 maxwin=29200 scale=7
14.687659 tcp_in_window:
14.687699 seq=1913287798 ack=505004284+(0) sack=505004284+(0) win=229 end=1913287798
14.687739 tcp_in_window: sender end=1913287798 maxend=1913316998 maxwin=29200 scale=7 receiver end=505004284 maxend=505004284 maxwin=29200 scale=7
14.687774 tcp_in_window: I=1 II=1 III=1 IV=1
14.687806 tcp_in_window: res=1 sender end=1913287798 maxend=1913316998 maxwin=29312 receiver end=505004284 maxend=505033596 maxwin=29200
ACK in the reply direction (no data). We still have scale=7 in both
directions.
14.688706 tcp_in_window: START
14.688733 tcp_in_window:
14.688762 seq=1913287798 ack=505004284+(0) sack=505004284+(0) win=229 end=1913287819
14.688793 tcp_in_window: sender end=1913287798 maxend=1913316998 maxwin=29312 scale=7 receiver end=505004284 maxend=505033596 maxwin=29200 scale=7
14.688824 tcp_in_window:
14.688852 seq=1913287798 ack=505004284+(0) sack=505004284+(0) win=229 end=1913287819
14.688882 tcp_in_window: sender end=1913287819 maxend=1913287819 maxwin=229 scale=0 receiver end=505004284 maxend=505033596 maxwin=29200 scale=7
14.688911 tcp_in_window: I=1 II=1 III=1 IV=1
14.688938 tcp_in_window: res=1 sender end=1913287819 maxend=1913287819 maxwin=229 receiver end=505004284 maxend=505033596 maxwin=29200
Data (21 bytes) packet in reply direction. And somewhere between the
first and second debugging print, we ended up with sender scale=0 and
that value is then preserved from now on.
The only place between the two debug prints where we could change only
one of the td_sender values are the two calls to tcp_options() but
neither should be called now unless I missed something. I'll try to
think about it some more.
Michal Kubecek
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-17 9:20 ` Michal Kubecek
@ 2018-04-17 12:34 ` Dominique Martinet
2018-04-17 13:00 ` Michal Kubecek
2018-04-17 13:29 ` Florian Westphal
0 siblings, 2 replies; 26+ messages in thread
From: Dominique Martinet @ 2018-04-17 12:34 UTC (permalink / raw)
To: Michal Kubecek
Cc: netdev, Florian Westphal, Marcelo Ricardo Leitner, Eric Dumazet
Michal Kubecek wrote on Tue, Apr 17, 2018:
> Data (21 bytes) packet in reply direction. And somewhere between the
> first and second debugging print, we ended up with sender scale=0 and
> that value is then preserved from now on.
>
> The only place between the two debug prints where we could change only
> one of the td_sender values are the two calls to tcp_options() but
> neither should be called now unless I missed something. I'll try to
> think about it some more.
Could it have something to do with the way I setup the connection?
I don't think the "both remotes call connect() with carefully selected
source/dest port" is a very common case..
If you look at the tcpdump outputs I attached the sequence usually is
something like
server > client SYN
client > server SYN
server > client SYNACK
client > server ACK
ultimately it IS a connection, but with an extra SYN packet in front of
it (that first SYN opens up the conntrack of the nat so that the
client's syn can come in, the client's conntrack will be that of a
normal connection since its first SYN goes in directly after the
server's (it didn't see the server's SYN))
Looking at my logs again, I'm seeing the same as you:
This looks like the actual SYN/SYN/SYNACK/ACK:
- 14.364090 seq=505004283 likely SYN coming out of server
- 14.661731 seq=1913287797 on next line it says receiver
end=505004284 so likely the matching SYN from client
Which this time gets a proper SYNACK from server:
14.662020 seq=505004283 ack=1913287798
And following final dataless ACK:
14.687570 seq=1913287798 ack=505004284
Then as you point out some data ACK, where the scale poofs:
14.688762 seq=1913287798 ack=505004284+(0) sack=505004284+(0) win=229 end=1913287819
14.688793 tcp_in_window: sender end=1913287798 maxend=1913316998 maxwin=29312 scale=7 receiver end=505004284 maxend=505033596 maxwin=29200 scale=7
14.688824 tcp_in_window:
14.688852 seq=1913287798 ack=505004284+(0) sack=505004284+(0) win=229 end=1913287819
14.688882 tcp_in_window: sender end=1913287819 maxend=1913287819 maxwin=229 scale=0 receiver end=505004284 maxend=505033596 maxwin=29200 scale=7
As you say, only tcp_options() will clear only on side of the scales.
We don't have sender->td_maxwin == 0 (printed) so I see no other way
than we are in the last else if:
- we have after(end, sender->td_end) (end=1913287819 > sender
end=1913287798)
- I assume the tcp state machine must be confused because of the
SYN/SYN/SYNACK/ACK pattern and we probably enter the next check,
but since this is a data packet it doesn't have the tcp option for scale
thus scale resets.
At least peeling the logs myself helped me follow the process, I'll
sprinkle some carefully crafted logs tomorrow to check if this is true
and will let you figure what is best of trying to preserve scale if it
was set before, setting a default to 14 or something else.
Thanks!
--
Dominique Martinet | Asmadeus
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-17 12:34 ` Dominique Martinet
@ 2018-04-17 13:00 ` Michal Kubecek
2018-04-17 13:29 ` Florian Westphal
1 sibling, 0 replies; 26+ messages in thread
From: Michal Kubecek @ 2018-04-17 13:00 UTC (permalink / raw)
To: netdev
Cc: Dominique Martinet, Florian Westphal, Marcelo Ricardo Leitner,
Eric Dumazet
On Tue, Apr 17, 2018 at 02:34:37PM +0200, Dominique Martinet wrote:
> Michal Kubecek wrote on Tue, Apr 17, 2018:
> > Data (21 bytes) packet in reply direction. And somewhere between the
> > first and second debugging print, we ended up with sender scale=0 and
> > that value is then preserved from now on.
> >
> > The only place between the two debug prints where we could change only
> > one of the td_sender values are the two calls to tcp_options() but
> > neither should be called now unless I missed something. I'll try to
> > think about it some more.
>
> Could it have something to do with the way I setup the connection?
> I don't think the "both remotes call connect() with carefully selected
> source/dest port" is a very common case..
>
> If you look at the tcpdump outputs I attached the sequence usually is
> something like
> server > client SYN
> client > server SYN
> server > client SYNACK
> client > server ACK
This must be what nf_conntrack_proto_tcp.c calls "simultaneous open".
> ultimately it IS a connection, but with an extra SYN packet in front of
> it (that first SYN opens up the conntrack of the nat so that the
> client's syn can come in, the client's conntrack will be that of a
> normal connection since its first SYN goes in directly after the
> server's (it didn't see the server's SYN))
>
>
> Looking at my logs again, I'm seeing the same as you:
>
> This looks like the actual SYN/SYN/SYNACK/ACK:
> - 14.364090 seq=505004283 likely SYN coming out of server
> - 14.661731 seq=1913287797 on next line it says receiver
> end=505004284 so likely the matching SYN from client
> Which this time gets a proper SYNACK from server:
> 14.662020 seq=505004283 ack=1913287798
> And following final dataless ACK:
> 14.687570 seq=1913287798 ack=505004284
>
> Then as you point out some data ACK, where the scale poofs:
> 14.688762 seq=1913287798 ack=505004284+(0) sack=505004284+(0) win=229 end=1913287819
> 14.688793 tcp_in_window: sender end=1913287798 maxend=1913316998 maxwin=29312 scale=7 receiver end=505004284 maxend=505033596 maxwin=29200 scale=7
> 14.688824 tcp_in_window:
> 14.688852 seq=1913287798 ack=505004284+(0) sack=505004284+(0) win=229 end=1913287819
> 14.688882 tcp_in_window: sender end=1913287819 maxend=1913287819 maxwin=229 scale=0 receiver end=505004284 maxend=505033596 maxwin=29200 scale=7
>
> As you say, only tcp_options() will clear only on side of the scales.
> We don't have sender->td_maxwin == 0 (printed) so I see no other way
> than we are in the last else if:
> - we have after(end, sender->td_end) (end=1913287819 > sender
> end=1913287798)
> - I assume the tcp state machine must be confused because of the
> SYN/SYN/SYNACK/ACK pattern and we probably enter the next check,
> but since this is a data packet it doesn't have the tcp option for scale
> thus scale resets.
I agree that sender->td_maxwin is not zero so that the handshake above
probably left the conntrack in TCP_CONNTRACK_SYN_RECV state for some
reason. I'll try to go through the code with the pattern you mentioned
in mind.
Michal Kubecek
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-17 12:34 ` Dominique Martinet
2018-04-17 13:00 ` Michal Kubecek
@ 2018-04-17 13:29 ` Florian Westphal
2018-04-18 8:13 ` Jozsef Kadlecsik
1 sibling, 1 reply; 26+ messages in thread
From: Florian Westphal @ 2018-04-17 13:29 UTC (permalink / raw)
To: Michal Kubecek, netdev, Florian Westphal,
Marcelo Ricardo Leitner, Eric Dumazet
Cc: Jozsef Kadlecsik
Dominique Martinet <asmadeus@codewreck.org> wrote:
[ CC Jozsef ]
> Could it have something to do with the way I setup the connection?
> I don't think the "both remotes call connect() with carefully selected
> source/dest port" is a very common case..
>
> If you look at the tcpdump outputs I attached the sequence usually is
> something like
> server > client SYN
> client > server SYN
> server > client SYNACK
> client > server ACK
>
> ultimately it IS a connection, but with an extra SYN packet in front of
> it (that first SYN opens up the conntrack of the nat so that the
> client's syn can come in, the client's conntrack will be that of a
> normal connection since its first SYN goes in directly after the
> server's (it didn't see the server's SYN))
>
> Looking at my logs again, I'm seeing the same as you:
>
> This looks like the actual SYN/SYN/SYNACK/ACK:
> - 14.364090 seq=505004283 likely SYN coming out of server
> - 14.661731 seq=1913287797 on next line it says receiver
> end=505004284 so likely the matching SYN from client
> Which this time gets a proper SYNACK from server:
> 14.662020 seq=505004283 ack=1913287798
> And following final dataless ACK:
> 14.687570 seq=1913287798 ack=505004284
>
> Then as you point out some data ACK, where the scale poofs:
> 14.688762 seq=1913287798 ack=505004284+(0) sack=505004284+(0) win=229 end=1913287819
> 14.688793 tcp_in_window: sender end=1913287798 maxend=1913316998 maxwin=29312 scale=7 receiver end=505004284 maxend=505033596 maxwin=29200 scale=7
> 14.688824 tcp_in_window:
> 14.688852 seq=1913287798 ack=505004284+(0) sack=505004284+(0) win=229 end=1913287819
> 14.688882 tcp_in_window: sender end=1913287819 maxend=1913287819 maxwin=229 scale=0 receiver end=505004284 maxend=505033596 maxwin=29200 scale=7
>
> As you say, only tcp_options() will clear only on side of the scales.
> We don't have sender->td_maxwin == 0 (printed) so I see no other way
> than we are in the last else if:
> - we have after(end, sender->td_end) (end=1913287819 > sender
> end=1913287798)
> - I assume the tcp state machine must be confused because of the
> SYN/SYN/SYNACK/ACK pattern and we probably enter the next check,
> but since this is a data packet it doesn't have the tcp option for scale
> thus scale resets.
Yes, this looks correct. Jozsef, can you please have a look?
Problem seems to be that conntrack believes that ACK packet
re-initializes the connection:
595 /*
596 * RFC 793: "if a TCP is reinitialized ... then it need
597 * not wait at all; it must only be sure to use sequence
598 * numbers larger than those recently used."
599 */
600 sender->td_end =
601 sender->td_maxend = end;
602 sender->td_maxwin = (win == 0 ? 1 : win);
603
604 tcp_options(skb, dataoff, tcph, sender);
and last line clears the scale value (no wscale option in data packet).
Transitions are:
server > client SYN sNO -> sSS
client > server SYN sSS -> sS2
server > client SYNACK sS2 -> sSR /* here */
client > server ACK sSR -> sES
SYN/ACK was observed in original direction so we hit
state->state == TCP_CONNTRACK_SYN_RECV && dir == IP_CT_DIR_REPLY test
when we see the ack packet and end up in the 'TCP is reinitialized' branch.
AFAICS, without this, connection would move to sES just fine,
as the data ack is in window.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-17 13:29 ` Florian Westphal
@ 2018-04-18 8:13 ` Jozsef Kadlecsik
2018-04-18 8:30 ` Dominique Martinet
0 siblings, 1 reply; 26+ messages in thread
From: Jozsef Kadlecsik @ 2018-04-18 8:13 UTC (permalink / raw)
To: Florian Westphal
Cc: Michal Kubecek, netdev, Marcelo Ricardo Leitner, Eric Dumazet
Hi,
On Tue, 17 Apr 2018, Florian Westphal wrote:
> Dominique Martinet <asmadeus@codewreck.org> wrote:
>
> [ CC Jozsef ]
>
> > Could it have something to do with the way I setup the connection?
> > I don't think the "both remotes call connect() with carefully selected
> > source/dest port" is a very common case..
> >
> > If you look at the tcpdump outputs I attached the sequence usually is
> > something like
> > server > client SYN
> > client > server SYN
> > server > client SYNACK
> > client > server ACK
> >
> > ultimately it IS a connection, but with an extra SYN packet in front of
> > it (that first SYN opens up the conntrack of the nat so that the
> > client's syn can come in, the client's conntrack will be that of a
> > normal connection since its first SYN goes in directly after the
> > server's (it didn't see the server's SYN))
> >
> > Looking at my logs again, I'm seeing the same as you:
> >
> > This looks like the actual SYN/SYN/SYNACK/ACK:
> > - 14.364090 seq=505004283 likely SYN coming out of server
> > - 14.661731 seq=1913287797 on next line it says receiver
> > end=505004284 so likely the matching SYN from client
> > Which this time gets a proper SYNACK from server:
> > 14.662020 seq=505004283 ack=1913287798
> > And following final dataless ACK:
> > 14.687570 seq=1913287798 ack=505004284
> >
> > Then as you point out some data ACK, where the scale poofs:
> > 14.688762 seq=1913287798 ack=505004284+(0) sack=505004284+(0) win=229 end=1913287819
> > 14.688793 tcp_in_window: sender end=1913287798 maxend=1913316998 maxwin=29312 scale=7 receiver end=505004284 maxend=505033596 maxwin=29200 scale=7
> > 14.688824 tcp_in_window:
> > 14.688852 seq=1913287798 ack=505004284+(0) sack=505004284+(0) win=229 end=1913287819
> > 14.688882 tcp_in_window: sender end=1913287819 maxend=1913287819 maxwin=229 scale=0 receiver end=505004284 maxend=505033596 maxwin=29200 scale=7
> >
> > As you say, only tcp_options() will clear only on side of the scales.
> > We don't have sender->td_maxwin == 0 (printed) so I see no other way
> > than we are in the last else if:
> > - we have after(end, sender->td_end) (end=1913287819 > sender
> > end=1913287798)
> > - I assume the tcp state machine must be confused because of the
> > SYN/SYN/SYNACK/ACK pattern and we probably enter the next check,
> > but since this is a data packet it doesn't have the tcp option for scale
> > thus scale resets.
>
> Yes, this looks correct. Jozsef, can you please have a look?
>
> Problem seems to be that conntrack believes that ACK packet
> re-initializes the connection:
>
> 595 /*
> 596 * RFC 793: "if a TCP is reinitialized ... then it need
> 597 * not wait at all; it must only be sure to use sequence
> 598 * numbers larger than those recently used."
> 599 */
> 600 sender->td_end =
> 601 sender->td_maxend = end;
> 602 sender->td_maxwin = (win == 0 ? 1 : win);
> 603
> 604 tcp_options(skb, dataoff, tcph, sender);
>
> and last line clears the scale value (no wscale option in data packet).
>
>
> Transitions are:
> server > client SYN sNO -> sSS
> client > server SYN sSS -> sS2
> server > client SYNACK sS2 -> sSR /* here */
> client > server ACK sSR -> sES
>
> SYN/ACK was observed in original direction so we hit
> state->state == TCP_CONNTRACK_SYN_RECV && dir == IP_CT_DIR_REPLY test
> when we see the ack packet and end up in the 'TCP is reinitialized' branch.
>
> AFAICS, without this, connection would move to sES just fine,
> as the data ack is in window.
Yes, the state transition is wrong for simultaneous open, because the
tcp_conntracks table is not (cannot be) smart enough. Could you verify the
next untested patch?
diff --git a/include/uapi/linux/netfilter/nf_conntrack_tcp.h b/include/uapi/linux/netfilter/nf_conntrack_tcp.h
index 74b9115..bcba72d 100644
--- a/include/uapi/linux/netfilter/nf_conntrack_tcp.h
+++ b/include/uapi/linux/netfilter/nf_conntrack_tcp.h
@@ -46,6 +46,9 @@ enum tcp_conntrack {
/* Marks possibility for expected RFC5961 challenge ACK */
#define IP_CT_EXP_CHALLENGE_ACK 0x40
+/* Simultaneous open initialized */
+#define IP_CT_TCP_SIMULTANEOUS_OPEN 0x80
+
struct nf_ct_tcp_flags {
__u8 flags;
__u8 mask;
diff --git a/net/netfilter/nf_conntrack_proto_tcp.c b/net/netfilter/nf_conntrack_proto_tcp.c
index e97cdc1..8e67910 100644
--- a/net/netfilter/nf_conntrack_proto_tcp.c
+++ b/net/netfilter/nf_conntrack_proto_tcp.c
@@ -981,6 +981,17 @@ static int tcp_packet(struct nf_conn *ct,
return NF_ACCEPT; /* Don't change state */
}
break;
+ case TCP_CONNTRACK_SYN_SENT2:
+ /* tcp_conntracks table is not smart enough to handle
+ * simultaneous open.
+ */
+ ct->proto.tcp.last_flags |= IP_CT_TCP_SIMULTANEOUS_OPEN;
+ break;
+ case TCP_CONNTRACK_SYN_RECV:
+ if (dir == IP_CT_DIR_REPLY && index == TCP_ACK_SET &&
+ ct->proto.tcp.last_flags & IP_CT_TCP_SIMULTANEOUS_OPEN)
+ new_state = TCP_CONNTRACK_ESTABLISHED;
+ break;
case TCP_CONNTRACK_CLOSE:
if (index == TCP_RST_SET
&& (ct->proto.tcp.seen[!dir].flags & IP_CT_TCP_FLAG_MAXACK_SET)
Best regards,
Jozsef
-
E-mail : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-18 8:13 ` Jozsef Kadlecsik
@ 2018-04-18 8:30 ` Dominique Martinet
2018-04-18 9:36 ` Dominique Martinet
0 siblings, 1 reply; 26+ messages in thread
From: Dominique Martinet @ 2018-04-18 8:30 UTC (permalink / raw)
To: Jozsef Kadlecsik
Cc: Florian Westphal, Michal Kubecek, netdev,
Marcelo Ricardo Leitner, Eric Dumazet
Jozsef Kadlecsik wrote on Wed, Apr 18, 2018:
> Yes, the state transition is wrong for simultaneous open, because the
> tcp_conntracks table is not (cannot be) smart enough. Could you verify the
> next untested patch?
Thanks for the patch; I'll give it a try (probably won't make it today
so will report tomorrow)
--
Dominique Martinet | Asmadeus
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-18 8:30 ` Dominique Martinet
@ 2018-04-18 9:36 ` Dominique Martinet
2018-04-18 10:27 ` Jozsef Kadlecsik
0 siblings, 1 reply; 26+ messages in thread
From: Dominique Martinet @ 2018-04-18 9:36 UTC (permalink / raw)
To: Jozsef Kadlecsik, Florian Westphal, Michal Kubecek, netdev,
Marcelo Ricardo Leitner, Eric Dumazet
Dominique Martinet wrote on Wed, Apr 18, 2018:
> Jozsef Kadlecsik wrote on Wed, Apr 18, 2018:
> > Yes, the state transition is wrong for simultaneous open, because the
> > tcp_conntracks table is not (cannot be) smart enough. Could you verify the
> > next untested patch?
>
> Thanks for the patch; I'll give it a try (probably won't make it today
> so will report tomorrow)
Actually had time; I can confirm (added printks) we did get in that if
that was pointed at, and we no longer get there now.
The connection no longer gets in invalid state, so that looks like it
nailed it.
I'm now confused what this has to do with tcp_timestamp though, since
setting that off also seemed to work around the issue, but if we get
something like that in I'll be happy anyway.
Big props to everyone involved, I would have taken much longer alone,
--
Dominique Martinet | Asmadeus
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-18 9:36 ` Dominique Martinet
@ 2018-04-18 10:27 ` Jozsef Kadlecsik
2018-04-18 11:30 ` Dominique Martinet
0 siblings, 1 reply; 26+ messages in thread
From: Jozsef Kadlecsik @ 2018-04-18 10:27 UTC (permalink / raw)
To: Dominique Martinet
Cc: Florian Westphal, Michal Kubecek, netdev,
Marcelo Ricardo Leitner, Eric Dumazet
On Wed, 18 Apr 2018, Dominique Martinet wrote:
> Dominique Martinet wrote on Wed, Apr 18, 2018:
> > Jozsef Kadlecsik wrote on Wed, Apr 18, 2018:
> > > Yes, the state transition is wrong for simultaneous open, because the
> > > tcp_conntracks table is not (cannot be) smart enough. Could you verify the
> > > next untested patch?
> >
> > Thanks for the patch; I'll give it a try (probably won't make it today
> > so will report tomorrow)
>
> Actually had time; I can confirm (added printks) we did get in that if
> that was pointed at, and we no longer get there now. The connection no
> longer gets in invalid state, so that looks like it nailed it.
>
> I'm now confused what this has to do with tcp_timestamp though, since
> setting that off also seemed to work around the issue, but if we get
> something like that in I'll be happy anyway.
Thanks for the testing! One more line is required, however: we have to get
the assured bit set for the connection, see the new patch below.
The tcp_conntracks state table could be fixed with introducing a new
state, but that part is exposed to userspace (ctnetlink) and ugly
compatibility code would be required for backward compatibility.
diff --git a/include/uapi/linux/netfilter/nf_conntrack_tcp.h b/include/uapi/linux/netfilter/nf_conntrack_tcp.h
index 74b9115..bcba72d 100644
--- a/include/uapi/linux/netfilter/nf_conntrack_tcp.h
+++ b/include/uapi/linux/netfilter/nf_conntrack_tcp.h
@@ -46,6 +46,9 @@ enum tcp_conntrack {
/* Marks possibility for expected RFC5961 challenge ACK */
#define IP_CT_EXP_CHALLENGE_ACK 0x40
+/* Simultaneous open initialized */
+#define IP_CT_TCP_SIMULTANEOUS_OPEN 0x80
+
struct nf_ct_tcp_flags {
__u8 flags;
__u8 mask;
diff --git a/net/netfilter/nf_conntrack_proto_tcp.c b/net/netfilter/nf_conntrack_proto_tcp.c
index e97cdc1..2c1fc7e 100644
--- a/net/netfilter/nf_conntrack_proto_tcp.c
+++ b/net/netfilter/nf_conntrack_proto_tcp.c
@@ -981,6 +981,20 @@ static int tcp_packet(struct nf_conn *ct,
return NF_ACCEPT; /* Don't change state */
}
break;
+ case TCP_CONNTRACK_SYN_SENT2:
+ /* tcp_conntracks table is not smart enough to handle
+ * simultaneous open.
+ */
+ ct->proto.tcp.last_flags |= IP_CT_TCP_SIMULTANEOUS_OPEN;
+ break;
+ case TCP_CONNTRACK_SYN_RECV:
+ if (dir == IP_CT_DIR_REPLY && index == TCP_ACK_SET &&
+ ct->proto.tcp.last_flags & IP_CT_TCP_SIMULTANEOUS_OPEN) {
+ /* We want to set the assured bit */
+ old_state = TCP_CONNTRACK_SYN_RECV;
+ new_state = TCP_CONNTRACK_ESTABLISHED;
+ }
+ break;
case TCP_CONNTRACK_CLOSE:
if (index == TCP_RST_SET
&& (ct->proto.tcp.seen[!dir].flags & IP_CT_TCP_FLAG_MAXACK_SET)
Best regards,
Jozsef
-
E-mail : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-18 10:27 ` Jozsef Kadlecsik
@ 2018-04-18 11:30 ` Dominique Martinet
2018-04-18 11:37 ` Jozsef Kadlecsik
0 siblings, 1 reply; 26+ messages in thread
From: Dominique Martinet @ 2018-04-18 11:30 UTC (permalink / raw)
To: Jozsef Kadlecsik
Cc: Florian Westphal, Michal Kubecek, netdev,
Marcelo Ricardo Leitner, Eric Dumazet
Jozsef Kadlecsik wrote on Wed, Apr 18, 2018:
> Thanks for the testing! One more line is required, however: we have to get
> the assured bit set for the connection, see the new patch below.
I think it actually was better before. If I understand things correctly
at this point (when we get in the case TCP_CONNTRACK_SYN_RECV) we will
have seen SYN(out) SYN(in) SYNACK(out), but not the final ACK(in) yet.
Leaving old state as it was will not set the assured bit, but that will
be set on the next packet because old_state == new_state == established
at that point and the connection will really be setup then.
I don't think anything will blow up if we do either way, but strictly
speaking I'm more comfortable with the former.
I'll test the new patch regardless, I left work so can't reproduce
anymore but will yell tomorrow if it does explode ;)
> The tcp_conntracks state table could be fixed with introducing a new
> state, but that part is exposed to userspace (ctnetlink) and ugly
> compatibility code would be required for backward compatibility.
I agree a new state is more work than it is worth, I'm happy to leave it
as is.
--
Dominique Martinet | Asmadeus
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: tcp hang when socket fills up ?
2018-04-18 11:30 ` Dominique Martinet
@ 2018-04-18 11:37 ` Jozsef Kadlecsik
0 siblings, 0 replies; 26+ messages in thread
From: Jozsef Kadlecsik @ 2018-04-18 11:37 UTC (permalink / raw)
To: Dominique Martinet
Cc: Florian Westphal, Michal Kubecek, netdev,
Marcelo Ricardo Leitner, Eric Dumazet
On Wed, 18 Apr 2018, Dominique Martinet wrote:
> Jozsef Kadlecsik wrote on Wed, Apr 18, 2018:
> > Thanks for the testing! One more line is required, however: we have to get
> > the assured bit set for the connection, see the new patch below.
>
> I think it actually was better before. If I understand things correctly
> at this point (when we get in the case TCP_CONNTRACK_SYN_RECV) we will
> have seen SYN(out) SYN(in) SYNACK(out), but not the final ACK(in) yet.
>
> Leaving old state as it was will not set the assured bit, but that will
> be set on the next packet because old_state == new_state == established
> at that point and the connection will really be setup then.
Yes, you are right: the first patch is better than the second one.
Overthinking :-)
Best regards,
Jozsef
-
E-mail : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2018-04-18 11:37 UTC | newest]
Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-04-06 9:07 tcp hang when socket fills up ? Dominique Martinet
2018-04-13 9:42 ` Dominique Martinet
2018-04-13 15:01 ` Eric Dumazet
2018-04-13 16:32 ` Michal Kubecek
2018-04-14 1:09 ` Dominique Martinet
2018-04-14 1:39 ` Eric Dumazet
2018-04-14 1:55 ` Dominique Martinet
2018-04-16 1:47 ` Dominique Martinet
2018-04-16 2:26 ` Eric Dumazet
2018-04-16 3:55 ` Dominique Martinet
2018-04-16 4:03 ` Dominique Martinet
2018-04-16 11:01 ` Florian Westphal
2018-04-17 3:52 ` Dominique Martinet
2018-04-17 5:28 ` Eric Dumazet
2018-04-17 6:52 ` Michal Kubecek
2018-04-17 9:20 ` Michal Kubecek
2018-04-17 12:34 ` Dominique Martinet
2018-04-17 13:00 ` Michal Kubecek
2018-04-17 13:29 ` Florian Westphal
2018-04-18 8:13 ` Jozsef Kadlecsik
2018-04-18 8:30 ` Dominique Martinet
2018-04-18 9:36 ` Dominique Martinet
2018-04-18 10:27 ` Jozsef Kadlecsik
2018-04-18 11:30 ` Dominique Martinet
2018-04-18 11:37 ` Jozsef Kadlecsik
2018-04-16 20:43 ` Marcelo Ricardo Leitner
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.