TCP LAST ACK incorrectly treated as invalid

TCP LAST ACK incorrectly treated as invalid

[vDev]

It looks like TCP conntrack is incorrectly treating "LAST ACK" (ACK
from peer for last FIN) as the sequence number comparison does not
take into account that the last ACK sequence number will be one
greater than the previous sequence number.

Here's the code that seems to be the cause of it in tcp_in_window()
[nf_conntrack_proto_tcp.c]:

       pr_debug("tcp_in_window: I=%i II=%i III=%i IV=%i\n",
                 before(seq, sender->td_maxend + 1),
                 after(end, sender->td_end - receiver->td_maxwin - 1),
                 before(sack, receiver->td_end + 1),
                 after(sack, receiver->td_end - MAXACKWINDOW(sender) - 1));

        if (before(seq, sender->td_maxend + 1) &&
            after(end, sender->td_end - receiver->td_maxwin - 1) &&
            before(sack, receiver->td_end + 1) &&
            after(sack, receiver->td_end - MAXACKWINDOW(sender) - 1)) {

III in pr_debug will be 0 as "before(sack, receiver->td_end + 1)" will
evaluate to false as sack will be equal to (receiver->td_end + 1) for
the last ACK. The sequence number will be one greater in the last ACK.

Any feedback will be helpful. Thanks for looking into this.

Re: TCP LAST ACK incorrectly treated as invalid

[Jozsef Kadlecsik]

On Mon, 20 Oct 2014, vDev wrote:

> It looks like TCP conntrack is incorrectly treating "LAST ACK" (ACK
> from peer for last FIN) as the sequence number comparison does not
> take into account that the last ACK sequence number will be one
> greater than the previous sequence number.
> 
> Here's the code that seems to be the cause of it in tcp_in_window()
> [nf_conntrack_proto_tcp.c]:
> 
>        pr_debug("tcp_in_window: I=%i II=%i III=%i IV=%i\n",
>                  before(seq, sender->td_maxend + 1),
>                  after(end, sender->td_end - receiver->td_maxwin - 1),
>                  before(sack, receiver->td_end + 1),
>                  after(sack, receiver->td_end - MAXACKWINDOW(sender) - 1));
> 
>         if (before(seq, sender->td_maxend + 1) &&
>             after(end, sender->td_end - receiver->td_maxwin - 1) &&
>             before(sack, receiver->td_end + 1) &&
>             after(sack, receiver->td_end - MAXACKWINDOW(sender) - 1)) {
> 
> III in pr_debug will be 0 as "before(sack, receiver->td_end + 1)" will
> evaluate to false as sack will be equal to (receiver->td_end + 1) for
> the last ACK. The sequence number will be one greater in the last ACK.
> 
> Any feedback will be helpful. Thanks for looking into this.

Maybe you are not taking into account that the last ACK acks FIN+ACK and 
FIN is counted in the sequence numbers and it's stored in td_end. Please 
see segment_seq_plus_len() in nf_conntrack_proto_tcp.c.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: TCP LAST ACK incorrectly treated as invalid

[vDev]

Thanks, Jozsef. I see sequence number compensation in
segment_seq_plus_len() but does that not cover the ACK sequence
numbers. The variables (sack and receiver->td_end) are off by 1. It
seems like receiver->td_end must be incremented by 1 for last ACK.
Here's a trace:

[  775.980000] seq=726609914 ack=94044192+(0) sack=94044192+(0)
win=14600 end=726609914
[  775.980000] tcp_in_window: sender end=726609914 maxend=726611414
maxwin=14600 scale=0 receiver end=94044191 maxend=94058791 maxwin=1500
scale=0
[  775.980000] tcp_in_window: I=1 II=1 III=0 IV=1
[  775.980000] tcp_in_window: res=0 sender end=726609914
maxend=726611414 maxwin=14600 receiver end=94044191 maxend=94058791
maxwin=1500

As you can see above sack is 94044192 and receiver end is 94044191. As
a result III is 0 and res=0 which causes -NF_ACCEPT to be returned.



On Tue, Oct 21, 2014 at 1:57 AM, Jozsef Kadlecsik
<kadlec@blackhole.kfki.hu> wrote:
> On Mon, 20 Oct 2014, vDev wrote:
>
>> It looks like TCP conntrack is incorrectly treating "LAST ACK" (ACK
>> from peer for last FIN) as the sequence number comparison does not
>> take into account that the last ACK sequence number will be one
>> greater than the previous sequence number.
>>
>> Here's the code that seems to be the cause of it in tcp_in_window()
>> [nf_conntrack_proto_tcp.c]:
>>
>>        pr_debug("tcp_in_window: I=%i II=%i III=%i IV=%i\n",
>>                  before(seq, sender->td_maxend + 1),
>>                  after(end, sender->td_end - receiver->td_maxwin - 1),
>>                  before(sack, receiver->td_end + 1),
>>                  after(sack, receiver->td_end - MAXACKWINDOW(sender) - 1));
>>
>>         if (before(seq, sender->td_maxend + 1) &&
>>             after(end, sender->td_end - receiver->td_maxwin - 1) &&
>>             before(sack, receiver->td_end + 1) &&
>>             after(sack, receiver->td_end - MAXACKWINDOW(sender) - 1)) {
>>
>> III in pr_debug will be 0 as "before(sack, receiver->td_end + 1)" will
>> evaluate to false as sack will be equal to (receiver->td_end + 1) for
>> the last ACK. The sequence number will be one greater in the last ACK.
>>
>> Any feedback will be helpful. Thanks for looking into this.
>
> Maybe you are not taking into account that the last ACK acks FIN+ACK and
> FIN is counted in the sequence numbers and it's stored in td_end. Please
> see segment_seq_plus_len() in nf_conntrack_proto_tcp.c.
>
> Best regards,
> Jozsef
> -
> E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
> PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
> Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
>           H-1525 Budapest 114, POB. 49, Hungary

Re: TCP LAST ACK incorrectly treated as invalid

[Jozsef Kadlecsik]

On Tue, 21 Oct 2014, vDev wrote:

> Thanks, Jozsef. I see sequence number compensation in
> segment_seq_plus_len() but does that not cover the ACK sequence
> numbers. The variables (sack and receiver->td_end) are off by 1. It
> seems like receiver->td_end must be incremented by 1 for last ACK.
> Here's a trace:
> 
> [  775.980000] seq=726609914 ack=94044192+(0) sack=94044192+(0)
> win=14600 end=726609914
> [  775.980000] tcp_in_window: sender end=726609914 maxend=726611414
> maxwin=14600 scale=0 receiver end=94044191 maxend=94058791 maxwin=1500
> scale=0
> [  775.980000] tcp_in_window: I=1 II=1 III=0 IV=1
> [  775.980000] tcp_in_window: res=0 sender end=726609914
> maxend=726611414 maxwin=14600 receiver end=94044191 maxend=94058791
> maxwin=1500
> 
> As you can see above sack is 94044192 and receiver end is 94044191. As
> a result III is 0 and res=0 which causes -NF_ACCEPT to be returned.

It's a single packet, it doesn't tell anything about the TCP stream.
A full TCP stream dump is required from the very first SYN to the very 
last packet.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: TCP LAST ACK incorrectly treated as invalid

[vDev]

OK. Here it is. Please look for III=0 and subsequent res=0.

<7>[ 2238.060000] tcp_new: sender end=4004184065 maxend=4004184065
maxwin=1500 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
<7>[ 2238.060000] tcp_in_window: START
<7>[ 2238.060000] tcp_in_window:
<7>[ 2238.060000] seq=4004184064 ack=0+(0) sack=0+(0) win=1500 end=4004184065
<7>[ 2238.060000] tcp_in_window: sender end=4004184065
maxend=4004184065 maxwin=1500 scale=0 receiver end=0 maxend=0 maxwin=0
scale=0
<7>[ 2238.060000] tcp_in_window:
<7>[ 2238.060000] seq=4004184064 ack=0+(0) sack=0+(0) win=1500 end=4004184065
<7>[ 2238.060000] tcp_in_window: sender end=4004184065
maxend=4004184065 maxwin=1500 scale=0 receiver end=0 maxend=0 maxwin=0
scale=0
<7>[ 2238.060000] tcp_in_window: I=1 II=1 III=1 IV=1
<7>[ 2238.060000] tcp_in_window: res=1 sender end=4004184065
maxend=4004184065 maxwin=1500 receiver end=0 maxend=1500 maxwin=0
<7>[ 2238.060000] tcp_conntracks:
<7>[ 2238.060000] syn=1 ack=0 fin=0 rst=0 old=0 new=1
<7>[ 2238.110000] tcp_in_window: START
<7>[ 2238.110000] tcp_in_window:
<7>[ 2238.110000] seq=201772607 ack=4004184065+(0) sack=4004184065+(0)
win=14600 end=201772608
<7>[ 2238.110000] tcp_in_window: sender end=0 maxend=1500 maxwin=0
scale=0 receiver end=4004184065 maxend=4004184065 maxwin=1500 scale=0
<7>[ 2238.110000] tcp_in_window:
<7>[ 2238.110000] seq=201772607 ack=4004184065+(0) sack=4004184065+(0)
win=14600 end=201772608
<7>[ 2238.110000] tcp_in_window: sender end=201772608 maxend=201772608
maxwin=14600 scale=0 receiver end=4004184065 maxend=4004184065
maxwin=1500 scale=0
<7>[ 2238.110000] tcp_in_window: I=1 II=1 III=1 IV=1
<7>[ 2238.110000] tcp_in_window: res=1 sender end=201772608
maxend=201772608 maxwin=14600 receiver end=4004184065
maxend=4004198665 maxwin=1500
<7>[ 2238.110000] tcp_conntracks:
<7>[ 2238.110000] syn=1 ack=1 fin=0 rst=0 old=1 new=2
<7>[ 2238.120000] tcp_in_window: START
<7>[ 2238.120000] tcp_in_window:
<7>[ 2238.120000] seq=4004184065 ack=201772608+(0) sack=201772608+(0)
win=1500 end=4004184065
<7>[ 2238.120000] tcp_in_window: sender end=4004184065
maxend=4004198665 maxwin=1500 scale=0 receiver end=201772608
maxend=201772608 maxwin=14600 scale=0
<7>[ 2238.120000] tcp_in_window:
<7>[ 2238.120000] seq=4004184065 ack=201772608+(0) sack=201772608+(0)
win=1500 end=4004184065
<7>[ 2238.120000] tcp_in_window: sender end=4004184065
maxend=4004198665 maxwin=1500 scale=0 receiver end=201772608
maxend=201772608 maxwin=14600 scale=0
<7>[ 2238.120000] tcp_in_window: I=1 II=1 III=1 IV=1
<7>[ 2238.120000] tcp_in_window: res=1 sender end=4004184065
maxend=4004198665 maxwin=1500 receiver end=201772608 maxend=201774108
maxwin=14600
<7>[ 2238.120000] tcp_conntracks:
<7>[ 2238.120000] syn=0 ack=1 fin=0 rst=0 old=2 new=3
<7>[ 2238.810000] tcp_in_window: START
<7>[ 2238.810000] tcp_in_window:
<7>[ 2238.810000] seq=4004184065 ack=201772608+(0) sack=201772608+(0)
win=1500 end=4004184095
<7>[ 2238.810000] tcp_in_window: sender end=4004184065
maxend=4004198665 maxwin=1500 scale=0 receiver end=201772608
maxend=201774108 maxwin=14600 scale=0
<7>[ 2238.810000] tcp_in_window:
<7>[ 2238.810000] seq=4004184065 ack=201772608+(0) sack=201772608+(0)
win=1500 end=4004184095
<7>[ 2238.810000] tcp_in_window: sender end=4004184065
maxend=4004198665 maxwin=1500 scale=0 receiver end=201772608
maxend=201774108 maxwin=14600 scale=0
<7>[ 2238.810000] tcp_in_window: I=1 II=1 III=1 IV=1
<7>[ 2238.810000] tcp_in_window: res=1 sender end=4004184095
maxend=4004198665 maxwin=1500 receiver end=201772608 maxend=201774108
maxwin=14600
<7>[ 2238.810000] tcp_conntracks:
<7>[ 2238.810000] syn=0 ack=1 fin=0 rst=0 old=3 new=3
<7>[ 2238.870000] tcp_in_window: START
<7>[ 2238.870000] tcp_in_window:
<7>[ 2238.870000] seq=201772608 ack=4004184095+(0) sack=4004184095+(0)
win=14600 end=201772608
<7>[ 2238.870000] tcp_in_window: sender end=201772608 maxend=201774108
maxwin=14600 scale=0 receiver end=4004184095 maxend=4004198665
maxwin=1500 scale=0
<7>[ 2238.870000] tcp_in_window:
<7>[ 2238.870000] seq=201772608 ack=4004184095+(0) sack=4004184095+(0)
win=14600 end=201772608
<7>[ 2238.870000] tcp_in_window: sender end=201772608 maxend=201774108
maxwin=14600 scale=0 receiver end=4004184095 maxend=4004198665
maxwin=1500 scale=0
<7>[ 2238.870000] tcp_in_window: I=1 II=1 III=1 IV=1
<7>[ 2238.870000] tcp_in_window: res=1 sender end=201772608
maxend=201774108 maxwin=14600 receiver end=4004184095
maxend=4004198695 maxwin=1500
<7>[ 2238.870000] tcp_conntracks:
<7>[ 2238.870000] syn=0 ack=1 fin=0 rst=0 old=3 new=3
<7>[ 2238.870000] tcp_in_window: START
<7>[ 2238.870000] tcp_in_window:
<7>[ 2238.870000] seq=201772608 ack=4004184095+(0) sack=4004184095+(0)
win=14600 end=201772626
<7>[ 2238.870000] tcp_in_window: sender end=201772608 maxend=201774108
maxwin=14600 scale=0 receiver end=4004184095 maxend=4004198695
maxwin=1500 scale=0
<7>[ 2238.870000] tcp_in_window:
<7>[ 2238.870000] seq=201772608 ack=4004184095+(0) sack=4004184095+(0)
win=14600 end=201772626
<7>[ 2238.870000] tcp_in_window: sender end=201772608 maxend=201774108
maxwin=14600 scale=0 receiver end=4004184095 maxend=4004198695
maxwin=1500 scale=0
<7>[ 2238.870000] tcp_in_window: I=1 II=1 III=1 IV=1
<7>[ 2238.870000] tcp_in_window: res=1 sender end=201772626
maxend=201774108 maxwin=14600 receiver end=4004184095
maxend=4004198695 maxwin=1500
<7>[ 2238.870000] tcp_conntracks:
<7>[ 2238.870000] syn=0 ack=1 fin=0 rst=0 old=3 new=3
<7>[ 2238.870000] tcp_conntracks:
<7>[ 2238.870000] syn=0 ack=1 fin=1 rst=0 old=3 new=4
<7>[ 2238.880000] tcp_in_window: START
<7>[ 2238.880000] tcp_in_window:
<7>[ 2238.880000] seq=4004184095 ack=201772626+(0) sack=201772626+(0)
win=1500 end=4004184095
<7>[ 2238.880000] tcp_in_window: sender end=4004184095
maxend=4004198695 maxwin=1500 scale=0 receiver end=201772626
maxend=201774108 maxwin=14600 scale=0
<7>[ 2238.880000] tcp_in_window:
<7>[ 2238.880000] seq=4004184095 ack=201772626+(0) sack=201772626+(0)
win=1500 end=4004184095
<7>[ 2238.880000] tcp_in_window: sender end=4004184095
maxend=4004198695 maxwin=1500 scale=0 receiver end=201772626
maxend=201774108 maxwin=14600 scale=0
<7>[ 2238.880000] tcp_in_window: I=1 II=1 III=1 IV=1
<7>[ 2238.880000] tcp_in_window: res=1 sender end=4004184095
maxend=4004198695 maxwin=1500 receiver end=201772626 maxend=201774126
maxwin=14600
<7>[ 2238.880000] tcp_conntracks:
<7>[ 2238.880000] syn=0 ack=1 fin=0 rst=0 old=4 new=5
<7>[ 2238.880000] tcp_conntracks:
<7>[ 2238.880000] syn=0 ack=1 fin=1 rst=0 old=5 new=6
<7>[ 2238.940000] tcp_in_window: START
<7>[ 2238.940000] tcp_in_window:
<7>[ 2238.940000] seq=201772627 ack=4004184096+(0) sack=4004184096+(0)
win=14600 end=201772627
<7>[ 2238.940000] tcp_in_window: sender end=201772626 maxend=201774126
maxwin=14600 scale=0 receiver end=4004184095 maxend=4004198695
maxwin=1500 scale=0
<7>[ 2238.940000] tcp_in_window:
<7>[ 2238.940000] seq=201772626 ack=4004184096+(0) sack=4004184096+(0)
win=14600 end=201772626
<7>[ 2238.940000] tcp_in_window: sender end=201772626 maxend=201774126
maxwin=14600 scale=0 receiver end=4004184095 maxend=4004198695
maxwin=1500 scale=0
<7>[ 2238.940000] tcp_in_window: I=1 II=1 III=0 IV=1
<7>[ 2238.940000] tcp_in_window: res=0 sender end=201772626
maxend=201774126 maxwin=14600 receiver end=4004184095
maxend=4004198695 maxwin=1500
<7>[ 2238.940000] nf_ct_tcp: invalid new deleting.
<7>[ 2241.930000] tcp_conntracks:
<7>[ 2241.930000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
<7>[ 2244.990000] tcp_conntracks:
<7>[ 2244.990000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
<7>[ 2248.060000] tcp_conntracks:
<7>[ 2248.060000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
<7>[ 2251.120000] tcp_conntracks:
<7>[ 2251.120000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
<7>[ 2254.190000] tcp_conntracks:
<7>[ 2254.190000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
<7>[ 2257.250000] tcp_conntracks:
<7>[ 2257.250000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
<7>[ 2260.320000] tcp_conntracks:
<7>[ 2260.320000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
<7>[ 2263.380000] tcp_conntracks:
<7>[ 2263.380000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
<7>[ 2266.440000] tcp_conntracks:
<7>[ 2266.440000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
<7>[ 2269.510000] tcp_conntracks:
<7>[ 2269.510000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
<7>[ 2272.570000] tcp_conntracks:
<7>[ 2272.570000] syn=0 ack=0 fin=0 rst=1 old=6 new=8


On Tue, Oct 21, 2014 at 1:39 PM, Jozsef Kadlecsik
<kadlec@blackhole.kfki.hu> wrote:
> On Tue, 21 Oct 2014, vDev wrote:
>
>> Thanks, Jozsef. I see sequence number compensation in
>> segment_seq_plus_len() but does that not cover the ACK sequence
>> numbers. The variables (sack and receiver->td_end) are off by 1. It
>> seems like receiver->td_end must be incremented by 1 for last ACK.
>> Here's a trace:
>>
>> [  775.980000] seq=726609914 ack=94044192+(0) sack=94044192+(0)
>> win=14600 end=726609914
>> [  775.980000] tcp_in_window: sender end=726609914 maxend=726611414
>> maxwin=14600 scale=0 receiver end=94044191 maxend=94058791 maxwin=1500
>> scale=0
>> [  775.980000] tcp_in_window: I=1 II=1 III=0 IV=1
>> [  775.980000] tcp_in_window: res=0 sender end=726609914
>> maxend=726611414 maxwin=14600 receiver end=94044191 maxend=94058791
>> maxwin=1500
>>
>> As you can see above sack is 94044192 and receiver end is 94044191. As
>> a result III is 0 and res=0 which causes -NF_ACCEPT to be returned.
>
> It's a single packet, it doesn't tell anything about the TCP stream.
> A full TCP stream dump is required from the very first SYN to the very
> last packet.
>
> Best regards,
> Jozsef
> -
> E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
> PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
> Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
>           H-1525 Budapest 114, POB. 49, Hungary

Re: TCP LAST ACK incorrectly treated as invalid

[Jozsef Kadlecsik]

On Tue, 21 Oct 2014, vDev wrote:

> OK. Here it is. Please look for III=0 and subsequent res=0.

This is not the dump of the TCP stream. Please send a tcpdump recording!

[...] 
> <7>[ 2238.870000] tcp_conntracks:
> <7>[ 2238.870000] syn=0 ack=1 fin=0 rst=0 old=3 new=3
> <7>[ 2238.870000] tcp_conntracks:
> <7>[ 2238.870000] syn=0 ack=1 fin=1 rst=0 old=3 new=4

It seems there was an ignored packet here. But without a tcpdump, nothing 
much can be said about it.

Best regard,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: TCP LAST ACK incorrectly treated as invalid

[Jozsef Kadlecsik]

On Tue, 21 Oct 2014, Jozsef Kadlecsik wrote:

> On Tue, 21 Oct 2014, vDev wrote:
> 
> > OK. Here it is. Please look for III=0 and subsequent res=0.
> 
> This is not the dump of the TCP stream. Please send a tcpdump recording!

And do not send it as an ascii dump but as a pcap file...

Best regards,
Joysef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: TCP LAST ACK incorrectly treated as invalid

[vDev]

I am unable to post a pcap file at this point but here is an ASCII
output. The last ACK (ID: 6200) was considered invalid due to III
evaluating to 0 and res=0 in the earlier trace.

tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
262144 bytes
16:39:34.462344 IP (tos 0x0, ttl 63, id 11, offset 0, flags [none],
proto TCP (6), length 44)
    Linux_Router.1026 > REMOTE_SERVER.63001: Flags [S], cksum 0xeaa4
(correct), seq 1278607360, win 1500, options [mss 536], length 0
16:39:34.518272 IP (tos 0x0, ttl 52, id 0, offset 0, flags [DF], proto
TCP (6), length 44)
    REMOTE_SERVER.63001 > Linux_Router.1026: Flags [S.], cksum 0xaf17
(correct), seq 913887703, ack 1278607361, win 14600, options [mss
536], length 0
16:39:34.529264 IP (tos 0x0, ttl 63, id 12, offset 0, flags [none],
proto TCP (6), length 40)
    Linux_Router.1026 > REMOTE_SERVER.63001: Flags [.], cksum 0xf664
(correct), seq 1, ack 1, win 1500, length 0
16:39:35.199013 IP (tos 0x0, ttl 63, id 13, offset 0, flags [none],
proto TCP (6), length 70)
    Linux_Router.1026 > REMOTE_SERVER.63001: Flags [P.], cksum 0x54a7
(correct), seq 1:31, ack 1, win 1500, length 30
16:39:35.255619 IP (tos 0x0, ttl 52, id 6197, offset 0, flags [DF],
proto TCP (6), length 40)
    REMOTE_SERVER.63001 > Linux_Router.1026: Flags [.], cksum 0xc31a
(correct), seq 1, ack 31, win 14600, length 0
16:39:35.256164 IP (tos 0x0, ttl 52, id 6198, offset 0, flags [DF],
proto TCP (6), length 58)
    REMOTE_SERVER.63001 > Linux_Router.1026: Flags [P.], cksum 0xc6ca
(correct), seq 1:19, ack 31, win 14600, length 18
16:39:35.256521 IP (tos 0x0, ttl 52, id 6199, offset 0, flags [DF],
proto TCP (6), length 40)
    REMOTE_SERVER.63001 > Linux_Router.1026: Flags [F.], cksum 0xc307
(correct), seq 19, ack 31, win 14600, length 0
16:39:35.266911 IP (tos 0x0, ttl 63, id 14, offset 0, flags [none],
proto TCP (6), length 40)
    Linux_Router.1026 > REMOTE_SERVER.63001: Flags [.], cksum 0xf634
(correct), seq 31, ack 19, win 1500, length 0
16:39:35.273232 IP (tos 0x0, ttl 63, id 15, offset 0, flags [none],
proto TCP (6), length 40)
    Linux_Router.1026 > REMOTE_SERVER.63001: Flags [F.], cksum 0xf632
(correct), seq 31, ack 20, win 1500, length 0
16:39:35.328857 IP (tos 0x0, ttl 52, id 6200, offset 0, flags [DF],
proto TCP (6), length 40)
    REMOTE_SERVER.63001 > Linux_Router.1026: Flags [.], cksum 0xc306
(correct), seq 20, ack 32, win 14600, length 0
16:39:35.329297 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
TCP (6), length 40)
    Linux_Router.1026 > REMOTE_SERVER.63001: Flags [R], cksum 0x047f
(correct), seq 1278607392, win 0, length 0
16:39:38.293532 IP (tos 0x0, ttl 63, id 16, offset 0, flags [none],
proto TCP (6), length 40)
    Linux_Router.1026 > REMOTE_SERVER.63001: Flags [F.], cksum 0xf632
(correct), seq 31, ack 20, win 1500, length 0
16:39:41.333370 IP (tos 0x0, ttl 63, id 17, offset 0, flags [none],
proto TCP (6), length 40)
    Linux_Router.1026 > REMOTE_SERVER.63001: Flags [F.], cksum 0xf632
(correct), seq 31, ack 20, win 1500, length 0
16:39:44.373300 IP (tos 0x0, ttl 63, id 18, offset 0, flags [none],
proto TCP (6), length 40)
    Linux_Router.1026 > REMOTE_SERVER.63001: Flags [F.], cksum 0xf632
(correct), seq 31, ack 20, win 1500, length 0
16:39:47.413089 IP (tos 0x0, ttl 63, id 19, offset 0, flags [none],
proto TCP (6), length 40)
    Linux_Router.1026 > REMOTE_SERVER.63001: Flags [F.], cksum 0xf632
(correct), seq 31, ack 20, win 1500, length 0
16:39:50.452954 IP (tos 0x0, ttl 63, id 20, offset 0, flags [none],
proto TCP (6), length 40)
    Linux_Router.1026 > REMOTE_SERVER.63001: Flags [F.], cksum 0xf632
(correct), seq 31, ack 20, win 1500, length 0
16:39:53.492753 IP (tos 0x0, ttl 63, id 21, offset 0, flags [none],
proto TCP (6), length 40)
    Linux_Router.1026 > REMOTE_SERVER.63001: Flags [F.], cksum 0xf632
(correct), seq 31, ack 20, win 1500, length 0
16:39:56.532626 IP (tos 0x0, ttl 63, id 22, offset 0, flags [none],
proto TCP (6), length 40)
    Linux_Router.1026 > REMOTE_SERVER.63001: Flags [F.], cksum 0xf632
(correct), seq 31, ack 20, win 1500, length 0
16:39:59.572464 IP (tos 0x0, ttl 63, id 23, offset 0, flags [none],
proto TCP (6), length 40)
    Linux_Router.1026 > REMOTE_SERVER.63001: Flags [F.], cksum 0xf632
(correct), seq 31, ack 20, win 1500, length 0
16:40:02.612307 IP (tos 0x0, ttl 63, id 24, offset 0, flags [none],
proto TCP (6), length 40)
    Linux_Router.1026 > REMOTE_SERVER.63001: Flags [F.], cksum 0xf632
(correct), seq 31, ack 20, win 1500, length 0
16:40:05.652150 IP (tos 0x0, ttl 63, id 25, offset 0, flags [none],
proto TCP (6), length 40)
    Linux_Router.1026 > REMOTE_SERVER.63001: Flags [F.], cksum 0xf632
(correct), seq 31, ack 20, win 1500, length 0
16:40:08.691960 IP (tos 0x0, ttl 63, id 26, offset 0, flags [none],
proto TCP (6), length 40)
    Linux_Router.1026 > REMOTE_SERVER.63001: Flags [R], cksum 0xcb77
(correct), seq 1278621991, win 0, length 0


On Tue, Oct 21, 2014 at 3:49 PM, Jozsef Kadlecsik
<kadlec@blackhole.kfki.hu> wrote:
> On Tue, 21 Oct 2014, Jozsef Kadlecsik wrote:
>
>> On Tue, 21 Oct 2014, vDev wrote:
>>
>> > OK. Here it is. Please look for III=0 and subsequent res=0.
>>
>> This is not the dump of the TCP stream. Please send a tcpdump recording!
>
> And do not send it as an ascii dump but as a pcap file...
>
> Best regards,
> Joysef
> -
> E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
> PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
> Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
>           H-1525 Budapest 114, POB. 49, Hungary

Re: TCP LAST ACK incorrectly treated as invalid

[Jozsef Kadlecsik]

On Tue, 21 Oct 2014, vDev wrote:

> I am unable to post a pcap file at this point but here is an ASCII
> output. The last ACK (ID: 6200) was considered invalid due to III
> evaluating to 0 and res=0 in the earlier trace.

That cannot be the last ACK, because it's a FIN, ACK packet:

>     Linux_Router.1026 > REMOTE_SERVER.63001: Flags [F.], cksum 0xf632
> (correct), seq 31, ack 20, win 1500, length 0
> 16:39:35.328857 IP (tos 0x0, ttl 52, id 6200, offset 0, flags [DF],
> proto TCP (6), length 40)

However, this is still not enough. The debug lines corresponding to the 
captured packets are required too. Moreover, previously there were packets 
skipping the window checking in the debug log, without enough debug data 
to identify them:

> <7>[ 2238.870000] tcp_conntracks:
> <7>[ 2238.870000] syn=0 ack=1 fin=0 rst=0 old=3 new=3
> <7>[ 2238.870000] tcp_conntracks:
> <7>[ 2238.870000] syn=0 ack=1 fin=1 rst=0 old=3 new=4

So please apply this small patch, which adds the missing debug info:

diff --git a/net/netfilter/nf_conntrack_proto_tcp.c b/net/netfilter/nf_conntrack_proto_tcp.c
index 44d1ea3..655170f 100644
--- a/net/netfilter/nf_conntrack_proto_tcp.c
+++ b/net/netfilter/nf_conntrack_proto_tcp.c
@@ -830,12 +830,16 @@ static int tcp_packet(struct nf_conn *ct,
 	th = skb_header_pointer(skb, dataoff, sizeof(_tcph), &_tcph);
 	BUG_ON(th == NULL);
 
+	pr_debug("tcp_packet: ");
 	spin_lock_bh(&ct->lock);
 	old_state = ct->proto.tcp.state;
 	dir = CTINFO2DIR(ctinfo);
 	index = get_conntrack_index(th);
 	new_state = tcp_conntracks[dir][index][old_state];
 	tuple = &ct->tuplehash[dir].tuple;
+	pr_debug("dir=%u, seq=%u ack=%u win=%u end=%u\n",
+		 dir, ntohl(th->seq), ntohl(th->ack_seq), ntohs(th->window),
+		 segment_seq_plus_len(ntohl(th->seq), skb->len, dataoff, th));
 
 	switch (new_state) {
 	case TCP_CONNTRACK_SYN_SENT:


and send me the kernel debug messages and the tcpdump recording of the 
packet flow. If you must send the tcpdump in ascii, then print absolute 
and not relative TCP sequence numbers.

At the moment the only thing I see about the packet flow is that the first 
reply packet answering the first FIN does not ack the FIN flag itself. 
However the TCP conntrack state table advances the state to CLOSE_WAIT, 
instead of keeping the FIN_WAIT state. But that's still not a problem 
in handling the next packets.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: TCP LAST ACK incorrectly treated as invalid

[vDev]

[-- Attachment #1: Type: text/plain, Size: 2987 bytes --]

Thanks, Jozsef. Attached is the new packet capture and trace with patch applied.



On Wed, Oct 22, 2014 at 2:37 PM, Jozsef Kadlecsik
<kadlec@blackhole.kfki.hu> wrote:
> On Tue, 21 Oct 2014, vDev wrote:
>
>> I am unable to post a pcap file at this point but here is an ASCII
>> output. The last ACK (ID: 6200) was considered invalid due to III
>> evaluating to 0 and res=0 in the earlier trace.
>
> That cannot be the last ACK, because it's a FIN, ACK packet:
>
>>     Linux_Router.1026 > REMOTE_SERVER.63001: Flags [F.], cksum 0xf632
>> (correct), seq 31, ack 20, win 1500, length 0
>> 16:39:35.328857 IP (tos 0x0, ttl 52, id 6200, offset 0, flags [DF],
>> proto TCP (6), length 40)
>
> However, this is still not enough. The debug lines corresponding to the
> captured packets are required too. Moreover, previously there were packets
> skipping the window checking in the debug log, without enough debug data
> to identify them:
>
>> <7>[ 2238.870000] tcp_conntracks:
>> <7>[ 2238.870000] syn=0 ack=1 fin=0 rst=0 old=3 new=3
>> <7>[ 2238.870000] tcp_conntracks:
>> <7>[ 2238.870000] syn=0 ack=1 fin=1 rst=0 old=3 new=4
>
> So please apply this small patch, which adds the missing debug info:
>
> diff --git a/net/netfilter/nf_conntrack_proto_tcp.c b/net/netfilter/nf_conntrack_proto_tcp.c
> index 44d1ea3..655170f 100644
> --- a/net/netfilter/nf_conntrack_proto_tcp.c
> +++ b/net/netfilter/nf_conntrack_proto_tcp.c
> @@ -830,12 +830,16 @@ static int tcp_packet(struct nf_conn *ct,
>         th = skb_header_pointer(skb, dataoff, sizeof(_tcph), &_tcph);
>         BUG_ON(th == NULL);
>
> +       pr_debug("tcp_packet: ");
>         spin_lock_bh(&ct->lock);
>         old_state = ct->proto.tcp.state;
>         dir = CTINFO2DIR(ctinfo);
>         index = get_conntrack_index(th);
>         new_state = tcp_conntracks[dir][index][old_state];
>         tuple = &ct->tuplehash[dir].tuple;
> +       pr_debug("dir=%u, seq=%u ack=%u win=%u end=%u\n",
> +                dir, ntohl(th->seq), ntohl(th->ack_seq), ntohs(th->window),
> +                segment_seq_plus_len(ntohl(th->seq), skb->len, dataoff, th));
>
>         switch (new_state) {
>         case TCP_CONNTRACK_SYN_SENT:
>
>
> and send me the kernel debug messages and the tcpdump recording of the
> packet flow. If you must send the tcpdump in ascii, then print absolute
> and not relative TCP sequence numbers.
>
> At the moment the only thing I see about the packet flow is that the first
> reply packet answering the first FIN does not ack the FIN flag itself.
> However the TCP conntrack state table advances the state to CLOSE_WAIT,
> instead of keeping the FIN_WAIT state. But that's still not a problem
> in handling the next packets.
>
> Best regards,
> Jozsef
> -
> E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
> PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
> Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
>           H-1525 Budapest 114, POB. 49, Hungary

[-- Attachment #2: Log.txt --]
[-- Type: text/plain, Size: 27717 bytes --]

root /tmp # tcpdump -n -XX -vv -S -i eth2 host Remote_Server
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size 262144 bytes
16:20:20.820187 IP (tos 0x0, ttl 63, id 1628, offset 0, flags [none], proto TCP (6), length 44)
    Linux_Router.1039 > Remote_Server.63001: Flags [S], cksum 0xde22 (correct), seq 2561327104, win 1500, options [mss 536], length 0
        0x0000:  001d b5cf 433d 0041 ac00 256a 0800 4500  ....C=.A..%j..E.
        0x0010:  002c 065c 0000 3f06 1081 434e 77fb 0cb3  .,.\..?...CNw...
        0x0020:  9cf3 040f f619 98aa c000 0000 0000 6002  ..............`.
        0x0030:  05dc de22 0000 0204 0218                 ..."......
16:20:20.876252 IP (tos 0x0, ttl 52, id 0, offset 0, flags [DF], proto TCP (6), length 44)
    Remote_Server.63001 > Linux_Router.1039: Flags [S.], cksum 0x428b (correct), seq 3661860374, ack 2561327105, win 14600, options [mss 536], length 0
        0x0000:  0041 ac00 256a 001d b5cf 433d 0800 4500  .A..%j....C=..E.
        0x0010:  002c 0000 4000 3406 e1dc 0cb3 9cf3 434e  .,..@.4.......CN
        0x0020:  77fb f619 040f da43 8e16 98aa c001 6012  w......C......`.
        0x0030:  3908 428b 0000 0204 0218 0000            9.B.........
16:20:20.886805 IP (tos 0x0, ttl 63, id 1629, offset 0, flags [none], proto TCP (6), length 40)
    Linux_Router.1039 > Remote_Server.63001: Flags [.], cksum 0x89d8 (correct), seq 2561327105, ack 3661860375, win 1500, length 0
        0x0000:  001d b5cf 433d 0041 ac00 256a 0800 4500  ....C=.A..%j..E.
        0x0010:  0028 065d 0000 3f06 1084 434e 77fb 0cb3  .(.]..?...CNw...
        0x0020:  9cf3 040f f619 98aa c001 da43 8e17 5010  ...........C..P.
        0x0030:  05dc 89d8 0000                           ......
16:20:21.593340 IP (tos 0x0, ttl 63, id 1630, offset 0, flags [none], proto TCP (6), length 70)
    Linux_Router.1039 > Remote_Server.63001: Flags [P.], cksum 0xe81a (correct), seq 2561327105:2561327135, ack 3661860375, win 1500, length 30
        0x0000:  001d b5cf 433d 0041 ac00 256a 0800 4500  ....C=.A..%j..E.
        0x0010:  0046 065e 0000 3f06 1065 434e 77fb 0cb3  .F.^..?..eCNw...
        0x0020:  9cf3 040f f619 98aa c001 da43 8e17 5018  ...........C..P.
        0x0030:  05dc e81a 0000 024d 3533 3033 3130 3030  .......M53031000
        0x0040:  4c30 3030 3038 301c 4035 1c1c 1c1c 302e  L000080.@5....0.
        0x0050:  3030 2303                                00#.
16:20:21.652968 IP (tos 0x0, ttl 52, id 24150, offset 0, flags [DF], proto TCP (6), length 40)
    Remote_Server.63001 > Linux_Router.1039: Flags [.], cksum 0x568e (correct), seq 3661860375, ack 2561327135, win 14600, length 0
        0x0000:  0041 ac00 256a 001d b5cf 433d 0800 4500  .A..%j....C=..E.
        0x0010:  0028 5e56 4000 3406 838a 0cb3 9cf3 434e  .(^V@.4.......CN
        0x0020:  77fb f619 040f da43 8e17 98aa c01f 5010  w......C......P.
        0x0030:  3908 568e 0000 0000 0000 0000            9.V.........
16:20:21.653394 IP (tos 0x0, ttl 52, id 24151, offset 0, flags [DF], proto TCP (6), length 58)
    Remote_Server.63001 > Linux_Router.1039: Flags [P.], cksum 0x5a3e (correct), seq 3661860375:3661860393, ack 2561327135, win 14600, length 18
        0x0000:  0041 ac00 256a 001d b5cf 433d 0800 4500  .A..%j....C=..E.
        0x0010:  003a 5e57 4000 3406 8377 0cb3 9cf3 434e  .:^W@.4..w....CN
        0x0020:  77fb f619 040f da43 8e17 98aa c01f 5018  w......C......P.
        0x0030:  3908 5a3e 0000 0245 5252 4f52 3a20 4241  9.Z>...ERROR:.BA
        0x0040:  4420 5343 414e 0339                      D.SCAN.9
16:20:21.653783 IP (tos 0x0, ttl 52, id 24152, offset 0, flags [DF], proto TCP (6), length 40)
    Remote_Server.63001 > Linux_Router.1039: Flags [F.], cksum 0x567b (correct), seq 3661860393, ack 2561327135, win 14600, length 0
        0x0000:  0041 ac00 256a 001d b5cf 433d 0800 4500  .A..%j....C=..E.
        0x0010:  0028 5e58 4000 3406 8388 0cb3 9cf3 434e  .(^X@.4.......CN
        0x0020:  77fb f619 040f da43 8e29 98aa c01f 5011  w......C.)....P.
        0x0030:  3908 567b 0000 0000 0000 0000            9.V{........
16:20:21.664588 IP (tos 0x0, ttl 63, id 1631, offset 0, flags [none], proto TCP (6), length 40)
    Linux_Router.1039 > Remote_Server.63001: Flags [.], cksum 0x89a8 (correct), seq 2561327135, ack 3661860393, win 1500, length 0
        0x0000:  001d b5cf 433d 0041 ac00 256a 0800 4500  ....C=.A..%j..E.
        0x0010:  0028 065f 0000 3f06 1082 434e 77fb 0cb3  .(._..?...CNw...
        0x0020:  9cf3 040f f619 98aa c01f da43 8e29 5010  ...........C.)P.
        0x0030:  05dc 89a8 0000                           ......
16:20:21.671235 IP (tos 0x0, ttl 63, id 1632, offset 0, flags [none], proto TCP (6), length 40)
    Linux_Router.1039 > Remote_Server.63001: Flags [F.], cksum 0x89a6 (correct), seq 2561327135, ack 3661860394, win 1500, length 0
        0x0000:  001d b5cf 433d 0041 ac00 256a 0800 4500  ....C=.A..%j..E.
        0x0010:  0028 0660 0000 3f06 1081 434e 77fb 0cb3  .(.`..?...CNw...
        0x0020:  9cf3 040f f619 98aa c01f da43 8e2a 5011  ...........C.*P.
        0x0030:  05dc 89a6 0000                           ......
16:20:21.726839 IP (tos 0x0, ttl 52, id 24153, offset 0, flags [DF], proto TCP (6), length 40)
    Remote_Server.63001 > Linux_Router.1039: Flags [.], cksum 0x567a (correct), seq 3661860394, ack 2561327136, win 14600, length 0
        0x0000:  0041 ac00 256a 001d b5cf 433d 0800 4500  .A..%j....C=..E.
        0x0010:  0028 5e59 4000 3406 8387 0cb3 9cf3 434e  .(^Y@.4.......CN
        0x0020:  77fb f619 040f da43 8e2a 98aa c020 5010  w......C.*....P.
        0x0030:  3908 567a 0000 0000 0000 0000            9.Vz........
16:20:21.727338 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    Linux_Router.1039 > Remote_Server.63001: Flags [R], cksum 0xf7fc (correct), seq 2561327136, win 0, length 0
        0x0000:  001d b5cf 433d 0041 ac00 256a 0800 4500  ....C=.A..%j..E.
        0x0010:  0028 0000 4000 4006 d5e0 434e 77fb 0cb3  .(..@.@...CNw...
        0x0020:  9cf3 040f f619 98aa c020 0000 0000 5004  ..............P.
        0x0030:  0000 f7fc 0000                           ......
16:20:24.676125 IP (tos 0x0, ttl 63, id 1633, offset 0, flags [none], proto TCP (6), length 40)
    Linux_Router.1039 > Remote_Server.63001: Flags [F.], cksum 0x89a6 (correct), seq 2561327135, ack 3661860394, win 1500, length 0
        0x0000:  001d b5cf 433d 0041 ac00 256a 0800 4500  ....C=.A..%j..E.
        0x0010:  0028 0661 0000 3f06 1080 434e 77fb 0cb3  .(.a..?...CNw...
        0x0020:  9cf3 040f f619 98aa c01f da43 8e2a 5011  ...........C.*P.
        0x0030:  05dc 89a6 0000                           ......
16:20:27.715963 IP (tos 0x0, ttl 63, id 1634, offset 0, flags [none], proto TCP (6), length 40)
    Linux_Router.1039 > Remote_Server.63001: Flags [F.], cksum 0x89a6 (correct), seq 2561327135, ack 3661860394, win 1500, length 0
        0x0000:  001d b5cf 433d 0041 ac00 256a 0800 4500  ....C=.A..%j..E.
        0x0010:  0028 0662 0000 3f06 107f 434e 77fb 0cb3  .(.b..?...CNw...
        0x0020:  9cf3 040f f619 98aa c01f da43 8e2a 5011  ...........C.*P.
        0x0030:  05dc 89a6 0000                           ......
16:20:30.755758 IP (tos 0x0, ttl 63, id 1635, offset 0, flags [none], proto TCP (6), length 40)
    Linux_Router.1039 > Remote_Server.63001: Flags [F.], cksum 0x89a6 (correct), seq 2561327135, ack 3661860394, win 1500, length 0
        0x0000:  001d b5cf 433d 0041 ac00 256a 0800 4500  ....C=.A..%j..E.
        0x0010:  0028 0663 0000 3f06 107e 434e 77fb 0cb3  .(.c..?..~CNw...
        0x0020:  9cf3 040f f619 98aa c01f da43 8e2a 5011  ...........C.*P.
        0x0030:  05dc 89a6 0000                           ......
16:20:33.795670 IP (tos 0x0, ttl 63, id 1636, offset 0, flags [none], proto TCP (6), length 40)
    Linux_Router.1039 > Remote_Server.63001: Flags [F.], cksum 0x89a6 (correct), seq 2561327135, ack 3661860394, win 1500, length 0
        0x0000:  001d b5cf 433d 0041 ac00 256a 0800 4500  ....C=.A..%j..E.
        0x0010:  0028 0664 0000 3f06 107d 434e 77fb 0cb3  .(.d..?..}CNw...
        0x0020:  9cf3 040f f619 98aa c01f da43 8e2a 5011  ...........C.*P.
        0x0030:  05dc 89a6 0000                           ......
16:20:36.835543 IP (tos 0x0, ttl 63, id 1637, offset 0, flags [none], proto TCP (6), length 40)
    Linux_Router.1039 > Remote_Server.63001: Flags [F.], cksum 0x89a6 (correct), seq 2561327135, ack 3661860394, win 1500, length 0
        0x0000:  001d b5cf 433d 0041 ac00 256a 0800 4500  ....C=.A..%j..E.
        0x0010:  0028 0665 0000 3f06 107c 434e 77fb 0cb3  .(.e..?..|CNw...
        0x0020:  9cf3 040f f619 98aa c01f da43 8e2a 5011  ...........C.*P.
        0x0030:  05dc 89a6 0000                           ......
16:20:39.875373 IP (tos 0x0, ttl 63, id 1638, offset 0, flags [none], proto TCP (6), length 40)
    Linux_Router.1039 > Remote_Server.63001: Flags [F.], cksum 0x89a6 (correct), seq 2561327135, ack 3661860394, win 1500, length 0
        0x0000:  001d b5cf 433d 0041 ac00 256a 0800 4500  ....C=.A..%j..E.
        0x0010:  0028 0666 0000 3f06 107b 434e 77fb 0cb3  .(.f..?..{CNw...
        0x0020:  9cf3 040f f619 98aa c01f da43 8e2a 5011  ...........C.*P.
        0x0030:  05dc 89a6 0000                           ......
16:20:42.915207 IP (tos 0x0, ttl 63, id 1639, offset 0, flags [none], proto TCP (6), length 40)
    Linux_Router.1039 > Remote_Server.63001: Flags [F.], cksum 0x89a6 (correct), seq 2561327135, ack 3661860394, win 1500, length 0
        0x0000:  001d b5cf 433d 0041 ac00 256a 0800 4500  ....C=.A..%j..E.
        0x0010:  0028 0667 0000 3f06 107a 434e 77fb 0cb3  .(.g..?..zCNw...
        0x0020:  9cf3 040f f619 98aa c01f da43 8e2a 5011  ...........C.*P.
        0x0030:  05dc 89a6 0000                           ......
16:20:45.955108 IP (tos 0x0, ttl 63, id 1641, offset 0, flags [none], proto TCP (6), length 40)
    Linux_Router.1039 > Remote_Server.63001: Flags [F.], cksum 0x89a6 (correct), seq 2561327135, ack 3661860394, win 1500, length 0
        0x0000:  001d b5cf 433d 0041 ac00 256a 0800 4500  ....C=.A..%j..E.
        0x0010:  0028 0669 0000 3f06 1078 434e 77fb 0cb3  .(.i..?..xCNw...
        0x0020:  9cf3 040f f619 98aa c01f da43 8e2a 5011  ...........C.*P.
        0x0030:  05dc 89a6 0000                           ......
16:20:48.994909 IP (tos 0x0, ttl 63, id 1642, offset 0, flags [none], proto TCP (6), length 40)
    Linux_Router.1039 > Remote_Server.63001: Flags [F.], cksum 0x89a6 (correct), seq 2561327135, ack 3661860394, win 1500, length 0
        0x0000:  001d b5cf 433d 0041 ac00 256a 0800 4500  ....C=.A..%j..E.
        0x0010:  0028 066a 0000 3f06 1077 434e 77fb 0cb3  .(.j..?..wCNw...
        0x0020:  9cf3 040f f619 98aa c01f da43 8e2a 5011  ...........C.*P.
        0x0030:  05dc 89a6 0000                           ......
16:20:52.034766 IP (tos 0x0, ttl 63, id 1643, offset 0, flags [none], proto TCP (6), length 40)
    Linux_Router.1039 > Remote_Server.63001: Flags [F.], cksum 0x89a6 (correct), seq 2561327135, ack 3661860394, win 1500, length 0
        0x0000:  001d b5cf 433d 0041 ac00 256a 0800 4500  ....C=.A..%j..E.
        0x0010:  0028 066b 0000 3f06 1076 434e 77fb 0cb3  .(.k..?..vCNw...
        0x0020:  9cf3 040f f619 98aa c01f da43 8e2a 5011  ...........C.*P.
        0x0030:  05dc 89a6 0000                           ......
16:20:55.074560 IP (tos 0x0, ttl 63, id 1644, offset 0, flags [none], proto TCP (6), length 40)
    Linux_Router.1039 > Remote_Server.63001: Flags [R], cksum 0xbef5 (correct), seq 2561341735, win 0, length 0
        0x0000:  001d b5cf 433d 0041 ac00 256a 0800 4500  ....C=.A..%j..E.
        0x0010:  0028 066c 0000 3f06 1075 434e 77fb 0cb3  .(.l..?..uCNw...
        0x0020:  9cf3 040f f619 98aa f927 0000 0000 5004  .........'....P.
        0x0030:  0000 bef5 0000                           ......

22 packets captured
22 packets received by filter
0 packets dropped by kernel
root /tmp # dmesg 
0] device eth2 entered promiscuous mode
[  251.250000] tcp_new: sender end=2054078465 maxend=2054078465 maxwin=1500 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
[  251.250000] tcp_packet: 
[  251.250000] dir=0, seq=2054078464 ack=0 win=1500 end=2054078465
[  251.250000] tcp_in_window: START
[  251.250000] tcp_in_window: 
[  251.250000] seq=2054078464 ack=0+(0) sack=0+(0) win=1500 end=2054078465
[  251.250000] tcp_in_window: sender end=2054078465 maxend=2054078465 maxwin=1500 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
[  251.250000] tcp_in_window: 
[  251.250000] seq=2054078464 ack=0+(0) sack=0+(0) win=1500 end=2054078465
[  251.250000] tcp_in_window: sender end=2054078465 maxend=2054078465 maxwin=1500 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
[  251.250000] tcp_in_window: I=1 II=1 III=1 IV=1
[  251.250000] tcp_in_window: res=1 sender end=2054078465 maxend=2054078465 maxwin=1500 receiver end=0 maxend=1500 maxwin=0
[  251.250000] tcp_conntracks: 
[  251.250000] syn=1 ack=0 fin=0 rst=0 old=0 new=1
[  254.300000] tcp_new: sender end=2054078465 maxend=2054078465 maxwin=1500 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
[  254.300000] tcp_packet: 
[  254.300000] dir=0, seq=2054078464 ack=0 win=1500 end=2054078465
[  254.300000] tcp_in_window: START
[  254.300000] tcp_in_window: 
[  254.300000] seq=2054078464 ack=0+(0) sack=0+(0) win=1500 end=2054078465
[  254.300000] tcp_in_window: sender end=2054078465 maxend=2054078465 maxwin=1500 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
[  254.300000] tcp_in_window: 
[  254.300000] seq=2054078464 ack=0+(0) sack=0+(0) win=1500 end=2054078465
[  254.300000] tcp_in_window: sender end=2054078465 maxend=2054078465 maxwin=1500 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
[  254.300000] tcp_in_window: I=1 II=1 III=1 IV=1
[  254.300000] tcp_in_window: res=1 sender end=2054078465 maxend=2054078465 maxwin=1500 receiver end=0 maxend=1500 maxwin=0
[  254.300000] tcp_conntracks: 
[  254.300000] syn=1 ack=0 fin=0 rst=0 old=0 new=1
[  257.370000] tcp_new: sender end=2054078465 maxend=2054078465 maxwin=1500 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
[  257.370000] tcp_packet: 
[  257.370000] dir=0, seq=2054078464 ack=0 win=1500 end=2054078465
[  257.370000] tcp_in_window: START
[  257.370000] tcp_in_window: 
[  257.370000] seq=2054078464 ack=0+(0) sack=0+(0) win=1500 end=2054078465
[  257.370000] tcp_in_window: sender end=2054078465 maxend=2054078465 maxwin=1500 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
[  257.370000] tcp_in_window: 
[  257.370000] seq=2054078464 ack=0+(0) sack=0+(0) win=1500 end=2054078465
[  257.370000] tcp_in_window: sender end=2054078465 maxend=2054078465 maxwin=1500 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
[  257.370000] tcp_in_window: I=1 II=1 III=1 IV=1
[  257.370000] tcp_in_window: res=1 sender end=2054078465 maxend=2054078465 maxwin=1500 receiver end=0 maxend=1500 maxwin=0
[  257.370000] tcp_conntracks: 
[  257.370000] syn=1 ack=0 fin=0 rst=0 old=0 new=1
[  258.860000] device eth2 left promiscuous mode
[  260.430000] nf_ct_tcp: invalid new deleting.
[  283.400000] tcp_new: sender end=736947074 maxend=736947074 maxwin=8192 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
[  283.400000] tcp_packet: 
[  283.400000] dir=0, seq=736947073 ack=0 win=8192 end=736947074
[  283.400000] tcp_in_window: START
[  283.400000] tcp_in_window: 
[  283.400000] seq=736947073 ack=0+(0) sack=0+(0) win=8192 end=736947074
[  283.400000] tcp_in_window: sender end=736947074 maxend=736947074 maxwin=8192 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
[  283.400000] tcp_in_window: 
[  283.400000] seq=736947073 ack=0+(0) sack=0+(0) win=8192 end=736947074
[  283.400000] tcp_in_window: sender end=736947074 maxend=736947074 maxwin=8192 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
[  283.400000] tcp_in_window: I=1 II=1 III=1 IV=1
[  283.400000] tcp_in_window: res=1 sender end=736947074 maxend=736947074 maxwin=8192 receiver end=0 maxend=8192 maxwin=0
[  283.400000] tcp_conntracks: 
[  283.400000] syn=1 ack=0 fin=0 rst=0 old=0 new=1
[  283.400000] tcp_packet: 
[  283.400000] dir=1, seq=0 ack=736947074 win=0 end=0
[  283.400000] tcp_conntracks: 
[  283.400000] syn=0 ack=1 fin=0 rst=1 old=1 new=8
[  354.430000] console setup
[  368.820000] device eth2 entered promiscuous mode
[  376.100000] tcp_new: sender end=2561327105 maxend=2561327105 maxwin=1500 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
[  376.100000] tcp_packet: 
[  376.100000] dir=0, seq=2561327104 ack=0 win=1500 end=2561327105
[  376.100000] tcp_in_window: START
[  376.100000] tcp_in_window: 
[  376.100000] seq=2561327104 ack=0+(0) sack=0+(0) win=1500 end=2561327105
[  376.100000] tcp_in_window: sender end=2561327105 maxend=2561327105 maxwin=1500 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
[  376.100000] tcp_in_window: 
[  376.100000] seq=2561327104 ack=0+(0) sack=0+(0) win=1500 end=2561327105
[  376.100000] tcp_in_window: sender end=2561327105 maxend=2561327105 maxwin=1500 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
[  376.100000] tcp_in_window: I=1 II=1 III=1 IV=1
[  376.100000] tcp_in_window: res=1 sender end=2561327105 maxend=2561327105 maxwin=1500 receiver end=0 maxend=1500 maxwin=0
[  376.100000] tcp_conntracks: 
[  376.100000] syn=1 ack=0 fin=0 rst=0 old=0 new=1
[  376.160000] tcp_packet: 
[  376.160000] dir=1, seq=3661860374 ack=2561327105 win=14600 end=3661860375
[  376.160000] tcp_in_window: START
[  376.160000] tcp_in_window: 
[  376.160000] seq=3661860374 ack=2561327105+(0) sack=2561327105+(0) win=14600 end=3661860375
[  376.160000] tcp_in_window: sender end=0 maxend=1500 maxwin=0 scale=0 receiver end=2561327105 maxend=2561327105 maxwin=1500 scale=0
[  376.160000] tcp_in_window: 
[  376.160000] seq=3661860374 ack=2561327105+(0) sack=2561327105+(0) win=14600 end=3661860375
[  376.160000] tcp_in_window: sender end=3661860375 maxend=3661860375 maxwin=14600 scale=0 receiver end=2561327105 maxend=2561327105 maxwin=1500 scale=0
[  376.160000] tcp_in_window: I=1 II=1 III=1 IV=1
[  376.160000] tcp_in_window: res=1 sender end=3661860375 maxend=3661860375 maxwin=14600 receiver end=2561327105 maxend=2561341705 maxwin=1500
[  376.160000] tcp_conntracks: 
[  376.160000] syn=1 ack=1 fin=0 rst=0 old=1 new=2
[  376.170000] tcp_packet: 
[  376.170000] dir=0, seq=2561327105 ack=3661860375 win=1500 end=2561327105
[  376.170000] tcp_in_window: START
[  376.170000] tcp_in_window: 
[  376.170000] seq=2561327105 ack=3661860375+(0) sack=3661860375+(0) win=1500 end=2561327105
[  376.170000] tcp_in_window: sender end=2561327105 maxend=2561341705 maxwin=1500 scale=0 receiver end=3661860375 maxend=3661860375 maxwin=14600 scale=0
[  376.170000] tcp_in_window: 
[  376.170000] seq=2561327105 ack=3661860375+(0) sack=3661860375+(0) win=1500 end=2561327105
[  376.170000] tcp_in_window: sender end=2561327105 maxend=2561341705 maxwin=1500 scale=0 receiver end=3661860375 maxend=3661860375 maxwin=14600 scale=0
[  376.170000] tcp_in_window: I=1 II=1 III=1 IV=1
[  376.170000] tcp_in_window: res=1 sender end=2561327105 maxend=2561341705 maxwin=1500 receiver end=3661860375 maxend=3661861875 maxwin=14600
[  376.170000] tcp_conntracks: 
[  376.170000] syn=0 ack=1 fin=0 rst=0 old=2 new=3
[  376.880000] tcp_packet: 
[  376.880000] dir=0, seq=2561327105 ack=3661860375 win=1500 end=2561327135
[  376.880000] tcp_in_window: START
[  376.880000] tcp_in_window: 
[  376.880000] seq=2561327105 ack=3661860375+(0) sack=3661860375+(0) win=1500 end=2561327135
[  376.880000] tcp_in_window: sender end=2561327105 maxend=2561341705 maxwin=1500 scale=0 receiver end=3661860375 maxend=3661861875 maxwin=14600 scale=0
[  376.880000] tcp_in_window: 
[  376.880000] seq=2561327105 ack=3661860375+(0) sack=3661860375+(0) win=1500 end=2561327135
[  376.880000] tcp_in_window: sender end=2561327105 maxend=2561341705 maxwin=1500 scale=0 receiver end=3661860375 maxend=3661861875 maxwin=14600 scale=0
[  376.880000] tcp_in_window: I=1 II=1 III=1 IV=1
[  376.880000] tcp_in_window: res=1 sender end=2561327135 maxend=2561341705 maxwin=1500 receiver end=3661860375 maxend=3661861875 maxwin=14600
[  376.880000] tcp_conntracks: 
[  376.880000] syn=0 ack=1 fin=0 rst=0 old=3 new=3
[  376.950000] tcp_packet: 
[  376.950000] dir=1, seq=3661860375 ack=2561327135 win=14600 end=3661860375
[  376.950000] tcp_in_window: START
[  376.950000] tcp_in_window: 
[  376.950000] seq=3661860375 ack=2561327135+(0) sack=2561327135+(0) win=14600 end=3661860375
[  376.950000] tcp_in_window: sender end=3661860375 maxend=3661861875 maxwin=14600 scale=0 receiver end=2561327135 maxend=2561341705 maxwin=1500 scale=0
[  376.950000] tcp_in_window: 
[  376.950000] seq=3661860375 ack=2561327135+(0) sack=2561327135+(0) win=14600 end=3661860375
[  376.950000] tcp_in_window: sender end=3661860375 maxend=3661861875 maxwin=14600 scale=0 receiver end=2561327135 maxend=2561341705 maxwin=1500 scale=0
[  376.950000] tcp_in_window: I=1 II=1 III=1 IV=1
[  376.950000] tcp_in_window: res=1 sender end=3661860375 maxend=3661861875 maxwin=14600 receiver end=2561327135 maxend=2561341735 maxwin=1500
[  376.950000] tcp_conntracks: 
[  376.950000] syn=0 ack=1 fin=0 rst=0 old=3 new=3
[  376.950000] tcp_packet: 
[  376.950000] dir=1, seq=3661860375 ack=2561327135 win=14600 end=3661860393
[  376.950000] tcp_in_window: START
[  376.950000] tcp_in_window: 
[  376.950000] seq=3661860375 ack=2561327135+(0) sack=2561327135+(0) win=14600 end=3661860393
[  376.950000] tcp_in_window: sender end=3661860375 maxend=3661861875 maxwin=14600 scale=0 receiver end=2561327135 maxend=2561341735 maxwin=1500 scale=0
[  376.950000] tcp_in_window: 
[  376.950000] seq=3661860375 ack=2561327135+(0) sack=2561327135+(0) win=14600 end=3661860393
[  376.950000] tcp_in_window: sender end=3661860375 maxend=3661861875 maxwin=14600 scale=0 receiver end=2561327135 maxend=2561341735 maxwin=1500 scale=0
[  376.950000] tcp_in_window: I=1 II=1 III=1 IV=1
[  376.950000] tcp_in_window: res=1 sender end=3661860393 maxend=3661861875 maxwin=14600 receiver end=2561327135 maxend=2561341735 maxwin=1500
[  376.950000] tcp_conntracks: 
[  376.950000] syn=0 ack=1 fin=0 rst=0 old=3 new=3
[  376.950000] tcp_packet: 
[  376.950000] dir=1, seq=3661860393 ack=2561327135 win=14600 end=3661860394
[  376.950000] tcp_conntracks: 
[  376.950000] syn=0 ack=1 fin=1 rst=0 old=3 new=4
[  376.960000] tcp_packet: 
[  376.960000] dir=0, seq=2561327135 ack=3661860393 win=1500 end=2561327135
[  376.960000] tcp_in_window: START
[  376.960000] tcp_in_window: 
[  376.960000] seq=2561327135 ack=3661860393+(0) sack=3661860393+(0) win=1500 end=2561327135
[  376.960000] tcp_in_window: sender end=2561327135 maxend=2561341735 maxwin=1500 scale=0 receiver end=3661860393 maxend=3661861875 maxwin=14600 scale=0
[  376.960000] tcp_in_window: 
[  376.960000] seq=2561327135 ack=3661860393+(0) sack=3661860393+(0) win=1500 end=2561327135
[  376.960000] tcp_in_window: sender end=2561327135 maxend=2561341735 maxwin=1500 scale=0 receiver end=3661860393 maxend=3661861875 maxwin=14600 scale=0
[  376.960000] tcp_in_window: I=1 II=1 III=1 IV=1
[  376.960000] tcp_in_window: res=1 sender end=2561327135 maxend=2561341735 maxwin=1500 receiver end=3661860393 maxend=3661861893 maxwin=14600
[  376.960000] tcp_conntracks: 
[  376.960000] syn=0 ack=1 fin=0 rst=0 old=4 new=5
[  376.960000] tcp_packet: 
[  376.960000] dir=0, seq=2561327135 ack=3661860394 win=1500 end=2561327136
[  376.960000] tcp_conntracks: 
[  376.960000] syn=0 ack=1 fin=1 rst=0 old=5 new=6
[  377.020000] tcp_packet: 
[  377.020000] dir=1, seq=3661860394 ack=2561327136 win=14600 end=3661860394
[  377.020000] tcp_in_window: START
[  377.020000] tcp_in_window: 
[  377.020000] seq=3661860394 ack=2561327136+(0) sack=2561327136+(0) win=14600 end=3661860394
[  377.020000] tcp_in_window: sender end=3661860393 maxend=3661861893 maxwin=14600 scale=0 receiver end=2561327135 maxend=2561341735 maxwin=1500 scale=0
[  377.020000] tcp_in_window: 
[  377.020000] seq=3661860393 ack=2561327136+(0) sack=2561327136+(0) win=14600 end=3661860393
[  377.020000] tcp_in_window: sender end=3661860393 maxend=3661861893 maxwin=14600 scale=0 receiver end=2561327135 maxend=2561341735 maxwin=1500 scale=0
[  377.020000] tcp_in_window: I=1 II=1 III=0 IV=1
[  377.020000] tcp_in_window: res=0 sender end=3661860393 maxend=3661861893 maxwin=14600 receiver end=2561327135 maxend=2561341735 maxwin=1500
[  377.020000] nf_ct_tcp: invalid new deleting.
[  379.990000] tcp_packet: 
[  379.990000] dir=0, seq=2561327135 ack=3661860394 win=1500 end=2561327136
[  379.990000] tcp_conntracks: 
[  379.990000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
[  383.060000] tcp_packet: 
[  383.060000] dir=0, seq=2561327135 ack=3661860394 win=1500 end=2561327136
[  383.060000] tcp_conntracks: 
[  383.060000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
[  386.120000] tcp_packet: 
[  386.120000] dir=0, seq=2561327135 ack=3661860394 win=1500 end=2561327136
[  386.120000] tcp_conntracks: 
[  386.120000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
[  389.190000] tcp_packet: 
[  389.190000] dir=0, seq=2561327135 ack=3661860394 win=1500 end=2561327136
[  389.190000] tcp_conntracks: 
[  389.190000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
[  392.250000] tcp_packet: 
[  392.250000] dir=0, seq=2561327135 ack=3661860394 win=1500 end=2561327136
[  392.250000] tcp_conntracks: 
[  392.250000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
[  395.310000] tcp_packet: 
[  395.310000] dir=0, seq=2561327135 ack=3661860394 win=1500 end=2561327136
[  395.310000] tcp_conntracks: 
[  395.310000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
[  398.380000] tcp_packet: 
[  398.380000] dir=0, seq=2561327135 ack=3661860394 win=1500 end=2561327136
[  398.380000] tcp_conntracks: 
[  398.380000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
[  401.440000] tcp_packet: 
[  401.440000] dir=0, seq=2561327135 ack=3661860394 win=1500 end=2561327136
[  401.440000] tcp_conntracks: 
[  401.440000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
[  404.510000] tcp_packet: 
[  404.510000] dir=0, seq=2561327135 ack=3661860394 win=1500 end=2561327136
[  404.510000] tcp_conntracks: 
[  404.510000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
[  407.570000] tcp_packet: 
[  407.570000] dir=0, seq=2561327135 ack=3661860394 win=1500 end=2561327136
[  407.570000] tcp_conntracks: 
[  407.570000] syn=0 ack=1 fin=1 rst=0 old=6 new=6
[  410.640000] tcp_packet: 
[  410.640000] dir=0, seq=2561341735 ack=0 win=0 end=2561341735
[  410.640000] tcp_conntracks: 
[  410.640000] syn=0 ack=0 fin=0 rst=1 old=6 new=8
[  416.860000] device eth2 left promiscuous mode
[  418.060000] tcp_new: sender end=499646465 maxend=499646465 maxwin=16384 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
[  418.060000] tcp_packet: 
[  418.060000] dir=0, seq=499646464 ack=0 win=16384 end=499646465
[  418.060000] tcp_in_window: START
[  418.060000] tcp_in_window: 
[  418.060000] seq=499646464 ack=0+(0) sack=0+(0) win=16384 end=499646465
[  418.060000] tcp_in_window: sender end=499646465 maxend=499646465 maxwin=16384 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
[  418.060000] tcp_in_window: 
[  418.060000] seq=499646464 ack=0+(0) sack=0+(0) win=16384 end=499646465
[  418.060000] tcp_in_window: sender end=499646465 maxend=499646465 maxwin=16384 scale=0 receiver end=0 maxend=0 maxwin=0 scale=0
[  418.060000] tcp_in_window: I=1 II=1 III=1 IV=1
[  418.060000] tcp_in_window: res=1 sender end=499646465 maxend=499646465 maxwin=16384 receiver end=0 maxend=16384 maxwin=0
[  418.060000] tcp_conntracks: 
[  418.060000] syn=1 ack=0 fin=0 rst=0 old=0 new=1
[  418.060000] tcp_packet: 
[  418.060000] dir=1, seq=0 ack=499646465 win=0 end=0
[  418.060000] tcp_conntracks: 
[  418.060000] syn=0 ack=1 fin=0 rst=1 old=1 new=8
root /tmp # 
 CTRL-A Z for help | 38400 8N1 | NOR | Minicom 2.2    | VT102 |      Offline                                                                             

Re: TCP LAST ACK incorrectly treated as invalid

[vDev]

That cannot be the last ACK, because it's a FIN, ACK packet:

>     Linux_Router.1026 > REMOTE_SERVER.63001: Flags [F.], cksum 0xf632
> (correct), seq 31, ack 20, win 1500, length 0
> 16:39:35.328857 IP (tos 0x0, ttl 52, id 6200, offset 0, flags [DF],
> proto TCP (6), length 40)

You looked at the previous packet. The correct packet is:

16:39:35.328857 IP (tos 0x0, ttl 52, id 6200, offset 0, flags [DF],
proto TCP (6), length 40)
    REMOTE_SERVER.63001 > Linux_Router.1026: Flags [.], cksum 0xc306
(correct), seq 20, ack 32, win 14600, length 0



On Wed, Oct 22, 2014 at 4:42 PM, vDev <vijaypas@gmail.com> wrote:
> Thanks, Jozsef. Attached is the new packet capture and trace with patch applied.
>
>
>
> On Wed, Oct 22, 2014 at 2:37 PM, Jozsef Kadlecsik
> <kadlec@blackhole.kfki.hu> wrote:
>> On Tue, 21 Oct 2014, vDev wrote:
>>
>>> I am unable to post a pcap file at this point but here is an ASCII
>>> output. The last ACK (ID: 6200) was considered invalid due to III
>>> evaluating to 0 and res=0 in the earlier trace.
>>
>> That cannot be the last ACK, because it's a FIN, ACK packet:
>>
>>>     Linux_Router.1026 > REMOTE_SERVER.63001: Flags [F.], cksum 0xf632
>>> (correct), seq 31, ack 20, win 1500, length 0
>>> 16:39:35.328857 IP (tos 0x0, ttl 52, id 6200, offset 0, flags [DF],
>>> proto TCP (6), length 40)
>>
>> However, this is still not enough. The debug lines corresponding to the
>> captured packets are required too. Moreover, previously there were packets
>> skipping the window checking in the debug log, without enough debug data
>> to identify them:
>>
>>> <7>[ 2238.870000] tcp_conntracks:
>>> <7>[ 2238.870000] syn=0 ack=1 fin=0 rst=0 old=3 new=3
>>> <7>[ 2238.870000] tcp_conntracks:
>>> <7>[ 2238.870000] syn=0 ack=1 fin=1 rst=0 old=3 new=4
>>
>> So please apply this small patch, which adds the missing debug info:
>>
>> diff --git a/net/netfilter/nf_conntrack_proto_tcp.c b/net/netfilter/nf_conntrack_proto_tcp.c
>> index 44d1ea3..655170f 100644
>> --- a/net/netfilter/nf_conntrack_proto_tcp.c
>> +++ b/net/netfilter/nf_conntrack_proto_tcp.c
>> @@ -830,12 +830,16 @@ static int tcp_packet(struct nf_conn *ct,
>>         th = skb_header_pointer(skb, dataoff, sizeof(_tcph), &_tcph);
>>         BUG_ON(th == NULL);
>>
>> +       pr_debug("tcp_packet: ");
>>         spin_lock_bh(&ct->lock);
>>         old_state = ct->proto.tcp.state;
>>         dir = CTINFO2DIR(ctinfo);
>>         index = get_conntrack_index(th);
>>         new_state = tcp_conntracks[dir][index][old_state];
>>         tuple = &ct->tuplehash[dir].tuple;
>> +       pr_debug("dir=%u, seq=%u ack=%u win=%u end=%u\n",
>> +                dir, ntohl(th->seq), ntohl(th->ack_seq), ntohs(th->window),
>> +                segment_seq_plus_len(ntohl(th->seq), skb->len, dataoff, th));
>>
>>         switch (new_state) {
>>         case TCP_CONNTRACK_SYN_SENT:
>>
>>
>> and send me the kernel debug messages and the tcpdump recording of the
>> packet flow. If you must send the tcpdump in ascii, then print absolute
>> and not relative TCP sequence numbers.
>>
>> At the moment the only thing I see about the packet flow is that the first
>> reply packet answering the first FIN does not ack the FIN flag itself.
>> However the TCP conntrack state table advances the state to CLOSE_WAIT,
>> instead of keeping the FIN_WAIT state. But that's still not a problem
>> in handling the next packets.
>>
>> Best regards,
>> Jozsef
>> -
>> E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
>> PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
>> Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
>>           H-1525 Budapest 114, POB. 49, Hungary

Re: TCP LAST ACK incorrectly treated as invalid

[Jozsef Kadlecsik]

On Wed, 22 Oct 2014, vDev wrote:

> Thanks, Jozsef. Attached is the new packet capture and trace with patch 
> applied.

Thanks, now packets and conntrack states can be compared.

Up to packet 6 everything is normal. However, look at packet 7:

16:20:21.653783 IP (tos 0x0, ttl 52, id 24152, offset 0, flags [DF],
    proto TCP (6), length 40)
    Remote_Server.63001 > Linux_Router.1039: Flags [F.], cksum 0x567b 
    (correct), seq 3661860393, ack 2561327135, win 14600, length 0

This is the first FIN packet, and the kernel debug log says:

[  376.950000] tcp_packet: 
[  376.950000] dir=1, seq=3661860393 ack=2561327135 win=14600 end=3661860394
[  376.950000] tcp_conntracks: 
[  376.950000] syn=0 ack=1 fin=1 rst=0 old=3 new=4

The previous conntrack state is in "old=3", i.e. 
TCP_CONNTRACK_ESTABLISHED. The new=4 means TCP_CONNTRACK_FIN_WAIT. However 
between

[  376.950000] tcp_packet:
[  376.950000] dir=1, seq=3661860393 ack=2561327135 win=14600 end=3661860394

and

[  376.950000] tcp_conntracks:
[  376.950000] syn=0 ack=1 fin=1 rst=0 old=3 new=4

there is a missing the call to tcp_in_window()! Therefore the internal 
counters of conntrack is not updated and later packets won't match the 
wrong internal states.

Why the call to tcp_in_window() is missing? Looking at tcp_packet(), 
there's nothing which could cause skipping it: the big switch about the 
new_state does not divert the handling of TCP_CONNTRACK_FIN_WAIT.

So, what's your kernel version number? You have got the source:
please post net/netfilter/nf_conntrack_proto_tcp.c

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: TCP LAST ACK incorrectly treated as invalid

[vDev]

Yes! I believe a piece of code that is machine dependent stole the
packet and thereby got things messed up in conntrack. A rebuild seems
to have taken care of the issue. Thanks for taking a look at this. The
kernel version is 2.6.35. Again, thanks!



On Thu, Oct 23, 2014 at 3:04 AM, Jozsef Kadlecsik
<kadlec@blackhole.kfki.hu> wrote:
> On Wed, 22 Oct 2014, vDev wrote:
>
>> Thanks, Jozsef. Attached is the new packet capture and trace with patch
>> applied.
>
> Thanks, now packets and conntrack states can be compared.
>
> Up to packet 6 everything is normal. However, look at packet 7:
>
> 16:20:21.653783 IP (tos 0x0, ttl 52, id 24152, offset 0, flags [DF],
>     proto TCP (6), length 40)
>     Remote_Server.63001 > Linux_Router.1039: Flags [F.], cksum 0x567b
>     (correct), seq 3661860393, ack 2561327135, win 14600, length 0
>
> This is the first FIN packet, and the kernel debug log says:
>
> [  376.950000] tcp_packet:
> [  376.950000] dir=1, seq=3661860393 ack=2561327135 win=14600 end=3661860394
> [  376.950000] tcp_conntracks:
> [  376.950000] syn=0 ack=1 fin=1 rst=0 old=3 new=4
>
> The previous conntrack state is in "old=3", i.e.
> TCP_CONNTRACK_ESTABLISHED. The new=4 means TCP_CONNTRACK_FIN_WAIT. However
> between
>
> [  376.950000] tcp_packet:
> [  376.950000] dir=1, seq=3661860393 ack=2561327135 win=14600 end=3661860394
>
> and
>
> [  376.950000] tcp_conntracks:
> [  376.950000] syn=0 ack=1 fin=1 rst=0 old=3 new=4
>
> there is a missing the call to tcp_in_window()! Therefore the internal
> counters of conntrack is not updated and later packets won't match the
> wrong internal states.
>
> Why the call to tcp_in_window() is missing? Looking at tcp_packet(),
> there's nothing which could cause skipping it: the big switch about the
> new_state does not divert the handling of TCP_CONNTRACK_FIN_WAIT.
>
> So, what's your kernel version number? You have got the source:
> please post net/netfilter/nf_conntrack_proto_tcp.c
>
> Best regards,
> Jozsef
> -
> E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
> PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
> Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
>           H-1525 Budapest 114, POB. 49, Hungary

TCP LAST ACK incorrectly treated as invalid

[vDev]

It looks like TCP conntrack is incorrectly treating "LAST ACK" (ACK
from peer for last FIN) as the sequence number comparison does not
take into account that the last ACK sequence number will be one
greater than the previous sequence number.

Here's the code that seems to be the cause of it in tcp_in_window()
[nf_conntrack_proto_tcp.c]:

       pr_debug("tcp_in_window: I=%i II=%i III=%i IV=%i\n",
                 before(seq, sender->td_maxend + 1),
                 after(end, sender->td_end - receiver->td_maxwin - 1),
                 before(sack, receiver->td_end + 1),
                 after(sack, receiver->td_end - MAXACKWINDOW(sender) - 1));

        if (before(seq, sender->td_maxend + 1) &&
            after(end, sender->td_end - receiver->td_maxwin - 1) &&
            before(sack, receiver->td_end + 1) &&
            after(sack, receiver->td_end - MAXACKWINDOW(sender) - 1)) {

III in pr_debug will be 0 as "before(sack, receiver->td_end + 1)" will
evaluate to false as sack will be equal to (receiver->td_end + 1) for
the last ACK. The sequence number will be one greater in the last ACK.

Any feedback will be helpful. Thanks for looking into this.