Connection tracking debugging?!

Connection tracking debugging?!

[Marco Gaiarin]

I've done some changes in a remote site, managed by a linux/netfilter
firewall; mostly i've added more clients, but also changed connectivity
(provider).


After that i've started to catch some little troubles, eg random
disconnection in videoconferencing (Zoom) and in and ICA client. Seems
to me vaguely a 'connection tracking' trouble...

I'v added this rules:

 iptables -A std-cleanup -m conntrack --ctstate INVALID -m limit --limit 1/sec -j LOG --log-prefix "C=std-cleanup A=inv L=err "
 iptables -A std-cleanup -m conntrack --ctstate INVALID -j DROP

linked to INPUT and FORWARD chain, ed effectively i catch 'invalid'
event:

 May 25 11:45:49 prosecco kernel: [789480.844612] C=std-cleanup A=inv L=err IN=enp0s25 OUT=ppp0 MACl:3b:e5:0f:02:e9:dc:4a:3e:42:19:29:08:00 SRC\x10.10.2.169 DST“.41.169.27 LEN@ TOS=0x00 PREC=0x00 TTL\x127 ID\x16685 DF PROTO=TCP SPTP944 DPTD3 WINDOW=0 RES=0x00 RST URGP=0 
 May 25 11:45:50 prosecco kernel: [789482.292680] C=std-cleanup A=inv L=err IN=enp0s25 OUT=ppp0 MACl:3b:e5:0f:02:e9:dc:4a:3e:42:19:29:08:00 SRC\x10.10.2.169 DST“.41.169.27 LEN@ TOS=0x00 PREC=0x00 TTL\x127 ID\x16873 DF PROTO=TCP SPTP940 DPTD3 WINDOW=0 RES=0x00 RST URGP=0 
 May 25 11:50:00 prosecco kernel: [789732.718655] C=std-cleanup A=inv L=err IN=enp0s25 OUT=ppp0 MACl:3b:e5:0f:02:e9:dc:4a:3e:42:19:29:08:00 SRC\x10.10.2.169 DST“.41.169.27 LEN@ TOS=0x00 PREC=0x00 TTL\x127 ID0802 DF PROTO=TCP SPTQ274 DPTD3 WINDOW=0 RES=0x00 RST URGP=0 

so seems to me that by some way the connection tracking 'loose' the
tracking, and clearly afterward the package get marked invalid, forcing
a reconnection.


Using 'conntrack' helper, lead nothing strange to me, or at least
nothing different from other similar installation that instead works as
expected.


How can i 'debug' this issue? Thanks.

-- 
dott. Marco Gaiarin				        GNUPG Key ID: 240A3D66
  Associazione ``La Nostra Famiglia''          http://www.lanostrafamiglia.it/
  Polo FVG   -   Via della Bontà, 7 - 33078   -   San Vito al Tagliamento (PN)
  marco.gaiarin(at)lanostrafamiglia.it   t +39-0434-842711   f +39-0434-842797

		Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
      http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
	(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)

Re: Connection tracking debugging?!

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 901 bytes --]

On 5/25/21 4:36 AM, Marco Gaiarin wrote:
> How can i 'debug' this issue? Thanks.

I'd check the output of dmesg to see if you're exhausting the state 
table.  If you are, you'll see all sorts of messages from the kernel. 
At least I did when I ran into this years ago.  Adding memory addressed 
the problem then.

Short of that low hanging fruit I'd start with packet captures so that 
you can watch the traffic flow.  I occasionally see invalid traffic 
after the flow should have been closed.

It looks like your client may be sending TCP Reset packets.  This could 
be directly related to how different systems terminate a TCP connection. 
  --  Even if the clients agree, they may be doing something different 
than the connection tracker helper expects, thus causing a subsequent 
packet to be considered invalid after a shorter shutdown.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4013 bytes --]

Re: Connection tracking debugging?!

[Marco Gaiarin]

Mandi! Grant Taylor
  In chel di` si favelave...

Sorry for the late answer, but i was busy in other things...

> > How can i 'debug' this issue? Thanks.
> I'd check the output of dmesg to see if you're exhausting the state table.
> If you are, you'll see all sorts of messages from the kernel. At least I did
> when I ran into this years ago.  Adding memory addressed the problem then.

No, i've not sayed that, but was the first things i've looked for, no
conntrack table overflow...


> Short of that low hanging fruit I'd start with packet captures so that you
> can watch the traffic flow.  I occasionally see invalid traffic after the
> flow should have been closed.
> 
> It looks like your client may be sending TCP Reset packets.  This could be
> directly related to how different systems terminate a TCP connection.  --
> Even if the clients agree, they may be doing something different than the
> connection tracker helper expects, thus causing a subsequent packet to be
> considered invalid after a shorter shutdown.

Could be that passing thru a proxy (SSL/CONNECT, squid) could lead to
more TCP resets? Seems that proxied connection reset more frequently...

I see in proxy cache.log file sometimes:

	2021/06/04 12:41:07| TunnelStateData::Connection::error: FD 20: read/write failure: (32) Broken pipe

-- 
dott. Marco Gaiarin				        GNUPG Key ID: 240A3D66
  Associazione ``La Nostra Famiglia''          http://www.lanostrafamiglia.it/
  Polo FVG   -   Via della Bontà, 7 - 33078   -   San Vito al Tagliamento (PN)
  marco.gaiarin(at)lanostrafamiglia.it   t +39-0434-842711   f +39-0434-842797

		Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
      http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
	(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)