[PATCH] doc: document danger of applying REJECT to INVALID CTs

[PATCH] doc: document danger of applying REJECT to INVALID CTs

[Jan Engelhardt]

Signed-off-by: Jan Engelhardt <jengelh@inai.de>
---
 extensions/libip6t_REJECT.man | 20 ++++++++++++++++++++
 extensions/libipt_REJECT.man  | 20 ++++++++++++++++++++
 2 files changed, 40 insertions(+)

diff --git a/extensions/libip6t_REJECT.man b/extensions/libip6t_REJECT.man
index 0030a51f..3c42768e 100644
--- a/extensions/libip6t_REJECT.man
+++ b/extensions/libip6t_REJECT.man
@@ -30,3 +30,23 @@ TCP RST packet to be sent back.  This is mainly useful for blocking
 hosts (which won't accept your mail otherwise).
 \fBtcp\-reset\fP
 can only be used with kernel versions 2.6.14 or later.
+.PP
+\fIWarning:\fP You should not indiscriminately apply the REJECT target to
+packets whose connection state is classified as INVALID; instead, you should
+only DROP these.
+.PP
+Consider a source host transmitting a packet P, with P experiencing so much
+delay along its path that the source host issues a retransmission, P_2, with
+P_2 being successful in reaching its destination and advancing the connection
+state normally. It is conceivable that the late-arriving P may be considered
+not to be associated with any connection tracking entry. Generating a reject
+response for a packet so classed would then terminate the healthy connection.
+.PP
+So, instead of:
+.PP
+-A INPUT ... -j REJECT
+.PP
+do consider using:
+.PP
+-A INPUT ... -m conntrack --ctstate INVALID -j DROP
+-A INPUT ... -j REJECT
diff --git a/extensions/libipt_REJECT.man b/extensions/libipt_REJECT.man
index 8a360ce7..cc47aead 100644
--- a/extensions/libipt_REJECT.man
+++ b/extensions/libipt_REJECT.man
@@ -30,3 +30,23 @@ TCP RST packet to be sent back.  This is mainly useful for blocking
 hosts (which won't accept your mail otherwise).
 .IP
 (*) Using icmp\-admin\-prohibited with kernels that do not support it will result in a plain DROP instead of REJECT
+.PP
+\fIWarning:\fP You should not indiscriminately apply the REJECT target to
+packets whose connection state is classified as INVALID; instead, you should
+only DROP these.
+.PP
+Consider a source host transmitting a packet P, with P experiencing so much
+delay along its path that the source host issues a retransmission, P_2, with
+P_2 being successful in reaching its destination and advancing the connection
+state normally. It is conceivable that the late-arriving P may be considered
+not to be associated with any connection tracking entry. Generating a reject
+response for a packet so classed would then terminate the healthy connection.
+.PP
+So, instead of:
+.PP
+-A INPUT ... -j REJECT
+.PP
+do consider using:
+.PP
+-A INPUT ... -m conntrack --ctstate INVALID -j DROP
+-A INPUT ... -j REJECT
-- 
2.26.2

Re: [PATCH] doc: document danger of applying REJECT to INVALID CTs

[Pablo Neira Ayuso]

Applied, thanks.

Re: [PATCH] document danger of '-j REJECT'ing of '-m state INVALID' packets

[Maciej Żenczykowski]

So I've never tried to figure out how things break, just observed that
they do - first many many years ago (close to 15ish) - between my wifi
connected laptop at home and my university server in the same city.
I've kept an INVALID->DROP rule in all my firewalls since then and not
had problems.  I vaguely recall seeing delayed packets when I debugged
it back then.

See for example: https://github.com/moby/libnetwork/issues/1090 for
others running into this.

Now we've hit an issue at work where a network misconfiguration has
asymmetric one way pathing with a result that some packets were
getting *massively* delayed, and it's been causing user firewalls to
generate tcp resets for 'too old' 'already ack'ed' packets (ie. dups).

While this is of course a misconfig, and it shouldn't happen, in
practice it sometimes simply does.
All it takes is for a packet to get into a long queue, and the network
path to shift (immediately after it) to a less congested path.
Due to bufferbloat those long queues can take seconds to drain and
exceed path rtt by orders of magnitude.

I *think* what happens is:

A non-final tcp packet gets massively delayed, the packet past that
makes it through to the receive, and triggers an ACK with SACK, which
makes it back to the sender and triggers a retransmit and the
connections keeps on making forward progress,  then eventually the
delayed packet arrives and it's no longer considered valid and
triggers a tcp reset.  Massively of course depends on the rtt and
retransmit aggressiveness.

Here's my attempt to demonstrate what I believe the problem to be:

(on a freshly booted clean/empty/idle fedora 31 vm)

iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -m state --state INVALID -j DROP
modprobe ifb
ip link set dev ifb0 up
tc qdisc add dev ifb0 root netem reorder 99% 0% delay 10s
tc qdisc add dev eth0 clsact
tc filter add dev eth0 ingress u32 match u32 0 0 action mirred egress
redirect dev ifb0
wget -O /dev/null https://git.kernel.org/torvalds/t/linux-5.7-rc4.tar.gz
iptables-save -c

...
/dev/null                             [     <=>
                           ] 169.58M  2.93MB/s    in 45s
2020-05-09 10:35:44 (3.81 MB/s) - ‘/dev/null’ saved [177819073]
...
[31750:181080717] -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
[244:1403178] -A INPUT -m state --state INVALID -j DROP


Now if I reboot, and run the same script, except instead of the
INVALID/DROP rule I do
  iptables -A INPUT -p tcp -j REJECT --reject-with tcp-reset
then the download never finishes (it hangs after 15MB @ 2MB/s and
eventually times out).

[4170:16758894] -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
[37:147454] -A INPUT -p tcp -j REJECT --reject-with tcp-reset

(arguably since this is a VM, and thus NAT'ed by my host, and then
again by the real ipv4 NAT, the setup isn't entirely clear, but I hope
it makes my point: INVALID state needs to be dropped, not rejected)

[PATCH] doc: document danger of applying REJECT to INVALID CTs

[Jan Engelhardt]

Signed-off-by: Jan Engelhardt <jengelh@inai.de>
---

Maciej's explanation on how INVALID+REJECT can lead to problems looks
convincing. I hereby present new manpage wording in the form of "if A, then B"
to better build the argument of avoiding REJECT. So the issue is not caused by
an _incoming_ TCP RST as the initial mail might have suggested,
but by RST generated by REJECT (--reject-with tcp-reset).

It is conceivable to me that a connection termination may occur with not only
TCP+RST, but also with TCP+ICMP and UDP+ICMP, so I trimmed any
protocol-specific wording too. Also trimmed is any mention of -j ACCEPT,
because rule order is not the point of the argument.


 extensions/libip6t_REJECT.man | 21 +++++++++++++++++++++
 extensions/libipt_REJECT.man  | 21 +++++++++++++++++++++
 2 files changed, 42 insertions(+)

diff --git a/extensions/libip6t_REJECT.man b/extensions/libip6t_REJECT.man
index 0030a51f..38183dd7 100644
--- a/extensions/libip6t_REJECT.man
+++ b/extensions/libip6t_REJECT.man
@@ -30,3 +30,24 @@ TCP RST packet to be sent back.  This is mainly useful for blocking
 hosts (which won't accept your mail otherwise).
 \fBtcp\-reset\fP
 can only be used with kernel versions 2.6.14 or later.
+.PP
+\fIWarning:\fP You should not indiscrimnately apply the REJECT target to
+packets whose connection state is classified as INVALID; instead, you should
+only DROP these:
+.PP
+Consider a source host retransmitting an original packet P as P_2 for any
+reason, and P_2 getting routed via a different path (load balancing/policy
+routing, or anything of the kind). Additionally, let P_2 experience so much
+delay that the source host issues \fIanother\fP retransmission, P_3, with P_3
+being succesful in reaching its destination and advancing the connection state
+normally. The delayed P_2, when it eventually is processed, may be considered
+to be not associated with any connection tracking entry. Generating a reject
+packet for such a belated packet would then terminate the healthy connection.
+.PP
+So, instead of:
+.PP
+-A INPUT -m conntrack --ctstate INVALID -j REJECT
+.PP
+do consider using:
+.PP
+-A INPUT -m conntrack --ctstate INVALID -j DROP
diff --git a/extensions/libipt_REJECT.man b/extensions/libipt_REJECT.man
index 8a360ce7..9e80d7ea 100644
--- a/extensions/libipt_REJECT.man
+++ b/extensions/libipt_REJECT.man
@@ -30,3 +30,24 @@ TCP RST packet to be sent back.  This is mainly useful for blocking
 hosts (which won't accept your mail otherwise).
 .IP
 (*) Using icmp\-admin\-prohibited with kernels that do not support it will result in a plain DROP instead of REJECT
+.PP
+\fIWarning:\fP You should not indiscrimnately apply the REJECT target to
+packets whose connection state is classified as INVALID; instead, you should
+only DROP these:
+.PP
+Consider a source host retransmitting an original packet P as P_2 for any
+reason, and P_2 getting routed via a different path (load balancing/policy
+routing, or anything of the kind). Additionally, let P_2 experience so much
+delay that the source host issues \fIanother\fP retransmission, P_3, with P_3
+being succesful in reaching its destination and advancing the connection state
+normally. The delayed P_2, when it eventually is processed, may be considered
+to be not associated with any connection tracking entry. Generating a reject
+packet for such a belated packet would then terminate the healthy connection.
+.PP
+So, instead of:
+.PP
+-A INPUT -m conntrack --ctstate INVALID -j REJECT
+.PP
+do consider using:
+.PP
+-A INPUT -m conntrack --ctstate INVALID -j DROP
-- 
2.26.2

Re: [PATCH] doc: document danger of applying REJECT to INVALID CTs

[Maciej Żenczykowski]

I *think* that your talk of 3 packets is not needed, ie. the initial
delayed packet doesn't have to be a retransmission.
It can be the first copy of that segment that gets massively delayed
and arrives late and causes problems,
by virtue of arriving after the retransmission already caused the
connection to move on.

Other than that this does seem perhaps a bit cleared than what I wrote.

Re: [PATCH] doc: document danger of applying REJECT to INVALID CTs

[Maciej Żenczykowski]

Also maybe the example should be:

instead of just:
-A INPUT ... -j REJECT
do:
-A INPUT ... -m conntrack --ctstate INVALID -j DROP
-A INPUT ... -j REJECT