"random" syn packets dropped

"random" syn packets dropped

[Bjørnar Ness]

Reposted from netfilter:

I am not sure if this is nftables related, but I post this issue here,
and see if any of you can come up with a clue to what might be
going on here.

Problem description:

When I create multiple tcp connections from the same client to
multiple dst hosts at the same time, the n'th syn packet is just
discarded by "something" in the kernel.

If I reorder the list of dst hosts, a different dst host will hang in SYN_SENT
on the client. This setup has been running for about a month, and we have
no changed that can explain this behavior.

What I am seeing on the firewall running kernel 4.8.1 is the following:

* the syn packet enters through the eth1.700 interface (tcdump)
* nft trace monitoring shows the packet beeing accepted on eth1.300 in
postrouting.
* tcpdump on the eth1.300 interface does not show the packet.
* rp_filter etc should not be kicking in here, (and also, "random"
hosts are dropped)
* conntrack table is not full
* this issue seem to suddenly appeared, is this a known bug?
* hint? All connections from the client is established from the same
source port.

-- 
Bj(/)rnar

Re: "random" syn packets dropped

[Florian Westphal]

Bjørnar Ness <bjornar.ness@gmail.com> wrote:
> I am not sure if this is nftables related, but I post this issue here,
> and see if any of you can come up with a clue to what might be
> going on here.
> 
> Problem description:
> 
> When I create multiple tcp connections from the same client to
> multiple dst hosts at the same time, the n'th syn packet is just
> discarded by "something" in the kernel.
> 
> If I reorder the list of dst hosts, a different dst host will hang in SYN_SENT
> on the client. This setup has been running for about a month, and we have
> no changed that can explain this behavior.
> 
> What I am seeing on the firewall running kernel 4.8.1 is the following:
> 
> * the syn packet enters through the eth1.700 interface (tcdump)
> * nft trace monitoring shows the packet beeing accepted on eth1.300 in
> postrouting.
> * tcpdump on the eth1.300 interface does not show the packet.
> * rp_filter etc should not be kicking in here, (and also, "random"
> hosts are dropped)
> * conntrack table is not full
> * this issue seem to suddenly appeared, is this a known bug?

No.

> * hint? All connections from the client is established from the same
> source port.

can you show conntrack -S output?

Is nat in use?

Does 'perf script net_dropmonitor' show anything?

Thanks.

Re: "random" syn packets dropped

[Bjørnar Ness]

2016-11-08 15:08 GMT+01:00 Florian Westphal <fw@strlen.de>:
> Bjørnar Ness <bjornar.ness@gmail.com> wrote:
>> * this issue seem to suddenly appeared, is this a known bug?
>
> No.
>
>> * hint? All connections from the client is established from the same
>> source port.
>
> can you show conntrack -S output?

cpu=0       searched=121227494 found=39167580 new=304 invalid=7359
ignore=323276 delete=58920554 delete_list=58920526 insert=304
insert_failed=0 drop=0 early_drop=0 error=7359 search_restart=0
cpu=1       searched=769896688 found=3223427856 new=1824070235
invalid=99676863 ignore=9456795 delete=1803700815
delete_list=1731666319 insert=1745298679 insert_failed=3 drop=3
early_drop=0 error=3674946 search_restart=2
cpu=2       searched=488983917 found=2793077405 new=1800015302
invalid=97556206 ignore=9424173 delete=1789649391
delete_list=1705077976 insert=1724463703 insert_failed=9 drop=9
early_drop=0 error=3748383 search_restart=2
cpu=3       searched=449939710 found=3134174470 new=1839530428
invalid=99941483 ignore=9734373 delete=1811219076
delete_list=1735637482 insert=1761666050 insert_failed=6 drop=6
early_drop=0 error=3914932 search_restart=2

> Is nat in use?

yes, but not on there connections

> Does 'perf script net_dropmonitor' show anything?

Sorry. no ftrace

-- 
Bj(/)rnar

Re: "random" syn packets dropped

[Bjørnar Ness]

Hello again.

After a restart of the machine, this problem disappeared for 10 days,
but unfortunately
showed up again yesterday. Seems like net_dropmonitor does not catch
this one, the
only output I get is:

nf_hook_slow                       171                         1

This is definately something you should look into. I would be glad to
assist if there
is anything I can do.

-- 
Bj(/)rnar

Re: "random" syn packets dropped

[Florian Westphal]

Bjørnar Ness <bjornar.ness@gmail.com> wrote:
> After a restart of the machine, this problem disappeared for 10 days,
> but unfortunately
> showed up again yesterday. Seems like net_dropmonitor does not catch
> this one, the
> only output I get is:
> 
> nf_hook_slow                       171                         1
>
> This is definately something you should look into. I would be glad to
> assist if there
> is anything I can do.

Maybe already fixed via

https://patchwork.ozlabs.org/patch/695586/
https://patchwork.ozlabs.org/patch/695587/

Re: "random" syn packets dropped

[Bjørnar Ness]

Hello, Florian

I applied the patches to net-next, but unfortunately this did fix the
bug I am seeing,
in addition, now the bug seems to appear immediately (also with
previous kernel),
not after ~10 days as I have seen before.

-- 
Bj(/)rnar

Re: "random" syn packets dropped

[Pablo Neira Ayuso]

On Thu, Nov 24, 2016 at 02:37:30PM +0100, Bjørnar Ness wrote:
> Hello, Florian
> 
> I applied the patches to net-next, but unfortunately this did fix the
> bug I am seeing,
> in addition, now the bug seems to appear immediately (also with
> previous kernel),
> not after ~10 days as I have seen before.

If you revert this, do you still hit problems?

commit 870190a9ec9075205c0fa795a09fa931694a3ff1
Author: Florian Westphal <fw@strlen.de>
Date:   Tue Jul 5 12:07:24 2016 +0200

    netfilter: nat: convert nat bysrc hash to rhashtable

It would be good to bisect to make sure this is related to the
rhashtable conversion.

Let us know,
Thanks Bjørnar.

Re: "random" syn packets dropped

[Bjørnar Ness]

Hello, Pablo.

Seems I was too quick to post this message, we were hit by an attack
just after the
upgrade, and the problems was this time caused by conntrack_max.
Currently it seems to
work with the patches, but the next week or two will probably show.

-- 
Bj(/)rnar

"random" syn packets dropped

[Bjørnar Ness]

I am not sure if this is nftables related, but I post this issue here,
and see if any of you can
come up with a clue to what might be going on.

Problem description:

When I create multiple tcp connections from the same client to
multiple dst hosts at the same
time, the n'th syn packet seems to be just discarded by "something".
If I reorder the list of dst
hosts, a different dst host will hang in SYN_SENT.

What I am seeing on the firewall running kernel 4.8.1 is the following:

* the syn packet enters through the eth1.700 interface
* the packet does _not_ exit through eth1.300 interface as supposed to.
* nft trace monitoring shows the packet beeing accepted on eth1.300 in
postrouting.
* rp_filter etc should not be kicking in here, (and also, "random"
hosts are dropped)
* conntrack table is not full
* this issue seem to suddenly appeared, is this a known bug?

-- 
Bj(/)rnar