From: Leandro Lucarella <leandro.lucarella@sociomantic.com>
To: Rick Jones <rick.jones2@hp.com>
Cc: Eric Dumazet <eric.dumazet@gmail.com>,
netdev@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: Doubts about listen backlog and tcp_max_syn_backlog
Date: Thu, 24 Jan 2013 20:21:26 +0100 [thread overview]
Message-ID: <20130124192125.GD4608@sociomantic.com> (raw)
In-Reply-To: <51018110.1070403@hp.com>
On Thu, Jan 24, 2013 at 10:44:32AM -0800, Rick Jones wrote:
> On 01/24/2013 04:22 AM, Leandro Lucarella wrote:
> >On Wed, Jan 23, 2013 at 11:28:08AM -0800, Rick Jones wrote:
> >>>Then if syncookies are enabled, the time spent in connect() shouldn't be
> >>>bigger than 3 seconds even if SYNs are being "dropped" by listen, right?
> >>
> >>Do you mean if "ESTABLISHED" connections are dropped because the
> >>listen queue is full? I don't think I would put that as "SYNs being
> >>dropped by listen" - too easy to confuse that with an actual
> >>dropping of a SYN segment.
> >
> >I was just kind of quoting the name given by netstat: "SYNs to LISTEN
> >sockets dropped" (for kernel 3.0, I noticed newer kernels don't have
> >this stat anymore, or the name was changed). I still don't know if we
> >are talking about the same thing.
>
> Are you sure those stats are not present in 3.X kernels? I just
> looked at /proc/net/netstat on a 3.7 system and noticed both the
> ListenMumble stats and the three cookie stats. And I see the code
> for them in the tree:
>
> aj@tardy:~/net-next/net/ipv4$ grep MIB_LISTEN *.c
> proc.c: SNMP_MIB_ITEM("ListenOverflows", LINUX_MIB_LISTENOVERFLOWS),
> proc.c: SNMP_MIB_ITEM("ListenDrops", LINUX_MIB_LISTENDROPS),
> tcp_ipv4.c: NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_LISTENOVERFLOWS);
> tcp_ipv4.c: NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_LISTENDROPS);
>
> raj@tardy:~/net-next/net/ipv4$ grep MIB_SYN *.c
> proc.c: SNMP_MIB_ITEM("SyncookiesSent", LINUX_MIB_SYNCOOKIESSENT),
> proc.c: SNMP_MIB_ITEM("SyncookiesRecv", LINUX_MIB_SYNCOOKIESRECV),
> proc.c: SNMP_MIB_ITEM("SyncookiesFailed", LINUX_MIB_SYNCOOKIESFAILED),
> syncookies.c: NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_SYNCOOKIESSENT);
> syncookies.c: NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_SYNCOOKIESFAILED);
> syncookies.c: NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_SYNCOOKIESRECV);
>
> I will sometimes be tripped-up by netstat's not showing a statistic
> with a zero value...
This is what I'm talking about:
pc1 $ uname -a
Linux labs09 3.5.0-18-generic #29~precise1-Ubuntu SMP Mon Oct 22 16:31:46 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
pc1 $ netstat --version | head -n2
net-tools 1.60
netstat 1.42 (2001-04-15)
pc1 $ netstat -s | grep -i syn
4 invalid SYN cookies received
pc2 $ uname -a
Linux eu-21 3.0.0-19-server #33-Ubuntu SMP Thu Apr 19 20:32:48 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
pc2 $ netstat --version | head -n2
net-tools 1.60
netstat 1.42 (2001-04-15)
pc2 $ netstat -s | grep -i syn
1996450 SYN cookies sent
2899079 SYN cookies received
410573 invalid SYN cookies received
10012473 resets received for embryonic SYN_RECV sockets
5659740 SYNs to LISTEN sockets dropped
1 connections reset due to unexpected SYN
I didn't take a look at the kernel or netstat sources about this, so I
don't know exactly how are they connected.
> >>But yes, I would not expect a connect() call to remain incomplete
> >>for any longer than it took to receive an SYN|ACK from the other
> >>end.
> >
> >So the only reason to experience these high times spent in connect()
> >should be because a SYN or SYN|ACK was actually loss in a lower layer,
> >like an error in the network device or a transmission error?
>
> Modulo the/some other drop-without-stat point such as Vijay
> mentioned yesterday.
So, in this cases a syncookie is not sent back? I had the impression
they were sent always...
> You might consider taking some packet traces. If you can I would
> start with a trace taken on the system(s) on which the long
> connect() calls are happening. I think the tcpdump manpage has an
> example of a tcpdump command with a filter expression that catches
> just SYNchronize and FINished segments which I suppose you could
> extend to include ReSeT segments. Such a filter expression would be
> missing the client's ACK of the SYN|ACK but unless you see
> incrementing stats relating to say checksum failures or other drops
> on the "client" side I suppose you could assume that the client
> ACKed the server's SYN|ACK.
Yes, I already did captures and we are definitely loosing packets
(including SYNs), but it looks like the amount of SYNs I'm loosing is
lower than the amount of long connect() times I observe. This is not
confirmed yet, I'm still investigating.
Thanks!
--
Leandro Lucarella
sociomantic labs GmbH
http://www.sociomantic.com
next prev parent reply other threads:[~2013-01-24 19:21 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-01-22 16:10 Doubts about listen backlog and tcp_max_syn_backlog Leandro Lucarella
2013-01-22 16:45 ` Eric Dumazet
2013-01-22 16:59 ` Leandro Lucarella
2013-01-22 17:13 ` Eric Dumazet
2013-01-22 18:17 ` Rick Jones
2013-01-22 18:42 ` Leandro Lucarella
2013-01-22 22:01 ` Rick Jones
2013-01-23 10:47 ` Leandro Lucarella
2013-01-23 19:28 ` Rick Jones
2013-01-24 12:22 ` Leandro Lucarella
2013-01-24 18:44 ` Rick Jones
2013-01-24 19:21 ` Leandro Lucarella [this message]
2013-01-25 6:12 ` Nivedita SInghvi
2013-01-25 10:05 ` Leandro Lucarella
2013-01-28 2:48 ` Nivedita Singhvi
2013-01-28 5:21 ` Vijay Subramanian
2013-01-28 14:40 ` Leandro Lucarella
2013-01-28 13:08 ` Leandro Lucarella
2013-01-28 2:49 ` Nivedita Singhvi
2013-01-23 20:48 ` Vijay Subramanian
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130124192125.GD4608@sociomantic.com \
--to=leandro.lucarella@sociomantic.com \
--cc=eric.dumazet@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=rick.jones2@hp.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).