40k+ outbound connections and bind() problem

40k+ outbound connections and bind() problem

[denys]

 I am running server (TCP accelerator), which is initiating more than 
 40K connections.
 Each instance of server bound to separate IP, and each of them handling 
 around 5-10K connections.
 At moments, when i have excessive connect/disconnect events if i try to 
 establish connection even from IP is not used,
 during bind() i am getting errno "Address already in use".
 What can be the issue?
 Will it help if i increase in kernel hardcoded limit from 64K TCP bind 
 hash entries to higher values?

Re: 40k+ outbound connections and bind() problem

[Daniel Baluta]

Hi,

On Wed, Jan 26, 2011 at 2:14 PM,  <denys@visp.net.lb> wrote:
> I am running server (TCP accelerator), which is initiating more than 40K
> connections.
> Each instance of server bound to separate IP, and each of them handling
> around 5-10K connections.
> At moments, when i have excessive connect/disconnect events if i try to
> establish connection even from IP is not used,
> during bind() i am getting errno "Address already in use".
> What can be the issue?
> Will it help if i increase in kernel hardcoded limit from 64K TCP bind hash
> entries to higher values?

Please provide more information. What kernel version are you using?
Are you  binding on a SO_REUSEADDR socket right?

thanks,
Daniel.

Re: 40k+ outbound connections and bind() problem

[denys]

 On Wed, 26 Jan 2011 14:23:44 +0200, Daniel Baluta wrote:
> Hi,
>
> On Wed, Jan 26, 2011 at 2:14 PM,  <denys@visp.net.lb> wrote:
>> I am running server (TCP accelerator), which is initiating more than 
>> 40K
>> connections.
>> Each instance of server bound to separate IP, and each of them 
>> handling
>> around 5-10K connections.
>> At moments, when i have excessive connect/disconnect events if i try 
>> to
>> establish connection even from IP is not used,
>> during bind() i am getting errno "Address already in use".
>> What can be the issue?
>> Will it help if i increase in kernel hardcoded limit from 64K TCP 
>> bind hash
>> entries to higher values?
>
> Please provide more information. What kernel version are you using?
> Are you  binding on a SO_REUSEADDR socket right?
 Latest stable, 2.6.37

 No, but trying with it now. At same time i think it is not correct, if 
 i try to bind to "unused" for any connections IP, and it will return 
 such error.
 It is not listening socket, it is outbound connection
 s = socket(...)
 memset(&name,0x0,sizeof(name));
 name.sin_family=AF_INET;
 name.sin_addr.s_addr=s_stx->src_ip;
 name.sin_port=0;
 bind()
 connect()

 Now i will try to put between socket and bind
 setsockopt(netsocket,SOL_SOCKET,SO_REUSEADDR,(void*)&tmpint,sizeof(tmpint));

 But IMHO it is not very correct for outgoing connections.

Re: 40k+ outbound connections and bind() problem

[Daniel Baluta]

Hi,

> Latest stable, 2.6.37

Please apply this patch [1] and see if it's still happening.

thanks,
Daniel

[1] http://www.spinics.net/lists/netdev/msg152204.html

Re: 40k+ outbound connections and bind() problem

[Eric Dumazet]

Le mercredi 26 janvier 2011 à 15:00 +0200, Daniel Baluta a écrit :
> Hi,
> 
> > Latest stable, 2.6.37
> 
> Please apply this patch [1] and see if it's still happening.
> 
> thanks,
> Daniel
> 
> [1] http://www.spinics.net/lists/netdev/msg152204.html
> -

I dont think it'll help ;)

Denys problem is if one source IP address is used, there is an absolute
64K limit to (IP, port) tuple

Only thing you can do is to tune /proc/sys/net/ipv4/ip_local_port_range
to get close to 64K limit

Re: 40k+ outbound connections and bind() problem

[denys]

 On Wed, 26 Jan 2011 14:17:14 +0100, Eric Dumazet wrote:
> Le mercredi 26 janvier 2011 à 15:00 +0200, Daniel Baluta a écrit :
>> Hi,
>>
>> > Latest stable, 2.6.37
>>
>> Please apply this patch [1] and see if it's still happening.
>>
>> thanks,
>> Daniel
>>
>> [1] http://www.spinics.net/lists/netdev/msg152204.html
>> -
>
> I dont think it'll help ;)
>
> Denys problem is if one source IP address is used, there is an 
> absolute
> 64K limit to (IP, port) tuple
>
> Only thing you can do is to tune 
> /proc/sys/net/ipv4/ip_local_port_range
> to get close to 64K limit
 No, it is not one ip source.
 As i mention before, i am binding each instance to different IP before 
 connect().
 And when i am having issues and such errors, i am trying httping and 
 binding it to some unused ip, but still getting this error.

Re: 40k+ outbound connections and bind() problem

[Eric Dumazet]

Le mercredi 26 janvier 2011 à 15:23 +0200, denys@visp.net.lb a écrit :
> On Wed, 26 Jan 2011 14:17:14 +0100, Eric Dumazet wrote:
> > Le mercredi 26 janvier 2011 à 15:00 +0200, Daniel Baluta a écrit :
> >> Hi,
> >>
> >> > Latest stable, 2.6.37
> >>
> >> Please apply this patch [1] and see if it's still happening.
> >>
> >> thanks,
> >> Daniel
> >>
> >> [1] http://www.spinics.net/lists/netdev/msg152204.html
> >> -
> >
> > I dont think it'll help ;)
> >
> > Denys problem is if one source IP address is used, there is an 
> > absolute
> > 64K limit to (IP, port) tuple
> >
> > Only thing you can do is to tune 
> > /proc/sys/net/ipv4/ip_local_port_range
> > to get close to 64K limit
>  No, it is not one ip source.
>  As i mention before, i am binding each instance to different IP before 
>  connect().
>  And when i am having issues and such errors, i am trying httping and 
>  binding it to some unused ip, but still getting this error.

Ah OK, you definitely want to use REUSEADDR then.