missing conntrack protocol on updates

missing conntrack protocol on updates

[Amin Azez]

I have the recent icmp extension to conntrack built and lsof shows that 
the library is loaded

If I leave conntrack -E running on a test box with a LOT of traffic 
going through it I get quite a few conntrack updates with missing 
protocols, detected with this grep spell:
# conntrack -E conntrack | egrep -v 'tcp|udp|icmp|DESTROY'

They seem to come in groups, and doesn't seem to be related to anything 
particular I can detect except that orig_packets and reply_packets seem 
to be 1, so I wonder if they are icmp responses.

Of course as I am using a custom conntrack kernel module which also 
dumps out the mac addresses the fault could be here, I wondered if you 
would leave that grep running for a while to see if the fault is a 
general one?

I've not been able to get /proc/net/ip_conntrack to have a missing protocol

Azez



[UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2118 
src=192.168.0.233 dst=192.168.0.252 sport=2118 dport=80 timeout=432000 
orig_packets=1 orig_bytes=52 reply_packets=1 reply_bytes=52 
src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
[UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2126 
src=192.168.0.233 dst=192.168.0.252 sport=2126 dport=80 timeout=432000 
orig_packets=1 orig_bytes=52 reply_packets=1 reply_bytes=52 
src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
[UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2128 
src=192.168.0.233 dst=192.168.0.252 sport=2128 dport=80 timeout=432000 
orig_packets=1 orig_bytes=52 reply_packets=1 reply_bytes=52 
src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
[UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2133 
src=192.168.0.233 dst=192.168.0.252 sport=2133 dport=80 timeout=432000 
orig_packets=1 orig_bytes=52 reply_packets=1 reply_bytes=52 
src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
[UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2134 
src=192.168.0.233 dst=192.168.0.252 sport=2134 dport=80 timeout=432000 
orig_packets=1 orig_bytes=52 reply_packets=1 reply_bytes=52 
src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
[UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2135 
src=192.168.0.233 dst=192.168.0.252 sport=2135 dport=80 timeout=432000 
orig_packets=1 orig_bytes=52 reply_packets=1 reply_bytes=52 
src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
[UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2136 
src=192.168.0.233 dst=192.168.0.252 sport=2136 dport=80 timeout=432000 
orig_packets=1 orig_bytes=52 reply_packets=1 reply_bytes=52 
src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
[UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2138 
src=192.168.0.233 dst=192.168.0.252 sport=2138 dport=80 timeout=432000 
orig_packets=1 orig_bytes=52 reply_packets=1 reply_bytes=52 
src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
[UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2139 
src=192.168.0.233 dst=192.168.0.252 sport=2139 dport=80 timeout=432000 
orig_packets=1 orig_bytes=52 reply_packets=1 reply_bytes=52 
src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
[UPDATE] src=192.168.0.1 dst=192.168.0.255 sport=138 dport=138 
src=192.168.0.255 dst=192.168.0.1 sport=138 dport=138 [UNREPLIED] 
orig_packets=2 orig_bytes=491 reply_packets=0 reply_bytes=0 
src_mac=00:40:63:d5:72:50 dst_mac=ff:ff:ff:ff:ff:ff

Re: missing conntrack protocol on updates

[Pablo Neira]

Hi Amin,

Amin Azez wrote:
> Of course as I am using a custom conntrack kernel module which also 
> dumps out the mac addresses the fault could be here, I wondered if you 
> would leave that grep running for a while to see if the fault is a 
> general one?
 >
> [UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2118 
> src=192.168.0.233 dst=192.168.0.252 sport=2118 dport=80 timeout=432000 
> orig_packets=1 orig_bytes=52 reply_packets=1 reply_bytes=52 
> src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
> [UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2128 
> src=192.168.0.233 dst=192.168.0.252 sport=2128 dport=80 timeout=432000 
> orig_packets=1 orig_bytes=52 reply_packets=1 reply_bytes=52 
> src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
> [UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2133 
> src=192.168.0.233 dst=192.168.0.252 sport=2133 dport=80 timeout=432000 
> orig_packets=1 orig_bytes=52 reply_packets=1 reply_bytes=52 
> src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
> [UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2134 
> src=192.168.0.233 dst=192.168.0.252 sport=2134 dport=80 timeout=432000 
> orig_packets=1 orig_bytes=52 reply_packets=1 reply_bytes=52 
> src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a

This seems related to you hack. All those update messages tell me that 
you are sending a netlink event message for every IPCT_PROTINFO_VOLATILE 
event, aren't you? Maybe you're doing something similar, I'd need to see 
the code anyway.

If my guess is correct, such loss of messages is related to the nature 
of the netlink sockets. Netlink is a unreliable protocol. Under *heavy* 
loads and if the messages are sent from interrupt context it will be 
likely to drop messages for spamming events.

--
Pablo

Re: missing conntrack protocol on updates

[Amin Azez]

[sorry for the delay in responding I've been away]

Pablo Neira wrote:
> Hi Amin,
> 
> Amin Azez wrote:
> 
>> Of course as I am using a custom conntrack kernel module which also 
>> dumps out the mac addresses the fault could be here, I wondered if you 
>> would leave that grep running for a while to see if the fault is a 
>> general one?
> 
>> [UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2118 
>> src=192.168.0.233 dst=192.168.0.252 sport=2118 dport=80 timeout=432000 
>> orig_packets=1 orig_bytes=52 reply_packets=1 reply_bytes=52 
>> src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
>> [UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2128 
>> src=192.168.0.233 dst=192.168.0.252 sport=2128 dport=80 timeout=432000 
>> orig_packets=1 orig_bytes=52 reply_packets=1 reply_bytes=52 
>> src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
>> [UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2133 
>> src=192.168.0.233 dst=192.168.0.252 sport=2133 dport=80 timeout=432000 
>> orig_packets=1 orig_bytes=52 reply_packets=1 reply_bytes=52 
>> src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
>> [UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2134 
>> src=192.168.0.233 dst=192.168.0.252 sport=2134 dport=80 timeout=432000 
>> orig_packets=1 orig_bytes=52 reply_packets=1 reply_bytes=52 
>> src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
> 
> 
> This seems related to you hack. All those update messages tell me that 
> you are sending a netlink event message for every IPCT_PROTINFO_VOLATILE 
> event, aren't you? 

As the dport addresses are all different I'm not sure how you conclude this.

> Maybe you're doing something similar, 

I'm not deliberately sending a message for every 
IPCT_PROTOINFO_VOLATILE, although - good guess - I will be wanting to 
get more frequent updates for long lived connections.

> I'd need to see the code anyway.

I'll send patches to conntrack and to kernel 2.6.12 seperately.

> If my guess is correct, such loss of messages is related to the nature 
> of the netlink sockets. Netlink is a unreliable protocol. Under *heavy* 
> loads and if the messages are sent from interrupt context it will be 
> likely to drop messages for spamming events.

Aye, I've been able to do this, but never drop only part of a message, 
usually the entire netlink packet is dropped, so its interesting that 
only the protocol part is missing.

Amin

Re: missing conntrack protocol on updates

[Pablo Neira]

Hi Amin,

Amin Azez wrote:
>> This seems related to you hack. All those update messages tell me that 
>> you are sending a netlink event message for every 
>> IPCT_PROTINFO_VOLATILE event, aren't you? 
> 
> As the dport addresses are all different I'm not sure how you conclude 
> this.

Could you assure you that you're losing messages under heavy loads (tons 
of updates) without your patch?

>> Maybe you're doing something similar, 
> 
> I'm not deliberately sending a message for every 
> IPCT_PROTOINFO_VOLATILE, although - good guess - I will be wanting to 
> get more frequent updates for long lived connections.

OK, I've caught you, you're spamming ;). In case that we'd want more 
frequent updates I'll need to implement a solution based on a buffer as 
Harald currently does in ULOG and do some benchmarks.

>> I'd need to see the code anyway.
> 
> I'll send patches to conntrack and to kernel 2.6.12 seperately.

I've got a major re-sync against ctnetlink and friends (conntrack tool 
included) lying around here. I'll pass it to Harald as soon as I finish 
  the testing. This update implies tons of changes, so you'll have to 
rework your patch. I know that it's a pain in the neck but don't forget 
that this stuff is still under development. There's a snapshot at 
people.netfilter.org.

>> If my guess is correct, such loss of messages is related to the nature 
>> of the netlink sockets. Netlink is a unreliable protocol. Under 
>> *heavy* loads and if the messages are sent from interrupt context it 
>> will be likely to drop messages for spamming events.
> 
> 
> Aye, I've been able to do this, but never drop only part of a message, 
> usually the entire netlink packet is dropped, so its interesting that 
> only the protocol part is missing.

I see the protocol part here, I think that since IPCT_PROTOINFO is only 
set when the protocol state changes, so I bet that this is related to 
you hack.

--
Pablo

Re: missing conntrack protocol on updates

[Amin Azez]

Pablo Neira wrote:

> Hi Amin,
>
> Amin Azez wrote:
>
>>> This seems related to you hack. All those update messages tell me 
>>> that you are sending a netlink event message for every 
>>> IPCT_PROTINFO_VOLATILE event, aren't you? 
>>
>>
>> As the dport addresses are all different I'm not sure how you 
>> conclude this.
>
>
> Could you assure you that you're losing messages under heavy loads 
> (tons of updates) without your patch?

I'm only losing messages when I re-invoke the enumeration of all 
contracks over the event connection AND when I am slow at processing 
these events through talking to other processes.

>
>>> Maybe you're doing something similar, 
>>
>>
>> I'm not deliberately sending a message for every 
>> IPCT_PROTOINFO_VOLATILE, although - good guess - I will be wanting to 
>> get more frequent updates for long lived connections.
>
>
> OK, I've caught you, you're spamming ;). In case that we'd want more 
> frequent updates I'll need to implement a solution based on a buffer 
> as Harald currently does in ULOG and do some benchmarks.

Aye, I did think of a variation of the enumeration where it only dumps 
conntracks that have been in existance more than so-long since their 
last dump or maybe to only dump conntracks that have been in existance 
for more than so-long and clients just invoke this dump periodically.

>
>>> I'd need to see the code anyway.
>>
>>
>> I'll send patches to conntrack and to kernel 2.6.12 seperately.
>
>
> I've got a major re-sync against ctnetlink and friends (conntrack tool 
> included) lying around here. I'll pass it to Harald as soon as I 
> finish  the testing. This update implies tons of changes, so you'll 
> have to rework your patch. I know that it's a pain in the neck but 
> don't forget that this stuff is still under development. There's a 
> snapshot at people.netfilter.org.
>
OK, I'll grab the snapshot thanks, and then send patches along with a 
patch to libnfnetlink with non-blocking read loop on the netlink socket 
and which permits the event handler to signify that the read-loop should 
exit. This allows the application using libnfnetlink to exit the read 
loop based on signals and such, and then re-enter the read loop later. 
I'm also adding in/out device name to /proc/net/ip_conntrack and related 
netlink events.

>>> If my guess is correct, such loss of messages is related to the 
>>> nature of the netlink sockets. Netlink is a unreliable protocol. 
>>> Under *heavy* loads and if the messages are sent from interrupt 
>>> context it will be likely to drop messages for spamming events.
>>
>>
>>
>> Aye, I've been able to do this, but never drop only part of a 
>> message, usually the entire netlink packet is dropped, so its 
>> interesting that only the protocol part is missing.
>
>
> I see the protocol part here, I think that since IPCT_PROTOINFO is 
> only set when the protocol state changes, so I bet that this is 
> related to you hack.
>
I'll look for this now then.

Thanks

Amin

solved Re: missing conntrack protocol on updates

[Amin Azez]

I had mistakenly thought that ctnetlink_fill_info was the only place 
that constructed conntrack netlink packets but now I see that I had 
missed out ctnetlink_conntrack_event which tries to optimize away the 
protocol information if it has not changed but always outputs the tuple.

This seems a mistake because the tuple identifies the conntrack by IP 
and port and protocol, and as there is no medium term conntrack ID 
available the protocol will also be needed to trace the conntrack 
updates as.

Sam

Pablo Neira wrote:

> Hi Amin,
>
> Amin Azez wrote:
>
>> Of course as I am using a custom conntrack kernel module which also 
>> dumps out the mac addresses the fault could be here, I wondered if 
>> you would leave that grep running for a while to see if the fault is 
>> a general one?
>
> >
>
>> [UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2118 
>> src=192.168.0.233 dst=192.168.0.252 sport=2118 dport=80 
>> timeout=432000 orig_packets=1 orig_bytes=52 reply_packets=1 
>> reply_bytes=52 src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
>> [UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2128 
>> src=192.168.0.233 dst=192.168.0.252 sport=2128 dport=80 
>> timeout=432000 orig_packets=1 orig_bytes=52 reply_packets=1 
>> reply_bytes=52 src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
>> [UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2133 
>> src=192.168.0.233 dst=192.168.0.252 sport=2133 dport=80 
>> timeout=432000 orig_packets=1 orig_bytes=52 reply_packets=1 
>> reply_bytes=52 src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
>> [UPDATE] src=192.168.0.252 dst=192.168.0.233 sport=80 dport=2134 
>> src=192.168.0.233 dst=192.168.0.252 sport=2134 dport=80 
>> timeout=432000 orig_packets=1 orig_bytes=52 reply_packets=1 
>> reply_bytes=52 src_mac=00:09:5b:bb:d2:aa dst_mac=00:01:02:12:c6:3a
>
>
> This seems related to you hack. All those update messages tell me that 
> you are sending a netlink event message for every 
> IPCT_PROTINFO_VOLATILE event, aren't you? Maybe you're doing something 
> similar, I'd need to see the code anyway.

Re: solved Re: missing conntrack protocol on updates

[Pablo Neira]

Amin Azez wrote:
> I had mistakenly thought that ctnetlink_fill_info was the only place 
> that constructed conntrack netlink packets but now I see that I had 
> missed out ctnetlink_conntrack_event which tries to optimize away the 
> protocol information if it has not changed but always outputs the tuple.
> 
> This seems a mistake because the tuple identifies the conntrack by IP 
> and port and protocol, and as there is no medium term conntrack ID 
> available the protocol will also be needed to trace the conntrack 
> updates as.

The protocol information you're referring to is contained by 
ip_conntrack_tuple, so it's always dumped since ctnetlink_dump_tuples is 
always called.

Don't get it wrong, ctnetlink_dump_protoinfo dumps the private protocol 
information, ie. in case of a TCP connection, if the it's established, 
closed, etc... and such info isn't using to hash a conntrack. I think 
you're getting confused because of this:

struct cta_proto {
         unsigned char num_proto;        /* Protocol number IPPROTO_X */
         union ip_conntrack_proto proto;
};

See that we always dump the protocol id number together with the private 
protocol information. Otherwise the user space program won't be able to 
handle the information about an update properly.

--
Pablo