xfrm by MARK: tcp problems when mark for in and out differ

xfrm by MARK: tcp problems when mark for in and out differ

[Gerd v. Egidy]

Hi,

I use current strongswan git to set up ipsec connections with the xfrm by MARK 
feature. When I configure xfrm policies with different marks for incoming and 
outgoing packets, incoming tcp connections can't be established anymore. The 
SYN-ACK packet is never sent through the tunnel.

An example policy looks like this:

src 192.168.5.0/24 dst 192.168.1.0/24
        dir out priority 1760
        mark 5/0xffffffff
        tmpl src 172.16.1.131 dst 172.16.1.130
                proto esp reqid 16384 mode tunnel
src 192.168.1.0/24 dst 192.168.5.0/24
        dir fwd priority 1760
        tmpl src 172.16.1.130 dst 172.16.1.131
                proto esp reqid 16384 mode tunnel
src 192.168.1.0/24 dst 192.168.5.0/24
        dir in priority 1760
        tmpl src 172.16.1.130 dst 172.16.1.131
                proto esp reqid 16384 mode tunnel

-> incoming packets are without mark, outgoing packets are marked with 5

I traced the packet in the xfrm code and found out that the problem is in the 
flow data. When the SYN-ACK hits __xfrm_lookup, the value in fl->mark is 0 
(more precisely: the mark value used in the incoming packet). This means that 
xfrm_policy_match will not match on the correct policy because the mark values 
differ.

I'm not too familiar with the kernel networking code. But I guess that the 
flow for the SYN-ACK is set up based on the data used for the SYN and is not 
updated when my iptables rule changes the mark of the packet:

iptables -t raw -A OUTPUT -s 192.168.5.0/255.255.255.0 -d 
192.168.1.0/255.255.255.0 -j MARK --set-mark 5

I guess that the flow data should be updated somewhere. But I don't know what 
the correct place for that code would be.

Can somebody more familiar with the network stack help me with this please?

Thank you very much.

Kind regards,

Gerd


-- 
Address (better: trap) for people I really don't want to get mail from:
jonas@cactusamerica.com

Re: xfrm by MARK: tcp problems when mark for in and out differ

[jamal]

On Wed, 2010-10-13 at 15:57 +0200, Gerd v. Egidy wrote:
> Hi,

> 
> -> incoming packets are without mark, outgoing packets are marked with 5
> 

You could use tc ingress path to mark incoming packets. Example:

----
tc qdisc add dev eth0 ingress
tc filter add dev eth0 parent ffff: protocol ip pref 9 u32 \
match ip src 192.168.1.0/32 flowid 1:5 action skbedit mark 5
---

just remember that src on outgoing is dst on incoming..


cheers,
jamal

Re: xfrm by MARK: tcp problems when mark for in and out differ

[Gerd v. Egidy]

Hi Jamal,

> > -> incoming packets are without mark, outgoing packets are marked with 5
> 
> You could use tc ingress path to mark incoming packets. Example:

In my full setup I do exactly that. What I posted was a minimized setup to 
just show the problem.

I have a complex set of netfilter rules and routing tables to allow several 
ways of doing NAT on ipsec. The rules need different marks on packets coming 
from / going to the vpn to correctly distinguish packets in the forwarding 
case.

I did further testing, 2.6.36-rc7 has the problem too.

> When the SYN-ACK hits __xfrm_lookup, the value in fl->mark is 0 
> (more precisely: the mark value used in the incoming packet).

this is wrong, the value in fl->mark is always 0. I must have confused some 
data in my debug printks.

So it seems like the fl->mark is never initialized with the packet mark in the 
first place. What would be the correct stage in the kernel network stack to do 
that?

Kind regards,

Gerd

-- 
Address (better: trap) for people I really don't want to get mail from:
jonas@cactusamerica.com

Re: xfrm by MARK: tcp problems when mark for in and out differ

[jamal]

Gerd,

On Thu, 2010-10-14 at 15:01 +0200, Gerd v. Egidy wrote:

> this is wrong, the value in fl->mark is always 0. I must have confused some 
> data in my debug printks.
>
> So it seems like the fl->mark is never initialized with the packet mark in the 
> first place. What would be the correct stage in the kernel network stack to do 
> that?

Can you try a simple setup without xfrm/ipsec and see if this reverse
path works? Was there a kernel where it worked?

cheers,
jamal

-j MARK in raw vs. mangle (was Re: xfrm by MARK: tcp problems when mark for in and out differ)

[Gerd v. Egidy]

Hi Jamal,

thanks for your help.

> > So it seems like the fl->mark is never initialized with the packet mark
> > in the first place. What would be the correct stage in the kernel
> > network stack to do that?
> 
> Can you try a simple setup without xfrm/ipsec and see if this reverse
> path works? Was there a kernel where it worked?

I just tried opening a simple tcp connection without any xfrm or other weird 
stuff. I just had one iptables rule in place:

-t raw -A OUTPUT -d 192.168.5.200 -j MARK --set-mark 99

192.168.5.200 is the other system I open the tcp connection from. So this 
should mark all response packets to the client.

But the moment __xfrm_lookup is called (this is where my debug printk sits), 
fl->mark is always 0.

By chance I changed the rule over to the mangle table:

-t mangle -A OUTPUT -d 192.168.5.200 -j MARK --set-mark 99

Now it works, the mark in the flow is 99!

So it seems this has nothing to do with xfrm, but that the MARK target has 
different effects when used in raw than in mangle. I was using raw because I 
had to set conntrack zones too and it was more conveniant to do both in one 
place.

Can one of the netfilter guys comment on this? Is using MARK in raw not fully 
supported or has known deficiencies?

Kind regards,

Gerd

-- 
Address (better: trap) for people I really don't want to get mail from:
jonas@cactusamerica.com

Re: xfrm by MARK: tcp problems when mark for in and out differ

[jamal]

Please dont cc subscriber-only mailing lists like
dev@lists.strongswan.org in the future.

cheers,
jamal

-j MARK in raw vs. mangle (was Re: xfrm by MARK: tcp problems when mark for in and out differ)

[Gerd v. Egidy]

Hi Jamal,

thanks for your help.

> > So it seems like the fl->mark is never initialized with the packet mark
> > in the first place. What would be the correct stage in the kernel
> > network stack to do that?
> 
> Can you try a simple setup without xfrm/ipsec and see if this reverse
> path works? Was there a kernel where it worked?

I just tried opening a simple tcp connection without any xfrm or other weird 
stuff. I just had one iptables rule in place:

-t raw -A OUTPUT -d 192.168.5.200 -j MARK --set-mark 99

192.168.5.200 is the other system I open the tcp connection from. So this 
should mark all response packets to the client.

But the moment __xfrm_lookup is called (this is where my debug printk sits), 
fl->mark is always 0.

By chance I changed the rule over to the mangle table:

-t mangle -A OUTPUT -d 192.168.5.200 -j MARK --set-mark 99

Now it works, the mark in the flow is 99!

So it seems this has nothing to do with xfrm, but that the MARK target has 
different effects when used in raw than in mangle. I was using raw because I 
had to set conntrack zones too and it was more conveniant to do both in one 
place.

Can one of the netfilter guys comment on this? Is using MARK in raw not fully 
supported or has known deficiencies?

Kind regards,

Gerd

PS: sorry for the double post, had an old netfilter-devel address in my 
mailer.

-- 
Address (better: trap) for people I really don't want to get mail from:
jonas@cactusamerica.com

Re: -j MARK in raw vs. mangle (was Re: xfrm by MARK: tcp problems when mark for in and out differ)

[Patrick McHardy]

On 14.10.2010 16:16, Gerd v. Egidy wrote:
> Hi Jamal,
> 
> thanks for your help.
> 
>>> So it seems like the fl->mark is never initialized with the packet mark
>>> in the first place. What would be the correct stage in the kernel
>>> network stack to do that?
>>
>> Can you try a simple setup without xfrm/ipsec and see if this reverse
>> path works? Was there a kernel where it worked?
> 
> I just tried opening a simple tcp connection without any xfrm or other weird 
> stuff. I just had one iptables rule in place:
> 
> -t raw -A OUTPUT -d 192.168.5.200 -j MARK --set-mark 99
> 
> 192.168.5.200 is the other system I open the tcp connection from. So this 
> should mark all response packets to the client.
> 
> But the moment __xfrm_lookup is called (this is where my debug printk sits), 
> fl->mark is always 0.
> 
> By chance I changed the rule over to the mangle table:
> 
> -t mangle -A OUTPUT -d 192.168.5.200 -j MARK --set-mark 99
> 
> Now it works, the mark in the flow is 99!
> 
> So it seems this has nothing to do with xfrm, but that the MARK target has 
> different effects when used in raw than in mangle. I was using raw because I 
> had to set conntrack zones too and it was more conveniant to do both in one 
> place.
> 
> Can one of the netfilter guys comment on this? Is using MARK in raw not fully 
> supported or has known deficiencies?

No, the problem is most likely that for outgoing packets, the XFRM
lookup is done with the route lookup before the packet is even sent,
so once it hits the raw or mangle table, it is too late. mangle however
performs rerouting when the mark value changes, at which point a new
XFRM lookup is performed.

Re: -j MARK in raw vs. mangle (was Re: xfrm by MARK: tcp problems when mark for in and out differ)

[Gerd v. Egidy]

Hi Patrick,

> > So it seems this has nothing to do with xfrm, but that the MARK target
> > has different effects when used in raw than in mangle. I was using raw
> > because I had to set conntrack zones too and it was more conveniant to
> > do both in one place.
> > 
> > Can one of the netfilter guys comment on this? Is using MARK in raw not
> > fully supported or has known deficiencies?
> 
> No, the problem is most likely that for outgoing packets, the XFRM
> lookup is done with the route lookup before the packet is even sent,
> so once it hits the raw or mangle table, it is too late. mangle however
> performs rerouting when the mark value changes, at which point a new
> XFRM lookup is performed.

ah, this would explain it. Thanks for the explanation. I'll just stick with 
mangle for marking.

Kind regards,

Gerd

-- 
Address (better: trap) for people I really don't want to get mail from:
jonas@cactusamerica.com