UDP packets sent with wrong source address after routing change [AV#3431]

UDP packets sent with wrong source address after routing change [AV#3431]

[Chris Wilson]

Dear sirs,

I think I've found a bug or limitation with the way conntrack works. It's 
been discovered several times:

* http://forum.voxilla.com/threads/pap2t-registers-only-on-manual-port-change.33252/
* http://tomatousb.org/forum/t-272891
* http://community.plus.net/forum/index.php?topic=61369.0;wap2
* http://www.spinics.net/lists/netfilter/msg53056.html
* https://www.google.co.uk/search?q=udp%20conntrack%20source%20address%20changes
* http://www.spinics.net/lists/netfilter/msg53056.html

Summary: UDP devices (e.g. SIP phones) that don't change their source 
port, behind NAT, when routing to their destination changes, keep a 
conntrack entry valid which contains the old (wrong) SNAT-to address and 
reply address, so they are sent with the wrong IP and replies would not be 
forwarded either.

(I apologise if this is already known and fixed; if so I would appreciate
a pointer to the fix or documentation).

The details:

This only seems to happen when you're using a UDP device behind a Linux
NAT router, and your routing to the destination host changes, because:

* You bring up a VPN tunnel and the SIP destination is at the other end of
that tunnel; or

* Your default route changes because you failover to another provider.

We observe packets from the UDP device leaving the router on the new
destination interface, but with the old source address, which is not
appropriate for that interface, and is thus discarded.

Ideally the UDP device would notice that its connection is down and try to
reconnect. If the source port changes (as it does with TCP), this works
around the problem, as a new conntrack entry is created and the old one
eventually discarded. However some UDP devices use a fixed port,
particularly ones that expect incoming connections, such as SIP devices.

One could argue that this UDP device behaviour is broken, but many devices 
do not have the option to use a random or changing source port (e.g. the 
popular Linksys PAP2T) and I think this case could and should be handled 
by iptables.

The logical option seems to be that if the packet matches an existing
conntrack entry, but is going to be sent with a different source address,
then it should not actually match that entry. So the tuple for lookup in
the conntrack table should include the source address and port after SNAT.
However I suspect this is not easily generalisable to other NAT types, and
violates layering (the source address and port after SNAT may not be
available at the time of the lookup) so I guess there may be a more
acceptable solution.

Another option which doesn't violate layering might be to update the 
NAT rule when the outgoing address is known (after routing), if it's 
different to the current outgoing address. This would break existing TCP 
connections if the routing changes, but in most cases the connection is 
already broken by the routing change, and what we're trying to fix is 
that future connection attempts, reusing the same conntrack, should not 
fail as well.

If conntrack-tools worked on centos 5 then I could use it to remove the
stale conntrack entry after the upstream connection fails over, but this
is hacky and specific to my use case, so people will continue to
run into this problem with commodity routers running Linux.

Another option might be to remember the destination interface as part of
the conntrack entry, and continue to send matching packets out of that
interface even if the routing changes. This could probably be hacked
together with fwmark for a fixed set of interfaces. It also doesn't
handle the case where there's a good reason for the failover, e.g. the
original interface has failed or is now down for some reason.

Please let me know if you'd like some more details or there's anything
else I can do to help. It should be fairly easy to create a test case now.

Cheers, Chris.
-- 
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Future Business, Cam City FC, Milton Rd, Cambridge, CB4 1UY, UK

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Jan Engelhardt]

On Thursday 2012-11-08 17:35, Chris Wilson wrote:
>
> Summary: UDP devices (e.g. SIP phones) that don't change their source port,
> behind NAT, when routing to their destination changes, keep a conntrack entry
> valid which contains the old (wrong) SNAT-to address

Actually, not {changing the tuple and thus breaking at least
stream-type connections} could be considered a feature, for example
when you migrate off a certain address and onto a new one.

This(^) thought applies to both SNAT and DNAT scenarios. In fact,
IPv6 (non-NAT) temporary addresses follow the same principle — they
continue to be valid for a certain time even after their
preferred_lifetime has run out.

You can remove "old" NFCT entries by use of conntrack(8) if you
the time between preferred_lft and valid_lft running out is too
short.

> However I suspect this is not easily generalisable to other NAT types, and
> violates layering (the source address and port after SNAT may not be
> available at the time of the lookup) so I guess there may be a more
> acceptable solution.
>
> Another option which doesn't violate layering might be to update the NAT rule
> when the outgoing address is known (after routing),

That is what MASQUERADE is usually for.

> if it's different to the
> current outgoing address. This would break existing TCP connections if the
> routing changes, but in most cases the connection is already broken by the
> routing change, and what we're trying to fix is that future connection
> attempts, reusing the same conntrack, should not fail as well.
>
> Another option might be to remember the destination interface as part of
> the conntrack entry, and continue to send matching packets out of that
> interface even if the routing changes. [...]

Consider this 2×2 event matrix (giving 4 cases a packet can end up
in):

  route out on {A, B} × use outgoing address {A, B}

It is certainly possible that a customer may be multi-linked (2 or
more ISPs), and each ISP accepting traffic from/to his multi-home (1
or more subnets), whilst he is doing round-robin loadbalancing.

An operating system can therefore never automatically know whether
it is ok to continue using certain tuples over a link or not, unless
the user intervenes in some way.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Chris Wilson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3848 bytes --]

Hi Jan,

On Thu, 8 Nov 2012, Jan Engelhardt wrote:
> On Thursday 2012-11-08 17:35, Chris Wilson wrote:
>>
>> Summary: UDP devices (e.g. SIP phones) that don't change their source port,
>> behind NAT, when routing to their destination changes, keep a conntrack entry
>> valid which contains the old (wrong) SNAT-to address
>
> Actually, not {changing the tuple and thus breaking at least
> stream-type connections} could be considered a feature, for example
> when you migrate off a certain address and onto a new one.

OK, I can see that, but in this case it results in pretty broken 
behaviour, which users (other than me) are encountering in practice.

> You can remove "old" NFCT entries by use of conntrack(8) if you the time 
> between preferred_lft and valid_lft running out is too short.

Unfortunately I don't think conntrack works on the CentOS 5 kernel 
(2.6.18). Perhaps it's just too old, but it's widely used, and other 2.6.x 
kernels are widely used on routers too.

I know that fixing/changing this behaviour would also require a kernel 
upgrade, but I'd prefer that at some point as users are upgraded to newer 
kernels, this bug stops affecting them.

I'd also prefer a more automatic solution than manually removing entries 
from the conntrack table.

>> Another option which doesn't violate layering might be to update the 
>> NAT rule when the outgoing address is known (after routing),
>
> That is what MASQUERADE is usually for.

Unfortunately I am using MASQUERADE and this still happens. If it could 
just be fixed in the MASQUERADE target that would be a big win.

>> Another option might be to remember the destination interface as part 
>> of the conntrack entry, and continue to send matching packets out of 
>> that interface even if the routing changes. [...]
>
> Consider this 2×2 event matrix (giving 4 cases a packet can end up
> in):
>
>  route out on {A, B} × use outgoing address {A, B}
>
> It is certainly possible that a customer may be multi-linked (2 or
> more ISPs), and each ISP accepting traffic from/to his multi-home (1
> or more subnets), whilst he is doing round-robin loadbalancing.

It is possible, especially in the round-robin case, but that is probably 
less often used than the simple failover case, because so few ISPs will 
allow you to do this, for spoofing prevention reasons. Also it's not much 
of a failover system if the backup route uses the same ISP as the main 
route.

Considing both cases where users with failover and load balancing, and 
considering whether they use use the same ISP for both connections AND the 
ISP supports spoofing the source address, or not, I'd guess that the first 
case (same ISP and spoofing works) is less likely than the second, but I 
don't have evidence to back it up.

> An operating system can therefore never automatically know whether
> it is ok to continue using certain tuples over a link or not, unless
> the user intervenes in some way.

I think it would rarely be acceptable to send a packet out of the wrong 
interface that doesn't match the source address (AB and BA in your 
matrix). Perhaps in that case we could require an extra flag to 
MASQUERADE?

Even a default-off MASQUERADE flag that says "--update-source-address" 
would give us the option to have this fixed automatically, provided that 
people know about the flag and use it.

The only alternative currently (when the device uses fixed source ports 
and the router doesn't support conntrack(8)) is to poll the device's web 
interface, check whether it's offline, and if so manually tweak its source 
port. That seems more hacky.

Cheers, Chris.
-- 
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Future Business, Cam City FC, Milton Rd, Cambridge, CB4 1UY, UK

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Jan Engelhardt]

On Thursday 2012-11-08 19:37, Chris Wilson wrote:
>
>> You can remove "old" NFCT entries by use of conntrack(8) if you the time
>> between preferred_lft and valid_lft running out is too short.
>
> Unfortunately I don't think conntrack works on the CentOS 5 kernel (2.6.18).
> Perhaps it's just too old, but it's widely used, and other 2.6.x kernels are
> widely used on routers too.

RHEL and its derivates do not really care about NF software.
Wide use is not going to fix that. (And that's why Windows
is considered an even greater mess ;)

>> An operating system can therefore never automatically know whether
>> it is ok to continue using certain tuples over a link or not, unless
>> the user intervenes in some way.
>
> I think it would rarely be acceptable to send a packet out of the wrong
> interface that doesn't match the source address (AB and BA in your matrix).

It is a standard practice that {the default gateway of one's
equipment} [=the first node on ISP's territory] and your own
subnet make a disjoint set.

$ ip r
2a01:4f8:161:3a0::1 dev eth0  metric 1024
2a01:4f8:161:fff::/64 dev eth1  proto kernel  metric 256 
default via 2a01:4f8:161:3a0::1 dev eth0  metric 1024 
$ ip -6 a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    inet6 2a01:4f8:161:fff::1/128 scope global 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 500
    inet6 2a01:4f8:161:fff::1/64 scope global 
       valid_lft forever preferred_lft forever

(This similarly works for IPv4.)


> Even a default-off MASQUERADE flag that says "--update-source-address" would
> give us the option to have this fixed automatically, provided that people know
> about the flag and use it.

I'll leave this to the nf_nat maintainer(s) and what direction it
should go.

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Chris Wilson]

Hi all,

On Thu, 8 Nov 2012, Jan Engelhardt wrote:

>>> An operating system can therefore never automatically know whether it 
>>> is ok to continue using certain tuples over a link or not, unless the 
>>> user intervenes in some way.
...
>> Even a default-off MASQUERADE flag that says "--update-source-address" would
>> give us the option to have this fixed automatically, provided that people know
>> about the flag and use it.
>
> I'll leave this to the nf_nat maintainer(s) and what direction it
> should go.

While thinking about this last night, I realised that if this is actually 
a bug or limitation in MASQUERADE, then anyone using a Linux router on 
dynamic IP (frequently changing) with one of these UDP devices behind it 
is almost certainly experiencing the same problem:

After the public IP changes (which might be every 8-24 hours in some 
cases), outbound packets from the device keep the old conntrack entry 
alive forever, with the wrong public IP address, and no replies will ever 
be received to those packets. Surely this is a more common scenario than a 
Linux box managing multihoming or failover connections.

I hope that either we will make the MASQUERADE target update the source 
(SNAT) address address automatically by default, or that all cheap router 
vendors will add rules like this:

iptables -t nat -A POSTROUTING -p udp -j MASQUERADE --update-source-address
iptables -t nat -A POSTROUTING -j MASQUERADE

Cheers, Chris.
-- 
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Future Business, Cam City FC, Milton Rd, Cambridge, CB4 1UY, UK

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Pablo Neira Ayuso]

On Thu, Nov 08, 2012 at 06:37:24PM +0000, Chris Wilson wrote:
[...]
> >>Another option which doesn't violate layering might be to update
> >>the NAT rule when the outgoing address is known (after routing),
> >
> >That is what MASQUERADE is usually for.
> 
> Unfortunately I am using MASQUERADE and this still happens. If it
> could just be fixed in the MASQUERADE target that would be a big
> win.

MASQUERADE already cleans up the entries in the conntrack table once
you get your device down, that code is still there in 2.6.18:

http://lxr.linux.no/#linux+v2.6.18/net/ipv4/netfilter/ipt_MASQUERADE.c#L111

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Jan Engelhardt]

On Saturday 2012-11-10 15:07, Pablo Neira Ayuso wrote:

>On Thu, Nov 08, 2012 at 06:37:24PM +0000, Chris Wilson wrote:
>[...]
>> >>Another option which doesn't violate layering might be to update
>> >>the NAT rule when the outgoing address is known (after routing),
>> >
>> >That is what MASQUERADE is usually for.
>> 
>> Unfortunately I am using MASQUERADE and this still happens. If it
>> could just be fixed in the MASQUERADE target that would be a big
>> win.
>
>MASQUERADE already cleans up the entries in the conntrack table once
>you get your device down, that code is still there in 2.6.18:
>
>http://lxr.linux.no/#linux+v2.6.18/net/ipv4/netfilter/ipt_MASQUERADE.c#L111

It looks like it only catches the case of a changing ifindex.

That may work for PPP links, but if you access the 'net over an
Ethernet-looking link as is (thankfully) done by cable ISPs,
the interface will never go away.

The DHCP client will simply change one or more addresses on it -- and
at the same time I wonder if that makes it a "you should run
conntrack -F or something from your dhclient.script" case.

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Jozsef Kadlecsik]

On Sat, 10 Nov 2012, Jan Engelhardt wrote:

> On Saturday 2012-11-10 15:07, Pablo Neira Ayuso wrote:
> 
> >On Thu, Nov 08, 2012 at 06:37:24PM +0000, Chris Wilson wrote:
> >[...]
> >> >>Another option which doesn't violate layering might be to update
> >> >>the NAT rule when the outgoing address is known (after routing),
> >> >
> >> >That is what MASQUERADE is usually for.
> >> 
> >> Unfortunately I am using MASQUERADE and this still happens. If it
> >> could just be fixed in the MASQUERADE target that would be a big
> >> win.
> >
> >MASQUERADE already cleans up the entries in the conntrack table once
> >you get your device down, that code is still there in 2.6.18:
> >
> >http://lxr.linux.no/#linux+v2.6.18/net/ipv4/netfilter/ipt_MASQUERADE.c#L111
> 
> It looks like it only catches the case of a changing ifindex.
> 
> That may work for PPP links, but if you access the 'net over an
> Ethernet-looking link as is (thankfully) done by cable ISPs,
> the interface will never go away.
> 
> The DHCP client will simply change one or more addresses on it -- and
> at the same time I wonder if that makes it a "you should run
> conntrack -F or something from your dhclient.script" case.

That's also handled by the MASQUERADE target: if the device was downed or 
its address deleted, the corresponding conntrack entries are cleaned up.

But the thread is drifted from the original cases:

> This only seems to happen when you're using a UDP device behind a Linux
> NAT router, and your routing to the destination host changes, because:
>
> * You bring up a VPN tunnel and the SIP destination is at the other end 
> of that tunnel; or

This case is not handled by MASQUERADE.

> * Your default route changes because you failover to another provider.

And this case is also not handled if the original default route device is 
still healthy, that is for example when you use some dynamic routing 
protocol which detects failure in the routing path.

I do not see any simple solution except to delete any NATed entry 
unconditionally when the routing changes. But that can easily be too much 
and may kill valid entries.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Pablo Neira Ayuso]

On Sat, Nov 10, 2012 at 10:47:55PM +0100, Jozsef Kadlecsik wrote:
> On Sat, 10 Nov 2012, Jan Engelhardt wrote:
> 
> > On Saturday 2012-11-10 15:07, Pablo Neira Ayuso wrote:
> > 
> > >On Thu, Nov 08, 2012 at 06:37:24PM +0000, Chris Wilson wrote:
> > >[...]
> > >> >>Another option which doesn't violate layering might be to update
> > >> >>the NAT rule when the outgoing address is known (after routing),
> > >> >
> > >> >That is what MASQUERADE is usually for.
> > >> 
> > >> Unfortunately I am using MASQUERADE and this still happens. If it
> > >> could just be fixed in the MASQUERADE target that would be a big
> > >> win.
> > >
> > >MASQUERADE already cleans up the entries in the conntrack table once
> > >you get your device down, that code is still there in 2.6.18:
> > >
> > >http://lxr.linux.no/#linux+v2.6.18/net/ipv4/netfilter/ipt_MASQUERADE.c#L111
> > 
> > It looks like it only catches the case of a changing ifindex.
> > 
> > That may work for PPP links, but if you access the 'net over an
> > Ethernet-looking link as is (thankfully) done by cable ISPs,
> > the interface will never go away.
> > 
> > The DHCP client will simply change one or more addresses on it -- and
> > at the same time I wonder if that makes it a "you should run
> > conntrack -F or something from your dhclient.script" case.
> 
> That's also handled by the MASQUERADE target: if the device was downed or 
> its address deleted, the corresponding conntrack entries are cleaned up.
> 
> But the thread is drifted from the original cases:
> 
> > This only seems to happen when you're using a UDP device behind a Linux
> > NAT router, and your routing to the destination host changes, because:
> >
> > * You bring up a VPN tunnel and the SIP destination is at the other end 
> > of that tunnel; or
> 
> This case is not handled by MASQUERADE.
> 
> > * Your default route changes because you failover to another provider.
> 
> And this case is also not handled if the original default route device is 
> still healthy, that is for example when you use some dynamic routing 
> protocol which detects failure in the routing path.
> 
> I do not see any simple solution except to delete any NATed entry 
> unconditionally when the routing changes. But that can easily be too much 
> and may kill valid entries.

I do not see it either and flushing the entire table seems to me like
way too much.

My suggestion is to use conntrack -D ... including the appropriate
selectors to delete only affected entries, and those selectors depend
on the setup.

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Chris Wilson]

Hi Joszef,

On Sat, 10 Nov 2012, Jozsef Kadlecsik wrote:

> But the thread is drifted from the original cases:
>
>> This only seems to happen when you're using a UDP device behind a Linux
>> NAT router, and your routing to the destination host changes, because:
>>
>> * You bring up a VPN tunnel and the SIP destination is at the other end
>> of that tunnel; or
>
> This case is not handled by MASQUERADE.
>
>> * Your default route changes because you failover to another provider.
>
> And this case is also not handled if the original default route device is
> still healthy, that is for example when you use some dynamic routing
> protocol which detects failure in the routing path.

Not necessarily dynamic routing. Even using pppd (pppoa/pppoe) for one 
connection and ethernet for the other can cause this problem. Any case 
that changes routing without bringing down the old device can be affected.

I'm glad you agree that these are actual problems or limitations with 
MASQUERADE.

> I do not see any simple solution except to delete any NATed entry
> unconditionally when the routing changes. But that can easily be too much
> and may kill valid entries.

But I'm sorry that you won't consider having a flag that changes 
MASQUERADE's behaviour to automatically change the source address in the 
conntrack entry.

Is that because you think it's impossible or infeasible to get access to 
the right information at the right time? Or some other reason?

Cheers, Chris.
-- 
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Future Business, Cam City FC, Milton Rd, Cambridge, CB4 1UY, UK

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Jozsef Kadlecsik]

On Mon, 12 Nov 2012, Chris Wilson wrote:

> On Sat, 10 Nov 2012, Jozsef Kadlecsik wrote:
> 
> > But the thread is drifted from the original cases:
> > 
> > > This only seems to happen when you're using a UDP device behind a Linux
> > > NAT router, and your routing to the destination host changes, because:
> > > 
> > > * You bring up a VPN tunnel and the SIP destination is at the other end
> > > of that tunnel; or
> > 
> > This case is not handled by MASQUERADE.
> > 
> > > * Your default route changes because you failover to another provider.
> > 
> > And this case is also not handled if the original default route device is
> > still healthy, that is for example when you use some dynamic routing
> > protocol which detects failure in the routing path.
> 
> Not necessarily dynamic routing. Even using pppd (pppoa/pppoe) for one
> connection and ethernet for the other can cause this problem. Any case that
> changes routing without bringing down the old device can be affected.
> 
> I'm glad you agree that these are actual problems or limitations with
> MASQUERADE.
>
> > I do not see any simple solution except to delete any NATed entry
> > unconditionally when the routing changes. But that can easily be too much
> > and may kill valid entries.
> 
> But I'm sorry that you won't consider having a flag that changes 
> MASQUERADE's behaviour to automatically change the source address in the 
> conntrack entry.

I don't think changing the source address were a good solution: if 
MASQUERADE could be changed to handle the cases then the wrong conntrack 
entries should be deleted.

But what would trigger the action? (Routing changed? I do not see such a 
kernel event but I might missed it.) And more importantly, what would 
identify the affected entries?

I do not see any good answer to those questions.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Chris Wilson]

Hi Jozsef,

On Mon, 12 Nov 2012, Jozsef Kadlecsik wrote:

>>> I do not see any simple solution except to delete any NATed entry
>>> unconditionally when the routing changes. But that can easily be too much
>>> and may kill valid entries.
>>
>> But I'm sorry that you won't consider having a flag that changes
>> MASQUERADE's behaviour to automatically change the source address in the
>> conntrack entry.
>
> I don't think changing the source address were a good solution: if
> MASQUERADE could be changed to handle the cases then the wrong conntrack
> entries should be deleted.
>
> But what would trigger the action? (Routing changed? I do not see such a
> kernel event but I might missed it.) And more importantly, what would
> identify the affected entries?
>
> I do not see any good answer to those questions.

I propose that:

* when the packet matches an existing conntrack rule, and

* is sent out of an interface that does not list the packet's new (SNAT-to) 
source address as one of its IP addresses (i.e. if this were a new 
connection, MASQUERADE would not choose this source address), and

* the --update-source-address flag is set on the MASQUERADE target

then update the source address on the conntrack rule to the new one.

That's the same thing that would happen if we deleted the conntrack entry 
first: MASQUERADE would choose a new source address and save it in the new 
conntrack entry.

Cheers, Chris.
-- 
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Future Business, Cam City FC, Milton Rd, Cambridge, CB4 1UY, UK

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Jozsef Kadlecsik]

On Mon, 12 Nov 2012, Chris Wilson wrote:

> On Mon, 12 Nov 2012, Jozsef Kadlecsik wrote:
> 
> > > > I do not see any simple solution except to delete any NATed entry
> > > > unconditionally when the routing changes. But that can easily be too
> > > > much
> > > > and may kill valid entries.
> > > 
> > > But I'm sorry that you won't consider having a flag that changes
> > > MASQUERADE's behaviour to automatically change the source address in the
> > > conntrack entry.
> > 
> > I don't think changing the source address were a good solution: if
> > MASQUERADE could be changed to handle the cases then the wrong conntrack
> > entries should be deleted.
> > 
> > But what would trigger the action? (Routing changed? I do not see such a
> > kernel event but I might missed it.) And more importantly, what would
> > identify the affected entries?
> > 
> > I do not see any good answer to those questions.
> 
> I propose that:
> 
> * when the packet matches an existing conntrack rule, and
> 
> * is sent out of an interface that does not list the packet's new (SNAT-to)
> source address as one of its IP addresses (i.e. if this were a new
> connection, MASQUERADE would not choose this source address), and

First, it'd mean a heavy checking for every matched packet.

Second, there's no callback, whatsoever in the netfilter conntrack 
framework which could execute this instruction set. And if such an 
internal hook were added, that'd mean an unnecessary "check the hook and 
call if exists" overhead for *every* other case.

An in-kernel "route changed" notification were great, because then we 
could delete all MASQUERAD-ed entries where --update-source-address flag 
is set.
 
> * the --update-source-address flag is set on the MASQUERADE target
> 
> then update the source address on the conntrack rule to the new one.

Why do you insist on updating the source address? Should the 
--update-source-address be limited to UDP only?

> That's the same thing that would happen if we deleted the conntrack entry
> first: MASQUERADE would choose a new source address and save it in the new
> conntrack entry.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Chris Wilson]

Hi Jozsef,

On Mon, 12 Nov 2012, Jozsef Kadlecsik wrote:

>>>> But I'm sorry that you won't consider having a flag that changes 
>>>> MASQUERADE's behaviour to automatically change the source address in 
>>>> the conntrack entry.
>>>
>>> I don't think changing the source address were a good solution: if 
>>> MASQUERADE could be changed to handle the cases then the wrong 
>>> conntrack entries should be deleted.

I'm not quite sure what you mean here. Are you saying that "invalid 
conntrack entries should be detected and deleted" or "changing MASQUERADE 
is pointless because if it would just delete conntrack entries that should 
not be deleted"?

>>> But what would trigger the action? (Routing changed? I do not see such 
>>> a kernel event but I might missed it.) And more importantly, what 
>>> would identify the affected entries?
[...]
> An in-kernel "route changed" notification were great, because then we 
> could delete all MASQUERAD-ed entries where --update-source-address flag 
> is set.

If we could register to receive routing changed events for SNAT source 
addresses, that would be great (more efficient) but if the mechanism does 
not yet exist, it would be much more difficult to create.

I suppose that if we just deleted them all on any routing change, then 
that would solve the immediate problem for us. It would be ideal if 
incoming packets could still be directed to the correct internal source, 
but dropping them is a smaller problem than continuing to send out invalid 
packets and keeping a broken entry alive forever.

>> I propose that:
>>
>> * when the packet matches an existing conntrack rule, and
>>
>> * is sent out of an interface that does not list the packet's new (SNAT-to)
>> source address as one of its IP addresses (i.e. if this were a new
>> connection, MASQUERADE would not choose this source address), and
>
> First, it'd mean a heavy checking for every matched packet.

Is it really so expensive to get the list of IP addresses for an interface 
and check down the list for presence of a given address? If so, perhaps 
that's a problem elsewhere in the kernel?

> Second, there's no callback, whatsoever in the netfilter conntrack
> framework which could execute this instruction set. And if such an
> internal hook were added, that'd mean an unnecessary "check the hook and
> call if exists" overhead for *every* other case.

Another option is to check, after routing, the source address that 
MASQUERADE would have assigned to the packet. There must be a mechanism 
for that, because we already do it for the first packet in the connection, 
and it surely cannot be expensive. If it's different to the current 
SNAT-to address in the conntrack, and the flag is set on the conntrack, 
then update the conntrack.

>> * the --update-source-address flag is set on the MASQUERADE target
>>
>> then update the source address on the conntrack rule to the new one.
>
> Why do you insist on updating the source address?

In these cases, the original source address is no longer valid. If we 
delete the conntrack, we will just end up creating a new one for the same 
packet. That results in more overhead and higher churn in the conntrack 
table, but admittedly since the source address has changed, we should 
probably treat the conntrack as UNREPLIED anyway, i.e. more like a new 
fresh conntrack.

Perhaps if we were to leave the old conntrack alone and just create a new 
one, then packets arriving from the peer destined to the old source 
address would still end up being forwarded to the local device, which 
could help to keep any existing connections kind-of alive (that might be 
seen as a bad thing as well).

> Should the --update-source-address be limited to UDP only?

I think the problem applies even more to portless protocols such as GRE. 
Devices have no choice to "change the source port" when making a new 
connection. Any packets that they send, will definitely keep a stale entry 
alive.

Thank you for considering and engaging with this proposal :)

Cheers, Chris.
-- 
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Future Business, Cam City FC, Milton Rd, Cambridge, CB4 1UY, UK

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Jozsef Kadlecsik]

Hi Chris,

On Mon, 12 Nov 2012, Chris Wilson wrote:

> On Mon, 12 Nov 2012, Jozsef Kadlecsik wrote:
> 
> > > > > But I'm sorry that you won't consider having a flag that changes
> > > > > MASQUERADE's behaviour to automatically change the source address in
> > > > > the conntrack entry.
> > > > 
> > > > I don't think changing the source address were a good solution: if
> > > > MASQUERADE could be changed to handle the cases then the wrong
> > > > conntrack entries should be deleted.
> 
> I'm not quite sure what you mean here. Are you saying that "invalid conntrack
> entries should be detected and deleted" or "changing MASQUERADE is pointless
> because if it would just delete conntrack entries that should not be
> deleted"?

What I meant was that instead of changing the IP address in the conntrack 
entry, it should simply be deleted and re-created.

> > > > But what would trigger the action? (Routing changed? I do not see such
> > > > a kernel event but I might missed it.) And more importantly, what would
> > > > identify the affected entries?
> [...]
> > An in-kernel "route changed" notification were great, because then we could
> > delete all MASQUERAD-ed entries where --update-source-address flag is set.
> 
> If we could register to receive routing changed events for SNAT source 
> addresses, that would be great (more efficient) but if the mechanism 
> does not yet exist, it would be much more difficult to create.

There's definitely no such an interface.
 
> I suppose that if we just deleted them all on any routing change, then that
> would solve the immediate problem for us. It would be ideal if incoming
> packets could still be directed to the correct internal source, but dropping
> them is a smaller problem than continuing to send out invalid packets and
> keeping a broken entry alive forever.
> 
> > > I propose that:
> > > 
> > > * when the packet matches an existing conntrack rule, and
> > > 
> > > * is sent out of an interface that does not list the packet's new
> > > (SNAT-to)
> > > source address as one of its IP addresses (i.e. if this were a new
> > > connection, MASQUERADE would not choose this source address), and
> > 
> > First, it'd mean a heavy checking for every matched packet.
> 
> Is it really so expensive to get the list of IP addresses for an interface
> and check down the list for presence of a given address? 

That'd mean that for *every* packet first the next routing hop should be 
calculated in netfilter too, then an IP address of the interface 
selected and then the old and the calculated addresses compared. For every 
packet, unconditionally, that's simply too much overhead.

> If so, perhaps that's a problem elsewhere in the kernel?

No, it's netfilter specific.
 
> > Second, there's no callback, whatsoever in the netfilter conntrack
> > framework which could execute this instruction set. And if such an
> > internal hook were added, that'd mean an unnecessary "check the hook and
> > call if exists" overhead for *every* other case.
> 
> Another option is to check, after routing, the source address that 
> MASQUERADE would have assigned to the packet. There must be a mechanism 
> for that, because we already do it for the first packet in the 
> connection, and it surely cannot be expensive. If it's different to the 
> current SNAT-to address in the conntrack, and the flag is set on the 
> conntrack, then update the conntrack.

The first packets are special: the (NAT) rules are evaluated for the very 
first packets only. Subsequent ones are looked up in the conntrack table 
and the nat table is completely skipped. As I wrote, there's no 
callback/hook by which such a functionality for MASQUERADE could be 
called. Also, SNAT/MASQUERADE is the very last action in netfilter.
 
> > > * the --update-source-address flag is set on the MASQUERADE target
> > > 
> > > then update the source address on the conntrack rule to the new one.
> > 
> > Why do you insist on updating the source address?
> 
> In these cases, the original source address is no longer valid. If we 
> delete the conntrack, we will just end up creating a new one for the 
> same packet. That results in more overhead and higher churn in the 
> conntrack table, but admittedly since the source address has changed, we 
> should probably treat the conntrack as UNREPLIED anyway, i.e. more like 
> a new fresh conntrack.

My concern is that there can for example be conntrack extensions attached 
to the conntrack entry, which may became broken if the IP address was
simply changed.
 
> Perhaps if we were to leave the old conntrack alone and just create a new
> one, then packets arriving from the peer destined to the old source address
> would still end up being forwarded to the local device, which could help to
> keep any existing connections kind-of alive (that might be seen as a bad
> thing as well).
> 
> > Should the --update-source-address be limited to UDP only?
> 
> I think the problem applies even more to portless protocols such as GRE.
> Devices have no choice to "change the source port" when making a new
> connection. Any packets that they send, will definitely keep a stale entry
> alive.
> 
> Thank you for considering and engaging with this proposal :)

It's an issue, which should be solved in MASQUERADE itself. The conntrack 
tool can be used to solve it, in theory. In practice it's not easy at all: 
a daemon should be run on the machine which catches the routing changes 
and then calls "conntrack".

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Jozsef Kadlecsik]

On Mon, 12 Nov 2012, Chris Wilson wrote:

> I propose that:
> 
> * when the packet matches an existing conntrack rule, and
> 
> * is sent out of an interface that does not list the packet's new (SNAT-to)
> source address as one of its IP addresses (i.e. if this were a new
> connection, MASQUERADE would not choose this source address), and
> 
> * the --update-source-address flag is set on the MASQUERADE target
> 
> then update the source address on the conntrack rule to the new one.
> 
> That's the same thing that would happen if we deleted the conntrack entry
> first: MASQUERADE would choose a new source address and save it in the new
> conntrack entry.

What do you think about this?

- add route change notification event to the net core
- add --update-source-address flag to the MASQUERADE target
- add a call for such events to the MASQUERADE target, when
  the flag is enabled

The called function then can scan the conntrack table and for every entry 
which has got the update-source-address flag, can check whether the source 
IP address should be changed. Those entries are then deleted.

Best regards,
Joysef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Ed W]

>> An in-kernel "route changed" notification were great, because then we 
>> could delete all MASQUERAD-ed entries where --update-source-address 
>> flag is set.
>
> If we could register to receive routing changed events for SNAT source 
> addresses, that would be great (more efficient) but if the mechanism 
> does not yet exist, it would be much more difficult to create.
>
> I suppose that if we just deleted them all on any routing change, then 
> that would solve the immediate problem for us. It would be ideal if 
> incoming packets could still be directed to the correct internal 
> source, but dropping them is a smaller problem than continuing to send 
> out invalid packets and keeping a broken entry alive forever.

I'm probably completely missing the point, but:

- Route changes such as you describe will be triggered by some user 
space event.
- Lets generically call that "dhcpcd" for the sake of a label
- These userspace events should all be capable of some kind of "hook" on 
change
- This userspace hook can zap the relevant conntrack entries

The pros/conns seem to be that:
- one more piece of userspace to maintain
- but there is now integration between the userspace routing decisions 
and the decision to zap certain conntrack entries


For example in my situation I dynamically tweak ipsets and an IP 
dropping into an IPset causes routing decisions to be made for just that 
user. When we add a user to an ipset we also zap previous conntrack 
entries relating to that user.

I'm not sure that I can see any way for the kernel to know when it 
should adjust conntrack entries in my situation?


Cheers

Ed W

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Chris Wilson]

Hi Jozsef,

On Mon, 12 Nov 2012, Jozsef Kadlecsik wrote:

>>>> * is sent out of an interface that does not list the packet's new
>>>> (SNAT-to)
>>>> source address as one of its IP addresses (i.e. if this were a new
>>>> connection, MASQUERADE would not choose this source address), and
>>>
>>> First, it'd mean a heavy checking for every matched packet.
>>
>> Is it really so expensive to get the list of IP addresses for an interface
>> and check down the list for presence of a given address?
>
> That'd mean that for *every* packet first the next routing hop should be
> calculated in netfilter too

Why "too"? Doesn't the kernel do that already? Why not use the results of 
the kernel's decision?

> then an IP address of the interface selected and then the old and the 
> calculated addresses compared. For every packet, unconditionally, that's 
> simply too much overhead.

This only needs to be done if the packet hits a conntrack with 
--update-source-address. And looking down the list of IP addresses of the 
interface in that case doesn't seem too bad. But I'm willing to concede on 
it.

>> If so, perhaps that's a problem elsewhere in the kernel?
>
> No, it's netfilter specific.

I mean, it shouldn't be hard (or expensive) to get a list of IP addresses 
for an interface, and it shouldn't be hard to check whether a given IP is 
in that list. If it is, then it seems to me that something must be broken. 
But I don't want to have a fight over it.

>> Another option is to check, after routing, the source address that 
>> MASQUERADE would have assigned to the packet. There must be a mechanism 
>> for that, because we already do it for the first packet in the 
>> connection, and it surely cannot be expensive. If it's different to the 
>> current SNAT-to address in the conntrack, and the flag is set on the 
>> conntrack, then update the conntrack.
>
> The first packets are special: the (NAT) rules are evaluated for the 
> very first packets only. Subsequent ones are looked up in the conntrack 
> table and the nat table is completely skipped. As I wrote, there's no 
> callback/hook by which such a functionality for MASQUERADE could be 
> called.

That doesn't mean that it's expensive, just that we don't do it at the 
moment. And the mangle table is called for every packet, so netfilter 
already does some work for every packet.

> Also, SNAT/MASQUERADE is the very last action in netfilter.

Sorry, but why does that make a difference?

> My concern is that there can for example be conntrack extensions attached
> to the conntrack entry, which may became broken if the IP address was
> simply changed.

Good point, I didn't realise that.

>> I think the problem applies even more to portless protocols such as 
>> GRE. Devices have no choice to "change the source port" when making a 
>> new connection. Any packets that they send, will definitely keep a 
>> stale entry alive.
>>
>> Thank you for considering and engaging with this proposal :)
>
> It's an issue, which should be solved in MASQUERADE itself. The 
> conntrack tool can be used to solve it, in theory. In practice it's not 
> easy at all: a daemon should be run on the machine which catches the 
> routing changes and then calls "conntrack".

Agreed :)

Cheers, Chris.
-- 
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Future Business, Cam City FC, Milton Rd, Cambridge, CB4 1UY, UK

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Chris Wilson]

Hi Jozsef,

On Mon, 12 Nov 2012, Jozsef Kadlecsik wrote:

> What do you think about this?
>
> - add route change notification event to the net core
> - add --update-source-address flag to the MASQUERADE target
> - add a call for such events to the MASQUERADE target, when
>  the flag is enabled
>
> The called function then can scan the conntrack table and for every entry
> which has got the update-source-address flag, can check whether the source
> IP address should be changed. Those entries are then deleted.

That sounds like a relatively easy implementation that would solve the 
main problem for us.

I think there might be one element missing from the above process: the 
actual conntrack entries created by the MASQUERADE --update-source-address 
rule should be marked with an UPDATE_SOURCE_ADDRESS flag, so that they can 
be found and deleted when the routing changes.

Perhaps the flag/option should be called "--remove-if-routing-changes" or 
"--routing-dependent" or something like that, since the source address is 
not really being (directly) changed as I proposed at the beginning?

Cheers, Chris.
-- 
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Future Business, Cam City FC, Milton Rd, Cambridge, CB4 1UY, UK

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Pablo Neira Ayuso]

Hi Jozsef,

On Mon, Nov 12, 2012 at 08:34:26PM +0100, Jozsef Kadlecsik wrote:
> On Mon, 12 Nov 2012, Chris Wilson wrote:
> 
> > I propose that:
> > 
> > * when the packet matches an existing conntrack rule, and
> > 
> > * is sent out of an interface that does not list the packet's new (SNAT-to)
> > source address as one of its IP addresses (i.e. if this were a new
> > connection, MASQUERADE would not choose this source address), and
> > 
> > * the --update-source-address flag is set on the MASQUERADE target
> > 
> > then update the source address on the conntrack rule to the new one.
> > 
> > That's the same thing that would happen if we deleted the conntrack entry
> > first: MASQUERADE would choose a new source address and save it in the new
> > conntrack entry.
> 
> What do you think about this?
> 
> - add route change notification event to the net core
> - add --update-source-address flag to the MASQUERADE target
> - add a call for such events to the MASQUERADE target, when
>   the flag is enabled
> 
> The called function then can scan the conntrack table and for every entry 
> which has got the update-source-address flag, can check whether the source 
> IP address should be changed. Those entries are then deleted.

It seems to me this can be implemented this from user-space. It would
require a new working mode for conntrackd that would:

1) subscribe to route events via rtnl and libmnl.
2) get new interface address for some monitored address, also via rtnl.
3) iterate over the table and remove those entries with outdated IP
address.

All the infrastructure is ready, and it would not require any kernel
upgrade. What do you think about this approach?

Regards.

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Stephen Clark]

On 11/12/2012 06:30 PM, Pablo Neira Ayuso wrote:
> Hi Jozsef,
>
> On Mon, Nov 12, 2012 at 08:34:26PM +0100, Jozsef Kadlecsik wrote:
>> On Mon, 12 Nov 2012, Chris Wilson wrote:
>>
>>> I propose that:
>>>
>>> * when the packet matches an existing conntrack rule, and
>>>
>>> * is sent out of an interface that does not list the packet's new (SNAT-to)
>>> source address as one of its IP addresses (i.e. if this were a new
>>> connection, MASQUERADE would not choose this source address), and
>>>
>>> * the --update-source-address flag is set on the MASQUERADE target
>>>
>>> then update the source address on the conntrack rule to the new one.
>>>
>>> That's the same thing that would happen if we deleted the conntrack entry
>>> first: MASQUERADE would choose a new source address and save it in the new
>>> conntrack entry.
>> What do you think about this?
>>
>> - add route change notification event to the net core
>> - add --update-source-address flag to the MASQUERADE target
>> - add a call for such events to the MASQUERADE target, when
>>    the flag is enabled
>>
>> The called function then can scan the conntrack table and for every entry
>> which has got the update-source-address flag, can check whether the source
>> IP address should be changed. Those entries are then deleted.
> It seems to me this can be implemented this from user-space. It would
> require a new working mode for conntrackd that would:
>
> 1) subscribe to route events via rtnl and libmnl.
> 2) get new interface address for some monitored address, also via rtnl.
> 3) iterate over the table and remove those entries with outdated IP
> address.
>
> All the infrastructure is ready, and it would not require any kernel
> upgrade. What do you think about this approach?
>
>
A similar problem exists in the following scenario:
You have two upstream isp that you are doing load balancing by having multiple
default routes:
default
         nexthop via 66.xxx.xxx.xxx  dev eth1 weight 1
         nexthop via 205.xxx.xxx.xxx  dev eth2 weight 1
On one of the external interface you have a DNAT to
an internal server on a private address. The DNAT makes
a conntrack entry that is going to in effect do a SNAT on reponses
from the internal server back out to the internet, but the load balancing
decision on routing happens before this implicit SNAT so you have packets
trying to go out an interface where the source address does not fall in the
subnet of that interface.

Why is routing decision done before the SNAT?




-- 

"They that give up essential liberty to obtain temporary safety,
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty
decreases."  (Thomas Jefferson)

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Jozsef Kadlecsik]

On Tue, 13 Nov 2012, Stephen Clark wrote:

> On 11/12/2012 06:30 PM, Pablo Neira Ayuso wrote:
> > 
> > On Mon, Nov 12, 2012 at 08:34:26PM +0100, Jozsef Kadlecsik wrote:
> > > On Mon, 12 Nov 2012, Chris Wilson wrote:
> > > 
> > > > I propose that:
> > > > 
> > > > * when the packet matches an existing conntrack rule, and
> > > > 
> > > > * is sent out of an interface that does not list the packet's new
> > > > (SNAT-to)
> > > > source address as one of its IP addresses (i.e. if this were a new
> > > > connection, MASQUERADE would not choose this source address), and
> > > > 
> > > > * the --update-source-address flag is set on the MASQUERADE target
> > > > 
> > > > then update the source address on the conntrack rule to the new one.
> > > > 
> > > > That's the same thing that would happen if we deleted the conntrack
> > > > entry
> > > > first: MASQUERADE would choose a new source address and save it in the
> > > > new
> > > > conntrack entry.
> > > What do you think about this?
> > > 
> > > - add route change notification event to the net core
> > > - add --update-source-address flag to the MASQUERADE target
> > > - add a call for such events to the MASQUERADE target, when
> > >    the flag is enabled
> > > 
> > > The called function then can scan the conntrack table and for every entry
> > > which has got the update-source-address flag, can check whether the source
> > > IP address should be changed. Those entries are then deleted.
> > It seems to me this can be implemented this from user-space. It would
> > require a new working mode for conntrackd that would:
> > 
> > 1) subscribe to route events via rtnl and libmnl.
> > 2) get new interface address for some monitored address, also via rtnl.
> > 3) iterate over the table and remove those entries with outdated IP
> > address.
> > 
> > All the infrastructure is ready, and it would not require any kernel
> > upgrade. What do you think about this approach?
> > 
> A similar problem exists in the following scenario:
> You have two upstream isp that you are doing load balancing by having multiple
> default routes:
> default
>         nexthop via 66.xxx.xxx.xxx  dev eth1 weight 1
>         nexthop via 205.xxx.xxx.xxx  dev eth2 weight 1
> On one of the external interface you have a DNAT to
> an internal server on a private address. The DNAT makes
> a conntrack entry that is going to in effect do a SNAT on reponses
> from the internal server back out to the internet, but the load balancing
> decision on routing happens before this implicit SNAT so you have packets
> trying to go out an interface where the source address does not fall in the
> subnet of that interface.

In my opinion this is a broken network design. The DNAT should not depend 
on the external interface, problem solved.
 
> Why is routing decision done before the SNAT?

Because that is the natural place: this way OUTPUT and FORWARD can be 
separated and we can use the original source IP addresses in the rules.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Jozsef Kadlecsik]

On Mon, 12 Nov 2012, Chris Wilson wrote:

> On Mon, 12 Nov 2012, Jozsef Kadlecsik wrote:
> 
> > > > > * is sent out of an interface that does not list the packet's new
> > > > > (SNAT-to)
> > > > > source address as one of its IP addresses (i.e. if this were a new
> > > > > connection, MASQUERADE would not choose this source address), and
> > > > 
> > > > First, it'd mean a heavy checking for every matched packet.
> > > 
> > > Is it really so expensive to get the list of IP addresses for an interface
> > > and check down the list for presence of a given address?
> > 
> > That'd mean that for *every* packet first the next routing hop should be
> > calculated in netfilter too
> 
> Why "too"? Doesn't the kernel do that already? Why not use the results of the
> kernel's decision?

I wrote from memory and I was wrong: routing info is reused.
 
> > then an IP address of the interface selected and then the old and the
> > calculated addresses compared. For every packet, unconditionally, that's
> > simply too much overhead.
> 
> This only needs to be done if the packet hits a conntrack with
> --update-source-address. And looking down the list of IP addresses of the
> interface in that case doesn't seem too bad. But I'm willing to concede on it.

In your proposal, the checking of the --update-source-address flag was the 
last one. So I commented it according to that logic.
 
> > > If so, perhaps that's a problem elsewhere in the kernel?
> > 
> > No, it's netfilter specific.
> 
> I mean, it shouldn't be hard (or expensive) to get a list of IP addresses for
> an interface, and it shouldn't be hard to check whether a given IP is in that
> list. If it is, then it seems to me that something must be broken. But I don't
> want to have a fight over it.

What I press is that unnecessary checkings simply waste resources and 
therefore expensive. Even if something is fast, if it can be avoided, then 
we must do so.
 
> > > Another option is to check, after routing, the source address that
> > > MASQUERADE would have assigned to the packet. There must be a mechanism
> > > for that, because we already do it for the first packet in the connection,
> > > and it surely cannot be expensive. If it's different to the current
> > > SNAT-to address in the conntrack, and the flag is set on the conntrack,
> > > then update the conntrack.
> > 
> > The first packets are special: the (NAT) rules are evaluated for the very
> > first packets only. Subsequent ones are looked up in the conntrack table and
> > the nat table is completely skipped. As I wrote, there's no callback/hook by
> > which such a functionality for MASQUERADE could be called.
> 
> That doesn't mean that it's expensive, just that we don't do it at the moment.
> And the mangle table is called for every packet, so netfilter already does
> some work for every packet.

Yes, and the proposal of adding all hooks to the mangle table was also 
heavily argued. Sorry, but I regard it expensive to add a new hook to NAT 
in general, just to check a new MASQUERADE flag for every packet.
 
> > Also, SNAT/MASQUERADE is the very last action in netfilter.
> 
> Sorry, but why does that make a difference?

The "missing" callback/hook from NAT could be "worked around" if there 
were a netfilter hook after SNAT/MASQUERADE.

> > My concern is that there can for example be conntrack extensions attached
> > to the conntrack entry, which may became broken if the IP address was
> > simply changed.
> 
> Good point, I didn't realise that.
> 
> > > I think the problem applies even more to portless protocols such as GRE.
> > > Devices have no choice to "change the source port" when making a new
> > > connection. Any packets that they send, will definitely keep a stale entry
> > > alive.
> > > 
> > > Thank you for considering and engaging with this proposal :)
> > 
> > It's an issue, which should be solved in MASQUERADE itself. The conntrack
> > tool can be used to solve it, in theory. In practice it's not easy at all: a
> > daemon should be run on the machine which catches the routing changes and
> > then calls "conntrack".
> 
> Agreed :)

Please note, I believe only the second example you wrote is relevant. 
Starting VPN just out of the blue *after* connections are built up is just 
not the right thing.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Jozsef Kadlecsik]

On Mon, 12 Nov 2012, Chris Wilson wrote:

> On Mon, 12 Nov 2012, Jozsef Kadlecsik wrote:
> 
> > What do you think about this?
> > 
> > - add route change notification event to the net core
> > - add --update-source-address flag to the MASQUERADE target
> > - add a call for such events to the MASQUERADE target, when
> >  the flag is enabled
> > 
> > The called function then can scan the conntrack table and for every entry
> > which has got the update-source-address flag, can check whether the source
> > IP address should be changed. Those entries are then deleted.
> 
> That sounds like a relatively easy implementation that would solve the main
> problem for us.
> 
> I think there might be one element missing from the above process: the 
> actual conntrack entries created by the MASQUERADE 
> --update-source-address rule should be marked with an 
> UPDATE_SOURCE_ADDRESS flag, so that they can be found and deleted when 
> the routing changes.

Yes, I thought that is implied by the the MASQUERADE flag.
 
> Perhaps the flag/option should be called "--remove-if-routing-changes" or
> "--routing-dependent" or something like that, since the source address is not
> really being (directly) changed as I proposed at the beginning?

I dunno, I'm not very good at naming. The "--routing-dependent" is 
shorter.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Chris Wilson]

Hi Jozsef,

On Tue, 13 Nov 2012, Jozsef Kadlecsik wrote:

>>>> I think the problem applies even more to portless protocols such as GRE.
>>>> Devices have no choice to "change the source port" when making a new
>>>> connection. Any packets that they send, will definitely keep a stale entry
>>>> alive.
>
> Please note, I believe only the second example you wrote is relevant.
> Starting VPN just out of the blue *after* connections are built up is just
> not the right thing.

I don't want to argue too strongly for the VPN case because I didn't 
experience it myself. I found it while researching to understand this 
problem, and it was never solved (properly and reported) by the sysadmin 
who experienced it.

The problem I have with that is that I don't know how to configure the 
Linux firewall and devices behind it correctly to avoid this.

Devices behind the firewall are trying to establish connections with 
devices outside, constantly. The firewall reboots. As soon as networking 
is up and iptables allows the devices' packets to pass, they start 
creating conntrack entries, even if the VPN is not up and therefore the 
packets are being routed out of the wrong interface.

The VPN cannot be brought up until networking is up, so the only ways I 
can see to prevent this are:

* the firewall ruleset forbids packets destined for VPN addresses to leave 
via the public interface. But the VPN destinations not be known until the 
tunnel comes up and the VPN server declares to the client which subnets 
should be routed through it. It might even change every time. Or it might 
be the default route. Should we then forbid all packets from leaving on 
the public interface? How will the VPN communication happen then?

* disable IP forwarding until the VPN comes up. Affects all non-VPN 
traffic as well.

* keep the internal interface down until VPN comes up. Affects all non-VPN 
traffic as well.

* shoot lusers/device vendors in the head. Maybe the best option?

* flush the existing conntrack entries when the VPN comes up.

* any others?

Cheers, Chris.
-- 
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Future Business, Cam City FC, Milton Rd, Cambridge, CB4 1UY, UK

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Jozsef Kadlecsik]

On Tue, 13 Nov 2012, Pablo Neira Ayuso wrote:

> On Mon, Nov 12, 2012 at 08:34:26PM +0100, Jozsef Kadlecsik wrote:
> > 
> > What do you think about this?
> > 
> > - add route change notification event to the net core
> > - add --update-source-address flag to the MASQUERADE target
> > - add a call for such events to the MASQUERADE target, when
> >   the flag is enabled
> > 
> > The called function then can scan the conntrack table and for every entry 
> > which has got the update-source-address flag, can check whether the source 
> > IP address should be changed. Those entries are then deleted.
> 
> It seems to me this can be implemented this from user-space. It would
> require a new working mode for conntrackd that would:
> 
> 1) subscribe to route events via rtnl and libmnl.
> 2) get new interface address for some monitored address, also via rtnl.
> 3) iterate over the table and remove those entries with outdated IP
> address.
> 
> All the infrastructure is ready, and it would not require any kernel
> upgrade. What do you think about this approach?

So far conntrackd implements conntrack replication. It could be extended 
with such functionality, yes. However, it'd be just good if MASQUERADE 
would not require an external component (i.e. userspace daemon) to work in 
all cases.

Step 2 does indeed required?

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Jozsef Kadlecsik]

Hi Chris,

On Tue, 13 Nov 2012, Chris Wilson wrote:

> On Tue, 13 Nov 2012, Jozsef Kadlecsik wrote:
> 
> > Please note, I believe only the second example you wrote is relevant.
> > Starting VPN just out of the blue *after* connections are built up is just
> > not the right thing.
> 
> I don't want to argue too strongly for the VPN case because I didn't
> experience it myself. I found it while researching to understand this problem,
> and it was never solved (properly and reported) by the sysadmin who
> experienced it.
> 
> The problem I have with that is that I don't know how to configure the Linux
> firewall and devices behind it correctly to avoid this.
> 
> Devices behind the firewall are trying to establish connections with devices
> outside, constantly. The firewall reboots. As soon as networking is up and
> iptables allows the devices' packets to pass, they start creating conntrack
> entries, even if the VPN is not up and therefore the packets are being routed
> out of the wrong interface.

Ahh, the niceties of living systems! :-))
 
> The VPN cannot be brought up until networking is up, so the only ways I can
> see to prevent this are:
> 
> * the firewall ruleset forbids packets destined for VPN addresses to leave via
> the public interface. But the VPN destinations not be known until the tunnel
> comes up and the VPN server declares to the client which subnets should be
> routed through it. It might even change every time. Or it might be the default
> route. Should we then forbid all packets from leaving on the public interface?
> How will the VPN communication happen then?

But *something* selects which traffic is forwarded via the VPN tunnels. As 
egress-ingress filtering, the same selection can be used to allow the 
proper interfaces only.
 
> * disable IP forwarding until the VPN comes up. Affects all non-VPN traffic as
> well.
> 
> * keep the internal interface down until VPN comes up. Affects all non-VPN
> traffic as well.
> 
> * shoot lusers/device vendors in the head. Maybe the best option?
> 
> * flush the existing conntrack entries when the VPN comes up.
> 
> * any others?

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Pablo Neira Ayuso]

On Tue, Nov 13, 2012 at 05:11:11PM +0100, Jozsef Kadlecsik wrote:
> On Tue, 13 Nov 2012, Pablo Neira Ayuso wrote:
> 
> > On Mon, Nov 12, 2012 at 08:34:26PM +0100, Jozsef Kadlecsik wrote:
> > > 
> > > What do you think about this?
> > > 
> > > - add route change notification event to the net core
> > > - add --update-source-address flag to the MASQUERADE target
> > > - add a call for such events to the MASQUERADE target, when
> > >   the flag is enabled
> > > 
> > > The called function then can scan the conntrack table and for every entry 
> > > which has got the update-source-address flag, can check whether the source 
> > > IP address should be changed. Those entries are then deleted.
> > 
> > It seems to me this can be implemented this from user-space. It would
> > require a new working mode for conntrackd that would:
> > 
> > 1) subscribe to route events via rtnl and libmnl.
> > 2) get new interface address for some monitored address, also via rtnl.
> > 3) iterate over the table and remove those entries with outdated IP
> > address.
> > 
> > All the infrastructure is ready, and it would not require any kernel
> > upgrade. What do you think about this approach?
> 
> So far conntrackd implements conntrack replication. It could be extended 
> with such functionality, yes. However, it'd be just good if MASQUERADE 
> would not require an external component (i.e. userspace daemon) to work in 
> all cases.

I see. If the change is more or small and it allows it to work for all
cases, I'll be OK.

> Step 2 does indeed required?

Hm, not really, sorry.

Your plan is to set some flag in ct->status so you can identify those
that you need to kill, is it precise?

Regards.

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Chris Wilson]

Hi Jozsef,

On Tue, 13 Nov 2012, Jozsef Kadlecsik wrote:

>> The VPN cannot be brought up until networking is up, so the only ways I 
>> can see to prevent this are:
>>
>> * the firewall ruleset forbids packets destined for VPN addresses to 
>> leave via the public interface. But the VPN destinations not be known 
>> until the tunnel comes up and the VPN server declares to the client 
>> which subnets should be routed through it. It might even change every 
>> time. Or it might be the default route. Should we then forbid all 
>> packets from leaving on the public interface? How will the VPN 
>> communication happen then?
>
> But *something* selects which traffic is forwarded via the VPN tunnels. 
> As egress-ingress filtering, the same selection can be used to allow the 
> proper interfaces only.

The VPN software may decide, based on configuration received from the 
remote server, what traffic is routed. Almost all tunneled VPNs work this 
way; the only exception I can think of is IPsec. OpenVPN and anything 
PPP-based (PPTP and L2TP) all create an interface with a subnet and mask 
assigned by the server, and the kernel automatically routes all traffic 
for that subnet through the VPN. But what address will be assigned? Often, 
only the VPN server knows, and sometimes only at the moment of assignment.

We could say "you must know which subnet will be assigned in order to 
write your policy" which is technically correct, but not user friendly. 
I'm very heartened to see that you want to make the default behaviour 
(without resort to external tools) more user-friendly :)

Cheers, Chris.
-- 
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Future Business, Cam City FC, Milton Rd, Cambridge, CB4 1UY, UK

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Jan Engelhardt]

On Tuesday 2012-11-13 18:02, Chris Wilson wrote:
>
> The VPN software may decide, based on configuration received from the remote
> server, what traffic is routed. Almost all tunneled VPNs work this way; the
> only exception I can think of is IPsec. OpenVPN and anything PPP-based (PPTP
> and L2TP) all create an interface with a subnet and mask assigned by the
> server, and the kernel automatically routes all traffic for that subnet through
> the VPN.

StrongSWAN (as a setup daemon for the kernel's IPsec) also supports
dynamic address assignment. It's been some years since I did that;
looking at ipsec.conf now would hint towards "leftsourceip=%config"
being the magic option.

Furthermore, also in the IPsec case does the kernel automatically
route traffic for the rightsubnet properly - because Strongswan will
add the required routes (in local table no. 220, if you care).

> We could say "you must know which subnet will be assigned in order to write
> your policy" which is technically correct, but not user friendly.

I'm digressing here, but I'll throw it in nevertheless. Allowing the
tunnel traffic is pretty much the same:

openvpn: iptables -A OUTPUT -d 1.2.3.4 ...
ipsec: iptables -A OUTPUT -m policy --tunnel-dst 1.2.3.4 ...

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Stephen Clark]

On 11/13/2012 10:25 AM, Jozsef Kadlecsik wrote:
> On Tue, 13 Nov 2012, Stephen Clark wrote:
>
>> On 11/12/2012 06:30 PM, Pablo Neira Ayuso wrote:
>>> On Mon, Nov 12, 2012 at 08:34:26PM +0100, Jozsef Kadlecsik wrote:
>>>> On Mon, 12 Nov 2012, Chris Wilson wrote:
>>>>
>>>>> I propose that:
>>>>>
>>>>> * when the packet matches an existing conntrack rule, and
>>>>>
>>>>> * is sent out of an interface that does not list the packet's new
>>>>> (SNAT-to)
>>>>> source address as one of its IP addresses (i.e. if this were a new
>>>>> connection, MASQUERADE would not choose this source address), and
>>>>>
>>>>> * the --update-source-address flag is set on the MASQUERADE target
>>>>>
>>>>> then update the source address on the conntrack rule to the new one.
>>>>>
>>>>> That's the same thing that would happen if we deleted the conntrack
>>>>> entry
>>>>> first: MASQUERADE would choose a new source address and save it in the
>>>>> new
>>>>> conntrack entry.
>>>> What do you think about this?
>>>>
>>>> - add route change notification event to the net core
>>>> - add --update-source-address flag to the MASQUERADE target
>>>> - add a call for such events to the MASQUERADE target, when
>>>>     the flag is enabled
>>>>
>>>> The called function then can scan the conntrack table and for every entry
>>>> which has got the update-source-address flag, can check whether the source
>>>> IP address should be changed. Those entries are then deleted.
>>> It seems to me this can be implemented this from user-space. It would
>>> require a new working mode for conntrackd that would:
>>>
>>> 1) subscribe to route events via rtnl and libmnl.
>>> 2) get new interface address for some monitored address, also via rtnl.
>>> 3) iterate over the table and remove those entries with outdated IP
>>> address.
>>>
>>> All the infrastructure is ready, and it would not require any kernel
>>> upgrade. What do you think about this approach?
>>>
>> A similar problem exists in the following scenario:
>> You have two upstream isp that you are doing load balancing by having multiple
>> default routes:
>> default
>>          nexthop via 66.xxx.xxx.xxx  dev eth1 weight 1
>>          nexthop via 205.xxx.xxx.xxx  dev eth2 weight 1
>> On one of the external interface you have a DNAT to
>> an internal server on a private address. The DNAT makes
>> a conntrack entry that is going to in effect do a SNAT on reponses
>> from the internal server back out to the internet, but the load balancing
>> decision on routing happens before this implicit SNAT so you have packets
>> trying to go out an interface where the source address does not fall in the
>> subnet of that interface.
> In my opinion this is a broken network design. The DNAT should not depend
> on the external interface, problem solved.
>   
Hmmm... what does this mean ^^^ ?
Say you have the follwoing:
eth1 with ips 66.xxx.xxx.1 and 66.xxx.xxx.2
eth2 with ip 205.xxx.xxx.xxx
eth0 with ip 10.0.1.254/24
with a server at 10.0.1.253.

iptables -A PREROUTING -i eth1 -d 66.xxx.xxx.2 -j DNAT --to-destination 10.0.1.253

How else would you access an internal server at
a private address without using a DNAT from an external public ip? Is there some
other way to do this that I am not aware of?



>> Why is routing decision done before the SNAT?
> Because that is the natural place: this way OUTPUT and FORWARD can be
> separated and we can use the original source IP addresses in the rules.
>
>
<snip>

-- 

"They that give up essential liberty to obtain temporary safety,
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty
decreases."  (Thomas Jefferson)

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Jozsef Kadlecsik]

On Tue, 13 Nov 2012, Stephen Clark wrote:

> On 11/13/2012 10:25 AM, Jozsef Kadlecsik wrote:
> > On Tue, 13 Nov 2012, Stephen Clark wrote:
> > 
> > > A similar problem exists in the following scenario:
> > > You have two upstream isp that you are doing load balancing by having
> > > multiple
> > > default routes:
> > > default
> > >          nexthop via 66.xxx.xxx.xxx  dev eth1 weight 1
> > >          nexthop via 205.xxx.xxx.xxx  dev eth2 weight 1
> > > On one of the external interface you have a DNAT to
> > > an internal server on a private address. The DNAT makes
> > > a conntrack entry that is going to in effect do a SNAT on reponses
> > > from the internal server back out to the internet, but the load balancing
> > > decision on routing happens before this implicit SNAT so you have packets
> > > trying to go out an interface where the source address does not fall in
> > > the
> > > subnet of that interface.
> > In my opinion this is a broken network design. The DNAT should not depend
> > on the external interface, problem solved.
> >   
> Hmmm... what does this mean ^^^ ?
> Say you have the follwoing:
> eth1 with ips 66.xxx.xxx.1 and 66.xxx.xxx.2
> eth2 with ip 205.xxx.xxx.xxx
> eth0 with ip 10.0.1.254/24
> with a server at 10.0.1.253.
> 
> iptables -A PREROUTING -i eth1 -d 66.xxx.xxx.2 -j DNAT --to-destination
> 10.0.1.253
> 
> How else would you access an internal server at a private address 
> without using a DNAT from an external public ip? Is there some other way 
> to do this that I am not aware of?

Everything depends on your backup provider: does it route the network 
66.xxx.xxx.xxx/y to you or not?

- If the answer is no, then the rule above is correct but the internal 
  server cannot be reached when the backup line is up. So it does not
  matter what's in the conntrack table, no answer is sent over the backup
  link to you.
- If the answer is yes, then the rule should not contain the "-i eth1"
  part and your internal server could be reached as 66.xxx.xxx.2,
  independent of the uplinks.

Best regards,
Jozsef 
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Stephen Clark]

On 11/13/2012 02:24 PM, Jozsef Kadlecsik wrote:
> On Tue, 13 Nov 2012, Stephen Clark wrote:
>
>> On 11/13/2012 10:25 AM, Jozsef Kadlecsik wrote:
>>> On Tue, 13 Nov 2012, Stephen Clark wrote:
>>>
>>>> A similar problem exists in the following scenario:
>>>> You have two upstream isp that you are doing load balancing by having
>>>> multiple
>>>> default routes:
>>>> default
>>>>           nexthop via 66.xxx.xxx.xxx  dev eth1 weight 1
>>>>           nexthop via 205.xxx.xxx.xxx  dev eth2 weight 1
>>>> On one of the external interface you have a DNAT to
>>>> an internal server on a private address. The DNAT makes
>>>> a conntrack entry that is going to in effect do a SNAT on reponses
>>>> from the internal server back out to the internet, but the load balancing
>>>> decision on routing happens before this implicit SNAT so you have packets
>>>> trying to go out an interface where the source address does not fall in
>>>> the
>>>> subnet of that interface.
>>> In my opinion this is a broken network design. The DNAT should not depend
>>> on the external interface, problem solved.
>>>
>> Hmmm... what does this mean ^^^ ?
>> Say you have the follwoing:
>> eth1 with ips 66.xxx.xxx.1 and 66.xxx.xxx.2
>> eth2 with ip 205.xxx.xxx.xxx
>> eth0 with ip 10.0.1.254/24
>> with a server at 10.0.1.253.
>>
>> iptables -A PREROUTING -i eth1 -d 66.xxx.xxx.2 -j DNAT --to-destination
>> 10.0.1.253
>>
>> How else would you access an internal server at a private address
>> without using a DNAT from an external public ip? Is there some other way
>> to do this that I am not aware of?
> Everything depends on your backup provider: does it route the network
> 66.xxx.xxx.xxx/y to you or not?
>
> - If the answer is no, then the rule above is correct but the internal
>    server cannot be reached when the backup line is up. So it does not
>    matter what's in the conntrack table, no answer is sent over the backup
>    link to you.
> - If the answer is yes, then the rule should not contain the "-i eth1"
>    part and your internal server could be reached as 66.xxx.xxx.2,
>    independent of the uplinks.
There is no intent for backup of the incoming connection to 66.xxx.xxx.2 - only 
load balancing outgoing
traffic.

-- 

"They that give up essential liberty to obtain temporary safety,
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty
decreases."  (Thomas Jefferson)

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Jozsef Kadlecsik]

On Tue, 13 Nov 2012, Stephen Clark wrote:

> On 11/13/2012 02:24 PM, Jozsef Kadlecsik wrote:
> > On Tue, 13 Nov 2012, Stephen Clark wrote:
> > 
> > > On 11/13/2012 10:25 AM, Jozsef Kadlecsik wrote:
> > > > On Tue, 13 Nov 2012, Stephen Clark wrote:
> > > > 
> > > > > A similar problem exists in the following scenario:
> > > > > You have two upstream isp that you are doing load balancing by having
> > > > > multiple
> > > > > default routes:
> > > > > default
> > > > >           nexthop via 66.xxx.xxx.xxx  dev eth1 weight 1
> > > > >           nexthop via 205.xxx.xxx.xxx  dev eth2 weight 1
> > > > > On one of the external interface you have a DNAT to
> > > > > an internal server on a private address. The DNAT makes
> > > > > a conntrack entry that is going to in effect do a SNAT on reponses
> > > > > from the internal server back out to the internet, but the load
> > > > > balancing
> > > > > decision on routing happens before this implicit SNAT so you have
> > > > > packets
> > > > > trying to go out an interface where the source address does not fall
> > > > > in
> > > > > the
> > > > > subnet of that interface.
> > > > In my opinion this is a broken network design. The DNAT should not
> > > > depend
> > > > on the external interface, problem solved.
> > > > 
> > > Hmmm... what does this mean ^^^ ?
> > > Say you have the follwoing:
> > > eth1 with ips 66.xxx.xxx.1 and 66.xxx.xxx.2
> > > eth2 with ip 205.xxx.xxx.xxx
> > > eth0 with ip 10.0.1.254/24
> > > with a server at 10.0.1.253.
> > > 
> > > iptables -A PREROUTING -i eth1 -d 66.xxx.xxx.2 -j DNAT --to-destination
> > > 10.0.1.253
> > > 
> > > How else would you access an internal server at a private address
> > > without using a DNAT from an external public ip? Is there some other way
> > > to do this that I am not aware of?
> > Everything depends on your backup provider: does it route the network
> > 66.xxx.xxx.xxx/y to you or not?
> > 
> > - If the answer is no, then the rule above is correct but the internal
> >    server cannot be reached when the backup line is up. So it does not
> >    matter what's in the conntrack table, no answer is sent over the backup
> >    link to you.
> > - If the answer is yes, then the rule should not contain the "-i eth1"
> >    part and your internal server could be reached as 66.xxx.xxx.2,
> >    independent of the uplinks.
> There is no intent for backup of the incoming connection to 66.xxx.xxx.2 -
> only load balancing outgoing
> traffic.

Then I don't understand, what is the problem. When the reply packet is 
sent out over the backup line, why should the source address fall into 
the subnet of the outgoing interface? Unless, of course if you yourself or 
your backup provider prevents it by egress filtering.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Stephen Clark]

On 11/14/2012 03:08 AM, Jozsef Kadlecsik wrote:
> On Tue, 13 Nov 2012, Stephen Clark wrote:
>
>> On 11/13/2012 02:24 PM, Jozsef Kadlecsik wrote:
>>> On Tue, 13 Nov 2012, Stephen Clark wrote:
>>>
>>>> On 11/13/2012 10:25 AM, Jozsef Kadlecsik wrote:
>>>>> On Tue, 13 Nov 2012, Stephen Clark wrote:
>>>>>
>>>>>> A similar problem exists in the following scenario:
>>>>>> You have two upstream isp that you are doing load balancing by having
>>>>>> multiple
>>>>>> default routes:
>>>>>> default
>>>>>>            nexthop via 66.xxx.xxx.xxx  dev eth1 weight 1
>>>>>>            nexthop via 205.xxx.xxx.xxx  dev eth2 weight 1
>>>>>> On one of the external interface you have a DNAT to
>>>>>> an internal server on a private address. The DNAT makes
>>>>>> a conntrack entry that is going to in effect do a SNAT on reponses
>>>>>> from the internal server back out to the internet, but the load
>>>>>> balancing
>>>>>> decision on routing happens before this implicit SNAT so you have
>>>>>> packets
>>>>>> trying to go out an interface where the source address does not fall
>>>>>> in
>>>>>> the
>>>>>> subnet of that interface.
>>>>> In my opinion this is a broken network design. The DNAT should not
>>>>> depend
>>>>> on the external interface, problem solved.
>>>>>
>>>> Hmmm... what does this mean ^^^ ?
>>>> Say you have the follwoing:
>>>> eth1 with ips 66.xxx.xxx.1 and 66.xxx.xxx.2
>>>> eth2 with ip 205.xxx.xxx.xxx
>>>> eth0 with ip 10.0.1.254/24
>>>> with a server at 10.0.1.253.
>>>>
>>>> iptables -A PREROUTING -i eth1 -d 66.xxx.xxx.2 -j DNAT --to-destination
>>>> 10.0.1.253
>>>>
>>>> How else would you access an internal server at a private address
>>>> without using a DNAT from an external public ip? Is there some other way
>>>> to do this that I am not aware of?
>>> Everything depends on your backup provider: does it route the network
>>> 66.xxx.xxx.xxx/y to you or not?
>>>
>>> - If the answer is no, then the rule above is correct but the internal
>>>     server cannot be reached when the backup line is up. So it does not
>>>     matter what's in the conntrack table, no answer is sent over the backup
>>>     link to you.
>>> - If the answer is yes, then the rule should not contain the "-i eth1"
>>>     part and your internal server could be reached as 66.xxx.xxx.2,
>>>     independent of the uplinks.
>> There is no intent for backup of the incoming connection to 66.xxx.xxx.2 -
>> only load balancing outgoing
>> traffic.
> Then I don't understand, what is the problem. When the reply packet is
> sent out over the backup line, why should the source address fall into
> the subnet of the outgoing interface? Unless, of course if you yourself or
> your backup provider prevents it by egress filtering.
>
A lot of ISPs in the U.S. do reverse path filtering and drop packets that could 
not originate from their provided subnet.
If they did not do this then of course there would be no problem.

Regards,
Steve

-- 

"They that give up essential liberty to obtain temporary safety,
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty
decreases."  (Thomas Jefferson)

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Chris Wilson]

Hi all,

On Wed, 14 Nov 2012, Stephen Clark wrote:
> On 11/14/2012 03:08 AM, Jozsef Kadlecsik wrote:
>
>> Then I don't understand, what is the problem. When the reply packet is 
>> sent out over the backup line, why should the source address fall into 
>> the subnet of the outgoing interface? Unless, of course if you yourself 
>> or your backup provider prevents it by egress filtering.
>
> A lot of ISPs in the U.S. do reverse path filtering and drop packets 
> that could not originate from their provided subnet. If they did not do 
> this then of course there would be no problem.

Not just in the US. It's common here in the UK too. IMHO all ISPs should 
do this to prevent spoofing attacks, so that attacks are traceable, unless 
you have a special agreement with them to use their connection for certain 
specific other source addresses which are also traceable to you.

Cheers, Chris.
-- 
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Future Business, Cam City FC, Milton Rd, Cambridge, CB4 1UY, UK

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Jozsef Kadlecsik]

On Wed, 14 Nov 2012, Stephen Clark wrote:

> On 11/14/2012 03:08 AM, Jozsef Kadlecsik wrote:
> > On Tue, 13 Nov 2012, Stephen Clark wrote:
> > 
> > > On 11/13/2012 02:24 PM, Jozsef Kadlecsik wrote:
> > > > On Tue, 13 Nov 2012, Stephen Clark wrote:
> > > > 
> > > > > On 11/13/2012 10:25 AM, Jozsef Kadlecsik wrote:
> > > > > > On Tue, 13 Nov 2012, Stephen Clark wrote:
> > > > > > 
> > > > > > > A similar problem exists in the following scenario:
> > > > > > > You have two upstream isp that you are doing load balancing by
> > > > > > > having
> > > > > > > multiple
> > > > > > > default routes:
> > > > > > > default
> > > > > > >            nexthop via 66.xxx.xxx.xxx  dev eth1 weight 1
> > > > > > >            nexthop via 205.xxx.xxx.xxx  dev eth2 weight 1
> > > > > > > On one of the external interface you have a DNAT to
> > > > > > > an internal server on a private address. The DNAT makes
> > > > > > > a conntrack entry that is going to in effect do a SNAT on reponses
> > > > > > > from the internal server back out to the internet, but the load
> > > > > > > balancing
> > > > > > > decision on routing happens before this implicit SNAT so you have
> > > > > > > packets
> > > > > > > trying to go out an interface where the source address does not
> > > > > > > fall
> > > > > > > in
> > > > > > > the
> > > > > > > subnet of that interface.
> > > > > > In my opinion this is a broken network design. The DNAT should not
> > > > > > depend
> > > > > > on the external interface, problem solved.
> > > > > > 
> > > > > Hmmm... what does this mean ^^^ ?
> > > > > Say you have the follwoing:
> > > > > eth1 with ips 66.xxx.xxx.1 and 66.xxx.xxx.2
> > > > > eth2 with ip 205.xxx.xxx.xxx
> > > > > eth0 with ip 10.0.1.254/24
> > > > > with a server at 10.0.1.253.
> > > > > 
> > > > > iptables -A PREROUTING -i eth1 -d 66.xxx.xxx.2 -j DNAT
> > > > > --to-destination
> > > > > 10.0.1.253
> > > > > 
> > > > > How else would you access an internal server at a private address
> > > > > without using a DNAT from an external public ip? Is there some other
> > > > > way
> > > > > to do this that I am not aware of?
> > > > Everything depends on your backup provider: does it route the network
> > > > 66.xxx.xxx.xxx/y to you or not?
> > > > 
> > > > - If the answer is no, then the rule above is correct but the internal
> > > >     server cannot be reached when the backup line is up. So it does not
> > > >     matter what's in the conntrack table, no answer is sent over the
> > > > backup
> > > >     link to you.
> > > > - If the answer is yes, then the rule should not contain the "-i eth1"
> > > >     part and your internal server could be reached as 66.xxx.xxx.2,
> > > >     independent of the uplinks.
> > > There is no intent for backup of the incoming connection to 66.xxx.xxx.2 -
> > > only load balancing outgoing
> > > traffic.
> > Then I don't understand, what is the problem. When the reply packet is
> > sent out over the backup line, why should the source address fall into
> > the subnet of the outgoing interface? Unless, of course if you yourself or
> > your backup provider prevents it by egress filtering.
> > 
> A lot of ISPs in the U.S. do reverse path filtering and drop packets that
> could not originate from their provided subnet.
> If they did not do this then of course there would be no problem.

But then this traffic is not load balanced at all and the reply packets 
must be sent out over eth1 only. So you have to add a routing rule which 
forces routing over eth1.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Stephen Clark]

On 11/14/2012 03:15 PM, Jozsef Kadlecsik wrote:
> On Wed, 14 Nov 2012, Stephen Clark wrote:
>
>> On 11/14/2012 03:08 AM, Jozsef Kadlecsik wrote:
>>> On Tue, 13 Nov 2012, Stephen Clark wrote:
>>>
>>>> On 11/13/2012 02:24 PM, Jozsef Kadlecsik wrote:
>>>>> On Tue, 13 Nov 2012, Stephen Clark wrote:
>>>>>
>>>>>> On 11/13/2012 10:25 AM, Jozsef Kadlecsik wrote:
>>>>>>> On Tue, 13 Nov 2012, Stephen Clark wrote:
>>>>>>>
>>>>>>>> A similar problem exists in the following scenario:
>>>>>>>> You have two upstream isp that you are doing load balancing by
>>>>>>>> having
>>>>>>>> multiple
>>>>>>>> default routes:
>>>>>>>> default
>>>>>>>>             nexthop via 66.xxx.xxx.xxx  dev eth1 weight 1
>>>>>>>>             nexthop via 205.xxx.xxx.xxx  dev eth2 weight 1
>>>>>>>> On one of the external interface you have a DNAT to
>>>>>>>> an internal server on a private address. The DNAT makes
>>>>>>>> a conntrack entry that is going to in effect do a SNAT on reponses
>>>>>>>> from the internal server back out to the internet, but the load
>>>>>>>> balancing
>>>>>>>> decision on routing happens before this implicit SNAT so you have
>>>>>>>> packets
>>>>>>>> trying to go out an interface where the source address does not
>>>>>>>> fall
>>>>>>>> in
>>>>>>>> the
>>>>>>>> subnet of that interface.
>>>>>>> In my opinion this is a broken network design. The DNAT should not
>>>>>>> depend
>>>>>>> on the external interface, problem solved.
>>>>>>>
>>>>>> Hmmm... what does this mean ^^^ ?
>>>>>> Say you have the follwoing:
>>>>>> eth1 with ips 66.xxx.xxx.1 and 66.xxx.xxx.2
>>>>>> eth2 with ip 205.xxx.xxx.xxx
>>>>>> eth0 with ip 10.0.1.254/24
>>>>>> with a server at 10.0.1.253.
>>>>>>
>>>>>> iptables -A PREROUTING -i eth1 -d 66.xxx.xxx.2 -j DNAT
>>>>>> --to-destination
>>>>>> 10.0.1.253
>>>>>>
>>>>>> How else would you access an internal server at a private address
>>>>>> without using a DNAT from an external public ip? Is there some other
>>>>>> way
>>>>>> to do this that I am not aware of?
>>>>> Everything depends on your backup provider: does it route the network
>>>>> 66.xxx.xxx.xxx/y to you or not?
>>>>>
>>>>> - If the answer is no, then the rule above is correct but the internal
>>>>>      server cannot be reached when the backup line is up. So it does not
>>>>>      matter what's in the conntrack table, no answer is sent over the
>>>>> backup
>>>>>      link to you.
>>>>> - If the answer is yes, then the rule should not contain the "-i eth1"
>>>>>      part and your internal server could be reached as 66.xxx.xxx.2,
>>>>>      independent of the uplinks.
>>>> There is no intent for backup of the incoming connection to 66.xxx.xxx.2 -
>>>> only load balancing outgoing
>>>> traffic.
>>> Then I don't understand, what is the problem. When the reply packet is
>>> sent out over the backup line, why should the source address fall into
>>> the subnet of the outgoing interface? Unless, of course if you yourself or
>>> your backup provider prevents it by egress filtering.
>>>
>> A lot of ISPs in the U.S. do reverse path filtering and drop packets that
>> could not originate from their provided subnet.
>> If they did not do this then of course there would be no problem.
> But then this traffic is not load balanced at all and the reply packets
> must be sent out over eth1 only. So you have to add a routing rule which
> forces routing over eth1.
>

You are correct and that is what we do. But it would be nice if the linux kernel was
smart enough to make sure the packet went out the correct interface without having
to add additional rules.

But I guess one could argue this gives the user more control.

-- 

"They that give up essential liberty to obtain temporary safety,
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty
decreases."  (Thomas Jefferson)

Re: UDP packets sent with wrong source address after routing change [AV#3431]

[Jozsef Kadlecsik]

On Thu, 15 Nov 2012, Stephen Clark wrote:

> On 11/14/2012 03:15 PM, Jozsef Kadlecsik wrote:
> > On Wed, 14 Nov 2012, Stephen Clark wrote:
> > 
> > > On 11/14/2012 03:08 AM, Jozsef Kadlecsik wrote:
> > > > On Tue, 13 Nov 2012, Stephen Clark wrote:
> > > > 
> > > > > On 11/13/2012 02:24 PM, Jozsef Kadlecsik wrote:
> > > > > > On Tue, 13 Nov 2012, Stephen Clark wrote:
> > > > > > 
> > > > > > > On 11/13/2012 10:25 AM, Jozsef Kadlecsik wrote:
> > > > > > > > On Tue, 13 Nov 2012, Stephen Clark wrote:
> > > > > > > > 
> > > > > > > > > A similar problem exists in the following scenario:
> > > > > > > > > You have two upstream isp that you are doing load balancing by
> > > > > > > > > having
> > > > > > > > > multiple
> > > > > > > > > default routes:
> > > > > > > > > default
> > > > > > > > >             nexthop via 66.xxx.xxx.xxx  dev eth1 weight 1
> > > > > > > > >             nexthop via 205.xxx.xxx.xxx  dev eth2 weight 1
> > > > > > > > > On one of the external interface you have a DNAT to
> > > > > > > > > an internal server on a private address. The DNAT makes
> > > > > > > > > a conntrack entry that is going to in effect do a SNAT on
> > > > > > > > > reponses
> > > > > > > > > from the internal server back out to the internet, but the
> > > > > > > > > load
> > > > > > > > > balancing
> > > > > > > > > decision on routing happens before this implicit SNAT so you
> > > > > > > > > have
> > > > > > > > > packets
> > > > > > > > > trying to go out an interface where the source address does
> > > > > > > > > not
> > > > > > > > > fall
> > > > > > > > > in
> > > > > > > > > the
> > > > > > > > > subnet of that interface.
> > > > > > > > In my opinion this is a broken network design. The DNAT should
> > > > > > > > not
> > > > > > > > depend
> > > > > > > > on the external interface, problem solved.
> > > > > > > > 
> > > > > > > Hmmm... what does this mean ^^^ ?
> > > > > > > Say you have the follwoing:
> > > > > > > eth1 with ips 66.xxx.xxx.1 and 66.xxx.xxx.2
> > > > > > > eth2 with ip 205.xxx.xxx.xxx
> > > > > > > eth0 with ip 10.0.1.254/24
> > > > > > > with a server at 10.0.1.253.
> > > > > > > 
> > > > > > > iptables -A PREROUTING -i eth1 -d 66.xxx.xxx.2 -j DNAT
> > > > > > > --to-destination
> > > > > > > 10.0.1.253
> > > > > > > 
> > > > > > > How else would you access an internal server at a private address
> > > > > > > without using a DNAT from an external public ip? Is there some
> > > > > > > other
> > > > > > > way
> > > > > > > to do this that I am not aware of?
> > > > > > Everything depends on your backup provider: does it route the
> > > > > > network
> > > > > > 66.xxx.xxx.xxx/y to you or not?
> > > > > > 
> > > > > > - If the answer is no, then the rule above is correct but the
> > > > > > internal
> > > > > >      server cannot be reached when the backup line is up. So it does
> > > > > > not
> > > > > >      matter what's in the conntrack table, no answer is sent over
> > > > > > the
> > > > > > backup
> > > > > >      link to you.
> > > > > > - If the answer is yes, then the rule should not contain the "-i
> > > > > > eth1"
> > > > > >      part and your internal server could be reached as 66.xxx.xxx.2,
> > > > > >      independent of the uplinks.
> > > > > There is no intent for backup of the incoming connection to
> > > > > 66.xxx.xxx.2 -
> > > > > only load balancing outgoing
> > > > > traffic.
> > > > Then I don't understand, what is the problem. When the reply packet is
> > > > sent out over the backup line, why should the source address fall into
> > > > the subnet of the outgoing interface? Unless, of course if you yourself
> > > > or
> > > > your backup provider prevents it by egress filtering.
> > > > 
> > > A lot of ISPs in the U.S. do reverse path filtering and drop packets that
> > > could not originate from their provided subnet.
> > > If they did not do this then of course there would be no problem.
> > But then this traffic is not load balanced at all and the reply packets
> > must be sent out over eth1 only. So you have to add a routing rule which
> > forces routing over eth1.
> 
> You are correct and that is what we do. But it would be nice if the 
> linux kernel was smart enough to make sure the packet went out the 
> correct interface without having to add additional rules.

You told the kernel that both link/interfaces can be used for default 
route, so it just obeys the rules. Source address is irrelevant at 
routing, unless explicitely used in the rules.
 
Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary