Re: [PATCH net-next] IPv6: enable TCP to use an anycast address

Re: [PATCH net-next] IPv6: enable TCP to use an anycast address

[François-Xavier Le Bail]

Hello,

> Actually, I was alerted by reset processing in your patch, it cannot be right.

Perhaps, I missed something.

My tests were:
1) enable anycast in tcp_v6_conn_request().
So, a client-serveur exchange with anycast destination address is possible.

2) try to connect with a port not listen by the server.
With unicast destination address, the client got a reset.
With anycast destination address, no reset, the client try several syn and give up.

3) enable anycast in tcp_v6_send_reset().
With anycast destination address, the client got a reset.
So unicast and anycast behave the same.

Do you see any problem with this behaviour ?

> Do not you think this must not be enabled for common use? At least
> some separate sysctl disabled by default.

We could indeed use a sysctl, disabled by default.

But my goal was to enable anycast address usage transparently, if possible.

It's only a possibility for some use cases and if it's dont break unicast TCP usage,
like I think it, I propose to enable this change without sysctl.

So, my proposals for the TCP case, in descending order of preference are:

1) No sysctl.
2) A sysctl disabled par default, used in tcp_v6_conn_request() and tcp_v6_send_reset().
3) A sysctl disabled par default, used in tcp_v6_send_reset(), if problem only here.

Greetings,
Francois-Xavier

Re: [PATCH net-next] IPv6: enable TCP to use an anycast address

[Alexey Kuznetsov]

Hello!

On Wed, Jan 22, 2014 at 6:32 PM, François-Xavier Le Bail
<fx.lebail@yahoo.com> wrote:
>> Actually, I was alerted by reset processing in your patch, it cannot be right.
>
> Perhaps, I missed something.

Perhaps, I missed something. According to my old knowledge, tcp cannot
be used with anycasts
by many reasons and I have no idea what could change to the moment.

F.e. what do you do when a segment arrives on listening socket w/o
connection associated?
If you sent reset, you could reset someone's valid connection (I mean
this, saying "it cannot be right").
If you do not, you violate protocol, those resets are required to
clean up stale states.

>> Do not you think this must not be enabled for common use? At least
>> some separate sysctl disabled by default.
>
> We could indeed use a sysctl, disabled by default.
>
> But my goal was to enable anycast address usage transparently, if possible.

IMHO it is logically impossible. That's why my first question was
about a document which
allowed anycasts with TCP. (BTW I found your answer unconvincing, at least.)
Traditional anycasting used to be stateless and it was impossible to
use with TCP
because unsynchronized TCP connections would be unavoidable.
AFAIK only suggestion from ancient RFC1546 could cure this problem,
but it is very tricky and implies client knows that destination is anycast.

Actually, I do not even understand how the idea of use of anycast with TCP
emerged. The only situation I can imagine: the router is expected to
do full scale
connection tracking and to bind connections to anycast destinations to
correct nodes.
(And the same moment you understand that the notion of anycast is
completely lost
in this picture: the same thing is done by load balancers in IPv4 for
unicast addresses
for ages) I still have no idea how it is going to work when both
client and servers bound
to anycasts are on the same link.

So, it looks like you need this sysctl to announce: "someone cares
about proper routing
on this network and tcp should work". It looks obvious this mode
cannot be enabled by default.

Alexey

Re: [PATCH net-next] IPv6: enable TCP to use an anycast address

[Hannes Frederic Sowa]

Hi!

On Thu, Jan 23, 2014 at 01:11:52AM +0400, Alexey Kuznetsov wrote:
> On Wed, Jan 22, 2014 at 6:32 PM, François-Xavier Le Bail
> <fx.lebail@yahoo.com> wrote:
> >> Actually, I was alerted by reset processing in your patch, it cannot be right.
> >
> > Perhaps, I missed something.
> 
> Perhaps, I missed something. According to my old knowledge, tcp cannot
> be used with anycasts
> by many reasons and I have no idea what could change to the moment.
> 
> F.e. what do you do when a segment arrives on listening socket w/o
> connection associated?
> If you sent reset, you could reset someone's valid connection (I mean
> this, saying "it cannot be right").
> If you do not, you violate protocol, those resets are required to
> clean up stale states.

When using anycast addresses with routing protocols, like e.g. anycast
bgp one cannot make the distinction in the kernel at all. Operators
currently depend on the BGP being stable enough so that the anycast
destination towards one server remains stable during the whole connection.

In case a flap happens, of course, packets can end up at a different server
and will get a RST. I guess this currently works well enough for anycast
network operators. Most DNS server use this model currently and DNS depends on
UDP and TCP being reachable for name resolution.

But this change does affect anycast IPv6 only in one subnet. If an anycast
address is allocated in the kernel, neighbour discovery logic tries as
hard to stick to one particular neighbour in the current subnet by always
sending neighbour advertisments with non-override flag. So in case we
need to reprobe the hardware address for the anycast neighbour, we will
get multiple answers and will pick that one which does not override the
current address. In case the anycast server really disappears from the
network, we could end up in a situation where the TCP packet gets send
to the wrong server which could result in a RST. IMHO this is acceptable,
but I may be wrong here. What is your opinion on that?

> >> Do not you think this must not be enabled for common use? At least
> >> some separate sysctl disabled by default.
> >
> > We could indeed use a sysctl, disabled by default.
> >
> > But my goal was to enable anycast address usage transparently, if possible.
> 
> IMHO it is logically impossible. That's why my first question was
> about a document which
> allowed anycasts with TCP. (BTW I found your answer unconvincing, at least.)
> Traditional anycasting used to be stateless and it was impossible to
> use with TCP
> because unsynchronized TCP connections would be unavoidable.
> AFAIK only suggestion from ancient RFC1546 could cure this problem,
> but it is very tricky and implies client knows that destination is anycast.
> 
> Actually, I do not even understand how the idea of use of anycast with TCP
> emerged. The only situation I can imagine: the router is expected to
> do full scale
> connection tracking and to bind connections to anycast destinations to
> correct nodes.
> (And the same moment you understand that the notion of anycast is
> completely lost
> in this picture: the same thing is done by load balancers in IPv4 for
> unicast addresses
> for ages) I still have no idea how it is going to work when both
> client and servers bound
> to anycasts are on the same link.
> 
> So, it looks like you need this sysctl to announce: "someone cares
> about proper routing
> on this network and tcp should work". It looks obvious this mode
> cannot be enabled by default.

IMHO it would be ok if a tcp socket binds to an anycast address, because the
administrator or software can check either for the pre-defined ones (the
subnet router anycast address) or the application allocated an anycast via
setsockopt specifically (the address could also be requested by another
process and the tcp socket could bind to it by accident, that might be a
problem. If we want to protect from this scenario we must add a new sysctl
knob.).

We don't use anycast addresses for normal source address selection, so if the
application does not specifically set the anycast address it will always use a
unicast one.

Greetings,

  Hannes

Re: [PATCH net-next] IPv6: enable TCP to use an anycast address

[David Miller]

From: Hannes Frederic Sowa <hannes@stressinduktion.org>
Date: Mon, 13 Jan 2014 02:11:46 +0100

> Hi!
> 
> On Sun, Jan 12, 2014 at 06:53:47AM -0800, François-Xavier Le Bail wrote:
>> On Sat, 1/11/14, Hannes Frederic Sowa <hannes@stressinduktion.org> wrote:
>> 
>> > On Sat, Jan 11, 2014 at 05:38:27PM +0400, Alexey Kuznetsov wrote:
>> > > On Sat, Jan 11, 2014 at 5:06 PM, François-Xavier Le Bail
>> > > <fx.lebail@yahoo.com> wrote:
>> > > > Many DNS root-servers use TCP with anycast (IPv4 and IPV6).
>> > > 
>> > > Actually, I was alerted by reset processing in your patch, it cannot be right.
>> > > 
>> > > Do not you think this must not be enabled for common use? At least
>> > > some separate sysctl disabled by default.
>> 
>> > The idea I had, was, that if a socket does knowingly bind to an anycast
>> > address, it is allowed to do so and process queries on it with both TCP and
>> > UDP. I don't think we need a sysctl for that? Anycast addresses are either
>> > pre-defined (e.g. the subnet router anycast address) or specified by a flag
>> > when the administrator adds one. Currently one can only add anycast addresses
>> > either by forwarding and gets the per-subnet anycast address or with a
>> > setsockopt IPV6_JOIN_ANYCAST.
>> 
>> > So the problem is what should be allowed when the socket listens on an any
>> > address? Maybe this should be protected by a sysctl?
>> 
>> TCP case:
>> With my two patches (the one for bind and this one for tcp), when a
>> SOCK_STREAM socket listen to in6addr_any, the server is able to 
>> send TCP reply with unicast or anycast source address, according
>> to the destination address used by the client.
>> 
>> dest request unicast => src reply unicast (current behavior)
>> dest resquet anycast => src reply anycast (new)
>> 
>> So, I don't think there is a need for a sysctl.
> 
> I am still thinking about the RST-case and am a bit unsure here. But I
> currently don't see any problems.

I think this needs much more discussion and analysis before I can really
seriously consider applying this, sorry.

Re: [PATCH net-next] IPv6: enable TCP to use an anycast address

[Hannes Frederic Sowa]

Hi!

On Sun, Jan 12, 2014 at 06:53:47AM -0800, François-Xavier Le Bail wrote:
> On Sat, 1/11/14, Hannes Frederic Sowa <hannes@stressinduktion.org> wrote:
> 
> > On Sat, Jan 11, 2014 at 05:38:27PM +0400, Alexey Kuznetsov wrote:
> > > On Sat, Jan 11, 2014 at 5:06 PM, François-Xavier Le Bail
> > > <fx.lebail@yahoo.com> wrote:
> > > > Many DNS root-servers use TCP with anycast (IPv4 and IPV6).
> > > 
> > > Actually, I was alerted by reset processing in your patch, it cannot be right.
> > > 
> > > Do not you think this must not be enabled for common use? At least
> > > some separate sysctl disabled by default.
> 
> > The idea I had, was, that if a socket does knowingly bind to an anycast
> > address, it is allowed to do so and process queries on it with both TCP and
> > UDP. I don't think we need a sysctl for that? Anycast addresses are either
> > pre-defined (e.g. the subnet router anycast address) or specified by a flag
> > when the administrator adds one. Currently one can only add anycast addresses
> > either by forwarding and gets the per-subnet anycast address or with a
> > setsockopt IPV6_JOIN_ANYCAST.
> 
> > So the problem is what should be allowed when the socket listens on an any
> > address? Maybe this should be protected by a sysctl?
> 
> TCP case:
> With my two patches (the one for bind and this one for tcp), when a
> SOCK_STREAM socket listen to in6addr_any, the server is able to 
> send TCP reply with unicast or anycast source address, according
> to the destination address used by the client.
> 
> dest request unicast => src reply unicast (current behavior)
> dest resquet anycast => src reply anycast (new)
> 
> So, I don't think there is a need for a sysctl.

I am still thinking about the RST-case and am a bit unsure here. But I
currently don't see any problems.

> UDP case:
> By default (no socket option), the server program don't know the
> destination address of the request. The ipv6_dev_get_saddr() is 
> used for choosing the unicast source address of the reply.
> 
> I am not sure a change is needed here.

If the socket did bind to anycast address, we may well send packets from
anycast source without switching a sysctl IMHO.

If not bound we need to keep current behaviour, yes. As DNS-Servers
normally do accept a list of addresses where to bind to, I find this
acceptable. RFC6724 additionally now allowes the selection of anycast
as source address, so we may think about this and maybe integrate this
with ip addrlabel and such).

We also need to check the implications to icmp here (error reporting if
an UDP packets is sent to an anycast address). Also we have to check for
icmps generated because of forwarding errors and their source selection.

> When using IPV6_RECVPKTINFO, a server is able to know
> the destination address of the request and can use it as source
> address for the reply.
> To enable anycast for this (don't get EINVAL), there is need for
> a patch like the one I posted ("IPv6: add option to use anycast
> addresses as source addresses for datagrams").
> I am working on a v2.

Ack.

> With the appropriate change:
> dest request unicast => src reply unicast (current behavior)
> dest resquet anycast => src reply anycast (new)
> 
> I don't think, there either, there is a need for a sysctl.

I agree here.

Greetings,

  Hannes

Re: [PATCH net-next] IPv6: enable TCP to use an anycast address

[François-Xavier Le Bail]

On Sat, 1/11/14, Hannes Frederic Sowa <hannes@stressinduktion.org> wrote:

> On Sat, Jan 11, 2014 at 05:38:27PM +0400, Alexey Kuznetsov wrote:
> > On Sat, Jan 11, 2014 at 5:06 PM, François-Xavier Le Bail
> > <fx.lebail@yahoo.com> wrote:
> > > Many DNS root-servers use TCP with anycast (IPv4 and IPV6).
> > 
> > Actually, I was alerted by reset processing in your patch, it cannot be right.
> > 
> > Do not you think this must not be enabled for common use? At least
> > some separate sysctl disabled by default.

> The idea I had, was, that if a socket does knowingly bind to an anycast
> address, it is allowed to do so and process queries on it with both TCP and
> UDP. I don't think we need a sysctl for that? Anycast addresses are either
> pre-defined (e.g. the subnet router anycast address) or specified by a flag
> when the administrator adds one. Currently one can only add anycast addresses
> either by forwarding and gets the per-subnet anycast address or with a
> setsockopt IPV6_JOIN_ANYCAST.

> So the problem is what should be allowed when the socket listens on an any
> address? Maybe this should be protected by a sysctl?

Hi,

TCP case:
With my two patches (the one for bind and this one for tcp), when a
SOCK_STREAM socket listen to in6addr_any, the server is able to 
send TCP reply with unicast or anycast source address, according
to the destination address used by the client.

dest request unicast => src reply unicast (current behavior)
dest resquet anycast => src reply anycast (new)

So, I don't think there is a need for a sysctl.

UDP case:
By default (no socket option), the server program don't know the
destination address of the request. The ipv6_dev_get_saddr() is 
used for choosing the unicast source address of the reply.

I am not sure a change is needed here.

When using IPV6_RECVPKTINFO, a server is able to know
the destination address of the request and can use it as source
address for the reply.
To enable anycast for this (don't get EINVAL), there is need for
a patch like the one I posted ("IPv6: add option to use anycast
addresses as source addresses for datagrams").
I am working on a v2.

With the appropriate change:
dest request unicast => src reply unicast (current behavior)
dest resquet anycast => src reply anycast (new)

I don't think, there either, there is a need for a sysctl.

What do you think about it?

BR,
Francois-Xavier

Re: [PATCH net-next] IPv6: enable TCP to use an anycast address

[Hannes Frederic Sowa]

Hi!

On Sat, Jan 11, 2014 at 05:38:27PM +0400, Alexey Kuznetsov wrote:
> On Sat, Jan 11, 2014 at 5:06 PM, François-Xavier Le Bail
> <fx.lebail@yahoo.com> wrote:
> > Many DNS root-servers use TCP with anycast (IPv4 and IPV6).
> >
> > see : http://tools.ietf.org/html/draft-jabley-dnsop-anycast-mapping-04#section-4
> >
> > "   L-Root service is provided using a single IPv4 address (199.7.83.42)
> >    and a single IPv6 address (2001:500:3::42).  It should be noted that
> >    it is preferable to refer to the service using its DNS name (L.ROOT-
> >    SERVERS.NET) rather than literal addresses, since addresses can
> >    change from time to time."
> 
> Is this all? It looks like this implies routing by deep packet inspection,
> fetching some creepy node identification options from inside DNS payload
> (not written directly, but  implied). This smells funky.
> 
> Actually, I was alerted by reset processing in your patch, it cannot be right.
> 
> Do not you think this must not be enabled for common use? At least
> some separate sysctl disabled by default.

RFC 4291 - IP Version 6 Addressing Architecture started to allow the use of
anycast addresses as source addresses.

This would be great to have DNS servers listening on them but they need to
respond to both, UDP and TCP.

The idea I had, was, that if a socket does knowingly bind to an anycast
address, it is allowed to do so and process queries on it with both TCP and
UDP. I don't think we need a sysctl for that? Anycast addresses are either
pre-defined (e.g. the subnet router anycast address) or specified by a flag
when the administrator adds one. Currently one can only add anycast addresses
either by forwarding and gets the per-subnet anycast address or with a
setsockopt IPV6_JOIN_ANYCAST.

So the problem is what should be allowed when the socket listens on an any
address? Maybe this should be protected by a sysctl?

Greetings,

  Hannes

Re: [PATCH net-next] IPv6: enable TCP to use an anycast address

[François-Xavier Le Bail]

On Sat, 1/11/14, Alexey Kuznetsov <kuznet@ms2.inr.ac.ru> wrote:

Hello,

> Actually, I was alerted by reset processing in your patch, it cannot be right.

Did I miss something, please explain why there is an error.

> Do not you think this must not be enabled for common use? At least
> some separate sysctl disabled by default.

Why not ? But if a patch can enable anycast case without break "normal" usage,
I think we can enable this by default. Those who don't need anycast will not use it.

Re: [PATCH net-next] IPv6: enable TCP to use an anycast address

[François-Xavier Le Bail]

On Sat, 1/11/14, Christoph Paasch <christoph.paasch@uclouvain.be> wrote:

>> On 11/01/14 - 05:06:10, François-Xavier Le Bail wrote:
>> On Sat, 1/11/14, Alexey Kuznetsov <kuznet@ms2.inr.ac.ru> wrote:
>> As said in http://tools.ietf.org/html/draft-iab-anycast-arch-implications-12.html#section-2.1
>> (this informational draft is in RCF queue)

> This draft does not say that TCP should be used with anycast
> addresses. Rather the opposite:

> "  This document does not treat in any depth
>   the fact that there are deployed services with TCP transport using
>   anycast today.  Evidence exists to suggest that such practice is not
>   "safe" in the traditional and architectural sense (as described in
>   Section 4.2). "

> And Section 4.2 gives more information.
> http://tools.ietf.org/html/draft-iab-anycast-arch-implications-12.html#section-4.2

The draft say there are use cases for TCP with anycast services.
The DNS use case is widely deployed in real word.

I think enabled such possibility in Linux kernel is useful.

It's only a possibility for such use cases and it's dont break usual TCP usage.

Francois-Xavier

Re: [PATCH net-next] IPv6: enable TCP to use an anycast address

[Christoph Paasch]

On 11/01/14 - 05:06:10, François-Xavier Le Bail wrote:
> On Sat, 1/11/14, Alexey Kuznetsov <kuznet@ms2.inr.ac.ru> wrote:
> As said in http://tools.ietf.org/html/draft-iab-anycast-arch-implications-12.html#section-2.1
> (this informational draft is in RCF queue)

This draft does not say that TCP should be used with anycast
addresses. Rather the opposite:

"  This document does not treat in any depth
   the fact that there are deployed services with TCP transport using
   anycast today.  Evidence exists to suggest that such practice is not
   "safe" in the traditional and architectural sense (as described in
   Section 4.2). "

And Section 4.2 gives more information.
http://tools.ietf.org/html/draft-iab-anycast-arch-implications-12.html#section-4.2


Christoph

> 
> "  No firm conclusion was reached regarding
>    use of TCP with anycasted services, but it was observed that
>    anycasting was useful for DNS, although it did introduce some new
>    complexities."
> 
> For this works, there is need for appropriate routing (BGP, ...).
> 
> Many DNS root-servers use TCP with anycast (IPv4 and IPV6).
> 
> see : http://tools.ietf.org/html/draft-jabley-dnsop-anycast-mapping-04#section-4
> 
> "   L-Root service is provided using a single IPv4 address (199.7.83.42)
>    and a single IPv6 address (2001:500:3::42).  It should be noted that
>    it is preferable to refer to the service using its DNS name (L.ROOT-
>    SERVERS.NET) rather than literal addresses, since addresses can
>    change from time to time."
> 
> François-Xavier Le Bail
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH net-next] IPv6: enable TCP to use an anycast address

[Alexey Kuznetsov]

Hello!

On Sat, Jan 11, 2014 at 5:06 PM, François-Xavier Le Bail
<fx.lebail@yahoo.com> wrote:
> Many DNS root-servers use TCP with anycast (IPv4 and IPV6).
>
> see : http://tools.ietf.org/html/draft-jabley-dnsop-anycast-mapping-04#section-4
>
> "   L-Root service is provided using a single IPv4 address (199.7.83.42)
>    and a single IPv6 address (2001:500:3::42).  It should be noted that
>    it is preferable to refer to the service using its DNS name (L.ROOT-
>    SERVERS.NET) rather than literal addresses, since addresses can
>    change from time to time."

Is this all? It looks like this implies routing by deep packet inspection,
fetching some creepy node identification options from inside DNS payload
(not written directly, but  implied). This smells funky.

Actually, I was alerted by reset processing in your patch, it cannot be right.

Do not you think this must not be enabled for common use? At least
some separate sysctl disabled by default.

Re: [PATCH net-next] IPv6: enable TCP to use an anycast address

[François-Xavier Le Bail]

On Sat, 1/11/14, Alexey Kuznetsov <kuznet@ms2.inr.ac.ru> wrote:

> Hello!

> I am just curious: since when is TCP allowed to use anycast addresses?

As said in http://tools.ietf.org/html/draft-iab-anycast-arch-implications-12.html#section-2.1
(this informational draft is in RCF queue)

"  No firm conclusion was reached regarding
   use of TCP with anycasted services, but it was observed that
   anycasting was useful for DNS, although it did introduce some new
   complexities."

For this works, there is need for appropriate routing (BGP, ...).

Many DNS root-servers use TCP with anycast (IPv4 and IPV6).

see : http://tools.ietf.org/html/draft-jabley-dnsop-anycast-mapping-04#section-4

"   L-Root service is provided using a single IPv4 address (199.7.83.42)
   and a single IPv6 address (2001:500:3::42).  It should be noted that
   it is preferable to refer to the service using its DNS name (L.ROOT-
   SERVERS.NET) rather than literal addresses, since addresses can
   change from time to time."

François-Xavier Le Bail

Re: [PATCH net-next] IPv6: enable TCP to use an anycast address

[Alexey Kuznetsov]

Hello!

I am just curious: since when is TCP allowed to use anycast addresses?

Alexey Kuznetsov

[PATCH net-next] IPv6: enable TCP to use an anycast address

[Francois-Xavier Le Bail]

- Use ipv6_anycast_destination() in tcp_v6_send_reset() and
  tcp_v6_conn_request().

Signed-off-by: Francois-Xavier Le Bail <fx.lebail@yahoo.com>
---
Tested with link-local and global anycast addresses.
Tested with SOCK_STREAM socket, bind and TCP traffic OK.

 net/ipv6/tcp_ipv6.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index ffd5fa8..85297b8 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -832,7 +832,8 @@ static void tcp_v6_send_reset(struct sock *sk, struct sk_buff *skb)
 	if (th->rst)
 		return;
 
-	if (!ipv6_unicast_destination(skb))
+	if (!ipv6_unicast_destination(skb) &&
+	    !ipv6_anycast_destination(skb))
 		return;
 
 #ifdef CONFIG_TCP_MD5SIG
@@ -963,7 +964,8 @@ static int tcp_v6_conn_request(struct sock *sk, struct sk_buff *skb)
 	if (skb->protocol == htons(ETH_P_IP))
 		return tcp_v4_conn_request(sk, skb);
 
-	if (!ipv6_unicast_destination(skb))
+	if (!ipv6_unicast_destination(skb) &&
+	    !ipv6_anycast_destination(skb))
 		goto drop;
 
 	if ((sysctl_tcp_syncookies == 2 ||