Possible bug: Association always uses the primary source address in replies

Possible bug: Association always uses the primary source address in replies

[Martin Schröder]

Hi,
we have a server and a client. Both are connected to two different networks.
Betweeen these four networks there is a complex routing structure.
Due to an outage we have an outage from the primary interface of the server
to all networks of the client; there is still a route with the secondary
interface.

And then this happens:
1. The server is listening, the client is starting.
2. Client begins to start Init to the reachable (secondary) interface of
   the server
3. Server answers with Init ACK from his *primary* interface, which of
   course does not reach the client

Why does the server not answer on his secondary interface when there is
still a route on this interface to the client but uses the primary interface
which has no route?

Tested with 4.9 and 4.18.rc1.

Best
   Martin

Re: Possible bug: Association always uses the primary source address in replies

[Xin Long]

On Wed, Jul 4, 2018 at 9:09 PM, Martin Schröder <martin@oneiros.de> wrote:
> Hi,
> we have a server and a client. Both are connected to two different networks.
> Betweeen these four networks there is a complex routing structure.
> Due to an outage we have an outage from the primary interface of the server
> to all networks of the client; there is still a route with the secondary
> interface.
>
> And then this happens:
> 1. The server is listening, the client is starting.
> 2. Client begins to start Init to the reachable (secondary) interface of
>    the server
> 3. Server answers with Init ACK from his *primary* interface, which of
>    course does not reach the client
>
> Why does the server not answer on his secondary interface when there is
> still a route on this interface to the client but uses the primary interface
> which has no route?
This is up to the src IP of your INIT packet, which INIT_ACK from server
will use as the primary transport path to select the out route.

Can you provide the interface addresses, the INIT and INIT ACK
packet info, and the route info on the server? Thanks.


>
> Tested with 4.9 and 4.18.rc1.
>
> Best
>    Martin
> --
> To unsubscribe from this list: send the line "unsubscribe linux-sctp" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Possible bug: Association always uses the primary source address in replies

[Martin Schröder]

[-- Attachment #1: Type: text/plain, Size: 3995 bytes --]

2018-07-04 16:19 GMT+02:00 Xin Long <lucien.xin@gmail.com>:
> This is up to the src IP of your INIT packet, which INIT_ACK from server
> will use as the primary transport path to select the out route.
>
> Can you provide the interface addresses, the INIT and INIT ACK
> packet info, and the route info on the server? Thanks.

-----------------
MSrv1> ip rule show
0: from all lookup local
32764: from 195.219.193.96/27 to 178.23.31.0/24 lookup secondary
32765: from 195.219.193.64/27 to 178.23.31.0/24 lookup primary
32766: from all lookup main
32767: from all lookup default

MSrv1>ip r s t all
178.23.31.0/24 via 195.219.193.65 dev eth1 table primary
178.23.31.0/24 via 195.219.193.97 dev eth0 table secondary
default via 195.219.193.65 dev eth1 onlink
10.3.4.0/24 via 192.168.148.1 dev eth2
10.3.8.3 via 192.168.148.1 dev eth2
10.3.8.5 via 192.168.148.1 dev eth2
10.3.192.0/22 via 192.168.148.1 dev eth2
10.113.0.0/24 via 192.168.148.1 dev eth2
192.168.148.0/22 dev eth2 proto kernel scope link src 192.168.151.184
195.219.193.64/27 dev eth1 proto kernel scope link src 195.219.193.90
195.219.193.96/27 dev eth0 proto kernel scope link src 195.219.193.122
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link
src 127.0.0.1
broadcast 192.168.148.0 dev eth2 table local proto kernel scope link
src 192.168.151.184
local 192.168.151.184 dev eth2 table local proto kernel scope host src
192.168.151.184
broadcast 192.168.151.255 dev eth2 table local proto kernel scope link
src 192.168.151.184
broadcast 195.219.193.64 dev eth1 table local proto kernel scope link
src 195.219.193.90
local 195.219.193.90 dev eth1 table local proto kernel scope host src
195.219.193.90
local 195.219.193.91 dev eth1 table local proto kernel scope host src
195.219.193.90
broadcast 195.219.193.95 dev eth1 table local proto kernel scope link
src 195.219.193.90
broadcast 195.219.193.96 dev eth0 table local proto kernel scope link
src 195.219.193.122
local 195.219.193.122 dev eth0 table local proto kernel scope host src
195.219.193.122
local 195.219.193.123 dev eth0 table local proto kernel scope host src
195.219.193.122
broadcast 195.219.193.127 dev eth0 table local proto kernel scope link
src 195.219.193.122
fe80::/64 dev eth1 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth2 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
local fe80::5054:ff:fe6b:271a dev eth0 table local proto kernel metric
0 pref medium
local fe80::5054:ff:fe97:127 dev eth2 table local proto kernel metric
0 pref medium
local fe80::5054:ff:fed0:4530 dev eth1 table local proto kernel metric
0 pref medium
ff00::/8 dev eth1 table local metric 256 pref medium
ff00::/8 dev eth0 table local metric 256 pref medium
ff00::/8 dev eth2 table local metric 256 pref medium
-----------------

I've attached a pcap file and a graphic of the network.

Our situation is that the gateway 195.219.193.65/27 is reachable but doesn't
forward packets (e.g. because of defect). We then try to establish a connection.
The init packets come from the client and reach the address 195.219.193.122/27,
but the server answers with one init ack from the address 195.219.193.90/27.
After that inits arrive on the server but no acks are sent.

After packet number 8 we shut down the local interface with address
195.219.193.90/27. And then the init ack is sent with the correct address
195.219.193.122/27.

It seems that if a server is bound with two addresses and for one address
all routes are down the server doesn't use the other source address.
Is this the correct behaviour?

Best
   Martin

PS: Personal note: I'm only the proxy here for Ralf who is the developer but
    whose english is not so good.

[-- Attachment #2: example.pcap --]
[-- Type: application/vnd.tcpdump.pcap, Size: 4860 bytes --]

[-- Attachment #3: network.pdf --]
[-- Type: application/pdf, Size: 41902 bytes --]

Re: Possible bug: Association always uses the primary source address in replies

[Xin Long]

On Thu, Jul 5, 2018 at 9:17 PM, Martin Schröder <martin@oneiros.de> wrote:
> 2018-07-04 16:19 GMT+02:00 Xin Long <lucien.xin@gmail.com>:
>> This is up to the src IP of your INIT packet, which INIT_ACK from server
>> will use as the primary transport path to select the out route.
>>
>> Can you provide the interface addresses, the INIT and INIT ACK
>> packet info, and the route info on the server? Thanks.
>
> -----------------
> MSrv1> ip rule show
> 0: from all lookup local
> 32764: from 195.219.193.96/27 to 178.23.31.0/24 lookup secondary
> 32765: from 195.219.193.64/27 to 178.23.31.0/24 lookup primary
> 32766: from all lookup main
> 32767: from all lookup default
>
> MSrv1>ip r s t all
> 178.23.31.0/24 via 195.219.193.65 dev eth1 table primary
> 178.23.31.0/24 via 195.219.193.97 dev eth0 table secondary
Oh, I got your problem (thanks for your topo file attached):

You were trying to make it be 4 paths, while it's actually 2 paths to SCTP.
From the server side:

  Path1: local IP A -> 178.23.31.30/25
  Path2: local IP B -> 178.23.31.130/25

Then SCTP server will choose local IP according to your routes and out dev
IPs.

After getting the INIT packet with src IP 178.23.31.130, the server would
use this IP as the dst IP of INIT_ACK packet and look up a route:

  # ip route get 178.23.31.130
  178.23.31.0/24 via 195.219.193.65 dev eth1 table primary [1]

I guess route [1] was chosen, then got the out dev 'eth1' from this route.
And also from eth1, 195.219.193.90 was set as the src IP (195.219.193.90)
of INIT_ACK packet

So it means your 'ip rule' never worked as you expected. (Check the packets
in the pcap file, it's impossible that a lookup would match your ip_rules)


Pls try like this, make it 2 paths by removing Path A2 and Path B1 on your
client route side, the paths left are:

  195.219.193.90/27  <--(195.219.193.65)--> 178.23.31.30/25
  195.219.193.122/27 <--(195.219.193.97)--> 178.23.31.130/25

Then on the server side, change to add the routes:

  # ip route add 178.23.31.30/25  via 195.219.193.65 dev eth1 [a]
  # ip route add 178.23.31.130/25 via 195.219.193.97 dev eth0 [b]

With this setup, it will work like this:

The client tries to connect to server by sending INIT packet to the IP
178.23.31.90 when the route (195.219.193.65/27) is down. After failing,
the client then changes to send it to the IP 178.23.31.122, and it will
arrive the server through the route (195.219.193.97).

After the server gets the INIT packet whose src IP must be
178.23.31.130, and the route [b] with out dev eth0 will be chosen when
looking up a route for INIT_ACK packet, which will get out from eth0
with the src IP 195.219.193.122.

Make sense?

Re: Possible bug: Association always uses the primary source address in replies

[Martin Schröder]

2018-07-06 18:34 GMT+02:00 Xin Long <lucien.xin@gmail.com>:
> Make sense?

Yes, but it doesn't make us happy. :-)

So we can get our setup working (but will be able to test that only in
some weeks).

The biggest problem for us is that we only control one side and can not
really prevent such a cross connection.

Btw: Why is there a primary address (local) AND a primary peer address
(remote) if only one path is possible as primary and the local and the
remote address is fixed?

Many many thanks for your help

Best
   Martin

Re: Possible bug: Association always uses the primary source address in replies

[Xin Long]

On Mon, Jul 9, 2018 at 7:38 PM, Martin Schröder <martin@oneiros.de> wrote:
> 2018-07-06 18:34 GMT+02:00 Xin Long <lucien.xin@gmail.com>:
>> Make sense?
>
> Yes, but it doesn't make us happy. :-)
>
> So we can get our setup working (but will be able to test that only in
> some weeks).
>
> The biggest problem for us is that we only control one side and can not
> really prevent such a cross connection.
OK, I understand what you really want. But I think it should be
something from the route, like some route's multipath policy that
could detect one route gw failure and change to another gw.
SCTP should not decide the route's out dev according to the
incoming INIT packet only for "INIT_ACK". Sorry about that.

>
> Btw: Why is there a primary address (local) AND a primary peer address
> (remote) if only one path is possible as primary and the local and the
> remote address is fixed?
There's no so-called "a primary local", but only "a primary peer" (remote)
address named "a primary path or transport", the local address is always
selected among the bind_addr_list, but the one also on the outdev of the
route has the higher priority to be used as the src IP of the outgoing
packets like INIT_ACK.

Like, it got the route for the path "178.23.31.130":
  178.23.31.130/25 via 195.219.193.97 dev eth0 [A]

In the bind_addr_list, there are two addrs:
  195.219.193.90, 195.219.193.122

If 195.219.193.122 is on eth0, which is the outdev of route [A],
195.219.193.122 will be select for this path's src IP.