some trouble when using vrf

some trouble when using vrf

[linmiaohe]

Hi all:
    I meet some trouble when using vrf, it's very nice of you if you
can help me fix it.

    1.If I bind a vrf address before I set SO_BINDTODEVICE opts to bind an
enslaved device, "Cannot assign requested address" will occurs. It's
because only after we set SO_BINDTODEVICE opts to bind an enslaved device,
then we can lookup the route in the specified l3mdev domain.
    I think we couldn't limit the order of bind ip addr and SO_BINDTODEVICE opts
otherwise many vrf unaware apps many need to change their code.
    I can't find a convenient way to fix it.Here is an example:

    ...
    bind(sock_fd, (struct sockaddr *)&addr_serv, sizeof(addr_serv));
    ...
    ret = setsockopt(sock_fd, SOL_SOCKET, SO_BINDTODEVICE, bind_dev, strlen(bind_dev)+1);
    ...
    This code snipet doesn't work if the ip address of addr_serv is not in default vrf. And
"Cannot assign requested address" will occurs.

    2.When I run a udp server in default vrf and set net.ipv4.udp_l3mdev_accept=1,
it's supposed to work accorss VRFs.
    But it occurs when udp server received a packet, it sends back with the source
ip address from default vrf which is supposed to from the specified l3mdev domain.
And the connection between server and client is broken. It's because udp server
run in default vrf and doesn't bind the enslaved device, so only the route table in
default vrf is avaliable.

    For example,I run a udp server in default vrf and set net.ipv4.udp_l3mdev_accept=1:
    [root@localhost vrf]# sysctl -a | grep l3mdev
    net.ipv4.raw_l3mdev_accept = 1
    net.ipv4.tcp_l3mdev_accept = 1
    net.ipv4.udp_l3mdev_accept = 1

    Then connect the server and send a udp packet to the specified vrf. Here is tcpdump output:
    17:28:54.925417 IP 10.0.0.2.43003 > 10.0.0.1.irdmi: UDP, length 17
    17:28:54.925953 IP 9.85.153.236.irdmi > 10.0.0.2.43003: UDP, length 17
    ps. 10.0.0.2 is client ip, 10.0.0.1 is specified vrf ip, and 9.85.153.236 is default vrf ip.
    10.0.0.1 and 9.85.153.236 are in the same host.

Thanks a lot if you can help me fix these. Have a nice day.

Re: some trouble when using vrf

[Mike Manning]

On 16/04/2019 12:26, linmiaohe wrote:
> Hi all:
>     I meet some trouble when using vrf, it's very nice of you if you
> can help me fix it.
>
>     1.If I bind a vrf address before I set SO_BINDTODEVICE opts to bind an
> enslaved device, "Cannot assign requested address" will occurs. It's
> because only after we set SO_BINDTODEVICE opts to bind an enslaved device,
> then we can lookup the route in the specified l3mdev domain.
>     I think we couldn't limit the order of bind ip addr and SO_BINDTODEVICE opts
> otherwise many vrf unaware apps many need to change their code.
>     I can't find a convenient way to fix it.Here is an example:
>
>     ...
>     bind(sock_fd, (struct sockaddr *)&addr_serv, sizeof(addr_serv));
>     ...
>     ret = setsockopt(sock_fd, SOL_SOCKET, SO_BINDTODEVICE, bind_dev, strlen(bind_dev)+1);
>     ...
>     This code snipet doesn't work if the ip address of addr_serv is not in default vrf. And
> "Cannot assign requested address" will occurs.
Or you can run instances of vrf-unaware applications per VRF with 'sudo
ip vrf exec <vrf> <cmd>', with the 3 sysctl you indicate below set to 0.
>     2.When I run a udp server in default vrf and set net.ipv4.udp_l3mdev_accept=1,
> it's supposed to work accorss VRFs.
>     But it occurs when udp server received a packet, it sends back with the source
> ip address from default vrf which is supposed to from the specified l3mdev domain.
> And the connection between server and client is broken. It's because udp server
> run in default vrf and doesn't bind the enslaved device, so only the route table in
> default vrf is avaliable.
>
>     For example,I run a udp server in default vrf and set net.ipv4.udp_l3mdev_accept=1:
>     [root@localhost vrf]# sysctl -a | grep l3mdev
>     net.ipv4.raw_l3mdev_accept = 1
>     net.ipv4.tcp_l3mdev_accept = 1
>     net.ipv4.udp_l3mdev_accept = 1
>
>     Then connect the server and send a udp packet to the specified vrf. Here is tcpdump output:
>     17:28:54.925417 IP 10.0.0.2.43003 > 10.0.0.1.irdmi: UDP, length 17
>     17:28:54.925953 IP 9.85.153.236.irdmi > 10.0.0.2.43003: UDP, length 17
>     ps. 10.0.0.2 is client ip, 10.0.0.1 is specified vrf ip, and 9.85.153.236 is default vrf ip.
>     10.0.0.1 and 9.85.153.236 are in the same host.
>
> Thanks a lot if you can help me fix these. Have a nice day.
>
>
>

Re: some trouble when using vrf

[linmiaohe]

On 2019/4/16 20:11, Mike Manning wrote:
> On 16/04/2019 12:26, linmiaohe wrote:
>> Hi all:
>>     I meet some trouble when using vrf, it's very nice of you if you
>> can help me fix it.
>>
>>     1.If I bind a vrf address before I set SO_BINDTODEVICE opts to bind an
>> enslaved device, "Cannot assign requested address" will occurs. It's
>> because only after we set SO_BINDTODEVICE opts to bind an enslaved device,
>> then we can lookup the route in the specified l3mdev domain.
>>     I think we couldn't limit the order of bind ip addr and SO_BINDTODEVICE opts
>> otherwise many vrf unaware apps many need to change their code.
>>     I can't find a convenient way to fix it.Here is an example:
>>
>>     ...
>>     bind(sock_fd, (struct sockaddr *)&addr_serv, sizeof(addr_serv));
>>     ...
>>     ret = setsockopt(sock_fd, SOL_SOCKET, SO_BINDTODEVICE, bind_dev, strlen(bind_dev)+1);
>>     ...
>>     This code snipet doesn't work if the ip address of addr_serv is not in default vrf. And
>> "Cannot assign requested address" will occurs.
> Or you can run instances of vrf-unaware applications per VRF with 'sudo
> ip vrf exec <vrf> <cmd>', with the 3 sysctl you indicate below set to 0.
>>     2.When I run a udp server in default vrf and set net.ipv4.udp_l3mdev_accept=1,
>> it's supposed to work accorss VRFs.
>>     But it occurs when udp server received a packet, it sends back with the source
>> ip address from default vrf which is supposed to from the specified l3mdev domain.
>> And the connection between server and client is broken. It's because udp server
>> run in default vrf and doesn't bind the enslaved device, so only the route table in
>> default vrf is avaliable.
>>
>>     For example,I run a udp server in default vrf and set net.ipv4.udp_l3mdev_accept=1:
>>     [root@localhost vrf]# sysctl -a | grep l3mdev
>>     net.ipv4.raw_l3mdev_accept = 1
>>     net.ipv4.tcp_l3mdev_accept = 1
>>     net.ipv4.udp_l3mdev_accept = 1
>>
>>     Then connect the server and send a udp packet to the specified vrf. Here is tcpdump output:
>>     17:28:54.925417 IP 10.0.0.2.43003 > 10.0.0.1.irdmi: UDP, length 17
>>     17:28:54.925953 IP 9.85.153.236.irdmi > 10.0.0.2.43003: UDP, length 17
>>     ps. 10.0.0.2 is client ip, 10.0.0.1 is specified vrf ip, and 9.85.153.236 is default vrf ip.
>>     10.0.0.1 and 9.85.153.236 are in the same host.
>>
>> Thanks a lot if you can help me fix these. Have a nice day.
>>
>>
>>
> 
> 
> .
> 
    Thanks for your reply. Run instances of vrf-unaware applications per VRF is an optional solution for problem 1.
But maybe it will limits the applications supposed to accross VRFS can't coexist with per VRF applications as
udp_l3mdev_accept is globally effective.
    And what about problem 2?
    Thanks again.

Re: some trouble when using vrf

[David Ahern]

On 4/16/19 5:26 AM, linmiaohe wrote:
> Hi all:
>     I meet some trouble when using vrf, it's very nice of you if you
> can help me fix it.
> 
>     1.If I bind a vrf address before I set SO_BINDTODEVICE opts to bind an
> enslaved device, "Cannot assign requested address" will occurs. It's
> because only after we set SO_BINDTODEVICE opts to bind an enslaved device,
> then we can lookup the route in the specified l3mdev domain.
>     I think we couldn't limit the order of bind ip addr and SO_BINDTODEVICE opts
> otherwise many vrf unaware apps many need to change their code.
>     I can't find a convenient way to fix it.Here is an example:
> 
>     ...
>     bind(sock_fd, (struct sockaddr *)&addr_serv, sizeof(addr_serv));
>     ...
>     ret = setsockopt(sock_fd, SOL_SOCKET, SO_BINDTODEVICE, bind_dev, strlen(bind_dev)+1);
>     ...
>     This code snipet doesn't work if the ip address of addr_serv is not in default vrf. And
> "Cannot assign requested address" will occurs.

Applications must bind to the device first and then bind to the local
address. As I recall this applies regardless of whether it is a VRF or
other.


> 
>     2.When I run a udp server in default vrf and set net.ipv4.udp_l3mdev_accept=1,
> it's supposed to work accorss VRFs.
>     But it occurs when udp server received a packet, it sends back with the source
> ip address from default vrf which is supposed to from the specified l3mdev domain.
> And the connection between server and client is broken. It's because udp server
> run in default vrf and doesn't bind the enslaved device, so only the route table in
> default vrf is avaliable.

UDP applications can use cmsg with IP_PKTINFO to send the reply out the
same device as the request was received. An app can also track the
relationship of the enslaved device to a VRF to use a VRF based lookup
to send a response. Choices.


> 
>     For example,I run a udp server in default vrf and set net.ipv4.udp_l3mdev_accept=1:
>     [root@localhost vrf]# sysctl -a | grep l3mdev
>     net.ipv4.raw_l3mdev_accept = 1
>     net.ipv4.tcp_l3mdev_accept = 1
>     net.ipv4.udp_l3mdev_accept = 1
> 
>     Then connect the server and send a udp packet to the specified vrf. Here is tcpdump output:
>     17:28:54.925417 IP 10.0.0.2.43003 > 10.0.0.1.irdmi: UDP, length 17
>     17:28:54.925953 IP 9.85.153.236.irdmi > 10.0.0.2.43003: UDP, length 17
>     ps. 10.0.0.2 is client ip, 10.0.0.1 is specified vrf ip, and 9.85.153.236 is default vrf ip.
>     10.0.0.1 and 9.85.153.236 are in the same host.
> 
> Thanks a lot if you can help me fix these. Have a nice day.
> 
> 
> 

Re: some trouble when using vrf

[linmiaohe]

On 2019/4/16 23:20, David Ahern wrote:
> On 4/16/19 5:26 AM, linmiaohe wrote:
>>     ...
>>     bind(sock_fd, (struct sockaddr *)&addr_serv, sizeof(addr_serv));
>>     ...
>>     ret = setsockopt(sock_fd, SOL_SOCKET, SO_BINDTODEVICE, bind_dev, strlen(bind_dev)+1);
>>     ...
>>     This code snipet doesn't work if the ip address of addr_serv is not in default vrf. And
>> "Cannot assign requested address" will occurs.
> 
> Applications must bind to the device first and then bind to the local
> address. As I recall this applies regardless of whether it is a VRF or
> other.
> 
> 
It should be, but unfortunately some applications break this.

>>
>>     2.When I run a udp server in default vrf and set net.ipv4.udp_l3mdev_accept=1,
>> it's supposed to work accorss VRFs.
> 
> UDP applications can use cmsg with IP_PKTINFO to send the reply out the
> same device as the request was received. An app can also track the
> relationship of the enslaved device to a VRF to use a VRF based lookup
> to send a response. Choices.
> 
> 

UDP applications using cmsg with IP_PKTINFO is really a good idea.

>>
>> Thanks a lot if you can help me fix these. Have a nice day.
>>
> .
> 

Thanks a lot for your advice.

Re: some trouble when using vrf

[David Ahern]

On 4/16/19 8:22 PM, linmiaohe wrote:
> 
> 
> On 2019/4/16 23:20, David Ahern wrote:
>> On 4/16/19 5:26 AM, linmiaohe wrote:
>>>     ...
>>>     bind(sock_fd, (struct sockaddr *)&addr_serv, sizeof(addr_serv));
>>>     ...
>>>     ret = setsockopt(sock_fd, SOL_SOCKET, SO_BINDTODEVICE, bind_dev, strlen(bind_dev)+1);
>>>     ...
>>>     This code snipet doesn't work if the ip address of addr_serv is not in default vrf. And
>>> "Cannot assign requested address" will occurs.
>>
>> Applications must bind to the device first and then bind to the local
>> address. As I recall this applies regardless of whether it is a VRF or
>> other.
>>
>>
> It should be, but unfortunately some applications break this.
> 

Sure, I have an open item on my to-do list about re-verifying bound
addresses when the device binding changes (and vice versa). This is not
a problem VRF introduced, but VRF use case does bring more attention to it.

Re: some trouble when using vrf

[linmiaohe]

On 2019/4/17 12:56, David Ahern wrote:
> On 4/16/19 8:22 PM, linmiaohe wrote:
>>
>>
>> On 2019/4/16 23:20, David Ahern wrote:
>>> On 4/16/19 5:26 AM, linmiaohe wrote:
>>>>     ...
>>>>     bind(sock_fd, (struct sockaddr *)&addr_serv, sizeof(addr_serv));
>>>>     ...
>>>>     ret = setsockopt(sock_fd, SOL_SOCKET, SO_BINDTODEVICE, bind_dev, strlen(bind_dev)+1);
>>>>     ...
>>>>     This code snipet doesn't work if the ip address of addr_serv is not in default vrf. And
>>>> "Cannot assign requested address" will occurs.
>>>
>>> Applications must bind to the device first and then bind to the local
>>> address. As I recall this applies regardless of whether it is a VRF or
>>> other.
>>>
>>>
>> It should be, but unfortunately some applications break this.
>>
> 
> Sure, I have an open item on my to-do list about re-verifying bound
> addresses when the device binding changes (and vice versa). This is not
> a problem VRF introduced, but VRF use case does bring more attention to it.
> 
> 

Cherrs! I can't wait to enjoy the source code of the item. Please let
me know if you work out it. Thanks a lot.