gateway field missing in netlink message

gateway field missing in netlink message

[Magesh M P]

Hi

I am trying to configure dual gateways with ip route command.

Ip route show command shows the dual gateway information.

I got a vpp stack that is running. The communication of route entries between Linux kernel and vpp stack is through netlink messages.

On parsing the netlink message for the route entry with dual gateways, we see that the message carries only single gateway. Is this a known bug ? Please suggest a solution to resolve this.

Regards
Magesh

Re: gateway field missing in netlink message

[David Ahern]

On Wed, May 04, 2022 at 06:46:05AM +0000, Magesh  M P wrote:
> Hi
> 
> I am trying to configure dual gateways with ip route command.
> 
> Ip route show command shows the dual gateway information.
> 
> I got a vpp stack that is running. The communication of route entries between Linux kernel and vpp stack is through netlink messages.
> 
> On parsing the netlink message for the route entry with dual gateways, we see that the message carries only single gateway. Is this a known bug ? Please suggest a solution to resolve this.
> 

If `ip route show` lists a multipath route, the bug is in your app. Are
you handling RTA_MULTIPATH attribute?

Re: gateway field missing in netlink message

[Magesh M P]

Hi Dave,

Thanks for responding.

The librtnl/netns.c contains the parser code as below which parses the MULTIPATH attribute. Could you please take a look at the code and see if anything is wrong ?
static int
rtnl_parse_rtattr(struct rtattr *db[], size_t max,
                  struct rtattr *rta, size_t len) {
int attrlen;
  for(; RTA_OK(rta, len); rta = RTA_NEXT(rta, len)) {
    if (rta->rta_type <= max){
                clib_warning("rattr type [%d] ",rta->rta_type);
         if (rta->rta_type == RTA_TABLE) {
           unsigned int rtable=*(unsigned int*)RTA_DATA(rta);
            clib_warning("RTA_RTABLE type rtable %x\n",rtable);
           }
         if (rta->rta_type == RTA_OIF) {
           unsigned int oif=*(unsigned int*)RTA_DATA(rta);
            clib_warning("RTA_OIF type oif %x\n",oif);
           }
         if (rta->rta_type == RTA_GATEWAY) {
           unsigned int gw=*(unsigned int*)RTA_DATA(rta);
            clib_warning("RTA_GATEWAY type gateway %x\n",gw);
           }
         if (rta->rta_type == RTA_DST) {
           unsigned int destip=*(unsigned int*)RTA_DATA(rta);
            clib_warning("RTA_DST type destip %x\n",destip);
           }
         if (rta->rta_type == RTA_MULTIPATH){
          struct rtnexthop *nhptr = (struct rtnexthop*)RTA_DATA(rta);
          int rtnhp_len = RTA_PAYLOAD(rta);
                 clib_warning("RTA_MULTIPATH type\n");
           if (rtnhp_len < (int) sizeof (*nhptr) ||
              nhptr->rtnh_len > rtnhp_len){
                continue;
               }
          attrlen = rtnhp_len - sizeof(struct rtnexthop);
           if (attrlen) {
              struct rtattr *attr = RTNH_DATA(nhptr);
                for(; RTA_OK(attr,attrlen);attr=RTA_NEXT(attr, attrlen)){
                 clib_warning("attr->rta_type %d\n",attr->rta_type);
                 if (attr->rta_type == RTA_GATEWAY) {
                 unsigned int nh= *(unsigned int*) RTA_DATA(attr);
                 clib_warning("gateway %x\n",nh);
                  }
                }
             }
          }
      db[rta->rta_type] = rta;
        }
#ifdef RTNL_CHECK
    else
      clib_warning("RTA type too high: %d", rta->rta_type);
#endif
  }

  if(len) {
    clib_warning("rattr lenght mistmatch %d %d len",
                 (int) len, (int) rta->rta_len);
    return -1;
  }
  return 0;
}



From: David Ahern <dsahern@kernel.org>
Sent: Thursday, May 5, 2022 4:01 AM
To: Magesh M P <magesh@digitizethings.com>
Cc: netdev@vger.kernel.org <netdev@vger.kernel.org>
Subject: Re: gateway field missing in netlink message 
 
On Wed, May 04, 2022 at 06:46:05AM +0000, Magesh  M P wrote:
> Hi
> 
> I am trying to configure dual gateways with ip route command.
> 
> Ip route show command shows the dual gateway information.
> 
> I got a vpp stack that is running. The communication of route entries between Linux kernel and vpp stack is through netlink messages.
> 
> On parsing the netlink message for the route entry with dual gateways, we see that the message carries only single gateway. Is this a known bug ? Please suggest a solution to resolve this.
> 

If `ip route show` lists a multipath route, the bug is in your app. Are
you handling RTA_MULTIPATH attribute?

Re: gateway field missing in netlink message

[Stephen Hemminger]

On Thu, 5 May 2022 03:43:45 +0000
"Magesh  M P" <magesh@digitizethings.com> wrote:

> Hi Dave,
> 
> Thanks for responding.
> 
> The librtnl/netns.c contains the parser code as below which parses the MULTIPATH attribute. Could you please take a look at the code and see if anything is wrong ?

That won't support IPv6.

Re: gateway field missing in netlink message

[Magesh M P]

 
Hi Steve/Dave,
 
Thank you very much for great support and sharing the knowledge.
 
Dave referred me the file iproute.c but this contains the flow from user space to kernel.
 
In my problematic scenario, the use case is from Linux kernel to VPP kernel and this is where my parser at vpp kernel is not doing its work properly. The VPP stack tries to synchronize the routing table information from Linux kernel through netlink messages.
 
My file netns.c looks like the following:
vpp-netlink/netns.c at master · Oryon/vpp-netlink · GitHub
 
I tried to implement the parser logic as shown in the following link but that did not help.
Parsing the RTA_MULTIPATH attribute from Rtnetlink | Eder L. Fernandes (ederlf.website)
 
I see only one gateway in parsing the RTA_MULTIPATH attribute inspite when I had configured dual gateways.
 
The existing code in netns.c works fine for single gateway ip route configuration and it fails in dual gateway ECMP case. 
 
Your help is greatly appreciated.
 
Best regards
Magesh
 
 
Sent from Mail for Windows
 
From: Stephen Hemminger
Sent: 05 May 2022 21:58
To: Magesh M P
Cc: David Ahern
Subject: Re: gateway field missing in netlink message
 
Have you considered using libmnl it is good way to handle the
low level parts of netlink parsing.

https://www.netfilter.org/projects/libmnl/
 

Re: gateway field missing in netlink message

[David Ahern]

On 5/9/22 12:59 AM, Magesh M P wrote:
> 
> 
> 
> 
>  
> Hi Steve/Dave,
>  
> Thank you very much for great support and sharing the knowledge.
>  
> Dave referred me the file iproute.c but this contains the flow from user space to kernel.

the reference is to a print function - dumping the state of the route
received from the kernel. It's caller parsed the message. The 2 have
everything you need to understand the message getting sent.

Re: gateway field missing in netlink message

[Magesh M P]

 
Hi Steve/Dave
 
Could you please confirm that VPP during synchronization of routing table with Linux kernel in case of dual gateway ECMP configuration gets only single route in the netlink message is a known bug ?? 
 
I am using VPP 21.06 version.
 
Regards
Magesh 
 
 
  
From: Stephen Hemminger
Sent: 09 May 2022 20:35
To: Magesh M P
Cc: David Ahern
Subject: Re: gateway field missing in netlink message
 
On Mon, 9 May 2022 06:55:46 +0000
"Magesh  M P" <magesh@digitizethings.com> wrote:

> Hi Steve/Dave,
> 
> Thank you very much for great support and sharing the knowledge.
> 
> Dave referred me the file iproute.c but this contains the flow from user space to kernel.
> 
> In my problematic scenario, the use case is from Linux kernel to VPP kernel and this is where my parser at vpp kernel is not doing its work properly. The VPP stack tries to synchronize the routing table information from Linux kernel through netlink messages.
> 
> My file netns.c looks like the following:
> vpp-netlink/netns.c at master · Oryon/vpp-netlink · GitHub<https://github.com/Oryon/vpp-netlink/blob/master/librtnl/netns.c>
> 
> I tried to implement the parser logic as shown in the following link but that did not help.
> Parsing the RTA_MULTIPATH attribute from Rtnetlink | Eder L. Fernandes (ederlf.website)<https://ederlf.website/post/netlink-multipath/>
> 
> I see only one gateway in parsing the RTA_MULTIPATH attribute inspite when I had configured dual gateways.
> 
> The existing code in netns.c works fine for single gateway ip route configuration and it fails in dual gateway ECMP case.
> 
> Your help is greatly appreciated.
> 
> Best regards
> Magesh
> 
> 
> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows
> 
> From: Stephen Hemminger<mailto:stephen@networkplumber.org>
> Sent: 05 May 2022 21:58
> To: Magesh M P<mailto:magesh@digitizethings.com>
> Cc: David Ahern<mailto:dsahern@gmail.com>
> Subject: Re: gateway field missing in netlink message
> 
> Have you considered using libmnl it is good way to handle the
> low level parts of netlink parsing.
> 
> https://www.netfilter.org/projects/libmnl/


Have you looked at other routing projects that handle multipath like:

https://github.com/FRRouting/frr/blob/master/zebra/rt_netlink.c

 

Re: gateway field missing in netlink message

[Stephen Hemminger]

On Wed, 11 May 2022 05:35:21 +0000
"Magesh  M P" <magesh@digitizethings.com> wrote:

>  
> Hi Steve/Dave
>  
> Could you please confirm that VPP during synchronization of routing table with Linux kernel in case of dual gateway ECMP configuration gets only single route in the netlink message is a known bug ?? 
>  
> I am using VPP 21.06 version.
>  
> Regards
> Magesh 
>  

I don't work on VPP.

There is no kernel bug here.

Re: gateway field missing in netlink message

[David Ahern]

On 5/11/22 9:56 AM, Stephen Hemminger wrote:
> On Wed, 11 May 2022 05:35:21 +0000
> "Magesh  M P" <magesh@digitizethings.com> wrote:
> 
>>  
>> Hi Steve/Dave
>>  
>> Could you please confirm that VPP during synchronization of routing table with Linux kernel in case of dual gateway ECMP configuration gets only single route in the netlink message is a known bug ?? 
>>  
>> I am using VPP 21.06 version.
>>  
>> Regards
>> Magesh 
>>  
> 
> I don't work on VPP.
> 
> There is no kernel bug here.

 +1

if there was a bug, FRR would not work. Bird would not work. iproute2
would not show multipath routes, etc. That's why we gave you references
to known working code.

Re: gateway field missing in netlink message

[Stephen Hemminger]

On Thu, 5 May 2022 03:43:45 +0000
"Magesh  M P" <magesh@digitizethings.com> wrote:

> The librtnl/netns.c contains the parser code as below which parses the MULTIPATH attribute. Could you please take a look at the code and see if anything is wrong ?

Also assuming byte order and assuming sizeof(unsigned int) == sizeof(uint32_t) is likely
to lead you astray.

RE: gateway field missing in netlink message

[Magesh M P]

 Hi Steve

The netlink parser code is checking only for IPv4 messages
 
Sent from Mail for Windows
 

From: Stephen Hemminger <stephen@networkplumber.org>
Sent: Thursday, May 5, 2022 9:20:23 AM
To: Magesh M P <magesh@digitizethings.com>
Cc: David Ahern <dsahern@kernel.org>; netdev@vger.kernel.org <netdev@vger.kernel.org>
Subject: Re: gateway field missing in netlink message 
 
On Thu, 5 May 2022 03:43:45 +0000
"Magesh  M P" <magesh@digitizethings.com> wrote:

> The librtnl/netns.c contains the parser code as below which parses the MULTIPATH attribute. Could you please take a look at the code and see if anything is wrong ?

Also assuming byte order and assuming sizeof(unsigned int) == sizeof(uint32_t) is likely
to lead you astray.

Re: gateway field missing in netlink message

[David Ahern]

On 5/4/22 8:43 PM, Magesh M P wrote:
> Hi Dave,
> 
> Thanks for responding.
> 
> The librtnl/netns.c contains the parser code as below which parses the MULTIPATH attribute. Could you please take a look at the code and see if anything is wrong ?

Stephen gave you a few comments. In addition to that rather than me
reviewing your code for errors, here is a reference to iproute2 code
that is known to work:

https://git.kernel.org/pub/scm/network/iproute2/iproute2.git/tree/ip/iproute.c#n742

you can review it, add debug prints, etc to get your code working the same.

gateway field missing in netlink message

[Magesh M P]

Hi Dave/Steve

There is nothing to do with VPP code here.

There is a netlink related file librtnl/netns.c which calls function rtnl_parse_rtattr() which is not handling RTA_MULTIPATH. This function is called when netlink message is received from the kernel.

netns.c is netlink library file.

Any idea to look for the latest code in netns.c to handle RTA_MULTIPATH ?

I added code but I could see only one gateway RTA_GATEWAY inside RTA_MULTIPATH. 

Regards
Magesh

gateway field missing in netlink message

[Magesh M P]

Hi Dave/Steve,

Thanks for your support....

The Linux kernel is sending the route information with dual gateways properly to the netlink listener in the vpp stack.

The following parser logic is used in netlink listener to retrieve the two gateways in RTA_MULTIPATH attribute

      if (route_attribute->rta_type == RTA_MULTIPATH)
        {
           printf("AMG - RTA_MULTIPATH\n");
           /* Get RTA_MULTIPATH data */
           struct rtnexthop *nhptr = (struct rtnexthop*) RTA_DATA(route_attribute);
           int rtnhp_len = RTA_PAYLOAD(route_attribute);
           /* Is the message complete? */
           if (rtnhp_len < (int) sizeof(*nhptr) || 
              nhptr->rtnh_len > rtnhp_len)
           {
              continue;
           }
           /* Get the size of the attributes */
           unsigned int attrlen = rtnhp_len - sizeof(struct rtnexthop);
           if (attrlen) 
           {
             /* Retrieve attributes */
             struct rtattr *attr = RTNH_DATA(nhptr);
			 int len = RTA_PAYLOAD(route_attribute);
			 struct rtnexthop *nh = RTA_DATA(route_attribute);
				 
			 for(;;)
		     {
				 if (len < sizeof(*nh)) 
				 {
				 	printf("BREAK -attr->rta_len[%u]\n", len);
					 break;
				 }
				 if (nh->rtnh_len > len) 
				 {
				 	printf("BREAK2 -attr->rta_len[%u]\n", len);
				    break;
				 }

                 printf("RTA_MULTIPATH - attr=[%p] attr->rta_len=[%u]attr->rta_type=[%u] DATA=[%p]\n", attr, attr->rta_len, attr->rta_type, RTA_DATA(attr));

				 if (nh->rtnh_len > sizeof(*nh)) 
				 {
					unsigned short type;				 
					unsigned int len_nh=nh->rtnh_len - sizeof(*nh);
					struct rtattr* attr2=RTNH_DATA(nh);
				    while (RTA_OK(attr2, len_nh)) 
					{
					    type = attr2->rta_type & ~0;
                        if ((type <= RTA_MAX))
                        {
                             attr = attr2;
                        }
						attr2 = RTA_NEXT(attr2, len_nh);
					}
				 } 

                 if (attr->rta_type == RTA_GATEWAY) 
                 {
                     /* Next hops from RTA_MULTIPATH are 
                      * contained in RTA_GATEWAY attributes! 
                      */
                     inet_ntop(AF_INET, RTA_DATA(attr), gateway_address, sizeof(gateway_address)); 
                     printf("GATEWAY: route to destination --> %s/%d proto %d and gateway %s\n", destination_address, route_netmask, route_protocol, gateway_address);
                 }
				len -= NLMSG_ALIGN(nh->rtnh_len);
				nh = RTNH_NEXT(nh);			
              }
           }
         }