Re: [PATCH net-next] ipv4/fib: send RTM_DELROUTE notify when flush fib

[Nicolas Dichtel <nicolas.dichtel@6wind.com>]

Le 13/09/2023 à 16:53, Nicolas Dichtel a écrit :
> Le 13/09/2023 à 16:43, David Ahern a écrit :
>> On 9/13/23 8:11 AM, Nicolas Dichtel wrote:
>>> The compat_mode was introduced for daemons that doesn't support the nexthop
>>> framework. There must be a notification (RTM_DELROUTE) when a route is deleted
>>> due to a carrier down event. Right now, the backward compat is broken.
>>
>> The compat_mode is for daemons that do not understand the nexthop id
>> attribute, and need the legacy set of attributes for the route - i.e,
> Yes, it's my point.
> On my system, one daemon understands and configures nexthop id and another one
> doesn't understand nexthop id. This last daemon removes routes when an interface
> is put down but not when the carrier is lost.
> The kernel doc [1] says:
> 	Further, updates or deletes of a nexthop configuration generate route
> 	notifications for each fib entry using the nexthop.
> So, my understanding is that a RTM_DELROUTE msg should be sent when a nexthop is
> removed due to a carrier lost event.
> 
> [1]
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/ip-sysctl.rst#n2116

I dug a bit more about these (missing) notifications. I will try to describe
what should be done for cases where there is no notification:

When an interface is set down:
 - the single (!multipath) routes associated with this interface should be
   removed;
 - for multipath routes:
   + if all nh use this interface: the routes are deleted;
   + if only some nh uses this interface :
     ~ if all other nh already point to a down interface: the routes are deleted;
     ~ if at least one nh points to an up interface:
       o the nh are *temporarily* disabled if it's a plain nexthop;
       o the nh is *definitely* removed if it's a nexthop object;
When the interface is set up later, disabled nh are restored (ie only plain
nexthop of multipath routes).

When an interface loses its carrier:
 - for routes using plain nexthop: nothing happens;
 - for routes using nexthop objects:
   + for single routes: they are deleted;
   + for multipath routes, the nh is definitely removed if it's a nexthop
     object (ie the route is deleted if there is no other nexthop in the group);
When an interface recovers its carrier, there is nothing to do.

When the last ipv4 address of an interface is removed:
 - for routes using nexthop objects: nothing happens;
 - for routes using plain nexthop: the same rules as 'interface down' applies.
When an ipv4 address is added again on the interface, disabled nh are restored
(ie only plain nexthop of multipath routes).

I bet I miss some cases.

Conclusions:
 - legacy applications (that are not aware of nexthop objects) cannot maintain a
   routing cache (even with compat_mode enabled);
 - fixing only the legacy applications (aka compat_mode) seems too
   complex;
 - even if an application is aware of nexthop objects, the rules to maintain a
   cache are far from obvious.

I don't understand why there is so much reluctance to not send a notification
when a route is deleted. This would fix all cases.
I understand that the goal was to save netlink traffic, but in this case, the
daemons that are interested in maintaining a routing cache have to fully parse
their cache to mark/remove routes. For big routing tables, this will cost a lot
of cpu, so I wonder if it's really a gain for the system. On such systems, there
is probably more than one daemon in this case, so even more cpu to spend for
these operations.

As Thomas said, this discussion has come up for more than a decade. And with the
nexthop objects support, it's even more complex. There is obviously something to do.

At least, I would have expected an RTM_DELNEXTHOP msg for each deleted nexthop.
But this wouldn't solve the routing cache sync for legacy applications.


Regards,
Nicolas