Re: Race condition in route lookup

[Jesse Hathaway <jesse@mbuki-mvuki.org>]

On Thu, Oct 10, 2019 at 3:31 AM Ido Schimmel <idosch@idosch.org> wrote:
> I think it's working as expected. Here is my theory:
>
> If CPU0 is executing both the route get request and forwarding packets
> through the directly connected interface, then the following can happen:
>
> <CPU0, t0> - In process context, per-CPU dst entry cached in the nexthop
> is found. Not yet dumped to user space
>
> <Any CPU, t1> - Routes are added / removed, therefore invalidating the
> cache by bumping 'net->ipv4.rt_genid'
>
> <CPU0, t2> - In softirq, packet is forwarded through the nexthop. The
> cached dst entry is found to be invalid. Therefore, it is replaced by a
> newer dst entry. dst_dev_put() is called on old entry which assigns the
> blackhole netdev to 'dst->dev'. This netdev has an ifindex of 0 because
> it is not registered.
>
> <CPU0, t3> - After softirq finished executing, your route get request
> from t0 is resumed and the old dst entry is dumped to user space with
> ifindex of 0.
>
> I tested this on my system using your script to generate the route get
> requests. I pinned it to the same CPU forwarding packets through the
> nexthop. To constantly invalidate the cache I created another script
> that simply adds and removes IP addresses from an interface.
>
> If I stop the packet forwarding or the script that invalidates the
> cache, then I don't see any '*' answers to my route get requests.

Thanks for the reply and analysis Ido, I tested with an additional script which
adds and deletes a route in a loop, as you also saw this increased the
frequency of blackhole route replies from the first script.

Questions:

1. We saw this behavior occurring with TCP connections traversing our routers,
though I was able to reproduce it with only local route requests on our router.
Would you expect this same behavior for TCP traffic only in the kernel which
does not go to userspace?

2. These blackhole routes occur even though our main routing table is not
changing, however a separate route table managed by bird on the Linux router is
changing. Is this still expected behavior given that the ip-rules and main
route table used by these route requests are not changing?

3. We were previously rejecting these packets with an iptables rule which sent
an ICMP prohibited message to the sender, this caused TCP connections to break
with a EHOSTUNREACH, should we be silently dropping these packets instead?

4. If we should just be dropping these packets, why does the kernel not drop
them instead of letting them traverse the iptables rules?

> BTW, the blackhole netdev was added in 5.3. I assume (didn't test) that
> with older kernel versions you'll see 'lo' instead of '*'.

Yes indeed! Thanks for solving that mystery as well, our routers are running
5.1, but we upgraded to 5.4-rc2 to determine whether the issue was still
present in the latest kernel.