Re: Kernel oops with mlx5 and dual XDP redirect programs

[Saeed Mahameed <saeedm@mellanox.com>]

On Tue, 2018-10-23 at 12:10 +0200, Toke Høiland-Jørgensen wrote:
> Saeed Mahameed <saeedm@mellanox.com> writes:
> 
> > On Thu, 2018-10-18 at 23:53 +0200, Toke Høiland-Jørgensen wrote:
> > > Saeed Mahameed <saeedm@mellanox.com> writes:
> > > 
> > > > I think that the mlx5 driver doesn't know how to tell the other
> > > > device
> > > > to stop transmitting to it while it is resetting.. Maybe tariq
> > > > or
> > > > Jesper know more about this ?
> > > > I will look at this tomorrow after noon and will try to
> > > > repro...
> > > 
> > > Hi Saeed
> > > 
> > > Did you have a chance to poke at this? :)
> > 
> > HI Toke, yes i have been planing to respond but also i wanted to
> > dig
> > more,
> > 
> > so the root cause is very clear.
> > 
> > 1. core 1 is doing tx_dev->ndo_xdp_xmit()
> > 2. core 2 is doing tx_dev->xdp_set() //remove xdp program.
> 
> Right, it was also my guess that it was related to this interaction.
> Thanks for looking into it!
> 
> > and the problem is beyond mlx5, since we don't have a way to tell a
> > different core/different netdev to stop xmitting, or at least
> > synchronize with it.
> 
> Hmm, ideally there should be some way for the higher level XDP API to
> notice this and abort the call before it even reaches the driver on
> the
> TX side, shouldn't there? At LPC, Jesper and I will be talking about
> a
> proposal for decoupling the ndo_xdp_xmit() resource allocation from
> loading and unloading XDP programs, which I guess could be a way to
> deal
> with this as well.
> 
> In the meantime...
> 

Yes totally agree, this is why my fix is temporary. 
Good Idea about LPC, let's discuss this there.

> > I will be waiting for your confirmation that the fix did work.
> 
> I tested your patch, and it does indeed fix the crash. However, it
> also
> seems to have the effect that the XDP redirect continues to function
> even after removing the XDP program on the target device.
> 
> I.e., after the call to ./xdp_fwd -d $TX_IF, I still see packets
> being
> redirected out $TX_IF. Is this intentional?
> 

Interesting, shouldn't happen, unless there is something weird going on
when running xpd_fwd -d together with xdp_redirect_map, i just checked
the code and if ndo_xdp_set was called with null program we will remove
xdp tx resources, nothing suspicious in the driver.

I will look at this later this week.

> -Toke