Re: TC BPF Program Crashing With Bnx2x Drivers

Re: TC BPF Program Crashing With Bnx2x Drivers

[Jakub Kicinski]

On Thu, 9 Apr 2020 01:57:42 +0200 Daniel Borkmann wrote:
> On 4/9/20 1:30 AM, Christian Deacon wrote:
> > Hey Daniel,
> > 
> > 
> > Thank you for your response and I'm glad I'm in the correct area!
> > 
> > 
> > When the individual ran:
> > 
> > 
> > ```
> > 
> > ethtool -K eth0 tso off
> > 
> > ```
> > 
> > 
> > The program started operating without crashing. It has been around 20 minutes so far and no crash. Therefore, I'd assume that stopped the crashing considering it usually crashed 20 - 30 seconds after starting the program each time beforehand. I'm not entirely sure what TSO does with this network driver, but I'll try doing some research.  
> 
> Yep, don't think it should crash anymore after you turned it off and
> it survived since then. ;) I presume GSO is still on in your case,
> right (check via `ethtool -k eth0`)?
> 
> > I was suspecting it may be the 'bpf_skb_adjust_room()' function as
> > well since I'm using a mode that was implemented in later kernels.
> > This function removes the outer IP header in my program from the
> > outgoing IPIP packet. I'm not sure what would be causing the
> > crashing, though.  
> 
> Probably bnx2x folks might be able to help but as mentioned looks
> like the tso handling in there has an issue with the ipip which leads
> to the nic hang eventually.

IMHO this is not a bnx2x problem. The drivers should not have to
re-validate GSO flags..

Let's see if I get this right. We have an IPinIP encap, IPXIP4 GSO skb
comes down and TC bpf pulls the outer header off, but the gso flags
remain unchanged. The driver then sees IPXIP4 GSO but there are no
headers so it implodes. Is this correct?

And we have the ability to add the right gso flags for encap, not decap
(bpf_skb_net_grow() vs bpf_skb_net_shrink()).

RE: [EXT] Re: TC BPF Program Crashing With Bnx2x Drivers

[Ariel Elior]

> -----Original Message-----
> From: Jakub Kicinski <kuba@kernel.org>
> Sent: Thursday, April 9, 2020 10:18 PM
> To: Daniel Borkmann <daniel@iogearbox.net>
> Cc: Christian Deacon <gamemann@gflclan.com>; bpf@vger.kernel.org; Ariel
> Elior <aelior@marvell.com>; Sudarsana Reddy Kalluru
> <skalluru@marvell.com>; netdev@vger.kernel.org
> Subject: [EXT] Re: TC BPF Program Crashing With Bnx2x Drivers
> 
> External Email
> 
> ----------------------------------------------------------------------
> On Thu, 9 Apr 2020 01:57:42 +0200 Daniel Borkmann wrote:
> > On 4/9/20 1:30 AM, Christian Deacon wrote:
> > > Hey Daniel,
> > >
> > >
> > > Thank you for your response and I'm glad I'm in the correct area!
> > >
> > >
> > > When the individual ran:
> > >
> > >
> > > ```
> > >
> > > ethtool -K eth0 tso off
> > >
> > > ```
> > >
> > >
> > > The program started operating without crashing. It has been around 20
> minutes so far and no crash. Therefore, I'd assume that stopped the crashing
> considering it usually crashed 20 - 30 seconds after starting the program each
> time beforehand. I'm not entirely sure what TSO does with this network driver,
> but I'll try doing some research.
> >
> > Yep, don't think it should crash anymore after you turned it off and
> > it survived since then. ;) I presume GSO is still on in your case,
> > right (check via `ethtool -k eth0`)?
> >
> > > I was suspecting it may be the 'bpf_skb_adjust_room()' function as
> > > well since I'm using a mode that was implemented in later kernels.
> > > This function removes the outer IP header in my program from the
> > > outgoing IPIP packet. I'm not sure what would be causing the
> > > crashing, though.
> >
> > Probably bnx2x folks might be able to help but as mentioned looks
> > like the tso handling in there has an issue with the ipip which leads
> > to the nic hang eventually.
> 
> IMHO this is not a bnx2x problem. The drivers should not have to
> re-validate GSO flags..
> 
> Let's see if I get this right. We have an IPinIP encap, IPXIP4 GSO skb
> comes down and TC bpf pulls the outer header off, but the gso flags
> remain unchanged. The driver then sees IPXIP4 GSO but there are no
> headers so it implodes. Is this correct?
> 
> And we have the ability to add the right gso flags for encap, not decap
> (bpf_skb_net_grow() vs bpf_skb_net_shrink()).

Hi,
I would still like to get an ethtool -d output after this happens, to review
the device state. Even if the skb was illegal, the error we got was device
crash, which seems too severe. I wonder what would happen if a VF
would provide such an SKB.
Thanks,
Ariel