Re: [PATCH net-next v2 0/3] VGT+ support

[Saeed Mahameed <saeedm@mellanox.com>]

On Mon, 2019-11-04 at 18:35 -0800, Jakub Kicinski wrote:
> On Mon, 4 Nov 2019 18:47:32 -0700, David Ahern wrote:
> > On 11/4/19 6:38 PM, Saeed Mahameed wrote:
> > > On Fri, 2019-11-01 at 17:21 -0700, Jakub Kicinski wrote:  
> > > > On Fri, 1 Nov 2019 21:28:22 +0000, Saeed Mahameed wrote:  
> > > > > Bottom line, we tried to push this feature a couple of years
> > > > > ago,
> > > > > and
> > > > > due to some internal issues this submission ignored for a
> > > > > while,
> > > > > now as
> > > > > the legacy sriov customers are moving towards upstream, which
> > > > > is
> > > > > for me
> > > > > a great progress I think this feature worth the shot, also as
> > > > > Ariel
> > > > > pointed out, VF vlan filter is really a gap that should be
> > > > > closed.
> > > > > 
> > > > > For all other features it is true that the user must consider
> > > > > moving to
> > > > > witchdev mode or find a another community for support.
> > > > > 
> > > > > Our policy is still strong regarding obsoleting legacy mode
> > > > > and
> > > > > pushing
> > > > > all new feature to switchdev mode, but looking at the facts
> > > > > here  I
> > > > > do
> > > > > think there is a point here and ROI to close this gap in
> > > > > legacy
> > > > > mode.
> > > > > 
> > > > > I hope this all make sense.   
> > > > 
> > > > I understand and sympathize, you know full well the benefits of
> > > > working
> > > > upstream-first...
> > > > 
> > > > I won't reiterate the entire response from my previous email,
> > > > but the
> > > > bottom line for me is that we haven't added a single legacy VF
> > > > NDO
> > > > since 2016, I was hoping we never will add more and I was
> > > > trying to
> > > > stop anyone who tried.
> > > >  
> > > 
> > > The NDO is not the problem here, we can perfectly extend the
> > > current
> > > set_vf_vlan_ndo to achieve the same goal with minimal or even NO
> > > kernel
> > > changes, but first you have to look at this from my angel, i have
> > > been
> > > doing lots of research and there are many points for why this
> > > should be
> > > added to legacy mode:
> > > 
> > > 1) Switchdev mode can't replace legacy mode with a press of a
> > > button,
> > > many missing pieces.
> > > 
> > > 2) Upstream Legacy SRIOV is incomplete since it goes together
> > > with
> > > flexible vf vlan configuration, most of mlx5 legacy sriov users
> > > are
> > > using customized kernels and external drivers, since upstream is
> > > lacking this one basic vlan filtering feature, and many users
> > > decline
> > > switching to upstream kernel due to this missing features.
> > > 
> > > 3) Many other vendors have this feature in customized
> > > drivers/kernels,
> > > and many vendors/drivers don't even support switchdev mode (mlx4
> > > for
> > > example), we can't just tell the users of such device we are not
> > > supporting basic sriov legacy mode features in upstream kernel.
> > > 
> > > 4) the motivation for this is to slowly move sriov users to
> > > upstream
> > > and eventually to switchdev mode.  
> > 
> > If the legacy freeze started in 2016 and we are at the end of 2019,
> > what
> > is the migration path?
> 
> The migration path is to implement basic bridge offload which can
> take
> care of this trivially.
> 

it is not about implementation, it is also the user echo system that
needs to be migrated.
so this needs to happen in baby steps, you can't just ask ask someone
to move to something that is not even cooked yet, and will require
months or years of validation and deployment.

> Problem is people equate switchdev with OvS offload today, so bridge
> offload is not actually implemented. It's really hard to convince
> product marketing that it's work worth taking on.
> 
> Adding incremental features to legacy API is always going to be
> cheaper
> than migrating controllers to switchdev.
> 

Yes this feature is 1000 times cheaper than implementing the full
offload, and yet very trivial and necessary, the ROI is so big that it
worth doing it, more exposure to upstream.

> IDK if you remember the net_failover argument about in-kernel VF to
> virtio bonding. The controllers are doing the bare minimum and it's 
> hard for HW vendors to justify the expense of moving forward all
> parts 
> of the stack.
> 
> Which means SR-IOV is either stuck in pure-L2 middle ages or requires
> all the SDN complexity and overhead. Switchdev bridge offload can be
> trivially extended to support eVPN and simplify so many deployments,
> sigh..
> 
> > > Now if the only remaining problem is the uAPI, we can minimize
> > > kernel
> > > impact or even make no kernel changes at all, only ip route2 and
> > > drivers, by reusing the current set_vf_vlan_ndo.  
> > 
> > And this caught my eye as well -- iproute2 does not need the
> > baggage either.
> > 
> > Is there any reason this continued support for legacy sriov can not
> > be
> > done out of tree?
> 
> Exactly. Moving to upstream is only valuable if it doesn't require
> brining all the out-of-tree baggage.

this baggage is a very essential part for eth sriov, it is a missing
feature in both switchdev mode (bridge offloads) and legacy.

Guys, I need to know my options here and make some effort assessment.

1) implement bridge offloads: months of development, years for
deployment and migration
2) Close this gap in legacy mode: days.

I am all IN for bridge offloads, but you have to understand why i pick
2, not because it is cheaper, but because it is more realistic for my
current users. Saying no to this just because switchdev mode is the de
facto standard isn't fair and there should be an active clear
transition plan, with something available to work with ... not just
ideas.

Your claims are valid only when we are truly ready for migration. we
are simply not and no one has a clear plan in the horizon, so i don't
get this total freeze attitude of legacy mode, it should be just like
ethtool we want to to replace it but we know we are not there yet, so
we carefully add only necessary things with lots of auditing, same
should go here.