Re: phys_port_id in switchdev mode?

From: Jakub Kicinski <jakub.kicinski@netronome.com>
To: Marcelo Ricardo Leitner <marcelo.leitner@gmail.com>
Cc: Florian Fainelli <f.fainelli@gmail.com>,
	Or Gerlitz <gerlitz.or@gmail.com>,
	Simon Horman <simon.horman@netronome.com>,
	Andy Gospodarek <andy@greyhouse.net>,
	"mchan@broadcom.com" <mchan@broadcom.com>,
	Jiri Pirko <jiri@resnulli.us>,
	Alexander Duyck <alexander.duyck@gmail.com>,
	Frederick Botha <frederick.botha@netronome.com>,
	nick viljoen <nick.viljoen@netronome.com>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>
Subject: Re: phys_port_id in switchdev mode?
Date: Sat, 1 Sep 2018 13:34:12 +0200	[thread overview]
Message-ID: <20180901133412.2939466a@cakuba.netronome.com> (raw)
In-Reply-To: <20180831201321.GA4590@localhost.localdomain>

On Fri, 31 Aug 2018 17:13:22 -0300, Marcelo Ricardo Leitner wrote:
> On Tue, Aug 28, 2018 at 08:43:51PM +0200, Jakub Kicinski wrote:
> > Ugh, CC: netdev..
> > 
> > On Tue, 28 Aug 2018 20:05:39 +0200, Jakub Kicinski wrote:  
> > > Hi!
> > > 
> > > I wonder if we can use phys_port_id in switchdev to group together
> > > interfaces of a single PCI PF?  Here is the problem:  
> 
> On Mellanox cards, this is already possible via phys_switch_id, as
> each PF has its own phys_switch_id. So all VFs with a given
> phys_switch_id belong to the PF with that same phys_switch_id.

You mean Connect-X4 and on, Connect-X3 also shares PF between ports.
But you're right, in simpler designs there is usually a 1:1 relation
between an external networking port, a PCIe PF and a eswitch.  Which
does not mean it's correct to conflate the ports.

> I understand this is a vendor-specific design

It's hardly a strange design :)  If the card is a true switch
associating VFs on the PCIe side with an Ethernet port makes no sense.

> but if you have the same phys_switch_id across PFs, does it really
> matter on which PF the VF was created on?

It does, because one can have multiple PFs, potentially connected to
different hosts.  And legacy NDOs, for example, must target the right
one.

So for one thing it seems like the right thing to do to associate VFs
with PFs, not external ports.

The other thing is that if you look at your Mellanox card in switchdev
mode, you'll notice that systemd/udev is not able to name the
representor interfaces properly without custom rules, because they are
not linked to the PCIe PF.  They are not linked because VF reprs don't
provide legacy SR-IOV NDOs, so we prevent libvirt from poking at them.
phys_port_id should steer libvirt away from VF representors, so we can
get proper naming.  (And having legacy NDOs implemented everywhere is a
no-go, because ip link output grows quadratically.)