Re: [PATCH net-next 1/6] ethtool: Extend link modes settings uAPI with lanes

From: Jiri Pirko <jiri@resnulli.us>
To: Edwin Peer <edwin.peer@broadcom.com>
Cc: Ido Schimmel <idosch@idosch.org>, netdev <netdev@vger.kernel.org>,
	"David S . Miller" <davem@davemloft.net>,
	Jakub Kicinski <kuba@kernel.org>,
	jiri@nvidia.com, danieller@nvidia.com, andrew@lunn.ch,
	f.fainelli@gmail.com, mkubecek@suse.cz, mlxsw@nvidia.com,
	Ido Schimmel <idosch@nvidia.com>
Subject: Re: [PATCH net-next 1/6] ethtool: Extend link modes settings uAPI with lanes
Date: Tue, 1 Dec 2020 12:22:50 +0100	[thread overview]
Message-ID: <20201201112250.GK3055@nanopsycho.orion> (raw)
In-Reply-To: <CAKOOJTw54DxitbYHW7vNVWRv9BbsdmW_ARTgpMu5HBVjkTeQ5w@mail.gmail.com>

Mon, Nov 30, 2020 at 07:00:16PM CET, edwin.peer@broadcom.com wrote:
>On Mon, Nov 30, 2020 at 9:14 AM Jiri Pirko <jiri@resnulli.us> wrote:
>
>> >> There is a crucial difference. Split port is configured alwasy by user.
>> >> Each split port has a devlink instace, netdevice associated with it.
>> >> It is one level above the lanes.
>> >
>> >Right, but the one still implies the other. Splitting the port implies fewer
>> >lanes available.
>> >
>> >I understand the concern if the device cannot provide sufficient MAC
>> >resources to provide for the additional ports, but leaving a net device
>> >unused (with the option to utilize an additional, now spare, port) still
>> >seems better to me than leaving lanes unused and always wasted.
>>
>> I don't follow what exactly are you implying. Could you elaborate a bit
>> more?
>
>Perhaps an example...
>
>Consider a physical QSFP connector comprising 4 lanes. Today, if the
>speed is forced, we would achieve 100G speeds using all 4 lanes with
>NRZ encoding. If we configure the port for PAM4 encoding at the same
>speed, then we only require 2 of the available 4 lanes. The remaining 2
>lanes are wasted. If we only require 2 of the 4 lanes, why not split the
>port and request the same speed of one of the now split out ports? Now,
>this same speed is only achievable using PAM4 encoding (it is implied)
>and we have a spare, potentially usable, assuming an appropriate break-
>out cable, port instead of the 2 unused lanes.

I don't see how this dynamic split port could work in real life to be
honest. The split is something admin needs to configure and can rely
that netdevice exists all the time and not comes and goes under
different circumstances. Multiple obvious reasons why.

One way or another, I don't see the relation between this and this
patchset.

>
>So concretely, I'm suggesting that if we want to force PAM4 at the lower
>speeds, split the port and then we don't need an ethtool interface change
>at all to achieve the same goal. Having a spare (potentially usable) port
>is better than spare (unusable) lanes.

The admin has to decide, define.

>
>Now, if the port can't be split for some reason (perhaps there aren't
>sufficient device MAC resources, stats contexts, whatever), then that's
>a different story. But, even so, perhaps the port lane topology more
>appropriately belongs as part of a device configuration interface in
>devlink and the number of lanes available to a port should be a
>property of the port instead of a link mode knob?
>
>Regards,
>Edwin Peer