Re: [RFC PATCH net-next v4 2/6] devlink: Extend devlink-rate api with queues and new parameters

From: "Wilczynski, Michal" <michal.wilczynski@intel.com>
To: Jakub Kicinski <kuba@kernel.org>
Cc: Edward Cree <ecree.xilinx@gmail.com>, <netdev@vger.kernel.org>,
	<alexandr.lobakin@intel.com>, <dchumak@nvidia.com>,
	<maximmi@nvidia.com>, <jiri@resnulli.us>,
	<simon.horman@corigine.com>, <jacob.e.keller@intel.com>,
	<jesse.brandeburg@intel.com>, <przemyslaw.kitszel@intel.com>
Subject: Re: [RFC PATCH net-next v4 2/6] devlink: Extend devlink-rate api with queues and new parameters
Date: Thu, 22 Sep 2022 15:45:55 +0200	[thread overview]
Message-ID: <9656fcda-0d63-06dc-0803-bc5f90ee44fd@intel.com> (raw)
In-Reply-To: <20220922055040.7c869e9c@kernel.org>

On 9/22/2022 2:50 PM, Jakub Kicinski wrote:
> On Thu, 22 Sep 2022 13:44:14 +0200 Wilczynski, Michal wrote:
>> Below I'll paste the output of how initially the topology looks like for our
>> hardware.
>> If the devlink_port objects are present (as in switchdev mode), there
>> should also be vport nodes represented. It is NOT a requirement for
>> a queue to have a vport as it's ancestor.
> Thanks! How did you know that my preferred method of reading
> hierarchies is looking at a linear output!? 😕

Haha :D, sorry about that, that's the real output from the devlink tool 
that I
modified, so it's already like that for devlink-rate. At least I don't 
know about
any way to represent this better, besides drawing the hierarchy by hand. And
it's quite big of a hierarchy.

>
> Anyway. My gut feeling is that this is cutting a corner. Seems
> most natural for the VF/PF level to be controlled by the admin
> and the queue level by whoever owns the queue. The hypervisor
> driver/FW should reconcile the two and compile the full hierarchy.

We tried already tc-htb, and it doesn't work for a couple of reasons, 
even in this potential
hybrid with devlink-rate. One of the problems with tc-htb offload is 
that it forces you to allocate a new
queue, it doesn't allow for reassigning an existing queue to another 
scheduling node. This
is our main use case.

Here's a discussion about tc-htb: 
https://lore.kernel.org/netdev/20220704114513.2958937-1-michal.wilczynski@intel.com/

So either I would have to invent a new offload type (?) for tc, or 
completely rewrite and
probably break tc-htb that mellanox implemented.
Also in our use case it's possible to create completely new branches 
from the root and
reassigning queues there. This wouldn't be possible with the method 
you're proposing.

So existing interface doesn't allow us to do what is required.

BR,
Michał