Re: Potential regression/bug in net/mlx5 driver

From: Jakub Kicinski <kuba@kernel.org>
To: Saeed Mahameed <saeedm@nvidia.com>
Cc: Paul Moore <paul@paul-moore.com>,
	Leon Romanovsky <leon@kernel.org>,
	Linux regressions mailing list <regressions@lists.linux.dev>,
	Saeed Mahameed <saeed@kernel.org>, Shay Drory <shayd@nvidia.com>,
	netdev@vger.kernel.org, selinux@vger.kernel.org,
	Tariq Toukan <tariqt@nvidia.com>
Subject: Re: Potential regression/bug in net/mlx5 driver
Date: Fri, 14 Apr 2023 17:34:45 -0700	[thread overview]
Message-ID: <20230414173445.0800b7cf@kernel.org> (raw)
In-Reply-To: <ZDnRkVNYlHk4QVqy@x130>

On Fri, 14 Apr 2023 15:20:01 -0700 Saeed Mahameed wrote:
> >> Officially we test only 3 GA FWs back. The fact that mlx5 is a generic CX
> >> driver makes it really hard to test all the possible combinations, so we
> >> need to be strict with how back we want to officially support and test old
> >> generations.  
> >
> >Would you be able to pull the datapoints for what 3 GA FWs means
> >in case of CX4? Release number and date when it was released?
>
> https://network.nvidia.com/files/related-docs/eol/LCR-000821.pdf
> 
> Since CX4 was EOL last year, it is going to be hard to find this info but
> let me check my email archive.. 
> 
> 12.28.2006   27-Sep-20 - recommended version
> 12.26.xxxx   12-Dec-2019
> 12.24.1000   2-Dec-18

That's basically 3 years of support. Seems fairly reasonable.

> >> Upgrade FW when possible, it is always easier than upgrading the kernel.
> >> Anyways this was a very rare FW/Arch bug, We should've exposed an
> >> explicit cap for this new type of PF when we had the chance, now it's too
> >> late since a proper fix will require FW and Driver upgrades and breaking
> >> the current solution we have over other OSes as well.
> >>
> >> Yes I can craft an if condition to explicitly check for chip id and FW
> >> version for this corner case, which has no precedence in mlx5, but I prefer
> >> to ask to upgrade FW first, and if that's an acceptable solution, I would
> >> like to keep the mlx5 clean and device agnostic as much as possible.  
> >
> >IMO you either need a fully fleshed out FW update story, with advanced
> >warnings for a few releases, distributing the FW via linux-firmware or
> >fwupdmgr or such.  Or deal with the corner cases in the driver :(
> 
> Completely agree, I will start an internal discussion .. 
> 
> >We can get Paul to update, sure, but if he noticed so quickly the
> >question remains how many people out in the wild will get affected
> >and not know what the cause is?  
> 
> Right, I will make sure this will be addressed, will let you know how we
> will handle this, will try to post a patch early next cycle, but i will
> need to work with Arch and release managers for this, so it will take a
> couple of weeks to formalize a proper solution.

What do we do now, tho? If the main side effect of a revert is that
users of a newfangled device with an order of magnitude lower
deployment continue to see a warning/error in the logs - I'm leaning
towards applying it :(