Re: Potential regression/bug in net/mlx5 driver

From: Paul Moore <paul@paul-moore.com>
To: Jakub Kicinski <kuba@kernel.org>
Cc: Saeed Mahameed <saeedm@nvidia.com>,
	Leon Romanovsky <leon@kernel.org>,
	 Linux regressions mailing list <regressions@lists.linux.dev>,
	Saeed Mahameed <saeed@kernel.org>,  Shay Drory <shayd@nvidia.com>,
	netdev@vger.kernel.org, selinux@vger.kernel.org,
	 Tariq Toukan <tariqt@nvidia.com>
Subject: Re: Potential regression/bug in net/mlx5 driver
Date: Fri, 14 Apr 2023 10:37:07 -0400	[thread overview]
Message-ID: <CAHC9VhQrDSc65njFBQ8sJ_zr2AcP-qQEU-BcAk5h69XhC=H=dA@mail.gmail.com> (raw)
In-Reply-To: <20230413202631.7e3bd713@kernel.org>

On Thu, Apr 13, 2023 at 11:26 PM Jakub Kicinski <kuba@kernel.org> wrote:
> On Thu, 13 Apr 2023 20:03:18 -0700 Saeed Mahameed wrote:
> > On 13 Apr 15:51, Jakub Kicinski wrote:
> > >On Thu, 13 Apr 2023 15:34:21 -0700 Saeed Mahameed wrote:

...

> > >The question is who's supposed to be paying the price of mlx5 being
> > >used for old and new parts? What is fair to expect from the user
> > >when the FW Paul has presumably works just fine for him?
> > >
> > Upgrade FW when possible, it is always easier than upgrading the kernel.
> > Anyways this was a very rare FW/Arch bug, We should've exposed an
> > explicit cap for this new type of PF when we had the chance, now it's too
> > late since a proper fix will require FW and Driver upgrades and breaking
> > the current solution we have over other OSes as well.
> >
> > Yes I can craft an if condition to explicitly check for chip id and FW
> > version for this corner case, which has no precedence in mlx5, but I prefer
> > to ask to upgrade FW first, and if that's an acceptable solution, I would
> > like to keep the mlx5 clean and device agnostic as much as possible.
>
> IMO you either need a fully fleshed out FW update story, with advanced
> warnings for a few releases, distributing the FW via linux-firmware or
> fwupdmgr or such.  Or deal with the corner cases in the driver :(
>
> We can get Paul to update, sure, but if he noticed so quickly the
> question remains how many people out in the wild will get affected
> and not know what the cause is?

I think it is that last bit which is the real issue, at least from a
regression standpoint.  I didn't see anything on the console or in the
logs to indicate that ancient/buggy FW was the issue, even once I
bisected the kernel (which your average user isn't going to do) it
wasn't clear that it was a FW problem.  Perhaps the mlx5 driver should
perform a simple FW version check on initialization and
pr_warn()/pr_err() if the loaded FW is below a support threshold?
Seeing a "mlx5: hey idiot, your FW is ancient, you need to upgrade!"
line on my console/dmesg would have sent me in the right direction and
likely avoided all of this ...

-- 
paul-moore.com