Re: [PATCH] net/mlx5: reduce stack usage in FW tracer

From: Arnd Bergmann <arnd@arndb.de>
To: Saeed Mahameed <saeedm@mellanox.com>
Cc: "davem@davemloft.net" <davem@davemloft.net>,
	"leon@kernel.org" <leon@kernel.org>, "cai@lca.pw" <cai@lca.pw>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	Moshe Shemesh <moshe@mellanox.com>,
	Feras Daoud <ferasda@mellanox.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	Eran Ben Elisha <eranbe@mellanox.com>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	Erez Shitrit <erezsh@mellanox.com>
Subject: Re: [PATCH] net/mlx5: reduce stack usage in FW tracer
Date: Mon, 9 Sep 2019 22:18:48 +0200	[thread overview]
Message-ID: <CAK8P3a0_VhZ9hYmc6P3Qx+Z6WSHh3PVZ7JZh7Tr=R1CAKvqWmA@mail.gmail.com> (raw)
In-Reply-To: <383db08b6001503ac45c2e12ac514208dc5a4bba.camel@mellanox.com>

On Mon, Sep 9, 2019 at 9:39 PM Saeed Mahameed <saeedm@mellanox.com> wrote:
> On Fri, 2019-09-06 at 17:11 +0200, Arnd Bergmann wrote:
> > --- a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c
> > +++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c
> > @@ -557,16 +557,16 @@ static void mlx5_tracer_print_trace(struct
> > tracer_string_format *str_frmt,
> >                                   struct mlx5_core_dev *dev,
> >                                   u64 trace_timestamp)
> >  {
> > -     char    tmp[512];
> > -
>
> Hi Arnd, thanks for the patch,
> this function is very perfomance critical when fw traces are activated
> to pull some fw content on error situations, using kmalloc here might
> become a problem and stall the system further more if the problem was
> initially due to lack of memory.
>
> since this function only needs 512 bytes maybe we should mark it as
> noinline to avoid any extra stack usages on the caller function
> mlx5_fw_tracer_handle_traces ?

That would shut up the warning, but doesn't sound right either.

If it's performance critical indeed, maybe the best solution would
be to also avoid the snprintf(), as that is also a rather heavyweight
function?

I could not find an easy solution for this, but I did notice the unusual way
this deals with a variable format string passed into mlx5_tracer_print_trace
along with a set of parameters, which opens up a set of possible
format string vulnerabilities as well as making mlx5_tracer_print_trace()
a bit expensive. You also take a mutex and free memory in there,
which obviously then also got allocated in the fast path.

To do this right, a better approach may be to just rely on ftrace, storing
the (pointer to the) format string and the arguments in the buffer without
creating a string. Would that be an option here?

A more minimal approach might be to move what is now the on-stack
buffer into the mlx5_fw_tracer function. I see that you already store
a copy of the string in there from mlx5_fw_tracer_save_trace(),
which conveniently also holds a mutex already that protects
it from concurrent access.

       Arnd