[PATCH] net/mlx5: reduce stack usage in FW tracer

[PATCH] net/mlx5: reduce stack usage in FW tracer

[Arnd Bergmann]

It's generally not ok to put a 512 byte buffer on the stack, as kernel
stack is a scarce resource:

drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c:660:13: error: stack frame size of 1032 bytes in function 'mlx5_fw_tracer_handle_traces' [-Werror,-Wframe-larger-than=]

This is done in a context that is allowed to sleep, so using
dynamic allocation is ok as well. I'm not too worried about
runtime overhead, as this already contains an snprintf() and
other expensive functions.

Fixes: 70dd6fdb8987 ("net/mlx5: FW tracer, parse traces and kernel tracing support")
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
---
 .../mellanox/mlx5/core/diag/fw_tracer.c       | 21 ++++++++++---------
 1 file changed, 11 insertions(+), 10 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c
index 2011eaf15cc5..d81e78060f9f 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c
@@ -557,16 +557,16 @@ static void mlx5_tracer_print_trace(struct tracer_string_format *str_frmt,
 				    struct mlx5_core_dev *dev,
 				    u64 trace_timestamp)
 {
-	char	tmp[512];
-
-	snprintf(tmp, sizeof(tmp), str_frmt->string,
-		 str_frmt->params[0],
-		 str_frmt->params[1],
-		 str_frmt->params[2],
-		 str_frmt->params[3],
-		 str_frmt->params[4],
-		 str_frmt->params[5],
-		 str_frmt->params[6]);
+	char *tmp = kasprintf(GFP_KERNEL, str_frmt->string,
+			      str_frmt->params[0],
+			      str_frmt->params[1],
+			      str_frmt->params[2],
+			      str_frmt->params[3],
+			      str_frmt->params[4],
+			      str_frmt->params[5],
+			      str_frmt->params[6]);
+	if (!tmp)
+		return;
 
 	trace_mlx5_fw(dev->tracer, trace_timestamp, str_frmt->lost,
 		      str_frmt->event_id, tmp);
@@ -576,6 +576,7 @@ static void mlx5_tracer_print_trace(struct tracer_string_format *str_frmt,
 
 	/* remove it from hash */
 	mlx5_tracer_clean_message(str_frmt);
+	kfree(tmp);
 }
 
 static int mlx5_tracer_handle_string_trace(struct mlx5_fw_tracer *tracer,
-- 
2.20.0

Re: [PATCH] net/mlx5: reduce stack usage in FW tracer

[Saeed Mahameed]

On Fri, 2019-09-06 at 17:11 +0200, Arnd Bergmann wrote:
> It's generally not ok to put a 512 byte buffer on the stack, as
> kernel
> stack is a scarce resource:
> 
> drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c:660:13:
> error: stack frame size of 1032 bytes in function
> 'mlx5_fw_tracer_handle_traces' [-Werror,-Wframe-larger-than=]
> 
> This is done in a context that is allowed to sleep, so using
> dynamic allocation is ok as well. I'm not too worried about
> runtime overhead, as this already contains an snprintf() and
> other expensive functions.
> 
> Fixes: 70dd6fdb8987 ("net/mlx5: FW tracer, parse traces and kernel
> tracing support")
> Signed-off-by: Arnd Bergmann <arnd@arndb.de>
> ---
>  .../mellanox/mlx5/core/diag/fw_tracer.c       | 21 ++++++++++-------
> --
>  1 file changed, 11 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c
> b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c
> index 2011eaf15cc5..d81e78060f9f 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c
> @@ -557,16 +557,16 @@ static void mlx5_tracer_print_trace(struct
> tracer_string_format *str_frmt,
>  				    struct mlx5_core_dev *dev,
>  				    u64 trace_timestamp)
>  {
> -	char	tmp[512];
> -

Hi Arnd, thanks for the patch, 
this function is very perfomance critical when fw traces are activated
to pull some fw content on error situations, using kmalloc here might
become a problem and stall the system further more if the problem was
initially due to lack of memory.

since this function only needs 512 bytes maybe we should mark it as
noinline to avoid any extra stack usages on the caller function
mlx5_fw_tracer_handle_traces ?

> -	snprintf(tmp, sizeof(tmp), str_frmt->string,
> -		 str_frmt->params[0],
> -		 str_frmt->params[1],
> -		 str_frmt->params[2],
> -		 str_frmt->params[3],
> -		 str_frmt->params[4],
> -		 str_frmt->params[5],
> -		 str_frmt->params[6]);
> +	char *tmp = kasprintf(GFP_KERNEL, str_frmt->string,
> +			      str_frmt->params[0],
> +			      str_frmt->params[1],
> +			      str_frmt->params[2],
> +			      str_frmt->params[3],
> +			      str_frmt->params[4],
> +			      str_frmt->params[5],
> +			      str_frmt->params[6]);
> +	if (!tmp)
> +		return;
>  
>  	trace_mlx5_fw(dev->tracer, trace_timestamp, str_frmt->lost,
>  		      str_frmt->event_id, tmp);
> @@ -576,6 +576,7 @@ static void mlx5_tracer_print_trace(struct
> tracer_string_format *str_frmt,
>  
>  	/* remove it from hash */
>  	mlx5_tracer_clean_message(str_frmt);
> +	kfree(tmp);
>  }
>  
>  static int mlx5_tracer_handle_string_trace(struct mlx5_fw_tracer
> *tracer,

Re: [PATCH] net/mlx5: reduce stack usage in FW tracer

[Arnd Bergmann]

On Mon, Sep 9, 2019 at 9:39 PM Saeed Mahameed <saeedm@mellanox.com> wrote:
> On Fri, 2019-09-06 at 17:11 +0200, Arnd Bergmann wrote:
> > --- a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c
> > +++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c
> > @@ -557,16 +557,16 @@ static void mlx5_tracer_print_trace(struct
> > tracer_string_format *str_frmt,
> >                                   struct mlx5_core_dev *dev,
> >                                   u64 trace_timestamp)
> >  {
> > -     char    tmp[512];
> > -
>
> Hi Arnd, thanks for the patch,
> this function is very perfomance critical when fw traces are activated
> to pull some fw content on error situations, using kmalloc here might
> become a problem and stall the system further more if the problem was
> initially due to lack of memory.
>
> since this function only needs 512 bytes maybe we should mark it as
> noinline to avoid any extra stack usages on the caller function
> mlx5_fw_tracer_handle_traces ?

That would shut up the warning, but doesn't sound right either.

If it's performance critical indeed, maybe the best solution would
be to also avoid the snprintf(), as that is also a rather heavyweight
function?

I could not find an easy solution for this, but I did notice the unusual way
this deals with a variable format string passed into mlx5_tracer_print_trace
along with a set of parameters, which opens up a set of possible
format string vulnerabilities as well as making mlx5_tracer_print_trace()
a bit expensive. You also take a mutex and free memory in there,
which obviously then also got allocated in the fast path.

To do this right, a better approach may be to just rely on ftrace, storing
the (pointer to the) format string and the arguments in the buffer without
creating a string. Would that be an option here?

A more minimal approach might be to move what is now the on-stack
buffer into the mlx5_fw_tracer function. I see that you already store
a copy of the string in there from mlx5_fw_tracer_save_trace(),
which conveniently also holds a mutex already that protects
it from concurrent access.

       Arnd

Re: [PATCH] net/mlx5: reduce stack usage in FW tracer

[Saeed Mahameed]

On Mon, 2019-09-09 at 22:18 +0200, Arnd Bergmann wrote:
> On Mon, Sep 9, 2019 at 9:39 PM Saeed Mahameed <saeedm@mellanox.com>
> wrote:
> > On Fri, 2019-09-06 at 17:11 +0200, Arnd Bergmann wrote:
> > > --- a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c
> > > +++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c
> > > @@ -557,16 +557,16 @@ static void mlx5_tracer_print_trace(struct
> > > tracer_string_format *str_frmt,
> > >                                   struct mlx5_core_dev *dev,
> > >                                   u64 trace_timestamp)
> > >  {
> > > -     char    tmp[512];
> > > -
> > 
> > Hi Arnd, thanks for the patch,
> > this function is very perfomance critical when fw traces are
> > activated
> > to pull some fw content on error situations, using kmalloc here
> > might
> > become a problem and stall the system further more if the problem
> > was
> > initially due to lack of memory.
> > 
> > since this function only needs 512 bytes maybe we should mark it as
> > noinline to avoid any extra stack usages on the caller function
> > mlx5_fw_tracer_handle_traces ?
> 
> That would shut up the warning, but doesn't sound right either.
> 
> If it's performance critical indeed, maybe the best solution would
> be to also avoid the snprintf(), as that is also a rather heavyweight
> function?
> 
> I could not find an easy solution for this, but I did notice the
> unusual way
> this deals with a variable format string passed into
> mlx5_tracer_print_trace
> along with a set of parameters, which opens up a set of possible
> format string vulnerabilities as well as making
> mlx5_tracer_print_trace()
> a bit expensive. You also take a mutex and free memory in there,
> which obviously then also got allocated in the fast path.
> 
> To do this right, a better approach may be to just rely on ftrace,
> storing
> the (pointer to the) format string and the arguments in the buffer
> without
> creating a string. Would that be an option here?

I am not sure how this would work, since the format parameters can
changes depending on the FW string and the specific traces.

> 
> A more minimal approach might be to move what is now the on-stack
> buffer into the mlx5_fw_tracer function. I see that you already store
> a copy of the string in there from mlx5_fw_tracer_save_trace(),
> which conveniently also holds a mutex already that protects
> it from concurrent access.
> 

This sounds plausible.

So for now let's do this or the noinline approach, Please let me know
which one do you prefer, if it is the mutex protected buffer, i can do
it myself.

I will open an internal task and discussion then address your valuable
points in a future submission, since we already in rc8 I don't want to
take the risk now.

Thanks for your feedback !
Saeed.

>        Arnd

Re: [PATCH] net/mlx5: reduce stack usage in FW tracer

[Arnd Bergmann]

On Mon, Sep 9, 2019 at 11:53 PM Saeed Mahameed <saeedm@mellanox.com> wrote:
> On Mon, 2019-09-09 at 22:18 +0200, Arnd Bergmann wrote:

> > To do this right, a better approach may be to just rely on ftrace,
> > storing
> > the (pointer to the) format string and the arguments in the buffer
> > without
> > creating a string. Would that be an option here?
>
> I am not sure how this would work, since the format parameters can
> changes depending on the FW string and the specific traces.

Ah, so the format string comes from the firmware? I didn't look
at the code in enough detail to understand why it's done like this,
only enough to notice that it's rather unusual.

Possibly trace_mlx5_fw might still get away with copying the format
string and the arguments, leaving the snprintf() to the time we read
the buffer, but I don't know enough about ftrace to be sure that
would actually work, and you'd need to duplicate it in
mlx5_devlink_fmsg_fill_trace().

> > A more minimal approach might be to move what is now the on-stack
> > buffer into the mlx5_fw_tracer function. I see that you already store
> > a copy of the string in there from mlx5_fw_tracer_save_trace(),
> > which conveniently also holds a mutex already that protects
> > it from concurrent access.
> >
>
> This sounds plausible.
>
> So for now let's do this or the noinline approach, Please let me know
> which one do you prefer, if it is the mutex protected buffer, i can do
> it myself.
>
> I will open an internal task and discussion then address your valuable
> points in a future submission, since we already in rc8 I don't want to
> take the risk now.

Yes, that sounds like a good plan. If you can't avoid the snprintf
entirely, then the mutex protected buffer should be helpful, and
also avoid a strncpy() along with the stack buffer.

      Arnd

RE: [PATCH] net/mlx5: reduce stack usage in FW tracer

[David Laight]

From: Arnd Bergmann
> Sent: 10 September 2019 09:15
...
> > I am not sure how this would work, since the format parameters can
> > changes depending on the FW string and the specific traces.
> 
> Ah, so the format string comes from the firmware? I didn't look
> at the code in enough detail to understand why it's done like this,
> only enough to notice that it's rather unusual.

If the format string comes from the firmware you really shouldn't
pass it to any standard printf function.
You must ensure that it doesn't contain any format effectors
that might dereference parameters.
(The code might try to do that.)

Given that 'pointer' format effectors can't be used, the firmware
must also supply the relevant integer ones?
This should mean that all the processing is deferrable until the
trace record is read.

'noinline' just papers over the cracks.
Especially since vasprintf() is likely to use a lot of stack.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: [PATCH] net/mlx5: reduce stack usage in FW tracer

[Saeed Mahameed]

On Tue, 2019-09-10 at 15:38 +0000, David Laight wrote:
> From: Arnd Bergmann
> > Sent: 10 September 2019 09:15
> ...
> > > I am not sure how this would work, since the format parameters
> > > can
> > > changes depending on the FW string and the specific traces.
> > 
> > Ah, so the format string comes from the firmware? I didn't look
> > at the code in enough detail to understand why it's done like this,
> > only enough to notice that it's rather unusual.
> 
> If the format string comes from the firmware you really shouldn't
> pass it to any standard printf function.
> You must ensure that it doesn't contain any format effectors
> that might dereference parameters.
> (The code might try to do that.)
> 
> Given that 'pointer' format effectors can't be used, the firmware
> must also supply the relevant integer ones?
> This should mean that all the processing is deferrable until the
> trace record is read.
> 

Point taken, i will discuss this with the team next week, since i am
traveling this week. and we will provide a proper solution. for now,
off the top of my head, FW strings and parameters are well defined and
very simple, many of the problems here do not apply, but will verify.

> 'noinline' just papers over the cracks.
> Especially since vasprintf() is likely to use a lot of stack.

Right i also tend to agree with noinline as a temporary solution until
we address all the points provided here.

> 
> 	David
> 
> -
> Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes,
> MK1 1PT, UK
> Registration No: 1397386 (Wales)