[PATCH v3] tracing: Fix infinite loop in tracing_read_pipe on overflowed print_trace_line

[PATCH v3] tracing: Fix infinite loop in tracing_read_pipe on overflowed print_trace_line

[Yang Jihong]

print_trace_line may overflow seq_file buffer. If the event is not
consumed, the while loop keeps peeking this event, causing a infinite loop.

Signed-off-by: Yang Jihong <yangjihong1@huawei.com>
---
 kernel/trace/trace.c | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index a7fe0e115272..55733224fa88 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -6787,7 +6787,27 @@ tracing_read_pipe(struct file *filp, char __user *ubuf,
 
 		ret = print_trace_line(iter);
 		if (ret == TRACE_TYPE_PARTIAL_LINE) {
-			/* don't print partial lines */
+			/*
+			 * If one trace_line of the tracer overflows seq_file
+			 * buffer, trace_seq_to_user returns -EBUSY.
+			 * In this case, we need to consume it, otherwise,
+			 * while loop will peek this event next time,
+			 * resulting in an infinite loop.
+			 */
+			if (trace_seq_has_overflowed(&iter->seq)) {
+				/*
+				 * Here we only consider the case that one
+				 * print_trace_line() fills the entire trace_seq
+				 * in one shot, in that case, iter->seq.seq.len is zero,
+				 * we simply output a log of too long line to inform the user.
+				 */
+				iter->seq.full = 0;
+				trace_seq_puts(&iter->seq, "[LINE TOO BIG]\n");
+				trace_consume(iter);
+				break;
+			}
+
+			/* In other cases, don't print partial lines */
 			iter->seq.seq.len = save_len;
 			break;
 		}
-- 
2.30.GIT

Re: [PATCH v3] tracing: Fix infinite loop in tracing_read_pipe on overflowed print_trace_line

[Steven Rostedt]

On Thu, 24 Nov 2022 20:58:50 +0800
Yang Jihong <yangjihong1@huawei.com> wrote:

> print_trace_line may overflow seq_file buffer. If the event is not
> consumed, the while loop keeps peeking this event, causing a infinite loop.
> 
> Signed-off-by: Yang Jihong <yangjihong1@huawei.com>
> ---
>  kernel/trace/trace.c | 22 +++++++++++++++++++++-
>  1 file changed, 21 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
> index a7fe0e115272..55733224fa88 100644
> --- a/kernel/trace/trace.c
> +++ b/kernel/trace/trace.c
> @@ -6787,7 +6787,27 @@ tracing_read_pipe(struct file *filp, char __user *ubuf,
>  
>  		ret = print_trace_line(iter);
>  		if (ret == TRACE_TYPE_PARTIAL_LINE) {
> -			/* don't print partial lines */
> +			/*
> +			 * If one trace_line of the tracer overflows seq_file
> +			 * buffer, trace_seq_to_user returns -EBUSY.
> +			 * In this case, we need to consume it, otherwise,
> +			 * while loop will peek this event next time,
> +			 * resulting in an infinite loop.
> +			 */
> +			if (trace_seq_has_overflowed(&iter->seq)) {

The only way to get here is if the above is true, and that is not going to
cause the infinite loop. What does is if save_len == 0. In fact, that's
all you need to check for:

			if (save_len == 0) {

Should do the trick.

-- Steve



> +				/*
> +				 * Here we only consider the case that one
> +				 * print_trace_line() fills the entire trace_seq
> +				 * in one shot, in that case, iter->seq.seq.len is zero,
> +				 * we simply output a log of too long line to inform the user.
> +				 */
> +				iter->seq.full = 0;
> +				trace_seq_puts(&iter->seq, "[LINE TOO BIG]\n");
> +				trace_consume(iter);
> +				break;
> +			}
> +
> +			/* In other cases, don't print partial lines */
>  			iter->seq.seq.len = save_len;
>  			break;
>  		}

Re: [PATCH v3] tracing: Fix infinite loop in tracing_read_pipe on overflowed print_trace_line

[Yang Jihong]

On 2022/11/29 0:46, Steven Rostedt wrote:
> On Thu, 24 Nov 2022 20:58:50 +0800
> Yang Jihong <yangjihong1@huawei.com> wrote:
> 
>> print_trace_line may overflow seq_file buffer. If the event is not
>> consumed, the while loop keeps peeking this event, causing a infinite loop.
>>
>> Signed-off-by: Yang Jihong <yangjihong1@huawei.com>
>> ---
>>   kernel/trace/trace.c | 22 +++++++++++++++++++++-
>>   1 file changed, 21 insertions(+), 1 deletion(-)
>>
>> diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
>> index a7fe0e115272..55733224fa88 100644
>> --- a/kernel/trace/trace.c
>> +++ b/kernel/trace/trace.c
>> @@ -6787,7 +6787,27 @@ tracing_read_pipe(struct file *filp, char __user *ubuf,
>>   
>>   		ret = print_trace_line(iter);
>>   		if (ret == TRACE_TYPE_PARTIAL_LINE) {
>> -			/* don't print partial lines */
>> +			/*
>> +			 * If one trace_line of the tracer overflows seq_file
>> +			 * buffer, trace_seq_to_user returns -EBUSY.
>> +			 * In this case, we need to consume it, otherwise,
>> +			 * while loop will peek this event next time,
>> +			 * resulting in an infinite loop.
>> +			 */
>> +			if (trace_seq_has_overflowed(&iter->seq)) {
> 
> The only way to get here is if the above is true, and that is not going to
> cause the infinite loop. What does is if save_len == 0. In fact, that's
> all you need to check for:
> 
> 			if (save_len == 0) {
> 
> Should do the trick.
Yes, Has been tested to be feasible, will change in next version.

By the way, the problem mentioned in the v1 patch:
"Anyway, what is triggering this?"

The problem is caused by the print_line callback function 
(blk_tracer_print_line) of blktrace.
The blktrace does not filter out the events whose type is !=TRACK_BLK.
It parses and prints all trace events in the ring buffer as blktrace 
events. As a result, the seq_file buffer may overflow in the 
blk_log_dump_pdu function:

static void blk_log_dump_pdu()
{
...
         for (i = 0; i < pdu_len; i++) { // pdu_len may exceed the 
seq_file buffer size.

                 trace_seq_printf(s, "%s%02x",
                                  i == 0 ? "" : " ", pdu_buf[i]);

                 /*
                  * stop when the rest is just zeros and indicate so
                  * with a ".." appended
                  */
                 if (i == end && end != pdu_len - 1) {
                         trace_seq_puts(s, " ..) ");
                         return;
                 }
         }
...
}

The fixed patch has been sent:
https://lore.kernel.org/lkml/20221122040410.85113-1-yangjihong1@huawei.com/T/


Thanks,
Yang