Fwd: [REGRESSION 6.0.x / 6.1.x] NULL dereferencing at tracing

Fwd: [REGRESSION 6.0.x / 6.1.x] NULL dereferencing at tracing

[postix]

> Can you apply this to see if it fixes it?
>
> I'm guessing there's a path to the release of the file descriptor where
> the ring buffer isn't allocated (and this expected it to be).
>
> I'll investigate further to see if I can find that path.
>
> -- Steve
>
> diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c 
> index 199759c73519..c1c7ce4c6ddb 100644 --- 
> a/kernel/trace/ring_buffer.c +++ b/kernel/trace/ring_buffer.c @@ 
> -937,6 +937,9 @@ void ring_buffer_wake_waiters(struct trace_buffer 
> *buffer, int cpu)   	struct ring_buffer_per_cpu *cpu_buffer;
>   	struct rb_irq_work *rbwork;
>   
> + if (!buffer) + return; +   	if (cpu == RING_BUFFER_ALL_CPUS) {
>   
>   		/* Wake up individual ones too. One level recursion */

Dear Steve,


I have tested your suggested patch using kernel 6.1.0-rc2, but 
unfortunately it didn't fix the issue for me.

Thank you for looking into it though!


Best Regards

--AD

Re: Fwd: [REGRESSION 6.0.x / 6.1.x] NULL dereferencing at tracing

[postix]

Hello everyone,

I have added lot's of debug printk's to see what's happening and I found 
that the "cpu" counter, which is used to access the buffer's array 
elements (cpu_buffer = buffer->buffers[cpu]) in the ring_buffer_wake_waiters
function, exceeds the maximum number of total of total cores, namely in 
my case 24, which means, it should only run from 0..23. However, upon 
debugging, it runs up to 31, and thus causing a NULL pointer dereference 
(&cpu_buffer->irq_work).

After adding a return statement in case cpu > 24, the bug is no longer 
reproducible.

You can find the diff between v6.1-rc2 and the patched version with 
added debug log in [1].
The corresponding dmesg output can be found in [2].

I hope this gives you a good hint to find the root cause!

[1] https://paste.opensuse.org/e60601aa
[2] https://paste.opensuse.org/bf1398ce

Re: [REGRESSION 6.0.x / 6.1.x] NULL dereferencing at tracing

[Steven Rostedt]

On Wed,  2 Nov 2022 15:57:56 +0000
postix@posteo.eu wrote:

> Hello everyone,
> 
> I have added lot's of debug printk's to see what's happening and I found 
> that the "cpu" counter, which is used to access the buffer's array 
> elements (cpu_buffer = buffer->buffers[cpu]) in the ring_buffer_wake_waiters
> function, exceeds the maximum number of total of total cores, namely in 
> my case 24, which means, it should only run from 0..23. However, upon 
> debugging, it runs up to 31, and thus causing a NULL pointer dereference 
> (&cpu_buffer->irq_work).
> 

Could you add this patch.

 https://lore.kernel.org/all/20221101191009.1e7378c8@rorschach.local.home/

Thanks,

-- Steve

Re: [REGRESSION 6.0.x / 6.1.x] NULL dereferencing at tracing

[postix]

On 02.11.22 17:03, Steven Rostedt wrote:
 > Could you add this patch.
 >
 > https://lore.kernel.org/all/20221101191009.1e7378c8@rorschach.local.home/


Thanks, this patch fixes the issue for me! Please see the final dmesg 
output [1].

[1] https://paste.opensuse.org/e8d4fa46


All the best

--AD

Re: [REGRESSION 6.0.x / 6.1.x] NULL dereferencing at tracing

[Steven Rostedt]

On Wed,  2 Nov 2022 16:36:29 +0000
postix@posteo.eu wrote:

> On 02.11.22 17:03, Steven Rostedt wrote:
>  > Could you add this patch.
>  >
>  > https://lore.kernel.org/all/20221101191009.1e7378c8@rorschach.local.home/  
> 
> 
> Thanks, this patch fixes the issue for me! Please see the final dmesg 
> output [1].
> 
> [1] https://paste.opensuse.org/e8d4fa46
> 

Yes that's known too. rasdaemon needs to be updated to use the
libtracefs library, which should fix all this.

-- Steve