Debug cache miss

Debug cache miss

[Shay Gover]

Hi everyone,

Recently I'm getting complete I/O freezes (mouse still working).
I think, it's a cache miss.
Sometimes I see on dmesg this error:
kernel: perf: interrupt took too long (2575 > 2500), lowering
kernel.perf_event_max_sample_rate to 77400
(Numbers change)

How can I debug this? Where should I look?
Currently I'm watching cache_bypass_misses

Thanks,

Shay

Re: Debug cache miss

[Killian De Volder]

Cache misses should not be causing this, if the data is not found, a BIO (Kernel Block IO request) is submitted to the backing device, and everything should be fine.

I'm not familiar enough with the kernel code;
but if bcache is the cause of this, this would imply the bcache code is ran during the interrupt until it is done.
I suspect the kernel does not do this (I could be wrong).

Now in principle this is a warning, your computers should continue to run.

I have personally had bug with bcache in the past, but it either froze the kernel with a panic.
Or continue working (with some buts but still working).

Questions / tests :

1) Can you confirm the problem disappears when not using bcache ?
(But please try to still use all block devices you use when you are testing without bcache.)

2) When did this problem start ?

3) Does the computer freeze and stop, or continue ?

4) Check the smart of your devices: "smartctl -a /dev/sdX"

Killian De Volder


On 14-04-17 15:21, Shay Gover wrote:
> Hi everyone,
> 
> Recently I'm getting complete I/O freezes (mouse still working).
> I think, it's a cache miss.
> Sometimes I see on dmesg this error:
> kernel: perf: interrupt took too long (2575 > 2500), lowering
> kernel.perf_event_max_sample_rate to 77400
> (Numbers change)
> 
> How can I debug this? Where should I look?
> Currently I'm watching cache_bypass_misses
> 
> Thanks,
> 
> Shay
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Debug cache miss

[Kai Krakow]

Am Fri, 14 Apr 2017 16:21:00 +0300
schrieb Shay Gover <govershay@gmail.com>:

> Hi everyone,
> 
> Recently I'm getting complete I/O freezes (mouse still working).
> I think, it's a cache miss.
> Sometimes I see on dmesg this error:
> kernel: perf: interrupt took too long (2575 > 2500), lowering
> kernel.perf_event_max_sample_rate to 77400
> (Numbers change)
> 
> How can I debug this? Where should I look?
> Currently I'm watching cache_bypass_misses

Are you using VirtualBox somewhere? I only see such messages when
running VirtualBox guests with high IO load. But the IO stack is not
the problem here.

-- 
Regards,
Kai

Replies to list-only preferred.