Re: [fuse-devel] Writing to FUSE via mmap extremely slow (sometimes) on some machines?

[Michael Stapelberg <michael+lkml@stapelberg.ch>]

Here’s a /proc/<pid>/stack from when the issue is happening:

[<0>] balance_dirty_pages_ratelimited+0x2ca/0x3b0
[<0>] __handle_mm_fault+0xe6e/0x1280
[<0>] handle_mm_fault+0xbe/0x1d0
[<0>] __do_page_fault+0x249/0x4f0
[<0>] page_fault+0x1e/0x30

How can I obtain the numbers for the next step?

Thanks,

On Tue, Mar 3, 2020 at 2:04 PM Tejun Heo <tj@kernel.org> wrote:
>
> Hello,
>
> Sorry about the delay.
>
> On Wed, Feb 26, 2020 at 08:59:55PM +0100, Miklos Szeredi wrote:
> > - apparently memcpy is copying downwards (from largest address to
> > smallest address).  Not sure why, when I run the reproducer, it copies
> > upwards.
> > - there's a slow batch of reads of the first ~4MB of data, then a
> > quick writeback
> > - there's a quick read of the rest (~95MB) of data, then a quick
> > writeback of the same
> >
> > Plots of the whole and closeups of slow and quick segments attached.
> > X axis is time, Y axis is offset.
> >
> > Tejun, could this behavior be attributed to dirty throttling?  What
> > would be the best way to trace this?
>
> Yeah, seems likely. Can you please try offcputime (or just sample
> /proc/PID/stack) and see whether it's in balance dirty pages?
>
>   https://github.com/iovisor/bcc/blob/master/tools/offcputime.py
>
> If it's dirty throttling, the next step would be watching the bdp
> tracepoints to find out what kind of numbers it's getting.
>
> Thanks.
>
> --
> tejun