All of lore.kernel.org
 help / color / mirror / Atom feed
From: Tejun Heo <tj@kernel.org>
To: Miklos Szeredi <miklos@szeredi.hu>
Cc: Michael Stapelberg <michael+lkml@stapelberg.ch>,
	Jack Smith <smith.jack.sidman@gmail.com>,
	fuse-devel <fuse-devel@lists.sourceforge.net>,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org
Subject: Re: [fuse-devel] Writing to FUSE via mmap extremely slow (sometimes) on some machines?
Date: Tue, 3 Mar 2020 08:04:21 -0500	[thread overview]
Message-ID: <20200303130421.GA5186@mtj.thefacebook.com> (raw)
In-Reply-To: <CAJfpegtkEU9=3cvy8VNr4SnojErYFOTaCzUZLYvMuQMi050bPQ@mail.gmail.com>

Hello,

Sorry about the delay.

On Wed, Feb 26, 2020 at 08:59:55PM +0100, Miklos Szeredi wrote:
> - apparently memcpy is copying downwards (from largest address to
> smallest address).  Not sure why, when I run the reproducer, it copies
> upwards.
> - there's a slow batch of reads of the first ~4MB of data, then a
> quick writeback
> - there's a quick read of the rest (~95MB) of data, then a quick
> writeback of the same
> 
> Plots of the whole and closeups of slow and quick segments attached.
> X axis is time, Y axis is offset.
> 
> Tejun, could this behavior be attributed to dirty throttling?  What
> would be the best way to trace this?

Yeah, seems likely. Can you please try offcputime (or just sample
/proc/PID/stack) and see whether it's in balance dirty pages?

  https://github.com/iovisor/bcc/blob/master/tools/offcputime.py

If it's dirty throttling, the next step would be watching the bdp
tracepoints to find out what kind of numbers it's getting.

Thanks.

-- 
tejun

  parent reply	other threads:[~2020-03-03 13:04 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-02-24 13:29 Writing to FUSE via mmap extremely slow (sometimes) on some machines? Michael Stapelberg
     [not found] ` <CACQJH27s4HKzPgUkVT+FXWLGqJAAMYEkeKe7cidcesaYdE2Vog@mail.gmail.com>
     [not found]   ` <CANnVG6=Ghu5r44mTkr0uXx_ZrrWo2N5C_UEfM59110Zx+HApzw@mail.gmail.com>
     [not found]     ` <CAJfpegvzhfO7hg1sb_ttQF=dmBeg80WVkV8srF3VVYHw9ybV0w@mail.gmail.com>
     [not found]       ` <CANnVG6kSJJw-+jtjh-ate7CC3CsB2=ugnQpA9ACGFdMex8sftg@mail.gmail.com>
2020-02-26 19:59         ` [fuse-devel] " Miklos Szeredi
2020-03-03 10:34           ` Michael Stapelberg
2020-03-03 10:34             ` Michael Stapelberg
2020-03-03 13:04           ` Tejun Heo [this message]
2020-03-03 14:03             ` Michael Stapelberg
2020-03-03 14:03               ` Michael Stapelberg
2020-03-03 14:13               ` Tejun Heo
2020-03-03 14:21                 ` Michael Stapelberg
2020-03-03 14:21                   ` Michael Stapelberg
2020-03-03 14:25                   ` Tejun Heo
2020-03-05 14:45                     ` Michael Stapelberg
2020-03-09 14:32                       ` Michael Stapelberg
2020-03-09 14:32                         ` Michael Stapelberg
2020-03-09 14:36                         ` Miklos Szeredi
2020-03-09 14:36                           ` Miklos Szeredi
2020-03-09 15:11                           ` Michael Stapelberg
2020-03-09 15:11                             ` Michael Stapelberg
2020-03-12 15:45                             ` Michael Stapelberg
2020-03-12 15:45                               ` Michael Stapelberg

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200303130421.GA5186@mtj.thefacebook.com \
    --to=tj@kernel.org \
    --cc=fuse-devel@lists.sourceforge.net \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=michael+lkml@stapelberg.ch \
    --cc=miklos@szeredi.hu \
    --cc=smith.jack.sidman@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.