From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3F1EC3F2C6 for ; Tue, 3 Mar 2020 14:14:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 829CA20838 for ; Tue, 3 Mar 2020 14:14:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=stapelberg-ch.20150623.gappssmtp.com header.i=@stapelberg-ch.20150623.gappssmtp.com header.b="WxAW7Dq+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 829CA20838 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=stapelberg.ch Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 19A8B6B0007; Tue, 3 Mar 2020 09:14:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 14AE56B0008; Tue, 3 Mar 2020 09:14:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05FE36B000A; Tue, 3 Mar 2020 09:14:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0192.hostedemail.com [216.40.44.192]) by kanga.kvack.org (Postfix) with ESMTP id E21D56B0007 for ; Tue, 3 Mar 2020 09:13:59 -0500 (EST) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 9950A22000 for ; Tue, 3 Mar 2020 14:13:59 +0000 (UTC) X-FDA: 76554244998.09.van86_2493a6b27e913 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id C5F151803DCE7 for ; Tue, 3 Mar 2020 14:04:11 +0000 (UTC) X-HE-Tag: van86_2493a6b27e913 X-Filterd-Recvd-Size: 4712 Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Tue, 3 Mar 2020 14:04:11 +0000 (UTC) Received: by mail-oi1-f176.google.com with SMTP id t24so3116232oij.3 for ; Tue, 03 Mar 2020 06:04:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stapelberg-ch.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=5SOsGdj13iAaTZ5WnVwP1r7WHAp30ZWrAWiWCxl86dM=; b=WxAW7Dq+cdbY0wtzLnAM2VlH2TSqf4NXmWBVUykvoeNKu0upZemvsUnyBkF6IVuIIi ge8LIPJGIc7tyrcvEi9PkbtSuNZqCtSW326CHfIEAuKScUYB3QAWYP6rjvWNYRrZVarD vDAU2Bz+ZyBQ/HQYTcnjBzps25tyC/mU4q9mqLUd72qwxTT7BW17y+WZZuvkJ0UouMZw y1F57GzeMqh3Lpt1FHacKHEjZgeSd+0o8Qp8f+ThBtHSTmcZPaWT6BigNNyqaIAVpQWL 0ZtY5+S2XM7utg8TdrU/zn9Ta5TszvzcKNVgNs34lSQOx3EnqjVJbhbtjVbWmxuRhTTU Ilig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=5SOsGdj13iAaTZ5WnVwP1r7WHAp30ZWrAWiWCxl86dM=; b=ccgYemVNkOUjkK5MdYbOr3LENl4gjgJZ22IWEdwl6CEblCnWspfOfGINAPF1rEsqjh zrNOi/iymJkTVc1Fibs+pRntKh0NQynH7l0jByeIffxSIF3gPX9PzIn5ZVzJMoQWwf8c 3k0XIXVMBlWf7SjXilr0/Xsyr8aULpUirwHd7JxrSl0W3jrzFg3FCsMqyEsdkvaqkVti vqIGNnJ4mMAJHn6SFWsD3CU8q5KDEKRMeopXxjsi8mmJiYRdTF/TxLlQBwIclOPdTOkS Jh9/haoakqk0sgg4nlcR7Yl8ynXndokIZdXGrwapIe9/zkT2JyQOGMBOi4p6J8KKqBgd gXKQ== X-Gm-Message-State: ANhLgQ1adiIOIBfve5c2pSktQoPGNiRwXWgHqMQsOqvNleu1FMyrdGsy qS4omJIvD76gp5npMt7P+f20H0jVHHIMqopz1HM1sg== X-Google-Smtp-Source: ADFU+vvDhfQa1++FCih/IiclAQTyIDsZwsxz4995CMfE9/X3UT6uuPnEn6drjU/bt5ti5p0OT6FOt9cdLWOZlWUaKhA= X-Received: by 2002:aca:c45:: with SMTP id i5mr2392065oiy.111.1583244250138; Tue, 03 Mar 2020 06:04:10 -0800 (PST) MIME-Version: 1.0 References: <20200303130421.GA5186@mtj.thefacebook.com> In-Reply-To: <20200303130421.GA5186@mtj.thefacebook.com> From: Michael Stapelberg Date: Tue, 3 Mar 2020 15:03:58 +0100 Message-ID: Subject: Re: [fuse-devel] Writing to FUSE via mmap extremely slow (sometimes) on some machines? To: Tejun Heo Cc: Miklos Szeredi , Jack Smith , fuse-devel , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.016267, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Here=E2=80=99s a /proc//stack from when the issue is happening: [<0>] balance_dirty_pages_ratelimited+0x2ca/0x3b0 [<0>] __handle_mm_fault+0xe6e/0x1280 [<0>] handle_mm_fault+0xbe/0x1d0 [<0>] __do_page_fault+0x249/0x4f0 [<0>] page_fault+0x1e/0x30 How can I obtain the numbers for the next step? Thanks, On Tue, Mar 3, 2020 at 2:04 PM Tejun Heo wrote: > > Hello, > > Sorry about the delay. > > On Wed, Feb 26, 2020 at 08:59:55PM +0100, Miklos Szeredi wrote: > > - apparently memcpy is copying downwards (from largest address to > > smallest address). Not sure why, when I run the reproducer, it copies > > upwards. > > - there's a slow batch of reads of the first ~4MB of data, then a > > quick writeback > > - there's a quick read of the rest (~95MB) of data, then a quick > > writeback of the same > > > > Plots of the whole and closeups of slow and quick segments attached. > > X axis is time, Y axis is offset. > > > > Tejun, could this behavior be attributed to dirty throttling? What > > would be the best way to trace this? > > Yeah, seems likely. Can you please try offcputime (or just sample > /proc/PID/stack) and see whether it's in balance dirty pages? > > https://github.com/iovisor/bcc/blob/master/tools/offcputime.py > > If it's dirty throttling, the next step would be watching the bdp > tracepoints to find out what kind of numbers it's getting. > > Thanks. > > -- > tejun