From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2FE5C10F29 for ; Mon, 9 Mar 2020 15:11:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A4AF0227BF for ; Mon, 9 Mar 2020 15:11:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=stapelberg-ch.20150623.gappssmtp.com header.i=@stapelberg-ch.20150623.gappssmtp.com header.b="kck6cnyF" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726893AbgCIPL4 (ORCPT ); Mon, 9 Mar 2020 11:11:56 -0400 Received: from mail-oi1-f193.google.com ([209.85.167.193]:32982 "EHLO mail-oi1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726814AbgCIPLz (ORCPT ); Mon, 9 Mar 2020 11:11:55 -0400 Received: by mail-oi1-f193.google.com with SMTP id q81so10504396oig.0 for ; Mon, 09 Mar 2020 08:11:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stapelberg-ch.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Pe7aGFw1rGoFxsniL3x3JXEjzeRykIDaBsTSdNntehk=; b=kck6cnyFOnxDUrLv5kIPXBPcnnmUki5xOjXr0Ll4OdHSy9KRg2k9gembkzBAaB3FX5 YPKdeBOoBICplH3R0P3Ed92zr6aRtUoTIDA70hEJ9QwnOjuCP9Ef9dPUv/ya0desWlpF aDZ86yQD48xEpu+1br/0JM2DvbncR38vrrZl+yDLUFLS/Y/q1wz+QvcwkRNGVD26+ifB KXjRXNmiuJUXdBeZZyBGwAlevBcE4k/dWyBBzHI/uAdRPyD7svpfAaE0cUy9QPFQw8oz F6Ek57Q0dfX71YBJOmGimLGdMQkJZGL52HgkqIN8movB/Hxy/ZtMPhYZmEWA9D4y5/dS VrRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Pe7aGFw1rGoFxsniL3x3JXEjzeRykIDaBsTSdNntehk=; b=c9p2SMtxAcldm39Djo+FjqS4/QXlvzxnBVuDTMTAjOWc3WgGwYmig3zVNv9pm+WHPd +rI1jS9gZr8JyLdgZa9Wtq8o9CUzCqISaXs8cC7RY+NLLLsWfu/dC2PExh8CQUsTszuD WM1lFlAMJqEn7o+hclXWu6SsjSfQfQBGX2T87ofmZWMn0Bo3iO7drMCe1FjxfPOiOyWn ttxZ0cs47fgTjkWPurENzMKNWkOR3rwy/VLdxJ51IbZBMzKaoBgXIg8aVwxaELH5zz1Y fsP8HIrhzpX3P+n5DiZUQGimK4PCObczU7VHSvkatiovqS7g9QnWJano+8ksFlmSOKMX hJUw== X-Gm-Message-State: ANhLgQ2d4csMULGsyhmHtXHr9tWI6mMeW51Pgg/L38ofPIf40MZuH1XP khakVoofb9U4+oatZgXtb/pyJeLArsNRyG40kKst+g== X-Google-Smtp-Source: ADFU+vus//NY1DKPpgrUSxgdA5Fo4qf81eTNXVXm9Fv6HrbESdN4nB6VGYPe85+xgUn+zMyQzxroKyM/OOowOynLuMI= X-Received: by 2002:aca:a98a:: with SMTP id s132mr4183617oie.75.1583766712761; Mon, 09 Mar 2020 08:11:52 -0700 (PDT) MIME-Version: 1.0 References: <20200303130421.GA5186@mtj.thefacebook.com> <20200303141311.GA189690@mtj.thefacebook.com> <20200303142512.GC189690@mtj.thefacebook.com> In-Reply-To: From: Michael Stapelberg Date: Mon, 9 Mar 2020 16:11:41 +0100 Message-ID: Subject: Re: [fuse-devel] Writing to FUSE via mmap extremely slow (sometimes) on some machines? To: Miklos Szeredi Cc: Tejun Heo , Jack Smith , fuse-devel , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Content-Type: multipart/mixed; boundary="00000000000052a69505a06d6c41" Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org --00000000000052a69505a06d6c41 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks for clarifying. I have modified the mmap test program (see attached) to optionally read in the entire file when the WORKAROUND=3D environment variable is set, thereby preventing the FUSE reads in the write phase. I can now see a batch of reads, followed by a batch of writes. What=E2=80=99s interesting: when polling using =E2=80=9Cwhile :; do grep ^B= di /sys/kernel/debug/bdi/0:93/stats; sleep 0.1; done=E2=80=9D and running the mmap test program, I see: BdiDirtied: 3566304 kB BdiWritten: 3563616 kB BdiWriteBandwidth: 13596 kBps BdiDirtied: 3566304 kB BdiWritten: 3563616 kB BdiWriteBandwidth: 13596 kBps BdiDirtied: 3566528 kB (+224 kB) <-- starting to dirty pages BdiWritten: 3564064 kB (+448 kB) <-- starting to write BdiWriteBandwidth: 10700 kBps <-- only bandwidth update! BdiDirtied: 3668224 kB (+ 101696 kB) <-- all pages dirtied BdiWritten: 3565632 kB (+1568 kB) BdiWriteBandwidth: 10700 kBps BdiDirtied: 3668224 kB BdiWritten: 3665536 kB (+ 99904 kB) <-- all pages written BdiWriteBandwidth: 10700 kBps BdiDirtied: 3668224 kB BdiWritten: 3665536 kB BdiWriteBandwidth: 10700 kBps This seems to suggest that the bandwidth measurements only capture the rising slope of the transfer, but not the bulk of the transfer itself, resulting in inaccurate measurements. This effect is worsened when the test program doesn=E2=80=99t pre-read the output file and hence the kernel gets fewer FUSE write requests out. On Mon, Mar 9, 2020 at 3:36 PM Miklos Szeredi wrote: > > On Mon, Mar 9, 2020 at 3:32 PM Michael Stapelberg > wrote: > > > > Here=E2=80=99s one more thing I noticed: when polling > > /sys/kernel/debug/bdi/0:93/stats, I see that BdiDirtied and BdiWritten > > remain at their original values while the kernel sends FUSE read > > requests, and only goes up when the kernel transitions into sending > > FUSE write requests. Notably, the page dirtying throttling happens in > > the read phase, which is most likely why the write bandwidth is > > (correctly) measured as 0. > > > > Do we have any ideas on why the kernel sends FUSE reads at all? > > Memory writes (stores) need the memory page to be up-to-date wrt. the > backing file before proceeding. This means that if the page hasn't > yet been cached by the kernel, it needs to be read first. > > Thanks, > Miklos --00000000000052a69505a06d6c41 Content-Type: text/x-csrc; charset="US-ASCII"; name="mmap.c" Content-Disposition: attachment; filename="mmap.c" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_k7kl7x8h0 I2luY2x1ZGUgPHN5cy90eXBlcy5oPgojaW5jbHVkZSA8c3lzL3N0YXQuaD4KI2luY2x1ZGUgPHN5 cy9tbWFuLmg+IAojaW5jbHVkZSA8ZmNudGwuaD4KI2luY2x1ZGUgPHN0cmluZy5oPgojaW5jbHVk ZSA8c3RkaW8uaD4KI2luY2x1ZGUgPHN0ZGxpYi5oPgojaW5jbHVkZSA8dW5pc3RkLmg+CiNpbmNs dWRlIDxzdGRpbnQuaD4KCi8qCiAqIEFuIGltcGxlbWVudGF0aW9uIG9mIGNvcHkgKCJjcCIpIHRo YXQgdXNlcyBtZW1vcnkgbWFwcy4gIFZhcmlvdXMKICogZXJyb3IgY2hlY2tpbmcgaGFzIGJlZW4g cmVtb3ZlZCB0byBwcm9tb3RlIHJlYWRhYmlsaXR5CiAqLwoKLy8gV2hlcmUgd2Ugd2FudCB0aGUg c291cmNlIGZpbGUncyBtZW1vcnkgbWFwIHRvIGxpdmUgaW4gdmlydHVhbCBtZW1vcnkKLy8gVGhl IGRlc3RpbmF0aW9uIGZpbGUgcmVzaWRlcyBpbW1lZGlhdGVseSBhZnRlciB0aGUgc291cmNlIGZp bGUKI2RlZmluZSBNQVBfTE9DQVRJT04gMHg2MTAwCgppbnQgbWFpbiAoaW50IGFyZ2MsIGNoYXIg KmFyZ3ZbXSkgewogaW50IGZkaW4sIGZkb3V0OwogY2hhciAqc3JjLCAqZHN0Owogc3RydWN0IHN0 YXQgc3RhdGJ1ZjsKIG9mZl90IGZpbGVTaXplID0gMDsKCiBpZiAoYXJnYyAhPSAzKSB7CiAgIHBy aW50ZiAoInVzYWdlOiBhLm91dCA8ZnJvbWZpbGU+IDx0b2ZpbGU+XG4iKTsKICAgZXhpdCgwKTsK IH0KCiAvKiBvcGVuIHRoZSBpbnB1dCBmaWxlICovCiBpZiAoKGZkaW4gPSBvcGVuIChhcmd2WzFd LCBPX1JET05MWSkpIDwgMCkgewogICBwcmludGYgKCJjYW4ndCBvcGVuICVzIGZvciByZWFkaW5n XG4iLCBhcmd2WzFdKTsKICAgZXhpdCgwKTsKIH0KCiAvKiBvcGVuL2NyZWF0ZSB0aGUgb3V0cHV0 IGZpbGUgKi8KIGlmICgoZmRvdXQgPSBvcGVuIChhcmd2WzJdLCBPX1JEV1IgfCBPX0NSRUFUIHwg T19UUlVOQywgMDYwMCkpIDwgMCkgewogICBwcmludGYgKCJjYW4ndCBjcmVhdGUgJXMgZm9yIHdy aXRpbmdcbiIsIGFyZ3ZbMl0pOwogICBleGl0KDApOwogfQogCiAvKiBmaW5kIHNpemUgb2YgaW5w dXQgZmlsZSAqLwogZnN0YXQgKGZkaW4sJnN0YXRidWYpIDsKIGZpbGVTaXplID0gc3RhdGJ1Zi5z dF9zaXplOwogCiAvKiBnbyB0byB0aGUgbG9jYXRpb24gY29ycmVzcG9uZGluZyB0byB0aGUgbGFz dCBieXRlICovCiBpZiAobHNlZWsgKGZkb3V0LCBmaWxlU2l6ZSAtIDEsIFNFRUtfU0VUKSA9PSAt MSkgewogICBwcmludGYgKCJsc2VlayBlcnJvclxuIik7CiAgIGV4aXQoMCk7CiB9CiAKIC8qIHdy aXRlIGEgZHVtbXkgYnl0ZSBhdCB0aGUgbGFzdCBsb2NhdGlvbiAqLwogd3JpdGUgKGZkb3V0LCAi IiwgMSk7CiAKIC8qIAogICogbWVtb3J5IG1hcCB0aGUgaW5wdXQgZmlsZS4gIE9ubHkgdGhlIGZp cnN0IHR3byBhcmd1bWVudHMgYXJlCiAgKiBpbnRlcmVzdGluZzogMSkgdGhlIGxvY2F0aW9uIGFu ZCAyKSB0aGUgc2l6ZSBvZiB0aGUgbWVtb3J5IG1hcCAKICAqIGluIHZpcnR1YWwgbWVtb3J5IHNw YWNlLiBOb3RlIHRoYXQgdGhlIGxvY2F0aW9uIGlzIG9ubHkgYSAiaGludCI7CiAgKiB0aGUgT1Mg Y2FuIGNob29zZSB0byByZXR1cm4gYSBkaWZmZXJlbnQgdmlydHVhbCBtZW1vcnkgYWRkcmVzcy4K ICAqIFRoaXMgaXMgaWxsdXN0cmF0ZWQgYnkgdGhlIHByaW50ZiBjb21tYW5kIGJlbG93LgogKi8K CiBzcmMgPSBtbWFwICgodm9pZCopIE1BUF9MT0NBVElPTiwgZmlsZVNpemUsIAoJICAgICBQUk9U X1JFQUQsIE1BUF9TSEFSRUQgfCBNQVBfUE9QVUxBVEUsIGZkaW4sIDApOwoKIC8qIG1lbW9yeSBt YXAgdGhlIG91dHB1dCBmaWxlIGFmdGVyIHRoZSBpbnB1dCBmaWxlICovCiBkc3QgPSBtbWFwICgo dm9pZCopIE1BUF9MT0NBVElPTiArIGZpbGVTaXplICwgZmlsZVNpemUgLCAKCSAgICAgUFJPVF9S RUFEIHwgUFJPVF9XUklURSwgTUFQX1NIQVJFRCwgZmRvdXQsIDApOwoKCiBwcmludGYoInBpZDog JWRcbiIsIGdldHBpZCgpKTsKIHByaW50ZigiTWFwcGVkIHNyYzogMHglcCAgYW5kIGRzdDogMHgl cFxuIixzcmMsZHN0KTsKCiBpZiAoZ2V0ZW52KCJXT1JLQVJPVU5EIikgIT0gTlVMTCkgewogICBw cmludGYoIndvcmthcm91bmQ6IHJlYWRpbmcgb3V0cHV0IGZpbGUgYmVmb3JlIGRpcnR5aW5nIGl0 cyBwYWdlc1xuIik7CiAgIHVpbnQ4X3Qgc3VtID0gMDsKICAgdWludDhfdCAqcHRyID0gKHVpbnQ4 X3QqKWRzdDsKICAgZm9yIChvZmZfdCBpID0gMDsgaSA8IGZpbGVTaXplOyBpKyspIHsKICAgICBz dW0gKz0gKnB0cjsKICAgICBwdHIrKzsKICAgfQogICBwcmludGYoInN1bTogJWRcbiIsIHN1bSk7 CiAgIHNsZWVwKDEpOwogICBwcmludGYoIndyaXRpbmdcbiIpOwogfQoKIC8qIENvcHkgdGhlIGlu cHV0IGZpbGUgdG8gdGhlIG91dHB1dCBmaWxlICovCiBtZW1jcHkgKGRzdCwgc3JjLCBmaWxlU2l6 ZSk7CgogcHJpbnRmKCJtZW1jcHkgZG9uZVxuIik7CgogLy8gd2Ugc2hvdWxkIHByb2JhYmx5IHVu bWFwIG1lbW9yeSBhbmQgY2xvc2UgdGhlIGZpbGVzCn0gLyogbWFpbiAqLwo= --00000000000052a69505a06d6c41-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E10BC10F27 for ; Mon, 9 Mar 2020 15:11:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4273320873 for ; Mon, 9 Mar 2020 15:11:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=stapelberg-ch.20150623.gappssmtp.com header.i=@stapelberg-ch.20150623.gappssmtp.com header.b="kck6cnyF" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4273320873 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=stapelberg.ch Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DFAFB6B0005; Mon, 9 Mar 2020 11:11:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DD2FF6B0006; Mon, 9 Mar 2020 11:11:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE8256B0007; Mon, 9 Mar 2020 11:11:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0230.hostedemail.com [216.40.44.230]) by kanga.kvack.org (Postfix) with ESMTP id B70C56B0005 for ; Mon, 9 Mar 2020 11:11:54 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 7A20B5DF1 for ; Mon, 9 Mar 2020 15:11:54 +0000 (UTC) X-FDA: 76576163748.06.can73_67debeb86a115 X-HE-Tag: can73_67debeb86a115 X-Filterd-Recvd-Size: 10046 Received: from mail-oi1-f196.google.com (mail-oi1-f196.google.com [209.85.167.196]) by imf32.hostedemail.com (Postfix) with ESMTP for ; Mon, 9 Mar 2020 15:11:53 +0000 (UTC) Received: by mail-oi1-f196.google.com with SMTP id l12so10456536oil.9 for ; Mon, 09 Mar 2020 08:11:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stapelberg-ch.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Pe7aGFw1rGoFxsniL3x3JXEjzeRykIDaBsTSdNntehk=; b=kck6cnyFOnxDUrLv5kIPXBPcnnmUki5xOjXr0Ll4OdHSy9KRg2k9gembkzBAaB3FX5 YPKdeBOoBICplH3R0P3Ed92zr6aRtUoTIDA70hEJ9QwnOjuCP9Ef9dPUv/ya0desWlpF aDZ86yQD48xEpu+1br/0JM2DvbncR38vrrZl+yDLUFLS/Y/q1wz+QvcwkRNGVD26+ifB KXjRXNmiuJUXdBeZZyBGwAlevBcE4k/dWyBBzHI/uAdRPyD7svpfAaE0cUy9QPFQw8oz F6Ek57Q0dfX71YBJOmGimLGdMQkJZGL52HgkqIN8movB/Hxy/ZtMPhYZmEWA9D4y5/dS VrRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Pe7aGFw1rGoFxsniL3x3JXEjzeRykIDaBsTSdNntehk=; b=U188hYjWmeEKBQtf6awuOM/BtjayCnnheE3xk+QELxfmU6x3Lk/Y3bCgEDtAVm//9Y PHqzt1opG+Uad/CRW42t1ZxIbuvGaivWnFi/3+B4H9LLp9nytUNyQEOQY9FeGvBtzLaX M0oPRXIbyrL1MhvJUCBF4V4tN7FBJkrGXO5L/Kbv3qOohC04icOlOto0/dPYzbl5Y7Ro LiTFUbV7uvS3eltXXiWz3w9gusVTcd3F4Z7dvtdfQRqbUdqpEeUPqQwFvO29CaR9GShn lIsse9LVWGg6BeVYQU0dIpU4cYkSstGyPe4LSffdciAn9zjZX1mSxM2th4LsHZqZMD4O KFAg== X-Gm-Message-State: ANhLgQ0Ta87EPErtbPBZ0iDUFEeJN4OTOmzEayOgeTcZhiyrlkPB8EFw NBZLIGRhTc6MQit3REG4RAMpOcSUbdw8xMtHKGLkOA== X-Google-Smtp-Source: ADFU+vus//NY1DKPpgrUSxgdA5Fo4qf81eTNXVXm9Fv6HrbESdN4nB6VGYPe85+xgUn+zMyQzxroKyM/OOowOynLuMI= X-Received: by 2002:aca:a98a:: with SMTP id s132mr4183617oie.75.1583766712761; Mon, 09 Mar 2020 08:11:52 -0700 (PDT) MIME-Version: 1.0 References: <20200303130421.GA5186@mtj.thefacebook.com> <20200303141311.GA189690@mtj.thefacebook.com> <20200303142512.GC189690@mtj.thefacebook.com> In-Reply-To: From: Michael Stapelberg Date: Mon, 9 Mar 2020 16:11:41 +0100 Message-ID: Subject: Re: [fuse-devel] Writing to FUSE via mmap extremely slow (sometimes) on some machines? To: Miklos Szeredi Cc: Tejun Heo , Jack Smith , fuse-devel , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Content-Type: multipart/mixed; boundary="00000000000052a69505a06d6c41" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --00000000000052a69505a06d6c41 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks for clarifying. I have modified the mmap test program (see attached) to optionally read in the entire file when the WORKAROUND=3D environment variable is set, thereby preventing the FUSE reads in the write phase. I can now see a batch of reads, followed by a batch of writes. What=E2=80=99s interesting: when polling using =E2=80=9Cwhile :; do grep ^B= di /sys/kernel/debug/bdi/0:93/stats; sleep 0.1; done=E2=80=9D and running the mmap test program, I see: BdiDirtied: 3566304 kB BdiWritten: 3563616 kB BdiWriteBandwidth: 13596 kBps BdiDirtied: 3566304 kB BdiWritten: 3563616 kB BdiWriteBandwidth: 13596 kBps BdiDirtied: 3566528 kB (+224 kB) <-- starting to dirty pages BdiWritten: 3564064 kB (+448 kB) <-- starting to write BdiWriteBandwidth: 10700 kBps <-- only bandwidth update! BdiDirtied: 3668224 kB (+ 101696 kB) <-- all pages dirtied BdiWritten: 3565632 kB (+1568 kB) BdiWriteBandwidth: 10700 kBps BdiDirtied: 3668224 kB BdiWritten: 3665536 kB (+ 99904 kB) <-- all pages written BdiWriteBandwidth: 10700 kBps BdiDirtied: 3668224 kB BdiWritten: 3665536 kB BdiWriteBandwidth: 10700 kBps This seems to suggest that the bandwidth measurements only capture the rising slope of the transfer, but not the bulk of the transfer itself, resulting in inaccurate measurements. This effect is worsened when the test program doesn=E2=80=99t pre-read the output file and hence the kernel gets fewer FUSE write requests out. On Mon, Mar 9, 2020 at 3:36 PM Miklos Szeredi wrote: > > On Mon, Mar 9, 2020 at 3:32 PM Michael Stapelberg > wrote: > > > > Here=E2=80=99s one more thing I noticed: when polling > > /sys/kernel/debug/bdi/0:93/stats, I see that BdiDirtied and BdiWritten > > remain at their original values while the kernel sends FUSE read > > requests, and only goes up when the kernel transitions into sending > > FUSE write requests. Notably, the page dirtying throttling happens in > > the read phase, which is most likely why the write bandwidth is > > (correctly) measured as 0. > > > > Do we have any ideas on why the kernel sends FUSE reads at all? > > Memory writes (stores) need the memory page to be up-to-date wrt. the > backing file before proceeding. This means that if the page hasn't > yet been cached by the kernel, it needs to be read first. > > Thanks, > Miklos --00000000000052a69505a06d6c41 Content-Type: text/x-csrc; charset="US-ASCII"; name="mmap.c" Content-Disposition: attachment; filename="mmap.c" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_k7kl7x8h0 I2luY2x1ZGUgPHN5cy90eXBlcy5oPgojaW5jbHVkZSA8c3lzL3N0YXQuaD4KI2luY2x1ZGUgPHN5 cy9tbWFuLmg+IAojaW5jbHVkZSA8ZmNudGwuaD4KI2luY2x1ZGUgPHN0cmluZy5oPgojaW5jbHVk ZSA8c3RkaW8uaD4KI2luY2x1ZGUgPHN0ZGxpYi5oPgojaW5jbHVkZSA8dW5pc3RkLmg+CiNpbmNs dWRlIDxzdGRpbnQuaD4KCi8qCiAqIEFuIGltcGxlbWVudGF0aW9uIG9mIGNvcHkgKCJjcCIpIHRo YXQgdXNlcyBtZW1vcnkgbWFwcy4gIFZhcmlvdXMKICogZXJyb3IgY2hlY2tpbmcgaGFzIGJlZW4g cmVtb3ZlZCB0byBwcm9tb3RlIHJlYWRhYmlsaXR5CiAqLwoKLy8gV2hlcmUgd2Ugd2FudCB0aGUg c291cmNlIGZpbGUncyBtZW1vcnkgbWFwIHRvIGxpdmUgaW4gdmlydHVhbCBtZW1vcnkKLy8gVGhl IGRlc3RpbmF0aW9uIGZpbGUgcmVzaWRlcyBpbW1lZGlhdGVseSBhZnRlciB0aGUgc291cmNlIGZp bGUKI2RlZmluZSBNQVBfTE9DQVRJT04gMHg2MTAwCgppbnQgbWFpbiAoaW50IGFyZ2MsIGNoYXIg KmFyZ3ZbXSkgewogaW50IGZkaW4sIGZkb3V0OwogY2hhciAqc3JjLCAqZHN0Owogc3RydWN0IHN0 YXQgc3RhdGJ1ZjsKIG9mZl90IGZpbGVTaXplID0gMDsKCiBpZiAoYXJnYyAhPSAzKSB7CiAgIHBy aW50ZiAoInVzYWdlOiBhLm91dCA8ZnJvbWZpbGU+IDx0b2ZpbGU+XG4iKTsKICAgZXhpdCgwKTsK IH0KCiAvKiBvcGVuIHRoZSBpbnB1dCBmaWxlICovCiBpZiAoKGZkaW4gPSBvcGVuIChhcmd2WzFd LCBPX1JET05MWSkpIDwgMCkgewogICBwcmludGYgKCJjYW4ndCBvcGVuICVzIGZvciByZWFkaW5n XG4iLCBhcmd2WzFdKTsKICAgZXhpdCgwKTsKIH0KCiAvKiBvcGVuL2NyZWF0ZSB0aGUgb3V0cHV0 IGZpbGUgKi8KIGlmICgoZmRvdXQgPSBvcGVuIChhcmd2WzJdLCBPX1JEV1IgfCBPX0NSRUFUIHwg T19UUlVOQywgMDYwMCkpIDwgMCkgewogICBwcmludGYgKCJjYW4ndCBjcmVhdGUgJXMgZm9yIHdy aXRpbmdcbiIsIGFyZ3ZbMl0pOwogICBleGl0KDApOwogfQogCiAvKiBmaW5kIHNpemUgb2YgaW5w dXQgZmlsZSAqLwogZnN0YXQgKGZkaW4sJnN0YXRidWYpIDsKIGZpbGVTaXplID0gc3RhdGJ1Zi5z dF9zaXplOwogCiAvKiBnbyB0byB0aGUgbG9jYXRpb24gY29ycmVzcG9uZGluZyB0byB0aGUgbGFz dCBieXRlICovCiBpZiAobHNlZWsgKGZkb3V0LCBmaWxlU2l6ZSAtIDEsIFNFRUtfU0VUKSA9PSAt MSkgewogICBwcmludGYgKCJsc2VlayBlcnJvclxuIik7CiAgIGV4aXQoMCk7CiB9CiAKIC8qIHdy aXRlIGEgZHVtbXkgYnl0ZSBhdCB0aGUgbGFzdCBsb2NhdGlvbiAqLwogd3JpdGUgKGZkb3V0LCAi IiwgMSk7CiAKIC8qIAogICogbWVtb3J5IG1hcCB0aGUgaW5wdXQgZmlsZS4gIE9ubHkgdGhlIGZp cnN0IHR3byBhcmd1bWVudHMgYXJlCiAgKiBpbnRlcmVzdGluZzogMSkgdGhlIGxvY2F0aW9uIGFu ZCAyKSB0aGUgc2l6ZSBvZiB0aGUgbWVtb3J5IG1hcCAKICAqIGluIHZpcnR1YWwgbWVtb3J5IHNw YWNlLiBOb3RlIHRoYXQgdGhlIGxvY2F0aW9uIGlzIG9ubHkgYSAiaGludCI7CiAgKiB0aGUgT1Mg Y2FuIGNob29zZSB0byByZXR1cm4gYSBkaWZmZXJlbnQgdmlydHVhbCBtZW1vcnkgYWRkcmVzcy4K ICAqIFRoaXMgaXMgaWxsdXN0cmF0ZWQgYnkgdGhlIHByaW50ZiBjb21tYW5kIGJlbG93LgogKi8K CiBzcmMgPSBtbWFwICgodm9pZCopIE1BUF9MT0NBVElPTiwgZmlsZVNpemUsIAoJICAgICBQUk9U X1JFQUQsIE1BUF9TSEFSRUQgfCBNQVBfUE9QVUxBVEUsIGZkaW4sIDApOwoKIC8qIG1lbW9yeSBt YXAgdGhlIG91dHB1dCBmaWxlIGFmdGVyIHRoZSBpbnB1dCBmaWxlICovCiBkc3QgPSBtbWFwICgo dm9pZCopIE1BUF9MT0NBVElPTiArIGZpbGVTaXplICwgZmlsZVNpemUgLCAKCSAgICAgUFJPVF9S RUFEIHwgUFJPVF9XUklURSwgTUFQX1NIQVJFRCwgZmRvdXQsIDApOwoKCiBwcmludGYoInBpZDog JWRcbiIsIGdldHBpZCgpKTsKIHByaW50ZigiTWFwcGVkIHNyYzogMHglcCAgYW5kIGRzdDogMHgl cFxuIixzcmMsZHN0KTsKCiBpZiAoZ2V0ZW52KCJXT1JLQVJPVU5EIikgIT0gTlVMTCkgewogICBw cmludGYoIndvcmthcm91bmQ6IHJlYWRpbmcgb3V0cHV0IGZpbGUgYmVmb3JlIGRpcnR5aW5nIGl0 cyBwYWdlc1xuIik7CiAgIHVpbnQ4X3Qgc3VtID0gMDsKICAgdWludDhfdCAqcHRyID0gKHVpbnQ4 X3QqKWRzdDsKICAgZm9yIChvZmZfdCBpID0gMDsgaSA8IGZpbGVTaXplOyBpKyspIHsKICAgICBz dW0gKz0gKnB0cjsKICAgICBwdHIrKzsKICAgfQogICBwcmludGYoInN1bTogJWRcbiIsIHN1bSk7 CiAgIHNsZWVwKDEpOwogICBwcmludGYoIndyaXRpbmdcbiIpOwogfQoKIC8qIENvcHkgdGhlIGlu cHV0IGZpbGUgdG8gdGhlIG91dHB1dCBmaWxlICovCiBtZW1jcHkgKGRzdCwgc3JjLCBmaWxlU2l6 ZSk7CgogcHJpbnRmKCJtZW1jcHkgZG9uZVxuIik7CgogLy8gd2Ugc2hvdWxkIHByb2JhYmx5IHVu bWFwIG1lbW9yeSBhbmQgY2xvc2UgdGhlIGZpbGVzCn0gLyogbWFpbiAqLwo= --00000000000052a69505a06d6c41--