From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22153C3F2CD for ; Tue, 3 Mar 2020 14:25:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D7F6820838 for ; Tue, 3 Mar 2020 14:25:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="W07icVy3" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D7F6820838 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8494B6B0005; Tue, 3 Mar 2020 09:25:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7D3D06B0006; Tue, 3 Mar 2020 09:25:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 674706B0007; Tue, 3 Mar 2020 09:25:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0130.hostedemail.com [216.40.44.130]) by kanga.kvack.org (Postfix) with ESMTP id 493C76B0005 for ; Tue, 3 Mar 2020 09:25:15 -0500 (EST) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id E05DDB788 for ; Tue, 3 Mar 2020 14:25:14 +0000 (UTC) X-FDA: 76554273348.02.event51_4add763679944 X-HE-Tag: event51_4add763679944 X-Filterd-Recvd-Size: 4465 Received: from mail-qt1-f196.google.com (mail-qt1-f196.google.com [209.85.160.196]) by imf17.hostedemail.com (Postfix) with ESMTP for ; Tue, 3 Mar 2020 14:25:14 +0000 (UTC) Received: by mail-qt1-f196.google.com with SMTP id o10so2891931qtr.7 for ; Tue, 03 Mar 2020 06:25:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=qz8bhnv0I8TXyq1XMurQya8Qmu0LrE3c6IwRtArQcqU=; b=W07icVy3fNTM0LqYTX3/r2GpDfvgpWX1kFWwW/kHp/eoQeC7MVE9uLi1RmAF6d8kix P8EMRVI2CKrF1rksdOdFwqd9iTKrUK+ro5WSp/xHU0j/DvYkQ4m645UWU9RKRFFJrHvA AwH/2jkSgmRXpj2Asprh6lDKSVC0TfIK+rli5kfRICBhI9WYwbMEwwVJsewDI4E9etvt 7iOWAzF8CTjJB6U+pBd7EkMAAbXp9QeVGJRU+aHQVkyqFUQf5oZHv+stfZAHhWv4o0ox Xxf4VRclR19n9S4TgwKY1mCShsOq9VZ7zHaz3UPlDebPZb7GhwnrMw52yvpb8Z1A83Za OpeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to; bh=qz8bhnv0I8TXyq1XMurQya8Qmu0LrE3c6IwRtArQcqU=; b=uQpyagHJV7ha8uR+rxYZoAsPoFXCzZcGzBSI0P1w0Iaoxtemdnxe7EJT6bntjtkVSb bYcR/WVqM9jOw86O6vGiZYeP7vudp8bFr8FSYxMEX7otSNeADrlYVHhmOcS+ZATIaf62 Q1KFSdoBWlGyhOgvdpkC+0SxWuO2uJWn7p5MMNlBVpooXhR6HXv6WIXhFFCsZEofq+0u NchiGY4RVvvITHYkzRNi+8PmXZkRdrdjavJfryYhipqLHjfcr5pQTdO0cq5+J6+6fUyf 08JtkEdk3ly3FuQ4cwSHarzegKyPpLnRTQymoMxbISkD8w8V729bAe7w6ZcwARB5f97+ nivQ== X-Gm-Message-State: ANhLgQ2ypWxXsMCyxx6EurTcoz2fedHgT0+pCcjiZT0ryIl5uMMP41FS YSobBt54o0tNZK7u8qbk03s= X-Google-Smtp-Source: ADFU+vvUdf/gf6lmbS0a1RQDQG8eIk8sCAy+gJCldsN0M4lFk9ta89/AvaDgb2th4n6W+s3FHhxv9g== X-Received: by 2002:ac8:1688:: with SMTP id r8mr4681278qtj.144.1583245513690; Tue, 03 Mar 2020 06:25:13 -0800 (PST) Received: from localhost ([2620:10d:c091:500::7f70]) by smtp.gmail.com with ESMTPSA id f89sm3436621qtb.7.2020.03.03.06.25.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Mar 2020 06:25:13 -0800 (PST) Date: Tue, 3 Mar 2020 09:25:12 -0500 From: Tejun Heo To: Michael Stapelberg Cc: Miklos Szeredi , Jack Smith , fuse-devel , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [fuse-devel] Writing to FUSE via mmap extremely slow (sometimes) on some machines? Message-ID: <20200303142512.GC189690@mtj.thefacebook.com> References: <20200303130421.GA5186@mtj.thefacebook.com> <20200303141311.GA189690@mtj.thefacebook.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Bogosity: Ham, tests=bogofilter, spamicity=0.000003, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Mar 03, 2020 at 03:21:47PM +0100, Michael Stapelberg wrote: > Find attached trace.log (cat /sys/kernel/debug/tracing/trace) and > fuse-debug.log (FUSE daemon with timestamps). > > Does that tell you something, or do we need more data? (If so, how?) This is likely the culprit. .... 1319822.406198: balance_dirty_pages: ... bdi_dirty=68 dirty_ratelimit=28 ... For whatever reason, bdp calculated that the dirty throttling threshold for the fuse device is 28 pages which is extremely low. Need to track down how that number came to be. I'm afraid from here on it'd mostly be reading source code and sprinkling printks around but the debugging really comes down to figuring out how we ended up with 68 and 28. Thanks. -- tejun