From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E3BDC3F2C6 for ; Tue, 3 Mar 2020 10:34:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D7D3020838 for ; Tue, 3 Mar 2020 10:34:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=stapelberg-ch.20150623.gappssmtp.com header.i=@stapelberg-ch.20150623.gappssmtp.com header.b="qCfEHrQ7" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D7D3020838 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=stapelberg.ch Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8D3F96B0007; Tue, 3 Mar 2020 05:34:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8AA9A6B0008; Tue, 3 Mar 2020 05:34:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E9926B000A; Tue, 3 Mar 2020 05:34:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0244.hostedemail.com [216.40.44.244]) by kanga.kvack.org (Postfix) with ESMTP id 692AB6B0007 for ; Tue, 3 Mar 2020 05:34:35 -0500 (EST) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 3BDB46D95 for ; Tue, 3 Mar 2020 10:34:35 +0000 (UTC) X-FDA: 76553692110.13.joke49_623650f620b50 X-HE-Tag: joke49_623650f620b50 X-Filterd-Recvd-Size: 5152 Received: from mail-oi1-f195.google.com (mail-oi1-f195.google.com [209.85.167.195]) by imf08.hostedemail.com (Postfix) with ESMTP for ; Tue, 3 Mar 2020 10:34:34 +0000 (UTC) Received: by mail-oi1-f195.google.com with SMTP id 5so2516200oiy.4 for ; Tue, 03 Mar 2020 02:34:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stapelberg-ch.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=y9VmsIZewgFG4BQQWz1oxJk1kr5h4oup+Ad6oeZpXNM=; b=qCfEHrQ7hnwBLGa3e/zU5LGGzRjhKWQjwTHHZagx7KQCJEQMsUlf8RwsdIAo/1eg8t FxEnEM6KCt/vMuMdmwwGC7D6Jw0YEZ2B105Kuyt01r7gmJy1e4R69UBhRxvzgLAaVwkW zNcHaS3X2JSojJM/8uN7uRH/4WwLKME9guXU1izGZ2gdWEAx/dwWL2QdQ8Ukn/8AzFWp fH/Vy5XSCrE9wy5GTSQ8KdGgTnvA3ZY9jd7job4qwA3cNx04Elr923Pxk015XeCe5Xv9 s7pgAg5pcUvd3dEXLe4nBUS353e3eaGz1fS3bcjdHi0iS/TWoPswpN7+yrkO+yMZOOuX 2hCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=y9VmsIZewgFG4BQQWz1oxJk1kr5h4oup+Ad6oeZpXNM=; b=d1/2jLLMvKw/NGKfcbD22ZaWapNx0jATnJE9qXNi62A+Jltdwcmtg9LohQ8jPe7igx EWpRpX84w9zn1lpagmjcBQiXRB+t0CzgFjxsysOfUWiqStJyA0sLdBSP48tniuHzhXFJ ZG6rf8FLgCnnQetfsVcktB6kSXjEUjVNuTL8fugf7y+Uopgcf3M9+At09/+wPvrlE8Pg 0/G+FoovezIoOeZQDR2Gdk+NnUfTJnkSAUj658NKlmp1dqQfa8Hkp0Ur7i30Mrd1F/ZK 1nDgIhnKthikv1sM+3XIBPNYWgiaOD1cg+5dCj3wxqV1eQHwfzqPr9W9nsaAxqIH+SAm ReSQ== X-Gm-Message-State: ANhLgQ1B68+y45GKhA3SjODYU1TtPLoOSGGpRACnm8JAfcEFjo3scux3 gniJ/uEvs4/bkqj65bsjPm9ioiMZ2yytECIgBdDHwP6EH2U= X-Google-Smtp-Source: ADFU+vvji4fARHK9nDfdxs/FQYCRzfLzhXebo3fJ5pM2RqxUk8XqmFRERwkXmdEzjcSwIVnh/yyIGfUSe5I0Dhortsk= X-Received: by 2002:aca:ed86:: with SMTP id l128mr2018604oih.75.1583231673902; Tue, 03 Mar 2020 02:34:33 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Michael Stapelberg Date: Tue, 3 Mar 2020 11:34:22 +0100 Message-ID: Subject: Re: [fuse-devel] Writing to FUSE via mmap extremely slow (sometimes) on some machines? To: Miklos Szeredi Cc: Tejun Heo , Jack Smith , fuse-devel , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.193381, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Tejun, friendly ping? Any thoughts on this? Thanks! On Wed, Feb 26, 2020 at 9:00 PM Miklos Szeredi wrote: > > Adding more CC and re-attaching the reproducer and the 25s log. > > On Wed, Feb 26, 2020 at 11:03 AM Michael Stapelberg > wrote: > > > > Find attached two logs: > > > > fuse-1s.log shows the expected case > > > > fuse-25s.log shows the issue. Note that the log spans 2 days. I > > started the cp at 10:54:53.251395. Note how the first WRITE opcode is > > only received at 10:55:18.094578! > > Observations: > > - apparently memcpy is copying downwards (from largest address to > smallest address). Not sure why, when I run the reproducer, it copies > upwards. > - there's a slow batch of reads of the first ~4MB of data, then a > quick writeback > - there's a quick read of the rest (~95MB) of data, then a quick > writeback of the same > > Plots of the whole and closeups of slow and quick segments attached. > X axis is time, Y axis is offset. > > Tejun, could this behavior be attributed to dirty throttling? What > would be the best way to trace this? > > Thanks, > Miklos > > > > > > Is there some sort of readahead going on that=E2=80=99s then being thro= ttled somewhere? > > > > Thanks, > > > > On Mon, Feb 24, 2020 at 3:23 PM Miklos Szeredi wrot= e: > > > > > > On Mon, Feb 24, 2020 at 3:18 PM Michael Stapelberg > > > wrote: > > > > > > > > Sorry, to clarify: the hang is always in the memcpy call. I.e., the > > > > =E2=80=9CMapped=E2=80=9D message is always printed, and it takes a = long time until > > > > =E2=80=9Cmemcpy done=E2=80=9D is printed. > > > > > > Have you tried running the fuse daemon with debugging enabled? Is > > > there any observable difference between the fast and the slow runs? > > > > > > Thanks, > > > Miklos