From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66983C2B9F4 for ; Tue, 22 Jun 2021 12:12:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0DA6760FF1 for ; Tue, 22 Jun 2021 12:12:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0DA6760FF1 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A30A56B0075; Tue, 22 Jun 2021 08:12:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A071C6B0078; Tue, 22 Jun 2021 08:12:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8A9136B007B; Tue, 22 Jun 2021 08:12:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0123.hostedemail.com [216.40.44.123]) by kanga.kvack.org (Postfix) with ESMTP id 58B756B0075 for ; Tue, 22 Jun 2021 08:12:08 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id F2343180AD82F for ; Tue, 22 Jun 2021 12:12:07 +0000 (UTC) X-FDA: 78281246694.16.E7FC07C Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf01.hostedemail.com (Postfix) with ESMTP id 72F78500172C for ; Tue, 22 Jun 2021 12:12:07 +0000 (UTC) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out1.suse.de (Postfix) with ESMTP id 0774D218D6; Tue, 22 Jun 2021 12:12:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1624363926; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P8FU8UVi5nPdDY+zE8FGh4fOpPiQip+w88J/iISrayw=; b=ypU7A5mpbI/uy5MkStBMLsOR+WDc7p78YosCeNjDBnSRUVgWBfv0V2v/pK1iW++XVgJ1rS UkqJVXlbl+Ir0ONRFh8OPhT9hRTFZfZZwOsywOVS1fj61Dqv5GG/OQvVwgzrw9Xrn5wFrA h2+uDzPyJyi8Txsw7PQBPl693bbveZw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1624363926; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P8FU8UVi5nPdDY+zE8FGh4fOpPiQip+w88J/iISrayw=; b=7OBHe4E1/tMHiC0lrsPWECWykj4GeSeFoh3TBYu8HNoyZQAIlbRuQcXQmSsfREOknhnomI O/U28lFtOhj6+3AA== Received: from quack2.suse.cz (unknown [10.100.224.230]) by relay2.suse.de (Postfix) with ESMTP id B70A7A3B94; Tue, 22 Jun 2021 12:12:05 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id 8E4111E1515; Tue, 22 Jun 2021 14:12:05 +0200 (CEST) Date: Tue, 22 Jun 2021 14:12:05 +0200 From: Jan Kara To: Michael Stapelberg Cc: Miklos Szeredi , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm , linux-fsdevel@vger.kernel.org, Tejun Heo , Dennis Zhou , Jens Axboe , Roman Gushchin , Johannes Thumshirn , Jan Kara , Song Liu , David Sterba Subject: Re: [PATCH] backing_dev_info: introduce min_bw/max_bw limits Message-ID: <20210622121205.GG14261@quack2.suse.cz> References: <20210617095309.3542373-1-stapelberg+linux@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=ypU7A5mp; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=7OBHe4E1; spf=pass (imf01.hostedemail.com: domain of jack@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=jack@suse.cz; dmarc=none X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 72F78500172C X-Stat-Signature: 8nuehtfqb3ir5pncjj6mtukjg61k8cqj X-HE-Tag: 1624363927-846926 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon 21-06-21 11:20:10, Michael Stapelberg wrote: > Hey Miklos >=20 > On Fri, 18 Jun 2021 at 16:42, Miklos Szeredi wrote: > > > > On Fri, 18 Jun 2021 at 10:31, Michael Stapelberg > > wrote: > > > > > Maybe, but I don=E2=80=99t have the expertise, motivation or time t= o > > > investigate this any further, let alone commit to get it done. > > > During our previous discussion I got the impression that nobody els= e > > > had any cycles for this either: > > > https://lore.kernel.org/linux-fsdevel/CANnVG6n=3DySfe1gOr=3D0ituQid= p56idGARDKHzP0hv=3DERedeMrMA@mail.gmail.com/ > > > > > > Have you had a look at the China LSF report at > > > http://bardofschool.blogspot.com/2011/? > > > The author of the heuristic has spent significant effort and time > > > coming up with what we currently have in the kernel: > > > > > > """ > > > Fengguang said he draw more than 10K performance graphs and read ev= en > > > more in the past year. > > > """ > > > > > > This implies that making changes to the heuristic will not be a qui= ck fix. > > > > Having a piece of kernel code sitting there that nobody is willing to > > fix is certainly not a great situation to be in. >=20 > Agreed. >=20 > > > > And introducing band aids is not going improve the above situation, > > more likely it will prolong it even further. >=20 > Sounds like =E2=80=9CPerfect is the enemy of good=E2=80=9D to me: you=E2= =80=99re looking for a > perfect hypothetical solution, > whereas we have a known-working low risk fix for a real problem. >=20 > Could we find a solution where medium-/long-term, the code in question > is improved, > perhaps via a Summer Of Code project or similar community efforts, > but until then, we apply the patch at hand? >=20 > As I mentioned, I think adding min/max limits can be useful regardless > of how the heuristic itself changes. >=20 > If that turns out to be incorrect or undesired, we can still turn the > knobs into a no-op, if removal isn=E2=80=99t an option. Well, removal of added knobs is more or less out of question as it can break some userspace. Similarly making them no-op is problematic unless w= e are pretty certain it cannot break some existing setup. That's why we hav= e to think twice (or better three times ;) before adding any knobs. Also honestly the knobs you suggest will be pretty hard to tune when there are multiple cgroups with writeback control involved (which can be affected b= y the same problems you observe as well). So I agree with Miklos that this = is not the right way to go. Speaking of tunables, did you try tuning /sys/devices/virtual/bdi//min_ratio? I suspect that may workaround your problems... Looking into your original report and tracing you did (thanks for that, really useful), it seems that the problem is that writeback bandwidth is updated at most every 200ms (more frequent calls are just ignored) and ar= e triggered only from balance_dirty_pages() (happen when pages are dirtied)= and inode writeback code so if the workload tends to have short spikes of act= ivity and extended periods of quiet time, then writeback bandwidth may indeed b= e seriously miscomputed because we just won't update writeback throughput after most of writeback has happened as you observed. I think the fix for this can be relatively simple. We just need to make sure we update writeback bandwidth reasonably quickly after the IO finishes. I'll write a patch and see if it helps. Honza --=20 Jan Kara SUSE Labs, CR