From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 933DFFC6196 for ; Fri, 8 Nov 2019 21:00:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4AF6B2178F for ; Fri, 8 Nov 2019 21:00:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="BDm7OoLZ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4AF6B2178F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C61586B0003; Fri, 8 Nov 2019 16:00:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C12CD6B0006; Fri, 8 Nov 2019 16:00:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B28B16B0007; Fri, 8 Nov 2019 16:00:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0149.hostedemail.com [216.40.44.149]) by kanga.kvack.org (Postfix) with ESMTP id 9CABB6B0003 for ; Fri, 8 Nov 2019 16:00:20 -0500 (EST) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with SMTP id 4D61E45CD for ; Fri, 8 Nov 2019 21:00:20 +0000 (UTC) X-FDA: 76134328200.09.bait24_774bf4fd4ba2b X-HE-Tag: bait24_774bf4fd4ba2b X-Filterd-Recvd-Size: 7121 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Fri, 8 Nov 2019 21:00:19 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0EECF214DB; Fri, 8 Nov 2019 21:00:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1573246818; bh=YdoZzimebFdQV6O7Wb5LCogyJVEvxDbTFEtZFm0YoJw=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=BDm7OoLZQoFXeZFGioW90fJ5lk6UvxKXvNJcW02TE6ixkVJGNNmaIKxvncUklmY0l dhKA9J8H01YNa8KBUc4U2zzFIdxe1TP2kSrVoywIVopxa72SUzWscau//ILU10IAF/ QtZXjgUYH7v9Mgh7/ZAQ0uZlbRAGjsCU+kUbaCCs= Date: Fri, 8 Nov 2019 13:00:17 -0800 From: Andrew Morton To: Hillf Danton Cc: linux-mm , fsdev , linux-kernel , Fengguang Wu , Tejun Heo , Jan Kara , Johannes Weiner , Shakeel Butt , Minchan Kim , Mel Gorman Subject: Re: [RFC v2] writeback: add elastic bdi in cgwb bdp Message-Id: <20191108130017.335cd353dd603dbba80b63dd@linux-foundation.org> In-Reply-To: <20191026104656.15176-1-hdanton@sina.com> References: <20191026104656.15176-1-hdanton@sina.com> X-Mailer: Sylpheed 3.5.1 (GTK+ 2.24.31; x86_64-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sat, 26 Oct 2019 18:46:56 +0800 Hillf Danton wrote: > > The elastic bdi is the mirror bdi of spinning disks, SSD, USB and > other storage devices/instruments on market. The performance of > ebdi goes up and down as the pattern of IO dispatched changes, as > approximately estimated as below. > > P = j(..., IO pattern); > > In ebdi's view, the bandwidth currently measured in balancing dirty > pages has close relation to its performance because the former is a > part of the latter. > > B = y(P); > > The functions above suggest there may be a layer violation if it > could be better measured somewhere below fs. > > It is measured however to the extent that makes every judge happy, > and is playing a role in dispatching IO with the IO pattern entirely > ignored that is volatile in nature. > > And it helps to throttle the dirty speed, with the figure ignored > that DRAM in general is x10 faster than ebdi. If B is half of P for > instance, then it is near 5% of dirty speed, just 2 points from the > figure in the snippet below. > > /* > * If ratelimit_pages is too high then we can get into dirty-data overload > * if a large number of processes all perform writes at the same time. > * If it is too low then SMP machines will call the (expensive) > * get_writeback_state too often. > * > * Here we set ratelimit_pages to a level which ensures that when all CPUs are > * dirtying in parallel, we cannot go more than 3% (1/32) over the dirty memory > * thresholds. > */ > > To prevent dirty speed from running away from laundry speed, ebdi > suggests the walk-dog method to put in bdp as a leash seems to > churn less in IO pattern. I'm finding both the changelog and the patch rather hard to understand. The absence of code comments doesn't help. But the test robot performance results look nice. Presumably you did your own performance testing. Please share the results of that in the changelog. > > --- a/include/linux/backing-dev-defs.h > +++ b/include/linux/backing-dev-defs.h > @@ -170,6 +170,8 @@ struct bdi_writeback { > > struct list_head bdi_node; /* anchored at bdi->wb_list */ > > + struct wait_queue_head bdp_waitq; Please add comments which help the reader find out what "bdp" stands for. > #ifdef CONFIG_CGROUP_WRITEBACK > struct percpu_ref refcnt; /* used only for !root wb's */ > struct fprop_local_percpu memcg_completions; > --- a/mm/backing-dev.c > +++ b/mm/backing-dev.c > @@ -324,6 +324,8 @@ static int wb_init(struct bdi_writeback > goto out_destroy_stat; > } > > + init_waitqueue_head(&wb->bdp_waitq); > + > return 0; > > out_destroy_stat: > --- a/mm/page-writeback.c > +++ b/mm/page-writeback.c > @@ -1551,6 +1551,39 @@ static inline void wb_dirty_limits(struc > } > } > > +static bool cgwb_bdp_should_throttle(struct bdi_writeback *wb) Please document this function. Describe the "why" not the "what". Comment should help readers understand what "cgwb" means. > +{ > + struct dirty_throttle_control gdtc = { GDTC_INIT_NO_WB }; > + > + if (fatal_signal_pending(current)) > + return false; > + > + gdtc.avail = global_dirtyable_memory(); > + > + domain_dirty_limits(&gdtc); > + > + gdtc.dirty = global_node_page_state(NR_FILE_DIRTY) + > + global_node_page_state(NR_UNSTABLE_NFS) + > + global_node_page_state(NR_WRITEBACK); > + > + if (gdtc.dirty < gdtc.bg_thresh) > + return false; > + > + if (!writeback_in_progress(wb)) > + wb_start_background_writeback(wb); Add a comment explaining what's going on here and why we're doing this. > + return gdtc.dirty > gdtc.thresh && > + wb_stat(wb, WB_DIRTIED) > > + wb_stat(wb, WB_WRITTEN) + > + wb_stat_error(); Why? > +} > + > +static inline void cgwb_bdp(struct bdi_writeback *wb) > +{ > + wait_event_interruptible_timeout(wb->bdp_waitq, > + !cgwb_bdp_should_throttle(wb), HZ); > +} > + > /* > * balance_dirty_pages() must be called by processes which are generating dirty > * data. It looks at the number of dirty pages in the machine and will force > @@ -1910,7 +1943,7 @@ void balance_dirty_pages_ratelimited(str > preempt_enable(); > > if (unlikely(current->nr_dirtied >= ratelimit)) > - balance_dirty_pages(wb, current->nr_dirtied); > + cgwb_bdp(wb); > > wb_put(wb); > } > --- a/fs/fs-writeback.c > +++ b/fs/fs-writeback.c > @@ -811,6 +811,8 @@ static long wb_split_bdi_pages(struct bd > if (nr_pages == LONG_MAX) > return LONG_MAX; > > + return nr_pages; > + > /* > * This may be called on clean wb's and proportional distribution > * may not make sense, just use the original @nr_pages in those > @@ -1604,6 +1606,7 @@ static long writeback_chunk_size(struct > pages = min(pages, work->nr_pages); > pages = round_down(pages + MIN_WRITEBACK_PAGES, > MIN_WRITEBACK_PAGES); > + pages = work->nr_pages; > } > > return pages; > @@ -2092,6 +2095,9 @@ void wb_workfn(struct work_struct *work) > wb_wakeup_delayed(wb); > > current->flags &= ~PF_SWAPWRITE; > + > + if (waitqueue_active(&wb->bdp_waitq)) > + wake_up_all(&wb->bdp_waitq); > } Does this patch affect both cgroup writeback and global writeback? Were both tested? Performance results of both?