From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753226AbbJWMQw (ORCPT ); Fri, 23 Oct 2015 08:16:52 -0400 Received: from mx1.redhat.com ([209.132.183.28]:43289 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750938AbbJWMQt (ORCPT ); Fri, 23 Oct 2015 08:16:49 -0400 Date: Fri, 23 Oct 2015 08:16:47 -0400 From: Vivek Goyal To: Tejun Heo Cc: Johannes Weiner , Li Zefan , cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Jens Axboe , Michal Hocko , Peter Zijlstra , Ingo Molnar , Paul Turner , kernel-team@fb.com Subject: Re: [PATCH cgroup/for-4.4 3/3] cgroup: replace unified-hierarchy.txt with a proper cgroup v2 documentation Message-ID: <20151023121647.GA22547@redhat.com> References: <20151022012212.GA20931@mtj.duckdns.org> <20151022012302.GB20931@mtj.duckdns.org> <20151022012354.GC20931@mtj.duckdns.org> <20151022143427.GC20847@redhat.com> <20151022143605.GE30579@mtj.duckdns.org> <20151023011807.GB5442@mtj.duckdns.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151023011807.GB5442@mtj.duckdns.org> User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Oct 23, 2015 at 10:18:07AM +0900, Tejun Heo wrote: > Hello, > > On Thu, Oct 22, 2015 at 11:36:05PM +0900, Tejun Heo wrote: > > It works with ext2 and 4 and btrfs. Will document it. Thanks. > > Updated to include all writeback information from > blkio-controller.txt. > Thanks Tejun. Looks good. Vivek > 5-3-2. Writeback > > Page cache is dirtied through buffered writes and shared mmaps and > written asynchronously to the backing filesystem by the writeback > mechanism. Writeback sits between the memory and IO domains and > regulates the proportion of dirty memory by balancing dirtying and > write IOs. > > The io controller, in conjunction with the memory controller, > implements control of page cache writeback IOs. The memory controller > defines the memory domain that dirty memory ratio is calculated and > maintained for and the io controller defines the io domain which > writes out dirty pages for the memory domain. Both system-wide and > per-cgroup dirty memory states are examined and the more restrictive > of the two is enforced. > > cgroup writeback requires explicit support from the underlying > filesystem. Currently, cgroup writeback is implemented on ext2, ext4 > and btrfs. On other filesystems, all writeback IOs are attributed to > the root cgroup. > > There are inherent differences in memory and writeback management > which affects how cgroup ownership is tracked. Memory is tracked per > page while writeback per inode. For the purpose of writeback, an > inode is assigned to a cgroup and all IO requests to write dirty pages > from the inode are attributed to that cgroup. > > As cgroup ownership for memory is tracked per page, there can be pages > which are associated with different cgroups than the one the inode is > associated with. These are called foreign pages. The writeback > constantly keeps track of foreign pages and, if a particular foreign > cgroup becomes the majority over a certain period of time, switches > the ownership of the inode to that cgroup. > > While this model is enough for most use cases where a given inode is > mostly dirtied by a single cgroup even when the main writing cgroup > changes over time, use cases where multiple cgroups write to a single > inode simultaneously are not supported well. In such circumstances, a > significant portion of IOs are likely to be attributed incorrectly. > As memory controller assigns page ownership on the first use and > doesn't update it until the page is released, even if writeback > strictly follows page ownership, multiple cgroups dirtying overlapping > areas wouldn't work as expected. It's recommended to avoid such usage > patterns. > > The sysctl knobs which affect writeback behavior are applied to cgroup > writeback as follows. > > vm.dirty_background_ratio > vm.dirty_ratio > > These ratios apply the same to cgroup writeback with the > amount of available memory capped by limits imposed by the > memory controller and system-wide clean memory. > > vm.dirty_background_bytes > vm.dirty_bytes > > For cgroup writeback, this is calculated into ratio against > total available memory and applied the same way as > vm.dirty[_background]_ratio. > > > P. Information on Kernel Programming > > This section contains kernel programming information in the areas > where interacting with cgroup is necessary. cgroup core and > controllers are not covered. > > > P-1. Filesystem Support for Writeback > > A filesystem can support cgroup writeback by updating > address_space_operations->writepage[s]() to annotate bio's using the > following two functions. > > wbc_init_bio(@wbc, @bio) > > Should be called for each bio carrying writeback data and > associates the bio with the inode's owner cgroup. Can be > called anytime between bio allocation and submission. > > wbc_account_io(@wbc, @page, @bytes) > > Should be called for each data segment being written out. > While this function doesn't care exactly when it's called > during the writeback session, it's the easiest and most > natural to call it as data segments are added to a bio. > > With writeback bio's annotated, cgroup support can be enabled per > super_block by setting SB_I_CGROUPWB in ->s_iflags. This allows for > selective disabling of cgroup writeback support which is helpful when > certain filesystem features, e.g. journaled data mode, are > incompatible. > > wbc_init_bio() binds the specified bio to its cgroup. Depending on > the configuration, the bio may be executed at a lower priority and if > the writeback session is holding shared resources, e.g. a journal > entry, may lead to priority inversion. There is no one easy solution > for the problem. Filesystems can try to work around specific problem > cases by skipping wbc_init_bio() or using bio_associate_blkcg() > directly. > > -- > tejun From mboxrd@z Thu Jan 1 00:00:00 1970 From: Vivek Goyal Subject: Re: [PATCH cgroup/for-4.4 3/3] cgroup: replace unified-hierarchy.txt with a proper cgroup v2 documentation Date: Fri, 23 Oct 2015 08:16:47 -0400 Message-ID: <20151023121647.GA22547@redhat.com> References: <20151022012212.GA20931@mtj.duckdns.org> <20151022012302.GB20931@mtj.duckdns.org> <20151022012354.GC20931@mtj.duckdns.org> <20151022143427.GC20847@redhat.com> <20151022143605.GE30579@mtj.duckdns.org> <20151023011807.GB5442@mtj.duckdns.org> Mime-Version: 1.0 Return-path: Content-Disposition: inline In-Reply-To: <20151023011807.GB5442-qYNAdHglDFBN0TnZuCh8vA@public.gmane.org> Sender: cgroups-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org List-ID: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Tejun Heo Cc: Johannes Weiner , Li Zefan , cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Jens Axboe , Michal Hocko , Peter Zijlstra , Ingo Molnar , Paul Turner , kernel-team-b10kYP2dOMg@public.gmane.org On Fri, Oct 23, 2015 at 10:18:07AM +0900, Tejun Heo wrote: > Hello, > > On Thu, Oct 22, 2015 at 11:36:05PM +0900, Tejun Heo wrote: > > It works with ext2 and 4 and btrfs. Will document it. Thanks. > > Updated to include all writeback information from > blkio-controller.txt. > Thanks Tejun. Looks good. Vivek > 5-3-2. Writeback > > Page cache is dirtied through buffered writes and shared mmaps and > written asynchronously to the backing filesystem by the writeback > mechanism. Writeback sits between the memory and IO domains and > regulates the proportion of dirty memory by balancing dirtying and > write IOs. > > The io controller, in conjunction with the memory controller, > implements control of page cache writeback IOs. The memory controller > defines the memory domain that dirty memory ratio is calculated and > maintained for and the io controller defines the io domain which > writes out dirty pages for the memory domain. Both system-wide and > per-cgroup dirty memory states are examined and the more restrictive > of the two is enforced. > > cgroup writeback requires explicit support from the underlying > filesystem. Currently, cgroup writeback is implemented on ext2, ext4 > and btrfs. On other filesystems, all writeback IOs are attributed to > the root cgroup. > > There are inherent differences in memory and writeback management > which affects how cgroup ownership is tracked. Memory is tracked per > page while writeback per inode. For the purpose of writeback, an > inode is assigned to a cgroup and all IO requests to write dirty pages > from the inode are attributed to that cgroup. > > As cgroup ownership for memory is tracked per page, there can be pages > which are associated with different cgroups than the one the inode is > associated with. These are called foreign pages. The writeback > constantly keeps track of foreign pages and, if a particular foreign > cgroup becomes the majority over a certain period of time, switches > the ownership of the inode to that cgroup. > > While this model is enough for most use cases where a given inode is > mostly dirtied by a single cgroup even when the main writing cgroup > changes over time, use cases where multiple cgroups write to a single > inode simultaneously are not supported well. In such circumstances, a > significant portion of IOs are likely to be attributed incorrectly. > As memory controller assigns page ownership on the first use and > doesn't update it until the page is released, even if writeback > strictly follows page ownership, multiple cgroups dirtying overlapping > areas wouldn't work as expected. It's recommended to avoid such usage > patterns. > > The sysctl knobs which affect writeback behavior are applied to cgroup > writeback as follows. > > vm.dirty_background_ratio > vm.dirty_ratio > > These ratios apply the same to cgroup writeback with the > amount of available memory capped by limits imposed by the > memory controller and system-wide clean memory. > > vm.dirty_background_bytes > vm.dirty_bytes > > For cgroup writeback, this is calculated into ratio against > total available memory and applied the same way as > vm.dirty[_background]_ratio. > > > P. Information on Kernel Programming > > This section contains kernel programming information in the areas > where interacting with cgroup is necessary. cgroup core and > controllers are not covered. > > > P-1. Filesystem Support for Writeback > > A filesystem can support cgroup writeback by updating > address_space_operations->writepage[s]() to annotate bio's using the > following two functions. > > wbc_init_bio(@wbc, @bio) > > Should be called for each bio carrying writeback data and > associates the bio with the inode's owner cgroup. Can be > called anytime between bio allocation and submission. > > wbc_account_io(@wbc, @page, @bytes) > > Should be called for each data segment being written out. > While this function doesn't care exactly when it's called > during the writeback session, it's the easiest and most > natural to call it as data segments are added to a bio. > > With writeback bio's annotated, cgroup support can be enabled per > super_block by setting SB_I_CGROUPWB in ->s_iflags. This allows for > selective disabling of cgroup writeback support which is helpful when > certain filesystem features, e.g. journaled data mode, are > incompatible. > > wbc_init_bio() binds the specified bio to its cgroup. Depending on > the configuration, the bio may be executed at a lower priority and if > the writeback session is holding shared resources, e.g. a journal > entry, may lead to priority inversion. There is no one easy solution > for the problem. Filesystems can try to work around specific problem > cases by skipping wbc_init_bio() or using bio_associate_blkcg() > directly. > > -- > tejun