From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751710AbbHBPsM (ORCPT ); Sun, 2 Aug 2015 11:48:12 -0400 Received: from mx1.redhat.com ([209.132.183.28]:34968 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751640AbbHBPsK (ORCPT ); Sun, 2 Aug 2015 11:48:10 -0400 Date: Sun, 2 Aug 2015 17:48:07 +0200 From: Martin Kletzander To: Marcelo Tosatti Cc: Vikas Shivappa , "Auld, Will" , Vikas Shivappa , "linux-kernel@vger.kernel.org" , "x86@kernel.org" , "hpa@zytor.com" , "tglx@linutronix.de" , "mingo@kernel.org" , "tj@kernel.org" , "peterz@infradead.org" , "Fleming, Matt" , "Williamson, Glenn P" , "Juvva, Kanaka D" Subject: Re: [PATCH 3/9] x86/intel_rdt: Cache Allocation documentation and cgroup usage guide Message-ID: <20150802154807.GA19188@wheatley> References: <1435789270-27010-1-git-send-email-vikas.shivappa@linux.intel.com> <1435789270-27010-4-git-send-email-vikas.shivappa@linux.intel.com> <20150728231516.GA16204@amt.cnet> <96EC5A4F3149B74492D2D9B9B1602C27461EB932@ORSMSX105.amr.corp.intel.com> <20150729193208.GC3201@amt.cnet> <20150730200812.GA10832@amt.cnet> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="vkogqOf2sHV7VnPd" Content-Disposition: inline In-Reply-To: <20150730200812.GA10832@amt.cnet> User-Agent: Mutt/1.5.23.1 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org --vkogqOf2sHV7VnPd Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline On Thu, Jul 30, 2015 at 05:08:13PM -0300, Marcelo Tosatti wrote: >On Thu, Jul 30, 2015 at 10:47:23AM -0700, Vikas Shivappa wrote: >> >> >> Marcello, >> >> >> On Wed, 29 Jul 2015, Marcelo Tosatti wrote: >> > >> >How about this: >> > >> >desiredclos (closid p1 p2 p3 p4) >> > 1 1 0 0 0 >> > 2 0 0 0 1 >> > 3 0 1 1 0 >> >> #1 Currently in the rdt cgroup , the root cgroup always has all the >> bits set and cant be changed (because the cgroup hierarchy would by >> default make this to have all bits as all the children need to have >> a subset of the root's bitmask). So if the user creates a cgroup and >> not put any task in it , the tasks in the root cgroup could be still >> using that part of the cache. Thats the reason i say we can have >> really 'exclusive' masks. >> >> Or in other words - there is always a desired clos (0) which has all >> parts set which acts like a default pool. >> >> Also the parts can overlap. Please apply this for all the below >> comments which will change the way they work. >> >> > >> >p means part. >> >> I am assuming p = (a contiguous cache capacity bit mask) > >Yes. > >> >closid 1 is a exclusive cgroup. >> >closid 2 is a "cache hog" class. >> >closid 3 is "default closid". >> > >> >Desiredclos is what user has specified. >> > >> >Transition 1: desiredclos --> effectiveclos >> >Clean all bits of unused closid's >> >(that must be updated whenever a >> >closid1 cgroup goes from empty->nonempty >> >and vice-versa). >> > >> >effectiveclos (closid p1 p2 p3 p4) >> > 1 0 0 0 0 >> > 2 0 0 0 1 >> > 3 0 1 1 0 >> >> > >> >Transition 2: effectiveclos --> expandedclos >> >expandedclos (closid p1 p2 p3 p4) >> > 1 0 0 0 0 >> > 2 0 0 0 1 >> > 3 1 1 1 0 >> >Then you have different inplacecos for each >> >CPU (see pseudo-code below): >> > >> >On the following events. >> > >> >- task migration to new pCPU: >> >- task creation: >> > >> > id = smp_processor_id(); >> > for (part = desiredclos.p1; ...; part++) >> > /* if my cosid is set and any other >> > cosid is clear, for the part, >> > synchronize desiredclos --> inplacecos */ >> > if (part[mycosid] == 1 && >> > part[any_othercosid] == 0) >> > wrmsr(part, desiredclos); >> > >> >> Currently the root cgroup would have all the bits set which will act >> like a default cgroup where all the otherwise unused parts (assuming >> they are a set of contiguous cache capacity bits) will be used. >> >> Otherwise the question is in the expandedclos - who decides to >> expand the closx parts to include some of the unused parts.. - that >> could just be a default root always ? > >Right, so the problem is for certain closid's you might never want >to expand (because doing so would cause data to be cached in a >cache way which might have high eviction rate in the future). >See the example from Will. > >But for the default cache (that is "unclassified applications" >i suppose it is beneficial to expand in most cases, that is, >use maximum amount of cache irrespective of eviction rate, which >is the behaviour that exists now without CAT). > >So perhaps a new flag "expand=y/n" can be added to the cgroup >directories... What do you say? > >Userspace representation of CAT >------------------------------- > >Usage model: >1) measure application performance without L3 cache reservation. >2) measure application perf with L3 cache reservation and >X number of cache ways until desired performance is attained. > >Requirements: >1) Persistency of CLOS configuration across hardware. On migration >of operating system or application between different hardware >systems we'd like the following to be maintained: > - exclusive number of bytes (*) reserved to a certain CLOSid. > - shared number of bytes (*) reserved between a certain group > of CLOSid's. > >For both code and data, rounded down or up in cache way size. > >2) Reasoning: >Different CBM masks in different hardware platforms might be necessary >to specify the same CLOS configuration, in terms of exclusive number of >bytes and shared number of bytes. (cache-way rounded number of bytes). >For example, due to L3 allocation by other hardware entities in certain parts >of the cache it might be necessary to relocate CBM mask to achieve >the same CLOS configuration. > >3) Proposed format: > Few questions from a random listener, I apologise if some of them are in a wrong place due to me missing some information from past threads. I'm not sure whether the following proposal to the format is the internal structure or what's going to be in cgroups. If this is user-visible interface, I think it could be a little less detailed. >sharedregionK.exclusive - Number of exclusive cache bytes reserved for > shared region. >sharedregionK.excl_data - Number of exclusive cache data bytes reserved for > shared region. >sharedregionK.excl_bytes - Number of exclusive cache code bytes reserved for > shared region. >sharedregionK.round_down - Round down to cache way bytes from respective number > specification (default is round up). >sharedregionK.expand - y/n - Expand shared region to more cache ways > when available (default N). > >cgroupN.exclusive - Number of exclusive L3 cache bytes reserved > for cgroup. >cgroupN.excl_data - Number of exclusive L3 data cache bytes reserved > for cgroup. >cgroupN.excl_code - Number of exclusive L3 code cache bytes reserved > for cgroup. By exclusive, you mean that it's exclusive to the tasks in this cgroup? The thing is that we must differentiate between limiting some process's from hogging the memory (like example 2 below) and making some part of the cache exclusive for particular application (example 1 below). I just hope we won't need to add something similar to 'isolcpus=' just so we can make sure none of the tasks in the root cgroup can spoil the part of the cache we need to have exclusive. I'm not sure creating a new subgroup and moving all the tasks there would work, It certainly is not possible with other cgroups, like the cpuset cgroup mentioned beforehand. I also don't quite fully understand how the co-mounting with the cpuset cgroup should work, but that's not design-related. One more question, how does this work on systems with multiple L3 caches (e.g. large NUMA node systems)? I'm guessing if the process is running only on some CPUs, the wrmsr() will be called on that particular CPU(s), right? >cgroupN.round_down - Round down to cache way bytes from respective number > specification (default is round up). >cgroupN.expand - y/n - Expand shared region to more cache ways when > available (default N). >cgroupN.shared = { sharedregion1, sharedregion2, ... } (list of shared >regions) > >Example 1: >One application with 2M exclusive cache, two applications >with 1M exclusive each, sharing an expansive shared region of 1M. > >cgroup1.exclusive = 2M > >sharedregion1.exclusive = 1M >sharedregion1.expand = Y > >cgroup2.exclusive = 1M >cgroup2.shared = sharedregion1 > >cgroup3.exclusive = 1M >cgroup3.shared = sharedregion1 > >Example 2: >3 high performance applications running, one of which is a cache hog >with no cache locality. > >cgroup1.exclusive = 8M >cgroup2.exclusive = 8M > >cgroup3.exclusive = 512K >cgroup3.round_down = Y > >In all cases the default cgroup (which requires no explicit >specification) is expansive and uses the remaining cache >ways, including the ways shared by other hardware entities. > >-- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ --vkogqOf2sHV7VnPd Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJVvju3AAoJEAgfwp8kF4bdTIoP/RKQOcMxOT1Yjt05Juea/tCH 5gdc9tEqlTl6sRsE70HYurD59xPoZl1YAHHLuKsf2ylrS3ym2AexdPrCxblEPmDv qnryiSC62KQP85Dwo81Zqy7I1Tzdpb0MvDrs/PRLPPL6wJ+5Ba9Nb48AmcRptR5A SGXGeCq/vmV6vXYaZQkwWJDzlxMPiTgJS+QFNooSVdxMKFU5zNdJxf9mSJ/uDMvG 82Hj76Cvt2IaVz4AI5ZRwCwofFeJb6UQ9NMN9qPvMO9t1yit/nQTcDiRZ/h4LqMz uoMHHcCktIHT9AMklWw5xiXpsfq7O7fWzk+NNaXCTkqbU4iySj+GQ4tVUKaGkfNW kzid0ngv5Cic4SRdLK3URKg/PHo1s5xJEcAwx5fZdvUSn7X9zXggisjifUoPFWLN XQQmhvj5io7hV+TDViKWpQSMi4/4ZJhsHcINSObedgtWvSV+Kvn+tbaIY443Vtdc LBL09ltnAsyuCb3MnQWh9YNjuyJ31uJVti7iyynzj5lbna/b0UZAdaSIKL0CT5LX qemvixkhv5I8ooDQUzUTQvskU3unImJ8on5Vh3nNNPfjF25GCz8TzU0N7A45tCA5 tFEisVEytrQ44izDBPmHpgLEsD+J1R7AUtVMRLxYMrG8qe6z9+EmXeJY7XURL1pT GR2on9zcqw0al/75Ex55 =3igN -----END PGP SIGNATURE----- --vkogqOf2sHV7VnPd--