Re: [PATCH RFC] ioctl based CAT interface

From: Marcelo Tosatti <mtosatti@redhat.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Luiz Capitulino <lcapitulino@redhat.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	Vikas Shivappa <vikas.shivappa@intel.com>,
	Tejun Heo <tj@kernel.org>, Yu Fenghua <fenghua.yu@intel.com>,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH RFC] ioctl based CAT interface
Date: Mon, 16 Nov 2015 12:37:08 -0200	[thread overview]
Message-ID: <20151116143708.GA26877@amt.cnet> (raw)
In-Reply-To: <20151116090756.GM17308@twins.programming.kicks-ass.net>

On Mon, Nov 16, 2015 at 10:07:56AM +0100, Peter Zijlstra wrote:
> On Fri, Nov 13, 2015 at 03:33:04PM -0200, Marcelo Tosatti wrote:
> > On Fri, Nov 13, 2015 at 05:51:00PM +0100, Peter Zijlstra wrote:
> > > On Fri, Nov 13, 2015 at 02:39:33PM -0200, Marcelo Tosatti wrote:
> > > > + * 	* one tcrid entry can be in different locations 
> > > > + * 	  in different sockets.
> > > 
> > > NAK on that without cpuset integration.
> > > 
> > > I do not want freely migratable tasks having radically different
> > > performance profiles depending on which CPU they land.
> > 
> > Ok, so, configuration:
> > 
> > 
> > Socket-1				Socket-2
> > 
> > pinned thread-A with			100% L3 free
> > 80% of L3 
> > reserved
> > 
> > 
> > So it is a problem if a thread running on socket-2 is scheduled to 
> > socket-1 because performance is radically different, fine.
> > 
> > Then one way to avoid that is to not allow freely migratable tasks
> > to move to Socket-1. Fine.
> > 
> > Then you want to use cpusets for that.
> > 
> > Can you fill in the blanks what is missing here?
> 
> I'm still not seeing what the problem with CAT-cgroup is.
> 
> /cgroups/cpuset/
>   socket-1/cpus = $socket-1
>            tasks = $thread-A
> 
>   socket-2/cpus = $socket-2
>            tasks = $thread-B
> 
> /cgroups/cat/
>   group-A/bitmap = 0x3F / 0xFF
>   group-A/tasks = $thread-A
> 
>   group-B/bitmap = 0xFF / 0xFF
>   group-B/tasks = $thread-B
> 
> 
> That gets you thread-A on socket-1 with 6/8 of the L3 and thread-B on
> socket-2 with 8/8 of the L3.

- need bitmasks per socket (optionally).
- format kept in kernel is not universal (have to convert every time
L3 cache size changes).
- need to specify type (i-cache or d-cache, differentiation supported on newer processors), 
ok can add more bitmaps.
- position in bitmask represents nothing other than identification of
  reservation and size, so:

		group-A = 0x3F, group-B = 0xFF
  is the same as
		group-A = 0xFC, group-B = 0xFF

- have to locate a free region every time in the bitmasks. 
So userspace has to do:

# lock write access to /cgroups/cat/
create group-C, taking into account bitmasks of 
group-A and group-B.
# unlock write access to /cgroups/cat.

But OK, it works, lets use that.