archive mirror
 help / color / mirror / Atom feed
From: Valentin Schneider <>
To: Steven Sistare <>,,
Subject: Re: [PATCH v3 03/10] sched/topology: Provide cfs_overload_cpus bitmap
Date: Mon, 3 Dec 2018 16:56:31 +0000	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

Hi Steve,

On 26/11/2018 19:06, Steven Sistare wrote:
> [...]
>> Mmm I was thinking we could abuse the wrap() and start at
>> (fls(prev_span) + 1), but we're not guaranteed to have contiguous spans -
>> the Arm Juno for instance has [0, 3, 4], [1, 2] as MC-level domains, so
>> that goes down the drain.
>> Another thing that has been trotting in my head would be some helper to
>> create a cpumask from a sparsemask (some sort of sparsemask_span()),
>> which would let us use the standard mask operators:
>> ----->8-----
>> struct cpumask *overload_span = sparsemask_span(overload_cpus)
>> for_each_domain(this_cpu, sd)
>> 	for_each_cpu_and(src_cpu, overload_span, sched_domain_span(sd))
>> 		<steal_from here>
>> -----8>-----
>> The cpumask could be part of the sparsemask struct to save us the
>> allocation, and only updated when calling sparsemask_span().
> I thought of providing something like this along with other sparsemask
> utility functions, but I decided to be minimalist, and let others add
> more functions if/when they become needed.  this_cpu_cpumask_var_ptr(select_idle_mask) 
> is a temporary that could be used as the destination of the conversion.
> Also, conversion adds cost, particularly on larger systems.  When comparing a
> cpumask and a sparsemask, it is more efficient to iterate over the smaller
> set, and test for membership in the larger, such as in try_steal:
>     for_each_cpu(src_cpu, cpu_smt_mask(dst_cpu)) {
>             if (sparsemask_test_elem(src_cpu, overload_cpus)
>>> To extend stealing across LLC, I would like to keep the per-LLC sparsemask, 
>>> but add to each SD a list of sparsemask pointers.  The list nodes would be
>>> private, but the sparsemask structs would be shared.  Each list would include
>>> the masks that overlap the SD's members.  The list would be a singleton at the
>>> core and LLC levels (same as the socket level for most processors), and would 
>>> have multiple elements at the NUMA level.
>> I see. As for misfit, creating asym_cpucapacity siblings of the sd_llc_*()
>> functions seems a bit much - there'd be a lot of redundancy for basically
>> just a single shared sparsemask, which is why I was rambling about moving
>> things to root_domain.
>> Having different locations where sparsemasks are stored is a bit of a
>> pain which I'd like to avoid, but if it can't be unified I suppose we'll
>> have to live with it.
> I don't follow.  A per-LLC sparsemask representing misfits can be allocated with
> one more line in sd_llc_alloc, and you can steal across LLC using the list I
> briefly described above.

Ah yes, that would work. Thing is, I had excluded having the misfit masks
being in the sd_llc_shareds, since from a logical standpoint they don't
really belong there.

With asymmetric CPU capacities we kind of disregard the cache landscape
and focus on, well, CPU asymmetries. There's a few commits laying around
that forgo some cache usage optimisations for asymmetric systems -
this one comes to mind:

    9c63e84db29b ("sched/core: Disable SD_PREFER_SIBLING on asymmetric CPU capacity domains")

So in truth I was envisioning separate SD_ASYM_CPUCAPACITY-based 
sparsemasks, which is why I was rambling about SD_ASYM_CPUCAPACITY siblings
of sd_llc_*()... *But* after I had a go at it, it looked to me like that
was a lot of duplicated code.

My root_domain suggestion stems from the fact that we only really need one
single sparsemask for misfit stealing, and it provides a unique location
to store the sparsemasks (and you mask them however you want when it comes
to using them).

Sadly I think that doesn't work as well for cfs_overload_cpus since you
can't split a sparsemask's chunks over several NUMA nodes, so we'd be
stuck with an allocation on a single node (but we already do that in some
places, e.g. for nohz.idle_cpus_mask, so... Is it that bad?).

> - Steve

  reply	other threads:[~2018-12-03 16:56 UTC|newest]

Thread overview: 30+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-11-09 12:50 [PATCH v3 00/10] steal tasks to improve CPU utilization Steve Sistare
2018-11-09 12:50 ` [PATCH v3 01/10] sched: Provide sparsemask, a reduced contention bitmap Steve Sistare
2018-11-27 15:16   ` Steven Sistare
2018-11-28  1:19     ` Omar Sandoval
2018-12-06 16:07       ` Steven Sistare
2018-12-06 18:19         ` Omar Sandoval
2018-11-09 12:50 ` [PATCH v3 02/10] sched/topology: Provide hooks to allocate data shared per LLC Steve Sistare
2018-11-09 12:50 ` [PATCH v3 03/10] sched/topology: Provide cfs_overload_cpus bitmap Steve Sistare
2018-11-09 17:38   ` Valentin Schneider
2018-11-19 17:32     ` Steven Sistare
2018-11-20 12:52       ` Valentin Schneider
2018-11-12 16:42   ` Valentin Schneider
2018-11-19 17:33     ` Steven Sistare
2018-11-20 12:42       ` Valentin Schneider
2018-11-26 19:06         ` Steven Sistare
2018-12-03 16:56           ` Valentin Schneider [this message]
2018-12-06 16:40             ` Steven Sistare
2018-12-06 17:28               ` Valentin Schneider
2018-11-09 12:50 ` [PATCH v3 04/10] sched/fair: Dynamically update cfs_overload_cpus Steve Sistare
2018-11-09 12:50 ` [PATCH v3 05/10] sched/fair: Hoist idle_stamp up from idle_balance Steve Sistare
2018-11-09 19:07   ` Valentin Schneider
2018-11-19 17:31     ` Steven Sistare
2018-11-20 10:24       ` Valentin Schneider
2018-11-09 12:50 ` [PATCH v3 06/10] sched/fair: Generalize the detach_task interface Steve Sistare
2018-11-09 12:50 ` [PATCH v3 07/10] sched/fair: Provide can_migrate_task_llc Steve Sistare
2018-11-09 12:50 ` [PATCH v3 08/10] sched/fair: Steal work from an overloaded CPU when CPU goes idle Steve Sistare
2018-11-09 12:50 ` [PATCH v3 09/10] sched/fair: disable stealing if too many NUMA nodes Steve Sistare
2018-11-09 12:50 ` [PATCH v3 10/10] sched/fair: Provide idle search schedstats Steve Sistare
2018-11-10 17:08   ` kbuild test robot
2018-11-09 15:02 ` hackbench run scripts Steven Sistare

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).