Re: [PATCH] KVM: x86: add a new field 'is_idle' to kvm_steal_time

[Vitaly Kuznetsov <vkuznets@redhat.com>]

Tianqiang Xu <skyele@sjtu.edu.cn> writes:

> This patch aims to fix performance issue caused by current
> para-virtualized scheduling design.
>
> The current para-virtualized scheduling design uses 'preempted' field of
> kvm_steal_time to avoid scheduling task on the preempted vCPU.
> However, when the pCPU where the preempted vCPU most recently run is idle,
> it will result in low cpu utilization, and consequently poor performance.
>
> The new field: 'is_idle' of kvm_steal_time can precisely reveal the status of
> pCPU where preempted vCPU most recently run, and then improve cpu utilization.
>
> Host OS sets this field to 1 if cpu_rq(this_cpu)->nr_running is 1 before
> a vCPU being scheduled out. On this condition, there is no other task on
> this pCPU to run. Thus, is_idle == 1 means the pCPU where the preempted
> vCPU most recently run is idle.
>
> Guest OS uses this field to know if a pCPU is idle and decides whether
> to schedule a task to a preempted vCPU or not. If the pCPU is idle,
> scheduling a task to this pCPU will improve cpu utilization. If not,
> avoiding scheduling a task to this preempted vCPU can avoid host/guest
> switch, hence improving performance.
>
> Experiments on a VM with 16 vCPUs show that the patch can reduce around
> 50% to 80% execution time for most PARSEC benchmarks. 
> This also holds true for a VM with 112 vCPUs.
>
> Experiments on 2 VMs with 112 vCPUs show that the patch can reduce around
> 20% to 80% execution time for most PARSEC benchmarks. 
>
> Test environment:
> -- PowerEdge R740
> -- 56C-112T CPU Intel(R) Xeon(R) Gold 6238R CPU
> -- Host 190G DRAM
> -- QEMU 5.0.0
> -- PARSEC 3.0 Native Inputs
> -- Host is idle during the test
> -- Host and Guest kernel are both kernel-5.14.0
>
> Results:
> 1. 1 VM, 16 VCPU, 16 THREAD.
>    Host Topology: sockets=2 cores=28 threads=2
>    VM Topology:   sockets=1 cores=16 threads=1
>    Command: <path to parsec>/bin/parsecmgmt -a run -p <benchmark> -i native -n 16
>    Statistics below are the real time of running each benchmark.(lower is better)
>
>                     before patch    after patch     improvements
> bodytrack           52.866s         22.619s         57.21%
> fluidanimate        84.009s         38.148s         54.59%
> streamcluster       270.17s         42.726s         84.19%
> splash2x.ocean_cp   31.932s         9.539s          70.13%
> splash2x.ocean_ncp  36.063s         14.189s         60.65%
> splash2x.volrend    134.587s        21.79s          83.81%
>
> 2. 1VM, 112 VCPU. Some benchmarks require the number of threads to be the power of 2,
> so we run them with 64 threads and 128 threads.
>    Host Topology: sockets=2 cores=28 threads=2
>    VM Topology:   sockets=1 cores=112 threads=1
>    Command: <path to parsec>/bin/parsecmgmt -a run -p <benchmark> -i native -n <64,112,128>
>    Statistics below are the real time of running each benchmark.(lower is better)
>
>    
>
>                                 before patch    after patch     improvements
> fluidanimate(64 thread)         124.235s        27.924s         77.52%
> fluidanimate(128 thread)        169.127s        64.541s         61.84%
> streamcluster(112 thread)       861.879s        496.66s         42.37%
> splash2x.ocean_cp(64 thread)    46.415s         18.527s         60.08%
> splash2x.ocean_cp(128 thread)   53.647s         28.929s         46.08%
> splash2x.ocean_ncp(64 thread)   47.613s         19.576s         58.89%
> splash2x.ocean_ncp(128 thread)  54.94s          29.199s         46.85%
> splash2x.volrend(112 thread)    801.384s        144.824s        81.93%
>
> 3. 2VM, each VM: 112 VCPU. Some benchmarks require the number of threads to
> be the power of 2, so we run them with 64 threads and 128 threads.
>    Host Topology: sockets=2 cores=28 threads=2
>    VM Topology:   sockets=1 cores=112 threads=1
>    Command: <path to parsec>/bin/parsecmgmt -a run -p <benchmark> -i native -n <64,112,128>
>    Statistics below are the average real time of running each benchmark in 2 VMs.(lower is better)
>                                 before patch    after patch     improvements
> fluidanimate(64 thread)         135.2125s       49.827s         63.15%
> fluidanimate(128 thread)        178.309s        86.964s         51.23%
> splash2x.ocean_cp(64 thread)    47.4505s        20.314s         57.19%
> splash2x.ocean_cp(128 thread)   55.5645s        30.6515s        44.84%
> splash2x.ocean_ncp(64 thread)   49.9775s        23.489s         53.00%
> splash2x.ocean_ncp(128 thread)  56.847s         28.545s         49.79%
> splash2x.volrend(112 thread)    838.939s        239.632s        71.44%
>
> For space limit, we list representative statistics here.
>
> --
> Authors: Tianqiang Xu, Dingji Li, Zeyu Mi
>          Shanghai Jiao Tong University
>
> Signed-off-by: Tianqiang Xu <skyele@sjtu.edu.cn>
> ---
>  arch/x86/hyperv/hv_spinlock.c         |  7 +++
>  arch/x86/include/asm/cpufeatures.h    |  1 +
>  arch/x86/include/asm/kvm_host.h       |  1 +
>  arch/x86/include/asm/paravirt.h       |  8 +++
>  arch/x86/include/asm/paravirt_types.h |  1 +
>  arch/x86/include/asm/qspinlock.h      |  6 ++
>  arch/x86/include/uapi/asm/kvm_para.h  |  4 +-
>  arch/x86/kernel/asm-offsets_64.c      |  1 +
>  arch/x86/kernel/kvm.c                 | 23 +++++++
>  arch/x86/kernel/paravirt-spinlocks.c  | 15 +++++
>  arch/x86/kernel/paravirt.c            |  2 +
>  arch/x86/kvm/x86.c                    | 90 ++++++++++++++++++++++++++-
>  include/linux/sched.h                 |  1 +
>  kernel/sched/core.c                   | 17 +++++
>  kernel/sched/sched.h                  |  1 +
>  15 files changed, 176 insertions(+), 2 deletions(-)

Thank you for the patch! Please split this patch into a series, e.g.:

- Introduce .pcpu_is_idle() stub infrastructure
- Scheduler changes
- Preparatory patch[es] for KVM creating 'is_idle'
- KVM host implementation
- KVM guest implementation
- ...

so it can be reviewed and ACKed. Just a couple of nitpicks below

>
> diff --git a/arch/x86/hyperv/hv_spinlock.c b/arch/x86/hyperv/hv_spinlock.c
> index 91cfe698bde0..b8c32b719cab 100644
> --- a/arch/x86/hyperv/hv_spinlock.c
> +++ b/arch/x86/hyperv/hv_spinlock.c
> @@ -66,6 +66,12 @@ __visible bool hv_vcpu_is_preempted(int vcpu)
>  }
>  PV_CALLEE_SAVE_REGS_THUNK(hv_vcpu_is_preempted);
>  
> +__visible bool hv_pcpu_is_idle(int vcpu)
> +{
> +	return false;
> +}
> +PV_CALLEE_SAVE_REGS_THUNK(hv_pcpu_is_idle);
> +
>  void __init hv_init_spinlocks(void)
>  {
>  	if (!hv_pvspin || !apic ||
> @@ -82,6 +88,7 @@ void __init hv_init_spinlocks(void)
>  	pv_ops.lock.wait = hv_qlock_wait;
>  	pv_ops.lock.kick = hv_qlock_kick;
>  	pv_ops.lock.vcpu_is_preempted = PV_CALLEE_SAVE(hv_vcpu_is_preempted);
> +	pv_ops.lock.pcpu_is_idle = PV_CALLEE_SAVE(hv_pcpu_is_idle);
>  }
>  
>  static __init int hv_parse_nopvspin(char *arg)
> diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
> index d0ce5cfd3ac1..8a078619c9de 100644
> --- a/arch/x86/include/asm/cpufeatures.h
> +++ b/arch/x86/include/asm/cpufeatures.h
> @@ -238,6 +238,7 @@
>  #define X86_FEATURE_VMW_VMMCALL		( 8*32+19) /* "" VMware prefers VMMCALL hypercall instruction */
>  #define X86_FEATURE_PVUNLOCK		( 8*32+20) /* "" PV unlock function */
>  #define X86_FEATURE_VCPUPREEMPT		( 8*32+21) /* "" PV vcpu_is_preempted function */
> +#define X86_FEATURE_PCPUISIDLE		( 8*32+22) /* "" PV pcpu_is_idle function */
>  
>  /* Intel-defined CPU features, CPUID level 0x00000007:0 (EBX), word 9 */
>  #define X86_FEATURE_FSGSBASE		( 9*32+ 0) /* RDFSBASE, WRFSBASE, RDGSBASE, WRGSBASE instructions*/
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index 974cbfb1eefe..bed0ab7233be 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -742,6 +742,7 @@ struct kvm_vcpu_arch {
>  
>  	struct {
>  		u8 preempted;
> +		u8 is_idle;
>  		u64 msr_val;
>  		u64 last_steal;
>  		struct gfn_to_pfn_cache cache;
> diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h
> index da3a1ac82be5..f34dec6eb515 100644
> --- a/arch/x86/include/asm/paravirt.h
> +++ b/arch/x86/include/asm/paravirt.h
> @@ -609,8 +609,16 @@ static __always_inline bool pv_vcpu_is_preempted(long cpu)
>  				ALT_NOT(X86_FEATURE_VCPUPREEMPT));
>  }
>  
> +static __always_inline bool pv_pcpu_is_idle(long cpu)
> +{
> +	return PVOP_ALT_CALLEE1(bool, lock.pcpu_is_idle, cpu,
> +				"xor %%" _ASM_AX ", %%" _ASM_AX ";",
> +				ALT_NOT(X86_FEATURE_PCPUISIDLE));
> +}
> +
>  void __raw_callee_save___native_queued_spin_unlock(struct qspinlock *lock);
>  bool __raw_callee_save___native_vcpu_is_preempted(long cpu);
> +bool __raw_callee_save___native_pcpu_is_idle(long cpu);
>  
>  #endif /* SMP && PARAVIRT_SPINLOCKS */
>  
> diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/paravirt_types.h
> index d9d6b0203ec4..7d9b5906580c 100644
> --- a/arch/x86/include/asm/paravirt_types.h
> +++ b/arch/x86/include/asm/paravirt_types.h
> @@ -257,6 +257,7 @@ struct pv_lock_ops {
>  	void (*kick)(int cpu);
>  
>  	struct paravirt_callee_save vcpu_is_preempted;
> +	struct paravirt_callee_save pcpu_is_idle;
>  } __no_randomize_layout;
>  
>  /* This contains all the paravirt structures: we get a convenient
> diff --git a/arch/x86/include/asm/qspinlock.h b/arch/x86/include/asm/qspinlock.h
> index d86ab942219c..1832dd8308ca 100644
> --- a/arch/x86/include/asm/qspinlock.h
> +++ b/arch/x86/include/asm/qspinlock.h
> @@ -61,6 +61,12 @@ static inline bool vcpu_is_preempted(long cpu)
>  {
>  	return pv_vcpu_is_preempted(cpu);
>  }
> +
> +#define pcpu_is_idle pcpu_is_idle
> +static inline bool pcpu_is_idle(long cpu)
> +{
> +	return pv_pcpu_is_idle(cpu);
> +}
>  #endif
>  
>  #ifdef CONFIG_PARAVIRT
> diff --git a/arch/x86/include/uapi/asm/kvm_para.h b/arch/x86/include/uapi/asm/kvm_para.h
> index 5146bbab84d4..2af305ba030a 100644
> --- a/arch/x86/include/uapi/asm/kvm_para.h
> +++ b/arch/x86/include/uapi/asm/kvm_para.h
> @@ -63,12 +63,14 @@ struct kvm_steal_time {
>  	__u32 version;
>  	__u32 flags;
>  	__u8  preempted;
> -	__u8  u8_pad[3];
> +	__u8  is_idle;
> +	__u8  u8_pad[2];
>  	__u32 pad[11];
>  };
>  
>  #define KVM_VCPU_PREEMPTED          (1 << 0)
>  #define KVM_VCPU_FLUSH_TLB          (1 << 1)
> +#define KVM_PCPU_IS_IDLE          (1 << 0)
>  
>  #define KVM_CLOCK_PAIRING_WALLCLOCK 0
>  struct kvm_clock_pairing {
> diff --git a/arch/x86/kernel/asm-offsets_64.c b/arch/x86/kernel/asm-offsets_64.c
> index b14533af7676..b587bbe44470 100644
> --- a/arch/x86/kernel/asm-offsets_64.c
> +++ b/arch/x86/kernel/asm-offsets_64.c
> @@ -22,6 +22,7 @@ int main(void)
>  
>  #if defined(CONFIG_KVM_GUEST) && defined(CONFIG_PARAVIRT_SPINLOCKS)
>  	OFFSET(KVM_STEAL_TIME_preempted, kvm_steal_time, preempted);
> +	OFFSET(KVM_STEAL_TIME_is_idle, kvm_steal_time, is_idle);
>  	BLANK();
>  #endif
>  
> diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
> index a26643dc6bd6..274d205b744c 100644
> --- a/arch/x86/kernel/kvm.c
> +++ b/arch/x86/kernel/kvm.c
> @@ -900,11 +900,20 @@ __visible bool __kvm_vcpu_is_preempted(long cpu)
>  }
>  PV_CALLEE_SAVE_REGS_THUNK(__kvm_vcpu_is_preempted);
>  
> +__visible bool __kvm_pcpu_is_idle(long cpu)
> +{
> +	struct kvm_steal_time *src = &per_cpu(steal_time, cpu);
> +
> +	return !!(src->is_idle & KVM_PCPU_IS_IDLE);
> +}
> +PV_CALLEE_SAVE_REGS_THUNK(__kvm_pcpu_is_idle);
> +
>  #else
>  
>  #include <asm/asm-offsets.h>
>  
>  extern bool __raw_callee_save___kvm_vcpu_is_preempted(long);
> +extern bool __raw_callee_save___kvm_pcpu_is_idle(long);
>  
>  /*
>   * Hand-optimize version for x86-64 to avoid 8 64-bit register saving and
> @@ -922,6 +931,18 @@ asm(
>  ".size __raw_callee_save___kvm_vcpu_is_preempted, .-__raw_callee_save___kvm_vcpu_is_preempted;"
>  ".popsection");
>  
> +asm(
> +".pushsection .text;"
> +".global __raw_callee_save___kvm_pcpu_is_idle;"
> +".type __raw_callee_save___kvm_pcpu_is_idle, @function;"
> +"__raw_callee_save___kvm_pcpu_is_idle:"
> +"movq	__per_cpu_offset(,%rdi,8), %rax;"
> +"cmpb	$0, " __stringify(KVM_STEAL_TIME_is_idle) "+steal_time(%rax);"
> +"setne	%al;"
> +"ret;"
> +".size __raw_callee_save___kvm_pcpu_is_idle, .-__raw_callee_save___kvm_pcpu_is_idle;"
> +".popsection");
> +
>  #endif
>  
>  /*
> @@ -970,6 +991,8 @@ void __init kvm_spinlock_init(void)
>  	if (kvm_para_has_feature(KVM_FEATURE_STEAL_TIME)) {
>  		pv_ops.lock.vcpu_is_preempted =
>  			PV_CALLEE_SAVE(__kvm_vcpu_is_preempted);
> +		pv_ops.lock.pcpu_is_idle =
> +			PV_CALLEE_SAVE(__kvm_pcpu_is_idle);
>  	}
>  	/*
>  	 * When PV spinlock is enabled which is preferred over
> diff --git a/arch/x86/kernel/paravirt-spinlocks.c b/arch/x86/kernel/paravirt-spinlocks.c
> index 9e1ea99ad9df..d7f6a461d0a5 100644
> --- a/arch/x86/kernel/paravirt-spinlocks.c
> +++ b/arch/x86/kernel/paravirt-spinlocks.c
> @@ -27,12 +27,24 @@ __visible bool __native_vcpu_is_preempted(long cpu)
>  }
>  PV_CALLEE_SAVE_REGS_THUNK(__native_vcpu_is_preempted);
>  
> +__visible bool __native_pcpu_is_idle(long cpu)
> +{
> +	return false;
> +}
> +PV_CALLEE_SAVE_REGS_THUNK(__native_pcpu_is_idle);
> +
>  bool pv_is_native_vcpu_is_preempted(void)
>  {
>  	return pv_ops.lock.vcpu_is_preempted.func ==
>  		__raw_callee_save___native_vcpu_is_preempted;
>  }
>  
> +bool pv_is_native_pcpu_is_idle(void)

Just 'pv_native_pcpu_is_idle' or 'pv_is_native_pcpu_idle' maybe?

> +{
> +	return pv_ops.lock.pcpu_is_idle.func ==
> +		__raw_callee_save___native_pcpu_is_idle;
> +}
> +
>  void __init paravirt_set_cap(void)
>  {
>  	if (!pv_is_native_spin_unlock())
> @@ -40,4 +52,7 @@ void __init paravirt_set_cap(void)
>  
>  	if (!pv_is_native_vcpu_is_preempted())
>  		setup_force_cpu_cap(X86_FEATURE_VCPUPREEMPT);
> +
> +	if (!pv_is_native_pcpu_is_idle())
> +		setup_force_cpu_cap(X86_FEATURE_PCPUISIDLE);
>  }
> diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c
> index 04cafc057bed..4489ca6d28c3 100644
> --- a/arch/x86/kernel/paravirt.c
> +++ b/arch/x86/kernel/paravirt.c
> @@ -366,6 +366,8 @@ struct paravirt_patch_template pv_ops = {
>  	.lock.kick			= paravirt_nop,
>  	.lock.vcpu_is_preempted		=
>  				PV_CALLEE_SAVE(__native_vcpu_is_preempted),
> +	.lock.pcpu_is_idle              =
> +				PV_CALLEE_SAVE(__native_pcpu_is_idle),
>  #endif /* SMP */
>  #endif
>  };
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index e5d5c5ed7dd4..61bd01f82cdb 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -3181,6 +3181,72 @@ static void kvm_vcpu_flush_tlb_guest(struct kvm_vcpu *vcpu)
>  	static_call(kvm_x86_tlb_flush_guest)(vcpu);
>  }
>  
> +static void kvm_steal_time_set_is_idle(struct kvm_vcpu *vcpu)
> +{
> +	struct kvm_host_map map;
> +	struct kvm_steal_time *st;
> +
> +	if (!(vcpu->arch.st.msr_val & KVM_MSR_ENABLED))
> +		return;
> +
> +	if (vcpu->arch.st.is_idle)
> +		return;
> +
> +	if (kvm_map_gfn(vcpu, vcpu->arch.st.msr_val >> PAGE_SHIFT, &map,
> +			&vcpu->arch.st.cache, true))
> +		return;
> +
> +	st = map.hva +
> +		offset_in_page(vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS);
> +
> +	st->is_idle = vcpu->arch.st.is_idle = KVM_PCPU_IS_IDLE;
> +
> +	kvm_unmap_gfn(vcpu, &map, &vcpu->arch.st.cache, true, true);
> +}
> +
> +static void kvm_steal_time_clear_is_idle(struct kvm_vcpu *vcpu)
> +{
> +	struct kvm_host_map map;
> +	struct kvm_steal_time *st;
> +
> +	if (!(vcpu->arch.st.msr_val & KVM_MSR_ENABLED))
> +		return;
> +
> +	if (!vcpu->arch.st.is_idle)
> +		return;
> +
> +	if (kvm_map_gfn(vcpu, vcpu->arch.st.msr_val >> PAGE_SHIFT, &map,
> +			&vcpu->arch.st.cache, false))
> +		return;
> +
> +	st = map.hva +
> +		offset_in_page(vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS);
> +
> +	if (guest_pv_has(vcpu, KVM_FEATURE_PV_TLB_FLUSH))
> +		xchg(&st->is_idle, 0);
> +	else
> +		st->is_idle = 0;
> +
> +	vcpu->arch.st.is_idle = 0;
> +
> +	kvm_unmap_gfn(vcpu, &map, &vcpu->arch.st.cache, true, false);
> +}
> +
> +
> +static DEFINE_PER_CPU(struct kvm_vcpu *, this_cpu_pre_run_vcpu);
> +
> +static void vcpu_load_update_pre_vcpu_callback(struct kvm_vcpu *new_vcpu, struct kvm_steal_time *st)
> +{
> +	struct kvm_vcpu *old_vcpu = __this_cpu_read(this_cpu_pre_run_vcpu);
> +
> +	if (!old_vcpu)
> +		return;
> +	if (old_vcpu != new_vcpu)
> +		kvm_steal_time_clear_is_idle(old_vcpu);
> +	else
> +		st->is_idle = new_vcpu->arch.st.is_idle = KVM_PCPU_IS_IDLE;
> +}
> +
>  static void record_steal_time(struct kvm_vcpu *vcpu)
>  {
>  	struct kvm_host_map map;
> @@ -3219,6 +3285,8 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
>  
>  	vcpu->arch.st.preempted = 0;
>  
> +	vcpu_load_update_pre_vcpu_callback(vcpu, st);
> +
>  	if (st->version & 1)
>  		st->version += 1;  /* first time write, random junk */
>  
> @@ -4290,6 +4358,8 @@ static void kvm_steal_time_set_preempted(struct kvm_vcpu *vcpu)
>  	kvm_unmap_gfn(vcpu, &map, &vcpu->arch.st.cache, true, true);
>  }
>  
> +extern int get_cpu_nr_running(int cpu);
> +
>  void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
>  {
>  	int idx;
> @@ -4304,8 +4374,15 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
>  	idx = srcu_read_lock(&vcpu->kvm->srcu);
>  	if (kvm_xen_msr_enabled(vcpu->kvm))
>  		kvm_xen_runstate_set_preempted(vcpu);
> -	else
> +	else {
>  		kvm_steal_time_set_preempted(vcpu);
> +
> +		if (get_cpu_nr_running(smp_processor_id()) <= 1)
> +			kvm_steal_time_set_is_idle(vcpu);
> +		else
> +			kvm_steal_time_clear_is_idle(vcpu);
> +	}
> +
>  	srcu_read_unlock(&vcpu->kvm->srcu, idx);
>  
>  	static_call(kvm_x86_vcpu_put)(vcpu);
> @@ -9693,6 +9770,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
>  	local_irq_enable();
>  	preempt_enable();
>  
> +	__this_cpu_write(this_cpu_pre_run_vcpu, vcpu);
> +
>  	vcpu->srcu_idx = srcu_read_lock(&vcpu->kvm->srcu);
>  
>  	/*
> @@ -11253,6 +11332,15 @@ void kvm_arch_pre_destroy_vm(struct kvm *kvm)
>  
>  void kvm_arch_destroy_vm(struct kvm *kvm)
>  {
> +	int cpu;
> +	struct kvm_vcpu *vcpu;
> +
> +	for_each_possible_cpu(cpu) {
> +		vcpu = per_cpu(this_cpu_pre_run_vcpu, cpu);
> +		if (vcpu && vcpu->kvm == kvm)
> +			per_cpu(this_cpu_pre_run_vcpu, cpu) = NULL;
> +	}
> +
>  	if (current->mm == kvm->mm) {
>  		/*
>  		 * Free memory regions allocated on behalf of userspace,
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index ec8d07d88641..dd4c41d2d8d3 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -1736,6 +1736,7 @@ extern int can_nice(const struct task_struct *p, const int nice);
>  extern int task_curr(const struct task_struct *p);
>  extern int idle_cpu(int cpu);
>  extern int available_idle_cpu(int cpu);
> +extern int available_idle_cpu_sched(int cpu);
>  extern int sched_setscheduler(struct task_struct *, int, const struct sched_param *);
>  extern int sched_setscheduler_nocheck(struct task_struct *, int, const struct sched_param *);
>  extern void sched_set_fifo(struct task_struct *p);
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 20ffcc044134..1bcc023ce581 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -6664,6 +6664,17 @@ int available_idle_cpu(int cpu)
>  	return 1;
>  }
>  
> +int available_idle_cpu_sched(int cpu)
> +{
> +	if (!idle_cpu(cpu))
> +		return 0;
> +
> +	if (!pcpu_is_idle(cpu))
> +		return 0;
> +
> +	return 1;
> +}
> +
>  /**
>   * idle_task - return the idle task for a given CPU.
>   * @cpu: the processor in question.
> @@ -10413,3 +10424,9 @@ void call_trace_sched_update_nr_running(struct rq *rq, int count)
>  {
>          trace_sched_update_nr_running_tp(rq, count);
>  }
> +
> +int get_cpu_nr_running(int cpu)
> +{
> +	return cpu_rq(cpu)->nr_running;
> +}
> +EXPORT_SYMBOL_GPL(get_cpu_nr_running);
> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> index 14a41a243f7b..49daefa91470 100644
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -101,6 +101,7 @@ extern void calc_global_load_tick(struct rq *this_rq);
>  extern long calc_load_fold_active(struct rq *this_rq, long adjust);
>  
>  extern void call_trace_sched_update_nr_running(struct rq *rq, int count);
> +

Stray change?

>  /*
>   * Helpers for converting nanosecond timing to jiffy resolution
>   */

-- 
Vitaly