3 preempted variables in kvm

3 preempted variables in kvm

[Alex Shi]

Hi All,

I am newbie on KVM side, so probably I am wrong on the following.
Please correct me if it is.

There are 3 preempted variables in kvm:
     1, kvm_vcpu.preempted  in include/linux/kvm_host.h
     2, kvm_steal_time.preempted 3, kvm_vcpu_arch.st.preempted in arch/x86
Seems all of them are set or cleared at the same time. Like,

vcpu_put:
        kvm_sched_out()-> set 3 preempted
                kvm_arch_vcpu_put():
                        kvm_steal_time_set_preempted

vcpu_load:
        kvm_sched_in() : clear above 3 preempted
                kvm_arch_vcpu_load() -> kvm_make_request(KVM_REQ_STEAL_UPDATE, vcpu);
                request dealed in vcpu_enter_guest() -> record_steal_time

Except the 2nd one reuse with KVM_FEATURE_PV_TLB_FLUSH bit which could be used
separately, Could we combine them into one, like just bool kvm_vcpu.preempted? and 
move out the KVM_FEATURE_PV_TLB_FLUSH. Believe all arch need this for a vcpu overcommit.

Thanks
Alex

Re: 3 preempted variables in kvm

[Sean Christopherson]

On Fri, Jan 22, 2021, Alex Shi wrote:
> Hi All,
> 
> I am newbie on KVM side, so probably I am wrong on the following.
> Please correct me if it is.
> 
> There are 3 preempted variables in kvm:
>      1, kvm_vcpu.preempted  in include/linux/kvm_host.h
>      2, kvm_steal_time.preempted
>      3, kvm_vcpu_arch.st.preempted in arch/x86
> Seems all of them are set or cleared at the same time. Like,

Not quite.  kvm_vcpu.preempted is set only in kvm_sched_out(), i.e. when the
vCPU was running and preempted by the host scheduler.  This is used by KVM when
KVM detects that a guest task appears to be waiting on a lock, in which case KVM
will bump the priority of preempted guest kernel threads in the hope that
scheduling in the preempted vCPU will release the lock.

kvm_steal_time.preempted is a paravirt struct that is shared with the guest.  It
is set on any call to kvm_arch_vcpu_put(), which covers kvm_sched_out() and adds
the case where the vCPU exits to userspace, e.g. for IO.  KVM itself hasn't been
preempted, but from the guest's perspective the CPU has been "preempted" in the
sense that CPU (from the guest's perspective) is not executing guest code.
Similar to KVM's vCPU scheduling heuristics, the guest kernel uses this info to
inform its scheduling, e.g. to avoid waiting on a lock owner to drop the lock
since the lock owner is not actively running.

kvm_vcpu_arch.st.preempted is effectively a host-side cache of
kvm_steal_time.preempted that's used to optimize kvm_arch_vcpu_put() by avoiding
the moderately costly mapping of guest.  It could be dropped, but it's a single
byte per vCPU so worth keeping even if the performance benefits are modest.

> vcpu_put:
>         kvm_sched_out()-> set 3 preempted
>                 kvm_arch_vcpu_put():
>                         kvm_steal_time_set_preempted
> 
> vcpu_load:
>         kvm_sched_in() : clear above 3 preempted
>                 kvm_arch_vcpu_load() -> kvm_make_request(KVM_REQ_STEAL_UPDATE, vcpu);
>                 request dealed in vcpu_enter_guest() -> record_steal_time
> 
> Except the 2nd one reuse with KVM_FEATURE_PV_TLB_FLUSH bit which could be used
> separately, Could we combine them into one, like just bool kvm_vcpu.preempted? and 
> move out the KVM_FEATURE_PV_TLB_FLUSH. Believe all arch need this for a vcpu overcommit.

Moving KVM_VCPU_FLUSH_TLB out of kvm_steal_time.preempted isn't viable. The
guest kernel is only allowed to rely on the host to flush the vCPU's TLB if it
knows the vCPU is preempted (from its perspective), as that's the only way it
can guarantee that KVM will observe the TLB flush request before enterring the
vCPU.  KVM_VCPU_FLUSH_TLB and KVM_VCPU_PREEMPTED need to be in the same word so
KVM can read and clear them atomically, otherwise there would be a window where
KVM would miss the flush request.

Re: 3 preempted variables in kvm

[Alex Shi]

Hi Sean,

Thanks a lot for the detailed explanations.
Yes, they are all meaningful variables on x86. But for more archs, guess the lock
issue on different arch's guest are similar. Maybe a abstraction on the point would
be very helpful. Any comments?

Thanks
Alex

在 2021/1/23 上午3:30, Sean Christopherson 写道:
> On Fri, Jan 22, 2021, Alex Shi wrote:
>> Hi All,
>>
>> I am newbie on KVM side, so probably I am wrong on the following.
>> Please correct me if it is.
>>
>> There are 3 preempted variables in kvm:
>>      1, kvm_vcpu.preempted  in include/linux/kvm_host.h
>>      2, kvm_steal_time.preempted
>>      3, kvm_vcpu_arch.st.preempted in arch/x86
>> Seems all of them are set or cleared at the same time. Like,
> 
> Not quite.  kvm_vcpu.preempted is set only in kvm_sched_out(), i.e. when the
> vCPU was running and preempted by the host scheduler.  This is used by KVM when
> KVM detects that a guest task appears to be waiting on a lock, in which case KVM
> will bump the priority of preempted guest kernel threads in the hope that
> scheduling in the preempted vCPU will release the lock.
> 
> kvm_steal_time.preempted is a paravirt struct that is shared with the guest.  It
> is set on any call to kvm_arch_vcpu_put(), which covers kvm_sched_out() and adds
> the case where the vCPU exits to userspace, e.g. for IO.  KVM itself hasn't been
> preempted, but from the guest's perspective the CPU has been "preempted" in the
> sense that CPU (from the guest's perspective) is not executing guest code.
> Similar to KVM's vCPU scheduling heuristics, the guest kernel uses this info to
> inform its scheduling, e.g. to avoid waiting on a lock owner to drop the lock
> since the lock owner is not actively running.
> 
> kvm_vcpu_arch.st.preempted is effectively a host-side cache of
> kvm_steal_time.preempted that's used to optimize kvm_arch_vcpu_put() by avoiding
> the moderately costly mapping of guest.  It could be dropped, but it's a single
> byte per vCPU so worth keeping even if the performance benefits are modest.
> 
>> vcpu_put:
>>         kvm_sched_out()-> set 3 preempted
>>                 kvm_arch_vcpu_put():
>>                         kvm_steal_time_set_preempted
>>
>> vcpu_load:
>>         kvm_sched_in() : clear above 3 preempted
>>                 kvm_arch_vcpu_load() -> kvm_make_request(KVM_REQ_STEAL_UPDATE, vcpu);
>>                 request dealed in vcpu_enter_guest() -> record_steal_time
>>
>> Except the 2nd one reuse with KVM_FEATURE_PV_TLB_FLUSH bit which could be used
>> separately, Could we combine them into one, like just bool kvm_vcpu.preempted? and 
>> move out the KVM_FEATURE_PV_TLB_FLUSH. Believe all arch need this for a vcpu overcommit.
> 
> Moving KVM_VCPU_FLUSH_TLB out of kvm_steal_time.preempted isn't viable. The
> guest kernel is only allowed to rely on the host to flush the vCPU's TLB if it
> knows the vCPU is preempted (from its perspective), as that's the only way it
> can guarantee that KVM will observe the TLB flush request before enterring the
> vCPU.  KVM_VCPU_FLUSH_TLB and KVM_VCPU_PREEMPTED need to be in the same word so
> KVM can read and clear them atomically, otherwise there would be a window where
> KVM would miss the flush request.
> 

Re: 3 preempted variables in kvm

[Marc Zyngier]

On 2021-01-25 05:49, Alex Shi wrote:
> Hi Sean,
> 
> Thanks a lot for the detailed explanations.
> Yes, they are all meaningful variables on x86. But for more archs,
> guess the lock
> issue on different arch's guest are similar. Maybe a abstraction on
> the point would
> be very helpful. Any comments?

It depends on what you have in mind. The whole "yield to another
vcpu" mechanism is already abstracted behind kvm_vcpu_on_spin(),
which at arm64, s390 and x86 are using. You could investigate
whether this is a valid approach for Power and MIPS.

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...