What does KVM_HINTS_REALTIME do?

What does KVM_HINTS_REALTIME do?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1866 bytes --]

Hi,
I was just in a discussion where we realized KVM_HINTS_REALTIME is a
little underdocumented. Here is attempt to address that. Please correct
me if there are inaccuracies or reply if you have additional questions:

KVM_HINTS_REALTIME (aka QEMU kvm-hint-dedicated) is defined as follows
in Documentation/virt/kvm/cpuid.rst:

  guest checks this feature bit to determine that vCPUs are never
  preempted for an unlimited time allowing optimizations

Users or management tools set this flag themselves (it is not set
automatically). This raises the question of what effects this flag has
and when it should be set.

When should I set KVM_HINTS_REALTIME?
-------------------------------------
When vCPUs are pinned to dedicated pCPUs. Even better if the isolcpus=
kernel parameter is used on the host so there are no disturbances.

Is the flag guest-wide or per-vCPU?
-----------------------------------
This flag is guest-wide so all vCPUs should be dedicated, not just some
of them.

Which Linux guest features are affected?
----------------------------------------
PV spinlocks, PV TLB flush, and PV sched yield are disabled by
KVM_HINTS_REALTIME. This is because no other vCPUs or host tasks will be
running on the pCPUs, so there is no benefit in involving the host.

The cpuidle-haltpoll driver is enabled by KVM_HINTS_REALTIME. This
driver performs busy waiting inside the guest before halting the CPU in
order to avoid the vCPU's wakeup latency. This driver also has a boolean
"force" module parameter if you wish to enable it without setting
KVM_HINTS_REALTIME.

When KVM_HINTS_REALTIME is set, the KVM_CAP_X86_DISABLE_EXITS capability
can also be used to disable MWAIT/HLT/PAUSE/CSTATE exits. This improves
the latency of these operations. The user or management tools need to
disable these exits themselves, e.g. with QEMU's -overcommit cpu-pm=on.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: What does KVM_HINTS_REALTIME do?

[Marcelo Tosatti]

On Tue, Aug 03, 2021 at 03:11:04PM +0100, Stefan Hajnoczi wrote:
> Hi,
> I was just in a discussion where we realized KVM_HINTS_REALTIME is a
> little underdocumented. Here is attempt to address that. Please correct
> me if there are inaccuracies or reply if you have additional questions:
> 
> KVM_HINTS_REALTIME (aka QEMU kvm-hint-dedicated) is defined as follows
> in Documentation/virt/kvm/cpuid.rst:
> 
>   guest checks this feature bit to determine that vCPUs are never
>   preempted for an unlimited time allowing optimizations
> 
> Users or management tools set this flag themselves (it is not set
> automatically). This raises the question of what effects this flag has
> and when it should be set.
> 
> When should I set KVM_HINTS_REALTIME?
> -------------------------------------
> When vCPUs are pinned to dedicated pCPUs. Even better if the isolcpus=
> kernel parameter is used on the host so there are no disturbances.
> 
> Is the flag guest-wide or per-vCPU?
> -----------------------------------
> This flag is guest-wide so all vCPUs should be dedicated, not just some
> of them.
> 
> Which Linux guest features are affected?
> ----------------------------------------
> PV spinlocks, PV TLB flush, and PV sched yield are disabled by
> KVM_HINTS_REALTIME. This is because no other vCPUs or host tasks will be
> running on the pCPUs, so there is no benefit in involving the host.
> 
> The cpuidle-haltpoll driver is enabled by KVM_HINTS_REALTIME. This
> driver performs busy waiting inside the guest before halting the CPU in
> order to avoid the vCPU's wakeup latency. This driver also has a boolean
> "force" module parameter if you wish to enable it without setting
> KVM_HINTS_REALTIME.
> 
> When KVM_HINTS_REALTIME is set, the KVM_CAP_X86_DISABLE_EXITS capability
> can also be used to disable MWAIT/HLT/PAUSE/CSTATE exits. This improves
> the latency of these operations. The user or management tools need to
> disable these exits themselves, e.g. with QEMU's -overcommit cpu-pm=on.
> 
> Stefan

Looks good.

Re: What does KVM_HINTS_REALTIME do?

[Wanpeng Li]

On Tue, 3 Aug 2021 at 22:14, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> Hi,
> I was just in a discussion where we realized KVM_HINTS_REALTIME is a
> little underdocumented. Here is attempt to address that. Please correct
> me if there are inaccuracies or reply if you have additional questions:
>
> KVM_HINTS_REALTIME (aka QEMU kvm-hint-dedicated) is defined as follows
> in Documentation/virt/kvm/cpuid.rst:
>
>   guest checks this feature bit to determine that vCPUs are never
>   preempted for an unlimited time allowing optimizations
>
> Users or management tools set this flag themselves (it is not set
> automatically). This raises the question of what effects this flag has
> and when it should be set.
>
> When should I set KVM_HINTS_REALTIME?
> -------------------------------------
> When vCPUs are pinned to dedicated pCPUs. Even better if the isolcpus=
> kernel parameter is used on the host so there are no disturbances.
>
> Is the flag guest-wide or per-vCPU?
> -----------------------------------
> This flag is guest-wide so all vCPUs should be dedicated, not just some
> of them.
>
> Which Linux guest features are affected?
> ----------------------------------------
> PV spinlocks, PV TLB flush, and PV sched yield are disabled by
> KVM_HINTS_REALTIME. This is because no other vCPUs or host tasks will be
> running on the pCPUs, so there is no benefit in involving the host.
>
> The cpuidle-haltpoll driver is enabled by KVM_HINTS_REALTIME. This
> driver performs busy waiting inside the guest before halting the CPU in
> order to avoid the vCPU's wakeup latency. This driver also has a boolean
> "force" module parameter if you wish to enable it without setting
> KVM_HINTS_REALTIME.
>
> When KVM_HINTS_REALTIME is set, the KVM_CAP_X86_DISABLE_EXITS capability
> can also be used to disable MWAIT/HLT/PAUSE/CSTATE exits. This improves
> the latency of these operations. The user or management tools need to
> disable these exits themselves, e.g. with QEMU's -overcommit cpu-pm=on.

Looks good to me, too. :)

    Wanpeng

Re: What does KVM_HINTS_REALTIME do?

[Pankaj Gupta]

> Hi,
> I was just in a discussion where we realized KVM_HINTS_REALTIME is a
> little underdocumented. Here is attempt to address that. Please correct
> me if there are inaccuracies or reply if you have additional questions:
>
> KVM_HINTS_REALTIME (akaalso  QEMU kvm-hint-dedicated) is defined as follows
> in Documentation/virt/kvm/cpuid.rst:
>
>   guest checks this feature bit to determine that vCPUs are never
>   preempted for an unlimited time allowing optimizations
>
> Users or management tools set this flag themselves (it is not set
> automatically). This raises the question of what effects this flag has
> and when it should be set.
>
> When should I set KVM_HINTS_REALTIME?
> -------------------------------------
> When vCPUs are pinned to dedicated pCPUs. Even better if the isolcpus=
> kernel parameter is used on the host so there are no disturbances.
>
> Is the flag guest-wide or per-vCPU?
> -----------------------------------
> This flag is guest-wide so all vCPUs should be dedicated, not just some
> of them.
>
> Which Linux guest features are affected?
> ----------------------------------------
> PV spinlocks, PV TLB flush, and PV sched yield are disabled by
> KVM_HINTS_REALTIME. This is because no other vCPUs or host tasks will be
> running on the pCPUs, so there is no benefit in involving the host.

Do we need to mention "halt_poll_ns" at host side also will also be disabled?

>
> The cpuidle-haltpoll driver is enabled by KVM_HINTS_REALTIME. This
> driver performs busy waiting inside the guest before halting the CPU in
> order to avoid the vCPU's wakeup latency. This driver also has a boolean
> "force" module parameter if you wish to enable it without setting
> KVM_HINTS_REALTIME.
>
> When KVM_HINTS_REALTIME is set, the KVM_CAP_X86_DISABLE_EXITS capability
> can also be used to disable MWAIT/HLT/PAUSE/CSTATE exits. This improves
> the latency of these operations. The user or management tools need to
> disable these exits themselves, e.g. with QEMU's -overcommit cpu-pm=on.

This looks good. Thank you.

Thanks,
Pankaj

Re: What does KVM_HINTS_REALTIME do?

[Pankaj Gupta]

> > Hi,
> > I was just in a discussion where we realized KVM_HINTS_REALTIME is a
> > little underdocumented. Here is attempt to address that. Please correct
> > me if there are inaccuracies or reply if you have additional questions:
> >
> > KVM_HINTS_REALTIME (akaalso  QEMU kvm-hint-dedicated) is defined as follows
> > in Documentation/virt/kvm/cpuid.rst:
> >
> >   guest checks this feature bit to determine that vCPUs are never
> >   preempted for an unlimited time allowing optimizations
> >
> > Users or management tools set this flag themselves (it is not set
> > automatically). This raises the question of what effects this flag has
> > and when it should be set.
> >
> > When should I set KVM_HINTS_REALTIME?
> > -------------------------------------
> > When vCPUs are pinned to dedicated pCPUs. Even better if the isolcpus=
> > kernel parameter is used on the host so there are no disturbances.
> >
> > Is the flag guest-wide or per-vCPU?
> > -----------------------------------
> > This flag is guest-wide so all vCPUs should be dedicated, not just some
> > of them.
> >
> > Which Linux guest features are affected?
> > ----------------------------------------
> > PV spinlocks, PV TLB flush, and PV sched yield are disabled by
> > KVM_HINTS_REALTIME. This is because no other vCPUs or host tasks will be
> > running on the pCPUs, so there is no benefit in involving the host.
>
> Do we need to mention "halt_poll_ns" at host side also will also be disabled?
with KVM_FEATURE_POLL_CONTROL

Sorry, pressed enter quickly in previous email.
>
> >
> > The cpuidle-haltpoll driver is enabled by KVM_HINTS_REALTIME. This
> > driver performs busy waiting inside the guest before halting the CPU in
> > order to avoid the vCPU's wakeup latency. This driver also has a boolean
> > "force" module parameter if you wish to enable it without setting
> > KVM_HINTS_REALTIME.
> >
> > When KVM_HINTS_REALTIME is set, the KVM_CAP_X86_DISABLE_EXITS capability
> > can also be used to disable MWAIT/HLT/PAUSE/CSTATE exits. This improves
> > the latency of these operations. The user or management tools need to
> > disable these exits themselves, e.g. with QEMU's -overcommit cpu-pm=on.
>
> This looks good. Thank you.
>
> Thanks,
> Pankaj

Re: What does KVM_HINTS_REALTIME do?

[Paolo Bonzini]

On 04/08/21 11:06, Pankaj Gupta wrote:
>> Do we need to mention "halt_poll_ns" at host side also will also be disabled?
> with KVM_FEATURE_POLL_CONTROL
> 
> Sorry, pressed enter quickly in previous email.

That is done by the cpuidle-haltpoll driver, not just by the presence of 
the feature.

Paolo

Re: What does KVM_HINTS_REALTIME do?

[Pankaj Gupta]

> >> Do we need to mention "halt_poll_ns" at host side also will also be disabled?
> > with KVM_FEATURE_POLL_CONTROL
> >
> > Sorry, pressed enter quickly in previous email.
>
> That is done by the cpuidle-haltpoll driver, not just by the presence of
> the feature.

Okay.

Thanks,
Pankaj

>
> Paolo
>