Timer delays in VM

Timer delays in VM

[Eugene Huang]

[-- Attachment #1.1: Type: text/plain, Size: 1003 bytes --]

Hi,

I am running qemu on an arm64 CentOS host. Inside a ubuntu VM, a process runs a timer created using timer_t:
ev.sigev_notify_function = m_callback;
...
timer_create(CLOCK_MONOTONIC, &ev, &m_timer_t);

This timer sometimes has significant delays. For example, the 50 ms timer can have a callback delay of 100ms.

I did a host kernel trace and see a lot of WFx kvm_exits, and  the following events between kvm_exit and kvm_entry:
kvm_exit
kvm_wfx_arm64
kvm_get_timer_map
sched_switch
kvm_timer_save_state
kvm_timer_update_irq
vgic_update_irq_pending
kvm_timer_restore_state
kvm_vcpu_wakeup
kvm_arm_setup_debug
kvm_arm_set_dreg32
kvm_entry

I have the following questions:

  1.  Why there are a lot WFx exits? Is the timer dependent on it?
  2.  Does this timer rely on kvm timer irq injection?
  3.  What can be any possible causes for the timer delay? Are there some locking mechanisms which can cause the delay?
  4.  What parameters can tune this timer?

Thanks.


[-- Attachment #1.2: Type: text/html, Size: 7310 bytes --]

[-- Attachment #2: Type: text/plain, Size: 151 bytes --]

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Timer delays in VM

[Marc Zyngier]

[Please don't send HTML email and stick to plain text]

On 2022-02-28 18:02, Eugene Huang wrote:
> Hi,
> 
> I am running qemu on an arm64 CentOS host. Inside a ubuntu VM, a

I assume that by this you mean QEMU as the VMM for a KVM guest, right?

> process runs a timer created using timer_t:
> 
> ev.sigev_notify_function = m_callback;
> 
> …
> 
> timer_create(CLOCK_MONOTONIC, &ev, &m_timer_t);
> 
> This timer sometimes has significant delays. For example, the 50 ms
> timer can have a callback delay of 100ms.
> 
> I did a host kernel trace and see a lot of WFx kvm_exits, and  the
> following events between kvm_exit and kvm_entry:
> 
> kvm_exit
> 
> kvm_wfx_arm64
> 
> kvm_get_timer_map
> 
> sched_switch
> 
> kvm_timer_save_state
> 
> kvm_timer_update_irq
> 
> vgic_update_irq_pending
> 
> kvm_timer_restore_state
> 
> kvm_vcpu_wakeup
> 
> kvm_arm_setup_debug
> 
> kvm_arm_set_dreg32
> 
> kvm_entry

All of this is perfectly normal (guest hits WFI from its idle
loop, no interrupt is pending, trap to EL2, schedule out,
schedule back in, reenter the guest).

> 
> I have the following questions:
> 
>  	* Why there are a lot WFx exits? Is the timer dependent on it?

That's most probably because your vcpu goes idle and execute WFI to
Wait For an Interrupt. As no interrupt is pending, the vcpu exits
so that the host can do something useful until it gets an interrupt
that is targeted at the vcpu. On an idle VM, this probably happens
100s of times a second.

> 	* Does this timer rely on kvm timer irq injection?

Yes. A timer interrupt is always injected in SW. But the timer
interrupt can either come from the HW timer itself (the VM
was running while the timer expired), or from a SW timer that
KVM as setup if the guest was blocked on WFI.

> 	* What can be any possible causes for the timer delay? Are there some
> locking mechanisms which can cause the delay?

This completely depend on how loaded your host is, the respective
priorities of the various processes, and a million of other things.
This is no different from the same userspace running on the host.
It also depends on the *guest* kernel, by the way.

There are of course locks all over the place, but that's the very
nature of the beast.

> 	* What parameters can tune this timer?

None. You may want to check whether the delay is observed when the
VM has hit WFI or not.

You also don't mention what host kernel version you are running.
In general, please try and reproduce the issue using the latest
kernel version (5.16 at the moment). Please also indicate what
HW you are using.

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Timer delays in VM

[Andrew Jones]

On Mon, Feb 28, 2022 at 09:02:47PM +0000, Marc Zyngier wrote:
> 
> You also don't mention what host kernel version you are running.
> In general, please try and reproduce the issue using the latest
> kernel version (5.16 at the moment). Please also indicate what
> HW you are using.
>

Yes, please reply with the kernel version these delays are seen on
and also try to reproduce with a latest upstream kernel version. If
the delays are not present with the upstream kernel version, then
we can open a CentOS bug.

(You may want to experiment with host and guest kernel configs as
well.)

Thanks,
drew

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

RE: Timer delays in VM

[Eugene Huang]

Thanks Marc for the reply. Please see inline below marked with [EH].

> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Monday, February 28, 2022 1:03 PM
> To: Eugene Huang <eugeneh@nvidia.com>
> Cc: kvmarm@lists.cs.columbia.edu
> Subject: Re: Timer delays in VM
> 
> External email: Use caution opening links or attachments
> 
> 
> [Please don't send HTML email and stick to plain text]
> 
> On 2022-02-28 18:02, Eugene Huang wrote:
> > Hi,
> >
> > I am running qemu on an arm64 CentOS host. Inside a ubuntu VM, a
> 
> I assume that by this you mean QEMU as the VMM for a KVM guest, right?

[EH] Yes.

> 
> > process runs a timer created using timer_t:
> >
> > ev.sigev_notify_function = m_callback;
> >
> > …
> >
> > timer_create(CLOCK_MONOTONIC, &ev, &m_timer_t);
> >
> > This timer sometimes has significant delays. For example, the 50 ms
> > timer can have a callback delay of 100ms.
> >
> > I did a host kernel trace and see a lot of WFx kvm_exits, and  the
> > following events between kvm_exit and kvm_entry:
> >
> > kvm_exit
> >
> > kvm_wfx_arm64
> >
> > kvm_get_timer_map
> >
> > sched_switch
> >
> > kvm_timer_save_state
> >
> > kvm_timer_update_irq
> >
> > vgic_update_irq_pending
> >
> > kvm_timer_restore_state
> >
> > kvm_vcpu_wakeup
> >
> > kvm_arm_setup_debug
> >
> > kvm_arm_set_dreg32
> >
> > kvm_entry
> 
> All of this is perfectly normal (guest hits WFI from its idle loop, no interrupt is
> pending, trap to EL2, schedule out, schedule back in, reenter the guest).
> 
> >
> > I have the following questions:
> >
> >       * Why there are a lot WFx exits? Is the timer dependent on it?
> 
> That's most probably because your vcpu goes idle and execute WFI to Wait
> For an Interrupt. As no interrupt is pending, the vcpu exits so that the host
> can do something useful until it gets an interrupt that is targeted at the vcpu.
> On an idle VM, this probably happens 100s of times a second.
> 
> >       * Does this timer rely on kvm timer irq injection?
> 
> Yes. A timer interrupt is always injected in SW. But the timer interrupt can
> either come from the HW timer itself (the VM was running while the timer
> expired), or from a SW timer that KVM as setup if the guest was blocked on
> WFI.

<EH> Here for arm64, EL1Virtual Timer is used. EL1 Virtual Timer is a HW timer, correct?
There is an armvtimer implementation in QEMU 6.1+. Does this armvtimer make a difference?

> 
> >       * What can be any possible causes for the timer delay? Are there
> > some locking mechanisms which can cause the delay?
> 
> This completely depend on how loaded your host is, the respective priorities
> of the various processes, and a million of other things.
> This is no different from the same userspace running on the host.
> It also depends on the *guest* kernel, by the way.

<EH> Our guest kernel is 5.4. How is the *guest* kernel involved? Can you give an example? Do you have suggestions on the guest kernel version as well.

> 
> There are of course locks all over the place, but that's the very nature of the
> beast.
> 
> >       * What parameters can tune this timer?
> 
> None. You may want to check whether the delay is observed when the VM
> has hit WFI or not.

<EH> Yes, delay is observed after vm_exit because of WFx (not sure WFI or WFE) but only when on a different vCPU in the same VM some workload is started. Since we pin that workload to its own vCPU, in theory, it should not affect the timing of another vCPU.

> 
> You also don't mention what host kernel version you are running.
> In general, please try and reproduce the issue using the latest kernel version
> (5.16 at the moment). Please also indicate what HW you are using.

<EH> Tried 5.15 and 5.4 kernels. Both have the issue. Do you think 5.16 can make a difference? The HW is an Ampere Altra system.

> 
> Thanks,
> 
>          M.
> --
> Jazz is not dead. It just smells funny...
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

RE: Timer delays in VM

[Eugene Huang]

Update: tried 5.16.11 kernel. Same behavior.

> -----Original Message-----
> From: Eugene Huang
> Sent: Tuesday, March 1, 2022 11:04 AM
> To: Marc Zyngier <maz@kernel.org>
> Cc: kvmarm@lists.cs.columbia.edu
> Subject: RE: Timer delays in VM
> 
> Thanks Marc for the reply. Please see inline below marked with [EH].
> 
> > -----Original Message-----
> > From: Marc Zyngier <maz@kernel.org>
> > Sent: Monday, February 28, 2022 1:03 PM
> > To: Eugene Huang <eugeneh@nvidia.com>
> > Cc: kvmarm@lists.cs.columbia.edu
> > Subject: Re: Timer delays in VM
> >
> > External email: Use caution opening links or attachments
> >
> >
> > [Please don't send HTML email and stick to plain text]
> >
> > On 2022-02-28 18:02, Eugene Huang wrote:
> > > Hi,
> > >
> > > I am running qemu on an arm64 CentOS host. Inside a ubuntu VM, a
> >
> > I assume that by this you mean QEMU as the VMM for a KVM guest, right?
> 
> [EH] Yes.
> 
> >
> > > process runs a timer created using timer_t:
> > >
> > > ev.sigev_notify_function = m_callback;
> > >
> > > …
> > >
> > > timer_create(CLOCK_MONOTONIC, &ev, &m_timer_t);
> > >
> > > This timer sometimes has significant delays. For example, the 50 ms
> > > timer can have a callback delay of 100ms.
> > >
> > > I did a host kernel trace and see a lot of WFx kvm_exits, and  the
> > > following events between kvm_exit and kvm_entry:
> > >
> > > kvm_exit
> > >
> > > kvm_wfx_arm64
> > >
> > > kvm_get_timer_map
> > >
> > > sched_switch
> > >
> > > kvm_timer_save_state
> > >
> > > kvm_timer_update_irq
> > >
> > > vgic_update_irq_pending
> > >
> > > kvm_timer_restore_state
> > >
> > > kvm_vcpu_wakeup
> > >
> > > kvm_arm_setup_debug
> > >
> > > kvm_arm_set_dreg32
> > >
> > > kvm_entry
> >
> > All of this is perfectly normal (guest hits WFI from its idle loop, no
> > interrupt is pending, trap to EL2, schedule out, schedule back in, reenter
> the guest).
> >
> > >
> > > I have the following questions:
> > >
> > >       * Why there are a lot WFx exits? Is the timer dependent on it?
> >
> > That's most probably because your vcpu goes idle and execute WFI to
> > Wait For an Interrupt. As no interrupt is pending, the vcpu exits so
> > that the host can do something useful until it gets an interrupt that is
> targeted at the vcpu.
> > On an idle VM, this probably happens 100s of times a second.
> >
> > >       * Does this timer rely on kvm timer irq injection?
> >
> > Yes. A timer interrupt is always injected in SW. But the timer
> > interrupt can either come from the HW timer itself (the VM was running
> > while the timer expired), or from a SW timer that KVM as setup if the
> > guest was blocked on WFI.
> 
> <EH> Here for arm64, EL1Virtual Timer is used. EL1 Virtual Timer is a HW
> timer, correct?
> There is an armvtimer implementation in QEMU 6.1+. Does this armvtimer
> make a difference?
> 
> >
> > >       * What can be any possible causes for the timer delay? Are
> > > there some locking mechanisms which can cause the delay?
> >
> > This completely depend on how loaded your host is, the respective
> > priorities of the various processes, and a million of other things.
> > This is no different from the same userspace running on the host.
> > It also depends on the *guest* kernel, by the way.
> 
> <EH> Our guest kernel is 5.4. How is the *guest* kernel involved? Can you
> give an example? Do you have suggestions on the guest kernel version as
> well.
> 
> >
> > There are of course locks all over the place, but that's the very
> > nature of the beast.
> >
> > >       * What parameters can tune this timer?
> >
> > None. You may want to check whether the delay is observed when the VM
> > has hit WFI or not.
> 
> <EH> Yes, delay is observed after vm_exit because of WFx (not sure WFI or
> WFE) but only when on a different vCPU in the same VM some workload is
> started. Since we pin that workload to its own vCPU, in theory, it should not
> affect the timing of another vCPU.
> 
> >
> > You also don't mention what host kernel version you are running.
> > In general, please try and reproduce the issue using the latest kernel
> > version
> > (5.16 at the moment). Please also indicate what HW you are using.
> 
> <EH> Tried 5.15 and 5.4 kernels. Both have the issue. Do you think 5.16 can
> make a difference? The HW is an Ampere Altra system.
> 
> >
> > Thanks,
> >
> >          M.
> > --
> > Jazz is not dead. It just smells funny...
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Timer delays in VM

[Marc Zyngier]

On Tue, 01 Mar 2022 19:03:33 +0000,
Eugene Huang <eugeneh@nvidia.com> wrote:
> 
> > >       * Does this timer rely on kvm timer irq injection?
> > 
> > Yes. A timer interrupt is always injected in SW. But the timer interrupt can
> > either come from the HW timer itself (the VM was running while the timer
> > expired), or from a SW timer that KVM as setup if the guest was blocked on
> > WFI.
> 
> <EH> Here for arm64, EL1Virtual Timer is used. EL1 Virtual Timer is
> a HW timer, correct?  There is an armvtimer implementation in QEMU
> 6.1+. Does this armvtimer make a difference?

KVM only deals with the EL1 timers (both physical and virtual). I
guess that by 'armvtimer', you mean libvirt's front-end for the stolen
time feature to expose to the guest how wall clock and CPU time
diverge (i.e. it isn't a timer at all, but a dynamic correction for
it).

> > >       * What can be any possible causes for the timer delay? Are there
> > > some locking mechanisms which can cause the delay?
> > 
> > This completely depend on how loaded your host is, the respective priorities
> > of the various processes, and a million of other things.
> > This is no different from the same userspace running on the host.
> > It also depends on the *guest* kernel, by the way.
> 
> <EH> Our guest kernel is 5.4. How is the *guest* kernel involved?
> Can you give an example? Do you have suggestions on the guest kernel
> version as well.

It is the guest kernel that programs the timer, and KVM isn't involved
at all, specially on your HW (direct access to both timers on
VHE-capable systems).

> > >       * What parameters can tune this timer?
> > 
> > None. You may want to check whether the delay is observed when the VM
> > has hit WFI or not.
> 
> <EH> Yes, delay is observed after vm_exit because of WFx (not sure
> WFI or WFE) but only when on a different vCPU in the same VM some
> workload is started.

Let me see if I understand what you mean:

- vcpu-0 is running your timer test, everything is fine
- vcpu-1 starts some other workload, and this affects the timer test
  on the other vcpu

Is that correct? It so, this would tend to indicate that both vcpu
share some physical resources such as a physical CPU. How do you run
your VM?

Also, please work out whether you exit because of a blocking WFI or
WFE, as they are indicative of different guest behaviour.

> Since we pin that workload to its own vCPU, in
> theory, it should not affect the timing of another vCPU.

Why not? a vcpu is just a host thread, and if they share a physical
CPU at some point, there is a knock-on effect.

> > You also don't mention what host kernel version you are running.
> > In general, please try and reproduce the issue using the latest kernel version
> > (5.16 at the moment). Please also indicate what HW you are using.
> 
> <EH> Tried 5.15 and 5.4 kernels. Both have the issue. Do you think
> 5.16 can make a difference? The HW is an Ampere Altra system.

Unlikely. The Altra is a mostly sane system, as long as you make sure
that VMs don't migrate across sockets (at which point it becomes
laughably bad). Nothing to do with KVM though.

Are these kernels compiled from scratch? Or are they whatever the
distro ships? Same question for the guest.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

RE: Timer delays in VM

[Eugene Huang]

> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Tuesday, March 1, 2022 11:29 PM
> To: Eugene Huang <eugeneh@nvidia.com>
> Cc: kvmarm@lists.cs.columbia.edu
> Subject: Re: Timer delays in VM
> 
> 
> On Tue, 01 Mar 2022 19:03:33 +0000,
> Eugene Huang <eugeneh@nvidia.com> wrote:
> >
> > > >       * Does this timer rely on kvm timer irq injection?
> > >
> > > Yes. A timer interrupt is always injected in SW. But the timer
> > > interrupt can either come from the HW timer itself (the VM was
> > > running while the timer expired), or from a SW timer that KVM as
> > > setup if the guest was blocked on WFI.
> >
> > <EH> Here for arm64, EL1Virtual Timer is used. EL1 Virtual Timer is a
> > HW timer, correct?  There is an armvtimer implementation in QEMU 6.1+.
> > Does this armvtimer make a difference?
> 
> KVM only deals with the EL1 timers (both physical and virtual). I guess that by
> 'armvtimer', you mean libvirt's front-end for the stolen time feature to
> expose to the guest how wall clock and CPU time diverge (i.e. it isn't a timer
> at all, but a dynamic correction for it).

<EH> Yes, I mean the libvirt front-end setting.  Okay, got it. Thanks.

> 
> > > >       * What can be any possible causes for the timer delay? Are
> > > > there some locking mechanisms which can cause the delay?
> > >
> > > This completely depend on how loaded your host is, the respective
> > > priorities of the various processes, and a million of other things.
> > > This is no different from the same userspace running on the host.
> > > It also depends on the *guest* kernel, by the way.
> >
> > <EH> Our guest kernel is 5.4. How is the *guest* kernel involved?
> > Can you give an example? Do you have suggestions on the guest kernel
> > version as well.
> 
> It is the guest kernel that programs the timer, and KVM isn't involved at all,
> specially on your HW (direct access to both timers on VHE-capable systems).
> 
> > > >       * What parameters can tune this timer?
> > >
> > > None. You may want to check whether the delay is observed when the
> > > VM has hit WFI or not.
> >
> > <EH> Yes, delay is observed after vm_exit because of WFx (not sure WFI
> > or WFE) but only when on a different vCPU in the same VM some workload
> > is started.
> 
> Let me see if I understand what you mean:
> 
> - vcpu-0 is running your timer test, everything is fine
> - vcpu-1 starts some other workload, and this affects the timer test
>   on the other vcpu
> 
> Is that correct? It so, this would tend to indicate that both vcpu share some
> physical resources such as a physical CPU. How do you run your VM?

<EH> We have the following further 1-to-1 mappings:
pcpu-20 - vcpu-0 is running your timer test, everything is fine
pcpu-21 - vcpu-1 starts some other workload, and this affects the timer test
on the other vcpu

- Each vCPU thread is pinned to its individual pCPU on the host (vcpupin in libvirt).
- Each pCPU on which a vCPU thread runs is isolated on the host (isolcpus).
- Each vCPU that runs the workload is isolated in the guest VM (isolcpus).

So we are pretty sure the workloads are separated.

> 
> Also, please work out whether you exit because of a blocking WFI or WFE, as
> they are indicative of different guest behaviour.

<EH> Will do. Somehow our current trace does not show this information.

> 
> > Since we pin that workload to its own vCPU, in theory, it should not
> > affect the timing of another vCPU.
> 
> Why not? a vcpu is just a host thread, and if they share a physical CPU at
> some point, there is a knock-on effect.

<EH> Again, because of vcpupin in libvirt, there is no sharing of a pCPU among vCPUs. At least that is our configuration intention.

> 
> > > You also don't mention what host kernel version you are running.
> > > In general, please try and reproduce the issue using the latest
> > > kernel version
> > > (5.16 at the moment). Please also indicate what HW you are using.
> >
> > <EH> Tried 5.15 and 5.4 kernels. Both have the issue. Do you think
> > 5.16 can make a difference? The HW is an Ampere Altra system.
> 
> Unlikely. The Altra is a mostly sane system, as long as you make sure that
> VMs don't migrate across sockets (at which point it becomes laughably bad).
> Nothing to do with KVM though.

<EH> Right, there is no migration of VMs.
I see kvm arm timer related code is very different between 5.4 and 5.15/5.16.  Can we still use 5.4 for both the host and the guest?

> 
> Are these kernels compiled from scratch? Or are they whatever the distro
> ships? Same question for the guest.

<EH> Yes. Both host and guest kernels are compiled from scratch. 

Thanks,
Eugene

> 
> Thanks,
> 
>         M.
> 
> --
> Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Timer delays in VM

[Marc Zyngier]

On Thu, 03 Mar 2022 05:49:00 +0000,
Eugene Huang <eugeneh@nvidia.com> wrote:
> 
> <EH> We have the following further 1-to-1 mappings:
> pcpu-20 - vcpu-0 is running your timer test, everything is fine
> pcpu-21 - vcpu-1 starts some other workload, and this affects the timer test
> on the other vcpu
> 
> - Each vCPU thread is pinned to its individual pCPU on the host (vcpupin in libvirt).
> - Each pCPU on which a vCPU thread runs is isolated on the host (isolcpus).
> - Each vCPU that runs the workload is isolated in the guest VM (isolcpus).
> 
> So we are pretty sure the workloads are separated.

Hmmm. Isolcpus certainly is something I never use. You may want to
check whether this has an influence on what your test's behaviour. You
may also want to post your full libvirt config, just in case someone
spots an issue there (I won't, as I know next to nothing about
libvirt).

> 
> > 
> > Also, please work out whether you exit because of a blocking WFI or WFE, as
> > they are indicative of different guest behaviour.
> 
> <EH> Will do. Somehow our current trace does not show this information.
> 
> > 
> > > Since we pin that workload to its own vCPU, in theory, it should not
> > > affect the timing of another vCPU.
> > 
> > Why not? a vcpu is just a host thread, and if they share a physical CPU at
> > some point, there is a knock-on effect.
> 
> <EH> Again, because of vcpupin in libvirt, there is no sharing of a
> pCPU among vCPUs. At least that is our configuration intention.

Which may or may not be what happens in reality. libvirt is largely
opaque, and because you ask it to do something doesn't mean it happens
the way you hope it does.

> 
> > 
> > > > You also don't mention what host kernel version you are running.
> > > > In general, please try and reproduce the issue using the latest
> > > > kernel version
> > > > (5.16 at the moment). Please also indicate what HW you are using.
> > >
> > > <EH> Tried 5.15 and 5.4 kernels. Both have the issue. Do you think
> > > 5.16 can make a difference? The HW is an Ampere Altra system.
> > 
> > Unlikely. The Altra is a mostly sane system, as long as you make sure that
> > VMs don't migrate across sockets (at which point it becomes laughably bad).
> > Nothing to do with KVM though.
> 
> <EH> Right, there is no migration of VMs.
> I see kvm arm timer related code is very different between 5.4 and
> 5.15/5.16.  Can we still use 5.4 for both the host and the guest?

That's your call. I've stopped looking at 5.4 a couple of minutes
after it was released. If I'm going to look for something, that will
be on top of upstream.

	M.

-- 
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

RE: Timer delays in VM

[Eugene Huang]

Hi Marc,

Looks like to effectively troubleshoot, we need to upgrade our guest kernel to the latest version 5.17. I have these questions.

1. Can you please point out the kernel source that programs the arm virtual timer?
2. is The Generic Timer section in ARM Architecture Reference Manual ARMv8 the correct reference that we should refer to?
3. From the kernel community point of view, which kernel in the guest is the better - the Ubuntu one or the upstream one?

Thanks,
Eugene

> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Thursday, March 3, 2022 6:43 AM
> To: Eugene Huang <eugeneh@nvidia.com>
> Cc: kvmarm@lists.cs.columbia.edu
> Subject: Re: Timer delays in VM
> 
> External email: Use caution opening links or attachments
> 
> 
> On Thu, 03 Mar 2022 05:49:00 +0000,
> Eugene Huang <eugeneh@nvidia.com> wrote:
> >
> > <EH> We have the following further 1-to-1 mappings:
> > pcpu-20 - vcpu-0 is running your timer test, everything is fine
> > pcpu-21 - vcpu-1 starts some other workload, and this affects the
> > timer test on the other vcpu
> >
> > - Each vCPU thread is pinned to its individual pCPU on the host (vcpupin in
> libvirt).
> > - Each pCPU on which a vCPU thread runs is isolated on the host (isolcpus).
> > - Each vCPU that runs the workload is isolated in the guest VM (isolcpus).
> >
> > So we are pretty sure the workloads are separated.
> 
> Hmmm. Isolcpus certainly is something I never use. You may want to check
> whether this has an influence on what your test's behaviour. You may also
> want to post your full libvirt config, just in case someone spots an issue there
> (I won't, as I know next to nothing about libvirt).
> 
> >
> > >
> > > Also, please work out whether you exit because of a blocking WFI or
> > > WFE, as they are indicative of different guest behaviour.
> >
> > <EH> Will do. Somehow our current trace does not show this information.
> >
> > >
> > > > Since we pin that workload to its own vCPU, in theory, it should
> > > > not affect the timing of another vCPU.
> > >
> > > Why not? a vcpu is just a host thread, and if they share a physical
> > > CPU at some point, there is a knock-on effect.
> >
> > <EH> Again, because of vcpupin in libvirt, there is no sharing of a
> > pCPU among vCPUs. At least that is our configuration intention.
> 
> Which may or may not be what happens in reality. libvirt is largely opaque,
> and because you ask it to do something doesn't mean it happens the way
> you hope it does.
> 
> >
> > >
> > > > > You also don't mention what host kernel version you are running.
> > > > > In general, please try and reproduce the issue using the latest
> > > > > kernel version
> > > > > (5.16 at the moment). Please also indicate what HW you are using.
> > > >
> > > > <EH> Tried 5.15 and 5.4 kernels. Both have the issue. Do you think
> > > > 5.16 can make a difference? The HW is an Ampere Altra system.
> > >
> > > Unlikely. The Altra is a mostly sane system, as long as you make
> > > sure that VMs don't migrate across sockets (at which point it becomes
> laughably bad).
> > > Nothing to do with KVM though.
> >
> > <EH> Right, there is no migration of VMs.
> > I see kvm arm timer related code is very different between 5.4 and
> > 5.15/5.16.  Can we still use 5.4 for both the host and the guest?
> 
> That's your call. I've stopped looking at 5.4 a couple of minutes after it was
> released. If I'm going to look for something, that will be on top of upstream.
> 
>         M.
> 
> --
> Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Timer delays in VM

[Marc Zyngier]

Hi Eugene,

On Tue, 08 Mar 2022 07:50:50 +0000,
Eugene Huang <eugeneh@nvidia.com> wrote:
> 
> Hi Marc,
> 
> Looks like to effectively troubleshoot, we need to upgrade our guest
> kernel to the latest version 5.17. I have these questions.
> 
> 1. Can you please point out the kernel source that programs the arm
> virtual timer?

drivers/clocksource/arm_arch_timer.c

> 2. is The Generic Timer section in ARM Architecture Reference Manual
> ARMv8 the correct reference that we should refer to?

Yes, section D7. You'll also need the GIC specifications (IHI0069H in
your case), because you can't really understand how KVM works without
looking at both.

> 3. From the kernel community point of view, which kernel in the
> guest is the better - the Ubuntu one or the upstream one?

Upstream. In general, distro kernels are only a nuisance when
debugging things.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Timer delays in VM

[Eugene Huang]

[-- Attachment #1: Type: text/plain, Size: 1019 bytes --]

Hi,

I am running qemu on an arm64 CentOS host. Inside a ubuntu VM, a process runs a timer created using timer_t:
ev.sigev_notify_function = m_callback;
...
timer_create(CLOCK_MONOTONIC, &ev, &m_timer_t);

This timer sometimes has significant delays. For example, the 50 ms timer can have a callback delay of 100ms.

I did a host kernel trace and see a lot of WFx kvm_exits, and  the following events between kvm_exit and kvm_entry:
kvm_exit
kvm_wfx_arm64
kvm_get_timer_map
sched_switch
kvm_timer_save_state
kvm_timer_update_irq
vgic_update_irq_pending
kvm_timer_restore_state
kvm_vcpu_wakeup
kvm_arm_setup_debug
kvm_arm_set_dreg32
kvm_entry

I have the following questions:

1.       Why there are a lot WFx exits? Is the timer dependent on it?

2.       Does this timer rely on kvm timer irq injection?

3.       What can be any possible causes for the timer delay? Are there some locking mechanisms which can cause the delay?

4.       What parameters can tune this timer?

Thanks.

[-- Attachment #2: Type: text/html, Size: 7895 bytes --]

Re: Timer delays in VM

[Peter Maydell]

.

On Mon, 28 Feb 2022 at 07:04, Eugene Huang <eugeneh@nvidia.com> wrote:
> I am running qemu on an arm64 CentOS host. Inside a ubuntu VM, a process runs a timer created using timer_t:
>
> ev.sigev_notify_function = m_callback;
>
> …
>
> timer_create(CLOCK_MONOTONIC, &ev, &m_timer_t);
>
>
>
> This timer sometimes has significant delays. For example, the 50 ms timer can have a callback delay of 100ms.
>
>
>
> I did a host kernel trace and see a lot of WFx kvm_exits, and  the following events between kvm_exit and kvm_entry:

For questions that are really about KVM and not QEMU, you'll probably
find more experts on kvmarm@lists.cs.columbia.edu. My non-expert
look at this series of events:

> kvm_exit
> kvm_wfx_arm64

The guest kernel has nothing to do, so it idles,
waiting for the next interrupt.

> kvm_get_timer_map
> sched_switch
> kvm_timer_save_state

So the host schedules something else instead of this vCPU
(I think).

> kvm_timer_update_irq
> vgic_update_irq_pending

At some later date, the timer IRQ fires...

> kvm_timer_restore_state
> kvm_vcpu_wakeup
> kvm_arm_setup_debug
> kvm_arm_set_dreg32
> kvm_entry

...so we reschedule the vCPU and re-enter the guest.

I don't know why the timer might have significant delays,
but of course if your host system is very heavily loaded we
might not get round to rescheduling the vCPU for a bit.

thanks
-- PMM