vPT rework (and timer mode)

vPT rework (and timer mode)

[Roger Pau Monné]

Hello,

I've been doing some work with the virtual timers infrastructure in
order to improve some of it's shortcomings. See:

https://lists.xenproject.org/archives/html/xen-devel/2020-06/msg00919.html

For an example of such issues, and how the emulated timers are not
architecturally correct.

It's my understanding that the purpose of pt_update_irq and
pt_intr_post is to attempt to implement the "delay for missed ticks"
mode, where Xen will accumulate timer interrupts if they cannot be
injected. As shown by the patch above, this is all broken when the
timer is added to a vCPU (pt->vcpu) different than the actual target
vCPU where the interrupt gets delivered (note this can also be a list
of vCPUs if routed from the IO-APIC using Fixed mode).

I'm at lost at how to fix this so that virtual timers work properly
and we also keep the "delay for missed ticks" mode without doing a
massive rework and somehow keeping track of where injected interrupts
originated, which seems an overly complicated solution.

My proposal hence would be to completely remove the timer_mode, and
just treat virtual timer interrupts as other interrupts, ie: they will
be injected from the callback (pt_timer_fn) and the vCPU(s) would be
kicked. Whether interrupts would get lost (ie: injected when a
previous one is still pending) depends on the contention on the
system. I'm not aware of any current OS that uses timer interrupts as
a way to track time. I think current OSes know the differences between
a timer counter and an event timer, and will use them appropriately.

This would allow to get rid of pt_update_irq and pt_intr_post calls in
the VMX/SVM interrupt injection paths, and likely simplify the virtual
timers code quite a lot. Note the guest would also always track the
real wallclock.

AFAICT such change would also allow to get rid of the per-vCPU vpt
lists.

Wanted to get some feedback on this approach before starting to do the
work, since as said above it will involve dropping the timer modes.

Thanks, Roger.

Re: vPT rework (and timer mode)

[Jan Beulich]

On 01.07.2020 11:02, Roger Pau Monné wrote:
> It's my understanding that the purpose of pt_update_irq and
> pt_intr_post is to attempt to implement the "delay for missed ticks"
> mode, where Xen will accumulate timer interrupts if they cannot be
> injected. As shown by the patch above, this is all broken when the
> timer is added to a vCPU (pt->vcpu) different than the actual target
> vCPU where the interrupt gets delivered (note this can also be a list
> of vCPUs if routed from the IO-APIC using Fixed mode).
> 
> I'm at lost at how to fix this so that virtual timers work properly
> and we also keep the "delay for missed ticks" mode without doing a
> massive rework and somehow keeping track of where injected interrupts
> originated, which seems an overly complicated solution.
> 
> My proposal hence would be to completely remove the timer_mode, and
> just treat virtual timer interrupts as other interrupts, ie: they will
> be injected from the callback (pt_timer_fn) and the vCPU(s) would be
> kicked. Whether interrupts would get lost (ie: injected when a
> previous one is still pending) depends on the contention on the
> system. I'm not aware of any current OS that uses timer interrupts as
> a way to track time. I think current OSes know the differences between
> a timer counter and an event timer, and will use them appropriately.

Fundamentally - why not, the more that this promises to be a
simplification. The question we need to answer up front is whether
we're happy to possibly break old OSes (presumably ones no-one
ought to be using anymore these days, due to their support life
cycles long having ended).

Jan

Re: vPT rework (and timer mode)

[Andrew Cooper]

On 03/07/2020 15:50, Jan Beulich wrote:
> On 01.07.2020 11:02, Roger Pau Monné wrote:
>> It's my understanding that the purpose of pt_update_irq and
>> pt_intr_post is to attempt to implement the "delay for missed ticks"
>> mode, where Xen will accumulate timer interrupts if they cannot be
>> injected. As shown by the patch above, this is all broken when the
>> timer is added to a vCPU (pt->vcpu) different than the actual target
>> vCPU where the interrupt gets delivered (note this can also be a list
>> of vCPUs if routed from the IO-APIC using Fixed mode).
>>
>> I'm at lost at how to fix this so that virtual timers work properly
>> and we also keep the "delay for missed ticks" mode without doing a
>> massive rework and somehow keeping track of where injected interrupts
>> originated, which seems an overly complicated solution.
>>
>> My proposal hence would be to completely remove the timer_mode, and
>> just treat virtual timer interrupts as other interrupts, ie: they will
>> be injected from the callback (pt_timer_fn) and the vCPU(s) would be
>> kicked. Whether interrupts would get lost (ie: injected when a
>> previous one is still pending) depends on the contention on the
>> system. I'm not aware of any current OS that uses timer interrupts as
>> a way to track time. I think current OSes know the differences between
>> a timer counter and an event timer, and will use them appropriately.
> Fundamentally - why not, the more that this promises to be a
> simplification. The question we need to answer up front is whether
> we're happy to possibly break old OSes (presumably ones no-one
> ought to be using anymore these days, due to their support life
> cycles long having ended).

The various timer modes were all compatibility, and IIRC, mostly for
Windows XP and older which told time by counting the number of timer
interrupts.

Paul - you might remember better than me?

Its possibly worth noting that issues in this are cause triple faults in
OVMF (it seems to enable interrupts in its timer handler), and breaks
in-guest kexec (because our timer-targetting logic doesn't work in a way
remotely close to real hardware when the kexec kernel is booting on a
non-zero vCPU).

~Andrew

RE: vPT rework (and timer mode)

[Paul Durrant]

> -----Original Message-----
> From: Andrew Cooper <andrew.cooper3@citrix.com>
> Sent: 03 July 2020 16:03
> To: Jan Beulich <jbeulich@suse.com>; Roger Pau Monné <roger.pau@citrix.com>
> Cc: xen-devel@lists.xenproject.org; Wei Liu <wl@xen.org>; Paul Durrant <paul@xen.org>
> Subject: Re: vPT rework (and timer mode)
> 
> On 03/07/2020 15:50, Jan Beulich wrote:
> > On 01.07.2020 11:02, Roger Pau Monné wrote:
> >> It's my understanding that the purpose of pt_update_irq and
> >> pt_intr_post is to attempt to implement the "delay for missed ticks"
> >> mode, where Xen will accumulate timer interrupts if they cannot be
> >> injected. As shown by the patch above, this is all broken when the
> >> timer is added to a vCPU (pt->vcpu) different than the actual target
> >> vCPU where the interrupt gets delivered (note this can also be a list
> >> of vCPUs if routed from the IO-APIC using Fixed mode).
> >>
> >> I'm at lost at how to fix this so that virtual timers work properly
> >> and we also keep the "delay for missed ticks" mode without doing a
> >> massive rework and somehow keeping track of where injected interrupts
> >> originated, which seems an overly complicated solution.
> >>
> >> My proposal hence would be to completely remove the timer_mode, and
> >> just treat virtual timer interrupts as other interrupts, ie: they will
> >> be injected from the callback (pt_timer_fn) and the vCPU(s) would be
> >> kicked. Whether interrupts would get lost (ie: injected when a
> >> previous one is still pending) depends on the contention on the
> >> system. I'm not aware of any current OS that uses timer interrupts as
> >> a way to track time. I think current OSes know the differences between
> >> a timer counter and an event timer, and will use them appropriately.
> > Fundamentally - why not, the more that this promises to be a
> > simplification. The question we need to answer up front is whether
> > we're happy to possibly break old OSes (presumably ones no-one
> > ought to be using anymore these days, due to their support life
> > cycles long having ended).
> 
> The various timer modes were all compatibility, and IIRC, mostly for
> Windows XP and older which told time by counting the number of timer
> interrupts.
> 
> Paul - you might remember better than me?

I think it is only quite recently that Windows has started favouring enlightened time sources rather than counting ticks but an admin may still turn all the viridian enlightenments off so just dropping ticks will probably still cause time to drift backwards.

  Paul

> 
> Its possibly worth noting that issues in this are cause triple faults in
> OVMF (it seems to enable interrupts in its timer handler), and breaks
> in-guest kexec (because our timer-targetting logic doesn't work in a way
> remotely close to real hardware when the kexec kernel is booting on a
> non-zero vCPU).
> 
> ~Andrew

Re: vPT rework (and timer mode)

[Roger Pau Monné]

On Mon, Jul 06, 2020 at 08:03:50AM +0100, Paul Durrant wrote:
> > -----Original Message-----
> > From: Andrew Cooper <andrew.cooper3@citrix.com>
> > Sent: 03 July 2020 16:03
> > To: Jan Beulich <jbeulich@suse.com>; Roger Pau Monné <roger.pau@citrix.com>
> > Cc: xen-devel@lists.xenproject.org; Wei Liu <wl@xen.org>; Paul Durrant <paul@xen.org>
> > Subject: Re: vPT rework (and timer mode)
> > 
> > On 03/07/2020 15:50, Jan Beulich wrote:
> > > On 01.07.2020 11:02, Roger Pau Monné wrote:
> > >> It's my understanding that the purpose of pt_update_irq and
> > >> pt_intr_post is to attempt to implement the "delay for missed ticks"
> > >> mode, where Xen will accumulate timer interrupts if they cannot be
> > >> injected. As shown by the patch above, this is all broken when the
> > >> timer is added to a vCPU (pt->vcpu) different than the actual target
> > >> vCPU where the interrupt gets delivered (note this can also be a list
> > >> of vCPUs if routed from the IO-APIC using Fixed mode).
> > >>
> > >> I'm at lost at how to fix this so that virtual timers work properly
> > >> and we also keep the "delay for missed ticks" mode without doing a
> > >> massive rework and somehow keeping track of where injected interrupts
> > >> originated, which seems an overly complicated solution.
> > >>
> > >> My proposal hence would be to completely remove the timer_mode, and
> > >> just treat virtual timer interrupts as other interrupts, ie: they will
> > >> be injected from the callback (pt_timer_fn) and the vCPU(s) would be
> > >> kicked. Whether interrupts would get lost (ie: injected when a
> > >> previous one is still pending) depends on the contention on the
> > >> system. I'm not aware of any current OS that uses timer interrupts as
> > >> a way to track time. I think current OSes know the differences between
> > >> a timer counter and an event timer, and will use them appropriately.
> > > Fundamentally - why not, the more that this promises to be a
> > > simplification. The question we need to answer up front is whether
> > > we're happy to possibly break old OSes (presumably ones no-one
> > > ought to be using anymore these days, due to their support life
> > > cycles long having ended).
> > 
> > The various timer modes were all compatibility, and IIRC, mostly for
> > Windows XP and older which told time by counting the number of timer
> > interrupts.
> > 
> > Paul - you might remember better than me?
> 
> I think it is only quite recently that Windows has started favouring enlightened time sources rather than counting ticks but an admin may still turn all the viridian enlightenments off so just dropping ticks will probably still cause time to drift backwards.

Even when not using the viridian enlightenments, Windows should rely
on emulated time counters (or the TSC) rather than counting ticks?

I guess I could give it a try with one of the emulated Windows versions
that we test on osstest.

Thanks, Roger.

RE: vPT rework (and timer mode)

[Paul Durrant]

> -----Original Message-----
> From: Roger Pau Monné <roger.pau@citrix.com>
> Sent: 06 July 2020 09:32
> To: paul@xen.org
> Cc: 'Andrew Cooper' <andrew.cooper3@citrix.com>; 'Jan Beulich' <jbeulich@suse.com>; xen-
> devel@lists.xenproject.org; 'Wei Liu' <wl@xen.org>
> Subject: Re: vPT rework (and timer mode)
> 
> On Mon, Jul 06, 2020 at 08:03:50AM +0100, Paul Durrant wrote:
> > > -----Original Message-----
> > > From: Andrew Cooper <andrew.cooper3@citrix.com>
> > > Sent: 03 July 2020 16:03
> > > To: Jan Beulich <jbeulich@suse.com>; Roger Pau Monné <roger.pau@citrix.com>
> > > Cc: xen-devel@lists.xenproject.org; Wei Liu <wl@xen.org>; Paul Durrant <paul@xen.org>
> > > Subject: Re: vPT rework (and timer mode)
> > >
> > > On 03/07/2020 15:50, Jan Beulich wrote:
> > > > On 01.07.2020 11:02, Roger Pau Monné wrote:
> > > >> It's my understanding that the purpose of pt_update_irq and
> > > >> pt_intr_post is to attempt to implement the "delay for missed ticks"
> > > >> mode, where Xen will accumulate timer interrupts if they cannot be
> > > >> injected. As shown by the patch above, this is all broken when the
> > > >> timer is added to a vCPU (pt->vcpu) different than the actual target
> > > >> vCPU where the interrupt gets delivered (note this can also be a list
> > > >> of vCPUs if routed from the IO-APIC using Fixed mode).
> > > >>
> > > >> I'm at lost at how to fix this so that virtual timers work properly
> > > >> and we also keep the "delay for missed ticks" mode without doing a
> > > >> massive rework and somehow keeping track of where injected interrupts
> > > >> originated, which seems an overly complicated solution.
> > > >>
> > > >> My proposal hence would be to completely remove the timer_mode, and
> > > >> just treat virtual timer interrupts as other interrupts, ie: they will
> > > >> be injected from the callback (pt_timer_fn) and the vCPU(s) would be
> > > >> kicked. Whether interrupts would get lost (ie: injected when a
> > > >> previous one is still pending) depends on the contention on the
> > > >> system. I'm not aware of any current OS that uses timer interrupts as
> > > >> a way to track time. I think current OSes know the differences between
> > > >> a timer counter and an event timer, and will use them appropriately.
> > > > Fundamentally - why not, the more that this promises to be a
> > > > simplification. The question we need to answer up front is whether
> > > > we're happy to possibly break old OSes (presumably ones no-one
> > > > ought to be using anymore these days, due to their support life
> > > > cycles long having ended).
> > >
> > > The various timer modes were all compatibility, and IIRC, mostly for
> > > Windows XP and older which told time by counting the number of timer
> > > interrupts.
> > >
> > > Paul - you might remember better than me?
> >
> > I think it is only quite recently that Windows has started favouring enlightened time sources rather
> than counting ticks but an admin may still turn all the viridian enlightenments off so just dropping
> ticks will probably still cause time to drift backwards.
> 
> Even when not using the viridian enlightenments, Windows should rely
> on emulated time counters (or the TSC) rather than counting ticks?

Microsoft implementations... sensible... two different things.

> 
> I guess I could give it a try with one of the emulated Windows versions
> that we test on osstest.
> 

Pick an old-ish version. I think osstest has copy of Windows 7.

Cheers,

  Paul


> Thanks, Roger.

Re: vPT rework (and timer mode)

[Roger Pau Monné]

On Mon, Jul 06, 2020 at 09:58:53AM +0100, Paul Durrant wrote:
> > -----Original Message-----
> > From: Roger Pau Monné <roger.pau@citrix.com>
> > Sent: 06 July 2020 09:32
> > To: paul@xen.org
> > Cc: 'Andrew Cooper' <andrew.cooper3@citrix.com>; 'Jan Beulich' <jbeulich@suse.com>; xen-
> > devel@lists.xenproject.org; 'Wei Liu' <wl@xen.org>
> > Subject: Re: vPT rework (and timer mode)
> > 
> > On Mon, Jul 06, 2020 at 08:03:50AM +0100, Paul Durrant wrote:
> > > > -----Original Message-----
> > > > From: Andrew Cooper <andrew.cooper3@citrix.com>
> > > > Sent: 03 July 2020 16:03
> > > > To: Jan Beulich <jbeulich@suse.com>; Roger Pau Monné <roger.pau@citrix.com>
> > > > Cc: xen-devel@lists.xenproject.org; Wei Liu <wl@xen.org>; Paul Durrant <paul@xen.org>
> > > > Subject: Re: vPT rework (and timer mode)
> > > >
> > > > On 03/07/2020 15:50, Jan Beulich wrote:
> > > > > On 01.07.2020 11:02, Roger Pau Monné wrote:
> > > > >> It's my understanding that the purpose of pt_update_irq and
> > > > >> pt_intr_post is to attempt to implement the "delay for missed ticks"
> > > > >> mode, where Xen will accumulate timer interrupts if they cannot be
> > > > >> injected. As shown by the patch above, this is all broken when the
> > > > >> timer is added to a vCPU (pt->vcpu) different than the actual target
> > > > >> vCPU where the interrupt gets delivered (note this can also be a list
> > > > >> of vCPUs if routed from the IO-APIC using Fixed mode).
> > > > >>
> > > > >> I'm at lost at how to fix this so that virtual timers work properly
> > > > >> and we also keep the "delay for missed ticks" mode without doing a
> > > > >> massive rework and somehow keeping track of where injected interrupts
> > > > >> originated, which seems an overly complicated solution.
> > > > >>
> > > > >> My proposal hence would be to completely remove the timer_mode, and
> > > > >> just treat virtual timer interrupts as other interrupts, ie: they will
> > > > >> be injected from the callback (pt_timer_fn) and the vCPU(s) would be
> > > > >> kicked. Whether interrupts would get lost (ie: injected when a
> > > > >> previous one is still pending) depends on the contention on the
> > > > >> system. I'm not aware of any current OS that uses timer interrupts as
> > > > >> a way to track time. I think current OSes know the differences between
> > > > >> a timer counter and an event timer, and will use them appropriately.
> > > > > Fundamentally - why not, the more that this promises to be a
> > > > > simplification. The question we need to answer up front is whether
> > > > > we're happy to possibly break old OSes (presumably ones no-one
> > > > > ought to be using anymore these days, due to their support life
> > > > > cycles long having ended).
> > > >
> > > > The various timer modes were all compatibility, and IIRC, mostly for
> > > > Windows XP and older which told time by counting the number of timer
> > > > interrupts.
> > > >
> > > > Paul - you might remember better than me?
> > >
> > > I think it is only quite recently that Windows has started favouring enlightened time sources rather
> > than counting ticks but an admin may still turn all the viridian enlightenments off so just dropping
> > ticks will probably still cause time to drift backwards.
> > 
> > Even when not using the viridian enlightenments, Windows should rely
> > on emulated time counters (or the TSC) rather than counting ticks?
> 
> Microsoft implementations... sensible... two different things.
> 
> > 
> > I guess I could give it a try with one of the emulated Windows versions
> > that we test on osstest.
> > 
> 
> Pick an old-ish version. I think osstest has copy of Windows 7.

Tried on Windows 7 (with viridian disabled) setting
timer_mode="one_missed_tick_pending" and limiting the capacity of the
domain to 1 (1% CPU utilization) in order to start missing ticks, and
the clock does indeed start lagging behind.

When not using one_missed_tick_pending mode and limiting the capacity
to 1 the clock also lags a bit (I guess with 1% CPU utilization
delayed ticks accumulate too much), but the clock doesn't seem to be
skewed that much.

Both modes will catch up at some point, I assume Windows does sync time
periodically with the wallclock, but I don't think we want to resort
to that.

I will draft a plan about how to proceed in order to fix the emulated
timers event delivery while keeping the accumulated ticks mode and
send it to the list, as I would like to fix this.

Roger.

RE: vPT rework (and timer mode)

[Paul Durrant]

> -----Original Message-----
> From: Roger Pau Monné <roger.pau@citrix.com>
> Sent: 21 July 2020 12:53
> To: paul@xen.org
> Cc: 'Andrew Cooper' <andrew.cooper3@citrix.com>; 'Jan Beulich' <jbeulich@suse.com>; xen-
> devel@lists.xenproject.org; 'Wei Liu' <wl@xen.org>; Igor Druzhinin <igor.druzhinin@citrix.com>
> Subject: Re: vPT rework (and timer mode)
> 
> On Mon, Jul 06, 2020 at 09:58:53AM +0100, Paul Durrant wrote:
> > > -----Original Message-----
> > > From: Roger Pau Monné <roger.pau@citrix.com>
> > > Sent: 06 July 2020 09:32
> > > To: paul@xen.org
> > > Cc: 'Andrew Cooper' <andrew.cooper3@citrix.com>; 'Jan Beulich' <jbeulich@suse.com>; xen-
> > > devel@lists.xenproject.org; 'Wei Liu' <wl@xen.org>
> > > Subject: Re: vPT rework (and timer mode)
> > >
> > > On Mon, Jul 06, 2020 at 08:03:50AM +0100, Paul Durrant wrote:
> > > > > -----Original Message-----
> > > > > From: Andrew Cooper <andrew.cooper3@citrix.com>
> > > > > Sent: 03 July 2020 16:03
> > > > > To: Jan Beulich <jbeulich@suse.com>; Roger Pau Monné <roger.pau@citrix.com>
> > > > > Cc: xen-devel@lists.xenproject.org; Wei Liu <wl@xen.org>; Paul Durrant <paul@xen.org>
> > > > > Subject: Re: vPT rework (and timer mode)
> > > > >
> > > > > On 03/07/2020 15:50, Jan Beulich wrote:
> > > > > > On 01.07.2020 11:02, Roger Pau Monné wrote:
> > > > > >> It's my understanding that the purpose of pt_update_irq and
> > > > > >> pt_intr_post is to attempt to implement the "delay for missed ticks"
> > > > > >> mode, where Xen will accumulate timer interrupts if they cannot be
> > > > > >> injected. As shown by the patch above, this is all broken when the
> > > > > >> timer is added to a vCPU (pt->vcpu) different than the actual target
> > > > > >> vCPU where the interrupt gets delivered (note this can also be a list
> > > > > >> of vCPUs if routed from the IO-APIC using Fixed mode).
> > > > > >>
> > > > > >> I'm at lost at how to fix this so that virtual timers work properly
> > > > > >> and we also keep the "delay for missed ticks" mode without doing a
> > > > > >> massive rework and somehow keeping track of where injected interrupts
> > > > > >> originated, which seems an overly complicated solution.
> > > > > >>
> > > > > >> My proposal hence would be to completely remove the timer_mode, and
> > > > > >> just treat virtual timer interrupts as other interrupts, ie: they will
> > > > > >> be injected from the callback (pt_timer_fn) and the vCPU(s) would be
> > > > > >> kicked. Whether interrupts would get lost (ie: injected when a
> > > > > >> previous one is still pending) depends on the contention on the
> > > > > >> system. I'm not aware of any current OS that uses timer interrupts as
> > > > > >> a way to track time. I think current OSes know the differences between
> > > > > >> a timer counter and an event timer, and will use them appropriately.
> > > > > > Fundamentally - why not, the more that this promises to be a
> > > > > > simplification. The question we need to answer up front is whether
> > > > > > we're happy to possibly break old OSes (presumably ones no-one
> > > > > > ought to be using anymore these days, due to their support life
> > > > > > cycles long having ended).
> > > > >
> > > > > The various timer modes were all compatibility, and IIRC, mostly for
> > > > > Windows XP and older which told time by counting the number of timer
> > > > > interrupts.
> > > > >
> > > > > Paul - you might remember better than me?
> > > >
> > > > I think it is only quite recently that Windows has started favouring enlightened time sources
> rather
> > > than counting ticks but an admin may still turn all the viridian enlightenments off so just
> dropping
> > > ticks will probably still cause time to drift backwards.
> > >
> > > Even when not using the viridian enlightenments, Windows should rely
> > > on emulated time counters (or the TSC) rather than counting ticks?
> >
> > Microsoft implementations... sensible... two different things.
> >
> > >
> > > I guess I could give it a try with one of the emulated Windows versions
> > > that we test on osstest.
> > >
> >
> > Pick an old-ish version. I think osstest has copy of Windows 7.
> 
> Tried on Windows 7 (with viridian disabled) setting
> timer_mode="one_missed_tick_pending" and limiting the capacity of the
> domain to 1 (1% CPU utilization) in order to start missing ticks, and
> the clock does indeed start lagging behind.
> 
> When not using one_missed_tick_pending mode and limiting the capacity
> to 1 the clock also lags a bit (I guess with 1% CPU utilization
> delayed ticks accumulate too much), but the clock doesn't seem to be
> skewed that much.
> 
> Both modes will catch up at some point, I assume Windows does sync time
> periodically with the wallclock, but I don't think we want to resort
> to that.
> 

IIRC it normally syncs once an hour or thereabouts. PV drivers will force a re-sync every 10 mins if they are installed.

> I will draft a plan about how to proceed in order to fix the emulated
> timers event delivery while keeping the accumulated ticks mode and
> send it to the list, as I would like to fix this.

Ok.

Cheers,

  Paul

> 
> Roger.