Re: Could you please answer some questions regarding Solaris PVHVM pvclock support.

Re: Could you please answer some questions regarding Solaris PVHVM pvclock support.

[Stefano Stabellini]

CC'ing xen-devel and Roger.

On Wed, 15 Jan 2014, Qin Li wrote:
> Hi Stefano,
> 
> How do you do?
> 
> Currently, Solaris only works as PV-on-HVM guest on Xen, but recently,

Actually now you have a better way of running Solaris on Xen: PVH.

http://wiki.xen.org/wiki/Xen_Overview#PV_in_an_HVM_Container_.28PVH.29_-_New_in_Xen_4.4

Roger already ported FreeBSD to Xen as a PVH guest:

http://marc.info/?l=freebsd-current&m=138971161228874&w=2


> we decide to changes this situation by work out below 2 features for
> Solaris guest OS:
> 
> /* x86: Does this Xen host support the HVM callback vector type? */
> #define XENFEAT_hvm_callback_vector 8
> 
> /* x86: pvclock algorithm is safe to use on HVM */
> #define XENFEAT_hvm_safe_pvclock

FYI both these features are available and used by PVH guests.


> For XENFEAT_hvm_callback_vector, it's straightforward that the original
> apic interrupt handler needs to be registered onto each vCPU's IDT.
> But for XENFEAT_hvm_safe_pvclock, I have some confusions as following:
> . Why the pvclock implementation within guest OS, has to depend on
> XENFEAT_hvm_callback_vector?

Because you need to be able to receive timer interrupts on multiple
vcpus and without XENFEAT_hvm_callback_vector you would only receive
interrupts from the Xen Platform PCI device.


> . For a PV-on-HVM guest OS, the "shared_info->vcpu_info->vcpu_time_info"
> is already visiable. Does guest OS still need any action to ask
> hypervisor to update this piece of memory periodically?

I don't think you need to ask the hypervisor to update vcpu_time_info
periodically, what gave you that idea?

Re: Could you please answer some questions regarding Solaris PVHVM pvclock support.

[Mukesh Rathor]

On Thu, 16 Jan 2014 12:17:59 +0000
Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:

> CC'ing xen-devel and Roger.
> 
> On Wed, 15 Jan 2014, Qin Li wrote:
> > Hi Stefano,
> > 
> > How do you do?
> > 
> > Currently, Solaris only works as PV-on-HVM guest on Xen, but
> > recently,
> 
> Actually now you have a better way of running Solaris on Xen: PVH.
> 
> http://wiki.xen.org/wiki/Xen_Overview#PV_in_an_HVM_Container_.28PVH.29_-_New_in_Xen_4.4
> 
> Roger already ported FreeBSD to Xen as a PVH guest:
> 
> http://marc.info/?l=freebsd-current&m=138971161228874&w=2


Qin,

I will be giving a talk internally on PVH in the next couple weeks
to the PMs and others, so if you are interested, ping me offline.

thanks
Mukesh

Re: Could you please answer some questions regarding Solaris PVHVM pvclock support.

[Qin Li]

[-- Attachment #1.1: Type: text/plain, Size: 2364 bytes --]


? 2014/1/16 20:17, Stefano Stabellini ??:
>> . For a PV-on-HVM guest OS, the "shared_info->vcpu_info->vcpu_time_info"
>> >is already visiable. Does guest OS still need any action to ask
>> >hypervisor to update this piece of memory periodically?
> I don't think you need to ask the hypervisor to update vcpu_time_info
> periodically, what gave you that idea?
Hi Stefano,

Now, I see it's the hypervisor that will update vcpu_time_info, but 
another thing confuse me:
HVM guest has time drift issue because TSC on different vCPU could be 
out-of-sync, especially after domain suspend/resume.
But how pvclock actually fix this issue? Let's see how FreeBSD port 
calculate the system time:

==================
static uint64_t
get_nsec_offset(struct vcpu_time_info *tinfo)
{

     return (scale_delta*(rdtsc() -* tinfo->tsc_timestamp,
         tinfo->tsc_to_system_mul, tinfo->tsc_shift));
}

/**
  * \brief Get the current time, in nanoseconds, since the hypervisor 
booted.
  *
  * \note This function returns the current CPU's idea of this value, unless
  *       it happens to be less than another CPU's previously determined 
value.
  */
static uint64_t
xen_fetch_vcpu_time(void)
{
     struct vcpu_time_info dst;
     struct vcpu_time_info *src;
     uint32_t pre_version;
     uint64_t now;
     volatile uint64_t last;
     struct vcpu_info *vcpu = DPCPU_GET(vcpu_info);

     src = &vcpu->time;

     critical_enter();
     do {
         pre_version = xen_fetch_vcpu_tinfo(&dst, src);
         barrier();
         now = dst.system_time + *get_nsec_offset(&dst);*
         barrier();
     } while (pre_version != src->version);

     /*
      * Enforce a monotonically increasing clock time across all
      * VCPUs.  If our time is too old, use the last time and return.
      * Otherwise, try to update the last time.
      */
     do {
         last = last_time;
         if (last > now) {
             now = last;
             break;
         }
     } while (!atomic_cmpset_64(&last_time, last, now));

     critical_exit();

     return (now);
}
==================================

I guest linux guest will do the same thing, rdtsc() fetch current 
timestamp from current running vCPU, TSC out-of-sync issue is still there.
It seems to me pvclock finally fix the time drift issue just because the 
workaround enforced as above, right?

Thanks,
Michael



[-- Attachment #1.2: Type: text/html, Size: 4304 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Could you please answer some questions regarding Solaris PVHVM pvclock support.

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 2947 bytes --]

On Wed, 26 Feb 2014, Qin Li wrote:
> 
> 于 2014/1/16 20:17, Stefano Stabellini 写道:
> 
> . For a PV-on-HVM guest OS, the "shared_info->vcpu_info->vcpu_time_info"
> > is already visiable. Does guest OS still need any action to ask
> > hypervisor to update this piece of memory periodically?
> 
> I don't think you need to ask the hypervisor to update vcpu_time_info
> periodically, what gave you that idea?
> 
> Hi Stefano,
> 
> Now, I see it's the hypervisor that will update vcpu_time_info, but another thing confuse me:
> HVM guest has time drift issue because TSC on different vCPU could be out-of-sync, especially after domain suspend/resume.
> But how pvclock actually fix this issue? Let's see how FreeBSD port calculate the system time:
> 
> ==================
> static uint64_t
> get_nsec_offset(struct vcpu_time_info *tinfo)
> {
> 
>     return (scale_delta(rdtsc() - tinfo->tsc_timestamp,
>         tinfo->tsc_to_system_mul, tinfo->tsc_shift));
> }
> 
> /**
>  * \brief Get the current time, in nanoseconds, since the hypervisor booted.
>  *
>  * \note This function returns the current CPU's idea of this value, unless
>  *       it happens to be less than another CPU's previously determined value.
>  */
> static uint64_t
> xen_fetch_vcpu_time(void)
> {
>     struct vcpu_time_info dst;
>     struct vcpu_time_info *src;
>     uint32_t pre_version;
>     uint64_t now;
>     volatile uint64_t last;
>     struct vcpu_info *vcpu = DPCPU_GET(vcpu_info);
> 
>     src = &vcpu->time;
> 
>     critical_enter();
>     do {
>         pre_version = xen_fetch_vcpu_tinfo(&dst, src);
>         barrier();
>         now = dst.system_time + get_nsec_offset(&dst);
>         barrier();
>     } while (pre_version != src->version);
> 
>     /*
>      * Enforce a monotonically increasing clock time across all
>      * VCPUs.  If our time is too old, use the last time and return.
>      * Otherwise, try to update the last time.
>      */
>     do {
>         last = last_time;
>         if (last > now) {
>             now = last;
>             break;
>         }
>     } while (!atomic_cmpset_64(&last_time, last, now));
> 
>     critical_exit();
> 
>     return (now);
> }
> ==================================
> 
> I guest linux guest will do the same thing, rdtsc() fetch current timestamp from current running vCPU, TSC out-of-sync issue is still there.
> It seems to me pvclock finally fix the time drift issue just because the workaround enforced as above, right?

First you should know that TSC is not always guaranteed to be
synchronized across multiple processors, especially on older systems.
On "TSC-safe" systems, Xen would export a consistent TSC to guests, by
setting the vtsc offset and scale appropriately.

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Could you please answer some questions regarding Solaris PVHVM pvclock support.

[Qin Li]

于 2014/2/27 21:03, Stefano Stabellini 写道:
>> >I guest linux guest will do the same thing, rdtsc() fetch current timestamp from current running vCPU, TSC out-of-sync issue is still there.
>> >It seems to me pvclock finally fix the time drift issue just because the workaround enforced as above, right?
> First you should know that TSC is not always guaranteed to be
> synchronized across multiple processors, especially on older systems.
> On "TSC-safe" systems, Xen would export a consistent TSC to guests, by
> setting the vtsc offset and scale appropriately.
Stefano,

Is there a easy way to check whether the system is "TSC-safe"? Do you 
mean "X86_FEATURE_TSC_RELIABLE" bit in "boot_cpu_data.x86_capability"?

Thanks,
Michael

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Could you please answer some questions regarding Solaris PVHVM pvclock support.

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 875 bytes --]

On Fri, 28 Feb 2014, Qin Li wrote:
> 于 2014/2/27 21:03, Stefano Stabellini 写道:
> > > >I guest linux guest will do the same thing, rdtsc() fetch current
> > > timestamp from current running vCPU, TSC out-of-sync issue is still there.
> > > >It seems to me pvclock finally fix the time drift issue just because the
> > > workaround enforced as above, right?
> > First you should know that TSC is not always guaranteed to be
> > synchronized across multiple processors, especially on older systems.
> > On "TSC-safe" systems, Xen would export a consistent TSC to guests, by
> > setting the vtsc offset and scale appropriately.
> Stefano,
> 
> Is there a easy way to check whether the system is "TSC-safe"? Do you mean
> "X86_FEATURE_TSC_RELIABLE" bit in "boot_cpu_data.x86_capability"?

Yes, I think that's it.
Maybe you want to read docs/misc/tscmode.txt.

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel