RFC: xen config changes v4

RFC: xen config changes v4

[Luis R. Rodriguez]

OK here's the state of affairs after some further discussion on some v3 patch
RFC changes and issues I've found after trying to build front end drivers
without CONFIG_XEN.

Option                          Selects                 Depends
----------------------------------------------------------------------
XEN
                                PARAVIRT(x86)
                                PARAVIRT_CLOCK(x86)
  XEN_PV(x86)                   XEN_HAVE_PVMMU(x86)
  XEN_PVH(x86)                  XEN_FRONTEND
  XEN_BACKEND                   SWIOTLB_XEN(arm,arm64)  XEN_PV(x86) ||
                                                        XEN_PVH(x86) ||
                                                        XEN_FRONTEND
   XEN_BLKDEV_BACKEND
   XEN_PCIDEV_BACKEND(x86)
   XEN_SCSI_BACKEND
   XEN_NETDEV_BACKEND
  PCI_XEN(x86)                  SWIOTLB_XEN
  XEN_DOM0                      XEN_BACKEND             XEN_PV(x86) ||
                                PCI_XEN(x86)            XEN_PVH(x86)
    XEN_ACPI_HOTPLUG_MEMORY                             XEN_STUB
    XEN_ACPI_HOTPLUG_CPU                                XEN_STUB
    XEN_MCE_LOG(x86)
    XEN_ACPI_PROCESSOR(x86)				ACPI_PROCESSOR
							CPU_FREQ
  XEN_SAVE_RESTORE(x86)
  XEN_DEBUG_FS
  XEN_WDT
  XEN_MAX_DOMAIN_MEMORY(x86)				XEN_HAVE_PVMMU(x86)
  XEN_BALLOON
    XEN_SELFBALLOONING                                  XEN_TMEM(x86)
    XEN_BALLOON_MEMORY_HOTPLUG
    XEN_SCRUB_PAGES
  XENFS                         XEN_PRIVCMD
    XEN_COMPAT_XENFS
  XEN_TMEM(x86)
  XEN_PRIVCMD
  XEN_SYS_HYPERVISOR
  XEN_DEV_EVTCHN
  XEN_GNTDEV
  XEN_GRANT_DEV_ALLOC
  SWIOTLB_XEN
  XEN_STUB(x86_64)                                      BROKEN
  XEN_ACPI_PROCESSOR(x86)
  XEN_HAVE_PVMMU(x86)					XEN_PV(x86)
  XEN_EFI(x64)
  XEN_XENBUS_FRONTEND
XEN_FRONTEND                    XEN			X86_LOCAL_APIC(x86)
                                XEN_XENBUS_FRONTEND	PCI(x86)
  XEN_FBDEV_FRONTEND            INPUT_XEN_KBDDEV_FRONTEND
  XEN_BLKDEV_FRONTEND
  HVC_XEN_FRONTEND              HVC_XEN
  TCG_XEN
  XEN_PCIDEV_FRONTEND(x86)      PCI_XEN(x86)
  XEN_SCSI_FRONTEND
  INPUT_XEN_KBDDEV_FRONTEND
  XEN_NETDEV_FRONTEND

Additionally I will now note some expected code collateral, none of
this is obviously final but I figure I'll give a heads up as what
I have seen so far or what has helped. Perhaps not all is necessary,
but its best to discuss in case anyone sees anything worth barking
over early.

  * Folding XEN_PVHVM under XEN_FRONTEND should enable good
    integration and building of frontend drivers without requiring
    building the whole xen tamale. That has some obvious code changes
    required in place. As a build collateral since XEN_PVH used to
    depend on XEN_PVHVM we'll now obviously have XEN_PVH select
    XEN_FRONTEND.

  * Although technically we never wanted to build XEN_PVHVM without
    XEN_PVH we obviously want to build XEN_FRONTEND without XEN_PV
    and since we are folding XEN_PVHVM under XEN_FRONTEND we want
    to build XEN_FRONTEND without XEN_PV or XEN_PVH. This means
    we need to build some old XEN_PVHVM code without requiring
    a series of things -- none of this is scary reallyt, its just
    putting code into common files and building them when either
    a PV guest is Xen frontend drivers are desired. The split of
    code also tries to aim to compartamentalize XEN_PV code so
    that in the future a swift git rm would enable removal of
    XEN_PV code should that be desirable. In order to make the
    building of common code and non-common code easier to read
    I've added two Kconfig options:

config XEN_BUILD_PV_GUEST                                                       
        def_bool y if XEN_PV || XEN_PVH                                         
                                                                                
config XEN_BUILD_HYPERVISOR                                                     
        def_bool y if XEN_PV || XEN_FRONTEND  

Naming is perhaps not the best, I welcome other ideas.

Example of where we'd use this:

arch/x86/xen/xen-head.S:#ifdef CONFIG_XEN_BUILD_PV_GUEST

  * arch/x86/xen/setup.c has code which only needs to be
    built under certain conditions for simple memory set up.
    To help with this I've added to arch/x86/xen/Kconfig:

config XEN_BUILD_SIMPLE_MEMORY_SETUP
	def_bool y if !XEN_FRONTEND

    and on it folded in all the xen_extra_mem, xen_released_pages,
    xen_remap_mfn stuff (xen_memory_setup() and friends). As
    collateral that also means:

--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -1560,8 +1566,10 @@ asmlinkage __visible void __init xen_start_kernel(void)

        if (xen_feature(XENFEAT_auto_translated_physmap))
                x86_init.resources.memory_setup = xen_auto_xlated_memory_setup;
+#ifdef CONFIG_XEN_BUILD_SIMPLE_MEMORY_SETUP
        else
                x86_init.resources.memory_setup = xen_memory_setup;
+#endif
        x86_init.oem.arch_setup = xen_arch_setup;
        x86_init.oem.banner = xen_banner;

There is some setup.c code which is common, so this helps compartamentalize the
non shared code.

  * In terms of shared code we end up with something that looks like this,
    desired naming preferences are welcomed, I can also just fold this into
    proper xen/ directory but was just lazy for my initial build test:

--- a/arch/x86/Kbuild
+++ b/arch/x86/Kbuild
@@ -1,7 +1,21 @@
 obj-$(CONFIG_KVM) += kvm/

+obj-$(CONFIG_XEN_BUILD_PV_GUEST) += xen/
+
 # Xen paravirtualization support
-obj-$(CONFIG_XEN) += xen/
+obj-$(CONFIG_XEN_BUILD_HYPERVISOR) += xen/common.o
+obj-$(CONFIG_XEN_BUILD_HYPERVISOR) += xen/time-common.o
+obj-$(CONFIG_XEN_BUILD_HYPERVISOR) += xen/mmu-common.o
+obj-$(CONFIG_XEN_BUILD_HYPERVISOR) += xen/p2m-common.o
+obj-$(CONFIG_XEN_BUILD_HYPERVISOR) += xen/setup-common.o
+obj-$(CONFIG_XEN_BUILD_HYPERVISOR) += xen/irq.o
+obj-$(CONFIG_XEN_BUILD_HYPERVISOR) += xen/xen-asm.o
+obj-$(CONFIG_XEN_BUILD_HYPERVISOR) += xen/xen-asm_$(BITS).o
+obj-$(CONFIG_XEN_BUILD_HYPERVISOR) += xen/grant-table.o
+obj-$(CONFIG_XEN_FRONTEND) += xen/hvm.o
+obj-$(CONFIG_XEN_FRONTEND) += xen/hvm-time.o
+obj-$(CONFIG_XEN_FRONTEND) += xen/mmu-hvm.o
+obj-$(CONFIG_XEN_FRONTEND) += xen/platform-pci-unplug.o

If any of this looks fuzzy still you could just wait for a clean
patch series to evaluate better.

  Luis

-- 
Luis Rodriguez, SUSE LINUX GmbH
Maxfeldstrasse 5; D-90409 Nuernberg

Re: RFC: xen config changes v4

[Juergen Gross]

On 02/26/2015 02:53 AM, Luis R. Rodriguez wrote:
> OK here's the state of affairs after some further discussion on some v3 patch
> RFC changes and issues I've found after trying to build front end drivers
> without CONFIG_XEN.
>
> Option                          Selects                 Depends
> ----------------------------------------------------------------------
> XEN
>                                  PARAVIRT(x86)
>                                  PARAVIRT_CLOCK(x86)
>    XEN_PV(x86)                   XEN_HAVE_PVMMU(x86)
>    XEN_PVH(x86)                  XEN_FRONTEND
>    XEN_BACKEND                   SWIOTLB_XEN(arm,arm64)  XEN_PV(x86) ||
>                                                          XEN_PVH(x86) ||
>                                                          XEN_FRONTEND
>     XEN_BLKDEV_BACKEND
>     XEN_PCIDEV_BACKEND(x86)
>     XEN_SCSI_BACKEND
>     XEN_NETDEV_BACKEND
>    PCI_XEN(x86)                  SWIOTLB_XEN
>    XEN_DOM0                      XEN_BACKEND             XEN_PV(x86) ||
>                                  PCI_XEN(x86)            XEN_PVH(x86)
>      XEN_ACPI_HOTPLUG_MEMORY                             XEN_STUB
>      XEN_ACPI_HOTPLUG_CPU                                XEN_STUB
>      XEN_MCE_LOG(x86)
>      XEN_ACPI_PROCESSOR(x86)				ACPI_PROCESSOR
> 							CPU_FREQ
>    XEN_SAVE_RESTORE(x86)
>    XEN_DEBUG_FS
>    XEN_WDT
>    XEN_MAX_DOMAIN_MEMORY(x86)				XEN_HAVE_PVMMU(x86)
>    XEN_BALLOON
>      XEN_SELFBALLOONING                                  XEN_TMEM(x86)
>      XEN_BALLOON_MEMORY_HOTPLUG
>      XEN_SCRUB_PAGES
>    XENFS                         XEN_PRIVCMD
>      XEN_COMPAT_XENFS
>    XEN_TMEM(x86)
>    XEN_PRIVCMD
>    XEN_SYS_HYPERVISOR
>    XEN_DEV_EVTCHN
>    XEN_GNTDEV
>    XEN_GRANT_DEV_ALLOC
>    SWIOTLB_XEN
>    XEN_STUB(x86_64)                                      BROKEN
>    XEN_ACPI_PROCESSOR(x86)
>    XEN_HAVE_PVMMU(x86)					XEN_PV(x86)
>    XEN_EFI(x64)
>    XEN_XENBUS_FRONTEND
> XEN_FRONTEND                    XEN			X86_LOCAL_APIC(x86)
>                                  XEN_XENBUS_FRONTEND	PCI(x86)
>    XEN_FBDEV_FRONTEND            INPUT_XEN_KBDDEV_FRONTEND
>    XEN_BLKDEV_FRONTEND
>    HVC_XEN_FRONTEND              HVC_XEN
>    TCG_XEN
>    XEN_PCIDEV_FRONTEND(x86)      PCI_XEN(x86)
>    XEN_SCSI_FRONTEND
>    INPUT_XEN_KBDDEV_FRONTEND
>    XEN_NETDEV_FRONTEND

So we are again in the situation that pv-drivers always imply the pvops
kernel (PARAVIRT selected). I started the whole Kconfig rework to
eliminate this dependency.

Can't we move selection of PARAVIRT[-CLOCK] to XEN_PV[H]? This could
be done in a separate series after this one together with the effort
to be able to build the frontend drivers without PARAVIRT, if this
requires some large code rework.

>
> Additionally I will now note some expected code collateral, none of
> this is obviously final but I figure I'll give a heads up as what
> I have seen so far or what has helped. Perhaps not all is necessary,
> but its best to discuss in case anyone sees anything worth barking
> over early.
>
>    * Folding XEN_PVHVM under XEN_FRONTEND should enable good
>      integration and building of frontend drivers without requiring
>      building the whole xen tamale. That has some obvious code changes
>      required in place. As a build collateral since XEN_PVH used to
>      depend on XEN_PVHVM we'll now obviously have XEN_PVH select
>      XEN_FRONTEND.
>
>    * Although technically we never wanted to build XEN_PVHVM without
>      XEN_PVH we obviously want to build XEN_FRONTEND without XEN_PV
>      and since we are folding XEN_PVHVM under XEN_FRONTEND we want
>      to build XEN_FRONTEND without XEN_PV or XEN_PVH. This means
>      we need to build some old XEN_PVHVM code without requiring
>      a series of things -- none of this is scary reallyt, its just
>      putting code into common files and building them when either
>      a PV guest is Xen frontend drivers are desired. The split of
>      code also tries to aim to compartamentalize XEN_PV code so
>      that in the future a swift git rm would enable removal of
>      XEN_PV code should that be desirable. In order to make the
>      building of common code and non-common code easier to read
>      I've added two Kconfig options:
>
> config XEN_BUILD_PV_GUEST
>          def_bool y if XEN_PV || XEN_PVH
>
> config XEN_BUILD_HYPERVISOR

XEN_BUILD_HYPERVISOR_SUPPORT?

>          def_bool y if XEN_PV || XEN_FRONTEND
>
> Naming is perhaps not the best, I welcome other ideas.
>
> Example of where we'd use this:
>
> arch/x86/xen/xen-head.S:#ifdef CONFIG_XEN_BUILD_PV_GUEST
>
>    * arch/x86/xen/setup.c has code which only needs to be
>      built under certain conditions for simple memory set up.
>      To help with this I've added to arch/x86/xen/Kconfig:
>
> config XEN_BUILD_SIMPLE_MEMORY_SETUP
> 	def_bool y if !XEN_FRONTEND

This naming is strange. The memory setup via xen_memory_setup() is
surely more complex than without it. XEN_BUILD_PV_MEMORY_SETUP
perhaps?


Juergen

>      and on it folded in all the xen_extra_mem, xen_released_pages,
>      xen_remap_mfn stuff (xen_memory_setup() and friends). As
>      collateral that also means:
>
> --- a/arch/x86/xen/enlighten.c
> +++ b/arch/x86/xen/enlighten.c
> @@ -1560,8 +1566,10 @@ asmlinkage __visible void __init xen_start_kernel(void)
>
>          if (xen_feature(XENFEAT_auto_translated_physmap))
>                  x86_init.resources.memory_setup = xen_auto_xlated_memory_setup;
> +#ifdef CONFIG_XEN_BUILD_SIMPLE_MEMORY_SETUP
>          else
>                  x86_init.resources.memory_setup = xen_memory_setup;
> +#endif
>          x86_init.oem.arch_setup = xen_arch_setup;
>          x86_init.oem.banner = xen_banner;
>
> There is some setup.c code which is common, so this helps compartamentalize the
> non shared code.
>
>    * In terms of shared code we end up with something that looks like this,
>      desired naming preferences are welcomed, I can also just fold this into
>      proper xen/ directory but was just lazy for my initial build test:
>
> --- a/arch/x86/Kbuild
> +++ b/arch/x86/Kbuild
> @@ -1,7 +1,21 @@
>   obj-$(CONFIG_KVM) += kvm/
>
> +obj-$(CONFIG_XEN_BUILD_PV_GUEST) += xen/
> +
>   # Xen paravirtualization support
> -obj-$(CONFIG_XEN) += xen/
> +obj-$(CONFIG_XEN_BUILD_HYPERVISOR) += xen/common.o
> +obj-$(CONFIG_XEN_BUILD_HYPERVISOR) += xen/time-common.o
> +obj-$(CONFIG_XEN_BUILD_HYPERVISOR) += xen/mmu-common.o
> +obj-$(CONFIG_XEN_BUILD_HYPERVISOR) += xen/p2m-common.o
> +obj-$(CONFIG_XEN_BUILD_HYPERVISOR) += xen/setup-common.o
> +obj-$(CONFIG_XEN_BUILD_HYPERVISOR) += xen/irq.o
> +obj-$(CONFIG_XEN_BUILD_HYPERVISOR) += xen/xen-asm.o
> +obj-$(CONFIG_XEN_BUILD_HYPERVISOR) += xen/xen-asm_$(BITS).o
> +obj-$(CONFIG_XEN_BUILD_HYPERVISOR) += xen/grant-table.o
> +obj-$(CONFIG_XEN_FRONTEND) += xen/hvm.o
> +obj-$(CONFIG_XEN_FRONTEND) += xen/hvm-time.o
> +obj-$(CONFIG_XEN_FRONTEND) += xen/mmu-hvm.o
> +obj-$(CONFIG_XEN_FRONTEND) += xen/platform-pci-unplug.o
>
> If any of this looks fuzzy still you could just wait for a clean
> patch series to evaluate better.
>
>    Luis
>

Re: RFC: xen config changes v4

[David Vrabel]

On 26/02/15 04:59, Juergen Gross wrote:
> 
> So we are again in the situation that pv-drivers always imply the pvops
> kernel (PARAVIRT selected). I started the whole Kconfig rework to
> eliminate this dependency.

Yes.  Can you produce a series that just addresses this one issue.

In the absence of any concrete requirement for this big Kconfig reorg I
I don't think it is helpful.

David

Re: RFC: xen config changes v4

[Stefano Stabellini]

On Thu, 26 Feb 2015, David Vrabel wrote:
> On 26/02/15 04:59, Juergen Gross wrote:
> > 
> > So we are again in the situation that pv-drivers always imply the pvops
> > kernel (PARAVIRT selected). I started the whole Kconfig rework to
> > eliminate this dependency.
> 
> Yes.  Can you produce a series that just addresses this one issue.
> 
> In the absence of any concrete requirement for this big Kconfig reorg I
> I don't think it is helpful.

I clearly missed some context as I didn't realize that this was the
intended goal. Why do we want this? Please explain as it won't come
for free.


We have a few PV interfaces for HVM guests that need PARAVIRT in Linux
in order to be used, for example pv_time_ops and HVMOP_pagetable_dying.
They are critical performance improvements and from the interface
perspective, small enough that doesn't make much sense having a separate
KConfig option for them.


In order to reach the goal above we necessarily need to introduce a
differentiation in terms of PV on HVM guests in Linux:

1) basic guests with PV network, disk, etc but no PV timers, no
   HVMOP_pagetable_dying, no PV IPIs
2) full PV on HVM guests that have PV network, disk, timers,
   HVMOP_pagetable_dying, PV IPIs and anything else that makes sense.

2) is much faster than 1) on Xen and 2) is only a tiny bit slower than
1) on native x86


>From Xen perspective and from code maintenance perspective I don't think
it makes sense to have the separation, actually it would make things
slower and harder to maintain.

Re: RFC: xen config changes v4

[Luis R. Rodriguez]

On Thu, Feb 26, 2015 at 11:08:20AM +0000, Stefano Stabellini wrote:
> On Thu, 26 Feb 2015, David Vrabel wrote:
> > On 26/02/15 04:59, Juergen Gross wrote:
> > > 
> > > So we are again in the situation that pv-drivers always imply the pvops
> > > kernel (PARAVIRT selected). I started the whole Kconfig rework to
> > > eliminate this dependency.
> > 
> > Yes.  Can you produce a series that just addresses this one issue.
> > 
> > In the absence of any concrete requirement for this big Kconfig reorg I
> > I don't think it is helpful.
> 
> I clearly missed some context as I didn't realize that this was the
> intended goal. Why do we want this? Please explain as it won't come
> for free.
> 
> 
> We have a few PV interfaces for HVM guests that need PARAVIRT in Linux
> in order to be used, for example pv_time_ops and HVMOP_pagetable_dying.
> They are critical performance improvements and from the interface
> perspective, small enough that doesn't make much sense having a separate
> KConfig option for them.
> 
> 
> In order to reach the goal above we necessarily need to introduce a
> differentiation in terms of PV on HVM guests in Linux:
> 
> 1) basic guests with PV network, disk, etc but no PV timers, no
>    HVMOP_pagetable_dying, no PV IPIs
> 2) full PV on HVM guests that have PV network, disk, timers,
>    HVMOP_pagetable_dying, PV IPIs and anything else that makes sense.
> 
> 2) is much faster than 1) on Xen and 2) is only a tiny bit slower than
> 1) on native x86

Also don't we shove 2) down hvm guests right now? Even when everything is
built in I do not see how we opt out for HVM for 1) at run time right now.
If this is true then the question of motivation for this becomes even
stronger I think.

  Luis

Re: RFC: xen config changes v4

[Stefano Stabellini]

On Thu, 26 Feb 2015, Luis R. Rodriguez wrote:
> On Thu, Feb 26, 2015 at 11:08:20AM +0000, Stefano Stabellini wrote:
> > On Thu, 26 Feb 2015, David Vrabel wrote:
> > > On 26/02/15 04:59, Juergen Gross wrote:
> > > > 
> > > > So we are again in the situation that pv-drivers always imply the pvops
> > > > kernel (PARAVIRT selected). I started the whole Kconfig rework to
> > > > eliminate this dependency.
> > > 
> > > Yes.  Can you produce a series that just addresses this one issue.
> > > 
> > > In the absence of any concrete requirement for this big Kconfig reorg I
> > > I don't think it is helpful.
> > 
> > I clearly missed some context as I didn't realize that this was the
> > intended goal. Why do we want this? Please explain as it won't come
> > for free.
> > 
> > 
> > We have a few PV interfaces for HVM guests that need PARAVIRT in Linux
> > in order to be used, for example pv_time_ops and HVMOP_pagetable_dying.
> > They are critical performance improvements and from the interface
> > perspective, small enough that doesn't make much sense having a separate
> > KConfig option for them.
> > 
> > 
> > In order to reach the goal above we necessarily need to introduce a
> > differentiation in terms of PV on HVM guests in Linux:
> > 
> > 1) basic guests with PV network, disk, etc but no PV timers, no
> >    HVMOP_pagetable_dying, no PV IPIs
> > 2) full PV on HVM guests that have PV network, disk, timers,
> >    HVMOP_pagetable_dying, PV IPIs and anything else that makes sense.
> > 
> > 2) is much faster than 1) on Xen and 2) is only a tiny bit slower than
> > 1) on native x86
> 
> Also don't we shove 2) down hvm guests right now? Even when everything is
> built in I do not see how we opt out for HVM for 1) at run time right now.
>
> If this is true then the question of motivation for this becomes even
> stronger I think.

Yes, indeed there is no way to do 1) at the moment. And for good
reasons, see above.

Re: RFC: xen config changes v4

[Luis R. Rodriguez]

On Thu, Feb 26, 2015 at 05:42:57PM +0000, Stefano Stabellini wrote:
> On Thu, 26 Feb 2015, Luis R. Rodriguez wrote:
> > On Thu, Feb 26, 2015 at 11:08:20AM +0000, Stefano Stabellini wrote:
> > > On Thu, 26 Feb 2015, David Vrabel wrote:
> > > > On 26/02/15 04:59, Juergen Gross wrote:
> > > > > 
> > > > > So we are again in the situation that pv-drivers always imply the pvops
> > > > > kernel (PARAVIRT selected). I started the whole Kconfig rework to
> > > > > eliminate this dependency.
> > > > 
> > > > Yes.  Can you produce a series that just addresses this one issue.
> > > > 
> > > > In the absence of any concrete requirement for this big Kconfig reorg I
> > > > I don't think it is helpful.
> > > 
> > > I clearly missed some context as I didn't realize that this was the
> > > intended goal. Why do we want this? Please explain as it won't come
> > > for free.
> > > 
> > > 
> > > We have a few PV interfaces for HVM guests that need PARAVIRT in Linux
> > > in order to be used, for example pv_time_ops and HVMOP_pagetable_dying.
> > > They are critical performance improvements and from the interface
> > > perspective, small enough that doesn't make much sense having a separate
> > > KConfig option for them.
> > > 
> > > 
> > > In order to reach the goal above we necessarily need to introduce a
> > > differentiation in terms of PV on HVM guests in Linux:
> > > 
> > > 1) basic guests with PV network, disk, etc but no PV timers, no
> > >    HVMOP_pagetable_dying, no PV IPIs
> > > 2) full PV on HVM guests that have PV network, disk, timers,
> > >    HVMOP_pagetable_dying, PV IPIs and anything else that makes sense.
> > > 
> > > 2) is much faster than 1) on Xen and 2) is only a tiny bit slower than
> > > 1) on native x86
> > 
> > Also don't we shove 2) down hvm guests right now? Even when everything is
> > built in I do not see how we opt out for HVM for 1) at run time right now.
> >
> > If this is true then the question of motivation for this becomes even
> > stronger I think.
> 
> Yes, indeed there is no way to do 1) at the moment. And for good
> reasons, see above.

OK if the goal is to be able to build front end drivers by avoiding building
PARAVIRT / PARAVIRT_CLOCK and if the gains to be able to do so (which haven't
been stated other than just the ability to do so) are small (as Stefano notes
simple hvm containers do not perform great) but requires a bit of work, I'd
rather ask -- why not address *why* we are avoiding PARAVIRT /
PARAVIRT_CLOCK and stick to the original goals behind the pvops model by
addressing what is required to be able to continue to be happy with one single
kernel. The work required to do that might be more than to just be able to
build simple Xen hvm containers without PARAVIRT / PARAVIRT_CLOCK  but I'd
think the gains would be much higher.

If this resonates well then I'd like to ask: what are the current most pressing
issues with enabling PARAVIRT / PARAVIRT_CLOCK.

  Luis

Re: RFC: xen config changes v4

[Juergen Gross]

On 02/26/2015 06:42 PM, Stefano Stabellini wrote:
> On Thu, 26 Feb 2015, Luis R. Rodriguez wrote:
>> On Thu, Feb 26, 2015 at 11:08:20AM +0000, Stefano Stabellini wrote:
>>> On Thu, 26 Feb 2015, David Vrabel wrote:
>>>> On 26/02/15 04:59, Juergen Gross wrote:
>>>>>
>>>>> So we are again in the situation that pv-drivers always imply the pvops
>>>>> kernel (PARAVIRT selected). I started the whole Kconfig rework to
>>>>> eliminate this dependency.
>>>>
>>>> Yes.  Can you produce a series that just addresses this one issue.
>>>>
>>>> In the absence of any concrete requirement for this big Kconfig reorg I
>>>> I don't think it is helpful.
>>>
>>> I clearly missed some context as I didn't realize that this was the
>>> intended goal. Why do we want this? Please explain as it won't come
>>> for free.
>>>
>>>
>>> We have a few PV interfaces for HVM guests that need PARAVIRT in Linux
>>> in order to be used, for example pv_time_ops and HVMOP_pagetable_dying.
>>> They are critical performance improvements and from the interface
>>> perspective, small enough that doesn't make much sense having a separate
>>> KConfig option for them.
>>>
>>>
>>> In order to reach the goal above we necessarily need to introduce a
>>> differentiation in terms of PV on HVM guests in Linux:
>>>
>>> 1) basic guests with PV network, disk, etc but no PV timers, no
>>>     HVMOP_pagetable_dying, no PV IPIs
>>> 2) full PV on HVM guests that have PV network, disk, timers,
>>>     HVMOP_pagetable_dying, PV IPIs and anything else that makes sense.
>>>
>>> 2) is much faster than 1) on Xen and 2) is only a tiny bit slower than
>>> 1) on native x86
>>
>> Also don't we shove 2) down hvm guests right now? Even when everything is
>> built in I do not see how we opt out for HVM for 1) at run time right now.
>>
>> If this is true then the question of motivation for this becomes even
>> stronger I think.
>
> Yes, indeed there is no way to do 1) at the moment. And for good
> reasons, see above.

Hmm, after checking the code I'm not convinced:

- HVMOP_pagetable_dying is obsolete on modern hardware supporting
   EPT/HAP

- PV IPIs are not needed on single-vcpu guests

- PARAVIRT_CLOCK doesn't need PARAVIRT (in fact the SUSEs kernel configs
   for all x86_64 kernels have CONFIG_PARAVIRT_CLOCK=y)

So I think we really should enable building Xen frontends without
PARAVIRT, implying at least no XEN_PV and no XEN_PVH.

I'll have a try setting up patches.


Juergen

Re: RFC: xen config changes v4

[Juergen Gross]

On 02/26/2015 07:48 PM, Luis R. Rodriguez wrote:
> On Thu, Feb 26, 2015 at 05:42:57PM +0000, Stefano Stabellini wrote:
>> On Thu, 26 Feb 2015, Luis R. Rodriguez wrote:
>>> On Thu, Feb 26, 2015 at 11:08:20AM +0000, Stefano Stabellini wrote:
>>>> On Thu, 26 Feb 2015, David Vrabel wrote:
>>>>> On 26/02/15 04:59, Juergen Gross wrote:
>>>>>>
>>>>>> So we are again in the situation that pv-drivers always imply the pvops
>>>>>> kernel (PARAVIRT selected). I started the whole Kconfig rework to
>>>>>> eliminate this dependency.
>>>>>
>>>>> Yes.  Can you produce a series that just addresses this one issue.
>>>>>
>>>>> In the absence of any concrete requirement for this big Kconfig reorg I
>>>>> I don't think it is helpful.
>>>>
>>>> I clearly missed some context as I didn't realize that this was the
>>>> intended goal. Why do we want this? Please explain as it won't come
>>>> for free.
>>>>
>>>>
>>>> We have a few PV interfaces for HVM guests that need PARAVIRT in Linux
>>>> in order to be used, for example pv_time_ops and HVMOP_pagetable_dying.
>>>> They are critical performance improvements and from the interface
>>>> perspective, small enough that doesn't make much sense having a separate
>>>> KConfig option for them.
>>>>
>>>>
>>>> In order to reach the goal above we necessarily need to introduce a
>>>> differentiation in terms of PV on HVM guests in Linux:
>>>>
>>>> 1) basic guests with PV network, disk, etc but no PV timers, no
>>>>     HVMOP_pagetable_dying, no PV IPIs
>>>> 2) full PV on HVM guests that have PV network, disk, timers,
>>>>     HVMOP_pagetable_dying, PV IPIs and anything else that makes sense.
>>>>
>>>> 2) is much faster than 1) on Xen and 2) is only a tiny bit slower than
>>>> 1) on native x86
>>>
>>> Also don't we shove 2) down hvm guests right now? Even when everything is
>>> built in I do not see how we opt out for HVM for 1) at run time right now.
>>>
>>> If this is true then the question of motivation for this becomes even
>>> stronger I think.
>>
>> Yes, indeed there is no way to do 1) at the moment. And for good
>> reasons, see above.
>
> OK if the goal is to be able to build front end drivers by avoiding building
> PARAVIRT / PARAVIRT_CLOCK and if the gains to be able to do so (which haven't
> been stated other than just the ability to do so) are small (as Stefano notes
> simple hvm containers do not perform great) but requires a bit of work, I'd
> rather ask -- why not address *why* we are avoiding PARAVIRT /
> PARAVIRT_CLOCK and stick to the original goals behind the pvops model by
> addressing what is required to be able to continue to be happy with one single
> kernel. The work required to do that might be more than to just be able to
> build simple Xen hvm containers without PARAVIRT / PARAVIRT_CLOCK  but I'd
> think the gains would be much higher.

I absolutely agree. I think this is a long term goal we should work on.
PVH should address most of the issues, BTW.

> If this resonates well then I'd like to ask: what are the current most pressing
> issues with enabling PARAVIRT / PARAVIRT_CLOCK.

PARAVIRT: performance, especially memory management

PARAVIRT_CLOCK: none


Juergen

Re: RFC: xen config changes v4

[Stefano Stabellini]

On Fri, 27 Feb 2015, Juergen Gross wrote:
> On 02/26/2015 06:42 PM, Stefano Stabellini wrote:
> > On Thu, 26 Feb 2015, Luis R. Rodriguez wrote:
> > > On Thu, Feb 26, 2015 at 11:08:20AM +0000, Stefano Stabellini wrote:
> > > > On Thu, 26 Feb 2015, David Vrabel wrote:
> > > > > On 26/02/15 04:59, Juergen Gross wrote:
> > > > > > 
> > > > > > So we are again in the situation that pv-drivers always imply the
> > > > > > pvops
> > > > > > kernel (PARAVIRT selected). I started the whole Kconfig rework to
> > > > > > eliminate this dependency.
> > > > > 
> > > > > Yes.  Can you produce a series that just addresses this one issue.
> > > > > 
> > > > > In the absence of any concrete requirement for this big Kconfig reorg
> > > > > I
> > > > > I don't think it is helpful.
> > > > 
> > > > I clearly missed some context as I didn't realize that this was the
> > > > intended goal. Why do we want this? Please explain as it won't come
> > > > for free.
> > > > 
> > > > 
> > > > We have a few PV interfaces for HVM guests that need PARAVIRT in Linux
> > > > in order to be used, for example pv_time_ops and HVMOP_pagetable_dying.
> > > > They are critical performance improvements and from the interface
> > > > perspective, small enough that doesn't make much sense having a separate
> > > > KConfig option for them.
> > > > 
> > > > 
> > > > In order to reach the goal above we necessarily need to introduce a
> > > > differentiation in terms of PV on HVM guests in Linux:
> > > > 
> > > > 1) basic guests with PV network, disk, etc but no PV timers, no
> > > >     HVMOP_pagetable_dying, no PV IPIs
> > > > 2) full PV on HVM guests that have PV network, disk, timers,
> > > >     HVMOP_pagetable_dying, PV IPIs and anything else that makes sense.
> > > > 
> > > > 2) is much faster than 1) on Xen and 2) is only a tiny bit slower than
> > > > 1) on native x86
> > > 
> > > Also don't we shove 2) down hvm guests right now? Even when everything is
> > > built in I do not see how we opt out for HVM for 1) at run time right now.
> > > 
> > > If this is true then the question of motivation for this becomes even
> > > stronger I think.
> > 
> > Yes, indeed there is no way to do 1) at the moment. And for good
> > reasons, see above.
> 
> Hmm, after checking the code I'm not convinced:
> 
> - HVMOP_pagetable_dying is obsolete on modern hardware supporting
>   EPT/HAP

That might be true, but what about older hardware?
Even on modern hardware a few workloads still run faster on shadow.
But if HVMOP_pagetable_dying is the only reason to keep PARAVIRT for HVM
guests, then I agree with you that we should remove it.


> - PV IPIs are not needed on single-vcpu guests
>
> - PARAVIRT_CLOCK doesn't need PARAVIRT (in fact the SUSEs kernel configs
>   for all x86_64 kernels have CONFIG_PARAVIRT_CLOCK=y)
> 
> So I think we really should enable building Xen frontends without
> PARAVIRT, implying at least no XEN_PV and no XEN_PVH.
> 
> I'll have a try setting up patches.
 
If we are doing this as a performance improvement, I would like to see a
couple of benchmarks (kernbench, hackbench) to show that on a
single-vcpu guest and multi-vcpu guest (let's say 4 vcpus) disabling
PARAVIRT leads to better performance on Xen on EPT hardware.

Re: RFC: xen config changes v4

[Juergen Gross]

On 02/27/2015 10:41 AM, Stefano Stabellini wrote:
> On Fri, 27 Feb 2015, Juergen Gross wrote:
>> On 02/26/2015 06:42 PM, Stefano Stabellini wrote:
>>> On Thu, 26 Feb 2015, Luis R. Rodriguez wrote:
>>>> On Thu, Feb 26, 2015 at 11:08:20AM +0000, Stefano Stabellini wrote:
>>>>> On Thu, 26 Feb 2015, David Vrabel wrote:
>>>>>> On 26/02/15 04:59, Juergen Gross wrote:
>>>>>>>
>>>>>>> So we are again in the situation that pv-drivers always imply the
>>>>>>> pvops
>>>>>>> kernel (PARAVIRT selected). I started the whole Kconfig rework to
>>>>>>> eliminate this dependency.
>>>>>>
>>>>>> Yes.  Can you produce a series that just addresses this one issue.
>>>>>>
>>>>>> In the absence of any concrete requirement for this big Kconfig reorg
>>>>>> I
>>>>>> I don't think it is helpful.
>>>>>
>>>>> I clearly missed some context as I didn't realize that this was the
>>>>> intended goal. Why do we want this? Please explain as it won't come
>>>>> for free.
>>>>>
>>>>>
>>>>> We have a few PV interfaces for HVM guests that need PARAVIRT in Linux
>>>>> in order to be used, for example pv_time_ops and HVMOP_pagetable_dying.
>>>>> They are critical performance improvements and from the interface
>>>>> perspective, small enough that doesn't make much sense having a separate
>>>>> KConfig option for them.
>>>>>
>>>>>
>>>>> In order to reach the goal above we necessarily need to introduce a
>>>>> differentiation in terms of PV on HVM guests in Linux:
>>>>>
>>>>> 1) basic guests with PV network, disk, etc but no PV timers, no
>>>>>      HVMOP_pagetable_dying, no PV IPIs
>>>>> 2) full PV on HVM guests that have PV network, disk, timers,
>>>>>      HVMOP_pagetable_dying, PV IPIs and anything else that makes sense.
>>>>>
>>>>> 2) is much faster than 1) on Xen and 2) is only a tiny bit slower than
>>>>> 1) on native x86
>>>>
>>>> Also don't we shove 2) down hvm guests right now? Even when everything is
>>>> built in I do not see how we opt out for HVM for 1) at run time right now.
>>>>
>>>> If this is true then the question of motivation for this becomes even
>>>> stronger I think.
>>>
>>> Yes, indeed there is no way to do 1) at the moment. And for good
>>> reasons, see above.
>>
>> Hmm, after checking the code I'm not convinced:
>>
>> - HVMOP_pagetable_dying is obsolete on modern hardware supporting
>>    EPT/HAP
>
> That might be true, but what about older hardware?
> Even on modern hardware a few workloads still run faster on shadow.
> But if HVMOP_pagetable_dying is the only reason to keep PARAVIRT for HVM
> guests, then I agree with you that we should remove it.
>
>
>> - PV IPIs are not needed on single-vcpu guests
>>
>> - PARAVIRT_CLOCK doesn't need PARAVIRT (in fact the SUSEs kernel configs
>>    for all x86_64 kernels have CONFIG_PARAVIRT_CLOCK=y)
>>
>> So I think we really should enable building Xen frontends without
>> PARAVIRT, implying at least no XEN_PV and no XEN_PVH.
>>
>> I'll have a try setting up patches.
>
> If we are doing this as a performance improvement, I would like to see a
> couple of benchmarks (kernbench, hackbench) to show that on a
> single-vcpu guest and multi-vcpu guest (let's say 4 vcpus) disabling
> PARAVIRT leads to better performance on Xen on EPT hardware.

This is not meant to be a performance improvement. It is meant to enable
a standard distro kernel configured without PARAVIRT to be able to run
as a HVM guest using the pv-drivers.

Juergen

Re: RFC: xen config changes v4

[Ian Campbell]

On Thu, 2015-02-26 at 19:48 +0100, Luis R. Rodriguez wrote:
> On Thu, Feb 26, 2015 at 05:42:57PM +0000, Stefano Stabellini wrote:
> > On Thu, 26 Feb 2015, Luis R. Rodriguez wrote:
> > > On Thu, Feb 26, 2015 at 11:08:20AM +0000, Stefano Stabellini wrote:
> > > > On Thu, 26 Feb 2015, David Vrabel wrote:
> > > > > On 26/02/15 04:59, Juergen Gross wrote:
> > > > > > 
> > > > > > So we are again in the situation that pv-drivers always imply the pvops
> > > > > > kernel (PARAVIRT selected). I started the whole Kconfig rework to
> > > > > > eliminate this dependency.
> > > > > 
> > > > > Yes.  Can you produce a series that just addresses this one issue.
> > > > > 
> > > > > In the absence of any concrete requirement for this big Kconfig reorg I
> > > > > I don't think it is helpful.
> > > > 
> > > > I clearly missed some context as I didn't realize that this was the
> > > > intended goal. Why do we want this? Please explain as it won't come
> > > > for free.
> > > > 
> > > > 
> > > > We have a few PV interfaces for HVM guests that need PARAVIRT in Linux
> > > > in order to be used, for example pv_time_ops and HVMOP_pagetable_dying.
> > > > They are critical performance improvements and from the interface
> > > > perspective, small enough that doesn't make much sense having a separate
> > > > KConfig option for them.
> > > > 
> > > > 
> > > > In order to reach the goal above we necessarily need to introduce a
> > > > differentiation in terms of PV on HVM guests in Linux:
> > > > 
> > > > 1) basic guests with PV network, disk, etc but no PV timers, no
> > > >    HVMOP_pagetable_dying, no PV IPIs
> > > > 2) full PV on HVM guests that have PV network, disk, timers,
> > > >    HVMOP_pagetable_dying, PV IPIs and anything else that makes sense.
> > > > 
> > > > 2) is much faster than 1) on Xen and 2) is only a tiny bit slower than
> > > > 1) on native x86
> > > 
> > > Also don't we shove 2) down hvm guests right now? Even when everything is
> > > built in I do not see how we opt out for HVM for 1) at run time right now.
> > >
> > > If this is true then the question of motivation for this becomes even
> > > stronger I think.
> > 
> > Yes, indeed there is no way to do 1) at the moment. And for good
> > reasons, see above.
> 
> OK if the goal is to be able to build front end drivers by avoiding building
> PARAVIRT / PARAVIRT_CLOCK and if the gains to be able to do so (which haven't
> been stated other than just the ability to do so) are small (as Stefano notes
> simple hvm containers do not perform great)

I may have misunderstood this bit, WRT this last parenthetical: adding
PV I/O drivers to an HVM guest is AFAIAA the single biggest improvement
you can make to a bare HVM guest in terms of performance.

There are indeed additional gains to be had from other PV stuff which
Stefano mentions (clocks etc), but I believe those are all mostly
incremental and not as impressive as the PV I/O gains (but still good
improvements).

That's not to say that there's an argument in the context of Linux that
if you can enable PV I/O then you can also enable other PV 
optimisations, but I thought I would mention it.

Wasn't part of the original point here to be able to enable PV I/O (and
perhaps other PV stuff) for non-PAE 32-bit x86, i.e. in a context where
PVMMU isn't available. (That doesn't necessarily conflict with "if you
can enable PV I/O then you can also enable other PV 
optimisations" though)

Ian.

Re: RFC: xen config changes v4

[Stefano Stabellini]

On Fri, 27 Feb 2015, Juergen Gross wrote:
> On 02/27/2015 10:41 AM, Stefano Stabellini wrote:
> > On Fri, 27 Feb 2015, Juergen Gross wrote:
> > > On 02/26/2015 06:42 PM, Stefano Stabellini wrote:
> > > > On Thu, 26 Feb 2015, Luis R. Rodriguez wrote:
> > > > > On Thu, Feb 26, 2015 at 11:08:20AM +0000, Stefano Stabellini wrote:
> > > > > > On Thu, 26 Feb 2015, David Vrabel wrote:
> > > > > > > On 26/02/15 04:59, Juergen Gross wrote:
> > > > > > > > 
> > > > > > > > So we are again in the situation that pv-drivers always imply
> > > > > > > > the
> > > > > > > > pvops
> > > > > > > > kernel (PARAVIRT selected). I started the whole Kconfig rework
> > > > > > > > to
> > > > > > > > eliminate this dependency.
> > > > > > > 
> > > > > > > Yes.  Can you produce a series that just addresses this one issue.
> > > > > > > 
> > > > > > > In the absence of any concrete requirement for this big Kconfig
> > > > > > > reorg
> > > > > > > I
> > > > > > > I don't think it is helpful.
> > > > > > 
> > > > > > I clearly missed some context as I didn't realize that this was the
> > > > > > intended goal. Why do we want this? Please explain as it won't come
> > > > > > for free.
> > > > > > 
> > > > > > 
> > > > > > We have a few PV interfaces for HVM guests that need PARAVIRT in
> > > > > > Linux
> > > > > > in order to be used, for example pv_time_ops and
> > > > > > HVMOP_pagetable_dying.
> > > > > > They are critical performance improvements and from the interface
> > > > > > perspective, small enough that doesn't make much sense having a
> > > > > > separate
> > > > > > KConfig option for them.
> > > > > > 
> > > > > > 
> > > > > > In order to reach the goal above we necessarily need to introduce a
> > > > > > differentiation in terms of PV on HVM guests in Linux:
> > > > > > 
> > > > > > 1) basic guests with PV network, disk, etc but no PV timers, no
> > > > > >      HVMOP_pagetable_dying, no PV IPIs
> > > > > > 2) full PV on HVM guests that have PV network, disk, timers,
> > > > > >      HVMOP_pagetable_dying, PV IPIs and anything else that makes
> > > > > > sense.
> > > > > > 
> > > > > > 2) is much faster than 1) on Xen and 2) is only a tiny bit slower
> > > > > > than
> > > > > > 1) on native x86
> > > > > 
> > > > > Also don't we shove 2) down hvm guests right now? Even when everything
> > > > > is
> > > > > built in I do not see how we opt out for HVM for 1) at run time right
> > > > > now.
> > > > > 
> > > > > If this is true then the question of motivation for this becomes even
> > > > > stronger I think.
> > > > 
> > > > Yes, indeed there is no way to do 1) at the moment. And for good
> > > > reasons, see above.
> > > 
> > > Hmm, after checking the code I'm not convinced:
> > > 
> > > - HVMOP_pagetable_dying is obsolete on modern hardware supporting
> > >    EPT/HAP
> > 
> > That might be true, but what about older hardware?
> > Even on modern hardware a few workloads still run faster on shadow.
> > But if HVMOP_pagetable_dying is the only reason to keep PARAVIRT for HVM
> > guests, then I agree with you that we should remove it.
> > 
> > 
> > > - PV IPIs are not needed on single-vcpu guests
> > > 
> > > - PARAVIRT_CLOCK doesn't need PARAVIRT (in fact the SUSEs kernel configs
> > >    for all x86_64 kernels have CONFIG_PARAVIRT_CLOCK=y)
> > > 
> > > So I think we really should enable building Xen frontends without
> > > PARAVIRT, implying at least no XEN_PV and no XEN_PVH.
> > > 
> > > I'll have a try setting up patches.
> > 
> > If we are doing this as a performance improvement, I would like to see a
> > couple of benchmarks (kernbench, hackbench) to show that on a
> > single-vcpu guest and multi-vcpu guest (let's say 4 vcpus) disabling
> > PARAVIRT leads to better performance on Xen on EPT hardware.
> 
> This is not meant to be a performance improvement. It is meant to enable
> a standard distro kernel configured without PARAVIRT to be able to run
> as a HVM guest using the pv-drivers.
 
This is not a convincing explanation.  Debian, Ubuntu and Fedora seems
to be able to cope with it just fine.

Why do you want to do that, even though it will cause a performance
regression and a maintenance pain?  You haven't provided a reason yet.

Re: RFC: xen config changes v4

[Ian Campbell]

On Fri, 2015-02-27 at 10:11 +0000, Stefano Stabellini wrote:

(for some reason I initially thought this was in reply to my mail, so
it's written in a way which assumes that, so sprinkle IMHO around the
place and/or take it as a follow on to my previous mail in this thread,
I guess)

> This is not a convincing explanation.  Debian, Ubuntu and Fedora seems
> to be able to cope with it just fine.

Debian doesn't really, for an i386 Debian installation you need to go
and find some slightly obscure media which has a PAE kernel on it in
order to install with PV drivers. If you just download the most obvious
i386 installation media you get no PV drivers of any description in an
HVM guest.

Fedora IIRC has moved everything over to PAE by default (no non-PAE
support), so they are probably OK.

I've no idea what Ubuntu does.

> Why do you want to do that, even though it will cause a performance
> regression and a maintenance pain?  You haven't provided a reason yet.

Where is the performance regression?

For a non-PAE x86 guest, which currently has 0 PV optimisations enabled
(no PV I/O, no PV clock, nothing) being able to enable PV I/O is a
useful performance improvement.

I'm also not saying that it *only* makes sense to enable PV I/O, if it
was also possible to enable other PV things, like PV clocks etc for
non-PAE x86 guests then that would also be worthwhile.

But I am saying that if enabling those extra optimisations for non-PAE
x86 guests is too invasive or problematic or whatever then it would
*still* be worth enabling PV I/O if that is more possible.

Note that in no case am I suggesting turning off something which is
possible today. In particular I see no reason to want to disable PV
optimisations for PAE enabled x86 guests.

Re: RFC: xen config changes v4

[Stefano Stabellini]

On Fri, 27 Feb 2015, Ian Campbell wrote:
> On Fri, 2015-02-27 at 10:11 +0000, Stefano Stabellini wrote:
> 
> (for some reason I initially thought this was in reply to my mail, so
> it's written in a way which assumes that, so sprinkle IMHO around the
> place and/or take it as a follow on to my previous mail in this thread,
> I guess)
> 
> > This is not a convincing explanation.  Debian, Ubuntu and Fedora seems
> > to be able to cope with it just fine.
> 
> Debian doesn't really, for an i386 Debian installation you need to go
> and find some slightly obscure media which has a PAE kernel on it in
> order to install with PV drivers. If you just download the most obvious
> i386 installation media you get no PV drivers of any description in an
> HVM guest.
> 
> Fedora IIRC has moved everything over to PAE by default (no non-PAE
> support), so they are probably OK.
> 
> I've no idea what Ubuntu does.
> 
> > Why do you want to do that, even though it will cause a performance
> > regression and a maintenance pain?  You haven't provided a reason yet.
> 
> Where is the performance regression?

For PAE and x86_64 guests.


> For a non-PAE x86 guest, which currently has 0 PV optimisations enabled
> (no PV I/O, no PV clock, nothing) being able to enable PV I/O is a
> useful performance improvement.
> 
> I'm also not saying that it *only* makes sense to enable PV I/O, if it
> was also possible to enable other PV things, like PV clocks etc for
> non-PAE x86 guests then that would also be worthwhile.
> 
> But I am saying that if enabling those extra optimisations for non-PAE
> x86 guests is too invasive or problematic or whatever then it would
> *still* be worth enabling PV I/O if that is more possible.

Yes, I think that would be valuable.


> Note that in no case am I suggesting turning off something which is
> possible today. In particular I see no reason to want to disable PV
> optimisations for PAE enabled x86 guests.

Right, this is what I am trying to avoid.  I would also like to avoid
having two levels of PVHVM guests for PAE enabled and x86_64 guests.

Re: RFC: xen config changes v4

[Juergen Gross]

On 02/27/2015 11:11 AM, Stefano Stabellini wrote:
> On Fri, 27 Feb 2015, Juergen Gross wrote:
>> On 02/27/2015 10:41 AM, Stefano Stabellini wrote:
>>> On Fri, 27 Feb 2015, Juergen Gross wrote:
>>>> On 02/26/2015 06:42 PM, Stefano Stabellini wrote:
>>>>> On Thu, 26 Feb 2015, Luis R. Rodriguez wrote:
>>>>>> On Thu, Feb 26, 2015 at 11:08:20AM +0000, Stefano Stabellini wrote:
>>>>>>> On Thu, 26 Feb 2015, David Vrabel wrote:
>>>>>>>> On 26/02/15 04:59, Juergen Gross wrote:
>>>>>>>>>
>>>>>>>>> So we are again in the situation that pv-drivers always imply
>>>>>>>>> the
>>>>>>>>> pvops
>>>>>>>>> kernel (PARAVIRT selected). I started the whole Kconfig rework
>>>>>>>>> to
>>>>>>>>> eliminate this dependency.
>>>>>>>>
>>>>>>>> Yes.  Can you produce a series that just addresses this one issue.
>>>>>>>>
>>>>>>>> In the absence of any concrete requirement for this big Kconfig
>>>>>>>> reorg
>>>>>>>> I
>>>>>>>> I don't think it is helpful.
>>>>>>>
>>>>>>> I clearly missed some context as I didn't realize that this was the
>>>>>>> intended goal. Why do we want this? Please explain as it won't come
>>>>>>> for free.
>>>>>>>
>>>>>>>
>>>>>>> We have a few PV interfaces for HVM guests that need PARAVIRT in
>>>>>>> Linux
>>>>>>> in order to be used, for example pv_time_ops and
>>>>>>> HVMOP_pagetable_dying.
>>>>>>> They are critical performance improvements and from the interface
>>>>>>> perspective, small enough that doesn't make much sense having a
>>>>>>> separate
>>>>>>> KConfig option for them.
>>>>>>>
>>>>>>>
>>>>>>> In order to reach the goal above we necessarily need to introduce a
>>>>>>> differentiation in terms of PV on HVM guests in Linux:
>>>>>>>
>>>>>>> 1) basic guests with PV network, disk, etc but no PV timers, no
>>>>>>>       HVMOP_pagetable_dying, no PV IPIs
>>>>>>> 2) full PV on HVM guests that have PV network, disk, timers,
>>>>>>>       HVMOP_pagetable_dying, PV IPIs and anything else that makes
>>>>>>> sense.
>>>>>>>
>>>>>>> 2) is much faster than 1) on Xen and 2) is only a tiny bit slower
>>>>>>> than
>>>>>>> 1) on native x86
>>>>>>
>>>>>> Also don't we shove 2) down hvm guests right now? Even when everything
>>>>>> is
>>>>>> built in I do not see how we opt out for HVM for 1) at run time right
>>>>>> now.
>>>>>>
>>>>>> If this is true then the question of motivation for this becomes even
>>>>>> stronger I think.
>>>>>
>>>>> Yes, indeed there is no way to do 1) at the moment. And for good
>>>>> reasons, see above.
>>>>
>>>> Hmm, after checking the code I'm not convinced:
>>>>
>>>> - HVMOP_pagetable_dying is obsolete on modern hardware supporting
>>>>     EPT/HAP
>>>
>>> That might be true, but what about older hardware?
>>> Even on modern hardware a few workloads still run faster on shadow.
>>> But if HVMOP_pagetable_dying is the only reason to keep PARAVIRT for HVM
>>> guests, then I agree with you that we should remove it.
>>>
>>>
>>>> - PV IPIs are not needed on single-vcpu guests
>>>>
>>>> - PARAVIRT_CLOCK doesn't need PARAVIRT (in fact the SUSEs kernel configs
>>>>     for all x86_64 kernels have CONFIG_PARAVIRT_CLOCK=y)
>>>>
>>>> So I think we really should enable building Xen frontends without
>>>> PARAVIRT, implying at least no XEN_PV and no XEN_PVH.
>>>>
>>>> I'll have a try setting up patches.
>>>
>>> If we are doing this as a performance improvement, I would like to see a
>>> couple of benchmarks (kernbench, hackbench) to show that on a
>>> single-vcpu guest and multi-vcpu guest (let's say 4 vcpus) disabling
>>> PARAVIRT leads to better performance on Xen on EPT hardware.
>>
>> This is not meant to be a performance improvement. It is meant to enable
>> a standard distro kernel configured without PARAVIRT to be able to run
>> as a HVM guest using the pv-drivers.
>
> This is not a convincing explanation.  Debian, Ubuntu and Fedora seems
> to be able to cope with it just fine.
>
> Why do you want to do that, even though it will cause a performance
> regression and a maintenance pain?  You haven't provided a reason yet.
>

Either we are talking about different things, or I really don't
understand your problem here. I don't want to disable something. I
just want to enable kernels without PARAVIRT to run under Xen better
than today. Being it 32 bit non-PAE kernels as Ian pointed out or
distro kernels like e.g. SLES and probably RHEL.

Using PV frontends is completely orthogonal to other PV enhancements
like PARAVIRT_CLOCK, HVMOP_pagetable_dying or PV IPIs. So why do you
object enabling the PV frontends for those kernels?


Juergen

Re: RFC: xen config changes v4

[Stefano Stabellini]

On Fri, 27 Feb 2015, Juergen Gross wrote:
> On 02/27/2015 11:11 AM, Stefano Stabellini wrote:
> > On Fri, 27 Feb 2015, Juergen Gross wrote:
> > > On 02/27/2015 10:41 AM, Stefano Stabellini wrote:
> > > > On Fri, 27 Feb 2015, Juergen Gross wrote:
> > > > > On 02/26/2015 06:42 PM, Stefano Stabellini wrote:
> > > > > > On Thu, 26 Feb 2015, Luis R. Rodriguez wrote:
> > > > > > > On Thu, Feb 26, 2015 at 11:08:20AM +0000, Stefano Stabellini
> > > > > > > wrote:
> > > > > > > > On Thu, 26 Feb 2015, David Vrabel wrote:
> > > > > > > > > On 26/02/15 04:59, Juergen Gross wrote:
> > > > > > > > > > 
> > > > > > > > > > So we are again in the situation that pv-drivers always
> > > > > > > > > > imply
> > > > > > > > > > the
> > > > > > > > > > pvops
> > > > > > > > > > kernel (PARAVIRT selected). I started the whole Kconfig
> > > > > > > > > > rework
> > > > > > > > > > to
> > > > > > > > > > eliminate this dependency.
> > > > > > > > > 
> > > > > > > > > Yes.  Can you produce a series that just addresses this one
> > > > > > > > > issue.
> > > > > > > > > 
> > > > > > > > > In the absence of any concrete requirement for this big
> > > > > > > > > Kconfig
> > > > > > > > > reorg
> > > > > > > > > I
> > > > > > > > > I don't think it is helpful.
> > > > > > > > 
> > > > > > > > I clearly missed some context as I didn't realize that this was
> > > > > > > > the
> > > > > > > > intended goal. Why do we want this? Please explain as it won't
> > > > > > > > come
> > > > > > > > for free.
> > > > > > > > 
> > > > > > > > 
> > > > > > > > We have a few PV interfaces for HVM guests that need PARAVIRT in
> > > > > > > > Linux
> > > > > > > > in order to be used, for example pv_time_ops and
> > > > > > > > HVMOP_pagetable_dying.
> > > > > > > > They are critical performance improvements and from the
> > > > > > > > interface
> > > > > > > > perspective, small enough that doesn't make much sense having a
> > > > > > > > separate
> > > > > > > > KConfig option for them.
> > > > > > > > 
> > > > > > > > 
> > > > > > > > In order to reach the goal above we necessarily need to
> > > > > > > > introduce a
> > > > > > > > differentiation in terms of PV on HVM guests in Linux:
> > > > > > > > 
> > > > > > > > 1) basic guests with PV network, disk, etc but no PV timers, no
> > > > > > > >       HVMOP_pagetable_dying, no PV IPIs
> > > > > > > > 2) full PV on HVM guests that have PV network, disk, timers,
> > > > > > > >       HVMOP_pagetable_dying, PV IPIs and anything else that
> > > > > > > > makes
> > > > > > > > sense.
> > > > > > > > 
> > > > > > > > 2) is much faster than 1) on Xen and 2) is only a tiny bit
> > > > > > > > slower
> > > > > > > > than
> > > > > > > > 1) on native x86
> > > > > > > 
> > > > > > > Also don't we shove 2) down hvm guests right now? Even when
> > > > > > > everything
> > > > > > > is
> > > > > > > built in I do not see how we opt out for HVM for 1) at run time
> > > > > > > right
> > > > > > > now.
> > > > > > > 
> > > > > > > If this is true then the question of motivation for this becomes
> > > > > > > even
> > > > > > > stronger I think.
> > > > > > 
> > > > > > Yes, indeed there is no way to do 1) at the moment. And for good
> > > > > > reasons, see above.
> > > > > 
> > > > > Hmm, after checking the code I'm not convinced:
> > > > > 
> > > > > - HVMOP_pagetable_dying is obsolete on modern hardware supporting
> > > > >     EPT/HAP
> > > > 
> > > > That might be true, but what about older hardware?
> > > > Even on modern hardware a few workloads still run faster on shadow.
> > > > But if HVMOP_pagetable_dying is the only reason to keep PARAVIRT for HVM
> > > > guests, then I agree with you that we should remove it.
> > > > 
> > > > 
> > > > > - PV IPIs are not needed on single-vcpu guests
> > > > > 
> > > > > - PARAVIRT_CLOCK doesn't need PARAVIRT (in fact the SUSEs kernel
> > > > > configs
> > > > >     for all x86_64 kernels have CONFIG_PARAVIRT_CLOCK=y)
> > > > > 
> > > > > So I think we really should enable building Xen frontends without
> > > > > PARAVIRT, implying at least no XEN_PV and no XEN_PVH.
> > > > > 
> > > > > I'll have a try setting up patches.
> > > > 
> > > > If we are doing this as a performance improvement, I would like to see a
> > > > couple of benchmarks (kernbench, hackbench) to show that on a
> > > > single-vcpu guest and multi-vcpu guest (let's say 4 vcpus) disabling
> > > > PARAVIRT leads to better performance on Xen on EPT hardware.
> > > 
> > > This is not meant to be a performance improvement. It is meant to enable
> > > a standard distro kernel configured without PARAVIRT to be able to run
> > > as a HVM guest using the pv-drivers.
> > 
> > This is not a convincing explanation.  Debian, Ubuntu and Fedora seems
> > to be able to cope with it just fine.
> > 
> > Why do you want to do that, even though it will cause a performance
> > regression and a maintenance pain?  You haven't provided a reason yet.
> > 
> 
> Either we are talking about different things, or I really don't
> understand your problem here. I don't want to disable something. I
> just want to enable kernels without PARAVIRT to run under Xen better
> than today. Being it 32 bit non-PAE kernels as Ian pointed out or
> distro kernels like e.g. SLES and probably RHEL.
> 
> Using PV frontends is completely orthogonal to other PV enhancements
> like PARAVIRT_CLOCK, HVMOP_pagetable_dying or PV IPIs. So why do you
> object enabling the PV frontends for those kernels?

I am for it.  I would like to avoid two user visible XEN enablement
options (XEN_FRONTEND vs. XEN_PVHVM) for x86_64 and PAE HVM guests to
avoid configurations with just XEN_FRONTEND, that can be considered a
performance regression compared to what we have now (on x86_64 and PAE).

Re: RFC: xen config changes v4

[Juergen Gross]

On 02/27/2015 01:24 PM, Stefano Stabellini wrote:
> On Fri, 27 Feb 2015, Juergen Gross wrote:
>> On 02/27/2015 11:11 AM, Stefano Stabellini wrote:
>>> On Fri, 27 Feb 2015, Juergen Gross wrote:
>>>> On 02/27/2015 10:41 AM, Stefano Stabellini wrote:
>>>>> On Fri, 27 Feb 2015, Juergen Gross wrote:
>>>>>> On 02/26/2015 06:42 PM, Stefano Stabellini wrote:
>>>>>>> On Thu, 26 Feb 2015, Luis R. Rodriguez wrote:
>>>>>>>> On Thu, Feb 26, 2015 at 11:08:20AM +0000, Stefano Stabellini
>>>>>>>> wrote:
>>>>>>>>> On Thu, 26 Feb 2015, David Vrabel wrote:
>>>>>>>>>> On 26/02/15 04:59, Juergen Gross wrote:
>>>>>>>>>>>
>>>>>>>>>>> So we are again in the situation that pv-drivers always
>>>>>>>>>>> imply
>>>>>>>>>>> the
>>>>>>>>>>> pvops
>>>>>>>>>>> kernel (PARAVIRT selected). I started the whole Kconfig
>>>>>>>>>>> rework
>>>>>>>>>>> to
>>>>>>>>>>> eliminate this dependency.
>>>>>>>>>>
>>>>>>>>>> Yes.  Can you produce a series that just addresses this one
>>>>>>>>>> issue.
>>>>>>>>>>
>>>>>>>>>> In the absence of any concrete requirement for this big
>>>>>>>>>> Kconfig
>>>>>>>>>> reorg
>>>>>>>>>> I
>>>>>>>>>> I don't think it is helpful.
>>>>>>>>>
>>>>>>>>> I clearly missed some context as I didn't realize that this was
>>>>>>>>> the
>>>>>>>>> intended goal. Why do we want this? Please explain as it won't
>>>>>>>>> come
>>>>>>>>> for free.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> We have a few PV interfaces for HVM guests that need PARAVIRT in
>>>>>>>>> Linux
>>>>>>>>> in order to be used, for example pv_time_ops and
>>>>>>>>> HVMOP_pagetable_dying.
>>>>>>>>> They are critical performance improvements and from the
>>>>>>>>> interface
>>>>>>>>> perspective, small enough that doesn't make much sense having a
>>>>>>>>> separate
>>>>>>>>> KConfig option for them.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> In order to reach the goal above we necessarily need to
>>>>>>>>> introduce a
>>>>>>>>> differentiation in terms of PV on HVM guests in Linux:
>>>>>>>>>
>>>>>>>>> 1) basic guests with PV network, disk, etc but no PV timers, no
>>>>>>>>>        HVMOP_pagetable_dying, no PV IPIs
>>>>>>>>> 2) full PV on HVM guests that have PV network, disk, timers,
>>>>>>>>>        HVMOP_pagetable_dying, PV IPIs and anything else that
>>>>>>>>> makes
>>>>>>>>> sense.
>>>>>>>>>
>>>>>>>>> 2) is much faster than 1) on Xen and 2) is only a tiny bit
>>>>>>>>> slower
>>>>>>>>> than
>>>>>>>>> 1) on native x86
>>>>>>>>
>>>>>>>> Also don't we shove 2) down hvm guests right now? Even when
>>>>>>>> everything
>>>>>>>> is
>>>>>>>> built in I do not see how we opt out for HVM for 1) at run time
>>>>>>>> right
>>>>>>>> now.
>>>>>>>>
>>>>>>>> If this is true then the question of motivation for this becomes
>>>>>>>> even
>>>>>>>> stronger I think.
>>>>>>>
>>>>>>> Yes, indeed there is no way to do 1) at the moment. And for good
>>>>>>> reasons, see above.
>>>>>>
>>>>>> Hmm, after checking the code I'm not convinced:
>>>>>>
>>>>>> - HVMOP_pagetable_dying is obsolete on modern hardware supporting
>>>>>>      EPT/HAP
>>>>>
>>>>> That might be true, but what about older hardware?
>>>>> Even on modern hardware a few workloads still run faster on shadow.
>>>>> But if HVMOP_pagetable_dying is the only reason to keep PARAVIRT for HVM
>>>>> guests, then I agree with you that we should remove it.
>>>>>
>>>>>
>>>>>> - PV IPIs are not needed on single-vcpu guests
>>>>>>
>>>>>> - PARAVIRT_CLOCK doesn't need PARAVIRT (in fact the SUSEs kernel
>>>>>> configs
>>>>>>      for all x86_64 kernels have CONFIG_PARAVIRT_CLOCK=y)
>>>>>>
>>>>>> So I think we really should enable building Xen frontends without
>>>>>> PARAVIRT, implying at least no XEN_PV and no XEN_PVH.
>>>>>>
>>>>>> I'll have a try setting up patches.
>>>>>
>>>>> If we are doing this as a performance improvement, I would like to see a
>>>>> couple of benchmarks (kernbench, hackbench) to show that on a
>>>>> single-vcpu guest and multi-vcpu guest (let's say 4 vcpus) disabling
>>>>> PARAVIRT leads to better performance on Xen on EPT hardware.
>>>>
>>>> This is not meant to be a performance improvement. It is meant to enable
>>>> a standard distro kernel configured without PARAVIRT to be able to run
>>>> as a HVM guest using the pv-drivers.
>>>
>>> This is not a convincing explanation.  Debian, Ubuntu and Fedora seems
>>> to be able to cope with it just fine.
>>>
>>> Why do you want to do that, even though it will cause a performance
>>> regression and a maintenance pain?  You haven't provided a reason yet.
>>>
>>
>> Either we are talking about different things, or I really don't
>> understand your problem here. I don't want to disable something. I
>> just want to enable kernels without PARAVIRT to run under Xen better
>> than today. Being it 32 bit non-PAE kernels as Ian pointed out or
>> distro kernels like e.g. SLES and probably RHEL.
>>
>> Using PV frontends is completely orthogonal to other PV enhancements
>> like PARAVIRT_CLOCK, HVMOP_pagetable_dying or PV IPIs. So why do you
>> object enabling the PV frontends for those kernels?
>
> I am for it.  I would like to avoid two user visible XEN enablement
> options (XEN_FRONTEND vs. XEN_PVHVM) for x86_64 and PAE HVM guests to
> avoid configurations with just XEN_FRONTEND, that can be considered a
> performance regression compared to what we have now (on x86_64 and PAE).

Would you be okay with making this an expert configuration alternative
for PAE/x86_64? This would enable the possibility to use PV drivers for
native-performance-tuned kernels. I would explicitly mention the better
alternative XEN_PVHVM in the Kconfig help text.


Juergen

Re: RFC: xen config changes v4

[Ian Campbell]

On Fri, 2015-02-27 at 12:24 +0000, Stefano Stabellini wrote:
> > Using PV frontends is completely orthogonal to other PV enhancements
> > like PARAVIRT_CLOCK, HVMOP_pagetable_dying or PV IPIs. So why do you
> > object enabling the PV frontends for those kernels?
> 
> I am for it.  I would like to avoid two user visible XEN enablement
> options (XEN_FRONTEND vs. XEN_PVHVM) for x86_64 and PAE HVM guests to
> avoid configurations with just XEN_FRONTEND, that can be considered a
> performance regression compared to what we have now (on x86_64 and PAE).

Probably some suitable select/depends could be used to allow this
configuration only for non-PAE/x86_32 and x86_64 configurations?

Ian.

Re: RFC: xen config changes v4

[Stefano Stabellini]

On Fri, 27 Feb 2015, Juergen Gross wrote:
> On 02/27/2015 01:24 PM, Stefano Stabellini wrote:
> > On Fri, 27 Feb 2015, Juergen Gross wrote:
> > > On 02/27/2015 11:11 AM, Stefano Stabellini wrote:
> > > > On Fri, 27 Feb 2015, Juergen Gross wrote:
> > > > > On 02/27/2015 10:41 AM, Stefano Stabellini wrote:
> > > > > > On Fri, 27 Feb 2015, Juergen Gross wrote:
> > > > > > > On 02/26/2015 06:42 PM, Stefano Stabellini wrote:
> > > > > > > > On Thu, 26 Feb 2015, Luis R. Rodriguez wrote:
> > > > > > > > > On Thu, Feb 26, 2015 at 11:08:20AM +0000, Stefano Stabellini
> > > > > > > > > wrote:
> > > > > > > > > > On Thu, 26 Feb 2015, David Vrabel wrote:
> > > > > > > > > > > On 26/02/15 04:59, Juergen Gross wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > So we are again in the situation that pv-drivers always
> > > > > > > > > > > > imply
> > > > > > > > > > > > the
> > > > > > > > > > > > pvops
> > > > > > > > > > > > kernel (PARAVIRT selected). I started the whole Kconfig
> > > > > > > > > > > > rework
> > > > > > > > > > > > to
> > > > > > > > > > > > eliminate this dependency.
> > > > > > > > > > > 
> > > > > > > > > > > Yes.  Can you produce a series that just addresses this
> > > > > > > > > > > one
> > > > > > > > > > > issue.
> > > > > > > > > > > 
> > > > > > > > > > > In the absence of any concrete requirement for this big
> > > > > > > > > > > Kconfig
> > > > > > > > > > > reorg
> > > > > > > > > > > I
> > > > > > > > > > > I don't think it is helpful.
> > > > > > > > > > 
> > > > > > > > > > I clearly missed some context as I didn't realize that this
> > > > > > > > > > was
> > > > > > > > > > the
> > > > > > > > > > intended goal. Why do we want this? Please explain as it
> > > > > > > > > > won't
> > > > > > > > > > come
> > > > > > > > > > for free.
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > We have a few PV interfaces for HVM guests that need
> > > > > > > > > > PARAVIRT in
> > > > > > > > > > Linux
> > > > > > > > > > in order to be used, for example pv_time_ops and
> > > > > > > > > > HVMOP_pagetable_dying.
> > > > > > > > > > They are critical performance improvements and from the
> > > > > > > > > > interface
> > > > > > > > > > perspective, small enough that doesn't make much sense
> > > > > > > > > > having a
> > > > > > > > > > separate
> > > > > > > > > > KConfig option for them.
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > In order to reach the goal above we necessarily need to
> > > > > > > > > > introduce a
> > > > > > > > > > differentiation in terms of PV on HVM guests in Linux:
> > > > > > > > > > 
> > > > > > > > > > 1) basic guests with PV network, disk, etc but no PV timers,
> > > > > > > > > > no
> > > > > > > > > >        HVMOP_pagetable_dying, no PV IPIs
> > > > > > > > > > 2) full PV on HVM guests that have PV network, disk, timers,
> > > > > > > > > >        HVMOP_pagetable_dying, PV IPIs and anything else that
> > > > > > > > > > makes
> > > > > > > > > > sense.
> > > > > > > > > > 
> > > > > > > > > > 2) is much faster than 1) on Xen and 2) is only a tiny bit
> > > > > > > > > > slower
> > > > > > > > > > than
> > > > > > > > > > 1) on native x86
> > > > > > > > > 
> > > > > > > > > Also don't we shove 2) down hvm guests right now? Even when
> > > > > > > > > everything
> > > > > > > > > is
> > > > > > > > > built in I do not see how we opt out for HVM for 1) at run
> > > > > > > > > time
> > > > > > > > > right
> > > > > > > > > now.
> > > > > > > > > 
> > > > > > > > > If this is true then the question of motivation for this
> > > > > > > > > becomes
> > > > > > > > > even
> > > > > > > > > stronger I think.
> > > > > > > > 
> > > > > > > > Yes, indeed there is no way to do 1) at the moment. And for good
> > > > > > > > reasons, see above.
> > > > > > > 
> > > > > > > Hmm, after checking the code I'm not convinced:
> > > > > > > 
> > > > > > > - HVMOP_pagetable_dying is obsolete on modern hardware supporting
> > > > > > >      EPT/HAP
> > > > > > 
> > > > > > That might be true, but what about older hardware?
> > > > > > Even on modern hardware a few workloads still run faster on shadow.
> > > > > > But if HVMOP_pagetable_dying is the only reason to keep PARAVIRT for
> > > > > > HVM
> > > > > > guests, then I agree with you that we should remove it.
> > > > > > 
> > > > > > 
> > > > > > > - PV IPIs are not needed on single-vcpu guests
> > > > > > > 
> > > > > > > - PARAVIRT_CLOCK doesn't need PARAVIRT (in fact the SUSEs kernel
> > > > > > > configs
> > > > > > >      for all x86_64 kernels have CONFIG_PARAVIRT_CLOCK=y)
> > > > > > > 
> > > > > > > So I think we really should enable building Xen frontends without
> > > > > > > PARAVIRT, implying at least no XEN_PV and no XEN_PVH.
> > > > > > > 
> > > > > > > I'll have a try setting up patches.
> > > > > > 
> > > > > > If we are doing this as a performance improvement, I would like to
> > > > > > see a
> > > > > > couple of benchmarks (kernbench, hackbench) to show that on a
> > > > > > single-vcpu guest and multi-vcpu guest (let's say 4 vcpus) disabling
> > > > > > PARAVIRT leads to better performance on Xen on EPT hardware.
> > > > > 
> > > > > This is not meant to be a performance improvement. It is meant to
> > > > > enable
> > > > > a standard distro kernel configured without PARAVIRT to be able to run
> > > > > as a HVM guest using the pv-drivers.
> > > > 
> > > > This is not a convincing explanation.  Debian, Ubuntu and Fedora seems
> > > > to be able to cope with it just fine.
> > > > 
> > > > Why do you want to do that, even though it will cause a performance
> > > > regression and a maintenance pain?  You haven't provided a reason yet.
> > > > 
> > > 
> > > Either we are talking about different things, or I really don't
> > > understand your problem here. I don't want to disable something. I
> > > just want to enable kernels without PARAVIRT to run under Xen better
> > > than today. Being it 32 bit non-PAE kernels as Ian pointed out or
> > > distro kernels like e.g. SLES and probably RHEL.
> > > 
> > > Using PV frontends is completely orthogonal to other PV enhancements
> > > like PARAVIRT_CLOCK, HVMOP_pagetable_dying or PV IPIs. So why do you
> > > object enabling the PV frontends for those kernels?
> > 
> > I am for it.  I would like to avoid two user visible XEN enablement
> > options (XEN_FRONTEND vs. XEN_PVHVM) for x86_64 and PAE HVM guests to
> > avoid configurations with just XEN_FRONTEND, that can be considered a
> > performance regression compared to what we have now (on x86_64 and PAE).
> 
> Would you be okay with making this an expert configuration alternative
> for PAE/x86_64? This would enable the possibility to use PV drivers for
> native-performance-tuned kernels. I would explicitly mention the better
> alternative XEN_PVHVM in the Kconfig help text.

I would prefer to hide it on PAE and x86_64.

Re: RFC: xen config changes v4

[Juergen Gross]

On 02/27/2015 02:38 PM, Stefano Stabellini wrote:
> On Fri, 27 Feb 2015, Juergen Gross wrote:
>> On 02/27/2015 01:24 PM, Stefano Stabellini wrote:
>>> On Fri, 27 Feb 2015, Juergen Gross wrote:
>>>> On 02/27/2015 11:11 AM, Stefano Stabellini wrote:
>>>>> On Fri, 27 Feb 2015, Juergen Gross wrote:
>>>>>> On 02/27/2015 10:41 AM, Stefano Stabellini wrote:
>>>>>>> On Fri, 27 Feb 2015, Juergen Gross wrote:
>>>>>>>> On 02/26/2015 06:42 PM, Stefano Stabellini wrote:
>>>>>>>>> On Thu, 26 Feb 2015, Luis R. Rodriguez wrote:
>>>>>>>>>> On Thu, Feb 26, 2015 at 11:08:20AM +0000, Stefano Stabellini
>>>>>>>>>> wrote:
>>>>>>>>>>> On Thu, 26 Feb 2015, David Vrabel wrote:
>>>>>>>>>>>> On 26/02/15 04:59, Juergen Gross wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> So we are again in the situation that pv-drivers always
>>>>>>>>>>>>> imply
>>>>>>>>>>>>> the
>>>>>>>>>>>>> pvops
>>>>>>>>>>>>> kernel (PARAVIRT selected). I started the whole Kconfig
>>>>>>>>>>>>> rework
>>>>>>>>>>>>> to
>>>>>>>>>>>>> eliminate this dependency.
>>>>>>>>>>>>
>>>>>>>>>>>> Yes.  Can you produce a series that just addresses this
>>>>>>>>>>>> one
>>>>>>>>>>>> issue.
>>>>>>>>>>>>
>>>>>>>>>>>> In the absence of any concrete requirement for this big
>>>>>>>>>>>> Kconfig
>>>>>>>>>>>> reorg
>>>>>>>>>>>> I
>>>>>>>>>>>> I don't think it is helpful.
>>>>>>>>>>>
>>>>>>>>>>> I clearly missed some context as I didn't realize that this
>>>>>>>>>>> was
>>>>>>>>>>> the
>>>>>>>>>>> intended goal. Why do we want this? Please explain as it
>>>>>>>>>>> won't
>>>>>>>>>>> come
>>>>>>>>>>> for free.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> We have a few PV interfaces for HVM guests that need
>>>>>>>>>>> PARAVIRT in
>>>>>>>>>>> Linux
>>>>>>>>>>> in order to be used, for example pv_time_ops and
>>>>>>>>>>> HVMOP_pagetable_dying.
>>>>>>>>>>> They are critical performance improvements and from the
>>>>>>>>>>> interface
>>>>>>>>>>> perspective, small enough that doesn't make much sense
>>>>>>>>>>> having a
>>>>>>>>>>> separate
>>>>>>>>>>> KConfig option for them.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> In order to reach the goal above we necessarily need to
>>>>>>>>>>> introduce a
>>>>>>>>>>> differentiation in terms of PV on HVM guests in Linux:
>>>>>>>>>>>
>>>>>>>>>>> 1) basic guests with PV network, disk, etc but no PV timers,
>>>>>>>>>>> no
>>>>>>>>>>>         HVMOP_pagetable_dying, no PV IPIs
>>>>>>>>>>> 2) full PV on HVM guests that have PV network, disk, timers,
>>>>>>>>>>>         HVMOP_pagetable_dying, PV IPIs and anything else that
>>>>>>>>>>> makes
>>>>>>>>>>> sense.
>>>>>>>>>>>
>>>>>>>>>>> 2) is much faster than 1) on Xen and 2) is only a tiny bit
>>>>>>>>>>> slower
>>>>>>>>>>> than
>>>>>>>>>>> 1) on native x86
>>>>>>>>>>
>>>>>>>>>> Also don't we shove 2) down hvm guests right now? Even when
>>>>>>>>>> everything
>>>>>>>>>> is
>>>>>>>>>> built in I do not see how we opt out for HVM for 1) at run
>>>>>>>>>> time
>>>>>>>>>> right
>>>>>>>>>> now.
>>>>>>>>>>
>>>>>>>>>> If this is true then the question of motivation for this
>>>>>>>>>> becomes
>>>>>>>>>> even
>>>>>>>>>> stronger I think.
>>>>>>>>>
>>>>>>>>> Yes, indeed there is no way to do 1) at the moment. And for good
>>>>>>>>> reasons, see above.
>>>>>>>>
>>>>>>>> Hmm, after checking the code I'm not convinced:
>>>>>>>>
>>>>>>>> - HVMOP_pagetable_dying is obsolete on modern hardware supporting
>>>>>>>>       EPT/HAP
>>>>>>>
>>>>>>> That might be true, but what about older hardware?
>>>>>>> Even on modern hardware a few workloads still run faster on shadow.
>>>>>>> But if HVMOP_pagetable_dying is the only reason to keep PARAVIRT for
>>>>>>> HVM
>>>>>>> guests, then I agree with you that we should remove it.
>>>>>>>
>>>>>>>
>>>>>>>> - PV IPIs are not needed on single-vcpu guests
>>>>>>>>
>>>>>>>> - PARAVIRT_CLOCK doesn't need PARAVIRT (in fact the SUSEs kernel
>>>>>>>> configs
>>>>>>>>       for all x86_64 kernels have CONFIG_PARAVIRT_CLOCK=y)
>>>>>>>>
>>>>>>>> So I think we really should enable building Xen frontends without
>>>>>>>> PARAVIRT, implying at least no XEN_PV and no XEN_PVH.
>>>>>>>>
>>>>>>>> I'll have a try setting up patches.
>>>>>>>
>>>>>>> If we are doing this as a performance improvement, I would like to
>>>>>>> see a
>>>>>>> couple of benchmarks (kernbench, hackbench) to show that on a
>>>>>>> single-vcpu guest and multi-vcpu guest (let's say 4 vcpus) disabling
>>>>>>> PARAVIRT leads to better performance on Xen on EPT hardware.
>>>>>>
>>>>>> This is not meant to be a performance improvement. It is meant to
>>>>>> enable
>>>>>> a standard distro kernel configured without PARAVIRT to be able to run
>>>>>> as a HVM guest using the pv-drivers.
>>>>>
>>>>> This is not a convincing explanation.  Debian, Ubuntu and Fedora seems
>>>>> to be able to cope with it just fine.
>>>>>
>>>>> Why do you want to do that, even though it will cause a performance
>>>>> regression and a maintenance pain?  You haven't provided a reason yet.
>>>>>
>>>>
>>>> Either we are talking about different things, or I really don't
>>>> understand your problem here. I don't want to disable something. I
>>>> just want to enable kernels without PARAVIRT to run under Xen better
>>>> than today. Being it 32 bit non-PAE kernels as Ian pointed out or
>>>> distro kernels like e.g. SLES and probably RHEL.
>>>>
>>>> Using PV frontends is completely orthogonal to other PV enhancements
>>>> like PARAVIRT_CLOCK, HVMOP_pagetable_dying or PV IPIs. So why do you
>>>> object enabling the PV frontends for those kernels?
>>>
>>> I am for it.  I would like to avoid two user visible XEN enablement
>>> options (XEN_FRONTEND vs. XEN_PVHVM) for x86_64 and PAE HVM guests to
>>> avoid configurations with just XEN_FRONTEND, that can be considered a
>>> performance regression compared to what we have now (on x86_64 and PAE).
>>
>> Would you be okay with making this an expert configuration alternative
>> for PAE/x86_64? This would enable the possibility to use PV drivers for
>> native-performance-tuned kernels. I would explicitly mention the better
>> alternative XEN_PVHVM in the Kconfig help text.
>
> I would prefer to hide it on PAE and x86_64.

Okay, as long as it is still _possible_ somehow to configure it.

Juergen

Re: RFC: xen config changes v4

[Luis R. Rodriguez]

On Fri, Feb 27, 2015 at 6:30 AM, Juergen Gross <jgross@suse.com> wrote:
> On 02/27/2015 02:38 PM, Stefano Stabellini wrote:
>>
>> On Fri, 27 Feb 2015, Juergen Gross wrote:
>>>
>>> On 02/27/2015 01:24 PM, Stefano Stabellini wrote:
>>>>
>>>> On Fri, 27 Feb 2015, Juergen Gross wrote:
>>>>>
>>>>> On 02/27/2015 11:11 AM, Stefano Stabellini wrote:
>>>>>>
>>>>>> On Fri, 27 Feb 2015, Juergen Gross wrote:
>>>>>>>
>>>>>>> On 02/27/2015 10:41 AM, Stefano Stabellini wrote:
>>>>>>>>
>>>>>>>> On Fri, 27 Feb 2015, Juergen Gross wrote:
>>>>>>>>>
>>>>>>>>> On 02/26/2015 06:42 PM, Stefano Stabellini wrote:
>>>>>>>>>>
>>>>>>>>>> On Thu, 26 Feb 2015, Luis R. Rodriguez wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Feb 26, 2015 at 11:08:20AM +0000, Stefano Stabellini
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, 26 Feb 2015, David Vrabel wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 26/02/15 04:59, Juergen Gross wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So we are again in the situation that pv-drivers always
>>>>>>>>>>>>>> imply
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> pvops
>>>>>>>>>>>>>> kernel (PARAVIRT selected). I started the whole Kconfig
>>>>>>>>>>>>>> rework
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> eliminate this dependency.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes.  Can you produce a series that just addresses this
>>>>>>>>>>>>> one
>>>>>>>>>>>>> issue.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In the absence of any concrete requirement for this big
>>>>>>>>>>>>> Kconfig
>>>>>>>>>>>>> reorg
>>>>>>>>>>>>> I
>>>>>>>>>>>>> I don't think it is helpful.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I clearly missed some context as I didn't realize that this
>>>>>>>>>>>> was
>>>>>>>>>>>> the
>>>>>>>>>>>> intended goal. Why do we want this? Please explain as it
>>>>>>>>>>>> won't
>>>>>>>>>>>> come
>>>>>>>>>>>> for free.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> We have a few PV interfaces for HVM guests that need
>>>>>>>>>>>> PARAVIRT in
>>>>>>>>>>>> Linux
>>>>>>>>>>>> in order to be used, for example pv_time_ops and
>>>>>>>>>>>> HVMOP_pagetable_dying.
>>>>>>>>>>>> They are critical performance improvements and from the
>>>>>>>>>>>> interface
>>>>>>>>>>>> perspective, small enough that doesn't make much sense
>>>>>>>>>>>> having a
>>>>>>>>>>>> separate
>>>>>>>>>>>> KConfig option for them.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> In order to reach the goal above we necessarily need to
>>>>>>>>>>>> introduce a
>>>>>>>>>>>> differentiation in terms of PV on HVM guests in Linux:
>>>>>>>>>>>>
>>>>>>>>>>>> 1) basic guests with PV network, disk, etc but no PV timers,
>>>>>>>>>>>> no
>>>>>>>>>>>>         HVMOP_pagetable_dying, no PV IPIs
>>>>>>>>>>>> 2) full PV on HVM guests that have PV network, disk, timers,
>>>>>>>>>>>>         HVMOP_pagetable_dying, PV IPIs and anything else that
>>>>>>>>>>>> makes
>>>>>>>>>>>> sense.
>>>>>>>>>>>>
>>>>>>>>>>>> 2) is much faster than 1) on Xen and 2) is only a tiny bit
>>>>>>>>>>>> slower
>>>>>>>>>>>> than
>>>>>>>>>>>> 1) on native x86
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Also don't we shove 2) down hvm guests right now? Even when
>>>>>>>>>>> everything
>>>>>>>>>>> is
>>>>>>>>>>> built in I do not see how we opt out for HVM for 1) at run
>>>>>>>>>>> time
>>>>>>>>>>> right
>>>>>>>>>>> now.
>>>>>>>>>>>
>>>>>>>>>>> If this is true then the question of motivation for this
>>>>>>>>>>> becomes
>>>>>>>>>>> even
>>>>>>>>>>> stronger I think.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Yes, indeed there is no way to do 1) at the moment. And for good
>>>>>>>>>> reasons, see above.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hmm, after checking the code I'm not convinced:
>>>>>>>>>
>>>>>>>>> - HVMOP_pagetable_dying is obsolete on modern hardware supporting
>>>>>>>>>       EPT/HAP
>>>>>>>>
>>>>>>>>
>>>>>>>> That might be true, but what about older hardware?
>>>>>>>> Even on modern hardware a few workloads still run faster on shadow.
>>>>>>>> But if HVMOP_pagetable_dying is the only reason to keep PARAVIRT for
>>>>>>>> HVM
>>>>>>>> guests, then I agree with you that we should remove it.
>>>>>>>>
>>>>>>>>
>>>>>>>>> - PV IPIs are not needed on single-vcpu guests
>>>>>>>>>
>>>>>>>>> - PARAVIRT_CLOCK doesn't need PARAVIRT (in fact the SUSEs kernel
>>>>>>>>> configs
>>>>>>>>>       for all x86_64 kernels have CONFIG_PARAVIRT_CLOCK=y)
>>>>>>>>>
>>>>>>>>> So I think we really should enable building Xen frontends without
>>>>>>>>> PARAVIRT, implying at least no XEN_PV and no XEN_PVH.
>>>>>>>>>
>>>>>>>>> I'll have a try setting up patches.
>>>>>>>>
>>>>>>>>
>>>>>>>> If we are doing this as a performance improvement, I would like to
>>>>>>>> see a
>>>>>>>> couple of benchmarks (kernbench, hackbench) to show that on a
>>>>>>>> single-vcpu guest and multi-vcpu guest (let's say 4 vcpus) disabling
>>>>>>>> PARAVIRT leads to better performance on Xen on EPT hardware.
>>>>>>>
>>>>>>>
>>>>>>> This is not meant to be a performance improvement. It is meant to
>>>>>>> enable
>>>>>>> a standard distro kernel configured without PARAVIRT to be able to
>>>>>>> run
>>>>>>> as a HVM guest using the pv-drivers.
>>>>>>
>>>>>>
>>>>>> This is not a convincing explanation.  Debian, Ubuntu and Fedora seems
>>>>>> to be able to cope with it just fine.
>>>>>>
>>>>>> Why do you want to do that, even though it will cause a performance
>>>>>> regression and a maintenance pain?  You haven't provided a reason yet.
>>>>>>
>>>>>
>>>>> Either we are talking about different things, or I really don't
>>>>> understand your problem here. I don't want to disable something. I
>>>>> just want to enable kernels without PARAVIRT to run under Xen better
>>>>> than today. Being it 32 bit non-PAE kernels as Ian pointed out or
>>>>> distro kernels like e.g. SLES and probably RHEL.
>>>>>
>>>>> Using PV frontends is completely orthogonal to other PV enhancements
>>>>> like PARAVIRT_CLOCK, HVMOP_pagetable_dying or PV IPIs. So why do you
>>>>> object enabling the PV frontends for those kernels?
>>>>
>>>>
>>>> I am for it.  I would like to avoid two user visible XEN enablement
>>>> options (XEN_FRONTEND vs. XEN_PVHVM) for x86_64 and PAE HVM guests to
>>>> avoid configurations with just XEN_FRONTEND, that can be considered a
>>>> performance regression compared to what we have now (on x86_64 and PAE).
>>>
>>>
>>> Would you be okay with making this an expert configuration alternative
>>> for PAE/x86_64? This would enable the possibility to use PV drivers for
>>> native-performance-tuned kernels. I would explicitly mention the better
>>> alternative XEN_PVHVM in the Kconfig help text.
>>
>>
>> I would prefer to hide it on PAE and x86_64.
>
>
> Okay, as long as it is still _possible_ somehow to configure it.

That begs the question, all this just for 32-bit non-PAE ?

 Luis

Re: RFC: xen config changes v4

[Konrad Rzeszutek Wilk]

> > This is not meant to be a performance improvement. It is meant to enable
> > a standard distro kernel configured without PARAVIRT to be able to run
> > as a HVM guest using the pv-drivers.
>  
> This is not a convincing explanation.  Debian, Ubuntu and Fedora seems
> to be able to cope with it just fine.

No they are not. The 32-bit Fedora Core 21 LiveISO is non-PAE. I think the
same situation was with Ubuntu.
> 
> Why do you want to do that, even though it will cause a performance
> regression and a maintenance pain?  You haven't provided a reason yet.

Re: RFC: xen config changes v4

[Konrad Rzeszutek Wilk]

On Fri, Feb 27, 2015 at 09:53:46AM -0800, Luis R. Rodriguez wrote:
> On Fri, Feb 27, 2015 at 6:30 AM, Juergen Gross <jgross@suse.com> wrote:
> > On 02/27/2015 02:38 PM, Stefano Stabellini wrote:
> >>
> >> On Fri, 27 Feb 2015, Juergen Gross wrote:
> >>>
> >>> On 02/27/2015 01:24 PM, Stefano Stabellini wrote:
> >>>>
> >>>> On Fri, 27 Feb 2015, Juergen Gross wrote:
> >>>>>
> >>>>> On 02/27/2015 11:11 AM, Stefano Stabellini wrote:
> >>>>>>
> >>>>>> On Fri, 27 Feb 2015, Juergen Gross wrote:
> >>>>>>>
> >>>>>>> On 02/27/2015 10:41 AM, Stefano Stabellini wrote:
> >>>>>>>>
> >>>>>>>> On Fri, 27 Feb 2015, Juergen Gross wrote:
> >>>>>>>>>
> >>>>>>>>> On 02/26/2015 06:42 PM, Stefano Stabellini wrote:
> >>>>>>>>>>
> >>>>>>>>>> On Thu, 26 Feb 2015, Luis R. Rodriguez wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Feb 26, 2015 at 11:08:20AM +0000, Stefano Stabellini
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, 26 Feb 2015, David Vrabel wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 26/02/15 04:59, Juergen Gross wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> So we are again in the situation that pv-drivers always
> >>>>>>>>>>>>>> imply
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>> pvops
> >>>>>>>>>>>>>> kernel (PARAVIRT selected). I started the whole Kconfig
> >>>>>>>>>>>>>> rework
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>> eliminate this dependency.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Yes.  Can you produce a series that just addresses this
> >>>>>>>>>>>>> one
> >>>>>>>>>>>>> issue.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> In the absence of any concrete requirement for this big
> >>>>>>>>>>>>> Kconfig
> >>>>>>>>>>>>> reorg
> >>>>>>>>>>>>> I
> >>>>>>>>>>>>> I don't think it is helpful.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> I clearly missed some context as I didn't realize that this
> >>>>>>>>>>>> was
> >>>>>>>>>>>> the
> >>>>>>>>>>>> intended goal. Why do we want this? Please explain as it
> >>>>>>>>>>>> won't
> >>>>>>>>>>>> come
> >>>>>>>>>>>> for free.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> We have a few PV interfaces for HVM guests that need
> >>>>>>>>>>>> PARAVIRT in
> >>>>>>>>>>>> Linux
> >>>>>>>>>>>> in order to be used, for example pv_time_ops and
> >>>>>>>>>>>> HVMOP_pagetable_dying.
> >>>>>>>>>>>> They are critical performance improvements and from the
> >>>>>>>>>>>> interface
> >>>>>>>>>>>> perspective, small enough that doesn't make much sense
> >>>>>>>>>>>> having a
> >>>>>>>>>>>> separate
> >>>>>>>>>>>> KConfig option for them.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> In order to reach the goal above we necessarily need to
> >>>>>>>>>>>> introduce a
> >>>>>>>>>>>> differentiation in terms of PV on HVM guests in Linux:
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1) basic guests with PV network, disk, etc but no PV timers,
> >>>>>>>>>>>> no
> >>>>>>>>>>>>         HVMOP_pagetable_dying, no PV IPIs
> >>>>>>>>>>>> 2) full PV on HVM guests that have PV network, disk, timers,
> >>>>>>>>>>>>         HVMOP_pagetable_dying, PV IPIs and anything else that
> >>>>>>>>>>>> makes
> >>>>>>>>>>>> sense.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2) is much faster than 1) on Xen and 2) is only a tiny bit
> >>>>>>>>>>>> slower
> >>>>>>>>>>>> than
> >>>>>>>>>>>> 1) on native x86
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Also don't we shove 2) down hvm guests right now? Even when
> >>>>>>>>>>> everything
> >>>>>>>>>>> is
> >>>>>>>>>>> built in I do not see how we opt out for HVM for 1) at run
> >>>>>>>>>>> time
> >>>>>>>>>>> right
> >>>>>>>>>>> now.
> >>>>>>>>>>>
> >>>>>>>>>>> If this is true then the question of motivation for this
> >>>>>>>>>>> becomes
> >>>>>>>>>>> even
> >>>>>>>>>>> stronger I think.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Yes, indeed there is no way to do 1) at the moment. And for good
> >>>>>>>>>> reasons, see above.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Hmm, after checking the code I'm not convinced:
> >>>>>>>>>
> >>>>>>>>> - HVMOP_pagetable_dying is obsolete on modern hardware supporting
> >>>>>>>>>       EPT/HAP
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> That might be true, but what about older hardware?
> >>>>>>>> Even on modern hardware a few workloads still run faster on shadow.
> >>>>>>>> But if HVMOP_pagetable_dying is the only reason to keep PARAVIRT for
> >>>>>>>> HVM
> >>>>>>>> guests, then I agree with you that we should remove it.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> - PV IPIs are not needed on single-vcpu guests
> >>>>>>>>>
> >>>>>>>>> - PARAVIRT_CLOCK doesn't need PARAVIRT (in fact the SUSEs kernel
> >>>>>>>>> configs
> >>>>>>>>>       for all x86_64 kernels have CONFIG_PARAVIRT_CLOCK=y)
> >>>>>>>>>
> >>>>>>>>> So I think we really should enable building Xen frontends without
> >>>>>>>>> PARAVIRT, implying at least no XEN_PV and no XEN_PVH.
> >>>>>>>>>
> >>>>>>>>> I'll have a try setting up patches.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> If we are doing this as a performance improvement, I would like to
> >>>>>>>> see a
> >>>>>>>> couple of benchmarks (kernbench, hackbench) to show that on a
> >>>>>>>> single-vcpu guest and multi-vcpu guest (let's say 4 vcpus) disabling
> >>>>>>>> PARAVIRT leads to better performance on Xen on EPT hardware.
> >>>>>>>
> >>>>>>>
> >>>>>>> This is not meant to be a performance improvement. It is meant to
> >>>>>>> enable
> >>>>>>> a standard distro kernel configured without PARAVIRT to be able to
> >>>>>>> run
> >>>>>>> as a HVM guest using the pv-drivers.
> >>>>>>
> >>>>>>
> >>>>>> This is not a convincing explanation.  Debian, Ubuntu and Fedora seems
> >>>>>> to be able to cope with it just fine.
> >>>>>>
> >>>>>> Why do you want to do that, even though it will cause a performance
> >>>>>> regression and a maintenance pain?  You haven't provided a reason yet.
> >>>>>>
> >>>>>
> >>>>> Either we are talking about different things, or I really don't
> >>>>> understand your problem here. I don't want to disable something. I
> >>>>> just want to enable kernels without PARAVIRT to run under Xen better
> >>>>> than today. Being it 32 bit non-PAE kernels as Ian pointed out or
> >>>>> distro kernels like e.g. SLES and probably RHEL.
> >>>>>
> >>>>> Using PV frontends is completely orthogonal to other PV enhancements
> >>>>> like PARAVIRT_CLOCK, HVMOP_pagetable_dying or PV IPIs. So why do you
> >>>>> object enabling the PV frontends for those kernels?
> >>>>
> >>>>
> >>>> I am for it.  I would like to avoid two user visible XEN enablement
> >>>> options (XEN_FRONTEND vs. XEN_PVHVM) for x86_64 and PAE HVM guests to
> >>>> avoid configurations with just XEN_FRONTEND, that can be considered a
> >>>> performance regression compared to what we have now (on x86_64 and PAE).
> >>>
> >>>
> >>> Would you be okay with making this an expert configuration alternative
> >>> for PAE/x86_64? This would enable the possibility to use PV drivers for
> >>> native-performance-tuned kernels. I would explicitly mention the better
> >>> alternative XEN_PVHVM in the Kconfig help text.
> >>
> >>
> >> I would prefer to hide it on PAE and x86_64.
> >
> >
> > Okay, as long as it is still _possible_ somehow to configure it.
> 
> That begs the question, all this just for 32-bit non-PAE ?

There was another reason. Some distros remove the CONFIG_XEN_DOM0 altogether
even thought they do enable the rest of the pieces (backends, frontends, etc).

Which begs the question - why do we care about DOM0 at all.

What we care about is drivers - either frontend or backend. If we want
backends and we want PV - then we want to build an kernel that can boot as
a normal PV or as an dom0 PV.

Ditto for HVM - if you want to build an kernel that won't do PV but
can do backends - we should be able to do that.

Or PVH  - we want an domain that can be an backend (or frontend).

That does mean the "PV" gets broken down further to be concrete
pieces and have nothing to do with drivers. 

The idea would be that you would just select four knobs:

 Yes/No Backend PV drivers [and maybe remove the PV part?]
 Yes/No Frontend PV drivers [and maybe remove the PV part?]
 Yes/No PV support (so utilizing the PV ABI)
 Yes/No PVH support (a stricter subset of PV ABI - with less pieces)

The HVM support would automatically be picked if the config had
the 'baremetal' type support - like IOAPIC, APIC, ACPI, etc.

So if you said Y, N, N, N, the kernel would only be able to
boot in HVM mode but still have pciback, netback, scsiback, blkback, and usbback.
(good for an device backend). And it could be an PAE or non-PAE kernel.

If you said N,Y,Y,Y then it could boot under HVM, PV, PVH, and only
have pcifront, netfront, scsifront, blkfront, and usbfront.
(not very good for an initial domain).

And so on.

I hope I hadn't confused the matter?
> 
>  Luis
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: RFC: xen config changes v4

[Luis R. Rodriguez]

On Fri, Feb 27, 2015 at 07:14:32AM +0100, Juergen Gross wrote:
> On 02/26/2015 07:48 PM, Luis R. Rodriguez wrote:
>> On Thu, Feb 26, 2015 at 05:42:57PM +0000, Stefano Stabellini wrote:
>>> On Thu, 26 Feb 2015, Luis R. Rodriguez wrote:
>>>> On Thu, Feb 26, 2015 at 11:08:20AM +0000, Stefano Stabellini wrote:
>>>>> On Thu, 26 Feb 2015, David Vrabel wrote:
>>>>>> On 26/02/15 04:59, Juergen Gross wrote:
>>>>>>>
>>>>>>> So we are again in the situation that pv-drivers always imply the pvops
>>>>>>> kernel (PARAVIRT selected). I started the whole Kconfig rework to
>>>>>>> eliminate this dependency.
>>>>>>
>>>>>> Yes.  Can you produce a series that just addresses this one issue.
>>>>>>
>>>>>> In the absence of any concrete requirement for this big Kconfig reorg I
>>>>>> I don't think it is helpful.
>>>>>
>>>>> I clearly missed some context as I didn't realize that this was the
>>>>> intended goal. Why do we want this? Please explain as it won't come
>>>>> for free.
>>>>>
>>>>>
>>>>> We have a few PV interfaces for HVM guests that need PARAVIRT in Linux
>>>>> in order to be used, for example pv_time_ops and HVMOP_pagetable_dying.
>>>>> They are critical performance improvements and from the interface
>>>>> perspective, small enough that doesn't make much sense having a separate
>>>>> KConfig option for them.
>>>>>
>>>>>
>>>>> In order to reach the goal above we necessarily need to introduce a
>>>>> differentiation in terms of PV on HVM guests in Linux:
>>>>>
>>>>> 1) basic guests with PV network, disk, etc but no PV timers, no
>>>>>     HVMOP_pagetable_dying, no PV IPIs
>>>>> 2) full PV on HVM guests that have PV network, disk, timers,
>>>>>     HVMOP_pagetable_dying, PV IPIs and anything else that makes sense.
>>>>>
>>>>> 2) is much faster than 1) on Xen and 2) is only a tiny bit slower than
>>>>> 1) on native x86
>>>>
>>>> Also don't we shove 2) down hvm guests right now? Even when everything is
>>>> built in I do not see how we opt out for HVM for 1) at run time right now.
>>>>
>>>> If this is true then the question of motivation for this becomes even
>>>> stronger I think.
>>>
>>> Yes, indeed there is no way to do 1) at the moment. And for good
>>> reasons, see above.
>>
>> OK if the goal is to be able to build front end drivers by avoiding building
>> PARAVIRT / PARAVIRT_CLOCK and if the gains to be able to do so (which haven't
>> been stated other than just the ability to do so) are small (as Stefano notes
>> simple hvm containers do not perform great) but requires a bit of work, I'd
>> rather ask -- why not address *why* we are avoiding PARAVIRT /
>> PARAVIRT_CLOCK and stick to the original goals behind the pvops model by
>> addressing what is required to be able to continue to be happy with one single
>> kernel. The work required to do that might be more than to just be able to
>> build simple Xen hvm containers without PARAVIRT / PARAVIRT_CLOCK  but I'd
>> think the gains would be much higher.
>
> I absolutely agree. I think this is a long term goal we should work on.
> PVH should address most of the issues, BTW.
>
>> If this resonates well then I'd like to ask: what are the current most pressing
>> issues with enabling PARAVIRT / PARAVIRT_CLOCK.
>
> PARAVIRT: performance, especially memory management

Do we have studies on specific areas? I'd be very interested in the exact routines.

> PARAVIRT_CLOCK: none

Great!

 Luis

Re: RFC: xen config changes v4

[Stefano Stabellini]

On Fri, 27 Feb 2015, Konrad Rzeszutek Wilk wrote:
> On Fri, Feb 27, 2015 at 09:53:46AM -0800, Luis R. Rodriguez wrote:
> > On Fri, Feb 27, 2015 at 6:30 AM, Juergen Gross <jgross@suse.com> wrote:
> > > On 02/27/2015 02:38 PM, Stefano Stabellini wrote:
> > >>
> > >> On Fri, 27 Feb 2015, Juergen Gross wrote:
> > >>>
> > >>> On 02/27/2015 01:24 PM, Stefano Stabellini wrote:
> > >>>>
> > >>>> On Fri, 27 Feb 2015, Juergen Gross wrote:
> > >>>>>
> > >>>>> On 02/27/2015 11:11 AM, Stefano Stabellini wrote:
> > >>>>>>
> > >>>>>> On Fri, 27 Feb 2015, Juergen Gross wrote:
> > >>>>>>>
> > >>>>>>> On 02/27/2015 10:41 AM, Stefano Stabellini wrote:
> > >>>>>>>>
> > >>>>>>>> On Fri, 27 Feb 2015, Juergen Gross wrote:
> > >>>>>>>>>
> > >>>>>>>>> On 02/26/2015 06:42 PM, Stefano Stabellini wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> On Thu, 26 Feb 2015, Luis R. Rodriguez wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Thu, Feb 26, 2015 at 11:08:20AM +0000, Stefano Stabellini
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Thu, 26 Feb 2015, David Vrabel wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On 26/02/15 04:59, Juergen Gross wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> So we are again in the situation that pv-drivers always
> > >>>>>>>>>>>>>> imply
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>> pvops
> > >>>>>>>>>>>>>> kernel (PARAVIRT selected). I started the whole Kconfig
> > >>>>>>>>>>>>>> rework
> > >>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>> eliminate this dependency.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Yes.  Can you produce a series that just addresses this
> > >>>>>>>>>>>>> one
> > >>>>>>>>>>>>> issue.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> In the absence of any concrete requirement for this big
> > >>>>>>>>>>>>> Kconfig
> > >>>>>>>>>>>>> reorg
> > >>>>>>>>>>>>> I
> > >>>>>>>>>>>>> I don't think it is helpful.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I clearly missed some context as I didn't realize that this
> > >>>>>>>>>>>> was
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>> intended goal. Why do we want this? Please explain as it
> > >>>>>>>>>>>> won't
> > >>>>>>>>>>>> come
> > >>>>>>>>>>>> for free.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> We have a few PV interfaces for HVM guests that need
> > >>>>>>>>>>>> PARAVIRT in
> > >>>>>>>>>>>> Linux
> > >>>>>>>>>>>> in order to be used, for example pv_time_ops and
> > >>>>>>>>>>>> HVMOP_pagetable_dying.
> > >>>>>>>>>>>> They are critical performance improvements and from the
> > >>>>>>>>>>>> interface
> > >>>>>>>>>>>> perspective, small enough that doesn't make much sense
> > >>>>>>>>>>>> having a
> > >>>>>>>>>>>> separate
> > >>>>>>>>>>>> KConfig option for them.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> In order to reach the goal above we necessarily need to
> > >>>>>>>>>>>> introduce a
> > >>>>>>>>>>>> differentiation in terms of PV on HVM guests in Linux:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1) basic guests with PV network, disk, etc but no PV timers,
> > >>>>>>>>>>>> no
> > >>>>>>>>>>>>         HVMOP_pagetable_dying, no PV IPIs
> > >>>>>>>>>>>> 2) full PV on HVM guests that have PV network, disk, timers,
> > >>>>>>>>>>>>         HVMOP_pagetable_dying, PV IPIs and anything else that
> > >>>>>>>>>>>> makes
> > >>>>>>>>>>>> sense.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 2) is much faster than 1) on Xen and 2) is only a tiny bit
> > >>>>>>>>>>>> slower
> > >>>>>>>>>>>> than
> > >>>>>>>>>>>> 1) on native x86
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Also don't we shove 2) down hvm guests right now? Even when
> > >>>>>>>>>>> everything
> > >>>>>>>>>>> is
> > >>>>>>>>>>> built in I do not see how we opt out for HVM for 1) at run
> > >>>>>>>>>>> time
> > >>>>>>>>>>> right
> > >>>>>>>>>>> now.
> > >>>>>>>>>>>
> > >>>>>>>>>>> If this is true then the question of motivation for this
> > >>>>>>>>>>> becomes
> > >>>>>>>>>>> even
> > >>>>>>>>>>> stronger I think.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Yes, indeed there is no way to do 1) at the moment. And for good
> > >>>>>>>>>> reasons, see above.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Hmm, after checking the code I'm not convinced:
> > >>>>>>>>>
> > >>>>>>>>> - HVMOP_pagetable_dying is obsolete on modern hardware supporting
> > >>>>>>>>>       EPT/HAP
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> That might be true, but what about older hardware?
> > >>>>>>>> Even on modern hardware a few workloads still run faster on shadow.
> > >>>>>>>> But if HVMOP_pagetable_dying is the only reason to keep PARAVIRT for
> > >>>>>>>> HVM
> > >>>>>>>> guests, then I agree with you that we should remove it.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> - PV IPIs are not needed on single-vcpu guests
> > >>>>>>>>>
> > >>>>>>>>> - PARAVIRT_CLOCK doesn't need PARAVIRT (in fact the SUSEs kernel
> > >>>>>>>>> configs
> > >>>>>>>>>       for all x86_64 kernels have CONFIG_PARAVIRT_CLOCK=y)
> > >>>>>>>>>
> > >>>>>>>>> So I think we really should enable building Xen frontends without
> > >>>>>>>>> PARAVIRT, implying at least no XEN_PV and no XEN_PVH.
> > >>>>>>>>>
> > >>>>>>>>> I'll have a try setting up patches.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> If we are doing this as a performance improvement, I would like to
> > >>>>>>>> see a
> > >>>>>>>> couple of benchmarks (kernbench, hackbench) to show that on a
> > >>>>>>>> single-vcpu guest and multi-vcpu guest (let's say 4 vcpus) disabling
> > >>>>>>>> PARAVIRT leads to better performance on Xen on EPT hardware.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> This is not meant to be a performance improvement. It is meant to
> > >>>>>>> enable
> > >>>>>>> a standard distro kernel configured without PARAVIRT to be able to
> > >>>>>>> run
> > >>>>>>> as a HVM guest using the pv-drivers.
> > >>>>>>
> > >>>>>>
> > >>>>>> This is not a convincing explanation.  Debian, Ubuntu and Fedora seems
> > >>>>>> to be able to cope with it just fine.
> > >>>>>>
> > >>>>>> Why do you want to do that, even though it will cause a performance
> > >>>>>> regression and a maintenance pain?  You haven't provided a reason yet.
> > >>>>>>
> > >>>>>
> > >>>>> Either we are talking about different things, or I really don't
> > >>>>> understand your problem here. I don't want to disable something. I
> > >>>>> just want to enable kernels without PARAVIRT to run under Xen better
> > >>>>> than today. Being it 32 bit non-PAE kernels as Ian pointed out or
> > >>>>> distro kernels like e.g. SLES and probably RHEL.
> > >>>>>
> > >>>>> Using PV frontends is completely orthogonal to other PV enhancements
> > >>>>> like PARAVIRT_CLOCK, HVMOP_pagetable_dying or PV IPIs. So why do you
> > >>>>> object enabling the PV frontends for those kernels?
> > >>>>
> > >>>>
> > >>>> I am for it.  I would like to avoid two user visible XEN enablement
> > >>>> options (XEN_FRONTEND vs. XEN_PVHVM) for x86_64 and PAE HVM guests to
> > >>>> avoid configurations with just XEN_FRONTEND, that can be considered a
> > >>>> performance regression compared to what we have now (on x86_64 and PAE).
> > >>>
> > >>>
> > >>> Would you be okay with making this an expert configuration alternative
> > >>> for PAE/x86_64? This would enable the possibility to use PV drivers for
> > >>> native-performance-tuned kernels. I would explicitly mention the better
> > >>> alternative XEN_PVHVM in the Kconfig help text.
> > >>
> > >>
> > >> I would prefer to hide it on PAE and x86_64.
> > >
> > >
> > > Okay, as long as it is still _possible_ somehow to configure it.
> > 
> > That begs the question, all this just for 32-bit non-PAE ?
> 
> There was another reason. Some distros remove the CONFIG_XEN_DOM0 altogether
> even thought they do enable the rest of the pieces (backends, frontends, etc).
> 
> Which begs the question - why do we care about DOM0 at all.
> 
> What we care about is drivers - either frontend or backend. If we want
> backends and we want PV - then we want to build an kernel that can boot as
> a normal PV or as an dom0 PV.
> 
> Ditto for HVM - if you want to build an kernel that won't do PV but
> can do backends - we should be able to do that.
> 
> Or PVH  - we want an domain that can be an backend (or frontend).
> 
> That does mean the "PV" gets broken down further to be concrete
> pieces and have nothing to do with drivers. 
> 
> The idea would be that you would just select four knobs:
> 
>  Yes/No Backend PV drivers [and maybe remove the PV part?]
>  Yes/No Frontend PV drivers [and maybe remove the PV part?]
>  Yes/No PV support (so utilizing the PV ABI)
>  Yes/No PVH support (a stricter subset of PV ABI - with less pieces)
> 
> The HVM support would automatically be picked if the config had
> the 'baremetal' type support - like IOAPIC, APIC, ACPI, etc.
> 
> So if you said Y, N, N, N, the kernel would only be able to
> boot in HVM mode but still have pciback, netback, scsiback, blkback, and usbback.
> (good for an device backend). And it could be an PAE or non-PAE kernel.
> 
> If you said N,Y,Y,Y then it could boot under HVM, PV, PVH, and only
> have pcifront, netfront, scsifront, blkfront, and usbfront.
> (not very good for an initial domain).
> 
> And so on.

It makes sense.


> I hope I hadn't confused the matter?

Nope, I think it clarifies things, thanks.

In this context the issue we were discussing is what to do with the
other PV interfaces for PV on HVM guests, such as HVMOP_pagetable_dying.
I think it would be natural to enable them when Frontend PV drivers are
enabled, without any additional Kconfig options.

Re: RFC: xen config changes v4

[Konrad Rzeszutek Wilk]

> > > >> I would prefer to hide it on PAE and x86_64.
> > > >
> > > >
> > > > Okay, as long as it is still _possible_ somehow to configure it.
> > > 
> > > That begs the question, all this just for 32-bit non-PAE ?
> > 
> > There was another reason. Some distros remove the CONFIG_XEN_DOM0 altogether
> > even thought they do enable the rest of the pieces (backends, frontends, etc).
> > 
> > Which begs the question - why do we care about DOM0 at all.
> > 
> > What we care about is drivers - either frontend or backend. If we want
> > backends and we want PV - then we want to build an kernel that can boot as
> > a normal PV or as an dom0 PV.
> > 
> > Ditto for HVM - if you want to build an kernel that won't do PV but
> > can do backends - we should be able to do that.
> > 
> > Or PVH  - we want an domain that can be an backend (or frontend).
> > 
> > That does mean the "PV" gets broken down further to be concrete
> > pieces and have nothing to do with drivers. 
> > 
> > The idea would be that you would just select four knobs:
> > 
> >  Yes/No Backend PV drivers [and maybe remove the PV part?]
> >  Yes/No Frontend PV drivers [and maybe remove the PV part?]
> >  Yes/No PV support (so utilizing the PV ABI)
> >  Yes/No PVH support (a stricter subset of PV ABI - with less pieces)
> > 
> > The HVM support would automatically be picked if the config had
> > the 'baremetal' type support - like IOAPIC, APIC, ACPI, etc.
> > 
> > So if you said Y, N, N, N, the kernel would only be able to
> > boot in HVM mode but still have pciback, netback, scsiback, blkback, and usbback.
> > (good for an device backend). And it could be an PAE or non-PAE kernel.
> > 
> > If you said N,Y,Y,Y then it could boot under HVM, PV, PVH, and only
> > have pcifront, netfront, scsifront, blkfront, and usbfront.
> > (not very good for an initial domain).
> > 
> > And so on.
> 
> It makes sense.
> 
> 
> > I hope I hadn't confused the matter?
> 
> Nope, I think it clarifies things, thanks.

Thought it does mean that it would add more #ifdery or cleanups
to the existing drivers so that they can be compiled under
different platforms without any assumptions.

> 
> In this context the issue we were discussing is what to do with the
> other PV interfaces for PV on HVM guests, such as HVMOP_pagetable_dying.
> I think it would be natural to enable them when Frontend PV drivers are
> enabled, without any additional Kconfig options.

I would put this in 'Enlightment support for Xen' - which would be
the basic foundation to make any kernel work under Xen. This would
pull in some _infrastructure_. Regardless of it being a backend
or frontend we need grant ops,event channels, support for migration.

Perhaps add that as a new option under CONFIG_HYPERVISOR. And not
depend on CONFIG_PARAVIRT - as in theory you can have an non-PARAVIRT HVM
guest running with PV drivers (I haven't tried it, but I would think
it can be done?)

Regarding the 'HVMOP_pagetable_drying' - If it is part of foundation
'enlightment for Xen' - then it would be folded in. If it is not, but
the platform looks to be a non-PV kernel (APIC, ACPI, IOPAPIC, MSI,
PCI, etc) then it would be automatically enabled.

BTW, when I think PV kernel - it is an non-APIC, non-ACPI, non.. a lot
of stuff. I did build one like that way back for 3.0 and it was quite
slim. lHm, maybe we should even provide an 'defconfig' just to make sure
we can test this kind of build?

Luis, sorry for hijacking this thread and expanding the scope of this work!

I think it would fantastical to make this work and would help a lot in
the future - but right now it is a bit of complex riddle to untangle!

Re: RFC: xen config changes v4

[Stefano Stabellini]

On Mon, 2 Mar 2015, Konrad Rzeszutek Wilk wrote:
> > > > >> I would prefer to hide it on PAE and x86_64.
> > > > >
> > > > >
> > > > > Okay, as long as it is still _possible_ somehow to configure it.
> > > > 
> > > > That begs the question, all this just for 32-bit non-PAE ?
> > > 
> > > There was another reason. Some distros remove the CONFIG_XEN_DOM0 altogether
> > > even thought they do enable the rest of the pieces (backends, frontends, etc).
> > > 
> > > Which begs the question - why do we care about DOM0 at all.
> > > 
> > > What we care about is drivers - either frontend or backend. If we want
> > > backends and we want PV - then we want to build an kernel that can boot as
> > > a normal PV or as an dom0 PV.
> > > 
> > > Ditto for HVM - if you want to build an kernel that won't do PV but
> > > can do backends - we should be able to do that.
> > > 
> > > Or PVH  - we want an domain that can be an backend (or frontend).
> > > 
> > > That does mean the "PV" gets broken down further to be concrete
> > > pieces and have nothing to do with drivers. 
> > > 
> > > The idea would be that you would just select four knobs:
> > > 
> > >  Yes/No Backend PV drivers [and maybe remove the PV part?]
> > >  Yes/No Frontend PV drivers [and maybe remove the PV part?]
> > >  Yes/No PV support (so utilizing the PV ABI)
> > >  Yes/No PVH support (a stricter subset of PV ABI - with less pieces)
> > > 
> > > The HVM support would automatically be picked if the config had
> > > the 'baremetal' type support - like IOAPIC, APIC, ACPI, etc.
> > > 
> > > So if you said Y, N, N, N, the kernel would only be able to
> > > boot in HVM mode but still have pciback, netback, scsiback, blkback, and usbback.
> > > (good for an device backend). And it could be an PAE or non-PAE kernel.
> > > 
> > > If you said N,Y,Y,Y then it could boot under HVM, PV, PVH, and only
> > > have pcifront, netfront, scsifront, blkfront, and usbfront.
> > > (not very good for an initial domain).
> > > 
> > > And so on.
> > 
> > It makes sense.
> > 
> > 
> > > I hope I hadn't confused the matter?
> > 
> > Nope, I think it clarifies things, thanks.
> 
> Thought it does mean that it would add more #ifdery or cleanups
> to the existing drivers so that they can be compiled under
> different platforms without any assumptions.
> 
> > 
> > In this context the issue we were discussing is what to do with the
> > other PV interfaces for PV on HVM guests, such as HVMOP_pagetable_dying.
> > I think it would be natural to enable them when Frontend PV drivers are
> > enabled, without any additional Kconfig options.
> 
> I would put this in 'Enlightment support for Xen' - which would be
> the basic foundation to make any kernel work under Xen. This would
> pull in some _infrastructure_. Regardless of it being a backend
> or frontend we need grant ops,event channels, support for migration.

Sounds good.


> Perhaps add that as a new option under CONFIG_HYPERVISOR. And not
> depend on CONFIG_PARAVIRT - as in theory you can have an non-PARAVIRT HVM
> guest running with PV drivers (I haven't tried it, but I would think
> it can be done?)

That is the bit I disagree with: not depending on CONFIG_PARAVIRT means
that many good optimizations for PV on HVM guests won't be enabled.
Doing so will cause a performance regression compared to the PV on HVM
guests of today.

Most x86_64 distro kernels have CONFIG_PARAVIRT and CONFIG_XEN, I
wouldn't want them to switch to CONFIG_XEN_FRONTEND without
CONFIG_PARAVIRT in the future as a consequence of this work.


> Regarding the 'HVMOP_pagetable_drying' - If it is part of foundation
> 'enlightment for Xen' - then it would be folded in. If it is not, but
> the platform looks to be a non-PV kernel (APIC, ACPI, IOPAPIC, MSI,
> PCI, etc) then it would be automatically enabled.
> 
> BTW, when I think PV kernel - it is an non-APIC, non-ACPI, non.. a lot
> of stuff. I did build one like that way back for 3.0 and it was quite
> slim. lHm, maybe we should even provide an 'defconfig' just to make sure
> we can test this kind of build?
> 
> Luis, sorry for hijacking this thread and expanding the scope of this work!
> 
> I think it would fantastical to make this work and would help a lot in
> the future - but right now it is a bit of complex riddle to untangle!

Re: RFC: xen config changes v4

[Konrad Rzeszutek Wilk]

On Mon, Mar 02, 2015 at 05:07:00PM +0000, Stefano Stabellini wrote:
> On Mon, 2 Mar 2015, Konrad Rzeszutek Wilk wrote:
> > > > > >> I would prefer to hide it on PAE and x86_64.
> > > > > >
> > > > > >
> > > > > > Okay, as long as it is still _possible_ somehow to configure it.
> > > > > 
> > > > > That begs the question, all this just for 32-bit non-PAE ?
> > > > 
> > > > There was another reason. Some distros remove the CONFIG_XEN_DOM0 altogether
> > > > even thought they do enable the rest of the pieces (backends, frontends, etc).
> > > > 
> > > > Which begs the question - why do we care about DOM0 at all.
> > > > 
> > > > What we care about is drivers - either frontend or backend. If we want
> > > > backends and we want PV - then we want to build an kernel that can boot as
> > > > a normal PV or as an dom0 PV.
> > > > 
> > > > Ditto for HVM - if you want to build an kernel that won't do PV but
> > > > can do backends - we should be able to do that.
> > > > 
> > > > Or PVH  - we want an domain that can be an backend (or frontend).
> > > > 
> > > > That does mean the "PV" gets broken down further to be concrete
> > > > pieces and have nothing to do with drivers. 
> > > > 
> > > > The idea would be that you would just select four knobs:
> > > > 
> > > >  Yes/No Backend PV drivers [and maybe remove the PV part?]
> > > >  Yes/No Frontend PV drivers [and maybe remove the PV part?]
> > > >  Yes/No PV support (so utilizing the PV ABI)
> > > >  Yes/No PVH support (a stricter subset of PV ABI - with less pieces)
> > > > 
> > > > The HVM support would automatically be picked if the config had
> > > > the 'baremetal' type support - like IOAPIC, APIC, ACPI, etc.
> > > > 
> > > > So if you said Y, N, N, N, the kernel would only be able to
> > > > boot in HVM mode but still have pciback, netback, scsiback, blkback, and usbback.
> > > > (good for an device backend). And it could be an PAE or non-PAE kernel.
> > > > 
> > > > If you said N,Y,Y,Y then it could boot under HVM, PV, PVH, and only
> > > > have pcifront, netfront, scsifront, blkfront, and usbfront.
> > > > (not very good for an initial domain).
> > > > 
> > > > And so on.
> > > 
> > > It makes sense.
> > > 
> > > 
> > > > I hope I hadn't confused the matter?
> > > 
> > > Nope, I think it clarifies things, thanks.
> > 
> > Thought it does mean that it would add more #ifdery or cleanups
> > to the existing drivers so that they can be compiled under
> > different platforms without any assumptions.
> > 
> > > 
> > > In this context the issue we were discussing is what to do with the
> > > other PV interfaces for PV on HVM guests, such as HVMOP_pagetable_dying.
> > > I think it would be natural to enable them when Frontend PV drivers are
> > > enabled, without any additional Kconfig options.
> > 
> > I would put this in 'Enlightment support for Xen' - which would be
> > the basic foundation to make any kernel work under Xen. This would
> > pull in some _infrastructure_. Regardless of it being a backend
> > or frontend we need grant ops,event channels, support for migration.
> 
> Sounds good.
> 
> 
> > Perhaps add that as a new option under CONFIG_HYPERVISOR. And not
> > depend on CONFIG_PARAVIRT - as in theory you can have an non-PARAVIRT HVM
> > guest running with PV drivers (I haven't tried it, but I would think
> > it can be done?)
> 
> That is the bit I disagree with: not depending on CONFIG_PARAVIRT means
> that many good optimizations for PV on HVM guests won't be enabled.
> Doing so will cause a performance regression compared to the PV on HVM
> guests of today.
> 
> Most x86_64 distro kernels have CONFIG_PARAVIRT and CONFIG_XEN, I
> wouldn't want them to switch to CONFIG_XEN_FRONTEND without
> CONFIG_PARAVIRT in the future as a consequence of this work.

The CONFIG_PARAVIRT is a giant umbrella for everything 'virt' nowadays.
It has runtime patches for the low-level functions, clock functions,
spinlocks, etc.

Unraveling that to be simpler (clock functions can be in their own
indirect structure that would be wrapped by CONFIG_VIRT_CLOCK for example?)
would be nice but an headache to unravel.

Hence your point is valid and correct - we need to select CONFIG_PARAVIRT - even
if we just end up using an fraction of it. The unravelling of
CONFIG_PARAVIRT can be done in the future.

> 
> 
> > Regarding the 'HVMOP_pagetable_drying' - If it is part of foundation
> > 'enlightment for Xen' - then it would be folded in. If it is not, but
> > the platform looks to be a non-PV kernel (APIC, ACPI, IOPAPIC, MSI,
> > PCI, etc) then it would be automatically enabled.
> > 
> > BTW, when I think PV kernel - it is an non-APIC, non-ACPI, non.. a lot
> > of stuff. I did build one like that way back for 3.0 and it was quite
> > slim. lHm, maybe we should even provide an 'defconfig' just to make sure
> > we can test this kind of build?
> > 
> > Luis, sorry for hijacking this thread and expanding the scope of this work!
> > 
> > I think it would fantastical to make this work and would help a lot in
> > the future - but right now it is a bit of complex riddle to untangle!
>  

Re: RFC: xen config changes v4

[Luis R. Rodriguez]

On Fri, Feb 27, 2015 at 10:27 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> There was another reason. Some distros remove the CONFIG_XEN_DOM0 altogether
> even thought they do enable the rest of the pieces (backends, frontends, etc).

Interesting, what distros do this? Also when one does not have dom0
what other forms are there to use Linux Xen backend drivers on GPL
kernels (because obviously non-GPL kernels are not an option)?

> Which begs the question - why do we care about DOM0 at all.

For the distros that do care about dom0? Not sure what those..

> What we care about is drivers - either frontend or backend. If we want
> backends and we want PV - then we want to build an kernel that can boot as
> a normal PV or as an dom0 PV.

Given we only have non-PAE 32-bit builds (big whoop) to be useful for
non-PV kernels I wonder how many distros are disabling PARAVIRT and if
they do are they going to be putting out PARAVIRT kernels specifically
just for Xen use? If we want traction on Xen we should make it easier
to manage Xen and that should be targeting to fold under one kernel.

> Ditto for HVM - if you want to build an kernel that won't do PV but
> can do backends - we should be able to do that.

Sure but if performance is brutally dumb -- and if this is legacy
stuff why care so much about it (32-bit non-PAE)?

> Or PVH  - we want an domain that can be an backend (or frontend).
>
> That does mean the "PV" gets broken down further to be concrete
> pieces and have nothing to do with drivers.

I was originally echoing this but some folks seem to not believe this
was required, it was only after illustrating the build issues without
PARAVIRT and Stefano jumping in that this is now more or less an
accepted build issue and something we'd need to consider if we do or
not. Your points still don't make a big case for the effort enabling
the thin Xen HVM solutions other than just having them available.

> The idea would be that you would just select four knobs:
>
>  Yes/No Backend PV drivers [and maybe remove the PV part?]

OK this is a new topic and I did not explore to what extend one can
have backend alone without PV.

>  Yes/No Frontend PV drivers [and maybe remove the PV part?]
>  Yes/No PV support (so utilizing the PV ABI)
>  Yes/No PVH support (a stricter subset of PV ABI - with less pieces)

Sure.

> The HVM support would automatically be picked if the config had
> the 'baremetal' type support - like IOAPIC, APIC, ACPI, etc.
>
> So if you said Y, N, N, N, the kernel would only be able to
> boot in HVM mode but still have pciback, netback, scsiback, blkback, and usbback.
> (good for an device backend). And it could be an PAE or non-PAE kernel.

But for 64-bit systems Stefano pointed out this would be silly to do.
The only good use case combination expressed for this so far would be
32-bit non-PAE systems no?

> If you said N,Y,Y,Y then it could boot under HVM, PV, PVH, and only
> have pcifront, netfront, scsifront, blkfront, and usbfront.
> (not very good for an initial domain).

Sure, thin HVM only if it disabled the PV part, maybe a boot param for
it. Again why the trouble?

> And so on.
>
> I hope I hadn't confused the matter?

Correct me if I'm wrong but I only see added to the table here was the
idea of clearing backend driver support also from the PV ties -- and
clarifying that this is all just about the big matrix of all possible
options and having the flexibility to let folks pick and choose. The
challenge I believe is figuring out what options in the matrix we
don't really care to enable or support, for instance it was noted that
distros didn't know what to do with XEN_PVHVM when exposed to them, so
there might be some things to consider also in this big matrix. Since
PVH is the long term, and I'm in no way a kernel hoarder, I'd advocate
only enabling things to let old PV to be compartmentalized well now
and later be ripped out. Personally that's the only reason why I'd
even bother with enabling thin HVM guests / backends / frontends, but
you guys would know better what path to take here.

In any case Juergen has expressed interest to taking this on again so
I'll go focus on some other items at the moment.

  Luis

Re: RFC: xen config changes v4

[Luis R. Rodriguez]

On Mon, Mar 02, 2015 at 11:07:02AM -0500, Konrad Rzeszutek Wilk wrote:
> > > > >> I would prefer to hide it on PAE and x86_64.
> > > > >
> > > > >
> > > > > Okay, as long as it is still _possible_ somehow to configure it.
> > > > 
> > > > That begs the question, all this just for 32-bit non-PAE ?
> > > 
> > > There was another reason. Some distros remove the CONFIG_XEN_DOM0 altogether
> > > even thought they do enable the rest of the pieces (backends, frontends, etc).
> > > 
> > > Which begs the question - why do we care about DOM0 at all.
> > > 
> > > What we care about is drivers - either frontend or backend. If we want
> > > backends and we want PV - then we want to build an kernel that can boot as
> > > a normal PV or as an dom0 PV.
> > > 
> > > Ditto for HVM - if you want to build an kernel that won't do PV but
> > > can do backends - we should be able to do that.
> > > 
> > > Or PVH  - we want an domain that can be an backend (or frontend).
> > > 
> > > That does mean the "PV" gets broken down further to be concrete
> > > pieces and have nothing to do with drivers. 
> > > 
> > > The idea would be that you would just select four knobs:
> > > 
> > >  Yes/No Backend PV drivers [and maybe remove the PV part?]
> > >  Yes/No Frontend PV drivers [and maybe remove the PV part?]
> > >  Yes/No PV support (so utilizing the PV ABI)
> > >  Yes/No PVH support (a stricter subset of PV ABI - with less pieces)
> > > 
> > > The HVM support would automatically be picked if the config had
> > > the 'baremetal' type support - like IOAPIC, APIC, ACPI, etc.
> > > 
> > > So if you said Y, N, N, N, the kernel would only be able to
> > > boot in HVM mode but still have pciback, netback, scsiback, blkback, and usbback.
> > > (good for an device backend). And it could be an PAE or non-PAE kernel.
> > > 
> > > If you said N,Y,Y,Y then it could boot under HVM, PV, PVH, and only
> > > have pcifront, netfront, scsifront, blkfront, and usbfront.
> > > (not very good for an initial domain).
> > > 
> > > And so on.
> > 
> > It makes sense.
> > 
> > 
> > > I hope I hadn't confused the matter?
> > 
> > Nope, I think it clarifies things, thanks.
> 
> Thought it does mean that it would add more #ifdery or cleanups
> to the existing drivers so that they can be compiled under
> different platforms without any assumptions.
> 
> > 
> > In this context the issue we were discussing is what to do with the
> > other PV interfaces for PV on HVM guests, such as HVMOP_pagetable_dying.
> > I think it would be natural to enable them when Frontend PV drivers are
> > enabled, without any additional Kconfig options.
> 
> I would put this in 'Enlightment support for Xen' - which would be
> the basic foundation to make any kernel work under Xen. This would
> pull in some _infrastructure_. Regardless of it being a backend
> or frontend we need grant ops,event channels, support for migration.
>
> 
> Perhaps add that as a new option under CONFIG_HYPERVISOR. And not
> depend on CONFIG_PARAVIRT - as in theory you can have an non-PARAVIRT HVM
> guest running with PV drivers (I haven't tried it, but I would think
> it can be done?)

It was noted that that it was decided to hide XEN_PVHVM from being user
selectable, if so since thin HVM support hasn't been supported before
and since PVH is the way of the future -- why even bother enabling it?
It'd just add the option of a new type of guest, perhaps confuse things
more and then make it harder to remove support for if we wanted to.

> Regarding the 'HVMOP_pagetable_drying' - If it is part of foundation
> 'enlightment for Xen' - then it would be folded in. If it is not, but
> the platform looks to be a non-PV kernel (APIC, ACPI, IOPAPIC, MSI,
> PCI, etc) then it would be automatically enabled.
> 
> BTW, when I think PV kernel - it is an non-APIC, non-ACPI, non.. a lot
> of stuff. I did build one like that way back for 3.0 and it was quite
> slim. lHm, maybe we should even provide an 'defconfig' just to make sure
> we can test this kind of build?

This relates to the 'dead code' topic and BTW it is my next goal to address
while Juergen takes this Kconfig changes back on. If you are interested
in this topic I can send you my initial thoughts.

> Luis, sorry for hijacking this thread and expanding the scope of this work!

No worries, its what it is. Since Juergen is taking this on now the only
thing I'd caution against here is kernel hoarding. I see potential for it
and I was more in the hopes to get things here to be an opportunity to
pave the way for easy clean up later of things we likely do want to remove.
The removal topic probably should come to the table if we really are
serious about it.

> I think it would fantastical to make this work and would help a lot in
> the future - but right now it is a bit of complex riddle to untangle!

Since I'm new here I can only chime in here with a hope that we just
simplify things further and tie them in with more long term goals:

  * dynamic run time PVness - striving towards single build kernels
    without performance impact for non PV kernels

  * pave the way to nuke code we really are serious about removing,
    that might entail actually annotating things as obsolete now
    and to be removed later (note: 9c0ece069b32e8e1a Linus removed
    Documentation/feature-removal-schedule.txt so really its up
    to us and good judgement)

  Luis

Re: RFC: xen config changes v4

[Luis R. Rodriguez]

On Mon, Mar 02, 2015 at 12:30:02PM -0500, Konrad Rzeszutek Wilk wrote:
> On Mon, Mar 02, 2015 at 05:07:00PM +0000, Stefano Stabellini wrote:
> > On Mon, 2 Mar 2015, Konrad Rzeszutek Wilk wrote:
> > > > > > >> I would prefer to hide it on PAE and x86_64.
> > > > > > >
> > > > > > >
> > > > > > > Okay, as long as it is still _possible_ somehow to configure it.
> > > > > > 
> > > > > > That begs the question, all this just for 32-bit non-PAE ?
> > > > > 
> > > > > There was another reason. Some distros remove the CONFIG_XEN_DOM0 altogether
> > > > > even thought they do enable the rest of the pieces (backends, frontends, etc).
> > > > > 
> > > > > Which begs the question - why do we care about DOM0 at all.
> > > > > 
> > > > > What we care about is drivers - either frontend or backend. If we want
> > > > > backends and we want PV - then we want to build an kernel that can boot as
> > > > > a normal PV or as an dom0 PV.
> > > > > 
> > > > > Ditto for HVM - if you want to build an kernel that won't do PV but
> > > > > can do backends - we should be able to do that.
> > > > > 
> > > > > Or PVH  - we want an domain that can be an backend (or frontend).
> > > > > 
> > > > > That does mean the "PV" gets broken down further to be concrete
> > > > > pieces and have nothing to do with drivers. 
> > > > > 
> > > > > The idea would be that you would just select four knobs:
> > > > > 
> > > > >  Yes/No Backend PV drivers [and maybe remove the PV part?]
> > > > >  Yes/No Frontend PV drivers [and maybe remove the PV part?]
> > > > >  Yes/No PV support (so utilizing the PV ABI)
> > > > >  Yes/No PVH support (a stricter subset of PV ABI - with less pieces)
> > > > > 
> > > > > The HVM support would automatically be picked if the config had
> > > > > the 'baremetal' type support - like IOAPIC, APIC, ACPI, etc.
> > > > > 
> > > > > So if you said Y, N, N, N, the kernel would only be able to
> > > > > boot in HVM mode but still have pciback, netback, scsiback, blkback, and usbback.
> > > > > (good for an device backend). And it could be an PAE or non-PAE kernel.
> > > > > 
> > > > > If you said N,Y,Y,Y then it could boot under HVM, PV, PVH, and only
> > > > > have pcifront, netfront, scsifront, blkfront, and usbfront.
> > > > > (not very good for an initial domain).
> > > > > 
> > > > > And so on.
> > > > 
> > > > It makes sense.
> > > > 
> > > > 
> > > > > I hope I hadn't confused the matter?
> > > > 
> > > > Nope, I think it clarifies things, thanks.
> > > 
> > > Thought it does mean that it would add more #ifdery or cleanups
> > > to the existing drivers so that they can be compiled under
> > > different platforms without any assumptions.
> > > 
> > > > 
> > > > In this context the issue we were discussing is what to do with the
> > > > other PV interfaces for PV on HVM guests, such as HVMOP_pagetable_dying.
> > > > I think it would be natural to enable them when Frontend PV drivers are
> > > > enabled, without any additional Kconfig options.
> > > 
> > > I would put this in 'Enlightment support for Xen' - which would be
> > > the basic foundation to make any kernel work under Xen. This would
> > > pull in some _infrastructure_. Regardless of it being a backend
> > > or frontend we need grant ops,event channels, support for migration.
> > 
> > Sounds good.
> > 
> > 
> > > Perhaps add that as a new option under CONFIG_HYPERVISOR. And not
> > > depend on CONFIG_PARAVIRT - as in theory you can have an non-PARAVIRT HVM
> > > guest running with PV drivers (I haven't tried it, but I would think
> > > it can be done?)
> > 
> > That is the bit I disagree with: not depending on CONFIG_PARAVIRT means
> > that many good optimizations for PV on HVM guests won't be enabled.
> > Doing so will cause a performance regression compared to the PV on HVM
> > guests of today.
> > 
> > Most x86_64 distro kernels have CONFIG_PARAVIRT and CONFIG_XEN, I
> > wouldn't want them to switch to CONFIG_XEN_FRONTEND without
> > CONFIG_PARAVIRT in the future as a consequence of this work.
> 
> The CONFIG_PARAVIRT is a giant umbrella for everything 'virt' nowadays.
> It has runtime patches for the low-level functions, clock functions,
> spinlocks, etc.
> 
> Unraveling that to be simpler (clock functions can be in their own
> indirect structure that would be wrapped by CONFIG_VIRT_CLOCK for example?)
> would be nice but an headache to unravel.

IMHO this is the sort of thing that needs to be addressed for a run time
dynamic change, thereby making it less of an issue for folks to consider
to enable / disable CONFIG_PARAVIRT. The split you mentioned *should* be
addressed cleanly as well.

> Hence your point is valid and correct - we need to select CONFIG_PARAVIRT - even
> if we just end up using an fraction of it. The unravelling of
> CONFIG_PARAVIRT can be done in the future.

I think it should have been done yesterday, addressing this would make
make this discussion simply about how thin you want your kernel rather
than one way avenues as they are now.

  Luis

Re: RFC: xen config changes v4

[Juergen Gross]

On 03/02/2015 06:07 PM, Stefano Stabellini wrote:
> On Mon, 2 Mar 2015, Konrad Rzeszutek Wilk wrote:
>>>>>>> I would prefer to hide it on PAE and x86_64.
>>>>>>
>>>>>>
>>>>>> Okay, as long as it is still _possible_ somehow to configure it.
>>>>>
>>>>> That begs the question, all this just for 32-bit non-PAE ?
>>>>
>>>> There was another reason. Some distros remove the CONFIG_XEN_DOM0 altogether
>>>> even thought they do enable the rest of the pieces (backends, frontends, etc).
>>>>
>>>> Which begs the question - why do we care about DOM0 at all.
>>>>
>>>> What we care about is drivers - either frontend or backend. If we want
>>>> backends and we want PV - then we want to build an kernel that can boot as
>>>> a normal PV or as an dom0 PV.
>>>>
>>>> Ditto for HVM - if you want to build an kernel that won't do PV but
>>>> can do backends - we should be able to do that.
>>>>
>>>> Or PVH  - we want an domain that can be an backend (or frontend).
>>>>
>>>> That does mean the "PV" gets broken down further to be concrete
>>>> pieces and have nothing to do with drivers.
>>>>
>>>> The idea would be that you would just select four knobs:
>>>>
>>>>   Yes/No Backend PV drivers [and maybe remove the PV part?]
>>>>   Yes/No Frontend PV drivers [and maybe remove the PV part?]
>>>>   Yes/No PV support (so utilizing the PV ABI)
>>>>   Yes/No PVH support (a stricter subset of PV ABI - with less pieces)
>>>>
>>>> The HVM support would automatically be picked if the config had
>>>> the 'baremetal' type support - like IOAPIC, APIC, ACPI, etc.
>>>>
>>>> So if you said Y, N, N, N, the kernel would only be able to
>>>> boot in HVM mode but still have pciback, netback, scsiback, blkback, and usbback.
>>>> (good for an device backend). And it could be an PAE or non-PAE kernel.
>>>>
>>>> If you said N,Y,Y,Y then it could boot under HVM, PV, PVH, and only
>>>> have pcifront, netfront, scsifront, blkfront, and usbfront.
>>>> (not very good for an initial domain).
>>>>
>>>> And so on.
>>>
>>> It makes sense.
>>>
>>>
>>>> I hope I hadn't confused the matter?
>>>
>>> Nope, I think it clarifies things, thanks.
>>
>> Thought it does mean that it would add more #ifdery or cleanups
>> to the existing drivers so that they can be compiled under
>> different platforms without any assumptions.
>>
>>>
>>> In this context the issue we were discussing is what to do with the
>>> other PV interfaces for PV on HVM guests, such as HVMOP_pagetable_dying.
>>> I think it would be natural to enable them when Frontend PV drivers are
>>> enabled, without any additional Kconfig options.
>>
>> I would put this in 'Enlightment support for Xen' - which would be
>> the basic foundation to make any kernel work under Xen. This would
>> pull in some _infrastructure_. Regardless of it being a backend
>> or frontend we need grant ops,event channels, support for migration.
>
> Sounds good.
>
>
>> Perhaps add that as a new option under CONFIG_HYPERVISOR. And not
>> depend on CONFIG_PARAVIRT - as in theory you can have an non-PARAVIRT HVM
>> guest running with PV drivers (I haven't tried it, but I would think
>> it can be done?)
>
> That is the bit I disagree with: not depending on CONFIG_PARAVIRT means
> that many good optimizations for PV on HVM guests won't be enabled.
> Doing so will cause a performance regression compared to the PV on HVM
> guests of today.

I think we should have some (hidden) config options to enable/disable
the basic features like frontends, backends, optimizations. Those
features would then be selected by user visible config options.

We should avoid mixing technically independent options under the same
config item.


Juergen

Re: RFC: xen config changes v4

[Luis R. Rodriguez]

Hey Konrad, just a friendly reminder, the backend question for GPL
kernels interests me specially in light of recent events.

On Mon, Mar 2, 2015 at 12:39 PM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> On Fri, Feb 27, 2015 at 10:27 AM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
>> There was another reason. Some distros remove the CONFIG_XEN_DOM0 altogether
>> even thought they do enable the rest of the pieces (backends, frontends, etc).
>
> Interesting, what distros do this? Also when one does not have dom0
> what other forms are there to use Linux Xen backend drivers on GPL
> kernels (because obviously non-GPL kernels are not an option)?
>
>> Which begs the question - why do we care about DOM0 at all.
>
> For the distros that do care about dom0? Not sure what those..

Re: RFC: xen config changes v4

[Konrad Rzeszutek Wilk]

On Fri, Mar 06, 2015 at 09:17:36AM -0800, Luis R. Rodriguez wrote:
> Hey Konrad, just a friendly reminder, the backend question for GPL
> kernels interests me specially in light of recent events.

GPL kernels? Aren't all Linux kernels GPL?

> 
> On Mon, Mar 2, 2015 at 12:39 PM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> > On Fri, Feb 27, 2015 at 10:27 AM, Konrad Rzeszutek Wilk
> > <konrad.wilk@oracle.com> wrote:
> >> There was another reason. Some distros remove the CONFIG_XEN_DOM0 altogether
> >> even thought they do enable the rest of the pieces (backends, frontends, etc).
> >
> > Interesting, what distros do this? Also when one does not have dom0
> > what other forms are there to use Linux Xen backend drivers on GPL
> > kernels (because obviously non-GPL kernels are not an option)?

FreeBSD (PVH - in progress) or NetBSD can run as initial domains.

The distro is Red Hat.

> >
> >> Which begs the question - why do we care about DOM0 at all.
> >
> > For the distros that do care about dom0? Not sure what those..

Re: RFC: xen config changes v4

[Luis R. Rodriguez]

On Fri, Mar 06, 2015 at 01:02:03PM -0500, Konrad Rzeszutek Wilk wrote:
> On Fri, Mar 06, 2015 at 09:17:36AM -0800, Luis R. Rodriguez wrote:
> > Hey Konrad, just a friendly reminder, the backend question for GPL
> > kernels interests me specially in light of recent events.
> 
> GPL kernels? Aren't all Linux kernels GPL?

I asked given some in theory some folks think they can use Linux
drivers on non GPL platforms.

> > On Mon, Mar 2, 2015 at 12:39 PM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> > > On Fri, Feb 27, 2015 at 10:27 AM, Konrad Rzeszutek Wilk
> > > <konrad.wilk@oracle.com> wrote:
> > >> There was another reason. Some distros remove the CONFIG_XEN_DOM0 altogether
> > >> even thought they do enable the rest of the pieces (backends, frontends, etc).
> > >
> > > Interesting, what distros do this? Also when one does not have dom0
> > > what other forms are there to use Linux Xen backend drivers on GPL
> > > kernels (because obviously non-GPL kernels are not an option)?
> 
> FreeBSD (PVH - in progress) or NetBSD can run as initial domains.
> 
> The distro is Red Hat.

Why would Red Hat enable Linux Xen backend drivers without dom0?

  Luis

Re: RFC: xen config changes v4

[Roger Pau Monné]

El 06/03/15 a les 19.08, Luis R. Rodriguez ha escrit:
> On Fri, Mar 06, 2015 at 01:02:03PM -0500, Konrad Rzeszutek Wilk wrote:
>> The distro is Red Hat.
> 
> Why would Red Hat enable Linux Xen backend drivers without dom0?

Driver domains? That's the only reason to have backends in a non-dom0
kernel AFAIK.

Roger.