PVH Whitelist Results / Windows Dom0

PVH Whitelist Results / Windows Dom0

[Rian Quinn]

[-- Attachment #1.1: Type: text/plain, Size: 2405 bytes --]

We finally have a Linux PVH guest up and running (using an initramfs right
now). I have posted a quick status update video on YouTube that shows our
progress of getting a Windows Dom0 working (which is one of the many goals
of our research).
https://www.youtube.com/watch?v=xzTKBek-g0k

As promised in the x86 Community Call, here is the list of things that a
PVH Linux guest requires. You can see the code for this here:
https://github.com/rianquinn/hyperkernel/blob/hyperkernel_1/bfvmm/src/hve/arch/intel_x64/xen/xen_op.cpp
and here:
https://github.com/rianquinn/hyperkernel/blob/hyperkernel_1/bfexec/src/main.c

I would love to put this information somewhere in Xen's project (i.e. wiki
or source), but I am not sure what you would prefer. Any ideas?

Finally, keep in mind that we will likely keep adding to this list as we
add more features (like front/back support, xenstore, etc...)

Thanks,
- Rian

CPUID:
- XEN_CPUID_LEAF(0)
- XEN_CPUID_LEAF(1)
- XEN_CPUID_LEAF(2)
- XEN_CPUID_LEAF(4)
- 0x0, 0x1, 0x2, 0x4, 0x6, 0x7, 0xA, 0xB, 0xD, 0xF, 0x10, 0x15, 0x16
- 0x80000000, 0x80000001, 0x80000002, 0x80000003, 0x80000004
- 0x80000007, 0x80000008

MSRs:
- Hypercall page (dynamic)
- ia32_star
- ia32_lstar
- ia32_cstar
- ia32_fmask
- ia32_kernel_gs_base
- ia32_pat
- ia32_efer
- ia32_fs_base
- ia32_gs_base
- ia32_sysenter_cs
- ia32_sysenter_eip
- ia32_sysenter_esp
- ia32_apic_base
- platform_info
- 0x34, 0x64E, 0x140, 0x1A0, 0x6e0

IO Ports (some of these are odd):
- 0xCF8 - 0xCFF
- 0x4D0 (odd since PIT and ACPI is disable for everything that might need
this)
- 0x4D1
- 0x70
- 0x71
- 0x3FE (any ideas)?
- 0x42, 0x43, 0x61
- XEN_IOPORT_BASE (since QEMU is not used, why is this needed?)

Hypercalls:
- XENMEM_decrease_reservation
- XENMEM_add_to_physmap_handler
- XENMEM_memory_map_handler
- XENVER_get_features_handler
- GNTTABOP_query_size_handler
- GNTTABOP_set_version_handler
- EVTCHNOP_init_control_handler
- EVTCHNOP_expand_array_handler
- EVTCHNOP_alloc_unbound_handler
- EVTCHNOP_bind_ipi_handler
- EVTCHNOP_bind_virq_handler
- EVTCHNOP_bind_vcpu_handler
- EVTCHNOP_send_handler
- HVMOP_set_param_handler
- HVMOP_get_param_handler
- HVMOP_pagetable_dying_handler

Memory:
- Shared info page
- Start info struct (PVH)
- Initial GDT, IDT, TSS
- Command line page
- ACPI (FSDT, DSDT, MADT)
- xAPIC page
- Real-mode trampoline (this was weird)
- DMI, Video Bios, MP Table, and some legacy ROMs

[-- Attachment #1.2: Type: text/html, Size: 5015 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH Whitelist Results / Windows Dom0

[Wei Liu]

On Sat, Dec 01, 2018 at 09:21:00AM -0700, Rian Quinn wrote:
> We finally have a Linux PVH guest up and running (using an initramfs right
> now). I have posted a quick status update video on YouTube that shows our
> progress of getting a Windows Dom0 working (which is one of the many goals
> of our research).
> https://www.youtube.com/watch?v=xzTKBek-g0k

This is very cool.

> 
> As promised in the x86 Community Call, here is the list of things that a
> PVH Linux guest requires. You can see the code for this here:
> https://github.com/rianquinn/hyperkernel/blob/hyperkernel_1/bfvmm/src/hve/arch/intel_x64/xen/xen_op.cpp
> and here:
> https://github.com/rianquinn/hyperkernel/blob/hyperkernel_1/bfexec/src/main.c
> 
> I would love to put this information somewhere in Xen's project (i.e. wiki
> or source), but I am not sure what you would prefer. Any ideas?
> 
> Finally, keep in mind that we will likely keep adding to this list as we
> add more features (like front/back support, xenstore, etc...)
> 

wiki.xenproject.org seems to be the right place.

Please tell me your handle so I can grant you write access.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH Whitelist Results / Windows Dom0

[Juergen Gross]

On 01/12/2018 17:21, Rian Quinn wrote:
> We finally have a Linux PVH guest up and running (using an
> initramfs right now). I have posted a quick status update video on
> YouTube that shows our progress of getting a Windows Dom0 working (which
> is one of the many goals of our research).  
> https://www.youtube.com/watch?v=xzTKBek-g0k
> 
> As promised in the x86 Community Call, here is the list of things that a
> PVH Linux guest requires. You can see the code for this here:
> https://github.com/rianquinn/hyperkernel/blob/hyperkernel_1/bfvmm/src/hve/arch/intel_x64/xen/xen_op.cpp
> and here:
> https://github.com/rianquinn/hyperkernel/blob/hyperkernel_1/bfexec/src/main.c
> 
> I would love to put this information somewhere in Xen's project (i.e.
> wiki or source), but I am not sure what you would prefer. Any ideas?
> 
> Finally, keep in mind that we will likely keep adding to this list as we
> add more features (like front/back support, xenstore, etc...)
> 
> Thanks,
> - Rian
> 

> IO Ports (some of these are odd):
> - XEN_IOPORT_BASE (since QEMU is not used, why is this needed?)

Linux kernel commit e6111161c0a02d58919d776eec94b313bb57911f
(in 4.20-rc1) addresses this already.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH Whitelist Results / Windows Dom0

[Roger Pau Monné]

Hello,

Thanks, this is very interesting.

On Sat, Dec 01, 2018 at 09:21:00AM -0700, Rian Quinn wrote:
> We finally have a Linux PVH guest up and running (using an initramfs right
> now). I have posted a quick status update video on YouTube that shows our
> progress of getting a Windows Dom0 working (which is one of the many goals
> of our research).
> https://www.youtube.com/watch?v=xzTKBek-g0k
> 
> As promised in the x86 Community Call, here is the list of things that a
> PVH Linux guest requires. You can see the code for this here:
> https://github.com/rianquinn/hyperkernel/blob/hyperkernel_1/bfvmm/src/hve/arch/intel_x64/xen/xen_op.cpp
> and here:
> https://github.com/rianquinn/hyperkernel/blob/hyperkernel_1/bfexec/src/main.c
> 
> I would love to put this information somewhere in Xen's project (i.e. wiki
> or source), but I am not sure what you would prefer. Any ideas?
> 
> Finally, keep in mind that we will likely keep adding to this list as we
> add more features (like front/back support, xenstore, etc...)
> 
> Thanks,
> - Rian
> 
> CPUID:
> - XEN_CPUID_LEAF(0)
> - XEN_CPUID_LEAF(1)
> - XEN_CPUID_LEAF(2)
> - XEN_CPUID_LEAF(4)
> - 0x0, 0x1, 0x2, 0x4, 0x6, 0x7, 0xA, 0xB, 0xD, 0xF, 0x10, 0x15, 0x16
> - 0x80000000, 0x80000001, 0x80000002, 0x80000003, 0x80000004
> - 0x80000007, 0x80000008
> 
> MSRs:
> - Hypercall page (dynamic)
> - ia32_star
> - ia32_lstar
> - ia32_cstar
> - ia32_fmask
> - ia32_kernel_gs_base
> - ia32_pat
> - ia32_efer
> - ia32_fs_base
> - ia32_gs_base
> - ia32_sysenter_cs
> - ia32_sysenter_eip
> - ia32_sysenter_esp
> - ia32_apic_base
> - platform_info
> - 0x34, 0x64E, 0x140, 0x1A0, 0x6e0
> 
> IO Ports (some of these are odd):
> - 0xCF8 - 0xCFF
> - 0x4D0 (odd since PIT and ACPI is disable for everything that might need
> this)

Likely some poking for EISA devices? (same for 0x4D1)

Can you trace this to the Linux code that's actually making the call
by injecting a trap when this happens?

> - 0x4D1
> - 0x70
> - 0x71
> - 0x3FE (any ideas)?

Serial port poking?

Again would be interesting to know the Linux code that's poking
this.

> - 0x42, 0x43, 0x61
> - XEN_IOPORT_BASE (since QEMU is not used, why is this needed?)

IIRC the PVH code path in Linux is almost the same as the HVM one,
that's why this port is poked in order to see whether there are
emulated devices to disable. I think this is expected and perfectly
fine.

> 
> Hypercalls:
> - XENMEM_decrease_reservation
> - XENMEM_add_to_physmap_handler
> - XENMEM_memory_map_handler
> - XENVER_get_features_handler
> - GNTTABOP_query_size_handler
> - GNTTABOP_set_version_handler
> - EVTCHNOP_init_control_handler
> - EVTCHNOP_expand_array_handler
> - EVTCHNOP_alloc_unbound_handler
> - EVTCHNOP_bind_ipi_handler
> - EVTCHNOP_bind_virq_handler
> - EVTCHNOP_bind_vcpu_handler
> - EVTCHNOP_send_handler
> - HVMOP_set_param_handler
> - HVMOP_get_param_handler
> - HVMOP_pagetable_dying_handler
> 
> Memory:
> - Shared info page
> - Start info struct (PVH)
> - Initial GDT, IDT, TSS
> - Command line page
> - ACPI (FSDT, DSDT, MADT)
> - xAPIC page
> - Real-mode trampoline (this was weird)

APs for PVH can be started using the native way, which means they are
started in real mode, that's why Linux uses the real mode trampoline.

> - DMI, Video Bios, MP Table, and some legacy ROMs

Legacy ROMs from which device?

Also there's no MP tables or video BIOS at all, so I guess this is
Linux trying to find the BDA and friends in the low 1MB?

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH Whitelist Results / Windows Dom0

[Rian Quinn]

[-- Attachment #1.1: Type: text/plain, Size: 5459 bytes --]

> Can you trace this to the Linux code that's actually making the call
> by injecting a trap when this happens?

Yes, we can. In some cases, we have to manually backtrace, but so far
we have been able to map resources to the actual source code.

> Serial port poking?

This would be a great one to locate in the kernel. I suspect that
serial is the case, but if that is true, something is a bit wrong as
once again, this device doesn't exist without QEMU. There is also a
little bit of testing that we should do here. Right now we manually
pass-through a serial device for UART debugging, and that might have
the side effect of this port showing up so I would want to rule that
out first.

> APs for PVH can be started using the native way, which means they are
> started in real mode, that's why Linux uses the real mode trampoline.

Ah... ok. That makes sense. Uhg... emulating INIT/SIPI is no fun. That
is some pretty fragile code.

> Legacy ROMs from which device?

Video BIOS was one of them. There are several memory regions within
legacy BIOS that are being scanned so my assumption is that these
regions are some ROMs, and I am not really sure why PVH would execute
that logic at all. I am pretty sure that it is scanning for MP tables
as I think I traced that specific logic back to the Linux kernel. I
know for sure that DMI is being scanned as well. Right now we map in a
read-only zero page and that works fine, but I would think that a lot
of this logic would not be needed in the Guest case. Dom0 is another
story.

On Mon, Dec 3, 2018 at 4:42 AM Roger Pau Monné <roger.pau@citrix.com> wrote:

> Hello,
>
> Thanks, this is very interesting.
>
> On Sat, Dec 01, 2018 at 09:21:00AM -0700, Rian Quinn wrote:
> > We finally have a Linux PVH guest up and running (using an initramfs
> right
> > now). I have posted a quick status update video on YouTube that shows our
> > progress of getting a Windows Dom0 working (which is one of the many
> goals
> > of our research).
> > https://www.youtube.com/watch?v=xzTKBek-g0k
> >
> > As promised in the x86 Community Call, here is the list of things that a
> > PVH Linux guest requires. You can see the code for this here:
> >
> https://github.com/rianquinn/hyperkernel/blob/hyperkernel_1/bfvmm/src/hve/arch/intel_x64/xen/xen_op.cpp
> > and here:
> >
> https://github.com/rianquinn/hyperkernel/blob/hyperkernel_1/bfexec/src/main.c
> >
> > I would love to put this information somewhere in Xen's project (i.e.
> wiki
> > or source), but I am not sure what you would prefer. Any ideas?
> >
> > Finally, keep in mind that we will likely keep adding to this list as we
> > add more features (like front/back support, xenstore, etc...)
> >
> > Thanks,
> > - Rian
> >
> > CPUID:
> > - XEN_CPUID_LEAF(0)
> > - XEN_CPUID_LEAF(1)
> > - XEN_CPUID_LEAF(2)
> > - XEN_CPUID_LEAF(4)
> > - 0x0, 0x1, 0x2, 0x4, 0x6, 0x7, 0xA, 0xB, 0xD, 0xF, 0x10, 0x15, 0x16
> > - 0x80000000, 0x80000001, 0x80000002, 0x80000003, 0x80000004
> > - 0x80000007, 0x80000008
> >
> > MSRs:
> > - Hypercall page (dynamic)
> > - ia32_star
> > - ia32_lstar
> > - ia32_cstar
> > - ia32_fmask
> > - ia32_kernel_gs_base
> > - ia32_pat
> > - ia32_efer
> > - ia32_fs_base
> > - ia32_gs_base
> > - ia32_sysenter_cs
> > - ia32_sysenter_eip
> > - ia32_sysenter_esp
> > - ia32_apic_base
> > - platform_info
> > - 0x34, 0x64E, 0x140, 0x1A0, 0x6e0
> >
> > IO Ports (some of these are odd):
> > - 0xCF8 - 0xCFF
> > - 0x4D0 (odd since PIT and ACPI is disable for everything that might need
> > this)
>
> Likely some poking for EISA devices? (same for 0x4D1)
>
> Can you trace this to the Linux code that's actually making the call
> by injecting a trap when this happens?
>
> > - 0x4D1
> > - 0x70
> > - 0x71
> > - 0x3FE (any ideas)?
>
> Serial port poking?
>
> Again would be interesting to know the Linux code that's poking
> this.
>
> > - 0x42, 0x43, 0x61
> > - XEN_IOPORT_BASE (since QEMU is not used, why is this needed?)
>
> IIRC the PVH code path in Linux is almost the same as the HVM one,
> that's why this port is poked in order to see whether there are
> emulated devices to disable. I think this is expected and perfectly
> fine.
>
> >
> > Hypercalls:
> > - XENMEM_decrease_reservation
> > - XENMEM_add_to_physmap_handler
> > - XENMEM_memory_map_handler
> > - XENVER_get_features_handler
> > - GNTTABOP_query_size_handler
> > - GNTTABOP_set_version_handler
> > - EVTCHNOP_init_control_handler
> > - EVTCHNOP_expand_array_handler
> > - EVTCHNOP_alloc_unbound_handler
> > - EVTCHNOP_bind_ipi_handler
> > - EVTCHNOP_bind_virq_handler
> > - EVTCHNOP_bind_vcpu_handler
> > - EVTCHNOP_send_handler
> > - HVMOP_set_param_handler
> > - HVMOP_get_param_handler
> > - HVMOP_pagetable_dying_handler
> >
> > Memory:
> > - Shared info page
> > - Start info struct (PVH)
> > - Initial GDT, IDT, TSS
> > - Command line page
> > - ACPI (FSDT, DSDT, MADT)
> > - xAPIC page
> > - Real-mode trampoline (this was weird)
>
> APs for PVH can be started using the native way, which means they are
> started in real mode, that's why Linux uses the real mode trampoline.
>
> > - DMI, Video Bios, MP Table, and some legacy ROMs
>
> Legacy ROMs from which device?
>
> Also there's no MP tables or video BIOS at all, so I guess this is
> Linux trying to find the BDA and friends in the low 1MB?
>
> Thanks, Roger.
>

[-- Attachment #1.2: Type: text/html, Size: 6865 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH Whitelist Results / Windows Dom0

[Roger Pau Monné]

Hello,

On Mon, Dec 03, 2018 at 09:06:37AM -0700, Rian Quinn wrote:
> > Can you trace this to the Linux code that's actually making the call
> > by injecting a trap when this happens?
> 
> Yes, we can. In some cases, we have to manually backtrace, but so far
> we have been able to map resources to the actual source code.
> 
> > Serial port poking?
> 
> This would be a great one to locate in the kernel. I suspect that
> serial is the case, but if that is true, something is a bit wrong as
> once again, this device doesn't exist without QEMU.

Maybe Linux pokes at this port in order to check whether the device
exists?

The fact that the device doesn't exist doesn't prevent a guest from
poking at this port, and IMO it's a legit thing to do. Returning all
1s (like bare metal) should be OK and would actually signal Linux
there's no register there and thus no device.

> There is also a
> little bit of testing that we should do here. Right now we manually
> pass-through a serial device for UART debugging, and that might have
> the side effect of this port showing up so I would want to rule that
> out first.
> 
> > APs for PVH can be started using the native way, which means they are
> > started in real mode, that's why Linux uses the real mode trampoline.
> 
> Ah... ok. That makes sense. Uhg... emulating INIT/SIPI is no fun. That
> is some pretty fragile code.

It's the same code that we already use for HVM guests, since PVH
guests get an emulated LAPIC like HVM ones.

> > Legacy ROMs from which device?
> 
> Video BIOS was one of them. There are several memory regions within
> legacy BIOS that are being scanned so my assumption is that these
> regions are some ROMs, and I am not really sure why PVH would execute
> that logic at all.

Xen signals in the FADT that there's no VGA, but I won't be surprised
that some OSes simply ignore this bit because there are systems with
broken ACPI tables out there with the bit set and VGA.

> I am pretty sure that it is scanning for MP tables
> as I think I traced that specific logic back to the Linux kernel.

There's no other way to detect MP tables rather that scanning the
different positions where they can be found, so I think it's fine for
Linux to do so.

> I
> know for sure that DMI is being scanned as well. Right now we map in a
> read-only zero page and that works fine, but I would think that a lot
> of this logic would not be needed in the Guest case. Dom0 is another
> story.

IMO we should try to limit as much as possible the PVH specific
modifications that we have to make to guests. So it's better to let
the guest scan memory or poke at IO ports rather than add a specific
'is running on PVH' check to each device driver that we know it's not
available when running as PVH.

Poking at such ports or scanning memory is exactly the same that's
done on bare metal, and should work fine on PVH to detect the absence
of certain devices.

Thanks, Roger.

> On Mon, Dec 3, 2018 at 4:42 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
> 
> > Hello,
> >
> > Thanks, this is very interesting.
> >
> > On Sat, Dec 01, 2018 at 09:21:00AM -0700, Rian Quinn wrote:
> > > We finally have a Linux PVH guest up and running (using an initramfs
> > right
> > > now). I have posted a quick status update video on YouTube that shows our
> > > progress of getting a Windows Dom0 working (which is one of the many
> > goals
> > > of our research).
> > > https://www.youtube.com/watch?v=xzTKBek-g0k
> > >
> > > As promised in the x86 Community Call, here is the list of things that a
> > > PVH Linux guest requires. You can see the code for this here:
> > >
> > https://github.com/rianquinn/hyperkernel/blob/hyperkernel_1/bfvmm/src/hve/arch/intel_x64/xen/xen_op.cpp
> > > and here:
> > >
> > https://github.com/rianquinn/hyperkernel/blob/hyperkernel_1/bfexec/src/main.c
> > >
> > > I would love to put this information somewhere in Xen's project (i.e.
> > wiki
> > > or source), but I am not sure what you would prefer. Any ideas?
> > >
> > > Finally, keep in mind that we will likely keep adding to this list as we
> > > add more features (like front/back support, xenstore, etc...)
> > >
> > > Thanks,
> > > - Rian
> > >
> > > CPUID:
> > > - XEN_CPUID_LEAF(0)
> > > - XEN_CPUID_LEAF(1)
> > > - XEN_CPUID_LEAF(2)
> > > - XEN_CPUID_LEAF(4)
> > > - 0x0, 0x1, 0x2, 0x4, 0x6, 0x7, 0xA, 0xB, 0xD, 0xF, 0x10, 0x15, 0x16
> > > - 0x80000000, 0x80000001, 0x80000002, 0x80000003, 0x80000004
> > > - 0x80000007, 0x80000008
> > >
> > > MSRs:
> > > - Hypercall page (dynamic)
> > > - ia32_star
> > > - ia32_lstar
> > > - ia32_cstar
> > > - ia32_fmask
> > > - ia32_kernel_gs_base
> > > - ia32_pat
> > > - ia32_efer
> > > - ia32_fs_base
> > > - ia32_gs_base
> > > - ia32_sysenter_cs
> > > - ia32_sysenter_eip
> > > - ia32_sysenter_esp
> > > - ia32_apic_base
> > > - platform_info
> > > - 0x34, 0x64E, 0x140, 0x1A0, 0x6e0
> > >
> > > IO Ports (some of these are odd):
> > > - 0xCF8 - 0xCFF
> > > - 0x4D0 (odd since PIT and ACPI is disable for everything that might need
> > > this)
> >
> > Likely some poking for EISA devices? (same for 0x4D1)
> >
> > Can you trace this to the Linux code that's actually making the call
> > by injecting a trap when this happens?
> >
> > > - 0x4D1
> > > - 0x70
> > > - 0x71
> > > - 0x3FE (any ideas)?
> >
> > Serial port poking?
> >
> > Again would be interesting to know the Linux code that's poking
> > this.
> >
> > > - 0x42, 0x43, 0x61
> > > - XEN_IOPORT_BASE (since QEMU is not used, why is this needed?)
> >
> > IIRC the PVH code path in Linux is almost the same as the HVM one,
> > that's why this port is poked in order to see whether there are
> > emulated devices to disable. I think this is expected and perfectly
> > fine.
> >
> > >
> > > Hypercalls:
> > > - XENMEM_decrease_reservation
> > > - XENMEM_add_to_physmap_handler
> > > - XENMEM_memory_map_handler
> > > - XENVER_get_features_handler
> > > - GNTTABOP_query_size_handler
> > > - GNTTABOP_set_version_handler
> > > - EVTCHNOP_init_control_handler
> > > - EVTCHNOP_expand_array_handler
> > > - EVTCHNOP_alloc_unbound_handler
> > > - EVTCHNOP_bind_ipi_handler
> > > - EVTCHNOP_bind_virq_handler
> > > - EVTCHNOP_bind_vcpu_handler
> > > - EVTCHNOP_send_handler
> > > - HVMOP_set_param_handler
> > > - HVMOP_get_param_handler
> > > - HVMOP_pagetable_dying_handler
> > >
> > > Memory:
> > > - Shared info page
> > > - Start info struct (PVH)
> > > - Initial GDT, IDT, TSS
> > > - Command line page
> > > - ACPI (FSDT, DSDT, MADT)
> > > - xAPIC page
> > > - Real-mode trampoline (this was weird)
> >
> > APs for PVH can be started using the native way, which means they are
> > started in real mode, that's why Linux uses the real mode trampoline.
> >
> > > - DMI, Video Bios, MP Table, and some legacy ROMs
> >
> > Legacy ROMs from which device?
> >
> > Also there's no MP tables or video BIOS at all, so I guess this is
> > Linux trying to find the BDA and friends in the low 1MB?
> >
> > Thanks, Roger.
> >

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH Whitelist Results / Windows Dom0

[Rian Quinn]

[-- Attachment #1.1: Type: text/plain, Size: 9478 bytes --]

> Xen signals in the FADT that there's no VGA, but I won't be surprised
> that some OSes simply ignore this bit because there are systems with
> broken ACPI tables out there with the bit set and VGA.

We do the same thing, and yeah it appears that Linux is ignoring this. We
noticed the same thing WRT some other ACPI specific things like SCI that
are not being disabled by Linux even though the ACPI tables say they are
disabled. IIRC, the PIT is in the boat as well. Either way, I agree that
returning nothing in these cases is a legit way to handle it.

> There's no other way to detect MP tables rather that scanning the
> different positions where they can be found, so I think it's fine for
> Linux to do so.

Agreed

> IMO we should try to limit as much as possible the PVH specific
> modifications that we have to make to guests. So it's better to let
> the guest scan memory or poke at IO ports rather than add a specific
> 'is running on PVH' check to each device driver that we know it's not
> available when running as PVH.

> Poking at such ports or scanning memory is exactly the same that's
> done on bare metal, and should work fine on PVH to detect the absence
> of certain devices.

Agreed. We still have to sort out some of these registers, but for now, I
think things are ok in general. I am more worried about Xen specific
things. We will know more once we try to plug all of the holes we had to
open. I also agree 100% that for PVH and HVM we should minimize the number
of things that need to be changed to support. Its better for the hypervisor
to return "no supported" than for the Linux kernel to need mods to support
PVH. This is also really important in order to support PVH Dom0 and
eventually someday a PVH version of Windows, which should be possible.

- Rian


On Mon, Dec 3, 2018 at 10:05 AM Roger Pau Monné <roger.pau@citrix.com>
wrote:

> Hello,
>
> On Mon, Dec 03, 2018 at 09:06:37AM -0700, Rian Quinn wrote:
> > > Can you trace this to the Linux code that's actually making the call
> > > by injecting a trap when this happens?
> >
> > Yes, we can. In some cases, we have to manually backtrace, but so far
> > we have been able to map resources to the actual source code.
> >
> > > Serial port poking?
> >
> > This would be a great one to locate in the kernel. I suspect that
> > serial is the case, but if that is true, something is a bit wrong as
> > once again, this device doesn't exist without QEMU.
>
> Maybe Linux pokes at this port in order to check whether the device
> exists?
>
> The fact that the device doesn't exist doesn't prevent a guest from
> poking at this port, and IMO it's a legit thing to do. Returning all
> 1s (like bare metal) should be OK and would actually signal Linux
> there's no register there and thus no device.
>
> > There is also a
> > little bit of testing that we should do here. Right now we manually
> > pass-through a serial device for UART debugging, and that might have
> > the side effect of this port showing up so I would want to rule that
> > out first.
> >
> > > APs for PVH can be started using the native way, which means they are
> > > started in real mode, that's why Linux uses the real mode trampoline.
> >
> > Ah... ok. That makes sense. Uhg... emulating INIT/SIPI is no fun. That
> > is some pretty fragile code.
>
> It's the same code that we already use for HVM guests, since PVH
> guests get an emulated LAPIC like HVM ones.
>
> > > Legacy ROMs from which device?
> >
> > Video BIOS was one of them. There are several memory regions within
> > legacy BIOS that are being scanned so my assumption is that these
> > regions are some ROMs, and I am not really sure why PVH would execute
> > that logic at all.
>
> Xen signals in the FADT that there's no VGA, but I won't be surprised
> that some OSes simply ignore this bit because there are systems with
> broken ACPI tables out there with the bit set and VGA.
>
> > I am pretty sure that it is scanning for MP tables
> > as I think I traced that specific logic back to the Linux kernel.
>
> There's no other way to detect MP tables rather that scanning the
> different positions where they can be found, so I think it's fine for
> Linux to do so.
>
> > I
> > know for sure that DMI is being scanned as well. Right now we map in a
> > read-only zero page and that works fine, but I would think that a lot
> > of this logic would not be needed in the Guest case. Dom0 is another
> > story.
>
> IMO we should try to limit as much as possible the PVH specific
> modifications that we have to make to guests. So it's better to let
> the guest scan memory or poke at IO ports rather than add a specific
> 'is running on PVH' check to each device driver that we know it's not
> available when running as PVH.
>
> Poking at such ports or scanning memory is exactly the same that's
> done on bare metal, and should work fine on PVH to detect the absence
> of certain devices.
>
> Thanks, Roger.
>
> > On Mon, Dec 3, 2018 at 4:42 AM Roger Pau Monné <roger.pau@citrix.com>
> wrote:
> >
> > > Hello,
> > >
> > > Thanks, this is very interesting.
> > >
> > > On Sat, Dec 01, 2018 at 09:21:00AM -0700, Rian Quinn wrote:
> > > > We finally have a Linux PVH guest up and running (using an initramfs
> > > right
> > > > now). I have posted a quick status update video on YouTube that
> shows our
> > > > progress of getting a Windows Dom0 working (which is one of the many
> > > goals
> > > > of our research).
> > > > https://www.youtube.com/watch?v=xzTKBek-g0k
> > > >
> > > > As promised in the x86 Community Call, here is the list of things
> that a
> > > > PVH Linux guest requires. You can see the code for this here:
> > > >
> > >
> https://github.com/rianquinn/hyperkernel/blob/hyperkernel_1/bfvmm/src/hve/arch/intel_x64/xen/xen_op.cpp
> > > > and here:
> > > >
> > >
> https://github.com/rianquinn/hyperkernel/blob/hyperkernel_1/bfexec/src/main.c
> > > >
> > > > I would love to put this information somewhere in Xen's project (i.e.
> > > wiki
> > > > or source), but I am not sure what you would prefer. Any ideas?
> > > >
> > > > Finally, keep in mind that we will likely keep adding to this list
> as we
> > > > add more features (like front/back support, xenstore, etc...)
> > > >
> > > > Thanks,
> > > > - Rian
> > > >
> > > > CPUID:
> > > > - XEN_CPUID_LEAF(0)
> > > > - XEN_CPUID_LEAF(1)
> > > > - XEN_CPUID_LEAF(2)
> > > > - XEN_CPUID_LEAF(4)
> > > > - 0x0, 0x1, 0x2, 0x4, 0x6, 0x7, 0xA, 0xB, 0xD, 0xF, 0x10, 0x15, 0x16
> > > > - 0x80000000, 0x80000001, 0x80000002, 0x80000003, 0x80000004
> > > > - 0x80000007, 0x80000008
> > > >
> > > > MSRs:
> > > > - Hypercall page (dynamic)
> > > > - ia32_star
> > > > - ia32_lstar
> > > > - ia32_cstar
> > > > - ia32_fmask
> > > > - ia32_kernel_gs_base
> > > > - ia32_pat
> > > > - ia32_efer
> > > > - ia32_fs_base
> > > > - ia32_gs_base
> > > > - ia32_sysenter_cs
> > > > - ia32_sysenter_eip
> > > > - ia32_sysenter_esp
> > > > - ia32_apic_base
> > > > - platform_info
> > > > - 0x34, 0x64E, 0x140, 0x1A0, 0x6e0
> > > >
> > > > IO Ports (some of these are odd):
> > > > - 0xCF8 - 0xCFF
> > > > - 0x4D0 (odd since PIT and ACPI is disable for everything that might
> need
> > > > this)
> > >
> > > Likely some poking for EISA devices? (same for 0x4D1)
> > >
> > > Can you trace this to the Linux code that's actually making the call
> > > by injecting a trap when this happens?
> > >
> > > > - 0x4D1
> > > > - 0x70
> > > > - 0x71
> > > > - 0x3FE (any ideas)?
> > >
> > > Serial port poking?
> > >
> > > Again would be interesting to know the Linux code that's poking
> > > this.
> > >
> > > > - 0x42, 0x43, 0x61
> > > > - XEN_IOPORT_BASE (since QEMU is not used, why is this needed?)
> > >
> > > IIRC the PVH code path in Linux is almost the same as the HVM one,
> > > that's why this port is poked in order to see whether there are
> > > emulated devices to disable. I think this is expected and perfectly
> > > fine.
> > >
> > > >
> > > > Hypercalls:
> > > > - XENMEM_decrease_reservation
> > > > - XENMEM_add_to_physmap_handler
> > > > - XENMEM_memory_map_handler
> > > > - XENVER_get_features_handler
> > > > - GNTTABOP_query_size_handler
> > > > - GNTTABOP_set_version_handler
> > > > - EVTCHNOP_init_control_handler
> > > > - EVTCHNOP_expand_array_handler
> > > > - EVTCHNOP_alloc_unbound_handler
> > > > - EVTCHNOP_bind_ipi_handler
> > > > - EVTCHNOP_bind_virq_handler
> > > > - EVTCHNOP_bind_vcpu_handler
> > > > - EVTCHNOP_send_handler
> > > > - HVMOP_set_param_handler
> > > > - HVMOP_get_param_handler
> > > > - HVMOP_pagetable_dying_handler
> > > >
> > > > Memory:
> > > > - Shared info page
> > > > - Start info struct (PVH)
> > > > - Initial GDT, IDT, TSS
> > > > - Command line page
> > > > - ACPI (FSDT, DSDT, MADT)
> > > > - xAPIC page
> > > > - Real-mode trampoline (this was weird)
> > >
> > > APs for PVH can be started using the native way, which means they are
> > > started in real mode, that's why Linux uses the real mode trampoline.
> > >
> > > > - DMI, Video Bios, MP Table, and some legacy ROMs
> > >
> > > Legacy ROMs from which device?
> > >
> > > Also there's no MP tables or video BIOS at all, so I guess this is
> > > Linux trying to find the BDA and friends in the low 1MB?
> > >
> > > Thanks, Roger.
> > >
>

[-- Attachment #1.2: Type: text/html, Size: 12137 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH Whitelist Results / Windows Dom0

[Connor Davis]

[-- Attachment #1.1: Type: text/plain, Size: 8544 bytes --]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, December 3, 2018 1:07 PM, Rian Quinn <rianquinn@gmail.com> wrote:

>> Xen signals in the FADT that there's no VGA, but I won't be surprised
>> that some OSes simply ignore this bit because there are systems with
>> broken ACPI tables out there with the bit set and VGA.
>
> We do the same thing, and yeah it appears that Linux is ignoring this. We noticed the same thing WRT some other ACPI specific things like SCI that are not being disabled by Linux even though the ACPI tables say they are disabled. IIRC, the PIT is in the boat as well. Either way, I agree that returning nothing in these cases is a legit way to handle it.
>
>> There's no other way to detect MP tables rather that scanning the
>> different positions where they can be found, so I think it's fine for
>> Linux to do so.
>
> Agreed
>
>> IMO we should try to limit as much as possible the PVH specific
>> modifications that we have to make to guests. So it's better to let
>> the guest scan memory or poke at IO ports rather than add a specific
>> 'is running on PVH' check to each device driver that we know it's not
>> available when running as PVH.
>
>> Poking at such ports or scanning memory is exactly the same that's
>> done on bare metal, and should work fine on PVH to detect the absence
>> of certain devices.
>
> Agreed. We still have to sort out some of these registers, but for now, I think things are ok in general. I am more worried about Xen specific things. We will know more once we try to plug all of the holes we had to open. I also agree 100% that for PVH and HVM we should minimize the number of things that need to be changed to support. Its better for the hypervisor to return "no supported" than for the Linux kernel to need mods to support PVH. This is also really important in order to support PVH Dom0 and eventually someday a PVH version of Windows, which should be possible.
>
> - Rian
>
> On Mon, Dec 3, 2018 at 10:05 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
>
>> Hello,
>>
>> On Mon, Dec 03, 2018 at 09:06:37AM -0700, Rian Quinn wrote:
>>> > Can you trace this to the Linux code that's actually making the call
>>> > by injecting a trap when this happens?
>>>
>>> Yes, we can. In some cases, we have to manually backtrace, but so far
>>> we have been able to map resources to the actual source code.
>>>
>>> > Serial port poking?
>>>
>>> This would be a great one to locate in the kernel. I suspect that
>>> serial is the case, but if that is true, something is a bit wrong as
>>> once again, this device doesn't exist without QEMU.
>>
>> Maybe Linux pokes at this port in order to check whether the device
>> exists?
>>
>> The fact that the device doesn't exist doesn't prevent a guest from
>> poking at this port, and IMO it's a legit thing to do. Returning all
>> 1s (like bare metal) should be OK and would actually signal Linux
>> there's no register there and thus no device.
>>
>>> There is also a
>>> little bit of testing that we should do here. Right now we manually
>>> pass-through a serial device for UART debugging, and that might have
>>> the side effect of this port showing up so I would want to rule that
>>> out first.
>>>
>>> > APs for PVH can be started using the native way, which means they are
>>> > started in real mode, that's why Linux uses the real mode trampoline.
>>>
>>> Ah... ok. That makes sense. Uhg... emulating INIT/SIPI is no fun. That
>>> is some pretty fragile code.
>>
>> It's the same code that we already use for HVM guests, since PVH
>> guests get an emulated LAPIC like HVM ones.
>>
>>> > Legacy ROMs from which device?
>>>
>>> Video BIOS was one of them. There are several memory regions within
>>> legacy BIOS that are being scanned so my assumption is that these
>>> regions are some ROMs, and I am not really sure why PVH would execute
>>> that logic at all.
>>
>> Xen signals in the FADT that there's no VGA, but I won't be surprised
>> that some OSes simply ignore this bit because there are systems with
>> broken ACPI tables out there with the bit set and VGA.
>>
>>> I am pretty sure that it is scanning for MP tables
>>> as I think I traced that specific logic back to the Linux kernel.
>>
>> There's no other way to detect MP tables rather that scanning the
>> different positions where they can be found, so I think it's fine for
>> Linux to do so.
>>
>>> I
>>> know for sure that DMI is being scanned as well. Right now we map in a
>>> read-only zero page and that works fine, but I would think that a lot
>>> of this logic would not be needed in the Guest case. Dom0 is another
>>> story.
>>
>> IMO we should try to limit as much as possible the PVH specific
>> modifications that we have to make to guests. So it's better to let
>> the guest scan memory or poke at IO ports rather than add a specific
>> 'is running on PVH' check to each device driver that we know it's not
>> available when running as PVH.
>>
>> Poking at such ports or scanning memory is exactly the same that's
>> done on bare metal, and should work fine on PVH to detect the absence
>> of certain devices.
>>
>> Thanks, Roger.
>>
>>> On Mon, Dec 3, 2018 at 4:42 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>
>>> > Hello,
>>> >
>>> > Thanks, this is very interesting.
>>> >
>>> > On Sat, Dec 01, 2018 at 09:21:00AM -0700, Rian Quinn wrote:
>>> > > We finally have a Linux PVH guest up and running (using an initramfs
>>> > right
>>> > > now). I have posted a quick status update video on YouTube that shows our
>>> > > progress of getting a Windows Dom0 working (which is one of the many
>>> > goals
>>> > > of our research).
>>> > > https://www.youtube.com/watch?v=xzTKBek-g0k
>>> > >
>>> > > As promised in the x86 Community Call, here is the list of things that a
>>> > > PVH Linux guest requires. You can see the code for this here:
>>> > >
>>> > https://github.com/rianquinn/hyperkernel/blob/hyperkernel_1/bfvmm/src/hve/arch/intel_x64/xen/xen_op.cpp
>>> > > and here:
>>> > >
>>> > https://github.com/rianquinn/hyperkernel/blob/hyperkernel_1/bfexec/src/main.c
>>> > >
>>> > > I would love to put this information somewhere in Xen's project (i.e.
>>> > wiki
>>> > > or source), but I am not sure what you would prefer. Any ideas?
>>> > >
>>> > > Finally, keep in mind that we will likely keep adding to this list as we
>>> > > add more features (like front/back support, xenstore, etc...)
>>> > >
>>> > > Thanks,
>>> > > - Rian
>>> > >
>>> > > CPUID:
>>> > > - XEN_CPUID_LEAF(0)
>>> > > - XEN_CPUID_LEAF(1)
>>> > > - XEN_CPUID_LEAF(2)
>>> > > - XEN_CPUID_LEAF(4)
>>> > > - 0x0, 0x1, 0x2, 0x4, 0x6, 0x7, 0xA, 0xB, 0xD, 0xF, 0x10, 0x15, 0x16
>>> > > - 0x80000000, 0x80000001, 0x80000002, 0x80000003, 0x80000004
>>> > > - 0x80000007, 0x80000008
>>> > >
>>> > > MSRs:
>>> > > - Hypercall page (dynamic)
>>> > > - ia32_star
>>> > > - ia32_lstar
>>> > > - ia32_cstar
>>> > > - ia32_fmask
>>> > > - ia32_kernel_gs_base
>>> > > - ia32_pat
>>> > > - ia32_efer
>>> > > - ia32_fs_base
>>> > > - ia32_gs_base
>>> > > - ia32_sysenter_cs
>>> > > - ia32_sysenter_eip
>>> > > - ia32_sysenter_esp
>>> > > - ia32_apic_base
>>> > > - platform_info
>>> > > - 0x34, 0x64E, 0x140, 0x1A0, 0x6e0
>>> > >
>>> > > IO Ports (some of these are odd):
>>> > > - 0xCF8 - 0xCFF
>>> > > - 0x4D0 (odd since PIT and ACPI is disable for everything that might need
>>> > > this)
>>> >
>>> > Likely some poking for EISA devices? (same for 0x4D1)
>>> >
>>> > Can you trace this to the Linux code that's actually making the call
>>> > by injecting a trap when this happens?
>>> >
>>> > > - 0x4D1

0x4D0 and 0x4D1 were touched from apci_early_init -> acpi_pic_sci_set_trigger since
we don't emulate any IOAPICs yet.

>>> > > - 0x70
>>> > > - 0x71

These are accessed from reassert_nmi. This is only called from default_do_nmi
in the version the guest is based on (4.20-rc2).

>>> > > - 0x3FE (any ideas)?
>>> >
>>> > Serial port poking?
>>> >
>>> > Again would be interesting to know the Linux code that's poking
>>> > this.
>>> >
>>> > > - 0x42, 0x43, 0x61

These are in pit_calibrate_tsc. The guest was taking this path initially, but
right now we are abusing cpuid.15h by returning the TSC hz in ecx rather
than the core crystal. We made a small, temporary hack to the pv interface/linux
to get this working:

https://github.com/connojd/linux/commit/6b620ebc4f56129ed068d9995d59e46521abc80b

In all our attempts to convince linux that TSC is reliable, it inevitably decided that it
in fact was not. Any thoughts on how to make this not be so dirty?

- Connor

[-- Attachment #1.2: Type: text/html, Size: 13592 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH Whitelist Results / Windows Dom0

[Roger Pau Monné]

On Tue, Dec 04, 2018 at 05:46:38AM +0000, Connor Davis wrote:
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Monday, December 3, 2018 1:07 PM, Rian Quinn <rianquinn@gmail.com> wrote:
> >>> > > IO Ports (some of these are odd):
> >>> > > - 0xCF8 - 0xCFF
> >>> > > - 0x4D0 (odd since PIT and ACPI is disable for everything that might need
> >>> > > this)
> >>> >
> >>> > Likely some poking for EISA devices? (same for 0x4D1)
> >>> >
> >>> > Can you trace this to the Linux code that's actually making the call
> >>> > by injecting a trap when this happens?
> >>> >
> >>> > > - 0x4D1
> 
> 0x4D0 and 0x4D1 were touched from apci_early_init -> acpi_pic_sci_set_trigger since
> we don't emulate any IOAPICs yet.

Hm, yes. When I looked into this I couldn't find a way to signal from
ACPI that there's no SCI. The only way seems to be to report
hardware-reduced ACPI, but that has a lot of other implications.

Maybe we could look into adding something to the ACPI spec to signal
there's no SCI even when not using hardware-reduced ACPI?

The FADT IA-PC boot flags field still has plenty of space.

> >>> > > - 0x70
> >>> > > - 0x71
> 
> These are accessed from reassert_nmi. This is only called from default_do_nmi
> in the version the guest is based on (4.20-rc2).
> 
> >>> > > - 0x3FE (any ideas)?
> >>> >
> >>> > Serial port poking?
> >>> >
> >>> > Again would be interesting to know the Linux code that's poking
> >>> > this.
> >>> >
> >>> > > - 0x42, 0x43, 0x61
> 
> These are in pit_calibrate_tsc.

Right, there's no way I'm aware of to signal there's no PIT. Again we
could consider trying to add a field to the IA-PC boot flags field for
this, but I'm not sure how useful it would be since quite a lot of
stuff relies on the existence of a PIT, and by the time FADT gets
checked it might be too late. I've seen the PIT being used very early
during boot due to the fact that it's assumed to be always present on
PC systems.

> The guest was taking this path initially, but
> right now we are abusing cpuid.15h by returning the TSC hz in ecx rather
> than the core crystal. We made a small, temporary hack to the pv interface/linux
> to get this working:
> 
> https://github.com/connojd/linux/commit/6b620ebc4f56129ed068d9995d59e46521abc80b
> 
> In all our attempts to convince linux that TSC is reliable, it inevitably decided that it
> in fact was not. Any thoughts on how to make this not be so dirty?

I'm not that familiar with Linux, do you know why Linux decides the TSC
is not reliable?

Is this based on CPU model, CPUID features, or some kind of test?

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH Whitelist Results / Windows Dom0

[Roger Pau Monné]

On Tue, Dec 04, 2018 at 05:46:38AM +0000, Connor Davis wrote:
> >>> > > - 0x70
> >>> > > - 0x71
> 
> These are accessed from reassert_nmi. This is only called from default_do_nmi
> in the version the guest is based on (4.20-rc2).

Ops, forgot to answer this one.

Xen sets the 'CMOS RTC Not Present' flag in the IA-PC bootflags, those
registers however are shared with the CMOS RAM.

I have to admit I'm not sure how this CMOS RTC flag is expected to
behave, since the CMOS RTC / RAM implementation seems to be completely
intertwined. IMO it would make sense to have a flag to disable CMOS
completely, but then there are NMI related registers in those IO
ports, so I'm afraid there isn't much that can be done here, apart
from making sure the non-RTC functionality is good enough for
guests.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH Whitelist Results / Windows Dom0

[Jan Beulich]

>>> On 04.12.18 at 18:50, <roger.pau@citrix.com> wrote:
> On Tue, Dec 04, 2018 at 05:46:38AM +0000, Connor Davis wrote:
>> >>> > > - 0x70
>> >>> > > - 0x71
>> 
>> These are accessed from reassert_nmi. This is only called from default_do_nmi
>> in the version the guest is based on (4.20-rc2).
> 
> Ops, forgot to answer this one.
> 
> Xen sets the 'CMOS RTC Not Present' flag in the IA-PC bootflags, those
> registers however are shared with the CMOS RAM.
> 
> I have to admit I'm not sure how this CMOS RTC flag is expected to
> behave, since the CMOS RTC / RAM implementation seems to be completely
> intertwined.

The first so many bytes in this space can be either RTC registers
or further bytes of CMOS RAM (to which various vendors have
created various other means of access mechanisms afaik). Hence
the flag set simply means there are no RTC registers there.
Whether CMOS RAM can be accessed through these ports is
unspecified by this flag being either set or clear, aiui (albeit I
think one might imply there is such RAM when the flag is set).

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel