About two-dimensional page translation (e.g., Intel EPT) and shadow page table in Linux QEMU/KVM

About two-dimensional page translation (e.g., Intel EPT) and shadow page table in Linux QEMU/KVM

[harry harry]

Hi all,

I hope you are very well! May I know whether it is possible to enable
two-dimensional page translation (e.g., Intel EPT) mechanisms and
shadow page table mechanisms in Linux QEMU/KVM at the same time on a
physical server? For example, if the physical server has 80 cores, is
it possible to let 40 cores use Intel EPT mechanisms for page
translation and the other 40 cores use shadow page table mechanisms?
Thanks!

Best,
Harry

Re: About two-dimensional page translation (e.g., Intel EPT) and shadow page table in Linux QEMU/KVM

[Maxim Levitsky]

On Sun, 2021-07-11 at 15:13 -0500, harry harry wrote:
> Hi all,
> 
> I hope you are very well! May I know whether it is possible to enable
> two-dimensional page translation (e.g., Intel EPT) mechanisms and
> shadow page table mechanisms in Linux QEMU/KVM at the same time on a
> physical server? For example, if the physical server has 80 cores, is
> it possible to let 40 cores use Intel EPT mechanisms for page
> translation and the other 40 cores use shadow page table mechanisms?
> Thanks!

Nope sadly. EPT/NPT is enabled by a module param.

Best regards,
	Maxim Levitsky

> 
> Best,
> Harry
> 

Re: About two-dimensional page translation (e.g., Intel EPT) and shadow page table in Linux QEMU/KVM

[harry harry]

Dear Maxim,

Thanks for your reply. I knew, in our current design/implementation,
EPT/NPT is enabled by a module param. I think it is possible to modify
the QEMU/KVM code to let it support EPT/NPT and show page table (SPT)
simultaneously (e.g., for an 80-core server, 40 cores use EPT/NPT and
the other 40 cores use SPT). What do you think? Thanks!

Best regards,
Harry

On Mon, Jul 12, 2021 at 4:49 AM Maxim Levitsky <mlevitsk@redhat.com> wrote:
>
> On Sun, 2021-07-11 at 15:13 -0500, harry harry wrote:
> > Hi all,
> >
> > I hope you are very well! May I know whether it is possible to enable
> > two-dimensional page translation (e.g., Intel EPT) mechanisms and
> > shadow page table mechanisms in Linux QEMU/KVM at the same time on a
> > physical server? For example, if the physical server has 80 cores, is
> > it possible to let 40 cores use Intel EPT mechanisms for page
> > translation and the other 40 cores use shadow page table mechanisms?
> > Thanks!
>
> Nope sadly. EPT/NPT is enabled by a module param.
>
> Best regards,
>         Maxim Levitsky
>
> >
> > Best,
> > Harry
> >
>
>

Re: About two-dimensional page translation (e.g., Intel EPT) and shadow page table in Linux QEMU/KVM

[Maxim Levitsky]

On Mon, 2021-07-12 at 08:02 -0500, harry harry wrote:
> Dear Maxim,
> 
> Thanks for your reply. I knew, in our current design/implementation,
> EPT/NPT is enabled by a module param. I think it is possible to modify
> the QEMU/KVM code to let it support EPT/NPT and show page table (SPT)
> simultaneously (e.g., for an 80-core server, 40 cores use EPT/NPT and
> the other 40 cores use SPT). What do you think? Thanks!
> 
> Best regards,
> Harry
> 
> On Mon, Jul 12, 2021 at 4:49 AM Maxim Levitsky <mlevitsk@redhat.com> wrote:
> > On Sun, 2021-07-11 at 15:13 -0500, harry harry wrote:
> > > Hi all,
> > > 
> > > I hope you are very well! May I know whether it is possible to enable
> > > two-dimensional page translation (e.g., Intel EPT) mechanisms and
> > > shadow page table mechanisms in Linux QEMU/KVM at the same time on a
> > > physical server? For example, if the physical server has 80 cores, is
> > > it possible to let 40 cores use Intel EPT mechanisms for page
> > > translation and the other 40 cores use shadow page table mechanisms?
> > > Thanks!
> > 
> > Nope sadly. EPT/NPT is enabled by a module param.
> > 
> > Best regards,
> >         Maxim Levitsky
> > 
> > > Best,
> > > Harry
> > > 
For same VM, I don't think it is feasable.

For multiple VMs make some use NPT/EPT and some don't,
this should be possible to implement.

Why do you need it?

Best regards,
	Maxim Levitsky

Re: About two-dimensional page translation (e.g., Intel EPT) and shadow page table in Linux QEMU/KVM

[Sean Christopherson]

On Mon, Jul 12, 2021, Maxim Levitsky wrote:
> On Mon, 2021-07-12 at 08:02 -0500, harry harry wrote:
> > Dear Maxim,
> > 
> > Thanks for your reply. I knew, in our current design/implementation,
> > EPT/NPT is enabled by a module param. I think it is possible to modify
> > the QEMU/KVM code to let it support EPT/NPT and show page table (SPT)
> > simultaneously (e.g., for an 80-core server, 40 cores use EPT/NPT and
> > the other 40 cores use SPT). What do you think? Thanks!
> > 
> > Best regards,
> > Harry
> > 
> > On Mon, Jul 12, 2021 at 4:49 AM Maxim Levitsky <mlevitsk@redhat.com> wrote:
> > > On Sun, 2021-07-11 at 15:13 -0500, harry harry wrote:
> > > > Hi all,
> > > > 
> > > > I hope you are very well! May I know whether it is possible to enable
> > > > two-dimensional page translation (e.g., Intel EPT) mechanisms and
> > > > shadow page table mechanisms in Linux QEMU/KVM at the same time on a
> > > > physical server? For example, if the physical server has 80 cores, is
> > > > it possible to let 40 cores use Intel EPT mechanisms for page
> > > > translation and the other 40 cores use shadow page table mechanisms?
> > > > Thanks!
> > > 
> > > Nope sadly. EPT/NPT is enabled by a module param.
> > >
> For same VM, I don't think it is feasable.

Heh, because the MMUs are all per-vCPU, it actually wouldn't be that much effort
beyond supporting !TDP and TDP for different VMs...

> For multiple VMs make some use NPT/EPT and some don't,
> this should be possible to implement.

...but supporting !TDP and TDP in a single KVM instance isn't going to happen.
It's certainly possible, but comes with a very high complexity cost, and likely
even performance costs.

The more sane way to support !TDP and TDP on a single host would be to support
multiple instances of KVM, e.g. /dev/kvm0, /dev/kvm1, etc...  Being able to use
!TDP and TDP isn't strong justification for the work required, but supporting
multiple KVM instances would allow upgrading KVM without having to migrate VMs
off the host, which is very desirable.  If multiple KVM instances are supported,
running !TDP and TDP KVM instances should Just Work.

Re: About two-dimensional page translation (e.g., Intel EPT) and shadow page table in Linux QEMU/KVM

[harry harry]

Dear Sean,

Thanks for the comments!


> Heh, because the MMUs are all per-vCPU, it actually wouldn't be that much effort
> beyond supporting !TDP and TDP for different VMs...
>

Sorry, may I know what do you mean by "MMUs are all per-vCPU"? Do you
mean the MMUs walk the page tables of each vCPU?


> ...but supporting !TDP and TDP in a single KVM instance isn't going to happen.
> It's certainly possible, but comes with a very high complexity cost, and likely
> even performance costs.

For one KVM instance, I think it might be possible to let several
physical cores use !TDP and other cores use TDP but I am not sure
about the implementation complexity.

>
> The more sane way to support !TDP and TDP on a single host would be to support
> multiple instances of KVM, e.g. /dev/kvm0, /dev/kvm1, etc...  Being able to use
> !TDP and TDP isn't strong justification for the work required, but supporting
> multiple KVM instances would allow upgrading KVM without having to migrate VMs
> off the host, which is very desirable.  If multiple KVM instances are supported,
> running !TDP and TDP KVM instances should Just Work.

Yes, for different KVM instances, it may be much easier but there
might be some other issues, e.g., communication overhead between
different instances. I think the upgrading idea is great but is very
limited to local upgrading.

Best,
Harry

Re: About two-dimensional page translation (e.g., Intel EPT) and shadow page table in Linux QEMU/KVM

[Sean Christopherson]

On Wed, Jul 14, 2021, harry harry wrote:
> > Heh, because the MMUs are all per-vCPU, it actually wouldn't be that much effort
> > beyond supporting !TDP and TDP for different VMs...
> 
> Sorry, may I know what do you mean by "MMUs are all per-vCPU"? Do you
> mean the MMUs walk the page tables of each vCPU?

No, each vCPU has its own MMU instance, where an "MMU instance" is (mostly) a KVM
construct.  Per-vCPU MMU instances are necessary because each vCPU has its own
relevant state, e.g. CR0, CR4, EFER, etc..., that affects the MMU instance in
some way.  E.g. the MMU instance is used to walk guest page tables when
translating GVA->GPA for emulation, so per-vCPU MMUs are necessary even when
using TDP.

However, shadow/TDP PTEs are shared between compatible MMU instances.  E.g. in
the common case where all vCPUs in a VM use identical settings, there will
effectively be a single set of TDP page tables shared by all vCPUs.

Re: About two-dimensional page translation (e.g., Intel EPT) and shadow page table in Linux QEMU/KVM

[harry harry]

Hi Sean,

> No, each vCPU has its own MMU instance, where an "MMU instance" is (mostly) a KVM
> construct.  Per-vCPU MMU instances are necessary because each vCPU has its own
> relevant state, e.g. CR0, CR4, EFER, etc..., that affects the MMU instance in
> some way.  E.g. the MMU instance is used to walk guest page tables when
> translating GVA->GPA for emulation, so per-vCPU MMUs are necessary even when
> using TDP.
>
> However, shadow/TDP PTEs are shared between compatible MMU instances.  E.g. in
> the common case where all vCPUs in a VM use identical settings, there will
> effectively be a single set of TDP page tables shared by all vCPUs.

What do you mean by "MMU instance"? Do you mean VMCS? MMU is hardware.
Could you please share me the code of the MMU instance in KVM? Thanks!

Re: About two-dimensional page translation (e.g., Intel EPT) and shadow page table in Linux QEMU/KVM

[Sean Christopherson]

On Thu, Jul 15, 2021, harry harry wrote:
> Hi Sean,
> 
> > No, each vCPU has its own MMU instance, where an "MMU instance" is (mostly) a KVM
> > construct.  Per-vCPU MMU instances are necessary because each vCPU has its own
> > relevant state, e.g. CR0, CR4, EFER, etc..., that affects the MMU instance in
> > some way.  E.g. the MMU instance is used to walk guest page tables when
> > translating GVA->GPA for emulation, so per-vCPU MMUs are necessary even when
> > using TDP.
> >
> > However, shadow/TDP PTEs are shared between compatible MMU instances.  E.g. in
> > the common case where all vCPUs in a VM use identical settings, there will
> > effectively be a single set of TDP page tables shared by all vCPUs.
> 
> What do you mean by "MMU instance"? Do you mean VMCS? MMU is hardware.

No, an MMU is not a hardware-exclusive term, e.g. a software emulator will have
an MMU to emulate the MMU of the target hardware.

The terminology we use in KVM is roughly that a KVM MMU is KVM's presentation of
a hardware MMU to the guest.  E.g. when shadow paging is used, there is both the
hardware MMU that is stuffed with KVM's shadow PTEs, and the KVM MMU that models
the guest's MMU (the guest thinks its configuring a hardware MMU, but in reality
KVM is intercepting (some) guest PTE modifications).  When TDP (EPT) is used, the
hardware MMU has two parts: the TDP PTEs that are controlled by KVM, and the IA32
PTEs that are controlled by the guest.  And there's still a KVM MMU for the guest;
the KVM MMU in that case knows how to connfigure the TDP PTEs in hardware _and_
walk the guest IA32 PTEs, e.g. to handle memory accesses during emulation.

Even more fun, when nested TDP is used, there is a KVM MMU for L1, a KVM MMU for
L1's EPT for L2, a KVM MMU for L2 (L2's legacy page tables), and the hardware MMU.

> Could you please share me the code of the MMU instance in KVM? Thanks!

struct kvm_mmu, and generally speaking everything under arch/x86/kvm/mmu/.

Re: About two-dimensional page translation (e.g., Intel EPT) and shadow page table in Linux QEMU/KVM

[harry harry]

Hi Sean,

Thanks for the explanations. Please see my comments below. Thanks!

>  When TDP (EPT) is used, the
> hardware MMU has two parts: the TDP PTEs that are controlled by KVM, and the IA32
> PTEs that are controlled by the guest.  And there's still a KVM MMU for the guest;
> the KVM MMU in that case knows how to connfigure the TDP PTEs in hardware _and_
> walk the guest IA32 PTEs, e.g. to handle memory accesses during emulation.

Sorry, I could not understand why the emulated MMU is still needed
when TDP (e.g., Intel EPT) is used?
In particular, in what situations, we need the emulated MMU to
configure the TDP PTEs in hardware and walk the guest IA32 PTEs?
Why do we need the emulated MMU in these situations?

Best,
Harry

Re: About two-dimensional page translation (e.g., Intel EPT) and shadow page table in Linux QEMU/KVM

[Sean Christopherson]

On Thu, Jul 15, 2021, harry harry wrote:
> Hi Sean,
> 
> Thanks for the explanations. Please see my comments below. Thanks!
> 
> >  When TDP (EPT) is used, the hardware MMU has two parts: the TDP PTEs that
> >  are controlled by KVM, and the IA32 PTEs that are controlled by the guest.
> >  And there's still a KVM MMU for the guest; the KVM MMU in that case knows
> >  how to connfigure the TDP PTEs in hardware _and_ walk the guest IA32 PTEs,
> >  e.g. to handle memory accesses during emulation.
> 
> Sorry, I could not understand why the emulated MMU is still needed
> when TDP (e.g., Intel EPT) is used?
> In particular, in what situations, we need the emulated MMU to
> configure the TDP PTEs in hardware and walk the guest IA32 PTEs?

Ignoring some weird corner cases that blur the lines between emulation and
hardware configuration, the emulated IA32 MMU isn't used to configure TDP PTEs in
hardware, it's only used to walk the the guest page tables.

> Why do we need the emulated MMU in these situations?

For emulation of any instruction/flow that starts with a guest virtual address.
On Intel CPUs, that includes quite literally any "full" instruction emulation,
since KVM needs to translate CS:RIP to a guest physical address in order to fetch
the guest's code stream.  KVM can't avoid "full" emulation unless the guest is
heavily enlightened, e.g. to avoid string I/O, among many other things.

Re: About two-dimensional page translation (e.g., Intel EPT) and shadow page table in Linux QEMU/KVM

[harry harry]

Sean, sorry for the late reply. Thanks for your careful explanations.

> For emulation of any instruction/flow that starts with a guest virtual address.
> On Intel CPUs, that includes quite literally any "full" instruction emulation,
> since KVM needs to translate CS:RIP to a guest physical address in order to fetch
> the guest's code stream.  KVM can't avoid "full" emulation unless the guest is
> heavily enlightened, e.g. to avoid string I/O, among many other things.

Do you mean the emulated MMU is needed when it *only* wants to
translate GVAs to GPAs in the guest level?
In such cases, the hardware MMU cannot be used because hardware MMU
can only translate GVAs to HPAs, right?

Re: About two-dimensional page translation (e.g., Intel EPT) and shadow page table in Linux QEMU/KVM

[Sean Christopherson]

On Wed, Jul 28, 2021, harry harry wrote:
> Sean, sorry for the late reply. Thanks for your careful explanations.
> 
> > For emulation of any instruction/flow that starts with a guest virtual address.
> > On Intel CPUs, that includes quite literally any "full" instruction emulation,
> > since KVM needs to translate CS:RIP to a guest physical address in order to fetch
> > the guest's code stream.  KVM can't avoid "full" emulation unless the guest is
> > heavily enlightened, e.g. to avoid string I/O, among many other things.
> 
> Do you mean the emulated MMU is needed when it *only* wants to
> translate GVAs to GPAs in the guest level?

Not quite, though gva_to_gpa() is the main use.  The emulated MMU is also used to
inject guest #PF and to load/store guest PDTPRs.  

> In such cases, the hardware MMU cannot be used because hardware MMU
> can only translate GVAs to HPAs, right?

Sort of.  The hardware MMU does translate GVA to GPA, but the GPA value is not
visible to software (unless the GPA->HPA translation faults).  That's also true
for VA to PA (and GVA to HPA).  Irrespective of virtualization, x86 ISA doesn't
provide an instruction to retrive the PA for a given VA.

If such an instruction did exist, and it was to be usable for a VMM to do a
GVA->GPA translation, the magic instruction would need to take all MMU params as
operands, e.g. CR0, CR3, CR4, and EFER.  When KVM is active (not the guest), the
hardware MMU is loaded with the host MMU configuration, not the guest.  In both
VMX and SVM, vCPU state is mostly ephemeral in the sense that it ceases to exist
in hardware when the vCPU exits to the host.  Some state is retained in hardware,
e.g. TLB and cache entries, but those are associated with select properties of
the vCPU, e.g. EPTP, CR3, etc..., not with the vCPU itself, i.e. not with the
VMCS (VMX) / VMCB (SVM).

Re: About two-dimensional page translation (e.g., Intel EPT) and shadow page table in Linux QEMU/KVM

[harry harry]

Sean, understood with many thanks!

Good luck,
Harry

On Wed, Jul 28, 2021 at 3:01 PM Sean Christopherson <seanjc@google.com> wrote:
>
> On Wed, Jul 28, 2021, harry harry wrote:
> > Sean, sorry for the late reply. Thanks for your careful explanations.
> >
> > > For emulation of any instruction/flow that starts with a guest virtual address.
> > > On Intel CPUs, that includes quite literally any "full" instruction emulation,
> > > since KVM needs to translate CS:RIP to a guest physical address in order to fetch
> > > the guest's code stream.  KVM can't avoid "full" emulation unless the guest is
> > > heavily enlightened, e.g. to avoid string I/O, among many other things.
> >
> > Do you mean the emulated MMU is needed when it *only* wants to
> > translate GVAs to GPAs in the guest level?
>
> Not quite, though gva_to_gpa() is the main use.  The emulated MMU is also used to
> inject guest #PF and to load/store guest PDTPRs.
>
> > In such cases, the hardware MMU cannot be used because hardware MMU
> > can only translate GVAs to HPAs, right?
>
> Sort of.  The hardware MMU does translate GVA to GPA, but the GPA value is not
> visible to software (unless the GPA->HPA translation faults).  That's also true
> for VA to PA (and GVA to HPA).  Irrespective of virtualization, x86 ISA doesn't
> provide an instruction to retrive the PA for a given VA.
>
> If such an instruction did exist, and it was to be usable for a VMM to do a
> GVA->GPA translation, the magic instruction would need to take all MMU params as
> operands, e.g. CR0, CR3, CR4, and EFER.  When KVM is active (not the guest), the
> hardware MMU is loaded with the host MMU configuration, not the guest.  In both
> VMX and SVM, vCPU state is mostly ephemeral in the sense that it ceases to exist
> in hardware when the vCPU exits to the host.  Some state is retained in hardware,
> e.g. TLB and cache entries, but those are associated with select properties of
> the vCPU, e.g. EPTP, CR3, etc..., not with the vCPU itself, i.e. not with the
> VMCS (VMX) / VMCB (SVM).

Re: About two-dimensional page translation (e.g., Intel EPT) and shadow page table in Linux QEMU/KVM

[harry harry]

Dear Maxim,

Thanks for your reply!

> For same VM, I don't think it is feasable.
>
> For multiple VMs make some use NPT/EPT and some don't,
> this should be possible to implement.
>
> Why do you need it?
>

I am just curious about it :).


Best,
Harry