Re: [RFC Design Doc] Add vNVDIMM support for Xen

[Haozhong Zhang <haozhong.zhang@intel.com>]

On 02/02/16 14:15, Konrad Rzeszutek Wilk wrote:
> > 3. Design of vNVDIMM in Xen
> 
> Thank you for this design!
> 
> > 
> >  Similarly to that in KVM/QEMU, enabling vNVDIMM in Xen is composed of
> >  three parts:
> >  (1) Guest clwb/clflushopt/pcommit enabling,
> >  (2) Memory mapping, and
> >  (3) Guest ACPI emulation.
> 
> 
> .. MCE? and vMCE?
>

Specifications on my hand seem not mention much about MCE for NVDIMM,
but I remember that NVDIMM driver in Linux kernel does have MCE
code. I'll have a look at that code and add this part later.

> > 
> >  The rest of this section present the design of each part
> >  respectively. The basic design principle to reuse existing code in
> >  Linux NVDIMM driver and QEMU as much as possible. As recent
> >  discussions in the both Xen and QEMU mailing lists for the v1 patch
> >  series, alternative designs are also listed below.
> > 
> > 
> > 3.1 Guest clwb/clflushopt/pcommit Enabling
> > 
> >  The instruction enabling is simple and we do the same work as in KVM/QEMU.
> >  - All three instructions are exposed to guest via guest cpuid.
> >  - L1 guest pcommit is never intercepted by Xen.
> 
> I wish there was some watermarks like the PLE has.
> 
> My fear is that an unfriendly guest can issue sfence all day long
> flushing out other guests MMC queue (the writes followed by pcommits).
> Which means that an guest may have degraded performance as their
> memory writes are being flushed out immediately as if they were
> being written to UC instead of WB memory. 
>

pcommit takes no parameter and it seems hard to solve this problem
from hardware for now. And the current VMX does not provide mechanism
to limit the commit rate of pcommit like PLE for pause.

> In other words - the NVDIMM resource does not provide any resource
> isolation. However this may not be any different than what we had
> nowadays with CPU caches.
>

Does Xen have any mechanism to isolate multiple guests' operations on
CPU caches?

> 
> >  - L1 hypervisor is allowed to intercept L2 guest pcommit.
> 
> clwb?
>

VMX is not capable to intercept clwb. Any reason to intercept it?

> > 
> > 
> > 3.2 Address Mapping
> > 
> > 3.2.1 My Design
> > 
> >  The overview of this design is shown in the following figure.
> > 
> >                  Dom0                         |               DomU
> >                                               |
> >                                               |
> >  QEMU                                         |
> >      +...+--------------------+...+-----+     |
> >   VA |   | Label Storage Area |   | buf |     |
> >      +...+--------------------+...+-----+     |
> >                      ^            ^     ^     |
> >                      |            |     |     |
> >                      V            |     |     |
> >      +-------+   +-------+        mmap(2)     |
> >      | vACPI |   | v_DSM |        |     |     |        +----+------------+
> >      +-------+   +-------+        |     |     |   SPA  |    | /dev/pmem0 |
> >          ^           ^     +------+     |     |        +----+------------+
> >  --------|-----------|-----|------------|--   |             ^            ^
> >          |           |     |            |     |             |            |
> >          |    +------+     +------------~-----~-------------+            |
> >          |    |            |            |     |        XEN_DOMCTL_memory_mapping
> >          |    |            |            +-----~--------------------------+
> >          |    |            |            |     |
> >          |    |       +----+------------+     |
> >  Linux   |    |   SPA |    | /dev/pmem0 |     |     +------+   +------+
> >          |    |       +----+------------+     |     | ACPI |   | _DSM |
> >          |    |                   ^           |     +------+   +------+
> >          |    |                   |           |         |          |
> >          |    |               Dom0 Driver     |   hvmloader/xl     |
> >  --------|----|-------------------|---------------------|----------|---------------
> >          |    +-------------------~---------------------~----------+
> >  Xen     |                        |                     |
> >          +------------------------~---------------------+
> >  ---------------------------------|------------------------------------------------
> >                                   +----------------+
> >                                                    |
> >                                             +-------------+
> >  HW                                         |    NVDIMM   |
> >                                             +-------------+
> > 
> > 
> >  This design treats host NVDIMM devices as ordinary MMIO devices:
> 
> Nice.
> 
> But it also means you need Xen to 'share' the ranges of an MMIO device.
> 
> That is you may need dom0 _DSM method to access certain ranges
> (the AML code may need to poke there) - and the guest may want to access
> those as well.
>

Currently, we are going to support _DSM that queries supported _DSM
commands and accesses vNVDIMM's label storage area. Both are emulated
by QEMU and not applied to host NVDIMM.

> And keep in mind that this NVDIMM management may not need to be always
> in initial domain.

I guess you mean it can be in a dedicated driver domain,

> As in you could have NVDIMM device drivers that would
> carve out the ranges to guests.

but I don't get what you mean here. More hints?

[...] 
> > 3.2.2 Alternative Design
> > 
> >  Jan Beulich's comments [7] on my question "why must pmem resource
> >  management and partition be done in hypervisor":
> >  | Because that's where memory management belongs. And PMEM,
> >  | other than PBLK, is just another form of RAM.
> >  | ...
> >  | The main issue is that this would imo be a layering violation
> > 
> >  George Dunlap's comments [8]:
> >  | This is not the case for PMEM.  The whole point of PMEM (correct me if
> >    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ used as fungible ram
> >  | I'm wrong) is to be used for long-term storage that survives over
> >  | reboot.  It matters very much that a guest be given the same PRAM
> >  | after the host is rebooted that it was given before.  It doesn't make
> >  | any sense to manage it the way Xen currently manages RAM (i.e., that
> >  | you request a page and get whatever Xen happens to give you).
> >  |
> >  | So if Xen is going to use PMEM, it will have to invent an entirely new
> >  | interface for guests, and it will have to keep track of those
> >  | resources across host reboots.  In other words, it will have to
> >  | duplicate all the work that Linux already does.  What do we gain from
> >  | that duplication?  Why not just leverage what's already implemented in
> >  | dom0?
> >  and [9]:
> >  | Oh, right -- yes, if the usage model of PRAM is just "cheap slow RAM",
> >  | then you're right -- it is just another form of RAM, that should be
> >  | treated no differently than say, lowmem: a fungible resource that can be
> >  | requested by setting a flag.
> > 
> >  However, pmem is used more as persistent storage than fungible ram,
> >  and my design is for the former usage. I would like to leave the
> >  detection, driver and partition (either through namespace or file
> >  systems) of NVDIMM in Dom0 Linux kernel.
> > 
> >  I notice that current XEN_DOMCTL_memory_mapping does not make santiy
> >  check for the physical address and size passed from caller
> >  (QEMU). Can QEMU be always trusted? If not, we would need to make Xen
> >  aware of the SPA range of pmem so that it can refuse map physical
> >  address in neither the normal ram nor pmem.
> 
> /me nods.
> > 
> >  Instead of duplicating the detection code (parsing NFIT and
> >  evaluating _FIT) in Dom0 Linux kernel, we decide to patch Dom0 Linux
> >  kernel to pass parameters of host pmem NVDIMM devices to Xen
> >  hypervisor:
> >  (1) Add a global
> >        struct rangeset pmem_rangeset
> >      in Xen hypervisor to record all SPA ranges of detected pmem devices.
> >      Each range in pmem_rangeset corresponds to a pmem device.
> > 
> >  (2) Add a hypercall
> >        XEN_SYSCTL_add_pmem_range
> >      (should it be a sysctl or a platform op?)
> >      that receives a pair of parameters (addr: starting SPA of pmem
> >      region, len: size of pmem region) and add a range (addr, addr +
> >      len - 1) in nvdimm_rangset.
> > 
> >  (3) Add a hypercall
> >        XEN_DOMCTL_pmem_mapping
> >      that takes the same parameters as XEN_DOMCTL_memory_mapping and
> >      maps a given host pmem range to guest. It checks whether the
> >      given host pmem range is in the pmem_rangeset before making the
> >      actual mapping.
> > 
> >  (4) Patch Linux NVDIMM driver to call XEN_SYSCTL_add_pmem_range
> >      whenever it detects a pmem device.
> > 
> >  (5) Patch QEMU to use XEN_DOMCTL_pmem_mapping for mapping host pmem
> >      devices.
> 
> That is nice - as you can instrument this on existing hardware and
> create 'fake' starting SPA for real memory - which Xen may not see
> due to being booted with 'mem=X'.
>

'mem=X' only limits the maximum address of normal ram. NVDIMM or other
MMIO devices are limited by it as well or not?

Thanks,
Haozhong