Re: [RFC Design Doc v2] Add vNVDIMM support for Xen

[Haozhong Zhang <haozhong.zhang@intel.com>]

On 08/03/16 19:16, Konrad Rzeszutek Wilk wrote:
> > > 1.4 clwb/clflushopt
> > > 
> > >  Writes to NVDIMM may be cached by caches, so certain flushing
> > >  operations should be performed to make them persistent on
> > >  NVDIMM. clwb is used in favor of clflushopt and clflush to flush
> > >  writes from caches to memory.
> > > 
> > >  Details of clwb/clflushopt can be found in Chapter 10 of [6].
> > 
> > Didn't that opcode get dropped in favour of poking in some register?
> 
> Nevermind. I got this confused with pcommit which was deprecated.
> 
> But looking at chapter 10.2.2 it mentions that to commit to
> persistent memory you need to use pcommit. So what is the story here?

The document has not been updated yet, though patches to revert
pcommit support for both Linux and Xen had been merged.

> .. snip...
> > >  A part not put in above figure is enabling guest clwb/clflushopt
> > >  which exposes those instructions to guest via guest cpuid.
> > 
> > And aren't those deprecated?
> 
> And again. Ignore that comment.
> .. snip..
> > > 4. Design of vNVDIMM in Xen
> > > 
> > >  As KVM/QEMU, our design currently only provides pmem vNVDIMM.
> > > 
> > >  Similarly to that in KVM/QEMU, enabling vNVDIMM in Xen is composed of
> > >  three parts:
> > >  (1) Guest clwb/clflushopt enabling,
> > >  (2) pmem address management, and
> > >  (3) Guest ACPI emulation.
> > > 
> > >  The rest of this section present the design of each part
> > >  respectively. The basic design principle to reuse existing code in
> > >  Linux NVDIMM driver, QEMU and Xen as much as possible.
> > > 
> > > 
> > > 4.1 Guest clwb/clflushopt Enabling
> > > 
> > >  The instruction enabling is simple and we do the same work as in KVM/QEMU:
> > >  - clwb/clflushopt are exposed to guest via guest cpuid.
> > > 
> > 
> > Again, isn't that deprecated and the new mechanism (pokng at some register)
> > has to be used?
> 
> So clflushopt can be used for flushing out a cacheline. But what
> to do about store in the non-volatile memory? I recall that you could
> do an sfence and then pcommit, which would be aking to an SCSI SYNC
> command.
> 
> But with pcommit being deprecated (albeit the URL you pointed too
> still lists pcommit) - at least in Xen and Linux - how do you
> enforce this wholesale flush?
>

After deprecating pcommit, at least one of following two approaches
should be provided by HW to guarantee persistent:

1) Asynchronous DRAM refresh (ADR)
   If the platform supports ADR, flush CPU cache lines (e.g. by
   clwb/clflushopt/clflush) will result result in flush write pending
   queues in memory controller to NVDIMM.
   
2) ACPI Flush Hint Address Structure
   If ACPI flush hint address structure is available for a NVDIMM
   region, software can write to that structure to flush any preceding
   stores to that NVDIMM region. (Section 5.2.25.8 of ACPI Spec 6.1)

ADR is preferred, as guest clwb/clflushopt/clflush do not introduce
VMEXITs.

However, I'm also going to emulate ACPI flush hint address structure
in case of the lack of ADR. Basically, guest writes to ACPI flush hint
address structure will be trapped to QEMU which will submit them to
the host ACPI flush hint address structure via Dom0 NVDIMM driver.

If neither ADR nor ACPI flush hint address structure is available,
persistent can not be guaranteed.

Thanks,
Haozhong

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel