Re: [RFC Design Doc] Add vNVDIMM support for Xen

From: Haozhong Zhang <haozhong.zhang@intel.com>
To: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Juergen Gross <jgross@suse.com>,
	Kevin Tian <kevin.tian@intel.com>, Wei Liu <wei.liu2@citrix.com>,
	Ian Campbell <ian.campbell@citrix.com>,
	George Dunlap <George.Dunlap@eu.citrix.com>,
	Stefano Stabellini <stefano.stabellini@eu.citrix.com>,
	Ian Jackson <ian.jackson@eu.citrix.com>,
	xen-devel@lists.xen.org, Jan Beulich <jbeulich@suse.com>,
	Jun Nakajima <jun.nakajima@intel.com>,
	Xiao Guangrong <guangrong.xiao@linux.intel.com>,
	Keir Fraser <keir@xen.org>
Subject: Re: [RFC Design Doc] Add vNVDIMM support for Xen
Date: Tue, 2 Feb 2016 11:44:35 +0800	[thread overview]
Message-ID: <20160202034435.GH6293@hz-desktop.sh.intel.com> (raw)
In-Reply-To: <56AFA319.8070801@citrix.com>

On 02/01/16 18:25, Andrew Cooper wrote:
> On 01/02/16 05:44, Haozhong Zhang wrote:
> > Hi,
> >
> > The following document describes the design of adding vNVDIMM support
> > for Xen. Any comments are welcome.
> >
> > Thanks,
> > Haozhong
> 
> Thankyou for doing this.  It is a very comprehensive document, and a
> fantastic example for future similar situations.
> 
> 
> To start with however, I would like to clear up my confusion over the
> the usecases of pmem vs pblk.
> 
> pblk, using indirect access, is less efficient than pmem.  NVDIMMs
> themselves are slower (and presumably more expensive) than equivalent
> RAM, and presumably still has a finite number of write cycles,  so I
> don't buy an argument suggesting that they are a plausible replacement
> for real RAM.
> 
> I presume therefore that a system would only choose to use pblk mode in
> situations where the host physical address space is a limiting factor. 
> Are there other situations which I have overlooked?
>

Limited physical address space is one concern. Another concern is that
pblk can be used by drivers to provide better RAS, like better error
detection and power-fail write atomicity. See Section "NVDIMM Driver"
in Chapter 1 of [3] for more details.

> Secondly, I presume that pmem vs pblk will be a firmware decision and
> fixed from the point of view of the Operating System?
>

Specifications on my hands [1-4] do not mention which one is in charge
for partitioning NVDIMM into pmem and pblk. However, as NFIT uses
separated SPA range structures for pmem and pblk regions, I also
presume that firmware (BIOS/EFI, or firmware on NVDIMM devices)
determines the partition.

In addition, some NVDIMM vendors may provide specific _DSM commands to
allow software (OS/drivers) to reconfigure the pmem/pblk partition,
but those changes only take effect after reboot. If OS/drivers or
system administrators decide to do so, IMO they should make sure no
users are currently using those NVDIMMs and data on NVDIMMs is already
properly handled.

Haozhong