Microsoft and Intel NVDIMM ACPI _DSM interfaces status?

Microsoft and Intel NVDIMM ACPI _DSM interfaces status?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 725 bytes --]

Hi,
Microsoft and Intel developed two different ACPI NVDIMM _DSM interfaces.

The specs for the Intel interface are available here:
https://pmem.io/documents/NVDIMM_DSM_Interface_Example.pdf

This is the interface that QEMU emulates. It has been reported that
Windows 2016 Server and 2019 Server guests do not recognize QEMU's
emulated NVDIMM devices using the Microsoft driver.

I'd like to understand the path forward that will allow both Linux and
Windows guests to successfully use QEMU's emulated NVDIMM device
(https://gitlab.com/qemu-project/qemu/-/blob/master/hw/acpi/nvdimm.c).

Are/have these two interfaces being/been unified?

Should QEMU emulate both of them to make running Windows guests easy?

Thanks,
Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Microsoft and Intel NVDIMM ACPI _DSM interfaces status?

[Laszlo Ersek]

(Adding Dexuan Cui to the CC list, comments below.)

On 03/17/21 12:49, Stefan Hajnoczi wrote:
> Hi,
> Microsoft and Intel developed two different ACPI NVDIMM _DSM
> interfaces.
>
> The specs for the Intel interface are available here:
> https://pmem.io/documents/NVDIMM_DSM_Interface_Example.pdf
>
> This is the interface that QEMU emulates. It has been reported that
> Windows 2016 Server and 2019 Server guests do not recognize QEMU's
> emulated NVDIMM devices using the Microsoft driver.

The Microsoft specification seems to be available at:

  https://uefi.org/NVDIMM%20RFIC%20Registry
  ->
  https://uefi.org/RFIC_LIST
  ->
  https://www.uefi.org/sites/default/files/resources/_DSM%20for%20Virtual%20NVDIMMs%20v1.01.docx

It defines:

- Region Format Interface Code (RFIC) 0x1901,

- with a _DSM command set (for non-root devices) that is identified by
  UUID 5746C5F2-A9A2-4264-AD0E-E4DDC9E09E80.

On the other hand, the Intel document defines:

- RFIC 0x0201,

- with a _DSM command set (for non-root devices) that is identified by
  UUID 4309AC30-0D11-11E4-9191-0800200C9A66.

In the Linux kernel,

- the Microsoft UUID is called UUID_NFIT_DIMM_N_HYPERV (1194c4133195,
  "nfit: Add Hyper-V NVDIMM DSM command set to white list", 2019-01-29),

- while the Intel one is called UUID_NFIT_DIMM (commit b94d5230d06e,
  "libnvdimm, nfit: initial libnvdimm infrastructure and NFIT support",
  2015-06-24).

An interesting excerpt from commit 1194c4133195:

> +        * There are 4 "legacy" NVDIMM command sets
> +        * (NVDIMM_FAMILY_{INTEL,MSFT,HPE1,HPE2}) that were created before
> +        * an EFI working group was established to constrain this
> +        * proliferation. The nfit driver probes for the supported command
> +        * set by GUID. Note, if you're a platform developer looking to add
> +        * a new command set to this probe, consider using an existing set,
> +        * or otherwise seek approval to publish the command set at
> +        * http://www.uefi.org/RFIC_LIST.

And presently, the "official RFIC list" *only* contains Microsoft's
definition.

So my guess is that the QEMU device model (emulating Intel) predates
both the standardization and the registry, and that the Microsoft driver
only recognizes their own format (RFIC 0x1901 / UUID
5746C5F2-A9A2-4264-AD0E-E4DDC9E09E80).

Back to your email:

On 03/17/21 12:49, Stefan Hajnoczi wrote:
> I'd like to understand the path forward that will allow both Linux and
> Windows guests to successfully use QEMU's emulated NVDIMM device
> (https://gitlab.com/qemu-project/qemu/-/blob/master/hw/acpi/nvdimm.c).
>
> Are/have these two interfaces being/been unified?

I wouldn't think so. In the Linux kernel, UUID_NFIT_DIMM_N_HYPERV is
mapped to NFIT_DEV_DIMM_N_HYPERV is mapped to NVDIMM_FAMILY_HYPERV,
which seems to translate to "DSM mask" and "flags" manipulations...
which I don't understand. :)

FWIW, I believe the Linux kernel implements a "generic set" of NVDIMM
operations, and then it cherry-picks those features/operations that the
hardware (virtual or otherwise) advertizes, or seems to support.

This, and more closely the above-quoted code comment, appear to indicate
that there's no unification, at the interface level. The Linux guest
driver may have some implementation-level unification.

The first link I pasted above,
<https://uefi.org/NVDIMM%20RFIC%20Registry>, refers to "NVST Workgroup
Chairperson". After logging in to my <members.uefi.org> account, I
managed to resolve "NVST" as follows:

- ACPI Specification Working Group [ASWG]

  - NVDIMM Subteam [NVST]

    - https://members.uefi.org/apps/org/workgroup/nvst/description.php

      """
      Public Description

      The NVDIMM Subteam was created to review ACPI and UEFI related
      topics pertaining to persistent memory devices. All relevant ECR's
      need to be reviewed by this subcommittee prior to review by the
      USWG and ASWG. Please contact the group chair with questions or to
      add items to the regular meeting agenda.
      """

Some other abbreviations resolved, for interpreting the blurb:

- USWG: UEFI Spec Working Group

- ECR: Engineering Change Request -- basically a ticket filed for one of
  the UEFI, ACPI, and Platform Init specs, in the (members only) Mantis
  bug tracker, at <https://mantis.uefi.org/>. ECRs are usually proposed
  as stand-alone documents that are attached to Mantis tickets.


> Should QEMU emulate both of them to make running Windows guests easy?

In my (uneducated) opinion: yes. Microsoft standarized their Region
Format Interface, with their _DSM UUID and all; and right now, that spec
seems to be the *only* officially approved format in the RFIC registry.
So it's plausible to me that, unlike the Linux kernel, Microsoft's
driver doesn't support the -- technically unapproved, nonstandard --
Intel Region Format Interface.

Dexuan, please correct me if I'm wrong.

Thanks,
Laszlo

RE: Microsoft and Intel NVDIMM ACPI _DSM interfaces status?

[Dexuan Cui]

> From: Laszlo Ersek <lersek@redhat.com>
> Sent: Wednesday, March 17, 2021 3:45 PM
> > The specs for the Intel interface are available here:
> > ...
> > This is the interface that QEMU emulates. It has been reported that
> > Windows 2016 Server and 2019 Server guests do not recognize QEMU's
> > emulated NVDIMM devices using the Microsoft driver.

I'm not sure why this happens -- sorry, I have no Windows knowledge.
 
> > Should QEMU emulate both of them to make running Windows guests easy?

I'm not sure about the background here, but since it looks like QEMU is already
able to emulate the Intel NVDIMM, I suppose it should be quick to add the
emulation of the Hyper-V NVDIMM. I think they're pretty similar, and the
_DSM interface supported by Hyper-V NVDIMM is simple.
 
> In my (uneducated) opinion: yes. Microsoft standarized their Region
> Format Interface, with their _DSM UUID and all; and right now, that spec
> seems to be the *only* officially approved format in the RFIC registry.
> So it's plausible to me that, unlike the Linux kernel, Microsoft's
> driver doesn't support the -- technically unapproved, nonstandard --
> Intel Region Format Interface.
> 
> Dexuan, please correct me if I'm wrong.
> 
> Thanks,
> Laszlo

Hi Laszlo, I'm not 100% sure, but I guess your may be correct.

BTW, earlier in 2019, we made the below patches (which are in the mainline):

2019-02-28    libnvdimm/btt: Fix LBA masking during 'free list' population
2019-02-12    acpi/nfit: Require opt-in for read-only label configurations
2019-02-02    libnvdimm/dimm: Add a no-BLK quirk based on NVDIMM family
2019-01-29    nfit: Add Hyper-V NVDIMM DSM command set to white list
2019-01-29    nfit: acpi_nfit_ctl(): Check out_obj->type in the right place

The patches improve the interoperability between Windows VM and 
Linux VM, e.g. the same Hyper-V NVDIMM device can work this way:
the Windows VM creates an NTFS partition based on the device, and
creates a text file in the partition, and later we shut down the Windows VM
and assign the device to Linux VM, and Linux VM is able to read the text file.

Before the patches, IIRC, Linux VM could only use the Hyper-V NVIDMM
device in label-less mode.

Let me share some old 2019 notes about Hyper-V NVDIMM, in case the
info may be helpful to you: 

"
In Linux VM, IMO the label-less mode is preferred for Hyper-V NVDIMM,
because Hyper-V does not support _LSW (I'm not sure about the latest
status), so Dan made the patch "acpi/nfit: Require opt-in for read-only
label configurations" to not use the Hyper-V label info, by default.
In label-less mode, when creating a namespace, Linux can set it to
one of the 4 namespace modes: fsdax, devdax, sector, and raw (these
namespace modes are Linux-specific and can not be recognized
by Windows.). 

With the "nfit.force_labels" bootup-time kernel parameter, Linux can
be forced to be in label mode, and then if Hyper-V initializes the 4KB
BTT Info Block(s) with the standard EFI_BTT_ABSTRACTION_GUID
(which is defined in UEFI 2.7 spec), we're supposed to support the
"interoperability" between Windows VM and Linux VM.

Note: label-less mode is incompatible with label mode. A namespace
created in one mode can't be recognized when Linux runs in the other
mode. In label mode, so far, only 2 namespace modes (raw and sector)
can be supported by the Hyper-V NVDIMM, because Hyper-V doesn't
support _LSW, yet. If Hyper-V sets the EFI_BTT_ABSTRACTION_GUID,
the namespace is "BTT-capable" and can be in sector namespace
mode, otherwise it's in raw namespace mode.

After a Windows VM initializes a BTT-capable namespace in a Hyper-V
NVDIMM device by creating a NTFS file system in the namespace, we
can re-assign the Hyper-V NVDIMM device to Linux VM, and in label
mode Linux VM is supposed to be able to read and write the files in
the NTFS file system.
"

Thanks,
-- Dexuan

Re: Microsoft and Intel NVDIMM ACPI _DSM interfaces status?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4362 bytes --]

On Thu, Mar 18, 2021 at 02:00:29AM +0000, Dexuan Cui wrote:
> > From: Laszlo Ersek <lersek@redhat.com>
> > Sent: Wednesday, March 17, 2021 3:45 PM
> > > The specs for the Intel interface are available here:
> > > ...
> > > This is the interface that QEMU emulates. It has been reported that
> > > Windows 2016 Server and 2019 Server guests do not recognize QEMU's
> > > emulated NVDIMM devices using the Microsoft driver.
> 
> I'm not sure why this happens -- sorry, I have no Windows knowledge.
>  
> > > Should QEMU emulate both of them to make running Windows guests easy?
> 
> I'm not sure about the background here, but since it looks like QEMU is already
> able to emulate the Intel NVDIMM, I suppose it should be quick to add the
> emulation of the Hyper-V NVDIMM. I think they're pretty similar, and the
> _DSM interface supported by Hyper-V NVDIMM is simple.
>  
> > In my (uneducated) opinion: yes. Microsoft standarized their Region
> > Format Interface, with their _DSM UUID and all; and right now, that spec
> > seems to be the *only* officially approved format in the RFIC registry.
> > So it's plausible to me that, unlike the Linux kernel, Microsoft's
> > driver doesn't support the -- technically unapproved, nonstandard --
> > Intel Region Format Interface.
> > 
> > Dexuan, please correct me if I'm wrong.
> > 
> > Thanks,
> > Laszlo
> 
> Hi Laszlo, I'm not 100% sure, but I guess your may be correct.
> 
> BTW, earlier in 2019, we made the below patches (which are in the mainline):
> 
> 2019-02-28    libnvdimm/btt: Fix LBA masking during 'free list' population
> 2019-02-12    acpi/nfit: Require opt-in for read-only label configurations
> 2019-02-02    libnvdimm/dimm: Add a no-BLK quirk based on NVDIMM family
> 2019-01-29    nfit: Add Hyper-V NVDIMM DSM command set to white list
> 2019-01-29    nfit: acpi_nfit_ctl(): Check out_obj->type in the right place
> 
> The patches improve the interoperability between Windows VM and 
> Linux VM, e.g. the same Hyper-V NVDIMM device can work this way:
> the Windows VM creates an NTFS partition based on the device, and
> creates a text file in the partition, and later we shut down the Windows VM
> and assign the device to Linux VM, and Linux VM is able to read the text file.
> 
> Before the patches, IIRC, Linux VM could only use the Hyper-V NVIDMM
> device in label-less mode.
> 
> Let me share some old 2019 notes about Hyper-V NVDIMM, in case the
> info may be helpful to you: 
> 
> "
> In Linux VM, IMO the label-less mode is preferred for Hyper-V NVDIMM,
> because Hyper-V does not support _LSW (I'm not sure about the latest
> status), so Dan made the patch "acpi/nfit: Require opt-in for read-only
> label configurations" to not use the Hyper-V label info, by default.
> In label-less mode, when creating a namespace, Linux can set it to
> one of the 4 namespace modes: fsdax, devdax, sector, and raw (these
> namespace modes are Linux-specific and can not be recognized
> by Windows.). 
> 
> With the "nfit.force_labels" bootup-time kernel parameter, Linux can
> be forced to be in label mode, and then if Hyper-V initializes the 4KB
> BTT Info Block(s) with the standard EFI_BTT_ABSTRACTION_GUID
> (which is defined in UEFI 2.7 spec), we're supposed to support the
> "interoperability" between Windows VM and Linux VM.
> 
> Note: label-less mode is incompatible with label mode. A namespace
> created in one mode can't be recognized when Linux runs in the other
> mode. In label mode, so far, only 2 namespace modes (raw and sector)
> can be supported by the Hyper-V NVDIMM, because Hyper-V doesn't
> support _LSW, yet. If Hyper-V sets the EFI_BTT_ABSTRACTION_GUID,
> the namespace is "BTT-capable" and can be in sector namespace
> mode, otherwise it's in raw namespace mode.
> 
> After a Windows VM initializes a BTT-capable namespace in a Hyper-V
> NVDIMM device by creating a NTFS file system in the namespace, we
> can re-assign the Hyper-V NVDIMM device to Linux VM, and in label
> mode Linux VM is supposed to be able to read and write the files in
> the NTFS file system.
> "

Thank you, Laszlo and Dexuan!

I wonder if there are existing Windows drivers available that work with
QEMU's NVDIMM device. Otherwise it may make sense to implement the
Hyper-V interface.

Stefan

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Microsoft and Intel NVDIMM ACPI _DSM interfaces status?

[Dan Williams]

On Wed, Mar 17, 2021 at 4:49 AM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> Hi,
> Microsoft and Intel developed two different ACPI NVDIMM _DSM interfaces.
>
> The specs for the Intel interface are available here:
> https://pmem.io/documents/NVDIMM_DSM_Interface_Example.pdf
>
> This is the interface that QEMU emulates. It has been reported that
> Windows 2016 Server and 2019 Server guests do not recognize QEMU's
> emulated NVDIMM devices using the Microsoft driver.
>
> I'd like to understand the path forward that will allow both Linux and
> Windows guests to successfully use QEMU's emulated NVDIMM device
> (https://gitlab.com/qemu-project/qemu/-/blob/master/hw/acpi/nvdimm.c).
>
> Are/have these two interfaces being/been unified?

No, they service 2 different classes of NVDIMMs. The Microsoft
specification was defined for NVDIMM-N devices that are the
traditional battery/super-capacitor backed DDR with sometimes an equal
amount of flash to squirrel away data to non-volatile media on power
loss. The Intel one is for a persistent media class of device where
there is no need to communicate health attributes like "energy source
died" or "restore from flash" failed.

> Should QEMU emulate both of them to make running Windows guests easy?

Depends on how tolerant Windows is to different format-interface-code
implementations and what QEMU should return on each of the functions
it decides to implement.

Note that QEMU only implements a small subset of the Intel commands,
i.e. just enough to provision namespaces in the NVDIMM label area.
"NVDIMM label area" is a concept that is newer than the NVDIMM-N
definition which is why you don't see labels mentioned in the
Microsoft specification. Since then ACPI has developed common label
area access methods, see "6.5.10 NVDIMM Label Methods" in the ACPI 6.4
specification.

Note that you'll also see "9.20.8 NVDIMM Device Methods" in that spec
for some health management commands that overlap what the Microsoft
and Intel specs communicate. Linux does not support them since any
platform that might support them will also support the Intel
specification so there's no driving need for Linux to migrate. I do
not know the status of that command support in Windows, or the HPE
command support in Windows for that matter.

If you need to support guests that expect the Hyper-V command
definition, and will fail to attach NVDIMM support in the absence of
that definition then QEMU needs UUID_NFIT_DIMM_N_HYPERV support, note
that is a different command set than even UUID_NFIT_DIMM_N_MSFT
(include/acpi/acuuid.h). That would also require changes to virtual
ACPI NFIT to advertise the correlated format interface code. There may
be more... you would need someone from Hyper-V land to enumerate all
that is expected.

Re: Microsoft and Intel NVDIMM ACPI _DSM interfaces status?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3033 bytes --]

On Wed, Mar 17, 2021 at 04:52:03PM -0700, Dan Williams wrote:
> On Wed, Mar 17, 2021 at 4:49 AM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >
> > Hi,
> > Microsoft and Intel developed two different ACPI NVDIMM _DSM interfaces.
> >
> > The specs for the Intel interface are available here:
> > https://pmem.io/documents/NVDIMM_DSM_Interface_Example.pdf
> >
> > This is the interface that QEMU emulates. It has been reported that
> > Windows 2016 Server and 2019 Server guests do not recognize QEMU's
> > emulated NVDIMM devices using the Microsoft driver.
> >
> > I'd like to understand the path forward that will allow both Linux and
> > Windows guests to successfully use QEMU's emulated NVDIMM device
> > (https://gitlab.com/qemu-project/qemu/-/blob/master/hw/acpi/nvdimm.c).
> >
> > Are/have these two interfaces being/been unified?
> 
> No, they service 2 different classes of NVDIMMs. The Microsoft
> specification was defined for NVDIMM-N devices that are the
> traditional battery/super-capacitor backed DDR with sometimes an equal
> amount of flash to squirrel away data to non-volatile media on power
> loss. The Intel one is for a persistent media class of device where
> there is no need to communicate health attributes like "energy source
> died" or "restore from flash" failed.
> 
> > Should QEMU emulate both of them to make running Windows guests easy?
> 
> Depends on how tolerant Windows is to different format-interface-code
> implementations and what QEMU should return on each of the functions
> it decides to implement.
> 
> Note that QEMU only implements a small subset of the Intel commands,
> i.e. just enough to provision namespaces in the NVDIMM label area.
> "NVDIMM label area" is a concept that is newer than the NVDIMM-N
> definition which is why you don't see labels mentioned in the
> Microsoft specification. Since then ACPI has developed common label
> area access methods, see "6.5.10 NVDIMM Label Methods" in the ACPI 6.4
> specification.
> 
> Note that you'll also see "9.20.8 NVDIMM Device Methods" in that spec
> for some health management commands that overlap what the Microsoft
> and Intel specs communicate. Linux does not support them since any
> platform that might support them will also support the Intel
> specification so there's no driving need for Linux to migrate. I do
> not know the status of that command support in Windows, or the HPE
> command support in Windows for that matter.
> 
> If you need to support guests that expect the Hyper-V command
> definition, and will fail to attach NVDIMM support in the absence of
> that definition then QEMU needs UUID_NFIT_DIMM_N_HYPERV support, note
> that is a different command set than even UUID_NFIT_DIMM_N_MSFT
> (include/acpi/acuuid.h). That would also require changes to virtual
> ACPI NFIT to advertise the correlated format interface code. There may
> be more... you would need someone from Hyper-V land to enumerate all
> that is expected.
> 

Thanks!

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]