Building on Ubuntu; and persistence_domain:cpu_cache

Building on Ubuntu; and persistence_domain:cpu_cache

[Eliot Moss]

Dear NVDimm folk:

I write concerning ndctl verion 72.1+ running on Ubuntu 22.04 (Linux
5.15.0-41-generic x86_64).

The system is a brand new two socket Dell server with cpu model Xeon GOld 6346
and 4 Tb of Optane DC P200 memory.

I am able to ue ndctl to configure the two regions with one namespace each in
fsdax mode.  Here is what ndctl list --namespaces -R prints:

{
   "regions":[
     {
       "dev":"region1",
       "size":2177548419072,
       "align":16777216,
       "available_size":0,
       "max_available_extent":0,
       "type":"pmem",
       "iset_id":-953140445588584312,
       "persistence_domain":"memory_controller",
       "namespaces":[
         {
           "dev":"namespace1.0",
           "mode":"fsdax",
           "map":"dev",
           "size":2143522127872,
           "uuid":"ed74879e-4eb6-4f88-bb7f-ae018d659720",
           "sector_size":512,
           "align":2097152,
           "blockdev":"pmem1",
           "name":"namespace1"
         }
       ]
     },
     {
       "dev":"region0",
       "size":2177548419072,
       "align":16777216,
       "available_size":0,
       "max_available_extent":0,
       "type":"pmem",
       "iset_id":-3109801715871676280,
       "persistence_domain":"memory_controller",
       "namespaces":[
         {
           "dev":"namespace0.0",
           "mode":"fsdax",
           "map":"dev",
           "size":2143522127872,
           "uuid":"64c75dc0-3d7a-4ac0-8698-8914e67b18db",
           "sector_size":512,
           "align":2097152,
           "blockdev":"pmem0",
           "name":"namespace0"
         }
       ]
     }
   ]
}

What concerns me is that it shows "persistence_domain":"memory_controller"
when I think it should show the persistence domain as "cpu_cache", since this
system is supposed to support eADR.

I wondered if maybe I needed the very latest version of ndctl for it to print
that, but I cannot build it.  I did my best to obtain the pre-reqs -- they
mostly have different names under Ubunut -- but the first meson command,
"meson setup build" hangs and if I then try the compile step it complains.

I am hoping someone here might be able to shed light on how I can verify that
this system does support persistence_domain cpu_cache, or if something needs
to be enabled or turned on, etc.  I have also gotten my Dell sales rep to
contact the Dell product engineers about this, but have not yet heard back.

Thanks for any light you might shed on either issue ...

Regards - Eliot Moss
Professor Emeritus, Computer and Information Sciences
Univ of Massachusetts Amherst

RE: Building on Ubuntu; and persistence_domain:cpu_cache

[Dan Williams]

Eliot Moss wrote:
> Dear NVDimm folk:
> 
> I write concerning ndctl verion 72.1+ running on Ubuntu 22.04 (Linux
> 5.15.0-41-generic x86_64).
> 
> The system is a brand new two socket Dell server with cpu model Xeon GOld 6346
> and 4 Tb of Optane DC P200 memory.
> 
> I am able to ue ndctl to configure the two regions with one namespace each in
> fsdax mode.  Here is what ndctl list --namespaces -R prints:
> 
> {
>    "regions":[
>      {
>        "dev":"region1",
>        "size":2177548419072,
>        "align":16777216,
>        "available_size":0,
>        "max_available_extent":0,
>        "type":"pmem",
>        "iset_id":-953140445588584312,
>        "persistence_domain":"memory_controller",
>        "namespaces":[
>          {
>            "dev":"namespace1.0",
>            "mode":"fsdax",
>            "map":"dev",
>            "size":2143522127872,
>            "uuid":"ed74879e-4eb6-4f88-bb7f-ae018d659720",
>            "sector_size":512,
>            "align":2097152,
>            "blockdev":"pmem1",
>            "name":"namespace1"
>          }
>        ]
>      },
>      {
>        "dev":"region0",
>        "size":2177548419072,
>        "align":16777216,
>        "available_size":0,
>        "max_available_extent":0,
>        "type":"pmem",
>        "iset_id":-3109801715871676280,
>        "persistence_domain":"memory_controller",
>        "namespaces":[
>          {
>            "dev":"namespace0.0",
>            "mode":"fsdax",
>            "map":"dev",
>            "size":2143522127872,
>            "uuid":"64c75dc0-3d7a-4ac0-8698-8914e67b18db",
>            "sector_size":512,
>            "align":2097152,
>            "blockdev":"pmem0",
>            "name":"namespace0"
>          }
>        ]
>      }
>    ]
> }
> 
> What concerns me is that it shows "persistence_domain":"memory_controller"
> when I think it should show the persistence domain as "cpu_cache", since this
> system is supposed to support eADR.

This is determined by:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/acpi/nfit/core.c?h=v5.15#n3022

The first thing to check is whether your ACPI tables have that bit set,
where that flag is coming from this table:

https://uefi.org/specs/ACPI/6.4/05_ACPI_Software_Programming_Model/ACPI_Software_Programming_Model.html#platform-capabilities-structure

You can dump the table with the acpica-tools (acpidump and iasl).  Some
examples of how to extract and disassemble a table is here (see the
usage of iasl in the "how does it work" section):

https://docs.kernel.org/admin-guide/acpi/initrd_table_override.html

> I wondered if maybe I needed the very latest version of ndctl for it to print
> that, but I cannot build it.  I did my best to obtain the pre-reqs -- they
> mostly have different names under Ubunut -- but the first meson command,
> "meson setup build" hangs and if I then try the compile step it complains.

ndctl is just packaging up the kernel's sysfs attribute data into JSON.
That data is coming direct from:

/sys/bus/devices/nd/$region/persistence_domain

> I am hoping someone here might be able to shed light on how I can verify that
> this system does support persistence_domain cpu_cache, or if something needs
> to be enabled or turned on, etc.  I have also gotten my Dell sales rep to
> contact the Dell product engineers about this, but have not yet heard back.

I can't help on that side, but if you do get a contact I expect the
acpidump result from above will be useful.

> Thanks for any light you might shed on either issue ...
> 
> Regards - Eliot Moss
> Professor Emeritus, Computer and Information Sciences
> Univ of Massachusetts Amherst
> 

Re: Building on Ubuntu; and persistence_domain:cpu_cache

[Eliot Moss]

On 7/22/2022 2:39 PM, Dan Williams wrote:
> Eliot Moss wrote:

>> What concerns me is that it shows "persistence_domain":"memory_controller"
>> when I think it should show the persistence domain as "cpu_cache", since this
>> system is supposed to support eADR.
> 
> This is determined by:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/acpi/nfit/core.c?h=v5.15#n3022
> 
> The first thing to check is whether your ACPI tables have that bit set,
> where that flag is coming from this table:
> 
> https://uefi.org/specs/ACPI/6.4/05_ACPI_Software_Programming_Model/ACPI_Software_Programming_Model.html#platform-capabilities-structure
> 
> You can dump the table with the acpica-tools (acpidump and iasl).  Some
> examples of how to extract and disassemble a table is here (see the
> usage of iasl in the "how does it work" section):
> 
> https://docs.kernel.org/admin-guide/acpi/initrd_table_override.html
> 
>> I wondered if maybe I needed the very latest version of ndctl for it to print
>> that, but I cannot build it.  I did my best to obtain the pre-reqs -- they
>> mostly have different names under Ubunut -- but the first meson command,
>> "meson setup build" hangs and if I then try the compile step it complains.
> 
> ndctl is just packaging up the kernel's sysfs attribute data into JSON.
> That data is coming direct from:
> 
> /sys/bus/devices/nd/$region/persistence_domain
> 
>> I am hoping someone here might be able to shed light on how I can verify that
>> this system does support persistence_domain cpu_cache, or if something needs
>> to be enabled or turned on, etc.  I have also gotten my Dell sales rep to
>> contact the Dell product engineers about this, but have not yet heard back.
> 
> I can't help on that side, but if you do get a contact I expect the
> acpidump result from above will be useful.
> 

Thank you, Dan!  The table in question is, I believe, the NFIT (NVDIMM
Firmware Information Table).  I can see a dump of all 488 bytes of it,
though I am not certain how to pick it apart.  Still, it would seem
that, if this system really *does* support persistence_domain:cpu_cache,
the NFIT is not set up correctly by the vendor.  However, I do wonder,
with technology like this, how much of what's going on lies on Intel's
side in terms of what the P200 boards report and how much on Dell's
side.  How these lower-level pieces go together and are configured lies
beyond my current education :-) ...

I await the Dell engineers' response to my queries.

Regards - Eliot

Re: Building on Ubuntu; and persistence_domain:cpu_cache

[Eliot Moss]

On 7/22/2022 3:08 PM, Eliot Moss wrote:
> On 7/22/2022 2:39 PM, Dan Williams wrote:
>> Eliot Moss wrote:
> 
>>> What concerns me is that it shows "persistence_domain":"memory_controller"
>>> when I think it should show the persistence domain as "cpu_cache", since this
>>> system is supposed to support eADR.

> Thank you, Dan!  The table in question is, I believe, the NFIT (NVDIMM
> Firmware Information Table).  I can see a dump of all 488 bytes of it,
> though I am not certain how to pick it apart.

A quick followup: I figured out how to parse the table by hand, and sure
enough, the relevant bit is not set.  So ndctl and friends are doing the
right thing.  The issue is either that the platform does not have the
capability we expected or that the NFIT is wrong and not reporting the
capability that the hardware actually provides.

At this point it really does seem to be a manufacturer issue of one kind
or another.

Regards - Eliot

Re: Building on Ubuntu; and persistence_domain:cpu_cache

[Dan Williams]

Eliot Moss wrote:
> On 7/22/2022 3:08 PM, Eliot Moss wrote:
> > On 7/22/2022 2:39 PM, Dan Williams wrote:
> >> Eliot Moss wrote:
> > 
> >>> What concerns me is that it shows "persistence_domain":"memory_controller"
> >>> when I think it should show the persistence domain as "cpu_cache", since this
> >>> system is supposed to support eADR.
> 
> > Thank you, Dan!  The table in question is, I believe, the NFIT (NVDIMM
> > Firmware Information Table).  I can see a dump of all 488 bytes of it,
> > though I am not certain how to pick it apart.
> 
> A quick followup: I figured out how to parse the table by hand, and sure
> enough, the relevant bit is not set.  So ndctl and friends are doing the
> right thing.  The issue is either that the platform does not have the
> capability we expected or that the NFIT is wrong and not reporting the
> capability that the hardware actually provides.

Glad you got it parsed, for future reference use iasl:

# cp /sys/firmware/acpi/tables/NFIT ./
# iasl -d NFIT

Intel ACPI Component Architecture
ASL+ Optimizing Compiler/Disassembler version 20220331
Copyright (c) 2000 - 2022 Intel Corporation

File appears to be binary: found 190 non-ASCII characters, disassembling
Binary file appears to be a valid ACPI table, disassembling
Input file NFIT, Length 0xE0 (224) bytes
ACPI: NFIT 0x0000000000000000 0000E0 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
Acpi Data Table [NFIT] decoded
Formatted output:  NFIT.dsl - 5355 bytes

# cat NFIT.dsl

/*
 * Intel ACPI Component Architecture
 * AML/ASL+ Disassembler version 20220331 (64-bit version)
 * Copyright (c) 2000 - 2022 Intel Corporation
 * 
 * Disassembly of NFIT, Fri Jul 22 19:54:16 2022
 *
 * ACPI Data Table [NFIT]
 *
 * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue (in hex)
 */

[000h 0000   4]                    Signature : "NFIT"    [NVDIMM Firmware Interface Table]
[004h 0004   4]                 Table Length : 000000E0
[008h 0008   1]                     Revision : 01
[009h 0009   1]                     Checksum : E3
[00Ah 0010   6]                       Oem ID : "BOCHS "
[010h 0016   8]                 Oem Table ID : "BXPC    "
[018h 0024   4]                 Oem Revision : 00000001
[01Ch 0028   4]              Asl Compiler ID : "BXPC"
[020h 0032   4]        Asl Compiler Revision : 00000001

[024h 0036   4]                     Reserved : 00000000

[028h 0040   2]                Subtable Type : 0000 [System Physical Address Range]
[02Ah 0042   2]                       Length : 0038

[02Ch 0044   2]                  Range Index : 0002
[02Eh 0046   2]        Flags (decoded below) : 0003
                   Add/Online Operation Only : 1
                      Proximity Domain Valid : 1
                       Location Cookie Valid : 0
[030h 0048   4]                     Reserved : 00000000
[034h 0052   4]             Proximity Domain : 00000000
[038h 0056  16]             Region Type GUID : 66F0D379-B4F3-4074-AC43-0D3318B78CDB
[048h 0072   8]           Address Range Base : 0000000480000000
[050h 0080   8]         Address Range Length : 0000001FC0000000
[058h 0088   8]         Memory Map Attribute : 0000000000008008
[060h 0096   8]              Location Cookie : 0000000100300001

...

Re: Building on Ubuntu; and persistence_domain:cpu_cache

[Eliot Moss]

On 7/22/2022 10:56 PM, Dan Williams wrote:
> Eliot Moss wrote:
>> On 7/22/2022 3:08 PM, Eliot Moss wrote:
>>> On 7/22/2022 2:39 PM, Dan Williams wrote:
>>>> Eliot Moss wrote:

> Glad you got it parsed, for future reference use iasl:
> 
> # cp /sys/firmware/acpi/tables/NFIT ./
> # iasl -d NFIT
> 
> Intel ACPI Component Architecture
> ASL+ Optimizing Compiler/Disassembler version 20220331
> Copyright (c) 2000 - 2022 Intel Corporation
> 
> File appears to be binary: found 190 non-ASCII characters, disassembling
> Binary file appears to be a valid ACPI table, disassembling
> Input file NFIT, Length 0xE0 (224) bytes
> ACPI: NFIT 0x0000000000000000 0000E0 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
> Acpi Data Table [NFIT] decoded
> Formatted output:  NFIT.dsl - 5355 bytes
> 
> # cat NFIT.dsl
> 
> /*
>   * Intel ACPI Component Architecture
>   * AML/ASL+ Disassembler version 20220331 (64-bit version)
>   * Copyright (c) 2000 - 2022 Intel Corporation
>   *
>   * Disassembly of NFIT, Fri Jul 22 19:54:16 2022
>   *
>   * ACPI Data Table [NFIT]
>   *
>   * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue (in hex)
>   */
> 
> [000h 0000   4]                    Signature : "NFIT"    [NVDIMM Firmware Interface Table]
> [004h 0004   4]                 Table Length : 000000E0
> [008h 0008   1]                     Revision : 01
> [009h 0009   1]                     Checksum : E3
> [00Ah 0010   6]                       Oem ID : "BOCHS "
> [010h 0016   8]                 Oem Table ID : "BXPC    "
> [018h 0024   4]                 Oem Revision : 00000001
> [01Ch 0028   4]              Asl Compiler ID : "BXPC"
> [020h 0032   4]        Asl Compiler Revision : 00000001
> 
> [024h 0036   4]                     Reserved : 00000000
> 
> [028h 0040   2]                Subtable Type : 0000 [System Physical Address Range]
> [02Ah 0042   2]                       Length : 0038
> 
> [02Ch 0044   2]                  Range Index : 0002
> [02Eh 0046   2]        Flags (decoded below) : 0003
>                     Add/Online Operation Only : 1
>                        Proximity Domain Valid : 1
>                         Location Cookie Valid : 0
> [030h 0048   4]                     Reserved : 00000000
> [034h 0052   4]             Proximity Domain : 00000000
> [038h 0056  16]             Region Type GUID : 66F0D379-B4F3-4074-AC43-0D3318B78CDB
> [048h 0072   8]           Address Range Base : 0000000480000000
> [050h 0080   8]         Address Range Length : 0000001FC0000000
> [058h 0088   8]         Memory Map Attribute : 0000000000008008
> [060h 0096   8]              Location Cookie : 0000000100300001

Thank you, Dan - I had figured that out in the mean time.

For future readers of this thread, ipmctl is also handy for this:

ipmctl show -system NFIT

...
    ---TableType=0x7
       Length: 16 bytes
       TypeEquals: PlatformCapabilities
       HighestValidCapability: 0x02
       Capabilities: 0x00000002
       Capabilities.CPUCacheFlushToNVDIMM: 0
       Capabilities.MemoryControllerFlushToNVDIMM: 1
       Capabilities.MemoryMirroring: 0

-- Eliot

RE: Building on Ubuntu; and persistence_domain:cpu_cache

[Dan Williams]

Eliot Moss wrote:
> Dear NVDimm folk:
> 
> I write concerning ndctl verion 72.1+ running on Ubuntu 22.04 (Linux
> 5.15.0-41-generic x86_64).
> 
> The system is a brand new two socket Dell server with cpu model Xeon GOld 6346
> and 4 Tb of Optane DC P200 memory.
> 
> I am able to ue ndctl to configure the two regions with one namespace each in
> fsdax mode.  Here is what ndctl list --namespaces -R prints:
> 
> {
>    "regions":[
>      {
>        "dev":"region1",
>        "size":2177548419072,
>        "align":16777216,
>        "available_size":0,
>        "max_available_extent":0,
>        "type":"pmem",
>        "iset_id":-953140445588584312,
>        "persistence_domain":"memory_controller",
>        "namespaces":[
>          {
>            "dev":"namespace1.0",
>            "mode":"fsdax",
>            "map":"dev",
>            "size":2143522127872,
>            "uuid":"ed74879e-4eb6-4f88-bb7f-ae018d659720",
>            "sector_size":512,
>            "align":2097152,
>            "blockdev":"pmem1",
>            "name":"namespace1"
>          }
>        ]
>      },
>      {
>        "dev":"region0",
>        "size":2177548419072,
>        "align":16777216,
>        "available_size":0,
>        "max_available_extent":0,
>        "type":"pmem",
>        "iset_id":-3109801715871676280,
>        "persistence_domain":"memory_controller",
>        "namespaces":[
>          {
>            "dev":"namespace0.0",
>            "mode":"fsdax",
>            "map":"dev",
>            "size":2143522127872,
>            "uuid":"64c75dc0-3d7a-4ac0-8698-8914e67b18db",
>            "sector_size":512,
>            "align":2097152,
>            "blockdev":"pmem0",
>            "name":"namespace0"
>          }
>        ]
>      }
>    ]
> }
> 
> What concerns me is that it shows "persistence_domain":"memory_controller"
> when I think it should show the persistence domain as "cpu_cache", since this
> system is supposed to support eADR.

FYI, I notice this topic was discussed on the 'pmem' list:

https://groups.google.com/g/pmem/c/K35X70fzAMw/m/5qEhhzb8AAAJ