From: "Huang, Ying" <ying.huang@intel.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Mel Gorman <mgorman@suse.de>,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Feng Tang <feng.tang@intel.com>,
Andrew Morton <akpm@linux-foundation.org>,
Michal Hocko <mhocko@suse.com>, Rik van Riel <riel@surriel.com>,
Mel Gorman <mgorman@techsingularity.net>,
Dave Hansen <dave.hansen@linux.intel.com>,
Yang Shi <shy828301@gmail.com>, Zi Yan <ziy@nvidia.com>,
Wei Xu <weixugc@google.com>, osalvador <osalvador@suse.de>,
Shakeel Butt <shakeelb@google.com>,
Hasan Al Maruf <hasanalmaruf@fb.com>
Subject: Re: [PATCH -V10 RESEND 0/6] NUMA balancing: optimize memory placement for memory tiering system
Date: Thu, 13 Jan 2022 15:19:06 +0800 [thread overview]
Message-ID: <87sftsumqd.fsf@yhuang6-desk2.ccr.corp.intel.com> (raw)
In-Reply-To: <Yd79b6PptQMNzDRw@hirez.programming.kicks-ass.net> (Peter Zijlstra's message of "Wed, 12 Jan 2022 17:10:23 +0100")
[-- Attachment #1: Type: text/plain, Size: 3326 bytes --]
Hi, Peter,
Peter Zijlstra <peterz@infradead.org> writes:
> On Tue, Dec 07, 2021 at 10:27:51AM +0800, Huang Ying wrote:
>> After commit c221c0b0308f ("device-dax: "Hotplug" persistent memory
>> for use like normal RAM"), the PMEM could be used as the
>> cost-effective volatile memory in separate NUMA nodes. In a typical
>> memory tiering system, there are CPUs, DRAM and PMEM in each physical
>> NUMA node. The CPUs and the DRAM will be put in one logical node,
>> while the PMEM will be put in another (faked) logical node.
>
> So what does a system like that actually look like, SLIT table wise, and
> how does that affect init_numa_topology_type() ?
The SLIT table is as follows,
[000h 0000 4] Signature : "SLIT" [System Locality Information Table]
[004h 0004 4] Table Length : 0000042C
[008h 0008 1] Revision : 01
[009h 0009 1] Checksum : 59
[00Ah 0010 6] Oem ID : "INTEL "
[010h 0016 8] Oem Table ID : "S2600WF "
[018h 0024 4] Oem Revision : 00000001
[01Ch 0028 4] Asl Compiler ID : "INTL"
[020h 0032 4] Asl Compiler Revision : 20091013
[024h 0036 8] Localities : 0000000000000004
[02Ch 0044 4] Locality 0 : 0A 15 11 1C
[030h 0048 4] Locality 1 : 15 0A 1C 11
[034h 0052 4] Locality 2 : 11 1C 0A 1C
[038h 0056 4] Locality 3 : 1C 11 1C 0A
The `numactl -H` output is as follows,
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
node 0 size: 64136 MB
node 0 free: 5981 MB
node 1 cpus: 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
node 1 size: 64466 MB
node 1 free: 10415 MB
node 2 cpus:
node 2 size: 253952 MB
node 2 free: 253920 MB
node 3 cpus:
node 3 size: 253952 MB
node 3 free: 253951 MB
node distances:
node 0 1 2 3
0: 10 21 17 28
1: 21 10 28 17
2: 17 28 10 28
3: 28 17 28 10
init_numa_topology_type() set sched_numa_topology_type to NUMA_DIRECT.
The node 0 and node 1 are onlined during boot. While the PMEM node,
that is, node 2 and node 3 are onlined later. As in the following dmesg
snippet.
[ 2.252573][ T0] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x7fffffff]
[ 2.259224][ T0] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x107fffffff]
[ 2.266139][ T0] ACPI: SRAT: Node 2 PXM 2 [mem 0x1080000000-0x4f7fffffff] non-volatile
[ 2.274267][ T0] ACPI: SRAT: Node 1 PXM 1 [mem 0x4f80000000-0x5f7fffffff]
[ 2.281271][ T0] ACPI: SRAT: Node 3 PXM 3 [mem 0x5f80000000-0x9e7fffffff] non-volatile
[ 2.289403][ T0] NUMA: Initialized distance table, cnt=4
[ 2.294934][ T0] NUMA: Node 0 [mem 0x00000000-0x7fffffff] + [mem 0x100000000-0x107fffffff] -> [mem 0x00000000-0x107fffffff]
[ 2.306266][ T0] NODE_DATA(0) allocated [mem 0x107ffd5000-0x107fffffff]
[ 2.313115][ T0] NODE_DATA(1) allocated [mem 0x5f7ffd0000-0x5f7fffafff]
[ 5.391151][ T1] smp: Brought up 2 nodes, 96 CPUs
Full dmesg is attached.
Best Regards,
Huang, Ying
[-- Attachment #2: dmesg.xz --]
[-- Type: application/x-xz, Size: 27120 bytes --]
next prev parent reply other threads:[~2022-01-13 7:19 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-12-07 2:27 [PATCH -V10 RESEND 0/6] NUMA balancing: optimize memory placement for memory tiering system Huang Ying
2021-12-07 2:27 ` [PATCH -V10 RESEND 1/6] NUMA Balancing: add page promotion counter Huang Ying
2021-12-07 6:05 ` Hasan Al Maruf
2021-12-08 2:16 ` Huang, Ying
2021-12-17 7:25 ` Baolin Wang
2021-12-07 2:27 ` [PATCH -V10 RESEND 2/6] NUMA balancing: optimize page placement for memory tiering system Huang Ying
2021-12-07 6:36 ` Hasan Al Maruf
2021-12-08 3:16 ` Huang, Ying
2021-12-17 7:35 ` Baolin Wang
2021-12-07 2:27 ` [PATCH -V10 RESEND 3/6] memory tiering: skip to scan fast memory Huang Ying
2021-12-17 7:41 ` Baolin Wang
2021-12-07 2:27 ` [PATCH -V10 RESEND 4/6] memory tiering: hot page selection with hint page fault latency Huang Ying
2021-12-07 2:27 ` [PATCH -V10 RESEND 5/6] memory tiering: rate limit NUMA migration throughput Huang Ying
2021-12-07 2:27 ` [PATCH -V10 RESEND 6/6] memory tiering: adjust hot threshold automatically Huang Ying
2022-01-12 16:10 ` [PATCH -V10 RESEND 0/6] NUMA balancing: optimize memory placement for memory tiering system Peter Zijlstra
2022-01-13 7:19 ` Huang, Ying [this message]
2022-01-13 9:49 ` Peter Zijlstra
2022-01-13 12:06 ` Huang, Ying
2022-01-13 13:00 ` Peter Zijlstra
2022-01-13 13:13 ` Huang, Ying
2022-01-13 14:24 ` Huang, Ying
2022-01-14 5:24 ` Huang, Ying
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87sftsumqd.fsf@yhuang6-desk2.ccr.corp.intel.com \
--to=ying.huang@intel.com \
--cc=akpm@linux-foundation.org \
--cc=dave.hansen@linux.intel.com \
--cc=feng.tang@intel.com \
--cc=hasanalmaruf@fb.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@suse.de \
--cc=mgorman@techsingularity.net \
--cc=mhocko@suse.com \
--cc=osalvador@suse.de \
--cc=peterz@infradead.org \
--cc=riel@surriel.com \
--cc=shakeelb@google.com \
--cc=shy828301@gmail.com \
--cc=weixugc@google.com \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.