From: Muralidhara M K <muralimk@amd.com>
To: <linux-edac@vger.kernel.org>, <x86@kernel.org>
Cc: <linux-kernel@vger.kernel.org>, <bp@alien8.de>,
<mchehab@kernel.org>, <yazen.ghannam@amd.com>, <nchatrad@amd.com>,
Muralidhara M K <muralidhara.mk@amd.com>,
Naveen Krishna Chatradhi <naveenkrishna.chatradhi@amd.com>
Subject: [PATCH 3/5] EDAC/amd64: Document heterogeneous system enumeration
Date: Mon, 15 May 2023 11:35:35 +0000 [thread overview]
Message-ID: <20230515113537.1052146-4-muralimk@amd.com> (raw)
In-Reply-To: <20230515113537.1052146-1-muralimk@amd.com>
From: Muralidhara M K <muralidhara.mk@amd.com>
Document High Bandwidth Memory (HBM) and AMD heterogeneous system
topology and enumeration.
[ bp: Simplify and de-marketize, unify, massage. ]
Signed-off-by: Muralidhara M K <muralidhara.mk@amd.com>
Co-developed-by: Naveen Krishna Chatradhi <naveenkrishna.chatradhi@amd.com>
Signed-off-by: Naveen Krishna Chatradhi <naveenkrishna.chatradhi@amd.com>
Signed-off-by: Yazen Ghannam <yazen.ghannam@amd.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
---
Documentation/driver-api/edac.rst | 120 ++++++++++++++++++++++++++++++
1 file changed, 120 insertions(+)
diff --git a/Documentation/driver-api/edac.rst b/Documentation/driver-api/edac.rst
index b8c742aa0a71..f4f044b95c4f 100644
--- a/Documentation/driver-api/edac.rst
+++ b/Documentation/driver-api/edac.rst
@@ -106,6 +106,16 @@ will occupy those chip-select rows.
This term is avoided because it is unclear when needing to distinguish
between chip-select rows and socket sets.
+* High Bandwidth Memory (HBM)
+
+HBM is a new memory type with low power consumption and ultra-wide
+communication lanes. It uses vertically stacked memory chips (DRAM dies)
+interconnected by microscopic wires called "through-silicon vias," or
+TSVs.
+
+Several stacks of HBM chips connect to the CPU or GPU through an ultra-fast
+interconnect called the "interposer". Therefore, HBM's characteristics
+are nearly indistinguishable from on-chip integrated RAM.
Memory Controllers
------------------
@@ -176,3 +186,113 @@ nodes::
the L1 and L2 directories would be "edac_device_block's"
.. kernel-doc:: drivers/edac/edac_device.h
+
+
+Heterogeneous system support
+----------------------------
+
+An AMD heterogeneous system is built by connecting the data fabrics of
+both CPUs and GPUs via custom xGMI links. Thus, the data fabric on the
+GPU nodes can be accessed the same way as the data fabric on CPU nodes.
+
+The MI200 accelerators are data center GPUs. They have 2 data fabrics,
+and each GPU data fabric contains four Unified Memory Controllers (UMC).
+Each UMC contains eight channels. Each UMC channel controls one 128-bit
+HBM2e (2GB) channel (equivalent to 8 X 2GB ranks). This creates a total
+of 4096-bits of DRAM data bus.
+
+While the UMC is interfacing a 16GB (8high X 2GB DRAM) HBM stack, each UMC
+channel is interfacing 2GB of DRAM (represented as rank).
+
+Memory controllers on AMD GPU nodes can be represented in EDAC thusly:
+
+ GPU DF / GPU Node -> EDAC MC
+ GPU UMC -> EDAC CSROW
+ GPU UMC channel -> EDAC CHANNEL
+
+For example: a heterogeneous system with 1 AMD CPU is connected to
+4 MI200 (Aldebaran) GPUs using xGMI.
+
+Some more heterogeneous hardware details:
+
+- The CPU UMC (Unified Memory Controller) is mostly the same as the GPU UMC.
+ They have chip selects (csrows) and channels. However, the layouts are different
+ for performance, physical layout, or other reasons.
+- CPU UMCs use 1 channel, In this case UMC = EDAC channel. This follows the
+ marketing speak. CPU has X memory channels, etc.
+- CPU UMCs use up to 4 chip selects, So UMC chip select = EDAC CSROW.
+- GPU UMCs use 1 chip select, So UMC = EDAC CSROW.
+- GPU UMCs use 8 channels, So UMC channel = EDAC channel.
+
+The EDAC subsystem provides a mechanism to handle AMD heterogeneous
+systems by calling system specific ops for both CPUs and GPUs.
+
+AMD GPU nodes are enumerated in sequential order based on the PCI
+hierarchy, and the first GPU node is assumed to have a Node ID value
+following those of the CPU nodes after latter are fully populated::
+
+ $ ls /sys/devices/system/edac/mc/
+ mc0 - CPU MC node 0
+ mc1 |
+ mc2 |- GPU card[0] => node 0(mc1), node 1(mc2)
+ mc3 |
+ mc4 |- GPU card[1] => node 0(mc3), node 1(mc4)
+ mc5 |
+ mc6 |- GPU card[2] => node 0(mc5), node 1(mc6)
+ mc7 |
+ mc8 |- GPU card[3] => node 0(mc7), node 1(mc8)
+
+For example, a heterogeneous system with one AMD CPU is connected to
+four MI200 (Aldebaran) GPUs using xGMI. This topology can be represented
+via the following sysfs entries::
+
+ /sys/devices/system/edac/mc/..
+
+ CPU # CPU node
+ ├── mc 0
+
+ GPU Nodes are enumerated sequentially after CPU nodes have been populated
+ GPU card 1 # Each MI200 GPU has 2 nodes/mcs
+ ├── mc 1 # GPU node 0 == mc1, Each MC node has 4 UMCs/CSROWs
+ │ ├── csrow 0 # UMC 0
+ │ │ ├── channel 0 # Each UMC has 8 channels
+ │ │ ├── channel 1 # size of each channel is 2 GB, so each UMC has 16 GB
+ │ │ ├── channel 2
+ │ │ ├── channel 3
+ │ │ ├── channel 4
+ │ │ ├── channel 5
+ │ │ ├── channel 6
+ │ │ ├── channel 7
+ │ ├── csrow 1 # UMC 1
+ │ │ ├── channel 0
+ │ │ ├── ..
+ │ │ ├── channel 7
+ │ ├── .. ..
+ │ ├── csrow 3 # UMC 3
+ │ │ ├── channel 0
+ │ │ ├── ..
+ │ │ ├── channel 7
+ │ ├── rank 0
+ │ ├── .. ..
+ │ ├── rank 31 # total 32 ranks/dimms from 4 UMCs
+ ├
+ ├── mc 2 # GPU node 1 == mc2
+ │ ├── .. # each GPU has total 64 GB
+
+ GPU card 2
+ ├── mc 3
+ │ ├── ..
+ ├── mc 4
+ │ ├── ..
+
+ GPU card 3
+ ├── mc 5
+ │ ├── ..
+ ├── mc 6
+ │ ├── ..
+
+ GPU card 4
+ ├── mc 7
+ │ ├── ..
+ ├── mc 8
+ │ ├── ..
--
2.25.1
next prev parent reply other threads:[~2023-05-15 11:41 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-15 11:35 [PATCH 0/5] AMD64 EDAC GPU Updates Muralidhara M K
2023-05-15 11:35 ` [PATCH 1/5] x86/amd_nb: Add MI200 PCI IDs Muralidhara M K
2023-05-31 9:42 ` Borislav Petkov
2023-06-05 14:14 ` [tip: ras/core] x86/amd_nb: Re-sort and re-indent PCI defines tip-bot2 for Borislav Petkov (AMD)
2023-06-05 14:14 ` [tip: ras/core] x86/amd_nb: Add MI200 PCI IDs tip-bot2 for Yazen Ghannam
2023-05-15 11:35 ` [PATCH 2/5] x86/MCE/AMD, EDAC/mce_amd: Decode UMC_V2 ECC errors Muralidhara M K
2023-06-05 14:14 ` [tip: ras/core] " tip-bot2 for Yazen Ghannam
2023-05-15 11:35 ` Muralidhara M K [this message]
2023-06-05 14:14 ` [tip: ras/core] EDAC/amd64: Document heterogeneous system enumeration tip-bot2 for Muralidhara M K
2023-05-15 11:35 ` [PATCH 4/5] EDAC/amd64: Add support for AMD heterogeneous Family 19h Model 30h-3Fh Muralidhara M K
2023-06-05 14:14 ` [tip: ras/core] " tip-bot2 for Muralidhara M K
2023-05-15 11:35 ` [PATCH 5/5] EDAC/amd64: Cache and use GPU node map Muralidhara M K
2023-06-05 14:14 ` [tip: ras/core] " tip-bot2 for Yazen Ghannam
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230515113537.1052146-4-muralimk@amd.com \
--to=muralimk@amd.com \
--cc=bp@alien8.de \
--cc=linux-edac@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mchehab@kernel.org \
--cc=muralidhara.mk@amd.com \
--cc=naveenkrishna.chatradhi@amd.com \
--cc=nchatrad@amd.com \
--cc=x86@kernel.org \
--cc=yazen.ghannam@amd.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).