Linux-EDAC Archive on lore.kernel.org
 help / color / Atom feed
From: James Morse <james.morse@arm.com>
To: Robert Richter <rrichter@marvell.com>
Cc: Borislav Petkov <bp@alien8.de>, Tony Luck <tony.luck@intel.com>,
	Mauro Carvalho Chehab <mchehab@kernel.org>,
	"linux-edac@vger.kernel.org" <linux-edac@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 14/21] EDAC, ghes: Extract numa node information for each dimm
Date: Wed, 29 May 2019 18:51:00 +0100
Message-ID: <42d1cb7f-de7a-31e8-eb89-8d0651a3501a@arm.com> (raw)
In-Reply-To: <20190529084344.28562-15-rrichter@marvell.com>

Hi Robert,

On 29/05/2019 09:44, Robert Richter wrote:
> In a later patch we want to have one mc device per node. This patch
> extracts the numa node information for each dimm. This is done by
> collecting the physical address ranges from the DMI table (Memory
> Array Mapped Address - Type 19 of SMBIOS spec). The node information> for a physical address is already know to a numa aware system (e.g. by
> using the ACPI _PXM method or the ACPI SRAT table), so based on the PA
> we can assign the node id to the dimms.

I think you're letting the smbios information drive you here. We'd like to do as much as
possible without it, all its really good for is telling us the label on the PCB.

With this approach, you only get numa information by parsing more smbios, which we have to
try and validate, and fall back to some error path if it smells wrong. We end up needing
things like a 'fallback memory controller' in the case the firmware fault-time value is
missing, or nuts.

What bugs me is we already know the numa information from the address. We could expose
that without the smbios tables at all, and it would be useful to someone playing the
dimm-bisect game. Not making it depend on smbios means there is a good chance it can
become common with other edac drivers.

I don't think we need to know the dimm->node mapping up front. When we get an error,
pfn_to_nid() on the address tells us which node that memory is attached to. This should be
the only place nid information comes from, that way we don't need to check it. Trying to
correlate it with smbios tables is much more code. If the CPER comes with a DIMM handle,
we know the DIMM too.

So all we really need is to know at setup time is how many numa-nodes there are, and the
maximum DIMM per node if we don't want phantom-dimms. Type-17 already has a
Physical-Memory-Array-Handle, which points at Type-19... but we don't need to read it,
just count them and find the biggest.

If the type-19 information is missing from smbios, or not linked up properly, or the
values provided at fault-time don't overlap with the values in the table, or there is no
fault-time node information: you still get the numa-node information based on the address.

Using the minimum information should give us the least code, and the least exposure to
misdescribed tables.


> A fallback that disables numa is implemented in case the node
> information is inconsistent.

> diff --git a/drivers/edac/ghes_edac.c b/drivers/edac/ghes_edac.c
> index 50f4ee36b755..083452a48b42 100644
> --- a/drivers/edac/ghes_edac.c
> +++ b/drivers/edac/ghes_edac.c
> @@ -67,14 +67,34 @@ struct memdev_dmi_entry {
>  	u16 conf_mem_clk_speed;
>  } __attribute__((__packed__));
>  
> +/* Memory Array Mapped Address - Type 19 of SMBIOS spec */
> +struct memarr_dmi_entry {
> +	u8		type;
> +	u8		length;
> +	u16		handle;
> +	u32		start;
> +	u32		end;
> +	u16		phys_mem_array_handle;
> +	u8		partition_width;
> +	u64		ext_start;
> +	u64		ext_end;
> +} __attribute__((__packed__));

Any chance we could collect the structures from the smbios spec in a header file somewhere?

I'd prefer not to read this thing at all if we can help it.

>  struct ghes_dimm_info {
>  	struct dimm_info dimm_info;
>  	int		idx;
> +	int		numa_node;

(I thought nid was the preferred term)


> +	phys_addr_t	start;
> +	phys_addr_t	end;

I think start and end are deceptive as they overlap with other DIMMs on systems with
interleaving memory controllers. I'd prefer not to keep these values around.


> +	u16		phys_handle;
>  };
>  
>  struct ghes_mem_info {
> -	int num_dimm;
> +	int		num_dimm;
>  	struct ghes_dimm_info *dimms;
> +	int		num_nodes;

> +	int		num_per_node[MAX_NUMNODES];

Number of what?


> +	bool		enable_numa;

This is used locally in mem_info_setup(), but its not read from here by any of the later
patches in the series. Is it needed?

I don't like the idea that this is behaviour that is turned on/off. Its a property of the
system. I think it would be better if CONFIG_NUMA causes you to get multiple
memory-controllers created, but if its not actually a NUMA machine there would only be
one. This lets us test that code on not-really-numa systems.


>  };
>  
>  struct ghes_mem_info mem_info;
> @@ -97,10 +117,50 @@ static void ghes_dimm_info_init(void)
>  
>  	for_each_dimm(dimm) {
>  		dimm->idx	= idx;
> +		dimm->numa_node	= NUMA_NO_NODE;
>  		idx++;
>  	}
>  }
>  
> +static void ghes_edac_set_nid(const struct dmi_header *dh, void *arg)
> +{
> +	struct memarr_dmi_entry *entry = (struct memarr_dmi_entry *)dh;
> +	struct ghes_dimm_info *dimm;
> +	phys_addr_t start, end;
> +	int nid;
> +
> +	if (dh->type != DMI_ENTRY_MEM_ARRAY_MAPPED_ADDR)
> +		return;

> +	/* only support SMBIOS 2.7+ */
> +	if (entry->length < sizeof(*entry))
> +		return;

Lovely. I still hope we can get away without parsing this table.


> +	if (entry->start == 0xffffffff)
> +		start = entry->ext_start;
> +	else
> +		start = entry->start;
> +	if (entry->end == 0xffffffff)
> +		end = entry->ext_end;
> +	else
> +		end = entry->end;


> +	if (!pfn_valid(PHYS_PFN(start)))
> +		return;

Eh? Just because there is no struct page doesn't mean firmware won't report errors for it.
This is going to bite on arm64 if the 'start' page happens to have been reserved by
firmware, and thus doesn't have a struct page. Bottom-up allocation doesn't sound unlikely.


> +	nid = pfn_to_nid(PHYS_PFN(start));

... Ugh, because pfn_to_nid() goes via struct page.

You can make this robust by scanning start->end looking for a pfn_valid() you can pull the
nid out of. (no, I don't think its a good idea either!)

I'd like to see if we can get rid of the 'via address' part of this.


> +	if (nid < 0 || nid >= MAX_NUMNODES || !node_possible(nid))
> +		nid = NUMA_NO_NODE;

Can this happen? Does this indicate the firmware tables are wrong, or mm is about derail?


> +	for_each_dimm(dimm) {
> +		if (entry->phys_mem_array_handle == dimm->phys_handle) {
> +			dimm->numa_node	= nid;
> +			dimm->start	= start;
> +			dimm->end	= end;
> +		}
> +	}
> +}
> +
>  static int get_dimm_smbios_index(u16 handle)
>  {
>  	struct mem_ctl_info *mci = ghes_pvt->mci;
> @@ -213,8 +273,25 @@ static void ghes_edac_dmidecode(const struct dmi_header *dh, void *arg)
>  	}
>  }
>  
> +static void mem_info_disable_numa(void)
> +{
> +	struct ghes_dimm_info *dimm;
> +
> +	for_each_dimm(dimm) {
> +		if (dimm->numa_node != NUMA_NO_NODE)
> +			mem_info.num_per_node[dimm->numa_node] = 0;

> +		dimm->numa_node = 0;

NUMA_NO_NODE?

> +	}
> +
> +	mem_info.num_per_node[0] = mem_info.num_dimm;
> +	mem_info.num_nodes = 1;
> +	mem_info.enable_numa = false;
> +}
> +
>  static int mem_info_setup(void)
>  {
> +	struct ghes_dimm_info *dimm;
> +	bool enable_numa = true;
>  	int idx = 0;
>  
>  	memset(&mem_info, 0, sizeof(mem_info));
> @@ -231,6 +308,29 @@ static int mem_info_setup(void)
>  
>  	ghes_dimm_info_init();
>  	dmi_walk(ghes_edac_dmidecode, &idx);
> +	dmi_walk(ghes_edac_set_nid, NULL);
> +
> +	for_each_dimm(dimm) {
> +		if (dimm->numa_node == NUMA_NO_NODE) {
> +			enable_numa = false;
> +		} else {

> +			if (!mem_info.num_per_node[dimm->numa_node])
> +				mem_info.num_nodes++;

This is to try and hide empty nodes?


> +			mem_info.num_per_node[dimm->numa_node]++;

Could you do these two in your previous for_each_dimm() walk?


> +		}
> +
> +		edac_dbg(1, "DIMM%i: Found mem range [%pa-%pa] on node %d\n",
> +			dimm->idx, &dimm->start, &dimm->end, dimm->numa_node);
> +	}


> +	mem_info.enable_numa = enable_numa;
> +	if (enable_numa)
> +		return 0;
> +
> +	/* something went wrong, disable numa */
> +	if (num_possible_nodes() > 1)
> +		pr_warn("Can't get numa info, disabling numa\n");
> +	mem_info_disable_numa();

I'd like to find a way of doing this where we don't need this sort of thing!


Thanks,

James

  reply index

Thread overview: 43+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-05-29  8:44 [PATCH 00/21] EDAC, mc, ghes: Fixes and updates to improve memory error reporting Robert Richter
2019-05-29  8:44 ` [PATCH 01/21] EDAC, mc: Fix edac_mc_find() in case no device is found Robert Richter
2019-05-29  8:44 ` [PATCH 02/21] EDAC: Fixes to use put_device() after device_add() errors Robert Richter
2019-06-11 17:28   ` Borislav Petkov
2019-06-12 17:17     ` Robert Richter
2019-05-29  8:44 ` [PATCH 03/21] EDAC: Kill EDAC_DIMM_PTR() macro Robert Richter
2019-05-29  8:44 ` [PATCH 04/21] EDAC: Kill EDAC_DIMM_OFF() macro Robert Richter
2019-05-29  8:44 ` [PATCH 05/21] EDAC: Introduce mci_for_each_dimm() iterator Robert Richter
2019-05-29  8:44 ` [PATCH 06/21] EDAC, mc: Cleanup _edac_mc_free() code Robert Richter
2019-05-29  8:44 ` [PATCH 07/21] EDAC, mc: Remove per layer counters Robert Richter
2019-05-29  8:44 ` [PATCH 08/21] EDAC, mc: Rework edac_raw_mc_handle_error() to use struct dimm_info Robert Richter
2019-05-29  8:44 ` [PATCH 09/21] EDAC, ghes: Use standard kernel macros for page calculations Robert Richter
2019-05-29 15:13   ` James Morse
2019-05-29  8:44 ` [PATCH 10/21] EDAC, ghes: Remove pvt->detail_location string Robert Richter
2019-05-29 15:13   ` James Morse
2019-06-12 18:13     ` Robert Richter
2019-05-29  8:44 ` [PATCH 11/21] EDAC, ghes: Unify trace_mc_event() code with edac_mc driver Robert Richter
2019-05-29 15:12   ` James Morse
2019-06-03 13:10     ` Robert Richter
2019-06-04 17:15       ` James Morse
2019-06-13 22:23         ` Robert Richter
2019-05-29  8:44 ` [PATCH 12/21] EDAC, ghes: Add support for legacy API counters Robert Richter
2019-05-29 15:13   ` James Morse
2019-06-12 18:41     ` Robert Richter
2019-06-19 17:22       ` James Morse
2019-06-20  6:55         ` Robert Richter
2019-06-26  9:33           ` James Morse
2019-06-26 10:27             ` Robert Richter
2019-05-29  8:44 ` [PATCH 13/21] EDAC, ghes: Rework memory hierarchy detection Robert Richter
2019-05-29 15:06   ` James Morse
2019-05-31 13:41     ` Robert Richter
2019-05-29  8:44 ` [PATCH 14/21] EDAC, ghes: Extract numa node information for each dimm Robert Richter
2019-05-29 17:51   ` James Morse [this message]
2019-06-13 20:52     ` Robert Richter
2019-05-29  8:44 ` [PATCH 15/21] EDAC, ghes: Moving code around ghes_edac_register() Robert Richter
2019-05-29  8:44 ` [PATCH 16/21] EDAC, ghes: Create one memory controller device per node Robert Richter
2019-05-29  8:44 ` [PATCH 17/21] EDAC, ghes: Fill sysfs with the DMI DIMM label information Robert Richter
2019-05-29  8:44 ` [PATCH 18/21] EDAC, mc: Introduce edac_mc_alloc_by_dimm() for per dimm allocation Robert Richter
2019-05-29  8:44 ` [PATCH 19/21] EDAC, ghes: Identify dimm by node, card, module and handle Robert Richter
2019-05-29  8:44 ` [PATCH 20/21] EDAC, ghes: Enable per-layer reporting based on card/module Robert Richter
2019-05-29  8:44 ` [PATCH 21/21] EDAC, Documentation: Describe CPER module definition and DIMM ranks Robert Richter
2019-05-29 14:54 ` [PATCH 00/21] EDAC, mc, ghes: Fixes and updates to improve memory error reporting Borislav Petkov
2019-05-31 14:48   ` Robert Richter

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=42d1cb7f-de7a-31e8-eb89-8d0651a3501a@arm.com \
    --to=james.morse@arm.com \
    --cc=bp@alien8.de \
    --cc=linux-edac@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mchehab@kernel.org \
    --cc=rrichter@marvell.com \
    --cc=tony.luck@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-EDAC Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-edac/0 linux-edac/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-edac linux-edac/ https://lore.kernel.org/linux-edac \
		linux-edac@vger.kernel.org
	public-inbox-index linux-edac

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-edac


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git