From: Laurent Dufour <ldufour@linux.ibm.com>
To: "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
mpe@ellerman.id.au, benh@kernel.crashing.org, paulus@samba.org
Cc: nathanl@linux.ibm.com, Tyrel Datwyler <tyreld@linux.ibm.com>,
linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3] pseries/drmem: update LMBs after LPM
Date: Thu, 29 Apr 2021 13:35:30 +0200 [thread overview]
Message-ID: <66c1b88f-6614-52b1-9b53-5fe60a1a813b@linux.ibm.com> (raw)
In-Reply-To: <e3fcd8b6-6d43-85ed-7036-42430aad4979@linux.ibm.com>
Le 29/04/2021 à 13:31, Laurent Dufour a écrit :
> Le 29/04/2021 à 12:27, Aneesh Kumar K.V a écrit :
>> Laurent Dufour <ldufour@linux.ibm.com> writes:
>>
>>> After a LPM, the device tree node ibm,dynamic-reconfiguration-memory may be
>>> updated by the hypervisor in the case the NUMA topology of the LPAR's
>>> memory is updated.
>>>
>>> This is caught by the kernel, but the memory's node is updated because
>>> there is no way to move a memory block between nodes.
>>>
>>> If later a memory block is added or removed, drmem_update_dt() is called
>>> and it is overwriting the DT node to match the added or removed LMB. But
>>> the LMB's associativity node has not been updated after the DT node update
>>> and thus the node is overwritten by the Linux's topology instead of the
>>> hypervisor one.
>>>
>>> Introduce a hook called when the ibm,dynamic-reconfiguration-memory node is
>>> updated to force an update of the LMB's associativity.
>>>
>>> Cc: Tyrel Datwyler <tyreld@linux.ibm.com>
>>> Signed-off-by: Laurent Dufour <ldufour@linux.ibm.com>
>>> ---
>>>
>>> V3:
>>> - Check rd->dn->name instead of rd->dn->full_name
>>> V2:
>>> - Take Tyrel's idea to rely on OF_RECONFIG_UPDATE_PROPERTY instead of
>>> introducing a new hook mechanism.
>>> ---
>>> arch/powerpc/include/asm/drmem.h | 1 +
>>> arch/powerpc/mm/drmem.c | 35 +++++++++++++++++++
>>> .../platforms/pseries/hotplug-memory.c | 4 +++
>>> 3 files changed, 40 insertions(+)
>>>
>>> diff --git a/arch/powerpc/include/asm/drmem.h b/arch/powerpc/include/asm/drmem.h
>>> index bf2402fed3e0..4265d5e95c2c 100644
>>> --- a/arch/powerpc/include/asm/drmem.h
>>> +++ b/arch/powerpc/include/asm/drmem.h
>>> @@ -111,6 +111,7 @@ int drmem_update_dt(void);
>>> int __init
>>> walk_drmem_lmbs_early(unsigned long node, void *data,
>>> int (*func)(struct drmem_lmb *, const __be32 **, void *));
>>> +void drmem_update_lmbs(struct property *prop);
>>> #endif
>>> static inline void invalidate_lmb_associativity_index(struct drmem_lmb *lmb)
>>> diff --git a/arch/powerpc/mm/drmem.c b/arch/powerpc/mm/drmem.c
>>> index 9af3832c9d8d..f0a6633132af 100644
>>> --- a/arch/powerpc/mm/drmem.c
>>> +++ b/arch/powerpc/mm/drmem.c
>>> @@ -307,6 +307,41 @@ int __init walk_drmem_lmbs_early(unsigned long node,
>>> void *data,
>>> return ret;
>>> }
>>> +/*
>>> + * Update the LMB associativity index.
>>> + */
>>> +static int update_lmb(struct drmem_lmb *updated_lmb,
>>> + __maybe_unused const __be32 **usm,
>>> + __maybe_unused void *data)
>>> +{
>>> + struct drmem_lmb *lmb;
>>> +
>>> + /*
>>> + * Brut force there may be better way to fetch the LMB
>>> + */
>>> + for_each_drmem_lmb(lmb) {
>>> + if (lmb->drc_index != updated_lmb->drc_index)
>>> + continue;
>>> +
>>> + lmb->aa_index = updated_lmb->aa_index;
>>> + break;
>>> + }
>>> + return 0;
>>> +}
>>> +
>>> +/*
>>> + * Update the LMB associativity index.
>>> + *
>>> + * This needs to be called when the hypervisor is updating the
>>> + * dynamic-reconfiguration-memory node property.
>>> + */
>>> +void drmem_update_lmbs(struct property *prop)
>>> +{
>>> + if (!strcmp(prop->name, "ibm,dynamic-memory"))
>>> + __walk_drmem_v1_lmbs(prop->value, NULL, NULL, update_lmb);
>>> + else if (!strcmp(prop->name, "ibm,dynamic-memory-v2"))
>>> + __walk_drmem_v2_lmbs(prop->value, NULL, NULL, update_lmb);
>>> +}
>>> #endif
>>> static int init_drmem_lmb_size(struct device_node *dn)
>>> diff --git a/arch/powerpc/platforms/pseries/hotplug-memory.c
>>> b/arch/powerpc/platforms/pseries/hotplug-memory.c
>>> index 8377f1f7c78e..672ffbee2e78 100644
>>> --- a/arch/powerpc/platforms/pseries/hotplug-memory.c
>>> +++ b/arch/powerpc/platforms/pseries/hotplug-memory.c
>>> @@ -949,6 +949,10 @@ static int pseries_memory_notifier(struct notifier_block
>>> *nb,
>>> case OF_RECONFIG_DETACH_NODE:
>>> err = pseries_remove_mem_node(rd->dn);
>>> break;
>>> + case OF_RECONFIG_UPDATE_PROPERTY:
>>> + if (!strcmp(rd->dn->name,
>>> + "ibm,dynamic-reconfiguration-memory"))
>>> + drmem_update_lmbs(rd->prop);
>>> }
>>> return notifier_from_errno(err);
>>
>> How will this interact with DLPAR memory? When we dlpar memory,
>> ibm,configure-connector is used to fetch the new associativity details
>> and set drmem_lmb->aa_index correctly there. Once that is done kernel
>> then call drmem_update_dt() which will result in the above notifier
>> callback?
>
> When a memory DLPAR operation is done, the in memory DT property
> "ibm,dynamic-memory" or "ibm,dynamic-memory-v2" (if existing) have to be updated
> to reflect the added/removed memory part. This is done by calling
> drmem_update_dt().
>
> This patch is addressing the case where the hypervisor has updated the DT
> property mentioned above. In that case, the LMB tree should be updated so the
> aa_index fields are matching the DT one. This way the next time a memory DLPAR
> operation is done the DT properties "ibm,dynamic-memory" or
> "ibm,dynamic-memory-v2" will be rebuilt correctly.
>
>> IIUC, the call back then will update drmem_lmb->aa_index again?
Oh I missed what you pointed out.
Please ignore my previous answer, I need to double check code.
> drmem_update_dt() is not updating drmem_lmb->aa_index, that's the oppposite, it
> is rebuilding the in memory DT property "ibm,dynamic-memory" or
> "ibm,dynamic-memory-v2" based on the data stored in the LMB tree.
>
> Laurent.
next prev parent reply other threads:[~2021-04-29 11:35 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-28 9:47 [PATCH v3] pseries/drmem: update LMBs after LPM Laurent Dufour
2021-04-29 10:27 ` Aneesh Kumar K.V
2021-04-29 11:31 ` Laurent Dufour
2021-04-29 11:35 ` Laurent Dufour [this message]
2021-04-29 12:38 ` Laurent Dufour
2021-04-29 19:12 ` Tyrel Datwyler
2021-04-30 16:13 ` Laurent Dufour
2021-04-30 23:58 ` Tyrel Datwyler
2021-05-03 17:28 ` Laurent Dufour
2021-05-03 20:44 ` Tyrel Datwyler
2021-05-04 7:03 ` Laurent Dufour
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=66c1b88f-6614-52b1-9b53-5fe60a1a813b@linux.ibm.com \
--to=ldufour@linux.ibm.com \
--cc=aneesh.kumar@linux.ibm.com \
--cc=benh@kernel.crashing.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=mpe@ellerman.id.au \
--cc=nathanl@linux.ibm.com \
--cc=paulus@samba.org \
--cc=tyreld@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).