From: Gu Zheng <guz.fnst@cn.fujitsu.com>
To: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Cc: <linux-kernel@vger.kernel.org>, <tj@kernel.org>,
<laijs@cn.fujitsu.com>, <isimatu.yasuaki@jp.fujitsu.com>,
<tangchen@cn.fujitsu.com>, <izumi.taku@jp.fujitsu.com>
Subject: Re: [PATCH 1/2] x86/cpu hotplug: make apicid <--> cpuid mapping persistent
Date: Mon, 30 Mar 2015 17:58:44 +0800 [thread overview]
Message-ID: <55191E54.3000205@cn.fujitsu.com> (raw)
In-Reply-To: <5514345B.40306@jp.fujitsu.com>
Hi Kame-san,
On 03/27/2015 12:31 AM, Kamezawa Hiroyuki wrote:
> On 2015/03/26 13:55, Gu Zheng wrote:
>> Hi Kame-san,
>> On 03/26/2015 11:19 AM, Kamezawa Hiroyuki wrote:
>>
>>> On 2015/03/26 11:17, Gu Zheng wrote:
>>>> Previously, we build the apicid <--> cpuid mapping when the cpu is present, but
>>>> the relationship will be changed if the cpu/node hotplug happenned, because we
>>>> always choose the first free cpuid for the hot added cpu (whether it is new-add
>>>> or re-add), so this the cpuid <--> node mapping changed if node hot plug
>>>> occurred, and it causes the wq sub-system allocation failture:
>>>> ==
>>>> SLUB: Unable to allocate memory on node 2 (gfp=0x80d0)
>>>> cache: kmalloc-192, object size: 192, buffer size: 192, default
>>>> order:
>>>> 1, min order: 0
>>>> node 0: slabs: 6172, objs: 259224, free: 245741
>>>> node 1: slabs: 3261, objs: 136962, free: 127656
>>>> ==
>>>> So here we build the persistent [lapic id] <--> cpuid mapping when the cpu first
>>>> present, and never change it.
>>>>
>>>> Suggested-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
>>>> Signed-off-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
>>>> ---
>>>> arch/x86/kernel/apic/apic.c | 31 ++++++++++++++++++++++++++++++-
>>>> 1 files changed, 30 insertions(+), 1 deletions(-)
>>>>
>>>> diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
>>>> index ad3639a..d539ebc 100644
>>>> --- a/arch/x86/kernel/apic/apic.c
>>>> +++ b/arch/x86/kernel/apic/apic.c
>>>> @@ -2038,6 +2038,30 @@ void disconnect_bsp_APIC(int virt_wire_setup)
>>>> apic_write(APIC_LVT1, value);
>>>> }
>>>>
>>>> +/*
>>>> + * Logic cpu number(cpuid) to local APIC id persistent mappings.
>>>> + * Do not clear the mapping even if cpu hot removed.
>>>> + * */
>>>> +static int apicid_to_x86_cpu[MAX_LOCAL_APIC] = {
>>>> + [0 ... MAX_LOCAL_APIC - 1] = -1,
>>>> +};
>>>
>>>
>>> This patch cannot handle x2apic, which is 32bit.
>>
>> IMO, if the apicid is too big (larger than MAX_LOCAL_APIC), we will skip
>> generating a logic cpu number for it, so it seems no problem here.
>>
> you mean MAX_LOCAL_APIC=32768 ? ....isn't it too wasting ?
I use the big array here to keep the same format with the existed ones:
int apic_version[MAX_LOCAL_APIC];
s16 __apicid_to_node[MAX_LOCAL_APIC] = {
[0 ... MAX_LOCAL_APIC-1] = NUMA_NO_NODE
};
Or we should also say "NO" to them?
Regards,
Gu
>
> Anyway, APIC IDs are sparse values. Please use proper structure.
>
> Thanks,
> -Kame
>
>>>
>>> As far as I understand, it depends on CPU's spec and the newest cpu has 9bit apicid, at least.
>>>
>>> But you can't create inifinit array.
>>>
>>> If you can't allocate the array dynamically, How about adding
>>>
>>> static int cpuid_to_apicid[MAX_CPU] = {}
>>>
>>> or using idr library ? (please see lib/idr.c)
>>>
>>> I guess you can update this map after boot(after mm initialization)
>>> and make use of idr library.
>>>
>>> About this patch, Nack.
>>>
>>> -Kame
>>>
>>>
>>>
>>>> +
>>>> +/*
>>>> + * Internal cpu id bits, set the bit once cpu present, and never clear it.
>>>> + * */
>>>> +static cpumask_t cpuid_mask = CPU_MASK_NONE;
>>>> +
>>>> +static int get_cpuid(int apicid)
>>>> +{
>>>> + int cpuid;
>>>> +
>>>> + cpuid = apicid_to_x86_cpu[apicid];
>>>> + if (cpuid == -1)
>>>> + cpuid = cpumask_next_zero(-1, &cpuid_mask);
>>>> +
>>>> + return cpuid;
>>>> +}
>>>> +
>>>> int generic_processor_info(int apicid, int version)
>>>> {
>>>> int cpu, max = nr_cpu_ids;
>>>> @@ -2115,7 +2139,10 @@ int generic_processor_info(int apicid, int version)
>>>> */
>>>> cpu = 0;
>>>> } else
>>>> - cpu = cpumask_next_zero(-1, cpu_present_mask);
>>>> + cpu = get_cpuid(apicid);
>>>> +
>>>> + /* Store the mapping */
>>>> + apicid_to_x86_cpu[apicid] = cpu;
>>>>
>>>> /*
>>>> * Validate version
>>>> @@ -2144,6 +2171,8 @@ int generic_processor_info(int apicid, int version)
>>>> early_per_cpu(x86_cpu_to_logical_apicid, cpu) =
>>>> apic->x86_32_early_logical_apicid(cpu);
>>>> #endif
>>>> + /* Mark this cpu id as uesed (already mapping a local apic id) */
>>>> + cpumask_set_cpu(cpu, &cpuid_mask);
>>>> set_cpu_possible(cpu, true);
>>>> set_cpu_present(cpu, true);
>>>>
>>>>
>>>
>>>
>>> .
>>>
>>
>>
>
>
> .
>
next prev parent reply other threads:[~2015-03-30 10:17 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-03-26 2:17 [PATCH 0/2] workqueue: fix a bug when numa mapping is changed Gu Zheng
2015-03-26 2:17 ` [PATCH 1/2] x86/cpu hotplug: make apicid <--> cpuid mapping persistent Gu Zheng
2015-03-26 3:19 ` Kamezawa Hiroyuki
2015-03-26 4:55 ` Gu Zheng
2015-03-26 15:13 ` Tejun Heo
2015-03-26 16:31 ` Kamezawa Hiroyuki
2015-03-30 9:58 ` Gu Zheng [this message]
2015-04-01 2:59 ` Kamezawa Hiroyuki
2015-03-26 2:17 ` [PATCH 2/2] workqueue: update per cpu workqueue's numa affinity when cpu preparing online Gu Zheng
2015-03-26 3:12 ` [PATCH 0/2] workqueue: fix a bug when numa mapping is changed Kamezawa Hiroyuki
2015-03-26 5:04 ` Gu Zheng
2015-03-26 15:18 ` Tejun Heo
2015-03-26 16:42 ` Kamezawa Hiroyuki
2015-03-30 9:49 ` Gu Zheng
2015-03-30 9:49 ` Gu Zheng
2015-03-31 6:09 ` Kamezawa Hiroyuki
2015-03-31 15:28 ` Tejun Heo
2015-04-01 2:55 ` Kamezawa Hiroyuki
2015-04-01 3:02 ` Tejun Heo
2015-04-01 3:05 ` Tejun Heo
2015-04-01 8:30 ` Kamezawa Hiroyuki
2015-04-02 1:36 ` Gu Zheng
2015-04-02 2:54 ` Kamezawa Hiroyuki
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=55191E54.3000205@cn.fujitsu.com \
--to=guz.fnst@cn.fujitsu.com \
--cc=isimatu.yasuaki@jp.fujitsu.com \
--cc=izumi.taku@jp.fujitsu.com \
--cc=kamezawa.hiroyu@jp.fujitsu.com \
--cc=laijs@cn.fujitsu.com \
--cc=linux-kernel@vger.kernel.org \
--cc=tangchen@cn.fujitsu.com \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).