* nr_cpu_ids vs AMD 3970x(32 physical CPUs) @ 2020-07-03 15:57 Uladzislau Rezki 2020-07-03 16:56 ` Peter Zijlstra ` (2 more replies) 0 siblings, 3 replies; 15+ messages in thread From: Uladzislau Rezki @ 2020-07-03 15:57 UTC (permalink / raw) To: linux-kernel, linux-mm Cc: Andrew Morton, Linus Torvalds, GregKroah-Hartmangregkh, peterz Hello, folk. I have a system based on AMD 3970x CPUs. It has 32 physical cores and 64 threads. It seems that "nr_cpu_ids" variable is not correctly set on latest 5.8-rc3 kernel. Please have a look below on dmesg output: <snip> urezki@pc638:~$ sudo dmesg | grep 128 [ 0.000000] IOAPIC[0]: apic_id 128, version 33, address 0xfec00000, GSI 0-23 [ 0.000000] smpboot: Allowing 128 CPUs, 64 hotplug CPUs [ 0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:128 nr_node_ids:1 ... [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=128, Nodes=1 [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=128. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=128 urezki@pc638:~$ <snip> For example SLUB thinks that it deals with 128 CPUs in the system what is wrong if i do not miss something. Since nr_cpu_ids is broken(?), thus the "cpu_possible_mask" does not correspond to reality as well. Any thoughts? Thanks! -- Vlad Rezki ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: nr_cpu_ids vs AMD 3970x(32 physical CPUs) 2020-07-03 15:57 nr_cpu_ids vs AMD 3970x(32 physical CPUs) Uladzislau Rezki @ 2020-07-03 16:56 ` Peter Zijlstra 2020-07-03 17:09 ` Uladzislau Rezki 2020-07-03 17:07 ` Gabriel C 2020-07-03 19:05 ` peter enderborg 2 siblings, 1 reply; 15+ messages in thread From: Peter Zijlstra @ 2020-07-03 16:56 UTC (permalink / raw) To: Uladzislau Rezki Cc: linux-kernel, linux-mm, Andrew Morton, Linus Torvalds, GregKroah-Hartmangregkh On Fri, Jul 03, 2020 at 05:57:49PM +0200, Uladzislau Rezki wrote: > Hello, folk. > > I have a system based on AMD 3970x CPUs. It has 32 physical cores > and 64 threads. It seems that "nr_cpu_ids" variable is not correctly > set on latest 5.8-rc3 kernel. Please have a look below on dmesg output: > > <snip> > urezki@pc638:~$ sudo dmesg | grep 128 > [ 0.000000] IOAPIC[0]: apic_id 128, version 33, address 0xfec00000, GSI 0-23 > [ 0.000000] smpboot: Allowing 128 CPUs, 64 hotplug CPUs This is your BIOS saying it needs 128 ids, 64 of which are 'empty'. I have a box like that as well, if it bothers you boot with: "possible_cpus=64" or something. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: nr_cpu_ids vs AMD 3970x(32 physical CPUs) 2020-07-03 16:56 ` Peter Zijlstra @ 2020-07-03 17:09 ` Uladzislau Rezki 2020-07-03 17:38 ` Peter Zijlstra 0 siblings, 1 reply; 15+ messages in thread From: Uladzislau Rezki @ 2020-07-03 17:09 UTC (permalink / raw) To: Peter Zijlstra Cc: Uladzislau Rezki, linux-kernel, linux-mm, Andrew Morton, Linus Torvalds, GregKroah-Hartmangregkh On Fri, Jul 03, 2020 at 06:56:27PM +0200, Peter Zijlstra wrote: > On Fri, Jul 03, 2020 at 05:57:49PM +0200, Uladzislau Rezki wrote: > > Hello, folk. > > > > I have a system based on AMD 3970x CPUs. It has 32 physical cores > > and 64 threads. It seems that "nr_cpu_ids" variable is not correctly > > set on latest 5.8-rc3 kernel. Please have a look below on dmesg output: > > > > <snip> > > urezki@pc638:~$ sudo dmesg | grep 128 > > [ 0.000000] IOAPIC[0]: apic_id 128, version 33, address 0xfec00000, GSI 0-23 > > [ 0.000000] smpboot: Allowing 128 CPUs, 64 hotplug CPUs > > This is your BIOS saying it needs 128 ids, 64 of which are 'empty'. > > I have a box like that as well, if it bothers you boot with: > "possible_cpus=64" or something. > OK, i got it. I thought that "cpu_possible_mask" strictly follows the rule: the number of CPUs in a system that physically are present. Thanks, Peter! -- Vlad Rezki ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: nr_cpu_ids vs AMD 3970x(32 physical CPUs) 2020-07-03 17:09 ` Uladzislau Rezki @ 2020-07-03 17:38 ` Peter Zijlstra 2020-07-03 19:26 ` Uladzislau Rezki 0 siblings, 1 reply; 15+ messages in thread From: Peter Zijlstra @ 2020-07-03 17:38 UTC (permalink / raw) To: Uladzislau Rezki Cc: linux-kernel, linux-mm, Andrew Morton, Linus Torvalds, GregKroah-Hartmangregkh On Fri, Jul 03, 2020 at 07:09:41PM +0200, Uladzislau Rezki wrote: > On Fri, Jul 03, 2020 at 06:56:27PM +0200, Peter Zijlstra wrote: > > On Fri, Jul 03, 2020 at 05:57:49PM +0200, Uladzislau Rezki wrote: > > > Hello, folk. > > > > > > I have a system based on AMD 3970x CPUs. It has 32 physical cores > > > and 64 threads. It seems that "nr_cpu_ids" variable is not correctly > > > set on latest 5.8-rc3 kernel. Please have a look below on dmesg output: > > > > > > <snip> > > > urezki@pc638:~$ sudo dmesg | grep 128 > > > [ 0.000000] IOAPIC[0]: apic_id 128, version 33, address 0xfec00000, GSI 0-23 > > > [ 0.000000] smpboot: Allowing 128 CPUs, 64 hotplug CPUs > > > > This is your BIOS saying it needs 128 ids, 64 of which are 'empty'. > > > > I have a box like that as well, if it bothers you boot with: > > "possible_cpus=64" or something. > > > OK, i got it. I thought that "cpu_possible_mask" strictly follows > the rule: the number of CPUs in a system that physically are present. Nah, it's based on ACPI (SRAT IIRC) tables. The case of over-provisioning is useful for systems that support physical hotplug, but I've seen boards without that capability do it too. Just chalk it up to the foibles of BIOS. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: nr_cpu_ids vs AMD 3970x(32 physical CPUs) 2020-07-03 17:38 ` Peter Zijlstra @ 2020-07-03 19:26 ` Uladzislau Rezki 0 siblings, 0 replies; 15+ messages in thread From: Uladzislau Rezki @ 2020-07-03 19:26 UTC (permalink / raw) To: Peter Zijlstra Cc: Uladzislau Rezki, linux-kernel, linux-mm, Andrew Morton, Linus Torvalds, GregKroah-Hartmangregkh On Fri, Jul 03, 2020 at 07:38:14PM +0200, Peter Zijlstra wrote: > On Fri, Jul 03, 2020 at 07:09:41PM +0200, Uladzislau Rezki wrote: > > On Fri, Jul 03, 2020 at 06:56:27PM +0200, Peter Zijlstra wrote: > > > On Fri, Jul 03, 2020 at 05:57:49PM +0200, Uladzislau Rezki wrote: > > > > Hello, folk. > > > > > > > > I have a system based on AMD 3970x CPUs. It has 32 physical cores > > > > and 64 threads. It seems that "nr_cpu_ids" variable is not correctly > > > > set on latest 5.8-rc3 kernel. Please have a look below on dmesg output: > > > > > > > > <snip> > > > > urezki@pc638:~$ sudo dmesg | grep 128 > > > > [ 0.000000] IOAPIC[0]: apic_id 128, version 33, address 0xfec00000, GSI 0-23 > > > > [ 0.000000] smpboot: Allowing 128 CPUs, 64 hotplug CPUs > > > > > > This is your BIOS saying it needs 128 ids, 64 of which are 'empty'. > > > > > > I have a box like that as well, if it bothers you boot with: > > > "possible_cpus=64" or something. > > > > > OK, i got it. I thought that "cpu_possible_mask" strictly follows > > the rule: the number of CPUs in a system that physically are present. > > Nah, it's based on ACPI (SRAT IIRC) tables. The case of > over-provisioning is useful for systems that support physical hotplug, > but I've seen boards without that capability do it too. > > Just chalk it up to the foibles of BIOS. > Yes, i see that such information is propagated by the BIOS to the kernel, at least for x86 systems. Thad sad if i have single socket system then we do not have any physical hotplug ability, thus there is no need in over-provisioning. Agree that it can be hard to fix, since all that depends on ACPI interface. Like you stated in another mail. -- Vlad Rezki ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: nr_cpu_ids vs AMD 3970x(32 physical CPUs) 2020-07-03 15:57 nr_cpu_ids vs AMD 3970x(32 physical CPUs) Uladzislau Rezki 2020-07-03 16:56 ` Peter Zijlstra @ 2020-07-03 17:07 ` Gabriel C 2020-07-03 17:24 ` Uladzislau Rezki 2020-07-03 17:45 ` Peter Zijlstra 2020-07-03 19:05 ` peter enderborg 2 siblings, 2 replies; 15+ messages in thread From: Gabriel C @ 2020-07-03 17:07 UTC (permalink / raw) To: Uladzislau Rezki Cc: LKML, linux-mm, Andrew Morton, Linus Torvalds, GregKroah-Hartmangregkh, Peter Zijlstra (Intel) Am Fr., 3. Juli 2020 um 17:58 Uhr schrieb Uladzislau Rezki <urezki@gmail.com>: > > Hello, folk. Hello, > > I have a system based on AMD 3970x CPUs. It has 32 physical cores > and 64 threads. It seems that "nr_cpu_ids" variable is not correctly > set on latest 5.8-rc3 kernel. Please have a look below on dmesg output: > > <snip> > urezki@pc638:~$ sudo dmesg | grep 128 > [ 0.000000] IOAPIC[0]: apic_id 128, version 33, address 0xfec00000, GSI 0-23 > [ 0.000000] smpboot: Allowing 128 CPUs, 64 hotplug CPUs > [ 0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:128 nr_node_ids:1 > ... > [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=128, Nodes=1 > [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=128. > [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=128 > urezki@pc638:~$ > <snip> > > For example SLUB thinks that it deals with 128 CPUs in the system what is > wrong if i do not miss something. Since nr_cpu_ids is broken(?), thus the > "cpu_possible_mask" does not correspond to reality as well. > > Any thoughts? This is not a 5.8-rc3 problem. Almost all AMD CPUs and APUs are looking like this. The only CPUs I own are getting that right is a dual EPYC box, everything else is broken regarding the right C/T & socket(s) count, and that probably bc is using NUAM code to have the info. I reported that a while back and no-one ever cared. There is even a comment in the hotplug code saying setting the wrong CPU count is a waste of resources. I have a 2200G is reporting 48Cores. AMD Ryzen 7 3750H reporting twice the cores and twice the socket. ... [ 0.040578] smpboot: Allowing 16 CPUs, 8 hotplug CPUs ... [ 0.382122] smpboot: Max logical packages: 2 .. I boot all the boxes restricting the cores to the correct count on the command line. Wasted resource or not, this is still a bug IMO. Best Regards, Gabriel C ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: nr_cpu_ids vs AMD 3970x(32 physical CPUs) 2020-07-03 17:07 ` Gabriel C @ 2020-07-03 17:24 ` Uladzislau Rezki 2020-07-03 17:45 ` Peter Zijlstra 1 sibling, 0 replies; 15+ messages in thread From: Uladzislau Rezki @ 2020-07-03 17:24 UTC (permalink / raw) To: Gabriel C Cc: Uladzislau Rezki, LKML, linux-mm, Andrew Morton, Linus Torvalds, GregKroah-Hartmangregkh, Peter Zijlstra (Intel) > > > > I have a system based on AMD 3970x CPUs. It has 32 physical cores > > and 64 threads. It seems that "nr_cpu_ids" variable is not correctly > > set on latest 5.8-rc3 kernel. Please have a look below on dmesg output: > > > > <snip> > > urezki@pc638:~$ sudo dmesg | grep 128 > > [ 0.000000] IOAPIC[0]: apic_id 128, version 33, address 0xfec00000, GSI 0-23 > > [ 0.000000] smpboot: Allowing 128 CPUs, 64 hotplug CPUs > > [ 0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:128 nr_node_ids:1 > > ... > > [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=128, Nodes=1 > > [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=128. > > [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=128 > > urezki@pc638:~$ > > <snip> > > > > For example SLUB thinks that it deals with 128 CPUs in the system what is > > wrong if i do not miss something. Since nr_cpu_ids is broken(?), thus the > > "cpu_possible_mask" does not correspond to reality as well. > > > > Any thoughts? > > This is not a 5.8-rc3 problem. Almost all AMD CPUs and APUs are > looking like this. > The only CPUs I own are getting that right is a dual EPYC box, > everything else is broken > regarding the right C/T & socket(s) count, and that probably bc is > using NUAM code > to have the info. > > I reported that a while back and no-one ever cared. > > There is even a comment in the hotplug code saying setting the wrong CPU count > is a waste of resources. > > I have a 2200G is reporting 48Cores. > > AMD Ryzen 7 3750H reporting twice the cores and twice the socket. > > ... > > [ 0.040578] smpboot: Allowing 16 CPUs, 8 hotplug CPUs > ... > [ 0.382122] smpboot: Max logical packages: 2 > .. > > I boot all the boxes restricting the cores to the correct count on the > command line. > > Wasted resource or not, this is still a bug IMO. > I suspect that DEFINE_PER_CPU variables can be twice as big, but i have not checked it actually. So, if the code needs to identify real number of CPUs it can be a challenge :) Thanks. -- Vlad Rezki ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: nr_cpu_ids vs AMD 3970x(32 physical CPUs) 2020-07-03 17:07 ` Gabriel C 2020-07-03 17:24 ` Uladzislau Rezki @ 2020-07-03 17:45 ` Peter Zijlstra 2020-07-03 18:36 ` Gabriel C 1 sibling, 1 reply; 15+ messages in thread From: Peter Zijlstra @ 2020-07-03 17:45 UTC (permalink / raw) To: Gabriel C Cc: Uladzislau Rezki, LKML, linux-mm, Andrew Morton, Linus Torvalds, Rafael J. Wysocki On Fri, Jul 03, 2020 at 07:07:39PM +0200, Gabriel C wrote: > I boot all the boxes restricting the cores to the correct count on the > command line. This, because you're right about the wasted memory. > Wasted resource or not, this is still a bug IMO. Yeah, but not one we can do much about I think. It is the BIOS saying it wants more because it expects someone to come along and stick another CPU in. Possible we could say that for single socket machines overprovisioning is 'silly', but then, I've no idea how to detect that. You'll need to find an ACPI person. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: nr_cpu_ids vs AMD 3970x(32 physical CPUs) 2020-07-03 17:45 ` Peter Zijlstra @ 2020-07-03 18:36 ` Gabriel C 0 siblings, 0 replies; 15+ messages in thread From: Gabriel C @ 2020-07-03 18:36 UTC (permalink / raw) To: Peter Zijlstra Cc: Uladzislau Rezki, LKML, linux-mm, Andrew Morton, Linus Torvalds, Rafael J. Wysocki Am Fr., 3. Juli 2020 um 19:45 Uhr schrieb Peter Zijlstra <peterz@infradead.org>: > > On Fri, Jul 03, 2020 at 07:07:39PM +0200, Gabriel C wrote: > > > I boot all the boxes restricting the cores to the correct count on the > > command line. > > This, because you're right about the wasted memory. > > > Wasted resource or not, this is still a bug IMO. > > Yeah, but not one we can do much about I think. It is the BIOS saying it > wants more because it expects someone to come along and stick another > CPU in. > > Possible we could say that for single socket machines overprovisioning > is 'silly', but then, I've no idea how to detect that. You'll need to > find an ACPI person. I know the EPYC box got that problem too initially, it reported twice the cores and twice the sockets, but got fixed in some kernel versions. https://lkml.org/lkml/2018/5/20/102 I never really looked at how this is calculated but I still believe there is a bug somewhere. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: nr_cpu_ids vs AMD 3970x(32 physical CPUs) 2020-07-03 15:57 nr_cpu_ids vs AMD 3970x(32 physical CPUs) Uladzislau Rezki 2020-07-03 16:56 ` Peter Zijlstra 2020-07-03 17:07 ` Gabriel C @ 2020-07-03 19:05 ` peter enderborg 2020-07-03 19:28 ` Uladzislau Rezki 2 siblings, 1 reply; 15+ messages in thread From: peter enderborg @ 2020-07-03 19:05 UTC (permalink / raw) To: Uladzislau Rezki, linux-kernel, linux-mm Cc: Andrew Morton, Linus Torvalds, GregKroah-Hartmangregkh, peterz On 7/3/20 5:57 PM, Uladzislau Rezki wrote: > Hello, folk. > > I have a system based on AMD 3970x CPUs. It has 32 physical cores > and 64 threads. It seems that "nr_cpu_ids" variable is not correctly > set on latest 5.8-rc3 kernel. Please have a look below on dmesg output: > > <snip> > urezki@pc638:~$ sudo dmesg | grep 128 > [ 0.000000] IOAPIC[0]: apic_id 128, version 33, address 0xfec00000, GSI 0-23 > [ 0.000000] smpboot: Allowing 128 CPUs, 64 hotplug CPUs > [ 0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:128 nr_node_ids:1 My 3950 do [ 0.005271] ACPI: SSDT 0x00000000CA1F5000 0003F1 (v02 ALASKA CPUSSDT 01072009 AMI 01072009) [ 0.108266] smpboot: Allowing 32 CPUs, 0 hotplug CPUs [ 0.111384] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:32 nr_cpu_ids:32 nr_node_ids:1 (Fedora F32 5.6.18-300) What motherboard and BIOs do you use? > ... > [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=128, Nodes=1 > [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=128. > [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=128 > urezki@pc638:~$ > <snip> > > For example SLUB thinks that it deals with 128 CPUs in the system what is > wrong if i do not miss something. Since nr_cpu_ids is broken(?), thus the > "cpu_possible_mask" does not correspond to reality as well. > > Any thoughts? > > Thanks! > > -- > Vlad Rezki ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: nr_cpu_ids vs AMD 3970x(32 physical CPUs) 2020-07-03 19:05 ` peter enderborg @ 2020-07-03 19:28 ` Uladzislau Rezki 2020-07-03 20:08 ` Linus Torvalds 0 siblings, 1 reply; 15+ messages in thread From: Uladzislau Rezki @ 2020-07-03 19:28 UTC (permalink / raw) To: peter enderborg Cc: Uladzislau Rezki, linux-kernel, linux-mm, Andrew Morton, Linus Torvalds, GregKroah-Hartmangregkh, peterz > > Hello, folk. > > > > I have a system based on AMD 3970x CPUs. It has 32 physical cores > > and 64 threads. It seems that "nr_cpu_ids" variable is not correctly > > set on latest 5.8-rc3 kernel. Please have a look below on dmesg output: > > > > <snip> > > urezki@pc638:~$ sudo dmesg | grep 128 > > [ 0.000000] IOAPIC[0]: apic_id 128, version 33, address 0xfec00000, GSI 0-23 > > [ 0.000000] smpboot: Allowing 128 CPUs, 64 hotplug CPUs > > [ 0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:128 nr_node_ids:1 > > My 3950 do > > > [ 0.005271] ACPI: SSDT 0x00000000CA1F5000 0003F1 (v02 ALASKA CPUSSDT 01072009 AMI 01072009) > [ 0.108266] smpboot: Allowing 32 CPUs, 0 hotplug CPUs > [ 0.111384] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:32 nr_cpu_ids:32 nr_node_ids:1 > > (Fedora F32 5.6.18-300) What motherboard and BIOs do you use? > I have MSI TRX40 with latest BIOS. -- Vlad Rezki ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: nr_cpu_ids vs AMD 3970x(32 physical CPUs) 2020-07-03 19:28 ` Uladzislau Rezki @ 2020-07-03 20:08 ` Linus Torvalds 2020-07-03 21:20 ` Uladzislau Rezki 0 siblings, 1 reply; 15+ messages in thread From: Linus Torvalds @ 2020-07-03 20:08 UTC (permalink / raw) To: Uladzislau Rezki Cc: peter enderborg, Linux Kernel Mailing List, Linux-MM, Andrew Morton, GregKroah-Hartmangregkh, Peter Zijlstra On Fri, Jul 3, 2020 at 12:28 PM Uladzislau Rezki <urezki@gmail.com> wrote: > > I have MSI TRX40 with latest BIOS. I think it's just that the BIOS is set for the max possible, in case you'd have a 3990X. I compile my kernel with CONFIG_NR_CPUS's set to 64. That works around the issue. Lots of distros seem to set CONFIG_MAXSMP to true, which I guess is the most generic thing to do, but the problem with that is not just the silly problem with the BIOS, but it also means that the kernel does dynamic allocation for cpumasks even if you _don't_ have that problem, because at compile-time you don't know how big the cpumask will be. With CONFIG_NR_CPUS's set to 64, the kernel will just use a "unsigned long" on the stack (and in various data structures) and be done with it, and not do unnecessary dynamic allocations. Linus ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: nr_cpu_ids vs AMD 3970x(32 physical CPUs) 2020-07-03 20:08 ` Linus Torvalds @ 2020-07-03 21:20 ` Uladzislau Rezki 2020-07-03 21:51 ` Matthew Wilcox 0 siblings, 1 reply; 15+ messages in thread From: Uladzislau Rezki @ 2020-07-03 21:20 UTC (permalink / raw) To: Linus Torvalds Cc: Uladzislau Rezki, peter enderborg, Linux Kernel Mailing List, Linux-MM, Andrew Morton, GregKroah-Hartmangregkh, Peter Zijlstra > On Fri, Jul 3, 2020 at 12:28 PM Uladzislau Rezki <urezki@gmail.com> wrote: > > > > I have MSI TRX40 with latest BIOS. > > I think it's just that the BIOS is set for the max possible, in case > you'd have a 3990X. > 3990x is the top one in this series, so indeed it can be a case and explanation why nr_cpu_ids is set to 128. > > I compile my kernel with CONFIG_NR_CPUS's set to 64. That works around > the issue. > > Lots of distros seem to set CONFIG_MAXSMP to true, which I guess is > the most generic thing to do, but the problem with that is not just > the silly problem with the BIOS, but it also means that the kernel > does dynamic allocation for cpumasks even if you _don't_ have that > problem, because at compile-time you don't know how big the cpumask > will be. > > With CONFIG_NR_CPUS's set to 64, the kernel will just use a "unsigned > long" on the stack (and in various data structures) and be done with > it, and not do unnecessary dynamic allocations. > Thanks for proposed workaround! I will update the CONFIG_NR_CPUS with proper value in my .config Some background: Actually i have been thinking about making vmalloc address space to be per-CPU, i.e. divide it to per-CPU address space making an allocation lock-less. It will eliminate a high lock contention. When i have done a prototype i noticed and realized that there is a silly issue with nr_cpu_ids on some systems. Therefore i reported about it. Thanks, Linus! -- Vlad Rezki ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: nr_cpu_ids vs AMD 3970x(32 physical CPUs) 2020-07-03 21:20 ` Uladzislau Rezki @ 2020-07-03 21:51 ` Matthew Wilcox 2020-07-03 22:22 ` Uladzislau Rezki 0 siblings, 1 reply; 15+ messages in thread From: Matthew Wilcox @ 2020-07-03 21:51 UTC (permalink / raw) To: Uladzislau Rezki Cc: Linus Torvalds, peter enderborg, Linux Kernel Mailing List, Linux-MM, Andrew Morton, GregKroah-Hartmangregkh, Peter Zijlstra On Fri, Jul 03, 2020 at 11:20:47PM +0200, Uladzislau Rezki wrote: > Some background: > Actually i have been thinking about making vmalloc address space to > be per-CPU, i.e. divide it to per-CPU address space making an allocation > lock-less. It will eliminate a high lock contention. When i have done > a prototype i noticed and realized that there is a silly issue with > nr_cpu_ids on some systems. vfree() may happen on a different CPU from the one which called vmalloc(), so I'm not sure you're going to get as large a win as you think you will. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: nr_cpu_ids vs AMD 3970x(32 physical CPUs) 2020-07-03 21:51 ` Matthew Wilcox @ 2020-07-03 22:22 ` Uladzislau Rezki 0 siblings, 0 replies; 15+ messages in thread From: Uladzislau Rezki @ 2020-07-03 22:22 UTC (permalink / raw) To: Matthew Wilcox Cc: Uladzislau Rezki, Linus Torvalds, peter enderborg, Linux Kernel Mailing List, Linux-MM, Andrew Morton, GregKroah-Hartmangregkh, Peter Zijlstra On Fri, Jul 03, 2020 at 10:51:57PM +0100, Matthew Wilcox wrote: > On Fri, Jul 03, 2020 at 11:20:47PM +0200, Uladzislau Rezki wrote: > > Some background: > > Actually i have been thinking about making vmalloc address space to > > be per-CPU, i.e. divide it to per-CPU address space making an allocation > > lock-less. It will eliminate a high lock contention. When i have done > > a prototype i noticed and realized that there is a silly issue with > > nr_cpu_ids on some systems. > > vfree() may happen on a different CPU from the one which called vmalloc(), > so I'm not sure you're going to get as large a win as you think you will. > Hmm.. According to my tests the difference is approximately 7x/10x but i also need to say as of now those tests are draft. Indeed vfree() can be done on different CPU, but i do not think it is a big issue. The main goal is to make the vmalloc() to be scaled to number of CPUs in a system. Because as number of CPUs increase as tight an allocation becomes. Doing vfree() on another CPU would be kind of noise(critical section is short), whereas other ones will be able to do progress because of their own locks. -- Vlad Rezki ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2020-07-03 22:22 UTC | newest] Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-07-03 15:57 nr_cpu_ids vs AMD 3970x(32 physical CPUs) Uladzislau Rezki 2020-07-03 16:56 ` Peter Zijlstra 2020-07-03 17:09 ` Uladzislau Rezki 2020-07-03 17:38 ` Peter Zijlstra 2020-07-03 19:26 ` Uladzislau Rezki 2020-07-03 17:07 ` Gabriel C 2020-07-03 17:24 ` Uladzislau Rezki 2020-07-03 17:45 ` Peter Zijlstra 2020-07-03 18:36 ` Gabriel C 2020-07-03 19:05 ` peter enderborg 2020-07-03 19:28 ` Uladzislau Rezki 2020-07-03 20:08 ` Linus Torvalds 2020-07-03 21:20 ` Uladzislau Rezki 2020-07-03 21:51 ` Matthew Wilcox 2020-07-03 22:22 ` Uladzislau Rezki
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).