* [PATCH] debugobjects: scale the static pool size
@ 2018-11-20 6:42 Qian Cai
2018-11-20 13:50 ` Waiman Long
0 siblings, 1 reply; 4+ messages in thread
From: Qian Cai @ 2018-11-20 6:42 UTC (permalink / raw)
To: akpm, tglx; +Cc: longman, yang.shi, arnd, linux-kernel, Qian Cai
The current value of the early boot static pool size is not big enough
for systems with large number of CPUs with timer or/and workqueue
objects selected. As the results, systems have 60+ CPUs with both timer
and workqueue objects enabled could trigger "ODEBUG: Out of memory.
ODEBUG disabled". Hence, fixed it by computing it according to
CONFIG_NR_CPUS and CONFIG_DEBUG_OBJECTS_* options.
Signed-off-by: Qian Cai <cai@gmx.us>
---
lib/debugobjects.c | 53 +++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 52 insertions(+), 1 deletion(-)
diff --git a/lib/debugobjects.c b/lib/debugobjects.c
index 70935ed91125..372dc34206d5 100644
--- a/lib/debugobjects.c
+++ b/lib/debugobjects.c
@@ -23,7 +23,53 @@
#define ODEBUG_HASH_BITS 14
#define ODEBUG_HASH_SIZE (1 << ODEBUG_HASH_BITS)
+/*
+ * Some debug objects are allocated during the early boot. Enabling some
+ * options like timers or workqueue objects may increase the size required
+ * significantly with large number of CPUs. For example,
+ *
+ * No. CPUs x 2 (worker pool) objects:
+ *
+ * start_kernel
+ * workqueue_init_early
+ * init_worker_pool
+ * init_timer_key
+ * debug_object_init
+ *
+ * No. CPUs objects (CONFIG_HIGH_RES_TIMERS):
+ *
+ * sched_init
+ * hrtick_rq_init
+ * hrtimer_init
+ *
+ * CONFIG_DEBUG_OBJECTS_WORK:
+ * No. CPUs x 6 (workqueue) objects:
+ *
+ * workqueue_init_early
+ * alloc_workqueue
+ * __alloc_workqueue_key
+ * alloc_and_link_pwqs
+ * init_pwq
+ *
+ * Also, plus No. CPUs objects:
+ *
+ * perf_event_init
+ * __init_srcu_struct
+ * init_srcu_struct_fields
+ * init_srcu_struct_nodes
+ * __init_work
+ *
+ * Increase the number a bit more in case the implmentatins are changed in
+ * the future.
+ */
+#if defined(CONFIG_NR_CPUS) && defined(CONFIG_DEBUG_OBJECTS_TIMERS) && \
+!defined(CONFIG_DEBUG_OBJECTS_WORK)
+#define ODEBUG_POOL_SIZE (CONFIG_NR_CPUS * 10)
+#elif defined(CONFIG_NR_CPUS) && defined(CONFIG_DEBUG_OBJECTS_WORK)
+#define ODEBUG_POOL_SIZE (CONFIG_NR_CPUS * 30)
+#else
#define ODEBUG_POOL_SIZE 1024
+#endif /* CONFIG_NR_CPUS */
#define ODEBUG_POOL_MIN_LEVEL 256
#define ODEBUG_CHUNK_SHIFT PAGE_SHIFT
@@ -58,8 +104,13 @@ static int debug_objects_fixups __read_mostly;
static int debug_objects_warnings __read_mostly;
static int debug_objects_enabled __read_mostly
= CONFIG_DEBUG_OBJECTS_ENABLE_DEFAULT;
+/*
+ * This is only used after replaced static objects, so no need to scale it
+ * to use the early boot static pool size and it has already been scaled
+ * according to actual No. CPUs in the box within debug_objects_mem_init().
+ */
static int debug_objects_pool_size __read_mostly
- = ODEBUG_POOL_SIZE;
+ = 1024;
static int debug_objects_pool_min_level __read_mostly
= ODEBUG_POOL_MIN_LEVEL;
static struct debug_obj_descr *descr_test __read_mostly;
--
2.17.2 (Apple Git-113)
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] debugobjects: scale the static pool size
2018-11-20 6:42 [PATCH] debugobjects: scale the static pool size Qian Cai
@ 2018-11-20 13:50 ` Waiman Long
2018-11-20 15:12 ` Qian Cai
0 siblings, 1 reply; 4+ messages in thread
From: Waiman Long @ 2018-11-20 13:50 UTC (permalink / raw)
To: Qian Cai, akpm, tglx; +Cc: yang.shi, arnd, linux-kernel
On 11/20/2018 01:42 AM, Qian Cai wrote:
> The current value of the early boot static pool size is not big enough
> for systems with large number of CPUs with timer or/and workqueue
> objects selected. As the results, systems have 60+ CPUs with both timer
> and workqueue objects enabled could trigger "ODEBUG: Out of memory.
> ODEBUG disabled". Hence, fixed it by computing it according to
> CONFIG_NR_CPUS and CONFIG_DEBUG_OBJECTS_* options.
>
> Signed-off-by: Qian Cai <cai@gmx.us>
> ---
> lib/debugobjects.c | 53 +++++++++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 52 insertions(+), 1 deletion(-)
>
> diff --git a/lib/debugobjects.c b/lib/debugobjects.c
> index 70935ed91125..372dc34206d5 100644
> --- a/lib/debugobjects.c
> +++ b/lib/debugobjects.c
> @@ -23,7 +23,53 @@
> #define ODEBUG_HASH_BITS 14
> #define ODEBUG_HASH_SIZE (1 << ODEBUG_HASH_BITS)
>
> +/*
> + * Some debug objects are allocated during the early boot. Enabling some
> + * options like timers or workqueue objects may increase the size required
> + * significantly with large number of CPUs. For example,
> + *
> + * No. CPUs x 2 (worker pool) objects:
> + *
> + * start_kernel
> + * workqueue_init_early
> + * init_worker_pool
> + * init_timer_key
> + * debug_object_init
> + *
> + * No. CPUs objects (CONFIG_HIGH_RES_TIMERS):
> + *
> + * sched_init
> + * hrtick_rq_init
> + * hrtimer_init
> + *
> + * CONFIG_DEBUG_OBJECTS_WORK:
> + * No. CPUs x 6 (workqueue) objects:
> + *
> + * workqueue_init_early
> + * alloc_workqueue
> + * __alloc_workqueue_key
> + * alloc_and_link_pwqs
> + * init_pwq
> + *
> + * Also, plus No. CPUs objects:
> + *
> + * perf_event_init
> + * __init_srcu_struct
> + * init_srcu_struct_fields
> + * init_srcu_struct_nodes
> + * __init_work
> + *
> + * Increase the number a bit more in case the implmentatins are changed in
> + * the future.
> + */
> +#if defined(CONFIG_NR_CPUS) && defined(CONFIG_DEBUG_OBJECTS_TIMERS) && \
> +!defined(CONFIG_DEBUG_OBJECTS_WORK)
> +#define ODEBUG_POOL_SIZE (CONFIG_NR_CPUS * 10)
> +#elif defined(CONFIG_NR_CPUS) && defined(CONFIG_DEBUG_OBJECTS_WORK)
> +#define ODEBUG_POOL_SIZE (CONFIG_NR_CPUS * 30)
> +#else
> #define ODEBUG_POOL_SIZE 1024
> +#endif /* CONFIG_NR_CPUS */
> #define ODEBUG_POOL_MIN_LEVEL 256
>
CONFIG_NR_CPUS is always defined. You don't need to put that as a #if
condition. Where does the scaling factor 30 come from? It looks high to me.
For UP system, CONFIG_NR_CPUS will be 1. I think it is better to have a
guarantee minimum plus a multiplier of the # of configured CPUs.
Something like
512 + CONFIG_NR_CPUS * <multiplier>
where <multiplier> should be the sum of all early allocation objects
that scale with the number of cpus. The guarantee minimum will cover
other miscellaneous objects.
Cheers,
Longman
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] debugobjects: scale the static pool size
2018-11-20 13:50 ` Waiman Long
@ 2018-11-20 15:12 ` Qian Cai
2018-11-20 15:45 ` Waiman Long
0 siblings, 1 reply; 4+ messages in thread
From: Qian Cai @ 2018-11-20 15:12 UTC (permalink / raw)
To: Waiman Long
Cc: Andrew Morton, Thomas Gleixner, Yang Shi, Arnd Bergmann, linux kernel
> On Nov 20, 2018, at 8:50 AM, Waiman Long <longman@redhat.com> wrote:
>
> On 11/20/2018 01:42 AM, Qian Cai wrote:
>> The current value of the early boot static pool size is not big enough
>> for systems with large number of CPUs with timer or/and workqueue
>> objects selected. As the results, systems have 60+ CPUs with both timer
>> and workqueue objects enabled could trigger "ODEBUG: Out of memory.
>> ODEBUG disabled". Hence, fixed it by computing it according to
>> CONFIG_NR_CPUS and CONFIG_DEBUG_OBJECTS_* options.
>>
>> Signed-off-by: Qian Cai <cai@gmx.us>
>> ---
>> lib/debugobjects.c | 53 +++++++++++++++++++++++++++++++++++++++++++++-
>> 1 file changed, 52 insertions(+), 1 deletion(-)
>>
>> diff --git a/lib/debugobjects.c b/lib/debugobjects.c
>> index 70935ed91125..372dc34206d5 100644
>> --- a/lib/debugobjects.c
>> +++ b/lib/debugobjects.c
>> @@ -23,7 +23,53 @@
>> #define ODEBUG_HASH_BITS 14
>> #define ODEBUG_HASH_SIZE (1 << ODEBUG_HASH_BITS)
>>
>> +/*
>> + * Some debug objects are allocated during the early boot. Enabling some
>> + * options like timers or workqueue objects may increase the size required
>> + * significantly with large number of CPUs. For example,
>> + *
>> + * No. CPUs x 2 (worker pool) objects:
>> + *
>> + * start_kernel
>> + * workqueue_init_early
>> + * init_worker_pool
>> + * init_timer_key
>> + * debug_object_init
>> + *
>> + * No. CPUs objects (CONFIG_HIGH_RES_TIMERS):
>> + *
>> + * sched_init
>> + * hrtick_rq_init
>> + * hrtimer_init
>> + *
>> + * CONFIG_DEBUG_OBJECTS_WORK:
>> + * No. CPUs x 6 (workqueue) objects:
>> + *
>> + * workqueue_init_early
>> + * alloc_workqueue
>> + * __alloc_workqueue_key
>> + * alloc_and_link_pwqs
>> + * init_pwq
>> + *
>> + * Also, plus No. CPUs objects:
>> + *
>> + * perf_event_init
>> + * __init_srcu_struct
>> + * init_srcu_struct_fields
>> + * init_srcu_struct_nodes
>> + * __init_work
>> + *
>> + * Increase the number a bit more in case the implmentatins are changed in
>> + * the future.
>> + */
>> +#if defined(CONFIG_NR_CPUS) && defined(CONFIG_DEBUG_OBJECTS_TIMERS) && \
>> +!defined(CONFIG_DEBUG_OBJECTS_WORK)
>> +#define ODEBUG_POOL_SIZE (CONFIG_NR_CPUS * 10)
>> +#elif defined(CONFIG_NR_CPUS) && defined(CONFIG_DEBUG_OBJECTS_WORK)
>> +#define ODEBUG_POOL_SIZE (CONFIG_NR_CPUS * 30)
>> +#else
>> #define ODEBUG_POOL_SIZE 1024
>> +#endif /* CONFIG_NR_CPUS */
>> #define ODEBUG_POOL_MIN_LEVEL 256
>>
>
> CONFIG_NR_CPUS is always defined. You don't need to put that as a #if
> condition. Where does the scaling factor 30 come from? It looks high to me.
Hmm, looks like some architectures could have it undefined since it depends
on CONFIG_SMP where the later can be disabled. For example alpha,
config NR_CPUS
int "Maximum number of CPUs (2-32)"
range 2 32
depends on SMP
Scaling factor 30 came from the data, with all the debug_objects options
enabled, I have,
64-CPU: ODEBUG: 1114 of 1114 active objects replaced
256-CPU: ODEBUG: 4378 of 4378 active objects replaced
I also give a bit room for growth in the future since the implementation details
could always change.
>
> For UP system, CONFIG_NR_CPUS will be 1. I think it is better to have a
> guarantee minimum plus a multiplier of the # of configured CPUs.
> Something like
>
> 512 + CONFIG_NR_CPUS * <multiplier>
>
> where <multiplier> should be the sum of all early allocation objects
> that scale with the number of cpus. The guarantee minimum will cover
> other miscellaneous objects.
That is a good catch. I’ll fix that.
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [PATCH] debugobjects: scale the static pool size
2018-11-20 15:12 ` Qian Cai
@ 2018-11-20 15:45 ` Waiman Long
0 siblings, 0 replies; 4+ messages in thread
From: Waiman Long @ 2018-11-20 15:45 UTC (permalink / raw)
To: Qian Cai
Cc: Andrew Morton, Thomas Gleixner, Yang Shi, Arnd Bergmann, linux kernel
On 11/20/2018 10:12 AM, Qian Cai wrote:
>
>> On Nov 20, 2018, at 8:50 AM, Waiman Long <longman@redhat.com> wrote:
>>
>> On 11/20/2018 01:42 AM, Qian Cai wrote:
>>> The current value of the early boot static pool size is not big enough
>>> for systems with large number of CPUs with timer or/and workqueue
>>> objects selected. As the results, systems have 60+ CPUs with both timer
>>> and workqueue objects enabled could trigger "ODEBUG: Out of memory.
>>> ODEBUG disabled". Hence, fixed it by computing it according to
>>> CONFIG_NR_CPUS and CONFIG_DEBUG_OBJECTS_* options.
>>>
>>> Signed-off-by: Qian Cai <cai@gmx.us>
>>> ---
>>> lib/debugobjects.c | 53 +++++++++++++++++++++++++++++++++++++++++++++-
>>> 1 file changed, 52 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/lib/debugobjects.c b/lib/debugobjects.c
>>> index 70935ed91125..372dc34206d5 100644
>>> --- a/lib/debugobjects.c
>>> +++ b/lib/debugobjects.c
>>> @@ -23,7 +23,53 @@
>>> #define ODEBUG_HASH_BITS 14
>>> #define ODEBUG_HASH_SIZE (1 << ODEBUG_HASH_BITS)
>>>
>>> +/*
>>> + * Some debug objects are allocated during the early boot. Enabling some
>>> + * options like timers or workqueue objects may increase the size required
>>> + * significantly with large number of CPUs. For example,
>>> + *
>>> + * No. CPUs x 2 (worker pool) objects:
>>> + *
>>> + * start_kernel
>>> + * workqueue_init_early
>>> + * init_worker_pool
>>> + * init_timer_key
>>> + * debug_object_init
>>> + *
>>> + * No. CPUs objects (CONFIG_HIGH_RES_TIMERS):
>>> + *
>>> + * sched_init
>>> + * hrtick_rq_init
>>> + * hrtimer_init
>>> + *
>>> + * CONFIG_DEBUG_OBJECTS_WORK:
>>> + * No. CPUs x 6 (workqueue) objects:
>>> + *
>>> + * workqueue_init_early
>>> + * alloc_workqueue
>>> + * __alloc_workqueue_key
>>> + * alloc_and_link_pwqs
>>> + * init_pwq
>>> + *
>>> + * Also, plus No. CPUs objects:
>>> + *
>>> + * perf_event_init
>>> + * __init_srcu_struct
>>> + * init_srcu_struct_fields
>>> + * init_srcu_struct_nodes
>>> + * __init_work
>>> + *
>>> + * Increase the number a bit more in case the implmentatins are changed in
>>> + * the future.
>>> + */
>>> +#if defined(CONFIG_NR_CPUS) && defined(CONFIG_DEBUG_OBJECTS_TIMERS) && \
>>> +!defined(CONFIG_DEBUG_OBJECTS_WORK)
>>> +#define ODEBUG_POOL_SIZE (CONFIG_NR_CPUS * 10)
>>> +#elif defined(CONFIG_NR_CPUS) && defined(CONFIG_DEBUG_OBJECTS_WORK)
>>> +#define ODEBUG_POOL_SIZE (CONFIG_NR_CPUS * 30)
>>> +#else
>>> #define ODEBUG_POOL_SIZE 1024
>>> +#endif /* CONFIG_NR_CPUS */
>>> #define ODEBUG_POOL_MIN_LEVEL 256
>>>
>> CONFIG_NR_CPUS is always defined. You don't need to put that as a #if
>> condition. Where does the scaling factor 30 come from? It looks high to me.
> Hmm, looks like some architectures could have it undefined since it depends
> on CONFIG_SMP where the later can be disabled. For example alpha,
>
> config NR_CPUS
> int "Maximum number of CPUs (2-32)"
> range 2 32
> depends on SMP
include/linux/threads.h:
#ifndef CONFIG_NR_CPUS
/* FIXME: This should be fixed in the arch's Kconfig */
#define CONFIG_NR_CPUS 1
#endif
> Scaling factor 30 came from the data, with all the debug_objects options
> enabled, I have,
>
> 64-CPU: ODEBUG: 1114 of 1114 active objects replaced
> 256-CPU: ODEBUG: 4378 of 4378 active objects replaced
>
> I also give a bit room for growth in the future since the implementation details
> could always change.
(4378-1114)/(256-64) = 17
So the max scaling factor is 17. I would say you could round it up to 20
at most.
Cheers,
Longman
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2018-11-20 15:45 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-11-20 6:42 [PATCH] debugobjects: scale the static pool size Qian Cai
2018-11-20 13:50 ` Waiman Long
2018-11-20 15:12 ` Qian Cai
2018-11-20 15:45 ` Waiman Long
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).