LKML Archive on lore.kernel.org
 help / color / Atom feed
From: "Luis R. Rodriguez" <mcgrof@kernel.org>
To: shuah@kernel.org, jeyu@redhat.com, rusty@rustcorp.com.au,
	ebiederm@xmission.com, dmitry.torokhov@gmail.com,
	acme@redhat.com, corbet@lwn.net
Cc: martin.wilck@suse.com, mmarek@suse.com, pmladek@suse.com,
	hare@suse.com, rwright@hpe.com, jeffm@suse.com, DSterba@suse.com,
	fdmanana@suse.com, neilb@suse.com, linux@roeck-us.net,
	rgoldwyn@suse.com, subashab@codeaurora.org, xypron.glpk@gmx.de,
	keescook@chromium.org, atomlin@redhat.com, mbenes@suse.cz,
	paulmck@linux.vnet.ibm.com, dan.j.williams@intel.com,
	jpoimboe@redhat.com, davem@davemloft.net, mingo@redhat.com,
	akpm@linux-foundation.org, torvalds@linux-foundation.org,
	gregkh@linuxfoundation.org, linux-kselftest@vger.kernel.org,
	linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org,
	"Luis R. Rodriguez" <mcgrof@kernel.org>
Subject: [PATCH 1/6] kmod: add dynamic max concurrent thread count
Date: Thu, 18 May 2017 20:24:39 -0700
Message-ID: <20170519032444.18416-2-mcgrof@kernel.org> (raw)
In-Reply-To: <20170519032444.18416-1-mcgrof@kernel.org>

We currently statically limit the number of modprobe threads which
we allow to run concurrently to 50. As per Keith Owens, this was a
completely arbitrary value, and it was set in the 2.3.38 days [0]
over 16 years ago in year 2000.

Although we haven't yet hit our lower limits, experimentation [1]
shows that when and if we hit this limit in the worst case, will be
fatal -- consider get_fs_type() failures upon mount on a system which
has many partitions, some of which might even be with the same
filesystem. Its best to be prudent and increase and set this
value to something more sensible which ensures we're far from hitting
the limit and also allows default build/user run time override.

The worst case is fatal given that once a module fails to load there
is a period of time during which subsequent request for the same module
will fail, so in the case of partitions its not just one request that
could fail, but whole series of partitions. This later issue of a
module request failure domino effect can be addressed later, but
increasing the limit to something more meaninful should at least give us
enough cushion to avoid this for a while.

Set this value up with a bit more meaninful modern limits:

Bump this up to 64  max for small systems (CONFIG_BASE_SMALL)
Bump this up to 128 max for larger systems (!CONFIG_BASE_SMALL)

Also allow the default max limit to be further fine tuned at compile
time and at initialization at run time at boot up using the kernel
parameter: max_modprobes.

[0] https://git.kernel.org/cgit/linux/kernel/git/history/history.git/commit/?id=ab1c4ec7410f6ec64e1511d1a7d850fc99c09b44
[1] https://github.com/mcgrof/test_request_module

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 Documentation/admin-guide/kernel-parameters.txt |  7 ++++
 include/linux/kmod.h                            |  3 +-
 init/Kconfig                                    | 23 +++++++++++++
 init/main.c                                     |  1 +
 kernel/kmod.c                                   | 43 ++++++++++++++++---------
 5 files changed, 61 insertions(+), 16 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 15f79c27748d..1314a85c10c9 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -1748,6 +1748,13 @@
 
 	keepinitrd	[HW,ARM]
 
+	kmod.max_modprobes [KNL]
+			This lets you set the max allowed of concurrent
+			modprobes threads possible on a system overriding the
+			default heuristic of:
+
+				min(max_threads/2, 1 << CONFIG_MAX_KMOD_CONCURRENT)
+
 	kernelcore=	[KNL,X86,IA-64,PPC]
 			Format: nn[KMGTPE] | "mirror"
 			This parameter
diff --git a/include/linux/kmod.h b/include/linux/kmod.h
index c4e441e00db5..302c9dfdcda9 100644
--- a/include/linux/kmod.h
+++ b/include/linux/kmod.h
@@ -38,13 +38,14 @@ int __request_module(bool wait, const char *name, ...);
 #define request_module_nowait(mod...) __request_module(false, mod)
 #define try_then_request_module(x, mod...) \
 	((x) ?: (__request_module(true, mod), (x)))
+void init_kmod_umh(void);
 #else
 static inline int request_module(const char *name, ...) { return -ENOSYS; }
 static inline int request_module_nowait(const char *name, ...) { return -ENOSYS; }
+static inline void init_kmod_umh(void) { }
 #define try_then_request_module(x, mod...) (x)
 #endif
 
-
 struct cred;
 struct file;
 
diff --git a/init/Kconfig b/init/Kconfig
index 1d3475fc9496..5974ad7c9d0a 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -2191,6 +2191,29 @@ config TRIM_UNUSED_KSYMS
 
 	  If unsure, or if you need to build out-of-tree modules, say N.
 
+config MAX_KMOD_CONCURRENT
+	int "Max allowed concurrent request_module() calls (6=>64, 10=>1024)"
+	range 6 14
+	default 7 if !BASE_SMALL
+	default 6 if BASE_SMALL
+	help
+	  The kernel restricts the number of possible concurrent calls to
+	  request_module() to help avoid a recursive loop possible with
+	  modules. The default maximum number of concurrent threads allowed
+	  to run request_module() will be:
+
+	    max_modprobes = min(max_threads/2, 1 << CONFIG_MAX_KMOD_CONCURRENT);
+
+	  The value set in CONFIG_MAX_KMOD_CONCURRENT represents then the power
+	  of 2 value used at boot time for the above computation. You can
+	  override the default built value using the kernel parameter:
+
+		kmod.max_modprobes=4096
+
+	  We set this to default to 64 (2^6) concurrent modprobe threads for
+	  small systems, for larger systems this defaults to 128 (2^7)
+	  concurrent modprobe threads.
+
 endif # MODULES
 
 config MODULES_TREE_LOOKUP
diff --git a/init/main.c b/init/main.c
index f866510472d7..25669dfded42 100644
--- a/init/main.c
+++ b/init/main.c
@@ -650,6 +650,7 @@ asmlinkage __visible void __init start_kernel(void)
 	thread_stack_cache_init();
 	cred_init();
 	fork_init();
+	init_kmod_umh();
 	proc_caches_init();
 	buffer_init();
 	key_init();
diff --git a/kernel/kmod.c b/kernel/kmod.c
index 563f97e2be36..6fe6848787b2 100644
--- a/kernel/kmod.c
+++ b/kernel/kmod.c
@@ -46,6 +46,9 @@
 #include <trace/events/module.h>
 
 extern int max_threads;
+unsigned int max_modprobes;
+module_param(max_modprobes, uint, 0644);
+MODULE_PARM_DESC(max_modprobes, "Max number of allowed concurrent modprobes");
 
 #define CAP_BSET	(void *)1
 #define CAP_PI		(void *)2
@@ -127,10 +130,8 @@ int __request_module(bool wait, const char *fmt, ...)
 {
 	va_list args;
 	char module_name[MODULE_NAME_LEN];
-	unsigned int max_modprobes;
 	int ret;
 	static atomic_t kmod_concurrent = ATOMIC_INIT(0);
-#define MAX_KMOD_CONCURRENT 50	/* Completely arbitrary value - KAO */
 	static int kmod_loop_msg;
 
 	/*
@@ -154,19 +155,6 @@ int __request_module(bool wait, const char *fmt, ...)
 	if (ret)
 		return ret;
 
-	/* If modprobe needs a service that is in a module, we get a recursive
-	 * loop.  Limit the number of running kmod threads to max_threads/2 or
-	 * MAX_KMOD_CONCURRENT, whichever is the smaller.  A cleaner method
-	 * would be to run the parents of this process, counting how many times
-	 * kmod was invoked.  That would mean accessing the internals of the
-	 * process tables to get the command line, proc_pid_cmdline is static
-	 * and it is not worth changing the proc code just to handle this case. 
-	 * KAO.
-	 *
-	 * "trace the ppid" is simple, but will fail if someone's
-	 * parent exits.  I think this is as good as it gets. --RR
-	 */
-	max_modprobes = min(max_threads/2, MAX_KMOD_CONCURRENT);
 	atomic_inc(&kmod_concurrent);
 	if (atomic_read(&kmod_concurrent) > max_modprobes) {
 		/* We may be blaming an innocent here, but unlikely */
@@ -188,6 +176,31 @@ int __request_module(bool wait, const char *fmt, ...)
 	return ret;
 }
 EXPORT_SYMBOL(__request_module);
+
+/*
+ * If modprobe needs a service that is in a module, we get a recursive
+ * loop.  Limit the number of running kmod threads to max_threads/2 or
+ * CONFIG_MAX_KMOD_CONCURRENT, whichever is the smaller.  A cleaner method
+ * would be to run the parents of this process, counting how many times
+ * kmod was invoked.  That would mean accessing the internals of the
+ * process tables to get the command line, proc_pid_cmdline is static
+ * and it is not worth changing the proc code just to handle this case.
+ *
+ * "trace the ppid" is simple, but will fail if someone's
+ * parent exits.  I think this is as good as it gets.
+ *
+ * You can override with with a kernel parameter, for instance to allow
+ * 4096 concurrent modprobe instances:
+ *
+ *	kmod.max_modprobes=4096
+ */
+void __init init_kmod_umh(void)
+{
+	if (!max_modprobes)
+		max_modprobes = min(max_threads/2,
+				    1 << CONFIG_MAX_KMOD_CONCURRENT);
+}
+
 #endif /* CONFIG_MODULES */
 
 static void call_usermodehelper_freeinfo(struct subprocess_info *info)
-- 
2.11.0

  reply index

Thread overview: 69+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-05-19  3:24 [PATCH 0/6] kmod: few simple enhancements Luis R. Rodriguez
2017-05-19  3:24 ` Luis R. Rodriguez [this message]
2017-05-19 20:44   ` [PATCH 1/6] kmod: add dynamic max concurrent thread count Dmitry Torokhov
     [not found]     ` <CAB=NE6XGL24O+JfTNUG0HO4obhDc-v+HyL0SCrQELiZrj2-qNw@mail.gmail.com>
     [not found]       ` <CAB=NE6Wa4Nemh80yaCCwbjrNRLPD+GJMncg12APg9Vq63AWVng@mail.gmail.com>
     [not found]         ` <CAB=NE6Vc6RDAytn2Pkv2V58HFo8ncR0eOHZ3===kbZ2NF78ubg@mail.gmail.com>
     [not found]           ` <CAB=NE6Vqmx=y6muenpuQKynTP=pGWMF8tzoCA0BXD6d63q9wPg@mail.gmail.com>
2017-05-19 21:58             ` Dmitry Torokhov
2017-05-25 16:22               ` Luis R. Rodriguez
2017-05-25 16:38                 ` Dmitry Torokhov
2017-05-25 16:50                   ` Luis R. Rodriguez
2017-05-25 17:30                     ` Dmitry Torokhov
2017-05-25 17:38                       ` Luis R. Rodriguez
2017-05-25 18:06                         ` Luis R. Rodriguez
2017-05-25 18:26                           ` Dmitry Torokhov
2017-05-25 19:01                             ` Luis R. Rodriguez
2017-05-25 21:38                               ` Luis R. Rodriguez
2017-05-19  3:24 ` [PATCH 2/6] module: use list_for_each_entry_rcu() on find_module_all() Luis R. Rodriguez
2017-05-23 14:46   ` Miroslav Benes
2017-05-19  3:24 ` [PATCH 3/6] kmod: provide wrappers for kmod_concurrent inc/dec Luis R. Rodriguez
2017-05-19  3:24 ` [PATCH 4/6] kmod: return -EBUSY if modprobe limit is reached Luis R. Rodriguez
2017-05-19  3:24 ` [PATCH 5/6] kmod: preempt on kmod_umh_threads_get() Luis R. Rodriguez
2017-05-19 22:27   ` Dmitry Torokhov
2017-05-25  0:14     ` Luis R. Rodriguez
2017-05-25  0:45       ` Dmitry Torokhov
2017-05-25  1:00         ` Luis R. Rodriguez
2017-05-25  2:27           ` Dmitry Torokhov
2017-05-25 11:19             ` Petr Mladek
2017-05-25 15:38               ` Luis R. Rodriguez
2017-05-25 16:42               ` Dmitry Torokhov
2017-05-25 15:18             ` Jessica Yu
2017-05-19  3:24 ` [PATCH 6/6] kmod: use simplified rate limit printk Luis R. Rodriguez
2017-05-19 22:23   ` Dmitry Torokhov
2017-05-23  9:00     ` Petr Mladek
2017-05-26  0:16 ` [PATCH v2 0/5] kmod: help make deterministic Luis R. Rodriguez
2017-05-26  0:16   ` [PATCH v2 1/5] module: use list_for_each_entry_rcu() on find_module_all() Luis R. Rodriguez
2017-05-26  0:16   ` [PATCH v2 2/5] kmod: reduce atomic operations on kmod_concurrent Luis R. Rodriguez
2017-05-26  1:11     ` Dmitry Torokhov
2017-05-26 20:03       ` Luis R. Rodriguez
2017-05-26  0:16   ` [PATCH v2 3/5] kmod: add test driver to stress test the module loader Luis R. Rodriguez
2017-05-26  0:16   ` [PATCH v2 4/5] kmod: add helpers for getting kmod limit Luis R. Rodriguez
2017-05-26  0:56     ` Dmitry Torokhov
2017-05-26 20:27       ` Luis R. Rodriguez
2017-05-26  0:16   ` [PATCH v2 5/5] kmod: throttle kmod thread limit Luis R. Rodriguez
2017-05-26 21:12   ` [PATCH v3 0/4] kmod: help make deterministic Luis R. Rodriguez
2017-05-26 21:12     ` [PATCH v3 1/4] module: use list_for_each_entry_rcu() on find_module_all() Luis R. Rodriguez
2017-05-26 21:12     ` [PATCH v3 2/4] kmod: reduce atomic operations on kmod_concurrent and simplify Luis R. Rodriguez
2017-06-23 19:19       ` [PATCH v4 " Luis R. Rodriguez
2017-06-26 11:36       ` [PATCH v3 " Petr Mladek
2017-05-26 21:12     ` [PATCH v3 3/4] kmod: add test driver to stress test the module loader Luis R. Rodriguez
2017-05-26 21:12     ` [PATCH v3 4/4] kmod: throttle kmod thread limit Luis R. Rodriguez
2017-06-22 15:19       ` Petr Mladek
2017-06-23 16:16         ` Luis R. Rodriguez
2017-06-23 17:56           ` Luis R. Rodriguez
2017-06-23 19:16             ` Luis R. Rodriguez
2017-06-26 10:03               ` Petr Mladek
2017-06-26  9:55           ` Petr Mladek
2017-06-23 19:20       ` [PATCH v4 " Luis R. Rodriguez
2017-06-26 11:38         ` Petr Mladek
2017-06-28 22:11           ` Luis R. Rodriguez
2017-06-20 20:56     ` [PATCH v3 0/4] kmod: help make deterministic Luis R. Rodriguez
2017-06-21  0:23       ` Kees Cook
2017-06-26 21:37         ` Jessica Yu
2017-06-26 22:44           ` Luis R. Rodriguez
2017-06-27  0:27             ` Luis R. Rodriguez
2017-06-27  8:13               ` Petr Mladek
2017-06-27 10:04                 ` Jessica Yu
2017-06-27 15:26             ` Jessica Yu
2017-06-28  0:49               ` Luis R. Rodriguez
2017-06-28 22:31     ` [PATCH v4 0/3] " Luis R. Rodriguez
2017-06-28 22:31       ` [PATCH v4 1/3] MAINTAINERS: give kmod some maintainer love Luis R. Rodriguez
2017-06-28 22:31       ` [PATCH v4 2/3] kmod: add test driver to stress test the module loader Luis R. Rodriguez
2017-06-28 22:31       ` [PATCH v4 3/3] kmod: throttle kmod thread limit Luis R. Rodriguez

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20170519032444.18416-2-mcgrof@kernel.org \
    --to=mcgrof@kernel.org \
    --cc=DSterba@suse.com \
    --cc=acme@redhat.com \
    --cc=akpm@linux-foundation.org \
    --cc=atomlin@redhat.com \
    --cc=corbet@lwn.net \
    --cc=dan.j.williams@intel.com \
    --cc=davem@davemloft.net \
    --cc=dmitry.torokhov@gmail.com \
    --cc=ebiederm@xmission.com \
    --cc=fdmanana@suse.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=hare@suse.com \
    --cc=jeffm@suse.com \
    --cc=jeyu@redhat.com \
    --cc=jpoimboe@redhat.com \
    --cc=keescook@chromium.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=linux@roeck-us.net \
    --cc=martin.wilck@suse.com \
    --cc=mbenes@suse.cz \
    --cc=mingo@redhat.com \
    --cc=mmarek@suse.com \
    --cc=neilb@suse.com \
    --cc=paulmck@linux.vnet.ibm.com \
    --cc=pmladek@suse.com \
    --cc=rgoldwyn@suse.com \
    --cc=rusty@rustcorp.com.au \
    --cc=rwright@hpe.com \
    --cc=shuah@kernel.org \
    --cc=subashab@codeaurora.org \
    --cc=torvalds@linux-foundation.org \
    --cc=xypron.glpk@gmx.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git
	git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git