LKML Archive on lore.kernel.org
 help / color / Atom feed
From: "Luis R. Rodriguez" <mcgrof@kernel.org>
To: Dmitry Torokhov <dmitry.torokhov@gmail.com>
Cc: "Luis R. Rodriguez" <mcgrof@kernel.org>,
	akpm@linux-foundation.org, jeyu@redhat.com, shuah@kernel.org,
	rusty@rustcorp.com.au, ebiederm@xmission.com, acme@redhat.com,
	corbet@lwn.net, martin.wilck@suse.com, mmarek@suse.com,
	pmladek@suse.com, hare@suse.com, rwright@hpe.com, jeffm@suse.com,
	DSterba@suse.com, fdmanana@suse.com, neilb@suse.com,
	linux@roeck-us.net, rgoldwyn@suse.com, subashab@codeaurora.org,
	xypron.glpk@gmx.de, keescook@chromium.org, atomlin@redhat.com,
	mbenes@suse.cz, paulmck@linux.vnet.ibm.com,
	dan.j.williams@intel.com, jpoimboe@redhat.com,
	davem@davemloft.net, mingo@redhat.com, alan@linux.intel.com,
	tytso@mit.edu, gregkh@linuxfoundation.org,
	torvalds@linux-foundation.org, linux-kselftest@vger.kernel.org,
	linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 2/5] kmod: reduce atomic operations on kmod_concurrent
Date: Fri, 26 May 2017 22:03:40 +0200
Message-ID: <20170526200340.GX8951@wotan.suse.de> (raw)
In-Reply-To: <20170526011100.GF29859@dtor-ws>

On Thu, May 25, 2017 at 06:11:00PM -0700, Dmitry Torokhov wrote:
> On Thu, May 25, 2017 at 05:16:27PM -0700, Luis R. Rodriguez wrote:
> > When checking if we want to allow a kmod thread to kick off we increment,
> > then read to see if we should enable a thread. If we were over the allowed
> > limit limit we decrement. Splitting the increment far apart from decrement
> > means there could be a time where two increments happen potentially
> > giving a false failure on a thread which should have been allowed.
> > 
> > CPU1			CPU2
> > atomic_inc()
> > 			atomic_inc()
> > atomic_read()
> > 			atomic_read()
> > atomic_dec()
> > 			atomic_dec()
> > 
> > In this case a read on CPU1 gets the atomic_inc()'s and we could negate
> > it from getting a kmod thread. We could try to prevent this with a lock
> > or preemption but that is overkill. We can fix by reducing the number of
> > atomic operations. We do this by inverting the logic of of the enabler,
> > instead of incrementing kmod_concurrent as we get new kmod users, define the
> > variable kmod_concurrent_max as the max number of currently allowed kmod
> > users and as we get new kmod users just decrement it if its still positive.
> > This combines the dec and read in one atomic operation.
> > 
> > In this case we no longer get the same false failure:
> > 
> > CPU1			CPU2
> > atomic_dec_if_positive()
> > 			atomic_dec_if_positive()
> > atomic_inc()
> > 			atomic_inc()
> > 
> > Suggested-by: Petr Mladek <pmladek@suse.com>
> > Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> > ---
> >  include/linux/kmod.h |  2 ++
> >  init/main.c          |  1 +
> >  kernel/kmod.c        | 44 +++++++++++++++++++++++++-------------------
> >  3 files changed, 28 insertions(+), 19 deletions(-)
> > 
> > diff --git a/include/linux/kmod.h b/include/linux/kmod.h
> > index c4e441e00db5..8e2f302b214a 100644
> > --- a/include/linux/kmod.h
> > +++ b/include/linux/kmod.h
> > @@ -38,10 +38,12 @@ int __request_module(bool wait, const char *name, ...);
> >  #define request_module_nowait(mod...) __request_module(false, mod)
> >  #define try_then_request_module(x, mod...) \
> >  	((x) ?: (__request_module(true, mod), (x)))
> > +void init_kmod_umh(void);
> >  #else
> >  static inline int request_module(const char *name, ...) { return -ENOSYS; }
> >  static inline int request_module_nowait(const char *name, ...) { return -ENOSYS; }
> >  #define try_then_request_module(x, mod...) (x)
> > +static inline void init_kmod_umh(void) { }
> >  #endif
> >  
> >  
> > diff --git a/init/main.c b/init/main.c
> > index 9ec09ff8a930..9b20be716cf7 100644
> > --- a/init/main.c
> > +++ b/init/main.c
> > @@ -650,6 +650,7 @@ asmlinkage __visible void __init start_kernel(void)
> >  	thread_stack_cache_init();
> >  	cred_init();
> >  	fork_init();
> > +	init_kmod_umh();
> >  	proc_caches_init();
> >  	buffer_init();
> >  	key_init();
> > diff --git a/kernel/kmod.c b/kernel/kmod.c
> > index 563f97e2be36..cafd27b92d19 100644
> > --- a/kernel/kmod.c
> > +++ b/kernel/kmod.c
> > @@ -46,6 +46,7 @@
> >  #include <trace/events/module.h>
> >  
> >  extern int max_threads;
> > +unsigned int max_modprobes;
> >  
> >  #define CAP_BSET	(void *)1
> >  #define CAP_PI		(void *)2
> > @@ -56,6 +57,8 @@ static DEFINE_SPINLOCK(umh_sysctl_lock);
> >  static DECLARE_RWSEM(umhelper_sem);
> >  
> >  #ifdef CONFIG_MODULES
> > +static atomic_t kmod_concurrent_max = ATOMIC_INIT(0);
> > +#define MAX_KMOD_CONCURRENT 50	/* Completely arbitrary value - KAO */
> >  
> >  /*
> >  	modprobe_path is set via /proc/sys.
> > @@ -127,10 +130,7 @@ int __request_module(bool wait, const char *fmt, ...)
> >  {
> >  	va_list args;
> >  	char module_name[MODULE_NAME_LEN];
> > -	unsigned int max_modprobes;
> >  	int ret;
> > -	static atomic_t kmod_concurrent = ATOMIC_INIT(0);
> > -#define MAX_KMOD_CONCURRENT 50	/* Completely arbitrary value - KAO */
> >  	static int kmod_loop_msg;
> >  
> >  	/*
> > @@ -154,21 +154,7 @@ int __request_module(bool wait, const char *fmt, ...)
> >  	if (ret)
> >  		return ret;
> >  
> > -	/* If modprobe needs a service that is in a module, we get a recursive
> > -	 * loop.  Limit the number of running kmod threads to max_threads/2 or
> > -	 * MAX_KMOD_CONCURRENT, whichever is the smaller.  A cleaner method
> > -	 * would be to run the parents of this process, counting how many times
> > -	 * kmod was invoked.  That would mean accessing the internals of the
> > -	 * process tables to get the command line, proc_pid_cmdline is static
> > -	 * and it is not worth changing the proc code just to handle this case. 
> > -	 * KAO.
> > -	 *
> > -	 * "trace the ppid" is simple, but will fail if someone's
> > -	 * parent exits.  I think this is as good as it gets. --RR
> > -	 */
> > -	max_modprobes = min(max_threads/2, MAX_KMOD_CONCURRENT);
> > -	atomic_inc(&kmod_concurrent);
> > -	if (atomic_read(&kmod_concurrent) > max_modprobes) {
> > +	if (atomic_dec_if_positive(&kmod_concurrent_max) < 0) {
> >  		/* We may be blaming an innocent here, but unlikely */
> >  		if (kmod_loop_msg < 5) {
> >  			printk(KERN_ERR
> > @@ -184,10 +170,30 @@ int __request_module(bool wait, const char *fmt, ...)
> >  
> >  	ret = call_modprobe(module_name, wait ? UMH_WAIT_PROC : UMH_WAIT_EXEC);
> >  
> > -	atomic_dec(&kmod_concurrent);
> > +	atomic_inc(&kmod_concurrent_max);
> > +
> >  	return ret;
> >  }
> >  EXPORT_SYMBOL(__request_module);
> > +
> > +/*
> > + * If modprobe needs a service that is in a module, we get a recursive
> > + * loop.  Limit the number of running kmod threads to max_threads/2 or
> > + * MAX_KMOD_CONCURRENT, whichever is the smaller.  A cleaner method
> > + * would be to run the parents of this process, counting how many times
> > + * kmod was invoked.  That would mean accessing the internals of the
> > + * process tables to get the command line, proc_pid_cmdline is static
> > + * and it is not worth changing the proc code just to handle this case.
> > + *
> > + * "trace the ppid" is simple, but will fail if someone's
> > + * parent exits.  I think this is as good as it gets.
> > + */
> > +void __init init_kmod_umh(void)
> > +{
> > +	max_modprobes = min(max_threads/2, MAX_KMOD_CONCURRENT);
> > +	atomic_set(&kmod_concurrent_max, max_modprobes);
> 
> I would love if we could initialize atomic statically. So the trouble we
> are trying to solve here is we create more threads than kernel supports,
> with thread count being calculated as:
> 
> 	threads = div64_u64((u64) totalram_pages * (u64) PAGE_SIZE,
> 			    (u64) THREAD_SIZE * 8UL);
> 
> So to not being serve 50 threads we need to deal with system smaller
> than 3200 pages, or ~13M memory (assume thread size is 8 pages - 64 bit
> with kasan, smaller page sizes reduce memory even more). Can you run
> 4.12 with modules support on machine with such memory?
> 
> So maybe we shoudl simply say:
> 
> static atomic_t kmod_concurrent_max = ATOMIC_INIT(MAX_KMOD_CONCURRENT_MAX);
> 
> and call it a day? So we do not need init_kmod_umh() and don't need to
> call it from init/main.c.

I like this very much. If shit blows up we can simply use the kconfig thing
I had proposed earlier, however it would have to accept less than 50 threads
given we'd be computing this at build time, not at run time.

  Luis

  reply index

Thread overview: 69+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-05-19  3:24 [PATCH 0/6] kmod: few simple enhancements Luis R. Rodriguez
2017-05-19  3:24 ` [PATCH 1/6] kmod: add dynamic max concurrent thread count Luis R. Rodriguez
2017-05-19 20:44   ` Dmitry Torokhov
     [not found]     ` <CAB=NE6XGL24O+JfTNUG0HO4obhDc-v+HyL0SCrQELiZrj2-qNw@mail.gmail.com>
     [not found]       ` <CAB=NE6Wa4Nemh80yaCCwbjrNRLPD+GJMncg12APg9Vq63AWVng@mail.gmail.com>
     [not found]         ` <CAB=NE6Vc6RDAytn2Pkv2V58HFo8ncR0eOHZ3===kbZ2NF78ubg@mail.gmail.com>
     [not found]           ` <CAB=NE6Vqmx=y6muenpuQKynTP=pGWMF8tzoCA0BXD6d63q9wPg@mail.gmail.com>
2017-05-19 21:58             ` Dmitry Torokhov
2017-05-25 16:22               ` Luis R. Rodriguez
2017-05-25 16:38                 ` Dmitry Torokhov
2017-05-25 16:50                   ` Luis R. Rodriguez
2017-05-25 17:30                     ` Dmitry Torokhov
2017-05-25 17:38                       ` Luis R. Rodriguez
2017-05-25 18:06                         ` Luis R. Rodriguez
2017-05-25 18:26                           ` Dmitry Torokhov
2017-05-25 19:01                             ` Luis R. Rodriguez
2017-05-25 21:38                               ` Luis R. Rodriguez
2017-05-19  3:24 ` [PATCH 2/6] module: use list_for_each_entry_rcu() on find_module_all() Luis R. Rodriguez
2017-05-23 14:46   ` Miroslav Benes
2017-05-19  3:24 ` [PATCH 3/6] kmod: provide wrappers for kmod_concurrent inc/dec Luis R. Rodriguez
2017-05-19  3:24 ` [PATCH 4/6] kmod: return -EBUSY if modprobe limit is reached Luis R. Rodriguez
2017-05-19  3:24 ` [PATCH 5/6] kmod: preempt on kmod_umh_threads_get() Luis R. Rodriguez
2017-05-19 22:27   ` Dmitry Torokhov
2017-05-25  0:14     ` Luis R. Rodriguez
2017-05-25  0:45       ` Dmitry Torokhov
2017-05-25  1:00         ` Luis R. Rodriguez
2017-05-25  2:27           ` Dmitry Torokhov
2017-05-25 11:19             ` Petr Mladek
2017-05-25 15:38               ` Luis R. Rodriguez
2017-05-25 16:42               ` Dmitry Torokhov
2017-05-25 15:18             ` Jessica Yu
2017-05-19  3:24 ` [PATCH 6/6] kmod: use simplified rate limit printk Luis R. Rodriguez
2017-05-19 22:23   ` Dmitry Torokhov
2017-05-23  9:00     ` Petr Mladek
2017-05-26  0:16 ` [PATCH v2 0/5] kmod: help make deterministic Luis R. Rodriguez
2017-05-26  0:16   ` [PATCH v2 1/5] module: use list_for_each_entry_rcu() on find_module_all() Luis R. Rodriguez
2017-05-26  0:16   ` [PATCH v2 2/5] kmod: reduce atomic operations on kmod_concurrent Luis R. Rodriguez
2017-05-26  1:11     ` Dmitry Torokhov
2017-05-26 20:03       ` Luis R. Rodriguez [this message]
2017-05-26  0:16   ` [PATCH v2 3/5] kmod: add test driver to stress test the module loader Luis R. Rodriguez
2017-05-26  0:16   ` [PATCH v2 4/5] kmod: add helpers for getting kmod limit Luis R. Rodriguez
2017-05-26  0:56     ` Dmitry Torokhov
2017-05-26 20:27       ` Luis R. Rodriguez
2017-05-26  0:16   ` [PATCH v2 5/5] kmod: throttle kmod thread limit Luis R. Rodriguez
2017-05-26 21:12   ` [PATCH v3 0/4] kmod: help make deterministic Luis R. Rodriguez
2017-05-26 21:12     ` [PATCH v3 1/4] module: use list_for_each_entry_rcu() on find_module_all() Luis R. Rodriguez
2017-05-26 21:12     ` [PATCH v3 2/4] kmod: reduce atomic operations on kmod_concurrent and simplify Luis R. Rodriguez
2017-06-23 19:19       ` [PATCH v4 " Luis R. Rodriguez
2017-06-26 11:36       ` [PATCH v3 " Petr Mladek
2017-05-26 21:12     ` [PATCH v3 3/4] kmod: add test driver to stress test the module loader Luis R. Rodriguez
2017-05-26 21:12     ` [PATCH v3 4/4] kmod: throttle kmod thread limit Luis R. Rodriguez
2017-06-22 15:19       ` Petr Mladek
2017-06-23 16:16         ` Luis R. Rodriguez
2017-06-23 17:56           ` Luis R. Rodriguez
2017-06-23 19:16             ` Luis R. Rodriguez
2017-06-26 10:03               ` Petr Mladek
2017-06-26  9:55           ` Petr Mladek
2017-06-23 19:20       ` [PATCH v4 " Luis R. Rodriguez
2017-06-26 11:38         ` Petr Mladek
2017-06-28 22:11           ` Luis R. Rodriguez
2017-06-20 20:56     ` [PATCH v3 0/4] kmod: help make deterministic Luis R. Rodriguez
2017-06-21  0:23       ` Kees Cook
2017-06-26 21:37         ` Jessica Yu
2017-06-26 22:44           ` Luis R. Rodriguez
2017-06-27  0:27             ` Luis R. Rodriguez
2017-06-27  8:13               ` Petr Mladek
2017-06-27 10:04                 ` Jessica Yu
2017-06-27 15:26             ` Jessica Yu
2017-06-28  0:49               ` Luis R. Rodriguez
2017-06-28 22:31     ` [PATCH v4 0/3] " Luis R. Rodriguez
2017-06-28 22:31       ` [PATCH v4 1/3] MAINTAINERS: give kmod some maintainer love Luis R. Rodriguez
2017-06-28 22:31       ` [PATCH v4 2/3] kmod: add test driver to stress test the module loader Luis R. Rodriguez
2017-06-28 22:31       ` [PATCH v4 3/3] kmod: throttle kmod thread limit Luis R. Rodriguez

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20170526200340.GX8951@wotan.suse.de \
    --to=mcgrof@kernel.org \
    --cc=DSterba@suse.com \
    --cc=acme@redhat.com \
    --cc=akpm@linux-foundation.org \
    --cc=alan@linux.intel.com \
    --cc=atomlin@redhat.com \
    --cc=corbet@lwn.net \
    --cc=dan.j.williams@intel.com \
    --cc=davem@davemloft.net \
    --cc=dmitry.torokhov@gmail.com \
    --cc=ebiederm@xmission.com \
    --cc=fdmanana@suse.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=hare@suse.com \
    --cc=jeffm@suse.com \
    --cc=jeyu@redhat.com \
    --cc=jpoimboe@redhat.com \
    --cc=keescook@chromium.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=linux@roeck-us.net \
    --cc=martin.wilck@suse.com \
    --cc=mbenes@suse.cz \
    --cc=mingo@redhat.com \
    --cc=mmarek@suse.com \
    --cc=neilb@suse.com \
    --cc=paulmck@linux.vnet.ibm.com \
    --cc=pmladek@suse.com \
    --cc=rgoldwyn@suse.com \
    --cc=rusty@rustcorp.com.au \
    --cc=rwright@hpe.com \
    --cc=shuah@kernel.org \
    --cc=subashab@codeaurora.org \
    --cc=torvalds@linux-foundation.org \
    --cc=tytso@mit.edu \
    --cc=xypron.glpk@gmx.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git
	git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git