linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed
From: Dennis Zhou <dennis@kernel.org>
To: Wang Yugui <wangyugui@e16-tech.com>
Cc: Vlastimil Babka <vbabka@suse.cz>,
	linux-mm@kvack.org, linux-btrfs@vger.kernel.org
Subject: Re: unexpected -ENOMEM from percpu_counter_init()
Date: Wed, 7 Apr 2021 14:56:38 +0000	[thread overview]
Message-ID: <YG3IJnAjgikZ0PkC@google.com> (raw)
In-Reply-To: <20210407210905.F790.409509F4@e16-tech.com>

Hello,

On Wed, Apr 07, 2021 at 09:09:07PM +0800, Wang Yugui wrote:
> Hi,
> 
> > +CC btrfs
> > 
> > On 4/1/21 12:51 PM, Wang Yugui wrote:
> > > Hi,
> > > 
> > > an unexpected -ENOMEM from percpu_counter_init() happened when xfstest 
> > > with kernel 5.11.10 and 5.10.27
> > 
> > Is there a dmesg log showing allocation failure or something?
> 
> When unexpected -ENOMEM of percpu_counter_init(), btrfs as upper caller
> finally output something to dmesg.
> 
> And we add one trace to btrfs source to make sure that.
> >     if (ret == -ENOMEM) printk("ENOMEM btrfs_drew_lock_init\n");
> 
> 
> Now the reproduce frequency become from >50% to not happen or very slow
> with the flowing change.
> 
> diff --git a/mm/percpu.c b/mm/percpu.c
> index 6596a0a..0127be1 100644
> --- a/mm/percpu.c
> +++ b/mm/percpu.c
> @@ -104,8 +104,8 @@
>  /* chunks in slots below this are subject to being sidelined on failed alloc */
>  #define PCPU_SLOT_FAIL_THRESHOLD	3
>  
> -#define PCPU_EMPTY_POP_PAGES_LOW	2
> -#define PCPU_EMPTY_POP_PAGES_HIGH	4
> +#define PCPU_EMPTY_POP_PAGES_LOW	8
> +#define PCPU_EMPTY_POP_PAGES_HIGH	16
>  

These settings are from 2014 when Tejun initially implemented the atomic
allocation float. It is probably time to think about increasing the
number of pages. I'd prefer to do it in a dynamic way though (some X% of
a chunk instead of a fixed number increase).

>  #ifdef CONFIG_SMP
>  /* default addr <-> pcpu_ptr mapping, override in asm/percpu.h if necessary */
> diff --git a/include/linux/percpu.h b/include/linux/percpu.h
> index 5e76af7..8cc091b 100644
> --- a/include/linux/percpu.h
> +++ b/include/linux/percpu.h
> @@ -14,7 +14,7 @@
>  
>  /* enough to cover all DEFINE_PER_CPUs in modules */
>  #ifdef CONFIG_MODULES
> -#define PERCPU_MODULE_RESERVE		(8 << 10)
> +#define PERCPU_MODULE_RESERVE		(32 << 10)
>  #else
>  #define PERCPU_MODULE_RESERVE		0
>  #endif
> 

This is a reserved region purely for module static inits.
btrfs_drew_lock_init() is a dynamic init.

> 
> Just some guess,
> 1) maybe some releationship to the trigger of 'vm.dirty_bytes=10737418240'.
> 
> this problem happen in 
> server/T7610 with E5-2660v2 *2 and SSD/SAS(6Gb/s) and 192G memory
> but not happen in
> server/T620 with E5-2680v2 *2 and SSD/NVMe and 192G memory.
> 
> 2) maybe some releationship to numa.
> 128G memory in node1(CPU1), and 64G in node2(CPU2)
> 
> Best Regards
> Wang Yugui (wangyugui@e16-tech.com)
> 2021/04/07
> 
> 
> > > direct caller:
> > > int btrfs_drew_lock_init(struct btrfs_drew_lock *lock)
> > > {
> > >     int ret;
> > > 
> > >     ret = percpu_counter_init(&lock->writers, 0, GFP_KERNEL);
> > >     if (ret)
> > >         return ret;
> > > 
> > >     atomic_set(&lock->readers, 0);
> > >     init_waitqueue_head(&lock->pending_readers);
> > >     init_waitqueue_head(&lock->pending_writers);
> > > 
> > >     return 0;
> > > }
> > > 
> > > upper caller:
> > >     nofs_flag = memalloc_nofs_save();
> > >     ret = btrfs_drew_lock_init(&root->snapshot_lock);
> > >     memalloc_nofs_restore(nofs_flag);

The issue is here. nofs is set which means percpu attempts an atomic
allocation. If it cannot find anything already allocated it isn't happy.
This was done before memalloc_nofs_{save/restore}() were pervasive.

Percpu should probably try to allocate some pages if possible even if
nofs is set.

> > >     if (ret == -ENOMEM) printk("ENOMEM btrfs_drew_lock_init\n");
> > >     if (ret)
> > >         goto fail;
> > > 
> > > The hardware of this server:
> > > CPU:  Xeon(R) CPU E5-2660 v2(10 core)  *2
> > > memory:  192G, no swap
> > > 
> > > Only one xfstests job is running in this server, and about 7% of memory
> > > is used.
> > > 
> > > Any advice please.
> > > 
> > > Best Regards
> > > Wang Yugui (wangyugui@e16-tech.com)
> > > 2021/04/01
> > > 
> > > 
> 

Thanks,
Dennis


  reply	other threads:[~2021-04-07 14:56 UTC|newest]

Thread overview: 27+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-04-01 10:51 Wang Yugui
2021-04-02  1:49 ` Wang Yugui
2021-04-07 12:35 ` Vlastimil Babka
2021-04-07 13:09   ` Wang Yugui
2021-04-07 14:56     ` Dennis Zhou [this message]
2021-04-07 23:28       ` Wang Yugui
2021-04-08  2:44         ` Dennis Zhou
2021-04-08  9:20           ` Wang Yugui
2021-04-08 13:48             ` Dennis Zhou
2021-04-08 14:28               ` Filipe Manana
2021-04-08 15:02                 ` Dennis Zhou
2021-04-09 11:39                   ` Filipe Manana
2021-04-09 13:39                     ` Dennis Zhou
2021-04-09 13:42                       ` Filipe Manana
2021-04-09  0:08               ` Wang Yugui
2021-04-09  2:14                 ` Dennis Zhou
2021-04-09  4:02                   ` Wang Yugui
2021-04-09  7:36                     ` Wang Yugui
2021-04-09  7:48                       ` Wang Yugui
2021-04-09 13:56                       ` Dennis Zhou
2021-04-10 15:29                         ` Wang Yugui
2021-04-10 15:52                           ` Dennis Zhou
2021-04-10 16:08                             ` Wang Yugui
2021-04-11 15:20                               ` Wang Yugui
2021-04-12  4:03                                 ` Dennis Zhou
2021-04-12  5:24                                   ` Wang Yugui
2021-04-09  9:52   ` Wang Yugui

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=YG3IJnAjgikZ0PkC@google.com \
    --to=dennis@kernel.org \
    --cc=linux-btrfs@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=vbabka@suse.cz \
    --cc=wangyugui@e16-tech.com \
    --subject='Re: unexpected -ENOMEM from percpu_counter_init()' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
on how to clone and mirror all data and code used for this inbox