From: Roman Gushchin <guro@fb.com>
To: Christian Borntraeger <borntraeger@de.ibm.com>
Cc: "akpm@linux-foundation.org" <akpm@linux-foundation.org>,
"hannes@cmpxchg.org" <hannes@cmpxchg.org>,
Kernel Team <Kernel-team@fb.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-mm@kvack.org" <linux-mm@kvack.org>,
"longman@redhat.com" <longman@redhat.com>,
"shakeelb@google.com" <shakeelb@google.com>,
"vdavydov.dev@gmail.com" <vdavydov.dev@gmail.com>,
Heiko Carstens <heiko.carstens@de.ibm.com>,
Janosch Frank <frankja@linux.ibm.com>,
"kvm@vger.kernel.org" <kvm@vger.kernel.org>
Subject: Re: WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management)
Date: Thu, 21 Nov 2019 16:58:11 +0000 [thread overview]
Message-ID: <20191121165807.GA201621@localhost.localdomain> (raw)
In-Reply-To: <20191121111739.3054-1-borntraeger@de.ibm.com>
On Thu, Nov 21, 2019 at 12:17:39PM +0100, Christian Borntraeger wrote:
> Folks,
>
> I do get errors like the following when running a new testcase in our KVM CI.
> The test basically unloads kvm, reloads with with hpage=1 (enable huge page
> support for guests on s390) start a guest with libvirt and hugepages, shut the
> guest down and unload the kvm module.
>
> WARNING: CPU: 8 PID: 208 at lib/percpu-refcount.c:108 percpu_ref_exit+0x50/0x58
> Modules linked in: kvm(-) xt_CHECKSUM xt_MASQUERADE bonding xt_tcpudp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_na>
> CPU: 8 PID: 208 Comm: kworker/8:1 Not tainted 5.2.0+ #66
> Hardware name: IBM 2964 NC9 712 (LPAR)
> Workqueue: events sysfs_slab_remove_workfn
> Krnl PSW : 0704e00180000000 0000001529746850 (percpu_ref_exit+0x50/0x58)
> R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3
> Krnl GPRS: 00000000ffff8808 0000001529746740 000003f4e30e8e18 0036008100000000
> 0000001f00000000 0035008100000000 0000001fb3573ab8 0000000000000000
> 0000001fbdb6de00 0000000000000000 0000001529f01328 0000001fb3573b00
> 0000001fbb27e000 0000001fbdb69300 000003e009263d00 000003e009263cd0
> Krnl Code: 0000001529746842: f0a0000407fe srp 4(11,%r0),2046,0
> 0000001529746848: 47000700 bc 0,1792
> #000000152974684c: a7f40001 brc 15,152974684e
> >0000001529746850: a7f4fff2 brc 15,1529746834
> 0000001529746854: 0707 bcr 0,%r7
> 0000001529746856: 0707 bcr 0,%r7
> 0000001529746858: eb8ff0580024 stmg %r8,%r15,88(%r15)
> 000000152974685e: a738ffff lhi %r3,-1
> Call Trace:
> ([<000003e009263d00>] 0x3e009263d00)
> [<00000015293252ea>] slab_kmem_cache_release+0x3a/0x70
> [<0000001529b04882>] kobject_put+0xaa/0xe8
> [<000000152918cf28>] process_one_work+0x1e8/0x428
> [<000000152918d1b0>] worker_thread+0x48/0x460
> [<00000015291942c6>] kthread+0x126/0x160
> [<0000001529b22344>] ret_from_fork+0x28/0x30
> [<0000001529b2234c>] kernel_thread_starter+0x0/0x10
> Last Breaking-Event-Address:
> [<000000152974684c>] percpu_ref_exit+0x4c/0x58
> ---[ end trace b035e7da5788eb09 ]---
>
> I have bisected this to
> # first bad commit: [f0a3a24b532d9a7e56a33c5112b2a212ed6ec580] mm: memcg/slab: rework non-root kmem_cache lifecycle management
>
> unmounting /sys/fs/cgroup/memory/ before the test makes the problem go away so
> it really seems to be related to the new percpu-refs from this patch.
>
> Any ideas?
Hello, Christian!
It seems to be a race between releasing of the root kmem_cache (caused by rmmod)
and a memcg kmem_cache. Does delaying rmmod for say, a minute, "resolve" the
issue?
I'll take a look at this code path.
Thanks!
next prev parent reply other threads:[~2019-11-21 16:58 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-06-11 23:18 [PATCH v7 00/10] mm: reparent slab memory on cgroup removal Roman Gushchin
2019-06-11 23:18 ` [PATCH v7 01/10] mm: postpone kmem_cache memcg pointer initialization to memcg_link_cache() Roman Gushchin
2019-06-13 2:04 ` Andrew Morton
2019-06-13 16:25 ` Roman Gushchin
2019-06-11 23:18 ` [PATCH v7 02/10] mm: rename slab delayed deactivation functions and fields Roman Gushchin
2019-06-25 18:17 ` Shakeel Butt
2019-06-11 23:18 ` [PATCH v7 03/10] mm: generalize postponed non-root kmem_cache deactivation Roman Gushchin
2019-06-25 18:17 ` Shakeel Butt
2019-06-11 23:18 ` [PATCH v7 04/10] mm: introduce __memcg_kmem_uncharge_memcg() Roman Gushchin
2019-06-11 23:18 ` [PATCH v7 05/10] mm: unify SLAB and SLUB page accounting Roman Gushchin
2019-06-11 23:18 ` [PATCH v7 06/10] mm: don't check the dying flag on kmem_cache creation Roman Gushchin
2019-06-16 16:26 ` Vladimir Davydov
2019-06-25 18:31 ` Shakeel Butt
2019-06-11 23:18 ` [PATCH v7 07/10] mm: synchronize access to kmem_cache dying flag using a spinlock Roman Gushchin
2019-06-16 16:27 ` Vladimir Davydov
2019-06-25 18:33 ` Shakeel Butt
2019-06-11 23:18 ` [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management Roman Gushchin
2019-06-25 23:57 ` Shakeel Butt
2019-11-21 11:17 ` WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management) Christian Borntraeger
2019-11-21 13:08 ` Christian Borntraeger
2019-11-21 16:58 ` Roman Gushchin [this message]
2019-11-21 16:59 ` Christian Borntraeger
2019-11-21 18:45 ` Roman Gushchin
2019-11-21 20:43 ` Rik van Riel
2019-11-21 20:55 ` Roman Gushchin
2019-11-21 22:09 ` Roman Gushchin
2019-11-22 13:00 ` Christian Borntraeger
2019-11-22 16:28 ` Christian Borntraeger
2019-11-24 0:39 ` Roman Gushchin
2019-11-25 8:00 ` Christian Borntraeger
2019-11-25 18:07 ` Roman Gushchin
2019-06-11 23:18 ` [PATCH v7 09/10] mm: stop setting page->mem_cgroup pointer for slab pages Roman Gushchin
2019-06-26 0:15 ` Shakeel Butt
2019-06-11 23:18 ` [PATCH v7 10/10] mm: reparent memcg kmem_caches on cgroup removal Roman Gushchin
2019-06-16 16:29 ` Vladimir Davydov
2019-06-26 0:15 ` Shakeel Butt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191121165807.GA201621@localhost.localdomain \
--to=guro@fb.com \
--cc=Kernel-team@fb.com \
--cc=akpm@linux-foundation.org \
--cc=borntraeger@de.ibm.com \
--cc=frankja@linux.ibm.com \
--cc=hannes@cmpxchg.org \
--cc=heiko.carstens@de.ibm.com \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=longman@redhat.com \
--cc=shakeelb@google.com \
--cc=vdavydov.dev@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).