From: Michal Hocko <mhocko@kernel.org>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Sachin Sant <sachinp@linux.vnet.ibm.com>,
Pekka Enberg <penberg@kernel.org>,
Linux-Next Mailing List <linux-next@vger.kernel.org>,
Paul Mackerras <paulus@samba.org>,
David Rientjes <rientjes@google.com>,
Christopher Lameter <cl@linux.com>,
linuxppc-dev@lists.ozlabs.org,
Joonsoo Kim <iamjoonsoo.kim@lge.com>,
Kirill Tkhai <ktkhai@virtuozzo.com>
Subject: Re: [5.6.0-rc2-next-20200218/powerpc] Boot failure on POWER9
Date: Thu, 27 Feb 2020 19:26:50 +0100 [thread overview]
Message-ID: <20200227182650.GG3771@dhcp22.suse.cz> (raw)
In-Reply-To: <9a86f865-50b5-7483-9257-dbb08fecd62b@suse.cz>
[Cc ppc maintainers]
On Thu 27-02-20 17:16:41, Vlastimil Babka wrote:
> On 2/27/20 5:00 PM, Sachin Sant wrote:
> >
> >
> >> On 27-Feb-2020, at 5:42 PM, Michal Hocko <mhocko@kernel.org> wrote:
> >>
> >> A very good hint indeed. I would do this
> >> diff --git a/include/linux/topology.h b/include/linux/topology.h
> >> index eb2fe6edd73c..d9f1b6737e4d 100644
> >> --- a/include/linux/topology.h
> >> +++ b/include/linux/topology.h
> >> @@ -137,6 +137,8 @@ static inline void set_numa_mem(int node)
> >> {
> >> this_cpu_write(_numa_mem_, node);
> >> _node_numa_mem_[numa_node_id()] = node;
> >> + pr_info("%s %d -> %d\n", __FUNCTION__, numa_node_id(), node);
> >> + dump_stack();
> >> }
> >> #endif
> >>
> >> Btw. it would be also helpful to get
> >> `faddr2line ___slab_alloc+0x334' from your kernel Sachin.
> >
> > [linux-next]# ./scripts/faddr2line ./vmlinux ___slab_alloc+0x334
> > ___slab_alloc+0x334/0x760:
> > new_slab_objects at mm/slub.c:2478
> > (inlined by) ___slab_alloc at mm/slub.c:2628
> > [linux-next]#
>
> Hmm that doesn't look relevant, but that address was marked as unreliable, no?
> Don't we actually need this one?
>
> [ 8.768727] NIP [c0000000003d55f4] ___slab_alloc+0x1f4/0x760
>
> > I have also attached boot log with a kernel that include about change.
> > I see the following o/p during boot:
> >
> > [ 0.005269] set_numa_mem 1 -> 1
>
> So there's no "set_numa_mem 0 -> X", specifically not
> "set_numa_mem 0 -> 1" which I would have expected. That seems to confirm my
> suspicion that the arch code doesn't set up the memoryless node 0 properly.
Please have a look at http://lkml.kernel.org/r/52EF4673-7292-4C4C-B459-AF583951BA48@linux.vnet.ibm.com
for the boot log with the debugging patch which tracks set_numa_mem.
This seems to lead to a crash in the slab allocator bebcause
node_to_mem_node(0) for memory less node resolves to the memory less
node http://lkml.kernel.org/r/dd450314-d428-6776-af07-f92c04c7b967@suse.cz.
The original report is http://lkml.kernel.org/r/3381CD91-AB3D-4773-BA04-E7A072A63968@linux.vnet.ibm.com
>
> > [ 0.005270] CPU: 12 PID: 0 Comm: swapper/12 Not tainted 5.6.0-rc3-next-20200227-autotest+ #6
> > [ 0.005271] Call Trace:
> > [ 0.005272] [c0000008b37dfe80] [c000000000b5d948] dump_stack+0xbc/0x104 (unreliable)
> > [ 0.005274] [c0000008b37dfec0] [c000000000059320] start_secondary+0x600/0x6e0
> > [ 0.005277] [c0000008b37dff90] [c00000000000ac54] start_secondary_prolog+0x10/0x14
> >
> > Thanks
> > -Sachin
> >
--
Michal Hocko
SUSE Labs
next prev parent reply other threads:[~2020-02-27 18:29 UTC|newest]
Thread overview: 49+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-02-18 10:45 [5.6.0-rc2-next-20200218/powerpc] Boot failure on POWER9 Sachin Sant
2020-02-18 10:50 ` Kirill Tkhai
2020-02-18 11:01 ` Kirill Tkhai
2020-02-18 11:35 ` Kirill Tkhai
2020-02-18 11:40 ` Sachin Sant
2020-02-18 11:55 ` Michal Hocko
2020-02-18 14:00 ` Sachin Sant
2020-02-18 14:26 ` Michal Hocko
2020-02-18 15:11 ` Sachin Sant
2020-02-18 15:24 ` Michal Hocko
2020-02-22 3:38 ` Christopher Lameter
2020-02-24 8:58 ` Michal Hocko
2020-02-26 18:25 ` Christopher Lameter
2020-02-26 18:41 ` Michal Hocko
2020-02-26 18:44 ` Christopher Lameter
2020-02-26 19:01 ` Michal Hocko
2020-02-26 20:31 ` David Rientjes
2020-02-26 20:52 ` Michal Hocko
2020-02-26 21:45 ` Vlastimil Babka
2020-02-26 22:29 ` Vlastimil Babka
2020-02-27 12:12 ` Michal Hocko
2020-02-27 16:00 ` Sachin Sant
2020-02-27 16:16 ` Vlastimil Babka
2020-02-27 18:26 ` Michal Hocko [this message]
2020-03-10 15:01 ` Michal Hocko
2020-03-12 12:18 ` Michael Ellerman
2020-03-12 16:51 ` Sachin Sant
2020-03-13 10:48 ` Michael Ellerman
2020-03-13 11:12 ` Srikar Dronamraju
2020-03-13 11:35 ` Vlastimil Babka
2020-03-14 8:10 ` Sachin Sant
2020-02-27 12:02 ` Michal Hocko
2020-02-18 11:38 ` Sachin Sant
2020-02-18 11:53 ` Kirill Tkhai
2020-03-17 13:17 ` [PATCH 0/4] Fix kmalloc_node on offline nodes Srikar Dronamraju
2020-03-17 13:17 ` [PATCH 1/4] mm: Check for node_online in node_present_pages Srikar Dronamraju
2020-03-17 13:37 ` Srikar Dronamraju
2020-03-17 13:17 ` [PATCH 2/4] mm/slub: Use mem_node to allocate a new slab Srikar Dronamraju
2020-03-17 13:34 ` Vlastimil Babka
2020-03-17 13:45 ` Srikar Dronamraju
2020-03-17 13:53 ` Vlastimil Babka
2020-03-17 14:51 ` Srikar Dronamraju
2020-03-17 15:29 ` Vlastimil Babka
2020-03-18 7:29 ` Srikar Dronamraju
2020-03-17 16:41 ` Srikar Dronamraju
2020-03-17 13:17 ` [PATCH 3/4] mm: Implement reset_numa_mem Srikar Dronamraju
2020-03-17 13:17 ` [PATCH 4/4] powerpc/numa: Set fallback nodes for offline nodes Srikar Dronamraju
2020-03-17 14:22 ` Bharata B Rao
2020-03-17 14:29 ` Srikar Dronamraju
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200227182650.GG3771@dhcp22.suse.cz \
--to=mhocko@kernel.org \
--cc=cl@linux.com \
--cc=iamjoonsoo.kim@lge.com \
--cc=ktkhai@virtuozzo.com \
--cc=linux-next@vger.kernel.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=paulus@samba.org \
--cc=penberg@kernel.org \
--cc=rientjes@google.com \
--cc=sachinp@linux.vnet.ibm.com \
--cc=vbabka@suse.cz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).