Linux-Next Archive on lore.kernel.org
 help / color / Atom feed
From: Michal Hocko <mhocko@kernel.org>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Sachin Sant <sachinp@linux.vnet.ibm.com>,
	Pekka Enberg <penberg@kernel.org>,
	Linux-Next Mailing List <linux-next@vger.kernel.org>,
	David Rientjes <rientjes@google.com>,
	Christopher Lameter <cl@linux.com>,
	linuxppc-dev@lists.ozlabs.org,
	Joonsoo Kim <iamjoonsoo.kim@lge.com>,
	Kirill Tkhai <ktkhai@virtuozzo.com>,
	Benjamin Herrenschmidt <benh@kernel.crashing.org>,
	Paul Mackerras <paulus@samba.org>,
	Michael Ellerman <mpe@ellerman.id.au>
Subject: Re: [5.6.0-rc2-next-20200218/powerpc] Boot failure on POWER9
Date: Thu, 27 Feb 2020 19:26:50 +0100
Message-ID: <20200227182650.GG3771@dhcp22.suse.cz> (raw)
In-Reply-To: <9a86f865-50b5-7483-9257-dbb08fecd62b@suse.cz>

[Cc ppc maintainers]
On Thu 27-02-20 17:16:41, Vlastimil Babka wrote:
> On 2/27/20 5:00 PM, Sachin Sant wrote:
> > 
> > 
> >> On 27-Feb-2020, at 5:42 PM, Michal Hocko <mhocko@kernel.org> wrote:
> >> 
> >> A very good hint indeed. I would do this
> >> diff --git a/include/linux/topology.h b/include/linux/topology.h
> >> index eb2fe6edd73c..d9f1b6737e4d 100644
> >> --- a/include/linux/topology.h
> >> +++ b/include/linux/topology.h
> >> @@ -137,6 +137,8 @@ static inline void set_numa_mem(int node)
> >> {
> >> 	this_cpu_write(_numa_mem_, node);
> >> 	_node_numa_mem_[numa_node_id()] = node;
> >> +	pr_info("%s %d -> %d\n", __FUNCTION__, numa_node_id(), node);
> >> +	dump_stack();
> >> }
> >> #endif
> >> 
> >> Btw. it would be also helpful to get
> >> `faddr2line ___slab_alloc+0x334' from your kernel Sachin.
> > 
> > [linux-next]# ./scripts/faddr2line ./vmlinux ___slab_alloc+0x334 
> > ___slab_alloc+0x334/0x760:
> > new_slab_objects at mm/slub.c:2478
> > (inlined by) ___slab_alloc at mm/slub.c:2628
> > [linux-next]# 
> 
> Hmm that doesn't look relevant, but that address was marked as unreliable, no?
> Don't we actually need this one?
> 
> [    8.768727] NIP [c0000000003d55f4] ___slab_alloc+0x1f4/0x760
> 
> > I have also attached boot log with a kernel that include about change.
> > I see the following o/p during boot:
> > 
> > [    0.005269] set_numa_mem 1 -> 1
> 
> So there's no "set_numa_mem 0 -> X", specifically not
> "set_numa_mem 0 -> 1" which I would have expected. That seems to confirm my
> suspicion that the arch code doesn't set up the memoryless node 0 properly.

Please have a look at http://lkml.kernel.org/r/52EF4673-7292-4C4C-B459-AF583951BA48@linux.vnet.ibm.com
for the boot log with the debugging patch which tracks set_numa_mem.
This seems to lead to a crash in the slab allocator bebcause
node_to_mem_node(0) for memory less node resolves to the memory less
node http://lkml.kernel.org/r/dd450314-d428-6776-af07-f92c04c7b967@suse.cz.
The original report is http://lkml.kernel.org/r/3381CD91-AB3D-4773-BA04-E7A072A63968@linux.vnet.ibm.com

> 
> > [    0.005270] CPU: 12 PID: 0 Comm: swapper/12 Not tainted 5.6.0-rc3-next-20200227-autotest+ #6
> > [    0.005271] Call Trace:
> > [    0.005272] [c0000008b37dfe80] [c000000000b5d948] dump_stack+0xbc/0x104 (unreliable)
> > [    0.005274] [c0000008b37dfec0] [c000000000059320] start_secondary+0x600/0x6e0
> > [    0.005277] [c0000008b37dff90] [c00000000000ac54] start_secondary_prolog+0x10/0x14
> > 
> > Thanks
> > -Sachin
> > 

-- 
Michal Hocko
SUSE Labs

  reply index

Thread overview: 34+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-02-18 10:45 Sachin Sant
2020-02-18 10:50 ` Kirill Tkhai
2020-02-18 11:01   ` Kirill Tkhai
2020-02-18 11:35     ` Kirill Tkhai
2020-02-18 11:40     ` Sachin Sant
2020-02-18 11:55       ` Michal Hocko
2020-02-18 14:00         ` Sachin Sant
2020-02-18 14:26           ` Michal Hocko
2020-02-18 15:11             ` Sachin Sant
2020-02-18 15:24               ` Michal Hocko
2020-02-22  3:38                 ` Christopher Lameter
2020-02-24  8:58                   ` Michal Hocko
2020-02-26 18:25                     ` Christopher Lameter
2020-02-26 18:41                       ` Michal Hocko
2020-02-26 18:44                         ` Christopher Lameter
2020-02-26 19:01                           ` Michal Hocko
2020-02-26 20:31                             ` David Rientjes
2020-02-26 20:52                               ` Michal Hocko
2020-02-26 21:45                         ` Vlastimil Babka
2020-02-26 22:29                           ` Vlastimil Babka
2020-02-27 12:12                             ` Michal Hocko
2020-02-27 16:00                               ` Sachin Sant
2020-02-27 16:16                                 ` Vlastimil Babka
2020-02-27 18:26                                   ` Michal Hocko [this message]
2020-03-10 15:01                                     ` Michal Hocko
2020-03-12 12:18                                       ` Michael Ellerman
2020-03-12 16:51                                         ` Sachin Sant
2020-03-13 10:48                                           ` Michael Ellerman
2020-03-13 11:12                                             ` Srikar Dronamraju
2020-03-13 11:35                                               ` Vlastimil Babka
2020-03-14  8:10                                                 ` Sachin Sant
2020-02-27 12:02                           ` Michal Hocko
2020-02-18 11:38   ` Sachin Sant
2020-02-18 11:53     ` Kirill Tkhai

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200227182650.GG3771@dhcp22.suse.cz \
    --to=mhocko@kernel.org \
    --cc=benh@kernel.crashing.org \
    --cc=cl@linux.com \
    --cc=iamjoonsoo.kim@lge.com \
    --cc=ktkhai@virtuozzo.com \
    --cc=linux-next@vger.kernel.org \
    --cc=linuxppc-dev@lists.ozlabs.org \
    --cc=mpe@ellerman.id.au \
    --cc=paulus@samba.org \
    --cc=penberg@kernel.org \
    --cc=rientjes@google.com \
    --cc=sachinp@linux.vnet.ibm.com \
    --cc=vbabka@suse.cz \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Next Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-next/0 linux-next/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-next linux-next/ https://lore.kernel.org/linux-next \
		linux-next@vger.kernel.org
	public-inbox-index linux-next

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-next


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git