All of lore.kernel.org
 help / color / mirror / Atom feed
From: Vlastimil Babka <vbabka@suse.cz>
To: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Cc: kernel test robot <oliver.sang@intel.com>,
	Oliver Glitta <glittao@gmail.com>,
	lkp@lists.01.org, lkp@intel.com,
	LKML <linux-kernel@vger.kernel.org>,
	linux-mm@kvack.org, Mike Rapoport <rppt@kernel.org>
Subject: Re: [mm/slub] ae107fa919: BUG:unable_to_handle_page_fault_for_address
Date: Sat, 12 Mar 2022 10:21:25 +0100	[thread overview]
Message-ID: <a8dc8bc4-9a00-b2f7-1f68-273f7a14a14c@suse.cz> (raw)
In-Reply-To: <YivzD7PYilkFwjFt@ip-172-31-19-208.ap-northeast-1.compute.internal>

On 3/12/22 02:10, Hyeonggon Yoo wrote:
> On Fri, Mar 11, 2022 at 04:46:00PM +0000, Hyeonggon Yoo wrote:
>> On Fri, Mar 11, 2022 at 04:36:47PM +0100, Vlastimil Babka wrote:
>>> On 3/11/22 15:54, Hyeonggon Yoo wrote:
>>>> On Wed, Mar 09, 2022 at 10:15:31AM +0800, kernel test robot wrote:
>>>>>
>>>>>
>>>>> Greeting,
>>>>>
>>>>> FYI, we noticed the following commit (built with gcc-9):
>>>>>
>>>>> commit: ae107fa91914f098cd54ab77e68f83dd6259e901 ("mm/slub: use stackdepot to save stack trace in objects")
>>>>> https://git.kernel.org/cgit/linux/kernel/git/vbabka/linux.git slub-stackdepot-v3r0
>>>>>
>>>>> in testcase: boot
>>>>>
>>>>> on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G
>>>>>
>>>>> caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
>>>>>
>>>>
>>>> [+Cc Vlastimil and linux-mm]
>>>
>>> Thanks.
>>> lkp folks: it would be nice if I was CC'd automatically on this, it's a
>>> commit from my git tree and with by s-o-b :)
>>>
>>>> I _strongly_ suspect that this is because we don't initialize
>>>> stack_table[i] = NULL when we allocate it from memblock_alloc().
>>>
>>> No, Mike (CC'd) suggested to drop the array init cycle, because
>>> memblock_alloc would zero the area anyway.
>>
>> Ah, you are right. My mistake.
>>
>>> There has to be a different
>>> reason. Wondering if dmesg contains the stack depot initialization message
>>> at all...
>>
>> I think I found the reason.
>> This is because of CONFIG_SLUB_DEBUG_ON.
>> It can enable debugging without passing boot parameter.
>>
>> if CONFIG_SLUB_DEBUG_ON=y && slub_debug is not passed, we do not call
>> stack_depot_want_early_init(), but the debugging flags are set.
>>
>> And we only call stack_depot_init() later in kmem_cache_create_usercopy().
>>
>> so it crashed while creating boot cache.
> 
> I tested this, and this was the reason.
> It crashed on CONFIG_SLUB_DEBUG_ON=y because stackdepot always assume
> that it was initialized in boot step, or failed
> (stack_depot_disable=true).
> 
> But as it didn't even tried to initialize it, stack_table == NULL &&
> stack_depot_disable == false. So accessing *(NULL + <hash value>)

Thanks for finding the cause!

> Ideas? implementing something like kmem_cache_init_early() again?

I think we could simply make CONFIG_SLUB_DEBUG_ON select/depend on
STACKDEPOT_ALWAYS_INIT?

WARNING: multiple messages have this Message-ID (diff)
From: Vlastimil Babka <vbabka@suse.cz>
To: lkp@lists.01.org
Subject: Re: [mm/slub] ae107fa919: BUG:unable_to_handle_page_fault_for_address
Date: Sat, 12 Mar 2022 10:21:25 +0100	[thread overview]
Message-ID: <a8dc8bc4-9a00-b2f7-1f68-273f7a14a14c@suse.cz> (raw)
In-Reply-To: <YivzD7PYilkFwjFt@ip-172-31-19-208.ap-northeast-1.compute.internal>

[-- Attachment #1: Type: text/plain, Size: 2423 bytes --]

On 3/12/22 02:10, Hyeonggon Yoo wrote:
> On Fri, Mar 11, 2022 at 04:46:00PM +0000, Hyeonggon Yoo wrote:
>> On Fri, Mar 11, 2022 at 04:36:47PM +0100, Vlastimil Babka wrote:
>>> On 3/11/22 15:54, Hyeonggon Yoo wrote:
>>>> On Wed, Mar 09, 2022 at 10:15:31AM +0800, kernel test robot wrote:
>>>>>
>>>>>
>>>>> Greeting,
>>>>>
>>>>> FYI, we noticed the following commit (built with gcc-9):
>>>>>
>>>>> commit: ae107fa91914f098cd54ab77e68f83dd6259e901 ("mm/slub: use stackdepot to save stack trace in objects")
>>>>> https://git.kernel.org/cgit/linux/kernel/git/vbabka/linux.git slub-stackdepot-v3r0
>>>>>
>>>>> in testcase: boot
>>>>>
>>>>> on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G
>>>>>
>>>>> caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
>>>>>
>>>>
>>>> [+Cc Vlastimil and linux-mm]
>>>
>>> Thanks.
>>> lkp folks: it would be nice if I was CC'd automatically on this, it's a
>>> commit from my git tree and with by s-o-b :)
>>>
>>>> I _strongly_ suspect that this is because we don't initialize
>>>> stack_table[i] = NULL when we allocate it from memblock_alloc().
>>>
>>> No, Mike (CC'd) suggested to drop the array init cycle, because
>>> memblock_alloc would zero the area anyway.
>>
>> Ah, you are right. My mistake.
>>
>>> There has to be a different
>>> reason. Wondering if dmesg contains the stack depot initialization message
>>> at all...
>>
>> I think I found the reason.
>> This is because of CONFIG_SLUB_DEBUG_ON.
>> It can enable debugging without passing boot parameter.
>>
>> if CONFIG_SLUB_DEBUG_ON=y && slub_debug is not passed, we do not call
>> stack_depot_want_early_init(), but the debugging flags are set.
>>
>> And we only call stack_depot_init() later in kmem_cache_create_usercopy().
>>
>> so it crashed while creating boot cache.
> 
> I tested this, and this was the reason.
> It crashed on CONFIG_SLUB_DEBUG_ON=y because stackdepot always assume
> that it was initialized in boot step, or failed
> (stack_depot_disable=true).
> 
> But as it didn't even tried to initialize it, stack_table == NULL &&
> stack_depot_disable == false. So accessing *(NULL + <hash value>)

Thanks for finding the cause!

> Ideas? implementing something like kmem_cache_init_early() again?

I think we could simply make CONFIG_SLUB_DEBUG_ON select/depend on
STACKDEPOT_ALWAYS_INIT?

  reply	other threads:[~2022-03-12  9:22 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-03-09  2:15 [mm/slub] ae107fa919: BUG:unable_to_handle_page_fault_for_address kernel test robot
2022-03-09  2:15 ` kernel test robot
2022-03-11 14:54 ` Hyeonggon Yoo
2022-03-11 14:54   ` Hyeonggon Yoo
2022-03-11 15:36   ` Vlastimil Babka
2022-03-11 15:36     ` Vlastimil Babka
2022-03-11 16:46     ` Hyeonggon Yoo
2022-03-11 16:46       ` Hyeonggon Yoo
2022-03-12  1:10       ` Hyeonggon Yoo
2022-03-12  1:10         ` Hyeonggon Yoo
2022-03-12  9:21         ` Vlastimil Babka [this message]
2022-03-12  9:21           ` Vlastimil Babka
2022-03-12 11:04           ` Hyeonggon Yoo
2022-03-12 11:04             ` Hyeonggon Yoo
2022-03-12 12:06             ` Mike Rapoport
2022-03-12 12:06               ` Mike Rapoport
2022-03-12 13:08               ` Vlastimil Babka
2022-03-12 13:08                 ` Vlastimil Babka
2022-03-14 16:20                 ` Vlastimil Babka
2022-03-14 16:20                   ` Vlastimil Babka

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=a8dc8bc4-9a00-b2f7-1f68-273f7a14a14c@suse.cz \
    --to=vbabka@suse.cz \
    --cc=42.hyeyoo@gmail.com \
    --cc=glittao@gmail.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=lkp@intel.com \
    --cc=lkp@lists.01.org \
    --cc=oliver.sang@intel.com \
    --cc=rppt@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.