pending 4.14-rc boot failures on arm64

pending 4.14-rc boot failures on arm64

[Dan Rue]

Hi Greg,

It looks like the pending v4.14.8-159-gc2a94d1a609 fails to boot on all
arm64 boards. hikey and juno are failing for us, and I see similar arm64
boot failures reported by kernelci.

Still investigating root cause.

Dan

Re: pending 4.14-rc boot failures on arm64

[Greg Kroah-Hartman]

On Thu, Dec 21, 2017 at 04:35:30PM -0600, Dan Rue wrote:
> Hi Greg,
> 
> It looks like the pending v4.14.8-159-gc2a94d1a609 fails to boot on all
> arm64 boards. hikey and juno are failing for us, and I see similar arm64
> boot failures reported by kernelci.
> 
> Still investigating root cause.

Any help in bisecting to find the problem here would be great...

thanks,

greg k-h

Re: pending 4.14-rc boot failures on arm64

[Naresh Kamboju]

On 22 December 2017 at 12:41, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
> On Thu, Dec 21, 2017 at 04:35:30PM -0600, Dan Rue wrote:
>> Hi Greg,
>>
>> It looks like the pending v4.14.8-159-gc2a94d1a609 fails to boot on all
>> arm64 boards. hikey and juno are failing for us, and I see similar arm64
>> boot failures reported by kernelci.
>>
>> Still investigating root cause.
>
> Any help in bisecting to find the problem here would be great...

Booting now on arm64 Hikey board. After reverting 83e3c48729d.

git revert 83e3c48729d9ebb7af5a31a504f3fd6aff0348c4

Linux version 4.14.9-rc1-00160-gb610c43 (naresh@linaro) (gcc version
4.9.2 20140904 (prerelease) (crosstool-NG linaro-1.13.1-4.9-2014.09 -
Linaro GCC 4.9-
2014.09)) #1 SMP PREEMPT Fri Dec 22 14:21:46 IST 2017

Ref:
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1527427.html#nav
https://bugs.linaro.org/show_bug.cgi?id=3404

- Naresh

>
> thanks,
>
> greg k-h
> _______________________________________________
> Lkft-triage mailing list
> Lkft-triage@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/lkft-triage

Re: pending 4.14-rc boot failures on arm64

[Greg Kroah-Hartman]

On Fri, Dec 22, 2017 at 02:29:55PM +0530, Naresh Kamboju wrote:
> On 22 December 2017 at 12:41, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> > On Thu, Dec 21, 2017 at 04:35:30PM -0600, Dan Rue wrote:
> >> Hi Greg,
> >>
> >> It looks like the pending v4.14.8-159-gc2a94d1a609 fails to boot on all
> >> arm64 boards. hikey and juno are failing for us, and I see similar arm64
> >> boot failures reported by kernelci.
> >>
> >> Still investigating root cause.
> >
> > Any help in bisecting to find the problem here would be great...
> 
> Booting now on arm64 Hikey board. After reverting 83e3c48729d.
> 
> git revert 83e3c48729d9ebb7af5a31a504f3fd6aff0348c4

Any hint as to what the name of that commit is?  The tree is rebased a
lot, that commit is long gone :(

> Linux version 4.14.9-rc1-00160-gb610c43 (naresh@linaro) (gcc version
> 4.9.2 20140904 (prerelease) (crosstool-NG linaro-1.13.1-4.9-2014.09 -
> Linaro GCC 4.9-
> 2014.09)) #1 SMP PREEMPT Fri Dec 22 14:21:46 IST 2017
> 
> Ref:
> https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1527427.html#nav
> https://bugs.linaro.org/show_bug.cgi?id=3404

What does this mean?

confused,

greg k-h

Re: pending 4.14-rc boot failures on arm64

[Naresh Kamboju]

On 22 December 2017 at 14:36, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
> On Fri, Dec 22, 2017 at 02:29:55PM +0530, Naresh Kamboju wrote:
>> On 22 December 2017 at 12:41, Greg Kroah-Hartman
>> <gregkh@linuxfoundation.org> wrote:
>> > On Thu, Dec 21, 2017 at 04:35:30PM -0600, Dan Rue wrote:
>> >> Hi Greg,
>> >>
>> >> It looks like the pending v4.14.8-159-gc2a94d1a609 fails to boot on all
>> >> arm64 boards. hikey and juno are failing for us, and I see similar arm64
>> >> boot failures reported by kernelci.
>> >>
>> >> Still investigating root cause.
>> >
>> > Any help in bisecting to find the problem here would be great...
>>
>> Booting now on arm64 Hikey board. After reverting 83e3c48729d.
>>
>> git revert 83e3c48729d9ebb7af5a31a504f3fd6aff0348c4
>
> Any hint as to what the name of that commit is?  The tree is rebased a
> lot, that commit is long gone :(

After reverting the below commit the boot successful.

commit 83e3c48729d9ebb7af5a31a504f3fd6aff0348c4
mm/sparsemem: Allocate mem_section at runtime for CONFIG_SPARSEMEM_EXTREME=y

>From the history of linux-next,
I have reported linux-next 20171102 boot failure on arm64 on November 2nd
Bug 3404 - linux-next-20171102 doesn't boot on HiKey
https://bugs.linaro.org/show_bug.cgi?id=3404

one of the comments from bug says
"This bug is been fixed by applying this patch."
[PATCH] mm, sparse: Fix boot on arm64

[start quote]
Since 83e3c48729d9 ("mm/sparsemem: Allocate mem_section at runtime for
CONFIG_SPARSEMEM_EXTREME=y") we allocate mem_section dynamically in
sparse_memory_present_with_active_regions(). But some architectures, like
arm64, don't use the routine to initialize sparsemem.

Let's move the initialization into memory_present() it should cover all
architectures.

Signed-off-by: Kirill A. Shutemov <kirill.shute...@linux.intel.com>
Fixes: 83e3c48729d9 ("mm/sparsemem: Allocate mem_section at runtime for
CONFIG_SPARSEMEM_EXTREME=y")
---
 mm/page_alloc.c | 10 ----------
 mm/sparse.c     | 10 ++++++++++
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 8dfd13f724d9..77e4d3c5c57b 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5646,16 +5646,6 @@ void __init
sparse_memory_present_with_active_regions(int nid)
        unsigned long start_pfn, end_pfn;
        int i, this_nid;

-#ifdef CONFIG_SPARSEMEM_EXTREME
-       if (!mem_section) {
-               unsigned long size, align;
-
-               size = sizeof(struct mem_section) * NR_SECTION_ROOTS;
-               align = 1 << (INTERNODE_CACHE_SHIFT);
-               mem_section = memblock_virt_alloc(size, align);
-       }
-#endif
-
        for_each_mem_pfn_range(i, nid, &start_pfn, &end_pfn, &this_nid)
                memory_present(this_nid, start_pfn, end_pfn);
 }
diff --git a/mm/sparse.c b/mm/sparse.c
index b00a97398795..d294148ba395 100644
--- a/mm/sparse.c
+++ b/mm/sparse.c
@@ -206,6 +206,16 @@ void __init memory_present(int nid, unsigned long start,
unsigned long end)
 {
        unsigned long pfn;

+#ifdef CONFIG_SPARSEMEM_EXTREME
+       if (unlikely(!mem_section)) {
+               unsigned long size, align;
+
+               size = sizeof(struct mem_section) * NR_SECTION_ROOTS;
+               align = 1 << (INTERNODE_CACHE_SHIFT);
+               mem_section = memblock_virt_alloc(size, align);
+       }
+#endif
+
        start &= PAGE_SECTION_MASK;
        mminit_validate_memmodel_limits(&start, &end);
        for (pfn = start; pfn < end; pfn += PAGES_PER_SECTION) {

[end quote]

Please refer this link for more information,
Re: [tip:x86/mm] mm/sparsemem: Allocate mem_section at runtime for
CONFIG_SPARSEMEM_EXTREME=y
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1527427.html#nav

>
>> Linux version 4.14.9-rc1-00160-gb610c43 (naresh@linaro) (gcc version
>> 4.9.2 20140904 (prerelease) (crosstool-NG linaro-1.13.1-4.9-2014.09 -
>> Linaro GCC 4.9-
>> 2014.09)) #1 SMP PREEMPT Fri Dec 22 14:21:46 IST 2017
>>
>> Ref:
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1527427.html#nav
https://bugs.linaro.org/show_bug.cgi?id=3404
>
> What does this mean?
It was additional information.
Reference thread and bug links about this commit and bug investigation.

>
> confused,
>
> greg k-h

Re: pending 4.14-rc boot failures on arm64

[Naresh Kamboju]

On 22 December 2017 at 15:50, Naresh Kamboju <naresh.kamboju@linaro.org> wrote:
> On 22 December 2017 at 14:36, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
>> On Fri, Dec 22, 2017 at 02:29:55PM +0530, Naresh Kamboju wrote:
>>> On 22 December 2017 at 12:41, Greg Kroah-Hartman
>>> <gregkh@linuxfoundation.org> wrote:
>>> > On Thu, Dec 21, 2017 at 04:35:30PM -0600, Dan Rue wrote:
>>> >> Hi Greg,
>>> >>
>>> >> It looks like the pending v4.14.8-159-gc2a94d1a609 fails to boot on all
>>> >> arm64 boards. hikey and juno are failing for us, and I see similar arm64
>>> >> boot failures reported by kernelci.
>>> >>
>>> >> Still investigating root cause.
>>> >
>>> > Any help in bisecting to find the problem here would be great...
>>>
>>> Booting now on arm64 Hikey board. After reverting 83e3c48729d.
>>>
>>> git revert 83e3c48729d9ebb7af5a31a504f3fd6aff0348c4

Reverting above commit 83e3c48 (mm/sparsemem: Allocate mem_section at)
might not be the right way to fix this problem.
Instead cherry pick this commit 629a359bd to fix ARM64 boot crash problem.
(mm/sparsemem: Fix ARM64 boot crash when CONFIG_SPARSEMEM_EXTREME=y)

tree: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
branch: linux-4.14.y

git cherry-pick  629a359bdb0e0652a8227b4ff3125431995fec6e

$ git log --oneline -2
599755e mm/sparsemem: Fix ARM64 boot crash when CONFIG_SPARSEMEM_EXTREME=y
b483e86 Linux 4.14.9-rc1

Kernel version after the successful boot on Hikey.
Linux version 4.14.9-rc1-00161-g599755e (naresh@linaro) (gcc version
6.2.1 20161016 (Linaro GCC 6.2-2016.11-rc2)) #2 SMP PREEMPT Fri Dec 22
17:48:59 IST 2017

More information,
Found this commit from linux tree
----

>From 629a359bdb0e0652a8227b4ff3125431995fec6e Mon Sep 17 00:00:00 2001
From: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Date: Tue, 7 Nov 2017 11:33:37 +0300
Subject: mm/sparsemem: Fix ARM64 boot crash when CONFIG_SPARSEMEM_EXTREME=y

Since commit:

  83e3c48729d9 ("mm/sparsemem: Allocate mem_section at runtime for
CONFIG_SPARSEMEM_EXTREME=y")

we allocate the mem_section array dynamically in
sparse_memory_present_with_active_regions(),
but some architectures, like arm64, don't call the routine to
initialize sparsemem.

Let's move the initialization into memory_present() it should cover all
architectures.

Reported-and-tested-by: Sudeep Holla <sudeep.holla@arm.com>
Tested-by: Bjorn Andersson <bjorn.andersson@linaro.org>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-mm@kvack.org
Fixes: 83e3c48729d9 ("mm/sparsemem: Allocate mem_section at runtime
for CONFIG_SPARSEMEM_EXTREME=y")
Link: http://lkml.kernel.org/r/20171107083337.89952-1-kirill.shutemov@linux.intel.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 mm/page_alloc.c | 10 ----------
 mm/sparse.c     | 10 ++++++++++
 2 files changed, 10 insertions(+), 10 deletions(-)

(limited to 'mm')

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 8dfd13f..77e4d3c 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5646,16 +5646,6 @@ void __init
sparse_memory_present_with_active_regions(int nid)
  unsigned long start_pfn, end_pfn;
  int i, this_nid;

-#ifdef CONFIG_SPARSEMEM_EXTREME
- if (!mem_section) {
- unsigned long size, align;
-
- size = sizeof(struct mem_section) * NR_SECTION_ROOTS;
- align = 1 << (INTERNODE_CACHE_SHIFT);
- mem_section = memblock_virt_alloc(size, align);
- }
-#endif
-
  for_each_mem_pfn_range(i, nid, &start_pfn, &end_pfn, &this_nid)
  memory_present(this_nid, start_pfn, end_pfn);
 }
diff --git a/mm/sparse.c b/mm/sparse.c
index b00a973..d294148 100644
--- a/mm/sparse.c
+++ b/mm/sparse.c
@@ -206,6 +206,16 @@ void __init memory_present(int nid, unsigned long
start, unsigned long end)
 {
  unsigned long pfn;

+#ifdef CONFIG_SPARSEMEM_EXTREME
+ if (unlikely(!mem_section)) {
+ unsigned long size, align;
+
+ size = sizeof(struct mem_section) * NR_SECTION_ROOTS;
+ align = 1 << (INTERNODE_CACHE_SHIFT);
+ mem_section = memblock_virt_alloc(size, align);
+ }
+#endif
+
  start &= PAGE_SECTION_MASK;
  mminit_validate_memmodel_limits(&start, &end);
  for (pfn = start; pfn < end; pfn += PAGES_PER_SECTION) {
-- 
cgit v1.1

----
Reference link:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/patch/mm?id=d04fdafc066fed10cc9610b2f36f1d0ff0327864

- Naresh

>>
>> Any hint as to what the name of that commit is?  The tree is rebased a
>> lot, that commit is long gone :(
>
> After reverting the below commit the boot successful.
>
> commit 83e3c48729d9ebb7af5a31a504f3fd6aff0348c4
> mm/sparsemem: Allocate mem_section at runtime for CONFIG_SPARSEMEM_EXTREME=y
>
> From the history of linux-next,
> I have reported linux-next 20171102 boot failure on arm64 on November 2nd
> Bug 3404 - linux-next-20171102 doesn't boot on HiKey
> https://bugs.linaro.org/show_bug.cgi?id=3404
>
> one of the comments from bug says
> "This bug is been fixed by applying this patch."
> [PATCH] mm, sparse: Fix boot on arm64
>
> [start quote]
> Since 83e3c48729d9 ("mm/sparsemem: Allocate mem_section at runtime for
> CONFIG_SPARSEMEM_EXTREME=y") we allocate mem_section dynamically in
> sparse_memory_present_with_active_regions(). But some architectures, like
> arm64, don't use the routine to initialize sparsemem.
>
> Let's move the initialization into memory_present() it should cover all
> architectures.
>
> Signed-off-by: Kirill A. Shutemov <kirill.shute...@linux.intel.com>
> Fixes: 83e3c48729d9 ("mm/sparsemem: Allocate mem_section at runtime for
> CONFIG_SPARSEMEM_EXTREME=y")
> ---
>  mm/page_alloc.c | 10 ----------
>  mm/sparse.c     | 10 ++++++++++
>  2 files changed, 10 insertions(+), 10 deletions(-)
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 8dfd13f724d9..77e4d3c5c57b 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -5646,16 +5646,6 @@ void __init
> sparse_memory_present_with_active_regions(int nid)
>         unsigned long start_pfn, end_pfn;
>         int i, this_nid;
>
> -#ifdef CONFIG_SPARSEMEM_EXTREME
> -       if (!mem_section) {
> -               unsigned long size, align;
> -
> -               size = sizeof(struct mem_section) * NR_SECTION_ROOTS;
> -               align = 1 << (INTERNODE_CACHE_SHIFT);
> -               mem_section = memblock_virt_alloc(size, align);
> -       }
> -#endif
> -
>         for_each_mem_pfn_range(i, nid, &start_pfn, &end_pfn, &this_nid)
>                 memory_present(this_nid, start_pfn, end_pfn);
>  }
> diff --git a/mm/sparse.c b/mm/sparse.c
> index b00a97398795..d294148ba395 100644
> --- a/mm/sparse.c
> +++ b/mm/sparse.c
> @@ -206,6 +206,16 @@ void __init memory_present(int nid, unsigned long start,
> unsigned long end)
>  {
>         unsigned long pfn;
>
> +#ifdef CONFIG_SPARSEMEM_EXTREME
> +       if (unlikely(!mem_section)) {
> +               unsigned long size, align;
> +
> +               size = sizeof(struct mem_section) * NR_SECTION_ROOTS;
> +               align = 1 << (INTERNODE_CACHE_SHIFT);
> +               mem_section = memblock_virt_alloc(size, align);
> +       }
> +#endif
> +
>         start &= PAGE_SECTION_MASK;
>         mminit_validate_memmodel_limits(&start, &end);
>         for (pfn = start; pfn < end; pfn += PAGES_PER_SECTION) {
>
> [end quote]
>
> Please refer this link for more information,
> Re: [tip:x86/mm] mm/sparsemem: Allocate mem_section at runtime for
> CONFIG_SPARSEMEM_EXTREME=y
> https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1527427.html#nav
>
>>
>>> Linux version 4.14.9-rc1-00160-gb610c43 (naresh@linaro) (gcc version
>>> 4.9.2 20140904 (prerelease) (crosstool-NG linaro-1.13.1-4.9-2014.09 -
>>> Linaro GCC 4.9-
>>> 2014.09)) #1 SMP PREEMPT Fri Dec 22 14:21:46 IST 2017
>>>
>>> Ref:
> https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1527427.html#nav
> https://bugs.linaro.org/show_bug.cgi?id=3404
>>
>> What does this mean?
> It was additional information.
> Reference thread and bug links about this commit and bug investigation.
>
>>
>> confused,
>>
>> greg k-h

Re: pending 4.14-rc boot failures on arm64

[Greg Kroah-Hartman]

On Fri, Dec 22, 2017 at 06:09:00PM +0530, Naresh Kamboju wrote:
> On 22 December 2017 at 15:50, Naresh Kamboju <naresh.kamboju@linaro.org> wrote:
> > On 22 December 2017 at 14:36, Greg Kroah-Hartman
> > <gregkh@linuxfoundation.org> wrote:
> >> On Fri, Dec 22, 2017 at 02:29:55PM +0530, Naresh Kamboju wrote:
> >>> On 22 December 2017 at 12:41, Greg Kroah-Hartman
> >>> <gregkh@linuxfoundation.org> wrote:
> >>> > On Thu, Dec 21, 2017 at 04:35:30PM -0600, Dan Rue wrote:
> >>> >> Hi Greg,
> >>> >>
> >>> >> It looks like the pending v4.14.8-159-gc2a94d1a609 fails to boot on all
> >>> >> arm64 boards. hikey and juno are failing for us, and I see similar arm64
> >>> >> boot failures reported by kernelci.
> >>> >>
> >>> >> Still investigating root cause.
> >>> >
> >>> > Any help in bisecting to find the problem here would be great...
> >>>
> >>> Booting now on arm64 Hikey board. After reverting 83e3c48729d.
> >>>
> >>> git revert 83e3c48729d9ebb7af5a31a504f3fd6aff0348c4
> 
> Reverting above commit 83e3c48 (mm/sparsemem: Allocate mem_section at)
> might not be the right way to fix this problem.

No it isn't :)

> Instead cherry pick this commit 629a359bd to fix ARM64 boot crash problem.
> (mm/sparsemem: Fix ARM64 boot crash when CONFIG_SPARSEMEM_EXTREME=y)
> 
> tree: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
> branch: linux-4.14.y
> 
> git cherry-pick  629a359bdb0e0652a8227b4ff3125431995fec6e

I've taken this patch and added it to the queue, thanks.

greg k-h