ARM64 kernel Image size limitations?

ARM64 kernel Image size limitations?

[Florian Fainelli]

Hi,

I have been fighting with a reasonably large kernel Image: 46MB and
found unable to boot it. This is on v4.13-rc6, but I could go back as
far as v4.9~ish. It is so big, because it's got some debugging enabled,
and the initramfs compression is disabled, but still that ought to be
possible.

After some experimentation I found that the breaking size is somewhere
around 33MB total for arch/arm64/boot/Image and when that happens, I get
the following backtrace which suggests that there is some memory
corruption of some kind (or so it seems to me), other experiments
indicated that the backtrace may point back to where the bootloader had
set up its exception vector. I am fairly confident that the bootloader
is not responsible for corrupting the Image that is loaded.

Is there such a size limitation or am I possibly tripping over something
else?

(gdb) bt
#0  0xffffff800880721c in arch_spin_lock (lock=<optimized out>)
    at ./arch/arm64/include/asm/spinlock.h:89
#1  do_raw_spin_lock_flags (flags=<optimized out>, lock=<optimized out>)
    at ./include/linux/spinlock.h:155
#2  __raw_spin_lock_irqsave (lock=<optimized out>)
    at ./include/linux/spinlock_api_smp.h:119
#3  _raw_spin_lock_irqsave (lock=0xffffff800a1100f8)
    at kernel/locking/spinlock.c:159
#4  0xffffff80080888b4 in die (str=0xffffff800894a490 "Oops",
    regs=0xffffff800a013630, err=-1778384892) at
arch/arm64/kernel/traps.c:279
#5  0x80ffffff080952c4 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) info reg
x0             0xffffff800a1100f8       18446743524122624248
x1             0x1c0    448
x2             0x10000  65536
x3             0x1      1
x4             0x0      0
x5             0xffffff8008857dc8       18446743524096703944
x6             0xffffff800a00494f       18446743524121528655
x7             0x6572646461206c65       7310015527464758373
x8             0x65676e6172207373       7306930285237531507
x9             0x0      0
x10            0xffffff800a0134e0       18446743524121588960
x11            0xffffff800a0134e0       18446743524121588960
x12            0x40d00  265472
x13            0x72422820352e382e       8233187187582973998
x14            0x73206d6f6364616f       8295750838792773999
x15            0x0      0
x16            0x0      0
x17            0x0      0
x18            0x0      0
x19            0xffffff800a110000       18446743524122624000
x20            0x96000004       2516582404
x21            0xffffff800894a490       18446743524097696912
x22            0xffffff800a013630       18446743524121589296
x23            0xffffff800a013630       18446743524121589296
x24            0x25     37
x25            0xffffff800a018000       18446743524121608192
x26            0xffffff8008955622       18446743524097742370
x27            0xffffff8008955622       18446743524097742370
x28            0xffffff800a01dd80       18446743524121632128
x29            0xffffff800a0134d0       18446743524121588944
x30            0xffffff80080888b4       18446743524088514740
sp             0xffffff800a0134d0       0xffffff800a0134d0
pc             0xffffff800880721c       0xffffff800880721c
<_raw_spin_lock_irqsave+44>
CPSR           0x600001c5       1610613189
(gdb)

-- 
Florian

ARM64 kernel Image size limitations?

[Ard Biesheuvel]

Hi,

On 28 August 2017 at 20:55, Florian Fainelli <f.fainelli@gmail.com> wrote:
> Hi,
>
> I have been fighting with a reasonably large kernel Image: 46MB and
> found unable to boot it. This is on v4.13-rc6, but I could go back as
> far as v4.9~ish. It is so big, because it's got some debugging enabled,
> and the initramfs compression is disabled, but still that ought to be
> possible.
>
> After some experimentation I found that the breaking size is somewhere
> around 33MB total for arch/arm64/boot/Image and when that happens, I get
> the following backtrace which suggests that there is some memory
> corruption of some kind (or so it seems to me), other experiments
> indicated that the backtrace may point back to where the bootloader had
> set up its exception vector. I am fairly confident that the bootloader
> is not responsible for corrupting the Image that is loaded.
>
> Is there such a size limitation or am I possibly tripping over something
> else?
>

What boot environment are you using? Is it possible the DTB is copied
on top of the kernel by the boot loader?

ARM64 kernel Image size limitations?

[Florian Fainelli]

(fixed Will's address, apologies)

On 08/28/2017 01:01 PM, Ard Biesheuvel wrote:
> Hi,
> 
> On 28 August 2017 at 20:55, Florian Fainelli <f.fainelli@gmail.com> wrote:
>> Hi,
>>
>> I have been fighting with a reasonably large kernel Image: 46MB and
>> found unable to boot it. This is on v4.13-rc6, but I could go back as
>> far as v4.9~ish. It is so big, because it's got some debugging enabled,
>> and the initramfs compression is disabled, but still that ought to be
>> possible.
>>
>> After some experimentation I found that the breaking size is somewhere
>> around 33MB total for arch/arm64/boot/Image and when that happens, I get
>> the following backtrace which suggests that there is some memory
>> corruption of some kind (or so it seems to me), other experiments
>> indicated that the backtrace may point back to where the bootloader had
>> set up its exception vector. I am fairly confident that the bootloader
>> is not responsible for corrupting the Image that is loaded.
>>
>> Is there such a size limitation or am I possibly tripping over something
>> else?
>>
> 
> What boot environment are you using? Is it possible the DTB is copied
> on top of the kernel by the boot loader?

Bootloader is BOLT (Broadcom's own implementation, has PSCI and is
capable of doing FDT live patching), and yes the DTB is provided by the
bootloader at an address higher than the kernel.

Physical offset of the RAM on this platform is 0, so the load is going
to be at PA 0x80000, 0x80000 + size (35MB) = 0x2280000, DTB is placed at
0x771f000 and is 0x90ef bytes.


-- 
Florian

ARM64 kernel Image size limitations?

[Ard Biesheuvel]

On 28 August 2017 at 22:24, Florian Fainelli <f.fainelli@gmail.com> wrote:
> (fixed Will's address, apologies)
>
> On 08/28/2017 01:01 PM, Ard Biesheuvel wrote:
>> Hi,
>>
>> On 28 August 2017 at 20:55, Florian Fainelli <f.fainelli@gmail.com> wrote:
>>> Hi,
>>>
>>> I have been fighting with a reasonably large kernel Image: 46MB and
>>> found unable to boot it. This is on v4.13-rc6, but I could go back as
>>> far as v4.9~ish. It is so big, because it's got some debugging enabled,
>>> and the initramfs compression is disabled, but still that ought to be
>>> possible.
>>>
>>> After some experimentation I found that the breaking size is somewhere
>>> around 33MB total for arch/arm64/boot/Image and when that happens, I get
>>> the following backtrace which suggests that there is some memory
>>> corruption of some kind (or so it seems to me), other experiments
>>> indicated that the backtrace may point back to where the bootloader had
>>> set up its exception vector. I am fairly confident that the bootloader
>>> is not responsible for corrupting the Image that is loaded.
>>>
>>> Is there such a size limitation or am I possibly tripping over something
>>> else?
>>>
>>
>> What boot environment are you using? Is it possible the DTB is copied
>> on top of the kernel by the boot loader?
>
> Bootloader is BOLT (Broadcom's own implementation, has PSCI and is
> capable of doing FDT live patching), and yes the DTB is provided by the
> bootloader at an address higher than the kernel.
>
> Physical offset of the RAM on this platform is 0, so the load is going
> to be at PA 0x80000, 0x80000 + size (35MB) = 0x2280000, DTB is placed at
> 0x771f000 and is 0x90ef bytes.
>

Could you try to dump __log_buf from gdb?

ARM64 kernel Image size limitations?

[Florian Fainelli]

On 08/29/2017 01:15 AM, Ard Biesheuvel wrote:
> On 28 August 2017 at 22:24, Florian Fainelli <f.fainelli@gmail.com> wrote:
>> (fixed Will's address, apologies)
>>
>> On 08/28/2017 01:01 PM, Ard Biesheuvel wrote:
>>> Hi,
>>>
>>> On 28 August 2017 at 20:55, Florian Fainelli <f.fainelli@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> I have been fighting with a reasonably large kernel Image: 46MB and
>>>> found unable to boot it. This is on v4.13-rc6, but I could go back as
>>>> far as v4.9~ish. It is so big, because it's got some debugging enabled,
>>>> and the initramfs compression is disabled, but still that ought to be
>>>> possible.
>>>>
>>>> After some experimentation I found that the breaking size is somewhere
>>>> around 33MB total for arch/arm64/boot/Image and when that happens, I get
>>>> the following backtrace which suggests that there is some memory
>>>> corruption of some kind (or so it seems to me), other experiments
>>>> indicated that the backtrace may point back to where the bootloader had
>>>> set up its exception vector. I am fairly confident that the bootloader
>>>> is not responsible for corrupting the Image that is loaded.
>>>>
>>>> Is there such a size limitation or am I possibly tripping over something
>>>> else?
>>>>
>>>
>>> What boot environment are you using? Is it possible the DTB is copied
>>> on top of the kernel by the boot loader?
>>
>> Bootloader is BOLT (Broadcom's own implementation, has PSCI and is
>> capable of doing FDT live patching), and yes the DTB is provided by the
>> bootloader at an address higher than the kernel.
>>
>> Physical offset of the RAM on this platform is 0, so the load is going
>> to be at PA 0x80000, 0x80000 + size (35MB) = 0x2280000, DTB is placed at
>> 0x771f000 and is 0x90ef bytes.
>>
> 
> Could you try to dump __log_buf from gdb?

Silly me, this is a bootloader limitation, unlike what I thought the
load address is not at PA 0, it is at PA 0x06FFC000 and there is only
~34MB worth of space available, mayhem ensues.

Thanks!
-- 
Florian

ARM64 kernel Image size limitations?

[Florian Fainelli]

On 08/29/2017 09:51 AM, Florian Fainelli wrote:
> On 08/29/2017 01:15 AM, Ard Biesheuvel wrote:
>> On 28 August 2017 at 22:24, Florian Fainelli <f.fainelli@gmail.com> wrote:
>>> (fixed Will's address, apologies)
>>>
>>> On 08/28/2017 01:01 PM, Ard Biesheuvel wrote:
>>>> Hi,
>>>>
>>>> On 28 August 2017 at 20:55, Florian Fainelli <f.fainelli@gmail.com> wrote:
>>>>> Hi,
>>>>>
>>>>> I have been fighting with a reasonably large kernel Image: 46MB and
>>>>> found unable to boot it. This is on v4.13-rc6, but I could go back as
>>>>> far as v4.9~ish. It is so big, because it's got some debugging enabled,
>>>>> and the initramfs compression is disabled, but still that ought to be
>>>>> possible.
>>>>>
>>>>> After some experimentation I found that the breaking size is somewhere
>>>>> around 33MB total for arch/arm64/boot/Image and when that happens, I get
>>>>> the following backtrace which suggests that there is some memory
>>>>> corruption of some kind (or so it seems to me), other experiments
>>>>> indicated that the backtrace may point back to where the bootloader had
>>>>> set up its exception vector. I am fairly confident that the bootloader
>>>>> is not responsible for corrupting the Image that is loaded.
>>>>>
>>>>> Is there such a size limitation or am I possibly tripping over something
>>>>> else?
>>>>>
>>>>
>>>> What boot environment are you using? Is it possible the DTB is copied
>>>> on top of the kernel by the boot loader?
>>>
>>> Bootloader is BOLT (Broadcom's own implementation, has PSCI and is
>>> capable of doing FDT live patching), and yes the DTB is provided by the
>>> bootloader at an address higher than the kernel.
>>>
>>> Physical offset of the RAM on this platform is 0, so the load is going
>>> to be at PA 0x80000, 0x80000 + size (35MB) = 0x2280000, DTB is placed at
>>> 0x771f000 and is 0x90ef bytes.
>>>
>>
>> Could you try to dump __log_buf from gdb?
> 
> Silly me, this is a bootloader limitation, unlike what I thought the
> load address is not at PA 0, it is at PA 0x06FFC000 and there is only
> ~34MB worth of space available, mayhem ensues.

Most definitively -ENOCOFFEE it is loaded at PA 0x80000 as I expect it
to, I will try to get you a log after drinking some coffee ;)
-- 
Florian

ARM64 kernel Image size limitations?

[Mark Rutland]

On Mon, Aug 28, 2017 at 12:55:23PM -0700, Florian Fainelli wrote:
> Hi,
> 
> I have been fighting with a reasonably large kernel Image: 46MB and
> found unable to boot it. This is on v4.13-rc6, but I could go back as
> far as v4.9~ish. It is so big, because it's got some debugging enabled,
> and the initramfs compression is disabled, but still that ought to be
> possible.

I regularly boot 90+ MB kernels, so this is certainly possible, in
regular-ish configurations.

> After some experimentation I found that the breaking size is somewhere
> around 33MB total for arch/arm64/boot/Image and when that happens, I get
> the following backtrace which suggests that there is some memory
> corruption of some kind (or so it seems to me), other experiments
> indicated that the backtrace may point back to where the bootloader had
> set up its exception vector. I am fairly confident that the bootloader
> is not responsible for corrupting the Image that is loaded.
> 
> Is there such a size limitation or am I possibly tripping over something
> else?

Are you using 16K pages, by any chance?

With 16K pages, the early page tables can only map up to 32M, and thus
cannot map a kernel image larger than this.

When those go wrong, you'd see what looks like memory corruption, or
unmapped memory.

With 4K page, we can map up to 1G, and with 64K we can map up to 512M.

Thanks,
Mark.

> 
> (gdb) bt
> #0  0xffffff800880721c in arch_spin_lock (lock=<optimized out>)
>     at ./arch/arm64/include/asm/spinlock.h:89
> #1  do_raw_spin_lock_flags (flags=<optimized out>, lock=<optimized out>)
>     at ./include/linux/spinlock.h:155
> #2  __raw_spin_lock_irqsave (lock=<optimized out>)
>     at ./include/linux/spinlock_api_smp.h:119
> #3  _raw_spin_lock_irqsave (lock=0xffffff800a1100f8)
>     at kernel/locking/spinlock.c:159
> #4  0xffffff80080888b4 in die (str=0xffffff800894a490 "Oops",
>     regs=0xffffff800a013630, err=-1778384892) at
> arch/arm64/kernel/traps.c:279
> #5  0x80ffffff080952c4 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> (gdb) info reg
> x0             0xffffff800a1100f8       18446743524122624248
> x1             0x1c0    448
> x2             0x10000  65536
> x3             0x1      1
> x4             0x0      0
> x5             0xffffff8008857dc8       18446743524096703944
> x6             0xffffff800a00494f       18446743524121528655
> x7             0x6572646461206c65       7310015527464758373
> x8             0x65676e6172207373       7306930285237531507
> x9             0x0      0
> x10            0xffffff800a0134e0       18446743524121588960
> x11            0xffffff800a0134e0       18446743524121588960
> x12            0x40d00  265472
> x13            0x72422820352e382e       8233187187582973998
> x14            0x73206d6f6364616f       8295750838792773999
> x15            0x0      0
> x16            0x0      0
> x17            0x0      0
> x18            0x0      0
> x19            0xffffff800a110000       18446743524122624000
> x20            0x96000004       2516582404
> x21            0xffffff800894a490       18446743524097696912
> x22            0xffffff800a013630       18446743524121589296
> x23            0xffffff800a013630       18446743524121589296
> x24            0x25     37
> x25            0xffffff800a018000       18446743524121608192
> x26            0xffffff8008955622       18446743524097742370
> x27            0xffffff8008955622       18446743524097742370
> x28            0xffffff800a01dd80       18446743524121632128
> x29            0xffffff800a0134d0       18446743524121588944
> x30            0xffffff80080888b4       18446743524088514740
> sp             0xffffff800a0134d0       0xffffff800a0134d0
> pc             0xffffff800880721c       0xffffff800880721c
> <_raw_spin_lock_irqsave+44>
> CPSR           0x600001c5       1610613189
> (gdb)
> 
> -- 
> Florian
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel