regression: insmod module failed in VM with nvdimm on

regression: insmod module failed in VM with nvdimm on

[chenxiang (M)]

Hi,

We boot the VM using following commands (with nvdimm on)  (qemu version 
6.1.50, kernel 6.0-r4):

qemu-system-aarch64 -machine 
virt,kernel_irqchip=on,gic-version=3,nvdimm=on  -kernel 
/home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios 
/root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m 
2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0 
ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1' 
-object memory-backend-ram,id=ram1,size=10G -device 
nvdimm,id=dimm1,memdev=ram1  -device ioh3420,id=root_port1,chassis=1 
-device vfio-pci,host=7d:01.0,id=net0,bus=root_port1

Then in VM we insmod a module, vmalloc error occurs as follows (kernel 
5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4):

estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko
[    8.186563] vmap allocation for size 20480 failed: use vmalloc=<size> 
to increase size
[    8.187288] insmod: vmalloc error: size 16384, vm_struct allocation 
failed, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
[    8.188402] CPU: 1 PID: 235 Comm: insmod Not tainted 6.0.0-rc4+ #1
[    8.188958] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 
02/06/2015
[    8.189593] Call trace:
[    8.189825]  dump_backtrace.part.0+0xc4/0xd0
[    8.190245]  show_stack+0x24/0x40
[    8.190563]  dump_stack_lvl+0x68/0x84
[    8.190913]  dump_stack+0x18/0x34
[    8.191223]  warn_alloc+0x124/0x1b0
[    8.191555]  __vmalloc_node_range+0xe4/0x55c
[    8.191959]  module_alloc+0xf8/0x104
[    8.192305]  load_module+0x854/0x1e70
[    8.192655]  __do_sys_init_module+0x1e0/0x220
[    8.193075]  __arm64_sys_init_module+0x28/0x34
[    8.193489]  invoke_syscall+0x50/0x120
[    8.193841]  el0_svc_common.constprop.0+0x58/0x1a0
[    8.194296]  do_el0_svc+0x38/0xd0
[    8.194613]  el0_svc+0x2c/0xc0
[    8.194901]  el0t_64_sync_handler+0x1ac/0x1b0
[    8.195313]  el0t_64_sync+0x19c/0x1a0
[    8.195672] Mem-Info:
[    8.195872] active_anon:17641 inactive_anon:118549 isolated_anon:0
[    8.195872]  active_file:0 inactive_file:0 isolated_file:0
[    8.195872]  unevictable:0 dirty:0 writeback:0
[    8.195872]  slab_reclaimable:3439 slab_unreclaimable:3067
[    8.195872]  mapped:877 shmem:135976 pagetables:39 bounce:0
[    8.195872]  kernel_misc_reclaimable:0
[    8.195872]  free:353735 free_pcp:3210 free_cma:0
[    8.199119] Node 0 active_anon:70564kB inactive_anon:474196kB 
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB 
isolated(file):0kB mapped:3508kB dirty:0kB writeback:0kB shmem:543904kB 
shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB 
kernel_stack:1904kB pagetables:156kB all_unreclaimable? no
[    8.201683] Node 0 DMA free:1414940kB boost:0kB min:22528kB 
low:28160kB high:33792kB reserved_highatomic:0KB active_anon:70564kB 
inactive_anon:474196kB active_file:0kB inactive_file:0kB unevictable:0kB 
writepending:0kB present:2097152kB managed:2010444kB mlocked:0kB 
bounce:0kB free_pcp:12840kB local_pcp:2032kB free_cma:0kB
[    8.204158] lowmem_reserve[]: 0 0 0 0
[    8.204481] Node 0 DMA: 1*4kB (E) 1*8kB (U) 1*16kB (U) 2*32kB (UM) 
1*64kB (U) 1*128kB (U) 2*256kB (ME) 2*512kB (ME) 2*1024kB (M) 3*2048kB 
(UM) 343*4096kB (M) = 1414940kB
[    8.205881] Node 0 hugepages_total=0 hugepages_free=0 
hugepages_surp=0 hugepages_size=1048576kB
[    8.206644] Node 0 hugepages_total=0 hugepages_free=0 
hugepages_surp=0 hugepages_size=32768kB
[    8.207381] Node 0 hugepages_total=0 hugepages_free=0 
hugepages_surp=0 hugepages_size=2048kB
[    8.208111] Node 0 hugepages_total=0 hugepages_free=0 
hugepages_surp=0 hugepages_size=64kB
[    8.208826] 135976 total pagecache pages
[    8.209195] 0 pages in swap cache
[    8.209484] Free swap  = 0kB
[    8.209733] Total swap = 0kB
[    8.209989] 524288 pages RAM
[    8.210239] 0 pages HighMem/MovableOnly
[    8.210571] 21677 pages reserved
[    8.210852] 0 pages hwpoisoned
insmod: can't insert '/lib/modules/6.0.0-rc4+/hnae3.ko': Cannot allocate 
memory

We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr: 
defer initialization to initcall where permitted").

Do you have any idea about the issue?


Best Regards,

Xiang Chen


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: regression: insmod module failed in VM with nvdimm on

[Marc Zyngier]

On Wed, 30 Nov 2022 02:52:35 +0000,
"chenxiang (M)" <chenxiang66@hisilicon.com> wrote:
> 
> Hi,
> 
> We boot the VM using following commands (with nvdimm on)  (qemu
> version 6.1.50, kernel 6.0-r4):

How relevant is the presence of the nvdimm? Do you observe the failure
without this?

> 
> qemu-system-aarch64 -machine
> virt,kernel_irqchip=on,gic-version=3,nvdimm=on  -kernel
> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios
> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m
> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0
> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1'
> -object memory-backend-ram,id=ram1,size=10G -device
> nvdimm,id=dimm1,memdev=ram1  -device ioh3420,id=root_port1,chassis=1
> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1
> 
> Then in VM we insmod a module, vmalloc error occurs as follows (kernel
> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4):
> 
> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko
> [    8.186563] vmap allocation for size 20480 failed: use
> vmalloc=<size> to increase size

Have you tried increasing the vmalloc size to check that this is
indeed the problem?

[...]

> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr:
> defer initialization to initcall where permitted").

I guess you mean commit fc5a89f75d2a instead, right?

> Do you have any idea about the issue?

I sort of suspect that the nvdimm gets vmap-ed and consumes a large
portion of the vmalloc space, but you give very little information
that could help here...

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: regression: insmod module failed in VM with nvdimm on

[Ard Biesheuvel]

On Wed, 30 Nov 2022 at 08:53, Marc Zyngier <maz@kernel.org> wrote:
>
> On Wed, 30 Nov 2022 02:52:35 +0000,
> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote:
> >
> > Hi,
> >
> > We boot the VM using following commands (with nvdimm on)  (qemu
> > version 6.1.50, kernel 6.0-r4):
>
> How relevant is the presence of the nvdimm? Do you observe the failure
> without this?
>
> >
> > qemu-system-aarch64 -machine
> > virt,kernel_irqchip=on,gic-version=3,nvdimm=on  -kernel
> > /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios
> > /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m
> > 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0
> > ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1'
> > -object memory-backend-ram,id=ram1,size=10G -device
> > nvdimm,id=dimm1,memdev=ram1  -device ioh3420,id=root_port1,chassis=1
> > -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1
> >
> > Then in VM we insmod a module, vmalloc error occurs as follows (kernel
> > 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4):
> >
> > estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko
> > [    8.186563] vmap allocation for size 20480 failed: use
> > vmalloc=<size> to increase size
>
> Have you tried increasing the vmalloc size to check that this is
> indeed the problem?
>
> [...]
>
> > We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr:
> > defer initialization to initcall where permitted").
>
> I guess you mean commit fc5a89f75d2a instead, right?
>
> > Do you have any idea about the issue?
>
> I sort of suspect that the nvdimm gets vmap-ed and consumes a large
> portion of the vmalloc space, but you give very little information
> that could help here...
>

Ouch. I suspect what's going on here: that patch defers the
randomization of the module region, so that we can decouple it from
the very early init code.

Obviously, it is happening too late now, and the randomized module
region is overlapping with a vmalloc region that is in use by the time
the randomization occurs.

Does the below fix the issue?

diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c
index 37a9deed2aec..71fb18b2f304 100644
--- a/arch/arm64/kernel/kaslr.c
+++ b/arch/arm64/kernel/kaslr.c
@@ -90,4 +90,4 @@ static int __init kaslr_init(void)

        return 0;
 }
-subsys_initcall(kaslr_init)
+arch_initcall(kaslr_init)

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: regression: insmod module failed in VM with nvdimm on #forregzbot

[Thorsten Leemhuis]

[Note: this mail is primarily send for documentation purposes and/or for
regzbot, my Linux kernel regression tracking bot. That's why I removed
most or all folks from the list of recipients, but left any that looked
like a mailing lists. These mails usually contain '#forregzbot' in the
subject, to make them easy to spot and filter out.]

[TLDR: I'm adding this regression report to the list of tracked
regressions; all text from me you find below is based on a few templates
paragraphs you might have encountered already already in similar form.]

Hi, this is your Linux kernel regression tracker.

On 30.11.22 03:52, chenxiang (M) wrote:
>
> We boot the VM using following commands (with nvdimm on)  (qemu version
> 6.1.50, kernel 6.0-r4):
> 
> qemu-system-aarch64 -machine
> virt,kernel_irqchip=on,gic-version=3,nvdimm=on  -kernel
> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios
> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m
> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0
> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1'
> -object memory-backend-ram,id=ram1,size=10G -device
> nvdimm,id=dimm1,memdev=ram1  -device ioh3420,id=root_port1,chassis=1
> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1
> [...]
> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr:
> defer initialization to initcall where permitted").

Thanks for the report. To be sure below issue doesn't fall through the
cracks unnoticed, I'm adding it to regzbot, my Linux kernel regression
tracking bot:

#regzbot ^introduced fc5a89f75d2a
#regzbot title arm64: kaslr: vmalloc error when boothing a VM with nvdimm
#regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already
discussed somewhere else? It was fixed already? You want to clarify when
the regression started to happen? Or point out I got the title or
something else totally wrong? Then just reply -- ideally with also
telling regzbot about it, as explained here:
https://linux-regtracking.leemhuis.info/tracked-regression/

Reminder for developers: When fixing the issue, add 'Link:' tags
pointing to the report (the mail this one replies to), as explained for
in the Linux kernel's documentation; above webpage explains why this is
important for tracked regressions.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I deal with a lot of
reports and sometimes miss something important when writing mails like
this. If that's the case here, don't hesitate to tell me in a public
reply, it's in everyone's interest to set the public record straight.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: regression: insmod module failed in VM with nvdimm on

[chenxiang (M)]

Hi Marc,


在 2022/11/30 15:53, Marc Zyngier 写道:
> On Wed, 30 Nov 2022 02:52:35 +0000,
> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote:
>> Hi,
>>
>> We boot the VM using following commands (with nvdimm on)  (qemu
>> version 6.1.50, kernel 6.0-r4):
> How relevant is the presence of the nvdimm? Do you observe the failure
> without this?

We didn't see the failure without it.

>> qemu-system-aarch64 -machine
>> virt,kernel_irqchip=on,gic-version=3,nvdimm=on  -kernel
>> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios
>> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m
>> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0
>> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1'
>> -object memory-backend-ram,id=ram1,size=10G -device
>> nvdimm,id=dimm1,memdev=ram1  -device ioh3420,id=root_port1,chassis=1
>> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1
>>
>> Then in VM we insmod a module, vmalloc error occurs as follows (kernel
>> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4):
>>
>> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko
>> [    8.186563] vmap allocation for size 20480 failed: use
>> vmalloc=<size> to increase size
> Have you tried increasing the vmalloc size to check that this is
> indeed the problem?
>
> [...]

I didn't increase the vmalloc size, but i check the vmall size and i 
think it is big enough when the issue occurs:

estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko
[    4.879899] vmap allocation for size 20480 failed: use vmalloc=<size> 
to increase size
[    4.880643] insmod: vmalloc error: size 16384, vm_struct allocation 
failed, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
[    4.881802] CPU: 1 PID: 230 Comm: insmod Not tainted 6.1.0-rc4+ #21
[    4.882414] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 
02/06/2015
[    4.883082] Call trace:
[    4.883333]  dump_backtrace.part.0+0xc4/0xd0
[    4.883766]  show_stack+0x20/0x50
[    4.884091]  dump_stack_lvl+0x68/0x84
[    4.884450]  dump_stack+0x18/0x34
[    4.884778]  warn_alloc+0x11c/0x1bc
[    4.885124]  __vmalloc_node_range+0x50c/0x64c
[    4.885553]  module_alloc+0xf4/0x100
[    4.885902]  load_module+0x858/0x1e90
[    4.886265]  __do_sys_init_module+0x1c0/0x200
[    4.886699]  __arm64_sys_init_module+0x24/0x30
[    4.887147]  invoke_syscall+0x50/0x120
[    4.887516]  el0_svc_common.constprop.0+0x58/0x190
[    4.887993]  do_el0_svc+0x34/0xc0
[    4.888327]  el0_svc+0x2c/0xb4
[    4.888631]  el0t_64_sync_handler+0xb8/0xbc
[    4.889046]  el0t_64_sync+0x19c/0x1a0
[    4.889423] Mem-Info:
[    4.889639] active_anon:9679 inactive_anon:63094 isolated_anon:0
[    4.889639]  active_file:0 inactive_file:0 isolated_file:0
[    4.889639]  unevictable:0 dirty:0 writeback:0
[    4.889639]  slab_reclaimable:3322 slab_unreclaimable:3082
[    4.889639]  mapped:873 shmem:72569 pagetables:34
[    4.889639]  sec_pagetables:0 bounce:0
[    4.889639]  kernel_misc_reclaimable:0
[    4.889639]  free:416212 free_pcp:4414 free_cma:0
[    4.893362] Node 0 active_anon:38716kB inactive_anon:252376kB 
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB 
isolated(file):0kB mapped:3492kB dirty:0kB writeback:0kB shmem:290276kB 
shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB 
kernel_stack:1904kB pagetables:136kB sec_pagetables:0kB 
all_unreclaimable? no
[    4.896343] Node 0 DMA free:1664848kB boost:0kB min:22528kB 
low:28160kB high:33792kB reserved_highatomic:0KB active_anon:38716kB 
inactive_anon:252376kB active_file:0kB inactive_file:0kB unevictable:0kB 
writepending:0kB present:2097152kB managed:2010376kB mlocked:0kB 
bounce:0kB free_pcp:17704kB local_pcp:3668kB free_cma:0kB
[    4.899097] lowmem_reserve[]: 0 0 0 0 0
[    4.899466] Node 0 DMA: 2*4kB (UM) 1*8kB (M) 2*16kB (UM) 1*32kB (M) 
2*64kB (ME) 1*128kB (U) 2*256kB (ME) 2*512kB (M) 6*1024kB (UME) 5*2048kB 
(UM) 402*4096kB (M) = 1664848kB
[    4.900865] Node 0 hugepages_total=0 hugepages_free=0 
hugepages_surp=0 hugepages_size=1048576kB
[    4.901648] Node 0 hugepages_total=0 hugepages_free=0 
hugepages_surp=0 hugepages_size=32768kB
[    4.902526] Node 0 hugepages_total=0 hugepages_free=0 
hugepages_surp=0 hugepages_size=2048kB
[    4.903354] Node 0 hugepages_total=0 hugepages_free=0 
hugepages_surp=0 hugepages_size=64kB
[    4.904173] 72569 total pagecache pages
[    4.904524] 0 pages in swap cache
[    4.904831] Free swap  = 0kB
[    4.905109] Total swap = 0kB
[    4.905407] 524288 pages RAM
[    4.905696] 0 pages HighMem/MovableOnly
[    4.906085] 21694 pages reserved
[    4.906388] 0 pages hwpoisoned
insmod: can't insert '/lib/modules/6.1.0-rc4+/hnae3.ko': Cannot allocate 
memory
estuary:/$ insmod /lib/modules/$(uname -r)/hns3.ko
[    4.911599] vmap allocation for size 122880 failed: use 
vmalloc=<size> to increase size
insmod: can't insert '/lib/modules/6.1.0-rc4+/hns3.ko': Cannot allocate 
memory
estuary:/$ insmod /lib/modules/$(uname -r)/hclge.ko
[    4.917761] vmap allocation for size 319488 failed: use 
vmalloc=<size> to increase size
insmod: can't insert '/lib/modules/6.1.0-rc4+/hclge.ko': Cannot allocate 
memory
estuary:/$ insmod /lib/modules/$(uname -r)/hclgevf.ko
[    5.160299] vmap allocation for size 73728 failed: use vmalloc=<size> 
to increase size
insmod: can't insert '/lib/modules/6.1.0-rc4+/hclgevf.ko': Cannot 
allocate memory
estuary:/$
estuary:/$ cat /proc/meminfo
MemTotal:        2010376 kB
MemFree:         1664848 kB
MemAvailable:    1637744 kB
Buffers:               0 kB
Cached:           290276 kB
SwapCached:            0 kB
Active:            39456 kB
Inactive:         251628 kB
Active(anon):      39456 kB
Inactive(anon):   251628 kB
Active(file):          0 kB
Inactive(file):        0 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:           860 kB
Mapped:             3368 kB
Shmem:            290276 kB
KReclaimable:      13372 kB
Slab:              25732 kB
SReclaimable:      13372 kB
SUnreclaim:        12360 kB
KernelStack:        1872 kB
PageTables:          136 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1005188 kB
Committed_AS:     292076 kB
VmallocTotal:   133143592960 kB
VmallocUsed:        2612 kB
VmallocChunk:          0 kB
Percpu:              672 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
estuary:/$

>
>> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr:
>> defer initialization to initcall where permitted").
> I guess you mean commit fc5a89f75d2a instead, right?

Right

>
>> Do you have any idea about the issue?
> I sort of suspect that the nvdimm gets vmap-ed and consumes a large
> portion of the vmalloc space, but you give very little information
> that could help here...
>
> 	M.
>


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: regression: insmod module failed in VM with nvdimm on

[chenxiang (M)]

Hi Ard,


在 2022/11/30 16:18, Ard Biesheuvel 写道:
> On Wed, 30 Nov 2022 at 08:53, Marc Zyngier <maz@kernel.org> wrote:
>> On Wed, 30 Nov 2022 02:52:35 +0000,
>> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote:
>>> Hi,
>>>
>>> We boot the VM using following commands (with nvdimm on)  (qemu
>>> version 6.1.50, kernel 6.0-r4):
>> How relevant is the presence of the nvdimm? Do you observe the failure
>> without this?
>>
>>> qemu-system-aarch64 -machine
>>> virt,kernel_irqchip=on,gic-version=3,nvdimm=on  -kernel
>>> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios
>>> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m
>>> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0
>>> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1'
>>> -object memory-backend-ram,id=ram1,size=10G -device
>>> nvdimm,id=dimm1,memdev=ram1  -device ioh3420,id=root_port1,chassis=1
>>> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1
>>>
>>> Then in VM we insmod a module, vmalloc error occurs as follows (kernel
>>> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4):
>>>
>>> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko
>>> [    8.186563] vmap allocation for size 20480 failed: use
>>> vmalloc=<size> to increase size
>> Have you tried increasing the vmalloc size to check that this is
>> indeed the problem?
>>
>> [...]
>>
>>> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr:
>>> defer initialization to initcall where permitted").
>> I guess you mean commit fc5a89f75d2a instead, right?
>>
>>> Do you have any idea about the issue?
>> I sort of suspect that the nvdimm gets vmap-ed and consumes a large
>> portion of the vmalloc space, but you give very little information
>> that could help here...
>>
> Ouch. I suspect what's going on here: that patch defers the
> randomization of the module region, so that we can decouple it from
> the very early init code.
>
> Obviously, it is happening too late now, and the randomized module
> region is overlapping with a vmalloc region that is in use by the time
> the randomization occurs.
>
> Does the below fix the issue?

The issue still occurs, but it seems decrease the probability, before it 
occured almost every time, after the change, i tried 2-3 times, and it 
occurs.
But i change back "subsys_initcall" to "core_initcall", and i test more 
than 20 times, and it is still ok.

>
> diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c
> index 37a9deed2aec..71fb18b2f304 100644
> --- a/arch/arm64/kernel/kaslr.c
> +++ b/arch/arm64/kernel/kaslr.c
> @@ -90,4 +90,4 @@ static int __init kaslr_init(void)
>
>          return 0;
>   }
> -subsys_initcall(kaslr_init)
> +arch_initcall(kaslr_init)
> .
>


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: regression: insmod module failed in VM with nvdimm on

[Ard Biesheuvel]

On Thu, 1 Dec 2022 at 08:15, chenxiang (M) <chenxiang66@hisilicon.com> wrote:
>
> Hi Ard,
>
>
> 在 2022/11/30 16:18, Ard Biesheuvel 写道:
> > On Wed, 30 Nov 2022 at 08:53, Marc Zyngier <maz@kernel.org> wrote:
> >> On Wed, 30 Nov 2022 02:52:35 +0000,
> >> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote:
> >>> Hi,
> >>>
> >>> We boot the VM using following commands (with nvdimm on)  (qemu
> >>> version 6.1.50, kernel 6.0-r4):
> >> How relevant is the presence of the nvdimm? Do you observe the failure
> >> without this?
> >>
> >>> qemu-system-aarch64 -machine
> >>> virt,kernel_irqchip=on,gic-version=3,nvdimm=on  -kernel
> >>> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios
> >>> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m
> >>> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0
> >>> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1'
> >>> -object memory-backend-ram,id=ram1,size=10G -device
> >>> nvdimm,id=dimm1,memdev=ram1  -device ioh3420,id=root_port1,chassis=1
> >>> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1
> >>>
> >>> Then in VM we insmod a module, vmalloc error occurs as follows (kernel
> >>> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4):
> >>>
> >>> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko
> >>> [    8.186563] vmap allocation for size 20480 failed: use
> >>> vmalloc=<size> to increase size
> >> Have you tried increasing the vmalloc size to check that this is
> >> indeed the problem?
> >>
> >> [...]
> >>
> >>> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr:
> >>> defer initialization to initcall where permitted").
> >> I guess you mean commit fc5a89f75d2a instead, right?
> >>
> >>> Do you have any idea about the issue?
> >> I sort of suspect that the nvdimm gets vmap-ed and consumes a large
> >> portion of the vmalloc space, but you give very little information
> >> that could help here...
> >>
> > Ouch. I suspect what's going on here: that patch defers the
> > randomization of the module region, so that we can decouple it from
> > the very early init code.
> >
> > Obviously, it is happening too late now, and the randomized module
> > region is overlapping with a vmalloc region that is in use by the time
> > the randomization occurs.
> >
> > Does the below fix the issue?
>
> The issue still occurs, but it seems decrease the probability, before it
> occured almost every time, after the change, i tried 2-3 times, and it
> occurs.
> But i change back "subsys_initcall" to "core_initcall", and i test more
> than 20 times, and it is still ok.
>

Thank you for confirming. I will send out a patch today.

> >
> > diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c
> > index 37a9deed2aec..71fb18b2f304 100644
> > --- a/arch/arm64/kernel/kaslr.c
> > +++ b/arch/arm64/kernel/kaslr.c
> > @@ -90,4 +90,4 @@ static int __init kaslr_init(void)
> >
> >          return 0;
> >   }
> > -subsys_initcall(kaslr_init)
> > +arch_initcall(kaslr_init)
> > .
> >
>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: regression: insmod module failed in VM with nvdimm on

[Ard Biesheuvel]

On Thu, 1 Dec 2022 at 09:07, Ard Biesheuvel <ardb@kernel.org> wrote:
>
> On Thu, 1 Dec 2022 at 08:15, chenxiang (M) <chenxiang66@hisilicon.com> wrote:
> >
> > Hi Ard,
> >
> >
> > 在 2022/11/30 16:18, Ard Biesheuvel 写道:
> > > On Wed, 30 Nov 2022 at 08:53, Marc Zyngier <maz@kernel.org> wrote:
> > >> On Wed, 30 Nov 2022 02:52:35 +0000,
> > >> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote:
> > >>> Hi,
> > >>>
> > >>> We boot the VM using following commands (with nvdimm on)  (qemu
> > >>> version 6.1.50, kernel 6.0-r4):
> > >> How relevant is the presence of the nvdimm? Do you observe the failure
> > >> without this?
> > >>
> > >>> qemu-system-aarch64 -machine
> > >>> virt,kernel_irqchip=on,gic-version=3,nvdimm=on  -kernel
> > >>> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios
> > >>> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m
> > >>> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0
> > >>> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1'
> > >>> -object memory-backend-ram,id=ram1,size=10G -device
> > >>> nvdimm,id=dimm1,memdev=ram1  -device ioh3420,id=root_port1,chassis=1
> > >>> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1
> > >>>
> > >>> Then in VM we insmod a module, vmalloc error occurs as follows (kernel
> > >>> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4):
> > >>>
> > >>> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko
> > >>> [    8.186563] vmap allocation for size 20480 failed: use
> > >>> vmalloc=<size> to increase size
> > >> Have you tried increasing the vmalloc size to check that this is
> > >> indeed the problem?
> > >>
> > >> [...]
> > >>
> > >>> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr:
> > >>> defer initialization to initcall where permitted").
> > >> I guess you mean commit fc5a89f75d2a instead, right?
> > >>
> > >>> Do you have any idea about the issue?
> > >> I sort of suspect that the nvdimm gets vmap-ed and consumes a large
> > >> portion of the vmalloc space, but you give very little information
> > >> that could help here...
> > >>
> > > Ouch. I suspect what's going on here: that patch defers the
> > > randomization of the module region, so that we can decouple it from
> > > the very early init code.
> > >
> > > Obviously, it is happening too late now, and the randomized module
> > > region is overlapping with a vmalloc region that is in use by the time
> > > the randomization occurs.
> > >
> > > Does the below fix the issue?
> >
> > The issue still occurs, but it seems decrease the probability, before it
> > occured almost every time, after the change, i tried 2-3 times, and it
> > occurs.
> > But i change back "subsys_initcall" to "core_initcall", and i test more
> > than 20 times, and it is still ok.
> >
>
> Thank you for confirming. I will send out a patch today.
>

...but before I do that, could you please check whether the change
below fixes your issue as well?

diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c
index 6ccc7ef600e7c1e1..c8c205b630da1951 100644
--- a/arch/arm64/kernel/kaslr.c
+++ b/arch/arm64/kernel/kaslr.c
@@ -20,7 +20,11 @@
 #include <asm/sections.h>
 #include <asm/setup.h>

-u64 __ro_after_init module_alloc_base;
+/*
+ * Set a reasonable default for module_alloc_base in case
+ * we end up running with module randomization disabled.
+ */
+u64 __ro_after_init module_alloc_base = (u64)_etext - MODULES_VSIZE;
 u16 __initdata memstart_offset_seed;

 struct arm64_ftr_override kaslr_feature_override __initdata;
@@ -30,12 +34,6 @@ static int __init kaslr_init(void)
        u64 module_range;
        u32 seed;

-       /*
-        * Set a reasonable default for module_alloc_base in case
-        * we end up running with module randomization disabled.
-        */
-       module_alloc_base = (u64)_etext - MODULES_VSIZE;
-
        if (kaslr_feature_override.val & kaslr_feature_override.mask & 0xf) {
                pr_info("KASLR disabled on command line\n");
                return 0;

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: regression: insmod module failed in VM with nvdimm on

[chenxiang (M)]

在 2022/12/1 19:07, Ard Biesheuvel 写道:
> On Thu, 1 Dec 2022 at 09:07, Ard Biesheuvel <ardb@kernel.org> wrote:
>> On Thu, 1 Dec 2022 at 08:15, chenxiang (M) <chenxiang66@hisilicon.com> wrote:
>>> Hi Ard,
>>>
>>>
>>> 在 2022/11/30 16:18, Ard Biesheuvel 写道:
>>>> On Wed, 30 Nov 2022 at 08:53, Marc Zyngier <maz@kernel.org> wrote:
>>>>> On Wed, 30 Nov 2022 02:52:35 +0000,
>>>>> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> We boot the VM using following commands (with nvdimm on)  (qemu
>>>>>> version 6.1.50, kernel 6.0-r4):
>>>>> How relevant is the presence of the nvdimm? Do you observe the failure
>>>>> without this?
>>>>>
>>>>>> qemu-system-aarch64 -machine
>>>>>> virt,kernel_irqchip=on,gic-version=3,nvdimm=on  -kernel
>>>>>> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios
>>>>>> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m
>>>>>> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0
>>>>>> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1'
>>>>>> -object memory-backend-ram,id=ram1,size=10G -device
>>>>>> nvdimm,id=dimm1,memdev=ram1  -device ioh3420,id=root_port1,chassis=1
>>>>>> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1
>>>>>>
>>>>>> Then in VM we insmod a module, vmalloc error occurs as follows (kernel
>>>>>> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4):
>>>>>>
>>>>>> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko
>>>>>> [    8.186563] vmap allocation for size 20480 failed: use
>>>>>> vmalloc=<size> to increase size
>>>>> Have you tried increasing the vmalloc size to check that this is
>>>>> indeed the problem?
>>>>>
>>>>> [...]
>>>>>
>>>>>> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr:
>>>>>> defer initialization to initcall where permitted").
>>>>> I guess you mean commit fc5a89f75d2a instead, right?
>>>>>
>>>>>> Do you have any idea about the issue?
>>>>> I sort of suspect that the nvdimm gets vmap-ed and consumes a large
>>>>> portion of the vmalloc space, but you give very little information
>>>>> that could help here...
>>>>>
>>>> Ouch. I suspect what's going on here: that patch defers the
>>>> randomization of the module region, so that we can decouple it from
>>>> the very early init code.
>>>>
>>>> Obviously, it is happening too late now, and the randomized module
>>>> region is overlapping with a vmalloc region that is in use by the time
>>>> the randomization occurs.
>>>>
>>>> Does the below fix the issue?
>>> The issue still occurs, but it seems decrease the probability, before it
>>> occured almost every time, after the change, i tried 2-3 times, and it
>>> occurs.
>>> But i change back "subsys_initcall" to "core_initcall", and i test more
>>> than 20 times, and it is still ok.
>>>
>> Thank you for confirming. I will send out a patch today.
>>
> ...but before I do that, could you please check whether the change
> below fixes your issue as well?

Yes, but i can only reply to you tomorrow as other guy is testing on the 
only environment today.

>
> diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c
> index 6ccc7ef600e7c1e1..c8c205b630da1951 100644
> --- a/arch/arm64/kernel/kaslr.c
> +++ b/arch/arm64/kernel/kaslr.c
> @@ -20,7 +20,11 @@
>   #include <asm/sections.h>
>   #include <asm/setup.h>
>
> -u64 __ro_after_init module_alloc_base;
> +/*
> + * Set a reasonable default for module_alloc_base in case
> + * we end up running with module randomization disabled.
> + */
> +u64 __ro_after_init module_alloc_base = (u64)_etext - MODULES_VSIZE;
>   u16 __initdata memstart_offset_seed;
>
>   struct arm64_ftr_override kaslr_feature_override __initdata;
> @@ -30,12 +34,6 @@ static int __init kaslr_init(void)
>          u64 module_range;
>          u32 seed;
>
> -       /*
> -        * Set a reasonable default for module_alloc_base in case
> -        * we end up running with module randomization disabled.
> -        */
> -       module_alloc_base = (u64)_etext - MODULES_VSIZE;
> -
>          if (kaslr_feature_override.val & kaslr_feature_override.mask & 0xf) {
>                  pr_info("KASLR disabled on command line\n");
>                  return 0;
> .
>


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: regression: insmod module failed in VM with nvdimm on

[Ard Biesheuvel]

On Thu, 1 Dec 2022 at 13:07, chenxiang (M) <chenxiang66@hisilicon.com> wrote:
>
>
>
> 在 2022/12/1 19:07, Ard Biesheuvel 写道:
> > On Thu, 1 Dec 2022 at 09:07, Ard Biesheuvel <ardb@kernel.org> wrote:
> >> On Thu, 1 Dec 2022 at 08:15, chenxiang (M) <chenxiang66@hisilicon.com> wrote:
> >>> Hi Ard,
> >>>
> >>>
> >>> 在 2022/11/30 16:18, Ard Biesheuvel 写道:
> >>>> On Wed, 30 Nov 2022 at 08:53, Marc Zyngier <maz@kernel.org> wrote:
> >>>>> On Wed, 30 Nov 2022 02:52:35 +0000,
> >>>>> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> We boot the VM using following commands (with nvdimm on)  (qemu
> >>>>>> version 6.1.50, kernel 6.0-r4):
> >>>>> How relevant is the presence of the nvdimm? Do you observe the failure
> >>>>> without this?
> >>>>>
> >>>>>> qemu-system-aarch64 -machine
> >>>>>> virt,kernel_irqchip=on,gic-version=3,nvdimm=on  -kernel
> >>>>>> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios
> >>>>>> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m
> >>>>>> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0
> >>>>>> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1'
> >>>>>> -object memory-backend-ram,id=ram1,size=10G -device
> >>>>>> nvdimm,id=dimm1,memdev=ram1  -device ioh3420,id=root_port1,chassis=1
> >>>>>> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1
> >>>>>>
> >>>>>> Then in VM we insmod a module, vmalloc error occurs as follows (kernel
> >>>>>> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4):
> >>>>>>
> >>>>>> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko
> >>>>>> [    8.186563] vmap allocation for size 20480 failed: use
> >>>>>> vmalloc=<size> to increase size
> >>>>> Have you tried increasing the vmalloc size to check that this is
> >>>>> indeed the problem?
> >>>>>
> >>>>> [...]
> >>>>>
> >>>>>> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr:
> >>>>>> defer initialization to initcall where permitted").
> >>>>> I guess you mean commit fc5a89f75d2a instead, right?
> >>>>>
> >>>>>> Do you have any idea about the issue?
> >>>>> I sort of suspect that the nvdimm gets vmap-ed and consumes a large
> >>>>> portion of the vmalloc space, but you give very little information
> >>>>> that could help here...
> >>>>>
> >>>> Ouch. I suspect what's going on here: that patch defers the
> >>>> randomization of the module region, so that we can decouple it from
> >>>> the very early init code.
> >>>>
> >>>> Obviously, it is happening too late now, and the randomized module
> >>>> region is overlapping with a vmalloc region that is in use by the time
> >>>> the randomization occurs.
> >>>>
> >>>> Does the below fix the issue?
> >>> The issue still occurs, but it seems decrease the probability, before it
> >>> occured almost every time, after the change, i tried 2-3 times, and it
> >>> occurs.
> >>> But i change back "subsys_initcall" to "core_initcall", and i test more
> >>> than 20 times, and it is still ok.
> >>>
> >> Thank you for confirming. I will send out a patch today.
> >>
> > ...but before I do that, could you please check whether the change
> > below fixes your issue as well?
>
> Yes, but i can only reply to you tomorrow as other guy is testing on the
> only environment today.
>

That is fine, thanks.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: regression: insmod module failed in VM with nvdimm on

[chenxiang (M)]

Hi Ard,


在 2022/12/1 19:07, Ard Biesheuvel 写道:
> On Thu, 1 Dec 2022 at 09:07, Ard Biesheuvel <ardb@kernel.org> wrote:
>> On Thu, 1 Dec 2022 at 08:15, chenxiang (M) <chenxiang66@hisilicon.com> wrote:
>>> Hi Ard,
>>>
>>>
>>> 在 2022/11/30 16:18, Ard Biesheuvel 写道:
>>>> On Wed, 30 Nov 2022 at 08:53, Marc Zyngier <maz@kernel.org> wrote:
>>>>> On Wed, 30 Nov 2022 02:52:35 +0000,
>>>>> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> We boot the VM using following commands (with nvdimm on)  (qemu
>>>>>> version 6.1.50, kernel 6.0-r4):
>>>>> How relevant is the presence of the nvdimm? Do you observe the failure
>>>>> without this?
>>>>>
>>>>>> qemu-system-aarch64 -machine
>>>>>> virt,kernel_irqchip=on,gic-version=3,nvdimm=on  -kernel
>>>>>> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios
>>>>>> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m
>>>>>> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0
>>>>>> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1'
>>>>>> -object memory-backend-ram,id=ram1,size=10G -device
>>>>>> nvdimm,id=dimm1,memdev=ram1  -device ioh3420,id=root_port1,chassis=1
>>>>>> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1
>>>>>>
>>>>>> Then in VM we insmod a module, vmalloc error occurs as follows (kernel
>>>>>> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4):
>>>>>>
>>>>>> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko
>>>>>> [    8.186563] vmap allocation for size 20480 failed: use
>>>>>> vmalloc=<size> to increase size
>>>>> Have you tried increasing the vmalloc size to check that this is
>>>>> indeed the problem?
>>>>>
>>>>> [...]
>>>>>
>>>>>> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr:
>>>>>> defer initialization to initcall where permitted").
>>>>> I guess you mean commit fc5a89f75d2a instead, right?
>>>>>
>>>>>> Do you have any idea about the issue?
>>>>> I sort of suspect that the nvdimm gets vmap-ed and consumes a large
>>>>> portion of the vmalloc space, but you give very little information
>>>>> that could help here...
>>>>>
>>>> Ouch. I suspect what's going on here: that patch defers the
>>>> randomization of the module region, so that we can decouple it from
>>>> the very early init code.
>>>>
>>>> Obviously, it is happening too late now, and the randomized module
>>>> region is overlapping with a vmalloc region that is in use by the time
>>>> the randomization occurs.
>>>>
>>>> Does the below fix the issue?
>>> The issue still occurs, but it seems decrease the probability, before it
>>> occured almost every time, after the change, i tried 2-3 times, and it
>>> occurs.
>>> But i change back "subsys_initcall" to "core_initcall", and i test more
>>> than 20 times, and it is still ok.
>>>
>> Thank you for confirming. I will send out a patch today.
>>
> ...but before I do that, could you please check whether the change
> below fixes your issue as well?
>
> diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c
> index 6ccc7ef600e7c1e1..c8c205b630da1951 100644
> --- a/arch/arm64/kernel/kaslr.c
> +++ b/arch/arm64/kernel/kaslr.c
> @@ -20,7 +20,11 @@
>   #include <asm/sections.h>
>   #include <asm/setup.h>
>
> -u64 __ro_after_init module_alloc_base;
> +/*
> + * Set a reasonable default for module_alloc_base in case
> + * we end up running with module randomization disabled.
> + */
> +u64 __ro_after_init module_alloc_base = (u64)_etext - MODULES_VSIZE;
>   u16 __initdata memstart_offset_seed;
>
>   struct arm64_ftr_override kaslr_feature_override __initdata;
> @@ -30,12 +34,6 @@ static int __init kaslr_init(void)
>          u64 module_range;
>          u32 seed;
>
> -       /*
> -        * Set a reasonable default for module_alloc_base in case
> -        * we end up running with module randomization disabled.
> -        */
> -       module_alloc_base = (u64)_etext - MODULES_VSIZE;
> -
>          if (kaslr_feature_override.val & kaslr_feature_override.mask & 0xf) {
>                  pr_info("KASLR disabled on command line\n");
>                  return 0;
> .

We have tested this change, the issue is still and it doesn't fix the issue.


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: regression: insmod module failed in VM with nvdimm on

[Ard Biesheuvel]

On Fri, 2 Dec 2022 at 03:48, chenxiang (M) <chenxiang66@hisilicon.com> wrote:
>
> Hi Ard,
>
>
> 在 2022/12/1 19:07, Ard Biesheuvel 写道:
> > On Thu, 1 Dec 2022 at 09:07, Ard Biesheuvel <ardb@kernel.org> wrote:
> >> On Thu, 1 Dec 2022 at 08:15, chenxiang (M) <chenxiang66@hisilicon.com> wrote:
> >>> Hi Ard,
> >>>
> >>>
> >>> 在 2022/11/30 16:18, Ard Biesheuvel 写道:
> >>>> On Wed, 30 Nov 2022 at 08:53, Marc Zyngier <maz@kernel.org> wrote:
> >>>>> On Wed, 30 Nov 2022 02:52:35 +0000,
> >>>>> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> We boot the VM using following commands (with nvdimm on)  (qemu
> >>>>>> version 6.1.50, kernel 6.0-r4):
> >>>>> How relevant is the presence of the nvdimm? Do you observe the failure
> >>>>> without this?
> >>>>>
> >>>>>> qemu-system-aarch64 -machine
> >>>>>> virt,kernel_irqchip=on,gic-version=3,nvdimm=on  -kernel
> >>>>>> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios
> >>>>>> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m
> >>>>>> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0
> >>>>>> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1'
> >>>>>> -object memory-backend-ram,id=ram1,size=10G -device
> >>>>>> nvdimm,id=dimm1,memdev=ram1  -device ioh3420,id=root_port1,chassis=1
> >>>>>> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1
> >>>>>>
> >>>>>> Then in VM we insmod a module, vmalloc error occurs as follows (kernel
> >>>>>> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4):
> >>>>>>
> >>>>>> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko
> >>>>>> [    8.186563] vmap allocation for size 20480 failed: use
> >>>>>> vmalloc=<size> to increase size
> >>>>> Have you tried increasing the vmalloc size to check that this is
> >>>>> indeed the problem?
> >>>>>
> >>>>> [...]
> >>>>>
> >>>>>> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr:
> >>>>>> defer initialization to initcall where permitted").
> >>>>> I guess you mean commit fc5a89f75d2a instead, right?
> >>>>>
> >>>>>> Do you have any idea about the issue?
> >>>>> I sort of suspect that the nvdimm gets vmap-ed and consumes a large
> >>>>> portion of the vmalloc space, but you give very little information
> >>>>> that could help here...
> >>>>>
> >>>> Ouch. I suspect what's going on here: that patch defers the
> >>>> randomization of the module region, so that we can decouple it from
> >>>> the very early init code.
> >>>>
> >>>> Obviously, it is happening too late now, and the randomized module
> >>>> region is overlapping with a vmalloc region that is in use by the time
> >>>> the randomization occurs.
> >>>>
> >>>> Does the below fix the issue?
> >>> The issue still occurs, but it seems decrease the probability, before it
> >>> occured almost every time, after the change, i tried 2-3 times, and it
> >>> occurs.
> >>> But i change back "subsys_initcall" to "core_initcall", and i test more
> >>> than 20 times, and it is still ok.
> >>>
> >> Thank you for confirming. I will send out a patch today.
> >>
> > ...but before I do that, could you please check whether the change
> > below fixes your issue as well?
> >
> > diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c
> > index 6ccc7ef600e7c1e1..c8c205b630da1951 100644
> > --- a/arch/arm64/kernel/kaslr.c
> > +++ b/arch/arm64/kernel/kaslr.c
> > @@ -20,7 +20,11 @@
> >   #include <asm/sections.h>
> >   #include <asm/setup.h>
> >
> > -u64 __ro_after_init module_alloc_base;
> > +/*
> > + * Set a reasonable default for module_alloc_base in case
> > + * we end up running with module randomization disabled.
> > + */
> > +u64 __ro_after_init module_alloc_base = (u64)_etext - MODULES_VSIZE;
> >   u16 __initdata memstart_offset_seed;
> >
> >   struct arm64_ftr_override kaslr_feature_override __initdata;
> > @@ -30,12 +34,6 @@ static int __init kaslr_init(void)
> >          u64 module_range;
> >          u32 seed;
> >
> > -       /*
> > -        * Set a reasonable default for module_alloc_base in case
> > -        * we end up running with module randomization disabled.
> > -        */
> > -       module_alloc_base = (u64)_etext - MODULES_VSIZE;
> > -
> >          if (kaslr_feature_override.val & kaslr_feature_override.mask & 0xf) {
> >                  pr_info("KASLR disabled on command line\n");
> >                  return 0;
> > .
>
> We have tested this change, the issue is still and it doesn't fix the issue.
>

Thanks for the report.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: regression: insmod module failed in VM with nvdimm on

[Thorsten Leemhuis]

Hi, this is your Linux kernel regression tracker. Top-posting for once,
to make this easily accessible to everyone.

Was there some progress to get this regression resolved? From here it
looks stalled, but maybe I missed something.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I deal with a lot of
reports and sometimes miss something important when writing mails like
this. If that's the case here, don't hesitate to tell me in a public
reply, it's in everyone's interest to set the public record straight.

On 02.12.22 14:44, Ard Biesheuvel wrote:
> On Fri, 2 Dec 2022 at 03:48, chenxiang (M) <chenxiang66@hisilicon.com> wrote:
>>
>> Hi Ard,
>>
>>
>> 在 2022/12/1 19:07, Ard Biesheuvel 写道:
>>> On Thu, 1 Dec 2022 at 09:07, Ard Biesheuvel <ardb@kernel.org> wrote:
>>>> On Thu, 1 Dec 2022 at 08:15, chenxiang (M) <chenxiang66@hisilicon.com> wrote:
>>>>> Hi Ard,
>>>>>
>>>>>
>>>>> 在 2022/11/30 16:18, Ard Biesheuvel 写道:
>>>>>> On Wed, 30 Nov 2022 at 08:53, Marc Zyngier <maz@kernel.org> wrote:
>>>>>>> On Wed, 30 Nov 2022 02:52:35 +0000,
>>>>>>> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> We boot the VM using following commands (with nvdimm on)  (qemu
>>>>>>>> version 6.1.50, kernel 6.0-r4):
>>>>>>> How relevant is the presence of the nvdimm? Do you observe the failure
>>>>>>> without this?
>>>>>>>
>>>>>>>> qemu-system-aarch64 -machine
>>>>>>>> virt,kernel_irqchip=on,gic-version=3,nvdimm=on  -kernel
>>>>>>>> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios
>>>>>>>> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m
>>>>>>>> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0
>>>>>>>> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1'
>>>>>>>> -object memory-backend-ram,id=ram1,size=10G -device
>>>>>>>> nvdimm,id=dimm1,memdev=ram1  -device ioh3420,id=root_port1,chassis=1
>>>>>>>> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1
>>>>>>>>
>>>>>>>> Then in VM we insmod a module, vmalloc error occurs as follows (kernel
>>>>>>>> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4):
>>>>>>>>
>>>>>>>> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko
>>>>>>>> [    8.186563] vmap allocation for size 20480 failed: use
>>>>>>>> vmalloc=<size> to increase size
>>>>>>> Have you tried increasing the vmalloc size to check that this is
>>>>>>> indeed the problem?
>>>>>>>
>>>>>>> [...]
>>>>>>>
>>>>>>>> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr:
>>>>>>>> defer initialization to initcall where permitted").
>>>>>>> I guess you mean commit fc5a89f75d2a instead, right?
>>>>>>>
>>>>>>>> Do you have any idea about the issue?
>>>>>>> I sort of suspect that the nvdimm gets vmap-ed and consumes a large
>>>>>>> portion of the vmalloc space, but you give very little information
>>>>>>> that could help here...
>>>>>>>
>>>>>> Ouch. I suspect what's going on here: that patch defers the
>>>>>> randomization of the module region, so that we can decouple it from
>>>>>> the very early init code.
>>>>>>
>>>>>> Obviously, it is happening too late now, and the randomized module
>>>>>> region is overlapping with a vmalloc region that is in use by the time
>>>>>> the randomization occurs.
>>>>>>
>>>>>> Does the below fix the issue?
>>>>> The issue still occurs, but it seems decrease the probability, before it
>>>>> occured almost every time, after the change, i tried 2-3 times, and it
>>>>> occurs.
>>>>> But i change back "subsys_initcall" to "core_initcall", and i test more
>>>>> than 20 times, and it is still ok.
>>>>>
>>>> Thank you for confirming. I will send out a patch today.
>>>>
>>> ...but before I do that, could you please check whether the change
>>> below fixes your issue as well?
>>>
>>> diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c
>>> index 6ccc7ef600e7c1e1..c8c205b630da1951 100644
>>> --- a/arch/arm64/kernel/kaslr.c
>>> +++ b/arch/arm64/kernel/kaslr.c
>>> @@ -20,7 +20,11 @@
>>>   #include <asm/sections.h>
>>>   #include <asm/setup.h>
>>>
>>> -u64 __ro_after_init module_alloc_base;
>>> +/*
>>> + * Set a reasonable default for module_alloc_base in case
>>> + * we end up running with module randomization disabled.
>>> + */
>>> +u64 __ro_after_init module_alloc_base = (u64)_etext - MODULES_VSIZE;
>>>   u16 __initdata memstart_offset_seed;
>>>
>>>   struct arm64_ftr_override kaslr_feature_override __initdata;
>>> @@ -30,12 +34,6 @@ static int __init kaslr_init(void)
>>>          u64 module_range;
>>>          u32 seed;
>>>
>>> -       /*
>>> -        * Set a reasonable default for module_alloc_base in case
>>> -        * we end up running with module randomization disabled.
>>> -        */
>>> -       module_alloc_base = (u64)_etext - MODULES_VSIZE;
>>> -
>>>          if (kaslr_feature_override.val & kaslr_feature_override.mask & 0xf) {
>>>                  pr_info("KASLR disabled on command line\n");
>>>                  return 0;
>>> .
>>
>> We have tested this change, the issue is still and it doesn't fix the issue.
>>
> 
> Thanks for the report.
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#regzbot poke

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: regression: insmod module failed in VM with nvdimm on #forregzbot

[Linux regression tracking #update (Thorsten Leemhuis)]

[TLDR: This mail in primarily relevant for Linux regression tracking. A
change or fix related to the regression discussed in this thread was
posted or applied, but it did not use a Link: tag to point to the
report, as Linus and the documentation call for. Things happen, no
worries -- but now the regression tracking bot needs to be told manually
about the fix. See link in footer if these mails annoy you.]

On 30.11.22 11:10, Thorsten Leemhuis wrote:
>
> On 30.11.22 03:52, chenxiang (M) wrote:
>>
>> We boot the VM using following commands (with nvdimm on)  (qemu version
>> 6.1.50, kernel 6.0-r4):
>>
>> qemu-system-aarch64 -machine
>> virt,kernel_irqchip=on,gic-version=3,nvdimm=on  -kernel
>> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios
>> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m
>> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0
>> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1'
>> -object memory-backend-ram,id=ram1,size=10G -device
>> nvdimm,id=dimm1,memdev=ram1  -device ioh3420,id=root_port1,chassis=1
>> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1
>> [...]
>> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr:
>> defer initialization to initcall where permitted").
> 
> Thanks for the report. To be sure below issue doesn't fall through the
> cracks unnoticed, I'm adding it to regzbot, my Linux kernel regression
> tracking bot:
> 
> #regzbot ^introduced fc5a89f75d2a
> #regzbot title arm64: kaslr: vmalloc error when boothing a VM with nvdimm
> #regzbot ignore-activity

#regzbot fix: arm64: kaslr: don't pretend KASLR is enabled if offset <
MIN_KIMG_ALIGN
#regzbot ignore-activity

Backstory:

https://lore.kernel.org/lkml/CAMj1kXGWEaQXoKj=DzG9XpVGi4t5zfE-RSG0BodVL-b47nsj-A@mail.gmail.com/
https://lore.kernel.org/all/20230223204101.1500373-1-ardb@kernel.org/

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel