* regression: insmod module failed in VM with nvdimm on @ 2022-11-30 2:52 chenxiang (M) 2022-11-30 7:53 ` Marc Zyngier 2022-11-30 10:10 ` regression: insmod module failed in VM with nvdimm on #forregzbot Thorsten Leemhuis 0 siblings, 2 replies; 14+ messages in thread From: chenxiang (M) @ 2022-11-30 2:52 UTC (permalink / raw) To: ardb Cc: Marc Zyngier, will, mark.rutland, linux-arm-kernel, chenxiang via, linuxarm Hi, We boot the VM using following commands (with nvdimm on) (qemu version 6.1.50, kernel 6.0-r4): qemu-system-aarch64 -machine virt,kernel_irqchip=on,gic-version=3,nvdimm=on -kernel /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0 ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1' -object memory-backend-ram,id=ram1,size=10G -device nvdimm,id=dimm1,memdev=ram1 -device ioh3420,id=root_port1,chassis=1 -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1 Then in VM we insmod a module, vmalloc error occurs as follows (kernel 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4): estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko [ 8.186563] vmap allocation for size 20480 failed: use vmalloc=<size> to increase size [ 8.187288] insmod: vmalloc error: size 16384, vm_struct allocation failed, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0 [ 8.188402] CPU: 1 PID: 235 Comm: insmod Not tainted 6.0.0-rc4+ #1 [ 8.188958] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [ 8.189593] Call trace: [ 8.189825] dump_backtrace.part.0+0xc4/0xd0 [ 8.190245] show_stack+0x24/0x40 [ 8.190563] dump_stack_lvl+0x68/0x84 [ 8.190913] dump_stack+0x18/0x34 [ 8.191223] warn_alloc+0x124/0x1b0 [ 8.191555] __vmalloc_node_range+0xe4/0x55c [ 8.191959] module_alloc+0xf8/0x104 [ 8.192305] load_module+0x854/0x1e70 [ 8.192655] __do_sys_init_module+0x1e0/0x220 [ 8.193075] __arm64_sys_init_module+0x28/0x34 [ 8.193489] invoke_syscall+0x50/0x120 [ 8.193841] el0_svc_common.constprop.0+0x58/0x1a0 [ 8.194296] do_el0_svc+0x38/0xd0 [ 8.194613] el0_svc+0x2c/0xc0 [ 8.194901] el0t_64_sync_handler+0x1ac/0x1b0 [ 8.195313] el0t_64_sync+0x19c/0x1a0 [ 8.195672] Mem-Info: [ 8.195872] active_anon:17641 inactive_anon:118549 isolated_anon:0 [ 8.195872] active_file:0 inactive_file:0 isolated_file:0 [ 8.195872] unevictable:0 dirty:0 writeback:0 [ 8.195872] slab_reclaimable:3439 slab_unreclaimable:3067 [ 8.195872] mapped:877 shmem:135976 pagetables:39 bounce:0 [ 8.195872] kernel_misc_reclaimable:0 [ 8.195872] free:353735 free_pcp:3210 free_cma:0 [ 8.199119] Node 0 active_anon:70564kB inactive_anon:474196kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:3508kB dirty:0kB writeback:0kB shmem:543904kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:1904kB pagetables:156kB all_unreclaimable? no [ 8.201683] Node 0 DMA free:1414940kB boost:0kB min:22528kB low:28160kB high:33792kB reserved_highatomic:0KB active_anon:70564kB inactive_anon:474196kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:2097152kB managed:2010444kB mlocked:0kB bounce:0kB free_pcp:12840kB local_pcp:2032kB free_cma:0kB [ 8.204158] lowmem_reserve[]: 0 0 0 0 [ 8.204481] Node 0 DMA: 1*4kB (E) 1*8kB (U) 1*16kB (U) 2*32kB (UM) 1*64kB (U) 1*128kB (U) 2*256kB (ME) 2*512kB (ME) 2*1024kB (M) 3*2048kB (UM) 343*4096kB (M) = 1414940kB [ 8.205881] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [ 8.206644] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=32768kB [ 8.207381] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 8.208111] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=64kB [ 8.208826] 135976 total pagecache pages [ 8.209195] 0 pages in swap cache [ 8.209484] Free swap = 0kB [ 8.209733] Total swap = 0kB [ 8.209989] 524288 pages RAM [ 8.210239] 0 pages HighMem/MovableOnly [ 8.210571] 21677 pages reserved [ 8.210852] 0 pages hwpoisoned insmod: can't insert '/lib/modules/6.0.0-rc4+/hnae3.ko': Cannot allocate memory We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr: defer initialization to initcall where permitted"). Do you have any idea about the issue? Best Regards, Xiang Chen _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: regression: insmod module failed in VM with nvdimm on 2022-11-30 2:52 regression: insmod module failed in VM with nvdimm on chenxiang (M) @ 2022-11-30 7:53 ` Marc Zyngier 2022-11-30 8:18 ` Ard Biesheuvel 2022-12-01 7:01 ` chenxiang (M) 2022-11-30 10:10 ` regression: insmod module failed in VM with nvdimm on #forregzbot Thorsten Leemhuis 1 sibling, 2 replies; 14+ messages in thread From: Marc Zyngier @ 2022-11-30 7:53 UTC (permalink / raw) To: chenxiang (M) Cc: ardb, will, mark.rutland, linux-arm-kernel, chenxiang via, linuxarm On Wed, 30 Nov 2022 02:52:35 +0000, "chenxiang (M)" <chenxiang66@hisilicon.com> wrote: > > Hi, > > We boot the VM using following commands (with nvdimm on) (qemu > version 6.1.50, kernel 6.0-r4): How relevant is the presence of the nvdimm? Do you observe the failure without this? > > qemu-system-aarch64 -machine > virt,kernel_irqchip=on,gic-version=3,nvdimm=on -kernel > /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios > /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m > 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0 > ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1' > -object memory-backend-ram,id=ram1,size=10G -device > nvdimm,id=dimm1,memdev=ram1 -device ioh3420,id=root_port1,chassis=1 > -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1 > > Then in VM we insmod a module, vmalloc error occurs as follows (kernel > 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4): > > estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko > [ 8.186563] vmap allocation for size 20480 failed: use > vmalloc=<size> to increase size Have you tried increasing the vmalloc size to check that this is indeed the problem? [...] > We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr: > defer initialization to initcall where permitted"). I guess you mean commit fc5a89f75d2a instead, right? > Do you have any idea about the issue? I sort of suspect that the nvdimm gets vmap-ed and consumes a large portion of the vmalloc space, but you give very little information that could help here... M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: regression: insmod module failed in VM with nvdimm on 2022-11-30 7:53 ` Marc Zyngier @ 2022-11-30 8:18 ` Ard Biesheuvel 2022-12-01 7:15 ` chenxiang (M) 2022-12-01 7:01 ` chenxiang (M) 1 sibling, 1 reply; 14+ messages in thread From: Ard Biesheuvel @ 2022-11-30 8:18 UTC (permalink / raw) To: Marc Zyngier Cc: chenxiang (M), will, mark.rutland, linux-arm-kernel, chenxiang via, linuxarm On Wed, 30 Nov 2022 at 08:53, Marc Zyngier <maz@kernel.org> wrote: > > On Wed, 30 Nov 2022 02:52:35 +0000, > "chenxiang (M)" <chenxiang66@hisilicon.com> wrote: > > > > Hi, > > > > We boot the VM using following commands (with nvdimm on) (qemu > > version 6.1.50, kernel 6.0-r4): > > How relevant is the presence of the nvdimm? Do you observe the failure > without this? > > > > > qemu-system-aarch64 -machine > > virt,kernel_irqchip=on,gic-version=3,nvdimm=on -kernel > > /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios > > /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m > > 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0 > > ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1' > > -object memory-backend-ram,id=ram1,size=10G -device > > nvdimm,id=dimm1,memdev=ram1 -device ioh3420,id=root_port1,chassis=1 > > -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1 > > > > Then in VM we insmod a module, vmalloc error occurs as follows (kernel > > 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4): > > > > estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko > > [ 8.186563] vmap allocation for size 20480 failed: use > > vmalloc=<size> to increase size > > Have you tried increasing the vmalloc size to check that this is > indeed the problem? > > [...] > > > We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr: > > defer initialization to initcall where permitted"). > > I guess you mean commit fc5a89f75d2a instead, right? > > > Do you have any idea about the issue? > > I sort of suspect that the nvdimm gets vmap-ed and consumes a large > portion of the vmalloc space, but you give very little information > that could help here... > Ouch. I suspect what's going on here: that patch defers the randomization of the module region, so that we can decouple it from the very early init code. Obviously, it is happening too late now, and the randomized module region is overlapping with a vmalloc region that is in use by the time the randomization occurs. Does the below fix the issue? diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c index 37a9deed2aec..71fb18b2f304 100644 --- a/arch/arm64/kernel/kaslr.c +++ b/arch/arm64/kernel/kaslr.c @@ -90,4 +90,4 @@ static int __init kaslr_init(void) return 0; } -subsys_initcall(kaslr_init) +arch_initcall(kaslr_init) _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: regression: insmod module failed in VM with nvdimm on 2022-11-30 8:18 ` Ard Biesheuvel @ 2022-12-01 7:15 ` chenxiang (M) 2022-12-01 8:07 ` Ard Biesheuvel 0 siblings, 1 reply; 14+ messages in thread From: chenxiang (M) @ 2022-12-01 7:15 UTC (permalink / raw) To: Ard Biesheuvel, Marc Zyngier Cc: will, mark.rutland, linux-arm-kernel, chenxiang via, linuxarm Hi Ard, 在 2022/11/30 16:18, Ard Biesheuvel 写道: > On Wed, 30 Nov 2022 at 08:53, Marc Zyngier <maz@kernel.org> wrote: >> On Wed, 30 Nov 2022 02:52:35 +0000, >> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote: >>> Hi, >>> >>> We boot the VM using following commands (with nvdimm on) (qemu >>> version 6.1.50, kernel 6.0-r4): >> How relevant is the presence of the nvdimm? Do you observe the failure >> without this? >> >>> qemu-system-aarch64 -machine >>> virt,kernel_irqchip=on,gic-version=3,nvdimm=on -kernel >>> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios >>> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m >>> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0 >>> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1' >>> -object memory-backend-ram,id=ram1,size=10G -device >>> nvdimm,id=dimm1,memdev=ram1 -device ioh3420,id=root_port1,chassis=1 >>> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1 >>> >>> Then in VM we insmod a module, vmalloc error occurs as follows (kernel >>> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4): >>> >>> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko >>> [ 8.186563] vmap allocation for size 20480 failed: use >>> vmalloc=<size> to increase size >> Have you tried increasing the vmalloc size to check that this is >> indeed the problem? >> >> [...] >> >>> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr: >>> defer initialization to initcall where permitted"). >> I guess you mean commit fc5a89f75d2a instead, right? >> >>> Do you have any idea about the issue? >> I sort of suspect that the nvdimm gets vmap-ed and consumes a large >> portion of the vmalloc space, but you give very little information >> that could help here... >> > Ouch. I suspect what's going on here: that patch defers the > randomization of the module region, so that we can decouple it from > the very early init code. > > Obviously, it is happening too late now, and the randomized module > region is overlapping with a vmalloc region that is in use by the time > the randomization occurs. > > Does the below fix the issue? The issue still occurs, but it seems decrease the probability, before it occured almost every time, after the change, i tried 2-3 times, and it occurs. But i change back "subsys_initcall" to "core_initcall", and i test more than 20 times, and it is still ok. > > diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c > index 37a9deed2aec..71fb18b2f304 100644 > --- a/arch/arm64/kernel/kaslr.c > +++ b/arch/arm64/kernel/kaslr.c > @@ -90,4 +90,4 @@ static int __init kaslr_init(void) > > return 0; > } > -subsys_initcall(kaslr_init) > +arch_initcall(kaslr_init) > . > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: regression: insmod module failed in VM with nvdimm on 2022-12-01 7:15 ` chenxiang (M) @ 2022-12-01 8:07 ` Ard Biesheuvel 2022-12-01 11:07 ` Ard Biesheuvel 0 siblings, 1 reply; 14+ messages in thread From: Ard Biesheuvel @ 2022-12-01 8:07 UTC (permalink / raw) To: chenxiang (M) Cc: Marc Zyngier, will, mark.rutland, linux-arm-kernel, chenxiang via, linuxarm On Thu, 1 Dec 2022 at 08:15, chenxiang (M) <chenxiang66@hisilicon.com> wrote: > > Hi Ard, > > > 在 2022/11/30 16:18, Ard Biesheuvel 写道: > > On Wed, 30 Nov 2022 at 08:53, Marc Zyngier <maz@kernel.org> wrote: > >> On Wed, 30 Nov 2022 02:52:35 +0000, > >> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote: > >>> Hi, > >>> > >>> We boot the VM using following commands (with nvdimm on) (qemu > >>> version 6.1.50, kernel 6.0-r4): > >> How relevant is the presence of the nvdimm? Do you observe the failure > >> without this? > >> > >>> qemu-system-aarch64 -machine > >>> virt,kernel_irqchip=on,gic-version=3,nvdimm=on -kernel > >>> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios > >>> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m > >>> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0 > >>> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1' > >>> -object memory-backend-ram,id=ram1,size=10G -device > >>> nvdimm,id=dimm1,memdev=ram1 -device ioh3420,id=root_port1,chassis=1 > >>> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1 > >>> > >>> Then in VM we insmod a module, vmalloc error occurs as follows (kernel > >>> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4): > >>> > >>> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko > >>> [ 8.186563] vmap allocation for size 20480 failed: use > >>> vmalloc=<size> to increase size > >> Have you tried increasing the vmalloc size to check that this is > >> indeed the problem? > >> > >> [...] > >> > >>> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr: > >>> defer initialization to initcall where permitted"). > >> I guess you mean commit fc5a89f75d2a instead, right? > >> > >>> Do you have any idea about the issue? > >> I sort of suspect that the nvdimm gets vmap-ed and consumes a large > >> portion of the vmalloc space, but you give very little information > >> that could help here... > >> > > Ouch. I suspect what's going on here: that patch defers the > > randomization of the module region, so that we can decouple it from > > the very early init code. > > > > Obviously, it is happening too late now, and the randomized module > > region is overlapping with a vmalloc region that is in use by the time > > the randomization occurs. > > > > Does the below fix the issue? > > The issue still occurs, but it seems decrease the probability, before it > occured almost every time, after the change, i tried 2-3 times, and it > occurs. > But i change back "subsys_initcall" to "core_initcall", and i test more > than 20 times, and it is still ok. > Thank you for confirming. I will send out a patch today. > > > > diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c > > index 37a9deed2aec..71fb18b2f304 100644 > > --- a/arch/arm64/kernel/kaslr.c > > +++ b/arch/arm64/kernel/kaslr.c > > @@ -90,4 +90,4 @@ static int __init kaslr_init(void) > > > > return 0; > > } > > -subsys_initcall(kaslr_init) > > +arch_initcall(kaslr_init) > > . > > > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: regression: insmod module failed in VM with nvdimm on 2022-12-01 8:07 ` Ard Biesheuvel @ 2022-12-01 11:07 ` Ard Biesheuvel 2022-12-01 12:06 ` chenxiang (M) 2022-12-02 2:48 ` chenxiang (M) 0 siblings, 2 replies; 14+ messages in thread From: Ard Biesheuvel @ 2022-12-01 11:07 UTC (permalink / raw) To: chenxiang (M) Cc: Marc Zyngier, will, mark.rutland, linux-arm-kernel, chenxiang via, linuxarm On Thu, 1 Dec 2022 at 09:07, Ard Biesheuvel <ardb@kernel.org> wrote: > > On Thu, 1 Dec 2022 at 08:15, chenxiang (M) <chenxiang66@hisilicon.com> wrote: > > > > Hi Ard, > > > > > > 在 2022/11/30 16:18, Ard Biesheuvel 写道: > > > On Wed, 30 Nov 2022 at 08:53, Marc Zyngier <maz@kernel.org> wrote: > > >> On Wed, 30 Nov 2022 02:52:35 +0000, > > >> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote: > > >>> Hi, > > >>> > > >>> We boot the VM using following commands (with nvdimm on) (qemu > > >>> version 6.1.50, kernel 6.0-r4): > > >> How relevant is the presence of the nvdimm? Do you observe the failure > > >> without this? > > >> > > >>> qemu-system-aarch64 -machine > > >>> virt,kernel_irqchip=on,gic-version=3,nvdimm=on -kernel > > >>> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios > > >>> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m > > >>> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0 > > >>> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1' > > >>> -object memory-backend-ram,id=ram1,size=10G -device > > >>> nvdimm,id=dimm1,memdev=ram1 -device ioh3420,id=root_port1,chassis=1 > > >>> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1 > > >>> > > >>> Then in VM we insmod a module, vmalloc error occurs as follows (kernel > > >>> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4): > > >>> > > >>> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko > > >>> [ 8.186563] vmap allocation for size 20480 failed: use > > >>> vmalloc=<size> to increase size > > >> Have you tried increasing the vmalloc size to check that this is > > >> indeed the problem? > > >> > > >> [...] > > >> > > >>> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr: > > >>> defer initialization to initcall where permitted"). > > >> I guess you mean commit fc5a89f75d2a instead, right? > > >> > > >>> Do you have any idea about the issue? > > >> I sort of suspect that the nvdimm gets vmap-ed and consumes a large > > >> portion of the vmalloc space, but you give very little information > > >> that could help here... > > >> > > > Ouch. I suspect what's going on here: that patch defers the > > > randomization of the module region, so that we can decouple it from > > > the very early init code. > > > > > > Obviously, it is happening too late now, and the randomized module > > > region is overlapping with a vmalloc region that is in use by the time > > > the randomization occurs. > > > > > > Does the below fix the issue? > > > > The issue still occurs, but it seems decrease the probability, before it > > occured almost every time, after the change, i tried 2-3 times, and it > > occurs. > > But i change back "subsys_initcall" to "core_initcall", and i test more > > than 20 times, and it is still ok. > > > > Thank you for confirming. I will send out a patch today. > ...but before I do that, could you please check whether the change below fixes your issue as well? diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c index 6ccc7ef600e7c1e1..c8c205b630da1951 100644 --- a/arch/arm64/kernel/kaslr.c +++ b/arch/arm64/kernel/kaslr.c @@ -20,7 +20,11 @@ #include <asm/sections.h> #include <asm/setup.h> -u64 __ro_after_init module_alloc_base; +/* + * Set a reasonable default for module_alloc_base in case + * we end up running with module randomization disabled. + */ +u64 __ro_after_init module_alloc_base = (u64)_etext - MODULES_VSIZE; u16 __initdata memstart_offset_seed; struct arm64_ftr_override kaslr_feature_override __initdata; @@ -30,12 +34,6 @@ static int __init kaslr_init(void) u64 module_range; u32 seed; - /* - * Set a reasonable default for module_alloc_base in case - * we end up running with module randomization disabled. - */ - module_alloc_base = (u64)_etext - MODULES_VSIZE; - if (kaslr_feature_override.val & kaslr_feature_override.mask & 0xf) { pr_info("KASLR disabled on command line\n"); return 0; _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: regression: insmod module failed in VM with nvdimm on 2022-12-01 11:07 ` Ard Biesheuvel @ 2022-12-01 12:06 ` chenxiang (M) 2022-12-01 12:53 ` Ard Biesheuvel 2022-12-02 2:48 ` chenxiang (M) 1 sibling, 1 reply; 14+ messages in thread From: chenxiang (M) @ 2022-12-01 12:06 UTC (permalink / raw) To: Ard Biesheuvel Cc: Marc Zyngier, will, mark.rutland, linux-arm-kernel, chenxiang via, linuxarm 在 2022/12/1 19:07, Ard Biesheuvel 写道: > On Thu, 1 Dec 2022 at 09:07, Ard Biesheuvel <ardb@kernel.org> wrote: >> On Thu, 1 Dec 2022 at 08:15, chenxiang (M) <chenxiang66@hisilicon.com> wrote: >>> Hi Ard, >>> >>> >>> 在 2022/11/30 16:18, Ard Biesheuvel 写道: >>>> On Wed, 30 Nov 2022 at 08:53, Marc Zyngier <maz@kernel.org> wrote: >>>>> On Wed, 30 Nov 2022 02:52:35 +0000, >>>>> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote: >>>>>> Hi, >>>>>> >>>>>> We boot the VM using following commands (with nvdimm on) (qemu >>>>>> version 6.1.50, kernel 6.0-r4): >>>>> How relevant is the presence of the nvdimm? Do you observe the failure >>>>> without this? >>>>> >>>>>> qemu-system-aarch64 -machine >>>>>> virt,kernel_irqchip=on,gic-version=3,nvdimm=on -kernel >>>>>> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios >>>>>> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m >>>>>> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0 >>>>>> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1' >>>>>> -object memory-backend-ram,id=ram1,size=10G -device >>>>>> nvdimm,id=dimm1,memdev=ram1 -device ioh3420,id=root_port1,chassis=1 >>>>>> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1 >>>>>> >>>>>> Then in VM we insmod a module, vmalloc error occurs as follows (kernel >>>>>> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4): >>>>>> >>>>>> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko >>>>>> [ 8.186563] vmap allocation for size 20480 failed: use >>>>>> vmalloc=<size> to increase size >>>>> Have you tried increasing the vmalloc size to check that this is >>>>> indeed the problem? >>>>> >>>>> [...] >>>>> >>>>>> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr: >>>>>> defer initialization to initcall where permitted"). >>>>> I guess you mean commit fc5a89f75d2a instead, right? >>>>> >>>>>> Do you have any idea about the issue? >>>>> I sort of suspect that the nvdimm gets vmap-ed and consumes a large >>>>> portion of the vmalloc space, but you give very little information >>>>> that could help here... >>>>> >>>> Ouch. I suspect what's going on here: that patch defers the >>>> randomization of the module region, so that we can decouple it from >>>> the very early init code. >>>> >>>> Obviously, it is happening too late now, and the randomized module >>>> region is overlapping with a vmalloc region that is in use by the time >>>> the randomization occurs. >>>> >>>> Does the below fix the issue? >>> The issue still occurs, but it seems decrease the probability, before it >>> occured almost every time, after the change, i tried 2-3 times, and it >>> occurs. >>> But i change back "subsys_initcall" to "core_initcall", and i test more >>> than 20 times, and it is still ok. >>> >> Thank you for confirming. I will send out a patch today. >> > ...but before I do that, could you please check whether the change > below fixes your issue as well? Yes, but i can only reply to you tomorrow as other guy is testing on the only environment today. > > diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c > index 6ccc7ef600e7c1e1..c8c205b630da1951 100644 > --- a/arch/arm64/kernel/kaslr.c > +++ b/arch/arm64/kernel/kaslr.c > @@ -20,7 +20,11 @@ > #include <asm/sections.h> > #include <asm/setup.h> > > -u64 __ro_after_init module_alloc_base; > +/* > + * Set a reasonable default for module_alloc_base in case > + * we end up running with module randomization disabled. > + */ > +u64 __ro_after_init module_alloc_base = (u64)_etext - MODULES_VSIZE; > u16 __initdata memstart_offset_seed; > > struct arm64_ftr_override kaslr_feature_override __initdata; > @@ -30,12 +34,6 @@ static int __init kaslr_init(void) > u64 module_range; > u32 seed; > > - /* > - * Set a reasonable default for module_alloc_base in case > - * we end up running with module randomization disabled. > - */ > - module_alloc_base = (u64)_etext - MODULES_VSIZE; > - > if (kaslr_feature_override.val & kaslr_feature_override.mask & 0xf) { > pr_info("KASLR disabled on command line\n"); > return 0; > . > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: regression: insmod module failed in VM with nvdimm on 2022-12-01 12:06 ` chenxiang (M) @ 2022-12-01 12:53 ` Ard Biesheuvel 0 siblings, 0 replies; 14+ messages in thread From: Ard Biesheuvel @ 2022-12-01 12:53 UTC (permalink / raw) To: chenxiang (M) Cc: Marc Zyngier, will, mark.rutland, linux-arm-kernel, chenxiang via, linuxarm On Thu, 1 Dec 2022 at 13:07, chenxiang (M) <chenxiang66@hisilicon.com> wrote: > > > > 在 2022/12/1 19:07, Ard Biesheuvel 写道: > > On Thu, 1 Dec 2022 at 09:07, Ard Biesheuvel <ardb@kernel.org> wrote: > >> On Thu, 1 Dec 2022 at 08:15, chenxiang (M) <chenxiang66@hisilicon.com> wrote: > >>> Hi Ard, > >>> > >>> > >>> 在 2022/11/30 16:18, Ard Biesheuvel 写道: > >>>> On Wed, 30 Nov 2022 at 08:53, Marc Zyngier <maz@kernel.org> wrote: > >>>>> On Wed, 30 Nov 2022 02:52:35 +0000, > >>>>> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote: > >>>>>> Hi, > >>>>>> > >>>>>> We boot the VM using following commands (with nvdimm on) (qemu > >>>>>> version 6.1.50, kernel 6.0-r4): > >>>>> How relevant is the presence of the nvdimm? Do you observe the failure > >>>>> without this? > >>>>> > >>>>>> qemu-system-aarch64 -machine > >>>>>> virt,kernel_irqchip=on,gic-version=3,nvdimm=on -kernel > >>>>>> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios > >>>>>> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m > >>>>>> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0 > >>>>>> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1' > >>>>>> -object memory-backend-ram,id=ram1,size=10G -device > >>>>>> nvdimm,id=dimm1,memdev=ram1 -device ioh3420,id=root_port1,chassis=1 > >>>>>> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1 > >>>>>> > >>>>>> Then in VM we insmod a module, vmalloc error occurs as follows (kernel > >>>>>> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4): > >>>>>> > >>>>>> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko > >>>>>> [ 8.186563] vmap allocation for size 20480 failed: use > >>>>>> vmalloc=<size> to increase size > >>>>> Have you tried increasing the vmalloc size to check that this is > >>>>> indeed the problem? > >>>>> > >>>>> [...] > >>>>> > >>>>>> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr: > >>>>>> defer initialization to initcall where permitted"). > >>>>> I guess you mean commit fc5a89f75d2a instead, right? > >>>>> > >>>>>> Do you have any idea about the issue? > >>>>> I sort of suspect that the nvdimm gets vmap-ed and consumes a large > >>>>> portion of the vmalloc space, but you give very little information > >>>>> that could help here... > >>>>> > >>>> Ouch. I suspect what's going on here: that patch defers the > >>>> randomization of the module region, so that we can decouple it from > >>>> the very early init code. > >>>> > >>>> Obviously, it is happening too late now, and the randomized module > >>>> region is overlapping with a vmalloc region that is in use by the time > >>>> the randomization occurs. > >>>> > >>>> Does the below fix the issue? > >>> The issue still occurs, but it seems decrease the probability, before it > >>> occured almost every time, after the change, i tried 2-3 times, and it > >>> occurs. > >>> But i change back "subsys_initcall" to "core_initcall", and i test more > >>> than 20 times, and it is still ok. > >>> > >> Thank you for confirming. I will send out a patch today. > >> > > ...but before I do that, could you please check whether the change > > below fixes your issue as well? > > Yes, but i can only reply to you tomorrow as other guy is testing on the > only environment today. > That is fine, thanks. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: regression: insmod module failed in VM with nvdimm on 2022-12-01 11:07 ` Ard Biesheuvel 2022-12-01 12:06 ` chenxiang (M) @ 2022-12-02 2:48 ` chenxiang (M) 2022-12-02 13:44 ` Ard Biesheuvel 1 sibling, 1 reply; 14+ messages in thread From: chenxiang (M) @ 2022-12-02 2:48 UTC (permalink / raw) To: Ard Biesheuvel Cc: Marc Zyngier, will, mark.rutland, linux-arm-kernel, chenxiang via, linuxarm Hi Ard, 在 2022/12/1 19:07, Ard Biesheuvel 写道: > On Thu, 1 Dec 2022 at 09:07, Ard Biesheuvel <ardb@kernel.org> wrote: >> On Thu, 1 Dec 2022 at 08:15, chenxiang (M) <chenxiang66@hisilicon.com> wrote: >>> Hi Ard, >>> >>> >>> 在 2022/11/30 16:18, Ard Biesheuvel 写道: >>>> On Wed, 30 Nov 2022 at 08:53, Marc Zyngier <maz@kernel.org> wrote: >>>>> On Wed, 30 Nov 2022 02:52:35 +0000, >>>>> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote: >>>>>> Hi, >>>>>> >>>>>> We boot the VM using following commands (with nvdimm on) (qemu >>>>>> version 6.1.50, kernel 6.0-r4): >>>>> How relevant is the presence of the nvdimm? Do you observe the failure >>>>> without this? >>>>> >>>>>> qemu-system-aarch64 -machine >>>>>> virt,kernel_irqchip=on,gic-version=3,nvdimm=on -kernel >>>>>> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios >>>>>> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m >>>>>> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0 >>>>>> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1' >>>>>> -object memory-backend-ram,id=ram1,size=10G -device >>>>>> nvdimm,id=dimm1,memdev=ram1 -device ioh3420,id=root_port1,chassis=1 >>>>>> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1 >>>>>> >>>>>> Then in VM we insmod a module, vmalloc error occurs as follows (kernel >>>>>> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4): >>>>>> >>>>>> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko >>>>>> [ 8.186563] vmap allocation for size 20480 failed: use >>>>>> vmalloc=<size> to increase size >>>>> Have you tried increasing the vmalloc size to check that this is >>>>> indeed the problem? >>>>> >>>>> [...] >>>>> >>>>>> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr: >>>>>> defer initialization to initcall where permitted"). >>>>> I guess you mean commit fc5a89f75d2a instead, right? >>>>> >>>>>> Do you have any idea about the issue? >>>>> I sort of suspect that the nvdimm gets vmap-ed and consumes a large >>>>> portion of the vmalloc space, but you give very little information >>>>> that could help here... >>>>> >>>> Ouch. I suspect what's going on here: that patch defers the >>>> randomization of the module region, so that we can decouple it from >>>> the very early init code. >>>> >>>> Obviously, it is happening too late now, and the randomized module >>>> region is overlapping with a vmalloc region that is in use by the time >>>> the randomization occurs. >>>> >>>> Does the below fix the issue? >>> The issue still occurs, but it seems decrease the probability, before it >>> occured almost every time, after the change, i tried 2-3 times, and it >>> occurs. >>> But i change back "subsys_initcall" to "core_initcall", and i test more >>> than 20 times, and it is still ok. >>> >> Thank you for confirming. I will send out a patch today. >> > ...but before I do that, could you please check whether the change > below fixes your issue as well? > > diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c > index 6ccc7ef600e7c1e1..c8c205b630da1951 100644 > --- a/arch/arm64/kernel/kaslr.c > +++ b/arch/arm64/kernel/kaslr.c > @@ -20,7 +20,11 @@ > #include <asm/sections.h> > #include <asm/setup.h> > > -u64 __ro_after_init module_alloc_base; > +/* > + * Set a reasonable default for module_alloc_base in case > + * we end up running with module randomization disabled. > + */ > +u64 __ro_after_init module_alloc_base = (u64)_etext - MODULES_VSIZE; > u16 __initdata memstart_offset_seed; > > struct arm64_ftr_override kaslr_feature_override __initdata; > @@ -30,12 +34,6 @@ static int __init kaslr_init(void) > u64 module_range; > u32 seed; > > - /* > - * Set a reasonable default for module_alloc_base in case > - * we end up running with module randomization disabled. > - */ > - module_alloc_base = (u64)_etext - MODULES_VSIZE; > - > if (kaslr_feature_override.val & kaslr_feature_override.mask & 0xf) { > pr_info("KASLR disabled on command line\n"); > return 0; > . We have tested this change, the issue is still and it doesn't fix the issue. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: regression: insmod module failed in VM with nvdimm on 2022-12-02 2:48 ` chenxiang (M) @ 2022-12-02 13:44 ` Ard Biesheuvel 2022-12-15 17:33 ` Thorsten Leemhuis 0 siblings, 1 reply; 14+ messages in thread From: Ard Biesheuvel @ 2022-12-02 13:44 UTC (permalink / raw) To: chenxiang (M) Cc: Marc Zyngier, will, mark.rutland, linux-arm-kernel, chenxiang via, linuxarm On Fri, 2 Dec 2022 at 03:48, chenxiang (M) <chenxiang66@hisilicon.com> wrote: > > Hi Ard, > > > 在 2022/12/1 19:07, Ard Biesheuvel 写道: > > On Thu, 1 Dec 2022 at 09:07, Ard Biesheuvel <ardb@kernel.org> wrote: > >> On Thu, 1 Dec 2022 at 08:15, chenxiang (M) <chenxiang66@hisilicon.com> wrote: > >>> Hi Ard, > >>> > >>> > >>> 在 2022/11/30 16:18, Ard Biesheuvel 写道: > >>>> On Wed, 30 Nov 2022 at 08:53, Marc Zyngier <maz@kernel.org> wrote: > >>>>> On Wed, 30 Nov 2022 02:52:35 +0000, > >>>>> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote: > >>>>>> Hi, > >>>>>> > >>>>>> We boot the VM using following commands (with nvdimm on) (qemu > >>>>>> version 6.1.50, kernel 6.0-r4): > >>>>> How relevant is the presence of the nvdimm? Do you observe the failure > >>>>> without this? > >>>>> > >>>>>> qemu-system-aarch64 -machine > >>>>>> virt,kernel_irqchip=on,gic-version=3,nvdimm=on -kernel > >>>>>> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios > >>>>>> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m > >>>>>> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0 > >>>>>> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1' > >>>>>> -object memory-backend-ram,id=ram1,size=10G -device > >>>>>> nvdimm,id=dimm1,memdev=ram1 -device ioh3420,id=root_port1,chassis=1 > >>>>>> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1 > >>>>>> > >>>>>> Then in VM we insmod a module, vmalloc error occurs as follows (kernel > >>>>>> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4): > >>>>>> > >>>>>> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko > >>>>>> [ 8.186563] vmap allocation for size 20480 failed: use > >>>>>> vmalloc=<size> to increase size > >>>>> Have you tried increasing the vmalloc size to check that this is > >>>>> indeed the problem? > >>>>> > >>>>> [...] > >>>>> > >>>>>> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr: > >>>>>> defer initialization to initcall where permitted"). > >>>>> I guess you mean commit fc5a89f75d2a instead, right? > >>>>> > >>>>>> Do you have any idea about the issue? > >>>>> I sort of suspect that the nvdimm gets vmap-ed and consumes a large > >>>>> portion of the vmalloc space, but you give very little information > >>>>> that could help here... > >>>>> > >>>> Ouch. I suspect what's going on here: that patch defers the > >>>> randomization of the module region, so that we can decouple it from > >>>> the very early init code. > >>>> > >>>> Obviously, it is happening too late now, and the randomized module > >>>> region is overlapping with a vmalloc region that is in use by the time > >>>> the randomization occurs. > >>>> > >>>> Does the below fix the issue? > >>> The issue still occurs, but it seems decrease the probability, before it > >>> occured almost every time, after the change, i tried 2-3 times, and it > >>> occurs. > >>> But i change back "subsys_initcall" to "core_initcall", and i test more > >>> than 20 times, and it is still ok. > >>> > >> Thank you for confirming. I will send out a patch today. > >> > > ...but before I do that, could you please check whether the change > > below fixes your issue as well? > > > > diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c > > index 6ccc7ef600e7c1e1..c8c205b630da1951 100644 > > --- a/arch/arm64/kernel/kaslr.c > > +++ b/arch/arm64/kernel/kaslr.c > > @@ -20,7 +20,11 @@ > > #include <asm/sections.h> > > #include <asm/setup.h> > > > > -u64 __ro_after_init module_alloc_base; > > +/* > > + * Set a reasonable default for module_alloc_base in case > > + * we end up running with module randomization disabled. > > + */ > > +u64 __ro_after_init module_alloc_base = (u64)_etext - MODULES_VSIZE; > > u16 __initdata memstart_offset_seed; > > > > struct arm64_ftr_override kaslr_feature_override __initdata; > > @@ -30,12 +34,6 @@ static int __init kaslr_init(void) > > u64 module_range; > > u32 seed; > > > > - /* > > - * Set a reasonable default for module_alloc_base in case > > - * we end up running with module randomization disabled. > > - */ > > - module_alloc_base = (u64)_etext - MODULES_VSIZE; > > - > > if (kaslr_feature_override.val & kaslr_feature_override.mask & 0xf) { > > pr_info("KASLR disabled on command line\n"); > > return 0; > > . > > We have tested this change, the issue is still and it doesn't fix the issue. > Thanks for the report. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: regression: insmod module failed in VM with nvdimm on 2022-12-02 13:44 ` Ard Biesheuvel @ 2022-12-15 17:33 ` Thorsten Leemhuis 0 siblings, 0 replies; 14+ messages in thread From: Thorsten Leemhuis @ 2022-12-15 17:33 UTC (permalink / raw) To: Ard Biesheuvel, chenxiang (M) Cc: Marc Zyngier, will, mark.rutland, linux-arm-kernel, chenxiang via, linuxarm, regressions Hi, this is your Linux kernel regression tracker. Top-posting for once, to make this easily accessible to everyone. Was there some progress to get this regression resolved? From here it looks stalled, but maybe I missed something. Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) P.S.: As the Linux kernel's regression tracker I deal with a lot of reports and sometimes miss something important when writing mails like this. If that's the case here, don't hesitate to tell me in a public reply, it's in everyone's interest to set the public record straight. On 02.12.22 14:44, Ard Biesheuvel wrote: > On Fri, 2 Dec 2022 at 03:48, chenxiang (M) <chenxiang66@hisilicon.com> wrote: >> >> Hi Ard, >> >> >> 在 2022/12/1 19:07, Ard Biesheuvel 写道: >>> On Thu, 1 Dec 2022 at 09:07, Ard Biesheuvel <ardb@kernel.org> wrote: >>>> On Thu, 1 Dec 2022 at 08:15, chenxiang (M) <chenxiang66@hisilicon.com> wrote: >>>>> Hi Ard, >>>>> >>>>> >>>>> 在 2022/11/30 16:18, Ard Biesheuvel 写道: >>>>>> On Wed, 30 Nov 2022 at 08:53, Marc Zyngier <maz@kernel.org> wrote: >>>>>>> On Wed, 30 Nov 2022 02:52:35 +0000, >>>>>>> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> We boot the VM using following commands (with nvdimm on) (qemu >>>>>>>> version 6.1.50, kernel 6.0-r4): >>>>>>> How relevant is the presence of the nvdimm? Do you observe the failure >>>>>>> without this? >>>>>>> >>>>>>>> qemu-system-aarch64 -machine >>>>>>>> virt,kernel_irqchip=on,gic-version=3,nvdimm=on -kernel >>>>>>>> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios >>>>>>>> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m >>>>>>>> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0 >>>>>>>> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1' >>>>>>>> -object memory-backend-ram,id=ram1,size=10G -device >>>>>>>> nvdimm,id=dimm1,memdev=ram1 -device ioh3420,id=root_port1,chassis=1 >>>>>>>> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1 >>>>>>>> >>>>>>>> Then in VM we insmod a module, vmalloc error occurs as follows (kernel >>>>>>>> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4): >>>>>>>> >>>>>>>> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko >>>>>>>> [ 8.186563] vmap allocation for size 20480 failed: use >>>>>>>> vmalloc=<size> to increase size >>>>>>> Have you tried increasing the vmalloc size to check that this is >>>>>>> indeed the problem? >>>>>>> >>>>>>> [...] >>>>>>> >>>>>>>> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr: >>>>>>>> defer initialization to initcall where permitted"). >>>>>>> I guess you mean commit fc5a89f75d2a instead, right? >>>>>>> >>>>>>>> Do you have any idea about the issue? >>>>>>> I sort of suspect that the nvdimm gets vmap-ed and consumes a large >>>>>>> portion of the vmalloc space, but you give very little information >>>>>>> that could help here... >>>>>>> >>>>>> Ouch. I suspect what's going on here: that patch defers the >>>>>> randomization of the module region, so that we can decouple it from >>>>>> the very early init code. >>>>>> >>>>>> Obviously, it is happening too late now, and the randomized module >>>>>> region is overlapping with a vmalloc region that is in use by the time >>>>>> the randomization occurs. >>>>>> >>>>>> Does the below fix the issue? >>>>> The issue still occurs, but it seems decrease the probability, before it >>>>> occured almost every time, after the change, i tried 2-3 times, and it >>>>> occurs. >>>>> But i change back "subsys_initcall" to "core_initcall", and i test more >>>>> than 20 times, and it is still ok. >>>>> >>>> Thank you for confirming. I will send out a patch today. >>>> >>> ...but before I do that, could you please check whether the change >>> below fixes your issue as well? >>> >>> diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c >>> index 6ccc7ef600e7c1e1..c8c205b630da1951 100644 >>> --- a/arch/arm64/kernel/kaslr.c >>> +++ b/arch/arm64/kernel/kaslr.c >>> @@ -20,7 +20,11 @@ >>> #include <asm/sections.h> >>> #include <asm/setup.h> >>> >>> -u64 __ro_after_init module_alloc_base; >>> +/* >>> + * Set a reasonable default for module_alloc_base in case >>> + * we end up running with module randomization disabled. >>> + */ >>> +u64 __ro_after_init module_alloc_base = (u64)_etext - MODULES_VSIZE; >>> u16 __initdata memstart_offset_seed; >>> >>> struct arm64_ftr_override kaslr_feature_override __initdata; >>> @@ -30,12 +34,6 @@ static int __init kaslr_init(void) >>> u64 module_range; >>> u32 seed; >>> >>> - /* >>> - * Set a reasonable default for module_alloc_base in case >>> - * we end up running with module randomization disabled. >>> - */ >>> - module_alloc_base = (u64)_etext - MODULES_VSIZE; >>> - >>> if (kaslr_feature_override.val & kaslr_feature_override.mask & 0xf) { >>> pr_info("KASLR disabled on command line\n"); >>> return 0; >>> . >> >> We have tested this change, the issue is still and it doesn't fix the issue. >> > > Thanks for the report. > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel #regzbot poke _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: regression: insmod module failed in VM with nvdimm on 2022-11-30 7:53 ` Marc Zyngier 2022-11-30 8:18 ` Ard Biesheuvel @ 2022-12-01 7:01 ` chenxiang (M) 1 sibling, 0 replies; 14+ messages in thread From: chenxiang (M) @ 2022-12-01 7:01 UTC (permalink / raw) To: Marc Zyngier Cc: ardb, will, mark.rutland, linux-arm-kernel, chenxiang via, linuxarm Hi Marc, 在 2022/11/30 15:53, Marc Zyngier 写道: > On Wed, 30 Nov 2022 02:52:35 +0000, > "chenxiang (M)" <chenxiang66@hisilicon.com> wrote: >> Hi, >> >> We boot the VM using following commands (with nvdimm on) (qemu >> version 6.1.50, kernel 6.0-r4): > How relevant is the presence of the nvdimm? Do you observe the failure > without this? We didn't see the failure without it. >> qemu-system-aarch64 -machine >> virt,kernel_irqchip=on,gic-version=3,nvdimm=on -kernel >> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios >> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m >> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0 >> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1' >> -object memory-backend-ram,id=ram1,size=10G -device >> nvdimm,id=dimm1,memdev=ram1 -device ioh3420,id=root_port1,chassis=1 >> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1 >> >> Then in VM we insmod a module, vmalloc error occurs as follows (kernel >> 5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4): >> >> estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko >> [ 8.186563] vmap allocation for size 20480 failed: use >> vmalloc=<size> to increase size > Have you tried increasing the vmalloc size to check that this is > indeed the problem? > > [...] I didn't increase the vmalloc size, but i check the vmall size and i think it is big enough when the issue occurs: estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko [ 4.879899] vmap allocation for size 20480 failed: use vmalloc=<size> to increase size [ 4.880643] insmod: vmalloc error: size 16384, vm_struct allocation failed, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0 [ 4.881802] CPU: 1 PID: 230 Comm: insmod Not tainted 6.1.0-rc4+ #21 [ 4.882414] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [ 4.883082] Call trace: [ 4.883333] dump_backtrace.part.0+0xc4/0xd0 [ 4.883766] show_stack+0x20/0x50 [ 4.884091] dump_stack_lvl+0x68/0x84 [ 4.884450] dump_stack+0x18/0x34 [ 4.884778] warn_alloc+0x11c/0x1bc [ 4.885124] __vmalloc_node_range+0x50c/0x64c [ 4.885553] module_alloc+0xf4/0x100 [ 4.885902] load_module+0x858/0x1e90 [ 4.886265] __do_sys_init_module+0x1c0/0x200 [ 4.886699] __arm64_sys_init_module+0x24/0x30 [ 4.887147] invoke_syscall+0x50/0x120 [ 4.887516] el0_svc_common.constprop.0+0x58/0x190 [ 4.887993] do_el0_svc+0x34/0xc0 [ 4.888327] el0_svc+0x2c/0xb4 [ 4.888631] el0t_64_sync_handler+0xb8/0xbc [ 4.889046] el0t_64_sync+0x19c/0x1a0 [ 4.889423] Mem-Info: [ 4.889639] active_anon:9679 inactive_anon:63094 isolated_anon:0 [ 4.889639] active_file:0 inactive_file:0 isolated_file:0 [ 4.889639] unevictable:0 dirty:0 writeback:0 [ 4.889639] slab_reclaimable:3322 slab_unreclaimable:3082 [ 4.889639] mapped:873 shmem:72569 pagetables:34 [ 4.889639] sec_pagetables:0 bounce:0 [ 4.889639] kernel_misc_reclaimable:0 [ 4.889639] free:416212 free_pcp:4414 free_cma:0 [ 4.893362] Node 0 active_anon:38716kB inactive_anon:252376kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:3492kB dirty:0kB writeback:0kB shmem:290276kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:1904kB pagetables:136kB sec_pagetables:0kB all_unreclaimable? no [ 4.896343] Node 0 DMA free:1664848kB boost:0kB min:22528kB low:28160kB high:33792kB reserved_highatomic:0KB active_anon:38716kB inactive_anon:252376kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:2097152kB managed:2010376kB mlocked:0kB bounce:0kB free_pcp:17704kB local_pcp:3668kB free_cma:0kB [ 4.899097] lowmem_reserve[]: 0 0 0 0 0 [ 4.899466] Node 0 DMA: 2*4kB (UM) 1*8kB (M) 2*16kB (UM) 1*32kB (M) 2*64kB (ME) 1*128kB (U) 2*256kB (ME) 2*512kB (M) 6*1024kB (UME) 5*2048kB (UM) 402*4096kB (M) = 1664848kB [ 4.900865] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [ 4.901648] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=32768kB [ 4.902526] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 4.903354] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=64kB [ 4.904173] 72569 total pagecache pages [ 4.904524] 0 pages in swap cache [ 4.904831] Free swap = 0kB [ 4.905109] Total swap = 0kB [ 4.905407] 524288 pages RAM [ 4.905696] 0 pages HighMem/MovableOnly [ 4.906085] 21694 pages reserved [ 4.906388] 0 pages hwpoisoned insmod: can't insert '/lib/modules/6.1.0-rc4+/hnae3.ko': Cannot allocate memory estuary:/$ insmod /lib/modules/$(uname -r)/hns3.ko [ 4.911599] vmap allocation for size 122880 failed: use vmalloc=<size> to increase size insmod: can't insert '/lib/modules/6.1.0-rc4+/hns3.ko': Cannot allocate memory estuary:/$ insmod /lib/modules/$(uname -r)/hclge.ko [ 4.917761] vmap allocation for size 319488 failed: use vmalloc=<size> to increase size insmod: can't insert '/lib/modules/6.1.0-rc4+/hclge.ko': Cannot allocate memory estuary:/$ insmod /lib/modules/$(uname -r)/hclgevf.ko [ 5.160299] vmap allocation for size 73728 failed: use vmalloc=<size> to increase size insmod: can't insert '/lib/modules/6.1.0-rc4+/hclgevf.ko': Cannot allocate memory estuary:/$ estuary:/$ cat /proc/meminfo MemTotal: 2010376 kB MemFree: 1664848 kB MemAvailable: 1637744 kB Buffers: 0 kB Cached: 290276 kB SwapCached: 0 kB Active: 39456 kB Inactive: 251628 kB Active(anon): 39456 kB Inactive(anon): 251628 kB Active(file): 0 kB Inactive(file): 0 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 860 kB Mapped: 3368 kB Shmem: 290276 kB KReclaimable: 13372 kB Slab: 25732 kB SReclaimable: 13372 kB SUnreclaim: 12360 kB KernelStack: 1872 kB PageTables: 136 kB SecPageTables: 0 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 1005188 kB Committed_AS: 292076 kB VmallocTotal: 133143592960 kB VmallocUsed: 2612 kB VmallocChunk: 0 kB Percpu: 672 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB FileHugePages: 0 kB FilePmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB estuary:/$ > >> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr: >> defer initialization to initcall where permitted"). > I guess you mean commit fc5a89f75d2a instead, right? Right > >> Do you have any idea about the issue? > I sort of suspect that the nvdimm gets vmap-ed and consumes a large > portion of the vmalloc space, but you give very little information > that could help here... > > M. > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: regression: insmod module failed in VM with nvdimm on #forregzbot 2022-11-30 2:52 regression: insmod module failed in VM with nvdimm on chenxiang (M) 2022-11-30 7:53 ` Marc Zyngier @ 2022-11-30 10:10 ` Thorsten Leemhuis 2023-03-03 9:42 ` Linux regression tracking #update (Thorsten Leemhuis) 1 sibling, 1 reply; 14+ messages in thread From: Thorsten Leemhuis @ 2022-11-30 10:10 UTC (permalink / raw) To: regressions; +Cc: linux-arm-kernel, chenxiang via, linuxarm [Note: this mail is primarily send for documentation purposes and/or for regzbot, my Linux kernel regression tracking bot. That's why I removed most or all folks from the list of recipients, but left any that looked like a mailing lists. These mails usually contain '#forregzbot' in the subject, to make them easy to spot and filter out.] [TLDR: I'm adding this regression report to the list of tracked regressions; all text from me you find below is based on a few templates paragraphs you might have encountered already already in similar form.] Hi, this is your Linux kernel regression tracker. On 30.11.22 03:52, chenxiang (M) wrote: > > We boot the VM using following commands (with nvdimm on) (qemu version > 6.1.50, kernel 6.0-r4): > > qemu-system-aarch64 -machine > virt,kernel_irqchip=on,gic-version=3,nvdimm=on -kernel > /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios > /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m > 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0 > ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1' > -object memory-backend-ram,id=ram1,size=10G -device > nvdimm,id=dimm1,memdev=ram1 -device ioh3420,id=root_port1,chassis=1 > -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1 > [...] > We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr: > defer initialization to initcall where permitted"). Thanks for the report. To be sure below issue doesn't fall through the cracks unnoticed, I'm adding it to regzbot, my Linux kernel regression tracking bot: #regzbot ^introduced fc5a89f75d2a #regzbot title arm64: kaslr: vmalloc error when boothing a VM with nvdimm #regzbot ignore-activity This isn't a regression? This issue or a fix for it are already discussed somewhere else? It was fixed already? You want to clarify when the regression started to happen? Or point out I got the title or something else totally wrong? Then just reply -- ideally with also telling regzbot about it, as explained here: https://linux-regtracking.leemhuis.info/tracked-regression/ Reminder for developers: When fixing the issue, add 'Link:' tags pointing to the report (the mail this one replies to), as explained for in the Linux kernel's documentation; above webpage explains why this is important for tracked regressions. Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) P.S.: As the Linux kernel's regression tracker I deal with a lot of reports and sometimes miss something important when writing mails like this. If that's the case here, don't hesitate to tell me in a public reply, it's in everyone's interest to set the public record straight. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: regression: insmod module failed in VM with nvdimm on #forregzbot 2022-11-30 10:10 ` regression: insmod module failed in VM with nvdimm on #forregzbot Thorsten Leemhuis @ 2023-03-03 9:42 ` Linux regression tracking #update (Thorsten Leemhuis) 0 siblings, 0 replies; 14+ messages in thread From: Linux regression tracking #update (Thorsten Leemhuis) @ 2023-03-03 9:42 UTC (permalink / raw) To: regressions; +Cc: linux-arm-kernel, chenxiang via, linuxarm [TLDR: This mail in primarily relevant for Linux regression tracking. A change or fix related to the regression discussed in this thread was posted or applied, but it did not use a Link: tag to point to the report, as Linus and the documentation call for. Things happen, no worries -- but now the regression tracking bot needs to be told manually about the fix. See link in footer if these mails annoy you.] On 30.11.22 11:10, Thorsten Leemhuis wrote: > > On 30.11.22 03:52, chenxiang (M) wrote: >> >> We boot the VM using following commands (with nvdimm on) (qemu version >> 6.1.50, kernel 6.0-r4): >> >> qemu-system-aarch64 -machine >> virt,kernel_irqchip=on,gic-version=3,nvdimm=on -kernel >> /home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios >> /root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m >> 2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0 >> ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1' >> -object memory-backend-ram,id=ram1,size=10G -device >> nvdimm,id=dimm1,memdev=ram1 -device ioh3420,id=root_port1,chassis=1 >> -device vfio-pci,host=7d:01.0,id=net0,bus=root_port1 >> [...] >> We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr: >> defer initialization to initcall where permitted"). > > Thanks for the report. To be sure below issue doesn't fall through the > cracks unnoticed, I'm adding it to regzbot, my Linux kernel regression > tracking bot: > > #regzbot ^introduced fc5a89f75d2a > #regzbot title arm64: kaslr: vmalloc error when boothing a VM with nvdimm > #regzbot ignore-activity #regzbot fix: arm64: kaslr: don't pretend KASLR is enabled if offset < MIN_KIMG_ALIGN #regzbot ignore-activity Backstory: https://lore.kernel.org/lkml/CAMj1kXGWEaQXoKj=DzG9XpVGi4t5zfE-RSG0BodVL-b47nsj-A@mail.gmail.com/ https://lore.kernel.org/all/20230223204101.1500373-1-ardb@kernel.org/ Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) -- Everything you wanna know about Linux kernel regression tracking: https://linux-regtracking.leemhuis.info/about/#tldr That page also explains what to do if mails like this annoy you. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2023-03-03 9:43 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-11-30 2:52 regression: insmod module failed in VM with nvdimm on chenxiang (M) 2022-11-30 7:53 ` Marc Zyngier 2022-11-30 8:18 ` Ard Biesheuvel 2022-12-01 7:15 ` chenxiang (M) 2022-12-01 8:07 ` Ard Biesheuvel 2022-12-01 11:07 ` Ard Biesheuvel 2022-12-01 12:06 ` chenxiang (M) 2022-12-01 12:53 ` Ard Biesheuvel 2022-12-02 2:48 ` chenxiang (M) 2022-12-02 13:44 ` Ard Biesheuvel 2022-12-15 17:33 ` Thorsten Leemhuis 2022-12-01 7:01 ` chenxiang (M) 2022-11-30 10:10 ` regression: insmod module failed in VM with nvdimm on #forregzbot Thorsten Leemhuis 2023-03-03 9:42 ` Linux regression tracking #update (Thorsten Leemhuis)
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).