Arm64 crash while online/offline memory sections

Arm64 crash while online/offline memory sections

[Qian Cai (QUIC)]

Reverting the patchset "Allocate memmap from hotadded memory (per device)" [1] from today's linux-next fixed a crash while online/offline memory sections.

[1] https://lore.kernel.org/linux-mm/20210421102701.25051-1-osalvador@suse.de/

[  162.327720][ T1694] kernel BUG at include/linux/mm.h:1383!
[  162.333695][ T1694] Internal error: Oops - BUG: 0 [#1] SMP
[  162.339181][ T1694] Modules linked in: loop processor efivarfs ip_tables x_tables ext4 mbcache jbd2 dm_mod igb nvme i2c_algo_bit mlx5_core i2c_core nvme_core firmware_class
[  162.354604][ T1694] CPU: 13 PID: 1694 Comm: ranbug Not tainted 5.12.0-next-20210524+ #4
[  162.362601][ T1694] Hardware name: MiTAC RAPTOR EV-883832-X3-0001/RAPTOR, BIOS 1.6 06/28/2020
[  162.371116][ T1694] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
[  162.377811][ T1694] pc : memory_subsys_offline+0x1f8/0x250
[  162.383295][ T1694] lr : memory_subsys_offline+0x1f8/0x250
[  162.388773][ T1694] sp : ffff80002458f8e0
[  162.392773][ T1694] x29: ffff80002458f8e0 x28: ffff800010914d30 x27: 0000000000000000
[  162.400602][ T1694] x26: 0000000000002000 x25: 1fffe00002550401 x24: ffff000012a82008
[  162.408431][ T1694] x23: fffffc0000000000 x22: 0000000000008000 x21: 0000000000000001
[  162.416259][ T1694] x20: ffffffffffffffff x19: ffff000012a82018 x18: ffff0008527b6a70
[  162.424086][ T1694] x17: 0000000000000000 x16: 0000000000000007 x15: 00000000000000c8
[  162.431914][ T1694] x14: 0000000000000000 x13: ffff800011c6eea4 x12: ffff60136ceb8574
[  162.439742][ T1694] x11: 1fffe0136ceb8573 x10: ffff60136ceb8573 x9 : dfff800000000000
[  162.447570][ T1694] x8 : ffff009b675c2b9b x7 : 0000000000000001 x6 : ffff009b675c2b98
[  162.455398][ T1694] x5 : 00009fec93147a8d x4 : ffff009b675c2b98 x3 : 1fffe0010a4f6c09
[  162.463226][ T1694] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000034
[  162.471054][ T1694] Call trace:
[  162.474186][ T1694]  memory_subsys_offline+0x1f8/0x250
[  162.479318][ T1694]  device_offline+0x154/0x1d8
[  162.483844][ T1694]  online_store+0xa4/0x118
[  162.488107][ T1694]  dev_attr_store+0x44/0x78
[  162.492457][ T1694]  sysfs_kf_write+0xe8/0x138
[  162.496896][ T1694]  kernfs_fop_write_iter+0x26c/0x3d0
[  162.502028][ T1694]  new_sync_write+0x2bc/0x4f8
[  162.506552][ T1694]  vfs_write+0x718/0xc88
[  162.510643][ T1694]  ksys_write+0xf8/0x1e0
[  162.514732][ T1694]  __arm64_sys_write+0x74/0xa8
[  162.519342][ T1694]  invoke_syscall.constprop.0+0x78/0x1e8
[  162.524824][ T1694]  do_el0_svc+0xe4/0x298
[  162.528914][ T1694]  el0_svc+0x20/0x30
[  162.532658][ T1694]  el0_sync_handler+0xb0/0xb8
[  162.537181][ T1694]  el0_sync+0x178/0x180
[  162.541187][ T1694] Code: f00033e1 91318021 91090021 97e38d8b (d4210000)
[  162.547968][ T1694] ---[ end trace 2a1964462a219f20 ]---
[  162.553273][ T1694] Kernel panic - not syncing: Oops - BUG: Fatal exception
[  162.560250][ T1694] SMP: stopping secondary CPUs
[  162.564871][ T1694] Kernel Offset: disabled
[  162.569045][ T1694] CPU features: 0x00000251,20000846
[  162.574089][ T1694] Memory Limit: none
[  162.577849][ T1694] ---[ end Kernel panic - not syncing: Oops - BUG: Fatal exception ]---

Occasionally, it failed like this,

[  119.319732][ T1387] Unable to handle kernel paging request at virtual address dfff800000000000
[  119.328575][ T1387] Mem abort info:
[  119.332073][ T1387]   ESR = 0x96000004
[  119.335879][ T1387]   EC = 0x25: DABT (current EL), IL = 32 bits
[  119.341895][ T1387]   SET = 0, FnV = 0
[  119.345697][ T1387]   EA = 0, S1PTW = 0
[  119.349542][ T1387] Data abort info:
[  119.353121][ T1387]   ISV = 0, ISS = 0x00000004
[  119.358387][ T1387]   CM = 0, WnR = 0
[  119.362055][ T1387] [dfff800000000000] address between user and kernel address ranges
[  119.370166][ T1387] Internal error: Oops: 96000004 [#1] SMP
[  119.375739][ T1387] Modules linked in: loop processor efivarfs ip_tables x_tables ext4 mbcache jbd2 dm_mod igb i2c_algo_bit nvme mlx5_core i2c_core nvme_core firmware_class
[  119.391164][ T1387] CPU: 2 PID: 1387 Comm: ranbug Not tainted 5.13.0-rc3+ #16
[  119.398294][ T1387] Hardware name: MiTAC RAPTOR EV-883832-X3-0001/RAPTOR, BIOS 1.6 06/28/2020
[  119.406810][ T1387] pstate: 20000005 (nzCv daif -PAN -UAO -TCO BTYPE=--)
[  119.413504][ T1387] pc : hex_dump_to_buffer+0x3ac/0x9f0
[  119.418728][ T1387] lr : print_hex_dump+0x10c/0x1d8
[  119.423600][ T1387] sp : ffff800021b4f3a0
[  119.427600][ T1387] x29: ffff800021b4f3a0 x28: dfff800000000000 x27: ffff800021b4f500
[  119.435429][ T1387] x26: 0000000000000083 x25: fffffffffffffffe x24: 0000000000000008
[  119.443257][ T1387] x23: 0000000000000045 x22: 0000000000000083 x21: 0000000000000020
[  119.451084][ T1387] x20: 0000000000000000 x19: ffff800021b4f500 x18: ffff0008650187e8
[  119.458912][ T1387] x17: 0000000000000000 x16: 0000000000000007 x15: 00000000000000c8
[  119.466740][ T1387] x14: 0000000000000000 x13: ffff800011c6eea4 x12: 000000000000f1f1
[  119.474568][ T1387] x11: 0000000000000005 x10: ffff8000112eec80 x9 : 00000000f3f3f3f3
[  119.482396][ T1387] x8 : 0000000000000010 x7 : 0000000000000000 x6 : fffffffffffffffe
[  119.490224][ T1387] x5 : 0000000000000083 x4 : 0000000000000000 x3 : 1fffffffffffffff
[  119.498051][ T1387] x2 : ffff8000112eecc0 x1 : 0000000000000006 x0 : 0000000000000000
[  119.505879][ T1387] Call trace:
[  119.509011][ T1387]  hex_dump_to_buffer+0x3ac/0x9f0
[  119.513884][ T1387]  print_hex_dump+0x10c/0x1d8
[  119.518407][ T1387]  __dump_page+0x8c8/0xba8
[  119.522672][ T1387]  dump_page+0x20/0x50
[  119.526587][ T1387]  memory_subsys_offline+0x1f8/0x250
[  119.531722][ T1387]  device_offline+0x154/0x1d8
[  119.536247][ T1387]  online_store+0xa4/0x118
[  119.540511][ T1387]  dev_attr_store+0x44/0x78
[  119.544860][ T1387]  sysfs_kf_write+0xe8/0x138
[  119.549298][ T1387]  kernfs_fop_write_iter+0x26c/0x3d0
[  119.554429][ T1387]  new_sync_write+0x2bc/0x4f8
[  119.558953][ T1387]  vfs_write+0x718/0xc88
[  119.563042][ T1387]  ksys_write+0xf8/0x1e0
[  119.567130][ T1387]  __arm64_sys_write+0x74/0xa8
[  119.571740][ T1387]  invoke_syscall.constprop.0+0x78/0x1e8
[  119.577222][ T1387]  do_el0_svc+0xe4/0x298
[  119.581311][ T1387]  el0_svc+0x20/0x30
[  119.585054][ T1387]  el0_sync_handler+0xb0/0xb8
[  119.589578][ T1387]  el0_sync+0x178/0x180
[  119.593583][ T1387] Code: 1200080b 120008c1 d343fc00 38fc6864 (38fc6803)
[  119.600364][ T1387] ---[ end trace b03c958bc1accb66 ]---
[  119.605669][ T1387] Kernel panic - not syncing: Oops: Fatal exception
[  119.612124][ T1387] SMP: stopping secondary CPUs
[  119.616747][ T1387] Kernel Offset: disabled
[  119.620923][ T1387] CPU features: 0x00000251,20000846
[  119.625968][ T1387] Memory Limit: none
[  119.629728][ T1387] ---[ end Kernel panic - not syncing: Oops: Fatal exception ]---

Re: Arm64 crash while online/offline memory sections

[David Hildenbrand]

On 25.05.21 17:36, Qian Cai (QUIC) wrote:
> Reverting the patchset "Allocate memmap from hotadded memory (per device)" [1] from today's linux-next fixed a crash while online/offline memory sections.

Do we know which patch in particular is problematic?


-- 
Thanks,

David / dhildenb

RE: Arm64 crash while online/offline memory sections

[Qian Cai (QUIC)]

> -----Original Message-----
> From: David Hildenbrand <david@redhat.com>
> Sent: Tuesday, May 25, 2021 11:40 AM
> To: Qian Cai (QUIC) <quic_qiancai@quicinc.com>; Oscar Salvador
> <osalvador@suse.de>
> Cc: Andrew Morton <akpm@linux-foundation.org>; Anshuman Khandual
> <anshuman.khandual@arm.com>; Vlastimil Babka <vbabka@suse.cz>;
> Michal Hocko <mhocko@kernel.org>; Linux Memory Management List
> <linux-mm@kvack.org>; Linux Kernel Mailing List <linux-
> kernel@vger.kernel.org>
> Subject: Re: Arm64 crash while online/offline memory sections
> 
> On 25.05.21 17:36, Qian Cai (QUIC) wrote:
> > Reverting the patchset "Allocate memmap from hotadded memory (per
> device)" [1] from today's linux-next fixed a crash while online/offline
> memory sections.
> 
> Do we know which patch in particular is problematic?

I don't know yet. It could be messy to dive into a patchset which usually has less clear boundaries, but I'll try to dig a bit more.

RE: Arm64 crash while online/offline memory sections

[Qian Cai (QUIC)]

> Do we know which patch in particular is problematic?

Okay, the winner is "mm,memory_hotplug: Allocate memmap from the added memory range".

https://lore.kernel.org/linux-mm/20210421102701.25051-5-osalvador@suse.de/

That and the rest of the series are the minimal I have to revert to fix the crash.

Re: Arm64 crash while online/offline memory sections

[Oscar Salvador]

On Tue, May 25, 2021 at 05:57:34PM +0000, Qian Cai (QUIC) wrote:
> > Do we know which patch in particular is problematic?
> 
> Okay, the winner is "mm,memory_hotplug: Allocate memmap from the added memory range".
> 
> https://lore.kernel.org/linux-mm/20210421102701.25051-5-osalvador@suse.de/

Ok, which means that is irrelevant to having it enabled, as the latter
patch of that series actualy enables it for arm64.
Can you work out where exactly the crash happens?

I will have a look into it tomorrow.

Thanks for reporting.

-- 
Oscar Salvador
SUSE L3

Re: Arm64 crash while online/offline memory sections

[David Hildenbrand]

On 25.05.21 20:00, Oscar Salvador wrote:
> On Tue, May 25, 2021 at 05:57:34PM +0000, Qian Cai (QUIC) wrote:
>>> Do we know which patch in particular is problematic?
>>
>> Okay, the winner is "mm,memory_hotplug: Allocate memmap from the added memory range".
>>
>> https://lore.kernel.org/linux-mm/20210421102701.25051-5-osalvador@suse.de/
> 
> Ok, which means that is irrelevant to having it enabled, as the latter
> patch of that series actualy enables it for arm64.
> Can you work out where exactly the crash happens?
> 
> I will have a look into it tomorrow.
> 
> Thanks for reporting.
> 

I assume the following will work:

diff --git a/drivers/base/memory.c b/drivers/base/memory.c
index b31b3af5c490..6e661d106e96 100644
--- a/drivers/base/memory.c
+++ b/drivers/base/memory.c
@@ -218,14 +218,15 @@ static int memory_block_offline(struct memory_block *mem)
         struct zone *zone;
         int ret;
  
-       zone = page_zone(pfn_to_page(start_pfn));
-
         /*
          * Unaccount before offlining, such that unpopulated zone and kthreads
          * can properly be torn down in offline_pages().
          */
-       if (nr_vmemmap_pages)
+       if (nr_vmemmap_pages) {
+               /* Hotplugged memory has no holes. */
+               zone = page_zone(pfn_to_page(start_pfn));
                 adjust_present_page_count(zone, -nr_vmemmap_pages);
+       }
  
         ret = offline_pages(start_pfn + nr_vmemmap_pages,
                             nr_pages - nr_vmemmap_pages);


We must not touch pfn_to_page(start_pfn) if it might be a memory hole.
offline_pages() will make sure there are no holes, but that's too late.

-- 
Thanks,

David / dhildenb

RE: Arm64 crash while online/offline memory sections

[Qian Cai (QUIC)]

> -----Original Message-----
> From: David Hildenbrand <david@redhat.com>
> Sent: Tuesday, May 25, 2021 2:12 PM
> I assume the following will work:
> 
> diff --git a/drivers/base/memory.c b/drivers/base/memory.c
> index b31b3af5c490..6e661d106e96 100644
> --- a/drivers/base/memory.c
> +++ b/drivers/base/memory.c
> @@ -218,14 +218,15 @@ static int memory_block_offline(struct memory_block *mem)
>          struct zone *zone;
>          int ret;
> 
> -       zone = page_zone(pfn_to_page(start_pfn));
> -
>          /*
>           * Unaccount before offlining, such that unpopulated zone and kthreads
>           * can properly be torn down in offline_pages().
>           */
> -       if (nr_vmemmap_pages)
> +       if (nr_vmemmap_pages) {
> +               /* Hotplugged memory has no holes. */
> +               zone = page_zone(pfn_to_page(start_pfn));
>                  adjust_present_page_count(zone, -nr_vmemmap_pages);
> +       }
> 
>          ret = offline_pages(start_pfn + nr_vmemmap_pages,
>                              nr_pages - nr_vmemmap_pages);
> 
> 
> We must not touch pfn_to_page(start_pfn) if it might be a memory hole.
> offline_pages() will make sure there are no holes, but that's too late.

Good catch, David. This patch works well.

Re: Arm64 crash while online/offline memory sections

[Oscar Salvador]

On Tue, May 25, 2021 at 08:12:22PM +0200, David Hildenbrand wrote:
> diff --git a/drivers/base/memory.c b/drivers/base/memory.c
> index b31b3af5c490..6e661d106e96 100644
> --- a/drivers/base/memory.c
> +++ b/drivers/base/memory.c
> @@ -218,14 +218,15 @@ static int memory_block_offline(struct memory_block *mem)
>         struct zone *zone;
>         int ret;
> -       zone = page_zone(pfn_to_page(start_pfn));
> -
>         /*
>          * Unaccount before offlining, such that unpopulated zone and kthreads
>          * can properly be torn down in offline_pages().
>          */
> -       if (nr_vmemmap_pages)
> +       if (nr_vmemmap_pages) {
> +               /* Hotplugged memory has no holes. */
> +               zone = page_zone(pfn_to_page(start_pfn));
>                 adjust_present_page_count(zone, -nr_vmemmap_pages);
> +       }
>         ret = offline_pages(start_pfn + nr_vmemmap_pages,
>                             nr_pages - nr_vmemmap_pages);
> 
> 
> We must not touch pfn_to_page(start_pfn) if it might be a memory hole.
> offline_pages() will make sure there are no holes, but that's too late.

Yeah, definitely. I somehow didn't think about holes when writing that.
Thanks for catching it David. Would you be so kind to send a patch?

Thanks!


-- 
Oscar Salvador
SUSE L3