linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed
From: Sean Anderson <sean.anderson@seco.com>
To: David Hildenbrand <david@redhat.com>
Cc: akpm@linux-foundation.org, alexander.h.duyck@linux.intel.com,
	dave.hansen@intel.com, haiyangz@microsoft.com, kys@microsoft.com,
	linux-acpi@vger.kernel.org, linux-hyperv@vger.kernel.org,
	linux-kernel@vger.kernel.org, linux-mm@kvack.org,
	mgorman@techsingularity.net, mhocko@kernel.org, mhocko@suse.com,
	osalvador@suse.de, pankaj.gupta.linux@gmail.com,
	richard.weiyang@linux.alibaba.com, rppt@kernel.org,
	sthemmin@microsoft.com, vbabka@suse.cz, wei.liu@kernel.org,
	willy@infradead.org, xen-devel@lists.xenproject.org
Subject: Re: [PATCH v2 4/5] mm/page_alloc: place pages to tail in __free_pages_core()
Date: Fri, 10 Sep 2021 19:06:45 -0400	[thread overview]
Message-ID: <c9fc3cc7-2c6e-dc6f-b5f6-0b48fb49aedb@seco.com> (raw)
In-Reply-To: <528e8d9c-b148-30ec-d8cc-3dd072eaa7f2@redhat.com>



On 9/8/21 2:21 AM, David Hildenbrand wrote:
> On 08.09.21 00:40, Sean Anderson wrote:
>> Hi David,
>>
>> This patch breaks booting on my custom Xilinx ZynqMP board. Booting
>> fails just after/during GIC initialization:
>>
>> [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
>> [    0.000000] Linux version 5.14.0 (sean@plantagenet) (aarch64-linux-gnu-gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #251 SMP Tue Sep 7 18:11:50 EDT 2021
>> [    0.000000] Machine model: xlnx,zynqmp
>> [    0.000000] earlycon: cdns0 at MMIO 0x00000000ff010000 (options '115200n8')
>> [    0.000000] printk: bootconsole [cdns0] enabled
>> [    0.000000] efi: UEFI not found.
>> [    0.000000] Zone ranges:
>> [    0.000000]   DMA32    [mem 0x0000000000000000-0x00000000ffffffff]
>> [    0.000000]   Normal   [mem 0x0000000100000000-0x000000087fffffff]
>> [    0.000000] Movable zone start for each node
>> [    0.000000] Early memory node ranges
>> [    0.000000]   node   0: [mem 0x0000000000000000-0x000000007fefffff]
>> [    0.000000]   node   0: [mem 0x0000000800000000-0x000000087fffffff]
>> [    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000087fffffff]
>> [    0.000000] On node 0, zone Normal: 256 pages in unavailable ranges
>> [    0.000000] cma: Reserved 1000 MiB at 0x0000000041400000
>> [    0.000000] psci: probing for conduit method from DT.
>> [    0.000000] psci: PSCIv1.1 detected in firmware.
>> [    0.000000] psci: Using standard PSCI v0.2 function IDs
>> [    0.000000] psci: MIGRATE_INFO_TYPE not supported.
>> [    0.000000] psci: SMC Calling Convention v1.1
>> [    0.000000] percpu: Embedded 19 pages/cpu s46752 r0 d31072 u77824
>> [    0.000000] Detected VIPT I-cache on CPU0
>> [    0.000000] CPU features: detected: ARM erratum 845719
>> [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 1033987
>> [    0.000000] Kernel command line: earlycon clk_ignore_unused root=/dev/mmcblk0p2 rootwait rw cma=1000M
>> [    0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
>> [    0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
>> [    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
>> [    0.000000] software IO TLB: mapped [mem 0x000000003d400000-0x0000000041400000] (64MB)
>> [    0.000000] Memory: 3023384K/4193280K available (4288K kernel code, 514K rwdata, 1200K rodata, 896K init, 187K bss, 145896K reserved, 1024000K cma-reserved)
>> [    0.000000] rcu: Hierarchical RCU implementation.
>> [    0.000000] rcu:     RCU event tracing is enabled.
>> [    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
>> [    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
>> [    0.000000] GIC: Adjusting CPU interface base to 0x00000000f902f000
>> [    0.000000] Root IRQ handler: gic_handle_irq
>> [    0.000000] GIC: Using split EOI/Deactivate mode
>>
>> and I bisected it to this patch. Applying the following patch (for 5.14)
>> fixes booting again:
>
> Hi Sean,
>
> unfortunately that patch most likely (with 99.9999% confidence) revealed another latent BUG in your setup.

I suspected as much; however even after inspecting this patch I was
unsure what I should investigate further.

>
> Some memory that shouldn't be handed to the buddy as free memory is getting now allocated earlier than later, resulting in that issue.
>
>
> I had all different kinds of reports, but they were mostly
>
> a) Firmware bugs that result in uncached memory getting exposed to the buddy, resulting in severe performance degradation such that the system will no longer boot. [3]
>
> I wrote kstream [1] to be run under the old kernel, to identify these.
>
> b) BUGs that result in unsuitable memory getting exposed to either the buddy or devices, resulting in errors during device initialization. [6]
>
> c) Use after free BUGs.
>
> Exposing memory, such as used for ACPI tables, to the buddy as free memory although it's still in use. [4]
>
> d) Hypervisor BUGs
>
> The last report (heavy performance degradation) was due to a BUG in dpdk. [2]
>
>
> What the exact symptoms you're experiencing? Really slow boot/stall? Then it could be a) and kstream might help.

Well, as it turns out DDR chips of half the correct size were installed.
This caused the upper half of memory to alias to the lower half. As it
happened, due to some lucky circumstances this didn't initially cause
problems. Sorry for the noise.

--Sean


  reply	other threads:[~2021-09-10 23:06 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-05 12:15 [PATCH v2 0/5] mm: place pages to the freelist tail when onlining and undoing isolation David Hildenbrand
2020-10-05 12:15 ` [PATCH v2 1/5] mm/page_alloc: convert "report" flag of __free_one_page() to a proper flag David Hildenbrand
2020-10-05 12:15 ` [PATCH v2 2/5] mm/page_alloc: place pages to tail in __putback_isolated_page() David Hildenbrand
2020-10-20 17:18   ` Vlastimil Babka
2020-10-05 12:15 ` [PATCH v2 3/5] mm/page_alloc: move pages to tail in move_to_free_list() David Hildenbrand
2020-10-06 12:12   ` Michal Hocko
2020-10-20 17:20   ` Vlastimil Babka
2020-10-05 12:15 ` [PATCH v2 4/5] mm/page_alloc: place pages to tail in __free_pages_core() David Hildenbrand
2021-09-07 22:40   ` Sean Anderson
2021-09-08  6:21     ` David Hildenbrand
2021-09-10 23:06       ` Sean Anderson [this message]
2020-10-05 12:15 ` [PATCH v2 5/5] mm/memory_hotplug: update comment regarding zone shuffling David Hildenbrand
2020-10-20 17:21   ` Vlastimil Babka
2020-10-21 10:58   ` Pankaj Gupta

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=c9fc3cc7-2c6e-dc6f-b5f6-0b48fb49aedb@seco.com \
    --to=sean.anderson@seco.com \
    --cc=akpm@linux-foundation.org \
    --cc=alexander.h.duyck@linux.intel.com \
    --cc=dave.hansen@intel.com \
    --cc=david@redhat.com \
    --cc=haiyangz@microsoft.com \
    --cc=kys@microsoft.com \
    --cc=linux-acpi@vger.kernel.org \
    --cc=linux-hyperv@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mgorman@techsingularity.net \
    --cc=mhocko@kernel.org \
    --cc=mhocko@suse.com \
    --cc=osalvador@suse.de \
    --cc=pankaj.gupta.linux@gmail.com \
    --cc=richard.weiyang@linux.alibaba.com \
    --cc=rppt@kernel.org \
    --cc=sthemmin@microsoft.com \
    --cc=vbabka@suse.cz \
    --cc=wei.liu@kernel.org \
    --cc=willy@infradead.org \
    --cc=xen-devel@lists.xenproject.org \
    --subject='Re: [PATCH v2 4/5] mm/page_alloc: place pages to tail in __free_pages_core()' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).