Re: arm64 regression in kernel 5.12 related to the (n)VHE

From: Florian Fainelli <f.fainelli@gmail.com>
To: "Marc Zyngier" <maz@kernel.org>, "Rafał Miłecki" <zajec5@gmail.com>
Cc: David Brazdil <dbrazdil@google.com>,
	Catalin Marinas <catalin.marinas@arm.com>,
	Will Deacon <will@kernel.org>,
	linux-arm-kernel <linux-arm-kernel@lists.infradead.org>,
	Mark Rutland <mark.rutland@arm.com>,
	Ard Biesheuvel <ardb@kernel.org>, Marco Elver <elver@google.com>,
	BCM Kernel Feedback <bcm-kernel-feedback-list@broadcom.com>,
	Florian Fainelli <f.fainelli@gmail.com>
Subject: Re: arm64 regression in kernel 5.12 related to the (n)VHE
Date: Thu, 12 Aug 2021 01:33:10 -0700	[thread overview]
Message-ID: <a02714d9-9446-2834-eb84-1790c640a58a@gmail.com> (raw)
In-Reply-To: <878s179jsq.wl-maz@kernel.org>

On 8/12/2021 12:57 AM, Marc Zyngier wrote:
> On Thu, 12 Aug 2021 08:32:02 +0100,
> Rafał Miłecki <zajec5@gmail.com> wrote:
>>
>> On 12.08.2021 08:51, Marc Zyngier wrote:
>>> Interestingly, all your CPUs are booting at EL2. Which is great.  Can
>>> you try and enable KVM on your existing 5.10 kernel? Just selecting
>>> CONFIG_KVM should be enough. Does it boot correctly with KVM enabled?
>>>
>>> My suspicion is that the firmware doesn't set SCR_EL3.HCE, and that
>>> the HVC instruction UNDEFs at EL1. That would be bad news.
>>
>> Interesting! I had to enable CONFIG_VIRTUALIZATION and CONFIG_NET first.
>> First I verified kernel built with those options still boots. It does.
>>
>> Then I enabled CONFIG_KVM and kernel seems to hang around switching from
>> bootconsole to the console.
>>
>> Starting program at 0x0000000000080000
>> /memory = 0x40000000
>> WARNING: Node's property /reserved-memory/dt_reserved_buffer is not defined
>> WARNING: Node's property /reserved-memory/dt_reserved_flow is not defined
>> WARNING: Node's property /reserved-memory/dt_reserved_dhd2 is not defined
>> Booting Linux on physical CPU 0x0000000000 [0x420f1000]
>> Linux version 5.11.22-g0453a426c37b (rmilecki@localhost.localdomain) (aarch64-buildroot-linux-uclibc-gcc.br_real (Buildroot -g91617ed) 9.3.0, GNU ld (GNU Binutils) 2.33.1) #8 SMP Thu Aug 12 09:25:55 CEST 2021
>> Machine model: Asus GT-AC5300
>> earlycon: bcm63xx_uart0 at MMIO 0x00000000ff800640 (options '')
>> printk: bootconsole [bcm63xx_uart0] enabled
>> efi: UEFI not found.
>> [Firmware Bug]: Kernel image misaligned at boot, please fix your bootloader!
>> Zone ranges:
>>    DMA      [mem 0x0000000000000000-0x000000003fffffff]
>>    DMA32    empty
>>    Normal   empty
>> Movable zone start for each node
>> Early memory node ranges
>>    node   0: [mem 0x0000000000000000-0x000000003fffffff]
>> Initmem setup node 0 [mem 0x0000000000000000-0x000000003fffffff]
>> percpu: Embedded 18 pages/cpu s43904 r0 d29824 u73728
>> Detected VIPT I-cache on CPU0
>> CPU features: detected: ARM erratum 843419
>> Built 1 zonelists, mobility grouping on.  Total pages: 258048
>> Kernel command line: earlycon=bcm63xx_uart,0xff800640
>> Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
>> Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
>> mem auto-init: stack:off, heap alloc:off, heap free:off
>> Memory: 1019556K/1048576K available (4352K kernel code, 678K rwdata, 860K rodata, 2496K init, 232K bss, 29020K reserved, 0K cma-reserved)
>> SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
>> rcu: Hierarchical RCU implementation.
>> rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
>> NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
>> GIC: Using split EOI/Deactivate mode
>> random: get_random_bytes called from start_kernel+0x33c/0x52c with crng_init=0
>> arch_timer: cp15 timer(s) running at 50.00MHz (phys).
>> clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns
>> sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns
>> Console: colour dummy device 80x25
>> printk: console [tty0] enabled
>> printk: bootconsole [bcm63xx_uart0] disabled
>>
>>
>> (Unless it's a false conclusion and CONFIG_KVM just breaks console
>> somehow)
> 
> No, that's because you don't pass the right console to your
> kernel. Add something like "console=ttyS0,115200" to the kernel
> command line, which will show what you are missing, as well as stop
> the double-logging.
> 
> Anyway, the fact that it stops booting when you enable KVM confirms my
> suspicion. The firmware on this system is probably crap enough not to
> enable HVC. Let's confirm it further: please apply the patch below on
> top of mainline and tell me that it now boots fine...
> 
> Are you in a position where you can actually fix the firmware? Or is
> it some closed-source blob?

This is a closed source blob, but you ought to be able to load a custom 
EL3 stub that could fix things up, in premise at least.

> 
> Broadcom folks: can you lease check whether the firmware on this
> system correctly configures SCR_EL3.HCE?

Let me confirm that with the responsible folks.
-- 
Florian

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel