From mboxrd@z Thu Jan 1 00:00:00 1970 From: Greg Kurz Date: Fri, 25 May 2018 10:48:31 +0000 Subject: Re: [PATCH] KVM: PPC: Book3S PR: Enable use on POWER9 inside HPT-mode guests Message-Id: <20180525124831.54278f72@bahia.lan> List-Id: References: <20180519055638.GA24787@fergus.ozlabs.ibm.com> In-Reply-To: <20180519055638.GA24787@fergus.ozlabs.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: kvm-ppc@vger.kernel.org On Thu, 24 May 2018 09:12:09 +1000 Paul Mackerras wrote: > On Wed, May 23, 2018 at 07:04:21PM +0200, Greg Kurz wrote: > > On Sat, 19 May 2018 15:56:38 +1000 > > Paul Mackerras wrote: > > > > > This relaxes the restriction on using PR KVM on POWER9. The existing > > > code does work inside a guest partition running in HPT mode, because > > > hypercalls such as H_ENTER use the old HPTE format, not the new > > > format used by POWER9, and so no change to PR KVM's HPT manipulation > > > code is required. PR KVM will still refuse to run if the kernel is > > > using radix translation or if it is running bare-metal. > > > > > > Signed-off-by: Paul Mackerras > > > --- > > > > Paul, > > > > I have built a 4.16.0 kernel + this patch and booted the L1 guest > > with "disable_radix=on". I could then successfully boot a L2 guest, > > using the same kernel for simplicity. Both guests using identical > > fedora28 images. So it seems to be working at first sight. > > > > > > But, if I boot the L2 guest with the default fedora28 kernel, ie > > 4.16.9-300.fc28.ppc64le, the L2 guest hangs. > > > > OF stdout device is: /vdevice/vty@71000000 > > Preparing to boot Linux version 4.16.9-300.fc28.ppc64le (mockbuild@buildvm-ppc64le-05.ppc.fedoraproject.org) (gcc version 8.1.1 20180502 (Red Hat 8.1.1-1) (GCC)) #1 SMP Thu May 17 04:31:32 UTC 2018 > > Detected machine type: 0000000000000101 > > command line: BOOT_IMAGE=/boot/vmlinuz-4.16.9-300.fc28.ppc64le root=UUID"128c5c-30b1-4e0a-ac16-95853df31131 ro rhgb console=hvc0 early_printk LANG=en_US.UTF-8 > > Max number of cores passed to firmware: 1024 (NR_CPUS = 1024) > > Calling ibm,client-architecture-support... done > > memory layout at init: > > memory_limit : 0000000000000000 (16 MB aligned) > > alloc_bottom : 0000000004e70000 > > alloc_top : 0000000030000000 > > alloc_top_hi : 0000000100000000 > > rmo_top : 0000000030000000 > > ram_top : 0000000100000000 > > instantiating rtas at 0x000000002fff0000... done > > prom_hold_cpus: skipped > > copying OF device tree... > > Building dt strings... > > Building dt structure... > > Device tree strings 0x0000000004e80000 -> 0x0000000004e80aaf > > Device tree struct 0x0000000004e90000 -> 0x0000000004ea0000 > > Quiescing Open Firmware ... > > Booting Linux via __start() @ 0x0000000002000000 ... > > > > (qemu) p $pc > > 0xc000000000026aa0 > > (qemu) p $lr > > 0xc000000000119ff4 > > > > # addr2line -e /usr/lib/debug/lib/modules/4.16.9-300.fc28.ppc64le/vmlinux 0xc000000000026aa0 > > /usr/src/debug/kernel-4.16.fc28/linux-4.16.9-300.fc28.ppc64le/./arch/powerpc/include/asm/time.h:115 > > > > # addr2line -e /usr/lib/debug/lib/modules/4.16.9-300.fc28.ppc64le/vmlinux 0xc000000000119ff4 > > /usr/src/debug/kernel-4.16.fc28/linux-4.16.9-300.fc28.ppc64le/kernel/panic.c:300 > > > > ie, the final mdelay(PANIC_TIMER_STEP) in panic(). > > > > Not sure how to debug this further, any suggestion is welcome :) > > I suggest you find the address of log_buf from System.map, read that > via the qemu command line (log_buf is a pointer), then dump the memory > it points to, so you can see the panic message. > Hi Paul, Thanks for your suggestion. I could reproduced the problem if I boot the L2 guest with an upstream kernel (commit d7b66b4ab034). I've tried to dump the log_buf but things didn't go well: $ grep 'd log_buf' System.map c000000001304f08 d log_buf_len c000000001304f10 d log_buf (qemu) x 0xc000000001304f08 c000000001304f08: Cannot access memory Since 4.16.0 works, I could bisect down to: commit dbfcf3cb9c681aa0c5d0bb46068f98d5b1823dd3 Author: Paul Mackerras Date: Thu Feb 16 16:03:39 2017 +1100 powerpc/64: Call H_REGISTER_PROC_TBL when running as a HPT guest on POWER9 The hcall is handled by QEMU, which then calls the KVM_PPC_CONFIGURE_V3_MMU ioctl, which fails since PR KVM doesn't implement it, and H_REGISTER_PROC_TBL fails with H_PARAMETER. The panic hence come from... static int pseries_lpar_register_process_table(unsigned long base, unsigned long page_size, unsigned long table_size) { . . . for (;;) { rc = plpar_hcall_norets(H_REGISTER_PROC_TBL, flags, base, page_size, table_size); if (!H_IS_LONG_BUSY(rc)) break; mdelay(get_longbusy_msecs(rc)); } if (rc != H_SUCCESS) { pr_err("Failed to register process table (rc=%ld)\n", rc); BUG(); ^^^ here. The changelog of commit dbfcf3cb9c68 reads: " If the hypervisor is able to support both radix and HPT guests, it would be entitled to defer allocation of the HPT until the H_REGISTER_PROC_TBL call" But in our case, the hypervisor is QEMU/PR KVM in a L1 guest booted with radix disabled. It is hence not "entitled to defer allocation of the HPT", and QEMU allocates one during initial machine reset. If I patch QEMU to make H_REGISTER_PROC_TBL a nop when KVM_CAP_PPC_MMU_RADIX returns 0, then the L2 kernel boots like a charm. So I'm wondering if the guest should even call H_REGISTER_PROC_TBL in this case, since there's nothing to do ? Also, peeking into PAPR, I see that H_REGISTER_PROC_TBL is mandatory only "If the platform supports the In-Memory Table Translation Option", which isn't the case here. This is supposed to be advertised through the "hcall-imtt" function set in the OF property "ibm,hypertas-functions" in the /rtas node. I guess a correct behavior would be for QEMU to advertise "hcall-imtt" when it supports both radix and hash, and the kernel should only call H_REGISTER_PROC_TBL if it is available. Of course, neither QEMU, nor the kernel seem to care about "hcall-imtt" today... so I guess the easier way is to fix H_REGISTER_PROC_TBL in QEMU. > Another thing to try would be to do the same test on a POWER8. > No surprise, it continues to work on a POWER8, since: /* * On POWER9, we need to do a H_REGISTER_PROC_TBL hcall * to inform the hypervisor that we wish to use the HPT. */ if (cpu_has_feature(CPU_FTR_ARCH_300)) register_process_table(0, 0, 0); > Paul. Cheers, -- Greg