From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0596DC11F64 for ; Thu, 1 Jul 2021 09:08:07 +0000 (UTC) Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 108186146B for ; Thu, 1 Jul 2021 09:08:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 108186146B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=gerhold.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id EC24C82C08; Thu, 1 Jul 2021 11:08:03 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=gerhold.net Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gerhold.net header.i=@gerhold.net header.b="F9khGYCM"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 7F9348322E; Thu, 1 Jul 2021 11:08:01 +0200 (CEST) Received: from mo4-p00-ob.smtp.rzone.de (mo4-p00-ob.smtp.rzone.de [81.169.146.163]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 10E5582BF6 for ; Thu, 1 Jul 2021 11:07:57 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=gerhold.net Authentication-Results: phobos.denx.de; spf=none smtp.mailfrom=stephan@gerhold.net DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1625130475; s=strato-dkim-0002; d=gerhold.net; h=Message-ID:Subject:Cc:To:From:Date:Cc:Date:From:Subject:Sender; bh=wmTrCsoydGclfuAO0v/+Uu3LCoap9ArblnlVe7tOCaI=; b=F9khGYCMxe+II2TZpVb4uloenWVvvvWPqPBqvmOR1Lwu6uKc79ViNEgquX39kO+7Mw NLGI/0eHfFyai+f0NVQQuj5Ze1yVZzoj9N4nOEndeZEnSFmvOlA5221cUboxk5I8659F a5+l9/zxKc9yUVQ0zgKBAbbmaFmieWDfoDgn+Qwj8eHt5FXezY3lDE90XqbI6dmfR/2F D5Te1W3JprzlNl/LHTHSLySQx+X+aRQ0u5+PwL7WgVvyiOUN2H3cvV8mJ0X9OwO7kvBu N5VQqxWKmNFN2KyCrul3w/2xlUtjuNVjpglEdM33psmSX32RjxE6dMaE/Wfvq9hFsCut D3Fg== Authentication-Results: strato.com; dkim=none X-RZG-AUTH: ":P3gBZUipdd93FF5ZZvYFPugejmSTVR2nRPhVOQ/OcYgojyw4j34+u26zEodhPgRDZ8j5IcrDBg==" X-RZG-CLASS-ID: mo00 Received: from gerhold.net by smtp.strato.de (RZmta 47.28.1 SBL|AUTH) with ESMTPSA id Y070ccx6197t76N (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256 bits)) (Client did not present a certificate); Thu, 1 Jul 2021 11:07:55 +0200 (CEST) Date: Thu, 1 Jul 2021 11:07:45 +0200 From: Stephan Gerhold To: u-boot@lists.denx.de, Ramon Fried Cc: Jorge Ramirez-Ortiz , Nicolas Dechesne Subject: [RFC] Load U-Boot without LK on DragonBoard 410c (+ DB820c?) Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.2 at phobos.denx.de X-Virus-Status: Clean Hi! at the moment the U-Boot ports for both DragonBoard 410c and 820c are designed to be loaded as an Android boot image after Qualcomm's LK bootloader. This is simple to set up but LK is redundant in this case, since everything done by LK can be also done directly by U-Boot. Dropping LK entirely would have at least the following advantages: - Easier installation/board code (no need for Android boot images) - (Slightly) faster boot - Boot directly in 64-bit without a round trip to 32-bit for LK This was not possible so far because of some unsolved problems. For clarity I try to describe them together with some background here, but I want to apologize for the long text. It's all quite complicated. :) 1. "Signing" 64-bit U-Boot ========================== Ramon already tried to eliminate LK for DB410c 3 years ago [1]. One of the open problems back then was to have a proper "signing" tool with 64-bit support. The firmware expects an ELF image with a few Qualcomm-specific ELF headers. Normally this is used for secure boot setups. This is not used on DragonBoards, but the firmware still insists on having a dummy (self-signed) certificate chain in the ELF images. Linaro uses signlk [2] to sign their builds of LK. It looks like Nicolas extended it with ELF64 support after Ramon's mail [3]. However, for some reason signlk literally works only for LK for me. I tried to "sign" U-Boot and some other firmware, but everything except LK is always rejected with the following message on boot: B - 1031113 - Error code 302e at boot_config.c Line 296 I tried to track down the issue in the source code for quite some time but did not manage to find the problem. Perhaps it's some subtle mistake with some of the ELF modifications, I'm not sure. (For some reason, signlk makes subtle changes to all of the existing ELF headers...) After reading about the image format myself I decided to try to make my own "signing" tool, qtestsign: https://github.com/msm8916-mainline/qtestsign It's based on a mixture of the specification [4] and some missing bits taken from signlk, put in a simple and clean Python tool. I still don't know what exactly qtestsign does different, but unlike signlk it can successfully "sign" U-Boot and all other firmware from DragonBoard 410c. [1]: https://lore.kernel.org/u-boot/CA+Kvs9kS=DbJKNAixk_3tz+3iWnRaSP0gJdZ8eKrzasKOr6wcw@mail.gmail.com/ [2]: https://git.linaro.org/landing-teams/working/qualcomm/signlk.git/ [3]: https://git.linaro.org/landing-teams/working/qualcomm/signlk.git/commit/?id=1f61c03322c3728f35b3f0cd4ff04f73522f1e67 [4]: https://www.qualcomm.com/media/documents/files/secure-boot-and-image-authentication-technical-overview-v1-0.pdf My solution ----------- Now we have all we need to install U-Boot without LK. For DragonBoard 410c the following steps end up in the U-Boot prompt without going through LK: 1. Change dragonboard410c_defconfig as follows: -CONFIG_SYS_TEXT_BASE=0x80080000 +CONFIG_SYS_TEXT_BASE=0x8F600000 +CONFIG_OF_EMBED=y (I discuss this at the end of the mail) 2. $ make 3. Sign the ELF image: $ qtestsign.py aboot /u-boot [5] 4. Flash "/u-boot-test-signed.mbn" to the "aboot" partition [5]: https://github.com/msm8916-mainline/qtestsign 2. Linux gets stuck when loaded by 64-bit U-Boot without LK =========================================================== This should work well enough to get the U-Boot prompt on serial. However, once you load Linux you will likely notice a problem: [ 0.059043] smp: Bringing up secondary CPUs ... [ 5.120691] CPU1: failed to come online [ 10.246760] CPU2: failed to come online [ 15.372848] CPU3: failed to come online [ 15.406275] CPU: All CPU(s) started at EL1 ... [ 16.185527] genirq: irq_chip msmgpio did not update eff. affinity mask of irq 79 Board freezes forever. :( My investigations have shown this is a bug in the PSCI implementation on DB410c (part of the TrustZone/"tz" firmware). Shortly said, since we have never done the 32-bit -> 64-bit switch in LK, the PSCI implementation seems to believe we are still running in 32-bit mode and starts all further CPUs in 32-bit mode. The other CPU cores crash immediately when coming up and CPU 0 hangs once CPU idle suspends it for the first time. I have described this problem together with a workaround in detail here: https://github.com/msm8916-mainline/qhypstub#boot-flow The idea is to execute the TZ syscall to switch from 32-bit -> 64-bit even though we are already running in 64-bit mode. This will make the PSCI implementation aware that we want all further CPU cores booted in 64-bit mode as well. My solution ----------- The workaround is applied automatically when using my open-source "hyp" firmware replacement qhypstub: https://github.com/msm8916-mainline/qhypstub As a bonus, both U-Boot and Linux start in EL2, making it possible to use virtualization (e.g. KVM in Linux). $ git clone https://github.com/msm8916-mainline/qhypstub.git $ cd qhypstub $ make CROSS_COMPILE=aarch64-linux-gnu- $ qtestsign.py hyp qhypstub.elf # Flash "qhypstub-test-signed.mbn" to "hyp" partition and reboot. Now it works: [ 0.063411] CPU1: Booted secondary processor 0x0000000001 [0x410fd030] [ 0.064184] CPU2: Booted secondary processor 0x0000000002 [0x410fd030] [ 0.064906] CPU3: Booted secondary processor 0x0000000003 [0x410fd030] [ 0.123032] CPU: All CPU(s) started at EL2 [ 0.448743] kvm [1]: Hyp mode initialized successfully ... And with that U-Boot is fully working as far as I can tell. (I have only tested serial, SD card and USB so far. If something is broken, it's likely some missing register initialization that should be ported from LK/Linux...) 3. Remaining open questions =========================== I still see 3 questions that we need to discuss: 1. This is a quite fundamental change. Can we just make it to dragonboard410c_defconfig? Does it make sense to keep the old setup with LK? When would it be used? 2. Workaround for PSCI bug: I'm not sure if we want to make qhypstub [6] a requirement for U-Boot. On the one hand it's open-source, solves the problem nicely without changes in U-Boot and provides EL2 additionally. I'm also not aware of any problem/disadvantage when using it (if you find a problem, please let me know!). But I realize it's unofficial. If we want to support using Qualcomm's "hyp" firmware as well I could try porting the PSCI workaround from qhypstub to U-Boot. It should be ~10 lines of ARM64 assembly [7] placed e.g. in board/qualcomm/dragonboard410c/head.S. However, I will need to make sure to detect if U-Boot was started in EL2 by qhypstub because otherwise doing the workaround twice will conflict and U-Boot might demote itself back to EL1. 3. CONFIG_OF_EMBED: There is a big warning about this in the build log: "This option should only be used for debugging purposes. Please use CONFIG_OF_SEPARATE for boards in mainline." The important part here is that we need an ELF image with both U-Boot and the DTB. CONFIG_OF_EMBED is convenient for that because we can just use the ELF image built by the linker and it already contains the DTB. If CONFIG_OF_EMBED is really so bad it might be possible to build a new boot image based on "u-boot-dtb.bin" (which is U-Boot with DTB appended). I'm not sure if this is really much better though. Bonus question: Could something similar also work for DB820c? I don't have one myself but I think a similar setup short also work on it. If someone is interested in testing this I would be happy to help. :) Thanks for reading! Stephan [6]: https://github.com/msm8916-mainline/qhypstub [7]: https://github.com/msm8916-mainline/qhypstub/blob/c9c3fd0f66ea60032812b06b51da39f25e678638/qhypstub.s#L197-L204