From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ed1-f42.google.com (mail-ed1-f42.google.com [209.85.208.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8AE926FC7 for ; Thu, 2 Mar 2023 14:03:26 +0000 (UTC) Received: by mail-ed1-f42.google.com with SMTP id x3so5026881edb.10 for ; Thu, 02 Mar 2023 06:03:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=h/JXa3jl6N7Sx/fIVr6/rpxA4dj4ykaps8uQIwSRFHI=; b=SQe9A8zf+XRjs48CSJ+h4GnRYESyOz6AHJ2g25k/lwPj0Py8kP4rG8hhbJK3PRK/Ys g3JfMqTzkjAS07GFOOSo5vWwYNY4RjnyegQ/OwnuY/mIwkndxf/tRl69oSIjYCFEsha9 1uivUu44qu8TIqf1FwmyDphbHfzYdzuMHmHW0+vQiB1MI+CoTV1HvTPaMBmotfy/pIXM 3Ya/DW90qOpCzTz0iMBDoR/16Ny+HfrUtO9T+8TUzhnEeRvYG92vR7b848tcOs8uZSCk vTIm7M9BChxeqRZaVnDa7PfzMTULinMLJXWwlP6QFt4Pu4gBnY88Tnv+m57e1yTd1QRA McPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=h/JXa3jl6N7Sx/fIVr6/rpxA4dj4ykaps8uQIwSRFHI=; b=dK8gC2ZZY9RCTunU4wXoewV+83fJ4c5pQ5zTSx6WkipAQMtaZktxgD5lDAQkV7lRAm mxhGWSLwa2Fha9nmeSe+I3AYFYkXcAvge7btFgrUoZTq+a/CzG3NJiI65/X5wCmoNfgy 5tc1n02uahpSDDF1U7Y9geDakwCPkzjK1EPMzhCQ3EJjeLKr7snPo48Dh8BsQWJWURNG n/txATeFslxCppKcQ/bGcbcGVCR99pK3j86WU9FriCFcz7zTZSaOJ96vwTTlc9cNitJ5 xNP0HdDjUItcJYuNwSZS/dKgkm+vvnDmZ9q3hAxQY5gntWkrZRK4+i/FeQ9oITe/WDOU esHg== X-Gm-Message-State: AO0yUKU1/EAcukOMaxNo4yTAIqesPxTb76FK6O3AavuK1+oosvtWQccV x4x4NBy7EIBsFGDf1eeYLYw= X-Google-Smtp-Source: AK7set/gtIv/d5+qF714NjY0nEzkNgdHznl1CeO4qmEfZDHvPZB9UdsHo5ShI+zFg/5R6pNVkFnOVQ== X-Received: by 2002:a05:6402:18a:b0:4ac:c3c0:24d7 with SMTP id r10-20020a056402018a00b004acc3c024d7mr10632959edv.42.1677765804566; Thu, 02 Mar 2023 06:03:24 -0800 (PST) Received: from [192.168.5.8] (PC-176-101-165-146.tvk-net.pl. [176.101.165.146]) by smtp.gmail.com with ESMTPSA id t19-20020a50d713000000b004af73333d6esm6981406edi.53.2023.03.02.06.03.23 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 02 Mar 2023 06:03:24 -0800 (PST) Message-ID: <7a0e256d-3ce1-3218-c930-ed518a679b8b@gmail.com> Date: Thu, 2 Mar 2023 15:03:22 +0100 Precedence: bulk X-Mailing-List: linux-coco@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.7.1 Subject: Re: [RFC kvmtool 18/31] arm64: Populate initial realm contents Content-Language: en-US To: Suzuki K Poulose Cc: Alexandru Elisei , Andrew Jones , Christoffer Dall , Fuad Tabba , Jean-Philippe Brucker , Joey Gouly , Marc Zyngier , Mark Rutland , Oliver Upton , Paolo Bonzini , Quentin Perret , Steven Price , Thomas Huth , Will Deacon , Zenghui Yu , linux-coco@lists.linux.dev, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org References: <20230127112248.136810-1-suzuki.poulose@arm.com> <20230127113932.166089-1-suzuki.poulose@arm.com> <20230127113932.166089-19-suzuki.poulose@arm.com> From: Piotr Sawicki In-Reply-To: <20230127113932.166089-19-suzuki.poulose@arm.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Hi, > From: Alexandru Elisei > > Populate the realm memory with the initial contents, which include > the device tree blob, the kernel image, and initrd, if specified, > or the firmware image. > > Populating an image in the realm involves two steps: > a) Mark the IPA area as RAM - INIT_IPA_REALM > b) Load the contents into the IPA - POPULATE_REALM > > Wherever we know the actual size of an image in memory, we make > sure the "memory area" is initialised to RAM. > e.g., Linux kernel image size from the header which includes the bss etc. > The "file size" on disk for the Linux image is much smaller. > We mark the region of size Image.header.size as RAM (a), from the kernel > load address. And load the Image file into the memory (b) above. > At the moment we only detect the Arm64 Linux Image header format. > > Since we're already touching the code that copies the > initrd in guest memory, let's do a bit of cleaning and remove a > useless local variable. > > Signed-off-by: Alexandru Elisei > [ Make sure the Linux kernel image area is marked as RAM ] > Signed-off-by: Suzuki K Poulose > --- > arm/aarch32/include/asm/realm.h | 3 + > arm/aarch64/include/asm/realm.h | 3 + > arm/aarch64/realm.c | 112 ++++++++++++++++++++++++++++++++ > arm/fdt.c | 6 ++ > arm/kvm.c | 20 ++++-- > include/linux/kernel.h | 1 + > 6 files changed, 140 insertions(+), 5 deletions(-) > > diff --git a/arm/aarch32/include/asm/realm.h b/arm/aarch32/include/asm/realm.h > index 5aca6cca..fcff0e55 100644 > --- a/arm/aarch32/include/asm/realm.h > +++ b/arm/aarch32/include/asm/realm.h > @@ -6,5 +6,8 @@ > #include "kvm/kvm.h" > > static inline void kvm_arm_realm_create_realm_descriptor(struct kvm *kvm) {} > +static inline void kvm_arm_realm_populate_kernel(struct kvm *kvm) {} > +static inline void kvm_arm_realm_populate_initrd(struct kvm *kvm) {} > +static inline void kvm_arm_realm_populate_dtb(struct kvm *kvm) {} > > #endif /* ! __ASM_REALM_H */ > diff --git a/arm/aarch64/include/asm/realm.h b/arm/aarch64/include/asm/realm.h > index e176f15f..6e760ac9 100644 > --- a/arm/aarch64/include/asm/realm.h > +++ b/arm/aarch64/include/asm/realm.h > @@ -6,5 +6,8 @@ > #include "kvm/kvm.h" > > void kvm_arm_realm_create_realm_descriptor(struct kvm *kvm); > +void kvm_arm_realm_populate_kernel(struct kvm *kvm); > +void kvm_arm_realm_populate_initrd(struct kvm *kvm); > +void kvm_arm_realm_populate_dtb(struct kvm *kvm); > > #endif /* ! __ASM_REALM_H */ > diff --git a/arm/aarch64/realm.c b/arm/aarch64/realm.c > index fc7f8d6a..eddccece 100644 > --- a/arm/aarch64/realm.c > +++ b/arm/aarch64/realm.c > @@ -1,5 +1,7 @@ > #include "kvm/kvm.h" > > +#include > +#include > #include > > > @@ -80,3 +82,113 @@ void kvm_arm_realm_create_realm_descriptor(struct kvm *kvm) > if (ioctl(kvm->vm_fd, KVM_ENABLE_CAP, &rme_create_rd) < 0) > die_perror("KVM_CAP_RME(KVM_CAP_ARM_RME_CREATE_RD)"); > } > + > +static void realm_init_ipa_range(struct kvm *kvm, u64 start, u64 size) > +{ > + struct kvm_cap_arm_rme_init_ipa_args init_ipa_args = { > + .init_ipa_base = start, > + .init_ipa_size = size > + }; > + struct kvm_enable_cap rme_init_ipa_realm = { > + .cap = KVM_CAP_ARM_RME, > + .args[0] = KVM_CAP_ARM_RME_INIT_IPA_REALM, > + .args[1] = (u64)&init_ipa_args > + }; > + > + if (ioctl(kvm->vm_fd, KVM_ENABLE_CAP, &rme_init_ipa_realm) < 0) > + die("unable to intialise IPA range for Realm %llx - %llx (size %llu)", > + start, start + size, size); > + > +} > + > +static void __realm_populate(struct kvm *kvm, u64 start, u64 size) > +{ > + struct kvm_cap_arm_rme_populate_realm_args populate_args = { > + .populate_ipa_base = start, > + .populate_ipa_size = size > + }; > + struct kvm_enable_cap rme_populate_realm = { > + .cap = KVM_CAP_ARM_RME, > + .args[0] = KVM_CAP_ARM_RME_POPULATE_REALM, > + .args[1] = (u64)&populate_args > + }; > + > + if (ioctl(kvm->vm_fd, KVM_ENABLE_CAP, &rme_populate_realm) < 0) > + die("unable to populate Realm memory %llx - %llx (size %llu)", > + start, start + size, size); > +} > + > +static void realm_populate(struct kvm *kvm, u64 start, u64 size) > +{ > + realm_init_ipa_range(kvm, start, size); > + __realm_populate(kvm, start, size); > +} > + > +static bool is_arm64_linux_kernel_image(void *header) > +{ > + struct arm64_image_header *hdr = header; > + > + return memcmp(&hdr->magic, ARM64_IMAGE_MAGIC, sizeof(hdr->magic)) == 0; > +} > + > +static ssize_t arm64_linux_kernel_image_size(void *header) > +{ > + struct arm64_image_header *hdr = header; > + > + if (is_arm64_linux_kernel_image(header)) > + return le64_to_cpu(hdr->image_size); > + die("Not arm64 Linux kernel Image"); > +} > + > +void kvm_arm_realm_populate_kernel(struct kvm *kvm) > +{ > + u64 start, end, mem_size; > + void *header = guest_flat_to_host(kvm, kvm->arch.kern_guest_start); > + > + start = ALIGN_DOWN(kvm->arch.kern_guest_start, SZ_4K); > + end = ALIGN(kvm->arch.kern_guest_start + kvm->arch.kern_size, SZ_4K); > + > + if (is_arm64_linux_kernel_image(header)) > + mem_size = arm64_linux_kernel_image_size(header); > + else > + mem_size = end - start; > + > + realm_init_ipa_range(kvm, start, mem_size); > + __realm_populate(kvm, start, end - start); > +} > + > +void kvm_arm_realm_populate_initrd(struct kvm *kvm) > +{ > + u64 kernel_end, start, end; > + > + kernel_end = ALIGN(kvm->arch.kern_guest_start + kvm->arch.kern_size, SZ_4K); > + start = ALIGN_DOWN(kvm->arch.initrd_guest_start, SZ_4K); > + /* > + * Because we align the initrd to 4 bytes, it is theoretically possible > + * for the start of the initrd to overlap with the same page where the > + * kernel ends. > + */ > + if (start < kernel_end) > + start = kernel_end; > + end = ALIGN(kvm->arch.initrd_guest_start + kvm->arch.initrd_size, SZ_4K); > + if (end > start) > + realm_populate(kvm, start, end - start); > +} > + > +void kvm_arm_realm_populate_dtb(struct kvm *kvm) > +{ > + u64 initrd_end, start, end; > + > + initrd_end = ALIGN(kvm->arch.initrd_guest_start + kvm->arch.initrd_size, SZ_4K); > + start = ALIGN_DOWN(kvm->arch.dtb_guest_start, SZ_4K); > + /* > + * Same situation as with the initrd, but now it is the DTB which is > + * overlapping with the last page of the initrd, because the initrd is > + * populated first. > + */ > + if (start < initrd_end) > + start = initrd_end; > + end = ALIGN(kvm->arch.dtb_guest_start + FDT_MAX_SIZE, SZ_4K); > + if (end > start) > + realm_populate(kvm, start, end - start); > +} > diff --git a/arm/fdt.c b/arm/fdt.c > index 286ccadf..762a604d 100644 > --- a/arm/fdt.c > +++ b/arm/fdt.c > @@ -7,6 +7,8 @@ > #include "arm-common/gic.h" > #include "arm-common/pci.h" > > +#include > + > #include > > #include > @@ -231,6 +233,10 @@ static int setup_fdt(struct kvm *kvm) > > if (kvm->cfg.arch.dump_dtb_filename) > dump_fdt(kvm->cfg.arch.dump_dtb_filename, fdt_dest); > + > + if (kvm->cfg.arch.is_realm) > + kvm_arm_realm_populate_dtb(kvm); > + > return 0; > } > late_init(setup_fdt); > diff --git a/arm/kvm.c b/arm/kvm.c > index acb627b2..57c5b5f7 100644 > --- a/arm/kvm.c > +++ b/arm/kvm.c > @@ -6,6 +6,7 @@ > #include "kvm/fdt.h" > > #include "arm-common/gic.h" > +#include > > #include > > @@ -167,6 +168,9 @@ bool kvm__arch_load_kernel_image(struct kvm *kvm, int fd_kernel, int fd_initrd, > pr_debug("Loaded kernel to 0x%llx (%llu bytes)", > kvm->arch.kern_guest_start, kvm->arch.kern_size); I've noticed that multiple calling of the measurement test from the kvm-unit-tests suite results in different Realm Initial Measurements, although the kernel image is always the same. After short investigation, I've found that the RIM starts being different while populating the last 4kB chunk of the kernel image. The issue occurs when the image size is not aligned to the page size (4kB). After zeroing the unused area of the last chunk, the measurements become repeatable. > + if (kvm->cfg.arch.is_realm) > + kvm_arm_realm_populate_kernel(kvm); > + > /* > * Now load backwards from the end of memory so the kernel > * decompressor has plenty of space to work with. First up is > @@ -188,7 +192,6 @@ bool kvm__arch_load_kernel_image(struct kvm *kvm, int fd_kernel, int fd_initrd, > /* ... and finally the initrd, if we have one. */ > if (fd_initrd != -1) { > struct stat sb; > - unsigned long initrd_start; > > if (fstat(fd_initrd, &sb)) > die_perror("fstat"); > @@ -199,7 +202,6 @@ bool kvm__arch_load_kernel_image(struct kvm *kvm, int fd_kernel, int fd_initrd, > if (pos < kernel_end) > die("initrd overlaps with kernel image."); > > - initrd_start = guest_addr; > file_size = read_file(fd_initrd, pos, limit - pos); > if (file_size == -1) { > if (errno == ENOMEM) > @@ -208,11 +210,13 @@ bool kvm__arch_load_kernel_image(struct kvm *kvm, int fd_kernel, int fd_initrd, > die_perror("initrd read"); > } > > - kvm->arch.initrd_guest_start = initrd_start; > + kvm->arch.initrd_guest_start = guest_addr; > kvm->arch.initrd_size = file_size; > pr_debug("Loaded initrd to 0x%llx (%llu bytes)", > - kvm->arch.initrd_guest_start, > - kvm->arch.initrd_size); > + kvm->arch.initrd_guest_start, kvm->arch.initrd_size); > + > + if (kvm->cfg.arch.is_realm) > + kvm_arm_realm_populate_initrd(kvm); > } else { > kvm->arch.initrd_size = 0; > } > @@ -269,6 +273,8 @@ bool kvm__load_firmware(struct kvm *kvm, const char *firmware_filename) > > /* Kernel isn't loaded by kvm, point start address to firmware */ > kvm->arch.kern_guest_start = fw_addr; > + kvm->arch.kern_size = fw_sz; > + > pr_debug("Loaded firmware to 0x%llx (%zd bytes)", > kvm->arch.kern_guest_start, fw_sz); > > @@ -283,6 +289,10 @@ bool kvm__load_firmware(struct kvm *kvm, const char *firmware_filename) > kvm->arch.dtb_guest_start, > kvm->arch.dtb_guest_start + FDT_MAX_SIZE); > > + if (kvm->cfg.arch.is_realm) > + /* We hijack the kernel fields to describe the firmware. */ > + kvm_arm_realm_populate_kernel(kvm); > + > return true; > } > > diff --git a/include/linux/kernel.h b/include/linux/kernel.h > index 6c22f1c0..25f19c20 100644 > --- a/include/linux/kernel.h > +++ b/include/linux/kernel.h > @@ -9,6 +9,7 @@ > > #define DIV_ROUND_UP(n,d) (((n) + (d) - 1) / (d)) > > +#define ALIGN_DOWN(x,a) __ALIGN_MASK(x - (typeof(x))((a) - 1),(typeof(x))(a)-1) > #define ALIGN(x,a) __ALIGN_MASK(x,(typeof(x))(a)-1) > #define __ALIGN_MASK(x,mask) (((x)+(mask))&~(mask)) > #define IS_ALIGNED(x, a) (((x) & ((typeof(x))(a) - 1)) == 0) Kind regards, Piotr Sawicki From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6E3F5C678D4 for ; Thu, 2 Mar 2023 14:04:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To:Subject: MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=dVhqSQq0rQN4gebeQS5f7TRM9CcyowoJIkOkSXVrooU=; b=RyGghRg2KkYZk7 Cpa+G+oA0Db4PJFX1nwogp3Hb1P+6LnUvKG2J9f39QlrUrfilomLvtBRGaiHI3PwPnvAUr5CS2PwE k2uo5UGvYK2c+wGs7Ci56UB9ct/wM6/VVzbKsW/sibo3TGPBbJHK4t7Dfvjpt6V3Mns4yMcuddETP JiRFgqckEtmFoswnfTDIvzr8Ib72GpCWIp+Dlisvgabrh+Vtt2ldA10t2i2XIuqdM/HeWb2xaeLtZ UmzZIP0TLJS0fihcoSzsgbbRcIKrjd9MxUcS3EdUDlrsdOnBtOUHOYk89UzE/v4bFK6ys30qZPewR OKch8TcT+zb1Z9m5Bg4g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pXjWv-002PAp-Ux; Thu, 02 Mar 2023 14:03:34 +0000 Received: from mail-ed1-x52c.google.com ([2a00:1450:4864:20::52c]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pXjWr-002P8z-Qd for linux-arm-kernel@lists.infradead.org; Thu, 02 Mar 2023 14:03:31 +0000 Received: by mail-ed1-x52c.google.com with SMTP id i34so67975184eda.7 for ; Thu, 02 Mar 2023 06:03:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=h/JXa3jl6N7Sx/fIVr6/rpxA4dj4ykaps8uQIwSRFHI=; b=SQe9A8zf+XRjs48CSJ+h4GnRYESyOz6AHJ2g25k/lwPj0Py8kP4rG8hhbJK3PRK/Ys g3JfMqTzkjAS07GFOOSo5vWwYNY4RjnyegQ/OwnuY/mIwkndxf/tRl69oSIjYCFEsha9 1uivUu44qu8TIqf1FwmyDphbHfzYdzuMHmHW0+vQiB1MI+CoTV1HvTPaMBmotfy/pIXM 3Ya/DW90qOpCzTz0iMBDoR/16Ny+HfrUtO9T+8TUzhnEeRvYG92vR7b848tcOs8uZSCk vTIm7M9BChxeqRZaVnDa7PfzMTULinMLJXWwlP6QFt4Pu4gBnY88Tnv+m57e1yTd1QRA McPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=h/JXa3jl6N7Sx/fIVr6/rpxA4dj4ykaps8uQIwSRFHI=; b=W/2SOvlX8nUgMcU8Jgh7erq/Y2DZUk7an9GULo/ZndiF+YQo0SQhgLZfhGDolcH4ZG avLdisHkpZB/yQXu6xjLIUxk7GWsDSYSWgNWAGXMdxEd9KJAtUvzGJNXXGHEyJDmfXeF J3CbeJCsCa0/eDyr6dkhzmBMgexJjAnhbYkEMx979fZNxFrtBaH4jHLccAf+tChmq4tn sWO7wNb6eXdyEq9ToQT3c15CuBuzM14pEI+1TTg2Y89rMG9tJWXUWeykxYDhRUj0LYi6 OvJWJihHDO4ATmKn48b5OhYQwxprNKJUV9R3z+0MMvd7Q7LMog83k1KE5TEL2GZnsAFX NZZQ== X-Gm-Message-State: AO0yUKUdfZdvj9XccU8NKgRdLDx5PkfLBAhhHjAYzeONpKYyQHzXNwp9 4eFpmtC0sfQk6aD6dZN//SQ= X-Google-Smtp-Source: AK7set/gtIv/d5+qF714NjY0nEzkNgdHznl1CeO4qmEfZDHvPZB9UdsHo5ShI+zFg/5R6pNVkFnOVQ== X-Received: by 2002:a05:6402:18a:b0:4ac:c3c0:24d7 with SMTP id r10-20020a056402018a00b004acc3c024d7mr10632959edv.42.1677765804566; Thu, 02 Mar 2023 06:03:24 -0800 (PST) Received: from [192.168.5.8] (PC-176-101-165-146.tvk-net.pl. [176.101.165.146]) by smtp.gmail.com with ESMTPSA id t19-20020a50d713000000b004af73333d6esm6981406edi.53.2023.03.02.06.03.23 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 02 Mar 2023 06:03:24 -0800 (PST) Message-ID: <7a0e256d-3ce1-3218-c930-ed518a679b8b@gmail.com> Date: Thu, 2 Mar 2023 15:03:22 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.7.1 Subject: Re: [RFC kvmtool 18/31] arm64: Populate initial realm contents Content-Language: en-US To: Suzuki K Poulose Cc: Alexandru Elisei , Andrew Jones , Christoffer Dall , Fuad Tabba , Jean-Philippe Brucker , Joey Gouly , Marc Zyngier , Mark Rutland , Oliver Upton , Paolo Bonzini , Quentin Perret , Steven Price , Thomas Huth , Will Deacon , Zenghui Yu , linux-coco@lists.linux.dev, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org References: <20230127112248.136810-1-suzuki.poulose@arm.com> <20230127113932.166089-1-suzuki.poulose@arm.com> <20230127113932.166089-19-suzuki.poulose@arm.com> From: Piotr Sawicki In-Reply-To: <20230127113932.166089-19-suzuki.poulose@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230302_060329_926998_73628E0C X-CRM114-Status: GOOD ( 44.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi, > From: Alexandru Elisei > > Populate the realm memory with the initial contents, which include > the device tree blob, the kernel image, and initrd, if specified, > or the firmware image. > > Populating an image in the realm involves two steps: > a) Mark the IPA area as RAM - INIT_IPA_REALM > b) Load the contents into the IPA - POPULATE_REALM > > Wherever we know the actual size of an image in memory, we make > sure the "memory area" is initialised to RAM. > e.g., Linux kernel image size from the header which includes the bss etc. > The "file size" on disk for the Linux image is much smaller. > We mark the region of size Image.header.size as RAM (a), from the kernel > load address. And load the Image file into the memory (b) above. > At the moment we only detect the Arm64 Linux Image header format. > > Since we're already touching the code that copies the > initrd in guest memory, let's do a bit of cleaning and remove a > useless local variable. > > Signed-off-by: Alexandru Elisei > [ Make sure the Linux kernel image area is marked as RAM ] > Signed-off-by: Suzuki K Poulose > --- > arm/aarch32/include/asm/realm.h | 3 + > arm/aarch64/include/asm/realm.h | 3 + > arm/aarch64/realm.c | 112 ++++++++++++++++++++++++++++++++ > arm/fdt.c | 6 ++ > arm/kvm.c | 20 ++++-- > include/linux/kernel.h | 1 + > 6 files changed, 140 insertions(+), 5 deletions(-) > > diff --git a/arm/aarch32/include/asm/realm.h b/arm/aarch32/include/asm/realm.h > index 5aca6cca..fcff0e55 100644 > --- a/arm/aarch32/include/asm/realm.h > +++ b/arm/aarch32/include/asm/realm.h > @@ -6,5 +6,8 @@ > #include "kvm/kvm.h" > > static inline void kvm_arm_realm_create_realm_descriptor(struct kvm *kvm) {} > +static inline void kvm_arm_realm_populate_kernel(struct kvm *kvm) {} > +static inline void kvm_arm_realm_populate_initrd(struct kvm *kvm) {} > +static inline void kvm_arm_realm_populate_dtb(struct kvm *kvm) {} > > #endif /* ! __ASM_REALM_H */ > diff --git a/arm/aarch64/include/asm/realm.h b/arm/aarch64/include/asm/realm.h > index e176f15f..6e760ac9 100644 > --- a/arm/aarch64/include/asm/realm.h > +++ b/arm/aarch64/include/asm/realm.h > @@ -6,5 +6,8 @@ > #include "kvm/kvm.h" > > void kvm_arm_realm_create_realm_descriptor(struct kvm *kvm); > +void kvm_arm_realm_populate_kernel(struct kvm *kvm); > +void kvm_arm_realm_populate_initrd(struct kvm *kvm); > +void kvm_arm_realm_populate_dtb(struct kvm *kvm); > > #endif /* ! __ASM_REALM_H */ > diff --git a/arm/aarch64/realm.c b/arm/aarch64/realm.c > index fc7f8d6a..eddccece 100644 > --- a/arm/aarch64/realm.c > +++ b/arm/aarch64/realm.c > @@ -1,5 +1,7 @@ > #include "kvm/kvm.h" > > +#include > +#include > #include > > > @@ -80,3 +82,113 @@ void kvm_arm_realm_create_realm_descriptor(struct kvm *kvm) > if (ioctl(kvm->vm_fd, KVM_ENABLE_CAP, &rme_create_rd) < 0) > die_perror("KVM_CAP_RME(KVM_CAP_ARM_RME_CREATE_RD)"); > } > + > +static void realm_init_ipa_range(struct kvm *kvm, u64 start, u64 size) > +{ > + struct kvm_cap_arm_rme_init_ipa_args init_ipa_args = { > + .init_ipa_base = start, > + .init_ipa_size = size > + }; > + struct kvm_enable_cap rme_init_ipa_realm = { > + .cap = KVM_CAP_ARM_RME, > + .args[0] = KVM_CAP_ARM_RME_INIT_IPA_REALM, > + .args[1] = (u64)&init_ipa_args > + }; > + > + if (ioctl(kvm->vm_fd, KVM_ENABLE_CAP, &rme_init_ipa_realm) < 0) > + die("unable to intialise IPA range for Realm %llx - %llx (size %llu)", > + start, start + size, size); > + > +} > + > +static void __realm_populate(struct kvm *kvm, u64 start, u64 size) > +{ > + struct kvm_cap_arm_rme_populate_realm_args populate_args = { > + .populate_ipa_base = start, > + .populate_ipa_size = size > + }; > + struct kvm_enable_cap rme_populate_realm = { > + .cap = KVM_CAP_ARM_RME, > + .args[0] = KVM_CAP_ARM_RME_POPULATE_REALM, > + .args[1] = (u64)&populate_args > + }; > + > + if (ioctl(kvm->vm_fd, KVM_ENABLE_CAP, &rme_populate_realm) < 0) > + die("unable to populate Realm memory %llx - %llx (size %llu)", > + start, start + size, size); > +} > + > +static void realm_populate(struct kvm *kvm, u64 start, u64 size) > +{ > + realm_init_ipa_range(kvm, start, size); > + __realm_populate(kvm, start, size); > +} > + > +static bool is_arm64_linux_kernel_image(void *header) > +{ > + struct arm64_image_header *hdr = header; > + > + return memcmp(&hdr->magic, ARM64_IMAGE_MAGIC, sizeof(hdr->magic)) == 0; > +} > + > +static ssize_t arm64_linux_kernel_image_size(void *header) > +{ > + struct arm64_image_header *hdr = header; > + > + if (is_arm64_linux_kernel_image(header)) > + return le64_to_cpu(hdr->image_size); > + die("Not arm64 Linux kernel Image"); > +} > + > +void kvm_arm_realm_populate_kernel(struct kvm *kvm) > +{ > + u64 start, end, mem_size; > + void *header = guest_flat_to_host(kvm, kvm->arch.kern_guest_start); > + > + start = ALIGN_DOWN(kvm->arch.kern_guest_start, SZ_4K); > + end = ALIGN(kvm->arch.kern_guest_start + kvm->arch.kern_size, SZ_4K); > + > + if (is_arm64_linux_kernel_image(header)) > + mem_size = arm64_linux_kernel_image_size(header); > + else > + mem_size = end - start; > + > + realm_init_ipa_range(kvm, start, mem_size); > + __realm_populate(kvm, start, end - start); > +} > + > +void kvm_arm_realm_populate_initrd(struct kvm *kvm) > +{ > + u64 kernel_end, start, end; > + > + kernel_end = ALIGN(kvm->arch.kern_guest_start + kvm->arch.kern_size, SZ_4K); > + start = ALIGN_DOWN(kvm->arch.initrd_guest_start, SZ_4K); > + /* > + * Because we align the initrd to 4 bytes, it is theoretically possible > + * for the start of the initrd to overlap with the same page where the > + * kernel ends. > + */ > + if (start < kernel_end) > + start = kernel_end; > + end = ALIGN(kvm->arch.initrd_guest_start + kvm->arch.initrd_size, SZ_4K); > + if (end > start) > + realm_populate(kvm, start, end - start); > +} > + > +void kvm_arm_realm_populate_dtb(struct kvm *kvm) > +{ > + u64 initrd_end, start, end; > + > + initrd_end = ALIGN(kvm->arch.initrd_guest_start + kvm->arch.initrd_size, SZ_4K); > + start = ALIGN_DOWN(kvm->arch.dtb_guest_start, SZ_4K); > + /* > + * Same situation as with the initrd, but now it is the DTB which is > + * overlapping with the last page of the initrd, because the initrd is > + * populated first. > + */ > + if (start < initrd_end) > + start = initrd_end; > + end = ALIGN(kvm->arch.dtb_guest_start + FDT_MAX_SIZE, SZ_4K); > + if (end > start) > + realm_populate(kvm, start, end - start); > +} > diff --git a/arm/fdt.c b/arm/fdt.c > index 286ccadf..762a604d 100644 > --- a/arm/fdt.c > +++ b/arm/fdt.c > @@ -7,6 +7,8 @@ > #include "arm-common/gic.h" > #include "arm-common/pci.h" > > +#include > + > #include > > #include > @@ -231,6 +233,10 @@ static int setup_fdt(struct kvm *kvm) > > if (kvm->cfg.arch.dump_dtb_filename) > dump_fdt(kvm->cfg.arch.dump_dtb_filename, fdt_dest); > + > + if (kvm->cfg.arch.is_realm) > + kvm_arm_realm_populate_dtb(kvm); > + > return 0; > } > late_init(setup_fdt); > diff --git a/arm/kvm.c b/arm/kvm.c > index acb627b2..57c5b5f7 100644 > --- a/arm/kvm.c > +++ b/arm/kvm.c > @@ -6,6 +6,7 @@ > #include "kvm/fdt.h" > > #include "arm-common/gic.h" > +#include > > #include > > @@ -167,6 +168,9 @@ bool kvm__arch_load_kernel_image(struct kvm *kvm, int fd_kernel, int fd_initrd, > pr_debug("Loaded kernel to 0x%llx (%llu bytes)", > kvm->arch.kern_guest_start, kvm->arch.kern_size); I've noticed that multiple calling of the measurement test from the kvm-unit-tests suite results in different Realm Initial Measurements, although the kernel image is always the same. After short investigation, I've found that the RIM starts being different while populating the last 4kB chunk of the kernel image. The issue occurs when the image size is not aligned to the page size (4kB). After zeroing the unused area of the last chunk, the measurements become repeatable. > + if (kvm->cfg.arch.is_realm) > + kvm_arm_realm_populate_kernel(kvm); > + > /* > * Now load backwards from the end of memory so the kernel > * decompressor has plenty of space to work with. First up is > @@ -188,7 +192,6 @@ bool kvm__arch_load_kernel_image(struct kvm *kvm, int fd_kernel, int fd_initrd, > /* ... and finally the initrd, if we have one. */ > if (fd_initrd != -1) { > struct stat sb; > - unsigned long initrd_start; > > if (fstat(fd_initrd, &sb)) > die_perror("fstat"); > @@ -199,7 +202,6 @@ bool kvm__arch_load_kernel_image(struct kvm *kvm, int fd_kernel, int fd_initrd, > if (pos < kernel_end) > die("initrd overlaps with kernel image."); > > - initrd_start = guest_addr; > file_size = read_file(fd_initrd, pos, limit - pos); > if (file_size == -1) { > if (errno == ENOMEM) > @@ -208,11 +210,13 @@ bool kvm__arch_load_kernel_image(struct kvm *kvm, int fd_kernel, int fd_initrd, > die_perror("initrd read"); > } > > - kvm->arch.initrd_guest_start = initrd_start; > + kvm->arch.initrd_guest_start = guest_addr; > kvm->arch.initrd_size = file_size; > pr_debug("Loaded initrd to 0x%llx (%llu bytes)", > - kvm->arch.initrd_guest_start, > - kvm->arch.initrd_size); > + kvm->arch.initrd_guest_start, kvm->arch.initrd_size); > + > + if (kvm->cfg.arch.is_realm) > + kvm_arm_realm_populate_initrd(kvm); > } else { > kvm->arch.initrd_size = 0; > } > @@ -269,6 +273,8 @@ bool kvm__load_firmware(struct kvm *kvm, const char *firmware_filename) > > /* Kernel isn't loaded by kvm, point start address to firmware */ > kvm->arch.kern_guest_start = fw_addr; > + kvm->arch.kern_size = fw_sz; > + > pr_debug("Loaded firmware to 0x%llx (%zd bytes)", > kvm->arch.kern_guest_start, fw_sz); > > @@ -283,6 +289,10 @@ bool kvm__load_firmware(struct kvm *kvm, const char *firmware_filename) > kvm->arch.dtb_guest_start, > kvm->arch.dtb_guest_start + FDT_MAX_SIZE); > > + if (kvm->cfg.arch.is_realm) > + /* We hijack the kernel fields to describe the firmware. */ > + kvm_arm_realm_populate_kernel(kvm); > + > return true; > } > > diff --git a/include/linux/kernel.h b/include/linux/kernel.h > index 6c22f1c0..25f19c20 100644 > --- a/include/linux/kernel.h > +++ b/include/linux/kernel.h > @@ -9,6 +9,7 @@ > > #define DIV_ROUND_UP(n,d) (((n) + (d) - 1) / (d)) > > +#define ALIGN_DOWN(x,a) __ALIGN_MASK(x - (typeof(x))((a) - 1),(typeof(x))(a)-1) > #define ALIGN(x,a) __ALIGN_MASK(x,(typeof(x))(a)-1) > #define __ALIGN_MASK(x,mask) (((x)+(mask))&~(mask)) > #define IS_ALIGNED(x, a) (((x) & ((typeof(x))(a) - 1)) == 0) Kind regards, Piotr Sawicki _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel