From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40597C4338F for ; Mon, 9 Aug 2021 18:25:10 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DDE2260F8F for ; Mon, 9 Aug 2021 18:25:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org DDE2260F8F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.165200.301921 (Exim 4.92) (envelope-from ) id 1mD9x6-0003ly-V7; Mon, 09 Aug 2021 18:24:44 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 165200.301921; Mon, 09 Aug 2021 18:24:44 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mD9x6-0003lr-RE; Mon, 09 Aug 2021 18:24:44 +0000 Received: by outflank-mailman (input) for mailman id 165200; Mon, 09 Aug 2021 18:24:43 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mD9x5-0003ll-C4 for xen-devel@lists.xenproject.org; Mon, 09 Aug 2021 18:24:43 +0000 Received: from mail-lj1-x22e.google.com (unknown [2a00:1450:4864:20::22e]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 7b300149-eec5-478a-8c80-3db736f2c886; Mon, 09 Aug 2021 18:24:41 +0000 (UTC) Received: by mail-lj1-x22e.google.com with SMTP id a7so24908795ljq.11 for ; Mon, 09 Aug 2021 11:24:41 -0700 (PDT) Received: from [192.168.1.7] ([212.22.223.21]) by smtp.gmail.com with ESMTPSA id t142sm1802752lff.269.2021.08.09.11.24.39 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 09 Aug 2021 11:24:40 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 7b300149-eec5-478a-8c80-3db736f2c886 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=yWuXsEqJtZJ1mFJI5QsqhAlsVtYtcbWuN/ULVUqdVl0=; b=D6oRBlo7pl65BPDhfa9/NGFdZzaJzLhuukZy2EeMeNXpWTGBz9tzwQDl5O/R+q6xcL rcUZD5H+KB7RE3ekRJ7KvYPnnHS+r1vb/2l5CpjXslpKAubl+0nDMWoG2zjcf0PX0lKs SwZwaN+5V6PL4DZbKja2iwnBm04rr0i1yeAtmY+aHn2EP4AsSUflrBH++WBRMEP6+r+T 6Qch4AyuUUQDtQmDr4pZEsGiDtmZFG8W7mmY+BRyET1KyDcyCjOYTO/7b4n57wAhgmst 67AYb+El1smMP/pCes5rB3Hm6SSiBnzg16i9zOjbKaXtcfypIAsGPTi8li+E60HrA134 rN1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=yWuXsEqJtZJ1mFJI5QsqhAlsVtYtcbWuN/ULVUqdVl0=; b=DI0K+bv1s1T025BOxISVTSzCrRPtmMSAICqe6tKFxCeK1Eh8I33Mi/TDYAY/gHqnyA nd34mzBxKPTHtd+V4dwoURZW/fDyijJbbjW8C8lupJMrUCC2ARAKptP1TVCcfu8NUVSD P+aLFNnfq9SS9QlIidAvr2QMMWLaRIOOfyKRWhng+YwElvn84rRbI7vevQdVmDmYuycQ Az9xzLJ7UJH7rOAGikB7ySDFxLyUGiaxcgHZTsuIRuQK5CUgia0lxUWSewCh6Mma1hxR YFWy3pDxtfGaljjM7qAfab/duP2JBVf6k66uYg7FT58t7Bs0lONDuF4quaW9jHb7kCL6 /xaw== X-Gm-Message-State: AOAM532QMf/SZKDxCi0dRVcnjOqGq0Aw/p81aAF+fInpkztg9+FbD9qQ otWN1no+NwDMkf+3j/NXPMs= X-Google-Smtp-Source: ABdhPJyqsz0tFQpZ391Gbd2WtohWCiEYAQazBQSMyK1FU/M8NvvwgFhQYf4fxxl0yR10HGRivc24Zg== X-Received: by 2002:a2e:a364:: with SMTP id i4mr232528ljn.501.1628533480917; Mon, 09 Aug 2021 11:24:40 -0700 (PDT) Subject: Re: [RFC PATCH] xen/memory: Introduce a hypercall to provide unallocated space To: Julien Grall Cc: Stefano Stabellini , Andrew Cooper , xen-devel@lists.xenproject.org, Oleksandr Tyshchenko , Daniel De Graaf , "Daniel P. Smith" , Ian Jackson , Wei Liu , George Dunlap , Jan Beulich , Volodymyr Babchuk , =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= , Bertrand Marquis , Wei Chen References: <1627489110-25633-1-git-send-email-olekstysh@gmail.com> <80fafc01-f063-d6e5-1c08-7ad64550310c@citrix.com> <4de5ed21-379e-b618-44c8-924d88b1a519@citrix.com> <6a633f4e-13e0-4a2b-cf6e-35ef90ae977c@gmail.com> <5643d414-0b76-74a4-2c37-c7a99338d547@gmail.com> <6596ad08-8398-64dd-ef62-cd7bc6f7333e@gmail.com> From: Oleksandr Message-ID: <1d0ea55d-2e5a-daa7-9c60-c7a1c4b48fa1@gmail.com> Date: Mon, 9 Aug 2021 21:24:39 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US On 09.08.21 18:42, Julien Grall wrote: > Hi Oleksandr, Hi Julien. Thank you for the input. > > On 07/08/2021 18:03, Oleksandr wrote: >> >> On 06.08.21 03:30, Stefano Stabellini wrote: >> >> Hi Stefano >> >>> On Wed, 4 Aug 2021, Julien Grall wrote: >>>>> +#define GUEST_SAFE_RANGE_BASE xen_mk_ullong(0xDE00000000) /* >>>>> 128GB */ >>>>> +#define GUEST_SAFE_RANGE_SIZE xen_mk_ullong(0x0200000000) >>>>> >>>>> While the possible new DT bindings has not been agreed yet, I re-used >>>>> existing "reg" property under the hypervisor node to pass safe >>>>> range as a >>>>> second region, >>>>> https://elixir.bootlin.com/linux/v5.14-rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10: >>>>> >>>> So a single region works for a guest today, but for dom0 we will >>>> need multiple >>>> regions because it is may be difficult to find enough contiguous >>>> space for a >>>> single region. >>>> >>>> That said, as dom0 is mapped 1:1 (including some guest mapping), >>>> there is also >>>> the question where to allocate the safe region. For grant table, we >>>> so far >>>> re-use the Xen address space because it is assumed it will space >>>> will always >>>> be bigger than the grant table. >>>> >>>> I am not sure yet where we could allocate the safe regions. >>>> Stefano, do you >>>> have any ideas? >>> The safest choice would be the address range corresponding to memory >>> (/memory) not already allocated to Dom0. >>> >>> For instance from my last boot logs: >>> (XEN) Allocating 1:1 mappings totalling 1600MB for dom0: >>> (XEN) BANK[0] 0x00000010000000-0x00000070000000 (1536MB) >>> (XEN) BANK[1] 0x00000078000000-0x0000007c000000 (64MB) >>> >>> All the other ranges could be given as unallocated space: >>> >>> - 0x0 - 0x10000000 >>> - 0x70000000 - 0x78000000 >>> - 0x8_0000_0000 - 0x8_8000_0000 >> >> Thank you for the ideas. >> >> If I got the idea correctly, yes, as these ranges represent the real >> RAM, so no I/O would be in conflict with them and as the result - no >> overlaps would be expected. >> But, I wonder, would this work if we have IOMMU enabled for Dom0 and >> need to establish 1:1 mapping for the DMA devices to work with grant >> mappings... >> In arm_iommu_map_page() we call guest_physmap_add_entry() with gfn = >> mfn, so the question is could we end up with this new gfn replacing >> the valid mapping >> (with gfn allocated from the safe region)? > > Right, when we enable the IOMMU for dom0, Xen will add an extra > mapping with GFN == MFN for foreign and grant pages. This is because > Linux is not aware that whether a device is protected by an IOMMU. > Therefore it is assuming it is not and will use the MFN to configure > for DMA transaction. > > We can't remove the mapping without significant changes in Linux and > Xen. I would not mandate them for this work. > > That said, I think it would be acceptable to have different way to > find the region depending on the dom0 configuration. So we could use > the RAM not used by dom0 when the IOMMU is turned off. OK > >>> The second best choice would be an hole: an address range not used by >>> anybody else (no reg property) and also not even mappable by a bus (not >>> covered by a ranges property). This is not the best choice because >>> there >>> can cases where physical resources appear afterwards. > > Are you saying that the original device-tree doesn't even describe > them in any way (i.e. reserved...)? > >> >> Unfortunately, yes. > > So the decision where the safe region is located will be done by Xen. > There is no involvement of the domain (it will discover the region > from the DT). Therefore, I don't think we need to think about > everything right now as we could adapt this is exact region is not > part of the stable ABI. > > The hotplug is one I would defer because this is not supported (and > quite likely not working) in Xen upstream today. Sounds reasonable. > > > Now regarding the case where dom0 is using the IOMMU. The assumption > is Xen will be able to figure out all the regions used from the > firmware table (ACPI or DT). > > AFAIK, this assumption would be correct for DT. However, for ACPI, I > remember we were not able to find all the MMIOs region in Xen (see [1] > and [2]). So even this solution would not work for ACPI. > > If I am not mistaken, we don't support IOMMU with ACPI yet. So we > could defer the problem to when this is going to be supported. Sounds reasonable. To summarize: 0. Skip ACPI case for now, implement for DT case 1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as safe ranges I would take into the account holes >= 1MB. I am wondering, do we need a special alignment here other than a PAGE_SIZE? 2. If IOMMU is disabled for Dom0 -> provide RAM which not assigned to Dom0 as safe ranges We could even provide holes here as well. > > > Cheers, > > [1] https://marc.info/?l=linux-arm-kernel&m=148469169210500&w=2 > [2] Xen commit 80f9c316708400cea4417e36337267d3b26591db > -- Regards, Oleksandr Tyshchenko