From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.2 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_2 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA2D6C433B4 for ; Wed, 28 Apr 2021 11:05:11 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7AC32613BD for ; Wed, 28 Apr 2021 11:05:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7AC32613BD Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=xen.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.119225.225632 (Exim 4.92) (envelope-from ) id 1lbi04-0002H0-2K; Wed, 28 Apr 2021 11:05:00 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 119225.225632; Wed, 28 Apr 2021 11:05:00 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lbi03-0002Gt-VQ; Wed, 28 Apr 2021 11:04:59 +0000 Received: by outflank-mailman (input) for mailman id 119225; Wed, 28 Apr 2021 11:04:58 +0000 Received: from mail.xenproject.org ([104.130.215.37]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lbi02-0002Gm-Qp for xen-devel@lists.xenproject.org; Wed, 28 Apr 2021 11:04:58 +0000 Received: from xenbits.xenproject.org ([104.239.192.120]) by mail.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lbhzz-0003hQ-LJ; Wed, 28 Apr 2021 11:04:55 +0000 Received: from 54-240-197-239.amazon.com ([54.240.197.239] helo=ua82172827c7b5a.ant.amazon.com) by xenbits.xenproject.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lbhzz-00026Q-6P; Wed, 28 Apr 2021 11:04:55 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=xen.org; s=20200302mail; h=Content-Transfer-Encoding:Mime-Version:Content-Type: References:In-Reply-To:Date:Cc:To:From:Subject:Message-ID; bh=69MVAyGcIfT3FsUHh/jvNeUT2ENrdkUxLG6OCXyG6fk=; b=QaNvjseb5/dS22R0e5rjWavcSF nQk5gdR1E8361Qc0T6yvXXlxFEGkDQ45LN8/fbM9lQ64KFnM65fOuTwsfDU7Qo7nJEcFZbb+95clb ziqcRCU3SgFATKg+C52OR1Ze1Fk9YSfvBvqKqORmg33oKnn+awpd/7UWxS1KPOb3ihqQ=; Message-ID: <943480f695052163b540919c9c55655e5b4f741a.camel@xen.org> Subject: Re: [PATCH 13/16] xen/page_alloc: add a path for xenheap when there is no direct map From: Hongyan Xia To: Jan Beulich Cc: julien@xen.org, Andrew Cooper , George Dunlap , Ian Jackson , Stefano Stabellini , Wei Liu , xen-devel@lists.xenproject.org Date: Wed, 28 Apr 2021 12:04:51 +0100 In-Reply-To: References: <32ae7c14babf7e78b60febb53095a74c5e865456.1588278317.git.hongyxia@amazon.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.5-0ubuntu0.18.04.2 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit On Thu, 2021-04-22 at 14:31 +0200, Jan Beulich wrote: > On 30.04.2020 22:44, Hongyan Xia wrote: > > From: Hongyan Xia > > > > When there is not an always-mapped direct map, xenheap allocations > > need > > to be mapped and unmapped on-demand. > > > > Signed-off-by: Hongyan Xia > > This series has been left uncommented for far too long - I'm sorry. > While earlier patches here are probably reasonable (but would likely > need re-basing, so I'm not sure whether to try to get to look though > them before that makes much sense), No worries. This series depends on the domheap Xen page table conversion series anyway (which was just fully merged. Thanks.). I will re-base now since the dependency is resolved. > As I don't think we have many cases where code actually depends on > being able to apply __va() (or equivalent) to the address returned > from alloc_xenheap_pages(), I think this should instead involve > vmap(), with the vmap area drastically increased (perhaps taking all > of the space the direct map presently consumes). For any remaining > users of __va() or alike these should perhaps be converted into an > alias / derivation of vmap_to_{mfn,page}() then. That's true, and this was my first implementation (and also Wei's original proposal) which worked okay. But, several problems got in the way. 1. Partial unmap. Biggest offender is xmalloc which allocates and could then free part of it, which means we need to be able to partially unmap the region. vmap() does not support this. 2. Fast PA->VA. There is currently no way to go from PA to VA in vmapped pages, unless we somehow repurpose or add new fields in page_info. Also, VA->PA is possible but very slow now. There is not much PA->VA in the critical path but see 3. 3. EPT. Mapping and unmapping EPT in HVM hypercalls and MMIO are so many and so slow that it is probably not possible to keep them as domheap pages due to the big performance drop after removing the direct map. If we move them to xenheap pages on vmap, then this depends on 2 for page table walking. In the end, I could not find a way that met all 3 above without massive and intrusive changes. If there is a way, it certainly needs a design document. The "on-demand" direct map solves all the problems without breaking any APIs and is very easy to understand. We have been using Xen without the direct map for a while now with this approach with decent performance (in fact, you cannot tell that this is a Xen without the direct map by just real-world benchmarks alone). I too agree that this approach is a litte hacky and wastes a big chunk of virtual address space. Definitely wants some discussion if a better way can be found that solves the problems. Thanks, Hongyan