From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09F25C47247 for ; Thu, 30 Apr 2020 20:50:45 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D113C20836 for ; Thu, 30 Apr 2020 20:50:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=xen.org header.i=@xen.org header.b="HZ3Q9E4S" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D113C20836 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=xen.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jUG8j-0004bx-KG; Thu, 30 Apr 2020 20:50:37 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jUG8i-0004bO-Sh for xen-devel@lists.xenproject.org; Thu, 30 Apr 2020 20:50:36 +0000 X-Inumbo-ID: 2dcb9211-8b24-11ea-9ab3-12813bfff9fa Received: from mail.xenproject.org (unknown [104.130.215.37]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 2dcb9211-8b24-11ea-9ab3-12813bfff9fa; Thu, 30 Apr 2020 20:50:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=xen.org; s=20200302mail; h=References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Sender:Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=tF+dQ9SLZTmEavdmofAbgYKyBZJqPPcUUiyb5AWHxN8=; b=HZ3Q9E4S5yiVy8k/Mfc9UAUXsi O1sw7XgIi6nF741qocRN/95JoYovNWdGd3bBrj+aNrItkexAW57YfkTs4HkGKyEaNkKkFFYXdfVhU eMd1JKLU1yKKp4Z4KPUjwrS/sMnihHtyOyD16eQyYgRoJFaTzFqR1qwJ94oMjm7IhaEY=; Received: from xenbits.xenproject.org ([104.239.192.120]) by mail.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jUG8I-0004Un-SN; Thu, 30 Apr 2020 20:50:10 +0000 Received: from 54-240-197-234.amazon.com ([54.240.197.234] helo=u1bbd043a57dd5a.ant.amazon.com) by xenbits.xenproject.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.89) (envelope-from ) id 1jUG3G-0005wj-3A; Thu, 30 Apr 2020 20:44:58 +0000 From: Hongyan Xia To: xen-devel@lists.xenproject.org Subject: [PATCH 14/16] x86/setup: leave early boot slightly earlier Date: Thu, 30 Apr 2020 21:44:23 +0100 Message-Id: <446c0b3b1811574d155cb84827e4fc64e3425413.1588278317.git.hongyxia@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , julien@xen.org, Wei Liu , Jan Beulich , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" From: Hongyan Xia When we do not have a direct map, memory for metadata of heap nodes in init_node_heap() is allocated from xenheap, which needs to be mapped and unmapped on demand. However, we cannot just take memory from the boot allocator to create the PTEs while we are passing memory to the heap allocator. To solve this race, we leave early boot slightly sooner so that Xen PTE pages are allocated from the heap instead of the boot allocator. We can do this because the metadata for the 1st node is statically allocated, and by the time we need memory to create mappings for the 2nd node, we already have enough memory in the heap allocator in the 1st node. Signed-off-by: Hongyan Xia --- xen/arch/x86/setup.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c index 60fc4038be..dbb2ac1c8f 100644 --- a/xen/arch/x86/setup.c +++ b/xen/arch/x86/setup.c @@ -1507,6 +1507,22 @@ void __init noreturn __start_xen(unsigned long mbi_p) numa_initmem_init(0, raw_max_page); + /* + * When we do not have a direct map, memory for metadata of heap nodes in + * init_node_heap() is allocated from xenheap, which needs to be mapped and + * unmapped on demand. However, we cannot just take memory from the boot + * allocator to create the PTEs while we are passing memory to the heap + * allocator during end_boot_allocator(). + * + * To solve this race, we need to leave early boot before + * end_boot_allocator() so that Xen PTE pages are allocated from the heap + * instead of the boot allocator. We can do this because the metadata for + * the 1st node is statically allocated, and by the time we need memory to + * create mappings for the 2nd node, we already have enough memory in the + * heap allocator in the 1st node. + */ + system_state = SYS_STATE_boot; + if ( max_page - 1 > virt_to_mfn(HYPERVISOR_VIRT_END - 1) ) { unsigned long limit = virt_to_mfn(HYPERVISOR_VIRT_END - 1); @@ -1536,8 +1552,6 @@ void __init noreturn __start_xen(unsigned long mbi_p) else end_boot_allocator(); - system_state = SYS_STATE_boot; - console_init_ring(); vesa_init(); -- 2.24.1.AMZN