From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33389C433FE for ; Thu, 24 Nov 2022 00:42:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C4D6A6B0071; Wed, 23 Nov 2022 19:42:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BFD676B0072; Wed, 23 Nov 2022 19:42:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC52E6B0074; Wed, 23 Nov 2022 19:42:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 996056B0071 for ; Wed, 23 Nov 2022 19:42:22 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 63EDE12050A for ; Thu, 24 Nov 2022 00:42:22 +0000 (UTC) X-FDA: 80166484524.25.8A60254 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf16.hostedemail.com (Postfix) with ESMTP id B63EC180008 for ; Thu, 24 Nov 2022 00:42:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669250541; x=1700786541; h=message-id:date:mime-version:subject:to:cc:references: from:in-reply-to:content-transfer-encoding; bh=rwf2kOewvRWuB3KO93AmVzqlch5DrL4wzyL2Tt+r8ro=; b=gWceUQX0rs2TSK3J62EC+UPfB7D0yegxIMdcBGksaAUenRNC+U72V38/ SFGlTWAkRXdJpRPzCY6N0wLWP1CTwsGBtUGIOBvRYLF/gqjhIFm21wJI5 16ZyMQxtdUUt6kJrpNQOG3SiXAH+8KVmzzO7TMqW8KBN0qgmh874Zoiwj /8PW14JVmlchNHrrsGdaLv5Zm8Opd8tE/8+iDc7qEB+3x/tdb8N+4W4vF LlFe6gitS9h3Se8/hlpiNgT3wRSGd9jZSvCjTSkLND/bR13uuc4e1gNvx lKHCxzcusEI2SAGH5onw2aJEUGeVrizTJvnHDnPxZ/YLXL95jhKQ4xPog w==; X-IronPort-AV: E=McAfee;i="6500,9779,10540"; a="293891681" X-IronPort-AV: E=Sophos;i="5.96,189,1665471600"; d="scan'208";a="293891681" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Nov 2022 16:42:20 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10540"; a="784436506" X-IronPort-AV: E=Sophos;i="5.96,187,1665471600"; d="scan'208";a="784436506" Received: from vcbudden-mobl3.amr.corp.intel.com (HELO [10.212.129.67]) ([10.212.129.67]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Nov 2022 16:42:19 -0800 Message-ID: <4e19a7e1-9088-21d6-d521-a76c340d8179@intel.com> Date: Wed, 23 Nov 2022 16:42:17 -0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.2.2 Subject: Re: [PATCH v7 18/20] x86/virt/tdx: Initialize all TDMRs Content-Language: en-US To: Kai Huang , linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: linux-mm@kvack.org, seanjc@google.com, pbonzini@redhat.com, dan.j.williams@intel.com, rafael.j.wysocki@intel.com, kirill.shutemov@linux.intel.com, ying.huang@intel.com, reinette.chatre@intel.com, len.brown@intel.com, tony.luck@intel.com, peterz@infradead.org, ak@linux.intel.com, isaku.yamahata@intel.com, chao.gao@intel.com, sathyanarayanan.kuppuswamy@linux.intel.com, bagasdotme@gmail.com, sagis@google.com, imammedo@redhat.com References: <2337c8e9086a006aaa2c4b99caf478420d1fc640.1668988357.git.kai.huang@intel.com> From: Dave Hansen In-Reply-To: <2337c8e9086a006aaa2c4b99caf478420d1fc640.1668988357.git.kai.huang@intel.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1669250542; a=rsa-sha256; cv=none; b=XZEFPvqq2S8ZrOFYz0zNKmKC16/Bw4dtyw5PFCl8wWMfSaxrzZz5uhob33Qm/7sureoVYD c4rIjtn4dlJqEeBXhI6pInxs/68JvFdi4dtZnGJeZhUdK1toobLqpbvwIOGdfsC+amHW/f wVZTAOypeYn35mDqRpTZgzIIywBIiZk= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=gWceUQX0; spf=pass (imf16.hostedemail.com: domain of dave.hansen@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=dave.hansen@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1669250542; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VWGORbB/uNherJ3jvt14VgXVdERyWnrwctD453cuWmM=; b=FIBHJChhKkVdBrqwiaQbyGwubRRXAJtKSYjDtFd1R8ml5ihUVoJyd2P9t5wCW8fpZ4XEOb gwUXeZN+GyPLjy1qy9waLSnJuKvnq5uDyPU63IE+2NDu/b+GaJOSAtmDuJh4Zyxs7hXgzW RWSfR+plKvlSsIi2fO0RWKEw84NIn88= X-Rspam-User: X-Stat-Signature: i4qymmgch4tnmnm6derydxi8mfgyyg8b X-Rspamd-Queue-Id: B63EC180008 Authentication-Results: imf16.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=gWceUQX0; spf=pass (imf16.hostedemail.com: domain of dave.hansen@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=dave.hansen@intel.com; dmarc=pass (policy=none) header.from=intel.com X-Rspamd-Server: rspam07 X-HE-Tag: 1669250541-461180 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 11/20/22 16:26, Kai Huang wrote: > Initialize TDMRs via TDH.SYS.TDMR.INIT as the last step to complete the > TDX initialization. > > All TDMRs need to be initialized using TDH.SYS.TDMR.INIT SEAMCALL before > the memory pages can be used by the TDX module. The time to initialize > TDMR is proportional to the size of the TDMR because TDH.SYS.TDMR.INIT > internally initializes the PAMT entries using the global KeyID. > > To avoid long latency caused in one SEAMCALL, TDH.SYS.TDMR.INIT only > initializes an (implementation-specific) subset of PAMT entries of one > TDMR in one invocation. The caller needs to call TDH.SYS.TDMR.INIT > iteratively until all PAMT entries of the given TDMR are initialized. > > TDH.SYS.TDMR.INITs can run concurrently on multiple CPUs as long as they > are initializing different TDMRs. To keep it simple, just initialize > all TDMRs one by one. On a 2-socket machine with 2.2G CPUs and 64GB > memory, each TDH.SYS.TDMR.INIT roughly takes couple of microseconds on > average, and it takes roughly dozens of milliseconds to complete the > initialization of all TDMRs while system is idle. Any chance you could say TDH.SYS.TDMR.INIT a few more times in there? ;) Seriously, please try to trim that down. If you talk about the implementation in detail in the code comments, don't cover it in detail in the changelog too. Also, this is a momentous patch, right? It's the last piece. Maybe worth calling that out. > diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c > index 99d1be5941a7..9bcdb30b7a80 100644 > --- a/arch/x86/virt/vmx/tdx/tdx.c > +++ b/arch/x86/virt/vmx/tdx/tdx.c > @@ -1066,6 +1066,65 @@ static int config_global_keyid(void) > return seamcall_on_each_package_serialized(&sc); > } > > +/* Initialize one TDMR */ Does this comment add value? Even if it does, it is better than naming the dang function init_one_tdmr()? > +static int init_tdmr(struct tdmr_info *tdmr) > +{ > + u64 next; > + > + /* > + * Initializing PAMT entries might be time-consuming (in > + * proportion to the size of the requested TDMR). To avoid long > + * latency in one SEAMCALL, TDH.SYS.TDMR.INIT only initializes > + * an (implementation-defined) subset of PAMT entries in one > + * invocation. > + * > + * Call TDH.SYS.TDMR.INIT iteratively until all PAMT entries > + * of the requested TDMR are initialized (if next-to-initialize > + * address matches the end address of the TDMR). > + */ The PAMT discussion here is, IMNHO silly. If this were about initializing PAMT's then it should be renamed init_pamts() and the SEAMCALL should be called PAMT_INIT or soemthing. It's not, and the ABI is built around the TDMR and *its* addresses. Let's chop this comment down: /* * Initializing a TDMR can be time consuming. To avoid long * SEAMCALLs, the TDX module may only initialize a part of the * TDMR in each call. */ Talk about the looping logic in the loop. > + do { > + struct tdx_module_output out; > + int ret; > + > + ret = seamcall(TDH_SYS_TDMR_INIT, tdmr->base, 0, 0, 0, NULL, > + &out); > + if (ret) > + return ret; > + /* > + * RDX contains 'next-to-initialize' address if > + * TDH.SYS.TDMR.INT succeeded. > + */ > + next = out.rdx; > + /* Allow scheduling when needed */ That comment is a wee bit superfluous, don't you think? > + cond_resched(); /* Keep making SEAMCALLs until the TDMR is done */ > + } while (next < tdmr->base + tdmr->size); > + > + return 0; > +} > + > +/* Initialize all TDMRs */ Does this comment add value? > +static int init_tdmrs(struct tdmr_info *tdmr_array, int tdmr_num) > +{ > + int i; > + > + /* > + * Initialize TDMRs one-by-one for simplicity, though the TDX > + * architecture does allow different TDMRs to be initialized in > + * parallel on multiple CPUs. Parallel initialization could > + * be added later when the time spent in the serialized scheme > + * becomes a real concern. > + */ Trim down the comment: /* * This operation is costly. It can be parallelized, * but keep it simple for now. */ > + for (i = 0; i < tdmr_num; i++) { > + int ret; > + > + ret = init_tdmr(tdmr_array_entry(tdmr_array, i)); > + if (ret) > + return ret; > + } > + > + return 0; > +} > + > /* > * Detect and initialize the TDX module. > * > @@ -1172,11 +1231,11 @@ static int init_tdx_module(void) > if (ret) > goto out_free_pamts; > > - /* > - * Return -EINVAL until all steps of TDX module initialization > - * process are done. > - */ > - ret = -EINVAL; > + /* Initialize TDMRs to complete the TDX module initialization */ > + ret = init_tdmrs(tdmr_array, tdmr_num); > + if (ret) > + goto out_free_pamts; > + > out_free_pamts: > if (ret) { > /* > diff --git a/arch/x86/virt/vmx/tdx/tdx.h b/arch/x86/virt/vmx/tdx/tdx.h > index 768d097412ab..891691b1ea50 100644 > --- a/arch/x86/virt/vmx/tdx/tdx.h > +++ b/arch/x86/virt/vmx/tdx/tdx.h > @@ -19,6 +19,7 @@ > #define TDH_SYS_INFO 32 > #define TDH_SYS_INIT 33 > #define TDH_SYS_LP_INIT 35 > +#define TDH_SYS_TDMR_INIT 36 > #define TDH_SYS_LP_SHUTDOWN 44 > #define TDH_SYS_CONFIG 45 >