From: Dave Hansen <dave.hansen@intel.com>
To: "Huang, Kai" <kai.huang@intel.com>,
"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Cc: "Luck, Tony" <tony.luck@intel.com>,
"bagasdotme@gmail.com" <bagasdotme@gmail.com>,
"ak@linux.intel.com" <ak@linux.intel.com>,
"Wysocki, Rafael J" <rafael.j.wysocki@intel.com>,
"kirill.shutemov@linux.intel.com"
<kirill.shutemov@linux.intel.com>, "Christopherson,,
Sean" <seanjc@google.com>,
"Chatre, Reinette" <reinette.chatre@intel.com>,
"pbonzini@redhat.com" <pbonzini@redhat.com>,
"linux-mm@kvack.org" <linux-mm@kvack.org>,
"Yamahata, Isaku" <isaku.yamahata@intel.com>,
"tglx@linutronix.de" <tglx@linutronix.de>,
"Shahar, Sagi" <sagis@google.com>,
"imammedo@redhat.com" <imammedo@redhat.com>,
"Gao, Chao" <chao.gao@intel.com>,
"Brown, Len" <len.brown@intel.com>,
"peterz@infradead.org" <peterz@infradead.org>,
"sathyanarayanan.kuppuswamy@linux.intel.com"
<sathyanarayanan.kuppuswamy@linux.intel.com>,
"Huang, Ying" <ying.huang@intel.com>,
"Williams, Dan J" <dan.j.williams@intel.com>
Subject: Re: [PATCH v8 07/16] x86/virt/tdx: Use all system memory when initializing TDX module as TDX memory
Date: Tue, 10 Jan 2023 08:18:46 -0800 [thread overview]
Message-ID: <16f23950-2a27-29de-c0b4-e5f2d927c8b4@intel.com> (raw)
In-Reply-To: <a5d4de8b074b6d1a34bd3d424c5c4f0d677ec15f.camel@intel.com>
On 1/10/23 04:09, Huang, Kai wrote:
> On Mon, 2023-01-09 at 08:51 -0800, Dave Hansen wrote:
>> On 1/9/23 03:48, Huang, Kai wrote:
>>>>>>> This can also be enhanced in the future, i.e. by allowing adding non-TDX
>>>>>>> memory to a separate NUMA node. In this case, the "TDX-capable" nodes
>>>>>>> and the "non-TDX-capable" nodes can co-exist, but the kernel/userspace
>>>>>>> needs to guarantee memory pages for TDX guests are always allocated from
>>>>>>> the "TDX-capable" nodes.
>>>>>
>>>>> Why does it need to be enhanced? What's the problem?
>>>
>>> The problem is after TDX module initialization, no more memory can be hot-added
>>> to the page allocator.
>>>
>>> Kirill suggested this may not be ideal. With the existing NUMA ABIs we can
>>> actually have both TDX-capable and non-TDX-capable NUMA nodes online. We can
>>> bind TDX workloads to TDX-capable nodes while other non-TDX workloads can
>>> utilize all memory.
>>>
>>> But probably it is not necessarily to call out in the changelog?
>>
>> Let's say that we add this TDX-compatible-node ABI in the future. What
>> will old code do that doesn't know about this ABI?
>
> Right. The old app will break w/o knowing the new ABI. One resolution, I
> think, is we don't introduce new userspace ABI, but hide "TDX-capable" and "non-
> TDX-capable" nodes in the kernel, and let kernel to enforce always allocating
> TDX guest memory from those "TDX-capable" nodes.
That doesn't actually hide all of the behavior from users. Let's say
they do:
numactl --membind=6 qemu-kvm ...
In other words, take all of this guest's memory and put it on node 6.
There lots of free memory on node 6 which is TDX-*IN*compatible. Then,
they make it a TDX guest:
numactl --membind=6 qemu-kvm -tdx ...
What happens? Does the kernel silently ignore the --membind=6? Or does
it return -ENOMEM somewhere and confuse the user who has *LOTS* of free
memory on node 6.
In other words, I don't think the kernel can just enforce this
internally and hide it from userspace.
>> Is there something fundamental that keeps a memory area that spans two
>> nodes from being removed and then a new area added that is comprised of
>> a single node?
>> Boot time:
>>
>> | memblock | memblock |
>> <--Node=0--> <--Node=1-->
>>
>> Funky hotplug... nothing to see here, then:
>>
>> <--------Node=2-------->
>
> I must have missed something, but how can this happen?
>
> I had memory that this cannot happen because the BIOS always allocates address
> ranges for all NUMA nodes during machine boot. Those address ranges don't
> necessarily need to have DIMM fully populated but they don't change during
> machine's runtime.
Is your memory correct? Is there evidence, or requirements in any
specification to support your memory?
next prev parent reply other threads:[~2023-01-10 16:19 UTC|newest]
Thread overview: 84+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-12-09 6:52 [PATCH v8 00/16] TDX host kernel support Kai Huang
2022-12-09 6:52 ` [PATCH v8 01/16] x86/tdx: Define TDX supported page sizes as macros Kai Huang
2023-01-06 19:04 ` Dave Hansen
2022-12-09 6:52 ` [PATCH v8 02/16] x86/virt/tdx: Detect TDX during kernel boot Kai Huang
2023-01-06 17:09 ` Dave Hansen
2023-01-08 22:25 ` Huang, Kai
2022-12-09 6:52 ` [PATCH v8 03/16] x86/virt/tdx: Make INTEL_TDX_HOST depend on X86_X2APIC Kai Huang
2023-01-06 19:04 ` Dave Hansen
2022-12-09 6:52 ` [PATCH v8 04/16] x86/virt/tdx: Add skeleton to initialize TDX on demand Kai Huang
2023-01-06 17:14 ` Dave Hansen
2023-01-08 22:26 ` Huang, Kai
2022-12-09 6:52 ` [PATCH v8 05/16] x86/virt/tdx: Implement functions to make SEAMCALL Kai Huang
2023-01-06 17:29 ` Dave Hansen
2023-01-09 10:30 ` Huang, Kai
2023-01-09 19:54 ` Dave Hansen
2023-01-09 22:10 ` Huang, Kai
2022-12-09 6:52 ` [PATCH v8 06/16] x86/virt/tdx: Get information about TDX module and TDX-capable memory Kai Huang
2023-01-06 17:46 ` Dave Hansen
2023-01-09 10:25 ` Huang, Kai
2023-01-09 19:52 ` Dave Hansen
2023-01-09 22:07 ` Huang, Kai
2023-01-09 22:11 ` Dave Hansen
2022-12-09 6:52 ` [PATCH v8 07/16] x86/virt/tdx: Use all system memory when initializing TDX module as TDX memory Kai Huang
2023-01-06 18:18 ` Dave Hansen
2023-01-09 11:48 ` Huang, Kai
2023-01-09 16:51 ` Dave Hansen
2023-01-10 12:09 ` Huang, Kai
2023-01-10 16:18 ` Dave Hansen [this message]
2023-01-11 10:00 ` Huang, Kai
2023-01-12 0:56 ` Huang, Ying
2023-01-12 1:18 ` Huang, Kai
2023-01-12 1:59 ` Huang, Ying
2023-01-12 2:22 ` Huang, Kai
2023-01-12 11:33 ` Huang, Kai
2023-01-18 11:08 ` Huang, Kai
2023-01-18 13:57 ` David Hildenbrand
2023-01-18 19:38 ` Huang, Kai
2022-12-09 6:52 ` [PATCH v8 08/16] x86/virt/tdx: Add placeholder to construct TDMRs to cover all TDX memory regions Kai Huang
2023-01-06 19:24 ` Dave Hansen
2023-01-10 0:40 ` Huang, Kai
2023-01-10 0:47 ` Dave Hansen
2023-01-10 2:23 ` Huang, Kai
2023-01-10 19:12 ` Dave Hansen
2023-01-11 9:23 ` Huang, Kai
2022-12-09 6:52 ` [PATCH v8 09/16] x86/virt/tdx: Fill out " Kai Huang
2023-01-06 19:36 ` Dave Hansen
2023-01-10 0:45 ` Huang, Kai
2022-12-09 6:52 ` [PATCH v8 10/16] x86/virt/tdx: Allocate and set up PAMTs for TDMRs Kai Huang
2023-01-06 21:53 ` Dave Hansen
2023-01-10 0:49 ` Huang, Kai
2023-01-07 0:47 ` Dave Hansen
2023-01-10 0:47 ` Huang, Kai
2022-12-09 6:52 ` [PATCH v8 11/16] x86/virt/tdx: Designate reserved areas for all TDMRs Kai Huang
2023-01-06 22:07 ` Dave Hansen
2023-01-10 1:19 ` Huang, Kai
2023-01-10 1:22 ` Dave Hansen
2023-01-10 11:01 ` Huang, Kai
2023-01-10 15:19 ` Dave Hansen
2023-01-11 10:57 ` Huang, Kai
2023-01-11 16:16 ` Dave Hansen
2023-01-11 22:10 ` Huang, Kai
2023-01-10 11:01 ` Huang, Kai
2023-01-10 15:17 ` Dave Hansen
2022-12-09 6:52 ` [PATCH v8 12/16] x86/virt/tdx: Designate the global KeyID and configure the TDX module Kai Huang
2023-01-06 22:21 ` Dave Hansen
2023-01-10 10:48 ` Huang, Kai
2023-01-10 16:25 ` Dave Hansen
2023-01-10 23:33 ` Huang, Kai
2022-12-09 6:52 ` [PATCH v8 13/16] x86/virt/tdx: Configure global KeyID on all packages Kai Huang
2023-01-06 22:49 ` Dave Hansen
2023-01-10 10:15 ` Huang, Kai
2023-01-10 16:53 ` Dave Hansen
2023-01-11 0:06 ` Huang, Kai
2022-12-09 6:52 ` [PATCH v8 14/16] x86/virt/tdx: Initialize all TDMRs Kai Huang
2023-01-07 0:17 ` Dave Hansen
2023-01-10 10:23 ` Huang, Kai
2022-12-09 6:52 ` [PATCH v8 15/16] x86/virt/tdx: Flush cache in kexec() when TDX is enabled Kai Huang
2023-01-07 0:35 ` Dave Hansen
2023-01-10 11:29 ` Huang, Kai
2023-01-10 15:27 ` Dave Hansen
2023-01-11 0:13 ` Huang, Kai
2023-01-11 0:30 ` Dave Hansen
2023-01-11 1:58 ` Huang, Kai
2022-12-09 6:52 ` [PATCH v8 16/16] Documentation/x86: Add documentation for TDX host support Kai Huang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=16f23950-2a27-29de-c0b4-e5f2d927c8b4@intel.com \
--to=dave.hansen@intel.com \
--cc=ak@linux.intel.com \
--cc=bagasdotme@gmail.com \
--cc=chao.gao@intel.com \
--cc=dan.j.williams@intel.com \
--cc=imammedo@redhat.com \
--cc=isaku.yamahata@intel.com \
--cc=kai.huang@intel.com \
--cc=kirill.shutemov@linux.intel.com \
--cc=kvm@vger.kernel.org \
--cc=len.brown@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=pbonzini@redhat.com \
--cc=peterz@infradead.org \
--cc=rafael.j.wysocki@intel.com \
--cc=reinette.chatre@intel.com \
--cc=sagis@google.com \
--cc=sathyanarayanan.kuppuswamy@linux.intel.com \
--cc=seanjc@google.com \
--cc=tglx@linutronix.de \
--cc=tony.luck@intel.com \
--cc=ying.huang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).