From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C103FC00140 for ; Thu, 18 Aug 2022 21:52:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 495BD8D0002; Thu, 18 Aug 2022 17:52:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 41E918D0001; Thu, 18 Aug 2022 17:52:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 298DF8D0002; Thu, 18 Aug 2022 17:52:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 188418D0001 for ; Thu, 18 Aug 2022 17:52:26 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id DCAED1A060A for ; Thu, 18 Aug 2022 21:52:25 +0000 (UTC) X-FDA: 79814062650.08.94A368C Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf31.hostedemail.com (Postfix) with ESMTP id 17B4B2326B for ; Thu, 18 Aug 2022 21:44:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1660859068; x=1692395068; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=5TUL0pMN1Ch62fh0N5bHXldv3b4lhkGVjVf4Q20K7N8=; b=YZq+a/TsM955Aq2id9IXnqjSnL3/szeaXiYQAnkueRzIMx0pFAEYlHyc nYtgVjR4NOD8Cb7OzpvkYvATuVTvracmFCe5QfFjxspL7rMcPuZDI47+S Qc2qhXWqjHfdGnVfgGF2KWYpA8UDvSg4qRZLzJfinQ3OroRHcxUL1rpJx nvoW5lke7mRs8RI7XOk5P4snkmb7LRC+THXacDnZsXakUKFsW1nSBlCXc fqDjjh7HfnfM0fkOcl1TsaCcrecaUYzCU+NUfs4DIDcdgyKKccTyhz2Vf 4wRoZWYu0souwP/AfDPzUmmScLZffxqxJaDg/PRQu4Omwp9NHrMzFeBHR A==; X-IronPort-AV: E=McAfee;i="6500,9779,10442"; a="275802633" X-IronPort-AV: E=Sophos;i="5.93,246,1654585200"; d="scan'208";a="275802633" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Aug 2022 06:24:33 -0700 X-IronPort-AV: E=Sophos;i="5.93,246,1654585200"; d="scan'208";a="604253471" Received: from geigerri-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.251.215.246]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Aug 2022 06:24:24 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 68132104AA0; Thu, 18 Aug 2022 16:24:21 +0300 (+03) Date: Thu, 18 Aug 2022 16:24:21 +0300 From: "Kirill A . Shutemov" To: Hugh Dickins Cc: Chao Peng , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, linux-doc@vger.kernel.org, qemu-devel@nongnu.org, linux-kselftest@vger.kernel.org, Paolo Bonzini , Jonathan Corbet , Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H . Peter Anvin" , Jeff Layton , "J . Bruce Fields" , Andrew Morton , Shuah Khan , Mike Rapoport , Steven Price , "Maciej S . Szmigiero" , Vlastimil Babka , Vishal Annapurve , Yu Zhang , luto@kernel.org, jun.nakajima@intel.com, dave.hansen@intel.com, ak@linux.intel.com, david@redhat.com, aarcange@redhat.com, ddutile@redhat.com, dhildenb@redhat.com, Quentin Perret , Michael Roth , mhocko@suse.com, Muchun Song , "Gupta, Pankaj" Subject: Re: [PATCH v7 00/14] KVM: mm: fd-based approach for supporting KVM guest private memory Message-ID: <20220818132421.6xmjqduempmxnnu2@box> References: <20220706082016.2603916-1-chao.p.peng@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660859068; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LhMgiAtqYqXg9aZf+jRl/MchN9n3eO9crhybW8sQ1To=; b=IGALUzOfHxGXvVMnQH7TDxJ7c9d1cYuCJ6lPFX4owD/QDD2jL+SziPr2sDidEmjx9dQT7F TerFJCZBSeOLgjQExVIyLP/8IEkt0MkjIYlFRVbb66XEsFfP4HCigdRLslWisq0EF9PIBk 2D6n/Tu7KYIpISHlyHG3IJ+Q/cmOU6E= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b="YZq+a/Ts"; dmarc=fail reason="No valid SPF" header.from=intel.com (policy=none); spf=none (imf31.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.126) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660859068; a=rsa-sha256; cv=none; b=fCqsWDh1bApF+s5qdQeRdvGtwk9KKlOTPNGpg0FkkmAicaLXTee2iNom2zoWSdgup6EyFX xZv4tElqH3Z4AVbEp3UantOA/a4v06XSdsHk+Q1K0eR363V6gzKzNQq57STLpu6wlT9+H7 /hzH4kqur+PdhXXUb9WHXFT8GrZLxtE= Authentication-Results: imf31.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b="YZq+a/Ts"; dmarc=fail reason="No valid SPF" header.from=intel.com (policy=none); spf=none (imf31.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.126) smtp.mailfrom=kirill.shutemov@linux.intel.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 17B4B2326B X-Stat-Signature: urq9ns7zp9x4bzpnynewbj8es51sszi8 X-Rspam-User: X-HE-Tag: 1660859065-57334 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Aug 17, 2022 at 10:40:12PM -0700, Hugh Dickins wrote: > On Wed, 6 Jul 2022, Chao Peng wrote: > > This is the v7 of this series which tries to implement the fd-based KVM > > guest private memory. > > Here at last are my reluctant thoughts on this patchset. > > fd-based approach for supporting KVM guest private memory: fine. > > Use or abuse of memfd and shmem.c: mistaken. > > memfd_create() was an excellent way to put together the initial prototype. > > But since then, TDX in particular has forced an effort into preventing > (by flags, seals, notifiers) almost everything that makes it shmem/tmpfs. > > Are any of the shmem.c mods useful to existing users of shmem.c? No. > Is MFD_INACCESSIBLE useful or comprehensible to memfd_create() users? No. > > What use do you have for a filesystem here? Almost none. > IIUC, what you want is an fd through which QEMU can allocate kernel > memory, selectively free that memory, and communicate fd+offset+length > to KVM. And perhaps an interface to initialize a little of that memory > from a template (presumably copied from a real file on disk somewhere). > > You don't need shmem.c or a filesystem for that! > > If your memory could be swapped, that would be enough of a good reason > to make use of shmem.c: but it cannot be swapped; and although there > are some references in the mailthreads to it perhaps being swappable > in future, I get the impression that will not happen soon if ever. > > If your memory could be migrated, that would be some reason to use > filesystem page cache (because page migration happens to understand > that type of memory): but it cannot be migrated. Migration support is in pipeline. It is part of TDX 1.5 [1]. And swapping theoretically possible, but I'm not aware of any plans as of now. [1] https://www.intel.com/content/www/us/en/developer/articles/technical/intel-trust-domain-extensions.html > Some of these impressions may come from earlier iterations of the > patchset (v7 looks better in several ways than v5). I am probably > underestimating the extent to which you have taken on board other > usages beyond TDX and SEV private memory, and rightly want to serve > them all with similar interfaces: perhaps there is enough justification > for shmem there, but I don't see it. There was mention of userfaultfd > in one link: does that provide the justification for using shmem? > > I'm afraid of the special demands you may make of memory allocation > later on - surprised that huge pages are not mentioned already; > gigantic contiguous extents? secretmem removed from direct map? The design allows for extension to hugetlbfs if needed. Combination of MFD_INACCESSIBLE | MFD_HUGETLB should route this way. There should be zero implications for shmem. It is going to be separate struct memfile_backing_store. I'm not sure secretmem is a fit here as we want to extend MFD_INACCESSIBLE to be movable if platform supports it and secretmem is not migratable by design (without direct mapping fragmentations). > Here's what I would prefer, and imagine much easier for you to maintain; > but I'm no system designer, and may be misunderstanding throughout. > > QEMU gets fd from opening /dev/kvm_something, uses ioctls (or perhaps > the fallocate syscall interface itself) to allocate and free the memory, > ioctl for initializing some of it too. KVM in control of whether that > fd can be read or written or mmap'ed or whatever, no need to prevent it > in shmem.c, no need for flags, seals, notifications to and fro because > KVM is already in control and knows the history. If shmem actually has > value, call into it underneath - somewhat like SysV SHM, and /dev/zero > mmap, and i915/gem make use of it underneath. If shmem has nothing to > add, just allocate and free kernel memory directly, recorded in your > own xarray. I guess shim layer on top of shmem *can* work. I don't see immediately why it would not. But I'm not sure it is right direction. We risk creating yet another parallel VM with own rules/locking/accounting that opaque to core-mm. Note that on machines that run TDX guests such memory would likely be the bulk of memory use. Treating it as a fringe case may bite us one day. -- Kiryl Shutsemau / Kirill A. Shutemov