Linux-mm Archive on lore.kernel.org
 help / color / Atom feed
From: Sean Christopherson <sean.j.christopherson@intel.com>
To: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Cc: Borislav Petkov <bp@alien8.de>,
	x86@kernel.org, linux-sgx@vger.kernel.org,
	linux-kernel@vger.kernel.org, linux-mm@kvack.org,
	Jethro Beekman <jethro@fortanix.com>,
	Jordan Hand <jorhand@linux.microsoft.com>,
	Nathaniel McCallum <npmccallum@redhat.com>,
	Chunyang Hui <sanqian.hcy@antfin.com>,
	Seth Moore <sethmo@google.com>,
	akpm@linux-foundation.org, andriy.shevchenko@linux.intel.com,
	asapek@google.com, cedric.xing@intel.com,
	chenalexchen@google.com, conradparker@google.com,
	cyhanish@google.com, dave.hansen@intel.com,
	haitao.huang@intel.com, josh@joshtriplett.org,
	kai.huang@intel.com, kai.svahn@intel.com, kmoy@google.com,
	ludloff@google.com, luto@kernel.org, nhorman@redhat.com,
	puiterwijk@redhat.com, rientjes@google.com, tglx@linutronix.de,
	yaozhangx@google.com
Subject: Re: [PATCH v38 16/24] x86/sgx: Add a page reclaimer
Date: Mon, 28 Sep 2020 18:14:39 -0700
Message-ID: <20200929011438.GA31167@linux.intel.com> (raw)
In-Reply-To: <20200922140314.GA164527@linux.intel.com>

On Tue, Sep 22, 2020 at 05:03:23PM +0300, Jarkko Sakkinen wrote:
> On Tue, Sep 22, 2020 at 12:45:38PM +0200, Borislav Petkov wrote:
> > > +	spin_lock(&sgx_active_page_list_lock);
> > > +	for (i = 0; i < SGX_NR_TO_SCAN; i++) {
> > > +		if (list_empty(&sgx_active_page_list))
> > 
> > Isn't it enough to do this once, i.e., not in the loop? You're holding
> > sgx_active_page_list_lock...

Argh, I missed this until I looked at Jarkko's updated tree.

The reason for checking list_empty() on every iteration is that the loop is
greedy, i.e. it tries to grab and reclaim up to 16 (SGX_NR_TO_SCAN) EPC pages
at a time.

> I think that would make sense. Distantly analogous to the EINIT
> discussion. Too complex code for yet to be known problem workloads I'd
> say.

Nooooo.  Please no.

This will most definitely impact workloads running a single large enclave, or a
process running multiple enclaves, as we'll lose the batching of ETRACK and IPIs.
ETRACK isn't a big deal, but the IPIs are painful and could thrash the system.
E.g. in the current code, reclaiming 10 pages from an enclave with actively
running threads will generate one round of IPIs to CPUs associated with the
enclave (based on the mm struct).  Going to per-page reclaim will likely result
in 10 rounds of IPIs as threads will reenter the enclave immediately after each
IPI wave.

Having to grab the section spinlock for every page is also (very) problematic.
Reclaiming one page at a time makes it nigh impossible for the reclaimer to
keep up as every allocation attempt acquires the spinlock.  I.e. the reclaimer
has to contend with every CPU that is faulting in an EPC page or is adding a
page to the enclave.  In my experiments with the EPC cgroup, even batching 16
pages may be insufficient to make "forward progress".  That's not an apples to
apples comparison, but it's a rough equivalent of what will happen with the
reclaim thread versus EPC page faults.  We can (and should) play with scaling
the number of reclaim threads, but at this stage, that's an exercise best left
to post-upstreaming.

I can't do A/B testing to provide numbers, because the changes in Jarkko's tree
break basic reclaim.  Which is a nice segue into my last point: this is by far
the most subtle code in the SGX code base; I really, really don't want to be
making last minute changes in this area unless they are absolutely necessary or
obviously benign.  Even if/when we get the basic reclaim functional again, and
assuming it doesn't have performance issues, I wouldn't be comfortable merging
the code without hammering it with the EPC cgroup tests for multiple days (on
top of the 1+ days to rebased the EPC cgroup).  Past testing with the cgroup
found multiple bugs with races between CPUs that required hitting a window that
was open for less than 10 instructions.


  parent reply index

Thread overview: 81+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <20200915112842.897265-1-jarkko.sakkinen@linux.intel.com>
2020-09-15 11:28 ` [PATCH v38 10/24] mm: Add vm_ops->mprotect() Jarkko Sakkinen
2020-09-18 12:44   ` Borislav Petkov
2020-09-18 15:09   ` Andy Lutomirski
2020-09-18 23:24     ` [PATCH v38 10/24] mm: Add vm_ops->mprotect()' Jarkko Sakkinen
2020-09-18 23:53     ` [PATCH v38 10/24] mm: Add vm_ops->mprotect() Sean Christopherson
2020-09-19  0:15       ` Andy Lutomirski
2020-09-22 12:58         ` Jarkko Sakkinen
2020-09-22 15:11           ` Dave Hansen
2020-09-23 13:30             ` Jarkko Sakkinen
2020-09-23 13:43             ` Jarkko Sakkinen
2020-09-23 14:33             ` Jarkko Sakkinen
2020-09-24 14:50               ` Dave Hansen
2020-09-24 16:27                 ` Sean Christopherson
2020-09-24 19:35                 ` Jarkko Sakkinen
2020-09-21 12:49       ` Jarkko Sakkinen
2020-09-21 12:51         ` Jarkko Sakkinen
2020-09-21 13:14         ` Jarkko Sakkinen
2020-09-21 16:57         ` Sean Christopherson
2020-09-21 21:07           ` Jarkko Sakkinen
2020-09-21 21:18             ` Sean Christopherson
2020-09-22  5:29               ` Jarkko Sakkinen
2020-09-22  5:35                 ` Jarkko Sakkinen
2020-09-22 16:43                   ` Sean Christopherson
2020-09-23 13:50                     ` Jarkko Sakkinen
2020-09-24 19:11                       ` Haitao Huang
2020-09-24 19:28                         ` Sean Christopherson
2020-09-24 19:39                           ` Dave Hansen
2020-09-24 20:01                             ` Sean Christopherson
2020-09-24 20:10                               ` Dave Hansen
2020-09-24 20:25                                 ` Sean Christopherson
2020-09-24 20:54                                   ` Dave Hansen
2020-09-24 22:10                                     ` Jarkko Sakkinen
2020-09-24 23:05                                     ` Sean Christopherson
2020-09-24 23:09                                       ` Dave Hansen
2020-09-25  0:00                                         ` Sean Christopherson
2020-09-25 17:18                                           ` Dave Hansen
2020-09-25 19:43                                             ` Sean Christopherson
2020-09-25 19:53                                               ` Dave Hansen
2020-09-26  4:15                                                 ` Andy Lutomirski
2020-09-28  0:53                                                 ` Jarkko Sakkinen
2020-09-28 14:04                                                   ` Dave Hansen
2020-09-28 16:19                                                     ` Jarkko Sakkinen
2020-09-28 16:48                                                       ` Dave Hansen
2020-09-28 19:32                                                         ` Jarkko Sakkinen
2020-09-28 19:45                                                           ` Dave Hansen
2020-09-28 20:19                                                             ` Jarkko Sakkinen
2020-09-29  1:37                                                               ` Andy Lutomirski
2020-09-29  4:05                                                                 ` Jarkko Sakkinen
2020-09-29 14:24                                                                   ` Dave Hansen
2020-09-30  0:20                                                                     ` Jarkko Sakkinen
2020-09-30 14:35                                                                       ` Dave Hansen
2020-09-28 20:18                                                         ` Jarkko Sakkinen
2020-10-18  8:49                                                     ` Dr. Greg
2020-10-19 21:31                                                       ` Sean Christopherson
2020-10-20 10:01                                                         ` Dr. Greg
2020-10-20 16:40                                                           ` Sean Christopherson
2020-10-24 14:37                                                             ` Dr. Greg
2020-10-24 15:33                                                               ` Andy Lutomirski
2020-10-26 10:51                                                                 ` Dr. Greg
2020-10-26 22:59                                                                   ` Andy Lutomirski
2020-10-27  0:40                                                                     ` Sean Christopherson
2020-09-24 22:07                                 ` Jarkko Sakkinen
2020-09-24 21:58                           ` Jarkko Sakkinen
2020-09-24 21:55                         ` Jarkko Sakkinen
2020-09-15 11:28 ` [PATCH v38 11/24] x86/sgx: Add SGX enclave driver Jarkko Sakkinen
2020-09-21  9:30   ` Borislav Petkov
2020-09-21 12:09     ` Jarkko Sakkinen
2020-10-01 17:36   ` Sean Christopherson
2020-10-01 18:49     ` Jarkko Sakkinen
2020-09-15 11:28 ` [PATCH v38 16/24] x86/sgx: Add a page reclaimer Jarkko Sakkinen
2020-09-22 10:45   ` Borislav Petkov
2020-09-22 14:03     ` Jarkko Sakkinen
2020-09-22 14:24       ` Borislav Petkov
2020-09-23 14:52         ` Jarkko Sakkinen
2020-09-29  1:14       ` Sean Christopherson [this message]
2020-09-29  3:50         ` Jarkko Sakkinen
2020-09-29  8:35           ` Sean Christopherson
2020-09-22 16:24     ` Sean Christopherson
2020-09-22 18:02       ` Borislav Petkov
2020-09-23 15:25       ` Jarkko Sakkinen
     [not found] <20200915110522.893152-1-jarkko.sakkinen@linux.intel.com>
2020-09-15 11:05 ` Jarkko Sakkinen

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200929011438.GA31167@linux.intel.com \
    --to=sean.j.christopherson@intel.com \
    --cc=akpm@linux-foundation.org \
    --cc=andriy.shevchenko@linux.intel.com \
    --cc=asapek@google.com \
    --cc=bp@alien8.de \
    --cc=cedric.xing@intel.com \
    --cc=chenalexchen@google.com \
    --cc=conradparker@google.com \
    --cc=cyhanish@google.com \
    --cc=dave.hansen@intel.com \
    --cc=haitao.huang@intel.com \
    --cc=jarkko.sakkinen@linux.intel.com \
    --cc=jethro@fortanix.com \
    --cc=jorhand@linux.microsoft.com \
    --cc=josh@joshtriplett.org \
    --cc=kai.huang@intel.com \
    --cc=kai.svahn@intel.com \
    --cc=kmoy@google.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=linux-sgx@vger.kernel.org \
    --cc=ludloff@google.com \
    --cc=luto@kernel.org \
    --cc=nhorman@redhat.com \
    --cc=npmccallum@redhat.com \
    --cc=puiterwijk@redhat.com \
    --cc=rientjes@google.com \
    --cc=sanqian.hcy@antfin.com \
    --cc=sethmo@google.com \
    --cc=tglx@linutronix.de \
    --cc=x86@kernel.org \
    --cc=yaozhangx@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-mm Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-mm/0 linux-mm/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-mm linux-mm/ https://lore.kernel.org/linux-mm \
		linux-mm@kvack.org
	public-inbox-index linux-mm

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kvack.linux-mm


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git