sgx_reclaimer_write() refinement

sgx_reclaimer_write() refinement

[Jarkko Sakkinen]

Hi

I noticed that sgx_reclaimer_write() does not destory enclave albeit it
can fail. It should have a proper fallback mechanism.

I would take a long path to fix this by replacing sgx_reclaim_pages()
with

struct sgx_epc_page *sgx_reclaim_page(void)

It would greatly reduce the complexity of the flow. Would be much easier
to do fallback code paths. We have three trial EWB approach, which
should prevent already unnecesaary ETRACK's and IPI's.

The reclaimer thread would loop this until high watermark is reached.

The alloc function would change as:

struct sgx_epc_page *sgx_alloc_page(void *owner, bool reclaim)
{
	struct sgx_epc_page *entry;

	entry = sgx_try_alloc_page(owner);
	if (entry)
		break;

	if (list_empty(&sgx_active_page_list)) {
		entry = ERR_PTR(-ENOMEM);
		goto out;
	}

	if (!reclaim) {
		entry = ERR_PTR(-EBUSY);
		goto out;
	}

	entry = sgx_reclaim_page();

out:
	if (sgx_calc_free_cnt() < SGX_NR_LOW_PAGES)
		wake_up(&ksgxswapd_waitq);

	return entry;
}

We get rid of the loop because the reclaimer function will return the
EPC page. The reclaimer thread will add the page to the active page
list.

/Jarkko

Re: sgx_reclaimer_write() refinement

[Jarkko Sakkinen]

On Fri, Aug 23, 2019 at 10:08:09PM +0300, Jarkko Sakkinen wrote:
> It would greatly reduce the complexity of the flow. Would be much easier
> to do fallback code paths. We have three trial EWB approach, which
> should prevent already unnecesaary ETRACK's and IPI's.

I recall that I used a cluster ages ago to do minimize ETRACK's and
IPI's. I implemented three trial EWB during the Spring 2018 when the
reclaimer code was heavily refined but failed to notice that it rendered
cluster's meaningless...

/Jarkko

Re: sgx_reclaimer_write() refinement

[Jarkko Sakkinen]

On Fri, Aug 23, 2019 at 10:15:46PM +0300, Jarkko Sakkinen wrote:
> On Fri, Aug 23, 2019 at 10:08:09PM +0300, Jarkko Sakkinen wrote:
> > It would greatly reduce the complexity of the flow. Would be much easier
> > to do fallback code paths. We have three trial EWB approach, which
> > should prevent already unnecesaary ETRACK's and IPI's.
> 
> I recall that I used a cluster ages ago to do minimize ETRACK's and
> IPI's. I implemented three trial EWB during the Spring 2018 when the
> reclaimer code was heavily refined but failed to notice that it rendered
> cluster's meaningless...

One potential caveat here: EBLOCK's could never sequence before ETRACK
with a single thread.

However, the existing code does work in parallel already i.e. there can
be multiple sgx_alloc_page() calls on different going on. This change
would make sgx_alloc_page() success or fail (in the rare case of no
pages to swap) always in constant time and these would group naturally
behind ETRACK's.

What I would do for the reclaimer thread would be to convert it from a
kthread to a multi-threaded work queue (albeit even current single
threaded approach might work reasonably well as it goes in parallel with
the sgx_alloc_page() calls).

/Jarkko

Re: sgx_reclaimer_write() refinement

[Jarkko Sakkinen]

On Mon, Aug 26, 2019 at 06:23:40AM +0300, Jarkko Sakkinen wrote:
> On Fri, Aug 23, 2019 at 10:15:46PM +0300, Jarkko Sakkinen wrote:
> > On Fri, Aug 23, 2019 at 10:08:09PM +0300, Jarkko Sakkinen wrote:
> > > It would greatly reduce the complexity of the flow. Would be much easier
> > > to do fallback code paths. We have three trial EWB approach, which
> > > should prevent already unnecesaary ETRACK's and IPI's.
> > 
> > I recall that I used a cluster ages ago to do minimize ETRACK's and
> > IPI's. I implemented three trial EWB during the Spring 2018 when the
> > reclaimer code was heavily refined but failed to notice that it rendered
> > cluster's meaningless...
> 
> One potential caveat here: EBLOCK's could never sequence before ETRACK
> with a single thread.
> 
> However, the existing code does work in parallel already i.e. there can
> be multiple sgx_alloc_page() calls on different going on. This change

on differen threads

/Jarkko