From: "Jarkko Sakkinen" <jarkko@kernel.org>
To: "Haitao Huang" <haitao.huang@linux.intel.com>,
<dave.hansen@linux.intel.com>, <tj@kernel.org>,
<mkoutny@suse.com>, <linux-kernel@vger.kernel.org>,
<linux-sgx@vger.kernel.org>, <x86@kernel.org>,
<cgroups@vger.kernel.org>, <tglx@linutronix.de>,
<mingo@redhat.com>, <bp@alien8.de>, <hpa@zytor.com>,
<sohil.mehta@intel.com>
Cc: <zhiquan1.li@intel.com>, <kristen@linux.intel.com>,
<seanjc@google.com>, <zhanb@microsoft.com>,
<anakrish@microsoft.com>, <mikko.ylinen@linux.intel.com>,
<yangjie@microsoft.com>, <chrisyan@microsoft.com>
Subject: Re: [PATCH v8 05/15] x86/sgx: Add sgx_epc_lru_list to encapsulate LRU list
Date: Fri, 02 Feb 2024 01:28:32 +0200 [thread overview]
Message-ID: <CYU4S0Y9SVZ2.2G9RM5Y3K28XI@suppilovahvero> (raw)
In-Reply-To: <20240130020938.10025-6-haitao.huang@linux.intel.com>
On Tue Jan 30, 2024 at 4:09 AM EET, Haitao Huang wrote:
> From: Sean Christopherson <sean.j.christopherson@intel.com>
>
> Introduce a data structure to wrap the existing reclaimable list and its
> spinlock. Each cgroup later will have one instance of this structure to
> track EPC pages allocated for processes associated with the same cgroup.
> Just like the global SGX reclaimer (ksgxd), an EPC cgroup reclaims pages
> from the reclaimable list in this structure when its usage reaches near
> its limit.
>
> Use this structure to encapsulate the LRU list and its lock used by the
> global reclaimer.
>
> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> Co-developed-by: Kristen Carlson Accardi <kristen@linux.intel.com>
> Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
> Co-developed-by: Haitao Huang <haitao.huang@linux.intel.com>
> Signed-off-by: Haitao Huang <haitao.huang@linux.intel.com>
I'd put author as last sob but that said I'm not sure if there is rigid
rule to do so. Thus not saying must here.
> Cc: Sean Christopherson <seanjc@google.com>
> ---
> V6:
> - removed introduction to unreclaimables in commit message.
>
> V4:
> - Removed unneeded comments for the spinlock and the non-reclaimables.
> (Kai, Jarkko)
> - Revised the commit to add introduction comments for unreclaimables and
> multiple LRU lists.(Kai)
> - Reordered the patches: delay all changes for unreclaimables to
> later, and this one becomes the first change in the SGX subsystem.
>
> V3:
> - Removed the helper functions and revised commit messages.
> ---
> arch/x86/kernel/cpu/sgx/main.c | 39 +++++++++++++++++-----------------
> arch/x86/kernel/cpu/sgx/sgx.h | 15 +++++++++++++
> 2 files changed, 35 insertions(+), 19 deletions(-)
>
> diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c
> index c32f18b70c73..912959c7ecc9 100644
> --- a/arch/x86/kernel/cpu/sgx/main.c
> +++ b/arch/x86/kernel/cpu/sgx/main.c
> @@ -28,10 +28,9 @@ static DEFINE_XARRAY(sgx_epc_address_space);
>
> /*
> * These variables are part of the state of the reclaimer, and must be accessed
> - * with sgx_reclaimer_lock acquired.
> + * with sgx_global_lru.lock acquired.
> */
> -static LIST_HEAD(sgx_active_page_list);
> -static DEFINE_SPINLOCK(sgx_reclaimer_lock);
> +static struct sgx_epc_lru_list sgx_global_lru;
>
> static atomic_long_t sgx_nr_free_pages = ATOMIC_LONG_INIT(0);
>
> @@ -306,13 +305,13 @@ static void sgx_reclaim_pages(void)
> int ret;
> int i;
>
> - spin_lock(&sgx_reclaimer_lock);
> + spin_lock(&sgx_global_lru.lock);
> for (i = 0; i < SGX_NR_TO_SCAN; i++) {
> - if (list_empty(&sgx_active_page_list))
> + epc_page = list_first_entry_or_null(&sgx_global_lru.reclaimable,
> + struct sgx_epc_page, list);
> + if (!epc_page)
> break;
>
> - epc_page = list_first_entry(&sgx_active_page_list,
> - struct sgx_epc_page, list);
> list_del_init(&epc_page->list);
> encl_page = epc_page->owner;
>
> @@ -324,7 +323,7 @@ static void sgx_reclaim_pages(void)
> */
> epc_page->flags &= ~SGX_EPC_PAGE_RECLAIMER_TRACKED;
> }
> - spin_unlock(&sgx_reclaimer_lock);
> + spin_unlock(&sgx_global_lru.lock);
>
> for (i = 0; i < cnt; i++) {
> epc_page = chunk[i];
> @@ -347,9 +346,9 @@ static void sgx_reclaim_pages(void)
> continue;
>
> skip:
> - spin_lock(&sgx_reclaimer_lock);
> - list_add_tail(&epc_page->list, &sgx_active_page_list);
> - spin_unlock(&sgx_reclaimer_lock);
> + spin_lock(&sgx_global_lru.lock);
> + list_add_tail(&epc_page->list, &sgx_global_lru.reclaimable);
> + spin_unlock(&sgx_global_lru.lock);
>
> kref_put(&encl_page->encl->refcount, sgx_encl_release);
>
> @@ -380,7 +379,7 @@ static void sgx_reclaim_pages(void)
> static bool sgx_should_reclaim(unsigned long watermark)
> {
> return atomic_long_read(&sgx_nr_free_pages) < watermark &&
> - !list_empty(&sgx_active_page_list);
> + !list_empty(&sgx_global_lru.reclaimable);
> }
>
> /*
> @@ -432,6 +431,8 @@ static bool __init sgx_page_reclaimer_init(void)
>
> ksgxd_tsk = tsk;
>
> + sgx_lru_init(&sgx_global_lru);
> +
> return true;
> }
>
> @@ -507,10 +508,10 @@ struct sgx_epc_page *__sgx_alloc_epc_page(void)
> */
> void sgx_mark_page_reclaimable(struct sgx_epc_page *page)
> {
> - spin_lock(&sgx_reclaimer_lock);
> + spin_lock(&sgx_global_lru.lock);
> page->flags |= SGX_EPC_PAGE_RECLAIMER_TRACKED;
> - list_add_tail(&page->list, &sgx_active_page_list);
> - spin_unlock(&sgx_reclaimer_lock);
> + list_add_tail(&page->list, &sgx_global_lru.reclaimable);
> + spin_unlock(&sgx_global_lru.lock);
> }
>
> /**
> @@ -525,18 +526,18 @@ void sgx_mark_page_reclaimable(struct sgx_epc_page *page)
> */
> int sgx_unmark_page_reclaimable(struct sgx_epc_page *page)
> {
> - spin_lock(&sgx_reclaimer_lock);
> + spin_lock(&sgx_global_lru.lock);
> if (page->flags & SGX_EPC_PAGE_RECLAIMER_TRACKED) {
> /* The page is being reclaimed. */
> if (list_empty(&page->list)) {
> - spin_unlock(&sgx_reclaimer_lock);
> + spin_unlock(&sgx_global_lru.lock);
> return -EBUSY;
> }
>
> list_del(&page->list);
> page->flags &= ~SGX_EPC_PAGE_RECLAIMER_TRACKED;
> }
> - spin_unlock(&sgx_reclaimer_lock);
> + spin_unlock(&sgx_global_lru.lock);
>
> return 0;
> }
> @@ -578,7 +579,7 @@ struct sgx_epc_page *sgx_alloc_epc_page(void *owner, bool reclaim)
> break;
> }
>
> - if (list_empty(&sgx_active_page_list)) {
> + if (list_empty(&sgx_global_lru.reclaimable)) {
> page = ERR_PTR(-ENOMEM);
> break;
> }
> diff --git a/arch/x86/kernel/cpu/sgx/sgx.h b/arch/x86/kernel/cpu/sgx/sgx.h
> index a898d86dead0..0e99e9ae3a67 100644
> --- a/arch/x86/kernel/cpu/sgx/sgx.h
> +++ b/arch/x86/kernel/cpu/sgx/sgx.h
> @@ -88,6 +88,21 @@ static inline void *sgx_get_epc_virt_addr(struct sgx_epc_page *page)
> return section->virt_addr + index * PAGE_SIZE;
> }
>
> +/*
> + * Contains EPC pages tracked by the global reclaimer (ksgxd) or an EPC
> + * cgroup.
> + */
> +struct sgx_epc_lru_list {
> + spinlock_t lock;
> + struct list_head reclaimable;
> +};
> +
> +static inline void sgx_lru_init(struct sgx_epc_lru_list *lru)
> +{
> + spin_lock_init(&lru->lock);
> + INIT_LIST_HEAD(&lru->reclaimable);
> +}
> +
> struct sgx_epc_page *__sgx_alloc_epc_page(void);
> void sgx_free_epc_page(struct sgx_epc_page *page);
>
Reviewed-by: Jarkko Sakkinen <jarkko@kernel.org>
BR, Jarkko
next prev parent reply other threads:[~2024-02-01 23:28 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-01-30 2:09 [PATCH v8 00/15] Add Cgroup support for SGX EPC memory Haitao Huang
2024-01-30 2:09 ` [PATCH v8 01/15] cgroup/misc: Add per resource callbacks for CSS events Haitao Huang
2024-02-01 23:24 ` Jarkko Sakkinen
2024-02-02 16:34 ` Haitao Huang
2024-02-02 17:02 ` Tejun Heo
2024-02-12 19:31 ` Jarkko Sakkinen
2024-01-30 2:09 ` [PATCH v8 02/15] cgroup/misc: Export APIs for SGX driver Haitao Huang
2024-01-30 2:09 ` [PATCH v8 03/15] cgroup/misc: Add SGX EPC resource type Haitao Huang
2024-01-30 2:09 ` [PATCH v8 04/15] x86/sgx: Implement basic EPC misc cgroup functionality Haitao Huang
2024-01-30 15:22 ` Huang, Kai
2024-01-31 1:59 ` Haitao Huang
2024-02-01 23:27 ` Jarkko Sakkinen
2024-01-30 2:09 ` [PATCH v8 05/15] x86/sgx: Add sgx_epc_lru_list to encapsulate LRU list Haitao Huang
2024-02-01 23:28 ` Jarkko Sakkinen [this message]
2024-02-02 16:52 ` Haitao Huang
2024-01-30 2:09 ` [PATCH v8 06/15] x86/sgx: Abstract tracking reclaimable pages in LRU Haitao Huang
2024-02-01 23:29 ` Jarkko Sakkinen
2024-01-30 2:09 ` [PATCH v8 07/15] x86/sgx: Expose sgx_reclaim_pages() for cgroup Haitao Huang
2024-01-30 15:39 ` Huang, Kai
2024-01-31 1:35 ` Haitao Huang
2024-02-01 23:37 ` Jarkko Sakkinen
2024-01-30 2:09 ` [PATCH v8 08/15] x86/sgx: Implement EPC reclamation flows " Haitao Huang
2024-02-01 23:39 ` Jarkko Sakkinen
2024-01-30 2:09 ` [PATCH v8 09/15] x86/sgx: Charge mem_cgroup for per-cgroup reclamation Haitao Huang
2024-01-30 2:09 ` [PATCH v8 10/15] x86/sgx: Add EPC reclamation in cgroup try_charge() Haitao Huang
2024-01-30 2:09 ` [PATCH v8 11/15] x86/sgx: Abstract check for global reclaimable pages Haitao Huang
2024-01-30 2:09 ` [PATCH v8 12/15] x86/sgx: Expose sgx_epc_cgroup_reclaim_pages() for global reclaimer Haitao Huang
2024-01-30 2:09 ` [PATCH v8 13/15] x86/sgx: Turn on per-cgroup EPC reclamation Haitao Huang
2024-01-30 2:09 ` [PATCH v8 14/15] Docs/x86/sgx: Add description for cgroup support Haitao Huang
2024-01-30 2:09 ` [PATCH v8 15/15] selftests/sgx: Add scripts for EPC cgroup testing Haitao Huang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CYU4S0Y9SVZ2.2G9RM5Y3K28XI@suppilovahvero \
--to=jarkko@kernel.org \
--cc=anakrish@microsoft.com \
--cc=bp@alien8.de \
--cc=cgroups@vger.kernel.org \
--cc=chrisyan@microsoft.com \
--cc=dave.hansen@linux.intel.com \
--cc=haitao.huang@linux.intel.com \
--cc=hpa@zytor.com \
--cc=kristen@linux.intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-sgx@vger.kernel.org \
--cc=mikko.ylinen@linux.intel.com \
--cc=mingo@redhat.com \
--cc=mkoutny@suse.com \
--cc=seanjc@google.com \
--cc=sohil.mehta@intel.com \
--cc=tglx@linutronix.de \
--cc=tj@kernel.org \
--cc=x86@kernel.org \
--cc=yangjie@microsoft.com \
--cc=zhanb@microsoft.com \
--cc=zhiquan1.li@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).