From: "Haitao Huang" <haitao.huang@linux.intel.com>
To: jarkko@kernel.org, dave.hansen@linux.intel.com, tj@kernel.org,
mkoutny@suse.com, linux-kernel@vger.kernel.org,
linux-sgx@vger.kernel.org, x86@kernel.org,
cgroups@vger.kernel.org, tglx@linutronix.de, mingo@redhat.com,
bp@alien8.de, hpa@zytor.com, sohil.mehta@intel.com,
tim.c.chen@linux.intel.com, "Huang, Kai" <kai.huang@intel.com>
Cc: zhiquan1.li@intel.com, kristen@linux.intel.com,
seanjc@google.com, zhanb@microsoft.com, anakrish@microsoft.com,
mikko.ylinen@linux.intel.com, yangjie@microsoft.com,
chrisyan@microsoft.com
Subject: Re: [PATCH v12 08/14] x86/sgx: Add basic EPC reclamation flow for cgroup
Date: Tue, 23 Apr 2024 10:53:21 -0500 [thread overview]
Message-ID: <op.2moue7exwjvjmi@hhuan26-mobl.amr.corp.intel.com> (raw)
In-Reply-To: <8a9ae08a-c813-442d-9fc3-031a4c984700@intel.com>
On Wed, 17 Apr 2024 18:51:28 -0500, Huang, Kai <kai.huang@intel.com> wrote:
>
>
> On 16/04/2024 3:20 pm, Haitao Huang wrote:
>> From: Kristen Carlson Accardi <kristen@linux.intel.com>
>> Currently in the EPC page allocation, the kernel simply fails the
>> allocation when the current EPC cgroup fails to charge due to its usage
>> reaching limit. This is not ideal. When that happens, a better way is
>> to reclaim EPC page(s) from the current EPC cgroup (and/or its
>> descendants) to reduce its usage so the new allocation can succeed.
>> Add the basic building blocks to support per-cgroup reclamation.
>> Currently the kernel only has one place to reclaim EPC pages: the
>> global
>> EPC LRU list. To support the "per-cgroup" EPC reclaim, maintain an LRU
>> list for each EPC cgroup, and introduce a "cgroup" variant function to
>> reclaim EPC pages from a given EPC cgroup and its descendants.
>> Currently the kernel does the global EPC reclaim in sgx_reclaim_page().
>> It always tries to reclaim EPC pages in batch of SGX_NR_TO_SCAN (16)
>> pages. Specifically, it always "scans", or "isolates" SGX_NR_TO_SCAN
>> pages from the global LRU, and then tries to reclaim these pages at once
>> for better performance.
>> Implement the "cgroup" variant EPC reclaim in a similar way, but keep
>> the implementation simple: 1) change sgx_reclaim_pages() to take an LRU
>> as input, and return the pages that are "scanned" and attempted for
>> reclamation (but not necessarily reclaimed successfully); 2) loop the
>> given EPC cgroup and its descendants and do the new sgx_reclaim_pages()
>> until SGX_NR_TO_SCAN pages are "scanned".
>> This implementation, encapsulated in sgx_cgroup_reclaim_pages(), always
>> tries to reclaim SGX_NR_TO_SCAN pages from the LRU of the given EPC
>> cgroup, and only moves to its descendants when there's no enough
>> reclaimable EPC pages to "scan" in its LRU. It should be enough for
>> most cases.
>> Note, this simple implementation doesn't _exactly_ mimic the current
>> global EPC reclaim (which always tries to do the actual reclaim in batch
>> of SGX_NR_TO_SCAN pages): when LRUs have less than SGX_NR_TO_SCAN
>> reclaimable pages, the actual reclaim of EPC pages will be split into
>> smaller batches _across_ multiple LRUs with each being smaller than
>> SGX_NR_TO_SCAN pages.
>> A more precise way to mimic the current global EPC reclaim would be to
>> have a new function to only "scan" (or "isolate") SGX_NR_TO_SCAN pages
>> _across_ the given EPC cgroup _AND_ its descendants, and then do the
>> actual reclaim in one batch. But this is unnecessarily complicated at
>> this stage.
>> Alternatively, the current sgx_reclaim_pages() could be changed to
>> return the actual "reclaimed" pages, but not "scanned" pages. However,
>> the reclamation is a lengthy process, forcing a successful reclamation
>> of predetermined number of pages may block the caller for too long. And
>> that may not be acceptable in some synchronous contexts, e.g., in
>> serving an ioctl().
>> With this building block in place, add synchronous reclamation support
>> in sgx_cgroup_try_charge(): trigger a call to
>> sgx_cgroup_reclaim_pages() if the cgroup reaches its limit and the
>> caller allows synchronous reclaim as indicated by s newly added
>> parameter.
>> A later patch will add support for asynchronous reclamation reusing
>> sgx_cgroup_reclaim_pages().
>> Note all reclaimable EPC pages are still tracked in the global LRU thus
>> no per-cgroup reclamation is actually active at the moment. Per-cgroup
>> tracking and reclamation will be turned on in the end after all
>> necessary infrastructure is in place.
>
> Nit:
>
> "all necessary infrastructures are in place", or, "all necessary
> building blocks are in place".
>
> ?
>
>> Co-developed-by: Sean Christopherson <sean.j.christopherson@intel.com>
>> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
>> Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
>> Co-developed-by: Haitao Huang <haitao.huang@linux.intel.com>
>> Signed-off-by: Haitao Huang <haitao.huang@linux.intel.com>
>> Tested-by: Jarkko Sakkinen <jarkko@kernel.org>
>> ---
>
> Reviewed-by: Kai Huang <kai.huang@intel.com>
>
Thanks
> More nitpickings below:
>
> [...]
>
>> -static inline int sgx_cgroup_try_charge(struct sgx_cgroup *sgx_cg)
>> +static inline int sgx_cgroup_try_charge(struct sgx_cgroup *sgx_cg,
>> enum sgx_reclaim reclaim)
>
> Let's still wrap the text on 80-character basis.
>
> I guess most people are more used to that.
>
> [...]
>
>> - epc_page = list_first_entry_or_null(&sgx_global_lru.reclaimable,
>> - struct sgx_epc_page, list);
>> + epc_page = list_first_entry_or_null(&lru->reclaimable, struct
>> sgx_epc_page, list);
>
> Ditto.
>
Actually I changed to 100 char width based on comments from Jarkko IIRC.
I don't have personal preference, but will not change back to 80 unless
Jarkko also agrees.
Thanks
Haitao
next prev parent reply other threads:[~2024-04-23 15:53 UTC|newest]
Thread overview: 64+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-16 3:19 [PATCH v12 00/14] Add Cgroup support for SGX EPC memory Haitao Huang
2024-04-16 3:19 ` [PATCH v12 01/14] x86/sgx: Replace boolean parameters with enums Haitao Huang
2024-04-16 3:19 ` [PATCH v12 02/14] cgroup/misc: Add per resource callbacks for CSS events Haitao Huang
2024-04-16 3:20 ` [PATCH v12 03/14] cgroup/misc: Export APIs for SGX driver Haitao Huang
2024-04-16 3:20 ` [PATCH v12 04/14] cgroup/misc: Add SGX EPC resource type Haitao Huang
2024-04-16 3:20 ` [PATCH v12 05/14] x86/sgx: Implement basic EPC misc cgroup functionality Haitao Huang
2024-04-16 13:22 ` Huang, Kai
2024-04-18 22:41 ` Haitao Huang
2024-04-18 23:29 ` Huang, Kai
2024-04-19 18:15 ` Haitao Huang
2024-04-19 22:21 ` Huang, Kai
2024-04-16 22:23 ` Haitao Huang
2024-04-16 3:20 ` [PATCH v12 06/14] x86/sgx: Add sgx_epc_lru_list to encapsulate LRU list Haitao Huang
2024-04-16 3:20 ` [PATCH v12 07/14] x86/sgx: Abstract tracking reclaimable pages in LRU Haitao Huang
2024-04-16 14:07 ` Huang, Kai
2024-04-16 22:48 ` Haitao Huang
2024-04-16 3:20 ` [PATCH v12 08/14] x86/sgx: Add basic EPC reclamation flow for cgroup Haitao Huang
2024-04-17 23:51 ` Huang, Kai
2024-04-23 15:53 ` Haitao Huang [this message]
2024-04-16 3:20 ` [PATCH v12 09/14] x86/sgx: Implement async reclamation " Haitao Huang
2024-04-19 1:32 ` Huang, Kai
2024-04-19 18:55 ` Haitao Huang
2024-04-19 22:44 ` Huang, Kai
2024-04-20 1:14 ` Haitao Huang
2024-04-22 0:22 ` Huang, Kai
2024-04-22 16:17 ` Haitao Huang
2024-04-22 22:16 ` Huang, Kai
2024-04-23 13:08 ` Haitao Huang
2024-04-23 14:19 ` Huang, Kai
2024-04-23 15:30 ` Haitao Huang
2024-04-23 22:13 ` Huang, Kai
2024-04-24 0:26 ` Haitao Huang
2024-04-24 2:13 ` Huang, Kai
2024-04-16 3:20 ` [PATCH v12 10/14] x86/sgx: Charge mem_cgroup for per-cgroup reclamation Haitao Huang
2024-04-23 7:21 ` Huang, Kai
2024-04-16 3:20 ` [PATCH v12 11/14] x86/sgx: Abstract check for global reclaimable pages Haitao Huang
2024-04-16 3:20 ` [PATCH v12 12/14] x86/sgx: Turn on per-cgroup EPC reclamation Haitao Huang
2024-04-29 10:49 ` Huang, Kai
2024-04-29 16:05 ` Haitao Huang
2024-04-29 22:18 ` Huang, Kai
2024-04-30 1:31 ` Haitao Huang
2024-04-16 3:20 ` [PATCH v12 13/14] Docs/x86/sgx: Add description for cgroup support Haitao Huang
2024-04-21 7:18 ` Bagas Sanjaya
2024-04-23 7:29 ` Huang, Kai
2024-04-16 3:20 ` [PATCH v12 14/14] selftests/sgx: Add scripts for EPC cgroup testing Haitao Huang
2024-04-16 5:16 ` Haitao Huang
2024-04-16 5:42 ` Huang, Kai
2024-04-16 14:15 ` Jarkko Sakkinen
2024-04-26 14:28 ` Dave Hansen
2024-04-28 22:03 ` Jarkko Sakkinen
2024-04-29 16:18 ` Haitao Huang
2024-04-29 16:43 ` Jarkko Sakkinen
2024-04-29 17:14 ` Haitao Huang
2024-04-16 15:00 ` Haitao Huang
2024-04-16 14:05 ` Jarkko Sakkinen
2024-04-16 14:10 ` Jarkko Sakkinen
2024-04-16 14:54 ` Haitao Huang
2024-04-16 16:08 ` Jarkko Sakkinen
2024-04-16 22:04 ` Haitao Huang
2024-04-16 22:21 ` Haitao Huang
2024-04-17 3:05 ` Haitao Huang
2024-04-17 22:46 ` Jarkko Sakkinen
2024-04-24 19:42 ` Haitao Huang
2024-04-25 4:51 ` Jarkko Sakkinen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=op.2moue7exwjvjmi@hhuan26-mobl.amr.corp.intel.com \
--to=haitao.huang@linux.intel.com \
--cc=anakrish@microsoft.com \
--cc=bp@alien8.de \
--cc=cgroups@vger.kernel.org \
--cc=chrisyan@microsoft.com \
--cc=dave.hansen@linux.intel.com \
--cc=hpa@zytor.com \
--cc=jarkko@kernel.org \
--cc=kai.huang@intel.com \
--cc=kristen@linux.intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-sgx@vger.kernel.org \
--cc=mikko.ylinen@linux.intel.com \
--cc=mingo@redhat.com \
--cc=mkoutny@suse.com \
--cc=seanjc@google.com \
--cc=sohil.mehta@intel.com \
--cc=tglx@linutronix.de \
--cc=tim.c.chen@linux.intel.com \
--cc=tj@kernel.org \
--cc=x86@kernel.org \
--cc=yangjie@microsoft.com \
--cc=zhanb@microsoft.com \
--cc=zhiquan1.li@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).