From: "Huang, Kai" <kai.huang@intel.com>
To: "Mehta, Sohil" <sohil.mehta@intel.com>,
"mingo@redhat.com" <mingo@redhat.com>,
"jarkko@kernel.org" <jarkko@kernel.org>,
"x86@kernel.org" <x86@kernel.org>,
"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
"cgroups@vger.kernel.org" <cgroups@vger.kernel.org>,
"hpa@zytor.com" <hpa@zytor.com>,
"tim.c.chen@linux.intel.com" <tim.c.chen@linux.intel.com>,
"linux-sgx@vger.kernel.org" <linux-sgx@vger.kernel.org>,
"mkoutny@suse.com" <mkoutny@suse.com>,
"haitao.huang@linux.intel.com" <haitao.huang@linux.intel.com>,
"tglx@linutronix.de" <tglx@linutronix.de>,
"tj@kernel.org" <tj@kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"bp@alien8.de" <bp@alien8.de>
Cc: "mikko.ylinen@linux.intel.com" <mikko.ylinen@linux.intel.com>,
"seanjc@google.com" <seanjc@google.com>,
"anakrish@microsoft.com" <anakrish@microsoft.com>,
"Zhang, Bo" <zhanb@microsoft.com>,
"kristen@linux.intel.com" <kristen@linux.intel.com>,
"yangjie@microsoft.com" <yangjie@microsoft.com>,
"Li, Zhiquan1" <zhiquan1.li@intel.com>,
"chrisyan@microsoft.com" <chrisyan@microsoft.com>
Subject: Re: [PATCH v9 08/15] x86/sgx: Implement EPC reclamation flows for cgroup
Date: Wed, 21 Feb 2024 10:48:58 +0000 [thread overview]
Message-ID: <bf21f955c1b56ef836ad03bc42d522b6d020edbf.camel@intel.com> (raw)
In-Reply-To: <op.2jhao6nzwjvjmi@hhuan26-mobl.amr.corp.intel.com>
On Wed, 2024-02-21 at 00:23 -0600, Haitao Huang wrote:
> StartHi Kai
> On Tue, 20 Feb 2024 03:52:39 -0600, Huang, Kai <kai.huang@intel.com> wrote:
> [...]
> >
> > So you introduced the work/workqueue here but there's no place which
> > actually
> > queues the work. IMHO you can either:
> >
> > 1) move relevant code change here; or
> > 2) focus on introducing core functions to reclaim certain pages from a
> > given EPC
> > cgroup w/o workqueue and introduce the work/workqueue in later patch.
> >
> > Makes sense?
> >
>
> Starting in v7, I was trying to split the big patch, #10 in v6 as you and
> others suggested. My thought process was to put infrastructure needed for
> per-cgroup reclaim in the front, then turn on per-cgroup reclaim in [v9
> 13/15] in the end.
That's reasonable for sure.
>
> Before that, all reclaimables are tracked in the global LRU so really
> there is no "reclaim certain pages from a given EPC cgroup w/o workqueue"
> or reclaim through workqueue before that point, as suggested in #2. This
> patch puts down the implementation for both flows but neither used yet, as
> stated in the commit message.
I know it's not used yet. The point is how to split patches to make them more
self-contain and easy to review.
For #2, sorry for not being explicit -- I meant it seems it's more reasonable to
split in this way:
Patch 1)
a). change to sgx_reclaim_pages();
b). introduce sgx_epc_cgroup_reclaim_pages();
c). introduce sgx_epc_cgroup_reclaim_work_func() (use a better name),
which just takes an EPC cgroup as input w/o involving any work/workqueue.
These functions are all related to how to implement reclaiming pages from a
given EPC cgroup, and they are logically related in terms of implementation thus
it's easier to be reviewed together.
Then you just need to justify the design/implementation in changelog/comments.
Patch 2)
- Introduce work/workqueue, and implement the logic to queue the work.
Now we all know there's a function to reclaim pages for a given EPC cgroup, then
we can focus on when that is called, either directly or indirectly.
>
> #1 would force me go back and merge the patches again.
I don't think so. I am not asking to put all things together, but only asking
to split in better way (that I see).
You mentioned some function is "Scheduled by sgx_epc_cgroup_try_charge() to
reclaim pages", but I am not seeing any code doing that in this patch. This
needs fixing, either by moving relevant code here, or removing these not-done-
yet comments.
For instance (I am just giving an example), if after review we found the
queue_work() shouldn't be done in try_charge(), you will need to go back to this
patch and remove these comments.
That's not the best way. Each patch needs to be self-contained.
>
> Sorry I feel kind of lost on this whole thing by now. It seems so random
> to me. Is there hard rules on this?
See above. I am only offering my opinion on how to split patches in better way.
>
> I was hoping these statements would help reviewers on the flow of the
> patches.
>
> At the end of [v9 04/15]:
>
> For now, the EPC cgroup simply blocks additional EPC allocation in
> sgx_alloc_epc_page() when the limit is reached. Reclaimable pages are
> still tracked in the global active list, only reclaimed by the global
> reclaimer when the total free page count is lower than a threshold.
>
> Later patches will reorganize the tracking and reclamation code in the
> global reclaimer and implement per-cgroup tracking and reclaiming.
>
> At the end of [v9 06/15]:
>
> Next patches will first get the cgroup reclamation flow ready while
> keeping pages tracked in the global LRU and reclaimed by ksgxd before we
> make the switch in the end for sgx_lru_list() to return per-cgroup
> LRU.
>
> At the end of [v9 08/15]:
>
> Both synchronous and asynchronous flows invoke the same top level reclaim
> function, and will be triggered later by sgx_epc_cgroup_try_charge()
> when usage of the cgroup is at or near its limit.
>
> At the end of [v9 10/15]:
> Note at this point, all reclaimable EPC pages are still tracked in the
> global LRU and per-cgroup LRUs are empty. So no per-cgroup reclamation
> is activated yet.
They are useful in the changelog in each patch I suppose, but to me we are
discussing different things.
I found one pain in the review is I have to jump back and forth many times among
multiple patches to see whether one patch is reasonable. That's why I am asking
whether there's better way to split patches so that each patch can be self-
contained logically in someway and easier to review.
next prev parent reply other threads:[~2024-02-21 10:49 UTC|newest]
Thread overview: 106+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-02-05 21:06 [PATCH v9 00/15] Add Cgroup support for SGX EPC memory Haitao Huang
2024-02-05 21:06 ` [PATCH v9 01/15] cgroup/misc: Add per resource callbacks for CSS events Haitao Huang
2024-02-05 21:06 ` [PATCH v9 02/15] cgroup/misc: Export APIs for SGX driver Haitao Huang
2024-02-05 21:06 ` [PATCH v9 03/15] cgroup/misc: Add SGX EPC resource type Haitao Huang
2024-02-05 21:06 ` [PATCH v9 04/15] x86/sgx: Implement basic EPC misc cgroup functionality Haitao Huang
2024-02-19 12:47 ` Huang, Kai
2024-02-26 18:25 ` Michal Koutný
2024-02-27 21:35 ` Haitao Huang
2024-03-09 21:10 ` Haitao Huang
2024-02-05 21:06 ` [PATCH v9 05/15] x86/sgx: Add sgx_epc_lru_list to encapsulate LRU list Haitao Huang
2024-02-05 21:06 ` [PATCH v9 06/15] x86/sgx: Abstract tracking reclaimable pages in LRU Haitao Huang
2024-02-05 21:06 ` [PATCH v9 07/15] x86/sgx: Expose sgx_reclaim_pages() for cgroup Haitao Huang
2024-02-20 9:26 ` Huang, Kai
2024-02-05 21:06 ` [PATCH v9 08/15] x86/sgx: Implement EPC reclamation flows " Haitao Huang
2024-02-12 19:35 ` Jarkko Sakkinen
2024-02-20 9:52 ` Huang, Kai
2024-02-20 13:18 ` Michal Koutný
2024-02-20 20:09 ` Huang, Kai
2024-02-21 6:23 ` Haitao Huang
2024-02-21 10:48 ` Huang, Kai [this message]
2024-02-22 20:12 ` Haitao Huang
2024-02-22 22:24 ` Huang, Kai
2024-03-28 0:24 ` Haitao Huang
2024-02-21 6:44 ` Haitao Huang
2024-02-21 11:00 ` Huang, Kai
2024-02-22 17:20 ` Haitao Huang
2024-02-22 22:31 ` Huang, Kai
2024-02-22 18:09 ` Haitao Huang
2024-02-05 21:06 ` [PATCH v9 09/15] x86/sgx: Charge mem_cgroup for per-cgroup reclamation Haitao Huang
2024-02-12 19:46 ` Jarkko Sakkinen
2024-02-13 3:21 ` Haitao Huang
2024-02-15 23:43 ` Dave Hansen
2024-02-16 6:07 ` Haitao Huang
2024-02-16 15:15 ` Dave Hansen
2024-02-16 21:38 ` Haitao Huang
2024-02-16 21:55 ` Dave Hansen
2024-02-16 23:33 ` Haitao Huang
2024-02-05 21:06 ` [PATCH v9 10/15] x86/sgx: Add EPC reclamation in cgroup try_charge() Haitao Huang
2024-02-12 19:55 ` Jarkko Sakkinen
2024-02-12 23:15 ` Haitao Huang
2024-02-14 1:52 ` Jarkko Sakkinen
2024-02-19 15:12 ` Haitao Huang
2024-02-19 20:20 ` Jarkko Sakkinen
2024-02-19 15:39 ` [RFC PATCH] x86/sgx: Remove 'reclaim' boolean parameters Haitao Huang
2024-02-19 15:56 ` Dave Hansen
2024-02-19 20:42 ` Jarkko Sakkinen
2024-02-19 22:25 ` Haitao Huang
2024-02-19 22:43 ` Jarkko Sakkinen
2024-02-19 20:23 ` Jarkko Sakkinen
2024-02-21 11:06 ` [PATCH v9 10/15] x86/sgx: Add EPC reclamation in cgroup try_charge() Huang, Kai
2024-02-22 17:09 ` Haitao Huang
2024-02-22 21:26 ` Huang, Kai
2024-02-22 22:57 ` Haitao Huang
2024-02-23 10:18 ` Huang, Kai
2024-02-23 17:00 ` Haitao Huang
2024-02-26 1:38 ` Huang, Kai
2024-02-26 4:03 ` Haitao Huang
2024-02-26 11:36 ` Huang, Kai
2024-02-26 14:04 ` Dave Hansen
2024-02-26 21:48 ` Haitao Huang
2024-02-26 21:56 ` Dave Hansen
2024-02-26 22:34 ` Huang, Kai
2024-02-26 22:38 ` Dave Hansen
2024-02-26 22:46 ` Huang, Kai
2024-02-27 20:41 ` Jarkko Sakkinen
2024-02-27 9:26 ` Michal Koutný
2024-02-26 21:18 ` Haitao Huang
2024-02-26 22:24 ` Huang, Kai
2024-02-26 22:31 ` Dave Hansen
2024-02-26 22:38 ` Huang, Kai
2024-02-05 21:06 ` [PATCH v9 11/15] x86/sgx: Abstract check for global reclaimable pages Haitao Huang
2024-02-12 19:56 ` Jarkko Sakkinen
2024-02-21 11:34 ` Huang, Kai
2024-02-05 21:06 ` [PATCH v9 12/15] x86/sgx: Expose sgx_epc_cgroup_reclaim_pages() for global reclaimer Haitao Huang
2024-02-12 19:58 ` Jarkko Sakkinen
2024-02-21 11:10 ` Huang, Kai
2024-02-22 16:35 ` Haitao Huang
2024-02-05 21:06 ` [PATCH v9 13/15] x86/sgx: Turn on per-cgroup EPC reclamation Haitao Huang
2024-02-21 11:23 ` Huang, Kai
2024-02-22 16:36 ` Haitao Huang
2024-02-22 22:44 ` Huang, Kai
2024-02-23 18:46 ` Haitao Huang
2024-02-05 21:06 ` [PATCH v9 14/15] Docs/x86/sgx: Add description for cgroup support Haitao Huang
2024-02-05 21:06 ` [PATCH v9 15/15] selftests/sgx: Add scripts for EPC cgroup testing Haitao Huang
2024-03-27 12:55 ` Jarkko Sakkinen
2024-03-27 16:56 ` Jarkko Sakkinen
2024-03-28 0:57 ` Haitao Huang
2024-03-28 3:05 ` Haitao Huang
2024-03-30 11:23 ` Jarkko Sakkinen
2024-03-30 11:26 ` Jarkko Sakkinen
2024-04-02 11:23 ` Michal Koutný
2024-04-02 11:58 ` Jarkko Sakkinen
2024-04-02 16:20 ` Haitao Huang
2024-04-02 17:40 ` Michal Koutný
2024-04-02 18:20 ` Haitao Huang
2024-04-03 16:46 ` Jarkko Sakkinen
2024-04-03 15:33 ` Jarkko Sakkinen
2024-04-02 15:42 ` Dave Hansen
2024-04-03 15:16 ` Jarkko Sakkinen
2024-03-28 3:54 ` Haitao Huang
2024-03-30 11:15 ` Jarkko Sakkinen
2024-03-30 15:32 ` Haitao Huang
2024-03-31 16:19 ` Jarkko Sakkinen
2024-03-31 17:35 ` Haitao Huang
2024-04-01 14:10 ` Jarkko Sakkinen
2024-02-08 8:43 ` [PATCH v9 00/15] Add Cgroup support for SGX EPC memory Mikko Ylinen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=bf21f955c1b56ef836ad03bc42d522b6d020edbf.camel@intel.com \
--to=kai.huang@intel.com \
--cc=anakrish@microsoft.com \
--cc=bp@alien8.de \
--cc=cgroups@vger.kernel.org \
--cc=chrisyan@microsoft.com \
--cc=dave.hansen@linux.intel.com \
--cc=haitao.huang@linux.intel.com \
--cc=hpa@zytor.com \
--cc=jarkko@kernel.org \
--cc=kristen@linux.intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-sgx@vger.kernel.org \
--cc=mikko.ylinen@linux.intel.com \
--cc=mingo@redhat.com \
--cc=mkoutny@suse.com \
--cc=seanjc@google.com \
--cc=sohil.mehta@intel.com \
--cc=tglx@linutronix.de \
--cc=tim.c.chen@linux.intel.com \
--cc=tj@kernel.org \
--cc=x86@kernel.org \
--cc=yangjie@microsoft.com \
--cc=zhanb@microsoft.com \
--cc=zhiquan1.li@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).