Re: [PATCH v10 12/14] x86/sgx: Turn on per-cgroup EPC reclamation

["Haitao Huang" <haitao.huang@linux.intel.com>]

On Mon, 08 Apr 2024 07:20:23 -0500, Huang, Kai <kai.huang@intel.com> wrote:

>
>> --- a/arch/x86/kernel/cpu/sgx/epc_cgroup.h
>> +++ b/arch/x86/kernel/cpu/sgx/epc_cgroup.h
>> @@ -28,6 +28,10 @@ static inline int sgx_cgroup_try_charge(struct  
>> sgx_cgroup *sgx_cg, enum sgx_recl
>>  static inline void sgx_cgroup_uncharge(struct sgx_cgroup *sgx_cg) { }
>>
>>  static inline void sgx_cgroup_init(void) { }
>> +
>> +static inline void sgx_cgroup_reclaim_pages(struct misc_cg *root,  
>> struct mm_struct *charge_mm)
>> +{
>> +}
>>  #else
>>  struct sgx_cgroup {
>>  	struct misc_cg *cg;
>> @@ -65,6 +69,9 @@ static inline void sgx_put_cg(struct sgx_cgroup  
>> *sgx_cg)
>>
>>  int sgx_cgroup_try_charge(struct sgx_cgroup *sgx_cg, enum sgx_reclaim  
>> reclaim);
>>  void sgx_cgroup_uncharge(struct sgx_cgroup *sgx_cg);
>> +bool sgx_cgroup_lru_empty(struct misc_cg *root);
>> +bool sgx_cgroup_should_reclaim(struct sgx_cgroup *sgx_cg);
>> +void sgx_cgroup_reclaim_pages(struct misc_cg *root,  struct mm_struct  
>> *charge_mm);
>
> Seems the decision to choose whether to implement a stub function for  
> some
> function is quite random to me.
>
> My impression is people in general don't like #ifdef in the C file but  
> prefer to
> implementing it in the header in some helper function.
>
> I guess you might want to just implement a stub function for each of the  
> 3
> functions exposed, so that we can eliminate some #ifdefs in the  
> sgx/main.c (see
> below).
>
>>  void sgx_cgroup_init(void);
>>
>>  #endif
>> diff --git a/arch/x86/kernel/cpu/sgx/main.c  
>> b/arch/x86/kernel/cpu/sgx/main.c
>> index 7f92455d957d..68f28ff2d5ef 100644
>> --- a/arch/x86/kernel/cpu/sgx/main.c
>> +++ b/arch/x86/kernel/cpu/sgx/main.c
>> @@ -34,6 +34,16 @@ static struct sgx_epc_lru_list sgx_global_lru;
>>
>>  static inline struct sgx_epc_lru_list *sgx_lru_list(struct  
>> sgx_epc_page *epc_page)
>>  {
>> +#ifdef CONFIG_CGROUP_SGX_EPC
>> +	if (epc_page->sgx_cg)
>> +		return &epc_page->sgx_cg->lru;
>> +
>> +	/*
>> +	 * This should not happen when cgroup is enabled: Every page belongs
>> +	 * to a cgroup, or the root by default.
>> +	 */
>> +	WARN_ON_ONCE(1);
>
> In the case MISC cgroup is enabled in Kconfig but disabled by command  
> line, I
> think this becomes legal now?
>

I'm not sure actually. Never saw the warning even if I set  
"cgroup_disable=misc" in commandlibe. Tried both v1 and v2. Anyway, I  
think we can remove this warning and we handle the NULL sgx_cg now.

>> +#endif
>>  	return &sgx_global_lru;
>>  }
>>
>> @@ -42,7 +52,11 @@ static inline struct sgx_epc_lru_list  
>> *sgx_lru_list(struct sgx_epc_page *epc_pag
>>   */
>>  static inline bool sgx_can_reclaim(void)
>>  {
>> +#ifdef CONFIG_CGROUP_SGX_EPC
>> +	return !sgx_cgroup_lru_empty(misc_cg_root());
>> +#else
>>  	return !list_empty(&sgx_global_lru.reclaimable);
>> +#endif
>>  }
>>
>
> Here you are using #ifdef  CONFIG_CGRUP_SGX_EPC, but ...
>
>>  static atomic_long_t sgx_nr_free_pages = ATOMIC_LONG_INIT(0);
>> @@ -404,7 +418,10 @@ static bool sgx_should_reclaim(unsigned long  
>> watermark)
>>
>>  static void sgx_reclaim_pages_global(struct mm_struct *charge_mm)
>>  {
>> -	sgx_reclaim_pages(&sgx_global_lru, charge_mm);
>> +	if (IS_ENABLED(CONFIG_CGROUP_SGX_EPC))
>> +		sgx_cgroup_reclaim_pages(misc_cg_root(), charge_mm);
>> +	else
>> +		sgx_reclaim_pages(&sgx_global_lru, charge_mm);
>>  }
>
> ... here you are using IS_ENABLED(CONFIG_CGROUP_SGX_EPC).
>
> Any reason they are not consistent?

I was hesitant to expose sgx_global_lru needed for implementing  
sgx_cgroup_lru_empty() stub which would also be a random decision and  
overkill to just remove couple of #ifdefs in short functions.

>
> Also, in the case where MISC cgroup is disabled via commandline, I think  
> it
> won't work, because misc_cg_root() should be NULL in this case while
> IS_ENABLED(CONFIG_CGROUP_SGX_EPC) is true.
>
>>

The misc root cgroup is a static similar to sgx_cg_root. So  
misc_cg_root()  won't be NULL
However, based on how css_misc() was check NULL, I suppose  
sgx_get_current_cg() may be NULL when cgroup is disabled (again not 100%  
sure but we handle it anyway)

>>  /*
>> @@ -414,6 +431,16 @@ static void sgx_reclaim_pages_global(struct  
>> mm_struct *charge_mm)
>>   */
>>  void sgx_reclaim_direct(void)
>>  {
>> +#ifdef CONFIG_CGROUP_SGX_EPC
>> +	struct sgx_cgroup *sgx_cg = sgx_get_current_cg();
>> +
>> +	/* Make sure there are some free pages at cgroup level */
>> +	if (sgx_cg && sgx_cgroup_should_reclaim(sgx_cg)) {
>> +		sgx_cgroup_reclaim_pages(sgx_cg->cg, current->mm);
>> +		sgx_put_cg(sgx_cg);
>> +	}
>> +#endif
>
> This #ifdef CONFIG_CGROUP_SGX_EPC can be removed if we implement a stub  
> function
> for sgx_cgroup_should_reclaim().
>
Yes.
>> +	/* Make sure there are some free pages at global level */
>>  	if (sgx_should_reclaim(SGX_NR_LOW_PAGES))
>>  		sgx_reclaim_pages_global(current->mm);
>>  }
>

Thanks
Haitao