From: Haitao Huang <haitao.huang@linux.intel.com> To: jarkko@kernel.org, dave.hansen@linux.intel.com, tj@kernel.org, linux-kernel@vger.kernel.org, linux-sgx@vger.kernel.org, x86@kernel.org, cgroups@vger.kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, sohil.mehta@intel.com Cc: zhiquan1.li@intel.com, kristen@linux.intel.com, seanjc@google.com, zhanb@microsoft.com, anakrish@microsoft.com, mikko.ylinen@linux.intel.com, yangjie@microsoft.com Subject: [PATCH v5 17/18] Docs/x86/sgx: Add description for cgroup support Date: Fri, 22 Sep 2023 20:06:56 -0700 [thread overview] Message-ID: <20230923030657.16148-18-haitao.huang@linux.intel.com> (raw) In-Reply-To: <20230923030657.16148-1-haitao.huang@linux.intel.com> From: Sean Christopherson <sean.j.christopherson@intel.com> Add initial documentation of how to regulate the distribution of SGX Enclave Page Cache (EPC) memory via the Miscellaneous cgroup controller. Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com> Co-developed-by: Kristen Carlson Accardi <kristen@linux.intel.com> Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com> Co-developed-by: Haitao Huang<haitao.huang@linux.intel.com> Signed-off-by: Haitao Huang<haitao.huang@linux.intel.com> Cc: Sean Christopherson <seanjc@google.com> Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com> --- V4: - Fix indentation (Randy) - Change misc.events file to be read-only - Fix a typo for 'subsystem' - Add behavior when VMM overcommit EPC with a cgroup (Mikko) --- Documentation/arch/x86/sgx.rst | 82 ++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) diff --git a/Documentation/arch/x86/sgx.rst b/Documentation/arch/x86/sgx.rst index d90796adc2ec..65c211bd5342 100644 --- a/Documentation/arch/x86/sgx.rst +++ b/Documentation/arch/x86/sgx.rst @@ -300,3 +300,85 @@ to expected failures and handle them as follows: first call. It indicates a bug in the kernel or the userspace client if any of the second round of ``SGX_IOC_VEPC_REMOVE_ALL`` calls has a return code other than 0. + + +Cgroup Support +============== + +The "sgx_epc" resource within the Miscellaneous cgroup controller regulates +distribution of SGX EPC memory, which is a subset of system RAM that +is used to provide SGX-enabled applications with protected memory, +and is otherwise inaccessible, i.e. shows up as reserved in +/proc/iomem and cannot be read/written outside of an SGX enclave. + +Although current systems implement EPC by stealing memory from RAM, +for all intents and purposes the EPC is independent from normal system +memory, e.g. must be reserved at boot from RAM and cannot be converted +between EPC and normal memory while the system is running. The EPC is +managed by the SGX subsystem and is not accounted by the memory +controller. Note that this is true only for EPC memory itself, i.e. +normal memory allocations related to SGX and EPC memory, e.g. the +backing memory for evicted EPC pages, are accounted, limited and +protected by the memory controller. + +Much like normal system memory, EPC memory can be overcommitted via +virtual memory techniques and pages can be swapped out of the EPC +to their backing store (normal system memory allocated via shmem). +The SGX EPC subsystem is analogous to the memory subsystem, and +it implements limit and protection models for EPC memory. + +SGX EPC Interface Files +----------------------- + +For a generic description of the Miscellaneous controller interface +files, please see Documentation/admin-guide/cgroup-v2.rst + +All SGX EPC memory amounts are in bytes unless explicitly stated +otherwise. If a value which is not PAGE_SIZE aligned is written, +the actual value used by the controller will be rounded down to +the closest PAGE_SIZE multiple. + + misc.capacity + A read-only flat-keyed file shown only in the root cgroup. + The sgx_epc resource will show the total amount of EPC + memory available on the platform. + + misc.current + A read-only flat-keyed file shown in the non-root cgroups. + The sgx_epc resource will show the current active EPC memory + usage of the cgroup and its descendants. EPC pages that are + swapped out to backing RAM are not included in the current count. + + misc.max + A read-write single value file which exists on non-root + cgroups. The sgx_epc resource will show the EPC usage + hard limit. The default is "max". + + If a cgroup's EPC usage reaches this limit, EPC allocations, + e.g. for page fault handling, will be blocked until EPC can + be reclaimed from the cgroup. If EPC cannot be reclaimed in + a timely manner, reclaim will be forced, e.g. by ignoring LRU. + + The EPC pages allocated for KVM guests by the virtual EPC driver + are not reclaimable by the host kernel SGX reclaimers. If a VMM + tries to start a VM within a cgroup whose EPC usage reaches this + limit, the virtual EPC driver will stop allocating more EPC for the + VM, and return SIGBUS to the VMM which would abort the VM launch. + + misc.events + A read-only flat-keyed file which exists on non-root cgroups. + A value change in this file generates a file modified event. + + max + The number of times the cgroup has triggered a reclaim + due to its EPC usage approaching (or exceeding) its max + EPC boundary. + +Migration +--------- + +Once an EPC page is charged to a cgroup (during allocation), it +remains charged to the original cgroup until the page is released +or reclaimed. Migrating a process to a different cgroup doesn't +move the EPC charges that it incurred while in the previous cgroup +to its new cgroup. -- 2.25.1
WARNING: multiple messages have this Message-ID (diff)
From: Haitao Huang <haitao.huang-VuQAYsv1563Yd54FQh9/CA@public.gmane.org> To: jarkko-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, dave.hansen-VuQAYsv1563Yd54FQh9/CA@public.gmane.org, tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-sgx-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, x86-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, tglx-hfZtesqFncYOwBW4kG4KsQ@public.gmane.org, mingo-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org, bp-Gina5bIWoIWzQB+pC5nmwQ@public.gmane.org, hpa-YMNOUZJC4hwAvxtiuMwx3w@public.gmane.org, sohil.mehta-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org Cc: zhiquan1.li-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org, kristen-VuQAYsv1563Yd54FQh9/CA@public.gmane.org, seanjc-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org, zhanb-0li6OtcxBFHby3iVrkZq2A@public.gmane.org, anakrish-0li6OtcxBFHby3iVrkZq2A@public.gmane.org, mikko.ylinen-VuQAYsv1563Yd54FQh9/CA@public.gmane.org, yangjie-0li6OtcxBFHby3iVrkZq2A@public.gmane.org Subject: [PATCH v5 17/18] Docs/x86/sgx: Add description for cgroup support Date: Fri, 22 Sep 2023 20:06:56 -0700 [thread overview] Message-ID: <20230923030657.16148-18-haitao.huang@linux.intel.com> (raw) In-Reply-To: <20230923030657.16148-1-haitao.huang-VuQAYsv1563Yd54FQh9/CA@public.gmane.org> From: Sean Christopherson <sean.j.christopherson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> Add initial documentation of how to regulate the distribution of SGX Enclave Page Cache (EPC) memory via the Miscellaneous cgroup controller. Signed-off-by: Sean Christopherson <sean.j.christopherson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> Co-developed-by: Kristen Carlson Accardi <kristen-VuQAYsv1563Yd54FQh9/CA@public.gmane.org> Signed-off-by: Kristen Carlson Accardi <kristen-VuQAYsv1563Yd54FQh9/CA@public.gmane.org> Co-developed-by: Haitao Huang<haitao.huang-VuQAYsv1563Yd54FQh9/CA@public.gmane.org> Signed-off-by: Haitao Huang<haitao.huang-VuQAYsv1563Yd54FQh9/CA@public.gmane.org> Cc: Sean Christopherson <seanjc-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> Reviewed-by: Bagas Sanjaya <bagasdotme-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> --- V4: - Fix indentation (Randy) - Change misc.events file to be read-only - Fix a typo for 'subsystem' - Add behavior when VMM overcommit EPC with a cgroup (Mikko) --- Documentation/arch/x86/sgx.rst | 82 ++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) diff --git a/Documentation/arch/x86/sgx.rst b/Documentation/arch/x86/sgx.rst index d90796adc2ec..65c211bd5342 100644 --- a/Documentation/arch/x86/sgx.rst +++ b/Documentation/arch/x86/sgx.rst @@ -300,3 +300,85 @@ to expected failures and handle them as follows: first call. It indicates a bug in the kernel or the userspace client if any of the second round of ``SGX_IOC_VEPC_REMOVE_ALL`` calls has a return code other than 0. + + +Cgroup Support +============== + +The "sgx_epc" resource within the Miscellaneous cgroup controller regulates +distribution of SGX EPC memory, which is a subset of system RAM that +is used to provide SGX-enabled applications with protected memory, +and is otherwise inaccessible, i.e. shows up as reserved in +/proc/iomem and cannot be read/written outside of an SGX enclave. + +Although current systems implement EPC by stealing memory from RAM, +for all intents and purposes the EPC is independent from normal system +memory, e.g. must be reserved at boot from RAM and cannot be converted +between EPC and normal memory while the system is running. The EPC is +managed by the SGX subsystem and is not accounted by the memory +controller. Note that this is true only for EPC memory itself, i.e. +normal memory allocations related to SGX and EPC memory, e.g. the +backing memory for evicted EPC pages, are accounted, limited and +protected by the memory controller. + +Much like normal system memory, EPC memory can be overcommitted via +virtual memory techniques and pages can be swapped out of the EPC +to their backing store (normal system memory allocated via shmem). +The SGX EPC subsystem is analogous to the memory subsystem, and +it implements limit and protection models for EPC memory. + +SGX EPC Interface Files +----------------------- + +For a generic description of the Miscellaneous controller interface +files, please see Documentation/admin-guide/cgroup-v2.rst + +All SGX EPC memory amounts are in bytes unless explicitly stated +otherwise. If a value which is not PAGE_SIZE aligned is written, +the actual value used by the controller will be rounded down to +the closest PAGE_SIZE multiple. + + misc.capacity + A read-only flat-keyed file shown only in the root cgroup. + The sgx_epc resource will show the total amount of EPC + memory available on the platform. + + misc.current + A read-only flat-keyed file shown in the non-root cgroups. + The sgx_epc resource will show the current active EPC memory + usage of the cgroup and its descendants. EPC pages that are + swapped out to backing RAM are not included in the current count. + + misc.max + A read-write single value file which exists on non-root + cgroups. The sgx_epc resource will show the EPC usage + hard limit. The default is "max". + + If a cgroup's EPC usage reaches this limit, EPC allocations, + e.g. for page fault handling, will be blocked until EPC can + be reclaimed from the cgroup. If EPC cannot be reclaimed in + a timely manner, reclaim will be forced, e.g. by ignoring LRU. + + The EPC pages allocated for KVM guests by the virtual EPC driver + are not reclaimable by the host kernel SGX reclaimers. If a VMM + tries to start a VM within a cgroup whose EPC usage reaches this + limit, the virtual EPC driver will stop allocating more EPC for the + VM, and return SIGBUS to the VMM which would abort the VM launch. + + misc.events + A read-only flat-keyed file which exists on non-root cgroups. + A value change in this file generates a file modified event. + + max + The number of times the cgroup has triggered a reclaim + due to its EPC usage approaching (or exceeding) its max + EPC boundary. + +Migration +--------- + +Once an EPC page is charged to a cgroup (during allocation), it +remains charged to the original cgroup until the page is released +or reclaimed. Migrating a process to a different cgroup doesn't +move the EPC charges that it incurred while in the previous cgroup +to its new cgroup. -- 2.25.1
next prev parent reply other threads:[~2023-09-23 3:08 UTC|newest] Thread overview: 144+ messages / expand[flat|nested] mbox.gz Atom feed top 2023-09-23 3:06 [PATCH v5 00/18] Add Cgroup support for SGX EPC memory Haitao Huang 2023-09-23 3:06 ` Haitao Huang 2023-09-23 3:06 ` [PATCH v5 01/18] cgroup/misc: Add per resource callbacks for CSS events Haitao Huang 2023-09-23 3:06 ` Haitao Huang 2023-09-25 17:09 ` Jarkko Sakkinen 2023-09-25 17:09 ` Jarkko Sakkinen 2023-09-26 3:04 ` Haitao Huang 2023-09-26 13:10 ` Jarkko Sakkinen 2023-09-26 13:10 ` Jarkko Sakkinen 2023-09-26 13:13 ` Jarkko Sakkinen 2023-09-26 13:13 ` Jarkko Sakkinen 2023-09-27 1:56 ` Haitao Huang 2023-10-02 22:47 ` Jarkko Sakkinen 2023-10-02 22:55 ` Jarkko Sakkinen 2023-10-04 15:45 ` Haitao Huang 2023-10-04 17:18 ` Tejun Heo 2023-09-27 9:20 ` Huang, Kai 2023-10-03 14:29 ` Haitao Huang 2023-10-17 18:55 ` Michal Koutný 2023-09-23 3:06 ` [PATCH v5 02/18] cgroup/misc: Add SGX EPC resource type and export APIs for SGX driver Haitao Huang 2023-09-23 3:06 ` Haitao Huang 2023-09-25 18:50 ` Tejun Heo 2023-09-25 18:50 ` Tejun Heo 2023-09-28 3:59 ` Huang, Kai 2023-10-03 7:00 ` Haitao Huang 2023-10-03 19:33 ` Huang, Kai 2023-09-23 3:06 ` [PATCH v5 03/18] x86/sgx: Add sgx_epc_lru_lists to encapsulate LRU lists Haitao Huang 2023-09-23 3:06 ` [PATCH v5 04/18] x86/sgx: Use sgx_epc_lru_lists for existing active page list Haitao Huang 2023-09-23 3:06 ` Haitao Huang 2023-09-23 3:06 ` [PATCH v5 05/18] x86/sgx: Store reclaimable EPC pages in sgx_epc_lru_lists Haitao Huang 2023-09-23 3:06 ` Haitao Huang 2023-09-27 10:14 ` Huang, Kai 2023-09-23 3:06 ` [PATCH v5 06/18] x86/sgx: Introduce EPC page states Haitao Huang 2023-09-23 3:06 ` Haitao Huang 2023-09-25 17:11 ` Jarkko Sakkinen 2023-09-25 17:11 ` Jarkko Sakkinen 2023-09-27 10:28 ` Huang, Kai 2023-10-03 4:49 ` Haitao Huang 2023-10-03 20:03 ` Huang, Kai 2023-10-04 15:24 ` Haitao Huang 2023-10-04 21:05 ` Huang, Kai 2023-09-23 3:06 ` [PATCH v5 07/18] x86/sgx: Introduce RECLAIM_IN_PROGRESS state Haitao Huang 2023-09-23 3:06 ` Haitao Huang 2023-09-25 17:13 ` Jarkko Sakkinen 2023-09-25 17:13 ` Jarkko Sakkinen 2023-09-27 10:42 ` Huang, Kai 2023-09-23 3:06 ` [PATCH v5 08/18] x86/sgx: Use a list to track to-be-reclaimed pages Haitao Huang 2023-09-23 3:06 ` Haitao Huang 2023-09-28 9:28 ` Huang, Kai 2023-10-03 5:09 ` Haitao Huang 2023-09-23 3:06 ` [PATCH v5 09/18] x86/sgx: Store struct sgx_encl when allocating new VA pages Haitao Huang 2023-09-23 3:06 ` Haitao Huang 2023-09-27 11:14 ` Huang, Kai 2023-09-27 15:35 ` Haitao Huang 2023-09-27 21:21 ` Huang, Kai 2023-09-29 15:06 ` Haitao Huang 2023-10-02 11:05 ` Huang, Kai 2023-09-27 11:35 ` Huang, Kai 2023-10-03 6:45 ` Haitao Huang 2023-10-03 20:07 ` Huang, Kai 2023-10-04 15:03 ` Haitao Huang 2023-10-04 21:13 ` Huang, Kai 2023-10-05 4:22 ` Haitao Huang 2023-10-05 6:49 ` Huang, Kai 2023-09-23 3:06 ` [PATCH v5 10/18] x86/sgx: Add EPC page flags to identify owner types Haitao Huang 2023-09-23 3:06 ` Haitao Huang 2023-09-23 3:06 ` [PATCH v5 11/18] x86/sgx: store unreclaimable pages in LRU lists Haitao Huang 2023-09-23 3:06 ` Haitao Huang 2023-09-27 11:57 ` Huang, Kai 2023-10-03 5:42 ` Haitao Huang 2023-09-28 9:41 ` Huang, Kai 2023-10-03 5:15 ` Haitao Huang 2023-10-03 20:12 ` Huang, Kai 2023-09-23 3:06 ` [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC Haitao Huang 2023-09-23 3:06 ` Haitao Huang 2023-10-09 23:45 ` Huang, Kai 2023-10-10 0:23 ` Sean Christopherson 2023-10-10 0:50 ` Huang, Kai 2023-10-10 1:34 ` Huang, Kai 2023-10-10 16:49 ` Haitao Huang 2023-10-11 0:51 ` Huang, Kai 2023-10-12 13:27 ` Haitao Huang 2023-10-16 10:57 ` Huang, Kai 2023-10-16 19:52 ` Haitao Huang 2023-10-16 21:09 ` Huang, Kai 2023-10-17 0:10 ` Haitao Huang 2023-10-17 1:34 ` Huang, Kai 2023-10-17 12:58 ` Haitao Huang 2023-10-17 18:54 ` Michal Koutný 2023-10-17 19:13 ` Michal Koutný 2023-10-18 4:39 ` Haitao Huang 2023-10-18 4:37 ` Haitao Huang 2023-10-18 13:55 ` Dave Hansen 2023-10-18 15:26 ` Haitao Huang 2023-10-18 15:37 ` Dave Hansen 2023-10-18 15:52 ` Michal Koutný 2023-10-18 16:25 ` Haitao Huang 2023-10-16 21:32 ` Sean Christopherson 2023-10-17 0:09 ` Haitao Huang 2023-10-17 15:43 ` Sean Christopherson 2023-10-17 11:49 ` Mikko Ylinen 2023-10-11 1:14 ` Huang, Kai 2023-10-16 11:02 ` Huang, Kai 2023-10-10 1:42 ` Haitao Huang 2023-10-10 2:23 ` Huang, Kai 2023-10-10 13:26 ` Haitao Huang 2023-10-11 0:01 ` Sean Christopherson 2023-10-11 15:02 ` Haitao Huang 2023-10-10 1:04 ` Haitao Huang 2023-10-10 1:18 ` Huang, Kai 2023-10-10 1:38 ` Haitao Huang 2023-10-10 2:12 ` Huang, Kai 2023-10-10 17:05 ` Haitao Huang 2023-10-11 0:31 ` Huang, Kai 2023-10-11 16:04 ` Haitao Huang 2023-09-23 3:06 ` [PATCH v5 13/18] x86/sgx: Expose sgx_reclaim_pages() for use by EPC cgroup Haitao Huang 2023-09-23 3:06 ` Haitao Huang 2023-10-05 12:24 ` Huang, Kai 2023-10-05 19:23 ` Haitao Huang 2023-10-05 20:25 ` Huang, Kai 2023-09-23 3:06 ` [PATCH v5 14/18] x86/sgx: Add helper to grab pages from an arbitrary EPC LRU Haitao Huang 2023-09-23 3:06 ` Haitao Huang 2023-09-23 3:06 ` [PATCH v5 15/18] x86/sgx: Prepare for multiple LRUs Haitao Huang 2023-09-23 3:06 ` Haitao Huang 2023-10-05 12:30 ` Huang, Kai 2023-10-05 19:33 ` Haitao Huang 2023-10-05 20:38 ` Huang, Kai 2023-09-23 3:06 ` [PATCH v5 16/18] x86/sgx: Limit process EPC usage with misc cgroup controller Haitao Huang 2023-09-23 3:06 ` Haitao Huang 2023-09-25 17:15 ` Jarkko Sakkinen 2023-09-25 17:15 ` Jarkko Sakkinen 2023-10-05 21:01 ` Huang, Kai 2023-10-10 0:12 ` Huang, Kai 2023-10-10 0:16 ` Huang, Kai 2023-10-10 0:26 ` Huang, Kai 2023-10-22 18:26 ` Haitao Huang 2023-10-10 9:19 ` Huang, Kai 2023-10-10 9:32 ` Huang, Kai 2023-10-17 18:54 ` Michal Koutný 2023-10-19 16:05 ` Haitao Huang 2023-09-23 3:06 ` Haitao Huang [this message] 2023-09-23 3:06 ` [PATCH v5 17/18] Docs/x86/sgx: Add description for cgroup support Haitao Huang 2023-09-23 3:06 ` [PATCH v5 18/18] selftests/sgx: Add scripts for EPC cgroup testing Haitao Huang 2023-09-23 3:06 ` Haitao Huang
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20230923030657.16148-18-haitao.huang@linux.intel.com \ --to=haitao.huang@linux.intel.com \ --cc=anakrish@microsoft.com \ --cc=bp@alien8.de \ --cc=cgroups@vger.kernel.org \ --cc=dave.hansen@linux.intel.com \ --cc=hpa@zytor.com \ --cc=jarkko@kernel.org \ --cc=kristen@linux.intel.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-sgx@vger.kernel.org \ --cc=mikko.ylinen@linux.intel.com \ --cc=mingo@redhat.com \ --cc=seanjc@google.com \ --cc=sohil.mehta@intel.com \ --cc=tglx@linutronix.de \ --cc=tj@kernel.org \ --cc=x86@kernel.org \ --cc=yangjie@microsoft.com \ --cc=zhanb@microsoft.com \ --cc=zhiquan1.li@intel.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.