From: "Haitao Huang" <haitao.huang@linux.intel.com>
To: "Jarkko Sakkinen" <jarkko@kernel.org>,
dave.hansen@linux.intel.com, tj@kernel.org, mkoutny@suse.com,
linux-kernel@vger.kernel.org, linux-sgx@vger.kernel.org,
x86@kernel.org, cgroups@vger.kernel.org, tglx@linutronix.de,
mingo@redhat.com, bp@alien8.de, hpa@zytor.com,
sohil.mehta@intel.com, tim.c.chen@linux.intel.com
Cc: zhiquan1.li@intel.com, kristen@linux.intel.com,
seanjc@google.com, zhanb@microsoft.com, anakrish@microsoft.com,
mikko.ylinen@linux.intel.com, yangjie@microsoft.com,
chrisyan@microsoft.com
Subject: Re: [PATCH v9 15/15] selftests/sgx: Add scripts for EPC cgroup testing
Date: Wed, 27 Mar 2024 19:57:26 -0500 [thread overview]
Message-ID: <op.2lbjl0oawjvjmi@hhuan26-mobl.amr.corp.intel.com> (raw)
In-Reply-To: <D04OVW6I8MUA.1OAIHFQ8943SM@kernel.org>
On Wed, 27 Mar 2024 11:56:35 -0500, Jarkko Sakkinen <jarkko@kernel.org>
wrote:
> On Wed Mar 27, 2024 at 2:55 PM EET, Jarkko Sakkinen wrote:
>> On Mon, 2024-02-05 at 13:06 -0800, Haitao Huang wrote:
>> > The scripts rely on cgroup-tools package from libcgroup [1].
>> >
>> > To run selftests for epc cgroup:
>> >
>> > sudo ./run_epc_cg_selftests.sh
>> >
>> > To watch misc cgroup 'current' changes during testing, run this in a
>> > separate terminal:
>> >
>> > ./watch_misc_for_tests.sh current
>> >
>> > With different cgroups, the script starts one or multiple concurrent
>> > SGX
>> > selftests, each to run one unclobbered_vdso_oversubscribed test.> Each
>> > of such test tries to load an enclave of EPC size equal to the EPC
>> > capacity available on the platform. The script checks results against
>> > the expectation set for each cgroup and reports success or failure.
>> >
>> > The script creates 3 different cgroups at the beginning with
>> > following
>> > expectations:
>> >
>> > 1) SMALL - intentionally small enough to fail the test loading an
>> > enclave of size equal to the capacity.
>> > 2) LARGE - large enough to run up to 4 concurrent tests but fail some
>> > if
>> > more than 4 concurrent tests are run. The script starts 4 expecting
>> > at
>> > least one test to pass, and then starts 5 expecting at least one test
>> > to fail.
>> > 3) LARGER - limit is the same as the capacity, large enough to run
>> > lots of
>> > concurrent tests. The script starts 8 of them and expects all pass.
>> > Then it reruns the same test with one process randomly killed and
>> > usage checked to be zero after all process exit.
>> >
>> > The script also includes a test with low mem_cg limit and LARGE
>> > sgx_epc
>> > limit to verify that the RAM used for per-cgroup reclamation is
>> > charged
>> > to a proper mem_cg.
>> >
>> > [1] https://github.com/libcgroup/libcgroup/blob/main/README
>> >
>> > Signed-off-by: Haitao Huang <haitao.huang@linux.intel.com>
>> > ---
>> > V7:
>> > - Added memcontrol test.
>> >
>> > V5:
>> > - Added script with automatic results checking, remove the
>> > interactive
>> > script.
>> > - The script can run independent from the series below.
>> > ---
>> > .../selftests/sgx/run_epc_cg_selftests.sh | 246
>> > ++++++++++++++++++
>> > .../selftests/sgx/watch_misc_for_tests.sh | 13 +
>> > 2 files changed, 259 insertions(+)
>> > create mode 100755
>> > tools/testing/selftests/sgx/run_epc_cg_selftests.sh
>> > create mode 100755
>> > tools/testing/selftests/sgx/watch_misc_for_tests.sh
>> >
>> > diff --git a/tools/testing/selftests/sgx/run_epc_cg_selftests.sh
>> > b/tools/testing/selftests/sgx/run_epc_cg_selftests.sh
>> > new file mode 100755
>> > index 000000000000..e027bf39f005
>> > --- /dev/null
>> > +++ b/tools/testing/selftests/sgx/run_epc_cg_selftests.sh
>> > @@ -0,0 +1,246 @@
>> > +#!/bin/bash
>>
>> This is not portable and neither does hold in the wild.
>>
>> It does not even often hold as it is not uncommon to place bash
>> to the path /usr/bin/bash. If I recall correctly, e.g. NixOS has
>> a path that is neither of those two.
>>
>> Should be #!/usr/bin/env bash
>>
>> That is POSIX compatible form.
>>
>> Just got around trying to test this in NUC7 so looking into this in
>> more detail.
>>
>> That said can you make the script work with just "#!/usr/bin/env sh"
>> and make sure that it is busybox ash compatible?
>>
>> I don't see any necessity to make this bash only and it adds to the
>> compilation time of the image. Otherwise lot of this could be tested
>> just with qemu+bzImage+busybox(inside initramfs).
>>
>> Now you are adding fully glibc shenanigans for the sake of syntax
>> sugar.
>>
>> > +# SPDX-License-Identifier: GPL-2.0
>> > +# Copyright(c) 2023 Intel Corporation.
>> > +
>> > +TEST_ROOT_CG=selftest
>> > +cgcreate -g misc:$TEST_ROOT_CG
>>
>> How do you know that cgcreate exists? It is used a lot in the script
>> with no check for the existence. Please fix e.g. with "command -v
>> cgreate".
>>
>> > +if [ $? -ne 0 ]; then
>> > + echo "# Please make sure cgroup-tools is installed, and misc
>> > cgroup is mounted."
>> > + exit 1
>> > +fi
>>
>> And please do not do it this way. Also, please remove the advice for
>> "cgroups-tool". This is not meant to be debian only. Better would be
>> to e.g. point out the URL of the upstream project.
>>
>> And yeah the whole message should be based on "command -v", not like
>> this.
>>
>> > +TEST_CG_SUB1=$TEST_ROOT_CG/test1
>> > +TEST_CG_SUB2=$TEST_ROOT_CG/test2
>> > +# We will only set limit in test1 and run tests in test3
>> > +TEST_CG_SUB3=$TEST_ROOT_CG/test1/test3
>> > +TEST_CG_SUB4=$TEST_ROOT_CG/test4
>> > +
>> > +cgcreate -g misc:$TEST_CG_SUB1
>>
>>
>>
>> > +cgcreate -g misc:$TEST_CG_SUB2
>> > +cgcreate -g misc:$TEST_CG_SUB3
>> > +cgcreate -g misc:$TEST_CG_SUB4
>> > +
>> > +# Default to V2
>> > +CG_MISC_ROOT=/sys/fs/cgroup
>> > +CG_MEM_ROOT=/sys/fs/cgroup
>> > +CG_V1=0
>> > +if [ ! -d "/sys/fs/cgroup/misc" ]; then
>> > + echo "# cgroup V2 is in use."
>> > +else
>> > + echo "# cgroup V1 is in use."
>>
>> Is "#" prefix a standard for kselftest? I don't know this, thus asking.
>>
>> > + CG_MISC_ROOT=/sys/fs/cgroup/misc
>> > + CG_MEM_ROOT=/sys/fs/cgroup/memory
>> > + CG_V1=1
>>
>> Have you checked what is the indentation policy for bash scripts inside
>> kernel tree. I don't know what it is. That's why I'm asking.
>>
>> > +fi
>> > +
>> > +CAPACITY=$(grep "sgx_epc" "$CG_MISC_ROOT/misc.capacity" | awk
>> > '{print $2}')
>> > +# This is below number of VA pages needed for enclave of capacity
>> > size. So
>> > +# should fail oversubscribed cases
>> > +SMALL=$(( CAPACITY / 512 ))
>> > +
>> > +# At least load one enclave of capacity size successfully, maybe up
>> > to 4.
>> > +# But some may fail if we run more than 4 concurrent enclaves of
>> > capacity size.
>> > +LARGE=$(( SMALL * 4 ))
>> > +
>> > +# Load lots of enclaves
>> > +LARGER=$CAPACITY
>> > +echo "# Setting up limits."
>> > +echo "sgx_epc $SMALL" > $CG_MISC_ROOT/$TEST_CG_SUB1/misc.max
>> > +echo "sgx_epc $LARGE" > $CG_MISC_ROOT/$TEST_CG_SUB2/misc.max
>> > +echo "sgx_epc $LARGER" > $CG_MISC_ROOT/$TEST_CG_SUB4/misc.max
>> > +
>> > +timestamp=$(date +%Y%m%d_%H%M%S)
>> > +
>> > +test_cmd="./test_sgx -t unclobbered_vdso_oversubscribed"
>> > +
>> > +wait_check_process_status() {
>> > + local pid=$1
>> > + local check_for_success=$2 # If 1, check for success;
>> > + # If 0, check for failure
>> > + wait "$pid"
>> > + local status=$?
>> > +
>> > + if [[ $check_for_success -eq 1 && $status -eq 0 ]]; then
>> > + echo "# Process $pid succeeded."
>> > + return 0
>> > + elif [[ $check_for_success -eq 0 && $status -ne 0 ]]; then
>> > + echo "# Process $pid returned failure."
>> > + return 0
>> > + fi
>> > + return 1
>> > +}
>> > +
>> > +wai
>> > wait_and_detect_for_any() {
>>
>> what is "any"?
>>
>> Maybe for some key functions could have short documentation what they
>> are and for what test uses them. I cannot possibly remember all of this
>> just by hints such as "this waits for Any" ;-)
>>
>> I don't think there is actual kernel guideline to engineer the script
>> to work with just ash but at least for me that would inevitably
>> increase my motivation to test this patch set more rather than less.
>
> I also wonder is cgroup-tools dependency absolutely required or could
> you just have a function that would interact with sysfs?
I should have checked email before hit the send button for v10 :-).
It'd be more complicated and less readable to do all the stuff without the
cgroup-tools, esp cgexec. I checked dependency, cgroup-tools only depends
on libc so I hope this would not cause too much inconvenience.
I saw bash was also used in cgroup test scripts so at least that's
consistent :-)
I can look into ash if that's required. Let me know.
Certainly can add more docs as you suggested.
Thanks
Haitao
next prev parent reply other threads:[~2024-03-28 0:57 UTC|newest]
Thread overview: 106+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-02-05 21:06 [PATCH v9 00/15] Add Cgroup support for SGX EPC memory Haitao Huang
2024-02-05 21:06 ` [PATCH v9 01/15] cgroup/misc: Add per resource callbacks for CSS events Haitao Huang
2024-02-05 21:06 ` [PATCH v9 02/15] cgroup/misc: Export APIs for SGX driver Haitao Huang
2024-02-05 21:06 ` [PATCH v9 03/15] cgroup/misc: Add SGX EPC resource type Haitao Huang
2024-02-05 21:06 ` [PATCH v9 04/15] x86/sgx: Implement basic EPC misc cgroup functionality Haitao Huang
2024-02-19 12:47 ` Huang, Kai
2024-02-26 18:25 ` Michal Koutný
2024-02-27 21:35 ` Haitao Huang
2024-03-09 21:10 ` Haitao Huang
2024-02-05 21:06 ` [PATCH v9 05/15] x86/sgx: Add sgx_epc_lru_list to encapsulate LRU list Haitao Huang
2024-02-05 21:06 ` [PATCH v9 06/15] x86/sgx: Abstract tracking reclaimable pages in LRU Haitao Huang
2024-02-05 21:06 ` [PATCH v9 07/15] x86/sgx: Expose sgx_reclaim_pages() for cgroup Haitao Huang
2024-02-20 9:26 ` Huang, Kai
2024-02-05 21:06 ` [PATCH v9 08/15] x86/sgx: Implement EPC reclamation flows " Haitao Huang
2024-02-12 19:35 ` Jarkko Sakkinen
2024-02-20 9:52 ` Huang, Kai
2024-02-20 13:18 ` Michal Koutný
2024-02-20 20:09 ` Huang, Kai
2024-02-21 6:23 ` Haitao Huang
2024-02-21 10:48 ` Huang, Kai
2024-02-22 20:12 ` Haitao Huang
2024-02-22 22:24 ` Huang, Kai
2024-03-28 0:24 ` Haitao Huang
2024-02-21 6:44 ` Haitao Huang
2024-02-21 11:00 ` Huang, Kai
2024-02-22 17:20 ` Haitao Huang
2024-02-22 22:31 ` Huang, Kai
2024-02-22 18:09 ` Haitao Huang
2024-02-05 21:06 ` [PATCH v9 09/15] x86/sgx: Charge mem_cgroup for per-cgroup reclamation Haitao Huang
2024-02-12 19:46 ` Jarkko Sakkinen
2024-02-13 3:21 ` Haitao Huang
2024-02-15 23:43 ` Dave Hansen
2024-02-16 6:07 ` Haitao Huang
2024-02-16 15:15 ` Dave Hansen
2024-02-16 21:38 ` Haitao Huang
2024-02-16 21:55 ` Dave Hansen
2024-02-16 23:33 ` Haitao Huang
2024-02-05 21:06 ` [PATCH v9 10/15] x86/sgx: Add EPC reclamation in cgroup try_charge() Haitao Huang
2024-02-12 19:55 ` Jarkko Sakkinen
2024-02-12 23:15 ` Haitao Huang
2024-02-14 1:52 ` Jarkko Sakkinen
2024-02-19 15:12 ` Haitao Huang
2024-02-19 20:20 ` Jarkko Sakkinen
2024-02-19 15:39 ` [RFC PATCH] x86/sgx: Remove 'reclaim' boolean parameters Haitao Huang
2024-02-19 15:56 ` Dave Hansen
2024-02-19 20:42 ` Jarkko Sakkinen
2024-02-19 22:25 ` Haitao Huang
2024-02-19 22:43 ` Jarkko Sakkinen
2024-02-19 20:23 ` Jarkko Sakkinen
2024-02-21 11:06 ` [PATCH v9 10/15] x86/sgx: Add EPC reclamation in cgroup try_charge() Huang, Kai
2024-02-22 17:09 ` Haitao Huang
2024-02-22 21:26 ` Huang, Kai
2024-02-22 22:57 ` Haitao Huang
2024-02-23 10:18 ` Huang, Kai
2024-02-23 17:00 ` Haitao Huang
2024-02-26 1:38 ` Huang, Kai
2024-02-26 4:03 ` Haitao Huang
2024-02-26 11:36 ` Huang, Kai
2024-02-26 14:04 ` Dave Hansen
2024-02-26 21:48 ` Haitao Huang
2024-02-26 21:56 ` Dave Hansen
2024-02-26 22:34 ` Huang, Kai
2024-02-26 22:38 ` Dave Hansen
2024-02-26 22:46 ` Huang, Kai
2024-02-27 20:41 ` Jarkko Sakkinen
2024-02-27 9:26 ` Michal Koutný
2024-02-26 21:18 ` Haitao Huang
2024-02-26 22:24 ` Huang, Kai
2024-02-26 22:31 ` Dave Hansen
2024-02-26 22:38 ` Huang, Kai
2024-02-05 21:06 ` [PATCH v9 11/15] x86/sgx: Abstract check for global reclaimable pages Haitao Huang
2024-02-12 19:56 ` Jarkko Sakkinen
2024-02-21 11:34 ` Huang, Kai
2024-02-05 21:06 ` [PATCH v9 12/15] x86/sgx: Expose sgx_epc_cgroup_reclaim_pages() for global reclaimer Haitao Huang
2024-02-12 19:58 ` Jarkko Sakkinen
2024-02-21 11:10 ` Huang, Kai
2024-02-22 16:35 ` Haitao Huang
2024-02-05 21:06 ` [PATCH v9 13/15] x86/sgx: Turn on per-cgroup EPC reclamation Haitao Huang
2024-02-21 11:23 ` Huang, Kai
2024-02-22 16:36 ` Haitao Huang
2024-02-22 22:44 ` Huang, Kai
2024-02-23 18:46 ` Haitao Huang
2024-02-05 21:06 ` [PATCH v9 14/15] Docs/x86/sgx: Add description for cgroup support Haitao Huang
2024-02-05 21:06 ` [PATCH v9 15/15] selftests/sgx: Add scripts for EPC cgroup testing Haitao Huang
2024-03-27 12:55 ` Jarkko Sakkinen
2024-03-27 16:56 ` Jarkko Sakkinen
2024-03-28 0:57 ` Haitao Huang [this message]
2024-03-28 3:05 ` Haitao Huang
2024-03-30 11:23 ` Jarkko Sakkinen
2024-03-30 11:26 ` Jarkko Sakkinen
2024-04-02 11:23 ` Michal Koutný
2024-04-02 11:58 ` Jarkko Sakkinen
2024-04-02 16:20 ` Haitao Huang
2024-04-02 17:40 ` Michal Koutný
2024-04-02 18:20 ` Haitao Huang
2024-04-03 16:46 ` Jarkko Sakkinen
2024-04-03 15:33 ` Jarkko Sakkinen
2024-04-02 15:42 ` Dave Hansen
2024-04-03 15:16 ` Jarkko Sakkinen
2024-03-28 3:54 ` Haitao Huang
2024-03-30 11:15 ` Jarkko Sakkinen
2024-03-30 15:32 ` Haitao Huang
2024-03-31 16:19 ` Jarkko Sakkinen
2024-03-31 17:35 ` Haitao Huang
2024-04-01 14:10 ` Jarkko Sakkinen
2024-02-08 8:43 ` [PATCH v9 00/15] Add Cgroup support for SGX EPC memory Mikko Ylinen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=op.2lbjl0oawjvjmi@hhuan26-mobl.amr.corp.intel.com \
--to=haitao.huang@linux.intel.com \
--cc=anakrish@microsoft.com \
--cc=bp@alien8.de \
--cc=cgroups@vger.kernel.org \
--cc=chrisyan@microsoft.com \
--cc=dave.hansen@linux.intel.com \
--cc=hpa@zytor.com \
--cc=jarkko@kernel.org \
--cc=kristen@linux.intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-sgx@vger.kernel.org \
--cc=mikko.ylinen@linux.intel.com \
--cc=mingo@redhat.com \
--cc=mkoutny@suse.com \
--cc=seanjc@google.com \
--cc=sohil.mehta@intel.com \
--cc=tglx@linutronix.de \
--cc=tim.c.chen@linux.intel.com \
--cc=tj@kernel.org \
--cc=x86@kernel.org \
--cc=yangjie@microsoft.com \
--cc=zhanb@microsoft.com \
--cc=zhiquan1.li@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).