From: John Kacur <jkacur@redhat.com>
To: Crystal Wood <crwood@redhat.com>
Cc: Clark Williams <williams@redhat.com>, linux-rt-users@vger.kernel.org
Subject: Re: [PATCH 3/5] rteval: kcompile: Skip mrproper, and re-extract if clean fails
Date: Tue, 2 Apr 2024 15:07:03 -0400 (EDT) [thread overview]
Message-ID: <0f21e919-9bab-5abf-8844-d4ff8c84b82d@redhat.com> (raw)
In-Reply-To: <20240304211655.20174-4-crwood@redhat.com>
On Mon, 4 Mar 2024, Crystal Wood wrote:
> We only ever do out-of-tree builds, so the kernel directory should not be
> getting damaged except by incomplete extraction (which mrproper would
> probably not fix) or external meddling. So, skip the mrproper and
> instead re-extract if a "make clean" fails.
>
> Also, add -j to cleancmd to further speed things up. Startup speed may
> not seem all that important given how long rteval is typically run for,
> but this helps make quick tests (e.g. while debugging things, or when
> hunting a latency that shows up very quickly) less painful.
>
> On my 12-cpu laptop, this patch saves about 15 seconds of startup time.
>
> Signed-off-by: Crystal Wood <crwood@redhat.com>
> ---
> rteval/modules/loads/kcompile.py | 40 ++++++++++++++------------------
> 1 file changed, 18 insertions(+), 22 deletions(-)
>
> diff --git a/rteval/modules/loads/kcompile.py b/rteval/modules/loads/kcompile.py
> index de1539986723..db1941074157 100644
> --- a/rteval/modules/loads/kcompile.py
> +++ b/rteval/modules/loads/kcompile.py
> @@ -54,7 +54,7 @@ class KBuildJob:
> self.jobs = self.calc_jobs_per_cpu() * cpus_available
>
> self.runcmd = f"make O={self.objdir} -C {self.kdir} -j{self.jobs}"
> - self.cleancmd = f"make O={self.objdir} -C {self.kdir} clean allmodconfig"
> + self.cleancmd = f"make O={self.objdir} -C {self.kdir} clean allmodconfig -j${self.jobs}"
> self.cleancmd += f"&& pushd {self.objdir} && {self.kdir}/scripts/config -d CONFIG_MODULE_SIG_SHA1 -e CONFIG_MODULE_SIG_SHA512 && popd && make O={self.objdir} -C {self.kdir} olddefconfig"
> if self.binder:
> self.runcmd = self.binder + " " + self.runcmd
> @@ -90,8 +90,8 @@ class KBuildJob:
> def clean(self, sin=None, sout=None, serr=None):
> """ Runs command to clean any previous builds and configure kernel """
> self.log(Log.DEBUG, f"cleaning objdir {self.objdir}")
> - subprocess.call(self.cleancmd, shell=True,
> - stdin=sin, stdout=sout, stderr=serr)
> + return subprocess.call(self.cleancmd, shell=True,
> + stdin=sin, stdout=sout, stderr=serr)
>
> def run(self, sin=None, sout=None, serr=None):
> """ Use Popen to launch a kcompile job """
> @@ -234,6 +234,10 @@ class Kcompile(CommandLineLoad):
> self.logger, self.cpus[n] if self.cpulist else None)
> self.args.append(str(self.buildjobs[n])+";")
>
> + def _repair_tarball(self):
> + self._log(Log.DEBUG, "Invalid state in kernel build tree, reloading")
> + self._remove_build_dirs()
> + self._extract_tarball()
>
> def _WorkloadBuild(self):
> if self._donotrun:
> @@ -246,30 +250,22 @@ class Kcompile(CommandLineLoad):
> else:
> out = err = null
>
> - # clean up any damage from previous runs
> - try:
> - cmd = ["make", "-C", self.mydir, "mrproper"]
> - ret = subprocess.call(cmd, stdin=null, stdout=out, stderr=err)
> + # clean up object dirs and make sure each has a config file
> + for n in self.nodes:
> + ret = self.buildjobs[n].clean(sin=null, sout=out, serr=err)
> if ret:
> - # if the above make failed, remove and reinstall the source tree
> - self._log(Log.DEBUG, "Invalid state in kernel build tree, reloading")
> - self._remove_build_dirs()
> - self._extract_tarball()
> - ret = subprocess.call(cmd, stdin=null, stdout=out, stderr=err)
> - if ret:
> - # give up
> - raise rtevalRuntimeError(self, f"kcompile setup failed: {ret}")
> - except KeyboardInterrupt as m:
> - self._log(Log.DEBUG, "keyboard interrupt, aborting")
> - return
> - self._log(Log.DEBUG, "ready to run")
> + self._repair_tarball()
> + ret = self.buildjobs[n].clean(sin=null, sout=out, serr=err)
> + if ret:
> + raise rtevalRuntimeError(self, f"kcompile setup failed: {ret}")
> +
> if self._logging:
> os.close(out)
> os.close(err)
> - # clean up object dirs and make sure each has a config file
> - for n in self.nodes:
> - self.buildjobs[n].clean(sin=null, sout=null, serr=null)
> +
> os.close(null)
> +
> + self._log(Log.DEBUG, "ready to run")
> self._setReady()
>
> def _WorkloadPrepare(self):
> --
> 2.43.0
You would probably achieve close to the same or better by just using '-j'
with make mrproper
[jkacur@fionn tmp]$ time tar xJf ../linux-6.6.1.tar.xz
real 0m7.224s
user 0m7.233s
sys 0m2.204s
[jkacur@fionn tmp]$ cd linux-6.6.1/
[jkacur@fionn linux-6.6.1]$ time make mrproper
real 0m14.451s
user 0m7.998s
sys 0m6.307s
[jkacur@fionn linux-6.6.1]$ time make mrproper -j
real 0m1.336s
user 0m3.473s
sys 0m2.147s
John
next prev parent reply other threads:[~2024-04-02 19:07 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-04 21:16 [PATCH 0/5] rteval: Fixes and speedups Crystal Wood
2024-03-04 21:16 ` [PATCH 1/5] rteval: default_config_search: Return None on failure Crystal Wood
2024-03-28 19:10 ` John Kacur
2024-03-04 21:16 ` [PATCH 2/5] rteval: kcompile: Fix path lookups in _remove_build_dirs Crystal Wood
2024-03-28 21:07 ` John Kacur
2024-03-04 21:16 ` [PATCH 3/5] rteval: kcompile: Skip mrproper, and re-extract if clean fails Crystal Wood
2024-04-02 19:07 ` John Kacur [this message]
2024-04-03 14:57 ` Crystal Wood
2024-04-05 22:11 ` John Kacur
2024-03-04 21:16 ` [PATCH 4/5] rteval: Break out of main loop faster on interrupt/stoptime Crystal Wood
2024-04-02 20:39 ` John Kacur
2024-03-04 21:16 ` [PATCH 5/5] rteval: Remove 30 second "settling" period Crystal Wood
2024-04-02 20:52 ` John Kacur
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=0f21e919-9bab-5abf-8844-d4ff8c84b82d@redhat.com \
--to=jkacur@redhat.com \
--cc=crwood@redhat.com \
--cc=linux-rt-users@vger.kernel.org \
--cc=williams@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).