RCU Archive on lore.kernel.org
 help / color / Atom feed
From: paulmck@kernel.org
To: rcu@vger.kernel.org
Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com,
	mingo@kernel.org, jiangshanlai@gmail.com,
	akpm@linux-foundation.org, mathieu.desnoyers@efficios.com,
	josh@joshtriplett.org, tglx@linutronix.de, peterz@infradead.org,
	rostedt@goodmis.org, dhowells@redhat.com, edumazet@google.com,
	fweisbec@gmail.com, oleg@redhat.com, joel@joelfernandes.org,
	"Paul E. McKenney" <paulmck@kernel.org>
Subject: [PATCH tip/core/rcu 18/18] torture: Compress KASAN vmlinux files
Date: Wed,  6 Jan 2021 09:31:19 -0800
Message-ID: <20210106173119.23159-18-paulmck@kernel.org> (raw)
In-Reply-To: <20210106173056.GA23035@paulmck-ThinkPad-P72>

From: "Paul E. McKenney" <paulmck@kernel.org>

The sizes of vmlinux files built with KASAN enabled can approach a full
gigabyte, which can result in disk overflow sooner rather than later.
Fortunately, the xz command compresses them by almost an order of
magnitude.  This commit therefore uses xz to compress vmlinux file built
by torture.sh with KASAN enabled.

However, xz is not the fastest thing in the world.  In fact, it is way
slower than rotating-rust mass storage.  This commit therefore also adds a
--compress-kasan-vmlinux argument to specify the degree of xz concurrency,
which defaults to using all available CPUs if there are that many files in
need of compression.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
 tools/testing/selftests/rcutorture/bin/torture.sh | 48 ++++++++++++++++++++++-
 1 file changed, 46 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/rcutorture/bin/torture.sh b/tools/testing/selftests/rcutorture/bin/torture.sh
index 0867f30..ad7525b 100755
--- a/tools/testing/selftests/rcutorture/bin/torture.sh
+++ b/tools/testing/selftests/rcutorture/bin/torture.sh
@@ -36,7 +36,8 @@ configs_locktorture=
 configs_scftorture=
 kcsan_kmake_args=
 
-# Default duration and apportionment.
+# Default compression, duration, and apportionment.
+compress_kasan_vmlinux="`identify_qemu_vcpus`"
 duration_base=10
 duration_rcutorture_frac=7
 duration_locktorture_frac=1
@@ -65,6 +66,7 @@ function doyesno () {
 
 usage () {
 	echo "Usage: $scriptname optional arguments:"
+	echo "       --compress-kasan-vmlinux concurrency"
 	echo "       --configs-rcutorture \"config-file list w/ repeat factor (3*TINY01)\""
 	echo "       --configs-locktorture \"config-file list w/ repeat factor (10*LOCK01)\""
 	echo "       --configs-scftorture \"config-file list w/ repeat factor (2*CFLIST)\""
@@ -87,6 +89,11 @@ usage () {
 while test $# -gt 0
 do
 	case "$1" in
+	--compress-kasan-vmlinux)
+		checkarg --compress-kasan-vmlinux "(concurrency level)" $# "$2" '^[0-9][0-9]*$' '^error'
+		compress_kasan_vmlinux=$2
+		shift
+		;;
 	--config-rcutorture|--configs-rcutorture)
 		checkarg --configs-rcutorture "(list of config files)" "$#" "$2" '^[^/]\+$' '^--'
 		configs_rcutorture="$configs_rcutorture $2"
@@ -391,8 +398,45 @@ fi
 echo Started at $startdate, ended at `date`, duration `get_starttime_duration $starttime`. | tee -a $T/log
 echo Summary: Successes: $nsuccesses Failures: $nfailures. | tee -a $T/log
 tdir="`cat $T/successes $T/failures | head -1 | awk '{ print $NF }' | sed -e 's,/[^/]\+/*$,,'`"
+if test -n "$tdir" && test $compress_kasan_vmlinux -gt 0
+then
+	# KASAN vmlinux files can approach 1GB in size, so compress them.
+	echo Looking for KASAN files to compress: `date` > "$tdir/log-xz" 2>&1
+	find "$tdir" -type d -name '*-kasan' -print > $T/xz-todo
+	ncompresses=0
+	batchno=1
+	if test -s $T/xz-todo
+	then
+		echo Size before compressing: `du -sh $tdir | awk '{ print $1 }'` `date` 2>&1 | tee -a "$tdir/log-xz" | tee -a $T/log
+		for i in `cat $T/xz-todo`
+		do
+			echo Compressing vmlinux files in ${i}: `date` >> "$tdir/log-xz" 2>&1
+			for j in $i/*/vmlinux
+			do
+				xz "$j" >> "$tdir/log-xz" 2>&1 &
+				ncompresses=$((ncompresses+1))
+				if test $ncompresses -ge $compress_kasan_vmlinux
+				then
+					echo Waiting for batch $batchno of $ncompresses compressions `date` | tee -a "$tdir/log-xz" | tee -a $T/log
+					wait
+					ncompresses=0
+					batchno=$((batchno+1))
+				fi
+			done
+		done
+		if test $ncompresses -gt 0
+		then
+			echo Waiting for final batch $batchno of $ncompresses compressions `date` | tee -a "$tdir/log-xz" | tee -a $T/log
+		fi
+		wait
+		echo Size after compressing: `du -sh $tdir | awk '{ print $1 }'` `date` 2>&1 | tee -a "$tdir/log-xz" | tee -a $T/log
+		echo Total duration `get_starttime_duration $starttime`. | tee -a $T/log
+	else
+		echo No compression needed: `date` >> "$tdir/log-xz" 2>&1
+	fi
+fi
 if test -n "$tdir"
 then
-	cp $T/log $tdir
+	cp $T/log "$tdir"
 fi
 exit $ret
-- 
2.9.5


      parent reply index

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-01-06 17:30 [PATCH tip/core/rcu 0/18] Add a torture-all acceptance-test script for v5.12 Paul E. McKenney
2021-01-06 17:31 ` [PATCH tip/core/rcu 01/18] torture: Do Kconfig analysis only once per scenario paulmck
2021-01-06 17:31 ` [PATCH tip/core/rcu 02/18] torture: Add torture.sh torture-everything script paulmck
2021-01-06 17:31 ` [PATCH tip/core/rcu 03/18] torture: Make torture.sh use common time-duration bash functions paulmck
2021-01-06 17:31 ` [PATCH tip/core/rcu 04/18] torture: Remove use of "eval" in torture.sh paulmck
2021-01-06 17:31 ` [PATCH tip/core/rcu 05/18] torture: Add "make allmodconfig" to torture.sh paulmck
2021-01-06 17:31 ` [PATCH tip/core/rcu 06/18] torture: Auto-size SCF and scaling runs based on number of CPUs paulmck
2021-01-06 17:31 ` [PATCH tip/core/rcu 07/18] torture: Enable torture.sh argument checking paulmck
2021-01-06 17:31 ` [PATCH tip/core/rcu 08/18] torture: Make torture.sh rcuscale and refscale deal with allmodconfig paulmck
2021-01-06 17:31 ` [PATCH tip/core/rcu 09/18] torture: Make torture.sh refscale runs use verbose_batched module parameter paulmck
2021-01-06 17:31 ` [PATCH tip/core/rcu 10/18] torture: Create doyesno helper function for torture.sh paulmck
2021-01-06 17:31 ` [PATCH tip/core/rcu 11/18] torture: Make torture.sh allmodconfig retain and label output paulmck
2021-01-06 17:31 ` [PATCH tip/core/rcu 12/18] torture: Make torture.sh throttle VERBOSE_TOROUT_*() for refscale paulmck
2021-01-06 17:31 ` [PATCH tip/core/rcu 13/18] torture: Make torture.sh refuse to do zero-length runs paulmck
2021-01-06 17:31 ` [PATCH tip/core/rcu 14/18] torture: Drop log.long generation from torture.sh paulmck
2021-01-06 17:31 ` [PATCH tip/core/rcu 15/18] torture: Allow scenarios to be specified to torture.sh paulmck
2021-01-06 17:31 ` [PATCH tip/core/rcu 16/18] torture: Add command and results directory to torture.sh log paulmck
2021-01-06 17:31 ` [PATCH tip/core/rcu 17/18] torture: Add --kcsan-kmake-arg to torture.sh for KCSAN paulmck
2021-01-06 17:31 ` paulmck [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210106173119.23159-18-paulmck@kernel.org \
    --to=paulmck@kernel.org \
    --cc=akpm@linux-foundation.org \
    --cc=dhowells@redhat.com \
    --cc=edumazet@google.com \
    --cc=fweisbec@gmail.com \
    --cc=jiangshanlai@gmail.com \
    --cc=joel@joelfernandes.org \
    --cc=josh@joshtriplett.org \
    --cc=kernel-team@fb.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mathieu.desnoyers@efficios.com \
    --cc=mingo@kernel.org \
    --cc=oleg@redhat.com \
    --cc=peterz@infradead.org \
    --cc=rcu@vger.kernel.org \
    --cc=rostedt@goodmis.org \
    --cc=tglx@linutronix.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

RCU Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/rcu/0 rcu/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 rcu rcu/ https://lore.kernel.org/rcu \
		rcu@vger.kernel.org
	public-inbox-index rcu

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.rcu


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git