[PATCH] torture: Correctly fetch CPUs for kvm-build.sh with all native language

[PATCH] torture: Correctly fetch CPUs for kvm-build.sh with all native language

[Frederic Weisbecker]

Grepping for "CPU" on lscpu output isn't always successful, depending
on the local language setting. As a result, the build can be aborted
early with:

	"make: the '-j' option requires a positive integer argument"

Prefer a more generic solution.

Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
 tools/testing/selftests/rcutorture/bin/kvm-build.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/testing/selftests/rcutorture/bin/kvm-build.sh b/tools/testing/selftests/rcutorture/bin/kvm-build.sh
index 55f4fc102624..5ad973dca820 100755
--- a/tools/testing/selftests/rcutorture/bin/kvm-build.sh
+++ b/tools/testing/selftests/rcutorture/bin/kvm-build.sh
@@ -42,7 +42,7 @@ then
 fi
 
 # Tell "make" to use double the number of real CPUs on the build system.
-ncpus="`lscpu | grep '^CPU(' | awk '{ print $2 }'`"
+ncpus="`getconf _NPROCESSORS_ONLN`"
 make -j$((2 * ncpus)) $TORTURE_KMAKE_ARG > $resdir/Make.out 2>&1
 retval=$?
 if test $retval -ne 0 || grep "rcu[^/]*": < $resdir/Make.out | egrep -q "Stop|Error|error:|warning:" || egrep -q "Stop|Error|error:" < $resdir/Make.out
-- 
2.25.1

Re: [PATCH] torture: Correctly fetch CPUs for kvm-build.sh with all native language

[Paul E. McKenney]

On Thu, Apr 01, 2021 at 03:26:02PM +0200, Frederic Weisbecker wrote:
> Grepping for "CPU" on lscpu output isn't always successful, depending
> on the local language setting. As a result, the build can be aborted
> early with:
> 
> 	"make: the '-j' option requires a positive integer argument"
> 
> Prefer a more generic solution.
> 
> Signed-off-by: Frederic Weisbecker <frederic@kernel.org>

Good catch, applied, thank you!

There is a similar construct in kvm-remote.sh, so I added a similar
fix to your patch.

But what about this in functions.sh?

nt="`lscpu | grep '^NUMA node0' | sed -e 's/^[^,]*,\([0-9]*\),.*$/\1/'`"

I am guessing that "node0" is human-language-independent, but is "NUMA"?

							Thanx, Paul

------------------------------------------------------------------------

commit cf519a9b736cf7dcd47920065e4de8c1855b5532
Author: Frederic Weisbecker <frederic@kernel.org>
Date:   Thu Apr 1 15:26:02 2021 +0200

    torture: Correctly fetch number of CPUs for non-English languages
    
    Grepping for "CPU" on lscpu output isn't always successful, depending
    on the local language setting.  As a result, the build can be aborted
    early with:
    
            "make: the '-j' option requires a positive integer argument"
    
    This commit therefore uses the human-language-independent approach
    available via the getconf command, both in kvm-build.sh and in
    kvm-remote.sh.
    
    Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
    Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

diff --git a/tools/testing/selftests/rcutorture/bin/kvm-build.sh b/tools/testing/selftests/rcutorture/bin/kvm-build.sh
index 55f4fc1..5ad973d 100755
--- a/tools/testing/selftests/rcutorture/bin/kvm-build.sh
+++ b/tools/testing/selftests/rcutorture/bin/kvm-build.sh
@@ -42,7 +42,7 @@ then
 fi
 
 # Tell "make" to use double the number of real CPUs on the build system.
-ncpus="`lscpu | grep '^CPU(' | awk '{ print $2 }'`"
+ncpus="`getconf _NPROCESSORS_ONLN`"
 make -j$((2 * ncpus)) $TORTURE_KMAKE_ARG > $resdir/Make.out 2>&1
 retval=$?
 if test $retval -ne 0 || grep "rcu[^/]*": < $resdir/Make.out | egrep -q "Stop|Error|error:|warning:" || egrep -q "Stop|Error|error:" < $resdir/Make.out
diff --git a/tools/testing/selftests/rcutorture/bin/kvm-remote.sh b/tools/testing/selftests/rcutorture/bin/kvm-remote.sh
index 0adaf26..92dd1a3 100755
--- a/tools/testing/selftests/rcutorture/bin/kvm-remote.sh
+++ b/tools/testing/selftests/rcutorture/bin/kvm-remote.sh
@@ -136,7 +136,7 @@ chmod +x $T/bin/kvm-remote-*.sh
 # Check first to avoid the need for cleanup for system-name typos
 for i in $systems
 do
-	ncpus="`ssh $i lscpu | grep '^CPU(' | awk '{ print $2 }'`"
+	ncpus="`ssh $i getconf _NPROCESSORS_ONLN 2> /dev/null`"
 	echo $i: $ncpus CPUs " " `date` | tee -a "$oldrun/remote-log"
 	ret=$?
 	if test "$ret" -ne 0

Re: [PATCH] torture: Correctly fetch CPUs for kvm-build.sh with all native language

[Frederic Weisbecker]

On Thu, Apr 01, 2021 at 11:51:16AM -0700, Paul E. McKenney wrote:
> On Thu, Apr 01, 2021 at 03:26:02PM +0200, Frederic Weisbecker wrote:
> > Grepping for "CPU" on lscpu output isn't always successful, depending
> > on the local language setting. As a result, the build can be aborted
> > early with:
> > 
> > 	"make: the '-j' option requires a positive integer argument"
> > 
> > Prefer a more generic solution.
> > 
> > Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> 
> Good catch, applied, thank you!
> 
> There is a similar construct in kvm-remote.sh, so I added a similar
> fix to your patch.
> 
> But what about this in functions.sh?
> 
> nt="`lscpu | grep '^NUMA node0' | sed -e 's/^[^,]*,\([0-9]*\),.*$/\1/'`"
> 
> I am guessing that "node0" is human-language-independent, but is "NUMA"?

I thought they wouldn't bother translating that, but they did...

    NUMA node0 CPU(s):               0-7

becomes:

    Nœud NUMA 0 de processeur(s) : 0-7

Not sure about the best way to fix it.

Thanks.

Re: [PATCH] torture: Correctly fetch CPUs for kvm-build.sh with all native language

[Paul E. McKenney]

On Thu, Apr 01, 2021 at 10:31:12PM +0200, Frederic Weisbecker wrote:
> On Thu, Apr 01, 2021 at 11:51:16AM -0700, Paul E. McKenney wrote:
> > On Thu, Apr 01, 2021 at 03:26:02PM +0200, Frederic Weisbecker wrote:
> > > Grepping for "CPU" on lscpu output isn't always successful, depending
> > > on the local language setting. As a result, the build can be aborted
> > > early with:
> > > 
> > > 	"make: the '-j' option requires a positive integer argument"
> > > 
> > > Prefer a more generic solution.
> > > 
> > > Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> > 
> > Good catch, applied, thank you!
> > 
> > There is a similar construct in kvm-remote.sh, so I added a similar
> > fix to your patch.
> > 
> > But what about this in functions.sh?
> > 
> > nt="`lscpu | grep '^NUMA node0' | sed -e 's/^[^,]*,\([0-9]*\),.*$/\1/'`"
> > 
> > I am guessing that "node0" is human-language-independent, but is "NUMA"?
> 
> I thought they wouldn't bother translating that, but they did...
> 
>     NUMA node0 CPU(s):               0-7
> 
> becomes:
> 
>     Nœud NUMA 0 de processeur(s) : 0-7
> 
> Not sure about the best way to fix it.

The rude and crude fix is for the scripts to force the local language
to English.  ;-)

							Thanx, Paul

Re: [PATCH] torture: Correctly fetch CPUs for kvm-build.sh with all native language

[Frederic Weisbecker]

On Thu, Apr 01, 2021 at 01:40:22PM -0700, Paul E. McKenney wrote:
> On Thu, Apr 01, 2021 at 10:31:12PM +0200, Frederic Weisbecker wrote:
> > On Thu, Apr 01, 2021 at 11:51:16AM -0700, Paul E. McKenney wrote:
> > > On Thu, Apr 01, 2021 at 03:26:02PM +0200, Frederic Weisbecker wrote:
> > > > Grepping for "CPU" on lscpu output isn't always successful, depending
> > > > on the local language setting. As a result, the build can be aborted
> > > > early with:
> > > > 
> > > > 	"make: the '-j' option requires a positive integer argument"
> > > > 
> > > > Prefer a more generic solution.
> > > > 
> > > > Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> > > 
> > > Good catch, applied, thank you!
> > > 
> > > There is a similar construct in kvm-remote.sh, so I added a similar
> > > fix to your patch.
> > > 
> > > But what about this in functions.sh?
> > > 
> > > nt="`lscpu | grep '^NUMA node0' | sed -e 's/^[^,]*,\([0-9]*\),.*$/\1/'`"
> > > 
> > > I am guessing that "node0" is human-language-independent, but is "NUMA"?
> > 
> > I thought they wouldn't bother translating that, but they did...
> > 
> >     NUMA node0 CPU(s):               0-7
> > 
> > becomes:
> > 
> >     Nœud NUMA 0 de processeur(s) : 0-7
> > 
> > Not sure about the best way to fix it.
> 
> The rude and crude fix is for the scripts to force the local language
> to English.  ;-)

I don't have a better answer :o)

Re: [PATCH] torture: Correctly fetch CPUs for kvm-build.sh with all native language

[Paul E. McKenney]

On Thu, Apr 01, 2021 at 10:41:13PM +0200, Frederic Weisbecker wrote:
> On Thu, Apr 01, 2021 at 01:40:22PM -0700, Paul E. McKenney wrote:
> > On Thu, Apr 01, 2021 at 10:31:12PM +0200, Frederic Weisbecker wrote:
> > > On Thu, Apr 01, 2021 at 11:51:16AM -0700, Paul E. McKenney wrote:
> > > > On Thu, Apr 01, 2021 at 03:26:02PM +0200, Frederic Weisbecker wrote:
> > > > > Grepping for "CPU" on lscpu output isn't always successful, depending
> > > > > on the local language setting. As a result, the build can be aborted
> > > > > early with:
> > > > > 
> > > > > 	"make: the '-j' option requires a positive integer argument"
> > > > > 
> > > > > Prefer a more generic solution.
> > > > > 
> > > > > Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> > > > 
> > > > Good catch, applied, thank you!
> > > > 
> > > > There is a similar construct in kvm-remote.sh, so I added a similar
> > > > fix to your patch.
> > > > 
> > > > But what about this in functions.sh?
> > > > 
> > > > nt="`lscpu | grep '^NUMA node0' | sed -e 's/^[^,]*,\([0-9]*\),.*$/\1/'`"
> > > > 
> > > > I am guessing that "node0" is human-language-independent, but is "NUMA"?
> > > 
> > > I thought they wouldn't bother translating that, but they did...
> > > 
> > >     NUMA node0 CPU(s):               0-7
> > > 
> > > becomes:
> > > 
> > >     Nœud NUMA 0 de processeur(s) : 0-7
> > > 
> > > Not sure about the best way to fix it.
> > 
> > The rude and crude fix is for the scripts to force the local language
> > to English.  ;-)
> 
> I don't have a better answer :o)

If you set the environment variable LANG to en_US.UTF-8, does that
make things work for you?  Huh.  Setting it to fr_FR.UTF-8 does not
shift lscpu out of English for me, so I am guessing "no".

Help?

							Thanx, Paul

Re: [PATCH] torture: Correctly fetch CPUs for kvm-build.sh with all native language

[Frederic Weisbecker]

On Thu, Apr 01, 2021 at 02:02:53PM -0700, Paul E. McKenney wrote:
> On Thu, Apr 01, 2021 at 10:41:13PM +0200, Frederic Weisbecker wrote:
> > On Thu, Apr 01, 2021 at 01:40:22PM -0700, Paul E. McKenney wrote:
> > > On Thu, Apr 01, 2021 at 10:31:12PM +0200, Frederic Weisbecker wrote:
> > > > On Thu, Apr 01, 2021 at 11:51:16AM -0700, Paul E. McKenney wrote:
> > > > > On Thu, Apr 01, 2021 at 03:26:02PM +0200, Frederic Weisbecker wrote:
> > > > > > Grepping for "CPU" on lscpu output isn't always successful, depending
> > > > > > on the local language setting. As a result, the build can be aborted
> > > > > > early with:
> > > > > > 
> > > > > > 	"make: the '-j' option requires a positive integer argument"
> > > > > > 
> > > > > > Prefer a more generic solution.
> > > > > > 
> > > > > > Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> > > > > 
> > > > > Good catch, applied, thank you!
> > > > > 
> > > > > There is a similar construct in kvm-remote.sh, so I added a similar
> > > > > fix to your patch.
> > > > > 
> > > > > But what about this in functions.sh?
> > > > > 
> > > > > nt="`lscpu | grep '^NUMA node0' | sed -e 's/^[^,]*,\([0-9]*\),.*$/\1/'`"
> > > > > 
> > > > > I am guessing that "node0" is human-language-independent, but is "NUMA"?
> > > > 
> > > > I thought they wouldn't bother translating that, but they did...
> > > > 
> > > >     NUMA node0 CPU(s):               0-7
> > > > 
> > > > becomes:
> > > > 
> > > >     Nœud NUMA 0 de processeur(s) : 0-7
> > > > 
> > > > Not sure about the best way to fix it.
> > > 
> > > The rude and crude fix is for the scripts to force the local language
> > > to English.  ;-)
> > 
> > I don't have a better answer :o)
> 
> If you set the environment variable LANG to en_US.UTF-8, does that
> make things work for you?  Huh.  Setting it to fr_FR.UTF-8 does not
> shift lscpu out of English for me, so I am guessing "no".

Maybe that language isn't installed in your system. I would expect
en_US.UTF-8 to be supported pretty much everywhere though. At least it
works for me with: "LANG=en_US.UTF-8 lscpu".

Thanks.

> 
> Help?
> 
> 							Thanx, Paul

Re: [PATCH] torture: Correctly fetch CPUs for kvm-build.sh with all native language

[Paul E. McKenney]

On Thu, Apr 01, 2021 at 11:08:02PM +0200, Frederic Weisbecker wrote:
> On Thu, Apr 01, 2021 at 02:02:53PM -0700, Paul E. McKenney wrote:
> > On Thu, Apr 01, 2021 at 10:41:13PM +0200, Frederic Weisbecker wrote:
> > > On Thu, Apr 01, 2021 at 01:40:22PM -0700, Paul E. McKenney wrote:
> > > > On Thu, Apr 01, 2021 at 10:31:12PM +0200, Frederic Weisbecker wrote:
> > > > > On Thu, Apr 01, 2021 at 11:51:16AM -0700, Paul E. McKenney wrote:
> > > > > > On Thu, Apr 01, 2021 at 03:26:02PM +0200, Frederic Weisbecker wrote:
> > > > > > > Grepping for "CPU" on lscpu output isn't always successful, depending
> > > > > > > on the local language setting. As a result, the build can be aborted
> > > > > > > early with:
> > > > > > > 
> > > > > > > 	"make: the '-j' option requires a positive integer argument"
> > > > > > > 
> > > > > > > Prefer a more generic solution.
> > > > > > > 
> > > > > > > Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> > > > > > 
> > > > > > Good catch, applied, thank you!
> > > > > > 
> > > > > > There is a similar construct in kvm-remote.sh, so I added a similar
> > > > > > fix to your patch.
> > > > > > 
> > > > > > But what about this in functions.sh?
> > > > > > 
> > > > > > nt="`lscpu | grep '^NUMA node0' | sed -e 's/^[^,]*,\([0-9]*\),.*$/\1/'`"
> > > > > > 
> > > > > > I am guessing that "node0" is human-language-independent, but is "NUMA"?
> > > > > 
> > > > > I thought they wouldn't bother translating that, but they did...
> > > > > 
> > > > >     NUMA node0 CPU(s):               0-7
> > > > > 
> > > > > becomes:
> > > > > 
> > > > >     Nœud NUMA 0 de processeur(s) : 0-7
> > > > > 
> > > > > Not sure about the best way to fix it.
> > > > 
> > > > The rude and crude fix is for the scripts to force the local language
> > > > to English.  ;-)
> > > 
> > > I don't have a better answer :o)
> > 
> > If you set the environment variable LANG to en_US.UTF-8, does that
> > make things work for you?  Huh.  Setting it to fr_FR.UTF-8 does not
> > shift lscpu out of English for me, so I am guessing "no".
> 
> Maybe that language isn't installed in your system. I would expect
> en_US.UTF-8 to be supported pretty much everywhere though. At least it
> works for me with: "LANG=en_US.UTF-8 lscpu".
> 
> Thanks.

How about like this?  I put this only in kvm.sh for the moment, but
if these keep cropping up I will just hit all the scripts.  ;-)

							Thanx, Paul

------------------------------------------------------------------------

commit 4ca332016ed81c15ebb3b744dbfc462281c544b8
Author: Paul E. McKenney <paulmck@kernel.org>
Date:   Thu Apr 1 15:26:56 2021 -0700

    torture:  Set kvm.sh language to English
    
    Some of the code invoked directly and indirectly from kvm.sh parses
    the output of commands.  This parsing assumes English, which can cause
    failures if the user has set some other language.  In a few cases,
    there are language-independent commands available, but this is not
    always the case.  Therefore, as an alternative to polyglot parsing,
    this commit sets the LANG environment variable to en_US.UTF-8.
    
    Reported-by: Frederic Weisbecker <frederic@kernel.org>
    Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

diff --git a/tools/testing/selftests/rcutorture/bin/kvm.sh b/tools/testing/selftests/rcutorture/bin/kvm.sh
index fab3bd9..390bb97 100755
--- a/tools/testing/selftests/rcutorture/bin/kvm.sh
+++ b/tools/testing/selftests/rcutorture/bin/kvm.sh
@@ -20,6 +20,9 @@ mkdir $T
 
 cd `dirname $scriptname`/../../../../../
 
+# This script knows only English.
+LANG=en_US.UTF-8; export LANG
+
 dur=$((30*60))
 dryrun=""
 KVM="`pwd`/tools/testing/selftests/rcutorture"; export KVM

Re: [PATCH] torture: Correctly fetch CPUs for kvm-build.sh with all native language

[Frederic Weisbecker]

On Thu, Apr 01, 2021 at 03:33:29PM -0700, Paul E. McKenney wrote:
> On Thu, Apr 01, 2021 at 11:08:02PM +0200, Frederic Weisbecker wrote:
> How about like this?  I put this only in kvm.sh for the moment, but
> if these keep cropping up I will just hit all the scripts.  ;-)

Sure, works for me!

Thanks.

> 
> 							Thanx, Paul
> 
> ------------------------------------------------------------------------
> 
> commit 4ca332016ed81c15ebb3b744dbfc462281c544b8
> Author: Paul E. McKenney <paulmck@kernel.org>
> Date:   Thu Apr 1 15:26:56 2021 -0700
> 
>     torture:  Set kvm.sh language to English
>     
>     Some of the code invoked directly and indirectly from kvm.sh parses
>     the output of commands.  This parsing assumes English, which can cause
>     failures if the user has set some other language.  In a few cases,
>     there are language-independent commands available, but this is not
>     always the case.  Therefore, as an alternative to polyglot parsing,
>     this commit sets the LANG environment variable to en_US.UTF-8.
>     
>     Reported-by: Frederic Weisbecker <frederic@kernel.org>
>     Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> 
> diff --git a/tools/testing/selftests/rcutorture/bin/kvm.sh b/tools/testing/selftests/rcutorture/bin/kvm.sh
> index fab3bd9..390bb97 100755
> --- a/tools/testing/selftests/rcutorture/bin/kvm.sh
> +++ b/tools/testing/selftests/rcutorture/bin/kvm.sh
> @@ -20,6 +20,9 @@ mkdir $T
>  
>  cd `dirname $scriptname`/../../../../../
>  
> +# This script knows only English.
> +LANG=en_US.UTF-8; export LANG
> +
>  dur=$((30*60))
>  dryrun=""
>  KVM="`pwd`/tools/testing/selftests/rcutorture"; export KVM

Re: [PATCH] torture: Correctly fetch CPUs for kvm-build.sh with all native language

[Christian Kujau]

On Thu, 1 Apr 2021, Paul E. McKenney wrote:
> +# This script knows only English.
> +LANG=en_US.UTF-8; export LANG

This, too, will only work if en_US.UTF-8 is installed . Check with "locale 
-a" if it is. Also, Perl will complain loudly if the language is not 
installed (try: "LANG=en_US.UTF-9 perl"), a nice way to test if LANG works 
as expected.

So, wouldn't LANG=C be a more conservative fallback here?

</bikeshed>

Christian.
-- 
BOFH excuse #58:

high pressure system failure

Re: [PATCH] torture: Correctly fetch CPUs for kvm-build.sh with all native language

[Paul E. McKenney]

On Mon, Apr 05, 2021 at 04:23:09PM +0200, Christian Kujau wrote:
> On Thu, 1 Apr 2021, Paul E. McKenney wrote:
> > +# This script knows only English.
> > +LANG=en_US.UTF-8; export LANG
> 
> This, too, will only work if en_US.UTF-8 is installed . Check with "locale 
> -a" if it is. Also, Perl will complain loudly if the language is not 
> installed (try: "LANG=en_US.UTF-9 perl"), a nice way to test if LANG works 
> as expected.
> 
> So, wouldn't LANG=C be a more conservative fallback here?

If it works for Frederic, I am good with it.  And yes, we do need to
have a least-common-denominator solution.

> </bikeshed>

Plaid!!!  We must paint the bikeshed plaid!  ;-)

							Thanx, Paul