[rt-tests v2 v2 00/20] rt-numa.h cleanups

[rt-tests v2 v2 00/20] rt-numa.h cleanups

[Daniel Wagner]

This series cleans up the rt-numa.h header.

As we have a hard dependency on libnuma we can simplify the code in
cyclictest. This allows remove all the small helpers in rt_numa.h. And
with this we can remove the header and reduce the confusion with
rt-numa.h

While at it, I simplified the --smp vs --affinity vs --threads
logic. There is no need for additional variables to keep state. With
this we also make --affinity to behave as with the rest of
rt-tests. That is a plan -a will be the same as with -S. There is no
need for -S anymore but I think we should leave it in place for
backwards compatibility. I suspect, there must be a lot of muscle
memory out there :)

Since signaltest has the same code as cyclictest, cleanup this tool as
well.

With all those cleanups in cyclictest and signaltest, rt-numa.h
contains only a three functions. The final part of the series is to
remove unnecessary function arguments (max_cpus). This also fixes the
theoretical problem that a sparse cpumask would not be handled
correctly.

changes v2:
  - added more cleanup patches (7-20)

Daniel Wagner (20):
  cyclictest: Always use libnuma
  cyclictest: Use numa API directly
  cyclictest: Use affinity_mask for stearing thread placement
  cyclictest: Mimik --smp behavior with --affinity
  cyclictest: Simplify --smp vs --affinity vs --threads argument logic
  cyclictest: Move verbose message into main
  signaltest: Always use libnuma
  signaltest: Use affinity_mask for stearing thread placement
  signaltest:  Simplify --smp vs --affinity vs --threads argument logic
  rt-numa: Remove unused definitions and numa_initialize()
  rt-numa: Add generic cpu_for_thread() helper
  rt-numa: Use mask size for iterator limit
  rt-numa: Remove max_cpus argument from parse_cpusmask
  rt-numa: Use error message helpers
  signaltest: Remove unused max_cpus argument from process_options
  cyclictest: Remove unused max_cpus argument from process_options
  rt-numa: Use CPU_SETSIZE as upper loop limit
  rt-numa: Remove used max_cpus argument from cpu_for_thread()
  cyclictest: Remove max cpus used verbose information
  cyclictest: Remove unecessary local variable

 src/cyclictest/cyclictest.c | 177 +++++++++++++-----------------------
 src/cyclictest/rt_numa.h    |  98 --------------------
 src/include/rt-numa.h       |  13 +--
 src/lib/rt-numa.c           |  44 ++++-----
 src/oslat/oslat.c           |   3 +-
 src/signaltest/signaltest.c |  80 ++++------------
 6 files changed, 101 insertions(+), 314 deletions(-)
 delete mode 100644 src/cyclictest/rt_numa.h

-- 
2.29.2

[rt-tests v2 v2 01/20] cyclictest: Always use libnuma

[Daniel Wagner]

libnuma is hard dependency for cyclictest. Thus we can always call
numa_initialize(). This allows us to remove the global 'numa' variable
to track if libnuma has been initialized or not.

Signed-off-by: Daniel Wagner <dwagner@suse.de>
---
 src/cyclictest/cyclictest.c | 63 +++++++++++++++++--------------------
 src/cyclictest/rt_numa.h    |  2 --
 2 files changed, 29 insertions(+), 36 deletions(-)

diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
index f38c453f1975..514ed7b20fdb 100644
--- a/src/cyclictest/cyclictest.c
+++ b/src/cyclictest/cyclictest.c
@@ -1018,9 +1018,6 @@ static void process_options(int argc, char *argv[], int max_cpus)
 			/* smp sets AFFINITY_USEALL in OPT_SMP */
 			if (smp)
 				break;
-			if (numa_initialize())
-				fatal("Couldn't initialize libnuma");
-			numa = 1;
 			if (optarg) {
 				parse_cpumask(optarg, max_cpus, &affinity_mask);
 				setaffinity = AFFINITY_SPECIFIED;
@@ -1126,8 +1123,6 @@ static void process_options(int argc, char *argv[], int max_cpus)
 			use_system = MODE_SYS_OFFSET; break;
 		case 'S':
 		case OPT_SMP: /* SMP testing */
-			if (numa)
-				fatal("numa and smp options are mutually exclusive\n");
 			smp = 1;
 			num_threads = -1; /* update after parsing */
 			setaffinity = AFFINITY_USEALL;
@@ -1201,16 +1196,17 @@ static void process_options(int argc, char *argv[], int max_cpus)
 
 	/* if smp wasn't requested, test for numa automatically */
 	if (!smp) {
-		if (numa_initialize())
-			fatal("Couldn't initialize libnuma");
-		numa = 1;
 		if (setaffinity == AFFINITY_UNSPECIFIED)
 			setaffinity = AFFINITY_USEALL;
 	}
 
-	if (option_affinity) {
-		if (smp)
-			warn("-a ignored due to smp mode\n");
+	if (option_affinity && smp) {
+		warn("-a ignored due to smp mode\n");
+		if (affinity_mask) {
+			numa_bitmask_free(affinity_mask);
+			affinity_mask = NULL;
+		}
+		setaffinity = AFFINITY_USEALL;
 	}
 
 	if (smi) {
@@ -1744,6 +1740,12 @@ int main(int argc, char **argv)
 	int max_cpus = sysconf(_SC_NPROCESSORS_ONLN);
 	int i, ret = -1;
 	int status;
+	void *stack;
+	void *currstk;
+	size_t stksize;
+
+	if (numa_initialize())
+		fatal("Couldn't initialize libnuma");
 
 	process_options(argc, argv, max_cpus);
 
@@ -1926,34 +1928,27 @@ int main(int argc, char **argv)
 		default: cpu = -1;
 		}
 
-		node = -1;
-		if (numa) {
-			void *stack;
-			void *currstk;
-			size_t stksize;
+		/* find the memory node associated with the cpu i */
+		node = rt_numa_numa_node_of_cpu(cpu);
 
-			/* find the memory node associated with the cpu i */
-			node = rt_numa_numa_node_of_cpu(cpu);
+		/* get the stack size set for this thread */
+		if (pthread_attr_getstack(&attr, &currstk, &stksize))
+			fatal("failed to get stack size for thread %d\n", i);
 
-			/* get the stack size set for this thread */
-			if (pthread_attr_getstack(&attr, &currstk, &stksize))
-				fatal("failed to get stack size for thread %d\n", i);
+		/* if the stack size is zero, set a default */
+		if (stksize == 0)
+			stksize = PTHREAD_STACK_MIN * 2;
 
-			/* if the stack size is zero, set a default */
-			if (stksize == 0)
-				stksize = PTHREAD_STACK_MIN * 2;
+		/*  allocate memory for a stack on appropriate node */
+		stack = rt_numa_numa_alloc_onnode(stksize, node, cpu);
 
-			/*  allocate memory for a stack on appropriate node */
-			stack = rt_numa_numa_alloc_onnode(stksize, node, cpu);
+		/* touch the stack pages to pre-fault them in */
+		memset(stack, 0, stksize);
 
-			/* touch the stack pages to pre-fault them in */
-			memset(stack, 0, stksize);
-
-			/* set the thread's stack */
-			if (pthread_attr_setstack(&attr, stack, stksize))
-				fatal("failed to set stack addr for thread %d to 0x%x\n",
-				      i, stack+stksize);
-		}
+		/* set the thread's stack */
+		if (pthread_attr_setstack(&attr, stack, stksize))
+			fatal("failed to set stack addr for thread %d to 0x%x\n",
+				i, stack+stksize);
 
 		/* allocate the thread's parameter block  */
 		parameters[i] = par = threadalloc(sizeof(struct thread_param), node);
diff --git a/src/cyclictest/rt_numa.h b/src/cyclictest/rt_numa.h
index 46690941e0a6..8d02f419ed6d 100644
--- a/src/cyclictest/rt_numa.h
+++ b/src/cyclictest/rt_numa.h
@@ -13,8 +13,6 @@
 #include "rt-utils.h"
 #include "error.h"
 
-static int numa = 0;
-
 #include <numa.h>
 
 static void *
-- 
2.29.2

Re: [rt-tests v2 v2 01/20] cyclictest: Always use libnuma

[John Kacur]

On Fri, 18 Dec 2020, Daniel Wagner wrote:

> libnuma is hard dependency for cyclictest. Thus we can always call
> numa_initialize(). This allows us to remove the global 'numa' variable
> to track if libnuma has been initialized or not.
> 
> Signed-off-by: Daniel Wagner <dwagner@suse.de>
> ---
>  src/cyclictest/cyclictest.c | 63 +++++++++++++++++--------------------
>  src/cyclictest/rt_numa.h    |  2 --
>  2 files changed, 29 insertions(+), 36 deletions(-)
> 
> diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
> index f38c453f1975..514ed7b20fdb 100644
> --- a/src/cyclictest/cyclictest.c
> +++ b/src/cyclictest/cyclictest.c
> @@ -1018,9 +1018,6 @@ static void process_options(int argc, char *argv[], int max_cpus)
>  			/* smp sets AFFINITY_USEALL in OPT_SMP */
>  			if (smp)
>  				break;
> -			if (numa_initialize())
> -				fatal("Couldn't initialize libnuma");
> -			numa = 1;
>  			if (optarg) {
>  				parse_cpumask(optarg, max_cpus, &affinity_mask);
>  				setaffinity = AFFINITY_SPECIFIED;
> @@ -1126,8 +1123,6 @@ static void process_options(int argc, char *argv[], int max_cpus)
>  			use_system = MODE_SYS_OFFSET; break;
>  		case 'S':
>  		case OPT_SMP: /* SMP testing */
> -			if (numa)
> -				fatal("numa and smp options are mutually exclusive\n");
>  			smp = 1;
>  			num_threads = -1; /* update after parsing */
>  			setaffinity = AFFINITY_USEALL;
> @@ -1201,16 +1196,17 @@ static void process_options(int argc, char *argv[], int max_cpus)
>  
>  	/* if smp wasn't requested, test for numa automatically */
>  	if (!smp) {
> -		if (numa_initialize())
> -			fatal("Couldn't initialize libnuma");
> -		numa = 1;
>  		if (setaffinity == AFFINITY_UNSPECIFIED)
>  			setaffinity = AFFINITY_USEALL;
>  	}
>  
> -	if (option_affinity) {
> -		if (smp)
> -			warn("-a ignored due to smp mode\n");
> +	if (option_affinity && smp) {
> +		warn("-a ignored due to smp mode\n");
> +		if (affinity_mask) {
> +			numa_bitmask_free(affinity_mask);
> +			affinity_mask = NULL;
> +		}
> +		setaffinity = AFFINITY_USEALL;
>  	}
>  
>  	if (smi) {
> @@ -1744,6 +1740,12 @@ int main(int argc, char **argv)
>  	int max_cpus = sysconf(_SC_NPROCESSORS_ONLN);
>  	int i, ret = -1;
>  	int status;
> +	void *stack;
> +	void *currstk;
> +	size_t stksize;
> +
> +	if (numa_initialize())
> +		fatal("Couldn't initialize libnuma");
>  
>  	process_options(argc, argv, max_cpus);
>  
> @@ -1926,34 +1928,27 @@ int main(int argc, char **argv)
>  		default: cpu = -1;
>  		}
>  
> -		node = -1;
> -		if (numa) {
> -			void *stack;
> -			void *currstk;
> -			size_t stksize;
> +		/* find the memory node associated with the cpu i */
> +		node = rt_numa_numa_node_of_cpu(cpu);
>  
> -			/* find the memory node associated with the cpu i */
> -			node = rt_numa_numa_node_of_cpu(cpu);
> +		/* get the stack size set for this thread */
> +		if (pthread_attr_getstack(&attr, &currstk, &stksize))
> +			fatal("failed to get stack size for thread %d\n", i);
>  
> -			/* get the stack size set for this thread */
> -			if (pthread_attr_getstack(&attr, &currstk, &stksize))
> -				fatal("failed to get stack size for thread %d\n", i);
> +		/* if the stack size is zero, set a default */
> +		if (stksize == 0)
> +			stksize = PTHREAD_STACK_MIN * 2;
>  
> -			/* if the stack size is zero, set a default */
> -			if (stksize == 0)
> -				stksize = PTHREAD_STACK_MIN * 2;
> +		/*  allocate memory for a stack on appropriate node */
> +		stack = rt_numa_numa_alloc_onnode(stksize, node, cpu);
>  
> -			/*  allocate memory for a stack on appropriate node */
> -			stack = rt_numa_numa_alloc_onnode(stksize, node, cpu);
> +		/* touch the stack pages to pre-fault them in */
> +		memset(stack, 0, stksize);
>  
> -			/* touch the stack pages to pre-fault them in */
> -			memset(stack, 0, stksize);
> -
> -			/* set the thread's stack */
> -			if (pthread_attr_setstack(&attr, stack, stksize))
> -				fatal("failed to set stack addr for thread %d to 0x%x\n",
> -				      i, stack+stksize);
> -		}
> +		/* set the thread's stack */
> +		if (pthread_attr_setstack(&attr, stack, stksize))
> +			fatal("failed to set stack addr for thread %d to 0x%x\n",
> +				i, stack+stksize);
>  
>  		/* allocate the thread's parameter block  */
>  		parameters[i] = par = threadalloc(sizeof(struct thread_param), node);
> diff --git a/src/cyclictest/rt_numa.h b/src/cyclictest/rt_numa.h
> index 46690941e0a6..8d02f419ed6d 100644
> --- a/src/cyclictest/rt_numa.h
> +++ b/src/cyclictest/rt_numa.h
> @@ -13,8 +13,6 @@
>  #include "rt-utils.h"
>  #include "error.h"
>  
> -static int numa = 0;
> -
>  #include <numa.h>
>  
>  static void *
> -- 
> 2.29.2
> 
> 
Signed-off-by: John Kacur <jkacur@redhat.com>

Re: [rt-tests v2 v2 01/20] cyclictest: Always use libnuma

[Kurt Kanzenbach]

[-- Attachment #1: Type: text/plain, Size: 696 bytes --]

Hi,

On Fri Dec 18 2020, Daniel Wagner wrote:
> libnuma is hard dependency for cyclictest. Thus we can always call
> numa_initialize(). This allows us to remove the global 'numa' variable
> to track if libnuma has been initialized or not.
>
> Signed-off-by: Daniel Wagner <dwagner@suse.de>

It seems like with this particular commit, it's not possible to run
cyclictest on arm32 systems anymore. I guess due to missing NUMA
support?

Just tested on a dual core Cyclone V:

root@tsn:~/rt-tests# ./cyclictest -S -m -p 99 --secaligned
FATAL: Couldn't initialize libnuma
root@tsn:~/rt-tests# 

I've used the current unstable/devel/latest branch. Any suggestions?

Thanks,
Kurt

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [rt-tests v2 v2 01/20] cyclictest: Always use libnuma

[Daniel Wagner]

Hi Kurt,

On Fri, Feb 19, 2021 at 02:44:36PM +0100, Kurt Kanzenbach wrote:
> It seems like with this particular commit, it's not possible to run
> cyclictest on arm32 systems anymore. I guess due to missing NUMA
> support?

Yes, your distro needs to provide libnuma. cyclictest runs fine on arm32
with the library.

> Just tested on a dual core Cyclone V:
> 
> root@tsn:~/rt-tests# ./cyclictest -S -m -p 99 --secaligned
> FATAL: Couldn't initialize libnuma

I think you would see the same error when trying to use the '-a' option
without the patch. The dependency is not new.

> I've used the current unstable/devel/latest branch. Any suggestions?

The simplest thing is obviously to get libnuma on your system. I assume
this is not so simple in your case. In this case you could build
cyclictest a static binary.

First, build numactl as static libary:

  ./configure --enable-static && make

and then rt-tests with

  CFLAGS="-static -L../numactl/.libs/" make

Thanks,
Daniel

Re: [rt-tests v2 v2 01/20] cyclictest: Always use libnuma

[Kurt Kanzenbach]

[-- Attachment #1: Type: text/plain, Size: 1575 bytes --]

Hi Daniel,

On Fri Feb 19 2021, Daniel Wagner wrote:
> Hi Kurt,
>
> On Fri, Feb 19, 2021 at 02:44:36PM +0100, Kurt Kanzenbach wrote:
>> It seems like with this particular commit, it's not possible to run
>> cyclictest on arm32 systems anymore. I guess due to missing NUMA
>> support?
>
> Yes, your distro needs to provide libnuma. cyclictest runs fine on arm32
> with the library.

I'm using Debian and it provides libnuma.

>
>> Just tested on a dual core Cyclone V:
>> 
>> root@tsn:~/rt-tests# ./cyclictest -S -m -p 99 --secaligned
>> FATAL: Couldn't initialize libnuma
>
> I think you would see the same error when trying to use the '-a' option
> without the patch. The dependency is not new.
>
>> I've used the current unstable/devel/latest branch. Any suggestions?
>
> The simplest thing is obviously to get libnuma on your system. I assume
> this is not so simple in your case. In this case you could build
> cyclictest a static binary.
>
> First, build numactl as static libary:
>
>   ./configure --enable-static && make
>
> and then rt-tests with
>
>   CFLAGS="-static -L../numactl/.libs/" make

Static building with the newest numactl library also doesn't work:

|root@tsn:~/rt-tests# file cyclictest 
|cyclictest: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, BuildID[sha1]=248eaf8847544423dc51c6ceea18bbffc487991e, with debug_info, not stripped
|root@tsn:~/rt-tests# ./cyclictest 
|FATAL: Couldn't initialize libnuma
|root@tsn:~/rt-tests#

Hmm.

Thanks,
Kurt

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [rt-tests v2 v2 01/20] cyclictest: Always use libnuma

[Daniel Wagner]

On Fri, Feb 19, 2021 at 03:39:45PM +0100, Kurt Kanzenbach wrote:
> I'm using Debian and it provides libnuma.

Hmm, that is strange. Indeed. I have to update my BBB first to a more
recent Debian release to try this out.

> Static building with the newest numactl library also doesn't work:
> 
> |root@tsn:~/rt-tests# file cyclictest 
> |cyclictest: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, BuildID[sha1]=248eaf8847544423dc51c6ceea18bbffc487991e, with debug_info, not stripped
> |root@tsn:~/rt-tests# ./cyclictest 
> |FATAL: Couldn't initialize libnuma

I try to reproduce this here. Building libnuma from source right now.

Re: [rt-tests v2 v2 01/20] cyclictest: Always use libnuma

[Sebastian Andrzej Siewior]

On 2021-02-19 15:54:22 [+0100], Daniel Wagner wrote:
> I try to reproduce this here. Building libnuma from source right now.

|~# uname -m
|x86_64
|~# ./cyclictest 
|FATAL: Couldn't initialize libnuma~# 

Just do
 # CONFIG_NUMA is not set

in your .config and it is done.

Sebastian

Re: [rt-tests v2 v2 01/20] cyclictest: Always use libnuma

[Christian Eggers]

CONFIG_NUMA is not available on ARM32:

https://cateee.net/lkddb/web-lkddb/NUMA.html

regards,
Christian

On Friday, 19 February 2021, 16:17:17 CET, Sebastian Andrzej Siewior wrote:
> On 2021-02-19 15:54:22 [+0100], Daniel Wagner wrote:
> > I try to reproduce this here. Building libnuma from source right now.
> 
> |~# uname -m
> |x86_64
> |~# ./cyclictest 
> |FATAL: Couldn't initialize libnuma~# 
> 
> Just do
>  # CONFIG_NUMA is not set
> 
> in your .config and it is done.
> 
> Sebastian
> 

Re: [rt-tests v2 v2 01/20] cyclictest: Always use libnuma

[Daniel Wagner]

On Fri, Feb 19, 2021 at 04:21:46PM +0100, Christian Eggers wrote:
> CONFIG_NUMA is not available on ARM32:

libnuma is not happy if the kernel doesn't have the config
option. Well, in this case we just revert the patch I suppose.

Re: [rt-tests v2 v2 01/20] cyclictest: Always use libnuma

[John Kacur]

On Fri, 19 Feb 2021, Daniel Wagner wrote:

> On Fri, Feb 19, 2021 at 04:21:46PM +0100, Christian Eggers wrote:
> > CONFIG_NUMA is not available on ARM32:
> 
> libnuma is not happy if the kernel doesn't have the config
> option. Well, in this case we just revert the patch I suppose.
> 

Yeah, I'm not happy with this.
To be sure Daniel did a lot of good work cleaning-up some of the numa 
calls and cleaning up the somewhat artificial smp / numa divide.

That work needs to continue, but I want the ability to build cyclictest
without NUMA, and that also means we need the small helpers.

John

Re: [rt-tests v2 v2 01/20] cyclictest: Always use libnuma

[Christian Eggers]

On Friday, 19 February 2021, 17:21:21 CET, John Kacur wrote:
> 
> On Fri, 19 Feb 2021, Daniel Wagner wrote:
> 
> > On Fri, Feb 19, 2021 at 04:21:46PM +0100, Christian Eggers wrote:
> > > CONFIG_NUMA is not available on ARM32:
> > 
> > libnuma is not happy if the kernel doesn't have the config
> > option. Well, in this case we just revert the patch I suppose.
> > 
> 
> Yeah, I'm not happy with this.
> To be sure Daniel did a lot of good work cleaning-up some of the numa 
> calls and cleaning up the somewhat artificial smp / numa divide.
> 
> That work needs to continue, but I want the ability to build cyclictest
> without NUMA, and that also means we need the small helpers.

Just for clarification: Does this mean that cyclictest shall link against
libnuma (libnuma can be built on ARM without problems), but not call it
functions at runtime?

From the numa.h:

/* NUMA support available. If this returns a negative value all other function
   in this library are undefined. */
int numa_available(void);

regards,
Christian

Re: [rt-tests v2 v2 01/20] cyclictest: Always use libnuma

[Daniel Wagner]

On 19.02.21 17:27, Christian Eggers wrote:
> Just for clarification: Does this mean that cyclictest shall link against
> libnuma (libnuma can be built on ARM without problems), but not call it
> functions at runtime?

Yes, I think that is the idea of libnuma.

Re: [rt-tests v2 v2 01/20] cyclictest: Always use libnuma

[John Kacur]

On Fri, 19 Feb 2021, Christian Eggers wrote:

> On Friday, 19 February 2021, 17:21:21 CET, John Kacur wrote:
> > 
> > On Fri, 19 Feb 2021, Daniel Wagner wrote:
> > 
> > > On Fri, Feb 19, 2021 at 04:21:46PM +0100, Christian Eggers wrote:
> > > > CONFIG_NUMA is not available on ARM32:
> > > 
> > > libnuma is not happy if the kernel doesn't have the config
> > > option. Well, in this case we just revert the patch I suppose.
> > > 
> > 
> > Yeah, I'm not happy with this.
> > To be sure Daniel did a lot of good work cleaning-up some of the numa 
> > calls and cleaning up the somewhat artificial smp / numa divide.
> > 
> > That work needs to continue, but I want the ability to build cyclictest
> > without NUMA, and that also means we need the small helpers.
> 
> Just for clarification: Does this mean that cyclictest shall link against
> libnuma (libnuma can be built on ARM without problems), but not call it
> functions at runtime?
> 
> From the numa.h:
> 
> /* NUMA support available. If this returns a negative value all other function
>    in this library are undefined. */
> int numa_available(void);
> 
> regards,
> Christian

We used to be able to build cyclictest without libnuma.
The small helpers that wrapped the calls to numa contained versions
without numa calls. Most people did have libnuma, so we were requiring 
libnuma say on x86_64.

I think at some point before Daniel starting removing the wrappers this
support was broken.

So, I mean the ability to build cyclitest without libnuma at least
on some architectures.

John

Re: [rt-tests v2 v2 01/20] cyclictest: Always use libnuma

[John Kacur]

On Fri, 19 Feb 2021, Christian Eggers wrote:

> On Friday, 19 February 2021, 17:21:21 CET, John Kacur wrote:
> > 
> > On Fri, 19 Feb 2021, Daniel Wagner wrote:
> > 
> > > On Fri, Feb 19, 2021 at 04:21:46PM +0100, Christian Eggers wrote:
> > > > CONFIG_NUMA is not available on ARM32:
> > > 
> > > libnuma is not happy if the kernel doesn't have the config
> > > option. Well, in this case we just revert the patch I suppose.
> > > 
> > 
> > Yeah, I'm not happy with this.
> > To be sure Daniel did a lot of good work cleaning-up some of the numa 
> > calls and cleaning up the somewhat artificial smp / numa divide.
> > 
> > That work needs to continue, but I want the ability to build cyclictest
> > without NUMA, and that also means we need the small helpers.
> 
> Just for clarification: Does this mean that cyclictest shall link against
> libnuma (libnuma can be built on ARM without problems), but not call it
> functions at runtime?
> 
> From the numa.h:
> 
> /* NUMA support available. If this returns a negative value all other function
>    in this library are undefined. */
> int numa_available(void);

On the other hand, this is what was working for you before the changes 
right? We required libnuma for building but not for runtime?

That would be easier to maintain.

> 
> regards,
> Christian
> 
> 
> 
> 

Re: [rt-tests v2 v2 01/20] cyclictest: Always use libnuma

[John Kacur]

On Fri, 19 Feb 2021, Kurt Kanzenbach wrote:

> Hi,
> 
> On Fri Dec 18 2020, Daniel Wagner wrote:
> > libnuma is hard dependency for cyclictest. Thus we can always call
> > numa_initialize(). This allows us to remove the global 'numa' variable
> > to track if libnuma has been initialized or not.

Just a small note, the global 'numa' variable was NOT used to track 
whether libnuma had been intialized or not.

The user used to specify smp or numa, then we decided that we would remove 
the numa option.

What happened was that if numa was available and smp was not specified
the program would automatically use numa else it would use smp.

If smp was specified then smp was used. The numa variable would track 
that.

the smp option used to just collect a common set of options
-a -t and priorities all the same.

It made sense way back when, but it's less useful now-a-days

> >
> > Signed-off-by: Daniel Wagner <dwagner@suse.de>
> 
> It seems like with this particular commit, it's not possible to run
> cyclictest on arm32 systems anymore. I guess due to missing NUMA
> support?
> 
> Just tested on a dual core Cyclone V:
> 
> root@tsn:~/rt-tests# ./cyclictest -S -m -p 99 --secaligned
> FATAL: Couldn't initialize libnuma
> root@tsn:~/rt-tests# 
> 
> I've used the current unstable/devel/latest branch. Any suggestions?
> 
> Thanks,
> Kurt
> 

[rt-tests v2 v2 02/20] cyclictest: Use numa API directly

[Daniel Wagner]

There is no need for small libnuma wrappers functions as we always
use libnuma. Remove them and get rid of rt_numa.h, so there is
no confusion with rt-numa.h anymore.

Signed-off-by: Daniel Wagner <dwagner@suse.de>
---
 src/cyclictest/cyclictest.c | 41 ++++++++--------
 src/cyclictest/rt_numa.h    | 96 -------------------------------------
 2 files changed, 22 insertions(+), 115 deletions(-)
 delete mode 100644 src/cyclictest/rt_numa.h

diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
index 514ed7b20fdb..a5ca764da254 100644
--- a/src/cyclictest/cyclictest.c
+++ b/src/cyclictest/cyclictest.c
@@ -33,10 +33,10 @@
 #include <sys/utsname.h>
 #include <sys/mman.h>
 #include <sys/syscall.h>
-#include "rt_numa.h"
 
 #include "rt-utils.h"
 #include "rt-numa.h"
+#include "error.h"
 
 #include <bionic.h>
 
@@ -514,9 +514,9 @@ static void *timerthread(void *param)
 
 	memset(&stop, 0, sizeof(stop));
 
-	/* if we're running in numa mode, set our memory node */
-	if (par->node != -1)
-		rt_numa_set_numa_run_on_node(par->node, par->cpu);
+	if (numa_run_on_node(par->node))
+		warn("Could not set NUMA node %d for thread %d: %s\n",
+				par->node, par->cpu, strerror(errno));
 
 	if (par->cpu != -1) {
 		CPU_ZERO(&mask);
@@ -1275,7 +1275,7 @@ static void process_options(int argc, char *argv[], int max_cpus)
 	}
 	if (error) {
 		if (affinity_mask)
-			rt_bitmask_free(affinity_mask);
+			numa_bitmask_free(affinity_mask);
 		display_help(1);
 	}
 }
@@ -1929,7 +1929,7 @@ int main(int argc, char **argv)
 		}
 
 		/* find the memory node associated with the cpu i */
-		node = rt_numa_numa_node_of_cpu(cpu);
+		node = numa_node_of_cpu(cpu);
 
 		/* get the stack size set for this thread */
 		if (pthread_attr_getstack(&attr, &currstk, &stksize))
@@ -1940,7 +1940,10 @@ int main(int argc, char **argv)
 			stksize = PTHREAD_STACK_MIN * 2;
 
 		/*  allocate memory for a stack on appropriate node */
-		stack = rt_numa_numa_alloc_onnode(stksize, node, cpu);
+		stack = numa_alloc_onnode(stksize, node);
+		if (!stack)
+			fatal("failed to allocate %d bytes on node %d for cpu %d\n",
+				stksize, node, cpu);
 
 		/* touch the stack pages to pre-fault them in */
 		memset(stack, 0, stksize);
@@ -1951,13 +1954,13 @@ int main(int argc, char **argv)
 				i, stack+stksize);
 
 		/* allocate the thread's parameter block  */
-		parameters[i] = par = threadalloc(sizeof(struct thread_param), node);
+		parameters[i] = par = numa_alloc_onnode(sizeof(struct thread_param), node);
 		if (par == NULL)
 			fatal("error allocating thread_param struct for thread %d\n", i);
 		memset(par, 0, sizeof(struct thread_param));
 
 		/* allocate the thread's statistics block */
-		statistics[i] = stat = threadalloc(sizeof(struct thread_stat), node);
+		statistics[i] = stat = numa_alloc_onnode(sizeof(struct thread_stat), node);
 		if (stat == NULL)
 			fatal("error allocating thread status struct for thread %d\n", i);
 		memset(stat, 0, sizeof(struct thread_stat));
@@ -1966,8 +1969,8 @@ int main(int argc, char **argv)
 		if (histogram) {
 			int bufsize = histogram * sizeof(long);
 
-			stat->hist_array = threadalloc(bufsize, node);
-			stat->outliers = threadalloc(bufsize, node);
+			stat->hist_array = numa_alloc_onnode(bufsize, node);
+			stat->outliers = numa_alloc_onnode(bufsize, node);
 			if (stat->hist_array == NULL || stat->outliers == NULL)
 				fatal("failed to allocate histogram of size %d on node %d\n",
 				      histogram, i);
@@ -1977,14 +1980,14 @@ int main(int argc, char **argv)
 
 		if (verbose) {
 			int bufsize = VALBUF_SIZE * sizeof(long);
-			stat->values = threadalloc(bufsize, node);
+			stat->values = numa_alloc_onnode(bufsize, node);
 			if (!stat->values)
 				goto outall;
 			memset(stat->values, 0, bufsize);
 			par->bufmsk = VALBUF_SIZE - 1;
 			if (smi) {
 				int bufsize = VALBUF_SIZE * sizeof(long);
-				stat->smis = threadalloc(bufsize, node);
+				stat->smis = numa_alloc_onnode(bufsize, node);
 				if (!stat->smis)
 					goto outall;
 				memset(stat->smis, 0, bufsize);
@@ -2099,7 +2102,7 @@ int main(int argc, char **argv)
 				print_stat(stdout, parameters[i], i, 0, 0);
 		}
 		if (statistics[i]->values)
-			threadfree(statistics[i]->values, VALBUF_SIZE*sizeof(long), parameters[i]->node);
+			numa_free(statistics[i]->values, VALBUF_SIZE*sizeof(long));
 	}
 
 	if (trigger)
@@ -2108,8 +2111,8 @@ int main(int argc, char **argv)
 	if (histogram) {
 		print_hist(parameters, num_threads);
 		for (i = 0; i < num_threads; i++) {
-			threadfree(statistics[i]->hist_array, histogram*sizeof(long), parameters[i]->node);
-			threadfree(statistics[i]->outliers, histogram*sizeof(long), parameters[i]->node);
+			numa_free(statistics[i]->hist_array, histogram*sizeof(long));
+			numa_free(statistics[i]->outliers, histogram*sizeof(long));
 		}
 	}
 
@@ -2125,14 +2128,14 @@ int main(int argc, char **argv)
 	for (i=0; i < num_threads; i++) {
 		if (!statistics[i])
 			continue;
-		threadfree(statistics[i], sizeof(struct thread_stat), parameters[i]->node);
+		numa_free(statistics[i], sizeof(struct thread_stat));
 	}
 
  outpar:
 	for (i = 0; i < num_threads; i++) {
 		if (!parameters[i])
 			continue;
-		threadfree(parameters[i], sizeof(struct thread_param), parameters[i]->node);
+		numa_free(parameters[i], sizeof(struct thread_param));
 	}
  out:
 	/* close any tracer file descriptors */
@@ -2147,7 +2150,7 @@ int main(int argc, char **argv)
 		close(latency_target_fd);
 
 	if (affinity_mask)
-		rt_bitmask_free(affinity_mask);
+		numa_bitmask_free(affinity_mask);
 
 	/* Remove running status shared memory file if it exists */
 	if (rstat_fd >= 0)
diff --git a/src/cyclictest/rt_numa.h b/src/cyclictest/rt_numa.h
deleted file mode 100644
index 8d02f419ed6d..000000000000
--- a/src/cyclictest/rt_numa.h
+++ /dev/null
@@ -1,96 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * A numa library for cyclictest.
- *
- * (C) 2010 John Kacur <jkacur@redhat.com>
- * (C) 2010 Clark Williams <williams@redhat.com>
- *
- */
-
-#ifndef _RT_NUMA_H
-#define _RT_NUMA_H
-
-#include "rt-utils.h"
-#include "error.h"
-
-#include <numa.h>
-
-static void *
-threadalloc(size_t size, int node)
-{
-	if (node == -1)
-		return malloc(size);
-	return numa_alloc_onnode(size, node);
-}
-
-static void
-threadfree(void *ptr, size_t size, int node)
-{
-	if (node == -1)
-		free(ptr);
-	else
-		numa_free(ptr, size);
-}
-
-static void rt_numa_set_numa_run_on_node(int node, int cpu)
-{
-	int res;
-	res = numa_run_on_node(node);
-	if (res)
-		warn("Could not set NUMA node %d for thread %d: %s\n",
-				node, cpu, strerror(errno));
-	return;
-}
-
-static void *rt_numa_numa_alloc_onnode(size_t size, int node, int cpu)
-{
-	void *stack;
-	stack = numa_alloc_onnode(size, node);
-	if (stack == NULL)
-		fatal("failed to allocate %d bytes on node %d for cpu %d\n",
-				size, node, cpu);
-	return stack;
-}
-
-/*
- * Use new bit mask CPU affinity behavior
- */
-static int rt_numa_numa_node_of_cpu(int cpu)
-{
-	int node;
-	node = numa_node_of_cpu(cpu);
-	if (node == -1)
-		fatal("invalid cpu passed to numa_node_of_cpu(%d)\n", cpu);
-	return node;
-}
-
-static inline unsigned int rt_numa_bitmask_isbitset( const struct bitmask *mask,
-	unsigned long i)
-{
-	return numa_bitmask_isbitset(mask,i);
-}
-
-static inline struct bitmask* rt_numa_parse_cpustring(const char* s,
-	int max_cpus)
-{
-	return numa_parse_cpustring_all(s);
-}
-
-static inline void rt_bitmask_free(struct bitmask *mask)
-{
-	numa_bitmask_free(mask);
-}
-
-/** Returns number of bits set in mask. */
-static inline unsigned int rt_numa_bitmask_count(const struct bitmask *mask)
-{
-	unsigned int num_bits = 0, i;
-	for (i = 0; i < mask->size; i++) {
-		if (rt_numa_bitmask_isbitset(mask, i))
-			num_bits++;
-	}
-	/* Could stash this instead of recomputing every time. */
-	return num_bits;
-}
-
-#endif	/* _RT_NUMA_H */
-- 
2.29.2

Re: [rt-tests v2 v2 02/20] cyclictest: Use numa API directly

[John Kacur]

On Fri, 18 Dec 2020, Daniel Wagner wrote:

> There is no need for small libnuma wrappers functions as we always
> use libnuma. Remove them and get rid of rt_numa.h, so there is
> no confusion with rt-numa.h anymore.
> 
> Signed-off-by: Daniel Wagner <dwagner@suse.de>
> ---
>  src/cyclictest/cyclictest.c | 41 ++++++++--------
>  src/cyclictest/rt_numa.h    | 96 -------------------------------------
>  2 files changed, 22 insertions(+), 115 deletions(-)
>  delete mode 100644 src/cyclictest/rt_numa.h
> 
> diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
> index 514ed7b20fdb..a5ca764da254 100644
> --- a/src/cyclictest/cyclictest.c
> +++ b/src/cyclictest/cyclictest.c
> @@ -33,10 +33,10 @@
>  #include <sys/utsname.h>
>  #include <sys/mman.h>
>  #include <sys/syscall.h>
> -#include "rt_numa.h"
>  
>  #include "rt-utils.h"
>  #include "rt-numa.h"
> +#include "error.h"
>  
>  #include <bionic.h>
>  
> @@ -514,9 +514,9 @@ static void *timerthread(void *param)
>  
>  	memset(&stop, 0, sizeof(stop));
>  
> -	/* if we're running in numa mode, set our memory node */
> -	if (par->node != -1)
> -		rt_numa_set_numa_run_on_node(par->node, par->cpu);
> +	if (numa_run_on_node(par->node))
> +		warn("Could not set NUMA node %d for thread %d: %s\n",
> +				par->node, par->cpu, strerror(errno));
>  
>  	if (par->cpu != -1) {
>  		CPU_ZERO(&mask);
> @@ -1275,7 +1275,7 @@ static void process_options(int argc, char *argv[], int max_cpus)
>  	}
>  	if (error) {
>  		if (affinity_mask)
> -			rt_bitmask_free(affinity_mask);
> +			numa_bitmask_free(affinity_mask);
>  		display_help(1);
>  	}
>  }
> @@ -1929,7 +1929,7 @@ int main(int argc, char **argv)
>  		}
>  
>  		/* find the memory node associated with the cpu i */
> -		node = rt_numa_numa_node_of_cpu(cpu);
> +		node = numa_node_of_cpu(cpu);

You've eliminated the error handling that was in the wrapper here.

>  
>  		/* get the stack size set for this thread */
>  		if (pthread_attr_getstack(&attr, &currstk, &stksize))
> @@ -1940,7 +1940,10 @@ int main(int argc, char **argv)
>  			stksize = PTHREAD_STACK_MIN * 2;
>  
>  		/*  allocate memory for a stack on appropriate node */
> -		stack = rt_numa_numa_alloc_onnode(stksize, node, cpu);
> +		stack = numa_alloc_onnode(stksize, node);
> +		if (!stack)
> +			fatal("failed to allocate %d bytes on node %d for cpu %d\n",
> +				stksize, node, cpu);
>  
>  		/* touch the stack pages to pre-fault them in */
>  		memset(stack, 0, stksize);
> @@ -1951,13 +1954,13 @@ int main(int argc, char **argv)
>  				i, stack+stksize);
>  
>  		/* allocate the thread's parameter block  */
> -		parameters[i] = par = threadalloc(sizeof(struct thread_param), node);
> +		parameters[i] = par = numa_alloc_onnode(sizeof(struct thread_param), node);
>  		if (par == NULL)
>  			fatal("error allocating thread_param struct for thread %d\n", i);
>  		memset(par, 0, sizeof(struct thread_param));
>  
>  		/* allocate the thread's statistics block */
> -		statistics[i] = stat = threadalloc(sizeof(struct thread_stat), node);
> +		statistics[i] = stat = numa_alloc_onnode(sizeof(struct thread_stat), node);
>  		if (stat == NULL)
>  			fatal("error allocating thread status struct for thread %d\n", i);
>  		memset(stat, 0, sizeof(struct thread_stat));
> @@ -1966,8 +1969,8 @@ int main(int argc, char **argv)
>  		if (histogram) {
>  			int bufsize = histogram * sizeof(long);
>  
> -			stat->hist_array = threadalloc(bufsize, node);
> -			stat->outliers = threadalloc(bufsize, node);
> +			stat->hist_array = numa_alloc_onnode(bufsize, node);
> +			stat->outliers = numa_alloc_onnode(bufsize, node);
>  			if (stat->hist_array == NULL || stat->outliers == NULL)
>  				fatal("failed to allocate histogram of size %d on node %d\n",
>  				      histogram, i);
> @@ -1977,14 +1980,14 @@ int main(int argc, char **argv)
>  
>  		if (verbose) {
>  			int bufsize = VALBUF_SIZE * sizeof(long);
> -			stat->values = threadalloc(bufsize, node);
> +			stat->values = numa_alloc_onnode(bufsize, node);
>  			if (!stat->values)
>  				goto outall;
>  			memset(stat->values, 0, bufsize);
>  			par->bufmsk = VALBUF_SIZE - 1;
>  			if (smi) {
>  				int bufsize = VALBUF_SIZE * sizeof(long);
> -				stat->smis = threadalloc(bufsize, node);
> +				stat->smis = numa_alloc_onnode(bufsize, node);
>  				if (!stat->smis)
>  					goto outall;
>  				memset(stat->smis, 0, bufsize);
> @@ -2099,7 +2102,7 @@ int main(int argc, char **argv)
>  				print_stat(stdout, parameters[i], i, 0, 0);
>  		}
>  		if (statistics[i]->values)
> -			threadfree(statistics[i]->values, VALBUF_SIZE*sizeof(long), parameters[i]->node);
> +			numa_free(statistics[i]->values, VALBUF_SIZE*sizeof(long));
>  	}
>  
>  	if (trigger)
> @@ -2108,8 +2111,8 @@ int main(int argc, char **argv)
>  	if (histogram) {
>  		print_hist(parameters, num_threads);
>  		for (i = 0; i < num_threads; i++) {
> -			threadfree(statistics[i]->hist_array, histogram*sizeof(long), parameters[i]->node);
> -			threadfree(statistics[i]->outliers, histogram*sizeof(long), parameters[i]->node);
> +			numa_free(statistics[i]->hist_array, histogram*sizeof(long));
> +			numa_free(statistics[i]->outliers, histogram*sizeof(long));
>  		}
>  	}
>  
> @@ -2125,14 +2128,14 @@ int main(int argc, char **argv)
>  	for (i=0; i < num_threads; i++) {
>  		if (!statistics[i])
>  			continue;
> -		threadfree(statistics[i], sizeof(struct thread_stat), parameters[i]->node);
> +		numa_free(statistics[i], sizeof(struct thread_stat));
>  	}
>  
>   outpar:
>  	for (i = 0; i < num_threads; i++) {
>  		if (!parameters[i])
>  			continue;
> -		threadfree(parameters[i], sizeof(struct thread_param), parameters[i]->node);
> +		numa_free(parameters[i], sizeof(struct thread_param));
>  	}
>   out:
>  	/* close any tracer file descriptors */
> @@ -2147,7 +2150,7 @@ int main(int argc, char **argv)
>  		close(latency_target_fd);
>  
>  	if (affinity_mask)
> -		rt_bitmask_free(affinity_mask);
> +		numa_bitmask_free(affinity_mask);
>  
>  	/* Remove running status shared memory file if it exists */
>  	if (rstat_fd >= 0)
> diff --git a/src/cyclictest/rt_numa.h b/src/cyclictest/rt_numa.h
> deleted file mode 100644
> index 8d02f419ed6d..000000000000
> --- a/src/cyclictest/rt_numa.h
> +++ /dev/null
> @@ -1,96 +0,0 @@
> -// SPDX-License-Identifier: GPL-2.0-or-later
> -/*
> - * A numa library for cyclictest.
> - *
> - * (C) 2010 John Kacur <jkacur@redhat.com>
> - * (C) 2010 Clark Williams <williams@redhat.com>
> - *
> - */
> -
> -#ifndef _RT_NUMA_H
> -#define _RT_NUMA_H
> -
> -#include "rt-utils.h"
> -#include "error.h"
> -
> -#include <numa.h>
> -
> -static void *
> -threadalloc(size_t size, int node)
> -{
> -	if (node == -1)
> -		return malloc(size);
> -	return numa_alloc_onnode(size, node);
> -}
> -
> -static void
> -threadfree(void *ptr, size_t size, int node)
> -{
> -	if (node == -1)
> -		free(ptr);
> -	else
> -		numa_free(ptr, size);
> -}
> -
> -static void rt_numa_set_numa_run_on_node(int node, int cpu)
> -{
> -	int res;
> -	res = numa_run_on_node(node);
> -	if (res)
> -		warn("Could not set NUMA node %d for thread %d: %s\n",
> -				node, cpu, strerror(errno));
> -	return;
> -}
> -
> -static void *rt_numa_numa_alloc_onnode(size_t size, int node, int cpu)
> -{
> -	void *stack;
> -	stack = numa_alloc_onnode(size, node);
> -	if (stack == NULL)
> -		fatal("failed to allocate %d bytes on node %d for cpu %d\n",
> -				size, node, cpu);
> -	return stack;
> -}
> -
> -/*
> - * Use new bit mask CPU affinity behavior
> - */
> -static int rt_numa_numa_node_of_cpu(int cpu)
> -{
> -	int node;
> -	node = numa_node_of_cpu(cpu);
> -	if (node == -1)
> -		fatal("invalid cpu passed to numa_node_of_cpu(%d)\n", cpu);
> -	return node;
> -}
> -
> -static inline unsigned int rt_numa_bitmask_isbitset( const struct bitmask *mask,
> -	unsigned long i)
> -{
> -	return numa_bitmask_isbitset(mask,i);
> -}
> -
> -static inline struct bitmask* rt_numa_parse_cpustring(const char* s,
> -	int max_cpus)
> -{
> -	return numa_parse_cpustring_all(s);
> -}
> -
> -static inline void rt_bitmask_free(struct bitmask *mask)
> -{
> -	numa_bitmask_free(mask);
> -}
> -
> -/** Returns number of bits set in mask. */
> -static inline unsigned int rt_numa_bitmask_count(const struct bitmask *mask)
> -{
> -	unsigned int num_bits = 0, i;
> -	for (i = 0; i < mask->size; i++) {
> -		if (rt_numa_bitmask_isbitset(mask, i))
> -			num_bits++;
> -	}
> -	/* Could stash this instead of recomputing every time. */
> -	return num_bits;
> -}
> -
> -#endif	/* _RT_NUMA_H */
> -- 
> 2.29.2
> 
> 

Re: [rt-tests v2 v2 02/20] cyclictest: Use numa API directly

[Daniel Wagner]

On Tue, Jan 26, 2021 at 12:31:37AM -0500, John Kacur wrote:
> > @@ -1929,7 +1929,7 @@ int main(int argc, char **argv)
> >  		}
> >  
> >  		/* find the memory node associated with the cpu i */
> > -		node = rt_numa_numa_node_of_cpu(cpu);
> > +		node = numa_node_of_cpu(cpu);
> 
> You've eliminated the error handling that was in the wrapper here.

I was under the impression that cpu will always be a valid. You are
right, we can't be sure of that, because we only warn if the
affinity_mask is not correct (should this be a error?) and things could
change until we reach this point. I'll add the error handling back.

[rt-tests v2 v2 03/20] cyclictest: Use affinity_mask for stearing thread placement

[Daniel Wagner]

We don't need an extra variable to track the state if a bitmask is
available or not. Check directly if the mask is usable.

Signed-off-by: Daniel Wagner <dwagner@suse.de>
---
 src/cyclictest/cyclictest.c | 34 +++++++---------------------------
 1 file changed, 7 insertions(+), 27 deletions(-)

diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
index a5ca764da254..3dafef4e7e5a 100644
--- a/src/cyclictest/cyclictest.c
+++ b/src/cyclictest/cyclictest.c
@@ -893,7 +893,6 @@ static int interval = DEFAULT_INTERVAL;
 static int distance = -1;
 static struct bitmask *affinity_mask = NULL;
 static int smp = 0;
-static int setaffinity = AFFINITY_UNSPECIFIED;
 
 static int clocksources[] = {
 	CLOCK_MONOTONIC,
@@ -1020,19 +1019,13 @@ static void process_options(int argc, char *argv[], int max_cpus)
 				break;
 			if (optarg) {
 				parse_cpumask(optarg, max_cpus, &affinity_mask);
-				setaffinity = AFFINITY_SPECIFIED;
 			} else if (optind < argc &&
 				   (atoi(argv[optind]) ||
 				    argv[optind][0] == '0' ||
 				    argv[optind][0] == '!')) {
 				parse_cpumask(argv[optind], max_cpus, &affinity_mask);
-				setaffinity = AFFINITY_SPECIFIED;
-			} else {
-				setaffinity = AFFINITY_USEALL;
 			}
 
-			if (setaffinity == AFFINITY_SPECIFIED && !affinity_mask)
-				display_help(1);
 			if (verbose)
 				printf("Using %u cpus.\n",
 					numa_bitmask_weight(affinity_mask));
@@ -1125,7 +1118,6 @@ static void process_options(int argc, char *argv[], int max_cpus)
 		case OPT_SMP: /* SMP testing */
 			smp = 1;
 			num_threads = -1; /* update after parsing */
-			setaffinity = AFFINITY_USEALL;
 			break;
 		case 't':
 		case OPT_THREADS:
@@ -1194,23 +1186,16 @@ static void process_options(int argc, char *argv[], int max_cpus)
 		use_nanosleep = MODE_CLOCK_NANOSLEEP;
 	}
 
-	/* if smp wasn't requested, test for numa automatically */
-	if (!smp) {
-		if (setaffinity == AFFINITY_UNSPECIFIED)
-			setaffinity = AFFINITY_USEALL;
-	}
-
 	if (option_affinity && smp) {
 		warn("-a ignored due to smp mode\n");
 		if (affinity_mask) {
 			numa_bitmask_free(affinity_mask);
 			affinity_mask = NULL;
 		}
-		setaffinity = AFFINITY_USEALL;
 	}
 
 	if (smi) {
-		if (setaffinity == AFFINITY_UNSPECIFIED)
+		if (affinity_mask)
 			fatal("SMI counter relies on thread affinity\n");
 
 		if (!has_smi_counter())
@@ -1756,7 +1741,7 @@ int main(int argc, char **argv)
 		printf("Max CPUs = %d\n", max_cpus);
 
 	/* Restrict the main pid to the affinity specified by the user */
-	if (affinity_mask != NULL) {
+	if (affinity_mask) {
 		int res;
 
 		errno = 0;
@@ -1915,18 +1900,13 @@ int main(int argc, char **argv)
 		if (status != 0)
 			fatal("error from pthread_attr_init for thread %d: %s\n", i, strerror(status));
 
-		switch (setaffinity) {
-		case AFFINITY_UNSPECIFIED: cpu = -1; break;
-		case AFFINITY_SPECIFIED:
+		if (affinity_mask)
 			cpu = cpu_for_thread_sp(i, max_cpus, affinity_mask);
-			if (verbose)
-				printf("Thread %d using cpu %d.\n", i, cpu);
-			break;
-		case AFFINITY_USEALL:
+		else
 			cpu = cpu_for_thread_ua(i, max_cpus);
-			break;
-		default: cpu = -1;
-		}
+
+		if (verbose)
+			printf("Thread %d using cpu %d.\n", i, cpu);
 
 		/* find the memory node associated with the cpu i */
 		node = numa_node_of_cpu(cpu);
-- 
2.29.2

[rt-tests v2 v2 04/20] cyclictest: Mimik --smp behavior with --affinity

[Daniel Wagner]

Let --affinity without an argument behave in the same way as
--smp. Run a thread on each CPU. This makes cyclictest behave as the
rest of the rt-tests when --affinity is used.

Reviewed-by: Daniel Wagner <dwagner@suse.de>
---
 src/cyclictest/cyclictest.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
index 3dafef4e7e5a..23243ddfe9d3 100644
--- a/src/cyclictest/cyclictest.c
+++ b/src/cyclictest/cyclictest.c
@@ -1024,6 +1024,8 @@ static void process_options(int argc, char *argv[], int max_cpus)
 				    argv[optind][0] == '0' ||
 				    argv[optind][0] == '!')) {
 				parse_cpumask(argv[optind], max_cpus, &affinity_mask);
+			} else {
+				num_threads = -1;
 			}
 
 			if (verbose)
-- 
2.29.2

[rt-tests v2 v2 05/20] cyclictest: Simplify --smp vs --affinity vs --threads argument logic

[Daniel Wagner]

Allow each command line only to update one variable and do the final
decission at the end of the parsing step. With this the order
of the command line is not important.

Signed-off-by: Daniel Wagner <dwagner@suse.de>
---
 src/cyclictest/cyclictest.c | 34 ++++++++--------------------------
 1 file changed, 8 insertions(+), 26 deletions(-)

diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
index 23243ddfe9d3..a34b2140f940 100644
--- a/src/cyclictest/cyclictest.c
+++ b/src/cyclictest/cyclictest.c
@@ -885,14 +885,13 @@ static int timermode = TIMER_ABSTIME;
 static int use_system;
 static int priority;
 static int policy = SCHED_OTHER;	/* default policy if not specified */
-static int num_threads = 1;
+static int num_threads = -1;
 static int max_cycles;
 static int clocksel = 0;
 static int quiet;
 static int interval = DEFAULT_INTERVAL;
 static int distance = -1;
 static struct bitmask *affinity_mask = NULL;
-static int smp = 0;
 
 static int clocksources[] = {
 	CLOCK_MONOTONIC,
@@ -958,7 +957,7 @@ enum option_values {
 static void process_options(int argc, char *argv[], int max_cpus)
 {
 	int error = 0;
-	int option_affinity = 0;
+	int smp = 0;
 
 	for (;;) {
 		int option_index = 0;
@@ -1013,10 +1012,6 @@ static void process_options(int argc, char *argv[], int max_cpus)
 		switch (c) {
 		case 'a':
 		case OPT_AFFINITY:
-			option_affinity = 1;
-			/* smp sets AFFINITY_USEALL in OPT_SMP */
-			if (smp)
-				break;
 			if (optarg) {
 				parse_cpumask(optarg, max_cpus, &affinity_mask);
 			} else if (optind < argc &&
@@ -1024,8 +1019,6 @@ static void process_options(int argc, char *argv[], int max_cpus)
 				    argv[optind][0] == '0' ||
 				    argv[optind][0] == '!')) {
 				parse_cpumask(argv[optind], max_cpus, &affinity_mask);
-			} else {
-				num_threads = -1;
 			}
 
 			if (verbose)
@@ -1117,22 +1110,14 @@ static void process_options(int argc, char *argv[], int max_cpus)
 		case OPT_SYSTEM:
 			use_system = MODE_SYS_OFFSET; break;
 		case 'S':
-		case OPT_SMP: /* SMP testing */
-			smp = 1;
-			num_threads = -1; /* update after parsing */
-			break;
+		case OPT_SMP:
+			smp = 1; break;
 		case 't':
 		case OPT_THREADS:
-			if (smp) {
-				warn("-t ignored due to smp mode\n");
-				break;
-			}
 			if (optarg != NULL)
 				num_threads = atoi(optarg);
 			else if (optind < argc && atoi(argv[optind]))
 				num_threads = atoi(argv[optind]);
-			else
-				num_threads = -1; /* update after parsing */
 			break;
 		case OPT_TRIGGER:
 			trigger = atoi(optarg);
@@ -1188,13 +1173,10 @@ static void process_options(int argc, char *argv[], int max_cpus)
 		use_nanosleep = MODE_CLOCK_NANOSLEEP;
 	}
 
-	if (option_affinity && smp) {
-		warn("-a ignored due to smp mode\n");
-		if (affinity_mask) {
-			numa_bitmask_free(affinity_mask);
-			affinity_mask = NULL;
-		}
-	}
+	if (smp && num_threads != -1)
+		warn("--threads overwrites smp mode\n");
+	if (smp && affinity_mask)
+		warn("--affinity overwrites smp mode\n");
 
 	if (smi) {
 		if (affinity_mask)
-- 
2.29.2

[rt-tests v2 v2 06/20] cyclictest: Move verbose message into main

[Daniel Wagner]

By moving the verbose message down we print the message with the final
affinity_mask. This also handles the case where the bitmask is not set.

Signed-off-by: Daniel Wagner <dwagner@suse.de>
---
 src/cyclictest/cyclictest.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
index a34b2140f940..7e84c25f7070 100644
--- a/src/cyclictest/cyclictest.c
+++ b/src/cyclictest/cyclictest.c
@@ -1020,10 +1020,6 @@ static void process_options(int argc, char *argv[], int max_cpus)
 				    argv[optind][0] == '!')) {
 				parse_cpumask(argv[optind], max_cpus, &affinity_mask);
 			}
-
-			if (verbose)
-				printf("Using %u cpus.\n",
-					numa_bitmask_weight(affinity_mask));
 			break;
 		case 'A':
 		case OPT_ALIGNED:
@@ -1732,6 +1728,10 @@ int main(int argc, char **argv)
 		res = numa_sched_setaffinity(getpid(), affinity_mask);
 		if (res != 0)
 			warn("Couldn't setaffinity in main thread: %s\n", strerror(errno));
+
+		if (verbose)
+			printf("Using %u cpus.\n",
+				numa_bitmask_weight(affinity_mask));
 	}
 
 	if (trigger) {
-- 
2.29.2

Re: [rt-tests v2 v2 06/20] cyclictest: Move verbose message into main

[John Kacur]

On Fri, 18 Dec 2020, Daniel Wagner wrote:

> By moving the verbose message down we print the message with the final
> affinity_mask. This also handles the case where the bitmask is not set.
> 
> Signed-off-by: Daniel Wagner <dwagner@suse.de>
> ---
>  src/cyclictest/cyclictest.c | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
> index a34b2140f940..7e84c25f7070 100644
> --- a/src/cyclictest/cyclictest.c
> +++ b/src/cyclictest/cyclictest.c
> @@ -1020,10 +1020,6 @@ static void process_options(int argc, char *argv[], int max_cpus)
>  				    argv[optind][0] == '!')) {
>  				parse_cpumask(argv[optind], max_cpus, &affinity_mask);
>  			}
> -
> -			if (verbose)
> -				printf("Using %u cpus.\n",
> -					numa_bitmask_weight(affinity_mask));
>  			break;
>  		case 'A':
>  		case OPT_ALIGNED:
> @@ -1732,6 +1728,10 @@ int main(int argc, char **argv)
>  		res = numa_sched_setaffinity(getpid(), affinity_mask);
>  		if (res != 0)
>  			warn("Couldn't setaffinity in main thread: %s\n", strerror(errno));
> +
> +		if (verbose)
> +			printf("Using %u cpus.\n",
> +				numa_bitmask_weight(affinity_mask));
>  	}
>  
>  	if (trigger) {
> -- 
> 2.29.2
> 
> 
Signed-off-by: John Kacur <jkacur@redhat.com>

[rt-tests v2 v2 07/20] signaltest: Always use libnuma

[Daniel Wagner]

libnuma is hard dependency for signaltest. Thus we can always call
numa_initialize(). This allows us to remove the global 'numa' variable
to track if libnuma has been initialized or not.

Signed-off-by: Daniel Wagner <dwagner@suse.de>
---
 src/signaltest/signaltest.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/src/signaltest/signaltest.c b/src/signaltest/signaltest.c
index 918d2ab98f6e..b3a82f8c4f65 100644
--- a/src/signaltest/signaltest.c
+++ b/src/signaltest/signaltest.c
@@ -206,7 +206,6 @@ static int quiet;
 static int lockall;
 static struct bitmask *affinity_mask = NULL;
 static int smp = 0;
-static int numa = 0;
 static int setaffinity = AFFINITY_UNSPECIFIED;
 
 /* Process commandline options */
@@ -214,6 +213,7 @@ static void process_options(int argc, char *argv[], unsigned int max_cpus)
 {
 	int option_affinity = 0;
 	int error = 0;
+	int numa = 0;
 
 	for (;;) {
 		int option_index = 0;
@@ -242,8 +242,6 @@ static void process_options(int argc, char *argv[], unsigned int max_cpus)
 			/* smp sets AFFINITY_USEALL in OPT_SMP */
 			if (smp)
 				break;
-			if (numa_initialize())
-				fatal("Couldn't initialize libnuma");
 			numa = 1;
 			if (optarg) {
 				parse_cpumask(optarg, max_cpus, &affinity_mask);
@@ -298,9 +296,6 @@ static void process_options(int argc, char *argv[], unsigned int max_cpus)
 
 	/* if smp wasn't requested, test for numa automatically */
 	if (!smp) {
-		if (numa_initialize())
-			fatal("Couldn't initialize libnuma");
-		numa = 1;
 		if (setaffinity == AFFINITY_UNSPECIFIED)
 			setaffinity = AFFINITY_USEALL;
 	}
@@ -354,6 +349,9 @@ int main(int argc, char **argv)
 	int status, cpu;
 	int max_cpus = sysconf(_SC_NPROCESSORS_ONLN);
 
+	if (numa_initialize())
+		fatal("Couldn't initialize libnuma");
+
 	process_options(argc, argv, max_cpus);
 
 	if (check_privs())
-- 
2.29.2

Re: [rt-tests v2 v2 07/20] signaltest: Always use libnuma

[John Kacur]

On Fri, 18 Dec 2020, Daniel Wagner wrote:

> libnuma is hard dependency for signaltest. Thus we can always call
> numa_initialize(). This allows us to remove the global 'numa' variable
> to track if libnuma has been initialized or not.
> 
> Signed-off-by: Daniel Wagner <dwagner@suse.de>
> ---
>  src/signaltest/signaltest.c | 10 ++++------
>  1 file changed, 4 insertions(+), 6 deletions(-)
> 
> diff --git a/src/signaltest/signaltest.c b/src/signaltest/signaltest.c
> index 918d2ab98f6e..b3a82f8c4f65 100644
> --- a/src/signaltest/signaltest.c
> +++ b/src/signaltest/signaltest.c
> @@ -206,7 +206,6 @@ static int quiet;
>  static int lockall;
>  static struct bitmask *affinity_mask = NULL;
>  static int smp = 0;
> -static int numa = 0;
>  static int setaffinity = AFFINITY_UNSPECIFIED;
>  
>  /* Process commandline options */
> @@ -214,6 +213,7 @@ static void process_options(int argc, char *argv[], unsigned int max_cpus)
>  {
>  	int option_affinity = 0;
>  	int error = 0;
> +	int numa = 0;
>  
>  	for (;;) {
>  		int option_index = 0;
> @@ -242,8 +242,6 @@ static void process_options(int argc, char *argv[], unsigned int max_cpus)
>  			/* smp sets AFFINITY_USEALL in OPT_SMP */
>  			if (smp)
>  				break;
> -			if (numa_initialize())
> -				fatal("Couldn't initialize libnuma");
>  			numa = 1;
>  			if (optarg) {
>  				parse_cpumask(optarg, max_cpus, &affinity_mask);
> @@ -298,9 +296,6 @@ static void process_options(int argc, char *argv[], unsigned int max_cpus)
>  
>  	/* if smp wasn't requested, test for numa automatically */
>  	if (!smp) {
> -		if (numa_initialize())
> -			fatal("Couldn't initialize libnuma");
> -		numa = 1;
>  		if (setaffinity == AFFINITY_UNSPECIFIED)
>  			setaffinity = AFFINITY_USEALL;
>  	}
> @@ -354,6 +349,9 @@ int main(int argc, char **argv)
>  	int status, cpu;
>  	int max_cpus = sysconf(_SC_NPROCESSORS_ONLN);
>  
> +	if (numa_initialize())
> +		fatal("Couldn't initialize libnuma");
> +
>  	process_options(argc, argv, max_cpus);
>  
>  	if (check_privs())
> -- 
> 2.29.2
> 
> 
Signed-off-by: John Kacur <jkacur@redhat.com>

[rt-tests v2 v2 08/20] signaltest: Use affinity_mask for stearing thread placement

[Daniel Wagner]

We don't need an extra variable to track the state if a bitmask is
available or not. Check directly if the mask is usable.

Reviewed-by: Daniel Wagner <dwagner@suse.de>
---
 src/signaltest/signaltest.c | 44 ++++++++-----------------------------
 1 file changed, 9 insertions(+), 35 deletions(-)

diff --git a/src/signaltest/signaltest.c b/src/signaltest/signaltest.c
index b3a82f8c4f65..e19fc9a740a9 100644
--- a/src/signaltest/signaltest.c
+++ b/src/signaltest/signaltest.c
@@ -205,15 +205,13 @@ static int verbose;
 static int quiet;
 static int lockall;
 static struct bitmask *affinity_mask = NULL;
-static int smp = 0;
-static int setaffinity = AFFINITY_UNSPECIFIED;
 
 /* Process commandline options */
 static void process_options(int argc, char *argv[], unsigned int max_cpus)
 {
-	int option_affinity = 0;
 	int error = 0;
 	int numa = 0;
+	int smp = 0;
 
 	for (;;) {
 		int option_index = 0;
@@ -238,26 +236,19 @@ static void process_options(int argc, char *argv[], unsigned int max_cpus)
 			break;
 		switch (c) {
 		case 'a':
-			option_affinity = 1;
 			/* smp sets AFFINITY_USEALL in OPT_SMP */
 			if (smp)
 				break;
 			numa = 1;
 			if (optarg) {
 				parse_cpumask(optarg, max_cpus, &affinity_mask);
-				setaffinity = AFFINITY_SPECIFIED;
 			} else if (optind < argc &&
 				   (atoi(argv[optind]) ||
 				    argv[optind][0] == '0' ||
 				    argv[optind][0] == '!')) {
 				parse_cpumask(argv[optind], max_cpus, &affinity_mask);
-				setaffinity = AFFINITY_SPECIFIED;
-			} else {
-				setaffinity = AFFINITY_USEALL;
 			}
 
-			if (setaffinity == AFFINITY_SPECIFIED && !affinity_mask)
-				display_help(1);
 			if (verbose)
 				printf("Using %u cpus.\n",
 					numa_bitmask_weight(affinity_mask));
@@ -275,7 +266,6 @@ static void process_options(int argc, char *argv[], unsigned int max_cpus)
 				fatal("numa and smp options are mutually exclusive\n");
 			smp = 1;
 			num_threads = -1; /* update after parsing */
-			setaffinity = AFFINITY_USEALL;
 			break;
 		case 't': num_threads = atoi(optarg); break;
 		case 'v': verbose = 1; break;
@@ -294,16 +284,8 @@ static void process_options(int argc, char *argv[], unsigned int max_cpus)
 	if (num_threads < 2)
 		error = 1;
 
-	/* if smp wasn't requested, test for numa automatically */
-	if (!smp) {
-		if (setaffinity == AFFINITY_UNSPECIFIED)
-			setaffinity = AFFINITY_USEALL;
-	}
-
-	if (option_affinity) {
-		if (smp)
-			warn("-a ignored due to smp mode\n");
-	}
+	if (smp && affinity_mask)
+		warn("-a ignored due to smp mode\n");
 
 	if (error) {
 		if (affinity_mask)
@@ -365,7 +347,7 @@ int main(int argc, char **argv)
 		}
 
 	/* Restrict the main pid to the affinity specified by the user */
-	if (affinity_mask != NULL) {
+	if (affinity_mask) {
 		int res;
 
 		errno = 0;
@@ -400,21 +382,13 @@ int main(int argc, char **argv)
 			par[i].bufmsk = VALBUF_SIZE - 1;
 		}
 
-		switch (setaffinity) {
-		case AFFINITY_UNSPECIFIED:
-			cpu = -1;
-			break;
-		case AFFINITY_SPECIFIED:
+		if (affinity_mask)
 			cpu = cpu_for_thread_sp(i, max_cpus, affinity_mask);
-			if (verbose)
-				printf("Thread %d using cpu %d.\n", i, cpu);
-			break;
-		case AFFINITY_USEALL:
+		else
 			cpu = cpu_for_thread_ua(i, max_cpus);
-			break;
-		default:
-			cpu = -1;
-		}
+
+		if (verbose)
+			printf("Thread %d using cpu %d.\n", i, cpu);
 
 		par[i].id = i;
 		par[i].prio = priority;
-- 
2.29.2

Re: [rt-tests v2 v2 08/20] signaltest: Use affinity_mask for stearing thread placement

[John Kacur]

On Fri, 18 Dec 2020, Daniel Wagner wrote:

> We don't need an extra variable to track the state if a bitmask is
> available or not. Check directly if the mask is usable.
> 
> Reviewed-by: Daniel Wagner <dwagner@suse.de>
> ---
>  src/signaltest/signaltest.c | 44 ++++++++-----------------------------
>  1 file changed, 9 insertions(+), 35 deletions(-)
> 
> diff --git a/src/signaltest/signaltest.c b/src/signaltest/signaltest.c
> index b3a82f8c4f65..e19fc9a740a9 100644
> --- a/src/signaltest/signaltest.c
> +++ b/src/signaltest/signaltest.c
> @@ -205,15 +205,13 @@ static int verbose;
>  static int quiet;
>  static int lockall;
>  static struct bitmask *affinity_mask = NULL;
> -static int smp = 0;
> -static int setaffinity = AFFINITY_UNSPECIFIED;
>  
>  /* Process commandline options */
>  static void process_options(int argc, char *argv[], unsigned int max_cpus)
>  {
> -	int option_affinity = 0;
>  	int error = 0;
>  	int numa = 0;
> +	int smp = 0;
>  
>  	for (;;) {
>  		int option_index = 0;
> @@ -238,26 +236,19 @@ static void process_options(int argc, char *argv[], unsigned int max_cpus)
>  			break;
>  		switch (c) {
>  		case 'a':
> -			option_affinity = 1;
>  			/* smp sets AFFINITY_USEALL in OPT_SMP */
>  			if (smp)
>  				break;
>  			numa = 1;
>  			if (optarg) {
>  				parse_cpumask(optarg, max_cpus, &affinity_mask);
> -				setaffinity = AFFINITY_SPECIFIED;
>  			} else if (optind < argc &&
>  				   (atoi(argv[optind]) ||
>  				    argv[optind][0] == '0' ||
>  				    argv[optind][0] == '!')) {
>  				parse_cpumask(argv[optind], max_cpus, &affinity_mask);
> -				setaffinity = AFFINITY_SPECIFIED;
> -			} else {
> -				setaffinity = AFFINITY_USEALL;
>  			}
>  
> -			if (setaffinity == AFFINITY_SPECIFIED && !affinity_mask)
> -				display_help(1);
>  			if (verbose)
>  				printf("Using %u cpus.\n",
>  					numa_bitmask_weight(affinity_mask));
> @@ -275,7 +266,6 @@ static void process_options(int argc, char *argv[], unsigned int max_cpus)
>  				fatal("numa and smp options are mutually exclusive\n");
>  			smp = 1;
>  			num_threads = -1; /* update after parsing */
> -			setaffinity = AFFINITY_USEALL;
>  			break;
>  		case 't': num_threads = atoi(optarg); break;
>  		case 'v': verbose = 1; break;
> @@ -294,16 +284,8 @@ static void process_options(int argc, char *argv[], unsigned int max_cpus)
>  	if (num_threads < 2)
>  		error = 1;
>  
> -	/* if smp wasn't requested, test for numa automatically */
> -	if (!smp) {
> -		if (setaffinity == AFFINITY_UNSPECIFIED)
> -			setaffinity = AFFINITY_USEALL;
> -	}
> -
> -	if (option_affinity) {
> -		if (smp)
> -			warn("-a ignored due to smp mode\n");
> -	}
> +	if (smp && affinity_mask)
> +		warn("-a ignored due to smp mode\n");
>  
>  	if (error) {
>  		if (affinity_mask)
> @@ -365,7 +347,7 @@ int main(int argc, char **argv)
>  		}
>  
>  	/* Restrict the main pid to the affinity specified by the user */
> -	if (affinity_mask != NULL) {
> +	if (affinity_mask) {
>  		int res;
>  
>  		errno = 0;
> @@ -400,21 +382,13 @@ int main(int argc, char **argv)
>  			par[i].bufmsk = VALBUF_SIZE - 1;
>  		}
>  
> -		switch (setaffinity) {
> -		case AFFINITY_UNSPECIFIED:
> -			cpu = -1;
> -			break;
> -		case AFFINITY_SPECIFIED:
> +		if (affinity_mask)
>  			cpu = cpu_for_thread_sp(i, max_cpus, affinity_mask);
> -			if (verbose)
> -				printf("Thread %d using cpu %d.\n", i, cpu);
> -			break;
> -		case AFFINITY_USEALL:
> +		else
>  			cpu = cpu_for_thread_ua(i, max_cpus);
> -			break;
> -		default:
> -			cpu = -1;
> -		}
> +
> +		if (verbose)
> +			printf("Thread %d using cpu %d.\n", i, cpu);
>  
>  		par[i].id = i;
>  		par[i].prio = priority;
> -- 
> 2.29.2
> 
> 

In git you wrote Reviewed by instead of Signed-off-by:
Why?

Re: [rt-tests v2 v2 08/20] signaltest: Use affinity_mask for stearing thread placement

[Daniel Wagner]

> In git you wrote Reviewed by instead of Signed-off-by:
> Why?

I am using shortcuts for those things and I used my wrong muscle
memory. Obviously it should be

Signed-off-by: Daniel Wagner <dwagner@suse.de>

[rt-tests v2 v2 09/20] signaltest: Simplify --smp vs --affinity vs --threads argument logic

[Daniel Wagner]

Allow each command line only to update one variable and do the final
decission at the end of the parsing step. With this the order
of the command line is not important.

Reviewed-by: Daniel Wagner <dwagner@suse.de>
---
 src/signaltest/signaltest.c | 23 +++++++----------------
 1 file changed, 7 insertions(+), 16 deletions(-)

diff --git a/src/signaltest/signaltest.c b/src/signaltest/signaltest.c
index e19fc9a740a9..61420fa13347 100644
--- a/src/signaltest/signaltest.c
+++ b/src/signaltest/signaltest.c
@@ -210,7 +210,6 @@ static struct bitmask *affinity_mask = NULL;
 static void process_options(int argc, char *argv[], unsigned int max_cpus)
 {
 	int error = 0;
-	int numa = 0;
 	int smp = 0;
 
 	for (;;) {
@@ -236,10 +235,6 @@ static void process_options(int argc, char *argv[], unsigned int max_cpus)
 			break;
 		switch (c) {
 		case 'a':
-			/* smp sets AFFINITY_USEALL in OPT_SMP */
-			if (smp)
-				break;
-			numa = 1;
 			if (optarg) {
 				parse_cpumask(optarg, max_cpus, &affinity_mask);
 			} else if (optind < argc &&
@@ -261,12 +256,7 @@ static void process_options(int argc, char *argv[], unsigned int max_cpus)
 		case 'm': lockall = 1; break;
 		case 'p': priority = atoi(optarg); break;
 		case 'q': quiet = 1; break;
-		case 'S':
-			if (numa)
-				fatal("numa and smp options are mutually exclusive\n");
-			smp = 1;
-			num_threads = -1; /* update after parsing */
-			break;
+		case 'S': smp = 1; break;
 		case 't': num_threads = atoi(optarg); break;
 		case 'v': verbose = 1; break;
 		}
@@ -278,15 +268,16 @@ static void process_options(int argc, char *argv[], unsigned int max_cpus)
 	if (priority < 0 || priority > 99)
 		error = 1;
 
-	if (num_threads == -1)
-		num_threads = get_available_cpus(affinity_mask);
+	if (smp) {
+		if (affinity_mask)
+			warn("--affinity overwrites smp mode\n");
+		else
+			num_threads = get_available_cpus(affinity_mask);
+	}
 
 	if (num_threads < 2)
 		error = 1;
 
-	if (smp && affinity_mask)
-		warn("-a ignored due to smp mode\n");
-
 	if (error) {
 		if (affinity_mask)
 			numa_bitmask_free(affinity_mask);
-- 
2.29.2

[rt-tests v2 v2 10/20] rt-numa: Remove unused definitions and numa_initialize()

[Daniel Wagner]

There is no user left of the AFFINITY_* enum. Remove them.

Also there is no need for numa_initialize() to prodect from being
called several times. We can safely initialize libnuma at the begin of
each program.

Reviewed-by: Daniel Wagner <dwagner@suse.de>
---
 src/cyclictest/cyclictest.c |  2 +-
 src/include/rt-numa.h       |  8 --------
 src/lib/rt-numa.c           | 15 ---------------
 src/signaltest/signaltest.c |  2 +-
 4 files changed, 2 insertions(+), 25 deletions(-)

diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
index 7e84c25f7070..a6364c8e04c0 100644
--- a/src/cyclictest/cyclictest.c
+++ b/src/cyclictest/cyclictest.c
@@ -1709,7 +1709,7 @@ int main(int argc, char **argv)
 	void *currstk;
 	size_t stksize;
 
-	if (numa_initialize())
+	if (numa_available() == -1)
 		fatal("Couldn't initialize libnuma");
 
 	process_options(argc, argv, max_cpus);
diff --git a/src/include/rt-numa.h b/src/include/rt-numa.h
index ca86a45dab3a..54f5c3a240e9 100644
--- a/src/include/rt-numa.h
+++ b/src/include/rt-numa.h
@@ -4,14 +4,6 @@
 
 #include <numa.h>
 
-enum {
-	AFFINITY_UNSPECIFIED,
-	AFFINITY_SPECIFIED,
-	AFFINITY_USEALL
-};
-
-int numa_initialize(void);
-
 int get_available_cpus(struct bitmask *cpumask);
 int cpu_for_thread_sp(int thread_num, int max_cpus, struct bitmask *cpumask);
 int cpu_for_thread_ua(int thread_num, int max_cpus);
diff --git a/src/lib/rt-numa.c b/src/lib/rt-numa.c
index ba888dc38cda..db5178f93043 100644
--- a/src/lib/rt-numa.c
+++ b/src/lib/rt-numa.c
@@ -13,21 +13,6 @@
 #include "error.h"
 #include "rt-numa.h"
 
-/* numa_available() must be called before any other calls to the numa library */
-int numa_initialize(void)
-{
-	static int is_initialized;
-
-	if (is_initialized == 1)
-		return 0;
-
-	if (numa_available() == -1)
-		return -1;
-
-	is_initialized = 1;
-	return 0;
-}
-
 int get_available_cpus(struct bitmask *cpumask)
 {
 	if (cpumask)
diff --git a/src/signaltest/signaltest.c b/src/signaltest/signaltest.c
index 61420fa13347..41ebb2c87a2f 100644
--- a/src/signaltest/signaltest.c
+++ b/src/signaltest/signaltest.c
@@ -322,7 +322,7 @@ int main(int argc, char **argv)
 	int status, cpu;
 	int max_cpus = sysconf(_SC_NPROCESSORS_ONLN);
 
-	if (numa_initialize())
+	if (numa_available() == -1)
 		fatal("Couldn't initialize libnuma");
 
 	process_options(argc, argv, max_cpus);
-- 
2.29.2

[rt-tests v2 v2 11/20] rt-numa: Add generic cpu_for_thread() helper

[Daniel Wagner]

Add an simpler wrapper which calls the right implementation depending
on if the cpumask is valid or not.

Reviewed-by: Daniel Wagner <dwagner@suse.de>
---
 src/cyclictest/cyclictest.c |  6 +-----
 src/include/rt-numa.h       |  3 +--
 src/lib/rt-numa.c           | 13 ++++++++++---
 src/signaltest/signaltest.c |  6 +-----
 4 files changed, 13 insertions(+), 15 deletions(-)

diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
index a6364c8e04c0..092534abb709 100644
--- a/src/cyclictest/cyclictest.c
+++ b/src/cyclictest/cyclictest.c
@@ -1884,11 +1884,7 @@ int main(int argc, char **argv)
 		if (status != 0)
 			fatal("error from pthread_attr_init for thread %d: %s\n", i, strerror(status));
 
-		if (affinity_mask)
-			cpu = cpu_for_thread_sp(i, max_cpus, affinity_mask);
-		else
-			cpu = cpu_for_thread_ua(i, max_cpus);
-
+		cpu = cpu_for_thread(i, max_cpus, affinity_mask);
 		if (verbose)
 			printf("Thread %d using cpu %d.\n", i, cpu);
 
diff --git a/src/include/rt-numa.h b/src/include/rt-numa.h
index 54f5c3a240e9..189da3a804e1 100644
--- a/src/include/rt-numa.h
+++ b/src/include/rt-numa.h
@@ -5,8 +5,7 @@
 #include <numa.h>
 
 int get_available_cpus(struct bitmask *cpumask);
-int cpu_for_thread_sp(int thread_num, int max_cpus, struct bitmask *cpumask);
-int cpu_for_thread_ua(int thread_num, int max_cpus);
+int cpu_for_thread(int thread_num, int max_cpus, struct bitmask *cpumask);
 
 int parse_cpumask(char *str, int max_cpus, struct bitmask **cpumask);
 
diff --git a/src/lib/rt-numa.c b/src/lib/rt-numa.c
index db5178f93043..a246dbca7291 100644
--- a/src/lib/rt-numa.c
+++ b/src/lib/rt-numa.c
@@ -21,7 +21,7 @@ int get_available_cpus(struct bitmask *cpumask)
 	return numa_num_task_cpus();
 }
 
-int cpu_for_thread_sp(int thread_num, int max_cpus, struct bitmask *cpumask)
+static int cpu_for_thread_sp(int thread_num, int max_cpus, struct bitmask *cpumask)
 {
 	unsigned int m, cpu, i, num_cpus;
 
@@ -44,8 +44,7 @@ int cpu_for_thread_sp(int thread_num, int max_cpus, struct bitmask *cpumask)
 	return 0;
 }
 
-/* cpu_for_thread AFFINITY_USEALL */
-int cpu_for_thread_ua(int thread_num, int max_cpus)
+static int cpu_for_thread_ua(int thread_num, int max_cpus)
 {
 	int res, num_cpus, i, m, cpu;
 	pthread_t thread;
@@ -73,6 +72,14 @@ int cpu_for_thread_ua(int thread_num, int max_cpus)
 	return 0;
 }
 
+int cpu_for_thread(int thread_num, int max_cpus, struct bitmask *cpumask)
+{
+	if (cpumask)
+		return cpu_for_thread_sp(thread_num, max_cpus, cpumask);
+	else
+		return cpu_for_thread_ua(thread_num, max_cpus);
+}
+
 /*
  * After this function is called, affinity_mask is the intersection of
  * the user supplied affinity mask and the affinity mask from the run
diff --git a/src/signaltest/signaltest.c b/src/signaltest/signaltest.c
index 41ebb2c87a2f..2ea5e6b58946 100644
--- a/src/signaltest/signaltest.c
+++ b/src/signaltest/signaltest.c
@@ -373,11 +373,7 @@ int main(int argc, char **argv)
 			par[i].bufmsk = VALBUF_SIZE - 1;
 		}
 
-		if (affinity_mask)
-			cpu = cpu_for_thread_sp(i, max_cpus, affinity_mask);
-		else
-			cpu = cpu_for_thread_ua(i, max_cpus);
-
+		cpu = cpu_for_thread(i, max_cpus, affinity_mask);
 		if (verbose)
 			printf("Thread %d using cpu %d.\n", i, cpu);
 
-- 
2.29.2

[rt-tests v2 v2 12/20] rt-numa: Use mask size for iterator limit

[Daniel Wagner]

The bitmask structure knows its size use, thus use it as upper limit
in the loop.

Signed-off-by: Daniel Wagner <dwagner@suse.de>
---
 src/lib/rt-numa.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/lib/rt-numa.c b/src/lib/rt-numa.c
index a246dbca7291..1918f93b74eb 100644
--- a/src/lib/rt-numa.c
+++ b/src/lib/rt-numa.c
@@ -33,7 +33,7 @@ static int cpu_for_thread_sp(int thread_num, int max_cpus, struct bitmask *cpuma
 	m = thread_num % num_cpus;
 
 	/* there are num_cpus bits set, we want position of m'th one */
-	for (i = 0, cpu = 0; i < max_cpus; i++) {
+	for (i = 0, cpu = 0; i < cpumask->size; i++) {
 		if (numa_bitmask_isbitset(cpumask, i)) {
 			if (cpu == m)
 				return i;
@@ -97,7 +97,7 @@ static void use_current_cpuset(int max_cpus, struct bitmask *cpumask)
 	 * Clear bits that are not set in both the cpuset from the
 	 * environment, and in the user specified affinity.
 	 */
-	for (i = 0; i < max_cpus; i++) {
+	for (i = 0; i < cpumask->size; i++) {
 		if ((!numa_bitmask_isbitset(cpumask, i)) ||
 		    (!numa_bitmask_isbitset(curmask, i)))
 			numa_bitmask_clearbit(cpumask, i);
-- 
2.29.2

[rt-tests v2 v2 13/20] rt-numa: Remove max_cpus argument from parse_cpusmask

[Daniel Wagner]

Remove usused max_cpus argument from parse_cpumask().

Reviewed-by: Daniel Wagner <dwagner@suse.de>
---
 src/cyclictest/cyclictest.c | 4 ++--
 src/include/rt-numa.h       | 2 +-
 src/lib/rt-numa.c           | 6 +++---
 src/oslat/oslat.c           | 3 +--
 src/signaltest/signaltest.c | 4 ++--
 5 files changed, 9 insertions(+), 10 deletions(-)

diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
index 092534abb709..9e138c812fc8 100644
--- a/src/cyclictest/cyclictest.c
+++ b/src/cyclictest/cyclictest.c
@@ -1013,12 +1013,12 @@ static void process_options(int argc, char *argv[], int max_cpus)
 		case 'a':
 		case OPT_AFFINITY:
 			if (optarg) {
-				parse_cpumask(optarg, max_cpus, &affinity_mask);
+				parse_cpumask(optarg, &affinity_mask);
 			} else if (optind < argc &&
 				   (atoi(argv[optind]) ||
 				    argv[optind][0] == '0' ||
 				    argv[optind][0] == '!')) {
-				parse_cpumask(argv[optind], max_cpus, &affinity_mask);
+				parse_cpumask(argv[optind], &affinity_mask);
 			}
 			break;
 		case 'A':
diff --git a/src/include/rt-numa.h b/src/include/rt-numa.h
index 189da3a804e1..446ce54a6ba2 100644
--- a/src/include/rt-numa.h
+++ b/src/include/rt-numa.h
@@ -7,6 +7,6 @@
 int get_available_cpus(struct bitmask *cpumask);
 int cpu_for_thread(int thread_num, int max_cpus, struct bitmask *cpumask);
 
-int parse_cpumask(char *str, int max_cpus, struct bitmask **cpumask);
+int parse_cpumask(char *str, struct bitmask **cpumask);
 
 #endif
diff --git a/src/lib/rt-numa.c b/src/lib/rt-numa.c
index 1918f93b74eb..fc7adb4d467a 100644
--- a/src/lib/rt-numa.c
+++ b/src/lib/rt-numa.c
@@ -85,7 +85,7 @@ int cpu_for_thread(int thread_num, int max_cpus, struct bitmask *cpumask)
  * the user supplied affinity mask and the affinity mask from the run
  * time environment
  */
-static void use_current_cpuset(int max_cpus, struct bitmask *cpumask)
+static void use_current_cpuset(struct bitmask *cpumask)
 {
 	struct bitmask *curmask;
 	int i;
@@ -106,7 +106,7 @@ static void use_current_cpuset(int max_cpus, struct bitmask *cpumask)
 	numa_bitmask_free(curmask);
 }
 
-int parse_cpumask(char *str, int max_cpus, struct bitmask **cpumask)
+int parse_cpumask(char *str, struct bitmask **cpumask)
 {
 	struct bitmask *mask;
 
@@ -120,7 +120,7 @@ int parse_cpumask(char *str, int max_cpus, struct bitmask **cpumask)
 		return 0;
 	}
 
-	use_current_cpuset(max_cpus, mask);
+	use_current_cpuset(mask);
 	*cpumask = mask;
 
 	return 0;
diff --git a/src/oslat/oslat.c b/src/oslat/oslat.c
index 5b7e0d5b5d5c..0811079d9f04 100644
--- a/src/oslat/oslat.c
+++ b/src/oslat/oslat.c
@@ -716,7 +716,6 @@ int main(int argc, char *argv[])
 	struct thread *threads;
 	int i, n_cores;
 	struct bitmask *cpu_set = NULL;
-	int max_cpus = sysconf(_SC_NPROCESSORS_ONLN);
 
 #ifdef FRC_MISSING
 	printf("This architecture is not yet supported. "
@@ -743,7 +742,7 @@ int main(int argc, char *argv[])
 
 	if (!g.cpu_list)
 		g.cpu_list = strdup("all");
-	if (parse_cpumask(g.cpu_list, max_cpus, &cpu_set))
+	if (parse_cpumask(g.cpu_list, &cpu_set))
 		exit(1);
 	n_cores = numa_bitmask_weight(cpu_set);
 
diff --git a/src/signaltest/signaltest.c b/src/signaltest/signaltest.c
index 2ea5e6b58946..0d189483753d 100644
--- a/src/signaltest/signaltest.c
+++ b/src/signaltest/signaltest.c
@@ -236,12 +236,12 @@ static void process_options(int argc, char *argv[], unsigned int max_cpus)
 		switch (c) {
 		case 'a':
 			if (optarg) {
-				parse_cpumask(optarg, max_cpus, &affinity_mask);
+				parse_cpumask(optarg, &affinity_mask);
 			} else if (optind < argc &&
 				   (atoi(argv[optind]) ||
 				    argv[optind][0] == '0' ||
 				    argv[optind][0] == '!')) {
-				parse_cpumask(argv[optind], max_cpus, &affinity_mask);
+				parse_cpumask(argv[optind], &affinity_mask);
 			}
 
 			if (verbose)
-- 
2.29.2

[rt-tests v2 v2 14/20] rt-numa: Use error message helpers

[Daniel Wagner]

As we have our small error message helpers use them.

Signed-off-by: Daniel Wagner <dwagner@suse.de>
---
 src/lib/rt-numa.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/lib/rt-numa.c b/src/lib/rt-numa.c
index fc7adb4d467a..3a8441d5151c 100644
--- a/src/lib/rt-numa.c
+++ b/src/lib/rt-numa.c
@@ -40,7 +40,7 @@ static int cpu_for_thread_sp(int thread_num, int max_cpus, struct bitmask *cpuma
 			cpu++;
 		}
 	}
-	fprintf(stderr, "Bug in cpu mask handling code.\n");
+	warn("Bug in cpu mask handling code.\n");
 	return 0;
 }
 
@@ -68,7 +68,7 @@ static int cpu_for_thread_ua(int thread_num, int max_cpus)
 		}
 	}
 
-	fprintf(stderr, "Bug in cpu mask handling code.\n");
+	warn("Bug in cpu mask handling code.\n");
 	return 0;
 }
 
-- 
2.29.2

Re: [rt-tests v2 v2 14/20] rt-numa: Use error message helpers

[John Kacur]

On Fri, 18 Dec 2020, Daniel Wagner wrote:

> As we have our small error message helpers use them.
> 
> Signed-off-by: Daniel Wagner <dwagner@suse.de>
> ---
>  src/lib/rt-numa.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/src/lib/rt-numa.c b/src/lib/rt-numa.c
> index fc7adb4d467a..3a8441d5151c 100644
> --- a/src/lib/rt-numa.c
> +++ b/src/lib/rt-numa.c
> @@ -40,7 +40,7 @@ static int cpu_for_thread_sp(int thread_num, int max_cpus, struct bitmask *cpuma
>  			cpu++;
>  		}
>  	}
> -	fprintf(stderr, "Bug in cpu mask handling code.\n");
> +	warn("Bug in cpu mask handling code.\n");
>  	return 0;
>  }
>  
> @@ -68,7 +68,7 @@ static int cpu_for_thread_ua(int thread_num, int max_cpus)
>  		}
>  	}
>  
> -	fprintf(stderr, "Bug in cpu mask handling code.\n");
> +	warn("Bug in cpu mask handling code.\n");
>  	return 0;
>  }
>  
> -- 
> 2.29.2
> 
> 
Signed-off-by: John Kacur <jkacur@redhat.com>

[rt-tests v2 v2 15/20] signaltest: Remove unused max_cpus argument from process_options

[Daniel Wagner]

There is no use of this argument. Remove it.

Reviewed-by: Daniel Wagner <dwagner@suse.de>
---
 src/signaltest/signaltest.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/signaltest/signaltest.c b/src/signaltest/signaltest.c
index 0d189483753d..3ca26fb373bb 100644
--- a/src/signaltest/signaltest.c
+++ b/src/signaltest/signaltest.c
@@ -207,7 +207,7 @@ static int lockall;
 static struct bitmask *affinity_mask = NULL;
 
 /* Process commandline options */
-static void process_options(int argc, char *argv[], unsigned int max_cpus)
+static void process_options(int argc, char *argv[])
 {
 	int error = 0;
 	int smp = 0;
@@ -325,7 +325,7 @@ int main(int argc, char **argv)
 	if (numa_available() == -1)
 		fatal("Couldn't initialize libnuma");
 
-	process_options(argc, argv, max_cpus);
+	process_options(argc, argv);
 
 	if (check_privs())
 		exit(1);
-- 
2.29.2

[rt-tests v2 v2 16/20] cyclictest: Remove unused max_cpus argument from process_options

[Daniel Wagner]

There is no use of this argument. Remove it.

Reviewed-by: Daniel Wagner <dwagner@suse.de>
---
 src/cyclictest/cyclictest.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
index 9e138c812fc8..b7de32cf58d0 100644
--- a/src/cyclictest/cyclictest.c
+++ b/src/cyclictest/cyclictest.c
@@ -954,7 +954,7 @@ enum option_values {
 };
 
 /* Process commandline options */
-static void process_options(int argc, char *argv[], int max_cpus)
+static void process_options(int argc, char *argv[])
 {
 	int error = 0;
 	int smp = 0;
@@ -1712,7 +1712,7 @@ int main(int argc, char **argv)
 	if (numa_available() == -1)
 		fatal("Couldn't initialize libnuma");
 
-	process_options(argc, argv, max_cpus);
+	process_options(argc, argv);
 
 	if (check_privs())
 		exit(EXIT_FAILURE);
-- 
2.29.2

[rt-tests v2 v2 17/20] rt-numa: Use CPU_SETSIZE as upper loop limit

[Daniel Wagner]

The cpeset might not be dense, so user the CPU_SETSIZE as upper limit.

Reviewed-by: Daniel Wagner <dwagner@suse.de>
---
 src/lib/rt-numa.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/lib/rt-numa.c b/src/lib/rt-numa.c
index 3a8441d5151c..45d4f1193d5f 100644
--- a/src/lib/rt-numa.c
+++ b/src/lib/rt-numa.c
@@ -60,7 +60,7 @@ static int cpu_for_thread_ua(int thread_num, int max_cpus)
 	num_cpus = CPU_COUNT(&cpuset);
 	m = thread_num % num_cpus;
 
-	for (i = 0, cpu = 0; i < max_cpus; i++) {
+	for (i = 0, cpu = 0; i < CPU_SETSIZE; i++) {
 		if (CPU_ISSET(i, &cpuset)) {
 			if (cpu == m)
 				return i;
-- 
2.29.2

[rt-tests v2 v2 18/20] rt-numa: Remove used max_cpus argument from cpu_for_thread()

[Daniel Wagner]

There is no need to pass in the max_cpus anymore. Thus remove the
argument.

Reviewed-by: Daniel Wagner <dwagner@suse.de>
---
 src/cyclictest/cyclictest.c |  2 +-
 src/include/rt-numa.h       |  2 +-
 src/lib/rt-numa.c           | 10 +++++-----
 src/signaltest/signaltest.c |  3 +--
 4 files changed, 8 insertions(+), 9 deletions(-)

diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
index b7de32cf58d0..5d1c08e8b8e0 100644
--- a/src/cyclictest/cyclictest.c
+++ b/src/cyclictest/cyclictest.c
@@ -1884,7 +1884,7 @@ int main(int argc, char **argv)
 		if (status != 0)
 			fatal("error from pthread_attr_init for thread %d: %s\n", i, strerror(status));
 
-		cpu = cpu_for_thread(i, max_cpus, affinity_mask);
+		cpu = cpu_for_thread(i, affinity_mask);
 		if (verbose)
 			printf("Thread %d using cpu %d.\n", i, cpu);
 
diff --git a/src/include/rt-numa.h b/src/include/rt-numa.h
index 446ce54a6ba2..405e57869735 100644
--- a/src/include/rt-numa.h
+++ b/src/include/rt-numa.h
@@ -5,7 +5,7 @@
 #include <numa.h>
 
 int get_available_cpus(struct bitmask *cpumask);
-int cpu_for_thread(int thread_num, int max_cpus, struct bitmask *cpumask);
+int cpu_for_thread(int thread_num, struct bitmask *cpumask);
 
 int parse_cpumask(char *str, struct bitmask **cpumask);
 
diff --git a/src/lib/rt-numa.c b/src/lib/rt-numa.c
index 45d4f1193d5f..04f2e9adb4b1 100644
--- a/src/lib/rt-numa.c
+++ b/src/lib/rt-numa.c
@@ -21,7 +21,7 @@ int get_available_cpus(struct bitmask *cpumask)
 	return numa_num_task_cpus();
 }
 
-static int cpu_for_thread_sp(int thread_num, int max_cpus, struct bitmask *cpumask)
+static int cpu_for_thread_sp(int thread_num, struct bitmask *cpumask)
 {
 	unsigned int m, cpu, i, num_cpus;
 
@@ -44,7 +44,7 @@ static int cpu_for_thread_sp(int thread_num, int max_cpus, struct bitmask *cpuma
 	return 0;
 }
 
-static int cpu_for_thread_ua(int thread_num, int max_cpus)
+static int cpu_for_thread_ua(int thread_num)
 {
 	int res, num_cpus, i, m, cpu;
 	pthread_t thread;
@@ -72,12 +72,12 @@ static int cpu_for_thread_ua(int thread_num, int max_cpus)
 	return 0;
 }
 
-int cpu_for_thread(int thread_num, int max_cpus, struct bitmask *cpumask)
+int cpu_for_thread(int thread_num, struct bitmask *cpumask)
 {
 	if (cpumask)
-		return cpu_for_thread_sp(thread_num, max_cpus, cpumask);
+		return cpu_for_thread_sp(thread_num, cpumask);
 	else
-		return cpu_for_thread_ua(thread_num, max_cpus);
+		return cpu_for_thread_ua(thread_num);
 }
 
 /*
diff --git a/src/signaltest/signaltest.c b/src/signaltest/signaltest.c
index 3ca26fb373bb..5427db7f8d85 100644
--- a/src/signaltest/signaltest.c
+++ b/src/signaltest/signaltest.c
@@ -320,7 +320,6 @@ int main(int argc, char **argv)
 	struct thread_stat *stat;
 	int i, ret = -1;
 	int status, cpu;
-	int max_cpus = sysconf(_SC_NPROCESSORS_ONLN);
 
 	if (numa_available() == -1)
 		fatal("Couldn't initialize libnuma");
@@ -373,7 +372,7 @@ int main(int argc, char **argv)
 			par[i].bufmsk = VALBUF_SIZE - 1;
 		}
 
-		cpu = cpu_for_thread(i, max_cpus, affinity_mask);
+		cpu = cpu_for_thread(i, affinity_mask);
 		if (verbose)
 			printf("Thread %d using cpu %d.\n", i, cpu);
 
-- 
2.29.2

[rt-tests v2 v2 19/20] cyclictest: Remove max cpus used verbose information

[Daniel Wagner]

Since we always print the thread placement in verbose mode now, there
is no need to print the max_cpus anymore. With the last of max_cpus
gone, we can also remove the sysconf() call.

Reviewed-by: Daniel Wagner <dwagner@suse.de>
---
 src/cyclictest/cyclictest.c | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
index 5d1c08e8b8e0..35c57c782f30 100644
--- a/src/cyclictest/cyclictest.c
+++ b/src/cyclictest/cyclictest.c
@@ -1702,7 +1702,6 @@ int main(int argc, char **argv)
 	int signum = SIGALRM;
 	int mode;
 	int cpu;
-	int max_cpus = sysconf(_SC_NPROCESSORS_ONLN);
 	int i, ret = -1;
 	int status;
 	void *stack;
@@ -1717,9 +1716,6 @@ int main(int argc, char **argv)
 	if (check_privs())
 		exit(EXIT_FAILURE);
 
-	if (verbose)
-		printf("Max CPUs = %d\n", max_cpus);
-
 	/* Restrict the main pid to the affinity specified by the user */
 	if (affinity_mask) {
 		int res;
@@ -1728,10 +1724,6 @@ int main(int argc, char **argv)
 		res = numa_sched_setaffinity(getpid(), affinity_mask);
 		if (res != 0)
 			warn("Couldn't setaffinity in main thread: %s\n", strerror(errno));
-
-		if (verbose)
-			printf("Using %u cpus.\n",
-				numa_bitmask_weight(affinity_mask));
 	}
 
 	if (trigger) {
-- 
2.29.2

[rt-tests v2 v2 20/20] cyclictest: Remove unecessary local variable

[Daniel Wagner]

The return value is not used, thus we can avoid the unussed local
variable.

Reviewed-by: Daniel Wagner <dwagner@suse.de>
---
 src/cyclictest/cyclictest.c | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)

diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
index 35c57c782f30..5c738005f778 100644
--- a/src/cyclictest/cyclictest.c
+++ b/src/cyclictest/cyclictest.c
@@ -1718,18 +1718,13 @@ int main(int argc, char **argv)
 
 	/* Restrict the main pid to the affinity specified by the user */
 	if (affinity_mask) {
-		int res;
-
 		errno = 0;
-		res = numa_sched_setaffinity(getpid(), affinity_mask);
-		if (res != 0)
+		if (numa_sched_setaffinity(getpid(), affinity_mask) != 0)
 			warn("Couldn't setaffinity in main thread: %s\n", strerror(errno));
 	}
 
 	if (trigger) {
-		int retval;
-		retval = trigger_init();
-		if (retval != 0) {
+		if (trigger_init() != 0) {
 			fprintf(stderr, "trigger_init() failed\n");
 			exit(EXIT_FAILURE);
 		}
-- 
2.29.2

Re: [rt-tests v2 v2 00/20] rt-numa.h cleanups

[Daniel Wagner]

On Fri, Dec 18, 2020 at 05:18:23PM +0100, Daniel Wagner wrote:
> This series cleans up the rt-numa.h header.

ping

Re: [rt-tests v2 v2 00/20] rt-numa.h cleanups

[John Kacur]

On Fri, 18 Dec 2020, Daniel Wagner wrote:

> This series cleans up the rt-numa.h header.
> 
> As we have a hard dependency on libnuma we can simplify the code in
> cyclictest. This allows remove all the small helpers in rt_numa.h. And
> with this we can remove the header and reduce the confusion with
> rt-numa.h

I'm not entirely convinced. The code has been moving in this direction,
but I don't know what motivation there is for libnuma people to provide 
libnuma on architectures without numa. This is what the code used to do, 
it wrapped these calls in a way that we could call them whether numa was 
available or not. On Fedora there is no libnuma (numactl) for arm 
processors currently afaik as an example. But, there are a lot of good 
changes here and I  want the code to go forward, so there is no point 
holding on to functionality that we largely got rid of, but if in the
future we have arches that won't work on some distributions,
we may be forced to reimpliment the numa wrappers.

> 
> While at it, I simplified the --smp vs --affinity vs --threads
> logic. There is no need for additional variables to keep state. With
> this we also make --affinity to behave as with the rest of
> rt-tests. That is a plan -a will be the same as with -S. There is no
> need for -S anymore but I think we should leave it in place for
> backwards compatibility. I suspect, there must be a lot of muscle
> memory out there :)

This is really good. Note, the first step that got us here, was the code
that just detected whether numa was available and used it automatically
if it was, unless smp was specified. Believe it or not though, some of
the motivation for that was to reduce the alphabet soup of options on
cyclictest. If -S is no longer needed, it should be removed, people can
learn new muscle memory. (I'm typing this on a Canadian (English layout)
keyboard after using a German keyboard for 13+ years.)


> 
> Since signaltest has the same code as cyclictest, cleanup this tool as
> well.
> 
> With all those cleanups in cyclictest and signaltest, rt-numa.h
> contains only a three functions. The final part of the series is to
> remove unnecessary function arguments (max_cpus). This also fixes the
> theoretical problem that a sparse cpumask would not be handled
> correctly.
> 
> changes v2:
>   - added more cleanup patches (7-20)
> 
> Daniel Wagner (20):
>   cyclictest: Always use libnuma
>   cyclictest: Use numa API directly
>   cyclictest: Use affinity_mask for stearing thread placement
>   cyclictest: Mimik --smp behavior with --affinity
>   cyclictest: Simplify --smp vs --affinity vs --threads argument logic
>   cyclictest: Move verbose message into main
>   signaltest: Always use libnuma
>   signaltest: Use affinity_mask for stearing thread placement
>   signaltest:  Simplify --smp vs --affinity vs --threads argument logic
>   rt-numa: Remove unused definitions and numa_initialize()
>   rt-numa: Add generic cpu_for_thread() helper
>   rt-numa: Use mask size for iterator limit
>   rt-numa: Remove max_cpus argument from parse_cpusmask
>   rt-numa: Use error message helpers
>   signaltest: Remove unused max_cpus argument from process_options
>   cyclictest: Remove unused max_cpus argument from process_options
>   rt-numa: Use CPU_SETSIZE as upper loop limit
>   rt-numa: Remove used max_cpus argument from cpu_for_thread()
>   cyclictest: Remove max cpus used verbose information
>   cyclictest: Remove unecessary local variable
> 
>  src/cyclictest/cyclictest.c | 177 +++++++++++++-----------------------
>  src/cyclictest/rt_numa.h    |  98 --------------------
>  src/include/rt-numa.h       |  13 +--
>  src/lib/rt-numa.c           |  44 ++++-----
>  src/oslat/oslat.c           |   3 +-
>  src/signaltest/signaltest.c |  80 ++++------------
>  6 files changed, 101 insertions(+), 314 deletions(-)
>  delete mode 100644 src/cyclictest/rt_numa.h
> 
> -- 
> 2.29.2
> 
>