[PATCH v3 0/2] rt-tests: cyclictest: Add option to specify main pid affinity

[PATCH v3 0/2] rt-tests: cyclictest: Add option to specify main pid affinity

[Jonathan Schwender]

Hi John,

Changes in v3:
Rebased onto latest commit 95066b1a1a0a ("cyclicdeadline.c: Fix printf format specifier")
in unstable/devel/latest.


This patch adds the option --mainaffinity to specify the affinity of
the main pid.
This is mainly useful if you want to bind the main thread to a
different (e.g. housekeeping ) CPU than the measurement threads.

Some numbers on potential latency benefits are linked in the
v1 thread at https://lore.kernel.org/linux-rt-users/dd40b81d-7099-7740-c2ad-64b49e582234@gmail.com/

Regards

Jonathan


Jonathan Schwender (2):
  cyclictest: Move main pid setaffinity handling into a function
  cyclictest: Add --mainaffinity=[CPUSET] option.

 src/cyclictest/cyclictest.c | 28 +++++++++++++++++++---------
 src/include/rt-numa.h       |  2 ++
 src/lib/rt-numa.c           | 11 +++++++++++
 3 files changed, 32 insertions(+), 9 deletions(-)

-- 
2.30.2

[PATCH v3 1/2] cyclictest: Move main pid setaffinity handling into a function

[Jonathan Schwender]

Move error handling for setting the affinity of the main pid
into a separate function.
This prevents duplicating the code in the next commit,
where the main thread pid can be restricted to one of
two bitmasks depending on the passed parameters.

After feedback from John Kacur, the function is now
located in src/lib/rt-numa.c.
This allows other tests to reuse this, if they also
prefer to warn if numa_sched_setaffinity fails.

Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
---
 src/cyclictest/cyclictest.c |  8 +-------
 src/include/rt-numa.h       |  2 ++
 src/lib/rt-numa.c           | 11 +++++++++++
 3 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
index dca9610..460f6ae 100644
--- a/src/cyclictest/cyclictest.c
+++ b/src/cyclictest/cyclictest.c
@@ -1791,13 +1791,7 @@ int main(int argc, char **argv)
 
 	/* Restrict the main pid to the affinity specified by the user */
 	if (affinity_mask != NULL) {
-		int res;
-
-		errno = 0;
-		res = numa_sched_setaffinity(getpid(), affinity_mask);
-		if (res != 0)
-			warn("Couldn't setaffinity in main thread: %s\n", strerror(errno));
-
+		try_numa_sched_setaffinity(getpid(), affinity_mask);
 		if (verbose)
 			printf("Using %u cpus.\n",
 				numa_bitmask_weight(affinity_mask));
diff --git a/src/include/rt-numa.h b/src/include/rt-numa.h
index ca86a45..e9fa312 100644
--- a/src/include/rt-numa.h
+++ b/src/include/rt-numa.h
@@ -18,4 +18,6 @@ int cpu_for_thread_ua(int thread_num, int max_cpus);
 
 int parse_cpumask(char *str, int max_cpus, struct bitmask **cpumask);
 
+void try_numa_sched_setaffinity(__pid_t pid, struct bitmask *cpumask);
+
 #endif
diff --git a/src/lib/rt-numa.c b/src/lib/rt-numa.c
index babcc63..9a81cfb 100644
--- a/src/lib/rt-numa.c
+++ b/src/lib/rt-numa.c
@@ -138,3 +138,14 @@ int parse_cpumask(char *str, int max_cpus, struct bitmask **cpumask)
 
 	return 0;
 }
+
+void try_numa_sched_setaffinity(pid_t pid, struct bitmask *cpumask)
+{
+	int res;
+
+	errno = 0;
+	res = numa_sched_setaffinity(pid, cpumask);
+	if (res != 0)
+		warn("Couldn't setaffinity for thread %d: %s\n", pid,
+		     strerror(errno));
+}
-- 
2.30.2

[PATCH v3 2/2] cyclictest: Add --mainaffinity=[CPUSET] option.

[Jonathan Schwender]

This allows the user to specify a separate cpuset for the main pid,
e.g. on a housekeeping CPU.
If --mainaffinity is not specified, but --affinity is, then the
current behaviour is preserved and the main pid is bound
to the cpuset specified by --affinity

Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
---
 src/cyclictest/cyclictest.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
index 460f6ae..b7ca8de 100644
--- a/src/cyclictest/cyclictest.c
+++ b/src/cyclictest/cyclictest.c
@@ -836,6 +836,8 @@ static void display_help(int error)
 	       "	 --laptop	   Save battery when running cyclictest\n"
 	       "			   This will give you poorer realtime results\n"
 	       "			   but will not drain your battery so quickly\n"
+	       "         --mainaffinity=[CPUSET]  Run the main thread on CPU #N. This only affects\n"
+	       "                           the main thread and not the measurement threads\n"
 	       "-m       --mlockall        lock current and future memory allocations\n"
 	       "-M       --refresh_on_max  delay updating the screen until a new max\n"
 	       "			   latency is hit. Useful for low bandwidth.\n"
@@ -891,6 +893,7 @@ static int quiet;
 static int interval = DEFAULT_INTERVAL;
 static int distance = -1;
 static struct bitmask *affinity_mask = NULL;
+static struct bitmask *main_affinity_mask = NULL;
 static int smp = 0;
 static int setaffinity = AFFINITY_UNSPECIFIED;
 
@@ -944,7 +947,7 @@ enum option_values {
 	OPT_AFFINITY=1, OPT_BREAKTRACE, OPT_CLOCK,
 	OPT_DISTANCE, OPT_DURATION, OPT_LATENCY,
 	OPT_FIFO, OPT_HISTOGRAM, OPT_HISTOFALL, OPT_HISTFILE,
-	OPT_INTERVAL, OPT_LOOPS, OPT_MLOCKALL, OPT_REFRESH,
+	OPT_INTERVAL, OPT_LOOPS, OPT_MAINAFFINITY, OPT_MLOCKALL, OPT_REFRESH,
 	OPT_NANOSLEEP, OPT_NSECS, OPT_OSCOPE, OPT_PRIORITY,
 	OPT_QUIET, OPT_PRIOSPREAD, OPT_RELATIVE, OPT_RESOLUTION,
 	OPT_SYSTEM, OPT_SMP, OPT_THREADS, OPT_TRIGGER,
@@ -981,6 +984,7 @@ static void process_options(int argc, char *argv[], int max_cpus)
 			{"interval",         required_argument, NULL, OPT_INTERVAL },
 			{"laptop",	     no_argument,	NULL, OPT_LAPTOP },
 			{"loops",            required_argument, NULL, OPT_LOOPS },
+			{"mainaffinity",     required_argument, NULL, OPT_MAINAFFINITY},
 			{"mlockall",         no_argument,       NULL, OPT_MLOCKALL },
 			{"refresh_on_max",   no_argument,       NULL, OPT_REFRESH },
 			{"nsecs",            no_argument,       NULL, OPT_NSECS },
@@ -1083,6 +1087,16 @@ static void process_options(int argc, char *argv[], int max_cpus)
 		case 'l':
 		case OPT_LOOPS:
 			max_cycles = atoi(optarg); break;
+		case OPT_MAINAFFINITY:
+			if (optarg) {
+				parse_cpumask(optarg, max_cpus, &main_affinity_mask);
+			} else if (optind < argc &&
+			           (atoi(argv[optind]) ||
+			            argv[optind][0] == '0' ||
+			            argv[optind][0] == '!')) {
+				parse_cpumask(argv[optind], max_cpus, &main_affinity_mask);
+			}
+			break;
 		case 'm':
 		case OPT_MLOCKALL:
 			lockall = 1; break;
@@ -1790,7 +1804,9 @@ int main(int argc, char **argv)
 	}
 
 	/* Restrict the main pid to the affinity specified by the user */
-	if (affinity_mask != NULL) {
+	if (main_affinity_mask != NULL) {
+		try_numa_sched_setaffinity(getpid(), main_affinity_mask);
+	} else if (affinity_mask != NULL) {
 		try_numa_sched_setaffinity(getpid(), affinity_mask);
 		if (verbose)
 			printf("Using %u cpus.\n",
-- 
2.30.2

Re: [PATCH v3 1/2] cyclictest: Move main pid setaffinity handling into a function

[John Kacur]

On Thu, 25 Mar 2021, Jonathan Schwender wrote:

> Move error handling for setting the affinity of the main pid
> into a separate function.
> This prevents duplicating the code in the next commit,
> where the main thread pid can be restricted to one of
> two bitmasks depending on the passed parameters.
> 
> After feedback from John Kacur, the function is now
> located in src/lib/rt-numa.c.
> This allows other tests to reuse this, if they also
> prefer to warn if numa_sched_setaffinity fails.
> 
> Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
> ---
>  src/cyclictest/cyclictest.c |  8 +-------
>  src/include/rt-numa.h       |  2 ++
>  src/lib/rt-numa.c           | 11 +++++++++++
>  3 files changed, 14 insertions(+), 7 deletions(-)
> 
> diff --git a/src/cyclictest/cyclictest.c b/src/cyclictest/cyclictest.c
> index dca9610..460f6ae 100644
> --- a/src/cyclictest/cyclictest.c
> +++ b/src/cyclictest/cyclictest.c
> @@ -1791,13 +1791,7 @@ int main(int argc, char **argv)
>  
>  	/* Restrict the main pid to the affinity specified by the user */
>  	if (affinity_mask != NULL) {
> -		int res;
> -
> -		errno = 0;
> -		res = numa_sched_setaffinity(getpid(), affinity_mask);
> -		if (res != 0)
> -			warn("Couldn't setaffinity in main thread: %s\n", strerror(errno));
> -
> +		try_numa_sched_setaffinity(getpid(), affinity_mask);
>  		if (verbose)
>  			printf("Using %u cpus.\n",
>  				numa_bitmask_weight(affinity_mask));
> diff --git a/src/include/rt-numa.h b/src/include/rt-numa.h
> index ca86a45..e9fa312 100644
> --- a/src/include/rt-numa.h
> +++ b/src/include/rt-numa.h
> @@ -18,4 +18,6 @@ int cpu_for_thread_ua(int thread_num, int max_cpus);
>  
>  int parse_cpumask(char *str, int max_cpus, struct bitmask **cpumask);
>  
> +void try_numa_sched_setaffinity(__pid_t pid, struct bitmask *cpumask);
> +
>  #endif
> diff --git a/src/lib/rt-numa.c b/src/lib/rt-numa.c
> index babcc63..9a81cfb 100644
> --- a/src/lib/rt-numa.c
> +++ b/src/lib/rt-numa.c
> @@ -138,3 +138,14 @@ int parse_cpumask(char *str, int max_cpus, struct bitmask **cpumask)
>  
>  	return 0;
>  }
> +
> +void try_numa_sched_setaffinity(pid_t pid, struct bitmask *cpumask)
> +{
> +	int res;
> +
> +	errno = 0;
> +	res = numa_sched_setaffinity(pid, cpumask);
> +	if (res != 0)
> +		warn("Couldn't setaffinity for thread %d: %s\n", pid,
> +		     strerror(errno));
> +}
> -- 
> 2.30.2
> 
> 

Yuck. My bad, this is some pretty inelegant error code wrapped around a 
numa call, I don't think it really warrents trying to make an API out of 
it. That said, I might need to do this anyway, when I put back code that 
can run when numa is not available on a target machine.

For now, can you just leave this inline and respin?

Thanks

John Kacur