[PATCH 1/2] torture: Support randomized shuffling for proxy exec testing

[PATCH 1/2] torture: Support randomized shuffling for proxy exec testing

[John Stultz]

From: Connor O'Brien <connoro@google.com>

Currently shuffling sets the same cpu affinities for all tasks,
which makes us less likely to hit paths involving migrating
blocked tasks onto a cpu where they can't run.

This patch adds an element of randomness to allow affinities of
different writer tasks to diverge.

This has helped uncover issues in testing with Proxy Execution

Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Joel Fernandes <joel@joelfernandes.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Valentin Schneider <vschneid@redhat.com>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: kernel-team@android.com
Signed-off-by: Connor O'Brien <connoro@google.com>
Signed-off-by: John Stultz <jstultz@google.com>
---
 kernel/torture.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/kernel/torture.c b/kernel/torture.c
index 1a0519b836ac..8be83fdc6be1 100644
--- a/kernel/torture.c
+++ b/kernel/torture.c
@@ -54,6 +54,9 @@ module_param(verbose_sleep_frequency, int, 0444);
 static int verbose_sleep_duration = 1;
 module_param(verbose_sleep_duration, int, 0444);
 
+static int random_shuffle;
+module_param(random_shuffle, int, 0444);
+
 static char *torture_type;
 static int verbose;
 
@@ -518,6 +521,7 @@ static void torture_shuffle_task_unregister_all(void)
  */
 static void torture_shuffle_tasks(void)
 {
+	DEFINE_TORTURE_RANDOM(rand);
 	struct shuffle_task *stp;
 
 	cpumask_setall(shuffle_tmp_mask);
@@ -537,8 +541,10 @@ static void torture_shuffle_tasks(void)
 		cpumask_clear_cpu(shuffle_idle_cpu, shuffle_tmp_mask);
 
 	mutex_lock(&shuffle_task_mutex);
-	list_for_each_entry(stp, &shuffle_task_list, st_l)
-		set_cpus_allowed_ptr(stp->st_t, shuffle_tmp_mask);
+	list_for_each_entry(stp, &shuffle_task_list, st_l) {
+		if (!random_shuffle || torture_random(&rand) & 0x1)
+			set_cpus_allowed_ptr(stp->st_t, shuffle_tmp_mask);
+	}
 	mutex_unlock(&shuffle_task_mutex);
 
 	cpus_read_unlock();
-- 
2.41.0.rc2.161.g9c6817b8e7-goog

[PATCH 2/2] torture: Add lock_torture_writer_fifo module param

[John Stultz]

From: Dietmar Eggemann <dietmar.eggemann@arm.com>

Modifies locktorture writer to run as RT task.

To use it:
insmod /lib/modules/torture.ko random_shuffle=1 lock_torture_writer_fifo=1
                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^
insmod /lib/modules/locktorture.ko torture_type=mutex_lock rt_boost=1 rt_boost_factor=50 nested_locks=3

This patch has been helpful to uncover issues with the proxy-execution
seires.

Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Joel Fernandes <joel@joelfernandes.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Valentin Schneider <vschneid@redhat.com>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: kernel-team@android.com
Signed-off-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
[jstultz: Include header change to build, reword commit message]
Signed-off-by: John Stultz <jstultz@google.com>
---
 kernel/locking/locktorture.c |  3 ++-
 kernel/torture.c             | 11 ++++++++++-
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/kernel/locking/locktorture.c b/kernel/locking/locktorture.c
index 153ddc4c47ef..7cb044fc99b2 100644
--- a/kernel/locking/locktorture.c
+++ b/kernel/locking/locktorture.c
@@ -816,7 +816,8 @@ static int lock_torture_writer(void *arg)
 	bool skip_main_lock;
 
 	VERBOSE_TOROUT_STRING("lock_torture_writer task started");
-	set_user_nice(current, MAX_NICE);
+	if (!rt_task(current))
+		set_user_nice(current, MAX_NICE);
 
 	do {
 		if ((torture_random(&rand) & 0xfffff) == 0)
diff --git a/kernel/torture.c b/kernel/torture.c
index 8be83fdc6be1..db79197e257a 100644
--- a/kernel/torture.c
+++ b/kernel/torture.c
@@ -37,6 +37,7 @@
 #include <linux/ktime.h>
 #include <asm/byteorder.h>
 #include <linux/torture.h>
+#include <linux/sched/rt.h>
 #include "rcu/rcu.h"
 
 MODULE_LICENSE("GPL");
@@ -57,6 +58,9 @@ module_param(verbose_sleep_duration, int, 0444);
 static int random_shuffle;
 module_param(random_shuffle, int, 0444);
 
+static int lock_torture_writer_fifo;
+module_param(lock_torture_writer_fifo, int, 0444);
+
 static char *torture_type;
 static int verbose;
 
@@ -734,7 +738,7 @@ bool stutter_wait(const char *title)
 	cond_resched_tasks_rcu_qs();
 	spt = READ_ONCE(stutter_pause_test);
 	for (; spt; spt = READ_ONCE(stutter_pause_test)) {
-		if (!ret) {
+		if (!ret && !rt_task(current)) {
 			sched_set_normal(current, MAX_NICE);
 			ret = true;
 		}
@@ -944,6 +948,11 @@ int _torture_create_kthread(int (*fn)(void *arg), void *arg, char *s, char *m,
 		*tp = NULL;
 		return ret;
 	}
+
+	if (lock_torture_writer_fifo &&
+	    !strncmp(s, "lock_torture_writer", strlen(s)))
+		sched_set_fifo(*tp);
+
 	wake_up_process(*tp);  // Process is sleeping, so ordering provided.
 	torture_shuffle_task_register(*tp);
 	return ret;
-- 
2.41.0.rc2.161.g9c6817b8e7-goog

Re: [PATCH 2/2] torture: Add lock_torture_writer_fifo module param

[Paul E. McKenney]

On Fri, Jun 02, 2023 at 10:02:10PM +0000, John Stultz wrote:
> From: Dietmar Eggemann <dietmar.eggemann@arm.com>
> 
> Modifies locktorture writer to run as RT task.
> 
> To use it:
> insmod /lib/modules/torture.ko random_shuffle=1 lock_torture_writer_fifo=1
>                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
> insmod /lib/modules/locktorture.ko torture_type=mutex_lock rt_boost=1 rt_boost_factor=50 nested_locks=3
> 
> This patch has been helpful to uncover issues with the proxy-execution
> seires.
> 
> Cc: Davidlohr Bueso <dave@stgolabs.net>
> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> Cc: Josh Triplett <josh@joshtriplett.org>
> Cc: Joel Fernandes <joel@joelfernandes.org>
> Cc: Juri Lelli <juri.lelli@redhat.com>
> Cc: Valentin Schneider <vschneid@redhat.com>
> Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
> Cc: kernel-team@android.com
> Signed-off-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
> [jstultz: Include header change to build, reword commit message]
> Signed-off-by: John Stultz <jstultz@google.com>

Queued and pushed, thank you all!

							Thanx, Paul

> ---
>  kernel/locking/locktorture.c |  3 ++-
>  kernel/torture.c             | 11 ++++++++++-
>  2 files changed, 12 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/locking/locktorture.c b/kernel/locking/locktorture.c
> index 153ddc4c47ef..7cb044fc99b2 100644
> --- a/kernel/locking/locktorture.c
> +++ b/kernel/locking/locktorture.c
> @@ -816,7 +816,8 @@ static int lock_torture_writer(void *arg)
>  	bool skip_main_lock;
>  
>  	VERBOSE_TOROUT_STRING("lock_torture_writer task started");
> -	set_user_nice(current, MAX_NICE);
> +	if (!rt_task(current))
> +		set_user_nice(current, MAX_NICE);
>  
>  	do {
>  		if ((torture_random(&rand) & 0xfffff) == 0)
> diff --git a/kernel/torture.c b/kernel/torture.c
> index 8be83fdc6be1..db79197e257a 100644
> --- a/kernel/torture.c
> +++ b/kernel/torture.c
> @@ -37,6 +37,7 @@
>  #include <linux/ktime.h>
>  #include <asm/byteorder.h>
>  #include <linux/torture.h>
> +#include <linux/sched/rt.h>
>  #include "rcu/rcu.h"
>  
>  MODULE_LICENSE("GPL");
> @@ -57,6 +58,9 @@ module_param(verbose_sleep_duration, int, 0444);
>  static int random_shuffle;
>  module_param(random_shuffle, int, 0444);
>  
> +static int lock_torture_writer_fifo;
> +module_param(lock_torture_writer_fifo, int, 0444);
> +
>  static char *torture_type;
>  static int verbose;
>  
> @@ -734,7 +738,7 @@ bool stutter_wait(const char *title)
>  	cond_resched_tasks_rcu_qs();
>  	spt = READ_ONCE(stutter_pause_test);
>  	for (; spt; spt = READ_ONCE(stutter_pause_test)) {
> -		if (!ret) {
> +		if (!ret && !rt_task(current)) {
>  			sched_set_normal(current, MAX_NICE);
>  			ret = true;
>  		}
> @@ -944,6 +948,11 @@ int _torture_create_kthread(int (*fn)(void *arg), void *arg, char *s, char *m,
>  		*tp = NULL;
>  		return ret;
>  	}
> +
> +	if (lock_torture_writer_fifo &&
> +	    !strncmp(s, "lock_torture_writer", strlen(s)))
> +		sched_set_fifo(*tp);
> +
>  	wake_up_process(*tp);  // Process is sleeping, so ordering provided.
>  	torture_shuffle_task_register(*tp);
>  	return ret;
> -- 
> 2.41.0.rc2.161.g9c6817b8e7-goog
> 

Re: [PATCH 2/2] torture: Add lock_torture_writer_fifo module param

[Davidlohr Bueso]

On Sat, 03 Jun 2023, Paul E. McKenney wrote:

>On Fri, Jun 02, 2023 at 10:02:10PM +0000, John Stultz wrote:
>> From: Dietmar Eggemann <dietmar.eggemann@arm.com>
>>
>> Modifies locktorture writer to run as RT task.
>>
>> To use it:
>> insmod /lib/modules/torture.ko random_shuffle=1 lock_torture_writer_fifo=1
>>                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
>> insmod /lib/modules/locktorture.ko torture_type=mutex_lock rt_boost=1 rt_boost_factor=50 nested_locks=3
>>
>> This patch has been helpful to uncover issues with the proxy-execution
>> seires.
>>
>> Cc: Davidlohr Bueso <dave@stgolabs.net>
>> Cc: "Paul E. McKenney" <paulmck@kernel.org>
>> Cc: Josh Triplett <josh@joshtriplett.org>
>> Cc: Joel Fernandes <joel@joelfernandes.org>
>> Cc: Juri Lelli <juri.lelli@redhat.com>
>> Cc: Valentin Schneider <vschneid@redhat.com>
>> Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
>> Cc: kernel-team@android.com
>> Signed-off-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
>> [jstultz: Include header change to build, reword commit message]
>> Signed-off-by: John Stultz <jstultz@google.com>
>
>Queued and pushed, thank you all!

Both look good to me. Feel free to add my:

Acked-by: Davidlohr Bueso <dave@stgolabs.net>

Re: [PATCH 2/2] torture: Add lock_torture_writer_fifo module param

[Paul E. McKenney]

On Sat, Jun 03, 2023 at 10:34:14AM -0700, Davidlohr Bueso wrote:
> On Sat, 03 Jun 2023, Paul E. McKenney wrote:
> 
> > On Fri, Jun 02, 2023 at 10:02:10PM +0000, John Stultz wrote:
> > > From: Dietmar Eggemann <dietmar.eggemann@arm.com>
> > > 
> > > Modifies locktorture writer to run as RT task.
> > > 
> > > To use it:
> > > insmod /lib/modules/torture.ko random_shuffle=1 lock_torture_writer_fifo=1
> > >                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > insmod /lib/modules/locktorture.ko torture_type=mutex_lock rt_boost=1 rt_boost_factor=50 nested_locks=3
> > > 
> > > This patch has been helpful to uncover issues with the proxy-execution
> > > seires.
> > > 
> > > Cc: Davidlohr Bueso <dave@stgolabs.net>
> > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > Cc: Josh Triplett <josh@joshtriplett.org>
> > > Cc: Joel Fernandes <joel@joelfernandes.org>
> > > Cc: Juri Lelli <juri.lelli@redhat.com>
> > > Cc: Valentin Schneider <vschneid@redhat.com>
> > > Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
> > > Cc: kernel-team@android.com
> > > Signed-off-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
> > > [jstultz: Include header change to build, reword commit message]
> > > Signed-off-by: John Stultz <jstultz@google.com>
> > 
> > Queued and pushed, thank you all!
> 
> Both look good to me. Feel free to add my:
> 
> Acked-by: Davidlohr Bueso <dave@stgolabs.net>

Thank you!  I will apply this on my next rebase.

							Thanx, Paul