[PATCH 0/2] firmware/psci: PSCI checker cleanup

[PATCH 0/2] firmware/psci: PSCI checker cleanup

[Valentin Schneider]

Hi folks,

This is a small cleanup of the PSCI checker following Peter's objections
to its homegrown do_idle() implementation. It is based on his
sched_setscheduler() unexport series at [1].

I've never really used the thing before, but it still seems to behave
correctly on my Juno r0 & HiKey960.

Cheers,
Valentin

[1]: https://lore.kernel.org/lkml/20200422112719.826676174@infradead.org/T/#t

Valentin Schneider (2):
  firmware/psci: Make PSCI checker not bother with parking
  firmware/psci: Make PSCI checker use play_idle()

 drivers/firmware/psci/psci_checker.c | 159 +++++++--------------------
 1 file changed, 42 insertions(+), 117 deletions(-)

--
2.24.0

[PATCH 1/2] firmware/psci: Make PSCI checker not bother with parking

[Valentin Schneider]

kthread_stop() unconditionally calls kthread_unpark(). For kthreads
with KTHREAD_IS_PER_CPU, this leads to waiting for
wait_task_inactive(TASK_PARKED) before re-binding them. This is required
mainly for smpboot threads, as they are parked before their respective CPU
comes online which causes their affinity mask to be reset.

For users of kthread_create_on_cpu(), this means they must park their
threads before stopping them, which is a bit silly. While we could change
kthread_unpark() to only do the rebind if it is required, using
kthread_create_on_cpu() for anything else than smpboot threads is risky:
they won't get parked/unparked during a hotplug cycle, so their affinity
will just be reset as soon as their respective CPU goes out.

The PSCI checker only lives during initcalls and explicitly points out
that it cannot exist concurrently with hotplug operations, so let's just
make it use kthread_create_on_node() + kthread_bind() (similar to what
e.g. RCU torture does).

Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
---
 drivers/firmware/psci/psci_checker.c | 32 +++++++++++-----------------
 1 file changed, 12 insertions(+), 20 deletions(-)

diff --git a/drivers/firmware/psci/psci_checker.c b/drivers/firmware/psci/psci_checker.c
index a5279a430274..fa7bb1e8a461 100644
--- a/drivers/firmware/psci/psci_checker.c
+++ b/drivers/firmware/psci/psci_checker.c
@@ -347,19 +347,12 @@ static int suspend_test_thread(void *arg)
 	if (atomic_dec_return_relaxed(&nb_active_threads) == 0)
 		complete(&suspend_threads_done);
 
-	for (;;) {
-		/* Needs to be set first to avoid missing a wakeup. */
-		set_current_state(TASK_INTERRUPTIBLE);
-		if (kthread_should_park())
-			break;
-		schedule();
-	}
+	while (!kthread_should_stop())
+		schedule_timeout_interruptible(MAX_SCHEDULE_TIMEOUT);
 
 	pr_info("CPU %d suspend test results: success %d, shallow states %d, errors %d\n",
 		cpu, nb_suspend, nb_shallow_sleep, nb_err);
 
-	kthread_parkme();
-
 	return nb_err;
 }
 
@@ -395,13 +388,15 @@ static int suspend_tests(void)
 			continue;
 		}
 
-		thread = kthread_create_on_cpu(suspend_test_thread,
-					       (void *)(long)cpu, cpu,
-					       "psci_suspend_test");
-		if (IS_ERR(thread))
+		thread = kthread_create_on_node(suspend_test_thread,
+						(void *)(long)cpu, cpu_to_node(cpu),
+						"psci_suspend_test/%d", cpu);
+		if (IS_ERR(thread)) {
 			pr_err("Failed to create kthread on CPU %d\n", cpu);
-		else
+		} else {
 			threads[nb_threads++] = thread;
+			kthread_bind(thread, cpu);
+		}
 	}
 
 	if (nb_threads < 1) {
@@ -418,17 +413,14 @@ static int suspend_tests(void)
 	 */
 	for (i = 0; i < nb_threads; ++i)
 		wake_up_process(threads[i]);
-	complete_all(&suspend_threads_started);
 
+	complete_all(&suspend_threads_started);
 	wait_for_completion(&suspend_threads_done);
 
-
 	/* Stop and destroy all threads, get return status. */
-	for (i = 0; i < nb_threads; ++i) {
-		err += kthread_park(threads[i]);
+	for (i = 0; i < nb_threads; ++i)
 		err += kthread_stop(threads[i]);
-	}
- out:
+out:
 	cpuidle_resume_and_unlock();
 	kfree(threads);
 	return err;
-- 
2.24.0

[PATCH 2/2] firmware/psci: Make PSCI checker use play_idle()

[Valentin Schneider]

To test if we can reach all idle states, the checker implements a
minimalist open-coded version of do_idle(). While it gets the job done,
it's not the nicest way to go about it.

What we can do instead is to nudge cpuidle to go for the idle state we want
with cpuidle_use_deepest_state(), go idle via play_idle(), and check the
cpuidle usage stats to see which idle state we entered.

We don't directly get the state.enter() return value anymore, but if we
never enter a given idle state throughout the entire test, then that
should be a good indicator that something is off.

Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
---
 drivers/firmware/psci/psci_checker.c | 131 +++++++--------------------
 1 file changed, 32 insertions(+), 99 deletions(-)

diff --git a/drivers/firmware/psci/psci_checker.c b/drivers/firmware/psci/psci_checker.c
index fa7bb1e8a461..722df252af28 100644
--- a/drivers/firmware/psci/psci_checker.c
+++ b/drivers/firmware/psci/psci_checker.c
@@ -226,56 +226,15 @@ static int hotplug_tests(void)
 	return err;
 }
 
-static void dummy_callback(struct timer_list *unused) {}
-
-static int suspend_cpu(struct cpuidle_device *dev,
-		       struct cpuidle_driver *drv, int index)
-{
-	struct cpuidle_state *state = &drv->states[index];
-	bool broadcast = state->flags & CPUIDLE_FLAG_TIMER_STOP;
-	int ret;
-
-	arch_cpu_idle_enter();
-
-	if (broadcast) {
-		/*
-		 * The local timer will be shut down, we need to enter tick
-		 * broadcast.
-		 */
-		ret = tick_broadcast_enter();
-		if (ret) {
-			/*
-			 * In the absence of hardware broadcast mechanism,
-			 * this CPU might be used to broadcast wakeups, which
-			 * may be why entering tick broadcast has failed.
-			 * There is little the kernel can do to work around
-			 * that, so enter WFI instead (idle state 0).
-			 */
-			cpu_do_idle();
-			ret = 0;
-			goto out_arch_exit;
-		}
-	}
-
-	ret = state->enter(dev, drv, index);
-
-	if (broadcast)
-		tick_broadcast_exit();
-
-out_arch_exit:
-	arch_cpu_idle_exit();
-
-	return ret;
-}
-
-static int suspend_test_thread(void *arg)
+static int suspend_test_thread(void *__unused)
 {
-	int cpu = (long)arg;
-	int i, nb_suspend = 0, nb_shallow_sleep = 0, nb_err = 0;
+	DECLARE_BITMAP(states_reached, CPUIDLE_STATE_MAX);
 	struct cpuidle_device *dev;
 	struct cpuidle_driver *drv;
-	/* No need for an actual callback, we just want to wake up the CPU. */
-	struct timer_list wakeup_timer;
+	int cpu = smp_processor_id();
+	int i, nr_states;
+	int nr_shallows = 0;
+	bool pass;
 
 	/* Wait for the main thread to give the start signal. */
 	wait_for_completion(&suspend_threads_started);
@@ -286,11 +245,12 @@ static int suspend_test_thread(void *arg)
 
 	dev = this_cpu_read(cpuidle_devices);
 	drv = cpuidle_get_cpu_driver(dev);
+	nr_states = drv->state_count;
+	bitmap_zero(states_reached, CPUIDLE_STATE_MAX);
 
 	pr_info("CPU %d entering suspend cycles, states 1 through %d\n",
 		cpu, drv->state_count - 1);
 
-	timer_setup_on_stack(&wakeup_timer, dummy_callback, 0);
 	for (i = 0; i < NUM_SUSPEND_CYCLE; ++i) {
 		int index;
 		/*
@@ -298,62 +258,45 @@ static int suspend_test_thread(void *arg)
 		 * doesn't use PSCI).
 		 */
 		for (index = 1; index < drv->state_count; ++index) {
-			int ret;
 			struct cpuidle_state *state = &drv->states[index];
+			unsigned long long prev, next;
 
+			prev = READ_ONCE(dev->states_usage[index].usage);
 			/*
-			 * Set the timer to wake this CPU up in some time (which
-			 * should be largely sufficient for entering suspend).
-			 * If the local tick is disabled when entering suspend,
-			 * suspend_cpu() takes care of switching to a broadcast
-			 * tick, so the timer will still wake us up.
+			 * This will attempt to go into the deepest idle state
+			 * possible. By specifying the exit latency, we prevent
+			 * higher idle states from being picked, so we should
+			 * end up in the one idle state we want. This may cause
+			 * issues if there are several idle states with
+			 * identical exit latency, but who does that?
 			 */
-			mod_timer(&wakeup_timer, jiffies +
-				  usecs_to_jiffies(state->target_residency));
-
-			/* IRQs must be disabled during suspend operations. */
-			local_irq_disable();
+			play_idle_precise(state->target_residency_ns,
+					  state->exit_latency_ns);
 
-			ret = suspend_cpu(dev, drv, index);
+			next = READ_ONCE(dev->states_usage[index].usage);
 
-			/*
-			 * We have woken up. Re-enable IRQs to handle any
-			 * pending interrupt, do not wait until the end of the
-			 * loop.
-			 */
-			local_irq_enable();
-
-			if (ret == index) {
-				++nb_suspend;
-			} else if (ret >= 0) {
-				/* We did not enter the expected state. */
-				++nb_shallow_sleep;
-			} else {
-				pr_err("Failed to suspend CPU %d: error %d "
-				       "(requested state %d, cycle %d)\n",
-				       cpu, ret, index, i);
-				++nb_err;
-			}
+			if (prev != next)
+				bitmap_set(states_reached, index, 1);
+			else
+				nr_shallows++;
 		}
 	}
 
-	/*
-	 * Disable the timer to make sure that the timer will not trigger
-	 * later.
-	 */
-	del_timer(&wakeup_timer);
-	destroy_timer_on_stack(&wakeup_timer);
+	/* Assert we reached each state at least once */
+	pass = bitmap_weight(states_reached, CPUIDLE_STATE_MAX) == nr_states - 1;
 
 	if (atomic_dec_return_relaxed(&nb_active_threads) == 0)
 		complete(&suspend_threads_done);
 
+	pr_info("CPU %d suspend results: reached %d states out of %d (%d/%d misses)\n",
+		cpu, bitmap_weight(states_reached, CPUIDLE_STATE_MAX),
+		nr_states - 1, nr_shallows,
+		(nr_states - 1) * NUM_SUSPEND_CYCLE);
+
 	while (!kthread_should_stop())
 		schedule_timeout_interruptible(MAX_SCHEDULE_TIMEOUT);
 
-	pr_info("CPU %d suspend test results: success %d, shallow states %d, errors %d\n",
-		cpu, nb_suspend, nb_shallow_sleep, nb_err);
-
-	return nb_err;
+	return !pass;
 }
 
 static int suspend_tests(void)
@@ -367,15 +310,6 @@ static int suspend_tests(void)
 	if (!threads)
 		return -ENOMEM;
 
-	/*
-	 * Stop cpuidle to prevent the idle tasks from entering a deep sleep
-	 * mode, as it might interfere with the suspend threads on other CPUs.
-	 * This does not prevent the suspend threads from using cpuidle (only
-	 * the idle tasks check this status). Take the idle lock so that
-	 * the cpuidle driver and device look-up can be carried out safely.
-	 */
-	cpuidle_pause_and_lock();
-
 	for_each_online_cpu(cpu) {
 		struct task_struct *thread;
 		/* Check that cpuidle is available on that CPU. */
@@ -389,7 +323,7 @@ static int suspend_tests(void)
 		}
 
 		thread = kthread_create_on_node(suspend_test_thread,
-						(void *)(long)cpu, cpu_to_node(cpu),
+						NULL, cpu_to_node(cpu),
 						"psci_suspend_test/%d", cpu);
 		if (IS_ERR(thread)) {
 			pr_err("Failed to create kthread on CPU %d\n", cpu);
@@ -421,7 +355,6 @@ static int suspend_tests(void)
 	for (i = 0; i < nb_threads; ++i)
 		err += kthread_stop(threads[i]);
 out:
-	cpuidle_resume_and_unlock();
 	kfree(threads);
 	return err;
 }
-- 
2.24.0

Re: [PATCH 0/2] firmware/psci: PSCI checker cleanup

[Sudeep Holla]

On Fri, Apr 24, 2020 at 02:56:55PM +0100, Valentin Schneider wrote:
> Hi folks,
> 
> This is a small cleanup of the PSCI checker following Peter's objections
> to its homegrown do_idle() implementation. It is based on his
> sched_setscheduler() unexport series at [1].
> 
> I've never really used the thing before, but it still seems to behave
> correctly on my Juno r0 & HiKey960.
> 

Reviewed-by: Sudeep Holla <sudeep.holla@arm.com>
Tested-by: Sudeep Holla <sudeep.holla@arm.com>

-- 
Regards,
Sudeep

Re: [PATCH 0/2] firmware/psci: PSCI checker cleanup

[Valentin Schneider]

On 03/06/20 18:05, Sudeep Holla wrote:
> On Fri, Apr 24, 2020 at 02:56:55PM +0100, Valentin Schneider wrote:
>> Hi folks,
>>
>> This is a small cleanup of the PSCI checker following Peter's objections
>> to its homegrown do_idle() implementation. It is based on his
>> sched_setscheduler() unexport series at [1].
>>
>> I've never really used the thing before, but it still seems to behave
>> correctly on my Juno r0 & HiKey960.
>>
>
> Reviewed-by: Sudeep Holla <sudeep.holla@arm.com>
> Tested-by: Sudeep Holla <sudeep.holla@arm.com>

Thanks!

AIUI the plan is to have the base in for the following version, so we
can wait until then - or I can rebase this on top of mainline, and
whoever will be on the receiving end of the merge conflict will be
slightly annoyed :-)

I'm in no particular rush, and this isn't very hot code, so up to you.

Re: [PATCH 0/2] firmware/psci: PSCI checker cleanup

[Valentin Schneider]

Hi,

On 03/06/20 18:39, Valentin Schneider wrote:
> On 03/06/20 18:05, Sudeep Holla wrote:
>> On Fri, Apr 24, 2020 at 02:56:55PM +0100, Valentin Schneider wrote:
>>> Hi folks,
>>>
>>> This is a small cleanup of the PSCI checker following Peter's objections
>>> to its homegrown do_idle() implementation. It is based on his
>>> sched_setscheduler() unexport series at [1].
>>>
>>> I've never really used the thing before, but it still seems to behave
>>> correctly on my Juno r0 & HiKey960.
>>>
>>
>> Reviewed-by: Sudeep Holla <sudeep.holla@arm.com>
>> Tested-by: Sudeep Holla <sudeep.holla@arm.com>
>
> Thanks!
>
> AIUI the plan is to have the base in for the following version, so we
> can wait until then - or I can rebase this on top of mainline, and
> whoever will be on the receiving end of the merge conflict will be
> slightly annoyed :-)
>
> I'm in no particular rush, and this isn't very hot code, so up to you.
>

The sched_setscheduler() series is in 5.9. Patches apply cleanly atop 5.9,
and I double-checked on my Juno that they still actually work :-)

> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel