From: Daniel Bristot de Oliveira <bristot@redhat.com>
To: Steven Rostedt <rostedt@goodmis.org>
Cc: Phil Auld <pauld@redhat.com>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
Kate Carcia <kcarcia@redhat.com>,
Jonathan Corbet <corbet@lwn.net>, Ingo Molnar <mingo@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
Thomas Gleixner <tglx@linutronix.de>,
Alexandre Chartre <alexandre.chartre@oracle.com>,
Clark Willaims <williams@redhat.com>,
John Kacur <jkacur@redhat.com>,
Juri Lelli <juri.lelli@redhat.com>,
Borislav Petkov <bp@alien8.de>, "H. Peter Anvin" <hpa@zytor.com>,
x86@kernel.org, linux-doc@vger.kernel.org,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH V4 05/12] trace/hwlat: Support hotplug operations
Date: Tue, 22 Jun 2021 16:05:33 +0200 [thread overview]
Message-ID: <77575407-0332-8734-2697-bc2099fd9fe0@redhat.com> (raw)
In-Reply-To: <20210621134636.5b332226@oasis.local.home>
On 6/21/21 7:46 PM, Steven Rostedt wrote:
> On Mon, 21 Jun 2021 18:14:36 +0200
> Daniel Bristot de Oliveira <bristot@redhat.com> wrote:
>
>>>> Yep! I tried to take the trace_type_lock here, and got the lockdep info about
>>>> this problem.
>>>>
>>>>> The only thing I could think of is to wake up a worker thread to do the
>>>>> work. That is, this just wakes the worker thread, then the worker grabs
>>>>> the trace_types_lock, iterates through the cpu mask of expect running
>>>>> threads, and then starts or kills them depending on the hwlat_busy
>>>>> value.
>>>> So, it will not wait for the kworker to run?
>>> What wont wait?
>>
>> For example, at the shutdown, should the hotplug callback wait for the workqueue
>> to run & kill the thread, or not?
>
> Doing that won't help the deadlock situation.
yep, that is why I asked... :-(
> CPU 1 CPU 2
> ----- -----
> Start shutdown
> down online_cpus()
>
> mutex_lock(trace_types_lock);
> get_online_cpus()
> [BLOCK]
>
> wake_up_thread;
> [schedule worker]
>
> mutex_lock(trace_types_lock);
>
> [ DEADLOCK ]
>
>
> Make all access to save_cpumask and hwlat_per_cpu_data inside the
> get_online_cpus() protection. (like in move_to_next_cpu(),
> start_single_thread() expand the get_online_cpus()).
>
> Then in the cpu going down case, we can simply kill the thread and
> update the save_cpumask, as it will be protected by the
> get_online_cpus() code.
>
> That is, don't even check if hwlat_busy is set or not. Just simply do:
>
>
> CPU_DOWN:
>
> stop_cpu_kthead(cpu);
>
> That will stop the kthread if it is running. But we should update
> that function to also set per_cpu(hwlat_per_cpu_data).kthread = NULL;
> Like stop_single_kthread() does.
>
> But for CPU_UP, we should do the work via a worker thread.
>
> CPU_UP:
> schedule_work_on(&update_kthreads, cpu);
>
> Which in the work function for that update_kthreads work queue:
>
> mutex_lock(&trace_types_lock);
> if (!hwlat_busy || hwlat_data.thread_mode != MODE_PER_CPU)
> goto out_unlock;
>
> get_online_cpus();
> if (!this_cpu(hwlat_per_cpu_data).kthread)
> start_per_cpu_kthread(smp_processor_id());
> put_online_cpus();
>
> out_unlock:
> mutex_unlock(&trace_types_lock);
>
> Or something like that.
It works!
I will send a v5 with all the fixes requested, including this one.
Thanks Steven!
-- Daniel
next prev parent reply other threads:[~2021-06-22 14:05 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-15 9:28 [PATCH V4 00/12] hwlat improvements and osnoise/timerlat tracers Daniel Bristot de Oliveira
2021-06-15 9:28 ` [PATCH V4 01/12] trace/hwlat: Fix Clark's email Daniel Bristot de Oliveira
2021-06-15 9:28 ` [PATCH V4 02/12] trace/hwlat: Implement the mode config option Daniel Bristot de Oliveira
2021-06-15 9:28 ` [PATCH V4 03/12] trace/hwlat: Switch disable_migrate to mode none Daniel Bristot de Oliveira
2021-06-15 9:28 ` [PATCH V4 04/12] trace/hwlat: Implement the per-cpu mode Daniel Bristot de Oliveira
2021-06-15 9:28 ` [PATCH V4 05/12] trace/hwlat: Support hotplug operations Daniel Bristot de Oliveira
2021-06-18 16:45 ` Steven Rostedt
2021-06-18 19:00 ` Steven Rostedt
2021-06-21 11:34 ` Daniel Bristot de Oliveira
2021-06-21 15:25 ` Steven Rostedt
2021-06-21 16:14 ` Daniel Bristot de Oliveira
2021-06-21 17:46 ` Steven Rostedt
2021-06-22 14:05 ` Daniel Bristot de Oliveira [this message]
2021-06-15 9:28 ` [PATCH V4 06/12] trace: Add a generic function to read/write u64 values from tracefs Daniel Bristot de Oliveira
2021-06-18 16:56 ` Steven Rostedt
2021-06-21 11:50 ` Daniel Bristot de Oliveira
2021-06-15 9:28 ` [PATCH V4 07/12] trace/hwlat: Use trace_min_max_param for width and window params Daniel Bristot de Oliveira
2021-06-15 9:28 ` [PATCH V4 08/12] trace/hwlat: Remove printk from sampling loop Daniel Bristot de Oliveira
2021-06-15 9:28 ` [PATCH V4 09/12] trace: Add __print_ns_to_secs() and __print_ns_without_secs() helpers Daniel Bristot de Oliveira
2021-06-15 9:28 ` [PATCH V4 10/12] trace: Add osnoise tracer Daniel Bristot de Oliveira
2021-06-18 17:55 ` Steven Rostedt
2021-06-21 12:09 ` Daniel Bristot de Oliveira
2021-06-18 18:27 ` Steven Rostedt
2021-06-21 12:09 ` Daniel Bristot de Oliveira
2021-06-15 9:28 ` [PATCH V4 11/12] trace: Add timerlat tracer Daniel Bristot de Oliveira
2021-06-15 9:28 ` [PATCH V4 12/12] trace/osnoise: Support hotplug operations Daniel Bristot de Oliveira
2021-06-18 19:04 ` Steven Rostedt
2021-06-21 12:18 ` Daniel Bristot de Oliveira
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=77575407-0332-8734-2697-bc2099fd9fe0@redhat.com \
--to=bristot@redhat.com \
--cc=alexandre.chartre@oracle.com \
--cc=bigeasy@linutronix.de \
--cc=bp@alien8.de \
--cc=corbet@lwn.net \
--cc=hpa@zytor.com \
--cc=jkacur@redhat.com \
--cc=juri.lelli@redhat.com \
--cc=kcarcia@redhat.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=pauld@redhat.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
--cc=williams@redhat.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).