Re: About CPU hot-plug stress test failed in cpufreq driver

From: Viresh Kumar <viresh.kumar@linaro.org>
To: "Rafael J. Wysocki" <rjw@rjwysocki.net>, peterz@infradead.org
Cc: Anson Huang <anson.huang@nxp.com>, Jacky Bai <ping.bai@nxp.com>,
	"linux-pm@vger.kernel.org" <linux-pm@vger.kernel.org>
Subject: Re: About CPU hot-plug stress test failed in cpufreq driver
Date: Thu, 5 Dec 2019 16:47:16 +0530	[thread overview]
Message-ID: <20191205111716.eu73f22x4rz652x2@vireshk-i7> (raw)
In-Reply-To: <4619762.rfsBlt1pEd@kreacher>

+Peter

On 05-12-19, 12:10, Rafael J. Wysocki wrote:
> On Thursday, December 5, 2019 12:00:34 PM CET Viresh Kumar wrote:
> > On 05-12-19, 08:53, Anson Huang wrote:
> > > Hi, Rafael
> > > 	This issue is very weird, the irq_work used in cpufreq_governor.c is very simple, ONLY one entry to claim the irq_work, and cpufreq_governor's irq_work is a private irq_work structure, no other drivers use it. I added some trace event in cpufreq_governor.c and irq_work.c, every time, the issue happened at the point of CPU1/2/3 all off, and CPU1 start ON line, but when CPU1 tried to sync the irq_work in cpufreq_dbs_governor_stop(), the irq_work shows that previous work is pending on CPU3 which is offline, I also had the trace event in irq_work_claim(),  but no any log shows the cpufreq_governor irq_work is claimed on CPU3 after CPU3 offline, below is the debug patch I added and the log on 2 consoles:
> > > 	If I understand it correctly, the irq work used in cpufreq_governor ONLY has one entry of calling irq_work_queue() which will be ONLY claimed on the CPU calling the irq_work_queue(), but from trace result, I have NOT see where CPU3 could call irq_work_queue() after it finishes the irq work sync before offline. Could it something wrong related to cache maintain during CPU hotplug?
> > 
> > I think you earlier said that the issue wasn't there in 4.19 kernel,
> > right ? What about doing git bisect to see if we can find the
> > offending commit ?
> 
> That won't hurt, but I guess that it will be just the one that started to
> use irq_work ..

cpufreq_governor.c has hardly seen any patches since 4.19 and irq_work
is being used since a long time before 4.19 I think.

> Also note that schedutil has the same issue.

I agree, but at the same time both cpufreq_governor.c and schedutil
have almost exactly the same code around irq_work. It may be an update
to irq_work implementation, or something else that is causing the
corruption here.

And git blame may just point us to the offending patch if we are lucky
enough :)

-- 
viresh