stable.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Zhouyi Zhou <zhouzhouyi@gmail.com>
To: Joel Fernandes <joel@joelfernandes.org>
Cc: paulmck@kernel.org, linux-kernel@vger.kernel.org,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Frederic Weisbecker <frederic@kernel.org>,
	Will Deacon <will@kernel.org>, Marc Zyngier <maz@kernel.org>,
	rcu <rcu@vger.kernel.org>,
	stable@vger.kernel.org, Frederic Weisbecker <fweisbec@gmail.com>,
	Ingo Molnar <mingo@kernel.org>,
	"Rafael J. Wysocki" <rafael@kernel.org>,
	Thomas Gleixner <tglx@linutronix.de>
Subject: Re: [PATCH v2 RESEND] tick/nohz: Fix cpu_is_hotpluggable() by checking with nohz subsystem
Date: Sat, 28 Jan 2023 09:42:59 +0800	[thread overview]
Message-ID: <CAABZP2zr5oDgOOP0yE+TfwJz4hoAOG+4d5C8MJgs_LuSCS-kpw@mail.gmail.com> (raw)
In-Reply-To: <CAABZP2ztv8NkvGVQt8D+=6tNj4dg=6-Sp41+HGeNFxZ1oXsmNQ@mail.gmail.com>

On Thu, Jan 26, 2023 at 11:41 PM Zhouyi Zhou <zhouzhouyi@gmail.com> wrote:
>
> On Thu, Jan 26, 2023 at 11:14 PM Joel Fernandes <joel@joelfernandes.org> wrote:
> >
> > On Thu, Jan 26, 2023 at 10:01 AM Zhouyi Zhou <zhouzhouyi@gmail.com> wrote:
> > >
> > > On Thu, Jan 26, 2023 at 12:16 PM Zhouyi Zhou <zhouzhouyi@gmail.com> wrote:
> > > >
> > > > On Wed, Jan 25, 2023 at 8:13 AM Zhouyi Zhou <zhouzhouyi@gmail.com> wrote:
> > > > >
> > > > > On Wed, Jan 25, 2023 at 6:56 AM Paul E. McKenney <paulmck@kernel.org> wrote:
> > > > > >
> > > > > > On Tue, Jan 24, 2023 at 05:31:26PM +0000, Joel Fernandes (Google) wrote:
> > > > > > > For CONFIG_NO_HZ_FULL systems, the tick_do_timer_cpu cannot be offlined.
> > > > > > > However, cpu_is_hotpluggable() still returns true for those CPUs. This causes
> > > > > > > torture tests that do offlining to end up trying to offline this CPU causing
> > > > > > > test failures. Such failure happens on all architectures.
> > > > > > >
> > > > > > > Fix it by asking the opinion of the nohz subsystem on whether the CPU can
> > > > > > > be hotplugged.
> > > > > > >
> > > > > > > [ Apply Frederic Weisbecker feedback on refactoring tick_nohz_cpu_down(). ]
> > > > > > >
> > > > > > > For drivers/base/ portion:
> > > > > > > Acked-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> > > > > > >
> > > > > > > Cc: Frederic Weisbecker <frederic@kernel.org>
> > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > Cc: Zhouyi Zhou <zhouzhouyi@gmail.com>
> > > > > > > Cc: Will Deacon <will@kernel.org>
> > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > Cc: rcu <rcu@vger.kernel.org>
> > > > > > > Cc: stable@vger.kernel.org
> > > > > > > Fixes: 2987557f52b9 ("driver-core/cpu: Expose hotpluggability to the rest of the kernel")
> > > > > > > Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
> > > > > >
> > > > > > Queued for further review and testing, thank you both!
> > > > > >
> > > > > > It might be a few hours until it becomes publicly visible, but it will
> > > > > > get there.
> > > > > A new round of rcutorture test on fixed linux-5.15.y[3] has been
> > > > > performed in the PPC VM of Open Source Lab of Oregon State University
> > > > > [1], which will last about 29 hours. The test result on original
> > > > > linux-5.15.y is at [2].
> > > > From the result of [1], the HOTPLUG failure reports have been
> > > > eliminated, but a new kernel null point bug related to scsi module has
> > > > been reported [4] ;-(
> > > > [    5.178733][    C1] BUG: Kernel NULL pointer dereference on read at
> > > > 0x00000008
> > > > ...
> > > > [    5.231013][    C1] [c00000001ff9fca0] [c0000000009ffbc8]
> > > > scsi_end_request+0xd8/0x1f0 (unreliable)^M
> > > > [    5.234961][    C1] [c00000001ff9fcf0] [c000000000a00e68]
> > > > scsi_io_completion+0x88/0x700^M
> > > > [    5.237863][    C1] [c00000001ff9fda0] [c0000000009f5028]
> > > > scsi_finish_command+0xe8/0x150^M
> > > > [    5.240089][    C1] [c00000001ff9fdf0] [c000000000a00c70]
> > > > scsi_complete+0x90/0x140^M
> > > > [    5.242481][    C1] [c00000001ff9fe20] [c0000000007e5170]
> > > > blk_complete_reqs+0x80/0xa0^M
> > > > [    5.245187][    C1] [c00000001ff9fe50] [c000000000f0b5d0]
> > > > __do_softirq+0x1e0/0x4e0^M
> > > > [    5.248479][    C1] [c00000001ff9ff90] [c0000000000170e8]
> > > > do_softirq_own_stack+0x48/0x60^M
> > > > [    5.250919][    C1] [c00000000a5e7c40] [c00000000a5e7c80]
> > > > 0xc00000000a5e7c80^M
> > > > [    5.253792][    C1] [c00000000a5e7c70] [c0000000001534c0]
> > > > do_softirq+0xb0/0xc0^M
> > > > [    5.256824][    C1] [c00000000a5e7ca0] [c0000000001535ac]
> > > > __local_bh_enable_ip+0xdc/0x110^M
> > > > [    5.259414][    C1] [c00000000a5e7cc0] [c0000000001d75e8]
> > > > irq_forced_thread_fn+0xc8/0xf0^M
> > > > [    5.261921][    C1] [c00000000a5e7d00] [c0000000001d7ae4]
> > > > irq_thread+0x1b4/0x2a0^M
> > > > [    5.265298][    C1] [c00000000a5e7da0] [c00000000017d8c8]
> > > > kthread+0x1a8/0x1d0^M
> > > > [    5.269184][    C1] [c00000000a5e7e10] [c00000000000cee4]
> > > > ret_from_kernel_thread+0x5c/0x64^M
> > > >
> > > > But when I invoked [5]  by hand, the bug did not show again [6].
> > > >
> > > > [4] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/console.log
> > > > [5] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/qemu-cmd
> > > > [6] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/console.zzy.log
> > > >
> > > > I think the bug is not caused by Joel's patch, there must be some
> > > > other reason. I am starting the 29 hours' rcutorture test again. And I
> > > > can give ssh access to you if you are interested in it.
> > > >
> > > > Sorry for the inconvenience that I bring
> > > >
> > > > Thanks
> > > > Zhouyi
> > > Hi the above kernel null pointer dereference bug has nothing to do
> > > with Joel's fix because I can reproduce it on original 5.15.y [7]
> > > using as while loop [8] (after 36 iterations, the bug fires).
> > > So, Joel's patch is tested good!
> > > Tested-by: Zhouyi Zhou <zhouzhouyi@gmail.com>
> >
> > Interesting. I have been running rcutorture's TREE03 on 5.15.y quite a
> > lot and I don't see such an issue.
> >
> >  However, your logs showed the crash is SCSI related. These were the
> > recent SCSI commits in 5.15.y but I am not sure if it causes the
> > issue:
> >
> > 13259b6 scsi: mpi3mr: Refer CONFIG_SCSI_MPI3MR in Makefile
> > 513fdf0 scsi: ufs: Stop using the clock scaling lock in the error handler
> > 7c26d21 scsi: ufs: core: WLUN suspend SSU/enter hibern
> >
> > Perhaps report it to the scsi and/or stable lists, or do some web
> > searches for if someone else sees it.
> Thank Joel for your guidance!
> I will continue the research following your instructions!
> I will report to you and the community once I have any process.
Hi
I think I have chased down the reason behind the bug reported by SCSI
subsystem following your guidance, and have reported to SCSI
mainteners [1]
This is a fruitful journey, thank you all ;-)

So once again, Joel's patch is tested good ;-)
Tested-by: Zhouyi Zhou <zhouzhouyi@gmail.com>

Thanks for your attention
Thanks
Zhouyi
[1] https://lore.kernel.org/lkml/CAABZP2ywCbu4Po63BDBE7U1WEqx4DF7F2CZjTqFp0dSDw-uziQ@mail.gmail.com/T/#u
>
> Thank you all!
>
> Best Regards
> Zhouyi
> >
> > - Joel

      reply	other threads:[~2023-01-28  1:43 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-01-24 17:31 [PATCH v2 RESEND] tick/nohz: Fix cpu_is_hotpluggable() by checking with nohz subsystem Joel Fernandes (Google)
2023-01-24 22:56 ` Paul E. McKenney
2023-01-25  0:13   ` Zhouyi Zhou
2023-01-25  6:57     ` Joel Fernandes
2023-01-26  4:16     ` Zhouyi Zhou
2023-01-26 15:01       ` Zhouyi Zhou
2023-01-26 15:13         ` Joel Fernandes
2023-01-26 15:41           ` Zhouyi Zhou
2023-01-28  1:42             ` Zhouyi Zhou [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAABZP2zr5oDgOOP0yE+TfwJz4hoAOG+4d5C8MJgs_LuSCS-kpw@mail.gmail.com \
    --to=zhouzhouyi@gmail.com \
    --cc=frederic@kernel.org \
    --cc=fweisbec@gmail.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=joel@joelfernandes.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=maz@kernel.org \
    --cc=mingo@kernel.org \
    --cc=paulmck@kernel.org \
    --cc=rafael@kernel.org \
    --cc=rcu@vger.kernel.org \
    --cc=stable@vger.kernel.org \
    --cc=tglx@linutronix.de \
    --cc=will@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).