From: Ming Lei <ming.lei@redhat.com>
To: Luis Chamberlain <mcgrof@kernel.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>,
Paul Mackerras <paulus@samba.org>,
tj@kernel.org, gregkh@linuxfoundation.org,
akpm@linux-foundation.org, minchan@kernel.org, jeyu@kernel.org,
shuah@kernel.org, bvanassche@acm.org, dan.j.williams@intel.com,
joe@perches.com, tglx@linutronix.de, keescook@chromium.org,
rostedt@goodmis.org, linux-spdx@vger.kernel.org,
linux-doc@vger.kernel.org, linux-block@vger.kernel.org,
linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH v8 11/12] zram: fix crashes with cpu hotplug multistate
Date: Wed, 20 Oct 2021 00:39:22 +0800 [thread overview]
Message-ID: <YW70uklcBWrmJIn8@T590> (raw)
In-Reply-To: <YW7pQKi8AlV+ZemU@bombadil.infradead.org>
On Tue, Oct 19, 2021 at 08:50:24AM -0700, Luis Chamberlain wrote:
> On Tue, Oct 19, 2021 at 10:34:41AM +0800, Ming Lei wrote:
> > On Mon, Oct 18, 2021 at 12:32:11PM -0700, Luis Chamberlain wrote:
> > > On Sat, Oct 16, 2021 at 07:28:39PM +0800, Ming Lei wrote:
> > > > On Fri, Oct 15, 2021 at 10:31:31AM -0700, Luis Chamberlain wrote:
> > > > > On Fri, Oct 15, 2021 at 04:36:11PM +0800, Ming Lei wrote:
> > > > > > On Thu, Oct 14, 2021 at 05:22:40PM -0700, Luis Chamberlain wrote:
> > > > > > > On Fri, Oct 15, 2021 at 07:52:04AM +0800, Ming Lei wrote:
> > > > > > ...
> > > > > > > >
> > > > > > > > We need to understand the exact reason why there is still cpuhp node
> > > > > > > > left, can you share us the exact steps for reproducing the issue?
> > > > > > > > Otherwise we may have to trace and narrow down the reason.
> > > > > > >
> > > > > > > See my commit log for my own fix for this issue.
> > > > > >
> > > > > > OK, thanks!
> > > > > >
> > > > > > I can reproduce the issue, and the reason is that reset_store fails
> > > > > > zram_remove() when unloading module, then the warning is caused.
> > > > > >
> > > > > > The top 3 patches in the following tree can fix the issue:
> > > > > >
> > > > > > https://github.com/ming1/linux/commits/my_v5.15-blk-dev
> > > > >
> > > > > Thanks for trying an alternative fix! A crash stops yes, however this
> > > >
> > > > I doubt it is alternative since your patchset doesn't mention the exact
> > > > reason of 'Error: Removing state 63 which has instances left.', that is
> > > > simply caused by failing to remove zram because ->claim is set during
> > > > unloading module.
> > >
> > > Well I disagree because it does explain how the race can happen, and it
> > > also explains how since the sysfs interface is exposed until module
> > > removal completes, it leaves exposed knobs to allow re-initializing of a
> > > struct zcomp for a zram device before the exit.
> > >
> > > > Yeah, you mentioned the race between disksize_store() vs. zram_remove(),
> > > > however I don't think it is reproduced easily in the test because the race
> > > > window is pretty small, also it can be fixed easily in my 3rd path
> > > > without any complicated tricks.
> > >
> > > Reproducing for me is... extremely easy.
> >
> > In my observation, failing zram_remove() is extremely easy to trigger, which
> > is caused by reset_store() which sets ->reclaim as true, so
> > zram_remove() is failed and zram_reset_device() is bypassed , then the
> > failure of 'Error: Removing state 63 which has instances left.' is caused.
> >
> > We are in same page?
>
> The actual first issue is the CPU hotplug remove callback is long gone and
> in the meantime we allow a race to add a new "instance", in the zram
> driver's case a cpu struct zcomp instance though the sysfs interface.
>
> Regardless of if zram_remove() can fail or not, the above race needs to
> be addressed.
>
> > > > Not dig into details of your patchset via grabbing module reference
> > > > count during show/store attribute of kernfs which is done in your patch
> > > > 9, but IMO this way isn't necessary:
> > >
> > > That's to address the deadlock only.
> > >
> > > > 1) any driver module has to cleanup anything which may refer to symbols
> > > > or data defined in module_exit of this driver
> > >
> > > Yes, and as the cpu multistate hotplug documentation warns (although
> > > such documentation is kind of hidden) that driver authors need to be
> > > careful with module removal too, refer to the warning at the end of
> > > __cpuhp_remove_state_cpuslocked() about module removal.
> >
> > It is zram's bug. zram has to clean everything in module_exit(),
> > unfortunately zram_remove() can be failed when calling from
> > module_exit() because ->claim is set as true by reset_store(), then
> > zram_reset_device()(->zcomp_destroy) isn't called, and this failure should
> > not happen when unloading module, should it?
>
> You're addressing a possible failig zram_remove() while I address not
> allowing entry to muck with the zram driver at all once we're bailing
> on module removal.
>
> > > > 2) device_del() is often done in module_exit(), once device_del()
> > > > returns, no any new show/store on the device's kobject attribute
> > > > is possible.
> > >
> > > Right and if a syfs knob is exposed before device_del() completely
> > > and is allowed to do things, the driver should take care to prevent
> > > races for CPU multistate support. The small state machine I added ensures
> >
> > What is the race for CPU multistate support? If you mean 'Error: Removing
> > state 63 which has instances left.', it is zram's bug since zram has to
> > cleanup everything in module_exit().
>
> Yes. And it is what my out of tree yet Acked patch, 'zram: fix
> crashes with cpu hotplug multistate' does.
Unfortunately that patch adds new deadlock between hot_remove_store() and
disksize_store() & others, see my below comment.
>
> > > we don't run over any expectations from cpu hotplug multistate support.
> > >
> > > I've *never* suggested there cannot be alternatives to my solution with
> > > the small state machine, but for you to say it is incorrect is simply
> > > not right either.
> > >
> > > > 3) it is _not_ a must or pattern for fixing bugs to hold one lock before
> > > > calling device_del(), meantime the lock is required in the device's
> > > > attribute show()/store(), which causes AA deadlock easily. Your approach
> > > > just avoids the issue by not releasing module until all show/store are
> > > > done.
> > >
> > > Right, there are two approaches here:
> > >
> > > a) Your approach is to accept the deadlock as a requirement and so
> > > you would prefer to implement an alternative to using a shared lock
> > > on module exit and sysfs op.
> >
> > wrt. in-tree zram, there is neither any deadlock in linus tree, nor after
> > applying my 3 patches. If you think there is, please share us the code
> > or lockdep warning.
>
> Right, 'zram: fix crashes with cpu hotplug multistate' is not yet
> merged, my approach to fixing that does add a lock use on module removal
> which does introduce a possible deadlock with syfs, which is later addressed
> generically between sysfs and module removal for all drivers.
>
> > > b) While I address such a deadlock head on as I think this sort of locking
> > > be allowed for two reasons:
> > > b1) as we never documented such requirement otherwise.
> > > b2) There is a possibility that other drivers already exist too
> > > which *do* use a shared lock on module removal and sysfs ops
> > > (and I just confirmed this to be true)
> >
> > The 'deadlock' is actually caused by your out-of-tree patch of 'zram: fix
> > crashes with cpu hotplug multistate' which adds mutex_lock(zram_index_mutex)
> > in destroy_devices().
>
> Yes yes, but you are completely throwing out the window that other
> possible deadlocks can exist in the kernel *and* that *new* cases of
> the deadlock can easily also be added!
>
> > We can fix this issue easily without needing the global lock, please see the
> > attached(pre-V2) patch.
>
> So far your patches do not fix the issues though...
>
> > > So I *really* don't think it is wise for us to simply accept this new
> > > found deadlock as a *new* requirement, specially if we can fix it easily.
> > >
> > > A cursory review using Coccinelle potential issues with mutex lock
> > > directly used on module exit (so this doesn't cover drivers like zram
> > > which uses a routine and then grabs the lock through indirection) and a
> > > sysfs op shows these drivers are also affected by this deadlock:
> > >
> > > * arch/powerpc/sysdev/fsl_mpic_timer_wakeup.c
> >
> > In fsl_wakeup_sys_exit(), device_remove_file() is called before
> > acquiring &sysfs_lock, so there shouldn't be such AA deadlock.
> >
> > > * lib/test_firmware.c
> >
> > Yeah, there is the AA deadlock risk, but it should be fixed by moving
> > misc_deregister() out of &test_fw_mutex.
>
> And just like that you are ignoring other possible uses in the kernel
> which might have similar deadlocks.
>
> So do you want to take the position:
>
> Hey driver authors: you cannot use any shared lock on module removal and
> on sysfs ops?
IMO, yes, in your patch of 'zram: fix crashes with cpu hotplug multistate',
when you added mutex_lock(zram_index_mutex) to disksize_store() and
other attribute show() or store() method. You have added new deadlock
between hot_remove_store() and disksize_store() & others, which can't be
addressed by your approach of holding module refcnt.
So far not see ltp tests covers hot add/remove interface yet.
Thanks,
Ming
next prev parent reply other threads:[~2021-10-19 16:40 UTC|newest]
Thread overview: 94+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-27 16:37 [PATCH v8 00/12] syfs: generic deadlock fix with module removal Luis Chamberlain
2021-09-27 16:37 ` [PATCH v8 01/12] LICENSES: Add the copyleft-next-0.3.1 license Luis Chamberlain
[not found] ` <202110050907.35FBD2A1@keescook>
[not found] ` <YWR2ZrtzChamY1y4@bombadil.infradead.org>
2021-10-11 17:57 ` Kees Cook
2021-09-27 16:37 ` [PATCH v8 02/12] testing: use the copyleft-next-0.3.1 SPDX tag Luis Chamberlain
2021-10-05 16:11 ` Kees Cook
2021-09-27 16:37 ` [PATCH v8 03/12] selftests: add tests_sysfs module Luis Chamberlain
2021-10-05 14:16 ` Greg KH
2021-10-05 16:57 ` Tim.Bird
2021-10-11 17:40 ` Luis Chamberlain
2021-10-11 17:38 ` Luis Chamberlain
2021-10-07 14:23 ` Miroslav Benes
2021-10-11 19:11 ` Luis Chamberlain
[not found] ` <202110050912.3DF681ED@keescook>
2021-10-11 19:03 ` Luis Chamberlain
2021-09-27 16:37 ` [PATCH v8 04/12] kernfs: add initial failure injection support Luis Chamberlain
2021-10-05 19:47 ` Kees Cook
2021-10-11 20:44 ` Luis Chamberlain
2021-09-27 16:37 ` [PATCH v8 05/12] test_sysfs: add support to use kernfs failure injection Luis Chamberlain
2021-10-05 19:51 ` Kees Cook
2021-10-11 20:56 ` Luis Chamberlain
2021-09-27 16:37 ` [PATCH v8 06/12] kernel/module: add documentation for try_module_get() Luis Chamberlain
2021-10-05 19:58 ` Kees Cook
2021-10-11 21:16 ` Luis Chamberlain
2021-09-27 16:38 ` [PATCH v8 07/12] fs/kernfs/symlink.c: replace S_IRWXUGO with 0777 on kernfs_create_link() Luis Chamberlain
2021-10-05 19:59 ` Kees Cook
2021-09-27 16:38 ` [PATCH v8 08/12] fs/sysfs/dir.c: replace S_IRWXU|S_IRUGO|S_IXUGO with 0755 sysfs_create_dir_ns() Luis Chamberlain
2021-10-05 16:05 ` Kees Cook
2021-09-27 16:38 ` [PATCH v8 09/12] sysfs: fix deadlock race with module removal Luis Chamberlain
2021-10-05 9:24 ` Ming Lei
2021-10-11 21:25 ` Luis Chamberlain
2021-10-12 0:20 ` Ming Lei
2021-10-12 21:18 ` Luis Chamberlain
2021-10-13 1:07 ` Ming Lei
2021-10-13 12:35 ` Luis Chamberlain
2021-10-13 15:04 ` Ming Lei
2021-10-13 21:16 ` Luis Chamberlain
2021-10-05 20:50 ` Kees Cook
2021-10-11 22:26 ` Luis Chamberlain
2021-10-13 12:41 ` Luis Chamberlain
2021-09-27 16:38 ` [PATCH v8 10/12] test_sysfs: enable deadlock tests by default Luis Chamberlain
2021-09-27 16:38 ` [PATCH v8 11/12] zram: fix crashes with cpu hotplug multistate Luis Chamberlain
2021-10-05 20:55 ` Kees Cook
2021-10-11 18:27 ` Luis Chamberlain
2021-10-14 1:55 ` Ming Lei
2021-10-14 2:11 ` Ming Lei
2021-10-14 20:24 ` Luis Chamberlain
2021-10-14 23:52 ` Ming Lei
2021-10-15 0:22 ` Luis Chamberlain
2021-10-15 8:36 ` Ming Lei
2021-10-15 8:52 ` Greg KH
2021-10-15 17:31 ` Luis Chamberlain
2021-10-16 11:28 ` Ming Lei
2021-10-18 19:32 ` Luis Chamberlain
2021-10-19 2:34 ` Ming Lei
2021-10-19 6:23 ` Miroslav Benes
2021-10-19 9:23 ` Ming Lei
2021-10-20 6:43 ` Miroslav Benes
2021-10-20 7:49 ` Ming Lei
2021-10-20 8:19 ` Miroslav Benes
2021-10-20 8:28 ` Greg KH
2021-10-25 9:58 ` Miroslav Benes
2021-10-20 10:09 ` Ming Lei
2021-10-26 8:48 ` Petr Mladek
2021-10-26 15:37 ` Ming Lei
2021-10-26 17:01 ` Luis Chamberlain
2021-10-27 11:57 ` Miroslav Benes
2021-10-27 14:27 ` Luis Chamberlain
2021-11-02 15:24 ` Petr Mladek
2021-11-02 16:25 ` Luis Chamberlain
2021-11-03 0:01 ` Ming Lei
2021-11-03 12:44 ` Luis Chamberlain
2021-10-27 11:42 ` Miroslav Benes
2021-11-02 14:15 ` Petr Mladek
2021-11-02 14:51 ` Petr Mladek
2021-11-02 15:17 ` Ming Lei
2021-11-02 14:56 ` Ming Lei
2021-10-19 15:28 ` Luis Chamberlain
2021-10-19 16:29 ` Ming Lei
2021-10-19 19:36 ` Luis Chamberlain
2021-10-20 1:15 ` Ming Lei
2021-10-20 15:48 ` Luis Chamberlain
2021-10-21 0:39 ` Ming Lei
2021-10-21 17:18 ` Luis Chamberlain
2021-10-22 0:05 ` Ming Lei
2021-10-19 15:50 ` Luis Chamberlain
2021-10-19 16:25 ` Greg KH
2021-10-19 16:30 ` Luis Chamberlain
2021-10-19 17:28 ` Greg KH
2021-10-19 19:46 ` Luis Chamberlain
2021-10-19 16:39 ` Ming Lei [this message]
2021-10-19 19:38 ` Luis Chamberlain
2021-10-20 0:55 ` Ming Lei
2021-09-27 16:38 ` [PATCH v8 12/12] zram: use ATTRIBUTE_GROUPS to fix sysfs deadlock module removal Luis Chamberlain
2021-10-05 20:57 ` Kees Cook
2021-10-11 18:28 ` Luis Chamberlain
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YW70uklcBWrmJIn8@T590 \
--to=ming.lei@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=benh@kernel.crashing.org \
--cc=bvanassche@acm.org \
--cc=dan.j.williams@intel.com \
--cc=gregkh@linuxfoundation.org \
--cc=jeyu@kernel.org \
--cc=joe@perches.com \
--cc=keescook@chromium.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-spdx@vger.kernel.org \
--cc=mcgrof@kernel.org \
--cc=minchan@kernel.org \
--cc=paulus@samba.org \
--cc=rostedt@goodmis.org \
--cc=shuah@kernel.org \
--cc=tglx@linutronix.de \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).