From: Ming Lei <ming.lei@redhat.com>
To: Luis Chamberlain <mcgrof@kernel.org>
Cc: tj@kernel.org, gregkh@linuxfoundation.org,
akpm@linux-foundation.org, minchan@kernel.org, jeyu@kernel.org,
shuah@kernel.org, bvanassche@acm.org, dan.j.williams@intel.com,
joe@perches.com, tglx@linutronix.de, keescook@chromium.org,
rostedt@goodmis.org, linux-spdx@vger.kernel.org,
linux-doc@vger.kernel.org, linux-block@vger.kernel.org,
linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org,
linux-kernel@vger.kernel.org, ming.lei@redhat.com
Subject: Re: [PATCH v8 11/12] zram: fix crashes with cpu hotplug multistate
Date: Thu, 14 Oct 2021 09:55:48 +0800 [thread overview]
Message-ID: <YWeOJP2UJWYF94fu@T590> (raw)
In-Reply-To: <20210927163805.808907-12-mcgrof@kernel.org>
On Mon, Sep 27, 2021 at 09:38:04AM -0700, Luis Chamberlain wrote:
> Provide a simple state machine to fix races with driver exit where we
> remove the CPU multistate callbacks and re-initialization / creation of
> new per CPU instances which should be managed by these callbacks.
>
> The zram driver makes use of cpu hotplug multistate support, whereby it
> associates a struct zcomp per CPU. Each struct zcomp represents a
> compression algorithm in charge of managing compression streams per
> CPU. Although a compiled zram driver only supports a fixed set of
> compression algorithms, each zram device gets a struct zcomp allocated
> per CPU. The "multi" in CPU hotplug multstate refers to these per
> cpu struct zcomp instances. Each of these will have the CPU hotplug
> callback called for it on CPU plug / unplug. The kernel's CPU hotplug
> multistate keeps a linked list of these different structures so that
> it will iterate over them on CPU transitions.
>
> By default at driver initialization we will create just one zram device
> (num_devices=1) and a zcomp structure then set for the now default
> lzo-rle comrpession algorithm. At driver removal we first remove each
> zram device, and so we destroy the associated struct zcomp per CPU. But
> since we expose sysfs attributes to create new devices or reset /
> initialize existing zram devices, we can easily end up re-initializing
> a struct zcomp for a zram device before the exit routine of the module
> removes the cpu hotplug callback. When this happens the kernel's CPU
> hotplug will detect that at least one instance (struct zcomp for us)
> exists. This can happen in the following situation:
>
> CPU 1 CPU 2
>
> disksize_store(...);
> class_unregister(...);
> idr_for_each(...);
> zram_debugfs_destroy();
>
> idr_destroy(...);
> unregister_blkdev(...);
> cpuhp_remove_multi_state(...);
>
> The warning comes up on cpuhp_remove_multi_state() when it sees that the
> state for CPUHP_ZCOMP_PREPARE does not have an empty instance linked list.
> In this case, that a struct zcom still exists, the driver allowed its
> creation per CPU even though we could have just freed them per CPU
> though a call on another CPU, and we are then later trying to remove the
> hotplug callback.
>
> Fix all this by providing a zram initialization boolean
> protected the shared in the driver zram_index_mutex, which we
> can use to annotate when sysfs attributes are safe to use or
> not -- once the driver is properly initialized. When the driver
> is going down we also are sure to not let userspace muck with
> attributes which may affect each per cpu struct zcomp.
>
> This also fixes a series of possible memory leaks. The
> crashes and memory leaks can easily be caused by issuing
> the zram02.sh script from the LTP project [0] in a loop
> in two separate windows:
>
> cd testcases/kernel/device-drivers/zram
> while true; do PATH=$PATH:$PWD:$PWD/../../../lib/ ./zram02.sh; done
>
> You end up with a splat as follows:
>
> kernel: zram: Removed device: zram0
> kernel: zram: Added device: zram0
> kernel: zram0: detected capacity change from 0 to 209715200
> kernel: Adding 104857596k swap on /dev/zram0. <etc>
> kernel: zram0: detected capacitky change from 209715200 to 0
> kernel: zram0: detected capacity change from 0 to 209715200
> kernel: ------------[ cut here ]------------
> kernel: Error: Removing state 63 which has instances left.
> kernel: WARNING: CPU: 7 PID: 70457 at \
> kernel/cpu.c:2069 __cpuhp_remove_state_cpuslocked+0xf9/0x100
> kernel: Modules linked in: zram(E-) zsmalloc(E) <etc>
> kernel: CPU: 7 PID: 70457 Comm: rmmod Tainted: G \
> E 5.12.0-rc1-next-20210304 #3
> kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), \
> BIOS 1.14.0-2 04/01/2014
> kernel: RIP: 0010:__cpuhp_remove_state_cpuslocked+0xf9/0x100
> kernel: Code: <etc>
> kernel: RSP: 0018:ffffa800c139be98 EFLAGS: 00010282
> kernel: RAX: 0000000000000000 RBX: ffffffff9083db58 RCX: ffff9609f7dd86d8
> kernel: RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff9609f7dd86d0
> kernel: RBP: 0000000000000000i R08: 0000000000000000 R09: ffffa800c139bcb8
> kernel: R10: ffffa800c139bcb0 R11: ffffffff908bea40 R12: 000000000000003f
> kernel: R13: 00000000000009d8 R14: 0000000000000000 R15: 0000000000000000
> kernel: FS: 00007f1b075a7540(0000) GS:ffff9609f7dc0000(0000) knlGS:<etc>
> kernel: CS: 0010 DS: 0000 ES 0000 CR0: 0000000080050033
> kernel: CR2: 00007f1b07610490 CR3: 00000001bd04e000 CR4: 0000000000350ee0
> kernel: Call Trace:
> kernel: __cpuhp_remove_state+0x2e/0x80
> kernel: __do_sys_delete_module+0x190/0x2a0
> kernel: do_syscall_64+0x33/0x80
> kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
>
> The "Error: Removing state 63 which has instances left" refers
> to the zram per CPU struct zcomp instances left.
>
> [0] https://github.com/linux-test-project/ltp.git
>
> Acked-by: Minchan Kim <minchan@kernel.org>
> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
> ---
Hello Luis,
Can you test the following patch and see if the issue can be addressed?
Please see the idea from the inline comment.
Also zram_index_mutex isn't needed in zram disk's store() compared with
your patch, then the deadlock issue you are addressing in this series can
be avoided.
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index fcaf2750f68f..3c17927d23a7 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -1985,11 +1985,17 @@ static int zram_remove(struct zram *zram)
/* Make sure all the pending I/O are finished */
fsync_bdev(bdev);
- zram_reset_device(zram);
pr_info("Removed device: %s\n", zram->disk->disk_name);
del_gendisk(zram->disk);
+
+ /*
+ * reset device after gendisk is removed, so any change from sysfs
+ * store won't come in, then we can really reset device here
+ */
+ zram_reset_device(zram);
+
blk_cleanup_disk(zram->disk);
kfree(zram);
return 0;
@@ -2073,7 +2079,12 @@ static int zram_remove_cb(int id, void *ptr, void *data)
static void destroy_devices(void)
{
class_unregister(&zram_control_class);
+
+ /* hold the global lock so new device can't be added */
+ mutex_lock(&zram_index_mutex);
idr_for_each(&zram_index_idr, &zram_remove_cb, NULL);
+ mutex_unlock(&zram_index_mutex);
+
zram_debugfs_destroy();
idr_destroy(&zram_index_idr);
unregister_blkdev(zram_major, "zram");
Thanks,
Ming
next prev parent reply other threads:[~2021-10-14 1:56 UTC|newest]
Thread overview: 94+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-27 16:37 [PATCH v8 00/12] syfs: generic deadlock fix with module removal Luis Chamberlain
2021-09-27 16:37 ` [PATCH v8 01/12] LICENSES: Add the copyleft-next-0.3.1 license Luis Chamberlain
[not found] ` <202110050907.35FBD2A1@keescook>
[not found] ` <YWR2ZrtzChamY1y4@bombadil.infradead.org>
2021-10-11 17:57 ` Kees Cook
2021-09-27 16:37 ` [PATCH v8 02/12] testing: use the copyleft-next-0.3.1 SPDX tag Luis Chamberlain
2021-10-05 16:11 ` Kees Cook
2021-09-27 16:37 ` [PATCH v8 03/12] selftests: add tests_sysfs module Luis Chamberlain
2021-10-05 14:16 ` Greg KH
2021-10-05 16:57 ` Tim.Bird
2021-10-11 17:40 ` Luis Chamberlain
2021-10-11 17:38 ` Luis Chamberlain
2021-10-07 14:23 ` Miroslav Benes
2021-10-11 19:11 ` Luis Chamberlain
[not found] ` <202110050912.3DF681ED@keescook>
2021-10-11 19:03 ` Luis Chamberlain
2021-09-27 16:37 ` [PATCH v8 04/12] kernfs: add initial failure injection support Luis Chamberlain
2021-10-05 19:47 ` Kees Cook
2021-10-11 20:44 ` Luis Chamberlain
2021-09-27 16:37 ` [PATCH v8 05/12] test_sysfs: add support to use kernfs failure injection Luis Chamberlain
2021-10-05 19:51 ` Kees Cook
2021-10-11 20:56 ` Luis Chamberlain
2021-09-27 16:37 ` [PATCH v8 06/12] kernel/module: add documentation for try_module_get() Luis Chamberlain
2021-10-05 19:58 ` Kees Cook
2021-10-11 21:16 ` Luis Chamberlain
2021-09-27 16:38 ` [PATCH v8 07/12] fs/kernfs/symlink.c: replace S_IRWXUGO with 0777 on kernfs_create_link() Luis Chamberlain
2021-10-05 19:59 ` Kees Cook
2021-09-27 16:38 ` [PATCH v8 08/12] fs/sysfs/dir.c: replace S_IRWXU|S_IRUGO|S_IXUGO with 0755 sysfs_create_dir_ns() Luis Chamberlain
2021-10-05 16:05 ` Kees Cook
2021-09-27 16:38 ` [PATCH v8 09/12] sysfs: fix deadlock race with module removal Luis Chamberlain
2021-10-05 9:24 ` Ming Lei
2021-10-11 21:25 ` Luis Chamberlain
2021-10-12 0:20 ` Ming Lei
2021-10-12 21:18 ` Luis Chamberlain
2021-10-13 1:07 ` Ming Lei
2021-10-13 12:35 ` Luis Chamberlain
2021-10-13 15:04 ` Ming Lei
2021-10-13 21:16 ` Luis Chamberlain
2021-10-05 20:50 ` Kees Cook
2021-10-11 22:26 ` Luis Chamberlain
2021-10-13 12:41 ` Luis Chamberlain
2021-09-27 16:38 ` [PATCH v8 10/12] test_sysfs: enable deadlock tests by default Luis Chamberlain
2021-09-27 16:38 ` [PATCH v8 11/12] zram: fix crashes with cpu hotplug multistate Luis Chamberlain
2021-10-05 20:55 ` Kees Cook
2021-10-11 18:27 ` Luis Chamberlain
2021-10-14 1:55 ` Ming Lei [this message]
2021-10-14 2:11 ` Ming Lei
2021-10-14 20:24 ` Luis Chamberlain
2021-10-14 23:52 ` Ming Lei
2021-10-15 0:22 ` Luis Chamberlain
2021-10-15 8:36 ` Ming Lei
2021-10-15 8:52 ` Greg KH
2021-10-15 17:31 ` Luis Chamberlain
2021-10-16 11:28 ` Ming Lei
2021-10-18 19:32 ` Luis Chamberlain
2021-10-19 2:34 ` Ming Lei
2021-10-19 6:23 ` Miroslav Benes
2021-10-19 9:23 ` Ming Lei
2021-10-20 6:43 ` Miroslav Benes
2021-10-20 7:49 ` Ming Lei
2021-10-20 8:19 ` Miroslav Benes
2021-10-20 8:28 ` Greg KH
2021-10-25 9:58 ` Miroslav Benes
2021-10-20 10:09 ` Ming Lei
2021-10-26 8:48 ` Petr Mladek
2021-10-26 15:37 ` Ming Lei
2021-10-26 17:01 ` Luis Chamberlain
2021-10-27 11:57 ` Miroslav Benes
2021-10-27 14:27 ` Luis Chamberlain
2021-11-02 15:24 ` Petr Mladek
2021-11-02 16:25 ` Luis Chamberlain
2021-11-03 0:01 ` Ming Lei
2021-11-03 12:44 ` Luis Chamberlain
2021-10-27 11:42 ` Miroslav Benes
2021-11-02 14:15 ` Petr Mladek
2021-11-02 14:51 ` Petr Mladek
2021-11-02 15:17 ` Ming Lei
2021-11-02 14:56 ` Ming Lei
2021-10-19 15:28 ` Luis Chamberlain
2021-10-19 16:29 ` Ming Lei
2021-10-19 19:36 ` Luis Chamberlain
2021-10-20 1:15 ` Ming Lei
2021-10-20 15:48 ` Luis Chamberlain
2021-10-21 0:39 ` Ming Lei
2021-10-21 17:18 ` Luis Chamberlain
2021-10-22 0:05 ` Ming Lei
2021-10-19 15:50 ` Luis Chamberlain
2021-10-19 16:25 ` Greg KH
2021-10-19 16:30 ` Luis Chamberlain
2021-10-19 17:28 ` Greg KH
2021-10-19 19:46 ` Luis Chamberlain
2021-10-19 16:39 ` Ming Lei
2021-10-19 19:38 ` Luis Chamberlain
2021-10-20 0:55 ` Ming Lei
2021-09-27 16:38 ` [PATCH v8 12/12] zram: use ATTRIBUTE_GROUPS to fix sysfs deadlock module removal Luis Chamberlain
2021-10-05 20:57 ` Kees Cook
2021-10-11 18:28 ` Luis Chamberlain
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YWeOJP2UJWYF94fu@T590 \
--to=ming.lei@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=bvanassche@acm.org \
--cc=dan.j.williams@intel.com \
--cc=gregkh@linuxfoundation.org \
--cc=jeyu@kernel.org \
--cc=joe@perches.com \
--cc=keescook@chromium.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-spdx@vger.kernel.org \
--cc=mcgrof@kernel.org \
--cc=minchan@kernel.org \
--cc=rostedt@goodmis.org \
--cc=shuah@kernel.org \
--cc=tglx@linutronix.de \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).