From: Minchan Kim <minchan@kernel.org>
To: Luis Chamberlain <mcgrof@kernel.org>
Cc: ngupta@vflare.org, sergey.senozhatsky.work@gmail.com,
axboe@kernel.dk, mbenes@suse.com, jpoimboe@redhat.com,
tglx@linutronix.de, keescook@chromium.org, jikos@kernel.org,
rostedt@goodmis.org, peterz@infradead.org,
linux-block@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 1/4] zram: fix crashes due to use of cpu hotplug multistate
Date: Wed, 19 May 2021 12:54:02 -0700 [thread overview]
Message-ID: <YKVs2pSiTaRSQljC@google.com> (raw)
In-Reply-To: <20210423011108.11988-2-mcgrof@kernel.org>
On Fri, Apr 23, 2021 at 01:11:05AM +0000, Luis Chamberlain wrote:
> Provide a simple state machine to fix races with driver exit where we
> remove the CPU multistate callbacks and re-initialization / creation of
> new per CPU instances which should be managed by these callbacks.
>
> The zram driver makes use of cpu hotplug multistate support, whereby it
> associates a struct zcomp per CPU. Each struct zcomp represents a
> compression algorithm in charge of managing compression streams per CPU.
> Although a compiled zram driver only supports a fixed set of compression
> algorithms, each zram device gets a struct zcomp allocated per CPU. The
> "multi" in CPU hotplug multstate refers to these per cpu struct zcomp
> instances. Each of these will have the CPU hotplug callback called for
> it on CPU plug / unplug. The kernel's CPU hotplug multistate keeps a
> linked list of these different structures so that it will iterate over
> them on CPU transitions.
>
> By default at driver initialization we will create just one zram device
> (num_devices=1) and a zcomp structure then set for the now default
> lzo-rle comrpession algorithm. At driver removal we first remove each
> zram device, and so we destroy the associated struct zcomp per CPU. But
> since we expose sysfs attributes to create new devices or reset / initialize
> existing zram devices, we can easily end up re-initializing a struct zcomp
> for a zram device before the exit routine of the module removes the cpu
> hotplug callback. When this happens the kernel's CPU hotplug will detect
> that at least one instance (struct zcomp for us) exists. This can happen
> in the following situation:
>
> CPU 1 CPU 2
>
> class_unregister(...);
> idr_for_each(...);
> zram_debugfs_destroy();
> disksize_store(...);
> idr_destroy(...);
> unregister_blkdev(...);
> cpuhp_remove_multi_state(...);
>
> The warning comes up on cpuhp_remove_multi_state() when it sees that the
> state for CPUHP_ZCOMP_PREPARE does not have an empty instance linked list.
> In this case, that a struct zcom still exists, the driver allowed its
> creation per CPU even though we could have just freed them per CPU
> though a call on another CPU, and we are then later trying to remove the
> hotplug callback.
>
> Fix all this by providing a zram initialization boolean
> protected the the shared in the driver zram_index_mutex,
> which we can use to annotate when sysfs attributes are
> safe to use or not -- once the driver is properly initialized.
> When the driver is going down we also are sure to not let
> userspace muck with attributes which may affect each per cpu
> struct zcomp.
>
> This also fixes a series of possible memory leaks. The
> crashes and memory leaks can easily be caused by issuing
> the zram02.sh script from the LTP project [0] in a loop
> in two separate windows:
>
> cd testcases/kernel/device-drivers/zram
> while true; do PATH=$PATH:$PWD:$PWD/../../../lib/ ./zram02.sh; done
>
> You end up with a splat as follows:
>
> kernel: zram: Removed device: zram0
> kernel: zram: Added device: zram0
> kernel: zram0: detected capacity change from 0 to 209715200
> kernel: Adding 104857596k swap on /dev/zram0. Priority:-2 extents:1 across:104857596k SSFS
> kernel: zram0: detected capacitky change from 209715200 to 0
> kernel: zram0: detected capacity change from 0 to 209715200
> kernel: ------------[ cut here ]------------
> kernel: Error: Removing state 63 which has instances left.
> kernel: WARNING: CPU: 7 PID: 70457 at kernel/cpu.c:2069 __cpuhp_remove_state_cpuslocked+0xf9/0x100
> kernel: Modules linked in: zram(E-) zsmalloc(E) <etc>
> kernel: CPU: 7 PID: 70457 Comm: rmmod Tainted: G E 5.12.0-rc1-next-20210304 #3
> kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014
> kernel: RIP: 0010:__cpuhp_remove_state_cpuslocked+0xf9/0x100
> kernel: Code: <etc>
> kernel: RSP: 0018:ffffa800c139be98 EFLAGS: 00010282
> kernel: RAX: 0000000000000000 RBX: ffffffff9083db58 RCX: ffff9609f7dd86d8
> kernel: RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff9609f7dd86d0
> kernel: RBP: 0000000000000000i R08: 0000000000000000 R09: ffffa800c139bcb8
> kernel: R10: ffffa800c139bcb0 R11: ffffffff908bea40 R12: 000000000000003f
> kernel: R13: 00000000000009d8 R14: 0000000000000000 R15: 0000000000000000
> kernel: FS: 00007f1b075a7540(0000) GS:ffff9609f7dc0000(0000) knlGS:0000000000000000
> kernel: CS: 0010 DS: 0000 ES 0000 CR0: 0000000080050033
> kernel: CR2: 00007f1b07610490 CR3: 00000001bd04e000 CR4: 0000000000350ee0
> kernel: Call Trace:
> kernel: __cpuhp_remove_state+0x2e/0x80
> kernel: __do_sys_delete_module+0x190/0x2a0
> kernel: do_syscall_64+0x33/0x80
> kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
>
> The "Error: Removing state 63 which has instances left" refers
> to the zram per CPU struc zcomp instances left.
>
> [0] https://github.com/linux-test-project/ltp.git
>
> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
Acked-by: Minchan Kim <minchan@kernel.org>
next prev parent reply other threads:[~2021-05-19 19:54 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-23 1:11 [PATCH v2 0/4] zram: fix few sysfs races Luis Chamberlain
2021-04-23 1:11 ` [PATCH v2 1/4] zram: fix crashes due to use of cpu hotplug multistate Luis Chamberlain
2021-05-19 19:54 ` Minchan Kim [this message]
2021-04-23 1:11 ` [PATCH v2 2/4] zram: avoid disksize setting when device is being claimed Luis Chamberlain
2021-05-19 19:56 ` Minchan Kim
2021-04-23 1:11 ` [PATCH v2 3/4] zram: fix deadlock with sysfs attribute usage and driver removal Luis Chamberlain
2021-04-23 1:11 ` [PATCH v2 4/4] zram: fix possible races between sysfs use and bdev access Luis Chamberlain
2021-05-19 20:09 ` [PATCH v2 0/4] zram: fix few sysfs races Minchan Kim
2021-05-19 20:20 ` Luis Chamberlain
2021-05-21 20:01 ` Greg Kroah-Hartman
2021-05-21 20:16 ` Luis Chamberlain
2021-05-21 20:45 ` Greg Kroah-Hartman
2021-05-21 21:08 ` Luis Chamberlain
2021-05-22 7:48 ` Greg Kroah-Hartman
2021-05-25 1:16 ` Luis Chamberlain
2021-05-25 7:41 ` Greg Kroah-Hartman
2021-06-21 23:19 ` Luis Chamberlain
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YKVs2pSiTaRSQljC@google.com \
--to=minchan@kernel.org \
--cc=axboe@kernel.dk \
--cc=jikos@kernel.org \
--cc=jpoimboe@redhat.com \
--cc=keescook@chromium.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mbenes@suse.com \
--cc=mcgrof@kernel.org \
--cc=ngupta@vflare.org \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=sergey.senozhatsky.work@gmail.com \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).