From: Pavel Tatashin <pasha.tatashin@soleen.com>
To: pasha.tatashin@soleen.com, tyhicks@linux.microsoft.com,
axboe@kernel.dk, linux-block@vger.kernel.org,
linux-kernel@vger.kernel.org, sashal@kernel.org,
jmorris@namei.org, lukas.bulwahn@gmail.com, hch@lst.de,
pvorel@suse.cz, ming.lei@redhat.com, mzxreary@0pointer.de,
mcgrof@kernel.org, zhengbin13@huawei.com, maco@android.com,
colin.king@canonical.com, evgreen@chromium.org
Subject: [PATCH v4 0/1] scale loop device lock
Date: Tue, 26 Jan 2021 09:46:29 -0500 [thread overview]
Message-ID: <20210126144630.230714-1-pasha.tatashin@soleen.com> (raw)
Changelog
v4
- Added review-by Petr Vorel
- Addressed comments from Chaitanya Kulkarni
v3
- Added review-by Tyler
- Sync with mainline
v2
- Addressed Tyler Hicks comments
- added mutex_destroy()
- comment in lo_open()
- added lock around lo_disk
===
In our environment we are using systemd portable containers in
squashfs formats, convert them into loop device, and mount.
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop5 7:5 0 76.4M 0 loop
`-BaseImageM1908 252:3 0 76.4M 1 crypt /BaseImageM1908
loop6 7:6 0 20K 0 loop
`-test_launchperf20 252:17 0 1.3M 1 crypt /app/test_launchperf20
loop7 7:7 0 20K 0 loop
`-test_launchperf18 252:4 0 1.5M 1 crypt /app/test_launchperf18
loop8 7:8 0 8K 0 loop
`-test_launchperf8 252:25 0 28K 1 crypt app/test_launchperf8
loop9 7:9 0 376K 0 loop
`-test_launchperf14 252:29 0 45.7M 1 crypt /app/test_launchperf14
loop10 7:10 0 16K 0 loop
`-test_launchperf4 252:11 0 968K 1 crypt app/test_launchperf4
loop11 7:11 0 1.2M 0 loop
`-test_launchperf17 252:26 0 150.4M 1 crypt /app/test_launchperf17
loop12 7:12 0 36K 0 loop
`-test_launchperf19 252:13 0 3.3M 1 crypt /app/test_launchperf19
loop13 7:13 0 8K 0 loop
...
We have over 50 loop devices which are mounted during boot.
We observed contentions around loop_ctl_mutex.
The sample contentions stacks:
Contention 1:
__blkdev_get()
bdev->bd_disk->fops->open()
lo_open()
mutex_lock_killable(&loop_ctl_mutex); <- contention
Contention 2:
__blkdev_put()
disk->fops->release()
lo_release()
mutex_lock(&loop_ctl_mutex); <- contention
With total time waiting for loop_ctl_mutex ~18.8s during boot (across 8
CPUs) on our machine (69 loop devices): 2.35s per CPU.
Scaling this lock eliminates this contention entirely, and improves the boot
performance by 2s on our machine.
v3 https://lore.kernel.org/lkml/20210125201156.1330164-1-pasha.tatashin@soleen.com
v2 https://lore.kernel.org/lkml/20200723211748.13139-1-pasha.tatashin@soleen.com
v1 https://lore.kernel.org/lkml/20200717205322.127694-1-pasha.tatashin@soleen.com
Pavel Tatashin (1):
loop: scale loop device by introducing per device lock
drivers/block/loop.c | 93 +++++++++++++++++++++++++-------------------
drivers/block/loop.h | 1 +
2 files changed, 54 insertions(+), 40 deletions(-)
--
2.25.1
next reply other threads:[~2021-01-26 14:48 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-26 14:46 Pavel Tatashin [this message]
2021-01-26 14:46 ` [PATCH v4 1/1] loop: scale loop device by introducing per device lock Pavel Tatashin
2021-01-26 20:09 ` Jens Axboe
2021-01-27 16:12 ` Pavel Tatashin
2021-01-26 19:03 ` [PATCH v4 0/1] scale loop " Chaitanya Kulkarni
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210126144630.230714-1-pasha.tatashin@soleen.com \
--to=pasha.tatashin@soleen.com \
--cc=axboe@kernel.dk \
--cc=colin.king@canonical.com \
--cc=evgreen@chromium.org \
--cc=hch@lst.de \
--cc=jmorris@namei.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=lukas.bulwahn@gmail.com \
--cc=maco@android.com \
--cc=mcgrof@kernel.org \
--cc=ming.lei@redhat.com \
--cc=mzxreary@0pointer.de \
--cc=pvorel@suse.cz \
--cc=sashal@kernel.org \
--cc=tyhicks@linux.microsoft.com \
--cc=zhengbin13@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).