From: Kent Overstreet <koverstreet@google.com> To: Vivek Goyal <vgoyal@redhat.com> Cc: Mikulas Patocka <mpatocka@redhat.com>, linux-bcache@vger.kernel.org, linux-kernel@vger.kernel.org, dm-devel@redhat.com, tj@kernel.org, bharrosh@panasas.com, Jens Axboe <axboe@kernel.dk> Subject: Re: [PATCH v7 9/9] block: Avoid deadlocks with bio allocation by stacking drivers Date: Sun, 2 Sep 2012 18:26:42 -0700 [thread overview] Message-ID: <20120903012642.GD20060@moria.home.lan> (raw) In-Reply-To: <20120831150159.GB13483@redhat.com> On Fri, Aug 31, 2012 at 11:01:59AM -0400, Vivek Goyal wrote: > On Thu, Aug 30, 2012 at 06:43:59PM -0700, Kent Overstreet wrote: > > On Thu, Aug 30, 2012 at 06:07:45PM -0400, Vivek Goyal wrote: > > > On Wed, Aug 29, 2012 at 10:13:45AM -0700, Kent Overstreet wrote: > > > > > > [..] > > > > > Performance aside, punting submission to per device worker in case of deep > > > > > stack usage sounds cleaner solution to me. > > > > > > > > Agreed, but performance tends to matter in the real world. And either > > > > way the tricky bits are going to be confined to a few functions, so I > > > > don't think it matters that much. > > > > > > > > If someone wants to code up the workqueue version and test it, they're > > > > more than welcome... > > > > > > Here is one quick and dirty proof of concept patch. It checks for stack > > > depth and if remaining space is less than 20% of stack size, then it > > > defers the bio submission to per queue worker. > > > > I can't think of any correctness issues. I see some stuff that could be > > simplified (blk_drain_deferred_bios() is redundant, just make it a > > wrapper around blk_deffered_bio_work()). > > > > Still skeptical about the performance impact, though - frankly, on some > > of the hardware I've been running bcache on this would be a visible > > performance regression - probably double digit percentages but I'd have > > to benchmark it. That kind of of hardware/usage is not normal today, > > but I've put a lot of work into performance and I don't want to make > > things worse without good reason. > > Would you like to give this patch a quick try and see with bcache on your > hardware how much performance impact do you see. If I can get a test system I can publish numbers setup with a modern kernel, on I will. Will take a bit though. > Given the fact that submission through worker happens only in case of > when stack usage is high, that should reduce the impact of the patch > and common use cases should reamin unaffected. Except depending on how users have their systems configured, it'll either never happen or it'll happen for most every bio. That makes the performance overhead unpredictable, too. > > > > Have you tested/benchmarked it? > > No, I have not. I will run some simple workloads on SSD. Normal SATA ssds are not going to show the overhead - achi is a pig and it'll be lost in the noise. > There are so many places in kernel where worker threads do work on behalf > of each process. I think this is really a minor concern and I would not > be too worried about it. Yeah, but this is somewhat unprecedented in the amount of cpu time you're potentially moving to worker threads. It is a concern. > What is concerning though really is the greater stack usage due to > recursive nature of make_request() and performance impact of deferral > to a worker thread. Your patch shouldn't increase stack usage (at least if your threshold is safe - it's too high as is).
WARNING: multiple messages have this Message-ID (diff)
From: Kent Overstreet <koverstreet-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> To: Vivek Goyal <vgoyal-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> Cc: Mikulas Patocka <mpatocka-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-bcache-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org, tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, bharrosh-C4P08NqkoRlBDgjK7y7TUQ@public.gmane.org, Jens Axboe <axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org> Subject: Re: [PATCH v7 9/9] block: Avoid deadlocks with bio allocation by stacking drivers Date: Sun, 2 Sep 2012 18:26:42 -0700 [thread overview] Message-ID: <20120903012642.GD20060@moria.home.lan> (raw) In-Reply-To: <20120831150159.GB13483-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> On Fri, Aug 31, 2012 at 11:01:59AM -0400, Vivek Goyal wrote: > On Thu, Aug 30, 2012 at 06:43:59PM -0700, Kent Overstreet wrote: > > On Thu, Aug 30, 2012 at 06:07:45PM -0400, Vivek Goyal wrote: > > > On Wed, Aug 29, 2012 at 10:13:45AM -0700, Kent Overstreet wrote: > > > > > > [..] > > > > > Performance aside, punting submission to per device worker in case of deep > > > > > stack usage sounds cleaner solution to me. > > > > > > > > Agreed, but performance tends to matter in the real world. And either > > > > way the tricky bits are going to be confined to a few functions, so I > > > > don't think it matters that much. > > > > > > > > If someone wants to code up the workqueue version and test it, they're > > > > more than welcome... > > > > > > Here is one quick and dirty proof of concept patch. It checks for stack > > > depth and if remaining space is less than 20% of stack size, then it > > > defers the bio submission to per queue worker. > > > > I can't think of any correctness issues. I see some stuff that could be > > simplified (blk_drain_deferred_bios() is redundant, just make it a > > wrapper around blk_deffered_bio_work()). > > > > Still skeptical about the performance impact, though - frankly, on some > > of the hardware I've been running bcache on this would be a visible > > performance regression - probably double digit percentages but I'd have > > to benchmark it. That kind of of hardware/usage is not normal today, > > but I've put a lot of work into performance and I don't want to make > > things worse without good reason. > > Would you like to give this patch a quick try and see with bcache on your > hardware how much performance impact do you see. If I can get a test system I can publish numbers setup with a modern kernel, on I will. Will take a bit though. > Given the fact that submission through worker happens only in case of > when stack usage is high, that should reduce the impact of the patch > and common use cases should reamin unaffected. Except depending on how users have their systems configured, it'll either never happen or it'll happen for most every bio. That makes the performance overhead unpredictable, too. > > > > Have you tested/benchmarked it? > > No, I have not. I will run some simple workloads on SSD. Normal SATA ssds are not going to show the overhead - achi is a pig and it'll be lost in the noise. > There are so many places in kernel where worker threads do work on behalf > of each process. I think this is really a minor concern and I would not > be too worried about it. Yeah, but this is somewhat unprecedented in the amount of cpu time you're potentially moving to worker threads. It is a concern. > What is concerning though really is the greater stack usage due to > recursive nature of make_request() and performance impact of deferral > to a worker thread. Your patch shouldn't increase stack usage (at least if your threshold is safe - it's too high as is).
next prev parent reply other threads:[~2012-09-03 1:27 UTC|newest] Thread overview: 128+ messages / expand[flat|nested] mbox.gz Atom feed top 2012-08-28 17:37 [PATCH v7 0/9] Block cleanups, deadlock fix Kent Overstreet 2012-08-28 17:37 ` [PATCH v7 1/9] block: Generalized bio pool freeing Kent Overstreet 2012-08-28 17:37 ` Kent Overstreet 2012-08-28 17:37 ` [PATCH v7 2/9] dm: Use bioset's front_pad for dm_rq_clone_bio_info Kent Overstreet 2012-08-28 17:37 ` [PATCH v7 3/9] block: Add bio_reset() Kent Overstreet 2012-08-28 17:37 ` Kent Overstreet 2012-08-28 20:31 ` Tejun Heo 2012-08-28 20:31 ` Tejun Heo 2012-08-28 22:17 ` Kent Overstreet 2012-08-28 22:53 ` Kent Overstreet 2012-08-28 22:53 ` Kent Overstreet 2012-09-01 2:23 ` Tejun Heo 2012-09-01 2:23 ` Tejun Heo 2012-09-05 20:13 ` Kent Overstreet 2012-08-28 17:37 ` [PATCH v7 4/9] pktcdvd: Switch to bio_kmalloc() Kent Overstreet 2012-08-28 20:32 ` Tejun Heo 2012-08-28 20:32 ` Tejun Heo 2012-08-28 22:24 ` Kent Overstreet 2012-09-04 9:05 ` Jiri Kosina 2012-09-04 9:05 ` Jiri Kosina 2012-09-05 19:44 ` Kent Overstreet 2012-09-05 19:44 ` Kent Overstreet 2012-08-28 17:37 ` [PATCH v7 5/9] block: Kill bi_destructor Kent Overstreet 2012-08-28 20:36 ` Tejun Heo 2012-08-28 20:36 ` Tejun Heo 2012-08-28 22:07 ` Kent Overstreet 2012-08-28 17:37 ` [PATCH v7 6/9] block: Consolidate bio_alloc_bioset(), bio_kmalloc() Kent Overstreet 2012-08-28 17:37 ` Kent Overstreet 2012-08-28 20:41 ` Tejun Heo 2012-08-28 20:41 ` Tejun Heo 2012-08-28 22:03 ` Kent Overstreet 2012-08-28 22:03 ` Kent Overstreet 2012-09-01 2:17 ` Tejun Heo 2012-08-28 17:37 ` [PATCH v7 7/9] block: Add bio_clone_bioset(), bio_clone_kmalloc() Kent Overstreet 2012-08-28 20:44 ` Tejun Heo 2012-08-28 20:44 ` Tejun Heo 2012-08-28 22:05 ` Kent Overstreet 2012-09-01 2:19 ` Tejun Heo 2012-09-01 2:19 ` Tejun Heo 2012-08-28 17:37 ` [PATCH v7 8/9] block: Reorder struct bio_set Kent Overstreet 2012-08-28 17:37 ` [PATCH v7 9/9] block: Avoid deadlocks with bio allocation by stacking drivers Kent Overstreet 2012-08-28 20:49 ` Tejun Heo 2012-08-28 20:49 ` Tejun Heo 2012-08-28 22:28 ` Kent Overstreet 2012-08-28 23:01 ` Kent Overstreet 2012-08-28 23:01 ` Kent Overstreet 2012-08-29 1:31 ` Vivek Goyal 2012-08-29 1:31 ` Vivek Goyal 2012-08-29 3:25 ` Kent Overstreet 2012-08-29 12:57 ` Vivek Goyal 2012-08-29 12:57 ` Vivek Goyal 2012-08-29 14:39 ` [dm-devel] " Alasdair G Kergon 2012-08-29 14:39 ` Alasdair G Kergon 2012-08-29 16:26 ` Kent Overstreet 2012-08-29 16:26 ` Kent Overstreet 2012-08-29 21:01 ` John Stoffel 2012-08-29 21:08 ` Kent Overstreet 2012-08-29 21:08 ` Kent Overstreet 2012-08-28 22:06 ` Vivek Goyal 2012-08-28 22:06 ` Vivek Goyal 2012-08-28 22:23 ` Kent Overstreet 2012-08-28 22:23 ` Kent Overstreet 2012-08-29 16:24 ` Mikulas Patocka 2012-08-29 16:50 ` Kent Overstreet 2012-08-29 16:57 ` [dm-devel] " Alasdair G Kergon 2012-08-29 16:57 ` Alasdair G Kergon 2012-08-29 17:07 ` Vivek Goyal 2012-08-29 17:07 ` Vivek Goyal 2012-08-29 17:13 ` Kent Overstreet 2012-08-29 17:13 ` Kent Overstreet 2012-08-29 17:23 ` [dm-devel] " Alasdair G Kergon 2012-08-29 17:23 ` Alasdair G Kergon 2012-08-29 17:32 ` Kent Overstreet 2012-08-30 22:07 ` Vivek Goyal 2012-08-30 22:07 ` Vivek Goyal 2012-08-31 1:43 ` Kent Overstreet 2012-08-31 1:43 ` Kent Overstreet 2012-08-31 1:55 ` Kent Overstreet 2012-08-31 1:55 ` Kent Overstreet 2012-08-31 15:01 ` Vivek Goyal 2012-09-03 1:26 ` Kent Overstreet [this message] 2012-09-03 1:26 ` Kent Overstreet 2012-09-03 20:41 ` Mikulas Patocka 2012-09-03 20:41 ` Mikulas Patocka 2012-09-04 3:41 ` Kent Overstreet 2012-09-04 3:41 ` Kent Overstreet 2012-09-04 18:55 ` Tejun Heo 2012-09-04 18:55 ` Tejun Heo 2012-09-04 19:01 ` Tejun Heo 2012-09-04 19:43 ` Kent Overstreet 2012-09-04 19:43 ` Kent Overstreet 2012-09-04 19:42 ` Kent Overstreet 2012-09-04 21:03 ` Tejun Heo 2012-09-04 21:03 ` Tejun Heo 2012-09-04 19:26 ` Mikulas Patocka 2012-09-04 19:26 ` Mikulas Patocka 2012-09-04 19:39 ` Vivek Goyal 2012-09-04 19:39 ` Vivek Goyal 2012-09-04 19:51 ` [PATCH] dm: Use bioset's front_pad for dm_target_io Kent Overstreet 2012-09-04 19:51 ` Kent Overstreet 2012-09-04 21:20 ` Tejun Heo 2012-09-04 21:20 ` Tejun Heo 2012-09-11 19:28 ` [PATCH 2] " Mikulas Patocka 2012-09-11 19:28 ` Mikulas Patocka 2012-09-11 19:50 ` Kent Overstreet 2012-09-11 19:50 ` Kent Overstreet 2012-09-12 22:31 ` Mikulas Patocka 2012-09-12 22:31 ` Mikulas Patocka 2012-09-14 23:09 ` [dm-devel] " Alasdair G Kergon 2012-09-14 23:09 ` Alasdair G Kergon 2012-09-01 2:13 ` [PATCH v7 9/9] block: Avoid deadlocks with bio allocation by stacking drivers Tejun Heo 2012-09-01 2:13 ` Tejun Heo 2012-09-03 1:34 ` [PATCH v2] " Kent Overstreet 2012-09-03 1:34 ` Kent Overstreet 2012-09-04 15:00 ` [PATCH v7 9/9] " Vivek Goyal 2012-09-04 15:00 ` Vivek Goyal 2012-09-03 0:49 ` Dave Chinner 2012-09-03 0:49 ` Dave Chinner 2012-09-03 1:17 ` Kent Overstreet 2012-09-03 1:17 ` Kent Overstreet 2012-09-04 13:54 ` Vivek Goyal 2012-09-04 13:54 ` Vivek Goyal 2012-09-04 18:26 ` Tejun Heo 2012-09-04 18:26 ` Tejun Heo 2012-09-05 3:57 ` Dave Chinner 2012-09-05 3:57 ` Dave Chinner 2012-09-05 4:37 ` Tejun Heo 2012-09-05 4:37 ` Tejun Heo
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20120903012642.GD20060@moria.home.lan \ --to=koverstreet@google.com \ --cc=axboe@kernel.dk \ --cc=bharrosh@panasas.com \ --cc=dm-devel@redhat.com \ --cc=linux-bcache@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=mpatocka@redhat.com \ --cc=tj@kernel.org \ --cc=vgoyal@redhat.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.