[RFC PATCH 0/8] blk-throttle: Throttle buffered WRITE in balance_dirty_pages()

* [RFC PATCH 0/8] blk-throttle: Throttle buffered WRITE in balance_dirty_pages()
@ 2011-06-03 21:06 Vivek Goyal
  2011-06-03 21:06 ` [PATCH 1/8] blk-throttle: convert wait routines to return jiffies to wait Vivek Goyal
                   ` (7 more replies)
  0 siblings, 8 replies; 10+ messages in thread
From: Vivek Goyal @ 2011-06-03 21:06 UTC (permalink / raw)
  To: linux-kernel, linux-fsdevel, axboe
  Cc: vgoyal, arighi, fengguang.wu, jack, akpm

Hi,

I have been trying to find ways to solve two problems with block IO controller
cgroups.

- Current throttling logic in IO controller does not throttle buffered WRITES.
  Well it does throttle all the WRITEs at device and by that time buffered
  WRITE have lost the submitter's context and most of the IO comes in 
  flusher thread's context at device. Hence currently buffered write
  throttling is not supported.

- All WRITEs are throttled at device level and this can easily lead to
  filesystem serialization.

  One simple example is that if a process writes some pages to cache and
  then does fsync(), and process gets throttled then it locks up the
  filesystem. With ext4, I noticed that even a simple "ls" does not make
  progress. The reason boils down to the fact that filesystems are not
  aware of cgroups and one of the things which get serialized is journalling
  in ordered mode.

  So even if we do something to carry submitter's cgroup information
  to device and do throttling there, it will lead to serialization of
  filesystems and is not a good idea.

So how to go about fixing it. There seem to be two options.

- Throttling should still be done at device level. Make filesystems aware
  of cgroups so that multiple transactions can make progress in parallel
  (per cgroup) and there are no shared resources across cgroups in
  filesystems which can lead to serialization.

- Throttle WRITEs while they are entering the cache and not after that.
  Something like balance_dirty_pages(). Direct IO is still throttled
  at device level. That way, we can avoid these journalling related
  serialization issues w.r.t trottling.

  But the big issue with this approach is that we control the IO rate
  entering into the cache and not IO rate at the device. That way it
  can happen that flusher later submits lots of WRITEs to device and
  we will see a periodic IO spike on end node.

  So this mechanism helps a bit but is not the complete solution. It
  can primarily help those folks which have the system resources and
  plenty of IO bandwidth available but they don't want to give it to
  customer because it is not a premium customer etc.

Option 1 seem to be really hard to fix. Filesystems have not been written
keeping cgroups in mind. So I am really skeptical that I can convince file
system designers to make fundamental changes in filesystems and journalling
code to make them cgroup aware.

Hence with this patch series I have implemented option 2. Option 2 is not
the best solution but atleast it gives us some control then not having any
control on buffered writes. Andrea Righi did similar patches in the past
here.

https://lkml.org/lkml/2011/2/28/115

This patch series had issues w.r.t to interaction between bio and task
throttling, so I redid it.

Design
------

IO controller already has the capability to keep track of IO rates of
a group and enqueue the bio in internal queues if group exceeds the
rate and dispatch these bios later.

This patch series also introduce the capability to throttle a dirtying
task in balance_dirty_pages_ratelimited_nr(). Now no WRITES except
direct WRITES will be throttled at device level. If a dirtying task
exceeds its configured IO rate, it is put on a group wait queue and
woken up when it can dirty more pages.

No new interface has been introduced and both direct IO as well as buffered
IO make use of common IO rate limit.

How To
=====
- Create a cgroup and limit it to 1MB/s for writes.
  echo "8:16 1024000" > /cgroup/blk/test1/blkio.throttle.write_bps_device

- Launch dd thread in the cgroup
  dd if=/dev/zero of=zerofile bs=4K count=1K

 1024+0 records in
 1024+0 records out
 4194304 bytes (4.2 MB) copied, 4.00428 s, 1.0 MB/s

Any feedback is welcome.

Thanks
Vivek

Vivek Goyal (8):
  blk-throttle: convert wait routines to return jiffies to wait
  blk-throttle: do not enforce first queued bio check in
    tg_wait_dispatch
  blk-throttle: use IO size and direction as parameters to wait
    routines
  blk-throttle: Specify number of IOs during dispatch update
  blk-throttle: Get rid of extend slice trace message
  blk-throttle: core logic to throttle task while dirtying pages
  blk-throttle: Do not throttle WRITEs at device level except direct IO
  blk-throttle: enable throttling of task while dirtying pages

 block/blk-cgroup.c        |    6 +-
 block/blk-cgroup.h        |    2 +-
 block/blk-throttle.c      |  506 +++++++++++++++++++++++++++++++++++---------
 block/cfq-iosched.c       |    2 +-
 block/cfq.h               |    6 +-
 fs/direct-io.c            |    1 +
 include/linux/blk_types.h |    2 +
 include/linux/blkdev.h    |    5 +
 mm/page-writeback.c       |    3 +
 9 files changed, 421 insertions(+), 112 deletions(-)

-- 
1.7.4.4

^ permalink raw reply	[flat|nested] 10+ messages in thread