[GIT PULL] Core block IO bits for 4.2

[GIT PULL] Core block IO bits for 4.2

[Jens Axboe]

Hi Linus,

This is the core block pull request for 4.2. Nothing really major in
here, mostly a collection of smaller optimizations and cleanups, mixed
with various fixes. In more detail, this pull request contains:

- Addition of policy specific data to blkcg for block cgroups. From
  Arianna Avanzini.

- Various cleanups around command types from Christoph.

- Cleanup of the suspend block I/O path from Christoph.

- Plugging updates from Shaohua and Jeff Moyer, for blk-mq.

- Eliminating atomic inc/dec of both remaining IO count and reference
  count in a bio. From me.

- Fixes for SG gap and chunk size support for data-less (discards) IO,
  so we can merge these better. From me.

- Small restructuring of blk-mq shared tag support, freeing drivers from
  iterating hardware queues. From Keith Busch.

- A few cfq-iosched tweaks, from Tahsin Erdogan and me. Makes the IOPS
  mode the default for non-rotational storage.


Please pull!


  git://git.kernel.dk/linux-block.git for-4.2/core


----------------------------------------------------------------
Arianna Avanzini (1):
      block, cgroup: implement policy-specific per-blkcg data

Christoph Hellwig (11):
      block: rename REQ_TYPE_SPECIAL to REQ_TYPE_DRV_PRIV
      block: move REQ_TYPE_ATA_TASKFILE and REQ_TYPE_ATA_PC to ide.h
      block: move REQ_TYPE_SENSE to the ide driver
      block: remove REQ_TYPE_PM_SHUTDOWN
      block: move PM request support to IDE
      nbd: stop using req->cmd
      block: use an atomic_t for mq_freeze_depth
      block: remove BIO_EOPNOTSUPP
      block: remove unused BIO_RW_BLOCK and BIO_EOF flags
      suspend: simplify block I/O handling
      block, dm: don't copy bios for request clones

Jarod Wilson (1):
      block: export blkdev_reread_part() and __blkdev_reread_part()

Jeff Moyer (1):
      blk-mq: fix plugging in blk_sq_make_request

Jens Axboe (10):
      bio: skip atomic inc/dec of ->bi_remaining for non-chains
      bio: skip atomic inc/dec of ->bi_cnt for most use cases
      block: collapse bio bit space
      block: only honor SG gap prevention for merges that contain data
      block: don't honor chunk sizes for data-less IO
      block: add blk_set_queue_dying() to blkdev.h
      cfq-iosched: fix the setting of IOPS mode on SSDs
      cfq-iosched: move group scheduling functions under ifdef
      cfq-iosched: fix sysfs oops when attempting to read unconfigured weights
      cfq-iosched: fix other locations where blkcg_to_cfqgd() can return NULL

Julia Lawall (1):
      block: fix returnvar.cocci warnings

Keith Busch (1):
      blk-mq: Shared tag enhancements

Mike Snitzer (1):
      block: remove management of bi_remaining when restoring original bi_end_io

Ming Lei (1):
      block: replace trylock with mutex_lock in blkdev_reread_part()

Shaohua Li (5):
      blk: clean up plug
      sched: always use blk_schedule_flush_plug in io_schedule_out
      blk-mq: avoid re-initialize request which is failed in direct dispatch
      blk-mq: do limited block plug for multiple queue case
      blk-mq: make plug work for mutiple disks and queues

Steven Rostedt (1):
      blktrace: Add blktrace.c to BLOCK LAYER in MAINTAINERS file

Tahsin Erdogan (1):
      block: Make CFQ default to IOPS mode on SSDs

 MAINTAINERS                  |   1 +
 block/bio-integrity.c        |   4 +-
 block/bio.c                  |  77 +++++++++++--------
 block/blk-cgroup.c           |  92 ++++++++++++++++++++---
 block/blk-cgroup.h           |  40 ++++++++--
 block/blk-core.c             | 136 +++++++++-------------------------
 block/blk-exec.c             |  10 ---
 block/blk-merge.c            |   3 +-
 block/blk-mq-tag.c           |  38 ++++++++++
 block/blk-mq-tag.h           |   1 +
 block/blk-mq.c               | 160 +++++++++++++++++++++++++---------------
 block/blk.h                  |   5 +-
 block/bounce.c               |   3 -
 block/cfq-iosched.c          | 125 +++++++++++++++++++++++++------
 block/elevator.c             |   2 +
 block/ioctl.c                |  37 ++++++++--
 drivers/block/nbd.c          |  50 ++++++-------
 drivers/block/paride/pd.c    |   4 +-
 drivers/block/sx8.c          |   4 +-
 drivers/block/virtio_blk.c   |   6 +-
 drivers/ide/ide-atapi.c      |  10 +--
 drivers/ide/ide-cd.c         |  10 +--
 drivers/ide/ide-cd_ioctl.c   |   2 +-
 drivers/ide/ide-devsets.c    |   2 +-
 drivers/ide/ide-eh.c         |   4 +-
 drivers/ide/ide-floppy.c     |   8 +-
 drivers/ide/ide-io.c         |  12 +--
 drivers/ide/ide-ioctls.c     |   2 +-
 drivers/ide/ide-park.c       |   4 +-
 drivers/ide/ide-pm.c         |  56 ++++++++++----
 drivers/ide/ide-tape.c       |   6 +-
 drivers/ide/ide-taskfile.c   |   2 +-
 drivers/md/bcache/io.c       |   2 +-
 drivers/md/bcache/request.c  |   2 +-
 drivers/md/dm-cache-target.c |   6 --
 drivers/md/dm-raid1.c        |   2 -
 drivers/md/dm-snap.c         |   1 -
 drivers/md/dm-table.c        |  25 ++++---
 drivers/md/dm-thin.c         |   9 +--
 drivers/md/dm-verity.c       |   2 +-
 drivers/md/dm.c              | 171 ++++++++++---------------------------------
 drivers/md/dm.h              |   5 +-
 fs/btrfs/disk-io.c           |  13 +---
 fs/btrfs/extent_io.c         |   2 -
 fs/btrfs/volumes.c           |  18 ++---
 fs/btrfs/volumes.h           |   2 -
 fs/buffer.c                  |  13 +---
 fs/ext4/page-io.c            |   1 -
 fs/nilfs2/segbuf.c           |  12 ---
 fs/xfs/xfs_aops.c            |   1 -
 include/linux/bio.h          |  17 ++++-
 include/linux/blk-mq.h       |   4 +
 include/linux/blk_types.h    |  25 ++++---
 include/linux/blkdev.h       |  45 ++----------
 include/linux/elevator.h     |   2 +
 include/linux/fs.h           |   3 +
 include/linux/ide.h          |  27 +++++++
 include/linux/swap.h         |   1 -
 include/uapi/linux/nbd.h     |   2 -
 kernel/power/Makefile        |   3 +-
 kernel/power/block_io.c      | 103 --------------------------
 kernel/power/power.h         |   9 ---
 kernel/power/swap.c          | 159 ++++++++++++++++++++++++++++++----------
 kernel/sched/core.c          |   5 +-
 mm/page_io.c                 |   2 +-
 65 files changed, 853 insertions(+), 757 deletions(-)
 delete mode 100644 kernel/power/block_io.c

-- 
Jens Axboe

Re: [GIT PULL] Core block IO bits for 4.2

[Junichi Nomura]

Hi Jens,

On 06/25/15 23:37, Jens Axboe wrote:
>       block, dm: don't copy bios for request clones

this change should not be pushed to mainline yet.

Firstly, Christoph has a newer version of the patch that fixes silent
data corruption problem:
  https://www.redhat.com/archives/dm-devel/2015-May/msg00229.html

And the new version still depends on LLDDs to always complete requests
to the end when error happens, while block API doesn't enforce such a
requirement. If the assumption is ever broken, the inconsistency between
request and bio (e.g. rq->__sector and rq->bio) will cause silent data
corruption:
  https://www.redhat.com/archives/dm-devel/2015-June/msg00022.html

-- 
Jun'ichi Nomura, NEC Corporation

Re: [GIT PULL] Core block IO bits for 4.2

[Linus Torvalds]

[-- Attachment #1.1: Type: text/plain, Size: 330 bytes --]

On Jun 25, 2015 23:17, "Junichi Nomura" <j-nomura@ce.jp.nec.com> wrote:
>
> Hi Jens,
>
> On 06/25/15 23:37, Jens Axboe wrote:
> >       block, dm: don't copy bios for request clones
>
> this change should not be pushed to mainline yet.

Hmm. I merged all of Jens' pull requests today, so it's now in my tree.
What do?

     Linus

[-- Attachment #1.2: Type: text/html, Size: 527 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [GIT PULL] Core block IO bits for 4.2

[Junichi Nomura]

On 06/26/15 15:41, Linus Torvalds wrote:
> On Jun 25, 2015 23:17, "Junichi Nomura" <j-nomura@ce.jp.nec.com <mailto:j-nomura@ce.jp.nec.com>> wrote:
>> On 06/25/15 23:37, Jens Axboe wrote:
>> >       block, dm: don't copy bios for request clones
>>
>> this change should not be pushed to mainline yet.
> 
> Hmm. I merged all of Jens' pull requests today, so it's now in my tree. What do?

Then I think following 3 commits should be reverted.

2d76fff dm: cleanup methods that requeue requests
cbc4e3c dm: do not allocate any mempools for blk-mq request-based DM
5f1b670 block, dm: don't copy bios for request clones

Reverting 5f1b670 involves conflicts in drvers/md/dm.c for
these 2 hunks due to diff contexts difference.
It's ok to revert those changes ignoring the contexts.

  @@ -1089,8 +1038,6 @@ static void free_rq_clone(struct request *clone, bool must_be_mapped)
   
          WARN_ON_ONCE(must_be_mapped && !clone->q);
   
  -       blk_rq_unprep_clone(clone);
  -
          if (md->type == DM_TYPE_MQ_REQUEST_BASED)
                  /* stacked on blk-mq queue(s) */
                  tio->ti->type->release_clone_rq(clone);

  @@ -1973,11 +1889,7 @@ static int map_request(struct dm_rq_target_io *tio, struct request *rq,
                  }
                  if (IS_ERR(clone))
                          return DM_MAPIO_REQUEUE;
  -               if (setup_clone(clone, rq, tio, GFP_ATOMIC)) {
  -                       /* -ENOMEM */
  -                       ti->type->release_clone_rq(clone);
  -                       return DM_MAPIO_REQUEUE;
  -               }
  +               setup_clone(clone, rq, tio);
          }
   
          switch (r) {

-- 
Jun'ichi Nomura, NEC Corporation

[git pull] device mapper fixes for 4.2-rc1

[Mike Snitzer]

On Fri, Jun 26 2015 at  4:05P -0400,
Junichi Nomura <j-nomura@ce.jp.nec.com> wrote:

> On 06/26/15 15:41, Linus Torvalds wrote:
> > On Jun 25, 2015 23:17, "Junichi Nomura" <j-nomura@ce.jp.nec.com <mailto:j-nomura@ce.jp.nec.com>> wrote:
> >> On 06/25/15 23:37, Jens Axboe wrote:
> >> >       block, dm: don't copy bios for request clones
> >>
> >> this change should not be pushed to mainline yet.
> > 
> > Hmm. I merged all of Jens' pull requests today, so it's now in my tree. What do?
> 
> Then I think following 3 commits should be reverted.
> 
> 2d76fff dm: cleanup methods that requeue requests
> cbc4e3c dm: do not allocate any mempools for blk-mq request-based DM
> 5f1b670 block, dm: don't copy bios for request clones
> 
> Reverting 5f1b670 involves conflicts in drvers/md/dm.c for
> these 2 hunks due to diff contexts difference.
> It's ok to revert those changes ignoring the contexts.

That is more than I'd care to put on Linus at this point.  I've prepared
a DM pull request that fixes things up.  Apologies for not pressing this
request-based DM partial completion issue further, it was an oversight
on my part.  We'll have to get it fixed up properly and revisit for a
future release.

Linus,

The following changes since commit e262f34741522e0d821642e5449c6eeb512723fc:

  dm stats: add support for request-based DM devices (2015-06-17 12:40:41 -0400)

are available in the git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm tags/dm-4.2-fixes

for you to fetch changes up to b5451e456840af027b794afc2c7c84c2a17f569b:

  dm cache policy smq: fix "default" version to be 1.4.0 (2015-06-26 10:14:28 -0400)

----------------------------------------------------------------
- Revert block and DM core changes that removed request-based DM's
  ability to handle partial request completions -- otherwise with the
  current SCSI LLDs these changes could lead to silent data corruption.

- Fix two DM version bumps that were missing from the initial 4.2 DM
  pull request (enabled userspace lvm2 to know certain changes have been
  made).

----------------------------------------------------------------
Mike Snitzer (4):
      Revert "dm: do not allocate any mempools for blk-mq request-based DM"
      Revert "block, dm: don't copy bios for request clones"
      dm: bump the ioctl version to 4.32.0
      dm cache policy smq: fix "default" version to be 1.4.0

 block/blk-core.c                 |  94 +++++++++++++++--
 drivers/md/dm-cache-policy-smq.c |   2 +-
 drivers/md/dm-table.c            |  29 ++----
 drivers/md/dm.c                  | 220 +++++++++++++++++++++++++++------------
 drivers/md/dm.h                  |   5 +-
 include/linux/blk_types.h        |   2 -
 include/linux/blkdev.h           |   6 +-
 include/uapi/linux/dm-ioctl.h    |   4 +-
 8 files changed, 256 insertions(+), 106 deletions(-)

Re: [GIT PULL] Core block IO bits for 4.2

[Jens Axboe]

On 06/26/2015 12:15 AM, Junichi Nomura wrote:
> Hi Jens,
> 
> On 06/25/15 23:37, Jens Axboe wrote:
>>        block, dm: don't copy bios for request clones
> 
> this change should not be pushed to mainline yet.
> 
> Firstly, Christoph has a newer version of the patch that fixes silent
> data corruption problem:
>    https://www.redhat.com/archives/dm-devel/2015-May/msg00229.html
> 
> And the new version still depends on LLDDs to always complete requests
> to the end when error happens, while block API doesn't enforce such a
> requirement. If the assumption is ever broken, the inconsistency between
> request and bio (e.g. rq->__sector and rq->bio) will cause silent data
> corruption:
>    https://www.redhat.com/archives/dm-devel/2015-June/msg00022.html

Why was none of that sent to the block tree?

In any case, Linus, pick up Mike's pull request to fix this up.

-- 
Jens Axboe

Re: [GIT PULL] Core block IO bits for 4.2

[Christoph Hellwig]

On Fri, Jun 26, 2015 at 01:22:33PM -0600, Jens Axboe wrote:
> Why was none of that sent to the block tree?
> 
> In any case, Linus, pick up Mike's pull request to fix this up.

I didn't manage to driver the discussion forward, and hadn't really
grasped it was in the block tree.  For some reason I assumed
it only was in Mikes dm WIP tree.